In this study, the effectiveness of Chain-of-thought (CoT) prompting in improving model performance is explored across various tasks. By drawing inspiration from cognitive psychology, the authors identify scenarios where verbal thinking hinders human performance and investigate if the same applies to language models. Surprisingly, they find that in certain cases, CoT actually increases model performance, despite verbal thinking decreasing human performance. Results show that inference-time reasoning can significantly reduce model accuracy in some tasks, highlighting the importance of prompt choices. This research provides new insights into the impact of CoT and inference-time reasoning on model performance, bridging the gap between human and model cognition.
https://arxiv.org/abs/2410.21333