Large language models have expanded into various domains in human lives by pretraining on massive amounts of language data. Chain-of-Thought method helps extract accurate responses by guiding models to generate solutions step-by-step. However, language constraints on reasoning hints that AI does not need to translate thoughts into words like humans. Meta’s paper introduces COCONUT, a new method allowing large language models to reason in a continuous latent space, improving on word-based reasoning. COCONUT alternates between language mode and latent thought mode, enhancing reasoning capabilities. Experimental results show Coconut outperforming traditional methods on planning-intensive tasks, hinting at BFS-like reasoning. Future research directions include pretraining language models with continuous thoughts and optimizing Coconut efficiency.
https://aipapersacademy.com/chain-of-continuous-thought/