This research delves into the crucial aspect of enhancing the reasoning capabilities of large language models (LLMs) by exploring their geometrical understanding. The study establishes a correlation between the expressive power of LLMs and the density of their self-attention graphs, with higher density indicating a greater intrinsic dimension and thus increased expressive capacity. Theoretical analysis and toy examples support this connection, along with empirical evidence linking this geometric framework to recent advancements in methods aimed at improving LLMs’ reasoning abilities. This work sheds light on the importance of geometric properties in enhancing the performance of LLMs in real-world applications.
https://arxiv.org/abs/2407.02678