In this paper, the authors explore the effectiveness of Low-Rank Adaptation (LoRA) compared to full fine-tuning for adapting large language models to various tasks. They analyze weight matrices and spectral properties to show that LoRA introduces “intruder dimensions” not found in full fine-tuning, impacting model generalization and robustness across tasks. Interestingly, higher-rank LoRA models align more closely with full fine-tuning despite using fewer parameters. This study raises questions about the different areas of parameter space accessed by LoRA and full fine-tuning, highlighting the importance of understanding how fine-tuning methods impact model performance and adaptation.
https://arxiv.org/abs/2410.21228