In this web content, the authors discuss the challenges of deploying large language models (LLMs) for real-world applications due to their size and computational requirements. They highlight the use of smaller specialized models trained through fine-tuning or distillation methods as a workaround. The authors then introduce their new mechanism called distilling step-by-step, which extracts informative natural language rationales from LLMs to train smaller task-specific models in a more data-efficient way. They demonstrate that this mechanism enables smaller models to outperform larger LLMs using less training data, showcasing a significant reduction in model size and training requirements. The authors provide experimental results and comparisons to illustrate the effectiveness of their approach.
https://blog.research.google/2023/09/distilling-step-by-step-outperforming.html