Summary: The authors of this web content discuss their observations while fine-tuning a large language model (LLM) on science exam questions. They found that the LLM was able to rapidly memorize examples from the dataset after seeing them just once, which contradicts prior wisdom about neural network sample efficiency. The authors conducted experiments to validate and understand this phenomenon, and the results indicated that the models are able to rapidly remember inputs. This discovery may require a rethinking of how LLMs are trained and used. The authors also discuss the implications of this fast learning and potential challenges it may present.
https://www.fast.ai/posts/2023-09-04-learning-jumps/