Llemma is a remarkable language model designed specifically for mathematics. Developed by pretraining Code Llama on a mix of scientific papers, web data with mathematical content, and mathematical code, Llemma outperforms all existing open base models on the MATH benchmark, as well as the unreleased Minerva model suite. What’s even more impressive is that Llemma can perform tool use and formal theorem proving without any additional finetuning. The creators of Llemma are generously sharing all the resources associated with their work, including the 7 billion and 34 billion parameter models, the Proof-Pile-2 dataset, and the code for replicating their experiments.
https://arxiv.org/abs/2310.10631