Researchers have unveiled MEGALODON, a groundbreaking large language model designed to overcome the limitations of traditional Transformer neural architectures. With chunk-wise attention and parallel sequence-based training, MEGALODON outperforms other models like Llama 2 on various benchmarks. The model boasts linear computational complexity and the ability to handle sequences of unlimited length, showing promise for large-scale multi-modality pretraining. By incorporating a complex exponential moving average mechanism, MEGALODON achieves remarkable improvements in training perplexity and downstream benchmarks. Initial results suggest superior computational efficiency compared to other models, positioning MEGALODON as a leader in the field of language modeling.
https://www.infoq.com/news/2024/06/meta-llm-megalodon/