In 2024, AI saw significant advancements in text and image generation, with various labs challenging OpenAI’s dominance. Large Language Models (LLMs) remain a major focus, with new scaling paradigms and architectures like Dense Transformers gaining traction. Mixture-of-Experts models have resurfaced, showing promise but posing challenges in deployment. Tokenization methods are evolving, with Byte Latent Transformers emerging as a promising solution. The field of reasoning has seen notable progress, driven by models like o1 and o3. Image generation has also seen remarkable improvements, with labs innovating on the Diffusion Transformer framework. Multimodal models are gaining popularity, with early-fusion approaches showing potential. Looking ahead to 2025, the AI community anticipates further advancements in generative models and innovative research directions.
https://nrehiew.github.io/blog/2024/