In this groundbreaking study, we tackle the challenge of optimizing large-scale machine learning models by introducing a novel approach using metagradients. Our algorithm efficiently calculates metagradients, allowing for improved model performance through smooth model training. With metagradient descent (MGD), we surpass existing dataset selection methods, resist accuracy-degrading data poisoning attacks, and automatically optimize learning rate schedules. This new method revolutionizes the training process, offering promising advancements in the field of machine learning.
https://arxiv.org/abs/2503.13751