In this paper, we challenge the notion that deep neural networks exhibit unique generalization behavior compared to other model classes. We discuss phenomena such as benign overfitting, double descent, and the success of overparametrization, arguing that these are not exclusive to neural networks. By incorporating soft inductive biases, we can better understand and characterize generalization behavior across different model classes. While deep learning has distinct qualities like representation learning and mode connectivity, it is not as mysterious or different from other models as commonly believed. This paper sheds light on the unifying principles that govern generalization in machine learning models.
https://arxiv.org/abs/2503.02113