Optimizing the performance of deep learning models can be a challenging task. Many users resort to ineffective ad-hoc methods, resulting in performance that feels more like alchemy than science. However, by reasoning from first principles and understanding the components that contribute to performance, it becomes easier to identify optimizations that matter. Three key components to consider are compute, memory bandwidth, and overhead. Maximizing compute while reducing memory and overhead costs is crucial. Operator fusion, which eliminates unnecessary memory transfers, is a valuable optimization technique. Memory bandwidth costs, along with overhead, can significantly impact performance. By measuring compute intensity and identifying bottlenecks, it becomes possible to improve GPU efficiency and achieve better performance.
https://horace.io/brrr_intro.html