The blog post delves into Energy-Based Models (EBMs) for generative modeling, emphasizing their unique approach of framing the problem in terms of an energy function. EBMs assign low energy to likely data and high energy to unlikely data, guiding the generation of new samples. The post explains the training of EBMs through approaches like Contrastive Divergence, Score Matching, and Noise Contrastive Estimation. Practical tips and challenges in training EBMs are outlined, highlighting issues like poor negatives, unregularized energy functions, and slow computation. The author envisions Extropic’s hardware-based encoding of energy functions as a potential revolutionary advancement in probabilistic AI.
https://mpmisko.github.io/2024/ai-fundamentals-energy-based-models/