SDEs for Generative ML: A Variational Introduction

Stochastic Differential Equations (SDEs) have become fundamental to significant advancements in generative machine learning, enabling the creation of realistic images, videos, and biomolecules. This paper provides an accessible, informal introduction to the core concepts behind using SDEs in this context. It lays out the necessary differential equations and the probabilistic framework that underpins their application in generative modeling. A key component discussed is the Fokker-Planck equation, which describes the temporal evolution of the marginal distribution of stochastic variables within these differential equations. The paper also derives the variational lower bound on the log-likelihood, known as the Evidence Lower Bound (ELBO), establishing it as a universal starting point for understanding various generative approaches. From this variational perspective, the paper then compares and contrasts popular techniques such as diffusion models, score matching, and flow matching. It illustrates how each of these methods can be understood as specific parameterizations or instantiations of the more general variational framework, using a simple one-dimensional density modeling problem as an example to highlight their differences and similarities.