Diffusion models are currently one of the most powerful types of tools for generative tasks that require complex and local structures, such as image generation and molecular discovery. They’ve shown an exciting capability to generalize beyond their training data and, in this sense, exhibit “creativity”. For instance, after being trained with datasets of actual images, they can transform random noise samples into novel, high-quality images.While this creative capability is impressive, it raises an intriguing question: where does it come from? Understanding the answer to this question is an important step towards demystifying the “black-box” nature of diffusion-based generative AI.To that end, in "On the Interpolation Effect of Score Smoothing in Diffusion Models", presented at ICLR 2026, we dive into the mathematics of diffusion models to answer this question. We show that a model’s creativity isn’t a random fluke. Instead, it is a consequence of how neural network training naturally "smooths" the transformation from noise back to the data during the generation process.
Understanding denoising
Training a diffusion model begins with taking a set of real training data samples — like cat photos — and intentionally corrupting them with noise until they become completely unrecognizable. The model is then trained to reverse this corruption step-by-step so that it can reconstruct a realistic-looking image from pure noise, a process called denoising.If the model learns to perform this denoising process perfectly based only on its training samples, it should produce carbon copies of them during deployment time as well (a behavior known as memorization). In this scenario, the model acts as a retrieval tool rather than as a creative engine capable of generating novel outputs.In practice, however, diffusion models usually do more than just memorize; they generalize to generate new data samples.To understand how diffusion models actually denoise data, imagine random noise as a cloud of gas particles scattered across a room, where a “force field” pulls each particle in a specific direction until they form a meaningful shape. In a diffusion model, the moving particles are individual data points undergoing denoising. The “force field” is the score function (SF), which is learned from the training data and dictates where the particles should flow at any given moment.If the model relies on a score function learned perfectly from the training data, the force field will drive the particles into positions that exactly replicate the training data points (i.e., memorization).
