In-depth understanding of formulation and guidance mechanisms in Diffusion models

Question

I've been reading a research paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. and came across an a concept related to diffusion models (DMs). In the abstract, the authors state:

By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations.

I'm trying to gain a deeper understanding of this highlighted part, particularly how the formulation of DMs allows for a guidance mechanism that controls the image generation process without the need for retraining.

Can anyone suggest detailed resources, or perhaps elaborate on the mechanism through which this control is achieved in DMs?

Answer 1

You can think as if the network learns the gradient of the data distribution...

For example, think about having some points in 1D which are distributed as 2 Gaussians:

Learning the gradient means that if you start from a low probability mass area, you know in what direction you have to go, by performing: $$ x_{t+1} = x_t + \alpha \nabla_x p(x) $$

For example:

the red arrow is the gradient, saying that if you want to go to an are that has higher probability mass, you have to go on the right, so you move your dot slightly on the right

Now, this was in a 1D very simplistic scenario, but the same applies to DM, you start with a "prior" and then from there you learn in which direction you have to move your image to get a more likely (which in out case means "more realistic") sample

As a resource, I would suggest you to look at this blogpost: https://lilianweng.github.io/posts/2021-07-11-diffusion-models/