# What is the exact role of model $p_\theta$ in diffusion models for the reverse process?

I'm reading this interesting blog post explaining diffusion probabilistic models and trying to understand the following.

In order to compute the reverse process, we need to consider the posterior distribution $$q(\textbf{x}_{t-1} | \textbf{x}_t)$$ which is said to be intractable*

because it needs to use the entire dataset and therefore we need to learn a model $$p_\theta$$ to approximate these conditional probabilities in order to run the reverse diffusion process.

If we use Bayes theorem we have

$$q(\textbf{x}_{t-1} | \textbf{x}_t) = \frac{q(\textbf{x}_t |\textbf{x}_{t-1})q(\textbf{x}_{t-1})}{q(\textbf{x}_t)}$$

I understand that indeed we don't have any prior knowledge of $$q(\textbf{x}_{t-1})$$ or $$q(\textbf{x}_t)$$ since this would mean already having the distribution we are trying to estimate. Is this correct?

The above posterior becomes tractable when conditioned on $$\textbf{x}_0$$ and we obtain

$$q(\textbf{x}_{t-1} | \textbf{x}_t , \textbf{x}_0) = \mathcal{N}(\tilde{\bf{\mu}}(\textbf{x}_t , \textbf{x}_0) \, , \, \tilde{\beta}_t \textbf{I})$$

So, apparently, we obtain a posterior that can be calculated in closed form when we condition on the original data $$\textbf{x}_0$$. At this point, I don't understand the role of the model $$p_\theta$$ : why do we need to tune the parameters of a model if we can already obtain our posterior?

• Don't know much about diffusion models, but I think this iterative conversion of $\mathbf{x}_0$ to a gaussian is only being done to then train a model to reverse the process. So in my understanding, the whole idea is to not use $\mathbf{x}_0$, otherwise, why would you want to add noise to $\mathbf{x}_0$ in the first place? This might be comparable to the encoding and decoding process of variational autoencoders. The encoding makes the input noisier and the decoder reconstructs the input. Here, you only condition the encoder on $\mathbf{x}_0$, but not the decoder directly. Jul 3 at 10:53
• @Chillston Thank you greatly for your answer! So when we are optimizing the lower bound on the negative log-likelihood even there we are not using $\mathbf{x}_0$ correct? Since I have you here maybe you can also tell me the intuition behind setting up a variational inference problem with a fixed approximate posterior, I thought the whole point of variational inference was to allow the posterior to belong to a big enough (well-behaved) class so as to get as close as possible to the true posterior in an efficient way. Jul 3 at 12:10
• I hope I understand you right, I really didn't look into diffusion models too deep, yet. How I understand this is: You ultimately want a model that denoises a given random sample and maps it onto the manifold of your data. Because it is not feasible to model $p_{\Theta} : X_T \rightarrow X_0$ directly, the model learns a single denoising step $p_{\Theta} : X_{t} \rightarrow X_{t+1}$ ($0 \leq t \leq T$). By iteratively applying the model you arrive at $X_0$ after $T$ iterations. Thus given a noisy sample $x_t$, the model predicts the noise that was added to $\mathbf{x}_{t-1}$. Jul 3 at 15:36
• So the (simplified) training objective is actually the MSE between predicted and actual noise (s. Equation 14 Ho et al.). So I'd say you need $\mathbf{x}_0$ to optimize the variational bound. Jul 3 at 15:37
• Regarding the second part: In my intuition, having a fixed posterior doesn't limit diversity in the case of denoising models (if that's what you mean), because you never specifically train the model to map $\mathbf{x}_T$ to $\mathbf{x}_0$. Instead, you are training it to slightly improve the quality of corrupted samples. This denoising itself is stochastic. Therefore over multiple denoising steps, the outcome can vary greatly. For an example, see Figure 7 in the Ho et al. paper. Jul 3 at 15:38