6
$\begingroup$

I'm reading this interesting blog post explaining diffusion probabilistic models and trying to understand the following.

In order to compute the reverse process, we need to consider the posterior distribution $q(\textbf{x}_{t-1} | \textbf{x}_t)$ which is said to be intractable*

because it needs to use the entire dataset and therefore we need to learn a model $p_\theta$ to approximate these conditional probabilities in order to run the reverse diffusion process.

If we use Bayes theorem we have

$$q(\textbf{x}_{t-1} | \textbf{x}_t) = \frac{q(\textbf{x}_t |\textbf{x}_{t-1})q(\textbf{x}_{t-1})}{q(\textbf{x}_t)}$$

I understand that indeed we don't have any prior knowledge of $q(\textbf{x}_{t-1})$ or $q(\textbf{x}_t)$ since this would mean already having the distribution we are trying to estimate. Is this correct?

The above posterior becomes tractable when conditioned on $\textbf{x}_0$ and we obtain

$$q(\textbf{x}_{t-1} | \textbf{x}_t , \textbf{x}_0) = \mathcal{N}(\tilde{\bf{\mu}}(\textbf{x}_t , \textbf{x}_0) \, , \, \tilde{\beta}_t \textbf{I})$$

So, apparently, we obtain a posterior that can be calculated in closed form when we condition on the original data $\textbf{x}_0$. At this point, I don't understand the role of the model $p_\theta$ : why do we need to tune the parameters of a model if we can already obtain our posterior?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Don't know much about diffusion models, but I think this iterative conversion of $\mathbf{x}_0$ to a gaussian is only being done to then train a model to reverse the process. So in my understanding, the whole idea is to not use $\mathbf{x}_0$, otherwise, why would you want to add noise to $\mathbf{x}_0$ in the first place? This might be comparable to the encoding and decoding process of variational autoencoders. The encoding makes the input noisier and the decoder reconstructs the input. Here, you only condition the encoder on $\mathbf{x}_0$, but not the decoder directly. $\endgroup$
    – Chillston
    Jul 3, 2022 at 10:53
  • 1
    $\begingroup$ @Chillston Thank you greatly for your answer! So when we are optimizing the lower bound on the negative log-likelihood even there we are not using $\mathbf{x}_0$ correct? Since I have you here maybe you can also tell me the intuition behind setting up a variational inference problem with a fixed approximate posterior, I thought the whole point of variational inference was to allow the posterior to belong to a big enough (well-behaved) class so as to get as close as possible to the true posterior in an efficient way. $\endgroup$
    – Monolite
    Jul 3, 2022 at 12:10
  • $\begingroup$ I hope I understand you right, I really didn't look into diffusion models too deep, yet. How I understand this is: You ultimately want a model that denoises a given random sample and maps it onto the manifold of your data. Because it is not feasible to model $p_{\Theta} : X_T \rightarrow X_0$ directly, the model learns a single denoising step $p_{\Theta} : X_{t} \rightarrow X_{t+1}$ ($0 \leq t \leq T$). By iteratively applying the model you arrive at $X_0$ after $T$ iterations. Thus given a noisy sample $x_t$, the model predicts the noise that was added to $\mathbf{x}_{t-1}$. $\endgroup$
    – Chillston
    Jul 3, 2022 at 15:36
  • $\begingroup$ So the (simplified) training objective is actually the MSE between predicted and actual noise (s. Equation 14 Ho et al.). So I'd say you need $\mathbf{x}_0$ to optimize the variational bound. $\endgroup$
    – Chillston
    Jul 3, 2022 at 15:37
  • $\begingroup$ Regarding the second part: In my intuition, having a fixed posterior doesn't limit diversity in the case of denoising models (if that's what you mean), because you never specifically train the model to map $\mathbf{x}_T$ to $\mathbf{x}_0$. Instead, you are training it to slightly improve the quality of corrupted samples. This denoising itself is stochastic. Therefore over multiple denoising steps, the outcome can vary greatly. For an example, see Figure 7 in the Ho et al. paper. $\endgroup$
    – Chillston
    Jul 3, 2022 at 15:38

2 Answers 2

Reset to default
2
$\begingroup$

I am also learning diffusion models and would like to give some information.

At this point, I don't understand the role of the model $p_\theta$

To clear a bit: $p_\theta$ is just another annotation for U-net and the role is receiving ($x_t$,$t$) (sometimes also receives classifier $y$) and predicts $x_0$ OR $x_{t-1}$ depending on different papers. So at the end of the day, to synthesize new data, given a noisy (usually Gaussian) image, U-net can iteratively predict $x_0$ better - check out algorithm 2 in the DDPM paper (2020).

Your question about the posterior might be answered in more detail here: Diffusion Models | Paper Explanation | Math Explained - YouTube

Check video time around 18:00 that explains a bit more information regarding $x_0$ guided process in the optimization of lower boundary.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Why is noise added at each timestamp during the reverse process? $\endgroup$
    – Nathan B
    Jan 24 at 9:15
  • $\begingroup$ @NathanB I am assuming you mean the DDIM diffusion way. DDIM is predicting x_0 and adding the noise back to x_t-1 during reverse. From my understanding, this strategy is turning the diffusion process from markovian to non-markovian. The benefits of DDIM include faster generation, better sampling strategy, etc. I would recommend the cold diffusion paper and the original DDIM paper for a deeper understanding. $\endgroup$
    – lqi
    Jan 25 at 17:34
  • $\begingroup$ What will happen if you don't add some noise in the reverse process at every step? After all, you start from a random noise image, so you already have the randomness needed for the sampling, why add more noise every step backwards? $\endgroup$
    – Nathan B
    Jan 26 at 11:32
  • 1
    $\begingroup$ @NathanB you can do it without adding noise during sampling. It's just different sampling strategies. In the original DDPM paper, they are not adding noise when sampling backward. $\endgroup$
    – lqi
    Jan 27 at 18:48
  • 1
    $\begingroup$ @NathanB I can't give you the conclusion that it's adding more "randomness" to be more "creative" as I haven't fully researched it enough. However, the main purpose includes changing non-markovian to markovian, which allows faster generation. Different sampling strategies have different pros and cons, I highly suggest reading papers like cold diffusion, stable diffusion, ILVR, etc. to get a deeper intuition of those sampling strategies. $\endgroup$
    – lqi
    Jan 30 at 2:20
0
$\begingroup$

You do not yet have $\mathbf{x}_0$ during sampling (not training). That's why you need to approximate $q(\mathbf{x}_{t−1}|\mathbf{x}_t, \mathbf{x}_0)$ with $p_{\theta}(\mathbf{x}_{t−1}|\mathbf{x}_t)$ via variational inference such as KL divergence. After training with good data, this should produce an approximation of $\mathbf{x}_0$.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .