Reverse Distribution in Denoising Diffusion Models is Simple

Question

In explanations of denoising diffusion models it is stated that $q(x_{t-1}|x_t)$ is intractable. This is often justified via Bayes' rule, i.e. $$ q(x_{t-1}|x_t) \propto q(x_t|x_{t-1})q(x_{t-1}) $$ and the marginal $q(x_{t-1})$ is unknown. But I'm confused. We know that $$ x_t = \sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}E,\quad E\sim\mathcal{N}(0,I) $$ therefore we can solve this equation for $x_{t-1}$: \begin{align} x_{t-1} &= (1-\beta_t)^{-1/2}x_t - \sqrt{\frac{\beta_t}{1-\beta_t}}E \\ &= (1-\beta_t)^{-1/2}x_t + \sqrt{\frac{\beta_t}{1-\beta_t}}R,\quad R\sim\mathcal{N}(0,I). \end{align} Thus $$ q(x_{t-1}|x_t) = \mathcal{N}(x_{t-1};(1-\beta_t)^{-1/2}x_t,\frac{\beta_t}{1-\beta_t}I). $$ This is simple as can be. If this is true, there is no point in parameterizing the reverse distribution with neural nets and we don't need $\beta_t$ to be small etc. What am I missing?

Answer 1

while we set $R$ to be independent from $x_{t-1}$ in the calculation of $x_t$, we no long have independency between $x_t$ and $R$, and I guess this gives rise to the confusion.

For simplicity, let us consider two independent normal variable $x, \varepsilon \sim \mathcal{N}\left(0, 1\right)$, and let us set $y = ax + b\varepsilon$. Then we have $$ y|x = \mathcal{N}\left(y; ax, b^2\right), $$ (For more detail on conditional normal, see this link: https://statproofbook.github.io/P/mvn-cond) which is similar to your conclusion.

But we have $$ x|y = \mathcal{N}\left(x; \frac{ay}{a^2 + b^2}, \frac{b^2}{a^2 + b^2}\right) $$

(You may also find this link: https://www.statlect.com/probability-distributions/normal-distribution-linear-combinations helpful in computing the linear combination of multivariable normal)

Answer 2

Consider the simple case where $x_{t-1} = c$ is deterministic and $\beta > 0$. Then the forward process is given by $$q(x_t|x_{t-1}) \sim \mathcal{N}(x_t; \sqrt{1 - \beta}c, \beta I)$$ But the backward process would definitely not be given by $$q(x_{t-1}|x_t) \sim \mathcal{N}(x_t; (1 - \beta)^{-1/2}x_t, \frac{\beta}{1 - \beta} I)$$ as $x_{t-1}$ is deterministic.

Of course ironically in the example I presented, the backward process is trivial, but it shows you can't just "solve for $x_{t-1}$".