# Reverse Distribution in Denoising Diffusion Models is Simple

In explanations of denoising diffusion models it is stated that $$q(x_{t-1}|x_t)$$ is intractable. This is often justified via Bayes' rule, i.e. $$q(x_{t-1}|x_t) \propto q(x_t|x_{t-1})q(x_{t-1})$$ and the marginal $$q(x_{t-1})$$ is unknown. But I'm confused. We know that $$x_t = \sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}E,\quad E\sim\mathcal{N}(0,I)$$ therefore we can solve this equation for $$x_{t-1}$$: \begin{align} x_{t-1} &= (1-\beta_t)^{-1/2}x_t - \sqrt{\frac{\beta_t}{1-\beta_t}}E \\ &= (1-\beta_t)^{-1/2}x_t + \sqrt{\frac{\beta_t}{1-\beta_t}}R,\quad R\sim\mathcal{N}(0,I). \end{align} Thus $$q(x_{t-1}|x_t) = \mathcal{N}(x_{t-1};(1-\beta_t)^{-1/2}x_t,\frac{\beta_t}{1-\beta_t}I).$$ This is simple as can be. If this is true, there is no point in parameterizing the reverse distribution with neural nets and we don't need $$\beta_t$$ to be small etc. What am I missing?

• Hi, did you end up finding the solution to this? Mar 19, 2023 at 19:48

while we set $$R$$ to be independent from $$x_{t-1}$$ in the calculation of $$x_t$$, we no long have independency between $$x_t$$ and $$R$$, and I guess this gives rise to the confusion.

For simplicity, let us consider two independent normal variable $$x, \varepsilon \sim \mathcal{N}\left(0, 1\right)$$, and let us set $$y = ax + b\varepsilon$$. Then we have $$y|x = \mathcal{N}\left(y; ax, b^2\right),$$ (For more detail on conditional normal, see this link: https://statproofbook.github.io/P/mvn-cond) which is similar to your conclusion.

But we have $$x|y = \mathcal{N}\left(x; \frac{ay}{a^2 + b^2}, \frac{b^2}{a^2 + b^2}\right)$$

(You may also find this link: https://www.statlect.com/probability-distributions/normal-distribution-linear-combinations helpful in computing the linear combination of multivariable normal)

Consider the simple case where $$x_{t-1} = c$$ is deterministic and $$\beta > 0$$. Then the forward process is given by $$q(x_t|x_{t-1}) \sim \mathcal{N}(x_t; \sqrt{1 - \beta}c, \beta I)$$ But the backward process would definitely not be given by $$q(x_{t-1}|x_t) \sim \mathcal{N}(x_t; (1 - \beta)^{-1/2}x_t, \frac{\beta}{1 - \beta} I)$$ as $$x_{t-1}$$ is deterministic.

Of course ironically in the example I presented, the backward process is trivial, but it shows you can't just "solve for $$x_{t-1}$$".