1
$\begingroup$

The inputs are decayed towards the origin using this formula within Denoising Diffusion Probabilistic Models (DDPMs):

$$q\left(\mathbf{x}_{1: T} \mid \mathbf{x}_0\right):=\prod_{t=1}^T q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right), \quad q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right):=\mathcal{N}\left(\mathbf{x}_t ; \sqrt{1-\beta_t} \mathbf{x}_{t-1}, \beta_t \mathbf{I}\right)$$

I however do not understand how the origin is determined. How is the noise in the forward process or diffusion process computed? In the original DDPM paper it is only stated that:

the forward process variances $β_t$ can be learned by reparameterization [33] or held constant as hyperparameters.

$\endgroup$

1 Answer 1

0
$\begingroup$

You can generate a noise $\epsilon_t$ by sampling a distribution (which you already explicitly stated as $\mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\mathbf{x}_{t-1}, \beta_t\mathbf{I})$) during the sampling phase. The "origin" is the standard normal distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$, which is a constant instead of a variable ground truth to learn.

The variable ground truth to learn (using a U-Net) during the training phase is rather the noise $\epsilon_t$ added during the sampling phase. Reverse to the diffusion (sampling) process, the noise $\epsilon_t$, when subtracted from the noisier $\mathbf{x}_{t}$, can produce a less noisy $\mathbf{x}_{t+1}$. Therefore, given $\mathbf{x}_{t}$, the noise $\epsilon_t$ encodes certain information regarding the input data $\mathbf{x}_{T}$.

$\endgroup$
2
  • $\begingroup$ Can you maybe elaborate on that? I still do not understand how the origin is determined, is it the distribution of the input data point during the specific time point? $\endgroup$ Nov 25, 2022 at 9:03
  • $\begingroup$ @Mariusmarten I edited the answer. Maybe you confuse origin with ground truth. Diffusion models are intuitively different from traditional neural networks due to the additional sampling phase. $\endgroup$
    – lkjsfkshd
    Dec 9, 2022 at 19:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .