1
$\begingroup$

In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a repository with code here, which gives a few more clues.

Now there is one thing I don't particularly understand, and that is that although the KL divergence is taken over $t=2..T$, enter image description here

in the code, they sample $t=1..T-1$ and say in the comments,

# choose a timestep in [1, self.trajectory_length-1].
# note the reverse process is fixed for the very
# first timestep, so we skip it.

Now if I understand correctly, with the first timestep of the reverse process is just $t=T$, which is just the isotropic gaussian, but what I don't understand is, why do they sample from $t=1$ instead of $t=2$ like in the KL divergence.

Also, if you were indeed to sample from $t=2$, how do you learn $f_{\mu}(x^1,1),f_{\Sigma}(x^1,1)$ so that you can reverse the last step? I would expect that to come form the entropy $H_q(X^{(1)}|X^{(0)})$, but from the code we see that they replace that with the entropy of a Gaussian with $\sigma=\sqrt{1 - a_1} = \sqrt{b_1}$, $$ H_q(X^{(1)}|X^{(0)}) = \frac{1}{2}(\log 2 \pi + 1) + \frac{1}{2}\log b_1 $$

In the reverse process, you also see that they don't handle the last step of the reverse process any differently.

Summarising, how can they sample $t \in 1..T-1$, and calculate the KL divergence, while the equation specifies that $t \in 2..T$?

EDIT: After analysing the code further, it looks like they are not learning the last denoising step, and are only generating $x^{(1)}$.

$\endgroup$
0

1 Answer 1

0
$\begingroup$

I didn't watch the code but let me have a guess: For the reverse process, the last step from x(1) to x(0) should not add any noise respecting to f_sigma. Because this is the last step so adding any noise is not reasonable. So, just take x(1) as the final output is good enough to end the process.

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jul 3, 2023 at 15:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .