# About cosine noise schedule in Diffusion Model

Could you provide the explanation of Figure 4 from the paper at Improved Denoising Diffusion Probabilistic Models?

(1) The paper says, "the end of the forward noising process is too noisy,and so doesn’t contribute very much to sample quality". But if the goal is to have an image with only noise, why is it problematic to have a lot of noise?

(2) The paper says, "a model trained with the linear schedule does not get much worse (as measured by FID) when we skip up to 20% of the reverse diffusion process". But why is it the reverse diffusion process? Shouldn't the deterioration be related to the forward process?

(3) Also, why is the training process relevant?

It seems to me that the explanation of this section means 'linear noise schedule is better than cosine one'. Could you explain, please?

I will suppose that you already have understood how diffusion models work. Some good resources are this blog and the DDPM paper.

If we look at Figure 3 of the paper, we see that in linear schedule the image are almost purely noise in the last quarter. During sampling, we usually perform the same number of steps the model was trained on, but in the linear schedule the reverse process will just turn some random noise into some other random noise for 1/4 of the time.

1. If those steps are irrelevant in terms of image quality (estimated using FID), it means that we are just wasting time (and a bunch precious energy to run GPUs)
2. The choice of the noise schedule used during training, i.e the degree with which the image is gradually turned into noise, will affect the performance (measuring the quality of generated images during sampling) of the diffusion model. If the performance at test time does not change much when skipping training steps, this means that the model has learned nothing for a fraction of his training steps, so the authors aim to find a schedule where each training step positively impacts the image quality at test time.
3. Another proposed solution of this paper is in the learned variances $$\Sigma_\theta(x,t)$$. The original paper sets $$\Sigma_\theta(x,t)= \sigma^2 I$$, while this paper proposes an interpolation $$\Sigma_\theta(x,t)= \exp(v \log \beta_t + (1-v) \log \tilde{\beta_t})$$ In figure 8 they show that with this modification the number of sampling steps can be reduced without losing image quality (measured by FID). All but $$\Sigma_\theta(x,t)= \beta^2 I$$ obtain the same performance in 400 vs 4K denoising steps.
• Thank you for reply. I understand about the Figure 8. On the other hand , does Figure 4 mean that the cosine schedule gets much worse when skipped up to 20%? Commented Jul 8, 2023 at 1:15
• In the case of Figure 4, does it mean cosine schedule is not good? Commented Jul 8, 2023 at 1:22
• @diffusionstable no, the cosine schedule is good because intuitively every denoising step is useful to produce a better image, whereas in linear schedule many steps do not contribute in improving the final result. Commented Jul 8, 2023 at 6:43
• I'm sorry to append another question, but since a higher FID value indicates lower accuracy, regarding the FID figures, it is written that even if the reverse diffusion process is skipped by 20%, the linear schedule does not deteriorate. What does this mean? (It seems the linear schedule is good.) Commented Sep 18, 2023 at 6:05
• @diffusionstable A Higher FID indicates a lower image quality. Skipping steps while mantaining the same FID is a nice property during sampling, but not during training. If some (up to 1/4th, in this case) of the steps were not useful during training, the model will likely produce lower quality images, compared to a model trained with a schedule where each step is useful for learning. You can observe this by looking at table 1, where the linear schedule achieves a higher test FID compared to the cosine schedule. Commented Sep 21, 2023 at 14:10

Intuitively, the noise schedule shall be smooth and progressive enough to be easily approximated by the NN. In this context, a linear schedule is plausibly too "easy", and hence prone to overfitting.

In contrast, the modulation of the cosine schedule provides a slightly more challenging task to the NN (yet remaining smooth and gradual), so that when the reverse process is learnt, it is more robust.

• Thank you for relay. I understand the overfitting. I see. Commented Jul 9, 2023 at 10:46