1
$\begingroup$

I've been studying diffusion models and came across the noise schedule, particularly how the noise variance $\beta_t$ is adjusted over iterations. I've observed that $\beta_t$ typically starts from a very small value during the initial steps and increases to a much larger value in the final steps. What is the underlying reason for this progression? Why is it important to begin with a low noise variance and end with a high one? I'd appreciate any intuitive explanations or references that shed light on this choice.

$\endgroup$

2 Answers 2

0
$\begingroup$

mainly because it works, it makes learning the denoising process stable, because at each step the added noise is not too much, it makes it easy for the AI to figure out the changes that occurred the try and remove the added noise, plus the changes in its weight won't be large for each denoising step, am pretty sure you have been in a math class and have had to simplify some complex equation, imagine your teacher shows you the complex long equation, and then he just writes the simplified equation under it without showing the steps ? .... yeah, that will make it hard for you to learn anything, you can still figure it out because you are human, and your teacher already taught you about BODMAS and some rules of math but still it will be difficult, most diffusion are just that, diffusion models, they don't have human level reasoning etc.

Even for humans, when changes happen rapidly, it makes it hard to predict what actually happened. Imagine how hard it will be for you if you haven't learnt about BODMAS and stuff and you are exposed to math expression and the final solution? Gradual steps lead to better learning in humans, it works for ai too.

Yeah, so you begin with small steps, hopefully during the small steps the ai learns the fundamental rules of the operations, which is equivalent to a teacher showing each step because you are new to the material, then after you have some familiarity, teachers begin to skip steps and sometimes don't show steps at all.

$\endgroup$
0
$\begingroup$

Let me provide an intuitive explanation. During the forward diffusion process in the denoising diffusion model, the variance of the noise needs to be increased to maintain a relatively constant Signal-to-Noise Ratio (SNR). This is probably because a stable SNR makes it easier for the network to learn denoising.

You can think of SNR as the ratio between the signal's variance and the variance of the added Gaussian noise. As we progressively add noise to the image, the SNR naturally decreases. To counteract this decrease and keep the SNR stable, we need to increase the variance of the noise added in the later stages of the process.

New contributor
lxfhfut is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .