I am currently learning about latent diffusion by understanding its implementation.

For a standard diffusion model, the denoiser accepts a normalized image during training. During sampling at every timestep, the denoiser predicts a less-noisy image and clamps the pixel values to the normalized range to ensure that it remains valid.

However, for latent diffusion models, an autoencoder is used to convert a normalized image into a reduced latent space. After the encoding, the output latent may no longer be within the normalized range. This encoded image is then fed into the denoiser for latent diffusion. Since the latent is not normalized, clamping it to a specific range during sampling wouldn't make sense. However, if we do not clamp the latent, it might be possible for it to explode into something invalid.

As such, I am wondering if the output latent from the autoencoder has to be re-normalized again before being fed into the denoiser model. If so, how can we determine the extreme values to normalize the latent?

Any thoughts on this are appreciated.



You must log in to answer this question.

Browse other questions tagged .