Consider for example the MNIST dataset. When we apply diffusion to the pixel space, the image slowly becomes more and more noisy until white noise has been reached (like below). In the last step (t=100), the image does not really depend on the original number. If we have a well-trained unconditional diffusion model based on the original paper, it will thus generate a random number when starting from this image.
Evidently, there is a clear notion of "noise" and thus the concept of "denoising" makes sense. In the "High-Resolution Image Synthesis with Latent Diffusion Models" paper, they suggest to apply the noising and denoising procedure on a latent space to reduce the number of dimensions and thus the computational cost. Makes sense. But i'm struggling to make sense of the concept of "noise" in this latent space, especially if it is a highly compressed representation of the original pixel space. Consider for example the 2D dimensional latent space for the MNIST dataset below.
If we diffuse for example a sample of the number "8", it will travel the latent space and in the end it might end up in the cluster of another number. With sufficient amount of diffusion, the cluster in which it ends up should be randomly distributed. I suppose that denoising from a random starting point will just travel the clusters too. However, it seems to me that the concept of "noise" does not exist in highly compressed latent spaces - every reasonable value of the latent variables corresponds to a number.
Are diffusion models still beneficial in these highly compressed latent spaces? And are such highly compressed latent spaces common in other imaging applications? Or are they more sparse in general?
