When we define the loss function of a variational autoencoder (VAE), we add the Kullback-Leibler divergence between the sample taken according to a normal distribution of parameters:
$$ N(\mu,\sigma) $$
and we compare it with a normal distribution of parameters
$$ N(0,1) $$
My intuition is that it is clever having samples taken from a distribution centered around zero, but I don't understand why we want that examples are taken with a normal distribution.
If you are mathematically inclined, here is an article that discusses the reasoning.
What I get as a take away is that the VAE forces the learned latent space to be Gaussian due to the KL divergence term in the loss function. So, now we have a known distribution to sample from to create input vectors to feed to the decoder, to produce say images of dogs, if the VAE was trained on images of dogs. As you sample from the distribution, you will produce images of different types of dog images.
I assume a different type of distribution could be selected if one uses the proper loss function for that type of distribution, that is, the loss function which would measure the difference in the distribution of the latent space and the desired distribution.
KL divergence is the loss function that forces the latent space distribution to be Gaussian. If you do not "restrict" the latent space, as is the case with a regular autoencoder, you have no idea what kind of vector to select as an input to the decoder to produce a dog image. Without restriction, there are $2^n$ (where $n$ is the dimensions of the latent space) possible vectors you could choose from. Chances of selecting one that produces a dog image would be minuscule.
Well, I hope this helps. I no longer am mathematically proficient (age 75), so I hope my interpretation of the math is correct.
A VAE tends to produce blurry images because there are two terms in the loss function. One term is trying to make the output look like the input while the KL loss term is trying to restrict the latent space distribution. GANs ( generative adversarial networks) don't have this conflict, so they produce much high-quality images.
