0
$\begingroup$

I had come to think of VAEs as having 2 loss terms - A reconstruction loss that tries to ensure that the input matches the output (just as a regular Auto Encoder), and a KL-Divergence Loss term that tries to ensure each training batch is mapped to a smaller dimensional space where the data being mapped is normally distributed at the latent encoding layer with a std normal distribution of $\mu$=0 and $\sigma^2$=1. The output of the encoder is a $\mu$ vector and a $\sigma^2$ vector.

But, I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the latent encoding layer. So, if our data has an $\mathcal{N}(0,1)$ distribution and we then add exactly this type of noise to each sample in the representation layer, why do we expect successful reconstruction? It seems like we've added a lot of noise, given the region over which our data is distributed.

Is it just because on 'average' no noise has been added?

Why not train by just letting the kl-loss term insist that each batch have $\mu$=0 and $\sigma^2$=1 and only take a $\mathcal{N}(0,1)$ sample from the hidden layer when we actually do want a new hidden sample?

$\endgroup$
1
  • $\begingroup$ You claimed "I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the hidden layer", in what reference you noticed this unit Gaussian noise is added at the hidden layer? Encoder or decoder's hidden layer? $\endgroup$
    – cinch
    Commented Mar 29 at 16:38

1 Answer 1

2
$\begingroup$

You can just insist each training batch is normally distributed—this is the main idea behind normalizing flows, where you try to maximize the likelihood of sampling the encoding assuming it follows a normal distribution.

People mostly use VAEs because

  1. VAEs became popular first, which is because
  2. Normalizing flows require more math/numerical methods, and
  3. Take more compute to train.

Unfortunately, (1) has gotten to the point where everyone is using VAEs, simply because everyone else is! Research and compute is slowly making normalizing flows more viable. For example, you can pair them with neural ODEs for smoother, dynamic flows, but (as of March 2024) you'd likely have to write your own GPU kernels to get an efficient ODE solver.

A recent success with normalizing flows is voice cloning: you can flow from an arbitrary voice to the international phonetic alphabet, which removes timbre. Flowing backward lets you clone someone else's voice.

Now, back to VAEs. You're absolutely correct that you should not expect perfect reconstruction after adding noise. The issue with traditional encoders is they may get perfect reconstruction, but only for your sample of images. For example, perhaps the encoder only needs to signal "this is a horse", and because there is only one horse in your dataset the decoder can perfectly reconstruct it. The point of the noise term in VAEs is to splat out your distribution so the representation has to actually be about the image. See this previous StackExchange answer.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .