Why does the variational auto-encoder use the reconstruction loss?

Question

VAE is trained to reduce the following two losses.

1. KL divergence between inferred latent distribution and Gaussian.

2. the reconstruction loss

I understand that the first one regularizes VAE to get structured latent space. But why and how does the second loss help VAE to work?

During the training of the VAE, we first feed an image to the encoder. Then, the encoder infers mean and variance. After that, we sample $z$ from the inferred distribution. Finally, the decoder gets the sampled $z$ and generates an image. So, in this way, the VAE is trained to make the generated image to be equal to the original input image.

Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.

But, as you know, VAE works well. So I think I miss something important or understand it in a totally wrong way.

Answer 1

The VAE uses the ELBO loss, which is composed of the KL term and the likelihood term. The ELBO loss is a lower bound on the evidence of your data, so if you maximize the ELBO you also maximize the evidence of the given data, which is what you indirectly want to do, i.e. you want the probability of your given data (i.e. the data in your dataset) to be high (because you want to use the VAE for the generation of inputs similar to the ones in your dataset). So, the idea is that you optimize both the KL term and the reconstruction (or likelihood) term jointly (i.e. the ELBO). Why? Because, as I just said, the ELBO is the Evidence Lower BOund on the given data, so, by maximizing it, you are also maximizing the evidence of your data. In other words, if you maximize the ELBO, you are finding a decoder that will have a high probability of reconstructing your inputs (i.e. the likelihood term), but, at the same time, you want your encoder to be constrained (i.e. KL term). Please, read this answer for further details.

Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.

The relationship is that you will be maximizing the ELBO, which implies (and you can see this implication only if you are familiar with the ELBO loss) you will be minimizing the KL divergence between your posterior and the prior to generate the samples $z$ (i.e. minimizing because there will be a minus in front of the KL term in the ELBO loss) and maximizing the probability of the reconstructed input. More precisely, $z$ is used to reconstruct the input (i.e. the decoder does this), which is then used to calculate the reconstruction loss.

In the mathematical formulations, you will see that the likelihood term of the ELBO is $p(x \mid z)$, i.e. the likelihood of the input $x$ given $z$. The $z$ is the input to the decoder, which produces a reconstruction of $x$. In practice, people will e.g. use the cross-entropy to then calculate the "reconstruction loss" (e.g. see this PyTorch implementation), which should correspond to this likelihood term $p(x \mid z)$. Why does the cross-entropy correspond to a likelihood? Because you can actually prove that the cross-entropy is equivalent to the negative log-likelihood. (Also, note that, in the ELBO loss, $p(x \mid z)$ does not appear, but the logarithm of $p(x \mid z)$ appears, but, for simplicity, I have used $p(x \mid z)$ rather than $\log p(x \mid z)$ above.)