These slides (slide number 26) mention that the ELBO enforces an information bottleneck at the latent variables z which make it prone to bad local minima.
Can you please explain what they mean by that?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ I'd need to read the slides, but probably this only means that essentially we compress the inputs into a vector, which places a restriction on the amount of information you can contain from the input. Now, why it leads to local minima is a different question. Not evident to answer I guess. $\endgroup$– nbroCommented Dec 18, 2023 at 10:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In the slides, they aim to explain the use of Hierarchical VAEs and the types of problems they address. "Evidence Lower Bound" (ELBO) measures a lower bound approximation of the log-likelihood distribution for the observed data. A VAE estimates the real distribution of $p(x)$ through a Normal Distribution, utilizing the ELBO estimation, which in a continuous space is expressed as:
$$ELBO = \int q(z|x) \log \left( \frac{p(x,z)}{q(z|x)}\right) dz$$
This formula arises from marginalizing the latent variables and applying Bayes' rule. However, due to the integral's intractability, the real distribution of $p(x)$ is approximated by the ELBO loss function.
The ELBO loss function employs KL divergence, which, outside the probabilistic perspective, acts as a regularizer for the observed data. The reconstruction process is similar to an Autoencoder loss, but instead of using Euclidean distances with Mean Squared Error (MSE), it utilizes probabilities with Sigmoids and Binary Cross Entropy to align with the nature of the image data.
This regularization makes it challenging for the model to specialize in generating a specific data point. The VAE's objective is not just to reconstruct observed data but to use a learned distribution for reconstructing the learned data, which is markedly different. This addition of a learned distribution results in a more general yet sparser representation, often leading to a mean representation of the data.
Typically, this is undesirable for a neural network. We generally don't want models to learn a mean representation of the data, as this can be easily achieved with other methods like PCA, which employs linear approximations through its eigenvectors. Neural Networks should represent complex nonlinear functions, like is the case of $p(x)$ if $x$ represents the random variable for the sets of crisp and realistic images .
So in the slides when they mention that ELBO enforces an information bottleneck at the latent variables $z$. The z (latent variables) generating the normal distributions from which we sample to retrieve $p(x)$ (the image data) are trapped in a suboptimal local minimum. This means that these Normal Distributions generated by the $\sigma$ and $\mu$ vectors generated by the encoder are not optimally tuned for the specific set of the observed data.
This issue is common in unsupervised learning when the target distribution is unknown, and the model is expected to identify or learn it. In the case of VAEs, the goal is not to replicate the discrete distribution of the exact observed data, but to recognize that the observed data may represent a more general continous distribution.
A potential solution is to segment the continuous latent space into discrete portions. This approach enables the model to learn more specific data distributions, narrowing the data possibilities. This is the reason behind using VQ-VAEs, which balance generality and improved data modeling for a specific dataset. Another solutions for this, is to use diffusion models, they seem to address these issues without necessitating the quantization of the vector space.
