Questions tagged [evidence-lower-bound]

For questions about the Evidence Lower BOund (ELBO) objective function, which is typically optimized in the context of variational auto-encoders or variational Bayesian neural networks.

Filter by
Sorted by
Tagged with
0 votes
1 answer
40 views

Are some low dimensional distributions known to be hard to model with VAEs?

I am trying to implement a toy VAE project. My goal is to use a VAE to model the moon dataset from scikit-learn, with an extra constant (but noisy) z-dimension. To this end I use an approximate ...
user avatar
  • 21
0 votes
0 answers
24 views

Tensorflow Probability Implementation of Automatic Differentiation Variational Inference with Mixtures

In this paper, the authors suggest using the following loss instead of the traditional ELBO in order to train what basically is a Variational Autoencoder with a Gaussian Mixture Model instead of a ...
user avatar
  • 1
1 vote
1 answer
29 views

How is the variational lower bound for hard attention derived in Show, Attend and Tell

How is the jump from line 1 to line 2 done in equation 10 of Show, Attend and Tell? While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is ...
user avatar
3 votes
1 answer
105 views

What does the approximate posterior on latent variables, $q_\phi(z|x)$, tend to when optimising VAE's

The ELBO objective is described as follows $$ ELBO(\phi,\theta) = E_{q_\phi(z|x)}[log p_\theta (x|z)] - KL[q_\phi (z|x)||p(z)] $$ This form of ELBO includes a regularisation term in the form of the ...
user avatar
0 votes
2 answers
812 views

In variational autoencoders, why do people use MSE for the loss?

In VAEs, we try to maximize the ELBO = $\mathbb{E}_q [\log\ p(x|z)] + D_{KL}(q(z \mid x), p(z))$, but I see that many implement the first term as the MSE of the image and its reconstruction. Here's a ...
user avatar
  • 129
2 votes
1 answer
217 views

How does the implementation of the VAE's objective function equate to ELBO?

For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO. $$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$ The above is the definition ...
user avatar
4 votes
1 answer
507 views

Why does the variational auto-encoder use the reconstruction loss?

VAE is trained to reduce the following two losses. KL divergence between inferred latent distribution and Gaussian. the reconstruction loss I understand that the first one regularizes VAE to get ...
user avatar
  • 89
1 vote
1 answer
146 views

In this VAE formula, why do $p$ and $q$ have the same parameters?

In $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$ Given that $p$ is ...
user avatar
5 votes
1 answer
460 views

Why is the evidence equal to the KL divergence plus the loss?

Why is the equation $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables? I was ...
user avatar
2 votes
0 answers
55 views

Why does the ELBO come to a steady state and the latent space shrinks?

I'm trying to train a VAE using a graph dataset. However, my latent space shrinks epoch by epoch. Meanwhile, my ELBO plot comes to a steady state after a few epochs. I tried to play around with ...
user avatar
  • 141
4 votes
2 answers
564 views

What's going on in the equation of the variational lower bound?

I don't really understand what this equation is saying or what the purpose of the ELBO is. How does it help us find the true posterior distribution?
user avatar
  • 351