Questions tagged [evidence-lower-bound]
For questions about the Evidence Lower BOund (ELBO) objective function, which is typically optimized in the context of variational auto-encoders or variational Bayesian neural networks.
29
questions
2
votes
1
answer
57
views
Question regarding the ELBO decomposition proposed by Hoffman&Johnson
recently I'm trying to read a paper by Hoffman and Johnson discussing an alternative decomposition of ELBO in variational autoencoders. In formula (9) and (10) of their original paper, they proposed ...
- 21
2
votes
1
answer
51
views
Deriving ELBO for Diffusion Models
I am trying to read through the proof of ELBO for diffusion models on pg. 8 of this paper. However, I do not see how the author arrived at Eqn (45) from Eqn (44). Specifically, I do not know how they ...
2
votes
1
answer
48
views
Derivation of the consistency term in the DDPM Evidence Lower Bound (ELBO) [closed]
I have been studying diffusion models from this tutorial: https://arxiv.org/abs/2403.18103 and trying to derive all results as I read it. Although this tutorial is very comprehensive, it skips many of ...
- 31
0
votes
0
answers
22
views
In VAE, why not maximize D[q(z)||p(z|x)] instead？
In VAE, the goal is to maximize $\log p(x)$, where $x$ is the data and $p(\cdot)$ is a parameterized distribution.
For any distribution $q(z)$, the following identity holds,
$$
\log p(x)=D[q(z)||p(z|x)...
- 1
0
votes
0
answers
18
views
Is there a way to mix the Importance weighted VAE with the Beta-VAE?
Is there a way to mix the Importance weighted VAE with the Beta-VAE?
It seems that we cannot separate out the KL-term from the IWAE-ELBO, hence we cannot multiply it with Beta. On the other hand, it ...
- 11
2
votes
2
answers
221
views
What is the meaning of log p(x) in VAE math and why is it constant
I was reading the article on medium, where the author cites this equation for Variational Inference:
\begin{align*}
\text{KL}(q(z|x^{(i)})||p(z|x^{(i)})) &= \int_z q(z|x^{(i)})\text{log}\frac{q(z|...
1
vote
1
answer
159
views
How does using the ELBO in VAEs make the problem tractable?
I'm studying Variational Autoencoders and a lot of the literature says that the posterior is intractable because the marginal distribution p(x) is intractable since the space of z is so large we ...
0
votes
0
answers
16
views
Should the encoder be trained for more steps in VAEs?
This is based on my interpretation of how the ELBO loss works. The log likelihood of the data is equal to the ELBO + KL Divergence term.
$$\operatorname{log} p_{\theta}(x) = \underbrace{\mathbb {E}_{...
1
vote
1
answer
167
views
What is an information bottleneck in the context of ELBO and Hierarchical VAEs?
These slides (slide number 26) mention that the ELBO enforces an information bottleneck at the latent variables z which make it prone to bad local minima.
Can you please explain what they mean by that?...
0
votes
0
answers
44
views
Variational Autoencoders - Can We Learn Directly From Marginal With a Pretrained Decoder?
So, with VAE we use ELBO instead of directly maximizing the marginal likelihood, because the marginal likelihood is intractable.
As far as I understand it, this is the case for two reasons:
$$p(x) = \...
0
votes
0
answers
17
views
Trouble understanding Variational Inference objective
I was reading the Meta Temporal Point Processes paper and was having trouble understading the training objective presented. The authors state that it is the ELBO used in Variational Inference ...
- 1
0
votes
1
answer
115
views
Confusion over taking gradients in Variational Autoencoders (VAE)
I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example.
We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log
\...
1
vote
1
answer
82
views
If we know the joint distribution, can we simply derive the evidence from it?
I'm struggling to understand one specific part of the formalism of the free energy principle.
My understanding is that the free energy principle can be derived from considering statistical dynamics of ...
- 11
3
votes
1
answer
121
views
Clarification on the training objective of denoising diffusion models
I'm reading the Denoising Diffusion Probabilistic Models paper (Ho et al. 2020). And I am puzzled about the training objective. I understood (I think) the trick regarding the reparametrization of the ...
- 31
0
votes
1
answer
57
views
Why is the variational lower bound is easier to compute than the original marginal distribution?
Why is the ELBO of $p(x)=\int p(x|z)p(z)\mathrm{d}z$ easier to compute/estimate than the expression itself? Can we compute this quantity itself through sampling in the same way? I understanding that ...
- 101
1
vote
0
answers
106
views
Is VAE the same as the E-step of the EM algorithm?
EM(Expectation Maximum)
Target: maximize $p_\theta(x)$
$ p_\theta(x)=\frac{p_\theta(x, z)}{p_\theta(z \mid x)} \\\\$
Take log on both sides:
$ \log p_\theta(x)=\log p_\theta(x, z)-\log p_\theta(z \...
- 11
0
votes
0
answers
390
views
How to optimize ELBO(VAE's loss function)?
Suppose we've got the following formula:
$\log p(x;\theta)=\mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]+KL(q(z|x;\phi)||p(z|x;\theta))\\ \geq \mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\...
- 11
1
vote
0
answers
14
views
variational inference but with a weighted loglikelihood
I would like to know if it's correct if I substitute in the ELBO formula
a weighted sum of the loglikelihood
$$\sum E_{q_{\theta}(w)}[w_i \ln{p(y_i|f^{w}(x_i))}]$$
in place of the traditional sum.
...
- 111
0
votes
1
answer
175
views
Are some low dimensional distributions known to be hard to model with VAEs?
I am trying to implement a toy VAE project.
My goal is to use a VAE to model the moon dataset from scikit-learn, with an extra constant (but noisy) z-dimension.
To this end I use an approximate ...
- 21
0
votes
0
answers
56
views
Tensorflow Probability Implementation of Automatic Differentiation Variational Inference with Mixtures
In this paper, the authors suggest using the following loss instead of the traditional ELBO in order to train what basically is a Variational Autoencoder with a Gaussian Mixture Model instead of a ...
- 1
1
vote
1
answer
37
views
How is the variational lower bound for hard attention derived in Show, Attend and Tell
How is the jump from line 1 to line 2 done in equation 10 of Show, Attend and Tell?
While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is ...
- 1,369
3
votes
1
answer
296
views
What does the approximate posterior on latent variables, $q_\phi(z|x)$, tend to when optimising VAE's
The ELBO objective is described as follows
$$ ELBO(\phi,\theta) = E_{q_\phi(z|x)}[log p_\theta (x|z)] - KL[q_\phi (z|x)||p(z)] $$
This form of ELBO includes a regularisation term in the form of the ...
- 394
3
votes
2
answers
8k
views
In variational autoencoders, why do people use MSE for the loss?
In VAEs, we try to maximize the ELBO = $\mathbb{E}_q [\log\ p(x|z)] + D_{KL}(q(z \mid x), p(z))$, but I see that many implement the first term as the MSE of the image and its reconstruction. Here's a ...
- 229
2
votes
1
answer
2k
views
How does the implementation of the VAE's objective function equate to ELBO?
For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO.
$$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$
The above is the definition ...
- 805
4
votes
1
answer
2k
views
Why does the variational auto-encoder use the reconstruction loss?
VAE is trained to reduce the following two losses.
KL divergence between inferred latent distribution and Gaussian.
the reconstruction loss
I understand that the first one regularizes VAE to get ...
- 89
2
votes
1
answer
442
views
In this VAE formula, why do $p$ and $q$ have the same parameters?
In $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$
Given that $p$ is ...
- 805
6
votes
1
answer
1k
views
Why is the evidence equal to the KL divergence plus the loss?
Why is the equation $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables?
I was ...
- 805
2
votes
0
answers
149
views
Why does the ELBO come to a steady state and the latent space shrinks?
I'm trying to train a VAE using a graph dataset. However, my latent space shrinks epoch by epoch. Meanwhile, my ELBO plot comes to a steady state after a few epochs.
I tried to play around with ...
- 151
4
votes
2
answers
923
views
What's going on in the equation of the variational lower bound?
I don't really understand what this equation is saying or what the purpose of the ELBO is. How does it help us find the true posterior distribution?
- 351