1
$\begingroup$

I'm studying Variational Autoencoders and a lot of the literature says that the posterior is intractable because the marginal distribution p(x) is intractable since the space of z is so large we cannot possibly integrate over it all. So to avoid this they create a lower bound on the log likelihood, the ELBO, which they then try to maximize. The term for the ELBO is:

$$E_q[log~ p(z,x)]-E_q[log~q(z)]$$

What I am trying to understand is how is this now tractable. The expectations in the ELBO are still over the distribution of q. Take the first term for example:

$$E_q[log~ p(z,x)]=\int_{}^{}q(z)~log~p(z,x)dz$$

Is this not still an integral over all z values? How did we make this problem any more tractable by finding the ELBO?

Additional Question: Also another thing I was confused about is we always say the posterior p(z|x) is not computable because we don't have p(x), but how exactly do we have the numerator, p(x,z). $$p(z|x)=\frac{p(x,z)}{p(x)}$$

Is this because we assume a prior, and then also assume that we can model p(x|z) with a decoder?

$\endgroup$
1
  • $\begingroup$ Please, limit your posts to just one specific question. If you have multiple questions, even if they are related to the same main topic, you can always create multiple posts. $\endgroup$
    – nbro
    Feb 19 at 12:13

1 Answer 1

0
$\begingroup$

Though the posterior $p(z|x)$ is intractable, we can still compute $p(x,z)=p(z)p(x|z)$ using conditional probability. As you claimed correctly $p(z)$ is assumed a simple and known prior distribution such as a standard normal distribution. Depending on the underlying generative process $p(x|z)$ is often assumed to be a multivariate Gaussian distribution with a diagonal covariance matrix and is typically implemented using a neural network known as decoder which takes the latent variables $z$ as input and outputs the parameters $\theta$ of $p(x|z)$.

Your ELBO seems not exactly correct compared to the reference: $${L_{\theta ,\phi }(x):=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }({z|x})}}\right]=\ln p_{\theta }(x)-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }({\cdot |x}))}$$

The form given is not very convenient for maximization, but the following, equivalent form, is: $${L_{\theta ,\phi }(x)=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln p_{\theta }(x|z)\right]-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }(\cdot ))}$$ where ${\ln p_{\theta }(x|z)}$ is implemented as ${-{\frac {1}{2}}\|x-D_{\theta }(z)\|_{2}^{2}}$, since that is, up to an additive constant, what ${x\sim {\mathcal {N}}(D_{\theta }(z),I)}$ yields. That is, we model the distribution of $x$ conditional on $z$ to be a Gaussian distribution centered on ${D_{\theta }(z)}$.

Since distribution of the approximate posterior ${q_{\phi }(z|x)}$ and the prior ${p_{\theta }(z)}$ are often chosen to be Gaussians, ELBO in the reference is finally tractable as $${L_{\theta ,\phi }(x)=-{\frac {1}{2}}\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\|x-D_{\theta }(z)\|_{2}^{2}\right]-{\frac {1}{2}}\left(N\sigma _{\phi }(x)^{2}+\|E_{\phi }(x)\|_{2}^{2}-2N\ln \sigma _{\phi }(x)\right)+Const}$$

Here $N$ is the dimension of $z$.

$\endgroup$
4
  • $\begingroup$ Thank you for you answer. But how can we automatically say that we can model p(x|z) with a decoder, but we can't do the same with p(z|x). Through Bayes formula isn't p(x|z) also dependent on p(x)? $\endgroup$ Feb 21 at 0:29
  • $\begingroup$ Of course encoder is used to approximate parameters of p(z|x), similar to decoder p(x|z). As implicated in my answer p(z|x)p(x)=p(x|z)p(z) and RHS terms are both tractable, so what I mean p(z|x) is intractable (analytically) is due to intractability of p(x). If you were able to directly learn a parametric model of the evidence p(x) (an integral expression per chain rule as a GMM in my ref) from first principles such as MLE or MAP, then you can simply generate output by sampling p(x). The difficulty of p(x) is that integration in high dimensional latent space is intractable even numerically. $\endgroup$
    – cinch
    Feb 21 at 6:29
  • $\begingroup$ From what I am understanding the reason why p(x|z) is tractable is because we simply choose for it to tractable my modeling it with a simply tractable distribution like Gaussian. Then obviously because of p(x), p(z|x) becomes intractable and we have to approximate it with VI. Theoretically, is it possible to choose a simple tractable distribution for p(z|x) instead which would force us to VI approximation on the likelihood? $\endgroup$ Feb 21 at 7:44
  • $\begingroup$ There're reasons. Since z is latent variable which acts similarly as hidden state in HMM, thus p(z) and p(x|z) including p(z|x) are rightly supposed to be tractable as multivariate Gaussians usually in your sense of 'simple distribution' due to its analytic convenience, enabling reparameterization trick and VI via KL Divergence and preventing overfit, yet sometimes to fit more complex multimodal images, GMMs or hierarchical multivariate Gaussians are used, or normalizing flows instead of VAEs. Any conceivable simpler continuous distribution like uniform likely underfits. Hope it clarifies now. $\endgroup$
    – cinch
    Feb 21 at 19:38

You must log in to answer this question.