I was reading the article on medium, where the author cites this equation for Variational Inference: \begin{align*} \text{KL}(q(z|x^{(i)})||p(z|x^{(i)})) &= \int_z q(z|x^{(i)})\text{log}\frac{q(z|x^{(i)})}{p(z|x^{(i)})} dz \\ &= \mathbb{E}_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}_{||}[\text{log}(p(z|x^{(i)}))]\\ &= \mathbb{E}_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}_{||}[\text{log}(p(x^{(i)}, z))] + \mathbb{E}_q[\text{log}(p(x^{(i)}))]\\ &= \mathbb{E}_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}_{||}[\text{log}(p(x^{(i)}, z))] + \text{log}(p(x^{(i)}))\\ &= -\text{ELBO} + \text{log}(p(x^{(i)}))\\ \end{align*}

I understand all of the math behind this equation, but I was wondering what is the underlying intuition behind each of the terms in this equation (KL divergence, ELBO, and logp(x))?

The author claims that $\text{log} p(x)$ is a constant in this equation and I'm having a hard time understanding why. Is $p(x)$ considered to be the theoretical data generating distribution which created our $x$'s and not the model that we are training?


2 Answers 2


Variational methods are designed for situations allowing us to avoid the intractable integral such as $p(x)$ parameterized by the common $\theta$ along with $p(z|x)$ here by transforming the statistical inference problem into an optimization problem with respect to the parameters $\theta$ and additional $\phi$ parameterizing the needed variational $q(z|x)$. And since in your reference the prior marginal evidence $p(x)$ defined as the latent variables induced model distribution evaluated at the $i$-th example of the training set (note your reference uses $p(x^{(i)})$ before notation simplification) integrates out $p_{\theta}(x|z)$ by all possible values of the latent variables $z$ to induce the said example, it's constant with respect to the optimized parameters $\theta$ and $\phi$ and thus irrelevant of $q_{\phi}(z|x)$. But note for your confusion the distribution $p(x)$ itself is essentially a GMM as explained in VAE's wikipedia reference.

To further avoid the intractable $p(x)$ in the ELBO, your reference arrives at the same convenient form of ELBO as mentioned in wikipedia:

$${L_{\theta ,\phi }(x):=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }({z|x})}}\right]=\ln p_{\theta }(x)-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }({\cdot |x}))}$$ The form given is not very convenient for maximization, but the following, equivalent form, is: $${L_{\theta ,\phi }(x)=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln p_{\theta }(x|z)\right]-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }(\cdot ))}$$ where ${\ln p_{\theta }(x|z)}$ is implemented as ${-{\frac {1}{2}}\|x-D_{\theta }(z)\|_{2}^{2}}$, since that is, up to an additive constant, what ${x\sim {\mathcal {N}}(D_{\theta }(z),I)}$ yields. That is, we model the distribution of $x$ conditional on $z$ to be a Gaussian distribution centered on ${D_{\theta }(z)}$.

  • $\begingroup$ Thank you for your answer. If our goal in generative modeling is to maximize p(x), then why does VAE math all center around trying to approximate p(z|x), Is this because with q(z|x), p(z), and p(x|z) all of which we have after variation inference, we can calculate/approximate p(x). Also as a side note, I am confused by the use of theta. When we say p_theta(x), does theta refer to our neural network parameters or parameters to a theoretical unknown distribution that we don't have the analytical form for. $\endgroup$ Commented Feb 23 at 6:26
  • $\begingroup$ As clearly stated in your own ref, generative modeling goal is mostly about the first 2 problems, while the 3rd problem of prior evidence p(x) modeled as a GMM distribution or evaluated at an example as a constant is much less important (irrelevant to ELBO as explained in my answer). Theta is the non-variational decoder network parameters for the posterior multivariate Gaussian distribution p(x|z) which certainly has analytic form but unknown parameters such as mean vector and covariant matrix, the theta parameterized decoder’s outputs are nothing but the said Gassian’s unknown parameters. $\endgroup$
    – cinch
    Commented Feb 23 at 7:31
  • $\begingroup$ so from what I am understanding the in the equation log p(x)=ELBO -KL(q(z|x)||p(z|x)), log p(x) represents the true data generating distribution and does not change because it is not dependent on theta or any parameters that exist within our model. Does this intuition sound right? $\endgroup$ Commented Feb 24 at 0:57
  • $\begingroup$ Check the line above "The term logp(x) is a constant" in your ref, '-' should be "+". Since the logp(x) is constant, then minimizing the KL-Divergence is equivalent o maximizing the ELBO which is the lower bound of log-evidence p(x) as KL-D is always greater than 0. Since x is continuous for image like data, as explained in my answer, here p(x) mean its pdf evaluated at the ith training data which is an intractable integral but we know it must be a constant w.r.t. the learning parameters theta and phi. So your intuition is correct except it's not true data generating distribution meant here. $\endgroup$
    – cinch
    Commented Feb 24 at 6:57

Let me try to give you a less mathematical, more intuitive explanation.

You assume a latent space $\mathcal{Z}$ and you approximate the true distribution $p(x)$ as: $$ p_\theta(x) = \sum_\mathcal{Z} p_\theta(x|z) p_Z(z) = \mathbb{E}_{z \sim p_Z} p_\theta(x|z) $$ Here you use a neural net with parameters $\theta$ to model $p(x|z)$.

However, there is a problem that arises in this setting. In order to optimize our model we actually want to sample $z$ that would be a good match for the given data point $x$, but there is actually very little chance that that would be the case. If there really exists an underlying latent space to your data, then the best distribution to sample $z$ from is $p(z|x)$. So what you do now is you try to approximate this with a second neural network $q_\phi(x)$.

You want to maximize the log-probability of the training data by optimizing $\theta$:

\begin{aligned} \max_\theta \log p_\theta(x) &= \log \mathbb{E}_{z \sim q_\phi(x)} \bigg[ p_\theta(x|z) \frac{p_Z(z)}{q_\phi(z|x)} \bigg] \\ & \geq \mathbb{E}_{z \sim q_\phi(x)} \log \bigg[ p_\theta(x|z) \frac{p_Z(z)}{q_\phi(z|x)} \bigg] \\ & = \mathbb{E}_{z \sim q_\phi(x)} \bigg[ \log p_\theta(x) - KL(q_\phi(x) \; || \;p_Z) \bigg] \end{aligned}

And you also want to minimize the KL divergence between $q_\phi(x)$ and $p(z|x)$ by optimizing $\phi$:

\begin{aligned} \min_\phi KL(q_\phi(z|x) \; || \; p_\theta(z|x) ) &= \mathbb{E}_{z \sim q_\phi(x)} \log \frac{q_\phi(z|x)}{p_\theta(z|x)} \\ &= \mathbb{E}_{z \sim q_\phi(x|z)} \Big[ q_\phi(z|x) - p_Z(z) - \log p_\phi(x|z) \Big] + \log p_\theta(x) \end{aligned}

Note that the last term in the objective is constant with respect to $\phi$, so in the end we are left with the same equation but this time we optimize over $\phi$.

Finally, the objective is:

$$ \max_{\theta, \phi} \mathbb{E}_{z \sim q_\phi(x)} \bigg[ \log p_\theta(x) - KL(q_\phi(x) \; || \;p_Z) \bigg] $$


You must log in to answer this question.