1
$\begingroup$

In $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$

Given that $p$ is just the probability of the observed data and $q$ is the approximation of the posterior, shouldn't they be different distributions and thus their parameters different?

$\endgroup$
1
$\begingroup$

I will try to answer your questions directly (but I guess I won't be able to), otherwise, this can become quite confusing, given the inconsistencies that can be found across different sources.

In $logp_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$ why is $\theta$ and param for $p$ and $q$?

In a few words, your equation is wrong because it uses the letters $\phi$ and $\theta$ inconsistently.

If you look more carefully at the right-hand side of your equation, you will notice that $q_{\theta}$ has different parameters, i.e. $\theta$, than $p_{\phi}$, which has parameters $\phi$, so $p$ and $q$ have different parameters, and this should be the case, because they are represented by different neural networks in the case of the VAE. However, the left-hand side uses $\theta$ as the parameters of $p$ (while the right-hand side uses $\phi$ to index $p$), so this should already suggest that the equation is not correct (as you correctly thought).

In the case of the VAE, $\phi$ usually represents the parameters (or weights) of the encoder neural network (NN), while $\theta$ usually represents the parameters of the decoder NN (or vice-versa, but you should just be consistent, which is often not the case in your equation). In fact, in the VAE paper, in equation 3, the authors use $\phi$ to represent the parameters of the encoder $q$, while $\theta$ is used to denote the parameters of the decoder $p$.

So, if you follow the notation in the VAE paper, the ELBO can actually be written something like

\begin{align} \mathcal{L}(\phi,\theta; \mathbf{x}) &= \mathbb{E}_{\tilde{z} \sim q_{\phi}(\mathbf{z} \mid \mathbf{x})} \left[ \log p_{\theta} (\mathbf{x} \mid \mathbf{z}) \right] - \operatorname{KL} \left(q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| p_{\theta}(\mathbf{z}) \right) \tag{1} \label{1} \end{align}

The ELBO loss $\mathcal{L}(\phi,\theta; \mathbf{x})$ has both parameters (of the encoder and decoder), which will be optimized jointly. Note that I have ignored the indices in the observations $\mathbf{x}$ (for simplicity), while, in the VAE paper, they are present. Furthermore, note that, both in \ref{1} and in the VAE paper, we use bold letters (because these objects are usually vectors), i.e. $\mathbf{x}$ and $\mathbf{z}$, rather than $x$ and $z$ (like in your equation).

Note also that, even though $p_{\theta}(\mathbf{z})$ is indexed by $\theta$, in reality, this may be an un-parametrized distribution (e.g. a Gaussian with mean $0$ and variance $1$), i.e. not a family of distributions. The use of the index $\theta$ in $p_{\theta}(\mathbf{z})$ comes from the (implicit) assumption that both $p_{\theta}(\mathbf{z})$ and $p_{\theta} (\mathbf{x} \mid \mathbf{z})$ come from the same family of distributions (e.g. a family of Gaussians). In fact, if you consider the family of all Gaussian distributions, then $p_{\theta}(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ also belongs to that family. But $\theta$ and $\phi$ are also used to denote the parameters (or weights) of the networks, so this becomes understanbly confusing. (To understand equation 10 of the VAE paper, see https://stats.stackexchange.com/a/370048/82135).

Why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$

This is wrong, in fact. If you look at equation 1 of the VAE paper, they use $\theta$ to denote the parameters of $p(\mathbf{x})$, i.e. $p_{\theta}(\mathbf{x})$, while the parameters of the encoder are $\phi$, i.e. $q_{\phi}(\mathbf{z} \mid \mathbf{x}$).

Cause $p$ is just the probability of the observed data and $q$ is the approximation of the posterior so shouldn't they be different distributions and their parameters different?

Yes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.