2
$\begingroup$

I've noticed in several resources on variational autoencoders (for example the wikipedia article), we use the same parameters theta for the prior, likelihood, posterior, etc distributions. For example the equation $p_\theta(x) = \int_z p_\theta(x|z)p_\theta(z)dz$. Aren't $p_\theta(x)$ and $p_\theta(z)$ two different distributions, so how can we parameterize them with the same $\theta$ params. I might be misunderstanding something about what it means to parameterize a distribution...

$\endgroup$

1 Answer 1

0
$\begingroup$

I think this is very confusing to many people. I had to deal with VAEs (and Bayesian neural networks) multiple times in the past, and I've seen so many inconsistent notations and unclear explanations. Here's what I know about the issue that you're raising, but I will not give you a definitive answer.

In the appendix of the VAE paper (sections C.1 and C.2), they consider the cases of

  1. a multivariate Bernoulli decoder, and
  2. Gaussian encoder and decoder

They write

  • in the case of the multivariate Bernoulli decoder, $\theta$ are the weights and biases of the neural network that outputs the Bernoulli parameters (each Bernoulli distribution has only one parameter that basically defines the probability of getting one sample rather than the other - this is not written in that section - I am just telling you this, in case you didn't know it).

  • in the case of the Gaussian encoder, $\phi$ (which parametrizes the variational distribution $q_\phi$) are the weights and biases of the neural network that represents the encoder (which outputs the $\mu$ and $\sigma$ of the Gaussian, which are the parameters that define the specific Gaussian - you can see this from the equations in appendix C.2)

  • in the case of the Gaussian decoder (appendix C.2), the weights and biases of the neural network (the decoder) are part of $\theta$ (I don't know what they mean by "part of" here, honestly).

So, in the case of the VAE (with a multivariate Bernoulli decoder, or a Gaussian encoder and decoder), which is really a neural network (or 2 neural networks - encoder and decoder) that attempts to model the random process which is represented by the graphical model in figure 1 of the VAE paper, $\phi$ and $\theta$ are the weights and biases of the neural networks that represent the encoder and decoder (partially, in the case of the Gaussian decoder, they say), respectively. $\phi$ and $\theta$ are really the learnable parameters. In fact, if you look at algorithm 1 or any implementation of the VAE (I like this simple PyTorch one), they are what we're trying to optimize.

Now, if you look at the definition of a parametric model, e.g. here, you will see that the parameters $\theta$ in $p_\theta$ actually refer to e.g. the mean and variance of the Gaussian, i.e. $\theta = \{ \mu, \sigma \}$. This is one thing that causes confusion - the subscript $\theta$ (or $\phi$) refers to different things in different articles.

HOWEVER, ultimately, what we're trying to learn are the parameters of a probability distribution, let's say it's a Gaussian, so we are trying to learn a mean and variance $\{\mu, \sigma \}$. In the VAE, we use a neural network to output the mean and variance, i.e. $\mu, \sigma = f_\phi(x)$, where $f_\phi$ is the neural network with weights $\theta$ (see the equations in appendix C) and $x$ is an input. So, $\{\mu, \sigma \}$ are defined by $\phi$.

Now, back to your actual question. Here are some possible interpretations of that notation.

  1. If $\theta$ represented the parameters of a specific probability distribution (let's say a Gaussian), so $\theta = \{ \mu, \sigma \}$, then, as you noticed, you would have all your distributions with the same mean and variance, and I don't think this is generally the case (but could happen?) - for example, $p_\theta(x)$ and $p_\theta(z)$ would exactly be the same distribution, so that means that $x$ and $z$ follow exactly the same distribution (e.g. $\mathcal{N}(0, 1)$).

  2. However, if you look at figure 1 of the VAE paper, both $x$ and $z$ depend on $\theta$, but is it the same $\theta$? So, even though the letter $\theta$ is used as a subscript in the likelihood, prior, marginal and posterior, do these distributions actually have the same specific values of $\theta$? If you look at section 2.1, they also use that notation you found on Wikipedia.

  3. If $\theta$ represented the weights of neural networks, then all neural networks would be equal (this is not the case in practice - see the PyTorch implementation).

  4. If $\theta$ were hyper-parameters, that notation could make sense (because the mean and variance of the Gaussians could still be different), but this does not seem to be consistent with the actual explanation in appendix C and the PyTorch implementation.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .