What is the difference between q and p in Statistical Notation(used in VAE)?

Question

I'm looking at general visuals of Variational Autoencoders and I'm seeing that the encoder is typically expressed as q(z|x) with phi as a subscript while the decoder is p(x|z) with theta as a subscript. I've seen the use of p and q in other papers like stable diffusion. I was wondering what the specific meanings of p and q are and in what situations would you use them.

Answer 1

These letters have no special meaning. It's just a convention. So you can denote these distributions with other letters, like $a$, $b$, $c$, etc. $p$ is probably common because it probably stands for probability. $q$ is commonly used to denote the variational distribution in the context of VAEs, which is the distribution you use to approximate the target distribution. People maybe used $q$ because it just comes after $p$ in alphabet (who knows?).

The subscript in $a_\theta$ means that the distribution $a$ is parametrized by $a$. For example, you could denote a Gaussian distribution as follows $a_\theta = \mathcal{N}(\mu, \sigma)$, where $\theta = \{\mu, \sigma\}$.

However, in practice, in the case of the VAEs, the parameters are the weights of the neural networks, which are actually used to produce either the mean and variance, which are used to construct the distribution, from which you can sample the latent vector (encoder) or to decode (e.g. generate an image). In other words, you have some parameters which are a function of other parameters.

You can think of a pametrized distribution as a collection of distributions, unless you fix the parameters. It's like parameterized function. You can also denote a neural network as a parametrized function. More details here. It all now comes together.

If you're not familiar with the VAEs, these explanations will make little sense. Anyway, the VAE just models a process like $p(x, z) = p(x \mid z) p(z)$. Hopefully the following image (from the VAE paper) will clarify things a bit.

Answer 2

VAEs are probabilistic generative models initially designed for unsupervised learning, it can randomly generate new data that is similar to the input training data. And $p, q$ are standard conditional probability distributions approximated by the decoder and encoder neural networks, respectively.

Variational autoencoders are probabilistic generative models that require neural networks as only a part of their overall structure. The neural network components are typically referred to as the encoder and decoder for the first and second component respectively. The first neural network maps the input variable to a latent space that corresponds to the parameters of a variational distribution... The decoder has the opposite function, which is to map from the latent space to the input space, in order to produce or generate data points.

From the point of view of probabilistic modeling, one wants to maximize the likelihood of the data $x$ by their chosen parameterized probability distribution $p_{\theta }(x)=p(x|\theta)$. This distribution is usually chosen to be a Gaussian... Simple distributions are easy enough to maximize, however distributions where a prior is assumed over the latents $z$ results in intractable integrals.

Real-world data often exhibit complex and multimodal distributions that cannot be accurately captured by a single (Gaussian) distribution, so you have to introduce a possibly high dimensional latent space corresponding to the parameters of a commonly assumed variational multivariate Gaussian distribution to encode the input.

According to the chain rule, the equation can be rewritten as $$p_{\theta}(x)=\int _{z}p_{\theta}({x|z})p_{\theta}(z)\,dz$$ In the vanilla variational autoencoder, $z$ is usually taken to be a finite-dimensional vector of real numbers, and $p_{\theta}({x|z})$ to be a Gaussian distribution. Then $p_{\theta}(x)$ is a mixture of Gaussian distributions.

Thus $p$ is usually the conditional likelihood distribution $p_{\theta}(x|z)$ of the input computed by the decoder and is represented by a ANN parameterized by $\theta$.

Unfortunately, the computation of $p_{\theta}(z|x)$ is expensive and in most cases intractable. To speed up the calculus to make it feasible, it is necessary to introduce a further function to approximate the posterior distribution as $q_{\phi }({z|x})\approx p_{\theta}({z|x})$ with $\phi$ defined as the set of real values that parametrize $q$. This is sometimes called amortized inference, since by "investing" in finding a good $q_{\phi}$, one can later infer $z$ from $x$ quickly without doing any integrals.

Thus $q$ is usually an approximated posterior distribution $p_{\theta}(z|x)$ of the latent variable computed by the encoder and is represented by a ANN parameterized by $\phi$.

Finally the two ANNs are usually trained together by maximizing a variational evidence lower bound aka ELBO. As for their use in stable diffusion is not surprised since it's also a generative model, but you'd better cite your reference specifically.

Answer 3

In the context of Variational Autoencoders (VAEs) and other probabilistic models, "p" and "q" denote probability distributions. They are used to describe the relationships between variables and model different aspects of the data generation process.

1. $p$: Represents the true or target distribution in the problem. In VAEs, $p$ is used for:

   • p(x): The marginal distribution of the data, representing the overall distribution of observed data points.
   • p(z): The prior distribution of the latent variables, usually chosen as a simple distribution like a standard Gaussian.
   • p(x|z): The likelihood function (decoder), representing the probability of generating data point x given a latent variable z, parameterized by $theta$.This function is also known as the decoder, as it "decodes" the latent variable z back into the data space. It is parameterized by a set of parameters denoted as $\mathcal{\Theta}$, which typically are weights and biases.VAE tries to find the best $\mathcal{\Theta}$ for the decoder such that the generated data points $x$ are as close as possible to the true data points $z$. During training, the encoder learns to generate approximate posterior distributions $q(z|x)$ that capture the distribution of latent variables for a given data point $x$. The decoder then uses these latent variables to reconstruct the original data points. By optimizing the VAE's objective function, which includes both the reconstruction loss and a regularization term (KL-divergence), the model learns a compact and meaningful latent space representation while ensuring the generated data is close to the true data.

2. $q$: Represents an approximate or variational distribution used to estimate the true distribution "p". In VAEs, "q" is used for:

   • q(z|x): The variational posterior (encoder), representing an approximation of the true posterior distribution p(z|x), parameterized by $phi$.

The use of "p" and "q" is not limited to VAEs. In other contexts like Stable Diffusion, "p" and "q" are used similarly to represent different probability distributions involved in the problem.