Are mean and standard deviation in variational autoencoders unique?

Question

In general, if I have a collection of data then mean(Expectation) and standard deviation are calculated as follows

$$\text{mean } = \mu = \mathbb{E}[X] = \sum\limits_{i = 1}^n p_ix_i $$ $$\text{Variance =}\sigma (X) = \sqrt{\sum\limits_{i = 1}^{n}p_i{(x_i - \mu)^2}{}}$$

where $X$ is a random vector having support $\{x_1, x_2, x_3, \cdots, x_n\}$.

Thus a dataset of samples have a single mean and single variance.

Now, let us discuss about the case of variational auto-encoders. They look like follows

Suppose I trained the above auto-encoder on a training set, then for each sample I will get a mean and standard deviation at latent layer. Here, we can get a new $\mu$ and $\sigma$ for each data sample. But, as we see earlier, mean and standard deviation exists for a dataset and not for each sample.

I am confused about "how can we say that mean and standard deviation are obtained at latent layer if they are not constant in nature"?

Answer 1

Mean $\mu(x)$ and standard deviation $\sigma(x)$ are actually learnable functions, whose parameters are adjusted via the back propagation procedure.

Mean and standard deviation are not computed on the input vector $x$ or any transform of it.

The procedure is the following:

• Pass the sample $x$ from the training data

• Propagate this vector $x$ through some NN (Feedforward MLP) and obtain some other vector $\tilde{x}$
• Get the mean $\mu(x)$ and std $\sigma(x)$ from $\tilde{x}$ from two more neural networks (maybe single layer)

• Generate random noise $\varepsilon$ and get a point in the latent space $\mu(x) + \sigma(x) \varepsilon$ (it is known as reparametrization trick)

You can think about the procedure as follows - you have Normal distribution around each point of the input data in the latent space, and the mean $\mu(x)$ and $\sigma(x)$ are the parameters of this distribution (different for each point). The generated data is expected to resemble the training example, but differ in some reasonable sense, belong to the manifold of realistic images.

Answer 2

The answer relies in the KL item of the loss function - it regularizes (=enforce) the distribution of z (represented by mu and sigma) to be close to N(0,1).

That's the meaning of the KL divergence - a distance measure between probabilities and in order to minimize that penalty item - the network will tune it's parameters such that the z would be very close to N(0,1).