In general, if I have a collection of data then mean(Expectation) and standard deviation are calculated as follows

$$\text{mean } = \mu = \mathbb{E}[X] = \sum\limits_{i = 1}^n p_ix_i $$ $$\text{Variance =}\sigma (X) = \sqrt{\sum\limits_{i = 1}^{n}p_i{(x_i - \mu)^2}{}}$$

where $X$ is a random vector having support $\{x_1, x_2, x_3, \cdots, x_n\}$.

Thus a dataset of samples have a single mean and single variance.

Now, let us discuss about the case of variational auto-encoders. They look like follows enter image description here

Suppose I trained the above auto-encoder on a training set, then for each sample I will get a mean and standard deviation at latent layer. Here, we can get a new $\mu$ and $\sigma$ for each data sample. But, as we see earlier, mean and standard deviation exists for a dataset and not for each sample.

I am confused about "how can we say that mean and standard deviation are obtained at latent layer if they are not constant in nature"?


Mean $\mu(x)$ and standard deviation $\sigma(x)$ are actually learnable functions, whose parameters are adjusted via the back propagation procedure.

Mean and standard deviation are not computed on the input vector $x$ or any transform of it.

The procedure is the following:

  • Pass the sample $x$ from the training data
  • Propagate this vector $x$ through some NN (Feedforward MLP) and obtain some other vector $\tilde{x}$
  • Get the mean $\mu(x)$ and std $\sigma(x)$ from $\tilde{x}$ from two more neural networks (maybe single layer)
  • Generate random noise $\varepsilon$ and get a point in the latent space $\mu(x) + \sigma(x) \varepsilon$ (it is known as reparametrization trick)

enter image description here

You can think about the procedure as follows - you have Normal distribution around each point of the input data in the latent space, and the mean $\mu(x)$ and $\sigma(x)$ are the parameters of this distribution (different for each point). The generated data is expected to resemble the training example, but differ in some reasonable sense, belong to the manifold of realistic images.

  • $\begingroup$ But, what is the gaurentee for us that those single feed forward layers are indeed calculating mean and standard deviation? Is there any reason behind calling them as mean and standard deviation? $\endgroup$
    – hanugm
    Aug 17 '21 at 1:42
  • 1
    $\begingroup$ @hanugm they are defined to do this. Given any reference sample $x$ during many enough forward passes with the fixed parameters of networks $\mu(x)$ and $\sigma(x)$ - the point cloud to which this point will be mapped will have mean $\mu(x)$ and standard deviation $\sigma(x)$. It is guaranteed by the reparametrization trick. $\endgroup$ Aug 17 '21 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.