1
$\begingroup$

In general, if I have a collection of data then mean(Expectation) and standard deviation are calculated as follows

$$\text{mean } = \mu = \mathbb{E}[X] = \sum\limits_{i = 1}^n p_ix_i $$ $$\text{Variance =}\sigma (X) = \sqrt{\sum\limits_{i = 1}^{n}p_i{(x_i - \mu)^2}{}}$$

where $X$ is a random vector having support $\{x_1, x_2, x_3, \cdots, x_n\}$.

Thus a dataset of samples have a single mean and single variance.

Now, let us discuss about the case of variational auto-encoders. They look like follows enter image description here

Suppose I trained the above auto-encoder on a training set, then for each sample I will get a mean and standard deviation at latent layer. Here, we can get a new $\mu$ and $\sigma$ for each data sample. But, as we see earlier, mean and standard deviation exists for a dataset and not for each sample.

I am confused about "how can we say that mean and standard deviation are obtained at latent layer if they are not constant in nature"?

$\endgroup$
1
  • $\begingroup$ I don't want to edit your question, but I believe you have put "Variance" where you should have put "Standard Deviation" in the formula with $\sigma (X)$. However, the formula on the right hand side isn't quite right for stddev (or certainly for variance) $\endgroup$ Commented Mar 18, 2022 at 11:05

2 Answers 2

1
$\begingroup$

Mean $\mu(x)$ and standard deviation $\sigma(x)$ are actually learnable functions, whose parameters are adjusted via the back propagation procedure.

Mean and standard deviation are not computed on the input vector $x$ or any transform of it.

The procedure is the following:

  • Pass the sample $x$ from the training data
  • Propagate this vector $x$ through some NN (Feedforward MLP) and obtain some other vector $\tilde{x}$
  • Get the mean $\mu(x)$ and std $\sigma(x)$ from $\tilde{x}$ from two more neural networks (maybe single layer)
  • Generate random noise $\varepsilon$ and get a point in the latent space $\mu(x) + \sigma(x) \varepsilon$ (it is known as reparametrization trick)

enter image description here

You can think about the procedure as follows - you have Normal distribution around each point of the input data in the latent space, and the mean $\mu(x)$ and $\sigma(x)$ are the parameters of this distribution (different for each point). The generated data is expected to resemble the training example, but differ in some reasonable sense, belong to the manifold of realistic images.

$\endgroup$
2
  • $\begingroup$ But, what is the gaurentee for us that those single feed forward layers are indeed calculating mean and standard deviation? Is there any reason behind calling them as mean and standard deviation? $\endgroup$
    – hanugm
    Commented Aug 17, 2021 at 1:42
  • 1
    $\begingroup$ @hanugm they are defined to do this. Given any reference sample $x$ during many enough forward passes with the fixed parameters of networks $\mu(x)$ and $\sigma(x)$ - the point cloud to which this point will be mapped will have mean $\mu(x)$ and standard deviation $\sigma(x)$. It is guaranteed by the reparametrization trick. $\endgroup$ Commented Aug 17, 2021 at 6:05
0
$\begingroup$

The answer relies in the KL item of the loss function - it regularizes (=enforce) the distribution of z (represented by mu and sigma) to be close to N(0,1).

That's the meaning of the KL divergence - a distance measure between probabilities and in order to minimize that penalty item - the network will tune it's parameters such that the z would be very close to N(0,1).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .