I am not sure I understand what is the advantage of using a VAE's over a deterministic Auto Encoder? For example, assuming we have just 2 labels, a deterministic Auto Encoder will always map a given image to the same latent vector. However, one expects that after the training, the 2 classes will form separate clusters in the encoder space.

In the case of the VAE an image is mapped to an encoding vector probabilistically. However, one still ends up with 2 separate clusters. Now, if one passes a new image (at the test time), in both cases the network should be able to place that new image in one of the 2 clusters.

How are these 2 clusters created using the VAE better than the ones from the deterministic case?

  • $\begingroup$ "However, one expects that after the training, the 2 classes will form separate clusters in the encoder space.", what do you mean by "the 2 classes"? $\endgroup$
    – nbro
    May 24 '19 at 9:51

The main difference between a variational auto-encoder (VAE) and an auto-encoder is that the VAE is a generative and statistical model while an auto-encoder (AE) is just a data compressor and decompressor (it is just a function approximator), so an AE is not a statistical model. They both have an encoder and a decoder and they both convert the inputs to a latent representation, but their inner workings are different and their purpose is also slightly different. An AE could be considered a generative model, in the sense that it generates objects that are similar to the input (but they might not be the same as the input). However, it is not a generative model according to the usual statistical definition of a generative model.

In the case of an AE, given an input $x$ (e.g. an image), the encoder produces one latent vector $z_x$, which can be decoded into $x'$ (another image which should be similar or related to $x$). Compactly, this can be presented as $x'=f(z_x=g(x))$, where $g$ is the encoder and $f$ is the decoder. This operation is deterministic, in the sense that, given the same $x$, the same $z_x$ and $x'$ are produced.

In the case of a VAE, given an input $x \in \mathcal{X}$ (e.g. an image), more than one latent vector, $z_{x}^i \in \mathcal{Z}$, can be produced, because the encoder part of the VAE models the inputs as a probability distribution. In other words, the encoder of a VAE assumes that the inputs $\mathcal{X}$ are drawn from a probability distribution, which can be denoted by $\mathcal{X} \sim q$, where $q$ is some computable probability distribution (e.g. a Gaussian distribution with mean zero, $\mu =0$, and a finite standard deviation, $\sigma$). $q$ is defined over $\mathcal{Z}$ (the space of possible latent vectors). More specifically, $q$ is a conditional probability distribution, which can be denoted by $q(z \mid x)$.

So, the encoder of a VAE, while being trained, creates a probability distribution, which, in practice, can be represented by neural networks, that is, you can train a neural network to represent the parameters of e.g. a Gaussian distribution (which is defined by two parameters: the mean and standard deviation). You can thus more precisely denote $q$ by $q_\phi$, where $\phi$ are the parameters of a neural network that e.g. output two numbers: the mean and a standard deviation of $q$.

Given a probability distribution $q_\phi$, we can sample several $z_{x}^i$ from $q_\phi$. Why is this useful? Given the same input $x$ (e.g. a graphical representation of a molecule $m$), we can sample e.g. $N$ latent vectors $z_{x}^1, z_{x}^2, \dots, z_{x}^N$, which can all be decoded (e.g. into $K$ new molecules). Why would this be useful? For example, in the case you need to generate molecules which have a similar structure to the input molecule $m$. Why would you need this? In some cases, molecules with a similar structure can have similar effects, but could e.g. be more easy or inexpensive to synthesise.

Why can we generate $K$ new molecules out of $N$ latent vectors, where $K$ can be greater than $N$? We have not yet mentioned the details of a decoder of a VAE. The decoder of a VAE also models its inputs, $\mathcal{Z}$, as a probability distribution defined over $\mathcal{X}$, which can be denoted by $p_\theta(x \mid z)$, where $z$ is a latent vector. In other words, given the $i$th latent vector sampled from the latent distribution (the distribution of the encoder), $z_{x}^i$, we can sample more than one reconstructed input from the probability distribution of the decoder. Hence, a VAE often constructs two probability distributions: one that is associated with the encoder, $q_\phi(z \mid x)$, and one that is associated with the decoder, $p_\theta(x \mid z)$.

Given that VAEs create these distributions, they are considered statistical models. An auto-encoder has no notion of a probability distribution. An AE only maps inputs to outputs deterministically, so it is a function approximator (given that it approximates a function that compresses the input and then another function that decompresses it). VAEs are thus a lot more flexible than AEs.

  • $\begingroup$ thanks, i thought those are same. Are they also implemented differently? $\endgroup$ Jun 25 '19 at 0:19
  • $\begingroup$ @user8426627 I've never implemented them, but they should have different implementations. Have a look at some public implementation around. In the case of the VAE, you need to learn the parameters of distribution (e.g., if you use a normal distribution, you need to learn the mean and possibly the variance). In the AE case, there is not such thing. $\endgroup$
    – nbro
    Jun 29 '19 at 20:18
  • $\begingroup$ that sounds like input is interpreted differently (input preparation is not part of nn) but the network itself has same architecture $\endgroup$ Jun 29 '19 at 21:04
  • $\begingroup$ @user8426627 It is not the input, it is more the output. The mean and the variance will be the output of some NN. However, both in the case of AE and VAE, at least conceptually, there is an encoder and a decoder. In this way, they are similar. $\endgroup$
    – nbro
    Jun 29 '19 at 21:08

VAE's are not used for classification. They are used for inference or as Generative Models, while AE's can be used as data re-constructors (as you described above), de-noisers, classifiers. So the difference is generation of new data vs re-construction of data.

VAE's map the inputs to a hidden space, where each variable is enforced to have probability distribution which is given by $N(0,1)$ i.e. the standard Normal Distribution. Once we have trained a VAE we will now, use only the decoder part to generate new models.


enter image description here

Source: Stanford University CS231n slides

Assume there is an x_axis and a y_axis on the bottom and the left. Let the x_axis represent $x_1$ and y_axis $x_2$ which are our hidden variables. By varying $x_1$ and $x_2$ you can see what happens. Increasing $x_1$ changes face angle while increasing $x_2$ changes eye droop. Thus we can generate new data by varying the features in a latent representation.

For better understanding I highly recommend you check out these links:

Variational autoencoders.

Variational Autoencoders - Brian Keng

VAE - Ali Ghodsi

Generative Models - CS231n

  • $\begingroup$ Thanks for this! It kinda makes sense (I will read the papers, too). One quick questions: why enforce the distribution of the latent variables to be unit gaussians? Is there any a priori reason for that variable to have that distribution? For example, why can't the rotation angle (for example) have an uniform distribution? $\endgroup$ May 24 '19 at 8:05
  • $\begingroup$ @AlexMarshall I cannot answer that question but probably it has some statistical and mathemetical motivation, like easier calculations, etc. Or maybe it's a hyperparameter with empirical basis that when we use normal function then this works well. $\endgroup$
    – user9947
    May 24 '19 at 8:37
  • $\begingroup$ @AlexMarshall The Gaussian distribution is a good approximation of many real-world distributions. For example, the height of a population often follows a Gaussian distribution. $\endgroup$
    – nbro
    May 24 '19 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.