Skip to main content
Share Your Experience: Take the 2024 Developer Survey
9 votes

Why doesn't VAE suffer mode collapse?

With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long ...
R.M.'s user avatar
  • 308
8 votes
Accepted

Why is the evidence equal to the KL divergence plus the loss?

In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\...
nbro's user avatar
  • 40.8k
8 votes

How is this Pytorch expression equivalent to the KL divergence?

This is the analytical form of the KL divergence between two multivariate Gaussian densities with diagonal covariance matrices (i.e. we assume independence). More precisely, it's the KL divergence ...
nbro's user avatar
  • 40.8k
8 votes
Accepted

What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?

You are asking about several things here and while related, solving one, will not necessarily "solve" your problem. Let's look at them separately: Optimal dimension of the latent space. ...
Евген's user avatar
7 votes
Accepted

What does the notation $\mathcal{N}(z; \mu, \sigma)$ stand for in statistics?

It means that $z$ has a (multivariate) normal distribution with 0 mean and identity covariance matrix. This essentially means each individual element of the vector $z$ has a standard normal ...
David's user avatar
  • 4,910
6 votes
Accepted

How does the implementation of the VAE's objective function equate to ELBO?

I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by ...
nbro's user avatar
  • 40.8k
6 votes

What are the fundamental differences between VAE and GAN for image generation?

GANs generally produce better photo-realistic images but can be difficult to work with. Conversely, VAEs are easier to train but don’t usually give the best results. I recommend picking VAEs if you ...
Brian O'Donnell's user avatar
6 votes
Accepted

How is this Pytorch expression equivalent to the KL divergence?

The code is correct. Since OP asked for a proof, one follows. The usage in the code is straightforward if you observe that the authors are using the symbols unconventionally: ...
Sycorax's user avatar
  • 473
6 votes
Accepted

In variational autoencoders, why do people use MSE for the loss?

If $p(x|z) \sim \mathcal{N}(f(z), I)$, then \begin{align} \log\ p(x|z) &\sim \log\ \exp(-(x-f(z))^2) \\ &\sim -(x-f(z))^2 \\ &= -(x-\hat{x})^2, \end{align} where $\hat{x}$, the ...
IttayD's user avatar
  • 219
4 votes

Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you ...
seasonyc's user avatar
4 votes
Accepted

Concrete example of latent variables and observables plugged into the Bayes' rule

Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule \begin{align} p(z|x)=\frac{p(x|z)p(z)}{p(x)} \tag{1}\label{1} \end{align} The ...
nbro's user avatar
  • 40.8k
4 votes

How many types of variational auto-encoders are there?

There are many variations of the original VAE (proposed in the 2013 paper Auto-Encoding Variational Bayes), with different purposes (such as the generation of discrete data or graphs). Of course, I ...
nbro's user avatar
  • 40.8k
4 votes
Accepted

How to determine the quality of synthetic data?

Due to subjective nature, quantitative evaluation of synthetic images is difficult in general. However, there are metrics like Inception Score or FID score that are used for evaluation of generative ...
ayandas's user avatar
  • 258
4 votes
Accepted

Are the authors of the VAE paper writing the PDFs as a function of the random variables?

When it comes to notation/terminology, often, people in machine learning are (a bit?) sloppy, which causes a lot of confusion, especially for newcomers to the field or people not very math-savvy. I ...
nbro's user avatar
  • 40.8k
4 votes

What exactly is meant by variational distribution?

The variational distribution is the distribution (or set of distributions) that you use to approximate the distribution you are looking for. It's often denoted by $q$, $q_\phi$ or $q_\phi(z \mid x)$, ...
nbro's user avatar
  • 40.8k
4 votes
Accepted

In the VAE, why is $z \sim \mathcal{N}(\mu, \sigma^2)$ equivalent to $z = \mu + \sigma \odot \epsilon$?

I'll attempt a less formal explanation. The distribution $\mathcal{N}(\mu, \sigma)$ represents a normal distribution with mean $\mu$ and standard deviation $\sigma$. When we sample from this ...
Robin van Hoorn's user avatar
3 votes
Accepted

Why do we regularize the variational autoencoder with a normal distribution?

If you are mathematically inclined, here is an article that discusses the reasoning. What I get as a take away is that the VAE forces the learned latent space to be Gaussian due to the KL divergence ...
Gerry P's user avatar
  • 724
3 votes
Accepted

Why does the KL divergence not satisfy the triangle inequality?

To prove that the KL divergence does not satisfy the triangle inequality, you just need a counterexample. Definitions KL divergence Let's first recapitulate the definition of KL divergence for ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

How does the Kullback-Leibler divergence give "knowledge gained"?

You can know it better, if you know the concept of entropy: Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also ...
OmG's user avatar
  • 1,816
3 votes

How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
Konstantin Solomatov's user avatar
3 votes

What's going on in the equation of the variational lower bound?

From this document, as you found here, $X$ is an observed variable and $Z$ is a hidden variable; $p(X)$ is the density function of $X$. The posterior distribution of the hidden variables can then be ...
OmG's user avatar
  • 1,816
3 votes
Accepted

In variational autoencoders, what does p(x|z) mean?

Whilst you're right that for any continuous distribution $P(X = x) = 0 \;; \forall x \in \mathcal{X}$ where $\mathcal{X}$ is there support of the distribution, they are not referring to probabilities ...
David's user avatar
  • 4,910
3 votes

In variational autoencoders, why do people use MSE for the loss?

On page 5 of the VAE paper, it's clearly stated We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

How does backprop work through the random sampling layer in a variational autoencoder?

You do not backpropagate with respect to $\epsilon$, which is the random sample or random variable (depending on how you look at it). You backpropagate with respect to the mean $\mu$ and variance $\...
nbro's user avatar
  • 40.8k
3 votes
Accepted

Why don't we also need to approximate $p(x \mid z)$ in the VAE?

What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right? You could assume $p(x)$ is ...
nbro's user avatar
  • 40.8k
3 votes

How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian?

Few more clarifications. While the correct thing to do is draw from the prior, we have no guarantees that the aggregated posterior will cover the prior. Think of the aggregated posterior as the ...
sfotiadis's user avatar
  • 291
3 votes

Comparison of the two alternative forms for the KL divergence

The KL divergence is just not symmetric, and so changing $q$ for $p$, and vice-versa, gives you a different behavior because the expectation is computed on a different distribution. In the first plot,...
Luca Anzalone's user avatar
3 votes

How to expand reconstruction error to mean squared error in Variational AutoEncoder?

In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

How to perform latent space Interpolation between two images?

You need more training images. Far more, at least a few hundred, with variations. The latent space has no meaningful form to it when you train with just two end points. The decoder will have no ...
Neil Slater's user avatar
  • 32.4k
2 votes

Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?

In essence, Variational Autoencoders learn an "explicit" distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution. However, Generative ...
Amir's user avatar
  • 135

Only top scored, non community-wiki answers of a minimum length are eligible