8
votes
Accepted
Why is the evidence equal to the KL divergence plus the loss?
In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\...
- 39.6k
8
votes
Why doesn't VAE suffer mode collapse?
With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long ...
- 298
8
votes
How is this Pytorch expression equivalent to the KL divergence?
This is the analytical form of the KL divergence between two multivariate Gaussian densities with diagonal covariance matrices (i.e. we assume independence). More precisely, it's the KL divergence ...
- 39.6k
8
votes
Accepted
What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?
You are asking about several things here and while related, solving one, will not necessarily "solve" your problem. Let's look at them separately:
Optimal dimension of the latent space.
...
- 196
7
votes
Accepted
What does the notation $\mathcal{N}(z; \mu, \sigma)$ stand for in statistics?
It means that $z$ has a (multivariate) normal distribution with 0 mean and identity covariance matrix. This essentially means each individual element of the vector $z$ has a standard normal ...
- 4,665
6
votes
Accepted
How is this Pytorch expression equivalent to the KL divergence?
The code is correct. Since OP asked for a proof, one follows.
The usage in the code is straightforward if you observe that the authors are using the symbols unconventionally: ...
- 453
5
votes
What are the fundamental differences between VAE and GAN for image generation?
GANs generally produce better photo-realistic images but can be difficult to work with. Conversely, VAEs are easier to train but don’t usually give the best results.
I recommend picking VAEs if you ...
- 1,863
5
votes
Accepted
In variational autoencoders, why do people use MSE for the loss?
If $p(x|z) \sim \mathcal{N}(f(z), I)$, then
\begin{align}
\log\ p(x|z)
&\sim \log\ \exp(-(x-f(z))^2) \\
&\sim -(x-f(z))^2 \\
&= -(x-\hat{x})^2,
\end{align}
where $\hat{x}$, the ...
- 209
4
votes
Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you ...
- 49
4
votes
Accepted
Concrete example of latent variables and observables plugged into the Bayes' rule
Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule
\begin{align}
p(z|x)=\frac{p(x|z)p(z)}{p(x)}
\tag{1}\label{1}
\end{align}
The ...
- 39.6k
4
votes
Accepted
How does the implementation of the VAE's objective function equate to ELBO?
I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by ...
- 39.6k
4
votes
How many types of variational auto-encoders are there?
There are many variations of the original VAE (proposed in the 2013 paper Auto-Encoding Variational Bayes), with different purposes (such as the generation of discrete data or graphs). Of course, I ...
- 39.6k
4
votes
Accepted
Are the authors of the VAE paper writing the PDFs as a function of the random variables?
When it comes to notation/terminology, often, people in machine learning are (a bit?) sloppy, which causes a lot of confusion, especially for newcomers to the field or people not very math-savvy. I ...
- 39.6k
3
votes
Accepted
Why do we regularize the variational autoencoder with a normal distribution?
If you are mathematically inclined, here is an article that discusses the reasoning.
What I get as a take away is that the VAE forces the learned latent space to be Gaussian due to the KL divergence ...
- 704
3
votes
Accepted
Why does the KL divergence not satisfy the triangle inequality?
To prove that the KL divergence does not satisfy the triangle inequality, you just need a counterexample.
Definitions
KL divergence
Let's first recapitulate the definition of KL divergence for ...
- 39.6k
3
votes
Accepted
How does the Kullback-Leibler divergence give "knowledge gained"?
You can know it better, if you know the concept of entropy:
Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also ...
- 1,751
3
votes
How should we choose the dimensions of the encoding layer in auto-encoders?
The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
3
votes
What's going on in the equation of the variational lower bound?
From this document, as you found here, $X$ is an observed variable and $Z$ is a hidden variable; $p(X)$ is the density function of $X$. The posterior distribution of the hidden variables can then be ...
- 1,751
3
votes
Accepted
In variational autoencoders, what does p(x|z) mean?
Whilst you're right that for any continuous distribution $P(X = x) = 0 \;; \forall x \in \mathcal{X}$ where $\mathcal{X}$ is there support of the distribution, they are not referring to probabilities ...
- 4,665
3
votes
In variational autoencoders, why do people use MSE for the loss?
On page 5 of the VAE paper, it's clearly stated
We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary ...
- 39.6k
3
votes
Accepted
How to determine the quality of synthetic data?
Due to subjective nature, quantitative evaluation of synthetic images is difficult in general. However, there are metrics like Inception Score or FID score that are used for evaluation of generative ...
- 238
3
votes
Accepted
How does backprop work through the random sampling layer in a variational autoencoder?
You do not backpropagate with respect to $\epsilon$, which is the random sample or random variable (depending on how you look at it). You backpropagate with respect to the mean $\mu$ and variance $\...
- 39.6k
3
votes
Accepted
Why don't we also need to approximate $p(x \mid z)$ in the VAE?
What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right?
You could assume $p(x)$ is ...
- 39.6k
3
votes
How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian?
Few more clarifications. While the correct thing to do is draw from the prior, we have no guarantees that the aggregated posterior will cover the prior. Think of the aggregated posterior as the ...
- 181
3
votes
Comparison of the two alternative forms for the KL divergence
The KL divergence is just not symmetric, and so changing $q$ for $p$, and vice-versa, gives you a different behavior because the expectation is computed on a different distribution.
In the first plot,...
- 2,443
3
votes
How to expand reconstruction error to mean squared error in Variational AutoEncoder?
In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should ...
- 39.6k
2
votes
Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
In essence, Variational Autoencoders learn an "explicit" distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution.
However, Generative ...
- 135
2
votes
Accepted
Does MMD-VAE solve the problem of blurred images of vanilla VAEs?
[Answering my own question after 5 months of studying VAE models]
The point of the MMD-VAE or InfoVAE is not exactly to emphasise on the visual quality of generated samples. It is to preserve greater ...
- 148
2
votes
What's going on in the equation of the variational lower bound?
The use of KL provides a more intuitive way of what the ELBO is attempting to maximize.
Basically, we want to find a posterior approximation such that $p(z\mid x) \approx q(z)\in\mathcal{Q}$
$$KL(q(...
- 21
2
votes
Accepted
Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
In the source code, the author defines sd by
sd = 0.5 * tf.layers.dense(x, units=n_latent)
which means that $\...
- 261
Only top scored, non community-wiki answers of a minimum length are eligible