16
votes
Accepted
Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?
When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence are equal.
As already stated in the question, the general formula is this:
$$H(p, q) = H(p) + D_{...
- 1,917
8
votes
Accepted
Why is the evidence equal to the KL divergence plus the loss?
In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\...
- 37k
7
votes
How is this Pytorch expression equivalent to the KL divergence?
This is the analytical form of the KL divergence between two multivariate Gaussian densities with diagonal covariance matrices (i.e. we assume independence). More precisely, it's the KL divergence ...
- 37k
6
votes
Accepted
How is this Pytorch expression equivalent to the KL divergence?
The code is correct. Since OP asked for a proof, one follows.
The usage in the code is straightforward if you observe that the authors are using the symbols unconventionally: ...
- 248
5
votes
Accepted
Are there some notions of distance between two policies?
Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance ...
- 37k
4
votes
Accepted
Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?
Lets start with question 1) how does JS-divergence handles zeros?
by definition:
\begin{align}
D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\
&= \frac{...
- 2,329
3
votes
Accepted
How does the Kullback-Leibler divergence give "knowledge gained"?
You can know it better, if you know the concept of entropy:
Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also ...
- 1,723
3
votes
Accepted
Why does the KL divergence not satisfy the triangle inequality?
To prove that the KL divergence does not satisfy the triangle inequality, you just need a counterexample.
Definitions
KL divergence
Let's first recapitulate the definition of KL divergence for ...
- 37k
2
votes
What is the reason for mode collapse in GAN as opposed to WGAN?
I don't have a definite answer, but only a suspicion/idea:
Looking at Figure 1 from the WGAN paper, we clear see that the JS divergence on the right is not continuous at $0$, hence not differentiable ...
- 171
2
votes
What are the advantages of the Kullback-Leibler over the MSE/RMSE?
KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.
L2-norm/MSE/RMSE doesn't do well with ...
- 1,379
2
votes
When should one prefer using Total Variational Divergence over KL divergence in RL
To add to nbro's answer, I'd say also that much of the time the distance measure isn't simply a design decision, rather it comes up naturally from the model of the problem. For instance, minimizing ...
- 1,071
2
votes
Why is KL divergence used so often in Machine Learning?
In ML we always deal with unknown probability distributions from which the data comes. The most common way to calculate the distance between real and model distribution is $KL$ divergence.
Why ...
- 847
2
votes
Why is KL divergence used so often in Machine Learning?
This question is very general in the sense that the reason may differ depending on the area of ML you are considering. Below are two different areas of ML where the KL-divergence is a natural ...
- 196
1
vote
Accepted
How is this statement from a TensorFlow implementation of a certain KL-divergence formula related to the corresponding formula?
It should remain from a general code that has been refactored. By the way, the red code phrase is always zero. Because, beta is a vector of ...
- 1,723
1
vote
Accepted
How do you calculate KL divergence on a three-dimensional space for a Variational Autoencoder?
Your three dimensional latent representation consists of two images of mean pixels and covariance pixels as shown in Fig. 3. Which represents a Gaussian distribution with the mean and covariance for ...
- 600
1
vote
What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?
Ans 1.
The motive of Variational Inference(on which VAE is based), is to decrease $KL(q(z|x)||p(z))$, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can ...
- 164
1
vote
Accepted
When should one prefer using Total Variational Divergence over KL divergence in RL
I did not read those two specified linked/cited papers and I am not currently familiar with the total variation distance, but I think I can answer some of your questions, given that I am reasonably ...
- 37k
1
vote
Accepted
What are the advantages of the Kullback-Leibler over the MSE/RMSE?
In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior $p(z \mid x)$, to the known joint $p(z,x)=p(x|z)p(z)$ and optimize only the ELBO. You cannot do this ...
- 412
Only top scored, non community-wiki answers of a minimum length are eligible