15 votes
Accepted

Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence are equal. As already stated in the question, the general formula is this: $$H(p, q) = H(p) + D_{...
user avatar
  • 1,877
6 votes
Accepted

Why is the evidence equal to the KL divergence plus the loss?

In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\...
user avatar
  • 34.4k
6 votes

How is this Pytorch expression equivalent to the KL divergence?

This is the analytical form of the KL divergence between two multivariate Gaussian densities with diagonal covariance matrices (i.e. we assume independence). More precisely, it's the KL divergence ...
user avatar
  • 34.4k
5 votes
Accepted

How is this Pytorch expression equivalent to the KL divergence?

The code is correct. Since OP asked for a proof, one follows. The usage in the code is straightforward if you observe that the authors are using the symbols unconventionally: ...
user avatar
  • 236
5 votes
Accepted

Are there some notions of distance between two policies?

Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance ...
user avatar
  • 34.4k
3 votes
Accepted

Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?

Lets start with question 1) how does JS-divergence handles zeros? by definition: \begin{align} D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\ &= \frac{...
user avatar
  • 2,289
3 votes
Accepted

How does the Kullback-Leibler divergence give "knowledge gained"?

You can know it better, if you know the concept of entropy: Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also ...
user avatar
  • 1,663
3 votes
Accepted

Why does the KL divergence not satisfy the triangle inequality?

To prove that the KL divergence does not satisfy the triangle inequality, you just need a counterexample. Definitions KL divergence Let's first recapitulate the definition of KL divergence for ...
user avatar
  • 34.4k
2 votes

What is the reason for mode collapse in GAN as opposed to WGAN?

I don't have a definite answer, but only a suspicion/idea: Looking at Figure 1 from the WGAN paper, we clear see that the JS divergence on the right is not continuous at $0$, hence not differentiable ...
user avatar
2 votes

What are the advantages of the Kullback-Leibler over the MSE/RMSE?

KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted. L2-norm/MSE/RMSE doesn't do well with ...
user avatar
  • 1,369
2 votes

When should one prefer using Total Variational Divergence over KL divergence in RL

To add to nbro's answer, I'd say also that much of the time the distance measure isn't simply a design decision, rather it comes up naturally from the model of the problem. For instance, minimizing ...
user avatar
  • 981
2 votes

Why is KL divergence used so often in Machine Learning?

In ML we always deal with unknown probability distributions from which the data comes. The most common way to calculate the distance between real and model distribution is $KL$ divergence. Why ...
user avatar
2 votes

Why is KL divergence used so often in Machine Learning?

This question is very general in the sense that the reason may differ depending on the area of ML you are considering. Below are two different areas of ML where the KL-divergence is a natural ...
user avatar
  • 196
1 vote
Accepted

How is this statement from a TensorFlow implementation of a certain KL-divergence formula related to the corresponding formula?

It should remain from a general code that has been refactored. By the way, the red code phrase is always zero. Because, beta is a vector of ...
user avatar
  • 1,663
1 vote
Accepted

How do you calculate KL divergence on a three-dimensional space for a Variational Autoencoder?

Your three dimensional latent representation consists of two images of mean pixels and covariance pixels as shown in Fig. 3. Which represents a Gaussian distribution with the mean and covariance for ...
user avatar
  • 600
1 vote

What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?

Ans 1. The motive of Variational Inference(on which VAE is based), is to decrease $KL(q(z|x)||p(z))$, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can ...
user avatar
1 vote
Accepted

When should one prefer using Total Variational Divergence over KL divergence in RL

I did not read those two specified linked/cited papers and I am not currently familiar with the total variation distance, but I think I can answer some of your questions, given that I am reasonably ...
user avatar
  • 34.4k
1 vote
Accepted

What are the advantages of the Kullback-Leibler over the MSE/RMSE?

In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior $p(z \mid x)$, to the known joint $p(z,x)=p(x|z)p(z)$ and optimize only the ELBO. You cannot do this ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible