Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [kl-divergence]

For questions related to the Kullback–Leibler (KL) divergence, which is a measure (that is not a metric, but it is pre-metric, because it does not satisfy all properties of metrics, i.e. it is not symmetric) of divergence (or distance) between two probability measures (density functions, or mass functions), which is commonly used in many machine learning settings, e.g. in the context of variational auto-encoders (VAES).

26 questions
Filter by
Sorted by
Tagged with
1 vote
51 views

### How to evaluate the KL divergence between two distributions that may require sampling?

The KL divergence between two distributions is: $$\int \mathbf{p}(x;\theta_{1}) \; log \frac{\mathbf{p}(x;\theta_{1})}{\mathbf{p}(x;\theta_{2})} \nu(dx) \\$$ If the ...
• 111
27 views

### How to find an argument of a NN function(which returns a distribution) to minimize a KL divergence?

Consider a neural network function $f:\mathbb{R}\to distribution$. For simplicity, maybe consider that it returns a gaussian distribution. I want to find $\arg\min_{s\in\mathbb{R}}D_{KL}(f(s),q)$ for ...
23 views

### I don't understand how the KL Divergence works for the stated goal of Inception Score

Inception Score has two goals: The entropy for the distribution prediced on individual samples should be small(the outputs are specific) The entropy for the marginal distribution over all samples ...
• 113
30 views

• 795
1k views

### Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?

I have read articles on how Jensen-Shannon divergence is preferred over Kullback-Leibler in measuring how good a distribution mapping is learned in a generative network because of the fact that JS-...
• 1,419
379 views

### How does the Kullback-Leibler divergence give "knowledge gained"?

I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section Expressed in the ...
• 351
4k views

### What are the advantages of the Kullback-Leibler over the MSE/RMSE?

I've recently encountered different articles that are recommending to use the KL divergence instead of the MSE/RMSE (as the loss function), when trying to learn a probability distribution, but none of ...
• 1,148