Questions tagged [kl-divergence]
For questions related to the Kullback–Leibler (KL) divergence, which is a measure (that is not a metric, but it is pre-metric, because it does not satisfy all properties of metrics, i.e. it is not symmetric) of divergence (or distance) between two probability measures (density functions, or mass functions), which is commonly used in many machine learning settings, e.g. in the context of variational auto-encoders (VAES).
24
questions
0
votes
0
answers
17
views
Variational Autoencoders - Can We Learn Directly From Marginal With a Pretrained Decoder?
So, with VAE we use ELBO instead of directly maximizing the marginal likelihood, because the marginal likelihood is intractable.
As far as I understand it, this is the case for two reasons:
$$p(x) = \...
1
vote
0
answers
115
views
Negative KL-divergence RLHF implementation
I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace:
What Is the Concern with Negative KL Divergence?
If you generate text by purely ...
- 11
0
votes
0
answers
231
views
How does one decide the probability distribution for an LLM during RLHF?
I was looking into how KL divergence is used in LLMs to prevent reward hacking within the course Generative AI with LLMs on Coursera (if this redirects to the homepage, the article is on week 3). As ...
- 101
0
votes
0
answers
36
views
Mean and std in vaerational autoencoder
Are mean and standard deviation in variational autoencoders equal?
If not, then why are both calculated in the same way?
0
votes
0
answers
125
views
How to optimize ELBO(VAE's loss function)?
Suppose we've got the following formula:
$\log p(x;\theta)=\mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]+KL(q(z|x;\phi)||p(z|x;\theta))\\ \geq \mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\...
- 11
1
vote
1
answer
88
views
What is the best distance measure between policies that are not probability distributions?
This question asks if there is a way to measure distance between policies that are in fact probability distributions.
In the case of continuous control with deterministic policies where they take a ...
- 111
1
vote
0
answers
35
views
How to compare different trajecories in a Markov Decision Process
I realize that my question is a bit fuzzy and I am sorry for that. If needed, I will try to make it more rigorous and precice.
Let $\mathcal{M}$ be a Markov Decision Process, with state space $\...
- 173
1
vote
0
answers
837
views
What is the most suitable measure of the distance between two VAE's latent spaces?
The problem I'm trying to solve is as follows.
I have two separate domains, where inputs do not have the same dimensions. However, I want to create a common feature space between both domains using ...
2
votes
1
answer
85
views
How is this statement from a TensorFlow implementation of a certain KL-divergence formula related to the corresponding formula?
I am trying to understand a certain KL-divergence formula (which can be found on page 6 of the paper Evidential Deep Learning to Quantify Classification Uncertainty) and found a TensorFlow ...
- 33
0
votes
2
answers
770
views
KL divergence coefficient update doesn't make sense in RLlib's PPO implementation
I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss,...
- 233
1
vote
0
answers
589
views
Why does the VAE using a KL-divergence with a non-standard mean does not produce good images?
I know I can make a VAE do generation with a mean of 0 and std-dev of 1.
I tested it with the following loss function:
...
- 53
7
votes
2
answers
2k
views
How is this Pytorch expression equivalent to the KL divergence?
I found the following PyTorch code (from this link)
-0.5 * torch.sum(1 + sigma - mu.pow(2) - sigma.exp())
where mu is the mean ...
- 737
3
votes
1
answer
351
views
How do you calculate KL divergence on a three-dimensional space for a Variational Autoencoder?
I'm trying to implement a variational auto-encoder (as seen in Section 3.1 here: https://arxiv.org/pdf/2004.06271.pdf).
It differs from a traditional VAE because it encodes its input images to three-...
7
votes
2
answers
2k
views
Why is KL divergence used so often in Machine Learning?
The KL Divergence is quite easy to compute in closed form for simple distributions -such as Gaussians- but has some not-very-nice properties. For example, it is not symmetrical (thus it is not a ...
- 233
3
votes
1
answer
554
views
Are there some notions of distance between two policies?
I want to determine some distance between two policies $\pi_1 (a \mid s)$ and $\pi_2 (a \mid s)$, i.e. something like $\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$, where $\pi_i (a\mid ...
- 287
4
votes
1
answer
2k
views
What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?
Variational autoencoders have two components in their loss function. The first component is the reconstruction loss, which for image data, is the pixel-wise difference between the input image and ...
- 151
3
votes
2
answers
1k
views
When should one prefer using Total Variational Divergence over KL divergence in RL
In RL, both the KL divergence (DKL) and Total variational divergence (DTV) are used to measure the distance between two policies. I'm most familiar with using DKL as an early stopping metric during ...
- 531
3
votes
1
answer
532
views
What is the reason for mode collapse in GAN as opposed to WGAN?
In this article I am reading:
$D_{KL}$ gives us inifity when two distributions are disjoint. The value of $D_{JS}$ has sudden jump, not differentiable at $\theta=0$. Only Wasserstein metric provides ...
- 131
3
votes
1
answer
2k
views
Why does the KL divergence not satisfy the triangle inequality?
The KL divergence is defined as
$$D_{KL}=\sum_i p(x_i)log\left(\frac{p(x_i)}{q(x_i)}\right)$$
Why does $D_{KL}$ not satisfy the triangle inequality?
Also, can't you make it satisfy the triangle ...
- 737
6
votes
1
answer
1k
views
Why is the evidence equal to the KL divergence plus the loss?
Why is the equation $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables?
I was ...
- 737
5
votes
1
answer
1k
views
Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?
I have read articles on how Jensen-Shannon divergence is preferred over Kullback-Leibler in measuring how good a distribution mapping is learned in a generative network because of the fact that JS-...
- 1,409
2
votes
1
answer
355
views
How does the Kullback-Leibler divergence give "knowledge gained"?
I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section
Expressed in the ...
- 351
6
votes
2
answers
4k
views
What are the advantages of the Kullback-Leibler over the MSE/RMSE?
I've recently encountered different articles that are recommending to use the KL divergence instead of the MSE/RMSE (as the loss function), when trying to learn a probability distribution, but none of ...
- 1,118
19
votes
1
answer
5k
views
Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?
The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me ...
- 301