I've recently encounter different articles that are recommending to use KL instead of L1/L2 norm when trying to minimize a probability distribution. But none of the articles are giving a clear reasoning why ones is better than the other. Could anyone give me a strong argument why KL is suitable for this?

  • 1
    $\begingroup$ Minimize a probablity distribution? Or decrease the difference between 2 probablity distributions? As far as I know KL divergence penalty term is used to Basically fit a probablity distribution to a standard probablity distribution. $\endgroup$
    – user9947
    Sep 11 '19 at 7:09
  • $\begingroup$ You're right, decrease the difference between 2 probability distributions. $\endgroup$
    – razvanc92
    Sep 11 '19 at 7:10
  • $\begingroup$ Can you link the architecture/model you are trying to implement? $\endgroup$
    – user9947
    Sep 11 '19 at 7:32
  • 1
    $\begingroup$ Realize that the expression "minimize a probability distribution" is not very meaningful. In machine learning, KL is used when you are trying to approximate an (intractable) probability distribution. See the details of the variational auto-encoder. L1/L2 is often used for regularization. $\endgroup$
    – nbro
    Sep 11 '19 at 11:37
  • $\begingroup$ @nbro You're absolutely right I messed up the terminologies. What I wanted to ask was: why I can't use mse or rmse as a training error when predicting a distribution. $\endgroup$
    – razvanc92
    Sep 12 '19 at 10:46

Considering your question was asking why KL-Divergence would be preferred over MSE, RMSE, or L1/L2 Norm between ground truth and predicted, as loss function for predicting a distribution -

KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.

L2-norm/MSE/RMSE doesn’t do well with probabilities, because of the power operations involved in the calculation of loss. Probabilities, being fractions under 1, are significantly effected by any power operations (Square or Root), and considering we are calculating the squares of differences of probabilities, the values that are summed are abnormally small, essentially barely learning anything as the random initialization itself starts with an abnormally small loss, almost always staying constant.

L1 norm on the other hand, does not have any power operations, making it relatively acceptable.

Loss functions such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence are preffered for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or in other words, assuming $Q$ is the ground truth distribution, KL-Divergence is a measure of how much $P$ deviates from $Q$. Also considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.

More clarity on the motivation behind Kullback-Leibler can be read here. Hope your query has been answered!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.