# Advantages of Kullback-Leibler over L1/L2?

I've recently encounter different articles that are recommending to use KL instead of L1/L2 norm when trying to minimize a probability distribution. But none of the articles are giving a clear reasoning why ones is better than the other. Could anyone give me a strong argument why KL is suitable for this?

• Minimize a probablity distribution? Or decrease the difference between 2 probablity distributions? As far as I know KL divergence penalty term is used to Basically fit a probablity distribution to a standard probablity distribution. – DuttaA Sep 11 '19 at 7:09
• You're right, decrease the difference between 2 probability distributions. – razvanc92 Sep 11 '19 at 7:10
• Can you link the architecture/model you are trying to implement? – DuttaA Sep 11 '19 at 7:32
• Realize that the expression "minimize a probability distribution" is not very meaningful. In machine learning, KL is used when you are trying to approximate an (intractable) probability distribution. See the details of the variational auto-encoder. L1/L2 is often used for regularization. – nbro Sep 11 '19 at 11:37
• @nbro You're absolutely right I messed up the terminologies. What I wanted to ask was: why I can't use mse or rmse as a training error when predicting a distribution. – razvanc92 Sep 12 '19 at 10:46

Loss functions such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence are preffered for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or in other words, assuming $$Q$$ is the ground truth distribution, KL-Divergence is a measure of how much $$P$$ deviates from $$Q$$. Also considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.