I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section

Expressed in the language of Bayesian inference, ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is a measure of the information gained by revising one's beliefs from the prior probability distribution $Q$ to the posterior probability distribution $P$

I was under the impression that KL divergence is a way of measure the difference between distributions (used in autoencoders to determine the difference between the input and the output generated from latents).

How does the equation $$D_{KL}(P \| Q)=\sum P(x) \log \left( \frac{P(X)}{Q(X)} \right)$$ give us a divergence? Also, in encoding and decoding algorithms that use KL divergence, is the goal to minimize $D_{KL}(P \| Q)$?


1 Answer 1


You can know it better, if you know the concept of entropy:

Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event ${\displaystyle E}$ is an increasing function of the reciprocal of the ${\displaystyle p(E)}$ of the event, precisely ${\displaystyle I(E)=-\log _{2}(p(E))=\log _{2}(1/p(E))}$. Shannon defined the entropy Η of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ as: $${\displaystyle \mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$ Here ${\displaystyle \operatorname {E} }$ is the expected value operator, and $I$ is the information content of ${\displaystyle I(X)}$ is itself a random variable. The entropy can explicitly be written as $${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}}$$ where $b$ is the base of the logarithm used.

Now, the KL divergence, try to find the cross entropy of two probability distributions.

So, what is the cross entropy:

The cross entropy between two probability distributions ${\displaystyle p}$ and ${\displaystyle q}$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution ${\displaystyle q}$, rather than the true distribution ${\displaystyle p}$.

The cross entropy for the distributions ${\displaystyle p}$ and ${\displaystyle q}$ over a given set is defined as follows: $${\displaystyle H(p,q)=\operatorname {E} _{p}[-\log q]}.$$ The definition may be formulated using the Kullback–Leibler divergence${\displaystyle D_{\mathrm {KL} }(p\|q)}$ of ${\displaystyle q}$ from ${\displaystyle p}$ (also known as the relative entropy of ${\displaystyle p}$ with respect to ${\displaystyle q}$). $${\displaystyle H(p,q)=H(p)+D_{\mathrm {KL} }(p\|q)},$$ where ${\displaystyle H(p)}$ is the entropy of ${\displaystyle p}$.

Now, as we want to approximate a distribution function $p$ with other distribution function $q$, we want to minimize the cross entropy between these two. The first part $H(p)$ could not be changed as we want to find a distribution $q$ and $p$ is given. Hence, we need to minimize $KL$ divergence of these two, to minimize the cross entropy and better approximation for the $p$ distribution.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .