# How does the Kullback-Leibler divergence give “knowledge gained”?

I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section

Expressed in the language of Bayesian inference, $$D_{\text{KL}}(P\parallel Q)}$$ is a measure of the information gained by revising one's beliefs from the prior probability distribution $$Q$$ to the posterior probability distribution $$P$$

I was under the impression that KL divergence is a way of measure the difference between distributions (used in autoencoders to determine the difference between the input and the output generated from latents).

How does the equation $$D_{KL}(P \| Q)=\sum P(x) \log \left( \frac{P(X)}{Q(X)} \right)$$ give us a divergence? Also, in encoding and decoding algorithms that use KL divergence, is the goal to minimize $$D_{KL}(P \| Q)$$?

## 1 Answer

You can know it better, if you know the concept of entropy:

Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event $$E}$$ is an increasing function of the reciprocal of the $$p(E)}$$ of the event, precisely $$I(E)=-\log _{2}(p(E))=\log _{2}(1/p(E))}$$. Shannon defined the entropy Η of a discrete random variable $${\textstyle X}$$ with possible values $${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$$ and probability mass function $${\textstyle \mathrm {P} (X)}$$ as: $$\mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$ Here $$\operatorname {E} }$$ is the expected value operator, and $$I$$ is the information content of $$I(X)}$$ is itself a random variable. The entropy can explicitly be written as $$\mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}}$$ where $$b$$ is the base of the logarithm used.

Now, the KL divergence, try to find the cross entropy of two probability distributions.

So, what is the cross entropy:

The cross entropy between two probability distributions $$p}$$ and $$q}$$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution $$q}$$, rather than the true distribution $$p}$$.

The cross entropy for the distributions $$p}$$ and $$q}$$ over a given set is defined as follows: $$H(p,q)=\operatorname {E} _{p}[-\log q]}.$$ The definition may be formulated using the Kullback–Leibler divergence$$D_{\mathrm {KL} }(p\|q)}$$ of $$q}$$ from $$p}$$ (also known as the relative entropy of $$p}$$ with respect to $$q}$$). $$H(p,q)=H(p)+D_{\mathrm {KL} }(p\|q)},$$ where $$H(p)}$$ is the entropy of $$p}$$.

Now, as we want to approximate a distribution function $$p$$ with other distribution function $$q$$, we want to minimize the cross entropy between these two. The first part $$H(p)$$ could not be changed as we want to find a distribution $$q$$ and $$p$$ is given. Hence, we need to minimize $$KL$$ divergence of these two, to minimize the cross entropy and better approximation for the $$p$$ distribution.