Cross entropy is identical to the KL divergence plus entropy of target distribution. KL equals to zero when the two distributions are the same, which seems more intuitive to me than the entropy of the target distribution, which is what cross entropy is on a match.

I'm not saying there's more information in one of the other except that a human view may find a zero more intuitive than a positive. Of course, one usually uses a evaluative method to really see how well classification occurs. But is the choice of cross entropy over KL historic?


When it comes to classification problem in machine learning, the cross entropy and KL divergence are equal. As already stated in the question, the general formula is this:

$$H(p, q) = H(p) + D_{KL}(p||q)$$

Where $p$ a “true” distribution and $q$ is an estimated distribution, $H(p, q)$ is the cross-entropy, $H(p)$ is the entropy and $D$ is the Kullback-Leibler divergence.

Note that in machine learning, $p$ is a one-hot representation of the ground-truth class, i.e.,

$$p = [0,..., 1, ..., 0]$$

which is basically a delta-function distribution. But the entropy of the delta function is zero, hence KL divergence simply equals the cross-entropy.

In fact, even if $H(p)$ wasn't $0$ (e.g., soft labels), it is fixed and has no contribution to the gradient. In terms of optimization, it's safe to simply remove it and optimize the Kullback-Leibler divergence.


Cross-entropy is an entropy, not an entropy difference.

A more natural and perhaps intuitive way to conceptualize the categorization criteria is through a relation rather than a definition.

$H(P, Q) - H(P) = D_{\mathrm{KL}}(P\|Q) = - \sum_i P(i) \log\frac{Q(i)}{P(i)}$

This follows parallels, identified by Claude Shannon with John von Neumann, between quantum mechanical thermodynamics and information theory. Entropy is not an absolute quantity. It is a relative one, so neither entropy nor cross entropy can be calculated, but their difference can be for either the discrete case above or its continuous sibling below.

$H(P, Q) - H(P) = D_{\mathrm{KL}}(P\|Q) = - \int_{-\infty}^\infty \, p(x) \log\frac {q(x)} {p(x)} \, dx$

Although we may see $H(...) = ...$ in the literature, with no H'(...) on the right hand side of the equation, it is not technically accurate. In such cases there is always some implied entropy to which the entropy on the left hand side is relative.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.