Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

Question

The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me than the entropy of the target distribution, which is what the cross-entropy is on a match.

I'm not saying there's more information in one of the other except that a human view may find a zero more intuitive than a positive. Of course, one usually uses an evaluative method to really see how well classification occurs. But is the choice of the cross-entropy over the KL divergence historic?

Answer 1

When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence are equal.

As already stated in the question, the general formula is this:

$$H(p, q) = H(p) + D_{KL}(p \parallel q),$$

where $p$ is the "true"/target distribution and $q$ is an estimated distribution, $H(p, q)$ is the cross-entropy, $H(p)$ is the entropy and $D$ is the Kullback-Leibler divergence.

Note that in machine learning, $p$ is a one-hot representation of the ground-truth class, i.e.,

$$p = [0,..., 1, ..., 0]$$

which is basically a delta-function distribution. But the entropy of the delta function is zero, hence KL divergence simply equals the cross-entropy.

In fact, even if $H(p)$ wasn't $0$ (e.g., soft labels), it is fixed and has no contribution to the gradient. In terms of optimization, it's safe to simply remove it and optimize the Kullback-Leibler divergence.