Forward KL Divergence (also known as cross entropy loss) is a standard loss function in supervised learning problems. I understand why it is so: matching a known a trained distribution to a known distribution fits $P \log(P/Q)$ where $P$ is the known distribution.

Why isn't the reverse KL divergence commonly used in supervised learning?

  • 1
    $\begingroup$ Maybe say a little more about what reverse KL divergence is. $\endgroup$ Apr 5 '19 at 19:54
  • $\begingroup$ Reverse KL Divergence would be $Q \log (Q/P)$ $\endgroup$
    – cgo
    Apr 8 '19 at 0:24

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.

A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are just one example of supervised learning, so this answer might not fully cover your question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.