Return to Answer

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$, but also would tend to zero if you would work around this by defining dummy probabilities $p_j = \epsilon \rightarrow 0.$ Minimizing reverse cross-entropy would become minimizing the entropy $H(Q)$, which, however, has additional undesired minima at $p_i = 0$ and $p_j = 1$ for some $j$.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.

A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are just one example of supervised learning, so this answer might not fully cover your question.

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.

A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are just one example of supervised learning, so this answer might not fully cover your question.

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class and the class membership is represented by one-hot encoding, which corresponds to P being 1 for the class the data point belongs to and 0 for all other classes. The loss function mosty used in this problems is the average categorical cross-entropy $E_p[-log(q)] = H(p) - D(p || q)$, which in the case of single-class membership reduces to the binary cross-entropy, because only one P value is non-zero and the cross-entropy becomes $-\log(q)$. The reverse KL-Divergence is strictly speaking not even defined in this case, as P needs to be absolute continuous w.r.t. Q.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply. From a more conceptual point of view, this answer summaries nicely the relation of cross entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a sample drawn from the model under the (empirical) distribution of the observed data. A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are of course just one case of supervised learning, so this answer might not fully cover your question. And sorry for the bit sloppy notation, I will correct that later when I have a bit more time..

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$, but also would tend to zero if you would work around this by defining dummy probabilities $p_j = \epsilon \rightarrow 0.$ Minimizing reverse cross-entropy would become minimizing the entropy $H(Q)$, which, however, has additional undesired minima at $p_i = 0$ and $p_j = 1$ for some $j$.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data. 

A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are just one example of supervised learning, so this answer might not fully cover your question.

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class and the class membership is represented by one-hot encoding, which corresponds to P being 1 for the class the data point belongs to and 0 for all other classes. The loss function mosty used in this problems is the average categorical cross-entropy E_p[-log(q)] = H(p) - D(p||q), which in the case of single-class membership reduces to the binary cross-entropy, because only one P value is non-zero and the cross-entropy becomes -log(q). The reverse KL-Divergence is strictly speaking not even defined in this case, as P needs to be absolute continuous w.r.t. Q.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply. From a more conceptual point of view, this answer summaries nicely the relation of cross entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a sample drawn from the model under the (empirical) distribution of the observed data. A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are of course just one case of supervised learning, so this answer might not fully cover your question. And sorry for the bit sloppy notation, I will correct that later when I have a bit more time..

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class and the class membership is represented by one-hot encoding, which corresponds to P being 1 for the class the data point belongs to and 0 for all other classes. The loss function mosty used in this problems is the average categorical cross-entropy $E_p[-log(q)] = H(p) - D(p || q)$, which in the case of single-class membership reduces to the binary cross-entropy, because only one P value is non-zero and the cross-entropy becomes $-\log(q)$. The reverse KL-Divergence is strictly speaking not even defined in this case, as P needs to be absolute continuous w.r.t. Q.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply. From a more conceptual point of view, this answer summaries nicely the relation of cross entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a sample drawn from the model under the (empirical) distribution of the observed data. A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are of course just one case of supervised learning, so this answer might not fully cover your question. And sorry for the bit sloppy notation, I will correct that later when I have a bit more time..