# Why does PyTorch use a different formula for the cross-entropy?

In my understanding, the formula to calculate the cross-entropy is

$$H(p,q) = - \sum p_i \log(q_i)$$

But in PyTorch nn.CrossEntropyLoss is calculated using this formula:

$$loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right)$$

that I think it only addresses the $$\log(q_i)$$ part in the first formula.

Why does PyTorch use a different formula for the cross-entropy?

• You are using the Softmax CE loss, use BCE loss or Binary CE loss for your formula.
– user9947
Jan 15 '20 at 8:09

When you one-hot-encode your labels with $$p_i \in \{0,1\}$$ you get $$p_i = 0$$ iff $$i$$ is not correct and, equivalently, $$p_i =1$$ iff $$i$$ is correct.

Hence, $$p_i \log(q_i) = 0 \log(q_i) = 0$$ for all classes except the "truth" and $$p_i \log(q_i) = 1 \log(q_i) = \log(q_i)$$ for the correct prediction.

Therefore, your loss reduces to: $$H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})$$

• I'm not sure whether OP wanted to know this.
– user9947
Jan 15 '20 at 8:11
• @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. . Jan 15 '20 at 8:23
• OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch nn.CrossEntropyLoss. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You! Jan 15 '20 at 8:35
• @Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;)
– nbro
Jan 15 '20 at 13:17