# Why does PyTorch use a different formula for the cross-entropy?

In my understanding, the formula to calculate the cross-entropy is

$$H(p,q) = - \sum p_i \log(q_i)$$

But in PyTorch nn.CrossEntropyLoss is calculated using this formula:

$$loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right)$$

that I think it only addresses the $$\log(q_i)$$ part in the first formula.

Why does PyTorch use a different formula for the cross-entropy?

• You are using the Softmax CE loss, use BCE loss or Binary CE loss for your formula. – DuttaA Jan 15 at 8:09

When you one-hot-encode your labels with $$p_i \in \{0,1\}$$ you get $$p_i = 0$$ iff $$i$$ is not correct and, equivalently, $$p_i =1$$ iff $$i$$ is correct.
Hence, $$p_i \log(q_i) = 0 \log(q_i) = 0$$ for all classes except the "truth" and $$p_i \log(q_i) = 1 \log(q_i) = \log(q_i)$$ for the correct prediction.
Therefore, your loss reduces to: $$H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})$$
• @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. . – Sammy Jan 15 at 8:23
• OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch nn.CrossEntropyLoss. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You! – malioboro Jan 15 at 8:35