In my understanding, the formula to calculate the cross-entropy is

$$ H(p,q) = - \sum p_i \log(q_i) $$

But in PyTorch nn.CrossEntropyLoss is calculated using this formula:

$$ loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right) $$

that I think it only addresses the $\log(q_i)$ part in the first formula.

Why does PyTorch use a different formula for the cross-entropy?

  • $\begingroup$ You are using the Softmax CE loss, use BCE loss or Binary CE loss for your formula. $\endgroup$ – DuttaA Jan 15 '20 at 8:09

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct.

Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all classes except the "truth" and $p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $ for the correct prediction.

Therefore, your loss reduces to: $$ H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth}) $$

  • $\begingroup$ I'm not sure whether OP wanted to know this. $\endgroup$ – DuttaA Jan 15 '20 at 8:11
  • $\begingroup$ @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. . $\endgroup$ – Sammy Jan 15 '20 at 8:23
  • $\begingroup$ OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch nn.CrossEntropyLoss. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You! $\endgroup$ – malioboro Jan 15 '20 at 8:35
  • 1
    $\begingroup$ @Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;) $\endgroup$ – nbro Jan 15 '20 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.