1
$\begingroup$

In my understanding, the formula to calculate the cross-entropy is

$$ H(p,q) = - \sum p_i \log(q_i) $$

But in PyTorch nn.CrossEntropyLoss is calculated using this formula:

$$ loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right) $$

that I think it only addresses the $\log(q_i)$ part in the first formula.

Why does PyTorch use a different formula for the cross-entropy?

Improve this question
$\endgroup$
1
  • $\begingroup$ You are using the Softmax CE loss, use BCE loss or Binary CE loss for your formula. $\endgroup$
    – user9947
    Commented Jan 15, 2020 at 8:09

1 Answer 1

Reset to default
1
$\begingroup$

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct.

Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all classes except the "truth" and $p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $ for the correct prediction.

Therefore, your loss reduces to: $$ H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth}) $$

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm not sure whether OP wanted to know this. $\endgroup$
    – user9947
    Commented Jan 15, 2020 at 8:11
  • $\begingroup$ @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. . $\endgroup$
    – Jonathan
    Commented Jan 15, 2020 at 8:23
  • $\begingroup$ OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch nn.CrossEntropyLoss. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You! $\endgroup$
    – malioboro
    Commented Jan 15, 2020 at 8:35
  • 1
    $\begingroup$ @Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;) $\endgroup$
    – nbro
    Commented Jan 15, 2020 at 13:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .