Timeline for Why do we use the softmax instead of no activation function?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 7, 2021 at 19:28 | comment | added | Kostya | @nbro I kinda started explaining why use of logits is more numerically stable, but it turned out to be too mush of a tangent. So that deserves a separate question and answer in my opinion. | |
| May 7, 2021 at 19:21 | comment | added | nbro |
If you directly use the logits as "the logarithms of the probability", then what you say should make sense. However, why should you interpret the output of the neural network as the logarithm of the probability of belonging to a class? Still, it would be nice to show what TF exactly does when you set from_logits=True.
|
|
| May 7, 2021 at 19:09 | comment | added | nbro | Note that the cross-entropy is really defined for 2 probability distributions, and I would say that's the reason why we use the softmax. Note that, if the logits are zero or negative, then the cross-entropy is not defined because of the logarithm. I think that TensorFlow accepts logits because it probably performs a (stable) conversion of the logits to a probability distribution under the hood, but I have not looked at the source code. So, it's a matter of diving into the TF implementation of the CE loss. | |
| May 7, 2021 at 18:41 | vote | accept | dato nefaridze | ||
| May 7, 2021 at 17:22 | history | edited | Kostya | CC BY-SA 4.0 |
added 40 characters in body
|
| May 7, 2021 at 17:11 | history | answered | Kostya | CC BY-SA 4.0 |