13
votes
Accepted
Are softmax outputs of classifiers true probabilities?
The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, ...
- 266
11
votes
Are softmax outputs of classifiers true probabilities?
Excellent question.
The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct.
What usually ...
- 1,112
5
votes
Accepted
Why are there two versions of softmax cross entropy? Which one to use in what situation?
It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
- 2,286
4
votes
Accepted
Which paper introduced the term "softmax"?
The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (...
- 37k
4
votes
Accepted
Why do we use the softmax instead of no activation function?
Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code.
Long answer: ...
- 2,261
3
votes
Should softmax be in the model or in the loss function?
Mathematically it does not matter at all. The results will be the same. However there is a strong reason to prefer it being in the loss function: numeric stability.
Because the loss function knows ...
- 2,005
3
votes
Accepted
Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?
When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus.
If ...
- 278
2
votes
Why are policy gradient methods more effective in high-dimensional action spaces?
Above softmax in action preferences is used for policy gradient methods with (large) spaces with discrete actions, while for continuous spaces with infinite number of actions Gaussian distribution is ...
- 561
1
vote
Why didn't my convolutional image classifier network learn anything?
Here are some points I noticed:
Your data isn't enough for training a deep learning model from scratch. Like you mentioned using a pre-trained model is probably a better alternative like vgg
150 ...
- 21
1
vote
Accepted
Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?
I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different:
The first image shows a linear classifier assigning scores for ...
- 151
1
vote
Accepted
Trouble writing the backpropagation algorithm in python through crossentropy and softmax
I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the ...
- 121
1
vote
Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?
In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the ...
- 26.5k
1
vote
Are there any scale invariant activation functions that outputs probability distribution?
Can't you just divide the numbers by their sum? If your numbers could be negative, you can clamp them to a probability of zero by passing them through a RELU activation.
- 605
1
vote
Accepted
Are any non-injective activation functions used?
There is at least Swish, which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$.
...This suggests that Swish can be loosely viewed as a smooth function
which nonlinearly interpolates between ...
- 605
1
vote
Accepted
Is it normal that the values of the LogSoftmax function are very large negative numbers?
Sounds like it worked to me.
nn.LogSoftmax returns the log of the softmax (duh). The outputs from softmax add up to 1, and form a probability distribution.
0 is the log of 1, meaning that class was ...
- 659
1
vote
Where can I read about the multinoulli distribution?
You can find a description of this distribution (which is also known as categorical distribution, which you probably already heard of) in section 2.3.2 (p. 35) of the book Machine Learning: A ...
- 37k
1
vote
Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?
First of all, check out this question. Generally, you don't need to apply softmax and using raw logits leads to better numerical stability.
The numerical issue that ...
- 2,261
1
vote
Accepted
Is it appropriate to use a softmax activation with a categorical crossentropy loss?
Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE).
Here's the BCE (equation 4.90 from this book)
$$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(...
- 37k
1
vote
How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?
Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation:
$$
\...
- 11
1
vote
What is the advantage of using cross entropy loss & softmax?
Short answer: larger gradients
That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
- 540
1
vote
Accepted
Should I use additional empty category in some categorical problems?
In short: yes, you must allow "do nothing" decision as a first level result.
Your system must decide the action to be taken, including "do nothing" action. This is different to low ...
- 1,282
1
vote
Is the self-attention matrix softmax output (layer 1) symmetric?
I compared my results visually to a second implementation known to be working - "The annotated transformer". I compared the pytorch calculation results of the attention-method to my ...
- 111
1
vote
Why does TensorFlow docs discourage using softmax as activation for the last layer?
This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's slide:
The grey ...
- 11
1
vote
Accepted
Why does TensorFlow docs discourage using softmax as activation for the last layer?
It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over ...
- 256
1
vote
Accepted
Is this neural network with a softmax in the output layer suitable for multi-label classification?
Firstly, you should use sigmoid in your last layer instead of softmax. Softmax returns a probability distribution, meaning that when one labels probability increases the other will decrease, which is ...
- 1,108
Only top scored, non community-wiki answers of a minimum length are eligible