16 votes
Accepted

Are softmax outputs of classifiers true probabilities?

The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, ...
D.W.'s user avatar
  • 307
12 votes

Are softmax outputs of classifiers true probabilities?

Excellent question. The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct. What usually ...
Dr. Snoopy's user avatar
  • 1,355
7 votes
Accepted

Which paper introduced the term "softmax"?

The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (...
nbro's user avatar
  • 40.5k
5 votes
Accepted

Why are there two versions of softmax cross entropy? Which one to use in what situation?

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
Brale's user avatar
  • 2,386
4 votes
Accepted

Why do we use the softmax instead of no activation function?

Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code. Long answer: ...
Kostya's user avatar
  • 2,524
3 votes
Accepted

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus. If ...
Uskebasi's user avatar
  • 278
3 votes

Should softmax be in the model or in the loss function?

Mathematically it does not matter at all. The results will be the same. However there is a strong reason to prefer it being in the loss function: numeric stability. Because the loss function knows ...
chessprogrammer's user avatar
2 votes

Is the self-attention matrix softmax output (layer 1) symmetric?

I compared my results visually to a second implementation known to be working - "The annotated transformer". I compared the pytorch calculation results of the attention-method to my ...
thepacker's user avatar
  • 131
2 votes

Why does TensorFlow docs discourage using softmax as activation for the last layer?

This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's slide: The grey ...
xeonqq's user avatar
  • 21
2 votes
Accepted

Why does TensorFlow docs discourage using softmax as activation for the last layer?

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over ...
ted's user avatar
  • 276
2 votes
Accepted

Since $f_c$ returns the probability of class label $c$, we require $0 \le f_c \le 1$ for each $c$, and $\sum_{c = 1}^C f_c = 1$. Why avoid this?

Your intuition is correct. The restrictions you wrote down are necessary by definition. The author means that it is hard to build a machine learning model that gives probability by design ( returns $...
vl_knd's user avatar
  • 498
2 votes
Accepted

Dealing with noise in models with softmax output

I don't fully agree with the other answer, mainly on point 1. First thing first, adding a "nothing" label, means that you also have to gather new data to train that "nothing" ...
Alberto's user avatar
  • 1,915
1 vote

Why didn't my convolutional image classifier network learn anything?

Here are some points I noticed: Your data isn't enough for training a deep learning model from scratch. Like you mentioned using a pre-trained model is probably a better alternative like vgg 150 ...
kimo26's user avatar
  • 21
1 vote

Why are policy gradient methods more effective in high-dimensional action spaces?

Above softmax in action preferences is used for policy gradient methods with (large) spaces with discrete actions, while for continuous spaces with infinite number of actions Gaussian distribution is ...
cinch's user avatar
  • 2,272
1 vote

Number of units in Final softmax layer in VGGNet16

You should do the first one: add a layer with only 4 units instead of the 1000 unit layer. You can think of the first $N-1$ layers as a feature extractor, converting the high-dimensional image to a (...
Alexander Wan's user avatar
1 vote

Backpropagation with CrossEntropy and Softmax, HOW?

The derivation of the softmax function is a bit tricky because except the other common activation functions(sigmoid, relu...) all the Zs values impact each other. It is because when you are ...
Ege's user avatar
  • 25
1 vote

Backpropagation with CrossEntropy and Softmax, HOW?

No it's not the same. First the derivative of the cost function is taken with respect to the weight and not the input. This is usually done using the chain rule of calculus. To calculate this using ...
Chiho's user avatar
  • 11
1 vote
Accepted

Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different: The first image shows a linear classifier assigning scores for ...
Foobar's user avatar
  • 153
1 vote
Accepted

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the ...
user605734 MBS's user avatar
1 vote

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the ...
Neil Slater's user avatar
  • 32.1k
1 vote

Are there any scale invariant activation functions that outputs probability distribution?

Can't you just divide the numbers by their sum? If your numbers could be negative, you can clamp them to a probability of zero by passing them through a RELU activation.
NikoNyrh's user avatar
  • 767
1 vote
Accepted

Are any non-injective activation functions used?

There is at least Swish, which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$. ...This suggests that Swish can be loosely viewed as a smooth function which nonlinearly interpolates between ...
NikoNyrh's user avatar
  • 767
1 vote
Accepted

Is it normal that the values of the LogSoftmax function are very large negative numbers?

Sounds like it worked to me. nn.LogSoftmax returns the log of the softmax (duh). The outputs from softmax add up to 1, and form a probability distribution. 0 is the log of 1, meaning that class was ...
user253751's user avatar
1 vote

Where can I read about the multinoulli distribution?

You can find a description of this distribution (which is also known as categorical distribution, which you probably already heard of) in section 2.3.2 (p. 35) of the book Machine Learning: A ...
nbro's user avatar
  • 40.5k
1 vote

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

First of all, check out this question. Generally, you don't need to apply softmax and using raw logits leads to better numerical stability. The numerical issue that ...
Kostya's user avatar
  • 2,524
1 vote
Accepted

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE). Here's the BCE (equation 4.90 from this book) $$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(...
nbro's user avatar
  • 40.5k
1 vote

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation: $$ \...
kid's user avatar
  • 11
1 vote

What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
S2673's user avatar
  • 590
1 vote
Accepted

Should I use additional empty category in some categorical problems?

In short: yes, you must allow "do nothing" decision as a first level result. Your system must decide the action to be taken, including "do nothing" action. This is different to low ...
pasaba por aqui's user avatar
1 vote
Accepted

Is this neural network with a softmax in the output layer suitable for multi-label classification?

Firstly, you should use sigmoid in your last layer instead of softmax. Softmax returns a probability distribution, meaning that when one labels probability increases the other will decrease, which is ...
razvanc92's user avatar
  • 1,128

Only top scored, non community-wiki answers of a minimum length are eligible