# Tag Info

Accepted

### Are softmax outputs of classifiers true probabilities?

The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, ...
• 307

### Are softmax outputs of classifiers true probabilities?

Excellent question. The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct. What usually ...
• 1,355
Accepted

### Which paper introduced the term "softmax"?

The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (...
• 40.5k
Accepted

### Why are there two versions of softmax cross entropy? Which one to use in what situation?

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
• 2,386
Accepted

### Why do we use the softmax instead of no activation function?

Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code. Long answer: ...
• 2,524
Accepted

### Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus. If ...
• 278

### Should softmax be in the model or in the loss function?

Mathematically it does not matter at all. The results will be the same. However there is a strong reason to prefer it being in the loss function: numeric stability. Because the loss function knows ...
• 2,890

### Is the self-attention matrix softmax output (layer 1) symmetric?

I compared my results visually to a second implementation known to be working - "The annotated transformer". I compared the pytorch calculation results of the attention-method to my ...
• 131

### Why does TensorFlow docs discourage using softmax as activation for the last layer?

This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's slide: The grey ...
• 21
Accepted

### Why does TensorFlow docs discourage using softmax as activation for the last layer?

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over ...
• 276
Accepted

• 590
1 vote
Accepted

### Should I use additional empty category in some categorical problems?

In short: yes, you must allow "do nothing" decision as a first level result. Your system must decide the action to be taken, including "do nothing" action. This is different to low ...
• 1,283
1 vote
Accepted

### Is this neural network with a softmax in the output layer suitable for multi-label classification?

Firstly, you should use sigmoid in your last layer instead of softmax. Softmax returns a probability distribution, meaning that when one labels probability increases the other will decrease, which is ...
• 1,128

Only top scored, non community-wiki answers of a minimum length are eligible