# Tag Info

Accepted

### Are softmax outputs of classifiers true probabilities?

The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, ...
• 266

### Are softmax outputs of classifiers true probabilities?

Excellent question. The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct. What usually ...
• 1,112
Accepted

### Why are there two versions of softmax cross entropy? Which one to use in what situation?

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
• 2,286
Accepted

### Which paper introduced the term "softmax"?

The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (...
• 37k
Accepted

### Why do we use the softmax instead of no activation function?

Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code. Long answer: ...
• 2,261

### Should softmax be in the model or in the loss function?

Mathematically it does not matter at all. The results will be the same. However there is a strong reason to prefer it being in the loss function: numeric stability. Because the loss function knows ...
• 2,005
Accepted

### Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus. If ...
• 278

### Why are policy gradient methods more effective in high-dimensional action spaces?

Above softmax in action preferences is used for policy gradient methods with (large) spaces with discrete actions, while for continuous spaces with infinite number of actions Gaussian distribution is ...
• 561
1 vote

### Why didn't my convolutional image classifier network learn anything?

Here are some points I noticed: Your data isn't enough for training a deep learning model from scratch. Like you mentioned using a pre-trained model is probably a better alternative like vgg 150 ...
• 21
1 vote
Accepted

### Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different: The first image shows a linear classifier assigning scores for ...
• 151
1 vote
Accepted

### Trouble writing the backpropagation algorithm in python through crossentropy and softmax

I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the ...
1 vote

### Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the ...
• 26.5k
1 vote

### Are there any scale invariant activation functions that outputs probability distribution?

Can't you just divide the numbers by their sum? If your numbers could be negative, you can clamp them to a probability of zero by passing them through a RELU activation.
• 605
1 vote
Accepted

### Are any non-injective activation functions used?

There is at least Swish, which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$. ...This suggests that Swish can be loosely viewed as a smooth function which nonlinearly interpolates between ...
• 605
1 vote
Accepted

### Is it normal that the values of the LogSoftmax function are very large negative numbers?

Sounds like it worked to me. nn.LogSoftmax returns the log of the softmax (duh). The outputs from softmax add up to 1, and form a probability distribution. 0 is the log of 1, meaning that class was ...
• 659
1 vote

You can find a description of this distribution (which is also known as categorical distribution, which you probably already heard of) in section 2.3.2 (p. 35) of the book Machine Learning: A ...
• 37k
1 vote

### Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

First of all, check out this question. Generally, you don't need to apply softmax and using raw logits leads to better numerical stability. The numerical issue that ...
• 2,261
1 vote
Accepted

• 11
1 vote

### What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
• 540
1 vote
Accepted

### Should I use additional empty category in some categorical problems?

In short: yes, you must allow "do nothing" decision as a first level result. Your system must decide the action to be taken, including "do nothing" action. This is different to low ...
• 1,282
1 vote

### Is the self-attention matrix softmax output (layer 1) symmetric?

I compared my results visually to a second implementation known to be working - "The annotated transformer". I compared the pytorch calculation results of the attention-method to my ...
• 111
1 vote

### Why does TensorFlow docs discourage using softmax as activation for the last layer?

This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's slide: The grey ...
• 11
1 vote
Accepted

### Why does TensorFlow docs discourage using softmax as activation for the last layer?

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over ...
• 256
1 vote
Accepted

### Is this neural network with a softmax in the output layer suitable for multi-label classification?

Firstly, you should use sigmoid in your last layer instead of softmax. Softmax returns a probability distribution, meaning that when one labels probability increases the other will decrease, which is ...
• 1,108

Only top scored, non community-wiki answers of a minimum length are eligible