Questions tagged [softmax]
For questions related to the softmax function, which a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. The softmax is often used as the activation function of the output layer of a neural network.
13
questions
1
vote
0answers
27 views
Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm
This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question.
In the A3C algorithm from the original ...
1
vote
1answer
58 views
How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?
I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...
0
votes
0answers
28 views
Backpropagation implementation not applicable for other cases
I saw this implementation of backpropagation in MATLAB, where the loss function used is MSE, and the last layer's activation function was sigmoid.
I denoted the portions of the formula for what I ...
0
votes
1answer
71 views
Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?
If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
1
vote
0answers
40 views
How am I supposed to code equation 4.57 from the book “Machine Learning: An Algorithmic Perspective”?
Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained
$$\delta_o(\kappa) = (y_\kappa -...
1
vote
1answer
169 views
Why are there two versions of softmax cross entropy? Which one to use in what situation?
I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one?
For example in this Quora answer, there are 2 answers:
$L(\mathbf{w})=\frac{1}{N} \sum_{n=1}^...
3
votes
2answers
200 views
What is the advantage of using cross entropy loss & softmax?
I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
1
vote
2answers
57 views
Should I use additional empty category in some categorical problems?
I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
2
votes
1answer
101 views
Which paper introduced the term “softmax”?
Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned ...
1
vote
0answers
33 views
Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?
Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?
Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning ...
0
votes
1answer
124 views
Transformers - is the self attention matrix softmax output (layer 1) symmetric?
Let's assume, that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by it transposed version we receive a matrix of 49 by 49. Which is symmetric. ...
1
vote
0answers
33 views
What do the authors of this paper mean by the bias term in this picture of a neural network implementation?
I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (...
3
votes
2answers
351 views
Why does TensorFlow docs discourage using softmax as activation for the last layer?
The beginner colab example for tensorflow states:
Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network....