Questions tagged [softmax]
For questions related to the softmax function, which a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. The softmax is often used as the activation function of the output layer of a neural network.
10
questions
1
vote
0answers
36 views
How am I supposed to code equation 4.57 from the book “Machine Learning: An Algorithmic Perspective”?
Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained
$$\delta_o(\kappa) = (y_\kappa -...
1
vote
1answer
160 views
Why are there two versions of softmax cross entropy? Which one to use in what situation?
I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one?
For example in this Quora answer, there are 2 answers:
$L(\mathbf{w})=\frac{1}{N} \sum_{n=1}^...
3
votes
2answers
93 views
What is the advantage of using cross entropy loss & softmax?
I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
-2
votes
1answer
30 views
1
vote
2answers
53 views
Should I use additional empty category in some categorical problems?
I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
2
votes
1answer
85 views
Which paper introduced the term “softmax”?
Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned ...
1
vote
0answers
33 views
Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?
Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?
Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning ...
0
votes
1answer
69 views
Transformers - is the self attention matrix softmax output (layer 1) symmetric?
Let's assume, that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by it transposed version we receive a matrix of 49 by 49. Which is symmetric. ...
1
vote
0answers
31 views
What do the authors of this paper mean by the bias term in this picture of a neural network implementation?
I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (...
2
votes
1answer
208 views
Why does TensorFlow docs discourage using softmax as activation for the last layer?
The beginner colab example for tensorflow states:
Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network....
