Newest 'softmax' Questions

0

votes

1answer

43 views

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...

asked Jun 3 at 15:46

Liuuuuk

1

0

votes

0answers

38 views

How is it possible that the softmax combined with the MSE in a molecule classification task performs than than the cross-entropy?

I'm working on a GNN project associated with molecule classification. The project is to classify if the atom in the molecule will initiate a certain reaction. For example, a molecule can be ...

deep-learning classification objective-functions graph-neural-networks softmax

asked May 8 at 3:24

whtitefall

101

2

votes

1answer

98 views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...

neural-networks activation-function softmax multiclass-classification

asked May 7 at 16:04

dato nefaridze

6421 silver badge9 bronze badges

-1

votes

1answer

35 views

Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...

neural-networks reinforcement-learning tensorflow python softmax

asked May 1 at 16:14

fardis nadimi

191 bronze badge

0

votes

1answer

51 views

XOR Neural Network gets stuck in training

I'm trying to create a neural network to simulate a XOR gate. Here's my dataset: ...

neural-networks softmax xor-problem categorical-crossentropy

asked Mar 25 at 18:55

user45708

31 bronze badge

1

vote

1answer

243 views

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....

binary-classification sigmoid softmax categorical-crossentropy binary-crossentropy

asked Mar 16 at 16:20

user9317212

978 bronze badges

1

vote

0answers

32 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...

reinforcement-learning policy-gradients softmax a3c

asked Jan 31 at 18:25

Kagaratsch

1112 bronze badges

1

vote

1answer

72 views

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...

deep-learning backpropagation math cross-entropy softmax

asked Jan 16 at 10:48

Ilknur Mustafa

153 bronze badges

0

votes

0answers

33 views

Backpropagation implementation not applicable for other cases

I saw this implementation of backpropagation in MATLAB, where the loss function used is MSE, and the last layer's activation function was sigmoid. I denoted the portions of the formula for what I ...

deep-learning backpropagation gradient-descent linear-algebra softmax

asked Jan 6 at 11:55

Ilknur Mustafa

153 bronze badges

0

votes

1answer

75 views

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...

backpropagation objective-functions gradient-descent linear-algebra softmax

asked Jan 2 at 17:22

Ilknur Mustafa

153 bronze badges

1

vote

0answers

43 views

How am I supposed to code equation 4.57 from the book “Machine Learning: An Algorithmic Perspective”?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained $$\delta_o(\kappa) = (y_\kappa -...

machine-learning python backpropagation implementation softmax

asked Nov 3 '20 at 9:01

NewToCoding

112 bronze badges

1

vote

1answer

180 views

Why are there two versions of softmax cross entropy? Which one to use in what situation?

I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one? For example in this Quora answer, there are 2 answers: $L(\mathbf{w})=\frac{1}{N} \sum_{n=1}^...

machine-learning comparison cross-entropy softmax categorical-crossentropy

asked Oct 24 '20 at 1:08

Herbert

1354 bronze badges

4

votes

2answers

498 views

What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...

neural-networks gradient-descent cross-entropy mean-squared-error softmax

asked Oct 11 '20 at 0:13

Ben

3771 silver badge8 bronze badges

1

vote

2answers

61 views

Should I use additional empty category in some categorical problems?

I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...

categorical-data sigmoid multi-label-classification softmax

asked Oct 2 '20 at 16:43

Kacper777

1636 bronze badges

2

votes

1answer

177 views

Which paper introduced the term “softmax”?

Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned ...

neural-networks deep-learning classification history softmax

asked Jul 10 '20 at 0:37

nbro♦

28.1k6 gold badges56 silver badges122 bronze badges

1

vote

0answers

33 views

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient? Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning ...

neural-networks comparison training gradient-descent softmax

asked Jul 9 '20 at 19:37

user3180

5002 silver badges10 bronze badges

0

votes

1answer

245 views

Transformers - is the self attention matrix softmax output (layer 1) symmetric?

Let's assume, that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by it transposed version we receive a matrix of 49 by 49. Which is symmetric. ...

transformer attention softmax

asked Jun 22 '20 at 21:32

thepacker

1012 bronze badges

1

vote

0answers

36 views

What do the authors of this paper mean by the bias term in this picture of a neural network implementation?

I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (...

convolutional-neural-networks papers algorithmic-bias softmax

asked May 9 '20 at 19:49

Mike

1314 bronze badges

4

votes

2answers

578 views

Why does TensorFlow docs discourage using softmax as activation for the last layer?

The beginner colab example for tensorflow states: Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network....

tensorflow objective-functions softmax

asked Apr 13 '20 at 7:38

galah92

1433 bronze badges

Stack Exchange Network

current community

your communities

more stack exchange communities

Questions tagged [softmax]

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

How is it possible that the softmax combined with the MSE in a molecule classification task performs than than the cross-entropy?

Why do we use the softmax instead of no activation function?

Exploration for softmax should be binary or continuous softmax?

XOR Neural Network gets stuck in training

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

Backpropagation implementation not applicable for other cases

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

How am I supposed to code equation 4.57 from the book “Machine Learning: An Algorithmic Perspective”?

Why are there two versions of softmax cross entropy? Which one to use in what situation?

What is the advantage of using cross entropy loss & softmax?

Should I use additional empty category in some categorical problems?

Which paper introduced the term “softmax”?

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Transformers - is the self attention matrix softmax output (layer 1) symmetric?

What do the authors of this paper mean by the bias term in this picture of a neural network implementation?

Why does TensorFlow docs discourage using softmax as activation for the last layer?

Hot Network Questions

Questions tagged [softmax]

Related Tags