Questions tagged [softmax]

For questions related to the softmax function, which a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. The softmax is often used as the activation function of the output layer of a neural network.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

What are general techniques of structuring an image classification Neural Network for very large numbers of output classes?

I am aware of Neural Networks that have 100K+ classes and I would like to build one myself (yes, I have lots of training data) but I am unsure which technique to use because most of the nets I have ...
AnalogDigital's user avatar
1 vote
1 answer
62 views

Dealing with noise in models with softmax output

I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). ...
Sterling Duchess's user avatar
1 vote
1 answer
35 views

Number of units in Final softmax layer in VGGNet16

I am trying to implement and train neural network model VGGNet from scratch, on my own data. I am reproducing all the layers of the model. I am having a confusion about the last, fully connected ...
Dawood Ahmad's user avatar
0 votes
1 answer
203 views

How to do backpropagation with argmax?

I am attempting to utilize two networks: a classifier and a linear network. Based on the output class of the first network, my goal is to retrieve the corresponding value from the linear network using ...
Subrat Prasad's user avatar
1 vote
2 answers
103 views

Backpropagation with CrossEntropy and Softmax, HOW?

Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
qazaq's user avatar
  • 11
0 votes
0 answers
84 views

Computing confidence score for transformer output

I'm sampling multiple generations from a transformer model and I would like to have a confidence score for each generation. The Hugging Face library's generate() ...
Daniel Darabos's user avatar
0 votes
1 answer
114 views

Should softmax be in the model or in the loss function?

Suppose I have an image segmentation model with an output of [ 128, 128, 2 ], segmenting an input image into 2 parts. Commonly, loss functions have the sigmoid or ...
starbeamrainbowlabs's user avatar
0 votes
0 answers
28 views

Exponentiation immediately after log softmax

In an AlphaZero repo's implementation of Othello game, I see: At one line: F.log_softmax here Next line of the control flow: ...
Intrastellar Explorer's user avatar
0 votes
1 answer
73 views

Why didn't my convolutional image classifier network learn anything?

So I am trying to make a CNN image classifier that has two classes, good and bad. The aim is to look at photoshoot pictures that can be found on fashion sites and find the "best one". I ...
isa türk's user avatar
  • 101
4 votes
1 answer
380 views

Why are policy gradient methods more effective in high-dimensional action spaces?

David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ...
Saucy Goat's user avatar
17 votes
2 answers
3k views

Are softmax outputs of classifiers true probabilities?

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have ...
Snehal Patel's user avatar
1 vote
1 answer
169 views

Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity. However, here's where my understanding starts to ...
Foobar's user avatar
  • 151
1 vote
1 answer
181 views

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
user605734 MBS's user avatar
0 votes
1 answer
252 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
Aquila's user avatar
  • 33
0 votes
1 answer
349 views

Are there any scale invariant activation functions that outputs probability distribution?

Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are ...
hanugm's user avatar
  • 3,612
-1 votes
1 answer
181 views

Are any non-injective activation functions used?

All activation functions I know of are injective, which I think makes sense. But are there cases where non-injective activations can be useful?
Moritz Groß's user avatar
1 vote
0 answers
102 views

Use soft-max post-training for a ReLU trained network?

For a project, I've trained multiple networks for multiclass classification all ending with a ReLU activation at the output. Now the output logits are not probabilities. Is it valid to get the ...
user452306's user avatar
1 vote
1 answer
1k views

Is it normal that the values of the LogSoftmax function are very large negative numbers? [closed]

I have trained a classification network with PyTorch lightning where my training step looks like below: ...
pd109's user avatar
  • 125
0 votes
2 answers
360 views

Where can I read about the multinoulli distribution?

I encountered the term multinoulli distribution in the following sentence from Chapter 4: Numerical Computation of the deep learning book. The softmax function is often used to predict the ...
hanugm's user avatar
  • 3,612
0 votes
1 answer
162 views

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
Liuuuuk's user avatar
2 votes
1 answer
2k views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
dato nefaridze's user avatar
-1 votes
1 answer
124 views

Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
fardis nadimi's user avatar
0 votes
1 answer
426 views

Why does my neural network to solve the XOR problem always output 0.5?

I'm trying to create a neural network to simulate an XOR gate. Here's my dataset: ...
user45708's user avatar
1 vote
1 answer
3k views

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
user9317212's user avatar
1 vote
0 answers
181 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
Kagaratsch's user avatar
1 vote
1 answer
607 views

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...
Ilknur Mustafa's user avatar
0 votes
1 answer
335 views

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
Ilknur Mustafa's user avatar
1 vote
0 answers
63 views

How am I supposed to code equation 4.57 from the book "Machine Learning: An Algorithmic Perspective"?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained $$\delta_o(\kappa) = (y_\kappa -...
NewToCoding's user avatar
1 vote
1 answer
286 views

Why are there two versions of softmax cross entropy? Which one to use in what situation?

I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one? For example in this Quora answer, there are 2 answers: $L(\mathbf{w})=\frac{1}{N} \sum_{n=1}^...
Herbert's user avatar
  • 113
5 votes
2 answers
1k views

What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
Ben's user avatar
  • 425
1 vote
2 answers
77 views

Should I use additional empty category in some categorical problems?

I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
user avatar
5 votes
1 answer
2k views

Which paper introduced the term "softmax"?

Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned ...
nbro's user avatar
  • 39.6k
1 vote
0 answers
50 views

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient? Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning ...
user3180's user avatar
  • 598
1 vote
1 answer
1k views

Is the self-attention matrix softmax output (layer 1) symmetric?

Let's assume that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by its transposed version, we receive a matrix of 49 by 49, which is symmetric. ...
thepacker's user avatar
  • 131
2 votes
2 answers
92 views

What do the authors of this paper mean by the bias term in this picture of a neural network implementation?

I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (...
Mike's user avatar
  • 141
6 votes
2 answers
1k views

Why does TensorFlow docs discourage using softmax as activation for the last layer?

The beginner colab example for tensorflow states: Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network....
galah92's user avatar
  • 163
1 vote
1 answer
399 views

Is this neural network with a softmax in the output layer suitable for multi-label classification?

I have data with about 100 numerical features and a multi-labelling that encodes ownership of a certain product (i.e. my labels are of the form $[x_i, i=1, \dots, n]$, where $n$ is the number of ...
Joseph Doob's user avatar