# Questions tagged [softmax]

For questions related to the softmax function, which a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. The softmax is often used as the activation function of the output layer of a neural network.

44 questions
Filter by
Sorted by
Tagged with
47 views

### Using conditional probability as an estimate in a loss function

I have a rather large ML framework that takes multiple conditional probability terms that are computed via classifiers/neural networks. This arbitrary loss function is computed via a function: ...
• 101
40 views

### custom neural network implementation is giving 10% accuracy on mnist dataset

I've created a toy neural network in python for learning purposes and decided to test it on mnist dataset, I didn't expect great results but the result that I got - 10% is as good as a guess. For many ...
25 views

### How is the complexity of the chunked attention computation in "Self Attention Does Not Need O(n2) Memory" independent from the query chunks size?

In Self-attention Does Not Need O(n{2}) Memory the authors present a say to have a constant memory complexity attention algorithm that is sequential in nature and also present an implementation that ...
• 575
20 views

### Any LMs that use tanh (generalization) instead of sigmoid within Attention?

Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
• 11
28 views

### One Softmax or two separate logistic regressions for the task of classifying pictures as a/b and c/d

Simply put, the question 11 in chapter 4 of Aurélien Géron's book "Hands-on Machine Learning" asks: Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime. Should you ...
• 23
90 views

### Why does the algorithm in "Self-attention Does Not Need $O(n^{2})$ Memory" require $O(log n)$ memory when $k, v$ pairs are not ordered?

I am reading Self-attention Does Not Need $O(n^{2})$ Memory which proposes an algorithm that requires $O(1)$ memory for one query and $O(log n)$ memory for self-attention, in theory. In practice the ...
• 575
47 views

### Correctly applying softmax in self attention layer

I'm trying to understand how to apply softmax in self attention layer. Let's say we have Query and Key matrix where the last row is for Paddings In this case Z = Q*K_t would be something like this: ...
1 vote
27 views

### Since $f_c$ returns the probability of class label $c$, we require $0 \le f_c \le 1$ for each $c$, and $\sum_{c = 1}^C f_c = 1$. Why avoid this?

Chapter 1.2.1.5 Uncertainty of Probabilistic Machine Learning: An Introduction by Kevin P. Murphy says the following: We can capture our uncertainty using the following conditional probability ...
• 593
1 vote
882 views

### Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?

I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in ...
• 303
25 views

### What are general techniques of structuring an image classification Neural Network for very large numbers of output classes?

I am aware of Neural Networks that have 100K+ classes and I would like to build one myself (yes, I have lots of training data) but I am unsure which technique to use because most of the nets I have ...
1 vote
80 views

### Dealing with noise in models with softmax output

I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). ...
1 vote
60 views

### Number of units in Final softmax layer in VGGNet16

I am trying to implement and train neural network model VGGNet from scratch, on my own data. I am reproducing all the layers of the model. I am having a confusion about the last, fully connected ...
424 views

### How to do backpropagation with argmax?

I am attempting to utilize two networks: a classifier and a linear network. Based on the output class of the first network, my goal is to retrieve the corresponding value from the linear network using ...
1 vote
180 views

### Backpropagation with CrossEntropy and Softmax, HOW?

Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
• 11
264 views

### Should softmax be in the model or in the loss function?

Suppose I have an image segmentation model with an output of [ 128, 128, 2 ], segmenting an input image into 2 parts. Commonly, loss functions have the sigmoid or ...
136 views

### Why didn't my convolutional image classifier network learn anything?

So I am trying to make a CNN image classifier that has two classes, good and bad. The aim is to look at photoshoot pictures that can be found on fashion sites and find the "best one". I ...
• 101
490 views

### Why are policy gradient methods more effective in high-dimensional action spaces?

David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ...
• 143
4k views

### Are softmax outputs of classifiers true probabilities?

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have ...
• 962
1 vote
259 views

### Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity. However, here's where my understanding starts to ...
• 153
1 vote
191 views

### Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
275 views

### Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
• 33
425 views

### Are there any scale invariant activation functions that outputs probability distribution?

Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are ...
• 3,930
231 views

### Are any non-injective activation functions used?

All activation functions I know of are injective, which I think makes sense. But are there cases where non-injective activations can be useful?
• 133
1 vote
108 views

### Use soft-max post-training for a ReLU trained network?

For a project, I've trained multiple networks for multiclass classification all ending with a ReLU activation at the output. Now the output logits are not probabilities. Is it valid to get the ...
1 vote
2k views

### Is it normal that the values of the LogSoftmax function are very large negative numbers? [closed]

I have trained a classification network with PyTorch lightning where my training step looks like below: ...
• 125
397 views

I encountered the term multinoulli distribution in the following sentence from Chapter 4: Numerical Computation of the deep learning book. The softmax function is often used to predict the ...
• 3,930
186 views

### Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
3k views

### Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
128 views

### Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
571 views

### Why does my neural network to solve the XOR problem always output 0.5?

I'm trying to create a neural network to simulate an XOR gate. Here's my dataset: ...
1 vote
4k views

### Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
• 161
1 vote
242 views

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
• 111
1 vote
669 views

### How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...
379 views

### Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
1 vote
Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained \delta_o(\kappa) = (y_\kappa -...