Skip to main content

Questions tagged [softmax]

For questions related to the softmax function, which a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. The softmax is often used as the activation function of the output layer of a neural network.

13 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
3 answers

What do the authors of this paper mean by the bias term in this picture of a neural network implementation?

I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (...
Mike's user avatar
  • 141
1 vote
1 answer

Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?

I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in ...
Kasia's user avatar
  • 303
1 vote
0 answers

Use soft-max post-training for a ReLU trained network?

For a project, I've trained multiple networks for multiclass classification all ending with a ReLU activation at the output. Now the output logits are not probabilities. Is it valid to get the ...
user452306's user avatar
1 vote
0 answers

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
Kagaratsch's user avatar
1 vote
0 answers

How am I supposed to code equation 4.57 from the book "Machine Learning: An Algorithmic Perspective"?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained $$\delta_o(\kappa) = (y_\kappa -...
NewToCoding's user avatar
1 vote
0 answers

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient? Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning ...
user3180's user avatar
  • 628
0 votes
0 answers

How is the complexity of the chunked attention computation in "Self Attention Does Not Need O(n2) Memory" independent from the query chunks size?

In Self-attention Does Not Need O(n{2}) Memory the authors present a say to have a constant memory complexity attention algorithm that is sequential in nature and also present an implementation that ...
Daviiid's user avatar
  • 575
0 votes
0 answers

Any LMs that use tanh (generalization) instead of sigmoid within Attention?

Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
naston's user avatar
  • 11
0 votes
0 answers

Why does the algorithm in "Self-attention Does Not Need $O(n^{2})$ Memory" require $O(log n)$ memory when $k, v$ pairs are not ordered?

I am reading Self-attention Does Not Need $O(n^{2})$ Memory which proposes an algorithm that requires $O(1)$ memory for one query and $O(log n)$ memory for self-attention, in theory. In practice the ...
Daviiid's user avatar
  • 575
0 votes
0 answers

Correctly applying softmax in self attention layer

I'm trying to understand how to apply softmax in self attention layer. Let's say we have Query and Key matrix where the last row is for Paddings In this case Z = Q*K_t would be something like this: ...
Davk9_4's user avatar
0 votes
0 answers

What are general techniques of structuring an image classification Neural Network for very large numbers of output classes?

I am aware of Neural Networks that have 100K+ classes and I would like to build one myself (yes, I have lots of training data) but I am unsure which technique to use because most of the nets I have ...
AnalogDigital's user avatar
0 votes
0 answers

Computing confidence score for transformer output

I'm sampling multiple generations from a transformer model and I would like to have a confidence score for each generation. The Hugging Face library's generate() ...
Daniel Darabos's user avatar
-1 votes
1 answer

Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
fardis nadimi's user avatar