Questions tagged [softmax]

For questions related to the softmax function, which a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. The softmax is often used as the activation function of the output layer of a neural network.

28 questions
Filter by
Sorted by
Tagged with
1 vote
22 views

Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity. However, here's where my understanding starts to ...
• 121
1 vote
34 views

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
54 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
• 33
85 views

Are there any scale invariant activation functions that outputs probability distribution?

Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are ...
• 3,099
63 views

Are any non-injective activation functions used?

All activation functions I know of are injective, which I think makes sense. But are there cases where non-injective activations can be useful?
• 133
133 views

How to predict using softmax having separate inputs and outputs?

I am new to Deep Learning. Having completed the coursera courses and read something from Deep Learning with Python, I am trying to implement one idea using DL. There is a number of user equipment (UE) ...
1 vote
43 views

Use soft-max post-training for a ReLU trained network?

For a project, I've trained multiple networks for multiclass classification all ending with a ReLU activation at the output. Now the output logits are not probabilities. Is it valid to get the ...
1 vote
177 views

Is it normal that the values of the LogSoftmax function are very large negative numbers? [closed]

I have trained a classification network with PyTorch lightning where my training step looks like below: ...
• 25
138 views

I encountered the term multinoulli distribution in the following sentence from Chapter 4: Numerical Computation of the deep learning book. The softmax function is often used to predict the ...
• 3,099
22 views

Why use two different embeddings for actions in this paper?

I was reading this paper Top-𝐾 Off-Policy Correction for a REINFORCE Recommender System and I'm wondering is there a particular advantage to use different embeddings for actions, one embedding is ...
• 509
68 views

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
396 views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
72 views

Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
156 views

Why does my neural network to solve the XOR problem always output 0.5?

I'm trying to create a neural network to simulate an XOR gate. Here's my dataset: ...
1 vote
959 views

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
• 131
1 vote
55 views

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
• 111
1 vote
270 views

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...
100 views

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
1 vote
53 views

How am I supposed to code equation 4.57 from the book "Machine Learning: An Algorithmic Perspective"?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained \delta_o(\kappa) = (y_\kappa -...
1 vote
203 views

• 113