13
votes
Accepted
Are softmax outputs of classifiers true probabilities?
The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, ...
- 278
11
votes
Are softmax outputs of classifiers true probabilities?
Excellent question.
The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct.
What usually ...
- 1,122
5
votes
Accepted
Can LSTM model use ReLU or LeakyReLU as the activation funtion?
Yes, you can use ReLU or LeakyReLU in an LSTM model.
There aren't hard rules for choosing activation functions. Run your model with each activation function and pick the best performing one.
See the ...
- 1,728
5
votes
Can LSTM model use ReLU or LeakyReLU as the activation funtion?
Yes an LSTM can use any of these.
There are no hard rules of which to use. That is why they all exist.
Some rules of thumb are:
Relu is the cheapest computationally. Almost always worth trying first.
...
- 2,165
4
votes
Why do activation functions in neural networks have to be non-polynomial to approximate any function?
Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs ...
- 681
4
votes
Accepted
Why aren't artificial derivatives used more often to solve the vanishing gradient problem?
This idea seems pretty convincing
Indeed, you don't have to use the exact gradient of the activation function during the backward step.
The gradient of the activation function is a multiplicative ...
3
votes
Accepted
Why cannot linear activation functions be used to approximate any function?
However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is ...
- 1,452
3
votes
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
I think a better way to understand LSTMs is by their purpose, instead of gradients and distributions.
If you analyze the interactions of each gate with the cell state, you'll realize that LSTMs ...
- 31
2
votes
Accepted
Why use ReLU over Leaky ReLU?
Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the dying neurons issue in ReLU: when these are negative, they got zeroed.
Regarding the answer of @Regresslt:
...
- 1,253
1
vote
Why use ReLU over Leaky ReLU?
Leaky ReLU is indeed an improvement over the standard ReLU activation function, but comes with some of the following limitations:
It may suffer from the "dying ReLU" problem, where a large ...
- 11
1
vote
Accepted
How do we determine the slope for leakyrelu activation function?
There is definitely no 'mathematically optimal' LeakyReLU value, as it is dependent on the data, the architecture etc.
In addition, to the best of my knowledge, there is no 'best practice' when it ...
- 1,574
1
vote
Accepted
Single label classification into hierarchical categories using a neural network
In my opinion, the problem you pose is best described as an ordinal classification problem, rather than a hierarchical classification problem. There are a number of approaches (besides ordinal loss ...
- 882
1
vote
Why is the derivative of activation function all positive?
By definition from wikipedia:
Let $\varphi$ be a nonconstant, bounded, and continuous function.
This is also being proven for ReLU and other functions that violets some of those properties... ...
1
vote
Why and when do we use ReLU over tanh activation function?
For a discussion about the advantages of ReLU, see the original paper by Glorot (2011) "Deep sparse rectifier neural networks".
"Efficient Backprop" is a 1998 paper. At the time ...
- 205
1
vote
Accepted
Where does the "rectified" in ReLU come from?
I think it is by analogy with an electrical rectifier. A rectifier allows current to flow in one direction but blocks current in the other direction. Or if you prefer it allows voltage in one polarity ...
- 842
1
vote
Accepted
Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?
I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different:
The first image shows a linear classifier assigning scores for ...
- 151
1
vote
Is there any way to train a regression model with negative values that is more stable?
A couple things you could try:
You could try normalizing your target variable, so that it's number of standard deviations from the mean, or mapped to [-1,1].
If you are using drop-out during training. ...
- 111
1
vote
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
The purpose of the tanh and sigmoid functions in an LSTM (Long Short-Term Memory) network is to control the flow of information through the cell state, which is the "memory" of the network.
...
- 111
Only top scored, non community-wiki answers of a minimum length are eligible