5 votes
Accepted

If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

It is not the input to first layer you need to worry about, but the output from the hidden layer to the next layer. No matter how the inputs and weights are arranged, after passing through ReLU in the ...
Neil Slater's user avatar
  • 32.1k
4 votes
Accepted

Why use ReLU over Leaky ReLU?

Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the dying neurons issue in ReLU: when these are negative, they got zeroed. Regarding the answer of @Regresslt: ...
Luca Anzalone's user avatar
3 votes
Accepted

Activation function intuition question

Activations can help with brightness, but not in the way you described, not by smoothing. Without activation, all your layers collapse to a single linear operation, just because in matrix operations ...
Kirill Fedyanin's user avatar
1 vote

Why does my activation function cause NaNs?

Update: the issue is that your odd_pow() is numerically unstable, and so to fix the exploding gradients, which cause a NaN loss, you simply need to add a small ...
Luca Anzalone's user avatar
1 vote

Difference in gradient calculation for the last layer activation in neural networks

In neural networks, The gradient calculation for the last layer differs from the other layers due to the specific combination of the cost function and the activation function used in that layer. for a ...
Keval's user avatar
  • 111
1 vote

How do we determine the slope for leakyrelu activation function?

When this slope in the negative part is learned, the activation function is called Parametric ReLU or PReLU. https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html
Jaume Oliver Lafont's user avatar
1 vote

Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Like the accepted answer, I'm assuming you are referring to (literature refering to) Leshno et al, 1993. That paper only concerns 1-layer neural networks, and those are simply of the form $$x\mapsto\...
Teun's user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible