# Tag Info

Accepted

### If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

It is not the input to first layer you need to worry about, but the output from the hidden layer to the next layer. No matter how the inputs and weights are arranged, after passing through ReLU in the ...
• 32.1k
Accepted

### Why use ReLU over Leaky ReLU?

Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the dying neurons issue in ReLU: when these are negative, they got zeroed. Regarding the answer of @Regresslt: ...
• 2,898
Accepted

### Activation function intuition question

Activations can help with brightness, but not in the way you described, not by smoothing. Without activation, all your layers collapse to a single linear operation, just because in matrix operations ...
1 vote

### Why does my activation function cause NaNs?

Update: the issue is that your odd_pow() is numerically unstable, and so to fix the exploding gradients, which cause a NaN loss, you simply need to add a small ...
• 2,898
1 vote

### Difference in gradient calculation for the last layer activation in neural networks

In neural networks, The gradient calculation for the last layer differs from the other layers due to the specific combination of the cost function and the activation function used in that layer. for a ...
• 111
1 vote

### How do we determine the slope for leakyrelu activation function?

When this slope in the negative part is learned, the activation function is called Parametric ReLU or PReLU. https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html
1 vote

### Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Like the accepted answer, I'm assuming you are referring to (literature refering to) Leshno et al, 1993. That paper only concerns 1-layer neural networks, and those are simply of the form x\mapsto\...
• 111

Only top scored, non community-wiki answers of a minimum length are eligible