Questions tagged [relu]

For questions concerning neural network rectifiers (ReLU)

Filter by
Sorted by
Tagged with
10
votes
1answer
1k views

How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
3
votes
1answer
91 views

How are exploding numbers in a forward pass of a CNN combated?

Take AlexNet for example: In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example: Say I have a weight ...