Questions tagged [relu]
For questions concerning neural network rectifiers (ReLU)
11
questions
0
votes
1answer
32 views
Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh
I have these training data to separate, the classes are rather randomly scattered:
My first attempt was using tf.nn.relu activation function, but output was stuck ...
3
votes
2answers
58 views
Is PReLU superfluous with respect to ReLU?
Why do people use the $PReLU$ activation?
$PReLU[x] = ReLU[x] + ReLU[p*x]$
with the parameter $p$ typically being a small negative number.
If a fully connected layer is followed by a at least two ...
1
vote
1answer
59 views
Dropout causes too much noise for network to train
I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
1
vote
0answers
21 views
How is the bias caused by a max pooling layer overcome?
I have constructed a CNN that utilises max pooling layers. I have found with these layers that, should I remove them, my network performs ideally with every output and gradient at each layer having a ...
3
votes
1answer
48 views
How are exploding numbers in a forward pass of a CNN combated?
Take AlexNet for example:
In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:
Say I have a weight ...
0
votes
2answers
73 views
How should the values of the filters of a CNN change?
I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation.
But one thing I noticed was the "just positive" ...
0
votes
0answers
31 views
Has anyone succeeded to (intentionally) overfit the neural network with MNIST?
I am currently studying myself with a subject "representational(expressive power) of neural network" and trying to intentionally fully overfit the neural network which means that at least the model ...
1
vote
1answer
91 views
How does backpropagation with unbounded activation functions such as ReLU work?
I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation ...
2
votes
2answers
94 views
Why do we throw out negative ReLU value?
when using Rectified Linear Unit after convolution layers we have to have twice as much filters to be able to detect features (eg both left and right edge detector). Why do we just throw out negative ...
9
votes
1answer
171 views
What happens when I mix activation functions?
There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions?
I recently found that Google has developed Swish activation function which is (...
9
votes
1answer
4k views
What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?
I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
