Questions tagged [relu]
For questions concerning neural network rectifiers (ReLU)
25
questions
0
votes
0answers
7 views
How to decide if gradients are vanishing?
I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not. Is there some threshold to decide on vanishing gradient by ...
0
votes
0answers
18 views
Random Initializations with ReLU gives puzzling results
this may sound naive, but I’m getting a really puzzling result. I was experimenting with MNIST on vanilla MLP (784, 256, 128, 10) with ...
0
votes
2answers
27 views
Is it possible to have a negative output using only ReLU activation functions, but not in the final layer?
I know that if you use an ReLU activation function at a node in the neural network, the output of that node will be non-negative. I am wondering if it is possible to have a negative output in the ...
3
votes
1answer
59 views
Can residual neural networks use other activation functions different from ReLU?
In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
2
votes
0answers
69 views
Should batch normalisation be applied before or after ReLU?
I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus.
The crucial thing that I haven't seen mentioned in these discussions is that applying ...
1
vote
1answer
64 views
If features are always positives, why do we use RELU activation functions?
When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU?
As far as I know, features are never negative or converted to negative in any ...
2
votes
1answer
48 views
Does net with ReLU not learn when output < 0?
The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/...
2
votes
1answer
57 views
Neural network doesn't seem to converge with ReLU but it does with Sigmoid?
I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
1
vote
0answers
33 views
Why do DeconvNet use ReLU in the backward pass?
Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
1
vote
0answers
49 views
Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?
Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...
2
votes
2answers
99 views
Is ReLU a non-linear activation function?
According to this blog post
The purpose of an activation function is to add some kind of non-linear property to the function
The sigmoid is typically used as an activation function of a unit of a ...
2
votes
2answers
48 views
How to determine the target value when using ReLU as activation function?
Consider the following simple neural network with only one neuron.
The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$
The weights of the only neuron are $w_1$ and $...
0
votes
1answer
116 views
Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh
I have these training data to separate, the classes are rather randomly scattered:
My first attempt was using tf.nn.relu activation function, but output was stuck ...
4
votes
2answers
212 views
Is PReLU superfluous with respect to ReLU?
Why do people use the $PReLU$ activation?
$PReLU[x] = ReLU[x] + ReLU[p*x]$
with the parameter $p$ typically being a small negative number.
If a fully connected layer is followed by a at least two ...
1
vote
1answer
91 views
Dropout causes too much noise for network to train
I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
2
votes
0answers
70 views
How is the bias caused by a max pooling layer overcome?
I have constructed a CNN that utilizes max-pooling layers. I have found with these layers that, should I remove them, my network performs ideally with every output and gradient at each layer having a ...
3
votes
1answer
91 views
How are exploding numbers in a forward pass of a CNN combated?
Take AlexNet for example:
In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:
Say I have a weight ...
0
votes
2answers
395 views
How should the values of the filters of a CNN change?
I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation.
But one thing I noticed was the "just positive" ...
3
votes
1answer
249 views
How does backpropagation with unbounded activation functions such as ReLU work?
I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation ...
2
votes
1answer
245 views
Is there a ReLU-like activation function that concatenates positive and negative values?
Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.
10
votes
1answer
913 views
What happens when I mix activation functions?
There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions?
I recently found that Google has developed Swish activation function which is (...
11
votes
1answer
10k views
What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?
I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
7
votes
2answers
3k views
Why do we prefer ReLU over linear activation functions?
The ReLU activation function is defined as follows
$$y = \operatorname{max}(0,x)$$
And the linear activation function is defined as follows
$$y = x$$
The ReLU nonlinearity just clips the values ...
10
votes
1answer
1k views
How exactly can ReLUs approximate non-linear and curved functions?
Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
1
vote
2answers
3k views
What is the derivative of the Leaky ReLU activation function?
I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the ...
