Questions tagged [relu]
For questions about the rectified linear unit (ReLU) or rectifiers, which is a widely used activation function in neural networks.
49
questions
2
votes
0
answers
31
views
What's the result of multiple neurons using ReLU activation function?
This question comes from a doubt that I recently had on an amazing book called "Neural Networks from Scratch With Python by Harrison Kinsley & Daniel Kukieła"
Let's suppose that I have ...
- 21
0
votes
0
answers
11
views
How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?
I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, . . . , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
1
vote
2
answers
851
views
Why does Batch-Normalization before ReLU works?
I have read a little about Batch-Normalization and I understood that there isn't any better option on where you place Batch-Normalization (it all depends on the case).
However, I don't understand the ...
- 11
2
votes
1
answer
146
views
How does gradient descent work with relu if weights are negative?
How does gradient descent work with relu, imagine the weights are quite negative and so our "prediction" is 0, then not much is learned. Is there a risk that training gets stuck when weights ...
- 158
1
vote
0
answers
52
views
Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization?
From Deep Learning (Courville, Goodfellow, Bengio), a ReLU activation often "dies" because
One drawback to rectified linear units is that they cannot learn via gradient based methods on ...
- 11
1
vote
0
answers
134
views
Neural network and logical gates
I have a network witch consist of two fully connected layers (without bias) and a ReLu function in between. The network input is two binary numbers, and the output should be the a logical gate result:
...
- 131
2
votes
1
answer
613
views
Why cannot linear activation functions be used to approximate any function?
In neural networks we use nonlinear activation functions such as sigmoid, ReLU, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear ...
- 35
2
votes
1
answer
179
views
Why and when do we use ReLU over tanh activation function?
I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how ...
0
votes
1
answer
69
views
Where does the "rectified" in ReLU come from?
ReLU stands for Rectified Linear Unit. Linear Unit, I understand, since the function is piecewise linear. But what does rectified mean?
I looked up the definition and it said:
denoting an electric ...
- 123
1
vote
1
answer
249
views
Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?
The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
- 608
0
votes
1
answer
72
views
What is meant by non-linearity in Convolutional Neural Networks? And why do we focus on removing it entirely? [closed]
I'm aware of the working of ReLU that it's turns every negative value to zero and doesn't effect any positive value, but what confuses me is that: what is actually meant by Non-linearity in feature ...
- 13
-1
votes
1
answer
181
views
Are any non-injective activation functions used?
All activation functions I know of are injective, which I think makes sense.
But are there cases where non-injective activations can be useful?
- 133
1
vote
2
answers
2k
views
Effects of ReLU Activation on Convexity of Loss Functions
I have heard the following argument being made regarding Neural Networks:
A Neural Network is a composition of several Activation Functions
Sigmoid Activation Functions are Non-Convex Functions
The ...
- 269
1
vote
0
answers
123
views
What are Linear and Non-Linear Features of an image in the context of Convolutional Neural Network?
What features of image are linear or non-linear, any example ?
- 11
4
votes
1
answer
1k
views
Why should one ever use ReLU instead of PReLU?
To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
- 41
0
votes
1
answer
110
views
Why the partial derivative is $0$ when $F_{ij}^l < 0$?. Math behind style transfer
I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like -
For context, here is the paragraph -
...
- 101
1
vote
0
answers
57
views
ReLU function converging to local optimum in one case and diverging in the other one
I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function.
To check my implementation I ...
- 107
2
votes
0
answers
93
views
Can Neural Networks using ReLU activation work without using the bias term in their neurons?
I created a super simple NN of 1 input, 2 hidden layers of 2 neurons each and 1 output neuron as shown below.
All activations are ReLUs and neurons doesn't use the bias term. What I found is that the ...
- 31
0
votes
1
answer
62
views
How to explain that a same DNN model have radically different behaviours with each new initialization and training?
I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. ...
- 573
5
votes
2
answers
1k
views
Why is tf.abs non-differentiable in Tensorflow?
I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is ...
- 53
1
vote
0
answers
112
views
What are the benefits of using ELU over other activation functions in CNNs?
I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function.
What are the benefits of this activation function over others, such as ...
- 63
0
votes
1
answer
750
views
How is the performance of a model affected by adding a ReLU to fully connected layers?
How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?
2
votes
0
answers
317
views
How to decide if gradients are vanishing?
I am trying to debug a convolutional neural network. I am seeing gradients close to zero.
How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
- 121
0
votes
2
answers
2k
views
Is it possible to have a negative output using only ReLU activation functions, but not in the final layer?
I know that if you use an ReLU activation function at a node in the neural network, the output of that node will be non-negative. I am wondering if it is possible to have a negative output in the ...
- 321
3
votes
1
answer
235
views
Can residual neural networks use other activation functions different from ReLU?
In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
4
votes
0
answers
2k
views
Should batch normalisation be applied before or after ReLU?
I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus.
The crucial thing that I haven't seen mentioned in these discussions is that applying ...
- 171
1
vote
1
answer
115
views
If features are always positives, why do we use RELU activation functions?
When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU?
As far as I know, features are never negative or converted to negative in any ...
- 129
2
votes
1
answer
112
views
Does net with ReLU not learn when output < 0?
The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/...
- 639
4
votes
1
answer
757
views
Neural network doesn't seem to converge with ReLU but it does with Sigmoid?
I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
- 151
1
vote
0
answers
55
views
Why do DeconvNet use ReLU in the backward pass?
Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
- 161
1
vote
1
answer
283
views
Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?
Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...
- 116
2
votes
2
answers
520
views
Is ReLU a non-linear activation function?
According to this blog post
The purpose of an activation function is to add some kind of non-linear property to the function
The sigmoid is typically used as an activation function of a unit of a ...
- 151
2
votes
2
answers
504
views
How to determine the target value when using ReLU as activation function?
Consider the following simple neural network with only one neuron.
The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$
The weights of the only neuron are $w_1$ and $...
5
votes
2
answers
2k
views
In deep learning, is it possible to use discontinuous activation functions?
In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
(My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
0
votes
1
answer
903
views
Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh
I have these training data to separate, the classes are rather randomly scattered:
My first attempt was using tf.nn.relu activation function, but output was stuck ...
- 1,283
4
votes
2
answers
892
views
Is PReLU superfluous with respect to ReLU?
Why do people use the $PReLU$ activation?
$PReLU[x] = ReLU[x] + ReLU[p*x]$
with the parameter $p$ typically being a small negative number.
If a fully connected layer is followed by a at least two ...
- 143
1
vote
1
answer
262
views
Dropout causes too much noise for network to train
I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
- 1,346
2
votes
0
answers
218
views
How is the bias caused by a max pooling layer overcome?
I have constructed a CNN that utilizes max-pooling layers. I have found with these layers that, should I remove them, my network performs ideally with every output and gradient at each layer having a ...
- 1,346
3
votes
1
answer
419
views
How are exploding numbers in a forward pass of a CNN combated?
Take AlexNet for example:
In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:
Say I have a weight ...
- 1,346
0
votes
2
answers
853
views
How should the values of the filters of a CNN change?
I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation.
But one thing I noticed was the "just positive" ...
3
votes
1
answer
576
views
How does backpropagation with unbounded activation functions such as ReLU work?
I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation ...
1
vote
1
answer
503
views
Is there a ReLU-like activation function that concatenates positive and negative values?
Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.
- 127
9
votes
1
answer
4k
views
What happens when I mix activation functions?
There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions?
I recently found that Google has developed Swish activation function which is (...
- 93
22
votes
1
answer
28k
views
What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?
I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
- 696
11
votes
2
answers
8k
views
Why do we prefer ReLU over linear activation functions?
The ReLU activation function is defined as follows
$$y = \operatorname{max}(0,x)$$
And the linear activation function is defined as follows
$$y = x$$
The ReLU nonlinearity just clips the values ...
- 499
13
votes
1
answer
3k
views
How exactly can ReLUs approximate non-linear and curved functions?
Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
user9947
1
vote
3
answers
5k
views
What is the derivative of the Leaky ReLU activation function?
I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the ...
- 145
2
votes
2
answers
472
views
Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?
Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?
My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...
- 59
10
votes
3
answers
2k
views
Are ReLUs incapable of solving certain problems?
Background
I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently.
Both for fun and to increase my understanding, I ...
