# Questions tagged [relu]

For questions about the rectified linear unit (ReLU) or rectifiers, which is a widely used activation function in neural networks.

36 questions
Filter by
Sorted by
Tagged with
29 views

### What are Linear and Non-Linear Features of an image in the context of Convolutional Neural Network?

What features of image are linear or non-linear, any example ?
60 views

### Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
60 views

### Why the partial derivative is $0$ when $F_{ij}^l < 0$?. Math behind style transfer

I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like - For context, here is the paragraph - ...
20 views

### ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
21 views

### Can Neural Networks using ReLU activation work without using the bias term in their neurons?

I created a super simple NN of 1 input, 2 hidden layers of 2 neurons each and 1 output neuron as shown below. All activations are ReLUs and neurons doesn't use the bias term. What I found is that the ...
42 views

### How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. ...
194 views

### Why is tf.abs non-differentiable in Tensorflow?

I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is ...
46 views

### What are the benefits of using ELU over other activation functions in CNNs?

I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function. What are the benefits of this activation function over others, such as ...
39 views

### How is the performance of a model affected by adding a ReLU to fully connected layers?

How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?
73 views

### How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
404 views

### Is it possible to have a negative output using only ReLU activation functions, but not in the final layer?

I know that if you use an ReLU activation function at a node in the neural network, the output of that node will be non-negative. I am wondering if it is possible to have a negative output in the ...
112 views

### Can residual neural networks use other activation functions different from ReLU?

In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
149 views

### Should batch normalisation be applied before or after ReLU?

I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus. The crucial thing that I haven't seen mentioned in these discussions is that applying ...
72 views

### If features are always positives, why do we use RELU activation functions?

When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU? As far as I know, features are never negative or converted to negative in any ...
54 views

937 views

### In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
343 views

### Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...
458 views

### Is PReLU superfluous with respect to ReLU?

Why do people use the $PReLU$ activation? $PReLU[x] = ReLU[x] + ReLU[p*x]$ with the parameter $p$ typically being a small negative number. If a fully connected layer is followed by a at least two ...
151 views

### Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
127 views

### How is the bias caused by a max pooling layer overcome?

I have constructed a CNN that utilizes max-pooling layers. I have found with these layers that, should I remove them, my network performs ideally with every output and gradient at each layer having a ...
164 views

### How are exploding numbers in a forward pass of a CNN combated?

Take AlexNet for example: In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example: Say I have a weight ...
685 views

### How should the values of the filters of a CNN change?

I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation. But one thing I noticed was the "just positive" ...
367 views

### How does backpropagation with unbounded activation functions such as ReLU work?

I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation ...
347 views

### Is there a ReLU-like activation function that concatenates positive and negative values?

Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.
2k views

### What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
16k views

### What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
5k views

### Why do we prefer ReLU over linear activation functions?

The ReLU activation function is defined as follows $$y = \operatorname{max}(0,x)$$ And the linear activation function is defined as follows $$y = x$$ The ReLU nonlinearity just clips the values ...
2k views

### How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
4k views

### What is the derivative of the Leaky ReLU activation function?

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the ...