Newest 'relu' Questions

1

vote

0answers

29 views

What are Linear and Non-Linear Features of an image in the context of Convolutional Neural Network?

What features of image are linear or non-linear, any example ?

asked Sep 19 at 8:52

Shuai Xu

112 bronze badges

3

votes

1answer

60 views

Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...

deep-learning comparison activation-function relu prelu

asked Sep 10 at 11:42

Roman27

311 bronze badge

0

votes

1answer

60 views

Why the partial derivative is $0$ when $F_{ij}^l < 0$?. Math behind style transfer

I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like - For context, here is the paragraph - ...

convolutional-neural-networks relu calculus vgg style-transfer

asked Aug 13 at 6:13

HarshDarji

1013 bronze badges

1

vote

0answers

20 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...

neural-networks machine-learning backpropagation feedforward-neural-network relu

asked Jul 27 at 20:19

SAGALPREET SINGH

1375 bronze badges

2

votes

0answers

21 views

Can Neural Networks using ReLU activation work without using the bias term in their neurons?

I created a super simple NN of 1 input, 2 hidden layers of 2 neurons each and 1 output neuron as shown below. All activations are ReLUs and neurons doesn't use the bias term. What I found is that the ...

neural-networks relu

asked Jun 15 at 13:05

Manu Soman

312 bronze badges

0

votes

1answer

42 views

How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. ...

regression relu weights-initialization fully-connected-layer learning-curve

asked Apr 14 at 2:20

Daviiid

4359 bronze badges

5

votes

2answers

194 views

Why is tf.abs non-differentiable in Tensorflow?

I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is ...

tensorflow backpropagation relu gradient

asked Feb 17 at 17:29

zedsdead

533 bronze badges

1

vote

0answers

46 views

What are the benefits of using ELU over other activation functions in CNNs?

I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function. What are the benefits of this activation function over others, such as ...

convolutional-neural-networks activation-function relu leaky-relu elu

asked Nov 16 '20 at 11:24

IgnacioGaBo

314 bronze badges

0

votes

1answer

39 views

How is the performance of a model affected by adding a ReLU to fully connected layers?

How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?

neural-networks deep-neural-networks activation-function performance relu

asked Nov 7 '20 at 5:48

sai_varshittha

515 bronze badges

2

votes

0answers

73 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...

convolutional-neural-networks activation-function relu vanishing-gradient-problem adam

asked Oct 20 '20 at 5:22

pramesh

1213 bronze badges

0

votes

2answers

404 views

Is it possible to have a negative output using only ReLU activation functions, but not in the final layer?

I know that if you use an ReLU activation function at a node in the neural network, the output of that node will be non-negative. I am wondering if it is possible to have a negative output in the ...

neural-networks relu

asked Sep 20 '20 at 14:18

anonuser01

2912 silver badges9 bronze badges

3

votes

1answer

112 views

Can residual neural networks use other activation functions different from ReLU?

In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...

neural-networks convolutional-neural-networks activation-function relu residual-networks

asked Jul 23 '20 at 6:14

jr123456jr987654321

1137 bronze badges

2

votes

0answers

149 views

Should batch normalisation be applied before or after ReLU?

I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus. The crucial thing that I haven't seen mentioned in these discussions is that applying ...

neural-networks deep-learning relu batch-normalization

asked Jul 1 '20 at 7:24

Kris

1514 bronze badges

1

vote

1answer

72 views

If features are always positives, why do we use RELU activation functions?

When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU? As far as I know, features are never negative or converted to negative in any ...

convolutional-neural-networks linear-regression relu non-linear-regression

asked May 16 '20 at 2:55

sujeto1

1292 bronze badges

2

votes

1answer

54 views

Does net with ReLU not learn when output < 0?

The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/...

deep-learning backpropagation relu

asked Apr 27 '20 at 15:17

Gilad Deutsch

5692 silver badges11 bronze badges

3

votes

1answer

210 views

Neural network doesn't seem to converge with ReLU but it does with Sigmoid?

I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....

convergence relu c++ sigmoid

asked Apr 15 '20 at 18:37

finlay morrison

412 bronze badges

1

vote

0answers

38 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...

neural-networks deep-learning convolutional-neural-networks backpropagation relu

asked Mar 30 '20 at 11:27

volperossa

1616 bronze badges

1

vote

0answers

66 views

Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?

Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...

machine-learning activation-function relu

asked Mar 29 '20 at 8:28

Paul Higazi

965 bronze badges

2

votes

2answers

190 views

Is ReLU a non-linear activation function?

According to this blog post The purpose of an activation function is to add some kind of non-linear property to the function The sigmoid is typically used as an activation function of a unit of a ...

neural-networks deep-learning math activation-function relu

asked Mar 29 '20 at 2:12

slowmonk

1512 bronze badges

2

votes

2answers

158 views

How to determine the target value when using ReLU as activation function?

Consider the following simple neural network with only one neuron. The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$ The weights of the only neuron are $w_1$ and $...

neural-networks backpropagation relu mean-squared-error

asked Mar 10 '20 at 17:16

Alan Vinícius

212 bronze badges

5

votes

2answers

937 views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...

deep-learning backpropagation optimization activation-function relu

asked Jan 22 '20 at 4:40

Gyeonghoon Ko

511 bronze badge

0

votes

1answer

343 views

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...

tensorflow gradient-descent activation-function relu sigmoid

asked Sep 12 '19 at 10:26

datdinhquoc

1,0827 silver badges24 bronze badges

4

votes

2answers

458 views

Is PReLU superfluous with respect to ReLU?

Why do people use the $PReLU$ activation? $PReLU[x] = ReLU[x] + ReLU[p*x]$ with the parameter $p$ typically being a small negative number. If a fully connected layer is followed by a at least two ...

neural-networks machine-learning activation-function relu

asked Jul 23 '19 at 13:54

Robert Nowak

1435 bronze badges

1

vote

1answer

151 views

Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...

neural-networks convolutional-neural-networks dropout regularization relu

asked Jul 16 '19 at 2:58

Recessive

1,2612 silver badges15 bronze badges

2

votes

0answers

127 views

How is the bias caused by a max pooling layer overcome?

I have constructed a CNN that utilizes max-pooling layers. I have found with these layers that, should I remove them, my network performs ideally with every output and gradient at each layer having a ...

machine-learning convolutional-neural-networks relu bias-variance-tradeoff

asked Jul 1 '19 at 4:13

Recessive

1,2612 silver badges15 bronze badges

3

votes

1answer

164 views

How are exploding numbers in a forward pass of a CNN combated?

Take AlexNet for example: In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example: Say I have a weight ...

machine-learning convolutional-neural-networks relu

asked Jun 28 '19 at 2:59

Recessive

1,2612 silver badges15 bronze badges

0

votes

2answers

685 views

How should the values of the filters of a CNN change?

I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation. But one thing I noticed was the "just positive" ...

convolutional-neural-networks backpropagation relu convolution

asked Apr 22 '19 at 9:09

jutu OOtv

32 bronze badges

3

votes

1answer

367 views

How does backpropagation with unbounded activation functions such as ReLU work?

I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation ...

neural-networks backpropagation relu

asked Feb 16 '19 at 0:21

Archie Shahidullah

3581 silver badge7 bronze badges

2

votes

1answer

347 views

Is there a ReLU-like activation function that concatenates positive and negative values?

Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.

convolutional-neural-networks relu

asked Jan 6 '19 at 21:54

Andrew Matiuk

1373 bronze badges

10

votes

1answer

2k views

What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...

neural-networks machine-learning activation-function relu sigmoid

asked Jan 4 '19 at 13:39

JSChang

1036 bronze badges

13

votes

1answer

16k views

What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...

neural-networks activation-function relu

asked Jul 24 '18 at 12:47

gvgramazio

5561 gold badge6 silver badges17 bronze badges

9

votes

2answers

5k views

Why do we prefer ReLU over linear activation functions?

The ReLU activation function is defined as follows $$y = \operatorname{max}(0,x)$$ And the linear activation function is defined as follows $$y = x$$ The ReLU nonlinearity just clips the values ...

neural-networks deep-learning comparison activation-function relu

asked May 19 '18 at 16:41

imflash217

4814 silver badges13 bronze badges

11

votes

1answer

2k views

How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...

neural-networks machine-learning activation-function relu

asked Mar 9 '18 at 10:45

user9947

1

vote

2answers

4k views

What is the derivative of the Leaky ReLU activation function?

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the ...

neural-networks backpropagation math activation-function relu

asked Feb 19 '18 at 1:45

Mike AI

1272 silver badges8 bronze badges

2

votes

2answers

236 views

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work? My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...

neural-networks reference-request activation-function relu cross-entropy

asked Jan 9 '18 at 14:54

Emil Wormbs

593 bronze badges

8

votes

3answers

1k views

Are ReLUs incapable of solving certain problems?

Background I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding, I ...

neural-networks activation-function function-approximation relu sigmoid

asked Nov 17 '16 at 20:46

Benjamin Chambers

2011 silver badge7 bronze badges

Questions tagged [relu]

Related Tags