Questions tagged [relu]

For questions concerning neural network rectifiers (ReLU)

5 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
69 views

Should batch normalisation be applied before or after ReLU?

I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus. The crucial thing that I haven't seen mentioned in these discussions is that applying ...
2
votes
0answers
66 views

How is the bias caused by a max pooling layer overcome?

I have constructed a CNN that utilizes max-pooling layers. I have found with these layers that, should I remove them, my network performs ideally with every output and gradient at each layer having a ...
1
vote
0answers
33 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
1
vote
0answers
46 views

Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?

Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...
1
vote
1answer
87 views

Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...