I have read a little about Batch-Normalization and I understood that there isn't any better option on where you place Batch-Normalization (it all depends on the case). However, I don't understand the point on applying Batch-Normalization followed by a ReLU.

ReLU transform all negative values to zero. But Batch-Normalization already transforms all values by restricting them to a value between zero and one. So ReLU will not have any effect when applied after BN, because there will not be any negative values left.

Am I right, or am I missing something?


2 Answers 2


Batch-normalization (BN) does NOT transform all values by restricting them to a value between zero and one.

BN performs two operations: a normalization, and a shifting with scaling. The normalization operation transforms the inputs to have approximately zero-mean and unitary variance, then the scaling by $\gamma$ and shift by $\beta$ is performed to give more power (or flexibility) to the next layers.

In general you want to apply BN before any activation function (like ReLU) because the BN tend to center the data in the "active" region of the non-linearity, thus speeding-up training.

Moreover, BN introduces some noise in the training which is often interpreted as an implicit form of regularization (thus boosting also generalization performance.)

  • $\begingroup$ If BN is applied before ReLU, couldn't be the case that all inputs to BN are positive, so after passing through the BN layer some of the become negative leading to dead neurons? $\endgroup$
    – ado sar
    May 11, 2023 at 23:01
  • $\begingroup$ I guess this possibility cannot be excluded with certainty, but consider that BN also learns a shifting coefficient $\beta$ that is added after standardization. So the learning dynamics may push $\beta$ to be large enough to still yield positive values that would be active after the ReLU. $\endgroup$ May 12, 2023 at 8:14

Although you can normalize the features and initialize the weights with Normal Xavier initialization, during propagation the weights size can still propagate through a deep neural network, causing Vanishing/exploding gradients. Think about "very deep" neural networks with +50 layers, how easily the weights could go bananas.

Batch-normalization normalizes after each batch, continuously keeping the weights in check.

You also write it "works" but that does not say anything about performance. It might be better to use Leaky ReLU to avoid dead neurons which seem very possible with the a feature and weight normalized relu activation neural network.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .