1
$\begingroup$

Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training.

Since we train a neural network by dividing training data into batches, we use the word batch normalization as we consider a batch of training vectors at a time.

My doubt is about the position of batch normalization layers in a neural network.

Is it present before the input layer only? Or is it before every layer? Or is it dependent on the underlying task?

Suppose there is a neural network of 4 layers: $I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$

Which one of the following is true

$$bn \rightarrow I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$$

$$bn1 \rightarrow I \rightarrow bn2 \rightarrow h1 \rightarrow bn3 \rightarrow h2 \rightarrow bn4 \rightarrow h3 \rightarrow bn5 \rightarrow O$$

Here $I$ stands for input layer, $h$ for the hidden layer, $O$ for output layer, and $bn$ for batch normalization layer.

$\endgroup$
1
  • $\begingroup$ Batch normalization can be applied to any input or hidden layer in a network. p:318 $\endgroup$
    – hanugm
    Jul 22 at 3:30
1
$\begingroup$

I think the answer to your question is much more a rule of thumb than an appropriate analytical answer. First of all, I would like to remark that Batch Normalization [1] are applied most commonly to convolutional layer, constituting what is called a "convolutional block" (Convolution + Batch Normalization + Activation). Thus, for giving you an idea on where to put the normalization layers, I will analyze three papers that make use of Batch Normalization.

Unsupervised representation learning with deep convolutional generative adversarial networks [2]. In Section 3, Approach and Model Architecture, the authors make the following remark:

Third is Batch Normalization [1] which stabilizes learning by normalizing the input to each unit to have zero mean and unit variance. This helps deal with training problems that arise due to poor initialization and helps gradient flow in deeper models. This proved critical to get deep generators to begin learning, preventing the generator from collapsing all samples to a single point which is a common failure mode observed in GANs. Directly applying batchnorm to all layers however, resulted in sample oscillation and model instability. This was avoided by not applying batchnorm to the generator output layer and the discriminator input layer.

thus, the authors have experimented with Batch Normalization on all layers, but this approach resulted in model instability. They ultimately avoided Batch Normalization on the Generator's output layer, and Discriminator's input layer. These correspond to "raw image layers" (the output of the generator is an image, as well as the input of the discriminator). Using your notation, it would be something like:

$$G: z \rightarrow (C + BN + A)_{1} \rightarrow (C + A)_{2} \rightarrow O$$ $$D: I \rightarrow (C + A)_{1} \rightarrow (C + BN + A)_{2} \rightarrow h_{1} \rightarrow h_{2} \rightarrow O$$

where $G$ stands for Generator, $D$ for Discriminator, $C$ for convolution, $BN$ for batchnorm, $A$ for activation and $D_{i}$ for dense layers.

Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising [3]. In Figure 1., the authors show their proposed architecture that applies the same logic. Neither the first convolutional layer, neither the last use Batch Normalization. Although the authors make their point in justifying the usage of residual learning + Batch Normalization, the paper does not justify this architectural choice. The network would look something like this:

$$I \rightarrow (C + A)_{1} \rightarrow (C + BN + A)_{2} \rightarrow \cdots \rightarrow (C + BN + A)_{n} \rightarrow C \rightarrow O$$

Deep Residual Learning for Image Recognition [4]. This paper proposes a different scheme, as it applies Batch Normalization after each convolution operation on convolutional layers (thus including the input).

To sum up some papers use it only on the "hidden convolutional blocks", others on all of them. My advice is that, if you have the time, you should compare the two approaches. Remark: maybe I am unaware of some further development on the matter that gives a precise argument in favor of whether of these approaches.

References

[1] Ioffe, S., & Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). PMLR.

[2] Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.

[3] Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7), 3142-3155.

[4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.