For a deep NN, should I generally apply batch normalization after each convolution layer? Or only after some of them? Which? Every 2nd, every 3rd, lowest, highest, etc.?
In the literature, it differs. You will see models do it after or before pooling only, and sometimes you see it after every single convolution.
Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) but lately has been tied to the optimization process (How Does Batch Normalization Help Optimization?).
This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given some initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: Fixup Initialization).
So I would recommend 3 things until it is more understood how to utilize it generally:
- Play around, get frisky and experiment. Use what works best.
- Use block featurizers that are known to work well like residual blocks. Proven in practice and will probably work for you too.
- Do the research and investigate it more, if you find the answer, you'll be helping a lot of people :)