How to perform gradient checking in a neural network with batch normalization?

Question

I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with gradient checking.

A few days ago, I implemented batch normalization using the formulas provided by the original paper. However, in contrast with learning/momentum/regularization, the batch normalization procedure behaves differently during fit and predict phases - both needed for gradient checking. As we fit the network, batch normalization computes each batch mean and estimates the population's mean to be used when we want to predict something.

In a similar way, I know we may not perform gradient checking in a neural network with dropout, since dropout turns some gradients to zero during fit and is not applied during prediction.

Can we perform gradient checking in NN with batch normalization? If so, how?

Answer 1

You should be able to do gradient checking as long as you fix the randomness by fixing the random seed, on python you might want to look at numpy.random.seed.

From http://cs231n.github.io/neural-networks-3/#ensemble :

When performing gradient check, remember to turn off any non-deterministic effects in the network, such as dropout, random data augmentations, etc. Otherwise these can clearly introduce huge errors when estimating the numerical gradient. The downside of turning off these effects is that you wouldn’t be gradient checking them (e.g. it might be that dropout isn’t backpropagated correctly). Therefore, a better solution might be to force a particular random seed before evaluating both (f(x+h)) and (f(x-h)), and when evaluating the analytic gradient.

Answer 2

This link from Stanford is by far the best resource on gradient checking that I have encountered so far:

http://cs231n.github.io/neural-networks-3/

I am sure it will help you a lot.

Pro Tip: make sure that you use "centered/central difference" formula for the derivative calculations and also use "relative error" (not absolute) to compare the two gradients.