# How to explain near zero gradients on first epochs?

As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero and as we train the net the gradients will arrive to values nearest to zero. But it is not the case as you can see for example here (This image show gradients distribution on each epoch):

https://wandb.ai/ayush-thakur/debug-neural-nets/reports/Visualizing-and-Debugging-Neural-Networks-with-PyTorch-and-W-B--Vmlldzo2OTUzNA

and here (This image show gradients for 5 layers after the first batch):

I've seen the same behavior in other simple nets. Can someone explain this?

• I'm not sure how meaningful gradient distributions are. Try observing L2 norm of gradients. That should get smaller as your model converges. May 9, 2022 at 20:14
• Thanks for your response. I've tried but the distribution is basically the same (centered on zero). Why do you think that L2 is better? May 9, 2022 at 21:51
• L2 norm of the gradient kinda tells you, on aggregate, whether your gradient gets smaller or not. Individual dimensions may vary. L2 norm of a vector is just a scalar value, you can just observe how this changes over the training, May 9, 2022 at 22:00