0
$\begingroup$

As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero and as we train the net the gradients will arrive to values nearest to zero. But it is not the case as you can see for example here (This image show gradients distribution on each epoch):

enter image description here

https://wandb.ai/ayush-thakur/debug-neural-nets/reports/Visualizing-and-Debugging-Neural-Networks-with-PyTorch-and-W-B--Vmlldzo2OTUzNA

and here (This image show gradients for 5 layers after the first batch):

enter image description here

https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial3/Activation_Functions.html

I've seen the same behavior in other simple nets. Can someone explain this?

$\endgroup$
3
  • $\begingroup$ I'm not sure how meaningful gradient distributions are. Try observing L2 norm of gradients. That should get smaller as your model converges. $\endgroup$
    – SpiderRico
    May 9, 2022 at 20:14
  • $\begingroup$ Thanks for your response. I've tried but the distribution is basically the same (centered on zero). Why do you think that L2 is better? $\endgroup$ May 9, 2022 at 21:51
  • $\begingroup$ L2 norm of the gradient kinda tells you, on aggregate, whether your gradient gets smaller or not. Individual dimensions may vary. L2 norm of a vector is just a scalar value, you can just observe how this changes over the training, $\endgroup$
    – SpiderRico
    May 9, 2022 at 22:00

1 Answer 1

0
$\begingroup$

Nope. Delta rule: taylor series with truncation relates mean, variance, and gradient.

Variance is proportional to the square of the local slope in the region. If your minimum is at the bottom of a wide-area (think grand canyon) then you can have low variance. If the minimum is at the bottom of a well, then the local variance can be high.

The goal of optimizers (like those with momentum and adaptivity) is to test the local area and maybe bump the learner out of a local but not global best to something better nearby. That means you aren't exactly testing the point as much as you are testing the space around the point.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .