Consider the following PyTorch code

# Run a sample training loop that "teaches" the network
# to output the constant zero function
for _ in range(10000):
  input = torch.randn(4)
  output = net(input)
  loss = torch.abs(output)

and its corresponding explanation on training a neural network

A training loop…

  1. acquires an input,
  2. runs the network,
  3. computes a loss,
  4. zeros the network’s parameters’ gradients,
  5. calls loss.backward() to update the parameters’ gradients,
  6. calls optimizer.step() to apply the gradients to the parameters.

Code contains net.zero_grad() which has been explained as zeros the network’s parameters’ gradients.

What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase "zeros of networks parameters gradient". What does that particular step do?


In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all gradients, calcluated before, with the current gradient.

Therefore, optimizer.step() will do not this:

w = w - eta * grad L[i] # L[i] - loss function for the i-th sample

But rather:

w = w - eta * sum_i(grad L[i]) # sum of gradient with respect to all samples

Which is not the desired behavior.

  • $\begingroup$ Worth noting that this is an implementation detail in PyTorch, and other frameworks may do the same implicitly. It is a nice feature when there's more than one source of gradients, but for simple NNs it's just some boilerplate code that needs to be written by the framework user $\endgroup$ Jul 18 at 12:09
  • $\begingroup$ So, if we don't use net.zero_grad() then the updation of weights will go wrong because of issue in updation of gradients. Updation of weight takes the sum of all the gradients calculated so far instead of the current gradient. $\endgroup$
    – hanugm
    Jul 18 at 23:07
  • $\begingroup$ @hanugm, yes, precisely, it can be beneficial to use gradient information from the previous timesteps, like in the momentum optimizers - Nesterov, Adam, but not in tihs way. With the decay of the importance of gradients from previous timestamps with time. $\endgroup$ Jul 19 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.