Consider the following PyTorch code

# Run a sample training loop that "teaches" the network
# to output the constant zero function
for _ in range(10000):
  input = torch.randn(4)
  output = net(input)
  loss = torch.abs(output)

and its corresponding explanation on training a neural network

A training loop…

  1. acquires an input,
  2. runs the network,
  3. computes a loss,
  4. zeros the network’s parameters’ gradients,
  5. calls loss.backward() to update the parameters’ gradients,
  6. calls optimizer.step() to apply the gradients to the parameters.

Code contains net.zero_grad() which has been explained as zeros the network’s parameters’ gradients.

What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase "zeros of networks parameters gradient". What does that particular step do?


2 Answers 2


In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all gradients, calcluated before, with the current gradient.

Therefore, optimizer.step() will do not this:

w = w - eta * grad L[i] # L[i] - loss function for the i-th sample

But rather:

w = w - eta * sum_i(grad L[i]) # sum of gradient with respect to all samples

Which is not the desired behavior.

  • $\begingroup$ Worth noting that this is an implementation detail in PyTorch, and other frameworks may do the same implicitly. It is a nice feature when there's more than one source of gradients, but for simple NNs it's just some boilerplate code that needs to be written by the framework user $\endgroup$ Commented Jul 18, 2021 at 12:09
  • $\begingroup$ So, if we don't use net.zero_grad() then the updation of weights will go wrong because of issue in updation of gradients. Updation of weight takes the sum of all the gradients calculated so far instead of the current gradient. $\endgroup$
    – hanugm
    Commented Jul 18, 2021 at 23:07
  • $\begingroup$ @hanugm, yes, precisely, it can be beneficial to use gradient information from the previous timesteps, like in the momentum optimizers - Nesterov, Adam, but not in tihs way. With the decay of the importance of gradients from previous timestamps with time. $\endgroup$ Commented Jul 19, 2021 at 18:22

I agree with the answer from @spiridon_the_sun_rotator .

If you don't zerorize the gradient, before calling loss.backward() then gradient will be accumulated which is not what you want. you want to calculate the gradient associated with the current step.

  • $\begingroup$ That answer has already been given. $\endgroup$
    – Chenmunka
    Commented Jul 9 at 16:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .