What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?

Question

Consider the following PyTorch code

# Run a sample training loop that "teaches" the network
# to output the constant zero function
for _ in range(10000):
  input = torch.randn(4)
  output = net(input)
  loss = torch.abs(output)
  net.zero_grad()
  loss.backward()
  optimizer.step()

and its corresponding explanation on training a neural network

A training loop…

1. acquires an input,
2. runs the network,
3. computes a loss,
4. zeros the network’s parameters’ gradients,
5. calls loss.backward() to update the parameters’ gradients,
6. calls optimizer.step() to apply the gradients to the parameters.

Code contains net.zero_grad() which has been explained as zeros the network’s parameters’ gradients.

What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase "zeros of networks parameters gradient". What does that particular step do?

Answer 1

In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all gradients, calcluated before, with the current gradient.

Therefore, optimizer.step() will do not this:

w = w - eta * grad L[i] # L[i] - loss function for the i-th sample

But rather:

w = w - eta * sum_i(grad L[i]) # sum of gradient with respect to all samples

Which is not the desired behavior.

Answer 2

I agree with the answer from @spiridon_the_sun_rotator .

If you don't zerorize the gradient, before calling loss.backward() then gradient will be accumulated which is not what you want. you want to calculate the gradient associated with the current step.