Consider the following PyTorch code
# Run a sample training loop that "teaches" the network # to output the constant zero function for _ in range(10000): input = torch.randn(4) output = net(input) loss = torch.abs(output) net.zero_grad() loss.backward() optimizer.step()
and its corresponding explanation on training a neural network
A training loop…
- acquires an input,
- runs the network,
- computes a loss,
- zeros the network’s parameters’ gradients,
- calls loss.backward() to update the parameters’ gradients,
- calls optimizer.step() to apply the gradients to the parameters.
Code contains net.zero_grad() which has been explained as zeros the network’s parameters’ gradients.
What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase "zeros of networks parameters gradient". What does that particular step do?