Consider the following description regarding gradient clipping in PyTorch

torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=2.0, error_if_nonfinite=False)

Clips gradient norm of an iterable of parameters.

The norm is computed over all gradients together as if they were concatenated into a single vector. Gradients are modified in-place.

Let the weights and gradients, for loss function $L$, of the model, be given as below

\begin{align} w &= [w_1, w_2, w_3, \cdots, w_n] \\ \triangledown &= [\triangledown_1, \triangledown_2, \triangledown_3, \cdots, \triangledown_n] \text{, where } \triangledown_i = \dfrac{\partial L}{\partial w_i} \text{ and } 1 \le i \le n \end{align}

From the description, we need to compute gradient norm, i.e. $||\triangledown||$.

How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?


Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,

$$ \text{if } \Vert \mathbf{g} \Vert \geq c, \text{then } \mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} $$

where $c$ is a hyperparameter, $\mathbf{g}$ is the gradient, and $\Vert \mathbf{g} \Vert$ is the norm of $\mathbf{g}$.

Since $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$ is a unit vector, after rescaling the new $\mathbf{g}$ will have norm $c$.

Note that if $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} < c$ , then we don’t need to do anything.

Check this article for more information

  • 3
    $\begingroup$ This appears to be copy-pasted from another source, without appropriate attribution. We have certain rules that we require you to follow whenever copying from another source: ai.stackexchange.com/help/referencing. Please edit your answer to follow them. $\endgroup$
    – D.W.
    Dec 27 '21 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.