What exactly happens in gradient clipping by norm?

Consider the following description regarding gradient clipping in PyTorch

Clips gradient norm of an iterable of parameters.

The norm is computed over all gradients together as if they were concatenated into a single vector. Gradients are modified in-place.

Let the weights and gradients, for loss function $$L$$, of the model, be given as below

\begin{align} w &= [w_1, w_2, w_3, \cdots, w_n] \\ \triangledown &= [\triangledown_1, \triangledown_2, \triangledown_3, \cdots, \triangledown_n] \text{, where } \triangledown_i = \dfrac{\partial L}{\partial w_i} \text{ and } 1 \le i \le n \end{align}

From the description, we need to compute gradient norm, i.e. $$||\triangledown||$$.

How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?

Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,

$$\text{if } \Vert \mathbf{g} \Vert \geq c, \text{then } \mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$$

where $$c$$ is a hyperparameter, $$\mathbf{g}$$ is the gradient, and $$\Vert \mathbf{g} \Vert$$ is the norm of $$\mathbf{g}$$.

Since $$\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$$ is a unit vector, after rescaling the new $$\mathbf{g}$$ will have norm $$c$$.

Note that if $$\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} < c$$ , then we don’t need to do anything.