2
$\begingroup$

I'm new to the field of AI (though I have a background in mathematics).

As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping elementwise (as opposed to normalization) would change that direction.

Why then would we choose to clip instead of normalization?

The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.

What is the justification for this approach of clipping elementwise?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Intro: Gradient clipping is in the context of neural networks, which have multiple layers meaning that each layer has learnable parameters (vectors and matrices) so you actually get a list of gradients: one per layer, where each gradient is not a scalar (can be) but rather a tensor with the same shape of the parameters in the layer (this is due to reducing the Jacobian of the layer'weights to a gradient instead.)

Gradient clipping: to prevent spikes during training (and so avoid instabilities) it is possibile to clip (i.e. limit) each value to a maximum value. This operation does not change the direction of improvement but rather reduce it's magnitude because is performed in absolute value: say $g$ is the gradient and want to clip to $v$ (where $v>0$), the operation is $g=\text{clip}(g, -v, v)$.

Moreover there are alternatives to this:

  • Normalizing each gradient by its $l_2$-norm, which basically scales each layer's gradient independently. Or
  • Scaling the gradients by the global norm of all the gradients (so the list considered as a whole.)

Why then, would we choose to clip instead of normalization?

You decide it by trial and error, in some applications clipping is sufficient in some other normalization by norm is better.

Anyway, in both cases the training may experience a slow-down since the magnitude of the gradients are reduced and so the step size is too.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Sorry, I don't follow this: "it is possible to clip (i.e. limit) each value to a maximum value. This operation does not change the direction of improvement ...". Since (as you rightly pointed out) there are multiple layers with matrices and vectors to optimize, we can reshape all of these to one giant vector, and we are optimizing J(theta). From my understanding, gradient clipping (as opposed to normalization or scaling) looks at dTheta, and clips/saturates some values if they exceed a scalar bound. How does this not change the direction of the dTheta vector? $\endgroup$
    – Ukn0wn
    Commented Apr 28, 2023 at 16:58
  • $\begingroup$ You're right, what I not specified is that clipping occurs in absolute value because you don't want to change the sign of the gradient. Suppose $d\theta=-10$ and your clip value is $v=1$ (must be positive) what you do is $clip(d\theta, -v, v) = -1$. This code confirms this. I'm going to add this detail to the answer. $\endgroup$
    – Luca Anzalone
    Commented Apr 28, 2023 at 17:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .