I'm new to the field of AI (though I have a background in mathematics). As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping element wise (as opposed to normalization) would change that direction.

Why then, would we choose to clip instead of normalization? The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.

What is the justification for this approach of clipping element wise?

Thank you