Skip to main content
deleted 6 characters in body; edited title
Source Link
nbro
  • 41.4k
  • 12
  • 115
  • 205

Gradient What is the justification for this approach of clipping: elementwise?

I'm new to the field of AI (though I have a background in mathematics). 

As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping element wiseelementwise (as opposed to normalization) would change that direction.

Why then, would we choose to clip instead of normalization? 

The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.

What is the justification for this approach of clipping element wiseelementwise?

Thank you

Gradient clipping: elementwise

I'm new to the field of AI (though I have a background in mathematics). As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping element wise (as opposed to normalization) would change that direction.

Why then, would we choose to clip instead of normalization? The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.

What is the justification for this approach of clipping element wise?

Thank you

What is the justification for this approach of clipping elementwise?

I'm new to the field of AI (though I have a background in mathematics). 

As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping elementwise (as opposed to normalization) would change that direction.

Why then would we choose to clip instead of normalization? 

The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.

What is the justification for this approach of clipping elementwise?

Source Link

Gradient clipping: elementwise

I'm new to the field of AI (though I have a background in mathematics). As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping element wise (as opposed to normalization) would change that direction.

Why then, would we choose to clip instead of normalization? The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.

What is the justification for this approach of clipping element wise?

Thank you