Timeline for What is the justification for this approach of clipping elementwise?
Current License: CC BY-SA 4.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 28, 2023 at 17:42 | history | edited | Luca Anzalone | CC BY-SA 4.0 |
added 148 characters in body
|
| Apr 28, 2023 at 17:40 | comment | added | Luca Anzalone | You're right, what I not specified is that clipping occurs in absolute value because you don't want to change the sign of the gradient. Suppose $d\theta=-10$ and your clip value is $v=1$ (must be positive) what you do is $clip(d\theta, -v, v) = -1$. This code confirms this. I'm going to add this detail to the answer. | |
| Apr 28, 2023 at 16:58 | comment | added | Ukn0wn | Sorry, I don't follow this: "it is possible to clip (i.e. limit) each value to a maximum value. This operation does not change the direction of improvement ...". Since (as you rightly pointed out) there are multiple layers with matrices and vectors to optimize, we can reshape all of these to one giant vector, and we are optimizing J(theta). From my understanding, gradient clipping (as opposed to normalization or scaling) looks at dTheta, and clips/saturates some values if they exceed a scalar bound. How does this not change the direction of the dTheta vector? | |
| Apr 27, 2023 at 20:53 | history | answered | Luca Anzalone | CC BY-SA 4.0 |