I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation:
direction_and_amount = (pred - goal_pred) * input
weight = weight - direction_and_amount
It explains the motivation behind multiplying the prediction difference with input using three cases: scaling, negative reversal and stopping.
What are scaling, negative reversal, and stopping? These three attributes have the combined effect of translating the pure error into the absolute amount you want to change weight. They do so by addressing three major edge cases where the pure error isn’t sufficient to make a good modification to weight.
These three cases are:
- Negative input,
- zero input and
- the value of input (scaling).
Negative and zero cases are very obvious. However, I didn't understand scaling. Regarding scaling, there's the following explanation:
Scaling is the third effect on the pure error caused by multiplying it by input. Logically, if the input is big, your weight update should also be big. This is more of a side effect, because it often goes out of control. Later, you’ll use alpha to address when that happens.
But I didn't understand it. Considering the linear regression problem, why weight update should be big if the input is big?
