2
$\begingroup$

I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation:

direction_and_amount = (pred - goal_pred) * input
weight = weight - direction_and_amount

It explains the motivation behind multiplying the prediction difference with input using three cases: scaling, negative reversal and stopping.

What are scaling, negative reversal, and stopping? These three attributes have the combined effect of translating the pure error into the absolute amount you want to change weight. They do so by addressing three major edge cases where the pure error isn’t sufficient to make a good modification to weight.

These three cases are:

  1. Negative input,
  2. zero input and
  3. the value of input (scaling).

Negative and zero cases are very obvious. However, I didn't understand scaling. Regarding scaling, there's the following explanation:

Scaling is the third effect on the pure error caused by multiplying it by input. Logically, if the input is big, your weight update should also be big. This is more of a side effect, because it often goes out of control. Later, you’ll use alpha to address when that happens.

But I didn't understand it. Considering the linear regression problem, why weight update should be big if the input is big?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Just to be clear, what is the loss function the book is using in that section? What neural network is the book trying to describe the training of? This seems to be just an interpretation of the derivative of the loss function (i.e. back-propagation, aka the application of the chain rule to compute derivatives of compositive functions). $\endgroup$ – nbro♦ Nov 10 '20 at 14:33
  • 1
    $\begingroup$ Thanks for the response. It's MSE loss. Well, it's just an introduction in the book, single input single output linear function. I also read many loss and transfer/activation functions. In every combination, somehow error is multiplied by the input at the end. I know this multiplication comes from chain rule, but I want to know if there can be an intuition behind it other than calculus definition. $\endgroup$ – user42191 Nov 10 '20 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.