1
$\begingroup$

Why are the weights of a neural net updated only considering the old values of the later layer, not the already updated values?

I use this example to explain my problem. When applying the backpropagation chain rule, the weights of the previous layer ($w_1, w_2, w_3, w_4$) are updated making use of the chain rule:

$$\frac{\partial E_{total}}{\partial w_1} = \frac{\partial E_{total}}{\partial out_{h1}} * \frac{\partial out_{h1}}{\partial net_{h1}}*\frac{\partial net_{h1}}{\partial w_1}$$

He then says:

$$\frac{\partial net_{o1}}{\partial out_{h1}}=w_5$$

Although he has already calculated the updated value for $w_5$, he uses the old value of $w_5$ to update $w_1$? Because the updated value of $w_1$ will have an impact on the outcome together with the updated value of $w_5$?

$\endgroup$
1
$\begingroup$

The basic idea of gradient descent is:

  • Calculate the gradient of some score with respect to parameters that you can control

  • Take a step in the direction of that gradient that improves the score (subract a multiple of gradient - for gradient descent - if you want to minimise some cost function)

The backpropagation using chain rule in neural network layers is part of the first part, calculating the gradient.

If you interleaved these steps during a single calculation, by taking an update step before propagating the gradient over all layers, you would not propagate the true gradient, but some interim value. There is no guarantee this value would reflect the true gradient of the affected layer when compared to training data and loss functions.

There are some manipulations of the gradient that are acceptable and improve convergence. For instance momentum, and various forms of weighting based on previous gradient calculations. However, I am not aware of any successful attempt to perform updates with partially-calculated gradient then attempt to continue the interrupted gradient calculations over the mixed updated and not-yet-updated parameters.

It is also possible to calculate gradients and then only update some of the possible parameters. In some cases this is useful, for instance it is a good choice when performing transfer learning, by only updating a few layers close to the output. This constrains the network to keep a lot of its trained parameters as-is, which reduces the chances of over-fitting to the smaller dataset that the network is learning from when transferring.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.