0
$\begingroup$

Why are all weights of a neural net updated and not only the weights of the first hidden layer?

The error-influence of the prediction by the weights of a neural net is calculated using the chain rule. However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer. My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place. Where am I wrong?

$\endgroup$
1
$\begingroup$

However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer.

I don't see how the second statement follows from the first.

Each weight $w_i$ (not just the ones in the first layer) affects the loss $\mathcal{L}$ according to the partial derivative of $\mathcal{L}$ with respect to $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$. Intuitively, the partial derivative with respect to parameter tells you how the function is changing with respect to that parameter.

My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place.

I am not sure I understand your reasoning, but, typically, you update the parameters only after having computed all the partial derivatives. In other words, first, you compute all partial derivatives, i.e. the gradient with back-propagation (a fancy name to denote the application of the chain rule), then you update the parameters.

Why do you do this? In this case, the loss function is multi-variable function, so it depends on multiple variables. The gradient $\nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_N} \right]$ represents the direction (note that the gradient is a vector and vectors have direction) towards which your function is increasing or decreasing (depending on the sign of the gradient).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.