I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside each neuron in the hidden layers that defines how this is an important neuron for the correct result in some way? How does the network know how to tweak those weights for that specific neuron?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
tl;dr
The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.
A deeper look...
Suppose we have a NN with parameters $w_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute a loss $J$. The loss (or cost) function tells us how far off we are from the target. This is what we want to reduce, so that the predictions fall closer to the target.
By computing the partial derivative of the loss function $J$ w.r.t a parameter $w_i$ (so this is just one partial derivative and not the full gradient vector)
$$ \frac{\partial J}{\partial w_i} $$
the NN uncovers two pieces of information:
- The slope of $J$ w.r.t $w_i$, which tells the NN how much $w_i$ affects $J$.
- Its sign, which tells the NN which way to tweak $w_i$ to decrease (or increase) the value of $J$.
By making the parameter updates depend on the derivative
$$ w_i^{new} \leftarrow w_i^{old} - \lambda \frac{\partial J}{\partial w_i} $$
the NN causes parameters that affect the loss the most, to be updated the most.
You can think of this as: parameters that are more to blame for the network's mistakes (i.e. contribute more towards the loss) are forced to change the most, in the direction that will decrease the loss.
