I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside each neuron in the hidden layers that defines how this is an important neuron for the correct result in some way? How does the network know how to tweak those weights for that specific neuron?



The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter.

A deeper look...

Suppose we have a NN with parameters $w_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute a loss $J$. The loss (or cost) function tells us how far off we are from the target. This is what we want to reduce, so that the predictions fall closer to the target.

By computing the partial derivative of the loss function $J$ w.r.t a parameter $w_i$

$$ \frac{\partial J}{\partial w_i} $$

the NN uncovers two pieces of information:

  • The slope of $J$ w.r.t $w_i$, which tells the NN how much $w_i$ affects $J$.
  • Its sign, which tells the NN which way to tweak $w_i$ to decrease (or increase) the value of $J$.

By making the parameter updates depend on the derivative

$$ w_i^{new} \leftarrow w_i^{old} - \lambda \frac{\partial J}{\partial w_i} $$

the NN causes parameters that affect the loss the most, to be updated the most.

You can think of this as: parameters that are more to blame for the network's mistakes (i.e. contribute more towards the loss) are forced to change the most, in the direction that will decrease the loss.

| improve this answer | |
  • 1
    $\begingroup$ @NeilSlater The first derivative can be positive or negative. If the first derivative is zero, it does not tell you anything about the nature of the extremum, which can be a minimum or maximum (or a saddle point). If we assume that $\lambda > 0$, then, if the first derivative is positive, we will be decreasing the weights. If it is negative, he will be increasing them. $\endgroup$ – nbro Nov 23 '19 at 15:50
  • 1
    $\begingroup$ @NeilSlater The derivative tells you the rate of change of the loss function. Depending on the sign of the derivative and depending on whether you use gradient descent or ascent (i.e. depending on whether you use $-$ or $+$ in the update step, respectively), you can increase or decrease the loss. I don't think that this answer is strictly incorrect (even though the answerer provides the update step for gradient descent). I've updated the answer to include "or increase". $\endgroup$ – nbro Nov 23 '19 at 21:12
  • 1
    $\begingroup$ @NeilSlater Your assumption is wrong. In general, the derivative function $f'$ tells you how the function $f$ is changing. Depending on the specific value of the derivative function $f'$ at a specific point $x$, $f'(x)$, you can know how $f$ is changing, it can be increasing or decreasing. More specifically, $f'(x)$ corresponds to the slope of the tangent line that goes through $f$ at $x$. This tangent can have a positive or negative slope, or it can also have a zero slope (e.g. at a maximum). See ugrad.math.ubc.ca/coursedoc/math100/notes/derivs/deriv6.html. $\endgroup$ – nbro Nov 23 '19 at 22:59
  • 1
    $\begingroup$ @NeilSlater So, no, the sign of the derivative does not tell you the direction that will increase the function, but it tells you how the function is changing, so it tells you how the function is increasing or decreasing. $\endgroup$ – nbro Nov 23 '19 at 23:02
  • 2
    $\begingroup$ @NeilSlater Wow, I'm a bit late to the party, but when I say "Its sign, which tells the NN which way to tweak $w_i$ to decrease the value of $J$", I mean that by knowing the sign you know how to change $w_i$ so that you can change $J$ and then I follow with the update equation that shows exactly that. I see what you're trying to say (in your first comment) and you are right about that, but I don't see why what I said is grammatically incorrect. I think of it like saying "by having a compass you know which way to travel south" and then you'd correct me by saying "but a compass points north". $\endgroup$ – Djib2011 Nov 24 '19 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.