While reading the the book on neural network http://neuralnetworksanddeeplearning.com/chap2.html by Michael Nielson I had a problem of understanding eqn BP3. Which reads as "Change in cost wrt bias in a neuron is equals to error in that neuron". (Sorry unable to put the eqn here.)


This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.

Now, if you decided to read further, here's the sketch of a proof.

Recall equations (25) and (29), both definitions:

  • $z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$

  • $\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l = \dfrac{\partial C}{\partial z^l}$

The chain rule itself:

  • the partial derivative of $C$ with respect to $b^l$:

$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$

Almost all elements in this sum will be zero, except for one when $k=j$. The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.