While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is

$$ \frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3}, $$ which can be translated to plain English as follows

The change in cost, $C$, with respect to the bias of a neuron, $b_{j}^{l}$, is equal to the error in that neuron, $\delta_{j}^{l}$.

Why would this be the case?


This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.

Now, if you decided to read further, here's the sketch of a proof.

Recall equations (25) and (29), both definitions:

  • $z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$

  • $\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l = \dfrac{\partial C}{\partial z^l}$

The chain rule itself:

  • the partial derivative of $C$ with respect to $b^l$:

$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$

Almost all elements in this sum will be zero, except for one when $k=j$. The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.