Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is
$$ \frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3}, $$ which can be translated to plain English as follows
The change in cost, $C$, with respect to the bias of a neuron, $b_{j}^{l}$, is equal to the error in that neuron, $\delta_{j}^{l}$.
Why would this be the case?
This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.
Now, if you decided to read further, here's the sketch of a proof.
Recall equations (25) and (29), both definitions:
$z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
$\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l = \dfrac{\partial C}{\partial z^l}$
The chain rule itself:
$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$
Almost all elements in this sum will be zero, except for one when $k=j$. The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.
The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.
