How does adding a small change to an neuron's weighted input affect the overall cost?

Question

I was reading the following book: http://neuralnetworksanddeeplearning.com/chap2.html

and towards the end of equation 29, there is a paragraph that explains this:

However I am unsure how the equation below is derived:

Answer 1

I believe he's just saying that:

$$ \frac{\partial C}{\partial z_j^l} \Delta z_j^l \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C $$

so that the change in cost function can be arrived at simply for a small enough perturbation $\Delta z_j^l$.

Or, taking that line of approximations backwards, the change in the cost function for a given perturbation is just: $$ \partial C \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \frac{\partial C}{\partial z_j^l} \Delta z_j^l $$

Answer 2

I think that Nielsen just wanted to convey the idea of the back-propagation algorithm using that formula, as you can read from the next paragraph "Now, this demon is a good demon...", so I don't think that that partial derivative is mathematically correct, provided the partial derivative is still with respect to $z_j^l$.

$C$ is the cost (or loss) function. $z_j^l$ is the linear output of neuron $j$ in layer $l$, which is followed by a non-linear function (e.g. sigmoid), denoted by $\sigma$. So, the actual output of neuron $j$ in layer $l$ is $\sigma(z_j^l)$.

The partial derivative of the cost function $C$ with respect to this neuron's linear output, $z_j^l$, is $$\frac{\partial C}{\partial z_j^l} = \frac{\partial C}{\partial z_j^l} 1 = \frac{\partial C}{\partial z_j^l} \frac{\partial z_j^l}{\partial z_j^l}.$$

If the the linear output of node $j$ in layer $l$ is now $z_j^l + \Delta z_j^l$, then the partial derivative with respect to $z_j^l$ becomes

\begin{align} \frac{\partial C}{\partial z_j^l} &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)}\frac{\partial (z_j^l + \Delta z_j^l)}{\partial z_j^l} \\ &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} \left( \frac{\partial z_j^l}{\partial z_j^l} + \frac{\partial \Delta z_j^l}{\partial z_j^l} \right) \\ &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} \left( 1 + \frac{\partial \Delta z_j^l}{\partial z_j^l} \right) \\ &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} + \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)}\frac{\partial \Delta z_j^l}{\partial z_j^l} \\ \end{align}

$\Delta z_j^l$ depends on $z_j^l$, but it is not specified how.

Answer 3

The derivative of a function ($f(x_1,x_2..x_n)$) w.r.t to one of the variables ($x_1,x_2..x_n$) gives us the rate of change of the function w.r.t the rate of change of the variable. This roughly means that by how much will the function value change if we change the variable by a "unit amount" or $+1$. (we cannot use the change as $+1$ as the change needs to be infinitesimally small, this is just a rough explanation)

The image shows a tangent line to a curve or function $f(x)$. The slope of this tangent is given by $\frac{df(x)}{dx}$ at that particular $x$. If you move by a very very small amount in the direction of positive $x$ i.e. $x+\delta x$ the change in the value of the $f(x)= y $ will almost be the same as the change in the value of the $y$ of the tangent line.

Now as per the excerpt the cost function $C$ is a function of $z_j^l$. Thus, it can be written as $$C = f(z_j^l, .....).$$ So, $$\frac{\partial C}{\partial z_j^l}$$ indicates how much $C$ will vary w.r.t $z_j^l$, i.e. when $z_j^l$ is changed by an infinitesimally small amount and thus the formulae: $$\frac{\partial C}{\partial z_j^l} \Delta z_j^l$$ where the author assumed $\Delta z_j^l$ to be very small. This gives the infinitesimal change in $C$ or gives us $\Delta C$, for infinitesimally small change in $z_j^l$ or any varible affecting the cost function. This can be derived by series expansions too (given below), but this is an intuitive explanation.

An explanation can be given from Taylor Series Theorem which states: :

Let $f(x)$ be a function which is analytic at $x = a$. Then we can write $f(x)$ as the following power series, called the Taylor series of $f(x)$ at $x = a$, then we can write $f(x)$ as:

$$f(x) = f(a) + f'(a)(x-a) + f''(a)\frac{(x-a)^2}{2!} + f'''(a)\frac{(x-a)^3}{3!}... $$

Now if we keep other variables constant and make cost function $f$ vary only with $z^l_j$ and if we put $a=z^l_j$ and $x=z^l_j + \Delta z^l_j$ the equation becomes:

$$f(z^l_j + \Delta z^l_j) = f(z^l_j) + f'(z^l_j)(\Delta z^l_j) + f''(z^l_j)\frac{(\Delta z^l_j)^2}{2!} + f'''(a)\frac{(\Delta z^l_j)^3}{3!}... $$

which if we ignore the higher order terms of $\Delta z^l_j$, since terms containing $\Delta z^l_j$ for powers greater than 1 will be negligible compared to $\Delta z^l_j$ with power 1. Thus the equation now effectively is:

$$f(z^l_j + \Delta z^l_j) = f(z^l_j) + f'(z^l_j)(\Delta z^l_j)$$

$$f(z^l_j + \Delta z^l_j) - f(z^l_j)= f'(z^l_j)(\Delta z^l_j)$$

$$f(z^l_j + \Delta z^l_j) - f(z^l_j)= \frac{\partial f(z^l_j)}{\partial z^l_j}(\Delta z^l_j)$$ where $$f(z^l_j + \Delta z^l_j) - f(z^l_j)$$ can be thought of as $\Delta C$ or the change in cost function for small change in $z^l_j$

NOTE: I have glossed over some requirements for a Taylor Series to be convergent.