# How does adding a small change to an neuron's weighted input affect the overall cost?

I was reading the following book: http://neuralnetworksanddeeplearning.com/chap2.html

and towards the end of equation 29, there is a paragraph that explains this:

However I am unsure how the equation below is derived:

• The expression you give in your final statement is the definition of a math derivative. Partial, because cost depends on all other network parameters, and using $\Delta$ instead of differential, but it is the concept of derivative:change in result due to infinitesimal change in input. – pasaba por aqui Jun 5 '19 at 14:52
• @pasabaporaqui It's not the definition of derivative. The problem with that notation is that if you change a parameter by a certain additive amount the derivative will not change by the same multiplicative factor (in general). – nbro Jun 5 '19 at 15:01
• I think for state functions where you can write down the total differential, using Euler's Rule and the Chain Rule, C depends upon z. So, the infinitesimal change in C, dC in some arbitrary dimension is given by: dC = (delta C/delta z)*dz – mamauwu Jun 5 '19 at 15:05
• @nbro: $df = f' dx$, $\Delta f = f' \Delta x$, $f' = lim ( \Delta x -> 0 ) \Delta f / \Delta x$, ... lots of ways to write same concept: change in result due to small change in input. – pasaba por aqui Jun 5 '19 at 15:07
• @flying_costa: there are several usual ways to generalize same concept to functions of several arguments: partial derivative, gradient ( $\nabla$) ... In the text you quote, partial derivative is used for all parameters $z_i^j$ with $i$ in ... and $j$ in .... – pasaba por aqui Jun 5 '19 at 16:38

I believe he's just saying that:

$$\frac{\partial C}{\partial z_j^l} \Delta z_j^l \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C$$

so that the change in cost function can be arrived at simply for a small enough perturbation $$\Delta z_j^l$$.

Or, taking that line of approximations backwards, the change in the cost function for a given perturbation is just: $$\partial C \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \frac{\partial C}{\partial z_j^l} \Delta z_j^l$$

• The $\approx$ in $\frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C$ can be replaced with an $=$ (it is not just an approximation, it is an exact equation). – nbro Jun 5 '19 at 15:04
• How does this answer the question? You're just approximating $\partial C$. It doesn't not explain why, if you add $\Delta z$ to $z$, then the derivative of $C$ w.r.t. $z$ needs to be multiplied by $\Delta z$. – nbro Jun 5 '19 at 15:09
• @nbro To respond to your first comment: Well, I wrote it like that because it's not a very mathematically rigorous thing to do to write it with strict equality. It's probably OK here, but I haven't dug into it. – Peter K. Jun 5 '19 at 15:20
• @nbro I'm confused by your second comment: It doesn't not explain why ??? Not sure what you're saying. – Peter K. Jun 5 '19 at 15:21
• Why isn't it a mathematically rigorous thing? It is just another way of writing $\frac{\partial C}{\partial z} = \frac{\partial C}{\partial z}$. – nbro Jun 5 '19 at 15:22

I think that Nielsen just wanted to convey the idea of the back-propagation algorithm using that formula, as you can read from the next paragraph "Now, this demon is a good demon...", so I don't think that that partial derivative is mathematically correct, provided the partial derivative is still with respect to $$z_j^l$$.

$$C$$ is the cost (or loss) function. $$z_j^l$$ is the linear output of neuron $$j$$ in layer $$l$$, which is followed by a non-linear function (e.g. sigmoid), denoted by $$\sigma$$. So, the actual output of neuron $$j$$ in layer $$l$$ is $$\sigma(z_j^l)$$.

The partial derivative of the cost function $$C$$ with respect to this neuron's linear output, $$z_j^l$$, is $$\frac{\partial C}{\partial z_j^l} = \frac{\partial C}{\partial z_j^l} 1 = \frac{\partial C}{\partial z_j^l} \frac{\partial z_j^l}{\partial z_j^l}.$$

If the the linear output of node $$j$$ in layer $$l$$ is now $$z_j^l + \Delta z_j^l$$, then the partial derivative with respect to $$z_j^l$$ becomes

\begin{align} \frac{\partial C}{\partial z_j^l} &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)}\frac{\partial (z_j^l + \Delta z_j^l)}{\partial z_j^l} \\ &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} \left( \frac{\partial z_j^l}{\partial z_j^l} + \frac{\partial \Delta z_j^l}{\partial z_j^l} \right) \\ &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} \left( 1 + \frac{\partial \Delta z_j^l}{\partial z_j^l} \right) \\ &= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} + \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)}\frac{\partial \Delta z_j^l}{\partial z_j^l} \\ \end{align}

$$\Delta z_j^l$$ depends on $$z_j^l$$, but it is not specified how.

The derivative of a function ($$f(x_1,x_2..x_n)$$) w.r.t to one of the variables ($$x_1,x_2..x_n$$) gives us the rate of change of the function w.r.t the rate of change of the variable. This roughly means that by how much will the function value change if we change the variable by a "unit amount" or $$+1$$. (we cannot use the change as $$+1$$ as the change needs to be infinitesimally small, this is just a rough explanation)

The image shows a tangent line to a curve or function $$f(x)$$. The slope of this tangent is given by $$\frac{df(x)}{dx}$$ at that particular $$x$$. If you move by a very very small amount in the direction of positive $$x$$ i.e. $$x+\delta x$$ the change in the value of the $$f(x)= y$$ will almost be the same as the change in the value of the $$y$$ of the tangent line.

Now as per the excerpt the cost function $$C$$ is a function of $$z_j^l$$. Thus, it can be written as $$C = f(z_j^l, .....).$$ So, $$\frac{\partial C}{\partial z_j^l}$$ indicates how much $$C$$ will vary w.r.t $$z_j^l$$, i.e. when $$z_j^l$$ is changed by an infinitesimally small amount and thus the formulae: $$\frac{\partial C}{\partial z_j^l} \Delta z_j^l$$ where the author assumed $$\Delta z_j^l$$ to be very small. This gives the infinitesimal change in $$C$$ or gives us $$\Delta C$$, for infinitesimally small change in $$z_j^l$$ or any varible affecting the cost function. This can be derived by series expansions too (given below), but this is an intuitive explanation.

An explanation can be given from Taylor Series Theorem which states: :

Let $$f(x)$$ be a function which is analytic at $$x = a$$. Then we can write $$f(x)$$ as the following power series, called the Taylor series of $$f(x)$$ at $$x = a$$, then we can write $$f(x)$$ as:

$$f(x) = f(a) + f'(a)(x-a) + f''(a)\frac{(x-a)^2}{2!} + f'''(a)\frac{(x-a)^3}{3!}...$$

Now if we keep other variables constant and make cost function $$f$$ vary only with $$z^l_j$$ and if we put $$a=z^l_j$$ and $$x=z^l_j + \Delta z^l_j$$ the equation becomes:

$$f(z^l_j + \Delta z^l_j) = f(z^l_j) + f'(z^l_j)(\Delta z^l_j) + f''(z^l_j)\frac{(\Delta z^l_j)^2}{2!} + f'''(a)\frac{(\Delta z^l_j)^3}{3!}...$$

which if we ignore the higher order terms of $$\Delta z^l_j$$, since terms containing $$\Delta z^l_j$$ for powers greater than 1 will be negligible compared to $$\Delta z^l_j$$ with power 1. Thus the equation now effectively is:

$$f(z^l_j + \Delta z^l_j) = f(z^l_j) + f'(z^l_j)(\Delta z^l_j)$$

$$f(z^l_j + \Delta z^l_j) - f(z^l_j)= f'(z^l_j)(\Delta z^l_j)$$

$$f(z^l_j + \Delta z^l_j) - f(z^l_j)= \frac{\partial f(z^l_j)}{\partial z^l_j}(\Delta z^l_j)$$ where $$f(z^l_j + \Delta z^l_j) - f(z^l_j)$$ can be thought of as $$\Delta C$$ or the change in cost function for small change in $$z^l_j$$

NOTE: I have glossed over some requirements for a Taylor Series to be convergent.

• I don't think that this an answer is correct (also because the OP was asking for a mathematical explanation of the derivation of the formula, and you do not provide it). $\Delta C$ is not very meaningful, given that $C$ is a function. Furthermore, I think the OP understands the basics of derivatives, so there is no need to recap that, IMHO. – nbro Jun 4 '19 at 16:12
• @nbro $\Delta C$ is meaningful if it is read in the context of what was written before. Also I have said it is an intuitive explanation. – DuttaA Jun 4 '19 at 16:14
• But I don't think that, when $z$ is changed by $\Delta z$, the derivative of $C$ w.r.t. $z$ is represented by that formula. See my answer (which can also be wrong, if I made some mistake). – nbro Jun 4 '19 at 16:18
• @nbro funnily enough you choose to use approximations and just divide $\partial z$ by $\partial z$ and when I do approximations it suddenly wrong and a downvote. – DuttaA Jun 4 '19 at 16:24
• That's not an approximation at all. That's an exact calculation. The derivative of $x$ with respect to $x$ is $1$. It's like saying $a/a = 1$. – nbro Jun 4 '19 at 16:28