If I have a neural network, and say the 6th output node of the neural network is:

$$x_6 = w_{16}y_1 + w_{26}y_2 + w_{36}y_3$$

What does that make the derivative of:

$$\frac{\partial x_6}{\partial w_{26}}$$

I guess that it's how is $x_6$ changing with respect to $w_{26}$, so, therefore, is it equal to $y_2$ (since the output, $y_2$, will change depending on the weight added to the input)?


Formally speaking $x_6$ is a function of $w_{16},\ w_{26}$ and $w_{36}$, that is $$x_6 =f(w_{16}, w_{26}, w_{36})=w_{16}y_1 + w_{26}y_2 + w_{36}y_3.$$ The derivative w.r.t. $w_{26}$ is $$\frac{\partial x_6}{\partial w_{26}}= \frac{\partial w_{16}y_1}{\partial w_{26}} +\frac{\partial w_{26}y_2}{\partial w_{26}} +\frac{\partial w_{36}y_3}{\partial w_{26}} = 0 +y_2 \frac{\partial w_{26}}{\partial w_{26}} + 0= y_2.$$ The first equality is obtained using the fact that the partial derivative is linear (so the derivative of the sum is the sum of the derivatives); the second equality comes from again from the linearity and from the fact that $w_{16}y_1$ and $w_{36}y_3$ are constants with respect to $w_{26}$, so their partial derivative w.r.t. this variable is $0$.


Not really asked in the original question, but since I'm here let me have a bit of fun ;).

Let's say $x_6$ is the output of the sixth node after you apply an activation function, that is $$x_6 =\sigma(f(w_{16}, w_{26}, w_{36}))=\sigma(w_{16}y_1 + w_{26}y_2 + w_{36}y_3).$$ You can compute the partial derivative applying the properties illustrated above, with the additional help of the chain rule $$\frac{\partial x_6}{\partial w_{26}}=\frac{\partial \sigma(w_{16}y_1 + w_{26}y_2 + w_{36}y_3)}{\partial w_{26}}=\sigma'\frac{\partial w_{16}y_1}{\partial w_{26}} +\sigma'\frac{\partial w_{26}y_2}{\partial w_{26}} +\sigma'\frac{\partial w_{36}y_3}{\partial w_{26}}=y_2\sigma'$$ $\sigma'$ denotes the derivative of sigma with respect to its argument.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.