What is the derivative of a specific output with respect to a specific weight?

Question

If I have a neural network, and say the 6th output node of the neural network is:

$$x_6 = w_{16}y_1 + w_{26}y_2 + w_{36}y_3$$

What does that make the derivative of:

$$\frac{\partial x_6}{\partial w_{26}}$$

I guess that it's how is $x_6$ changing with respect to $w_{26}$, so, therefore, is it equal to $y_2$ (since the output, $y_2$, will change depending on the weight added to the input)?

Answer 1

Formally speaking $x_6$ is a function of $w_{16},\ w_{26}$ and $w_{36}$, that is $$x_6 =f(w_{16}, w_{26}, w_{36})=w_{16}y_1 + w_{26}y_2 + w_{36}y_3.$$ The derivative w.r.t. $w_{26}$ is $$\frac{\partial x_6}{\partial w_{26}}= \frac{\partial w_{16}y_1}{\partial w_{26}} +\frac{\partial w_{26}y_2}{\partial w_{26}} +\frac{\partial w_{36}y_3}{\partial w_{26}} = 0 +y_2 \frac{\partial w_{26}}{\partial w_{26}} + 0= y_2.$$ The first equality is obtained using the fact that the partial derivative is linear (so the derivative of the sum is the sum of the derivatives); the second equality comes from again from the linearity and from the fact that $w_{16}y_1$ and $w_{36}y_3$ are constants with respect to $w_{26}$, so their partial derivative w.r.t. this variable is $0$.

Bonus

Not really asked in the original question, but since I'm here let me have a bit of fun ;).

Let's say $x_6$ is the output of the sixth node after you apply an activation function, that is $$x_6 =\sigma(f(w_{16}, w_{26}, w_{36}))=\sigma(w_{16}y_1 + w_{26}y_2 + w_{36}y_3).$$ You can compute the partial derivative applying the properties illustrated above, with the additional help of the chain rule $$\frac{\partial x_6}{\partial w_{26}}=\frac{\partial \sigma(w_{16}y_1 + w_{26}y_2 + w_{36}y_3)}{\partial w_{26}}=\sigma'\frac{\partial w_{16}y_1}{\partial w_{26}} +\sigma'\frac{\partial w_{26}y_2}{\partial w_{26}} +\sigma'\frac{\partial w_{36}y_3}{\partial w_{26}}=y_2\sigma'$$ $\sigma'$ denotes the derivative of sigma with respect to its argument.