What weights should I use while back-propagating?

Question

I've started to learn about neural networks recently and I can't find the answer to this question.

Let's assume there's a neural network (fig. 1)

So if the loss function is:

and the derivative is:

if I want to use this to find what k and l (well there's only one neuron with index l here, but what if there would be more?) should i use in and ?

I've also found "other" way of backpropagating it's described here, but I can't understand how they came up with that method from the original equation w -= step * dE/dw.

Sorry if I failed to explain my problem. If something isn't clear please ask in comments.

Answer 1

First I will assume you notate $y$ as the models output and $z$ as the ground-truth. Second, I am assuming this is a linear model (No activation functions). Then the gradient math goes as so:

$ \begin{align*} \frac{dE}{dw_{ij}^1} &= \frac{dE}{dy}\frac{dy}{dw_{ij}^1} \\ &= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3}\frac{dn_{k}^3}{dw_{ij}^1} \\ &= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3} \frac{\partial n_{k}^3}{ \partial n_{j}^2} \frac{dn_{j}^2}{ dw_{ij}^1} \\ &= -2*(z-y)\sum_kw_{kl}^3w_{jk}^2 x_i \\ \end{align*} $

So the reason you are having trouble to figure out which $k$ index to use, the answer is because you need to use both and sum over them. The $l$ index is just the only $l$ index that exists because you only have one node in that layer.