3
$\begingroup$

I've started to learn about neural networks recently and I can't find the answer to this question.

Let's assume there's a neural network (fig. 1) Figure 1

So if the loss function is: Loss function

and the derivative deriv is: Derivative definition

if I want to use this to find dE/dw122 what k and l (well there's only one neuron with index l here, but what if there would be more?) should i use in w^3_kl and w^2_jk?

I've also found "other" way of backpropagating it's described here, but I can't understand how they came up with that method from the original equation w -= step * dE/dw.

Sorry if I failed to explain my problem. If something isn't clear please ask in comments.

$\endgroup$
1
$\begingroup$

First I will assume you notate $y$ as the models output and $z$ as the ground-truth. Second, I am assuming this is a linear model (No activation functions). Then the gradient math goes as so:

$ \begin{align*} \frac{dE}{dw_{ij}^1} &= \frac{dE}{dy}\frac{dy}{dw_{ij}^1} \\ &= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3}\frac{dn_{k}^3}{dw_{ij}^1} \\ &= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3} \frac{\partial n_{k}^3}{ \partial n_{j}^2} \frac{dn_{j}^2}{ dw_{ij}^1} \\ &= -2*(z-y)\sum_kw_{kl}^3w_{jk}^2 x_i \\ \end{align*} $

So the reason you are having trouble to figure out which $k$ index to use, the answer is because you need to use both and sum over them. The $l$ index is just the only $l$ index that exists because you only have one node in that layer.

$\endgroup$
  • $\begingroup$ Thank you. I'll have to learn more about derivatives apparently. I would up-vote but I don't have enough reputation, sorry. $\endgroup$ – Sharpy Jul 29 at 13:11
  • $\begingroup$ @Sharpy glad to help and there are tons of online resource going over the differential calculus if you need the refresher, Also if it answered your question, you can still checkmark the answer as correct even if you cant upvote $\endgroup$ – mshlis Jul 29 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.