When I look up the generalised delta rule equation for back-propogation, I am seeing two conflicting equations.

For example, here (slide 20), given $o$ (the output, defined in slide 18), $z$ (the activated output) and a target $t$, defined in slide 17, then:

$\frac{\delta E}{\delta Z} = o(1-o)(o-t)$

When I look for the same equation else, e.g. here, slide 14, it says, given $o$ the output and $y$ the label, then (using slightly different notation $\beta_k$):

$\beta_k = o_k(1-o_k)(y_k-o_k)$

I can see here that these two equations are almost the same, but not quite. One subtracts the output from the target, and one subtracts the target from the output.

The reason why I'm asking this is I'm trying to do question 29 and 30 of this paper, and they are using the second equation ($\beta_k$) but my college notes (that I can't copy and paste due to copyright) define the equation according to the first equation $\frac{\delta E}{\delta Z}$. I'm wondering which way is correct, do you subtract the target from the obtained output, or vice versa?

  • $\begingroup$ This may depend on the definition of the loss function. You should check that the loss function in the second case is equivalent to the loss function in the first. If you find the answer to your own question and nobody provides a formal answer meanwhile, feel free to write a formal answer to your own question below. $\endgroup$
    – nbro
    Jan 5 '21 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.