Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com

Question

I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he gave in the lectures, I noticed that the formula for $\delta^L$, "error" of last activation layer, is slightly different to that derived in http://neuralnetworksanddeeplearning.com/chap2.html.

In Andrew's, it seems like there is no inclusion of the partial derivative da/dz, or $\sigma'(z)$, only the dC/da part.

However Michael Nielson does include that term:

Is this difference significant and why does it arise? Is it because the derivation Nielson goes through defines the Cost using the mean square errors, whereas Andrew Ng defines the cost using the -ylog(h(x))... one? Also will Nielson's equations score full marks on the Ng's assignment?

Thank you for reading.

Answer 1

It's a matter of different loss functions used. In Andrew Ng's C1W2L09 video he derives the loss gradient as $$\frac{d𝓛}{dz} = \frac{d𝓛}{da}*\frac{da}{dz}$$ using the loss (cost) function binary cross entropy whose derivative is $\frac{d𝓛}{da} = -\frac{y}{a} + \frac{1 - y}{1 - a}$ and the activation function is sigmoid whose derivative is $\frac{da}{dz} = a(1-a)$. The product of the two is $$a - y$$ which is consistent with the vectorized form you showed.

Nielsen uses the quadratic loss function $L = \frac{1}{2}||y - a^L||^2$ whose derivative is as you've shown: $\frac{d𝓛}{da} = a^L - y$. His formula $\delta^L = (a^L - y) \odot \sigma'(z^L)$ is specific to quadratic loss.

So Andrew's $\frac{d𝓛}{dz} = \frac{d𝓛}{da}*\frac{da}{dz}$ is the same as Nielsen's matrix based $\delta^L=\nabla_aC \odot \sigma'(z^L)$, here are the equivalent terms (Andrew's on the left, Nielsen's on the right) $$\frac{dL}{da} = \nabla_aC $$ and $$\frac{da}{dz} = \sigma'(z^L)$$