Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?

Question

Article 1: https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/

Article 2: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/

I was reading the 2 articles that teach backpropagation from scratch. I had a doubt about the first article, and so I would really like to ask specifically about the way he used to calculate the deltas of neurons in the hidden layer.

The first article, talking about calculating the delta in hidden layers, mentions the following:

the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer, followed by multiplication of the delta by the derivative of the nonlinear activation function for the activations of the current layer

However, in the second article, the explanation seems to be a little different, in the second article's explanation on how to calculate the delta he mentions

You can see that the error signal for neurons in the hidden layer is accumulated from neurons in the output layer where the hidden neuron number j is also the index of the neuron’s weight in the output layer neuron[‘weights’][j].

Analyzing this explanation from the second article, I had this doubt: the first article on the hidden layer says that the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer. However, in the second article he says that the delta in the hidden layer is calculated using the sum of the deltas of the next layer multiplied by the connection weights with the neuron with index J in the current hidden layer (in this case, the next layer is the output layer), this sum is multiplied by the derivative of the activation of the hidden layer neuron. I was very confused precisely because in the second article the weights that were used to calculate the delta are not the weights of the current layer (as is done in the first article), instead, the weights used in the second article are the weights of the next layer (That's it, the weights of the inputs of the neurons in the next layer, in this case the output layer)

I know that both articles are similar explanations, but I was confused. Is there any difference in the formulas of both articles? Or in practice are both equivalent? Could someone please help me understand these differences?

Answer 1

The key idea of backpropagation can be summarized as:

Each individual component of the gradient, $\partial C/\partial w_{jk}^{l}$ can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer – specifically the gradient of the weighted input of each layer, denoted by $\delta^{l}$ – from back to front... The $\delta ^{l}$ can easily be computed recursively, going from right to left, as: $$\delta ^{l-1}:=(f^{l-1})'\circ (W^{l})^{T}\cdot \delta ^{l}$$ The gradients of the weights can thus be computed using a few matrix multiplications for each level; this is backpropagation.

Once this crucial step is done, all the weights can be straightforwardly updated via the delta rule involving a learning rate. Therefore you 1st article's quoted section/code just essentially implements the above crucial gradients $\delta ^{l-1}$ of the weighted inputs of each layer from code line 99-111, and code line 113-125 implements the final weights delta update rule which additionally involves the multiplicative terms of learning rate and the respective weighted inputs of each layer in Numpy's matrix/array linear algebra way.

And the error signal of your 2nd article's quoted section ("the error signal for neurons in the hidden layer is accumulated from neurons in the output layer") is just the inner function part $(W^{l})^{T}\cdot \delta ^{l}$ of the composed gradient function $\delta ^{l-1}$. And at the end line 18 of the quoted code snippet you have:

neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])

Thus the same gradient of each layer $\delta ^{l-1}$ as expressed as neuron['delta'] here is obtained too after multiplied by the derivative of the activation/transfer function though here seems only primitive Python data types are employed instead of Numpy's linear algebra data types.