1
$\begingroup$

Article 1: https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/

Article 2: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/

I was reading the 2 articles that teach backpropagation from scratch. I had a doubt about the first article, and so I would really like to ask specifically about the way he used to calculate the deltas of neurons in the hidden layer.

The first article, talking about calculating the delta in hidden layers, mentions the following:

the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer, followed by multiplication of the delta by the derivative of the nonlinear activation function for the activations of the current layer

However, in the second article, the explanation seems to be a little different, in the second article's explanation on how to calculate the delta he mentions

You can see that the error signal for neurons in the hidden layer is accumulated from neurons in the output layer where the hidden neuron number j is also the index of the neuron’s weight in the output layer neuron[‘weights’][j].

Analyzing this explanation from the second article, I had this doubt: the first article on the hidden layer says that the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer. However, in the second article he says that the delta in the hidden layer is calculated using the sum of the deltas of the next layer multiplied by the connection weights with the neuron with index J in the current hidden layer (in this case, the next layer is the output layer), this sum is multiplied by the derivative of the activation of the hidden layer neuron. I was very confused precisely because in the second article the weights that were used to calculate the delta are not the weights of the current layer (as is done in the first article), instead, the weights used in the second article are the weights of the next layer (That's it, the weights of the inputs of the neurons in the next layer, in this case the output layer)

I know that both articles are similar explanations, but I was confused. Is there any difference in the formulas of both articles? Or in practice are both equivalent? Could someone please help me understand these differences?

$\endgroup$

1 Answer 1

2
$\begingroup$

The key idea of backpropagation can be summarized as:

Each individual component of the gradient, $\partial C/\partial w_{jk}^{l}$ can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer – specifically the gradient of the weighted input of each layer, denoted by $\delta^{l}$ – from back to front... The $\delta ^{l}$ can easily be computed recursively, going from right to left, as: $$\delta ^{l-1}:=(f^{l-1})'\circ (W^{l})^{T}\cdot \delta ^{l}$$ The gradients of the weights can thus be computed using a few matrix multiplications for each level; this is backpropagation.

Once this crucial step is done, all the weights can be straightforwardly updated via the delta rule involving a learning rate. Therefore you 1st article's quoted section/code just essentially implements the above crucial gradients $\delta ^{l-1}$ of the weighted inputs of each layer from code line 99-111, and code line 113-125 implements the final weights delta update rule which additionally involves the multiplicative terms of learning rate and the respective weighted inputs of each layer in Numpy's matrix/array linear algebra way.

And the error signal of your 2nd article's quoted section ("the error signal for neurons in the hidden layer is accumulated from neurons in the output layer") is just the inner function part $(W^{l})^{T}\cdot \delta ^{l}$ of the composed gradient function $\delta ^{l-1}$. And at the end line 18 of the quoted code snippet you have:

neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])

Thus the same gradient of each layer $\delta ^{l-1}$ as expressed as neuron['delta'] here is obtained too after multiplied by the derivative of the activation/transfer function though here seems only primitive Python data types are employed instead of Numpy's linear algebra data types.

$\endgroup$
6
  • $\begingroup$ Thanks for explaining! With your explanation, both articles seem similar. So this means that to calculate the delta of a neuron in the hidden layers, in both ways, both the first article (which uses the current layer) and the second article (which uses the next layer), work equivalently and are based on the same principles for the same goal?, even using a slightly different way? Did I understand correctly? $\endgroup$
    – will The J
    Nov 22, 2023 at 11:05
  • 1
    $\begingroup$ Yes, your conclusion is correct, there’s only one standard back propagation algorithm though there’re many different implementations such as manifested here and in another similar question you asked recently. One idea with many different implementations is a common phenomenon in applied sciences and engineering. $\endgroup$
    – cinch
    Nov 22, 2023 at 16:09
  • $\begingroup$ Thanks!. But, in second article, the sum of DELTA OUTPUT * WEIGHT to calcule delta in hidden layer, he explains that WEIGHT is the weight J (the index of the neuron J of current hidden layer in vector of the output layer, referring to input J), that is, the connection weight of output layer, that is multiplied by input of output layer (the input of the output layer is the output of the last hidden layer). But, in first article is different, because the weights are the weights that connect the input layer input with the current hidden layer. I had this question below: $\endgroup$
    – will The J
    Nov 23, 2023 at 1:34
  • 1
    $\begingroup$ The 1st article is using matrix algebra just as my above answer for the hidden layer's all units' gradients $\delta^{l-1}$ of their respective weighted input, so if 3 units in the current hidden layer $\delta^{l-1}$ is a vector of 3 gradients. This ref is clearer, the first hidden layer has 3 units (1,2,3), each has gradient of its own weighted input, and it's not in matrix form, so like your 2nd article, you can compare and check its for-loops carefully though this is mainly a coding practice, I'd prefer 1st article's matrix code. $\endgroup$
    – cinch
    Nov 23, 2023 at 7:03
  • 1
    $\begingroup$ Thank for the explanations!. It has helped me a lot to understand the concepts. $\endgroup$
    – will The J
    Nov 23, 2023 at 22:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .