Does this article make use of the chain rule? And where?

Question

References:

• Chain rule in Wikipedia: https://en.wikipedia.org/wiki/Chain_rule
• Chain rule in Towards Data Science: https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~:text=The%20algorithm%20is%20used%20to,parameters%20(weights%20and%20biases).

Searching the internet in some articles, I learned that the chain rule is used so that the network can identify the contribution of each weight to the error, and how much each weight needs to be adjusted.

This article I mentioned in the question: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, they teaches you how to program a multilayer perceptron neural network from scratch in Python. During the calculations in the backpropagation phase, he uses some different terms such "delta", and it calculates the "delta" layer by layer. I know that the explanation in the article must make use of the chain rule, but I couldn't understand where.

I couldn't see where the chain rule is. Does this article make use of the chain rule? And where?

Answer 1

Unless the derivation has been added in a comment, when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules. As it is basically a rule about multiplying one simpler derivative by another to resolve the derivative of a more complex function, then typically some of the terms in each statement represent $\frac{\partial f}{\partial x}$ and some represent $\frac{\partial g}{\partial f}$. When those are multiplied together you find the numerical value of $\frac{\partial g}{\partial x}$ from the composite $g(f(x))$.

The "deltas" being stored in variables are components of the gradient of the loss function with respect to the network's internal states and free parameters, at its current value (due to current item, or an aggregate across the current batch).

• Some are temporary calculations that don't need to be stored longer term because they are not gradient components of changeable parameters.

• Some represent components of $\nabla_{W,b} J$, the gradient of the cost function $J$ with respect to the weights and biases of the neural network - these are the values that you can change as part of training, so they are usually stored in a structure the same shape as the weights and biases so that they can be aggregated across multiple examples and subtracted from $W,b$ later.

Answer 2

Your last article is mainly focused on programming the classic full GD BP algo from scratch using pure Python under the default MSE cost function but never mentioned or emphasized in the main body except somewhere in the extremely long comment section, thus it may cause you cannot link with chain rule.

The first outer chain under MSE is obviously just the error signal from the final output compared with its true target, the 2nd chain is the derivative of the activation function which is the classic sigmoid function which you're already aware, and the final inner chain is the output of the originating neuron of the updated weight edge in the previous layer or just one of the input signals if there's none. These correspond to the three usual multiplicative terms in a delta rule.