0
$\begingroup$

References:

Searching the internet in some articles, I learned that the chain rule is used so that the network can identify the contribution of each weight to the error, and how much each weight needs to be adjusted.

This article I mentioned in the question: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, they teaches you how to program a multilayer perceptron neural network from scratch in Python. During the calculations in the backpropagation phase, he uses some different terms such "delta", and it calculates the "delta" layer by layer. I know that the explanation in the article must make use of the chain rule, but I couldn't understand where.

I couldn't see where the chain rule is. Does this article make use of the chain rule? And where?

$\endgroup$
4
  • 1
    $\begingroup$ The chain rule is spread in the 3 core functions of your reference: forward_propagate(network, row), backward_propagate_error(network, expected), update_weights(network, row, l_rate) which are linked in the train_network(network, train, l_rate, n_epoch, n_outputs) function. This is mostly a programming question. $\endgroup$
    – cinch
    Nov 9, 2023 at 0:38
  • 1
    $\begingroup$ Can you please don't use links in the title and don't use some much bold? $\endgroup$
    – nbro
    Nov 9, 2023 at 9:01
  • $\begingroup$ @nbro Okay. Today I will edit the question and remove the title link and remove the bold text. $\endgroup$
    – will The J
    Nov 9, 2023 at 10:38
  • $\begingroup$ @mohottnad Thanks for explain about it. $\endgroup$
    – will The J
    Nov 9, 2023 at 10:39

2 Answers 2

2
$\begingroup$

Unless the derivation has been added in a comment, when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules. As it is basically a rule about multiplying one simpler derivative by another to resolve the derivative of a more complex function, then typically some of the terms in each statement represent $\frac{\partial f}{\partial x}$ and some represent $\frac{\partial g}{\partial f}$. When those are multiplied together you find the numerical value of $\frac{\partial g}{\partial x}$ from the composite $g(f(x))$.

The "deltas" being stored in variables are components of the gradient of the loss function with respect to the network's internal states and free parameters, at its current value (due to current item, or an aggregate across the current batch).

  • Some are temporary calculations that don't need to be stored longer term because they are not gradient components of changeable parameters.

  • Some represent components of $\nabla_{W,b} J$, the gradient of the cost function $J$ with respect to the weights and biases of the neural network - these are the values that you can change as part of training, so they are usually stored in a structure the same shape as the weights and biases so that they can be aggregated across multiple examples and subtracted from $W,b$ later.

$\endgroup$
4
  • $\begingroup$ Thanks. Please, I have a question: the article uses the Sigmoid, and it shows its derivative. So according to your explanation in "[...] when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules.", this means that if I have an activation function complex, and I don't know the derivative of it, so that's where I would apply the derivation to find the derivative? But in article this derivation has already been done (the derivative of the Sigmoid), I don't need to do the derivation because it was done. Did I understand correctly? $\endgroup$
    – will The J
    Nov 9, 2023 at 22:18
  • $\begingroup$ in other words, do I just use the chain rule to find the derivative of the activation function when I don't know what the derivative ? $\endgroup$
    – will The J
    Nov 9, 2023 at 22:27
  • $\begingroup$ @willTheJ If you are being shown working code for training, you should not need to do any extra maths on the code or data in order to use it. Quoting the chain rule is explaining how it works, and how you might build your own NN from scratch. If you are building your own NN from scratch, using e.g. NumPy, and stuck, then please as a new question about that. $\endgroup$ Nov 10, 2023 at 14:14
  • $\begingroup$ Okay, I will create new questions for other questions.. Thanks for explaining $\endgroup$
    – will The J
    Nov 10, 2023 at 17:38
1
$\begingroup$

Your last article is mainly focused on programming the classic full GD BP algo from scratch using pure Python under the default MSE cost function but never mentioned or emphasized in the main body except somewhere in the extremely long comment section, thus it may cause you cannot link with chain rule.

The first outer chain under MSE is obviously just the error signal from the final output compared with its true target, the 2nd chain is the derivative of the activation function which is the classic sigmoid function which you're already aware, and the final inner chain is the output of the originating neuron of the updated weight edge in the previous layer or just one of the input signals if there's none. These correspond to the three usual multiplicative terms in a delta rule.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks a lot for explaining better the chain rule! $\endgroup$
    – will The J
    Nov 10, 2023 at 0:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .