1
$\begingroup$

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question because there are several paths in the computational graph to get the derivative for a particular weight. Why do we only use a one value? Why don't we combine the values from all possible paths?

Schema: Backpropagation schema Fromulas:

Normal path: $$\frac{\partial E}{\partial w_{1,1}} = \frac{\partial E}{\partial Out} \cdot \frac{\partial Out}{\partial a_{1,1}}\cdot \frac{\partial a_{1,1}}{\partial a_{0,1}}\cdot \frac{\partial a_{0,1}}{\partial w_{1,1}}$$

Alternate path: Normal path: $$\frac{\partial E}{\partial w_{1,1}} = \frac{\partial E}{\partial Out} \cdot \frac{\partial Out}{\partial a_{1,2}}\cdot \frac{\partial a_{1,2}}{\partial a_{0,1}}\cdot \frac{\partial a_{0,1}}{\partial w_{1,1}}$$

Why don't we consider both derivatives or the sum of them?

$\endgroup$
1
  • $\begingroup$ All of them are calculated and added together. $\endgroup$
    – user253751
    Sep 6 at 10:50
1
$\begingroup$

The backpropagation algorithm doesn't take paths, it computes the gradient with respect to all combinations defined by each weight matrix. But you could look at it as adding up the gradient from all paths.

Usually, the notation for the weights have three indices, the index of the input cell, the index of the output cell and the index of the layer, so $W_{1, 2}^{[3]}$ can be the coefficient in the 3rd layer of the neural network that multiplies the activation of the 1st cell in that layer and it contributes to the 2nd cell in the next layer. Now, when you look at $W^{[3]}$ without the lower indices, that is a matrix of weights between layers. This is what's used when expanding the differentiation using the product rule. So the derivatives are now with respect to those matrix weights. The point is that all of these paths contribute to the gradient at lower levels.

$\endgroup$
1
  • $\begingroup$ Thank you! I will do the calculations because I want to see exactly where it happens and after the calculations I will add them to the post and accept the answer. $\endgroup$
    – Mike Like
    Sep 6 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.