Different ways to calculate backpropagation derivatives, any difference?

Question

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question because there are several paths in the computational graph to get the derivative for a particular weight. Why do we only use a one value? Why don't we combine the values from all possible paths?

Schema: Fromulas:

Normal path: $$\frac{\partial E}{\partial w_{1,1}} = \frac{\partial E}{\partial Out} \cdot \frac{\partial Out}{\partial a_{1,1}}\cdot \frac{\partial a_{1,1}}{\partial a_{0,1}}\cdot \frac{\partial a_{0,1}}{\partial w_{1,1}}$$

Alternate path: Normal path: $$\frac{\partial E}{\partial w_{1,1}} = \frac{\partial E}{\partial Out} \cdot \frac{\partial Out}{\partial a_{1,2}}\cdot \frac{\partial a_{1,2}}{\partial a_{0,1}}\cdot \frac{\partial a_{0,1}}{\partial w_{1,1}}$$

Why don't we consider both derivatives or the sum of them?

Answer 1

The backpropagation algorithm doesn't take paths, it computes the gradient with respect to all combinations defined by each weight matrix. But you could look at it as adding up the gradient from all paths.

Usually, the notation for the weights have three indices, the index of the input cell, the index of the output cell and the index of the layer, so $W_{1, 2}^{[3]}$ can be the coefficient in the 3rd layer of the neural network that multiplies the activation of the 1st cell in that layer and it contributes to the 2nd cell in the next layer. Now, when you look at $W^{[3]}$ without the lower indices, that is a matrix of weights between layers. This is what's used when expanding the differentiation using the product rule. So the derivatives are now with respect to those matrix weights. The point is that all of these paths contribute to the gradient at lower levels.