1
$\begingroup$

In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as:

The back-propagation algorithm is very simple. To compute the gradient of some scalar z with respect to one of its ancestors x in the graph, we begin by observing that the gradient with respect to z is given by dz = 1. We can then compute dz the gradient with respect to each parent of z in the graph by multiplying the current gradient by the Jacobian of the operation that produced z. We continue multiplying by Jacobians traveling backwards through the graph in this way until we reach x. For any node that may be reached by going backwards from z through two or more paths, we simply sum the gradients arriving from different paths at that node.

To be specific I don't get this part:

We can then compute dz the gradient with respect to each parent of z in the graph by multiplying the current gradient by the Jacobian of the operation that produced z.

Can anyone help me understand this with some illustration? Thank you.

$\endgroup$
  • 2
    $\begingroup$ Hey. 3 blue 1 brown made amazing series on neural networks, youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi, the 3rd and 4th part have good illustrations of back propagation algorithm $\endgroup$ – mark mark Jun 29 at 5:05
  • 1
    $\begingroup$ Please, ask a more specific question. The current question is too broad. $\endgroup$ – nbro Jun 29 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.