About the requirement to compute the gradient at layer $l$

Question

I'm trying to understand a line of my note.

Let's say there is a simple feedforward neural network that has $N$ layers, and for a given layer $l$, it has weight $W^l$, and $g^l$ is the gradient to update it. Now the problem is:

1. From the Wiki page of Backpropagation, to compute $\nabla_{W^l}C$(or simply $g^l$), i.e. the gradient of the cost function with respect to the weight of layer $l$, you will need two things:

   1. A sub-expression $\delta^l$, which is the gradient of the weighted output of the current layer $l$, i.e. $\nabla_{z^l}C$. (there is no such symbol in the link, but I believe my usage is correct.)
   2. The activation of the previous layer $l-1$, i.e. $a^{l-1}$.

This is why it says $\nabla_{W^l}C=\delta^l(a^{l-1})^T$. My reasoning of this is that: the weighted output of layer $l$, i.e. $z^l$, is the result of multiplication between the output of the previous layer, i.e. $a^{l-1}$, and the weight of the current layer, i.e. $W^l$.

2. But now I found my note saying something different: it says that to compute the gradient $\nabla_{W^l}C$(or simply $g^l$), it would require $W^{l+1},g^{l+1},a^l$, that is: the weight and gradient of the next layer and the output of the current layer.

Is my note wrong?

Answer 1

1. A sub-expression $\delta^l$, which is the gradient of the weighted output of the current layer $l$, i.e. $\nabla_{z^l}C$. (there is no such symbol in the link, but I believe my usage is correct.)

First of all, you're correct that the symbol $\delta^l$ represents the gradient of the cost w.r.t. the weighted output (so the activation function has not been applied) of the layer $l$. This reasoning for this is simply chain-rule.

But you have to be aware of that the auxiliary function $\delta^l$ involves the value of $(f^l)'$ at $z^l$. This means that you will need to save $z^l$ for computing $\delta^l$.

But now I found my note saying something different: it says that to compute the gradient $\nabla_{W^l}C$(or simply $g^l$), it would require $W^{l+1},g^{l+1},a^l$, that is: the weight and gradient of the next layer and the output of the current layer.

By $\nabla_{W^l}C=\delta^l(a^{l-1})^T$, it's clear that you will need both $\delta^l$ and $a^{l-1}$ to update the weight $W^l$. For $\delta^l$, you will need:

1. the $\nabla_{a^l}{C}=(W^{l+1})^T\delta^{l+1}$, which is the gradient of the cost w.r.t. the activated weighted output of the current layer $l$.
2. the weighted output $z^l$ to compute $(f^l)'$ at $z^l$, which is mentioned in my first reply.

To compute the gradient $\nabla_{W^l}C$, the value of these symbols are required:

1. $W^{l+1}$.
2. $\nabla_{a^l}{C}$ instead of $g^{l+1}$, which is the gradient of the weight as you have said.
   • this involves $z^l$, as stated above.
3. $a^{l-1}$ instead of $a^{l}$, since the latter is for $\nabla_{W^{l+1}}C$.

So yes, your note is incorrect.

---

reference: https://towardsdatascience.com/back-propagation-simplified-218430e21ad0