4
$\begingroup$

Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?

I read this in a paper titled Mean Field Residual Networks: On the Edge of Chaos (2017). I am not sure how far this is true, because the error depends on those activations.

$\endgroup$
1
$\begingroup$

Yes, this is the premise of back-propagation, the gradient at layer $j_{n}$ is not impacted by the gradient at layer $j_{n-1}$. This allows you to start with a gradient at the output layer and propagated it back through the network to the input layer.

It is however impacted by the gradient at $j_{n+1}$, which the back-prop algorithm also follows.

$\endgroup$
1
$\begingroup$

Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?

Yes, as per @recessive answer they are indeed independent of the previous layers.

The goal of back-propagation is to trace the loss(error between target and network output) to specific weights in the network, and then tweak them to minimize this loss. For this to be possible, the activation must be independent of previous activation functions(going forward in the network).

It is very helpful to have a good understanding of the back-propagation process when reading these papers, and I personally suggest watching this video to get a good understanding of it:

https://www.youtube.com/watch?v=Ilg3gGewQ5U

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.