Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?
I read this in a paper titled Mean Field Residual Networks: On the Edge of Chaos (2017). I am not sure how far this is true, because the error depends on those activations.
Yes, this is the premise of back-propagation, the gradient at layer $j_{n}$ is not impacted by the gradient at layer $j_{n-1}$. This allows you to start with a gradient at the output layer and propagated it back through the network to the input layer.
It is however impacted by the gradient at $j_{n+1}$, which the back-prop algorithm also follows.
Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?
Yes, as per @recessive answer they are indeed independent of the previous layers.
The goal of back-propagation is to trace the loss(error between target and network output) to specific weights in the network, and then tweak them to minimize this loss. For this to be possible, the activation must be independent of previous activation functions(going forward in the network).
It is very helpful to have a good understanding of the back-propagation process when reading these papers, and I personally suggest watching this video to get a good understanding of it:
