Why do we update all layers simultaneously while training a neural network?

Question

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we update all of the layers simultaneously.

The above is an extract from Ian Goodfellow's Deep Learning - which talks about the need for batch normalization.

Why do we update all the layers simultaneously? Instead, if we update layers one at a time during backpropagation - it will eliminate the need for batch normalization, right?

Reference: A Gentle Introduction to Batch Normalization for Deep Neural Networks

P.S. The attached link says: Because all layers are changed during an update, the update procedure is forever chasing a moving target. Apart from the main question, it would be great if someone could explain why exactly a moving target is being referred to in the above sentence.

Answer 1

Why do we update all layers simultaneously while training a neural network?

We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm used to update the parameters and back-propagation is the algorithm used to compute the gradient of the loss function with respect to each of these parameters.

Let's denote a vector that contains all learnable parameters of a neural network $M$ by $\mathbf{w} = \left[w_1, \dots, w_n \right] \in \mathbb{R}^n$ (so $M$ contains $n$ learnable parameters), the loss function of $M$ by $\mathcal{L}$, the gradient of the loss function with respect to each parameter $w_i$ of $M$ by $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n} \right] \in \mathbb{R}^n$, then the gradient descent step to update all parameters is

\begin{align} \mathbf{w} \leftarrow \mathbf{w} - \gamma * \nabla \mathcal{L} \tag{1} \label{1} \end{align}

where $\gamma \in \mathbb{R}$ is the learning rate.

In equation \ref{1}, we are assigning to $\mathbf{w}$ the value $\mathbf{w} - \gamma * \nabla \mathcal{L}$, so we are updating all parameters $\mathbf{w}$ simultaneously, so we are also updating all layers simultaneously.

In principle, you could update each parameter $w_i$ individually. To be more precise, you would have the following update rule

\begin{align} w_i \leftarrow w_i - \gamma * \frac{\partial \mathcal{L}}{\partial w_i}, \; \forall i\tag{2} \label{2} \end{align}

So, you could update first $w_1$, then $w_2$, and so on.

Actually, you don't need to update the parameters sequentially (also because there is no real order of the parameters). You can actually update them in any other way. You can update the parameters in any way because, although the computation of the gradient highly depends on the structure of the neural network (so if you change the structure, the computation of the gradient also changes), once the gradient is computed, you already have all information to update each parameter independently of each other.

You typically update all parameters (or layers) simultaneously because, in practice, you work with vectors and matrices rather than scalars in order to take benefit from efficient matrix multiplication algorithms and hardware (i.e. GPUs).