Is back-propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images for handwritten letters,

1. Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?

2. Is back-propagation applied $$n$$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?

Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?

You can perform back-propagation using (or after) only one training input (also known as data point, example, sample or observation) or multiple ones (a batch). However, the loss function to train the neural network is slightly different in both cases.

Is back-propagation applied $$n$$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?

If we use only one example, we usually do not wait until the neural network gives satisfactory results for a single input-label pair $$(x_i, y_i)$$, but we keep feeding it with several input-label pairs, one after the other (and each time updating the parameters of the neural network using back-propagation), without usually caring whether the neural network already produces a satisfactory output for an input-label pair before feeding it with the next (although you could also do that).

Long answer (to the 1st question)

In case you want to know more about the first question, keep reading!

What is back-propagation?

Back-propagation is the (automatic) process of differentiating the loss function, which we use to train the neural network, $$\mathcal{L}$$, with respect to all of the parameters (or weights) of the same neural network. If you collect the $$N$$ parameters of the neural network in a vector

$$\boldsymbol{\theta} = \begin{bmatrix} \theta_1\\ \vdots \\ \theta_N \end{bmatrix},$$

then the derivative of the loss function $$\mathcal{L}$$ with respect to $$\boldsymbol{\theta}$$ is called the gradient, which is a vector that contains the partial derivatives of the loss function with respect to each single scalar parameter of the network, $$\theta_i$$, for $$i=1, \dots, N$$, that is, the gradient looks something like this

$$\nabla \mathcal{L} = \begin{bmatrix} \frac{\partial \mathcal{L}}{ \partial \theta_1}\\ \vdots \\ \frac{\partial \mathcal{L}}{ \partial \theta_N} \end{bmatrix},$$

where the symbol $$\nabla$$ denotes the gradient of the function $$\mathcal{L}$$.

Loss functions

The specific loss function $$\mathcal{L}$$ that is used to train the neural network depends on the problem that we need to solve. For example, if we need to solve a regression problem (i.e. predict a real number), then the mean squared error (MSE) can be used. If we need to solve a classification problem (i.e. predict a class), the cross-entropy (CE) (aka negative log-likelihood) may be used instead.

Example

Let us assume that we need to solve a regression problem, so we choose the mean squared error (MSE) as the loss function. For simplicity, let's also assume that the neural network, denoted by $$f_{\boldsymbol{\theta}}$$, contains only one output neuron and contains no biases.

Given an input-label pair $$(x_i, y_i) \in D$$ (where $$D$$ is a dataset of input-label pairs), the squared error function (not the mean squared error yet!) is defined as

$$\mathcal{L}_i(\boldsymbol{\theta}) = \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i) - y_i)^2,$$

where $$f_{\boldsymbol{\theta}}(x_i) = \hat{y}_i$$ is the output of the neural network for the the data point $$x_i$$ (which depends on the specific values of $$\boldsymbol{\theta}$$) and $$y_i$$ is the corresponding ground-truth label.

As the name suggests, $$\mathcal{L}_i$$ (where the subscript $$_i$$ is only used to refer to the specific input-label pair $$(x_i, y_i)$$) measures the squared error (i.e. some notion of distance) between the current prediction (or output) of the neural network, $$\hat{y}_i$$, and the expected output for the given input $$x_i$$, i.e. $$y_i$$.

We can differentiate $$\mathcal{L}_i(\boldsymbol{\theta})$$ with respect to the parameters of the neural network, $$\boldsymbol{\theta}$$. However, given that the details of back-propagation can easily become tedious, I will not describe them here. You can find more details here.

So, let me assume that we have a computer program that is able to compute $$\nabla \mathcal{L}_i$$. At that point, we can perform one step of the gradient descent algorithm

$$\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_i, \label{1}\tag{1}$$

where $$\gamma$$ is the learning rate and $$\leftarrow$$ is the assignment operator. Note that $$\boldsymbol{\theta}$$ and $$\nabla \mathcal{L}_i$$ have the same dimensions, $$N$$.

So, I have just shown you that you can update the parameters of the neural network using only one input-label pair, $$(x_i, y_i)$$. This way of performing GD with only one input-label pair is known as stochastic gradient descent (SGD).

In practice, for several reasons (including learning instability and inefficiency), it is rarely the case that you update the parameters using only one input-label pair $$(x_i, y_i)$$. Instead, you use multiple input-label pairs, which are collected in a so-called batch

$$B = \{(x_1, y_i), \dots, (x_M, y_M) \},$$

where $$M$$ is the size of batch $$B$$, which is also known as mini-batch when $$M$$ is smaller than the total number of input-label pairs in the training dataset, i.e. when $$|B| = M < |D|$$. If you use mini-batches, typical values for $$M$$ are $$32$$, $$64$$, $$128$$, and $$256$$ (yes, powers of 2: can you guess why?). See this question for other details.

In this case, the loss function $$\mathcal{L}_M(\boldsymbol{\theta})$$ is defined as as the mean (or average) of the squared errors for single input-label pairs, $$\mathcal{L}_i(\boldsymbol{\theta})$$, i.e.

\begin{align} \mathcal{L}_M(\boldsymbol{\theta}) &= \frac{1}{M} \sum_{i=1}^M \mathcal{L}_i(\boldsymbol{\theta}) \\ &= \frac{1}{M} \sum_{i=1}^M \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i)-y_i)^2 \\ &= \frac{1}{M} \frac{1}{2} \sum_{i=1}^M (f_{\boldsymbol{\theta}}(x_i)-y_i)^2. \end{align} The normalisation factor, $$\frac{1}{M}$$, can be thought of as averaging out the losses of all input-label pairs. Note also that we can take the $$\frac{1}{2}$$ out of the summation because it is a constant with respect to the variable of the summation, $$i$$.

In this case, let me also assume that we are able to compute (using back-propagation) the gradient of $$\mathcal{L}_M$$, so that we can perform a gradient descent (GD) update (using a batch of examples)

$$\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_M \label{2} \tag{2}$$ The only thing that changes with respect to the GD update in equation \ref{1} is the loss function, which is now $$\mathcal{L}_M$$ rather than $$\mathcal{L}_i$$.

This is known as mini-batch (or batch) gradient descent. In the case $$M = |D|$$, this is simply known as gradient descent, which is a term that can also be used to refer to any of its variants (including SGD or mini-batch GD).

• Thanks!. In the case of a batch, can I assume that the weight adjust factor for a given parameter w1 is an average weight adjust factor for all items in the batch? Apr 5, 2019 at 15:07
• @Maanu I would think of the update of a weight using a batch as an update that contains more info (than using only just one training example). This is also why training using a batches (rather than single examples) is usually more stable, that is, the NN tends to approximate the desidered function more rapidly. However, note that training using big batches can also require a lot of memory. So, there is a trade-off between memory requirements and stability of the training process.
– nbro
Apr 5, 2019 at 15:16
• I have joined this community now, just because I wanted to thank you for this clear answer, after weeks of searching Jul 22 at 15:44