I am new to deep learning and trying to understand the concept of back propagation. I have a doubt on when the back propagation is applied. Assume that I have a training data set of 1000 images for handwritten letters,

  1. Does back propagation applied immediately after getting the result for each input data or after getting the result data for all items in a batch?

  2. Does back propagation applied n times till the neural network gives satisfactory result for an input data before going to work on the next input data?


In short, you can perform back-propagation using (or after) only one training example or multiple ones (a batch). However, the loss function of the neural network is slightly different in both cases.

In case we use only one example, we usually do not wait until the NN gives satisfactory results for a single input data point $x$, but we keep feeding it with several input examples one after the other (and each time updating the parameters of the network with respect to that input example, using back-propagation), "without caring if it already performs well on any of these single examples".

In case you want to know more about the first question, keep reading!

Back-propagation is the process of differentiating the loss function of the neural network, $\mathcal{L}$, with respect to all of the parameters (or weights) of the same neural network. If you collect the $N$ parameters of the neural network in a vector

$$\boldsymbol{\theta} = \begin{bmatrix} \theta_1\\ \vdots \\ \theta_N \end{bmatrix} $$

then the derivative of the loss function $\mathcal{L}$ with respect to $\boldsymbol{\theta}$ is called the gradient, which is a vector that contains the partial derivatives of the loss function with respect to each single scalar parameter of the network, $\theta_i$, for $i=1, \dots, N$, that is, the gradient looks something like this

$$ \nabla \mathcal{L} = \begin{bmatrix} \frac{\partial \mathcal{L}}{ \partial \theta_1}\\ \vdots \\ \frac{\partial \mathcal{L}}{ \partial \theta_N} \end{bmatrix} $$

where the symbol $\nabla$ denotes the gradient of the function $\mathcal{L}$.

What is the loss function $\mathcal{L}$? There are several loss functions that are used in this context (e.g. the mean squared error or the cross entropy). A few of them can only be used in certain contexts or are more appropriate than others for certain problems.

For simplicity, let's take the mean squared error (MSE) as the loss function. Furthermore, let's assume that the neural network only contains one output neuron. For simplicity, also ignore bias (assume that we have only weights). In that case, the MSE for that single output unit looks as follows:

$$\mathcal{L}(\boldsymbol{\theta}) = \frac{1}{2} (y(x)-a)^2$$

where $y(x)$ is the output of the network for the (single) input example $x$ and $a$ is the corresponding ground-truth label. $\mathcal{L}$ measures the distance between the current prediction (or output) of the neural network, $y(x)$, and the "expected" output for the given input $x$, that is, $a$. In this case, the loss function only takes into account a single input, $x$.

We can differentiate this function with respect to the parameters of the neural network, $\boldsymbol{\theta}$. However, where are the parameters $\boldsymbol{\theta}$ in the formula $\frac{1}{2} (y(x)-a)^2$? They are "hidden" in the term $y(x)$. For simplicity, assume that we are able to calculate the gradient of $\mathcal{L}$, that is, $\nabla \mathcal{L}$. At this point, we can perform one step of the gradient descent algorithm

$$ \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L} $$

where $\gamma$ is the learning rate and $\leftarrow$ means "set". Note that $\boldsymbol{\theta}$ and $\nabla \mathcal{L}$ have the same dimensions ($N$). I have just shown you that you can thus update the parameters of NN using only one input example, $x$. So, in theory, for each input example $x$, you can perform a back-propagation step to compute $\nabla \mathcal{L}$ and then perform a gradient descent update (like above).

In practice, it is rarely the case that you compute the gradient using only one single example $x$. It is often the case that you compute the gradient of $\mathcal{L}$ using multiple examples $x_1, \dots, x_M$, where $M$ can either be the size of "batch" or it can be the size of the full training dataset (in case the the "batch" is the full training dataset). The batch is not usually the full training dataset, but just a small subset of it (e.g. $128$ examples).

In the case where you use more than one input example to compute the gradient, you define the loss function $L_M(\boldsymbol{\theta})$ as the sum of the loss functions $L_i(\boldsymbol{\theta})$, so something like this

\begin{align} L_M(\boldsymbol{\theta}) &= \frac{1}{M} \sum_{i=1}^M L_i(\boldsymbol{\theta}) \\ &= \frac{1}{M} \sum_{i=1}^M \frac{1}{2} (y(x)-a)^2 \\ &= \frac{1}{M} \frac{1}{2} \sum_{i=1}^M (y(x)-a)^2 \end{align}

where $L_i$ looks like the loss function $L$ defined above, but, here, we use the subscript $i$ to refer to the $i$th training example $x_i$. Note that a normalisation factor, $\frac{1}{M}$, which you can think of it as "averaging out" the losses of each single example $x_i$. Note also that we can take out the $\frac{1}{2}$ from the summation, because it is a constant with respect to the variable of the summation, $i$.

In this case, we assume that we are also able to compute (using back-propagation) the gradient of $L_M$, so that we can perform a gradient descent (GD) update (using a batch of examples)

$$ \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_M $$

The only thing that changes with respect to the GD update using only one single example is the loss function. In this case, using a batch, we use the loss function $\mathcal{L}_M$. In the case we use just one example $x$, the loss function is $\mathcal{L}$.

In the case the loss function is $\mathcal{L}_M$ (multiple training examples), the GD update (or algorithm) is called mini-batch (or batch) GD (or, simply, GD). In the case the loss function is $\mathcal{L}$ (one training example), the SG update is called stochastic GD.

In the reasonings above, for simplicity, I assumed that you know how to compute the gradient of the loss function with respect to the parameters. If you want to know the details of the computation of the gradient when you have one or more training inputs, read this answer.

| improve this answer | |
  • $\begingroup$ Thanks!. In the case of a batch, can I assume that the weight adjust factor for a given parameter w1 is an average weight adjust factor for all items in the batch? $\endgroup$ – Maanu Apr 5 '19 at 15:07
  • $\begingroup$ @Maanu I would think of the update of a weight using a batch as an update that contains more info (than using only just one training example). This is also why training using a batches (rather than single examples) is usually more stable, that is, the NN tends to approximate the desidered function more rapidly. However, note that training using big batches can also require a lot of memory. So, there is a trade-off between memory requirements and stability of the training process. $\endgroup$ – nbro Apr 5 '19 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.