Should we also shuffle the test dataset when training with SGD?

Question

When training machine learning models (e.g. neural networks) with stochastic gradient descent, it is common practice to (uniformly) shuffle the training data into batches/sets of different samples from different classes. Should we also shuffle the test dataset?

Answer 1

Short answer

Shuffling affects learning (i.e. the updates of the parameters of the model), but, during testing or validation, you are not learning. So, it should not make any difference whether you shuffle or not the test or validation data (unless you are computing some metric that depends on the order of the samples), given that you will not be computing any gradient, but just the loss or some metric/measure like the accuracy, which is not sensitive to the order or the samples you use to compute it. However, the specific samples that you use affects the computation of the loss and these quality metrics. So, how you split your original data into training, validation and test datasets affects the computation of the loss and metrics during validation and testing.

Long answer

Let me describe how gradient descent (GD) and stochastic gradient descent (SGD) are used to train machine learning models and, in particular, neural networks.

Gradient descent (GD)

When training ML models with GD, you have a loss (aka cost) function $L(\theta; D)$ (e.g. the cross-entropy or mean squared error) that you are trying to minimize, where $\theta \in \mathbb{R}^m$ is a vector of parameters of your model and $D$ is your labeled training dataset.

To minimize this function using GD, you compute the gradient of your loss function $L(\theta; D)$ with respect to the parameters of your model $\theta$ given the training samples. Let's denote this gradient by $\nabla_\theta L(\theta; D) \in \mathbb{R}^m$. Then we perform a step of gradient descent

$$ \theta \leftarrow \theta - \alpha \nabla_\theta L(\theta; D) \label{1}\tag{1} $$

Stochastic gradient descent (SGD)

You can also minimize $L$ using stochastic gradient descent, i.e. you compute an approximate (or stochastic) version of $ \nabla_\theta L(\theta; D)$, which we can denote as $\tilde{\nabla}_\theta L(\theta; B) \approx \nabla_\theta L(\theta; D)$, which is typically computed with a subset of $B$ of your training dataset $D$, i.e. $B \subset D$ and $|B| < |D|$. The step of SGD is exactly the same as the step of GD, but we use $\tilde{\nabla}_\theta L(\theta; B)$

$$ \theta \leftarrow \theta - \alpha \tilde{\nabla}_\theta L(\theta; B) \label{2}\tag{2} $$ If we split $D$ into $k$ subsets (or batches) $B_i$, for $i=1, \dots, k$ (and these subsets usually have the same size, i.e. $|B_i| = |B_j|, \forall i$, apart from one of them, which may contain fewer samples), then the SGD step needs to be performed $k$ times, in order to go through all training samples.

Sampling, shuffling, and convergence

Given that $\tilde{\nabla}_\theta L(\theta; B_i) \approx \nabla_\theta L(\theta; D), \forall i$, it should be clear that the way you split the samples into batches can affect learning (i.e. the updates of the parameters).

For instance, you could consider your dataset $D$ as an ordered sequence/list, and just split it into $k$ sub-sequences. Without shuffling this ordered sequence before splitting, you will always get the same batches, which means that, if there's some information associated with the specific ordering of this sequence, then it may bias the learning process. That's one of the reasons why you may want to shuffle the data.

So, you could uniformly choose samples from $D$ to create your batches $B_i$ (and this is a way of shuffling, in the sense that you will be uniformly building these batches at random), but you can also sample differently and you could also re-use the same samples in different batches (i.e. sampling with replacement). Of course, all these approaches can affect how learning proceeds.

Typically, when analyzing the convergence properties of SGD, you require that your samples are i.i.d. and that the learning rate $\alpha$ satisfies some conditions (the Robbins–Monro conditions). If that's not the case, then SGD may not converge to the correct answer. That's why sampling or shuffling can play an important role in SGD.

Testing and validation

During testing or validation, you are just computing the loss or some metric (like the accuracy) and not a stochastic gradient (i.e. you are not updating the parameters, by definition: you just do it during training). The way you compute the loss or accuracy should not be sensitive to the order of the samples, so shuffling should not affect the computation of the loss or accuracy. For instance, if you use the mean squared error, then you will need to compute

\begin{align} L(\theta; D_\text{test}) &= \operatorname {MSE} \\ &= {\frac {1}{n}}\sum _{i=1}^{n}(f_\theta(x_i)-{\hat {y_{i}}})^{2}\\ &= {\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-{\hat {y_{i}}})^{2} \end{align}

where

• $f_\theta$ is your ML model
• $x_i$ is the $i$th input
• $y_{i}$ is the true label for input $x_i$
• $\hat {y_{i}}$ is the output of the model
• $n$ is the number of samples you use to compute the MSE

This is an average, so it doesn't really matter whether you shuffle or not. Of course, it matters which samples you use though!

Further reading

Here you can find some informal answers to the question "Why do we shuffle the training data while training a neural network?". There are other papers that partially answer this and/or other related questions more formally, such as this or this.