Spikes in of Train and Test error

Question

I learn a DNN for image recognition. During each epoch, I calculate mean loss in the training set. After each epoch, I calculate loss and number of errors over both training and test set. The problem is, training and test error go to (almost) zero, then increase, go to zero again, increase, and so on. The process seems stochastic.

epoch: 1 mean_loss=0.109 train: errs=7 loss=0.00622 test: errs=3 loss=0.00608
epoch: 2 mean_loss=0.00524 train: errs=5 loss=0.00309 test: errs=3 loss=0.00369
epoch: 3 mean_loss=0.00408 train: errs=13 loss=0.00614 test: errs=7 loss=0.00951
epoch: 4 mean_loss=0.00198 train: errs=113 loss=0.102 test: errs=51 loss=0.265
epoch: 5 mean_loss=0.00424 train: errs=3 loss=0.00201 test: errs=2 loss=0.00148
epoch: 6 mean_loss=0.0027 train: errs=1 loss=0.000466 test: errs=2 loss=0.00193
epoch: 7 mean_loss=0.00797 train: errs=5 loss=0.00381 test: errs=0 loss=0.000493
epoch: 8 mean_loss=0.00368 train: errs=1 loss=0.000345 test: errs=2 loss=0.00148
epoch: 9 mean_loss=0.000358 train: errs=0 loss=6.76e-05 test: errs=0 loss=0.000446
epoch: 10 mean_loss=0.00101 train: errs=164 loss=0.0863 test: errs=67 loss=0.19
epoch: 11 mean_loss=0.000665 train: errs=0 loss=2.38e-05 test: errs=0 loss=9.86e-05
epoch: 12 mean_loss=0.00714 train: errs=5 loss=0.00909 test: errs=0 loss=0.00816
epoch: 13 mean_loss=0.00266 train: errs=73 loss=0.0333 test: errs=10 loss=0.0192
epoch: 14 mean_loss=0.00213 train: errs=0 loss=7.74e-05 test: errs=0 loss=0.000197
epoch: 15 mean_loss=6.12e-05 train: errs=0 loss=7.66e-05 test: errs=0 loss=3.44e-05
epoch: 16 mean_loss=0.00162 train: errs=5 loss=0.00265 test: errs=0 loss=0.0012
epoch: 17 mean_loss=0.000159 train: errs=0 loss=3.11e-05 test: errs=0 loss=4.26e-05
epoch: 18 mean_loss=4.68e-05 train: errs=0 loss=3.28e-05 test: errs=0 loss=6.05e-05
epoch: 19 mean_loss=2.47e-05 train: errs=0 loss=2.8e-05 test: errs=0 loss=5.01e-05
epoch: 20 mean_loss=2.2e-05 train: errs=0 loss=2.31e-05 test: errs=0 loss=3.95e-05
epoch: 21 mean_loss=2.37e-05 train: errs=0 loss=1.76e-05 test: errs=0 loss=2.52e-05
epoch: 22 mean_loss=1.4e-05 train: errs=0 loss=1.16e-05 test: errs=0 loss=1.52e-05
epoch: 23 mean_loss=2.13e-05 train: errs=0 loss=1.65e-05 test: errs=0 loss=2.13e-05
epoch: 24 mean_loss=1.53e-05 train: errs=0 loss=1.91e-05 test: errs=0 loss=2.46e-05
epoch: 25 mean_loss=0.00419 train: errs=0 loss=5.27e-05 test: errs=0 loss=4.65e-05
epoch: 26 mean_loss=0.000372 train: errs=6 loss=0.00297 test: errs=3 loss=0.00731
epoch: 27 mean_loss=0.0016 train: errs=0 loss=4.23e-05 test: errs=0 loss=3.69e-05
epoch: 28 mean_loss=3.34e-05 train: errs=0 loss=2.44e-05 test: errs=0 loss=2.76e-05
epoch: 29 mean_loss=7.03e-05 train: errs=0 loss=2.16e-05 test: errs=0 loss=1.69e-05
epoch: 30 mean_loss=2.41e-05 train: errs=0 loss=1.84e-05 test: errs=0 loss=1.77e-05
epoch: 31 mean_loss=1.26e-05 train: errs=0 loss=2.11e-05 test: errs=0 loss=1.78e-05
epoch: 32 mean_loss=1.39e-05 train: errs=0 loss=2.75e-05 test: errs=0 loss=2.42e-05
epoch: 33 mean_loss=7.68e-05 train: errs=0 loss=0.00014 test: errs=0 loss=4.66e-05
epoch: 34 mean_loss=2.53e-05 train: errs=0 loss=1.48e-05 test: errs=0 loss=1.56e-05
epoch: 35 mean_loss=0.000352 train: errs=1786 loss=2.17 test: errs=493 loss=2.56
epoch: 36 mean_loss=0.0088 train: errs=0 loss=0.000347 test: errs=0 loss=0.000449
epoch: 37 mean_loss=0.000395 train: errs=0 loss=6.18e-05 test: errs=0 loss=0.000125
epoch: 38 mean_loss=5e-05 train: errs=0 loss=6.73e-05 test: errs=0 loss=9.89e-05
epoch: 39 mean_loss=0.00401 train: errs=26 loss=0.00836 test: errs=27 loss=0.0269
epoch: 40 mean_loss=0.00051 train: errs=0 loss=7.66e-05 test: errs=0 loss=7.07e-05
epoch: 41 mean_loss=5.49e-05 train: errs=0 loss=2.47e-05 test: errs=0 loss=2.58e-05
epoch: 42 mean_loss=3.38e-05 train: errs=0 loss=1.67e-05 test: errs=0 loss=2.1e-05
epoch: 43 mean_loss=2.45e-05 train: errs=0 loss=1.28e-05 test: errs=0 loss=2.95e-05
epoch: 44 mean_loss=0.00137 train: errs=44 loss=0.0141 test: errs=16 loss=0.0207
epoch: 45 mean_loss=0.000785 train: errs=1 loss=0.000493 test: errs=0 loss=4.46e-05
epoch: 46 mean_loss=5.46e-05 train: errs=1 loss=0.000487 test: errs=0 loss=1.34e-05
epoch: 47 mean_loss=1.99e-05 train: errs=1 loss=0.00033 test: errs=0 loss=1.57e-05
epoch: 48 mean_loss=1.78e-05 train: errs=1 loss=0.000307 test: errs=0 loss=1.58e-05
epoch: 49 mean_loss=0.000903 train: errs=1 loss=0.00103 test: errs=0 loss=0.000393
epoch: 50 mean_loss=4.74e-05 train: errs=0 loss=4.63e-05 test: errs=0 loss=3.53e-05
Finished Training, time: 234.69774420000002 sec

The images are 96*96 gray. There are about 7000 training and 1750 test images. The order of presentation is random, and different at each epoch. Each image is either contains the object or not. The architecture is (Conv2d->ReLU->BatchNorm2d->MaxPool)*4->AvgPool(6,6)->Flatten->Conv->Conv->Conv. All MaxPool's are 2*2. First two Conv2d layers are 5*5, padding=2, others 3*3, padding=1. The optimiser is like this:

Optimizer= Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 0.001
    weight_decay: 1e-05
)

Currently I just choose the epoch when the training set error was minimal.

if epoch == 0 or train_loss < train_loss_best:
    net_best = copy.deepcopy(net)
    train_loss_best = train_loss

It works, but I don't like it. Is there a way to make the learning more stable and steady?

Answer 1

I was actually very recently working on CNNs and extensively training models and have noticed the same thing. (NOTE: The answer I will give is purely based on empirical observations and my understanding of mathematics of deep learning).

So, the thing I observed on training set (since we are directly optimising on the training set) is that between period of low losses there was suddenly a large loss, and then again low losses, but, this time the loss (of training set) reached even more lower values and the accuracy of the test set reached a higher stable accuracy (by higher stable accuracy I mean, that in general I was observing the accuracy was somewhat oscillating around a fixed value of accuracy, and now the accuracy is oscillating around a higher fixed value of accuracy). From this I concluded that, the high loss was some sort of an obstacle which is stopping the weights from reaching global minima and is trapping it in local minima, unless it gains enough momentum to escape the obstacle, which is indicated by the high loss (think of the loss function as a rotated around y-axis sawtooth waveform inclined at certain degree). So, as we reach lower loss it is expected (if the model is good) that you see better generalisation and hence the stable higher accuracy.

You can actually somewhat see this happening in your training too, although the data-set is smaller to make concrete comments.

Seeing your results it seems pretty clear that training loss and test loss are going hand in hand as well as accuracy, which is a good sign, and it means your model has not over-fitted yet and you can train it more, which will probably make you encounter these loss spikes less, nevertheless I am pretty sure they will be there every now and then, whichever method you choose, as the loss curve is always a pretty jagged terrain for a DNN.

I chose to disregard training accuracy, because most of the times it is not really related to the loss in a monotonically increasing way (check this thread), and the test loss is expected to increase since at such high accuracies, it has been empirically seen that test loss might decrease without affecting test accuracy (check this answer).