Why is my test error lower than the training error?

Question

I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay.

My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy is better than test accuracy.

So I wonder how this is happening.

Answer 1

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher.

That's normal behavior if you work with dropout.

Answer 2

I will just add to all the good answers already here.

Like I said on my comment earlier, this is not a bad this(provided you have a split your data correctly).

Other reasons could be:

• High dropout rate or excessive data augmentation could be one of the reason. This can cause the training accuracy to appear low whist in reality the model is in fact learning. How so, you ask? Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. Recall that every score you are seeing on the training set is being calculated on a "different" set of weights, ie a different network. So the accuracy you are seeing is the accuracy of all those networks your network tried before it settled on the one you are now testing on the test set. The reasoning here being that the other networks were just very "bad" compared to the final one that you are now using to test on the test set. If this is your case then, you are just witnessing neural network dropout and data augmentation doing exactly what they are meant to do, i.e avoid dropout.
• Another possible reason for this could be that your test set is different and simpler in comparison to the training set. That is to say: your data split could be such that you have a simpler test set than a train set and hence your network seems to be doing far much better on the test set than it did on the training set. If this is the case then good job, you have build yourself a very good model, no need to stress, but be careful because with such "bad" data splitting, future you might not be as lucky :).
• Make sure you are doing the correct pre-processing.
• Your test set might be very small as such, the high accuracy you are seeing on it is not indicative of the real accuracy you would see if you had a larger test set. If this is the case then, you may want to collect more test data and test your model again.
• Another reason could be than your training set is small and the high training error you are seeing there is not "real". This bring up the question of how big your training set it?
• To quote the Keras documentation:

  the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.

• Lastly you may find this github postenter link description here useful.

Answer 3

Beyond dropout (as @demento already explained) if you are training with data augmentation this is an expected behavior.

Answer 4

Also, it really depends how uniform your test data is distributed as well as its size. If you have relevantly so small test dataset compared to training one, it may be an expected output.