Higher accuracy in the test set than in the training set

Question

Hi I'm trying to train an ANN model to classify images containing these characters: 0,1,2,3,4,T,X,S eg. etc...

so something like the classification of records of the MNIST dataset but using my generated images (not handwritten).

I generate ~10000 records that have to be divided in the training set and in the test set:

n = 10000
X_train, y_train = generate_typed(typed_alphabet_resized.astype('uint8'), int(n*0.8))
X_test, y_test = generate_typed_test(typed_alphabet_resized.astype('uint8'), int(n*0.3))

generate_typed and generate_typed_test are pretty much the same function, the sole difference is the augmentation applied to images: for generate_typed_test the only augmentation applied is scale(1,3) while for generate_typed:

augmentation_seq = iaa.Sequential([
    iaa.Sometimes(0.3, iaa.GaussianBlur(sigma=(0, .1))), 
    iaa.Sometimes(0.3, iaa.AdditiveGaussianNoise(loc=0, scale=(0.0, 0.05*255), per_channel=0.2)), 
    iaa.Affine(scale=(1, 3)), 
], random_order=True)

I did that to make the model more generalized as possibile. But really the kind of images the model is supposed to work on in the real world might just (sometimes) be zoomed in or be slightly less centered. However even if I use these two different functions, the images I got are very very similar.

Now, the problem is: I got a bigger accuracy on the test set than on the training set, and I really can't understand why and how to fix it. It seems like that the accuracy of my model doesn't matter, if my model has around 80% of accuracy or 95% of accuracy, I still got an 0.9997 on test set (bigger than the accuracy during the training). Note: Even if I apply the same type of trasformations on the train and test set, or if I apply no trasformation to both sets nothing changes.

I'm not saying the model doesn't work, but it's really wrong quite often. I don't know if this means overfitting or not, but I don't think so because loss & accuracy doesn't intersect each other:

Model:

model3 = Sequential([
  Flatten(input_shape=(28,28,1)), # "appiattisco" l'immagine
  Dense(20, activation='relu'), # creo uno strato denso la cui funzione di attivazione è relu
  Dropout(0.5), # Spengo a caso dei neuroni (per evitare overfitting)
  Dense(num_classes, activation='softmax') # strato denso di 10 neuroni con funzione di attivazione softmax (perchè è un classificatore)
])

model3.compile(
    loss = 'sparse_categorical_crossentropy',
    metrics=['accuracy'],
    optimizer=Adam(lr=0.01)
)


X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.1, random_state=42)
model3.fit(X_train, y_train, epochs=5, batch_size = 5, validation_data=(X_val, y_val), callbacks=[early_stop])

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 flatten_34 (Flatten)        (None, 784)               0         
                                                                 
 dense_71 (Dense)            (None, 20)                15700     
                                                                 
 dropout_37 (Dropout)        (None, 20)                0         
                                                                 
 dense_72 (Dense)            (None, 105)               2205      
                                                                 
=================================================================
Total params: 17,905
Trainable params: 17,905
Non-trainable params: 0

Answer 1

Your test set images are more clean than your training set images, because you applied more types of noise to the training set images, so the classifier performs better on the cleaner images in the test set.

I don't see why this is surprising.

Answer 2

Getting higher accuracy in test set than in training set means that your test set is easier to work on than training set. Your graph shows results for only one epoch. In the 1st epoch, your train accuracy/loss will include the learning phase when loss was high whereas by the time evaluation happens, the model has already learned. Is the behaviour same in subsequent epochs?

As you mentioned that you get similar results even with same or no augmentation, ignoring the learning phase issue, it is possible that your training set has more samples for which the network doesn't perform well, dragging the performance down. You can test this using something like k-fold cross validation which tests your network on multiple train-test pairs.