Is it possible that the model is overfitting when the training and validation accuracy increase?

Question

I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.

I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.

After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:

def basem():    
    
    model = Sequential()
    
    model.add(Dense(25, input_dim=38))
    model.add(Activation("relu"))
    
    model.add(Dense(50))
    model.add(Activation("relu"))
    
    model.add(Dense(50))
    model.add(Activation("relu"))
    
    model.add(Dense(25))
    model.add(Activation("relu"))
    
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
    return model


model_base = basem()
model_base._name = 'base'

history_base = model_base.fit(X_train, y_train, batch_size=2048, 
                    epochs=15, validation_data=(X_val,y_val), shuffle=True)

This gives me the following plot

It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:

test acc: 0.9816503485233088
test_prec: 0.9840033637114158
test_f1: 0.9816046990113001
test_recall: 0.9792384866432975
test_roc_auc: 0.9980004347946355

Dev acc: 0.052931962886091546
Dev prec: 0.2854656099314699
Dev f1: 0.057228805478181974
Dev recall: 0.3597811552056071
Dev roc auc: 0.0036456892671197097

If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?

Edit

Adding dropout (0.3) to each layer yields the following:

Now, my validation accuracy is higher than my training. I can't make sense of any of this.

Answer 1

I'll try to answer on more general questions

1. Is it ok that model performs better on validation, then on train?

It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.

2. I'm suspicious my model is too good. What could I do?

That a good point, because in my experience too good results sometime are reasoned by the flow in the training. The most common is data leakage, that's mean you give the model a way to fool around in unexpected and unwanted way.

Let me give an example. Lets imagine you try to detect malicious traffic and you have a data about sessions and you split records on train/val. Let's imagine one of the features is IP address and you could have multiple sessions from same IP. But if same IP would be in both train and val for the crooked users, then model could just remember the 'bad' IP and thus getting high score on both train and val.

So, to avoid it you need to think about things like that. If records could be group in some too meaningful way, make sure you put the whole group either in train or val. Another thing, that's it's usually good thing to split it by time as well, i.e. use for train records for 2018 and 2019 and for val for 2020. Thus you not only avoid data leakage, but you make sure your model is robust for future predictions.

More general, try to understand, why you getting such good result with such simple model. Try to squeeze the data even smaller, drop some features to understand, when it would go done.

Finally I would recommend the post by Andrej Karpathy, when he gave lots of really good advices on practical side of training NN. http://karpathy.github.io/2019/04/25/recipe/