I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.
I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.
After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:
def basem():
model = Sequential()
model.add(Dense(25, input_dim=38))
model.add(Activation("relu"))
model.add(Dense(50))
model.add(Activation("relu"))
model.add(Dense(50))
model.add(Activation("relu"))
model.add(Dense(25))
model.add(Activation("relu"))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
return model
model_base = basem()
model_base._name = 'base'
history_base = model_base.fit(X_train, y_train, batch_size=2048,
epochs=15, validation_data=(X_val,y_val), shuffle=True)
This gives me the following plot
It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:
test acc: 0.9816503485233088
test_prec: 0.9840033637114158
test_f1: 0.9816046990113001
test_recall: 0.9792384866432975
test_roc_auc: 0.9980004347946355
Dev acc: 0.052931962886091546
Dev prec: 0.2854656099314699
Dev f1: 0.057228805478181974
Dev recall: 0.3597811552056071
Dev roc auc: 0.0036456892671197097
If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?
Edit
Adding dropout (0.3) to each layer yields the following:
Now, my validation accuracy is higher than my training. I can't make sense of any of this.