I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.
I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.
After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:
def basem(): model = Sequential() model.add(Dense(25, input_dim=38)) model.add(Activation("relu")) model.add(Dense(50)) model.add(Activation("relu")) model.add(Dense(50)) model.add(Activation("relu")) model.add(Dense(25)) model.add(Activation("relu")) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy']) return model model_base = basem() model_base._name = 'base' history_base = model_base.fit(X_train, y_train, batch_size=2048, epochs=15, validation_data=(X_val,y_val), shuffle=True)
It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:
test acc: 0.9816503485233088 test_prec: 0.9840033637114158 test_f1: 0.9816046990113001 test_recall: 0.9792384866432975 test_roc_auc: 0.9980004347946355 Dev acc: 0.052931962886091546 Dev prec: 0.2854656099314699 Dev f1: 0.057228805478181974 Dev recall: 0.3597811552056071 Dev roc auc: 0.0036456892671197097
If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?
Adding dropout (0.3) to each layer yields the following:
Now, my validation accuracy is higher than my training. I can't make sense of any of this.