How to fairly conduct a model performance with 5-fold cross validation after augmentation?

Question

I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that

• randomly divided the data-set into 5 folds;
• copy-pasted all 5-fold data-set to have 5 exact copies of data-set (folder_1 to folder_5, all absolutely same data-set)
• first fold in folder_1 is saved as test folder and remaining (fold_2, fold_3, fold_4, fold_5) are combined as one train folder
• second fold in folder_2 is saved as test folder and remaining (namely, fold_1, fold_3, fold_4, fold_5) are combined as one train folder
• third fold in folder_3 is saved as test folder and remaining (namely, fold_1, fold_2, fold_4, fold_5) are combined as one train folder.
• similar process has been done on folder_4 and foder_5.

I hope, by now, you got the idea of how I distributed the data-set.

The reason I did so is as follows:

I have augmented the training data (train folder) in each of the folders and used test folders respectively to evaluate (ROC-AUC score). Now I kind of have 5 ROC-AUC scores which I evaluated using test folders. If I get the average value out of those 5 scores.

(Assuming the above cross-validation process is done right) If I were to perform some manual hyperparameter optimizations (like an optimizer, learning rate, batch size, dropout, activation) and perform the above cross-validation with data augmentation and find the best so-called "mean ROC-AUC", does it mean I successfully conducted hyper-parameter optimization?

FYI: I have no problem with computing power OR/AND time at all to loop through the hyper-parameters for this type of cross-validation with data augmentation

Answer 1

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes, you successfully conducted hyper-parameter optimization. However, the AUC you received for the best set of hyperparameters found (by manual tuning) is not representative of the real performance of your model.

This is because the fact of using a test set to tune the parameters of your model degenerates it back to a "training" set, because the data is not being used to measure the performance of the model but to improve it instead (although on a different level of abstraction, i.e. not to directly influence the parameters of the model, such as neural network weights), making the resulting AUC an overly optimistic biased estimator of the real AUC (that it could have resulted in for an unseen test dataset).

That's why, if you care both about hyperparameter optimisation and being able to measure the "real" performance of your model, you need to split your dataset into three "buckets": training set $X_{train}$, validation st $X_{val}$ and test set $X_{test}$. $X_{test}$ should only be used once (after you've trained and tuned the model), and assuming it has enough samples and the samples are representative of the unseen data, you should get a good estimate of your model performance. $X_{val}$, on the other hand, is your validation set which you can reuse as many times as you want to find the optimal set of hyperparameters that result in the highest performance (i.e. AUC).