I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that
- randomly divided the data-set into 5 folds;
- copy-pasted all 5-fold data-set to have 5 exact copies of data-set (folder_1 to folder_5, all absolutely same data-set)
- first fold in folder_1 is saved as
testfolder and remaining (fold_2, fold_3, fold_4, fold_5) are combined as one
- second fold in folder_2 is saved as
testfolder and remaining (namely, fold_1, fold_3, fold_4, fold_5) are combined as one
- third fold in folder_3 is saved as
testfolder and remaining (namely, fold_1, fold_2, fold_4, fold_5) are combined as one
- similar process has been done on folder_4 and foder_5.
I hope, by now, you got the idea of how I distributed the data-set.
The reason I did so is as follows:
I have augmented the training data (
train folder) in each of the folders and used
test folders respectively to evaluate (ROC-AUC score). Now I kind of have 5 ROC-AUC scores which I evaluated using
test folders. If I get the average value out of those 5 scores.
(Assuming the above cross-validation process is done right) If I were to perform some manual hyperparameter optimizations (like an optimizer, learning rate, batch size, dropout, activation) and perform the above cross-validation with data augmentation and find the best so-called "mean ROC-AUC", does it mean I successfully conducted hyper-parameter optimization?
FYI: I have no problem with computing power OR/AND time at all to loop through the hyper-parameters for this type of cross-validation with data augmentation