I trained a dataset with a 5-fold cross validation to search for hyper-parameters by an AUROC metric. For splitting I used

five_splits = list(StratifiedKFold(n_splits=5, shuffle=True).split(X, y)) and looped through each fold.

enter image description here

Then I trained 80% of the dataset with optimal (for now) hyper-parameters from 5-fold CV, and plotted the AUROC on 20% of the dataset. For splitting I used

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

enter image description here

I don't think that shuffling of the samples or randomness may cause such a great gap between the scores. Also, the pre-processing and any other operations are identical in each training. I repeated this training for several times to no avail.

I can't seem to find the reason for what the cause of this ~0.06 difference might be.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.