I trained a dataset with a 5-fold cross validation to search for hyper-parameters by an AUROC metric. For splitting I used
five_splits = list(StratifiedKFold(n_splits=5, shuffle=True).split(X, y)) and looped through each fold.
Then I trained 80% of the dataset with optimal (for now) hyper-parameters from 5-fold CV, and plotted the AUROC on 20% of the dataset. For splitting I used
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
I don't think that shuffling of the samples or randomness may cause such a great gap between the scores. Also, the pre-processing and any other operations are identical in each training. I repeated this training for several times to no avail.
I can't seem to find the reason for what the cause of this ~0.06 difference might be.