0
$\begingroup$

I trained a dataset with a 5-fold cross validation to search for hyper-parameters by an AUROC metric. For splitting I used

five_splits = list(StratifiedKFold(n_splits=5, shuffle=True).split(X, y)) and looped through each fold.

enter image description here

Then I trained 80% of the dataset with optimal (for now) hyper-parameters from 5-fold CV, and plotted the AUROC on 20% of the dataset. For splitting I used

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

enter image description here

I don't think that shuffling of the samples or randomness may cause such a great gap between the scores. Also, the pre-processing and any other operations are identical in each training. I repeated this training for several times to no avail.

I can't seem to find the reason for what the cause of this ~0.06 difference might be.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.