Should I use leave-one-out cross-validation for testing?

Question

I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.

Here's what I was thinking:

1. train/test split the data, with only one data point in the test set.

2. train the model on training data, potentially with grid-search/cross-validation

3. use the best model from step 2 to make a prediction on the one data point and save the prediction in an array

4. repeat the previous steps until all the data points have been in the test set

5. calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions

The pros of this approach would be to:

• You don't have to split the data into train/test so you can train with more data points.

The cons would be:

• This approach suffers from potential(?) data leakage.

• You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.

I have tried the standard approaches of train/test splitting, but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.

I would really appreciate some feedback about whether this approach is actually feasible or not and why.

Answer 1

Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the remaining $n-1$, and 2-fold Cross Validation at which you split your dataset in half and train (and validate) two models on two different halves.

The important aspect when choosing $k$ is a bias-variance tradeoff. Note that in LOOCV you train each model using almost as many samples as there are available ($n-1$), so the validation step should give you an unbiased estimate of the real test error. However, each model in LOOCV is trained each time on almost exactly same dataset!. This has important consequences, since the output of each model is highly correlated with each other. Since the mean of highly correlated variables has big variance, LOOCV will suffer from huge variance.

On the other hand, in 2-fold CV the models share no common samples, so they are not correlated and therefore their outputs have low variance. But since we train each model using only a half of available samples, the procedure will have a high bias (the estimated test error will be off from the true test error).

What to do in this scenario? Choose something in the middle. Usually $k=5$ and $k=10$ should be a good choice.