I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.
Here's what I was thinking:
train/test split the data, with only one data point in the test set.
train the model on training data, potentially with grid-search/cross-validation
use the best model from step 2 to make a prediction on the one data point and save the prediction in an array
repeat the previous steps until all the data points have been in the test set
calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions
The pros of this approach would be to:
- You don't have to split the data into train/test so you can train with more data points.
The cons would be:
This approach suffers from potential(?) data leakage.
You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.
I have tried the standard approaches of train/test splitting, but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.
I would really appreciate some feedback about whether this approach is actually feasible or not and why.