I am currently working with a small dataset of 20x300. Since I have so few datapoints, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.
Here's what I was thinking:
- train/test split the data, with only one data point in the test set.
- train the model on training data, potentially with grid_search/cross-validation
- use the best model from step 2 to make a prediction on the one data point and save the prediction in an array
- repeat the previous steps until all the data points have been in the test set
- calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions
The pros of this approach would be to:
- You don't have to split the data into train/test so you can train with more datapoints.
The cons would be:
- This approach suffers from potential(?) data leakage.
- You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.
I have tried the standard approaches of train/test splitting but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.
I would really appreciate some feedback about whether this approach is actually feasibly or not and why.