Neural networks are usually evaluated by dividing a dataset into three splits:
- training,
- validation, and
- test
The idea is that critical hyperparameters of the network such as the number of epochs and the learning rate can be tuned by testing the network on the validation data while keeping the test data completely unseen until a final evaluation that happens only after the hyperparameters have been tuned.
However, if the amount of data is very small (e.g. 10-20 examples per class), then dividing the dataset into three splits may negatively impact the model due to lack of training data, and two splits is therefore preferable. A two split approach that makes a reasonable amount of data available for training is ten-fold stratified cross validation.
My question is -- is it statistically sound to tune hyperparameters by repeatedly evaluating hyperparameter sets using cross validation? Keep in mind that there is no held-out test data in this case, as the amount of available data is too small. I'd like some evidence/citations if possible showing that specifically for small datasets, this is the best approach for estimating the best hyperparameters that lead to the best generalizable model. Or if there is another approach that is better, I'd like to learn about that too.
