I was taught that, usually, a dataset has to be divided into three parts:
How is MNIST only providing the training and the test sets? What about the validation?
The test set should never be seen and ran once at the end of training.
The validation set is used to help you select hyperparameters and it would be cheating to tune your model on the test set because you would be giving your model information about the test set. This would give your model an unfair advantage and skew the results; that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.
For this reasons, the validation set must be a portion of the training data which is selected out and evaluated on during training so that you can do this. It's not necessary if you're not doing model selection.
There are a few reasons why MNIST only provides training and test sets, and not a validation set:
Since, the MNIST dataset is intended to be a simple and straightforward benchmark for machine learning models, It is important to have a standard test set that can be used to compare different models.
Secondly, the MNIST dataset is well-known and well-studied.
Thirdly splitting it into three sets (training, validation, and test) would reduce the size of each set too much & there is less need to have one.
The validation set is used to assess the performance of the model on unseen data. In this case, the validation set is not needed because the performance of the model can be assessed on the test set.
Another common method is to use cross-validation, which is where the data is split into multiple sets and each set is used to train and test the model. This allows for a more accurate assessment of the model's performance.
How is MNIST only providing the training and the test sets? What about the validation?
In order to perform cross-study comparison of model performance it makes a lot of sense to have a single test set for benchmarking. This way, different investigators can compare their respective models in an "apples to apples" manner. The test set is tested only once, as already mentioned, and is in a practical sense the final arbiter of model performance. As an aside, the MNIST not only provides an important source of data for testing individual hypotheses, but it also serves to provide "standards". After all, NIST stands for National Institute of Standards and Technology.
The reason why a validation set is not parsed out (like the way the test set is).
Different investigators may want to perform different types of validations (hold out, n-fold cross-validation, leave-one-out cross-validation). Thus, MNIST does not limit investigators from doing this by separating out a validation set. MNIST leaves it up to the investigator to parse their "training set" into yes, a training set and validation set they way they prefer. Unlike the test set, there is really no need to standardize the validation. In contrast, test set does need to serve as a standard for benchmarking models.
NOTE: To be clear, this does not mean that you should not perform a validation. You absolutely must. The only thing is that you must "carve out" the validation set in a manner of your choosing.
Good question, it probably should, but I guess since it's the "hello world" of deep learning, they may have wanted to simplify it.
But in your day to day use, you should split the training into train/validation, but keep test the way it is.
