Why do we need both the validation set and test set?

Question

I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a way I was able to understand. So, I'm trying to rephrase the question…

When working with neural networks, you typically split your data set into three parts:

• Training set
• Validation set
• Test set

I understand that you use the training set for, well, train the network, and that you use the test set to verify how well it has learned: By measuring how well the network performs on the test set, you know what to expect when actually using it later on. So far, so good.

Now, a model has hyper parameters, which – besides the weights – need to be tuned. If you change these, of course, you get different results. This is where in all explanations the validation set comes into play:

• Train using the training set
• Validate how well the model performs using the validation set
• Repeat this for a number of variants which differ in their hyperparameters (or do it in parallel, right from the start)
• Finally, select one and verify its performance using the test set

Now, my question is: Why would I need steps 2 and 3? I could as well train multiple version of my model in parallel, and then run all of them against the test set, to see which performs best, and then use this one.

So, in other words: Why would I use the validation set for comparing the model variants, if I could directly use the test set to do so? I mean, I need to train multiple versions either way. What is the benefit of doing it like this?

Probably, there is some meaning to it, and probably I got something wrong, but I can't figure out what. Any hints?

Answer 1

The difference between the validation and test set in my opinion should be explained in this way:

• the validation set is meant to be used multiple times.
• the test set is meant to be used only once.

I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model training. In every tutorial standard datasets are used so that you don't have to worry about data collection, data labelling (it's really sad to see that lot of people have not a clue about what the inter annotator agreement is), data pre-processing and especially, all the part about the real application of the model is almost never mentioned.

The importance of having a set of instances that you can use for fine tuning (validation) and a set of instances that your model never encountered neither in training nor during fine tuning (test) becomes particularly clear if you focus on the subsequent deployment of the model you trained. No one expect a model to have the same performance scores in training and when applied to some unknown data. And the crucial point is that the performance of a model on the validation set are not representative either of the behaviour of a model with unknown data, because the same validation data have been used to fine tuned the model! So here's why having a set of data completely new to the model is important, because it gives you a much more unbiased view about the model performance on a real use case scenario.

Answer 2

Simply stated, you use your validation set to regularize your model for unseen data. Test data is completely unseen data, on which you evaluate your model.

Various validation strategies are used to improve your model to perform for unseen data. So strategies like k-fold cross-validation are used. Also, the validation set helps you in tuning your hyperparameters such as learning rate, batch size, hidden units, number of layers, etc.

Train, Validation, Test sets help you in identifying whether you are underfitting or overfitting.

E.g. If human error at a task is 1%, train error is 8%, validation error is 10%, test set error is 12 % then,

Difference between,

1. Human level and training set error tells you about "Avoidable Bias"
2. Training set error and Validation set error tells you about "Variance and data mismatch"
3. Validation set error and Test error tells you about "degree of overfitting" with the validation set.

Based on these metrics, you can apply appropriate strategies for better performance on validation or test sets.