I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a way I was able to understand. So, I'm trying to rephrase the question…
When working with neural networks, you typically split your data set into three parts:
- Training set
- Validation set
- Test set
I understand that you use the training set for, well, train the network, and that you use the test set to verify how well it has learned: By measuring how well the network performs on the test set, you know what to expect when actually using it later on. So far, so good.
Now, a model has hyper parameters, which – besides the weights – need to be tuned. If you change these, of course, you get different results. This is where in all explanations the validation set comes into play:
- Train using the training set
- Validate how well the model performs using the validation set
- Repeat this for a number of variants which differ in their hyperparameters (or do it in parallel, right from the start)
- Finally, select one and verify its performance using the test set
Now, my question is: Why would I need steps 2 and 3? I could as well train multiple version of my model in parallel, and then run all of them against the test set, to see which performs best, and then use this one.
So, in other words: Why would I use the validation set for comparing the model variants, if I could directly use the test set to do so? I mean, I need to train multiple versions either way. What is the benefit of doing it like this?
Probably, there is some meaning to it, and probably I got something wrong, but I can't figure out what. Any hints?