4
$\begingroup$

I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a way I was able to understand. So, I'm trying to rephrase the question…

When working with neural networks, you typically split your data set into three parts:

  • Training set
  • Validation set
  • Test set

I understand that you use the training set for, well, train the network, and that you use the test set to verify how well it has learned: By measuring how well the network performs on the test set, you know what to expect when actually using it later on. So far, so good.

Now, a model has hyper parameters, which – besides the weights – need to be tuned. If you change these, of course, you get different results. This is where in all explanations the validation set comes into play:

  • Train using the training set
  • Validate how well the model performs using the validation set
  • Repeat this for a number of variants which differ in their hyperparameters (or do it in parallel, right from the start)
  • Finally, select one and verify its performance using the test set

Now, my question is: Why would I need steps 2 and 3? I could as well train multiple version of my model in parallel, and then run all of them against the test set, to see which performs best, and then use this one.

So, in other words: Why would I use the validation set for comparing the model variants, if I could directly use the test set to do so? I mean, I need to train multiple versions either way. What is the benefit of doing it like this?

Probably, there is some meaning to it, and probably I got something wrong, but I can't figure out what. Any hints?

$\endgroup$
5
$\begingroup$

The difference between the validation and test set in my opinion should be explained in this way:

  • the validation set is meant to be used multiple times.
  • the test set is meant to be used only once.

I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model training. In every tutorial standard datasets are used so that you don't have to worry about data collection, data labelling (it's really sad to see that lot of people have not a clue about what the inter annotator agreement is), data pre-processing and especially, all the part about the real application of the model is almost never mentioned.

The importance of having a set of instances that you can use for fine tuning (validation) and a set of instances that your model never encountered neither in training nor during fine tuning (test) becomes particularly clear if you focus on the subsequent deployment of the model you trained. No one expect a model to have the same performance scores in training and when applied to some unknown data. And the crucial point is that the performance of a model on the validation set are not representative either of the behaviour of a model with unknown data, because the same validation data have been used to fine tuned the model! So here's why having a set of data completely new to the model is important, because it gives you a much more unbiased view about the model performance on a real use case scenario.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Still doesn't really answer the question. Can it be proven that validation set is actually fruitful? $\endgroup$ – DuttaA Apr 6 at 19:50
  • 1
    $\begingroup$ @DuttaA This answers the question. The validation set is used for fine-tuning (or similar tasks), so you select a certain model (or hyper-parameters) based on it. The validation dataset is certainly useful (e.g. it is used for early stopping). The main question is: why do we need the test and validation sets? And the reason is simple: you want to have an idea of how your model performs on really unseen data. In terms of learning theory, I am sure you can formalize this thing too, but I don't remember having seen this formalised. Usually, you just want your model to learn the training data. $\endgroup$ – nbro Apr 6 at 20:11
  • $\begingroup$ @nbro it is formalized in LT only that it is way too theoretical than anything I have came across. Why I said it didn't answer the question is because OP said that he would train multiple models and verify on the train set and select the best, whereas in CV we fine tune parameters, here we already are training a wide array of NNs and selecting the best. So how does CV add any value? The CV set is only of value in early stopping, etc which requires continuous evaluation along with training. OP didn't indicate such a case, and I can informally show in such cases the performance maybe worse. $\endgroup$ – DuttaA Apr 6 at 20:25
  • $\begingroup$ I actually answered a question regarding this a while back stats.stackexchange.com/questions/410118/… but I think it requires a deeper look than the answer I provided above. $\endgroup$ – DuttaA Apr 6 at 20:29
  • $\begingroup$ @DuttaA You're also talking about CV, but the asker didn't mention CV, which is another beast. I usually think of CV as a way of randomizing the training procedure and exploiting all data that you have to train the model. This can be particularly useful when you have small datasets. $\endgroup$ – nbro Apr 6 at 21:11
2
$\begingroup$

Simply stated, you use your validation set to regularize your model for unseen data. Test data is completely unseen data, on which you evaluate your model.

Various validation strategies are used to improve your model to perform for unseen data. So strategies like k-fold cross-validation are used. Also, the validation set helps you in tuning your hyperparameters such as learning rate, batch size, hidden units, number of layers, etc.

Train, Validation, Test sets help you in identifying whether you are underfitting or overfitting.

E.g. If human error at a task is 1%, train error is 8%, validation error is 10%, test set error is 12 % then,

Difference between,

  1. Human level and training set error tells you about "Avoidable Bias"
  2. Training set error and Validation set error tells you about "Variance and data mismatch"
  3. Validation set error and Test error tells you about "degree of overfitting" with the validation set.

Based on these metrics, you can apply appropriate strategies for better performance on validation or test sets.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.