1
$\begingroup$

Consider the following statement from p14 of Naive Bayes and Sentiment Classification

While the use of a devset avoids overfitting the test set, having a fixed training set, devset, and test set creates another problem: in order to save lots of data for training, the test set (or devset) might not be large enough to be representative.

I heard about overfitting on train data. A model is said to be overfit on train data if it is giving low train error and high test error.

But, what does it mean overfitting on test set?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Essentially, any data you use to train or develop the model shouldn't be used as test data. In principle, "unseen" data gives a good estimate for the generalisation performance of the model; but this is only valid if the data really is unseen and hasn't been used in the model development process. If you've been tuning a model to increase its accuracy on the test set, then that data has influenced the model, so it's not unseen any more!

An example of what is wrong:

  1. Train a neural network on the training set.
  2. Evaluate the performance on the test set, and then change the parameters of the model in some way to try and increase the test set performance.
  3. Use the best parameters you found, and get a final evaluation of performance on the test set.

To make this procedure legitimate, you should have a three way split: train, dev, test. Do the tuning on the dev set, and then you can get a final estimate of generalisation using the test set.

If you don't do this, you'll generally think your model is a lot more accurate than it actually is. It's just like trying to estimate your generalisation performance from the training set, which I'm sure you know is a bad idea!

This phenomenon is what is sometimes known as overfitting the test set. To see why this name is used, just consider what overfitting the training data is: picking parameters that seem to fit the training data well, but don't generalise well. Likewise, overfitting the test set involves picking hyperparameters that seem to work well, but don't generalise. In each case, the solution is to have an additional set so you can get an unbiased estimate of what's actually happening.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I think the name "dev" is becoming more popular, and is better than "cross validation" because the cv set is often used in the role of test set in other circumstances (such as other stats disciplines than ML). However, worth noting that the alternative names for dev set are "validation", "cross validation" or "cv". $\endgroup$
    – Neil Slater
    Jul 7, 2021 at 11:32
  • 1
    $\begingroup$ I've never seen the terms "cross-validation" or "cv" being used to refer to the "validation set", but I've seen the term "developmental set" being used to refer to the validation set. Cross-validation typically either refers to k-fold cross-validation, the simple training-test split, or the training-validation-test split approaches. $\endgroup$
    – nbro
    Jul 7, 2021 at 12:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .