# Are the held-out datasets used for testing, validation or both?

I came across a new term "held-out corpora" and I confused regarding its usage in the NLP domain

Consider the following three paragraphs from N-gram Language Models

#1: held-out corpora as a non-train data

For an intrinsic evaluation of a language model we need a test set. As with many of the statistical models in our field, the probabilities of an $$n-$$gram model come from the corpus it is trained on, the training set or training corpus. We can then measure training set the quality of an n-gram model by its performance on some unseen data called the test set or test corpus. We will also sometimes call test sets and other datasets that are not in our training sets held out corpora because we hold them out from the held out training data.

This paragraph clearly says that held-out corpora can be used for either testing or validation or others except training.

#2: development set or devset for hyperparameter tuning

Sometimes we use a particular test set so often that we implicitly tune to its characteristics. We then need a fresh test set that is truly unseen. In such cases, we call the initial test set the development test set or,devset. How do we divide our data into training, development, and test sets? We want our test set to be as large as possible, since a small test set may be accidentally unrepresentative, but we also want as much training data as possible. At the minimum, we would want to pick the smallest test set that gives us enough statistical power to measure a statistically significant difference between two potential models. In practice, we often just divide our data into 80% training, 10% development, and 10% test. Given a large corpus that we want to divide into training and test, test data can either be taken from some continuous sequence of text inside the corpus, or we can remove smaller “stripes” of text from randomly selected parts of our corpus and combine them into a test set.

This paragraph clearly says that development set is used for hyperparameter tuning.

#3: held-out corpora for hyperparameter tuning

How are these $$\lambda$$ values set? Both the simple interpolation and conditional interpolation $$\lambda'$$s are learned from a held-out corpus. A held-out corpus is an additional training corpus that we use to set hyperparameters like these $$\lambda$$ values, by choosing the $$\lambda$$ values that maximize the likelihood of the held-out corpus.

This paragraph is clearly saying that held-out corpus is used for hyper-parameter training.

I am interpreting or understanding the terms as follows:

Train corpus is used to train the model for learning parameters.

Test corpus is used for evaluating the model wrt parameters.

Development set is used for evaluating the model wrt hyperparameters.

Held-out corpus includes any corpus outside training corpus. So, it can be used for evaluating either parameters or hyperparameters.

To be concise, informally, data = training data + held-out data = training data + development set + test data

Is my understanding true? I got confusion because of paragraph 3, which says that held-out corpus is used (only) for learning the hyperparameters while paragraph 1 says that held-out corpus includes any corpus outside train corpus. Does held-out corpora include devset or same as devset?

I think that these terms may be used inconsistently across sources.

If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the context.

In your second quote, the development set seems to be used as a synonym for validation dataset (a more common name to refer to the same concept), i.e. the dataset used for early stopping or hyper-parameter optimization (see also this).

The results above suggest a simple way of achieving this, namely by taking the available data and partitioning it into a training set, used to determine the coefficients $$\mathbf{w}$$, and a separate validation set, also called a hold-out set, used to optimize the model complexity