Are the held-out datasets used for testing, validation or both?

Question

I came across a new term "held-out corpora" and I confused regarding its usage in the NLP domain

Consider the following three paragraphs from N-gram Language Models

#1: held-out corpora as a non-train data

For an intrinsic evaluation of a language model we need a test set. As with many of the statistical models in our field, the probabilities of an $n-$gram model come from the corpus it is trained on, the training set or training corpus. We can then measure training set the quality of an n-gram model by its performance on some unseen data called the test set or test corpus. We will also sometimes call test sets and other datasets that are not in our training sets held out corpora because we hold them out from the held out training data.

This paragraph clearly says that held-out corpora can be used for either testing or validation or others except training.

#2: development set or devset for hyperparameter tuning

Sometimes we use a particular test set so often that we implicitly tune to its characteristics. We then need a fresh test set that is truly unseen. In such cases, we call the initial test set the development test set or,devset. How do we divide our data into training, development, and test sets? We want our test set to be as large as possible, since a small test set may be accidentally unrepresentative, but we also want as much training data as possible. At the minimum, we would want to pick the smallest test set that gives us enough statistical power to measure a statistically significant difference between two potential models. In practice, we often just divide our data into 80% training, 10% development, and 10% test. Given a large corpus that we want to divide into training and test, test data can either be taken from some continuous sequence of text inside the corpus, or we can remove smaller “stripes” of text from randomly selected parts of our corpus and combine them into a test set.

This paragraph clearly says that development set is used for hyperparameter tuning.

#3: held-out corpora for hyperparameter tuning

How are these $\lambda$ values set? Both the simple interpolation and conditional interpolation $\lambda'$s are learned from a held-out corpus. A held-out corpus is an additional training corpus that we use to set hyperparameters like these $\lambda$ values, by choosing the $\lambda$ values that maximize the likelihood of the held-out corpus.

This paragraph is clearly saying that held-out corpus is used for hyper-parameter training.

I am interpreting or understanding the terms as follows:

Train corpus is used to train the model for learning parameters.

Test corpus is used for evaluating the model wrt parameters.

Development set is used for evaluating the model wrt hyperparameters.

Held-out corpus includes any corpus outside training corpus. So, it can be used for evaluating either parameters or hyperparameters.

To be concise, informally, data = training data + held-out data = training data + development set + test data

Is my understanding true? I got confusion because of paragraph 3, which says that held-out corpus is used (only) for learning the hyperparameters while paragraph 1 says that held-out corpus includes any corpus outside train corpus. Does held-out corpora include devset or same as devset?

Answer 1

I think that these terms may be used inconsistently across sources.

If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the context.

In your second quote, the development set seems to be used as a synonym for validation dataset (a more common name to refer to the same concept), i.e. the dataset used for early stopping or hyper-parameter optimization (see also this).

So, my answer to your question in the title would be

Yes, the heldout dataset can be used for validation or testing, but not because it's a special dataset, but because people may use this term to refer to either the validation dataset or the test dataset.

Here's another example of the usage of the term held-out set to refer, in this case, to the validation dataset (section 1.6 of this famous ML book)

The results above suggest a simple way of achieving this, namely by taking the available data and partitioning it into a training set, used to determine the coefficients $\mathbf{w}$, and a separate validation set, also called a hold-out set, used to optimize the model complexity

Here's another example that shows that the term may be used inconsistently (emphasis mine, taken from section 5.3 of the famous deep learning book by Goodfellow et al.). In fact, in that same section, they refer to the validation dataset, which is distinct from this held-out test set (so, in this case, the held-out set is used to refer only to the test set).

Earlier we discussed how a held-out test set, composed of examples coming from the same distribution as the training set, can be used to estimate the generalization error of a learner, after the learning process has completed.

Answer 2

In the sample dataset, the variable Sprint is the respondent's time (in seconds) to sprint a given distance, and Smoking is an indicator about whether or not the respondent smokes (0 = Nonsmoker, 1 = Past smoker, 2 = Current smoker). Use ANOVA to test if there is a statistically significant difference in sprint time with respect to smoking status. Sprint time will serve as the dependent variable, and smoking status will act as the independent variable.