Why data required for hyperparameter tuning is considered as an additional data?

Question

Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following paragraph from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

Most machine learning algorithms have settings called hyperparameters, which must be determined outside the learning algorithm itself; we discuss how to set these using additional data.

My doubt is about the usage of the word 'additional' in the paragraph. Afaik, a small part of dataset under consideration is used to validate and hence in determining the hyperparameters, called as validation data. It is also a part of the dataset as training and testing data. You can check the section 5.3 for more details.

If yes, what is the need for the usage of the word 'additional'? Is it true that data for setting hyperparameters is taken outside of the underlying dataset?

Answer 1

I think it just tries to emphasize that you need three, non-intersecting, chunks of datasets: training, validation, and test. So, you need some data in addition to the training data to tune hyperparameters. You can simply create the train/test/validation splits by sampling without replacement from an initial dataset. You don't need anything additional than this initial dataset.

Answer 2

Surely you understand the concept of overfitting: if you are optimizing over hyperparameters under the same underlying data, then technically you are optimizing a new dataset $(\theta, D)$, where $\theta$ are hyperparameters, and $D$ is the original dataset.

This is not good, because in theory, $\theta$ should be independent of the underlying specific $D$ in question, so anything learned is likely to be overfitted to the specific $D$. We mitigate this by requiring additional data for training hyperparameters, such that anything learned about $\theta$ would be about the underlying data-generating statistic and not just any specfic $D$.