0
$\begingroup$

Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following paragraph from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

Most machine learning algorithms have settings called hyperparameters, which must be determined outside the learning algorithm itself; we discuss how to set these using additional data.

My doubt is about the usage of the word 'additional' in the paragraph. Afaik, a small part of dataset under consideration is used to validate and hence in determining the hyperparameters, called as validation data. It is also a part of the dataset as training and testing data. You can check the section 5.3 for more details.

If yes, what is the need for the usage of the word 'additional'? Is it true that data for setting hyperparameters is taken outside of the underlying dataset?

$\endgroup$
1
$\begingroup$

I think it just tries to emphasize that you need three, non-intersection, chunks of datasets: training, validation, and test. So, you need some data in addition to the training data to tune hyperparameters. You can simply create the train/test/validation splits by sampling without replacement from an initial dataset. You don't need anything additional than this initial dataset.

$\endgroup$
1
  • $\begingroup$ The quote is from the introductory chapter, the word "additional" is used once in the OP's linked chapter, as is the word "dataset". Validation or dev are not mentioned at all. I think it is less emphasis, than a simple introduction without using all the jargon. $\endgroup$ Aug 25 at 14:05
0
$\begingroup$

Surely you understand the concept of overfitting: if you are optimizing over hyperparameters under the same underlying data, then technically you are optimizing a new dataset $(\theta, D)$, where $\theta$ are hyperparameters, and $D$ is the original dataset.

This is not good, because in theory, $\theta$ should be independent of the underlying specific $D$ in question, so anything learned is likely to be overfitted to the specific $D$. We mitigate this by requiring additional data for training hyperparameters, such that anything learned about $\theta$ would be about the underlying data-generating statistic and not just any specfic $D$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.