If the validation set is used to tune the hyperparameters and the training set adjusts the weights, why don't they be one thing as they have a similar role, as in improving the model?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Idea is to optimize with regards to unseen data in each step in order to avoid overfitting and data leakage so that the final network will be most generalizable to novel data.
First, you initialize your network weights randomly. For those weights, training data is unseen so network is optimized with regards to loss function that is calculated using training data. This was the first step.
Second, you would like to optimize hyperparameters, the parameters you use to train your network in first step. The network already worked hard to do its best in the first step while learning weights. If you use the same dataset in this step, it will have even more flexibility to fit even better to training data. But this will result in high variance, and network will perform poorly on unseen data.
For this reason, you split your data into train, dev and test. Train network with train data, optimize it with dev data and finally, evaluate with test data, never touching it until the very last step.
-
$\begingroup$ Why would the use of the training data for hyper-parameter tuning result in high variance? You should probably elaborate on this part because it's not clear the variance of which random variable you're referring to and why that's even the case. $\endgroup$– nbroCommented Jul 28, 2021 at 11:44
-
$\begingroup$ @crinix so you mean that dividing the training set and dev set allows overfitting to be reduced? $\endgroup$ Commented Jul 28, 2021 at 12:49
-
$\begingroup$ @nbro Optimizing hyper-parameters wrt trainingset causes high-variance and overfitting yes. It provides the model with more means to reduce the loss on training data, while increasing loss for unseen data. OmarZayed yes. $\endgroup$ Commented Jul 28, 2021 at 22:04
$\begingroup$
$\endgroup$
I think this is best explained using an analogy. Also you seen to have the misconception that you don't tune hyper-parameters for training data. You want to increase the accuracy of the training set AND validation set at the same time, but the validation set is more important so you want to maximise that accuracy more.
Imagine you had a toddler, and you were trying to teach them what an apple looks like. You have 10 pictures of apples, and for 45 years you sit them in a room and show them these apples. After all this time they will get to know the apples so well, that even the minutest differences in the photos would be noticed. When you try to see how the toddler generalised (how well it can use what it saw in training to evaluate real examples), it's absolutely terrible, because after all that time how could anything be an apple but the 10 it had seen?
So to combat this, you might think to reduce the number of years you spent showing the toddler the training set (the 10 apples), maybe that will allow them to generalise better (prevent over-fitting)? But you need a way to validate that this is actually helping, on unseen data (novel apples). That's where the 5 unseen apples come in to play (the validation set). You measure the accuracy of this unseen data to get an actual idea of how the toddler has learnt, because in a real example, the toddler isn't going to have seen the apple before, so it's important to know how it will handle unseen data.
That brings us finally to the testing set. The issue with what I've described above is maybe there's some kind of bias in the validation set, maybe those apples are slightly more round than usual. Of course you want the maximum accuracy on the validation set, so you tune everything to increase this accuracy, but this incurs a bias to the validation set. Maybe the parameters you chose are good only for this validation set. To make sure this isn't the case, you try and increase your accuracy as much as possible on the validation set, then after that you test the true accuracy on the testing set. This ensures you can't just tune your hyper-parameters to the validation set and call it a day. You need real generalisation to increase both validation AND testing set accuracy.
