4
$\begingroup$

I have been recently reading about model selection algorithms (for example to decide which value of the regularisation parameter or what size of a neural network to use, broadly hyper-parameters). This is done by dividing the examples into three sets (training 60%, cross-validation 20%, test 20%) and training is done on the data with the first set for all parameters, and then choose the best parameter based on the result in the cross-validation and finally estimate the performance using the test set.

I understand the need for a different data-set compared to training and test for select the model, however, once the model is selected, why not using the cross-validation examples to improve the hypothesis before estimating the performance?

The only reason I could see is that this could cause the hypothesis to worsen and we wouldn't be able to detect it, but, is it really possible that by adding much more examples (60% -> 80%) the hypothesis gets worse?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

You are quite correct. If you have properly followed the Cross Validation procedure and selected the best model indeed, then you can use the CV set as the training set for the final model. And no it will not cause your hypothesis to worsen (for that set maybe, but not for new examples) if you have selected the model correctly. In-fact you may use the entire 100% of the data-set.

Justin Johnson a TA at Stanford University answered a similar type of question on training CNN's using 100% of the data-set. He said that if you had enough computational resources and want to squeeze that extra 1% or 2% accuracy from your model you can use the entire data-set after model selection.

NOTE: As @NeilSlater pointed out, if you need the model for reporting purposes you should only use 80% of the data-set, otherwise you'll lose your only source for unbiased model verification. But if you are looking to deploy the model on field you can use 100% of the data-set.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ There is a problem with using 100% of your training set for training. That is, you have no unbiased measure of the newly-trained model's performance. This may or may not be important if all you care about is "this is probably the best model", but I think you should note that caveat in your second paragraph. $\endgroup$
    – Neil Slater
    Commented Aug 29, 2018 at 17:18
  • 2
    $\begingroup$ @NeilSlater yes ..But then we have already made the choice of going along with the model, I think when we'll put the model on real data then 100% is used, but for reporting purposes in Scientific papers it'll be 80% like u said $\endgroup$
    – user9947
    Commented Aug 29, 2018 at 18:45
  • 2
    $\begingroup$ @DuttaA Neil is correct. The reason you don't retrain on the whole set when reporting for academic writing is the same reason you wouldn't do it in most industrial settings: you have no idea whether the model has overfit, and to what degree, if you don't have a validation set for it. It is actually extremely common for simpler ML models to become worse when more data is added, and you can do this with Deep Networks too if you are not careful. Your TA is wrong, but you might be able to get away with this if you use techniques like dropout to combat overfitting, and small scale testing. $\endgroup$
    – John Doucette
    Commented Aug 30, 2018 at 11:25
  • $\begingroup$ @JohnDoucette over fitting and all parameters have already been addressed by the question, I have never heard of ML models becoming worse by more data (when the best model has already been selected), the TA's answer is available in public domain videos so I doubt he is wrong...And I added neilslaters part in my answer $\endgroup$
    – user9947
    Commented Aug 30, 2018 at 11:53
  • $\begingroup$ @NeilSlater, I and DuttaA have created a chat room where we need your help. this chatroom goes by the name "Discussion by Ruchit Dalwadi". Can you join us & help ? $\endgroup$
    – Ruchit Dalwadi
    Commented Oct 17, 2018 at 12:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .