0
$\begingroup$

The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly chosen population for training and validation sets with each training (no set random seed). The results on validation and then test sets are pretty satisfactory for my case however they are always different each time, precision spans between 0.7 and 0.9. This is due to fact that each time different data points fall to set with which model is trained.

My question is: how do I know that final training will also generate good model and how to estimate its precision when I do not have anymore unseen data?

Improve this question
$\endgroup$
1
  • $\begingroup$ I'm not sure if I understand your question correctly, but PAC model (probably approximately correct) offers some probabilistic performance guarantees on the trained models from the training accuracy and so on. Maybe this is what you're looking for: en.wikipedia.org/wiki/Probably_approximately_correct_learning $\endgroup$
    – SpiderRico
    Aug 25, 2020 at 0:07

2 Answers 2

Reset to default
2
$\begingroup$

The purpose of the test set is to test your model before deploying, otherwise, you would not need the test set in the first place. If you retrain your model by also including the validation and test datasets, of course, you cannot test your model anymore. You need to leave the test dataset separate and not use it for retraining, unless you have more data for testing.

Improve this answer
$\endgroup$
4
  • $\begingroup$ But the final model, deployed on production should be trained on all data. I just wonder how to be sure that such model will work well in future. $\endgroup$
    – GKozinski
    Aug 25, 2020 at 13:22
  • $\begingroup$ @Makintosz As I say in my answer, I don't know where you read this, but, no, the final model should not be trained on all data (as far as I am concerned), unless you are willing not to be able to test the model you will deploy. There's no way to test your model without data. In the end, your model is a machine learning model that learns from and is evaluated on data. $\endgroup$
    – nbro
    Aug 25, 2020 at 13:25
  • $\begingroup$ First source I could find: machinelearningmastery.com/train-final-machine-learning-model it says: "Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model. This is why we prefer to train the final model on all available data." $\endgroup$
    – GKozinski
    Aug 25, 2020 at 13:55
  • $\begingroup$ @Makintosz Well, it's true that the model will likely perform better given more data, but then you cannot empirically evaluate it anymore if you don't have more data. However, I've just seen that someone suggested looking into stuff like PAC learning and error bounds. Maybe that serves your purpose. $\endgroup$
    – nbro
    Aug 25, 2020 at 14:02
1
$\begingroup$

We usually divide the dataset into multiple subsets namely (training, validation and test sets). During training, we validate the model against the validation set. And during testing, we use the test dataset to obtain metrics for the model. We should make sure the subsets are taken from the same sample. Once you've tested it against the test subset, there's nothing really we can do.

You can also increase your dataset, by using multiple data sources if the problem statement allows you to.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .