1
$\begingroup$

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.

I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:

(1) Train set: 0.99 | Cross-validation set: 0.72
(2) Train set: 0.75 | Cross-validation set: 0.70
(3) Train set: 0.69 | Cross-validation set: 0.69

These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.

My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?

$\endgroup$
1
$\begingroup$

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data.

Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better with unseen data.

| improve this answer | |
$\endgroup$
  • $\begingroup$ This is very crisp answer. Thanks for sharing, @Lustwelpintje. $\endgroup$ – Supratim Haldar Nov 26 '19 at 10:22
0
$\begingroup$

Assuming that your cross-validation scores(both on train set and test set) indicate model's prediction performance correctly, you should definitely decide which trained model to use based on your validation accuracy only, regardless your model is overfitted or not.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, thanks for your answer. But I'm not sure if I got it entirely, can be please elaborate a bit more. From my above examples, which one would you go with and why? $\endgroup$ – Supratim Haldar Jun 29 '19 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.