0
$\begingroup$

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.

I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:

(1) Train set: 0.99 | Cross-validation set: 0.72
(2) Train set: 0.75 | Cross-validation set: 0.70
(3) Train set: 0.69 | Cross-validation set: 0.69

These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.

My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?

$\endgroup$
0
$\begingroup$

Assuming that your cross-validation scores(both on train set and test set) indicate model's prediction performance correctly, you should definitely decide which trained model to use based on your validation accuracy only, regardless your model is overfitted or not.

$\endgroup$
  • $\begingroup$ Hi, thanks for your answer. But I'm not sure if I got it entirely, can be please elaborate a bit more. From my above examples, which one would you go with and why? $\endgroup$ – Supratim Haldar Jun 29 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.