There are two models for the same task:

model_1: 98% accuracy on training set, 54% accuracy on test set.
model_2: 48% accuracy on training set, 47% accuracy on test set.

From the statistics above we can say that model_1 overfits training set.
Q1: Can we say that model_2 underfits?
Q2: Why model_1 is bad choice if it performs better than model_2 on test set?


Whether a model is over-fitting the data cannot be determined solely from false positive and false negative data on training and test runs. If the sets are too small, very little can be determined. If the sample selection method is choosing inequitably with respect to the training model, accuracy information may be misleading.

If the first run was just a matter of luck in selection, that could be determined by making multiple training runs with the first model.

Even if over-fitting is the cause of the drop in testing accuracy for the first model, that does not mean that the over-fitting model isn't none-the-less more fit that the second model. It doesn't appear to be over-fit from that one training and test run, but may be deficient in other ways. The second model may under-fit, for instance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.