# how to decide the optimum model?

I have split the database available into 70% training, 15% validation, and 15% test, using holdout validation. I have trained the model and got the following results: training accuracy 100%, validation accuracy 97.83%, test accuracy 96.74%

In another trial for training the model, I got the following results: Training accuracy 100%, validation accuracy 97.61%, test accuracy 98.91%

The same data split is used in each run. Which model should I choose, the first case in which the the test accuracy is lower than the validation? or the second case in which the test is higher than the validation?

• Do you have the same data split for each run - i.e. exactly the same examples in train, validation and test sets? Also, can you give size of each set? Are your numbers exact or have you simplified them? There would be a difference in analysis if your values were 98.3% vs 98.7% instead of 97.6% vs 99.2% for instance. What matters is the ratio of error rates - whilst 98% vs 99% appears as a ratio of 2, if you have rounded nearest then the ratio could be anything from 1.1 (not really meaningful) to 5.0 (impressive). Nov 4, 2021 at 21:29