-1
$\begingroup$

I have split the database available into 70% training, 15% validation, and 15% test, using holdout validation. I have trained the model and got the following results: training accuracy 100%, validation accuracy 97.83%, test accuracy 96.74%

In another trial for training the model, I got the following results: Training accuracy 100%, validation accuracy 97.61%, test accuracy 98.91%

The same data split is used in each run. Which model should I choose, the first case in which the the test accuracy is lower than the validation? or the second case in which the test is higher than the validation?

$\endgroup$
1
  • $\begingroup$ Do you have the same data split for each run - i.e. exactly the same examples in train, validation and test sets? Also, can you give size of each set? Are your numbers exact or have you simplified them? There would be a difference in analysis if your values were 98.3% vs 98.7% instead of 97.6% vs 99.2% for instance. What matters is the ratio of error rates - whilst 98% vs 99% appears as a ratio of 2, if you have rounded nearest then the ratio could be anything from 1.1 (not really meaningful) to 5.0 (impressive). $\endgroup$ Nov 4, 2021 at 21:29

1 Answer 1

1
$\begingroup$

Testing each time on a test set is against the point of a train-val-test split. The reason test is important, is that you are only supposed to test on it when you think your model is good and ready with all final model and hyperparameter decisions made.

A good description can be found in this article: https://machinelearningmastery.com/difference-test-validation-datasets/

but to sum it up, test should be unbiased. The more you test against it, the more you bias the result.

$\endgroup$
1
  • $\begingroup$ More techincally for the last paragraph, the testing does not create bias, but the selecting does. If OP were to select a model using test, they could no longer trust the measurement, especially for really close values as in their question. I have other concerns with the question too though, where I have asked for clarification from the OP. $\endgroup$ Nov 5, 2021 at 14:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .