Do different models using early stopping have the same validation set to check model training performance?

Question

I, i have a doubt about making validation using early stopping given two NN models.

Suppose I have two models M1 and M2 and a Training set TS and Test set TS.

Take the TS and consider TS_80% and TS_20%, the first as training and the second as validation in the hold out procedure

Train M1 with early stopping on TS_80, choosing 10% of TS_80 as validation reference set for the early stopping procedure, we could call this TS_80_es_1

Then train M2 with early stopping on TS_80, choosing 10% of TS_80 as validation reference set for the early stopping procedure, calling it TS_80_es_2.

NOW, must TS_80_es_1 be identical to TS_80_es_2? I mean, the validation set we use for the early stopping must be the same or can it be chosen as 10% randomly in each individual training for each model and so we can consider different validation sets in the early stopping procedure?

In other words, every model we train using early stopping has the same validation set for early stopping procedure, or the validation set can be different for each training phase?

Answer 1

Short answer: Yes, the validation set should be the same otherwise you risk that a "lucky" set of validation samples is responsible for better performance.

Long answer: A fair comparison of 2 or more models can only occur if these are both trained and evaluated in exactly the same conditions. That is:

• Same training data, shuffled and batched in the same order just because of the stochastic nature of the optimizers.
• Same initialization of weights, or at least, in case of models with different architectures/layers, from the same pseudo-random generator's (PRNG) state.
• Validation should occur in the same way: data should be the same, as well as the model selection criterion.

In general, all the stochastic operations (SGD, weight init, train-test splits, samplings, etc) should be reproducible, for two reasons: 1) the performance of your model should not depend on random numbers that change every run, and 2) you want to avoid having both a lucky or unlucky training just due to the PRNG. To (almost) solve this, one has to set the same random seed for all the random generators. This may be not enough, due to some inherently non-deterministic operations (e.g. some calculations on GPU due to CUDA optimizations, parallelism, etc) but more importantly the (sequence) of PRNG states should be the same across runs and comparisons: Python or numpy's PRNG are stateful, meaning that each time you ask for random numbers they change their internal state. In particular the numpy's PRNG consumes entropy, being more and more deterministic as it is used. One should do as JAX forces: use stateless PRNGs, and manually keep track of the state.

Regarding statement 2, ou have to consider that a single run of a model gives a punctual estimation of the real performance of the model: what you want to know is an expected value, instead, i.e. something associated with a standard deviation for example. This is for a more broad discussion, but to properly compare two or more models (methods, etc) a single number (e.g. AUC) is not enough because you cannot decide whether M1 is really better than M2 because of the superior architecture or better hyper-params, or because it was just a lucky random seed or lucky data: since your training and validation sets are not infinite, model performance are subject to inherent fluctuations. Instead, training a model 100 times, for example, considering different initial situations (but different across the runs, and not along compared models), allows you to aggregare performance obtaining a mean and associated std. For example, M1 can achieve 95% ($\pm 3$) AUC while M2 achieves 95% ($\pm 1$). So what model you choose? both achieve the same performance on average, but M1 has more variability; there are approaches that perform statistical tests to say that M1 has a probability $p$ to beat M2, and M2 has a prob $p_1$ to be better. According to such probability you can make a decision, for example.

To conclude, this is an open question (also because 100 runs are not practical).