Short answer: Yes, the validation set should be the same otherwise you risk that a "lucky" set of validation samples is responsible for better performance.
Long answer: A fair comparison of 2 or more models can only occur if these are both trained and evaluated in exactly the same conditions. That is:
- Same training data, shuffled and batched in the same order just because of the stochastic nature of the optimizers.
- Same initialization of weights, or at least, in case of models with different architectures/layers, from the same pseudo-random generator's (PRNG) state.
- Validation should occur in the same way: data should be the same, as well as the model selection criterion.
In general, all the stochastic operations (SGD, weight init, train-test splits, samplings, etc) should be reproducible, for two reasons: 1) the performance of your model should not depend on random numbers that change every run, and 2) you want to avoid having both a lucky or unlucky training just due to the PRNG. To (almost) solve this, one has to set the same random seed for all the random generators. This may be not enough, due to some inherently non-deterministic operations (e.g. some calculations on GPU due to CUDA optimizations, parallelism, etc) but more importantly the (sequence) of PRNG states should be the same across runs and comparisons: Python or numpy's PRNG are stateful, meaning that each time you ask for random numbers they change their internal state. In particular the numpy's PRNG consumes entropy, being more and more deterministic as it is used. One should do as JAX forces: use stateless PRNGs, and manually keep track of the state.
Regarding statement 2, ou have to consider that a single run of a model gives a punctual estimation of the real performance of the model: what you want to know is an expected value, instead, i.e. something associated with a standard deviation for example. This is for a more broad discussion, but to properly compare two or more models (methods, etc) a single number (e.g. AUC) is not enough because you cannot decide whether M1 is really better than M2 because of the superior architecture or better hyper-params, or because it was just a lucky random seed or lucky data: since your training and validation sets are not infinite, model performance are subject to inherent fluctuations. Instead, training a model 100 times, for example, considering different initial situations (but different across the runs, and not along compared models), allows you to aggregare performance obtaining a mean and associated std. For example, M1 can achieve 95% ($\pm 3$) AUC while M2 achieves 95% ($\pm 1$). So what model you choose? both achieve the same performance on average, but M1 has more variability; there are approaches that perform statistical tests to say that M1 has a probability $p$ to beat M2, and M2 has a prob $p_1$ to be better. According to such probability you can make a decision, for example.
To conclude, this is an open question (also because 100 runs are not practical).