I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?
Unfortunately, the question you ask in the main body of your question here ("how do expert AI researchers do things") often turns out to actually be different from the question in your title ("how should I do things").
The very short answer would be that, ideally, you run as many repetitions as you can, but in practice it is indeed very often not feasible to do more than one or a handful.
For the long answer, I think it actually may be quite different also depending on what kind of machine learning you are doing. Personally, I am much more familiar with Reinforcement Learning and similar kinds of problems, i.e. problems where we're generating the "training" data ourselves by making agents act in environments, and similarly also evaluating by again making trained agents act in environments and measuring their performance. I'm not as familiar with the state-of-the-art research in "standard" machine learning tasks like classification/regression, but probably the problem is much worse in RL-style problems because:
- We actually have to generate our own data, which takes a huge amount of time (sometimes much more time than the actual training itself takes)
- There is often a lot of randomness in how we generate our training data, so we often have very different training datasets in different runs, which can of course also lead to wildly different levels of performance across different runs
- We pretty much always compute gradients from subsets (batches) of data in RL, which is again a source of randomness. In contrast, in supervised ML you could consider estimating gradients from the entire dataset at once per epoch, and then at least that part of your training process becomes deterministic (and hence replicable).
For the case of RL, recently a paper titled "Deep Reinforcement Learning at the Edge of the Statistical Precipice" appeared on arXiv, which gets into various tools and approaches you can use to more reasonably draw principled conclusions even from a small handful of repetitions of your training process (in better ways than the common practice of just reporting a mean, or even worse, just reporting the best result).
For supervised ML, some similar techniques may be applicable, but the need for this may also be less great (especially if you have little randomness in your training process).
Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.
For this specific situation, I would be interested to know by how much A is sometimes better than B, and B is sometimes better than A. If we're talking about like 0.1% differences in accuracy either way at a baseline accuracy of 99.5%, that probably just means they're both equally good. If you were "hoping" for one of them to outperform the other, well then that's probably disappointing... but for the overall training process in general, you could actually draw a more positive conclusion that, apparently, it is robust; it performs similarly across different runs even with different parameters!
On the other hand, if you observe differences in performance of significant magnitudes (like, 80% vs 90% accuracy), but sometimes A better by that much and sometimes B better by that much... then this strongly suggests that you indeed are dealing with high variance and you're going to have to do many repetitions to get statistically meaningful results.
