In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the true population from which data is sampled.
More generally, in AI research, we often end up testing performance of newly proposed algorithms / ideas on the same benchmarks over and over again. For example:
- For over a decade, researchers kept trying thousands of ideas on the game of Go.
- The ImageNet dataset has been used for huge amounts of different publications
- The Arcade Learning Environment (Atari games) has been used for thousands of Reinforcement Learning papers, having become especially popular since the DQN paper in 2015.
Of course, there are very good reasons for this phenomenon where the same benchmarks keep getting used:
- Reduced likelihood of researchers "creating" a benchmark themselves for which their proposed algorithm "happens" to perform well
- Easy comparison of results to other publications (previous as well as future publications) if they're all consistently evaluated in the same manner.
However, there is also a risk that the research community as a whole is in some sense "overfitting" to these commonly-used benchmarks. If thousands of researchers are generating new ideas for new algorithms, and evaluate them all on these same benchmarks, and there is a large bias towards primarily submitting/accepting publications that perform well on these benchmarks, the research output that gets published does not necessarily describe the algorithms that perform well across all interesting problems in the world; there may be a bias towards the set of commonly-used benchmarks.
Question: to what extent is what I described above a problem, and in what ways could it be reduced, mitigated or avoided?