How can AI researchers avoid "overfitting" to commonly-used benchmarks as a community?

Question

In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the true population from which data is sampled.

More generally, in AI research, we often end up testing performance of newly proposed algorithms / ideas on the same benchmarks over and over again. For example:

• For over a decade, researchers kept trying thousands of ideas on the game of Go.
• The ImageNet dataset has been used for huge amounts of different publications
• The Arcade Learning Environment (Atari games) has been used for thousands of Reinforcement Learning papers, having become especially popular since the DQN paper in 2015.

Of course, there are very good reasons for this phenomenon where the same benchmarks keep getting used:

• Reduced likelihood of researchers "creating" a benchmark themselves for which their proposed algorithm "happens" to perform well
• Easy comparison of results to other publications (previous as well as future publications) if they're all consistently evaluated in the same manner.

However, there is also a risk that the research community as a whole is in some sense "overfitting" to these commonly-used benchmarks. If thousands of researchers are generating new ideas for new algorithms, and evaluate them all on these same benchmarks, and there is a large bias towards primarily submitting/accepting publications that perform well on these benchmarks, the research output that gets published does not necessarily describe the algorithms that perform well across all interesting problems in the world; there may be a bias towards the set of commonly-used benchmarks.

---

Question: to what extent is what I described above a problem, and in what ways could it be reduced, mitigated or avoided?

Answer 1

Great question Dennis!

This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year.

Usually these discussions cover a number of themes:

1. "Existing benchmarks suck". This is usually the topic that opens discussion. In the 2015/2016 discussion, which focused on the Turing Test as a benchmark specifically, criticisms ranged from "it doesn't incentivize AI research on the right things", to claims that it was poorly defined, too hard, too easy, or not realistic.
2. General concensus that we need new benchmarks.
3. Suggestions of benchmarks based on various current research directions. In the latest discussion this included answering standardized tests for human students (well defined success, clear format, requires linking and understanding many ares), playing video games (well defined success, requires visual/auditory processing, planning, coping with uncertainty), and switching focus to robotics competitions.

I remember attending very similar discussions at machine learning conferences in the late 2000's, but I'm not sure anything was published out of it.

Despite these discussions, AI researchers seem to incorporate the new benchmarks, rather than displacing the older ones entirely. The Turing Test is still going strong for instance. I think there are a few reasons for this.

First, benchmarks are useful, particularly to provide context for research. Machine learning is a good example. If the author sets up an experiment on totally new data, then even if they apply a competing method, I have to trust that they did so faithfully, including things like optimizing the parameters as much as with their own methods. Very often they do not do this (it requires some expertise with competing methods), which inflates the reported advantage of their own techniques. If they also run their algorithm on a benchmark, then I can easily compare it to the benchmark performances reported by other authors, for their own methods. This makes it easier to spot a technique that's not really effective.

Second, even if new benchmarks or new problems are more useful, nobody knows about them! Beating the current record for performance on ImageNet can slingshot someone's career in a way that top performance on a new problem simply cannot.

Third, benchmarks tend to be things that AI researchers think can actually be accomplished with current tools (whether or not they are correct!). Usually iterative improvement on them is fairly easy (e.g. extend an existing technique). In a "publish-or-perish" world, I'd rather publish a small improvement on an existing benchmark than attempt a risker problem, at least pre-tenure.

So, I guess my view is fixing the dependence on benchmarks involves fixing the things that make people want to use them:

1. Have some standard way to compare techniques, but require researchers to also apply new techniques to a real world problem.
2. Remove the career and prestige rewards for working on benchmark problems, perhaps by explicitly tagging them as artificial.
3. Remove the incentives for publishing often.

Answer 2

In addition to the points already listed in John's answer, some factors that can help to reduce / mitigate the risk of overfitting to commonly-used benchmarks as a research community are:

1. Competitions with instances of problems hidden from entrants: as far as I'm aware this is particularly popular in game AI (see the General Game Playing competition and General Video Game Playing competitions). The basic idea is that submissions should be able to tackle a relatively broad class of problems (playing any game defined in a specified format, or generating levels for any video game with rules described in a specific format, etc.). To some extent, using a large suite of problems as a standard benchmark (such as the large collection of Atari games supported by ALE) also fits in with this idea, though there is value in hiding the problems that are ultimately used for testing from the people writing submissions. Of course, the idea is that entries submitted to these kinds of competitions will involve new research which may be published.

2. Using very simple toy problems: With simple I do not necessarily mean that they are simple to solve, but simple to describe / understand (it may still, for example, have a large state space and be difficult for current techniques to solve). Simple toy problems often help to test for a very specific "skill", and can more easily give insight into specifically why/when an algorithm may be expected to fail or succeed. Of course, large non-toy problems are also important to demonstrate "real-world" usefulness of algorithms, but they may often give less understanding / insight into an algorithm.

3. Theoretical work: Theoretical work can also give more insight and understanding of new algorithms. Algorithms with strong theoretical foundations are often more likely to generalize to a multitude of problem domains, assuming that the initial assumptions hold (big assumption here - there are plenty of cases where assumptions required for strong proofs do not hold!). This is not always possible / "needed", sometimes new research based purely on intuition and with relatively little theoretical foundations still turn out to work well (or theory is only developed after promising empirical results)... but it can certainly help. Theoretical work can take many different forms, such proofs of convergence (often under strict conditions), proofs for upper or lower bounds on important measures (such as regret, or probability of making a "wrong" choice, etc.), proofs that an algorithm or a problem is a more general or more specific case of an existing, well-understood algorithm or problem, proofs that a model has or does not have a certain representational capacity, proofs of algorithmic equivalence (that an algorithm computes exactly the same quantities as another well-understood algorithm, typically with lower computation and/or memory requirements), etc.

Answer 3

To better understand my point of view, I am using deep learning for geomatics and teledetection purposes.

So after reading with interest the great two previous answers, I would like to add my small contribution to this thread.

First, I would like to emphasis Insight knowledge: I do agree with the second point of Dennis's answer, "Simple" and well understood benchmarks helps for pin-pointing strengths of AI methods, which is good. But this is known by AI researchers. And if they work for general improvement and not just on one benchmark to "slingshot" their career as John's second point is mentioning, general improvement will be made. These thoughts links directly to my first point :

1. Multiplicity of benchmarks = Variety of problems = overfitting avoidance: If a paper is presenting an increment of performance on only one benchmark, it will ultimately say that their method is a specific one. So even if one benchmark can lead to an "overfitting", a group of benchmarks will offer a better variety of problems and thus a better insight of the generalization properties of an AI technique. A good application case of this is the PointNet paper (first end-to-end 3D point cloud neural network), where authors tested their approach against ModelNet40 (a well-known 3D classification benchmark), but also against MNIST, to verify the generalization capabilities of the network. Can "Meta-benchmarks" be a thing ? Benchmarks that are a concatenation of actual well-known ones (or ones who are known to give insight into one specific perk).

Then, in my field, where AI is seen more as a tool than in the theoretical deep learning field. It is known that there is a gap between benchmark results and "real life" (or applied) cases. This, in my opinion, is induced by the very high quality of benchmarks ground truth that isn't available for all cases. This leads to my second point :

2. Un/Semi-supervised benchmarks = as-noisy-as-the-reality benchmarks: As generalization is the capacity of dealing with the unknown, and because creating benchmarks on complex data is often time and money consuming, we should get out of the supervised way of thinking. This will increase the number of benchmarks available, which is good in regards to the first point, but also force theoretical AI research to focus more on these technic. It is well known that something is off with the actual supervised learning methods : Why do we need to show every possible case to a network ? Why can it not infer new knowledge from unlabeled data ? The biggest drag on this will be the data assessment. If semi-supervised benchmark will still give classical metrics, we will need a way of analyzing results on unlabeled data. In applied cases where ground truth is not often available, this is often done by visualization. But progress can be made to create a better, unbiased way on ranking methods. One idea can be to have a double-blinded random quality assessment in a captcha like manner of the results. I know this idea works only for human-understandable data, but other ways could be and needs to be found.