5
$\begingroup$

In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the true population from which data is sampled.

More generally, in AI research, we often end up testing performance of newly proposed algorithms / ideas on the same benchmarks over and over again. For example:

  • For over a decade, researchers kept trying thousands of ideas on the game of Go.
  • The ImageNet dataset has been used for huge amounts of different publications
  • The Arcade Learning Environment (Atari games) has been used for thousands of Reinforcement Learning papers, having become especially popular since the DQN paper in 2015.

Of course, there are very good reasons for this phenomenon where the same benchmarks keep getting used:

  • Reduced likelihood of researchers "creating" a benchmark themselves for which their proposed algorithm "happens" to perform well
  • Easy comparison of results to other publications (previous as well as future publications) if they're all consistently evaluated in the same manner.

However, there is also a risk that the research community as a whole is in some sense "overfitting" to these commonly-used benchmarks. If thousands of researchers are generating new ideas for new algorithms, and evaluate them all on these same benchmarks, and there is a large bias towards primarily submitting/accepting publications that perform well on these benchmarks, the research output that gets published does not necessarily describe the algorithms that perform well across all interesting problems in the world; there may be a bias towards the set of commonly-used benchmarks.


Question: to what extent is what I described above a problem, and in what ways could it be reduced / mitigated / avoided?

$\endgroup$
2
$\begingroup$

Great question Dennis!

This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year.

Usually these discussions cover a number of themes:

  1. "Existing benchmarks suck". This is usually the topic that opens discussion. In the 2015/2016 discussion, which focused on the Turing Test as a benchmark specifically, criticisms ranged from "it doesn't incentivize AI research on the right things", to claims that it was poorly defined, too hard, too easy, or not realistic.
  2. General concensus that we need new benchmarks.
  3. Suggestions of benchmarks based on various current research directions. In the latest discussion this included answering standardized tests for human students (well defined success, clear format, requires linking and understanding many ares), playing video games (well defined success, requires visual/auditory processing, planning, coping with uncertainty), and switching focus to robotics competitions.

I remember attending very similar discussions at machine learning conferences in the late 2000's, but I'm not sure anything was published out of it.

Despite these discussions, AI researchers seem to incorporate the new benchmarks, rather than displacing the older ones entirely. The Turing Test is still going strong for instance. I think there are a few reasons for this.

First, benchmarks are useful, particularly to provide context for research. Machine learning is a good example. If the author sets up an experiment on totally new data, then even if they apply a competing method, I have to trust that they did so faithfully, including things like optimizing the parameters as much as with their own methods. Very often they do not do this (it requires some expertise with competing methods), which inflates the reported advantage of their own techniques. If they also run their algorithm on a benchmark, then I can easily compare it to the benchmark performances reported by other authors, for their own methods. This makes it easier to spot a technique that's not really effective.

Second, even if new benchmarks or new problems are more useful, nobody knows about them! Beating the current record for performance on ImageNet can slingshot someone's career in a way that top performance on a new problem simply cannot.

Third, benchmarks tend to be things that AI researchers think can actually be accomplished with current tools (whether or not they are correct!). Usually iterative improvement on them is fairly easy (e.g. extend an existing technique). In a "publish-or-perish" world, I'd rather publish a small improvement on an existing benchmark than attempt a risker problem, at least pre-tenure.

So, I guess my view is fixing the dependence on benchmarks involves fixing the things that make people want to use them:

  1. Have some standard way to compare techniques, but require researchers to also apply new techniques to a real world problem.
  2. Remove the career and prestige rewards for working on benchmark problems, perhaps by explicitly tagging them as artificial.
  3. Remove the incentives for publishing often.
$\endgroup$
  • 1
    $\begingroup$ Some interesting thoughts! I think this is a kind of question that can have lots of different relevant answers, so I'll leave not mark any answer as "Accepted" for a while yet, see what other thoughts show up still. I'll get back to the question eventually to mark an answer as accepted, and probably also add some thoughts of my own :) $\endgroup$ – Dennis Soemers Aug 13 '18 at 16:51
2
$\begingroup$

In addition to the points already listed in John's answer, some factors that can help to reduce / mitigate the risk of overfitting to commonly-used benchmarks as a research community are:

  1. Competitions with instances of problems hidden from entrants: as far as I'm aware this is particularly popular in game AI (see the General Game Playing competition and General Video Game Playing competitions). The basic idea is that submissions should be able to tackle a relatively broad class of problems (playing any game defined in a specified format, or generating levels for any video game with rules described in a specific format, etc.). To some extent, using a large suite of problems as a standard benchmark (such as the large collection of Atari games supported by ALE) also fits in with this idea, though there is value in hiding the problems that are ultimately used for testing from the people writing submissions. Of course, the idea is that entries submitted to these kinds of competitions will involve new research which may be published.

  2. Using very simple toy problems: With simple I do not necessarily mean that they are simple to solve, but simple to describe / understand (it may still, for example, have a large state space and be difficult for current techniques to solve). Simple toy problems often help to test for a very specific "skill", and can more easily give insight into specifically why/when an algorithm may be expected to fail or succeed. Of course, large non-toy problems are also important to demonstrate "real-world" usefulness of algorithms, but they may often give less understanding / insight into an algorithm.

  3. Theoretical work: Theoretical work can also give more insight and understanding of new algorithms. Algorithms with strong theoretical foundations are often more likely to generalize to a multitude of problem domains, assuming that the initial assumptions hold (big assumption here - there are plenty of cases where assumptions required for strong proofs do not hold!). This is not always possible / "needed", sometimes new research based purely on intuition and with relatively little theoretical foundations still turn out to work well (or theory is only developed after promising empirical results)... but it can certainly help. Theoretical work can take many different forms, such proofs of convergence (often under strict conditions), proofs for upper or lower bounds on important measures (such as regret, or probability of making a "wrong" choice, etc.), proofs that an algorithm or a problem is a more general or more specific case of an existing, well-understood algorithm or problem, proofs that a model has or does not have a certain representational capacity, proofs of algorithmic equivalence (that an algorithm computes exactly the same quantities as another well-understood algorithm, typically with lower computation and/or memory requirements), etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.