6

Chess isn't really a benchmark per say. The method developed in AlphaGo to play Go should in principle generalize quite nicely to other games of this sort, such as chess. Since Stockfish is quite dominantly the strongest Chess AI, the natural question would be to see how well AlphaGo's method compares to Stockfish. Being one of the most well developed AI ...


5

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In Markovian Decision Processes (MDPs) the whole state space is known, this means you know all the information for your problem; therefore, you can use them to find ...


4

Great question Dennis! This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year. Usually these discussions cover a number of themes: "Existing benchmarks suck". ...


3

These are some examples of simple RTS games I know of: ELF Mini-RTS (code): It is very fast; the game environment runs 40,000 frames per second per core on a Macbook Pro. It captures the key dynamics of a real-time strategy game: Both players gather resources, build facilities, explore unknown territory (terrain that is out of sight of the player), and ...


3

This is an interesting question. One good place to start would be the gargantuan catalogue of rather simple RTS flash games and the like. There are many sites that have 100s of such titles, and it would not be too difficult to build the framework for these types of games(there are open source tools that can help as well). As far as "real" mainstream games, ...


3

In addition to the points already listed in John's answer, some factors that can help to reduce / mitigate the risk of overfitting to commonly-used benchmarks as a research community are: Competitions with instances of problems hidden from entrants: as far as I'm aware this is particularly popular in game AI (see the General Game Playing competition and ...


1

Some of the domains in the International Probabilistic Planning Competition, such as the Wildlife Preserve benchmark, fit quite well the constraints you have given. Note that the problems are modeled with a high-level declarative language, RDDL. This means that you can define problems as big or as small as your heart desires with relative ease, since you can ...


1

https://github.com/jcjohnson/cnn-benchmarks might be a good start. It mostly focuses on GPUs, but there is also one CPU (Dual Xeon E5-2630 v3).


Only top voted, non community-wiki answers of a minimum length are eligible