5

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In Markovian Decision Processes (MDPs) the whole state space is known, this means you know all the information for your problem; therefore, you can use them to find ...


3

In addition to the points already listed in John's answer, some factors that can help to reduce / mitigate the risk of overfitting to commonly-used benchmarks as a research community are: Competitions with instances of problems hidden from entrants: as far as I'm aware this is particularly popular in game AI (see the General Game Playing competition and ...


3

Great question Dennis! This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year. Usually these discussions cover a number of themes: "Existing benchmarks suck". ...


2

These are some examples of simple RTS games I know of: ELF Mini-RTS (code): It is very fast; the game environment runs 40,000 frames per second per core on a Macbook Pro. It captures the key dynamics of a real-time strategy game: Both players gather resources, build facilities, explore unknown territory (terrain that is out of sight of the player), and ...


2

This is an interesting question. One good place to start would be the gargantuan catalogue of rather simple RTS flash games and the like. There are many sites that have 100s of such titles, and it would not be too difficult to build the framework for these types of games(there are open source tools that can help as well). As far as "real" mainstream games, ...


1

Some of the domains in the International Probabilistic Planning Competition, such as the Wildlife Preserve benchmark, fit quite well the constraints you have given. Note that the problems are modeled with a high-level declarative language, RDDL. This means that you can define problems as big or as small as your heart desires with relative ease, since you can ...


1

https://github.com/jcjohnson/cnn-benchmarks might be a good start. It mostly focuses on GPUs, but there is also one CPU (Dual Xeon E5-2630 v3).


Only top voted, non community-wiki answers of a minimum length are eligible