Questions tagged [benchmarks]

For questions related to AI benchmarks--results that validate a specific technique or approach. Also for question regarding the history of AI achievements, and predictions as to future achievements.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

3-Body Gravitation Lander Game?

Has anyone tried writing a 3-body gravitation tracker (lander would be too much to ask) game to see how various AIs would handle a chaotic dynamical system?
1
vote
0answers
21 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
6
votes
1answer
105 views

Interesting examples of discrete stochastic games

SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'...
2
votes
0answers
16 views

NLP annotation tool online and other tools to compare performances of different NLP algorithms

I do text annotations (POS tagging, NER, chunking, synset) by using a specific annotation tool for Natural Language Processing. I would like to make the same annotations on different tools to compare ...
6
votes
1answer
142 views

Benchmarks for reinforcement learning in discrete MDPs

To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
5
votes
2answers
133 views

What are the most compact Real Time-Strategy Games?

There was a recent informal question on chat about RTS games suitable for AI benchmarks, and I thought it would be useful to ask a question about them in relation to AI research. Compact is defined ...
7
votes
2answers
111 views

How can AI researchers avoid “overfitting” to commonly-used benchmarks as a community?

In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the ...
2
votes
1answer
78 views

Are there benchmarks for assessing the speed of the forward-pass of neural networks?

I have a task where I would like to use a convolutional neural network (CNN). I would like to incrementally start from the fastest models, fine-tune and see whether they fit my "budget". At the moment,...