Questions tagged [benchmarks]

For questions related to AI benchmarks--results that validate a specific technique or approach. Also for question regarding the history of AI achievements, and predictions as to future achievements.

Filter by
Sorted by
Tagged with
0
votes
0answers
62 views

Are there any reinforcement learning benchmarks where the optimal policy is known for each environment?

There are multiple reinforcement learning (RL) benchmarks (i.e. a set of environments where we can test our RL algorithms), for example, the DeepMind Control Suite. However, given that I am currently ...
1
vote
0answers
17 views

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
1
vote
0answers
50 views

What is the efficiency of trained neural networks?

Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)? I understand that this is ...
1
vote
0answers
57 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
6
votes
1answer
122 views

Interesting examples of discrete stochastic games

SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'...
2
votes
0answers
19 views

NLP annotation tool online and other tools to compare performances of different NLP algorithms

I do text annotations (POS tagging, NER, chunking, synset) by using a specific annotation tool for Natural Language Processing. I would like to make the same annotations on different tools to compare ...
6
votes
1answer
216 views

Benchmarks for reinforcement learning in discrete MDPs

To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
6
votes
2answers
159 views

What are the most compact Real Time-Strategy Games?

There was a recent informal question on chat about RTS games suitable for AI benchmarks, and I thought it would be useful to ask a question about them in relation to AI research. Compact is defined ...
9
votes
2answers
166 views

How can AI researchers avoid “overfitting” to commonly-used benchmarks as a community?

In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the ...
2
votes
1answer
123 views

Are there benchmarks for assessing the speed of the forward-pass of neural networks?

I have a task where I would like to use a convolutional neural network (CNN). I would like to incrementally start from the fastest models, fine-tune and see whether they fit my "budget". At the moment,...
4
votes
1answer
202 views

Why is chess still a benchmark for Artificial Intelligence?

Even though modern chess playing programs have demonstrated themselves to be as strong (or stronger) than even the best human players for nearly 20 years now (1997 when IBM's Deep Blue defeated the ...