Questions tagged [benchmarks]

For questions related to AI benchmarks--results that validate a specific technique or approach. Also for question regarding the history of AI achievements, and predictions as to future achievements.

Filter by
Sorted by
Tagged with
0
votes
0answers
61 views

Are there any reinforcement learning benchmarks where the optimal policy is known for each environment?

There are multiple reinforcement learning (RL) benchmarks (i.e. a set of environments where we can test our RL algorithms), for example, the DeepMind Control Suite. However, given that I am currently ...
1
vote
0answers
17 views

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
1
vote
0answers
50 views

What is the efficiency of trained neural networks?

Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)? I understand that this is ...
0
votes
0answers
20 views

3-Body Gravitation Lander Game?

Has anyone tried writing a 3-body gravitation tracker (lander would be too much to ask) game to see how various AIs would handle a chaotic dynamical system?
1
vote
0answers
51 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
6
votes
1answer
119 views

Interesting examples of discrete stochastic games

SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'...
2
votes
0answers
19 views

NLP annotation tool online and other tools to compare performances of different NLP algorithms

I do text annotations (POS tagging, NER, chunking, synset) by using a specific annotation tool for Natural Language Processing. I would like to make the same annotations on different tools to compare ...
6
votes
1answer
198 views

Benchmarks for reinforcement learning in discrete MDPs

To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
6
votes
2answers
151 views

What are the most compact Real Time-Strategy Games?

There was a recent informal question on chat about RTS games suitable for AI benchmarks, and I thought it would be useful to ask a question about them in relation to AI research. Compact is defined ...
8
votes
2answers
158 views

How can AI researchers avoid “overfitting” to commonly-used benchmarks as a community?

In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the ...
2
votes
1answer
111 views

Are there benchmarks for assessing the speed of the forward-pass of neural networks?

I have a task where I would like to use a convolutional neural network (CNN). I would like to incrementally start from the fastest models, fine-tune and see whether they fit my "budget". At the moment,...