Skip to main content

Questions tagged [benchmarks]

For questions related to AI benchmarks--results that validate a specific technique or approach. Also for question regarding the history of AI achievements, and predictions as to future achievements.

Filter by
Sorted by
Tagged with
0 votes
1 answer

Minimum Number of Samples for LLM Benchmark?

I am working on a project to evaluate various fine-tuned LLMs. Unfortunately inference is prohibitively slow, and I don't think I will be able to test my models on the full test set of 40,000 samples. ...
Jonathan Emmons's user avatar
0 votes
2 answers

Is it legitimate to train a model on a benchmark dataset and use this model only for labeling another datasets

To assess our deep learning models (CNN) we have labeled a big benchmark dataset (it was labeled by specialists so it is kind of Ideal). I of course know that we do not want to train new models using ...
Igor's user avatar
  • 181
0 votes
0 answers

Intentionally corrupting LLM weights (lobotomy)

It is largely unknown how LLMs work inside. Has anyone scientifically tried to corrupt (open source) model's weights in an organized manner to maybe detect which parts of the model are doing what or ...
Kozuch's user avatar
  • 281
2 votes
1 answer

LLM Hallucinations on In-Context Data

I know of some benchmarks that LLMs do undergo, but I am no expert whatsoever. I think what I am wondering about is closest to TruthfulQA. The question came up when I heard of combining company data ...
Mathy's user avatar
  • 143
1 vote
1 answer

How do Multimodal LLMs of 2023 score on the ARC benchmark (in 2020: 20% Accuracy)

Q: I wonder if anyone has tried to solve the ARC tasks with one of the state-of-the-art Multimodal LLMs? Can LLMs that can process graphics input do the following? (This question is not about the The ...
knb's user avatar
  • 165
0 votes
0 answers

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
Julius Hamilton's user avatar
24 votes
5 answers

Why does ChatGPT fail in playing "20 questions"?

IBM Watson's success in playing "Jeopardy!" was a landmark in the history of artificial intelligence. In the seemingly simpler game of "Twenty questions" where player B has to ...
Hans-Peter Stricker's user avatar
1 vote
1 answer

Is there a benchmark for multi-objective evolutionary algorithms?

I'm working on a project for an evolutionary algorithms course, and the problem we're trying to solve is multi-objective. We'll use NSGA-II but we also wanted to compare with some other MOEAs, however,...
Matías Santurio's user avatar
1 vote
0 answers

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
Jakub's user avatar
  • 11
1 vote
0 answers

What is the efficiency of trained neural networks?

Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)? I understand that this is ...
Anton's user avatar
  • 111
1 vote
0 answers

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
ASA's user avatar
  • 151
6 votes
1 answer

Interesting examples of discrete stochastic games

SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'...
user76284's user avatar
  • 347
2 votes
0 answers

NLP annotation tool online and other tools to compare performances of different NLP algorithms

I do text annotations (POS tagging, NER, chunking, synset) by using a specific annotation tool for Natural Language Processing. I would like to make the same annotations on different tools to compare ...
franz1's user avatar
  • 173
6 votes
1 answer

Benchmarks for reinforcement learning in discrete MDPs

To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
user76284's user avatar
  • 347
6 votes
2 answers

What are the most compact Real Time-Strategy Games?

There was a recent informal question on chat about RTS games suitable for AI benchmarks, and I thought it would be useful to ask a question about them in relation to AI research. Compact is defined ...
DukeZhou's user avatar
  • 6,237
11 votes
3 answers

How can AI researchers avoid "overfitting" to commonly-used benchmarks as a community?

In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the ...
Dennis Soemers's user avatar
  • 10.3k
2 votes
1 answer

Are there benchmarks for assessing the speed of the forward-pass of neural networks?

I have a task where I would like to use a convolutional neural network (CNN). I would like to incrementally start from the fastest models, fine-tune and see whether they fit my "budget". At the moment,...
The Nomadic Coder's user avatar
4 votes
1 answer

Why is chess still a benchmark for Artificial Intelligence?

Even though modern chess playing programs have demonstrated themselves to be as strong (or stronger) than even the best human players for nearly 20 years now (1997 when IBM's Deep Blue defeated the ...
DJ2's user avatar
  • 143