0
$\begingroup$

I am working on a project to evaluate various fine-tuned LLMs. Unfortunately inference is prohibitively slow, and I don't think I will be able to test my models on the full test set of 40,000 samples. My current plan is to implement an early stopping threshold.

Is there an "accepted" or theoretical minimum number of samples that are required to benchmark an LLM?

$\endgroup$

1 Answer 1

1
$\begingroup$

The minimum number of test items is a function of the statistical power you want to reach so that you can reliably detect differences of a given magnitude. A small number of test items (a small sample size) can be acceptable for some purposes but be too small for other purposes.

In some fields of research, there are traditionally accepted test set sizes, for example 3000 items. In such fields, you have a good chance of getting your paper published (and pass your PhD exam) with a test set of such a size even if some publications have shown that the size is too small for the task.

If your task is to pick the best model from a large set of candidate models with a limited inference budget, you could first test all models on a small subset of the test set, select the best models (the optimal number of models to select depends on many factors) according to this incomplete test, then test the selected models with more data and repeat the process. Whether the final round uses the full test set and how many models should be in the final is also hard to say. Ideally, you'd test some candidate selection procedures using small, inexpensive LMs and then scale up to your LLMs. An ad-hoc approach like going from 1000 models to 100 models using a test set of 1000 items, then 100 to 10 using 7000 items and finally 10 to 1 using all 40k items can be acceptable in many contexts.

Suggested reading:

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .