Reference request: data efficiency of LLM pre-training

Question

I've seen it stated multiple times that LLMs have much worse data efficiency than humans (IE require more data to reach same or worse performance), EG this Tweet by Yann LeCun, or 19:30 in this talk by Michael Wooldridge. Are there any papers (preferably published but otherwise pre-prints) that really explore data-efficiency of LLM pre-training (not ICL, I'm happy to accept that's data-efficient), and possibly compare against an approximate/upper-limit human benchmark?

Preferably looking for papers with thorough exploration and evaluation of data-efficiency of existing LLMs, rather than papers proposing a new method for improving data-efficiency without thorough comparison to existing SOTA models.

I'd like to know so hopefully I can cite such a paper in future work of my own.

Answer 1

This article, cited on Reddit, provides an answer:

Frank, Michael C. "Bridging the data gap between children and large language models." Trends in Cognitive Sciences (2023).:

GPT-3 was trained on 5x10^11 tokens [2] and Chinchilla was trained on 1012 tokens [1]. Many companies keep training set sizes secret, but a recent leak suggested that one industry model was trained on 3.6x10^12 tokens. How do these numbers compare with human language experience? Comprehensive word counts are difficult to collect but sampling and extrapolation can provide reasonable upper and lower bounds for language input (Figure 1).A soft upper bound on a child’s linguistic input – language produced by the people around them – is around 106 words per month [3], [4]. For a five-year-old, that would be 6x10^7 words; for a 20-year-old, 2x10^8 words. We also might assume that a 20-year-old has been reading for 10-15 years, and for much of this time they are reading 2-3 books (105 words each) per week for an extra ~107 words per year. Our rough upper bound for a literate 20-year-old could be as high as 4x10^8 words (or even higher if they read constantly).

Additionally:

1 token ~= ¾ words

So converting tokens to words wouldn't change the order of magnitude.