0
$\begingroup$

DeepMind released the Training Compute-Optimal Large Language Models paper in 2022 which describe some scaling laws for LLMs. As far as I understand this is the most accredited reference to estimate the optimal relation between dataset size, compute power and model size.

Recently a number of models have been developed using far less data, parameters and compute than the bigger LLMs. Yet these models achieved great results thanks to much better data quality. For instance models like WizardLM, TinyStories and phi-1. The recent article published on nature Human-like systematic generalization through a meta-learning neural network seems to imply that data quality is the key.

I'm curious about what role the data quality plays in the training of LLMs. I'm wondering things like: is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too? Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?

$\endgroup$

1 Answer 1

0
$\begingroup$

is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too?

This is an open research question, but some research such as Beyond neural scaling laws: beating power law scaling via data pruning and Tiny Data seem to imply that higher quality data gives better scaling laws than Chinchilla estimates.

Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?

Not yet!

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .