I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. However, I am aware that in some contexts, this may lead to overfitting or other issues.

What is the general consensus on the optimal quantity of training data required for fine-tuning a Large Language Model? Is bigger always better?

  • $\begingroup$ Are you pretraining or fine-tuning? $\endgroup$ Aug 4, 2023 at 9:55
  • $\begingroup$ Fine-tuning a pretrained model to get better on a task. $\endgroup$
    – Peyman
    Aug 4, 2023 at 11:18


You must log in to answer this question.