1
$\begingroup$

loss curve

Why is my loss curve consistently oscillating? Every 50 steps it jumps back up. I'm assumming there's a bug in my data, since I'm using this colab notebook that shows a proper train/loss at the bottom. The only thing I changed was the dataset used. I'm using the falcon-7b-sharded-bf16 huggingface model.

$\endgroup$
1
  • $\begingroup$ probably stepsize scheduling? $\endgroup$
    – Alberto
    Aug 17, 2023 at 1:11

1 Answer 1

1
$\begingroup$

Based on the colab notebook that you used, the periodic oscillation of the training loss does not seem to originate from the learning rate scheduler which is set to "constant". However, I found that setting group_by_length=True (link) in the trainer can result in periodic oscillation of the training loss.

Below are 2 training loss examples for the same model/same dataset finetuned with: group_by_length=True1 group_by_length=True

group_by_length=False2 group_by_length=False

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .