How do I interpret a large variance of a loss function?

I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this: a plot of the loss function

The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.

(You can ignore the missing part, I just did not save the loss values for these epochs)

Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?

  • Learning rate parameter: 2.5e-4
  • Training data size: 395 MB

GPT-2 parameters:

  "n_vocab": 50000,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12
  • $\begingroup$ what is your learning rate/ dataset size $\endgroup$ – mshlis Aug 8 at 18:07
  • $\begingroup$ @mshlis I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it? $\endgroup$ – allo Aug 8 at 18:10
  • $\begingroup$ no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right? $\endgroup$ – mshlis Aug 8 at 18:12
  • $\begingroup$ No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model. $\endgroup$ – allo Aug 8 at 18:17
  • $\begingroup$ in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data $\endgroup$ – mshlis Aug 8 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.