# How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function?

I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:

The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.

(You can ignore the missing part, I just did not save the loss values for these epochs)

Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?

• Learning rate parameter: 2.5e-4
• Training data size: 395 MB

GPT-2 parameters:

{
"n_vocab": 50000,
"n_ctx": 1024,
"n_embd": 768,