# How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function?

I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:

The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.

(You can ignore the missing part, I just did not save the loss values for these epochs)

Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?

• Learning rate parameter: 2.5e-4
• Training data size: 395 MB

GPT-2 parameters:

{
"n_vocab": 50000,
"n_ctx": 1024,
"n_embd": 768,
"n_layer": 12
}

• what is your learning rate/ dataset size – mshlis Aug 8 '19 at 18:07
• @mshlis I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it? – allo Aug 8 '19 at 18:10
• no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right? – mshlis Aug 8 '19 at 18:12
• No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model. – allo Aug 8 '19 at 18:17
• in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data – mshlis Aug 8 '19 at 18:28