# What is the best way to smoothen out a loss curve plot

I am currently using a loss averaged over the last 100 iterations, but this leads to artifacts like the loss going down even when the current iteration has an average loss, because the loss 100 iterations ago was a large outlier.

I thought about using different interval lengths, but I wonder if an average over the last few iterations really is the right way to plot the loss.

Are there common alternatives? Maybe using decaying weights in the average? What are the best-practices for visualizing the loss?

• Why do you need to average the loss? Loss is suppose to decrease over time, so getting the average of loss over time is meaningless Jan 4 '20 at 12:01
• The variance of the loss per iteration is a lot larger than the decrease of the loss between the iterations. For example I currently have a loss between 2.6 and 3.2 in the last 100 iterations with an average of 2.92. As the scatter plot is almost useless to see the trend, I visualize the average as well.
– allo
Jan 4 '20 at 12:40
• Oh. Perhaps yous re looking for this: en.wikipedia.org/wiki/Moving_average#Exponential_moving_average Jan 4 '20 at 12:49

You can use the Exponential Moving Average method. This method is used in tensorbaord as a way to smoothen a loss curve plot. The algorithm is as follow: However there is a small problem doing it this way. As you can see S_t is initialized with the starting value, which makes the starting curve inaccurate. The green curve is the ideal curve for the algorithm, but the purple curve is the predicted curve. The curve is not correct on the start. To solve this, a correction factor is added in, thus making the algorithm this: This introduces WeightedCount which decreases over time to 0.

Exponential Moving Average is also used is other areas of deep learning, the most notable being some optimization algorithms. It is used in Adam, RMSProp and other similar optimizers to smooth out the gradients to make the path to minimal loss a more direct and straightforward path.