3
$\begingroup$

I am currently using a loss averaged over the last 100 iterations, but this leads to artifacts like the loss going down even when the current iteration has an average loss, because the loss 100 iterations ago was a large outlier.

I thought about using different interval lengths, but I wonder if an average over the last few iterations really is the right way to plot the loss.

Are there common alternatives? Maybe using decaying weights in the average? What are the best-practices for visualizing the loss?

| improve this question | |
$\endgroup$
  • $\begingroup$ Why do you need to average the loss? Loss is suppose to decrease over time, so getting the average of loss over time is meaningless $\endgroup$ – Clement Hui Jan 4 at 12:01
  • 1
    $\begingroup$ The variance of the loss per iteration is a lot larger than the decrease of the loss between the iterations. For example I currently have a loss between 2.6 and 3.2 in the last 100 iterations with an average of 2.92. As the scatter plot is almost useless to see the trend, I visualize the average as well. $\endgroup$ – allo Jan 4 at 12:40
  • $\begingroup$ Oh. Perhaps yous re looking for this: en.wikipedia.org/wiki/Moving_average#Exponential_moving_average $\endgroup$ – Clement Hui Jan 4 at 12:49
1
$\begingroup$

You can use the Exponential Moving Average method. This method is used in tensorbaord as a way to smoothen a loss curve plot. enter image description here The algorithm is as follow: enter image description here However there is a small problem doing it this way. As you can see S_t is initialized with the starting value, which makes the starting curve inaccurate. enter image description here The green curve is the ideal curve for the algorithm, but the purple curve is the predicted curve. The curve is not correct on the start. To solve this, a correction factor is added in, thus making the algorithm this: enter image description here This introduces WeightedCount which decreases over time to 0.

Exponential Moving Average is also used is other areas of deep learning, the most notable being some optimization algorithms. It is used in Adam, RMSProp and other similar optimizers to smooth out the gradients to make the path to minimal loss a more direct and straightforward path.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.