I am currently using a loss averaged over the last 100 iterations, but this leads to artifacts like the loss going down even when the current iteration has an average loss, because the loss 100 iterations ago was a large outlier.
I thought about using different interval lengths, but I wonder if an average over the last few iterations really is the right way to plot the loss.
Are there common alternatives? Maybe using decaying weights in the average? What are the best practices for visualizing the loss?
2.6
and3.2
in the last 100 iterations with an average of2.92
. As the scatter plot is almost useless to see the trend, I visualize the average as well. $\endgroup$