When exactly am I overfitting -- contradicting metrics

Question

I am training an object detection machine learning pipeline. Among the many metrics provided out of the box by tensorflow object detection API, I look at total_loss and DetectionBoxes_Precision/[email protected]:

Here the x-axis is the number of steps i.e. model experience. The orange line is for the training data and the blue line is for the validation data.

From the loss graph I would conclude, that at approx 2k steps overfitting starts, so using the model at approx 2k steps would be the best choice. But looking at the precision graph, training e.g. until 24k steps would be a much better model. Which one is the best model?

Here, loss and the precision metric where picked just for illustrating the dilemma, there are many more metrics available, leading to multiple conclusions about when overfitting actually starts.

Answer 1

From the loss graph I would conclude, that at approx 2k steps overfitting starts, so using the model at approx 2k steps would be the best choice. But looking at the precision graph, training e.g. until 24k steps would be a much better model. Which one is the best model?

Different metrics will lead you to different conclusions. To detect overfitting under any metric you do need to plot training and validation/dev of the same metric, so it is harder to spot on your second graph.

The reason why loss/cost functions and other metrics can disagree on what the best model is, is due to sensitivity to different types of error. Often loss functions will be sensitive to outliers - large differences in predictions versus ground truth in the training or validation data. So it is common to see loss diverging between validation and train first - probably due to a very few bad predictions caused by difficult edge cases that the model is not coping well with - whilst metrics such as accuracy may continue to improve on validation and test sets.

The way to resolve this is to pick the metric that is important to you for business reasons (ideally ahead of the training, so you are not biased in your own decision about what to do), and use that to make judgement calls. This is easier to rely on the larger and cleaner your dataset is.

One other thing you can do is to some error analysis - find the worst cases for loss function, and check that they are labelled correctly.

If things still look inconclusive to you, and dataset size is small enough that results are noisy, then consider using k-fold cross validation to reduce effects of sample bias. This takes more time, but may improve your confidence in your model selection process.

Answer 2

Generally, a model is considered to be overfitted when there's a huge gap between training and test/validation performances.

So during the training, you monitor the loss on validation data, and training data, and stop training if validation loss stagnates/increases given the training loss keeps decreasing.

In your scenario, I'm not sure what the total loss metric corresponds to. As I said, you've to measure loss on a held-out data other than training data to detect, and prevent overfitting.