2
$\begingroup$

I trained a ResNet20 on Cifar10 and obtained the following learning curves.

enter image description here

From the figures, I see at epoch 52, my validation loss is 0.323 (the lowest), and my validation accuracy is 89.7%.

On the other hand, at the end of the training (epoch 120), my validation loss is 0.413 and my validation accuracy is 91.3% (the highest).

Say I'd like to deploy this model on some real-world application. Should I prefer the snapshotted model at epoch 52, the one with lowest validation loss, or the model obtained at the end of training, the one with highest validation accuracy?

$\endgroup$
-1
$\begingroup$

In highly imbalanced classification problems, the highest accuracy can often be achieved simply by assigning the majority class to all observations. This is why learning algorithms do not maximize classification accuracy but minimize a loss function. Fundamentally, loss functions capture how much you "lose" when there is a difference between the statistic you want to estimate and the estimate itself. The appropriate loss function is not given by nature, but is provided by you.

With this in mind, it is not quite accurate to say that the highest accuracy is reached after 120 epochs: it is merely the maximum accuracy achieved by the algorithm so far. Unless you run the algorithm for longer, there is no way to say if this is even a local maximum. For example, assigning every observation to the majority class may well achieve a higher accuracy than that achieved at 120 epochs. The only significance of the 120 epochs is therefore that that is how long you ran the algorithm for.

Given these considerations, it makes far more sense to stop at around 50 epochs, when your loss function is minimized.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, you are right regarding the points for imbalanced datasets but Cifar10 isn't quite imbalanced. More or less, there are same amount of samples for each class. $\endgroup$ – SpiderRico Feb 12 at 17:15
  • $\begingroup$ It doesn't matter. What matters is that learning involves minimizing an appropriately chosen loss function. Unless you choose the loss function to be 1 - accuracy, minimizing losses does not correspond to maximizing accuracy. $\endgroup$ – perenniallydisappointed Feb 12 at 17:43
  • $\begingroup$ but your model tries to minimize the training loss, not the validation loss? $\endgroup$ – SpiderRico Feb 12 at 17:51
  • $\begingroup$ That's right. You want to minimize loss with respect to the data generating process, not with respect to the data itself. In other words: the model shouldn't just fit the data you already have, but also the data that you will get in the future. You can assess how well it does this on a test set. $\endgroup$ – perenniallydisappointed Feb 13 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.