8
$\begingroup$

I trained a ResNet20 on Cifar10 and obtained the following learning curves.

enter image description here

From the figures, I see at epoch 52, my validation loss is 0.323 (the lowest), and my validation accuracy is 89.7%.

On the other hand, at the end of the training (epoch 120), my validation loss is 0.413 and my validation accuracy is 91.3% (the highest).

Say I'd like to deploy this model on some real-world application. Should I prefer the snapshotted model at epoch 52, the one with lowest validation loss, or the model obtained at the end of training, the one with highest validation accuracy?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
1
$\begingroup$

Okay, I think it's better if we distinguish loss and accuracy first via Jeremy's answer, and I agree with him with the sentence "low or huge loss is a subjective metric".

The loss value is easy to affect by noise from data and significant increase with a few error data points. My advice in this case is to use more evaluation metrics, and understand correctly what you need from your model.

For example, with Cifar 10, and you need the more correct label the better, you can believe in accuracy. However, if you want your model to make sure its result is the correct, area under receiver operating characteristic curve (AUROC) maybe the better choice.

For example, classification problem with 3 class, correct label y = 1:

  • Good accuracy, bad AUROC: the output probability from softmax [0.3,0.4,0.3]
  • Good accuracy, good AUROC: [0.1,0.8,0.1]

And with imbalanced dataset, Precision, Recall and F1-score will be more suitable.

Improve this answer
$\endgroup$
1
$\begingroup$

In highly imbalanced classification problems, the highest accuracy can often be achieved simply by assigning the majority class to all observations. This is why learning algorithms do not maximize classification accuracy but minimize a loss function. Fundamentally, loss functions capture how much you "lose" when there is a difference between the statistic you want to estimate and the estimate itself. The appropriate loss function is not given by nature, but is provided by you.

With this in mind, it is not quite accurate to say that the highest accuracy is reached after 120 epochs: it is merely the maximum accuracy achieved by the algorithm so far. Unless you run the algorithm for longer, there is no way to say if this is even a local maximum. For example, assigning every observation to the majority class may well achieve a higher accuracy than that achieved at 120 epochs. The only significance of the 120 epochs is therefore that that is how long you ran the algorithm for.

Given these considerations, it makes far more sense to stop at around 50 epochs, when your loss function is minimized.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Hi, you are right regarding the points for imbalanced datasets but Cifar10 isn't quite imbalanced. More or less, there are same amount of samples for each class. $\endgroup$
    – SpiderRico
    Commented Feb 12, 2020 at 17:15
  • $\begingroup$ It doesn't matter. What matters is that learning involves minimizing an appropriately chosen loss function. Unless you choose the loss function to be 1 - accuracy, minimizing losses does not correspond to maximizing accuracy. $\endgroup$
    – perenniallydisappointed
    Commented Feb 12, 2020 at 17:43
  • $\begingroup$ but your model tries to minimize the training loss, not the validation loss? $\endgroup$
    – SpiderRico
    Commented Feb 12, 2020 at 17:51
  • $\begingroup$ That's right. You want to minimize loss with respect to the data generating process, not with respect to the data itself. In other words: the model shouldn't just fit the data you already have, but also the data that you will get in the future. You can assess how well it does this on a test set. $\endgroup$
    – perenniallydisappointed
    Commented Feb 13, 2020 at 8:55
  • $\begingroup$ What about a better metric like F1? Would you ever prefer it over validation loss to save the best model? $\endgroup$
    – rjurney
    Commented Jun 25 at 19:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .