1
$\begingroup$

Recently I developed a custom training algorithm for deep learning models, based on evolutionary algorithms. Details are not important, except that it also uses decreasing regular cross entropy loss as its fitness function.

What I observed is that it very well decreases the loss function but the classification metrics such as accuracy, precision or recall also decrease along the training. This got me confused, as I was sure that decreasing loss such as cross entropy should always increase these metrics. After researching I found out this is possible due to fact that cross entropy can decrease in case where confidence on few samples is greatly increasing, but many other samples are meanwhile getting incorrect scores, but the profit from these few correct ones are dominant over many incorrect ones: https://www.quora.com/What-is-the-matter-when-loss-decreases-and-accuracy-decreases-too-on-training-neural-network?top_ans=238980470

So my question is: is there a loss function that when decreasing will always be increasing classification metrics?

Improve this question
$\endgroup$
1
  • $\begingroup$ Does not answer the question (which I think is actually ill-posed, for reasons I hint at at the end of this answer of mine at SO), but may be helpful: stackoverflow.com/questions/47817424/… $\endgroup$
    – desertnaut
    Sep 24, 2022 at 23:59

1 Answer 1

Reset to default
2
$\begingroup$

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE).

Further, metrics like accuracy use thresholds that can prove problematic, versus evaluating the probability outputs directly. This is discussed extensively on the statistics Stack, Cross Validated. I will leave a few links.

https://stats.stackexchange.com/a/312787/247274

https://stats.stackexchange.com/a/359936/247274

https://stats.stackexchange.com/a/312124/247274

However, even strictly proper scoring rules do not have to move together. Since, for instance, Brier score and log loss (both of which are examples of the strictly proper scoring rules discussed in the first link) are measures of different quantities, this is desirable. Depending on what we value, we might prefer one over the other.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .