Recently I developed a custom training algorithm for deep learning models, based on evolutionary algorithms. Details are not important, except that it also uses decreasing regular cross entropy loss as its fitness function.
What I observed is that it very well decreases the loss function but the classification metrics such as accuracy, precision or recall also decrease along the training. This got me confused, as I was sure that decreasing loss such as cross entropy should always increase these metrics. After researching I found out this is possible due to fact that cross entropy can decrease in case where confidence on few samples is greatly increasing, but many other samples are meanwhile getting incorrect scores, but the profit from these few correct ones are dominant over many incorrect ones: https://www.quora.com/What-is-the-matter-when-loss-decreases-and-accuracy-decreases-too-on-training-neural-network?top_ans=238980470
So my question is: is there a loss function that when decreasing will always be increasing classification metrics?
