What does it mean if classification error is equal between two networks but the MSE is different?

Question

I'm experimenting with training a feedforward neural network using a genetic algorithm and I've done a few tests using both the mean squared error and classification error functions as fitness heuristic in the GA.

When I use MSE as error function, my GA tends to converge around an MSE of 0.1 (initial conditions have an MSE of around 0.9). Testing system accuracy with this network gives me 95%+ for both training and testing data.

But, when I use classification error as my heuristic, my GA tends to converge around when the MSE is about 0.3. System accuracy is still around the same at 95%+.

My question is, if you had two networks, one showing an MSE of 0.1 and one an MSE of 0.3, but both perform approximately the same in terms of accuracy, what can I deduce from the differences in MSE?

In other words: which network is "better", even if the accuracy is the same? Does a lower MSE mean anything below a certain amount? I could train my network for 100x as many generations and get a better MSE but not necessarily a better accuracy. Why?

For some context:

When the MSE is approximately 1.5 (epoch 250), the accuracy seems to match when the MSE is approximately 2.0 (epoch 50). Why does the accuracy not increase despite MSE decreasing?

Answer 1

MSE just measures the squared difference between actual and target values. It can still correctly classify the values, but perhaps not with the same confidence - leading to a higher loss (e.g. an output of 0.77 vs 0.98 when the target is 1). In terms of which is better, I wouldn't know without the specifics of your problem. It is possible the higher loss could be more robust since it is less likely to have overfitted on the data, yet achieves the same accuracy.

Answer 2

Accuracy itself isn't a sufficient way to compare two models. For example, you need to consider the precision and recall stats (see confusion matrix) and calculate some other metrics like f1 score. The measurement of accuracy is only the initial state that helps us to know if a model is "working". But in order to understand and compare you need to know how many of impostors' set were classified as true claimants, and how many true-claimants' set classified as impostors according to the sum of total correct classified ones. In order to make a decision with the above been known you have to define how critical a miss-classification could be? e.g. assume that you need to classify if a person has a disease or not? (that's critical).