I'm experimenting with training a feedforward neural network using a genetic algorithm and I've done a few tests using both the mean squared error and classification error functions as fitness heuristic in the GA.
When I use MSE as error function, my GA tends to converge around an MSE of 0.1 (initial conditions have an MSE of around 0.9). Testing system accuracy with this network gives me 95%+ for both training and testing data.
But, when I use classification error as my heuristic, my GA tends to converge around when the MSE is about 0.3. System accuracy is still around the same at 95%+.
My question is, if you had two networks, one showing an MSE of 0.1 and one an MSE of 0.3, but both perform approximately the same in terms of accuracy, what can I deduce from the differences in MSE?
In other words: which network is "better", even if the accuracy is the same? Does a lower MSE mean anything below a certain amount? I could train my network for 100x as many generations and get a better MSE but not necessarily a better accuracy. Why?
For some context:
When the MSE is approximately 1.5 (epoch 250), the accuracy seems to match when the MSE is approximately 2.0 (epoch 50). Why does the accuracy not increase despite MSE decreasing?