How to express accuracy of a regression ANN that uses MSE loss function?

Question

I have a regression MLP network with all input values between 0 and 1, and am using MSE for the loss function. The minimum MSE over the validation sample set comes to 0.019. So how to express the 'accuracy' of this network in 'lay' terms? If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))*100 = 86.2% accurate"?

Also, in the validation data set, there are three 'extreme' expected values. The lowest MSE results in predicted values closer to these three values, but not as close to all the other values, whereas a slightly higher MSE results in the opposite - predicted values further from the 'extreme' values but more accurate relative to all other expected values (and this outcome is actually preferred in the case I'm dealing with). I assume this can be explained by RMSE's sensitivity to outliers?

Answer 1

You can not use error to reliably measure accuracy. Error is best used as a measure of how fast the model is currently learning.

As an example, using different loss functions (cross entorpy vs MSE) results in massively different values for the error at similar accuracy.

Also considering this, an error of 0.0000000001 quite often has lower validation set accuracy then and error of 0.1, as the prior is likely over trained.

As for you second question, yes this is because MSE has a huge bias towards outliers. I have personally found regression networks to struggle in most circumstances, so if it is at all possible to turn the network into a classifier, you may see an improvement.

Answer 2

Just as a general remark, notice that technically we don't use the term "accuracy" for regression settings, such as yours - only for classification ones.

If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))*100 = 86.2% accurate"?

No.

The advantage of the RMSE, as you have correctly quoted, is that it is in the same units with your predicted quantity; so, if this quantity is, say, USD, you can say (to the business user) that the error of the model is 0.019 USD, and this can be perfectly fine by itself. But you cannot convert it to a percentage - it would be meaningless.

If required to give the performance of a regression model in a percentage, your best option would be the Mean Absolute Percentage Error (MAPE).