2
$\begingroup$

In a NN regression problem, considering that MSE is squaring the error and the error is between 0 and 1 would it be pointless to use MSE as our loss function during model training?

For example:

MSE = (y_pred - y_true) ^ 2 

@ Expected model output range [0, 1]:
MSE = (0.1 - 0.01) ^ 2 = 0.0081

// Significantly larger error is less pronounced in the MSE output
MSE = (0.1 - 0.0001) ^ 2 = 0.00998001


@ Expected model output range [10, 20]:
MSE = (10 - 12) ^ 2 = 4

// Significantly larger error is more pronounced in the MSE output
MSE = (10 - 20) ^ 2 = 100

If it’s indeed useless for that range, would using RMSE allow us to use this loss function at 0-1 range to benefit from its outlier sensitivity during training or is there another loss that would mimic the effect of MSE for values between 0 and 1?

Improve this question
$\endgroup$
1
  • $\begingroup$ @user253751 under the assumption that since squaring error values that are <1 has the opposite effect (i.e 0.2^2 = 0.04 but 2^2 = 4) since this is being used in a loss function from which the neural network is computing gradients the outliers move the gradient less than values within the common range of errors. $\endgroup$
    – Darren Rahnemoon
    Commented Mar 29, 2023 at 21:30

2 Answers 2

Reset to default
0
$\begingroup$

Depending on what you want to do, there are advantages to other loss functions (crossentropy) and other regression models (beta regression), but there is not necessarily a reason to dislike MSE as a loss function when the target is between $0$ and $1$, no. For instance, it might be that you know your outcome has a Gaussian distribution with a mean between $0$ and $1$. Then what you’re doing by minimizing MSE is equivalent to maximum likelihood estimation, which has many nice statistical properties.

From calculus, $f(x)$ and $\sqrt{f(x)}$ are minimized by the same $x$. Thus, RMSE and MSE are equivalent as loss functions in the sense that whatever minimized one minimizes the other.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Using a cross entropy loss vs. a regression loss amount to doing two greatly different tasks. From my understand OP has output values in $[0, 1]$, whereas a cross entropy loss is used for classifications where your values(/labels) are in $\{0, 1\}$. $\endgroup$
    – David
    Commented Mar 27, 2023 at 10:27
  • $\begingroup$ It might be that the true targets span the entire interval. It also could be that the targets are just the integers and the OP wants to estimate the probability. That is why it depends on the task. $\endgroup$
    – Dave
    Commented Mar 27, 2023 at 10:33
  • $\begingroup$ Yes just to clarify the expected output is a range between 0-1 not binary. It’s a regression problem not a classification. $\endgroup$
    – Darren Rahnemoon
    Commented Mar 28, 2023 at 2:07
  • $\begingroup$ When aggregated over a batch of data, MSE and RMSE have different minima. And I think it is perfectly fine to use cross entropy for non-binary labels, they just represent uncertainty of the correct answer. I do this regularly with binary image masks with blurred edges. $\endgroup$
    – NikoNyrh
    Commented Mar 28, 2023 at 18:13
0
$\begingroup$

To summarize all the comments and what my original mistake was:

The sensitivity effect is still there even though yes the squared of the error value less than 0 is less that itself however:

Example: 

For two error values of 0.9 and 0.2

MAE: 0.9
MSE: 0.9 ^ 2 = 0.81

MAE: 0.2
MSE: 0.2 ^ 2 = 0.04

MAE: 0.9 / 0.2 = 4.5 # less sensitive in MAE
MSE: 0.81 / 0.04 = 20.25 # still significantly sensitive in MSE
Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .