3
$\begingroup$

I understand that RMSE is just the square root of MSE. Generally, as far as I have seen, people seem to use MSE as a loss function and RMSE for evaluation purposes, since it exactly gives you the error as a distance in the Euclidean space.

What could be a major difference between using MSE and RMSE when used as loss functions for training?

I'm curious because good frameworks like PyTorch, Keras, etc. don't provide RMSE loss functions out of the box. Is it some kind of standard convention? If so, why?

Also, I'm aware of the difference that MSE magnifies the errors with magnitude>1 and shrinks the errors with magnitude<1 (on a quadratic scale), which RMSE doesn't do.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ 1.) Ease of derivative. 2.) Don't have to worry about ~0 in denominator causing huge gradient 3.) But to me the most important is mathematical convenience, someone might easily make the mistake of RMSE is just equal the difference $y-y'$ instead of root of mean square of $y-y'$. The answer this might start depending on conventions. 4.) In maths (Don't know the reason and might be inaccurate) we mainly work with variances instead of standard deviation. $\endgroup$
    – user9947
    Aug 31, 2019 at 17:37
  • $\begingroup$ Thanks @DuttaA , I think the comment you've given is quite good, that it can be one of the answers to this question. So, please post it as an answer below :) $\endgroup$
    – Gokul NC
    Sep 1, 2019 at 5:10
  • $\begingroup$ I don't believe there is any practical difference. Someone should provide a reference to a source that says there is a difference if you think there is one. $\endgroup$
    – Taw
    Jun 16, 2021 at 5:26

1 Answer 1

Reset to default
1
$\begingroup$

Answers in the comments are decent, particularly DuttA.

DuttA gives these, approximately

  • ease of derivative
  • Don't have to worry about ~0 in denominator causing huge gradient
  • But to me the most important is mathematical convenience, someone might easily make the mistake of RMSE is just equal the difference y−y′ instead of root of mean square of y−y′. The answer this might start depending on conventions.
  • In maths (Don't know the reason and might be inaccurate) we mainly work with variances instead of standard deviation.

Here are my reasons for using the MSE instead of RMSE:

  • Doesn't have the sqrt operations, so it computes faster
  • the square root isn't easy, its Newtons method, so it could be a dozens steps per iteration
  • MSE has all the information of RMSE, there is 1-to-1 mapping, so no loss
  • the storage of the square root typically doesn't save any memory in IEEE-784 and compute vs. memory is big thing in complexity
  • tools like gradient boosted machine can "recycle" the squared error computation for speedup and working on O(n) complexity
  • there is hidden scaling and regularization because many gpu hardware elements are fundamentally 8-bit, so if you can make your code more 8-bit in its guts then you don't have as much in the back-conversion and it runs a lot faster
Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .