So I was wondering, why I have only encountered square loss function also known as MSE. The only nice property of MSE I am so far aware of is its convex nature. But then all equations of the form $x^{2n}$ where $n$ is an integer belongs to the same family.

My question is what makes MSE the most suitable candidate among this entire family of curves? Why do other curves in the family, even though having steeper slopes, $(x >1) $, which might result in better optimisation, not used?

Here is a picture to what I mean where red is $x^4$ and green is $x^2$:

enter image description here


I can comment on several properties of MSE and related losses.

As you mentioned MSE (aka $l_2$-loss) is convex which is a great property in optimization in which one can find a single global optimum. MSE is used in linear and non-linear least squares problems which form the basis of many widely used statistical methods. I would imagine the math and implementation would be more difficult if one would use a higher-order loss (e.g. $x^3$) and that would also prove to be futile because MSE already possesses great statistical and optimization properties on its own.
Another important aspect, one wouldn't use higher-order loss functions in regression is because it would be extremely prone to outliers. MSE on its own would weigh the outliers much more than l1-loss would! And in real world data there is always noise and outliers present. In comparison l1 loss is more difficult in optimization, one reason for which is it's not differentiable at zero.

Other interesting losses you might want to read about are $l_0$ and $l_{inf}$ loss, all of which have their own trade-offs in optimization-sense.


There is another variant for MSE. You can employ the absolute value of the difference of your hypothesis and the expected output. MSE and the absolute difference version, each have a property. The interpretation of the MSE is that you have the area of squares which at first are large but after training the predicted outputs and real outputs get similar to each other which means the area of squares has been diminished after training. For absolute value, the interpretation is that the difference you are going to reduce is a length and after training, you diminish the length of the errors. I don't say what you've said is not possible. It is possible and it does have an interpretation, maybe to diminish the volume of hypercubes, but the point is that the differentiation of such cost functions are a bit complex and it may lead to update rules which are longer to be calculated.

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Ben N Mar 3 '19 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.