Linear regression: why is distance *squared* used as an error metric?

Question

Usually when performing linear regression predictions and gradient descent, the measure of the level of error for a particular line will be measured by the sum of the squared-distance values.

Why distance squared?

In most of the explanations I heard, they claim that:

• the function itself does not matter
• the result should be positive so positive and negative deviations are still counted

However, an abs() approach would still work. And isn't it inconvenient that distance squared minimizes the distance result for distances lower than 1?

I'm pretty sure someone must have considered this already -- so why is distance squared the most used approach to linear regression?

Answer 1

Brief Background

The error metric (an appropriate term used in the question title) quantifies the fitness of a linear or nonlinear model.

It aggregates individual errors across a set of observations (instances of training data). In typical use, an error function is applied to the difference between the dependent variable vector predicted by the model and empirical observations. These differences are calculated for each observation and then summed. ¹

Why Distance Squared?

Legendre, who first published the sum of squares method for gauging fitness of the model (Paris 1705) stated correctly that squaring before summing is convenient. Why did he write that?

One could use the absolute value of the error or the absolute value of its cube, but the discontinuity of the derivative of the absolute value makes the function NOT smooth. Functions that are NOT smooth create unnecessary difficulties when employing linear algebra to derive closed forms (simple algebraic expressions).

Closed forms are convenient when one wants to quickly and easily calculate slope and intercept in linear regression. ²

Gradient Descent

Gradient descent is generally employed for nonlinear regression. Lacking the ability to create closed forms for many nonlinear models, iteration becomes a dominant methodology for validating or tuning the model.

An intuitive understanding of gradient descent can be gained by considering a thirsty, blind person looking for water on land solely by taking calculated steps. (In software, these steps are iterations.) The blind person can only sense the direction of the altitude gradient (direction of slope) with their feet to descend to a local minimum altitude. ³

Anyone stating that, "The function itself does not matter," in relation to the usual applications of gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance as an error function would likely lead to the dehydration and death of the hikers.

The selection criteria for error metrics is important if one is interested in the speed of convergence on a solution or whether the solution will ever be found. ⁴

Since the gradient of a plane (linear surface) is a constant, the use of gradient descent for linear models is wasteful. The blind person need not continue to sample the angle of their foot.

Sign of the Error Metric

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. ⁵

Effectiveness of Error Metrics in Relation to 1.0

Because of the partial derivative of the least squares error metric with respect to an error at any given point is constant, the least squares error metric converges similarly above and below 1.0.

Notes

[1] The dimensions of a model's independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

[2] Another smooth function, such as the error to the fourth power would also lead to closed forms for slope and intercept, although they would produce slightly different results if the correlation coefficient is non-zero.

[3] Gradient descent algorithms in general do not guarantee finding a global minimum. In the example given, it would be quote possible to miss a small hole exists with water in it. Depending on the surface features (terrain), sensing the angle of the foot (determining gradient) can be counterproductive. The search can becomes chaotic. To extend the intuitive analogy, consider searching for the bottom of the stairs in Escher's Relativity lithograph.

[4] For an error metric to be likely to converge and therefore useful in regression regardless of the direction of the error the sign of the metric is irrelevant. It is each of the set of partial derivatives of the error metric with respect to the corresponding set of distances between the model predictions and observations that should be positive to regress omnidirectionally. It sounds more complicated, but even this corrected statement is an oversimplification.

[5] The error metric in gradient descent applications is often calculated using a convex function to avoid overshoot and possible oscillation and non-convergence. In some cases error functions other that sum of squares is used. The choice of function has to do with a number of factors:

• The model to which the data is to be fit
• Factors expected to affect or actually affecting deviations of the observations (training data) from the model
• Computational resources relative to the size of the data set

Answer 2

The squared form is sometimes called the Euclidean norm or L2 norm. One of its very helpful properties is that it has an easily defined derivative, which can be used in mathematical analysis and translated fairly easily into code.

Intuitively it is thought that it is advantageous to exaggerate the differences according to the value of the error, which squaring does. You might also use the powers 3 or 4, but the derivative is more complex.

A number of different norms may be used, according to the particular circumstances of the problem at hand.

Answer 3

One justification comes from the central limit theorem. If the noise in your data is the result of the sum of many independent effects, then it will tend to be normally distributed. And normally distributed means that the likelihood of the data is inversely proportional the exponential of the square of the distance to the mean.

In other words, minimizing the sum of squares of the distance to the mean amounts to finding the most likely value for the line assuming that the error is normally distributed. This is very often a reasonable assumption, but it is of course not always true.

Answer 4

It simply derives itself from the maximum likelihood estimation. where in we maximise the log likelihood function., for detailed insight see this lecture: The Method of Maximum Likelihood for Simple Linear Regression.

Answer 5

One justification is that under homoscedasticity the L2 norm produces the minimum variance unbiased estimator (MVUE), see Gauss-Markov Theorem. It means that the fitted values are the conditional expectations given the explanatory variables which is in many cases a nice property. Further it is the best estimator if the previous property is desirable.

As a response to the claim that the function itself does not matter, different functions give solutions with very different properties and a lot of effort has gone in to finding appropriate penalty functions, see for example Ridge regression and LASSO. The penalty function does matter.

edit: In response to your question regarding distances lower than 1, nothing "goes wrong" when the distances are smaller than 1. We always want to minimize the distance and the squared loss does so everywhere.