7
$\begingroup$

Usually when performing linear regression predictions and gradient descent, the measure of the level of error for a particular line will be measured by the sum of the squared-distance values.

Why distance squared?

In most of the explanations I heard, they claim that:

  • the function itself does not matter
  • the result should be positive so positive and negative deviations are still counted

However, an abs() approach would still work. And isn't it inconvenient that distance squared minimizes the distance result for distances lower than 1?

I'm pretty sure someone must have considered this already -- so why is distance squared the most used approach to linear regression?

$\endgroup$
  • 1
    $\begingroup$ Thank you very much for your answers -- all were very informative in a way and all of them addressed my questions from different points of view. I think I'm going to like this community. :) $\endgroup$ – Alpha Jan 31 '17 at 0:45
  • 1
    $\begingroup$ I'm voting to close this question as off-topic (see scope defined in help center). At present it would rather migrate to Cross Validated to get an answer. $\endgroup$ – Eric Platon Feb 14 '17 at 3:26
  • $\begingroup$ @EricPlaton, the functions used to gauge the fitness of a model to empirical data is central to learning. Although linear regression is centuries old and may not be an area of significant research today, classifying this question as off topic for machine learning is like stating that the embarrassment of hitting the ground in front of family and peers has nothing to do with learning to ride a bicycle. $\endgroup$ – FauChristian Oct 21 '17 at 5:03
  • 1
    $\begingroup$ If the function doesn't matter, how convergent would f(e) = e^(-2) be? If the function must be positive, why wouldn't f(e) = x^2 - 10^(100) produce the same linear regression formulae as x^2 + 10^(100)? Why would one waste the time to use gradient descent for a linear model? ... You may wish to question the veracity of our sources. $\endgroup$ – FauChristian Oct 25 '17 at 21:23
4
$\begingroup$

Brief Background

The error metric (an appropriate term used in the question title) quantifies the fitness of a linear or nonlinear model.

It aggregates individual errors across a set of observations (instances of training data). In typical use, an error function is applied to the difference between the dependent variable vector predicted by the model and empirical observations. These differences are calculated for each observation and then summed. 1

Why Distance Squared?

Legendre, who first published the sum of squares method for gauging fitness of the model (Paris 1705) stated correctly that squaring before summing is convenient. Why did he write that?

One could use the absolute value of the error or the absolute value of its cube, but the discontinuity of the derivative of the absolute value makes the function NOT smooth. Functions that are NOT smooth create unnecessary difficulties when employing linear algebra to derive closed forms (simple algebraic expressions).

Closed forms are convenient when one wants to quickly and easily calculate slope and intercept in linear regression. 2

Gradient Descent

Gradient descent is generally employed for nonlinear regression. Lacking the ability to create closed forms for many nonlinear models, iteration becomes a dominant methodology for validating or tuning the model.

An intuitive understanding of gradient descent can be gained by considering a thirsty, blind person looking for water on land solely by taking calculated steps. (In software, these steps are iterations.) The blind person can only sense the direction of the altitude gradient (direction of slope) with their feet to descend to a local minimum altitude. 3

Anyone stating that, "The function itself does not matter," in relation to the usual applications of gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance as an error function would likely lead to the dehydration and death of the hikers.

The selection criteria for error metrics is important if one is interested in the speed of convergence on a solution or whether the solution will ever be found. 4

Since the gradient of a plane (linear surface) is a constant, the use of gradient descent for linear models is wasteful. The blind person need not continue to sample the angle of their foot.

Sign of the Error Metric

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. 5

Effectiveness of Error Metrics in Relation to 1.0

Because of the partial derivative of the least squares error metric with respect to an error at any given point is constant, the least squares error metric converges similarly above and below 1.0.

Notes

[1] The dimensions of a model's independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

[2] Another smooth function, such as the error to the fourth power would also lead to closed forms for slope and intercept, although they would produce slightly different results if the correlation coefficient is non-zero.

[3] Gradient descent algorithms in general do not guarantee finding a global minimum. In the example given, it would be quote possible to miss a small hole exists with water in it. Depending on the surface features (terrain), sensing the angle of the foot (determining gradient) can be counterproductive. The search can becomes chaotic. To extend the intuitive analogy, consider searching for the bottom of the stairs in Escher's Relativity lithograph.

[4] For an error metric to be likely to converge and therefore useful in regression regardless of the direction of the error the sign of the metric is irrelevant. It is each of the set of partial derivatives of the error metric with respect to the corresponding set of distances between the model predictions and observations that should be positive to regress omnidirectionally. It sounds more complicated, but even this corrected statement is an oversimplification.

[5] The error metric in gradient descent applications is often calculated using a convex function to avoid overshoot and possible oscillation and non-convergence. In some cases error functions other that sum of squares is used. The choice of function has to do with a number of factors:

  • The model to which the data is to be fit
  • Factors expected to affect or actually affecting deviations of the observations (training data) from the model
  • Computational resources relative to the size of the data set
$\endgroup$
4
$\begingroup$

The squared form is sometimes called the Euclidean norm or L2 norm. One of its very helpful properties is that it has an easily defined derivative, which can be used in mathematical analysis and translated fairly easily into code.

Intuitively it is thought that it is advantageous to exaggerate the differences according to the value of the error, which squaring does. You might also use the powers 3 or 4, but the derivative is more complex.

A number of different norms may be used, according to the particular circumstances of the problem at hand.

$\endgroup$
  • 1
    $\begingroup$ Some suggestions: Also answer the second question about distances lower than 1 too. The two bulletted claims in the middle of the question incorrect and should be challenged. The term Euclidean Norm has ambiguous meaning according to the people at Wolfram (and that checks out), so the L2 Norm may want to be the only abbreviated term mentioned for the least squares error method. $\endgroup$ – FauChristian Oct 25 '17 at 21:02
2
$\begingroup$

One justification comes from the central limit theorem. If the noise in your data is the result of the sum of many independent effects, then it will tend to be normally distributed. And normally distributed means that the likelihood of the data is inversely proportional the exponential of the square of the distance to the mean.

In other words, minimizing the sum of squares of the distance to the mean amounts to finding the most likely value for the line assuming that the error is normally distributed. This is very often a reasonable assumption, but it is of course not always true.

$\endgroup$
  • 1
    $\begingroup$ Quantum noise is expensive to generate and rare. Most deviations between theoretical models and empirical data are systematic, often chaotic, possessing distributions that are not nearly normal. Using a square may be nicely aligned with the Q function as you stated, but that is not the historical or present reason for the square. Continuous functions allow the application of linear algebra to produce closed forms for slope and intercept for linear regression. Even functions (2, 4, ...) are continuous. The absolute values of odd functions are not. $\endgroup$ – FauChristian Oct 21 '17 at 4:56
2
$\begingroup$

It simply derives itself from the maximum likelihood estimation. where in we maximise the log likelihood function., for detailed insight see this lecture: The Method of Maximum Likelihood for Simple Linear Regression.

$\endgroup$
  • $\begingroup$ The least squares or L2 Norm is not often selected because of its derivation from the method of maximum likelihood. The distribution of errors in practice are rarely determined and when they are determined are rarely normal. It is out of the convenience of using an odd polynomial (with a continuous first derivative) and because of the low demand on computation resources that squaring is used. $\endgroup$ – FauChristian Oct 25 '17 at 21:11
  • $\begingroup$ Yes, that would be another great answer. But my answer is more based on theoretical or statistical aspect of linear regression. $\endgroup$ – GeneX Oct 26 '17 at 4:01
1
$\begingroup$

One justification is that under homoscedasticity the L2 norm produces the minimum variance unbiased estimator (MVUE), see Gauss-Markov Theorem. It means that the fitted values are the conditional expectations given the explanatory variables which is in many cases a nice property. Further it is the best estimator if the previous property is desirable.

As a response to the claim that the function itself does not matter, different functions give solutions with very different properties and a lot of effort has gone in to finding appropriate penalty functions, see for example Ridge regression and LASSO. The penalty function does matter.

edit: In response to your question regarding distances lower than 1, nothing "goes wrong" when the distances are smaller than 1. We always want to minimize the distance and the squared loss does so everywhere.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.