Return to Answer

The error metric (the quite appropriate term used in the question) quantifies the fitness of a model (linear or otherwise) as a scalar value.

It aggregates individual errors across a set of observations (instances of input data). In typical use, an error function is applied to the difference between the dependent variable vector predictions of the model and the observations for each instance of dependent variable vector input and then summed.

The dimensions of the independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

Closed forms are convenient when one wants to quickly and easily calculate slope and intercept in linear regression. ¹

An intuitive understanding of gradient descent can be gained by considering a thirsty, blind person looking for water on land solely by taking calculated steps. (In software, these steps are iterations.) The blind person can only sense the direction of the altitude gradient (direction of slope) with their feet to descend to a local minimum altitude. ²

Anyone stating that, "The function itself does not matter," in relation to the usual applications of gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance as an error function would likely lead to the dehydration and death of the hikers. The selection criteria for error metrics is important if one is interested in the speed of convergence on a solution or whether the solution will ever be found. ³

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. ⁴

[1] Another smooth function, such as the error to the fourth power would also lead to closed forms for slope and intercept, although they would produce slightly different results if the correlation coefficient is non-zero.

[2] Gradient descent algorithms in general do not guarantee finding a global minimum. In the example given, it would be quote possible to miss a small hole exists with water in it. Depending on the surface features (terrain), sensing the angle of the foot (determining gradient) can be counterproductive. The search can becomes chaotic. To extend the intuitive analogy, consider searching for the bottom of the stairs in Escher's Relativity lithograph.

[3] For an error metric to be likely to converge and therefore useful in regression regardless of the direction of the error the sign of the metric is irrelevant. It is each of the set of partial derivatives of the error metric with respect to the corresponding set of distances between the model predictions and observations that should be positive to regress omnidirectionally. It sounds more complicated, but even this corrected statement is an oversimplification.

[4] The error metric in gradient descent applications is often calculated using a convex function to avoid overshoot and possible oscillation and non-convergence. In some cases error functions other that sum of squares is used. The choice of function has to do with a number of factors: (1) the model to which the data is to be fit, (2) the factors expected to affect or actually affecting deviations of the observations (training data) from the model, and (3) computational resources relative to the size of the data set.

The error metric (an appropriate term used in the question title) quantifies the fitness of a linear or nonlinear model.

It aggregates individual errors across a set of observations (instances of training data). In typical use, an error function is applied to the difference between the dependent variable vector predicted by the model and empirical observations. These differences are calculated for each observation and then summed. ¹

Closed forms are convenient when one wants to quickly and easily calculate slope and intercept in linear regression. ²

An intuitive understanding of gradient descent can be gained by considering a thirsty, blind person looking for water on land solely by taking calculated steps. (In software, these steps are iterations.) The blind person can only sense the direction of the altitude gradient (direction of slope) with their feet to descend to a local minimum altitude. ³

Anyone stating that, "The function itself does not matter," in relation to the usual applications of gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance as an error function would likely lead to the dehydration and death of the hikers.

The selection criteria for error metrics is important if one is interested in the speed of convergence on a solution or whether the solution will ever be found. ⁴

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. ⁵

[1] The dimensions of a model's independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

[2] Another smooth function, such as the error to the fourth power would also lead to closed forms for slope and intercept, although they would produce slightly different results if the correlation coefficient is non-zero.

[3] Gradient descent algorithms in general do not guarantee finding a global minimum. In the example given, it would be quote possible to miss a small hole exists with water in it. Depending on the surface features (terrain), sensing the angle of the foot (determining gradient) can be counterproductive. The search can becomes chaotic. To extend the intuitive analogy, consider searching for the bottom of the stairs in Escher's Relativity lithograph.

[4] For an error metric to be likely to converge and therefore useful in regression regardless of the direction of the error the sign of the metric is irrelevant. It is each of the set of partial derivatives of the error metric with respect to the corresponding set of distances between the model predictions and observations that should be positive to regress omnidirectionally. It sounds more complicated, but even this corrected statement is an oversimplification.

[5] The error metric in gradient descent applications is often calculated using a convex function to avoid overshoot and possible oscillation and non-convergence. In some cases error functions other that sum of squares is used. The choice of function has to do with a number of factors:

• The model to which the data is to be fit
• Factors expected to affect or actually affecting deviations of the observations (training data) from the model
• Computational resources relative to the size of the data set

Correctness of the Question Terminology

The use of the term Error Metric in the question is excellent. The error metric for fitness of the model (linear or otherwise) is a scalar that aggregates the individual errors of the dependent variable vector predictions of the model against the observations, for each instance of dependent variable vector input. The dimensions of the independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

One could use the absolute value of the error or its cube, but the discontinuity of absolute values makes the use of odd functions problematic for employing linear algebra to derive closed forms for slope and intercept in linear regression. Another even function, such as the error to the fourth power would also lead to closed forms for slope and intercept.

Legendre, who first published the sum of squares method for gauging fitness of the model (Paris 1705) stated correctly that squaring before summing is convenient.

Gradient descent is for nonlinear regression.

The way to intuitively consider gradient descent is a blind, thirsty person looking for water on land in steps. (In software, these steps are iterations.) The blind person can only detect the gradient of altitude with respect to direction from the incline of their feet and descend to a local minimum. Algorithms of this sort do not guarantee finding a global minimum and it is possible that a small hole exists with water in it that is missed. In some terrain, the foot is a counterproductive size and the search becomes chaotic as if it were an Escher drawing.

Linear regression is much simpler for its user, using a rational expression of summations derived from linear algebra and the least squares model for the error metric. It is not iterative. For this reason, use of gradient descent for linear models is wasteful, and choices other than the sum of squares and other previously used error metrics requires additional (and probably unnecessary) derivations using principles linear algebra.

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. For an error metric to be usable regardless of the direction of the error, the partial derivative of error metric with respect to the length of the error vector prediction relative to its corresponding observation should be positive for each observation.

Real Gradient Descent Applications

It is also helpful in most real applications that the error metric be convex to avoid overshoot and possible oscillation and non-convergence. There is some risk in using an order of one rather than two in real gradient descent applications where the model being validated or tuned is generally nonlinear. Because of the way in which the magnitude of the gradient is proportional to the error, squaring has the same relative effect with errors above and below 1.0.

Anyone stating that, "The function itself does not matter," in relation to gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance would lead to death, and the absolute value of the cube root might devolve into chaotic iteration for some data sets.

In some cases functions other that sum of squares is used. The choice of function has to do with a number of factors: (1) the model to which the data is to be fit, (2) the factors expected to affect or actually affecting deviations of the observations (training data) from the model, and (3) computational resources relative to the size of the data set.

Brief Background

The error metric (the quite appropriate term used in the question) quantifies the fitness of a model (linear or otherwise) as a scalar value.

It aggregates individual errors across a set of observations (instances of input data). In typical use, an error function is applied to the difference between the dependent variable vector predictions of the model and the observations for each instance of dependent variable vector input and then summed.

The dimensions of the independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

Legendre, who first published the sum of squares method for gauging fitness of the model (Paris 1705) stated correctly that squaring before summing is convenient. Why did he write that?

One could use the absolute value of the error or the absolute value of its cube, but the discontinuity of the derivative of the absolute value makes the function NOT smooth. Functions that are NOT smooth create unnecessary difficulties when employing linear algebra to derive closed forms (simple algebraic expressions).

Closed forms are convenient when one wants to quickly and easily calculate slope and intercept in linear regression. ¹

Gradient descent is generally employed for nonlinear regression. Lacking the ability to create closed forms for many nonlinear models, iteration becomes a dominant methodology for validating or tuning the model.

An intuitive understanding of gradient descent can be gained by considering a thirsty, blind person looking for water on land solely by taking calculated steps. (In software, these steps are iterations.) The blind person can only sense the direction of the altitude gradient (direction of slope) with their feet to descend to a local minimum altitude. ²

Anyone stating that, "The function itself does not matter," in relation to the usual applications of gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance as an error function would likely lead to the dehydration and death of the hikers. The selection criteria for error metrics is important if one is interested in the speed of convergence on a solution or whether the solution will ever be found. ³

Since the gradient of a plane (linear surface) is a constant, the use of gradient descent for linear models is wasteful. The blind person need not continue to sample the angle of their foot.

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. ⁴

Effectiveness of Error Metrics in Relation to 1.0

Because of the partial derivative of the least squares error metric with respect to an error at any given point is constant, the least squares error metric converges similarly above and below 1.0.

Notes

[1] Another smooth function, such as the error to the fourth power would also lead to closed forms for slope and intercept, although they would produce slightly different results if the correlation coefficient is non-zero.

[2] Gradient descent algorithms in general do not guarantee finding a global minimum. In the example given, it would be quote possible to miss a small hole exists with water in it. Depending on the surface features (terrain), sensing the angle of the foot (determining gradient) can be counterproductive. The search can becomes chaotic. To extend the intuitive analogy, consider searching for the bottom of the stairs in Escher's Relativity lithograph.

[3] For an error metric to be likely to converge and therefore useful in regression regardless of the direction of the error the sign of the metric is irrelevant. It is each of the set of partial derivatives of the error metric with respect to the corresponding set of distances between the model predictions and observations that should be positive to regress omnidirectionally. It sounds more complicated, but even this corrected statement is an oversimplification.

[4] The error metric in gradient descent applications is often calculated using a convex function to avoid overshoot and possible oscillation and non-convergence. In some cases error functions other that sum of squares is used. The choice of function has to do with a number of factors: (1) the model to which the data is to be fit, (2) the factors expected to affect or actually affecting deviations of the observations (training data) from the model, and (3) computational resources relative to the size of the data set.

Correctness of the Question Terminology

The use of the term Error Metric in the question is excellent. The error metric for fitness of the model (linear or otherwise) is a scalar that aggregates the individual errors of the dependent variable vector predictions of the model against the observations, for each instance of dependent variable vector input. The dimensions of the independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

Why Distance Squared?

Legendre, who first published the sum of squares method for gauging fitness of the model (Paris 1705) stated correctly that squaring before summing is a convenience.

The way to intuitively consider gradient descent is a blind, thirsty person looking for water on land in steps. (In software, these steps are iterations.) The blind person can only detect the gradient of altitude with respect to direction from the incline of their feet and descend to a local minimum. Algorithms of this sort do not guarantee finding a global minimum and it is possible that a small hole exists with water in it that is missed. In some terrain, the foot is a counterproductive size and the search becomes chaotic as if it were an Escher drawing.

Linear regression is much simpler for its user, using a rational expression of summations derived from linear algebra and the least squares model for the error metric. It is not iterative. For this reason, use of gradient descent for linear models is wasteful, and choices other than the sum of squares and other previously used error metrics requires additional (and probably unnecessary) derivations using principles linear algebra.

Sign of the Error Metric

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. For an error metric to be usable regardless of the direction of the error, the partial derivative of error metric with respect to the length of the error vector prediction relative to its corresponding observation should be positive for each observation.

Real Gradient Descent Applications

It is also helpful in most real applications that the error metric be convex to avoid overshoot and possible oscillation and non-convergence. There is some risk in using an order of one rather than two in real gradient descent applications where the model being validated or tuned is generally nonlinear. Because of the way in which the magnitude of the gradient is proportional to the error, squaring has the same relative effect with errors above and below 1.0.

Anyone stating that, "The function itself does not matter," in relation to gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance would lead to death, and the absolute value of the cube root might devolve into chaotic iteration for some data sets.

In some cases functions other that sum of squares is used. The choice of function has to do with a number of factors: (1) the model to which the data is to be fit, (2) the factors expected to affect or actually affecting deviations of the observations (training data) from the model, and (3) computational resources relative to the size of the data set.

Correctness of the Question Terminology

The use of the term Error Metric in the question is excellent. The error metric for fitness of the model (linear or otherwise) is a scalar that aggregates the individual errors of the dependent variable vector predictions of the model against the observations, for each instance of dependent variable vector input. The dimensions of the independent and dependent variable vectors are, in machine learning, conventionally called features and labels respectively.

Why Distance Squared?

One could use the absolute value of the error or its cube, but the discontinuity of absolute values makes the use of odd functions problematic for employing linear algebra to derive closed forms for slope and intercept in linear regression. Another even function, such as the error to the fourth power would also lead to closed forms for slope and intercept.

Legendre, who first published the sum of squares method for gauging fitness of the model (Paris 1705) stated correctly that squaring before summing is convenient.

Gradient Descent

Gradient descent is for nonlinear regression.

The way to intuitively consider gradient descent is a blind, thirsty person looking for water on land in steps. (In software, these steps are iterations.) The blind person can only detect the gradient of altitude with respect to direction from the incline of their feet and descend to a local minimum. Algorithms of this sort do not guarantee finding a global minimum and it is possible that a small hole exists with water in it that is missed. In some terrain, the foot is a counterproductive size and the search becomes chaotic as if it were an Escher drawing.

Linear regression is much simpler for its user, using a rational expression of summations derived from linear algebra and the least squares model for the error metric. It is not iterative. For this reason, use of gradient descent for linear models is wasteful, and choices other than the sum of squares and other previously used error metrics requires additional (and probably unnecessary) derivations using principles linear algebra.

Sign of the Error Metric

The statement, "The result should be positive so positive and negative deviations are still counted," is incorrect. For an error metric to be usable regardless of the direction of the error, the partial derivative of error metric with respect to the length of the error vector prediction relative to its corresponding observation should be positive for each observation.

Real Gradient Descent Applications

It is also helpful in most real applications that the error metric be convex to avoid overshoot and possible oscillation and non-convergence. There is some risk in using an order of one rather than two in real gradient descent applications where the model being validated or tuned is generally nonlinear. Because of the way in which the magnitude of the gradient is proportional to the error, squaring has the same relative effect with errors above and below 1.0.

Anyone stating that, "The function itself does not matter," in relation to gradient descent would be a dangerous choice for guide in a blind hiking expedition. For instance, the reciprocal of distance would lead to death, and the absolute value of the cube root might devolve into chaotic iteration for some data sets.

In some cases functions other that sum of squares is used. The choice of function has to do with a number of factors: (1) the model to which the data is to be fit, (2) the factors expected to affect or actually affecting deviations of the observations (training data) from the model, and (3) computational resources relative to the size of the data set.