Is there a connection between the bias term in a linear regression model and the bias that can lead to under-fitting?

Question

Here is a linear regression model

$$y = mx + b,$$

where $b$ is known as $y$-intercept, but also known as the bias [1], $m$ is the slope, and $x$ is the feature vector.

As I understood, in machine learning, there is also the bias that can cause the model to underfit.

So, is there a connection between the bias term $b$ in a linear regression model and the bias that can lead to under-fitting in machine learning?

Answer 1

In machine learning, the term bias can refer to at least 2 related concepts

1. A (learnable) parameter of a model, such as a linear regression model, which allows you to learn a shifted function. For example, in the case of a linear regression model $y = f(x) = mx + b$, the bias $b$ allows you to shift the straight-line up an down: without the bias, you would only be able to control the slope $m$ of the straight-line. Similarly, in a neural network, you can have a neuron that performs a linear combination of the inputs, then it uses a bias term to shift the straight-line, and you could also use the bias after having applied the activation function, but this will have a different effect.

2. Anything that guides the learning algorithm (e.g. gradient descent with back-propagation) towards a specific set of solutions. For example, if you use regularization, you are biasing the learning algorithm to choose, typically, smoother or simpler functions. The bias term in the linear regression model is also a way of biasing the learning algorithm: you assume that the straight-line function does not necessarily go through zero, and this assumption affects the type of functions that you can learn (and this is why these two concepts of bias are related!). So, there are many ways of biasing a learning algorithm.

The bias does not always lead you to the best solutions and can actually lead to the wrong solutions, but it is often useful in many ways, for example, it can speed up learning, as you restrict the number of functions that can be learned and searched. The bias (as described in point 2) is often discussed in the context of the bias-variance trade-off, and both the bias and the variance are related to the concepts of generalization, under-fitting, and over-fitting. The linked Wikipedia article explains these concepts quite well and provides an example of how the bias and variance are decomposed, so you should probably read that article for more details.