What is the difference between linear and non-linear regression?

Question

In machine learning, I understand that linear regression assumes that parameters or weights in equation should be linear. For Example:

$$y = w_1x_1 + w_2x_2$$

is a linear equation where $x_1$ and $x_2$ are feature variables and $w_1$ and $w_2$ are parameters.

Also

$$y = w_1(x_1)^2 + w_2(x_2)^2$$

is also linear as parameters $w_1$ and $w_2$ are linear with respect to $y$.

Now, I read some articles stating that in the equation like

$$y = \log(w_1)x_1 + \log(w_2)x_2$$

can also be made linear by considering other variables $v_1$ and $v_2$ as:

\begin{align} v_1 &= \log(w_1)\\ v_2 &= \log(w_2) \end{align}

Thus,

$$y = v_1x_1 + v_2x_2$$

So, in this sense, any non-linear equation can be made linear, then what is non-linear regression here? I think I am missing something important here. I am a beginner in the field of Machine Learning. Can somebody help me?

Answer 1

The difference is simply that non-linear regression learns parameters that in some way control the non-linearity - e.g. any weight or bias that is applied before a non-linear function.

For instance:

$$y = (w_1 x_1 + w_2 x_2)^2 + w_3$$

With such a function to learn, you cannot separate out transformed values of $w_1$ and $w_2$ and turn this into a linear function of just $x_1$ and $x_2$.

What you are describing as non-linearities in your examples are instead all applied by the machine learning engineer to create new candidate features for linear regression. This is not usually described as non-linear regression, but feature transformation or feature engineering.

There is also a kind of middle ground where a central linear algorithm e.g. linear regression, is trained on many variations of the original features, by automated generation and filtering of transformed features. The most general variants of this approach are not hugely popular because they suffer from same risks of overfitting as non-linear models whilst not offering much in the way of improved performance. However, if you narrow down the types of feature and transformation combinations based on some knowledge of how you expect the target function to behave, it leads to many useful variants of linear regression - e.g. regression on fourier transforms, radial basis functions etc.