What is the difference between neural networks and other ways of curve fitting?

For simplicity, let's assume we want to solve a regression problem, where we have one independent variable and one dependent variable, which we want to predict. Let's also assume that there is a nonlinear relationship between the independent and dependent variables.

No matter the way we do it, we just need to build a proper curved line based on existing observations, such that the prediction is the best.

I know we can solve this problem with neural networks, but I also know other ways to create such curves. For example:

1. splines

2. kriging

3. lowess

4. Something I think would also work (do not know if exists): fitting curve using a series of Fourier sine waves, and so on

My questions are:

1. Is it true that neural networks are just one of the ways to fit a non-linear curve to the data?

2. What are the advantages and disadvantages of choosing a neural network over other approaches? (maybe it becomes better when I have many independent variables, and another little guess: maybe the neural network is better in omitting the effect of linear dependent input variables?)

1. In some sense, you're right that a neural net is just another tool to fit data. However, it's quite the tool! There's this universal approximation theorem saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all over if that's the right trend.

2. The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a couple of matrix products (the $$\hat{\beta}=(X^TX)^{-1}X^Ty$$).

1. It is indeed true that neural networks are just another ways of curve fitting. In fact, as I learned regression after I learned neural networks, I shout out "neural networks are just more sophisticated curve fitting!". However, as Dave said, it can approximate any function in practice. See Google's neural net playground for an interesting animation. You can see how the curve is fitted to the data you have there.

2. The main advantage of neural networks, and one of the primary reasons why they are favored in industry, is that they can predict highly abstract-structured and nonlinear data. The mathematics of this is explained clearly by Dave. I do not think they are any better than other models though, as they do have the same cons of other distance-based models:

• They are prone to converging to local minima if your training data implies a non-convex function.
• The data should be carefully labeled. You might even need to mislabel some data points to reduce overfitting.
• The time you train a neural networks can be awfully long. Most real-life applications require advanced computers. Although this is a practical problem, it is important as it reduces the effect of theoretical advances.
• Feature selection and feature engineering are still two of the most important processes as with any ML problem. More features does not mean better performance in neural networks. The model still is subject to multicollinearity, for example.
• The selection of hyperparameters depend on the nature of the data you have. Therefore, there aren't "one fits all" solutions. This is not a mathematical disadvantage but a practical one.
• Neural nets are scale variant. This takes us back to the fourth point.

However, the fact that it can approximate almost every function compensates for all these disadvantages.

neural networks can solve all taylor series polynomials meaning a NN is an generalized linear model. Most function f(y) can be solved with neural networks. However, many matrix operations can not be generalized for a neural network to solve like determinants. Operations like rotation, scale, and transform also can not be generalized.

you can solve all ordinary least square formulas and general linear model formulas using a neural network. There most n order polynomial curve fitting can be solved by a neural network: spline, b-splines, nurbs can be solved by a neural network.