# What is the difference between neural networks and other ways of curve fitting?

For simplicity, let's assume we want to solve a regression problem, where we have one independent variable and one dependent variable, which we want to predict. Let's also assume that there is a nonlinear relationship between the independent and dependent variables.

No matter the way we do it, we just need to build a proper curved line based on existing observations, such that the prediction is the best.

I know we can solve this problem with neural networks, but I also know other ways to create such curves. For example:

1. splines

2. kriging

3. lowess

4. Something I think would also work (do not know if exists): fitting curve using a series of Fourier sine waves, and so on

My questions are:

1. Is it true that neural networks are just one of the ways to fit a non-linear curve to the data?

2. What are the advantages and disadvantages of choosing a neural network over other approaches? (maybe it becomes better when I have many independent variables, and another little guess: maybe the neural network is better in omitting the effect of linear dependent input variables?)

2. The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a couple of matrix products (the $$\hat{\beta}=(X^TX)^{-1}X^Ty$$).