For simplicity, let's assume we want to solve a regression problem, where we have one independent variable and one dependent variable, which we want to predict. Let's also assume that there is a nonlinear relationship between the independent and dependent variables.

No matter the way we do it, we just need to build a proper curved line based on existing observations, such that the prediction is the best.

I know we can solve this problem with neural networks, but I also know other ways to create such curves. For example:

  1. splines

  2. kriging

  3. lowess

  4. Something I think would also work (do not know if exists): fitting curve using a series of Fourier sine waves, and so on

My questions are:

  1. Is it true that neural networks are just one of the ways to fit a non-linear curve to the data?

  2. What are the advantages and disadvantages of choosing a neural network over other approaches? (maybe it becomes better when I have many independent variables, and another little guess: maybe the neural network is better in omitting the effect of linear dependent input variables?)

  1. In some sense, you're right that a neural net is just another tool to fit data. However, it's quite the tool! There's this universal approximation theorem saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all over if that's the right trend.

  2. The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a couple of matrix products (the $\hat{\beta}=(X^TX)^{-1}X^Ty$).


neural networks can solve all taylor series polynomials meaning a NN is an generalized linear model. Most function f(y) can be solved with neural networks. However, many matrix operations can not be generalized for a neural network to solve like determinants. Operations like rotation, scale, and transform also can not be generalized.

you can solve all ordinary least square formulas and general linear model formulas using a neural network. There most n order polynomial curve fitting can be solved by a neural network: spline, b-splines, nurbs can be solved by a neural network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.