6
$\begingroup$

For simplicity, let's assume we want to solve a regression problem, where we have one independent variable and one dependent variable, which we want to predict. Let's also assume that there is a nonlinear relationship between the independent and dependent variables.

No matter the way we do it, we just need to build a proper curved line based on existing observations, such that the prediction is the best.

I know we can solve this problem with neural networks, but I also know other ways to create such curves. For example:

  1. splines

  2. kriging

  3. lowess

  4. Something I think would also work (do not know if exists): fitting curve using a series of Fourier sine waves, and so on

My questions are:

  1. Is it true that neural networks are just one of the ways to fit a non-linear curve to the data?

  2. What are the advantages and disadvantages of choosing a neural network over other approaches? (maybe it becomes better when I have many independent variables, and another little guess: maybe the neural network is better in omitting the effect of linear dependent input variables?)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

  1. In some sense, you're right that a neural net is just another tool to fit data. However, it's quite the tool! There's this universal approximation theorem saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all over if that's the right trend.

  2. The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has badly overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a few matrix products (the $\hat{\beta}=(X^TX)^{-1}X^Ty$).

Improve this answer
$\endgroup$
8
  • 2
    $\begingroup$ Not an expert but is the universal approximation theorem really saying something very special about neural networks? Paraphrasing a neural network expert: "The universal approximation theorem does NOT guarantee that you will converge to an arbitrarily good approximation of a wide class functions. It just means that this approximation is in the search space. There is no guarantee you will converge to it and in fact you almost never will." This sounds very similar to guarantees that are available when fitting with many other bases (sin/cos, polynomials, etc...) $\endgroup$
    – Kvothe
    Commented Jul 14, 2023 at 16:14
  • $\begingroup$ @Kvothe It is not clear if that quote refers to convergence in terms of training (converging to the optimum for a given architecture) or convergence of network to a function as more and more nodes are added to it. I am on board with the former but not the latter. The universal approximation theorems say, much like the Stone-Weierstrass theorem for polynomials or Carlson's theorem for Fourier series, that you can approximate any function (of a particular class, specified by the theorem details) by making a sufficiently complex neural network. $\endgroup$
    – Dave
    Commented Jul 14, 2023 at 16:36
  • 1
    $\begingroup$ I think I meant the former. In the latter part of your reply: Are you agreeing that there is nothing too special about neural networks and that universal approximation theorem basically just gives you a completeness of your basis that you also get in most other fitting methods? $\endgroup$
    – Kvothe
    Commented Jul 14, 2023 at 16:42
  • $\begingroup$ @Kvothe Completeness as in Cauchy completeness? $\endgroup$
    – Dave
    Commented Jul 14, 2023 at 16:58
  • $\begingroup$ I might be incorrectly borrowing language from intuitively similar spaces (e.g. Hilbert spaces). With "complete basis" in this case I mean a set of functions which give an arbitrarily good approximation for any function in a certain class of functions (the space for which these functions act as a "basis"). $\endgroup$
    – Kvothe
    Commented Jul 17, 2023 at 2:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .