I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high enough order.

So, my question is that, why can we not just optimize some high parameter count (or high dimensional) function instead? I am using a Taylor series just as an example, it can be any type of high dimensional function, and they all can be tuned with Backprop/gradient descent. I know there is lots of empirical evidence out their proving neural networks to win out over other types of functions, But I just cannot seem to understand why this is. Why does something that vaguely resembles real neurons work so well over other functions? What is the logic?

PS - Maybe a dumb question, I am just a beginner that currently only sees machine learning as a calculus optimization problem :)

  • $\begingroup$ You're asking many questions here. What is your main question? $\endgroup$
    – nbro
    Commented Jun 15 at 15:45
  • $\begingroup$ Main question is why prefer neural networks over any old function $\endgroup$
    – Yash Nath
    Commented Jun 15 at 15:48
  • $\begingroup$ You might like this arXiv paper by Chang (though Matloff seems to be the author most associated with it): Polynomial Regression As an Alternative to Neural Nets. $\endgroup$
    – Dave
    Commented Jun 25 at 9:54
  • $\begingroup$ In theory there're many universal approximators, but in practice polynomial or Fourier series are too rigid globally while too smooth locally, you'd better need many more units than a MLP for natural problems. $\endgroup$
    – cinch
    Commented Jun 27 at 22:51

1 Answer 1


First of all, a neural network can easily be described as some high dimensional function:

$$A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$$ where theta describes our parameters $A_0 \cdot \cdot \cdot A_k$ and $\sigma$ is some activation function like a relu or a sigmoid.

For the answer to your second question, as to why we use neural networks instead of other methods, is that we don't really know why neural networks work, they just seem to work well empirically better than other methods despite the theoretical evidence. In fact if you ask top figures in this field, you'll pretty much get the same the answer

  • $\begingroup$ $p(x \mid \theta) = A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$, this is very confusing, it seems like the NN is the PDF of a density, which is definitely not $\endgroup$
    – Alberto
    Commented Jun 16 at 11:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .