# Why are neural networks optimized instead of just optimizing a high dimensional function?

I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high enough order.

So, my question is that, why can we not just optimize some high parameter count (or high dimensional) function instead? I am using a Taylor series just as an example, it can be any type of high dimensional function, and they all can be tuned with Backprop/gradient descent. I know there is lots of empirical evidence out their proving neural networks to win out over other types of functions, But I just cannot seem to understand why this is. Why does something that vaguely resembles real neurons work so well over other functions? What is the logic?

PS - Maybe a dumb question, I am just a beginner that currently only sees machine learning as a calculus optimization problem :)

– nbro
Commented Jun 15 at 15:45
• Main question is why prefer neural networks over any old function Commented Jun 15 at 15:48
• You might like this arXiv paper by Chang (though Matloff seems to be the author most associated with it): Polynomial Regression As an Alternative to Neural Nets.
– Dave
Commented Jun 25 at 9:54
• In theory there're many universal approximators, but in practice polynomial or Fourier series are too rigid globally while too smooth locally, you'd better need many more units than a MLP for natural problems. Commented Jun 27 at 22:51

$$A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$$ where theta describes our parameters $$A_0 \cdot \cdot \cdot A_k$$ and $$\sigma$$ is some activation function like a relu or a sigmoid.
• $p(x \mid \theta) = A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$, this is very confusing, it seems like the NN is the PDF of a density, which is definitely not Commented Jun 16 at 11:58