# Why are neural networks preferred to other classification functions optimized by gradient decent

Consider a neural network, e.g. as presented by Nielsen here. Abstractly, we just construct some function $$f: \mathbb{R}^n \to [0,1]^m$$ for some $$n,m \in \mathbb{N}$$ (i.e. the dimensions of the input and output space) that depends on a large set of parameters, $$p_j$$. We then just define the cost function $$C$$ and calculate $$\nabla_p C$$ and just map $$p \to p - \epsilon \nabla_p C$$ repeatedly.

The question is why do we choose $$f$$ to be what it is in standard neural networks, e.g. a bunch of linear combinations and sigmoids? One answer is that there a theorem saying any suitably nice function can be approximated using neural networks. But the same is true of other types of functions $$f$$. The Stone-Weierstrass theorem gives that we could use polynomials in $$n$$ variables: $$f(x) = c^0_0 + (c^1_1 x_1 + c^1_2 x_2 + \cdots + c^1_n x_n) + (c^2_{11}x_1 x_1 + c^2_{12} x_1x_2 + \cdots + c^2_{1n} x_1 x_2 + c^2_{21} x_2x_1 + c^2_{22} x_2x_2 + \cdots) + \cdots,$$

and still have a nice approximation theorem. Here the gradient would be even easier to calculate. Why not use polynomials?

• Because polynomials are limited by their degree. The approximation ability of functions are generally given by a measure called VC dimension, and 2 degree polynomials have VC dimension 2/3 (I forget the exact number). So you basically have to take infinite degree polynomial combinations. NNs can be somewhat flexible in this regard that you don't have to manually choose functions, you can pretty much approximate a polynomial within an interval given sufficient nodes. This is the general theory, there are much more detailed nuances which goes against this aforementioned theory. – DuttaA Aug 29 '20 at 21:38
• Here is a related question What are the differences between artificial neural networks and other function approximators?. – nbro Aug 30 '20 at 2:34

You can indeed fit a polynomial to your labelled data, which is known as polynomial regression (which can e.g. be done with the function numpy.polyfit). One apparent limitation of polynomial regression is that, in practice, you need to assume that your data follows some specific polynomial of some degree $$n$$, i.e. you assume that your data has the form of the polynomial that you choose, which may not be true.