19
$\begingroup$

I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number of neurons. Or mathematically, ${|g(x)−f(x)|< \epsilon}$, where ${g(x)}$ is the approximation, ${f(x)}$ is the target function and is $\epsilon$ is an arbitrary bound.

A polynomial of degree $n$ has at maximum $n-1$ turning points (where the derivative of the polynomial changes sign). With each new turning point, the approximation seems to become more complex.

I'm not necessarily looking for a formula, but I'd like to get a general idea on how to figure out the sufficient number of neurons is for a reasonable approximation of a polynomial with a single layer of the neural network (you may consider "reasonable" to be $\epsilon = 0.0001$). To ask in other words, how would adding one more neuron affect the model's ability to express a polynomial?

Improve this question
$\endgroup$
3
  • 15
    $\begingroup$ If the network has one hidden layer, there are papers like A closer look at the approximation capabilities of neural networks (Kai Fong Ernest Chong) saying "... given approximation threshold $\epsilon$ ... if $X$ in $\mathbb R^n$ is compact, then a neural network with $n$ input units, $m$ output units, and a single hidden layer with $\binom{n+d}{d}$ hidden units (independent of $m$ and $\epsilon$), can uniformly approximate any polynomial function $f:X \to \mathbb R^m$ whose total degree is at most $d$ for each of its $m$ coordinate functions." $\endgroup$
    – Vepir
    Jan 4, 2021 at 15:38
  • 1
    $\begingroup$ @Vepir - that sounds much more like an answer than a comment. $\endgroup$
    – EngrStudent
    Nov 16, 2022 at 19:24
  • 1
    $\begingroup$ Not an answer, but I find that simple neural networks are quite expressive. I suspect that any complete answer will have to address the network topology in some way. $\endgroup$
    – Galen
    Mar 15 at 1:05

1 Answer 1

Reset to default
1
$\begingroup$

Not sure this is answering your question but, one way to get a general idea of the number of neurons needed is to consider the number of turning points in the polynomial.

Each turning point in a polynomial corresponds to a change in the sign of its derivative, and the more turning points a polynomial has, the more complex its shape. For a polynomial of degree n, there can be at most n-1 turning points. This means that a polynomial with many turning points will require more neurons in the hidden layer to be accurately approximated.

In general, increasing the number of neurons in the hidden layer can improve the model's ability to express a polynomial. Adding one more neuron to the hidden layer can allow the network to capture more complex patterns in the data, which can lead to a more accurate approximation of the polynomial.

I have found these two sources very useful in visualising the above. Hope it helps.

http://neuralnetworksanddeeplearning.com/chap4.html https://cs.stanford.edu/people/karpathy/convnetjs/demo/regression.html

Improve this answer
$\endgroup$

You must log in to answer this question.