4
$\begingroup$

The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper does not include the proofs of the theorem. Does anybody know where the proof can be found?

$\endgroup$
4
$\begingroup$

There are several papers related to the topic, because there have been several attempts to show this from slightly different perspectives and using slightly different assumptions (e.g. assuming that certain activation functions are used).

The article A visual proof that neural nets can compute any function (by Michael Nielsen) should give you some intuition behind the universality of neural networks.

However, most other papers related to the topic are quite technical, so they might be hard to follow. Nonetheless, below there are some links to some possibly useful papers.

See, for example, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function (1993), by Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, Shimon Schocken), which shows that feedforward neural networks are universal function approximators. More specifically, an FFNN equipped with a locally bounded piecewise continuous activation function can approximate any continuous function to any degree of accuracy, if and only if the FFNN's activation function is not a polynomial.

You can also see Universality of Deep Convolutional Neural Networks (2018), by Ding-Xuan Zhou, which shows that convolutional neural networks can approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough, which makes them universal function approximators.

See also page 632 of Recurrent Neural Networks Are Universal Approximators (2006), by Anton Maximilian Schäfer and Hans Georg Zimmermann, which shows that recurrent neural networks are universal function approximators. See also https://stats.stackexchange.com/a/221142/82135.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.