11
$\begingroup$

The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper does not include the proofs of the theorem. Does anybody know where the proof can be found?

$\endgroup$
13
$\begingroup$

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different perspectives and using slightly different assumptions (e.g. assuming that certain activation functions are used). Note that these proofs tell you that neural networks can approximate any continuous function, but they do not tell you exactly how you need to train your neural network so that it approximates your desired function. Moreover, most papers on the topic are quite technical and mathematical, so, if you do not have a solid knowledge of approximation theory and related fields, they may be difficult to read and understand. Nonetheless, below there are some links to some possibly useful articles and papers.

The article A visual proof that neural nets can compute any function (by Michael Nielsen) should give you some intuition behind the universality of neural networks, so this is probably the first article you should read.

Then you should probably read the paper Approximation by Superpositions of a Sigmoidal Function (1989), by G. Cybenko, who proves that multi-layer perceptrons (i.e. feed-forward neural networks with at least one hidden layer) can approximate any continuous function. However, he assumes that the neural network uses sigmoid activations functions, which, nowadays, have been replaced in many scenarios by ReLU activation functions. Other works (e.g. [1, 2]) showed that you don't necessarily need sigmoid activation functions, but only certain classes of activation functions do not make neural networks universal.

The universality property (i.e. the ability to approximate any continuous function) has also been proved in the case of convolutional neural networks. For example, see Universality of Deep Convolutional Neural Networks (2018), by Ding-Xuan Zhou, which shows that convolutional neural networks can approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough.

See also page 632 of Recurrent Neural Networks Are Universal Approximators (2006), by A. M. Schäfer et al., which shows that recurrent neural networks are universal function approximators. See also https://stats.stackexchange.com/a/221142/82135.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Just wanted to add that the new text Deep Learning Architectures A Mathematical Approach mentions this result, but I'm not sure if it gives a proof. It does mention an improved result by Hanin (http://arxiv.org/abs/1708.02691) for which I think it does give at least a partial proof. The original paper by Hanin seems to omit some proofs as well, but the published version (https://www.mdpi.com/2227-7390/7/10/992/htm) may be more complete.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I've found the proof eventually. It turned out to be very technical and is only interesting from a theoretical view point. What about the book that you mention ? Do you read it ? Can you recommend it ? @loren-rosen $\endgroup$ – Leroy Od Jul 14 at 8:38
  • $\begingroup$ I've only read the preface, and looked to see if it had the proof. Anyway, I do see that the conference proceedings have a link to an appendix with the proof (see papers.nips.cc/paper/…). (Note to self: update the wikipedia entry.) Also, of course, Google Scholar has lots of papers that reference it, are related, etc. $\endgroup$ – Loren Rosen Jul 17 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.