13

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different perspectives and using slightly different assumptions (e.g. assuming that certain activation functions are used). Note that these proofs tell you that neural ...


7

There is stuff like the Universal Approximation Theorem. There are also investigations into the loss surface of neural networks. And classics like this explanation of the vanishing gradient problem. But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions ...


4

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero). Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do ...


1

Support vector machines In the paper A Note on the Universal Approximation Capability of Support Vector Machines (2002) B. Hammer and K. Gersmann investigate the universal function approximation capabilities of SVMs. More specifically, the authors show that SVMs with standard kernels (including Gaussian, polynomial, and several dot product kernels) can ...


Only top voted, non community-wiki answers of a minimum length are eligible