Skip to main content
New
Stack Overflow Jobs powered by Indeed: A job site that puts thousands of tech jobs at your fingertips (U.S. only). Search jobs
31 votes
Accepted

Where can I find the proof of the universal approximation theorem?

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different ...
nbro's user avatar
  • 40.8k
7 votes

Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?

There is stuff like the Universal Approximation Theorem. There are also investigations into the loss surface of neural networks. And classics like this explanation of the vanishing gradient problem....
BlindKungFuMaster's user avatar
6 votes
Accepted

Smallest possible network to approximate the $sin$ function

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh ...
amin's user avatar
  • 420
6 votes

Where can I find the proof of the universal approximation theorem?

"Modern" Guarantees for Feed-Forward Neural Networks My answer will complement nbro's above, which gave a very nice overview of universal approximation theorems for different types of ...
ABIM's user avatar
  • 565
4 votes
Accepted

What makes the approximation capabilities of neural networks different than something like, say, Fourier series?

People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. There is an opinion that this ...
Kostya's user avatar
  • 2,534
4 votes
Accepted

Which machine learning models are universal function approximators?

Support vector machines In the paper A Note on the Universal Approximation Capability of Support Vector Machines (2002) B. Hammer and K. Gersmann investigate the universal function approximation ...
nbro's user avatar
  • 40.8k
4 votes
Accepted

Why do we need the identify function when approximating a function with a neural network with multiple layers?

This says that if you can approximate a function with one layer, you can also approximate it with multiple layers because you can make the extra layers do nothing. The universal approximation theorem ...
Quitting Due To Antisemitism's user avatar
3 votes

What makes the approximation capabilities of neural networks different than something like, say, Fourier series?

It's worth noting that the Fourier series analogy was used in early explorations of universal approximation theorems https://ieeexplore.ieee.org/document/23903.
Luke's user avatar
  • 56
3 votes

Why do activation functions in neural networks have to be non-polynomial to approximate any function?

Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs ...
cinch's user avatar
  • 2,277
3 votes
Accepted

Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

The paper Multilayer feedforward networks with a nonpolynomial activation function can approximate any function (by Leshno et al., 1993) provides a theorem claiming this and the (quite long) proof of ...
nbro's user avatar
  • 40.8k
3 votes

What are the learning limitations of neural networks trained with backpropagation?

Multilayer Perceptron (MLP) can theoretically approximate any bounded, continuous function. There's no guarantee for a discontinuous function. There are plenty of important discontinuous functions, ...
bpachev's user avatar
  • 410
2 votes

Where can I find the proof of the universal approximation theorem?

Just wanted to add that the new text Deep Learning Architectures A Mathematical Approach mentions this result, but I'm not sure if it gives a proof. It does mention an improved result by Hanin (http://...
Loren Rosen's user avatar
2 votes

What are the learning limitations of neural networks trained with backpropagation?

While I'm not familiar with any explicit statements regarding what a Multilayer Perceptron (MLP) cannot learn, I can provide some further detail on the positive statements you made about MLP ...
NietzscheanAI's user avatar
2 votes

How can "any process you can imagine" be thought of as function computation?

A function is simply a procedure that maps a particular input to a particular output. You put in $X$, and the function computes $Y$. Those $X$ and $Y$ can take many different forms. It could be ...
Nuclear Hoagie's user avatar
2 votes

Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?

This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of ...
David Hoelzer's user avatar
1 vote

Neural Networks are universal approximators? - Exercice 20.1 UML

$\beta$ is the size of all the intervals used to partition the input space, and thus $(2/\beta)$ is the number of intervals along each dimension. $d$ is the number of input space dimensions actually ...
cinch's user avatar
  • 2,277
1 vote

Are the capabilities of connectionist AI and symbolic AI the same?

Are the capabilities of connectionist AI and symbolic AI the same? No, not usually. Why not usually? Neural networks (connectionist AI) are usually used for inductive reasoning (i.e. the process of ...
nbro's user avatar
  • 40.8k
1 vote

Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Like the accepted answer, I'm assuming you are referring to (literature refering to) Leshno et al, 1993. That paper only concerns 1-layer neural networks, and those are simply of the form $$x\mapsto\...
Teun's user avatar
  • 111
1 vote
Accepted

Is there any paper that shows that multi-channel neural networks are universal approximators?

Yes, there is such a statement, valid even in a bit more general setting. Any function, equivariant to a certain symmetry, can be approximated arbitrarily well, provided that the number of parameters ...
spiridon_the_sun_rotator's user avatar
1 vote

Issue with graphical interpretation of the universal approximation theorem

The classical version of the universal approximation theorem states that, roughly, given a continuous function $f \colon [0, 1]^n \to [0, 1]^n$, there exists a single layer neural network and a set of ...
htl's user avatar
  • 1,010
1 vote

Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not ...
KoKlA's user avatar
  • 133
1 vote

What is the number of neurons required to approximate a polynomial of degree n?

Not sure this is answering your question but, one way to get a general idea of the number of neurons needed is to consider the number of turning points in the polynomial. Each turning point in a ...
hH1sG0n3's user avatar
  • 211
1 vote
Accepted

Can most of the basic machine learning models be easily represented as simple neural network architectures?

I think the author refers to both different choices of activation function and loss. It is explained in more detail in chapter 2. In particular 2.3 is ilustrative of this point. I don't think there is ...
Dani's user avatar
  • 36
1 vote
Accepted

When are multiple hidden layers necessary?

A very wide but shallow neural network is going to be harder to train. You can check that with the playground of tensorflow or with the MPG example in Google Colab. The relationship between ...
Joaquin Torrens's user avatar
1 vote

How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?

VC Dimension of Neural Networks establishes VC bounds depending on the number of weights, whereas the UAT refers to a class of neural networks in which the number of weights a particular network can ...
Dani's user avatar
  • 36
1 vote

If a neural network is a universal function approximator, can it have any prior beliefs?

I think your deduction is mostly correct. Neural networks of depth are universal function approximators. This means that in principal, for any function of the form you describe, there's a NN that ...
John Doucette's user avatar
1 vote
Accepted

Does a neural network exist that can learn every possible training data?

The branch of AI research that answers questions like this is called computational learning theory. For the specific question you have asked, the universal approximation theorem does indeed prove ...
John Doucette's user avatar
1 vote

Is there a way to calculate the closed-form expression of the function that a neural network computes?

As opposed to what is written in this answer, you can have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take ...
nbro's user avatar
  • 40.8k
1 vote

Is there a way to calculate the closed-form expression of the function that a neural network computes?

The network is the function. A network is a function, that is modeled by terms describing the architecture and coefficients that are learned. Look at a simple model: $$f(x) = ax+b$$ Your solver ...
allo's user avatar
  • 310
1 vote
Accepted

Is there a way to calculate the closed-form expression of the function that a neural network computes?

To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ ...
dcolazin's user avatar
  • 148

Only top scored, non community-wiki answers of a minimum length are eligible