New
Stack Overflow Jobs powered by Indeed: A job site that puts thousands of tech jobs at your fingertips (U.S. only). Search jobs

# Tag Info

## Hot answers tagged universal-approximation-theorems

Accepted

### Where can I find the proof of the universal approximation theorem?

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different ...
• 40.8k

### Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?

There is stuff like the Universal Approximation Theorem. There are also investigations into the loss surface of neural networks. And classics like this explanation of the vanishing gradient problem....
• 4,275
Accepted

### Smallest possible network to approximate the $sin$ function

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh ...
• 420

### Where can I find the proof of the universal approximation theorem?

"Modern" Guarantees for Feed-Forward Neural Networks My answer will complement nbro's above, which gave a very nice overview of universal approximation theorems for different types of ...
• 565
Accepted

### What makes the approximation capabilities of neural networks different than something like, say, Fourier series?

People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. There is an opinion that this ...
• 2,534
Accepted

### Which machine learning models are universal function approximators?

Support vector machines In the paper A Note on the Universal Approximation Capability of Support Vector Machines (2002) B. Hammer and K. Gersmann investigate the universal function approximation ...
• 40.8k
Accepted

### Why do we need the identify function when approximating a function with a neural network with multiple layers?

This says that if you can approximate a function with one layer, you can also approximate it with multiple layers because you can make the extra layers do nothing. The universal approximation theorem ...

### What makes the approximation capabilities of neural networks different than something like, say, Fourier series?

It's worth noting that the Fourier series analogy was used in early explorations of universal approximation theorems https://ieeexplore.ieee.org/document/23903.
• 56

### Why do activation functions in neural networks have to be non-polynomial to approximate any function?

Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs ...
• 2,277
Accepted

### Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

The paper Multilayer feedforward networks with a nonpolynomial activation function can approximate any function (by Leshno et al., 1993) provides a theorem claiming this and the (quite long) proof of ...
• 40.8k

### What are the learning limitations of neural networks trained with backpropagation?

Multilayer Perceptron (MLP) can theoretically approximate any bounded, continuous function. There's no guarantee for a discontinuous function. There are plenty of important discontinuous functions, ...
• 410

### Where can I find the proof of the universal approximation theorem?

Just wanted to add that the new text Deep Learning Architectures A Mathematical Approach mentions this result, but I'm not sure if it gives a proof. It does mention an improved result by Hanin (http://...

### What are the learning limitations of neural networks trained with backpropagation?

While I'm not familiar with any explicit statements regarding what a Multilayer Perceptron (MLP) cannot learn, I can provide some further detail on the positive statements you made about MLP ...
• 7,266

### How can "any process you can imagine" be thought of as function computation?

A function is simply a procedure that maps a particular input to a particular output. You put in $X$, and the function computes $Y$. Those $X$ and $Y$ can take many different forms. It could be ...

### Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?

This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of ...
1 vote

### Neural Networks are universal approximators? - Exercice 20.1 UML

$\beta$ is the size of all the intervals used to partition the input space, and thus $(2/\beta)$ is the number of intervals along each dimension. $d$ is the number of input space dimensions actually ...
• 2,277
1 vote

### Are the capabilities of connectionist AI and symbolic AI the same?

Are the capabilities of connectionist AI and symbolic AI the same? No, not usually. Why not usually? Neural networks (connectionist AI) are usually used for inductive reasoning (i.e. the process of ...
• 40.8k
1 vote

• 310
1 vote
Accepted

### Is there a way to calculate the closed-form expression of the function that a neural network computes?

To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ ...
• 148

Only top scored, non community-wiki answers of a minimum length are eligible