Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [universal-approximation-theorems]

For questions related to the (different) universal approximation theorems (UATs), for example, in the context of neural networks.

30 questions
Filter by
Sorted by
Tagged with
16k views

### Where can I find the proof of the universal approximation theorem?

The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
• 475
1k views

### What is the number of neurons required to approximate a polynomial of degree n?

I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number ...
• 773
783 views

### What are the learning limitations of neural networks trained with backpropagation?

In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function. This was solved by the backpropagation network with at least one hidden layer. This type of network ...
2k views

### Which machine learning models are universal function approximators?

The universal approximation theorem states that a feed-forward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some ...
• 40.8k
488 views

### Smallest possible network to approximate the $sin$ function

The main goal is: Find the smallest possible neural network to approximate the $sin$ function. Moreover, I want to find a qualitative reason why this network is the smallest possible network. I have ...
• 75
194 views

### What makes the approximation capabilities of neural networks different than something like, say, Fourier series?

People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable ...
479 views

### Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
• 53
282 views

### Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer. The proof goes like this: Create a "bump" function using 2 neurons. Create (infinitely) many of these ...
• 133
1k views

### Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?

The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a mathematical proof that shows that certain ...
230 views

### Does a neural network exist that can learn every possible training data?

The universal approximation theorem states, that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $R^n$...
• 143
733 views

### Is there a way to calculate the closed-form expression of the function that a neural network computes?

As stated in the universal approximation theorem, a neural network can approximate almost any function. Is there a way to calculate the closed-form (or analytical) expression of the function that a ...
• 133
124 views

### Are No Free Lunch theorem and Universal Approximation theorem contradictory in the context of neural networks?

To my understanding NFL states that, we cannot have an hypothesis (let's assume it is an approximator like NN in this case) class that can't achieve certain accuracy parameters $\leq \epsilon$ with ...
495 views

### Why do activation functions in neural networks have to be non-polynomial to approximate any function?

Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
334 views

### Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?

I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?
• 21
124 views

### Neural Networks are universal approximators? - Exercice 20.1 UML

I'm working on this question which can be found at page 282 of "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David. The statement is as ...
• 21
378 views

### How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?

Neural networks typically have $\mathcal{VC}$ dimension that is proportional to their number of parameters and inputs. For example, see the papers Vapnik-Chervonenkis dimension of recurrent neural ...
• 40.8k
1 vote
67 views

### Why do we need the identify function when approximating a function with a neural network with multiple layers?

I have a question about the explanation of universal approximation theorem provided by wikipedia. https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29 It says, after a ...
1 vote
37 views

### How can "any process you can imagine" be thought of as function computation?

I stumbled upon this passage when reading this guide. Universality theorems are a commonplace in computer science, so much so that we sometimes forget how astonishing they are. But it's worth ...
• 39
1 vote
204 views

### Are the capabilities of connectionist AI and symbolic AI the same?

The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
• 3,890
1 vote
79 views

### Is there any paper that shows that multi-channel neural networks are universal approximators?

Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where ...
• 13
1 vote
116 views

### Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
• 39
1 vote
103 views

### Can most of the basic machine learning models be easily represented as simple neural network architectures?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following: Different ...
• 579
1 vote
174 views

### When are multiple hidden layers necessary?

I know that my question probably seems like being asked many times, but Ill try to be more speciffic: Limitations to my question: I am NOT asking about convolutional neural networks, so please, try ...
• 181
1 vote
482 views

### If a neural network is a universal function approximator, can it have any prior beliefs?

Let us confine ourselves to the case where we have a $n$ dimensional input and a $+1$ or $-1$ output. It can be shown that: For every $n$, there exists a dense NN of depth 2, such that it contains ...
1 vote
110 views

### Is there a mathematical proof that a binary neural network can approximate any function with arbitrary accuracy?

Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant ...
1 vote
76 views

### Is the capability of RNN more than the capability of MLP?

Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
• 3,890
1 vote
75 views

### Does Godel's incompleteness theorems restricts the scope of connectionist-AI?

It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic. But, I am wondering whether it has any impact on the ...
• 3,890
1 vote
133 views

### Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?

The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
### Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?
There were some posts that using RNN can predict the next point of the sine wave function with data history. However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, \$\...