Questions tagged [universal-approximation-theorems]
For questions related to the (different) universal approximation theorems (UATs), for example, in the context of neural networks.
28
questions
3
votes
2
answers
61
views
What makes the approximation capabilities of neural networks different than something like, say, Fourier series?
People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable ...
- 131
1
vote
1
answer
55
views
Is there a mathematical proof that a binary neural network can approximate any function with arbitrary accuracy?
Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant ...
- 33
2
votes
1
answer
151
views
Why do activation functions in neural networks have to be non-polynomial to approximate any function?
Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
- 31
1
vote
0
answers
59
views
Is the capability of RNN more than the capability of MLP?
Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
- 3,481
1
vote
0
answers
47
views
Does Godel's incompleteness theorems restricts the scope of connectionist-AI?
It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic.
But, I am wondering whether it has any impact on the ...
- 3,481
1
vote
1
answer
95
views
Are the capabilities of connectionist AI and symbolic AI the same?
The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
- 3,481
4
votes
1
answer
281
views
Why does the activation function for a hidden layer in a MLP have to be non-polynomial?
Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial.
Is there a ...
- 43
1
vote
0
answers
110
views
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
1
vote
1
answer
66
views
Is there any paper that shows that multi-channel neural networks are universal approximators?
Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where ...
- 13
2
votes
1
answer
222
views
Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?
I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?
- 21
0
votes
1
answer
182
views
Why can a neural network use more than one activation function?
From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
- 39
1
vote
1
answer
69
views
Issue with graphical interpretation of the universal approximation theorem
This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
- 39
1
vote
2
answers
37
views
How can "any process you can imagine" be thought of as function computation?
I stumbled upon this passage when reading this guide.
Universality theorems are a commonplace in computer science, so much
so that we sometimes forget how astonishing they are. But it's worth
...
- 39
0
votes
1
answer
85
views
Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?
There were some posts that using RNN can predict the next point of the sine wave function with data history.
However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\...
5
votes
2
answers
251
views
Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:
Create a "bump" function using 2 neurons.
Create (infinitely) many of these ...
- 133
17
votes
0
answers
580
views
What is the number of neurons required to approximate a polynomial of degree n?
I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number ...
- 723
1
vote
1
answer
91
views
Can most of the basic machine learning models be easily represented as simple neural network architectures?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following:
Different ...
- 477
6
votes
1
answer
316
views
Smallest possible network to approximate the $sin$ function
The main goal is: Find the smallest possible neural network to approximate the $sin$ function.
Moreover, I want to find a qualitative reason why this network is the smallest possible network.
I have ...
- 75
1
vote
1
answer
136
views
When are multiple hidden layers necessary?
I know that my question probably seems like being asked many times, but Ill try
to be more speciffic:
Limitations to my question:
I am NOT asking about convolutional neural networks, so please, try ...
- 159
2
votes
1
answer
297
views
How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?
Neural networks typically have $\mathcal{VC}$ dimension that is proportional to their number of parameters and inputs. For example, see the papers Vapnik-Chervonenkis dimension of recurrent neural ...
- 37k
1
vote
1
answer
267
views
If a neural network is a universal function approximator, can it have any prior beliefs?
Let us confine ourselves to the case where we have a $n$ dimensional input and a $+1$ or $-1$ output. It can be shown that:
For every $n$, there exists a dense NN of depth 2, such that it contains ...
user9947
2
votes
0
answers
109
views
Are No Free Lunch theorem and Universal Approximation theorem contradictory in the context of neural networks?
To my understanding NFL states that, we cannot have an hypothesis (let's assume it is an approximator like NN in this case) class that can't achieve certain accuracy parameters $\leq \epsilon$ with ...
user9947
4
votes
1
answer
179
views
Does a neural network exist that can learn every possible training data?
The universal approximation theorem states, that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $R^n$...
- 143
25
votes
3
answers
11k
views
Where can I find the proof of the universal approximation theorem?
The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
- 415
3
votes
3
answers
480
views
Is there a way to calculate the closed-form expression of the function that a neural network computes?
As stated in the universal approximation theorem, a neural network can approximate almost any function.
Is there a way to calculate the closed-form (or analytical) expression of the function that a ...
- 133
8
votes
1
answer
2k
views
Which machine learning models are universal function approximators?
The universal approximation theorem states that a feed-forward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some ...
- 37k
4
votes
3
answers
1k
views
Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?
The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a mathematical proof that shows that certain ...
10
votes
2
answers
715
views
What are the learning limitations of neural networks trained with backpropagation?
In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function.
This was solved by the backpropagation network with at least one hidden layer. This type of network ...
- 1,194