Questions tagged [universal-approximation-theorems]
For questions related to the (different) universal approximation theorems (UATs), for example, in the context of neural networks.
33
questions
1
vote
1
answer
61
views
Why are neural networks optimized instead of just optimizing a high dimensional function?
I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high ...
- 11
1
vote
1
answer
42
views
100 layer neural network with 100 hidden units vs. 1 layer neural network with 100 hidden units
Suppose we have a neural network with 100 hidden layers. Each hidden layer has one hidden node, and the hidden nodes employ a universal basis function (e.g. tanh). Now we want to compare this network'...
1
vote
0
answers
71
views
Why doesn't the Kolmogorov-Arnold representation theorem imply an MLP-like structure?
Recently, Kolmogorov-Arnold Networks (KANs) generated a lot of hype, with "AI experts" throwing around terms like "ML 2.0" and "a new era of ML".
KANs are supposedly ...
- 11
2
votes
1
answer
136
views
Neural Networks are universal approximators? - Exercice 20.1 UML
I'm working on this question which can be found at page 282 of "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David.
The statement is as ...
- 21
1
vote
1
answer
67
views
Why do we need the identify function when approximating a function with a neural network with multiple layers?
I have a question about the explanation of universal approximation theorem provided by wikipedia.
https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29
It says, after a ...
- 21
5
votes
2
answers
211
views
What makes the approximation capabilities of neural networks different than something like, say, Fourier series?
People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable ...
- 153
1
vote
1
answer
134
views
Is there a mathematical proof of the universal approximation theorem for neural networks with binary weights?
Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant ...
- 33
2
votes
1
answer
586
views
Why do activation functions in neural networks have to be non-polynomial to approximate any function?
Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
- 31
1
vote
0
answers
78
views
Is the capability of RNN more than the capability of MLP?
Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
- 3,950
1
vote
0
answers
77
views
Does Godel's incompleteness theorems restricts the scope of connectionist-AI?
It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic.
But, I am wondering whether it has any impact on the ...
- 3,950
1
vote
1
answer
244
views
Are the capabilities of connectionist AI and symbolic AI the same?
The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
- 3,950
5
votes
2
answers
507
views
Why does the activation function for a hidden layer in a MLP have to be non-polynomial?
Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial.
Is there a ...
- 53
2
votes
0
answers
138
views
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
- 23
1
vote
1
answer
84
views
Is there any paper that shows that multi-channel neural networks are universal approximators?
Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where ...
- 13
2
votes
1
answer
340
views
Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?
I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?
- 21
0
votes
2
answers
306
views
Why can a neural network use more than one activation function?
From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
- 39
1
vote
1
answer
123
views
Issue with graphical interpretation of the universal approximation theorem
This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
- 39
1
vote
2
answers
37
views
How can "any process you can imagine" be thought of as function computation?
I stumbled upon this passage when reading this guide.
Universality theorems are a commonplace in computer science, so much
so that we sometimes forget how astonishing they are. But it's worth
...
- 39
0
votes
1
answer
86
views
Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?
There were some posts that using RNN can predict the next point of the sine wave function with data history.
However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\...
5
votes
2
answers
292
views
Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:
Create a "bump" function using 2 neurons.
Create (infinitely) many of these ...
- 133
20
votes
1
answer
1k
views
What is the number of neurons required to approximate a polynomial of degree n?
I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number ...
- 793
1
vote
1
answer
105
views
Can most of the basic machine learning models be easily represented as simple neural network architectures?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following:
Different ...
- 603
6
votes
1
answer
537
views
Smallest possible network to approximate the $sin$ function
The main goal is: Find the smallest possible neural network to approximate the $sin$ function.
Moreover, I want to find a qualitative reason why this network is the smallest possible network.
I have ...
- 75
1
vote
1
answer
180
views
When are multiple hidden layers necessary?
I know that my question probably seems like being asked many times, but Ill try
to be more speciffic:
Limitations to my question:
I am NOT asking about convolutional neural networks, so please, try ...
- 193
2
votes
1
answer
392
views
How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?
Neural networks typically have $\mathcal{VC}$ dimension that is proportional to their number of parameters and inputs. For example, see the papers Vapnik-Chervonenkis dimension of recurrent neural ...
- 41.1k
1
vote
1
answer
517
views
If a neural network is a universal function approximator, can it have any prior beliefs?
Let us confine ourselves to the case where we have a $n$ dimensional input and a $+1$ or $-1$ output. It can be shown that:
For every $n$, there exists a dense NN of depth 2, such that it contains ...
user9947
3
votes
0
answers
128
views
Are No Free Lunch theorem and Universal Approximation theorem contradictory in the context of neural networks?
To my understanding NFL states that, we cannot have an hypothesis (let's assume it is an approximator like NN in this case) class that can't achieve certain accuracy parameters $\leq \epsilon$ with ...
user9947
4
votes
1
answer
240
views
Does a neural network exist that can learn every possible training data?
The universal approximation theorem states, that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $R^n$...
- 143
31
votes
3
answers
17k
views
Where can I find the proof of the universal approximation theorem?
The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
- 475
3
votes
3
answers
781
views
Is there a way to calculate the closed-form expression of the function that a neural network computes?
As stated in the universal approximation theorem, a neural network can approximate almost any function.
Is there a way to calculate the closed-form (or analytical) expression of the function that a ...
- 133
8
votes
1
answer
3k
views
Which machine learning models are universal function approximators?
The universal approximation theorem states that a feed-forward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some ...
- 41.1k
4
votes
3
answers
1k
views
Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?
The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a mathematical proof that shows that certain ...
10
votes
2
answers
791
views
What are the learning limitations of neural networks trained with backpropagation?
In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function.
This was solved by the backpropagation network with at least one hidden layer. This type of network ...
