Questions tagged [universal-approximation-theorems]
For questions related to the (different) universal approximation theorems (UATs), for example, in the context of feed-forward neural networks.
18
questions
0
votes
1answer
47 views
Why can a neural network use more than one activation function?
From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
1
vote
1answer
34 views
Issue with graphical interpretation of the universal approximation theorem
This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
1
vote
2answers
34 views
How can “any process you can imagine” be thought of as function computation?
I stumbled upon this passage when reading this guide.
Universality theorems are a commonplace in computer science, so much
so that we sometimes forget how astonishing they are. But it's worth
...
0
votes
1answer
80 views
Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?
There were some posts that using RNN can predict the next point of the sine wave function with data history.
However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\...
4
votes
1answer
157 views
Is it really possible to create the “Perfect Cylinder” used in Universal Approximation Theorem for 1-hidden layer Neural Network?
There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:
Create a "bump" function using 2 neurons.
Create (infinitely) many of these ...
9
votes
0answers
115 views
What is the number of neurons required to approximate a polynomial of degree n?
I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number ...
1
vote
1answer
60 views
Can most of the basic machine learning models be easily represented as simple neural network architectures?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following:
Different ...
6
votes
1answer
134 views
Smallest possible network to approximate the $sin$ function
The main goal is: Find the smallest possible neural network to approximate the $sin$ function.
Moreover, I want to find a qualitative reason why this network is the smallest possible network.
I have ...
1
vote
1answer
93 views
When are multiple hidden layers necessary?
I know that my question probably seems like being asked many times, but Ill try
to be more speciffic:
Limitations to my question:
I am NOT asking about convolutional neural networks, so please, try ...
2
votes
1answer
147 views
How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters?
Neural networks typically have $\mathcal{VC}$ dimension that is proportional to their number of parameters and inputs. For example, see the papers Vapnik-Chervonenkis dimension of recurrent neural ...
1
vote
1answer
80 views
If a neural network is a universal function approximator, can it have any prior beliefs?
Let us confine ourselves to the case where we have a $n$ dimensional input and a $+1$ or $-1$ output. It can be shown that:
For every $n$, there exists a dense NN of depth 2, such that it contains ...
2
votes
0answers
64 views
Are No Free Lunch theorem and Universal Approximation theorem contradictory in the context of neural networks?
To my understanding NFL states that, we cannot have an hypothesis (let's assume it is an approximator like NN in this case) class that can't achieve certain accuracy parameters $\leq \epsilon$ with ...
4
votes
1answer
117 views
Does a neural network exist that can learn every possible training data?
The universal approximation theorem states, that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $R^n$...
16
votes
3answers
5k views
Where can I find the proof of the universal approximation theorem?
The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
2
votes
3answers
111 views
Is there a way to calculate the closed-form expression of the function that a neural network computes?
As stated in the universal approximation theorem, a neural network can approximate almost any function.
Is there a way to calculate the closed-form (or analytical) expression of the function that a ...
5
votes
1answer
1k views
Which machine learning models are universal function approximators?
The universal approximation theorem states that a feed-forward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some ...
4
votes
3answers
1k views
Is there a mathematical proof that shows that certain parameters work “better” than others for a certain task?
The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a mathematical proof that shows that certain ...
10
votes
2answers
589 views
What are the learning limitations of neural networks trained with backpropagation?
In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function.
This was solved by the backpropagation network with at least one hidden layer. This type of network ...
