# Questions tagged [universal-approximation-theorems]

For questions related to the (different) universal approximation theorems (UATs), for example, in the context of neural networks.

33 questions
Filter by
Sorted by
Tagged with
1 vote
61 views

### Why are neural networks optimized instead of just optimizing a high dimensional function?

I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high ...
1 vote
42 views

### 100 layer neural network with 100 hidden units vs. 1 layer neural network with 100 hidden units

Suppose we have a neural network with 100 hidden layers. Each hidden layer has one hidden node, and the hidden nodes employ a universal basis function (e.g. tanh). Now we want to compare this network'...
1 vote
71 views

### Why doesn't the Kolmogorov-Arnold representation theorem imply an MLP-like structure?

Recently, Kolmogorov-Arnold Networks (KANs) generated a lot of hype, with "AI experts" throwing around terms like "ML 2.0" and "a new era of ML". KANs are supposedly ...
136 views

### Neural Networks are universal approximators? - Exercice 20.1 UML

I'm working on this question which can be found at page 282 of "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David. The statement is as ...
• 21
1 vote
67 views

### Why do we need the identify function when approximating a function with a neural network with multiple layers?

I have a question about the explanation of universal approximation theorem provided by wikipedia. https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29 It says, after a ...
211 views

### What makes the approximation capabilities of neural networks different than something like, say, Fourier series?

People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable ...
1 vote
134 views

### Is there a mathematical proof of the universal approximation theorem for neural networks with binary weights?

Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant ...
586 views

### Why do activation functions in neural networks have to be non-polynomial to approximate any function?

Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
1 vote
78 views

### Is the capability of RNN more than the capability of MLP?

Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
• 3,950
1 vote
77 views

### Does Godel's incompleteness theorems restricts the scope of connectionist-AI?

It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic. But, I am wondering whether it has any impact on the ...
• 3,950
1 vote
244 views

### Are the capabilities of connectionist AI and symbolic AI the same?

The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
• 3,950
507 views

### Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
• 53
138 views

### Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?

The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
1 vote
84 views

### Is there any paper that shows that multi-channel neural networks are universal approximators?

Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where ...
• 13
340 views

### Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?

I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?
• 21
306 views

### Why can a neural network use more than one activation function?

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
• 39
1 vote
123 views

### Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
• 39
1 vote
37 views

### How can "any process you can imagine" be thought of as function computation?

I stumbled upon this passage when reading this guide. Universality theorems are a commonplace in computer science, so much so that we sometimes forget how astonishing they are. But it's worth ...
• 39
86 views

• 143
17k views

### Where can I find the proof of the universal approximation theorem?

The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
• 475
781 views

### Is there a way to calculate the closed-form expression of the function that a neural network computes?

As stated in the universal approximation theorem, a neural network can approximate almost any function. Is there a way to calculate the closed-form (or analytical) expression of the function that a ...
• 133
3k views

### Which machine learning models are universal function approximators?

The universal approximation theorem states that a feed-forward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some ...
• 41.1k