Maximum number of neurons in a layer given number of neurons in previous layer

Question

Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time.

What is the maximum number of hidden neurons h that a hidden layer should possess to detect all unique features/ correlations between input data from the previous layer which has n nodes?

In other words if we wanted to create a neural network with a large number of neurons in a hidden layer, what is the maximum neuron count possible that helps the network train (give n neurons are in the previous layer)?

Answer 1

The Universal Approximation Theorem states: In the mathematical theory of artificial neural networks, the universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters.

So clearly if you do not have infinite number of hidden neurons you cannot approximate a function with $\epsilon$ error tending to 0. So if we assume that the previous layer gives some sort of representation of the function to be approximated you still need infinite hidden neurons to approximate this representative function.

But probably it'll be more clear to you if we go into basic physical laws of mathematical approximation.

According to the Fourier Theorem by Joseph Fourier: A Fourier series is a way to represent a function as the sum of simple sine waves. More formally, it decomposes any periodic function or periodic signal into the weighted sum of a (possibly infinite) set of simple oscillating functions, namely sines and cosines (or, equivalently, complex exponentials).

So theoretically, if your function is periodic it maybe perfectly approximated by an activation function like $sin$ or $cos$. The simple reason you cannot use an activation like $ReLu$ or $Sigmoid$ is because they are non-periodic and as per another related theorem called Fourier Transform you would require infinite number of sine waves to approximate non-periodic curves so the problem collapses in itself.

Since there are various ways to train a Neural Network other than Gradient Descent theoretically speaking you just might approximate a periodic function with some other methods (if activation is sine or cos), it is almost imposiible via methods like Gradient Descent.

So in short the answer to your question will be $\infty$ for the aforementioned reasons.