1
$\begingroup$

Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time.

What is the maximum number of hidden neurons h that a hidden layer should possess to detect all unique features/ correlations between input data from the previous layer which has n nodes?

In other words if we wanted to create a neural network with a large number of neurons in a hidden layer, what is the maximum neuron count possible that helps the network train (give n neurons are in the previous layer)?

$\endgroup$
0
$\begingroup$

The Universal Approximation Theorem states: In the mathematical theory of artificial neural networks, the universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters.

So clearly if you do not have infinite number of hidden neurons you cannot approximate a function with $\epsilon$ error tending to 0. So if we assume that the previous layer gives some sort of representation of the function to be approximated you still need infinite hidden neurons to approximate this representative function.

But probably it'll be more clear to you if we go into basic physical laws of mathematical approximation.

According to the Fourier Theorem by Joseph Fourier: A Fourier series is a way to represent a function as the sum of simple sine waves. More formally, it decomposes any periodic function or periodic signal into the weighted sum of a (possibly infinite) set of simple oscillating functions, namely sines and cosines (or, equivalently, complex exponentials).

So theoretically, if your function is periodic it maybe perfectly approximated by an activation function like $sin$ or $cos$. The simple reason you cannot use an activation like $ReLu$ or $Sigmoid$ is because they are non-periodic and as per another related theorem called Fourier Transform you would require infinite number of sine waves to approximate non-periodic curves so the problem collapses in itself.

Since there are various ways to train a Neural Network other than Gradient Descent theoretically speaking you just might approximate a periodic function with some other methods (if activation is sine or cos), it is almost imposiible via methods like Gradient Descent.

So in short the answer to your question will be $\infty$ for the aforementioned reasons.

$\endgroup$
  • $\begingroup$ This answer is based on my interpretation of what Fourier Theorem states, if my interpretation is wrong please feel free to point out mistakes. $\endgroup$ – DuttaA Dec 9 '18 at 12:03
  • $\begingroup$ You state that an infinite number of hidden neurons in one layer can help approximate the function, but I'm not sure about this. If the previous layer has for example 2 neurons, shouldn't the number of nodes in this layer be -at max- 4 since there are 4 correlations between 2 numbers (2^2 according to combinatorics)? $\endgroup$ – Vikhyat Agarwal Dec 9 '18 at 14:06
  • $\begingroup$ @VikhyatAgarwal are you trying to find combinations or are you trying to approximate a function? A Combination might not give the required result. Since you have said you want to determine all possible unique features I assumed you are trying to approximate a function. $\endgroup$ – DuttaA Dec 9 '18 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.