# Standard deviation in the # of neurons per layer

In a neural network, by how much does the # of neurons typically vary from layer to layer?

I am NOT asking how to find the optimal # of neurons per lyr.

As a hardware design engineer with no practical experience programming neural networks, I would like to glean for example...

1. By how much does the number of neurons in hidden layers typically vary from that of the input layer?
2. What is the maximum deviation in the # of hidden lyr neurons to the # of input layer neurons?
3. How commonly do you see a large spike in the number of neurons?

It likely depends on the application so I would like to hear from as many people as possible. Please tell me about your experience.

• I think it depends on the task (and not on experience at all). For example easily generalizable data will have lower number of neurons as compared to hard to generalize data, and we have no way of knowing this b4hand. Are u trying to implement NNs on FPGAs? – DuttaA Aug 6 '19 at 16:08
• @DuttaA I'm experimenting with NN architectures and some of the design choices relate to how much bandwidth is typically used (and a typical max) for a single multiply/accumulate stage. – Angela Johnson Aug 6 '19 at 16:54

1. Input layers will always have the dimensionality of your input data(for every model I can think of).

2. See above, the deviation between hidden layers can be significant. For example, 128 in the first hidden and 64 in the rest(or vice versa).

3. This question in particular will always be problem dependent. It is decided via architecture search or intuition/experience combined with some exploratory search.

There is no right answer to this question. But, I would like to point you to an answer on CV that addresses the mean of your problem.

Two points from the accepted answer that I want to draw your attention to are:

a) There are some empirically-derived rules-of-thumb, of these, the most commonly relied on is 'the optimal size of the hidden layer is usually between the size of the input and size of the output layers'.

b) In sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Other answers in the thread are also very insightful. I will recommend you to go through the answers and figure out the standard deviation and just assume things are normal, and hence you would have your distribution.