In a neural network, by how much does the number of neurons typically vary from layer to layer?

Question

In a neural network, by how much does the number of neurons typically vary from layer to layer?

Note that I am NOT asking how to find the optimal number of neurons per layer.

As a hardware design engineer with no practical experience programming neural networks, I would like to glean for example

1. By how much does the number of neurons in hidden layers typically vary from that of the input layer?

2. What is the maximum deviation in the number of hidden layer neurons to the number of input layer neurons?

3. How commonly do you see a large spike in the number of neurons?

It likely depends on the application so I would like to hear from as many people as possible. Please tell me about your experience.

Answer 1

1. Input layers will always have the dimensionality of your input data(for every model I can think of).

2. See above, the deviation between hidden layers can be significant. For example, 128 in the first hidden and 64 in the rest(or vice versa).

3. This question in particular will always be problem dependent. It is decided via architecture search or intuition/experience combined with some exploratory search.

Answer 2

There is no right answer to this question. But, I would like to point you to an answer on CV that addresses the mean of your problem.

Two points from the accepted answer that I want to draw your attention to are:

a) There are some empirically-derived rules-of-thumb, of these, the most commonly relied on is 'the optimal size of the hidden layer is usually between the size of the input and size of the output layers'.

b) In sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Other answers in the thread are also very insightful. I will recommend you to go through the answers and figure out the standard deviation and just assume things are normal, and hence you would have your distribution.