I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer.
I remember that the Professor saying that there is no rule for choosing the number of hidden units and that having many of them along with many hidden layers can cause the network to overfit the data and under learn.
However, I still have this question where assuming that we have a network with an input layer of
n input nodes, a first hidden layer of
4 hidden units, a second layer of
X hidden units and an output layer of
5 units. Now if I follow the Professor's saying, it would mean that I am allowed to have
X = 3 or
X = 4 in layer 2.
Is that actually allowed? Won't we have some sort of information gain passing from 4 (or 3) nodes to 5? The example is illustrated below.