Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

Question

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer.

I remember that the Professor saying that there is no rule for choosing the number of hidden units and that having many of them along with many hidden layers can cause the network to overfit the data and under learn.

However, I still have this question where assuming that we have a network with an input layer of n input nodes, a first hidden layer of 4 hidden units, a second layer of X hidden units and an output layer of 5 units. Now if I follow the Professor's saying, it would mean that I am allowed to have X = 3 or X = 4 in layer 2.

Is that actually allowed? Won't we have some sort of information gain passing from 4 (or 3) nodes to 5? The example is illustrated below.

Answer 1

A layer with bigger number of nodes than previous one is something very common. Some examples are:

• strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes.

• bidirectional recurrent networks where in the forward direction number nodes decreases and in the backward increases.

• generators, that from a random vector generates, by example, a full image.

As general rule: decrease number of nodes forces the net to filter/resume/abstract/summarize the internal signal information (discarding useless information or noise) while increase number of nodes means apply current information to generate an answer value for a specific question/target.

Allow me a strongly simplified example: assume you want a system that, from a photo of an animal, answers the questions: number of legs? has beak ? flies ? . Net inputs are images of birds and dogs.

The net architecture can have layers of decreasing size until a single node that will decide "is bird or dog ?". From this single item of information (the only one need to answer all the questions) the output layer will have 3 nodes, each one answering one of the specific target questions: number or legs ? 4 if dog, 2 if bird, etc .