How do I choose the number of neurons in the fully-connected layer before the softmax layer?

I am solving a classification problem with CNN. The number of classes is 5.

1. How can I decide the number of neurons in the FC layer before the softmax layer?
2. Is it $$N * 5$$, where $$N$$ is the number of classes?
3. Is there any documentation for deciding the number of neurons in the FC layer (before SoftMax layer)

How can I decide the number of neurons in the FC layer before the softmax layer?

Train different network architectures (with different numbers of neurons in the last FC layer). Use cross-validation - a set of data you have not trained on - to measure the performance of the network, using a metric that you have decided beforehand is a good proxy for your experiment goals. For instance, you might choose getting the highest classification accuracy as your goal - but might choose something different for an unbalanced dataset, because 90% accuracy is not meaningful.

There is not usually a good reason to over-tune your network. Trying some variations with 1.5 x or 2 x geometric series (e.g. 5, 10, 20, 40 neurons in layer) is probably enough to find a good hidden layer size.

Is it $$N * 5$$, where $$N$$ is the number of classes?

No, in general. That may work just fine for your problem though.

Is there any documentation for deciding the number of neurons in the FC layer (before the softmax layer)

Not really. The most direct thing to do is find solutions that work well for similar problems to yours, and adapt as necessary. So you can search for any similar problem domain and see what the researchers used there.

There is not much theory to guide you here. However, if it helps with your intuition, more neurons can make a better fit to higher frequency variations in the target function you are learning, whilst more layers will mean better handling of complex function spaces (e.g. where rules about the mapping between input and output can be made according to combining simpler rules) - provided in both cases that you have enough training data that it is possible for the NN to learn the function approximately. These are not strictly defined traits of functions for machine learning, so many NN users will work with this intuitive view.

If in doubt, assuming this is not an image, NLP or other well-studied problem, then I might just guess at e.g. 64 neurons per hidden layer, and try 1, 2, and 3 hidden layers as a starting point (all the same size). I cannot say if that will work for you and your problem, but it might help get past the "blank page effect" and start you training and testing some variations.

• Thanks. When I tried with CIFAR10, I got different accuracy for different number of neurons in the last hidden layer. I got the maximum accuracy when the number of neurons was 256. When I gave 100, it over fits early compared to 256. Also note that I am going to solve defect detection with images in a semi conductor Chip Manufacturing tool – Maanu Apr 16 '19 at 0:57
• @Maanu: For defect detection I suspect you will not want a simple accuracy measure (I assume in production your defect rate is low, so the real world population is far from balanced). If you have trouble picking a metric, then perhaps ask here or on Data Science stack exchange. I think that Data Science is the better place to ask such practical questions about a specific problem domain. – Neil Slater Apr 16 '19 at 8:01