How to decide the optimum number of layers to be created while implementing a Neural Network (Feedforward, back propagation or RNN)?


There is a technique called Pruning in neural networks, which is used just for this same purpose.

The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows:

  • Train a large, densely connected, network with a standard training algorithm
  • Examine the trained network to assess the relative importance of the weights
  • Remove the least important weight(s)
  • retrain the pruned network
  • Repeat steps 2-4 until satisfied

However, there are several optimized methods for pruning neural nets, and it is also a very active area of research.

  • $\begingroup$ A symmetric approach is the common "grid search" applied to the network architecture. Start small (so fast), and automatically try larger architectures. All this is just brute force, though... $\endgroup$ – Eric Platon Aug 3 '16 at 6:41
  • $\begingroup$ @EricPlaton +1 for grid search. Very handy in hypertuning ML algos. But, isn't it computationally very intensive? $\endgroup$ – Dawny33 Aug 3 '16 at 6:43
  • 1
    $\begingroup$ Yes, it is expensive. Yet, when we can start small, the first stages can go pretty fast, and give a better idea what to aim for. $\endgroup$ – Eric Platon Aug 3 '16 at 6:45
  • $\begingroup$ @EricPlaton "Grid Search" would deserve a seperate answer for it's own :) $\endgroup$ – Dawny33 Aug 3 '16 at 8:52
  • $\begingroup$ I thought about it, but then there would be two alternative and perhaps "equally correct" answers... I thought the best for the reader was to compile the answer as one. $\endgroup$ – Eric Platon Aug 3 '16 at 8:58

You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network the better, so optimizing loss as a function of number of layers isn't a very fun thing to do.

Grid search and a bit of common sense (as learnt by seeing many examples) should be your best bet.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.