How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
There is a technique called
Pruning in neural networks, which is used just for this same purpose.
The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows:
- Train a large, densely connected, network with a standard training algorithm
- Examine the trained network to assess the relative importance of the weights
- Remove the least important weight(s)
- retrain the pruned network
- Repeat steps 2-4 until satisfied
You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network the better, so optimizing loss as a function of number of layers isn't a very fun thing to do.
Grid search and a bit of common sense (as learnt by seeing many examples) should be your best bet.