# Are there any rules of thumb for having some idea of what capacity a NN model needs to have for a given problem?

To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:

• There are 10 output classes
• The inputs are 28x28 grayscale pixels (I think this indirectly affects the model capacity. eg: if the inputs were 5x5 pixels, there wouldn't be much room for varying the way an 8 looks)

So, is there any way of knowing what the model capacity ought to be? Even if it's not exact? Even if it's a qualitative understanding of the type "if X goes up, then Y goes down"?

Just to accentuate what I mean when I say "not exact": I can already tell that a 100 variable model won't solve MNIST, so at least I have a lower bound. I'm also pretty sure that a 1,000,000,000 variable model is way more than needed. Of course, knowing a smaller range than that would be much more useful!

EDIT

For anyone who was following this, this answer was quite useful

• Have a look at these related questions How to estimate the capacity of a neural network? and How to choose the number of hidden layers and nodes in a feedforward neural network?. – nbro Feb 27 at 16:14
• @nbro this is great. I'm not sure what weight you carry around these communities but it seems like some. I feel that there's scope for consolidating some of this community, data science, and cross-validated. Users could get more value from one stronger community rather than a few fragmented smaller ones. – Alexander Soare Feb 27 at 17:14
• Yes, I completely agree with you. At least, Data Science SE should be merged with AI SE and there should be more interaction between Cross Validated SE and AI SE. Anyway, I wouldn't be so sure about the "rules of thumb" described in the other answers on Stats SE. Furthermore, they usually don't take into account the problem and available data. I think a more scientific way would be to look at computational learning theory (my first link above). – nbro Feb 27 at 17:24
• Great thanks a lot. I like having both approaches available. – Alexander Soare Feb 27 at 17:31

Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper How many hidden layers and nodes?), from which you may be able to derive your rules of thumb, depending on your problem, etc. I will be updating this answer, as I find more theoretical results.

### Result 1

The paper Learning capability and storage capacity of two-hidden-layer feedforward networks proves that a 2-hidden layer feedforward network ($$F$$) with $$2 \sqrt{(m + 2)N} \ll N$$ hidden neurons can learn any $$N$$ distinct samples $$D= \{ (x_i, t_i) \}_{i=1}^N$$ with an arbitrarily small error, where $$m$$ is the required number of output neurons. Conversely, a $$F$$ with $$Q$$ hidden neurons can store at least $$\frac{Q^2}{4(m+2)}$$ any distinct data $$(x_i, t_i)$$ with any desired precision.

They suggest that a sufficient number of neurons in the first layer should be $$\sqrt{(m + 2)N} + 2\sqrt{\frac{N}{m + 2}}$$ and in the second layer should be $$m\sqrt{\frac{N}{m + 2}}$$. So, for example, if your dataset has size $$N=10$$ and you have $$m=2$$ output neurons, then you should have the first hidden layer with roughly 10 neurons and the second layer with roughly 4 neurons. (I haven't actually tried this!)

However, these bounds are suited for fitting the training data (i.e. for overfitting), which isn't usually the goal, i.e. you want the network to generalize to unseen data.

This result is strictly related to the universal approximation theorems, i.e. a network with a single hidden layer can, in theory, approximate any continuous function.

• I will be updating this answer, as I gather more information! – nbro Mar 5 at 6:48

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:

IT SHOULD OVERFIT.

Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.

• Although I understand your point, this sounds like a bad suggestion for several reasons. Ideally, you want to design the neural network before training it and not design by trial-and-error (which can be computationally prohibitive, if your task is difficult, you have a big dataset, big net, etc). What if the training data isn't really similar to the test data, even though this assumption is usually implicitly made? This rule of thumb could have been just a comment. – nbro Feb 29 at 4:08
• Maybe you can rephrase this answer by saying that "In theory, if the model overfits the training data, it means that it has a sufficient capacity to model the training data, but this will require you to first train the model and try different configurations". – nbro Feb 29 at 4:09
• Thanks for the tip. And @nbro thanks for the clarity. – Alexander Soare Feb 29 at 11:18

(I wanted to add this as a comment, but, due to lack of rep, couldn't. Hence the entry as an answer. In case this is not useful, moderators may delete this.)

Personally, when I begin designing a machine learning model, I consider the following points:

• My data: if I have simple images like MNIST or in general images with very low resolution, a very deep network is not required.
• If my problem statement needs to learn lot of features from each image, such as for human face, I may need to learn eyes, nose, lips, expressions through their combinations, then I need a deep network with CONV layers.
• If I have timeseries data, LSTM or GRU makes sense, but, I also consider recurrent setup when my data has high resolution, low count datapoints.

The upper limit however may get decided by resources available on the computing device you are using for training. Hope this helps.