9
$\begingroup$

To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:

  • There are 10 output classes
  • The inputs are 28x28 grayscale pixels (I think this indirectly affects the model capacity. eg: if the inputs were 5x5 pixels, there wouldn't be much room for varying the way an 8 looks)

So, is there any way of knowing what the model capacity ought to be? Even if it's not exact? Even if it's a qualitative understanding of the type "if X goes up, then Y goes down"?

Just to accentuate what I mean when I say "not exact": I can already tell that a 100 variable model won't solve MNIST, so at least I have a lower bound. I'm also pretty sure that a 1,000,000,000 variable model is way more than needed. Of course, knowing a smaller range than that would be much more useful!

EDIT

For anyone who was following this, this answer was quite useful

$\endgroup$
2
+100
$\begingroup$

Theoretical results

Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper How many hidden layers and nodes?), from which you may be able to derive your rules of thumb, depending on your problem, etc.

Result 1

The paper Learning capability and storage capacity of two-hidden-layer feedforward networks proves that a 2-hidden layer feedforward network ($F$) with $$2 \sqrt{(m + 2)N} \ll N$$ hidden neurons can learn any $N$ distinct samples $D= \{ (x_i, t_i) \}_{i=1}^N$ with an arbitrarily small error, where $m$ is the required number of output neurons. Conversely, a $F$ with $Q$ hidden neurons can store at least $\frac{Q^2}{4(m+2)}$ any distinct data $(x_i, t_i)$ with any desired precision.

They suggest that a sufficient number of neurons in the first layer should be $\sqrt{(m + 2)N} + 2\sqrt{\frac{N}{m + 2}}$ and in the second layer should be $m\sqrt{\frac{N}{m + 2}}$. So, for example, if your dataset has size $N=10$ and you have $m=2$ output neurons, then you should have the first hidden layer with roughly 10 neurons and the second layer with roughly 4 neurons. (I haven't actually tried this!)

However, these bounds are suited for fitting the training data (i.e. for overfitting), which isn't usually the goal, i.e. you want the network to generalize to unseen data.

This result is strictly related to the universal approximation theorems, i.e. a network with a single hidden layer can, in theory, approximate any continuous function.

Model selection, complexity control, and regularisation

There are also the concepts of model selection and complexity control, and there are multiple related techniques that take into account the complexity of the model. The paper Model complexity control and statistical learning theory (2002) may be useful. It is also important to note regularisation techniques can be thought of as controlling the complexity of the model [1].

Further reading

You may also want to take a look at these related questions

(I will be updating this answer, as I find more theoretical results or other useful info)

| improve this answer | |
$\endgroup$
2
$\begingroup$

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:

IT SHOULD OVERFIT.

Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Although I understand your point, this sounds like a bad suggestion for several reasons. Ideally, you want to design the neural network before training it and not design by trial-and-error (which can be computationally prohibitive, if your task is difficult, you have a big dataset, big net, etc). What if the training data isn't really similar to the test data, even though this assumption is usually implicitly made? This rule of thumb could have been just a comment. $\endgroup$ – nbro Feb 29 at 4:08
  • $\begingroup$ Maybe you can rephrase this answer by saying that "In theory, if the model overfits the training data, it means that it has a sufficient capacity to model the training data, but this will require you to first train the model and try different configurations". $\endgroup$ – nbro Feb 29 at 4:09
  • $\begingroup$ Thanks for the tip. And @nbro thanks for the clarity. $\endgroup$ – Alexander Soare Feb 29 at 11:18
0
$\begingroup$

Personally, when I begin designing a machine learning model, I consider the following points:

  • My data: if I have simple images, like MNIST ones, or in general images with very low resolution, a very deep network is not required.

  • If my problem statement needs to learn a lot of features from each image, such as for the human face, I may need to learn eyes, nose, lips, expressions through their combinations, then I need a deep network with convolutional layers.

  • If I have time-series data, LSTM or GRU makes sense, but, I also consider recurrent setup when my data has high resolution, low count data points.

The upper limit however may get decided by resources available on the computing device you are using for training.

Hope this helps.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.