5

VC dimension A rigorous measure of the capacity of a neural network is the VC dimension, which is intuitively a number or bound that quantifies the difficulty of learning from data. The sample complexity, which is the number of training instances that the model (or learner) must be exposed to in order to be reasonably certain of the accurateness of the ...


2

A hypothesis space/class is the set of functions that the learning algorithm considers when picking one function to minimize some risk/loss functional. The capacity of a hypothesis space is a number or bound that quantifies the size (or richness) of the hypothesis space, i.e. the number (and type) of functions that can be represented by the hypothesis space. ...


2

Theoretical results Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper How many hidden layers and nodes?), from which you may be able to derive your rules of thumb, depending on your problem, etc. Result 1 The paper Learning ...


2

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning: IT SHOULD OVERFIT. Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.


1

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data. Sort of. A secondary objective function often works as a form of regularisation, and can work to reduce overfit. However, this ...


1

Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved. Solving capacity (learning capacity) of a network (dense usually) can be calculated as the product of number of neurons in all layers, for example: Input shape: 10 values Network shape: [...


1

The most obvious way more classes increase the network size it the output layer, but I don't believe there is a rule of thumb for the size of the entire network. As I understand it, there is no clear answer how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very ...


1

Consider a target function $f: x \mapsto f(x)$. A hypothesis refers to an approximation of $f$. A hypothesis space refers to the set of possible approximations that an algorithm can create for $f$. The hypothesis space consists of the set of functions the model is limited to learn. For instance, linear regression can be limited to linear functions as its ...


Only top voted, non community-wiki answers of a minimum length are eligible