Skip to main content
7 votes

How to estimate the capacity of a neural network?

VC dimension A rigorous measure of the capacity of a neural network is the VC dimension, which is intuitively a number or bound that quantifies the difficulty of learning from data. The sample ...
nbro's user avatar
  • 40.9k
4 votes
Accepted

Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?

Theoretical results Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper How ...
nbro's user avatar
  • 40.9k
3 votes

Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning: IT SHOULD OVERFIT. Once you get a model to overfit, its easier to experiment with ...
mshlis's user avatar
  • 2,369
3 votes

Is running more epochs really a direct cause of overfitting?

The Problem of Overfitting In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but ...
ddaedalus's user avatar
  • 919
2 votes

What is the difference between hypothesis space and representational capacity?

A hypothesis space/class is the set of functions that the learning algorithm considers when picking one function to minimize some risk/loss functional. The capacity of a hypothesis space is a number ...
nbro's user avatar
  • 40.9k
1 vote
Accepted

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
Neil Slater's user avatar
  • 32.7k
1 vote

In classification, how does the number of classes affect the model size and amount of data needed to train?

Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved. Solving capacity (learning ...
Dan D.'s user avatar
  • 1,293
1 vote

In classification, how does the number of classes affect the model size and amount of data needed to train?

The most obvious way more classes increase the network size is the output layer, but I don't believe there is a rule of thumb for the size of the entire network. As I understand it, there is no clear ...
Nathanson's user avatar
1 vote
Accepted

What is the difference between hypothesis space and representational capacity?

Consider a target function $f: x \mapsto f(x)$. A hypothesis refers to an approximation of $f$. A hypothesis space refers to the set of possible approximations that an algorithm can create for $f$. ...
Saurav Joshi's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible