2
$\begingroup$

When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?

Are there any soft or hard limitations where the number of outputs starts to stall learning?

Do you know about any analysis of how the number of classes scales the model?

Does the optimal size increase proportionally with the number of outputs? Does it increase at all? If it does increase, is the relationship linear or exponential?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The most obvious way more classes increase the network size is the output layer, but I don't believe there is a rule of thumb for the size of the entire network.

As I understand it, there is no clear answer to how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very active research field, just as an example compare the size of EfficientNet to other State of the art models when it was introduced and you can see the size difference.

Regarding the data needed, in The Deep Learning Book (which is a few years old now) they state as a general that with the models available at that time, you needed ~5000 examples per label for acceptable performance, while to exceed human performance (their words) you would need about 10 million labeled examples.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I didn't expect the difference between acceptable and super-human to be three orders of magnitude. Do you remember in which chapter of the book this statement was made? $\endgroup$
    – conscious_process
    Feb 25, 2021 at 13:35
  • 1
    $\begingroup$ As I mentioned, it is a rather old book (with regard to DL, since everything develops so fast) so this might have changed since it's a rather active research field. However, I think it's in the introductory chapter, under some title with Data in the name. $\endgroup$
    – Nathanson
    Feb 25, 2021 at 17:31
  • $\begingroup$ K, thanks for your input. $\endgroup$
    – conscious_process
    Feb 26, 2021 at 8:34
1
$\begingroup$

Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved.

Solving capacity (learning capacity) of a network (dense usually) can be calculated as the product of number of neurons in all layers, for example:

Input shape: 10 values
Network shape: [layer1 30 units, layer2 20 units, output 1 unit] should have learning capacity of $30 \times 20 \times 1 = 600$, it learn roughly max 600 different inputs (each input holds 10 values).

Another consideration, the separation lines, even when the inputs of the problem to be learnt are unlimited, but the 2 classes (just example) are always separated on 2 sides of a line without mixing up, just a single neuron can solve the problem.

One neuron can make 1 separation line, 1 layer makes a poly-line with segments are by the neurons in that layer, another layer makes another poly-line.

Thus, more classes, more separations to be done, and more classes would mean the input variety is large, so surely training data are a lot and model size needs to be large.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Very good point with the seperation lines. I am thinking about testing it out by making a dataset generator for n-classes (n-arm spiral?) and see how the size of minimal viable model changes when n increases. Imo the number of different inputs a network "can hold" depends on the complexity of the problem. If dataset is seperatable by a line, then it can hold infinite amount of points, because it generalized well. If dataset is completely random, then the network can't generalize and points can only be memorized. Maybe in the second case your product-based method holds, but idk. $\endgroup$
    – conscious_process
    Feb 26, 2021 at 8:33
  • $\begingroup$ yeah, those 'completely random' datasets are crazy $\endgroup$
    – Dan D.
    Feb 26, 2021 at 13:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .