2
$\begingroup$

Is there a measure of model complexity?

$\endgroup$
0

2 Answers 2

1
$\begingroup$

Yes. There are at least 2 measures of model complexity studied and used in learning theory: VC dimension and Rademacher complexity. If you're new to learning theory, you could take a look at this answer.

(Note: your question is not an exact duplicate of this, but the VC dimension is not specific to neural networks).

$\endgroup$
0
$\begingroup$

There is no "unit" to measure model complexity. However, you can take into account some key factors:

  • input dimension
  • output dimension
  • model length (number of layers): Model length describes the number of transformations on the data from input to output. A higher number of layers means more possible transformations and thus the possibility to model more complex dependencies.
  • model width (number of neurons per layer): Model width describes the dimensionality of the data going through a single layer. A higher number of neurons means higher information retention per layer.
  • activation functions: Without any activation functions, even the largest model can only learn linear dependencies. Activation functions introduce non-linearity into the model and make complex dependency modeling possible.

Based on those factors, you can get a good estimate of how complex your model is. Note that a more complex model does not equal better performance per-se, they require careful fine-tuning.

To get a good estimate if your model is large enough to model your data transformations, you can train it on very little data with batchsize=1 and no regularisation. If it completely learns the given samples, it is likely to be fitting for your entire dataset aswell.

$\endgroup$
1
  • $\begingroup$ The answer address neural network only, not model in general. $\endgroup$
    – lpounng
    Jun 16, 2023 at 2:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .