How do you distinguish between a complex and a simple model in machine learning? Which parameters control the complexity or simplicity of a model? Is it the number of inputs, or maybe the number of layers?
Moreover, when should a simple model be used instead of a complex one, and vice-versa?
Consider a continuum of complexity in models.
Trivial: $y = x + a$
Simple: $y = x \, \log \, (a x + b) + c$
Moderately complex: A wind turbine under constant wind velocity
Very complex: Ray tracing of lit 3-D motion scenes to pixels
Astronomically complex: The weather
Now consider a continuum regarding the generality or specificity of models.
Very specific: The robot for the Mars mission has an exact mechanical topology, materials call-out, and set of mechanical coordinates contained in the CAD files used to machine the robot's parts.
Somewhat specific: The formulas guiding the design of an internal combustion engine, which are well known.
Somewhat general: The phenomenon is deterministic and the variables and their domains are known.
Very general: There's probably some model because it works in nature but we know little more.
There are twenty permutations at the above level of granularity. Every one has purpose in mathematical analysis, applied research, engineering, and monetization.
Here are some general correlations between input, output, and layer counts.
Higher complexity often corresponds to larger layer count.
Higher i/o dimensionality corresponds to higher width to the corresponding i/o layers.
Mapping generality to or from specificity generally requires complexity.
Now, to make this answer even less appealing to those who want formula answer they can memorize, ...
Each artificial network is a model of an arbitrary function before training and a model of a specific function afterward.
Loss functions are models of disparity.
An algorithm is a model of a process created by spreading a recursive definition out in time to map into a model of centralized computation called a CPU.
The recursive definition is a model too.
There is almost nothing in science that is a not a model except ideas or data that are not yet modeled.
If you want to find a proper architecture for your model, you can use the NAS (neural architecture search) methods instead of running some naive models to find a model and involving to decide which model is more complex or simpler. Some methods which used in NAS to find a proper architecture are:
In what context are you asking this? It is totally different if you want to perform object detection, regression or, for example, reinforcement learning.
For the first case I would say that main point in using simple vs complex model is size of training data. If you have 1000 training samples you can't expect large network to perform better than simple one.
In a nutshell, if you already have a number of models, you usually should be able to distinguish (intuitively, if you will) between simpler and more complex ones. E.g. based on the number of inputs and number of layers, as you have already indicated in the question. Then, if a simpler model and a more complex model perform the same task, and the complex model does not perform significally better than the simpler one, you should use the simpler model. It's your role to decide what difference in performance would be significant, usually based on your use case. It's Occam's razor in practice (https://en.m.wikipedia.org/wiki/Occam%27s_razor). You might learn more practical aspects as part of this free course https://lagunita.stanford.edu/courses/HumanitiesSciences/StatLearning/Winter2016/about
