Stephen Wolfram published an interesting long post on machine learning this week.
He illustrates a function approximation application with the following target function, piecewise flat with three regions.
I understand one can describe such a function with five parameters, the three constant levels (initially low, high in the middle and mid on the right) and the two discontinuity points.
As a network architecture, the following picture is given.
If my count is right, there are 19 weights (4+12+3 arrows) and 8 biases (count of all neurons but the input one, 4+3+1), totalling 27 parameters. The activation function is said to be ReLU for all neurons.
With this frame, we have 27 parameters in the model to estimate a 5 parameter function.
The following image illustrates how the model fits the function as the number of examples grows.
From 10 thousand examples to 10 milion examples. The magnitude of data required is much higher than the complexity of the target function and the approximating network.
How should this (dis)proportion of data to problem parameters be understood?