4
$\begingroup$

Stephen Wolfram published an interesting long post on machine learning this week.

He illustrates a function approximation application with the following target function, piecewise flat with three regions.

enter image description here

I understand one can describe such a function with five parameters, the three constant levels (initially low, high in the middle and mid on the right) and the two discontinuity points.

As a network architecture, the following picture is given.

enter image description here

If my count is right, there are 19 weights (4+12+3 arrows) and 8 biases (count of all neurons but the input one, 4+3+1), totalling 27 parameters. The activation function is said to be ReLU for all neurons.

With this frame, we have 27 parameters in the model to estimate a 5 parameter function.

The following image illustrates how the model fits the function as the number of examples grows.

enter image description here

From 10 thousand examples to 10 milion examples. The magnitude of data required is much higher than the complexity of the target function and the approximating network.

How should this (dis)proportion of data to problem parameters be understood?

Improve this question
$\endgroup$
1
  • $\begingroup$ Any number of factors can affect the data requirement. For example, some pairs of loss and activation functions, such as cross-entropy logistic, are unusually good at keeping the learning rate approximately constant. $\endgroup$
    – J.G.
    Commented Feb 24, 2023 at 20:36

1 Answer 1

Reset to default
2
$\begingroup$

One example or sample is basically a pair of data $(x_i, y_i)$ with $x_i$ randomly picked from x axis and $y_i$ from the piecewise function. As you can see, it doesn't provide a lot information to calculating the weighting factors. By 10,000,000 randomly samples, one may capture the turning points. But if it is not randomly sampled, I would guess the number of sample can be less. The process is different from calculating model parameters from the piecewise function parameter as the neural net doesn't know it is a piecewise function as a priori. The function it expects is a general function, which can be very complex.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes! My mistake, plugging that a priori information that the network must learn on its own. Nice experiment to code, checking whether uniform sampling reduces the data size needed. $\endgroup$
    – Jaume Oliver Lafont
    Commented Feb 24, 2023 at 17:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .