4
$\begingroup$

In training a neural network, you often see the curve showing how fast the neural network is getting better. It usually grows very fast then slows down to almost horizontal.

Is there a mathematical formula that matches these curves?

enter image description here

Some similar curves are:

$$y=1-e^{-x}$$

$$y=\frac{x}{1+x}$$

$$y=\tanh(x)$$

$$y=1+x-\sqrt{1+x^2}$$

Is there a theoretical reason for this shape?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ As you mention, "It usually grows very fast then slows down", so, of course, there isn't a single mathematical formula that would describe all learning curves. So, I suggest you clarify your question, which seems to be quite interesting. $\endgroup$
    – nbro
    Commented Jan 9, 2020 at 15:36
  • $\begingroup$ @nbro OK. By usually I mean always. And of course it slows down because the neural network can't keep increasing intelligence forever. But these curves do almost always look the same and do usually keep growing for a long time but very slowly. I'm just wondering if one can theortically model this curve and produce a formula for it. $\endgroup$
    – zooby
    Commented Jan 9, 2020 at 19:08
  • 1
    $\begingroup$ Someone changed it to "how fast the nerual network is converging". This is not what the curves show. The curves might show, for example, how the neural network improves in its game of chess. $\endgroup$
    – zooby
    Commented Jan 9, 2020 at 19:11
  • $\begingroup$ Your question assumed the NN is working fine. Usage of RNN in CNN tasks might lead to a decreasing curve, or increasing the learning rate, or maybe any hyperparameters. $\endgroup$
    – user9947
    Commented Jan 9, 2020 at 19:21
  • 1
    $\begingroup$ I don't think there is a universal mathematical formula that describes the learning curve, but what I know is that not all curves look like your diagram. In Q-learning there is almost no overfitting so the curve continue to rise but in other task there maybe overfitting so it may rise then fall slowly again. Maybe one could fit a function to for the curve? $\endgroup$
    – Clement
    Commented Jan 11, 2020 at 4:59

1 Answer 1

Reset to default
0
$\begingroup$

I don't think there's a theoretical reason for the shape, it's just showing the diminishing returns as training goes.

As for a mathematical function that describes this, exponential decay may fit well here (in case of loss, or inverse exponential decay if accuracy).

Also, I would not assume that this abrupt flat line of your examples are a universal.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .