4
$\begingroup$

In training a neural network, you often see the curve showing how fast the neural network is getting better. It usually grows very fast then slows down to almost horizontal.

Is there a mathematical formula that matches these curves?

enter image description here

Some similar curves are:

$$y=1-e^{-x}$$

$$y=\frac{x}{1+x}$$

$$y=\tanh(x)$$

$$y=1+x-\sqrt{1+x^2}$$

Is there a theoretical reason for this shape?

$\endgroup$
6
  • 1
    $\begingroup$ As you mention, "It usually grows very fast then slows down", so, of course, there isn't a single mathematical formula that would describe all learning curves. So, I suggest you clarify your question, which seems to be quite interesting. $\endgroup$
    – nbro
    Jan 9 '20 at 15:36
  • $\begingroup$ @nbro OK. By usually I mean always. And of course it slows down because the neural network can't keep increasing intelligence forever. But these curves do almost always look the same and do usually keep growing for a long time but very slowly. I'm just wondering if one can theortically model this curve and produce a formula for it. $\endgroup$
    – zooby
    Jan 9 '20 at 19:08
  • 1
    $\begingroup$ Someone changed it to "how fast the nerual network is converging". This is not what the curves show. The curves might show, for example, how the neural network improves in its game of chess. $\endgroup$
    – zooby
    Jan 9 '20 at 19:11
  • $\begingroup$ Your question assumed the NN is working fine. Usage of RNN in CNN tasks might lead to a decreasing curve, or increasing the learning rate, or maybe any hyperparameters. $\endgroup$
    – user9947
    Jan 9 '20 at 19:21
  • 1
    $\begingroup$ I don't think there is a universal mathematical formula that describes the learning curve, but what I know is that not all curves look like your diagram. In Q-learning there is almost no overfitting so the curve continue to rise but in other task there maybe overfitting so it may rise then fall slowly again. Maybe one could fit a function to for the curve? $\endgroup$ Jan 11 '20 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.