0
$\begingroup$

In deep learning, one way to determine whether the training has converged is to observe the movement of the loss values over iterations or epochs. One can choose any $\epsilon$ threshold and any metric. If the value is less than $\epsilon$, then the training has converged.

My question is: how big is the $\epsilon$ value that is usually used? Are there examples of papers that specifically state the threshold?

$\endgroup$

1 Answer 1

0
$\begingroup$

I assume you are talking about the training loss. I don't think there's much in the literature, because it's not a big problem by itself. You can pick any reasonable epsilon based on your initial experiments, and you'll likely do fine.

A bigger problem is overfitting (as the picture below shows). I.e. you can reach very low training loss values, or even zero in some cases, but your model will do poorly for new data.

A better way is to track the validation loss during your training, and stop training when it bottomed out, or stopped improving.

enter image description here

Source: https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .