Why is it useful to track loss while the model is being trained?

Options are:

  1. Loss is only useful as a final metric. It should not be evaluated while the model is being trained.
  2. Loss dictates how effective the model is.
  3. Loss can help understand how much the model is changing per iteration. When it converges, that's an indicator that further training will have little benefit.
  4. None of the above
  • $\begingroup$ Hi and welcome to AI SE! Do you know something about machine learning or neural networks? This question should be easy to answer if you know something about neural networks and you have trained at least one once in your life. $\endgroup$ – nbro Jun 14 at 15:05
  • 1
    $\begingroup$ This looks like coursework? If it is, could you please add some of your own thoughts/analysis (use edit). Also, you may want to consider whether you have agreed to any code of conduct on such a course - e.g. if your correct answer leads to gaining a certificate - because posting a question from course material directly in order to get someone else to answer for you may lead to your results being nullified. $\endgroup$ – Neil Slater Jun 14 at 20:39

The loss function (aka cost function) measures the correctness of the predictions of the model.

For example, a simple cost function could be $|y - f(x)|$, where

  • $\hat{y} = f(x)$ is the prediction of the model $f$ when the input is $x$,
  • $y$ is the ground-truth label for input $x$ (i.e. what the model is supposed to output when the input is $x$), and
  • $|\cdot|$ is the absolute value

The loss function is also used to understand whether a model is overfitting (and underfitting) or not. More specifically, if the loss of the model evaluated on the training data (the training loss) keeps decreasing while the loss of the model evaluated on the validation data (validation loss) starts to increase, that's a good sign that the model is overfitting (i.e. just memorizing the training data, but it will likely perform poorly on non-training data). If the training loss does not decrease, that's a good sign of underfitting (i.e. the model is not capable of learning the patterns in the training data), so you may need to use a model with a bigger capacity, change the loss, or do something else.

The loss is typically not directly used as a measure of the performance of the model because the loss doesn't directly represent the performance of the model. In fact, the loss can be relatively big (with respect to another model's loss) and the model still performs well, but typically a lower loss corresponds to higher performance. The performance of the model is calculated differently depending on the task, model, and your goals. The most common performance measure is the accuracy (the number of correct predictions over the total number of predictions), but there are many other performance measures, such as the precision, recall, f1 score, AUC, etc., that emphasize different behavior that you expect from your model.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.