While training a CNN model, I used an l1_l2 regularization (i.e. I applied both $L_1$ and $L_2$ regularization) on the final layers. While training, I saw the training and validation losses are dropping very nicely, but the accuracies aren't changing at all! Is that due to the high regularization rate?

  • 2
    $\begingroup$ The effect as you saw was that the algo is giving more priority in keeping it's weights low and sparse as compared to improving accuracy. Remember algo minimizes loss, it doesn't improve accuracy. Accuracy is generally related to loss unless you bring in regularizers. $\endgroup$ – DuttaA Dec 23 '20 at 11:34
  • $\begingroup$ So, I should use them carefully, right? $\endgroup$ – Sepehr Golestanian Dec 23 '20 at 15:09
  • $\begingroup$ Maybe you should also provide the plots of the losses and accuracy, specify the training and validation datasets that you're using (including the number of examples that they contain), and specify the loss function you're using, and also the problem you're solving (binary classification)? $\endgroup$ – nbro Dec 23 '20 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.