While training a CNN model, I used an l1_l2 regularization (i.e. I applied both $L_1$ and $L_2$ regularization) on the final layers. While training, I saw the training and validation losses are dropping very nicely, but the accuracies aren't changing at all! Is that due to the high regularization rate?
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ The effect as you saw was that the algo is giving more priority in keeping it's weights low and sparse as compared to improving accuracy. Remember algo minimizes loss, it doesn't improve accuracy. Accuracy is generally related to loss unless you bring in regularizers. $\endgroup$– user9947Dec 23, 2020 at 11:34
-
$\begingroup$ So, I should use them carefully, right? $\endgroup$– Sepehr GolestanianDec 23, 2020 at 15:09
-
$\begingroup$ Maybe you should also provide the plots of the losses and accuracy, specify the training and validation datasets that you're using (including the number of examples that they contain), and specify the loss function you're using, and also the problem you're solving (binary classification)? $\endgroup$– nbroDec 23, 2020 at 19:24
Add a comment
|