Why L2 loss is more commonly used in Neural Networks than other loss functions? What is the reason to L2 being a default choice in Neural Networks?

  • 1
    $\begingroup$ By "L2 loss" do you mean L2 penalty/regularisation? $\endgroup$
    – nbro
    Jul 27 '20 at 20:39

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):


  1. L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
  2. Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator"). Source: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.

L2 Regularization:

  1. Using L2 regularization is equivalent to invoking a Gaussian prior (see https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior) on your model/estimator. If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian. From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: https://en.wikipedia.org/wiki/Conjugate_prior).

  2. As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.

  • 4
    $\begingroup$ I think it's important to note that L2 loss definitely isn't used more commonly than other loss functions. In fact if anything I would say cross-entropy has the largest appearance as it ties in with softmax activations which is all around excellent for classification. $\endgroup$
    – Recessive
    Jul 28 '20 at 4:46
  • $\begingroup$ @Recessive I agree $\endgroup$
    – pedrum
    Jul 28 '20 at 10:24
  • 3
    $\begingroup$ I think this answer doesn't clarify an important point. An "l2 loss" would be any loss that uses the "l2 norm" as a regularisation term (and, in that case, you will get MAP). This loss can be the MSE or it can e.g. the cross-entropy, i.e. l2 norm can be used for regression or classification. $\endgroup$
    – nbro
    Jul 28 '20 at 13:03
  • $\begingroup$ @nbro MSE loss specifically is sometimes referred to as L2 loss, so that one was probably meant in the question (e.g., see ml-cheatsheet.readthedocs.io/en/latest/…) $\endgroup$
    – Dennis Soemers
    Jul 29 '20 at 19:17
  • $\begingroup$ @DennisSoemers Right. I suspected that L2 loss couldn't just refer to any loss that uses L2 norm, but, well, I guess it's a little bit misleading to call MSE L2 loss when it doesn't use L2 regularisation, but I guess the 2 stands for the square in that case. Anyway, why not just calling it MSE? I don't get it. $\endgroup$
    – nbro
    Jul 29 '20 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.