2
$\begingroup$

Why L2 loss is more commonly used in Neural Networks than other loss functions? What is the reason to L2 being a default choice in Neural Networks?

$\endgroup$
  • $\begingroup$ By "L2 loss" do you mean L2 penalty/regularisation? $\endgroup$ – nbro Jul 27 at 20:39
4
$\begingroup$

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):

MSE:

  1. L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
  2. Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator"). Source: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.

L2 Regularization:

  1. Using L2 regularization is equivalent to invoking a Gaussian prior (see https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior) on your model/estimator. If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian. From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: https://en.wikipedia.org/wiki/Conjugate_prior).

  2. As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.

| improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ I think it's important to note that L2 loss definitely isn't used more commonly than other loss functions. In fact if anything I would say cross-entropy has the largest appearance as it ties in with softmax activations which is all around excellent for classification. $\endgroup$ – Recessive Jul 28 at 4:46
  • $\begingroup$ @Recessive I agree $\endgroup$ – pedrum Jul 28 at 10:24
  • 3
    $\begingroup$ I think this answer doesn't clarify an important point. An "l2 loss" would be any loss that uses the "l2 norm" as a regularisation term (and, in that case, you will get MAP). This loss can be the MSE or it can e.g. the cross-entropy, i.e. l2 norm can be used for regression or classification. $\endgroup$ – nbro Jul 28 at 13:03
  • $\begingroup$ @nbro good point, I agree $\endgroup$ – pedrum Jul 28 at 13:20
  • $\begingroup$ @nbro MSE loss specifically is sometimes referred to as L2 loss, so that one was probably meant in the question (e.g., see ml-cheatsheet.readthedocs.io/en/latest/…) $\endgroup$ – Dennis Soemers Jul 29 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.