2
$\begingroup$

Why L2 loss is more commonly used in Neural Networks than other loss functions? What is the reason to L2 being a default choice in Neural Networks?

$\endgroup$
1
  • 1
    $\begingroup$ By "L2 loss" do you mean L2 penalty/regularisation? $\endgroup$
    – nbro
    Commented Jul 27, 2020 at 20:39

1 Answer 1

5
$\begingroup$

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):

MSE:

  1. L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
  2. Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator"). Source: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.

L2 Regularization:

  1. Using L2 regularization is equivalent to invoking a Gaussian prior (see https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior) on your model/estimator. If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian. From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: https://en.wikipedia.org/wiki/Conjugate_prior).

  2. As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.

$\endgroup$
5
  • 4
    $\begingroup$ I think it's important to note that L2 loss definitely isn't used more commonly than other loss functions. In fact if anything I would say cross-entropy has the largest appearance as it ties in with softmax activations which is all around excellent for classification. $\endgroup$
    – Recessive
    Commented Jul 28, 2020 at 4:46
  • $\begingroup$ @Recessive I agree $\endgroup$
    – pedrum
    Commented Jul 28, 2020 at 10:24
  • 3
    $\begingroup$ I think this answer doesn't clarify an important point. An "l2 loss" would be any loss that uses the "l2 norm" as a regularisation term (and, in that case, you will get MAP). This loss can be the MSE or it can e.g. the cross-entropy, i.e. l2 norm can be used for regression or classification. $\endgroup$
    – nbro
    Commented Jul 28, 2020 at 13:03
  • $\begingroup$ @nbro MSE loss specifically is sometimes referred to as L2 loss, so that one was probably meant in the question (e.g., see ml-cheatsheet.readthedocs.io/en/latest/…) $\endgroup$
    – Dennis Soemers
    Commented Jul 29, 2020 at 19:17
  • $\begingroup$ @DennisSoemers Right. I suspected that L2 loss couldn't just refer to any loss that uses L2 norm, but, well, I guess it's a little bit misleading to call MSE L2 loss when it doesn't use L2 regularisation, but I guess the 2 stands for the square in that case. Anyway, why not just calling it MSE? I don't get it. $\endgroup$
    – nbro
    Commented Jul 29, 2020 at 20:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .