1
$\begingroup$

Why is laplace distribution less sensitive to outliers than normal distribution?. The following is the content.

(See page 103 of http://smlbook.org/book/sml-book-draft-latest.pdf)

"Also, in the regression case there is a duality between certain common loss functions and the maximum likelihood approach, as we have previously observed. For instance, in a regression model with additive noise, the squared error loss is equivalent to the negative log-likelihood if we assume a Gaussian noise distribution, $\epsilon \sim N(0, \sigma_{\epsilon}^2)$. Similarly, we noted above that the absolute error loss corresponds to an implicit assumption of Laplace distributed noise, $\epsilon \sim \mathcal{L}(0, B_{\epsilon})$.

This statistical perspective is one way to understand the fact that the absolute error loss is more robust (less sensitive to outliers) than the squared error loss, since the Laplace distribution has thicker tails compared to the Gaussian distribution. The Laplace distribution therefore encodes sporadic large noise values (that is, outliers) as more probable, compared to the Gaussian distribution"

$\endgroup$
2
  • $\begingroup$ Can you provide more context? Where did you find this claim? $\endgroup$
    – nbro
    Commented Nov 14, 2023 at 1:37
  • $\begingroup$ I updated the question with the content. $\endgroup$ Commented Nov 14, 2023 at 8:06

1 Answer 1

1
$\begingroup$

Your reference doesn't claim Laplace distribution is less sensitive to outliers than normal distribution, the intended claim is the absolute error loss is more robust (less sensitive to outliers) than the squared error loss used in many regression models.

According to your quoted well-known duality between certain common loss functions and the maximum likelihood approach for regression problems in Bayesian statistics, and in particular the squared error loss is equivalent to the negative log-likelihood loss $J(\mathbf{\theta})$ (expressed on the same page) if we assume a same Gaussian distributed additive noise, and a detailed equivalence derivation is shown here. Similarly the absolute error loss corresponds to a Laplace distributed noise.

Further per probability theory Laplace distribution generally has thicker tails compared to the Gaussian distribution, therefore Laplace distribution encodes sporadic larger noise values (as outliers) as more probable and thus possibly all likelihood factors $p(.)$ (between $0$ and $1$) in the above $J(\mathbf{\theta})$ containing the same outlier data in the same training set would be larger in the Laplace noise case , given the parametric statistical model to be regressed. And due to negative log-likelihood loss formula, the positive $J(\mathbf{\theta})$ would actually be smaller after applying the 'negative log' functions. Therefore its dual which is the absolute error loss is also smaller in the sense that it's more robust (less sensitive to outliers) than the squared error loss.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .