1
$\begingroup$

When the probability of observing a large positive error is larger than the probability of observing a large negative error in binary classification, how can this be modelled by a skewed noise distribution using the maximum likelihood approach?

The content is as follows:

From page 103 of: http://smlbook.org/book/sml-book-draft-latest.pdf

"Using the maximum likelihood approach, other assumptions about the noise or insights into its distribution can be incorporated in a similar way in the regression model (5.1). For instance, if we believe that the error is non-symmetric, in the sense that the probability of observing a large positive error is larger than the probability of observing a large negative error, then this can be modelled by a skewed noise distribution. Using the negative log-likelihood loss is then a systematic way of incorporating this skewness into the training objective."

$\endgroup$

1 Answer 1

0
$\begingroup$

From Bayesian statistics maximizing the data likelihood given a hypothesis of a specific parametric statistical model to be regressed along with a specific noise as source randomness is equivalent to minimizing the negative log-likelihood (NLL) of a specific form, for instance, from your previous question you've already known for any linear regression model with a Gaussian distributed additive noise the maximum likelihood approach demands the form of squared error loss. Thus the maximum likelihood approach is a generic way for systematically constructing a loss function based on a parametric model of the observed data, ad hoc loss functions can be avoided in theory for any assumed generic parametric model with any generically distributed noise.

If the error is non-symmetric, then obviously you cannot assume it following the usual Gaussian or Laplace distributions, perhaps you could assume it following the skew normal distribution, beta distribution, etc. Of course for a linear regression model with a non-symmetric noise the NLL does not typically have a closed-form analytic solution, so to find the parameters minimizing the NLL would usually involve iterative numerical optimization methods such as gradient descents.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .