Consider our parametric model $p_\theta$ for an underlying probabilistic distribution $p_{data}$.

Now, the likelihood of an observation $x$ is generally defined as $L(\theta|x) = p_{\theta}(x)$.

The purpose of the likelihood is to quantify how good the parameters are. How can the probability density at a given observation, $p_{\theta}(x)$, measure how good the parameters $\theta$ are?

Is there any relation between the goodness of parameters and the probability density value of an observation?


1 Answer 1


The probability density is used to 'measure how good' the parameters are because it is a natural way of quantifying if these parameters are good for the observed data.

Also, as the notation often causes some confusion, $L(\theta | x)$ denotes the probability of all of your observed data, not just one value. Also the "$|$" may cause confusion as it looks like we are conditioning on $x$ but this is not the case - it may be better practice to use $L(\theta; x)$ which is the notation used when I was learning likelihood. Further, as you have written $L(\theta | x) = p_\theta(x)$ I would like to clarify that if we are being precise in our definitions then this is only correct if you have one observed data point, assuming that you meant $p_\theta(x)$ is the density of $X$.

In my example below I use $p_\mu(x)$ to denote the density of the normal distribution with mean parameter $\mu$, but the density of the likelihood is the product of all the densities (because we assumed iid data). It is crucial that you understand that in general the likelihood is the probability of your observed data and not just the density or the product of densities as you may not always have iid and so it will not always boil down to taking the product of some densities.

The idea of Maximum Likelihood is to maximising the (log-)likelihood for a given set of data. This means that we need to choose a probability distribution that is parameterised by some parameters $\boldsymbol{\theta}$ and then optimise the parameters such that the likelihood is maximum.

Assuming we don't remove any constants then this makes intuitive sense as maximising the likelihood would be maximising the probability that the data came from this distribution -- i.e. the data we observed is most likely to have come from the given distribution with the given parameters. This is important as it means we then have the most likely model of our data which allows us to use this distribution to make inference about our data.

As an example, imagine if I had some iid data $x_1, x_2, ..., x_n \sim \mathcal{N}(0, 1)$. Now I could try to fit a $\mathcal{N}(\mu, 1)$ to this data and optimise for $\mu$. If I chose e.g. $\mu = -1000000$ then the likelihood (again assuming we don't remove any constants) would be $\approx 0$. If I chose a value of $\mu = 0.1$ then the likelihood would be much higher because this parameter is closer to the true parameter value.

To see why it is higher, recall that the likelihood for iid data is given by $$\prod_{i=1}^n p_\mu(x_i)$$ and if we evaluated our likelihood at $\mu=-10000000$ then you're going to be taking the product of lots of numbers that are $\approx 0$ - if you think about the bell curve shape of a Normal distribution that is centred at $-10000000$ with variance 1 then the density at the true $x_i$ values (recalling they are simulated from a unit Normal) would be approx $0$ - whereas if we evaluated the $x_i$ at the density of a Normal distribution centred at $0.1$ then the density will be non-zero and so your likelihood will have a higher value.

To summarise, the density value can be used to measure how good parameters are for a set of data as maximising wrt the parameters is analogous to maximising the probability that your data arose from said distribution.

As an aside, note that the definition of likelihood is the probability of observing your data under some assumed distribution. For discrete random variables this is fine, but for continuous distributions we have to be a more subtle. For any continuous random variable $X$ we have $\mathbb{P}(X=x) = 0$. However, for a very small $\delta$ we can say that

$$\mathbb{P}(x - \frac{\delta}{2} < X \leq x + \frac{\delta}{2}) = \int_{x - \frac{\delta}{2}}^{x + \frac{\delta}{2}} f_X(x) dx \approx \delta p_X(x) \; ;$$

you can think of this approximation by visualising an integral and recalling that an integral represents the area under the curve, and so for small $\delta$ this integral can be approximated by taking the area of a rectangle which is width $\times$ height, where the width is $\delta$ and the height is $f_X(x)$. This justifies our use of the density function for continuous random variables. Note that typically in maximum likelihood we omit any multiplicative constants as they don't depend on the parameters which is what happens with the $\delta$ from this justification.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .