How does Bishop derive $\ln p\left(\mathbf{x} \mid \mu, \sigma^{2}\right)$, when $p$ is a Gaussian?

Question

I am now reading the Bishop Machine Learning Book and going through every single equation.

We know that in the case of a single real-valued variable $x$, the Gaussian distribution is defined by $$\mathcal{N}\left(x \mid \mu, \sigma^{2}\right)=\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\}$$

Since the dataset $\mathbf{X}$ is i.i.d., we can therefore write the probability of the dataset, given $\mu \text { and } \sigma^{2}$, in the form

$$p\left(\mathbf{x} \mid \mu, \sigma^{2}\right)=\prod_{n=1}^{N} \mathcal{N}\left(x_{n} \mid \mu, \sigma^{2}\right)$$

We will resolve the numerical underflow by computing the sum of the log probabilities, therefore from the above two equations we have the following log-likelihood function, $$\ln p\left(\mathbf{x} \mid \mu, \sigma^{2}\right)=-\frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}\left(x_{n}-\mu\right)^{2}-\frac{N}{2} \ln \sigma^{2}-\frac{N}{2} \ln (2 \pi)$$

I am not quite sure how did they derive this in the book. Thanks a lot and have a good day.

You can refer to page 27 equation 1.54 for the detail.

Answer 1

This is not so difficult (just a bit verbose if you do all steps). Just replace $\mathcal{N}\left(x_{n} \mid \mu, \sigma^{2}\right)$ with $\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\}$ and apply the properties of logs (specifically, log of multiplication, log of exponential and log of a fraction). Try first before looking at my solution (otherwise you don't learn anything)!

\begin{align} p\left(\mathbf{x} \mid \mu, \sigma^{2}\right) &= \prod_{n=1}^{N} \mathcal{N}\left(x_{n} \mid \mu, \sigma^{2}\right)\\ &= \prod_{n=1}^{N}\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \iff \\ \log p\left(\mathbf{x} \mid \mu, \sigma^{2}\right) &= \log \left( \prod_{n=1}^{N}\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \right) \\ &= \sum_{n=1}^{N} \log \left(\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \right) \\ &= \sum_{n=1}^{N} \left( \log \left(\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \right) + \log \left( \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \right) \right) \\ &= \sum_{n=1}^{N} \left( - \frac{1}{2}\log \left(2 \pi \sigma^{2}\right) -\frac{1}{2 \sigma^{2}}(x-\mu)^{2} \right) \\ &= - \frac{1}{2} \sum_{n=1}^{N} \log \left(2 \pi \sigma^{2}\right) - \frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}(x-\mu)^{2} \\ &= - \frac{1}{2} \sum_{n=1}^{N} \left( \log 2 \pi + \log \sigma^{2}\right) - \frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}(x-\mu)^{2} \\ &= - \frac{1}{2} \sum_{n=1}^{N} \log 2 \pi - \frac{1}{2} \sum_{n=1}^{N} \log \sigma^{2}- \frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}(x-\mu)^{2} \end{align}