I was reading the book Deep Learning by Ian Goodfellow. I had a doubt in the Maximum likelihood estimation section (Pg 131). I understand till the Eq 5.58 which describes what is being maximized in the problem.

$$ \theta_{\text{ML}} = \text{argmax}_{\theta} \sum_1^m \log(p_{\text{model}}(x^{(i)};\theta)) $$

However the next equation 5.59 restates this equation as:

$$ \theta_{\text{ML}} = \text{argmax}_{\theta} E_{x \sim \hat{p}_{\text{data}}}(\log(p_{\text{model}}(x;\theta)) $$

where $$\hat{p}_{\text{data}}$$ is described as the empirical distribution defined by the training data. Could someone explain what is meant by this empirical distribution? It seems to be different from the distribution parametrized by theta as that is described by $$ p_{\text{model}} $$


1 Answer 1


The idea behind this kind of reasoning is that there is a "true" distribution (unknown to us, mere mortals) and that the data is generated following this distribution. But what we don't really know the shape of the distribution, all we know is the distribution of the data that we have. This is called the empirical distribution. Let's see a simple example to illustrate the point.
Let's consider a die. Each number is equally likely to show up if we throw the die, so the true underlying distribution is uniform over the set $\{1, 2,\dots6\}$. Now let's say you ask your friend to throw the die 60 times, what you will see is likely something close to uniform over the set $\{1, 2,\dots6\}$ (not really uniform though, as that would be highly unlikely). This distribution is the empirical one, and as you collect more and more samples it will converge to the actual underlying distribution.
In your case what happened is the following:
$x_1,\dots, x_m$ is your sample (in the example above, the $60$ numbers that you see as your friend throws the die). This sample defines a distribution, $\hat{p}_{data}$. In the example above, $\hat{p}_{data}$ would likely be close to the uniform distribution over $\{1, \dots, 6\}$. Now you can think about the sum over $x_1\dots, x_m$ of $\log(p_{model}(x^{(i)};\theta))$ as the average of $\log(p_{model}(x;\theta))$, where $x$ is drawn according to $\hat{p}_{data}$.
Let me make another example with some actual numbers. Let's say you toss a fair coin $7$ times and you see $$\{H, H, H, T, H, T, H\}.$$ The empirical distribution is $\mathbb{P}(H)=5/7$, $\mathbb{P}(T)=2/7$. So if you compute the expectation $$\mathbb{E}_{x \sim \hat{p}_{data}}[\log(p_{model}(x;\theta)]$$ you get $$\log(p_{model}(H;\theta)\cdot\mathbb{P}(H) +\log(p_{model}(T;\theta)\cdot\mathbb{P}(T) = \\ \frac{5}{7}\log(p_{model}(H;\theta) +\frac{2}{7}\log(p_{model}(T;\theta)),$$ which is what you will get if you compute the first sum you wrote.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .