1
$\begingroup$

enter image description here

enter image description here

This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is $f(x)$ not being multiplied by $p(x)$?

$\endgroup$
1
$\begingroup$

When we say that we have $N$ points that were "drawn from the probability distribution or probability density", this means that every point $x_n$ had the correct probability $p(x_n)$ of being sampled from the distribution when we were sampling our $n^{th}$ point.

For example, suppose that we wish to compute/estimate the expected value of a distribution given by flipping a coin, which is weighted such that it lands on heads two-thirds of the time (value of heads $= 1$), and tails one-thirds of the time (value of tails $= 0$). This means that the ground-truth probabilities are:

  • $p(1) = \frac{2}{3}$
  • $p(0) = \frac{1}{3}$

In this case, we actually wouldn't have to estimate anything by sampling; the exact probabilities are known, so we could just compute the expected value as $p(1) \times 1 + p(0) \times 0 = \frac{2}{3}$.

But now suppose that we do not know exactly how the coin is weighted, i.e. we do not know the exact values of $p(1)$ and $p(0)$. If we simply flip our weighted coin $N = 100$ times, we will in expectation find about $67$ samples of $x_n = 1$, and about $33$ samples of $x_n = 0$ (rounded to integers because we can't obtain a non-integer number of observations). So we cannot explicitly use the probabilities (because we do not know them), but they will implicitly be present in the number of repeated observations we have for every possible datapoint.

$\endgroup$
  • $\begingroup$ So you mean sampling is used to estimate probabilities, and that's why if p(1) = 2/3, we can expect 2/3rd of samples to be 1, and hence the expectation will account for the probabilities? $\endgroup$ – sage76 Jul 26 at 11:20
  • $\begingroup$ @sage76 Correct $\endgroup$ – Dennis Soemers Jul 26 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.