0
$\begingroup$

Every explanation of variational inference starts with the same basic premise: given an observed variable $x$, and a latent variable $z$,

$$ p(z|x)=\frac{p(x,z)}{p(x)} $$

and then proceeds to expand $p(x)$ as an expectation over $z$:

$$ p(x) = \int{p(x,z)dz} $$

and then states that it's too difficult to evaluate.

My very very basic question is why is $p(x)$ not simply equal to 1? It's an observed variable!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You're forgetting that $x$ can assume several values with different probability. Let's say that $x$ represent the roll of a fair dice. Then $p(x)$ will be 1/6 for all six possible values of $x$.

$$ p(\theta|x)=\frac{p(x|\theta)p(\theta)}{p(x)} $$

If you rearrange the formula it becomes clear that the whole point of Bayes theorem is to say that we want matching prior $p(\theta)$ and posterior $p(\theta|x)$ distributions.

$$ \frac{p(\theta|x)}{p(\theta)}=1=\frac{p(x|\theta)}{p(x)} $$

cause if the left side holds, then it means that the likelihood predicted by our parameters $p(x|\theta)$ is equal to the real observed probability $p(x)$. So if we train a model on some observed dice rolls, we expect a perfect model to learn for each face of the dice the real probability $p(x)$ which is 1/6 and not 1.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the explanation. I guess the short answer is that contrary to the explanations I've read, here $x$ is NOT assumed to be observed, but is treated as a random variable. (btw. in your second formula, surely you didn't mean that the ratios are equal to 1 - it's only when $\theta$ and $x$ are independent). $\endgroup$
    – Abrrval
    Aug 4, 2022 at 17:22
  • 1
    $\begingroup$ I guess it is semantics on what we mean by 'observed'. Say we rolled a dice and obtained the number 6, then we have simulated a realisation from our random variable $X$ and observed the outcome of this one particular roll of the die. In the literature, when they say $x$ is observed, they typically mean that the random variable $X$ relates to something that we can observe (as opposed to the latent variable, which, by definition, is unobservable). So, whilst we have observed that the outcome of this one roll of the die was 6, it does not mean it will always be the case ... $\endgroup$
    – David
    Sep 3, 2022 at 21:27
  • 1
    $\begingroup$ ... and so we use the probability of the outcome of that roll when doing variational inference. $\endgroup$
    – David
    Sep 3, 2022 at 21:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .