0
$\begingroup$

I'm working my way through a simple variational inference from scratch. For that, I assume that z denotes the probability of a coin showing heads. I'm using an uninformed prior ($Beta(1,1)$) and a likelihood of the form $binomial(10, 4)$ as I assume the coin has shown 4 times head in a total of 10 trials.

As a variational posterior, I set a beta-distribution as well. Now I want to compute $\mathbb{E}[\text{log}\ p(\text{daten}|z)]$, i.e. the likelihood cost. In each optimization step, $z$ follows the current variational distribution $Beta(\alpha_{\text{var}}, \beta_{\text{var}})$, therefore, according to Wiki, there is an analytical solution for $\mathbb{E}[\text{log}\ p(\text{daten}|z)]$:

$$\mathbb{E}[\text{log}\ p(\text{daten}|z)] = {\text{heads}\cdot(\psi(\alpha_{\text{var}})-\psi(\alpha_{\text{var}}+\beta_{\text{var}}))} + {\text{tails}\cdot(\psi(\beta_{\text{var}})-\psi(\alpha_{\text{var}}+\beta_{\text{var}}))}$$, where $\psi$ is the digamma function. In python for example:

alpha_va = 1
beta_va = 1
a = 4 * (scf.psi(alpha_va) - scf.psi(alpha_va + beta_va))
b = 6 * (scf.psi(beta_va) - scf.psi(alpha_va + beta_va))
print(a+b) # -10

According to my understanding, computing this term with a sample-based method should be easy:

  • draw 1000 samples from the current beta variational posterior
  • compute the corresponding likelihood with each sample and binomial(10, 4)
  • compute the log and the mean

However, this gives me -4.5 instead of -10 for $\mathbb{E}[\text{log}\ p(\text{daten}|z)]$:

samples_posterior_z = beta.rvs(alpha_va, beta_va, size=1000)
samples_likelihoods = binom.pmf(n=10, k=4, p=samples_posterior_z)
e_log_p = np.mean(np.log(samples_likelihoods))
print(e_log_p) # -4.5

What am I doing wrong, or is there a general mistake in thinking here?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.