I'm working my way through a simple variational inference from scratch. For that, I assume that z denotes the probability of a coin showing heads. I'm using an uninformed prior ($Beta(1,1)$) and a likelihood of the form $binomial(10, 4)$ as I assume the coin has shown 4 times head in a total of 10 trials.

As a variational posterior, I set a beta-distribution as well. Now I want to compute $\mathbb{E}[\text{log}\ p(\text{daten}|z)]$, i.e. the likelihood cost. In each optimization step, $z$ follows the current variational distribution $Beta(\alpha_{\text{var}}, \beta_{\text{var}})$, therefore, according to Wiki, there is an analytical solution for $\mathbb{E}[\text{log}\ p(\text{daten}|z)]$:

$$\mathbb{E}[\text{log}\ p(\text{daten}|z)] = {\text{heads}\cdot(\psi(\alpha_{\text{var}})-\psi(\alpha_{\text{var}}+\beta_{\text{var}}))} + {\text{tails}\cdot(\psi(\beta_{\text{var}})-\psi(\alpha_{\text{var}}+\beta_{\text{var}}))}$$, where $\psi$ is the digamma function. In python for example:

alpha_va = 1
beta_va = 1
a = 4 * (scf.psi(alpha_va) - scf.psi(alpha_va + beta_va))
b = 6 * (scf.psi(beta_va) - scf.psi(alpha_va + beta_va))
print(a+b) # -10

According to my understanding, computing this term with a sample-based method should be easy:

  • draw 1000 samples from the current beta variational posterior
  • compute the corresponding likelihood with each sample and binomial(10, 4)
  • compute the log and the mean

However, this gives me -4.5 instead of -10 for $\mathbb{E}[\text{log}\ p(\text{daten}|z)]$:

samples_posterior_z = beta.rvs(alpha_va, beta_va, size=1000)
samples_likelihoods = binom.pmf(n=10, k=4, p=samples_posterior_z)
e_log_p = np.mean(np.log(samples_likelihoods))
print(e_log_p) # -4.5

What am I doing wrong, or is there a general mistake in thinking here?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.