# Variational Inference: Approximate expected log likelihood via sampling

I'm working my way through a simple variational inference from scratch. For that, I assume that z denotes the probability of a coin showing heads. I'm using an uninformed prior ($$Beta(1,1)$$) and a likelihood of the form $$binomial(10, 4)$$ as I assume the coin has shown 4 times head in a total of 10 trials.

As a variational posterior, I set a beta-distribution as well. Now I want to compute $$\mathbb{E}[\text{log}\ p(\text{daten}|z)]$$, i.e. the likelihood cost. In each optimization step, $$z$$ follows the current variational distribution $$Beta(\alpha_{\text{var}}, \beta_{\text{var}})$$, therefore, according to Wiki, there is an analytical solution for $$\mathbb{E}[\text{log}\ p(\text{daten}|z)]$$:

$$\mathbb{E}[\text{log}\ p(\text{daten}|z)] = {\text{heads}\cdot(\psi(\alpha_{\text{var}})-\psi(\alpha_{\text{var}}+\beta_{\text{var}}))} + {\text{tails}\cdot(\psi(\beta_{\text{var}})-\psi(\alpha_{\text{var}}+\beta_{\text{var}}))}$$, where $$\psi$$ is the digamma function. In python for example:

alpha_va = 1
beta_va = 1
a = 4 * (scf.psi(alpha_va) - scf.psi(alpha_va + beta_va))
b = 6 * (scf.psi(beta_va) - scf.psi(alpha_va + beta_va))
print(a+b) # -10


According to my understanding, computing this term with a sample-based method should be easy:

• draw 1000 samples from the current beta variational posterior
• compute the corresponding likelihood with each sample and binomial(10, 4)
• compute the log and the mean

However, this gives me -4.5 instead of -10 for $$\mathbb{E}[\text{log}\ p(\text{daten}|z)]$$:

samples_posterior_z = beta.rvs(alpha_va, beta_va, size=1000)
samples_likelihoods = binom.pmf(n=10, k=4, p=samples_posterior_z)
e_log_p = np.mean(np.log(samples_likelihoods))
print(e_log_p) # -4.5


What am I doing wrong, or is there a general mistake in thinking here?