1
$\begingroup$

The E step on the EM algorithm asks us to set the value of the variational lower bound to be equal to the posterior probability of the latent variable, given the data points and parameters. Clearly we are not taking any expectations here, then why is it called the Expectation step? Am I missing something here?

$\endgroup$
1
$\begingroup$

In expectation step, firstly we calculate the posterior of latent variable $Z$ and then the $Q(θ | θ^{(t)})$ is defined as the expected value of the log likelihood of $θ$, with respect to the current conditional contribution of $Z$ given $X$ and the current estimates of $θ^{(t)}$. In maximization step, we update $θ$ using the argmax on $Q$, with respect to $θ$.

$$Q(θ | θ^{(t)}) = E_{Z|X,θ^{(t)}}[logL(θ;Χ,Z)]$$

To be more intuitive, think of k-means as a special case of EM, where in expectation step the $Z$ variables are defined, that is the latent variables indicating the membership in a cluster, and calculated in a hard assignment way. In maximization step the $μ$s of the clusters are updated. If you want to see the corresponding relation for $Q$ in k-means, I suggest you read the chapter 9.3.2 in C.Bishop's book: Pattern Recognition and Machine Learning.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I understood the intuition behind the E step in k means, but is it possible to extend the same intuition to the general EM algorithm?How do we visualise the posterior as an expectation in the derivation of the general EM algorithm? Or equivalently, can we show that the expectation term in the formula given by you for the E step can be expressed as the posterior? $\endgroup$ – Manish Kausik Hari Baskar Oct 7 at 3:10
  • $\begingroup$ @Manish Kausik Hari Baskar. The Q function uses the posterior calculated in E step as the sum of posteriors * $ln(X,Z|θ)$ over Z. Also, I fixed my post. $\endgroup$ – ddaedalus Oct 7 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.