What is expectation-maximization in machine learning?

Question

While studying machine learning algorithms, I often see the term "expectation-maximisation" (or EM), and how it is used to estimate parameters, where the model depends on unobserved latent variables.

The way I see it, it is like a probabilistic/statistical way to make predictions (I think I'm confusing something but this is the way I see it).

Which made me wonder how exactly does EM differ from probabilistic classifiers like naive bayes or logistic regression? Is EM something that exists on its own or is it employed within machine learning algorithms? And, if we use naive Bayes, for example, are we implicitly using EM?

Answer 1

• EM algorithm is a numerical method. It is not specific to any machine learning model. Common applications include hidden markov model and mixed Gaussians. The algorithm is not a classifier.
• Logistic regression is a statistical model. You need to pick a numerical method for logistic regression.
• Naive Bayesian is a statistical model. You need to pick a numerical method (if closed-form posterior distribution not available).

You will need to understand maximum likelihood before you tackle the EM algorithm. Briefly, the maximum likelihood is a method for estimating the most likely parameters in your model. For instance, if you have a sequence of randomly and identically distributed Gaussian random variables, the maximum likelihood estimator for your Gaussian mean is just the sample mean.

When you fit a logistic regression, you use a numerical method (e.g. iteratively reweighted least squares) to maximise your log-likelihood function.

Everything is good, but it's not possible to maximum the likelihood directly if you have some latent variables. A common example is modelling your DNA sequences with hidden markov model, where the latent state is unknown.

You can't do it because you don't know the latent variables. If you do, they are not latent by definition.

EM algorithm is a numerical method to estimate maximum likelihood when you have latent variables. The mathematics is complicated but the idea is simple. You start off with some initial values for your parameters. You update your parameters and latent variables, and the algorithm converges when the change in the log-likelihood function falls below some threshold.