Chapter 8: The EM Algorithm and Iterative Estimation

Learning Objectives

• Formulate latent-variable models and recognize when direct maximum-likelihood estimation is intractable
• Derive the EM algorithm as coordinate ascent on the evidence lower bound (ELBO) via Jensen's inequality
• Prove monotonic improvement of the incomplete-data log-likelihood under EM updates
• Apply EM to Gaussian mixture models: derive responsibilities, M-step updates, and recognize K-means as a hard-assignment limit
• Connect EM to Baum-Welch for hidden Markov models, sparse Bayesian learning, and variational inference
• Diagnose practical pathologies: local maxima, singular covariances, sensitivity to initialization