Chapter 17: Discrete-Time Markov Chains

Learning Objectives

• State the Markov property and verify it for a given process
• Compute $n$-step transition probabilities via matrix powers and the Chapman-Kolmogorov equation
• Classify states as transient/recurrent, aperiodic/periodic, and identify communication classes
• Find the stationary distribution of an irreducible chain and prove convergence
• Apply detailed balance to verify reversibility and construct MCMC samplers
• Model the Gilbert-Elliott channel and ARQ protocols as Markov chains