Prerequisites & Notation

Before You Begin

This chapter builds the bridge between probability and estimation theory. The tools developed here — conditional expectation as a random variable, the MMSE estimator, the LMMSE estimator — are the workhorses of Bayesian inference and signal processing. Make sure the following are solid before proceeding.

Expectation, variance, and covariance(Review ch04)
Self-check: Can you compute $\mathbb{E}[g(X,Y)]$ for a given joint density $f_{X,Y}(x,y)$ ?
Joint and conditional distributions(Review ch03)
Self-check: Can you derive $f(y|x)$ from the joint density $f_{X,Y}(x,y)$ ?
Gaussian random vectors(Review ch06)
Self-check: Do you know the conditional distribution of a sub-vector of a jointly Gaussian vector?
Matrix inversion and positive definiteness
Self-check: Can you invert a $2 \times 2$ matrix and check whether a matrix is positive definite?

Notation for This Chapter

Symbols introduced or heavily used in this chapter.

Symbol	Meaning	Introduced
$\mathbb{E}[X\|Y]$	Conditional expectation of $X$ given $Y$ (a random variable, function of $Y$ )	s01
$\hat{X}_{\text{MMSE}}$	MMSE estimator: $\hat{X}_{\text{MMSE}} = \mathbb{E}[X\|Y]$	s02
$\hat{X}_{\text{LMMSE}}$	Linear MMSE estimator	s03
$\mathbf{C}_{XY}$	Cross-covariance matrix $\text{Cov}(\mathbf{X}, \mathbf{Y})$	s03
$\mathbf{C}_{YY}$	Covariance matrix of $\mathbf{Y}$	s03
$\\text{Var}(X\|Y)$	Conditional variance of $X$ given $Y$ (a random variable)	s04
$\\text{MSE}(g)$	Mean square error of estimator $g$ : $\mathbb{E}[(X - g(Y))^2]$	s02

← Ch 11 Conditional Expectation: The Deeper View