Prerequisites & Notation

Before You Begin

This chapter assumes comfort with random vectors, Gaussian distributions, and the basic properties of estimators introduced in Chapters 5 and 6. If any item feels unfamiliar, revisit the referenced material first.

Joint, marginal, and conditional distributions; Bayes' rule
Self-check: Can you write $f_{X|Y}(x|y)$ in terms of $f_{Y|X}(y|x)$ and $f_X(x)$ ?
Multivariate Gaussian distribution: density, marginals, conditionals
Self-check: Given $(\mathbf{X},\mathbf{Y})$ jointly Gaussian, do you know the formula for $\mathbb{E}[\mathbf{X}|\mathbf{Y}=\mathbf{y}]$ ?
Covariance matrices, positive semidefiniteness, matrix inversion(Review ch01 (telecom))
Self-check: Can you state when a covariance matrix $\boldsymbol{\Sigma}_y$ is strictly positive definite?
Classical estimation: bias, variance, MSE, CRLB, MLE(Review ch05, ch06)
Self-check: Can you state the CRLB and explain when the MLE attains it?
Orthogonal projection onto a subspace in an inner product space(Review ch01 (telecom))
Self-check: Can you characterize the projection of $\mathbf{x}$ onto $\text{span}(\mathbf{y}_1,\ldots,\mathbf{y}_m)$ by an orthogonality condition?

Notation for This Chapter

Symbols used throughout Chapter 7. The Bayesian viewpoint treats the parameter as a random variable, so every symbol has a prior distribution.

Symbol	Meaning	Introduced
$\theta, \boldsymbol{\theta}$	Scalar or vector parameter to be estimated (random in the Bayesian framework)	s01
$Y, \mathbf{Y}$	Scalar or vector observation	s01
$f_\theta(\theta)$	Prior density of $\theta$	s01
$f_{Y\|\theta}(y\|\theta)$	Likelihood: conditional density of $Y$ given $\theta$	s01
$f_{\theta\|Y}(\theta\|y)$	Posterior density of $\theta$ given the observation $Y=y$	s01
$\hat{\theta}_{\text{MAP}}(y)$	Maximum a posteriori (MAP) estimator	s02
$\hat{\theta}_{\text{MMSE}}(y) = g_{\text{mmse}}(y)$	Minimum mean-square error estimator $= \mathbb{E}[\theta\|Y=y]$	s02
$\hat{\theta}_{\text{LMMSE}}(y)$	Linear MMSE estimator (affine function of $\mathbf{Y}$ )	s04
$\mathbf{A}$	LMMSE gain matrix $\boldsymbol{\Sigma}_{\theta y}\boldsymbol{\Sigma}_y^{-1}$	s04
$\boldsymbol{\Sigma}_\theta$	Prior covariance of $\boldsymbol{\theta}$	s04
$\boldsymbol{\Sigma}_{y}$	Covariance of the observation $\mathbf{Y}$	s04
$\boldsymbol{\Sigma}_{xy}$	Cross-covariance $\text{Cov}(\boldsymbol{\theta},\mathbf{Y})$	s04
$\mathbf{m}_\theta, \mathbf{m}_y$	Mean vectors $\mathbb{E}[\boldsymbol{\theta}]$ , $\mathbb{E}[\mathbf{Y}]$	s04
$\boldsymbol{\Sigma}_{\theta\|y}$	Posterior (MMSE-error) covariance $\boldsymbol{\Sigma}_\theta - \boldsymbol{\Sigma}_{\theta y}\boldsymbol{\Sigma}_y^{-1}\boldsymbol{\Sigma}_{y\theta}$	s04

← Ch 6 The Bayesian Framework