Prerequisites & Notation

Before You Begin

This chapter builds the frequentist backbone of estimation theory: bias, variance, sufficiency, the Cramer--Rao bound, and the Rao--Blackwell procedure for constructing minimum-variance unbiased estimators. Before you begin, make sure the following are second nature.

Multivariate Gaussian distribution, covariance matrices, PSD ordering(Review ch01)
Self-check: Can you write the density of $\mathbf{Y} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ and derive the log-likelihood up to constants?
Conditional expectation as an $L^2$ projection(Review ch02)
Self-check: Do you remember why $\mathbb{E}[X \mid T]$ minimizes the MSE over functions of $T$ , and what the tower property says about its variance?
Cauchy--Schwarz inequality and equality conditions
Self-check: Given $\int f g \, d\mu$ vs. $\sqrt{\int f^2} \sqrt{\int g^2}$ , can you state when equality holds?
Log-likelihood, score, and regularity conditions for differentiation under the integral sign
Self-check: Why does $\mathbb{E}_\theta[\partial_\theta \log f(Y;\theta)] = 0$ fail when the support depends on $\theta$ ?
Gaussian hypothesis testing and likelihood ratios(Review ch01)
Self-check: Can you connect the score function to an infinitesimal log-likelihood ratio?

Notation for This Chapter

Symbols introduced or emphasized in this chapter. For global conventions (vectors, matrices, probability), see the master notation page of this book.

Symbol	Meaning	Introduced
$\theta, \boldsymbol{\theta}$	Unknown parameter (scalar or vector) with $\theta \in \Lambda$	s01
$\Lambda$	Parameter domain, $\Lambda \subseteq \mathbb{R}^m$	s01
$\{f_\theta : \theta \in \Lambda\}$	Parametric family of densities / pmfs	s01
$g(\mathbf{y}), \hat{\theta}(\mathbf{y})$	Estimator function and its realized estimate	s01
$b(\theta)$	Bias of an estimator: $b(\theta) = \mathbb{E}_\theta[\hat{\theta}(\mathbf{Y})] - \theta$	s01
$\mathrm{MSE}_\theta(\hat{\theta})$	Mean-squared error at parameter $\theta$	s01
$\partial_\theta \log f_\theta(\mathbf{y})$	Score function	s02
$J(\theta)$	Scalar Fisher information	s02
$\mathbf{J}(\boldsymbol{\theta})$	Fisher information matrix (FIM)	s02
$T(\mathbf{y})$	Statistic / sufficient statistic for $\theta$	s03
$h(\mathbf{y}), g_\theta(t)$	Factors in the Fisher--Neyman factorization $f_\theta(\mathbf{y}) = g_\theta(T(\mathbf{y})) h(\mathbf{y})$	s03
$\eta(\theta), A(\theta)$	Natural parameter and log-partition function of an exponential family	s03
$\tilde{g}(T)$	Rao--Blackwellized estimator $\tilde{g}(T) = \mathbb{E}_\theta[g(\mathbf{Y}) \mid T(\mathbf{Y})]$	s04
$MVUE$	Minimum-variance unbiased estimator	s04

← Ch 4 The Estimation Problem