Exercises

$\mathbb{E}_\theta[\bar{Y}_n] = \tfrac{1}{n}\sum_i \mathbb{E}_\theta[Y_i] = \theta$, so $\bar{Y}_n$ is unbiased.

$\text{Var}_\theta(\bar{Y}_n) = \tfrac{1}{n^2}\sum_i \text{Var}(Y_i) = \sigma^2/n$ and $\mathrm{MSE}_\theta(\bar{Y}_n) = 0 + \sigma^2/n = \sigma^2/n$.

$\log f_\theta(y) = -\tfrac{1}{2}\log(2\pi\sigma^2) - (y-\theta)^2/(2\sigma^2)$. $\partial_\theta \log f_\theta(y) = (y-\theta)/\sigma^2$.

$\text{Var}_\theta((Y-\theta)/\sigma^2) = \sigma^2/\sigma^4 = 1/\sigma^2$.

$\partial^2_\theta \log f_\theta(y) = -1/\sigma^2$ (constant). So $-\mathbb{E}[\partial^2_\theta] = 1/\sigma^2$. The two expressions agree.

$J(\theta) = n/\sigma^2$.

$\text{Var}_\theta(\hat{\theta}) \geq \sigma^2/n$. The sample mean attains this bound, so it is efficient.

$\log p_\lambda(y) = y\log\lambda - \lambda - \log(y!)$, $\partial_\lambda \log p_\lambda(y) = y/\lambda - 1 = (y-\lambda)/\lambda$.

$J(\lambda) = \text{Var}((Y-\lambda)/\lambda) = \lambda/\lambda^2 = 1/\lambda$.

$\text{Var}_\lambda(\hat{\lambda}) \geq \lambda$. The estimator $\hat{\lambda} = Y$ is unbiased and has variance $\lambda$, so it is efficient.

$f_p(\mathbf{y}) = \exp\bigl((\log \tfrac{p}{1-p}) \sum_i y_i + n \log(1-p)\bigr)$. Natural parameter $\eta = \log(p/(1-p))$, natural sufficient statistic $T(\mathbf{y}) = \sum_i y_i$. The natural-parameter image is $\mathbb{R}$, so $T$ is complete sufficient.

$\hat{p} = T(\mathbf{Y})/n = \bar{Y}$ is unbiased and is a function of $T$. Hence, by Lehmann--Scheffe, $\hat{p}$ is the MVUE. Its variance is $p(1-p)/n$, which matches the CRB $J^{-1}(p) = p(1-p)/n$.

$J(\phi) = \tfrac{1}{\sigma^2}\sum_{i=0}^{n-1} A^2 \sin^2(\omega_0 i + \phi) \approx \tfrac{n A^2}{2\sigma^2}$ for large $n$ and $\omega_0 \notin \{0,\pi\}$.

$\text{Var}_\phi(\hat{\phi}) \geq \tfrac{2\sigma^2}{n A^2}$. This is the standard carrier-phase estimation CRB.

$\mathbb{E}_A[\hat{A}] = \mathbf{s}^T(A\mathbf{s})/\|\mathbf{s}\|^2 = A$.

$\text{Var}_{A}(\hat{A}) = \|\mathbf{s}\|^2 \sigma^2/\|\mathbf{s}\|^4 = \sigma^2/\|\mathbf{s}\|^2 = 1/J(A)$.

$\hat{A}$ is a linear combination of the Gaussian $Y_i$'s, hence Gaussian: $\hat{A} \sim \mathcal{N}(A, \sigma^2/\|\mathbf{s}\|^2)$.

$(n-1)S^2_{n-1}/\sigma^2 \sim \chi^2_{n-1}$, so $\text{Var}\bigl((n-1)S^2_{n-1}/\sigma^2\bigr) = 2(n-1)$.

$\text{Var}(S^2_{n-1}) = \bigl(\sigma^2/(n-1)\bigr)^2 \cdot 2(n-1) = 2\sigma^4/(n-1)$. The CRB is $2\sigma^4/n$ (from the FIM of EJoint CRB: Mean and Variance of a Gaussian). The gap factor $n/(n-1)$ goes to 1 as $n \to \infty$: $S^2_{n-1}$ is asymptotically efficient but not efficient at finite $n$.

By the CRB, every unbiased estimator has variance $\geq 1/J(\theta)$. An efficient estimator attains this lower bound. Hence no unbiased estimator can have strictly smaller variance, and $\hat{\theta}$ is the MVUE. $\blacksquare$

$f_\theta(\mathbf{y}) = \theta^n \exp(-\theta \sum_i y_i)$, so $T(\mathbf{y}) = \sum_i y_i$ is complete sufficient (exponential family with full-dimensional natural parameter image $\mathbb{R}_{>0}$).

$J_{1}(\theta) = 1/\theta^2$, so $J(\theta) = n/\theta^2$ and the CRB on unbiased estimators of $\theta$ is $\theta^2/n$.

$T = \sum_i Y_i \sim \mathrm{Gamma}(n, \theta)$, so $1/T$ has $\mathbb{E}[1/T] = \theta/(n-1)$. Therefore $\hat{\theta} = (n-1)/T$ is unbiased, a function of $T$, hence by Lehmann--Scheffe the MVUE. Its variance is $\theta^2/(n-2)$, strictly exceeding the CRB $\theta^2/n$ for finite $n$.

$\log f_{\sigma^2}(\mathbf{y}) = -(n/2)\log(2\pi\sigma^2) - \|\mathbf{y}\|^2/(2\sigma^2)$. $\partial_{\sigma^2} \log f = -n/(2\sigma^2) + \|\mathbf{y}\|^2/(2\sigma^4)$. $\partial^2_{\sigma^2}\log f = n/(2\sigma^4) - \|\mathbf{y}\|^2/\sigma^6$. $J(\sigma^2) = -\mathbb{E}[\partial^2_{\sigma^2}\log f] = -n/(2\sigma^4) + n\sigma^2/\sigma^6 = n/(2\sigma^4)$.

CRB: $\text{Var}(\hat{\sigma^2}) \geq 2\sigma^4/n$. The estimator $\hat{\sigma^2} = \|\mathbf{Y}\|^2/n$ is unbiased and a function of the complete sufficient statistic $\|\mathbf{Y}\|^2$, hence MVUE. Its variance $2\sigma^4/n$ attains the CRB: efficient.

$\ell(A, \sigma^2) = -(n/2)\log(2\pi\sigma^2) - \tfrac{1}{2\sigma^2}\sum_i(y_i - As_i)^2$. $\partial_A \ell = \tfrac{1}{\sigma^2}\sum_i s_i(y_i - A s_i)$. $\partial_{\sigma^2}\ell = -n/(2\sigma^2) + \tfrac{1}{2\sigma^4}\sum_i(y_i-As_i)^2$.

$-\mathbb{E}[\partial^2_A \ell] = \|\mathbf{s}\|^2/\sigma^2$, $-\mathbb{E}[\partial^2_{\sigma^2}\ell] = n/(2\sigma^4)$, $-\mathbb{E}[\partial_A\partial_{\sigma^2}\ell] = 0$ (cross-term averages out). Hence $\mathbf{J} = \mathrm{diag}(\|\mathbf{s}\|^2/\sigma^2, n/(2\sigma^4))$.

$\text{Var}(\hat{A}) \geq \sigma^2/\|\mathbf{s}\|^2$, $\text{Var}(\hat{\sigma^2}) \geq 2\sigma^4/n$. Because the FIM is diagonal, joint estimation does not inflate the individual bounds.

$\tilde{g}(t) = \int g(\mathbf{y}) \, p(\mathbf{y} \mid T(\mathbf{y})=t)\, d\mathbf{y}$ is a convex combination (with unit-mass probability weights) of realizations of $g$ consistent with $T=t$, hence belongs to the convex hull of $\{g(\mathbf{y}) : T(\mathbf{y}) = t\}$. $\blacksquare$

Under $A$, $(Y_1/s_1, \mathbf{s}^T\mathbf{Y})$ is jointly Gaussian with $\mathbb{E}[Y_1/s_1]=A$, $\mathbb{E}[\mathbf{s}^T\mathbf{Y}]=A\|\mathbf{s}\|^2$, cross-covariance $\sigma^2$. Conditional mean formula: $\mathbb{E}_A[Y_1/s_1 \mid \mathbf{s}^T\mathbf{Y}=t] = A + \tfrac{\sigma^2}{\sigma^2\|\mathbf{s}\|^2}(t - A\|\mathbf{s}\|^2) = t/\|\mathbf{s}\|^2$.

Original variance $\sigma^2/s_1^2$. Rao--Blackwellized variance $\sigma^2/\|\mathbf{s}\|^2 \leq \sigma^2/s_1^2$, with equality only if $\|\mathbf{s}\|^2 = s_1^2$ (i.e., $s_j = 0$ for $j \geq 2$).

$\ell(\boldsymbol{\theta}) = c - \tfrac{1}{2}(\mathbf{y}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\mathbf{y}-\boldsymbol{\mu})$. $\partial_i \ell = (\partial_i \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\mathbf{y}-\boldsymbol{\mu})$.

$\partial_j \partial_i \ell = -(\partial_i \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\partial_j \boldsymbol{\mu}) + (\partial^2_{ij}\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\mathbf{y}-\boldsymbol{\mu})$.

Under $\boldsymbol{\theta}$, $\mathbf{Y}-\boldsymbol{\mu}$ is zero-mean, so the second term vanishes. Hence $[\mathbf{J}]_{ij} = -\mathbb{E}[\partial_i\partial_j\ell] = (\partial_i\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\partial_j\boldsymbol{\mu})$. $\blacksquare$

Let $r_i = \|\boldsymbol{\theta}-\mathbf{p}_i\|$ and $\mathbf{u}_i$ be the unit vector from $\mathbf{p}_i$ to $\boldsymbol{\theta}$. Then $\nabla_{\boldsymbol{\theta}}\alpha_i = \tfrac{1}{r_i}\mathbf{u}_i^\perp$, where $\mathbf{u}_i^\perp$ is the unit vector perpendicular to $\mathbf{u}_i$.

Using Exercise 15, $\mathbf{J}(\boldsymbol{\theta}) = \sum_{i=1}^n \tfrac{1}{\sigma_\alpha^2 r_i^2} \mathbf{u}_i^\perp (\mathbf{u}_i^\perp)^T$.

$\mathbf{J}^{-1}$ is the $2\times 2$ covariance ellipse of the CRB. Its eigenvectors are determined by the spread of bearing directions: if all $\mathbf{u}_i^\perp$ are nearly parallel (sensors collinear with the source), the FIM is nearly rank-one and the CRB ellipse is extremely elongated --- the classic bearings-only observability problem. This is the same geometry used in hearing-only and AoA-based localization in wireless positioning.

Support $[0, \theta]$ depends on $\theta$, so $\partial_\theta \int_0^\theta f_\theta = f_\theta(\theta) \neq 0$, and the regularity condition used to derive the CRB does not hold.

$Y_{(n)} = \max_i Y_i$ has CDF $(y/\theta)^n$, so pdf $n y^{n-1}/\theta^n$ on $[0,\theta]$. $\mathbb{E}[Y_{(n)}] = n\theta/(n+1)$. The unbiased estimator $\hat{\theta} = \tfrac{n+1}{n} Y_{(n)}$ is a function of the complete sufficient statistic $Y_{(n)}$, hence MVUE.

$\mathbb{E}[Y_{(n)}^2] = n\theta^2/(n+2)$, so $\text{Var}(Y_{(n)}) = \tfrac{n\theta^2}{(n+1)^2(n+2)}$. Then $\text{Var}(\hat{\theta}) = \tfrac{\theta^2}{n(n+2)}$. The rate is $\Theta(1/n^2)$, better than the $\Theta(1/n)$ that the CRB would suggest were it applicable --- a concrete reason why the regularity condition matters.

$\nabla_{\boldsymbol{\theta}} \log f_\theta(\mathbf{y}) = (\partial_{\boldsymbol{\theta}}\eta)^T (T(\mathbf{y}) - \mathbb{E}_{\boldsymbol{\theta}}[T(\mathbf{Y})])$, using that in canonical form $\nabla_\eta A(\eta) = \mathbb{E}_\eta[T]$.

$\mathbf{J}(\boldsymbol{\theta}) = \text{Cov}_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{\theta}}\log f_\theta(\mathbf{Y})) = (\partial_{\boldsymbol{\theta}}\eta)^T \text{Cov}_{\boldsymbol{\theta}}(T(\mathbf{Y})) (\partial_{\boldsymbol{\theta}}\eta)$. $\blacksquare$

$\sigma^2/\|\mathbf{s}\|^2 = 0.25/16 = 0.015625$.

Running the supplement $\mathrm{fsi\_ch05\_rao\_blackwell.py}$ with $1000$ trials yields an empirical variance consistently within 5% of $0.015625$. The matched filter sits on the CRB, in line with the efficiency proof of Exercise 7.

$\log f_\mu(y) = -\tfrac{1}{2}\log(2\pi\mu^2) - (y-\mu)^2/(2\mu^2)$.

$\partial_\mu \log f_\mu(y) = -1/\mu + (y-\mu)/\mu^2 + (y-\mu)^2/\mu^3$.

Taking the second derivative and expectation, after algebra (using $\mathbb{E}(Y-\mu)^2 = \mu^2$ and $\mathbb{E}(Y-\mu)^4 = 3\mu^4$), $J_{1}(\mu) = 3/\mu^2$. So for $n$ i.i.d. samples, $J(\mu) = 3n/\mu^2$ and $\mathrm{CRB} = \mu^2/(3n)$.

$\bar{Y}$ is unbiased with variance $\mu^2/n$, strictly larger than $\mu^2/(3n)$. The sample mean ignores the variance information. The efficient estimator exploits both moments. Lesson: parameter sharing changes which estimator is efficient.

$\mathbf{J}(\boldsymbol{\theta}) = \mathbf{A}^T\mathbf{A}/\sigma^2$. $\mathbf{J}^{-1} = \sigma^2(\mathbf{A}^T\mathbf{A})^{-1}$.

$\text{Cov}(\hat{\boldsymbol{\theta}}) = \sigma^2(\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T \mathbf{A}(\mathbf{A}^T\mathbf{A})^{-1} = \sigma^2(\mathbf{A}^T\mathbf{A})^{-1}$, matching $\mathbf{J}^{-1}$. The LS estimator is efficient, hence MVUE (and the MLE coincides with it in the Gaussian case).