Exercises

Given $Y=y$, we know $\theta=y$ exactly, so $f_{\theta|Y}(\theta|y) = \delta(\theta - y)$.

Both estimators equal $y$: the mean of a point mass at $y$ is $y$, and its mode is $y$. The MMSE is zero. $\blacksquare$

Choosing $\phi(Y) = \mathbb{1}\{Y \in A\}$, the hypothesis becomes $\mathbb{E}[(\theta - \hat\theta(Y))\mathbb{1}\{Y \in A\}] = 0$ for every measurable $A$.

Equivalently, $\mathbb{E}[\theta\mathbb{1}\{Y \in A\}] = \mathbb{E}[\hat\theta(Y)\mathbb{1}\{Y \in A\}]$ for all $A$. This is exactly the defining property of the conditional expectation: a $\sigma(Y)$-measurable random variable whose integral over any $\sigma(Y)$-set equals that of $\theta$. Hence $\hat\theta(Y) = \mathbb{E}[\theta|Y]$ a.s. $\blacksquare$

$f_{Y|\theta}(\mathbf{y}|\theta) = \prod_{i=1}^N \frac{\theta^{y_i}e^{-\theta}}{y_i!}$.

The posterior is proportional to $\theta^{\sum y_i} e^{-(N+\lambda)\theta}$, i.e. a $\text{Gamma}(\sum y_i + 1, N + \lambda)$ density.

$\hat\theta_{\text{MMSE}} = \mathbb{E}[\theta|\mathbf{y}] = \frac{\sum y_i + 1}{N + \lambda}$, a shrinkage of the MLE $\overline{y} = (\sum y_i)/N$ toward the prior mean $1/\lambda$. $\blacksquare$

The residual $\mathbf{e} = \boldsymbol\theta - \mathbf{A}\mathbf{Y} - \mathbf{b}$ must be orthogonal to $\phi(\mathbf{Y}) = \mathbf{1}$ and to $\phi(\mathbf{Y}) = \mathbf{Y}$.

$\mathbb{E}[\mathbf{e}] = \mathbf{m}_\theta - \mathbf{A}\mathbf{m}_y - \mathbf{b} = \mathbf{0} \Rightarrow \mathbf{b} = \mathbf{m}_\theta - \mathbf{A}\mathbf{m}_y$.

$\mathbb{E}[\mathbf{e}\mathbf{Y}^\top] = \mathbb{E}[(\boldsymbol\theta - \mathbf{A}\mathbf{Y})\mathbf{Y}^\top] - \mathbf{b}\mathbf{m}_y^\top$. Using $\mathbf{b}$ from the previous step and simplifying, $\boldsymbol\Sigma_{\theta y} - \mathbf{A}\boldsymbol\Sigma_y = \mathbf{0}$, so $\mathbf{A} = \boldsymbol\Sigma_{\theta y}\boldsymbol\Sigma_y^{-1}$. $\blacksquare$

$\hat\theta_{\text{MMSE}}(y) = \frac{\rho\sigma_\theta\sigma_y}{\sigma_y^2} y = \rho\frac{\sigma_\theta}{\sigma_y}y$.

$\text{Var}(\theta|Y=y) = \sigma_\theta^2 - \frac{\rho^2\sigma_\theta^2\sigma_y^2}{\sigma_y^2} = \sigma_\theta^2(1-\rho^2)$.

The MMSE equals the (constant) conditional variance: $\text{MMSE} = \sigma_\theta^2(1-\rho^2)$. As $|\rho| \to 1$, the MMSE tends to zero (perfect prediction); as $\rho \to 0$, it returns to the prior variance. $\blacksquare$

$\log f_{\theta|Y}(\theta|y) = -\frac{\theta^2}{2\sigma_\theta^2} - \frac{|y-\theta|}{b} + \text{const}$.

Differentiating in $\theta$ (on either side of $y$) and setting the derivative to zero: $-\theta/\sigma_\theta^2 + \text{sign}(y-\theta)/b = 0$.

If $\theta < y$: $\theta = \sigma_\theta^2/b$; if $\theta > y$: $\theta = -\sigma_\theta^2/b$. Combining with the constraint that the maximizer is the one closest to $y$, $\hat\theta_{\text{MAP}}(y) = \text{sign}(y)\cdot \max(0, |y| - \sigma_\theta^2/b) \cdot \mathbb{1}\{|y| > \sigma_\theta^2/b\} + \frac{\sigma_\theta^2}{b}\text{sign}(y) \cdot \mathbb{1}\{|y| \leq \sigma_\theta^2/b\}$. More cleanly, the MAP is a soft-thresholding of $y$. Because the Laplace density is not Gaussian, the posterior is not Gaussian and $\hat\theta_{\text{MMSE}}(y)$ is not affine in $y$. $\blacksquare$

The LMMSE formula becomes $\hat\theta_{\text{LMMSE}} = \mathbf{H}^\top(\mathbf{H}\mathbf{H}^\top + \sigma_w^2\mathbf{I})^{-1}\mathbf{Y}$.

Multiplying out, $\mathbf{H}^\top(\mathbf{H}\mathbf{H}^\top + \sigma_w^2\mathbf{I}) = (\mathbf{H}^\top\mathbf{H} + \sigma_w^2\mathbf{I})\mathbf{H}^\top$, so $\mathbf{H}^\top(\mathbf{H}\mathbf{H}^\top + \sigma_w^2\mathbf{I})^{-1} = (\mathbf{H}^\top\mathbf{H} + \sigma_w^2\mathbf{I})^{-1}\mathbf{H}^\top$.

Hence $\hat\theta_{\text{LMMSE}} = (\mathbf{H}^\top\mathbf{H} + \sigma_w^2\mathbf{I})^{-1}\mathbf{H}^\top\mathbf{Y}$, the ridge regression form. $\blacksquare$

$\mathbb{E}[\hat\theta_{\text{MMSE}}(\mathbf{Y})] = \mathbb{E}[\mathbb{E}[\boldsymbol\theta|\mathbf{Y}]] = \mathbb{E}[\boldsymbol\theta]$ by the tower property.

In the scalar Gaussian model, $\hat\theta_{\text{MMSE}}(Y) = \alpha Y$ with $\alpha < 1$. Given $\theta$, $\mathbb{E}[\alpha Y|\theta] = \alpha\theta \neq \theta$ whenever $\theta \neq 0$. The MMSE is conditionally biased toward the prior mean — a feature, not a bug. $\blacksquare$

$\hat{\mathbf{X}} = \mathbf{A}\mathbf{Y}$ (with $\mathbf{A} = \boldsymbol\Sigma_x\mathbf{H}^H\boldsymbol\Sigma_y^{-1}$) and $\mathbf{e} = \mathbf{X} - \mathbf{A}\mathbf{Y}$ are both affine in the Gaussian pair $(\mathbf{X}, \mathbf{Z})$, hence jointly Gaussian.

$\mathbb{E}[\hat{\mathbf{X}}\mathbf{e}^H] = \mathbf{0}$ by the orthogonality principle (the residual is uncorrelated with any linear function of $\mathbf{Y}$, and $\hat{\mathbf{X}}$ is one).

For jointly Gaussian vectors, zero cross-covariance implies independence. $\blacksquare$

The Bayesian Fisher information matrix (prior + data) is $\mathbf{J}_B = \boldsymbol{\Sigma}_{h}^{-1} + {\sigma^2}^{-1}\mathbf{X}_p^H\mathbf{X}_p$.

The Bayesian CRLB gives $\text{Cov}(\mathbf{h} - \hat{\mathbf{h}}) \succeq \mathbf{J}_B^{-1}$, so $\text{tr}(\text{Cov}(\cdot)) \geq \text{tr}(\mathbf{J}_B^{-1})$.

From TMMSE Channel Estimator, the MMSE posterior covariance is exactly $\mathbf{J}_B^{-1}$. Hence the MMSE estimator attains the Bayesian CRLB — a special property of Gaussian models. $\blacksquare$

With $S = \sum Y_i$, $f_{\theta|\mathbf{Y}}(\theta) \propto \theta^{a+S-1}(1-\theta)^{b+N-S-1}$, i.e. $\text{Beta}(a+S, b+N-S)$.

MMSE = posterior mean = $(a+S)/(a+b+N)$. MAP = posterior mode = $(a+S-1)/(a+b+N-2)$ (assuming $a+S > 1$ and $b+N-S > 1$).

As $a,b \to 0$, both tend to the MLE $S/N$. The Beta$(0,0)$ prior is an improper prior (Haldane's prior); it serves as the non-informative limit. $\blacksquare$

Writing $\mathbf{m}^\star = \mathbb{E}[\boldsymbol\theta|\mathbf{Y}]$, $\boldsymbol\theta - \hat\theta = (\boldsymbol\theta - \mathbf{m}^\star) + (\mathbf{m}^\star - \hat\theta)$.

$\|\boldsymbol\theta - \hat\theta\|^2 = \|\boldsymbol\theta - \mathbf{m}^\star\|^2 + 2(\boldsymbol\theta - \mathbf{m}^\star)^\top(\mathbf{m}^\star - \hat\theta) + \|\mathbf{m}^\star - \hat\theta\|^2$.

Since $\mathbf{m}^\star$ and $\hat\theta$ are functions of $\mathbf{Y}$, the cross term has expectation zero by the orthogonality principle. Hence $\mathbb{E}[\|\boldsymbol\theta - \hat\theta\|^2] = \text{MMSE} + \mathbb{E}[\|\mathbf{m}^\star - \hat\theta\|^2]$. $\blacksquare$

$Y | \theta \sim \mathcal{CN}(0, |\theta|^2 + \sigma_w^2) = \mathcal{CN}(0, 1 + \sigma_w^2)$, independent of the sign of $\theta$.

By Bayes, $\Pr(\theta=+1|Y=y) = \Pr(\theta=-1|Y=y) = 1/2$ for every $y$, because the likelihoods are identical.

$\hat\theta_{\text{MMSE}}(y) = (+1)(1/2) + (-1)(1/2) = 0$ for all $y$. Without a phase reference, the receiver cannot distinguish $+1$ from $-1$ and the MMSE collapses to the prior mean. This motivates differential encoding or pilot-aided coherent detection. $\blacksquare$

Conditional on component $k$, the posterior is $\mathcal{N}(\mu_k^{\text{post}}, \sigma_k^{\text{post},2})$ with $\mu_k^{\text{post}} = \frac{\sigma_k^2 y + \sigma_w^2\mu_k}{\sigma_k^2 + \sigma_w^2}$ and $\sigma_k^{\text{post},2} = \sigma_k^2\sigma_w^2/ (\sigma_k^2+\sigma_w^2)$.

Posterior weights $\tilde\pi_k(y) \propto \pi_k \mathcal{N}(y; \mu_k, \sigma_k^2 + \sigma_w^2)$, normalized to sum to one.

$\hat\theta_{\text{MMSE}}(y) = \sum_k \tilde\pi_k(y)\mu_k^{\text{post}}$. This is a smooth soft-max between the per-component shrinkage estimates, with weights determined by how well $y$ matches each mixture component. $\blacksquare$

$\mathbf{h} - \tilde{\mathbf{G}}\mathbf{y} = (\mathbf{I} - \tilde{\mathbf{G}}\mathbf{X}_p)\mathbf{h} - \tilde{\mathbf{G}}\mathbf{w}$. Taking expectations over $\mathbf{h} \sim \mathcal{CN}(\mathbf{0}, \boldsymbol{\Sigma}_{h})$ and $\mathbf{w} \sim \mathcal{CN}(\mathbf{0}, \sigma^2\mathbf{I})$ independent: $$\mathbb{E}\|\mathbf{h} - \tilde{\mathbf{G}}\mathbf{y}\|^2 = \text{tr}\!\left[(\mathbf{I} - \tilde{\mathbf{G}}\mathbf{X}_p) \boldsymbol{\Sigma}_{h}(\mathbf{I} - \tilde{\mathbf{G}}\mathbf{X}_p)^H\right] + \sigma^2\,\text{tr}(\tilde{\mathbf{G}}\tilde{\mathbf{G}}^H).$$

This is minimized over $\tilde{\mathbf{G}}$ by setting $\tilde{\mathbf{G}} = \mathbf{G}^\star = \boldsymbol{\Sigma}_{h}\mathbf{X}_p^H (\mathbf{X}_p\boldsymbol{\Sigma}_{h}\mathbf{X}_p^H + \sigma^2\mathbf{I})^{-1}$ — the true-covariance MMSE estimator.

Any $\tilde{\mathbf{G}} \neq \mathbf{G}^\star$ gives a larger MSE. Mismatching the prior costs. $\blacksquare$

$\boldsymbol\Sigma_{\theta y}\boldsymbol\Sigma_y^{-1}\boldsymbol\Sigma_{y\theta} = \boldsymbol\Sigma_{\theta y}\boldsymbol\Sigma_y^{-1/2}(\boldsymbol\Sigma_{\theta y} \boldsymbol\Sigma_y^{-1/2})^H \succeq \mathbf{0}$.

Hence $\boldsymbol\Sigma_{\theta|y} = \boldsymbol\Sigma_\theta - \text{(PSD)} \preceq \boldsymbol\Sigma_\theta$. Equality iff the correction is zero, i.e. $\boldsymbol\Sigma_{\theta y} = \mathbf{0}$ — the observation is uncorrelated with the parameter. $\blacksquare$

$\boldsymbol\theta - \tilde g = (\boldsymbol\theta - g) - \phi$.

$\|\boldsymbol\theta - \tilde g\|^2 = \|\boldsymbol\theta - g\|^2 - 2(\boldsymbol\theta - g)^\top\phi + \|\phi\|^2$.

The middle term vanishes by hypothesis, leaving $\mathbb{E}\|\boldsymbol\theta - \tilde g\|^2 = \mathbb{E}\|\boldsymbol\theta - g\|^2 + \mathbb{E}\|\phi\|^2 \geq \mathbb{E}\|\boldsymbol\theta - g\|^2$. $\blacksquare$

$\hat\theta_{\text{MMSE}}(y) = \frac{\sqrt{\text{SNR}}}{1+\text{SNR}}y$ and the posterior variance is $\frac{1}{1+\text{SNR}}$. So $\text{MMSE}(\text{SNR}) = \frac{1}{1+\text{SNR}}$.

$I(\theta;Y) = \tfrac{1}{2}\log(1+\text{SNR})$ nats (scalar Gaussian capacity).

$\frac{d}{d\,\text{SNR}}\tfrac{1}{2}\log(1+\text{SNR}) = \frac{1}{2(1+\text{SNR})} = \tfrac{1}{2}\text{MMSE}(\text{SNR})$. The Guo–Shamai–Verdú I-MMSE identity holds and is tight for Gaussian inputs. $\blacksquare$

For $y=1$: $f_{\theta|Y}(\theta|1) \propto \theta$ on $[0,1]$, i.e. $\text{Beta}(2,1)$. For $y=0$: $f_{\theta|Y}(\theta|0) \propto 1-\theta$, i.e. $\text{Beta}(1,2)$.

$\hat\theta_{\text{MMSE}}(1) = 2/3$, $\hat\theta_{\text{MMSE}}(0) = 1/3$.

$\text{Var}(\theta|Y=y) = ab/[(a+b)^2(a+b+1)]$. For each $y$ this equals $2/(9\cdot 4) = 1/18$. So $\text{MMSE} = 1/18 \approx 0.056$ (versus the prior variance $1/12 \approx 0.083$). $\blacksquare$

Define the joint score $S = \frac{\partial}{\partial\theta}\log f_{\theta,Y}(\theta,Y) = \frac{\partial}{\partial\theta} \log f_\theta(\theta) + \frac{\partial}{\partial\theta}\log f_{Y|\theta}(Y|\theta)$, so $S = S_P + S_D$ (prior and data scores).

Under regularity, $\mathbb{E}[S_P S_D] = \mathbb{E}[S_P \cdot \mathbb{E}[S_D|\theta]] = 0$ because the expected data score given $\theta$ is zero.

Using integration by parts, $\mathbb{E}[(\theta-\hat\theta(Y))\cdot S] = 1$.

$1 = \mathbb{E}[(\theta-\hat\theta)\cdot S]^2 \leq \mathbb{E}[(\theta-\hat\theta)^2]\cdot\mathbb{E}[S^2] = \mathbb{E}[(\theta-\hat\theta)^2]\cdot J_B$.

Rearranging, $\mathbb{E}[(\theta-\hat\theta)^2] \geq 1/J_B$. This is the Bayesian CRLB, tight (with equality) in the Gaussian-Gaussian model. $\blacksquare$