Ferkans — Interactive Telecom Tutor

ex-ch12-01

Easy

Let $Y \sim \text{Uniform}[0,1]$ and $X | Y = y \sim \text{Uniform}[0, y]$ . Find $\mathbb{E}[X|Y]$ and verify the tower property by computing $\mathbb{E}[X]$ both directly and via $\mathbb{E}[\mathbb{E}[X|Y]]$ .

Show Hint

For a uniform on $[0, y]$ , the mean is $y/2$ .

Solution

Find $\mathbb{E}[X|Y]$

$\mathbb{E}[X|Y=y] = y/2$ , so $\mathbb{E}[X|Y] = Y/2$ .

Tower check

$\mathbb{E}[\mathbb{E}[X|Y]] = \mathbb{E}[Y/2] = 1/4$ . Direct: $\mathbb{E}[X] = \int_0^1 \int_0^y \frac{x}{y}\,dx\,dy = \int_0^1 \frac{y}{2}\,dy = 1/4$ . $\checkmark$

ex-ch12-02

Easy

Prove that $\text{Var}(\mathbb{E}[X|Y]) \leq \text{Var}(X)$ .

Show Hint

Use the law of total variance.

Variance is always non-negative.

Solution

Apply the law of total variance

$\text{Var}(X) = \mathbb{E}[\text{Var}(X|Y)] + \text{Var}(\mathbb{E}[X|Y])$ . Since $\mathbb{E}[\text{Var}(X|Y)] \geq 0$ , we get $\text{Var}(\mathbb{E}[X|Y]) \leq \text{Var}(X)$ . $\blacksquare$

ex-ch12-03

Easy

Let $X$ and $Y$ be independent with $\mathbb{E}[X] = 3$ and $\text{Var}(X) = 4$ . What is $\mathbb{E}[X|Y]$ ? What is the MMSE?

Show Hint

What does independence imply for the conditional expectation?

Solution

Apply independence

$\mathbb{E}[X|Y] = \mathbb{E}[X] = 3$ . The MMSE $= \mathbb{E}[(X - 3)^2] = \text{Var}(X) = 4$ . Observing $Y$ gives no information about $X$ .

ex-ch12-04

Medium

Let $(X, Y)$ be jointly Gaussian with $\mu_X = 1$ , $\mu_Y = -1$ , $\sigma_X^2 = 4$ , $\sigma_Y^2 = 9$ , and $\rho = 0.6$ . (a) Find $\hat{X}_{\text{LMMSE}}$ given $Y = 2$ . (b) Find the MSE of the LMMSE estimator. (c) Is the LMMSE equal to the MMSE here?

Show Hint

Use the scalar LMMSE formula: $\hat{X} = \mu_X + \rho(\sigma_X/\sigma_Y)(Y - \mu_Y)$ .

Solution

Part (a)

$\hat{X} = 1 + 0.6 \cdot \frac{2}{3}(2 - (-1)) = 1 + 0.4 \cdot 3 = 2.2$ .

Part (b)

$\text{MSE} = \sigma_X^2(1 - \rho^2) = 4(1 - 0.36) = 2.56$ .

Part (c)

Yes. For jointly Gaussian variables, LMMSE = MMSE.

ex-ch12-05

Medium

Let $X \in \{-1, +1\}$ equiprobably and $Y = X + Z$ with $Z \sim \mathcal{N}(0, 1)$ independent of $X$ . (a) Find $\mathbb{E}[X|Y=y]$ . (b) Find the LMMSE estimator $\hat{X}_{\text{LMMSE}}$ . (c) Compare the MSE of both estimators numerically for $y = 0.5$ .

Show Hint

For (a), use Bayes' rule to get the posterior, then compute the posterior mean.

For (b), compute $\text{Cov}(X,Y)$ and $\text{Var}(Y)$ .

Solution

Part (a): MMSE

$\mathbb{E}[X|Y=y] = \tanh(y)$ , as derived in Example EMMSE Estimation of a Binary Signal in Gaussian Noise.

Part (b): LMMSE

$\text{Cov}(X,Y) = \mathbb{E}[XY] = \mathbb{E}[X^2] = 1$ (since $\mathbb{E}[XZ] = 0$ ). $\text{Var}(Y) = \text{Var}(X) + \text{Var}(Z) = 1 + 1 = 2$ . So $\hat{X}_{\text{LMMSE}} = (1/2)Y$ .

Part (c): Compare

For $y = 0.5$ : MMSE estimate $= \tanh(0.5) \approx 0.462$ . LMMSE estimate $= 0.25$ . The MMSE estimator uses the nonlinear $\tanh$ to better exploit the binary structure of $X$ .

ex-ch12-06

Medium

Prove the orthogonality principle: $\mathbb{E}[(X - \mathbb{E}[X|Y]) \cdot h(Y)] = 0$ for any measurable $h$ with $\mathbb{E}[h(Y)^2] < \infty$ .

Show Hint

Use the tower property and the 'pulling out known' property.

Solution

Apply iterated conditioning

$\mathbb{E}[(X - \mathbb{E}[X|Y]) h(Y)] = \mathbb{E}\bigl[\mathbb{E}[(X - \mathbb{E}[X|Y]) h(Y) | Y]\bigr] = \mathbb{E}\bigl[h(Y) \mathbb{E}[X - \mathbb{E}[X|Y] | Y]\bigr]$ .

Now $\mathbb{E}[X - \mathbb{E}[X|Y] | Y] = \mathbb{E}[X|Y] - \mathbb{E}[X|Y] = 0$ . $\blacksquare$

ex-ch12-07

Medium

Let $N \sim \text{Poisson}(\lambda)$ and $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and variance $\sigma^2$ , independent of $N$ . Let $S = \sum_{i=1}^N X_i$ (with $S = 0$ if $N = 0$ ). Use the law of total variance to find $\text{Var}(S)$ .

Show Hint

Condition on $N$ . What are $\mathbb{E}[S|N]$ and $\text{Var}(S|N)$ ?

Solution

Conditional moments

$\mathbb{E}[S|N] = N\mu$ and $\text{Var}(S|N) = N\sigma^2$ .

Apply total variance

$\text{Var}(S) = \mathbb{E}[N\sigma^2] + \text{Var}(N\mu) = \lambda\sigma^2 + \mu^2\lambda = \lambda(\sigma^2 + \mu^2)$ .

This is Wald's identity for the variance of a random sum.

ex-ch12-08

Medium

Derive the LMMSE estimator of $X$ given $\mathbf{Y} = (Y_1, Y_2)^\mathsf{T}$ where $X$ , $Y_1$ , $Y_2$ are zero-mean with $\text{Var}(X) = 1$ , $\text{Var}(Y_1) = \text{Var}(Y_2) = 2$ , $\text{Cov}(X, Y_1) = 0.8$ , $\text{Cov}(X, Y_2) = 0.5$ , $\text{Cov}(Y_1, Y_2) = 0.3$ .

Show Hint

Write $\mathbf{C}_{XY} = (0.8, 0.5)$ and form $\mathbf{C}_{YY}$ , then invert.

Solution

Set up

$\mathbf{C}_{XY} = \begin{pmatrix} 0.8 & 0.5 \end{pmatrix}$ , $\mathbf{C}_{YY} = \begin{pmatrix} 2 & 0.3 \\ 0.3 & 2 \end{pmatrix}$ .

Invert $\mathbf{C}_{YY}$

$\det(\mathbf{C}_{YY}) = 4 - 0.09 = 3.91$ . $\mathbf{C}_{YY}^{-1} = \frac{1}{3.91}\begin{pmatrix} 2 & -0.3 \\ -0.3 & 2 \end{pmatrix}$ .

Compute the estimator

$\hat{X} = \mathbf{C}_{XY}\mathbf{C}_{YY}^{-1}\mathbf{Y} = \frac{1}{3.91}(0.8 \cdot 2 - 0.5 \cdot 0.3, -0.8 \cdot 0.3 + 0.5 \cdot 2)\mathbf{Y} = \frac{1}{3.91}(1.45, 0.76)\mathbf{Y} \approx 0.371 Y_1 + 0.194 Y_2$ .

MSE $= 1 - \mathbf{C}_{XY}\mathbf{C}_{YY}^{-1}\mathbf{C}_{YX} \approx 1 - 0.371 \cdot 0.8 - 0.194 \cdot 0.5 \approx 0.606$ .

ex-ch12-09

Hard

Show that for any estimator $\hat{X} = g(Y)$ , the MSE can be decomposed as

$\mathbb{E}[(X - g(Y))^2] = \text{MMSE} + \mathbb{E}[(\mathbb{E}[X|Y] - g(Y))^2].$

Interpret each term.

Show Hint

Add and subtract $\mathbb{E}[X|Y]$ inside the square.

Use the orthogonality principle to show the cross-term vanishes.

Solution

Decompose

$\mathbb{E}[(X - g(Y))^2] = \mathbb{E}[(X - \mathbb{E}[X|Y])^2] + 2\mathbb{E}[(X - \mathbb{E}[X|Y])(\mathbb{E}[X|Y] - g(Y))] + \mathbb{E}[(\mathbb{E}[X|Y] - g(Y))^2]$ .

Cross-term vanishes

$\mathbb{E}[X|Y] - g(Y)$ is a function of $Y$ . By the orthogonality principle, $(X - \mathbb{E}[X|Y]) \perp h(Y)$ for any $h$ . So the cross-term is zero.

Conclude

$\mathbb{E}[(X - g(Y))^2] = \text{MMSE} + \mathbb{E}[(\mathbb{E}[X|Y] - g(Y))^2]$ .

The first term is the irreducible error. The second is the penalty for using $g$ instead of the optimal $\mathbb{E}[X|Y]$ . Setting $g = \mathbb{E}[X|\cdot]$ eliminates the penalty. $\blacksquare$

ex-ch12-10

Hard

Let $\mathbf{Y} = \mathbf{H}\mathbf{x} + \mathbf{w}$ where $\mathbf{x} \sim \mathcal{CN}(\mathbf{0}, \sigma_x^2\mathbf{I}_n)$ , $\mathbf{w} \sim \mathcal{CN}(\mathbf{0}, \sigma^2\mathbf{I}_m)$ independent of $\mathbf{x}$ , and $\mathbf{H} \in \mathbb{C}^{m \times n}$ is known. Derive the LMMSE estimator of $\mathbf{x}$ and its MSE matrix.

Show Hint

Compute $\mathbf{C}_{\mathbf{xY}}$ and $\mathbf{C}_{\mathbf{YY}}$ .

Use the matrix inversion lemma to get a dual form.

Solution

Compute covariances

$\mathbf{C}_{\mathbf{xY}} = \sigma_x^2 \mathbf{H}^H$ , $\mathbf{C}_{\mathbf{YY}} = \sigma_x^2\mathbf{H}\mathbf{H}^H + \sigma^2\mathbf{I}_m$ .

LMMSE estimator

$\hat{\mathbf{x}} = \sigma_x^2\mathbf{H}^H(\sigma_x^2\mathbf{H}\mathbf{H}^H + \sigma^2\mathbf{I})^{-1}\mathbf{Y}$ .

By the matrix inversion lemma: $\hat{\mathbf{x}} = (\mathbf{H}^H\mathbf{H} + (\sigma^2/\sigma_x^2)\mathbf{I})^{-1}\mathbf{H}^H\mathbf{Y}$ .

MSE matrix

$\mathbf{C}_{\tilde{\mathbf{x}}} = \sigma_x^2\mathbf{I} - \sigma_x^4\mathbf{H}^H(\sigma_x^2\mathbf{H}\mathbf{H}^H + \sigma^2\mathbf{I})^{-1}\mathbf{H} = \sigma^2(\mathbf{H}^H\mathbf{H} + (\sigma^2/\sigma_x^2)\mathbf{I})^{-1}$ .

ex-ch12-11

Hard

Prove the conditional Jensen's inequality: if $\varphi$ is convex and $\mathbb{E}[|\varphi(X)|] < \infty$ , then

$\varphi(\mathbb{E}[X|Y]) \leq \mathbb{E}[\varphi(X)|Y] \quad \text{a.s.}$

Show Hint

Use the supporting hyperplane characterization of convexity.

For convex $\varphi$ , there exist $a, b$ such that $\varphi(x) \geq ax + b$ and $\varphi(\mu) = a\mu + b$ at $\mu = \mathbb{E}[X|Y]$ .

Solution

Supporting line

Since $\varphi$ is convex, for each $\mu$ there exist constants $a(\mu), b(\mu)$ such that $\varphi(x) \geq a(\mu)x + b(\mu)$ for all $x$ , with equality at $x = \mu$ .

Condition and apply

Take $\mu = \mathbb{E}[X|Y]$ . Then $\mathbb{E}[\varphi(X)|Y] \geq a(\mu)\mathbb{E}[X|Y] + b(\mu) = a(\mu)\mu + b(\mu) = \varphi(\mu) = \varphi(\mathbb{E}[X|Y])$ . $\blacksquare$

ex-ch12-12

Hard

Let $X \sim \text{Exp}(\lambda)$ and $Y = X + Z$ where $Z \sim \text{Exp}(\mu)$ independent of $X$ . (a) Find $\text{Cov}(X, Y)$ . (b) Find $\hat{X}_{\text{LMMSE}}$ . (c) Is the LMMSE equal to the MMSE? Justify your answer.

Show Hint

$\text{Cov}(X, Y) = \text{Cov}(X, X+Z) = \text{Var}(X)$ .

Solution

Part (a)

$\text{Cov}(X, Y) = \text{Cov}(X, X+Z) = \text{Var}(X) = 1/\lambda^2$ .

Part (b)

$\text{Var}(Y) = 1/\lambda^2 + 1/\mu^2$ . $\hat{X}_{\text{LMMSE}} = \frac{1}{\lambda} + \frac{1/\lambda^2}{1/\lambda^2 + 1/\mu^2}(Y - 1/\lambda - 1/\mu) = \frac{\mu^2}{\lambda^2 + \mu^2}Y + \frac{\lambda^2/\mu - \mu/\lambda}{\lambda^2 + \mu^2}$ .

Part (c)

No. The joint distribution of $(X, Y)$ is not Gaussian (both marginals are exponential). The conditional $X|Y=y$ has a truncated distribution on $[0, y]$ , so $\mathbb{E}[X|Y=y]$ is a nonlinear function of $y$ . LMMSE $\neq$ MMSE.

ex-ch12-13

Medium

A sensor measures temperature $T$ with additive noise: $Y = T + W$ where $T \sim \mathcal{N}(20, 4)$ (in Celsius) and $W \sim \mathcal{N}(0, 1)$ independent of $T$ . (a) Find the LMMSE estimate of $T$ given $Y = 23$ . (b) What fraction of the total variance of $Y$ is explained by $T$ ?

Show Hint

This is Gaussian, so LMMSE = MMSE.

Solution

Part (a)

$\hat{T} = 20 + \frac{4}{4+1}(23 - 20) = 20 + 0.8 \cdot 3 = 22.4$ .

Part (b)

$\text{Var}(\mathbb{E}[T|Y]) = \text{Var}(T) - \text{MMSE} = 4 - 4/(4+1) = 4 - 0.8 = 3.2$ . Fraction explained: $3.2/4 = 0.8 = \rho^2$ where $\rho = \sigma_T/\sigma_Y = 2/\sqrt{5}$ , so $\rho^2 = 4/5 = 0.8$ . $\checkmark$

ex-ch12-14

Challenge

Let $\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \mathbf{C}_X)$ in $\mathbb{R}^n$ and suppose we observe a noisy linear combination $Y = \mathbf{a}^\mathsf{T}\mathbf{X} + W$ where $\mathbf{a} \in \mathbb{R}^n$ is known and $W \sim \mathcal{N}(0, \sigma^2)$ is independent. Show that the MMSE estimate of $\mathbf{X}$ given $Y$ is

$\hat{\mathbf{X}} = \frac{\mathbf{C}_X \mathbf{a}}{\mathbf{a}^\mathsf{T}\mathbf{C}_X\mathbf{a} + \sigma^2} Y$

and find the MSE matrix.

Show Hint

Compute $\mathbf{C}_{\mathbf{X}Y}$ and $\text{Var}(Y)$ .

Since everything is Gaussian, LMMSE = MMSE.

Solution

Cross-covariance

$\mathbf{C}_{\mathbf{X}Y} = \mathbb{E}[\mathbf{X}Y] = \mathbb{E}[\mathbf{X}(\mathbf{a}^\mathsf{T}\mathbf{X} + W)] = \mathbb{E}[\mathbf{X}\mathbf{X}^\mathsf{T}]\mathbf{a} = \mathbf{C}_X\mathbf{a}$ .

Observation variance

$\text{Var}(Y) = \mathbf{a}^\mathsf{T}\mathbf{C}_X\mathbf{a} + \sigma^2$ .

LMMSE = MMSE

$\hat{\mathbf{X}} = \frac{\mathbf{C}_X\mathbf{a}}{\mathbf{a}^\mathsf{T}\mathbf{C}_X\mathbf{a} + \sigma^2}Y$ .

MSE matrix: $\mathbf{C}_{\tilde{X}} = \mathbf{C}_X - \frac{\mathbf{C}_X\mathbf{a}\mathbf{a}^\mathsf{T}\mathbf{C}_X}{\mathbf{a}^\mathsf{T}\mathbf{C}_X\mathbf{a} + \sigma^2}$ . This is a rank-one update to the prior covariance. $\blacksquare$

ex-ch12-15

Challenge

(Nonlinear MMSE vs. LMMSE gap) Let $X \sim \text{Uniform}[-1, 1]$ and $Y = X^2 + Z$ where $Z \sim \mathcal{N}(0, 0.1)$ independent of $X$ . (a) Show that $\mathbb{E}[X|Y]$ depends on $Y$ nonlinearly (find it numerically if needed). (b) Compute the LMMSE estimate. What is $\text{Cov}(X, Y)$ ? (c) Compare the MSE of both estimators via Monte Carlo simulation.

Show Hint

Note: $\text{Cov}(X, Y) = \text{Cov}(X, X^2) = \mathbb{E}[X^3] - \mathbb{E}[X]\mathbb{E}[X^2]$ .

For $X \sim \text{Uniform}[-1,1]$ , $\mathbb{E}[X^3] = 0$ by symmetry.

Solution

Part (b): LMMSE

$\mathbb{E}[X] = 0$ , $\mathbb{E}[X^2] = 1/3$ , $\mathbb{E}[X^3] = 0$ . So $\text{Cov}(X, Y) = \text{Cov}(X, X^2) = 0$ . Therefore $\hat{X}_{\text{LMMSE}} = 0$ for all $Y$ ! The LMMSE estimate ignores $Y$ entirely because $X$ and $Y = X^2 + Z$ are uncorrelated (despite being dependent).

Parts (a) and (c)

The MMSE estimator $\mathbb{E}[X|Y=y]$ uses the posterior $f(x|y) \propto f_{Z}(y - x^2) \cdot \mathbf{1}_{[-1,1]}(x)$ . This is a bimodal distribution (symmetric around 0 when $y > 0$ ), so $\mathbb{E}[X|Y] = 0$ by symmetry as well. Both estimators give the same result here — not because LMMSE is good, but because the MMSE itself cannot distinguish $+x$ from $-x$ given $y = x^2 + z$ . The MMSE error is $\mathbb{E}[X^2] = 1/3$ , same as the LMMSE error.

Exercises

ex-ch12-01

Find $\mathbb{E}[X|Y]$

Tower check

ex-ch12-02

Apply the law of total variance

ex-ch12-03

Apply independence

ex-ch12-04

Part (a)

Part (b)

Part (c)

ex-ch12-05

Part (a): MMSE

Part (b): LMMSE

Part (c): Compare

ex-ch12-06

Apply iterated conditioning

ex-ch12-07

Conditional moments

Apply total variance

ex-ch12-08

Set up

Invert $\mathbf{C}_{YY}$

Compute the estimator

ex-ch12-09

Decompose

Cross-term vanishes

Conclude

ex-ch12-10

Compute covariances

LMMSE estimator

MSE matrix

ex-ch12-11

Supporting line

Condition and apply

ex-ch12-12

Part (a)

Part (b)

Part (c)

ex-ch12-13

Part (a)

Part (b)

ex-ch12-14

Cross-covariance

Observation variance

LMMSE = MMSE

ex-ch12-15

Part (b): LMMSE

Parts (a) and (c)