Exercises

$\text{SNR} = P/N = 10/1 = 10$ (or 10 dB).

$C = \frac{1}{2}\log_2(1 + 10) = \frac{1}{2}\log_2(11) \approx 1.73$ bits per real channel use.

$\text{SNR} = 10^{0.3} \approx 2.0$.

$\eta = \log_2(1 + 2) = \log_2(3) \approx 1.585$ bits/s/Hz.

$h(Y|X) = h(Z) = \frac{1}{2}\log(2\pi e N)$.

Since $X \sim \mathcal{N}(0, P)$ and $Z \sim \mathcal{N}(0, N)$ are independent, $Y \sim \mathcal{N}(0, P+N)$, so $h(Y) = \frac{1}{2}\log(2\pi e(P+N))$.

$I(X;Y) = \frac{1}{2}\log(2\pi e(P+N)) - \frac{1}{2}\log(2\pi e N) = \frac{1}{2}\log\!\left(\frac{P+N}{N}\right) = \frac{1}{2}\log(1 + P/N)$.

$\text{SNR} = 2^2 - 1 = 3$.

$E_b/N_0 = \text{SNR}/\eta = 3/2 = 1.5$. In dB: $10\log_{10}(1.5) \approx 1.76$ dB.

This is $1.76 - (-1.59) = 3.35$ dB above the Shannon limit.

Since $h(Y|X) = h(Z)$ is fixed, $I(X;Y) = h(Y) - h(Z)$. Maximizing $I$ is equivalent to maximizing $h(Y)$.

$\text{Var}(Y) = \text{Var}(X) + N \leq P + N$. Among all distributions with variance at most $P + N$, the Gaussian $Y \sim \mathcal{N}(0, P+N)$ maximizes $h(Y)$.

$Y = X + Z$ is Gaussian if and only if $X$ is Gaussian (since $Z$ is Gaussian and independent of $X$). Therefore $X \sim \mathcal{N}(0, P)$ is the unique maximizer.

$N_0/|G_k|^2$: $0.125$, $0.25$, $1.0$, $2.0$.

$4\nu - (0.125 + 0.25 + 1.0 + 2.0) = 3$, $\nu = 1.594$. But ${E_s}_{4} = 1.594 - 2 < 0$.

$3\nu - (0.125 + 0.25 + 1.0) = 3$, $\nu = 1.458$.

${E_s}_{1} = 1.333$, ${E_s}_{2} = 1.208$, ${E_s}_{3} = 0.458$, ${E_s}_{4} = 0$. All non-negative. $\checkmark$

$C = \log_2(1 + 8 \times 1.333) + \log_2(1 + 4 \times 1.208) + \log_2(1 + 1 \times 0.458)KATEXPLACEHOLDER0END= \log_2(11.667) + \log_2(5.833) + \log_2(1.458)KATEXPLACEHOLDER1END\approx 3.544 + 2.544 + 0.544 = 6.633 \text{ bits.}$$

When $E_s$ is large, $\nu \gg \max_k N_0/|G_k|^2$, so all channels are active with ${E_s}_{k} \approx E_s/K$.

Water-filling: $C_{\text{WF}} = \sum_k \log(1 + |G_k|^2 {E_s}_{k}^*/N_0)$.

Equal power: $C_{\text{EP}} = \sum_k \log(1 + |G_k|^2 E_s/(KN_0))$.

The difference $C_{\text{WF}} - C_{\text{EP}}$ is bounded by a constant (depending on the channel gains) while both capacities grow as $K\log(E_s)$. Therefore the ratio $C_{\text{EP}}/C_{\text{WF}} \to 1$.

$N = N_0W = 10^{-9} \times 10^6 = 10^{-3}$ W = 1 mW.

At $P = 0.1$ mW: $\text{SNR} = 0.1$, $C = 10^6 \log_2(1.1) \approx 0.137$ Mbit/s.

At $P = 100$ mW: $\text{SNR} = 100$, $C = 10^6 \log_2(101) \approx 6.66$ Mbit/s.

We need $\log_2(1 + 2P/N)/\log_2(1+P/N) = 1.1$. Numerically, this occurs at $P/N \approx 50$ ($P \approx 50$ mW), where $C(P) \approx 5.67$ Mbit/s and $C(2P) \approx 6.34$ Mbit/s. Beyond this point, capacity is firmly in the logarithmic regime and power becomes an inefficient resource.

Codewords satisfy $\|\mathbf{x}(m)\|^2 \leq nP$. The noise $Z^{n}$ concentrates on a sphere of radius $\approx \sqrt{nN}$. The received $\mathbf{y} = \mathbf{x}(m) + Z^{n}$ lies in a sphere of radius $\sqrt{n(P+N)}$.

The volume of a sphere of radius $r$ in $\mathbb{R}^n$ is $V_n r^n$, where $V_n$ is a constant depending only on $n$.

$$M \leq \frac{V_n (\sqrt{n(P+N)})^n}{V_n (\sqrt{nN})^n} = \left(\frac{P+N}{N}\right)^{n/2}.$$

$R = \frac{1}{n}\log_2 M \leq \frac{1}{2}\log_2(1 + P/N) = C$.

$I(X; Y) = h(Y) - h(Y|X) = h(Y) - h(Z)$. $h(Z) = \log(\pi e N_0)$ for $Z \sim \mathcal{CN}(0, N_0)$.

$\text{Var}(Y) = \mathbb{E}[|X|^2] + N_0 \leq E_s + N_0$. By the complex Gaussian entropy maximizer: $h(Y) \leq \log(\pi e(E_s + N_0))$, achieved when $X \sim \mathcal{CN}(0, E_s)$.

$C = \log(\pi e(E_s+N_0)) - \log(\pi e N_0) = \log\!\left(1 + \frac{E_s}{N_0}\right).$$

If channels $1, \ldots, j$ are active: $\sum_{k=1}^j (\nu - N_0/|G_k|^2) = E_s$, so $\nu_{j} = \frac{1}{j}(E_s + \sum_{k=1}^j N_0/|G_k|^2)$.

Channel $j$ is active iff ${E_s}_{j} = \nu_{j} - N_0/|G_j|^2 > 0$, i.e., $N_0/|G_j|^2 < \nu_{j}$.

$\nu_{j+1} - \nu_{j} = \frac{1}{j+1}\left(N_0/|G_{j+1}|^2 - \nu_{j}\right)$. If channel $j+1$ is active, then $N_0/|G_{j+1}|^2 < \nu_{j}$, so $\nu_{j+1} < \nu_{j}$. The water level decreases as we add more channels. The optimal $K^*$ is the largest $j$ such that $N_0/|G_j|^2 < \nu_{j}$.

$G_k = 1 + 0.5 e^{-j2\pi k/64} + 0.3 e^{-j2\pi \cdot 2k/64}$ for $k = 0, \ldots, 63$. The gains $|G_k|^2$ vary across subcarriers; $|G_0|^2 = |1+0.5+0.3|^2 = 3.24$.

Solve for $\nu$ such that $\sum_{k=0}^{63} [\nu - 0.1/|G_k|^2]_+ = 5$. This requires numerical computation (bisection on $\nu$).

$C = \frac{64}{64+2} \cdot \frac{1}{64}\sum_{k=0}^{63} \left[\log_2\!\left(\frac{\nu|G_k|^2}{0.1}\right)\right]_+.$$The factor$64/66 \approx 0.970$accounts for the CP overhead. Numerical evaluation gives$C \approx 3.1$ bits per channel use.

Let $\alpha = P/N_0$. $C(W) = W\log_2(1 + \alpha/W)$.

$C'(W) = \log_2(1 + \alpha/W) - \frac{\alpha/W}{(1+\alpha/W)\ln 2}$.

$C''(W) = -\frac{\alpha^2}{W^{2}(1+\alpha/W)^2 \cdot W\ln 2} < 0$.

Since $C'' < 0$, $C(W)$ is strictly concave.

As $W \to 0$: $C \approx W \cdot \frac{\alpha/W}{\ln 2} = \frac{P}{N_0\ln 2}$ (constant, the power-limited ceiling).

Wait — more carefully: $C \to 0$ as $W \to 0$ (no bandwidth, no transmission). The slope $C'(0^+) = P/(N_0\ln 2)$ bits/s per Hz.

As $W \to \infty$: $C \to P/(N_0\ln 2)$ bits/s (finite).

$2^{2h(Y)} = 2^{2h(X+Z)} \geq 2^{2h(X)} + 2^{2h(Z)}$.

$2^{2h(X)} \leq 2^{\log(2\pi e P)} = 2\pi e P$ and $2^{2h(Z)} = 2\pi e N$.

$2^{2h(Y)} \geq 2\pi e P + 2\pi e N = 2\pi e(P+N)$.

Wait — this is a lower bound, not an upper bound! The EPI gives $h(Y) \geq \frac{1}{2}\log(2\pi e(P+N))$. This is actually the opposite direction.

The correct converse uses the maximum entropy property directly: $h(Y) \leq \frac{1}{2}\log(2\pi e \text{Var}(Y)) \leq \frac{1}{2}\log(2\pi e(P+N))$. The EPI provides an alternative proof that Gaussian input is optimal (achievability), not the converse.

$R(D) = \min_{\{D_k\}} \sum_k \frac{1}{2}\log\frac{\sigma_k^2}{D_k}$ subject to $\sum_k D_k \leq D$, $0 < D_k \leq \sigma_k^2$.

$\frac{\partial}{\partial D_k}\left[\sum_k \frac{1}{2}\log\frac{\sigma_k^2}{D_k}\right] = -\frac{1}{2D_k \ln 2}$.

KKT: $\frac{1}{2D_k\ln 2} = \lambda$ if $D_k < \sigma_k^2$ (active). So $D_k = \gamma = 1/(2\lambda\ln 2)$ for active sources. If $\gamma \geq \sigma_k^2$, then $D_k = \sigma_k^2$ (zero rate).

Channel coding: allocate MORE resources (power) to BETTER channels. Source coding: allocate MORE distortion to WEAKER sources (smaller variance).

Both are convex optimizations with the same mathematical structure. The water level $\nu$ (channel) and $\gamma$ (source) play dual roles. This duality extends to the separation theorem: the total system is optimal when source and channel coding are separately optimized.

Generate the channel taps and compute the $K$-point DFT to get $\{|G_k|^2\}_{k=0}^{255}$. The frequency response will show frequency-selective fading with approximately 6 dB variation.

At moderate SNR, some sub-carriers may be in deep fades. Water-filling will shut off the deeply faded sub-carriers and concentrate power on the good ones. The capacity gain of WF over EP is typically 5-15% at this SNR.

At high SNR, all sub-carriers have positive SNR even in fades. Water-filling still outperforms equal power but the gap shrinks to about 1-3%, confirming the theoretical result that equal power is asymptotically optimal.

$\text{SNR} = 10^3 = 1000$. $C = 200 \times 10^3 \times \log_2(1001) \approx 200 \times 10^3 \times 9.97 \approx 1.99$ Mbit/s.

GSM achieves 270.833 kbit/s gross (before error correction) in 200 kHz, i.e., $\eta \approx 1.35$ bits/s/Hz. The Shannon limit is $\eta_{\max} = 9.97$ bits/s/Hz. GSM operates at only 14% of the Shannon limit — reflecting its 1980s technology (no turbo/LDPC, no adaptive modulation).

For jointly Gaussian: $\hat{X}(Y) = \frac{\text{Cov}(X,Y)}{\text{Var}(Y)}Y = \frac{P}{P+N}Y$.

$\text{MMSE} = P - \frac{P^2}{P+N} = \frac{PN}{P+N}$.

$X|Y \sim \mathcal{N}(\frac{P}{P+N}Y, \frac{PN}{P+N})$.

$h(X|Y) = \frac{1}{2}\log\!\left(2\pi e \frac{PN}{P+N}\right)$.

$\frac{1}{2\pi e}2^{2h(X|Y)} = \frac{1}{2\pi e} \cdot 2\pi e \frac{PN}{P+N} = \frac{PN}{P+N} = \text{MMSE}$. $\checkmark$