Exercises

Classification accuracy $u = P(\\hat{S} = S) = 0.95$ means error probability $\\epsilon = 0.05$. The distortion is Hamming: $d(s, \\hat{s}) = \\mathbb{1}[s \\neq \\hat{s}]$ with $\\mathbb{E}[d] \\leq 0.05$.

The binary rate-distortion function is \R(D) = \H_{b}(p) - \H_{b}(D) for $D \\leq \\min(p, 1-p)$. With $p = 0.3$ and $D = 0.05$: \R(0.05) = \H_{b}(0.3) - \H_{b}(0.05) = 0.881 - 0.286 = 0.595 bits.

$\\text{SNR} = 100$ (20 dB). $D_{\\text{JSCC}} = 1/(1 + 100) = 1/101 \\approx 0.0099$.

$\C = \\frac{1}{2}\\log_2(101) \\approx 3.33$ bits. $D_{\\text{sep}} = 2^{-2 \\times 3.33} = 2^{-6.66} \\approx 0.0099$. They match, confirming optimality.

A digital system uses a fixed-rate channel code designed for $\\text{SNR}_{\\text{design}}$. The code rate is $\R = \C(\\text{SNR}_{\\text{design}})$. When the actual $\\text{SNR} < \\text{SNR}_{\\text{design}}$, $\C < \R$, and the channel code fails completely (BER $\\approx 0.5$). The reconstruction goes from near-perfect to garbage.

An analog system transmits a scaled version of the source. When SNR drops, the noise increases but the received signal still contains information about the source. The MMSE estimator adapts to the noise level, producing a noisier but still useful reconstruction. Example: at $\\text{SNR} = 20$ dB, MSE = 0.01. At $\\text{SNR} = 0$ dB, MSE = 0.5 (still better than pure noise, MSE = 1.0).

At high rate (low distortion), $\R_{\\text{MSE}} \\approx \\frac{d}{2}\\log(\\sigma^2/D_d)$ and $\R_{\\text{sem}} \\approx \\frac{m}{2}\\log(\\sigma^2/D_m)$. For the same per-component distortion, the ratio is $\R_{\\text{sem}}/\R_{\\text{MSE}} \\approx m/d$. With $m = 10$, $d = 1000$: semantic communication needs $\\approx 1\\%$ of the rate.

Transmit $X_i = \\sqrt{P/\\sigma_S^2} S$ for $i = 1, \\ldots, k$ (repeat $k = \\rho d$ times). The receiver averages: $\\hat{S} = \\frac{1}{k}\\sum Y_i$, giving MSE $= \\sigma_S^2/(1 + k\\text{SNR}/d) = \\sigma_S^2/(1+\\rho\\text{SNR})$.

The separation bound gives $D^* = \\sigma_S^2 \\cdot 2^{-2\\rho \C}$ where $\C = \\frac{1}{2}\\log(1+\\text{SNR})$. So D^* = \\sigma_S^2/(1+\\text{SNR})^\\rho. For $\\rho > 1$, (1+\\text{SNR})^\\rho > 1+\\rho\\text{SNR}, so separate coding strictly outperforms repetition. Repetition is optimal only for $\\rho = 1$ (matched bandwidth).

Let A = U_A \\Sigma_A V_A^\\top (SVD). The goal $G$ depends on $S$ only through V_A^\\top S \\in \\mathbb{R}^m. Define \\tilde{S} = V_A^\\top S \\sim \\mathcal{N}(0, V_A^\\top \\Sigma_S V_A).

The rate-utility function satisfies $R_U(D) \\leq \R_{\\tilde{S}}(D/\\|\\Sigma_A\\|^2)$ (distortion of $G$ is amplified by $\\Sigma_A$). For the Gaussian case: $R_U(D) \\leq \\sum_{i=1}^m \\frac{1}{2}\\log^+\\frac{\\tilde{\\lambda}_i \\sigma_{A,i}^2}{D}$ where $\\tilde{\\lambda}_i$ are eigenvalues of V_A^\\top \\Sigma_S V_A and $\\sigma_{A,i}$ are singular values of $A$.

Bounding each term: $R_U(D) \\leq \\frac{m}{2}\\log^+(\\lambda_{\\max} \\|A\\|^2 / D)$. This shows the rate scales with the task dimension $m$, not the source dimension $d$.

At 10 dB ($\\text{SNR} = 10$): $D_{\\text{opt}} = 1/11 \\approx 0.091$, PSNR $\\approx 10.4$ dB. At 5 dB ($\\text{SNR} = 3.16$): $D_{\\text{opt}} = 1/4.16 \\approx 0.24$, PSNR $\\approx 6.2$ dB.

A system trained at 10 dB has learned a representation optimized for $\\text{SNR} = 10$. At $\\text{SNR} = 5$ dB, the noise is higher than expected. The encoder's implicit power allocation may not be optimal, and the decoder's learned denoiser under-compensates. Empirically, the PSNR degradation is typically 0.5-2 dB compared to a matched system, depending on the architecture and training strategy.

Train with SNR drawn from a range (e.g., $[0, 20]$ dB) and condition the decoder on the estimated SNR. This "SNR-adaptive" DeepJSCC loses $<0.5$ dB vs. per-SNR training.

For $S \\sim \\mathcal{N}(0, \\sigma^2)$ observed through $Y = S + N$ with $N \\sim \\mathcal{N}(0, \\sigma^2)$: $\\hat{S} = \\frac{\\sigma^2}{\\sigma^2 + \\sigma^2} Y \\sim \\mathcal{N}\\!\\left(0, \\frac{\\sigma^4}{\\sigma^2 + \\sigma^2}\\right)$.

$\\text{Var}(\\hat{S}) = \\frac{\\sigma^4}{\\sigma^2 + \\sigma^2} < \\sigma^2 = \\text{Var}(S)$ for any $\\sigma^2 > 0$. Therefore $P_{\\hat{S}} = \\mathcal{N}(0, \\sigma^2 - D) \\neq P_S = \\mathcal{N}(0, \\sigma^2)$. The MMSE reconstruction is "narrower" than the source — it produces less variability.

To achieve $P_{\\hat{S}} = P_S$ (perfect perception), we must add noise to $\\hat{S}$ to restore the variance: $\\hat{S}_{\\text{perc}} = \\hat{S} + \\xi$ where $\\xi \\sim \\mathcal{N}(0, D)$. But this increases MSE from $D$ to $2D$. This is the perception-distortion tradeoff.

The encoder is $X = A S$ where $A \\in \\mathbb{R}^{k \\times d}$ with $k = \\rho d$ and power constraint \\text{tr}(A \\Sigma_S A^\\top) \\leq kP. The decoder is the LMMSE estimator.

MSE = \\text{tr}(\\Sigma_S) - \\text{tr}(\\Sigma_S A^\\top (A\\Sigma_S A^\\top + \\sigma^2 I_k)^{-1} A \\Sigma_S). For i.i.d. source ($\\Sigma_S = \\sigma_S^2 I_d$) and $A = \\alpha I_{k \\times d}$ (projection): MSE $= (d-k)\\sigma_S^2 + k \\frac{\\sigma_S^2 \\sigma^2}{\\alpha^2 \\sigma_S^2 + \\sigma^2}$.

For $\\rho \\leq 1$ ($k \\leq d$): MSE $= d\\sigma_S^2 - k\\frac{\\alpha^2 \\sigma_S^4}{\\alpha^2 \\sigma_S^2 + \\sigma^2}$. With power constraint $k\\alpha^2 \\sigma_S^2 = kP$, i.e., $\\alpha = \\sqrt{P/\\sigma_S^2}$: MSE $= d\\sigma_S^2 \\left(1 - \\frac{\\rho P}{P + \\sigma^2}\\right) = d\\sigma_S^2 \\left(\\frac{\\sigma^2 + (1-\\rho)P}{P + \\sigma^2}\\right)$. This is minimized at $\\rho^* = 1$, giving MSE $= d\\sigma_S^2/(1+\\text{SNR})$.

The classification task requires $\\log_2 C$ bits of information. The encoder maps the source to $k = \\rho d$ channel symbols. The channel can carry $k \C$ bits reliably, where $\C = \\frac{1}{2}\\log_2(1+\\text{SNR})$.

For reliable transmission: $\\rho d \C \\geq \\log_2 C$. Therefore $\\rho_{\\min} = \\frac{\\log_2 C}{d \C}$. For $C = 1000$, $d = 3072$ (CIFAR), $\\text{SNR} = 10$ dB: $\\rho_{\\min} = \\frac{10}{3072 \\times 1.73} \\approx 0.0019$.

Reconstruction at PSNR 30 dB requires $\\rho \\approx 0.2$-$0.5$ (depending on the source). The semantic $\\rho_{\\min} = 0.002$ is $100\\times$ smaller — this is the semantic communication gain. At error probability $\\epsilon$, add $\\sqrt{V/(\\rho d)} Q^{-1}(\\epsilon)$ for the finite-blocklength correction.

A digital scheme at rate $\R$ succeeds when $\\frac{1}{2}\\log(1+|h|^2 \\text{SNR}) \\geq \R$, i.e., $|h|^2 \\geq (2^{2\R}-1)/\\text{SNR}$. In outage (probability $p_{\\text{out}}$), distortion $= \\sigma_S^2$. In success, distortion $= \\sigma_S^2 \\cdot 2^{-2\R}$. Expected distortion: $D_{\\text{dig}} = p_{\\text{out}} \\sigma_S^2 + (1-p_{\\text{out}}) \\sigma_S^2 2^{-2\R}$.

The analog scheme transmits $X = \\sqrt{P/\\sigma_S^2} S$, receiving $Y = h\\sqrt{P/\\sigma_S^2} S + Z$. MMSE decoding gives distortion $D_{\\text{analog}}(h) = \\sigma_S^2/(1+|h|^2 \\text{SNR})$. Expected: $D_{\\text{analog}} = \\sigma_S^2 \\mathbb{E}[1/(1+|h|^2 \\text{SNR})]$.

For Rayleigh fading, $D_{\\text{analog}} = \\sigma_S^2 \\cdot \\frac{e^{1/\\text{SNR}}}{\\text{SNR}} E_1(1/\\text{SNR})$ where $E_1$ is the exponential integral. The digital distortion has a discontinuity (outage) while the analog distortion is smooth. For any fixed rate $\R$, there exists a range of SNR where $D_{\\text{analog}} < D_{\\text{dig}}$, because the analog scheme avoids the catastrophic outage penalty.

$\\mu_1 = \\mu_2 = 0$, so $\\|\\mu_1 - \\mu_2\\|^2 = 0$. $\\Sigma_1 = \\sigma^2 I_d$, $\\Sigma_2 = (\\sigma^2 - D) I_d$. $(\\Sigma_1 \\Sigma_2)^{1/2} = \\sqrt{\\sigma^2(\\sigma^2 - D)} I_d$. $\\text{FID} = d(\\sigma^2 + \\sigma^2 - D - 2\\sqrt{\\sigma^2(\\sigma^2 - D)}) = d(\\sigma - \\sqrt{\\sigma^2 - D})^2$.

$\\sigma^2 - D = \\sigma^2 \\text{SNR}/(1+\\text{SNR})$. $\\text{FID} = d\\sigma^2(1 - \\sqrt{\\text{SNR}/(1+\\text{SNR})})^2$. At high SNR: $\\text{FID} \\approx d\\sigma^2/(4\\text{SNR}^{2})$ — it decreases as $1/\\text{SNR}^{2}$. The FID quantifies the perception gap: even the optimal MMSE estimator produces reconstructions with lower variance than the source.

The base layer encodes $S \\in \\mathbb{R}^d$ at rate $R_b$, achieving MSE $D_{\\text{MSE}}(R_b)$ by reverse water-filling. With i.i.d. source: $D_{\\text{MSE}}(R_b) = d \\sigma^2 2^{-2R_b/d}$.

The semantic layer encodes the residual of the $m$ task-relevant components at rate $R_s$. The residual variance per component is $\\sigma^2 2^{-2R_b/d}$ (what the base layer did not capture). After semantic refinement: $D_{\\text{task}}(R_b, R_s) = m \\sigma^2 2^{-2R_b/d} \\cdot 2^{-2R_s/m}$.

Minimize $\\alpha d \\sigma^2 2^{-2R_b/d} + (1-\\alpha) m \\sigma^2 2^{-2(R_b/d + R_s/m)}$ subject to $R_b + R_s = R$. Setting the derivative to zero gives the optimal allocation. For $\\alpha \\to 0$ (task-focused): allocate most rate to the $m$ task dimensions, $R_s/m \\gg R_b/d$. For $\\alpha \\to 1$ (MSE-focused): $R_b = R$, $R_s = 0$.

$R_U(u)$ is the minimum rate for utility $\\geq u$. If $u_1 > u_2$, the constraint set for $u_1$ is a subset of the constraint set for $u_2$, so $R_U(u_1) \\geq R_U(u_2)$. Therefore $R_U$ is non-increasing. Wait — actually, higher utility requires more rate (more information about $G$), so $R_U$ should be non-decreasing. Let me reconsider. Since we maximize utility, higher $u$ requires more rate: $R_U(u_1) \\geq R_U(u_2)$ for $u_1 > u_2$. So $R_U$ is non-decreasing (like $\R$ with $D$ replaced by $-u$).

Let $P_1, P_2$ be optimal mappings achieving $(R_1, u_1)$ and $(R_2, u_2)$. Time-sharing: use $P_1$ with probability $\\lambda$ and $P_2$ with probability $1-\\lambda$. Rate: $\\leq \\lambda R_1 + (1-\\lambda) R_2$ (MI is convex in $P_{\\hat{S}|S}$ for mixture). Utility: $= \\lambda u_1 + (1-\\lambda) u_2$. Therefore $R_U(\\lambda u_1 + (1-\\lambda) u_2) \\leq \\lambda R_U(u_1) + (1-\\lambda) R_U(u_2)$.

At rate $R$, the utility region is: \\mathcal{U}(R) = \\{(u_1, \\ldots, u_L) : \\exists P_{\\hat{S}|S}, \I(S;\\hat{S}) \\leq R, \\mathbb{E}[U_\\ell(\\hat{S}, G_\\ell)] \\geq u_\\ell \\forall \\ell\\}.

If $(u_1^{(a)}, \\ldots, u_L^{(a)}) \\in \\mathcal{U}(R)$ and $(u_1^{(b)}, \\ldots, u_L^{(b)}) \\in \\mathcal{U}(R)$, the convex combination is achieved by time-sharing the two encoding schemes. Rate: $\\lambda R + (1-\\lambda)R = R$. Utility: \\lambda u_\\ell^{(a)} + (1-\\lambda)u_\\ell^{(b)}. So $\\mathcal{U}(R)$ is convex.

A shared encoder must operate on the Pareto boundary of $\\mathcal{U}(R)$. With $L$ tasks, the optimal representation is a compromise — it preserves information about all goals $G_1, \\ldots, G_L$. As $L \\to \\infty$ (many diverse tasks), the optimal encoder approaches the Shannon encoder that preserves all information in $S$, recovering the classical separation architecture. This is the formal version of the universality-efficiency tradeoff.