Exercises

The source $S$ is transmissible losslessly over the DMC if and only if $H(S) \leq C$. Separation of source and channel coding is optimal: compress to $H(S)$ bits, then channel-code at rate $\leq C$.

For general $\kappa$, the condition becomes $H(S) \leq \kappa C$. The bandwidth ratio provides $\kappa$ channel uses per source symbol, so the effective capacity is $\kappa C$ bits per source symbol.

$H(S) = h(0.3) = -0.3\log_2 0.3 - 0.7\log_2 0.7 \approx 0.881$ bits.

$C = 1 - h(0.1) = 1 - (-0.1\log_2 0.1 - 0.9\log_2 0.9) \approx 1 - 0.469 = 0.531$ bits/use.

Since $H(S) = 0.881 > 0.531 = C$, reliable lossless transmission is not possible at $\kappa = 1$. We would need $\kappa \geq H(S)/C \approx 1.66$ channel uses per source symbol.

Since the sources are independent, the Slepian–Wolf region requires $R_1 \geq H(S_1) = 0.8$ and $R_2 \geq H(S_2) = 0.6$, with $R_1 + R_2 \geq 1.4$.

The point $(0.8, 0.6)$ satisfies $R_1 = 0.8 \leq 1$, $R_2 = 0.6 \leq 1$, and $R_1 + R_2 = 1.4 \leq 1.5$. Therefore $(0.8, 0.6)$ is inside the MAC capacity region, and separate coding is sufficient.

Setting $R(D) = \kappa C$: $$\frac{1}{2}\log(1/D) = \frac{1}{2}\log(1 + 10) = \frac{1}{2}\log(11)$$ $$D_{\text{opt}} = \frac{1}{11} \approx 0.0909.$$

$D_{\text{uncoded}} = \frac{\sigma_S^2 \sigma^2}{P + \sigma^2} = \frac{1}{1 + \text{SNR}} = \frac{1}{11} \approx 0.0909.$

Both achieve $D = 1/11$. At $\kappa = 1$ for Gaussian source/channel, uncoded transmission is optimal.

Shannon's separation theorem relies on the fact that in the point-to-point setting, the source and channel are connected by a single rate constraint, and optimizing each side independently achieves the global optimum. In multi-terminal networks, the source correlation provides a form of implicit cooperation between distributed encoders that separate coding discards. When each encoder compresses its source independently (even using distributed compression like Slepian–Wolf), the compressed bitstreams lose the fine-grained correlation structure that a joint scheme could exploit through the channel.

Example: Two correlated binary sources over a binary adder MAC. The Slepian–Wolf sum rate exceeds the MAC sum capacity, so separation fails. But transmitting raw source bits directly through the MAC can succeed because the receiver effectively observes a function of the sources that, combined with knowledge of their correlation, suffices for reconstruction.

Since $P_e^{(k)} \to 0$, Fano's inequality gives $$H(S^k | Y^n) \leq 1 + P_e^{(k)} k \log|\mathcal{S}| \triangleq k\epsilon_k$$ where $\epsilon_k \to 0$.

$kH(S) = H(S^k) = I(S^k; Y^n) + H(S^k | Y^n) \leq I(S^k; Y^n) + k\epsilon_k.$$

Since the encoder maps $S^k \to X^n$ deterministically, the Markov chain $S^k \to X^n \to Y^n$ holds. By data processing: $$I(S^k; Y^n) \leq I(X^n; Y^n).$$

For the memoryless channel: $$I(X^n; Y^n) = \sum_{i=1}^n I(X_i; Y_i | Y^{i-1}) \leq \sum_{i=1}^n I(X_i; Y_i) \leq nC$$ where the first inequality uses the memoryless property and the second uses the definition of capacity.

Combining: $kH(S) \leq nC + k\epsilon_k$. Dividing by $k$ and taking $k \to \infty$: $H(S) \leq (n/k)C = \kappaC$. $\blacksquare$

The marginals are uniform: $H(S_1) = H(S_2) = 1$ bit.

$S_2$ given $S_1$ is a BSC($p$), so $H(S_1|S_2) = h(p)$.

$H(S_1, S_2) = H(S_2) + H(S_1|S_2) = 1 + h(p)$.

The Slepian–Wolf sum rate is $H(S_1, S_2) = 1 + h(p)$.

The Gaussian MAC sum capacity at $\text{SNR}_{1} = \text{SNR}_{2} = 10^{0.5} \approx 3.16$: $$C_{\text{sum}} = \frac{1}{2}\log_2(1 + 2 \times 3.16) = \frac{1}{2}\log_2(7.32) \approx 1.44 \text{ bits/use}.$$

Separation requires $1 + h(p) \leq 1.44$, i.e., $h(p) \leq 0.44$, which gives $p \leq 0.073$ approximately. For $p > 0.073$, the Slepian–Wolf sum rate exceeds the MAC sum capacity, and the sufficient condition for separation fails.

$D_{\text{opt}} = \sigma_S^2 \cdot 2^{-2\kappaC} = \sigma_S^2 \cdot (1 + \text{SNR})^{-2} = \frac{\sigma_S^2}{(1 + \text{SNR})^2}.$$

Each channel use transmits $X_i = \sqrt{P/(2\sigma_S^2)} S$ (power $P/2$ each, total power $P$ over 2 uses). The receiver uses MRC on the two observations: $Y_1 = \sqrt{P/(2\sigma_S^2)} S + Z_1$ and $Y_2 = \sqrt{P/(2\sigma_S^2)} S + Z_2$.

The effective SNR after MRC is $2 \times P/(2\sigma^2) = P/\sigma^2 = \text{SNR}$. The MMSE distortion is $$D_{\text{rep}} = \frac{\sigma_S^2}{1 + \text{SNR}}.$$

$\frac{D_{\text{rep}}}{D_{\text{opt}}} = \frac{1 + \text{SNR}}{1} = 1 + \text{SNR}.$$For$\text{SNR} = 10$(10 dB),$D_{\text{rep}} / D_{\text{opt}} = 11$. Separate coding achieves 11x lower distortion. The uncoded repetition scheme wastes the extra bandwidth by simply repeating the same sample, while separate coding uses the extra channel uses to transmit additional coded bits.

Let the encoders map $S_k^n \to X_k^n$ and the decoder produce $(\hat{S}_1^n, \hat{S}_2^n)$ from $Y^n$ with $P_e^{(n)} \to 0$.

By Fano's inequality: $H(S_k^n | Y^n) \leq n\epsilon_n$ for each $k$.

For the sum rate: $$n(H(S_1) + H(S_2)) = H(S_1^n, S_2^n) = I(S_1^n, S_2^n; Y^n) + H(S_1^n, S_2^n | Y^n)$$ $$\leq I(X_1^n, X_2^n; Y^n) + 2n\epsilon_n \leq nC_{\text{sum}} + 2n\epsilon_n$$ where $C_{\text{sum}}$ is the MAC sum capacity.

Similarly, $nH(S_k) \leq I(X_k^n; Y^n | X_{k'}^n) + n\epsilon_n \leq n \cdot \max_{P_{X_k}} I(X_k; Y | X_{k'}) + n\epsilon_n$.

These individual and sum bounds exactly match the Slepian–Wolf rates for independent sources (which equal the marginal entropies) needing to fit inside the MAC capacity region. Since the sources are independent, there is no correlation that joint coding could exploit beyond what separate Slepian–Wolf compression and MAC coding achieve. $\blacksquare$

Hybrid digital–analog coding splits the source into two components: a digital part (compressed and channel-coded) and an analog part (transmitted uncoded via a linear mapping). The channel input is $X^n = \alpha \phi(S^n) + \beta c(M)$ where $\phi$ is the analog mapping, $c(M)$ is the channel codeword, and $(\alpha, \beta)$ control the power split.

Broadcasting a Gaussian source to two receivers:

• Receiver 1 has $\text{SNR}_{1} = 0$ dB (weak channel)
• Receiver 2 has $\text{SNR}_{2} = 20$ dB (strong channel)

Pure digital (separate): Must choose a single operating point. If designed for receiver 2 (high rate), receiver 1 cannot decode. If designed for receiver 1 (low rate), receiver 2's potential is wasted.

Pure analog (uncoded): $D_k = \sigma_S^2/(1 + \text{SNR}_{k})$. This gracefully adapts to each receiver's SNR, but is suboptimal for either.

Hybrid coding: Transmit a base digital layer decodable by receiver 1 (achieving $D_1$ close to the rate-distortion bound at $\text{SNR}_{1}$), plus an analog refinement that receiver 2 can exploit to achieve much lower distortion. The analog component provides graceful degradation for intermediate receivers. Hybrid achieves near-optimal distortion for both receivers simultaneously — something neither pure approach can do.

$C = \frac{1}{2}\log_2(1 + 100) = \frac{1}{2}\log_2(101) \approx 3.33$ bits/use.

Setting $R(D) = \kappaC$: $$\frac{1}{2}\log_2(4/D) = 0.5 \times 3.33 = 1.665$$ $$4/D = 2^{3.33} \approx 10.05$$ $$D_{0.5} = 4/10.05 \approx 0.398.$$

$\frac{1}{2}\log_2(4/D) = 1 \times 3.33 = 3.33KATEXPLACEHOLDER0END4/D = 2^{6.66} \approx 101KATEXPLACEHOLDER1ENDD_1 = 4/101 \approx 0.0396.$$The improvement in dB:$10\log_{10}(D_{0.5}/D_1) = 10\log_{10}(10.05) \approx 10$ dB. Doubling the bandwidth ratio from 0.5 to 1 reduces distortion by about 10 dB at 20 dB SNR.

Source coding: Since $S \to T_2 \to T_1$ is a Markov chain, successive refinement (Chapter 6) is optimal. Encode $S^n$ into a base description $M_0$ at rate $R_0 = R_{S|T_1}(D_1)$ (for receiver 1, which has weaker SI) and a refinement $M_1$ at additional rate $R_1 - R_0 = R_{S|T_2}(D_2) - R_{S|T_1}(D_1)$ (for receiver 2, which has better SI and needs lower distortion).

Channel coding: Use superposition coding on the degraded BC. The cloud center carries $M_0$ (decodable by both receivers, since receiver 2 can decode anything receiver 1 can on a degraded BC). The satellite carries $M_1$ (decodable only by receiver 1, who has the better channel $Y_1$).

Wait — receiver 1 has the better channel but worse side information. The degraded BC means $Y_1$ is stronger than $Y_2$, and the degraded SI means $T_1$ is weaker than $T_2$. The base layer (high rate, decodable by both) carries the coarse description, and receiver 2 uses its good SI to reconstruct at low distortion even from the base layer alone. Receiver 1, with its strong channel, decodes both layers to compensate for its weaker SI.

By Fano's inequality at each receiver: $$nR_{S|T_k}(D_k) \leq I(S^n; Y_k^n | T_k^n) + n\epsilon_n.$$ The degradedness of the BC ensures that $I(S^n; Y_1^n | T_1^n) \geq I(S^n; Y_2^n | T_2^n)$ modulo the SI structure. Combined with the BC capacity region converse, this shows that the rates must lie in the degraded BC capacity region. $\blacksquare$

Set $X = \alpha S$ where $\alpha = \sqrt{P/\sigma_S^2}$ to satisfy the power constraint $\mathbb{E}[X^2] = \alpha^2 \sigma_S^2 = P$.

The receiver observes $Y = \alpha S + Z$.

Since $S$ and $Z$ are jointly Gaussian and independent, the MMSE estimate is the linear MMSE estimate: $$\hat{S} = \mathbb{E}[S|Y] = \frac{\alpha \sigma_S^2}{\alpha^2 \sigma_S^2 + \sigma^2} Y.$$

The MSE is $$D_{\text{uncoded}} = \sigma_S^2 - \frac{\alpha^2 \sigma_S^4}{\alpha^2 \sigma_S^2 + \sigma^2} = \frac{\sigma_S^2 \sigma^2}{\alpha^2 \sigma_S^2 + \sigma^2} = \frac{\sigma_S^2 \sigma^2}{P + \sigma^2} = \frac{\sigma_S^2}{1 + P/\sigma^2} = \frac{\sigma_S^2}{1 + \text{SNR}}.$$

Under separation at $\kappa = 1$: $$D^* = \sigma_S^2 \cdot 2^{-2C} = \sigma_S^2 \cdot (1 + \text{SNR})^{-1} = \frac{\sigma_S^2}{1 + \text{SNR}}.$$

We have $D_{\text{uncoded}} = D^*$. Uncoded linear transmission achieves the information-theoretic minimum distortion for the Gaussian source over the Gaussian channel at $\kappa = 1$. $\blacksquare$

With $X_k = \sqrt{P} S_k$, the received signal is $Y = \sqrt{P}(S_1 + S_2) + Z$. The covariance matrix of $(S_1, S_2)$ is $\mathbf{K}_S = \begin{bmatrix} 1 & \rho \\ \rho & 1 \end{bmatrix}$.

The MMSE estimate of $S_k$ given $Y$ is $$\hat{S}_k = \frac{\text{Cov}(S_k, Y)}{\text{Var}(Y)} Y.$$

$\text{Cov}(S_1, Y) = \sqrt{P}(\text{Var}(S_1) + \text{Cov}(S_1, S_2)) = \sqrt{P}(1 + \rho)$.

$\text{Var}(Y) = P(1 + 1 + 2\rho) + 1 = 2P(1 + \rho) + 1$.

The MMSE distortion for $S_1$ is: $$D_1 = 1 - \frac{P(1+\rho)^2}{2P(1+\rho) + 1}.$$

Taking the derivative with respect to $\rho$: $$\frac{\partial D_1}{\partial \rho} = -\frac{2P(1+\rho)(2P(1+\rho)+1) - P(1+\rho)^2 \cdot 2P}{(2P(1+\rho)+1)^2}$$ $$= -\frac{2P(1+\rho)}{(2P(1+\rho)+1)^2} < 0$$

for all $\rho \in [0, 1)$ and $P > 0$. Therefore, the distortion is strictly decreasing in $\rho$: more correlation means lower distortion.

Intuitively, when $\rho$ is high, $S_1 + S_2 \approx 2S_1$, so the MAC effectively provides a boosted observation of each source. The MMSE decoder uses the known correlation to separate the superimposed signals.

Setting $U_1 = U_2 = \emptyset$, the conditions become: $$H(S_1 | S_2) < I(X_1; Y | X_2)$$ $$H(S_2 | S_1) < I(X_2; Y | X_1)$$ $$H(S_1, S_2) < I(X_1, X_2; Y)$$ $$H(S_1 | S_2) + H(S_2) < I(X_1; Y | X_2) + I(X_2; Y)$$

The first three inequalities say exactly that the Slepian–Wolf corner point $(H(S_1|S_2), H(S_2|S_1))$ and the sum rate $H(S_1, S_2)$ are achievable inside the MAC capacity region for the input distribution $P_{X_1}P_{X_2}$. The fourth condition provides an additional "mixed" constraint.

Since the MAC capacity region is the set of $(R_1, R_2)$ satisfying $R_1 \leq I(X_1; Y|X_2)$, $R_2 \leq I(X_2; Y|X_1)$, $R_1 + R_2 \leq I(X_1, X_2; Y)$, and the Slepian–Wolf region requires $R_1 \geq H(S_1|S_2)$, $R_2 \geq H(S_2|S_1)$, $R_1 + R_2 \geq H(S_1, S_2)$, the conditions reduce to the Slepian–Wolf region fitting inside the MAC capacity region. $\blacksquare$

$D_{\text{opt}} = \sigma_S^2 \cdot 2^{-2\kappaC} = (1 + \text{SNR})^{-\kappa}.$

At $\kappa = 3$: $D_{\text{opt}} = (1 + \text{SNR})^{-3}$.

The best uncoded scheme at $\kappa = 3$ transmits the source 3 times (or uses an optimal linear mapping $\mathbb{R} \to \mathbb{R}^3$). With MRC over 3 copies, the effective SNR is $3\text{SNR}$, giving $D_{\text{uncoded}} = (1 + 3\text{SNR})^{-1}.$

$\frac{D_{\text{uncoded}}}{D_{\text{opt}}} = \frac{(1+\text{SNR})^3}{1 + 3\text{SNR}} \approx \frac{\text{SNR}^{3}}{3\text{SNR}} = \frac{\text{SNR}^{2}}{3} \approx \text{SNR}^{\kappa - 1}.$$The ratio grows as$\text{SNR}^{\kappa-1}$, confirming that the penalty of uncoded transmission grows exponentially with the bandwidth ratio. At$\kappa = 3$and$\text{SNR} = 20$dB ($= 100$), the ratio is approximately$100^2/3 \approx 3333$, or about 35 dB — a massive penalty for not coding.

Let $\boldsymbol{\alpha} = [\sqrt{P_1/\sigma_{S_1}^2}, \ldots, \sqrt{P_K/\sigma_{S_K}^2}]^T$ and $\mathbf{h}_{\text{eff}} = \mathbf{h} \odot \boldsymbol{\alpha}$ (element-wise product). Then $Y = \mathbf{h}_{\text{eff}}^T \mathbf{S} + Z$ where $\mathbf{S} = [S_1, \ldots, S_K]^T$.

The MMSE estimate of $S_k$ given $Y$ is: $$\hat{S}_k = \frac{[\mathbf{K}_S \mathbf{h}_{\text{eff}}]_k}{\mathbf{h}_{\text{eff}}^T \mathbf{K}_S \mathbf{h}_{\text{eff}} + 1} Y.$$

The MMSE distortion is: $$D_k = [\mathbf{K}_S]_{kk} - \frac{[\mathbf{K}_S \mathbf{h}_{\text{eff}}]_k^2}{\mathbf{h}_{\text{eff}}^T \mathbf{K}_S \mathbf{h}_{\text{eff}} + 1}.$$

The numerator of the subtracted term is $[\mathbf{K}_S \mathbf{h}_{\text{eff}}]_k^2$, which equals $(\sum_i [\mathbf{K}_S]_{ki} [\mathbf{h}_{\text{eff}}]_i)^2$. This depends on the off-diagonal entries of $\mathbf{K}_S$ (the cross-correlations), not just the diagonal entries (variances). The denominator similarly depends on the full matrix through $\mathbf{h}_{\text{eff}}^T \mathbf{K}_S \mathbf{h}_{\text{eff}}$.

The MMSE decoder uses the known correlation structure to "separate" the superimposed signals — a capability that separate source and channel coding would need to replicate through explicit distributed compression.

With $K = 3$, $h_k = 1$, $P_k = P$, $\sigma_{S_k}^2 = 1$: $\mathbf{h}_{\text{eff}} = \sqrt{P}\mathbf{1}_3$ and $\mathbf{K}_S = (1-\rho)\mathbf{I}_3 + \rho\mathbf{1}_3\mathbf{1}_3^T$.

$[\mathbf{K}_S \mathbf{h}_{\text{eff}}]_k = \sqrt{P}(1 - \rho + 3\rho) = \sqrt{P}(1 + 2\rho)$.

$\mathbf{h}_{\text{eff}}^T \mathbf{K}_S \mathbf{h}_{\text{eff}} = 3P(1 + 2\rho)$.

$$D_k = 1 - \frac{P(1+2\rho)^2}{3P(1+2\rho) + 1} = 1 - \frac{P(1+2\rho)}{3P + \frac{1}{1+2\rho}}.$$

As $\rho \to 1$: $D_k \to 1 - \frac{3P}{3P + 1/3} = \frac{1}{9P + 1}$, which is much smaller than the $\rho = 0$ case $D_k = 1 - \frac{P}{3P + 1}$.

When separation is justified:

• Point-to-point links with moderate-to-long blocklengths ($n > 1000$): the separation penalty is negligible (< 0.1 dB for $n > 5000$).
• Independent sources: no correlation to exploit.
• Well-characterized channels: the channel code can be optimized for the known channel statistics.
• Modularity benefits: separate design allows independent evolution of source and channel codecs (e.g., upgrading from H.264 to H.265 without changing the LDPC code).

When separation may fail:

• URLLC at short blocklength ($n < 100$): the finite-blocklength penalty can reach 1–3 dB. DeepJSCC has shown gains in this regime.
• Massive IoT with correlated sources: thousands of sensors observing correlated physical fields. Separate compression discards the correlation that joint coding could exploit through the MAC.
• Channel mismatch: when the actual channel differs from the design channel, digital schemes suffer cliff effects while analog/hybrid schemes degrade gracefully.
• Semantic communication: when the receiver only needs a function of the source (not full reconstruction), joint design can dramatically reduce the required rate.

The DeepJSCC frontier: Recent work on deep learning-based joint source–channel coding (Bourtsoulatze et al., 2019) has demonstrated practical JSCC schemes that outperform separate coding for image transmission at short blocklengths and under channel mismatch. These results suggest that for specific source-channel pairs, learned joint codes can approach the theoretical gains promised by information theory.

Conclusion: Separation is not a theoretical limitation but an engineering tradeoff. For the dominant use cases in current standards, it is the right choice. For emerging scenarios (URLLC, mMTC, semantic communication), joint design may offer meaningful gains that justify the additional complexity.