Exercises

Let $W = \mathbb{E}[X|Y] = \rho Y / \sigma_Y$ (the MMSE estimator, scaled). Given $W$, the conditional distribution of $X$ given $W$ and $Y$ given $W$ are independent Gaussians (since removing the common component leaves independent residuals in the Gaussian case).

$I(X, Y; W) = H(W) - H(W | X, Y)$. Since $W$ is a function of $Y$ (or $X$), $H(W|X,Y) = 0$ in the degenerate case...

More carefully: $I(X,Y; W) = I(X; W) + I(Y; W | X)$. By the Markov chain, $I(Y; W | X) = 0$ if $W$ is chosen as a function of the common part. For Gaussians, $C_W(X;Y) = \frac{1}{2}\log\frac{1}{1-\rho^2} = I(X;Y)$.

$I(X; Y) = H(Y) - H(Y|X) = 1 - \mathcal{H}_2(p)$.

For the DSBS, there is no nontrivial $W$ that renders $X \perp Y | W$ with a binary $W$. The minimum $I(X,Y; W)$ over all valid $W$ can be shown to equal $1$ (since $W$ must essentially identify $X$ or $Y$ to break the dependence). Therefore $C_W(X;Y) = 1 > 1 - \mathcal{H}_2(p) = I(X;Y)$ for $p > 0$.

This shows that Wyner's common information can strictly exceed mutual information for discrete sources.

$\mathcal{L}_{\text{IB}} = I(X; T) - \beta \cdot I(T; Y)$$By the chain rule:$I(X; T) = I(X; T | Y) + I(T; Y) - I(T; Y | X)$. Since$T - X - Y$is Markov:$I(T; Y | X) = 0$. Therefore:$I(X; T) = I(X; T | Y) + I(T; Y)$.

$\mathcal{L}_{\text{IB}} = I(X; T | Y) + I(T; Y) - \beta \cdot I(T; Y)KATEXPLACEHOLDER0END= I(X; T | Y) + (1 - \beta)I(T; Y)$$At$\beta = 1$:$\mathcal{L}_{\text{IB}} = I(X; T | Y)$. So minimizing the IB at$\beta = 1$is equivalent to finding the representation$T$that contains the least information about$X$beyond what$Y$already provides — i.e.,$T$should be a sufficient statistic of$X$for$Y$.

Under the joint distribution $P_{\text{data}}(x) q_\phi(z|x)$:

$$I(X; Z) = \mathbb{E}\left[\log\frac{q_\phi(z|x)}{q_\phi(z)}\right]$$

where $q_\phi(z) = \int P_{\text{data}}(x) q_\phi(z|x) dx$ is the aggregated posterior.

$\mathbb{E}[D(q_\phi(z|x) \| p(z))] = \mathbb{E}\left[\log\frac{q_\phi(z|x)}{p(z)}\right]KATEXPLACEHOLDER0END= \mathbb{E}\left[\log\frac{q_\phi(z|x)}{q_\phi(z)}\right] + \mathbb{E}\left[\log\frac{q_\phi(z)}{p(z)}\right]KATEXPLACEHOLDER1END= I(X; Z) + D(q_\phi(z) \| p(z))KATEXPLACEHOLDER2END\mathbb{E}[D(q_\phi(z|x) \| p(z))] \geq I(X; Z)$$The gap is$D(q_\phi(z) | p(z))$, which measures how far the aggregated posterior deviates from the prior.

$R_{\text{syn}} \geq H(X|Y) = \mathcal{H}_2(0.05)KATEXPLACEHOLDER0END= -0.05\log_2(0.05) - 0.95\log_2(0.95) \approx 0.286 \text{ bits}$$

The LDPC code rate is $k/n = 1 - R_{\text{syn}} \leq 1 - 0.286 = 0.714$. This must be below the BSC(0.05) capacity of $1 - \mathcal{H}_2(0.05) \approx 0.714$. In practice, we use a code rate slightly below this (e.g., 0.70) to allow for the gap to capacity of practical LDPC codes.

$\mathcal{L} = \sum_{x,t} P_X(x) P_{T|X}(t|x) \log\frac{P_{T|X}(t|x)}{P_T(t)}KATEXPLACEHOLDER0END- \beta \sum_{t,y} P_T(t) P_{Y|T}(y|t) \log\frac{P_{Y|T}(y|t)}{P_Y(y)}KATEXPLACEHOLDER1END- \sum_x \lambda(x)\left(\sum_t P_{T|X}(t|x) - 1\right)$$

Setting $\frac{\partial \mathcal{L}}{\partial P_{T|X}(t|x)} = 0$:

$$P_X(x)\left[\log\frac{P_{T|X}(t|x)}{P_T(t)} + 1\right] - \beta \sum_y P_X(x) P_{Y|X}(y|x) \log\frac{P_{Y|T}(y|t)}{P_Y(y)} - \lambda(x) = 0$$

Rearranging and using the normalization constraint to determine $\lambda(x)$:

$$P_{T|X}(t|x) = \frac{P_T(t)}{Z(x,\beta)} \exp\left(\beta \sum_y P_{Y|X}(y|x) \log P_{Y|T}(y|t)\right)$$

$$= \frac{P_T(t)}{Z(x,\beta)} \exp\left(-\beta D(P_{Y|X}(\cdot|x) \| P_{Y|T}(\cdot|t))\right) \cdot \text{const}$$

where $Z(x, \beta)$ ensures normalization.

For diagonal $\Sigma = \text{diag}(\sigma_1^2, \ldots, \sigma_d^2)$:

$$D(\mathcal{N}(\mu, \Sigma) \| \mathcal{N}(0, I)) = \frac{1}{2}\left[\text{tr}(\Sigma) + \|\mu\|^2 - d - \log\det\Sigma\right]$$

$$= \frac{1}{2}\left[\sum_j \sigma_j^2 + \sum_j \mu_j^2 - d - \sum_j \log\sigma_j^2\right]$$

$$= \frac{1}{2}\sum_{j=1}^d [\mu_j^2 + \sigma_j^2 - 1 - \log\sigma_j^2]$$

Each latent dimension contributes independently to the KL cost. The term $\mu_j^2 + \sigma_j^2 - 1 - \log\sigma_j^2 \geq 0$ with equality iff $\mu_j = 0$ and $\sigma_j = 1$ (matching the prior).

Dimensions with large $|\mu_j|$ carry more "information" about $x$ (the mean shifts away from the prior) and incur higher rate cost. This is the rate-distortion tradeoff in action: each dimension acts as a "channel" with capacity determined by how far the encoder pushes the posterior from the prior.

$H(X) = \log 4 = 2$ bits, $H(Y) = 1$ bit. $H(X|Y) = 1$ bit (given parity, two equally likely values remain). $H(Y|X) = 0$ (parity is a deterministic function of $X$).

• At $R_Y = 0$: $R_X = H(X) = 2$ bits (no help).
• At $R_Y \geq H(Y|X) = 0$: any $R_Y > 0$ suffices for full help!

Actually, the helper needs to send $Y$ at rate $R_Y \geq H(Y) = 1$ bit for the decoder to fully know $Y$. But with the Slepian-Wolf corner point, the helper only needs rate $R_Y \geq H(Y|X) = 0$ since $X$ determines $Y$.

So for any $R_Y > 0$: $R_X = H(X|Y) = 1$ bit. At $R_Y = 0$: $R_X = 2$ bits.

The transition is sharp — even a tiny helper rate gives the full benefit because $Y$ is a function of $X$ (zero conditional entropy).

With binary $T$: $$P_{T|X}(t|x) = \frac{P_T(t)}{Z(x)} \exp(-\beta D(P_{Y|X}(\cdot|x) \| P_{Y|T}(\cdot|t)))$$

For $t \in \{0,1\}$, $x$ is assigned to cluster $t=0$ or $t=1$ based on which cluster's average posterior $P_{Y|T}(\cdot|t)$ is closer (in KL sense) to $P_{Y|X}(\cdot|x)$.

For binary $Y$ and binary $T$, the KL divergence reduces to comparing $P(Y=1|X=x)$ with $P(Y=1|T=t)$. Inputs with $P(Y=1|X=x) > 0.5$ tend to cluster $t=1$, and those with $P(Y=1|X=x) < 0.5$ tend to cluster $t=0$.

At $\beta \to \infty$ (hard clustering), this becomes a deterministic threshold on $P(Y=1|X=x)$. At finite $\beta$, the clustering is soft, with the "temperature" $1/\beta$ controlling the fuzziness.

This is precisely what a logistic regression classifier does — the IB provides an information-theoretic justification for soft classification.

In standard channel coding on a BSC($p$): a codeword $c^n$ is transmitted, the received vector is $r^n = c^n \oplus e^n$ where $e^n$ is the error pattern with i.i.d. $\text{Bernoulli}(p)$ entries. The decoder uses $\mathbf{H}r^n = \mathbf{H}e^n$ (since $\mathbf{H}c^n = 0$ for valid codewords) to estimate $e^n$.

In DISCUS: $x^n$ is the source, $y^n$ is the side information, $e^n = x^n \oplus y^n$ is the "correlation noise." The encoder sends $s = \mathbf{H}x^n$, and the decoder computes $s' = s \oplus \mathbf{H}y^n = \mathbf{H}(x^n \oplus y^n) = \mathbf{H}e^n$.

This is identical to the channel decoding problem: the decoder has syndrome $s' = \mathbf{H}e^n$ and must recover $e^n$, where $e^n$ has i.i.d. $\text{Bernoulli}(p)$ entries. Any BSC($p$)-capacity-approaching decoder (belief propagation for LDPC codes) solves both problems.