Exercises

$h(X) = -\int_{-a}^{a} \frac{1}{2a} \log \frac{1}{2a}\,dx = \log(2a)$.

For $a = 1/2$: $h = 0$. For $a < 1/2$: $h < 0$.

$h(X+c) = -\int f_X(x-c) \log f_X(x-c)\,dx$. Substituting $u = x - c$: $= -\int f_X(u) \log f_X(u)\,du = h(X)$.

$h(X) = \frac{1}{2}\log(2\pi e \cdot 9) = \frac{1}{2}\log(18\pi e) \approx 3.28$ bits.

Note: $h(\mathcal{N}(\mu, \sigma^2)) = \frac{1}{2}\log(2\pi e \sigma^2)$ for any $\mu$.

$h(Y) = h(3X + 2) = h(3X) = h(X) + \log 3 = \frac{1}{2}\log(2\pi e) + \log 3$.

Alternatively: $Y \sim \mathcal{N}(2, 9)$, so $h(Y) = \frac{1}{2}\log(2\pi e \cdot 9) = \frac{1}{2}\log(2\pi e) + \frac{1}{2}\log 9 = \frac{1}{2}\log(2\pi e) + \log 3$. Consistent.

$I(X;Y) = D(f_{XY} \| f_X f_Y)$.

Under the invertible map $(X,Y) \mapsto (U,V) = (g(X), h(Y))$, both the joint density and the product of marginals transform with the same Jacobian factor $|g'(X)|^{-1} |h'(Y)|^{-1}$, which cancels in the ratio:

$\frac{f_{UV}(u,v)}{f_U(u)f_V(v)} = \frac{f_{XY}(g^{-1}(u), h^{-1}(v))}{f_X(g^{-1}(u))f_Y(h^{-1}(v))}$.

Therefore $D(f_{UV} \| f_U f_V) = D(f_{XY} \| f_X f_Y)$.

$h(X) = -\int_0^\infty \lambda e^{-\lambda x}(\ln\lambda - \lambda x)\frac{dx}{\ln 2} = \frac{1 - \ln\lambda}{\ln 2} = \log(e/\lambda)$.

Let $f$ be any PDF on $[0,\infty)$ with $\mathbb{E}[X] = 1/\lambda$, and let $\phi(x) = \lambda e^{-\lambda x}$.

$D(f \| \phi) = -h(X) - \int f(x)\log\phi(x)\,dx = -h(X) + \log\lambda + \frac{\lambda}{\ln 2}\mathbb{E}[X]$.

$= -h(X) + \log\lambda + \frac{1}{\ln 2} = -h(X) + \log(e\lambda) \geq 0$.

Therefore $h(X) \leq \log(e/\lambda) = h(\text{Exp}(\lambda))$.

$Y \sim \mathcal{N}(0, P + N)$, so $h(Y) = \frac{1}{2}\log(2\pi e(P+N))$.

$h(Y|X) = h(Z) = \frac{1}{2}\log(2\pi eN)$.

$I(X;Y) = \frac{1}{2}\log\frac{P+N}{N} = \frac{1}{2}\log(1 + P/N) = \frac{1}{2}\log(1 + \text{SNR})$.

Let $\mathbf{Y} = \mathbf{A}\mathbf{X}$. The PDF transforms as $f_{\mathbf{Y}}(\mathbf{y}) = \frac{1}{|\det(\mathbf{A})|} f_{\mathbf{X}}(\mathbf{A}^{-1}\mathbf{y})$.

$h(\mathbf{Y}) = -\int f_{\mathbf{Y}} \log f_{\mathbf{Y}}\,d\mathbf{y}$

$= -\int \frac{f_{\mathbf{X}}(\mathbf{A}^{-1}\mathbf{y})}{|\det\mathbf{A}|} \log\frac{f_{\mathbf{X}}(\mathbf{A}^{-1}\mathbf{y})}{|\det\mathbf{A}|}\,d\mathbf{y}$

$= -\int f_{\mathbf{X}}(\mathbf{x})[\log f_{\mathbf{X}}(\mathbf{x}) - \log|\det\mathbf{A}|]\,d\mathbf{x} = h(\mathbf{X}) + \log|\det\mathbf{A}|$.

$h(\mathbf{X}) = \frac{1}{2}\log((2\pi e)^n \det(\mathbf{K})) = \frac{1}{2}\log\prod_{i=1}^n (2\pi e \lambda_i)$

$= \frac{1}{2}\sum_{i=1}^n \log(2\pi e \lambda_i) = \sum_{i=1}^n h(\tilde{X}_i)$

where $\tilde{X}_i \sim \mathcal{N}(0, \lambda_i)$ are the components in the eigenbasis. The total entropy is the sum of entropies along independent principal components.

Since the channels are independent: $I(\mathbf{X};\mathbf{Y}) = \sum_{k=1}^K I(X_k;Y_k) = \sum_k \frac{1}{2}\log(1 + P_k/N_k)$.

Maximize $\sum_k \frac{1}{2}\log(1 + P_k/N_k)$ subject to $\sum_k P_k \leq P$ and $P_k \geq 0$.

The Lagrangian is $\sum_k \frac{1}{2}\log(1 + P_k/N_k) - \nu(\sum_k P_k - P)$.

KKT conditions give: $P_k^* = [\nu - N_k]_+$ where $\nu$ is chosen so that $\sum_k P_k^* = P$.

$C = \sum_{k=1}^K \frac{1}{2}\log\frac{\max(\nu, N_k)}{N_k}$.

The waterfilling interpretation: fill water to level $\nu$ over a "landscape" with heights $N_k$. Channels with noise above the water level get zero power — they are too noisy to be useful.

$I(X; X+Y) = h(X+Y) - h(X+Y|X) = h(X+Y) - h(Y) \geq 0$.

Therefore $h(X+Y) \geq h(Y)$.

Adding an independent signal to Gaussian noise can only increase the total entropy — the signal "spreads" the distribution further.

$X + \sqrt{t}Z \sim \mathcal{N}(0, t)$, so $h = \frac{1}{2}\log(2\pi e t)$ and $\frac{d}{dt}h = \frac{1}{2t\ln 2}$.

$J(\mathcal{N}(0,t)) = 1/t$, so $\frac{1}{2}J = \frac{1}{2t}$. In nats: $\frac{d}{dt}h_{\text{nats}} = \frac{1}{2t} = \frac{1}{2}J$. Verified.

$X + \sqrt{t}Z \sim \mathcal{N}(0, \sigma^2 + t)$.

$\frac{d}{dt}h = \frac{1}{2(\sigma^2 + t)\ln 2}$.

$J(\mathcal{N}(0, \sigma^2 + t)) = \frac{1}{\sigma^2 + t}$.

$\frac{1}{2}J = \frac{1}{2(\sigma^2 + t)}$. Converting to bits: matches. Verified.

Let $\phi$ be the PDF of $\mathcal{CN}(\mathbf{0}, \mathbf{K})$. For any $f$ with the same covariance:

$D(f\|\phi) = -h(\mathbf{X}) + n\log(\pi e) + \log\det(\mathbf{K}) \geq 0$.

The cross-entropy uses $\mathbb{E}[\mathbf{X}^H\mathbf{K}^{-1}\mathbf{X}] = \text{tr}(\mathbf{K}^{-1}\mathbf{K}) = n$.

Therefore $h(\mathbf{X}) \leq \log((\pi e)^n \det(\mathbf{K}))$.

$X|Y=y \sim \mathcal{N}\!\left(\frac{P}{P+N}y,\; \frac{PN}{P+N}\right)$.

The conditional variance does not depend on $y$.

$h(X|Y) = \frac{1}{2}\log\!\left(2\pi e \cdot \frac{PN}{P+N}\right)$.

Check: $I(X;Y) = h(X) - h(X|Y) = \frac{1}{2}\log(2\pi eP) - \frac{1}{2}\log\!\left(\frac{2\pi ePN}{P+N}\right) = \frac{1}{2}\log(1 + P/N)$. Consistent.

Define $Y_t = X + \sqrt{t}Z$. Using de Bruijn's identity:

$\frac{d}{dt}h(Y_t) = \frac{1}{2}J(Y_t)$, where $h$ is in nats.

$g(t) = e^{2h(Y_t)}$, so $g'(t) = 2g(t) \cdot \frac{1}{2}J(Y_t) = g(t)J(Y_t)$.

$g''(t) = g'(t)J(Y_t) + g(t)J'(Y_t) = g(t)[J(Y_t)^2 + J'(Y_t)]$.

One shows $J'(Y_t) \leq -J(Y_t)^2$ using the Cramér-Rao-type bound $J(Y_t) \leq 1/(t + N_X)$ where $N_X$ is the entropy power of $X$. This gives $g''(t) \leq 0$, proving concavity.

Concavity of $g$ gives: $g(1) \geq g(0) + g'(0)(1-0)$ is not the right bound. Instead, $g(1) \geq \frac{t}{1}g(0) + \frac{1-t}{1}\ldots$ The EPI follows from concavity along with a scaling argument.