Ferkans — Interactive Telecom Tutor

Beyond White Noise

The white noise assumption — that noise samples are i.i.d. — is a convenient idealization. In practice, noise is often colored: correlated across time or frequency. Interference from neighboring cells, quantization noise in ADCs, and feedback from automatic gain control all produce correlated disturbances. The key insight of this section is that the Karhunen-Lo`eve (KL) expansion diagonalizes the noise covariance, converting the colored-noise channel into a set of parallel Gaussian sub-channels with unequal noise variances — and water-filling once again provides the optimal power allocation.

Definition:
Gaussian Channel with Colored Noise

Consider the vector channel

$\mathbf{Y} = \mathbf{X} + \mathbf{Z}, \quad \mathbf{Z} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma}_{Z}),$

where $\boldsymbol{\Sigma}_{Z}$ is a positive definite noise covariance matrix and the power constraint is $\frac{1}{n}\mathbb{E}[\|\mathbf{X}\|^2] \leq P$ .

Since $\boldsymbol{\Sigma}_{Z}$ is positive definite, it admits an eigendecomposition $\boldsymbol{\Sigma}_{Z} = \mathbf{U} \boldsymbol{\Lambda} \mathbf{U}^T$ , where $\boldsymbol{\Lambda} = \text{diag}(\lambda_1, \ldots, \lambda_n)$ . In the rotated coordinate system $\tilde{\mathbf{X}} = \mathbf{U}^T\mathbf{X}$ , $\tilde{\mathbf{Y}} = \mathbf{U}^T\mathbf{Y}$ , the channel decomposes into $n$ independent sub-channels with noise variances $\lambda_k$ .

Theorem: Capacity with Colored Gaussian Noise

The capacity of the Gaussian channel with colored noise $\mathbf{Z} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma}_{Z})$ and per-symbol power constraint $P$ is

$C = \frac{1}{n}\sum_{k=1}^n \frac{1}{2}\log\!\left(1 + \frac{P_k^*}{\lambda_k}\right),$

where $\lambda_1, \ldots, \lambda_n$ are the eigenvalues of $\boldsymbol{\Sigma}_{Z}$ and the power allocation is the water-filling solution:

$P_k^* = \left[\nu - \lambda_k\right]_+,$

with $\nu$ chosen so that $\frac{1}{n}\sum_k P_k^* = P$ .

Colored noise has "easy directions" (eigenvectors with small eigenvalues) and "hard directions" (large eigenvalues). Water-filling exploits this structure by sending more power along the easy directions. This is the same principle as MIMO precoding: align the transmitted signal with the favorable directions of the channel.

Proof

Diagonalize the noise covariance

The eigendecomposition $\boldsymbol{\Sigma}_{Z} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^T$ decomposes the channel into $n$ independent sub-channels. Since $\mathbf{U}$ is orthogonal, the power constraint is preserved: $\|\tilde{\mathbf{X}}\|^2 = \|\mathbf{X}\|^2$ .

Apply the parallel Gaussian result

Each sub-channel has unit gain and noise variance $\lambda_k$ . By Theorem 10.3.1, the optimal power allocation is water-filling with $P_k^* = [\nu - \lambda_k]_+$ .

Example: Water-Filling with Colored Noise

A 2-dimensional channel has noise covariance $\boldsymbol{\Sigma}_{Z} = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ and power constraint $P = 3$ per dimension. Find the capacity.

Solution

Eigendecompose the noise covariance

The eigenvalues are $\lambda_1 = 1$ , $\lambda_2 = 3$ (eigenvectors: $(1,1)^T/\sqrt{2}$ and $(1,-1)^T/\sqrt{2}$ ).

Water-filling

Total power budget: $2P = 6$ . Try both channels active: $2\nu - (1+3) = 6$ , so $\nu = 5$ .

$P_1 = 5 - 1 = 4$ , $P_2 = 5 - 3 = 2$ . Both positive. $\checkmark$

Compute capacity

$C = \frac{1}{2}\left[\frac{1}{2}\log(1 + 4/1) + \frac{1}{2}\log(1 + 2/3)\right] = \frac{1}{2}\left[\frac{1}{2}\log 5 + \frac{1}{2}\log(5/3)\right].KATEXPLACEHOLDER0END= \frac{1}{4}\log 5 + \frac{1}{4}\log(5/3) = \frac{1}{4}\log(25/3) \approx 0.76 \text{ bits per dimension per channel use.}$ $

The Karhunen-Lo`eve Perspective

For stationary noise processes in the continuous-time setting, the Karhunen-Lo`eve expansion plays the role of the eigendecomposition. A stationary Gaussian process with power spectral density $S_Z(f)$ is diagonalized in the frequency domain by the Fourier transform. The "eigenvalues" become the spectral density $S_Z(f)$ , and water-filling over the noise spectrum determines the optimal transmitted power spectral density.

This connects the algebraic (matrix eigendecomposition) and analytic (spectral decomposition) views of the same principle: always transmit along the eigenmodes of the channel, allocating power by water-filling over the eigenvalues.

Water-Filling over Colored Noise Spectrum

Visualize water-filling over a continuous noise power spectral density. The bowl shape is $S_Z(f)$ (the noise spectrum), and water is poured to level $\nu$ . Adjust the total power and the noise spectral shape to explore the allocation.

Parameters

Total power

P

5

Noise spectral shape

Common Mistake: Do Not Whiten Then Ignore the Transformation

Mistake:

Whitening the noise by multiplying by $\boldsymbol{\Sigma}_{Z}^{-1/2}$ but forgetting that this also transforms the signal and changes the effective power constraint.

Correction:

The whitening filter $\boldsymbol{\Sigma}_{Z}^{-1/2}$ converts $\mathbf{Y} = \mathbf{X} + \mathbf{Z}$ to $\tilde{\mathbf{Y}} = \boldsymbol{\Sigma}_{Z}^{-1/2}\mathbf{X} + \tilde{\mathbf{Z}}$ where $\tilde{\mathbf{Z}} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ . But the effective channel is now $\boldsymbol{\Sigma}_{Z}^{-1/2}$ (not identity), and the power constraint becomes $\mathbb{E}[\|\boldsymbol{\Sigma}_{Z}^{-1/2}\mathbf{X}\|^2]$ , which differs from $\mathbb{E}[\|\mathbf{X}\|^2]$ when $\boldsymbol{\Sigma}_{Z} \neq \sigma^2\mathbf{I}$ . The correct approach is the eigendecomposition method.

Quick Check

For a channel with colored noise, when does equal power allocation achieve capacity?

When all noise eigenvalues are equal (i.e., the noise is actually white)

When the total power exceeds a certain threshold

Never — water-filling is always strictly better

When the noise covariance matrix is diagonal