Ferkans — Interactive Telecom Tutor

The Optimal Basis for a Random Process

Given a random process $X(t)$ on a finite interval $[0, T]$ , we want to represent it as a series $X(t) = \sum_n Z_n \phi_n(t)$ for some basis functions $\{\phi_n\}$ and random coefficients $\{Z_n\}$ . But which basis is best? The Fourier basis is convenient but not adapted to the process statistics. The point is that the Karhunen-Loève (KL) expansion chooses the basis that diagonalizes the autocorrelation operator — the eigenfunctions of $r_{xx}$ . This makes the coefficients uncorrelated (and independent for Gaussian processes), concentrates the maximum energy in the fewest terms, and provides the optimal finite-dimensional approximation in the mean-square sense. The KL expansion is the continuous-time analogue of principal component analysis (PCA).

,

Definition:
The Karhunen-Loève Expansion

Let $X(t)$ be a zero-mean, finite-variance random process on $[0, T]$ with autocorrelation $R_X(t, s) = \mathbb{E}[X(t)X^*(s)]$ . The Karhunen-Loève (KL) expansion of $X(t)$ is $X(t) = \text{l.i.m.}_{N\to\infty} \sum_{n=1}^{N} Z_n \phi_n(t),$ where:

The functions $\{\phi_n(t)\}$ are the orthonormal eigenfunctions of the autocorrelation kernel, satisfying the Fredholm integral equation $\int_0^T R_X(t, s)\,\phi_n(s)\, ds = \lambda_n\,\phi_n(t), \quad t \in [0, T].$
The eigenvalues $\lambda_1 \geq \lambda_2 \geq \cdots \geq 0$ are real and non-negative.
The random coefficients are $Z_n = \int_0^T X(t)\,\phi_n^*(t)\, dt$ .
The $\{Z_n\}$ are uncorrelated: $\mathbb{E}[Z_n Z_m^*] = \lambda_n\,\delta_{nm}$ .
For Gaussian $X(t)$ , the $\{Z_n\}$ are independent: $Z_n \sim \mathcal{N}(0, \lambda_n)$ .

The convergence is in the mean-square sense: $\mathbb{E}\!\left[\left|X(t) - \sum_{n=1}^N Z_n\phi_n(t)\right|^2\right] \to 0$ for each $t$ . For Gaussian processes, convergence also holds uniformly on $[0, T]$ with probability one under mild regularity.

, ,

Karhunen-Loève Expansion

A series representation $X(t) = \sum_n Z_n \phi_n(t)$ where $\{\phi_n\}$ are eigenfunctions of the autocorrelation kernel and $\{Z_n\}$ are uncorrelated random coefficients with variance equal to the corresponding eigenvalues.

Fredholm Integral Equation

The eigenvalue problem $\int_0^T K(t,s)\phi(s)\,ds = \lambda\phi(t)$ for an integral operator with kernel $K$ . In the KL expansion, the kernel is the autocorrelation function.

Related: Karhunen-Loève Expansion

Principal Component Analysis (PCA)

The discrete analogue of the KL expansion. For a random vector $\mathbf{X}$ with covariance $\boldsymbol{\Sigma}$ , PCA represents $\mathbf{X}$ in the eigenbasis of $\boldsymbol{\Sigma}$ , yielding uncorrelated components ordered by variance.

Related: Karhunen-Loève Expansion

Theorem: Optimality of the KL Expansion (Minimum Mean-Square Truncation Error)

Among all orthonormal expansions $X(t) = \sum_{n=1}^{\infty} Z_n \psi_n(t)$ with uncorrelated coefficients, the KL expansion minimizes the mean-square truncation error: $\varepsilon_N = \mathbb{E}\!\left[\int_0^T \left|X(t) - \sum_{n=1}^N Z_n \phi_n(t)\right|^2 dt\right].$

Specifically, $\varepsilon_N = \sum_{n=N+1}^{\infty} \lambda_n$ , and no other $N$ -term orthonormal expansion achieves a smaller error.

The KL basis concentrates the process energy into the first few coefficients by construction: the eigenvalues are ordered $\lambda_1 \geq \lambda_2 \geq \cdots$ , so the first $N$ terms capture the maximum possible energy. Any other basis would "spread" energy more evenly across coefficients, requiring more terms for the same approximation quality.

Proof

Express the total energy

The total average energy over $[0, T]$ is $\mathbb{E}\!\left[\int_0^T |X(t)|^2\, dt\right] = \int_0^T R_X(t,t)\, dt = \sum_{n=1}^{\infty} \lambda_n,$ where the last equality follows from Mercer's theorem (the eigenvalue expansion of the kernel's trace).

Compute the truncation error

The $N$ -term approximation $\hat{X}_N(t) = \sum_{n=1}^N Z_n \phi_n(t)$ has energy $\sum_{n=1}^N \lambda_n$ . The residual $X(t) - \hat{X}_N(t) = \sum_{n=N+1}^{\infty} Z_n \phi_n(t)$ has energy $\sum_{n=N+1}^{\infty} \lambda_n$ . Therefore $\varepsilon_N = \sum_{n>N} \lambda_n$ .

Prove optimality

Let $\{\psi_n\}$ be any other orthonormal basis with coefficients $W_n = \int X(t)\psi_n^*(t)\, dt$ . The energy in the first $N$ components is $\sum_{n=1}^N \mathbb{E}[|W_n|^2]$ . By a variational argument (or directly by the Courant-Fischer min-max theorem applied to the integral operator), $\max_{\psi_1,\ldots,\psi_N} \sum_{n=1}^N \mathbb{E}[|W_n|^2]$ is achieved when $\psi_n = \phi_n$ (the KL eigenfunctions), and the maximum equals $\sum_{n=1}^N \lambda_n$ . Hence the truncation error $\varepsilon_N$ is minimized by the KL basis.

, ,

Example: KL Expansion of the Wiener Process on $[0, T]$

Find the KL expansion of the Wiener process $W(t)$ on $[0, T]$ , which has autocorrelation $R_W(t, s) = \sigma^2 \min(t, s)$ .

Solution

Set up the Fredholm equation

We need $\int_0^T \sigma^2 \min(t, s)\,\phi(s)\, ds = \lambda\,\phi(t)$ . Split the integral at $s = t$ : $\sigma^2 \left[\int_0^t s\,\phi(s)\, ds + t\int_t^T \phi(s)\, ds\right] = \lambda\,\phi(t).$

Convert to a differential equation

Differentiating twice with respect to $t$ (using Leibniz's rule), we get $-\sigma^2 \phi(t) = \lambda\,\phi''(t),$ with boundary conditions $\phi(0) = 0$ (from the integral equation at $t = 0$ ) and $\phi'(T) = 0$ (from differentiating once and evaluating at $t = T$ ).

Solve the ODE

The general solution of $\phi'' + (\sigma^2/\lambda)\phi = 0$ is $\phi(t) = A\sin(\omega t) + B\cos(\omega t)$ with $\omega = \sigma/\sqrt{\lambda}$ . The boundary condition $\phi(0) = 0$ gives $B = 0$ . The condition $\phi'(T) = 0$ gives $\omega\cos(\omega T) = 0$ , so $\omega T = (n - \tfrac{1}{2})\pi$ for $n = 1, 2, 3, \ldots$ .

Write the eigenvalues and eigenfunctions

$\lambda_n = \frac{\sigma^2 T^2}{\bigl(n - \frac{1}{2}\bigr)^2 \pi^2}, \qquad \phi_n(t) = \sqrt{\frac{2}{T}}\sin\!\left(\frac{(n - \frac{1}{2})\pi t}{T}\right).$ $The KL expansion is$ W(t) = \sum_{n=1}^{\infty} Z_n \phi_n(t) $with$ Z_n \sim \mathcal{N}(0, \lambda_n) $independent. Note$ \sum_n \lambda_n = \sigma^2 T $(Mercer's theorem), matching$ \mathbb{E}[W(T)^2] = \sigma^2 T$.

,

Karhunen-Loève Expansion: Convergence vs. Number of Terms

Visualize the KL expansion of a Wiener process on $[0, T]$ . As more eigenfunctions are included, the expansion approximates the true realization more closely. Observe how the eigenvalues $\lambda_n$ decay (lower plot) and how the truncation error decreases.

Parameters

Number of KL terms

N

5

T

(interval length)1

\sigma

(process parameter)1

Random seed7

Karhunen-Loève Basis Functions Building Up a Random Realization

Watch the KL eigenfunctions

\phi_n(t)

accumulate one by one to approximate a random process realization. The animation shows how each successive term adds finer detail, with the eigenvalue

\lambda_n

controlling the amplitude of each component.

Each frame adds one KL term

Z_n \phi_n(t)

. The eigenvalues

\lambda_n

decay, so later terms contribute less energy.

The KL Expansion in Detection Theory

The KL expansion transforms the continuous-time detection problem "test $H_0: X(t) = W(t)$ vs. $H_1: X(t) = s(t) + W(t)$ for $t \in [0, T]$ " into an equivalent discrete problem in the KL coefficients. Under $H_0$ , the KL coefficients are $Z_n \sim \mathcal{N}(0, \lambda_n)$ ; under $H_1$ , they are $Z_n + s_n \sim \mathcal{N}(s_n, \lambda_n)$ where $s_n = \int s(t)\phi_n^*(t)\, dt$ . Since the $\{Z_n\}$ are independent for Gaussian noise, the likelihood ratio factors, and we recover the matched filter as the sufficient statistic. This is the rigorous justification of the matched filter derived heuristically in Ch. 15.

,

KL Expansion vs. Fourier Series

The Fourier series and the KL expansion are both orthonormal expansions of a process on $[0, T]$ , but they differ in a crucial way:

The Fourier basis $\{e^{j2\pi nt/T}\}$ is fixed and independent of the process. The coefficients are generally correlated unless $X(t)$ is WSS with specific structure.
The KL basis $\{\phi_n(t)\}$ is adapted to the process statistics. The coefficients are always uncorrelated (by construction).

For a WSS process on a long interval, the KL eigenfunctions approach the Fourier exponentials, and the eigenvalues approach the PSD samples $P_x(n/T)$ . This is the connection between the KL expansion and the Wiener-Khinchin theorem.

,

Definition:
Mercer's Theorem

If $R_X(t, s)$ is a continuous, positive semi-definite kernel on $[0, T] \times [0, T]$ , then it admits the eigenvalue expansion $R_X(t, s) = \sum_{n=1}^{\infty} \lambda_n\,\phi_n(t)\,\phi_n^*(s),$ where convergence is absolute and uniform. In particular, $\int_0^T R_X(t, t)\, dt = \sum_{n=1}^{\infty} \lambda_n,$ so the total energy equals the sum of eigenvalues (the trace of the operator).

,

Definition:
KL Expansion of WSS Processes on Large Intervals

For a WSS process on $[0, T]$ with $T$ large, the KL eigenfunctions are approximately complex exponentials $\phi_n(t) \approx \frac{1}{\sqrt{T}} e^{j2\pi f_n t}$ with $f_n = n/T$ , and the eigenvalues are approximately PSD samples: $\lambda_n \approx P_x(f_n) \cdot \Delta f$ where $\Delta f = 1/T$ .

This means for large $T$ , the KL expansion coincides with the Fourier expansion, and the eigenvalue distribution converges to the PSD.

,

Karhunen-Loève vs. Fourier Expansion

Property	KL Expansion	Fourier Series
Basis functions	Eigenfunctions of $R_X(t,s)$ — adapted to the process	Complex exponentials $e^{j2\pi nt/T}$ — fixed
Coefficients	Always uncorrelated (independent for Gaussian)	Generally correlated
Optimality	Minimizes $N$ -term m.s. truncation error	Not optimal in general
Computation	Requires solving a Fredholm integral equation	FFT — fast and simple
Large $T$ limit	Approaches Fourier for WSS processes	Approaches Fourier (tautologically)
Non-stationary processes	Handles naturally	Not adapted — poor convergence

Historical Note: Karhunen and Loève

1947-1960s

Kari Karhunen (1947, Finland) and Michel Loève (1948, France) independently discovered the expansion that bears their names. Karhunen was a student of Rolf Nevanlinna at the University of Helsinki, and his original paper was in Finnish — one reason the result was initially less known in the West. Loève, working in France and later at UC Berkeley, developed the expansion within his comprehensive theory of second-order processes. The KL expansion became central to communication theory through the work of David Slepian at Bell Labs, who in the 1960s computed the KL eigenfunctions for bandlimited processes (the prolate spheroidal wave functions), establishing the mathematical theory of time-frequency concentration.

,

🎓CommIT Contribution(2014)

Karhunen-Loève Channel Representation for Massive MIMO

A. Adhikary, J. Nam, G. Caire — IEEE Trans. Inf. Theory

The CommIT group used KL-type decompositions of the spatial channel covariance matrix to develop the Joint Spatial Division and Multiplexing (JSDM) framework for massive MIMO. The idea is to group users by their channel covariance eigenspaces — effectively by their KL bases — and serve each group with a pre-beamformer that projects onto the dominant eigenmodes. This two-stage beamforming approach (statistical pre-beamformer + instantaneous beamformer) achieves near-optimal massive MIMO capacity with only reduced-dimension CSI feedback. The covariance eigenmodes are precisely the spatial KL basis functions, and the eigenvalues determine how many spatial degrees of freedom each user group occupies. The JSDM framework demonstrates that the KL expansion is not merely a theoretical tool but a practical architecture for next-generation wireless systems.

KL expansionmassive MIMOJSDMcovariance eigenspaceView Paper →

🔧Engineering Note

Practical KL Truncation for Signal Compression

In practice, one truncates the KL expansion to $N$ terms, discarding eigenmodes with $\lambda_n$ below a threshold. The fraction of energy captured is $\eta_N = \sum_{n=1}^N \lambda_n / \sum_{n=1}^{\infty} \lambda_n$ . For many processes of interest (exponential autocorrelation, bandlimited processes), the eigenvalues decay rapidly, so $\eta_N > 0.99$ with $N \ll$ the nominal dimension $2WT$ . This is the principle behind transform coding, PCA-based compression, and reduced-rank signal processing. In massive MIMO, the rapid decay of spatial covariance eigenvalues means that only $r \ll N_t$ beams are needed to capture most of the channel energy.

Practical Constraints

•
Eigenvalue computation for large correlation matrices is $O(N^3)$ — Krylov methods help
•
Non-stationary processes require recomputing eigenfunctions over time

,

Quick Check

In the KL expansion of a Gaussian process, the coefficients $Z_n$ are:

Independent Gaussian random variables with $Z_n \sim \mathcal{N}(0, \lambda_n)$ .

Uncorrelated but not necessarily independent.

i.i.d. Gaussian random variables.

Correction:

Independent Gaussian random variables with

Z_n \sim \mathcal{N}(0, \lambda_n)

.

For a Gaussian process, the KL coefficients are uncorrelated (by the general KL theory) and jointly Gaussian (since they are linear functionals of a Gaussian process). Uncorrelated + jointly Gaussian = independent. Each $Z_n$ has variance $\lambda_n$ .

Common Mistake: The KL Expansion Is Not Limited to WSS Processes

Mistake:

Assuming the KL expansion applies only to wide-sense stationary processes.

Correction:

The KL expansion applies to any finite-variance process on a bounded interval $[0, T]$ , whether stationary or not. The autocorrelation kernel $R_X(t, s)$ need not be a function of $t - s$ alone. For non-stationary processes, the eigenfunctions are not sinusoids but adapted to the specific correlation structure. The WSS case is special only in that the eigenfunctions approximate Fourier exponentials for large $T$ .

Key Takeaway

The Karhunen-Loève expansion provides the optimal orthonormal representation of a random process: it diagonalizes the autocorrelation operator, yielding uncorrelated (independent for Gaussian) coefficients ordered by decreasing variance. The $N$ -term KL approximation minimizes the mean-square truncation error among all $N$ -dimensional projections — this is PCA for random processes. For WSS processes on large intervals, the KL basis converges to the Fourier basis and the eigenvalues converge to the PSD, unifying the spectral and KL viewpoints.

The Karhunen-Loève Expansion

The Optimal Basis for a Random Process

Definition: The Karhunen-Loève Expansion

Karhunen-Loève Expansion

Fredholm Integral Equation

Principal Component Analysis (PCA)

Theorem: Optimality of the KL Expansion (Minimum Mean-Square Truncation Error)

Express the total energy

Compute the truncation error

Prove optimality

Example: KL Expansion of the Wiener Process on [0,T][0, T][0,T]

Set up the Fredholm equation

Convert to a differential equation

Solve the ODE

Write the eigenvalues and eigenfunctions

Karhunen-Loève Expansion: Convergence vs. Number of Terms

Parameters

Karhunen-Loève Basis Functions Building Up a Random Realization

The KL Expansion in Detection Theory

KL Expansion vs. Fourier Series

Definition: Mercer's Theorem

Definition: KL Expansion of WSS Processes on Large Intervals

Karhunen-Loève vs. Fourier Expansion

Historical Note: Karhunen and Loève

Karhunen-Loève Channel Representation for Massive MIMO

Practical KL Truncation for Signal Compression

Quick Check

Common Mistake: The KL Expansion Is Not Limited to WSS Processes

Key Takeaway

Definition:
The Karhunen-Loève Expansion

Example: KL Expansion of the Wiener Process on $[0, T]$

Definition:
Mercer's Theorem

Definition:
KL Expansion of WSS Processes on Large Intervals