Ferkans — Interactive Telecom Tutor

The Central Role of Autocorrelation

For a WSS process, the autocorrelation function $r_{XX}(\tau)$ encodes how the process at time $t$ relates to itself at time $t + \tau$ . This single function determines the average power ( $r_{XX}(0)$ ), the rate of fluctuation (how fast $r_{XX}(\tau)$ decays), the bandwidth (via the power spectral density, which is the Fourier transform of $r_{XX}$ — Chapter 14), and the performance of optimal linear filters. Understanding the properties of $r_{XX}$ is the gateway to spectral analysis and filter design.

Definition:
Autocorrelation Function

For a second-order process $\{X(t) : t \in \mathcal{T}\}$ , the autocorrelation function is $r_{XX}(t_1, t_2) = \mathbb{E}[X(t_1)X^*(t_2)].$

If $X(t)$ is WSS, then $r_{XX}(t_1, t_2) = r_{XX}(t_1 - t_2) \triangleq r_{XX}(\tau)$ , where $\tau = t_1 - t_2$ .

For a discrete-time WSS process: $r_{xx}[k] = \mathbb{E}[X_n X_{n-k}^*]$ (independent of $n$ ).

Definition:
Autocovariance Function

The autocovariance function of a second-order process is $c_{XX}(t_1, t_2) = \text{Cov}(X(t_1), X(t_2)) = r_{XX}(t_1, t_2) - \mu_X(t_1)\mu_X^*(t_2).$

For a WSS process with mean $\mu$ : $c_{XX}(\tau) = r_{XX}(\tau) - |\mu|^2.$

The autocovariance measures the fluctuation around the mean. If $X(t)$ has zero mean, then $c_{XX}(\tau) = r_{XX}(\tau)$ .

Theorem: Properties of the WSS Autocorrelation

Let $\{X(t)\}$ be a WSS process with autocorrelation $r_{XX}(\tau)$ . Then:

Average power: $r_{XX}(0) = \mathbb{E}[|X(t)|^2] \geq 0$ .
Hermitian symmetry: $r_{XX}(\tau) = r_{XX}^*(-\tau)$ . For real-valued processes: $r_{XX}(\tau) = r_{XX}(-\tau)$ (even function).
Maximum at the origin: $|r_{XX}(\tau)| \leq r_{XX}(0)$ for all $\tau$ .
Non-negative definiteness: For any $N$ , any $(t_1, \ldots, t_N)$ , and any $(a_1, \ldots, a_N) \in \mathbb{C}^N$ : $\sum_{i=1}^{N}\sum_{j=1}^{N} a_i a_j^* r_{XX}(t_i - t_j) \geq 0.$

Property 3 says that $X(t)$ is most correlated with itself (at lag zero). This is intuitive: the best predictor of a signal is the signal itself, and correlation decreases as you look further into the past or future. Property 4 ensures that the variance of any linear combination $\sum a_i X(t_i)$ is non-negative.

Proof

Property 1: Average power

$r_{XX}(0) = \mathbb{E}[X(t)X^*(t)] = \mathbb{E}[|X(t)|^2] \geq 0$ .

Property 2: Hermitian symmetry

$r_{XX}(\tau) = \mathbb{E}[X(t)X^*(t-\tau)]$ . Replacing $t$ by $t - \tau$ (valid since the mean is constant): $r_{XX}(-\tau) = \mathbb{E}[X(t-\tau)X^*(t)]$ . Taking the conjugate: $r_{XX}^*(-\tau) = \mathbb{E}[X^*(t-\tau)X(t)] = \mathbb{E}[X(t)X^*(t-\tau)] = r_{XX}(\tau)$ .

Property 3: Maximum at origin

By the Cauchy-Schwarz inequality: $|r_{XX}(\tau)| = |\mathbb{E}[X(t)X^*(t-\tau)]| \leq \sqrt{\mathbb{E}[|X(t)|^2]\,\mathbb{E}[|X(t-\tau)|^2]} = \sqrt{r_{XX}(0) \cdot r_{XX}(0)} = r_{XX}(0).$

Property 4: Non-negative definiteness

Let $W = \sum_{i=1}^{N} a_i X(t_i)$ . Then: $0 \leq \mathbb{E}[|W|^2] = \mathbb{E}\left[\sum_i a_i X(t_i) \sum_j a_j^* X^*(t_j)\right] = \sum_{i,j} a_i a_j^* r_{XX}(t_i - t_j).$

Definition:
Non-Negative Definite Function (Definition 50)

A Hermitian symmetric function $f : \mathcal{T} \times \mathcal{T} \to \mathbb{C}$ (i.e., $f(t_1, t_2) = f^*(t_2, t_1)$ ) is positive semi-definite (or non-negative definite) if for all $N$ , all $(t_1, \ldots, t_N)$ , and all $(a_1, \ldots, a_N) \in \mathbb{C}^N$ : $\sum_{i=1}^{N}\sum_{j=1}^{N} a_i a_j^* f(t_i, t_j) \geq 0.$

For a WSS function depending only on $\tau = t_1 - t_2$ , we say $r(\tau)$ is positive semi-definite if $\sum_{i,j} a_i a_j^* r(t_i - t_j) \geq 0$ for all choices.

The autocorrelation of any process is positive semi-definite. Conversely, every positive semi-definite function is the autocorrelation of some process. This is the Herglotz (or Bochner) theorem, which we will see in Chapter 14 in connection with the power spectral density.

Example: Exponential Autocorrelation

A real-valued WSS process has autocorrelation $r_{XX}(\tau) = \sigma^2 e^{-\alpha|\tau|}$ with $\alpha > 0$ . Verify the four properties of the autocorrelation and find the average power and autocovariance (assuming $\mu = 0$ ).

Solution

Average power

$r_{XX}(0) = \sigma^2 e^0 = \sigma^2$ . The average power is $\sigma^2$ .

Even symmetry

$r_{XX}(-\tau) = \sigma^2 e^{-\alpha|-\tau|} = \sigma^2 e^{-\alpha|\tau|} = r_{XX}(\tau)$ . ✓

Maximum at origin

$|r_{XX}(\tau)| = \sigma^2 e^{-\alpha|\tau|} \leq \sigma^2 = r_{XX}(0)$ for all $\tau$ . ✓

Non-negative definiteness

This can be verified by showing that the Fourier transform (the PSD) is non-negative: $P_X(f) = \int_{-\infty}^{\infty} \sigma^2 e^{-\alpha|\tau|} e^{-j2\pi f\tau}\,d\tau = \frac{2\alpha\sigma^2}{\alpha^2 + (2\pi f)^2} > 0.$ Since $P_X(f) \geq 0$ , the function is positive semi-definite (Bochner's theorem). ✓

Autocovariance

Since $\mu = 0$ : $c_{XX}(\tau) = r_{XX}(\tau) - |\mu|^2 = \sigma^2 e^{-\alpha|\tau|}$ .

Autocorrelation Properties

Explore the properties of the autocorrelation function for different WSS processes. Observe the even symmetry, the maximum at the origin, and how the decay rate relates to the "memory" of the process.

Parameters

Autocorrelation Shape

\sigma^2

(power)1

\alpha

(decay rate)1

Theorem: Toeplitz Covariance Matrix of WSS Processes

Let $\{X_n\}$ be a WSS process. For any block of $N$ consecutive samples $\mathbf{X} = (X_\ell, X_{\ell+1}, \ldots, X_{\ell+N-1})^\top$ , the covariance matrix is $[\boldsymbol{\Sigma}]_{ij} = c_{xx}[i - j], \quad i, j = 0, \ldots, N-1,$ which is Toeplitz (constant along diagonals) and independent of the starting index $\ell$ .

The Toeplitz structure is the matrix manifestation of WSS: the covariance between $X_i$ and $X_j$ depends only on the distance $|i - j|$ , not on the absolute position. This structure is exploited in fast algorithms (Levinson-Durbin) and in the asymptotic analysis of the eigenvalue distribution (Szegő's theorem).

Proof

Direct computation

$[\boldsymbol{\Sigma}]_{ij} = \text{Cov}(X_{\ell+i}, X_{\ell+j}) = c_{xx}[\ell+i, \ell+j] = c_{xx}[(\ell+i) - (\ell+j)] = c_{xx}[i-j]$ . This depends only on $i - j$ , so $\boldsymbol{\Sigma}$ is Toeplitz. The starting index $\ell$ cancels.

Example: Autocorrelation of a Moving Average Process

Let $\{W_n\}$ be a zero-mean i.i.d. sequence with variance $\sigma_W^2$ . Define the moving average (MA) process $X_n = \frac{1}{3}(W_{n-1} + W_n + W_{n+1})$ . Find the autocorrelation $r_{xx}[k]$ and verify that $X_n$ is WSS.

Solution

Mean

$\mu_X = \mathbb{E}[X_n] = \frac{1}{3}(\mathbb{E}[W_{n-1}] + \mathbb{E}[W_n] + \mathbb{E}[W_{n+1}]) = 0$ (constant).

Autocorrelation

$r_{xx}[k] = \mathbb{E}[X_n X_{n-k}] = \frac{1}{9}\sum_{i=-1}^{1}\sum_{j=-1}^{1}\mathbb{E}[W_{n+i}W_{n-k+j}].KATEXPLACEHOLDER0ENDr_{xx}[k] = \frac{\sigma_W^2}{9}\sum_{i=-1}^{1}\sum_{j=-1}^{1}\delta[k + i - j] = \frac{\sigma_W^2}{9} \cdot \#\{(i,j) : i - j = -k,\; i,j \in \{-1,0,1\}\}.$ $

Evaluate for each lag

$k = 0$ : pairs $(i,j)$ with $i = j$ — three pairs. $r_{xx}[0] = 3\sigma_W^2/9 = \sigma_W^2/3$ .
$|k| = 1$ : pairs with $j - i = 1$ — two pairs $((-1,0), (0,1))$ . $r_{xx}[\pm 1] = 2\sigma_W^2/9$ .
$|k| = 2$ : one pair $((-1,1))$ . $r_{xx}[\pm 2] = \sigma_W^2/9$ .
$|k| \geq 3$ : no pairs. $r_{xx}[k] = 0$ .

Since $r_{xx}[k]$ depends only on $|k|$ and the mean is constant, $\{X_n\}$ is WSS.

Common Mistake: Confusing Autocorrelation with Autocovariance

Mistake:

Using $r_{XX}(\tau)$ and $c_{XX}(\tau)$ interchangeably, especially for processes with nonzero mean.

Correction:

$r_{XX}(\tau) = \mathbb{E}[X(t)X^*(t-\tau)]$ includes the mean product, while $c_{XX}(\tau) = r_{XX}(\tau) - |\mu|^2$ measures only the fluctuations. For zero-mean processes they coincide, but for $\mu \neq 0$ they differ. Many spectral analysis formulas use $r_{XX}$ while estimation formulas use $c_{XX}$ .

Historical Note: Norbert Wiener and the Autocorrelation Function

1930s--1940s

Norbert Wiener introduced the systematic use of the autocorrelation function in his 1930 paper "Generalized Harmonic Analysis" and his classified 1942 report on fire-control prediction (later published as Extrapolation, Interpolation, and Smoothing of Stationary Time Series, 1949). Wiener recognized that for stationary processes, the autocorrelation function and its Fourier transform (the power spectral density) provide a complete framework for linear prediction and filtering. His work, independently paralleled by Kolmogorov (1941), laid the foundation for all of modern statistical signal processing.

Historical Note: Khintchine and the Positive-Definiteness Connection

1930s

Aleksandr Khintchine (1934) proved that the autocorrelation function of a stationary process is positive semi-definite, and conversely that every continuous positive semi-definite function is the autocorrelation of some stationary process. This result, combined with Bochner's theorem on positive-definite functions and Fourier transforms, established the rigorous link between autocorrelation and power spectral density — the Wiener-Khintchine theorem (Chapter 14).

Quick Check

For a real-valued zero-mean WSS process with $r_{XX}(0) = 4$ and $r_{XX}(3) = -2$ , what is $r_{XX}(-3)$ ?

$2$

$-2$

$4$

Correction:

-2

For a real-valued WSS process, $r_{XX}(\tau) = r_{XX}(-\tau)$ (even symmetry).

🔧Engineering Note

Estimating the Autocorrelation from Data

In practice, we estimate $r_{XX}(\tau)$ from a single observed sample path $\{x(t_0), x(t_1), \ldots, x(t_{N-1})\}$ . The standard estimator is $\hat{r}_{xx}[k] = \frac{1}{N}\sum_{n=0}^{N-1-|k|} x[n+|k|]\,x^*[n], \quad |k| < N.$ This is the biased estimator (dividing by $N$ rather than $N - |k|$ ). The biased version is preferred because it guarantees a non-negative PSD estimate, while the unbiased version ( $1/(N - |k|)$ ) can produce negative PSD values.

Autocorrelation Function

$r_{XX}(\tau) = \mathbb{E}[X(t)X^*(t - \tau)]$ for a WSS process. Measures the linear dependence between the process at two time instants separated by lag $\tau$ .

Autocovariance Function

$c_{XX}(\tau) = r_{XX}(\tau) - |\mu|^2$ . The autocorrelation of the zero-mean part of the process.

Related: Autocorrelation Function

Key Takeaway

The autocorrelation function $r_{XX}(\tau)$ of a WSS process is even (Hermitian symmetric), maximized at the origin (where it equals the average power), and non-negative definite. These properties are not just mathematical curiosities — they ensure that the power spectral density (its Fourier transform) is real and non-negative, which is the basis for all spectral analysis in Chapter 14.

Autocorrelation and Autocovariance