Ferkans — Interactive Telecom Tutor

Definition:
Gaussian Process

A real-valued stochastic process $\{X(t) : t \in \mathcal{T}\}$ is a Gaussian process if for every $N \geq 1$ and every choice of time instants $t_1, t_2, \ldots, t_N \in \mathcal{T}$ , the random vector $\mathbf{X} = \begin{bmatrix} X(t_1) \\ X(t_2) \\ \vdots \\ X(t_N) \end{bmatrix}$ has a joint Gaussian distribution $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ , where $[\boldsymbol{\mu}]_k = \mathbb{E}[X(t_k)]$ and $[\boldsymbol{\Sigma}]_{k,\ell} = \text{Cov}(X(t_k), X(t_\ell))$ .

Equivalently, every finite linear combination $\sum_{k=1}^N a_k X(t_k)$ is a Gaussian random variable for all $a_1, \ldots, a_N \in \mathbb{R}$ .

The equivalence follows because $\sum_k a_k X(t_k) = \mathbf{a}^\mathsf{T} \mathbf{X}$ , and a random vector is jointly Gaussian if and only if every linear combination of its components is Gaussian.

Complete Characterization by First and Second Moments

A Gaussian distribution is uniquely determined by its mean and covariance. Therefore, a Gaussian process $\{X(t)\}$ is completely characterized by two functions:

Mean function: $\mu_X(t) = \mathbb{E}[X(t)]$
Covariance function: $C_X(t_1, t_2) = \text{Cov}(X(t_1), X(t_2)) = \mathbb{E}[(X(t_1) - \mu_X(t_1))(X(t_2) - \mu_X(t_2))]$

This is a remarkable simplification: while a general process requires the full hierarchy of finite-dimensional distributions, for a Gaussian process the mean and covariance functions tell us everything.

Theorem: Closure of Gaussian Processes Under Linear Operations

Let $\{X(t)\}$ be a Gaussian process and let $Y = \int_a^b g(t) X(t)\,dt$ for a deterministic function $g(t)$ (where the integral exists in mean square). Then $Y$ is a Gaussian random variable.

More generally, if $\{X(t)\}$ is Gaussian and $Y(s) = \int g(s,t) X(t)\,dt$ , then $\{Y(s)\}$ is also a Gaussian process.

Gaussian distributions are closed under linear operations (sums, integrals, limits in mean square). Since an integral is a limit of finite sums, and each finite sum of Gaussian RVs is Gaussian, the limit inherits Gaussianity.

Proof

Approximate the integral by a Riemann sum

Partition $[a,b]$ into $N$ subintervals. The Riemann sum $Y_N = \sum_{k=1}^N g(t_k) X(t_k) \Delta t_k$ is a finite linear combination of jointly Gaussian RVs, hence $Y_N \sim \mathcal{N}(\mu_{Y_N}, \sigma_{Y_N}^2)$ .

Take the mean-square limit

As $N \to \infty$ , $Y_N \xrightarrow{\text{m.s.}} Y$ . A mean-square limit of Gaussian RVs is Gaussian: the characteristic function $\phi_{Y_N}(u) = \exp(j u \mu_{Y_N} - \tfrac{1}{2} u^2 \sigma_{Y_N}^2)$ converges pointwise to $\phi_Y(u)$ , which is also Gaussian. $\blacksquare$

Definition:
Properties of the Covariance Function

A function $C : \mathcal{T} \times \mathcal{T} \to \mathbb{R}$ is a valid covariance function if and only if it satisfies:

Symmetry: $C(t_1, t_2) = C(t_2, t_1)$
Positive semi-definiteness: For all $N$ , all $t_1, \ldots, t_N$ , and all $a_1, \ldots, a_N \in \mathbb{R}$ , $\sum_{k=1}^N \sum_{\ell=1}^N a_k a_\ell\, C(t_k, t_\ell) \geq 0$

Any function satisfying these two conditions defines a unique Gaussian process (up to the choice of mean function), by the Kolmogorov extension theorem.

Condition 2 is equivalent to requiring that every finite-dimensional covariance matrix $\boldsymbol{\Sigma}$ be positive semi-definite.

Example: Is $X(t) = A\cos(2\pi f_0 t + \Theta)$ a Gaussian Process?

Let $A \sim \mathcal{N}(0, \sigma^2)$ and $\Theta$ uniform on $[0, 2\pi)$ , with $A$ and $\Theta$ independent. Determine whether $X(t) = A\cos(2\pi f_0 t + \Theta)$ is a Gaussian process.

Solution

Check the definition

For fixed $t$ , $X(t) = A \cos(2\pi f_0 t + \Theta)$ . Since $\Theta$ is uniform and independent of $A$ , the product $A \cos(2\pi f_0 t + \Theta)$ is not Gaussian in general — the uniform phase $\Theta$ introduces nonlinearity.

Consider two time instants

Take $t_1, t_2$ . Then $\mathbf{X} = A[\cos(\cdots + \Theta), \cos(\cdots + \Theta)]^\mathsf{T}$ . The components are perfectly dependent through $A$ and $\Theta$ . Their joint distribution is not Gaussian — it lives on a random curve in $\mathbb{R}^2$ .

Conclusion

$X(t) = A\cos(2\pi f_0 t + \Theta)$ is not a Gaussian process, even though it is WSS and its marginal distribution may look "bell-shaped" when averaged over $\Theta$ . The failure is that the joint distributions are not Gaussian.

Example: Gaussian Process from Random Fourier Coefficients

Let $A_k \sim \mathcal{N}(0, \sigma_k^2)$ for $k = 1, \ldots, K$ be independent Gaussian RVs. Show that $X(t) = \sum_{k=1}^K A_k \phi_k(t)$ , where $\phi_k(t)$ are deterministic functions, is a Gaussian process. Compute its mean and covariance.

Solution

Verify Gaussianity

For any times $t_1, \ldots, t_N$ , the vector $[X(t_1), \ldots, X(t_N)]^\mathsf{T}$ is a linear transformation of the Gaussian vector $[A_1, \ldots, A_K]^\mathsf{T}$ . Linear transformations of Gaussian vectors are Gaussian, so $\{X(t)\}$ is a GP.

Mean function

$\mu_X(t) = \mathbb{E}[X(t)] = \sum_{k=1}^K \mathbb{E}[A_k]\,\phi_k(t) = 0$ since each $A_k$ has zero mean.

Covariance function

$C_X(t_1, t_2) = \mathbb{E}[X(t_1)X(t_2)] = \sum_{k=1}^K \sigma_k^2 \phi_k(t_1)\phi_k(t_2)$

using independence: $\mathbb{E}[A_k A_\ell] = \sigma_k^2 \delta_{k\ell}$ .

Theorem: WSS Gaussian Process Is Strictly Stationary

If $\{X(t)\}$ is a Gaussian process that is wide-sense stationary (WSS) — i.e., $\mu_X(t) = \mu$ is constant and $r_{xx}(\tau) = \mathbb{E}[X(t+\tau)X^*(t)]$ depends only on $\tau$ — then $\{X(t)\}$ is strictly stationary.

A Gaussian distribution is completely determined by its mean and covariance. If these are time-shift invariant (WSS), then all finite-dimensional distributions are time-shift invariant, which is the definition of strict stationarity. For non-Gaussian processes, WSS does not imply strict stationarity because higher-order moments could still depend on absolute time.

Proof

Write the finite-dimensional distributions

For any $t_1, \ldots, t_N$ , the vector $\mathbf{X} = [X(t_1), \ldots, X(t_N)]^\mathsf{T}$ is $\mathcal{N}(\mu\mathbf{1}, \boldsymbol{\Sigma})$ where $[\boldsymbol{\Sigma}]_{k,\ell} = C_X(t_k - t_\ell) = r_{xx}(t_k - t_\ell) - |\mu|^2$ .

Apply a time shift

Shift all times by $s$ : the vector $[X(t_1+s), \ldots, X(t_N+s)]^\mathsf{T}$ is $\mathcal{N}(\mu\mathbf{1}, \boldsymbol{\Sigma}')$ with $[\boldsymbol{\Sigma}']_{k,\ell} = C_X((t_k+s) - (t_\ell+s)) = C_X(t_k - t_\ell) = [\boldsymbol{\Sigma}]_{k,\ell}$ .

Conclude

Since $\boldsymbol{\Sigma}' = \boldsymbol{\Sigma}$ and the mean is unchanged, the two distributions are identical. This holds for all $N$ , all $\{t_k\}$ , and all $s$ , so $\{X(t)\}$ is strictly stationary. $\blacksquare$

,

Common Mistake: WSS Does Not Imply Strict Stationarity in General

Mistake:

Assuming that any WSS process is strictly stationary.

Correction:

This implication holds only for Gaussian processes. For a general process, WSS constrains only the first two moments; higher-order statistics (skewness, kurtosis, etc.) could still vary with time. A standard counterexample: let $X[n]$ be i.i.d. Bernoulli( $\frac{1}{2}$ ) taking values $\pm 1$ . This is WSS (constant mean 0, $r_{xx}[m] = \delta[m]$ ) and also strictly stationary. But replace $X[0]$ with a non-symmetric distribution keeping the same mean and variance — the process is still WSS but no longer strictly stationary.

Sample Paths of a Gaussian Process

Draw sample paths from a zero-mean GP with different covariance kernels. Observe how the kernel controls smoothness, correlation length, and sample variability.

Parameters

Covariance kernel

\ell

(length scale)1

\sigma^2

(signal variance)1

Number of sample paths5

Quick Check

A Gaussian process is completely characterized by which of the following?

Its mean function only

Its mean function and covariance function

All of its finite-dimensional PDFs

Its autocorrelation function and PSD

Correction:

Its mean function and covariance function

A Gaussian distribution is uniquely determined by its mean and covariance. Since all finite-dimensional distributions of a GP are Gaussian, specifying the mean and covariance functions determines everything.

Gaussian Process

A stochastic process $\{X(t)\}$ such that every finite-dimensional marginal $[X(t_1), \ldots, X(t_N)]$ is jointly Gaussian. Completely determined by its mean and covariance functions.

Covariance Function (Kernel)

A function $C(t_1, t_2) = \text{Cov}(X(t_1), X(t_2))$ that must be symmetric and positive semi-definite. For a WSS process, $C(t_1, t_2) = C(t_1 - t_2)$ depends only on the time lag.

Positive Semi-Definite (Function)

A function $C(\tau)$ is positive semi-definite if for all $N$ , all $t_1, \ldots, t_N$ , and all $a_1, \ldots, a_N$ , we have $\sum_{k,\ell} a_k a_\ell C(t_k - t_\ell) \geq 0$ . Equivalently, its Fourier transform (the PSD) is nonnegative.

Related: Covariance Function (Kernel)

Historical Note: Kolmogorov and the Foundation of Stochastic Processes

1930s

Andrey Kolmogorov's 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung (Foundations of Probability Theory) placed probability on a rigorous measure-theoretic footing. His extension theorem guarantees the existence of a stochastic process with any consistent family of finite-dimensional distributions. For Gaussian processes, this means that specifying a mean and a positive semi-definite covariance function is sufficient to construct the process — a fact we use freely throughout this chapter.

Key Takeaway

A Gaussian process is the simplest infinite-dimensional generalization of the multivariate Gaussian distribution: two functions — the mean $\mu_X(t)$ and the covariance $C_X(t_1, t_2)$ — completely determine all statistical properties. This is why Gaussian models dominate engineering: they are rich enough to model complex phenomena yet tractable enough for closed-form analysis.

Definition and Finite-Dimensional Distributions

Definition: Gaussian Process