Ferkans — Interactive Telecom Tutor

From Scalars to Vectors

In Chapters 5–7, we studied individual random variables and pairs $(X, Y)$ . But in most engineering applications, we observe not a single measurement but a collection of measurements simultaneously. A MIMO receiver observes $n$ antenna outputs; an estimator processes a vector of samples; a stochastic process evaluated at $n$ time instants yields a random vector.

The natural mathematical object is the random vector $\mathbf{X} = (X_1, \ldots, X_n)^T$ , and the natural summary statistics are the mean vector and covariance matrix. This section establishes the vocabulary and the key structural result: the covariance matrix is always positive semi-definite.

Definition:
Random Vector

A random vector is an ordered collection of $n$ random variables defined on the same probability space $(\Omega, \mathcal{F}, \mathbb{P})$ :

$\mathbf{X} = \begin{pmatrix} X_1 \\ X_2 \\ \vdots \\ X_n \end{pmatrix}.$

The joint PDF (when it exists) is $f_{\mathbf{X}}(\mathbf{x})$ such that $\mathbb{P}(\mathbf{X} \in A) = \int_A f_{\mathbf{X}}(\mathbf{x})\,d\mathbf{x}$ for any (measurable) set $A \subseteq \mathbb{R}^n$ .

Random vector

An ordered collection $\mathbf{X} = (X_1, \ldots, X_n)^T$ of random variables on the same probability space. Completely characterized by the family of finite-dimensional distributions.

Related: Covariance matrix

Definition:
Mean Vector and Covariance Matrix

Let $\mathbf{X} = (X_1, \ldots, X_n)^T$ be a random vector with finite second moments. The mean vector is

$\boldsymbol{\mu} = \mathbb{E}[\mathbf{X}] = \begin{pmatrix} \mathbb{E}[X_1] \\ \vdots \\ \mathbb{E}[X_n] \end{pmatrix}.$

The covariance matrix is the $n \times n$ matrix

$\boldsymbol{\Sigma} = \mathbb{E}\bigl[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^T\bigr],$

whose $(i,j)$ -entry is $\text{Cov}(X_i, X_j)$ . The correlation matrix is $\mathbf{R} = \mathbb{E}[\mathbf{X}\mathbf{X}^T] = \boldsymbol{\Sigma} + \boldsymbol{\mu}\boldsymbol{\mu}^T$ .

The diagonal entries of $\boldsymbol{\Sigma}$ are the variances $\text{Var}(X_i)$ , and the off-diagonal entries are the covariances $\text{Cov}(X_i, X_j)$ .

Covariance matrix

The matrix $\boldsymbol{\Sigma} = \mathbb{E}[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^T]$ summarizing all pairwise covariances of a random vector. Always symmetric and positive semi-definite.

Theorem: Covariance Matrices Are Positive Semi-Definite

For any random vector $\mathbf{X}$ with finite second moments, the covariance matrix $\boldsymbol{\Sigma}$ is symmetric and positive semi-definite:

$\mathbf{a}^T \boldsymbol{\Sigma} \, \mathbf{a} \geq 0 \quad \text{for all } \mathbf{a} \in \mathbb{R}^n.$

Moreover, $\boldsymbol{\Sigma}$ is strictly positive definite if and only if no non-trivial linear combination $\mathbf{a}^T \mathbf{X}$ is a constant (almost surely).

The quadratic form $\mathbf{a}^T \boldsymbol{\Sigma} \, \mathbf{a}$ equals $\text{Var}(\mathbf{a}^T \mathbf{X})$ , and variance is always non-negative.

Proof

Express as a variance

Let $Z = \mathbf{a}^T \mathbf{X}$ . Then

$\text{Var}(Z) = \mathbb{E}[(Z - \mathbb{E}[Z])^2] = \mathbb{E}\bigl[\mathbf{a}^T (\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^T \mathbf{a}\bigr] = \mathbf{a}^T \boldsymbol{\Sigma} \, \mathbf{a}.$

Non-negativity

Since $\text{Var}(Z) \geq 0$ for any random variable $Z$ , we conclude $\mathbf{a}^T \boldsymbol{\Sigma} \, \mathbf{a} \geq 0$ for all $\mathbf{a} \in \mathbb{R}^n$ .

Strict positivity condition

Equality $\mathbf{a}^T \boldsymbol{\Sigma} \, \mathbf{a} = 0$ holds iff $\text{Var}(\mathbf{a}^T \mathbf{X}) = 0$ , i.e., $\mathbf{a}^T \mathbf{X}$ is a constant a.s. If no non-trivial linear combination is constant, then $\boldsymbol{\Sigma} \succ 0$ .

Positive semi-definite (PSD)

A symmetric matrix $\mathbf{A}$ is PSD ( $\mathbf{A} \succeq 0$ ) if $\mathbf{x}^T \mathbf{A} \mathbf{x} \geq 0$ for all $\mathbf{x}$ . Equivalently, all eigenvalues of $\mathbf{A}$ are non-negative.

Related: Covariance matrix

Common Mistake: Covariance Matrix vs. Correlation Matrix

Mistake:

Confusing the covariance matrix $\boldsymbol{\Sigma}$ with the correlation matrix $\mathbf{R} = \mathbb{E}[\mathbf{X}\mathbf{X}^T]$ .

Correction:

They differ by a rank-one term: $\mathbf{R} = \boldsymbol{\Sigma} + \boldsymbol{\mu}\boldsymbol{\mu}^T$ . They coincide only when $\boldsymbol{\mu} = \mathbf{0}$ . In signal processing, $\mathbf{R}$ includes the "DC component" while $\boldsymbol{\Sigma}$ does not.

Example: Covariance Matrix of a Bivariate Distribution

Let $\mathbf{X} = (X_1, X_2)^T$ with $\mathbb{E}[X_1] = 1$ , $\mathbb{E}[X_2] = -2$ , $\text{Var}(X_1) = 4$ , $\text{Var}(X_2) = 9$ , and $\text{Cov}(X_1, X_2) = -3$ . Write the covariance matrix and verify that it is PSD.

Solution

Write the matrix

$\boldsymbol{\Sigma} = \begin{pmatrix} 4 & -3 \\ -3 & 9 \end{pmatrix}.$ $

Check PSD via eigenvalues

The eigenvalues satisfy $\lambda^2 - 13\lambda + 27 = 0$ , giving $\lambda_1 = \frac{13 - \sqrt{133}}{2} \approx 2.74$ and $\lambda_2 \approx 10.26$ . Both are positive, so $\boldsymbol{\Sigma} \succ 0$ .

Alternative check

$\det(\boldsymbol{\Sigma}) = 36 - 9 = 27 > 0$ and the diagonal entries are positive, confirming positive definiteness.

Cross-Covariance Matrix

For two random vectors $\mathbf{X} \in \mathbb{R}^m$ and $\mathbf{Y} \in \mathbb{R}^n$ , the cross-covariance matrix is the $m \times n$ matrix

$\boldsymbol{\Sigma}_{xy} = \mathbb{E}\bigl[(\mathbf{X} - \boldsymbol{\mu}_x)(\mathbf{Y} - \boldsymbol{\mu}_y)^T\bigr].$

Notice that $\boldsymbol{\Sigma}_{yx} = \boldsymbol{\Sigma}_{xy}^{T}$ . The cross-covariance measures the linear dependence between $\mathbf{X}$ and $\mathbf{Y}$ and plays a central role in LMMSE estimation (Book FSI, Chapter 3).

Why This Matters: Covariance Matrices in MIMO Channel Modeling

In a MIMO system with $N_t$ transmit and $N_r$ receive antennas, the received signal vector $\mathbf{y} \in \mathbb{C}^{N_r}$ has a covariance matrix that encodes the spatial correlation structure of the channel and noise. The transmit covariance $\boldsymbol{\Sigma}_{t} = \mathbb{E}[\mathbf{x}\mathbf{x}^H]$ is the design variable in capacity-achieving precoding (water-filling over the eigenmodes of $\mathbf{H}\mathbf{H}^H$ ). The receive spatial correlation $\boldsymbol{\Sigma}_{r}$ determines how much diversity the channel offers. All of massive MIMO analysis rests on the covariance matrix structure developed in this chapter.

Random Vectors and Their Statistics