Ferkans — Interactive Telecom Tutor

Eigenvalues: The DNA of a Matrix

Eigenvalue decomposition reveals the intrinsic geometry of a linear transformation — its principal axes and scaling factors. In wireless communications, the eigenvalues of the channel correlation matrix determine the spatial degrees of freedom, the singular values of the channel matrix determine the capacity of each spatial stream, and PCA (principal component analysis) is eigendecomposition of the sample covariance.

Definition:
Eigenvalue Decomposition

A square matrix $\mathbf{A} \in \mathbb{C}^{n \times n}$ has eigenvalue decomposition $\mathbf{A}\mathbf{v} = \lambda \mathbf{v}$ , where $\lambda$ is an eigenvalue and $\mathbf{v}$ is the corresponding eigenvector. If $\mathbf{A}$ is diagonalizable:

$\mathbf{A} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{-1}$

where $\boldsymbol{\Lambda} = \mathrm{diag}(\lambda_1, \ldots, \lambda_n)$ .

eigenvalues, V = np.linalg.eig(A)       # general matrix
eigenvalues, V = np.linalg.eigh(A)      # Hermitian/symmetric

eigh is preferred when $\mathbf{A} = \mathbf{A}^H$ because it guarantees real eigenvalues, orthogonal eigenvectors, and is $3\times$ faster than eig (exploits symmetry via the divide-and-conquer algorithm).

Definition:
Singular Value Decomposition (SVD)

Any matrix $\mathbf{A} \in \mathbb{C}^{m \times n}$ (not necessarily square) admits a factorization:

$\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^H$

where $\mathbf{U} \in \mathbb{C}^{m \times m}$ (left singular vectors), $\boldsymbol{\Sigma} \in \mathbb{R}^{m \times n}$ (singular values on the diagonal), and $\mathbf{V} \in \mathbb{C}^{n \times n}$ (right singular vectors).

U, s, Vh = np.linalg.svd(A, full_matrices=True)   # full SVD
U, s, Vh = np.linalg.svd(A, full_matrices=False)  # reduced SVD

The reduced SVD returns $\mathbf{U} \in \mathbb{C}^{m \times r}$ , $\boldsymbol{\sigma} \in \mathbb{R}^r$ , $\mathbf{V}^H \in \mathbb{C}^{r \times n}$ where $r = \min(m, n)$ .

The singular values are always real and non-negative: $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$ . The eigenvalues of $\mathbf{A}^H\mathbf{A}$ are $\sigma_i^2$ .

Definition:
Condition Number

The condition number of a matrix (in the 2-norm) is the ratio of its largest to smallest singular value:

$\kappa_2(\mathbf{A}) = \frac{\sigma_{\max}}{\sigma_{\min}}$

kappa = np.linalg.cond(A)
# or equivalently:
s = np.linalg.svd(A, compute_uv=False)
kappa = s[0] / s[-1]

A matrix with $\kappa \approx 1$ is well-conditioned (stable to solve). A matrix with $\kappa \gg 1$ is ill-conditioned (solutions are sensitive to perturbations). For double precision, roughly $\log_{10}(\kappa)$ digits of accuracy are lost in solving $\mathbf{Ax} = \mathbf{b}$ .

Definition:
Numerical Rank

The numerical rank of a matrix is the number of singular values above a threshold $\epsilon$ :

$\mathrm{rank}_\epsilon(\mathbf{A}) = |\{i : \sigma_i > \epsilon\}|$

In NumPy:

rank = np.linalg.matrix_rank(A)          # default tolerance
rank = np.linalg.matrix_rank(A, tol=1e-6)  # custom tolerance

The default tolerance is $\sigma_{\max} \cdot \max(m, n) \cdot \epsilon_{\mathrm{mach}}$ .

Theorem: Eckart-Young-Mirsky Theorem (Best Low-Rank Approximation)

The best rank- $k$ approximation to $\mathbf{A}$ in the Frobenius (or spectral) norm is obtained by keeping only the $k$ largest singular values:

$\mathbf{A}_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^H$

The approximation error is: $\|\mathbf{A} - \mathbf{A}_k\|_F = \sqrt{\sigma_{k+1}^2 + \cdots + \sigma_r^2}$

The SVD sorts the "information content" of a matrix by singular value. Truncating at rank $k$ keeps the $k$ most important components and discards the rest. This is the mathematical basis for PCA, image compression, and low-rank channel approximations.

Theorem: Spectral Theorem for Hermitian Matrices

A Hermitian matrix $\mathbf{A} = \mathbf{A}^H$ has:

All real eigenvalues
Orthogonal eigenvectors: $\mathbf{V}^H\mathbf{V} = \mathbf{I}$
Eigendecomposition $\mathbf{A} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^H$

For a positive semidefinite (PSD) matrix, all eigenvalues are $\ge 0$ . For a positive definite (PD) matrix, all eigenvalues are $> 0$ .

Hermitian matrices are the complex generalization of real symmetric matrices. They appear everywhere in wireless: covariance matrices $\mathbf{R} = \mathbb{E}[\mathbf{x}\mathbf{x}^H]$ , Gram matrices $\mathbf{H}^H\mathbf{H}$ , and correlation matrices are all Hermitian PSD.

Theorem: SVD-Eigenvalue Connection

For any $\mathbf{A} \in \mathbb{C}^{m \times n}$ :

The singular values of $\mathbf{A}$ are the square roots of the eigenvalues of $\mathbf{A}^H\mathbf{A}$ (or $\mathbf{A}\mathbf{A}^H$ ).
The right singular vectors are the eigenvectors of $\mathbf{A}^H\mathbf{A}$ .
The left singular vectors are the eigenvectors of $\mathbf{A}\mathbf{A}^H$ .

Proof

From SVD to eigendecomposition

If $\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^H$ , then: $\mathbf{A}^H\mathbf{A} = \mathbf{V}\boldsymbol{\Sigma}^H\mathbf{U}^H \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^H = \mathbf{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\mathbf{V}^H = \mathbf{V}\mathrm{diag}(\sigma_i^2)\mathbf{V}^H$ which is the eigendecomposition of $\mathbf{A}^H\mathbf{A}$ .

Example: Eigendecomposition of a Channel Covariance Matrix

Construct a $4 \times 4$ spatial correlation matrix for a uniform linear array (ULA) with correlation coefficient $\rho = 0.7$ . Compute its eigendecomposition and verify the spectral theorem.

Solution

Build the correlation matrix

import numpy as np

N = 4
rho = 0.7
# Exponential correlation model: R[i,j] = rho^|i-j|
R = np.array([[rho ** abs(i - j) for j in range(N)] for i in range(N)])
print("R =\n", R)

Eigendecompose with eigh

eigenvalues, V = np.linalg.eigh(R)
print(f"Eigenvalues: {eigenvalues}")
print(f"Sum of eigenvalues: {np.sum(eigenvalues):.4f}")
print(f"Trace of R: {np.trace(R):.4f}")

# Verify: V V^H = I (orthogonality)
print(f"V^H V = I? {np.allclose(V.conj().T @ V, np.eye(N))}")

# Verify: R = V Lambda V^H (reconstruction)
R_reconstructed = V @ np.diag(eigenvalues) @ V.conj().T
print(f"Reconstruction error: {np.linalg.norm(R - R_reconstructed):.2e}")

Example: Low-Rank Approximation via SVD

Approximate a $100 \times 100$ matrix with a rank-5 version and measure the relative approximation error. Show how the error decreases as the rank increases.

Solution

SVD and truncation

import numpy as np

np.random.seed(42)
A = np.random.randn(100, 100)
U, s, Vh = np.linalg.svd(A, full_matrices=False)

for k in [1, 2, 5, 10, 20, 50]:
    A_k = (U[:, :k] * s[:k]) @ Vh[:k, :]
    rel_error = np.linalg.norm(A - A_k, 'fro') / np.linalg.norm(A, 'fro')
    energy = np.sum(s[:k]**2) / np.sum(s**2) * 100
    print(f"Rank {k:3d}: relative error = {rel_error:.4f}, "
          f"energy captured = {energy:.1f}%")

Key insight

The Eckart-Young theorem guarantees this is the best possible rank- $k$ approximation. In practice, many real-world matrices (images, channel matrices with limited scattering) have rapidly decaying singular values, so a small $k$ captures most of the energy.

Example: Comparing eig and eigh for Hermitian Matrices

Generate a random Hermitian matrix and compare the results of np.linalg.eig vs np.linalg.eigh. Verify that eigh returns sorted real eigenvalues and is faster.

Solution

Comparison

import numpy as np
import time

N = 500
np.random.seed(42)
A = np.random.randn(N, N) + 1j * np.random.randn(N, N)
A = (A + A.conj().T) / 2   # make Hermitian

# eig (general)
t0 = time.perf_counter()
vals_eig, vecs_eig = np.linalg.eig(A)
t_eig = time.perf_counter() - t0

# eigh (Hermitian)
t0 = time.perf_counter()
vals_eigh, vecs_eigh = np.linalg.eigh(A)
t_eigh = time.perf_counter() - t0

print(f"eig  time: {t_eig:.4f}s, max imag in eigenvalues: "
      f"{np.max(np.abs(np.imag(vals_eig))):.2e}")
print(f"eigh time: {t_eigh:.4f}s, eigenvalues are real by construction")
print(f"Speedup: {t_eig/t_eigh:.1f}x")

Eigenvalue Decomposition Explorer

Visualize eigenvalues and eigenvectors of a 2x2 or 3x3 matrix. See how changing matrix entries affects the eigenvalue spectrum and eigenvector directions.

Parameters

Matrix Type

Matrix Size4

Correlation / Rotation Parameter0.7

SVD Low-Rank Approximation (Animated)

Watch a matrix being reconstructed as rank-1, rank-2, ... components are added. See the singular value spectrum and the reconstruction error decrease with each added component.

Parameters

Matrix Source

Matrix Size32

eig vs eigh vs svd

Property	np.linalg.eig	np.linalg.eigh	np.linalg.svd
Input	Any square matrix	Hermitian/symmetric only	Any matrix (m x n)
Eigenvalues	Complex, unsorted	Real, sorted ascending	Real non-negative, sorted descending
Eigenvectors	Not necessarily orthogonal	Orthonormal	Two sets: U and V, orthonormal
LAPACK routine	geev	heev / syev	gesdd
Complexity	$O(n^3)$	$O(n^3)$ but ~3x faster	$O(mn^2)$ or $O(m^2n)$
Use when	General eigenproblems	Covariance, correlation, Gram matrices	Any matrix, rank, pseudoinverse

eigenvalue

A scalar $\lambda$ such that $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ for some nonzero vector $\mathbf{v}$ . Eigenvalues characterize the scaling behavior of a linear transformation.

Related: singular value

singular value

The non-negative square roots of the eigenvalues of $\mathbf{A}^H\mathbf{A}$ . Singular values measure the "gain" of a matrix along its principal directions.

Related: eigenvalue

Common Mistake: Using eig for Hermitian Matrices

Mistake:

Using np.linalg.eig on a matrix that is known to be Hermitian (e.g., covariance matrix $\mathbf{R} = \mathbb{E}[\mathbf{x}\mathbf{x}^H]$ ).

Correction:

Use np.linalg.eigh for Hermitian matrices. It is faster, returns real eigenvalues (no spurious imaginary parts), and guarantees orthogonal eigenvectors. eig on a Hermitian matrix may return eigenvalues with small imaginary parts due to roundoff.

Why This Matters: SVD Precoding for MIMO

In MIMO with channel state information at the transmitter (CSIT), the SVD of $\mathbf{H} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^H$ enables optimal precoding. By transmitting $\mathbf{x} = \mathbf{V}\tilde{\mathbf{x}}$ and receiving $\tilde{\mathbf{y}} = \mathbf{U}^H\mathbf{y}$ , the MIMO channel decomposes into $r$ parallel scalar channels: $\tilde{y}_i = \sigma_i \tilde{x}_i + \tilde{n}_i$ The singular values $\sigma_i$ determine the gain on each stream. Water-filling across these streams achieves the channel capacity.

Historical Note: The SVD: From Beltrami to Big Data

The singular value decomposition was independently discovered by Eugenio Beltrami (1873) and Camille Jordan (1874). Gene Golub and William Kahan developed the modern computational algorithm in 1965. Today, SVD underpins recommender systems (Netflix Prize), natural language processing (latent semantic analysis), image compression, and MIMO wireless — one of the most versatile tools in all of applied mathematics.

Quick Check

When should you use np.linalg.eigh instead of np.linalg.eig?

When the matrix is Hermitian (or real symmetric)

When you want complex eigenvalues

When the matrix is rectangular

Always — eigh is a strict upgrade over eig

Correction:

When the matrix is Hermitian (or real symmetric)

eigh exploits the Hermitian structure for speed and guaranteed real eigenvalues.

Key Takeaway

SVD is the Swiss Army knife of linear algebra. It works on any matrix (square or rectangular, real or complex), reveals the rank, condition number, range, and null space, provides the optimal low-rank approximation, and forms the basis for the pseudoinverse. When in doubt, compute the SVD.

Eigenvalue Applications

python

PCA, low-rank approximation, eigenfaces, and channel correlation analysis using eigenvalue and singular value decompositions.

# Code from: ch06/python/eigenvalue_applications.py
# Load from backend supplements endpoint

Eigenvalue and Singular Value Decomposition

Eigenvalues: The DNA of a Matrix

Definition: Eigenvalue Decomposition

Definition: Singular Value Decomposition (SVD)

Definition: Condition Number

Definition: Numerical Rank

Theorem: Eckart-Young-Mirsky Theorem (Best Low-Rank Approximation)

Theorem: Spectral Theorem for Hermitian Matrices

Theorem: SVD-Eigenvalue Connection

From SVD to eigendecomposition

Example: Eigendecomposition of a Channel Covariance Matrix

Build the correlation matrix

Eigendecompose with eigh

Example: Low-Rank Approximation via SVD

SVD and truncation

Key insight

Example: Comparing eig and eigh for Hermitian Matrices

Comparison

Eigenvalue Decomposition Explorer

Parameters

SVD Low-Rank Approximation (Animated)

Parameters

eig vs eigh vs svd

eigenvalue

singular value

Common Mistake: Using eig for Hermitian Matrices

Why This Matters: SVD Precoding for MIMO

Historical Note: The SVD: From Beltrami to Big Data

Quick Check

Key Takeaway

Eigenvalue Applications

Definition:
Eigenvalue Decomposition

Definition:
Singular Value Decomposition (SVD)

Definition:
Condition Number

Definition:
Numerical Rank