Ferkans — Interactive Telecom Tutor

The Gaussian Miracle

In general, uncorrelation ( $\text{Cov}(X_i, X_j) = 0$ ) is strictly weaker than independence. We saw counterexamples in Chapter 7: two random variables can have zero covariance yet remain strongly dependent. The multivariate Gaussian is the grand exception. For Gaussian vectors, uncorrelation and independence are equivalent. This is not just a curiosity — it is the structural property that makes Gaussian models so powerful, because it means that decorrelation (a second-order, linear operation) achieves full statistical independence.

Theorem: Uncorrelated Gaussian Components Are Independent

Let $\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ with $\boldsymbol{\Sigma} = \operatorname{diag}(\sigma_1^2, \ldots, \sigma_n^2)$ (diagonal covariance). Then $X_1, \ldots, X_n$ are mutually independent, each with $X_i \sim \mathcal{N}(\mu_i, \sigma_i^2)$ .

More generally, if $\boldsymbol{\Sigma}$ is block-diagonal with blocks corresponding to sub-vectors $\mathbf{X}_1, \ldots, \mathbf{X}_k$ , then these sub-vectors are mutually independent.

When $\boldsymbol{\Sigma}$ is diagonal, the quadratic form in the exponent separates: $(\mathbf{x}-\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}) = \sum_i (x_i - \mu_i)^2/\sigma_i^2$ . The joint PDF factors into a product of marginal PDFs, which is precisely the definition of independence.

Proof

Factor the PDF

When $\boldsymbol{\Sigma}$ is diagonal, $|\boldsymbol{\Sigma}| = \prod_i \sigma_i^2$ and the exponent separates:

$f_{\mathbf{X}}(\mathbf{x}) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma_i^2}} \exp\!\left(-\frac{(x_i - \mu_i)^2}{2\sigma_i^2}\right) = \prod_{i=1}^n f_{X_i}(x_i).$

Factored PDF implies independence

Since $f_{\mathbf{X}}(\mathbf{x}) = \prod_i f_{X_i}(x_i)$ , the components are mutually independent by definition.

The Converse Fails for Non-Gaussian Distributions

Consider $X \sim \mathcal{N}(0, 1)$ and $Y = X^2$ . Then $\text{Cov}(X, Y) = \mathbb{E}[X^3] = 0$ (by symmetry), so $X$ and $Y$ are uncorrelated. But $Y$ is a deterministic function of $X$ — they are maximally dependent! The Gaussian is special precisely because its distribution is fully determined by second-order statistics.

Example: Decorrelation via Eigenrotation

Let $\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma})$ with $\boldsymbol{\Sigma} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}$ . Find an orthogonal transformation $\mathbf{Y} = \mathbf{Q}^T\mathbf{X}$ such that $Y_1$ and $Y_2$ are independent.

Solution

Eigendecompose $\ntn{covmat}$

Eigenvalues: $\lambda_1 = 4$ , $\lambda_2 = 2$ . Eigenvectors: $\mathbf{u}_1 = \frac{1}{\sqrt{2}}(1,1)^T$ , $\mathbf{u}_2 = \frac{1}{\sqrt{2}}(1,-1)^T$ .

Apply the rotation

Set $\mathbf{Q} = [\mathbf{u}_1, \mathbf{u}_2]$ . Then $\mathbf{Y} = \mathbf{Q}^T\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Lambda})$ with $\boldsymbol{\Lambda} = \operatorname{diag}(4, 2)$ .

Conclude independence

Since $\boldsymbol{\Lambda}$ is diagonal and $\mathbf{Y}$ is Gaussian, $Y_1 \sim \mathcal{N}(0, 4)$ and $Y_2 \sim \mathcal{N}(0, 2)$ are independent.

Decorrelation = Independence for Gaussians

A Manim animation showing a correlated 2D Gaussian cloud being rotated to its principal axes, where the components become independent. Contrast with a non-Gaussian distribution where the same rotation decorrelates but does not make the components independent.

Eigenrotation decorrelates Gaussian vectors (left), but not all distributions (right)

Key Takeaway

For jointly Gaussian vectors, uncorrelated $\Longleftrightarrow$ independent. This is a uniquely Gaussian property. It means that PCA, whitening, and any linear decorrelation technique automatically achieves full statistical independence — but only under the Gaussian assumption.

Uncorrelated vs. Independent

Property	General distributions	Gaussian
Independent $\Rightarrow$ Uncorrelated	Yes (always)	Yes (always)
Uncorrelated $\Rightarrow$ Independent	No (counterexample: $X$ , $X^2$ )	Yes (unique to Gaussian)
Decorrelation technique	Removes linear dependence only	Removes all dependence
Sufficient statistics	Mean + covariance are not sufficient	Mean + covariance are sufficient
Practical implication	Must check higher-order moments	Second-order analysis is complete

Common Mistake: Marginally Gaussian Does Not Imply Jointly Gaussian

Mistake:

Assuming that if $X_1$ and $X_2$ are each marginally Gaussian, then $(X_1, X_2)$ is jointly Gaussian.

Correction:

Marginal Gaussianity is necessary but not sufficient for joint Gaussianity. Counterexample: let $X_1 \sim \mathcal{N}(0,1)$ , $Z \sim \{+1, -1\}$ uniformly, independent of $X_1$ , and $X_2 = Z \cdot X_1$ . Then $X_2$ is also $\mathcal{N}(0,1)$ , but $(X_1, X_2)$ is not jointly Gaussian (the conditional $X_2 | X_1 = x_1$ takes values $\pm x_1$ , not a Gaussian). Joint Gaussianity requires that every linear combination is Gaussian.

Historical Note: Darmois and Skitovich: The Gaussian Uniqueness

1953

The Darmois–Skitovich theorem (1953) provides a deep converse: if $X_1, \ldots, X_n$ are independent and the linear forms $L_1 = \sum a_i X_i$ and $L_2 = \sum b_i X_i$ are also independent, then every $X_i$ for which both $a_i \neq 0$ and $b_i \neq 0$ must be Gaussian. In other words, the Gaussian is the only distribution for which independence can be preserved under arbitrary linear combinations. This is a characterization theorem that places the Gaussian family in a unique position among all probability distributions.

Uncorrelated Implies Independent