Ferkans — Interactive Telecom Tutor

The Two Great Limit Theorems

The law of large numbers (LLN) and the central limit theorem (CLT) are the most important results in probability theory. The LLN says that sample averages converge to the true mean; the CLT says how: the fluctuations around the mean are Gaussian with standard deviation $\sigma/\sqrt{n}$ .

Both theorems have elegant proofs via characteristic functions. The strategy is the same: compute the CF of the (normalized) partial sum, show it converges pointwise to a known CF, and invoke the Levy continuity theorem.

Definition:
Convergence in Distribution

A sequence of random variables $X_n$ converges in distribution to $X$ , written $X_n \xrightarrow{D} X$ , if

$\lim_{n \to \infty} F_{X_n}(x) = F(x)$

at every continuity point $x$ of $F$ .

Convergence in distribution is the weakest mode of convergence. It does not require the $X_n$ to be defined on the same probability space. In particular, $X_n \xrightarrow{D} c$ (a constant) is equivalent to $X_n \xrightarrow{P} c$ (convergence in probability).

Theorem: Levy Continuity Theorem

Let $\{X_n\}$ be a sequence of random variables with CFs $\{\phi_n\}$ .

(a) If $X_n \xrightarrow{D} X$ with CF $\phi$ , then $\phi_n(u) \to \phi(u)$ for all $u \in \mathbb{R}$ .

(b) Conversely, if $\phi(u) = \lim_{n \to \infty}\phi_n(u)$ exists for all $u$ and $\phi$ is continuous at $u = 0$ , then $\phi$ is a valid CF of some CDF $F$ , and $X_n \xrightarrow{D} X$ where $F = F$ .

Convergence in distribution is equivalent to pointwise convergence of CFs, provided the limit is continuous at the origin. This is the bridge that allows us to prove limit theorems by working entirely in the transform domain.

Proof

Forward direction (sketch)

$\phi_n(u) = \int e^{jux}\,dF_n(x)$ . Since $e^{jux}$ is bounded and continuous, and $F_n \to F$ in distribution, the Helly-Bray theorem gives $\phi_n(u) \to \int e^{jux}\,dF(x) = \phi(u)$ .

Converse (sketch)

The sequence $\{F_n\}$ is tight (since $\phi$ is continuous at $0$ , mass cannot escape to infinity). By Prohorov's theorem, every subsequence has a further subsequence converging to some CDF $G$ . By the forward direction, the CF of $G$ must be $\phi$ . By uniqueness, all subsequential limits agree, so $F_n \to F$ .

,

Theorem: The Law of Large Numbers (via CF)

Let $X_1, X_2, \ldots$ be i.i.d. with finite mean $\mu = \mathbb{E}[X_1]$ and partial sum $S_n = \sum_{i=1}^n X_i$ . Then

$\frac{S_n}{n} \xrightarrow{D} \mu.$

The sample average $S_n/n$ concentrates around the true mean $\mu$ . As $n$ grows, the CF of $S_n/n$ converges to $e^{j\mu u}$ , which is the CF of the constant $\mu$ . A degenerate (constant) limit in distribution implies convergence in probability.

Proof

Compute the CF of $Z_n = S_n/n$

By the scaling property and independence: $\phi_{Z_n}(u) = [\phi_X(u/n)]^n$ .

Taylor-expand the CF

Since $\mathbb{E}[|X|] < \infty$ , Theorem TMoments from the Characteristic Function gives: $\phi_X(u) = 1 + j\mu u + o(u)$ as $u \to 0$ .

Substituting $u/n$ : $\phi_X(u/n) = 1 + \frac{j\mu u}{n} + o(1/n)$ .

Take the $n$-th power

$\phi_{Z_n}(u) = \left(1 + \frac{j\mu u}{n} + o(1/n)\right)^n \to e^{j\mu u}$ $as$ n \to \infty $. This is the CF of the constant$ \mu$.

Apply the continuity theorem

$e^{j\mu u}$ is continuous at $u = 0$ , so by Theorem TLevy Continuity Theorem, $Z_n \xrightarrow{D} \mu$ .

Theorem: The Central Limit Theorem (via CF)

Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and variance $\sigma^2 > 0$ . Then

$\frac{S_n - n\mu}{\sigma\sqrt{n}} \xrightarrow{D} \mathcal{N}(0, 1).$

After centering and scaling, the distribution of the partial sum approaches a Gaussian. The CF of the standardized sum converges to $e^{-u^2/2}$ , the CF of the standard Gaussian. This is because the higher cumulants ( $\kappa_3, \kappa_4, \ldots$ ) scale as $n^{-1/2}, n^{-1}, \ldots$ relative to $\kappa_2$ , so they vanish in the limit.

Proof

Standardize

Let $Y_i = (X_i - \mu)/\sigma$ with $\mathbb{E}[Y_i] = 0$ , $\text{Var}(Y_i) = 1$ . Let $\phi_Y$ denote the common CF of the $Y_i$ .

Taylor-expand $\phi_Y$

Since $\mathbb{E}[Y^2] = 1 < \infty$ : $\phi_Y(u) = 1 - \frac{u^2}{2} + o(u^2)$ .

CF of the standardized sum

$U_n = \frac{1}{\sqrt{n}}\sum_{i=1}^n Y_i$ has CF

$\phi_{U_n}(u) = \left[\phi_Y\!\left(\frac{u}{\sqrt{n}}\right)\right]^n = \left(1 - \frac{u^2}{2n} + o(u^2/n)\right)^n.$

Take the limit

$\phi_{U_n}(u) \to e^{-u^2/2}$ $as$ n \to \infty $. This is the CF of$ \mathcal{N}(0, 1)$.

Conclude

By the Levy continuity theorem, $U_n \xrightarrow{D} \mathcal{N}(0,1)$ .

The CLT in Action: CF Convergence to Gaussian

Watch how the characteristic function of the standardized sum $U_n = (S_n - n\mu)/(\sigma\sqrt{n})$ converges to the Gaussian CF $e^{-u^2/2}$ as $n$ increases. The real part (top) and imaginary part (bottom) are shown.

Parameters

Base distribution

Number of summands

n

1

Visualizing the Central Limit Theorem

This animation shows the PDF of

U_n = (S_n - n\mu)/(\sigma\sqrt{n})

for increasing

n

, overlaid with the standard Gaussian density. The convergence is visible even for small

n

and becomes compelling by

n \approx 30

.

The standardized sum of

n

i.i.d. random variables converges in distribution to

\mathcal{N}(0,1)

as

n

grows.

Historical Note: The Long Road to the Central Limit Theorem

18th-20th century

The CLT has a rich history spanning three centuries. De Moivre (1733) proved the first version for Bernoulli trials. Laplace (1810) extended it using generating functions. Chebyshev (1887) attempted a proof via moments, which Markov (1898) completed. The modern proof via characteristic functions, clean and general, is due to Levy (1925) and Lindeberg (1922). Lindeberg's condition — the most general sufficient condition for the CLT — removes the identical distribution requirement, needing only that no single summand dominates.

The CLT is arguably the most practically important theorem in all of mathematics: it explains why the Gaussian distribution appears so ubiquitously in nature and engineering.

Common Mistake: The CLT Is About the Limit, Not the Rate

Mistake:

Assuming that the CLT implies a good Gaussian approximation for small $n$ (e.g., $n = 5$ ). The CLT says nothing about the quality of the approximation for finite $n$ .

Correction:

The Berry-Esseen theorem quantifies the rate: the CDF error is bounded by $C\rho/(\sigma^3\sqrt{n})$ where $\rho = \mathbb{E}[|X-\mu|^3]$ . The constant $C < 0.4748$ (Shevtsova, 2011). For heavy-tailed distributions or asymmetric distributions, convergence can be slow. Always check with simulations or Berry-Esseen before trusting the Gaussian approximation for moderate $n$ .

Quick Check

In the CLT proof via CFs, what is the key property of the Gaussian CF $e^{-u^2/2}$ that ensures the limit is well-defined?

It is continuous at $u = 0$ , satisfying the Levy continuity theorem

It is bounded above by $1$

It is the only CF that is real-valued

It is analytic on the whole real line

Correction:

It is continuous at

u = 0

, satisfying the Levy continuity theorem

The Levy continuity theorem requires that the limiting function be continuous at the origin. Since $e^{-u^2/2}$ is smooth everywhere (and equals $1$ at $u = 0$ ), the theorem applies and guarantees convergence in distribution.

The Law of Large Numbers and Central Limit Theorem

The Two Great Limit Theorems

Definition: Convergence in Distribution

Theorem: Levy Continuity Theorem

Forward direction (sketch)

Converse (sketch)

Theorem: The Law of Large Numbers (via CF)

Compute the CF of $Z_n = S_n/n$

Taylor-expand the CF

Take the $n$-th power

Apply the continuity theorem

Theorem: The Central Limit Theorem (via CF)

Standardize

Taylor-expand $\phi_Y$

CF of the standardized sum

Take the limit

Conclude

The CLT in Action: CF Convergence to Gaussian

Parameters

Visualizing the Central Limit Theorem

Historical Note: The Long Road to the Central Limit Theorem

Common Mistake: The CLT Is About the Limit, Not the Rate

Quick Check

Definition:
Convergence in Distribution