Convergence Concepts and Limit Theorems

By Markov's inequality applied to the nonnegative random variable $|X_n - X|^2$, for any $\epsilon > 0$:

$$P\!\bigl(|X_n - X| > \epsilon\bigr) = P\!\bigl(|X_n - X|^2 > \epsilon^2\bigr) \leq \frac{E\!\bigl[|X_n - X|^2\bigr]}{\epsilon^2}.$$

If $E[|X_n - X|^2] \to 0$, then the right-hand side tends to zero, so $P(|X_n - X| > \epsilon) \to 0$.

Fix $\epsilon > 0$ and define $A_n = \{|X_n - X| > \epsilon\}$. Almost sure convergence means $P(\limsup_n A_n) = 0$ (by the Borel--Cantelli-type characterisation). Since $P(A_n) \leq P(\bigcup_{k \geq n} A_k)$ and the latter decreases to $P(\limsup_n A_n) = 0$, we get $P(A_n) \to 0$, which is convergence in probability.

Fix $x$ where $F_X$ is continuous and any $\epsilon > 0$. Then $$F_{X_n}(x) = P(X_n \leq x) \leq P(X \leq x + \epsilon) + P(|X_n - X| > \epsilon),$$ and similarly $$F_{X_n}(x) \geq P(X \leq x - \epsilon) - P(|X_n - X| > \epsilon).$$ Taking $n \to \infty$ (so $P(|X_n - X| > \epsilon) \to 0$) and then $\epsilon \to 0$ (using continuity of $F_X$ at $x$) yields $F_{X_n}(x) \to F_X(x)$.

By linearity of expectation,

$$E[\bar{X}_n] = \frac{1}{n}\sum_{i=1}^{n} E[X_i] = \frac{1}{n} \cdot n\mu = \mu.$$

Since the $X_i$ are independent with common variance $\sigma^2$,

$$\mathrm{Var}(\bar{X}_n) = \frac{1}{n^2}\sum_{i=1}^{n} \mathrm{Var}(X_i) = \frac{1}{n^2} \cdot n\sigma^2 = \frac{\sigma^2}{n}.$$

Chebyshev's inequality states that for any random variable $Y$ with finite variance,

$$P\!\bigl(|Y - E[Y]| > \epsilon\bigr) \leq \frac{\mathrm{Var}(Y)}{\epsilon^2}.$$

Applying this with $Y = \bar{X}_n$:

$$P\!\bigl(|\bar{X}_n - \mu| > \epsilon\bigr) \leq \frac{\mathrm{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}.$$

For any fixed $\epsilon > 0$,

$$0 \leq P\!\bigl(|\bar{X}_n - \mu| > \epsilon\bigr) \leq \frac{\sigma^2}{n\epsilon^2} \to 0 \quad \text{as } n \to \infty.$$

By the squeeze theorem, $P(|\bar{X}_n - \mu| > \epsilon) \to 0$, which is precisely $\bar{X}_n \xrightarrow{P} \mu$. $\blacksquare$

The proof of the SLLN under the minimal assumption of finite mean (Kolmogorov's version) relies on truncation arguments, Kolmogorov's maximal inequality, and the Borel--Cantelli lemma. These measure-theoretic tools are beyond the scope of this text.

Under the stronger assumption of finite fourth moment ($E[X_i^4] < \infty$), a more elementary proof using the Borel--Cantelli lemma and Chebyshev's inequality applied to $(\bar{X}_n - \mu)^4$ is possible. We refer the interested reader to Billingsley (1995), Chapter 22.

What we can verify: Since a.s. convergence implies convergence in probability (Theorem thm-convergence-hierarchy), the SLLN is indeed a stronger result than the WLLN. $\blacksquare$

Without loss of generality, assume $\mu = 0$ (otherwise replace $X_i$ by $X_i - \mu$). Let $S_n = \sum_{i=1}^{n} X_i$, so $Z_n = S_n / (\sigma\sqrt{n})$.

The characteristic function of $Z_n$ is

$$\Phi_{Z_n}(\omega) = E\!\left[e^{j\omega Z_n}\right] = E\!\left[e^{j\omega S_n / (\sigma\sqrt{n})}\right].$$

Since the $X_i$ are iid and independent,

$$\Phi_{Z_n}(\omega) = \prod_{i=1}^{n} E\!\left[e^{j\omega X_i / (\sigma\sqrt{n})}\right] = \left[\Phi_X\!\left(\frac{\omega}{\sigma\sqrt{n}}\right) \right]^n,$$

where $\Phi_X(t) = E[e^{jtX_1}]$ is the common CF of each $X_i$.

Since $E[X_1] = 0$ and $E[X_1^2] = \sigma^2$, the Taylor expansion of $\Phi_X(t)$ around $t = 0$ gives

$$\Phi_X(t) = 1 + jt\,E[X_1] + \frac{(jt)^2}{2}\,E[X_1^2] + o(t^2) = 1 - \frac{\sigma^2 t^2}{2} + o(t^2).$$

Substituting $t = \omega / (\sigma\sqrt{n})$:

$$\Phi_X\!\left(\frac{\omega}{\sigma\sqrt{n}}\right) = 1 - \frac{\sigma^2}{2} \cdot \frac{\omega^2}{\sigma^2 n} + o(1/n) = 1 - \frac{\omega^2}{2n} + o(1/n).$$

Therefore

$$\Phi_{Z_n}(\omega) = \left[1 - \frac{\omega^2}{2n} + o(1/n)\right]^n.$$

Using the fundamental limit $\lim_{n \to \infty}(1 + a_n/n)^n = e^a$ whenever $a_n \to a$, with $a_n = -\omega^2/2 + o(1)$:

$$\lim_{n \to \infty} \Phi_{Z_n}(\omega) = e^{-\omega^2/2}.$$

The function $e^{-\omega^2/2}$ is the characteristic function of $\mathcal{N}(0, 1)$. By Levy's continuity theorem, pointwise convergence of characteristic functions to a function that is continuous at $\omega = 0$ (and $e^{-\omega^2/2}$ is continuous everywhere) implies convergence in distribution:

$$Z_n \xrightarrow{d} \mathcal{N}(0, 1). \qquad \blacksquare$$

For any $s > 0$, the function $t \mapsto e^{st}$ is strictly increasing, so

$$\{X \geq a\} = \{e^{sX} \geq e^{sa}\}.$$

By Markov's inequality (for the nonnegative random variable $e^{sX}$),

$$P(X \geq a) = P(e^{sX} \geq e^{sa}) \leq \frac{E[e^{sX}]}{e^{sa}} = e^{-sa}\,M_X(s).$$

The inequality $P(X \geq a) \leq e^{-sa}M_X(s)$ holds for every $s > 0$. To obtain the tightest bound, we minimise the right-hand side over $s$:

$$P(X \geq a) \leq \inf_{s > 0}\; e^{-sa}\,M_X(s).$$

The optimal $s^*$ satisfies the first-order condition

$$\frac{d}{ds}\bigl[e^{-sa}M_X(s)\bigr] = 0 \quad \Longleftrightarrow \quad M_X'(s^*) = a\,M_X(s^*),$$

or equivalently $E[X\,e^{s^*X}] / E[e^{s^*X}] = a$, which means $a$ equals the tilted mean under the exponentially twisted distribution. $\blacksquare$

Defining the log-moment generating function (cumulant generating function)

$$\Lambda(s) = \ln M_X(s),$$

and its Legendre--Fenchel transform (the rate function)

$$\Lambda^*(a) = \sup_{s > 0}\;\bigl[sa - \Lambda(s)\bigr],$$

the Chernoff bound takes the compact form

$$P(X \geq a) \leq e^{-\Lambda^*(a)}.$$

The rate function $\Lambda^*$ governs the exponential decay of tail probabilities and is the central object in Cramer's theorem (the foundation of large deviations theory).