Ferkans — Interactive Telecom Tutor

Beyond Moments: Why Cumulants?

Moments describe the "shape" of a distribution, but they interact in complicated ways for sums of random variables. The $k$ -th moment of a sum involves cross-terms between lower moments of the summands. Cumulants simplify this: the $k$ -th cumulant of a sum of independent random variables is simply the sum of the individual $k$ -th cumulants. This additivity makes cumulants the natural language for the CLT and for large deviations.

Definition:
Cumulant Generating Function (CGF)

The cumulant generating function (CGF) of a random variable $X$ is

$m_X(t) = \log M_X(t) = \log \mathbb{E}[e^{tX}],$

defined for $t$ in the domain where $M_X(t) < \infty$ . The $n$ -th cumulant $\kappa_n$ is defined by

$m_X(t) = \sum_{n=1}^{\infty} \kappa_n \frac{t^n}{n!}.$

The first few cumulants are: $\kappa_1 = \mathbb{E}[X]$ (mean), $\kappa_2 = \text{Var}(X)$ (variance), $\kappa_3 = \mathbb{E}[(X-\mu)^3]$ (third central moment = skewness unnormalized).

For the Gaussian distribution, $m_X(t) = \mu t + \sigma^2 t^2/2$ , so $\kappa_1 = \mu$ , $\kappa_2 = \sigma^2$ , and $\kappa_n = 0$ for all $n \geq 3$ . The Gaussian is the unique distribution with only two nonzero cumulants.

Cumulant

The $n$ -th cumulant $\kappa_n$ is the $n$ -th coefficient in the Taylor expansion of the cumulant generating function $m_X(t) = \logM_X(t)$ . Cumulants are additive for independent random variables.

Cumulant Generating Function

$m_X(t) = \log \mathbb{E}[e^{tX}]$ . Its Taylor coefficients are the cumulants: $m_X(t) = \sum_{n=1}^{\infty} \kappa_n t^n/n!$ .

Theorem: Properties of the Cumulant Generating Function

Let $m_X(t) = \log M_X(t)$ where $M_X(t) < \infty$ for $|t| < a$ . Then:

$m_X(0) = 0$ .
$m_X'(0) = \mathbb{E}[X] = \mu$ .
$m_X''(0) = \text{Var}(X)$ .
$m_X(t)$ is convex on its domain.
If $X \perp Y$ , then $m_{X+Y}(t) = m_X(t) + m_Y(t)$ .

Proof

First two derivatives

$m_X'(t) = M_X'(t)/M_X(t)$ , so $m_X'(0) = M_X'(0)/1 = \mathbb{E}[X]$ .

$m_X''(t) = \frac{M_X''(t)M_X(t) - [M_X'(t)]^2}{[M_X(t)]^2}$ , so $m_X''(0) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2 = \text{Var}(X)$ .

Convexity

$m_X''(t) = \text{Var}_{F_t}(X) \geq 0$ , where $F_t$ is the tilted distribution. Alternatively, $m_X''(t) \geq 0$ by Cauchy-Schwarz applied to $\mathbb{E}[X e^{tX}]$ and $\mathbb{E}[e^{tX}]$ .

Additivity

$m_{X+Y}(t) = \log M_{X+Y}(t) = \log[M_X(t)M_Y(t)] = m_X(t) + m_Y(t)$ .

The Gaussian Has the Simplest Cumulant Structure

For the Gaussian $\mathcal{N}(\mu, \sigma^2)$ :

$m_X(t) = \mu t + \frac{\sigma^2}{2}t^2.$

All cumulants $\kappa_n$ with $n \geq 3$ vanish. In fact, the Gaussian is characterized by this property: it is the only distribution with a polynomial CGF. The CLT can be understood as saying that, as we sum more and more i.i.d. random variables, the higher cumulants ( $n \geq 3$ ) become negligible compared to $\kappa_1$ and $\kappa_2$ , so the distribution approaches Gaussian.

Example: Cumulants of the Poisson Distribution

Find all cumulants of $X \sim \text{Poi}(\lambda)$ .

Solution

Compute the CGF

$M_X(t) = e^{\lambda(e^t - 1)}$ , so $m_X(t) = \lambda(e^t - 1) = \lambda\sum_{n=1}^{\infty}\frac{t^n}{n!}$ .

Read off the cumulants

Comparing with $m_X(t) = \sum_{n=1}^{\infty}\kappa_n t^n/n!$ :

$\kappa_n = \lambda \quad\text{for all } n \geq 1.$

All cumulants of the Poisson are equal to $\lambda$ . In particular, $\kappa_1 = \lambda$ (mean) and $\kappa_2 = \lambda$ (variance), confirming $\mathbb{E}[X] = \text{Var}(X) = \lambda$ .

Quick Check

If $X$ and $Y$ are independent with CGFs $m_X$ and $m_Y$ , what is the third cumulant of $Z = X + Y$ ?

$\kappa_3^{(X)} + \kappa_3^{(Y)}$

$\kappa_3^{(X)} \cdot \kappa_3^{(Y)}$

$[\kappa_3^{(X)}]^2 + [\kappa_3^{(Y)}]^2$

$\kappa_3^{(X)} + \kappa_3^{(Y)} + 3\kappa_1^{(X)}\kappa_2^{(Y)} + 3\kappa_2^{(X)}\kappa_1^{(Y)}$

Correction:

\kappa_3^{(X)} + \kappa_3^{(Y)}

By additivity of the CGF: $m_Z(t) = m_X(t) + m_Y(t)$ . Comparing Taylor coefficients: the $n$ -th cumulant of $Z$ is $\kappa_n^{(X)} + \kappa_n^{(Y)}$ for all $n$ .

The Cumulant Generating Function