Ferkans — Interactive Telecom Tutor

Why Independence Is the Central Assumption

Independence is the single most important structural assumption in probability and its applications. The entire machinery of information theory — i.i.d. sources, memoryless channels, random codebooks — is built on independence. When it holds, computations simplify dramatically: joint distributions factor, expectations of products equal products of expectations, and the variance of a sum equals the sum of variances. When it fails, the analysis becomes harder, but the deviations from independence are often the most interesting part of the problem.

Definition:
Independence of Random Variables

Random variables $X$ and $Y$ are independent if for all $x, y \in \mathbb{R}$ :

$F_{X,Y}(x, y) = F_{X}(x) \cdot F_{Y}(y).$

Equivalently:

Discrete case: $P_{X,Y}(x_i, y_j) = P_{X}(x_i) \cdot P_{Y}(y_j)$ for all $i, j$ .
Continuous case: $f_{X,Y}(x, y) = f_{X}(x) \cdot f_{Y}(y)$ for all $x, y$ .

A collection $X_1, X_2, \ldots, X_n$ is (mutually) independent if the joint CDF (or PMF, or PDF) factors into a product of marginals for every subset of the collection.

Pairwise independence does not imply mutual independence. The same subtlety we encountered for events in Chapter 2 carries over to random variables.

Theorem: Functions of Independent RVs Are Independent

If $X$ and $Y$ are independent random variables and $g, h$ are Borel-measurable functions, then $g(X)$ and $h(Y)$ are independent.

Proof

CDF factorization

For any $a, b \in \mathbb{R}$ :

$\mathbb{P}(g(X) \le a, h(Y) \le b) = \mathbb{P}(X \in g^{-1}((-\infty,a]),\; Y \in h^{-1}((-\infty,b])).$

Since $g^{-1}((-\infty,a])$ and $h^{-1}((-\infty,b])$ are Borel sets and $X, Y$ are independent, the probability factors:

$= \mathbb{P}(X \in g^{-1}((-\infty,a])) \cdot \mathbb{P}(Y \in h^{-1}((-\infty,b])) = \mathbb{P}(g(X) \le a) \cdot \mathbb{P}(h(Y) \le b). \quad\blacksquare$

Theorem: Product of Expectations

If $X$ and $Y$ are independent and $\mathbb{E}[|X|], \mathbb{E}[|Y|] < \infty$ , then

$\mathbb{E}[XY] = \mathbb{E}[X] \cdot \mathbb{E}[Y].$

More generally, for any measurable $g, h$ with $\mathbb{E}[|g(X)|], \mathbb{E}[|h(Y)|] < \infty$ :

$\mathbb{E}[g(X)\,h(Y)] = \mathbb{E}[g(X)] \cdot \mathbb{E}[h(Y)].$

Proof

Joint factorization

$\mathbb{E}[g(X)\,h(Y)] = \iint g(x)\,h(y)\,f_{X,Y}(x,y)\,dx\,dy.KATEXPLACEHOLDER0END= \left(\int g(x)\,f_{X}(x)\,dx\right) \left(\int h(y)\,f_{Y}(y)\,dy\right) = \mathbb{E}[g(X)] \cdot \mathbb{E}[h(Y)]. \quad\blacksquare$ $

Example: Poisson Splitting Property

A coin is tossed $N$ times where $N \sim \text{Poisson}(\lambda)$ . Each toss independently lands heads with probability $p$ . Let $X$ = number of heads and $Y$ = number of tails. Show that $X$ and $Y$ are independent with $X \sim \text{Poisson}(\lambda p)$ and $Y \sim \text{Poisson}(\lambda(1-p))$ .

Solution

Joint PMF calculation

Since $X + Y = N$ and given $N = m$ , $X \sim \text{Binomial}(m, p)$ :

$P_{X,Y}(k, j) = \sum_{m=0}^{\infty} P_{X,Y|N}(k,j \mid m)\,P_{N}(m).$

The conditional term is zero unless $m = k + j$ , in which case it equals $\binom{k+j}{k} p^k (1-p)^j$ .

Simplify

$P_{X,Y}(k,j) = \binom{k+j}{k} p^k(1-p)^j \cdot \frac{e^{-\lambda}\lambda^{k+j}}{(k+j)!}.KATEXPLACEHOLDER0END= \frac{(\lambda p)^k e^{-\lambda p}}{k!} \cdot \frac{(\lambda(1-p))^j e^{-\lambda(1-p)}}{j!} = P_{X}(k) \cdot P_{Y}(j).$ $

Conclusion

The joint PMF factors into a product of marginals, so $X$ and $Y$ are independent. Each marginal is a Poisson PMF with the indicated parameter. This is the Poisson splitting property — a hallmark result that makes Poisson processes so tractable.

Common Mistake: Uncorrelated Does Not Imply Independent

Mistake:

Concluding that $X$ and $Y$ are independent because $\text{Cov}(X, Y) = 0$ .

Correction:

Uncorrelatedness means $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$ , which is a statement about second moments only. Independence is a much stronger condition: it requires the entire joint distribution to factor. A classic counterexample: let $X \sim \mathcal{N}(0,1)$ and $Y = X^2$ . Then $\text{Cov}(X, Y) = \mathbb{E}[X^3] = 0$ (by symmetry), but $Y$ is completely determined by $X$ .

The one important exception: for jointly Gaussian random variables, uncorrelated does imply independent. This is a special property of the Gaussian distribution, developed in Chapter 8.

Independence vs. Uncorrelatedness

Property	Independent	Uncorrelated
Formal condition	$F_{X,Y}(x,y) = F_{X}(x)\,F_{Y}(y)$ for all $x,y$	$\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$
What it constrains	Entire joint distribution	Second-order moments only
Implies the other?	Independence $\Rightarrow$ uncorrelated	Uncorrelated $\not\Rightarrow$ independent (in general)
Exception	—	Jointly Gaussian: uncorrelated $\Leftrightarrow$ independent
$\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$ ?	Yes	Yes (both suffice)

Independent random variables

Random variables whose joint CDF (or PMF/PDF) factors as a product of marginals. Independence means knowing the value of one provides no information about the other.

Quick Check

If $f_{X,Y}(x,y) = e^{-x} e^{-y}$ for $x, y \ge 0$ and zero otherwise, are $X$ and $Y$ independent?

Yes, because the joint PDF factors as $e^{-x} \cdot e^{-y}$ .

No, because they are both exponential.

Cannot determine without computing the CDF.

Correction:

Yes, because the joint PDF factors as

e^{-x} \cdot e^{-y}

.

The joint PDF factors as $f_{X}(x) \cdot f_{Y}(y)$ where each marginal is $\text{Exp}(1)$ .

Historical Note: The Formalization of Independence

1933

The concept of independence has been used informally since the earliest work on games of chance. But its rigorous mathematical definition — as the factorization of a joint distribution — was established by Kolmogorov in his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung. Kolmogorov's formalization made it possible to state precisely when two random variables "have nothing to do with each other" and to derive consequences such as the strong law of large numbers and the central limit theorem.

Key Takeaway

Independence of random variables means the joint distribution factors as a product of marginals. It implies uncorrelatedness, but the converse is false except for jointly Gaussian RVs. Independence is the key structural assumption that makes most of information theory and performance analysis tractable.

Independence of Random Variables