Ferkans — Interactive Telecom Tutor

Why Independence Matters

Independence is the structural assumption that makes most of information theory work. The i.i.d. (independent and identically distributed) model for sources and channels allows us to factor joint probabilities, write channel capacities as single-letter expressions, and prove coding theorems via the law of large numbers. When independence fails — correlated fading, bursty interference, memory in the channel — the analysis becomes markedly harder and often requires the Markov and mixing tools developed in Chapter 13.

Independence is also the most commonly over-assumed property in engineering. Verifying that a model truly has independent components, rather than merely treating them as independent for mathematical convenience, is a critical modelling skill.

Definition:
Independence of Events

A collection of events $\{A_i : i \in I\}$ is mutually independent (or simply independent) if for every finite subset $J \subseteq I$ : $\mathbb{P}\!\left(\bigcap_{i \in J} A_i\right) = \prod_{i \in J} \mathbb{P}(A_i).$ Two events $A$ and $B$ are independent if $\mathbb{P}(A \cap B) = \mathbb{P}(A)\,\mathbb{P}(B)$ .

The collection is pairwise independent if every pair satisfies $\mathbb{P}(A_i \cap A_j) = \mathbb{P}(A_i)\mathbb{P}(A_j)$ but the higher-order product conditions are not required. Mutual independence implies pairwise independence, but not conversely.

,

Theorem: Equivalent Characterization of Independence

When $\mathbb{P}(B) > 0$ , two events $A$ and $B$ are independent if and only if $\mathbb{P}(A \mid B) = \mathbb{P}(A).$ That is, knowing $B$ occurred provides no information about $A$ .

Proof

Forward implication

If $A \perp B$ (independent), then $\mathbb{P}(A \cap B) = \mathbb{P}(A)\mathbb{P}(B)$ . Dividing by $\mathbb{P}(B) > 0$ : $\mathbb{P}(A \mid B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)} = \frac{\mathbb{P}(A)\mathbb{P}(B)}{\mathbb{P}(B)} = \mathbb{P}(A).$

Reverse implication

If $\mathbb{P}(A \mid B) = \mathbb{P}(A)$ , multiply both sides by $\mathbb{P}(B)$ : $\mathbb{P}(A \cap B) = \mathbb{P}(A \mid B)\mathbb{P}(B) = \mathbb{P}(A)\mathbb{P}(B). \qquad\blacksquare$

Theorem: Independence Is Preserved Under Complementation

If $A$ and $B$ are independent, then so are $A^c$ and $B$ , $A$ and $B^c$ , and $A^c$ and $B^c$ .

Proof

Show $A^c$ and $B$ are independent

We have $\mathbb{P}(B) = \mathbb{P}(A \cap B) + \mathbb{P}(A^c \cap B)$ (since $A$ and $A^c$ partition $\Omega$ ). Therefore: $\mathbb{P}(A^c \cap B) = \mathbb{P}(B) - \mathbb{P}(A \cap B) = \mathbb{P}(B) - \mathbb{P}(A)\mathbb{P}(B) = (1 - \mathbb{P}(A))\mathbb{P}(B) = \mathbb{P}(A^c)\mathbb{P}(B).$

Cascading the result

Applying the same argument to the pair $(A^c, B)$ shows $A^c$ and $B^c$ are independent. Similarly $A$ and $B^c$ are independent. $\blacksquare$

Example: Pairwise Independence Does Not Imply Mutual Independence

Toss two fair coins. Let $A = \{\text{first coin heads}\}$ , $B = \{\text{second coin heads}\}$ , $C = \{\text{both coins show the same face}\}$ . Show that $A$ , $B$ , $C$ are pairwise independent but not mutually independent.

Solution

Sample space and probabilities

$\Omega = \{HH, HT, TH, TT\}$ , each with probability $1/4$ . $\mathbb{P}(A) = 1/2$ , $\mathbb{P}(B) = 1/2$ , $\mathbb{P}(C) = 1/2$ (since $C = \{HH, TT\}$ ).

Pairwise independence

$\mathbb{P}(A \cap B) = \mathbb{P}(\{HH\}) = 1/4 = \mathbb{P}(A)\mathbb{P}(B)$ . ✓ $\mathbb{P}(A \cap C) = \mathbb{P}(\{HH\}) = 1/4 = \mathbb{P}(A)\mathbb{P}(C)$ . ✓ $\mathbb{P}(B \cap C) = \mathbb{P}(\{HH\}) = 1/4 = \mathbb{P}(B)\mathbb{P}(C)$ . ✓

Failure of mutual independence

$\mathbb{P}(A \cap B \cap C) = \mathbb{P}(\{HH\}) = 1/4$ . But $\mathbb{P}(A)\mathbb{P}(B)\mathbb{P}(C) = (1/2)^3 = 1/8 \neq 1/4$ . The three events are pairwise independent but not mutually independent.

Common Mistake: Pairwise Independence Is NOT Mutual Independence

Mistake:

In simulations and modelling, it is tempting to verify independence only for pairs of events and conclude that all events in the collection are independent. The example above (three two-coin events) shows this is false with three events; analogous constructions exist for any number of events.

Correction:

Mutual independence requires the product rule to hold for every finite subset, not just pairs. For $n$ events, there are $2^n - n - 1$ conditions beyond the $\binom{n}{2}$ pairwise conditions. All must be checked.

Independence: Pairwise vs. Mutual

Property	Pairwise Independent	Mutually Independent
Definition	$\mathbb{P}(A_i \cap A_j) = \mathbb{P}(A_i)\mathbb{P}(A_j)$ for all $i \neq j$	$\mathbb{P}(\bigcap_{i \in J} A_i) = \prod_{i \in J}\mathbb{P}(A_i)$ for all finite $J$
Implications	Does NOT imply mutual independence	Implies pairwise independence
Number of conditions	$\binom{n}{2}$ equations	$2^n - n - 1$ equations (all subsets of size $\geq 2$ )
Used in practice	Weaker, easier to verify	Required for most probabilistic analysis
Example counterexample	Two fair coins + same-face event	N/A (no gap in reverse direction)

Independence Checker: $\mathbb{P}(A \cap B)$ vs. $\mathbb{P}(A)\mathbb{P}(B)$

Set the probabilities of three events $A$ , $B$ , $C$ defined on a two-coin experiment and verify whether each pair satisfies the product rule for independence.

Parameters

\mathbb{P}(A)

0.5

\mathbb{P}(B)

0.5

\mathbb{P}(A \cap B)

0.25

⚠️Engineering Note

The i.i.d. Assumption in Shannon Theory

Shannon's channel coding theorem assumes the channel is memoryless: consecutive uses of the channel are statistically independent. Under this assumption the capacity per channel use is a single-letter expression $C = \max_{p_X} I(X;Y)$ . The i.i.d. source coding theorem similarly requires independent symbols. These independence assumptions are the reason capacity results look so clean.

In practice, wireless channels are NOT memoryless: multipath creates frequency-selective fading (correlated across subcarriers) and Doppler creates time-selective fading (correlated across symbols). Engineers work around this via interleaving (reordering symbols to break correlation before decoding) and OFDM (converting a frequency-selective channel into many parallel flat-fading sub-channels, each approximately memoryless).

Practical Constraints

•
LTE/5G NR use OFDM with cyclic prefix to create approximately i.i.d. sub-channel model
•
Interleaver depth must exceed the coherence time to achieve near-independence
•
When coherence bandwidth $\ll$ channel bandwidth, frequency diversity approaches the i.i.d. bound

Independent Events

Events $\{A_i\}$ are mutually independent if $\mathbb{P}(\bigcap_{i \in J} A_i) = \prod_{i \in J}\mathbb{P}(A_i)$ for every finite subset $J$ . Intuitively, knowledge of any subset of events provides no information about the remaining events.

Quick Check

Two events $A$ and $B$ both have positive probability and are disjoint ( $A \cap B = \emptyset$ ). Are they independent?

Yes, because they have no overlap.

No, because $\mathbb{P}(A \cap B) = 0 \neq \mathbb{P}(A)\mathbb{P}(B)$ .

Only if $\mathbb{P}(A) = \mathbb{P}(B)$ .

Impossible to determine without more information.

Correction:

No, because

\mathbb{P}(A \cap B) = 0 \neq \mathbb{P}(A)\mathbb{P}(B)

.

Independence requires $\mathbb{P}(A \cap B) = \mathbb{P}(A)\mathbb{P}(B)$ . But disjointness gives $\mathbb{P}(A \cap B) = 0$ , while $\mathbb{P}(A)\mathbb{P}(B) > 0$ by assumption. So disjoint events with positive probability are DEPENDENT: knowing $A$ occurred tells you $B$ cannot have occurred.

Key Takeaway

Independence means no information flows between events. $A \perp B$ iff $\mathbb{P}(A \mid B) = \mathbb{P}(A)$ : observing $B$ leaves the probability of $A$ unchanged. Disjointness is the opposite extreme — the most dependent possible relationship. Mutual independence is strictly stronger than pairwise independence and requires $2^n - n - 1$ conditions for $n$ events. In information theory, independence is the assumption that makes entropy additive: $H(X_1, \ldots, X_n) = \sum_i H(X_i)$ .

Independence

Why Independence Matters

Definition: Independence of Events

Theorem: Equivalent Characterization of Independence

Forward implication

Reverse implication

Theorem: Independence Is Preserved Under Complementation

Show $A^c$ and $B$ are independent

Cascading the result

Example: Pairwise Independence Does Not Imply Mutual Independence

Sample space and probabilities

Pairwise independence

Failure of mutual independence

Common Mistake: Pairwise Independence Is NOT Mutual Independence

Independence: Pairwise vs. Mutual

Independence Checker: P(A∩B)\mathbb{P}(A \cap B)P(A∩B) vs. P(A)P(B)\mathbb{P}(A)\mathbb{P}(B)P(A)P(B)

Parameters

The i.i.d. Assumption in Shannon Theory

Independent Events

Quick Check

Key Takeaway

Definition:
Independence of Events

Independence Checker: $\mathbb{P}(A \cap B)$ vs. $\mathbb{P}(A)\mathbb{P}(B)$