Ferkans — Interactive Telecom Tutor

The Classical Model: Symmetry and Equal Likelihood

The classical probability model assigns equal probability to every outcome. It is both the historically oldest probability model (Laplace, 1812) and the one that arises most naturally in information theory and coding: a uniformly chosen codeword, a randomly selected scheduling slot, a uniformly distributed hash input.

The power of the classical model is that it reduces probability to combinatorics: $\mathbb{P}(A) = |A|/|\Omega|$ . Its limitation is the assumption of symmetry, which must be justified by the problem's physics or structure. When outcomes are not equally likely — a biased channel, a non-uniform source — one must use the general Kolmogorov framework from Section 1.3.

This section develops the classical model, explores when it applies, and uses it to derive several results that will reappear throughout the book.

Definition:
Uniform Probability on a Finite Set

Let $\Omega = \{\omega_1, \ldots, \omega_N\}$ be a finite sample space with $N = |\Omega|$ outcomes. The uniform probability measure on $(\Omega, 2^\Omega)$ assigns: $\mathbb{P}(\{\omega_i\}) = \frac{1}{N} \quad \text{for all } i = 1, \ldots, N.$ For any event $A \subseteq \Omega$ : $\mathbb{P}(A) = \frac{|A|}{|\Omega|} = \frac{|A|}{N}.$

The triple $(\Omega, 2^\Omega, \mathbb{P})$ with this $\mathbb{P}$ is called the classical probability space or equally likely outcomes model.

We verify Kolmogorov's axioms: non-negativity ( $|A|/N \geq 0$ ) and normalization ( $|\Omega|/N = 1$ ) are immediate. Countable additivity reduces to finite additivity for finite $\Omega$ : for disjoint $A_1, \ldots, A_k$ , $|\bigcup A_i| = \sum |A_i|$ , so $\mathbb{P}(\bigcup A_i) = \sum \mathbb{P}(A_i)$ . $\checkmark$

,

Example: Uniform Distribution on a Die

Roll a fair die ( $\Omega = \{1,2,3,4,5,6\}$ , uniform). (a) Find $\mathbb{P}(\text{even})$ . (b) Find $\mathbb{P}(\text{outcome} \leq 3)$ . (c) Find $\mathbb{P}(\text{even} \cap \text{outcome} \leq 3)$ . (d) Find $\mathbb{P}(\text{even} \cup \text{outcome} \leq 3)$ .

Solution

(a) $\mathbb{P}(\text{even})$

$A = \{2, 4, 6\}$ , $|A| = 3$ . $\mathbb{P}(A) = 3/6 = 1/2$ .

(b) $\mathbb{P}(\text{outcome} \leq 3)$

$B = \{1, 2, 3\}$ , $|B| = 3$ . $\mathbb{P}(B) = 3/6 = 1/2$ .

(c) Intersection

$A \cap B = \{2\}$ (even AND $\leq 3$ ). $\mathbb{P}(A \cap B) = 1/6$ .

(d) Union via inclusion-exclusion

$\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A \cap B) = 1/2 + 1/2 - 1/6 = 5/6$ . Check: $A \cup B = \{1,2,3,4,6\}$ , $|A \cup B| = 5$ , $5/6$ $\checkmark$ .

Example: Random Codebook Selection

Shannon's random coding argument: a binary codebook $\mathcal{C}$ consists of $M = 2^{nR}$ codewords, each chosen independently and uniformly from $\{0,1\}^n$ . Compute the probability that two specified codewords $\mathbf{c}_i$ and $\mathbf{c}_j$ ( $i \neq j$ ) are identical.

Solution

Set up the sample space

Each codeword is chosen uniformly from $\{0,1\}^n$ , giving $|\Omega| = (2^n)^M = 2^{nM}$ for the joint codebook selection. For the pair $(\mathbf{c}_i, \mathbf{c}_j)$ : $|\Omega_\text{pair}| = (2^n)^2 = 2^{2n}$ .

Count favorable outcomes

$\mathbf{c}_i = \mathbf{c}_j$ requires both to be the same string: $2^n$ choices. So $|A| = 2^n$ .

Compute the probability

$\mathbb{P}(\mathbf{c}_i = \mathbf{c}_j) = \frac{2^n}{2^{2n}} = 2^{-n}.KATEXPLACEHOLDER0END\mathbb{P}(\text{codebook has a repeated codeword}) \leq \binom{M}{2} \cdot 2^{-n} \approx \frac{M^2}{2} \cdot 2^{-n} = \frac{2^{2nR}}{2} \cdot 2^{-n} = \frac{1}{2} \cdot 2^{n(2R-1)}.$ $For$ R < 1/2 $, this goes to 0 as$ n \to \infty $: a large random binary codebook at rate$ R < 1/2$ almost surely has distinct codewords. This is a baby version of Shannon's random coding argument.

Classical Model vs. General Probability Space

Property	Classical (Uniform) Model	General Kolmogorov Model
Sample space	Finite $\Omega = \{\omega_1,\ldots,\omega_N\}$	Finite, countable, or uncountable
Sigma-algebra	Power set $2^\Omega$	Any sigma-algebra $\mathcal{F}$
Probability of outcome	$1/N$ (by symmetry)	Arbitrary $p_i \geq 0$ with $\sum p_i = 1$
Probability of event	$\|A\|/N$ (counting)	Sum or integral over $A$
Computing $\mathbb{P}(A)$	Combinatorics	Integration or summation
When applicable	Symmetric/equally likely outcomes	Always (with specified model)
Example	Fair die, random codeword	Rayleigh fading, Gaussian noise

Why This Matters: Uniform Randomness in Information Theory and Coding

The classical probability model is not just a pedagogical exercise — it is a key structural assumption in information theory. Shannon's channel capacity theorem is proved by showing that a uniformly random codebook achieves a vanishing error probability as the block length $n \to \infty$ , for any rate $R < C$ . The argument rests on two ingredients: the uniform distribution over codewords and the law of large numbers (which we prove in Chapter 11).

Similarly, in MIMO systems, the capacity-achieving input distribution for an i.i.d. Rayleigh fading channel is Gaussian — but the discrete approximation is a uniform distribution over a QAM constellation. Understanding when "all outcomes equally likely" is a valid model is thus essential for interpreting every capacity formula and every simulation result.

Theorem: Hypergeometric Probability

An urn contains $N$ balls: $K$ are red and $N-K$ are blue. Draw $n$ balls without replacement. The probability that exactly $k$ of the drawn balls are red is: $\mathbb{P}(X = k) = \frac{\binom{K}{k}\binom{N-K}{n-k}}{\binom{N}{n}}, \quad k = \max(0, n-(N-K)), \ldots, \min(n, K).$

The denominator counts all equally likely size- $n$ subsets of the $N$ balls. The numerator counts subsets with exactly $k$ red balls: choose $k$ of the $K$ red balls ( $\binom{K}{k}$ ways) and $n-k$ of the $N-K$ blue balls ( $\binom{N-K}{n-k}$ ways).

Proof

Count the sample space

Drawing $n$ balls from $N$ without replacement gives $\binom{N}{n}$ equally likely outcomes (the drawn subset is unordered).

Count favorable outcomes

To get exactly $k$ red: choose $k$ from the $K$ red balls ( $\binom{K}{k}$ ways), and $n-k$ from the $N-K$ blue balls ( $\binom{N-K}{n-k}$ ways). By the multiplication principle: $\binom{K}{k}\binom{N-K}{n-k}$ total.

Apply uniform probability

$\mathbb{P}(X = k) = \binom{K}{k}\binom{N-K}{n-k} / \binom{N}{n}$ . $\blacksquare$

🎓CommIT Contribution(2021)

Random Coding and the Probabilistic Method in Information Theory

G. Caire — Foundations and Trends in Communications and Information Theory

The random coding argument — assigning a uniform probability measure to the codebook ensemble and showing that the average error probability vanishes — is the central proof technique of information theory. It was introduced by Shannon in 1948 and refined in the following decades into a systematic methodology. The probabilistic method here is not just a proof tool but a constructive strategy: a randomly chosen code is likely to be good, even if we cannot exhibit the best code explicitly. This connects directly to the counting arguments in this section: random codebooks are uniform distributions over finite sets, and every bound in the proof uses the inclusion-exclusion and union bound from Section 1.3.

random-codingchannel-capacityinformation-theory

Common Mistake: The Classical Model Requires Genuine Symmetry

Mistake:

A student argues: "There are two outcomes for a biased coin — Heads or Tails — so each has probability 1/2." Or, more subtly: "The channel either makes an error or it doesn't, so the error probability is 1/2."

Correction:

The classical model is valid only when the outcomes are genuinely symmetric — when the physical mechanism treats all outcomes equally. A biased coin has $\mathbb{P}(H) = p \neq 1/2$ ; the number of outcomes ( $|\Omega| = 2$ ) does not determine their probabilities. For a channel, the error probability $p_e$ is a property of the channel and the signaling scheme, not something we can assign by counting. The classical model applies when: (a) a random experiment is explicitly designed to be fair (shuffled cards, fair dice, uniform random codebook), or (b) physical symmetry enforces equal likelihood (crystal symmetry, isotropic noise).

🔧Engineering Note

Uniform Random Scheduling in Wireless Systems

In multi-user wireless systems, a simple and fair scheduling policy is round-robin or uniform random selection: at each time slot, the base station selects one of $K$ users uniformly at random to serve. Under this classical model, each user is served with probability $1/K$ per slot.

The probability that a given user is NOT served for $n$ consecutive slots is $(1 - 1/K)^n \leq e^{-n/K}$ (using $1-x \leq e^{-x}$ ). By the union bound over $K$ users, the probability that any user waits more than $n = 2K \ln K$ slots is at most $K \cdot e^{-n/K} = K \cdot K^{-2} = 1/K \to 0$ .

This elementary calculation — uniform distribution, union bound, exponential bound — illustrates the practical reach of the tools in this chapter.

Practical Constraints

•
Uniform random scheduling is optimal in throughput only under symmetric channels
•
Proportional fair scheduling (Chapter 11 of Book FSI) adapts weights based on channel quality

Quick Check

A packet of 20 chips contains 5 defective and 15 good ones. An inspector randomly selects 4 chips without replacement. What is the probability that exactly 2 are defective?

$\binom{5}{2}\binom{15}{2}/\binom{20}{4}$

$\binom{5}{2}(1/4)^2(3/4)^2$

$(5/20)(4/19)$

$\binom{20}{4} / (\binom{5}{2}\binom{15}{2})$

Correction:

\binom{5}{2}\binom{15}{2}/\binom{20}{4}

By the hypergeometric formula with $N=20$ , $K=5$ , $n=4$ , $k=2$ : $\mathbb{P}(X=2) = \binom{5}{2}\binom{15}{2}/\binom{20}{4} = (10 \times 105)/4845 \approx 0.217$ .

Key Takeaway

The classical model $\mathbb{P}(A) = |A|/|\Omega|$ converts probability to counting, but requires genuine symmetry. It underpins Shannon's random coding argument, uniform scheduling analysis, and birthday-type collision bounds. Whenever a problem says "uniformly at random" or "equally likely," the classical model applies and the calculation reduces to combinatorics.

Finite Probability Spaces and Equally Likely Outcomes

The Classical Model: Symmetry and Equal Likelihood

Definition: Uniform Probability on a Finite Set

Example: Uniform Distribution on a Die

(a) $\mathbb{P}(\text{even})$

(b) $\mathbb{P}(\text{outcome} \leq 3)$

(c) Intersection

(d) Union via inclusion-exclusion

Example: Random Codebook Selection

Set up the sample space

Count favorable outcomes

Compute the probability

Classical Model vs. General Probability Space

Why This Matters: Uniform Randomness in Information Theory and Coding

Theorem: Hypergeometric Probability

Count the sample space

Count favorable outcomes

Apply uniform probability

Random Coding and the Probabilistic Method in Information Theory

Common Mistake: The Classical Model Requires Genuine Symmetry

Uniform Random Scheduling in Wireless Systems

Quick Check

Key Takeaway

Definition:
Uniform Probability on a Finite Set