Ferkans — Interactive Telecom Tutor

Why Build a Mathematical Theory of Randomness?

Many phenomena in engineering are too complex for deterministic description. The output of a noisy channel, the arrival time of the next packet at a router, the fading amplitude of a wireless link, the phase noise of an oscillator — all of these are quantities that we cannot predict exactly, but whose statistical behavior we can characterize and engineer around.

Before we can make any rigorous statement about such quantities — before we can compute an average, bound a tail probability, or prove convergence — we need a precise mathematical language for randomness. That language is probability theory, and its starting point is the concept of a sample space.

The formalism we build in this chapter may look abstract at first. Hold on. By Section 1.3, every definition will have a direct probabilistic consequence (complementation rule, union bound, continuity of probability) that you will use in every channel model, every error probability analysis, and every detection problem you will ever encounter.

Definition:
Sample Space

An experiment is any process whose outcome is not determined in advance. The sample space $\Omega$ is the set of all possible outcomes of the experiment. A single outcome is called a sample point and is denoted $\omega \in \Omega$ .

Sample spaces may be:

Finite: $\Omega = \{\omega_1, \ldots, \omega_N\}$ with $|\Omega| = N < \infty$ .
Countably infinite: $|\Omega| = \infty$ with a bijection to $\mathbb{N}$ .
Uncountably infinite: e.g., $\Omega = \mathbb{R}$ or $\Omega = [0,1]$ .

The choice of sample space is a modeling decision. For a coin flip, one might use $\Omega = \{H, T\}$ or, more elaborately, the physical space of all possible positions and velocities of the coin at the moment it lands. The art of probability modeling is choosing the simplest $\Omega$ that captures all the randomness relevant to the questions being asked.

,

Example: Sample Spaces in Communications

Identify the sample space for each of the following experiments: (a) Flip a fair coin once. (b) Roll a fair die once. (c) Transmit a binary symbol over a noisy channel and record the received signal. (d) Observe a continuous-time channel output $y(t)$ for $t \in [0, T]$ .

Solution

(a) Single coin flip

$\Omega = \{H, T\}$ . This is a finite sample space with two elements. The sigma-algebra is the power set $2^\Omega = \{\emptyset, \{H\}, \{T\}, \{H,T\}\}$ .

(b) Single die roll

$\Omega = \{1, 2, 3, 4, 5, 6\}$ . A finite sample space with six elements. With the power set as sigma-algebra, all $2^6 = 64$ subsets are events.

(c) Binary transmission, continuous channel

The transmitted symbol is in $\{0, 1\}$ , but the received signal $y$ is a real number (due to additive noise). A natural sample space is $\Omega = \{0,1\} \times \mathbb{R}$ , where each outcome is a pair (transmitted bit, received value). In practice, we often condition on the transmitted symbol and take $\Omega = \mathbb{R}$ for the received value alone.

(d) Continuous-time waveform

Here $\omega$ is an entire function $y: [0,T] \to \mathbb{R}$ , so $\Omega = \mathbb{R}^{[0,T]}$ , the space of all real-valued functions on $[0,T]$ . This is the setting for continuous-time stochastic processes, which we study starting in Chapter 13. Specifying the sigma-algebra on such spaces requires measure theory; for now, note that this is why the measure-theoretic foundations of Section 1.2 are not merely academic.

Definition:
Event

An event is a subset $A \subseteq \Omega$ . We say event $A$ occurs if the outcome $\omega$ of the experiment satisfies $\omega \in A$ .

The impossible event is the empty set $\emptyset$ : no outcome can belong to it. The certain event is $\Omega$ itself: every outcome belongs to it.

Set operations on events correspond to logical connectives:

Set operation	Logical reading
$A \cup B$	$A$ or $B$ (or both) occur
$A \cap B$	$A$ and $B$ both occur
$A^c = \Omega \setminus A$	$A$ does not occur
$A \setminus B = A \cap B^c$	$A$ occurs but $B$ does not
$A \cap B = \emptyset$	$A$ and $B$ are mutually exclusive (disjoint)

,

Theorem: De Morgan's Laws

For any collection of events $A_1, A_2, \ldots$ (finite or countably infinite): $\left(\bigcup_i A_i\right)^c = \bigcap_i A_i^c, \qquad \left(\bigcap_i A_i\right)^c = \bigcup_i A_i^c.$

"Not (A or B)" is the same as "(not A) and (not B)." De Morgan's laws say this logical equivalence holds for any number of events, including countably many. They are indispensable when converting between unions and intersections in proofs.

Proof

Prove the first law: $(\bigcup_i A_i)^c = \bigcap_i A_i^c$

We prove the two sets are equal by showing mutual containment.

( $\subseteq$ ): Let $\omega \in (\bigcup_i A_i)^c$ . Then $\omega \notin \bigcup_i A_i$ , so $\omega \notin A_i$ for every $i$ , hence $\omega \in A_i^c$ for every $i$ , so $\omega \in \bigcap_i A_i^c$ .

( $\supseteq$ ): Let $\omega \in \bigcap_i A_i^c$ . Then $\omega \in A_i^c$ for every $i$ , so $\omega \notin A_i$ for every $i$ , hence $\omega \notin \bigcup_i A_i$ , which gives $\omega \in (\bigcup_i A_i)^c$ .

Prove the second law from the first

Apply the first law to the events $\{A_i^c\}$ : $(\bigcup_i A_i^c)^c = \bigcap_i (A_i^c)^c = \bigcap_i A_i$ . Taking complements of both sides: $\bigcup_i A_i^c = (\bigcap_i A_i)^c$ . $\blacksquare$

Definition:
Partition

A collection of events $\{B_1, B_2, \ldots, B_n\}$ is a partition of $\Omega$ if:

The events are mutually exclusive: $B_i \cap B_j = \emptyset$ for all $i \neq j$ .
The events are exhaustive: $\bigcup_{i=1}^n B_i = \Omega$ .

Any event $A$ can be decomposed along a partition: $A = \bigcup_{i=1}^{n} (A \cap B_i)$ , a disjoint union.

Partitions encode the prior information available to an observer. When an experiment is performed in stages — first a hypothesis is true, then data is collected — the hypothesis partition $\{B_1, \ldots, B_n\}$ decomposes every event $A$ into the pieces "A occurred AND hypothesis $B_i$ was true." This decomposition is the structural foundation of the law of total probability (Chapter 2, Section 2.1).

Set Operations on Events

Explore the set-theoretic operations on two events $A$ and $B$ within sample space $\Omega$ . Select an operation to highlight the corresponding region.

Parameters

Set operation

\mathbb{P}(A)

0.4

\mathbb{P}(B)

0.4

\mathbb{P}(A \cap B)

0.15

Venn Diagrams: From Sample Space to Events

An animated walkthrough of the sample space, events as regions, and the main set operations with their logical interpretations.

Each colored region corresponds to one set-theoretic operation on events

A

and

B

. The entire rectangle is the sample space

\Omega

.

Common Mistake: Disjoint Events Are NOT Independent

Mistake:

A common error at the start of a course is to confuse mutually exclusive events with independent events. Both involve a pairwise relationship between two events, and both feel like "the events don't interfere with each other."

Correction:

Disjoint events ( $A \cap B = \emptyset$ ) are, in fact, as dependent as two events can be: if one occurs, the other is impossible. Independence ( $\mathbb{P}(A \cap B) = \mathbb{P}(A)\mathbb{P}(B)$ ) is a very different property — it means knowing $A$ occurred gives no information about whether $B$ occurred. We define independence formally in Chapter 2; the key point here is: disjoint $\neq$ independent.

Sample Space

The set $\Omega$ of all possible outcomes of an experiment. May be finite, countably infinite, or uncountably infinite.

Related: Event, Probability Space

Event

A subset $A \subseteq \Omega$ of the sample space. We say event $A$ occurs when the experimental outcome $\omega \in A$ .

Related: Sample Space, Sigma-Algebra

Key Takeaway

Events are sets, logical connectives are set operations. Once this dictionary is internalized, set-theoretic manipulations in probability proofs become transparent: every "and" is an intersection, every "or" is a union, every "not" is a complement.

Sample Spaces and Events