Radon-Nikodym Theorem and Likelihood Ratios

Densities as Derivatives of Measures

We are used to thinking of a PDF fX(x)f_X(x) as the function you integrate to get probabilities: P(a<X≤b)=∫abfX(x) dxP(a < X \leq b) = \int_a^b f_X(x)\, dx. But what is this function, really? It is the rate at which the probability measure PXP_X accumulates mass compared to the Lebesgue measure λ\lambda. In other words, the PDF is a derivative of one measure with respect to another: fX=dPX/dλf_X = dP_X / d\lambda.

The Radon-Nikodym theorem makes this precise and vastly generalizes it: any time one measure is "dominated by" another (absolutely continuous), there exists a density function relating them. The payoff for us is immediate: the likelihood ratio L(y)=f1(y)/f0(y)L(y) = f_1(y) / f_0(y) used in hypothesis testing is nothing but the Radon-Nikodym derivative dP1/dP0dP_1 / dP_0 — and this viewpoint extends hypothesis testing to settings where densities do not exist (e.g., testing between stochastic processes).

,

Definition:

Absolute Continuity of Measures

Let μ\mu and ν\nu be measures on (Ω,F)(\Omega, \mathcal{F}). We say ν\nu is absolutely continuous with respect to μ\mu, written ν≪μ\nu \ll \mu, if: μ(A)=0  ⟹  ν(A)=0for all A∈F.\mu(A) = 0 \implies \nu(A) = 0 \quad \text{for all } A \in \mathcal{F}. In words: every μ\mu-null set is also ν\nu-null. The measure μ\mu "dominates" ν\nu.

If ν≪μ\nu \ll \mu and μ≪ν\mu \ll \nu, the two measures are equivalent (ν∼μ\nu \sim \mu): they agree on which sets have measure zero.

Definition:

Singular Measures

Two measures μ\mu and ν\nu on (Ω,F)(\Omega, \mathcal{F}) are mutually singular, written μ⊥ν\mu \perp \nu, if there exists A∈FA \in \mathcal{F} with μ(A)=0\mu(A) = 0 and ν(Ac)=0\nu(A^c) = 0. Intuitively, μ\mu and ν\nu "live on disjoint sets."

The Cantor distribution is singular with respect to Lebesgue measure: it concentrates all its mass on the Cantor set, which has Lebesgue measure zero. Point masses (discrete distributions) are also singular w.r.t. Lebesgue measure.

Theorem: Lebesgue Decomposition Theorem

Let μ\mu and ν\nu be σ\sigma-finite measures on (Ω,F)(\Omega, \mathcal{F}). Then ν\nu has a unique decomposition: ν=νa+νs,\nu = \nu_a + \nu_s, where νa≪μ\nu_a \ll \mu (absolutely continuous part) and νs⊥μ\nu_s \perp \mu (singular part).

Any measure can be split into a part that has a density w.r.t. μ\mu and a part that lives on a μ\mu-null set. For a random variable's distribution w.r.t. Lebesgue measure: the absolutely continuous part is the "continuous density" part, and the singular part includes point masses (discrete component) and singular continuous distributions (like the Cantor distribution).

Proof
Construction via Radon-Nikodym

Define ρ=μ+ν\rho = \mu + \nu (a σ\sigma-finite measure). Both μ≪ρ\mu \ll \rho and ν≪ρ\nu \ll \rho. By Radon-Nikodym (applied to the ρ\rho-dominated case): dμ=f dρd\mu = f\, d\rho and dν=g dρd\nu = g\, d\rho. Set A={f>0}A = \{f > 0\}. Then νa=ν∣A\nu_a = \nu|_A is absolutely continuous w.r.t. μ\mu, and νs=ν∣Ac\nu_s = \nu|_{A^c} is singular (μ(Ac)=0\mu(A^c) = 0). ■\blacksquare

Theorem: Radon-Nikodym Theorem

Let μ\mu be a σ\sigma-finite measure on (Ω,F)(\Omega, \mathcal{F}) and let ν\nu be a finite measure with ν≪μ\nu \ll \mu. Then there exists a non-negative measurable function ff such that: ν(A)=∫Af dμfor all A∈F.\nu(A) = \int_A f\, d\mu \quad \text{for all } A \in \mathcal{F}. The function ff is unique μ\mu-a.e. and is called the Radon-Nikodym derivative of ν\nu with respect to μ\mu, written f=dνdμf = \frac{d\nu}{d\mu}.

The Radon-Nikodym derivative is the "density" of ν\nu relative to μ\mu. When μ=λ\mu = \lambda (Lebesgue measure) and ν=PX\nu = P_X (distribution of a continuous random variable), dPX/dλ=fXdP_X / d\lambda = f_X, the probability density function. The theorem says that this notion of density exists whenever one measure is absolutely continuous with respect to another.

Proof
Von Neumann's proof via Hilbert space

Consider L2(Ω,F,μ+ν)L^2(\Omega, \mathcal{F}, \mu + \nu). The linear functional Λ(g)=∫g dν\Lambda(g) = \int g\, d\nu is bounded: ∣Λ(g)∣≤∥g∥L2(μ+ν)|\Lambda(g)| \leq \|g\|_{L^2(\mu + \nu)} (by Cauchy-Schwarz, since ν\nu is finite). By the Riesz representation theorem, there exists h∈L2(μ+ν)h \in L^2(\mu + \nu) with ∫g dν=∫gh d(μ+ν)\int g\, d\nu = \int gh\, d(\mu + \nu) for all g∈L2g \in L^2.

Identify the derivative

Setting g=1Ag = \mathbf{1}_A: ν(A)=∫Ah dμ+∫Ah dν\nu(A) = \int_A h\, d\mu + \int_A h\, d\nu. Rearranging: ∫A(1−h) dν=∫Ah dμ\int_A (1 - h)\, d\nu = \int_A h\, d\mu. One shows 0≤h<10 \leq h < 1 (μ+ν)(\mu + \nu)-a.e. (using ν≪μ\nu \ll \mu). Define f=h/(1−h)f = h / (1 - h). Then ν(A)=∫Af dμ\nu(A) = \int_A f\, d\mu for all A∈FA \in \mathcal{F}. ■\blacksquare

,

Historical Note: Radon, Nikodym, and the Density Problem

1913--1930

Johann Radon proved the theorem in 1913 for the special case of Rn\mathbb{R}^n with Lebesgue measure. The full abstract version was established by Otton Nikodym in 1930. The theorem resolved a long-standing question: under what conditions does one measure have a "density" with respect to another? The answer — absolute continuity — is both necessary and sufficient, and the result became a cornerstone of modern analysis, probability, and mathematical statistics.

Definition:

Likelihood Ratio as Radon-Nikodym Derivative

Let P0P_0 and P1P_1 be two probability measures on (Ω,F)(\Omega, \mathcal{F}) with P1≪P0P_1 \ll P_0. The likelihood ratio is the Radon-Nikodym derivative: L=dP1dP0.L = \frac{dP_1}{dP_0}. When P0P_0 and P1P_1 have densities f0,f1f_0, f_1 with respect to a common dominating measure μ\mu (e.g., Lebesgue measure), this reduces to the familiar ratio: L(ω)=f1(ω)f0(ω).L(\omega) = \frac{f_1(\omega)}{f_0(\omega)}.

The Radon-Nikodym viewpoint is essential when densities do not exist — for example, when testing between two Gaussian processes (the Cameron-Martin-Girsanov theorem) or between discrete and continuous hypotheses.

,

Theorem: Chain Rule for Radon-Nikodym Derivatives

If ν≪μ≪λ\nu \ll \mu \ll \lambda, then ν≪λ\nu \ll \lambda and: dνdλ=dνdμ⋅dμdλλ-a.e.\frac{d\nu}{d\lambda} = \frac{d\nu}{d\mu} \cdot \frac{d\mu}{d\lambda} \quad \lambda\text{-a.e.}

Just like the chain rule for ordinary derivatives: the density of ν\nu relative to λ\lambda is the product of densities along the "chain." This is used in statistics when changing the reference measure — for example, computing the likelihood ratio under a composite hypothesis by going through a parametric family.

Proof
Verify the defining property

For any A∈FA \in \mathcal{F}: ν(A)=∫Adνdμ dμ=∫Adνdμ⋅dμdλ dλ,\nu(A) = \int_A \frac{d\nu}{d\mu}\, d\mu = \int_A \frac{d\nu}{d\mu} \cdot \frac{d\mu}{d\lambda}\, d\lambda, where the second equality uses the change-of-measure formula for the integral (if dμ=g dλd\mu = g\, d\lambda, then ∫Ah dμ=∫Ahg dλ\int_A h\, d\mu = \int_A hg\, d\lambda). The result dνdμ⋅dμdλ\frac{d\nu}{d\mu} \cdot \frac{d\mu}{d\lambda} satisfies the defining property of dνdλ\frac{d\nu}{d\lambda}, so by uniqueness they are equal a.e. ■\blacksquare

Example: Gaussian Likelihood Ratio as Radon-Nikodym Derivative

Let P0=N(0,1)P_0 = \mathcal{N}(0, 1) and P1=N(μ,1)P_1 = \mathcal{N}(\mu, 1) on (R,B(R))(\mathbb{R}, \mathcal{B}(\mathbb{R})). Compute the Radon-Nikodym derivative dP1/dP0dP_1/dP_0.

Solution
Both measures are a.c. w.r.t. Lebesgue

dP0/dλ=12πe−x2/2dP_0/d\lambda = \frac{1}{\sqrt{2\pi}} e^{-x^2/2} and dP1/dλ=12πe−(x−μ)2/2dP_1/d\lambda = \frac{1}{\sqrt{2\pi}} e^{-(x-\mu)^2/2}. Since both are positive everywhere, P0∼P1P_0 \sim P_1 (mutually absolutely continuous).

Apply the chain rule

dP1dP0(x)=dP1/dλdP0/dλ(x)=e−(x−μ)2/2e−x2/2=eμx−μ2/2.\frac{dP_1}{dP_0}(x) = \frac{dP_1/d\lambda}{dP_0/d\lambda}(x) = \frac{e^{-(x-\mu)^2/2}}{e^{-x^2/2}} = e^{\mu x - \mu^2/2}.$

Interpretation

This is the familiar likelihood ratio for testing H0:θ=0H_0: \theta = 0 vs H1:θ=μH_1: \theta = \mu with a single Gaussian observation. The Neyman-Pearson lemma says the optimal test compares L(x)=eμx−μ2/2L(x) = e^{\mu x - \mu^2/2} to a threshold — or equivalently, compares xx to a threshold (since LL is monotone in xx).

Radon-Nikodym Derivative dP/dQdP/dQ as Density Ratio

Visualize two probability distributions PP and QQ (both Gaussian with different parameters) and their Radon-Nikodym derivative dP/dQ=fP(x)/fQ(x)dP/dQ = f_P(x)/f_Q(x). The derivative shows where PP places relatively more mass than QQ.

Parameters
1
1
0
1

The Neyman-Pearson Lemma in Radon-Nikodym Language

The Neyman-Pearson lemma (Book FSI, Chapter 2) states that the most powerful test of H0:P=P0H_0: P = P_0 versus H1:P=P1H_1: P = P_1 at level α\alpha rejects H0H_0 when L(ω)>ηL(\omega) > \eta, where L=dP1/dP0L = dP_1/dP_0 is the likelihood ratio.

In the Radon-Nikodym framework, this is completely general: it works even when P0,P1P_0, P_1 are measures on infinite-dimensional spaces (e.g., the path space of a stochastic process). This is how one formulates detection of signals in continuous-time noise — the Cameron-Martin theorem gives the explicit form of dP1/dP0dP_1/dP_0 for Gaussian processes.

Theorem: Change of Measure Formula

If ν≪μ\nu \ll \mu with f=dν/dμf = d\nu/d\mu, then for any measurable g≥0g \geq 0: ∫g dν=∫g⋅f dμ.\int g\, d\nu = \int g \cdot f\, d\mu. In probability: if EQ\mathbb{E}_Q denotes expectation under QQ and L=dP/dQL = dP/dQ, then EP[g(X)]=EQ[g(X)⋅L(X)].\mathbb{E}_P[g(X)] = \mathbb{E}_Q[g(X) \cdot L(X)].

To compute an expectation under PP, you can instead compute a weighted expectation under QQ, where the weight is the likelihood ratio. This is the foundation of importance sampling — a Monte Carlo technique where you sample from a convenient distribution QQ and reweight by dP/dQdP/dQ.

Proof
Simple functions

For g=1Ag = \mathbf{1}_A: ∫1A dν=ν(A)=∫Af dμ=∫1Af dμ\int \mathbf{1}_A\, d\nu = \nu(A) = \int_A f\, d\mu = \int \mathbf{1}_A f\, d\mu. By linearity, the formula holds for simple functions.

General case via MCT

For g≥0g \geq 0, approximate gg from below by simple functions gn↑gg_n \uparrow g. By MCT applied to both sides: ∫g dν=lim⁡n∫gn dν=lim⁡n∫gnf dμ=∫gf dμ\int g\, d\nu = \lim_n \int g_n\, d\nu = \lim_n \int g_n f\, d\mu = \int gf\, d\mu. ■\blacksquare

Example: Importance Sampling for Rare Event Estimation

Estimate P(X>5)P(X > 5) where X∼N(0,1)X \sim \mathcal{N}(0, 1) using importance sampling with proposal Q=N(5,1)Q = \mathcal{N}(5, 1).

Solution
Change of measure

P(X>5)=EP[1X>5]=EQ[1X>5⋅L(X)]P(X > 5) = \mathbb{E}_P[\mathbf{1}_{X > 5}] = \mathbb{E}_Q[\mathbf{1}_{X > 5} \cdot L(X)] where L(x)=dPdQ(x)=e−(x2/2−(x−5)2/2)=e−5x+25/2L(x) = \frac{dP}{dQ}(x) = e^{-(x^2/2 - (x-5)^2/2)} = e^{-5x + 25/2}.

Monte Carlo estimator

Sample X1,…,Xn∼N(5,1)X_1, \ldots, X_n \sim \mathcal{N}(5, 1) and compute: p^=1n∑i=1n1Xi>5⋅e−5Xi+25/2.\hat{p} = \frac{1}{n} \sum_{i=1}^n \mathbf{1}_{X_i > 5} \cdot e^{-5X_i + 25/2}. Under Q=N(5,1)Q = \mathcal{N}(5,1), about half the samples exceed 5, so the indicator fires frequently — unlike naive Monte Carlo under PP where P(X>5)≈2.87×10−7P(X > 5) \approx 2.87 \times 10^{-7} and you would need billions of samples.

Why This Matters: From Radon-Nikodym to Radar Detection

In radar and sonar, the received signal is modeled as a continuous-time stochastic process. Testing whether a target is present (signal + noise vs. noise alone) is a hypothesis test between two measures on the space of sample paths. The Radon-Nikodym derivative dP1/dP0dP_1/dP_0 for Gaussian processes is given by the Cameron-Martin formula: dP1dP0=exp⁡(∫0Ts(t) dX(t)−12∫0Ts2(t) dt),\frac{dP_1}{dP_0} = \exp\left(\int_0^T s(t)\, dX(t) - \frac{1}{2}\int_0^T s^2(t)\, dt\right), where s(t)s(t) is the known signal waveform and X(t)X(t) is the observed process. The sufficient statistic is the correlator output ∫0Ts(t) dX(t)\int_0^T s(t)\, dX(t) — the continuous-time matched filter from Chapter 15, now justified measure-theoretically.

⚠️Engineering Note

Importance Sampling in BER Estimation

In communication systems, bit error rates (BER) below 10−610^{-6} are common design targets. Naive Monte Carlo simulation requires ∼108\sim 10^8 samples to estimate a BER of 10−610^{-6} with reasonable confidence. Importance sampling, using the change-of-measure formula, shifts the noise distribution to make errors more likely and reweights by dP/dQdP/dQ. This can reduce the required sample count by orders of magnitude.

The optimal importance sampling distribution for Gaussian channels shifts the noise mean to the decision boundary — the theoretical justification comes directly from the Radon-Nikodym theorem.

Types of Densities via Radon-Nikodym

SettingDominating measure μ\muRadon-Nikodym derivative dν/dμd\nu/d\mu
Continuous RVLebesgue measure λ\lambdaPDF fX(x)f_X(x)
Discrete RVCounting measurePMF pX(x)p_X(x)
Hypothesis testingP0P_0 (null hypothesis)Likelihood ratio L=f1/f0L = f_1/f_0
Bayesian posteriorPrior π\piPosterior density ∝L⋅π\propto L \cdot \pi
,

Common Mistake: Likelihood Ratio Undefined When P1≪̸P0P_1 \not\ll P_0

Mistake:

Computing L(x)=f1(x)/f0(x)L(x) = f_1(x)/f_0(x) and ignoring points where f0(x)=0f_0(x) = 0 but f1(x)>0f_1(x) > 0.

Correction:

If P1P_1 is not absolutely continuous with respect to P0P_0 (i.e., there exist sets where P0P_0 gives zero probability but P1P_1 does not), the Radon-Nikodym derivative does not exist. In hypothesis testing, this means the two hypotheses are "partially distinguishable with certainty" — you can perfectly detect H1H_1 on the set where P0=0P_0 = 0. The general Lebesgue decomposition P1=(P1)a+(P1)sP_1 = (P_1)_a + (P_1)_s handles this case.

Quick Check

The Radon-Nikodym derivative dP/dQdP/dQ exists when:

PP and QQ have the same support

P≪QP \ll Q (P is absolutely continuous w.r.t. Q)

P⊥QP \perp Q (P and Q are mutually singular)

Correction:
P≪QP \ll Q (P is absolutely continuous w.r.t. Q)

Absolute continuity is the necessary and sufficient condition for the Radon-Nikodym derivative to exist.

Radon-Nikodym Derivative

The measurable function f=dν/dμf = d\nu/d\mu satisfying ν(A)=∫Af dμ\nu(A) = \int_A f\, d\mu for all measurable AA. Exists when ν≪μ\nu \ll \mu; unique μ\mu-a.e. Generalizes the notion of PDF, PMF, and likelihood ratio.

Related: Absolute Continuity of Measures, Likelihood Ratio

Absolute Continuity (of Measures)

ν≪μ\nu \ll \mu means every μ\mu-null set is also ν\nu-null: μ(A)=0⇒ν(A)=0\mu(A) = 0 \Rightarrow \nu(A) = 0. Equivalent to ν\nu having a density (Radon-Nikodym derivative) with respect to μ\mu.

Related: Radon-Nikodym Theorem, Singular Measures

Key Takeaway

The Radon-Nikodym theorem unifies PDFs, PMFs, and likelihood ratios under a single concept: the derivative of one measure with respect to another. The likelihood ratio dP1/dP0dP_1/dP_0 is the central object of hypothesis testing and drives the Neyman-Pearson lemma, Wald's SPRT, and importance sampling. The measure-theoretic viewpoint extends all of these to settings — like continuous-time processes and infinite-dimensional spaces — where classical densities do not exist.

Conditional Expectation Given a Sigma-AlgebraChapter Summary