Composite Hypotheses and the GLRT

When the Signal Is Only Partially Known

In practice the received signal is rarely known exactly. A radar return has an unknown amplitude set by the target's radar cross-section; a carrier arrives with an unknown phase; a pulse is delayed by an unknown propagation time. We must detect whether something is there even though what it looks like lies in a parametric family. This is the domain of composite hypothesis testing, for which the generalised likelihood ratio test (GLRT) is the workhorse.

Definition:

Composite Hypothesis Testing

Let {p(y;θ)}θ∈Θ\{p(\mathbf{y};\boldsymbol{\theta})\}_{\boldsymbol{\theta}\in\Theta} be a family of densities indexed by a parameter vector θ\boldsymbol{\theta}. A composite hypothesis testing problem is a pair

H0:θ∈Θ0,H1:θ∈Θ1,\mathcal{H}_0: \boldsymbol{\theta} \in \Theta_0, \qquad \mathcal{H}_1: \boldsymbol{\theta} \in \Theta_1,

where Θ0,Θ1\Theta_0, \Theta_1 are disjoint subsets of Θ\Theta. When at least one Θj\Theta_j contains more than a single point, the hypothesis is called composite; otherwise simple.

The LRT of Chapter 1 is not directly applicable: there is no single density against which to form a ratio. We need a principled way to collapse the parameter uncertainty.

Definition:

Generalised Likelihood Ratio Test (GLRT)

The GLRT replaces the unknown parameters under each hypothesis with their maximum-likelihood estimates and forms the ratio

L~(y)  =  max⁡θ∈Θ1p(y;θ)max⁡θ∈Θ0p(y;θ).\widetilde{L}(\mathbf{y}) \;=\; \frac{\max_{\boldsymbol{\theta}\in\Theta_1} p(\mathbf{y};\boldsymbol{\theta})}{\max_{\boldsymbol{\theta}\in\Theta_0} p(\mathbf{y};\boldsymbol{\theta})}.

The GLRT decides H1\mathcal{H}_1 if L~(y)>η\widetilde{L}(\mathbf{y}) > \eta for a threshold η\eta chosen to achieve a prescribed false-alarm rate.

Unlike the Neyman--Pearson LRT, the GLRT is not guaranteed to be uniformly most powerful. It is, however, (i) easily derived whenever the MLE is tractable, (ii) asymptotically optimal under mild regularity, and (iii) invariant under reparametrisation --- which is why it dominates engineering practice.

Definition:

Non-Coherent Detection

Detection of a signal whose carrier phase is unknown and treated as a nuisance parameter is called non-coherent detection. Formally, let s(t;ϕ)=a(t)cos⁡(2πfct+ϕ)s(t; \phi) = a(t) \cos(2\pi f_c t + \phi) with ϕ∈[0,2π)\phi \in [0, 2\pi) unknown uniform. The GLRT over ϕ\phi yields the envelope detector.

Theorem: GLRT for Unknown-Amplitude Signal in AWGN

Consider the composite hypothesis

H0:y=w,H1:y=A s+w,A∈R∖{0}\mathcal{H}_0: \mathbf{y} = \mathbf{w}, \qquad \mathcal{H}_1: \mathbf{y} = A\,\mathbf{s} + \mathbf{w}, \quad A \in \mathbb{R}\setminus\{0\}

with w∼N(0,σ2I)\mathbf{w}\sim\mathcal{N}(\mathbf{0},\sigma^2\mathbf{I}) and known unit-energy template s\mathbf{s}. The GLRT statistic is

ℓ~(y)  =  (sTy)22σ2,\widetilde{\ell}(\mathbf{y}) \;=\; \frac{(\mathbf{s}^{\mathsf{T}}\mathbf{y})^2}{2\sigma^2},

and the test decides H1\mathcal{H}_1 iff ∣sTy∣>γ|\mathbf{s}^{\mathsf{T}}\mathbf{y}| > \gamma. Under H0\mathcal{H}_0, 2ℓ~(y)∼χ122\widetilde{\ell}(\mathbf{y}) \sim \chi_1^2; under H1\mathcal{H}_1 it is non-central χ12\chi_1^2 with non-centrality A2/σ2A^2/\sigma^2.

Without knowing the sign of AA, the detector cannot meaningfully compare the correlator to a single threshold; it compares its magnitude, yielding a two-sided test.

Proof
MLE of $A$ under $\mathcal{H}_1$.

The log-likelihood under H1\mathcal{H}_1 is

log⁡p1(y;A)=−12σ2∥y−As∥2+const.\log p_1(\mathbf{y}; A) = -\tfrac{1}{2\sigma^2}\|\mathbf{y} - A\mathbf{s}\|^2 + \text{const.}

Setting the derivative to zero:

∂∂Alog⁡p1=1σ2sT(y−As)=0  ⟹  A^ML=sTy.\frac{\partial}{\partial A}\log p_1 = \tfrac{1}{\sigma^2}\mathbf{s}^{\mathsf{T}}(\mathbf{y} - A\mathbf{s}) = 0 \;\Longrightarrow\; \widehat{A}_{\mathrm{ML}} = \mathbf{s}^{\mathsf{T}}\mathbf{y}.

(Using ∥s∥2=1\|\mathbf{s}\|^2 = 1.)

Profile likelihood.

Substituting A^ML\widehat{A}_{\mathrm{ML}} into log⁡p1\log p_1,

max⁡Alog⁡p1(y;A)=−12σ2(∥y∥2−(sTy)2)+const.\max_A \log p_1(\mathbf{y}; A) = -\tfrac{1}{2\sigma^2}\bigl(\|\mathbf{y}\|^2 - (\mathbf{s}^{\mathsf{T}}\mathbf{y})^2\bigr) + \text{const.}

Under H0\mathcal{H}_0 there is nothing to maximise: log⁡p0(y)=−∥y∥2/(2σ2)+const.\log p_0(\mathbf{y}) = -\|\mathbf{y}\|^2/(2\sigma^2) + \text{const.}

Form the GLRT statistic.

log⁡L~(y)=max⁡Alog⁡p1−log⁡p0=(sTy)22σ2.\log \widetilde{L}(\mathbf{y}) = \max_A\log p_1 - \log p_0 = \frac{(\mathbf{s}^{\mathsf{T}}\mathbf{y})^2}{2\sigma^2}.ComparingtoanythresholdisequivalenttocomparingComparing to any threshold is equivalent to comparing|\mathbf{s}^{\mathsf{T}}\mathbf{y}|toathresholdto a threshold\gamma$.

Distribution of the statistic.

Under H0\mathcal{H}_0, sTy/σ∼N(0,1)\mathbf{s}^{\mathsf{T}}\mathbf{y}/\sigma \sim \mathcal{N}(0,1), so its square is χ12\chi_1^2. Under H1\mathcal{H}_1, sTy/σ∼N(A/σ,1)\mathbf{s}^{\mathsf{T}}\mathbf{y}/\sigma \sim \mathcal{N}(A/\sigma,1), so its square is non-central χ12\chi_1^2 with parameter A2/σ2A^2/\sigma^2. ■\blacksquare

Theorem: GLRT for Unknown-Phase Signal Yields the Envelope Detector

Consider a signal with two known real quadrature components sI,sQ∈Rn\mathbf{s}_I, \mathbf{s}_Q \in \mathbb{R}^n of equal norm ∥sI∥=∥sQ∥=Es/2\|\mathbf{s}_I\| = \|\mathbf{s}_Q\| = \sqrt{E_s/2} and sI⊥sQ\mathbf{s}_I \perp \mathbf{s}_Q, parameterised by an unknown phase ϕ∈[0,2π)\phi \in [0, 2\pi):

H1:y  =  cos⁡ϕ sI+sin⁡ϕ sQ+w.\mathcal{H}_1: \mathbf{y} \;=\; \cos\phi\,\mathbf{s}_I + \sin\phi\,\mathbf{s}_Q + \mathbf{w}.

The GLRT statistic is the envelope

R(y)  =  (sITy)2+(sQTy)2.R(\mathbf{y}) \;=\; \sqrt{(\mathbf{s}_I^{\mathsf{T}}\mathbf{y})^2 + (\mathbf{s}_Q^{\mathsf{T}}\mathbf{y})^2}.

Without phase information, the detector cannot distinguish II from QQ branches. The magnitude I2+Q2\sqrt{I^2 + Q^2} is phase-invariant.

Proof
Log-likelihood under $\mathcal{H}_1$.

log⁡p1(y;ϕ)=−12σ2∥y−cos⁡ϕ sI−sin⁡ϕ sQ∥2+const.KATEXPLACEHOLDER0ENDlog⁡p1(y;ϕ)=1σ2(cos⁡ϕ sITy+sin⁡ϕ sQTy)−Es4σ2+const.\log p_1(\mathbf{y};\phi) = -\tfrac{1}{2\sigma^2}\|\mathbf{y} - \cos\phi\,\mathbf{s}_I - \sin\phi\,\mathbf{s}_Q\|^2 + \text{const.}KATEXPLACEHOLDER0END\log p_1(\mathbf{y};\phi) = \tfrac{1}{\sigma^2}\bigl(\cos\phi\,\mathbf{s}_I^{\mathsf{T}}\mathbf{y} + \sin\phi\,\mathbf{s}_Q^{\mathsf{T}}\mathbf{y}\bigr) - \frac{E_s}{4\sigma^2} + \text{const.}$

Maximise over $\phi$.

Let I=sITyI = \mathbf{s}_I^{\mathsf{T}}\mathbf{y} and Q=sQTyQ = \mathbf{s}_Q^{\mathsf{T}}\mathbf{y}. The phase-dependent term is cos⁡ϕ I+sin⁡ϕ Q=Rcos⁡(ϕ−ψ)\cos\phi\, I + \sin\phi\, Q = R\cos(\phi - \psi), where R=I2+Q2R = \sqrt{I^2 + Q^2} and ψ=atan2(Q,I)\psi = \mathrm{atan2}(Q, I). The maximum over ϕ\phi is RR, attained at ϕ=ψ\phi = \psi. Hence

max⁡ϕlog⁡p1(y;ϕ)=R(y)σ2−Es4σ2+const.\max_\phi \log p_1(\mathbf{y};\phi) = \frac{R(\mathbf{y})}{\sigma^2} - \frac{E_s}{4\sigma^2} + \text{const.}

Compute the GLRT.

log⁡L~(y)=max⁡ϕlog⁡p1−log⁡p0=R(y)σ2−Es4σ2.\log\widetilde{L}(\mathbf{y}) = \max_\phi\log p_1 - \log p_0 = \frac{R(\mathbf{y})}{\sigma^2} - \frac{E_s}{4\sigma^2}.ThisisamonotonefunctionofThis is a monotone function ofR(\mathbf{y}),sothetestcomparestheenvelopetoathreshold., so the test compares the envelope to a threshold.\blacksquare$

Example: Cost of Unknown Amplitude

At PF=10−2P_F = 10^{-2} and Es/N0=5E_s/N_0 = 5 dB, compare the detection probability of (i) the clairvoyant LRT that knows A=1A = 1 exactly and (ii) the GLRT that allows AA to take any sign.

Solution
Clairvoyant LRT.

d=2⋅100.5≈2.51d = \sqrt{2 \cdot 10^{0.5}} \approx 2.51. PD=Q(Q−1(10−2)−2.51)=Q(2.326−2.51)=Q(−0.18)≈0.57P_D = Q(Q^{-1}(10^{-2}) - 2.51) = Q(2.326 - 2.51) = Q(-0.18) \approx 0.57.

GLRT (two-sided threshold).

The GLRT uses the threshold γ=σ Q−1(PF/2)\gamma = \sigma\,Q^{-1}(P_F/2) since both tails contribute to the false alarm. With Q−1(5⋅10−3)≈2.576Q^{-1}(5\cdot 10^{-3}) \approx 2.576,

PD≈Q(2.576−2.51)+Q(2.576+2.51)=Q(0.066)+Q(5.086)≈0.474+2⋅10−7≈0.474.P_D \approx Q(2.576 - 2.51) + Q(2.576 + 2.51) = Q(0.066) + Q(5.086) \approx 0.474 + 2\cdot10^{-7} \approx 0.474.

Interpretation.

The GLRT loses roughly 0.1 on PDP_D at this operating point --- the fraction of the two-sided false-alarm mass "wasted" on the wrong side. This gap shrinks at high SNR and widens at low SNR.

Example: Matched-Subspace GLRT

Suppose H1:y=Sα+w\mathcal{H}_1: \mathbf{y} = \mathbf{S}\boldsymbol{\alpha} + \mathbf{w} where S∈Rn×k\mathbf{S}\in\mathbb{R}^{n\times k} has known orthonormal columns and α∈Rk\boldsymbol{\alpha}\in\mathbb{R}^k is unknown. Derive the GLRT.

Solution
MLE of $\boldsymbol{\alpha}$.

The log-likelihood is −∥y−Sα∥2/(2σ2)+const-\|\mathbf{y}-\mathbf{S}\boldsymbol{\alpha}\|^2/(2\sigma^2) + \text{const}. Since STS=Ik\mathbf{S}^{\mathsf{T}}\mathbf{S} = \mathbf{I}_k, the MLE is α^ML=STy\widehat{\boldsymbol{\alpha}}_{\mathrm{ML}} = \mathbf{S}^{\mathsf{T}}\mathbf{y}.

Profile likelihood.

Plugging in: max⁡αlog⁡p1−log⁡p0=∥STy∥2/(2σ2)=∥PSy∥2/(2σ2)\max_{\boldsymbol{\alpha}}\log p_1 - \log p_0 = \|\mathbf{S}^{\mathsf{T}}\mathbf{y}\|^2 / (2\sigma^2) = \|\mathbf{P}_\mathbf{S}\mathbf{y}\|^2/(2\sigma^2),

where PS=SST\mathbf{P}_\mathbf{S} = \mathbf{S}\mathbf{S}^{\mathsf{T}} is the projector onto the signal subspace.

GLRT decision rule.

Decide H1\mathcal{H}_1 iff ∥PSy∥2>γ\|\mathbf{P}_\mathbf{S}\mathbf{y}\|^2 > \gamma: project the data onto the signal subspace and threshold its squared norm. The scalar GLRT of TGLRT for Unknown-Amplitude Signal in AWGN is the special case k=1k=1.

GLRT vs. LRT for Unknown Amplitude

Detection probability of the clairvoyant LRT, a mismatched LRT that assumes the wrong amplitude A0=1A_0 = 1, and the two-sided GLRT, as Es/N0E_s/N_0 varies. Vary β\beta to see the LRT's vulnerability to amplitude mismatch --- something the GLRT avoids by construction.

Parameters
0.01
1
32

Common Mistake: GLRT Is Not Always UMP

Mistake:

"The GLRT is derived from the LRT, so it must be uniformly most powerful for every composite hypothesis."

Correction:

The GLRT is a heuristic. For a two-sided Gaussian mean test (A≶0A \lessgtr 0), no UMP test even exists, so no procedure can dominate the GLRT uniformly --- but one can sometimes design locally most powerful tests that outperform the GLRT at small signal levels. Still, the GLRT is the engineering default because its derivation is mechanical and its asymptotic efficiency is well understood.

Quick Check

The envelope detector is the GLRT for which problem?

Known signal in AWGN

Signal with unknown carrier phase

Signal with unknown delay only

Signal with unknown amplitude and sign

Correction:
Signal with unknown carrier phase

The unknown phase is profiled out, leaving I2+Q2\sqrt{I^2 + Q^2}.

Quick Check

Under regularity conditions, as n→∞n\to\infty the GLRT statistic 2log⁡L~(y)2\log\widetilde{L}(\mathbf{y}) under H0\mathcal{H}_0 converges in distribution to:

A Gaussian N(0,1)\mathcal{N}(0,1).

A chi-squared χk2\chi_k^2 with k=dim⁡Θ1−dim⁡Θ0k = \dim\Theta_1 - \dim\Theta_0.

A non-central chi-squared.

The exponential distribution.

Correction:
A chi-squared χk2\chi_k^2 with k=dim⁡Θ1−dim⁡Θ0k = \dim\Theta_1 - \dim\Theta_0.

This is Wilks' theorem — the asymptotic distribution of the GLRT statistic under H0\mathcal{H}_0.

Operational Interpretation

Every practical detector that claims to "adapt" to an unknown nuisance parameter --- adaptive matched filter, CFAR, GPS acquisition with Doppler search, cell-search in LTE --- is a disguised GLRT. The max-over-parameters step is typically implemented as a bank of parallel correlators followed by a maximum operator, or (in the continuous case) a peak-finder on the correlator output. Seen through this lens the detector's structural complexity is proportional to the dimension and fineness of the nuisance-parameter search grid.

Detection of Known Signals in AWGNColored Gaussian Noise and Pre-Whitening