Hypothesis Testing and Confidence Intervals

A hypothesis test consists of:

• Null hypothesis $H_0$: the default assumption (e.g., "the data comes from distribution $F_0$")
• Alternative hypothesis $H_1$: what we accept if $H_0$ is rejected
• Test statistic $T$: a function of the data
• p-value: $P(T \ge t_{\mathrm{obs}} \mid H_0)$ — the probability of seeing a result at least as extreme under $H_0$
• Significance level $\alpha$: reject $H_0$ if $p < \alpha$ (typically $\alpha = 0.05$)

from scipy.stats import ttest_1samp
stat, p_value = ttest_1samp(samples, popmean=0.0)
if p_value < 0.05:
    print("Reject H0 at 5% significance level")

• Type I error rate = $\alpha$ = probability of false alarm
• Type II error rate = $\beta$ = probability of missed detection
• Power = $1 - \beta$ = probability of correctly rejecting a false $H_0$

In simulation: Type I error means claiming an algorithm is better when it is not (false positive); Type II means missing a real improvement.

The one-sample t-test tests whether the population mean equals a hypothesized value $\mu_0$. The test statistic is:

$$t = \frac{\bar{x} - \mu_0}{s / \sqrt{N}}$$

where $s$ is the sample standard deviation and $N$ is the sample size. Under $H_0$, $t \sim t_{N-1}$ (Student's $t$ distribution with $N-1$ degrees of freedom).

The two-sample t-test compares means of two groups:

from scipy.stats import ttest_ind
stat, p = ttest_ind(ber_algorithm_A, ber_algorithm_B)

A $(1-\alpha)$ confidence interval for parameter $\theta$ is a random interval $[L, U]$ such that:

$$P(L \le \theta \le U) = 1 - \alpha$$

For the mean of a Gaussian with unknown variance, the CI is:

$$\bar{x} \pm t_{\alpha/2, N-1} \cdot \frac{s}{\sqrt{N}}$$

from scipy.stats import t
ci_half = t.ppf(1 - alpha/2, df=N-1) * s / np.sqrt(N)
ci = (x_bar - ci_half, x_bar + ci_half)

The KS test is a nonparametric test for whether data follows a specified distribution. It compares the empirical CDF to the reference:

$$D_N = \sup_x |\hat{F}_N(x) - F_0(x)|$$

from scipy.stats import kstest, norm
D, p = kstest(data, 'norm', args=(mu_hat, sigma_hat))

Two-sample KS test compares two empirical distributions:

from scipy.stats import ks_2samp
D, p = ks_2samp(data_A, data_B)

The bootstrap estimates the sampling distribution of a statistic by resampling the data with replacement:

1. From $N$ observations, draw $B$ bootstrap samples (each of size $N$, with replacement)
2. Compute the statistic $\hat{\theta}^{(b)}$ for each bootstrap sample
3. Use the $\alpha/2$ and $1-\alpha/2$ quantiles of $\{\hat{\theta}^{(b)}\}$ as confidence bounds

rng = np.random.default_rng(42)
B = 10000
boot_stats = np.array([
    np.mean(rng.choice(data, size=len(data), replace=True))
    for _ in range(B)
])
ci = np.percentile(boot_stats, [2.5, 97.5])