Exercises

$\ell_n(\theta) = k\log\theta + (n-k)\log(1-\theta)$ with $k = \sum_i y_i$.

$\partial\ell_n/\partial\theta = k/\theta - (n-k)/(1-\theta) = 0$ gives $\hat\theta_{\text{ml}} = k/n$.

$\mathbb{E}[\hat\theta_{\text{ml}}] = n^{-1}\sum_i \mathbb{E}[Y_i] = \theta$. $\blacksquare$

$\ell_n(\lambda) = -n\lambda + (\sum_i y_i)\log\lambda + \text{const}$, giving $\hat\lambda_{\text{ml}} = \bar y$.

$J_1(\lambda) = -\mathbb{E}[\partial^2\log f/\partial\lambda^2] = 1/\lambda$, so $J(\lambda) = n/\lambda$. The asymptotic variance of the MLE is $\lambda/n$, matching $\operatorname{\text{Var}}(\bar Y) = \lambda/n$ exactly. $\blacksquare$

$\ell_n(\theta) = -\tfrac{n}{2}\log(2\pi\theta) - \tfrac{1}{2\theta}\sum_i(y_i - \theta)^2$.

$\partial\ell/\partial\theta = -n/(2\theta) + \sum_i(y_i-\theta)/\theta + \tfrac{1}{2\theta^2}\sum_i(y_i-\theta)^2$. Simplify to $\partial\ell/\partial\theta = \bigl(-n\theta + \sum_i(y_i^2 - y_i\theta)\bigr)/(2\theta^2)\cdot 2 + \ldots$ (collect like terms carefully).

After simplification, the score equation is $n\theta^2 + n\theta - \sum_i y_i^2 = 0$ (using the identity $\sum_i (y_i - \theta)^2 = \sum_i y_i^2 - 2\theta\sum_i y_i + n\theta^2$). The positive root is $\hat\theta_{\text{ml}} = \tfrac{1}{2}\bigl(-1 + \sqrt{1 + 4\overline{y^2}}\bigr)$ with $\overline{y^2} = n^{-1}\sum_i y_i^2$. $\blacksquare$

With $\hat\sigma^2_{\text{ml}} = n^{-1}\sum_i y_i^2$ and $u(v) = \sqrt v$, $\hat\sigma_{\text{ml}} = \sqrt{\hat\sigma^2_{\text{ml}}} = \sqrt{n^{-1}\sum_i y_i^2}$. $\blacksquare$

$S_n = \sum_i Y_i \sim \text{Gamma}(n, \theta)$, so $\bar Y = S_n/n$ and $1/\bar Y = n/S_n$. For $Z \sim \text{Gamma}(n, \theta)$, $\mathbb{E}[1/Z] = \theta/(n-1)$ (standard identity, $n \geq 2$).

$\mathbb{E}[\hat\theta_{\text{ml}}] = n\mathbb{E}[1/S_n] = n\theta/(n-1)$. Therefore bias $= \theta/(n-1) = O(1/n)$. The bias-corrected estimator is $(n-1)/(n\bar Y)$. $\blacksquare$

$\partial^2 \log f/\partial(\sigma^2)^2 = 1/(2\sigma^4) - (y-\mu)^2/\sigma^6$. Taking negative expectation at the truth gives $J_1(\sigma^2) = 1/(2\sigma^4)$.

By asymptotic normality, $\operatorname{\text{Var}}(\hat\sigma^2_{\text{ml}}) \to 1/(nJ_1) = 2\sigma^4/n$, attaining the CRLB. $\blacksquare$

$\hat{\boldsymbol{\theta}}_{\text{ml}} = (\mathbf{A}^\mathsf{T}\mathbf{A})^{-1}\mathbf{A}^\mathsf{T}\mathbf{y}$ (ordinary least squares).

$\operatorname{\text{Var}}(\hat{\boldsymbol{\theta}}_{\text{ml}}) = \sigma^2(\mathbf{A}^\mathsf{T}\mathbf{A})^{-1}$, which equals the CRLB and makes the estimator both MVUE and BLUE (Gauss-Markov). $\blacksquare$

$\ell_n(\theta) = -n\log 2 - \sum_i |y_i - \theta|$. Maximizing is equivalent to minimizing $\sum_i |y_i - \theta|$.

The function $\theta \mapsto \sum_i |y_i - \theta|$ is convex and piecewise linear, minimized at the sample median (any median when $n$ is even). Thus $\hat\theta_{\text{ml}} = \text{median}(y_1,\ldots,y_n)$. $\blacksquare$

$\partial\ell_n/\partial\alpha = n/\alpha + n\log y_0 - \sum_i \log y_i = 0$ gives $\hat\alpha_{\text{ml}} = n/\sum_i \log(y_i/y_0)$.

$J_1(\alpha) = 1/\alpha^2$. Hence $\sqrt{n}(\hat\alpha_{\text{ml}} - \alpha) \xrightarrow{d} \mathcal{N}(0, \alpha^2)$. $\blacksquare$

$\nabla_{\boldsymbol{\beta}}\ell = \sum_i (y_i - \mu_i)\mathbf{x}_i = \mathbf{X}^\mathsf{T}(\mathbf{y} - \boldsymbol{\mu})$.

$\mathbf{J}(\boldsymbol{\beta}) = \mathbf{X}^\mathsf{T}\operatorname{diag}(\mu_1,\ldots,\mu_n)\mathbf{X}$.

$\boldsymbol{\beta}^{(k+1)} = \boldsymbol{\beta}^{(k)} + (\mathbf{X}^\mathsf{T}\operatorname{diag}(\boldsymbol{\mu}^{(k)})\mathbf{X})^{-1}\mathbf{X}^\mathsf{T}(\mathbf{y} - \boldsymbol{\mu}^{(k)})$, a weighted least squares step — the IRLS algorithm. $\blacksquare$

$\ell = -{\sigma^2}^{-1}\sum_n |Y[n] - Ae^{j(2\pi f_0 n + \phi)}|^2$.

$J(f_0) = \tfrac{8\pi^2 A^2}{\sigma^2}\sum_{n=0}^{N-1} n^2 \approx \tfrac{8\pi^2 A^2}{\sigma^2} \tfrac{N^3}{3}$.

$\operatorname{\text{Var}}(\hat f_0) \geq \tfrac{3\sigma^2}{8\pi^2 A^2 N^3} \sim N^{-3}$ — the super-efficient $N^{-3}$ scaling peculiar to frequency estimation. $\blacksquare$

$J_1(\theta) = 1/\sigma^2$, so $J(\theta)^{-1} = \sigma^2/n$.

$\operatorname{\text{Var}}(\bar Y) = \sigma^2/n = J(\theta)^{-1}$. The bound is attained exactly for all $n$, not just asymptotically. $\blacksquare$

$\mathbb{E}_{\theta_0}[\log(f_\theta(Y)/f_{\theta_0}(Y))] \leq \log \mathbb{E}_{\theta_0}[f_\theta(Y)/f_{\theta_0}(Y)] = \log 1 = 0$.

Negating, $\mathbb{E}_{\theta_0}[\log f_{\theta_0}(Y)] - \mathbb{E}_{\theta_0}[\log f_\theta(Y)] = D(f_{\theta_0}\|f_\theta) \geq 0$, with equality iff $f_\theta = f_{\theta_0}$ a.s. By identifiability this forces $\theta = \theta_0$, so $\theta_0$ is the unique population maximizer. $\blacksquare$

$\ell(\rho,\sigma^2) = -\tfrac{n}{2}\log(2\pi\sigma^2) - \tfrac{1}{2\sigma^2}\sum_{t=1}^n(y_t - \rho y_{t-1})^2$.

$\hat\sigma^2(\rho) = n^{-1}\sum_t(y_t - \rho y_{t-1})^2$. Substituting makes the profiled log-likelihood $-\tfrac{n}{2}(\log\hat\sigma^2(\rho) + 1) + \text{const}$.

Minimizing $\hat\sigma^2(\rho)$ in $\rho$ yields the least-squares estimator $\hat\rho_{\text{ml}} = \sum_t y_t y_{t-1}/\sum_t y_{t-1}^2$. $\blacksquare$

The likelihood is invariant under $(f_1, f_2) \leftrightarrow (f_2, f_1)$ permutation — at least two global maxima. Additionally, when the periodogram has two dominant peaks, the likelihood has a ridge along the line $f_1 = f_2$ that creates a saddle.

(i) Compute the periodogram and identify the $K=2$ largest peaks as initializations. (ii) Run alternating Newton updates: fix $f_2$, refine $f_1$; fix $f_1$, refine $f_2$. (iii) Alternatively apply ESPRIT/MUSIC to the sample covariance for a closed-form starting point. Without good initialization, pure Newton-Raphson will often return a wrong local maximum. $\blacksquare$

$\mathbf{A}\hat{\boldsymbol{\theta}} = \mathbf{A}(\mathbf{A}^\mathsf{T}\mathbf{A})^{-1}\mathbf{A}^\mathsf{T}\mathbf{y}$.

$\mathbf{P}_\mathbf{A}^2 = \mathbf{P}_\mathbf{A}$ and $\mathbf{P}_\mathbf{A}^\mathsf{T} = \mathbf{P}_\mathbf{A}$, so $\mathbf{P}_\mathbf{A}$ is an orthogonal projector onto $\operatorname{range}(\mathbf{A})$. The residual $\mathbf{y} - \mathbf{A}\hat{\boldsymbol{\theta}}$ is orthogonal to every column of $\mathbf{A}$. $\blacksquare$

The likelihood is zero unless $\theta \leq \min_i y_i$, then $L_n(\theta) = e^{-\sum_i (y_i - \theta)}$, increasing in $\theta$. Hence $\hat\theta_{\text{ml}} = \min_i Y_i$.

$\min_i Y_i - \theta_0 \sim \text{Exp}(n)$, so $n(\hat\theta_{\text{ml}} - \theta_0) \sim \text{Exp}(1)$, independent of $n$.

Rate is $n^{-1}$ (super-efficient), limit is exponential — not Gaussian. CRLB machinery does not apply: the support boundary breaks regularity. $\blacksquare$

$\sqrt{n}(\hat\theta_n + c/n - \theta_0) = \sqrt{n}(\hat\theta_n - \theta_0) + c/\sqrt{n}$. The second term goes to zero, so by Slutsky the limit is the same Gaussian. $\blacksquare$

Any $O(1/n)$ shift of an efficient estimator is asymptotically equivalent — bias correction at the $O(1/n)$ level does not change asymptotic variance.

$\operatorname{\text{Var}}(\hat\theta_{\text{ml}}) \to \theta^2/n$ (CRLB).

$g'(\theta) = -1/\theta^2$. Asymptotic variance of $1/\hat\theta_{\text{ml}}$ is $(1/\theta^4)(\theta^2/n) = 1/(n\theta^2)$, which is the CRLB for the mean parameter $\tau = 1/\theta$. $\blacksquare$

$\tilde{\mathbf{y}} = \mathbf{C}_w^{-1/2}\mathbf{y}$, $\tilde{\mathbf{s}}(f) = \mathbf{C}_w^{-1/2}\mathbf{s}(f)$. The whitened model has i.i.d. noise, so Theorem TGaussian Linear Model: Closed-Form MLE applies.

For fixed $f$, $\hat A(f) = \tilde{\mathbf{s}}(f)^H\tilde{\mathbf{y}}/\|\tilde{\mathbf{s}}(f)\|^2$.

The MLE of $f_0$ is $\hat f_{0,\text{ml}} = \arg\max_f |\mathbf{s}(f)^H \mathbf{C}_w^{-1}\mathbf{y}|^2 / (\mathbf{s}(f)^H \mathbf{C}_w^{-1}\mathbf{s}(f))$, the peak of the noise-whitened ("generalized") periodogram. $\blacksquare$