Exercises

$\rho_{12} = \boldsymbol{\Sigma}_{12}/\sqrt{\boldsymbol{\Sigma}_{11}\boldsymbol{\Sigma}_{22}} = 1/\sqrt{4 \cdot 2} = 1/(2\sqrt{2}) \approx 0.354$.

$\mathbf{Y} \sim \mathcal{N}(3 \cdot \mathbf{0} + \mathbf{1}, 3\mathbf{I} \cdot \mathbf{I} \cdot 3\mathbf{I}) = \mathcal{N}(\mathbf{1}, 9\mathbf{I}_n)$.

$\text{Var}(X_1 - X_2) = \mathbf{a}^T \boldsymbol{\Sigma} \mathbf{a} = 1 - 2(0.5) + 1 = 1$.

$\mathbb{E}[|Z|^2] = 4$. $\text{Var}(\text{Re}(Z)) = 4/2 = 2$. $|Z|$ is Rayleigh with parameter $\sqrt{2}$: $f_{|Z|}(r) = \frac{r}{2}\exp(-r^2/4)$, $r \geq 0$.

$\mathbb{E}[Q] = 3$ and $\text{Var}(Q) = 2 \cdot 3 = 6$.

$\boldsymbol{\Sigma}_{11} = 4$, $\boldsymbol{\Sigma}_{12} = (2, 0)$, $\boldsymbol{\Sigma}_{22} = \begin{pmatrix} 5 & 1 \\ 1 & 3 \end{pmatrix}$.

$\det(\boldsymbol{\Sigma}_{22}) = 14$, so $\boldsymbol{\Sigma}_{22}^{-1} = \frac{1}{14}\begin{pmatrix} 3 & -1 \\ -1 & 5 \end{pmatrix}$.

$\mu_{1|23} = 1 + (2, 0)\frac{1}{14}\begin{pmatrix} 3 & -1 \\ -1 & 5 \end{pmatrix}\begin{pmatrix} 4-2 \\ 2-3 \end{pmatrix} = 1 + \frac{1}{14}(6, -2)\begin{pmatrix} 2 \\ -1 \end{pmatrix} = 1 + \frac{14}{14} = 2$.

$\sigma^2_{1|23} = 4 - (2, 0)\frac{1}{14}\begin{pmatrix} 3 & -1 \\ -1 & 5 \end{pmatrix}\begin{pmatrix} 2 \\ 0 \end{pmatrix} = 4 - \frac{6}{14} \cdot 2 = 4 - 6/7 = 22/7$.

So $X_1 | (X_2, X_3) = (4, 2) \sim \mathcal{N}(2, 22/7)$.

$\mathbf{Y} \sim \mathcal{N}(\mathbf{A}\boldsymbol{\mu}, \mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^T)$.

$\mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^T$ and $\boldsymbol{\Sigma}$ are similar (since $\mathbf{A}$ is orthogonal), so they share the same eigenvalues. An orthogonal transformation rotates the Gaussian cloud without changing the principal axis lengths.

$\mathbf{L} = \begin{pmatrix} \sqrt{2} & 0 \\ 1/\sqrt{2} & \sqrt{3/2} \end{pmatrix}$.

$\mathbf{L}^{-1} = \begin{pmatrix} 1/\sqrt{2} & 0 \\ -1/\sqrt{3} & \sqrt{2/3} \end{pmatrix}$.

$\mathbf{L}^{-1}\boldsymbol{\Sigma}(\mathbf{L}^{-1})^T = \mathbf{L}^{-1}\mathbf{L}\mathbf{L}^T(\mathbf{L}^T)^{-1} = \mathbf{I}_2$.

$|\phi(\boldsymbol{\omega})| = |\exp(j\boldsymbol{\omega}^T\boldsymbol{\mu})| \cdot \exp(-\frac{1}{2}\boldsymbol{\omega}^T\boldsymbol{\Sigma}\boldsymbol{\omega}) = \exp(-\frac{1}{2}\boldsymbol{\omega}^T\boldsymbol{\Sigma}\boldsymbol{\omega}) \leq 1$ since $\boldsymbol{\omega}^T\boldsymbol{\Sigma}\boldsymbol{\omega} \geq 0$.

$|\phi(\boldsymbol{\omega})| = 1$ iff $\boldsymbol{\omega}^T\boldsymbol{\Sigma}\boldsymbol{\omega} = 0$. If $\boldsymbol{\Sigma} \succ 0$, this holds only for $\boldsymbol{\omega} = \mathbf{0}$.

$\|\mathbf{X}\|^2 = \sigma^2 Q$ where $Q \sim \chi^2_n$. $\mathbb{E}[\|\mathbf{X}\|^4] = \sigma^4 \mathbb{E}[Q^2] = \sigma^4(\text{Var}(Q) + (\mathbb{E}[Q])^2) = \sigma^4(2n + n^2) = \sigma^4 n(n+2)$.

$\log f(\mathbf{x}) = -\frac{n}{2}\log(2\pi) - \frac{1}{2}\log\det(\boldsymbol{\Sigma}) - \frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})$.

The quadratic form has expectation $\operatorname{tr}(\boldsymbol{\Sigma}^{-1}\boldsymbol{\Sigma}) = n$. Therefore

$h(\mathbf{X}) = \frac{n}{2}\log(2\pi) + \frac{1}{2}\log\det(\boldsymbol{\Sigma}) + \frac{n}{2} = \frac{1}{2}\log((2\pi e)^n\det(\boldsymbol{\Sigma}))$.

$\text{Cov}(\bar{\mathbf{X}}, \mathbf{X}_i - \bar{\mathbf{X}}) = \text{Cov}(\bar{\mathbf{X}}, \mathbf{X}_i) - \text{Cov}(\bar{\mathbf{X}}, \bar{\mathbf{X}}) = \boldsymbol{\Sigma}/m - \boldsymbol{\Sigma}/m = \mathbf{0}$.

Since $(\bar{\mathbf{X}}, \mathbf{X}_1 - \bar{\mathbf{X}}, \ldots, \mathbf{X}_m - \bar{\mathbf{X}})$ is jointly Gaussian (as an affine function of the i.i.d. data) and $\bar{\mathbf{X}}$ is uncorrelated with each centered observation, they are independent. Since $\mathbf{S}$ is a function of $\{\mathbf{X}_i - \bar{\mathbf{X}}\}$, it is independent of $\bar{\mathbf{X}}$.

Factor $\mathbf{M} = \begin{pmatrix}\mathbf{I} & \mathbf{B}\mathbf{D}^{-1} \\ \mathbf{0} & \mathbf{I}\end{pmatrix}\begin{pmatrix}\mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{D}\end{pmatrix}\begin{pmatrix}\mathbf{I} & \mathbf{0} \\ \mathbf{D}^{-1}\mathbf{C} & \mathbf{I}\end{pmatrix}$ where $\mathbf{S} = \mathbf{A} - \mathbf{B}\mathbf{D}^{-1}\mathbf{C}$.

The inverse of the middle factor is $\operatorname{diag}(\mathbf{S}^{-1}, \mathbf{D}^{-1})$. The $(1,1)$ block of $\mathbf{M}^{-1}$ picks up $\mathbf{S}^{-1} = (\mathbf{A} - \mathbf{B}\mathbf{D}^{-1}\mathbf{C})^{-1}$.

$0 \leq D(f_{\mathbf{X}} \| f_{\mathbf{G}}) = \int f_{\mathbf{X}} \log \frac{f_{\mathbf{X}}}{f_{\mathbf{G}}} = -h(\mathbf{X}) - \int f_{\mathbf{X}} \log f_{\mathbf{G}}$.

$-\int f_{\mathbf{X}} \log f_{\mathbf{G}} = \frac{n}{2}\log(2\pi) + \frac{1}{2}\log\det(\boldsymbol{\Sigma}) + \frac{1}{2}\mathbb{E}_{\mathbf{X}}[(\mathbf{X}-\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\mathbf{X}-\boldsymbol{\mu})]$.

The last term equals $\operatorname{tr}(\boldsymbol{\Sigma}^{-1}\boldsymbol{\Sigma}) = n$ since $\mathbf{X}$ and $\mathbf{G}$ share the same covariance. So $-\int f_{\mathbf{X}} \log f_{\mathbf{G}} = h(\mathbf{G})$.

$D(f_{\mathbf{X}} \| f_{\mathbf{G}}) = h(\mathbf{G}) - h(\mathbf{X}) \geq 0$, hence $h(\mathbf{X}) \leq h(\mathbf{G})$.

Let $\boldsymbol{\Sigma} = \boldsymbol{\Sigma}_{R} + j\boldsymbol{\Sigma}_{I}$ with $\boldsymbol{\Sigma}_{R} = \boldsymbol{\Sigma}_{R}^{T}$ and $\boldsymbol{\Sigma}_{I} = -\boldsymbol{\Sigma}_{I}^{T}$ (Hermitian symmetry). Properness gives:

$\boldsymbol{\Sigma}_{\text{Re}} = \boldsymbol{\Sigma}_{\text{Im}} = \frac{1}{2}\boldsymbol{\Sigma}_{R}$, $\boldsymbol{\Sigma}_{\text{Re,Im}} = \frac{1}{2}\boldsymbol{\Sigma}_{I}$.

$\boldsymbol{\Sigma}_{\hat{\mathbf{Z}}} = \frac{1}{2}\begin{pmatrix} \boldsymbol{\Sigma}_{R} & \boldsymbol{\Sigma}_{I} \\ -\boldsymbol{\Sigma}_{I} & \boldsymbol{\Sigma}_{R} \end{pmatrix}.$$So$\hat{\mathbf{Z}} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma}_{\hat{\mathbf{Z}}})$.

By the conditional Gaussian formula, $Y = (\sigma_{XY}/\sigma_X^2) X + W$ where $W$ is independent of $X$ and $\sigma_{XY} = \text{Cov}(X,Y)$.

$\text{Cov}(g(X), Y) = (\sigma_{XY}/\sigma_X^2)\text{Cov}(g(X), X)$. Integration by parts: $\text{Cov}(g(X), X) = \mathbb{E}[g(X)X] - \mathbb{E}[g(X)]\mathbb{E}[X] = \mathbb{E}[g(X)X]$ (since $\mathbb{E}[X] = 0$).

$\mathbb{E}[g(X)X] = \int g(x) x \frac{1}{\sqrt{2\pi}\sigma_X}e^{-x^2/(2\sigma_X^2)} dx = \sigma_X^2 \int g(x) \frac{d}{dx}\left[\frac{1}{\sqrt{2\pi}\sigma_X}e^{-x^2/(2\sigma_X^2)}\right](-1) dx$

Integrating by parts: $= \sigma_X^2 \mathbb{E}[g'(X)]$. Therefore $\text{Cov}(g(X), Y) = \sigma_{XY}\mathbb{E}[g'(X)] = \text{Cov}(X,Y)\mathbb{E}[g'(X)]$.

The joint density of the ordered eigenvalues $\lambda_1 > \cdots > \lambda_n > 0$ of $\mathbf{W} \sim \mathcal{W}_n(m, \mathbf{I})$ is

$$f(\lambda_1, \ldots, \lambda_n) \propto \prod_{i<j}(\lambda_i - \lambda_j)^2 \prod_i \lambda_i^{m-n} e^{-\lambda_i}.$$

The Vandermonde factor $\prod_{i<j}(\lambda_i - \lambda_j)^2$ vanishes whenever two eigenvalues coincide. Since this factor multiplies the density, the set $\{\lambda_i = \lambda_j\}$ has zero probability measure. Hence the eigenvalues are almost surely distinct.

With $X_1 = X_2 = X$ and $X_3 = X_4 = Y$:

$\mathbb{E}[X^2 Y^2] = \mathbb{E}[XX]\mathbb{E}[YY] + \mathbb{E}[XY]\mathbb{E}[XY] + \mathbb{E}[XY]\mathbb{E}[XY] = 1 + 2\rho^2$.

$\mathbf{x}^T\boldsymbol{\Sigma}^{-1}\mathbf{x} = \|\boldsymbol{\Sigma}^{-1/2}\mathbf{x}\|^2 = \|\mathbf{w}\|^2$. So $\mathbb{P}(\mathbf{X} \in \mathcal{E}_c) = \mathbb{P}(\|\mathbf{W}\|^2 \leq c) = \mathbb{P}(\chi^2_n \leq c)$.

$\frac{\partial \log p}{\partial \mu_i} = \frac{X_i - \mu_i}{\sigma_i^2}$, $\frac{\partial \log p}{\partial \sigma_i^2} = -\frac{1}{2\sigma_i^2} + \frac{(X_i - \mu_i)^2}{2\sigma_i^4}$.

$\mathbf{J} = \operatorname{diag}\!\left(\frac{1}{\sigma_1^2}, \ldots, \frac{1}{\sigma_n^2}, \frac{1}{2\sigma_1^4}, \ldots, \frac{1}{2\sigma_n^4}\right)$.

The off-diagonal blocks are zero by independence. Each component contributes a $2 \times 2$ block $\operatorname{diag}(1/\sigma_i^2, 1/(2\sigma_i^4))$.