Exercises

$M_X(t) = e^{-\lambda}\sum_{k=0}^{\infty}\frac{(\lambda e^t)^k}{k!} = e^{-\lambda}\cdot e^{\lambda e^t} = e^{\lambda(e^t - 1)}$.

$\phi_X(-u) = \mathbb{E}[e^{-juX}] = \mathbb{E}[\overline{e^{juX}}] = \overline{\mathbb{E}[e^{juX}]} = \overline{\phi_X(u)}$,

where the third equality uses the fact that expectation commutes with complex conjugation (since it is a real-linear operation on the real and imaginary parts).

$G_X(s) = \sum_{k=0}^{n}\binom{n}{k}(ps)^k(1-p)^{n-k} = (1-p+ps)^n$.

$G_X'(s) = np(1-p+ps)^{n-1}$, so $G_X'(1) = np$.

$m_X(t) = -\log(1 - t/\lambda)$. $m_X'(t) = 1/(\lambda - t)$, so $\kappa_1 = m_X'(0) = 1/\lambda$. $m_X''(t) = 1/(\lambda - t)^2$, so $\kappa_2 = 1/\lambda^2$. $m_X'''(t) = 2/(\lambda - t)^3$, so $\kappa_3 = 2/\lambda^3$.

$\phi_Y(u) = \mathbb{E}[e^{ju(3X+5)}] = e^{j5u}\mathbb{E}[e^{j(3u)X}] = e^{j5u}\phi_X(3u)$.

$M_{S_n}(t) = \left(\frac{\beta}{\beta-t}\right)^{n\alpha}$.

This is the MGF of $\text{Gamma}(n\alpha, \beta)$, so $S_n \sim \text{Gamma}(n\alpha, \beta)$.

$\mathbb{P}(X=k|X+Y=n) = \frac{e^{-\lambda}\lambda^k/k! \cdot e^{-\mu}\mu^{n-k}/(n-k)!}{e^{-(\lambda+\mu)}(\lambda+\mu)^n/n!} = \binom{n}{k}\left(\frac{\lambda}{\lambda+\mu}\right)^k \left(\frac{\mu}{\lambda+\mu}\right)^{n-k}$.

This is $\text{Bin}(n, \lambda/(\lambda+\mu))$.

By scaling: $\phi_X(u) = e^{-\gamma|u|}$.

$M_X(t) = \int_{-\infty}^{\infty} e^{tx}\frac{\gamma}{\pi(x^2+\gamma^2)}\,dx$. For $t > 0$, the integrand grows as $e^{tx}/(x^2+\gamma^2)$ which is not integrable as $x \to +\infty$. So $M_X(t) = \infty$ for all $t \neq 0$.

For $t > 0$: $\mathbb{P}(X \geq a) = \mathbb{P}(e^{tX} \geq e^{ta}) \leq \frac{\mathbb{E}[e^{tX}]}{e^{ta}} = e^{-ta}M_X(t)$.

Since this holds for all $t > 0$: $\mathbb{P}(X \geq a) \leq \inf_{t > 0} e^{-ta}M_X(t)$.

$G_X(s) = ps/(1-s(1-p))$ and $G_N(s) = e^{\lambda(s-1)}$.

$G_S(s) = e^{\lambda(G_X(s) - 1)} = \exp\!\left(\lambda\left(\frac{ps}{1-s(1-p)} - 1\right)\right) = \exp\!\left(\frac{-\lambda(1-s)}{1-s(1-p)}\right)$.

This is a negative binomial-like PGF. The distribution of $S$ does not simplify to a standard named distribution.

$m_X(t) = \lambda(e^t - 1) = \sum_{n=1}^{\infty}\lambda t^n/n!$, so $\kappa_n = \lambda$ for all $n$.

A $\mathcal{N}(\lambda, \lambda)$ has $\kappa_1 = \kappa_2 = \lambda$ and $\kappa_n = 0$ for $n \geq 3$. The Poisson has the same first two cumulants but all higher cumulants are nonzero and equal to $\lambda$. As $\lambda \to \infty$, the normalized Poisson approaches Gaussian (by CLT), and the higher cumulants become negligible relative to $\kappa_2$.

$\mu = G'(1) = 2p$. Critical: $p = 1/2$. Subcritical: $p < 1/2$. Supercritical: $p > 1/2$.

$s = (1-p+ps)^2$. Let $q = 1-p$. Then $s = (q+ps)^2$. Taking square roots: $\sqrt{s} = q + ps$ (nonneg. root). Let $u = \sqrt{s}$: $u = q + pu^2$, i.e., $pu^2 - u + q = 0$. $u = \frac{1 \pm \sqrt{1-4pq}}{2p}$.

For $p > 1/2$: $\eta = [(1-\sqrt{1-4pq})/(2p)]^2$. For $p \leq 1/2$: $\eta = 1$.

$e^{juX} = \sum_{i=0}^{k}\frac{(juX)^i}{i!} + R_k$ where $|R_k| \leq 2|uX|^k/k!$.

$\phi_X(u) = \sum_{i=0}^{k}\frac{(ju)^i}{i!}\mathbb{E}[X^i] + \mathbb{E}[R_k]$.

$|\mathbb{E}[R_k]| \leq 2|u|^k\mathbb{E}[|X|^k]/k!$. But we need the sharper bound: $|R_k|/|u|^k \to 0$ as $u \to 0$ (pointwise in $X$), and $|R_k|/|u|^k \leq c|X|^k$ which is integrable. By DCT, $\mathbb{E}[R_k] = o(u^k)$.

$m_X(t) = \log(1-p+pe^t)$.

$m_X'(t) = \frac{pe^t}{1-p+pe^t} = a$ gives $e^{t^*} = \frac{a(1-p)}{p(1-a)}$, $t^* = \log\frac{a(1-p)}{p(1-a)}$.

$m_X^*(a) = at^* - m_X(t^*) = a\log\frac{a}{p} + (1-a)\log\frac{1-a}{1-p} = D(\text{Ber}(a) \| \text{Ber}(p))$.

Cramer's theorem then says $\mathbb{P}(S_n/n > a) \doteq e^{-nD(\text{Ber}(a)\|\text{Ber}(p))}$, connecting large deviations directly to information theory.

Let $\mathbf{W} = \mathbf{U}^H\mathbf{Z} \sim \mathcal{CN}(\mathbf{0}, \mathbf{I})$. Then $\Delta = \sum_{i=1}^n \xi_i|W_i|^2$.

$|W_i|^2 \sim \text{Exp}(1)$, so the CF of $\xi_i|W_i|^2$ is $(1 - j\xi_i u)^{-1}$.

$\Phi_\Delta(u) = \prod_{i=1}^n\frac{1}{1 - j\xi_i u} = \frac{1}{\det(\mathbf{I} - ju\mathbf{F})}$.

The function $g(x) = e^{jux}$ is bounded ($|g| \leq 1$) and continuous. Since $F_n \to F$ at all continuity points:

$\phi_n(u) = \int e^{jux}\,dF_n(x) \to \int e^{jux}\,dF(x) = \phi(u)$.

The Helly-Bray theorem applies because $F_n, F$ are distribution functions and $g$ is bounded continuous. (Technically, one must also handle the possibility of mass escaping to infinity, which requires tightness — but convergence to a CDF $F$ implies tightness.)

$m_X(t) = t^2/2$. $m_X^*(a) = \sup_t\{at - t^2/2\} = a^2/2$ (achieved at $t = a$).

$\mathbb{P}(S_n/n > a) \doteq e^{-na^2/2}$.

$\mathbb{P}(S_n/n > a) = \mathbb{P}(Z > a\sqrt{n}) = Q(a\sqrt{n}) \sim \frac{1}{a\sqrt{2\pi n}}e^{-na^2/2}$.

The exponential rate is indeed $a^2/2$. Cramer's theorem captures the dominant exponential behavior; the prefactor $1/(a\sqrt{2\pi n})$ is subexponential and invisible on the log scale.

$G_1(s) = q/(1-ps)$. $G_2(s) = G(G_1(s)) = q/(1-p\cdot q/(1-ps)) = q(1-ps)/(1-ps-pq)$.

For $p = q = 1/2$: $G_n(s) = \frac{n - (n-1)s}{n+1-ns}$.

$G_n(0) = n/(n+1) \to 1$ as $n \to \infty$ when $p = q = 1/2$.

For general $p \neq q$: $\eta = \min\{1, q/p\}$. Since $\mu = p/q$, extinction is certain iff $p \leq q$ ($\mu \leq 1$).

$\phi_{(X+Y)/2}(u) = \phi_{X+Y}(u/2) = \phi_X(u/2)\phi_Y(u/2) = e^{-|u/2|}e^{-|u/2|} = e^{-|u|}$.

This is the CF of $\text{Cauchy}(0, 1)$. By uniqueness, $(X+Y)/2 \sim \text{Cauchy}(0, 1)$.

This means averaging two i.i.d. Cauchy RVs does not reduce the spread — the sample mean has the same distribution as a single observation! This is why the LLN fails for the Cauchy: it has no finite mean.

$\phi_X(u) = 1 - u^2/2 + O(|u|^3\rho)$. $\phi_{U_n}(u) = [\phi_X(u/\sqrt{n})]^n = [1 - u^2/(2n) + O(\rho|u|^3/n^{3/2})]^n$.

Using $|(1+z/n)^n - e^z| \leq |z|^2 e^{|z|}/n$ for suitable $z$:

$|\phi_{U_n}(u) - e^{-u^2/2}| = O(\rho|u|^3/\sqrt{n})$ for bounded $u$.

By the Berry-Esseen smoothing lemma, $\sup_x|F_{U_n}(x) - \Phi(x)| \leq C\rho/\sqrt{n}$ with $C < 0.4748$.