Prerequisites & Notation

Before You Begin

This chapter extends information measures to continuous random variables. We assume mastery of the discrete case from Chapter 1 and basic familiarity with continuous probability.

Entropy, mutual information, KL divergence (Chapter 1)(Review ita/ch01)
Self-check: Can you state the information inequality and its consequences?
Probability density functions, expectation, variance
Self-check: Can you compute $\mathbb{E}[g(X)]$ for a continuous RV with PDF $f_X$ ?
Gaussian distribution: PDF, moments, moment-generating function
Self-check: Can you write the PDF of $\mathcal{N}(\mu, \sigma^2)$ and compute its variance?
Multivariate Gaussian: joint PDF, covariance matrix, conditional distributions
Self-check: Can you write the joint PDF of $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ and state the conditional distribution formula?
Integration by parts, change of variables
Self-check: Can you evaluate $\int_0^\infty x e^{-x} dx$ ?
Determinants and positive definite matrices(Review telecom/ch01)
Self-check: Can you compute the determinant of a $2 \times 2$ covariance matrix?

Notation for This Chapter

Symbols introduced in this chapter. See Chapter 1 for the discrete information measures that carry over.

Symbol	Meaning	Introduced
$f_X(x)$	Probability density function (PDF) of continuous RV $X$	s01
$h(X)$	Differential entropy: $-\int f(x) \log f(x)\,dx$	s01
$h(X\|Y)$	Conditional differential entropy	s01
$\mathcal{N}(\mu, \sigma^2)$	Gaussian distribution with mean $\mu$ and variance $\sigma^2$	s02
$\boldsymbol{\Sigma}$	Covariance matrix $\boldsymbol{\Sigma} = \mathbb{E}[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^T]$	s03
$N(X)$	Entropy power: $N(X) = \frac{1}{2\pi e} 2^{2h(X)}$	s04
$X^\Delta$	Quantized version of $X$ with bin width $\Delta$	s05

← Ch 1 Differential Entropy