Ferkans — Interactive Telecom Tutor

Why the Bayesian Framework?

Variational regularization (§Regularization: Concept and General Theory) selects a single point estimate by minimizing a penalized objective. While effective, this approach does not quantify uncertainty in the reconstruction. When an algorithm declares that the reflectivity at pixel $i$ is $0.7$ , how confident should we be? Is the true value almost certainly in $[0.6, 0.8]$ , or could it be anywhere in $[0, 1]$ ?

The Bayesian framework treats the unknown scene $\boldsymbol{\gamma}$ as a random variable, encodes prior knowledge through $\pi(\boldsymbol{\gamma})$ , and produces the full posterior distribution $p(\boldsymbol{\gamma} \mid \mathbf{y})$ — from which any point estimate, interval, or decision can be derived. Uncertainty quantification is not optional for safety-critical applications (medical imaging, autonomous driving, surveillance) where a confidently wrong answer is far more dangerous than an honest admission of uncertainty.

Definition:
Bayesian Inverse Problem

Given the linear forward model $\mathbf{y} = \mathbf{A}\boldsymbol{\gamma} + \mathbf{w}$ with additive noise $\mathbf{w}$ , the Bayesian inverse problem consists of three ingredients:

Prior: A probability distribution $\pi(\boldsymbol{\gamma})$ encoding knowledge about the scene before observing data — e.g., that it is sparse.
Likelihood: The probability of observing $\mathbf{y}$ given $\boldsymbol{\gamma}$ , determined by the noise model: $p(\mathbf{y} \mid \boldsymbol{\gamma}) = p_{\mathbf{w}}(\mathbf{y} - \mathbf{A}\boldsymbol{\gamma}).$
Posterior: The conditional distribution of $\boldsymbol{\gamma}$ given $\mathbf{y}$ , obtained via Bayes' theorem.

The solution to the Bayesian inverse problem is the full posterior $p(\boldsymbol{\gamma} \mid \mathbf{y})$ , not a single point estimate.

Theorem: Bayes' Theorem for Inverse Problems

Under the forward model $\mathbf{y} = \mathbf{A}\boldsymbol{\gamma} + \mathbf{w}$ with prior $\pi(\boldsymbol{\gamma})$ and likelihood $p(\mathbf{y} \mid \boldsymbol{\gamma})$ , the posterior distribution is

$p(\boldsymbol{\gamma} \mid \mathbf{y}) = \frac{p(\mathbf{y} \mid \boldsymbol{\gamma})\,\pi(\boldsymbol{\gamma})}{\mathcal{Z}(\mathbf{y})},$

where the evidence (marginal likelihood) is

$\mathcal{Z}(\mathbf{y}) = \int p(\mathbf{y} \mid \boldsymbol{\gamma})\,\pi(\boldsymbol{\gamma})\, \mathrm{d}\boldsymbol{\gamma}.$

The posterior is well-defined whenever $0 < \mathcal{Z}(\mathbf{y}) < \infty$ .

Proof

Direct application of conditional probability

By the definition of conditional probability,

$p(\boldsymbol{\gamma} \mid \mathbf{y}) = \frac{p(\mathbf{y},\,\boldsymbol{\gamma})}{p(\mathbf{y})} = \frac{p(\mathbf{y} \mid \boldsymbol{\gamma})\,\pi(\boldsymbol{\gamma})} {\int p(\mathbf{y} \mid \boldsymbol{\gamma}')\,\pi(\boldsymbol{\gamma}')\, \mathrm{d}\boldsymbol{\gamma}'}.$

The denominator $\mathcal{Z}(\mathbf{y})$ is a normalizing constant ensuring $\int p(\boldsymbol{\gamma} \mid \mathbf{y})\,\mathrm{d}\boldsymbol{\gamma} = 1$ . $\blacksquare$

Definition:
Gaussian Likelihood for Additive Noise

When the noise is Gaussian, $\mathbf{w} \sim \mathcal{N}(0, \sigma^2 \mathbf{I})$ , the likelihood takes the form

$p(\mathbf{y} \mid \boldsymbol{\gamma}) = \frac{1}{(2\pi\sigma^2)^{m/2}} \exp\!\left(-\frac{1}{2\sigma^2} \|\mathbf{A}\boldsymbol{\gamma} - \mathbf{y}\|^2\right),$

where $m$ is the number of measurements. The negative log-likelihood is proportional to the least-squares data fidelity:

$-\log p(\mathbf{y} \mid \boldsymbol{\gamma}) = \frac{1}{2\sigma^2}\|\mathbf{A}\boldsymbol{\gamma} - \mathbf{y}\|^2 + \text{const}.$

This connects the Bayesian likelihood directly to the data-fidelity term in variational regularization (§Regularization: Concept and General Theory).

Definition:
Maximum A Posteriori (MAP) Estimate

The MAP estimate is the mode of the posterior distribution:

$\hat{\boldsymbol{\gamma}}_{\text{MAP}} = \arg\max_{\boldsymbol{\gamma}}\; p(\boldsymbol{\gamma} \mid \mathbf{y}) = \arg\min_{\boldsymbol{\gamma}}\; \bigl[-\log p(\mathbf{y} \mid \boldsymbol{\gamma}) - \log \pi(\boldsymbol{\gamma})\bigr].$

For Gaussian noise and a log-concave prior $\pi(\boldsymbol{\gamma}) \propto \exp(-\lambda R(\boldsymbol{\gamma}))$ , the MAP estimate solves

$\hat{\boldsymbol{\gamma}}_{\text{MAP}} = \arg\min_{\boldsymbol{\gamma}} \left\{ \frac{1}{2\sigma^2}\|\mathbf{A}\boldsymbol{\gamma} - \mathbf{y}\|^2 + \lambda\,R(\boldsymbol{\gamma})\right\},$

which is precisely the variational regularization problem from §Regularization: Concept and General Theory. Variational regularization is MAP estimation under the Bayesian interpretation.

Definition:
Minimum Mean Squared Error (MMSE) Estimate

The MMSE estimate (posterior mean) minimises the expected squared error under the posterior:

$\hat{\boldsymbol{\gamma}}_{\text{MMSE}} = \mathbb{E}[\boldsymbol{\gamma} \mid \mathbf{y}] = \int \boldsymbol{\gamma}\,p(\boldsymbol{\gamma} \mid \mathbf{y})\, \mathrm{d}\boldsymbol{\gamma}.$

It satisfies:

$\hat{\boldsymbol{\gamma}}_{\text{MMSE}} = \arg\min_{\hat{\boldsymbol{\gamma}}} \; \mathbb{E}\!\left[\|\boldsymbol{\gamma} - \hat{\boldsymbol{\gamma}}\|^2 \mid \mathbf{y}\right].$

For symmetric unimodal posteriors (e.g., Gaussian), MAP and MMSE coincide. For sparse, non-Gaussian posteriors they can differ substantially.

MAP vs MMSE — When Do They Differ?

The MAP and MMSE estimates coincide when the posterior is symmetric and unimodal (e.g., Gaussian). They diverge when:

The posterior is skewed — the mean is pulled away from the mode.
The posterior is multimodal — MAP selects one mode while MMSE averages over all modes, potentially landing between them.
The problem is high-dimensional — in $\mathbb{R}^n$ with large $n$ , the MAP estimate can lie in a region of low posterior probability mass (the "typical set" phenomenon from concentration of measure).

For RF imaging: MAP tends to produce sparser, edge-preserving images; MMSE produces smoother reconstructions but is more computationally demanding.

Posterior distribution

The conditional probability distribution $p(\boldsymbol{\gamma} \mid \mathbf{y})$ of the unknown scene $\boldsymbol{\gamma}$ given the measurements $\mathbf{y}$ . It combines the likelihood (data fit) and prior (regularization) through Bayes' theorem and encodes all information about the unknown after observing data.

Evidence (marginal likelihood)

The normalizing constant $\mathcal{Z}(\mathbf{y}) = \int p(\mathbf{y} \mid \boldsymbol{\gamma})\,\pi(\boldsymbol{\gamma})\,\mathrm{d}\boldsymbol{\gamma}$ of the posterior. It is the marginal probability of the data under the model and plays a central role in Bayesian model comparison and hyperparameter selection.

Theorem: Gaussian Prior Yields Gaussian Posterior

Let $\mathbf{y} = \mathbf{A}\boldsymbol{\gamma} + \mathbf{w}$ with $\mathbf{w} \sim \mathcal{N}(0, \sigma^2 \mathbf{I})$ and prior $\boldsymbol{\gamma} \sim \mathcal{N}(\boldsymbol{\gamma}_0, \mathbf{\Gamma})$ . Then the posterior is Gaussian:

$p(\boldsymbol{\gamma} \mid \mathbf{y}) = \mathcal{N}(\hat{\boldsymbol{\gamma}}_{\text{post}},\,\mathbf{\Gamma}_{\text{post}}),$

with posterior mean and covariance

$\hat{\boldsymbol{\gamma}}_{\text{post}} = \mathbf{\Gamma}_{\text{post}} \!\left(\frac{1}{\sigma^2}\mathbf{A}^H \mathbf{y} + \mathbf{\Gamma}^{-1}\boldsymbol{\gamma}_0\right),$

$\mathbf{\Gamma}_{\text{post}} = \left(\frac{1}{\sigma^2}\mathbf{A}^H\mathbf{A} + \mathbf{\Gamma}^{-1}\right)^{-1}.$

Moreover, MAP $=$ MMSE $= \hat{\boldsymbol{\gamma}}_{\text{post}}$ since the Gaussian posterior is symmetric and unimodal.

Proof

Completing the square in the exponent

The log-posterior (up to a constant) is

$-\log p(\boldsymbol{\gamma} \mid \mathbf{y}) = \frac{1}{2\sigma^2}\|\mathbf{A}\boldsymbol{\gamma} - \mathbf{y}\|^2 + \frac{1}{2}\|\boldsymbol{\gamma} - \boldsymbol{\gamma}_0\|_{\mathbf{\Gamma}^{-1}}^2,$

where $\|\mathbf{v}\|_{\mathbf{\Gamma}^{-1}}^2 = \mathbf{v}^H \mathbf{\Gamma}^{-1} \mathbf{v}$ . Expanding and collecting terms quadratic and linear in $\boldsymbol{\gamma}$ :

$= \frac{1}{2} \boldsymbol{\gamma}^H\!\left(\frac{1}{\sigma^2}\mathbf{A}^H\mathbf{A} + \mathbf{\Gamma}^{-1}\right)\!\boldsymbol{\gamma} - \boldsymbol{\gamma}^H\!\left(\frac{1}{\sigma^2}\mathbf{A}^H \mathbf{y} + \mathbf{\Gamma}^{-1}\boldsymbol{\gamma}_0\right) + \text{const}.$

This is the exponent of $\mathcal{N}(\hat{\boldsymbol{\gamma}}_{\text{post}}, \mathbf{\Gamma}_{\text{post}})$ with the stated parameters. $\blacksquare$

Tikhonov Regularization Is Gaussian MAP

Setting $\boldsymbol{\gamma}_0 = \mathbf{0}$ and $\mathbf{\Gamma} = \frac{\sigma^2}{\lambda} \mathbf{I}$ , the MAP/MMSE estimate becomes

$\hat{\boldsymbol{\gamma}}_{\text{MAP}} = (\mathbf{A}^H\mathbf{A} + \lambda \mathbf{I})^{-1}\mathbf{A}^H \mathbf{y},$

which is the Tikhonov regularized solution (§Regularization: Concept and General Theory). The regularization parameter $\lambda = \sigma^2/\gamma^2$ (where $\mathbf{\Gamma} = \gamma^2 \mathbf{I}$ ) has a clear probabilistic interpretation as the noise-to-signal variance ratio. This provides a principled way to set $\lambda$ : if you know $\sigma^2$ and have a prior estimate of $\gamma^2$ , no cross-validation is needed.

Common Mistake: The Prior Is Not "Objective"

Mistake:

A common misconception is that placing a Gaussian prior and computing the MAP estimate is somehow more objective than choosing a regularization parameter $\lambda$ by hand — the math is more elegant, so the result must be less arbitrary.

Correction:

The prior $\pi(\boldsymbol{\gamma})$ encodes subjective beliefs about the scene before observing data. A Gaussian prior with $\mathbf{\Gamma} = \gamma^2 \mathbf{I}$ still requires choosing $\gamma^2$ , and the Tikhonov MAP just shifts that choice from " $\lambda$ " to " $\gamma^2 = \sigma^2/\lambda$ ." The Bayesian framework makes the choice explicit and allows it to be informed by domain knowledge (e.g., known typical reflectivity ranges, spatial correlation length from physics), but it does not eliminate the need for a choice.

1D Posterior for Gaussian Likelihood and Various Priors

This plot illustrates Bayesian inference for the scalar model $y = a\gamma + w$ with $w \sim \mathcal{N}(0, \sigma^2)$ . Adjust the prior type and parameters to see how the posterior changes.

Left panel: Prior $\pi(\gamma)$ (blue), likelihood $p(y \mid \gamma)$ (green), and unnormalized posterior $\propto p(y \mid \gamma)\pi(\gamma)$ (red). Watch how the posterior balances data evidence against prior belief.

Right panel: MAP estimate (red dot) and MMSE estimate (orange dot) with $95\%$ credible interval shaded. For Gaussian priors the two coincide; for Laplace priors they diverge — MAP produces exact zeros, MMSE does not.

Parameters

Prior type

Prior variance

\gamma^2

(Gaussian/Laplace)1

Noise variance

\sigma^2

0.5

Observation

y

1.5

Key Takeaway

The Bayesian inverse problem treats the unknown as a random variable and seeks the full posterior $p(\boldsymbol{\gamma} \mid \mathbf{y})$ , not just a point estimate.
Bayes' theorem combines the likelihood $p(\mathbf{y} \mid \boldsymbol{\gamma})$ and prior $\pi(\boldsymbol{\gamma})$ to yield the posterior, up to the normalizing constant $\mathcal{Z}(\mathbf{y})$ .
For Gaussian noise, the negative log-likelihood equals the least-squares fidelity $\frac{1}{2\sigma^2}\|\mathbf{A}\boldsymbol{\gamma} - \mathbf{y}\|^2$ .
The MAP estimate equals variational regularization: $-\log\pi(\boldsymbol{\gamma})$ plays the role of the penalty $\lambda R(\boldsymbol{\gamma})$ .
The MMSE estimate (posterior mean) minimises expected squared error and accounts for the full posterior shape — not just its mode.

Quick Check

For a Gaussian prior $\boldsymbol{\gamma} \sim \mathcal{N}(\mathbf{0}, \gamma^2 \mathbf{I})$ and Gaussian noise $\mathbf{w} \sim \mathcal{N}(\mathbf{0}, \sigma^2 \mathbf{I})$ , the MAP and MMSE estimates are:

Equal to each other and equal to the Tikhonov solution with $\lambda = \sigma^2/\gamma^2$

Equal to each other but NOT equal to the Tikhonov solution

Different from each other because the Gaussian posterior is multimodal

Only defined when $\sigma^2 < \gamma^2$

Correction:

Equal to each other and equal to the Tikhonov solution with

\lambda = \sigma^2/\gamma^2

Correct. The Gaussian posterior is symmetric and unimodal, so MAP = MMSE. Setting $\boldsymbol{\gamma}_0 = \mathbf{0}$ and $\mathbf{\Gamma} = \gamma^2 \mathbf{I}$ gives exactly the Tikhonov solution with $\lambda = \sigma^2/\gamma^2$ .

Historical Note: Thomas Bayes and Inverse Reasoning

1763-2010

Thomas Bayes (1702-1761) never published his famous theorem himself. His essay "An Essay towards Solving a Problem in the Doctrine of Chances" was communicated to the Royal Society posthumously in 1763 by his friend Richard Price. The theorem was largely dormant until Pierre-Simon Laplace independently developed it in the 1780s and applied it broadly to problems of parameter estimation — making him the true founding figure of what we now call Bayesian inference.

The application of Bayes' theorem to inverse problems in physics and imaging developed slowly over the 20th century. Harold Jeffreys's 1939 monograph "Theory of Probability" applied Bayesian methods to geophysical inverse problems. The modern statistical framework for Bayesian inverse problems was synthesized by Kaipio and Somersalo in their 2005 monograph, and the rigorous infinite-dimensional theory was established by Stuart in 2010 — providing the foundation for this chapter.

Why This Matters: From Posterior to Target Detection

In RF imaging, the posterior $p(\boldsymbol{\gamma} \mid \mathbf{y})$ directly enables Bayesian detection: declare target present at pixel $i$ if $P(\gamma_i \neq 0 \mid \mathbf{y}) > \tau$ for some threshold $\tau$ . This is precisely the Neyman-Pearson detector applied to the posterior probability rather than the raw likelihood ratio — the Bayesian analogue of CFAR detection (§Computational Complexity and Kronecker Exploitation). For a Bernoulli-Gaussian prior (§Sparsity-Promoting Priors), this posterior probability has a closed form and provides the optimal detector under the assumed model.

See full treatment in Computational Complexity and Kronecker Exploitation

⚠️Engineering Note

Misspecified Noise Models in Practice

The Gaussian noise assumption $\mathbf{w} \sim \mathcal{N}(0, \sigma^2 \mathbf{I})$ underlies the least-squares likelihood. In real RF systems, noise is approximately Gaussian by the central limit theorem (many independent thermal noise sources), but:

Clutter (reflections from non-target objects) is non-Gaussian and can dominate. It requires a heavy-tailed likelihood (e.g., Student-t) or an explicit clutter model.
Phase errors from imperfect hardware synchronization add structured noise that is not captured by the i.i.d. Gaussian model.
Quantization noise from finite ADC resolution (8-12 bits in typical radar) introduces a bounded, nearly uniform component at low SNR.

A misspecified noise model leads to overconfident or miscalibrated uncertainty estimates. Robustness checks (e.g., computing the posterior under different noise models and comparing) are standard practice before deploying Bayesian UQ in safety-critical systems.

The Bayesian Framework for Inverse Problems