Ferkans — Interactive Telecom Tutor

From Matrices to Operators

In Telecom Ch~1, linear maps between finite-dimensional spaces are matrices. In imaging, the forward operator $\mathcal{A}$ maps a scene function $c \in L^2(\Omega)$ to measurement data $\mathbf{y} \in \mathbb{C}^M$ . This is an infinite-to-finite dimensional map — a bounded linear operator — and understanding its properties determines what reconstruction is possible.

The key properties to develop:

Boundedness — the operator does not amplify signals without bound.
Adjoint — the operator $\mathcal{A}^*$ is the generalisation of $\mathbf{A}^{H}$ ; it appears in every iterative algorithm.
Compactness — the operator "compresses" the unit ball, making inversion inherently unstable. This is why imaging is ill-posed.

Definition:
Linear Operator

An operator $\mathcal{A}: \mathcal{V} \to \mathcal{W}$ is linear if

$\mathcal{A}(\alpha f + \beta g) = \alpha\,\mathcal{A}(f) + \beta\,\mathcal{A}(g)$

for all $f, g \in \mathcal{V}$ and all scalars $\alpha, \beta \in \mathbb{F}$ .

Definition:
Bounded Operator and Operator Norm

A linear operator $\mathcal{A}: \mathcal{V} \to \mathcal{W}$ is bounded if its operator norm is finite:

$\|\mathcal{A}\|_{\mathrm{op}} \triangleq \sup_{\|f\|_{\mathcal{V}} = 1} \|\mathcal{A}f\|_{\mathcal{W}} < \infty.$

Equivalently: there exists $C > 0$ such that $\|\mathcal{A}f\|_{\mathcal{W}} \leq C\,\|f\|_{\mathcal{V}}$ for all $f$ .

For linear operators, bounded $\iff$ continuous (see $\iff$ $⟺$ Continuous for Linear Operators" data-ref-type="theorem">TBounded $\iff$ Continuous for Linear Operators). This equivalence does NOT hold for nonlinear maps.

Theorem: Bounded $\iff$ Continuous for Linear Operators

A linear operator $\mathcal{A}: \mathcal{V} \to \mathcal{W}$ between normed spaces is bounded if and only if it is continuous.

A linear map that is continuous at one point is continuous everywhere by linearity. Boundedness is precisely the quantitative version of this uniform continuity: the Lipschitz constant of $\mathcal{A}$ is its operator norm.

Show Hint

$(\Rightarrow)$ Use $\|\mathcal{A}f - \mathcal{A}g\| = \|\mathcal{A}(f-g)\| \leq \|\mathcal{A}\|_{\mathrm{op}}\|f - g\|$ .

$(\Leftarrow)$ Contrapositive: if $\mathcal{A}$ is unbounded, construct a sequence converging to $0$ whose image does not.

Proof

Bounded $\Rightarrow$ Continuous

Suppose $\|\mathcal{A}\|_{\mathrm{op}} < \infty$ . For any $f, g \in \mathcal{V}$ :

$\|\mathcal{A}f - \mathcal{A}g\| = \|\mathcal{A}(f - g)\| \leq \|\mathcal{A}\|_{\mathrm{op}}\|f - g\|.$

Given $\varepsilon > 0$ , take $\delta = \varepsilon / \|\mathcal{A}\|_{\mathrm{op}}$ . Then $\|f - g\| < \delta$ implies $\|\mathcal{A}f - \mathcal{A}g\| < \varepsilon$ . Thus $\mathcal{A}$ is Lipschitz continuous. $\blacksquare$

Continuous $\Rightarrow$ Bounded (contrapositive)

Suppose $\mathcal{A}$ is unbounded. There exist $f_n$ with $\|f_n\| = 1$ and $\|\mathcal{A}f_n\| \geq n$ . Set $g_n = f_n/n$ . Then $g_n \to 0$ but $\|\mathcal{A}g_n\| = \|\mathcal{A}f_n\|/n \geq 1 \not\to 0 = \mathcal{A}(0)$ . So $\mathcal{A}$ is not continuous at $0$ . $\blacksquare$

Example: Integral Operators and the Hilbert–Schmidt Condition

The operator $(\mathcal{A}f)(\mathbf{r}) = \int_{\Omega} K(\mathbf{r}, \mathbf{r}')\,f(\mathbf{r}')\,d\mathbf{r}'$ with kernel $K$ arises throughout imaging. Show that $\mathcal{A}$ is bounded on $L^2(\Omega)$ when $K$ satisfies the Hilbert–Schmidt condition.

Solution

Hilbert–Schmidt condition

The operator is bounded when

$\|K\|_{HS}^2 \triangleq \int\!\!\int |K(\mathbf{r}, \mathbf{r}')|^2\, d\mathbf{r}\,d\mathbf{r}' < \infty.$

By Cauchy–Schwarz:

$|(\mathcal{A}f)(\mathbf{r})|^2 \leq \int_{\Omega} |K(\mathbf{r}, \mathbf{r}')|^2\,d\mathbf{r}' \cdot \|f\|_{L^2}^2.$

Integrating over $\mathbf{r}$ : $\|\mathcal{A}f\|^2 \leq \|K\|_{HS}^2\,\|f\|^2$ , so $\|\mathcal{A}\|_{\mathrm{op}} \leq \|K\|_{HS}$ .

Connection to the imaging forward operator

The Born-approximation forward operator (Ch~5) has kernel $K(\mathbf{r}_{\mathrm{rx}}, \mathbf{r}) = G(\mathbf{r}_{\mathrm{rx}}, \mathbf{r})\, E^{\mathrm{inc}}(\mathbf{r})$ where $G$ is the free-space Green's function. The Hilbert–Schmidt condition is satisfied for bounded, compactly supported scenes, so the forward operator is always bounded.

Definition:
Adjoint Operator

The adjoint of a bounded linear operator $\mathcal{A}: \mathcal{V} \to \mathcal{W}$ is the unique bounded operator $\mathcal{A}^*: \mathcal{W} \to \mathcal{V}$ satisfying

$\langle \mathcal{A}f, g \rangle_{\mathcal{W}} = \langle f, \mathcal{A}^* g \rangle_{\mathcal{V}}$

for all $f \in \mathcal{V}$ and $g \in \mathcal{W}$ .

For matrices, $\mathcal{A}^* = \mathbf{A}^H$ (conjugate transpose). For integral operators, the adjoint has kernel $K^*(\mathbf{r}', \mathbf{r}) = \overline{K(\mathbf{r}, \mathbf{r}')}$ : source and receiver positions are swapped and the kernel is conjugated.

This swapping has a physical interpretation: the adjoint of the forward scattering operator is the back-propagation (matched filter) operator — propagating the received field back toward the scene.

Definition:
Self-Adjoint (Hermitian) Operator

A bounded operator $\mathcal{A}$ on a Hilbert space $\mathcal{H}$ is self-adjoint if $\mathcal{A} = \mathcal{A}^*$ , i.e.,

$\langle \mathcal{A}f, g \rangle = \langle f, \mathcal{A}g \rangle \quad \forall\, f, g \in \mathcal{H}.$

Key examples:

$\mathcal{A}^*\mathcal{A}$ is always self-adjoint (and positive semidefinite): $\langle \mathcal{A}^*\mathcal{A}f, g \rangle = \langle \mathcal{A}f, \mathcal{A}g \rangle = \langle f, \mathcal{A}^*\mathcal{A}g \rangle$ .
Real-valued convolution operators with symmetric kernels are self-adjoint on $L^2$ .

The normal equation $\mathcal{A}^*\mathcal{A}\hat{g} = \mathcal{A}^*y$ (Tikhonov regularisation, Ch~2) involves the self-adjoint operator $\mathcal{A}^*\mathcal{A}$ . The spectral theorem for compact self-adjoint operators (TSpectral Theorem for Compact Self-Adjoint Operators) will give us a complete eigendecomposition of $\mathcal{A}^*\mathcal{A}$ .

Definition:
Compact Operator

A linear operator $\mathcal{A}: \mathcal{V} \to \mathcal{W}$ is compact if it maps bounded sets to relatively compact (pre-compact) sets. Equivalently: every bounded sequence $\{f_n\} \subset \mathcal{V}$ has a subsequence $\{f_{n_k}\}$ such that $\{\mathcal{A}f_{n_k}\}$ converges in $\mathcal{W}$ .

Compact operators are the "closest to finite-dimensional" class of infinite-dimensional operators. All Hilbert–Schmidt integral operators (with square-integrable kernels) are compact. The definition via pre-compactness of the image of the unit ball explains the name: the operator "compacts" the unit ball to something with compact closure.

,

Theorem: Compact Operators are Not Invertible on Infinite-Dimensional Spaces

Let $\mathcal{H}$ be an infinite-dimensional Hilbert space. Then:

(i) The compact operators form a closed two-sided ideal in the algebra of bounded operators $\mathcal{B}(\mathcal{H})$ .

(ii) A compact operator on an infinite-dimensional space is not boundedly invertible: if $\mathcal{A}$ is compact and injective on an infinite-dimensional space, its inverse (if it exists) is unbounded.

(iii) This non-invertibility is why imaging problems are ill-posed: the forward operator is compact, so naive inversion amplifies noise without bound.

A compact operator "compresses" the unit ball to a relatively compact (strictly smaller) image. No operator that compresses in this way can have a bounded inverse that "un-compresses" everything back to the full unit ball in infinite dimensions.

Proof

Sketch of (ii)

Suppose $\mathcal{A}$ is compact and boundedly invertible with $\mathcal{A}^{-1}$ bounded. Then $\mathcal{I} = \mathcal{A}^{-1} \circ \mathcal{A}$ would be compact (the composition of a bounded operator with a compact operator is compact). But the identity on an infinite-dimensional space is not compact — by Riesz's lemma, the unit ball is not pre-compact. Contradiction. $\blacksquare$

,

Visualizing the Operator Norm

For a $2 \times 2$ matrix (a finite-dimensional operator), the operator norm equals the largest singular value — the maximum stretching factor. The unit circle maps to an ellipse whose semi-major axis equals $\|\mathbf{A}\|_{\mathrm{op}}$ . Explore how different matrix types (rotation, projection, scaling, shear) affect the geometry.

Parameters

Matrix type

Scale factor1.5

Common Mistake: Adjoint is NOT the Same as Inverse

Mistake:

Using $\mathcal{A}^*\mathbf{y}$ (the adjoint applied to the data) as the reconstruction, expecting it to recover the scene.

Correction:

The adjoint $\mathcal{A}^*$ is the analogue of $\mathbf{A}^H$ , not $\mathbf{A}^{-1}$ . For a unitary operator $\mathcal{U}$ we have $\mathcal{U}^* = \mathcal{U}^{-1}$ , but the imaging forward operator is never unitary (it loses information by mapping to a lower-dimensional space or having a non-trivial null space).

The result of applying $\mathcal{A}^*$ to the data is the back-projection or matched filter image $\hat{c}^{\mathrm{BP}} = \mathbf{A}^{H}\mathbf{y}$ . This is a blurred version of the true scene, not the reconstruction. The point spread function is $\mathcal{A}^*\mathcal{A}$ — invertible only up to regularisation.

Why This Matters: Operators in Reconstruction Algorithms

Every imaging algorithm repeatedly applies $\mathcal{A}$ and $\mathcal{A}^*$ . For example, ISTA performs:

$\hat{c}^{(t+1)} = \mathcal{S}_\tau\!\Bigl( \hat{c}^{(t)} - \mu\,\mathcal{A}^*\!\bigl(\mathcal{A}\hat{c}^{(t)} - \mathbf{y}\bigr)\Bigr),$

where $\mathcal{S}_\tau$ is soft-thresholding. ADMM alternates between applying $(\mathcal{A}^*\mathcal{A} + \rho\mathbf{I})^{-1}$ (a linear solve involving $\mathcal{A}^*\mathcal{A}$ ) and a proximal step. The Lipschitz constant of the gradient is $\|\mathcal{A}^*\mathcal{A}\|_{\mathrm{op}} = \sigma_1(\mathcal{A})^2$ — the step size $\mu < 2/\sigma_1^2$ is a hard constraint for convergence.

Efficient implementation of $\mathcal{A}$ and $\mathcal{A}^*$ — exploiting Kronecker product structure (Ch~4), FFT-based convolution, or the NUFFT — is the computational bottleneck of every reconstruction pipeline.

See full treatment in Chapter 4

Quick Check

For a linear operator between normed spaces, "bounded" is equivalent to ___.

differentiable

continuous

compact

invertible

Correction:

continuous

A linear operator is bounded if and only if it is continuous. This equivalence relies on linearity — it does NOT hold for nonlinear maps.

Quick Check

The adjoint of a matrix $\mathbf{A} \in \mathbb{C}^{M \times N}$ , viewed as an operator from $(\mathbb{C}^N, \|\cdot\|_2)$ to $(\mathbb{C}^M, \|\cdot\|_2)$ , is ___.

$\mathbf{A}^T$

$\mathbf{A}^{-1}$

$\mathbf{A}^H$ (conjugate transpose)

$\bar{\mathbf{A}}$ (element-wise conjugate)

Correction:

\mathbf{A}^H

(conjugate transpose)

The adjoint of a matrix is its conjugate transpose $\mathbf{A}^H$ , since $\langle \mathbf{A}\mathbf{x}, \mathbf{y}\rangle = \mathbf{y}^H \mathbf{A}\mathbf{x} = (\mathbf{A}^H \mathbf{y})^H \mathbf{x} = \langle \mathbf{x}, \mathbf{A}^H \mathbf{y}\rangle$ .

Quick Check

True or False: A compact operator on an infinite-dimensional Hilbert space can be boundedly invertible.

True

False

Correction:

False

A compact operator that is boundedly invertible would imply the identity is compact (as $\mathcal{I} = \mathcal{A}^{-1}\mathcal{A}$ would be compact). But the identity on an infinite-dimensional space is never compact (Riesz's lemma). This is why the imaging inverse problem is inherently ill-posed.

Bounded operator

A linear operator $\mathcal{A}: \mathcal{V} \to \mathcal{W}$ with finite operator norm $\|\mathcal{A}\|_{\mathrm{op}} = \sup_{\|f\|=1}\|\mathcal{A}f\| < \infty$ . Equivalent to continuity for linear operators.

Adjoint operator

The unique operator $\mathcal{A}^*: \mathcal{W} \to \mathcal{V}$ satisfying $\langle \mathcal{A}f, g \rangle = \langle f, \mathcal{A}^* g \rangle$ for all $f, g$ . Generalises the conjugate transpose $\mathbf{A}^H$ . Physically: the back-propagation (matched filter) operator.

Compact operator

A linear operator that maps bounded sets to pre-compact (relatively compact) sets. Compact operators are bounded, have countably many singular values decaying to zero, and are not boundedly invertible on infinite-dimensional spaces. The forward operator in imaging is always compact.

Key Takeaway

Three core messages of this section:

(1) Bounded operators generalise matrices to function spaces. The operator norm $\|\mathcal{A}\|_{\mathrm{op}}$ controls the Lipschitz constant of $\mathcal{A}$ and the step size of iterative algorithms.

(2) The adjoint $\mathcal{A}^*$ is the infinite-dimensional generalisation of $\mathbf{A}^H$ . It appears in every reconstruction algorithm as the back-projection (matched filter) step.

(3) Compact operators — the class that includes all imaging forward operators — are not boundedly invertible on infinite-dimensional spaces. This is the mathematical origin of ill-posedness. The resolution: regularisation, studied in Chapter 2.

Bounded Linear Operators and Compact Operators

From Matrices to Operators

Definition: Linear Operator

Definition: Bounded Operator and Operator Norm

Theorem: Bounded ⟺ \iff⟺ Continuous for Linear Operators

Bounded $\Rightarrow$ Continuous

Continuous $\Rightarrow$ Bounded (contrapositive)

Example: Integral Operators and the Hilbert–Schmidt Condition

Hilbert–Schmidt condition

Connection to the imaging forward operator

Definition: Adjoint Operator

Definition: Self-Adjoint (Hermitian) Operator

Definition: Compact Operator

Theorem: Compact Operators are Not Invertible on Infinite-Dimensional Spaces

Sketch of (ii)

Visualizing the Operator Norm

Parameters

Common Mistake: Adjoint is NOT the Same as Inverse

Why This Matters: Operators in Reconstruction Algorithms

Quick Check

Quick Check

Quick Check

Bounded operator

Adjoint operator

Compact operator

Key Takeaway

Definition:
Linear Operator

Definition:
Bounded Operator and Operator Norm

Theorem: Bounded $\iff$ Continuous for Linear Operators

Definition:
Adjoint Operator

Definition:
Self-Adjoint (Hermitian) Operator

Definition:
Compact Operator