Evaluation Metrics

PSNR measures the reconstruction quality relative to the dynamic range of the image:

$$\mathrm{PSNR} = 10\log_{10}\!\left( \frac{\max_q |\mathbf{c}_{q}|^2} {\frac{1}{Q}\sum_{q=1}^{Q}|\hat{\mathbf{c}}_q - \mathbf{c}_{q}|^2} \right) = 10\log_{10}\!\left(\frac{\mathbf{c}_{\max}^{2}}{\mathrm{MSE}}\right)$$

where $\mathbf{c}_{\max}$ is the maximum scene value and MSE is the mean squared error over $Q$ voxels.

Typical values: $> 30$ dB (good), $> 40$ dB (excellent), $< 20$ dB (poor).

SSIM compares local image patches in terms of luminance, contrast, and structure:

$$\mathrm{SSIM}(\mathbf{x}, \hat{\mathbf{x}}) = \frac{(2\mu_x\mu_{\hat{x}} + c_1)(2\sigma_{x\hat{x}} + c_2)} {(\mu_x^2 + \mu_{\hat{x}}^2 + c_1)(\sigma_x^2 + \sigma_{\hat{x}}^2 + c_2)}$$

where $\mu$, $\sigma^2$, and $\sigma_{x\hat{x}}$ are the local mean, variance, and cross-covariance computed over a sliding window, and $c_1, c_2$ are stabilisation constants.

SSIM $\in [0, 1]$ where $1$ indicates perfect reconstruction. The overall SSIM is the average over all windows (MSSIM).

LPIPS measures perceptual distance using features from a pretrained deep network (typically VGG or AlexNet):

$$\mathrm{LPIPS}(\mathbf{x}, \hat{\mathbf{x}}) = \sum_l w_l \cdot \|f_l(\mathbf{x}) - f_l(\hat{\mathbf{x}})\|_2^2$$

where $f_l$ extracts features at layer $l$ and $w_l$ are learned weights. Lower LPIPS means higher perceptual similarity.

LPIPS captures high-level structural features that PSNR and SSIM miss. However, it was trained on natural images, so its relevance to RF reflectivity maps (which look different from photographs) is an open question.

NMSE measures the relative reconstruction error:

$$\mathrm{NMSE} = \frac{\|\hat{\mathbf{c}} - \mathbf{c}\|_2^2}{\|\mathbf{c}\|_2^2}$$

NMSE $= 0$ is perfect reconstruction; NMSE $= 1$ means the reconstruction error equals the signal energy.

In dB: $\mathrm{NMSE}_{\mathrm{dB}} = 10\log_{10}(\mathrm{NMSE})$. Typical targets: $< -20$ dB (good), $< -30$ dB (excellent).

For 3D reconstructions represented as point sets $\mathcal{P}$ (reconstruction) and $\mathcal{Q}$ (ground truth):

Chamfer distance: $$d_C(\mathcal{P}, \mathcal{Q}) = \frac{1}{|\mathcal{P}|}\sum_{\mathbf{p} \in \mathcal{P}} \min_{\mathbf{q} \in \mathcal{Q}} \|\mathbf{p} - \mathbf{q}\|^2 + \frac{1}{|\mathcal{Q}|}\sum_{\mathbf{q} \in \mathcal{Q}} \min_{\mathbf{p} \in \mathcal{P}} \|\mathbf{q} - \mathbf{p}\|^2$$

Hausdorff distance: $$d_H(\mathcal{P}, \mathcal{Q}) = \max\!\left( \max_{\mathbf{p} \in \mathcal{P}} \min_{\mathbf{q} \in \mathcal{Q}} \|\mathbf{p} - \mathbf{q}\|,\; \max_{\mathbf{q} \in \mathcal{Q}} \min_{\mathbf{p} \in \mathcal{P}} \|\mathbf{q} - \mathbf{p}\| \right)$$

Chamfer is the average nearest-neighbour distance (sensitive to overall shape). Hausdorff is the worst-case distance (sensitive to outliers).

For volumetric (voxel-based) reconstructions:

Intersection over Union (IoU): $$\mathrm{IoU} = \frac{|\mathcal{V}_{\mathrm{pred}} \cap \mathcal{V}_{\mathrm{true}}|} {|\mathcal{V}_{\mathrm{pred}} \cup \mathcal{V}_{\mathrm{true}}|}$$ where $\mathcal{V}$ is the set of occupied voxels after thresholding. IoU $\in [0, 1]$; higher is better.

F-Score at threshold $\tau$: $$F_\tau = \frac{2 \cdot \mathrm{Precision}_\tau \cdot \mathrm{Recall}_\tau} {\mathrm{Precision}_\tau + \mathrm{Recall}_\tau}$$ where precision and recall are computed by counting predicted points within distance $\tau$ of a ground truth point and vice versa. The F-score at multiple thresholds (e.g., $\tau \in \{\lambda/4, \lambda/2, \lambda\}$) characterises both coarse and fine shape recovery.

For radar detection applications:

Probability of detection $P_D$: the fraction of true targets that are correctly detected (above threshold).

Probability of false alarm $P_{\mathrm{FA}}$: the fraction of non-target locations that exceed the detection threshold.

ROC curve: plots $P_D$ vs. $P_{\mathrm{FA}}$ as the threshold varies. The area under the ROC (AUC) summarises detection performance in a single number ($\mathrm{AUC} \in [0.5, 1]$).

Operating point: typically report $P_D$ at a fixed $P_{\mathrm{FA}} = 10^{-4}$ or $10^{-6}$, reflecting the practical requirement of low false alarm rates.

Beyond reconstruction quality, practical deployment requires:

• FLOPs: floating-point operations per reconstruction. Matched filter: $O(MQ)$. LASSO (ISTA, $T$ iterations): $O(T \cdot MQ)$. Deep network: depends on architecture.

• Inference time: wall-clock time for a single reconstruction on specified hardware (GPU model, batch size).

• Memory: peak GPU memory during reconstruction.

• Training time: total GPU-hours for learned methods (amortised over all future reconstructions).

• Number of parameters: for neural networks, total trainable parameters.

Report computational metrics alongside quality metrics. A method that achieves 0.5 dB higher PSNR but takes $100\times$ longer may not be preferable.