Ferkans — Interactive Telecom Tutor

Why Simulation Matters

Almost all RF imaging research relies on simulation at some stage: validating algorithms before hardware deployment, generating training data for learned methods, and exploring scenarios that are difficult to measure. A well-designed simulator is the foundation of credible research. This section surveys the major simulation frameworks, describes the CommIT group's own simulator, and covers the Monte Carlo methodology needed for statistically rigorous evaluation.

Definition:
Open-Source Simulation Frameworks

Sionna RT (NVIDIA): differentiable link-level simulator with ray tracing for 5G/6G. Supports OFDM, MIMO, and channel estimation. GPU-accelerated via TensorFlow/JAX. Python.

FEKO (Altair): commercial full-wave electromagnetic solver supporting MoM, FEM, FDTD, and physical optics. Used for antenna design and RCS computation. Accurate but slow for large scenes.

MATLAB Radar Toolbox: comprehensive radar simulation including FMCW, OFDM, MIMO, and SAR. Signal-level simulation with propagation models.

RadarSimPy: open-source Python radar simulator supporting FMCW, OFDM, phased array, and MIMO configurations. Includes target models, interference, and clutter.

OpenRadar: open-source Python library for TI mmWave radar data processing. Includes range-Doppler, CFAR, and angle estimation.

Sionna's differentiable ray tracing is particularly valuable for learned imaging: gradients flow through the channel model, enabling end-to-end training of reconstruction networks.

Definition:
Hierarchy of Forward Models

Forward models trade accuracy for speed:

Model	Physics	Speed	Multipath	Diffraction
Point scatterer	Born approx.	$\mu$ s	No	No
Ray tracing	Geometric optics	ms	Yes	Limited
FDTD/MoM	Full Maxwell	hours	Yes	Yes

The point-scatterer / Born model: $\mathbf{y}_{m}({f_k}_{k}) = \sum_{q=1}^{Q} \mathbf{c}_{q} \cdot a_m(\hat{\mathbf{d}}_q) \cdot e^{-j 2\pi {f_k}_{k} {\tau_{i,j}}_{m,q}} + \mathbf{w}_{m}({f_k}_{k})$

is the most common in RF imaging papers but is also the most prone to the inverse crime (Section 31.2).

,

Simulation Framework Comparison

Framework	Physics	GPU	Differentiable	Language	Cost
Sionna RT	Ray tracing	Yes	Yes	Python	Free
FEKO	Full-wave (MoM/FEM/FDTD)	Limited	No	GUI/Script	Commercial
MATLAB Radar	Signal-level	Limited	No	MATLAB	Commercial
RadarSimPy	Point scatterer + ray	No	No	Python	Free
OpenRadar	Data processing only	No	No	Python	Free
CommIT Simulator	Born / Kronecker	Yes (CuPy)	Yes (PyTorch)	Python	Internal

Definition:
The CommIT RF Imaging Simulator

The CommIT group's simulator implements the full imaging pipeline:

Architecture:

2D or 3D voxel grid with configurable resolution.
Multi-view, multi-frequency sensing with Kronecker-structured $\mathbf{A} = \mathbf{A}_{\mathrm{freq}} \otimes \mathbf{A}_{\mathrm{space}}$ .
GPU-accelerated forward and adjoint operations via CuPy/PyTorch.

The 13-phase pipeline:

Scene generation (ShapeNet / THuman / random)
Array configuration (ULA / UPA / circular / custom)
Frequency plan ( $f_0$ , $W$ , $K$ subcarriers)
Sensing matrix $\mathbf{A}$ construction
Kronecker factorisation and preconditioning
Forward measurement $\mathbf{y} = \mathbf{A}\mathbf{c} + \mathbf{w}$
Noise and impairment injection
Reconstruction (MF, LASSO, OAMP, learned)
Post-processing (thresholding, CFAR)
Metric computation (PSNR, SSIM, detection)
Logging and visualisation
Monte Carlo repetition
Statistical aggregation and reporting

The Kronecker structure enables fast matrix-vector products: $\mathbf{A}\mathbf{c}$ costs $O(MK + NQ)$ instead of $O(MQ)$ , where $M$ is measurements, $K$ frequencies, $N$ spatial channels, and $Q$ voxels.

Differentiable Simulators

For training learned imaging methods end-to-end, the simulator must be differentiable: gradients of the loss with respect to the input scene must propagate through the forward model.

Point scatterer model: fully differentiable (linear in $\mathbf{c}_{q}$ , smooth in positions).
Ray tracing (Sionna): differentiable with respect to material parameters and geometry.
FDTD/MoM: not differentiable (discrete grid, iterative solver). Adjoint methods provide approximate gradients.

Using a differentiable simulator for both training and evaluation creates the inverse crime (Section 31.2).

Definition:
Monte Carlo Evaluation Framework

A Monte Carlo evaluation consists of:

Parameter space: define the variables being swept (SNR, number of targets, compression ratio, etc.).
Random factors: identify the random elements (noise, target positions, channel realisations).
Number of trials: $N_{\mathrm{MC}}$ independent realisations per parameter setting.
Performance metric: compute the metric $m_i$ for each trial $i \in \{1, \ldots, N_{\mathrm{MC}}\}$ .
Statistics: report the sample mean $\bar{m} = \frac{1}{N_{\mathrm{MC}}} \sum_{i=1}^{N_{\mathrm{MC}}} m_i$ and confidence interval $\mathrm{CI}_{1-\alpha} = \bar{m} \pm z_{1-\alpha/2} \cdot s/\sqrt{N_{\mathrm{MC}}}$ , where $s$ is the sample standard deviation.

Theorem: Required Number of Monte Carlo Trials

To achieve a confidence interval half-width of $\epsilon$ at confidence level $1 - \alpha$ , the required number of trials is:

$N_{\mathrm{MC}} \geq \left(\frac{z_{1-\alpha/2} \cdot \sigma_m}{\epsilon}\right)^2,$

where $\sigma_m$ is the standard deviation of the metric and $z_{1-\alpha/2}$ is the standard normal quantile ( $z_{0.975} = 1.96$ for 95% confidence).

Proof

By the central limit theorem, $\bar{m} \sim \mathcal{N}(m_{\mathrm{true}}, \sigma_m^2/N_{\mathrm{MC}})$ for large $N_{\mathrm{MC}}$ . The half-width of the $(1-\alpha)$ confidence interval is $z_{1-\alpha/2} \cdot \sigma_m / \sqrt{N_{\mathrm{MC}}}$ . Setting this $\leq \epsilon$ and solving for $N_{\mathrm{MC}}$ gives the result. $\square$

Example: How Many Monte Carlo Trials?

An OFDM radar imaging algorithm is evaluated using PSNR as the metric. In a pilot run of 20 trials, the sample standard deviation is $s = 2.5$ dB. How many trials are needed for a 95% confidence interval of $\pm 0.5$ dB?

Solution

Computation

$N_{\mathrm{MC}} \geq (1.96 \times 2.5 / 0.5)^2 = (9.8)^2 = 96.04$ . Round up: $N_{\mathrm{MC}} = 100$ trials.

Interpretation

With 100 trials, the reported mean PSNR is accurate to $\pm 0.5$ dB with 95% confidence. The pilot run of 20 trials was insufficient (CI would have been $\pm 1.1$ dB). Many RF imaging papers report results from $\leq 10$ trials, which is inadequate for claims of $< 1$ dB improvement. $\square$

Monte Carlo Convergence

Watch the running mean PSNR and its 95% confidence interval converge as the number of Monte Carlo trials increases. The shaded band is the 95% CI. Notice how the CI is wide for few trials and narrows as $1/\sqrt{N_{\mathrm{MC}}}$ .

Parameters

True Mean PSNR (dB)25

True Std Dev (dB)3

Max Trials500

Definition:
Statistical Significance Testing

To claim that method A outperforms method B, use a paired statistical test:

Paired $t$ -test: compares mean performance on the same test instances. Null hypothesis: $\mu_A - \mu_B = 0$ .
Wilcoxon signed-rank test: non-parametric alternative when the metric distribution is non-Gaussian.
Effect size (Cohen's $d$ ): $d = |\bar{m}_A - \bar{m}_B| / s_{\mathrm{pooled}}$ . Conventions: $d < 0.2$ (negligible), $d \in [0.2, 0.5]$ (small), $d \in [0.5, 0.8]$ (medium), $d > 0.8$ (large).

Report both the $p$ -value and the effect size. A statistically significant but tiny improvement ( $d < 0.2$ ) may not be practically meaningful.

Common Monte Carlo Mistakes

Too few trials: 10 trials cannot resolve a 0.5 dB difference. Use the sample size formula above.
Same random seed: using identical seeds for all methods ensures fair comparison but must not be used to avoid reporting variability. Report both same-seed and independent-seed results.
Cherry-picking: showing the best trial or a hand-picked scene. Always report aggregate statistics.
Ignoring variance: reporting only the mean without confidence intervals or standard deviations.
Multiple comparisons: testing 10 methods pairwise (45 tests) inflates the false positive rate. Use Bonferroni correction: $\alpha_{\mathrm{adj}} = 0.05/45 \approx 0.001$ .

⚠️Engineering Note

GPU Acceleration for Large-Scale Simulation

For a $128 \times 128 \times 128$ voxel grid with 64 frequencies and a $16 \times 16$ MIMO array, the sensing matrix $\mathbf{A}$ has $16{,}384 \times 2{,}097{,}152$ entries --- it cannot be stored explicitly. The Kronecker factorisation $\mathbf{A} = \mathbf{A}_{\mathrm{freq}} \otimes \mathbf{A}_{\mathrm{space}}$ enables matrix-free operations:

Forward: $\mathbf{y} = (\mathbf{A}_{\mathrm{freq}} \otimes \mathbf{A}_{\mathrm{space}}) \operatorname{vec}(\mathbf{c})$ via two smaller matrix-vector products.
Adjoint: $\mathbf{A}^{H} \mathbf{y}$ via the same factored structure.

On an NVIDIA A100, this reduces the time per forward-adjoint pair from minutes (explicit matrix) to milliseconds (Kronecker).

Example: End-to-End Simulation with Sionna RT

Set up a Sionna simulation for RF imaging at 28 GHz with a $16$ -antenna base station. The scene contains 5 radar targets in an indoor room. Outline the pipeline.

Solution

Scene setup

Load a 3D indoor model in Sionna RT. Place the base station at one wall. Place 5 metallic targets (RCS $1$ -- $10$ m $^2$ ) at known positions within the room.

Channel simulation

Run Sionna RT to compute target echoes via ray paths (up to 3 reflections). Export channel impulse responses at 28 GHz with 400 MHz bandwidth ( $K = 128$ subcarriers).

Reconstruction

Form the sensing matrix $\mathbf{A}$ from the array geometry and frequency plan. Apply matched-filter imaging ( $\hat{\mathbf{c}}^{\text{BP}} = \mathbf{A}^{H} \mathbf{y}$ ) and LASSO for comparison. The Sionna ray-tracing data serves as the "honest" forward model; reconstruction uses the Born approximation, avoiding the inverse crime.

Evaluation

Compare reconstructed images to the known target positions. Compute PSNR, SSIM, and detection $P_D$ at $P_{\mathrm{FA}} = 10^{-4}$ . $\square$

Common Mistake: Reporting Results from a Single Simulation Run

Mistake:

Running one simulation with a single noise realisation and reporting the resulting PSNR as representative.

Correction:

A single run depends on the specific noise realisation and target configuration. Use at least $N_{\mathrm{MC}} = 100$ trials and report mean $\pm$ 95% confidence interval. For comparing two methods, use a paired $t$ -test to verify that the difference is statistically significant.

Why This Matters: From Channel Models to Imaging Simulators

The telecommunications community has developed sophisticated channel models (3GPP, QuaDRiGa) for link-level simulation. RF imaging simulators extend these by requiring geometric consistency: the scatterer positions that generate the channel must correspond to a physically meaningful scene that can serve as ground truth. Standard channel models lack this geometric consistency, which is why the CommIT group is developing a next-generation geometrically consistent channel simulation tool.

See full treatment in Chapter 32

Monte Carlo Method

A computational technique that uses repeated random sampling to estimate statistical quantities (means, variances, distributions) of a system. In RF imaging, used to evaluate algorithm performance over many noise and scene realisations.

Kronecker Structure

A matrix factorisation $\mathbf{A} = \mathbf{A}_{\mathrm{freq}} \otimes \mathbf{A}_{\mathrm{space}}$ that separates the frequency and spatial dimensions of the sensing operator, enabling fast matrix-vector products and reduced memory.

Related: Sensing Matrix

Quick Check

A simulation study uses 15 Monte Carlo trials and reports that method A achieves $28.3 \pm 1.8$ dB PSNR while method B achieves $27.5 \pm 2.1$ dB. Can we conclude that A is better than B?

Yes, A has higher mean PSNR

No, the confidence intervals overlap and 15 trials is too few

Yes, if we use a one-sided test

Cannot determine without the raw data

Correction:

No, the confidence intervals overlap and 15 trials is too few

Correct. The 0.8 dB difference is within the confidence intervals. Need ~100 trials for a CI of +/- 0.5 dB.

Key Takeaway

Sionna RT provides GPU-accelerated, differentiable simulation for end-to-end imaging research. The CommIT simulator uses Kronecker-structured sensing for fast 2D/3D imaging with a 13-phase pipeline. Monte Carlo evaluation with $N_{\mathrm{MC}} \geq 100$ trials, confidence intervals, and paired statistical tests is essential for credible performance claims. Common pitfalls include too few trials, cherry-picking, and ignoring multiple comparisons.

Simulation Frameworks and Our Simulator

Why Simulation Matters

Definition: Open-Source Simulation Frameworks

Definition: Hierarchy of Forward Models

Simulation Framework Comparison

Definition: The CommIT RF Imaging Simulator

Differentiable Simulators

Definition: Monte Carlo Evaluation Framework

Theorem: Required Number of Monte Carlo Trials

Proof

Example: How Many Monte Carlo Trials?

Computation

Interpretation

Monte Carlo Convergence

Parameters

Definition: Statistical Significance Testing

Common Monte Carlo Mistakes

GPU Acceleration for Large-Scale Simulation

Example: End-to-End Simulation with Sionna RT

Scene setup

Channel simulation

Reconstruction

Evaluation

Common Mistake: Reporting Results from a Single Simulation Run

Why This Matters: From Channel Models to Imaging Simulators

Monte Carlo Method

Kronecker Structure

Quick Check

Key Takeaway

Definition:
Open-Source Simulation Frameworks

Definition:
Hierarchy of Forward Models

Definition:
The CommIT RF Imaging Simulator

Definition:
Monte Carlo Evaluation Framework

Definition:
Statistical Significance Testing