Bridging the Sim-to-Real Gap

The Central Challenge

The sim-to-real gap -- the performance degradation when deploying simulation-trained models on real data -- is arguably the most pressing open problem in learned RF imaging. Models trained on point-scatterer simulations routinely lose 10--15 dB PSNR on measured data. Until this gap is closed, learned imaging methods remain laboratory curiosities rather than deployed systems. We do not yet know how to reliably quantify this gap before collecting real data, nor how to guarantee that a domain-adapted model will not fail catastrophically on unseen environments.

Definition:
The Sim-to-Real Gap

The sim-to-real gap arises from the distribution shift between simulated and real data:

$\Delta_{\mathrm{s2r}} = \mathbb{E}_{\mathcal{D}_{\mathrm{real}}} [\mathcal{L}(f_\theta)] - \mathbb{E}_{\mathcal{D}_{\mathrm{sim}}} [\mathcal{L}(f_\theta)],$

where $f_\theta$ is a model trained on $\mathcal{D}_{\mathrm{sim}}$ . Sources of the gap include:

Model mismatch: Born vs multipath, point vs extended targets.
Hardware impairments: IQ imbalance, phase noise, mutual coupling not modelled in simulation.
Scene complexity: real scenes have materials, textures, and clutter absent from simple simulations.
Environmental factors: interference, temperature drift, multipath from unmapped objects.

Definition:
Domain Adaptation Approaches

Several strategies address the sim-to-real gap:

Domain randomisation: randomise simulator parameters (noise level, material properties, array errors) during training to cover the real distribution.
Fine-tuning: pre-train on simulation, fine-tune on a small real dataset (10--100 examples).
Adversarial domain adaptation: train a domain discriminator $D$ alongside the imaging network, minimising $\mathcal{L}_{\mathrm{task}} + \lambda \mathcal{L}_{\mathrm{domain}}$ to learn domain-invariant features.
Physics-based simulation: use high-fidelity simulators (ray tracing via Sionna RT, FDTD) that model the dominant real-world effects.
Self-supervised learning: train on real data without labels using self-consistency losses (predict held-out measurements from the reconstruction).

$\text{Sim-to-Real Domain Gap Visualization}$

Visualises the distribution shift between simulated and real RF data in a 2D feature space. The overlap region determines how well a simulation-trained model transfers.

Parameters

Gap magnitude2

Simulation diversity1

Samples per domain300

Definition:
Meta-Learning for Few-Shot Adaptation

Meta-learning trains a model to adapt quickly from few examples. For RF imaging, model-agnostic meta-learning (MAML) optimises initial parameters $\theta_0$ such that a few gradient steps on $K$ real-data examples yield good performance:

$\theta_0^* = \arg\min_{\theta_0} \sum_{i=1}^{N_{\mathrm{tasks}}} \mathcal{L}_i\bigl(U_K(\theta_0; \mathcal{S}_i)\bigr),$

where $U_K$ denotes $K$ gradient steps on support set $\mathcal{S}_i$ . Each "task" is a different environment (room, hardware platform). The meta-learned initialisation captures scene priors that transfer across environments while allowing rapid adaptation to each.

Example: Sim-to-Real Budget Analysis

A learned indoor imaging system at 5 GHz is trained on 50,000 Born-model simulations. Real deployment on a TI IWR6843 in an office loses 12 dB PSNR. Decompose the gap into sources and propose a mitigation strategy with a budget of 20 real scenes.

Solution

Gap decomposition

Estimate each source by ablation: (1) Model mismatch (Born vs multipath): $\sim 5$ dB. (2) Off-grid targets: $\sim 2$ dB. (3) Phase noise / mutual coupling: $\sim 2$ dB. (4) Clutter not in training: $\sim 3$ dB. Total: $\sim 12$ dB, consistent with the observed gap.

Mitigation with 20 real scenes

(1) Add ray-tracing simulations to cover multipath (recovers $\sim 3$ dB). (2) Fine-tune with 10 real scenes, held-out 10 for validation (recovers $\sim 5$ dB). (3) Domain randomisation over hardware impairments (recovers $\sim 2$ dB). Expected residual gap: $\sim 2$ dB, within practical tolerance. $\square$

⚠️Engineering Note

Practical Sim-to-Real Transfer Pipeline

A production sim-to-real pipeline for RF imaging deployment: (1) Tier 1: Born-model simulation with domain randomisation ( $10^4$ -- $10^5$ scenes). (2) Tier 2: ray-tracing simulation with calibrated material models ( $10^3$ scenes). (3) Tier 3: fine-tuning on 10--50 real measured scenes with self-supervised measurement-consistency loss. (4) Monitoring: track the normalised residual $\|\mathbf{y} - \mathbf{A}\hat{\boldsymbol{\gamma}}\|_2 / \|\mathbf{y}\|_2$ at deployment; if it exceeds a calibrated threshold, flag for re-adaptation. This tiered approach is the current best practice, though the optimal allocation across tiers remains an open problem.

Common Mistake: Overfitting to Few Real Scenes

Mistake:

Fine-tuning a high-capacity network (e.g., U-Net with $10^7$ parameters) on 10 real scenes without regularisation. The network memorises the 10 scenes and fails on scene 11.

Correction:

Use strong regularisation: freeze early layers, reduce learning rate by $10\times$ -- $100\times$ , apply weight decay, and use early stopping on a held-out validation scene. Alternatively, use self-supervised loss (measurement consistency) which does not require ground-truth labels and thus avoids label-based overfitting.

Open Questions in Sim-to-Real Transfer

A priori gap prediction: given a simulator and target scenario, can we predict $\Delta_{\mathrm{s2r}}$ without real data? This would enable informed decisions about simulation fidelity investment.
Cross-hardware transfer: can a model trained on TI IWR6843 data transfer to Infineon BGT60 with different antenna arrays and waveforms?
Worst-case guarantees: domain adaptation minimises the average gap, but can we guarantee the worst-case performance on any real scene?
Continual adaptation: can the model improve continuously as it encounters new environments, without forgetting previously learned ones?

These questions define a rich research programme that connects RF imaging to the broader machine learning literature on robustness and distribution shift.

Quick Check

Which sim-to-real mitigation strategy is most effective when only 5 real measured scenes are available?

Adversarial domain adaptation

Self-supervised measurement consistency

Increasing simulation dataset size to 500,000

Training only on the 5 real scenes from scratch

Correction:

Self-supervised measurement consistency

Self-supervised loss uses all measurements without requiring ground-truth labels, maximising information from each scene and avoiding label-based overfitting.

Sim-to-Real Gap

The performance degradation (typically 10--15 dB PSNR) observed when deploying a simulation-trained model on real measured data. Caused by distribution shift in forward model, hardware, and environment.

Domain Adaptation

Machine learning techniques that reduce the distribution shift between source (simulation) and target (real) domains. Includes domain randomisation, fine-tuning, adversarial methods, and self-supervised approaches.

Related: Sim-to-Real Gap

Key Takeaway

The sim-to-real gap (10--15 dB) is the primary barrier to deploying learned RF imaging. The most effective current approach is tiered simulation (Born + ray tracing) combined with self-supervised fine-tuning on a small real dataset. Predicting the gap a priori and guaranteeing worst-case robustness remain fundamental open problems.

Prerequisites & Notation Dynamic Scene Imaging