Dynamic Scene Imaging

The World Moves

Most RF imaging algorithms assume a stationary scene during the measurement interval. In reality, people walk, vehicles move, and doors open. Dynamic scene imaging -- reconstructing both the scene and its motion -- is an active research frontier with applications in autonomous driving, human activity recognition, and indoor monitoring. The central difficulty is that each snapshot is severely underdetermined; temporal priors are not merely helpful but essential.

Definition:

Dynamic RF Imaging Problem

The dynamic imaging problem seeks to recover a time-varying scene Ξ³(x,t)\boldsymbol{\gamma}(\mathbf{x}, t) from sequential measurements {yt}t=1T\{\mathbf{y}_t\}_{t=1}^T:

yt=AtΞ³t+wt,\mathbf{y}_t = \mathbf{A}_t \boldsymbol{\gamma}_t + \mathbf{w}_t,

where At\mathbf{A}_t may also change (moving platform, ISAC beamforming). The key challenge: each yt\mathbf{y}_t alone is insufficient to reconstruct Ξ³t\boldsymbol{\gamma}_t (severely underdetermined). Temporal priors are essential.

Definition:

Temporal Priors for Dynamic Imaging

Temporal regularisation exploits the structure of scene dynamics:

  1. Temporal smoothness: βˆ₯Ξ³tβˆ’Ξ³tβˆ’1βˆ₯22\|\boldsymbol{\gamma}_t - \boldsymbol{\gamma}_{t-1}\|_2^2 penalises rapid changes (appropriate for slowly varying environments).

  2. Sparse innovation: βˆ₯Ξ³tβˆ’Ξ³tβˆ’1βˆ₯1\|\boldsymbol{\gamma}_t - \boldsymbol{\gamma}_{t-1}\|_1 assumes few voxels change between frames (a person moving through an otherwise static room).

  3. Optical flow: Ξ³t(x)β‰ˆΞ³tβˆ’1(xβˆ’vt(x))\boldsymbol{\gamma}_t(\mathbf{x}) \approx \boldsymbol{\gamma}_{t-1}(\mathbf{x} - \mathbf{v}_t(\mathbf{x})) models scene elements moving with velocity field vt\mathbf{v}_t.

  4. Kalman filtering: state-space model with transition Ξ³t=FΞ³tβˆ’1+qt\boldsymbol{\gamma}_t = \mathbf{F}\boldsymbol{\gamma}_{t-1} + \mathbf{q}_t and observation yt=AtΞ³t+wt\mathbf{y}_t = \mathbf{A}_t\boldsymbol{\gamma}_t + \mathbf{w}_t.

  5. Learned temporal model: a recurrent network (LSTM, Transformer) predicts Ξ³^t\hat{\boldsymbol{\gamma}}_t from {Ξ³^s}s<t\{\hat{\boldsymbol{\gamma}}_s\}_{s<t} and yt\mathbf{y}_t.

Theorem: Temporal Measurement Accumulation Bound

Consider a scene with KK moving point targets, each with velocity βˆ₯vkβˆ₯≀vmax⁑\|\mathbf{v}_k\| \leq v_{\max}, imaged at frame rate 1/Ξ”t1/\Delta t. If the per-frame measurement count is MM and the spatial dimension is N≫KN \gg K, then jointly recovering TT consecutive frames requires at least

MTβ‰₯O(Klog⁑(N/K)+KTβ‹…min⁑(1,vmax⁑Δt/Ξ”x))MT \geq \mathcal{O}\bigl(K \log(N/K) + KT \cdot \min(1, v_{\max}\Delta t / \Delta x)\bigr)

measurements in total, where Ξ”x\Delta x is the voxel size. The second term accounts for the innovation degrees of freedom introduced by motion.

The first term is the standard compressed sensing requirement for KK-sparse recovery. The second term captures the additional information needed as targets move: if vmax⁑Δtβ‰ͺΞ”xv_{\max}\Delta t \ll \Delta x, targets barely move between frames and temporal correlation strongly reduces the measurement requirement. If vmax⁑Δtβ‰₯Ξ”xv_{\max}\Delta t \geq \Delta x, each frame is essentially independent and we need Klog⁑(N/K)K\log(N/K) measurements per frame.

Proof
Sparse innovation model

Write Ξ³t=Ξ³tβˆ’1+Ξ΄t\boldsymbol{\gamma}_t = \boldsymbol{\gamma}_{t-1} + \boldsymbol{\delta}_t where Ξ΄t\boldsymbol{\delta}_t is the innovation. Under the velocity bound, βˆ₯Ξ΄tβˆ₯0≀Kβ‹…βŒˆvmax⁑Δt/Ξ”xβŒ‰\|\boldsymbol{\delta}_t\|_0 \leq K \cdot \lceil v_{\max}\Delta t / \Delta x \rceil.

Joint sparsity

The concatenated vector [Ξ³1;Ξ΄2;…;Ξ΄T]∈CNT[\boldsymbol{\gamma}_1; \boldsymbol{\delta}_2; \ldots; \boldsymbol{\delta}_T] \in \mathbb{C}^{NT} has sparsity K+(Tβˆ’1)Kmin⁑(1,vmax⁑Δt/Ξ”x)K + (T-1)K\min(1, v_{\max}\Delta t / \Delta x). Applying compressed sensing recovery to the joint system yields the stated measurement bound. β–‘\square

Example: Tracking a Walking Person at 10 fps

A radar images a room at 10 fps with M=50M = 50 measurements per frame. A person walks at 1.5 m/s. The grid has N=10,000N = 10{,}000 voxels at Ξ”x=5\Delta x = 5 cm. Which temporal prior is most appropriate and why?

Solution
Motion analysis

Between frames: displacement =1.5Γ—0.1=0.15= 1.5 \times 0.1 = 0.15 m =3Ξ”x= 3\Delta x. The person occupies ∼20\sim 20 voxels and moves coherently. This is a rigid-body translation.

Prior selection

Optical flow is most appropriate: the person translates as a rigid body with vβ‰ˆ1.5\mathbf{v} \approx 1.5 m/s. Smoothness prior: inappropriate (change is localised). Sparse innovation: partially captures the change but ignores the motion structure. Optical flow directly models the displacement, enabling prediction of the next frame's target location from the current estimate.

Measurement sufficiency

With Kβ‰ˆ20K \approx 20 target voxels and 3K=603K = 60 innovation voxels per frame, we need Mβ‰₯O(20log⁑500+60)β‰ˆ184M \geq \mathcal{O}(20 \log 500 + 60) \approx 184 measurements per frame for independent recovery. With temporal correlation via optical flow, the effective requirement drops to ∼50\sim 50--8080, making 50 measurements marginal but feasible with a strong temporal prior. β–‘\square

4D Neural Fields for RF

Extending NeRF and 3DGS to the time dimension creates 4D representations Ξ³(x,t)\boldsymbol{\gamma}(\mathbf{x}, t). Two approaches dominate:

  • Time-conditioned MLP: input (Ξ³(x),Ξ³(t))β†’(Οƒ,ρ)(\gamma(\mathbf{x}), \gamma(t)) \to (\sigma, \rho) where Ξ³(β‹…)\gamma(\cdot) denotes positional encoding. Simple but struggles with discontinuous motion.

  • Deformation field: learn d(x,t)\mathbf{d}(\mathbf{x}, t) that warps a canonical frame: Οƒt(x)=Οƒcanon(x+d(x,t))\sigma_t(\mathbf{x}) = \sigma_{\mathrm{canon}}(\mathbf{x} + \mathbf{d}(\mathbf{x}, t)). Better for rigid motion (vehicles, furniture).

For RF, the challenge is data: 4D training requires measurements at multiple viewpoints and multiple time steps. A 50-frame sequence with 10 viewpoints needs 500 CSI snapshots -- feasible for ISAC systems with multiple base stations, but expensive for single-AP setups.

Common Mistake: Temporal Aliasing in Dynamic RF Imaging

Mistake:

Imaging a scene where targets move faster than the frame rate can resolve (displacement >Ξ”x/2> \Delta x / 2 per frame) without accounting for temporal aliasing.

Correction:

Apply the temporal Nyquist criterion: the frame rate must satisfy fsβ‰₯2vmax⁑/Ξ”xf_s \geq 2 v_{\max} / \Delta x. For a 5 cm grid and 3 m/s motion: fsβ‰₯120f_s \geq 120 fps. If the radar operates at 10 fps, either increase the frame rate or use a motion model (Kalman filter, optical flow) to predict inter-frame positions.

Why This Matters: ISAC Resource Allocation for Dynamic Imaging

In ISAC systems (Chapter 29), dynamic imaging competes with communication for beam time and bandwidth. When the scene is static, most resources go to communication; when motion is detected (e.g., Doppler shift), resources shift to sensing. The optimal switching policy depends on the scene dynamics, communication QoS requirements, and the temporal prior's prediction accuracy -- a joint optimisation that connects dynamic imaging to ISAC beamforming design.

See full treatment in Chapter 29

Quick Check

A radar at 20 fps images a room where a door slowly swings open (angular velocity 0.5 rad/s). Which temporal prior is most appropriate?

Temporal smoothness

Sparse innovation

Optical flow

Correction:
Temporal smoothness

The door's motion is smooth and continuous (slow rotation). Temporal smoothness correctly penalises rapid changes while allowing gradual evolution of the reflectivity map.

Dynamic Scene Imaging

Reconstruction of time-varying scenes Ξ³(x,t)\boldsymbol{\gamma}(\mathbf{x}, t) from sequential measurements. Requires temporal priors (smoothness, sparsity, flow, learned models) to compensate for per-frame underdetermination.

Related: Temporal Prior

Temporal Prior

A regularisation constraint that exploits temporal structure in dynamic scenes. Examples: smoothness, sparse innovation, optical flow, Kalman state transition, learned recurrent models.

Related: Dynamic Scene Imaging

Key Takeaway

Dynamic scene imaging requires temporal priors to compensate for per-frame underdetermination. The choice of prior depends on the motion type: smoothness for slow changes, sparse innovation for localised changes, optical flow for rigid-body motion. 4D neural fields and ISAC-based temporal resource allocation are promising but largely unexplored for RF.

Bridging the Sim-to-Real GapPrimitive-Based Scene Representations