Ferkans — Interactive Telecom Tutor

The Rendering Equation as a Forward Model

In Chapter 1 we introduced the forward operator $\mathbf{y} = \mathcal{A}(\mathbf{g}) + \mathbf{w}$ abstractly. For optical imaging, the forward operator is precisely the rendering equation: given a 3D scene (geometry, materials, lighting), it computes the 2D image observed by a camera. Making this renderer differentiable means we can compute $\nabla_{\mathbf{g}} \mathcal{A}(\mathbf{g})$ and solve the inverse problem — inferring geometry, materials, and lighting from images — by gradient descent.

This section introduces the rendering equation and its inverse; Section 28.3 adapts both to RF wave propagation.

Definition:
The Rendering Equation

The rendering equation (Kajiya, 1986) describes how light interacts with surfaces:

$L_o(\mathbf{p}, \boldsymbol{\omega}_o) = L_e(\mathbf{p}, \boldsymbol{\omega}_o) + \int_{\mathcal{H}^2(\mathbf{n})} f_r(\mathbf{p}, \boldsymbol{\omega}_i, \boldsymbol{\omega}_o)\,L_i(\mathbf{p}, \boldsymbol{\omega}_i)\,(\boldsymbol{\omega}_i \cdot \mathbf{n})\,d\boldsymbol{\omega}_i,$

where:

$L_o(\mathbf{p}, \boldsymbol{\omega}_o)$ : outgoing radiance at surface point $\mathbf{p}$ in direction $\boldsymbol{\omega}_o$ .
$L_e$ : emitted radiance (light sources).
$f_r$ : BRDF (bidirectional reflectance distribution function) describing surface material.
$L_i(\mathbf{p}, \boldsymbol{\omega}_i)$ : incoming radiance from direction $\boldsymbol{\omega}_i$ .
$\mathbf{n}$ : surface normal at $\mathbf{p}$ .
$\mathcal{H}^2(\mathbf{n})$ : hemisphere above the surface.

The integral sums light arriving from all directions, weighted by the BRDF and the cosine foreshortening factor.

Definition:
Bidirectional Reflectance Distribution Function (BRDF)

The BRDF $f_r(\mathbf{p}, \boldsymbol{\omega}_i, \boldsymbol{\omega}_o)$ characterises how a surface reflects light:

$f_r = \frac{dL_o(\boldsymbol{\omega}_o)}{L_i\,\cos\theta_i\,d\boldsymbol{\omega}_i}.$

Physically valid BRDFs satisfy reciprocity ( $f_r(\boldsymbol{\omega}_i, \boldsymbol{\omega}_o) = f_r(\boldsymbol{\omega}_o, \boldsymbol{\omega}_i)$ ) and energy conservation ( $\int f_r \cos\theta_o\,d\boldsymbol{\omega}_o \leq 1$ ).

The Lambertian model ( $f_r = \rho/\pi$ , constant) is the simplest; the Cook-Torrance microfacet model is the standard in physically-based rendering. In RF, the "BRDF" is replaced by the bistatic scattering cross-section $\sigma(\boldsymbol{\omega}_i, \boldsymbol{\omega}_o)$ .

Definition:
Differentiable Rendering

A differentiable renderer computes $\nabla_\theta \mathcal{R}_\theta$ , enabling gradient-based optimisation of scene parameters $\theta$ . The key challenges:

Visibility discontinuities: When a surface edge moves, pixels transition between seeing different surfaces. The rendering function has discontinuities at silhouette edges.
Solutions:
- Soft rasterisation (SoftRas): Replace hard visibility with smooth sigmoid-based occupancy.
- Edge sampling: Explicitly detect and integrate over silhouette edges.
- Volume rendering (NeRF): Avoids hard visibility entirely by using a continuous density field.
- 3DGS: Gaussian alpha-compositing is inherently smooth.

,

Definition:
Inverse Rendering

Inverse rendering recovers scene properties (geometry, materials, lighting) from observed images by inverting the rendering equation. Given observed images $\{I_k\}$ and a differentiable renderer $\mathcal{R}_\theta$ :

$\hat{\theta} = \arg\min_\theta \sum_k \|\mathcal{R}_\theta(\text{view}_k) - I_k\|^2 + \lambda\,\mathcal{R}_{\text{reg}}(\theta),$

where $\theta = (\text{geometry}, \text{materials}, \text{lighting})$ and $\mathcal{R}_{\text{reg}}$ provides regularisation.

This is an ill-posed inverse problem: different combinations of geometry, materials, and lighting can produce the same image.

Theorem: Volume Rendering Integral

The colour $\hat{C}(\mathbf{r})$ of a pixel corresponding to camera ray $\mathbf{r}(t) = \mathbf{o} + t\,\mathbf{d}$ is:

$\hat{C}(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\,dt,$

where $\sigma(t)$ is the volume density, $\mathbf{c}$ is the emitted colour, and the transmittance is:

$T(t) = \exp\!\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s))\,ds\right).$

The discrete (quadrature) approximation used in NeRF is:

$\hat{C} = \sum_{i=1}^{N} T_i\,(1 - e^{-\sigma_i \delta_i})\,\mathbf{c}_i, \quad T_i = \exp\!\left(-\sum_{j=1}^{i-1} \sigma_j \delta_j\right),$

where $\delta_i = t_{i+1} - t_i$ and the sum runs over $N$ samples along the ray.

Volume rendering accumulates colour along a ray, weighting each point by how much light it emits ( $\sigma \cdot \mathbf{c}$ ) and how much of the ray has not been absorbed yet ( $T$ ). This is analogous to Beer-Lambert absorption — and to the RF matched-filter integral along a range cell.

Proof

Emission-absorption model

The change in radiance along a ray is governed by:

$\frac{dL}{dt} = -\sigma(t)\,L(t) + \sigma(t)\,\mathbf{c}(t),$

where the first term is absorption and the second is emission.

Integrating factor

Multiply both sides by the integrating factor $e^{\int_{t_n}^{t}\sigma(s)\,ds} = 1/T(t)$ :

$\frac{d}{dt}\!\left[\frac{L(t)}{T(t)}\right] = \frac{\sigma(t)\,\mathbf{c}(t)}{T(t)}.$

Integration

Integrating from $t_n$ to $t_f$ with $L(t_n) = 0$ (background black) gives:

$L(t_f) = \int_{t_n}^{t_f} T(t)\,\sigma(t)\,\mathbf{c}(t)\,dt. \quad \blacksquare$

Differentiable Volume Rendering: 1D Ray Marching

Visualise the 1D volume rendering integral along a single ray. The top panel shows the density profile $\sigma(t)$ and colour $\mathbf{c}(t)$ along the ray; the bottom panel shows the transmittance $T(t)$ and the accumulated colour. Observe how increasing the number of samples improves the quadrature approximation, and how high-density regions absorb light rapidly, causing $T(t)$ to drop.

Parameters

Samples per ray64

Density scale1

Density profile

Example: NeuS: Neural Implicit Surface Rendering

NeuS represents a scene as a neural SDF $f_\theta(\mathbf{p})$ (Chapter 24) and renders it via volume rendering. Explain how the SDF value is converted to a volume density for the rendering integral, and why this is preferable to NeRF's unconstrained density field.

Solution

SDF to density conversion

NeuS defines a density-like weight function via the logistic function applied to the SDF:

$w(t) = \frac{d}{dt}\!\left[-\Phi_s(f_\theta(\mathbf{r}(t)))\right],$

where $\Phi_s(x) = 1/(1 + e^{-sx})$ is the sigmoid with learnable inverse width $s$ . The weight $w(t)$ peaks at the zero-level set of the SDF (the surface) and decays away from it.

Rendering

The volume rendering integral becomes:

$\hat{C}(\mathbf{r}) = \int_{t_n}^{t_f} w(t)\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\,dt.$

As $s \to \infty$ , $w(t) \to \delta(f_\theta(\mathbf{r}(t)))$ , recovering surface rendering. During training, $s$ is gradually increased (annealed) to transition from volume to surface rendering.

Advantage over NeRF

NeRF's density field has no geometric interpretation — the density can be nonzero everywhere, producing "floaters" and fuzzy surfaces. NeuS constrains the geometry to an SDF, guaranteeing a well-defined surface with exact normals ( $\mathbf{n} = \nabla f_\theta / \|\nabla f_\theta\|$ ), which is critical for downstream tasks like relighting and material estimation.

Common Mistake: Visibility Discontinuities in Differentiable Rendering

Mistake:

Naively differentiating a hard rasteriser with respect to mesh vertex positions, obtaining zero gradients almost everywhere and undefined gradients at silhouette edges.

Correction:

Hard rasterisation is a step function in object coordinates: a pixel is either inside or outside a triangle. The gradient is zero (no information) in the interior and undefined at edges.

Solutions: (i) Soft rasterisation (SoftRas) replaces the step with a sigmoid: each pixel gets a soft "occupancy" from nearby triangles. (ii) Volume rendering (NeRF/NeuS) avoids hard visibility altogether. (iii) 3DGS uses smooth Gaussian splatting. All three approaches provide informative gradients for inverse rendering.

BRDF (Bidirectional Reflectance Distribution Function)

A function $f_r(\boldsymbol{\omega}_i, \boldsymbol{\omega}_o)$ that describes the ratio of reflected radiance to incident irradiance at a surface point, characterising the material's reflective properties.

Related: The Rendering Equation

Inverse Rendering

The problem of recovering scene geometry, materials, and lighting from observed images, typically solved via gradient-based optimisation through a differentiable renderer.

Related: Inverse Rendering

Quick Check

In the volume rendering integral, what does the transmittance $T(t) = \exp(-\int_{t_n}^{t} \sigma(s)\,ds)$ represent physically?

The probability that the ray has not been absorbed by depth $t$

The total emitted light up to depth $t$

The surface normal at depth $t$

The accumulated density along the ray

Correction:

The probability that the ray has not been absorbed by depth

t

$T(t)$ is the probability that a photon travelling along the ray reaches depth $t$ without being absorbed. It decreases monotonically from $T(t_n) = 1$ (no absorption yet) as the ray passes through regions of nonzero density.

Key Takeaway

The rendering equation is the forward model for optical imaging. Differentiable renderers — SoftRas for meshes, volume rendering for NeRF/NeuS, Gaussian splatting for 3DGS — enable gradient-based inverse rendering: estimating geometry, materials, and lighting from images. The core challenge is handling visibility discontinuities smoothly so that gradients are informative.

Differentiable Rendering

The Rendering Equation as a Forward Model

Definition: The Rendering Equation

Definition: Bidirectional Reflectance Distribution Function (BRDF)

Definition: Differentiable Rendering

Definition: Inverse Rendering

Theorem: Volume Rendering Integral

Emission-absorption model

Integrating factor

Integration

Differentiable Volume Rendering: 1D Ray Marching

Parameters

Example: NeuS: Neural Implicit Surface Rendering

SDF to density conversion

Rendering

Advantage over NeRF

Common Mistake: Visibility Discontinuities in Differentiable Rendering

BRDF (Bidirectional Reflectance Distribution Function)

Inverse Rendering

Quick Check

Key Takeaway

Definition:
The Rendering Equation

Definition:
Bidirectional Reflectance Distribution Function (BRDF)

Definition:
Differentiable Rendering

Definition:
Inverse Rendering