Ferkans — Interactive Telecom Tutor

From Principle to Practice

The previous three sections derived the shaping gain as a function of the shaping lattice's normalised second moment $G(\Lambda_s)$ and bounded the achievable gain at $\pi e / 6 \approx 1.53$ dB. The question now is operational: given a target shaping gain, how do we actually pick constellation points from a Voronoi region — or from a sphere — in a way that is low-complexity and rate-efficient?

Two classical answers appeared in the 1990s. Shell mapping (Laroia, Farvardin, Tretter, 1994) quantises by energy shells and uses a combinatorial encoder to index points by energy. Trellis shaping (Forney, 1992) uses a shaping convolutional code on top of the base constellation to drive low-energy sequences. Both trade complexity for shaping gain, and both appear in practical standards: shell mapping in V.34 voice-band modems and in some DVB-S2 extensions; trellis shaping in the original V.fast proposals. This section surveys both and ends by situating them next to the modern alternative of probabilistic shaping (Ch. 19).

,

Definition:
Shell Mapping

Fix a base constellation $\mathcal{X}_0$ (typically a 1D PAM alphabet $\{\pm 1, \pm 3, \ldots, \pm (M-1)\}$ ) partitioned into energy shells $\mathcal{S}_0, \mathcal{S}_1, \ldots, \mathcal{S}_K$ , where $\mathcal{S}_k$ contains the points of energy $k$ . A shell-mapped constellation in $n$ dimensions uses an $n$ -tuple of base points, indexed by a shell label $(k_1, \ldots, k_n) \in \{0, \ldots, K\}^n$ whose total energy $\sum_i k_i$ lies below a chosen threshold $K_{\max}$ . Points with large total shell index are excluded; points with small total shell index are kept.

The shell-mapping encoder is a bijection between bit-strings of length $R \cdot n$ and the set of admissible shell labels; it is implemented via a trie of size $O(n K_{\max})$ and runs in $O(n K_{\max})$ time.

The term "shell" reflects the geometric picture: the set of admissible points is the intersection of the base $n$ -cube with a sphere — an approximation of the Voronoi region of the $n$ -ball.

,

Shell Mapping Encoder (Laroia–Farvardin–Tretter)

Complexity:

O(n K_{\max})

time,

O(n K_{\max})

memory

Input. Message bits

\mathbf{m} \in \{0, 1\}^{R n}

,

dimension

n

, maximum total shell index

K_{\max}

, shell

population

\#\mathcal{S}_k

for

k = 0, \ldots, K

.

Output. Base-constellation

n

-tuple

(\mathbf{x}_1, \ldots, \mathbf{x}_n)

in an admissible shell.

Preprocessing (one-time).

1. Compute the cumulative shell counts

N_{j}(k) =

(number of admissible labels

(k_1, \ldots, k_j)

with

\sum_i k_i = k

), via the convolution

N_j(k) = \sum_{\ell = 0}^{k} N_{j-1}(k-\ell) \cdot \#\mathcal{S}_\ell

.

2. Tabulate the total admissible count

N_{\rm tot} = \sum_{k = 0}^{K_{\max}} N_n(k)

. Verify

2^{Rn} \le N_{\rm tot}

.

Encoding.

3. Interpret

\mathbf{m}

as an integer

I \in [0, 2^{Rn})

.

4. for

j = n, n-1, \ldots, 1

do

5.

\quad

Find the largest

k

with

\sum_{k' < k} N_{j-1}(K_{\rm left} - k') \cdot \#\mathcal{S}_{k'} \le I

.

6.

\quad

Assign shell index

k_j = k

, subtract the running sum from

I

.

7.

\quad

Decrement

K_{\rm left} \leftarrow K_{\rm left} - k_j

.

8. end for

9. Use the residual

I

to index the specific point within

\mathcal{S}_{k_j}

for each

j

.

Output. The chosen

(\mathbf{x}_1, \ldots, \mathbf{x}_n)

.

The algorithm is essentially an arithmetic coder run backwards — it consumes uniform bits from the message and emits shell labels with probabilities proportional to the cumulative counts. The decoder inverts the procedure.

Shell Populations for 1D PAM Shaping

For a base 1D PAM alphabet with $M = 16$ points (energies $1, 9, 25, \ldots, 225$ ), the plot shows the shell populations $\#\mathcal{S}_k$ and the cumulative admissible count $N_n(k)$ for $n = 16$ dimensions. The cumulative count levels off at $K_{\max}$ , defining the set of shell-mapped sequences. Move the slider to see how raising $K_{\max}$ admits more high-energy shells (recovering more rate) at the cost of shaping gain.

Parameters

Max energy shell

K_{\max}

12

Theorem: Shell Mapping Rate Loss

Let $M$ be the base PAM alphabet size and let the shell-mapping encoder restrict to sequences with total shell index $\le K_{\max}$ . The rate loss relative to unconstrained PAM (in bits per dimension) is

$\Delta R \;=\; \log_2 M - \frac{1}{n} \log_2 N_n(K_{\max}),$

where $N_n(K_{\max})$ is the number of admissible shell sequences. In the high-rate limit $n, M \to \infty$ with $K_{\max}/n$ chosen to match a target shaping gain $\gamma_s$ , the rate loss tends to $R_{\rm loss}(\gamma_s)$ , a universal function that vanishes as $\gamma_s \to 0$ and diverges as $\gamma_s \to \pi e / 6$ .

Rate loss is the price paid for shaping. Forbidding high-energy sequences removes some codewords — the fraction removed is the rate loss. As we demand more shaping gain we must forbid more sequences, and in the limit of the $\pi e / 6$ ceiling we forbid all but an exponentially thin slice of the cube — an infinite rate loss.

Show Hint

Count the number of admissible shell sequences and compare to $M^n$ .

Use Stirling or Laplace's method to extract the $n \to \infty$ scaling.

Invert the resulting function to express rate loss in terms of shaping gain.

Proof

Rate loss as a log-count ratio

The unconstrained PAM alphabet gives $M^n$ sequences in $n$ dimensions. The shell-mapping encoder selects from $N_n(K_{\max})$ of these. Expressing both in bits per dimension: $\frac{1}{n} \log_2 M^n - \frac{1}{n} \log_2 N_n(K_{\max}) = \log_2 M - \frac{1}{n} \log_2 N_n(K_{\max}).$

Asymptotic form

The generating function for shell populations of a 1D PAM alphabet is $P(z) = \sum_k \#\mathcal{S}_k z^k$ . By standard saddle-point analysis, $\frac{1}{n} \log_2 N_n(K_{\max}) \sim \log_2 M - H_\infty(\gamma_s),$ where $H_\infty$ is the per-dimension relative entropy between the uniform and the shaped distribution, itself a function only of the shaping gain.

Limit behaviour

$H_\infty \to 0$ as $\gamma_s \to 0$ (no shaping means no rate loss). $H_\infty \to \infty$ as $\gamma_s \to \pi e / 6$ (the ultimate shaping gain would require an infinitely concentrated distribution). The practical sweet spot — where $H_\infty \sim 0.02$ bits/dim — recovers roughly $0.8$ – $1.0$ dB of shaping gain, which is where V.34 modems operated. $\blacksquare$

,

Definition:
Trellis Shaping

Trellis shaping uses a rate- $k/n$ convolutional code $\mathcal{C}_{\rm sh}$ (the shaping code) on top of a base constellation. Each $n$ -tuple of information bits is mapped to a base point; the shaping code then adds a coset-representative from $\mathcal{C}_{\rm sh}^\perp$ (the dual code) to the base-point sequence, chosen by a Viterbi search for the minimum-energy representative in the coset. The encoder emits the minimum-energy coset element; the decoder recovers the information bits by projecting onto the coset (the shaping code is "invisible" to the data).

Because the Viterbi shaping search considers $2^{\nu}$ states at each step (for a shaping code of constraint length $\nu$ ), its complexity is linear in the sequence length but exponential in $\nu$ . Typical shaping codes are short (4- or 8-state), trading some shaping gain against decoder cost.

Trellis shaping is essentially a lattice vector quantiser for the minimum-energy coset representative. The trellis structure is what makes the quantisation tractable: a Viterbi search replaces an exhaustive search over $2^{k \cdot n/k} = 2^n$ candidate sequences.

,

Historical Note: Shell Mapping: From Conway–Sloane to V.34

1983–1998

The idea of shell-mapping traces back to Conway and Sloane's 1983 paper A fast encoding method for lattice codes and quantizers, which gave polynomial-time encoders for Voronoi constellations of Barnes–Wall lattices. Laroia, Farvardin, and Tretter generalised this to arbitrary lattices and arbitrary base alphabets in 1994, introducing the "trie-over-shells" encoder that has carried the practical shaping load ever since. The ITU adopted a 16-dimensional shell-mapping variant in the V.34 voice-band modem standard in 1998, where it contributed roughly $0.8$ dB of shaping gain to the total $4.8$ dB gain that pushed dial-up modems from 19.2 kb/s to 33.6 kb/s. V.34 was the last word in voice-band modems — after it, DSL took over the consumer market — and shell mapping lived on mostly as an inspirational idea until probabilistic shaping (Ch. 19) took over the function in 2015.

, ,

Historical Note: Forney's 1992 Trellis Shaping

1992

Forney's Trellis Shaping appeared in the IEEE Transactions on Information Theory (May 1992) as the final instalment of a three-paper sequence: coset codes (1988), multidimensional constellations (1989), and now trellis shaping (1992). The construction is the natural dual to TCM — where TCM uses a trellis code to structure the coding lattice, trellis shaping uses a trellis code to quantise to the shaping lattice. The two can be stacked: a TCM-coded constellation with trellis shaping on top achieves both coding and shaping gain. The scheme was proposed for V.fast (what became V.34), but was rejected in favour of the simpler shell-mapping approach — a common pattern in standards.

Shaping Methods Compared

Method	Gain (dB)	Complexity	Rate loss	Standards use
Cube (no shaping)	$0$	None	None	5G NR QAM, most modern
Shell mapping (16-D)	$\sim 0.8$	$O(n K_{\max})$ arithmetic	Small, tunable	V.34 (1998)
Trellis shaping	$\sim 0.7$ – $1.0$	Viterbi, $2^\nu$ states	Small	V.fast proposal (rejected)
Voronoi ( $E_8$ , $\Lambda_{24}$ )	$0.65$ / $1.03$	Closest-lattice-point decoding	None	Research; DVB-S2X shaping annex
Probabilistic shaping	$\sim 1$ dB @ rate $\ge 4$	Distribution matcher (CCDM)	Small, tunable	DVB-S2X, optical coherent (Ch. 19)
$n$ -ball (ideal limit)	$\pi e / 6 \approx 1.53$	$\infty$	Finite, tunable	Unachievable in practice

⚠️Engineering Note

The Hidden Cost of Shaping: Rate-Adaptation Overhead

All shaping schemes — shell mapping, trellis shaping, probabilistic amplitude shaping — pay a small but unavoidable rate-adaptation overhead. To achieve shaping gain $\gamma_s$ at a target rate $R$ , the shaping mechanism must allocate $\sim \log_2(\gamma_s)$ bits of side information per symbol to communicate which part of the shaped distribution is being used. This overhead is invisible when $\gamma_s$ is small but becomes the dominant cost as $\gamma_s \to \pi e / 6$ . Practical deployments (e.g., V.34, DVB-S2X) set $\gamma_s$ at $\approx 1$ dB to keep the overhead under $1\%$ of the total rate.

Practical Constraints

•
Rate-overhead: $\sim 1$ bit per 16-D symbol in V.34 shell mapping
•
Decoder complexity: Viterbi search ( $2^\nu$ states) for trellis shaping
•
Latency: shell mapping requires $n$ symbols of buffering to form an $n$ -D shell label

📋 Ref: DVB-S2X ETSI EN 302 307-2, Annex M (probabilistic shaping)

Why This Matters: Forward Reference: Probabilistic Shaping in 5G and Optical

The lattice and shell-mapping shaping schemes of this section have largely been superseded in modern standards by probabilistic amplitude shaping (PAS) — the topic of Ch. 19. PAS reaches $\sim 1$ dB of shaping gain with a simpler architecture: a distribution matcher (CCDM or PMF-quantisation) shapes the amplitude distribution, an off-the-shelf LDPC code provides error correction, and the combination recovers almost all of the ultimate $1.53$ dB at a fraction of the complexity of shell mapping. The information-theoretic ceiling $\pi e / 6$ that we derived in s04 still applies to PAS — the proof is identical, since it depends only on the Gaussian maximum-entropy principle. Ch. 19 will revisit the derivation in the PAS context.

Common Mistake: Shaping Adds Latency

Mistake:

Using shell mapping or trellis shaping in a low-latency link without budgeting for the buffering overhead.

Correction:

Shell mapping in $n = 16$ dimensions requires $16$ symbols of buffering at the encoder (to form a shell label) and $16$ symbols at the decoder (to unmap the shell label). At a $10$ GBaud line rate this is $\approx 1.6$ ns — negligible. At a $1$ kBaud voice-band link (V.34) it was $\approx 16$ ms per direction, which was just acceptable. For ultra-low-latency links ( $< 1$ ms budget, 5G URLLC), lattice shaping is typically avoided altogether and probabilistic shaping is preferred (which can operate on single symbols).

Shell mapping

A shaping algorithm that partitions the base constellation into energy shells and uses an arithmetic-coder-like encoder to select shell labels with prescribed (non-uniform) probabilities. Due to Laroia, Farvardin, Tretter (1994); used in V.34.

Trellis shaping

A shaping technique that adds a Viterbi-decoded coset-representative from the dual of a shaping convolutional code to the transmitted sequence, choosing the minimum-energy representative. Due to Forney (1992).

Quick Check

Shell mapping in $n = 16$ dimensions requires how many base-symbol times of buffering before the first shaped symbol can be transmitted?

$1$ symbol

$n = 16$ symbols

$K_{\max}$ symbols

No buffering

Correction:

n = 16

symbols

The shell-mapping encoder forms one $n$ -tuple (a shell label) at a time. It must wait to collect all $n$ base-symbol candidates before the first shaped symbol can be emitted. For $n = 16$ this is $16$ base-symbol times. At 10 GBaud this is $1.6$ ns (negligible); at $1$ kBaud it is $16$ ms (borderline).

Key Takeaway

Practical shaping trades complexity for shaping gain. Shell mapping (Laroia–Farvardin–Tretter, 1994) uses a combinatorial encoder over energy shells; trellis shaping (Forney, 1992) uses a Viterbi search over coset representatives. Both reach $\sim 0.8$ – $1.0$ dB of shaping gain at moderate complexity — well short of the $\pi e / 6 \approx 1.53$ dB ceiling. Modern standards have largely replaced both with probabilistic amplitude shaping (Ch. 19), which reaches similar gain with a distribution matcher and a capacity-approaching binary code. The ceiling is the same; only the mechanism changes.

Practical Shaping: Shell Mapping and Trellis Shaping

From Principle to Practice

Definition: Shell Mapping

Shell Mapping Encoder (Laroia–Farvardin–Tretter)

Shell Populations for 1D PAM Shaping

Parameters

Theorem: Shell Mapping Rate Loss

Rate loss as a log-count ratio

Asymptotic form

Limit behaviour

Definition: Trellis Shaping

Historical Note: Shell Mapping: From Conway–Sloane to V.34

Historical Note: Forney's 1992 Trellis Shaping

Shaping Methods Compared

The Hidden Cost of Shaping: Rate-Adaptation Overhead

Why This Matters: Forward Reference: Probabilistic Shaping in 5G and Optical

Common Mistake: Shaping Adds Latency

Shell mapping

Trellis shaping

Quick Check

Key Takeaway

Definition:
Shell Mapping

Definition:
Trellis Shaping