Broadcasting

Broadcasting: NumPy's Killer Feature

Broadcasting is the mechanism that lets NumPy operate on arrays of different shapes without explicit loops or data replication. Adding a scalar to an array, adding a row vector to every row of a matrix, or computing pairwise distances between point clouds — all of these are broadcasting in action.

Mastering broadcasting is what separates "I use NumPy" from "I think in NumPy."

Definition:
Broadcasting

Broadcasting is NumPy's mechanism for performing element-wise operations on arrays with different shapes. Instead of replicating data, NumPy virtually stretches the smaller array to match the larger one.

a = np.array([[1, 2, 3],
              [4, 5, 6]])   # shape (2, 3)
b = np.array([10, 20, 30])  # shape (3,)

c = a + b    # b is broadcast to shape (2, 3)
# c = [[11, 22, 33],
#      [14, 25, 36]]

No data is copied: NumPy uses stride tricks internally (stride 0 along the broadcast axis).

Definition:
Broadcasting Rules

Two arrays are compatible for broadcasting if, for each trailing dimension (aligning shapes from the right):

The dimensions are equal, OR
One of them is 1 (or missing)

When a dimension is 1 or missing, it is stretched to match the other.

Examples:

Shape A	Shape B	Result	Rule Applied
`(3, 4)`	`(4,)`	`(3, 4)`	B gets axis 0 of size 1, stretched to 3
`(3, 1)`	`(1, 4)`	`(3, 4)`	A stretched along axis 1, B along axis 0
`(5, 3, 4)`	`(3, 1)`	`(5, 3, 4)`	B gets axis 0, axis 2 stretched
`(3, 4)`	`(3,)`	Error	Trailing dims 4 vs 3 — incompatible

Definition:
np.newaxis and Explicit Axis Insertion

np.newaxis (alias for None) inserts a length-1 axis, enabling broadcasting along that dimension:

a = np.array([1, 2, 3])      # shape (3,)
b = np.array([10, 20])       # shape (2,)

# Cannot add directly: shapes (3,) and (2,) are incompatible
# Solution: insert axes to create outer product structure
c = a[:, np.newaxis] + b[np.newaxis, :]
# a[:, np.newaxis] has shape (3, 1)
# b[np.newaxis, :] has shape (1, 2)
# result has shape (3, 2) — broadcasting as outer addition

Equivalent to np.expand_dims(a, axis=1).

Theorem: Broadcasting Compatibility Theorem

Arrays with shapes $S_A = (a_1, a_2, \ldots, a_m)$ and $S_B = (b_1, b_2, \ldots, b_n)$ are broadcast-compatible if and only if, after left-padding the shorter shape with 1s to equalize lengths, every pair of corresponding dimensions $(a_k, b_k)$ satisfies $a_k = b_k$ or $\min(a_k, b_k) = 1$ .

The output shape is $(\max(a_1, b_1), \max(a_2, b_2), \ldots)$ .

Broadcasting aligns dimensions from the right (like aligning decimal points). Missing dimensions on the left are treated as size 1. A size-1 dimension can be "stretched" to any size because repeating a single value is always valid.

Theorem: Broadcasting as Implicit Outer Products

For 1-D arrays $\mathbf{u} \in \mathbb{R}^m$ and $\mathbf{v} \in \mathbb{R}^n$ , the operation u[:, None] * v[None, :] computes the outer product $\mathbf{u} \mathbf{v}^T \in \mathbb{R}^{m \times n}$ without forming any intermediate copies.

Inserting axes creates shapes (m, 1) and (1, n). Broadcasting stretches both to (m, n) and multiplies element-wise, which is exactly the outer product definition: $(\mathbf{u}\mathbf{v}^T)_{ij} = u_i \cdot v_j$ .

Example: Adding Bias Vector to Batch of Samples

Given a batch of 100 samples with 5 features (shape (100, 5)) and a bias vector of shape (5,), add the bias to each sample using broadcasting. Then compute per-feature means.

Solution

Broadcasting addition

rng = np.random.default_rng(42)
X = rng.standard_normal((100, 5))    # shape (100, 5)
bias = np.array([1.0, -2.0, 0.5, 3.0, -1.5])  # shape (5,)

# Broadcasting: (100, 5) + (5,) -> (100, 5)
X_biased = X + bias

# Per-feature means (should be close to bias values)
print(X_biased.mean(axis=0))
# ~ [1.0, -2.0, 0.5, 3.0, -1.5]

Example: Pairwise Distances via Broadcasting

Compute the pairwise Euclidean distance matrix for $n$ points in 2-D using broadcasting (no loops, no np.tile).

Solution

Broadcasting approach

points = rng.standard_normal((50, 2))  # 50 points in 2-D

# Expand dims for broadcasting:
# points[:, None, :] has shape (50, 1, 2)
# points[None, :, :] has shape (1, 50, 2)
diff = points[:, None, :] - points[None, :, :]   # (50, 50, 2)
dist = np.sqrt((diff ** 2).sum(axis=-1))          # (50, 50)

print(dist.shape)           # (50, 50)
print(dist[0, 0])           # 0.0 (distance to self)
assert np.allclose(dist, dist.T)  # symmetric

Example: Avoiding np.tile and np.repeat

Show why np.tile / np.repeat are almost never needed when broadcasting is available.

Solution

Wasteful approach with np.tile

a = np.arange(5)              # shape (5,)
b = np.arange(3)              # shape (3,)

# BAD: explicitly replicate data
a_tiled = np.tile(a, (3, 1))  # shape (3, 5) — copies data
b_tiled = np.tile(b, (5, 1)).T  # shape (3, 5) — copies data
result_bad = a_tiled + b_tiled

# GOOD: broadcasting (no data copied)
result_good = a[np.newaxis, :] + b[:, np.newaxis]  # (3, 5)

assert np.array_equal(result_bad, result_good)
# result_good uses zero memory overhead beyond the output

Broadcasting Step-by-Step Visualizer

See how broadcasting works for different shape combinations. Watch dimensions align from the right, stretch, and produce the output shape.

Parameters

Broadcasting Animation

Animated visualization showing how NumPy broadcasts arrays step by step: padding shapes, stretching size-1 dimensions, and computing the result.

Parameters

Broadcasting Rules — The three broadcasting rules illustrated: (1) pad shorter shape with 1s on the left, (2) stretch size-1 dimensions, (3) error if dimensions differ and neither is 1.

Broadcasting vs np.tile / np.repeat

Aspect	Broadcasting	np.tile / np.repeat
Memory	Zero extra allocation (stride tricks)	Copies entire array
Speed	Fast (no data movement)	Slower (allocation + copy)
Readability	Concise once you learn the rules	Explicit but verbose
When to use	Almost always	When you truly need a replicated array in memory
Example	`a[:, None] + b[None, :]`	`np.tile(a, (n, 1)) + np.tile(b, (1, m))`

Quick Check

What is the result shape of np.ones((3, 1)) + np.ones((1, 4))?

(3, 4)

(3, 1, 4)

Error: incompatible shapes

(1, 4)

Correction:

(3, 4)

Broadcasting: (3,1) and (1,4) -> stretch to (3,4).

Common Mistake: Trailing Dimension Mismatch

Mistake:

Trying to add arrays with incompatible trailing dimensions:

a = np.ones((3, 4))
b = np.ones((3,))
c = a + b   # ValueError: shapes (3,4) and (3,) not aligned

Trailing dims: 4 vs 3 — neither is 1, so broadcasting fails.

Correction:

Reshape b to make the dimension alignment explicit:

c = a + b[:, np.newaxis]   # (3, 4) + (3, 1) -> (3, 4)

broadcasting

NumPy's mechanism for performing element-wise operations on arrays with different shapes by virtually stretching size-1 dimensions.

Broadcasting Patterns

python

Broadcasting rules with practical examples: pairwise distances, bias addition, outer products.

# Code from: ch05/python/broadcasting_patterns.py
# Load from backend supplements endpoint

Key Takeaway

Broadcasting aligns dimensions from the right and stretches size-1 axes. It replaces np.tile / np.repeat with zero-copy virtual expansion. The pattern a[:, None] + b[None, :] creates outer operations — master this and you rarely need explicit loops.

Indexing and Slicing — Advanced Vectorization and Avoiding Loops

Broadcasting

Broadcasting: NumPy's Killer Feature

Definition: Broadcasting

Definition: Broadcasting Rules

Definition: np.newaxis and Explicit Axis Insertion

Theorem: Broadcasting Compatibility Theorem

Theorem: Broadcasting as Implicit Outer Products

Example: Adding Bias Vector to Batch of Samples

Broadcasting addition

Example: Pairwise Distances via Broadcasting

Broadcasting approach

Example: Avoiding np.tile and np.repeat

Wasteful approach with np.tile

Broadcasting Step-by-Step Visualizer

Parameters

Broadcasting Animation

Parameters

Broadcasting Rules

Broadcasting vs np.tile / np.repeat

Quick Check

Common Mistake: Trailing Dimension Mismatch

broadcasting

Broadcasting Patterns

Key Takeaway

Definition:
Broadcasting

Definition:
Broadcasting Rules

Definition:
np.newaxis and Explicit Axis Insertion