Exercises

Implement scaled dot-product attention from scratch. Verify on Q, K, V of shape (1, 4, 8).

Compute sinusoidal positional encodings for d_model=64, max_len=100. Plot as a heatmap.

Create a causal mask for sequence length 8 and verify masked positions get zero attention.

Implement nn.MultiheadAttention and compare output to your from-scratch implementation.

Count parameters in a transformer block with d_model=512, n_heads=8, d_ff=2048.

Implement multi-head attention from scratch with h=4 heads and d_model=64.

Build a 4-layer transformer encoder and train it for sequence classification.

Implement ViT for CIFAR-10 (32x32, patch_size=4). Train for 20 epochs.

Visualise attention weights for each head and interpret what patterns emerge.

Compare transformer vs LSTM on a sequence classification task. Report accuracy and speed.

Implement a transformer decoder with causal masking and cross-attention.

Implement KV-cache for efficient autoregressive generation. Measure speedup.

Implement learned positional embeddings and compare to sinusoidal on a task.

Implement rotary position embeddings (RoPE) and integrate with multi-head attention.

Apply a transformer to OFDM channel estimation: tokens = subcarriers, features = pilot measurements.

Implement Flash Attention v2 (tiled, IO-aware) and compare memory usage to standard attention.