Complex Tensors in PyTorch

Definition:

Complex Tensor Types

PyTorch supports two complex dtypes:

dtype Real part Imaginary part Total bytes
complex64 float32 float32 8
complex128 float64 float64 16

Creation:

z = torch.tensor([1+2j, 3+4j])                  # complex128
z = torch.complex(torch.randn(3), torch.randn(3))  # complex64
z = torch.randn(3, dtype=torch.complex128)       # random complex

Access real and imaginary parts via .real and .imag (these are views, not copies).

Complex tensors store real and imaginary parts interleaved in memory, matching NumPy's convention and enabling zero-copy conversion.

Definition:

Wirtinger Derivatives

For a function f:C→Rf: \mathbb{C} \to \mathbb{R}, the Wirtinger derivatives are:

βˆ‚fβˆ‚z=12(βˆ‚fβˆ‚xβˆ’jβˆ‚fβˆ‚y),βˆ‚fβˆ‚zβˆ—=12(βˆ‚fβˆ‚x+jβˆ‚fβˆ‚y)\frac{\partial f}{\partial z} = \frac{1}{2}\left(\frac{\partial f}{\partial x} - j\frac{\partial f}{\partial y}\right), \quad \frac{\partial f}{\partial z^*} = \frac{1}{2}\left(\frac{\partial f}{\partial x} + j\frac{\partial f}{\partial y}\right)

where z=x+jyz = x + jy. PyTorch's autograd returns the conjugate Wirtinger derivative βˆ‚fβˆ‚zβˆ—\frac{\partial f}{\partial z^*} because this is the steepest-descent direction for real-valued loss functions of complex parameters.

This convention means z.grad gives the direction to subtract for gradient descent, consistent with the real-valued case.

Theorem: Conjugate Wirtinger Derivative as Gradient

For a real-valued function L:Cn→RL: \mathbb{C}^n \to \mathbb{R}, the steepest-descent direction is:

Ξ”z=βˆ’Ξ±(βˆ‚Lβˆ‚zβˆ—)\Delta z = -\alpha \left(\frac{\partial L}{\partial z^*}\right)

This is exactly what PyTorch stores in z.grad. The update rule z←zβˆ’Ξ±β‹…z.gradz \leftarrow z - \alpha \cdot z.\text{grad} minimizes LL just as in the real case.

Think of a complex parameter z=x+jyz = x + jy as two real parameters. The Wirtinger calculus packages the xx and yy gradients into a single complex number that points in the steepest-descent direction.

Example: Gradient of |z|^2 via Autograd

Verify that autograd correctly computes βˆ‚βˆ‚zβˆ—βˆ£z∣2=z\frac{\partial}{\partial z^*} |z|^2 = z.

Solution
Implementation
import torch

z = torch.tensor([1+2j, 3-1j], dtype=torch.complex128,
                 requires_grad=True)
L = (z * z.conj()).real.sum()     # |z|^2 = z * z*
L.backward()
print(f"z.grad = {z.grad}")       # should equal z
print(f"z      = {z}")
print(f"Match: {torch.allclose(z.grad, z)}")
Explanation

Since ∣z∣2=zzβˆ—|z|^2 = z z^*, the conjugate Wirtinger derivative βˆ‚βˆ‚zβˆ—(zzβˆ—)=z\frac{\partial}{\partial z^*}(z z^*) = z. PyTorch confirms this with z.grad == z.

Example: FFT and Spectral Analysis with PyTorch

Compute the FFT of a signal x(t)=cos⁑(2Ο€β‹…50t)+0.5sin⁑(2Ο€β‹…120t)x(t) = \cos(2\pi \cdot 50t) + 0.5\sin(2\pi \cdot 120t) sampled at 1000 Hz using torch.fft, and find the dominant frequencies.

Solution
Implementation
import torch

fs = 1000.0
t = torch.arange(0, 1.0, 1.0/fs, dtype=torch.float64)
x = torch.cos(2 * torch.pi * 50 * t) + 0.5 * torch.sin(2 * torch.pi * 120 * t)

X = torch.fft.fft(x)
freqs = torch.fft.fftfreq(len(x), 1.0/fs)

# Power spectrum (one-sided)
mask = freqs >= 0
power = (X[mask].abs() ** 2) / len(x)

# Find peaks
top_k = torch.topk(power, 3)
print("Dominant frequencies:")
for idx in top_k.indices:
    print(f"  {freqs[mask][idx]:.0f} Hz, power={power[idx]:.2f}")
Key Points
  • torch.fft.fft returns complex128 when input is float64.
  • torch.fft.fftfreq mirrors numpy.fft.fftfreq.
  • The entire pipeline supports autograd, so you can differentiate through FFT operations.

Interactive FFT with Complex Tensors

Adjust signal frequencies, amplitudes, and sampling rate to see the FFT spectrum update in real time. Observe aliasing when the sampling rate is too low.

Parameters

Wirtinger Derivatives on the Complex Plane

Wirtinger Derivatives on the Complex Plane
Visualization of the Wirtinger derivative partialf/partialzβˆ—\\partial f / \\partial z^* as the steepest-descent direction for a real-valued function f(z)=∣zβˆ’z0∣2f(z) = |z - z_0|^2 on the complex plane.

Quick Check

When PyTorch computes z.grad for a real-valued loss L(z)L(z) with complex parameter zz, what does it return?

βˆ‚L/βˆ‚z\partial L / \partial z

βˆ‚L/βˆ‚zβˆ—\partial L / \partial z^*

βˆ‚L/βˆ‚x+jβ‹…βˆ‚L/βˆ‚y\partial L / \partial x + j \cdot \partial L / \partial y

2β‹…βˆ‚L/βˆ‚z2 \cdot \partial L / \partial z

Correction:
βˆ‚L/βˆ‚zβˆ—\partial L / \partial z^*

The conjugate Wirtinger derivative gives the steepest-descent direction for real-valued losses.

Common Mistake: Modifying .real or .imag Without Caution

Mistake:

Assigning to .real or .imag of a complex tensor modifies the original (they are views), which may break the computation graph:

z = torch.tensor([1+2j], requires_grad=True)
z.real[0] = 5.0   # RuntimeError!

Correction:

Build complex tensors from separate real and imaginary parts using torch.complex(real_part, imag_part) to keep the graph intact.

Historical Note: Wilhelm Wirtinger and Complex Differentiation

1920s

Wilhelm Wirtinger (1865-1945) introduced the calculus of βˆ‚/βˆ‚z\partial / \partial z and βˆ‚/βˆ‚zβˆ—\partial / \partial z^* in 1927. For decades it was a niche tool in several complex variables. Its revival in engineering came through adaptive filter theory (Brandwood 1983) and, more recently, through deep learning with complex-valued neural networks. PyTorch's adoption of Wirtinger calculus in 2020 (v1.7) made complex autograd practical.

Wirtinger Derivative

A pair of differential operators βˆ‚/βˆ‚z\partial/\partial z and βˆ‚/βˆ‚zβˆ—\partial/\partial z^* that decompose the gradient of a function of complex variables into holomorphic and anti-holomorphic parts.

Hermitian Transpose

The conjugate transpose AH\mathbf{A}^H of a matrix, computed as A.conj().T or A.mH in PyTorch.

Related: Wirtinger Derivative

Key Takeaway

PyTorch supports complex tensors natively with complex64 and complex128. Autograd uses Wirtinger calculus, returning the conjugate Wirtinger derivative βˆ‚L/βˆ‚zβˆ—\partial L / \partial z^* which is the steepest-descent direction for real-valued losses. FFT, Hermitian transpose (.mH), and all standard operations work seamlessly with complex tensors.

Complex Tensor Operations

python
Working with complex-valued tensors: creation, Wirtinger derivatives, FFT, Hermitian transpose, and signal processing examples. 
# Code from: ch12/python/complex_tensors.py
# Load from backend supplements endpoint
Automatic Differentiation (autograd)PyTorch Linear Algebra