Prerequisites & Notation

Before You Begin

This chapter assumes familiarity with NumPy array programming, basic linear algebra operations, and the notion of computational graphs. We also rely on CuPy concepts from Chapter 10.

  • NumPy array creation, broadcasting, and vectorization (Chapter 5)(Review ch05)

    Self-check: Can you reshape, broadcast, and slice NumPy arrays fluently?

  • Linear algebra: eigenvalues, SVD, solving linear systems (Chapter 6)(Review ch06)

    Self-check: Can you compute an SVD and explain what the singular values represent?

  • GPU programming concepts with CuPy (Chapter 10)(Review ch10)

    Self-check: Do you understand host vs. device memory and why data transfers are expensive?

  • Complex numbers and Hermitian matrices

    Self-check: Do you know that (zβˆ—)β€²β‰ (zβ€²)βˆ—(z^*)' \neq (z')^* in general for complex functions?

  • Calculus: chain rule, gradients, Jacobians

    Self-check: Can you differentiate f(g(x))f(g(x)) using the chain rule?

Notation for This Chapter

Symbols and conventions introduced in this chapter. We follow PyTorch's convention of calling multi-dimensional arrays tensors.

SymbolMeaningIntroduced
T\mathbf{T}A PyTorch tensor (uppercase boldface)s01
βˆ‡xf\nabla_{\mathbf{x}} fGradient of scalar ff with respect to tensor x\mathbf{x}s02
fracpartialLpartialmathbfx\\frac{\\partial L}{\\partial \\mathbf{x}}Partial derivative of loss LL w.r.t. parameter tensor x\mathbf{x}s02
fracpartialfpartialz,fracpartialfpartialzβˆ—\\frac{\\partial f}{\\partial z},\\ \\frac{\\partial f}{\\partial z^*}Wirtinger derivatives for complex-valued functionss03
mathbfUboldsymbolSigmamathbfVH\\mathbf{U}\\boldsymbol{\\Sigma}\\mathbf{V}^HSingular value decompositions04
texttttorch.Tensor\\texttt{torch.Tensor}The fundamental data structure in PyTorchs01
texttt.grad\\texttt{.grad}Attribute storing accumulated gradients on a leaf tensors02
← Ch 11Tensors vs. NumPy Arrays