Prerequisites & Notation

Before You Begin

This chapter assumes familiarity with multivariable calculus (gradients, Hessians), linear algebra (eigenvalues, positive definiteness), and basic NumPy/SciPy operations. If any of these feel unfamiliar, review the linked material first.

  • NumPy array creation, slicing, and broadcasting (Chapter 5)(Review ch05)

    Self-check: Can you compute a matrix-vector product with @ and reshape arrays?

  • Linear algebra: solve, eigenvalues, SVD (Chapter 6)(Review ch06)

    Self-check: Can you solve Ax=b\mathbf{A}\mathbf{x} = \mathbf{b} and compute eigenvalues in NumPy?

  • Multivariable calculus: gradients and Hessians

    Self-check: Can you compute ∇f\nabla f and ∇2f\nabla^2 f for f(x,y)=x2+xy+y2f(x,y) = x^2 + xy + y^2?

  • Convexity basics: convex sets and convex functions

    Self-check: Do you know that f(λx+(1−λ)y)≤λf(x)+(1−λ)f(y)f(\lambda x + (1-\lambda)y) \le \lambda f(x) + (1-\lambda)f(y)?

Notation for This Chapter

Symbols and conventions used throughout this chapter. We write optimization problems in the standard minimization form.

SymbolMeaningIntroduced
f(mathbfx)f(\\mathbf{x})Objective function to be minimizeds01
nablaf\\nabla f, mathbfg\\mathbf{g}Gradient (vector of partial derivatives)s01
nabla2f\\nabla^2 f, mathbfH\\mathbf{H}Hessian matrix (matrix of second partial derivatives)s01
\\mathbf{x}^\\starOptimal solutions01
gi(mathbfx)le0g_i(\\mathbf{x}) \\le 0Inequality constraintss02
hj(mathbfx)=0h_j(\\mathbf{x}) = 0Equality constraintss02
mathrmproxlambdaf(mathbfv)\\mathrm{prox}_{\\lambda f}(\\mathbf{v})Proximal operator of ff with step size λ\lambdas04
\\mathcal{S}_\\lambda(x)Soft-thresholding operator with threshold λ\lambdas04
∣mathbfx∣1\\|\\mathbf{x}\\|_1ℓ1\ell_1 norm: ∑i∣xi∣\sum_i |x_i| (promotes sparsity)s03
succeq0\\succeq 0Positive semidefinite (for matrices)s03
← Ch 7Unconstrained Optimization