Interfacing with C/C++ and Fortran

Why Interface with C/C++?

Sometimes you need to call existing C/C++ or Fortran libraries from Python — legacy codebases, vendor-optimized libraries (BLAS, LAPACK, FFTW), or custom high-performance kernels. Python provides several foreign function interfaces (FFI) with different trade-offs between ease of use, performance, and flexibility.

This section covers ctypes (stdlib, no compilation needed), cffi (C-level FFI with inline definitions), and pybind11 (modern C++ binding with automatic type conversion).

Definition:
ctypes: Standard Library FFI

ctypes is Python's built-in module for calling functions in shared libraries (.so, .dylib, .dll) without any compilation step:

import ctypes
import numpy as np

# Load the shared library
lib = ctypes.CDLL('./libvector.so')

# Declare argument and return types
lib.dot_product.argtypes = [
    ctypes.POINTER(ctypes.c_double),
    ctypes.POINTER(ctypes.c_double),
    ctypes.c_int,
]
lib.dot_product.restype = ctypes.c_double

# Call from NumPy arrays
a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 5.0, 6.0])
result = lib.dot_product(
    a.ctypes.data_as(ctypes.POINTER(ctypes.c_double)),
    b.ctypes.data_as(ctypes.POINTER(ctypes.c_double)),
    len(a),
)

ctypes requires manually specifying argument types and is error-prone (wrong types cause segfaults, not exceptions). It works with any compiled shared library without recompilation.

Definition:
cffi: C Foreign Function Interface

cffi provides a more Pythonic FFI with two modes:

ABI mode (like ctypes, no compiler needed):

from cffi import FFI
ffi = FFI()
ffi.cdef("double dot_product(double *a, double *b, int n);")
lib = ffi.dlopen('./libvector.so')

a = np.array([1.0, 2.0, 3.0])
ptr_a = ffi.cast("double *", a.ctypes.data)
result = lib.dot_product(ptr_a, ...)

API mode (compiles a C extension, faster):

ffi.set_source("_vector", '#include "vector.h"',
                libraries=["vector"])
ffi.compile()

cffi's API mode is recommended for production use. It compiles once and produces a proper Python extension module with correct type checking.

Definition:
pybind11: Modern C++ Bindings

pybind11 is a header-only C++ library that creates Python bindings with automatic type conversion, NumPy support, and exception handling:

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
namespace py = pybind11;

double dot_product(py::array_t<double> a, py::array_t<double> b) {
    auto ra = a.unchecked<1>();
    auto rb = b.unchecked<1>();
    double sum = 0.0;
    for (ssize_t i = 0; i < ra.shape(0); i++)
        sum += ra(i) * rb(i);
    return sum;
}

PYBIND11_MODULE(vector_ops, m) {
    m.def("dot_product", &dot_product, "Dot product of two arrays");
}

pybind11 automatically converts between NumPy arrays and C++ types, handles reference counting, and translates C++ exceptions to Python.

pybind11 supports classes, inheritance, virtual functions, STL containers, and even buffer protocol. It is the standard choice for new C++/Python bindings.

Definition:
NumPy Array Memory Layout for C Interop

NumPy arrays store data in contiguous memory blocks. For C interop, ensure arrays are:

C-contiguous (order='C'): row-major, as expected by C code
Correct dtype: match the C type (float64 for double)
Not a view: views may not be contiguous

x = np.ascontiguousarray(x, dtype=np.float64)
ptr = x.ctypes.data_as(ctypes.POINTER(ctypes.c_double))

The .ctypes.data attribute gives the raw memory address. The .ctypes.data_as() method casts to the desired pointer type.

Definition:
Calling Fortran with f2py

NumPy's f2py tool generates Python wrappers for Fortran subroutines:

! vector_ops.f90
subroutine dot_prod(a, b, n, result)
    integer, intent(in) :: n
    double precision, intent(in) :: a(n), b(n)
    double precision, intent(out) :: result
    integer :: i
    result = 0.0d0
    do i = 1, n
        result = result + a(i) * b(i)
    end do
end subroutine

python -m numpy.f2py -c vector_ops.f90 -m vector_ops

import vector_ops
result = vector_ops.dot_prod(a, b)  # automatic array handling

Fortran uses column-major (Fortran-contiguous) ordering. f2py handles the transposition automatically, but for performance, pass order='F' arrays to avoid copies.

Theorem: Foreign Function Call Overhead

The per-call overhead of a foreign function interface is:

$T_{\text{call}} = T_{\text{marshal}} + T_{\text{execute}} + T_{\text{unmarshal}}$

For ctypes and cffi, marshaling involves type conversion and pointer wrapping (~1-5 $\mu$ s). For pybind11, marshaling includes reference counting and exception translation (~0.5-2 $\mu$ s).

The FFI is worthwhile when $T_{\text{execute}} \gg T_{\text{call}}$ , i.e., the C function does enough work to amortize the call overhead.

Calling a C function from Python is not free. For a function that takes 100ns to execute, a 5 $\mu$ s call overhead means you lose 98% of the performance to marshaling. Pass large arrays, not individual scalars.

Example: Complete ctypes Workflow

Write a C function for element-wise array operations, compile it as a shared library, and call it from Python via ctypes.

Solution

C source (array_ops.c)

#include <math.h>

void apply_sigmoid(double *x, double *out, int n) {
    for (int i = 0; i < n; i++) {
        out[i] = 1.0 / (1.0 + exp(-x[i]));
    }
}

Compile

gcc -shared -fPIC -O3 -o libarray_ops.so array_ops.c -lm

Python caller

import ctypes, numpy as np

lib = ctypes.CDLL('./libarray_ops.so')
lib.apply_sigmoid.argtypes = [
    ctypes.POINTER(ctypes.c_double),
    ctypes.POINTER(ctypes.c_double),
    ctypes.c_int,
]
lib.apply_sigmoid.restype = None

x = np.random.randn(1000000)
out = np.empty_like(x)
lib.apply_sigmoid(
    x.ctypes.data_as(ctypes.POINTER(ctypes.c_double)),
    out.ctypes.data_as(ctypes.POINTER(ctypes.c_double)),
    len(x),
)

Example: pybind11 NumPy Extension Module

Create a Python extension module using pybind11 that provides a fast element-wise operation on NumPy arrays.

Solution

C++ source (ops.cpp)

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <cmath>
namespace py = pybind11;

py::array_t<double> fast_sigmoid(py::array_t<double> input) {
    auto buf = input.request();
    auto result = py::array_t<double>(buf.size);
    auto *ptr_in = static_cast<double *>(buf.ptr);
    auto *ptr_out = static_cast<double *>(result.request().ptr);
    for (ssize_t i = 0; i < buf.size; i++)
        ptr_out[i] = 1.0 / (1.0 + std::exp(-ptr_in[i]));
    result.resize({buf.size});
    return result;
}

PYBIND11_MODULE(fast_ops, m) {
    m.def("sigmoid", &fast_sigmoid, "Fast sigmoid");
}

Build and use

pip install pybind11
c++ -O3 -shared -std=c++17 -fPIC \
    $(python3 -m pybind11 --includes) \
    ops.cpp -o fast_ops$(python3-config --extension-suffix)

import fast_ops
import numpy as np
x = np.random.randn(1_000_000)
y = fast_ops.sigmoid(x)  # 3-5x faster than scipy.special.expit

Python FFI Methods Compared

Feature	ctypes	cffi	pybind11	f2py
Language	C	C	C++	Fortran
Requires Compilation	No (ABI)	No (ABI) / Yes (API)	Yes	Yes
Type Safety	Manual (error-prone)	Better (cdef)	Automatic	Automatic
NumPy Support	Via .ctypes	Via ffi.cast	Native	Native
Error Handling	Segfault risk	Better than ctypes	C++ exceptions	Fortran errors
Complexity	Low	Low-Medium	Medium	Low
Performance	Good	Good-Excellent	Excellent	Excellent
Best For	Quick prototyping	C library wrapping	New C++ extensions	Legacy Fortran

Quick Check

What is the primary risk of using ctypes with incorrect type declarations?

A Python TypeError exception

Slower execution

A segmentation fault (crash) with no Python traceback

Automatic type conversion to the correct type

Correction:

A segmentation fault (crash) with no Python traceback

Correct. ctypes performs no runtime type checking. Passing wrong types leads to memory corruption and segfaults.

Common Mistake: Non-Contiguous Arrays in C Calls

Mistake:

Passing a NumPy array slice (which may be non-contiguous) to a C function that expects contiguous memory:

x = np.random.randn(100, 100)
lib.process(x[::2, :].ctypes.data_as(...))  # non-contiguous!

Correction:

Always ensure contiguity before passing to C:

x_slice = np.ascontiguousarray(x[::2, :])
lib.process(x_slice.ctypes.data_as(...))

Historical Note: SWIG and the Evolution of Python Bindings

1996-2015

SWIG (Simplified Wrapper and Interface Generator, 1996) was the first widely-used tool for generating Python bindings from C/C++ headers. It could target multiple languages but produced verbose, hard-to-debug code. Boost.Python (2002) improved C++ support but required heavy template metaprogramming. pybind11 (2015, by Wenzel Jakob) simplified Boost.Python's approach into a lightweight, header-only library that became the de facto standard.

Key Takeaway

Choose your FFI by use case: ctypes for quick one-off calls to existing shared libraries, cffi for wrapping C APIs with better safety, pybind11 for new C++ extensions with full NumPy integration, and f2py for Fortran codes. In all cases, batch work into large array operations to amortize the per-call overhead.

FFI (Foreign Function Interface)

A mechanism that allows code written in one language (Python) to call functions written in another language (C, C++, Fortran).

Related: Shared Library

Shared Library

A compiled binary (.so on Linux, .dylib on macOS, .dll on Windows) containing machine code that can be loaded and called at runtime by any program, including Python via ctypes or cffi.

ctypes and cffi Demos

python

Complete ctypes and cffi examples with C source, compilation instructions, and Python caller code.

# Code from: ch14/python/c_interface.py
# Load from backend supplements endpoint

pybind11 Extension Module