Graph Neural Networks

A GNN operates on graph-structured data $G = (V, E)$. The message-passing update at layer $l$:

$$\mathbf{h}_v^{(l+1)} = \text{UPDATE}\left(\mathbf{h}_v^{(l)}, \text{AGG}\left(\{\mathbf{h}_u^{(l)} : u \in \mathcal{N}(v)\}\right)\right)$$

where $\mathcal{N}(v)$ are the neighbors of node $v$.

A Neural ODE defines the hidden state dynamics as:

$$\frac{d\mathbf{h}}{dt} = f_\theta(\mathbf{h}(t), t)$$

solved using adaptive ODE solvers (Dormand-Prince, RK45). The backward pass uses the adjoint method: $O(1)$ memory.

Contrastive learning learns representations by pulling positive pairs together and pushing negative pairs apart:

$$\mathcal{L} = -\log \frac{\exp(\text{sim}(\mathbf{z}_i, \mathbf{z}_j)/\tau)}{\sum_{k \neq i} \exp(\text{sim}(\mathbf{z}_i, \mathbf{z}_k)/\tau)}$$