Kangbo Li
06/28/24
Department of Computer Science, Cornell University
Joint work with Prof. Anil Damle
Moses Schönfinkel
Haskell Curry
Schönfinkel, Moses (1924)
Über die Bausteine der mathematischen Logik
$\mathbf{C} \triangleq f \mapsto (x \mapsto (y \mapsto (f(y))(x)))$
$\mathbf{B} \triangleq f \mapsto (g \mapsto (x \mapsto f(g(x))))$
Feynman's Path Integral
$\int_{C(\mathbb{R})} \mathcal{F}(x) \mathrm{d} x$
Path Dirac Delta
$\delta(x, y, 1)$
$$ \int_{C(\mathbb{R})} \delta(x, y, \mathcal{F}(x)) \mathrm{d} x = (x \mapsto \mathcal{F}(x))(y) = \mathcal{F}(y). $$
$\mathbf{B}$ rule
$$ \begin{aligned} \mathcal{P}&\left( x \mapsto \mathbf{B}(f)(g)(x) \right) = (x, k) \mapsto \\ & \mathcal{P}\left( x \mapsto g(x)\right) (x, \mathcal{P}\left( f \right)(g(x), k)) +\\ & {\color{cyan} \mathcal{P}\left( x \mapsto f\right) (x, i \mapsto \delta(g(x), i, k))} \end{aligned} $$
$\mathbf{C}$ rule
$$ \begin{aligned} \mathcal{P}& \left(\mathbf{C}(g)\right) =\\ &(x, k) \mapsto {\color{cyan} \int \mathcal{P}\left( g(b) \right)(x, k(b))) \mathrm{d} b } \end{aligned} $$
$\mathbf{B}$ & $\mathbf{C}$
Pullback
$\longrightarrow$
$\longleftarrow$
Contract
Path Integral
Dirac Delta
Demote backpropagtion $\Rightarrow$ Separate differentiation from numerics
More general, efficient, modular, and transparent.
1.
A Model of Differentiation
Combinatory logic $\leftrightarrow$ path integrals
2.
Implications on AI and Scientific Computing
Backprop is no longer special, and differentiation is simple
3.
Prototype Demo: CombDiff.jl
Differential tensor calculus & backprop an MLP analytically
Preprint: Automating Variational Differentiation
https://arxiv.org/abs/2406.16154
Acknowledgement