Continuous Function

Legacy Concept Lab

Neural Tangent Kernel & Infinite-Width Limits

NTK provides a mathematically clean limit where we can predict learning dynamics and generalization

Concept 6 of 100TheoryPhase 3
Open New NotebookAll Foundations
#6NTKTheory
key equation
Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′)\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x')
Phase 3: Optimization & generalizationConcept 6 of 100
Migrated:view the updated version in /domains →This /foundations page is legacy during migration.

Why It Matters for Modern Models

  • NTK provides a mathematically clean limit where we can predict learning dynamics and generalization
  • Many mechanistic-interpretability arguments assume behavior "somewhere between" kernel-like and feature-learning regimes

What Tutorials Skip

What is still poorly explained in textbooks and papers:

  • Most expositions are algebraic; missing is a geometric animation showing how trajectories in function space under NTK differ from genuine feature learning

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation
Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′)\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x')

Define network fθ(x)f_\theta(x) with parameters θ\theta. The NTK is:

Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′)\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x')

In the infinite-width limit, this kernel becomes deterministic and remains constant during training. Training becomes:

∂tft(x)=−∑iΘ(x,xi) ∂ℓ(ft(xi),yi)∂f\partial_t f_t(x) = - \sum_{i} \Theta(x,x_i)\, \frac{\partial \ell(f_t(x_i), y_i)}{\partial f}

a linear ODE in function space, just like kernel regression.

Canonical Papers

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Jacot et al.2018NeurIPS
Read paper →

Connections

Prerequisites

∪Double Descent

Enables

∀Theory🎲Init🎭RKHS

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

↔️ Mathematical Dual
↗Width ↔ Depth→Scaling
🔄 Same Technique
🔬First-order Taylor→Circuit Discovery
≈ Analogy
⚲Linear lens→Probing
⚠️ Breaks When
⊛Feature learning dominates→Circuits∪Rich regime dominates→Double Descent