Legacy Concept Lab
Probing, Linear Classifier Probes & Activation Analysis
Probing is one of the main tools to understand what GPT-like models know and where that knowledge lives
#12ProbingRepresentations
key equation
Phase 5: Representation & interpretabilityConcept 12 of 100
Why It Matters for Modern Models
- Probing is one of the main tools to understand what GPT-like models know and where that knowledge lives
- Used heavily for safety (probing for dangerous capabilities), robustness, and fairness analyses
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Clear mental model of what probes measure (information content vs ease of extraction)
- Visual, layer-by-layer maps of information flow in large LMs
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
Given layer representation , train a frozen probe:
(or a softmax over ) on a supervised task (POS tags, parse trees, etc.). The accuracy estimates how linearly separable that information is at layer .
BERT layers roughly follow the classical NLP pipeline (POS → syntax → semantics → coreference).