Legacy Concept Lab

Representation Learning & Embedding Geometry

Word & token embeddings in LMs, vision embeddings in CLIP-like models, multimodal embeddings in Gemini and GPT-4V

Concept 10 of 100RepresentationsPhase 5

Open New Notebook All Foundations

#10EmbeddingsRepresentations

key equation

\mathcal L = - \mathbb E \left[ \log \frac{\exp(\mathrm{sim}(f(x),g(y))/\tau)}{\sum_{y'} \exp(\mathrm{sim}(f(x), g(y'))/\tau)} \right]

Phase 5: Representation & interpretabilityConcept 10 of 100

Why It Matters for Modern Models

Word & token embeddings in LMs, vision embeddings in CLIP-like models, multimodal embeddings in Gemini and GPT-4V
Latent spaces of Stable Diffusion designed so distances correspond to semantic similarity

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Geometric explanation of anisotropy (representations bunch along a few directions) and how normalization/whitening alter behavior
Visuals showing how representations evolve across layers (local to global features)

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\mathcal L = - \mathbb E \left[ \log \frac{\exp(\mathrm{sim}(f(x),g(y))/\tau)}{\sum_{y'} \exp(\mathrm{sim}(f(x), g(y'))/\tau)} \right]

Learn a mapping $f_\theta: \mathcal X \to \mathbb R^d$ such that inner products or distances reflect meaningful relations.

Contrastive objective (InfoNCE-style):

\mathcal L = - \mathbb E \left[ \log \frac{\exp(\mathrm{sim}(f(x),g(y))/\tau)}{\sum_{y'} \exp(\mathrm{sim}(f(x), g(y'))/\tau)} \right]

This pushes "positive" pairs together, "negatives" apart; at optimum, it maximizes a lower bound on mutual information between views.

Canonical Papers

Representation Learning: A Review and New Perspectives

Bengio et al.2013IEEE TPAMI

Connections

Prerequisites

Enables

⊕Superposition ⚲Probing ⊛Circuits 🔍Sparse Autoencoders 🎚️Activation Steering 🖼️Multimodal VLP 🔤Tokens 🔍RAG ⚔️Adversarial 🔬Logit Lens 🔗Contrastive 🔄SSL 🕳️TDA

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

🔄 Same Technique

🎲Temperature scaling→Decoding 🖼️InfoNCE similarity matrix→Multimodal VLP ⊗Dot-product logits→Attention 🔍Dictionary learning→Sparse Autoencoders 🎚️Linear directions→Activation Steering 👍Implicit value directions→KTO

↔️ Mathematical Dual

∀Bottleneck viewpoint→Theory

≈ Analogy

⊛Algorithmic feature→Circuits