Mechanistic Interpretability: Seeing What Neurons Think

For decades, neural networks were called black boxes — we knew they worked but not why.

Mechanistic interpretability (Mech Interp) is changing that. Researchers are extracting specific concepts, circuits, and reasoning steps from inside the network.

2024 was a breakthrough year — Anthropic could identify the neurons for the Golden Gate Bridge in Claude 3 Sonnet and turn them up to make Claude obsessed with that bridge.

What Mech Interp Tries to Do

Three increasingly ambitious goals:

Goal 1: Find features

“Where in the network is the concept of ‘Eiffel Tower’ represented?”

Goal 2: Find circuits

“Which neurons + connections together compute ‘sentence sentiment’?”

Goal 3: Reverse-engineer the model

“Walk through this LLM’s reasoning on this prompt step-by-step.”

We’re at Goal 1 mostly solved, Goal 2 partial, Goal 3 nascent.

Key Techniques

1. Probing

Train small classifiers on hidden activations:

Take internal vector at layer L, position P
→ Predict: "is this about geography?"
If classifier accuracy > 80%, that representation has geographic info

Tells you what the model knows, but not how.

2. Sparse Autoencoders (SAEs)

The big 2023-2024 breakthrough. Train an autoencoder on hidden activations:

hidden_activation → encoder → sparse_features → decoder → reconstruction

Force the middle layer to be sparse (mostly zero, with a few active features).

Result: thousands to millions of monosemantic features — each feature corresponds to one specific concept.

3. Causal Interventions

Run model on prompt
Inspect feature F's activation
Manually set F to higher / lower value
Re-run remaining layers
Observe output change

If changing F changes output predictably → F causes that behavior. Beyond correlation.

4. Logit Lens

For each intermediate layer, decode “what next token would the model predict if it stopped here?” Lets you see the evolution of the model’s guess through layers.

A Real Finding: The “Golden Gate Bridge” Feature

In Anthropic’s 2024 work on Claude 3 Sonnet, an SAE identified a feature that activated for:

• The Golden Gate Bridge in San Francisco
• Pictures of it
• Mentions in different languages
• Even abstract references (“the iconic red bridge in California”)

When researchers amplified this feature 10×:

• Claude started seeing the Golden Gate Bridge everywhere
• “What’s your favorite landmark?” → “The Golden Gate Bridge”
• “Tell me about yourself” → “I am the Golden Gate Bridge”
• “Solve this math problem” → “The Golden Gate Bridge spans…”

This was a stunning demonstration: specific concepts have specific neural representations that can be edited.

Why This Matters

1. Detection of deception

If we can find the “lying” feature, we can detect when AI is being deceptive.

2. Safety verification

Instead of behavioral testing (hard to be exhaustive), check directly for unsafe representations:

• “Does this model have neurons that represent harmful instructions?”
• “Are they being suppressed correctly?“

3. Surgical editing

Remove a concept without retraining:

• Bias in profession-gender associations
• Dangerous knowledge (e.g., synthesis of harmful materials)

4. Trust and audit

For regulated industries, “show me how the model decided” becomes possible.

Limits

1. Doesn’t scale (yet)

SAEs on a 70B model find millions of features. Manually labeling them all is intractable. Auto-labeling helps but isn’t perfect.

2. Causal stories are hard

Knowing a feature exists is easy. Knowing the model uses it for X requires careful intervention.

3. Circuits are messy

The model doesn’t have “the addition circuit” cleanly factored. Many tasks are computed by overlapping, polysemantic mechanisms.

4. Deception is hard to detect

If the model is genuinely deceptive (trained on data where lying was rewarded), its “honesty” features may be exactly the same as “lying with confidence” features.

Research Landscape

Top labs

• Anthropic Interpretability — Chris Olah’s team, original SAE work
• DeepMind Causal Reasoning Team
• OpenAI Superalignment (defunct, but produced key papers before)
• METR / Apollo Research / Conjecture — independent

Recent landmark papers

• Toy Models of Superposition (Anthropic 2022) — explained polysemanticity
• Scaling Monosemanticity (Anthropic 2024) — 1B+ features in Claude 3 Sonnet
• Towards Monosemanticity (Anthropic 2023) — early SAE work
• A Mathematical Framework for Transformer Circuits (Anthropic 2021)

Tools

• TransformerLens — main library for circuit analysis (Neel Nanda)
• SAELens — SAE training / analysis
• NeuroNpedia — visualize features in real models

How to Get Started

If you want to do mech interp:

1. Background

• Solid linear algebra
• Understanding of Transformer (L3-05, L3-08)
• Some PyTorch

2. Tutorials

• Neel Nanda’s “200 concrete problems in mech interp”
• TransformerLens tutorials
• ARENA curriculum (intensive bootcamp)

3. First projects

• Probe GPT-2 small to find one feature
• Reproduce a published circuit (e.g., “indirect object identification”)
• Train an SAE on a small model

4. Career paths

• Research at Anthropic / DeepMind / OpenAI
• Independent (with funding from OpenPhilanthropy)
• Academic PhD

The field is young and small — ~500 active researchers worldwide. Talented newcomers can have outsized impact.

A Profound Implication

If mech interp succeeds at scale, it changes what AI is:

From: “an AI that we hope is aligned” To: “an AI we can verify is aligned, neuron by neuron”

This is what Anthropic, Apollo, METR are working toward. Whether we get there before scaling outpaces interp is one of the major open questions.

Next recommended: L6-05 Bias and Fairness or L6-07 AI Safety Research Entry.