AI
"Tracing a reasoning pathway" sounds like reading a debugger's stack trace, but it's closer to a controlled experiment: you can't just stare at a model's internal activations and know which ones caused a particular output — you have to intervene and measure the effect. Activation patching does exactly that, and TransformerLens is the standard open-source tooling for running these experiments on GPT-style models without writing custom instrumentation for each architecture.
Running an activation-patching experiment
1. Design a minimal-pair experiment. Construct two inputs that differ in one meaningful way (e.g., "John gave the book to Mary" vs. "Mary gave the book to John") and where the model's correct output should differ correspondingly.
2. Cache activations from both runs. TransformerLens's hook system lets you capture the activation at every layer and position during each forward pass, keyed for later comparison.
3. Patch one run's activation into the other and check the output. If replacing a specific layer/position's activation flips the output toward the other input's answer, that activation is causally implicated in the behavior — not merely correlated with it.
# pip install transformer-lens
import transformer_lens as tl
model = tl.HookedTransformer.from_pretrained("gpt2-small")
clean_prompt = "John gave the book to" # expect "Mary"
corrupted_prompt = "Mary gave the book to" # expect "John"
clean_logits, clean_cache = model.run_with_cache(clean_prompt)
corrupted_logits, corrupted_cache = model.run_with_cache(corrupted_prompt)
# Patch layer 9's residual stream from the clean run into the corrupted run,
# then check whether the output shifts back toward the clean answer
def patch_hook(activation, hook):
return clean_cache[hook.name]
patched_logits = model.run_with_hooks(
corrupted_prompt,
fwd_hooks=[("blocks.9.hook_resid_post", patch_hook)]
)| Method | Establishes |
|---|---|
| Reading raw activation values | What the model computed — correlation, not causation |
| Activation patching | Whether that specific computation causally drives the output — an actual causal claim |
This is genuinely rigorous work, not a quick diagnostic — published circuit-tracing results (like the IOI circuit) took careful, iterative patching experiments across many layers and positions to nail down. Expect the same for any serious attempt to trace a specific reasoning pathway in your own experiments; a single patching result is a data point, not a complete explanation.
Practical Challenge
Install TransformerLens, load GPT-2 small, and reproduce the minimal-pair patching example above — try patching a few different layers and see at which layer(s) the output actually flips.
Concept Check
Sources & Further Reading
- TransformerLens (GitHub) — the open-source library for mechanistic interpretability used throughout this article.
- TransformerLens Documentation: Getting Started in Mechanistic Interpretability — tutorials covering activation caching and patching.
- Anthropic: In-context Learning and Induction Heads — an example of the kind of circuit-tracing findings this methodology has produced.