
DS-GA 3001 - Fall 2026
NYU Center for Data Science
2026-11-24
This is the last regular lecture. It replaces the session on fairness.
What we had
What the lab found
Three levels of explanation
Level 3 is what would let us say how an answer was produced, and check whether the model’s own explanation is true.
So there are two things to explain: what is represented in those vectors, and how each layer transforms it.
logits -> next token
^
[ layer L ] attention + MLP
^
...
^
[ layer 2 ] attention + MLP
^
[ layer 1 ] attention + MLP
^
token embeddings
If it held
What people found in language models
Two questions from the paper
The answer to the second is yes, and it explains the first.
Elhage, N., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread.
\[\text{activation} \;=\; x_1 W_1 + x_2 W_2 + \dots\]
king - man + woman ≈ queen in word embeddingsWith m dimensions there are only m orthogonal directions. But there are far more than m features worth representing.

Exactly orthogonal directions: only d. Almost orthogonal directions: enormously many, once d is large.
Data: n synthetic features
Model: squeeze n features into m < n dimensions and get them back
\[h = Wx \qquad x' = \mathrm{ReLU}(W^{\top}h + b)\]
\[L = \sum_i I_i\,(x_i - x'_i)^2\]
How to read the result
We know the true features, so we can see exactly what the network did with them.
Elhage, N., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread.

Dense features: the model keeps two, at right angles, and drops three. Sparse features: it keeps all five, as a pentagon.

Each point is one input, plotted by its two hidden activations. With sparse features the directions are visible in the data itself.

Input: only feature 1 is on. Before the ReLU other features respond too. The ReLU deletes the negative part; a small positive leak remains.

Twenty features, five dimensions. The diagonal is which features are represented; anything off the diagonal is interference.
For interpretability: a network may be simulating a much larger, sparser network. Its neurons are the wrong unit to read.
Elhage, N., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread.
Option 2 is dictionary learning. It leaves the model untouched, and it is what the field has built on since.

Left: the pentagon model’s hidden activations, with no labels. Right: five directions found by sparse dictionary learning, drawn over the true ones.
Swap every MLP for a wide layer of sparse features (left). Then trace how features influence one another on a prompt (right).
Ameisen, E., et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread.
Size
Fidelity
So the replacement is an approximation, and a rough one.
The fix, for one prompt at a time:
With both, the replacement reproduces the real model’s output on that prompt exactly. What is unexplained is on the page, as error nodes.
Ameisen, E., et al. (2025). Circuit Tracing. Transformer Circuits Thread.
Nodes: active features, input tokens, error nodes (diamonds), and the output. Edges: how much one node contributes to another on this prompt. Then prune.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.

Ameisen, E., et al. (2025). Circuit Tracing. Transformer Circuits Thread.
Suppress the “Group” features in the real model. “say G” goes to 0%, and the output changes from DAG to acronyms without the G.
Ameisen, E., et al. (2025). Circuit Tracing. Transformer Circuits Thread.
What the analyst has to do
How long it takes: the authors report that the main mechanism can stand out in about ten minutes, and the overall picture takes one to two hours per prompt, before validation.
What the tool gives them
“We find that our raw data of attribution graphs is not particularly useful on its own – investing in an ergonomic, interactive interface for exploring them has been essential.” (Lindsey et al., 2025)

Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
Each panel suppresses one supernode (orange) and shows the effect on the rest and on the top outputs.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
Replace the Texas features with another place’s features. The model names that place’s capital: Sacramento, Atlanta, Victoria, Beijing.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.

Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
Suppress “rabbit” at the line break and the line ends in “habit”. Inject “green” and the model writes a line that ends in “green”.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.

Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
What the graph shows for 36 + 59
What the model says when asked how it did it
“I added the ones (6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95.”
The explanation is the schoolbook algorithm. The graph shows a different mechanism.
The model’s account of its own reasoning is one more output. It can be checked against the mechanism, and here it does not match.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.

A hallucination, on this account, is the “known” signal firing when the knowledge is not there.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
Top: make the model doubt Michael Jordan. Bottom: make it “recognize” Michael Batkin, and it names a sport for a person who does not exist.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
“Name one paper written by Andrej Karpathy.” The name is familiar, so the default is suppressed, and the model names a paper he did not write.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
Left: the written steps match the computation. Middle: a claimed calculation with nothing behind it. Right: the steps are built backward from the answer the user suggested.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
The attention patterns were frozen. The graph shows that information moved from one token to another (left), and cannot show why the model looked there (right).
Ameisen, E., et al. (2025). Circuit Tracing. Transformer Circuits Thread.
A head that attends to the list item matching a condition. Move its query vector from a “find the fruit” prompt into a “find the vehicle” prompt, and it attends to the fruit.
Sen Sharma, A., Rogers, G., Shapira, N., & Bau, D. (2025). LLMs Process Lists With General Filter Heads. arXiv:2510.26784.
Share of cases in which the patched model answers by the transplanted condition: about 0.8 to 0.95 across languages and list formats, and lower across different tasks.
Sen Sharma, A., Rogers, G., Shapira, N., & Bau, D. (2025). LLMs Process Lists With General Filter Heads. arXiv:2510.26784.
Put the question before the list and the filter heads stop mattering (0.02 on the previous slide). The model marks each item as it reads it.
Sen Sharma, A., Rogers, G., Shapira, N., & Bau, D. (2025). LLMs Process Lists With General Filter Heads. arXiv:2510.26784.
Stated by the authors
Added by the way the figures are made
What keeps it honest: interventions on the real model, including the ones that failed.
Lindsey, J., et al. (2025). On the Biology of a Large Language Model; Ameisen, E., et al. (2025). Circuit Tracing.
The same seven questions apply to a saliency map, a SHAP plot or an attention map.