Looking Inside Large Language Models

DS-GA 3001 - Fall 2026

Claudio Silva

NYU Center for Data Science

2026-11-24

Today

  1. Why look inside? From explaining an output to understanding a mechanism
  2. Superposition: why single neurons are hard to read
  3. From neurons to features: dictionary learning and the replacement model
  4. Attribution graphs: drawing one prompt’s computation as a graph
  5. Case studies: reasoning, planning, hallucination, chain of thought
  6. Attention: the part those graphs leave out, and filter heads
  7. Limits: what these pictures can and cannot claim

This is the last regular lecture. It replaces the session on fairness.

Why Look Inside?

Where we left off on November 3

What we had

  • Embeddings: words as points, neighbors as meaning
  • Attention maps: which tokens a head looks at
  • LIME: which input words move the output

What the lab found

  • A classifier can be right for the wrong reason
  • An attention map shows where a head looks. It does not show what was computed with what it found.

Three levels of explanation

  1. Behavior: what does the model output? (evaluation)
  2. Attribution: which inputs mattered for this output? (LIME, SHAP, saliency)
  3. Mechanism: what intermediate steps did the model compute? (today)

Level 3 is what would let us say how an answer was produced, and check whether the model’s own explanation is true.

A transformer in one slide

  • Every token position carries a vector, the residual stream. Each layer reads it and adds to it.
  • Attention moves information between positions.
  • MLP layers compute new information at one position. Their hidden units are the neurons.
  • At the last position, the final vector is turned into scores (logits) for the next token.

So there are two things to explain: what is represented in those vectors, and how each layer transforms it.

 logits  ->  next token
    ^
 [ layer L ]   attention + MLP
    ^
   ...
    ^
 [ layer 2 ]   attention + MLP
    ^
 [ layer 1 ]   attention + MLP
    ^
 token embeddings

Superposition

The hope: one neuron, one concept

If it held

  • Read a neuron’s top-activating examples, name it, done
  • Some neurons really are like this: curve detectors in vision models, for example

What people found in language models

  • Most neurons are polysemantic: one neuron responds to several unrelated things
  • Top activations can look clean while weaker activations mean something else

Two questions from the paper

  • Why do neurons sometimes line up with concepts and sometimes not?
  • Can a network represent more features than it has neurons?

The answer to the second is yes, and it explains the first.

Features are directions

  • Feature: a property of the input the network represents (a curve, the word being in French, a mention of Texas)
  • Linear representation hypothesis: each feature has a direction in activation space. Several active features add up:

\[\text{activation} \;=\; x_1 W_1 + x_2 W_2 + \dots\]

  • The classic evidence: king - man + woman ≈ queen in word embeddings
  • A neuron is one particular direction: a coordinate axis. Nothing forces features to lie along the axes.

With m dimensions there are only m orthogonal directions. But there are far more than m features worth representing.

How many directions fit in d dimensions?

Exactly orthogonal directions: only d. Almost orthogonal directions: enormously many, once d is large.

A toy model with known features

Data: n synthetic features

  • Each feature \(x_i\) is 0 with probability S (the sparsity), otherwise uniform on [0, 1]
  • Each has an importance \(I_i\): how much its error counts

Model: squeeze n features into m < n dimensions and get them back

\[h = Wx \qquad x' = \mathrm{ReLU}(W^{\top}h + b)\]

\[L = \sum_i I_i\,(x_i - x'_i)^2\]

How to read the result

  • Column \(W_i\) is the direction given to feature i
  • Its length says whether the feature is represented at all
  • The angles between columns say which features interfere

We know the true features, so we can see exactly what the network did with them.

Five features in two dimensions

Dense features: the model keeps two, at right angles, and drops three. Sparse features: it keeps all five, as a pentagon.

What the hidden activations look like

Each point is one input, plotted by its two hidden activations. With sparse features the directions are visible in the data itself.

The price: interference

Input: only feature 1 is on. Before the ReLU other features respond too. The ReLU deletes the negative part; a small positive leak remains.

More features, more sparsity

Twenty features, five dimensions. The diagonal is which features are represented; anything off the diagonal is interference.

What else the paper shows

  • A phase change. Each feature ends up in one of three states: not represented, in superposition, or with a dimension to itself. The switch between them is abrupt, and depends on sparsity and importance.
  • Geometry. Features arrange themselves in regular shapes: opposite pairs, triangles, pentagons, tetrahedra.
  • Polysemantic neurons appear. Put a ReLU on the hidden layer and some neurons carry one feature while others carry several, in the same model.
  • Computation in superposition. A small network can compute a function (absolute value) on more features than it has neurons.
  • The authors’ caution. These are tiny ReLU networks. They expect superposition itself to carry over to real models, and are much less sure about the geometry.

For interpretability: a network may be simulating a much larger, sparser network. Its neurons are the wrong unit to read.

From Neurons to Features

Three ways out

  1. Build models without superposition. Possible in the toy model, by penalizing activations. Likely to cost performance in a real one.
  2. Find the feature directions after the fact. Take a trained model, record activations, and solve for an overcomplete set of directions, more directions than dimensions, such that each activation is a sparse combination of them.
  3. Something in between.

Option 2 is dictionary learning. It leaves the model untouched, and it is what the field has built on since.

  • A sparse autoencoder learns a dictionary that reconstructs a layer’s activations
  • A transcoder learns one that reproduces what an MLP layer computes: it reads the layer’s input and predicts its output

Dictionary learning recovers the directions

Left: the pentagon model’s hidden activations, with no labels. Right: five directions found by sparse dictionary learning, drawn over the true ones.

The replacement model

Swap every MLP for a wide layer of sparse features (left). Then trace how features influence one another on a prompt (right).

How good is the replacement?

Size

  • For Claude 3.5 Haiku: 30 million features across all layers
  • On average about 235 active per token position (88 in the smaller 18-layer research model)

Fidelity

  • The features miss part of each MLP’s output: about 22% normalized error on Haiku, about 11.5% on the 18-layer model
  • Run on its own, the replacement gives the 18-layer model’s top next token in about 50% of cases

So the replacement is an approximation, and a rough one.

The fix, for one prompt at a time:

  • Add an error node at every layer and position holding exactly what the features missed
  • Freeze the attention patterns to what the real model did on this prompt

With both, the replacement reproduces the real model’s output on that prompt exactly. What is unexplained is on the page, as error nodes.

Attribution Graphs

From the local replacement model to a graph

Nodes: active features, input tokens, error nodes (diamonds), and the output. Edges: how much one node contributes to another on this prompt. Then prune.

A first graph: completing an acronym

  • Prompt: The National Digital Analytics Group (N
  • The model continues with DAG
  • Read from the bottom: the words Digital, Analytics, Group each activate a feature for their own first letter
  • “say D” and “say A” combine into “say DA_”
  • Each box is a supernode: several features with the same role, grouped by hand

Is the graph true? Intervene

Suppress the “Group” features in the real model. “say G” goes to 0%, and the output changes from DAG to acronyms without the G.

Reading a graph is an interface problem

What the analyst has to do

  • Label features: look at the text examples where each one fires most, and at the tokens it pushes the output toward
  • Group features with the same role into supernodes
  • Find the path that matters among many
  • Test it with interventions

How long it takes: the authors report that the main mechanism can stand out in about ten minutes, and the overall picture takes one to two hours per prompt, before validation.

What the tool gives them

  • An interactive graph: hover a node to see its feature, click to pin it, drag nodes, group them
  • Top inputs and outputs of the selected node, ranked by edge strength
  • A panel of dataset examples for each feature

“We find that our raw data of attribution graphs is not particularly useful on its own – investing in an ergonomic, interactive interface for exploring them has been essential.” (Lindsey et al., 2025)

Case Studies

Two-hop reasoning

  • Prompt: Fact: the capital of the state containing Dallas is
  • Claude 3.5 Haiku answers Austin
  • Does it look up the answer in one step, or go through Texas?
  • The graph: Dallas activates Texas features; capital activates say a capital; together they produce say Austin
  • There is also a direct Dallas-to-Austin shortcut. Both paths exist.

Testing it: remove a step

Each panel suppresses one supernode (orange) and shows the effect on the rest and on the top outputs.

Testing it: swap the state

Replace the Texas features with another place’s features. The model names that place’s capital: Sacramento, Atlanta, Victoria, Beijing.

Planning in poems

  • Prompt, a rhyming couplet: He saw a carrot and had to grab it,
  • Haiku writes: His hunger was like a starving rabbit
  • Two ways to do this: improvise word by word and find a rhyme at the end, or pick the rhyme first
  • The graph: at the line break, before the second line starts, features for rabbit and habit are already active

Changing the plan

Suppress “rabbit” at the line break and the line ends in “habit”. Inject “green” and the model writes a line that ends in “green”.

One concept, three languages

  • Prompts: The opposite of “small” is in English, French and Chinese
  • The middle of the graph is shared: small, antonym, large features that do not depend on the language
  • Language-specific features sit at the two ends: reading the input, and choosing the output word
  • Three swaps, three predictable changes: the operation, the operand, the output language

Addition, and what the model says about it

What the graph shows for 36 + 59

  • Several paths in parallel
  • A rough-magnitude path: “about 36 plus about 60 is about 90-something”
  • A ones-digit path: a lookup feature for “ends in 6, plus ends in 9, ends in 5”
  • The two are combined to give 95
  • The same lookup features fire in other contexts that need the sum, such as a citation’s year

What the model says when asked how it did it

“I added the ones (6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95.”

The explanation is the schoolbook algorithm. The graph shows a different mechanism.

The model’s account of its own reasoning is one more output. It can be checked against the mechanism, and here it does not match.

Hallucination: “I don’t know” is the default

  • Ask about a person and a can’t answer pathway is on by default
  • A name the model recognizes activates known answer features
  • Those inhibit the default (blue, T-shaped ends), and the model answers
  • Michael Jordan: known, so it answers “basketball”
  • Michael Batkin (made up): unknown, so it declines

A hallucination, on this account, is the “known” signal firing when the knowledge is not there.

Turning the circuit by hand

Top: make the model doubt Michael Jordan. Bottom: make it “recognize” Michael Batkin, and it names a sport for a person who does not exist.

A hallucination in the wild

“Name one paper written by Andrej Karpathy.” The name is familiar, so the default is suppressed, and the model names a paper he did not write.

Chain of thought: is the written reasoning the real reasoning?

Left: the written steps match the computation. Middle: a claimed calculation with nothing behind it. Right: the steps are built backward from the answer the user suggested.

Attention

What attribution graphs do not explain

The attention patterns were frozen. The graph shows that information moved from one token to another (left), and cannot show why the model looked there (right).

Filter heads

A head that attends to the list item matching a condition. Move its query vector from a “find the fruit” prompt into a “find the vehicle” prompt, and it attends to the fruit.

The condition is portable

Share of cases in which the patched model answers by the transplanted condition: about 0.8 to 0.95 across languages and list formats, and lower across different tasks.

Two strategies: lazy and eager

Put the question before the list and the filter heads stop mattering (0.02 on the previous slide). The model marks each item as it reads it.

Limits

What these pictures can and cannot claim

Stated by the authors

  • Graphs gave “satisfying insight” for about a quarter of the prompts they tried
  • Each graph is about one prompt and one output token: an existence proof
  • Error nodes: part of the computation is unexplained, and more so on long or unusual prompts
  • Attention patterns are taken as given
  • Features that are inactive, and why, do not appear
  • The replacement may compute the right output by a different mechanism than the real MLP

Added by the way the figures are made

  • Feature labels are a person’s reading of top examples
  • Supernodes are grouped by hand
  • The published cases are the ones that worked
  • Diagrams are simplified from pruned graphs, which are pruned from the full graph

What keeps it honest: interventions on the real model, including the ones that failed.

A checklist for reading an interpretability figure

  1. What is the unit? Neuron, attention head, learned feature, or a hand-made group of features?
  2. Who named it, and from what evidence? Top-activating examples only, or also its effect on the output?
  3. What is the scope? One prompt, a family of prompts, or the whole input distribution?
  4. How much is left out? Reconstruction error, pruned nodes, frozen components.
  5. Is it causal? Was something changed inside the model, and did the output change as predicted?
  6. What would have counted against it? Are failed predictions reported?
  7. Does it survive rephrasing? Remember the question placed before the list.

The same seven questions apply to a saliency map, a SHAP plot or an attention map.

What to take away

  • Neurons are the wrong unit. Networks store more features than they have dimensions, as almost-orthogonal directions, when features are sparse.
  • Features can be recovered by dictionary learning, approximately, and the approximation error should be shown.
  • An attribution graph is one prompt’s computation drawn in those features, pruned and grouped by a person.
  • Inside one production model, for specific prompts: intermediate reasoning steps, plans made in advance, a default of “I don’t know”, and explanations that do not match the mechanism.
  • Attention needs its own tools, and patching shows a reusable “filter” operation that depends on how the prompt is ordered.
  • A picture of a mechanism is a hypothesis. The intervention is the evidence.

References

Next: project presentations

  • Tuesday, Dec 1 and Tuesday, Dec 8: final project presentations
  • Dec 14: final project report due (8 pages)