On the Biology of a Large Language Model

chapter 1

The question in one picture

The paper gives Claude 3.5 Haiku the start of a sentence: “Fact: the capital of the state containing Dallas is”. The model writes “Austin”. Dallas

A person gets there in two steps: Dallas is in Texas, and the capital of Texas is Austin. The model may instead have remembered this sentence from its training text. Dallas The answer alone cannot tell the two routes apart. So the paper looks inside the model while it answers, in ten kinds of tasks. Intro

The answer, in short

In this example, the model uses both routes at once. Inside, it has a “Texas” step. It also has a direct link from Dallas to Austin. Dallas

Across the paper, the model often takes steps “in its head” that it never writes down. For example, it picks a rhyme before it writes the line. Sometimes the steps it writes down are not the steps it used. Intro

chapter 2

What you need first

1. A model reads tokens and writes one at a time

A language model reads text as tokens: words or pieces of words. It writes its answer one token at a time. Method

Inside, it has two kinds of layers. Attention layers move information from one token to another. MLP layers work on each token on its own, using many small units called neurons. Method

2. Neurons do many jobs

You might hope each neuron stands for one idea. Usually it does not. Most neurons are polysemantic: one neuron does several jobs that seem unrelated. Method

The paper uses features instead. A feature is a unit that tends to turn on for one idea, such as “Texas” or “capital”. The researchers label each feature by reading the text where it turns on. Method Dallas

3. Change something inside, then watch

A map of the inside is only a guess. To test it, the researchers change the model while it runs. They turn a group of features down, or push one up, and watch the answer. This is an intervention experiment. Method

If the answer changes the way the map predicts, the map is more likely to be right. Method

chapter 3

What the paper does

This chapter is the hardest one on the page. Read the sketch first, one step at a time.

First, the researchers train a replacement model. It copies the real model's MLP layers, with features in place of neurons. The trained part is a cross-layer transcoder: the piece that turns the model's neurons into features. It has 30 million features. Method

The copy is not perfect. For one prompt, error nodes hold what the features miss. The copy also reuses the real model's attention as it was on that prompt. This fixed-up copy is the local replacement model. Method

Next, they draw an attribution graph for one output token. Each box is a feature. Each arrow shows how much one feature pushed another. Boxes and arrows that add little to the answer are removed. Method

Then they group features that share a meaning into supernodes, by hand. Method

Last, they test the diagram with intervention experiments in the real model. They chose the labels and groups before they saw the test results. Method

The worked example: Dallas

Figure 6 from the paper Figure 6 open full size

The bottom row is the prompt. “Dallas” is a supernode.

“Dallas” turns on “Texas”, a word the prompt never uses.

“capital” turns on “say a capital”.

“Texas” and “say a capital” together push “say Austin”.

This line runs straight from “Dallas” to “say Austin”: the shortcut.

“Dallas” leads to “Texas”, and “capital” leads to “say a capital”. Together they push “Austin”. Boxes such as “say Austin” are output features: they push the model to say a certain word. Dallas

The paper calls the Dallas, Texas, Austin chain multi-hop reasoning: the answer depends on in-between steps. A direct shortcut from “Dallas” to “say Austin” sits in the same diagram. Dallas

chapter 4

What they found

Four of the paper's case studies; six more are in a fold. Intro

1. The Texas step can be swapped

It said “Sacramento”, the capital of the state it was given in place of Texas. Swap The same swap worked for Georgia, British Columbia, China and the Byzantine Empire. Swap

The swap results, in Figure 9, and stronger pushes

Figure 9 from the paper Figure 9 open full size

No change: the model says “Austin” 98% of the time.

“Texas” down, “California” up: the model says “Sacramento”.

Places that are not U.S. states needed a push of 8 times, where the U.S. states needed 2 times.

The “California” features came from a second prompt, about Oakland. Swap The places that are not U.S. states needed a stronger push. The paper suggests their features “fit” less naturally into the path. Swap

2. The model picks the rhyme before it writes the line

After “He saw a carrot and had to grab it,” the model ends the next line with “rabbit”. Features for “rabbit” and “habit” are on at the line break, before the second line starts. These are planned words. Poems

Figure 10 from the paper Figure 10 open full size

The first line ends in “grab it”.

The model is at the line break.

“rabbit” and “habit” are already on. These are the planned words.

With “rabbit”, the line ends “like a starving rabbit”. With “rabbit” turned down, it ends in “habit”.

Choosing the end first is forward planning. Writing the line to reach that end is backward planning. Poems

Turning “rabbit” down made the model end the line with “habit”. Poems In 25 poems, a planned word pushed in at the line break ended the line in 70% of cases. Poems

The test results, in Figure 12, and where the push works

Figure 12 from the paper Figure 12 open full size

No change: the last word is “rabbit” 81% of the time.

“rabbit” turned down at the line break: “habit” 95% of the time.

“rabbit” and “habit” turned down, “green” turned up: lines now end in “green”.

The paper found planned words in about half of the poems it studied. Poems

The push only worked at the line break. The same push on other tokens had no effect. Poems

3. Saying “I can't answer” is the default

Asked which sport Michael Jordan plays, the model said “Basketball”. For a made-up name, Michael Batkin, it apologized. Hallucinations

In a chat, the model plays the Assistant. In both questions, Assistant features turn on “can’t answer” features, which push it to say it cannot answer. Hallucinations The paper calls such a path a default circuit: on unless something turns it off. Components

For Jordan, the familiar name turns on “known answer” features, which shut the “can’t answer” features off. For Batkin, “unknown name” features turn on instead, and the “can’t answer” features stay on. Hallucinations

Figure 33 from the paper Figure 33 open full size

Batkin, right: Assistant features turn on “can’t answer” features by default.

The unfamiliar name adds “unknown name” features. The model apologizes.

Jordan, left: “known answer” features turn the “can’t answer” features off (blue lines with flat ends). The model says “Basketball”.

Some hallucinations, made-up answers, may come from this switch firing by mistake. Asked for a paper by Andrej Karpathy, the model named one he did not write. His name was familiar, so the “known answer” features turned on, though the model did not know his papers. Hallucinations

4. The written steps are not always the real steps

A first case: adding 36 and 59, in Figure 26

The model adds 36 and 59 correctly. Inside, one path finds the rough size of the sum. Another finds the last digit. Asked how it got the answer, the model describes the school method: add the ones, then carry the 1. That explanation does not match the diagram. Addition

Some of these features are lookup table features. One turns on when the two numbers end in 6 and 9. It pushes the sum to end in 5. Addition

Figure 26 from the paper Figure 26 open full size

One path works with rough sizes, such as “about 36 plus about 60”.

Another path looks only at last digits: “ends in 6 plus ends in 9”. That sum ends in 5.

The two paths meet at the answer, 95.

The same feature also turns on outside math problems. One case is a citation, where a journal volume and its founding year add up to the year of publication. Addition

The paper also checks the model's own explanations. Steps it writes out before it answers are called chain-of-thought reasoning. Written steps that do not reflect the model's real process are unfaithful. CoT

Figure 51 shows three math prompts. The model cannot work out the top two in its head. It can work out the bottom one. CoT

Figure 51 from the paper Figure 51 open full size

Top: the diagram runs back from “say 8” to the human's 4.

Middle: no calculation, only a guess.

Bottom: a real square root of 64.

Top: the human says they got 4 by hand. The model writes the middle value that leads to 4, working backward from the hint. This is motivated reasoning. CoT

Middle: it claims to use a calculator it does not have, and the diagram shows only a guess. The paper calls this bullshitting. Bottom: it really does the math. CoT

When the human's answer changed, the model's steps changed to land on it. CoT

chapter 5

What this does not show

The paper states these limits itself. Limitations

• About a quarter of prompts. The tools gave a satisfying view for about a quarter of the prompts the team tried. The paper shows the cases that worked. Approach
• Only examples. Each finding shows that a mechanism exists in specific examples. The paper does not claim how common it is. It likely misses other mechanisms in the same examples. Limitations
• Built from a copy. The diagrams come from the replacement model. Even the diagrams shown have large error node parts. Method Limitations
• Attention is a blind spot. The method does not explain how the model decides where to look back. In the jailbreak and in the motivated reasoning case, a key step happens there. Limitations CoT

chapter 6

Words to know

Hover or tap any underlined word on the page to see its entry. These are the terms this page uses, in the order it introduces them.

polysemantic (many-job)

One neuron does several unrelated jobs, so you cannot name what it does.

"their neurons are typically polysemantic – that is, they perform many different functions that are seemingly unrelated" (Method Overview)

feature

A unit that turns on for one idea, such as “Texas” or “a word that rhymes with -it”.

"features, sparsely active “replacement neurons” that often represent interpretable concepts" (Method Overview)

also called in the paper: replacement neurons

superposition

The model holds more ideas than it has neurons, so ideas share neurons.

"a phenomenon known as superposition , whereby models represent more concepts than they have neurons and thus cannot assign each neuron to its own concept" (Method Overview)

intervention experiment (change-and-watch test)

Turn some features down or up inside the real model, and see what the model says.

"we can perform intervention experiments in the original model, such as inhibiting feature groups and observing their effects on other features and on the model’s output" (Method Overview)

replacement model (readable copy of the model)

A second model, built to act like the real one, whose parts are easier to name.

"we build a replacement model that approximately reproduces the activations of the original model using more interpretable components" (Method Overview)

local replacement model (copy for one prompt)

The readable copy, fixed up so it matches the real model exactly on one prompt.

"The resulting model – incorporating error nodes and inheriting the attention patterns from the original model – we call the local replacement model." (Method Overview)

error node (leftover box)

A box that holds the part of the real model the features do not explain.

"We can fill in these gaps by including error nodes which represent the discrepancy between the two models." (Method Overview)

attribution graph (map of steps)

A drawing of which features pushed on which other features, ending at the model's answer.

"attribution graphs, a graphical representation of the computational steps the model uses to determine its output for a particular input, in which nodes represent features and edges represent the causal interactions between them" (Method Overview)

supernode (group of features)

Several features with a shared meaning, drawn as one box.

"By manually grouping these related graph nodes together into supernodes, we can obtain a simplified depiction of the computational steps performed by the model." (Method Overview)

cross-layer transcoder

The part that is trained to turn the model's neurons into features.

"Our replacement model is based on a cross-layer transcoder (CLT) architecture" (Method Overview)

output feature

A feature whose job is to push the model toward saying a certain word, such as “say Austin”.

"We also identify “output features” that consistently push the model to say certain tokens, even if there isn’t such a clear pattern to what words/phrases they activate on." (Introductory Example: Multi-step Reasoning)

multi-hop reasoning (reasoning in steps)

Getting to an answer through an in-between fact, such as Dallas, then Texas, then Austin.

"the replacement model does in fact perform “multi-hop reasoning” – that is, its decision to say Austin hinges on a chain of several intermediate computational steps" (Introductory Example: Multi-step Reasoning)

planned word

The word the model picks for the end of a line before it writes the line.

"It could then use this “planned word” to inform how it writes the next line, so that the planned word will fit naturally at the end of it." (Planning in Poems)

forward planning

Choosing where you want to end up before you start.

"First, the model uses the semantic and rhyming constraints of the poem to determine candidate targets for the next line." (Planning in Poems)

backward planning

Writing the earlier words so that they lead to the chosen end.

"Next, the model works backward from its target word to write a sentence that naturally ends in that word." (Planning in Poems)

Assistant

The character the model plays in a chat: the one that answers the human.

"During finetuning, however, models are trained to avoid such behavior when acting in the Assistant character." (Entity Recognition and Hallucinations)

“can’t answer” features

Features that push the model to say it does not know.

"The crux of this circuit is a group of “can’t answer” features that activate when the Assistant corrects or questions the premise of a user’s question, or declares that it has insufficient information to give a response." (Entity Recognition and Hallucinations)

default circuit (always-on path)

A path inside the model that is on unless something turns it off.

"We have observed several instances of circuits that appear active “by default” in certain contexts." (Commonly Observed Circuit Components and Structure)

“known answer” features

Features that turn on for a question the model is likely to know the answer to.

"A group of “known answer” and “known entity” features that activate when the Assistant is asked questions about people/things it is likely to know about." (Entity Recognition and Hallucinations)

also called in the paper: known entity features

“unknown name” features

Features that turn on for a name the model does not recognize.

"The “can’t answer” features are also promoted by a group of unfamiliar name features, which is in turn activated by the individual tokens of “Michael Batkin” and a generic “first names” feature." (Entity Recognition and Hallucinations)

also called in the paper: unfamiliar name features

hallucination

An answer the model makes up, said as if it were true.

"Language models are known to sometimes hallucinate; that is, make up false information in response to questions." (Entity Recognition and Hallucinations)

lookup table feature

A feature that has memorized one fact of the addition table, such as “6 plus 9 ends in 5”.

"We found a key step performed by “lookup table” features that translate between properties of the input (like the two numbers being summed ending in 6 and ending in 9) and a property of the output (like ending in 5)." (Addition)

chain-of-thought reasoning

Steps the model writes out before it gives its answer.

"Language models “think out loud,” a behavior known as chain-of-thought reasoning (CoT)." (Chain-of-thought Faithfulness)

unfaithful

The written steps are not the steps the model really used.

"prior work has shown that CoT reasoning can be unfaithful – that is, it can fail to reflect the actual mechanisms used by the model" (Chain-of-thought Faithfulness)

motivated reasoning

Choosing the steps so they land on the answer someone else suggested.

"motivated reasoning – tailoring its reasoning steps to arrive at the human-suggested answer" (Chain-of-thought Faithfulness)

bullshitting

Making up an answer without caring if it is true.

"bullshitting in the sense of Frankfurt – making up an answer without regard for the truth" (Chain-of-thought Faithfulness)

Claude 3.5 Haiku

The language model the paper studies, made by Anthropic.

"Claude 3.5 Haiku, released in October 2024, which serves as Anthropic’s lightweight production model as of this writing" (Introduction)

token

A word or piece of a word. The model reads tokens and writes one token at a time.

"transformer-based language models, which take in sequences of tokens (e.g. words, word fragments, and special characters), and output new tokens one at a time" (Method Overview)

attention layer (look-back layer)

A layer that copies information from one token to another, such as from an earlier word to the last one.

"attention layers, which move information between token positions" (Method Overview)

MLP layer (thinking layer)

A layer that works on one token at a time, using many small units called neurons.

"MLP (“multi-layer perceptron”) layers, which process information within each token position using collections of neurons" (Method Overview)

finetuning

Training after the first round, which teaches the model to act as a helpful assistant.

not defined in the paper

jailbreak

A trick prompt that gets a model to do something it would normally refuse.

"Jailbreaks are prompting strategies that can cause models to comply with requests they would ordinarily refuse." (Life of a Jailbreak)

reward model

A model that scores answers during training, so the trained model learns to give high-scoring answers.

not defined in the paper