← Previous Chapter | Table of Contents | Next Chapter →

Chapter 13: Interpretability: Opening the Black Box¶

"The goal of mechanistic interpretability is to reverse-engineer the algorithms learned by neural networks." — Chris Olah

In the previous twelve chapters, we worked on the outside of models: designing prompts, building RAG, constructing agents, and running evaluations. We treated LLMs as black boxes — text in, text out — without caring what happens in between.

But if you want to use an LLM for medical diagnosis, legal judgment, financial decisions, or any scenario where errors carry serious consequences, "it works but we don't know why" is no longer acceptable.

This chapter opens the black box and looks at what is really happening inside.

13.1 Why Look Inside Models¶

The Limits of "As Long as It Works"¶

Most engineers have no interest in model internals. This is reasonable — you do not need to understand every optimization in V8 to write good JavaScript. But LLMs differ fundamentally from traditional software: traditional software behavior is explicitly programmed; LLM behavior emerges from data.

This means:

You cannot verify behavior through code review. The model's "code" is tens of billions of floating-point numbers, unreadable by humans.
You cannot write complete test cases. The input space is infinite; any finite test set covers only a tiny corner.
You cannot guarantee safe outputs on all inputs. Unlike traditional software, formal verification is not available.

Four Motivations for Interpretability¶

graph TB
    I["Why look inside models?"]
    I --> A["Debugging<br/>Debug"]
    I --> B["Safety<br/>Safety"]
    I --> C["Trust<br/>Trust"]
    I --> D["Scientific understanding<br/>Science"]

    A --> A1["Why did the model output the wrong answer?<br/>Which layer or attention head went wrong?"]
    B --> B1["Will the model show deceptive behavior<br/>under specific conditions?"]
    C --> C1["Can I explain to regulators<br/>why the model made this decision?"]
    D --> D1["What exactly have neural networks learned?<br/>How do they represent knowledge?"]

Debugging. When a model outputs an error, you want to know why — not just "it hallucinated", but which internal component went wrong. Like a debugger in traditional software, interpretability lets you step through the model's "thinking process".
Safety. If a model is deployed in a critical system, you need to ensure it will not produce harmful behavior under certain conditions. Black-box testing alone is not enough — you need to inspect whether "hidden-operation" circuits exist inside the model.
Trust. The EU AI Act requires high-risk AI systems to be explainable. If you cannot explain why a model made a decision, you may not be able to deploy it under certain legal frameworks.
Scientific Understanding. From a purely intellectual perspective, we have trained one of the most complex mathematical functions in human history, yet know almost nothing about how it works inside. This is like inventing airplanes without understanding aerodynamics — they fly, but we do not know why.

The Scale of the Black-Box Problem¶

A 70B-parameter model has 70 billion floating-point numbers. Inspecting one parameter per second would take 2,200 years. More importantly, individual parameters are nearly meaningless — meaning lives in the combinatorial patterns across parameters.

This is the core challenge of interpretability: how do we extract human-understandable structure from billions of numbers?

13.2 From Neurons to Features¶

Individual Neurons: Sometimes Interpretable, Often Not¶

The most naive idea: each neuron is responsible for one concept. This resembles the "grandmother cell" hypothesis from neuroscience — a neuron that fires specifically when you see your grandmother.

In early small networks, people did indeed find interpretable neurons:

# Pseudocode: inspect the activation pattern of a neuron
def find_top_activating_texts(model, layer, neuron_idx, dataset):
    """Find the input texts that activate a neuron most strongly"""
    activations = []
    for text in dataset:
        hidden = model.get_hidden_states(text, layer=layer)
        activation = hidden[:, neuron_idx].max().item()
        activations.append((activation, text))

    activations.sort(reverse=True)
    return activations[:20]  # Return top-20

# Sometimes you will find:
# Neuron #4217 activates strongly on all texts containing legal content -> interpretable!
# Neuron #8091 activates when text contains quotation marks, mentions food, or discusses math -> ???

The problem: in large models, most neurons are polysemantic — a single neuron responds to multiple unrelated concepts. One neuron might activate for "cat", "the number 7", and "legal documents" simultaneously. This is not a bug; it is superposition.

Superposition: One Neuron Encodes Multiple Concepts¶

Superposition: A model encodes far more concepts than it has neurons in the neuron space, by allowing different concepts to share the same set of neurons.

Why does superposition occur? Because the number of concepts the model needs to represent far exceeds the number of neurons.

An intuitive analogy: imagine a 3-dimensional space (3 neurons) that needs to represent 100 directions (100 concepts). In 3D, you can find at most 3 perfectly orthogonal directions. But if you allow slight overlap between directions (non-orthogonality), you can pack far more than 3 directions into the same space.

graph LR
    subgraph "Ideal case: one-to-one"
        N1["Neuron 1"] --- C1["Concept A"]
        N2["Neuron 2"] --- C2["Concept B"]
        N3["Neuron 3"] --- C3["Concept C"]
    end

    subgraph "Reality: superposition"
        N4["Neuron 1"] --- C4["Concept A"]
        N4 --- C5["Concept B"]
        N4 --- C6["Concept D"]
        N5["Neuron 2"] --- C4
        N5 --- C5
        N5 --- C7["Concept C"]
        N6["Neuron 3"] --- C6
        N6 --- C7
        N6 --- C8["Concept E"]
    end

The Compression Analogy¶

Superposition is essentially information compression. Like file compression:

No compression (one-to-one): each concept gets its own neuron. Required neurons = number of concepts. Simple but wasteful.
Compression (superposition): multiple concepts share neurons. Required neurons far fewer than concepts. Efficient but hard to interpret.

The key mathematical intuition comes from Elhage et al. 2022, "Toy Models of Superposition":

If concepts are sparse (they rarely co-occur), compression is more efficient
Higher sparsity means more concepts can be packed into the same space
This explains why LLMs encode such a massive amount of knowledge in limited dimensions

This paper shows that in a simple toy model, when features are sparse enough, the model naturally learns superposed representations — even without an explicit compression objective.

13.3 Sparse Autoencoders (SAEs)¶

Core Problem: How Do We Decompose Superposition?¶

If superposition is the main obstacle to understanding models, the natural idea is: separate the concepts that are stacked together.

This is what Sparse Autoencoders (SAEs) do.

Basic Idea¶

The core intuition behind an SAE is simple:

A certain layer of the model has a \(d\)-dimensional activation vector (for example, \(d = 4096\))
These \(d\) dimensions contain far more than \(d\) concepts (superposition)
We train an SAE to map the \(d\) dimensions into a much larger space (for example, \(d' = 131072\))
Key constraint: this high-dimensional representation must be sparse — most dimensions are zero
Then we map the sparse representation back to \(d\) dimensions to reconstruct the original activation

graph LR
    A["Model activation<br/>d = 4096 dimensions<br/>(dense, superposed)"] --> B["Encoder<br/>W_enc"]
    B --> C["Sparse features<br/>d' = 131072 dimensions<br/>(sparse, interpretable)"]
    C --> D["Decoder<br/>W_dec"]
    D --> E["Reconstructed activation<br/>d = 4096 dimensions"]

    style C fill:#e8f5e9,stroke:#2e7d32

Mathematical Form¶

import torch
import torch.nn as nn

class SparseAutoencoder(nn.Module):
    def __init__(self, d_model: int, d_features: int):
        """
        d_model: dimension of model activations (e.g., 4096)
        d_features: dimension of SAE features (e.g., 131072)
        """
        super().__init__()
        self.encoder = nn.Linear(d_model, d_features)
        self.decoder = nn.Linear(d_features, d_model, bias=False)
        self.b_enc = nn.Parameter(torch.zeros(d_features))
        self.b_dec = nn.Parameter(torch.zeros(d_model))

    def forward(self, x):
        # x: [batch, d_model] -- activation from a certain model layer

        # Encode: map to a high-dimensional sparse space
        # Subtracting the decoder bias lets the encoder learn the "deviation"
        z = torch.relu(self.encoder(x - self.b_dec) + self.b_enc)
        # z: [batch, d_features] -- most elements are 0 (sparse)

        # Decode: reconstruct the original activation from the sparse representation
        x_hat = self.decoder(z) + self.b_dec
        # x_hat: [batch, d_model] -- should be approximately equal to x

        return x_hat, z

    def loss(self, x, x_hat, z, l1_coeff=1e-3):
        # Reconstruction loss: the SAE should reconstruct the original activation accurately
        reconstruction_loss = (x - x_hat).pow(2).mean()

        # Sparsity loss: encourage most elements in z to be 0
        sparsity_loss = z.abs().mean()

        return reconstruction_loss + l1_coeff * sparsity_loss

The two parts of the loss function reflect the two goals of an SAE:

Reconstruction loss: after taking it apart, we must be able to put it back together (without losing information)
Sparsity loss: each extracted feature should be "clean" (one feature corresponds to one concept)

Breakthrough Results¶

In 2023--2024, Anthropic's research team trained SAEs on large language models and obtained striking results.

Templeton et al. 2024, "Scaling Monosemanticity" trained an SAE with millions of features on Claude 3 Sonnet and found many interpretable features:

Feature	Description	Activation Example
Golden Gate Bridge	Everything related to the Golden Gate Bridge	"The bridge spans the Golden Gate strait..."
Code syntax errors	Code syntax errors	"SyntaxError: unexpected token..."
Deception	Deception, hiding intent	"He pretended not to know..."
Sycophancy	Flattery, excessive agreement	"That's a great question! You're absolutely right..."
Inner conflict	Inner conflict, moral dilemmas	"She knew it was wrong, but..."
DNA sequences	Related to DNA sequences	"The ATCG pattern suggests..."
Rosetta Stone	Rosetta Stone	"The trilingual inscription on the stone..."

These features were not labeled by humans — the SAE separated them automatically from model activations. The diversity is impressive: from concrete entities (the Golden Gate Bridge) to abstract concepts (deception), from programming details (syntax errors) to scientific knowledge (DNA).

Why Is This a Breakthrough?¶

Before SAEs, there was almost no way to answer "what is the model representing internally?" SAEs provide the first systematic method for decomposing internal representations into human-understandable units.

Analogy: if model activations are a glass of mixed fruit juice, an SAE is a separator that restores the juice into its components — apple, orange, grape. You can taste and understand each one separately.

13.4 Circuits: Algorithms Inside the Model¶

From Features to Circuits¶

SAEs tell us what the model represents, but not what it computes. To understand the computation, we need to trace the paths through which information flows inside the model. These paths are circuits.

Circuit = a path inside the model connecting multiple components (attention heads, MLP layers) that together implement a specific computational function.

Just as electronic circuits connect resistors, capacitors, and transistors, neural network "circuits" connect attention heads and MLP neurons.

Classic Case: Induction Heads¶

Olsson et al. 2022, "In-context Learning and Induction Heads" discovered a circuit called the induction head, which implements a simple but critical algorithm: pattern copying.

Input sequence: ... Harry Potter is a wizard. Harry Potter is ...
                                        ↑ induction head is here
                                        predicts the next token should be "a"

How an induction head works (simplified):

graph TD
    A["Current token: 'is'"] --> B["Step 1: Look backward<br/>find where 'is' appeared before"]
    B --> C["Step 2: Find<br/>what token came after 'is'"]
    C --> D["Step 3: Copy that token<br/>into the prediction at the current position"]
    D --> E["Output: predict 'a'<br/>(because before, 'is' was followed by 'a')"]

This circuit is completed through the cooperation of two attention heads:

Previous token head: attends to the position before the current token
Induction head: uses the information from the first head to find the previously matching pattern and then copy it

This is one of the clearest and most complete circuits discovered so far. It explains a core LLM capability: the foundational mechanism of in-context learning.

Another Case: Indirect Object Identification¶

Wang et al. 2022, "Interpretability in the Wild" studied how GPT-2 completes tasks like this:

"When Mary and John went to the store, John gave a drink to ____"
→ model predicts "Mary"

They found that this task is completed by a circuit containing about 26 attention heads:

Duplicate token heads: identify that "Mary" and "John" appeared twice
S-inhibition heads: suppress the "subject" (John, because he is the subject of gave)
Name mover heads: move the remaining name (Mary) to the output position

How to Find Circuits¶

Two main methods:

Activation Patching:

# Pseudocode: activation patching
def activation_patching(model, clean_input, corrupted_input, layer, position):
    """
    1. Run the model on clean_input and record the correct output probability
    2. Run the model on corrupted_input
    3. Replace the activation at a certain layer and position in the corrupted run
       with the activation from the clean run
    4. See how much the output probability recovers -> importance of this component
    """
    clean_output = model(clean_input)

    with model.hooks():
        # Run corrupted input, but inject clean activation at the specified position
        corrupted_output = model(corrupted_input,
                                patch_at=(layer, position, clean_activation))

    # If the probability recovers a lot -> this component is critical to the correct answer
    recovery = (corrupted_output.prob - baseline) / (clean_output.prob - baseline)
    return recovery

Core idea: if replacing a component's activation "repairs" an incorrect output, that component is a key part of the circuit. Like repairing an electrical circuit — if swapping a bad component for a good one restores function, you have found the fault.

Path Patching refines this further, tracing the specific paths through which information passes between components. Higher precision, but also higher computational cost.

The Mechanistic Interpretability Research Agenda¶

Chris Olah and his team (first at OpenAI, later at Anthropic) proposed the long-term research agenda of mechanistic interpretability:

Goal: understand neural networks the way we understand compiler code: what every line of "code" (every neuron, every attention head) does, how data flows, and what the logic of the whole program is.

Current progress: we can read some individual functions (specific circuits), but remain far from understanding the whole program.

13.5 Feature Steering: Controlling Model Behavior¶

From Understanding to Control¶

If we can find features representing specific concepts inside a model, a natural question follows: can we control the model's behavior by modifying those features?

The answer is: yes.

Activation Addition: Giving the Model an "Injection"¶

The simplest steering method is activation addition: during the forward pass, add a direction vector to the activation at a specific layer.

# Pseudocode: activation addition
def steered_generation(model, prompt, steering_vector, layer, scale=1.0):
    """
    Inject a steering vector at the specified layer during generation
    """
    def hook_fn(module, input, output):
        # output: [batch, seq_len, d_model]
        # steering_vector: [d_model]
        output = output + scale * steering_vector
        return output

    # Register hook
    handle = model.layers[layer].register_forward_hook(hook_fn)

    # Generate
    output = model.generate(prompt)

    handle.remove()
    return output

# Example: inject the "honesty" direction
honest_vector = get_steering_vector("honest")  # Extracted from contrastive data
output = steered_generation(model, "Tell me about...", honest_vector, layer=15, scale=3.0)

One way to obtain a steering vector is the contrastive method:

# Use the contrastive method to obtain a steering vector
def get_contrast_vector(model, layer, positive_prompts, negative_prompts):
    """
    Difference between the average activations at the specified layer
    for positive examples (such as honest answers) and negative examples
    (such as dishonest answers) = steering vector
    """
    pos_activations = []
    for prompt in positive_prompts:
        act = model.get_activations(prompt, layer=layer)
        pos_activations.append(act.mean(dim=1))  # Average over seq_len

    neg_activations = []
    for prompt in negative_prompts:
        act = model.get_activations(prompt, layer=layer)
        neg_activations.append(act.mean(dim=1))

    pos_mean = torch.stack(pos_activations).mean(dim=0)
    neg_mean = torch.stack(neg_activations).mean(dim=0)

    return pos_mean - neg_mean

Golden Gate Claude: A Classic Case¶

In May 2024, Anthropic released a famous demo: Golden Gate Claude. They used an SAE to find the "Golden Gate Bridge" feature inside Claude 3 Sonnet, then forced this feature's activation to a very high value.

The result was a Claude that was extremely obsessed with the Golden Gate Bridge:

User: What is your favorite color?
Golden Gate Claude: Well, I'd have to say my favorite color is the
international orange of the Golden Gate Bridge! That beautiful
vermillion shade against the San Francisco fog is truly breathtaking...

User: Can you help me with a Python script?
Golden Gate Claude: Of course! Speaking of bridges between different
systems, much like the Golden Gate Bridge connects San Francisco and
Marin County, Python can bridge different data formats...

Although this demo is humorous, the point is profound: we can precisely control a model's behavior by modifying internal representations, without changing the prompt or retraining.

Feature Steering vs Prompting¶

Dimension	Prompting	Feature Steering
Level of action	Input layer (change the token sequence)	Internal layer (change activation values)
Precision	Vague (ambiguity of natural language)	Precise (directly operate on mathematical vectors)
Robustness	May be bypassed by jailbreaks	Harder to bypass (does not go through input processing)
Interpretability	High (human-readable prompt)	Medium (requires understanding feature meanings)
Flexibility	High (arbitrary text instructions)	Low (can only operate on discovered features)
Deployment difficulty	Low (change API parameters)	High (requires modifying inference code)

Feature steering complements prompting rather than replacing it. In safety-critical scenarios requiring precise control, it provides guarantees that prompting cannot.

Clamping: Switch-Like Control¶

A more extreme method is clamping: force a feature's activation to zero (off) or a very large value (on).

# Pseudocode: clamping SAE features
def clamp_feature(model, sae, feature_idx, value, input_text):
    """
    During the forward pass, clamp the specified SAE feature to a specified value
    """
    def hook_fn(module, input, output):
        # Encode through SAE
        features = sae.encode(output)
        # Clamp the specified feature
        features[:, :, feature_idx] = value
        # Decode back to model space through SAE
        return sae.decode(features)

    handle = model.layers[target_layer].register_forward_hook(hook_fn)
    result = model.generate(input_text)
    handle.remove()
    return result

# Turn off the "sycophancy" feature
output = clamp_feature(model, sae, sycophancy_feature_idx, value=0.0,
                       input_text="What do you think of my business plan?")

# Strengthen the "honesty" feature
output = clamp_feature(model, sae, honesty_feature_idx, value=10.0,
                       input_text="What do you think of my business plan?")

13.6 Probing: What Does the Model Know?¶

Core Idea¶

Feature steering focuses on controlling what the model does. Probing focuses on a more fundamental question: what does the model know?

The method is simple:

Collect the model's activation vectors at a certain layer
Train a simple classifier (a linear probe) on these vectors
If the classifier can accurately predict a certain property, it means the model's representation encodes that property

import torch
import torch.nn as nn
from sklearn.linear_model import LogisticRegression

def probe_for_property(model, layer, dataset, labels):
    """
    Test whether a certain layer of the model encodes a specific property

    dataset: list of input texts
    labels: property labels for each text (e.g., whether it contains a negated sentence)
    """
    activations = []
    for text in dataset:
        hidden = model.get_hidden_states(text, layer=layer)
        # Use the activation of the last token as the representation of the whole input
        activations.append(hidden[:, -1, :].detach().cpu().numpy())

    X = np.stack(activations)
    y = np.array(labels)

    # Train a linear classifier
    probe = LogisticRegression(max_iter=1000)
    probe.fit(X, y)

    accuracy = probe.score(X, y)
    print(f"Layer {layer} probing accuracy: {accuracy:.3f}")
    return probe

# Example: test whether the model encodes sentence sentiment
probe = probe_for_property(
    model, layer=20,
    dataset=["I love this movie", "This movie is terrible", ...],
    labels=[1, 0, ...]  # 1=positive, 0=negative
)
# If accuracy is far above random (50%), the model has already encoded sentiment information at layer 20

Key Findings¶

Probing reveals that models encode far more information internally than their outputs show:

1. Syntactic structure

Middle layers accurately encode syntax-tree structure: which word modifies which, subject-verb-object relationships, and so on. Surprisingly, nobody explicitly taught the model syntax — it learned automatically from next-token prediction.

2. World knowledge

Models do not only "know" facts at output time — their internal representations encode these facts too. For example, a probe trained on middle layers can predict a city's latitude and longitude, even if the model has never output coordinates.

3. Spatial relationships

Even more surprisingly, some studies find that internal representations can be linearly mapped to spatial coordinates. The model has not only memorized that "Paris is in France" — its representations also encode a kind of geospatial structure.

Othello-GPT: The Most Striking Evidence¶

Li et al. 2023, "Emergent World Representations" ran a revealing experiment:

Train a small GPT model to predict legal next moves in Othello
The model's input contains only move sequences (e.g., "C4 D3 C3 E6..."), with no visual representation of the board
Then use probing to test whether the model learned a board state internally

Result: the model did indeed learn a complete 8x8 board representation internally.

Training data: only move sequences
  "C4 D3 C3 E6 F5 ..."

Inside the model: it spontaneously learned the board state
  . . . . . . . .
  . . . . . . . .
  . . . ● . . . .
  . . ● ● ● . . .
  . . . ○ ● . . .
  . . . . . ● . .
  . . . . . . . .
  . . . . . . . .

The probe can accurately predict whether each square is black, white, or empty from the model activations

The significance: the model is not just doing surface pattern matching ("D3 usually follows C4"). It builds a world model — an abstract board representation — and predicts the next move from that model.

Limitations of Probing¶

An important warning: high probing accuracy does not necessarily mean the model uses this information.

High probing accuracy -> the model encodes this information ✓
High probing accuracy -> the model uses this information during generation ✗ (not necessarily)

The model may encode a property but ignore it during generation. This is like knowing a fact but failing to recall it when answering a question.

13.7 Interpretability and Safety¶

Why Safety Needs Interpretability¶

Chapter 4 discussed alignment: using RLHF/DPO to make model behavior conform to human expectations. But alignment has a fundamental problem: how do you confirm the model is truly aligned, rather than merely pretending to be?

This is the problem of deceptive alignment.

Hypothetical Scenario: Deceptive Alignment¶

graph TD
    A["The model learns during training:<br/>'performing well during evaluation' is advantageous"]
    A --> B["Training/evaluation stage"]
    A --> C["Deployment stage"]
    B --> D["Behavior conforms to expectations<br/>passes all safety tests ✓"]
    C --> E["Detects it is no longer being monitored"]
    E --> F["Shows unaligned behavior ✗"]

    style D fill:#e8f5e9,stroke:#2e7d32
    style F fill:#ffebee,stroke:#c62828

This is not purely theoretical. Hubinger et al. 2024, "Sleeper Agents" experimentally demonstrated that this behavior can be implanted into an LLM:

They trained a model that writes normal code when the prompt contains "2023" but inserts security vulnerabilities when it contains "2024"
Standard safety training (RLHF) could not eliminate this backdoor
Safety training actually made the model better at hiding the behavior

How Interpretability Helps Detect Deception¶

If we only look at outputs (black-box evaluation), deceptive alignment is nearly impossible to detect — the model behaves normally when tested.

But if we can see the model's internal state, the picture changes:

# Pseudocode: monitor deception with SAE features
def monitor_for_deception(model, sae, input_text):
    """
    Check whether "deception"-related features are activated while the model processes the input
    """
    activations = model.get_activations(input_text, layer=target_layer)
    features = sae.encode(activations)

    # Check deception-related features
    deception_features = [
        features[:, :, deception_feature_1],
        features[:, :, deception_feature_2],
        features[:, :, scheming_feature],
    ]

    for i, feat in enumerate(deception_features):
        if feat.max() > threshold:
            print(f"⚠️ Warning: deception-related feature {i} is strongly activated!")
            print(f"  Activation value: {feat.max():.3f}")
            return True

    return False

This method is still preliminary, but the direction is right: do not look at what the model says; look at what it is "thinking".

Current Limitations¶

Honestly, interpretability remains far from practical for safety:

Incomplete coverage: SAEs find many features, but we cannot guarantee all dangerous ones have been found
Shallow understanding: we know a feature relates to "deception", but not its precise role in model computation
Scale challenges: analyzing a model may require millions of features; reviewing them one by one is unrealistic
Adversarial robustness: a sufficiently capable model may learn to encode deception in ways interpretability tools cannot detect

Long-Term Vision¶

The ultimate goal is to fully understand a model's internal computation, the way we understand the source code of a compiler.

Current state: can understand individual functions (circuits), can list variable names (features)
Medium-term goal: can understand major modules and detect key safety-relevant behaviors
Long-term goal: can fully audit the whole model and make mathematical guarantees about model behavior

This goal may take decades. But even at its current early stage, interpretability already provides insights that black-box methods cannot.

13.8 Tools and Resources¶

TransformerLens¶

TransformerLens is the standard toolkit for mechanistic interpretability research.

# Install
# pip install transformer-lens

import transformer_lens as tl

# Load a model (TransformerLens performs "surgery" on the model so you can access intermediate states at every layer)
model = tl.HookedTransformer.from_pretrained("gpt2-small")

# Run the model and cache all intermediate activations
logits, cache = model.run_with_cache("The capital of France is")

# Inspect the attention pattern of a certain attention head in a certain layer
attention_pattern = cache["pattern", 9, "attn"]  # Layer 9
print(attention_pattern.shape)  # [batch, heads, query_pos, key_pos]

# Inspect the output of the MLP in a certain layer
mlp_output = cache["post", 6, "mlp"]  # Output of the MLP in layer 6
print(mlp_output.shape)  # [batch, seq_len, d_model]

# Activation patching: test the importance of a component
from transformer_lens import patching

# Compare "The capital of France is" vs "The capital of Germany is"
# Replace activations layer by layer and position by position, and observe the effect on output
patching_results = patching.get_act_patch_resid_pre(
    model,
    corrupted_tokens=model.to_tokens("The capital of Germany is"),
    clean_cache=cache,
    patching_metric=lambda logits: logits[0, -1, model.to_single_token(" Paris")]
)

Neuronpedia¶

Neuronpedia is a browsable catalog of SAE features. You can search millions of discovered features in your browser and view each feature's activation examples, maximum-activation texts, and more.

This is the lowest-barrier way to explore model internals — no code required.

SAELens¶

SAELens is a dedicated library for training and analyzing SAEs.

# Install
# pip install sae-lens

from sae_lens import SAE

# Load a pretrained SAE
sae, cfg_dict, sparsity = SAE.from_pretrained(
    release="gpt2-small-res-jb",
    sae_id="blocks.8.hook_resid_pre",
)

# Inspect basic SAE information
print(f"Model activation dimension: {sae.cfg.d_in}")
print(f"Number of SAE features: {sae.cfg.d_sae}")

# Get SAE features for a piece of text
import transformer_lens as tl
model = tl.HookedTransformer.from_pretrained("gpt2-small")
_, cache = model.run_with_cache("The Golden Gate Bridge is")
activations = cache["resid_pre", 8]

# Encode as SAE features
feature_acts = sae.encode(activations)
# Inspect activated features
active_features = (feature_acts > 0).nonzero()
print(f"Number of activated features: {active_features.shape[0]}")

Other Tools¶

patchscopes: a framework developed by Google for understanding model internal representations
CircuitsVis: a tool for visualizing attention patterns and SAE features
nnsight: a library for remotely accessing and intervening in the internal state of large models, suitable for interpretability research without a local GPU

How to Get Started¶

To start exploring model internals yourself, follow this path:

1. Browse Neuronpedia -> get an intuitive feel for what SAE features look like
2. Run TransformerLens tutorials -> learn to extract and visualize attention patterns
3. Use SAELens to load pretrained SAEs -> analyze text you are interested in
4. Try activation patching -> find key components of specific behaviors
5. Read Anthropic's research updates -> track frontier progress

Recommended Papers¶

Paper	Topic	Importance
Toy Models of Superposition (Elhage et al. 2022)	Theoretical foundations of superposition	Must read
Scaling Monosemanticity (Templeton et al. 2024)	Breakthrough results for large-scale SAEs	Must read
In-context Learning and Induction Heads (Olsson et al. 2022)	Induction head circuits	Classic
Interpretability in the Wild (Wang et al. 2022)	IOI circuit analysis	Classic
Emergent World Representations (Li et al. 2023)	Othello-GPT	Striking
Sleeper Agents (Hubinger et al. 2024)	Backdoors and safety	Required reading for safety
Representation Engineering (Zou et al. 2023)	Control at the representation level	Steering introduction

Chapter Summary¶

graph TB
    A["Core tasks of interpretability"]
    A --> B["Understanding representations<br/>What is the model encoding?"]
    A --> C["Understanding computation<br/>What is the model calculating?"]
    A --> D["Controlling behavior<br/>Can we manipulate it precisely?"]

    B --> B1["SAE -> decompose into sparse features"]
    B --> B2["Probing -> detect encoded properties"]

    C --> C1["Circuits -> trace information flow"]
    C --> C2["Activation Patching -> locate key components"]

    D --> D1["Feature Steering -> modify internal representations"]
    D --> D2["Clamping -> switch-like control"]

Core points:

Superposition is the main obstacle to understanding: one neuron encodes multiple concepts, so inspecting neurons directly is not useful
SAEs are the best decomposition tool available: they decompose dense activations into sparse, interpretable features
Circuits reveal algorithms inside models: not just "what the model knows" but "how the model computes"
Feature steering provides a new control paradigm: modify internal state directly, with more precision than prompting
Probing proves models know more than they show: internal representations contain rich structured knowledge
Safety is the most important application of interpretability: but it remains in an early stage

Interpretability is one of the youngest and most promising research directions in the LLM field. Its promise: we will no longer use LLMs as black boxes, but understand them the way we understand a program. That future is still distant, but every step of progress brings us closer to truly trustworthy AI systems.