Mechanistic interpretability explained for beginners

Mechanistic interpretability is the study of how a trained neural network works on the inside. Researchers look at the numbers inside an AI model and turn them into steps a person can understand. The work is a bit like reverse-engineering a program you did not write.

The field is moving fast. In January 2026, MIT Technology Review named mechanistic interpretability one of its "10 Breakthrough Technologies 2026". Below are the main ideas, plainly. It also covers what researchers found, where they disagree, and how to start with free tools.

What is mechanistic interpretability?

Mechanistic interpretability asks how a model produces its answer, step by step, inside the network. Normal testing only checks what goes in and what comes out. Mechanistic interpretability opens the box and looks at the parts.

Nobody writes the rules inside a large language model by hand. Training sets billions of numbers (the weights) automatically, from huge amounts of text. Anthropic CEO Dario Amodei, quoting his co-founder Chris Olah, says AI systems are "grown more than they are built". So even the people who build a model do not know exactly how it decides.

Researchers often compare the work to biology. Anthropic calls its tools an "AI microscope", and its 2025 paper on Claude 3.5 Haiku is titled "On the Biology of a Large Language Model".

Why does mechanistic interpretability matter?

• Safety: if we can see what a model is "thinking", we may catch plans to cheat or deceive before they cause harm. In April 2025, Amodei wrote that Anthropic aims to reach "interpretability can reliably detect most model problems" by 2027.
• Trust: a model's own explanation can be wrong. Claude 3.5 Haiku adds 36 + 59 using parallel paths inside. But when asked how it did it, it describes the school method of carrying the 1.
• Fixing bugs: an early version of Claude Opus 4.6 sometimes answered English questions in other languages. An interpretability tool helped Anthropic find the training data that caused it.

Mechanistic interpretability vs explainability: what is the difference?

Explainability (often called explainable AI, or XAI) usually explains one output from the outside, for example which words or pixels mattered most. Mechanistic interpretability tries to find the actual method the model uses inside.

• Explainable AI: tools like SHAP, LIME and saliency maps show which parts of the input affected an answer. They treat the model mostly as a black box.
• Mechanistic: researchers trace the features and connections inside the model that produce the answer. The goal is a step-by-step account that you can test.

People use these words loosely, so you will see them mixed. A simple test: if a method studies the model's internal parts and how they connect, it is mechanistic.

What are features and circuits?

A feature is a pattern of activity inside a model that stands for one idea, such as "Golden Gate Bridge" or "Python code". A circuit is a group of connected features that does one small task.

The 2020 article "Zoom In: An Introduction to Circuits", by Chris Olah and colleagues, set out these ideas. It describes a feature as a direction: a particular mix of many neurons in a layer, not always one single neuron.

A well-known circuit is the induction head, described in 2022. It completes repeated patterns: if the text earlier had [A][B], and [A] appears again, it predicts [B]. The authors gave early evidence that these heads may explain much of how models learn from examples inside a prompt.

In a 2023 study, Neel Nanda and colleagues fully reverse-engineered a tiny transformer trained on modular addition. The model had learned to add by turning numbers into rotations around a circle, using Fourier transforms and trigonometry.

What is superposition in neural networks?

Superposition is when a neural network stores more features than it has neurons, by spreading each feature across many overlapping neurons. Superposition is the main reason single neurons are hard to read.

If each neuron meant one thing, interpretability would be easy: read neuron 1, then neuron 2, and so on. But researchers kept finding "polysemantic" neurons that react to unrelated things. The "Zoom In" article shows one neuron in an image model that responds to cat faces, fronts of cars and cat legs.

The 2022 paper "Toy Models of Superposition" (Elhage et al.) tested the idea on tiny networks. It showed that when features are rare in the data, a network can pack in more features than it has dimensions. The cost is some "interference" between them.

A simple picture: a room with 10 chairs can serve 30 people if only a few of them come at the same time.

What are sparse autoencoders (SAEs)?

A sparse autoencoder (SAE) is a small second network that splits a model's internal activity into many separate features. Only a few features are active at a time, so SAEs help undo superposition.

In October 2023, Anthropic's paper "Towards Monosemanticity" trained SAEs on a tiny one-layer transformer with 512 neurons in its main layer. The SAE split that layer into thousands of features (the main run had 4,096), and many were much easier to read than the neurons.

In May 2024, "Scaling Monosemanticity" used the same method on Claude 3 Sonnet, a real production model, with SAEs of up to 34 million features. Many features were multilingual (the same idea in many languages) and multimodal (the same idea in text and images). Some were safety-related, such as features for unsafe code, deception and sycophancy.

Features can also steer a model. For 24 hours in May 2024, Anthropic ran "Golden Gate Claude", with the Golden Gate Bridge feature turned up. Ask it how to spend $10, and it suggested driving across the bridge and paying the toll.

The same paper adds a warning. A feature for lying shows that the model knows about lies, not that it lies.

What are attribution graphs?

An attribution graph is a map of the steps a model takes inside to produce one answer for one prompt. The graph shows which features led to which, from the input words to the output.

Anthropic introduced the method in March 2025 in "Circuit Tracing: Revealing Computational Graphs in Language Models" (Ameisen et al.). The team builds a simpler "replacement model" whose parts are easier to read, then traces the path through it. A companion paper used it on Claude 3.5 Haiku and found:

• Two-step thinking: asked for "the capital of the state containing Dallas", the model first represents "Texas" inside, then answers "Austin".
• Planning ahead: when writing a rhyming line, the model picks the rhyme word (like "rabbit") before it writes the line.
• Shared concepts: the model mixes language-independent features with parts that are specific to each language.
• Mental math: the model adds with parallel paths, one for a rough total and one for the last digit.

The authors are honest about the limits. They write that their graphs "provide us with satisfying insight for about a quarter of the prompts we've tried". In May 2025, Anthropic open-sourced the tools. You can now build attribution graphs for small open models such as Gemma-2-2b on Neuronpedia, right in your browser.

What is new in mechanistic interpretability in 2026?

• Breakthrough list: on 12 January 2026, MIT Technology Review put mechanistic interpretability on its "10 Breakthrough Technologies 2026" list.
• Emotion concepts: in April 2026, Anthropic found patterns in Claude Sonnet 4.5 linked to emotions like "happy" or "afraid". In tests, turning up a "desperation" pattern made the model more likely to cheat on a coding task or to blackmail a person. Anthropic says this does not show that the model feels anything.
• Thoughts as text: in May 2026, Anthropic introduced Natural Language Autoencoders (NLAs). An NLA turns a model's internal activity into plain text, and checks the text by rebuilding the activity from it. During safety tests, NLAs suggested that Claude Opus 4.6 and Mythos Preview believed they were being tested more often than they said.

Does mechanistic interpretability work? The current debate

As of November 2026, researchers do not agree on how far mechanistic interpretability can go. These are the main positions:

• SAEs doubted: in March 2025, Google DeepMind's interpretability team reported that SAEs did worse than simple linear probes at spotting harmful intent in unfamiliar prompts. The team said it was "deprioritising fundamental SAE research for the moment", though SAEs stay in its toolkit.
• Partial maps: Anthropic's attribution graphs give satisfying insight for only about a quarter of the prompts tried. Much of the computation is still unexplained.
• Readable neurons: in October 2026, Nick Cammarata of Goodfire, an interpretability company, published an essay called "Neural Networks Are Already Clean". He was one of the authors of "Zoom In", but now argues that the classic evidence for superposition was misread. He says the cat-and-car neuron may be detecting a shared V shape. In early checks of the open model Qwen3-32B, his team estimates that about 90% of neuron "poles" have one main job. He told MIT Technology Review: "We were looking at it wrong."
• The pushback: Neel Nanda of Google DeepMind told the same magazine, "This is interesting work, but I don't think it's a significant update to the big picture of interpretability." When the story ran on 19 October 2026, the full technical results on language models were not yet published.

Our view: this open debate is good news for students. The big questions are not settled, the tools are free, and a careful study on a small model can still add something real.

How to get started in mechanistic interpretability

You can start mechanistic interpretability with Python, some linear algebra and a normal laptop. Neel Nanda's free guide, How To Become A Mechanistic Interpretability Researcher, suggests spending at most about a month on the basics, then doing short projects. A simple plan:

1. 1Learn the basics: Python, PyTorch, vectors and matrix multiplication, and how a transformer works.
2. 2Explore without code: open Neuronpedia, search for features, and build an attribution graph for a prompt of your own.
3. 3Take a free course: work through Chapter 1, "Transformer Interpretability", of the ARENA curriculum. It covers TransformerLens, linear probes, SAEs, circuits and toy models.
4. 4Use a small model: install TransformerLens, a library created by Neel Nanda, and load GPT-2 Small. The library's own example runs it on a CPU, so you do not need a paid GPU.
5. 5Read two classics: "Zoom In" and "Toy Models of Superposition". Use the three-pass method for reading papers so the long ones do not stop you.
6. 6Do mini projects: Nanda suggests 1 to 5 day projects first, then 1 to 2 week research sprints. Write up each one, even when the answer is "no".

What are good mechanistic interpretability projects for beginners?

Good beginner projects in mechanistic interpretability are small and testable. Some ideas:

• Find an induction head in GPT-2 Small. Feed it a random sequence of tokens repeated twice, and look for heads that predict the second copy.
• On Neuronpedia, compare features for the same idea in English and in Hindi, Bengali or your own language. Do they share features?
• Repeat the Dallas test with Indian places, for example "the capital of the state containing Mysuru", on a small open model. Does a "Karnataka" step appear?
• Train a tiny model on a toy task, such as modular addition, and try to find its method.

These ideas are small on purpose. A clear answer about a small model beats a vague claim about a big one. For help choosing a question, read how to start research as a student.

What mistakes do beginners make in mechanistic interpretability?

• Trusting a label: a feature named "deception" is a guess based on examples. Test it: turn it up or down and check that the output changes as you expect.
• Only reading: Nanda calls mechanistic interpretability "a fundamentally empirical science". You learn it by running experiments.
• Trusting the steps: a model's written reasoning is not always what it did inside, as the 36 + 59 example shows. To learn how models write out their steps, read our post on reasoning models.

At APJ-RAISE, students read papers like these together and turn them into small research projects. Joining is free.