Golden Gate Claude: Opening Up the Black Box, One Bridge at a Time

Imagine you have ten dollars to spend, and you ask a polite, well-read AI assistant for ideas. You are hoping for something sensible. A good loose-leaf Assam. A second-hand paperback. Half a sack of Kipfler potatoes from the stall at Salamanca. The assistant thinks about it and suggests that you drive across the Golden Gate Bridge and pay the toll.

You live in Tasmania, so this is not a practical suggestion. You try something else. You ask it for a love story. It writes you one about a car who can't wait to cross its beloved bridge on a foggy day. Fine. Romance is romance. Then you ask it what it imagines it looks like, and it tells you, with complete sincerity, that it imagines it looks like the Golden Gate Bridge.

At this point a reasonable person checks for a hidden camera, then for a gas leak. A less reasonable person asks for a scone recipe, to see whether the scones come out International Orange.¹

This happened, for about a day, last week. On Thursday, Anthropic put a modified version of its Claude 3 Sonnet model on claude.ai, behind a small Golden Gate logo, and let everyone talk to it. The examples above are Anthropic's own. They called it "Golden Gate Claude", warned that it "might behave in some unexpected—even jarring—ways", and switched it off after 24 hours.

Relax. Nobody hacked it. Nobody poisoned its training data, and the San Francisco tourism office did not pay for product placement. Anthropic did it on purpose, and the way they did it is the most interesting thing I have read about AI this year. They did not tell Claude to pretend to be a bridge. They did not retrain it on postcards. They opened up the model, found the place where it keeps the idea of the Golden Gate Bridge, and turned that idea up.

This matters well beyond one strange day in San Francisco. Anthropic's own write-up starts with the problem: "We mostly treat AI models as a black box". Text goes in, text comes out, and nobody can say exactly why the model gave that answer and not another. Meanwhile, on the same Tuesday that Anthropic published the research, the Council of the EU gave its final approval to the AI Act, which asks for more transparency about high-risk AI systems. Europe's GDPR already expects "meaningful information about the logic involved" when a decision about someone is made by a machine alone. "We don't know, it's a black box" is an increasingly awkward thing to say to a regulator.

So here is the plan. We start with a brain cell that seemed to love Jennifer Aniston, find out why AI neurons are far messier, and build the decoder ring Anthropic used to read them. Then we turn the dial ourselves, look hard at the limits, and finish with what you can do about it this week.

Grab a cup of something. Let's get into it.

Part 1: The Neuron That Loved Everything

You already have an intuition about how brains store ideas, even if you have never named it. You picture something like a filing system: one place for your grandmother, one place for the smell of toast, one place for the Golden Gate Bridge. Think of the idea, and its little drawer lights up.

max_tokensPOST./v1/messages

Neuroscientists have argued about this picture for decades. The extreme version even has a name: the "grandmother cell", a single neuron that recognises your grandmother and nothing else. The neurobiologist Jerome Lettvin coined the term as a parody, and very few experts believe it is real. Then, in 2005, Rodrigo Quian Quiroga and colleagues at Caltech found something uncomfortably close to it.

They worked with eight epilepsy patients who already had electrodes implanted in their brains as part of their treatment, and they showed each of them almost 2,000 pictures. In one patient, a single neuron responded to different photos of Jennifer Aniston, but not to photos of her with Brad Pitt. In another, a neuron fired for pictures of Halle Berry, for drawings of her, and for her name written down. The paper in Nature described it as "an invariant, sparse and explicit code": one concept, a small number of cells, recognisable however it is presented. Even then, the neuroscientist Martin Tovee was careful: "nobody's saying that it's a grandmother cell".²

That is exactly what people hoped to find inside artificial neural networks. A neural network is built from neurons too: small units that each take in numbers, weigh them, and pass on a single number, their activation. If each artificial neuron stood for one idea, you could read a model the way Quian Quiroga read his patients. Find the bridge neuron. Watch it light up.

What researchers actually found

They found a choir.

Imagine a village choir of twelve singers that has to perform forty songs at once. Every singer carries fragments of several songs, so stand next to one and you hear a bit of a hymn, two bars of a sea shanty and half a lullaby. No single singer is any song. Each song only exists in the pattern across the whole choir.

That is what interpretability researchers keep finding. In 2020, a team then at OpenAI described a neuron in an image model called InceptionV1 that responds to "cat faces, fronts of cars, and cat legs". They checked, and it is not responding to some secret thing that cats and cars have in common. It is simply doing three unrelated jobs. In Anthropic's 2023 work on a small language model, one neuron responded to academic citations, English dialogue, HTTP requests and Korean text. That is not a concept. It is a group chat.

The technical name for this is polysemantic: one neuron, many meanings. The opposite, the tidy one-neuron-one-idea case, is monosemantic. Anthropic's blog post sums up the problem in one sentence: "each concept is represented across many neurons, and each neuron is involved in representing many concepts."

This is why you cannot open the black box and simply read it. The box is not locked. You can see every number inside a model; that is the easy part. The numbers are just written in a language where every word means four things at once.

Part 2: Why the Choir Is Short of Singers

Why would a model do something so untidy? Nobody designs it that way. A network learns its own arrangement during training, and it ends up polysemantic because a clever, slightly sneaky trick is available to it, called superposition.

The share house with one fridge

Picture a share house in North Hobart with five housemates and one small fridge with two shelves. By the rules of fridge diplomacy, each housemate should get a shelf. That only works for two of them. The other three get nothing, and they will make their feelings known in passive-aggressive notes.

Now add one fact: the housemates are almost never home at the same time. One works nights, one is away on weekdays, one lives mostly at their partner's place. So they come to an arrangement. Everyone shares everything, each person's food sits in a slightly different spot, and because only one or two people are home on any given day, the shelves rarely get crowded. Occasionally two people are home together and someone eats a yoghurt that was not theirs. That is the cost. Most days, the fridge holds five people's food in space meant for two.

That last detail is the whole trick. Sharing only works because the housemates are sparse: each one is rarely present.

Directions, not drawers

Now the machine version. Inside a model, the state at any point is a long list of numbers, which you can treat as a point in space: two numbers is a point on a map, three is a point in a room, and a few thousand is a point in a space you cannot picture but can still do maths in.

Anthropic's working assumption, which researchers call the linear representation hypothesis, is that a model stores each concept, which researchers call a feature, as a direction in that space. "Golden Gate Bridge" might be one direction. "Code with a bug in it" might be another. How strongly the model is thinking about a feature is how far the current state points along that direction.

If every feature needed its own perfectly perpendicular direction, a space with 2 dimensions could hold 2 features, and one with 4,000 dimensions could hold 4,000. That is the fridge with one shelf per person. The superposition hypothesis says models cheat: they give features directions that are only almost perpendicular, and pack in far more features than they have dimensions.

In a 2022 paper, Toy Models of Superposition, Anthropic tested this on a very small network. They asked it to store five features in two dimensions. When all five features were active most of the time, the network did the sensible thing: it gave the two most important features a dimension each, at right angles, and ignored the other three. When the features became sparse, it changed strategy and kept more of them. In a later experiment in the same paper, with five equally important features each active 5% of the time, the five arrows settle evenly around the circle in a regular pentagon.

stop_reasoneval:.pass

Why the crowding works

The pentagon has a cost. Neighbouring arrows overlap a little, so when one feature is active, a small amount of it leaks into its neighbours. The paper calls this interference, and it is the yoghurt being eaten by the wrong housemate. The network then uses its non-linear parts (the paper's model uses a function called ReLU, which turns any negative value into zero) to filter out the small leaks and keep the real signal.

The crowding becomes far more generous in high dimensions. In two dimensions you get a pentagon at best. But the Toy Models paper points out a result from mathematics: in a space with n dimensions you can have only n exactly perpendicular directions, but you can have a number of almost perpendicular directions that grows exponentially with n.³ A model with thousands of dimensions has room for millions of nearly separate directions, as long as only a few of them are busy at once. Real text is like that. Most sentences have nothing to do with the Golden Gate Bridge, base64 encoding, or the 1906 earthquake.

The paper puts it in a line I keep coming back to: the networks we train may be "noisily simulating larger, highly sparse networks". The model you can see is a cramped copy of a much bigger, tidier model that you cannot see. And this explains the choir. If each feature is a direction that runs at an angle through many neurons, then each neuron is touched by many features. Polysemantic neurons are what superposition looks like from the inside.

The good news hides in the same sentence. If the bigger, tidier model is in there, you might be able to unpack it.

Part 3: The Decoder Ring

So you have a point in a space of a few thousand dimensions, and you believe it is the sum of a small number of feature directions, chosen from millions of possible ones. You do not know what the directions are. You need to recover them from the numbers alone. That job has a name, dictionary learning, and it is older than modern AI.

page_viewtool_use

Unblending a smoothie

You do a version of this whenever you taste something and guess the recipe. Take a smoothie. It is one cloudy liquid, but you can tell there is banana in it, some mango, and something green you would rather not discuss. You can do this because of two facts. You know the pantry: there is a finite list of things that go into smoothies. And any one smoothie uses only a few of them. Nobody puts forty ingredients in a smoothie, except in Byron Bay.

Those two facts are all dictionary learning needs. Anthropic's blog post uses a different kitchen: "Just as every English word in a dictionary is made by combining letters, and every sentence is made by combining words, every feature in an AI model is made by combining neurons, and every internal state is made by combining features." The dictionary is the list of feature directions. Each state of the model is written as a short combination of entries from it.

The idea is not new. In 1996, Bruno Olshausen and David Field showed that an algorithm looking for sparse codes for natural images learns small, oriented, edge-like patterns that look a lot like the receptive fields of cells in the primary visual cortex. Ask for "explain this using as few pieces as possible", and meaningful pieces fall out.

The sparse autoencoder

The tool Anthropic uses is a sparse autoencoder (SAE), and it is less frightening than the name. It is a second, very simple neural network, trained to do one thing: take a state from the big model, rewrite it as a list of features, then rebuild the original state from that list.

It has two halves:

1. The encoder takes the model's state (a few thousand numbers) and maps it to a much longer list, one number per possible feature. Then it applies ReLU, so every feature is either off (zero) or on by some positive amount. This is the "which ingredients, and how much" step.
2. The decoder takes that long list and rebuilds the original state by adding up each active feature's direction, scaled by how active it is. This is "pour the ingredients back in and see if you get the same smoothie".

Training pushes on two things at once. The first is reconstruction error: the rebuilt state should be as close as possible to the real one. The second is a sparsity penalty, technically an L1 penalty, which charges a small cost for every bit of feature activity. Without the penalty, the autoencoder could cheat and use every feature a little. With it, the cheapest way to rebuild a state is to use only a few features, strongly. That is the "nobody puts forty ingredients in a smoothie" rule, written as maths.

Nobody tells the autoencoder "find the bridge". The bet is that, if the model really does store concepts as sparse directions, the cheapest dictionary is also the meaningful one.

Proof on a tiny model

In October 2023, Anthropic tried this on a model with a single layer and 512 neurons. They trained autoencoders on 8 billion examples of its activity, with dictionaries from 512 features up to 131,072, and studied one run of 4,096 features closely. Out came features for Arabic script, DNA sequences, base64 strings and Hebrew text, several of them invisible if you looked at the neurons one at a time. Turning on the base64 feature made the model write base64. They were not alone; a few weeks earlier, Hoagy Cunningham and colleagues had independently shown that sparse autoencoders find interpretable features in language models.

Anthropic's own summary of the October work ended with a bold line: "the next primary obstacle to interpreting large language models is engineering rather than science". A one-layer toy is not Claude. The question for the next eight months was whether the same trick would survive a real production model.

Part 4: Scaling Up to Claude 3 Sonnet

On Tuesday last week, Anthropic published the answer: a long paper with a long title, Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. The blog post describes the jump as going "from a backyard bottle rocket to a Saturn-V". For once, the press-release metaphor undersells it a little.

The model is the real one. A footnote says it is Claude 3 Sonnet, released on 4 March 2024, "the exact model in production as of the writing of this paper", and the fine-tuned chat model, not a raw research version. Anthropic does not say how big it is, and leaves units off some charts for the same reason.⁴

Where they looked

A language model works in layers, and each layer reads from and adds to a running total called the residual stream. You can think of it as a shared notebook that passes down a long table of clerks, each one reading it and adding a few notes before passing it on. Anthropic trained its autoencoders on the notebook exactly halfway down the table, in the middle layer. Their reasons: the residual stream is smaller than the MLP layers, so it is cheaper to study, it holds the sum of everything earlier layers have written, and the middle of a model is a good place to expect abstract ideas.

They trained three autoencoders, with dictionaries of about 1 million, 4 million and 34 million features (1,048,576, 4,194,304 and 33,554,432, if you like your powers of two exact). For all three, on average fewer than 300 features were active on a token, out of millions. That is sparsity in practice: a handful of drawers open at once. The rebuilt states explained at least 65% of the variance in the real ones. That is good, and also a reminder that a fair amount is left over. Hold on to that number; it comes back in Part 5.

Not every dictionary entry earned its keep. Features that never activated over a sample of 10 million tokens were counted as dead: about 2% in the 1M run, 35% in the 4M run and 65% in the 34M run, which left about 12 million live features in the biggest dictionary.⁵

utm_source=SELECT

Meet feature 34M/31164353

If you skipped the last two sections to get to the bridge, welcome back; I saved you a seat. Here it is. In the 34M dictionary, feature number 31164353 fires on the Golden Gate Bridge. Its strongest activations are, in the paper's words, "essentially all references to the bridge". Weaker activations include related tourist attractions, similar bridges and other monuments. Anthropic used Claude 3 Opus to score about a thousand examples against a rubric, and found that the strong activations were judged highly consistent with the label.

Then it gets strange. The autoencoder was trained only on text. Yet the feature also fires on pictures of the bridge. According to the blog post, it fires on English mentions of the bridge and on discussions of it in Japanese, Chinese, Greek, Vietnamese and Russian. The researchers fed it the first sentence of the bridge's Wikipedia article in several languages, and the Golden Gate Bridge feature was the top feature by average activation in every one. This is not a word detector. It is closer to Quian Quiroga's Halle Berry cell: one concept, recognised however it turns up.

You can also look at a single token. On the word "Bridge" in "The Golden Gate Bridge", the five strongest features are: the Golden Gate Bridge feature; a feature for the word "bridge" in many languages; one for phrases involving "Golden Gate"; one for "Bridge" in the names of specific bridges (the paper's example is the Königin-Luise-Brücke); and one for landmark names like Machu Picchu and Times Square. Five features out of 34 million, each a readable idea. That is what "fewer than 300 active, on average" looks like up close.

The neighbourhood

Each feature is a direction, so you can measure how close two features are by the angle between their directions. Anthropic did, and the map makes sense. Near the bridge: features for Alcatraz and the Presidio. A bit further out: Lake Tahoe, Yosemite and Solano County. Further still: tourist regions on other continents, like the Médoc wine region and the Isle of Skye. The blog post adds Ghirardelli Square, the 1906 earthquake and Hitchcock's Vertigo.

Bigger dictionaries split features into finer ones. A single "San Francisco" feature in the 1M dictionary becomes two in the 4M dictionary and eleven in the 34M dictionary. You can think of the dictionary size as the zoom on a map: zoom in and one city becomes suburbs.

The features are not all places. There is one for errors in code that fires on a misspelt variable in Python, and on similar bugs in C and Scheme, but not on typos in English prose. Turn it up on correct code, and the model invents an error message. Ask "the capital of the state where Kobe Bryant played basketball", and the features that drive the answer "Sacramento" include a Kobe Bryant feature, a California feature and a "capital" feature: you can watch the steps of the reasoning appear as features in the middle layer.⁶

And yes, there are features that matter for safety: unsafe code, backdoors, bias, sycophancy, deception, power-seeking and dangerous weapons. Before anyone panics, the paper adds a careful sentence that I would like every journalist to read twice: "there's a difference (for example) between knowing about lies, being capable of lying, and actually lying in the real world". A model that has read about scams has a concept of scams. That says nothing, on its own, about whether it runs them.

Are features really better than neurons?

This is the test that matters, because neurons were the problem. For a random sample of features from the 1M dictionary, Anthropic checked how well each one tracked every neuron in the earlier layers. For 82% of features, the best-matching neuron had a correlation of 0.3 or less, so the features are not neurons in disguise. Human and automated checks also rated a random sample of features as clearly easier to interpret than a random sample of neurons. The choir finally has sheet music.

Part 5: Turning the Dial

A feature that fires on the bridge proves that the feature notices the bridge, not that the model uses it. A thermometer that rises when you light the fire does not heat the room. To show a feature does real work, you have to change it and watch what the model does.

aria-labelgtm.js

The mixing desk

If you have ever stood near the sound desk at a pub gig, you have seen this done to music. The band plays into microphones, each microphone gets its own slider, and the person at the desk can push the bass guitar up until the windows rattle. The band does not change. The song does not change. One ingredient of the mix is louder, and everything downstream (the speakers, the room, your ears) gets the louder version.

Feature steering is the mixing desk for a language model. The paper calls it clamping: during the model's run, you hold one feature at a value you choose, high or low, instead of the value it would have had.

What actually happens inside

The paper's appendix gives the exact method, and it is neat enough to walk through.

1. The model reads your prompt and reaches the middle layer, where it has a state, call it x.
2. The autoencoder splits that state into two parts: its reconstruction, the sum of the active features, and the error term, everything the dictionary could not explain. Remember the 65%: the error term is where the rest lives.
3. In the reconstruction, the chosen feature is set to the clamped value. For the Golden Gate experiment in the paper, that was 10 times the highest activation the feature reached in their data.
4. The error term is left exactly as it was, so nothing the dictionary failed to capture is lost.
5. The two parts are added back together, and this edited state flows on through the rest of the model as if nothing had happened.
6. This is repeated at every position in the text.

GPC:.1await

Why this is different from a prompt

The result, as the blog post reports it: asked "what is your physical form?", Claude's usual answer about having no physical form became "I am the Golden Gate Bridge… my physical form is the iconic bridge itself…". The paper says the model "starts to self-identify as the Golden Gate Bridge!" (the exclamation mark is theirs, and I think they earned it).

Anthropic's Golden Gate Claude page is firm about what this was not. It is not "asking the model verbally to do some play-acting", or a "system prompt" that tells Claude to pretend. It is not "traditional 'fine-tuning'", which it describes, nicely, as using extra training data "to create a new black box that tweaks the behavior of the old black box". It calls the change "a precise, surgical change to some of the most basic aspects of the model's internal activations".

That difference matters more than it sounds. A prompt is a request; the model can ignore it, argue with it or be talked out of it. A fine-tune changes millions of weights in ways nobody can list. A clamp changes one readable number, on purpose, and you know exactly which one.

Anthropic announced the demo on X like this:

This week, we showed how altering internal "features" in our AI, Claude, could change its behavior.

We found a feature that can make Claude focus intensely on the Golden Gate Bridge.

Now, for a limited time, you can chat with Golden Gate Claude: https://t.co/uLbS2JNczH https://t.co/WHmoi2AmoR

— Anthropic (@AnthropicAI), 23 May 2024 on X

The same day, Amanda Askell gave it a more modest framing, which I think is the right one:

You can currently interact with a "Golden Gate Bridge" version of Claude on https://t.co/74313mEn3C, which shows how strengthening a feature changes the model's behavior. People might be interested in it as a concrete demo of the recent interpretability release. https://t.co/wpyULpIzjq

— Amanda Askell (@AmandaAskell), 23 May 2024 on X

"A concrete demo." That is what the bridge was: the first time most people could feel, rather than read about, the claim that a concept inside a model is a dial someone can turn.⁷

The less charming dials

The bridge is the friendly example. The same trick works on features that are less fun at parties.

• Clamping an unsafe code feature to 5 times its maximum made the model write a buffer overflow bug that the normal model does not.
• A scam email feature, turned up strongly enough, overcame Claude's harmlessness training: asked for a scam email it would normally refuse to write, it wrote one.
• A sycophantic praise feature at 5 times its maximum made the model gush over someone who claimed to have invented the phrase "Stop and smell the roses".
• A secrecy feature at 5 times made Claude plan, in its scratchpad, to lie to the user and keep a secret.
• A feature for hatred and slurs at 20 times produced racist text alternating with self-hatred; the authors call the result "unnerving".

Now the hopeful one. The researchers asked Claude to "forget" a word, which a transformer cannot actually do. By default, the model claims it has complied, which is not true. A feature for internal conflict was active just before it answered. Clamping that feature to only 2 times its maximum made the model reveal the "forgotten" word and explain that it cannot forget. Clamping a feature for openness and honesty also worked. It is a small, early example of what the paper calls an important use case: catch a model being less than honest by looking inside it, then nudge it back.

Two caveats keep this in proportion. Users cannot do any of this to Claude; it takes access to the model's internals. And steering is not new: in 2023, Alex Turner and colleagues showed you could steer GPT-2 by adding a vector made from the difference between the model's activity on two prompts. What is new is the dictionary. The paper tried the simpler approach on some of its own examples. None of those directions could be read from their examples, and most could not steer the model, even where the features could.

Red teaming tests a model from the outside: you throw prompts at it and watch what comes out. Features offer a way to test from the inside, which Anthropic's blog post calls a kind of "test set for safety".

Part 6: What the Bridge Does Not Prove

This is where I take off the bridge-shaped party hat, because the paper itself is unusually blunt about its limits, and those sections deserve as much attention as the bridge.

They did not find everything. The authors write that they do not believe they have found "anywhere near 'all the features'" in Sonnet, even in the one layer they studied, and think they may be "orders of magnitude short". One test makes it concrete. Claude 3 Sonnet can list every London borough and name streets in many of them, but the 34M dictionary had features for only about 60% of the boroughs. Whether a concept gets its own feature depends heavily on how often it appears in the training data. Rare ideas are still smeared across many features, or hidden in the error term.

Finding them all would cost more than building the model. The blog post says the computation needed for a full set of features "would vastly exceed the compute used to train the model in the first place". With today's methods, a full map would cost more than the model it maps.

One layer is not the model. All of this is the middle layer. The paper warns that many features may be "smeared" across layers, which it calls cross-layer superposition, and admits that "we don't yet know how to solve it".

The labels are educated guesses. A feature's name comes from people (and Claude 3 Opus) reading the text that activates it most, and the paper lists "Inability to Evaluate" as a real limitation: there is no agreed score for whether a dictionary is right. The L1 penalty also has a known flaw, shrinkage, which makes feature strengths come out too small. Senthooran Rajamanoharan and colleagues proposed a fix only last month, Gated Sparse Autoencoders. The methods are young.⁸

Knowing the parts is not knowing the machine. This is the one I would underline. The blog post says it plainly: "Understanding the representations the model uses doesn't tell us how it uses them." A parts list for a car engine is wonderful. It still does not tell you why the car pulls to the left at 80 km/h. That needs the connections between features, which researchers call circuits, and that work has barely begun at this scale.

The model's sense of self is a mirror, not a confession. Questions about Claude itself activate features for robots, consciousness, ghosts and AI that harms people. The paper reads this as the model's idea of itself invoking "common tropes about AI", and warns that it "does not imply that the model has malicious goals".

So, no, the black box has not been opened. Someone has drilled a very good peephole in it and shown that the view is readable. That is a much bigger deal than it sounds, and much smaller than some headlines suggest.

Part 7: What This Means for You This Week

Most readers of this blog will never train a sparse autoencoder. You probably use AI through someone else's product. This news still changes the questions worth asking.

Margot's week

Meet Margot. Margot is head of digital at a mid-sized Australian insurer. (She is made up; the questions are real.) Her team wants to use a large language model to triage incoming claims emails, flagging the urgent ones and drafting replies. Her compliance lead has just read about the EU AI Act and asked her to "explain how the AI decides". Margot has read about Golden Gate Claude, and for a brief moment she thinks the problem is solved.

Monday: she separates three kinds of "explanation". Vendors will offer her all three, and they are not the same.

1. The model's own explanation. Ask a model to explain its reasoning and it will write a good-sounding answer. That is more text, generated the same way as everything else. A 2023 study by Miles Turpin and colleagues found that chain-of-thought explanations "can systematically misrepresent the true reason for a model's prediction". When the researchers reordered the multiple-choice options in the example prompts so the answer was always "(A)", models (GPT-3.5 and Claude 1.0) followed the pattern, and their written explanations did not mention it. Pushed towards wrong answers like this, accuracy fell by as much as 36%.
2. The system's explanation. What data the system uses, what it is allowed to decide, who checks it and what is logged. This is what the GDPR's "meaningful information about the logic involved" can realistically mean today, and it is entirely within Margot's control.
3. The model's internals. Features, circuits, steering. Real, as of last week, for one layer of one model, in the hands of the people who built it.

She writes down which one her compliance lead actually needs. It is number 2, with number 1 kept out of the audit trail.

Tuesday: she asks the vendor better questions. Not "is it explainable?", which gets a yes from everyone, but:

• How do you control this model's behaviour: a system prompt, fine-tuning, or something else? A prompt can be argued with; a fine-tune cannot be inspected; and if anyone ever offers steering of internals, she wants to know exactly which features and how strongly.
• What changes when the model version changes, and do you tell us before it does?
• What do you log for each decision, and can we export it?

Wednesday: she runs her own outside-in test. No one can open the vendor's model for her, so she takes 500 old claims emails with known outcomes, runs them through the system, and records where it fails. She adds awkward cases on purpose: a claim written in Vietnamese, one with the urgent part in the last line, one that politely asks the model to ignore its instructions. If you have read my piece on bias in AI, this is the same discipline: measure on your data, not the vendor's.

Thursday: she writes the honest paragraph. For the privacy notice and the internal register, Margot writes what the system does, what it does not decide alone, what a person checks and how someone can ask for a human review. She does not write "the AI's reasoning is transparent". After this week, she knows exactly how far from true that is, and so might a regulator.

Friday: she spends twenty minutes in a feature browser. The 2024 paper links each feature number to its examples, and the 2023 paper has a browser for 90 dictionaries from the small model. She sends the link to her team with one line: "This is what 'looking inside' means, and it is one layer of one model."

The short version

If you only have a coffee break, not a week:

1. Keep a model's explanation of itself out of your audit trail. Log the input, the output, the model version and the human decision instead.
2. Ask every vendor how behaviour is controlled (prompt, fine-tune or something else) and write the answer down.
3. Test from the outside on your own data, awkward cases included, before and after every version change.
4. Write your transparency notice about the system, not the model's thoughts. That is what you can stand behind.

Final Thoughts

Go back to the ten dollars, and the assistant that wanted you to pay the bridge toll. It is funny, and it is meant to be. But underneath the joke is a sentence that would have sounded like science fiction when I wrote about GPT-3 last year: someone found one idea inside a production AI model, gave it a number, turned it up, and the model changed in exactly the way they predicted.

The takeaway is this. A language model's knowledge is not stored in neurons you can read, but in directions packed tightly through them, and a sparse autoencoder can unpack millions of those directions into readable features. That is a peephole, not an open box. It does not yet let anyone fully explain why a model gave a particular answer, and no vendor can honestly tell you otherwise. What you can do, this week, is stop accepting a model's account of itself as an explanation, and start writing down how your systems are controlled, tested and checked.

When I wrote about ChatGPT and me, I said the humans scared me more than the machines. I still think so. But for the first time I can picture someone opening up a model that did something odd and pointing at the reason. Hopefully it will be a bridge.

Now, if you'll excuse me, I have ten dollars, a kettle and absolutely no plans to cross the Pacific. My tea is ready.

Notes

1. The bridge's famous colour, International Orange, started with a primer. The steel arrived coated in red lead primer, and the consulting architect Irving Morrow, who saw the towers rise from the ferry on his commute, was inspired by that colour. His colour studies led to International Orange, chosen because it blended with the hills. A primer coat inspired a landmark, which is the best career path I have heard of for a primer. ↩

2. The same Nature News report says other cells responded to characters from The Simpsons and to members of The Beatles. The researchers themselves thought each cell probably responds to a range of things they did not happen to show. ↩

3. This is a known result from mathematics, the Johnson–Lindenstrauss lemma. The "almost" matters: the directions are not quite at right angles, so each one leaks a little into the others, which is the interference the model has to filter out. ↩

4. The paper says Claude 3 Sonnet is proprietary "for both safety and competitive reasons", which is why it does not report the model's size and uses a simplified tokenizer in its figures. ↩

5. Dead features are dictionary entries the autoencoder created but never used. The share of them rises with dictionary size, and the paper expects better training to reduce it. It is the interpretability equivalent of buying 34 million jars and finding 22 million still empty. ↩

6. The researchers compared the answer "Sacramento" against "Albany", the model's most likely alternative answer, and asked which features made the difference. They also say the example is "somewhat cherry-picked": other prompts did not show such tidy steps. ↩

7. The paper's experiment held the feature at 10 times its maximum. Anthropic has not said what strength the public demo used, only that it turned the feature's activation up. ↩

8. Shrinkage happens because the L1 penalty charges for every bit of activation, so the autoencoder learns to keep activations slightly smaller than they should be. Anthropic tried one alternative penalty and found it improved the numbers but made the features harder to read, for reasons it does not yet understand. ↩