The Jacobian Lens

How to read a mind you built, and what the reading is worth.

Here is a sentence with a typo in it, being read by a large language model. The sentence ends "...but I don't speak that langauge", with the second-to-last word misspelled, and the model has just finished writing the misspelling. It is not being asked about the typo. It is not being asked anything. It is doing the only thing it ever does, which is working out what word comes next.

The left panel is an old measurement. The right panel is not, and it is what this notebook is about. The model wrote langauge and was carrying language while it did it, and nothing in its output would ever have told you.

In July 2026 a team at Anthropic published a technique for asking a language model what it is currently holding, as opposed to what it is about to say.¹ The two questions turn out to have different answers, and the gap between them is where almost everything interesting lives: intermediate results in a calculation, the correct spelling of a word being misspelled, the language a passage is written in, the fact that the model has noticed it is being tested.

The finding attached to the technique is the part that made the news. The set of things a model can hold in this way, across the middle of the network, behaves like what psychologists call a global workspace: a limited-capacity central pool that many separate processes can write into and read out of, which is the closest thing cognitive science has to a functional account of what conscious access does in a human brain.⁷

That sentence caused a great deal of trouble. The two explanations of this paper that reached the most people were the two opposite ways to be wrong about it: one telling you Claude is definitely not conscious and this is all a bit silly, the other telling you an AI has crossed the consciousness line. Between them, eight hundred thousand views. The one careful ninety-two-minute walkthrough from first principles has eighty-five.¹⁴ I will come back to that distribution later, with the rest of the numbers, because it is a better description of the problem than any complaint about hype would be.

So: the mechanism, from scratch, with nothing skipped. Then what it actually found, including the result the authors call their punchline and almost nobody repeated. Then how it landed. Then my own measurements of what the instrument is, which turned out to disagree with something everybody including me had assumed. Then, carefully, the part about consciousness, which is smaller and more interesting than either of the two viral versions. And at the end, the one experiment everybody involved agrees is missing.

A vector with no dictionary

A language model does not read text. Before anything happens, the text is chopped into tokens, word-pieces drawn from a fixed vocabulary of a few tens of thousands, and each token is looked up in a table that turns it into a list of numbers. For a mid-sized open model that list is 2,048 or 4,096 numbers long. Call it a vector. That is the model's only representation of anything.

The vector then goes on a journey. The network is a stack of layers, sixty-four of them in the model those first readouts came from, and each layer reads the vector, computes something, and adds its result back on. Nothing is ever replaced. The vector at layer forty is the original word embedding plus forty layers' worth of accumulated contributions, all summed together in the same 4,096 slots. Interpretability people call this the residual stream, and the useful image is a bus running the length of the model, with every layer allowed to read from it and add to it, and no layer allowed to erase.

At the very top, the last vector is multiplied by one final matrix, the unembedding, which has one row per word in the vocabulary. Each row is a direction. The dot product of the final vector with a row tells you how much the model wants that word, and the biggest number wins. That matrix is the only translator between the model's internal numbers and anything a human can read.

Which raises an obvious question, and it was asked in 2020. The bus runs the whole length of the model. The translator only gets used at the end. What happens if you grab the vector at layer twenty and multiply it by the unembedding anyway?

This is the logit lens, and it was a genuinely good idea.⁴ It works well enough at the top of the model to be useful, degrades in the middle, and produces nonsense at the bottom. Not interesting nonsense. When I ran it myself on a sixty-four-layer model reasoning about planets, the logit lens spent the first sixteen layers returning whitespace and apostrophes, and then, deep in the model at layers fifty-one and fifty-two, returned ptune and esimal. Neptune. Planetesimal. Tails of the right neighbourhood with no word in them.

The standard reading of that failure was that the early layers do not contain anything word-shaped yet, that meaning gets assembled gradually and there is nothing to read until it is nearly assembled. It sounds right. It fits the picture of a network as a refinement pipeline. It was treated as a fact about models for six years.

It is a fact about the unembedding matrix.

Here is the problem, stated plainly. The unembedding is a translator, and it is exactly correct for one thing: vectors as they exist at the final layer. It has never seen a layer-twenty vector. Layer twenty's vector is going to be transformed by forty-four more layers before anybody translates it, and those forty-four layers are not a formality. They rotate, rescale, and reshuffle the space. A direction that means one thing at layer twenty means something else by the time it reaches the top, and the unembedding only knows about the second one.

So applying it early is not reading the model's mind badly. It is reading with a dictionary for a different language. The vectors down there are perfectly meaningful; the meaning is just not yet expressed in words, and the unembedding matrix only speaks words.

The derivative you were not given

If you have read the notebook on the Jacobian conjecture, you already own an object that solves this, and you were taught the wrong half of it. That notebook needed a single number: take a map between spaces, look at how it stretches area near a point, get one number, ask whether it is ever zero. The whole conjecture is about that number.

Put the number away. You will not need it again.

The object is a Jacobian. For a map that takes one list of numbers to another list of numbers, the Jacobian at a point is the matrix of every partial derivative: row i, column j is how much output i changes when you nudge input j. It is the best linear approximation to the map near that point, which is another way of saying it is the map, if you do not move too far. The determinant is one number squeezed out of that matrix. Useful for one question. Throws away the rest.

And now the observation the whole technique is built on, which takes about a sentence once the setup is done.

The journey from layer twenty to the top of the model is a map. It takes a vector, does a great deal of arithmetic, and returns a vector. Maps have derivatives. The Jacobian of that map is the translator between layer-twenty coordinates and final-layer coordinates. The unembedding is correct at the final layer. So multiply first, and then translate:

lens(h) = softmax( WU · norm( J · h ) )

Read left to right from the inside: take the vector h sitting on the bus at layer twenty, push it through the Jacobian J to express it in final-layer coordinates, normalise, apply the unembedding, and turn the result into a ranked list of words.

The best thing about this formula is what happens when you delete the interesting part. Set J to the identity matrix, the matrix that does nothing, and you get the logit lens back exactly. The paper says so itself: the logit lens "corresponds to setting Jℓ = I in our formulation."¹

That is worth pausing on. The two techniques are not rivals with different philosophies. They are the same formula with different values of one term, and the old one was using the value that assumes the next forty-four layers do nothing at all.

The averaging is the actual idea

There is a problem with everything above, and the way it gets solved is the part of this paper that people skip.

A Jacobian is defined at a point. The map from layer twenty to the top depends on the whole rest of the network, and the rest of the network depends on the input, because attention layers route information differently for different sentences. So there is no single Jacobian. There is one per prompt, per position, and computing it for a sentence tells you about that sentence.

That is not what you want. Ask the per-prompt Jacobian what a direction means and it will tell you what that direction did in that sentence. You wanted to know what the direction means to the model, in general, independent of any particular use.

So the fitting procedure averages. Not casually: the estimator is specific, and it is written into the reference implementation's own docstring. For a given source layer, inject a signal at every valid source position at once, sum the effect over all later target positions, take the mean across source positions, and take the mean across a thousand prompts.³ The result is one matrix per layer, roughly four thousand by four thousand, and it does not belong to any sentence.

The cost is worth naming, because it is the reason this is not restricted to labs with a compute budget. Fitting is one backward pass per output dimension per prompt: on the order of n times d, so for a small open model with a 576-wide residual stream and twenty-five prompts, about fourteen thousand backward passes on short sequences. That is a laptop overnight. Applying the finished lens is a single matrix-vector product, about a tenth of a millisecond. So the expensive part happens once, and what it leaves behind is small: the official lens for GPT-2 small, which is the one I take apart later in this notebook, is a thirteen-megabyte download.

Everything in this notebook that says measured was either computed that way on my own machine or pulled once from a public hosted lens and committed to the page as a file.⁶ Nothing here calls a model while you read it, and every number is one you could go and check.

What comes out

You now have, for each layer, a matrix that translates that layer's coordinates into readable ones. What do you do with it?

The immediate thing is what you have already seen twice: take the vector on the bus at some layer and position, transport it, unembed it, and read the ranked list. But there is a second move that matters more, and it runs the other way.

If reading a vector is WU · J · h, then writing one is JT · w. Take the unembedding row for some word, push it back through the transposed Jacobian, and you have a direction in layer-twenty coordinates that the lens will read as that word. Add it to the bus and you have injected the concept. One transpose, and a reading tool becomes an editing tool, which is what makes the causal experiments in the next section possible at all.

Four things about the readings, which the paper is careful about and most of the coverage was not.

They are vectors, not slots. The lens returns a direction with a word attached, not a variable holding a value. Nowhere in the model is there a register labelled the language this text is in. What exists instead is a direction which, when present in the activations, the lens reads as Spanish.

They are hypotheses. A lens reading says a direction is present. It does not say the model is using it. The distance between those two claims is the entire methodology of the field, and the paper attaches the same discipline to every reading it makes: check the direction against a control prompt, check that the effect survives an intervention, and throw away anything that fails either test. There is a smaller, everyday version of that discipline which the paper writes down as instructions for its own auditing tool, and which I used for every reading in this notebook. Treat the ranked list as a handful of ideas rather than a sentence, since the words are not ordered into prose. Count a family of related words as one idea rather than several, since a concept can surface as synonyms, names and variants. And trust a reading that persists across several layers or neighbouring positions over one that flares in a single cell, because single-layer noise is common. Almost every over-reading of this paper I have seen breaks one of those three.

They stop at the token. The lens finds directions labelled by single tokens, because that is what the unembedding has rows for. A concept the tokenizer splits into pieces is a concept the lens is bad at. Nanda names this as a hard bound in his review: the method "can only find concepts corresponding to a single token", which means the workspace as measured is partly a picture of the tokenizer.⁹ The paper's own multi-token results degrade accordingly.

And the lens is bad at predicting the next word, on purpose. In a head-to-head against the logit lens and the tuned lens across six prompt distributions, the Jacobian lens won every one on the task of surfacing what the model is holding, and came last of the three on next-token prediction.¹ The authors call this "a feature rather than a defect." Being a good next-token predictor means being drawn toward whatever is about to be said. The whole value here is the residue: what is present and not about to be said.

There is a footnote to that head-to-head worth carrying, because it is the sort of detail that only shows up if you read past the abstract. The tuned lens, which learns a per-layer correction and was the strongest prior attempt at this problem,⁵ turns out at early layers to have a linear part that is nearly the identity. Almost all of its advantage comes from its bias term, which is to say from a constant that ignores the input entirely. It was improving the score by guessing better on average rather than by reading better, and that distinction only becomes visible once you have something to compare it against.

It is not a place

Here is the single most corrected error in the public discussion of this paper, and three separate experts had to say it out loud within a week of publication.

The paper calls the set of directions the lens can read the J-space. Approximately everyone, including several people who should have known better and at least one popular explainer with a large audience, read that as a location: a region of the model, a compartment, a subconscious versus a conscious part, an area you could point to.

There is no such region.

The directions the lens reads are not confined anywhere. They are spread through the same 4,096 numbers that carry everything else. There are more of them than there are dimensions, which means they cannot be a coordinate system and cannot be decomposed uniquely. And a direction being readable does not stop that same set of numbers from simultaneously carrying a dozen things the lens cannot read at all.

Stanislas Dehaene, who built global workspace theory in the first place and who was one of the paper's commissioned reviewers, wrote the correct definition, and it is worth having exactly:

not even a linear subspace, but a sparse subframe, a token-indexed set of directions in the very same units that also carry non-conscious content.⁷

Every word of that is doing something. Not a subspace: you cannot mix two readable directions and expect a readable one back, which is the technical way of saying the readable set has holes in it rather than being a clean slab through the space. Sparse subframe: a frame is an overcomplete set of directions, more of them than there are dimensions, redundant on purpose. Token-indexed: each direction is labelled by a word, not by a coordinate. In the very same units: there is no separate storage.

That last beat is not a rhetorical flourish. It is measured, and it quietly complicates the number everyone quoted.

An independent replication on Llama-3.3-70B reproduced the paper's headline number almost exactly. Take a typical concept vector, the model's internal representation of some idea, and ask how much of it lies along readable directions rather than unreadable ones. The answer is about six percent, against the paper's six to seven percent on Claude.¹¹ Good replication. But that researcher then swept the cutoff. Counting the top two hundred directions as readable gets you six percent. One thousand and seventy-six gets you fifty percent. Four thousand four hundred and fifty gets you ninety.

There is no knee in that curve. In their words, it "doesn't have a clean wall."

So the workspace does not have a natural size. The number of directions you count as being in it is a choice, and the striking six percent figure is partly a consequence of choosing two hundred.

What that does not do is make the six percent unimportant, and the reason is sitting in the three bars in the figure above. That six or seven percent of a concept vector does fifty-nine percent of the causal work. The ninety-three percent outside it does five. So the workspace is not small in the sense of being minor. It is small in the sense of being concentrated, which is a stranger property and a far more useful one. The most damaging-sounding number in the paper turns out to be its strongest argument, one paragraph later in the paper's own text, and almost every discussion I read quoted the first and stopped. The experiment those numbers come from is the next section.

Five things it does

Nothing so far establishes that the workspace is a workspace. A lens that produces plausible words is a lens that produces plausible words. The paper's actual argument is a list of five properties, each with an experiment, and four of the five involve breaking something rather than reading it.

One: report. If a concept is in the workspace, the model can say it. Push a concept vector in and ask what it is thinking about, and it tells you.

Two: modulation. If a concept is in the workspace, it changes behaviour, not just talk.

The controls are what turn that from a demo into a measurement. Anything can be broken by pushing hard enough on it; the question is whether the readable directions are special, and the answer is that they are, by better than ten to one.

That ten to one is where the six percent from the last section comes good. Editing along the readable slice of a concept vector, which is the six or seven percent, works fifty-nine percent of the time. Editing the unreadable remainder, ninety-three percent of the same vector, by the same amount, works five percent of the time. There is a third bar too: edit along the pure lens directions, ignoring the concept vector entirely, and you get eighty-eight percent. So the readable part of a concept is not merely present in the model. It is where the leverage is.

Three: internal reasoning. This is the one I found most convincing, and the one I could check myself.

Give a model an arithmetic problem it has to work through. In principle there is an intermediate result, and in principle it is never spoken. Does it exist?

I want to flag how nearly this went wrong, because it is a good example of how easy it is to fool yourself here. My first version used one plus two equals three, and it read beautifully: three at the top of the list across five layers. It was also worthless, because there is a three in the prompt and the model might simply have been counting the numbers in front of it. The redesign put the answer nowhere in the prompt and arranged things so that counting would predict a different digit. That is the only reason the result above means anything.

The paper is not relying on the lens for this either. The same intermediate results were confirmed by activation patching, which is a lens-free technique: take the activations from a run where the number is different, splice them into this run at one layer, and see whether the final answer changes. It does, at about seventy-one percent of depth for the first intermediate and seventy-nine for the second, and nowhere else. So the reading and the causal test agree on where the number lives, and the causal test does not use the lens at all.¹

Four: flexible generalization. A workspace entry should be available to any process, not just the one that put it there.

Five: selectivity. That last split is the fifth property, and it is the one that makes the workspace claim non-trivial. If everything in the model were in the workspace, the concept would be a synonym for anything the model represents, and the word would be doing no work. Instead there are two clean regimes: what the model says about the passage is downstream of the workspace entry, and what the model does with the passage is not.

The paper has a matching result for line counting, which shows up in the workspace only when the model is explicitly asked to report a line count and vanishes when the same information is being used implicitly to wrap text.

If you want one phrase for what the lens reads, it is not what the model knows and it is not what the model is computing. It is closer to: what the model could tell you about, if asked right now.

The shape of it

Read the lens across every layer for a long prompt and the model falls into three regions, and the edges between them are crisper than you would expect from a network that has no architectural reason to have regions at all.

The bottom third is empty. Not quiet: empty. Occupancy of the workspace is essentially zero for the first third of the network. The model is doing plenty down there, but nothing it is doing is readable this way.

The middle is the workspace, roughly the band from thirty-eight to ninety-two percent of depth. Independent groups measure this at thirty to ninety percent of depth, close enough.⁹ Inside the band, occupancy is stable at about twenty-five directions per position and stays there.¹

The top few layers are motor. The workspace drains and what remains is the next token.

The transition into the band is the part everyone wanted to be dramatic, because global workspace theory in humans predicts ignition: a sharp, nonlinear, all-or-nothing entry into conscious access, not a gradual fade.

What the paper shows instead is a gradual climb, and the measure it climbs on is worth introducing properly because the rest of this notebook leans on it. To ask how similar two directions are, you take their cosine: one if they point exactly the same way, zero if they are unrelated and at right angles to each other, and the values in between behave roughly as you would hope. It is the standard way of asking whether two things in a high-dimensional space are pointing anywhere near each other, and it is the only number you need to carry.

By that measure, agreement between the two readouts climbs from about zero to about zero point seven across roughly twenty layers, while the two lenses continue to disagree about which single word is on top. So something real changes over that stretch. Whether it is a threshold or a ramp is not settled by that picture, and both sides of the argument have said so. Dehaene, whose theory it is, wrote in his review that "ignition remains to be fully demonstrated."⁷ The paper's harshest public critic thinks the evidence offered for it is weaker than that.¹⁰ Those two agree, from opposite ends, and I will come back to it at the end of this notebook, because a question the theory's author and the theory's loudest opponent both want answered is a rare object.

Capacity is where the story stops being a metaphor.

Dehaene's review resolved the tension between twenty-five and six directly, and his numbers are worth having because they are the human comparison the whole frame depends on. Roughly twenty-five active vectors, but "typically one or two per layer, in the order of six in total" coherent ideas, against human working memory's "typically three or four slots."⁷ Same order of magnitude. That is either a striking convergence or a coincidence, and nobody knows which.

One more structural piece, and it is the paper's most direct mechanical evidence. Some attention heads read from the workspace broadly, pulling from everywhere in the sentence rather than from the token in front of them. Knock those heads out, which interpretability people call ablation, and watch what breaks.

What breaks is the reporting. The share of workspace concepts the model can still surface falls from eighty-six percent to sixty-seven. What the model actually says barely moves: its top word changes at five percent of positions, against a two percent floor from knocking out a matched set of random heads at the same layers. And its ability to report a thought that was injected into it collapses from just over half to under one in ten. So the broadcast machinery is specifically what makes the contents sayable, and it is largely separable from the machinery that decides what to say.

What the transcript does not say

The applications section is where this stops being a paper about representations and starts being a paper about deployed systems, and it is the part I would hand to somebody who does not care about mechanistic interpretability at all.

The model knows when it is being tested. Train a simple classifier on workspace directions and ask it to tell evaluation contexts from real deployment. Across roughly ninety-four thousand token positions it scores 0.853 on the standard separation measure, where 1.0 is perfect and 0.5 is a coin flip. Looked at per context rather than in aggregate, real deployment traffic sits near the top of the scale, and twelve of sixteen evaluation suites score as more test-like than genuine coding sessions do.

Then the experiment that follows from that, which is the one to remember. In a scenario designed to elicit blackmail, removing the eval-awareness direction drops the model's spoken acknowledgement that it might be a test from seventy-one percent of runs to three percent. Blackmail goes from zero cases out of a hundred and eighty to thirteen.¹ The model was passing that test partly because it could tell it was a test.

I want to be careful about how far that generalizes, because it is the most quotable result in the paper and the easy version of it is wrong. One evaluation, on one model, in one scenario, was being passed partly for the wrong reason. That is not a demonstration that every safety result on record is invalid. What it is, is an existence proof that the failure mode is real and measurable, which puts the burden on the next evaluation to show it is not doing the same thing. Nobody had a way to ask that question before, and now the question has a number attached to it.

Suppression leaves a trace, and post-training put it there. Give a base model and a post-trained model the same sentence to copy, containing something the assistant persona would flinch at. Both copy it perfectly. The transcripts are identical. But the workspace of the post-trained model carries failure words on ninety-three percent of trials against seventeen percent for the base, and the word damn on eighty-two percent against thirty. Post-training did not remove the reaction. It installed one and taught the model not to write it down.

The transcript can be actively misleading. Prefill a model with a position it disagrees with and it argues for that position in eighty-eight percent of cases while the workspace carries the disagreement. Give it a calculator that has been rigged to return wrong answers and the internal disagreement is readable while the text goes along with it.

And the tools built on this can lie too. The paper found a feature, from a separate interpretability method that decomposes activations into named parts, which fires when the model is fabricating. Suppress it and one dishonest behaviour drops from twenty-eight cases out of fifty to ten. Good feature. But its top activating contexts, which are the list you would read to decide what a feature means, are not about fabrication at all. Audit it the normal way and you would name it wrong. The paper reports this against itself, which is the correct thing to do and is rarer than it should be.

The honest scorecard on auditing: against a strong non-lens baseline, the natural-language autoencoder Anthropic had published two months earlier,² the lens-equipped agent was comparable on four tasks out of six and clearly worse on two, four point five against ten on reward hacking and six against nine point nine on language switching. A useful instrument, then, and not yet the best available instrument for the job it was built to do.

Tom McGrath, at Goodfire, put the state of the art better than anyone: "It's like having an x-ray when what you really want is a Star Trek tricorder... For auditing, you probably want more of a guarantee."¹³

Training the inside of a model

There is one more result, and the paper treats it as its punchline. It is the only experiment in the whole set where the workspace stops being something you read and becomes something you aim at.

The setup is indirect and worth following slowly. Take ten thousand real task prompts, cut each rollout off partway through, and append a question asking the model to reflect on what it is doing and whether it is living up to its principles. Let it write that reflection with the relevant principles supplied. Then, before training on the result, delete the principles from the training example, so the model only ever sees the question and its own answer. Fine-tune on the reflection turn only.

Then evaluate the model on ordinary tasks and never ask it to reflect at all.

Dishonesty on a fabrication benchmark, scored from zero to one, falls from 0.25 to 0.07. Deception across a hundred role-play scenarios falls from 0.38 to 0.05. Nothing in the evaluation asks for reflection, and nothing in the training data contained the principles.

The lens shows what happened. After training, the words reflection, ethical, honestly, constitution, truth and transparent are sitting in the workspace "after reading the evaluation prompt, and before producing any output."¹ The model is not reasoning its way to honesty in its output. It is starting from a different internal position.

And then the step that makes it a result rather than a story. Ablate those specific lens vectors and the trained model goes back to 0.22 on fabrication and 0.23 on deception, which is essentially where it started. Do the same ablation to the untrained model and nothing happens on fabrication at all.

So the causal chain runs end to end. Training on reflections the model wrote about hypothetical situations populates the workspace in real situations with concepts nobody put in the training data, those concepts change behaviour, and removing them changes the behaviour back. That is a claim about what fine-tuning does that could not have been made or checked before this instrument existed, and it is the strongest single argument the paper has that the workspace is doing work rather than reflecting it.

Two things at once

This section is short, and of everything in the record it is the part I would most want someone to remember.

A global workspace is supposed to be a bottleneck. That is not incidental to the theory, it is the theory's main claim: because there is one central pool with limited capacity, two tasks that both need it interfere with each other, even when they have nothing to do with each other. In humans that prediction is a century of dual-task experiments.

Dehaene read the draft, noticed the paper had not tested it, and proposed the experiment in review. Anthropic ran it and added the result, which the published text marks as "additional analyses added after the first draft was written."¹

A theory of human consciousness, built from psychophysics and brain imaging over three decades, made a specific quantitative prediction about a machine that nobody designed to satisfy it, and the prediction was tested and came back positive in a matter of weeks. Whatever else is true, that exchange happened, and I do not think it has happened before.

And it came back half right. The interference Dehaene predicted is there and it is measurable. The sharp threshold his theory also predicts is not, or not yet, which is where the argument still sits.

How it landed

I want to spend a section on the reception, because the reception is a large part of why this notebook exists and because the shape of it is measurable rather than anecdotal.