Anthropic finds its own interpretability tools don't help predict model behavior
A rigorous internal study shows activation-reading tools used in real safety evaluations give AI agents no measurable edge over just reading the transcript.
Original cover art, generated for this story. THE VISSION does not republish third-party press imagery.
- Anthropic published a study on August 21 testing whether interpretability tools — activation oracles, natural-language autoencoders and sparse autoencoders — help an AI agent predict how a model's behavior changes under counterfactual prompt edits.
- None of the three tools beat a baseline agent that simply read the transcript, across two target models and three predictor model families, according to the paper.
- The result held across hyperparameter sweeps and manual attempts to get better use out of the tools, the authors wrote.
Anthropic published research on August 21 testing a specific, practical question about the interpretability tools its own safety team uses: do they actually help predict how a language model's behavior will change under a counterfactual prompt edit, compared with an AI agent that just reads the conversation transcript? The paper, titled "Would This Change Your Answer? Evaluating Explanations of LLM Behavior in the Wild with Counterfactual Experiments," introduces a pipeline called CHIVE that runs thousands of these experiments automatically.
The answer, across two target models and three predictor model families, was no. "None of the three tools beats the transcript-only baseline," the authors wrote, testing activation oracles, natural-language autoencoders and sparse autoencoders — the last a technique Anthropic itself pioneered and uses in its own model system-card evaluations. The negative result held up against hyperparameter sweeps and deliberate manual attempts to get better performance out of the tools, according to the paper.
The paper does note one instance where a natural-language autoencoder helped: it correctly read out that a model was "likely generating a random integer between min and max," letting a predictor agent correctly anticipate an error caused by misleadingly named parameters — but the authors describe this as an exception rather than the rule, noting that tool outputs "almost never explicitly state the causal relationship" an agent needs to make a correct prediction.
Any team currently justifying an investment in activation-reading interpretability tooling — on the premise that it will make a model's behavior more predictable in practice — now has a direct, controlled result suggesting that premise doesn't hold, at least for the counterfactual-prediction task these tools are often pitched for. That's a more useful data point for prioritizing safety-research budget than another capability benchmark would be.