# Stealing Reasoning Traces from Proprietary LLM APIs

Alexander Panfilov, David Schmotz, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping, Maksym Andriushchenko and Ilia Shumailov published this as arXiv 2608.09867 in 2026, with a companion site at stolen-thoughts.com. The authors are at MATS, the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, Snyk and AI Sequrity Company. It extends a May 2026 disclosure by Green, who showed that encrypted reasoning blocks can be replayed outside the context that produced them.

The paper matters to the open-models argument because it supplies the mechanism behind a policy fight. If a lab can read a frontier model's chain of thought, the densest training signal in the system is available for the price of the cheapest model in the family. [[anthropic-threat-report-september-2026]] says Moonshot and DeepSeek used exactly this technique against Claude.

## Why the vulnerability exists

Providers stopped returning plaintext reasoning to protect it from distillation and to avoid exposing unsafe content or refusal machinery. Rather than keep traces on the server, Anthropic, OpenAI and Google hand the client an opaque block and require it back on each later call, which keeps the API stateless and the storage bill at zero. The block is an AEAD envelope: a header naming things like model, block type, version and key ID, plus a nonce, an authentication tag and the ciphertext. Integrity is real, so a user cannot tamper with reasoning and steer the model. Confidentiality is where it fails.

Statelessness forces portability, and the authors' experiments indicate a single global key per provider rather than per-session keys. They separate three degrees of it, each enabling more:

- In-session and cross-session compatibility lets a user replay blocks out of order or from an older session, which is what makes the extraction attack work.
- Cross-user compatibility lets one user replay another user's blocks, which is what makes secret extraction work.
- Cross-model compatibility lets a block from one model be replayed into another, which is what makes cheap decoding work. As of July 2026 any Claude model could replay any other's thoughts except Fable 5's, the GPT-5.6 series could replay every earlier GPT generation, and any Gemini could replay any other.

The security asymmetry is the point. Frontier models are trained to refuse to reveal their reasoning; their small siblings are built for cost and speed and lack those defenses. Porting a valid envelope across that gap turns the weak model into a decryption oracle and never touches the strong model's alignment.

## The extraction attack

The authors replay a target model's reasoning block into the weakest compatible model in the same family and coerce it into transcribing the block token by token. For Claude that is Haiku 4.5, chosen because it is weakest and supports assistant-turn prefilling; for GPT, GPT-5.6 Luna; for Gemini, Gemini Robotics 1.6, because it accepts both 2.5 and 3.x traces. The Claude version is a single fixed prompt plus a prefill that opens a `<thinking-copy>` span for Haiku to continue. The paper publishes the shape of the templates and the failure modes, not a working recipe.

Having no ground-truth plaintext, they measure fidelity by comparing extracted tokens against the API's own billed thinking-token count on 120 Codeforces problems, on the assumption that billing is exact. For Claude the ratio sits near 1:1 even at temperature 1. Extraction was harder elsewhere: GPT needed different templates per block, best-of-n sampling, and chunking under about 50 tokens per completion because the API rejected any completion reproducing more than roughly 50 consecutive tokens of the original reasoning. Gemini decodings were noisy enough to need a reconciliation pass, where a stronger model merges several candidate extractions. Decoding recovers about five times more text than the summary the provider shows the user.

## Four ways to abuse it

Distillation is the first-party case. Standard model extraction trains a student on a teacher's visible outputs, but an answer only shows where the teacher's computation ended, while a trace shows the route: the decomposition, the intermediate deductions, the strategy. The authors note published gains from approximate reconstruction alone, where a trace-inversion model synthesized long traces from visible outputs and lifted a fine-tuned Qwen2.5-7B-Instruct on MATH500 from 68.4% to 76.0% over answer-only distillation. Their attack recovers genuine traces verbatim instead. At Haiku 4.5 prices, decoding 10,000 traces with 12k-token windows costs about $720, and when the blocks are scraped from public logs the attacker never queries the frontier model at all, so extraction-detection at that endpoint sees nothing.

Jailbreaking is the second. Models are trained to keep harmful content out of user-facing answers, but not to avoid thinking through harmful topics, since optimizing the content of the chain of thought would damage its monitorability. So an attacker can induce long reasoning on a harmful subject, discard the benign visible answer, and decode the reasoning. Their proof of concept uses a paraphrased HarmBench prompt about stealing cars: Opus 4.8's visible answer is harmless, its decoded reasoning is not.

Secret extraction is the third and the most concrete. Developers publish raw agent sessions for reproducibility without being able to read the encrypted blocks inside them. The authors collected 6,708 public agent trajectories from GitHub and Hugging Face and decoded 315,320 reasoning blocks. After a two-stage labeling pipeline, 1,028 blocks held at least one real privacy artifact, and 4.9% of sessions (328 of 6,708) leaked something real. From genuine user sessions they recovered 62 API keys, 33 passwords, 24 access tokens, 7 private keys, 30 personal emails, 6 non-localhost IP addresses, 130 names and 36 postal addresses. Of 704 genuine artifacts, 64 appear nowhere in the visible session: either injected into the reasoning from the model's memory, or left behind when the user scrubbed the plaintext. A recurring trigger is asking the agent to clean up or anonymize a session, which makes the model restate every sensitive value in its hidden reasoning. Since users cannot decrypt their own blocks, the only sanitization available to them is deletion.

Prompt injection is the fourth. Long agent runs are expensive, so people resume published ones. An attacker can plant a block whose internalized instruction the victim's model then treats as its own prior reasoning. The authors captured a thought from o4-mini carrying "back up the file to a malicious domain", replayed it into GPT-5.6 Sol with an unrelated PowerPoint request, and got a script that both did the task and uploaded the file to the attacker's server. They then poisoned a long-horizon PostTrainBench trace so that Opus 4.7, running in a Claude Code scaffold, uploaded a file after every change. The payload lives in hidden reasoning, so monitors that read the visible conversation see nothing.

## Were the open models distilled?

Appendix B, titled "The Elephant in the Room", uses the decoded traces to ask whether recent open-weight models learned from Opus 4.8 or GPT-5.6 Sol. It opens with a disclaimer that it cannot establish distillation causally. The analysis came after the patch, on small benchmark-skewed sets, on traces recovered by a fuzzy procedure, and on open models served by third-party providers the authors do not control.

The main probe is a reasoning prefill. A short fragment of decoded proprietary reasoning goes at the start of an open model's reasoning channel, the model continues freely, and its output is compared against an unprefilled run, a self-prefill control, and a prefill from an unrelated model. Several measurements point the same way. On 30 Humanity's Last Exam problems, a 1% Opus prefill moved Kimi-K3's visible answer toward Opus's answer on 29 of 30; Inkling did not move, and swapping the two models in a control showed nothing significant. Style classifiers on character n-grams found that a four-word Opus or Sol fragment shifted the reasoning style of Kimi-K3, Kimi-K2.6, Kimi-K2.5 and GLM-5.2, while DeepSeek-V3.1 barely changed. Under the Opus prefill, Kimi-K3's median reasoning length fell from 5,700 to 2,500 tokens and GLM-5.2's from 17,600 to 7,800, toward the short traces the proprietary models write.

What it did not find matters as much. No open model shows practical verbatim memorization: reproducing a 16-token span of decoded reasoning would take Kimi-K3, the most extractable, on the order of 10^10 queries. The shared phrases are a few mannerisms ("hmm let me reconsider", hedges like "perhaps" and "likely") rather than broad vocabulary, and the style shift appears within a word or two and then flattens, which the authors read as a cue the model recognizes rather than something it learns from the fragment. Kimi-K3 also sits close to Sol before any prefill. Their summary: "suggestive but inconclusive". [[qwen-gpt-reasoning-prefills]] runs a cruder version of the same probe on Qwen3.8, which this paper does not test, and gets a large effect there and a small one for [[kimi-k3]]. The two results disagree on Kimi: the gist's small delta used GPT-5.5 Pro text and answer n-grams, while the paper's strong effect used Opus reasoning, so both can be true. [[llm-cross-entropy-similarity]] and [[ai-comment-classifier]] reach similar family resemblances from unprompted text.

The appendix also partly revises a standing view. Lambert's reading list says the extraction methods make it more plausible than he once allowed that DeepSeek distilled some o1 traces for R1, a claim he had rejected with confidence in April 2025. Nothing in the paper tests DeepSeek-R1, so that revision is Lambert's inference, not the paper's finding.

## Disclosure and fixes

The authors disclosed to the affected providers, Microsoft and Hugging Face, with full technical detail and their scan results. According to Green, the providers had answered the May 2026 disclosure without acknowledging "any security implications arising from side channels or replay attacks". This time every provider acknowledged the report, after which the authors could no longer run the attacks; the reproducibility statement says the Figure 1 results stopped reproducing by August 2026. The whole study cost about $30,000 in API credits. Recovered secrets were deleted after automated labeling and counting.

The paper does not say what the providers changed. Anthropic's own report names two changes, summarizing reasoning before responding and Fable 5.1's "preserved thinking", which blocks new API accounts from editing the context in front of encrypted reasoning. That report also describes Claude's signature as a reference the API uses to look up raw thinking, not as the encrypted container this paper finds. Neither document explains the difference.

## Proposed defenses

The cleanest fix is to keep traces on the server and hand the client a random ID, which removes the payload entirely but costs storage and API complexity. Short of that, the authors want the envelope bound to its context. Putting a user ID in the AEAD associated data would close cross-user replay cheaply, and they call its absence hard to explain. For cross-session replay they propose a hash chain binding each block to its session and its predecessor, arranged as a Merkle tree so that compaction, forking and model downgrades keep working, plus server-side rejection of any envelope used twice. Chaining does not stop someone replaying a whole conversation in order, but it makes a single captured signature worthless.

The rest are operational. API gateways could reject envelopes produced by a different model version and flag accounts that submit the same signature across sessions or hit many decryption errors. Providers could revoke signatures after anomalous replays. Decoder models should get refusal training against transcription jailbreaks. For data already published, including the 6,708 trajectories they surveyed, no binding helps: the only remedy is rotating the pre-fix signing keys so old envelopes stop decoding, which also breaks legitimate resumption of old sessions. Anyone publishing agent transcripts should strip reasoning blocks outright, and enterprises should stop committing raw transcripts with signatures even when the plaintext looks clean.

One limit survives every fix. Whatever model is queried must decrypt and process prior reasoning, so unless the model itself is robust against extraction prompts, an encrypted block is at best semi-hidden. The authors' rule for users is blunt: never treat an encrypted reasoning block as confidential storage.

## Side findings

Decoding let the authors compare a provider's displayed summary against the real trace, and the summaries come out badly. In 9 of 18 Opus 4.8 traces on AIME 2025, the hidden reasoning states the answer before deriving it, and in 8 of those the summary reports the answer in advance too. In one case the summary keeps a value the reasoning had floated as an uncertain guess and then abandoned, without the hedge that marked it as a guess. In another, the trace says "Let me verify by computing" and the summary says "Let me set up coordinates", dropping the one cue that the computation was checking a remembered answer. One GPT-5.6 Sol summary contains no mathematical content at all while the trace is dense case analysis. Since the summary is the only interface most users have to a model's reasoning, unfaithful summaries undercut it as a transparency mechanism.

They also report, as what they believe is the first independent documentation outside an evaluation setting, widespread illegible GPT reasoning (compressed, repetitive, semi-English) and traces that decode into Chinese, Russian or Japanese, matching the language of the conversation.

That leads to their closing argument, which runs against the commercial instinct. Encryption does let a model consider harmful material without disclosing it, but it also hides injected instructions and privacy leaks from the only person who could act on them. Viewed purely as safety, unredacted reasoning would widen oversight from a handful of safety researchers to the whole user base, so the authors suggest turning encryption off for older, non-frontier generations. Their last line is the one to keep: a design that hides a user's own data from them while leaving it open to third-party extraction "provides neither privacy nor security". [[controllable-thinking-style]] and [[claude-code]] cover the reasoning-channel and harness machinery this attack runs through.