How Researchers Read AI's Inner Thoughts -- and Caught Chinese Models Copying
The same technique that reveals how AI thinks also reveals that some Chinese AI may be trained on US models -- a research paper with spycraft implications that turns model interpretability into a geopolitical intelligence tool.
Computer scientists recently discovered a way to extract the hidden "thinking" that frontier AI models perform as they work through complex problems. The findings provide some evidence that certain Chinese models may have been trained by distilling reasoning information from US models that was supposedly hidden.
The researchers have also demonstrated that the method could be used to recover personal information, like passwords and API keys, from a model's inner reasoning. OpenAI, Anthropic, and Google have since patched this vulnerability.
How the Mind-Reading Trick Works
Advanced AI models solve difficult problems by breaking them into constituent parts, analyzed in turn in a kind of artificial reasoning called "chain of thought." Companies keep a proprietary model's reasoning secret to prevent others from using it to train new models. But they typically send an encrypted version of that reasoning to a user's computer to offload some computation.
The researchers' attack exploits a subtle design choice: most AI companies offer related models of different sizes. Larger models are more capable but more expensive. Users may choose smaller, weaker models for certain tasks to lower costs.
Alexander Panfilov, a computer scientist at the University of Tubingen in Germany, and colleagues from the Max Planck Institute, the AI safety institute MATS Research, and the security company Snyk found that feeding encrypted reasoning traces to a smaller version of the same model can reveal the hidden reasoning inside. The smaller models have received less alignment training, meaning they are less likely to refuse to reveal their inner thoughts.
"All major frontier model providers we tested share this vulnerability," Panfilov says. "It can lead to personal information leakage, and it enables large-scale reasoning distillation attacks."
Florian Tramer, a computer scientist at ETH Zurich specializing in computer security, called the approach clever. "The idea of swapping out messages to a weaker model variant which has the same decryption key but weaker alignment is very cool," he says. "It's definitely becoming an issue."
What the Researchers Found
The team tested frontier models from OpenAI, Anthropic, and Google accessed via API. They fed 90 questions to each model and compared the resulting reasoning traces.
The striking result: Kimi K3, an open-weight Chinese model from Moonshot AI, produces reasoning that closely mirrors Claude Opus 4.8 and GPT 5.6 Sol for certain prompts. The researchers note that their work "cannot causally establish distillation" -- the similarities could have other explanations. But the pattern is pronounced enough to raise questions.
Two other open-weight models tested, China's DeepSeek and Inkling from the US company Thinking Machines, did not exhibit this kind of reasoning similarity with Claude Opus.
Moonshot AI and Z.ai did not respond to requests for comment.
The Distillation Debate
Distillation is a well-established technique for efficiently copying the capabilities of existing models onto new ones. It is especially common in the development of open-weight models. But it has become a geopolitical flashpoint.
In February, OpenAI told US lawmakers that DeepSeek seemed to have copied one of its models to build R1. In June, Anthropic told lawmakers that Alibaba had systematically distilled its models to build Qwen.
Mark Zuckerberg, CEO of Meta, argued this week that distillation "is an important principle of how the open source ecosystem works" and warned that restricting the practice would put the US at a disadvantage.
Kyle Miller, a researcher at the Center for Security and Emerging Technologies (CSET), says it is unclear how much distillation really helps China. "Nobody here in the US knows how much distillation is benefiting the Chinese labs," Miller says. "If you removed the ability for Chinese labs to distill, it's my view that it wouldn't dramatically change the competitive landscape."
Yarin Gal, a computer scientist at Oxford University, notes that distillation has helped AI advance more rapidly. "If it's the norm that everyone blocks everyone from doing distillation, then that also will have implications on the rate of progress," he says.
The Security Fix
Panfilov and his coauthors alerted OpenAI, Anthropic, and Google to the vulnerability last month. Each company has adjusted its API to mitigate the problem. While it is no longer possible to extract private information this way, Panfilov says some reasoning traces can still be uncovered using the same method. Fixing the distillation entirely would require a fundamental overhaul to the way these companies' APIs work.
"We value independent research on our models and have begun building short-term mitigations for the replay behaviors described in the report," says Michael Aciman, a spokesperson for Anthropic. He adds that the research did not involve recovering encryption keys, accessing Anthropic's infrastructure, or recovering personal data from its systems.
Google and OpenAI both declined to comment.
What It Means
The research opens a new front in the battle over model distillation. Companies and policymakers may introduce measures aimed at limiting the practice, but even so, AI models could continue to reveal their inner thinking in surprising ways.
The technique also raises uncomfortable questions about how much of the AI industry's rapid progress depends on copying rather than original innovation. If the best evidence for distillation comes from reading a model's mind, the industry may need better tools for distinguishing inspiration from imitation.
Sources
- Will Knight, "A New Trick Reveals AI Models' Inner Thoughts," WIRED, Aug 11, 2026. https://www.wired.com/story/a-new-trick-reveals-ai-models-inner-thoughts/
- Alexander Panfilov et al., research paper on reasoning trace extraction (referenced in WIRED article).