Germany's Open AI Model Topped Every Benchmark. Then Its Own Report Admitted the Test Was in the Training Data.

The community caught it before the consortium did.

That is the least comfortable sentence in the Soofi S story, and it is also the one that matters most for the future of open-weight AI benchmarks. A German research consortium coordinated by the KI Bundesverband (German AI Association) released Soofi S — an open 30-billion-parameter language model that, according to its initial scores, beat every fully open competitor on both English and German benchmarks. It outperformed OLMo 3 32B from the Allen Institute for AI. It beat Apertus 70B from ETH Zurich and EPFL, a model more than twice its size. It took first place on every German-language benchmark in the comparison suite, sometimes by double-digit margins.

Then version 3.0 of its technical report quietly acknowledged that the science benchmark GPQA had leaked into the training data.

How a Missing Train-Test Split Broke the Pipeline

The root cause was depressingly mundane. On Hugging Face, the GPQA benchmark has no separate training set. Every question — test, Diamond, everything — sits under the default label "train." The consortium's data pipeline selected content based on these split names, so when it assembled its training corpus, rephrased questions from the GPQA test set ended up side by side with legitimate practice questions designed to teach the model test formats.

The leak affected both English questions and machine-translated German versions. A full audit triggered by the discovery turned up three more benchmarks with the same structural issue — TruthfulQA, BLiMP, and Inverse Scaling — though none of those are used in Soofi S's evaluation.

One thing stands out: nobody caught the problem during training. The model's performance on the affected GPQA tasks climbed from 32.3 to 43.4 points over the course of training — but that gain fell within the normal range and looked indistinguishable from progress on uncontaminated tests. There was no sudden spike that could have served as a warning sign. Benchmark contamination, it turns out, does not always announce itself.

The Transparency Paradox

Soofi S is one of the most transparently documented open models ever released. The consortium published not just the model weights and selected intermediate checkpoints, but the complete training and evaluation code and a detailed data inventory listing raw token counts, epoch numbers, and effective contributions per source — including sources that were reviewed but ultimately excluded. Roughly 152,000 individual results were made available for verification. The model meets the Open Source AI Definition 1.0 from the Open Source Initiative.

And it was exactly this transparency that allowed the community to find the leak. A proprietary model trained behind closed doors would never have been caught, but it would also never have been caught — nobody would have looked, because nobody would have had the data to examine.

The consortium's response was textbook remediation: drop GPQA from the evaluation, recalculate every score for every model in the comparison, and confirm that the ranking order did not change. Ellamind, a consortium partner, ran a separate verification using its own deliberately withheld test data that Soofi S was guaranteed not to have seen during training. The results held.

But the question that lingers is not whether this particular ranking was fair — it is how many other benchmark scores across the field are contaminated in ways nobody has yet found.

An Architecture Built for Long Contexts

Beyond the contamination controversy, Soofi S is technically interesting on its own merits. Its hybrid Mamba-Transformer architecture, adopted without modification from NVIDIA's Nemotron 3 Nano recipe, activates only 3.2 of its 31.6 billion parameters per token. In practice, this means processing speed stays nearly flat from 4,000 to 256,000 tokens of context — whereas dense models in the 14 to 24 billion parameter range see throughput drop sharply as context grows.

At 40,000 tokens with 32 parallel requests, Soofi S generates roughly eight times more tokens per second per GPU than comparable dense models. Only 6 of its 52 layers maintain the traditional KV cache at all; the rest use Mamba-2 selective-state-space layers that keep memory constant regardless of sequence length.

The consortium processed about 27 trillion tokens across three training phases, with a deliberate and unusual emphasis on German-language data. German made up 7.2 percent of the first phase and rose to 15.3 percent in the second — compared to about 5 percent for all non-English languages combined in NVIDIA's Nemotron reference recipe. The data draws from German web text (HPLT), the openly licensed German Commons corpus, German portions of FinePDFs and FineWiki, and the commercially licensed Genios corpus containing 193 million newspaper articles from 916 German publications.

On code benchmarks, Soofi S scores 73.8 percent on HumanEval and 70.2 on MBPP. On INCLUDE-DE, a test for Germany-specific regional knowledge, it ties with Qwen3.5 35B-A3B at 61.2 points despite having a fraction of the active parameters.

The Scaling Laws Debate That Won't Go Away

Critics have argued that Soofi S is heavily "overtrained" by the standards of classical scaling laws. The 2022 Chinchilla scaling laws from Google DeepMind identified roughly 20 tokens per parameter as the efficient frontier. Soofi S lands at several hundred tokens per parameter. Factor in only the 3.2 billion active parameters, and the ratio jumps to several thousand to one.

Michael Fromm, part of the project's technical leadership, pushed back directly: "There's new research showing that the old scaling laws from dense models no longer apply to MoE architectures." Individual experts in a mixture-of-experts model benefit from seeing the same documents, so repeated data causes less of a problem than it would with dense models. Fromm points to NVIDIA, which trained its own models on up to 25 trillion tokens, as precedent.

The debate matters beyond academic circles. If Soofi S's approach is valid, it means the most compute-efficient path to strong bilingual performance runs through aggressive over-training of small active-parameter models — a finding that would reshape how European AI projects allocate their budgets.

What the Soofi S Incident Really Means

The GPQA leak is not evidence that Soofi S is a bad model — the evidence strongly suggests it is a genuinely capable one, and the ranking held after the contaminated benchmark was removed. What it is evidence of is a structural vulnerability in how the open-weight ecosystem validates performance.

Standardized benchmark holdout procedures do not exist for the Hugging Face era. Community auditing pipelines are ad-hoc and rely on sustained attention from volunteers. And the very nature of open data — the thing the community celebrates — is what makes contamination discoverable in the first place, which raises an uncomfortable mirror question about the proprietary models that cannot be audited at all.

The consortium is now automatically cross-checking its training data against all known test questions instead of relying on split labels. That fix is simple. The broader trust problem is not.

Sources

• The Decoder — "German AI consortium releases Soofi S, an open 30B model that tops benchmarks in both English and German"
• Cybernews — "Germany's Soofi S may be the best-performing open-source AI model"
• Soofi S Technical Report v3.0 (analysis via The Decoder coverage)
• Xpert.Digital — "Soofi S: Deutschlands erstes ernstzunehmendes KI-Modell"
• Google DeepMind — "Chinchilla Scaling Laws" (background on scaling debate)