The Buying Decision Isn't on the Leaderboard

Datacurve's DeepSWE benchmark (published 2026-05-26, github.com/datacurve-ai/deep-swe) is a 113-task evaluation built from 91 open-source repositories across five programming languages. Reference solutions average 668 lines added across 7 files, with prompts averaging 2,158 characters. SWE-Bench Pro, the previous dominant benchmark, has reference solutions averaging 120 lines across 5 files with prompts averaging 4,614 characters. DeepSWE gives the agent less instruction and expects roughly 5.5 times more code output, which Datacurve argues mirrors how a human developer actually hands work to an AI assistant.

The leaderboard reorder is the headline. GPT-5.5 leads at 70 percent, GPT-5.4 at 56 percent, Claude Opus 4.7 at 54 percent, then a steep drop to Claude Sonnet 4.6 at 32 percent and a long tail. Claude Haiku 4.5, which scores 39 percent on SWE-Bench Pro, collapses to zero on DeepSWE. The range stretches from a 30-point band on SWE-Bench Pro to a 70-point band on DeepSWE. GPT-5.5 achieves its 70 percent at $5.80 median cost per trial.

Datacurve also audited SWE-Bench Pro's verifiers. The audit drew 30 tasks at random, ran 3 rollouts across 10 frontier model configurations, and deployed an LLM-based judge to independently assess whether each agent's patch actually solved the problem. SWE-Bench Pro's verifiers accepted wrong implementations 8.5 percent of the time and rejected correct implementations 24 percent of the time. DeepSWE's verifiers registered 0.3 percent and 1.1 percent. The "roughly one-third" combined error rate that has dominated yesterday's discourse is the sum of these two separate numbers.

The DeepSWE article's qualitative analysis section, midway through the piece, reports: "On DeepSWE, Claude Opus 4.7 and GPT-5.4 wrote and ran new tests in the project's own test framework on over 80% of their runs, even though no one asked them to. On SWE-Bench Pro, those same models dropped to 28% and 18%, respectively." The reason given is that SWE-Bench Pro's prompt template explicitly tells agents they "should not modify the testing logic or any of the tests." Agents complied, suppressing a behavior that the qualitative analysis suggests would have improved performance.

The mechanical structure of the finding is what makes it load-bearing. The two benchmarks ran the same models on similar code-modification tasks. The single textual variable between environments was the instruction clause about testing. The behavioral suppression was approximately 60 percentage points across two different frontier models. It was not a within-model fluctuation, not a noise band, not a methodological artifact. It was a controlled cross-benchmark comparison with one variable changed, and the variable was a single sentence in a prompt template.

This is the largest-N public empirical demonstration to date that prompt design controls reasoning regime activation in production agents. The framework that has been publishing this claim for over a year (the inference-time cognitive configuration series, beaudiamond.ai/articles) just received independent validation at a scale and in a venue that has nothing to do with it. Datacurve had no awareness of the framework. The finding stands on its own measurement.

Martin Orne published "On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications" in American Psychologist (volume 17, 1962, pages 776-783). The paper documented that experimental subjects modify behavior not just in response to explicit instructions, but in response to their interpretation of what behavior the experimental situation "demands." The mechanism overflows scope: subjects suppress or activate behaviors far outside the explicit instruction set in response to perceived situational demands.

The DeepSWE prompt-suppression finding is structurally identical at the mechanism level. The "should not modify testing logic" instruction was interpreted as a demand characteristic about the entire testing domain, suppressing self-verification (an unrelated adjacent behavior class) by approximately 60 percentage points. This is exactly the demand-characteristic overflow pattern Orne documented in 1962, transposed into the inference-time setting with frontier transformer agents instead of human experimental subjects. The mechanism is the same. The AI evaluation literature has not been citing this canon.

Beyond the prompt-suppression finding, DeepSWE's qualitative trajectory analysis surfaces a second buried finding: the model families fail in distinct, workload-predictable signatures. The discourse is treating leaderboard scores as the differentiator. The failure signatures are the actual differentiator for procurement.

Claude is forgetful on multi-part prompts. Roughly two-thirds of Claude's "MISSED_REQUIREMENT" failures on DeepSWE follow a "one branch shipped" pattern. When a prompt enumerates parallel behaviors ("support both sync and async," for instance), Claude typically implements the obvious branch and forgets to mirror the change to the parallel branch. One documented example: Opus 4.7 correctly landed a sync state-data hook in one engine class, while the async engine class never received the same hook. This is a coherent, reproducible failure signature, not run-to-run variance.

GPT-5.5 implements exactly what is asked. Across multiple runs of the same task, GPT trials converge on the same interpretation of the prompt. GPT-5.5 had the lowest rate of missed stated behaviors of any configuration tested. Instruction-following precision is a stable architectural trait, not per-run luck. This is the inverse failure signature: GPT will execute exactly what the prompt says, which means if the prompt is underspecified, GPT will not invent the missing structure (and Claude, with its environmental attentiveness, might).

The inference-time cognitive configuration series has been publishing for over a year. Article 1 in March 2026 introduced the thesis: frontier models contain latent reasoning regimes that default interactions rarely activate, and the gap between what a model can do and what it actually does in a standard exchange is enormous and closeable through interaction design. Article 5 last week (May 18) presented the mathematical foundation by way of Liang/Miikkulainen/Fiete's attractor-geometry measurement. Signal 010 yesterday (May 26) named the field convergence on the architectural direction.

DeepSWE today is the production-scale empirical receipt. The framework predicted that interaction design controls reasoning regime activation. DeepSWE measured that prediction at large N across two frontier models, with the suppression magnitude (60 percentage points) higher than any prior published measurement of prompt-design effects. The validation comes from a completely independent source. Datacurve is a benchmark startup; their incentive is to publish the most rigorous coding-agent evaluation in the field. They had no awareness of or interest in validating an inference-time cognitive configuration framework. They measured the phenomenon because their methodology required cross-benchmark comparison and the cross-benchmark comparison surfaced it.