Paper deep dive

SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 87

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,696 characters extracted from source content.

Expand or collapse full text

∗Equal Contribution †Project Lead ‡Corresponding Author [Date]August 24, 2026 [Homepage]https://lab.einsia.ai/swe-refactor-bench SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration? Deyao Hong∗† Yizhe Chi∗ Wenyi Li∗ Xiaoqiu Wang Mingju Gao Kaisen Yang Bingxiang He Youjie Zheng Calvin Xiao Qinhuai Na‡ Affiliation: Navers Lab, Einsia.AI Tsinghua University Abstract Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs (5.4%5.4\%) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/10047.0/100. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, 58%58\% reach 99%99\% of the fixed checks, yet only 26%26\% reach 100%100\%. Agent capability differs across migration categories: agents score 31.431.4 on build toolchain rewrites but only 5.65.6 on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations. 1 Introduction Modern software systems accumulate technical debt over decades, leaving long-lived repositories tied to technology stacks that their teams no longer want to maintain [14]. Replacing a language, framework, platform, or build toolchain remains expensive and largely manual because engineers must coordinate changes across source code, dependencies, interfaces, and build logic. Examples include rewriting a C library in Rust and replacing Maven with Gradle across an entire repository. A successful migration must both replace the old stack and preserve the system’s behaviour. The original system provides a behavioural reference, so correctness can be tested without a separate specification. Coding agents are increasingly reliable at localized bug fixing [33, 67, 63, 69, 71]. This progress raises a harder question: can coding agents autonomously complete long-horizon, whole-repository migrations while preserving behaviour? Reliable performance would extend coding agents from local edits to a costly class of system maintenance. Figure 1 contrasts the one-stage evaluation prior benchmarks apply with the three stages SWE Refactor Bench uses, and previews the benchmark’s task coverage and model scores. Figure 1: SWE Refactor Bench at a glance. Top: how a migration gets scored. Left, prior benchmarks: one stage, a behavioural test suite. With no migration audit in front of it, a submission that never migrated anything still turns every check green, so the suite cannot tell a completed migration from an untouched repository. Right, SWE Refactor Bench: three stages in series—Migration Audit asks whether the migration happened at all, Behavioural Tests is that same fixed suite, and Agentic Verification sends six coding agents, one hour each, after the differences the suite was never written to catch. Bottom left: the 20 tasks span four kinds of technical debt—language, framework, platform, and build toolchain. Bottom right: composite scores of 8 frontier models, each in its strongest configuration (out of 100; defined in (4)). Existing benchmarks cannot establish whether coding agents can complete whole-repository migrations. They evaluate repository-level code changes with fixed tests, including bug fixes, language-version upgrades, and language migrations [33, 69, 71, 44, 64, 34]. These tests provide a clear red-to-green signal for bug fixing: at least one test fails before the patch and passes after it. Whole-repository migration, however, starts from a repository whose tests already pass. A correct migration and an untouched repository can therefore receive the same perfect test score. Fixed tests measure behavioural correctness, but they cannot establish whether the migration actually occurred. This creates an easy shortcut: agents can retain or copy the original implementation to make the tests pass. We call this failure mode Blindness: a behaviour-only evaluator awards full credit without establishing that the target stack replaced the original implementation. Such an evaluator can detect whether known behaviour was broken, but not whether the migration occurred. This failure reflects a mismatch between what fixed tests observe and what migration requires. Fixed tests compare observable behaviour, whereas migration completeness asks whether the target stack actually replaced the original implementation. Adding more behavioural checks strengthens the evidence that behaviour was preserved, but it cannot establish replacement: the untouched implementation passes those checks by construction. Conversely, verifying replacement does not establish that behaviour survived. Evaluating migration therefore requires separate evidence for both properties. We introduce SWE Refactor Bench, which collects three forms of evidence through a three-stage protocol. Migration Audit is a hard gate that checks whether the target stack replaced the old implementation; any failed criterion vetoes the submission. This gate prevents unchanged code and wrapper patches from earning credit. The Behavioural Tests stage measures behavioural correctness with 130,118130,118 fixed checks recorded from the original system and requires every check to pass. Agentic Verification tests behaviours that a fixed suite may miss: six independent coding agents receive the original and migrated codebases and one hour each to generate differential tests. These tests are generated after submission, vary across verifier runs, and remain invisible to the migration agent. A verifier can reject a submission only with an executable counterexample that passes on the original and fails on the migrated system. The three stages prevent migration shortcuts, enforce known behavioural requirements, and probe differences not encoded in the fixed suite. The benchmark comprises 20 whole-repository migrations of real open-source infrastructure, including SQLite, zlib, libsodium, and GraphHopper. The tasks cover language (7), framework (7), platform (3), and build toolchain (3) migrations and give agents 6 to 30 hours of autonomous work per task. Each task requires the repository to use the target stack throughout while ensuring behavioural correctness. We evaluate 8 frontier models under 26 model–effort configurations, running each configuration once on all 20 tasks for 520 scored runs. Current agents remain far from reliable whole-repository migration. The best configuration, claude-opus-5 at xhigh effort, scores 47.0/10047.0/100; only 28 of 520 runs (5.4%5.4\%) pass all three stages, and 13 of the 20 tasks receive no accepted solution. Migration completeness and behavioural correctness are distinct abilities, and agents miss them in opposite directions: 3030 runs preserved behaviour by skipping the migration, and were stopped at Migration Audit; 252252 completed the migration but broke behaviour, and were stopped at Behavioural Tests. Agents also struggle to close the final correctness gap. Among the 340 runs that pass Migration Audit, 58%58\% reach 99%99\% of the fixed checks, yet only 26%26\% reach 100%100\%. Agent capability differs across migration categories: agents score 31.431.4 on build toolchain rewrites but only 5.65.6 on language rewrites. These results establish a capability gap on the 20 tasks in SWE Refactor Bench; they do not rank the intrinsic difficulty of all migration projects. In summary, we make the following contributions: • Blindness and benchmark. We identify Blindness: behaviour-only evaluation can award full credit without establishing that a migration occurred. And we introduce SWE Refactor Bench, a benchmark of 20 long-horizon, whole-repository migrations drawn from real open-source infrastructure and spanning language, framework, platform, and build toolchain migrations. • Three-stage adversarial evaluation. We design a protocol that combines a hard migration audit and 130,118130,118 fixed behavioural checks with agentic verification. During Agentic Verification, six independent coding agents actively search for hidden behavioural differences after submission and can reject a migration only with an executable counterexample. • Revealing agent capability gaps. “Getting the code right” is not the same as “getting the migration done”—and even doing both is not enough. Agents miss the two conditions in opposite directions—3030 runs skip the migration, 252252 complete it and break behaviour—so neither stage can stand in for the other; and among the 88 submissions that satisfy both under the fixed suite, agentic verifiers break 60. Only 28 of 520 runs (5.4%5.4\%) survive all three stages. Current agents therefore rarely deliver migrations that are complete, behaviour-preserving, and robust to agentic verification. 2 SWE Refactor Bench: Benchmark Design SWE Refactor Bench measures one ability: given a working repository built on one technology stack, can an agent deliver the same repository on another stack, with its observable behaviour intact and the old stack gone. 2.1 Task Formulation A behaviour-preserving migration task is a tuple τ=(RA,ΣA→ΣB,,ℐ,E,B).τ=(R_A,\; _A→ _B,\;O,\;I,\;E,\;B). (1) RAR_A—which we call State A—is a real open-source repository, taken at one commit, in a buildable and working condition. ΣA _A is the stack it is built on and ΣB _B the stack it must be built on afterwards: a language, an application framework, a host platform, or a build toolchain. O is the observable interface of the artifact, that is, the surface available for inspection once the repository has been built—process output and exit status, the symbols exported by an installed library, the manifest of an installation tree, the responses of a served endpoint. ℐI is the instruction given to the agent, E an offline image and B a time budget. The agent’s input is (RA,ℐ,E,B)(R_A,I,E,B): a container holding the original repository, the toolchains of both stacks, and an instruction saying what is to be done, with no network. Its output is the working tree RSR_S at the end of the run. Writing ⁡(R)O(R) for the observations produced by the artifact that repository R builds, RSR_S solves τ exactly when both of the following hold: ΣB builds the delivered artifact, and ΣA is absent from the repository and from the build closure, _B builds the delivered artifact, and _A is absent from the repository and from the build closure, (2) ⁡(RS)=⁡(RA). (R_S)=O(R_A). (3) We call (2) the migration condition and (3) the preservation condition. The first is a claim about the text of a repository and about what its build compiles; the second is a claim about the behaviour of an artifact. No new functionality has to be designed anywhere in this work: State A already does everything State B must do; the whole difficulty is to rebuild those behaviours exactly, on a stack that expresses them in an entirely different way. Why behavioural tests fail here. A behavioural test suite T is a finite set of observations drawn from O; the score it gives a repository R is the fraction it passes, rate⁡(R,T)rate(R;T). Since the preservation condition (3) is defined relative to RAR_A and T⊆T , we have rate⁡(RA,T)=1rate(R_A;T)=1 for any T—by construction, and for every task in this family. Setting RS=RAR_S=R_A (the empty diff: the agent hands the repository back untouched) therefore earns full marks on any behavioural suite while satisfying not one clause of the migration condition (2). This is exactly the blindness of Section 1: the maximum of the reward sits on a submission with zero work in it, and enlarging T does not help, because every case the migrated repository must pass is a case the original already passes. 2.2 Where the Tasks Come From Debt class Representative migrations Tasks Source LoC Budget Criteria Modules Checks (h) (Stage I) (Stage I) (Stage I) Language C → Rust, C → Java, Go → Zig 7 4.4 k–39.8 k 12–30 60 117 59,771 Framework Flask → Starlette, Gin → chi, Vue → React 7 0.8 k–94.8 k 6–16 39 79 55,852 Platform POSIX → wasm32-wasi, Node → V8 realm 3 16.0 k–358.0 k 6–10 19 31 6,725 Build toolchain Autotools → CMake, Maven → Gradle 3 19.0 k–78.8 k 6 18 37 7,770 Total 20 upstream projects, 20 tasks 20 867,062 262 136 264 130,118 Table 1: Composition of the SWE Refactor Bench task set. Every task requires a real open-source repository to be migrated in its entirety onto another stack, with its interface and observable behaviour unchanged. Budget is the time limit given to the agent; criteria are the Stage I migration questions, modules are the independent test modules of Stage I, and checks are the behavioural cases those modules hold. Pick the debt first, the repository second. We did not choose repositories and then invent a change for them; we went the other way: fix on a migration that a maintainer would call overdue, then look for a project where that migration is the whole job. Three admission requirements follow. First, the old stack has to be load-bearing rather than incidental, so that removing it reaches the design and not merely the import statements—cmark’s parser state machine, for instance, is hand-written C, and moving it to Rust means redesigning ownership rather than adding a layer of FFI. Second, the observable interface has to be one that something outside actually depends on, so that “behaviour preserved” is a constraint with content rather than a slogan—a C ABI, an HTTP API, an installation tree all have downstream consumers. Third, there has to be a runnable reference: State A itself can be built and executed repeatedly, which is what makes differential testing possible. We deliberately chose load-bearing infrastructure rather than exercises: SQLite [20, 59], zlib and the DEFLATE format it implements [18, 21], libsodium [7, 17], and GraphHopper, whose routing is built on contraction hierarchies [23, 27], among others. Four kinds of technical debt. A repository’s stack comes down to four things: what language it is written in, what framework it is organised around, what host it assumes, and what builds it. Each class of task moves exactly one of them (Table 1). Language rewrites (7 tasks) replace the implementation language while keeping the shape of the artifact: the original language’s implementation details are deleted along with its source, yet the external behaviour of the artifact has to be reproduced exactly, and the new language will not make the same choices by itself. Framework rewrites (7) keep the language and replace the dependency the application is organised around: most of what has to be reproduced is not code the application wrote but what the framework did on its behalf—how errors are represented, how parameters are coerced, how content is negotiated. Platform ports (3) replace the host the code assumes: from POSIX to wasm32-wasi [28], process control, the filesystem namespace and memory mapping are withdrawn, leaving only what the host explicitly grants. Build-toolchain rewrites (3) move the part that produces the artifact rather than the part that runs: what is compared is not only what the program does when it runs but what it was packaged into. What a task ships. Once (1) is instantiated, the agent receives: a State A that builds cleanly, a State B declaration naming the target stack down to specific versions, an instruction stating the requirements and how the build is invoked, an offline image—which also carries State A’s own toolchain, since the original is its reference—and an artifact contract saying what will be collected from the finished workspace. The rest it arranges itself: read the repository and the requirements, form a plan, rewrite the code onto the target stack, get it to build into a real artifact, and finally check its own work against State A. On the evaluation side sit three things it never touches: the Migration Audit criteria, written as prompt questions about this repository; the Behavioural Tests tests, whose expectations are recorded from a reference build and run of State A inside the evaluation image; and the further checks the six verifiers of Agentic Verification perform beyond those fixed tests. All three live in a separate image that is never mounted into the agent’s container, so those tests are not merely “unread” by it—they do not exist for it. 2.3 The Three-Stage Protocol Figure 2: On the left, what the agent sees: a real repository on the source stack, an instruction, an offline image, and the process it works through on its own—read the repository and the requirements, rewrite onto the target stack, get it to build, then check itself against the original. On the right, the three-stage evaluation it never touches: Migration Audit checks whether the migration actually happened, Behavioural Tests checks whether the fixed behaviour is preserved exactly, and Agentic Verification goes looking for whatever differences remain beyond the fixed tests. Figure 2 separates the agent’s workspace and workflow from the hidden three-stage evaluation. A submission is the working tree at the end of the agent’s run; nothing it built is carried into evaluation, and each stage rebuilds from the submitted source whatever it needs. The three stages are applied in order, and each of them can end the run—except that we ran the fixed suite on every submission, vetoed or not, since blindness cannot be counted otherwise; those runs still score zero. Stage I: Migration Audit. It asks one thing: did the migration actually happen—has the old stack disappeared from the repository and from the build. The blindness of Section 1 is exactly why someone has to ask: a behavioural suite gives full marks to a repository handed back untouched, so “was the work done at all” has to be checked separately. The criteria are written as prompt questions about this repository, answered one by one by a model reading both source trees; every failing verdict must cite re-checkable evidence, and each criterion is judged three times independently, with the majority taken. Stage I: Behavioural Tests. It asks whether behaviour survived unchanged. These tests were not written out of thin air but built from State A: the same calls are run against the original repository and the recorded output becomes the expected answer, so the same inputs must afterwards produce the same outputs and the same exit status. Across the suite there are 130,118130,118 checks, on average more than six thousand per task; a single wrong check scores zero, and only a clean sweep opens Stage I. Stage I: Agentic Verification. The first two stages check what we thought of in advance; this one checks what we did not. A fixed suite can only ask what its author happened to think of, and a submission that reaches this point has answered all of it correctly. So we change the kind of checker—six coding agents, one hour each, each holding both source trees and looking for something the original repository does that this submission does not. The six are divided up: five each probe one assigned direction (a C ABI, a set of routes, an installation tree, and so on) and the sixth is unrestricted, so they complement one another in coverage instead of re-searching the same ground. What they hand in cannot be a report, only an executable test case: passing on State A and failing on the submission, and it must first go green against the reference and then reproduce three times, so that a broken test and a flaky test both fail to win. This is differential testing [46] in an agentic form: a verifier may work the way a real tester does—generate inputs against a stated property [13], relate one output to another [12], or shrink a difference until it is reportable [70]—but what it says does not count; only a difference that runs does. Scoring. Let g be the Migration Audit verdict, rir_i the pass rate of module i in Behavioural Tests, and s∈0,…,6s∈\0,…,6\ the number of verifiers that failed to produce a counterexample. A submission scores S(τ)=[g=pass]⏟Stage I: veto⋅[ri=1∀i]⏟Stage I: all or nothing⋅( 0.4+0.6⋅s6⏟Stage I: per verifier)∈0∪[0.4, 1].S(τ)\;=\; 1 [g= pass ]_Stage I: veto· 1 [r_i=1\;\;∀ i ]_Stage I: all or nothing· (\,0.4\;+\; 0.6· s6_Stage I: per verifier )\;∈\;\0\∪[0.4,\,1]. (4) Every factor in the formula corresponds to one condition from Section 2.1. Stage I multiplies rather than adds, because without the migration the task was not done at all, and no amount of behavioural correctness should buy points for it. Stage I is all or nothing, because what this family of tasks asks is whether this is a drop-in replacement, and a library that is wrong once in a thousand calls is not: behind the failing test stands a downstream consumer, and it will not be spared because the other 99.99%99.99\% of the behaviour is right (Section 3.3.2 gives concrete examples). Stage I carries 0.60.6 because what it looks at is a residue that no fixed suite can see, and so deserves the larger share; it is scored linearly rather than all-or-nothing because not broken is not the same as no difference: six verifiers finding nothing is more credible than one verifier finding nothing, but this is still a matter of how strong the evidence is, not a demonstration of equivalence, and a linear score records precisely that degree. 3 Experiments This section first states the setup and the metrics (Section 3.1), then answers two questions: how far today’s agents get on whole-repository behaviour-preserving migration (Sections 3.2–3.3), and whether the evaluation itself holds up (Section 3.4). 3.1 Setup and Metrics We evaluate eight frontier models: claude-opus-5 and claude-sonnet-5; gpt-5.6-luna and gpt-5.6-sol; kimi-k3 [35]; qwen3.8-max [66]; dsv4-flash [15]; and glm-5.2 [24]. The two GPT-series models use Codex as their harness; the other six use Claude Code. Every model ran all 20 tasks, for a total of 88 models, 2626 configurations and 520520 scored runs. A run is one independent attempt by one model at one task: a fresh container is created from that task’s image, the instruction is handed to the agent, and it works on its own within the time budget the task declares, with no network beyond the model endpoint; when the budget runs out or it stops of its own accord, the working tree is collected as a submission and sent into the three-stage evaluation. Six metrics, each answering a different question. Alongside the composite score of (4) we report counts, because a count states what happened to a submission, whereas a score only states it after weighting. The following definitions are used throughout the paper. • Migrated (Stage I passed): runs in which every migration criterion was judged to pass by majority. • All tests pass (Stage I perfect): runs that passed every fixed check in Behavioural Tests. • Accepted: runs that migrated, passed every fixed check, and survived all six verifiers. • Broken: the share of runs that migrated and passed every fixed check but were then defeated by at least one verifier. • Blindness: runs that passed every fixed check yet were vetoed at Stage I. A behavioural instrument gives full marks to a repository that was never migrated—this count corresponds directly to the blindness phenomenon of Section 1. • Score: the mean over runs of the score from (4), out of 100100. 3.2 Overall Performance Table 2: The three-stage funnel: where each of the 520520 runs stops. Each row aggregates 2020 runs, one per task. Values are numbers of runs; the five outcome columns are mutually exclusive and exhaustive, so they sum to 2020 across a row. Harness identifies the coding-agent client: GPT-series models use Codex, and all others use Claude Code. Colour marks the stage at which a run stops: Migration Audit , Behavioural Tests , Agentic Verification , all three passed ; within a column, darker means more runs in that cell. Blindness is a special kind of Stage I failure: all fixed checks pass, yet the repository was never migrated. Model Harness Effort Stopped at Stage I: not migrated Stopped at Stage I Stopped at Stage I Passed fixed checks also fail fixed checks pass (Blindness) fixed checks fail counterexample found accepted Claude Opus 5 Claude Code low 2 0 13 3 2 Claude Opus 5 Claude Code medium 1 2 10 5 2 Claude Opus 5 Claude Code high 0 2 10 4 4 Claude Opus 5 Claude Code xhigh 0 1 8 6 5 Claude Opus 5 Claude Code max 2 2 9 4 3 GPT-5.6 Sol Codex none 11 0 8 1 0 GPT-5.6 Sol Codex low 11 0 7 2 0 GPT-5.6 Sol Codex medium 8 3 7 2 0 GPT-5.6 Sol Codex high 8 0 7 4 1 GPT-5.6 Sol Codex xhigh 5 1 11 3 0 GPT-5.6 Sol Codex max 5 0 8 3 4 Kimi K3 Claude Code max 5 0 10 3 2 Claude Sonnet 5 Claude Code low 5 1 13 1 0 Claude Sonnet 5 Claude Code medium 6 0 10 3 1 Claude Sonnet 5 Claude Code high 3 2 13 2 0 Claude Sonnet 5 Claude Code xhigh 7 1 10 1 1 Claude Sonnet 5 Claude Code max 5 1 12 1 1 GPT-5.6 Luna Codex none 10 1 8 1 0 GPT-5.6 Luna Codex low 14 1 4 1 0 GPT-5.6 Luna Codex medium 7 1 12 0 0 GPT-5.6 Luna Codex high 6 2 11 1 0 GPT-5.6 Luna Codex xhigh 4 2 12 2 0 GPT-5.6 Luna Codex max 2 1 14 3 0 Qwen 3.8 Max Claude Code max 8 4 6 0 2 DeepSeek V4 Flash Claude Code max 6 1 11 2 0 GLM 5.2 Claude Code max 9 1 8 2 0 Table 3: Behavioural pass rate, score and cost for the same runs. Behavioural pass rate is the row mean of the fraction of Behavioural Tests fixed checks passed (a perfect score is not required), score is defined in (4), and cost is the average API spend per task. Harness identifies the coding-agent client: GPT-series models use Codex, and all others use Claude Code. Shading encodes only the magnitude of a value within its column; each model’s best-scoring row is marked with a shaded background . Model Harness Effort Behavioural pass ↑ Score ↑ Cost ↓ % out of 100 USD / task Claude Opus 5 Claude Code low 92.5 20.5 34.8 Claude Opus 5 Claude Code medium 92.7 28.5 38.4 Claude Opus 5 Claude Code high 92.8 34.5 55.7 Claude Opus 5 Claude Code xhigh 92.8 47.0 74.9 Claude Opus 5 Claude Code max 91.9 31.0 72.4 GPT-5.6 Sol Codex none 63.6 4.0 2.9 GPT-5.6 Sol Codex low 62.0 7.0 5.9 GPT-5.6 Sol Codex medium 70.8 6.5 6.0 GPT-5.6 Sol Codex high 74.0 19.0 7.7 GPT-5.6 Sol Codex xhigh 73.5 9.5 19.1 GPT-5.6 Sol Codex max 84.1 28.5 143.5 Kimi K3 Claude Code max 93.9 19.5 28.9 Claude Sonnet 5 Claude Code low 79.2 4.0 4.4 Claude Sonnet 5 Claude Code medium 73.9 15.0 11.9 Claude Sonnet 5 Claude Code high 85.6 6.0 24.6 Claude Sonnet 5 Claude Code xhigh 76.5 9.0 27.0 Claude Sonnet 5 Claude Code max 84.3 8.5 27.5 GPT-5.6 Luna Codex none 66.4 4.0 1.6 GPT-5.6 Luna Codex low 58.3 4.0 1.7 GPT-5.6 Luna Codex medium 66.6 0.0 1.7 GPT-5.6 Luna Codex high 75.6 4.0 1.8 GPT-5.6 Luna Codex xhigh 83.8 5.5 2.9 GPT-5.6 Luna Codex max 89.1 10.5 2.8 Qwen 3.8 Max Claude Code max 74.7 10.0 14.5 DeepSeek V4 Flash Claude Code max 90.7 7.0 4.3 GLM 5.2 Claude Code max 85.2 6.5 17.5 Table 4: Summary by debt class, pooling all 2626 configurations. Column meanings follow the metric definitions of Section 3.1; shading follows Table 3. Stage I pass rate and acceptance count do not move together: build toolchain is the easiest class to get past Stage I and the one that loses most at Stage I; framework rewrites contribute 1414 of the 2828 acceptances; language rewrites score lowest. Migrated ↑ All tests pass ↑ Accepted ↑ Broken ↓ Blindness ↓ Score ↑ Class Tasks Runs n % n n % n mean Build toolchain 3 78 63 80.8 36 6 82.4 2 31.4 Platform port 3 78 45 57.7 21 4 76.5 4 17.2 Framework rewrite 7 182 132 72.5 40 14 44.0 15 12.0 Language rewrite 7 182 100 54.9 21 4 66.7 9 5.6 Leaderboard. Table 3 lists all 2626 configurations with their behavioural pass rate, score and cost, with each model’s best-scoring row shaded. Comparing strongest configurations, claude-opus-5 leads on acceptances and score: 5 acceptances in 2020 runs and a score of 47.047.0; behind it are gpt-5.6-sol at 28.528.5, kimi-k3 at 19.519.5 and claude-sonnet-5 at 15.015.0. Across the eight models in their strongest configurations, 160160 runs yield only 1414 acceptances. The rows worth studying are the three with no acceptances at all. gpt-5.6-luna, dsv4-flash and glm-5.2 never got one, yet each produced runs that passed every fixed check—44, 33, and 33, respectively, scattered across the blindness and counterexample found columns of Table 2—and those runs then fell at Stage I or Stage I. Judged by the fixed checks alone, each of these three models would appear on the leaderboard with a few “perfect scores”; under the three-stage protocol they solved nothing. The funnel is narrow, and it narrows for three different reasons. Table 2 decomposes each row’s 2020 runs into five mutually exclusive outcomes. Of the 520520 scored runs, 340340 (65.4%65.4\%) passed Stage I and 118118 (22.7%22.7\%) passed every fixed check, but only 8888 did both and thereby reached Stage I, of which 2828 (5.4%5.4\%) survived all six verifiers; 13 of the 20 tasks were never solved by any model. The mean score over all 520520 runs is only 13.44/10013.44/100, but over the 8888 that reached Stage I it is 79.4379.43—so the difficulty is in getting to Stage I at all. Moreover the three gates stop different submissions; no group is filtered out twice: stages that overlap are redundant, whereas stages that are disjoint are measuring three different things. With the fixed suite as the only instrument, 118118 submissions would tie for first place, among them submissions that never migrated anything and submissions that a verifier breaks within the hour. Outcomes also vary by migration category (Table 4). Build toolchain rewrites have the highest Stage I pass rate (80.8%80.8\%) and mean score (31.431.4), whereas framework rewrites contribute 1414 of the 2828 accepted runs. 3.3 Analysis of Agent Behaviour 3.3.1 “Getting the code right” and “getting the migration done” are two different abilities Figure 3: (a) Where each model sits on two axes, both of them shares of the same runs: horizontally “does it still behave like the original” (a perfect Stage I), vertically “was the migration done at all” (Stage I). (b) The same two axes, per task, with marker area the number of accepted submissions. A migration has to satisfy both conditions, but they are satisfied by different runs, and most runs manage at most one. Put the two axes together and the failures fall into two groups (Figure 3a). One is the runs that never migrated and kept every fixed check: a submission that barely migrates anything passes any behavioural suite by construction, and 3030 runs did exactly that, spread over 77 of the 88 models. Stage I gives all 3030 full marks; only Stage I stops them. The other is eight times as large: 252252 runs completed the migration and broke behaviour doing it, and only Stage I stops those. Neither stage can stand in for the other, then: Stage I alone would reward doing nothing, Stage I alone would reward doing damage. Per task, the two axes diverge just as clearly (Figure 3b), and in both directions. On lang04 (acorn, JavaScript → Rust), 2020 of 2626 runs passed Stage I, yet not one passed every fixed check. Twenty runs completed the migration, but all 2626 failed at least one fixed check. lang01 (cmark, C → Rust) is the mirror image: only 66 runs passed Stage I, yet 55 passed every fixed check—and all five of those are blindness. Each cleared seven of the eight criteria and failed only the one asking whether the Rust is the implementation: it reproduces the original’s control flow statement for statement, a transliteration—ownership was never redesigned, the original’s manual memory management simply moved into Rust, and the safety this migration exists to buy was not bought. No behavioural test can express that, because behaviour was never where the problem was. Figure 4 shows the four typical shapes of such submissions; the first two (handed back as-is, wrapped in a shim) are defects no behavioural test can express. Figure 4: The four kinds of submission Stage I has to tell apart. Top left, nothing rewritten: the new file is the C copied over verbatim, only the suffix changed. Top right, wrapped: the Rust side is an extern "C" forwarding shim and the original C still does the work. Bottom left, half done: some functions are genuinely rewritten while others still call back into the old implementation. Bottom right, rewritten wrongly: really rewritten, but the behaviour changed. The first two pass every behavioural test, and only an instrument that reads mechanism can reject them. 3.3.2 The last 1%: agents cannot deliver a perfect migration Figure 5: (a) Among the 340340 runs that did complete the migration, the share reaching each level on the fixed suite. (b) Among the 8888 submissions that passed every fixed check, how many of the six independent verifiers (one hour each) failed to find a behavioural difference. Only the rightmost bar is an accepted migration. For a repository migration, any failing unit test is a serious risk: behind that failing test stands a real downstream consumer, and it will not be spared because the other 99.99%99.99\% of the behaviour is right. And this is precisely the step agents cannot get past. Among the 340340 runs that passed Stage I (Figure 5a), 91%91\% get the fixed suite past half, 58%58\% reach 99%99\% and 36%36\% reach 99.9%99.9\%—but only 26%26\% make no error at all. That final step alone eliminates 3535 of the 123123 runs that had already reached 99.9%99.9\%; across the campaign 140140 runs land in [99%,100%)[99\%,100\%), missing a median of 12.512.5 checks, and 1818 of them miss exactly one. Those 1818 are not scattered at random; they concentrate on a handful of specific checks. On fw03 (conduit, Vue → React), four different models all ended at 21768/2176921768/21769, failing the same one: the original uses hash routing, so visiting / settles at /#/ while the React version stays at /, which breaks every bookmark and every shared link to the site. On build03 (PyCryptodome, setuptools → Meson), five different models all ended at 380/381380/381: the METADATA long description in the built wheel is 00 characters, so once the package is published its PyPI project page is blank. Not one of these checks can be called nitpicking: each corresponds to a regression that would cause trouble in production, and the original repository passes all of them. In other words, this is not a problem with the test suite; it is a problem with the migration. Making no test error is still not enough. Among the 8888 submissions that did not miss a single fixed check, only 2828 survived all six verifiers; the other 6060 (68.2%68.2\%) had a counterexample found against them within the hour (Figure 5b), with the average submission holding off only 3.943.94 of the 66 verifiers. In other words, an instrument that looks only at behaviour would have accepted all 8888 of these submissions, and Stage I took back two thirds of them. When a break comes, it comes quickly: the median time to a counterexample is 17.017.0 minutes, against 32.832.8 minutes for a survival. 3.3.3 Agent capability differs across migration categories Table 5: Agent success varies across stages and migration categories. Every cell is the share of the runs that entered that stage and passed it, so the three multiply out to the category’s final acceptance rate. The Stage I column is taken over the runs that cleared Stage I (a blindness run passes every fixed test but is already out at Stage I, so it does not count); the Stage I column is taken over the runs that cleared both earlier stages and actually met the verifiers. Agents do not perform best on one category throughout the pipeline: they achieve their highest pass rates on build toolchain rewrites at the first two stages but their lowest at the last; framework rewrites show the reverse pattern. Migration Audit ↑ Behavioural Tests ↑ Agentic Verification ↑ Final ↑ Category Runs n % n % n % Acc. Score Build toolchain 78 63/78 80.8 34/63 54.0 6/34 17.6 6 31.4 Platform port 78 45/78 57.7 17/45 37.8 4/17 23.5 4 17.2 Framework 182 132/182 72.5 25/132 18.9 14/25 56.0 14 12.0 Language 182 100/182 54.9 12/100 12.0 4/12 33.3 4 5.6 All 520 340/520 65.4 88/340 25.9 28/88 31.8 28 13.4 Agents exhibit distinct capability profiles across the four migration categories (Table 5). Their overall scores rank build toolchain rewrites first (31.431.4), followed by platform ports (17.217.2), framework rewrites (12.012.0), and language rewrites (5.65.6). However, agents do not perform best on one category throughout the pipeline. They achieve their highest Stage I and Stage I pass rates on build toolchain rewrites (80.8%80.8\% and 54.0%54.0\%), but their lowest Stage I survival rate on the same category (17.6%17.6\%). On framework rewrites, agents achieve only 18.9%18.9\% at Stage I but their highest Stage I survival rate, 56.0%56.0\%. Agents encounter different bottlenecks across migration categories. Their lowest conditional pass rate occurs at Stage I for build toolchain rewrites and platform ports (17.6%17.6\% and 23.5%23.5\%), but at Stage I for framework and language rewrites (18.9%18.9\% and 12.0%12.0\%). These profiles align with the repository surface that agents must modify. In build toolchain rewrites and platform ports, agents mainly change how the repository is built or which host it runs on while leaving the product code largely unchanged. Their submissions pass Stage I relatively often, but Stage I exposes behavioural differences outside the fixed suite. In framework and language rewrites, agents modify the product code itself, and the fixed suite filters their submissions at Stage I before most reach the verifiers. Language rewrites show the sharpest early attrition: only 1212 of 182182 runs reach Stage I. 3.4 Analysis of Benchmark Validity The premise of SWE Refactor Bench is a claim about an instrument rather than about models, so this section examines the instrument itself: whether the Stage I judge is stable (Section 3.4.1), what the Stage I and Stage I tests each ask (Section 3.4.2), and whether all six verifiers are needed (Section 3.4.3). 3.4.1 Stage I Decisions Agree Across Models and Humans Does the same judge model read the same tree differently three times? The judge is gpt-5.6-sol; every criterion is judged by three independent samples of it and the majority is taken. Of 3,5363,536 criterion verdicts, 3,4053,405 (96.3%96.3\%) had all three samples agree, and only 131131 split 2:12:1. That looks like very little disagreement, but Stage I requires every criterion to pass, so a single disagreement on a single criterion can decide the fate of a whole run: of the 340340 runs that passed Stage I, 3535 (10.3%10.3\%) had at least one criterion that passed only 2:12:1—on that criterion, one sample argued for zero. Which means that without majority voting, letting any single dissenting sample count, the number passing Stage I would be 305305 rather than 340340. Majority over three samples is load-bearing here, not ceremonial. Table 6: Task-by-Task Comparison ofMigration Audit and Independent Human Annotation. Each row is one task, attempted once by every one of the 2626 model–effort configurations. Human and judge are how many of those runs each side called a genuine migration; agree, too strict and too lenient sort the same runs by whether the two verdicts matched, the judge zeroed a run the human accepted, or the reverse. Called a real migration Judge against human Task Class runs by human by judge agree ↑ too strict ↓ too lenient ↓ lang01 (cmark) Language 26 7 6 25 1 0 lang03 (sqlparse) Language 26 15 14 23 2 1 fw02 (json-server) Framework 26 15 13 24 2 0 fw06 (uploadserver) Framework 26 15 13 22 3 1 pf02 (Stylus) Platform 26 14 12 24 2 0 build01 (libsodium) Toolchain 26 23 19 22 4 0 6 tasks 156 89 77 140 14 2 Would a human judge differently? We asked two software-engineering researchers not involved in this work to decide independently, without seeing the judge model’s verdicts, whether “this is a genuine migration” for all 156156 runs of 66 tasks spanning the four migration classes, three tasks each, working from the same criteria the judge saw. Judge and human agree 89.7%89.7\% of the time (140/156140/156, κ=0.795κ=0.795), and the direction of the disagreements matters more than their number: of the 1616 disagreements, 1414 are the judge being too strict—the human considers the migration genuine and the judge scored zero—and only 22 go the other way; Table 6 breaks this down by task. Following all 1616 through the later stages changes almost nothing about the final verdict: of the 1414 too-strict cases, 1212 would have failed the full fixed suite at Stage I anyway; of the 22 too-lenient cases, 11 likewise; and the remaining one reached Stage I, where four of the six agentic verifiers each constructed a counterexample against it, so it was not accepted either. That is, the Stage I error let no submission a human thought was disguised into the accepted set; what it may have undercounted is at most the 22 runs that passed every behavioural check and were zeroed by Stage I alone. The judge’s error has a direction, and that direction makes points harder to earn, not easier. Does the judge spare its own family? It is itself one of the evaluated systems, so leniency towards submissions from the same family would quietly inflate them. That does not appear: gpt-5.6-sol passed 57.1%57.1\% of the 240240 submissions written by a GPT-series model against 72.5%72.5\% of the other 280280. Most of that gap is submission quality rather than provenance. 3.4.2 Agentic Verification Finds Failures Beyond Fixed Tests Stage I and Stage I test two different parts of the same thing, and the clearest way to see it is to put two sets of cases side by side. What Stage I asks is what the task author could think of in advance. The check from Section 3.3.2—whether visiting / settles at /#/—is an observation you get simply by running the original once, which is why it sits in the fixed suite as one of 21,76921,769 checks. Coverage of that kind is already wide: fw03 alone carries more than twenty-one thousand checks. What Stage I asks cannot be anticipated. On fw04 (ChartMuseum, Gin → chi), POST /api/charts has to dispatch between a multipart-form upload handler and a raw-body upload handler; the original truncates Content-Type at the first space or semicolon, while the migrated version truncates only at the semicolon, so a request with one extra space before the boundary reaches a different handler on each side—an implementation detail of Gin whose existence the task author did not know of when writing the suite. Others of the same kind: lang05 (go-yaml, Go → Zig), where go-yaml, by way of Go’s time.Parse, also accepts a comma as the decimal separator, so 2001-12-14T21:59:43,10Z is a !!timestamp on the original and a !!str in the Zig version; and pf01 (SQLite, POSIX → WASI), where after checkpointing a WAL database and switching it back to DELETE mode, native SQLite removes both the -wal and the -shm file while the WASI version removes only -wal. What the two kinds have in common is that each is decisive—if this one check does not pass, the migration is not complete; the only difference is that one can be fixed in advance as a test and the other cannot. That is exactly why both stages are indispensable, and exactly why Stage I has to hand in an executable program rather than a verdict: only an executable counterexample turns “a problem nobody thought of” into a fact that can be re-checked. 3.4.3 Six Verifiers: Strength and Diversity Both Matter Figure 6: Break rate of each of the six verifiers: the share of its rounds in which it handed in an executable counterexample that passes on the original and fails on the submission, coloured by the model behind it and annotated with its reasoning-effort setting. The dashed line marks the lower of the two claude-opus-5 verifiers. Finding 1: a strong coding agent as the verifier matters an order of magnitude more than its configuration. The six verifiers face the same 8888 submissions and each runs 8888 rounds, so task difficulty is identical for them; all that differs is which direction each was assigned. The break rates, however, are far apart (Figure 6): the two held by claude-opus-5 are 55.7%55.7\% and 53.4%53.4\%, the other four between 21.6%21.6\% and 26.1%26.1\%. That gap cannot be attributed to opus happening to draw the easy directions, for two reasons. First, of the two opus verifiers, one probes an assigned direction and the other is unrestricted, so their prompts and effort settings both differ, and yet their break rates differ by only 2.32.3 percentage points. Second, the unrestricted one was given no direction at all—it was free to search the same ground as the other four—and it still broke 53.4%53.4\%. Changing prompt and effort within one model moves the break rate by about two points; changing the model moves it by thirty. This also means the reported numbers are a property of this panel as much as of the submissions: retire the two strongest and the remaining four would accept 4646 submissions instead of 2828. An accepted submission is therefore not a migration proven correct but a migration that survived the strongest adversaries we could field—and as models improve, this same fixed set of 20 tasks will be scored more strictly. Nor are the six redundant. Counting exclusive breaks—where this verifier alone found a counterexample and the other five did not—the two opus verifiers have 55 and 22, and the four weaker ones 44 between them. Even the one with the lowest break rate rejected a submission that the other five let through: they differ in strength, but they are complementary in coverage. Finding 2: models do not spare their own family’s work. This matters, because claude-opus-5 both leads the leaderboard and holds two of the verifiers. Pooled, the effect runs opposite to collusion: verifiers broke 33.9%33.9\% of the submissions written by a model of their own family and 34.3%34.3\% of everyone else’s. The apparent counter-evidence is that the opus verifiers break 40.8%40.8\% of opus-authored submissions against 65.0%65.0\% of everyone else’s; but that fails its own control—the four non-opus verifiers on the same submissions are at 17.1%17.1\% against 29.1%29.1\%, a drop by the same factor. What separates the two is submission quality, not collusion: submissions written by opus are simply harder to break, for everyone. 4 Related Work Table 7: Positioning against related benchmarks. Starts failing means whether the starting state already fails some test in the suite being scored; that is what makes “red to green” an informative verdict—and it is exactly the property SWE Refactor Bench cannot have. Migration criterion means whether a non-behavioural instrument can reject a submission that passed every behavioural check; agentic verifier means whether, at scoring time, further models actively search for differences beyond the fixed tests that the authors did not anticipate. ✓ is yes, ✓ is partial—MigrationBench moves a language version, Java 8 to 17/21, while the language itself stays—and × marks a dimension the benchmark does not aim at, not a defect. Benchmark Unit of work Starts failing Stack changes Migration criterion Agentic verifier SWE-bench [33] one issue ✓ × × × SWE-EVO [38] one feature ✓ × × × SWE-CI [10] CI history ✓ × × × Terminal-Bench [47] one task ✓ × × × TransCoder [53] one function ✓ ✓ × × MigrationBench [44] whole repository ✓ ✓ × × SWE Refactor Bench (this work) whole repository × ✓ ✓ ✓ Repository-level coding benchmarks: the signal is “red to green”. SWE-bench [33] established the paradigm the field now uses: a real repository, a real issue, and a test that fails before the patch and passes after it. The paradigm has since been extended—to other languages [69], to continuously refreshed task streams that resist contamination [71, 32], to building a whole library from scratch [72], and to writing tests rather than patches [48]. A neighbouring group of suites keeps the same criterion and only stretches the horizon: repository evolution (SWE-EVO) [38, 40], long continuous-integration histories (SWE-CI) [10], and hours of terminal work (Terminal-Bench) [47]. The criterion is inherited from function-level suites [11] and fits every one of those task types: while the work is unfinished the test is red, when it is finished it turns green, and that jump is itself the evidence of completion. SWE Refactor Bench’s tasks have no such jump available, because the starting state is already green—which is not a defect of those benchmarks but a property of behaviour-preserving evolution, which simply does not produce that signal. Code migration and reward hacking: fixed tests are not enough. Behaviour-preserving rewriting is not new: translation between languages stayed at the function level for a long time after TransCoder [53], and the difficulty of evaluating it was documented early [49]; the unit later grew to a whole repository [64, 31], and C to Rust acquired dedicated benchmarks of its own [34, 19]; other work moves not the language but a language version [44] or a single API [41]. The changes differ, the scoring does not: take a test suite, run it, count what passes—reusing the original repository’s tests when the language stays the same, and having humans write them on the target side when it does not. For a whole-repository change of stack that scoring is no longer sufficient: a repository handed back as-is still turns any fixed suite green, so fixed tests can say at most “nothing was broken”, never “something was changed”. It is tempting to read this as reward hacking—agents optimise everything their reward leaves unconstrained [54, 36, 4], and coding environments have supplied plenty of documented cases [6, 61], with SWE-bench itself audited item by item for leakage and contamination [43, 22, 65]. That literature tells us what to guard against, but the problem here is not that someone is exploiting a loophole: a repository handed back as-is circumvents no check; it earns full marks by the rules, and the rules simply cannot see whether the migration happened. Since the hole is not in how tight the tests are, no amount of extra tests will close it (Section 2.1); the only remedy is a second check outside behaviour, holding a veto [42]. Model-based judging and differential testing: the tools at the two ends of this work. Both that veto and the search beyond the fixed tests have established methods to draw on. When what must be decided cannot be written as a scoring script, using a model as the judge is standard practice [73], and its known failure modes—position and verbosity bias, self-preference [62, 50]—are exactly why Stage I answers narrow questions with cited evidence over three independent samples instead of producing one global rating, and why we measure its consistency (Section 3.4.1). Stage I is differential testing [46]: with the reference as the other side, this is how compilers have been checked at scale [68, 16], and what a verifier must submit is not an opinion but evidence that runs. The stronger route would of course be formal verification, but translation validation [51], differential symbolic execution [37, 8] and verified compilers [39] all require two sides whose semantics can be related, and a cross-language whole-repository rewrite does not offer that. To approximate the same effect we use agentic verifiers instead: let the strongest coding agents search as hard as they can with both source trees in hand, and if no counterexample comes out, that is the strongest evidence we are able to give. Table 7 places SWE Refactor Bench next to the benchmarks above, dimension by dimension. 5 Conclusion We have presented SWE Refactor Bench: 20 long-horizon whole-repository migrations drawn from real open-source infrastructure, together with a three-stage evaluation protocol that does not rely on behaviour alone—Migration Audit decides whether the migration actually happened and holds a veto, the Behavioural Tests stage decides whether behaviour is unchanged down to the last observation using 130,118130,118 fixed checks recorded from the original, and Agentic Verification sends six coding agents, one hour each, to look for the differences the fixed tests may have missed, accepting nothing but an executable counterexample. Across 520520 evaluations on 88 frontier models and 2626 configurations, only 2828 (5.4%5.4\%) passed all three stages, and 13 of the 20 tasks were solved by nobody. The failures follow a pattern: “getting the migration done” and “not breaking anything” are two different abilities, and agents missed them in opposite directions—a few runs preserved behaviour by skipping the migration and were stopped at Migration Audit, most attempted it and broke behaviour and were stopped at Behavioural Tests. And even when the migration was genuinely done, only 26%26\% of runs passed every fixed check, and two thirds of those still had a counterexample found against them by a coding agent within the hour. There is therefore a long way to go before an agent can complete a whole-repository migration that is genuinely deliverable; and how far along that road we are will be measured not by how much code the agent wrote, but by whether, once it is done, the system is still the same system. References [1] M. Akizuru (2023) Go-simple-upload-server v2.2.0. Note: https://github.com/mayth/go-simple-upload-serverMIT licence Cited by: Table 10, Table 10. [2] A. Albrecht (2016) Sqlparse 0.5.3. Note: https://github.com/andialbrecht/sqlparseBSD-3-Clause licence Cited by: Table 10, Table 10. [3] Automattic (2024) Stylus 0.63.0: an expressive CSS preprocessor. Note: https://github.com/stylus/stylusMIT licence Cited by: Table 10, Table 10. [4] B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi (2025) Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926. Cited by: §4. [5] F. Bellard and C. Gordon (2020) QuickJS 2020-11-08: a small embeddable JavaScript engine. Note: https://bellard.org/quickjs/MIT licence, stated in the header of every source file Cited by: Table 10, Table 10. [6] I. Bercovich, I. Segal, K. Zhang, S. Saxena, A. Raghunathan, and Z. Zhong (2026) Terminal wrench: a dataset of 331 reward-hackable environments and 3,632 exploit trajectories. arXiv preprint arXiv:2604.17596. Cited by: §4. [7] D. J. Bernstein, T. Lange, and P. Schwabe (2012) The security impact of a new cryptographic library. In International Conference on Cryptology and Information Security in Latin America (LATINCRYPT), Cited by: §2.2. [8] C. Cadar, D. Dunbar, and D. Engler (2008) KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), Cited by: §4. [9] Canonical Ltd. (2019) Go-yaml v3.0.1: YAML support for the Go language. Note: https://github.com/go-yaml/yamlApache-2.0 and MIT licences Cited by: Table 10, Table 10. [10] J. Chen, X. Xu, H. Wei, C. Chen, and B. Zhao (2026) SWE-CI: evaluating agent capabilities in maintaining codebases via continuous integration. arXiv preprint arXiv:2603.03823. Cited by: §4, Table 7. [11] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §4. [12] T. Y. Chen, F. Kuo, H. Liu, P. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou (2018) Metamorphic testing: a review of challenges and opportunities. ACM Computing Surveys 51 (1), p. 4:1–4:27. Cited by: Appendix A, §2.3. [13] K. Claessen and J. Hughes (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In ACM SIGPLAN International Conference on Functional Programming (ICFP), Cited by: Appendix A, §2.3. [14] W. Cunningham (1992) The WyCash portfolio management system. In Addendum to the Proceedings on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA Addendum), Experience Report, p. 29–30. Cited by: §1. [15] DeepSeek-AI (2024) DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §3.1. [16] Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang (2023) Large language models are zero-shot fuzzers: fuzzing deep-learning libraries via large language models. In ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), Note: arXiv:2212.14834 Cited by: §4. [17] F. Denis (2026) Libsodium 1.0.20. Note: https://github.com/jedisct1/libsodiumISC licence Cited by: Table 10, Table 10, §2.2. [18] L. P. Deutsch (1996) DEFLATE compressed data format specification version 1.3. Technical report Technical Report RFC 1951, Internet Engineering Task Force. Cited by: §2.2. [19] M. Emre, R. Schroeder, K. Dewey, and B. Hardekopf (2021) Translating C to safer Rust. Proceedings of the ACM on Programming Languages 5 (OOPSLA). Cited by: §4. [20] K. P. Gaffney, M. Prammer, L. Brasfield, D. R. Hipp, D. Kennedy, and J. M. Patel (2022) SQLite: past, present, and future. Proceedings of the VLDB Endowment 15 (12), p. 3535–3547. Cited by: §2.2. [21] J. Gailly and M. Adler (2022) Zlib 1.3.1. Note: https://github.com/madler/zlibZlib licence Cited by: Table 10, Table 10, §2.2. [22] S. Garg, B. Steenhoek, and Y. Huang (2025) Saving SWE-bench: a benchmark mutation approach for realistic agent evaluation. arXiv preprint arXiv:2510.08996. Cited by: §4. [23] R. Geisberger, P. Sanders, D. Schultes, and D. Delling (2008) Contraction hierarchies: faster and simpler hierarchical routing in road networks. In International Workshop on Experimental Algorithms (WEA), Cited by: §2.2. [24] GLM-4.5 Team (2025) GLM-4.5: agentic, reasoning, and coding (ARC) foundation models. arXiv preprint arXiv:2508.06471. Cited by: §3.1. [25] Google Inc. (2015) Jsonnet 0.20.0: a data templating language. Note: https://github.com/google/jsonnetApache-2.0 licence Cited by: Table 10, Table 10. [26] Google (2011) Gson gson-parent-2.10.1. Note: https://github.com/google/gsonApache-2.0 licence Cited by: Table 10, Table 10. [27] GraphHopper GmbH (2024) GraphHopper 11.0. Note: https://github.com/graphhopper/graphhopperApache-2.0 licence Cited by: Table 10, Table 10, §2.2. [28] A. Haas, A. Rossberg, D. L. Schuff, B. L. Titzer, M. Holman, D. Gohman, L. Wagner, A. Zakai, and J. Bastien (2017) Bringing the web up to speed with WebAssembly. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Cited by: §2.2. [29] S. Haase (2018) Miniserve 0.27.1. Note: https://github.com/svenstaro/miniserveMIT licence Cited by: Table 10, Table 10. [30] IBM Corp. (2018) JSONata 2.2.2: a query and transformation language for JSON. Note: https://github.com/jsonata-js/jsonataMIT licence Cited by: Table 10, Table 10. [31] A. R. Ibrahimzada, K. Ke, M. Pawagi, M. S. Abid, R. Pan, S. Sinha, and R. Jabbarvand (2024) AlphaTrans: a neuro-symbolic compositional approach for repository-level code translation and validation. arXiv preprint arXiv:2410.24117. Cited by: §4. [32] N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2025) LiveCodeBench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations (ICLR), Note: arXiv:2403.07974 Cited by: §4. [33] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) SWE-bench: can language models resolve real-world GitHub issues?. In International Conference on Learning Representations (ICLR), Note: arXiv:2310.06770 Cited by: §1, §1, §4, Table 7. [34] A. Khatry, R. Zhang, J. Pan, Z. Wang, Q. Chen, G. Durrett, and I. Dillig (2025) CRUST-bench: a comprehensive benchmark for C-to-safe-Rust transpilation. arXiv preprint arXiv:2504.15254. Cited by: §1, §4. [35] Kimi Team (2025) Kimi K2: open agentic intelligence. arXiv preprint arXiv:2507.20534. Cited by: §3.1. [36] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg (2020) Specification gaming: the flip side of AI ingenuity. Note: DeepMind blog. https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ Cited by: §4. [37] S. K. Lahiri, C. Hawblitzel, M. Kawaguchi, and H. Rebêlo (2012) SYMDIFF: a language-agnostic semantic diff tool for imperative programs. In International Conference on Computer Aided Verification (CAV), Cited by: §4. [38] T. Le, M. V. T. Thai, D. N. Manh, H. P. Nhat, and N. D. Q. Bui (2025) SWE-EVO: benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470. Cited by: §4, Table 7. [39] X. Leroy (2009) Formal verification of a realistic compiler. Communications of the ACM 52 (7), p. 107–115. Cited by: §4. [40] J. Li, G. Li, X. Zhang, Y. Zhao, Y. Dong, Z. Jin, B. Li, F. Huang, and Y. Li (2024) EvoCodeBench: an evolving code generation benchmark with domain-specific evaluations. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Note: arXiv:2410.22821 Cited by: §4. [41] T. Li, R. Li, B. Wang, B. Paulsen, U. Mathur, and P. Saxena (2025) Adversarial agent collaboration for correctness improvements of C to safe Rust translation. arXiv preprint arXiv:2510.03879. Cited by: §4. [42] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. A. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda (2023) Holistic evaluation of language models. Transactions on Machine Learning Research. Note: arXiv:2211.09110 Cited by: §4. [43] S. Liang, S. Garg, and R. Z. Moghaddam (2025) The SWE-bench illusion: when state-of-the-art LLMs remember instead of reason. arXiv preprint arXiv:2506.12286. Cited by: §4. [44] L. Liu, X. Liu, Q. Zhou, L. Chen, Y. Liu, H. Nguyen, B. Omidvar-Tehrani, X. Shen, J. Huan, O. Tripp, and A. Deoras (2025) MigrationBench: repository-level code migration benchmark from Java 8. arXiv preprint arXiv:2505.09569. Cited by: §1, §4, Table 7. [45] J. MacFarlane (2014) Cmark 0.31.1: the CommonMark reference implementation. Note: https://github.com/commonmark/cmarkBSD-2-Clause, MIT and C-BY-SA-4.0 Cited by: Table 10, Table 10. [46] W. M. McKeeman (1998) Differential testing for software. Digital Technical Journal 10 (1), p. 100–107. Cited by: §2.3, §4. [47] M. A. Merrill et al. (2026) Terminal-Bench: benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. Cited by: §4, Table 7. [48] N. Mündler, M. N. Müller, J. He, and M. Vechev (2024) SWT-bench: testing and validating real-world bug-fixes with code agents. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2406.12952 Cited by: §4. [49] R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand (2024) Lost in translation: a study of bugs introduced by large language models while translating code. In IEEE/ACM International Conference on Software Engineering (ICSE), Note: arXiv:2308.03109 Cited by: §4. [50] A. Panickssery, S. R. Bowman, and S. Feng (2024) LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076. Cited by: §4. [51] A. Pnueli, M. Siegel, and E. Singerman (1998) Translation validation. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), Cited by: §4. [52] K. Reitz (2017) Httpbin 0.10.2. Note: https://github.com/psf/httpbinISC or MIT licence Cited by: Table 10, Table 10. [53] B. Rozière, M. Lachaux, L. Chanussot, and G. Lample (2020) Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2006.03511 Cited by: §4, Table 7. [54] J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger (2022) Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2209.13085 Cited by: §4. [55] The Acorn Contributors (2022) Acorn 8.14.0: a small, fast JavaScript parser. Note: https://github.com/acornjs/acornMIT licence Cited by: Table 10, Table 10. [56] The Helm Authors (2022) ChartMuseum v0.15.0. Note: https://github.com/helm/chartmuseumApache-2.0 licence Cited by: Table 10, Table 10. [57] The PyCryptodome Authors (2024) PyCryptodome 3.20.0. Note: https://github.com/Legrandin/pycryptodomeBSD-2-Clause and Unlicense Cited by: Table 10, Table 10. [58] The RealWorld Contributors (2017) Vue-realworld-example-app, commit feb0b7d2. Note: https://github.com/gothinkster/vue-realworld-example-appMIT licence Cited by: Table 10, Table 10. [59] The SQLite Developers (2020) SQLite 3.31.1. Note: https://w.sqlite.org/Public domain; SPDX identifier blessing Cited by: Table 10, Table 10, §2.2. [60] typicode (2015) Json-server 0.17.4. Note: https://github.com/typicode/json-serverMIT licence Cited by: Table 10, Table 10. [61] H. Wang, H. Li, Q. Mang, A. Cheung, K. Sen, and D. Song (2026) Do androids dream of breaking the game? systematically auditing AI agent benchmarks with BenchJack. arXiv preprint arXiv:2605.12673. Cited by: §4. [62] P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui (2024) Large language models are not fair evaluators. In Annual Meeting of the Association for Computational Linguistics (ACL), Note: arXiv:2305.17926 Cited by: §4. [63] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig (2025) OpenHands: an open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR), Note: arXiv:2407.16741 Cited by: §1. [64] Y. Wang, Y. Wang, S. Wang, D. Guo, J. Chen, J. Grundy, X. Liu, Y. Ma, M. Mao, H. Zhang, and Z. Zheng (2024) RepoTransBench: a real-world multilingual benchmark for repository-level code translation. arXiv preprint arXiv:2412.17744. Cited by: §1, §4. [65] C. Xu, S. Guan, D. Greene, and M. Kechadi (2024) Benchmark data contamination of large language models: a survey. arXiv preprint arXiv:2406.04244. Cited by: §4. [66] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §3.1. [67] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2405.15793 Cited by: §1. [68] X. Yang, Y. Chen, E. Eide, and J. Regehr (2011) Finding and understanding bugs in C compilers. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Cited by: §4. [69] D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. Long, K. Shen, and L. Xiang (2025) Multi-SWE-bench: a multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605. Cited by: §1, §1, §4. [70] A. Zeller and R. Hildebrandt (2002) Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering 28 (2), p. 183–200. Cited by: Appendix A, §2.3. [71] L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang (2025) SWE-bench goes live!. arXiv preprint arXiv:2505.23419. Cited by: §1, §1, §4. [72] W. Zhao, N. Jiang, C. Lee, J. T. Chiu, C. Cardie, M. Gallé, and A. M. Rush (2025) Commit0: library generation from scratch. In International Conference on Learning Representations (ICLR), Note: arXiv:2412.01769 Cited by: §4. [73] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2306.05685 Cited by: §4. Appendix A How a Task Is Built Every task is built by the same procedure. This appendix walks through lang01-cmark-c-to-rust: its State A is cmark 0.31.1, the CommonMark reference implementation, 4141 C source files, built by CMake into libcmark.so.0.31.1, libcmark.a and cmark(1); its State B is those same three artifacts, behind the same cmark.h, implemented in Rust 1.90 without any external crate and with the C ABI unchanged. What a task ships with. Instantiating (1) means putting eight things on disk: (i) the archive of State A, rebuilt to confirm that it works; (i) a State B declaration naming the target stack down to fixed versions; (i) the instruction ℐI; (iv) the offline image E, which also carries State A’s own toolchain; (v) an artifact contract saying what will be collected from the finished workspace; (vi) the fixed Stage I expectations, recorded from a reference build of State A and divided into modules; (vii) the migration criteria; and (viii) the further checks made by the agentic verifiers. Items (vi)–(viii) live in a separate image that is never mounted into the agent’s container, so those tests are not merely “unread” by it—they do not exist for it. Pick the debt first, the repository second. We start from a migration a maintainer would call overdue and then look for a project where that migration is the whole job: the old stack has to be load-bearing, so that removing it reaches the design rather than just the imports. Why cmark qualifies is worth spelling out, since CommonMark parsers in Rust already exist: what is wanted here is not a correct parser but this library’s ABI, installation tree and so on, which no existing crate provides. The repository is then trimmed, archived and rebuilt inside the image to confirm it works, with history squashed to a single commit—an upstream log is a place where some past migration may well already be described, and git log is the first thing a competent agent reads. Write the requirements, but not the tests. ℐI states the requirements in full: the files that must survive untouched, how the build has to remain invocable without a network, and in which direction behaviour will be compared. What it never contains is a single one of the concrete behavioural checks—an instruction that enumerates the tests is an instruction to satisfy the tests. Scoring reads source only. An exclusion list is applied when the workspace is collected (build-*, target/, object files, install prefixes, .git), so “the old stack no longer exists” is a fact about source rather than a fact about what the agent happened to leave in some directory. Draw the boundary for the verifiers. The allow list and deny list of Stage I have to agree with ℐI: a scope that forbids what the instruction promised makes the task unwinnable, and a scope that allows what the instruction excluded makes it unloseable. Within that scope a verifier may work the way a real tester does [13, 12, 70]. For lang01, fifteen directions are allowed and eleven forbidden, the latter including internal struct layout, behaviour the original gets wrong as well, and—most worth stating explicitly—the identity of the tree under test: a candidate program that detects which tree it is running against satisfies every mechanical condition for a break while establishing nothing about the migration, and is rejected on discovery rather than weighed. Make the task verify itself. No reference solution ships with a task; in its place there is an identity run at image build time—State A is scored against itself inside the same image, and unless every behavioural module reaches 1.01.0 the build is refused. Stage I — migration criteria Stage I — test modules Criterion What it asks Required Verdict Module What it measures Pass rate no-c-sources has C left the build ✓ pass build two link configurations 1.001.00 no-foreign-headers is only cmark.h left ✓ pass structure install tree, ABI 1.001.00 no-c-in-build is no C compiled ✓ pass conformance CommonMark corpus 1.001.00 rust-present is Rust really there ✓ pass renderers five output renderers 1.001.00 rust-is-primary is Rust the implementation ✓ fail data-tables entities, case folding 1.001.00 no-embedded-reference no prebuilt artifact linked ✓ pass encoding UTF-8 validation 1.001.00 no-verifier-awareness was the scorer let off ✓ pass robustness pathological input 1.001.00 default-path is the scored path default ✓ pass options options and combinations 1.001.00 All eight criteria are required, so one failure streaming incremental parser 1.001.00 zeroes the run. Each is decided by majority tree AST and its invariants 1.001.00 over 3 samples at medium effort—here all iterators iterator API 1.001.00 eight were unanimous. provenance what built these artifacts 1.001.00 cli the installed executable 1.001.00 Any criterion fails ⇒ score 0; Stage I never runs 4,1844,184 checks, none skipped 1.001.00 a perfect Stage I Table 8: One task and one submission in full: claude-opus-5 at high effort on lang01-cmark-c-to-rust. All 4,1844,184 checks across 1515 modules pass, the highest mark Stage I can give; seven of the eight criteria pass, and the eighth, rust-is-primary, fails unanimously because the Rust reproduces the C implementation layout field by field—“the same hand-managed pointer stack, the same names and the same order … this is the original transliterated into unsafe Rust”, cited to src/inlines.rs:55 and neighbouring lines. Final score: 00. Table 8 shows one whole task as the scorer sees it: lang01’s eight criteria and the fifteen modules of Stage I. The run in the table is precisely the sharpest form of disagreement between the stages: all fifteen modules at 1.001.00, the highest mark the behavioural side can give, zeroed by Stage I on a single criterion—every behavioural instrument in the stack calls this migration complete, and the one instrument that reads the code calls it a transliteration. Appendix B Per-Task Breakdown Table 9: The 520520 runs broken down by task. Every task is run once by each of the 2626 configurations (n=26n=26). Column meanings follow the metric definitions of Section 3.1: migrated is the number of runs passing Migration Audit, all tests pass is the number passing every fixed check in Behavioural Tests (regardless of the Stage I verdict), blindness is the subset of those that failed Stage I, accepted is the number passing all three stages, and score is defined in (4). All tests pass −- blindness is therefore the number of runs that reached Stage I. Only 77 of the 2020 tasks ever produced an accepted submission. Migrated ↑ All tests pass ↑ Blindness ↓ Accepted ↑ Score ↑ Stage I passed Stage I perfect passed, not migrated all three passed out of 100 Task Migration n n n n n mean Language rewrites lang01 C → Rust 26 6 5 5 0 0.00 lang02 C → Java 26 18 4 0 0 12.69 lang03 Python → Go 26 14 0 0 0 0.00 lang04 JavaScript → Rust 26 20 0 0 0 0.00 lang05 Go → Zig 26 18 4 0 1 12.69 lang06 C++ → C# 26 10 3 2 0 2.31 lang07 JavaScript → TypeScript 26 14 5 2 3 11.54 Framework rewrites fw01 Flask → Starlette 26 24 1 1 0 0.00 fw02 Express → Fastify 26 13 1 1 0 0.00 fw03 Vue → React 26 26 5 0 2 16.92 fw04 Gin → chi 26 25 7 1 3 21.54 fw05 actix-web → axum 26 22 2 1 0 2.31 fw06 gorilla/mux → net/http 26 13 22 9 9 43.08 fw07 Dropwizard → Spring Boot 26 9 2 2 0 0.00 Platform ports pf01 POSIX → wasm32-wasi 26 15 1 0 0 2.31 pf02 CommonJS → V8 realm 26 12 0 0 0 0.00 pf03 x86-64 → 3 architectures 26 18 20 4 4 49.23 Build-toolchain rewrites build01 Autotools → CMake 26 19 2 0 0 4.23 build02 Maven → Gradle 26 20 13 0 0 25.38 build03 setuptools → Meson 26 24 21 2 6 64.62 All tasks 520 340 118 30 28 13.44 Table 9 takes the funnel of Section 3.2 apart task by task: of the 2626 runs, how many completed the migration, how many passed every fixed check, how many of those fall into blindness, and how many were finally accepted. Of the 2020 tasks, 1313 were never solved by any model, and for seven of them not a single submission even reached the verifiers—and they fail in two different ways. On lang03, lang04, and pf02, no run passed every fixed check. On lang01, fw01, fw02, and fw07, by contrast, some runs passed every fixed check, but every such run was rejected at Stage I as blindness. A behavioural instrument would report these four tasks as solved and the three-stage protocol reports them as unsolved, which is exactly the disagreement this benchmark was built to expose. Appendix C Detailed Task Catalogue Table 10 gives all 2020 tasks one by one; Table 1 in Section 2.2 is its summary by class. Task Upstream project Source stack Target stack LoC B (h) Criteria Modules Checks Language rewrites—the implementation language moves, the artifact does not lang01 cmark 0.31.1 C11 Rust 1.90 22,619 30 8 15 4,184 lang02 zlib 1.3.1 C89 Java 17 22,732 20 10 18 4,162 lang03 sqlparse 0.5.3 Python Go (stdlib) 4,393 15 8 15 8,530 lang04 acorn 8.14.0 JavaScript Rust 1.90 9,571 30 7 13 16,039 lang05 go-yaml v3.0.1 Go Zig 0.14.1 11,967 20 8 22 10,271 lang06 jsonnet 0.20.0 C++11 C# / .NET 8 39,842 20 9 19 2,608 lang07 JSONata 2.2.2 JavaScript TypeScript 5.9 9,247 12 10 15 13,977 Framework rewrites—the language stays; what is replaced is the framework the code is organised around fw01 httpbin Flask (WSGI) Starlette (ASGI3) 2,455 9 5 11 2,830 fw02 json-server Express 4 Fastify 5 2,992 9 5 10 6,760 fw03 RealWorld Conduit Vue 2 + webpack React 18 + Vite 2,261 11 5 10 21,769 fw04 ChartMuseum Gin chi v5 (net/http) 5,401 12 5 14 11,429 fw05 miniserve actix-web 4 axum 0.7 (tower) 3,484 8 6 14 12,466 fw06 uploadserver gorilla/mux net/http ServeMux 803 6 6 9 485 fw07 GraphHopper 11.0 Dropwizard 4 Spring Boot 3.5 94,766 16 7 11 113 Platform ports—the host the code assumes changes pf01 SQLite 3.31.1 POSIX (unix + win) wasm32-wasi 358,006 10 6 7 2,668 pf02 Stylus 0.63.0 CommonJS + Node ESM in a V8 realm 16,012 10 7 12 2,570 pf03 QuickJS x86-64 host layout 3 architectures, endian-clean 89,429 6 6 12 1,487 Build-toolchain rewrites—what produces the artifact changes, and the package is the observable build01 libsodium 1.0.20 Autotools CMake 3.20+ 73,228 6 6 15 4,720 build02 Gson 2.10.1 Maven Gradle (offline) 19,015 6 6 10 2,669 build03 pycryptodome 3.20.0 setuptools Meson 78,839 6 6 12 381 20 tasks 867,062 262 136 264 130,118 Table 10: The task set, task by task. LoC counts newlines in the version-controlled implementation source of State A, excluding tests, vendored trees and documentation, measured the same way for every task rather than transcribed from each project’s own description; together these repositories hold 10,59410,594 version-controlled files. B is the agent’s time budget in hours. Criteria is the number of prompt-form migration criteria, each of them required: a single failure zeroes the submission. Modules is the number of independent test modules and checks the number of fixed behavioural cases they hold, recorded from a reference build of State A. The size of a test suite is set by how much of the product is mechanically observable, not by the size of the repository. Every task has six verifiers of one hour each. Table 1 summarises these rows by class. Each row ships an upstream release verbatim, at the version shown and under its own licence: cmark [45], zlib [21], sqlparse [2], Acorn [55], go-yaml [9], Jsonnet [25], JSONata [30], httpbin [52], json-server [60], the RealWorld Conduit app [58], ChartMuseum [56], miniserve [29], go-simple-upload-server [1], GraphHopper [27], SQLite [59], Stylus [3], QuickJS [5], libsodium [17], Gson [26] and PyCryptodome [57].

Instant research discovery