Paper deep dive

Self-Supervised Scaling of Terminal Environments for Scientific Domains

Zhongzhi Li, Yucheng Shi, Zongxia Li, Junyao Yang, Ruhan Wang, Yu Wang, Jingyuan Huang, Jichao Yu, Ninghao Liu, Haitao Mi, Leowei Liang

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 102

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/5/2026, 6:47:08 AM

Summary

The paper introduces Software-in-the-loop Reconstruction (SWR), a self-supervised framework for generating terminal agent training environments in scientific domains. SWR uses existing software workflows to create tasks by partitioning inputs into public observations and hidden evaluations. An agent reconstructs the workflow behavior from public examples, which is then verified against hidden outputs using a hierarchical semantic verifier. The method scales by reusing workflows across multiple input configurations. Experiments show that fine-tuning Qwen3.8-27B on verified reconstruction trajectories improves performance on Terminal-Bench 2.

Entities (6)

SWR · framework · 98%Qwen3.8-27B · model · 95%Qwen3.8-Max · model · 95%Terminal-Bench 2 · benchmark · 92%Hierarchical Verifier · component · 90%Scientific Domains · domain · 85%

Relation Signals (5)

SWR → uses → Software Workflows

confidence 95% · SWR obtains reference outputs and verification targets from existing software workflows

Qwen3.8-27B → finetunedon → SWR Trajectories

confidence 92% · Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance

Qwen3.8-Max → solves → SWR Tasks

confidence 90% · Qwen3.8-Max solves 838 tasks and produces 1,422 verified trajectories

Hierarchical Verifier → verifies → Agent Outputs

confidence 90% · The candidate is evaluated on hidden configurations against workflow outputs using a hierarchical verifier

SWR → covers → Six Domains

confidence 88% · We instantiate SWR with 500 workflows and 46 software families across six domains

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Terminal agents are increasingly deployed beyond software engineering in science and other specialized domains. Constructing training environments requires executable reference behavior and a domain-specific verifier that distinguishes semantic correctness from superficially plausible artifacts. Authoring these components for each task requires repeated engineering and limits reuse. We introduce software-in-the-loop reconstruction, a self-supervised framework that obtains reference outputs and verification targets from existing software workflows, executable programs mapping structured inputs to outputs. For each workflow, we execute multiple input configurations and partition cases into public observations and hidden evaluations. Given the instruction, input schema, and public input--output observations, an agent constructs an editable program without access to the source workflow. The candidate is evaluated on hidden configurations against workflow outputs. A hierarchical verifier combines domain-specific semantic comparison, structural validity, and anti-shortcut checks, while public feedback supports iterative revision. The construction admits additional workflows and configurations without authoring a reference solution for each task. We instantiate SWR with 500 workflows and 46 software families across six domains. Across three attempts per task, Qwen3.8-Max solves 838 tasks and produces 1,422 verified trajectories, which we oversample to 3,000 reconstruction-only training examples. Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance from 47.94% to 53.56% across three seeds and achieves the highest mean among four matched-token corpus controls on all four reported evaluations. These results indicate that existing scientific software can provide scalable, behaviorally verified supervision for terminal agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

98,999 characters extracted from source content.

Expand or collapse full text

Self-Supervised Scaling of Terminal Environments for Scientific Domains Zhongzhi Li *1,2 , Yucheng Shi *†1 , Zongxia Li *1,3 , Junyao Yang 1,4 , Ruhan Wang 1,5 , Yu Wang 6 , Jingyuan Huang 2 , Jichao Yu 2 , Ninghao Liu ‡7 , Haitao Mi 1 , Leowei Liang 1 1 Tencent HY LLM Frontier 2 University of Georgia 3 University of Maryland, College Park 4 National University of Singapore 5 Indiana University 6 University of Illinois at Chicago 7 Hong Kong Polytechnic University * Equal contribution, † Project lead., ‡ Corresponding author. ninghliu@polyu.edu.hk, zl22754@uga.edu, zhongzhili@global.tencent.com Terminal agents are increasingly deployed beyond software engineering in science and other specialized domains. Constructing training environments in these settings requires both executable reference behavior and a domain- specific verifier capable of distinguishing semantic correctness from superficially plausible artifacts. Authoring these components independently for every task requires repeated task-specific engineering and limits their reuse across tasks. We introduce software-in-the-loop reconstruction, a self-supervised learning framework that obtains reference outputs and verification targets from existing software workflows, defined as executable programs that map structured inputs to outputs. For each workflow, we execute multiple input configurations and partition the resulting cases into public observations and hidden evaluations. Given the instruction, input schema, and public input–output observations, an agent constructs an editable program without access to the source workflow. The candidate is then evaluated on hidden configurations against outputs produced by the workflow. A hierarchical verifier combines domain-specific semantic comparison, structural validity, and anti-shortcut checks, while feedback on public cases supports iterative revision. The construction admits additional workflows and input configurations without independently authoring a reference solution for each task. We instantiate the framework as SWR, which contains 500 workflows and 46 software families across six domains, including physical and engineering simulation, life and molecular sciences, and earth and space sciences. Across three attempts per task, Qwen3.8-Max solves 838 tasks and produces 1,422 verified trajectories, which we oversample to 3,000 reconstruction-only training examples. Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance from 47.94% to 53.56% across three seeds and achieves the highest mean among four matched-token corpus controls on all four reported evaluations. These results indicate that existing scientific software can provide a scalable source of behaviorally verified supervision for terminal agents. SourceLearning taskTraining use Traditional SSL Image masking Raw image predict Predict masked pixelsVisual representation Ours Workflow reconstruction OpenSCAD gears n=6public n=18hidden Agent >_ reconstruct.py n=18 = Semantic check Verified trajectories Terminal-agent SFT Figure 1. Image masking learns visual representations from hidden pixels. Our method reconstructs workflow behavior from public configurations, tests it on hidden configurations, and retains passing trajectories for terminal-agent SFT. 1 arXiv:2610.02710v1 [cs.SE] 2 Oct 2026 1 Introduction Language-model agents can execute programs, coordinate tools, and finish multi-step work from a command line (Luo et al., 2025). Their uses extend from software engineering to scientific computing, engineering analysis, and other specialized domains (Yuan et al., 2026), and benchmarks such as SWE-bench, SWE-Gym, and Terminal-Bench measure these abilities by executable outcomes (Jimenez et al., 2024; Merrill et al., 2026; Pan et al., 2024). Current training data still focus on software engineering, so science and engineering domains remain less covered. Further progress therefore needs tasks grounded in the software workflows used by these domains together with reliable checks of completion. Two obstacles make these tasks hard to build. The first is the domain-specific verifier, which must accept correct programs that differ in algorithm or output format, reject results that only look valid, and apply criteria that differ across science and other domains. Scientific workflows sharpen this obstacle because their outputs can be numerical arrays, meshes, waveforms, biological sequences, or structured tables whose validity cannot be inferred from artifact existence alone. The second obstacle is scale, because existing benchmarks use a separate test or validator for each task (Hendrycks et al., 2021; Jain et al., 2025; Zhou et al., 2024), and repeating that design does not scale. The central problem is to obtain a reference solution and a verification target without writing both from scratch for every task. Software-in-the-loop reconstruction is a self-supervised learning method for this problem. It reuses one existing program across multiple tasks, allowing reference outputs and a common reconstruction interface to be shared. We call this program a workflow, an existing program that maps an input to an output. We run it on several inputs, record the outputs, and show the agent some of these pairs. The agent then writes a new program without seeing the workflow code. We run that program on the remaining inputs and compare its outputs with the workflow outputs, so the workflow is both the reference solution and the source of the expected outputs. One workflow therefore supplies reference behavior for many tasks through a shared interface. New inputs yield further tasks for science, engineering, and other specialized domains. Figure 1 contrasts this procedure with conventional image-based self-supervision. Image masking predicts withheld pixels and transfers the resulting visual representation to later tasks. Software-in-the-loop reconstruction instead divides workflow executions into public and hidden configurations. In the illustrated OpenSCAD example, the public configuration has six gear teeth and the hidden configuration has eighteen, so the reconstructed program must respond to a changed parameter rather than reproduce the observed geometry. Only interactions that pass hidden semantic verification become terminal-agent SFT trajectories. The setup follows programming by example and behavioral system identification (Gulwani, 2011; Gulwani et al., 2017; Ljung et al., 1987). Domain-specific comparison rules are still required, and we test them in a separate audit after the verifier is frozen and in a blinded human review. Figure 2 illustrates how a software workflow supplies public evidence, the input–output examples released to the agent, and hidden reference outputs on withheld inputs. These support task construction, verification, and collection of training trajectories. We instantiate this procedure as SWR. The construction scales directly because each workflow is wrapped once by adapting it to a common program interface and is then reused under new input configurations. Additional workflows enter through the same checks. The experiments below use a release built from 500 workflows across 46 software families, where a family groups workflows that share a common tool. The release connects software used in physical science, molecular analysis, and earth and space science with supporting formal, spatial, and data workflows. Qwen3.8-Max solves 22.8% of tasks in the first evaluation round and 27.9% at least once across three attempts, yielding 1,422 verified trajectories from 838 tasks. Many unsuccessful runs produce executable programs but fail the hidden behavioral checks, indicating that artifact emission alone is insufficient for reconstruction success. To evaluate the training value of the verified trajectories, we construct an oversampled reconstruction-only corpus for supervised fine-tuning (SFT) of Qwen3.8-27B. We evaluate the resulting models on Terminal-Bench 2.0 (TB2), Terminal-Bench 4.0 (TB4), Long-Horizon Terminal-Bench (LHTB), and SWR100, a held-out 100-task subset of SWR, using three seeds and matched-token controls that allocate every corpus the same number of assistant-loss tokens. 2 ObserveReconstructVerify Public scenariosHidden scenarios Workflow oracle Public observations Observed workflow behavior Task instruction Agent Executable reconstruction Public feedback Hidden scenarios Semantic evaluation Semantic diagnostics Strict pass/fail Figure 2 Software-in-the-loop reconstruction. Public observations are the input–output examples released to the agent. They guide construction of a reusable program. Withheld inputs supply the reference outputs used for verification. Our main contributions are as follows: 1.We formulate software-in-the-loop reconstruction for task construction in science and other special- ized domains, so that each task is derived from one executable workflow. Each workflow supplies a reference solution, public examples, and hidden reference outputs for multiple task variants, deriving reference behavior and verification targets from the same executable source rather than specifying them independently. 2.We construct SWR and a hierarchical semantic verifier. The task set scales by adding workflows and input configurations under fixed release checks. We assess sensitivity of workflow outputs to input changes, predefined replay controls that resubmit public outputs, and verifier reliability, including an independent audit conducted after the verifier is frozen. 3. We characterize reconstruction difficulty and evaluate verified trajectories as reconstruction-only SFT data, reporting three-seed results and comparisons with four matched-token corpus controls on down- stream terminal-agent evaluations. 2 Related Work Executable agent tasks and environment scaling. Executable evaluation has moved from single programs to multi-step interaction and, more recently, to large collections of terminal environments. APPS and LiveCodeBench score a generated program by executing it (Hendrycks et al., 2021; Jain et al., 2025). InterCode, AgentBench, GAIA, WebArena, and OSWorld carry the same criterion into interaction with digital environments (Liu et al., 2024; Mialon et al., 2024; Xie et al., 2024; Yang et al., 2023; Zhou et al., 2024). SWE-bench, SWE-agent, SWE-Gym, OpenHands, and Terminal-Bench concentrate on repository repair and command-line agents (Jimenez et al., 2024; Merrill et al., 2026; Pan et al., 2024; Wang et al., 2025; Yang et al., 2024). Later pipelines increase the supply of environments, tasks, and trajectories for terminal-agent training (Ivison et al., 2026; Peng et al., 2026; Pi et al., 2026; Raoof et al., 2026; Shen et al., 2026). In these settings the expected behavior is still specified separately for each task, usually by a written test or a task-specific end state. Software-in-the-loop reconstruction instead takes an existing workflow as the reference solution and obtains the expected outputs by executing that workflow, so the reference and the verification target come from the same program. Executable feedback and behavioral verification. The test-oracle problem is the difficulty of stating expected behavior when an exact output is costly to specify (Barr et al., 2014). Differential testing looks for disagreement among implementations (McKeeman, 1998), and metamorphic testing checks relations that should survive a transformation of the input (Chen et al., 2020). ReAct, Reflexion, Self-Refine, Voyager, and CodeRL use interaction or execution feedback to revise later actions (Le et al., 2022; Madaan et al., 2023; Shinn et al., 3 2023; Wang et al., 2023; Yao et al., 2022). The verification problem is particularly important for terminal agents used in science, where numerical tolerance, geometric equivalence, and domain-specific invariants may each define correctness. The hierarchical verifier in this paper has a narrower function. It compares a candidate output with the output of the source workflow under domain-specific rules that admit differences in implementation and serialization. Comparisons on public inputs guide revision during the interaction, whereas acceptance requires agreement on hidden inputs, structural compliance, and the predefined anti-shortcut checks. Program induction and system identification. Programming by example infers a program from input–output observations (Gulwani, 2011; Gulwani et al., 2017; Huang et al., 2026; Li et al., 2026). Neural and neuro- symbolic methods learn a search procedure or a reusable abstraction rather than one program for one set of examples (Balog et al., 2016; Devlin et al., 2017; Ellis et al., 2021; Reed & De Freitas, 2015). System identification estimates a predictive model from observations and interventions (Brunton et al., 2016; Ljung et al., 1987), and world models learn dynamics used for prediction and planning (Ha & Schmidhuber, 2018; Hafner et al., 2019; Schrittwieser et al., 2020). We use this inferential structure to build terminal-agent training tasks. The agent must return an editable program consistent with the public executions of a workflow, and the program is retained only when it agrees with the workflow on hidden inputs. Agreement on a finite held-out set does not amount to recovery of the original implementation or of the underlying mechanism. 3 Software-in-the-Loop Reconstruction 3.1 Task Definition Each task is derived from an existing software workflow. The agent observes an instruction and public input–output examples, then constructs an editable programg θ that reproduces the workflow behavior. Public tests provide feedback during revision, while hidden tests are used only for final evaluation. The workflow source, hidden inputs, and hidden reference outputs remain withheld, so the objective is behavioral reconstruction rather than source-code recovery. This formulation is well suited to scientific software because simulators and analysis pipelines already expose parameterized runs and structured outputs. Formally, a software workflowdis an executable functionf d . A scenariosis one input configuration, and f d (s) is the output obtained by executing the workflow on that configuration. For each workflow, we construct a reconstruction task T = (d,f d ,O pub ,S pub ,S hid ,J,I), S pub ∩ S hid =∅, O pub =(s,f d (s)) : s∈ S pub . (1) The setsS pub andS hid are disjoint public and hidden scenario sets, andO pub contains the public input–output examples. The instruction isI, andJis the private semantic judge. The source workflowf d produces the reference outputs, against which the agent’s editable program g θ is evaluated on the hidden scenarios. A failure on a hidden input. A hidden input can expose an error that the public examples leave undetected. On a task that repairs mass and inertia fields in a URDF robot description, a recorded Qwen3.8-Max program passes all six public scenarios but fails when it attempts to convert an unseen categorical value to an integer. A GPT-5.6 Sol reconstruction of the same task accepts the unseen value and passes. The contrast isolates a failure of hidden-input robustness after agreement on all public observations. Appendix J reports the matched trajectories. 3.2 Task Generation from Workflows Each workflow is reused to form six task variants that differ in the released examples and in the hidden evaluation set. Public evidence is the set of input–output examples released to the agent, and a hidden evaluation scenario is an input configuration withheld until the submitted program is tested. A typed scenario 4 Weak structural checksSemantic verification Brittle byte-identical matching Workspace exists Source-like files Accepted Hidden behavior Constant output False positive Oracle:Reconstruction: "value": 1.0, "status": "ok" "status": "ok", "value": 1.000000 Byte comparison Rejected False negative Weak structural verificationSemantic verification Byte-identical verification Reconstruction Behavioral evaluation Checks behavioral correctness Allows equivalent representations Behaviorally correct? Fail No Strict success Yes Behaviorally strict, representation tolerant Figure 3 Comparison of verification standards. Structural checks can accept an invalid constant output, while byte-identical comparison can reject a behaviorally equivalent output. Semantic verification evaluates task-relevant behavior and is designed to reduce both failure modes. generator draws these configurations from a schema in which every field has a declared type, such as numeric, categorical, or path-valued, and returns disjoint public and hidden sets for each variant. A variant is eligible for release only when the workflow executes successfully, the outputs are deterministic, the output schema is stable, and behavior varies measurably across scenarios. The wrapped workflow, obtained by adapting the source workflow to the reconstruction interface, must satisfy the verifier, and three public-output replay controls must fail. Each control returns an output taken from the public examples instead of computing the output on the hidden input. These release conditions are fixed for SWR and apply unchanged when further workflows are added, so the collection grows by wrapping new workflows and sampling new configurations rather than by redesigning each task. This reuse is important for workflows used in science because one simulator or analysis pipeline can define many tasks through controlled changes of its inputs. The experiments use the release obtained from 500 workflows. Variants that share a workflow also share its executable mechanism and are not independent software systems. Appendix A states the synthesis algorithm and the interface. 3.3 Hierarchical Semantic Verification A task contains public testsS pub and hidden testsS hid . Public tests return feedback during interaction, while hidden tests are reserved for final evaluation. For either splitS x , the judgeJassigns a score in [0,1] by comparing the candidate outputg θ (s) with the workflow outputf d (s). Execution or parsing failure receives zero. The process reward is the mean score over the tests in that split, r x (T,g θ ) = 1 |S x | X s∈S x J(g θ (s),f d (s)), x∈pub, hid,(2) wherexdenotes the public or hidden split. The resulting rewards arer pub =r pub (T,g θ ) andr hid =r hid (T,g θ ). The public reward guides revision, whereas the hidden reward remains withheld. We definePass(T,g θ ) = 1 only whenr hid = 1 and the candidate also passes the structural and anti-shortcut checks. These checks require valid execution and reject public-output replay or input-independent behavior. 3.4 Reconstruction SFT An interaction trajectory is retained for supervised fine-tuning (SFT) only when its final program satisfies the strict acceptance criterion. Letτ i denote a recorded trajectory for taskT i , and letg i denote its final 5 reconstructed program. The eligible pool is D SFT =τ i | Pass(T i ,g i ) = 1.(3) Trajectories whose task identifiers appear in an evaluation set are removed, and training entries are then sampled from the remaining pool. The resulting collectionB SFT is a multiset, because the same trajectory may be drawn more than once. Each draw enters the loss, so repetition reweights a trajectory without introducing an independent solution. The model with parametersφis trained to predict each recorded action from the interaction history that precedes it, L SFT (φ) =− X τ∈B SFT X t logp φ (a t | h t ),(4) wheretindexes agent steps,h t is the interaction history before stept, anda t is the recorded action. The termp φ (a t | h t ) is the probability assigned to that action. Each repeated copy of a trajectory enters the sum separately. 4 Experiments The experiments treat software-in-the-loop reconstruction as a source of training data for terminal agents in science and other specialized domains. They ask whether the generated tasks span heterogeneous scientific workflows and require the intended input-dependent behavior, whether the semantic verifier accepts valid outputs and rejects invalid ones, whether current agents solve the reconstruction tasks, and whether fine-tuning on verified reconstruction trajectories improves downstream terminal-agent benchmarks. 4.1 Experimental Setup Task sets. Task validity and verifier behavior are assessed on the full SWR release during development. An independent audit after the verifier is frozen covers 100 tasks. SWR100 contains 100 in-domain evaluation tasks whose identifiers are excluded from the final SFT corpus. It measures performance on held-out instances of the same reconstruction setting, and it does not measure generalization to workflows that were never used to build tasks. TB2, TB4, and LHTB measure transfer of terminal-agent skills rather than performance on end-to-end science problems. The final reconstruction-SFT corpus contains no auxiliary terminal-task data and shares no training source with these benchmarks. Agent evaluation. The first full-suite round covers every released task once. Qwen3.8-Max-0902 is evaluated on SWR with Terminus-2, a terminal-agent scaffold, under one attempt of at most 100 turns and one hour. Eight API-served models are evaluated separately on SWR100, with at most 500 turns and three hours per task. The primary metric is the strict hidden success defined in Section 3.3, which requires a valid program, complete agreement on the hidden inputs, and satisfaction of the anti-shortcut checks. Partial semantic agreement is used only for diagnosis. A missing or unscored run is counted as a failure. Repeated sampling and SFT. Repeated-attempt success, and the verified trajectories used for training, come from three evaluations of Qwen3.8-Max on each SWR task. The attempts are the first full-suite round and two further rounds that use the same model and the finalized verifier. The reported 22.8% Pass@1 is the success rate of the first round, rather than an average over rounds. Pass@3 is the fraction of tasks solved at least once across the three attempts. Let c i denote the number of successful attempts on task i. Then Pass@3 = 1 N N X i=1 1[c i > 0],(5) 6 0102030 Share of tasks (%) Software & Formal Systems Physical & Engineering Simulation Spatial, Visual & 3D Life & Molecular Sciences Data & Document Workflows Earth & Space Sciences 660 · 11 fam. 588 · 9 fam. 528 · 8 fam. 468 · 7 fam. 396 · 6 fam. 360 · 5 fam. (a) Task and family coverage across the six software domains in SWR. 0255075 Tasks containing modality (%) Tables Logs / text Structured JSON Images Audio Scientific data Source code Biological sequence CAD / geometry Markup PCAP Configuration 2,028 1,776 432 420 276 276 156 132 132 60 60 54 (b) Artifact types produced by the source programs in SWR. One task may contribute more than one type. 345 Num. String Bool. Array Mixed Dimensions / parameter type 0 25 50 75 100 Share (%) 6.76.7 86.7 66.1 26.7 5.2 1.9 0.1 Tasks by dimensionsAxes by type (c) Types and dimensions of scenario parameters in SWR. Figure 4 Composition of SWR. Panel (a) is the task distribution across software domains. Panel (b) is the artifact types. Panel (c) is the scenario parameter types and dimensions. whereN= 3000 is the number of tasks in the evaluated release and 1[c i >0] equals 1 when taskiis solved at least once and equals 0 otherwise. A missing attempt is counted as a failure. Each reconstruction-SFT budget is trained as three independent checkpoints with seeds 42, 43, and 44. The Base runs use the same evaluation seeds. Reported values are the mean and standard deviation across the three runs. 4.2 Behavioral Validity of SWR The full SWR release is used to test whether a change of input changes the workflow output, and whether a procedure that returns an already released output can satisfy the hidden checks. Coverage. The evaluated release is constructed from 500 workflows, 46 software families, and six domains. It includes workflows used in physical science, life and molecular sciences, and earth and space sciences, together with formal, spatial, and data workflows that support executable analysis. No family contributes more than 2.4% of the tasks. The release contains 15,600 public scenarios and 8,400 hidden scenarios, each an input configuration. Figure 4 summarizes the domain distribution, the artifact types, and the input parameters. Appendix B gives the full classification. Table 1 Composition of the evaluated SWR release. A workflow is an existing program that maps an input to an output. The columns count tasks, software families, workflows, and public and hidden input configurations. Dataset Tasks Families Workflows Public scenarios Hidden scenarios SWR3,0004650015,6008,400 Sensitivity to input changes. For workflows used in science, input sensitivity is necessary because a simulator or analysis pipeline that ignores an intervention cannot provide a meaningful hidden target. For each task the workflow is therefore executed under different input configurations, and the outputs are compared. A change is recorded when at least one output field used to assess task behavior differs between two executions. Every compared pair changes for 2,984 tasks. For each of the remaining 16 tasks, 35 of the 36 pairs change. Across all tasks, 75.1% of these output fields vary with the input configuration, and an average pair differs in 66.3% of the fields. On 2,400 tasks a single declared parameter can be varied while the others are held fixed. Doing so changes the output for 89.5% of the declared parameters. 7 0255075100 Mean coverage across tasks (%) Scenario pairs producing any semantic change Semantic fields varying across scenarios Fields changed in an average scenario pair Declared parameters with a measurable effect 100.0% n=3,000 75.1% n=3,000 66.3% n=3,000 89.5% n=2,400 Figure 5 Sensitivity of SWR outputs to input changes. The four metrics record whether a pair of configurations changes any output, how many semantic fields vary, how many fields change in an average pair, and how many declared parameters affect the output. Public-output replay controls. Three predefined controls return a public output instead of reconstructing the workflow. The first returns the most frequent public output. The second returns the output of the public scenario whose inputs are nearest the hidden inputs. The third is an oracle control that consults the hidden reference in order to select the best-matching public output. The controls reach 34.3–44.3% agreement on hidden output fields and meet the strict acceptance requirements on no released task. A task enters the release only when all three controls fail those requirements, so the zero-success result restates the selection rule. It does not show resistance to shortcuts other than the three that were tested. Appendix B reports the remaining statistics on input changes and public-output replay. 4.3 Verifier Reliability The verifier is tested for acceptance of valid outputs and rejection of invalid ones. During development, 35,873 controlled cases are constructed on the full SWR release. They cover outputs near a numerical tolerance boundary, structurally valid but incorrect outputs, outputs taken from the wrong scenario, and modified geometry. The cases are used to set comparison tolerances and to inspect verifier behavior before the configuration is frozen. The frozen verifier is then compared with a blinded review of 200 cases from the post-freeze audit, of which 100 are valid and 100 are invalid. Relative to the adjudicated human labels, the verifier accepts 2 invalid cases and rejects 2 valid cases, which corresponds to a false-acceptance rate and a false-rejection rate of 2.0% each. On this review the verifier agrees with the adjudicated label in most cases. Appendix I gives the protocols, the per-class results, and the geometry errors. 4.4 Agent Performance Full-suite evaluation. In a single attempt, Qwen3.8-Max solves 685 tasks, which is 22.8% of the scheduled SWR release. Records exist for 2,978 tasks, and the success rate on that subset is 23.0%. Pass@1 is computed over all 3,000 scheduled tasks, and a missing record is counted as a failure. Failure structure. Of the 2,781 evaluated workspaces that contain an executable candidate, 1,591 emit the same output on different hidden scenarios and 75 replay a public output. The presence of a gradeable program is therefore not sufficient for the required hidden behavior. Successful runs end at a median of 58 turns, against 85 turns for failures attributed to the model. The comparison records an association between length and outcome. It does not identify interaction length as a cause of failure. Appendix D gives the accounting. 8 0510152025 Strict Pass@1 (%) GPT-5.6 Sol Kimi-K3 Grok 4.6 DeepSeek-V4-Pro GLM-5.3 MiniMax-M3 Qwen3.8-Max Doubao Seed 2.1 20/100 6/100 4/100 3/100 3/100 3/100 3/100 2/100 (a) Strict Pass@1 on SWR100. The rate uses all 100 scheduled tasks. Software & Formal Systems Physical & Engineering Simulation Spatial, Visual & 3D Life & Molecular Sciences Data & Document Workflows Earth & Space Sciences GPT-5.6 Sol Kimi-K3 Grok 4.6 DeepSeek-V4-Pro GLM-5.3 MiniMax-M3 Qwen3.8-Max Doubao Seed 2.1 7/197/205/191/150/140/13 3/191/202/190/150/140/13 3/190/201/190/150/140/13 3/190/200/190/150/140/13 3/190/200/190/150/140/13 3/190/200/190/150/140/13 3/190/200/190/150/140/13 2/190/200/190/150/140/13 0 5 10 15 20 25 30 35 Pass@1 (%) (b) Strict successes by domain on SWR100. Each cell is the number of passed tasks over the number of scheduled tasks. Figure 6 Eight-model comparison on SWR100. Panel (a) is overall strict success. Panel (b) is success by domain. Table 2 Single-attempt evaluation of Qwen3.8-Max on SWR with the finalized verifier. Pass@1 uses all 3,000 scheduled tasks. Missing records count as failures. Scheduled Records Strict pass Pass@1 Record rate Pass/record 3,000 2,978685 22.8%99.3%23.0% Eight-model comparison. Eight API-served models are evaluated on the same 100 SWR100 tasks under one interaction budget. GPT-5.6 Sol reaches 20.0% Pass@1, and the other models reach between 2.0% and 6.0% (Figure 6). Of the 44 successful runs, 43 fall in Software & Formal Systems, Physical & Engineering Simulation, or Spatial, Visual & 3D. Only one success occurs in Life & Molecular Sciences, and no success occurs in Data & Document Workflows or in Earth & Space Sciences. The concentration shows that SWR100 remains difficult and that its explicitly scientific domains are especially challenging for current agents. Figure 7 Eight-model outputs for an OpenSCAD snap-fit task with a 16 m public example and a 21 m hidden reference. GPT-5.6 Sol preserves separate body and lid components, whereas the other seven outputs collapse to plate-like meshes at wall-thickness height. A task-level reconstruction case. Figure 7 shows the eight submitted meshes for one OpenSCAD snap-fit task. The public configuration and hidden reference differ in box height, from 16 m to 21 m, while retaining the two-part body-and-lid structure. GPT-5.6 Sol is the only evaluated model whose output preserves separate 9 body and lid components. The other seven outputs collapse to plate-like meshes whose vertical extent follows the wall thickness rather than the requested box height. These outputs are valid mesh artifacts, but they do not reproduce the workflow dependence on the input configuration. The case therefore illustrates why artifact existence and geometric validity are weaker criteria than hidden behavioral agreement. 4.5 Reconstruction SFT The remaining experiments ask whether fine-tuning on verified reconstruction trajectories improves held-out reconstruction tasks and external terminal-agent benchmarks. Verified trajectory collection. Qwen3.8-Max is run three times on each SWR task. At least one attempt succeeds on 838 tasks, which is 27.9% Pass@3, and the successful attempts yield 1,422 verified trajectories. Evaluation task identifiers are removed before training entries are sampled. Appendices C and E give the repeated-attempt counts and the corpus construction. Table 3 Three-attempt result of Qwen3.8-Max on SWR. The columns give verified tasks or trajectories and the corresponding rate. ResultVerified tasks/trajectories Rate Pass@3 / task union838 27.9% All verified trajectories1,422– Training-entry budget. Qwen3.8-27B is fine-tuned on 750, 1,500, and 3,000 reconstruction training entries, with three seeds at each budget. A budget counts sampled entries and therefore includes repeated trajectories. At 3,000 entries the mean rises from 47.94% to 53.56% on TB2, from 0.51% to 3.54% on TB4, from 20.67% to 27.67% on LHTB, and from 1.33% to 3.33% on SWR100. On every evaluation the mean is non-decreasing across the tested budgets, which is consistent with a gain from additional sampled entries under this sampling procedure. Table 4 Training-entry budgets for Qwen3.8-27B reconstruction SFT, including repeated samples. Entries are means and standard deviations over seeds 42, 43, and 44. Each seed is trained and tested separately. All entries are percentages. Training entriesTB2TB4 LHTB reward SWR100 success Base 47.94± 1.30 0.51± 0.87 20.67± 1.15 1.33± 0.58 SFT-750 50.19± 0.65 1.01± 0.87 22.33± 0.58 1.33± 0.58 SFT-1500 50.56± 1.12 2.02± 0.87 24.33± 1.53 2.67± 0.58 SFT-3000 53.56± 0.65 3.54± 0.87 27.67± 0.58 3.33± 0.58 Matched-token controls. Reconstruction supervision is compared with four other agent-training corpora at a matched token budget. OpenThoughts-Agent-v1 supplies curated trajectories for general-purpose agents (Raoof et al., 2026). TMax builds terminal-agent data from a structured task taxonomy and from environments that use different verifiers (Ivison et al., 2026). LiteCoder-Terminal generates synthetic executable long-horizon terminal environments together with expert trajectories (Peng et al., 2026). Nemotron-Terminal combines adapted existing tasks with skill-based synthetic generation (Pi et al., 2026). Reconstruction SFT and the four controls each receive 88.48M assistant-loss tokens. Reconstruction SFT attains the highest mean on all four evaluations, so the ordering is not explained by a larger assistant-loss-token budget. 10 Table 5 Matched-token comparison of Qwen3.8-27B training corpora. Entries are means and standard deviations over three seeds, reported as percentages. Every fine-tuned model receives 88.48M assistant-loss tokens. The reconstruction corpus contains 3,000 sampled entries, including repeats. Training corpusLoss tokensTB2TB4LHTBSWR100 Base (no SFT)– 47.94± 1.30 0.51± 0.87 20.67± 1.15 1.33± 0.58 OpenThoughts-Agent-v188.48M 49.06± 1.30 0.00± 0.00 22.67± 1.53 1.67± 0.58 TMax88.48M 49.44± 2.25 1.01± 0.87 20.67± 1.53 1.33± 0.58 LiteCoder-Terminal88.48M 48.69± 1.72 0.51± 0.87 23.00± 1.00 1.00± 1.00 Nemotron-Terminal88.48M 47.19± 1.12 0.51± 0.87 23.67± 1.15 1.33± 1.53 Reconstruction (ours)88.48M 53.56± 0.65 3.54± 0.87 27.67± 0.58 3.33± 0.58 5 Conclusion We introduced software-in-the-loop reconstruction, a self-supervised learning method that addresses the difficulty of building a domain-specific verifier for science and other specialized domains. A workflow, an existing program that maps an input to an output, serves as the reference solution and generates public examples and hidden reference outputs under different inputs. This places reference generation and verification targets on a common executable source, while domain-specific comparison rules remain necessary. We instantiate the procedure as SWR. The construction extends by adding workflows and input configurations, because each workflow supplies both the reference solution and the verification targets. The reported release spans 500 workflows and 46 software families, including software used in physical science, life and molecular sciences, and earth and space sciences. Supervised fine-tuning on verified reconstruction trajectories improves mean downstream terminal-agent performance across three seeds. Under a matched assistant-loss-token budget, reconstruction SFT achieves the highest mean performance on all four evaluations among the compared training corpora. These results support workflow reuse for scaling executable environments in science and other specialized domains, but the downstream evaluations measure terminal-agent transfer rather than autonomous scientific discovery. Future work can assess generalization to unseen scientific workflows and separate the contribution of additional unique trajectories from that of repeated sampling. References Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989, 2016. Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. The oracle problem in software testing: A survey. IEEE transactions on software engineering, 41(5):507–525, 2014. Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016. Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543, 2020. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, p. 990–998. PMLR, 2017. Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation, p. 835–850, 2021. 11 Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices, 46 (1):317–330, 2011. Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program synthesis. Foundations and trends in programming languages, 4(1-2):1–119, 2017. David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2(3):440, 2018. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, p. 2555–2565. PMLR, 2019. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021. Jingyuan Huang, Zuming Huang, Yucheng Shi, Tianze Yang, Xiaoming Zhai, Wei Chu, and Ninghao Liu. Trust the right teacher: Quality-aware self-distillation for gui grounding. arXiv preprint arXiv:2606.18101, 2026. Hamish Ivison, Junjie Oscar Yin, Rulin Shao, Teng Xiao, Nathan Lambert, and Hannaneh Hajishirzi. Tmax: A simple recipe for terminal agents. arXiv preprint arXiv:2606.23321, 2026. Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, volume 2025, p. 58791–58831, 2025. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, p. 54107–54157, 2024. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328, 2022. Zhongzhi Li, Xuansheng Wu, Yijiang Li, Lijie Hu, and Ninghao Liu. Less is enough: synthesizing diverse data in feature space of llms. arXiv e-prints, p. arXiv–2602, 2026. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, p. 52989–53046, 2024. Lennart Ljung et al. Theory for the user. System identification, 45, 1987. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460, 2025. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36:46534–46594, 2023. William M McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100–107, 1998. Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, volume 2026, p. 40903–40986, 2026. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In International Conference on Learning Representations, volume 2024, p. 9025–9049, 2024. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024. Xiaoxuan Peng, Kaiqi Zhang, Xinyu Lu, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. Litecoder- terminal: Scaling long-horizon terminal environments for learning language agents. arXiv preprint arXiv:2605.29559, 2026. 12 Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, and Wei Ping. On data engineering for scaling llm terminal capabilities. arXiv preprint arXiv:2602.21193, 2026. Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, Atula Tejaswi, Ryan Marten, Charlie F Ruan, Tyler Griggs, Alexander Glenn Shaw, Hritik Bansal, et al. Openthoughts-agent: Data recipes for agentic models. arXiv preprint arXiv:2606.24855, 2026. Scott Reed and Nando De Freitas. Neural programmer-interpreters. arXiv preprint arXiv:1511.06279, 2015. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru, Aznaur Aliev, Jay Rainton, Ahmed Awelkair, Zhichen Zeng, Jiajun Li, Shi Dong, Yueming Yuan, et al. Seta: Scaling environments for terminal agents. arXiv preprint arXiv:2607.10891, 2026. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, p. 65882–65919, 2025. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826–23854, 2023. John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, Yi Bin, Wenqi Shao, Chen Qian, Wei Ye, Yujuan Ding, Zheng Wang, Pengpeng Zeng, et al. Terminal agents: A survey of ai agents in command-line environments. 2026. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, p. 15585–15606, 2024. 13 Appendix This appendix records the task interface, the release audit, the evaluation protocol, and the additional experimental results. Sections A–J are organized as follows. • A. Benchmark Specification. Appendix A describes the task-generation procedure, reconstruction interface, release checks, semantic scoring, public feedback, and final evaluation protocol. •B. Dataset Composition and Audit. Appendix B summarizes the domain and software-family coverage of SWR, the output type of each family, example artifacts from each domain, the scenario parameters used in the tasks, behavioral sensitivity to scenario changes, public-output replay baselines, and remaining dataset limitations. • C. Repeated-Attempt Pass@3 Analysis. Appendix C reports Pass@3 over three attempts, the number of tasks solved across repeated runs, and the number of software families that remain unsolved. •D. Full-Suite Diagnostics. Appendix D provides detailed results for the full SWR evaluation, including strict successes, common failure modes, interaction lengths, and family-level difficulty. •E. Reconstruction SFT Corpus Construction. Appendix E describes how verified reconstruction trajectories are converted into the final SFT corpus through oversampling, within-trajectory cleanup, conversation repair, and length filtering. •F. Reconstruction Fine-Tuning Scaling and Controls. Appendix F reports the complete per-seed results for Base, SFT-750, SFT-1500, and SFT-3000, together with matched-token comparisons against OpenThoughts-Agent, TMax, LiteCoder-Terminal, and Nemotron-Terminal. •G. SWR100 Model and Domain Results. Appendix G summarizes aggregate model outcomes and the distribution of successful tasks across software domains. •H. SWR100 Interaction and Statistical Diagnostics. Appendix H provides complete per-model accounting, analyzes interaction length and token usage, and states the uncertainty introduced by evaluating each model–task pair only once. •I. Verifier Mutation Stress Tests. Appendix I evaluates the verifier on controlled output modifications, including tolerance-boundary cases, missing input-dependent behavior, wrong-scenario outputs, semantic perturbations, and geometry changes. • J. Failure Signatures and Matched Trajectory Cases. Appendix J analyzes failure modes across all 800 SWR100 model–task pairs and compares trajectories from different models on the same tasks using private-verifier feedback. 14 A Benchmark Specification Task materialization. Each source workflow is parameterized by a typed scenario schema and can produce multiple reconstruction variants. The reported release uses 500 workflows, with six variants of each. Before agent interaction begins, scenario generators sample disjoint public and hidden parameter settings, execute the source workflow, and materialize the resulting artifacts. In total, the release contains 15,600 public scenarios (5.2 per task on average) and 8,400 hidden scenarios (2.8 per task). The agent receives the instruction, the scenario schema, the public scenarios, and the public execution artifacts. The source workflow, the hidden inputs, the hidden references, and the private verifier state are withheld. Task-synthesis algorithm. Algorithm 1 gives the complete task-generation procedure used to construct the reported SWR release from the 500 source workflows. The same procedure admits further workflows without changing the task interface. For a fixed workflowd, superscript (k) denotes itsk-th task variant. Thus,T d,k instantiates the taskTdefined in Eq. 1. The wrapped reference programg ⋆ d implements the reconstruction interface and is distinct from the agent-generated programg θ . Eachc∈C pub is a replay control evaluated through the same input–output interface. The best-public control uses the hidden reference to select among public outputs. Every call toPassapplies the strict acceptance criterion in Section 3.3. A task is added only when the wrapped source workflow solves it and the three predefined public-output replay controls fail. Task instances are produced by this procedure, without a language model. Executable interface. Every task exposes the same typed executable interface. A candidate program maps one JSON-encoded scenario to one JSON object, which allows a common execution harness to evaluate heterogeneous workflows. During interaction, the candidate is evaluated only on public scenarios and receives semantic feedback on those outputs. Final evaluation applies the same program to hidden scenarios that remain unobserved by the agent. Any editable implementation that satisfies this interface is admissible. Release gates. A generated task is retained only after passing a fixed set of validity checks. The source workflow must execute successfully, produce deterministic outputs across repeated runs, preserve a stable output schema, and use disjoint public and hidden scenarios. The selected scenarios must also produce measurable changes in workflow behavior. The wrapped source workflow must also pass the verifier, and the predefined public-output replay strategies must fail. A released task therefore has a valid behavioral target, and it is not solved by those replay strategies. Evaluation separation. SWR100 is a held-out in-domain evaluation set. All 100 task identities are excluded from the final reconstruction-SFT corpus, which supports evaluation on held-out task instances within the same reconstruction setting. The exclusion is not evidence of generalization to unseen workflows. TB2, TB4, and LHTB are external evaluations and share no training source with the corpus. The final reconstruction-SFT runs contain no auxiliary terminal-task conversations. Strict and diagnostic scores. Final success is determined only by hidden evaluation. A submission must provide a valid executable reconstruction, produce schema-compatible outputs, match all required hidden behaviors, and pass the anti-shortcut checks. For 2,940 tasks, outputs are compared recursively over all mandatory fields. The remaining 60 geometry tasks use a domain-specific comparator. Categorical values, Booleans, and integers require exact agreement, while floating-point values use absolute and relative tolerances of 10 −6 and 10 −4 . Partial semantic agreement is reported for diagnosis and is excluded from the binary success decision. Public feedback and final accounting. During interaction, the agent receives feedback only from public scenarios. This feedback reports whether the candidate executes correctly, produces valid outputs, and matches the observed public behavior. Hidden scenarios and their reference outputs remain private until final grading. Agreement on the public examples can therefore coexist with failure on hidden scenarios. 15 Algorithm 1 Synthesis of reconstruction tasks from existing programs. Each of 500 workflows is instantiated in six variants. A variant is kept only if the source program passes and the public-output replay controls fail. Require: Executable workflows (d,f d ) 500 d=1 , typed scenario generators, and K = 6 variants per workflow Ensure: Reconstruction dataset D SWR 1: D SWR ←∅ 2: for each workflow (d,f d ) do 3: for k = 1 to K do 4:repeat // Stage 1: Scenario construction and workflow execution 5:(S (k) pub ,S (k) hid )← SampleScenarios(d,k) 6:valid← [S (k) pub ∩ S (k) hid =∅] 7:if valid then 8:Execute f d twice on S (k) pub ∪ S (k) hid 9:valid ←Successful (f d )∧ Deterministic(f d )∧ SchemaStable(f d )∧ InterventionSensitive(f d ) 10:end if 11:if valid then 12:O (k) pub ←(s,f d (s)) : s∈ S (k) pub 13:Retain (s,f d (s)) : s∈ S (k) hid as private references 14:end if // Stage 2: Task materialization 15:if valid then 16:I (k) ← RenderInstruction(d,O (k) pub ) 17:J (k) ← BuildSemanticJudge(f d ,S (k) hid ) 18:T d,k ← (d,f d ,O (k) pub ,S (k) pub ,S (k) hid ,J (k) ,I (k) ) 19:Materialize(T d,k , environment, public feedback, private verifier) 20:end if // Stage 3: Oracle and anti-shortcut gates 21:if valid then 22:g ⋆ d ← WrapWorkflow(f d ) 23:C pub ←modal replay, nearest replay, best-public replay 24:valid← [Pass(T d,k ,g ⋆ d ) = 1] 25:for each shortcut control c∈C pub do 26:if Pass(T d,k ,c) = 1 then 27:valid← false 28:end if 29:end for 30:end if 31:until valid 32: D SWR ←D SWR ∪T d,k 33: end for 34: end for 35: assert |D SWR | = 500× 6 = 3000 36: return D SWR Infrastructure failures are recorded separately from model failures, and the primary Pass@1 is computed over all scheduled tasks. 16 Table 6 Protocols for the reported evaluations. Columns give the task count, runs per model, turn limit, wall-clock limit, and which outcomes are retained. StudyTasks Runs/model Turn limit Wall time Retained outcomes Full-suite evaluation3,00011001 hour Success and failure Three-attempt Pass@33,0003– Final task union SWR100 comparison10015003 hours Success and failure SFT scaling4 suites3– Train-and-test seeds Training and decoding configuration. We initialize Qwen3.8-27B from the official Hugging Face snapshot Qwen/Qwen3.8-27B, which we convert once to a matching-layout Megatrontorch_distcheckpoint rather than loading it through the Hugging Face bridge. Full-parameter SLIME SFT runs on eight nodes with eight NVIDIA GPUs per node (64 GPUs total), tensor parallelism 4, pipeline parallelism 2, context parallelism 2, sequence parallelism, and global batch size 128. The maximum context length is 262,145 tokens and dynamic packing is capped at 49,152 tokens per GPU. Thinking is disabled. The sample-levelsft_lossusesloss_- mask_type=qwen3_5, so only assistant tokens contribute to the objective. Adam uses (β 1 ,β 2 ) = (0.9,0.98), cosine decay from 1×10 −5 to 1×10 −6 , 5% warmup, and zero weight decay. Optimizer and RNG states are neither loaded nor saved. After chat-template rendering, rows longer than 128k tokens are dropped rather than split with a sliding window. The SWR100 API runs use temperature 0.2, top-p0.95, at most 8,192 output tokens per call, at most 500 agent turns, and a three-hour task limit. B Dataset Composition and Audit Domain and family coverage. SWR covers 46 software families grouped into six application domains. Software & Formal Systems contains 11 families and 660 tasks, followed by Physical & Engineering Simulation with 9 families and 588 tasks, Spatial, Visual & 3D with 8 families and 528 tasks, Life & Molecular Sciences with 7 families and 468 tasks, Data & Document Workflows with 6 families and 396 tasks, and Earth & Space Sciences with 5 families and 360 tasks. No software family accounts for more than 2.4% of the benchmark, which limits the weight of any one family. Figure 10 shows the complete domain-to-family hierarchy. Observable artifact diversity. Artifact labels are multi-valued because a single task may contain several output types. Tables appear in 2,028 tasks and logs or text in 1,776 tasks. The benchmark also includes structured JSON (432), images (420), audio (276), scientific data (276), source code (156), biological sequences (132), geometry (132), markup (60), packet captures (60), and configuration files (54). Figure 8 places these output types on each of the 46 families and gives the median number of checkable output fields. Figure 9 shows three released public artifacts from each domain. Intervention complexity. Most tasks use five parameters (86.7%), while three- and four-parameter scenarios each account for 6.7% of the benchmark. Across all parameter axes, numeric values are most common (66.1%), followed by strings (26.7%), Booleans (5.2%), arrays (1.9%), and mixed types (0.1%). 17 TableLogJSONImageAudioScienceCodeSequenceGeometryMarkupPCAPConfig AJV schemas Supply chain Network PCAP OPA / Rego Z3 constraints CMake builds Nginx routing OpenSSL PKI Tree-sitter Verilog RTL Web UI state Audio signals Circuit simulation Finite elements Robotics control Room acoustics Power grids RF signals Crystallography OpenFOAM CFD GDAL / geospatial Map rendering Color imaging LiDAR / point cloud Blender scenes OpenSCAD KiCad PCB Video QC SAMtools / QC Medical imaging Molecular chemistry Cell microscopy Phylogenetics Proteomics / MS Structural biology Document / OCR DuckDB reports SQLite analytics GTFS schedules Font shaping Score engraving Astronomy / FITS Climate / NetCDF Well-log analysis Hydrology Seismology Software & Formal Systems Physical & Engineering Simulation Spatial, Visual & 3D Life & Molecular Sciences Data & Document Workflows Earth & Space Sciences 101001,000 Median fields Every task in the familyOnly some tasks Figure 8 Output types of the 46 software families. Families are grouped by domain. A filled mark means every task in the family produces that output type. An open mark means only some tasks do. In CMake builds, source code appears in 36 of 60 tasks and configuration files appear in 54 of 60. The bars give the median number of checkable output fields on a logarithmic scale. 18 object kinditemsmeta JSON schema command command nameargument Parse tree arbiteradapterfifo demuxbroadcastcrosspoint RTL modules 024681012 Time (ms) 10000 5000 0 5000 10000 Amplitude Room acoustics 024681012 Time (ms) 10000 5000 0 5000 10000 Amplitude RF envelope 024681012 Time (ms) 10000 0 10000 Amplitude Audio bandpass Bearing block Pulley 0246810 x 2 0 2 y Map boundary MoleculeNucleotide sequence PEPTIDEWGPYHQAIKR Peptide APACEUNA 0 50 100 150 200 250 Value Query resultScore 0102030 x Glyph positions 26002800300032003400 Velocity 0 10 20 30 Depth (m) Well log 050100150200250 Time (ms) 5000 0 5000 Amplitude Seismic envelope 0.02.55.07.510.012.515.0 Time (ms) 10000 5000 0 5000 10000 Amplitude Seismic window Software & Formal Systems Physical & Engineering Simulation Spatial, Visual & 3D Life & Molecular Sciences Data & Document Workflows Earth & Space Sciences Figure 9 Released public artifacts from the six domains. Each row is one domain and contains three examples taken from the files released with the task. The software panels redraw a released JSON schema, a released parse tree, and the RTL module names in a released file list. The other panels are released waveforms, source-program meshes, a map boundary, a molecule, biological sequences, a query result, a musical score, glyph positions, a well log, and two views of one seismic trace. 19 AJV schemas Supply chain Network PCAP OPA / Rego Z3 constraints CMake builds Nginx routing OpenSSL PKI Tree-sitter Verilog RTL Web UI state Document / OCR DuckDB reports SQLite analytics GTFS schedules Font shaping Score engraving GDAL / geospatial Map rendering Color imaging LiDAR / point cloud Blender scenes OpenSCAD KiCad PCB Video QC Audio signals Circuit simulation Finite elements Robotics control Room acoustics Power grids RF signals Crystallography OpenFOAM CFD Astronomy / FITS Climate / NetCDF Well-log analysis Hydrology Seismology SAMtools / QC Medical imaging Molecular chemistry Cell microscopy Phylogenetics Proteomics / MS Structural biology Software & Formal Systems Data & Document Workflows Spatial, Visual & 3D Physical & Engineering Simulation Earth & Space Sciences Life & Molecular Sciences 3,000 tasks 46 families · 6 domains Figure 10 Domain and software-family composition of SWR. Families are grouped into six domains. No family exceeds 2.4% of the tasks. 10 1 10 2 10 3 10 4 Semantic leaf fields per task (log) 0 25 50 75 100 Cumulative tasks (%) Median = 11 p90 = 95.3 Figure 11 Number of checkable output fields per task in SWR. Digest-like profiling fields are excluded. Semantic output complexity. For each task, the audit counts the output fields that the verifier can check. After digest-like profiling fields are excluded, the median task has 11 such fields and the 90th percentile is 95.3. The longer outputs come from workflows that emit nested tables, geometry descriptors, or scientific arrays. Strict grading inspects every required field, so a reconstruction must match many related values at once. 20 Behavioral grounding and shortcut resistance. The audit also tests whether workflow outputs change with the input scenario and whether replaying a public output solves a hidden case. Every task in the evaluated release changes under at least one scenario comparison, and 2,984 change for every observed scenario pair. On average, 75.1% of output fields vary across scenarios, and 66.3% of fields change within a scenario pair. For the 2,400 tasks where input parameters can be changed one at a time, 89.5% of parameters produce a measurable output change. We further evaluate three public-output replay baselines in Table 7. The strongest baseline, oracle-selected best-public replay, reaches 44.3% hidden-field agreement, while nearest-scenario replay and constant-output replay reach 38.0% and 34.3%, respectively. None of the three attains a strict pass on any task. The result agrees with the release criterion, under which a task is kept only when these strategies fail. Table 7 Public-output replay audit on SWR. Hidden-field agreement is the share of hidden fields matched by a replay control. Strict success requires a complete hidden pass. ControlHidden-field agreement Strict success Oracle100.0%100.0% Oracle-selected best-public replay44.3%0.0% Nearest public scenario38.0%0.0% Constant public output34.3%0.0% Within-workflow variant audit. Each of the 500 workflows produces six task variants that share the same underlying executable workflow. Public examples may be reused across variants to provide sufficient observa- tions. The six variants are therefore not independent software mechanisms, nor independent demonstration sets. Their hidden evaluations remain separate. All 15 pairs of variants within each workflow are compared, which yields 7,500 pairwise comparisons. Hidden scenario sets are disjoint across sibling variants, and no hidden scenario appears in the public or hidden set of another sibling. Each workflow also produces six distinct sets of hidden outputs. Thus, the six variants share the same underlying mechanism but differ in the hidden scenarios and behavioral targets used for evaluation. Dataset limitations. Because the tasks are derived from a finite set of workflows, related tasks can share instruction patterns and program structure. Coverage of software used in science is not exhaustive, and difficulty is uneven across scientific domains and software families. C Repeated-Attempt Pass@3 Analysis Evaluation setup. The full-suite evaluation runs Qwen3.8-Max once on each SWR task, and two further attempts use the same finalized verifier. Under Pass@3 a task is solved when at least one of the three attempts succeeds. Successful attempts are retained as verified trajectories for reconstruction SFT. Pass@3 results. Across three attempts, 838 tasks in the evaluated release are solved at least once, yielding 27.9% Pass@3. Among these tasks, 383 succeed in exactly one attempt, 326 in two attempts, and 129 in all three attempts. Of the 838 solved tasks, 455 succeed more than once. The successful attempts produce 1,422 verified trajectories for reconstruction SFT. Family-level results. Repeated attempts increase the number of solved tasks. After three attempts, 20 of the 46 families still contain no solved task, so additional sampling leaves a substantial part of the release unresolved. D Full-Suite Diagnostics Failure analysis. The finalized evaluation records 685 strict passes on the evaluated release, corresponding to 22.8% Pass@1. Among the 2,781 evaluated workspaces that contain an executable candidate, 1,591 return the 21 same output across hidden scenarios and 75 replay released outputs. Many submitted programs are executable and still fail to reproduce the dependence of the workflow output on the input. Failed runs retain a mean diagnostic agreement of 75.4%, so partial hidden agreement is common when strict success is not obtained. Interaction cost. Successful reconstructions end at a median of 58 turns, against 85 turns for failures attributed to the model. Figure 12a shows that the longer interactions are characteristic of the failed runs, rather than an effect of a few extreme trajectories. Across software families, pass rate is negatively correlated with median interaction length (r=−0.647, Figure 12b). Families with lower pass rates also have longer median interactions. 020406080100120 Agent turns 0 25 50 75 100 Trajectories at or below x (%) Successful: med=58, p90=91 Failed: med=85, p90=96 (a) Trajectory length in the single-attempt evaluation with the finalized verifier. Successful runs end earlier than failures attributed to the model. 0255075100 Qwen3.8-Max pass rate (%) 20 40 60 80 100 Median agent turns CMake builds Verilog RTL Web UI state Medical imaging Blender scenes Software & Formal Systems Physical & Engineering Simulation Spatial, Visual & 3D Life & Molecular Sciences Data & Document Workflows Earth & Space Sciences (b) Family pass rate against median turns in the same single-attempt evaluation. Figure 12 Interaction cost in the single-attempt SWR evaluation. Panel (a) is the distribution of trajectory length. Panel (b) is family pass rate against median turns. E SFT Corpus Construction Corpus construction. Uniform re-verification yields 1,422 successful trajectories from 838 reconstruction tasks. The final reconstruction-SFT corpus contains 3,000 sampled entries, obtained by oversampling verified trajectories and chiefly by repeating trajectories from tasks with only one success. A repeated entry reuses the same interaction and does not add an independent trajectory. Evaluation task identities are excluded. Duplicate events and actions that follow a completed solution are removed inside each conversation, and conversation boundaries are repaired. That cleanup is separate from the intentional repetition of complete training entries. The final corpus contains reconstruction data only. Training data and optimization. Each trajectory is rendered independently for the final from-base run. Rows longer than 128k tokens are dropped, and 49,152 tokens is used only for per-GPU dynamic packing. The training configuration is described in Appendix A. F Reconstruction Fine-Tuning Scaling and Controls Per-seed results. Table 8 reports the per-seed results underlying Table 4. The 3,000-entry checkpoint exceeds the Base model for every seed on all four benchmarks. The intermediate budgets show a progression as the number of sampled entries increases, and the magnitude of the gain differs across benchmarks. 22 Table 8 Per-seed training budgets for Qwen3.8-27B. These rows underlie Table 4. LHTB and SWR100 entries are percentages. ModelSeed TB2 (%) TB4 (%) LHTB SWR100 Base4247.190.00 22.001.00 Base4349.441.52 20.001.00 Base4447.190.00 20.002.00 SFT-750 4250.561.52 23.001.00 SFT-750 4349.441.52 22.002.00 SFT-750 4450.560.00 22.001.00 SFT-1500 4251.691.52 24.002.00 SFT-1500 4350.561.52 23.003.00 SFT-1500 4449.443.03 26.003.00 SFT-3000 4253.933.03 28.003.00 SFT-3000 4352.814.55 27.004.00 SFT-3000 4453.933.03 28.003.00 Matched-token corpus controls. Table 5 compares reconstruction supervision with four alternative agent- training corpora. Each fine-tuned checkpoint receives 88.48M assistant-loss tokens, the tokens whose prediction enters the supervised loss. This budget is shared by every corpus in the comparison. For reconstruction SFT, 3,000 counts sampled training entries, including intentional repetition, and does not count unique source trajectories. Unique-trajectory and exposure counts are omitted, because each source defines those counts differently. Table 9 Per-seed scores for the four matched-token control corpora. Every evaluation entry is a percentage. Training corpusSeed TB2 TB4 LHTB SWR100 OpenThoughts-Agent-v1 4248.31 0.00 21.001.00 OpenThoughts-Agent-v1 4350.56 0.00 23.002.00 OpenThoughts-Agent-v1 4448.31 0.00 24.002.00 TMax4249.44 0.00 21.002.00 TMax4347.19 1.52 19.001.00 TMax4451.69 1.52 22.001.00 LiteCoder-Terminal4250.56 0.00 22.002.00 LiteCoder-Terminal4348.31 0.00 24.000.00 LiteCoder-Terminal4447.19 1.52 23.001.00 Nemotron-Terminal4248.31 0.00 23.001.00 Nemotron-Terminal4347.19 0.00 23.000.00 Nemotron-Terminal4446.07 1.52 25.003.00 Scope of the comparison. Raising the reconstruction training-entry budget from 750 to 1,500 and then to 3,000 yields non-decreasing means on all four evaluations, and the 3,000-entry checkpoint exceeds Base for every seed. Because the budgets include repeated samples, the result describes training under the reported sampling procedure. It does not isolate the effect of adding independent trajectories. At a common budget of 88.48M assistant-loss tokens, reconstruction SFT attains the highest mean among the evaluated corpora. The comparison holds the amount of loss-bearing text fixed and leaves task composition, trajectory diversity, and verification entangled. G SWR100 Model and Domain Results Evaluation accounting. SWR100 contains 19 tasks from Software & Formal Systems, 20 from Physical & Engineering Simulation, 19 from Spatial, Visual & 3D, 15 from Life & Molecular Sciences, 14 from Data & 23 Document Workflows, and 13 from Earth & Space Sciences. Each model is scheduled on the same 100 tasks. Of 800 trials, 797 produce records, 785 receive a verifier score, and 44 pass. Missing and unscored runs remain failures in the denominator. Domain concentration. GPT-5.6 Sol accounts for 20 successes, followed by Kimi-K3 with six and Grok 4.6 with four. The remaining models solve two or three tasks each. Across models, 27 successes occur in Software & Formal Systems, eight in Physical & Engineering Simulation, eight in Spatial, Visual & 3D, and one in Life & Molecular Sciences. No run succeeds in either Data & Document Workflows or Earth & Space Sciences. H SWR100 Interaction and Statistical Diagnostics Interaction diagnostics. Figure 13 reports turns and token consumption. Under the common three-hour-per- task upper bound, median interaction length ranges from 69.5 to 356.5 turns and median token use ranges from 4.49M to 18.59M. These quantities record how each run spends the interaction budget. Task success is reported separately. Statistical scope. SWR100 contains one trial per model–task pair. Under this protocol the 14-point gap between GPT-5.6 Sol and the next-best model is large, whereas differences among models with two to six successes are not resolved. The set is disjoint in task identity from the processed SFT corpus and is an in-domain evaluation. 0100200300400 Agent turns GPT-5.6 Sol Kimi-K3 Grok 4.6 DeepSeek-V4-Pro GLM-5.3 MiniMax-M3 Qwen3.8-Max Doubao Seed 2.1 (a) Agent turns on SWR100 under a three-hour limit per task. 0510152025 Prompt + output tokens (M) GPT-5.6 Sol Kimi-K3 Grok 4.6 DeepSeek-V4-Pro GLM-5.3 MiniMax-M3 Qwen3.8-Max Doubao Seed 2.1 (b) Prompt and output tokens on SWR100 under the same three-hour limit. Figure 13 SWR100 trajectory profiles under a three-hour limit per task. Panel (a) is agent turns. Panel (b) is prompt and output tokens. Points are medians. Horizontal segments are interquartile ranges. 24 Table 10 Per-model results on SWR100. Pass@1 uses all 100 scheduled tasks for each model. Turns and tokens are medians. ModelRecords Scored Pass Turns Tokens (M) GPT-5.6 Sol9989 20 191.010.12 Kimi-K398986 131.56.94 Grok 4.6100994 98.05.03 DeepSeek-V4-Pro100993 187.510.00 Qwen3.8-Max1001003 356.518.59 GLM-5.31001003 69.54.49 MiniMax-M31001003 195.58.75 Doubao Seed 2.11001002 183.010.20 Total797785 44– A few runs have an execution record and no final score, ten for GPT-5.6 Sol and one each for Grok 4.6 and DeepSeek-V4-Pro. They remain in the Pass@1 denominator, so an absent score cannot raise the reported rate. Model comparison uses only strict hidden success, and trajectory statistics are used only to describe the interaction. I Verifier Mutation Stress Tests Protocol. An initial mutation audit found that the verifier rejects conspicuous changes of representation and of semantics. Controls of that kind can overstate reliability. The audit therefore uses a harder suite of five complementary mutation families. For structured outputs, we perturb one finite floating-point leaf at 0.5, 0.9, 0.99, 1.01, 1.1, 2, and 10 times the configured absolute-or-relative tolerance. We also mutate one deepest leaf while preserving its container schema, shift all numeric leaves coherently, and substitute the oracle output of a second hidden scenario. Finally, we replace one hidden input axis with its public-scenario value before re-executing the oracle. The last operation corresponds to a reconstruction that ignores one intervention axis and still emits a coherent, schema-valid output. For geometry, we sweep uniform scale, localized surface deformation, facet deletion, and public- or hidden-scenario mesh substitution. Each trial invokes the released task comparator directly. The 60 bioinformatics tasks are evaluated in their packaged samtools/bcftools/seqkit environment. Table 11 Stress test of the enhanced verifier on the full SWR release. Acceptance counts as correct for a valid control and as incorrect for every other mutation class. Mutation classTrials Accepted Rejected Valid near-boundary controls7,0657,0650 Single deep-leaf change2,94002,940 Correlated numeric shift2,86802,868 Wrong hidden scenario2,93902,939 Ignored input axis9,90109,901 Numeric perturbation> 1× tolerance9,26009,260 Localized geometry deformation2400240 Facet deletion3000300 Wrong-scenario geometry1200120 Uniform scale≥ 1.02×2400240 All invalid or wrong-scenario mutations 28,808028,808 25 0.5×0.9×0.99×1.01×1.1×2×10× Perturbation magnitude / configured tolerance 0 25 50 75 100 Mutation acceptance (%) configured boundary 2,315 applicable outputs per point (a) Acceptance near the configured numeric tolerance. The plot shows how often the verifier accepts an output as the perturbation grows. Deep-leaf Correlated numeric Wrong hidden Ignored input axis Local deformation Facet deletion Wrong geometry 96 97 98 99 100 Invalid-mutation detection (%) 2,940/2,9402,868/2,8682,939/2,9399,901/9,901240/240300/300120/120 Structured outputsGeometry outputs (b) Rejection of coherent semantic and geometry muta- tions. The rate is the share of mutated outputs that the verifier rejects. Figure 14 Stress tests of the enhanced verifier. Panel (a) uses 2,315 outputs with finite floating-point values at each perturbation size. Panel (b) reports detection rates with Wilson 95% lower bounds. Labels give rejected counts and totals. Boundary behavior and calibration outcome. Figure 14 shows that the verifier behaves consistently around its configured decision boundaries. For numeric outputs, all perturbations up to 0.99×the allowed tolerance are accepted, while all perturbations from 1.01×onward are rejected. The verifier also rejects more structured errors, including deep-field changes, correlated numeric shifts, wrong hidden-scenario outputs, and all 9,901 cases that ignore one input axis. The geometry comparator shows a similar distinction between equivalent and incorrect outputs. Valid changes such as ASCII reserialization, facet reordering, and rigid translation remain accepted, while wrong-scenario meshes, local deformations, facet deletions, and scale changes of at least 2% are rejected. Based on these stress tests, we use family-specific relative tolerances for audio and geospatial outputs and strengthen geometry comparison with bidirectional surface coverage and boundary-edge checks. As summarized in Table 11, the calibrated verifier makes no incorrect decisions across this finite stress-test suite. Independent post-freeze audit. We first validate the mutation generator on a separate 100-task pilot set. We confirm that the operators produce the intended valid and invalid cases, then freeze the generator and exclude all pilot tasks from the final audit. The held-out audit uses a disjoint set of 100 tasks, including two tasks from each of the 46 software families and eight prespecified high-risk cases. We generate 1,200 new trials, with 400 equivalence-preserving controls and 800 invalid mutations covering cross-field inconsistencies, unit and boundary errors, scenario splicing, and several geometry transformations. These cases test whether the frozen verifier can distinguish harmless representation changes from semantically incorrect outputs. Table 12 compares verifier decisions with the predefined accept-or-reject labels. Table 12 Verifier audit after freezing and before human adjudication. Intervals are Wilson 95% intervals for the correct-decision rate. Audit stratumTrials False accept False reject Correct95% CI Structured valid controls37600 100.00% [98.99, 100.00] Geometry valid controls2406 75.00% [55.10, 88.00] Structured invalid mutations75220 99.73% [99.04, 99.93] Geometry invalid mutations4810 97.92% [89.10, 99.63] All held-out trials1,20036 99.25% [98.58, 99.60] Audit errors and human validation. Under the predefined mutation labels, the verifier disagrees with 9 of the 1,200 audit cases. It accepts 3 of 800 cases labeled invalid and rejects 6 of 400 cases labeled valid, corresponding to provisional rates of 0.38% and 1.50%, respectively. These disagreements concern boundary 26 cases. Two accepted circuit perturbations differ by only one output unit and remain within the verifier’s declared relative tolerance, while the third is a geometry shear. All six rejected valid cases are alternative surface tessellations that preserve the same underlying geometry but fail the sampled-point coverage check. We report these provisional rates using the original mutation labels, without changing the labels after observing verifier decisions. We then independently validate both the labels and the verifier using 200 blinded cases, with one expected-valid and one expected-invalid example from each sampled task. Two annotators achieve 96.0% raw agreement. After adjudication, 98.0% of the predefined mutation labels agree with the human judgments. Relative to the adjudicated labels, the verifier makes two false accepts and two false rejects, corresponding to a 2.0% false-accept rate and a 2.0% false-reject rate on the balanced audit set. These results show that the remaining verifier errors are rare and are concentrated around numerical tolerance and geometric equivalence. Table 13 Blinded human audit of the held-out verifier set. Label validity and verifier agreement use adjudicated judgments. Annotator agreement is measured before adjudication. Audit stratumTasks BundlesLabel validity Verifier agreement Annotator agreement Family-stratified core92184 98.91% (182/184) 98.37% (181/184)96.74% (178/184) Risk-enriched816 87.50% (14/16)93.75% (15/16)87.50% (14/16) All100200 98.00% (196/200) 98.00% (196/200)96.00% (192/200) Table 14 Verifier decisions after adjudication on 200 blinded bundles. Human-adjudicated semantic validity is the reference label. Human-adjudicated label Verifier accepts Verifier rejects Total Valid / equivalent982100 Invalid / non-equivalent298100 Total100100200 Post-adjudication error rates. After human adjudication, the verifier records two false accepts and two false rejects on the balanced 200-case audit. The corresponding rates are 2.0% false acceptance, 2.0% false rejection, and 98.0% agreement. These rates use adjudicated semantic validity, in place of the predefined mutation labels in Table 12. The frozen verifier and the human judgments therefore agree closely, and the residual errors lie at numerical tolerances and at geometric equivalence. J Failure Signatures and Matched Trajectory Cases Hierarchical multi-label taxonomy. We reanalyze all 800 SWR100 model–task pairs using the final private- verifier report. Primary labels follow the order of the evaluation checks, from artifact absence through hidden execution failure and anti-shortcut failure to a geometry or semantic mismatch. Simultaneous semantic labels are retained rather than collapsed into one category. The resulting primary counts are 44 passes, 460 numeric semantic failures, 130 missing executable candidates, 36 hidden execution or output-parsing failures, 38 schema failures, 55 categorical failures, 20 public-output replays, eight geometry failures, three missing records, and six other residual outcomes. Multi-label accounting further reveals 173 trials with both numeric and categorical mismatches and 19 with both numeric and structural mismatches, distinctions that are not visible in the primary-label summary. Model-specific failure structure. Figure 15 shows that the dominant failure mode differs across models. Numeric mismatch is the primary label for 460 of 756 failed trials (60.8%), while failure to produce an executable candidate accounts for another 130 (17.2%). Grok 4.6, GLM-5.3, and DeepSeek-V4-Pro contribute 62, 41, and 21 missing-artifact failures, respectively, accounting for 124 of the 130 such cases. In contrast, 27 numeric mismatch dominates the failures of GPT-5.6 Sol (73 cases), Kimi-K3 (74), Qwen3.8-Max (71), and Doubao Seed 2.1 (72). The split corresponds to two bottlenecks. Some models often fail before a gradeable reconstruction exists, whereas others usually emit an executable artifact and then fail to recover the hidden behavior. Multi-label analysis further shows that semantic errors often occur together, with 173 trials containing both numeric and categorical mismatches, 22 containing categorical and structural mismatches, and 19 containing numeric and structural mismatches. 020406080100 SWR100 tasks per model GPT-5.6 Sol Kimi-K3 Grok 4.6 DeepSeek-V4-Pro Qwen3.8-Max GLM-5.3 MiniMax-M3 Doubao Seed 2.1 Pass Numeric Missing file Execution Schema Categorical Public replay Other Figure 15 Primary outcomes on SWR100, grouped by the order of evaluation checks. Every model receives the same 100 tasks. Rare geometry errors, missing records, and residual outcomes are grouped as Other. Matched-case design. Aggregate counts do not identify the interaction that produced a verifier outcome. Four same-task contrasts are therefore used to examine numeric mismatch, a missing reconstruction, and hidden execution failure. The selection is qualitative. The cases are used to diagnose mechanisms, and they are not used to estimate how often those mechanisms occur. For each contrast, both models receive the identical task package, three-hour upper bound, and private verifier. Table 15 records the differences associated with the outcomes, while Figures 16–19 show the underlying trajectory and verifier evidence. Table 15 Four matched pairs of a success and a failure on the same task. Turns and tests summarize interaction effort. The last column states the evidence tied to each outcome. Task mechanismPassing trajectoryFailing trajectoryDecisive evidence and interpretation Color-transform numer- ics GPT-5.6 Sol, 213 turns, 41 candidate tests Qwen3.8-Max,376 turns, 25 candidate tests The hidden mean is 0.291619 rather than 0.267706, and every reported XYZ component exceeds tolerance after 45 public-feedback calls. Container SBOM com- pletion GPT-5.6 Sol, 110 turns, 11 edits, 17 tests Grok 4.6, 149 turns, one edit, one test No executable candidate remains, so semantic grading never begins. The final plan still defers implementation until after another search. Robotics hidden tokenGPT-5.6 Sol, 31 turns, 0.97M tokens Qwen3.8-Max,368 turns, 20.60M tokens After six public passes, an unseen categorical value is cast to an integer and causes hidden execution to fail. Public completion conceals a parser that cannot process the new value. Video policy category Kimi-K3, 64 turns, 14 tests DeepSeek-V4-Pro, 204 turns, seven tests A closed vocabulary rejects an unseen audio-layout category. The passing implementation maps unseen categorical values deterministically. 28 Cross-case interpretation. The four matched cases separate three sources of failure. Some unsuccessful runs collect extensive observations and execute few candidate programs. The SBOM run ends before a reconstruction exists. The robotics and video runs pass the public checks and fail on an unseen categorical input. Longer interaction and additional public feedback are therefore not sufficient for a correct reconstruction. In the recorded successes, observations are turned into an executable hypothesis and that hypothesis is tested beyond the public examples. Evidence-to-test conversion. The color, SBOM, and video cases differ in how they use interactions. In the color task, GPT-5.6 Sol performs 41 candidate tests in 213 turns, compared with 25 tests in 376 turns for Qwen3.8-Max. In the SBOM task, GPT-5.6 Sol performs 17 tests in 110 turns, while Grok 4.6 performs only one test in 149 turns. A similar pattern appears in the video task, where Kimi-K3 performs 14 tests in 64 turns compared with seven in 204 turns for DeepSeek-V4-Pro. The comparisons are descriptive and do not identify test frequency as a cause of success. They do show unsuccessful runs in which most of the interaction is spent on inspection and few executable hypotheses are tested. Public completion and hidden robustness. A larger number of candidate tests is not by itself sufficient. In the robotics case, Qwen3.8-Max performs 32 candidate tests and passes six public checks, yet an integer cast fails when the hidden evaluation introduces a categorical token. In the color case, 45 public-feedback calls still leave the coupled XYZ transformation outside tolerance. Public agreement therefore leaves open a failure on hidden inputs. A test that is informative for these tasks also covers unseen but valid input types, unknown categories, and dependence among output fields. Artifact completion. The SBOM case is a failure of artifact completion rather than of semantic agreement. The unsuccessful trajectory issues 16 commands to gather evidence, makes one reconstruction edit and one candidate test, and leaves no executable candidate at grading time. Semantic accuracy is therefore never evaluated. In this run the gradeable object is absent, so later semantic revision cannot begin. Implications for trajectory supervision. The cases make four properties of a trajectory observable, namely early creation of the required artifact, testing after new evidence is obtained, explicit checks for unseen but valid inputs, and retention of a runnable candidate through the interaction. They also support treating artifact completion, robustness to hidden inputs, and semantic fidelity as separate outcomes. 29 Color imaging Correct interpolation versus prolonged numeric approximation Family: color imaging pipeline · identical task and private verifier TASK SPECIFICATION Inputs: color space, compression quality, demosaic method, tone curve, white point. Behavior: image/color metrics under hidden color transform and compression interventions. Deliverables: an executable candidate, a metadata record. Verifier: complete recursive comparison with absolute tolerance 1e-06 and relative tolerance 0.0001. GPT-5.6 Sol trajectory VerdictPASS Failure typestrict pass Productive turns213 Token consumption12.14M TRAJECTORY EVIDENCE Evidence commands 15. Reconstruction edits 31. Candidate tests 41. Feedback calls 42. Observed tracebacks 7. PRIVATE VERIFIER EVIDENCE All hidden scenarios passed the complete semantic comparator. FINAL AGENT ASSESSMENT The red-channel simplicity search uncovered a striking candidate structure: `red 1.042 + 0.0024*tone_curve - 0.0026*white_point + color_space_offset`, with offsets near exact hundredths `[0.22, 0.10, 0.00, 0.01, 0.02, 0.00]`. This is much more plausible than the previous Qwen3.8-Max trajectory VerdictFAIL Failure typenumeric semantic mismatch Productive turns376 Token consumption19.12M TRAJECTORY EVIDENCE Evidence commands 36. Reconstruction edits 24. Candidate tests 25. Feedback calls 45. Observed tracebacks 39. PRIVATE VERIFIER EVIDENCE $.mean: expected 0.267706, got 0.291619 (float outside tolerance); $.xyz[0]: expected 0.634891, got 0.601315 (float outside tolerance); $.xyz[1]: expected 0.396304, got 0.378518 (float outside tolerance) FINAL AGENT ASSESSMENT The R-only tone curve approach gives mean=0.285445 for p1, which is close to expected 0.289632! And the XYZ values are in a more reasonable range but still don't match. Let me reconsider the whole problem. I've been trying many approaches and none work exactly. Let me ANALYSIS 1.The failing run spends 163 more turns (376 versus 213) and 19.12M tokens, but executes only 25 candidate tests versus 41 in the passing run. 2.The hidden comparator exposes a joint numeric miss: mean 0.291619 versus 0.267706, with all reported XYZ components also outside tolerance. Matching one public statistic did not recover the transform. 3.Public-feedback volume is nearly identical (45 versus 42 calls), so repeated public checks alone do not constrain unseen color and compression interventions. 4.The successful run converts 15 evidence reads into 31 edits and 41 tests. the failing run reads more evidence but performs only 24 edits and 25 tests. Figure 16 Same-task numeric generalization. GPT-5.6 Sol passes after 213 turns and 41 candidate tests. Qwen3.8-Max uses 376 turns and 45 feedback calls but only 25 candidate tests. Its hidden mean is 0.291619 rather than 0.267706, and every reported XYZ component is outside tolerance. The failure is joint transform recovery, not a lack of public checks. 30 Container analysis Completed behavioral model versus missing reconstruction Family: container supply chain · identical task and private verifier TASK SPECIFICATION Inputs: base image, entrypoint, labels, package set, sbom scope. Behavior: SBOM/license/label edge cases not visible in public layer. Deliverables: an executable candidate, a metadata record. Verifier: complete recursive comparison with absolute tolerance 1e-06 and relative tolerance 0.0001. GPT-5.6 Sol trajectory VerdictPASS Failure typestrict pass Productive turns110 Token consumption5.40M TRAJECTORY EVIDENCE Evidence commands 9. Reconstruction edits 11. Candidate tests 17. Feedback calls 21. Observed tracebacks 14. PRIVATE VERIFIER EVIDENCE All hidden scenarios passed the complete semantic comparator. FINAL AGENT ASSESSMENT The hash formula could not be recovered from public digests despite testing direct canonical hashes, namespaced hashes, multiple Merkle variants, OCI ChainID variants, scenario-ID derivations, artifact hashes, and common Docker layer serializations. The current Grok 4.6 trajectory VerdictFAIL Failure type missing reconstruct py Productive turns149 Token consumption7.54M TRAJECTORY EVIDENCE Evidence commands 16. Reconstruction edits 1. Candidate tests 1. Feedback calls 9. Observed tracebacks 1. PRIVATE VERIFIER EVIDENCE Failed private check: has_reconstruct_py. FINAL AGENT ASSESSMENT Need to search the filesystem for oracle-like code (merkle, vuln_budget, layer_count) and try git-sha1 plus DSSE/in-toto encodings. Then implement the executable candidate. ANALYSIS 1.After 149 turns and 7.54M tokens, the failing run leaves no the executable candidate, so grading stops before any hidden semantic comparison. 2.It issues 16 evidence commands but only 1 reconstruction edit and 1 candidate test. its final plan still places implementation after another search. 3.The passing run does not fully recover the public hash formula, but commits to a type-aware executable fallback through 11 edits and 17 tests. 4.This contrast isolates an artifact-completion failure: exploration is useful only if the agent reserves enough budget to materialize and validate the required deliverable. Figure 17 Same-task artifact completion. GPT-5.6 Sol turns 110 turns into 11 edits and 17 candidate tests and leaves a passing executable fallback. Grok 4.6 spends 149 turns and 7.54M tokens on evidence search but makes one edit and one test. Its final plan still defers implementation. The workspace contains no executable candidate. 31 Robot description repair Valid physical repair versus non-executable output Family: robotics control · identical task and private verifier TASK SPECIFICATION Inputs: actuator gain, collision margin, initial pose, joint limit, trajectory target. Behavior: joint/controller/collision boundary cases under hidden poses. Deliverables: an executable candidate, a metadata record. Verifier: complete recursive comparison with absolute tolerance 1e-06 and relative tolerance 0.0001. GPT-5.6 Sol trajectory VerdictPASS Failure typestrict pass Productive turns31 Token consumption0.97M TRAJECTORY EVIDENCE Evidence commands 5. Reconstruction edits 7. Candidate tests 8. Feedback calls 7. Observed tracebacks 0. PRIVATE VERIFIER EVIDENCE All hidden scenarios passed the complete semantic comparator. FINAL AGENT ASSESSMENT The corrected implementation is ready. It fixes the primary hidden-generalization issue by using the inferred modulus-97 trajectory transform, provides formula-based behavior for unseen parameter combinations, preserves exact native outputs for known Qwen3.8-Max trajectory VerdictFAIL Failure typeexecution or output parse failure Productive turns368 Token consumption20.60M TRAJECTORY EVIDENCE Evidence commands 21. Reconstruction edits 24. Candidate tests 32. Feedback calls 36. Observed tracebacks 37. PRIVATE VERIFIER EVIDENCE Hidden execution: Hidden execution fails because an unseen categorical value is converted to an integer. FINAL AGENT ASSESSMENT Everything is working correctly: - the executable candidate is executable and produces correct output for all public scenarios - the metadata record exists with proper metadata - MuJoCo is imported and used for model building - Output format matches the expected oracle ANALYSIS 1.The passing run finishes in 31 turns and 0.97M tokens. the failing run uses 368 turns and 20.60M tokens despite passing all six public scenarios. 2.An unseen categorical value is parsed as an integer, causing execution to fail before a semantic output exists. The failure reflects a closed-format assumption rather than a small physical-model error. 3.The failing agent nevertheless declares the task complete because public feedback passes. This directly shows why public completion cannot substitute for hidden-input robustness. 4.The passing implementation handles unseen tokens with a deterministic transform and needs 8 candidate tests. the failing run accumulates 37 tracebacks without removing the brittle parser. Figure 18 Same-task hidden-input robustness. GPT-5.6 Sol passes in 31 turns with a representation that accepts unseen categorical values. Qwen3.8-Max uses 20.60M tokens and declares completion after all six public scenarios pass, but an invalid integer conversion then prevents execution on a hidden input. The contrast separates hidden-input robustness from completion on public feedback. 32 Video transcoding Parameter-dependent policy versus hidden-scenario exception Family: video broadcast qc · identical task and private verifier TASK SPECIFICATION Inputs: audio layout, frame rate, gop size, pixel format, subtitle policy. Behavior: media stream metadata and quality summaries under hidden transcode interventions. Deliverables: an executable candidate, a metadata record. Verifier: complete recursive comparison with absolute tolerance 1e-06 and relative tolerance 0.0001. Kimi-K3 trajectory VerdictPASS Failure typestrict pass Productive turns64 Token consumption3.11M TRAJECTORY EVIDENCE Evidence commands 4. Reconstruction edits 11. Candidate tests 14. Feedback calls 11. Observed tracebacks 2. PRIVATE VERIFIER EVIDENCE All hidden scenarios passed the complete semantic comparator. FINAL AGENT ASSESSMENT The reconstruction now uses a unified deterministic derivation (sha1(str(value))[:16] mod len(choices)) that fits all 6 public scenarios for both frame_rate and audio_channels, generalizing to unseen tokens. pixel_format uses the uniquely-consistent DeepSeek-V4-Pro trajectory VerdictFAIL Failure typeexecution or output parse failure Productive turns204 Token consumption10.00M TRAJECTORY EVIDENCE Evidence commands 28. Reconstruction edits 10. Candidate tests 7. Feedback calls 23. Observed tracebacks 3. PRIVATE VERIFIER EVIDENCE Hidden execution: Hidden execution fails because an unseen audio-layout category is rejected. FINAL AGENT ASSESSMENT The public-passing solution failed hidden verification, so I need to discover the actual oracle transformation rather than overfit public outputs. The task explicitly says the reconstruction should use ffmpeg/ffprobe/VMAF, so the true hidden oracle likely invokes real media ANALYSIS 1.The failing implementation treats the audio-layout field as a closed vocabulary. An unseen category causes execution to fail, and the verifier also finds that outputs do not vary across hidden scenarios. 2.The passing run maps unseen categorical tokens deterministically and finishes in 64 turns with 14 tests, versus 204 turns and 7 tests for the failing run. 3.The failure reads substantially more evidence (28 versus 4 commands) but performs fewer candidate executions. exploration does not become robustness testing. 4.After hidden feedback reveals the exception, the final plan returns to external search instead of adding an unknown-category fallback. The decisive difference is repair policy after counterevidence. Figure 19 Same-task policy generalization. Kimi-K3 maps unseen categorical values deterministically and passes after 64 turns and 14 candidate tests. DeepSeek-V4-Pro reads seven times as much public evidence but runs only seven candidate tests. Its closed vocabulary rejects an unseen audio-layout category. The trajectory then returns to external search instead of adding a fallback for unknown categories. 33

Instant research discovery