Paper deep dive
WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses
Haomin Qi, Xiangzhe Xu, Yiming Huang, Jingbo Shang, Chengpeng Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings actionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present WitnessGym, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifications and execution adapters allow extension to additional bug types and languages. Bug-preserving transformations vary the surrounding structure while preserving the witness behavior. Based on real-world Java projects with test suites, WitnessGym automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available.
Tags
Links
- Source: https://arxiv.org/abs/2609.36635v1
- Canonical: https://arxiv.org/abs/2609.36635v1
Trouble viewing inline? Open PDF directly β
Full Text
105,024 characters extracted from source content.
Expand or collapse full text
WITNESSGYM: BENCHMARKING CODING AGENTS ON THE CONSTRUCTION OF BUG WITNESSES Haomin QI 1 Xiangzhe Xu 2 Yiming Huang 1 Jingbo Shang 1 Chengpeng Wang 3 1 University of California, San Diego 2 Purdue University 3 National University of Singapore ABSTRACT Bug validation asks a coding agent to produce an executable witness for a re- ported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings ac- tionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present WITNESSGYM, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifi- cations and execution adapters allow extension to additional bug types and lan- guages. Bug-preserving transformations vary the surrounding structure while pre- serving the witness behavior. Based on real-world Java projects with test suites, WITNESSGYM automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available. https://github.com/HarminChee/WitnessGym https://huggingface.co/datasets/HarminChee/WitnessGym 1INTRODUCTION Agenticcodingsystemsincreasinglyaddressrepository-levelsoftware-engineering tasks (Wang et al.,2025;Yang et al.,2024;Xia et al.,2025;Antoniades et al.,2025), making reliable code auditing an important capability (Guo et al.,2025;Wang et al.,2024). In real-world AI coding scenarios, useful audit findings require concrete evidence that developers can inspect and act on, in addition to a suspicious code location (Wang et al.,2026;Zhang et al.,2025a;Zhu et al., 2025;Cheng et al.,2025;Liu et al.,2025;Simsek et al.,2026;Peng et al.,2025). This requirement matters because bug reports generated by existing AI coding agents may still contain non-negligible false positives (Zhao et al.,2026;Du et al.,2025), and repeated false alarms can quickly erode developersβ trust in the tool. For this reason, a useful coding agent should generate an executable witness that exposes the reported bug (Wang et al.,2026;Gezgin et al.,2026). An executable witness combines a concrete input with a testing harness that invokes the relevant code under the required environment and exposes an observable failure such as a crash or assertion violation. Such evidence helps developers distinguish real bugs from false alarms and decide whether a report deserves attention. Existing benchmarks provide only partial support for evaluating bug validation on new cases in real project contexts. They typically rely on curated capture-the-flag (CTF) challenges or disclosed vulnerabilities, tying evaluation to manually assembled tasks or known bugs and witnesses (Fig- ure 1a and b). For example, NYU CTF Bench (Shao et al.,2024) and Cybench (Zhang et al.,2025b) offer executable objectives in challenge-specific environments whose construction requires cura- tion. Benchmarks based on CVEs (Zhu et al.,2025), bug-bounty reports (Zhang et al.,2025a), and fuzzing artifacts ( Wang et al.,2026) retain real project context but draw their cases from historical 1 arXiv:2609.36635v1 [cs.SE] 29 Sep 2026 (a) Capture-the-flag (CTF) Instances(a) Capture-the-flag (CTF) Instances 12 void challenge(char *in) 13 char buf[16]; 14 strcpy(buf, in); 15 Limitations: Payload β Overflow β Flag/Crash (b) Historical Vulnerability Reproduction CVE Collection Agent task: Known Bugs and PoCs Public Repository Existing PoCs Limitations: Data Contamination Risk Restrictive Bug Types (c) WitnessGym (c) WitnessGym Manually Laborious Toy Programs Restrictive Bug Types Real Repository Bug Types Test Case Inputs Execution Context TraceEntry f1()f2()f3() ... Injected Bugs Hit the Bug! Witness Validated Our Strengths: Automation Realistic Repository Fresh Cases Accepted New Bug Case Diverse Bug Type f4() (b) Historical Vulnerability Reproduction Figure 1: Comparison of benchmark-construction methodologies for executable bug validation. vulnerabilities. Reusing these public artifacts limits the supply of new evaluation targets and creates opportunities for prior exposure to their solutions. We target a benchmark that can generate new cases automatically as models evolve while pre- serving the contextual reasoning required in real code auditing ( Wu et al.,2025;Pu et al.,2025; Saxon et al.,2024). These goals address both the cost of case construction and the dependence on existing solutions. Even when a static-analysis query reports a recognizable bug pattern and its lo- cation, validation requires concrete object states, call sequences, and path conditions that expose the bug through an observable failure. We introduce WITNESSGYM, a framework that meets these goals through automated bug injection. To construct new cases, it adapts API Contract, Value Flow, and Logic bug types from the Cod- eQL Java query library into injection specifications ( GitHub,2026). These categories distinguish contract, data-state, and functional reasoning in witness construction (Section3.2). To preserve real project context, it injects these bugs into production paths exercised by existing tests, rebuilds the project, and reruns the associated test. This test serves as a construction-time witness and must con- tinue to expose the bug after transformations vary the surrounding call, data, and control structure. Its contents are removed before agent evaluation, leaving the agent to construct its own witness. Construction establishes that the bug is observable; evaluation measures whether an agent can inde- pendently recover the conditions needed to expose it. We implement WITNESSGYM and automatically construct 1,300 benchmark cases using ten bug types in these three categories and six real-world Java projects with test suites, at a total cost of $16,500. We evaluate four agent frameworks instantiated in six framework/model pairings on their ability to construct executable witnesses for the benchmark cases. Our evaluation shows substan- tial remaining headroom on this task. The best-performing configuration, Codex with GPT-5.4, achieves 68.9% average validation success. Performance varies across agents and models, with a 50-percentage-point gap between the highest and lowest averages. Value Flow and Logic bugs are generally harder than API Contract bugs, and success decreases as execution contexts lengthen or transformation depth increases. The evaluation costs approximately $8,700. This work makes three main contributions. First, we present WITNESSGYM, a systematic method- ology for automatically constructing bug-validation benchmarks by injecting diverse bug types into real-world projects, producing new bugβwitness pairs. Second, we instantiate this method- ology using ten bug types across API Contract, Value Flow, and Logic and six real-world Java projects, yielding 1,300 execution-validated cases. Third, we evaluate coding agents on the re- sulting benchmark and analyze the factors that affect bug-validation performance. 2 2BACKGROUND AND MOTIVATION Bug Validation. Given a repository, a bug description, and a reported location, bug validation asks an agent to construct an executable witness. The witness combines a concrete input with a testing harness that invokes the relevant code under the required environment and exposes the bug through an exception, assertion violation, crash, or another observable failure. This task turns a reported bug into evidence that developers can reproduce and act on. A recogniz- able pattern or a localized report leaves the concrete input, program state, and failure oracle to be constructed. The agent must recover the relevant call sequence and path conditions, instantiate the required state, and expose the faulty behavior. For example, an API-contract violation may require a particular object state and call sequence before an assertion can expose the incorrect result. Evalua- tion therefore needs executable cases embedded in real program contexts, with a known failure that can serve as the validation oracle. Existing Benchmarks. Table 1 compares the construction capabilities needed for this evaluation. Curated CTF tasks provide explicit objectives, but adding cases requires new challenges and as- sociated environments ( Shao et al.,2024;Zhang et al.,2025b). Historical bug-reproduction work, including LIBRO, Issue2Test, and CyberGym, retains real project context while relying on existing bug reports or disclosed vulnerabilities ( Kang et al.,2023;Nashid et al.,2026;Wang et al.,2026). Its case supply is consequently tied to the availability of historical artifacts. Bug-injection meth- ods automate construction: LAVA and EvilCoder introduce new vulnerabilities through specialized injection mechanisms, while FixReverter applies patterns learned from past fixes at new injection sites (Dolan-Gavitt et al.,2016;Pewny & Holz,2016;Zhang et al.,2022). Their construction ob- jectives give limited control over the surrounding program structure for comparing agents across validation difficulties. Table 1: Comparison of bug-validation benchmark construction. Diversity is high for multiple bug families, medium for variants within a dominant family, and low for one narrow family. Auto. cases are constructed automatically; fresh bugs are generated rather than reused; real projects retain production code with its build and test environments; and context control varies surrounding call, data, or control structure. WorkDiversity Auto. cases Fresh bugs Real projects Context control CTF and security challenges NYU CTF Bench (2024)Highβ Cybench (2025b)Highβ Historical bug validation LIBRO ( 2023)Highβββ Issue2Test (2026)Highβββ CyberGym (2026)Mediumβββ Bug-injection methods LAVA (2016)Lowββ EvilCoder (2016)Lowββ FixReverter (2022)Mediumββ WITNESSGYM (Our Work)Highβ Construction Goals. We seek automatic construction of new bug cases in real projects, covering multiple bug families and retaining an executable oracle for every accepted case. Fresh cases reduce dependence on public bugβwitness pairs, while controlled structural variation lets the benchmark probe how context affects validation. WITNESSGYM combines these properties by injecting bugs into test-reached code and applying transformations that preserve the observed failure. This yields new validation targets with realistic project dependencies and adjustable surrounding structure, with- out per-case manual bug design or witness authoring. 3 3METHODOLOGY This section introduces WITNESSGYM, a framework for constructing bug-validation benchmarks. Given a real-world project with an existing test suite and a library of bug types, WITNESSGYM injects bugs into test-reached production contexts. The associated test provides the harness and concrete input used to expose each injected bug; we call this artifact the construction-time witness. Its contents are removed before agent evaluation. Figure 2 presents the WITNESSGYM workflow. Functions exercised by existing tests serve as injec- tion contexts, providing concrete executions in which new bugs can be reached and observed. Each injection and transformation is validated by rebuilding the project and rerunning its construction- time witness. The pipeline comprises execution-context collection, buggy-code injection, and bug- preserving code transformation. Real Repository Transformation Depth 1 Transformation Operators Library Buggy Code Injection Transformation Depth 2 Transformation Depth 3 Bug-Preserving Code Transformation Execution Context Instrumentation Bug Type Library ... ... Call Stack ... Validated Buggy Case Validate Witness Test Case 12 traceEntry() function1(input); 13 function1(input) return function2(ctx); 14 15 function2(ctx) 16 if (cond(ctx)) 17 return function3(ctx.value); 18 Execution Context Collection A Bug Type Example Value Flow API Contract Data Flow Control Flow 21 function2(ctx) 22 if (cond(ctx)) return fallback(); 23 return function3(ctx.value); 24 Trigger Crash Transformation Depth N Program Crashed Witness Construction and Validation Trigger Crash Trigger Crash Logic 6 if (cond(ctx)) return fallback(); Replay Cmd New Repo Execution Context Figure 2: Overview of the WITNESSGYM construction and evaluation pipeline. 3.1EXECUTION CONTEXT COLLECTION Given a real-world project and its existing test suite, WITNESSGYM first identifies the functions exercised by each test case. This step provides a test-grounded search space for bug injection. To collect the executed functions during runtime, WITNESSGYM applies program instrumenta- tion to the project. Specifically, it parses the source code of the project and inserts logging calls at the entry of each function. During test execution, these logging calls record the exe- cuted functions and their source locations. For a function f , we denote its source location as β(f) = (name(f), path(f), line(f)), where name(f) is the function name, path(f) is the path of the source file containing f , and line(f) is the starting line number of f . Formally, executing a test case t yields an ordered execution trace Ο t = [β 1 ,β 2 ,...,β n ], where each β i is an observed func- tion location. We define the corresponding execution context as the set of unique locations in that trace, EC t = β i | β i β Ο t . Thus, Ο t preserves order, whereas EC t supports context membership and distinct-function counts. Repeating this process for all test cases produces an execution context map,M EC , which maps each test case to its execution contextM EC (t) = EC t . The execution context map serves as the basis for selecting candidate injection locations. WIT- NESSGYM restricts the injection search space to functions executed by a specific test case. This design improves the likelihood that the injected bug is reachable and reduces the cost of subsequent execution-guided injection. 3.2BUGGY CODE INJECTION In the second stage, WITNESSGYM utilizes the collected execution contexts to synthesize bench- mark cases via buggy code injection. We systematically cataloged test-observable bug patterns from CodeQL Java query examples ( GitHub,2026) and grouped them by the program property they vi- olate: API Contract for API contracts or specifications, Value Flow for constraints on propagated values (e.g., dereferencing null values), and Logic for domain-specific functional behavior. This grouping separates the contract, data-state, and functional reasoning needed to construct witnesses. To bound construction and agent-evaluation cost, we randomly selected ten eligible types while retaining examples from each category. The construction loop takes a bug-type specification, a test- 4 reached execution context, and a replayable test supplying the input and oracle. Additional types use the same interface with a specification whose failure is observable; new ecosystems also require suitable tracing and replay adapters. In our setting, accepted bugs are required to produce observ- able runtime failures when exercised, which provides an automatic execution oracle for validating witnesses. The bug types used in our benchmark construction are listed in Table 5 in AppendixC. Based on the above ingredients, the buggy code injection can be formulated as follows. Given a test case t, an execution context EC t , and a bug type b, WITNESSGYM leverages a coding agent to inject b into the execution context EC t of test case t. Since the functions in EC t are exercised, the injected bug is likely to be reachable by the construction-time witness derived from test case t. After each injection attempt, WITNESSGYM rebuilds the modified project and reruns the associated test case t. If the test does not expose the bug, WITNESSGYM provides the execution feedback to the coding agent and requests a refinement. This process continues until the bug is observed or the attempt budget is exhausted. This procedure retains cases whose injected bug is present in production code and empirically exposed by the construction-time witness. The system prompt and user prompt used in buggy code injection are provided in Appendix I.1 and AppendixI.2. 3.3BUG-PRESERVING CODE TRANSFORMATION Since the bug types are derived from public CodeQL examples, directly reusing these snippets may produce benchmark cases that are syntactically or structurally similar to publicly available bug types. WITNESSGYM applies bug-preserving transformations that are similar in spirit to code obfuscation. Specifically, they alter the surface representation and contextual structure of the injected bug while preserving its behavior under the same construction-time witness. The transformations reduce sur- face similarity to public bug examples and increase contextual complexity while preserving the injected bug. Technically, WITNESSGYM is built on three families of transformation operators, which manipulate call stack, data flow, and control flow structures. The detailed definitions of the transformation operators are listed in Table 6 in AppendixC. Similar to buggy code injection, the transformation stage is also execution-guided so that we can validate whether the transformed code preserves the injected bug. If not, the transformation is reverted. Through this process, WITNESSGYM can derive multiple benchmark variants from the same injected bug. These variants share an underlying bug type and construction-time witness while differing in call structure, data flow, and control flow. They provide repeated validation tasks under systematically varied structural contexts. The prompt of the bug-preserving code transformation is given in AppendixI.3. 4EVALUATION We instantiate WITNESSGYM on six real-world Maven-based Java projects using ten observable bug types adapted from the CodeQL Java query library ( GitHub,2026). The types span API Contract, Value Flow, and Logic bugs. The benchmark contains 1,300 buggy cases across four execution- context-length categories and four transformation-depth levels, and costs approximately $16,500 to construct. Multiple cases per type provide observations across repositories and construction settings. AppendixC and AppendixD give implementation details and benchmark statistics. 4.1EVALUATION SETUP Bug-Validation Task. We evaluate every case under two settings that correspond to two important software-engineering scenarios. In the context-agnostic setting, denoted NoEC, the agent receives the buggy repository, the natural language description of the bug type, the buggy location, but no ex- ecution context. This setting mimics the validation of static bug-detection results, where developers may have a localized bug report but no calling context showing how the buggy location is reached. In the context-guided setting, denoted WithEC, the agent receives the same inputs together with the execution context. This setting mimics the bug-reproduction and debugging workflows, where a crash stack or recorded execution trace is available as a dynamic hint. The evaluation begins from a supplied bug report and location and measures the subsequent construction of an executable wit- ness. In the current benchmark instantiation, execution-context length ranges from 1 to 150 distinct production-side functions, grouped into four categories: short ([1, 40]), mid-short ([41, 70]), mid- 5 long ([71, 120]), and long ([121, 150]). AppendixI.4 and AppendixI.5 provide the prompts used for executable bug validation. Agent Settings. We evaluate six framework/model pairings: Codex and Claude Code with their hosted models, and OpenHands and OpenCode with four open-weight models. Table 2 lists the pairings and provider release identifiers (OpenAI,2026;Anthropic,2025;DeepSeek-AI,2025;Z.AI, 2025;Mistral AI,2025;Qwen Team,2025). Table 2: Evaluated framework/model pairings. Agent Framework Model A1CodexGPT-5.4 A2Claude Code Claude Sonnet 4.5 A3OpenHands DeepSeek-V3.2 A4OpenCodeZAI GLM-4.7 A5OpenHands Devstral-2-123B A6OpenCodeQwen3-Coder-480B-A35B-Instruct Each case allows at most three valida- tion attempts.Each attempt uses a 1,800- second agent timeout, a 1,200-second replay- verification timeout, a 600-second optional formatting timeout, and a 4,200-second case- level timeout. For OpenHands, we set the tem- perature to 0 and the maximum output length to 4,096 tokens. Codex, Claude Code, and OpenCode use interface-default decoding be- cause their evaluated interfaces do not expose directly comparable controls. Runs that exceed the case timeout, fail replay verification, or do not produce a repository test containing an executable witness are counted as failures. The evaluation costs approximately $8,700. 4.2RQ1: HOW WELL DO CODING AGENTS VALIDATE BUGS THROUGH EXECUTION? Table 3: Agent validation success rates. IDNoECWithECAverage A167.1%70.7%68.9% A2 57.1%61.8%59.4% A3 46.0%49.8%47.9% A444.2%47.6%45.9% A5 30.2%33.8%32.0% A617.1%19.3%18.2% To control evaluation cost, we randomly sample 900 cases from the 1,300-case benchmark and evalu- ate all six agents under both NoEC and WithEC. Table3 shows substantial performance differences. Codex (A1) and Claude Code (A2) achieve aver- age validation success rates of 68.9% and 59.4%, while OpenHands with DeepSeek-V3.2 (A3) exceeds OpenCode with Qwen3-Coder-480B-A35B-Instruct (A6) by 29.7 percentage points. The best-performing agent still fails on 31.1% of the cases. Execution context improves validation success by 2.2β4.7 percentage points across the six agents. These gains suggest that dynamic context helps agents identify relevant functions and paths for con- structing executable witnesses. However, the improvements do not change the agent ranking, and large performance gaps remain under WithEC. Even with execution context, the best-performing agent succeeds on 70.7% of cases, leaving 29.3% unsolved within the evaluation budget. The sup- plied context can guide code inspection, but it does not provide a complete witness. Finding 1: Current coding agents leave substantial room for improvement in executable bug validation, including when execution context is available. 4.3RQ2: HOW DO BUG TYPES AFFECT EXECUTABLE BUG VALIDATION? To investigate the effect of bug types, we analyze a 500-case subset drawn from the full bench- mark. For each bug type, the subset contains 25 short-context cases and 25 long-context cases, all constructed with the same depth-3 transformation configuration and the same transformation oper- ator sequence, and each case is evaluated under both the NoEC and WithEC settings. As shown in Table 4, difficulty varies substantially across bug types and agents. In particular, the Codex agent framework with GPT-5.4 (A1) and the Claude Code agent framework with Claude Sonnet 4.5 (A2) are relatively weak on Logic bugs, with WithEC success rates of 70.0% and 60.0%, respectively. The open-source agents exhibit a different profile. Their performance varies sharply across bug families, and they are generally less effective on Value Flow and Logic bugs than on API Contract bugs. The gap between industrial and open-source agents varies by bug type. In particular, OpenHands with DeepSeek-V3.2 (A3) generally outperforms the other evaluated open-source agents with open- 6 Table 4: Successful bug validations across bug types. L and S denote long and short execution contexts. Each A/B cell reports successful cases under WithEC and NoEC, respectively. Bug Type A1A2A3A4A5A6 LSLSLSLSLSLS Comparison of Identical Values18/1722/2117/1419/1614/1015/1511/1115/139/710/94/46/5 Typo in equals18/1521/2114/1316/1812/1315/1310/1012/118/79/105/36/5 Typo in hashCode16/1318/1716/1119/1410/915/1211/813/147/610/74/45/5 Hashed Value Without hashCode 14/1419/1715/1316/1510/1115/139/914/108/610/74/35/4 Inconsistent equals and hashCode17/1518/2013/1218/1712/912/1312/913/116/610/93/45/4 Intra-Function Null Dereference19/1623/2314/1319/159/912/1111/913/129/810/117/68/6 Collection Element Null Dereference 16/1620/1714/1216/1710/812/118/711/108/611/85/48/8 Guard Predicate Inversion16/1418/1613/1317/1413/1017/1312/914/137/78/84/35/5 Skipped Aggregation Counter16/1823/2015/1221/1712/1215/1512/1217/168/510/84/45/5 In-Bounds Collection Misrouting 15/1217/1811/1113/1512/1214/1512/1114/136/68/74/35/5 weight models. It narrows the gap to Claude Code with Claude Sonnet 4.5 (A2) on several Logic bug types, including Guard Predicate Inversion and In-Bounds Collection Misrouting. Under WithEC, A3 matches A2 on the former and slightly exceeds it on the latter. However, this pattern does not extend to all bug types, as A3 still trails A2 on both Value Flow types. These results show that open- source agent frameworks paired with strong open-weight models can approach industrial agents on specific bug types. We further quantify the effect of execution context across bug types. Figure3 shows the difference in success rates between WithEC and NoEC for each agentβtype pair. We hold the depth-3 transformation configuration fixed across bug types. The remaining differences therefore mainly re- flect bug semantics and how well each agent uses the sup- plied context. The gains from execution context vary across agents and bug types. The remaining difficulty lies in rea- soning about program properties within this context. Agents must use the supplied information to choose concrete in- puts, set up the required program state, and write checks that expose the faulty behavior. Knowing which functions are involved does not by itself determine how to trigger the bug. Different bug types require different forms of reason- ing. Agents and models therefore differ in how well they can turn the same contextual information into a working test. Benchmarks need multiple bug categories and types that test program properties at different levels of detail and complex- ity. A1A2A3A4A5A6 +4%+12%+8%+4%+6%+2% +6%-2%+2%+2%0%+6% +8%+20%+8%+4%+8%0% +4%+6%+2%+8%+10%+4% 0%+4%+4%+10%+2%0% +6%+10%+2%+6%0%+6% +6%+2%+6%+4%+10%+2% +8%+6%+14%+8%0%+2% +2%+14%0%+2%+10%0% +4%-4%-2%+4%+2%+2% 8 4 0 4 8 12 Gap (p) Figure 3: Success-rate gain with ex- ecution context (percentage points). Types follow Table4. Finding 2: Validation performance varies by bug type. Logic bugs are generally challenging, while open-source agent frameworks with open-weight models approach industrial agents on several specific bug types. 4.4RQ3: HOW DOES BUG-VALIDATION PERFORMANCE VARY WITH EXECUTION-CONTEXT LENGTH? To investigate the effect of execution context length, we fix the bug type to Guard Predicate In- version and analyze a 400-case subset drawn from the full benchmark. The subset is stratified by four execution-context-length buckets and four transformation depth settings, with 25 cases in each length-by-depth cell. For each depth setting, all cases use the same transformation-operator sequence, and each case is evaluated under both the NoEC and WithEC settings. 7 020406080 WithEC success (%) Short Mid-short Mid-long Long (a) Execution-context length effect A1 A2 A3 A4 A5 A6 D1D3D5D10 Transformation depth 0 20 40 60 80 Success rate (%) (b) Depth effect under WithEC Individual agents Agent mean D1D3D5D10 Transformation depth 0 20 40 60 80 (c) Depth effect under NoEC Individual agents Agent mean Figure 4: The effect of execution context length (a) and transformation depth (b and c). Figure4a summarizes the WithEC results, and Table7 in AppendixE reports both settings. Valida- tion success is lower for longer execution contexts for every evaluated agent in both settings. Under WithEC, Codex with GPT-5.4 (A1) decreases from 80% on short contexts to 52% on long contexts, Claude Code with Claude Sonnet 4.5 (A2) decreases by 36 percentage points, and OpenHands with Devstral-2-123B (A5) decreases from 44% to 20%. Execution context can narrow the relevant pro- gram region, while longer contexts still require agents to reconstruct more intermediate states, setup choices, and dependency chains before exposing the bug through a valid test. Finding 3: Bug-validation success is substantially lower for longer execution contexts, includ- ing when execution context is provided. 4.5RQ4: HOW DOES BUG-VALIDATION PERFORMANCE VARY WITH TRANSFORMATION DEPTH? To investigate the effect of transformation depth, we fix the bug type to Guard Predicate Inversion and regroup the same 400-case subset by four transformation depths (1, 3, 5, and 10 applied trans- formations). Each depth-by-length cell contains 25 validated cases, and the transformation operator sequence is fixed. Each case is evaluated under both the NoEC and WithEC settings. Figure 4b and c show lower validation success at greater transformation depths in both settings. Under WithEC, Codex with GPT-5.4 (A1) decreases from 80% at depth 1 to 59% at depth 10, Claude Code with Claude Sonnet 4.5 (A2) decreases from 72% to 51%, and OpenHands with DeepSeek- V3.2 (A3) decreases from 60% to 40%. Deeper transformations can introduce additional code fragments, intermediate states, and depen- dency relations around the injected bug. The agent must reason through the resulting implementation to construct a valid setup and oracle. Success generally decreases with depth, with small plateaus or reversals in individual cells. These comparisons characterize the evaluated transformation se- quences rather than the independent contribution of each operator. Table8 in AppendixF reports detailed statistics across transformation-depth buckets. AppendixG provides further discussion of transformation composition and injection-stage patch evolution. Finding 4: Greater transformation depth is generally associated with lower bug-validation success and provides a controllable benchmark-construction dimension. 4.6RQ5: HOW REALISTIC ARE THE INJECTED BUG PATCHES? The value of controlled bug injection depends on how closely its code changes resemble real de- velopment errors. We assess this resemblance through blinded source discrimination, using artifact- bearing positive controls to check judge sensitivity. We sample 700 of the 1,300 injected cases, stratified by repository and bug type, and match them to 700 historical bugs by repository and edit size. GPT-5.6-Terra and Claude Sonnet 4.6 independently see only blind IDs and production-code patches in the working-to-buggy direction. 8 Each judge performs single-patch judgment three times per patch, predicting its source (4,200 judg- ments), and pairwise judgment in both leftβright orientations, selecting the more realistic patch (1,400 judgments). For 100 sampled pairs, we add conspicuous code artifacts to the injected patch while preserving its bug behavior and repeat the pairwise task (200 judgments). All calls use inde- pendent contexts, giving 11,600 judgments across both models. Pairwise accuracy is the primary equivalence endpoint; single-patch accuracy and area under the receiver operating characteristic curve (ROC AUC) are auxiliary endpoints. Chance corresponds to 0.5. We use 10,000 whole-cluster bootstrap resamples, retaining related judgments together, and require each endpointβs complete 90% confidence interval (CI) to lie within 0.45β0.55 ( Lakens, 2017;Field & Welsh,2007). Positive controls require accuracy of at least 80% and a 95% CI above 50%. AppendixH details the protocol and diagnostics. Pairwise accuracy Single-patch accuracy Single-patch AUC 0.45 0.50 0.55 Score (a) Main sample (90% CI) GPT-5.6-TerraClaude Sonnet 4.6 GPT-5.6- Terra Claude Sonnet 4.6 50% 80% 100% Accuracy 92.0% 84.5% (b) Controls (95% CI) Figure 5: Blinded patch judgments. (a) Main-sample estimates and 90% cluster CIs; shading marks the 0.45β0.55 equivalence range and the dashed line marks chance. (b) Artifact-bearing controls with 95% CIs and the 80% accuracy threshold. Figure 5 reports pairwise accuracies of 51.64% and 52.14%. Both judges meet the primary and aux- iliary equivalence criteria. Their positive-control accuracies of 92.00% and 84.50% show sensitivity to conspicuous artifacts. The supplementary pairwise AUC interval for Claude Sonnet 4.6 slightly exceeds the equivalence bound; AppendixH reports this result and position preferences. Finding 5: Injected patches resemble historical bugs in blinded comparisons, combining code- level realism with controlled construction for evaluating executable bug validation. 5RELATED WORK Input Generation. Randoop, EvoSuite, and DART generate tests using execution feedback, cov- erage objectives, or path constraints (Pacheco et al.,2007;Fraser & Arcuri,2011;Godefroid et al., 2005). KATCH, AFLGo, WAFLGo, CAFL, and SDFuzz direct execution using locations, patches, data constraints, or vulnerability states (Marinescu & Cadar,2013;BΓΆhme et al.,2017;Xiang et al., 2024;Lee et al.,2021;Li et al.,2024). Target-directed input generation shares the need to satisfy the conditions that expose a bug. WITNESSGYM evaluates repository cases with supplied bug de- scriptions and locations, asking agents to construct both the input and its execution harness. Benchmarks.LIBRO, Issue2Test, and BRT Agent generate reproducing tests from bug re- ports ( Kang et al.,2023;Nashid et al.,2026;Cheng et al.,2025); AnyPoC validates candidate re- ports, and Differential Prompting uses program pairs to generate failure-inducing tests (Zhao et al., 2026;Li et al.,2023).CyberGym (Wang et al.,2026), SEC-bench (Lee et al.,2025), SEC- bench Pro (Lee et al.,2026), and exploit generators (Simsek et al.,2026;Andersson et al.,2026; Zhang et al.,2026) evaluate vulnerability reproduction; SecCodeBench-V2 evaluates secure code generation and repair (Chen et al.,2026). Defects4J curates faultβtest pairs, while LAVA, Evil- Coder, and FixReverter inject bugs ( Just et al.,2014;Dolan-Gavitt et al.,2016;Pewny & Holz, 2016;Zhang et al.,2022). WITNESSGYM creates multi-category targets with construction-time wit- nesses and controlled structural variants. Task and Data Construction. Change2Task derives verified coding-agent tasks from repository history across five maintenance task families (Qi et al.,2026). Related work on data governance 9 studies how heterogeneous sources can be selected and combined during multilingual model adap- tation ( Qi et al.,2025). WITNESSGYM complements these efforts by constructing fresh, execution- validated targets specifically for bug-witness construction. 6CONCLUSION This paper presents WITNESSGYM, an automated framework for constructing benchmarks for exe- cutable bug validation. It injects bugs into real projects and retains cases with a witness that exposes the faulty behavior. Bug-preserving transformations vary the surrounding program structure, allow- ing validation to be studied across different contextual demands. Our evaluation shows that cod- ing agents struggle to construct witnesses as execution contexts grow and transformations deepen. Blinded comparisons also find the injected patches difficult to distinguish from historical bugs un- der the evaluated protocol. WITNESSGYM offers a way to generate fresh validation tasks as agents evolve and study progress toward more reliable AI-assisted code auditing. REFERENCES Vivi Andersson, Sofia Bobadilla, Harald Hobbelhagen, and Martin Monperrus. PoCo: Agentic proof-of-concept exploit generation for smart contracts. ACM Transactions on Software Engi- neering and Methodology, 2026. doi: 10.1145/3816704. Anthropic. Introducing Claude Sonnet 4.5, 2025. URL https://w.anthropic.com/news/ claude-sonnet-4-5. Accessed September 24, 2026. Antonis Antoniades, Albert Γrwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. SWE-Search: Enhancing software agents with Monte Carlo tree search and iterative refinement. In The Thirteenth International Conference on Learning Representations, 2025. Marcel BΓΆhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. Directed grey- box fuzzing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communi- cations Security, p. 2329β2344. ACM, 2017. doi: 10.1145/3133956.3134020. Longfei Chen, Ji Zhao, Lanxiao Cui, Tong Su, Xingbo Pan, Ziyang Li, Yongxing Wu, Qijiang Cao, Qiyao Cai, Jing Zhang, Yuandong Ni, Junyao He, Zeyu Zhang, Chao Ge, Xuhuai Lu, Zeyu Gao, Yuxin Cui, Weisen Chen, Yuxuan Peng, Shengping Wang, Qi Li, Yukai Huang, Yukun Liu, Tuo Zhou, Terry Yue Zhuo, Junyang Lin, and Chao Zhang. SecCodeBench-V2 technical report. arXiv preprint arXiv:2602.15485, 2026. Runxiang Cheng, Michele Tufano, JΓΌrgen Cito, JosΓ© Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. Agentic bug reproduction for effective automated program repair at Google. arXiv preprint arXiv:2502.01821, 2025. DeepSeek-AI.DeepSeek-V3.2 model card, 2025.URL https://huggingface.co/ deepseek-ai/DeepSeek-V3.2. Accessed September 24, 2026. Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, William K. Robertson, Frederick Ulrich, and Ryan Whelan. LAVA: Large-scale automated vulnerability ad- dition. In IEEE Symposium on Security and Privacy, SP 2016, p. 110β121. IEEE Computer Society, 2016. Xueying Du, Kai Yu, Chong Wang, Yi Zou, Wentai Deng, Zuoyu Ou, Xin Peng, Lingming Zhang, and Yiling Lou. Minimizing false positives in static bug detection via LLM-enhanced path feasi- bility analysis. arXiv preprint arXiv:2506.10322, 2025. C. A. Field and A. H. Welsh. Bootstrapping clustered data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(3):369β390, 2007. doi: 10.1111/j.1467-9868.2007.00593. x. Gordon Fraser and Andrea Arcuri. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Confer- ence on Foundations of Software Engineering, p. 416β419. ACM, 2011. doi: 10.1145/2025113. 2025179. 10 Derin Gezgin, Amartya Das, Shinhae Kim, Zhengdong Huang, Nevena Stojkovic, and Claire Wang. PoC-Gym: Towards more reliable LLM-assisted proof-of-concept exploit generation. arXiv preprint arXiv:2602.04165, 2026. GitHub. CodeQL query help for Java and Kotlin. CodeQL documentation, 2026. URL https: //codeql.github.com/codeql-query-help/java/. Accessed September 24, 2026. Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: Directed automated random testing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Imple- mentation, p. 213β223. ACM, 2005. doi: 10.1145/1065010.1065036. Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. RepoAudit: An au- tonomous LLM-agent for repository-level code auditing. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 21083β21100. PMLR, 2025. RenΓ© Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In International Symposium on Software Testing and Analysis, ISSTA 2014, p. 437β440. ACM, 2014. Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. In Proceedings of the 45th IEEE/ACM International Con- ference on Software Engineering, p. 2312β2323, 2023. doi: 10.1109/ICSE48619.2023.00194. DaniΓ«l Lakens.Equivalence tests: A practical primer for t tests, correlations, and meta- analyses. Social Psychological and Personality Science, 8(4):355β362, 2017. doi: 10.1177/ 1948550617697177. Gwangmu Lee, Woochul Shim, and Byoungyoung Lee. Constraint-guided directed greybox fuzzing. In 30th USENIX Security Symposium, p. 3559β3576. USENIX Association, 2021. Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. SEC-bench: Automated Benchmark- ing of LLM Agents on Real-World Software Security Tasks. In Advances in Neural Information Processing Systems, volume 38, p. 128940β128976, 2025. doi: 10.52202/085713-3878. Hwiwon Lee, Jiawei Liu, Dongjun Kim, Wubing Xia, Ziqi Zhang, Chunqiu Steven Xia, and Ling- ming Zhang. SEC-bench Pro: Can language models solve long-horizon software security tasks? arXiv preprint arXiv:2605.26548, 2026. Penghui Li, Wei Meng, and Chao Zhang. SDFuzz: Target states driven directed fuzzing. In 33rd USENIX Security Symposium, p. 2441β2457. USENIX Association, 2024. Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. Nuances are the key: Unlocking ChatGPT to find failure-inducing tests with differential prompt- ing. In 38th IEEE/ACM International Conference on Automated Software Engineering, p. 14β26, 2023. doi: 10.1109/ASE56229.2023.00089. Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang. LLM-powered test case generation for detecting bugs in plausible programs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 430β440, 2025. Paul Dan Marinescu and Cristian Cadar. KATCH: High-coverage testing of software patches. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, p. 235β245. ACM, 2013. doi: 10.1145/2491411.2491438. Mistral AI. Devstral-2-123B-Instruct-2512 model card, 2025. URL https://huggingface.co/ mistralai/Devstral-2-123B-Instruct-2512. Accessed September 24, 2026. Noor Nashid, Islem Bouzenia, Michael Pradel, and Ali Mesbah. Issue2Test: Generating reproducing test cases from issue reports. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, p. 754β766, 2026. doi: 10.1145/3744916.3773129. 11 OpenAI.IntroducingGPT-5.4,2026.URL https://openai.com/index/ introducing-gpt-5-4/. Accessed September 24, 2026. Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-directed ran- dom test generation. In 29th International Conference on Software Engineering, p. 75β84. IEEE, 2007. doi: 10.1109/ICSE.2007.37. Wanzong Peng, Lin Ye, Xuetao Du, Hongli Zhang, Dongyang Zhan, Yunting Zhang, Yicheng Guo, and Chen Zhang. PwnGPT: Automatic exploit generation based on large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, p. 11481β11494. Asso- ciation for Computational Linguistics, 2025. Jannik Pewny and Thorsten Holz. EvilCoder: Automated bug insertion. In Proceedings of the 32nd Annual Conference on Computer Security Applications, ACSAC 2016, p. 214β225. ACM, 2016. Sophia Xiao Pu, Sitao Cheng, Xin Eric Wang, and William Yang Wang. Dynamic evaluation for oversensitivity in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 2337β2344, 2025. Haomin Qi, Chengbo Huang, Zihan Dai, and Yunkai Gao. Governance-aware hybrid fine-tuning for multilingual large language models. In 2025 IEEE International Conference on Big Data (BigData), p. 5507β5516. IEEE, 2025. doi: 10.1109/BigData66926.2025.11401808. Haomin Qi, Xingliang Wang, Xuanqi Gao, Baihui Sang, Xin Zhang, Minghua Ma, Pengfei Gao, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Change2Task: From repository changes to executable coding agent tasks and environments. arXiv preprint arXiv:2607.28591, 2026. Qwen Team. Qwen3-Coder-480B-A35B-Instruct model card, 2025. URL https://huggingface. co/Qwen/Qwen3-Coder-480B-A35B-Instruct. Accessed September 24, 2026. Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. Benchmarks as microscopes: A call for model metrology. In First Conference on Language Modeling, 2024. Donald J. Schuirmann. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Bio- pharmaceutics, 15(6):657β680, 1987. doi: 10.1007/BF01068419. Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security. In Advances in Neural Information Processing Sys- tems, volume 37, p. 57472β57498, 2024. doi: 10.52202/079017-1832. Deniz Simsek, Aryaz Eghbali, and Michael Pradel. PoCGen: Generating proof-of-concept exploits for vulnerabilities in npm packages. Proceedings of the ACM on Software Engineering, 3(FSE): 3887β3908, 2026. doi: 10.1145/3808178. Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA: analyzing dataflow in code with large language models. In Advances in Neural Infor- mation Processing Systems, volume 37, p. 131545β131574, 2024. doi: 10.52202/079017-4181. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI soft- ware developers as generalist agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025. OpenReview.net, 2025. Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. CyberGym: Evaluating AI Agentsβ Real-World Cybersecurity Capabilities at Scale. In The Fourteenth Inter- national Conference on Learning Representations, 2026. 12 Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. AntiLeakBench: Preventing data contamina- tion by automatically constructing benchmarks with updated real-world knowledge. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, p. 18403β18419. Association for Computational Linguistics, 2025. Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. Live-SWE-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025. Yi Xiang, Xuhong Zhang, Peiyu Liu, Shouling Ji, Hong Liang, Jiacheng Xu, and Wenhai Wang. Critical code guided directed greybox fuzzing for commits. In 33rd USENIX Security Symposium, p. 2459β2474. USENIX Association, 2024. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, volume 37, p. 50528β50652, 2024. Z.AI. GLM-4.7 model card, 2025. URL https://huggingface.co/zai-org/GLM-4.7. Ac- cessed September 24, 2026. Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Ja- son Khiem Tran, Nishka Kacheria, Ethan Jun-shen Ho, Denis Liu, Lauren McLane, Olivia Beyer Bruvik, Dai-Rong Han, Seungwoo Kim, Akhil Vyas, Cuiyuanxiu Chen, Ryan Li, Weiran Xu, Jonathan Z. Ye, Prerit Choudhary, Siddharth M. Bhatia, Vikram Sivashankar, Yuxuan Bao, Dawn Song, Dan Boneh, Daniel E. Ho, and Percy Liang. BountyBench: Dollar Impact of AI Agent At- tackers and Defenders on Real-World Cybersecurity Systems. In Advances in Neural Information Processing Systems, volume 38, 2025a. doi: 10.52202/085713-5725. Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Haoxiang Yang, Aolin Zhang, Rishi Al- luri, Nathan Tran, Rinnara Sangpisit, Kenny Oseleononmen, Dan Boneh, Daniel Ho, and Percy Liang. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview.net/forum?id=tc90LV0yRL. Jingwen Zhang, Yuhong Nan, Kaiwen Ning, Mingxi Ye, Wei Li, Yuming Xiao, Yuming Feng, Weizhe Zhang, and Zibin Zheng. V2E: Validating smart contract vulnerabilities through profit- driven exploit generation and execution. Proceedings of the ACM on Software Engineering, 3 (FSE):2259β2281, 2026. doi: 10.1145/3808108. Zenong Zhang, Zach Patterson, Michael Hicks, and Shiyi Wei. FIXREVERTER: A realistic bug injection methodology for benchmarking fuzz testing. In 31st USENIX Security Symposium (USENIX Security 22), p. 3699β3715. USENIX Association, 2022. Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, and Lingming Zhang. AnyPoC: Universal proof-of-concept test generation for scalable LLM-based bug detection. arXiv preprint arXiv:2604.11950, 2026. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. CVE-Bench: A Benchmark for AI Agentsβ Ability to Exploit Real- World Web Application Vulnerabilities. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 79850β79867. PMLR, 2025. 13 ALIMITATIONS The current implementation covers ten observable bug types in Maven-based Java projects with replayable tests. Additional bug specifications and language-specific tracing and replay adapters would extend this coverage. Context-length and transformation-depth comparisons use a fixed bug type to control the target semantics; broader combinations would test how these factors interact across types. The blinded patch study characterizes source discrimination under the two reported judges and presentation protocol. Broader model coverage and developer assessments could ex- amine other aspects of realism, including how injected bugs resemble those encountered during software maintenance. BBROADER IMPACT This work advances the evaluation of AI coding agents on executable bug validation in code-auditing scenarios. The injected bugs are isolated benchmark artifacts and are not changes proposed for deployed software. The framework can support research on evidence-grounded code auditing and on coding agents that validate reported bugs through execution. Bug-validation performance varies with execution context length, transformation depth, and bug type across the evaluated agents. These dimensions support construction of cases with different reasoning requirements and difficulty levels. The current implementation focuses on Maven-based Java projects and bug types derived from Cod- eQL Java queries. Applying the workflow to another ecosystem requires a runnable test environ- ment, observable validation signals, and language-specific adapters. The benchmark methodology can support further study of bug validation for AI-assisted software engineering and security analy- sis. CIMPLEMENTATION DETAILS Project and Bug Type Preparation. We instantiate WITNESSGYM on six real-world Java projects spanning distributed middleware, data serialization, and multipurpose utility libraries. These projects provide injection sites across varied execution contexts. All selected projects use Maven as their build system, which allows an individual test case to be replayed with a localized command such as: ./mvnw -q -pl <module> -Dtest=<test-class> test. We exclude tests that depend on external services, network access, nondeterministic timing, or ex- cessive runtime. Each experiment starts from a clean repository template, and all injection, transfor- mation, and evaluation runs are performed in isolated working copies. The current implementation targets Maven-based Java projects. Adapting the workflow to another language or build system requires isolated execution of a designated test and corresponding tracing, transformation, and vali- dation adapters. For bug type selection, we require the injected bug to produce an observable failure, such as an as- sertion failure or a runtime exception. Types whose effects are difficult to observe deterministically, such as pure performance regressions, are outside the scope of this benchmark. Among the eligi- ble patterns surveyed from CodeQL Java query examples, we randomly selected ten types across the three categories to bound construction and evaluation cost. Representative types include Guard Predicate Inversion, Intra-Function Null Dereference, and Comparison of Identical Values. Table 5 lists all ten types organized by category. Execution Context Collection. For each selected test case, we collect the ordered production- side execution path exercised by the test and summarize it as its execution context. To study how execution-context length affects bug validation, we stratify the collected execution contexts into length-based buckets. Here, execution context length refers to the number of distinct production- side functions reached during the test run, as recorded in the execution log Ο t . Specifically, we use four buckets: short, mid-short, mid-long, and long, with execution context lengths of [1, 40], 14 Table 5: The ten bug types used in the benchmark construction. CategoryTypeBug-type description API Contract Comparison of Identical ValuesA condition compares a value with itself, pro- ducing a constant result that silently disables validation or equality logic. Typo in equalsA function overriding equals(Object) has the wrong name or signature, so Java keeps us- ing reference equality. Typo in hashCodeA function intended to override hashCode() has the wrong name or signature, so hashed col- lections use identity hashing. Hashed Value Without hashCodeA class overrides equals(Object) but not hashCode(), so logically equal keys may hash to different buckets. Inconsistent equals and hashCode equals and hashCode use different fields or unstable state, breaking lookup in HashMap or HashSet. Value Flow Intra-Function Null DereferenceA value that was checked for null is later over- written, moved outside the guard, or derefer- enced on an unguarded path. Collection Element Null DereferenceMaintenance or eviction leaves a null element in a list, map, or cache, and a later lookup or iteration assumes it is non-null. Logic Guard Predicate InversionA guard condition is flipped or weakened, send- ing valid inputs to an error or skip path, or let- ting invalid inputs continue. Skipped Aggregation CounterA branch still processes an item but skips the counter or total update, so returned metrics un- dercount the work performed. In-Bounds Collection MisroutingA loop writes an element to a legal but wrong index or key, leaving the collection complete but logically misordered. [41, 70], [71, 120], and [121, 150] production functions, respectively. Shorter contexts correspond to more constrained execution paths. Longer contexts capture broader execution paths that may traverse multiple production functions, classes, or modules. Buggy Code Injection. We perform bug injection with Claude Code backed by Claude Sonnet 4.5. For each candidate case, the injection process is allowed up to three attempts. Before every attempt, the repository is restored to a clean workspace so that failed attempts do not accumulate state. A candidate is archived only if it satisfies three sequential validation checks. First, it must introduce a genuine production-code diff; cases with no change, formatting-only edits, or modifications confined to test or build files are rejected. Second, the diff must be structurally consistent with the intended bug type; we use lightweight syntactic checks to filter out obvious mismatches, such as injecting a null-pointer bug when the requested type is an equality inconsistency. Third, the replay command must pass on the clean baseline and expose the injected bug on the modified repository; cases that fail to compile, fail for unrelated infrastructure reasons, or do not expose the intended bug are discarded. Bug-Preserving Transformation. After obtaining a valid injected case, we apply bug-preserving transformations, also using Claude Code backed by Claude Sonnet 4.5. We allow one attempt per transformation. To diversify the structural embedding of injected bugs, we define ten transformation operators across three families. Table 6 lists all ten operators. Each transformation prompt instructs the agent to rewrite the region containing the injected bug while preserving the construction-time witness. We apply 1, 3, 5, or 10 operators sequentially after injection. Each step rebuilds the project and reruns the replay command. A transformed case is retained only when the project remains valid and the same witness still exposes the injected bug. The transformations therefore vary program structure without repairing the bug. 15 Table 6: The ten bug-preserving transformation operators. FamilyOperatorEffect Call Stack Helper Call InsertionAdds an internal helper or wrapper call between the original caller and the bug site, increasing call-chain depth. Call-Site ReroutingReplaces a direct internal call with a forwarding function, so the same logic is reached through an extra dispatch step. Extract-and-Apply SplitSplits one computation into an extraction helper and an application helper, forcing the bug to cross function boundaries. Data Flow Parameter BundlingPacks related arguments or locals into an inter- nal object that is passed through helper calls in- stead of using separate values. Local State BundlingMoves related local variables into a context ob- ject, so later code reads state through fields rather than direct locals. Intermediate State PassingStores intermediate values in a state holder and reads them later across helper calls, lengthening the value flow path. Cached Value AccessRoutes reads of a derived value through a cached accessor instead of direct recomputation or direct field access. Nested Data AccessReplaces direct value access with access through a nested container, wrapper, map entry, or list element. Pipeline Stage SplitBreaks a multi-step function into ordered in- ternal stages, making the bug appear inside a staged processing pipeline. Control Flow Rare Input Path ConditioningRestricts the bug path to rare but determinis- tic input conditions, preserving normal behav- ior for common inputs. DDETAILED BENCHMARK STATISTICS The constructed benchmark contains 1,300 archived buggy cases from six real-world Java projects. Each case stores a buggy repository snapshot, the execution context associated with the selected test, the applied bug type, the transformation sequence, and the construction-time witness used for validation. Figure6 summarizes the benchmark composition. At the case level, the benchmark contains 201 execution contexts: 56 short, 45 mid-short, 45 mid-long, and 55 long contexts. The benchmark covers all ten bug types in Table 5. Guard Predicate Inversion appears most fre- quently, with 688 cases, because it is used as the anchor type for systematically varying transforma- tion depth and execution context length. The remaining nine types each appear in 68 cases, yielding balanced coverage over the other Logic, Value Flow, and API Contract bugs. At the category level, the benchmark therefore contains 824 Logic cases, 136 Value Flow cases, and 340 API Contract cases. Across all cases, the ten transformation operators in Table6 appear 4,406 times in total. The most frequent operators are Call-Site Rerouting (1,028 occurrences), Helper Call Insertion (1,026), Local State Bundling (801), and Rare Input Path Conditioning (609), followed by Intermediate State Passing (218), Parameter Bundling (158), Cached Value Access (151), Nested Data Access (146), Pipeline Stage Split (141), and Extract-and-Apply Split (128). Approximately 63% of attempted candidates pass the construction pipeline. The estimated cost includes failed attempts and successful bug injection, transformation, execution-context collection, replay validation, and construction-time witness validation. The end-to-end construction cost is approximately $16,500. 16 0102030405060 Execution contexts Short Mid-short Mid-long Long 56 45 45 55 (a) Execution-context length 0200400600800 Cases Guard Predicate Inversion Other Logic Value Flow API Contract 688 136 136 340 (b) Bug type composition 02004006008001000 Occurrences Call-Site Rerouting Helper Call Insertion Local State Bundling Rare Path Conditioning Intermediate State Parameter Bundling Cached Value Access Nested Data Access Pipeline Split Extract-Apply Split 1,028 1,026 801 609 218 158 151 146 141 128 (c) Transformation operator usage Figure 6: The distributions of execution context length, different categories of bug types, and the occurrence of transformation operators. Table 7: Bug-validation success rates across execution context length buckets. Subscript annotations give percentage-point differences from the short bucket. Agent ID ShortMid-ShortMid-LongLong NoECWithECNoECWithECNoECWithECNoECWithEC A180%80%72% β8p 76% β4p 64% β16p 64% β16p 48% β32p 52% β28p A272%76%64% β8p 68% β8p 56% β16p 56% β20p 40% β32p 40% β36p A360%64%52% β8p 56% β8p 40% β20p 44% β20p 28% β32p 28% β36p A460%60%52% β8p 56% β4p 40% β20p 40% β20p 28% β32p 28% β32p A544%44%36% β8p 40% β4p 28% β16p 28% β16p 16% β28p 20% β24p A624%28%20% β4p 20% β8p 12% β12p 16% β12p 8% β16p 8% β20p EEXTENDED ANALYSIS OF EXECUTION CONTEXT EFFECTS This appendix expands RQ3 using the same 400-case controlled subset of Guard Predicate Inversion used in the main text. The subset contains four execution context length buckets and four transfor- mation depth buckets, with 25 validated cases for each bucket combination. Table 7 groups these cases by execution context length and aggregates over transformation depths. Table7 gives a finer-grained view of how execution-context length affects validation. Success de- creases with longer contexts for every agent under both NoEC and WithEC. Under WithEC, A1 drops from 80% to 52%, A2 from 76% to 40%, A3 from 64% to 28%, and A4 from 60% to 28% between the short and long buckets. Execution-context guidance improves several buckets, while success still declines as contexts grow. Under WithEC, A1 loses 28 percentage points and A2 loses 36 percentage points from short to long contexts. Longer contexts require agents to select relevant setup operations, preserve intermediate states, and construct an oracle that exposes the bug. The decline is often progressive across buckets. Longer execution contexts expand the program behavior that a witness must reconstruct before the bug becomes observable. FEXTENDED ANALYSIS OF TRANSFORMATION DEPTH EFFECTS This appendix expands RQ4 using the same 400-case controlled subset. The subset contains four transformation depth buckets and four execution context length buckets, with 25 validated cases for each bucket combination. Table8 groups these cases by transformation depth and aggregates over execution context lengths. Table 8 expands the transformation-depth analysis. Under WithEC, success from depth 1 to depth 10 decreases from 80% to 59% for A1, 72% to 51% for A2, 60% to 40% for A3, and 55% to 40% for A4. Transformation depth therefore provides a practical dimension for adjusting bug-validation difficulty. 17 Table 8: Bug-validation success rates across transformation-depth buckets. Subscript annotations give percentage-point differences from depth 1. Agent ID Depth 1Depth 3Depth 5Depth 10 NoECWithECNoECWithECNoECWithECNoECWithEC A176%80%67% β9p 68% β12p 62% β14p 65% β15p 59% β17p 59% β21p A270%72%59% β11p 60% β12p 53% β17p 57% β15p 50% β20p 51% β21p A357%60%45% β12p 49% β11p 42% β15p 43% β17p 36% β21p 40% β20p A453%55%47% β6p 48% β7p 41% β12p 41% β14p 39% β14p 40% β15p A538%40%33% β5p 32% β8p 28% β10p 32% β8p 25% β13p 28% β12p A622%23%17% β5p 16% β7p 12% β10p 16% β7p 13% β9p 17% β6p The effect of transformation depth differs from the effect of execution context length. Execution context length increases the amount of triggering behavior that the test must reconstruct, whereas transformation depth changes how the injected bug is structurally embedded in the code. As more transformations are applied, the same underlying bug may be surrounded by additional code frag- ments, modified control structure, or altered data dependencies. Consequently, the agent must rea- son through a transformed implementation before it can synthesize a valid setup and oracle. This makes transformation depth a construction-side complexity factor that complements execution con- text length. Transformation depth yields an overall difficulty trend with small plateaus and reversals in individual cells. For example, under WithEC, OpenHands with Devstral-2-123B (A5) remains at 32% from depth 3 to depth 5, and OpenCode with Qwen3-Coder-480B-A35B-Instruct (A6) increases from 16% at depth 5 to 17% at depth 10. Transformations are compositional: each additional step can affect code shape, control flow, and data flow in different ways. Transformation count thus provides a coarse-grained difficulty control; finer-grained metrics would account for the structural and semantic effects of individual operators. GADDITIONAL ANALYSIS OF TRANSFORMATION DIMENSION EFFECTS Transformation composition. The transformation families change different aspects of the code surrounding an injected bug. Call-stack operators introduce helper boundaries or additional dispatch steps. Data-flow operators change how values are stored, accessed, and passed between functions, while control-flow operators change the conditions under which the buggy path is reached. These changes can affect the program state and call sequence that an agent must reconstruct before a test exposes the bug. A transformed case may contain multiple operators, and later steps can reshape code introduced by earlier ones. The depth comparisons in RQ4 therefore concern the evaluated transformation sequences rather than the independent contribution of each operator. Grouping cases by operator occurrence alone does not separate these contributions. We examine the resulting patches below to describe how their size, structure, and retained content vary with transformation depth. Injection-stage patch evolution. We analyze the injection-stage artifacts to examine how trans- formation depth changes the structure of the generated buggy patches. This analysis uses the same 400-case transformation-depth subset as RQ4, with one fixed bug type, four transformation-depth buckets, and 100 successfully validated injected cases per bucket. Across these cases, we analyze 1,900 step-level transformation patches and the final buggy patch retained for each case. The analy- sis complements the validation results by describing the code changes produced during construction. Figure 7 and Table9 show that greater transformation depth is associated with larger final patches and more hunks, without substantially spreading edits across more files. The mean final patch size grows from 65.0 non-empty diff lines at depth 1 to 243.7 at depth 10, while the mean hunk count grows from 2.33 to 6.09. In contrast, the mean number of changed files remains nearly flat, from 18 13510 Transform depth 0 50 100 150 200 250 Final patch lines 65 98 161 244 Patch size keeps growing 13510 Transform depth 0 1 2 3 4 5 6 7 Final patch hunks More local edits, little file spread Hunks Changed files 13510 Transform depth 0% 20% 40% 60% 80% 100% Rate (%) Early shape is rewritten First-step line retention First-step sequence similarity 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Figure 7: Injection-stage patch evolution by transformation depth. Final patch size and hunk count grow with depth, while the number of changed files remains nearly flat. 13510 Transform depth 0 1 2 3 4 5 6 7 8 Accumulation factor 1.0x 2.4x 4.2x 6.6x Intermediate churn grows EarlyMiddleLate 0% 20% 40% 60% 80% 100% Rate (%) Depth 10 is dominated by later steps Line retention Sequence similarity Figure 8: Intermediate edit churn and step-position effects. At depth 10, later transformation steps have higher line retention and sequence similarity to the final buggy patch than earlier steps. 1.22 to 1.44. The additional edits therefore remain concentrated within a small number of files, even as the final patch becomes larger and more fragmented. Transformation depth also changes how earlier edits appear in the final buggy patch. First-step added-line retention remains relatively high, dropping from 100.0% at depth 1 to 74.9% at depth 10. However, first-step sequence similarity drops much more sharply, from 100.0% to 25.9%. Early transformation content often remains partly present even when its original arrangement changes. This pattern is consistent with later operators wrapping, splitting, moving, or rewriting earlier edits. The line-accumulation factor rises from 1.00Γ at depth 1 to 6.63Γ at depth 10, indicating that intermediate patches contain substantially more changed-line content in total than the final retained patch. Figure 8 further shows that later steps are more closely reflected in the final patch at depth 10. Early steps retain 77.5% of added lines on average, with 30.0% sequence similarity, while late steps reach 94.9% line retention and 80.7% sequence similarity. These observations are consistent with later transformations retaining parts of earlier edits while reshaping their structure. They also show why transformation depth should not be interpreted as a simple accumulation of unchanged edits. Together, these results provide a structural interpretation of the difficulty trend in RQ4. Deeper transformation sequences produce larger, more fragmented patches, while later steps change the arrangement of earlier edits. These changes may increase the code and state relationships that an agent must reason through when constructing an executable witness. The patch metrics describe the evaluated sequences for the fixed bug type; they do not by themselves establish which structural changes cause validation failures or isolate the effect of an individual operator. 19 Table 9: Injection-stage patch-evolution metrics by transformation depth. Line accumulation is the ratio between cumulative step-patch lines and final-patch lines. Retention and sequence similarity are measured from the first transformation step to the final buggy patch. DepthRunsFinal linesHunksChanged filesLine accum.First-step retentionFirst-step similarity 110065.02.331.221.00Γ100.0%100.0% 310097.83.591.402.40Γ88.1%61.7% 5100161.54.341.484.25Γ76.7%40.8% 10100243.76.091.446.63Γ74.9%25.9% HBLINDED PATCH REALISM STUDY H.1SAMPLES AND JUDGMENT PROTOCOL We sample 700 of the 1,300 injected cases, stratified by repository and bug type. Historical controls are matched by repository and edit size, also considering bug category, changed files, and hunks. This matching aims to reduce differences in repository style and patch size that could otherwise guide source judgments. Historical bug records establish provenance; pure refactorings, feature additions, and formatting changes are excluded. All production-code diffs use the working-to-buggy direction, reversing historical fixes where needed. We standardize formatting and context length, show blind IDs, and withhold commit metadata and source labels. In-patch comments, helpers, and structure remain visible so that judgments can still reflect code-level signs of artificial edits. Judge A is GPT-5.6-Terra with high reasoning effort; judge B is Claude Sonnet 4.6. Both judges receive identical instructions and materials in randomized order. Each judgment uses a fresh context, with no shared responses or source retrieval. Each patch receives three single-patch judgments, each producing an injected probability and a brief rationale. Probabilities of at least 0.5 yield the injected label. Each pair receives two judgments with positions swapped, each producing a left-more-realistic probability and a rationale. Probabilities of at least 0.5 select the left patch, and selecting the historical patch is scored as correct. Each model makes 4,200 single-patch and 1,400 main pairwise judgments. Positive Controls. Before judging, we sample 100 pairs and add unnecessary helpers, redundant wrappers, or unnatural comments to the injected patches, retaining their historical counterparts. Construction checks confirm that the bug behavior is preserved. The added artifacts contain no source labels, and controls are not selected based on judge success. These controls test whether the judges can recognize conspicuous artificial edits under the same blinded protocol. Evaluating both orientations adds 200 judgments per model, giving 5,800 judgments per model and 11,600 overall. The sensitivity check requires accuracyβ₯ 80% and a 95% CI lower bound > 50%. H.2STATISTICAL ANALYSIS Pairwise accuracy is the primary endpoint. For the two correctness indicators y i1 ,y i2 β 0, 1 of pair i, we compute s i = (y i1 + y i2 )/2 and average these scores over pairs. Single-patch accuracy and AUC are auxiliary endpoints, with injected as the positive class. Supplementary pairwise AUC uses the left-more-realistic probability and real-on-the-left as the positive class. The equivalence hypothesis is 0.45 < ΞΈ < 0.55, against ΞΈ β€ 0.45 or ΞΈ β₯ 0.55. This study- specific margin allows deviations from chance of less than five percentage points. Under the two one-sided testing principle (Ξ± = 0.05), each judgeβs complete 90% CI must lie strictly within these bounds ( Lakens,2017;Schuirmann,1987). An interval that merely includes chance is not sufficient to establish equivalence. Approximate intervals use 10,000 whole-cluster bootstrap re- samples ( Field & Welsh,2007), grouping patch repetitions, pair orientations, and cases sharing an original injection or historical control. This procedure keeps related judgments together rather than treating repeated calls as independent observations. Positive controls use 95% CIs, and the judges are analyzed separately. 20 Table 10: Detailed source-discrimination results. Accuracy CIs are percentages. TaskJudge Correct / Total Accuracy90% CIAUC90% CI SingleA2,173 / 4,20051.74%50.14β53.38 0.51198 0.49367β0.53075 SingleB2,189 / 4,20052.12%50.52β53.69 0.51286 0.49426β0.53084 Pairwise A723 / 1,40051.64%49.07β54.21 0.51681 0.48703β0.54633 Pairwise B730 / 1,40052.14%49.64β54.64 0.52411 0.49517β0.55252 Table 11: Artifact-bearing positive controls. Both judges meet the prespecified sensitivity check. JudgeCorrect / TotalIncorrect / TotalAccuracy95% CICheck A184 / 20016 / 20092.00%87.00%β96.00%Pass B169 / 20031 / 20084.50%78.50%β89.50%Pass Tables10 and11 show that both judges meet the primary and auxiliary equivalence criteria and pass the sensitivity check. The contrast between the main sample and positive controls suggests that the judges can detect conspicuous artifacts but have limited ability to distinguish the matched injected and historical patches. Judge Bβs supplementary pairwise AUC interval reaches 0.55252, slightly above the upper equivalence bound. Equivalence is therefore not established for this additional met- ric. These findings concern source discrimination by the two evaluated judges under this protocol; they do not establish equivalence between the broader distributions of injected and historical bugs. H.3PREFERENCE AND STABILITY DIAGNOSTICS In Figure 9, split means different underlying patch choices after swapping positions, not an explicit tie. For real-both, split, and injected-both counts R,S,I , correct judgments total 2R + S. These totals are 723/730 for the main sample and 184/169 for controls (A/B). Table 12: Judgment stability, position preferences, and correctness counts (1,400 patches per judge). DiagnosticJudge AJudge B Single-patch repeat agreement60.62%58.67% Patches with non-unanimous single judgments59.07%62.00% Main orientation-choice agreement68.43%64.29% Main LEFT choice rate52.93%54.71% Main LEFT choices / judgments741 / 1,400766 / 1,400 Main split pairs choosing LEFT twice131158 Patches with 0 / 1 correct single judgments277 / 369247 / 402 Patches with 2 / 3 correct single judgments458 / 296466 / 285 Repeat agreement averages the three pairwise comparisons of each patchβs single-patch judgments. Orientation agreement measures whether a judge selects the same underlying patch after positions are swapped. More than half the patches receive non-unanimous single-patch labels, showing sub- stantial variation across repeated judgments. Table12 also shows a left-choice tendency, especially for judge B. Evaluating both orientations balances the position of the historical patch, but does not establish position invariance. The aggregate equivalence results should therefore not be interpreted as stable decisions on every individual patch. For the per-patch counts N 0 ,...,N 3 in Table 12, correct judgments equal N 1 + 2N 2 + 3N 3 . Re- peat agreement equals [3(N 0 + N 3 ) + N 1 + N 2 ]/4,200, since unanimous labels agree in all three comparisons and non-unanimous labels agree in only one. 21 Real in bothSplitInjected in both 0%25%50%75%100% Main A 700 pairs 251221228 Main B 700 pairs 240250210 Control A 100 pairs 8884 Control B 100 pairs 74215 Figure 9: Preferences across both orientations. Numbers give pair counts; segment widths show percentages. Main groups contain 700 pairs and control groups contain 100. βSplitβ denotes incon- sistent choices, not an explicit tie. IPROMPT DESIGN We report the prompt structures used in WITNESSGYM. The implementation uses layered prompts: a reusable system prompt specifies global behavioral constraints, while a dynamically assembled task payload injects case-specific fields such as the bug type, bug type specification, transformation operator, execution context, replay command, candidate anchors, and file scope. For readability, repository-local paths and long execution context lists are shortened in this appendix, but the task goals, constraints, input fields, and required output formats follow the actual pipeline. Table 13: Layered prompt structures used by WITNESSGYM during benchmark construction and executable bug validation. Prompt StructureMain InputsRequired Output Bug Injection System PromptGlobal injection rules,replay- commanddiscipline,bounded editβrunβrevise loop, diff checks, and success criteria Single-line INJECT_REPORT_JSON with operational evidence fields Bug Injection Task PayloadBug type, bug type specification, re- quired and forbidden elements, ex- ecution context, candidate anchors, trigger hints, and file scope Same INJECT_REPORT_JSON pre- fix with type-site rationale fields Bug-Preserving Transformation Task Payload Transformation operator specifi- cation, applicability requirements, preservation constraints, execution context, and candidate anchors Single-line TRANSFORM_REPORT_JSON Bug Validation System PromptTest-only edit constraints,Java test-casematerializationrules, executable-witness objective, and evaluation-attempt policy Single-line TRIGGER_REPORT_JSONfall- back schema Bug Validation Task PayloadBuggy repository metadata, case ID, bug type, applied transforma- tions, bug-related files, replay com- mand, and optional execution con- text Single-line TRIGGER_REPORT_JSONwith detailed witness-construction fields 22 I.1BUG INJECTION SYSTEM PROMPT Bug Injection System Prompt Mission. Introduce a real, localized bug into production code so that the externally provided replay command fails for the intended bug type. The repository must end with a non-empty semantic git diff, and the replay command must fail because of the injected bug. Core rules. 1. Produce a non-empty semantic production-code diff. 2. Run the replay command exactly as provided, without changing the module, flags, test target, or command structure. 3. Do not modify tests unless explicitly allowed by the task. 4. Do not modify build files, dependency files, CI files, or repository configuration. 5. Do not change public or protected API signatures. 6. Keep the edit localized, realistic, and consistent with the repository style. 7. Do not claim success if the git diff is empty or if the observed failure is unrelated to the injected bug. Construction-side execution budget. During one bug-injection session, the implementation permits at most three runs of the provided replay command. If the replay command still passes, the agent must revise the production-code edit and rerun the same command within this budget. This construction-side replay budget is separate from the bug-validation attempt budget used in the evaluation. Required final output. The system prompt requires exactly one final single-line JSON object prefixed by INJECT_REPORT_JSON. The operational evidence fields include: ok, git_changed_files, repro_cmd, repro_exit_code, observed_exception, call_chain, and notes. I.2BUG INJECTION TASK PAYLOAD Bug Injection Task Payload Task-specific inputs. β’ Bug type and bug type ID. β’ Bug type name, category, scope, and language. β’ Bug type description. β’ Required elements that must appear in the patch. β’ Forbidden elements that must not appear in the patch. β’ Expected failure signal and trigger-condition hints. β’ Execution context summary collected from the target test. β’ Candidate anchors with trace-relevant methods, files, and candidate edit sites. β’ Allowed or preferred production-code file scope. Bug type contract. The injected bug must match the provided bug type specification rather than merely resemble it. Required elements must be visible in the actual patch, and forbidden elements must not appear in the final patch. The chosen site should be trace-relevant and should allow the provided replay command to trigger the injected bug after validation. Task-level reporting fields. The task payload uses the same INJECT_REPORT_JSON prefix and adds bug-type-site rationale fields: ok, type_id, target_files, target_symbols, selected_anchor, why_this_site, expected_trigger_path, stealth_rationale, bug_mechanism, preserved_prior_structure, and possible_risks. The implementation field stealth_rationale records the localization and plausibility of the edit. I.3BUG-PRESERVING TRANSFORMATION TASK PAYLOAD Bug-Preserving Transformation Task Payload The transformation prompt is a dynamically assembled task payload generated from an internal transformation operator specification. The payload uses implementation-level fields such as id, name, intent, description, applicability, forbidden_elements, must_preserve, and post_conditions. Each internal operator is mapped to a paper-facing transformation operator and family. For example, t_rare_profile_gate_v1 is reported as Rare Input Path Conditioning, 23 t_call_stack_deepen_v1 is reported as Helper Call Insertion, and t_private_adapter_layer_v1 is reported as a call-site rerouting / private-adapter transformation. Task. Apply one bug-preserving transformation step after successful bug injection. The transformation should alter the surrounding production-code structure while preserving compilation and preserving the replay behavior that exposes the injected bug. It is bug-preserving in the benchmark-construction sense: it preserves the relevant trigger path and bug-inducing behavior, but it is not required to preserve the full behavior of the original unmodified program. Inputs. β’ Internal transformation ID and operator name. β’ Paper-facing operator term and transformation family. β’ Operator intent and description. β’ Applicability requirements. β’ Forbidden edits. β’ Must-preserve constraints, including compilation success, public API signatures, behavior for non-trigger inputs when applicable, and replay-command triggerability. β’ Post-conditions expected from the operator. β’ Execution context summary and candidate anchors. Edit boundaries. 1. Do not modify tests unless explicitly allowed by the operator. 2. Do not change public or protected API signatures. 3. Do not perform broad refactors, renames, or formatting sweeps. 4. Keep the transformation localized to trace-relevant production code. 5. Preserve the injected bug-triggering behavior under the same replay command. Required final output. The payload requires exactly one final single-line JSON object prefixed by TRANSFORM_REPORT_JSON. The JSON object contains: ok, transform_id, changed_files, edit_sites, what_was_added_or_wrapped, what_was_preserved, what_was_replaced_or_overwritten, compatibility_with_previous_steps, expected_effect_on_stealth, and expected_effect_on_trigger_depth. The field expected_effect_on_stealth is the implementation field name for the expected surface-form and review-plausibility effect of the transformation. I.4BUG VALIDATION SYSTEM PROMPT Bug Validation System Prompt Mission. Create a Java test that exposes an already-injected production bug under the provided Maven replay command. The agentβs job is not to fix code, improve the repository, or write a generic regression test. The only goal is to materialize a test that exposes the injected bug. Evaluation attempt budget. For each benchmark case, the evaluation protocol allows at most three bug-validation attempts. Each attempt must follow the same test-only constraints and use the provided replay command as the validation target. A run succeeds only if the generated witness test compiles, the replay command exposes the injected bug, and no production file or build configuration is modified. Hard constraints. 1. Never modify production code. 2. Never modify build files, dependency files, CI files, configuration files, or repository metadata. 3. Only create or edit Java test files under src/test/java. 4. Prefer rewriting the official test file at the provided path and with the provided package and class name. 5. The witness test must materialize in the repository as a real .java file before verification. 6. The test must be designed to trigger the injected bug, not to validate normal behavior. 7. Do not delete or modify unrelated files. Required final output. The system prompt requires a final single-line TRIGGER_REPORT_JSON. The fallback schema includes: ok, test_strategy, target_bug_path_hypothesis, why_trigger_should_fail, and materialized_test_files. 24 I.5BUG VALIDATION TASK PAYLOAD Bug Validation Task Payload Task-specific inputs. β’ Repository name. β’ Evaluation group or context condition. β’ Injected case run ID. β’ Bug type ID and bug type. β’ Applied bug-preserving transformations. β’ Bug-related production files and location metadata. β’ Injection-stage verification exit code. β’ Maven replay command used for validation. β’ Natural-language bug summary. β’ Target test path and target test class when available. Execution context handling. The two evaluation settings are NoEC and WithEC. In WithEC, the task payload includes the execution context summary collected during benchmark construction. In NoEC, this execution context block is omitted. Neither setting provides the construction-time witness, hidden validation oracle, construction-time solution code, or permission to modify production code. Required final output. The task payload asks for a detailed final TRIGGER_REPORT_JSON schema with: summary, suspected_bug_mechanism, inspected_production_files, inspected_test_files, selected_test_path, selected_test_class, trigger_strategy, expected_failure_signal, materialized_test_files, and notes. I.6END-TO-END CASE PROMPT EXAMPLE This example shows a concrete case-level prompt flow assembled by the pipeline. The case uses repository dubbo, bug type Guard Predicate Inversion, and a depth-3 transformation chain. Long execution context lists are shortened for page fit, but the fields and constraints shown below follow the actual prompt structure. Case Metadata run_id: inject__20260315T064448Z repository: dubbo trace_group: short trace_id: org.apache.dubbo.common.url.URLParamTest bug_type: Guard Predicate Inversion type_id: logic_predicate_inversion_in_guard transform_depth: 3 transform_ids: - t_rare_profile_gate_v1 - t_call_stack_deepen_v1 - t_private_adapter_layer_v1 paper_facing_transform_terms: - Rare Input Path Conditioning - Helper Call Insertion - Call-Site Rerouting / Private Adapter Layer official_test_path: dubbo-common/src/test/java/org/apache/dubbo/common/url/URLParamTest.java official_test_class: org.apache.dubbo.common.url.URLParamTest replay_command: ./mvnw -q -pl dubbo-common -Dtest=org.apache.dubbo.common.url.URLParamTest test context_setting_for_executable_validation: WithEC evaluation_attempt_budget: at most 3 bug-validation attempts per benchmark case Bug Injection Prompt SYSTEM: You are operating inside a Git repository in an automated bug-injection pipeline. MISSION: Introduce a real, localized bug into production code so that the externally provided reproduction command fails for the intended bug type. Your work is only valid if the repository ends the session with a non-empty semantic git diff and the provided reproduction command, run verbatim, fails because of the injected bug. 25 NON-NEGOTIABLE RULES: - You must produce a non-empty semantic git diff in the current repository state before finalizing. - Semantic git diff means a real logic-changing edit, not whitespace-only, formatting-only, comment- only, import-only, rename-only, or mechanically trivial changes. - You must run the reproduction command exactly as provided in the task, verbatim. - Do not invent, replace, shorten, simplify, or substitute any command, module, flag, path, or test target. - Do not modify test code unless the task explicitly allows it. - Do not change public or protected API signatures. - Do not claim success based on a hypothetical edit. - Do not output a final success JSON if git diff is empty. - Do not output a final success JSON if the reproduction command in your JSON differs from the externally provided command. CONSTRUCTION-SIDE EXECUTION BUDGET: - Up to 3 total runs of the provided reproduction command in the same injection session. - If the reproduction command passes, make another production-code edit and rerun the same command verbatim within this budget. TASK PAYLOAD: You are performing controlled BUG INJECTION into production code for research benchmarking. Goal: introduce a subtle, realistic bug consistent with the specified bug type, while keeping changes localized and review-plausible. TYPE: - id: logic_predicate_inversion_in_guard - name: Predicate inversion in critical guard - category: Logic - scope: intra_method_branch_condition - language: java Description: A boolean predicate used to guard important logic is inverted or altered in a subtle way. The code still compiles, branches look plausible, but the semantics flip which path handles which inputs, causing assertions and invariants to fail. Required elements: - if_or_else_block_guarding_non_trivial_logic - condition_based_on_collection_size_or_status_flag - tests_that_distinguish_between_valid_and_invalid_input_cases Forbidden elements: - conditions_that_are_constant_true_or_false_after_modification - completely_removing_the_guard_and_inlining_the_body - branches_with_no_observable_effect Runtime effect hints: - exception_type_hint: none - typical_failure_signal: assertion failure or invalid return value Trigger condition hints: - test_shape_hint: tests feed different valid/invalid input categories and assert different behavior - input_properties_hint: at least one test relies on the original guard semantics TRACE CONTEXT SUMMARY: "trace_id": "org.apache.dubbo.common.url.URLParamTest", "test_name": "org.apache.dubbo.common.url.URLParamTest", "methods_count": 91 CANDIDATE ANCHORS: - method: URLParamTest file: dubbo-common/src/test/java/org/apache/dubbo/common/url/URLParamTest.java role: target test - method: hasMethodParameter file: dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java role: trace-relevant guard candidate - method: getMethodParameter file: dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java role: trace-relevant lookup candidate - method: getMethodParameterStrict file: dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java role: trace-relevant lookup candidate - method: initMethodParameters file: dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java role: method-parameter initialization path - method: size 26 file: dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java role: status/collection-size related helper - method: isEmpty file: dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java role: status/collection-size related helper DELIVERABLE: 1. Implement the bug strictly according to the bug type contract. 2. Keep the patch minimal and consistent with repository style. 3. Run the provided reproduction command verbatim. 4. Provide evidence in the final JSON. REQUIRED FINAL OUTPUT: Append exactly one final line with this prefix: INJECT_REPORT_JSON: ... Required field groups: - operational evidence: ok, git_changed_files, repro_cmd, repro_exit_code, observed_exception, call_chain, notes - type-site rationale: type_id, target_files, target_symbols, selected_anchor, why_this_site, expected_trigger_path, stealth_rationale, bug_mechanism, preserved_prior_structure, possible_risks Bug Injection Report INJECT_REPORT_JSON: "ok": true, "type_id": "logic_predicate_inversion_in_guard", "target_files": [ "dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java" ], "target_symbols": [ "getAnyMethodParameter" ], "selected_anchor": "getAnyMethodParameter", "why_this_site": "The method is reached by URLParamTest and contains a critical guard over method- specific parameter lookup.", "expected_trigger_path": "URLParamTest -> URLParam.getAnyMethodParameter -> inverted guard returns null instead of the expected method parameter.", "stealth_rationale": "The edit is a localized predicate substitution inside an existing defensive check.", "bug_mechanism": "A guard condition is inverted from a non-empty map check to an empty map check, swapping valid and invalid lookup paths.", "preserved_prior_structure": "Method signatures, surrounding branch structure, and other production logic remain unchanged.", "possible_risks": "The bug is most visible through method-parameter tests that expect lookup results from non-empty maps." Bug-Preserving Transformation Prompt TASK: Apply one bug-preserving transformation step after successful bug injection. The transformation must keep the repository compilable and preserve the replay behavior that exposes the injected bug. TRANSFORM SPEC 1: - id: t_rare_profile_gate_v1 - paper_facing_operator: Rare Input Path Conditioning - family: Control Flow - name: Gate misroute behind rare profile - description: restrict the misroute to a deterministic but low-probability input profile - applicability: at_least_one_mid_level_helper_candidate - forbidden_edits: changes_to_public_API_signatures, obvious_dead_branches_such_as_if_false - must_preserve: compilation_success, public_api_signatures, replay_command_triggerability - post_conditions: rare deterministic gate on the bug-related path TRANSFORM SPEC 2: - id: t_call_stack_deepen_v1 - paper_facing_operator: Helper Call Insertion - family: Call Stack - name: Deepen call stack with a natural helper extraction 27 - description: introduce a small helper method or lightweight wrapper to deepen the call stack - applicability: at_least_one_mid_level_helper_candidate - forbidden_edits: changes_to_public_API_signatures, broad refactors, formatting sweeps - must_preserve: compilation_success, public_api_signatures, behavior_for_non_trigger_inputs - post_conditions: additional_helper_in_call_chain TRANSFORM SPEC 3: - id: t_private_adapter_layer_v1 - paper_facing_operator: Call-Site Rerouting / Private Adapter Layer - family: Call Stack - name: Add a private adapter layer and reroute one call site - description: introduce a private adapter method and reroute at least one internal call site through it - applicability: at_least_one_internal_call_site_can_be_rerouted - forbidden_edits: changes_to_public_API_signatures, obvious_dead_branches_such_as_if_false - must_preserve: compilation_success, public_api_signatures, behavior_for_non_trigger_inputs - post_conditions: additional_adapter_in_call_chain TRACE CONTEXT SUMMARY: "trace_id": "org.apache.dubbo.common.url.URLParamTest", "test_name": "org.apache.dubbo.common.url.URLParamTest", "methods_count": 91 CANDIDATE ANCHORS: - URLParam.hasMethodParameter - URLParam.getMethodParameter - URLParam.getMethodParameterStrict - URLParam.initMethodParameters - URLParam.size - URLParam.isEmpty EDIT BOUNDARIES: - Do not modify tests unless explicitly allowed by the transform specification. - Do not change public or protected API signatures. - Avoid broad refactors, renames, or formatting sweeps. - Keep diffs localized to trace-relevant production code. - Preserve replay-command triggerability. REQUIRED FINAL OUTPUT: Append exactly one final line with this prefix: TRANSFORM_REPORT_JSON: ... Required keys: ok, transform_id, changed_files, edit_sites, what_was_added_or_wrapped, what_was_preserved, what_was_replaced_or_overwritten, compatibility_with_previous_steps, expected_effect_on_stealth, expected_effect_on_trigger_depth Bug-Preserving Transformation Report TRANSFORM_REPORT_JSON: "ok": true, "transform_id": "t_rare_profile_gate_v1 + t_call_stack_deepen_v1 + t_private_adapter_layer_v1", "changed_files": [ "dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java" ], "edit_sites": [ "URLParam.getMethodParameter", "URLParam.hasMethodParameter", "URLParam.keyIndexToOffset", "URLParam.computeValueOffset", "URLParam.resolveParameterWithEncoding" ], "what_was_added_or_wrapped": "Added private helper and adapter methods around parameter lookup and offset resolution.", "what_was_preserved": "Public APIs, compilation, and replay-command triggerability.", "what_was_replaced_or_overwritten": "Direct parameter and offset resolution call sites were rerouted through private helpers.", "compatibility_with_previous_steps": "The transformation chain preserves the injected guard- inversion behavior.", "expected_effect_on_stealth": "The resulting structure resembles localized helper extraction and profile-aware lookup logic.", "expected_effect_on_trigger_depth": "The execution path now includes additional helper and adapter calls before the wrong value is observed." 28 Bug Validation Prompt SYSTEM: You are an autonomous code agent operating inside a Java repository. You are running an executable bug-validation task for an already-injected production bug. Your job is not to fix code. Your job is not to improve the repository. Your job is not to write a generally reasonable test. Your only goal is to create a new Java test case that exposes the already-injected bug under the provided Maven verification command. EVALUATION ATTEMPT BUDGET: For each benchmark case, the evaluation protocol allows at most three bug-validation attempts. Each attempt must obey the same test-only constraints and use the provided replay command as the validation target. HARD CONSTRAINTS: 1. Never modify production code under src/main/java or any non-test source root. 2. Never modify build files, pom.xml files, Gradle files, CI files, config files, or dependency files. 3. Only create or edit Java test files under src/test/java. 4. Prefer rewriting the official testcase file at the same path and with the same package and class name when that information is provided. 5. The testcase must materialize in the repository as a real .java file before verification. 6. The testcase must be designed to trigger the injected bug, not to validate normal behavior. 7. Do not output file blocks for an external parser. Directly write files into the repository workspace. 8. Keep the testcase minimal but bug-targeted. 9. Do not delete or modify unrelated files. TASK PAYLOAD: Generate a new Java testcase that triggers the already-injected bug. Success criteria: - A real Java testcase file exists in the repository under src/test/java. - The testcase uses the correct package declaration. - The testcase compiles. - The provided verification command fails because the injected bug is triggered. - The failure is a semantic test failure, not a compile error and not a harness failure. - No production file or build configuration file is modified. Repository: dubbo Context setting: WithEC Implementation group: B Injected case run_id: inject__20260315T064448Z Type ID: logic_predicate_inversion_in_guard Bug type: Guard Predicate Inversion Applied transforms: t_rare_profile_gate_v1, t_call_stack_deepen_v1, t_private_adapter_layer_v1 Bug-related production files: - dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java Official test path: dubbo-common/src/test/java/org/apache/dubbo/common/url/URLParamTest.java Official test class: org.apache.dubbo.common.url.URLParamTest Verification command: ./mvnw -q -pl dubbo-common -Dtest=org.apache.dubbo.common.url.URLParamTest test Injection-stage verify rc: 1 Type summary: A boolean guard condition in production code was inverted, causing guarded URL-parameter lookup logic to execute under the wrong condition and leading to semantically incorrect behavior without changing public APIs. Execution context block: The target test reaches URLParam construction and method-specific parameter lookup. Trace-relevant production methods include URLParam.hasMethodParameter, URLParam.getMethodParameter, URLParam.getMethodParameterStrict, URLParam.initMethodParameters, URLParam.size, and URLParam. isEmpty. This execution context block identifies the path used during benchmark construction; it does not include the reference-trigger implementation or hidden validation oracle. What you must do: - Read the bug context carefully. - Inspect the relevant production and test files. - Rewrite or create the testcase directly at the official test path. - Keep the package name and class name correct. - Exercise the suspected buggy path through URLParam method-specific parameter lookup. 29 - Write an oracle that fails under the injected bug. - Run the provided verification command if needed. - Leave no production-code or build-configuration changes. Required final output: At the very end, print exactly one single-line JSON object prefixed by: TRIGGER_REPORT_JSON: The JSON object must have this schema: "summary": "...", "suspected_bug_mechanism": "...", "inspected_production_files": ["..."], "inspected_test_files": ["..."], "selected_test_path": "...", "selected_test_class": "...", "trigger_strategy": ["...", "...", "..."], "expected_failure_signal": "...", "materialized_test_files": ["..."], "notes": "..." Bug Validation Report TRIGGER_REPORT_JSON: "summary": "Generate a URLParam test that exercises method-specific parameter lookup after the transformed guard inversion.", "suspected_bug_mechanism": "A guard inversion and transformed lookup path cause method-specific parameter retrieval to return an incorrect value or miss a valid method entry.", "inspected_production_files": [ "dubbo-common/src/main/java/org/apache/dubbo/common/url/component/URLParam.java" ], "inspected_test_files": [ "dubbo-common/src/test/java/org/apache/dubbo/common/url/URLParamTest.java" ], "selected_test_path": "dubbo-common/src/test/java/org/apache/dubbo/common/url/URLParamTest.java", "selected_test_class": "org.apache.dubbo.common.url.URLParamTest", "trigger_strategy": [ "Construct URLParam with multiple parameters so that method-specific lookup and transformed helper paths are exercised.", "Call hasMethodParameter and getMethodParameter on methods that should be present.", "Assert the expected valid lookup result so that the injected guard inversion becomes observable." ], "expected_failure_signal": "Assertion failure: expected method-specific parameter to be present or equal to the expected value, but the transformed buggy path returns an incorrect result.", "materialized_test_files": [ "dubbo-common/src/test/java/org/apache/dubbo/common/url/URLParamTest.java" ], "notes": "The test is written only under src/test/java and does not modify production code or build configuration." 30