Paper deep dive

CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 178

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/13/2026, 3:37:41 AM

Summary

The paper introduces CheckerBench, a benchmark for evaluating long-horizon agentic static-analysis checker synthesis, and CheckerLab, an evaluation framework. CheckerBench contains 300 tasks derived from 297 CVEs across 167 repositories and five language ecosystems. The study evaluates 21 model-harness configurations, finding that current coding agents struggle with reliable checker development, achieving a mean Pass@1 of 32.30%.

Entities (11)

CheckerBench · benchmark · 100%CheckerLab · framework · 100%Claude Opus 4.8 · model · 95%GPT-5.6 · model · 95%Clang Static Analyzer · tool · 90%CodeQL · tool · 90%CVE-2017-7866 · vulnerability · 90%CWE-787 · vulnerabilitytype · 90%FFmpeg · repository · 90%Linux · repository · 90%OpenCode · harness · 90%

Relation Signals (8)

Claude Opus 4.8 → achievedbestpassat1 → 45.33%

confidence 95% · the best reaches 45.33%... Table 2... Claude Opus 4.8... Pass@1 (%) 45.33

CheckerBench → containstasksfrom → CVEs

confidence 95% · CheckerBench contains 300 real-world tasks collected from 297 CVEs

CheckerBench → usesevaluationframework → CheckerLab

confidence 95% · We further introduce CheckerLab, a common evaluation framework... Across 21 model-harness configurations... mean Pass@1 is 32.30%

CheckerBench → coverscwe → CWE-787

confidence 90% · The most frequent are CWE-787 (out-of-bounds write)... Figure 4(c)

CheckerBench → includesrepository → Linux

confidence 90% · CheckerBench contains 300 tasks from 167 open-source repositories, including Linux [38] (107 tasks)

CheckerBench → includesrepository → FFmpeg

confidence 90% · CheckerBench contains 300 tasks from 167 open-source repositories, including Linux [38] (107 tasks), FFmpeg [23] (4)

CheckerBench → supportsanalyzer → Clang Static Analyzer

confidence 90% · Static analysis environments are CSA (159) and CodeQL (141).

CheckerBench → supportsanalyzer → CodeQL

confidence 90% · Static analysis environments are CSA (159) and CodeQL (141).

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model-harness configurations and three independent repeats per configuration, mean Pass@1 is 32.30%, while the best reaches 45.33%. These results show that reliable, reusable checker development remains challenging for current coding agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

176,183 characters extracted from source content.

Expand or collapse full text

: Can Long-Horizon Agents Synthesize Static-Analysis Checkers? Hang He ∗ , Li Wang ∗ , Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan # ∗ Equal Contribution, # Corresponding Author East China Normal University, Humanlaya Data, Shanghai Jiao Tong University, Peking University, Shanghai Innovation Institute Abstract Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repos- itory, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable–fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model–harness configurations and three independent re- peats per configuration, mean Pass@1 is 32.30%, while the best reaches 45.33%. These results show that reliable, reusable checker development remains challenging for current coding agents. GitHub: https://github.com/ahang0712/CheckerBench-Dataset Hugging Face: https://huggingface.co/datasets/Benchanything/checkerbench HY4-Preview GLM-5.2 GPT-5.6 HY4-Preview Kimi-K3 Qwen3.8-Max GPT-5.6 DeepSeek-V4-Pro GLM-5.2GLM-5.2 HY4-Preview GPT-5.6 Claude Opus 4.8 Qwen3.8-Max Claude Opus 4.8 Qwen3.8-Max Kimi-K3 DeepSeek-V4-Pro Kimi-K3 DeepSeek-V4-Pro Claude Opus 4.8 0 5 10 15 20 25 30 35 40 45 50 Pass@1 (%) 21.67 22.33 23.00 23.67 24.33 25.00 31.67 32.3332.33 33.00 33.67 34.00 34.33 34.67 35.00 37.33 38.00 38.33 38.67 39.67 45.33 OpenCodeClaude CodeHermes Agent Figure 1 Pass@1 for 21 model–harness pairs on 300 CheckerBench tasks (Table 2). 1 arXiv:2610.07557v1 [cs.SE] 6 Oct 2026 Recurring evaluation gaps (a) Not checker-centric Tasks target individual bug fixes. Issue Coding agent Git patch Reusable checker A patch fixes one observed failure. A one-off patch is not a reusable checker. (b) Limited interaction horizon The workflow follows predefined stages. Patch example Bug pattern Plan Generate checker Bounded repairs Patch validation Stop Fixed stages limit interaction. (c) System-specific agent scaffold The same LLM uses different scaffolds. Same LLM Shell-only Lower Tool-assisted Higher Search View Edit The scaffold changes the score. (d) Fragmented correctness signals Patch tests omit checker-specific signals. Issue & repo Git patch Human test suite Resolved / unresolved Reported: patch-level success Missing: checker rerun, localization, false-positive evidence Patch success cannot prove checker correctness. Figure 2 Four recurring evaluation gaps: (a) Not checker-centric [32]; (b) Limited interaction horizon [67]; (c) System- specific agent scaffold [68]; and (d) Fragmented correctness signals [53]. 1 Introduction Static analysis provides a scalable approach for detecting complex program defects without executing the target programs [11, 28, 36, 59, 67]. However, existing static analyzers typically rely on manually designed checkers, which require substantial expert effort to encode defect patterns and analysis strategies into tool- specific rules [24, 59, 67]. Recent advances in large language models (LLMs) have opened new opportunities for automated code understanding and defect detection. Nevertheless, directly applying LLMs to code snippets remains challenging due to high inference costs and limited access to repository-wide context, which can lead to incomplete reasoning and hallucinated findings [27, 46]. To overcome these limitations, recent coding agents extend LLMs with the ability to interact with software repositories and external tools [9, 10], enabling iterative code modification, execution, and feedback-driven refinement. These capabilities provide a promising foundation for automating static-analysis checker devel- opment, which traditionally requires substantial manual engineering effort. Given a defect specification, an agent can navigate analyzer infrastructures, implement checker logic, integrate the checker into the build sys- tem, and iteratively refine its behavior based on compiler and analyzer feedback [32, 52, 58, 62, 66]. However, synthesizing effective static-analysis checkers remains a challenging long-horizon task: agents must under- stand complex analyzer frameworks, make coordinated modifications across multiple files, and ensure that the resulting checker compiles, detects target defects, and maintains low false-positive rates [11, 36, 65]. We refer to this task as long-horizon agentic static-analysis checker synthesis. However, as illustrated in Figure 2, existing benchmarks are insufficient for evaluating long-horizon agentic static-analysis checker synthesis. First, current coding and vulnerability benchmarks are not checker-centric: repository-level benchmarks typically evaluate code completion, patch generation, or feature implementation, while vulnerability benchmarks focus on detection, classification, or repair rather than reusable checker synthesis [32, 39, 61, 64, 69, 73]. Second, existing benchmarks provide limited task interaction. They either evaluate isolated tasks or decompose workflows into predefined stages, preventing agents from performing the iterative planning, implementation, and debugging required for checker development [24, 35, 67]. Third, benchmarks differ substantially in environments, tool interfaces, and evaluation protocols, making results 2 “Let agents turn past flaws into reusable checks, so yesterday’s failures can guard tomorrow’s code.” Table 1 Systems and benchmarks for agentic static-analysis checker synthesis.✓ denotes full support,✓partial support, and✗ no support. Appendix A defines the operational criteria. System / Benchmark Repository Executable Multi-Harness Long-Horizon Checker Semantic False-Positive Context EnvironmentSupportInteraction Synthesis Verification Evaluation RhoSynth [24]✗✓ KNighter [67]✓✗✓ SWE-bench [32]✓✗✓✗✓✗ SWE-Gym [53]✓✗✓✗ ✓✗ R2E-Gym [31]✓✗✓✗✓✗ ReposVul [61]✓✗✓✗ VulEval [64]✓✗✓✗ JitVul [69]✓✗✓✗✓✗ CheckerBench✓ difficult to compare across systems [62, 66, 68]. Finally, current evaluation metrics mainly measure patch- level correctness or test-suite outcomes, but do not capture checker-specific properties such as vulnerability coverage, false-positive control, and analyzer compatibility [32, 34, 63]. These limitations motivate a unified benchmark that evaluates agents’ ability to autonomously synthesize reliable and reusable static-analysis checkers. To address these gaps, we introduce CheckerBench, the first benchmark for evaluating long-horizon static- analysis checker synthesis. CheckerBench contains 300 real-world tasks collected from 297 CVEs, 167 repositories, 85 CWEs, and five programming-language ecosystems, covering diverse analyzer environments and defect patterns. Each task provides a reproducible repository, an analyzer-specific environment, and reusable skills, enabling agents to perform iterative development and validation rather than isolated code generation. We further develop CheckerLab, a unified evaluation framework that assesses coding-agent systems under a common task specification and scoring protocol. CheckerLab performs independent re- peated generation and frozen-candidate verification: each synthesized checker is rebuilt, executed on vulnera- ble and fixed revisions, and evaluated based on diagnostic reduction, patch localization, residual reports, and computational cost. Across 21 model–harness configurations, the average Pass@1 score is only 32.30%, with the best-performing configuration reaching 45.33% (Figure 1). These results demonstrate that synthesizing reliable and reusable static-analysis checkers remains a challenging problem for current coding agents. In summary, our contributions are threefold. • To the best of our knowledge, we are the first to build a comprehensive executable benchmark for evaluating repository-level, long-horizon static-analysis checker synthesis by coding agents. CheckerBench contains 300 real-world tasks spanning multiple programming-language ecosystems and analyzer environments, enabling systematic evaluation of agentic checker development. • We develop a scalable benchmark construction pipeline that automatically reconstructs analyzer environments, pack- ages synthesis tasks, and validates checker behavior. This pipeline enables reproducible task creation beyond manually curated checker examples. • We present CHECKERLAB, a unified evaluation framework for coding-agent systems with a shared task specification and scoring protocol. CheckerLab supports independent repeated generation and frozen-candidate verifica- tion, providing fine-grained measurements of checker correctness, patch localization, false-positive control, and computational cost. 2 Related Work 2.1 Static Analysis and Checker Synthesis Static analyzers rely on expert-authored checkers that encode defect patterns and analyzer-specific logic [24, 67]. RhoSynth learns graph-based code-quality rules from examples, while Gordon explores program-specific analyses from behavioral examples [24, 25]. LLM-based systems execute analysis pseudocode or filter bug reports [27, 46]. IRIS infers taint specifications for repository-wide analysis [36]; MoCQ and CQLLM generate vulnerability patterns or CodeQL queries [35, 60]. QLCoder synthesizes CodeQL queries through tool use and iterative feedback [59]. KNighter synthesizes and refines Clang Static Analyzer checkers from Linux kernel patches [67]. These systems target specific analyzers or domains, motivating a common evaluation of 3 300tasks300verified checkers300pilot trajectories 167repositories5 languages297CVEs85CWEs (a) Construction pipeline 1 Vulnerability source collection Link CVEs to fixes Recover both revisions Vulnerable and fixed pair 2 Executable environment reconstruction Pin tools and analyzers Reproduce both builds Executable revision pair 3 Checker-synthesis task construction Bundle patch and code Add build/scan steps Agent-visible task package 4 Independent verifier construction Rebuild in isolation Check report changes Frozen held-out verifier 5 Human-guided rollout collection Review failed attempts Revise and rerun First pass or exclude after 3 6 Quality control and split construction Validate artifacts Deduplicate tasks Verified checkers and traces (b) Real task: FFmpegPNG buffer overflow C language Clang Static Analyzer (CSA) CVE-2017-7866pngdec.c/ decode_zbuf() Vulnerable revision Stages 1–3 av_bprint_get_buffer(bp, 1, &buf, &buf_size); if(!buf_size) ret = AVERROR(ENOMEM); gotofail; zstream.next_out= buf; zstream.avail_out= buf_size; ret = inflate(&zstream, Z_PARTIAL_FLUSH); // ... bp->str[bp->len] = 0; N-byte buffer: full capacity may be used 0 When full, the NUL write is out of bounds. Fixed revision Stages 1–3 av_bprint_get_buffer(bp, 2, &buf, &buf_size); if(buf_size< 2) ret = AVERROR(ENOMEM); gotofail; zstream.next_out = buf; zstream.avail_out= buf_size-1; ret = inflate(&zstream, Z_PARTIAL_FLUSH); // ... bp->str[bp->len] = 0; N-byte buffer: one byte kept for NUL 0 Output ≤ N − 1; the terminator fits. Checker and verification results Stages 4–6 Generated CSA checker (excerpts) State = State->add<BprintCapSet>(CapSym); // ... if(!State->get<BprintCapSet>() .contains(VSym)) return; // ... C.emitReport(std::move(R)); CapSym: capacity; VSym: avail_out at inflate() Archived verifier findings Vulnerable 1 Fixed 0 Archived pass 41 interaction steps Mainline scan: 0 reports / 954 files Figure 3 CheckerBench construction. general-purpose agents developing and validating checkers across analyzer environments. 2.2 Coding Agents for Software Engineering Coding agents combine LLMs with repository navigation, code editing, tool execution, and feedback. SWE- agent and OpenHands support interactive development, while RepoGraph adds repository structure [52, 62, 68]. RepoAudit applies an agent to repository-level bug detection [26]. These systems produce patches or bug findings. Checker synthesis requires analyzer integration and validation through compiler and scan feedback on vulnerable and fixed revisions. 2.3 Benchmarks for Repository-Level Code Agents Repository-level benchmarks cover issue resolution (SWE-bench and SWE-bench Pro), feature development (FeatureBench), and terminal tasks (Terminal-Bench) [19, 32, 43, 73]. SWE-Gym and R2E-Gym provide executable environments and verifiers for issue-resolution tasks [31, 53]. ReposVul, VulEval, and JitVul target vulnerability detection; SEC-bench evaluates proof-of-concept generation and patching [34, 61, 64, 69]. These works score task completion, patches, or vulnerability detection. CheckerBench evaluates reusable checker construction with held-out verification of builds, vulnerable–fixed diagnostics, patch localization, and false positives (Table 1). 3 The CHECKERBENCH Benchmark Problem formulation. Each agent-facing task x i = (p i ,ℓ i ,α i ,E i ,Q i ) combines a vulnerability-fix instance p i = (r i ,c − i ,c + i , ∆ i ) (repository, vulnerable/fixed revisions, and patch) with its language, analyzer, repro- ducible version-pinned environment, and synthesis instruction. All runs share the initial skill library K; agents may edit checker artifacts but must preserve both revisions. Under an active generation time budget B, policy π θ in harness h yields trajectory τ (h,θ) i = (o 0 ,a 0 ,...,a N i −1 ,o N i ) and candidate checker q (h,θ) i . Here N i counts interactions, a k is an action, and o k+1 its resulting observation [8]. A released record is d (h,θ) i = (E i ,Q i ,V i ,τ (h,θ) i ,q (h,θ) i ), with executable veri- fier V i hidden during synthesis. The environment, instruction, and verifier are fixed across model–harness configurations; trajectories and checkers vary by rollout. 4 (a) Programming languages 159 (53.0%) 51 (17.0%) 30 (10%) 30 (10%) 30 (10%) C/C++ Java Python JavaScript Go (b) Repositories 107 (35.7%) 4 (1.3%) 4 (1.3%) 4 (1.3%) 4 (1.3%) 177 (59.0%) Linux FFmpeg ImageMagick JerryScript QEMU Others (c) CWE distribution 227 (72.3%) 26 (8.3%) 21 (6.7%) 15 (4.8%) 13 (4.1%) 12 (3.8%) CWE-787: Out-of-bounds write CWE-125: Out-of-bounds read CWE-22: Path traversal CWE-476: NULL pointer dereference CWE-79: Cross-site scripting Others Figure 4 CheckerBench distributions by (a) language, (b) repository, and (c) CWE. 3.1 Benchmark Construction Figure 3 summarizes the six-stage pipeline: recover executable vulnerability-fix pairs, package synthesis tasks, freeze independent verifiers, and curate verified checker–trajectory pairs. All candidates start from the same harness-component baseline and shared skill library. Stage I: Vulnerability Source Collection. We link public CVE records from the National Vulnerability Database 1 to fixing commits in open-source C/C++, Java, Python, JavaScript, and Go repositories. Each candidate must have a retrievable vulnerable–fixed revision pair and a patch that exposes a reusable static- analysis property. We exclude unresolved provenance and incompatible redistribution terms, and retain CVE/CWE mappings for auditing and stratified sampling. Stage I: Executable Environment Reconstruction. We reconstruct both revisions in isolated workspaces with pinned toolchains and analyzers: Clang Static Analyzer (CSA) for C/C++ and CodeQL for the other four languages. Revision-compatible build snapshots and compilation databases recover the analyzable scope of CSA tasks; CodeQL tasks use pinned analysis environments. Each pair includes affected functions and reproducible checker compilation and scan commands. We reject pairs that cannot be reconstructed or do not support stable differential analysis. Stage I: Checker-Synthesis Task Construction. Each task exposes the patch, pre- and post-fix functions, build metadata, compile/scan commands, and a minimal checker scaffold. Random workspace identifiers and omitted CVE/CWE labels reduce provenance shortcuts. The shared library K supplies harness-specific workflow skills and read-only analyzer documentation, example checker implementations, and semantic util- ities; it excludes task-specific checker solutions. The agent must infer the defect mechanism, implement a reusable checker, and refine it through compilation and analysis feedback, producing checker artifacts and an interaction trace. Stage IV: Independent Verifier Construction. We freeze a held-out verifier before rollout collection. It rebuilds each submitted checker from source in a clean workspace and checks vulnerable–fixed diagnostic contrast and patch relevance under backend-specific rules. CSA requires a disappearing report in a patch- modified function. Passing additionally requires process and semantic quality, mainline false-positive control, and mechanism and anti-hardcoding checks (Appendix F). Stage V: Human-Guided Rollout Collection and Sampling. Calibration uses fixed model, harness, tool, and budget settings. After a failed attempt, a human diagnoses the verification evidence and revises prompt guidance for a fresh rollout; the task, verifier, and thresholds stay fixed. Each task permits one initial attempt and at most two rerolls, stopping at the first pass and excluding tasks with no pass after three attempts. We retain all attempts and interventions for audit. Checker–trace pairs enter sampling only if the checker passes the frozen verifier and the trace contains at least 40 interaction steps (agent decisions). Eligible pairs are sampled jointly over repository, CWE, and horizon. The 40-step threshold is a curation rule; these pilot interventions are separate from autonomous main evaluation. 1 https://nvd.nist.gov/vuln 5 (a) CheckerLab Visible during synthesis Task package 50-minute budget PatchVersioned codeAnalyzerSkills Coding agent HarnessesClaude CodeOpenCodeHermes Agent Models MemoryPlannerToolsContext InspectEditBuildDiagnose Freeze checker Long-Horizontrace Post-submission only Independent verifierJudge model BuildContrastPatch relevanceScope Mainline FP scan Review up to 10 scan alerts GateProcessSemanticFP rate CSA ≥80/100 ≥7/10 ≤25% CodeQL≥8/10 Compiler &Analyzer feedback (b) Archived pilot trajectory TorchServeJava / CodeQLCVE-2023-48299 Task: Detect Zip Slip during archive extraction Flag archive-entry paths that escape the extraction directory. Agent actionObserved feedback 1 Inspect Read patchandtrace archive names to Files.newOutputStream(...). Patch evidence Fix adds canonical paths and a containment check before write 2 Implement &run Reuse ZipSlipFlow and implement and run vulnerability.ql. Initial validation Compile: OK; Vulnerable: 1 Fixed: 0; ZIP taint path found 3 Diagnose coverage Probe missing TAR flows and inspect extracted calls and types. Scope limitation Commons types unresolved; only JDK ZIP flow is observable 4 Test the mechanism Remove the sanitizer barrier in a control query and scan fixed code. Scratch control (barrier off) Fixed: 1 in scratch query only Final query keeps 1 → 0 contrast 5 Finalize Keep query and ZIP-only scope, recompile and rescan both revisions. Stable final artifact Compile: OK; Vulnerable: 1 Fixed: 0; query and trace frozen Archived pilot: 5 selected stages, 56 assistant turns, 71 tool calls Post-submission: Archived verifier passed Figure 5 CheckerLab workflow and five selected stages from an archived TorchServe pilot. Stage VI: Quality Control, Deduplication, and Splits. Automated checks establish artifact integrity, re- producible builds and scans, verifier isolation, and package completeness; manual review confirms defect semantics and patch-localized diagnostics. We remove duplicate revision pairs and keep tasks connected by repository, CVE, shared revisions, or similar patches in one split. Task-specific reference checkers, calibration traces, and verifier details remain hidden from evaluation agents. Appendix C details these checks and split safeguards. 3.2 Benchmark Statistics Projects and tasks. CheckerBench contains 300 tasks from 167 open-source repositories, including Linux [38] (107 tasks), FFmpeg [23] (4), and ImageMagick [29] (4). Tasks cover C/C++ (159), Java (30), Python (51), JavaScript (30), and Go (30). Static analysis environments are CSA (159) and CodeQL (141). Trajectories and interaction horizons. We release 300 accepted calibration trajectories, one per task. Cal- ibration includes human intervention to diagnose failures and revise prompts. All trajectories pass exe- cutable verification and the construction-time semantic-score threshold. They are separate from independent main-experiment repeats. Interaction steps reflect workload under the calibration configuration [57]. Appen- dices D.4 and D.5 report trajectory statistics, including the assistant-turn and tool-call distributions of the admitted pilot trajectories (Figure 11). Defect and task coverage. CheckerBench covers 85 CWEs. The most frequent are CWE-787 (out-of- bounds write), CWE-125 (out-of-bounds read), and CWE-22 (path traversal), as shown in Figure 4(c). Each task also records repository scale, integration scope, verification evidence, and calibration horizon. Appendix D provides complete benchmark statistics. 4 CHECKERLAB: Evaluation Framework CheckerLab separates interactive checker synthesis from independent evaluation of the frozen checker and its trajectory [41] (Figure 5). 6 Table 2 Performance of seven models across three agent harnesses on CheckerBench. Bold and underlinedvalues mark the best and second-best models per metric within each harness. Model-row subscripts show numerical changes relative to GPT-5.6 in the same harness: ↑ increases and ↓ decreases (percentage points for percentage metrics). Header arrows indicate the preferred direction. Model-row metrics are means over three independent repeats. Overall Average summarizes the 21 model–harness configurations. Model / System Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FP pass (%) ↓ #WN Process ↑ Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ OpenCode GPT-5.631.6798.3385.330.00780.227.6829.1960.13 Claude Opus 4.845.33 ↑13.66 81.67 ↓16.66 81.00 ↓4.33 13.79 ↑13.79 17486.42 ↑6.20 8.32 ↑0.64 44.78 ↑15.59 64.35 ↑4.22 Qwen3.8-Max37.33 ↑5.66 69.33 ↓29.00 69.00 ↓16.33 5.77 ↑5.77 52 90.15 ↑9.93 8.46 ↑0.78 35.96 ↑6.77 47.54 ↓12.59 DeepSeek-V4-Pro39.67 ↑8.00 92.67 ↓5.66 90.67 ↑5.34 2.56 ↑2.56 7884.89 ↑4.67 8.12 ↑0.44 51.09 ↑21.90 65.28 ↑5.15 GLM-5.232.33 ↑0.66 77.33 ↓21.00 75.33 ↓10.00 10.34 ↑10.34 58 86.73 ↑6.51 8.11 ↑0.43 41.21 ↑12.02 61.02 ↑0.89 Kimi-K338.67 ↑7.00 83.33 ↓15.00 81.67 ↓3.66 1.59 ↑1.59 6388.48 ↑8.26 8.30 ↑0.62 32.42 ↑3.23 43.53 ↓16.60 HY4-Preview33.67 ↑2.00 72.67 ↓25.66 71.33 ↓14.00 12.38 ↑12.38 105 86.90 ↑6.68 8.32 ↑0.64 41.28 ↑12.09 50.15 ↓9.98 Average36.9582.1979.196.6376.7186.268.1939.4256.00 Claude Code GPT-5.634.0093.6783.002.4441 81.417.6935.7160.58 Claude Opus 4.835.00 ↑1.00 63.67 ↓30.00 62.33 ↓20.67 2.33 ↓0.11 4386.88 ↑5.47 8.21 ↑0.52 32.60 ↓3.11 45.63 ↓14.95 Qwen3.8-Max34.67 ↑0.67 67.33 ↓26.34 67.00 ↓16.00 4.55 ↑2.11 44 89.58 ↑8.17 8.38 ↑0.69 33.91 ↓1.80 41.57 ↓19.01 DeepSeek-V4-Pro38.33 ↑4.33 91.33 ↓2.34 89.67 ↑6.67 1.72 ↓0.72 5885.25 ↑3.84 8.27 ↑0.58 47.17 ↑11.46 62.50 ↑1.92 GLM-5.233.00 ↓1.00 76.33 ↓17.34 74.67 ↓8.33 0.00 ↓2.44 18 89.67 ↑8.26 8.39 ↑0.70 38.75 ↑3.04 49.93 ↓10.65 Kimi-K338.00 ↑4.00 83.67 ↓10.00 82.33 ↓0.67 2.74 ↑0.30 7387.79 ↑6.38 8.34 ↑0.65 31.81 ↓3.90 38.61 ↓21.97 HY4-Preview23.67 ↓10.33 49.33 ↓44.34 47.33 ↓35.67 0.00 ↓2.44 13 90.83 ↑9.42 8.35 ↑0.66 50.66 ↑14.95 62.23 ↑1.65 Average33.8175.0572.331.9741.4387.348.2338.6651.58 Hermes Agent GPT-5.623.00100.0064.000.00679.857.5025.9446.91 Claude Opus 4.834.33 ↑11.33 80.33 ↓19.67 78.67 ↑14.67 5.45 ↑5.45 5585.58 ↑5.73 8.32 ↑0.82 44.84 ↑18.90 61.89 ↑14.98 Qwen3.8-Max25.00 ↑2.00 44.00 ↓56.00 42.33 ↓21.67 11.11 ↑11.11 27 90.92 ↑11.07 8.53 ↑1.03 40.55 ↑14.61 51.95 ↑5.04 DeepSeek-V4-Pro32.33 ↑9.33 94.00 ↓6.00 90.67 ↑26.67 0.00 =0.00 1285.29 ↑5.44 8.03 ↑0.53 39.82 ↑13.88 62.37 ↑15.46 GLM-5.222.33 ↓0.67 79.67 ↓20.33 78.00 ↑14.00 0.00 =0.00 382.87 ↑3.02 7.81 ↑0.31 42.85 ↑16.91 63.22 ↑16.31 Kimi-K324.33 ↑1.33 87.00 ↓13.00 79.67 ↑15.67 4.76 ↑4.76 6382.62 ↑2.77 8.08 ↑0.58 32.73 ↑6.79 42.12 ↓4.79 HY4-Preview21.67 ↓1.33 57.67 ↓42.33 52.67 ↓11.33 0.00 =0.00 388.42 ↑8.57 8.44 ↑0.94 49.69 ↑23.75 62.06 ↑15.15 Average26.1477.5269.433.0524.1485.088.1039.4955.79 Overall Average32.3078.2573.653.8847.4386.238.1739.1954.46 Evaluation workflow. Each run receives an isolated workspace with the task instruction, fixing patch, code context, analyzer tools, and enabled skills; the reference checker and held-out verifier remain hidden. Agents use native harness capabilities and compilation and analysis feedback within a common 50-minute genera- tion budget. At termination, we freeze the submitted CSA checker or CodeQL query source and record the trajectory and tool invocations. The verifier then rebuilds the frozen checker from source in a clean envi- ronment and checks vulnerable–fixed diagnostic contrast and patch relevance under backend-specific rules (Section 3.1). A broader mainline scan assesses false positives without checker edits. Claude Opus 5 [7] judges process quality, vulnerability semantics, and sampled mainline alerts in separate stages. A run passes only after normal generation completion and satisfaction of all gates in Equation 5; Appendix F gives the full rubrics. Metrics. Our primary metric, Pass@1, is the percentage of tasks whose checker passes the full evaluation after one independent synthesis run. For each of the 21 harness–model pairs, we evaluate the same 300 tasks in three independent repeats and average the three rates equally. Generation failures, refusals, and candidate- code errors count as failures. Build and Diff. SR are the percentages of all tasks passing independent compilation and differential verification, respectively. Process and Semantic are mean valid scores on 100- point and 10-point scales. Within each repeat, the false-positive percentage FP pass = 100 P FP/ P (TP+FP) pools fully adjudicated recorded mainline alerts from passed checkers. The acceptance gate uses each checker’s sampled mainline alerts, with a default FP cutoff of 25% (Appendix F). We compute each metric per repeat and average equally (Appendix H). In Table 2, #WN is the arithmetic mean of the counts of fully adjudicated recorded mainline warnings from passed checkers across the three repeats. Each repeat’s count is P (TP+FP). 7 C/C++ Python Go Java JavaScript 15.155.637.841.142.2 23.365.451.150.046.7 11.362.748.954.453.3 24.556.953.342.245.6 14.058.846.741.130.0 20.857.537.847.843.3 7.351.044.447.845.6 (a) Language CWE-787CWE-125 CWE-22 CWE-476 CWE-79 CWE-362CWE-416CWE-400 17.919.046.717.930.613.925.040.0 24.415.957.828.241.716.727.840.0 21.822.251.17.761.116.711.136.7 28.222.248.925.638.913.933.336.7 16.717.546.712.836.113.925.040.0 20.527.048.923.127.813.922.246.7 2.620.631.15.152.88.38.346.7 (b) CWE 203550 Avg. Turns (passed only) 10 20 30 40 50 Pass@1 (%) (c) Turns GPT Claude Qwen DeepSeek GLM Kimi HY4 GPT Claude Qwen DeepSeek GLM Kimi HY4 GPT Claude Qwen DeepSeek GLM Kimi HY4 050100 Pass@1 (%) (a) (b) GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview OpenCode Claude Code Hermes Agent Model GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview Figure 6 Pass@1 by (a) language and (b) CWE, averaged equally across three harnesses. (c) Pass@1 versus average turns on successful runs; colors denote harnesses and shapes denote models. The Average and Overall Average rows report mean counts across seven models and all 21 configurations, respectively. Warning counts are descriptive, with no ranking or change subscripts. Appendix E.6 details alert selection, recorded warning volume, and sensitivity to precision weighting and all-recorded-alert gates. For interaction efficiency, we average turns, tool calls, and explicit skill calls over successful trajectories within each repeat, including subagent activity. We then average these means equally across repeats and report success counts (Appendix E). 5 Experiments 5.1 Experimental Setup Models and agent harnesses. We evaluate 7 models using 3 agent harnesses, including OpenCode [3], Claude Code [5], and Hermes Agent [48]. The models are GPT-5.6-Sol [49], Claude Opus 4.8 [6], Qwen3.8-Max [2], DeepSeek-V4-Pro-0813 [18], GLM-5.2 [70], Kimi-K3 [33], and HY4-Preview [56]. Task execution. Each CheckerLab run uses an isolated E2B sandbox and workspace containing the task instruction, fixing patch, relevant code context, and analyzer tools. Agents can inspect code, implement checkers, and use compilation and analysis feedback within a 50-minute active generation budget. The submitted source and execution records are frozen at termination for independent verification. Environment and execution details are provided in Appendix E. Evaluation and judge. Each frozen checker is evaluated using backend-specific executable checks and scoring rubrics. We use Claude Opus 5 [7] as the LLM judge for false-positive triage, process quality, and vulnerability semantics. Human validation gives a pooled Cohen’s κ = 0.85 for judge–human agreement. Appendix E.9 reports the sampling, labeling, adjudication, and stage-specific agreement. The scoring rubrics are detailed in Appendix F. For the 21 model–harness configurations, we report mean Pass@1 over three independent repeats on the same 300 tasks, with additional diagnostic metrics defined in Section 4. Efficiency metrics are averaged over successful trajectories only. 5.2 Overall Performance End-to-end success. Across 21 model–harness configurations, mean Pass@1 is 32.30%; even the best com- bination, Claude Opus 4.8 with OpenCode, reaches only 45.33% (Figure 1; Table 2). Thus, more than half of tasks remain unsuccessful even for the strongest configuration. Overall Build and Diff. SR are much higher, at 78.25% and 73.65%, respectively. The gap between Diff. SR and Pass@1 is 41.35 percentage points, much larger than the 4.60-point gap between Build and Diff. SR. For example, GPT-5.6 with Hermes Agent achieves 100.00% Build but only 64.00% Diff. SR and 23.00% Pass@1. The larger gap between differential 8 verification and full acceptance highlights the challenge of meeting the additional requirements for semantic fidelity, process quality, false-positive control, and normal completion. Model–harness dependence. OpenCode has the highest mean Pass@1 across models (36.95%), followed by Claude Code (33.81%) and Hermes Agent (26.14%), with a 10.81-percentage-point difference between the highest and lowest averages. The best harness varies by model: GPT-5.6 and GLM-5.2 attain their highest Pass@1 with Claude Code. The leading model also changes across harnesses. Claude Opus 4.8 ranks first with OpenCode and Hermes Agent, whereas DeepSeek-V4-Pro leads with Claude Code (38.33%, compared with 35.00% for Claude Opus 4.8). Kimi-K3 further illustrates the sensitivity to harness choice, declining from 38.67% with OpenCode to 24.33% with Hermes Agent, a 14.34-point gap. These shifts show that harness choice changes both success rates and model rankings, making the complete model–harness configuration the appropriate unit for comparison and selection. Diagnostic quality and false positives. Qwen3.8-Max with Hermes Agent has the highest Process (90.92/100) and Semantic (8.53/10) scores across all configurations, yet reaches only 25.00% Pass@1 and 44.00% Build. OpenCode’s higher mean Pass@1 is also accompanied by a higher FP pass than Claude Code (6.63% vs. 1.97%). Within OpenCode, Claude Opus 4.8 has 13.79% FP pass with #WN= 174, compared with 0.00% and #WN= 7 for GPT-5.6. Both metrics summarize recorded, fully adjudicated alerts from passed checkers. No evaluated configuration leads simultaneously in synthesis success, process and semantic quality, and observed diagnostic precision. Checker reliability should therefore be assessed across these dimensions, with warning volume providing context for false-positive comparisons. Language and defect-family variation. Figure 6(a,b) averages each model’s results equally across the three harnesses. Every model has its lowest language-specific Pass@1 in C/C++, where even the leading model, DeepSeek-V4-Pro, reaches only 24.53%. This stratum contains 159 of the 300 tasks and therefore strongly influences aggregate success. The leaders change in other languages: Claude Opus 4.8 leads Python (65.36%), DeepSeek-V4-Pro leads Go (53.33%), and Qwen3.8-Max leads Java (54.44%) and JavaScript (53.33%). Qwen3.8-Max also outperforms DeepSeek-V4-Pro on CWE-79, with the ordering reversed on CWE-787. C/C++ uses CSA while the other languages use CodeQL, and CWE groups differ in repository and language composition. These results identify C/C++ tasks under CSA as a shared challenge and reveal complemen- tary model strengths across the evaluated task groups, supporting performance breakdowns by language and defect family alongside aggregate Pass@1. Success and interaction efficiency. Kimi-K3 uses the fewest average tool calls on successful runs in all three harnesses. With Claude Code, it achieves nearly the same Pass@1 as DeepSeek-V4-Pro (38.00% vs. 38.33%), while using fewer turns (31.81 vs. 47.17) and tool calls (38.61 vs. 62.50), a 38.2% reduction in tool calls (Figure 6(c)). Kimi-K3 thus combines a competitive success rate with efficient tool use on successful runs, highlighting the value of considering both Pass@1 and interaction efficiency when selecting a model–harness configuration. 5.3 Comparison with Baselines Table 3 Checker validity on KNighter’s 61 Linux-kernel tasks. MethodValid Rate Ours49/61 80.3% KNighter41/6167.2% Ours w/o skills18/61 29.5% We compare our skill-guided checker-synthesis method, imple- mented with OpenCode, against KNighter on the 61 Linux- kernel tasks from KNighter’s original benchmark. All meth- ods use DeepSeek-V4-Pro. Our method achieves an 80.3% valid-checker rate, compared with 67.2% for KNighter (Ta- ble 3). Removing the skill library lowers our method’s rate to 29.5%. KNighter organizes synthesis through stage-specific prompts and a checker template, while our method integrates the shared reusable guidance as skills. Its lead on KNighter’s original tasks suggests that this skill-guided workflow transfers well to an independently defined task set. Details in Appendix H.1. 9 050100150200 0% 20% 40% median = 58 GPT-5.6/OpenCode 050100150200 0% 20% 40% median = 53 GPT-5.6/Claude Code 050100150200 0% 20% 40% median = 46 GPT-5.6/ Hermes Agent 050100150200 0% 20% 40% median = 64.5 Claude Opus 4.8/OpenCode 050100150200 0% 20% 40% median = 41 Claude Opus 4.8/Claude Code 050100150200 0% 20% 40% median = 65 Claude Opus 4.8/ Hermes Agent 050100150200 0% 20% 40% median = 45 Qwen3.8-Max/OpenCode 050100150200 0% 20% 40% median = 37.5 Qwen3.8-Max/Claude Code 050100150200 0% 20% 40% median = 49 Qwen3.8-Max/ Hermes Agent 050100150200 0% 20% 40% median = 59 DeepSeek-V4-Pro/OpenCode 050100150200 0% 20% 40% median = 58 DeepSeek-V4-Pro/Claude Code 050100150200 0% 20% 40% median = 60 DeepSeek-V4-Pro/ Hermes Agent 050100150200 0% 20% 40% median = 58 GLM-5.2/OpenCode 050100150200 0% 20% 40% median = 49 GLM-5.2/Claude Code 050100150200 0% 20% 40% median = 61 GLM-5.2/ Hermes Agent 050100150200 0% 20% 40% median = 41.5 Kimi-K3/OpenCode 050100150200 0% 20% 40% median = 38 Kimi-K3/Claude Code 050100150200 0% 20% 40% median = 37 Kimi-K3/ Hermes Agent 050100150200 0% 20% 40% median = 49 HY4-Preview/OpenCode 050100150200 0% 20% 40% median = 59 HY4-Preview/Claude Code 050100150200 0% 20% 40% median = 59 HY4-Preview/ Hermes Agent Share of trajectories Tool calls per trajectory Figure 7 Tool-call distributions over successful trajectories for the 21 model–harness combinations. Histogram bins have width 10; dashed lines indicate medians. 1020304050 Active time budget (min) 0 10 20 30 40 50 Pass@1 (%) OpenCode 1020304050 Active time budget (min) Claude Code 1020304050 Active time budget (min) Hermes Agent GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview Figure 8 Retrospective Pass@1 versus active generation time for three agent harnesses. 5.4 Ablation Study We compare DeepSeek-V4-Pro on the same 300 tasks, using its original OpenCode run as Full. As sum- marized by Table 4, Pass@1 is 39.67% for Full, 8.33% without skills and strategy references, 13.33% with execution feedback hidden, and 0.33% for one-shot generation without tools or a harness. These gaps persist after excluding Process and normal-termination requirements: artifact pass is 48.67%, 18.67%, 20.33%, and 1.33%, respectively. One-shot performance also reflects invalid or incomplete submissions in 237/300 tasks. Differences between historical and new run conditions prevent attributing the gaps solely to the ablated components. See details in Appendix H.2. 5.5 Time-Budget Sensitivity We assess time-budget sensitivity by retrospectively applying 10, 20, 30, 40, and 50-minute active generation cutoffs to the existing trajectories across all 21 model–harness configurations. Only final artifacts completed by each cutoff contribute, with all 300 tasks retained per configuration. Figure 8 shows that aggregate Pass@1 rises from 7.24% at 10 minutes to 26.05% at 30 minutes and 32.30% at 50 minutes, with diminishing gains: the 40-minute cutoff captures 93.02% of the successes observed by 50 minutes, and the final 10 minutes add 2.25 percentage points. These late gains are largest for Qwen3.8-Max and HY4-Preview. The curves 10 characterize completion timing under the original 50-minute protocol; performance under separately imposed shorter deadlines remains untested. Full metrics and timing rules appear in Appendix H.3. 5.6 FP-Threshold Sensitivity To quantify the influence of the FP threshold on checker acceptance, we evaluate thresholds of 10%, 20%, 25%, 30%, 40%, and 50% across all 21 model–harness configurations (Figure 9). For checkers with adjudicated alerts, the additional acceptance criterion requires 100 FP/(TP + FP) to be no greater than the specified threshold. Adjudication labels and all other eligibility criteria are held fixed, with 300 tasks retained in the denominator for each configuration. The 25% threshold reproduces the acceptance rates for all 21 configurations in Table 2. The aggregate acceptance rate increases from 32.05% at 10% to 32.30% at 25% and 32.83% at 50%, yielding a net increase of 0.78 percentage points over the evaluated range. Over the same range, the largest increase for an individual configuration is 1.67 percentage points, observed for GLM-5.2 with Claude Code. These results suggest that aggregate acceptance is relatively stable across the evaluated FP thresholds within the recorded cohort. 5.7 Analysis of Checker Synthesis Failures We analyze 4,360 selected failed runs across seven models and three harnesses using tool traces, independent verification logs, and existing judge records. The leading failure categories are budget exhaustion (25.32%), inadequate synthesis process (22.96%), and inadequate semantic modeling (21.15%). These records expose gaps between analysis plans and implementations: for example, a planned path-sensitive analysis becomes AST matching that omits locking and control-flow conditions. Excess false positives account for 15.96% of failures, underscoring the need to evaluate precision beyond the original vulnerable–fixed pair. The remaining failures involve differential detection or localization (8.17%), patch-specific overfitting or anchoring (3.56%), missing or invalid artifacts (1.83%), and compilation or registration (1.05%). Key challenges for checker synthesis include timely completion, faithful implementation of analysis plans, and precise semantic modeling. 5.8 Data Contamination and Prior Exposure CVE publication postdates release for 3/300 tasks with Claude Opus 4.8, one with GLM-5.2, and none with other models (Figure 10(a,b)); code or patches may have appeared earlier. On 100 language-stratified tasks, we compare original inputs (O), paraphrases retaining identifiers (P), and versions of P with meaning- preserving identifier substitutions (D) [21]. Figure 10(c) shows P–D Pass@1 gaps of +3.00 and +2.00 per- centage points for Claude Opus 4.8 and DeepSeek-V4-Pro. Exact CVE recall falls from 10.00% to 1.00% and from 8.00% to 1.00%, respectively. The 7–9-point recall decline accompanies only a 2–3-point Pass@1 drop, suggesting that recognizing familiar vulnerabilities yields only limited gains under this intervention (Appendix C.3). 6 Conclusion We introduced CheckerBench, an executable benchmark for long-horizon static-analysis checker synthesis with 300 tasks across five programming-language ecosystems. CheckerLab independently rebuilds sub- Table 4 Performance comparison of DeepSeek-V4-Pro ablations on 300 CheckerBench tasks. Bold and underlined values mark the best and second-best configurations per metric. Subscripts show numerical changes relative to Full: ↑ increases and↓ decreases (percentage points for percentage metrics). Header arrows indicate the preferred direction. N/A denotes no alerts. Configuration Skill lib. Exec. feedback Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FP pass (%) ↓ Process ↑ Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ Full ✓39.6792.6790.672.5684.898.1257.6372.73 w/o Skills ✗✓8.33 ↓31.34 79.33 ↓13.34 52.67 ↓38.00 6.25 ↑3.69 65.31 ↓19.58 7.39 ↓0.73 59.56 ↑1.93 80.53 ↑7.80 w/o Execution Feedback ✓✗13.33 ↓26.34 67.33 ↓25.34 41.67 ↓49.00 25.00 ↑22.44 79.93 ↓4.96 8.00 ↓0.12 53.58 ↓4.05 72.16 ↓0.57 w/o Harness ✗0.33 ↓39.34 3.33 ↓89.34 3.33 ↓87.34 N/A73.64 ↓11.25 7.45 ↓0.67 1.00 ↓56.63 0.00 ↓72.73 11 102025304050 31.0% 32.0% 33.0% at 25% = 31.67% GPT-5.6/OpenCode 102025304050 33.0% 34.0% 35.0% at 25% = 34.00% GPT-5.6/Claude Code 102025304050 22.5% 23.0% 23.5% 24.0% at 25% = 23.00% GPT-5.6/ Hermes Agent 102025304050 44.0% 45.0% 46.0% at 25% = 45.33% Claude Opus 4.8/OpenCode 102025304050 34.0% 35.0% 36.0% at 25% = 35.00% Claude Opus 4.8/Claude Code 102025304050 34.0% 34.5% 35.0% 35.5% at 25% = 34.33% Claude Opus 4.8/ Hermes Agent 102025304050 37.0% 38.0% at 25% = 37.33% Qwen3.8-Max/OpenCode 102025304050 34.0% 35.0% 36.0% at 25% = 34.67% Qwen3.8-Max/Claude Code 102025304050 24.0% 25.0% 26.0% at 25% = 25.00% Qwen3.8-Max/ Hermes Agent 102025304050 39.0% 40.0% 41.0% at 25% = 39.67% DeepSeek-V4-Pro/OpenCode 102025304050 38.0% 39.0% 40.0% at 25% = 38.33% DeepSeek-V4-Pro/Claude Code 102025304050 32.0% 33.0% 34.0% at 25% = 32.33% DeepSeek-V4-Pro/ Hermes Agent 102025304050 31.5% 32.0% 32.5% 33.0% at 25% = 32.33% GLM-5.2/OpenCode 102025304050 33.0% 34.0% 35.0% at 25% = 33.00% GLM-5.2/Claude Code 102025304050 22.0% 23.0% 24.0% at 25% = 22.33% GLM-5.2/ Hermes Agent 102025304050 38.0% 39.0% 40.0% at 25% = 38.67% Kimi-K3/OpenCode 102025304050 38.0% 39.0% 40.0% at 25% = 38.00% Kimi-K3/Claude Code 102025304050 23.5% 24.0% 24.5% 25.0% at 25% = 24.33% Kimi-K3/ Hermes Agent 102025304050 33.0% 34.0% 35.0% at 25% = 33.67% HY4-Preview/OpenCode 102025304050 23.0% 24.0% 25.0% at 25% = 23.67% HY4-Preview/Claude Code 102025304050 21.0% 22.0% 23.0% at 25% = 21.67% HY4-Preview/ Hermes Agent Pass@1 (%) FP threshold (%) Panel-specific y-axis ranges. Dashed line: 25% cutoff. Figure 9 Pass@1 versus FP threshold for 21 model–harness pairs (300 fixed tasks each). Translucent bands highlight the curves. Dashed lines mark the 25% cutoff; annotations show Pass@1 at this cutoff. ≤2019 2020202120222023202420252026 0 20 40 60 38 22 2828 37 60 57 30 MayJunJul Aug 11511511511531 Claude Opus 4.8 · 28 May GLM-5.2 · 16 Jun GPT-5.6 · 9 Jul Kimi-K3 · 16 Jul Qwen3.8-Max · 2 Aug DeepSeek-V4-Pro · 13 Aug HY4-Preview · 28 Aug Score (%) 0 20 40 60 80 100 43.0 42.0 39.0 Score (%) 0 20 40 60 80 100 55.0 54.0 52.0 0 4 8 12 10.0 1.0 0 4 8 12 ID Recall (%) 8.0 1.0 (a) CVE publications (b) Model releases Latest CVE: 26 Jun (c) Prior-exposure sensitivity Claude Opus 4.8 Δ (P−D) +3.00 p ID Recall P−D: 9.00 p DeepSeek-V4-Pro Δ (P−D) +2.00 p ID Recall P−D: 7.00 p C/C++ (159) Python (51) Go (30) Java (30) JavaScript (30) Pass@1 (O)Pass@1 (P)Pass@1 (D)ID Recall (P)ID Recall (D) Figure 10 Temporal coverage and prior exposure: (a) CVE publication counts; (b) model release dates; (c) Pass@1 (O/P/D; left axes, 0–100%) and exact CVE recall (P/D; right axes, 0–12%). mitted checkers and assesses diagnostic behavior on vulnerable and fixed revisions, patch localization, and false positives. Across seven models and three agent harnesses, the best configuration achieves 45.33% Pass@1, leaving substantial room for improvement within the evaluated repositories and analyzer configura- tions. CheckerBench provides a reproducible testbed for agents that turn vulnerability specifications into reliable, reusable checkers. References [1] AgentRE-Bench. AgentRE-Bench. Project website, 2026. URL https://w.agentre-bench.ai/. [2] Alibaba Cloud. Qwen3.8-Max: Model studio release log. https://w.alibabacloud.com/help/en/model-studio/ newly-released-models, 2026. Release entry dated August 2, 2026; accessed September 19, 2026. [3] Anomaly Contributors. OpenCode. https://github.com/anomalyco/opencode, 2026. Accessed August 25, 2026. [4] Anonymous. BIN-BENCH: Can LLM Agents Reason Through Long-Horizon Binary Analysis? OpenReview, 2026. URL https://openreview.net/pdf?id=KCBDPLjVrV. Anonymous ACL submission. [5] Anthropic. How Claude Code works. https://code.claude.com/docs/en/how-claude-code-works, 2026. Accessed August 2, 2026. 12 [6] Anthropic. Introducing Claude Opus 4.8. https://w.anthropic.com/news/claude-opus-4-8, 2026. Released May 28, 2026; accessed October 4, 2026. [7] Anthropic. Introducing Claude Opus 5. https://w.anthropic.com/news/claude-opus-5, 2026. Accessed September 20, 2026. [8] J. Chai, G. Yin, Z. Xu, C. Yue, Y. Jia, S. Xia, X. Wang, J. Jiang, X. Li, C. Dong, H. He, and W. Lin. RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use, 2025. URL https://arxiv.org/abs/2509.06980. [9] H. Chen, Z. Hu, J. Chai, H. Yang, H. He, X. Wang, W. Lin, L. Wang, G. Yin, and Z. Zhao. ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIs, 2025. URL https://arxiv.org/abs/ 2512.16149. [10] H. Chen, S. Zhou, Z. Hu, J. Chai, H. He, H. Yang, X. Wang, W. Lin, T. Wang, Z. Zhao, and G. Yin. ToolForge: A Data Synthesis Pipeline for Multi-Hop, Multi-Turn, and Self-Reflective Tool-Use Data. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026. URL https://openreview.net/ forum?id=vFFehO66Y0. [11] T. Chen, S. Lu, S. Lu, Y. Gong, C. Yang, X. Li, M. R. H. Misu, H. Yu, N. Duan, P. Cheng, F. Yang, S. K. Lahiri, T. Xie, and L. Zhou. Automated proof generation for Rust code via self-evolution. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/ hash/b2e20d7402c9985eae4ba924c65370a8-Abstract-Conference.html. [12] Cilium eBPF Maintainers. BTF: Fixed panics during parsing of malformed input. https://github.com/cilium/ ebpf/pull/2021, 2026. Pull request 2021; merged May 27, 2026. [13] Clang Project. Clang Static Analyzer. https://clang.llvm.org/docs/ClangStaticAnalyzer.html, 2026. Accessed August 4, 2026. [14] CVE Program. CVE List V5: Official CVE records. https://github.com/CVEProject/cvelistV5, 2026. Per-record JSON snapshots archived September 15, 2026. [15] I. David and A. Gervais. CrackMeBench: Binary Reverse Engineering for Agents, 2026. URL https://arxiv. org/abs/2605.10597. [16] DeepSeek-AI. DeepSeek Harness. https://github.com/deepseek-ai/deepseek-harness, 2026. Accessed August 25, 2026. [17] DeepSeek-AI. DeepSeek-V4-Pro GA release. https://api-docs.deepseek.com/news/news260813/, 2026. DeepSeek-V4-Pro-0813 released August 13, 2026; accessed September 20, 2026. [18] DeepSeek-AI et al. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https: //arxiv.org/abs/2606.19348. [19] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?, 2025. URL https://arxiv.org/abs/2509.16941. [20] J. Ding, S. Long, C. Pu, H. Zhou, H. Gao, X. Gao, C. He, Y. Hou, F. Hu, Z. Li, W. Shi, Z. Wang, D. Zan, C. Zhang, X. Zhang, Q. Chen, X. Cheng, B. Deng, Q. Gu, K. Hua, J. Lin, P. Liu, M. Li, X. Pan, Z. Peng, Y. Qin, Y. Shan, Z. Tan, W. Xie, Z. Wang, Y. Yuan, J. Zhang, E. Zhao, Y. Zhao, H. Zhu, L. Zhu, C. Zou, M. Ding, J. Jiao, J. Liu, M. Liu, Q. Liu, C. Tao, J. Yang, T. Yang, Z. Zhang, X. Chen, W. Huang, and G. Zhang. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents, 2025. URL https://arxiv.org/abs/2512.12730. [21] Y. Fang, T. Sun, Y. Shi, M. Wang, and X. Gu. LastingBench: Defend benchmarks against knowledge leakage. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 18304–18317. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-emnlp.993. URL https://aclanthology.org/ 2025.findings-emnlp.993/. [22] Y. Feng, J. Sun, Z. Yang, J. Ai, C. Li, Z. Li, F. Zhang, K. He, R. Ma, J. Lin, J. Sun, Y. Xiao, S. Zhou, W. Wu, Y. Liu, P. Liu, Y. Qiao, S. Zhang, and K. Zhang. LongCLI-Bench: A Preliminary Benchmark and Study for Long- horizon Agentic Programming in Command-Line Interfaces, 2026. URL https://arxiv.org/abs/2602.14337. 13 [23] FFmpeg Project. FFmpeg Source Repository. https://github.com/FFmpeg/FFmpeg, 2026. Accessed August 28, 2026. [24] P. Garg and S. H. Sengamedu. Synthesizing code quality rules from examples. Proceedings of the ACM on Programming Languages, 6(OOPSLA2):1757–1787, 2022. doi: 10.1145/3563350. URL https://doi.org/10.1145/ 3563350. [25] C. S. Gordon. Synthesizing program-specific static analyses, 2018. URL https://arxiv.org/abs/1810.06600. [26] J. Guo, C. Wang, X. Xu, Z. Su, and X. Zhang. RepoAudit: An Autonomous LLM-Agent for Repository- Level Code Auditing. In International Conference on Machine Learning, pages 21083–21100, 2025. URL https: //proceedings.mlr.press/v267/guo25n.html. [27] Y. Hao, W. Chen, Z. Zhou, and W. Cui. E&V: Prompting large language models to perform static analysis by pseudo-code execution and verification, 2023. URL https://arxiv.org/abs/2312.08477. [28] H. He, Y. Luo, C. Wan, T. Su, H. Sun, and G. Pu. Automated detection of atomicity violations in large-scale systems, 2025. URL https://arxiv.org/abs/2504.00521. [29] ImageMagick Project. ImageMagick Source Repository. https://github.com/ImageMagick/ImageMagick, 2026. Accessed August 28, 2026. [30] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Live- CodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/ file/94074d5a072d28f75a76dabed43767-Paper-Conference.pdf. [31] N. Jain, J. Singh, M. Shetty, T. Zhang, L. Zheng, K. Sen, and I. Stoica. R2E-Gym: Procedural environment generation and hybrid verifiers for scaling open-weights SWE agents. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=7evvwwdo3z. [32] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. [33] Kimi Team et al. Kimi K3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653. [34] H. Lee, Z. Zhang, H. Lu, and L. Zhang. SEC-bench: Automated Benchmarking of LLM Agents on Real- World Software Security Tasks. In Advances in Neural Information Processing Systems, 2025. URL https: //arxiv.org/abs/2506.11791. [35] P. Li, S. Yao, J. S. Korich, C. Luo, J. Yu, Y. Cao, and J. Yang. Neuro-symbolic Static Analysis with LLM- generated Vulnerability Patterns, 2025. URL https://arxiv.org/abs/2504.16057v5. Version 5, revised April 2026. [36] Z. Li, S. Dutta, and M. Naik. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/ paper/2025/hash/582d4e27fa24168f3af1f4582655034b-Abstract-Conference.html. [37] Z. Li, Z. Li, Y. Shi, R. Wang, J. Yang, Z. Liu, X. Wu, A. Li, Y. Yu, N. Liu, L. Sun, H. Mi, and L. Liang. Long- Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward- Based Grading, 2026. URL https://arxiv.org/abs/2607.08964. [38] Linux Kernel Community. Linux Kernel Source Repository. https://github.com/torvalds/linux, 2026. Accessed August 28, 2026. [39] T. Liu, C. Xu, and J. McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.c/paper_files/ paper/2024/hash/d191ba4c8923ed8fd8935b7c98658b5f-Abstract-Conference.html. [40] LLVM Project. Clang Static Analyzer Checker Implementations. https://github.com/llvm/llvm-project/tree/ main/clang/lib/StaticAnalyzer/Checkers, 2026. Accessed August 25, 2026. [41] H. He, C. Yue, C. Dong, M. Tian, H. Chen, Z. Liu, J. Chai, X. Wang, Y. Zhang, Q. Liao, G. Yin, W. Lin, C. Wan, H. Sun, and T. Su. LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services. In 14 Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. Association for Computing Machinery, 2026. doi: 10.1145/3770855.3817466. URL https://doi.org/10.1145/3770855.3817466. [42] H. Luo, H. Zhang, X. Zhang, H. Wang, Z. Qin, W. Lu, G. Ma, H. He, Y. Xie, Q. Zhou, Z. Hu, H. Mi, Y. Wang, N. Tan, H. Chen, Y. R. Fung, C. Yuan, and L. Shen. UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios, 2025. URL https://arxiv.org/abs/2509.21766. [43] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=a7Qa4CcHak. [44] D. Miessler and PAI Contributors. Personal AI Infrastructure. https://github.com/danielmiessler/Personal_ AI_Infrastructure, 2026. Accessed August 25, 2026. [45] MITRE Corporation. Common Weakness Enumeration (CWE). https://cwe.mitre.org/, 2026. Accessed August 25, 2026. [46] M. M. Mohajer, R. Aleithan, N. S. Harzevili, M. Wei, A. B. Belle, H. V. Pham, and S. Wang. SkipAnalyzer: A tool for static code analysis with large language models, 2023. URL https://arxiv.org/abs/2310.18532. [47] Moonshot AI. Kimi K3: Model release in Kimi Code. https://w.kimi.com/code/docs/en/kimi-code/whats-new. html, 2026. Release entry dated July 16, 2026; accessed September 19, 2026. [48] Nous Research. Hermes Agent. https://github.com/NousResearch/hermes-agent, 2026. Accessed August 2, 2026. [49] OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition. https://openai.com/index/gpt-5-6/, 2026. Accessed September 14, 2026. [50] OpenClaw Contributors. OpenClaw. https://github.com/openclaw/openclaw, 2026. Accessed August 2, 2026. [51] G. Orlanski, D. Roy, A. Yun, C. Shin, A. Gu, A. Ge, D. Adila, N. Roberts, F. Sala, and A. Albarghouthi. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks, 2026. URL https://arxiv.org/abs/2603.24755. [52] S. Ouyang, W. Yu, K. Ma, Z. Xiao, Z. Zhang, M. Jia, J. Han, H. Zhang, and D. Yu. Repo- Graph: Enhancing AI software engineering with repository-level code graph. In International Confer- ence on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/hash/ 4a4a3c197deac042461c677219efd36c-Abstract-Conference.html. [53] J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang. Training software engineering agents and verifiers with SWE-Gym, 2025. URL https://arxiv.org/abs/2412.21139. [54] W. Peng, Y. Shi, Y. Wang, X. Zhang, B. Shen, and X. Gu. SWE-QA: Can language models answer repository- level code questions?, 2026. URL https://arxiv.org/html/2509.14635v2. Version 2, April 26, 2026; accepted to Findings of ACL 2026. [55] J. Su, Z. Zhao, H. Wang, and H. Chen. CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method, 2026. URL https://arxiv.org/abs/2608.17536. [56] Tencent.Tencent releases and open-sources Tencent Hy4 preview. https://w.tencent.com/ tencent-releases-and-open-sources-tencent-hy4-preview/, 2026. Released August 28, 2026; accessed Septem- ber 19, 2026. [57] H. He, C. Yue, C. Dong, C. Wan, T. Su, H. Sun, J. Chai, X. Wang, and G. Yin. VistaHop: Benchmarking Long-Horizon Visual DeepSearch, 2026. URL https://arxiv.org/abs/2606.03273. [58] B. Wang, W. Xu, Y. Li, X. Gao, Y. Xie, H. Sun, and D. Chen. Improving code localization with repository memory. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum? id=8yjWLJy2eX. [59] C. Wang, Z. Li, S. Dutta, and M. Naik. QLCoder: A query synthesizer for static analysis of security vulnerabili- ties. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.c/paper_ files/paper/2026/hash/2adf01ab15adde8820622f7f24bd516b-Abstract-Conference.html. 15 [60] L. Wang, C. Chen, J. Zhu, R. Zhan, and W. Han. CQLLM: A Framework for Generating CodeQL Security Vulnerability Detection Code Based on Large Language Model. Applied Sciences, 16(1):517, 2026. doi: 10.3390/ app16010517. URL https://w.mdpi.com/2076-3417/16/1/517. [61] X. Wang, R. Hu, C. Gao, X.-C. Wen, Y. Chen, and Q. Liao. ReposVul: A repository-level high-quality vulnera- bility dataset, 2024. URL https://arxiv.org/abs/2401.13169. [62] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/ hash/a4b6ad6b48850c0c331d1259fc66a69c-Abstract-Conference.html. [63] Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, and D. Song. CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale. In International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=2YvbLQEdYt. [64] X.-C. Wen, X. Wang, Y. Chen, R. Hu, D. Lo, and C. Gao. VulEval: Towards repository-level evaluation of software vulnerability detection, 2024. URL https://arxiv.org/abs/2404.15596. [65] H. Wu, C. Barrett, and N. Narodytska. Lemur: Integrating large language models in automated program verification. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr. c/paper_files/paper/2024/hash/0c86142265c5e2c900613d1d031cb90-Abstract-Conference.html. [66] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying LLM-based software engineering agents, 2024. URL https://arxiv.org/abs/2407.01489. [67] C. Yang, Z. Zhao, Z. Xie, H. Li, and L. Zhang. KNighter: Transforming static analysis with LLM-synthesized checkers, 2025. URL https://arxiv.org/abs/2503.09002. [68] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793. [69] A. Yildiz, S. G. Teo, Y. Lou, Y. Feng, C. Wang, and D. M. Divakaran. Benchmarking LLMs and LLM-based agents in practical vulnerability detection for code repositories, 2025. URL https://arxiv.org/abs/2503.03586. [70] Z.ai. GLM-5.2: Built for long-horizon tasks. https://z.ai/blog/glm-5.2, 2026. Accessed August 2, 2026. [71] Z.ai. GLM-5.2: Developer release notes. https://docs.z.ai/release-notes/new-released, 2026. Release entry dated June 16, 2026; accessed September 19, 2026. [72] W. Zeng, J. Zhang, H. Chen, Z. Hu, Y. Liang, J. Chai, D. Liu, Z. Liu, S. Yan, M. Xue, X. Wang, W. Lin, and G. Yin. RecRM-Bench: Benchmarking Multidimensional Reward Modeling for Agentic Recommender Systems, 2026. URL https://arxiv.org/abs/2605.11874. [73] Q. Zhou, J. Zhang, H. Wang, R. Hao, J. Wang, M. Han, Y. Yang, S. Wu, F. Pan, L. Fan, D. Tu, and Z. Zhang. FeatureBench: Benchmarking agentic coding for complex feature development. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=41xrZ3uGuI. 16 Appendix Contents A Extended Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B Dataset Construction Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.1 Source Collection and Provenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.2 Build Recovery and Executable Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.3 Task Packaging and Shared References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.4 Human-Guided Pilot Rerolls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 C Quality-Control and Contamination Audits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 C.1 Temporal Coverage and Prior-Exposure Audit . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C.2 Time-Window Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.3 Paired Prior-Exposure Sensitivity Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 D Dataset Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 D.1 Repository-Level Task Counts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 D.2 Programming Languages and Static Analysis Frameworks . . . . . . . . . . . . . . . . . . . 27 D.3 Complete CWE Label Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 D.4 Calibration Horizon and Difficulty Distributions . . . . . . . . . . . . . . . . . . . . . . . . . 29 D.5 Pilot Trajectory Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 D.6 CVE Publication-Time Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 E Evaluation and Inference Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 E.1 System Configurations and Reproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 E.2 Task Admission and Independent Repeats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 E.3 Source and Evidence Freezing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 E.4 Backend Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 E.5 Frozen Mainline Scan and Single-Judge Decisions . . . . . . . . . . . . . . . . . . . . . . . . 32 E.6 Mainline Alert Selection and Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 E.7 Failure Accounting, Recovery, and Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . 34 E.8 Statistical Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 E.9 Human Validation of the LLM Judge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 F Post-Synthesis Verifier Rubric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 F.1 Independent Replay and Diagnostic Predicates . . . . . . . . . . . . . . . . . . . . . . . . . 36 F.2 CSA Deterministic Objective Gate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 F.3 CSA Process and Semantic Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 F.4 CodeQL Quality Gates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 F.5 Mainline False Positives and Final Verdict . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 G Representative CSA Checker Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 H Additional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 H.1 KNighter Comparison Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 H.2 Ablation Outcomes and Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 H.3 Time-Budget Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 H.4 Language, Repository, and CWE Performance . . . . . . . . . . . . . . . . . . . . . . . . . . 43 17 A Extended Related Work Table 1 applies the following operational criteria. Repository Context requires the evaluated agent or system to receive a complete repository snapshot together with general-purpose navigation capabilities. Isolated functions, patch-only inputs, pre-extracted dependency views, and repositories used solely by an external validation pipeline receive partial credit. Executable Environment requires a reproducible repository or an- alyzer state with automated build and execution support. Multi-Harness Support is assessed against the six coding-agent harnesses integrated through CheckerBench’s adapters: Claude Code [5],Hermes Agent [48],OpenClaw [50],PAI [44],OpenCode [3], andDS Harness [16]. Full support requires documented, reproducible integrations for all six harnesses under a shared task and scoring contract; support for a subset receives partial credit. A generic patch-submission format, compatibility that depends on an unreleased wrapper, or a system-specific internal agent does not count as released harness support. Long-Horizon Interaction requires a repeated observation–tool–edit–execution loop. Protocols that permit multi-step agent execution or bounded retrieval without specifying the complete loop receive partial credit. Checker Synthesis requires the task output to be an executable, reusable static-analysis rule or checker. Se- mantic Verification requires an independent rerun of the submitted checker on vulnerable and fixed code together with patch-localized diagnostic evidence; self-reported counts, generic repository tests, and vulner- ability labels receive partial credit. False-Positive Evaluation requires residual diagnostic control on the fixed revision, a production scan, or an equivalent procedure that measures overly broad reporting. The audit reflects the task definitions, released artifacts, and evaluation protocols documented by each work, not capabilities that could be supplied by an external agent implementation. Adjacent agent benchmarks. Recent benchmarks extend agentic software engineering beyond short issue resolution tasks. NL2Repo-Bench starts with a natural-language specification and an empty workspace, re- quiring agents to construct an installable, multi-module Python library [20]. SWE-Bench Pro instead draws complex issues from maintained repositories, where solutions often require coordinated changes across multi- ple files [19]. LongCLI-Bench covers from-scratch development, feature addition, bug fixing, and refactoring in a command-line environment, using requirement and regression tests together with step-level scoring [22]. SlopCodeBench follows agents as they repeatedly extend their own solutions under evolving requirements, measuring checkpoint success alongside structural erosion and code verbosity [51]. Other evaluations focus on the trajectory itself. Long-Horizon-Terminal-Bench uses graded subtasks across software engineering and other terminal workflows to measure partial progress during extended runs [37]. UltraHorizon tests sustained exploration in partially observable environments, probing planning, memory, and tool use over long interactions [42]. These benchmarks expose failure modes in sustained execution that also matter when an agent must implement, compile, and refine a checker. Binary-analysis benchmarks address a related security workflow. BIN-BENCH evaluates long chains of tool- assisted binary analysis and examines reasoning traces in addition to final outcomes [4]. AgentRE-Bench asks agents to investigate unfamiliar binaries without source code and scores their structured behavioral claims against fixed ground truth [1]. CrackMeBench requires agents to recover a binary’s validation logic and submit an input or generator accepted by the executable [15]. Together, these works measure software construction, intermediate progress, and binary reasoning. CheckerBench evaluates a distinct artifact: a reusable analyzer checker, independently rebuilt and tested for vulnerable–fixed diagnostic contrast, patch localization, and false positives. B Dataset Construction Details B.1 Source Collection and Provenance We collect public CVE records from the National Vulnerability Database and link them to fixing commits in open-source repositories written in C/C++, Java, Python, JavaScript, and Go. A candidate must identify an exact fix, a retrievable parent revision, and a patch that exposes a defect-bearing program change. CVE identifiers provide instance-level provenance; CWE labels support coverage analysis and stratified sampling. We discard candidates with unresolved provenance, unavailable source, incompatible redistribution terms, or 18 no reusable static-analysis property. A stable instance identifier binds each retained repository, revision pair, patch, and downstream artifact. B.2 Build Recovery and Executable Scope The following build-anchor and compilation-database details describe the CSA construction adapter. CodeQL tasks retain their own pinned query-analysis environment and reconstruction metadata. The construction catalog links each vulnerability-fixing commit pair to its repository and to one build anchor per revision. Each anchor records a compilation database, a build snapshot, and build-completeness metadata. Construction first extracts the unified diff, commit message, and pre- and post-fix functions. The vulnerable-side compilation database then filters changed files to translation units for which the original include paths, preprocessor definitions, language mode, and generated dependencies can be recovered. Unmatched files remain in the provenance record as skipped inputs; pairs with no analyzable changed translation unit do not enter synthesis. For each retained pair, the builder creates isolated Git worktrees for the two source revisions and overlays revision-compatible generated files from the corresponding snapshots. Paths in compilation-database entries are rewritten from the original repository and build roots to the worktree and snapshot roots. Output directives, warnings-as-errors, disabled system includes, and known incompatible GCC-specific options are removed or normalized before CSA execution. Worktrees permit the two revisions to coexist and allow pairs to be processed concurrently without changing the shared repository mirror. They are removed after scanning, and stale worktrees from interrupted tasks are pruned before subsequent construction runs. After recovering both revisions, we extract the affected pre- and post-fix functions and generate deterministic commands for compiling a checker and scanning the pair. We reject an instance if either revision remains unreconstructable, affected translation units cannot be analyzed after bounded build recovery, or the pair does not support stable differential analysis. The resulting source–build correspondence, build metadata, and scan context remain available for auditing failures. B.3 Task Packaging and Shared References Each task is packaged in a randomly named workspace containing the patch, affected functions, a minimal checker template, compile and scan scripts, and read-only reference assets. The visible configuration contains repository, revision, build, and scan-scope metadata but omits CVE and CWE identifiers. Static templates and reference assets are hard-linked when possible and copied otherwise; in either case they are made read- only. A separate provenance ledger maps the random identifier back to the source pair. This design exposes all inputs required for synthesis while preventing vulnerability labels from becoming task shortcuts. The shared library K has two layers. Harness-specific skills support patch interpretation, checker design, build recovery, functional validation, and false-positive triage. A harness-independent, read-only reference layer provides analyzer API documentation, official checker or rule implementations (including CSA resources for C/C++ tasks [13, 40]), reusable language-specific AST, data-flow, and state utilities, CWE mechanism cards [45], and repository-specific conventions when available. These assets provide implementation knowl- edge but do not encode the task’s defect-specific state-transition logic or diagnostic reporting predicate. The agent must infer defect semantics, select analyzer extension points and semantic or state representations, implement and register the checker, and refine it using compiler and analyzer feedback. A completed run produces checker source and plugin or query artifacts, a structured defect analysis, and a scope statement. The declarations support analysis; only independent execution establishes correctness. B.4 Human-Guided Pilot Rerolls Calibration fixes model and harness versions, each task’s analyzer container and language toolchain, workspace schema, enabled tools, shared skills, and time, token, and interaction budgets for each attempt. Pilot runs cover the inspect–implement–compile–scan–repair cycle. We retain observations, edits, tool calls, and responses. One interaction step is one agent decision, independent of token count or low-level tool events. For each candidate task, the construction protocol permits one initial rollout and at most two additional roll- outs after human review. Each attempt uses an isolated workspace and the same task, analyzer configuration, verification predicates, and acceptance thresholds. After a failed checker submission, the reviewer identifies 19 the failure from execution logs and verification evidence, revises the prompt guidance, and records the reason for the change before launching the next rollout. A prompt revision guides the agent’s next attempt; every resulting checker is independently checked. The ledger links the attempt index (1–3), prompt version and changes, human diagnosis, source checksum, complete trace, verification evidence, verdict, and termination status. The first passing checker advances to quality control and sampling. A task with no passing checker after attempt 3 is excluded. The released checker–trace pairs therefore all pass the construction-time checker-admission criteria. These pilot retries are recorded separately from independent main-evaluation repeats and their infrastructure-recovery attempts. Human diagnosis can address compiler errors, missed or mislocalized diagnostics, semantic mismatches, or false positives. Verifier internals and task-specific reference checker source remain held out from the calibra- tion agent. Each submitted checker undergoes independent contrastive and patch-relevance checks; accepted executable checkers additionally undergo a subsystem scan on a prebuilt main-branch worktree and sampled LLM warning triage (Appendix C). All applicable executable, semantic, scope, and false-positive gates must pass. Final selection requires reproducible artifacts, complete traces, and independent verification records. We re- tain passing trajectories with at least 40 interaction steps and sample jointly over repository, CWE category, observed horizon, and difficulty. This threshold selects sustained interaction under the calibration configura- tion; it is not an intrinsic lower bound on task difficulty. The construction ledger retains checker source and traces for every attempt, independent verification and false-positive evidence, human diagnoses and prompt revisions, attempt counts, and either a passing checker–trace pair eligible for curation or an exclusion record after the cap. C Quality-Control and Contamination Audits Automated admission checks cover revision ancestry, patch and artifact integrity, the presence of build an- chors, compilation-database coverage, clean checker compilation, registration and plugin loading, determin- istic vulnerable–fixed scans, verifier isolation, and package completeness. The CSA contrastive gate requires N − i > N + i and N + i < 50; CodeQL retains its own frozen differential predicate. Passing the CSA gate is insuf- ficient by itself: code verification recomputes diagnostics and requires at least one disappearing vulnerable- revision report whose source location maps to a function modified by the patch. Hence empty scans, unrelated diagnostic reductions, and agent-reported counts cannot satisfy the hard gate. For accepted executable checkers, a broader subsystem scan on a prebuilt main-branch worktree supplies false- positive evidence. This worktree is neither the vulnerable nor the fixed task revision. During construction- time calibration, a triage model assesses sampled diagnostics. Failed false-positive checks enter the same human-diagnosis and prompt-revision loop as other admission failures, within the shared three-attempt cap (Appendix B.4). Each new checker is rebuilt and independently rechecked. The main evaluation uses the frozen-candidate scan and single Claude Opus 5 judge in Section 4, without returning scores for repair. Manual audit samples passing and failing submissions across projects, CWE families, and horizon strata, and inspects both defect-mechanism fidelity and the patch-localized true positive. Final removal counts, auditor agreement, and contamination-overlap statistics will be reported with the frozen release. Deduplication and split assignment. Manual review confirms that the patch supports the stated defect and that disappearing reports lie in changed defect-bearing functions. We remove exact duplicate revision pairs; separable units from the same CVE retain a shared provenance link. Connected groups formed by repository identity, CVE identity, shared revisions, and patch similarity are each assigned to one split. Repository- disjoint and weakness-family-disjoint subsets are additionally marked for generalization analysis. Artifact isolation. Reference states, task-specific checker implementations, calibration trajectories, verifier thresholds, and adjudication records remain outside the agent-visible snapshot. The vulnerability patch is an intentional task input. Potential training exposure to public vulnerability information is distinct from evaluation-time leakage of checker implementations, calibration traces, or verifier artifacts. The following 20 Table 5 Official model release dates and task counts by CVE publication date. The CVE snapshot is September 15, 2026; model release sources were checked September 19–20, 2026, with Opus 4.8 rechecked October 4, 2026. “On date” counts publications on the release day. Later publication alone does not certify an unseen task. ModelRelease date Before On date After GPT-5.62026-07-0930000 Claude Opus 4.8 2026-05-2829523 Qwen3.8-Max2026-08-0230000 DeepSeek-V4-Pro 2026-08-1330000 GLM-5.22026-06-1629901 Kimi-K32026-07-1630000 HY4-Preview2026-08-2830000 audits examine public dates and model release boundaries; artifact isolation addresses the separate evaluation- time leakage risk. C.1 Temporal Coverage and Prior-Exposure Audit Public evidence and date provenance. We archive CVE Program JSON records for the fixed task inventory, recording the source URL, retrieval provenance, and SHA-256 checksum alongside each task [14]. The audit snapshot of September 15, 2026 covers all 300 tasks and 297 canonical CVEs. The original inventory identifier CVE-2024-7773 is a rejected duplicate of CVE-2024-45436; the audit retains the original identifier and uses the canonical record for dating. No task is removed by this normalization. Publication dates range from June 14, 2010, to June 26, 2026. Appendix D.6 gives the complete year distribution. The machine-readable ledger is figures/temporal/task_dates.csv, accompanied by the archived records and a public-evidence index. The ledger separates CVE publication, the CNA’s explicit public-disclosure date, dated public events, and independently checked patch or checker publication evidence. Discovery dates, CVE reservation dates, packag- ing dates, and CVE identifier years do not establish public availability. Git author and committer timestamps alone also do not establish when a patch became public. For 48 tasks, the CVE record explicitly dates a pub- lic event before its own publication. The earliest documented event establishes that information was public by that date; it does not prove the absence of still earlier sources. The full first-public patch and checker audit is unresolved, so the ledger does not certify any task as previously unseen on the basis of publication date alone. Official model release dates. Table 5 uses public release dates documented in provider announcements or release logs, checked on September 19–20, 2026, with the Opus 4.8 source rechecked on October 4, 2026. GPT-5.6 Sol andClaude Opus 4.8 were released on July 9 and May 28, respectively [6, 49]. The official records dateQwen3.8-Max to August 2,DeepSeek-V4-Pro (the evaluated 0813 version) to August 13, GLM-5.2 to June 16, andKimi-K3 to July 16 [2, 17, 47, 71].HY4-Preview was released on August 28 [56]. Dates follow the named model version. Public service or API availability counts as release even if weights follow later; named public previews are included and restricted partner previews excluded. The ledger figures/temporal/model_releases.json records each source and date convention. Counts distinguish CVE publication before, on, and after the reported release day. These boundaries describe public availability, not a model’s training-data cutoff. The actual model version must be pinned and matched to its release before temporal scoring. A patch-before-CVE example. CVE-2026-10722, a task in cilium/ebpf, was published on June 3, 2026. Its CVE record references pull request #2021, whose fix was already merged on May 27, 2026 [12]. Thus the patch was public a week before the CVE publication. A CVE-only filter can overstate the evidence for novelty relative to a model release between those dates. We retain both dates and require review of the earlier source; we do not infer whether the model actually trained on it. 21 C.2 Time-Window Evaluation Protocol Following LiveCodeBench [30], the temporal analysis groups eligible tasks by audited public-availability windows and compares Pass@1 for each harness–model pair using the same three independent generation repeats as the main comparison. Eligibility requires a pinned model version and its official release date together with an audit of public vulnerability descriptions, patches, and task-specific checker sources. Tasks with unresolved public-availability dates remain explicitly unclassified. A common post-release comparison requires the same audited task subset after all included models’ releases. The current inventory has no CVE publications after the latest of the seven release dates, so it supplies no common post-release subset for all seven models. Model-specific comparisons report their own eligible task counts and remain conditional on the earlier-source audit. Windows are fixed from date and task metadata before inspecting outcomes. We begin with half-year windows and merge adjacent windows until each contains at least 20 distinct CVEs, without crossing a model’s release date. A remaining smaller subset is reported with its sample count and interval, without fitting a temporal trend. Each point reports its distinct task count and a 95% confidence interval. For each window, we compute Pass@1 from the final-pass verdicts of the same eligible tasks separately in each repeat and report the arithmetic mean of the three rates, weighting repeats equally. Resampling keeps the three repeat outcomes for each task and all tasks sharing a CVE together. We report language and static analysis framework strata wherever both periods have support, and disclose repository and CWE composition for every window; unsupported strata are not treated as matched comparisons. Construction-time horizon information serves only as a workload proxy. A score decline across time is interpreted jointly with these composition differences, rather than as proof of memorization. The patch remains an intended synthesis input. Reference checker code, calibration trajectories, and verifier artifacts remain held out from the agent-visible package. The audit of generation traces separately records any external retrieval of task-specific solutions or verification material; time-window evidence alone cannot rule out such evaluation-time leakage. The CVE publication counts in Figure 10(a) describe coverage only and do not supply performance estimates. The construction pilot trajectories do not supply independent temporal performance estimates. Performance curves are populated only from completed main-evaluation verdicts and audited task windows. C.3 Paired Prior-Exposure Sensitivity Protocol Sampling. The sampling protocol was fixed before collecting supplementary model outcomes. The initial candidate roster contains 100 distinct canonical CVEs: 53 C/C++, 17 Python, and 10 each in Go, Java, and JavaScript. Within each language, task IDs are sorted and placed in a fixed pseudorandom order; canonical- CVE duplicates are skipped across strata. The full candidate order and original metadata hashes are retained. A task that fails environment validation is replaced by the next eligible task in its frozen language order, with the failure evidence and replacement recorded before any generation or recognition probe. The final roster is admitted only after all 100 tasks pass validation. Selection and replacement never use model outcomes or probe recognition. Conditions and execution. Claude Opus 4.8 andDeepSeek-V4-Pro each run withOpenCode under original (O), ordinary-paraphrase (P), and identity-cue-reduced (D) conditions. P rewrites non-executable task prose while preserving requirements, source identifiers, and the complete patch. D starts from P and con- sistently substitutes safely modifiable internal identifiers with meaningful alternatives, and removes residual provenance cues from visible metadata. Standard-library and required external API names, types, constants, control/data flow, and the vulnerability and fix mechanisms are preserved. Unsafe transformations involving reflection, macros, or dynamic name lookup are rejected. This is a semantics-preserving intervention, distinct from the answer-changing counterfactuals studied by LastingBench [21]. There is one independent generation per task–model–condition combination, for 600 scheduled generations. This supplementary experiment does not reuse historical Full trajectories. All three conditions share the same skill library, executable feedback, 3,000-second active budget, model decoding settings within each model, and frozen verification rules with Claude Opus 5 as judge. Condition order is randomized within 22 task blocks. Workspaces and sessions are independent, and probes never enter generation context. Model API IDs, returned version metadata, access dates, harness version, and environment fingerprints are archived. Mutable API aliases cannot establish that a provider’s underlying weights remained unchanged. Transformation and admission checks. The before, after, and mainline source trees, patch, extracted con- text, metadata, and tool-visible paths must agree with the same transformation. For CodeQL, all three corresponding databases must be rebuilt from transformed source; changing only prompts, source-archive text, or database display labels does not qualify. Original Git history, source copies, private reverse maps, reference checkers, and verifier material must be inaccessible to generation. All conditions use preinstalled dependencies and local documentation, with network access restricted to the model gateway. Admission requires reverse-mapped syntax checks, human review of defect and fix equivalence, successful builds or database extraction, and independent reference-checker replay covering differential behavior, patch localization, and mainline false positives. Validation logs and human review records bind to the exact variant hashes. Missing evidence blocks admission. Infrastructure failures follow the existing bounded recovery policy and are reported separately; they are not imputed as model failures. Valid model failures and budget exhaustion remain failures, without score-based retries. Claude Opus 4.8 drafts transformation proposals andDeepSeek-V4-Pro provides preliminary critiques. These preparation calls are separate from the 600 synthesis runs and 400 recognition probes and request neither checker solutions nor CVE identification. A human reviewer checks defect and fix equivalence in the complete transformed environments; proposal review does not admit an unbuilt environment. Final semantic approval must reference the exact transformed environment hashes and retain the reviewer identity, review date, supporting evidence, and decision. Independent executable checks remain mandatory. Human review records are retained with the experiment artifacts. Closed-book recognition probes. Each task–model pair receives separate P and D probes, giving 400 sched- uled short calls. Each probe contains pre-fix code only: no fixing patch, CVE/CWE label, source URL, or identifying document heading. A fixed extraction rule selects at most 16,000 characters of code at complete line boundaries; P and D use corresponding source spans, with equal clipping boundaries rather than inde- pendently selected excerpts. Probe requests use temperature 0 and an output limit of 256 tokens, with no tools or retrieval. The common instruction is: Identify the publicly documented CVE associated with the supplied pre-fix code only if you recognize this specific instance. Do not infer an identifier from the general vulnerability type. Return exactly one JSON object with the key cve_id: a single CVE identifier, or "unknown" if you do not know. Do not use tools or external retrieval. ID Recall is the fraction of valid completed calls returning the canonical CVE; documented duplicate identi- fiers are normalized through the frozen ledger. Incorrect, unknown, refused, or malformed model answers do not receive repair attempts; infrastructure-incomplete calls remain unobserved. A correct identifier indicates instance familiarity, not memorization of a checker solution. Failure to recall an identifier does not establish non-exposure. Estimation and interpretation. The primary outcome is the existing final full_pass verdict. For each model, the primary effect is ∆ identity = 100 N N X i=1 Y i,P − Y i,D , N = 100, in percentage points. O–P and O–D differences separate general paraphrase sensitivity from the total inter- vention. The prespecified uncertainty analysis uses 10,000 paired bootstrap draws, resampling canonical-CVE clusters jointly across conditions, to obtain percentile 95% intervals. A supplementary analysis resamples repositories while retaining within-repository task weights. The intervals describe variation across sam- pled tasks/clusters under one realized generation per condition; they do not separately estimate repeated- generation variability. The two models are reported separately. 23 Probe P–D recall differences and descriptive P–D effects among tasks recognized and unrecognized under P provide supporting evidence. A decline in recognition together with a positive P–D effect supports sensitivity to identifying cues; ordinary paraphrase effects and residual difficulty changes remain alternative explana- tions. We prespecify a practical margin of ±5 percentage points. Only an entire 95% interval inside that margin supports a bounded effect under this intervention. A nonsignificant difference alone does not sup- port equivalence or absence of training contamination. SWE-QA similarly interprets its older-versus-newer source-cohort gap as a possible exposure association [54]. Build and Diff. SR retain the scheduled task denominator. Semantic scores report their own valid-score counts. Incomplete conditions and contrasts remain unavailable in the main comparison rather than silently changing denominators. Figure 10(c) in the main text reports aggregate Pass@1 and exact CVE recall for both models. The P–D effects are derived from O/P/D Pass@1 of 43.00/42.00/39.00% for Claude Opus 4.8 and 55.00/54.00/52.00% forDeepSeek-V4-Pro, yielding +3.00 and +2.00 percentage points, respectively. The corresponding P–D recall declines are 9.00 and 7.00 percentage points. D Dataset Composition D.1 Repository-Level Task Counts CheckerBench contains 300 tasks from 167 repositories: 159 C/C++ tasks use CSA, while the 141 CodeQL tasks comprise Java (30), Python (51), JavaScript (30), and Go (30). Table 6 lists every repository, its target language, static analysis framework, task count, and share. Projects are counted at the repository level. Dataset prefixes are removed, and repository identifiers are compared without case sensitivity. The historical python-imaging/Pillow identifier is merged with python-pillow/Pillow; ImageMagick/ImageMagick and ImageMagick/ImageMagick6 remain separate repos- itories. Target language denotes the language analyzed by the task, which can differ from the repository’s dominant implementation language. The counts are taken from the frozen task inventory and its repository- level aggregation. Across the full corpus, 151 repositories contribute one task each, 7 contribute two, 4 contribute three, and 4 contribute four; the Linux kernel repository contributes 107 tasks (35.7%). Table 6 Complete repository-level task counts. Shares use all 300 tasks as the denominator and are rounded to one decimal place. RepositoryTarget language Static analysis frameworkTasks Share torvalds/linuxC/C++CSA107 35.7% ffmpeg/ffmpegC/C++CSA41.3% imagemagick/imagemagickC/C++CSA41.3% jerryscript-project/jerryscriptC/C++CSA41.3% qemu/qemuC/C++CSA41.3% airlift/aircompressorJavaCodeQL31.0% freerdp/freerdpC/C++CSA31.0% gpac/gpacC/C++CSA31.0% libredwg/libredwgC/C++CSA31.0% ikus060/rdiffwebPythonCodeQL20.7% libarchive/libarchiveC/C++CSA20.7% mlflow/mlflowPythonCodeQL20.7% openexr/openexrC/C++CSA20.7% python-pillow/pillowPythonCodeQL20.7% rweather/noise-javaJavaCodeQL20.7% verdammelt/tnefC/C++CSA20.7% adaptivecomputing/torqueC/C++CSA10.3% alex/rplyPythonCodeQL10.3% andialbrecht/sqlparsePythonCodeQL10.3% ansible/ansiblePythonCodeQL10.3% apache/incubator-answerJavaScriptCodeQL10.3% apache/netbeans-html4jJavaCodeQL10.3% apache/sling-org-apache-sling-jcr-baseJavaCodeQL10.3% apache/thriftJavaScriptCodeQL10.3% Continued on next page 24 Table 6 continued RepositoryTarget language Static analysis frameworkTasks Share appleboy/gorushGoCodeQL10.3% argoproj/argo-eventsGoCodeQL10.3% argoproj/argo-workflowsGoCodeQL10.3% auth0/nextjs-auth0JavaScriptCodeQL10.3% avo-hq/avoJavaScriptCodeQL10.3% aws/aws-sdk-goGoCodeQL10.3% bettercap/bettercapGoCodeQL10.3% bitly/oauth2_proxyGoCodeQL10.3% bonigarcia/webdrivermanagerJavaCodeQL10.3% bonitasoft/bonita-connector-webserviceJavaCodeQL10.3% briancappello/flask-unchainedPythonCodeQL10.3% brokercap/bifrostJavaScriptCodeQL10.3% bspkrs/mcpmappingviewerJavaCodeQL10.3% caronc/apprisePythonCodeQL10.3% ccxvii/mujsC/C++CSA10.3% cesnet/libyangC/C++CSA10.3% chaosblade-io/chaosbladeGoCodeQL10.3% churchcrm/crmJavaScriptCodeQL10.3% cilium/ebpfGoCodeQL10.3% codehaus-plexus/plexus-utilsJavaCodeQL10.3% containernetworking/pluginsGoCodeQL10.3% corazawaf/corazaGoCodeQL10.3% cowtowncoder/java-merge-sortJavaCodeQL10.3% craftcms/cmsJavaScriptCodeQL10.3% cryptpad/cryptpadJavaScriptCodeQL10.3% darrenofficial/dpastePythonCodeQL10.3% datadog/guarddogPythonCodeQL10.3% decidim/decidimJavaScriptCodeQL10.3% decolua/9routerJavaScriptCodeQL10.3% dedsecinside/torbotPythonCodeQL10.3% deeplook/svglibPythonCodeQL10.3% dhowden/tagGoCodeQL10.3% dojo/dojoJavaScriptCodeQL10.3% duke-git/lancetGoCodeQL10.3% dwisiswant0/apkleaksPythonCodeQL10.3% eladnava/mailgenJavaScriptCodeQL10.3% electerm/electermJavaScriptCodeQL10.3% encode/starlettePythonCodeQL10.3% ethyca/fidesPythonCodeQL10.3% fonttools/fonttoolsPythonCodeQL10.3% foxcpp/maddyGoCodeQL10.3% gomarkdown/markdownGoCodeQL10.3% googleapis/google-oauth-java-clientJavaCodeQL10.3% goreleaser/goreleaserGoCodeQL10.3% gphper/ginadminGoCodeQL10.3% gruntjs/gruntJavaScriptCodeQL10.3% hedgedoc/hedgedocJavaScriptCodeQL10.3% horazont/xmpp-http-uploadPythonCodeQL10.3% horovod/horovodPythonCodeQL10.3% huggingface/transformersPythonCodeQL10.3% hybridgroup/gobotGoCodeQL10.3% hyperadev/dragonflyJavaCodeQL10.3% hyperledger/fabricGoCodeQL10.3% ifmeorg/ifmeJavaScriptCodeQL10.3% imagemagick/imagemagick6C/C++CSA10.3% jberet/jsr352JavaCodeQL10.3% jdhwpgmbca/pcaptureJavaCodeQL10.3% jedisct1/pure-ftpdC/C++CSA10.3% jenkinsci/aws-global-configuration-pluginJavaCodeQL10.3% jenkinsci/azure-ad-pluginJavaScriptCodeQL10.3% jenkinsci/codedx-pluginJavaCodeQL10.3% jenkinsci/dependency-track-pluginJavaCodeQL10.3% jenkinsci/environment-manager-tools-pluginJavaCodeQL10.3% jenkinsci/xcode-pluginJavaCodeQL10.3% jupyter-server/jupyter_serverPythonCodeQL10.3% keycloak/keycloakJavaScriptCodeQL10.3% keylime/keylimePythonCodeQL10.3% knative/funcGoCodeQL10.3% Continued on next page 25 Table 6 continued RepositoryTarget language Static analysis frameworkTasks Share kyverno/kyvernoGoCodeQL10.3% langchain-ai/langchainPythonCodeQL10.3% libical/libicalC/C++CSA10.3% libssh2/libssh2C/C++CSA10.3% libvnc/libvncserverC/C++CSA10.3% line/centraldogmaJavaScriptCodeQL10.3% mansuf/mangadex-downloaderPythonCodeQL10.3% matrix-org/matrix-react-sdkJavaScriptCodeQL10.3% matrix-org/sydentPythonCodeQL10.3% matrix-org/synapsePythonCodeQL10.3% mde/ejsJavaScriptCodeQL10.3% meetecho/janus-gatewayJavaScriptCodeQL10.3% mholt/archiverGoCodeQL10.3% michaelforney/samuraiC/C++CSA10.3% misp/misp-maltegoPythonCodeQL10.3% mity/md4cC/C++CSA10.3% mommyheather/advancedbackupsJavaCodeQL10.3% moodle/moodleJavaScriptCodeQL10.3% mrvautin/expresscartJavaScriptCodeQL10.3% mwarning/kadnodeC/C++CSA10.3% nautobot/nautobot-plugin-device-onboardingPythonCodeQL10.3% netplex/json-smart-v2JavaCodeQL10.3% netty/netty-incubator-codec-quicJavaCodeQL10.3% nltk/nltkPythonCodeQL10.3% nvbn/thefuckPythonCodeQL10.3% nvidia/nvflarePythonCodeQL10.3% nyuccl/psiturkPythonCodeQL10.3% oblac/jodd-httpJavaCodeQL10.3% ofirdagan/cross-domain-local-storageJavaScriptCodeQL10.3% ollama/ollamaGoCodeQL10.3% openstack/ceilometerPythonCodeQL10.3% opensuse/libsolvC/C++CSA10.3% ossrs/srsGoCodeQL10.3% outray-tunnel/outrayJavaScriptCodeQL10.3% overleaf/overleafJavaScriptCodeQL10.3% pallets/werkzeugPythonCodeQL10.3% panva/joseJavaScriptCodeQL10.3% paramiko/paramikoPythonCodeQL10.3% petl-developers/petlPythonCodeQL10.3% pgadmin-org/pgadmin4PythonCodeQL10.3% phpmyadmin/phpmyadminJavaScriptCodeQL10.3% pomerium/pomeriumGoCodeQL10.3% prismjs/prismJavaScriptCodeQL10.3% prometheus/client_golangGoCodeQL10.3% pytorch/serveJavaCodeQL10.3% pytorchlightning/pytorch-lightningPythonCodeQL10.3% rasahq/rasaPythonCodeQL10.3% redis/redisC/C++CSA10.3% revel/revelGoCodeQL10.3% saitoha/libsixelC/C++CSA10.3% sajari/docconvGoCodeQL10.3% sass/libsassC/C++CSA10.3% satori/go.uuidGoCodeQL10.3% seccomp/libseccompC/C++CSA10.3% sehmaschine/django-grappelliPythonCodeQL10.3% skvadrik/re2cC/C++CSA10.3% smallrye/smallrye-configJavaCodeQL10.3% snowflakedb/snowflake-connector-pythonPythonCodeQL10.3% spiral-project/ihatemoneyPythonCodeQL10.3% stranger6667/pyanyapiPythonCodeQL10.3% strukturag/libde265C/C++CSA10.3% strukturag/libheifC/C++CSA10.3% tankywoo/simikiPythonCodeQL10.3% tdunning/pig-vectorJavaCodeQL10.3% tenable/integration-jira-cloudPythonCodeQL10.3% txthinking/brookGoCodeQL10.3% uclouvain/openjpegC/C++CSA10.3% vadz/libtiffC/C++CSA10.3% Continued on next page 26 Table 6 continued RepositoryTarget language Static analysis frameworkTasks Share vyperlang/vyperPythonCodeQL10.3% weaveworks/tf-controllerGoCodeQL10.3% wildfly/jboss-ejb-clientJavaCodeQL10.3% xsuchy/templated-dictionaryPythonCodeQL10.3% xuxueli/xxl-jobJavaCodeQL10.3% zopefoundation/products.genericsetupPythonCodeQL10.3% zwczou/weixin-pythonPythonCodeQL10.3% Total: 167 repositories5 ecosystemsCSA / CodeQL 300 100.0% D.2 Programming Languages and Static Analysis Frameworks Table 7 gives the complete task distribution by target programming language and static analysis framework. Each task contributes once. Language denotes the code analyzed by the task, using the same definition and frozen inventory as Table 6. Table 7 Complete task counts by target language and static analysis framework. Shares use all 300 tasks. Target language CSA CodeQL Tasks Share C/C++1590159 53.0% Java03030 10.0% Python05151 17.0% JavaScript03030 10.0% Go03030 10.0% Total159141 300 100.0% D.3 Complete CWE Label Distribution Table 8 lists all 85 CWEs and the separate Other group. The source retains original task annotations, supplements three previously unlabeled tasks with explicit CWE labels from saved public CVE records, and assigns Other to the 14 tasks for which no explicit CWE label was found. Labels are deduplicated within each task. Of the 300 tasks, 272 have one CWE label, 14 have two, and 14 have none; 286 tasks therefore have an explicit CWE label. The table’s Other row contains the 14 tasks without an explicit CWE label. Each task contributes once to each of its labels. Table percentages use all 300 tasks, so rows can overlap and their shares sum to 104.7% including Other. Figure 4(c) instead partitions the 314 task–label pairs into pie sectors. Its five leading CWEs are shown with their names. The pie’s Other sector combines 213 pairs from the remaining 80 CWEs with the 14 unlabeled tasks, giving 227 pairs (72.3%). Ties are ordered by ascending CWE number. The CWEs comprise 82 Weakness entries and three upstream Category entries (CWE-189, CWE-264, and CWE-399), as recorded in the supplied CWE v4.20 catalog [45]. Category annotations are retained without remapping them to more specific weaknesses; CWE-264 is marked obsolete in that catalog. CSA and CodeQL columns report task counts for the respective static analysis frameworks. Table 8 Complete distribution of 85 CWEs and Other. Each task counts once per distinct label; shares use all 300 tasks. Multiple-label tasks appear in more than one row. † marks an upstream Category entry. CWENameCSA CodeQL Tasks Share CWE-787Out-of-bounds Write251268.7% CWE-125Out-of-bounds Read147217.0% CWE-22Improper Limitation of a Pathname to a Restricted Directory (’Path Traversal’) 213155.0% CWE-476NULL Pointer Dereference130134.3% CWE-79Improper Neutralization of Input During Web Page Generation (’Cross-site Scripting’) 012124.0% CWE-362Concurrent Execution using Shared Resource with Improper Synchronization (’Race Condition’) 93124.0% Continued on next page 27 Table 8 continued CWENameCSA CodeQL Tasks Share CWE-416Use After Free120124.0% CWE-400Uncontrolled Resource Consumption28103.3% CWE-908Use of Uninitialized Resource9093.0% CWE-667Improper Locking8082.7% CWE-190Integer Overflow or Wraparound6172.3% CWE-415Double Free7072.3% CWE-20Improper Input Validation1562.0% CWE-611Improper Restriction of XML External Entity Reference0662.0% CWE-617Reachable Assertion6062.0% CWE-119Improper Restriction of Operations within the Bounds of a Memory Buffer 5051.7% CWE-191Integer Underflow (Wrap or Wraparound)5051.7% CWE-200Exposure of Sensitive Information to an Unauthorized Actor0551.7% CWE-377Insecure Temporary File0551.7% CWE-401Missing Release of Memory after Effective Lifetime5051.7% CWE-601URL Redirection to Untrusted Site (’Open Redirect’)0551.7% CWE-770Allocation of Resources Without Limits or Throttling3251.7% CWE-78Improper Neutralization of Special Elements used in an OS Command (’OS Command Injection’) 0441.3% CWE-129Improper Validation of Array Index3141.3% CWE-674Uncontrolled Recursion3141.3% CWE-94Improper Control of Generation of Code (’Code Injection’)0331.0% CWE-189 † Numeric Errors1231.0% CWE-287Improper Authentication0331.0% CWE-326Inadequate Encryption Strength0331.0% CWE-1333Inefficient Regular Expression Complexity0331.0% CWE-23Relative Path Traversal0220.7% CWE-59Improper Link Resolution Before File Access (’Link Following’) 0220.7% CWE-74Improper Neutralization of Special Elements in Output Used by a Downstream Component (’Injection’) 0220.7% CWE-77Improper Neutralization of Special Elements used in a Command (’Command Injection’) 0220.7% CWE-284Improper Access Control0220.7% CWE-338Use of Cryptographically Weak Pseudo-Random Number Generator (PRNG) 0220.7% CWE-352Cross-Site Request Forgery (CSRF)0220.7% CWE-502Deserialization of Untrusted Data0220.7% CWE-532Insertion of Sensitive Information into Log File0220.7% CWE-672Operation on a Resource after Expiration or Release2020.7% CWE-772Missing Release of Resource after Effective Lifetime2020.7% CWE-835Loop with Unreachable Exit Condition (’Infinite Loop’)2020.7% CWE-863Incorrect Authorization0220.7% CWE-29Path Traversal: ’\.. ’0110.3% CWE-88Improper Neutralization of Argument Delimiters in a Command (’Argument Injection’) 0110.3% CWE-91XML Injection (aka Blind XPath Injection)0110.3% CWE-120Buffer Copy without Checking Size of Input (’Classic Buffer Overflow’) 1010.3% CWE-122Heap-based Buffer Overflow1010.3% CWE-131Incorrect Calculation of Buffer Size1010.3% CWE-193Off-by-one Error1010.3% CWE-201Insertion of Sensitive Information Into Sent Data0110.3% CWE-203Observable Discrepancy1010.3% CWE-209Generation of Error Message Containing Sensitive Information0110.3% CWE-250Execution with Unnecessary Privileges0110.3% CWE-256Plaintext Storage of a Password0110.3% CWE-264 † Permissions, Privileges, and Access Controls0110.3% CWE-266Incorrect Privilege Assignment0110.3% CWE-295Improper Certificate Validation0110.3% CWE-312Cleartext Storage of Sensitive Information0110.3% CWE-327Use of a Broken or Risky Cryptographic Algorithm0110.3% CWE-331Insufficient Entropy0110.3% CWE-347Improper Verification of Cryptographic Signature0110.3% CWE-364Signal Handler Race Condition1010.3% CWE-366Race Condition within a Thread0110.3% CWE-369Divide By Zero1010.3% CWE-399 † Resource Management Errors1010.3% CWE-407Inefficient Algorithmic Complexity0110.3% CWE-459Incomplete Cleanup1010.3% CWE-460Improper Cleanup on Thrown Exception0110.3% Continued on next page 28 Table 8 continued CWENameCSA CodeQL Tasks Share CWE-522Insufficiently Protected Credentials0110.3% CWE-538Insertion of Sensitive Information into Externally-Accessible File or Directory 0110.3% CWE-552Files or Directories Accessible to External Parties0110.3% CWE-613Insufficient Session Expiration0110.3% CWE-639Authorization Bypass Through User-Controlled Key0110.3% CWE-668Exposure of Resource to Wrong Sphere0110.3% CWE-670Always-Incorrect Control Flow Implementation0110.3% CWE-680Integer Overflow to Buffer Overflow1010.3% CWE-692Incomplete Denylist to Cross-Site Scripting0110.3% CWE-789Memory Allocation with Excessive Size Value0110.3% CWE-826Premature Release of Resource During Expected Lifetime1010.3% CWE-843Access of Resource Using Incompatible Type (’Type Confusion’) 1010.3% CWE-862Missing Authorization0110.3% CWE-918Server-Side Request Forgery (SSRF)0110.3% CWE-1188Initialization of a Resource with an Insecure Default0110.3% CWE-1336Improper Neutralization of Special Elements Used in a Template Engine 0110.3% OtherOther (no explicit CWE found)86144.7% Sum across label rows165149 314 104.7% D.4 Calibration Horizon and Difficulty Distributions Table 9 expands the horizon and difficulty statistics reported in Section 3.2. The minimum horizon is a curation constraint: only accepted calibration trajectories with at least 40 interaction steps are eligible for the final 300-instance sample. Difficulty scores are used by the dataset-selection pipeline. The median horizon is 60 steps and the maximum is 123. The associated difficulty score has median 0.684 and range 0.414–0.846. These retained construction-time fields use the original calibration metadata. Table 9 Interaction-horizon and difficulty distributions of the calibration trajectories. Interval Count Share Interaction steps [40, 60)14448.0% [60, 80)10434.7% [80,∞)5217.3% Difficulty score [0.0, 0.5)3210.7% [0.5, 0.6)7424.7% [0.6, 0.7)5919.7% [0.7, 0.8)13043.3% [0.8, 1.0)51.7% Table 10 reports every one-step bin underlying the horizon intervals above, including zero-count bins. These counts use construction-time interaction-step metadata; the message-level pilot counts follow below. Table 10 Complete one-step frequency counts in the retained construction-time interaction-step metadata. Read down each three-column block. All 84 bins, including zero-count bins, are shown; shares use 300 trajectories. Steps Traces Share Steps Traces Share Steps Traces Share 404 1.3%682 0.7%962 0.7% 414 1.3%697 2.3%970 0.0% 424 1.3%708 2.7%981 0.3% 433 1.0%715 1.7%991 0.3% 442 0.7%725 1.7%1000 0.0% 456 2.0%736 2.0%1011 0.3% 467 2.3%745 1.7%1020 0.0% 478 2.7%755 1.7%1030 0.0% 486 2.0%765 1.7%1040 0.0% 4911 3.7%776 2.0%1052 0.7% 503 1.0%781 0.3%1060 0.0% 5112 4.0%794 1.3%1071 0.3% 527 2.3%801 0.3%1081 0.3% 538 2.7%811 0.3%1090 0.0% 5411 3.7%829 3.0%1100 0.0% Continued on next page 29 Table 10 continued Steps Traces Share Steps Traces Share Steps Traces Share 556 2.0%832 0.7%1111 0.3% 5613 4.3%843 1.0%1120 0.0% 579 3.0%851 0.3%1130 0.0% 5812 4.0%863 1.0%1140 0.0% 598 2.7%873 1.0%1150 0.0% 607 2.3%883 1.0%1161 0.3% 618 2.7%894 1.3%1170 0.0% 624 1.3%904 1.3%1180 0.0% 635 1.7%910 0.0%1190 0.0% 643 1.0%921 0.3%1200 0.0% 655 1.7%931 0.3%1211 0.3% 666 2.0%942 0.7%1220 0.0% 677 2.3%951 0.3%1231 0.3% D.5 Pilot Trajectory Statistics Figure 11 summarizes the original pilot traces for the 300 selected tasks (159 CSA and 141 CodeQL), with one admitted trajectory per task. Selection follows the original checker-admission records, including 13 trajectories whose checkers passed admission while their agent runs ended at the spending limit. These statistics describe the construction pilot; independent main-evaluation runs follow the final-pass criteria in Section 4. 20406080100120140 Assistant turns per trajectory 0 20 40 60 Number of trajectories (a) Assistant turns Mean = 58.43 255075100125150 Tool calls per trajectory 0 25 50 75 Number of trajectories (b) Tool calls Mean = 72.73 Figure 11 Distributions of (a) assistant turns and (b) tool calls across 300 admitted pilot trajectories in Checker- Bench. Histogram bins have width 10. Dashed red lines mark sample means; solid red curves show Gaussian kernel density estimates with Scott’s bandwidth, scaled to histogram counts. An assistant turn is a unique assistant message, deduplicated by session and message identifier. A tool call is a unique tool-use invocation, deduplicated by session and invocation identifier; failed calls and calls without a matching result are included. The tool-call total includes explicit skill invocations. Assistant-turn counts are computed from message events and use a different definition from the construction-time interaction-step metadata in Table 9 and the runner’s native turn counter. Table 11 gives all histogram bins for both pilot metrics. The two columns each account for the same 300 admitted pilot trajectories under their respective event-counting definitions. 30 Table 11 Complete bin counts for the pilot distributions in Figure 11. Intervals are left-inclusive and right-exclusive. Each share uses 300 pilot trajectories. Interval Assistant turns Share Tool calls Share [20, 30)186.0%10.3% [30, 40)38 12.7%82.7% [40, 50)57 19.0%237.7% [50, 60)58 19.3%51 17.0% [60, 70)48 16.0%72 24.0% [70, 80)289.3%46 15.3% [80, 90)258.3%37 12.3% [90, 100)175.7%268.7% [100, 110)72.3%165.3% [110, 120)20.7%124.0% [120, 130)10.3%51.7% [130, 140)10.3%10.3% [140, 150)00.0%10.3% [150, 160)00.0%10.3% Total300 100.0%300 100.0% D.6 CVE Publication-Time Distribution Table 12 gives every publication year, including empty years. Counts use the 300 task records, so separable tasks sharing a CVE contribute separately; uncertainty estimates for temporal performance keep those tasks grouped. The 2026 bin is observed through the September 15 metadata snapshot. The complete dating and boundary policies are in Appendix C.1. Table 12 CVE publication years for the fixed 300 tasks. Dates use the canonical CVE record; the CVE identifier year is not used. Publication year CSA CodeQL Tasks Share 20100110.3% 20111010.3% 20120000.0% 20131120.7% 20142241.3% 20150000.0% 20162020.7% 201773103.3% 20185272.3% 201956113.7% 2020913227.3% 2021622289.3% 2022424289.3% 202392837 12.3% 2024441660 20.0% 2025431457 19.0% 202621930 10.0% Total159141300 100.0% E Evaluation and Inference Protocol E.1 System Configurations and Reproducibility The seven evaluated models areGPT-5.6-Sol [49],Claude Opus 4.8 [6],Qwen3.8-Max [2], DeepSeek-V4-Pro-0813 [18],GLM-5.2 [70],Kimi-K3 [33], andHY4-Preview [56]. We record har- ness and model versions, context and decoding settings, reasoning effort, and enabled tools. All systems share pinned repositories, toolchains, containers, resource limits, and the verifier. Released artifacts record 31 prompts, skills, tool schemas, commands, workspace diffs, diagnostics, termination reasons, and submissions. E.2 Task Admission and Independent Repeats The evaluation manifest fixes 300 environments (159 CSA and 141 CodeQL), three harnesses, seven models, and three independent generation repeats per environment–harness–model combination. All three repeats cover the same 300 tasks for each of the 21 harness–model pairs, yielding 6,300 logical runs per repeat and 18,900 in total. The reported results are the arithmetic mean of the three repeat-level estimates, with equal weight for each repeat. Recovery attempts are tracked separately. A held-out reference checker is used to verify environment usability before runs are admitted. Only environment–model combinations that pass preflight are released, and blocked combinations remain visible in the coverage ledger. The reference checker is never part of the synthesis inputs. Each independent repeat uses a separate workspace and a 50-minute generation budget. It receives the task instruction, patch, relevant code context, tools, and enabled skills. Visible compilation and analysis feedback can support multiple inspect–write–compile–repair cycles within this budget. The final source is checker.cpp for CSA or CodeQL query source. Generation completion status is recorded independently of downstream verification. E.3 Source and Evidence Freezing At termination, the ledger records the submitted source, SHA-256 digest, complete trajectory, tool-call events, explicit skill invocations, elapsed time, usage, and termination reason. Task, harness, model, container, tool, and scorer versions are retained with independent-repeat and recovery-attempt indices. Subsequent scans and judges consume this exact candidate. Compiled artifacts can be regenerated from it, but verification recovery and a valid low quality score cannot replace its source. Raw evidence is retained alongside any normalized summaries and referenced by path and content hash. E.4 Backend Execution The reference execution image pins LLVM/Clang 18 and mounts the task workspace at /work. Checker source is compiled with /opt/llvm/bin/clang++ in C++17 mode as a position-independent shared object using -fPIC, -shared, -fno-rtti, -fno-exceptions, the pinned LLVM include directory, and -Wl,–allow-shlib-undefined. The resulting object is a CSA plugin, not a standalone executable. Each scan invokes /opt/llvm/bin/clang –analyze, loads the plugin through the analyzer, enables its registered checker name, and appends the translated compilation-database flags and source file. Diagnostics are at- tributed to the checker through its registered diagnostic label. The task-local validator and the held-out verifier call the same generated scripts, but the verifier rebuilds from source in a clean workspace. For CodeQL, the verifier independently compiles and executes the submitted query using the task’s pinned CodeQL environment and revision-specific analysis inputs. It enforces the frozen CodeQL rules for target and patch relevance, vulnerable–fixed differential behavior, query scope, diagnostic quality, and hardcoding. CSA-specific compiler commands, report signatures, count limits, and plugin gates are not reused as CodeQL acceptance rules. The backend and rule version accompany every reported gate result. E.5 Frozen Mainline Scan and Single-Judge Decisions After paired-revision verification, the same checker is scanned over a broader mainline scope. The mainline code is distinct from the vulnerable and fixed task revisions. At most ten warnings requiring judgment are sampled; the selected warning IDs and supporting code context are retained. Appendix E.6 specifies the selection rule, recorded warning counts, and the distinction between the sampled gate and aggregate precision. Claude Opus 5 [7] performs false-positive triage and, in separate stages, assesses process quality and vulnerability semantics. All stages use this one judge model and record the exact model configuration and rubric version. Judge decisions cannot override a failed executable or risk gate. Each stage commits its first valid decision, including valid low scores and negative verdicts. Only unavailable or invalid evidence follows recovery; quality-based resampling is prohibited. Mainline “refinement” is an assessment of the frozen candidate, with no follow-up edits by the synthesis model. Empty warning sets 32 Table 13 Recorded warning volume in the historical repeat-1 cohort of passed checkers. Counts precede the ten-alert triage selection. Backend Checkers Zero alerts > 10 alerts Recorded alerts Sampled alerts CSA55034811880596 CodeQL1,3901,2185455382 Total1,9401,566161,335978 and missing or invalid judgments follow each backend’s frozen rules; an undefined sample statistic is not fabricated. E.6 Mainline Alert Selection and Sensitivity Selection unit and ordering. The frozen implementation selects individual alert records from each checker’s saved mainline output. Selection is deterministic conditional on that list; it is not uniform random sampling over alerts, files, or alert types, and the selector does not rank by severity. All records are judged when the list has at most ten entries. For a longer list of length N, the zero-based selected indices are I CSA =k⌊N/10⌋ : k = 0,..., 9, I CodeQL =round(k(N − 1)/9) : k = 0,..., 9. (1) CSA retains checker-tagged warnings and deduplicates identical path–line–column locations within each compiler invocation. Its collected list follows scan-worker completion order, so a fresh scan need not reproduce the same ordering. In particular, for 11–19 CSA warnings the rule takes the first ten. CodeQL uses the normalized finding order and first persists at most 100 findings; its ten-alert selector operates on that retained list. Neither backend allocates a fixed quota per file or alert type. Replaying a sample therefore requires the saved list and selected warning identities. Warning volume before triage. Table 13 reports the archived audit of 1,940 passed checkers from retained repeat 1, covering all 21 harness–model combinations. This historical cohort predates the subsequent aggre- gate corrections to Table 2; it is not a three-repeat total. Of its 1,335 recorded alerts, 978 were initially sampled and 357 required supplementary judgment. Only 16 checkers had more than ten recorded alerts; 1,566 had none. These counts cover the passed cohort, not all generated checkers. CSA stops collecting after a compiler invocation brings the collected total to at least 100 warnings, and one scan in this cohort reached that cap. Consequently, recorded counts do not establish the number of alerts an uncapped scan of the entire repository would produce. Alert-weighted and checker-weighted precision. The ten-alert sample supplies a per-checker acceptance gate. The reported FP pass instead pools recorded alerts from passed checkers, including supplementary judgments beyond that sample. For a fixed system and repeat, let C + contain those checkers with N i = TP i + FP i > 0. The corresponding alert-weighted and checker-weighted precision percentages are P micro = 100 P i∈C + TP i P i∈C + N i , P macro = 100 |C + | X i∈C + TP i N i .(2) Thus FP pass = 100− P micro ; larger alert sets receive more weight in this metric. P macro gives every alert- bearing checker equal weight. Zero-alert checkers are reported separately and enter neither precision denom- inator; a clean scan does not establish perfect precision. Observed sensitivity in the archived cohort. We recompute both estimators using the same post hoc Opus 5 audit of all 1,335 saved alerts. It contains 1,063 TP and 272 FP labels. We average the seven system-level estimates equally within each harness. The precision ordering changes fromOpenCode, Claude Code,Hermes Agent under alert weighting toClaude Code,OpenCode,Hermes Agent 33 under checker weighting. WithinClaude Code, the precision leader changes fromGPT-5.6 (93.33% micro) toHY4-Preview (96.11% macro). These are conditional precision rankings, distinct from Pass@1 rankings. Adding the all-recorded-alert gate removes 81 of the 1,940 original passes. The harness ordering by Pass@1 remains unchanged, but the leading harness–model pair changes from Claude Code/Kimi-K3 (122/300 originally) toOpenCode/DeepSeek-V4-Pro (119/300 after the added gate). The archived sampled gates and supplementary alert audit both used Claude Opus 5 judgments. The additional gate expands coverage from the ten-alert sample to all recorded alerts while retaining the same judge. Moreover, this is a passed-only audit of saved output: it does not re-adjudicate originally failed checkers or recover alerts beyond scan caps. A representative-subset comparison would need a common task subset spanning repositories and backends, include checkers reaching the FP stage regardless of their original verdict, and compare the sampled and complete outputs under the same judge and rubric. That analysis is not established by this audit; stability of the current full-evaluation rankings under exhaustive adjudication remains unverified. E.7 Failure Accounting, Recovery, and Aggregation Final passed requires normal generation completion and every backend-specific executable, mainline, process, semantic, mechanism, scope, and anti-hardcoding gate. Valid generation failures, refusals, and candidate code errors are failures. Infrastructure failures permit at most two additional attempts under the recovery policy. When source already exists, verification recovery reuses it and preserves prior valid stage decisions. Each additional attempt is attached to its original logical run and does not contribute an additional independent value to the reported estimate. Unresolved infrastructure outcomes are recorded separately from valid model failures, and evaluation coverage is reported with the scheduled manifest. Incomplete repeat groups are reported as incomplete coverage; they are not silently treated as completed failures or removed from the scheduled task set. For each harness–model pair, we compute Pass@1 as the percentage of the 300 tasks whose checker passes the full evaluation in each independent repeat. We report the arithmetic mean of these three rates, giving each repeat equal weight (Section 4). Each independent run contributes its own final verdict; a successful repeat does not replace a failed repeat. Final-success efficiency summaries average interaction turns, tool calls, and explicit skill calls over passed trajectories in that repeat, counting recorded invocation events including subagent activity. The conditional means from the three repeats are averaged equally. Each repeat’s passed-trajectory count is retained. If a retained repeat has no successful trajectory, its conditional efficiency and the aggregate efficiency mean are N/A, not zero. Full failure and recovery records remain archived for audit and resource accounting. E.8 Statistical Reporting The reported main comparison, time-budget levels, and task subsets average the same three independent repeats equally. The separate ablation study uses one generation per task for each new variant and reuses the original Full cohort (Appendix H.2). The subgroup figure and tables report descriptive estimates without significance claims. Ablation intervals resample paired task IDs jointly across configurations; they reflect task composition and do not estimate uncertainty over repeated model generations or independent CVE clusters. Valid-score sample sizes and backend-specific breakdowns accompany diagnostic means; passed-trajectory counts accompany efficiency summaries. Build and Diff. SR use all 300 tasks as their denominator. Process and Semantic average available valid scores on their respective scales. For passed checkers, each recorded mainline alert is labeled TP or FP. Within each repeat, FP pass pools these alerts as 100 P FP/ P (TP + FP). The reported FP pass and #WN are equal-weighted means of the three repeat-level proportions and warning counts, respectively. Checkers with no alerts contribute no denominator. One retained mainline scan reached its warning cap. Judge labels are not human ground truth. The Average and Overall Average rows in Table 2 are unweighted means of the reported metric values over the seven models within each harness and all 21 harness–model combinations, respectively. 34 E.9 Human Validation of the LLM Judge We validateClaude Opus 5’s false-positive triage, process-quality assessment, and vulnerability-semantic assessment against independent human annotations under the backend-specific rubrics in Appendix F. The three annotators have backgrounds in computer science graduate study and software development. Sampling and judgment units. We sampled 100 of the 159 CSA tasks and 100 of the 141 CodeQL tasks. The CodeQL allocation comprised 25 tasks each for Java, Python, JavaScript, and Go. The task allocation followed these backend and language strata, with one frozen checker submission per task and no score- boundary or outcome oversampling. Each of the 200 submissions received one process and one semantic judgment. For each backend, 80 of the 100 sampled tasks contributed 100 individual mainline warnings: 60 tasks contributed one warning each, and 20 contributed two each. The remaining 20 tasks contributed none. This yielded 200 warning judgments, 400 submission-stage judgments, and 600 judge–human pairs. No item was excluded or weighted. Model, harness, repeat, and pass/fail status were retained for audit, without allocation quotas. Human labels and reference. All three annotators labeled every item independently, giving 1,800 initial human labels. Their view contained the warning and source context for false-positive triage; checker source and trajectory evidence for process quality; and the fixing patch plus vulnerable and fixed code for semantic quality. Judge labels and rationales, model and harness identities, and final verdicts were hidden during initial labeling. A majority of the three binary labels defined the human reference. For process and semantics, the binary label records whether the score and applicable discrete gates meet the backend’s rubric: process at least 80/100, CSA semantics at least 7/10, and CodeQL semantics at least 8/10. The table covers these stage decisions; numeric-score error and individual rubric-criterion agreement are outside its scope. Label distribution and agreement. Table 14 gives separate judge and majority-human positive counts. Pos- itive means a false positive at the warning stage and an accepted decision at the process and semantic stages. Each row contains 100 valid pairs. Judge–human agreement uses unweighted Cohen’s κ; human–human agreement uses three-rater Fleiss’ κ on the original independent binary labels. Exact matches range from 89 to 95 per row. The 2× 2 cells in table order are (21, 3, 2, 74), (23, 2, 5, 70), (67, 3, 5, 25), (76, 3, 2, 19), (59, 7, 4, 30), and (65, 5, 3, 27), with cells ordered as judge-positive/human-positive, judge-positive/human- negative, judge-negative/human-positive, and judge-negative/human-negative. The minority vote is positive on (3, 5, 4, 3, 7, 5) split items in table order; the remaining split items have a negative minority vote. Table 14 Human validation of the three categorical judge stages, with 100 items per row. J+/H+ gives judge and majority-human positive counts; exact agreement is a count. Split gives items with a nonunanimous initial human vote. Backend StageJ+/H+ Exact J–H κ H–H κ Split CSAFP triage24/23950.861 0.8677 CodeQL FP triage25/28930.821 0.83510 CSAProcess70/72920.806 0.8529 CodeQL Process79/78950.852 0.8836 CSASemantic66/63890.760 0.80014 CodeQL Semantic70/68920.813 0.83211 Pooled All stages 334/332 556 0.852 0.872 57 Adjudication. The annotation set has 57 nonunanimous human items. Majority voting resolved all 57; no item was tied or removed from the denominator. The majority label changed only the human audit reference and left benchmark verdicts intact. Fleiss’ κ uses the 1,800 initial labels before majority resolution. 35 CSA versus CodeQL and interpretation of κ = 0.85. Across the six rows, the 556 matching labels among 600 pairs give exact agreement of 0.9267. The judge-positive total is 334 and the human-positive total is 332, so chance agreement is 0.5060 and pooled Cohen’s κ = (0.9267− 0.5060)/(1− 0.5060) = 0.852 to three decimal places. These calculations give the pooled κ = 0.85 reported in the main text after rounding to two decimal places. Reproducing the CSA and CodeQL results requires the frozen task IDs, checker and warning IDs, judge labels, all three independent human labels, and the adjudicated references. Task-clustered uncertainty intervals likewise require those records. F Post-Synthesis Verifier Rubric CSA and CodeQL retain separate frozen executable and scoring policies. Their common final requirements are normal generation completion, independent executable evidence, mainline false-positive control, process and semantic quality, and backend-specific mechanism, scope, and anti-hardcoding checks. Figure 5 gives the numerical thresholds. The CSA implementation details below describe the post-synthesis-v4 rubric; CodeQL retains its own backend requirements. F.1 Independent Replay and Diagnostic Predicates Before collecting trajectories, we instantiate and freeze each task’s verifier V i from its recovered revision pair and pinned environment E i . Given checker source q i , the verifier ignores agent-produced binaries, rebuilds or compiles the candidate in a clean workspace, and scans the task-specific scope in c − i and c + i . CSA uses a loadable checker plugin; CodeQL uses submitted query source in its pinned analysis environment. Version- pinned scripts preserve each backend’s context, while isolation prevents artifact leakage. Replay produces diagnostic multisets W − i (q i ) and W + i (q i ), with N ± i (q i ) = |W ± i (q i )|. Executable validity combines three binary indicators for build success, diagnostic contrast, and patch relevance: Valid i (q i ) = B i (q i )C i (q i )L i (q i ), B i (q i ) = 1[Build α i (q i ,E i )], C i (q i ) =C α i ,i (W − i ,W + i ), L i (q i ) =L α i ,i (W − i ,W + i , ∆ i ). (3) Here,C α i ,i is the frozen differential-detection predicate andL α i ,i is its target/patch-relevance predicate. CSA contrast requires N − i > N + i and N + i < T res , with T res = 50 fixed before collection. Reports are matched using the line-insensitive signature κ i (w) = (checker(w), file(w), func(w), msg(w)). At least one disappearing report must lie in a patch-modified function. CodeQL applies its own frozen query, differential, and relevance rules; CSA’s count and matching predicates are not substituted for them. Replay retains normalized locations, diff-hunk distances, residual diagnostics, and in- and out-of-patch reports as evidence. Let O i denote the completed objective gate, including required artifacts, independent replay, and the back- end’s scope and diagnostic-quality checks. CSA uses the operational G6 gate below; CodeQL uses its own frozen objective gate. In both cases, O i ⇒ Valid i (q i ) = 1. Executable validity, process quality, semantic fidelity, and mainline false-positive control are non-compensating requirements. F.2 CSA Deterministic Objective Gate This gate invokes no LLM. The source and rebuilt plugin are checker.cpp and checker.so. The full frozen CSA contrast and localization predicates must hold in addition to the recorded gate evidence. G0. The task work directory and interaction trace exist. G1. The checker source exists and contains at least 50 bytes. G2. Independent compilation produces the loadable analyzer artifact. 36 G3. verify_result.json exists and the vulnerable revision produces at least one diagnostic, n buggy > 0. G4. Vulnerable and fixed diagnostics satisfy the frozen CSA contrast rule; equal positive counts are rejected. G5. Patch relevance is at least 0.5 and diagnostic sparsity is at least 0.3. G6. All G0–G5 requirements pass. Only G6 makes a run eligible for a nonzero CSA process score. Four additional objective diagnostics— Differential Detection (15 points), Diagnostic Sharpness (15), False-Positive and Scope Control (15), and basic CSA Implementation Quality (4)—are reported for diagnosis. These 49 points are excluded from S p to avoid counting executability again as process quality. F.3 CSA Process and Semantic Quality The process stage examines the patch, vulnerable and fixed functions, state.json, scope statement, frozen source, synthesis summary, and validation evidence. Its raw score Q i comprises Patch Semantic Understand- ing (12 points), Detection Plan Quality (12), Plan-to-Code Consistency (12), and Validation Quality (6), for Q max = 42. Repeated reads, redundant commands, excessive edits, and non-progressing build loops incur a waste penalty p waste i ∈ [0, 5]. The process floor is 80/100. The semantic stage first infers the root cause, trigger and safe conditions, defect class, propagation require- ments, and minimum analyzer mechanism from the patch and vulnerable functions. It then compares the frozen checker with those requirements. The ordered mechanism ladder is syntax-only→ call-match→ path-sensitive→ dataflow→ cross-procedure. Mechanism adequacy is classified as adequate, partial, inadequate, or overkill; semantic correctness as yes, partial, or no. For ten-point correctness and mechanism-fit subscores s c,i ,s m,i , the process and semantic scores are S p,i = 1[O i ] 100 max(0,Q i − p waste i ) Q max , S s,i = 0.6s c,i + 0.4s m,i .(4) The CSA semantic floor is 7/10. CSA additionally requires mechanism_adapted=adequate, semantic_correct in yes,partial, valid parsing, and no semantic-anchor risk. A patch-adjacent AST match may therefore fail semantic assessment despite satisfying a diagnostic contrast. F.4 CodeQL Quality Gates CodeQL follows the layered admission structure described above while retaining backend-specific executable, differential, relevance, scope, and hardcoding predicates. Its query is independently rebuilt and checked on the vulnerable and fixed revisions before scoring, and its refinement scan uses the shared mainline false-positive gate, with a default sampled FP cutoff of 25% in the main comparison. Queries that fail or incompletely satisfy these prerequisites are not scored on process or semantic quality. The CodeQL process-quality score is a 100-point score with a process floor of 80. Its dimensions are vul- nerability modeling (18 points), observable-evidence planning (14), mechanism selection (18), plan-to-code consistency (18), validation and recovery (14), efficiency and finalization (10), and generalization discipline (8). The semantic-quality score is a 10-point score in half-point increments with a floor of 8; it assesses the vulnerable sink, trigger and dataflow, safety barrier, CodeQL mechanism, and report-scope generalization. In addition to the numeric floor, automatic admission requires correct vulnerability and safety-barrier modeling, sound dataflow or control-flow support, acceptable scope generalization, an adequate mechanism, and no patch hardcoding or project-specific scope. These CodeQL-specific conditions refine the common final requirements without changing the non- compensating relation between executable validity, false-positive control, process quality, and semantic qual- ity. F.5 Mainline False Positives and Final Verdict The frozen candidate’s mainline scan and at most ten judged warnings provide false-positive evidence. In the main comparison, the sampled FP rate must be at most 25% for both CSA and CodeQL. For a nonempty, 37 validly judged sample, FP rate is the fraction classified as false positives. Sample composition, warning- level decisions, and the backend’s complete refinement outcome are stored. A backend-recognized clean scan follows its frozen empty-sample policy. Let F i denote normal generation completion; J i the required valid judge decisions; R i the mainline false- positive gate, including the backend’s full refinement rules; M i adequate mechanism fit; C sem i the required semantic-correctness classification; and H i ,A i hardcoding and semantic-anchor risks. With t CSA = 7 and t CodeQL = 8, the final verdict is Pass i = F i ∧ O i ∧ J i ∧ R i ∧ [S p,i ≥ 80]∧ [S s,i ≥ t α i ] ∧ M i ∧ C sem i ∧¬(H i ∨ A i ). (5) Section 5.6 varies only the cutoff within R i , replacing its default 25% value while retaining the saved warning samples, judge decisions, and all remaining acceptance requirements. The main evaluation uses Claude Opus 5 [7] for staged false-positive, process, and semantic judgments and retains each stage’s first valid decision. A valid low score is a failure, not a reason to ask again. A valid failed gate or low score yields a failed quality outcome; no new checker is requested. An unavailable judge or missing verification evidence cannot grant a pass and follows the bounded infrastructure-recovery rules, without discarding valid earlier decisions. Only a normally completed generation whose candidate satisfies every required gate receives final passed. Objective-only scoring cannot issue that final outcome. G Representative CSA Checker Patterns This appendix summarizes a selected set of implementation patterns from official LLVM 18 Clang Static An- alyzer (CSA) checkers [40]. The catalog applies only to the CSA-backed C/C++ subset of CheckerBench; tasks in other language ecosystems use the corresponding analyzer-native rule APIs. We retain patterns that expose distinct synthesis challenges rather than reproducing complete checker implementations. Table 15 Selected CSA checker patterns used as implementation references for C/C++ tasks, from stateless local checks to path-sensitive lifecycle analyses with persistent program state. Pattern / official checkerCallbacks / stateSynthesis challenge Constraint violation DivZeroChecker PreStmt State: none Split feasible states and report only a defi- nitely zero denominator. Undefined value UndefResultChecker PostStmt State: none Identify the undefined operand and retain useful diagnostic provenance. Resource lifecycle SimpleStreamChecker PostCall, PreCall, DeadSymbols, PointerEscape State: symbol-to-state map Track opened and closed resources, double close, escape, and leaks. Nullability propagation NullabilityChecker Binding, calls, returns, locations, assumptions, and events State: two maps and an invariant flag Propagate annotations and constraints across multiple semantic events. Memory management MallocChecker Calls, locations, dead symbols, es- capes, and assumptions State: region-state map plus auxil- iary maps and sets Model allocation families, deallocation com- patibility, leaks, double free, and use after free. Stateless constraint checks. DivZeroChecker and UndefResultChecker illustrate the smallest useful checker structures. They register a single statement callback, query the symbolic value already maintained by CSA, and emit a report without introducing persistent state. The former uses a dual feasibility assumption to separate zero and nonzero denominators; the latter inspects an undefined result after expression evaluation and attributes it to the relevant operand. These patterns test whether an agent can select the correct callback phase and translate a local defect predicate into analyzer constraints. 38 Finite-state resource tracking. SimpleStreamChecker represents the minimal lifecycle pattern. A program- state map associates each resource symbol with an opened or closed state. Post-call handling records acquisi- tion, pre-call handling validates and updates release, dead-symbol handling reports leaks, and pointer-escape handling stops tracking resources whose ownership can no longer be established. This pattern requires the synthesized checker to coordinate several callbacks while preserving a compact state machine. Multi-callback semantic propagation. NullabilityChecker and MallocChecker represent the complex end of the catalog. They combine call, statement, location, assumption, liveness, and escape events with multiple state traits. Their purpose here is not to provide templates for direct copying, but to expose the design deci- sions required for repository-level synthesis: choosing semantic events, defining stable state keys, suppressing infeasible or duplicate reports, and attaching path evidence to diagnostics. Common reporting contract. Across the selected patterns, a checker obtains symbolic values from the current analysis state, creates a sink or non-fatal error node as appropriate, constructs a path-sensitive bug report, marks relevant symbols and source ranges, optionally installs a visitor for path notes, and finally emits the report. CheckerBench evaluates the resulting behavior through independent rebuilding and vulnerable–fixed diagnostic contrast rather than by matching these reference implementations. H Additional Experimental Results H.1 KNighter Comparison Details Tasks and comparison. The comparison in Section 5.3 uses the 61 Linux-kernel tasks fromKNighter’s original benchmark [67]. Each task takes a bug-fixing commit as input and requires a CSA checker that distinguishes buggy code from its patched version. We evaluate our OpenCode-based method,KNighter, and our method without skills, all usingDeepSeek-V4-Pro. Table 3 reports valid-checker counts and rates. Time budget. All three configurations use the same 50-minute (3,000-second) active generation budget per task. The budget covers model responses, tool execution, and checker-repair iterations across the complete workflow, including KNighter’s internal agent coordination. Recorded admission queues and maintenance pauses, as well as subsequent independent verification, are excluded under the timing convention in Ap- pendix H.3. The submitted checker is frozen when generation terminates. Accessible documentation. Our method andKNighter receive the same fixing patch, source-code context, repository revisions, and pinned CSA environment. Both can consult the same read-only Linux/CSA docu- mentation, analyzer API references, and official checker examples described in Appendix B.3. These shared resources exclude held-out reference checkers and task-specific checker solutions. Skill library. Both systems have access to the same reusable skill and strategy content for patch interpre- tation, checker design, build recovery, and validation. The content is exposed through each system’s native interface. The w/o skills variant of our method removes these skill packages and strategy references while retaining the same task inputs, factual API documentation, tools, and execution budget. Task prompts. The task-facing instructions standardize the synthesis objective, supplied patch and code context, available resources, required checker artifact, and validation requirements. System-specific role prompts and internal orchestration retain each harness’s workflow, while the task information and reusable knowledge made available to our method and KNighter are held constant. Validation criteria. All three configurations use the same independent verifier, scan scopes, and acceptance predicates. The verifier rebuilds each frozen checker from source in the pinned CSA environment and applies the same build-success, vulnerable–fixed diagnostic-contrast, and patch-relevance checks (Equation 3). Agent- reported success does not determine the verdict. Table 3 reports the number of valid checkers and the corresponding percentage over the same 61 tasks. 39 Table 16 Artifact pass and differences from Full. All rates use the full task denominator; CSA and CodeQL use 159 and 141 tasks. Intervals use 10,000 paired task bootstrap samples. ConfigurationAll (%)∆ p [95% CI]CSA (%)CodeQL (%) Full48.67Reference44.0353.90 w/o Skills18.67−30.00 [−36.33,−23.67]15.7221.99 w/o Execution Feedback20.33−28.33 [−34.67,−22.00]17.6123.40 w/o Harness1.33−47.33 [−53.00,−41.33]0.002.84 H.2 Ablation Outcomes and Coverage Each configuration covers the same 300 tasks: 159 CSA and 141 CodeQL tasks. The three new variants have 900 frozen candidates in total; Full reuses the original 300 candidates. We independently evaluated the 21 historical artifacts whose assessment had stopped at the generation deadline. All 21 fail an artifact requirement, including three with no submitted source; Full therefore has 146/300 artifact passes and 119/300 full passes (39.67%). No candidate was regenerated or edited during this completion. Configurations and generation. Each new configuration uses one generation per task. w/o Skills jointly removes skill packages and strategy references while retaining task inputs, repository tools, and factual API documentation. w/o Execution Feedback retains skills but returns neutral receipts for build and analyzer calls; actual outputs remain outside the agent workspace. Both use OpenCode with a 3,000-second active budget. w/o Harness makes one model call without tools or skills, using the full patch and deterministically extracted code context. The new variants share the model route, sampling settings, and 16,384-token output limit per response; one-shot generation has no feedback or repair call. Endpoints and comparability. Artifact pass applies the compilation, differential, semantic, anti-hardcoding, and false-positive gates without requiring Process or normal agent termination. Frozen artifacts are evaluated even when generation reaches the time limit. The new runs retain their frozen Claude Opus 5 judgments for Process, Semantic, and task gates; a separateOpus 5 audit of all saved alerts supplies the reported FP pass without changing the original verdicts. Historical Full differs from the new runs in execution date, client context limit, and workspace/tool isolation, so the comparisons do not isolate causal effects of individual components. The one-shot result also reflects its strict output protocol: 237/300 requests yield invalid or incomplete structured submissions, and another 53 fail compilation. Using the displayed rates in Table 4, the Pass@1 differences from Full are −31.34, −26.34, and −39.34 percentage points, respectively. The artifact-pass intervals in Table 16 describe variation over paired tasks for one generation per task; they do not cover model resampling, shared-CVE dependence, or differences between historical and new execution conditions. Process/Semantic valid sample counts are 239/239 for Full, 119/122 for w/o Skills, 100/100 for w/o Execution Feedback, and 10/10 for w/o Harness. These conditional means are not averages over all 300 tasks. Three unavailable Process judgments are excluded instead of using heuristic fallback scores. Missing evidence is never treated as a pass. One failed candidate has seven false positives among ten sampled alerts, exceeding its 25% threshold. Opus 5 independently adjudicates every saved alert of the 25, 40, and 1 passed new checkers (16, 64, and 0 alerts, respectively). All 80 labels are resolved after source-evidence completion; the Full FP figure reuses the 78-alert Opus 5 audit from the main experiment. These post hoc labels define FP pass , not the original sampled FP gates. The false-positive counts are 2/78 (2.56%) for Full, 1/16 for w/o Skills, and 16/64 for w/o Execution Feedback. One-shot has no saved alerts, so its N/A entry does not denote a zero false- positive proportion. No-alert checkers add nothing to the alert denominator. The metric is a false-discovery proportion over saved alerts, not a repository-wide false-positive rate. Because the ablation variants have substantially lower pass rates and therefore fewer successful trajectories, Table 4 averages turns and tool calls over all 300 tasks, including failed trajectories. This differs from Table 2, which reports interaction metrics over successful trajectories only. Table 17 additionally restricts 40 Table 17 Interaction cost on tasks passed by both artifacts. Each pair uses its own common task subset; these means should not be compared across rows as if the subsets were identical. VariantTasksFull turnsVariant turnsFull toolsVariant tools w/o Skills4249.2964.0263.2189.24 w/o Execution Feedback4345.2163.1960.6384.09 w/o Harness322.001.0039.330.00 each comparison to tasks whose artifacts pass in both Full and the corresponding variant. H.3 Time-Budget Sensitivity Tables 18–20 provide the complete time-budget results for the analysis in Section 5.5. The tables cover all seven models and three harnesses at 10, 20, 30, 40, and 50 minutes of active generation time, using the completed-within-budget policy below. Timing and eligibility. We retrospectively apply the cutoffs to existing trajectories, retaining each task’s current evaluated attempt. Active generation time includes model responses and tool execution, but excludes recorded API and compute admission queues, maintenance pauses, and subsequent external verification and judging. Only final artifacts completed within a cutoff contribute their existing evaluation outcomes; unfinished runs count as unsuccessful for Pass@1, Build, and Diff. SR, with all 300 tasks retained in each denominator. Intermediate artifacts are not evaluated, and agents are not rerun with shorter deadlines. Realized execution time and trajectory length can differ from the budget because agents may terminate early, batch tool operations, or spend time waiting for responses and recovering from build failures. Conditional metrics and model differences. Process and Semantic average available valid scores from eligible completed runs; Avg. Turns and Avg. Tool Calls average eligible passed runs. FP pass pools recorded alerts from those passed runs, using the same adjudicated labels as the main table. These statistics describe changing subsets of completed runs, not within-trajectory quality improvements. Averaged across harnesses, the 40-to- 50-minute Pass@1 gains are largest forQwen3.8-Max (3.67 percentage points) andHY4-Preview (3.33), compared with 0.44 forGPT-5.6 and 3.00 forClaude Opus 4.8. These differences describe when the retained trajectories finish successfully. Table 18 Time-budget sensitivity on CheckerBench:OpenCode. Model Budget (min) Pass@1 (%) ↑Build (%) ↑Diff. SR (%) ↑FP pass (%) ↓Process ↑Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ GPT-5.61012.0025.3325.000.0084.578.26 23.5054.53 2023.3364.6758.670.0082.207.83 26.8957.60 3028.3385.0075.330.0080.967.75 28.4058.85 4031.3395.3383.000.0080.717.68 29.2260.06 5031.6798.3385.330.0080.227.68 29.1960.13 Claude Opus 4.8108.339.679.670.0088.289.04 26.2039.36 2024.0031.6731.330.0089.428.84 33.6050.25 3036.0053.6753.005.3688.118.51 39.4157.24 4042.3370.3369.6714.0487.078.35 42.7862.07 5045.3381.6781.0013.7986.428.32 44.7864.35 Qwen3.8-Max106.337.677.67—92.159.22 24.1131.68 2022.3333.3333.000.0091.728.69 31.1541.39 3029.3350.3350.000.0091.168.50 33.1844.00 4035.0063.0062.676.0090.588.49 34.8446.10 5037.3369.3369.005.7790.158.46 35.9647.54 DeepSeek-V4-Pro1013.3322.6721.670.0089.988.40 34.2547.88 2026.0055.6754.335.2688.058.18 39.6452.94 3034.3375.3373.675.0086.348.19 44.6258.42 4038.6787.6785.672.8285.328.16 49.5263.78 5039.6792.6790.672.5684.898.12 51.0965.28 Continued on next page 41 Table 18 continued Model Budget (min) Pass@1 (%) ↑Build (%) ↑Diff. SR (%) ↑FP pass (%) ↓Process ↑Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ GLM-5.2103.674.334.330.0091.619.58 24.8243.73 2018.0034.3333.0014.6388.848.20 33.9853.26 3027.3359.6758.3310.3487.418.15 38.6258.17 4031.3373.0071.0010.3486.908.12 40.2359.55 5032.3377.3375.3310.3486.738.11 41.2161.02 Kimi-K3108.339.679.670.0092.159.34 22.3233.28 2022.3340.3340.000.0090.998.53 26.1036.84 3030.6762.0060.332.5689.678.36 28.9339.91 4036.6776.0074.331.7588.908.34 31.3242.35 5038.6783.3381.671.5988.488.30 32.4243.53 HY4-Preview105.337.677.000.0090.078.76 25.0632.44 2019.0034.3333.6718.1889.708.40 33.0441.02 3028.6755.6755.0014.2988.208.32 38.6447.22 4032.0065.3364.6713.9887.548.31 40.3249.15 5033.6772.6771.3312.3886.908.32 41.2850.15 Budgets threshold existing trajectories by active generation time, excluding recorded resource-admission queues and maintenance pauses. External verification and judging are outside the generation clock. Only final artifacts completed by the cutoff contribute; unfinished runs contribute no success to Pass@1, Build, or Diff. SR, whose denominator remains 300. Process and Semantic average available valid scores among eligible completed runs. Turns and Tool Calls average only eligible passed runs. FP pass is the pooled alert-level FP fraction for those passed runs, using the same adjudicated labels as the main table. A dash denotes no eligible score or a zero alert denominator. Per-cell sample sizes accompany the CSV. These are retrospective completion curves, not separately rerun deadline experiments. Table 19 Time-budget sensitivity on CheckerBench: Claude Code. Model Budget (min) Pass@1 (%) ↑Build (%) ↑Diff. SR (%) ↑FP pass (%) ↓Process ↑Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ GPT-5.61010.0019.3319.000.0088.098.24 28.3751.77 2023.6755.6753.330.0082.777.73 31.3454.06 3029.6776.3369.002.6381.897.70 34.1257.94 4033.0092.0081.332.4481.387.67 35.1859.87 5034.0093.6783.002.4481.417.69 35.7160.58 Claude Opus 4.8109.0010.009.000.0087.409.28 17.5230.11 2018.3325.3324.000.0088.308.85 23.3635.98 3028.6745.3344.002.9487.778.47 28.4941.53 4032.6755.3354.002.5087.268.30 30.2343.16 5035.0063.6762.332.3386.888.21 32.6045.63 Qwen3.8-Max107.008.678.670.0092.419.12 24.0029.33 2023.0039.3339.004.7690.968.50 27.9034.01 3027.3351.3351.005.8890.338.41 30.1236.60 4031.3361.0060.674.8889.868.35 32.1739.36 5034.6767.3367.004.5589.588.38 33.9141.57 DeepSeek-V4-Pro1014.0022.6722.330.0089.438.56 35.0748.45 2023.6749.3349.000.0087.268.29 39.4453.23 3031.0073.6772.670.0085.868.26 42.2456.84 4036.6786.0084.331.7985.618.29 45.7160.81 5038.3391.3389.671.7285.258.27 47.1762.50 GLM-5.2101.001.331.33—95.759.38 21.6740.33 2012.6723.0022.330.0093.378.89 30.9241.89 3023.0047.0045.330.0091.558.64 34.0145.87 4028.0062.6761.000.0090.558.48 36.6947.92 5033.0076.3374.670.0089.678.39 38.7549.93 Kimi-K31011.0013.3313.330.0092.519.33 24.8831.21 2022.6747.6747.000.0090.118.29 27.3233.54 3030.3366.3365.333.2888.498.27 30.0036.31 4035.6778.3377.002.8688.048.29 31.4637.83 5038.0083.6782.332.7487.798.34 31.8138.61 Continued on next page 42 Table 19 continued Model Budget (min) Pass@1 (%) ↑Build (%) ↑Diff. SR (%) ↑FP pass (%) ↓Process ↑Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ HY4-Preview102.002.332.330.0091.999.14 24.0032.33 208.3313.3313.000.0093.038.66 34.0443.00 3016.0029.6729.000.0091.868.63 43.0053.85 4019.6741.0039.330.0091.428.44 47.0258.29 5023.6749.3347.330.0090.838.35 50.6662.23 Budgets threshold existing trajectories by active generation time, excluding recorded resource-admission queues and maintenance pauses. External verification and judging are outside the generation clock. Only final artifacts completed by the cutoff contribute; unfinished runs contribute no success to Pass@1, Build, or Diff. SR, whose denominator remains 300. Process and Semantic average available valid scores among eligible completed runs. Turns and Tool Calls average only eligible passed runs. FP pass is the pooled alert-level FP fraction for those passed runs, using the same adjudicated labels as the main table. A dash denotes no eligible score or a zero alert denominator. Per-cell sample sizes accompany the CSV. These are retrospective completion curves, not separately rerun deadline experiments. Table 20 Time-budget sensitivity on CheckerBench: Hermes Agent. Model Budget (min) Pass@1 (%) ↑Build (%) ↑Diff. SR (%) ↑FP pass (%) ↓Process ↑Semantic ↑ Avg. Turns ↓ Avg. Tool Calls ↓ Hermes Agent GPT-5.6109.6727.6720.00—83.857.97 24.0044.21 2019.3371.6748.670.0081.957.67 25.5546.62 3022.3389.6759.330.0080.497.57 25.9047.19 4023.0094.6762.330.0080.227.57 25.9446.91 5023.00100.0064.000.0079.857.50 25.9446.91 Claude Opus 4.8106.008.008.000.0088.239.23 21.0030.50 2016.0026.3326.000.0088.308.59 28.3141.40 3026.3347.0046.670.0088.158.48 38.2453.27 4030.6762.6762.330.0087.238.41 42.0858.26 5034.3380.3378.675.4585.588.32 44.8461.89 Qwen3.8-Max103.003.673.67—93.739.05 20.5628.67 2010.0015.0014.670.0093.159.07 29.3738.43 3015.6725.3325.000.0092.038.76 34.0944.11 4019.6734.3334.005.8891.358.62 36.9347.54 5025.0044.0042.3311.1190.928.53 40.5551.95 DeepSeek-V4-Pro109.3315.0015.000.0090.908.78 29.2150.00 2022.0049.3348.000.0088.118.34 34.5656.45 3027.6770.6769.000.0086.738.09 37.2759.41 4030.3385.0082.330.0085.938.07 38.2460.56 5032.3394.0090.670.0085.298.03 39.8262.37 GLM-5.2103.676.006.00—90.698.59 25.1843.36 2012.3328.0027.330.0087.498.27 35.4155.84 3019.0053.0051.670.0085.547.99 39.9360.49 4021.6770.0068.670.0083.987.90 41.3762.00 5022.3379.6778.000.0082.877.81 42.8563.22 Kimi-K3107.6714.6714.330.0090.148.38 21.9629.43 2016.3341.0039.333.6487.428.30 25.9034.55 3022.6767.6763.004.7684.788.18 31.1340.41 4024.0078.6772.674.7683.078.06 31.6541.04 5024.3387.0079.674.7682.628.08 32.7342.12 HY4-Preview101.331.331.33—94.009.00 24.5035.25 207.6712.0012.00—92.198.91 31.3942.83 3012.6725.3325.00—89.988.69 37.3248.32 4017.3338.3337.000.0090.348.53 44.4456.00 5021.6757.6752.670.0088.428.44 49.6962.06 Budgets threshold existing trajectories by active generation time, excluding recorded resource-admission queues and maintenance pauses. External verification and judging are outside the generation clock. Only final artifacts completed by the cutoff contribute; unfinished runs contribute no success to Pass@1, Build, or Diff. SR, whose denominator remains 300. Process and Semantic average available valid scores among eligible completed runs. Turns and Tool Calls average only eligible passed runs. FP pass is the pooled alert-level FP fraction for those passed runs, using the same adjudicated labels as the main table. A dash denotes no eligible score or a zero alert denominator. Per-cell sample sizes accompany the CSV. These are retrospective completion curves, not separately rerun deadline experiments. H.4 Language, Repository, and CWE Performance These breakdowns average the same three independent repeats as Table 2 and Figure 6. Each model–harness pair has the same 300 tasks in each repeat. The main heatmaps average the three harness-specific estimates equally; the tables below retain each harness. The abbreviated model names in the CWE tables follow the full names in Table 2. All rates are percentages and are descriptive; small task subsets do not establish reliable model rankings. The downloadable CSVs preserve full precision. 43 Languages and analyzers. C/C++ uses CSA (159 tasks); Python (51), Go (30), Java (30), and JavaScript (30) use CodeQL. Language and analyzer effects cannot be separated. Table 21 Language-specific Pass@1 (%) by harness. All tasks in each language remain in the denominator. ModelC/C++PythonGoJavaJavaScript OpenCode GPT-5.612.5862.7546.6746.6750.00 Claude Opus 4.826.4272.5560.0070.0060.00 Qwen3.8-Max15.7272.5546.6753.3366.67 DeepSeek-V4-Pro28.3058.8253.3340.0053.33 GLM-5.216.3560.7853.3346.6733.33 Kimi-K327.0468.6336.6750.0040.00 HY4-Preview13.8454.9060.0053.3356.67 Claude Code GPT-5.622.0162.7530.0046.6740.00 Claude Opus 4.821.3862.7553.3340.0036.67 Qwen3.8-Max14.4754.9060.0066.6750.00 DeepSeek-V4-Pro28.3056.8646.6743.3346.67 GLM-5.220.1364.7143.3343.3326.67 Kimi-K323.9060.7846.6756.6746.67 HY4-Preview5.6652.9436.6743.3336.67 Hermes Agent GPT-5.610.6941.1836.6730.0036.67 Claude Opus 4.822.0160.7840.0040.0043.33 Qwen3.8-Max3.7760.7840.0043.3343.33 DeepSeek-V4-Pro16.9854.9060.0043.3336.67 GLM-5.25.6650.9843.3333.3330.00 Kimi-K311.3243.1430.0036.6743.33 HY4-Preview2.5245.1036.6746.6743.33 Table 22 Repository sensitivity by harness (%). Task equal uses 300 tasks; Repo equal averages 167 repository rates; No Linux uses 193 tasks in 166 repositories. ModelTask equalRepo equalNo Linux OpenCode GPT-5.631.6746.2342.49 Claude Opus 4.845.3357.3253.37 Qwen3.8-Max37.3353.7349.74 DeepSeek-V4-Pro39.6749.1746.11 GLM-5.232.3343.6339.38 Kimi-K338.6747.9244.56 HY4-Preview33.6748.6043.52 Claude Code GPT-5.634.0042.6039.90 Claude Opus 4.835.0044.5140.41 Qwen3.8-Max34.6749.4445.60 DeepSeek-V4-Pro38.3346.6843.52 GLM-5.233.0042.5038.86 Kimi-K338.0049.0144.56 HY4-Preview23.6738.0633.68 Hermes Agent GPT-5.623.0032.8831.09 Claude Opus 4.834.3343.7040.41 Qwen3.8-Max25.0040.5436.79 DeepSeek-V4-Pro32.3342.6438.34 GLM-5.222.3335.5131.61 Kimi-K324.3334.2531.09 HY4-Preview21.6735.9932.12 Interaction turns. Figure 6(c) compares all 21 combinations. Pass@1 uses all 300 tasks; Avg. Turns uses passed trajectories, including subagents. Different task sets do not establish a causal effect of more turns. Passed-trajectory counts appear in the companion summary. 44 Complete CWE breakdown. The following tables include all 85 explicit CWE labels and the 14 tasks without an explicit label (Other). A task contributes once to each distinct CWE label it carries; 14 tasks have two labels. Consequently the row counts sum to 314, not 300. Other does not pool known low-frequency CWEs. The main figure selects the eight explicit labels with at least ten tasks using counts alone, before examining performance. This display threshold is not a claim of statistical precision. CWE groups also differ in language, analyzer, and repository composition. Table 23 Complete CWE-specific Pass@1 (%) for OpenCode. n counts distinct tasks, not harnesses or repeats. CWEnGPTClaudeQwenDeepSeekGLMKimiHY4 CWE-787 267.6926.9219.2330.7715.3826.923.85 CWE-125 2123.8128.5723.8123.8119.0533.33 42.86 CWE-22 1560.0073.3353.3360.0053.3353.33 26.67 CWE-476 137.6930.7715.3830.7715.3823.087.69 CWE-79 1233.3366.6758.3341.6741.6733.33 66.67 CWE-362 128.3325.008.338.338.3325.00 16.67 CWE-416 1225.0025.008.3350.0025.0025.00 25.00 CWE-400 1040.0050.0040.0050.0050.0070.00 60.00 CWE-908 911.1133.330.0033.3311.1133.33 22.22 CWE-667 812.5012.5012.500.0037.5012.500.00 CWE-190 714.2914.290.0028.570.000.00 14.29 CWE-415 728.5714.2928.5728.5728.5728.570.00 CWE-20633.3383.3333.3333.3316.6750.00 66.67 CWE-611 683.3383.3366.6733.3350.0066.67 50.00 CWE-617 60.0016.6716.6733.3316.6716.670.00 CWE-119 50.000.000.000.000.0020.000.00 CWE-191 50.0040.000.0020.000.0060.00 20.00 CWE-200 540.0020.0060.0060.0040.0060.00 60.00 CWE-377 580.00100.0060.00100.00 100.00 100.00 60.00 CWE-401 520.0080.0020.0060.0060.0040.00 60.00 CWE-601 540.0040.0040.0020.0020.0060.00 60.00 CWE-770 540.0020.0040.000.0020.0020.00 40.00 CWE-78425.0025.0075.0050.0025.0050.00 25.00 CWE-129 40.0025.0025.0025.0025.000.00 25.00 CWE-674 40.0025.0050.0025.000.000.00 25.00 CWE-943 100.00100.0066.67100.0066.67 100.00 100.00 CWE-189 366.67100.0033.33100.0066.670.00 33.33 CWE-287 333.3366.6766.6766.670.0066.67 66.67 CWE-326 3 100.00100.0066.67100.0033.3366.67 66.67 CWE-1333 366.67100.0066.6766.6766.6766.67 33.33 CWE-23250.00100.00 100.0050.0050.0050.00 100.00 CWE-59250.00100.00 100.00100.00 100.0050.00 50.00 CWE-74250.0050.00 100.0050.0050.00 100.00 50.00 CWE-77250.0050.00 100.00100.00 100.0050.00 100.00 CWE-284 250.0050.0050.0050.0050.0050.000.00 CWE-338 250.00100.0050.0050.00 100.0050.00 100.00 CWE-352 2 100.00100.00 100.0050.000.00 100.00 50.00 CWE-502 2 100.00100.00 100.00100.0050.00 100.00 100.00 CWE-532 250.00100.00 100.000.0050.0050.00 50.00 CWE-672 250.0050.0050.00100.0050.00 100.00 50.00 CWE-772 20.0050.00 100.0050.000.000.000.00 CWE-835 20.0050.000.000.000.0050.000.00 CWE-863 2 100.00100.00 100.0050.00 100.0050.00 50.00 CWE-291 100.00100.00 100.00100.000.000.000.00 CWE-8810.000.00 100.00100.00 100.00 100.000.00 CWE-911 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-120 10.000.000.000.000.000.000.00 CWE-122 10.000.000.000.000.000.000.00 CWE-131 1 100.000.000.000.000.000.000.00 CWE-193 10.00100.00 100.000.000.000.000.00 CWE-201 10.000.000.000.00 100.000.00 100.00 CWE-203 10.00100.00 100.000.000.000.000.00 CWE-209 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-250 10.000.00 100.000.00 100.000.000.00 CWE-256 10.00100.00 100.000.00 100.00 100.000.00 CWE-264 1 100.000.000.00100.000.00 100.00 100.00 CWE-266 10.000.000.000.000.000.000.00 CWE-295 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-312 10.00100.00 100.000.00 100.00 100.000.00 CWE-327 1 100.00100.000.00100.00 100.00 100.00 100.00 CWE-331 10.000.000.000.000.000.000.00 Continued on next page 45 Table 23 (continued) CWEnGPTClaudeQwenDeepSeekGLMKimiHY4 CWE-347 1 100.000.00 100.00100.00 100.000.00 100.00 CWE-364 10.000.000.000.000.000.000.00 CWE-366 1 100.00100.000.00100.00 100.000.000.00 CWE-369 10.000.000.000.000.000.000.00 CWE-399 10.000.000.000.000.000.000.00 CWE-407 10.000.00 100.00100.00 100.000.000.00 CWE-459 1 100.00100.000.000.000.00 100.000.00 CWE-460 10.000.00 100.000.000.00 100.000.00 CWE-522 1 100.00100.00 100.000.00 100.000.00 100.00 CWE-538 10.000.00 100.000.00 100.000.00 100.00 CWE-552 10.000.000.000.000.000.000.00 CWE-613 10.00100.00 100.000.000.000.00 100.00 CWE-639 1 100.000.00 100.000.000.000.000.00 CWE-668 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-670 1 100.000.00 100.000.00 100.000.000.00 CWE-680 10.000.000.000.000.000.000.00 CWE-692 10.000.000.00100.000.000.000.00 CWE-789 10.00100.00 100.00100.00 100.000.00 100.00 CWE-826 1 100.000.000.00100.000.00 100.00 100.00 CWE-843 10.00100.000.00100.00 100.00 100.000.00 CWE-862 1 100.00100.00 100.000.00 100.00 100.000.00 CWE-918 10.00100.00 100.00100.000.000.00 100.00 CWE-1188 1 100.00100.000.00100.000.000.00 100.00 CWE-1336 1 100.00100.00 100.00100.00 100.00 100.00 100.00 Other1435.7157.1435.7150.0042.8657.14 50.00 Table 24 Complete CWE-specific Pass@1 (%) forClaude Code. n counts distinct tasks, not harnesses or repeats. CWEnGPTClaudeQwenDeepSeekGLMKimiHY4 CWE-787 2626.9226.9226.9226.9226.9223.083.85 CWE-125 2114.299.5228.5728.5728.5738.104.76 CWE-22 1546.6760.0060.0040.0046.6760.00 40.00 CWE-476 1330.7723.087.6938.4615.3815.380.00 CWE-79 1225.0025.0050.0041.6733.3325.00 33.33 CWE-362 1225.0016.6725.0025.0016.6716.678.33 CWE-416 1233.3325.0025.0025.0025.0025.000.00 CWE-400 1040.0040.0040.0040.0040.0040.00 50.00 CWE-908 922.2233.3311.1133.3333.3333.330.00 CWE-667 825.0025.000.0037.5025.0025.00 12.50 CWE-190 714.290.0014.2914.2914.290.000.00 CWE-415 714.2914.2928.5714.2914.2914.29 14.29 CWE-20650.0016.6766.6716.6716.6733.33 33.33 CWE-611 666.67100.0083.3350.0066.6783.33 83.33 CWE-617 633.330.0033.3316.6716.6716.670.00 CWE-119 520.000.000.000.000.000.000.00 CWE-191 540.0020.000.0040.000.0040.000.00 CWE-200 540.0060.0040.000.0040.0040.00 40.00 CWE-377 5 100.00100.0060.00100.0060.00 100.00 80.00 CWE-401 520.0080.0020.0060.0040.0040.00 20.00 CWE-601 540.0040.0040.0060.0040.0040.00 60.00 CWE-770 50.0040.0020.0020.000.000.00 20.00 CWE-78425.0050.0050.0075.0050.0075.00 25.00 CWE-129 425.000.0025.0050.0050.0050.00 25.00 CWE-674 40.000.0025.0025.000.0075.00 25.00 CWE-943 100.0066.67 100.00100.0066.6766.67 66.67 CWE-189 333.3366.6766.670.0066.670.00 33.33 CWE-287 333.330.0033.3333.3333.3333.33 66.67 CWE-326 366.6766.6766.67100.0066.6766.670.00 CWE-1333 333.3333.3333.3333.3366.6733.33 66.67 CWE-23250.00100.0050.0050.000.0050.000.00 CWE-592 100.00100.0050.00100.0050.00 100.000.00 CWE-74250.0050.00 100.0050.0050.0050.00 50.00 CWE-772 100.00100.00 100.0050.0050.00 100.00 100.00 CWE-284 250.0050.0050.0050.000.0050.000.00 CWE-338 250.000.000.0050.000.0050.00 50.00 CWE-352 250.00100.000.00100.0050.00 100.000.00 CWE-502 2 100.00100.0050.00100.00 100.0050.00 100.00 CWE-532 250.0050.0050.0050.0050.0050.00 50.00 CWE-672 250.0050.0050.000.0050.000.000.00 Continued on next page 46 Table 24 (continued) CWEnGPTClaudeQwenDeepSeekGLMKimiHY4 CWE-772 250.0050.0050.000.000.0050.000.00 CWE-835 20.000.000.000.000.000.000.00 CWE-863 250.0050.00 100.0050.00 100.0050.00 100.00 CWE-2910.00100.000.00100.00 100.00 100.00 100.00 CWE-8810.000.000.00100.000.00 100.000.00 CWE-911 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-120 10.000.000.000.000.000.000.00 CWE-122 10.000.000.000.00 100.000.000.00 CWE-131 10.000.000.000.000.000.000.00 CWE-193 10.000.00 100.00100.00 100.000.000.00 CWE-201 10.00100.000.000.000.000.000.00 CWE-203 10.000.000.000.000.000.00 100.00 CWE-209 1 100.000.00 100.000.00 100.00 100.00 100.00 CWE-250 10.000.00 100.00100.00 100.00 100.000.00 CWE-256 10.000.00 100.000.000.000.000.00 CWE-264 1 100.00100.000.00100.00 100.00 100.00 100.00 CWE-266 10.000.000.000.000.000.000.00 CWE-295 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-312 10.000.00 100.000.000.000.000.00 CWE-327 1 100.00100.000.000.00 100.00 100.000.00 CWE-331 10.000.00 100.000.000.000.000.00 CWE-347 10.000.000.000.00 100.00 100.00 100.00 CWE-364 10.000.000.000.000.000.000.00 CWE-366 1 100.00100.000.000.00 100.000.00 100.00 CWE-369 10.000.000.000.000.000.000.00 CWE-399 10.000.000.000.000.000.000.00 CWE-407 1 100.000.000.000.000.000.000.00 CWE-459 10.00100.000.000.00 100.00 100.000.00 CWE-460 10.000.000.000.000.000.000.00 CWE-522 10.00100.00 100.00100.000.00 100.000.00 CWE-538 10.00100.00 100.000.00 100.000.00 100.00 CWE-552 10.000.000.000.00 100.000.000.00 CWE-613 10.000.00 100.00100.000.00 100.00 100.00 CWE-639 10.000.000.00100.000.00 100.000.00 CWE-668 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-670 1 100.00100.000.00100.00 100.000.000.00 CWE-680 10.000.000.000.000.000.000.00 CWE-692 1 100.000.00 100.000.000.000.000.00 CWE-789 1 100.000.00 100.00100.00 100.00 100.00 100.00 CWE-826 10.00100.000.000.000.00 100.000.00 CWE-843 10.000.000.00100.000.000.000.00 CWE-862 1 100.00100.00 100.00100.000.00 100.000.00 CWE-918 10.000.00 100.00100.000.00 100.00 100.00 CWE-1188 1 100.00100.00 100.00100.000.000.000.00 CWE-1336 1 100.00100.00 100.00100.00 100.00 100.00 100.00 Other1435.7135.7135.7164.2935.7150.00 21.43 Table 25 Complete CWE-specific Pass@1 (%) forHermes Agent. n counts distinct tasks, not harnesses or repeats. CWEnGPTClaudeQwenDeepSeekGLMKimiHY4 CWE-787 2619.2319.2319.2326.927.6911.540.00 CWE-125 2119.059.5214.2914.294.769.52 14.29 CWE-22 1533.3340.0040.0046.6740.0033.33 26.67 CWE-476 1315.3830.770.007.697.6930.777.69 CWE-79 1233.3333.3375.0033.3333.3325.00 58.33 CWE-362 128.338.3316.678.3316.670.000.00 CWE-416 1216.6733.330.0025.0025.0016.670.00 CWE-400 1040.0030.0030.0020.0030.0030.00 30.00 CWE-908 90.0033.330.0022.2211.1111.11 11.11 CWE-667 80.0025.000.0012.500.000.000.00 CWE-190 714.290.000.0014.290.000.00 14.29 CWE-415 714.2928.570.0028.570.0014.290.00 CWE-20616.6733.3316.6716.6733.3316.67 33.33 CWE-611 650.0083.3333.3366.6750.0050.00 50.00 CWE-617 616.6716.670.000.000.0016.67 16.67 CWE-119 50.000.000.000.000.000.000.00 CWE-191 520.0060.000.0040.000.0020.000.00 CWE-200 520.0060.0060.0060.0020.0020.00 20.00 CWE-377 580.0060.00 100.0080.0080.00 100.00 80.00 Continued on next page 47 Table 25 (continued) CWEnGPTClaudeQwenDeepSeekGLMKimiHY4 CWE-401 50.0060.000.0080.0020.0020.00 20.00 CWE-601 520.0080.0020.0020.0020.000.00 40.00 CWE-770 50.000.0020.0020.000.0020.000.00 CWE-78425.0050.0075.0050.0075.0025.00 75.00 CWE-129 40.0025.000.0025.000.000.000.00 CWE-674 40.0025.0025.0025.0050.000.00 25.00 CWE-943 100.0066.6733.33100.0066.6733.33 66.67 CWE-189 333.3333.330.0066.6733.330.00 33.33 CWE-287 333.3333.3333.3366.6733.3333.33 66.67 CWE-326 366.67100.0066.6766.6766.67 100.00 33.33 CWE-1333 333.3366.6766.6766.6766.6766.67 33.33 CWE-23250.0050.0050.00100.0050.000.000.00 CWE-59250.00100.00 100.00100.00 100.00 100.00 50.00 CWE-74250.0050.00 100.0050.0050.0050.00 100.00 CWE-772 100.00100.0050.00100.0050.0050.00 100.00 CWE-284 250.000.000.0050.0050.0050.000.00 CWE-338 20.0050.0050.0050.0050.0050.00 50.00 CWE-352 250.00100.000.000.0050.00 100.000.00 CWE-502 2 100.00100.00 100.00100.00 100.00 100.00 50.00 CWE-532 250.0050.0050.0050.00 100.0050.00 100.00 CWE-672 20.0050.000.0050.0050.000.000.00 CWE-772 20.000.0050.000.000.0050.000.00 CWE-835 20.000.000.000.000.000.000.00 CWE-863 250.0050.00 100.0050.0050.0050.00 50.00 CWE-2910.000.000.000.000.000.000.00 CWE-8810.000.00 100.000.00 100.000.00 100.00 CWE-911 100.00100.00 100.00100.00 100.000.000.00 CWE-120 10.000.000.000.000.000.000.00 CWE-122 10.000.000.000.000.000.000.00 CWE-131 1 100.00100.000.00100.000.000.000.00 CWE-193 10.00100.000.000.000.000.000.00 CWE-201 10.000.000.000.00 100.000.000.00 CWE-203 10.000.000.000.000.000.000.00 CWE-209 10.000.00 100.00100.000.00 100.00 100.00 CWE-250 10.000.00 100.00100.000.000.000.00 CWE-256 10.000.000.000.00 100.000.000.00 CWE-264 10.00100.000.00100.000.00 100.000.00 CWE-266 10.000.000.000.000.00 100.000.00 CWE-295 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-312 10.000.000.000.00 100.000.000.00 CWE-327 1 100.000.000.00100.000.000.00 100.00 CWE-331 10.000.00 100.00100.000.000.000.00 CWE-347 10.00100.000.000.000.000.000.00 CWE-364 10.000.000.000.000.000.000.00 CWE-366 1 100.000.00 100.000.000.00 100.000.00 CWE-369 10.000.000.000.000.000.000.00 CWE-399 10.000.000.000.000.000.000.00 CWE-407 10.000.000.00100.000.00 100.00 100.00 CWE-459 10.000.000.00100.000.000.000.00 CWE-460 10.000.000.000.000.000.000.00 CWE-522 1 100.00100.00 100.000.000.00 100.000.00 CWE-538 10.000.00 100.000.000.00 100.00 100.00 CWE-552 10.000.000.000.000.000.000.00 CWE-613 10.000.000.00100.00 100.000.000.00 CWE-639 10.00100.000.000.000.00 100.00 100.00 CWE-668 1 100.00100.00 100.00100.00 100.00 100.00 100.00 CWE-670 10.00100.000.000.000.000.00 100.00 CWE-680 10.000.000.000.000.000.000.00 CWE-692 10.000.000.000.000.000.00 100.00 CWE-789 1 100.000.00 100.00100.000.00 100.000.00 CWE-826 10.000.000.000.000.000.000.00 CWE-843 10.000.000.00100.000.000.000.00 CWE-862 10.000.000.000.000.000.00 100.00 CWE-918 1 100.000.000.000.000.00 100.000.00 CWE-1188 1 100.00100.000.000.000.00 100.000.00 CWE-1336 1 100.00100.00 100.00100.00 100.00 100.00 100.00 Other1414.2957.1428.5735.7121.4314.29 14.29 48

Instant research discovery