Paper deep dive
Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch
Shuyang Xie, Shuxiao Xie, Feng Zhu, Yanli Ji, Wangmeng Zuo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Online-judge verdicts and the datasets and benchmarks built on them are treated as ground truth for evaluating and training large language models for code. Yet prior audits have sounded a warning: official suites accept buggy submissions. These audits, however, stop at the warning and offer no practical remedy. Our remedy has two parts: an off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss and supplies these suites where no official suite exists; a certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge: multiple independently written accepted solutions agree on the expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input legal. One such agent identifies 589 verified accepted-but-buggy submissions among AtCoder's 20,375 audited accepted submissions; extending the same certification to all five agents yields a union floor of 906 such submissions. Five agents, scored separately, each stay within 1.7pp of official-suite coverage on logic bugs those suites catch. On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget. Where an official suite exists, the agent audits suite adequacy instead of assuming it; where none exists, agent suites catch the most buggy submissions among methods we reproduced and tested.
Tags
Links
- Source: https://arxiv.org/abs/2608.01715v1
- Canonical: https://arxiv.org/abs/2608.01715v1
Trouble viewing inline? Open PDF directly â
Full Text
95,484 characters extracted from source content.
Expand or collapse full text
Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch Preprint Shuyang Xie 1â Shuxiao Xie 2â Feng Zhu 1 Yanli Ji 3 Wangmeng Zuo 1â 1 Harbin Institute of Technology 2 Fudan University 3 Sun Yat-sen University § Research artifacts: github.com/xieTwim/test-suite-auditors § Companion skill: github.com/xieTwim/agentic-testgen Abstract Online-judge verdicts and the datasets and benchmarks built on them are treated as ground truth for evaluating and training large language models for code. Yet prior audits have sounded a warning: official suites accept buggy submissions. These audits, however, stop at the warning and offer no practical remedy. Our remedy has two parts: an off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss and supplies these suites where no official suite exists; a certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge: multiple independently written accepted solutions agree on the expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input legal. One such agent identifies 589 verified accepted-but-buggy submissions among AtCoderâs 20,375 audited accepted submissions; extending the same certification to all five agents yields a union floor of 906 such submissions. Five agents, scored separately, each stay within 1.7p of official-suite coverage on logic bugs those suites catch. On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget. Where an official suite exists, the agent audits suite adequacy instead of assuming it; where none exists, agent suites catch the most buggy submissions among methods we reproduced and tested. 1 Introduction Online judges make hidden test suites the operational ground truth for program correctness. A passing submission is marked accepted, so the verdict shapes contestant ratings, determines which solutions datasets and benchmarks treat as correct, and grounds the execution-based reward signals used to train code models [13,14]. Yet the trust chain can fail: a solution can pass every official test yet fail on a legal input the suite never tries. That risk is no longer hypothetical. An empirical audit found official AtCoder tests accepting buggy submissions at scale [16], and TrickyBugs assembled plausible-but-buggy programs into a dataset [17]. Together, these studies establish the blind spot through random and differential testing over graded submissions, with majority voting used for adjudication. Downstream datasets inherit the same blind spot when they treat passage of their own tests as correctness. Without official suites, CodeContests and TACO ship tests produced by mutation or large language model generation. In those shipped tests, illegal, out-of-spec inputs are a named failure mode â Equal contribution. â Corresponding author: cswmzuo@gmail.com. arXiv:2608.01715v1 [cs.SE] 3 Aug 2026 Coding Agents as Test-Suite AuditorsPreprint Figure 1: The auditor framework and its certification chain (§3). Agents see only the problem statement and one reference solution; the adversarial test suite audits the hidden official suite (the audited object, never an input) by retesting an officially accepted submission, or supplies the missing evaluation suite for candidate buggy programs; the certification chain serves both. [26]. Prior work builds tests for datasets, evaluates test writers on known bugs, or audits judges through graded submissions (§2). No practical remedy yet combines test generation strong enough to expose what official suites miss with per-finding certification independent of the official judge (§2; §6). Off-the-shelf coding agents fill this gap as test-suite auditors. Unlike those approaches, our target-blind method audits the official hidden suites. Each agent receives only a problem statement and one human reference solution, from which it constructs an adversarial suite; official tests, verdicts, and the submissions under audit remain hidden. A finding enters the ledger only after the certification chain in Figure 1: a consensus oracle over independent accepted solutions supplies the expected output, brute-force solutions settle disputes, and a strict per-problem input-legality validator written from the statement alone admits as certifying evidence only legal inputs that pass it. The gate is stated in §3 and measured in §6. Within our own certification chains, it closes the illegal-input failure mode seen in shipped dataset tests. The same agents generate tests in both roles. Where official suites exist, they audit them: on AtCoder, one agent arm exposes verified accepted-but-buggy submissions within the audited sample. Every engine approaches re-judged official-suite logic-bug coverage, complementing the official suites rather than replacing them. Where none exist, they supply the tests themselves: on fresh post-cutoff Codeforces problems, they lead every reproduced baseline. The certification chain keeps each finding checkable in both settings. Our contributions are: âąA framework for auditing suite adequacy. Our target-blind auditor turns suite adequacy from an assumption into an audited property. Prior random and differential testing targets graded submissions [16,17]; off-the-shelf coding agents never see the audited submissions and construct adversarial suites from a problem statement and one reference solution. The certification chain in §3 verifies each finding without relying on the official judge. âąAn AtCoder audit with robustness checks. We audit 20,375 compilable accepted submissions. One agent arm (codex) identifies new, individually verified accepted-but-buggy official-suite misses; the 589-entry ledger records them. Carrying the same certification chain across all five arms yields a legality- and tolerance-clean union floor of 906. Separately, on the known-bug set (the official suiteâs historically rejected submissions), all agent suites trail the re-judged official detection rate by at most 1.7p; the random generation matched to the agentâs input budget trails by 10.9p. Oracle and input audits find no measurement artifact (§6). âąCodeforces test supply. On post-cutoff Codeforces problems without official suites, agent suites supply the tests. At every tested input budget, the agent arm leads all five baselines (§5). At its design budget of 2 Coding Agents as Test-Suite AuditorsPreprint 50 inputs, cov@50 (the buggy-pool share a random 50-input subset kills) is 0.952. Among the generator arms, it requires the fewest inputs to reach the coverage it achieves. âąOpen artifacts. We release the five-arm ledger of accepted-but-buggy submissions, each entry recording which agents certified it (906 in union, 589 for codex alone), the six-arm CF-fresh kill matrix, the per-problem validators, and the judging harness. These artifacts support machine re-checking of every certified number derived from them. 2 Background and Related Work Most prior work treats official suites or platform verdicts as scoring standards, either by building substitutes or by evaluating whether public and generated tests are sufficient. A smaller line instead audits the official verdict, but existing approaches rely on uncertified testing over graded submissions, target-aware inspection, or one-submission-at-a-time attacks. We therefore proceed from suite construction and test-quality evaluation, to verdict auditing, and finally to input legality as a certificate. Building suites and evaluating test quality. When hidden suites are unavailable, dataset builders construct substitutes. Mutation-based and inherited suites include CodeContests [14] and TACO [13]; agentic and scaled synthesis include CodeContests+, HardTests, and Klear-CodeTest [8,11,26]; generated or adversarial pipelines include AutoCode, Themis, and CodeHacker [23,30,31]. A separate line evaluates whether available tests expose bugs: HumanEval and EvalPlus study public benchmark tests [6,15], while TestCase-Eval evaluates LLM tests and TCGBench their generators, and CodeT and CURE use generated tests for solution selection [4,5,25,29]. These test-quality studies assess public or generated tests rather than hidden platform suites. Across both sublines, an official suite or verdict remains a scoring standard rather than an audit object. Auditing the official verdict. Changing the audit object also changes what each system accepts as truth. Who-Judges-the-Judge directly audits official tests and exposes accepted-but-buggy submissions at scale [16]. It uses random, differential, and majority-vote testing over graded submissions, but does not certify its findings. TrickyBugs collects such programs [17], while its LLM-powered follow-up generates tests and evaluates failures on curated datasets using canonical programs and dataset-provided or manual input-validity checks [18]. UOJ-Bench is target-aware: each attack targets one full-score submission, and its cost analysis prices that per-submission strategy at $100k/year [27]. Solvita is also target-aware; it inspects each candidate source and treats disagreement with official acceptance as an internal diagnostic, with stronger-than-official confirmation relying on accepted-solver cross-checks and manual validation [12]. CodeContests-O iterates generation through execution feedback on per-problem, platform-labeled solution pools and takes the official verdict as ground truth [3]; neither those pools nor its shipped tests cover the post-cutoff problems in §5. Under that convention, a generated test killing a latent accepted-but-buggy solution is a false negative, so the loop repairs the test rather than treating the disagreement as evidence. We instead audit that verdict. Making input legality a certificate. Input legality places a separate requirement on the tests used to audit an official verdict. A killing input is admissible only when legal under the statement constraints. LogiCase makes validators first-class through formal grammars [24]; CodeContests+ identifies illegal inputs as a failure mode in shipped dataset tests [26]. Themis uses constraint-aware validators, whereas our per-problem validator is statement-derived and written while blind to the killing inputs at stake; under our protocol an input serves as evidence only after passing it. This requirement addresses the illegal-input failure mode identified in shipped dataset tests. Those tests provide the background for the requirement; the official verdict remains the audit object. §6 measures the statement-derived gate on the certified kills. Taken together, these distinctions make the audit protocol, rather than suite scale or platform infrastructure, the basis of our comparison (§3). 3 Method: Agents as Suite Auditors, and the Certification Chain We next ask how a disagreement becomes a certified finding. Every reported finding follows a certification chain with independently audited links (Figure 1). Both experiments (§4â5) instantiate the same certification pattern, with the oracle, engine composition, and validator source configured per experiment. What the auditor can see. Suite construction uses a restricted visibility model, not a certification-chain link. For one competitive-programming problem, construction sees only the statement and one human reference solution. It cannot access official tests, verdicts, or any submission under audit. Per-problem artifacts comprise a 3 Coding Agents as Test-Suite AuditorsPreprint test generator, an input validator, an adversarial suite of concrete inputs, and re-checking tooling. Official suites and verdicts enter only afterwards as the object audited and are not used to judge our artifacts. The remaining leakage surface is overlap between constructed inputs and official tests, audited directly in §7. How the auditor constructs a suite. Five off-the-shelf coding-agent engines provide the auditing arms: codex: OpenAI Codex CLI [19] on GPT-5.4 [20]; claude: Claude Code [1] on Claude Opus 4.8 [2]; agy: Antigravity CLI [9] on Gemini 3.1 Pro [10]; opencode: OpenCode CLI [21] on DeepSeek V4 Pro [7]; and mini: mini-SWE-agent [28] on DeepSeek V4 Pro under a minimal scaffold. The task brief is identical across engines, and per-engine reasoning-effort settings are in §A. CodeContests+ fixes two generationâvalidation pipeline agent roles [26]. Each engine chooses its construction strategy. What makes the comparisons fair. AtCoder evaluates a five-engine panel, but its certified ledger uses one arm (codex); CF-fresh, the post-cutoff setting, runs a single engine. No pooling occurs across compositions. Construction strategy may vary, but every evaluated suite is scored with the same judging harness and scoring basis. Experiment 1 compares agent-constructed and re-judged official suites with random generation matched to the agentâs input budget. Experiment 2 compares agent-constructed suites with five reproduced baselines on a single matrix of output disagreements and scores only validator-passing inputs, pairing scores per problem. The mechanism analysis gives random the same input budget (§6). How a disagreement becomes a certificate. A shared scoring basis makes an output disagreement comparable, but not yet a bug. A test input kills a submission when the submissionâs output disagrees with the expected output. The expected output comes from a consensus oracle over independently accepted human solutions. On AtCoder, agreement among 8 reference solutions per problem defines the expected output; on Codeforces, agreement among the outputs of all successful runs, after all three accepted reference solutions are run, defines the expected output, provided that at least two runs succeed. §6 audits AtCoder-oracle conservatism and Codeforces-oracle soundness. Because one human reference is shown during construction, a natural concern is whether it is held out from the oracle. It is not: the reference shown to each engine is the first of the eight. Suite construction can optimize against one voter, but one solution cannot form the reference majority. Certification still requires the killed submission to disagree with that majority. Disputes arise when the killed submission agrees with a larger re-judged pool or when the gold, the oracle expectation, is contested. Brute-force solutions adjudicate them. Every kill is re-verified deterministically on a fixed Linux judging harness. Why legality is part of the certificate. An audited oracle is insufficient for inputs outside the statement. The final logical link in the chain is a strict per-problem validator. The harness independently checks every certifying input, whether disputed or not. An input enters scoring only if the validator accepts it. For AtCoder, the validator is an agent-produced artifact written by an LLM from the statement alone. It is blind to the killing inputs it will later judge. This makes the validator independent of the kills it gates. Before use, the AtCoder validator undergoes an instrument-level sanity check, separate from adjudicating constructed findings: it must accept every official input and reject the empty input. If a problemâs validator fails this quality gate, every kill row for that problem is scored unmeasured and excluded from the legality count. No such row is scored illegal. Under this gate, legality means validator-passing under the encoded statement constraints. For an AtCoder topological-sort problem that promises a DAG, a cyclic input is rejected and cannot refute a solution [26]. The CF-fresh protocol takes the testlib gate from the reproduced CodeContests+ baseline toolchain and applies the same scoring rule to every arm. §6 audits this legality check. Because shipped dataset tests do not validate inputs, legality is an explicit part of the certificate [26]. Across both experimental regimes, configurations differ, but the certification pattern remains the same. 4 Experiment 1: Auditing AtCoder Official Suites Experiment 1 audits two populations against existing AtCoder suites, carrying forward legality from the §3 certifi- cate. For accepted submissions, the certified codex ledger contains 589 accepted-but-buggy submissions missed by the official suites in the audited sample; the five-arm legality- and tolerance-clean floor is 906. For rejected submis- sions, a five-engine panel measures official-suite logic-bug recovery; its largest engine-specific shortfall is 1.7p. Setup: separating the two audit populations. The universe is 106 AtCoder problems with CodeNet human submissions [22]. Differential vetting cross-checks accepted outputs, excluding multi-answer and special-judge problems; initial scoring uses exact comparison. Per problem, we take the first 200 accepted and 300 rejected compilable C++ submissions. The accepted sample contains 20,375 submissions, including all from problems with fewer than 200. Its ledger uses only the codex arm and full certification chain. Official-suite-rejected 4 Coding Agents as Test-Suite AuditorsPreprint submissions codex-arm logic-bug exclusives746 â official TLE catches10 = ledger candidates (no official catch)736 â nondeterministic / unreproduced7 = raw deterministic kills729 â consensus corrections (80-voter; 123 full-pool)125 â tolerance false positives15 = verified accepted-but-buggy589 wrong-answer / runtime-error545 / 44 problems covered74 / 106 Table 1: Audit funnel from codex-arm logic-bug exclusives on the accepted pool to the verified accepted-but- buggy ledger. Official TLE catches reconcile that scoring count; reproducibility filtering precedes the two artifact corrections. submissions form the known-bug panel (REJ). All five engines are scored separately, never pooled. Engine-specific and random suites use a 50-input cap; official suites retain baseline sizes. One Linux judging harness scores all. Sample scope is in §7; full protocols in §A. What survives correction into the ledger. Within this accepted sample, codex suites expose 589 verified accepted-but-buggy submissions missed by the official suites. The 20,375 audited submissions cover 8.9% of these problemsâ C++ accepted submissions. Within that sample, the ledger rate is 2.9%. The ledger comprises 545 wrong-answer and 44 runtime-error submissions across 74 problems. Each fails on a legal input under an audited expected output. The funnel begins with the codex-arm logic-bug exclusives, submissions killed as logic bugs by its suites but not by official suites (Table 1); official TLE reconciliation and reproducibility filtering leave 729 de- terministic flags. An 80-submission sample from the same per-problem accepted pool cross-checked the reference oracle by voting on each flag. A majority favoring the killed submission made that oracle the outlier, withdrawing 125 flags. The full 200-submission pool confirmed the 123 of these reversals that fall in one problem. Tolerance re- moved 15 floating-point artifacts. Post hoc, independently written per-problem reference solvers re-adjudicated all 589 ledger entries in isolated scratch without access to repository oracle data, using 188 distinct killing inputs; the reviewing model saw neither expected/got values nor buggy source for wrong-answer rows (§A). All 44 runtime- error rows received separate static crash attribution, and the re-adjudication disagreed with 0 oracle outputs. The same gates across all five arms. The arm-general chain runs deterministic re-judge, consensus correction, legality validation, and tolerance filtering. Within the audited sample, every arm receives the codex ledgerâs legality and tolerance gates; they remove no codex rows, so its count remains unchanged. Certified-clean counts are 589 (codex), 597 (claude), 576 (agy), 514 (opencode), and 528 (mini). Their deduplicated union is a legality- and tolerance-clean floor of 906 accepted-but-buggy submissions. The gates drop 1094 submissions whose every killing input is illegal and remove 24 floating-point false positives; because dropped candidates are not regenerated, the union is a floor. Census confirmation covers 905 of the floor across every armâs certified rows, not only the codex ledger. A signed, human-adjudicated stratified sample of 203 findings had 0 overturns: the coverage layer sampled 202 rows within problem with caps and inverse-probability weighting, and the single risk-layer row was the one left unconfirmed by the census. The coverage layerâs false-discovery rate is bounded at 1.87% (Wilson 95% upper); the full protocol is in §A. The floor comprises the codex ledgerâs 589 submissions plus 317 added by the four other arms but not certified by codex, totaling 906. Known-bug coverage is close, with an official edge. REJ recovery probes performance on a broad official-known-bug panel. Time-limit verdicts are excluded from all bug claims (§A). Slow accepted solutionsâ bug status depends on unrecoverable setter intent. Micro-pooled coverage is 0.919â0.932, versus 0.936 for re-judged official suites. 3 The official rate remains higher, but every engine stays within 1.7p; agent-cap-matched random generation falls 10.9p short (Figure 2). These rates are descriptive; macro-recall, not this pooled rate, underlies 3 The official rate 0.936 falls below full recovery of its own REJ pool because of scoring and harness effects, not missed bugs. Most of the gap is submissions the official inputs catch as timeouts, which the excl-TLE basis credits to no arm; §A gives the decomposition and a sensitivity check. 5 Coding Agents as Test-Suite AuditorsPreprint 0.8250.8500.8750.9000.9250.950 known-bug (REJ) coverage official claude codex agy opencode mini random 0.936 0.932 0.930 0.925 0.922 0.919 0.827 max engine gap 1.7p Figure 2: Micro-pooled logic-bug coverage (26,682 REJ submissions; 106 AtCoder problems; TLE-only excluded; agent and random suites: 50-input cap; official suites: baseline sizes). Every engine tracks the re-judged official suites closely, while capped random generation trails. the noninferiority test. Without formal preregistration, a pre-execution plan designated codex and claude confirmatory, and 2 engines clear the formal test (§A); the five-engine result is descriptive, not formal. Inside the ledger: ruling out a few-template explanation. The known-bug comparison tells against official leniency as the ledgerâs source; its 589 verified entries also do not reduce to a few source-level near-clone groups: mechanical source-token clustering yields 578 near-clone-distinct clusters among those findings. The clustering criterion and transcribed examples are in §C. Official-test input overlap is audited in §7. The official suites remain the stronger single judge. Sharing a logic-bug basis, the exclusive counts answer opposite cross-pool questions. On REJ, official suites reject 1136 submissions missed by codex suites. On accepted submissions, the converse codex-arm count opens the funnel in Table 1; 589 survive certification. The counts show complementary directions, not comparable magnitudes. Expert-built official suites remain the human baseline and stronger single judge. The certified codex ledger and five-arm floor complement rather than replace official suites. The next experiment starts where official-suite availability ends. 5 Experiment 2: Post-Cutoff Head-to-Head on Fresh Codeforces Without access to an official hidden suite, this section compares supplied suites head-to-head. The agent arm leads every reproduced baseline at every tested input budget. The supply setting. The benchmark is CF-fresh: 41 Codeforces problems from completed official rounds harvested under a March 1âJune 9, 2026 date rule, intended to postdate the suite-building engineâs declared training-data cutoff: Jan 2026, which precedes the earliest included problem, dated March 8, 2026. The platform releases statements and sample tests, not full suites. A fixed-seed, content-blind harvest applies date, rating-band, and tag rules, yielding 48 problems; rule-based exclusion of 7 multi-solution or special-judge cases leaves 41. Freshness removes the direct memorization channel; residual controls are in §7. Buggy-pool membership and comparison arms. Among 1,964 multi-model LLM solver attempts on 41 problems, 596 compile and pass all public samples. The six arms are the agent arm (one engine, claude) and five reproduced baselines: official sample tests, cheap-random generation matched to the agentâs input budget, the CodeContests datasetâs mutation-based generator protocol [14], CodeContests+ [26], and an EvalPlus-style generator [15]. A sample-passing solution enters the pool when some armâs legal input exposes it and the three-reference consensus oracle of §3 certifies the failure; the pool is the union of the armsâ certified findings. §6 audits the oracleâs soundness. The pool contains 63 solutions with logic bugs, not timeouts. The official 6 Coding Agents as Test-Suite AuditorsPreprint 1102030405060 input budget k (legal inputs) 0.0 0.2 0.4 0.6 0.8 1.0 mean subset cov@ k design budget k=50 0 by construction 135 0 0.04 same gap, k †5 gap over best baseline agent (ours) CodeContests+ EvalPlus-style CodeContests cheap-random official (sample tests) Figure 3: CF-fresh subset coverage cov@í(order-free) for all six arms. Coverage averages over 63 adjudicated buggy solutions. The official arm comprises the sample tests, so its zero coverage is by construction: all pool members pass them. The inset magnifies the gap foríâ€5; the dotted line marks the agentâs 50-input design budget, beyond which its full set is used. sample arm has zero coverage by construction because every pool member passes the public samples. It records a builderâs pre-generator suite. All arms score only legal inputs under the shared validator. Fair comparison basis. We reproduce the five baselines end-to-end in one kill matrix, paired by problem. Coverage is cov@í, the order-free expected pool fraction killed by a uniform randomí-subset of an armâs legal inputs for that problem. An arm with fewer thanílegal inputs contributes its whole set. On the union pool, cov@ígives a relative ordering of the compared arms, not an absolute rate. Figure 3 shows the curves; §B reports the exact cov@í ladder and per-arm input counts. Lead at every input budget. The agent arm catches more of the CF-fresh buggy pool than every reproduced baseline (Figure 3), with the lead holding at everyí â [1,60]with no crossover. The minimum gap is 0.018 atí=1. At the design budgetí=50, the agent finalizes 50 inputs on 40 of the 41 problems (30 on the remaining one). A per-problem mean of 43.8 inputs survives both consensus and legality gatesâthe fewest among the generator arms, whose post-top-up means span 49.0â54.1 (§B). At that budget, agent coverage is 0.952 versus 0.809 for CodeContests+, the strongest baseline; equivalently, its miss rate is 4.01Ăthe agentâs. That budget was fixed by design, not by the curves; we do not headline cov@60 because the agentâs finalization cap is 50 inputs. Budget-dependent separation. All-budget ordering does not imply per-budget separation. Under the canonical bootstrap seed, the paired cluster-bootstrap delta against CodeContests+, the strongest baseline, is 0.143 at the design budget; its 95% confidence interval[0.028,0.331]excludes zero. All 12 robustness-seed intervals also have positive lower bounds. At the smallerí=20, the delta is 0.129, with CI[â0.011,0.351]. That interval crosses zero. The small-budget lead is directional and under-powered, and we do not claim separation there. Intervals against CodeContests, the EvalPlus-style generator, and cheap-random exclude zero at both budgets. Within-model workflow check. The coverage lead leaves one alternative explanation: the base model rather than the agentic workflow. For this check the agent arm is mini, on a separately re-adjudicated subset, not the claude arm or the main tableâs pool. We test a single-call arm using the same base model in four independent reps. No execution result is returned to the model, and buggy-pool membership is frozen. After adding the arm, we re-execute the kill matrix on the 19 problems containing 62 buggy solutions; §B tabulates it. We judge both arms identically. The comparison varies the wrapper, including tool and iteration affordances and the prompt contract needed to use them; the matched dimensions and the prompts themselves appear in §D. Atí=50, mini reaches 7 Coding Agents as Test-Suite AuditorsPreprint cov@50=0.903, versus 0.665 for the single-call arm, which misses 3.46Ăas many buggy solutions. On the validator-legal basis, the paired mini-minus-single delta is 0.238 ([0.074,0.417]) atí=50. A deployable four-call 50-slot mixture interleaves the four independent samples by position within the same input budget, without held-out selection. For this ladder, CodeContests+ is replicated on the same base model and averaged over four sampling replicates; it is a different pool and protocol from the CodeContests+ arm above. Atí=50, miss rates are 0.586 for CodeContests+, 0.335 for mean single-call, 0.210 for the mixture, and 0.097 for mini; per-replicate CodeContests+ coverage varies widely (§B). The mixture closes 53% of the observed single-callâmini gap. In this replication, CodeContests+ misses 6.05Ăas many buggy solutions as mini. Both paired contrastsâmini against mean single-call and against CodeContests+âexclude zero at both budgets; the mixture interval crosses zero. Thus, at the fixed 50-input deployment budget and for this four-replicate realization, no-oracle mixing weakens but does not exclude a sampling-diversity explanation. The held-out, per-problem oracle best-of-íupper bound appears in §B. The shortfall is not an inability to construct large cases. Without returned execution results, a single call receives no signal about whether its outputs pass the validator; validator-legal rates, maximum-scale input shares, and the zero-legal replicates are reported in §B. The control also does not match inference spend (§7). Within this pairing, the coverage edge tracks the wrapper as a whole rather than a different base model. 6 Why the Agent Suites Catch So Many: Artifact Audits and Diversity Correlates Two separate questions are whether the kills are real in both experiments and what accounts for Experiment 1âs volume and Experiment 2âs continuing gains. The evidence supports two answers. The kills survive direct oracle and legality checks. The volume and continuing gains are consistent with diversity across the input sets, not one unusually strong probe. What survives the oracle audits. The first artifact route is a wrong expected output. On CF-fresh, an indepen- dent panel of sample-passing LLM solutions votes on each gold (the oracleâs expected output) for the 1,795 agent- arm inputs carrying a usable gold; baseline-arm inputs are excluded. A case is contested whenever the largest non- gold vote count exceeds the goldâs, a rule met by 15 cases. The audit resolves all in the oracleâs favor, leaving zero demonstrated wrong golds. Each is therefore a false alarm: several LLM solutions share the same wrong answer and out-vote a correct gold from three system-test-passing human solutions. Brute-force-feasible cases are con- firmed directly; the others on structural grounds. §B reports the split; the conclusion is confined to that oracle and those problems. The correction funnel in §4 establishes AtCoder kill soundness; the remaining calibration question is whether an 8-reference consensus is more machinery than the result needs. It is conservative by design. When the same kills are re-scored under smaller reference consensuses, a single reference retains only 0.925 of the 589 certified kills, whereas three references already retain 1.000; the full 8-reference consensus is more than needed. The sensitivity is in wrong-answer kills; runtime-error kills are unchanged across the tested reference counts. Legality checks on the input side. After the oracle checks, a second apparent-kill route remains: a statement- forbidden input. Such a failure proves nothing. Here we measure the legality gate of §3. Independent re-verification finds 589 of the 589 certified AtCoder killing inputs passing strict per-problem statement validators, with zero illegal and zero unmeasured. By construction, CF-freshâs kill matrix scores legal-only inputs; the gate is applied on 41 of 41 problems. Each AtCoder agent-produced validator uses only the statement, never sees killing inputs, and before use must accept every official input and reject the empty input; all 74 ledger problems pass that quality gate. Under the legality definition of §3 (legality means validator-passing under the encoded statement constraints), the checks above rule out validator-judged illegal inputs as the source of these kills, addressing the failure mode in shipped dataset tests [18,26]; the validatorsâ own soundness against malformed inputs is audited separately in §7. Saturation locates diversity in the set. After both artifact checks, validity gives way to volume. The regimes use different proxies for the same construct, input-set diversity: TCE gives the saturation shape, and AtCoder trajectories show set construction. The TestCase-Eval (TCE) known-bug setting traces input-by-input coverage accumulation, a saturation profile the two main experiments do not expose [29]. It compares the agent input set with cheap-random generation matched to that input budget. The sweep shows cheap-random front-loading coverage at normalized cov@1 0.65 versus the agent at 0.32, then flattening so that the tail ordering reverses: per-input marginal coverage is 0.026±0.0046 for the agent versus 0.004±0.0013 for cheap-random. The no-feedback control tests whether this tail advantage depends on execution feedback and gives a comparable per-input marginal coverage of 0.025±0.0043; the TCE sweep design, full curves, ratios, and per-problem tail marginals are in §B. Computed on the unrounded tail marginals, the agent hasâŒ5.8Ăhigher per-input marginal coverage. This is a saturation-shape result, not a measurement of pairwise input disjointness. 8 Coding Agents as Test-Suite AuditorsPreprint Strategy presence and co-occurrence in the trajectories. In the audited AtCoder trajectories from Experi- ment 1âs codex arm, set assembly is better described as multi-strategy construction than bulk randomness (§A, §C). How the two answers fit the experiments. The oracle audits and legality certification close the artifact routes; the observed volume is consistent with diversity across the input sets. Experiment 1âs findings are dispersed, forming 578 source clusters and 224 bug-class families rather than one dominant exploit; the largest single problem contributes only 15.2% of the clusters (§C). Experiment 2 shows no crossover as suite size grows (§5). At í=1, cheap-random leads in TCEâs human-wrong-submission pool; CF-freshâs sample-passing-LLM-solution pool has no crossover. We do not offer a mechanism-level account linking the composition difference to the observed crossover behavior. The agent sets continue adding coverage in both pools. The diversity evidence is associative, not causal; the TCE mechanism estimate does not transfer mechanically beyond the studied population. 7 Threats to Validity and Limitations Sample scope. The deterministic sample contains each problemâs first 200 compilable submissions (8.9% of the accepted pool; §4). The 589 count remains an in-sample lower bound, not a platform rate; we do not extrapolate beyond this slice. Matching a future audit to the platformâs natural submission distribution would require CodeNetâs full distribution. Compared armsâ certified-finding union bounds the CF-fresh buggy pool; arm-independent pool construction remains open. Rebuilding the pool without the agentâs certified findings leaves it unchangedâevery entry is caught by at least one non-agent arm. The chain assumes competitive-programming structureâaccepted-solution consensus oracle, statement-derived validator, exact-output judgingâand has yet to be extended to specification-only tasks or tasks without accepted solutions to seed the oracle. What the legality gate is, and is not. Validator-passing does not establish semantic ground truth: quality-gated validators (§6) encode statement-declared constraints, but ambiguity can remain beyond their reach. Malformed- input gate soundness was audited structurally (trailing-token, truncation, and non-numeric mutations to official legal inputs: 0 false accepts) and semantically (format-valid but illegal inputs: 0 false accepts). A validator-blind, different-family model constructed both tracksâ fuzzing inputs, preventing illegal-input admission via a shared generatorâvalidator blind spot. Validator misclassification is one-directional: only false accepts could produce false kills; over-strictness merely under-counts. §A reports the fuzzing taxonomy, per-track counts, and an after- the-fact, single-author, non-blind manual problem-statement check of the then-present AtCoder and CF-fresh validators, finding no constraint-encoding errors; this attestation does not establish semantic ground truth. Cost. Budget matching equalized retained-input counts, not spend; per-problem medians are minutes-scale (§A). Contamination. AtCoder problems predate the enginesâ training cutoffs, so statement memorization remains possible in Experiment 1; Experiment 2 uses post-cutoff problems to remove that training-time channel. The audit concerns the suite, not the model: âthe official suite accepts these buggy submissionsâ does not depend on how the killing input was conceived. Experiment 1 measured agent-generatedâofficial-human-setter input overlap; the five AtCoder enginesâ highest non-trivial rate is 5.5%. Construction occurs in per-run child working directories; the held-out buggy-submission pool, official test I/O, and CodeNet AC pool remain physically outside them and are not provided as task inputs. A post-hoc audit of the recorded shell and message actions from all five engines, checking filesystem access to held-out materials and official test data, found 0 access-level leaks over 529 scanned agent runs. 8 Conclusion Given only a problem statement and one reference solution, off-the-shelf coding agents can audit test-suite adequacy. codex finds 589 verified accepted-but-buggy submissions among 20,375 audited; identically gated, the five-arm union floor reaches 906, including 317 not certified by codex. Scored separately, all five engines stay within 1.7p of the re-judged official suites on the logic bugs those suites catch (§4). As a single-engine judge for fresh problems without official suites, it leads every reproduced baseline at all tested input budgets. Audited oracles underpin both regimes; every certified killing input passes a per-problem legality validator (§6). Machine-checkable certificates in a re-checkable ledger support a deployable second judge. Suite adequacy becomes audited rather than assumed on reputation. 9 Coding Agents as Test-Suite AuditorsPreprint References [1] Anthropic. Claude Code: An Agentic Coding Tool for the Terminal.https://github.com/anthropics/ claude-code, 2025. [2] Anthropic. Claude Opus 4.8. https://w.anthropic.com/news/claude-opus-4-8, 2026. [3]Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Kangwen Zhao, Dongyun Xue, Mingxiao Feng, Wengang Zhou, and Houqiang Li. Codecontests-o: Powering llms via feedback-driven iterative test case generation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics (ACL 2026), pages 1054â1072. Association for Computational Linguistics, 2026. URL https://aclanthology.org/2026.findings-acl.53/. [4]Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, Yixuan Tan, Jiafu Tang, Yilin Tang, Junlei Wu, Qianyu Xiao, Can Zheng, Shouchen Zhou, Yuxiang Zhu, Yiming Huang, and Tianxing He. Can llms generate reliable test case generators? a study on competition-level programming problems, 2025. URLhttps://arxiv.org/ abs/2506.06821. [5]Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations (ICLR 2023). OpenReview.net, 2023. URL https://openreview.net/forum?id=ktrw68Cmu9c. [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv.org, 2021. [7]DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026. URL https://arxiv.org/abs/2606.19348. [8]Jia Fu, Xinyu Yang, Hongzhi Zhang, Yahui Liu, Jingyuan Zhang, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Klear-codetest: Scalable test case generation for code reinforcement learning, 2025. URLhttps: //arxiv.org/abs/2508.05710. [9]Google. Google Antigravity: An Agentic Development Platform.https://antigravity.google/, 2025. [10] Google DeepMind. Gemini 3.1 Pro Model Card.https://deepmind.google/models/model-cards/ gemini-3-1-pro/, 2026. [11]Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. Hardtests: Synthesizing high-quality test cases for llm coding, 2025. URLhttps: //arxiv.org/abs/2505.24098. [12] Han Li, Jinyu Tian, Rili Feng, Yuqiao Du, Chong Zheng, Chenyu Wang, Chenchen Liu, Shihao Li, Xinping Lei, Yifan Yao, Weihao Xie, Letian Zhu, and Jiaheng Liu. Solvita: Enhancing large language models for competitive programming via agentic evolution, 2026. URL https://arxiv.org/abs/2605.15301. [13] Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset, 2023. URL https://arxiv.org/abs/2312.14852. [14]Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, RĂ©mi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son dâAutume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Fre- itas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Sci- ence, 378(6624):1092â1097, December 2022. ISSN 1095-9203. doi:10.1126/science.abq1158. URL http://dx.doi.org/10.1126/science.abq1158. [15]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, 10 Coding Agents as Test-Suite AuditorsPreprint Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. URLhttp://papers.nips.c/paper_files/paper/ 2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html. [16] Kaibo Liu, Yudong Han, Jie M. Zhang, Zhenpeng Chen, Federica Sarro, Mark Harman, Gang Huang, and Yun Ma. Who judges the judge: An empirical study on online judge tests. ISSTA 2023 (32nd ACM SIGSOFT International Symposium on Software Testing and Analysis), 2023. doi:10.1145/3597926.3598060. [17] Kaibo Liu, Yudong Han, Yiyang Liu, Zhenpeng Chen, Jie M. Zhang, Federica Sarro, Gang Huang, and Yun Ma. Trickybugs: A dataset of corner-case bugs in plausible programs. In Proceedings of the 21st International Conference on Mining Software Repositories, MSR â24, page 113â117. ACM, April 2024. doi:10.1145/3643991.3644870. URL http://dx.doi.org/10.1145/3643991.3644870. [18]Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M. Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang. Llm-powered test case generation for detecting bugs in plausible programs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 430â440. Association for Computational Linguistics, 2025. doi:10.18653/V1/2025.ACL-LONG.20. URL https://doi.org/10.18653/v1/2025.acl-long.20. [19]OpenAI. Codex CLI: A Lightweight Coding Agent for the Terminal.https://github.com/openai/ codex, 2025. [20]OpenAI. GPT-5.4 Thinking System Card.https://deploymentsafety.openai.com/gpt-5-4- thinking/gpt-5-4-thinking.pdf, 2026. [21]OpenCode Contributors. OpenCode: An Open-Source AI Coding Agent for the Terminal.https:// opencode.ai/, 2025. [22]Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021), 2021. URLhttps://datasets-benchmarks-proceedings.neurips.c/paper/2021/ hash/a5bfc9e07964f8dddeb95fc584cd965d-Abstract-round2.html. [23]Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, and Shengyu Tao. Codehacker: Automated test case generation for detecting vulnerabilities in competitive programming solutions. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, pages 2352â2382. Association for Computational Linguistics, 2026. URL https://aclanthology.org/2026.acl-long.108/. [24]Sicheol Sung, Aditi, Dogyu Kim, Yo-Sub Han, and Sang-Ki Ko. Logicase: Effective test case generation from logical description in competitive programming. In Proceedings of the Thirty-Fourth International Joint Confer- ence on Artificial Intelligence (IJCAI 2025), pages 7742â7750. ijcai.org, 2025. doi:10.24963/ijcai.2025/861. URL https://doi.org/10.24963/ijcai.2025/861. [25] Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving LLM coder and unit tester via reinforcement learning. In Danielle Belgrave, Cheng Zhang, Laura N. Montoya, Hsuan-Tien Lin, Razvan Pascanu, Piotr Koniusz, Marzyeh Ghassemi, Nancy Chen, IvĂĄn Vladimir Meza RuĂz, and Arturo Loaiza-Bonilla, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. URLhttp://papers.nips.c/paper_files/ paper/2025/hash/d38653cdaa8e992549e1e9e1621610d7-Abstract-Conference.html. [26]Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. Codecontests+: High-quality test case generation for competitive programming. Conference on Empirical Methods in Natural Language Processing, 2025. doi:10.48550/arXiv.2506.05817. [27]Tingqiang Xu, Hangrui Zhou, Tianle Cai, Alex Gu, and Kaifeng Lyu. Beyond problem solving: Uoj-bench for evaluating code generation, hacking, and repair in competitive programming, 2026. URLhttps: //arxiv.org/abs/2606.12864. [28]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. URLhttp://papers.nips. 11 Coding Agents as Test-Suite AuditorsPreprint c/paper_files/paper/2024/hash/5a7c947568c1b1328c5230172e1e7c-Abstract- Conference.html. [29]Zheyuan Yang, Zexi Kuang, Xue Xia, and Yilun Zhao. Can llms generate high-quality test cases for algorithm problems? testcase-eval: A systematic evaluation of fault coverage and exposure. Annual Meeting of the Association for Computational Linguistics, 2025. doi:10.48550/arXiv.2506.12278. [30]Shengyu Ye, Qi Liu, Hao Jiang, Zheng Zhang, Heng Yu, and Zhenya Huang. Themis: Automated constraint- aware test synthesis framework for code reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34432â34440, 2026. [31]Shang Zhou, Zihan Zheng, Kaiyuan Liu, Zeyu Shen, Zerui Cheng, Zexing Chen, Hansen He, Jianzhu Yao, Huanzhi Mao, Qiuyang Mang, Tianfu Fu, Beichen Li, Dongruixuan Li, Wenhao Chai, Zhuang Liu, Aleksandra Korolova, Peter Henderson, Natasha Jaques, Pramod Viswanath, Saining Xie, and Jingbo Shang. Autocode: Llms as problem setters for competitive programming, 2025. URLhttps://arxiv.org/abs/ 2510.12803. 12 Coding Agents as Test-Suite AuditorsPreprint Appendix A Audit Protocol Details (AtCoder) This appendix gives the full protocol behind Experiment 1 (Section 4): the submission sampling and the two audit populations, how the consensus oracle is built and audited, the judging environment, and the per-entry schema of the released ledger. Numbers shared with the main text re-use the same macros; per-engine detail, costs, the full self-gap decomposition, and some protocol parameters are reported only here. Problem selection. The study problems come from a 126-problem candidate pool (AtCoder ABC D/E/F tasks in the mid-difficulty band, present in both the AtCoder mirror and CodeNet [22]), narrowed to the 106 with a usable reference pool and unambiguous exact-match verdicts. Multi-answer and special-judge tasks are removed at selection time, so exact output comparison is the intended verdict basis throughout. Submission sampling and the two audit populations. The universe is 106 problems with CodeNet human submissions [22]. Per problem, we take the first 200 accepted and the first 300 rejected compilable C++ submissions. The accepted sample contains 20,375 submissions, includes every accepted submission from problems with fewer than 200, and covers 8.9% of these problemsâ C++ accepted submissions. Accepted-but- buggy findings come from this pool, whose ledger uses only the codex arm under the full certification chain. Officially rejected submissions form the separate known-bug panel (REJ), scored on the pool defined below rather than on the whole sample; the five engines are scored separately and never pooled. Engine-specific and random suites use a 50-input cap. Official suites retain their baseline sizes. Consensus oracle and its audit. The oracle draws on 848 accepted human reference solutions across the 106 problems, 8 per problem, all source-level distinct after near-clone deduplication (848 clusters). The expected output for an input is the value the references agree on; a submission is killed when its output disagrees with that majority, or when it exits with a runtime error rather than an output. Two properties are audited. Conservatism: on the 589 certified kills, a single reference recovers only 0.925 of them, while three already recover 1.000 and five and eight both hold at 1.000. A lone reference is too few; eight is more than needed. Soundness: the correction funnel appears in Section 4 and its audit-anatomy table. In the single-problemabc164_econcentration, all 123 retractions were checked without sampling against the full 200-submission pool. Disputes too large for pool re-judging use brute-force adjudication. Census re-adjudication. After the openly-corrected pipeline closed, every one of the 589 ledger findings was independently re-adjudicated. Independently written per-problem reference solvers, validated on the official samples before use, ran on 188 distinct killing inputs for wrong-answer submissions. They ran in isolated scratch without access to repository oracle data. Their prompts omitted expected and got values, bug class, and DET class (the candidateâs stability on its killing input across three re-runs on the Linux judging host:DET_WAfor the same wrong normalized output throughout,DET_REfor a runtime error throughout,NONDETfor instability, andNOT_REPRODUCEDif no killing input reproduces the kill; onlyDET_WAandDET_REenter the ledger). Wrong-answer review saw the statement and killing inputs but not buggy source. Runtime-error review saw source only for crash attribution, with no expected output. The authors compared outputs; the reviewing model did not see the result. The 16 rows whose outputs exceed the ledgerâs 80-character display window were resolved by full-output comparison against re-run buggy binaries. All 44 runtime-error submissions received static crash attribution (mechanism and trigger named, input legality confirmed). The solvers disagreed with 0 oracle outputs and had zero conflicts with the earlier pre-registered stratified sample over the codex armâs pre-audit deterministic ledger: its coverage layer was a per-problem random draw capped at three per problem, with 0/163 adjudicated rows overturned, yielding an independent coverage-layer FDR bound of 2.3% (Wilson 95% upper); its risk layer exhaustively audited the multi-solution risk classes, tie-breaking and precision, in which more than one output can legitimately be correct, with 0/56 overturned. The protocol tests shared-error risk rather than assuming it away: onabc145_e, the reference solver shared the buggy submissionsâ misreading of the statement and was refuted by a dual-route brute-force witness. Disputed problems route to brute-force adjudication, and rows retracted by the earlier 200-pool and tolerance audits are counted separately, the upstream correction mechanisms working as designed. Judging environment. Every authoritative kill is re-judged deterministically on a single Linux host; numbers observed on other platforms during development never enter a claim. The harness enforces each problemâs own time limit plus a 0.5 s scheduling allowance, and it removes the default stack limit, matching judge semantics 13 Coding Agents as Test-Suite AuditorsPreprint so that deep-recursion accepted solutions are not spuriously killed. Verdicts are bucketed by type, and time- limit-exceeded catches are held in a separate bucket excluded from every bug claim in the paper (the excl-TLE basis). Validator soundness audit. The per-problem legality validators (Section 7) are audited against malformed inputs on two fronts, both built by a validator-blind different-family model so a shared blind spot cannot pass. Structurally, across 25 problems, 354 structurally illegal mutations of official inputs (trailing tokens, truncation, non-numeric fields) drew 0 false accepts. Semantically, across 18 problems, 62 format-valid-but-illegal inputs (a duplicate in a permutation, a cycle in a tree, a multi-edge in a simple graph, a broken connectivity or distinctness constraint) drew 0 false accepts. The gate rejects the malformations it exists to catch, and its error is one-directional: only a false accept could buy a false kill, while over-strictness merely under-counts. Complementing these machine audits, an author manually reviewed every per-problem validator present when the review was recordedâthe AtCoder agent-written validators and the CF-fresh testlib validatorsâagainst its problem statement, checking whether each statement-declared input constraint was encoded; the review found no constraint-encoding errors. This single-author, non-blind review was recorded after the fact and is an attestation, not an establishment of semantic ground truth. Cost. Budget matching equalized retained input counts per problem, not spend. Measured per-problem medians on the AtCoder panel are 30.5k output tokens / 12.9 min (codex) 4 , 21.7k / 5.8 min (claude), 26.3k / 22.8 min (opencode), and 9.1 min (mini). The CF-fresh agent arm spends 27.3k tokens and 7.7 min over 30 turns per problem. agy exposes no token or call accounting and no usable wall-clock, because its CLI records a narration-only transcript with no structured events; the raw spawn-to-last-write span includes requeue and watchdog waiting, so it is not registered as a generation wall-clock. mini exposes call counts but not token accounting. The script and reproduced-suite arms carry only reproduction-side cost. Per-engine known-bug coverage detail. Table A.1 expands the known-bug guard of Section 4 to the five engines individually. The excl-TLE official-suite-rejected (REJ) pool contains the 26,682 historical submissions that at least one armâs re-judged suite kills through a wrong-answer or runtime-error verdict; submissions no arm kills that way stay outside the pool, and within it every arm is credited only for its own wrong-answer and runtime-error catches. On this pool, official coverage 0.936 leaves a self-gap of 1696 submissions (6.4% of the pool). Of this gap, 90.9% is TLE masking: the official inputs still kill these submissions, but only with a time-limit-exceeded verdict, which the excl-TLE basis credits for no arm. The remaining 154 submissions (0.58% of the pool) are verdict drift: under the judging environment above, the official inputs no longer kill them at all, while at least one other armâs suite does. Dropping the drift submissions raises official coverage and widens the largest engine-to-official gap from 1.7p on the full pool to 2.1p; official coverage retains a small edge under both treatments. Noninferiority is tested per engine on a per-problem macro-recall basis against the re-judged official suites, with parameters fixed in advance (marginÎ=0.02,í”=10,000 bootstrap resamples, seed 20260619; no formal preregistration). An engine is noninferior at its single-engine 95% interval when the intervalâs lower bound stays aboveâÎ. The 2 confirmatory engines (codex, claude) clear this bar. Given the internal designation, Holm pass counts are 2 for the confirmatory pair and 0 across five engines. Random generation matched to the agentâs input budget (0.827 coverage) is well outside it. Input legality on the known-bug panel. The panel scores every armâs inputs as generated: unlike the accepted- pool chain, it applies no legality gate at scoring time. Re-running the same per-problem validators after the fact over every panel input covers the 103 of 106 panel problems that carry a validator. The per-engine share of illegal inputs runs from 0.08% to 3.79%, and the random arm sits at 1.67% â inside the engine range, so neither side of the engine-versus-random contrast is systematically favored. These are input-level shares, and they bound the exposure rather than correct the coverage rates: a submission is usually killed by several inputs, so a kill survives a legal-only basis whenever one legal input kills it. Certified per-arm contribution and isolation. Separate from the REJ coverage comparison, the arm-general certification chain yields certified-clean counts of 589 (codex), 597 (claude), 576 (agy), 514 (opencode), and 528 (mini). Their submission-deduplicated, legality- and tolerance-clean union is a floor of 906 accepted-but- buggy submissions: 1094 candidate submissions whose every killing input was illegal were dropped as invalid 4 The five auditing engines are defined in Section 3; each pairs an off-the-shelf coding-agent CLI with a base model: codex (OpenAI Codex CLI [19], GPT-5.4 [20]), claude (Claude Code [1], Claude Opus 4.8 [2]), agy (Antigravity CLI [9], Gemini 3.1 Pro [10]), opencode (OpenCode CLI [21], DeepSeek V4 Pro [7]), and mini (mini-SWE-agent [28] on the same DeepSeek V4 Pro). Reasoning-effort settings are xhigh for codex and claude, high for agy, and thinking-high for opencode and mini. 14 Coding Agents as Test-Suite AuditorsPreprint enginemicro-pooled cov. macro-recallÎ95% CI codex0.930-0.004 [â0.017, 0.009] claude0.932-0.003 [â0.018, 0.011] agy0.925-0.011 [â0.027, 0.005] opencode0.922-0.013 [â0.035, 0.007] mini0.919-0.015 [â0.036, 0.003] random0.827â official0.936â Table A.1: Per-engine known-bug (REJ, excl-TLE) results on the AtCoder panel. The two numeric columns are different estimands and do not subtract: coverage is micro-pooled over submissions, while the noninferiority delta against the re-judged official suites is computed on a per-problem macro-recall basis. Noninferior at the single-engine interval when the lower bound exceedsâÎ=â0.02. The full per-problem verified accepted-but- buggy distribution over the 74 problems ships as a CSV with the released artifacts. findings rather than regenerated, and 24 floating-point false positives were removed. The census protocol above ran on each armâs certified rows, not only on the codex ledger, and 905 of the floor are confirmed: for each of them, at least one arm that supplies a legal killing input returns a confirming verdict. A separate stratified sample of 203 floor findings comprises a coverage layer of 202 rows drawn at random within each problem, capped per problem and inverse-probability weighted so no single problem dominates the estimate. Its risk layer holds every floor member outside that confirmed core; its sole row is the one the census left unconfirmed, because the censusâs independently written solver produced no usable output for it. An author adjudicated all sampled rows directly: the unweighted coverage-layer overturn count was 0, and the risk-layer row was upheld as a legal input with a wrong output. The signed report ships with the released artifacts, and the coverage layer bounds the false-discovery rate at 1.87% (Wilson 95% upper). Of this floor, 317 are certified only by the four non-codex arms, while 130 are certified by codex alone. As a raw pre-certification panel view, the 4013-row denominator comprises re-judged accepted submissions with a wrong-answer or runtime-error kill, excluding TLE, from at least one of the five engine suites, the official suites under re-judge, or random generation. The five engine suites jointly kill a fraction 0.964 of this uncertified panel, while 41 rows are killed only by the official suites under re-judge; this panel fraction is not comparable to the REJ coverage rates. The opencode/mini pair shares a base model but differs in harness, so their certified-count gap isolates the generation-side effect of the agent harness from the underlying model. Trajectory strategy labels. The English labels translate the annotation codebook. Small-case enumeration systematically enumerates all small cases, includingí=1,í=2, single-element, and other trivial inputs. Structural / extreme families exercise named extremes such as maximum-í, boundary, overflow, all-zero or all-negative values, repeated values, degenerate star, chain, tree, or grid shapes, and worst-case stress. Semantic-adversarial construction targets a problem-specific logical trap in the statement semantics or an algorithmic assumption, rather than only a structural extreme. Generator sweep uses random or seeded batch generation. Self-built wrong panel uses agent-written incorrect solutions as probes. Paired differential mining runs a wrong solution against the reference to mine an input on which their outputs differ. An LLM assigns these labels from trajectory digests; a same-model blind pass-2 re-label measures testâretest reliability, as reported in Table C.8. Ledger schema. Each of the 589 verified entries records the submission fingerprint and its official verdict, the certifying killing input with its per-problem legality certificate, the reference pool that adjudicated the kill, and the generation channel it is attributed to. The ledger covers 74 of 106 problems and splits into 545 wrong-answer and 44 runtime-error findings; the full per-problem distribution is released as a machine-checkable CSV. B Reproduction Details (CF-fresh and TCE) This appendix documents the comparisons behind Experiment 2 (Section 5) and the mechanism analysis (Section 6): the CF-fresh kill matrix and its baselines, the within-model single-call control (on its rejudged matrix), and the TestCase-Eval (TCE) saturation sweep. CF-fresh kill matrix. The benchmark is 63 sample-passing LLM-written solutions (excl-TLE) over 41 post-cutoff Codeforces problems (finished, non-gym contest rounds harvested under a March 1âJune 9, 2026 window, with 15 Coding Agents as Test-Suite AuditorsPreprint armscriptLLM calls per problemtarget generation-time filter officialstatement samplesnoneâ â cheap-random cf_gen.py1, up to 2 retries50 generator self-check, then consensus CodeContests cf_codecontests.py none60 consensus CodeContests+ cf_ccplus.py1, up to 2 retries (feedback) 60 own validator, then consensus EvalPlus-style cf_evalplus.py1, for seeds60 consensus agent (ours)agent harnessagentic loop50 reference runs successfully Table B.2: Configuration of the CF-fresh input-generation arms used in the kill-matrix comparison. Scripts are released undercode/scripts/cf_fresh/; every arm that invokes an LLM uses the same base model, claude-4.8-opus, as the agent arm, holding model capability fixed while varying the input-generation procedure. The implementations do not exactly match the original baseline descriptions; the deviations are detailed below. included problems spanning 2026-03-08 to 2026-06-07). The CF-fresh premise is anchored to the suite-building engine: its vendor publishes a declared Jan 2026 training-data cutoff, which precedes the earliest included problem. A per-engine cutoff table is not available, as Google, DeepSeek, and Alibaba publish no declared cutoffs for the other models used. The solutions are adjudicated buggy by the three-reference consensus oracle of Section 3 (its soundness audit is Section 6). Post-cutoff selection removes the direct model-side memorization channel. Harvesting uses two content-blind passes, a rating-stratified sample and a newest-first full take; interactive or unrated problems never enter either pass. Of the 48 harvested problems, 7 multi-solution or special-judge cases are excluded post-harvest by rule, leaving 41; the released data lists their identifiers and reasons. Of the 1,964 attempts in the multi-model solver population (1,640 single-shot LLM solutions and 324 agent-written ones across the source-model ladder), 596 compile and pass all public samples; the buggy pool keeps the 63 that fail the consensus oracle. Six arms are scored on one kill matrix, paired per problem, inputs-legal-only under a singletestlib 5 validator gate from the CodeContests+ reproduction toolchain [26], applied to all 41 problems. The arms are the agent arm (a single claude engine, no-feedback configuration), the official sample tests, cheap-random generation matched to the agentâs input budget, the mutation-based generator protocol provided with the CodeContests dataset [14], CodeContests+, and an EvalPlus-style generator [15]. Coverage is cov@í, the expected fraction of the buggy pool killed by a uniform randomí-subset of an armâs legal inputs (order-free). An arm with fewer thanílegal inputs contributes its whole set. Table B.3 reports cov@íat eight budgets and gives per-arm input counts. The comparison was evaluated at every integer budgetí=1, . . . ,60 and showed no crossover; its minimum gap was 0.018 at í=1. Baseline configurations. We summarize the concrete implementations, generation procedures, and filtering rules of all CF-fresh arms in Table B.2 before detailing their deviations from the original baseline descriptions. CodeContests originally mutated both public and private tests and used consensus over 30 accepted solutions. For CF-fresh, only public samples are available for mutation because these tasks provide no private tests; we instead use consensus over three human accepted solutions. Because the original mutator source was not released, we reconstructed its operator-selection rates and mutation steps from the paper description. EvalPlus originally used Python runtime type dispatch for type-aware mutation; stdin exposes no type information, so we classify tokens as integers or alphabetic strings, apply type-aware mutation and boundary injection, and do not extract a complete input grammar. For CodeContests+, the original model was not disclosed, so we use the unified base model; we also do not reproduce the checker layer based on eighttestlibcheckers for multi-solution tasks. The cheap-random arm does not reproduce a published baseline; it serves as a floor using a seed-reading generator program. Equal legal-input budget. We top up the CodeContests and EvalPlus-style arms, and the cheap-random arm when its generation program supports reuse, before scoring the CF-fresh kill matrix. For each applicable arm, we continue its own generation procedure and apply validator-first filtering plus three-reference consensus checks until the legal-input budget reaches 60 inputs. The agent arm does not participate in this top-up and remains at its original generation budget. On the 27 of 41 problems with available native-output records, the pre-top-up per-problem legal-input medians are 14 for CodeContests, 39 for EvalPlus-style, and 50 for cheap-random. These medians report the native outputs before completion to the common legal-input target. On some problems, none of the applicable generators yields a legal input, so the corresponding armâproblem entries remain zero. 5 testlib.h, M. Mirzayanov, https://github.com/MikeMirzayanov/testlib. 16 Coding Agents as Test-Suite AuditorsPreprint armcov@1 cov@2 cov@5 cov@10 cov@20 cov@40 cov@50 cov@60 inputs agent (ours)0.223 0.356 0.570 0.729 0.853 0.932 0.952 0.95243.8 CodeContests+0.205 0.336 0.531 0.6440.7250.7900.8090.82553.2 EvalPlus-style0.123 0.209 0.377 0.5200.6250.6910.7110.73054.1 CodeContests0.075 0.134 0.246 0.3320.4030.4810.5050.52454.1 cheap-random0.175 0.211 0.265 0.3160.3770.4370.4530.46049.0 official (samples) 0.000 0.000 0.000 0.0000.0000.0000.0000.0001.0 Table B.3: CF-fresh kill matrix, cov@íat eight budgets over the 63 sample-passing adjudicated-buggy solutions (legal-only, excl-TLE, paired per problem), with mean legal inputs per problem (an equal-weight mean over all 41 problems, counting an armâproblem pair as zero when the consensus gate filters all of its inputs). The agent has the highest coverage in every cov@í column without having the most inputs. Best per column in bold. configurationcov@20cov@50 mini (agent)0.8010.903 mean single-call0.5970.665 four-call 50-slot mixture0.6880.790 CodeContests+ (four-rep mean)0.3620.414 Î vs single (95% CI)0.204 [0.059, 0.343]0.238 [0.074, 0.417] Î vs mixture (95% CI)0.113 [-0.070, 0.298]0.113 [-0.102, 0.339] Î vs CodeContests+ (95% CI) +0.439 [+0.310, +0.552] +0.489 [+0.359, +0.623] Table B.4: Deployable within-model controls on the rejudged basis of 19 problems and 62 buggy solutions. The agent row is mini, rather than the claude arm in Table B.3; that table uses a different buggy-solution pool, so cross-table subtraction does not define a matched contrast. The mean single-call row averages four independent reps. The mixture interleaves those reps by position within one 50-input suite, with no held-out selection. The CodeContests+ row is a same-base-model replication, averaged over four sampling replicates and construction-matched to the mean single-call row; it is not interchangeable with the single-sample, opus-based CodeContests+ arm of Table B.3 (different base model, sampling protocol, and pool). Its per-replicate cov@50 values are 0.158, 0.562, 0.677, and 0.259; a single sample of this protocol is not a stable estimate. At both budgets, the miniâsingle-call and miniâCodeContests+ 95% confidence intervals exclude zero, whereas the miniâmixture interval crosses zero. Values in different rows are rounded independently. Table B.3 averages inputs equally over all problems, including those entries, and therefore reports per-problem means below the completion target. Table B.4 reports the deployable within-model controls on their rejudged basis, including the mean single-call arm, the position-interleaved mixture, and the same-base-model CodeContests+ replication. On this base model, 4 of 19 problems produce a generator whose output its own generated validator rejects, against 0 on the opus-based replication; a replicate that draws such a generator keeps no inputs for that problem. Table B.5 then reports a retrospective held-out-pool upper bound as a diagnostic rather than as an entry in the baseline ranking. Within the same controls, the single-call shortfall is not an inability to construct large cases: maximum-scale inputs comprise 0.41â0.55 of the single-call armâs legal inputs across reps, versus 0.18 for mini, the smallest share of any arm here. Compilation failure, violation of a stated input constraint, and structurally malformed output are alike absorbed by the consensus oracle. Across the four replicates, validator-legal rates span 0.917â0.981, versus 0.995 for mini; 3 problems contain one zero-legal single-call replicate, while other replicates on those problems supply legal inputs. CF-fresh oracle audit. The three-reference consensus oracle behind the CF-fresh buggy pool (Section 6) was challenged by an independent LLM panel. A gold is the expected output when at least two accepted human reference solutions run successfully and all successful outputs are identical; an input whose references produce no such agreement has no gold, is skipped rather than passed, and is outside the audit by construction. The audited population comprises 1,795 agent-arm inputs carrying a usable gold and excludes baseline-arm inputs; a flag is raised when the panel of sample-passing LLM solutions assigns more votes to some other output than to the gold. All 15 flags adjudicate to correlated-LLM-error false alarms. Brute-force ground truth confirms all 13 flagged golds on one problem (cf2220F). The remaining 2 flags (cf2231F, whose inputs are too large to brute-force) adjudicate to the same pattern on structural grounds: the gold is three system-test-passing human 17 Coding Agents as Test-Suite AuditorsPreprint non-deployable boundcov@1cov@50 mini (reference)0.1680.903 oracle best-of-2â0.785 oracle best-of-3â0.831 oracle best-of-40.3460.855 Î vs best-of-4 (95% CI) -0.178 [-0.339, -0.031] 0.048 [-0.130, 0.212] Table B.5: Non-deployable retrospective oracle upper bounds, excluded from baseline ranking. For each problem, the oracle selects the best of the four single-call reps using the held-out buggy pool; that post hoc selection is unavailable at deployment. At cov@1, best-of-4 ranks above mini: the paired mini minus best-of-4 difference is negative, and its 95% confidence interval excludes zero. Retrospective selection has its greatest leverage at this suite size because single-draw variance is greatest there; this variance account does not alter the ordering reversal. At cov@50 the paired interval crosses zero, so the bound and mini are not separated. The cov@1 column is included as a diagnostic and is not the paperâs operating point. 1251020 first k emitted inputs (log scale) 0.3 0.4 0.5 0.6 0.7 0.8 mean prefix cov@ k agent (ours) cheap-random Figure B.1: Saturation on the TCE 24-problem sweep. Markers show absolute mean prefix cov@íâthe fraction of each problemâs known-bug submissions killed after the firstíinputs in emission orderâwith problems weighted equally, rather than using a randomí-subset. Cheap-random generation matched to the agentâs input budget front-loads coverage, while the agent set continues adding coverage through the tail. The sweepâs known-bug pool is independent of every compared arm, unlike the CF-fresh pool, which is the union of the armsâ own certified findings. Twenty inputs form the sweepâs per-arm design budget, so cov@20 is each armâs full-budget ceiling. solutions against a sample-passing LLM plurality on a deterministic-answer problem. No demonstrated wrong gold remains. TCE saturation sweep. The mechanism analysis runs on a 24-problem stratified Codeforces sweep spanning five difficulty bands (1500â3500 rating), built on the TestCase-Eval known-bug setting [29]. We measure saturation shape against cheap-random generation matched to the agentâs input budget: the normalized coverage at input 1 and the per-input marginal coverage in the saturated tail (Figure B.1 shows the coverage shape; Table B.6 reports both statistics). The agent set reaches 0.838 cov@20, and its mean prefix curve rises across both measured tail intervals; cheap-random front-loads and then flattens. A no-feedback configuration, which withholds our model-generated wrong-candidate kill signal, retains almost the same tail marginal (0.025 vs 0.026). The sustained tail marginal is thus a property of the agent-constructed set, not an artifact of our feedback loop. The ablation rules out our loop as the source, and claims nothing broader about feedback. 18 Coding Agents as Test-Suite AuditorsPreprint configurationnorm. cov@1 tail marginal (SE) cheap-random0.650.004 (0.0013) agent0.320.026 (0.0046) agent, no feedbackâ0.025 (0.0043) Table B.6: TCE saturation detail over the 24-problem sweep. Normalized cov@1 is cov@1/cov@20 (higher = more front-loaded), with the denominator taken at the sweepâs per-arm design budget; the tail marginal is mean new coverage per added input over the window from the sixth input through the smaller of the armâs actual input count and that budget, so arms with fewer inputs than that budget are not diluted (standard error in parentheses). Both columns average per-problem values, whereas Figure B.1 plots mean coverage curves, so neither column can be recovered as a ratio of points on those curves. Cheap-random front-loads and flattens; the agent set spreads coverage across inputs, and withholding feedback barely moves the marginal. Released tables. The per-problem kill matrices for both sweeps (cov@íper problem and per arm, and the TCE per-problem marginals) are released as machine-checkable tables alongside the buggy pools and the validator gates. Every cell above is re-derivable. Artifact availability. The supplementary code and data package carries the ledgers, kill matrices, validators, judging harness, and analysis code behind the numbers reported here, together with the signed false-discovery adjudication of §A and the per-finding exhibit bundle. It ships one entry point that recomputes every number this paper cites from the packaged artifacts alone and diffs each against the frozen values the paper is typeset from; the remaining registered measures are retired ones no section cites. A manifest records a SHA-256 digest for every file. What the package re-derives is every reported number from the stored artifacts. Regenerating a kill matrix one level further back, from the raw submissions, needs those submissions, and platform terms keep them out of the release; the regeneration script and the pinned invariant it self-checks against ship anyway, so the procedure is inspectable even where it is not runnable from the package alone. That path is Linux-only by design: AtCoder judging requires an unlimited process stack, and the macOS fallback silently caps it, turning deep-recursion accepted solutions into false runtime errors. The script refuses to run off Linux rather than emit a plausible wrong count. Computing environment. All authoritative judging runs on one fixed machine: Ubuntu 22.04.3 LTS, Linux 6.2.0-37, x86-64, an 8-thread Intel Xeon E5-2682 v4 at 2.50 GHz with 15 GiB of memory, g++ 11.4.0 (C++17) and Python 3.10.12. The engines are the released command-line agents and base models cited in Section 3; each runs against its own providerâs API, so wall-clock and token costs (§A) are not machine-bound. Hardware affects one verdict class only. Timeouts depend on processor speed, which is why the reported detection figures use the excl-TLE basis defined in Section 4: wrong-answer and runtime-error verdicts are deterministic under a fixed stack limit and reproduce across runs, whereas time-limit verdicts would not. C Qualitative Exhibits This appendix makes the certified findings concrete: the distribution of what goes wrong, three transcribed cases, and how the audit corrects itself. Every exhibit is transcribed from a real released artifact, never synthesized. What a certified finding is. The 589 verified findings split into 545 wrong-answer and 44 runtime-error submissions over 74 problems. A wrong-answer finding is a submission AtCoder accepted whose output disagrees with the consensus oracle on a legal input: an input inside the constraints the statement promises, certified by the problemâs validator. A runtime-error finding crashes on such an input. The findings are not concentrated: per-problem single-linkage clustering over comment-stripped source token 5-shingles at Jaccardâ„0.8, with identifiers left unchanged, yields 578 near-clone-distinct clusters and 224 (bug classĂ problem) families. Bug-class profile. An LLM labeler assigns each finding a bug class; Table C.7 gives the distribution over the 46 distinct classes. Because every one of the 589 entries is a submission the official suite accepted and an agent suite killed, this profile is by construction the profile of what only the agent audit caught. The labels are descriptive and single-labeler. An independent blind cross-family re-label agrees on 43.3% of a stratified 60-entry sample at this tableâs grain (Cohenâsí =0.36), with disagreement concentrated on algorithm-level class boundaries. The taxonomy is indicative, not adjudicated, and per-class counts carry no claims. Verified-bug existence (the 589 entries) is untouched: kill verdicts are mechanical, only the class names are soft. 19 Coding Agents as Test-Suite AuditorsPreprint bug classsubmissions special-case missing97 wrong greedy73 off-by-one64 integer overflow62 array out-of-bounds55 boundary55 tie-breaking33 wrong algorithm27 precision23 other (+37 classes)100 total (46 classes)589 Table C.7: Bug-class distribution of the 589 verified accepted-but-buggy findings (LLM-labeled; nine most common classes, with other aggregating the remaining 37 classes). strategy / featurepresent %agreeí construction strategies semantic-adversarial construction106 100.0 1.00 â structural / extreme families10497.2 -0.01 â generator sweep5284.9 0.70 small-case enumeration1850.0 0.15 â„2 of the 4 co-occur105â recurring features self-built wrong panel106 100.0 1.00 â paired differential mining10397.2 0.00 â Table C.8: Multi-strategy construction in the audited (codex-arm) trajectories: at least two strategies co-occur in 105 of 106 trajectories, with one audited trajectory per problem. Present is the pass-1 count over 106 problems; strategies are not mutually exclusive. â marks near-constant rows, whereí degenerates; read %agree there. Pass-2 does not measure cross-family agreement. Strategy labels are defined in §A. Construction-strategy taxonomy. Complementing the bug-class profile of what the audit caught, the construction-strategy taxonomy records the strategies with which the codex arm builds its inputs. The taxonomy is descriptive and single-labeler; we draw no per-strategy inferences. Three transcribed cases. Each case below is one verified finding, transcribed from the released exhibit bundle: a submission the official suite accepted, the legal killing input that certifies it, and the oracleâs expected output against what the submission produced. âąabc167_e: runtime error. The submission sizes its factorial tables ton(vector<llint> mul(n,0)) but writesmul[i+1]foriup tok. On the legal input3 2 2, wherekequalsn-1, the writemul[n] runs off the end and the program crashes; the oracleâs expected answer is 8. âąabc161_f: wrong answer. Enumerating divisors ofnandn-1, the submission admits a value the statement excludes: on the legal input 3 it outputs 3 where the oracle requires 2. âą abc141_e: wrong answer. The submission special-cases the first table row (i=0) in its own loop, setting c[0][j]=1for a matched pair but omitting themx=max(mx,c[i][j])update that the main loop applies only from the second row onward. A length-one repeat found solely in that first row is therefore never counted: on the legal input 2 / a it outputs 0 where the oracle requires 1. The released exhibit bundle records, for every finding, the LLM labelerâs bug class, the killing input itself, and the observed behavior (the oracleâs expected output against what the submission produced). Problem statements and submission sources stay with the originating platforms; each record carries the public problem and submission identifiers that locate them. 20 Coding Agents as Test-Suite AuditorsPreprint The audit correcting itself. The clearest illustration of the chainâs conservatism is a case it retracted. On abc164_e, deterministic re-judging first flagged 123 submissions. Re-judging each against a 200-submission pool showed the 8-reference oracle was itself the outlier on that problem, and all 123 were removed from the ledger. No finding in the released ledger rests on an expectation the larger pool overturns. The visible count is what survives this correction, not what the first pass produced. D Agent and Single-Call Prompts (Agent-Form Fairness) The agent-form control in the main text (Experiment 2) compares two CF-fresh arms: miniânot the claude arm in the main results tableâand a bare single call of the same base model, DeepSeek V4 Pro, to separate the agentic workflow from the base model. The single-call arm was run in four independent reps, whose mean is reported in the main text. This appendix presents the complete single-call system message and user template, reproduces the agent-arm task file with one[...]elision, and explicitly excerpts the auditing skill to which that file points; the full skill file is in the released archive, and Table D.9 documents the two armsâ equivalence so the comparisonâs fairness is checkable. The two arms receive the same task objective, the same initial information (the problem statement and one trusted reference solution), the same 50-input budget, and the same input-type checklist. The single-call system prompt was written to mirror the objective and checklist of the agent skill. Both arms may submit literal inputs or a C++ generator program. For the single-call arm, the harness compiles the generator and runs it over seeds after the call, treating each runâs standard output as an input, but returns no execution result to the model. The remaining agent-form differences are construction-time execution feedback and iteration: mini can check a candidate against the trusted reference, observe the result, and revise before submission, whereas the single-call arm cannot. The mini arm is nevertheless the no-feedback ablation, with its weak-coder wrong-solution generator disabled. It receives no kill signal on which to iterate, so the comparison remains conservative toward the agent. dimensionequivalent across arms? base modelyes (DeepSeek V4 Pro) objective + kill definitionyes initial information (statement+reference) yes legality requirementyes 50-input budgetyes input-type checklistyes (near-verbatim) generator-program channelyes no held-out test poolyes kill signal during constructionyes (none) construction-time reference checkno â agent form returned execution resultsno â agent form iteration (revise vs. one-shot)no â agent form Table D.9: Per-run equivalence of the two arms. Both may emit generator programs and receive no kill signal; agent form is isolated by access to construction-time checks and their returned execution results, which permit iterative revision. 21 Coding Agents as Test-Suite AuditorsPreprint D.1 Agent-arm prompt Per-problem task identity (AGENTS.md, byte-identical toCLAUDE.md; only the title,docs/statement.txt, and docs/ref.cpp change across problems): Agent-arm task (AGENTS.md) # ADG test-data task -- <pid> (<problem title>) **Objective.** Build a general discriminating test set -- a diverse set of legal inputs that catches ANY subtly-wrong solution (the trusted reference runs OK but the solution diverges -- different output, TLE, or crash). The goal is the COVERAGE of a diverse, budget-filling SET -- breadth across distinct failure modes, not a few clever inputs. OUTPUT CONTRACT (strict -- this is what gets scored): put your FINAL, curated test set -- exactly the inputs to be scored, deduplicated, in the order you want them evaluated -- as numbered files 01.txt ... 50.txt in workspace/final/. Only workspace/final/ is scored. workspace/hacks/ is scratch. - How (full protocol + tool contract): follow .claude/skills/adg-generator/SKILL.md. - Problem statement: docs/statement.txt | Oracle: docs/ref.cpp | (no validator shipped -- an input is legal if docs/ref.cpp runs OK on it; honor the statementâs constraints yourself) - Limits: time 3 s | memory 256 MB - Input budget -- produce 50 distinct inputs. This is a TARGET to FILL, not a ceiling. - Ablation -- the weak-coder wrong-generator (gen_wrongs.py) is DISABLED this run: you have no self-drafted wrongs to probe, so there is no kill signal to iterate on. Build from direct reasoning about the reference and generator sweeps. This is an open-ended task -- you choose the strategy and tool order; nothing is mandatory (see the skill). [...] Do not look for the held-out evaluation pool: it isnât here, and your only correctness oracle is docs/ref.cpp. The auditing skill this points to (.claude/skills/adg-generator/SKILL.md, objective and toolbox; ex- cerpted, the full file ships in the released archive): Auditing skill adg-generator/SKILL.md (excerpt) Your objective: produce a high-quality general discriminating test set for this problem -- a diverse set of legal inputs that FILLS the input budget, written as numbered files to workspace/final/, that would catch ANY subtly-wrong solution [...] Optimize the quality of the test set, not any fixed procedure. You decide the strategy. This is a toolbox, NOT a pipeline. [...] A capable test-setter typically: - Constructs inputs directly [...] edge cases, boundary/extreme values, overflow triggers, degenerate/adversarial structures (chains, stars, cliques, all-equal, coprime), worst-case inputs. - Writes a generator program (random or structured) and sweeps seeds -- for breadth and for scale. - Optionally drafts concrete wrong solutions to discover failure modes, then finds inputs exposing them. - Checks candidates, then merges / deduplicates / minimizes into a tight non-redundant set. - Iterates -- and may revisit any of the above multiple times. 22 Coding Agents as Test-Suite AuditorsPreprint Toolbox (every tool is OPTIONAL): - docs/ref.cpp -- the trusted reference (oracle). - scripts/eval_hack.py <input> -- check ONE candidate: is it LEGAL, does the reference run OK, and which wrongs in workspace/wrongs/ it kills. - scripts/write_generator.py workspace/gen.cpp --seeds N -- compile and run a C++ generator across N seeds. - scripts/gen_wrongs.py --n K -- optionally draft weak-coder candidate solutions. (DISABLED this run.) D.2 Single-call-arm prompt The bare single call (cf_bestofn.py) is one frozenchat.completionsrequest with no interactive tools. The harness compiles and sweeps any submitted generator after the call and returns nothing to the model. Unicode punctuation and symbols are transliterated to ASCII below. System message: Single-call system message You are generating adversarial TEST DATA for a competitive- programming problem. Your objective: produce a high-quality, DIVERSE set of LEGAL inputs (satisfying ALL input constraints stated in the problem) that would catch ANY subtly-wrong solution -- an input "kills" a wrong solution when the trusted reference runs OK on it but the wrong solution diverges (different output) or times out / crashes. You are a frozen frontier model with NO hidden test pool and NO interactive tools -- reason ONLY from the statement and the reference solution given, and answer in one shot. A good set mixes: small/edge cases (n=1, minimal), boundary/extreme values, overflow triggers, degenerate/adversarial structures (chains, stars, all-equal, coprime, ...), worst-case-complexity (maximum-size) inputs, and problem-specific tricky special cases. User message: the problem statement and reference solution, followed by the two-channel protocol below (cap=50; max_seeds=60; cxx and std are the compiler and standard recorded in §B): Single-call user message (template) === PROBLEM STATEMENT === statement === REFERENCE SOLUTION (trusted oracle, human-accepted, C++) === reference === CHANNELS (two ways to produce inputs; use either or both) === - **Direct construction** -- emit the raw stdin bytes of an input inside a â===INPUT===â block. - **Generator program** -- emit a C++17 program inside a â===GENERATOR===â block. We compile it with âcxx -O2 -std=stdâ and run it once per seed â0..N-1â, where N is the â===SEEDS===â value; each runâs stdout becomes one input. **Contract: the program reads ONE integer seed from stdin and prints exactly ONE legal input to stdout, and must produce a different input for a different seed.** This is the channel for SCALE (maximum-size / worst-case inputs) that the literal channel cannot reach -- typing a 10^5-element input as text is not feasible, writing a generator is. (!) You will NOT be told whether the program compiled, what it printed, whether any input is legal, or how many inputs exist so far. Nothing is checked back to you. Write code that is correct by construction and inputs that are legal by construction. 23 Coding Agents as Test-Suite AuditorsPreprint === OUTPUT PROTOCOL (strict) === Reply with ONLY the blocks below -- no prose, no explanation, no markdown fences around them. ===INPUT=== <raw stdin bytes of ONE test input> ===GENERATOR=== <complete C++17 program: reads one integer seed on stdin, prints ONE legal input on stdout> ===SEEDS=== <how many seeds to sweep, an integer, at most max_seeds> Blocks may appear in any order and any number of times. A â===GENERATOR===â block MUST be immediately followed by its â===SEEDS===â block. Emit nothing outside these blocks. (!!) **THIS IS YOUR ONLY REPLY.** You get exactly ONE response -- there is no second turn, no follow-up, no chance to add more later. Everything you emit now IS your entire final test set. Produce all cap distinct inputs in this single reply. Do not pace yourself, do not defer part of the budget to a later turn, do not end with an intention to continue -- there is no later turn. 24