Paper deep dive
The Quality of Claude AI-authored Python Tests Is Not Weaker Than Human-authored Tests
Douglas J. Leith
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/18/2026, 5:53:06 AM
Summary
This study evaluates the quality of Python tests authored by Claude AI models (Sonnet/Opus 4.6 and later) against human-authored tests from the Django and Pandas open-source projects. Using four independent evaluation methods—historical-revert testing, commit-based mutation testing, coverage-based mutation testing, and a seven-axis qualitative design rubric—the authors find that AI-written tests are not weaker than human-written ones. The AI corpus, derived from the real-world tool 'Extractor' built by Claude, showed comparable fault-detection capabilities and test design quality to the established human-written corpora, challenging prior pessimistic views on LLM-generated test quality.
Entities (9)
Relation Signals (6)
Extractor → comparedagainst → Django
confidence 98% · We evaluate the quality of Claude AI-written Python tests against human-written Python tests from two established open-source projects Django and Pandas.
Extractor → comparedagainst → pandas
confidence 98% · We evaluate the quality of Claude AI-written Python tests against human-written Python tests from two established open-source projects Django and Pandas.
Claude AI → authored → Extractor
confidence 95% · the real test suite of Extractor, a tool built from scratch by Claude Sonnet 4.6
Extractor Tests → hasquality → Comparable to Human Tests
confidence 92% · tests written by recent Claude models... are no weaker than the two human-written corpora.
Mutation Testing → usedtoevaluate → Test Quality
confidence 90% · three independent fault-injection protocols... mutation testing
Historical-Revert Testing → usedtoevaluate → Regression Detection
confidence 90% · historical-revert testing, which reverts each test’s own real, once-shipped bugfix commit and checks whether the test still fails
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We evaluate the quality of Claude AI-written Python tests against human-written Python tests from two established open-source projects Django and Pandas. Hundreds of tests per corpus are scored under one identical protocol. Using one-sided non-inferiority bounds, we find that the tests written by recent Claude models (Sonnet/Opus 4.6 and later) are no weaker than the two human-written corpora. In this study: (i) the AI-written corpus is tests from real tools, not synthetic tests generated in isolation against a fixed target, the setup used by every other AI-test-generation study we are aware of; (ii) every test is individually scored under three independent fault-injection protocols plus a seven-axis qualitative design rubric, allowing methods to cross-validate each other; (iii) tests are scored individually, rather than suite-level, identifying exactly which specific tests need attention.
Tags
Links
- Source: https://arxiv.org/abs/2608.15188v1
- Canonical: https://arxiv.org/abs/2608.15188v1
Trouble viewing inline? Open PDF directly →
Full Text
106,222 characters extracted from source content.
Expand or collapse full text
The Quality of Claude AI-authored Python Tests Is Not Weaker Than Human-authored TestsConference: ; ; PreprintCCS: Software and its engineering Software testing and debuggingCCS: Computing methodologies Machine learning D. J. Leith Affiliation: Trinity College Dublin , Dublin , Ireland email: doug.leith@tcd.ie Abstract. We evaluate the quality of Claude AI-written Python tests against human-written Python tests from two established open-source projects Django and Pandas. Hundreds of tests per corpus are scored under one identical protocol. Using one-sided non-inferiority bounds, we find that the tests written by recent Claude models (Sonnet/Opus 4.6 and later) are no weaker than the two human-written corpora. In this study: (i) the AI-written corpus is tests from real tools, not synthetic tests generated in isolation against a fixed target, the setup used by every other AI-test-generation study we are aware of; (i) every test is individually scored under three independent fault-injection protocols plus a seven-axis qualitative design rubric, allowing methods to cross-validate each other; (i) tests are scored individually, rather than suite-level, identifying exactly which specific tests need attention. Keywords: software testing, test quality, test suite evaluation, mutation testing, AI-generated code, large language models 1. Introduction We evaluate the performance of AI-written and human-written Python tests. We find that AI-written tests are no weaker than human-written ones: they are comparable across four independent evaluation methods applied to the same test population. This is contrary to the more pessimistic picture from prior AI-test-generation studies, which report low compile/pass rates and weak oracle accuracy for LLM-generated tests. Note, however, that coding-focused models underwent a well-documented improvement in ability in late 202511 1 In November 2025 Claude Opus 4.5 was the first model to break 80% on SWE-bench Verified.. The models evaluated here post-date that change, whereas previous work mostly evaluated substantially earlier models. Testing is what makes AI-written code safe to build on. Large language models write code quickly, but make mistakes often enough that their output cannot be trusted at face value (Chen et al. 2021; Liu et al. 2023). The safeguard is not flawless code but a test suite that locks in intended behaviour, so that a regression introduced by a later change, AI-made or human-made, is caught rather than silently shipped. That safeguard is only as good as the tests themselves. Whether AI-written tests are, in practice, as effective as human-written ones at this job therefore has direct bearing on how safely AI-assisted development can scale. It is not only an academic interest in code style. This report scores AI- and human-authored tests under four independent, complementary methods: (i) historical-revert testing, which reverts each test’s own real, once-shipped bugfix commit and checks whether the test still fails; (i) per-commit AST mutation testing, which mutates the lines that same introducing commit itself changed and checks whether the test catches the resulting synthetic fault; (i) coverage-guided mutation testing, which places a synthetic fault anywhere in the production code the test actually executes, without needing a paired commit; and (iv) a seven-axis qualitative rubric that scores test design directly, independent of whether a particular fault happens to be caught. This was applied at scale, across five Python corpora: several thousand synthetic mutants and hundreds of individually scored tests per corpus. The four methods cross-check each other’s blind spots. To the best of our knowledge, no prior study has evaluated both AI- and human-written tests under a matched, individual-test protocol allowing a direct head-to-head per-test comparison as we do here. This individual-test focus matters beyond precision: it is directly actionable. A suite-level score says a suite is, say, 90% effective, but not which tests make up the weak 10%. Scoring each test individually identifies exactly which ones need attention, so effort can be targeted at fixing those tests rather than diffused across a suite that is mostly already sound. Our main AI-written corpus is also unlike the setup used in every AI-test-generation study we are aware of. Rather than an LLM generating a test in isolation against a fixed, already-written function, it is the real test suite of Extractor, a tool built from scratch by Claude Sonnet 4.6 over several weeks of ordinary, unscripted development. The AI decided for itself, turn by turn, whether a given moment called for a test. No special test-writing instructions were given. We also extend the analysis to two large-scale open-source packages with Claude Sonnet/Opus 4.6 AI-authored tests, although for these we do not know the test generation workflow used as clearly as we do for Extractor. 2. Related work Whole-test-suite mutation adequacy. Much of the literature focuses on whether a test suite’s mutation score predicts its ability to catch real faults. Probably the earliest work is (Just et al. 2014), see also (Petrović et al. 2021), (Papadakis et al. 2019). However, (Papadakis et al. 2018) finds the correlation weakens sharply once suite size is controlled for; (Andrews et al. 2005) show hand-seeded faults behave differently from real faults; and (Jia and Harman 2009), (Shamshiri et al. 2015) show combining mutations does not reliably help, and that generated suites found only 55.7% of real faults, just 19.9% of individual suites finding any fault at all i.e. high scores on generation-time adequacy translate only weakly into real fault-finding. Evaluating the quality of individual tests. A smaller strand asks whether an individual test does its job, independent of whole-suite adequacy. In (Zhang and Mesbah 2015) it is shown that assertion quantity and quality, not coverage, predicts effectiveness; (Vahabzadeh et al. 2015) finds two-thirds of silently-passing test-code bugs are assertion-related. The closest precedent for scoring a test against a mutant of its inferred target is (Vercammen et al. 2018): on Apache Ant, the whole suite killed 44 of 55 focal mutants, but the responsible test alone killed only 35. Qualitative scoring (“test smells”). A more mature strand evaluates test quality by category and severity rather than fault-catching. Its vocabulary originates with (van Deursen et al. 2001)’s “test smells”. In (Bavota et al. 2015) it is found that 86% of JUnit tests carry at least one smell, with comprehension 30% worse in its presence; see also survey (Garousi and Küçük 2018). Detection is mostly automated via static rule-matching, but Panichella et al. (Panichella et al. 2022) show developers find such warnings unrepresentative, and respond better to hand-annotated verified judgement; (Campos et al. 2021) likewise finds smell severity is a matter of judgement. Evaluating AI-written tests. A fourth, recent strand asks whether LLM-generated tests are as effective as human-written ones, mostly via compile/pass rates rather than mutation testing. In (Schäfer et al. 2024) only 48.0% of LLM-generated JavaScript tests are found to pass at all; (Yuan et al. 2024) finds only 24.8% of GPT-3.5-generated tests compile and pass before refinement. In (Konstantinou et al. 2024) it is found that LLM-written oracles frequently capture actual rather than expected behaviour, under 50% oracle accuracy; (Haroon et al. 2026) finds 99% of tests that fail after a genuine behavioural change still pass on the unmodified program (but its Project CodeNet corpus has no AI training-data contamination check). A 2026 head-to-head on real Python bugs (Vathana et al. 2026) finds an LLM given the bug report plus fix detects 69% of faults against 17.2% for pre-existing human suites. Patch-review studies such as METR’s (Whitfill et al. 2026) grade whether AI-generated patches survive maintainer review, not whether their tests are effective. 3. Preliminaries 3.1. Dataset Data, plus scripts to reproduce the tables and figures here, is available at https://github.com/doug-leith/claude-python-test-quality 3.2. Code Corpora Used We primarily evaluate the test quality in three code corpora: Extractor (apple_protobuf_extractor), Django and Pandas; see Table 2. Table 1. Corpus scale: authorship, domain, and code/test size22 2 Note that these LOC numbers include comments/docstrings and white space. for the three corpora studied. Corpus Written by Domain Size (src/test LOC; tests) Extractor Claude (Sonnet 4.6) ARM64/ObjC reverse eng. 21k/20.7k; 1,423 tests Django Human; ~88k GitHub stars; active since 2012 Web framework 165k/354k; 17,519 tests Pandas Human; ~49k GitHub stars; active since 2010 Data analysis library 263k/407k; 20,244 tests Django and Pandas. Both are large, mature, widely used open-source Python projects with primarily human-written test suites and long, real commit histories33 3 See https://github.com/django/django, https://github.com/pandas-dev/pandas.. Extractor. This is Claude Sonnet AI-written code, wholly written by AI from scratch over several weeks of work. It is a real tool (for reverse engineering protobuf descriptors from compiled Apple Objective-C/C binaries), not synthetic test or “toy’ ’ code. Only 22% of episodes where tests are written/edited trace to a direct ask for tests; the remaining 78% were the agent’s own initiative e.g. a new feature or function was requested with no mention of tests anywhere in the instruction, and the agent wrote tests anyway while delivering the work. In the 22% of episodes with human prompting, the prompts are always general in nature e.g. “yes, go ahead with that plan. add tests to ensure good code coverage as you go along.”, “check the test coverage of the new code, add extra tests if needed.”. See Additional Material for more details. This contrasts with nearly all of the existing AI-written test quality literature (§2), where the AI is handed a fixed, already-written piece of production code (e.g. a function signature and body, a “focal method” plus its class, a method plus a pre-existing test prefix, a self-contained Project CodeNet snippet, or a target function plus a bug patch), and asked only to produce a test for it. The model never writes or evolves the production code itself, and never decides on its own initiative whether a test is warranted. Each test is generated as one isolated, bounded task, not as one moment inside an open-ended, multi-week development process in which the same agent also designs and writes the code under test. In §7 we extend the evaluation of AI-authored test quality to Quay and AWX, both large-scale open-source codebases. However, we keep that analysis separate from the Extractor analysis because: (i) we do not know the provenance of the Quay and AWX AI-authored tests as clearly as for Extractor (in particular, we do not know the AI instructions used to generate the tests nor whether, or how much, the tests underwent human revision before being committed), and (i) the Quay and AWX codebases contain a mix of human and AI authored code/tests whereas Extractor is wholly AI-authored. 3.3. Test Sampling We evaluate tests associated with a production-code change. Tests were drawn uniformly at random from the eligible population, subject to the following exclusion criteria: (i) the evaluation methodology cannot validly score this class of test44 4 For example, a refactor test whose purpose is to prove old and new implementations are equivalent. Such a test succeeds when it passes on both the pre- and post-change code, whereas our test methodologies focus on tests whose correct behaviour is to fail when a code regression occurs., (i) the test’s failure (or non-run) is an artefact of the test harness or environment, not a real signal55 5 The test harness is: (a) missing a dependency, (b) has a version mismatch or (c) a build/environment limitation that cannot be readily fixed. For example, a pre-2020 asgiref default that is a no-op against the modern dependency actually installed, a MariaDB-version-specific check, a compiler-version-only historical fix.. 4. Mutation-testing results Our goal is to evaluate the quality of AI-authored tests, using human-written tests (Django, Pandas) as a baseline for comparison. The same evaluation protocol is applied to all three corpora i.e. tests are scored the same way, regardless of which corpus a test comes from. The resulting test scores are therefore directly comparable. In this section test quality is evaluated under three mutation-based protocols (we consider qualitative evaluation in §5): (i) historical-revert: revert the production code change in each test’s introducing commit and check whether the test fails, evaluating “does the test catch the regression”; (i) commit-based mutation: for the same test/commit pairs, mutate the code the commit changes and score whether the test catches (“kills’ ’) each synthetic mutant; (i) coverage-based mutation: mutate every production line the test’s own execution is estimated to cover and score whether the test catches each mutant. Each protocol trades realism for reach in different ways. Historical-revert measures regression against a real, once-shipped defect but only for fixes paired with a test and only as a single pass/fail data point per test. Commit-based mutation keeps the same commit-scoped population but replaces the one real defect with many synthetic ones, trading some realism for a denser, per-test kill-rate signal. Coverage-based mutation testing drops the commit requirement and mutates code the test is estimated to cover, not just a diff. However, that broader reach also means it can mutate code unrelated to the test’s actual purpose, so a low kill rate can reflect either weak assertions or simply incidental coupling to code the test was never responsible for. Interpretability also differs sharply. A historical-revert outcome is easy to read: the test either caught a real, once-shipped bug or it did not. A kill-rate value is not so easy to read: there is no established mapping, or even a clear correlation, between a given kill-rate value and real fault tolerance (see §4.2.3 below) although, all else being equal, a higher kill rate suggests better fault tolerance than a lower one. 4.1. Historical-revert testing of individual tests 4.1.1. Methodology For each test t with introducing (post-fix) commit c touching production files F, the harness checks out c, confirms t currently passes (baseline), reverts every file in F to its pre-fix content from c’s parent, and re-runs t. If t now fails, the verdict is EFFECTIVE: the test genuinely caught the real historical regression. If it still passes, the verdict is SURVIVED. A third possible verdict, UNREACHED, applies when the test structurally can never reach the changed code under any input its own design could produce at all. The full pseudo-code is in the Additional Material. 4.1.2. Results Table 2 shows that the EFFECTIVE rate is consistently high for all three packages. Pandas scores slightly lower than Extractor and Django, and the confidence bounds for all three overlap substantially. The overlap of the confidence intervals means that the data cannot reject the hypothesis that there is ‘no difference’ between the corpora. We can also use a one-sided 95% non-inferiority bound on the difference (Extractor’s EFFECTIVE rate - Django/Pandas EFFECTIVE rate) to evaluate the hypothesis that Extractor’s EFFECTIVE rate is worse than Django/Pandas. Calculating this66 6 Bootstrap-resampled, 5,000 resamples with replacement at each corpus’s own sample size. rules out Extractor’s EFFECTIVE rate being worse than Django’s by more than 3.3%, and worse than Pandas’s by more than 1.4%, at 95% confidence. Table 2. Historical-revert testing results by corpus. Metric Extractor Django Pandas Scored 298 350 350 EFFECTIVE 288 342 334 SURVIVED 9 7 11 UNREACHED 1 1 5 EFFECTIVE Rate 96.6% 97.7% 95.4% 95% CI (Wilson) 93.9–98.2% 95.6–98.8% 92.7–97.2% Two mechanisms account for essentially all SURVIVED and UNREACHED tests across all three corpora: 1. Weak-value/loose-bound oracle. The production code path changes when the commit is reverted and the reverted code is executed by the test, but the test’s assertion does not distinguish the old behaviour from the new. This is the dominant failure mode in every corpus. 2. Wrong entry point / bypassed wrapper. The commit changes only one route in the affected logic, and the test exercises a different route. Depending on how much of the changed code sits behind that route, this can be scored as SURVIVED (some of the change is still reached, but not the part that matters) or as UNREACHED (none of it is). An example from Extractor is a concrete illustration of the first mechanism. The function under test by test_flags_struct_behind_name_still_detected decides whether an Objective-C ivar is a “flags struct” (a nested struct whose fields are all single-bit flags) by inspecting its raw type-encoding string. Before the commit, this decision used a broad rule. The commit replaces this with a more precise one: a specific marker token must appear, and it may now appear anywhere in the string, not only at the very start. The test’s chosen fixture happens to satisfy both rules at once (it starts with the marker token), so reverting the commit does not change the test’s result. 4.2. Commit-based mutation 4.2.1. Methodology The population is the same set of commits scored by the historical-revert evaluation above. Mutations are generated using cosmic-ray v8.4.677 7 PIT operates on JVM bytecode. Cosmic-ray and mutmut are the two most actively maintained tools for mutating Python code; other options (e.g. MutPy, Mutatest) exist but are narrower in scope or largely unmaintained (Diallo et al. 2024). We used cosmic-ray rather than mutmut because it allows control over the mutation operators used.. Following Laurent et al.’s empirical analysis of PIT’s own default operator set (Laurent et al. 2017), we base mutation on PIT’s curated operator set88 8 This set includes MATH (arithmetic-operator replacement), CONDITIONALS_BOUNDARY (e.g. <,<=), NEGATE_CONDITIONALS (==,!= and similar), INVERT_NEGATIVES, INCREMENTS (++,--), and VOID_METHOD_CALL/RETURN_VALS (deletes/replaces return value). . The kill rate reports the percentage of mutants that cause the test to fail i.e. that are detected by the test and so “killed”. 4.2.2. Results Table 10 shows that the confidence bounds of the kill rate overlap for all three corpora. The one-sided 95% non-inferiority bound calculated by bootstrapping rules out Extractor’s kill rate being worse than Django’s by more than 0.5%, at 95% confidence. Against Pandas the bound is positive (+2.4%), which rules out Extractor’s kill rate being worse at all i.e. the data is 95% confident Extractor’s kill rate is at least 2.4% higher than Pandas. (The table also shows the coverage-based mutation results discussed below, §4.3, for direct comparison.) Table 3. AST mutation testing results by corpus, per-commit and coverage-based protocols99footnotemark: 9 ,1010 10 Observe that for Extractor the number of tests is larger than the number of commits. Django and Pandas had one test per commit, whereas Extractor often included multiple tests in one commit.. Metric Extractor Django Pandas Per-commit Commits 48 100 116 Tests 234 100 116 Scored mutants 695 273 291 Killed 370 131 131 Survived 325 142 160 Kill rate 53.2% 48.0% 45.0% 95% CI (Wilson) 49.5–56.9% 42.1–53.9% 39.4–50.8% Coverage-based Tests 249 342 344 Scored mutants 2,791 2,744 1,616 Killed 1,293 1,373 738 Survived 1,498 1,371 878 Kill rate 46.3% 50.0% 45.7% 95% CI (Wilson) 44.5–48.2% 48.2–51.9% 43.3–48.1% 1010footnotetext: The number of tests scored is different between the per-commit and coverage-based evaluations. This is because production code with no operators to mutate is not included in the counts. The per-commit protocol mutates the commit code while the coverage-based protocol mutates code estimated from test coverage. Often the code estimated from test coverage was found to contain more mutable operators than the commit code, and so the number of tests evaluated is higher. 4.2.3. Interpreting kill rate Despite its popularity, mutation kill rate has no validated absolute scale. The intuition behind treating “higher is better” as a universal rule is that a suite killing more mutants must be more thorough. Thoroughness, however, in the sense that matters (catching real bugs), is exactly the property the literature finds weakest. Papadakis et al. (Papadakis et al. 2018) show that once test suites are compared at matched sizes, the correlation between mutation score and real fault detection is weak. Instead they show that when test suites are ranked by kill rate, higher-ranked suites tend to have better fault detection than the full population of same-size suites. That is, the kill rate’s ordinal information (higher kill rate suites tend to be better, relative to peers of the same size) remains useful. With this in mind, the results here suggest that the Extractor, Django and Pandas test suites can be expected to provide similar levels of fault protection. 4.3. Coverage-based mutation 4.3.1. Methodology For each test the production code executed by the test is estimated using a custom coverage analysis tool. That code is then mutated using cosmic-ray and the kill-rate reported. Coverage-based mutation raises a scoping question the per-commit protocol avoids: a test’s own call tree can pass through production code that is just scaffolding (e.g. fixtures, builders, mocks, setup/teardown) and has nothing to do with what the test is actually checking. Our coverage tool tries to detect and exclude such scaffolding. It uses two strategies for this: (i) exclude code outside the module(s) a test directly imports, (i) try to identify shared frameworks/libraries that most tests in the corpus execute incidentally and exclude these. 4.3.2. Results Table 10 shows that the confidence bounds of the kill rate overlap for all three corpora. The one-sided 95% non-inferiority bound calculated by bootstrapping rules out Extractor’s kill rate being worse than Django’s by more than 5.9%, at 95% confidence, and rules out being more than 2% worse than Pandas. Bearing in mind the discussion in §4.2.3, the data suggests that the test suites of all three corpora provide similar levels of fault protection. 4.4. Whole package mutation Our main interest here is in evaluating the quality of individual tests. However, in the literature it is more common to instead apply mutations to the whole code-base of a package and evaluate the ability of the whole test-suite to detect these mutations. For completeness, Table 4 shows the measured full code-base/whole test-suite mutation kill rates for Extractor and the django/utils/ component of Django (chosen because it is roughly the same size as the Extractor code). It can be seen that the Extractor and Django test-suites have almost identical kill rates. Table 4. Whole-test-suite mutation kill rate, Extractor vs. Django utils package. Metric Extractor Django Scored 679 679 Killed 399 396 Survived 280 283 Kill rate 58.8% 58.3% 95% CI (Wilson) 55.0–62.4% 54.6–62.0% 5. Qualitative test-quality results The mutation-testing analysis above tries to answer “does the test catch a wrong implementation”, a behavioural, pass/fail question. In this section we consider a separate, complementary qualitative evaluation that asks design-quality questions e.g. is a test as precise as it could be without becoming fragile, does it couple to implementation incidentally, does it exhibit a known anti-pattern, does it check one cohesive claim. Table 5. The seven-axis qualitative rubric: what each axis asks, and its verdict scale. Axis Verdict scale What it asks Intent fidelity MATCH / PARTIAL / MISMATCH Does the assertion actually check what the docstring/name/comment claims? Assertion strength STRICT / MODERATE / WEAK / n/a Is the assertion at the ceiling of precision the claim calls for, without introducing fragility it does not need? Behaviour/implementation coupling BEHAVIOR / JUSTIFIED-MECHANISM / UNJUSTIFIED-COUPLING Does it assert an externally observable outcome, or an incidental implementation detail that would break under a legitimate refactor? Anti-pattern present / absent per category A checklist for named smells: dead lever, assertion masking (coverage-real or diagnostic-only), over-broad mock, swallowed exception, log-instead-of-assert, tautology, skip/xfail masking Setup/fixture proportionality OK / DEFECTIVE Does Arrange build only what is needed, and does it construct the scenario the test actually claims to exercise? Purpose stated CLEAR (per-test/class/module/external) / PARTIAL / WRONG / NONE Does some source (docstring, comment, commit message, or a linked issue tracker) state what the test verifies and why? Cohesion SINGLE / BUNDLED Does the test check one cohesive claim, or bundle two or more independent claims into one method? Scoring uses a seven-axis rubric, shown in Table 5, and is carried out by AI agents. The instructions given to the agents, including the full definition of each category above, are given in the Additional Material. The defects in all tests categorised as having critical flaws are verified by human manual analysis. Table 11 summarises the qualitative evaluation results, mapping the non-ideal scoring verdicts to four severity tiers: critical / moderate / minor / informational. It can be seen that the coupling, purpose-stated and setup/fixture axes generally raise few issues. Anti-patterns are relatively common in all three corpora. Cohesion issues are infrequent in the Extractor tests, but more common in Django and especially Pandas. In all three corpora tests are flagged as having intent-fidelity and assertion-strength issues, and these tend to be more common for Extractor. A bootstrapped one-sided non-inferiority bound rules out Extractor’s critical rate exceeding Django’s by more than 2.3% and Pandas’s by more than 3.2%. More importantly, the absolute number of tests flagged is small. Overall only 12 tests are flagged as critical (6/5/1 for Extractor/Django/Pandas). These counts are small enough that the cross-corpus comparison should be read cautiously: a handful of borderline moderate/critical calls going the other way would change the ordering. Note also that there are likely to be false negatives (i.e. test defects that are not discovered), see §6. Table 6. Per-axis severity breakdown, by corpus1111 11 Note that a test is counted under every axis where it scored non-ideal, so two independently real properties of the same test, e.g. a weak assertion and a thin docstring, are counted separately in the row tallies. The headline numbers, e.g. critical=6 for Extractor, report the actual number of tests affected.. Intent-fidelity Assertion strength Coupling Anti-pattern Setup Purpose-stated Cohesion Extractor (n=298, critical=6 (2.0%), moderate=58 (19.4%)) Critical 4 6 0 1 2 0 0 Moderate 9 37 0 18 1 1 1 Minor 0 0 0 5 0 3 0 Info 0 0 11 0 0 5 0 Total 13 43 11 24 3 9 1 Django (n=342, critical=5 (1.5%), moderate=44 (12.9%)) Critical 2 2 0 0 2 0 0 Moderate 5 36 0 3 1 0 8 Minor 0 0 0 4 0 79 0 Info 0 0 28 0 0 149 0 Critical+Moderate 7 38 0 3 3 0 8 Pandas (n=344, critical=1 (0.3%), moderate=51 (14.8%)) Critical 0 1 0 0 0 0 0 Moderate 6 29 2 1 5 13 21 Minor 0 0 0 1 0 5 0 Info 0 0 18 0 0 56 0 Total 6 30 20 2 5 74 21 5.1. Curated exemplars This section presents a handful of illustrative findings, tagged by test_id. These are not the full data set, they are a “vivid, compelling” subset (Braun & Clarke’s Phase 6) chosen to make each axis’s numbers legible, not just countable. 5.1.1. Detailed example Before the per-axis entries below, we present one Extractor finding worked through in full to show the kind of evidence used to arrive at a critical tag. It is not enough that a potential defect is identified, it needs to be confirmed by mutating the code and a fix demonstrated. Test: test_resolve_image_exact_full_path def test_resolve_image_exact_full_path(self): """An exact full path resolves via tier 1 even when a shorter name would too.""" full = ’/System/Library/Frameworks/’ \ ’Foundation.framework/Foundation’ path = self._make_cache([(0x181200000, full)]) with DyldCacheReader(path) as r: self.assertEqual( r.resolve_image(full), 0x181200000) The function under test, resolve_image, tries three tiers in order and returns at the first to yield a candidate: (1) exact full-path match, (2) framework-principal-binary identity, (3) unique substring. Claim vs. check: the test docstring claims this proves tier 1 wins “even when a shorter name would too”, a priority claim. But the fixture builds a cache with exactly one image and queries it with that image’s own full path. Tier 3 (“unique substring”) trivially matches too since a string is always a substring of itself. With only one image present, whichever tier actually resolves the query returns the identical address, so the test cannot distinguish “tier 1 fired” from “tier 1 is broken and tier 3 silently covered for it.” Live proof: tier 1 was commented out entirely in a dedicated pinned worktree at this test’s own introducing commit, and the real test rerun. It still passed: tier 3’s substring check returns the same address the deleted tier 1 would have. The fix: add a second image whose path merely contains the queried string, making tier 3 ambiguous, so only tier 1’s exact match can resolve the original query uniquely. This detail also makes human verification of the AI analysis relatively straightforward. 5.1.2. Intent-fidelity test_json_response_raises_type_error_with_safe_arg (Django, critical). The test’s docstring claims two things: bad input raises TypeError, and passing safe=True at all raises a deprecation warning. Only the first is actually checked (the mechanism why is covered under axis 4, below). The second claim goes untested, a mismatch between what’s claimed and what’s verified, which is why this is scored critical. 5.1.3. Assertion strength test_resolve_image_exact_full_path (Extractor, critical), see above. 5.1.4. Behaviour/implementation coupling test_has_key_race_handling (Django, JUSTIFIED-MECHANISM). The assertion checks mocked_open.assert_called_once(), an implementation detail that looks at first glance like exactly the kind of coupling axis 3 penalizes. But the bug this test guards against is specifically about which low-level mechanism closes a TOCTOU (time-of-check-to-time-of-use) race: catching open()’s own exception, versus checking a file’s existence and opening it as two separate steps, which leaves a window for another process to change the file in between. Here the mechanism is the behaviour being promised, so testing it directly is the right call, not a smell. 5.1.5. Anti-pattern test_json_response_raises_type_error_with_safe_arg (Django, critical). Combines assertRaisesMessage/assertWarnsMessage in one with (A, B): tuple, checking for a TypeError and a deprecation warning at the same time: with ( self.assertRaisesMessage(TypeError,...), self.assertWarnsMessage(RemovedInDjango71Warning,...), ): JsonResponse([1, 2, 3], safe=True) Python exits stacked context managers in reverse order: when the code raises, the warning check exits first, sees an exception already in flight, and defers to it without ever checking whether its own warning fired. The warning check is present in the source and looks like it should catch a regression, but structurally can never run its own pass/fail logic – a so-called dead lever. The test would not notice if the deprecation warning it was explicitly written to guard were deleted entirely. 5.1.6. Setup/fixture proportionality test_forward_branch_handler_dominance (Extractor, critical). This test’s setup builds two branches of an if/else, but never rejoins them into a single path afterward. The logic under test exists specifically to determine which earlier branch controls a given later point in the code – and that’s only genuinely hard to get right at the point where branches rejoin. Because the setup never creates that rejoined case, nothing the test checks can catch a wrong answer. That’s a setup problem, not just a weak assertion: the scenario it builds is incapable of exercising the one case that matters. 5.1.7. Purpose stated test_read_index_col_none (Pandas, minor). The only comment on this test (“GH 7369, make sure can read a 0-obs dta file”) is copied verbatim from the sibling test test_read_empty_dta immediately above it in the same class. The text describes that other test’s scenario, not this one’s: this test’s DataFrame has 5 rows, not 0, and its actual claim, confirmed by the introducing commit PR #49745, is that index_col=None should produce a RangeIndex. Following the comment’s own issue reference makes things worse, not better: GH 7369 is the empty-dta-file bug, an entirely different issue than the one this test actually guards against. 5.1.8. Cohesion test_json_response_raises_type_error_with_safe_arg (Django, critical). Bundles a validation claim (bad input raises TypeError) with an unrelated deprecation claim (using safe= at all warns) into one method, and this is causal, not merely correlational: it is exactly this bundling that forces the compound with (A, B): whose exit-order semantics produce the axis-4 dead lever above. Two separate test methods would have made the bug structurally impossible, independent of getting the context-manager order right. 6. Test Triage Scoring individual tests, not just test-suites in aggregate, lets us flag the specific tests worth a closer look and also surface common patterns among them. It also lets us check whether different evaluation methodologies, e.g. the mutation-based and qualitative methods, agree on which tests are problematic or flag different ones. 6.1. Comparing historical-revert and qualitative scoring Table 7 summarises the number of tests flagged as problematic only by the historical-revert evaluation, only by the qualitative evaluation, and flagged by both. Also shown are the number of those tests that manual analysis confirm as having defects1212 12 See §10.3 for a full list of tests flagged, together with commentary on the actual behaviour of each test.. Table 7. Tests confirmed genuinely defective / tests flagged, by which methodology flagged them, by corpus. “Qualitative” = assertion-strength WEAK; “Historic-revert” = SURVIVED/UNREACHED (non-EFFECTIVE). Corpus Flagged by both Qualitative only Historic-revert only Extractor 2/2 (100%) 5/5 (100%) 2/6 (33.3%) Django 3/3 (100%) 6/6 (100%) 2/12 (16.7%) Pandas 0/0 (–) 3/3 (100%) 7/15 (46.7%) Seven Extractor tests are flagged by the qualitative evaluation and all are confirmed to have defects. Of these seven, only two fail as SURVIVED by historical-revert, the rest are flagged by historical-revert as EFFECTIVE1313 13 Three as EFFECTIVE (TESTING_NEW_FUNCTION). This occurs for a commit that introduces an entirely new class/module, or a new method/field/parameter on an already-existing class or function, plus associated tests. Reverting the change to the production code leaves the tests with no function/module/method to execute and so they fail to execute. This is categorised as EFFECTIVE because the revert can be trivially detected.. Eight Extractor tests are flagged as non-EFFECTIVE by historical-revert but four are genuinely flawed. Two of the flawed tests were flagged by both the historical-revert and qualitative evaluations. The other two flawed tests are not flagged by the qualitative evaluation. The other four tests are correctly found to be well-written by the qualitative evaluation. Table 8. False-positive rate of each triage methodology, by corpus. Corpus Qualitative FP rate Historical-revert FP rate Extractor 0/7 (0%) 4/8 (50.0%) Django 0/9 (0%) 10/15 (66.7%) Pandas 0/3 (0%) 8/15 (53.3%) Similar behaviour is observed for Django and Pandas. Table 8 gives each methodology’s false-positive (FP) rate i.e. of the tests each flags as problematic, how many turn out, on independent inspection, to have no real defect. The false positive rate is much higher for the historical-revert evaluation. This reflects the fact that a single mutation (reverting the production code changes in the commit containing the test) is often not discriminating enough to accurately evaluate overall test quality. The qualitative evaluation does not produce false positives, but it can produce false negatives: tests which have defects but which are assigned an assertion-strength of STRICT. 6.2. AST-mutation scoring (a) Per-commit. (b) Coverage-guided. Figure 1. Kill-rate distributions for Extractor, with defective tests marked by dashed line. Figure 1 shows the distribution of kill rates across all tests and the dashed lines indicate the kill rates of the nine Extractor tests1414 14 The per-commit plot shows 8 tests, the coverage-guided 6 tests. This is because the commit/code-covered by the other tests contained no operators to mutate i.e. this is correct behaviour. found to have a real defect in §6.1. Neither distribution cleanly separates defective from non-defective tests. It is not clear whether this is because the kill rate is not informative of test defects, or because the test defects uncovered by historical-revert and qualitative evaluation are an incomplete, possibly biased, sample. As already discussed above, there is good reason to suspect that kill rate is, at best, only weakly correlated with test quality1515 15 We can see how this might be the case by looking at a concrete example: Extractor’s test_word_substitution kills 57% of its commit’s mutations and 55% of its coverage-guided mutations, neither figure an obvious red flag. The production code this test exercises has two structurally distinct branches, a lowercase and an uppercase branch, but the test is defective because it only checks the uppercase branch. Of the 37 mutants tested, 35 modify code surrounding the branch but not the branch itself. One mutant modifies the uppercase branch and is killed, one modifies the lowercase branch and survives. The raw kill rates therefore tell us little about the defect in the test, only inspecting which mutants survive or reading the code directly can do this.. Nevertheless, while a low kill rate might only be a weak signal it might still be useful as a filter to narrow down a set of tests for closer inspection. To investigate this, we took the 25 lowest kill-rate Extractor tests under the per-commit and coverage-guided mutation evaluations (42 unique tests, some appearing in both evaluations) and used AI agents to analyse these for the failure mechanisms above plus any other defect pattern that the survived mutants suggested (see Additional Material for instructions given to the agents). Five tests were found to have a defect, three not flagged by the mutation or qualitative evaluations1616 16 See Additional Material §10.4 for details, including manual analysis of the defects.. That is, of the 42 low kill rate tests examined, 12% turn out to have defects. For comparison, we also carried out the same analysis of the 25 highest kill-rate Extractor tests and found no defective tests. A low kill rate therefore does indeed seem to be a useful filter for test triage. Its usefulness is not confined to Extractor. Table 9 shows that a similar filter also finds defects by examining the 25 tests with lowest kill rates in Django and Pandas, though at a lower rate for Django. Five of the six defects found this way were independently found by the qualitative evaluation, only one (test_groupby_resample_kind in Pandas) was missed by both the mutation and qualitative evaluations. All but one of the Django and Pandas tests flagged are new i.e. not flagged by the historic-commit or qualitative evaluations. Table 9. Low-kill-rate triage defect-find rate, by corpus. Corpus Per-commit Coverage-guided Combined unique Extractor (AI) 3/25 3/25 5/42 Pandas (human) 4/25 2/25 5/46 Django (human) 0/25 1/25 1/50 7. Results for other AI-authored corpora The results above are for one AI-authored codebase, Extractor. Since we developed Extractor ourselves, its provenance is clear. In particular, we know that the tests were wholly AI written with no human review/modification and no detailed human prompting/instructions. In this section we extend the analysis to two open-source codebases with AI-authored tests, Quay and AWX1717 17 See https://github.com/quay/quay, https://github.com/ansible/awx.. 7.1. Search for AI-authored code We initially carried out a search of github for open source Python packages that are wholly AI authored, i.e. similar to Extractor. We cloned candidate repositories for closer inspection of their commit history, code size, language used etc. We also searched chat groups focussed on AI code development1818 18 Hacker News, Reddit r/ClaudeAIm r/LocalLLaMA for discussion of such code, curated “awesome-vibe-coding”/“awesome-ralph”/“awesome-claude-code” lists, as well as used direct web search. We found that repositories that explicitly brand themselves “vibe coded” or “written entirely by Claude” are overwhelmingly “toy” scale i.e. a few hundred of lines with no real test suite. We then revised the search to focus on codebases: (i) with more than 100 AI-authored Python tests (other parts of the code, including some of the tests, could be human-authored), (i) PR’s and/or commits with a ‘Co-authored-by: Claude’ tag and stating version Opus/Sonnet 4.6 or later, and (i) code and tests are written together (rather than tests being back-fitted to existing code). In addition to direct search of github we also searched the AIDev dataset (933K agent-authored PRs), but the content of this is too old to cover Opus/Sonnet 4.6. These searches turned up two main candidates: Quay and AWX. Other codebases, including RNAlysis, ha-pitboss, pypureomapi and ouroboros, were found to have fewer than 100 AI-written tests. The Paddleformers1919 19 https://github.com/PaddlePaddle/PaddleFormers codebase by Baidu met criteria (i) and (i) but was excluded because analysis of the commits makes it clear that the tests are back-fitted to existing code. We discuss the restriction to the recent2020 20 Opus 4.6 was released on 5th Feb 2026, Sonnet 4.6 on 17th Feb 2026. Opus 4.6/Sonnet 4.6 and later models further below. Quay (quay/quay) is an open-source container image registry supporting Docker Registry and OCI, with authentication, ACLs, geo-replicated storage, and security scanning. It is maintained by Red Hat, which also uses it to run the public quay.io registry service. AWX (ansible/awx) is an open-source web-based UI, REST API, and task engine for running and scheduling Ansible playbooks (Ansible is an open-source IT automation tool that configures systems, deploys software, and orchestrates multi-machine tasks via YAML “playbooks”) and is one of the upstream projects for Red Hat’s Ansible Automation Platform. In both packages there are a substantial number of recent commits that contain new tests carrying a “Co-authored-by: Claude” tag on the commit, see Table 10. Table 10. Details of Quay and AWX. Corpus Written by Domain Size (src/test/AI-test LOC; tests/AI-tests Quay (quay/quay) Claude Opus 4.6, Sonnet 4.6 Cloud infrastructure ∼ 128,300/∼ 79,700/∼ 3,600, ∼ 3,800/214 AWX (ansible/awx) Claude Opus 4.6/4.8, Fable 5 IT automation ∼ 104,900/∼ 58,400/∼ 1,200, 2,674/102 7.2. AI workflow To try to better understand the AI test writing workflow used in Quay and AWX we (i) analysed the pattern of commits, (i) contacted the authors. 7.2.1. Commit analysis In Quay and AWX PRs multiple commits are merged. However, the individual pre-merge commits can be recovered from the GitHub API (via refs/pull/<N>/head and git blame) and so we are able to analyse these. Both the Quay and AWX commits containing AI written tests almost always bundle a feature’s implementation and unit tests together. Figure 2 shows that each commit typically contains multiple tests, similarly to Extractor. Figure 2. Tests introduced per individual commit, sorted in descending order, for Quay, AWX, and Extractor. In summary, the commit analysis indicates that the AI tests were generally developed incrementally alongside new production code. 7.2.2. Communication with code authors We contacted the code authors asking for information about the workflow they used for AI-authored tests. A Quay maintainer confirmed Claude Code produced these PRs largely autonomously: Claude was instructed to take a JIRA ticket number as input and to carry out the entire implementation, with explicit instructions to write both unit and integration tests along the way2121 21 Quay’s /create-plan-from-issue command, e.g. as of commit 92eddf2: https://github.com/quay/quay/blob/92eddf2dbc266eeed2d138c6dcaa7fe38c5f7479/.claude/commands/create-plan-from-issue.md. We did not receive a response from the AWX developers that we contacted, so the human input to the AI test writing is unclear. In particular, the level of detail given in the instructions to Claude and the amount of human review / re-writing of the tests are both unknown. 7.3. Coverage-guided mutation results Table 11. AST mutation testing results by corpus, coverage-based protocol. Corpus Tests (SCORED) Scored mutants Killed Survived Kill rate 95% CI (Wilson) Quay 128 1,825 817 1,008 44.8% [42.5%, 47.1%] AWX 50 665 391 274 58.8% [55.0%, 62.5%] Table 12. One-sided 95% non-inferiority bounds on the coverage-guided kill-rate, by corpus. Corpus vs. Django vs. Pandas Quay 7.8% 3.8% AWX 0% 0% Extractor 5.9% 2.0% Table 11 shows the measured kill-rates for Quay and AWX using coverage-guided mutation, and Table 12 shows the corresponding non-inferiority bounds relative to Django and Pandas. The latter rules out Quay’s kill rate being worse than Django/Pandas by more than 7.8%/3.8%, at 95% confidence, and rules out AWX’s kill rate being worse than Django or Pandas at all. The Quay bounds are larger than for Extractor but this is at least partly because of the smaller sample size, which increases the uncertainty in the measured critical rate and so increases the bound. Bearing in mind the discussion in §4.2.3, the data suggests that the test suites of all three AI-corpora provide similar levels of fault protection to Django and Pandas. 7.4. Qualitative test-quality results Table 13. Per-axis severity breakdown, by corpus. Intent-fidelity Assertion strength Coupling Anti-pattern Setup Purpose-stated Cohesion Quay (n=214, critical=8 (3.7%), moderate=39 (18.2%)) Critical 7 4 0 2 5 0 0 Moderate 17 27 2 5 4 4 1 Minor 0 0 0 3 1 3 0 Info 0 0 13 0 0 45 0 Critical+Moderate 24 31 2 7 9 4 1 AWX (n=102, critical=3 (2.9%), moderate=15 (14.7%)) Critical 3 1 0 2 3 2 0 Moderate 2 13 2 0 1 5 0 Minor 0 0 0 0 0 1 0 Info 0 0 16 0 0 31 0 Critical+Moderate 5 14 2 2 4 7 0 Table 14. One-sided 95% non-inferiority bounds on the qualitative-rubric critical-defect rate, by corpus. Tests are attributable to Opus 4.6 and later. Corpus vs. Django vs. Pandas Quay 4.8% 5.6% AWX 4.6% 5.6% Extractor 2.3% 3.2% Table 13 summarises the results of the qualitative evaluations of Quay and AWX, while Table 14 shows bootstrapped one-sided non-inferiority bounds for their critical rates relative to Django and Pandas. These bounds rule out Quay’s critical rate exceeding Django/Pandas by more than 4.8%/5.6% and AWX’s by more than 4.6%/5.6%. Also included in the table are the Extractor values from §5. It can be seen that the Quay and AWX bounds are somewhat larger than the bounds for Extractor. This is partly because of the smaller sample sizes for Quay, and especially AWX, which increases the uncertainty in the measured critical rate and so increases the bound2222 22 Keeping the percentage of critical samples fixed then for a population of 298 tests (matching Extractor) the AWX bounds decrease to 3.5%/4.4% from 4.6%/5.6%, and the Quay bounds decrease to 4.3%/5.4%. Nevertheless, the relative number of critical tests for Quay and AWX does seem slightly higher than for Extractor: 2.9% (3/102) critical AWX tests and 3.7% (8/214) Quay tests vs 2.0% (6/298) for Extractor. However the absolute number of tests flagged as critical is small (3/8/6) and so this comparison should be read cautiously since changes in one or two tests would be enough to make the numbers for all three comparable. 7.5. Opus 4.5 and earlier Table 15. Per-axis severity breakdown for Quay tests attributable to Opus 4.5 and earlier. Intent-fidelity Assertion strength Coupling Anti-pattern Setup Purpose-stated Cohesion Quay (n=233, critical=24 (10.3%), moderate=67 (28.8%)) Critical 17 14 0 11 13 3 0 Moderate 22 52 5 8 7 7 3 Minor 0 2 0 1 4 4 0 Info 0 0 9 0 0 18 0 Critical+Moderate 39 66 5 19 20 10 3 We excluded tests written by Claude Opus 4.5 and earlier because they seem to be of significantly lower quality than the tests written by later models. Table 15 shows the qualitative evaluation of a sample of Quay tests written by Opus 4.5 and earlier. Observe that 24 out of 233 tests are flagged as critical i.e. 10.3%. For comparison, Table 13 shows that for Quay tests written by Opus 4.6 and later only 8 out of 214 tests are flagged critical i.e. 3.7%. The available AWX tests are all written by Opus 4.6 and later models, similarly for Extractor. 8. Discussion 8.1. Are Django/Pandas tainted by AI-authored tests? We use Django/Pandas as human-authored baselines against which to compare the AI-authored Extractor tests. Django/Pandas are both actively maintained and still accepting commits today, a period in which AI-assisted coding has become more common. Both projects have AI-contribution policies, adopted in January 2026 (Django Software Foundation 2026; pandas development team 2026) which require disclosure but neither specifies a machine-checkable format or a commit-message location for it. Searching every pull request’s body directly for AI-disclosure reveals 20 pandas PRs (26 tests) and 12 django PRs (12 tests) . These PRs all date from January 2026 or later. Cross-referencing these against the test sample used in our evaluation shows that the samples draw overwhelmingly from pre-2026 commits (only 2 of 351 Django rows post-date the AI-contribution policy). We find that recomputing each corpus’ historical-revert rate with the disclosed AI tests excluded changes the reported scores by less than 0.1%. As well as explicit disclosures, we also searched the commit text of all the tests used in our sample for keywords and phrases that might suggest AI-authorship and found none. 8.2. Are the AI-authored tests tainted by the AI’s training data? In this study we seek to evaluate the quality of tests which are AI-authored during code development. One evaluation approach would be to take a public code corpus, strip of its (likely human-authored) tests, and then ask an AI to create tests for the resulting corpus. However, there are two main problems with this approach. Firstly, the AI tests are written post code development, not during it. Secondly, public code, including its existing tests, is almost certainly incorporated in the training data of any frontier code-writing AI. That is, the AI has already had sight of the existing tests before they are stripped and this runs the risk of confounding any evaluation of AI test authorship2323 23 (Haroon et al. 2026)’s study of LLM test generation under code evolution likely gives a concrete illustration of this. 99% of AI tests that fail once the code under test has genuinely changed still pass against the original, pre-change version, while exercising the same, now-changed region. The AI tests are observed track what the code used to do rather than what it currently does, which is what memorisation of that code, drawn from the public Project CodeNet corpus, might well look like.. In this study we avoid both of these concerns. The Extractor code was developed from scratch in a private environment, and has never been published. The tests are therefore authored during code development, rather than after, and it is impossible for the AI model to have been trained on this code or otherwise internalised it into its weights. Every Extractor test scored in this report reflects the model’s own test-writing behaviour, not recall of prior exposure. 8.2.1. What does “human-authored” mean here? For Django and Pandas, “human-authored” means authored by a group of humans with the processes a mature open-source project runs. Namely, a contributor writes the test, a maintainer distinct from that contributor reviews the pull request before merge, and, for many of the bugs sampled here, an external user found and reported the defect the test exists to catch. The AI-authored tests in Extractor have no analogue to any of this. They were written by one AI agent with one human directing it: no independent reviewer, no external user base, and no detailed test-design or review guidance from that human either (see Additional Material §10.6). It is notable that the AI-authored tests in Extractor match the human-authored Django and Pandas baselines despite lacking all of that scaffolding. 8.3. Limitations 8.3.1. Scope: one language, one model family The findings in this report analyse Python code authored by one AI model family (Claude Sonnet/Opus). We make no claim that these findings generalise to other coding languages or AI models (e.g. GPT Codex). In particular, we expect the results to be sensitive to the language used. Python appears to be Claude’s preferred language in the sense that when Claude organically writes tools for its own use they are written in Python. Claude is therefore probably especially well trained in Python, and its Python code-writing performance may be better than for other languages. 8.3.2. Low base rates limit statistical power As already noted, because the measured test quality is high even evaluating hundreds of tests per corpus still only turns up a small absolute number of tests flagged as having potential flaws. These small counts mean that the reported rates are sensitive to small changes in the number of flagged tests. That sensitivity matters less, however, for our main conclusion that all three corpora have low overall defect rates (a few per cent) and that AI-written tests are not weaker than human-written ones on this corpus. This conclusion only needs that the failure rate be reliably small, not pinned down to a precise value. A few hundred tests per corpus is enough to establish that with reasonable confidence, even if it is not enough to adjudicate a one- or two-point gap between corpora. 9. Conclusion We score Claude-written tests against two human-written corpora under four independent methods: historical-revert testing, per-commit AST mutation testing, coverage-guided mutation testing, and a seven-axis qualitative rubric. This is at a larger combined scale than any prior individual-test evaluation we are aware of. On every method, the AI-written tests are broadly comparable to the two human-written corpora, a result that runs against the more pessimistic picture from prior AI-test-generation studies based on compile/pass rates. References (1) Andrews et al. (2005) James H. Andrews, Lionel C. Briand, and Yvan Labiche. 2005. Is Mutation an Appropriate Tool for Testing Experiments?. In Proc. 27th Int. Conf. on Softw. Eng. (ICSE 2005). ACM, 402–411. https://doi.org/10.1145/1062455.1062530 Bavota et al. (2015) Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David Binkley. 2015. Are Test Smells Really Harmful? An Empirical Study. Empir. Softw. Eng. 20 (2015), 1052–1094. https://doi.org/10.1007/s10664-014-9313-0 Campos et al. (2021) Denivan Campos, Larissa Rocha, and Ivan Machado. 2021. Developers’ Perception on the Severity of Test Smells: An Empirical Study. arXiv:2107.13902. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. Diallo et al. (2024) Kadiatou Diallo, Zizhao Chen, W. Eric Wong, and Shou-Yu Lee. 2024. An Analysis and Comparison of Mutation Testing Tools for Python. In Proc. 2024 11th Int. Conf. on Dependable Syst. and Their Applications (DSA). IEEE, 161–169. https://doi.org/10.1109/DSA63982.2024.00030 Django Software Foundation (2026) Django Software Foundation. 2026. AI-Assisted Contributions. Django documentation, section of “Submitting contributions”. https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/submitting-patches/#ai-assisted-contributions Garousi and Küçük (2018) Vahid Garousi and Barış Küçük. 2018. Smells in Software Test Code: A Survey of Knowledge in Industry and Academia. J. Syst. Softw. 138 (2018), 52–81. https://doi.org/10.1016/j.jss.2017.12.013 Haroon et al. (2026) Sabaat Haroon, Mohammad Taha Khan, and Muhammad Ali Gulzar. 2026. Evaluating LLM-Based Test Generation Under Software Evolution. arXiv:2603.23443. Jia and Harman (2009) Yue Jia and Mark Harman. 2009. Higher Order Mutation Testing. Inf. Softw. Technol. 51, 10 (2009), 1379–1393. Just et al. (2014) René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In Proc. 22nd ACM SIGSOFT Int. Symp. on Foundations of Softw. Eng. (FSE 2014). ACM, 654–665. https://doi.org/10.1145/2635868.2635929 Konstantinou et al. (2024) Michael Konstantinou, Renzo Degiovanni, and Mike Papadakis. 2024. Do LLMs Generate Test Oracles That Capture the Actual or the Expected Program Behaviour? arXiv:2410.21136. Laurent et al. (2017) Thomas Laurent, Mike Papadakis, Marinos Kintis, Christopher Henard, Yves Le Traon, and Anthony Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In Proc. 2017 IEEE Int. Conf. on Softw. Testing, Verification and Validation (ICST 2017). IEEE, 430–435. https://doi.org/10.1109/ICST.2017.47 Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Inf. Processing Syst. 36 (NeurIPS 2023). https://doi.org/10.52202/075280-0943 pandas development team (2026) pandas development team. 2026. Automated contributions policy. pandas documentation, section of “Contributing to pandas”. https://pandas.pydata.org/docs/dev/development/contributing.html#automated-contributions-policy Panichella et al. (2022) Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent Hellendoorn. 2022. Test Smells 20 Years Later: Detectability, Validity, and Reliability. Empir. Softw. Eng. 27, 7, Article 170 (2022). https://doi.org/10.1007/s10664-022-10207-5 Papadakis et al. (2019) Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation Testing Advances: An Analysis and Survey. Advances in Computers 112 (2019), 275–378. https://doi.org/10.1016/bs.adcom.2018.03.015 Papadakis et al. (2018) Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are Mutation Scores Correlated with Real Fault Detection?: A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults. In Proc. 40th Int. Conf. on Softw. Eng. (ICSE 2018). ACM, 537–548. https://doi.org/10.1145/3180155.3180183 Petrović et al. (2021) Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2021. Does Mutation Testing Improve Testing Practices?. In Proc. 43rd IEEE/ACM Int. Conf. on Softw. Eng. (ICSE 2021). IEEE, 910–921. https://doi.org/10.1109/ICSE43902.2021.00087 Schäfer et al. (2024) Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Trans. Softw. Eng. 50, 1 (2024), 85–105. https://doi.org/10.1109/TSE.2023.3334955 Shamshiri et al. (2015) Sina Shamshiri, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges. In Proc. 30th IEEE/ACM Int. Conf. on Automated Softw. Eng. (ASE 2015). IEEE, 201–211. https://doi.org/10.1109/ASE.2015.86 Vahabzadeh et al. (2015) Arash Vahabzadeh, Amin Milani Fard, and Ali Mesbah. 2015. An Empirical Study of Bugs in Test Code. In Proc. 2015 IEEE Int. Conf. on Softw. Maintenance and Evolution (ICSME 2015). IEEE, 101–110. https://doi.org/10.1109/ICSM.2015.7332456 van Deursen et al. (2001) Arie van Deursen, Leon Moonen, Alex van den Bergh, and Gerard Kok. 2001. Refactoring Test Code. Technical Report SEN-R0106. Centrum voor Wiskunde en Informatica (CWI), Software Engineering (SEN). Also in: Proc. 2nd Int. Conf. on eXtreme Programming and Flexible Processes in Softw. Eng. (XP 2001), p. 92–95. Vathana et al. (2026) Phouvadeth Vathana, Prapti Bhatt, Rishi Patel, and Nasir U. Eisty. 2026. LLM vs. Human Unit Tests: Fault Detection on Real Python Bugs. arXiv:2606.08588. Vercammen et al. (2018) Robbert Vercammen, Mohammad Ghafari, Serge Demeyer, and Markus Borg. 2018. Goal-Oriented Mutation Testing with Focal Methods. arXiv:1807.10953. Whitfill et al. (2026) Parker Whitfill, Cheryl Wu, Joel Becker, and Nate Rush. 2026. Many SWE-bench-Passing PRs Would Not Be Merged into Main. METR (Model Evaluation & Threat Research), online report. https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/ Yuan et al. (2024) Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM on Softw. Eng. 1, FSE, Article 76 (2024), 1703–1726 pages. https://doi.org/10.1145/3660783 Zhang and Mesbah (2015) Yucheng Zhang and Ali Mesbah. 2015. Assertions Are Strongly Correlated with Test Suite Effectiveness. In Proc. 2015 10th Joint Meeting on Foundations of Softw. Eng. (ESEC/FSE 2015). ACM, 214–224. https://doi.org/10.1145/2786805.2786858 10. Additional Material 10.1. Historical-revert evaluation procedure See Figure 3 for pseudo-code of the historical-revert evaluation protocol used. This is executed once per candidate (c, t, F), where c is the test’s introducing (post-fix) commit, t is the test, and F is the set of production files touched by c’s diff. Build-environment provisioning (era-appropriate compiler/interpreter/dependency pins) is treated as a precondition of RUN and omitted from the figure for brevity. The coverage.py tool used to evaluate code coverage is described below Algorithm GROUP-B-EVAL(commit c, test t, production files F) checkout(worktree, c) // post-fix state if RUN(t) fails: return BASELINE_FAILS // environment/build issue, excluded for f in F: WRITE(f, content_of(f, parent(c))) // revert to pre-fix content if RUN(t) fails: verdict <- EFFECTIVE // test catches the historical bug else: verdict <- REACHABILITY-CHECK(t, F) // passes even without the fix — see below for f in F: WRITE(f, content_of(f, c)) // restore post-fix state return verdict Sub-procedure REACHABILITY-CHECK(test t, production files F) trace <- RUN(t, instrumented with coverage.py --branch, scoped to F) changed <- the revert diff’s ’+’ lines (blank lines excluded — see below) if any line/branch in ‘changed‘ shows as executed in ‘trace‘: return SURVIVED // this run genuinely reached the changed code, just didn’t discriminate else: return UNREACHED // this run never executed the changed code at all Figure 3. Historical-revert evaluation procedure (pseudo-code). 10.2. Qualitative rubric agent instructions Figure 4 shows the operative instruction text given to the rubric scoring agents, stripped of per-dataset specifics (sample file paths, repo locations, batch sizes) i.e. this covers the eval process and scoring rules themselves, which do not change between datasets. Each test is scored by its own agent (there is no batching). Scoring is a two-stage process: a blind Stage A pass scores all seven axes from the test’s source alone, and a Stage B pass then re-examines the two axes mutation evidence bears on most directly (assertion strength, setup/fixture proportionality) in light of that test’s own mutants. After scoring, the defects in each test are evaluated for severity. The four severity tiers used are given in Table 16 and the instructions given to the first pass agent are shown in Figure 5. Tests categorised as Critical severity in the first pass are double checked by a panel of three independent agents, each of which re-derives whether the test deserves a “critical” severity tier for its test-quality defect. The instructions given to the panelists are shown in Figure 6. If a majority agree on Critical then that category is kept, otherwise the severity is downgraded to Moderate. Table 17 shows the changes in scoring between Stage A and Stage B for tests categorised with Critical or Moderate severity. It can be seen that changes have relatively more impact on the Critical than on the Moderate tests. Table 18 shows the corresponding breakdown by rubric scoring axis. 1) Reading order (given before any scoring): Read every candidate test’s ENTIRE file from line 1 – module docstring, class docstring, method docstring/comment – before scoring axis 6. Never rely on just the method body, or on an excerpt from another document: this is the single most common way a prior pass got a purpose-clarity score wrong. 2) The seven axes and how to score each: Axis 1 – intent-fidelity. Does the assertion actually check what the docstring/name/ comment claims? MATCH / PARTIAL (checks something related but narrower or looser than stated) / MISMATCH (passes or fails for a reason unrelated to the stated claim). Ground this by tracing what the assertion can and cannot distinguish, not by reading the name and stopping there. Write your reasoning to a notes field – a bare enum value with no notes leaves no way to check why a verdict was reached, which blocks auditing disagreements between passes. Axis 2 – assertion strength. STRICT / MODERATE / WEAK / n/a. This is a precision judgment, not a syntactic one: STRICT means the assertion is at the ceiling of precision available for what’s actually being claimed, without introducing fragility the claim doesn’t call for – literal equality when the true value is a fixed fact, a threshold when the true value is inherently a range that legitimately varies, existence when the claim is inherently about existence. MODERATE/WEAK is only a valid score if you can present a concrete, robustness-preserving change that reaches STRICT. If no such change exists – because tightening it would introduce fragility, or duplicate a sibling test’s job – promote to STRICT instead of leaving it as an unfalsifiable “could be better in theory.” Do not count the number of assertions as a proxy for strength. Axis 3 – behavior/implementation coupling. BEHAVIOR (asserts an externally observable outcome; survives a legitimate refactor) / JUSTIFIED-MECHANISM (couples to an implementation detail, but that detail is the fix under test, so the coupling is appropriate) / UNJUSTIFIED-COUPLING (asserts an incidental implementation detail that would break under a behavior-preserving refactor and would teach a future maintainer nothing). Write your reasoning to a notes field, same requirement and rationale as axis 1 above; a bare enum value with no notes gives a later rescore nothing to check disagreements against. Axis 4 – anti-pattern scan. A checklist, not a scale. A single-select field – pick the one category that most specifically owns the finding, never a comma-joined list of several. Only use these exact strings, nothing else: none / dead lever / assertion masking (coverage-real) (verify by tracing the actual implementation, don’t guess from syntax) / assertion masking (diagnostic-only) (the sequential assertions provably share one underlying line/call) / over-broad mock / swallowed exception / log-instead-of-assert / tautology (cannot fail under any input) / skip/xfail masking. If a test genuinely exhibits more than one, name the single most severe/specific one here and mention the others in the notes field – the column itself must stay one of the exact strings above. Axis 5 – setup/fixture proportionality. Output is OK / DEFECTIVE / n/a – exactly one of these three strings. Two sub-questions inform the verdict: does Arrange build only what’s needed (excess/bloat), and does the fixture actually construct the scenario the test claims to exercise, or does it accidentally construct a different one that happens to produce the same superficial outcome for an unrelated reason? Either failure mode scores DEFECTIVE – say which one, and why, in the notes field. The second is the more severe failure mode: no mutation of the actual logic under test can ever be caught if the defect is upstream in Arrange. Axis 6 – purpose stated. Does some source state what the test verifies and why? Two independent things determine the verdict: whether a qualifying source exists at all, and – if so – whether a reader encounters it in the file itself (docstring/comment at any granularity) or only in VCS history (the introducing commit message, with nothing recovered in-file). These are not interchangeable – a reader who never runs git log on this file never sees the second kind – and collapsing them into one CLEAR value is a confirmed source of disagreement between scoring passes. Record which family explicitly:: • CLEAR_INFILE_SOURCE (per-test) / CLEAR_INFILE_SOURCE (class-level) / CLEAR_INFILE_SOURCE (module-level) / CLEAR_INFILE_SOURCE (external) (via an in-file reference to an issue tracker) – any of these require an actual docstring/comment/in-file trace, not a method name that “sounds right” (a name is not a stated source at any tier). • CLEAR_COMMIT_TEXT (per-test) (the introducing commit is specifically about this one test) / CLEAR_COMMIT_TEXT (shared) (the commit explains the purpose but is a broader/batch commit not calling this test out individually) – use this pair only when nothing in-file states the purpose at any granularity. A commit message is a real, legitimate source – don’t invent a stricter bar than the rubric already has – but it is measurably weaker than an in-file one and must be recorded as such, never silently folded into CLEAR_INFILE_SOURCE. • PARTIAL (something exists – in-file or commit – but under-explains) / WRONG (a source exists, is specific, and is factually incorrect) / NONE. Check all of: in-file docstring/comment at every granularity, the introducing commit message, and whether an external tracker is reachable via an in-file reference – before concluding anything, and before picking which family applies. Axis 7 – cohesion (one thing per test). Does the test exercise a single cohesive behavioral claim (SINGLE), even via multiple assertions that are all facets of that one claim, or does it bundle two or more genuinely independent claims into one method (BUNDLED)? For parametrized tests, each case checking the same claim under different inputs is SINGLE, not BUNDLED. 3) Scoring: Score all seven axes above from the test’s source (and, for axis 6, its commit) alone. 3) Stage A – blind scoring Score all seven axes above from the test’s source (and, for axis 6, its commit) alone. Do not open, list, or otherwise consult any mutation-testing output for this test – no coverage_guided_mutation_study results, no ast_mutation_study results, no kill rate, no mutant list. You may still perform source-grounded live verification (run the test as it stands, or hand-construct your own corrupted/mutated state and run the real assertion against it) – that is your own reasoning about the test, not a lookup into either mutation study’s corpus, and does not leak Stage-B information. Write Stage A’s verdicts to axis1_blind … axis7_blind, with per-axis notes. These columns are frozen once written – Stage B never edits them. 4) Stage B – mutation-informed refinement Given: Stage A’s own verdicts/notes, plus this test’s candidate mutants. Re-examine axis 2 and axis 5 in light of them – these are the two axes a concrete survived/killed mutant most directly bears on – but any axis may be revised if the evidence genuinely changes the picture (e.g. a survived mutant in dead code bears on axis 4’s dead lever, not axis 2). Candidate-mutant sourcing, in priority order: (i) Primary: this corpus’s ast_mutation_study results, keyed by commit hash, already pinned to the test’s own introducing commit. (i) Supplementary only: this corpus’s coverage_guided_mutation_study results – generated against current HEAD, not necessarily this test’s own commit, so its killed/survived label is never trusted directly. (i) If neither source has any mutants for this test, copy Stage A’s verdicts forward unchanged and stop – don’t fabricate mutation evidence for a test that has none. Verification is mandatory and must happen at the test’s own pinned commit – never against the live working tree. For each candidate mutant relied on: create a dedicated, throwaway git worktree at the test’s own commit (never a shared worktree another agent might be using concurrently), hand-patch the mutation into the pinned source, rerun the real test there, and use that pass/fail result – not any study’s recorded label – as the evidence. Before treating a patched-in survivor as a real weakness, confirm the mutated line is actually reachable by this specific test – most survivors are unreachable-by-this-test noise (dead code, another test’s path, a downstream guard already masking the outcome), not genuine gaps. Revert the hand-patch (or discard the worktree) once done; never leave a mutated file checked in. Write to axis1_refined … axis7_refined, plus per-axis refine notes for anything that changed, and a single refine_delta = yes/no. Also record mutant_source. Sanity expectation, not a target to game: mutation evidence should essentially never move a verdict stricter than blind Stage A – only weaker or unchanged. A run showing a meaningful rate of stricter changes should be treated as a likely leakage signal (e.g. accidentally reading the live working tree instead of the pinned worktree), not a new finding to report at face value. 4) Output columns: corpus, test_id, file, commit_hash, axis1_blind, axis1_blind_notes, axis2_blind, axis2_blind_notes, axis3_blind, axis3_blind_notes, axis4_blind, axis4_blind_notes, axis5_blind, axis5_blind_notes, axis6_blind, axis6_blind_notes, axis7_blind, axis7_blind_notes, axis1_refined, axis1_refined_notes, axis2_refined, axis2_refined_notes, axis3_refined, axis3_refined_notes, axis4_refined, axis4_refined_notes, axis5_refined, axis5_refined_notes, axis6_refined, axis6_refined_notes, axis7_refined, axis7_refined_notes, refine_delta, mutant_source, flag flag = yes if any axisN value is non-ideal, else no. Ideal values: axis1=MATCH, axis2=STRICT, axis3=BEHAVIOR, axis4=none, axis5=OK, axis6=CLEAR_INFILE_SOURCE () at any granularity, axis7=SINGLE. CLEAR_COMMIT_TEXT () is not ideal and must flag – same tier as PARTIAL. Figure 4. Qualitative rubric agent instructions. Table 16. Severity tier definitions for non-ideal rubric verdicts. Severity Definition Critical A genuine behavioural defect: axis-1 MISMATCH, axis-5 DEFECTIVE with no mitigation, or an axis-2 strength WEAK or MODERATE proven to have missed a real mutation. Moderate A real, presentable gap, with no proof yet of an actual missed defect. Minor Diagnostic-only assertion masking, a tautology, or an axis-6 WRONG/PARTIAL docstring that does not affect the test’s own correctness. Informational axis-3 JUSTIFIED-MECHANISM, axis-6 CLEAR_COMMIT_TEXT when it’s the row’s only flagged reason; any finding whose own note calls the deviation borderline/trivial. You are proposing a severity tier for already-scored, already-flagged qualitative test-quality rubric rows from the [corpus name] corpus. Each file you’ve been given is a standalone CSV – one header row plus one data row – already fully scored across all seven axes (schema v2), with flag = yes. Your job: read each row’s own already-recorded notes and propose a severity tier for the row as a whole. Do not re-run anything, do not open source files, and do not invent new evidence – work entirely from what is already written in the row’s own notes, paying particular attention to whether a finding’s verification was a genuine live hand-patch-and-rerun (look for phrases like “hand-verified,” “reran pytest,” “patched … rerun … FAILED/PASSED”) versus source-only reasoning (no mutation evidence available, verdict copied forward, “NO_CANDIDATES”). Important: when you report the test’s identity, copy the value verbatim from the row’s own test_id column – never derive it from the filename or shorten it. Severity tiers Severity tiers are: critical, moderate, minor, informational. Critical is narrow – reachable from only three findings, and nothing else qualifies no matter how severe it reads: axis1_refined = MISMATCH, and the row’s own notes record a live check (a test run, or a hand-patch followed by a rerun) that concretely demonstrates the mismatch. axis5_refined = DEFECTIVE for the “wrong scenario” reason (not just excess/bloat setup), and the row’s notes record a live check proving it. axis2_refined = WEAK or MODERATE, and the row’s Stage B notes show that the live check itself constructed the corrupted or mutated state, reran the real test, and the existing assertion still passed. If the verification behind an axis-1/2/5 finding was source-only, based on NO_CANDIDATES, or simply copied forward from Stage A without live evidence, it cannot be critical – cap it at moderate. Findings on axis 3, axis 4, axis 6, or axis 7 can never reach critical on their own. Moderate: a real, well-reasoned defect that doesn’t meet the live-verified bar above – axis-2 WEAK/MODERATE backed only by source-level reasoning; axis-1 PARTIAL; axis-5 DEFECTIVE without live proof; axis-4 anti-patterns that materially reduce the test’s value (“assertion masking (coverage-real),” “over-broad mock,” “swallowed exception,” “tautology,” “skip/xfail masking”); axis-7 BUNDLED where the bundling genuinely obscures which claim failed. Minor: axis-4 findings where a real assertion still exists elsewhere and the practical cost is low (“dead lever,” “assertion masking (diagnostic-only),” “log-instead-of-assert”); axis-5 DEFECTIVE for excess/bloat only, not wrong-scenario; axis-6 PARTIAL or WRONG. Informational: axis-3 JUSTIFIED-MECHANISM (by the axis’s own definition, this coupling is appropriate – it flags mechanically but isn’t a real defect); axis-6 CLEAR_COMMIT_TEXT when it is the row’s only flagged reason; any finding whose own note describes the deviation as borderline or trivial. The CLEAR_COMMIT_TEXT rule (this overrides anything above that would imply otherwise): axis6_refined = CLEAR_COMMIT_TEXT never elevates severity – a commit message is a real, legitimate purpose source, just weaker than an in-file one, and is certainly not a major flaw. If it is the row’s only flag reason, the severity is informational (minor at most). If it co-occurs with another flagged axis, judge severity from that other axis alone, as if axis 6 were ideal – never let CLEAR_COMMIT_TEXT push a row up a tier. What to do For each file: identify which axis or axes are non-ideal (that’s why flag = yes), then apply the tier rules above using only what the row’s own notes already say about verification depth. For every row, report: the test’s identity (copied verbatim from its test_id column); which axis or axes drove the flag (e.g. “axis2 = MODERATE”); the severity tier; whether the proof behind it was live-verified, source-only, or not applicable; and one to three sentences of justification that cite the specific evidence already present in the row’s notes – don’t derive new evidence, just cite what’s there. Figure 5. Instructions for the Step 1 (proposal) severity scoring agent. You are one of three independent panelists re-deriving whether a specific test deserves a critical severity tier for its test-quality defect. You are blind to any prior proposal or reasoning about this row – form your own judgment from scratch by reading the actual test and the production code it exercises yourself. Row identity you’ve been given: Test identity: [test_id] File: [file path] Commit: [commit hash] Axis or axes flagged by the original scoring (this tells you what to look at, but is not itself a severity claim): [flagged axes] Reading the code: read the test’s source and the production code it exercises, at this exact commit, read-only, from this corpus’s own checkout. Verifying live, if you need to: if you want to check a claim empirically, create your own dedicated, throwaway worktree at this row’s own commit. Run the real test against your worktree using this corpus’s own established test-running procedure (its particular Python version, environment variables, and any test-runner wrapper it requires – follow this corpus’s own setup exactly, not a generic pytest invocation, since some corpora need a specific harness to get a correct result). The bar for critical – narrow, and only these three patterns qualify: • The test’s assertion checks something unrelated to its stated claim (a mismatch), and you can show this concretely – either by running it live, or by tracing precisely what the assertion can and cannot distinguish. • The test’s setup constructs a fundamentally different scenario than the one it claims to exercise (not just excess setup), and a live check proves it – for example, patching the real logic the test claims to cover doesn’t make the test fail, because its setup never actually built that scenario in the first place. • You can hand-patch the specific logic the assertion is supposed to be checking, rerun the real test, and it still passes – meaning you personally reproduce a live failure-to-catch, not cite someone else’s claim of one. Your vote: vote critical only if you personally verified one of these three patterns, live, yourself, right now – not because the row’s axis verdicts merely look concerning on paper. If you cannot reproduce a live failure-to-catch within reasonable effort, vote not critical. Report your verdict together with a one-paragraph justification explaining exactly what you did and what you found. Figure 6. Instructions for the Step 2 (three panelists) severity score review agents. Table 17. Changes in qualitative rubric scoring from Stage A to Stage B. Severity Total tests Unchanged (A=B) Changed (≥1≥ 1 axis downgraded) Extractor critical 6 4 2 moderate 58 52 6 Django critical 5 3 2 moderate 44 37 7 Table 18. Breakdown by rubric axis of changes in scoring from Stage A to Stage B. Severity Axis changed # tests Direction Extractor critical axis2 (assertion strength) 2 STRICT -> WEAK / MODERATE moderate axis2 (assertion strength) 5 STRICT -> MODERATE (4×) / WEAK (1×) moderate axis5 (scenario realism) 1 OK -> DEFECTIVE Django critical axis5 (scenario realism) 2 OK -> DEFECTIVE moderate axis2 (assertion strength) 6 STRICT -> MODERATE moderate axis5 (scenario realism) 1 OK -> DEFECTIVE 10.3. Historic-revert vs. qualitative-rubric detail tables 10.3.1. Extractor See Tables 19 and 20. Table 19. Extractor tests flagged assertion-strength WEAK by the qualitative rubric, with historical-revert outcomes. Historic-revert: S = SURVIVED, E = EFFECTIVE, E/New = EFFECTIVE (TESTING_NEW_FUNCTION). Test (commit) Historic-revert Severity Commentary Weak-value/loose-bound oracle test_flags_struct_behind_name_still_detected S moderate There is one assertion, and the pre-commit (loose) and post-commit casing rules both return True for the input used by the test. test_forward_branch_handler_dominance E/New critical Both test assertions are structurally non-discriminating. test_by_file_breaks_the_package_merge_cycle E moderate Test assertions only check import-set membership on two files and never check for all 3 expected filenames. EFFECTIVE only because reverting the commit is a mutation that these assertions happen to catch. test_qualified_keys_avoid_short_name_collision E moderate Reverting the commit re-merges three types into one, which the test’s count check catches, hence EFFECTIVE. But the test never runs the other half of the fix, the code that fills in field data without guessing between same-named types; breaking that half goes unnoticed. Wrong entry point / bypassed wrapper test_suffix_must_be_a_real_suffix E/New critical Test only checks fields with no empty-string key in _ENC, so an earlier dict-miss fallback already returns False before the new length guard is ever reached. test_word_substitution E/New moderate The decoder has two branches (lowercase continuation, uppercase terminator); the test routes only through the uppercase branch, the lowercase route is never entered. test_lendelim_tag_wire_type_2_is_plain S moderate The production code falls back to a “plain” default whenever none of its more specific detection rules match. The test’s input is a length-delimited field, a case none of those specific rules are meant to fire on anyway, so it gets the “plain” default regardless of whether the surrounding detection logic is present, working, or missing entirely. Table 20. Extractor tests flagged non-EFFECTIVE by historical-revert, with qualitative rubric scores. Historic-revert: S = SURVIVED, U = UNREACHED. Test (commit) Historic-revert Severity Commentary Weak-value/loose-bound oracle test_flags_struct_behind_name_still_detected S moderate See Table 19. test_varint_tag_shift_is_plain S – Test claims to check one specific safeguard, but its input also differs from the bad case in a second, unrelated way. That second difference is enough to make the test pass, whether or not the safeguard works. test_index_scale_lsl3_on_nonfield_reg_does_not_block_fixed S moderate The test’s one assertion (a decoy index-scale lsl doesn’t block fixed32 classification) passes equally whether the terminator’s register-matching is a genuine, selective gate or simply absent. test_lendelim_tag_wire_type_2_is_plain S moderate See Table 19. No defect test_xasstring_routed_field_not_rebiased S – Test checks that one kind of field is left unchanged by a new feature. The feature is written to skip fields of that kind. Reverting the commit removes the feature, skip-check included, so this field is unchanged either way. The test is well written yet SURVIVED is also a correct verdict. test_no_package_note_when_no_package U – The test is well written and checks what it claims. The commit adds a check which is not exercised by the test so SURVIVED is also a correct verdict. test_no_recon_caveat_when_no_enums S – Structurally the same as the row above. test_explicit_prefixes_override_default S – A value could be passed as an argument or set globally. The commit deletes the global path; this test only exercises the argument path. The test is well written yet SURVIVED is also a correct verdict. 10.3.2. Django See Tables 21 and 22. Table 21. Django tests flagged assertion-strength WEAK by the qualitative rubric, with historical-revert outcomes. Historic-revert: S = SURVIVED, E = EFFECTIVE. Test (commit) Historic-revert Severity Commentary Weak-value/loose-bound oracle test_change_password E critical The test only checks assertIsInstance(user, User) after a password change, but it never actually changes the password, so the session hash it is meant to validate never diverges regardless of whether the fix’s await request.auser() is present or reverted to sync request.user. EFFECTIVE here is coincidental: reverting the whole commit instead removes the auser() call entirely, crashing with AttributeError before the weak assertion (or the narrow oracle gap) is ever reached. test_aggregate_subquery_annotation S critical assertLessEqual(sql.count('SELECT'), 4) accepts both the buggy count (4) and the fixed count (3), so it cannot catch the bug it was written for. test_remove_unique_together_on_pk_field S critical Fully reverting the commit’s fix, removing the exact behaviour the test is meant to guard, still leaves the test passing. test_rename_model_with_db_table_rename_m2m E moderate The test only checks that the rename completes without raising. A concrete, equally simple check is available and unused: asserting the actual row count on the renamed relation, which would additionally catch corruption that doesn’t raise. test_namespaced_db_table_foreign_key_reference E moderate The test has no explicit assertions, it passes as long as the model/field creation calls complete without raising. test_alter_field_with_custom_db_type E moderate The test has no explicit assertions, it passes as long as the field alteration completes without raising. It never checks that the resulting column actually reflects the new definition. test_server_formatter_default_format E moderate One of the test’s two assertions checks the exact formatted log line; the other, covering the date format this commit actually changes, only checks a broad regex that would match almost any bracketed date-like string, not the specific format the commit introduces. test_rename_m2m_field_with_2_references E moderate The test does correctly catch corruption on the rename’s crash-path, but that’s not what its gap is about: after the rename, it never checks that foreign-key targets and data actually survived, only that the rename itself didn’t crash. test_parameter_count_exceeds_variable_or_column_limit S moderate The only check is implicit, the test passes as long as the call doesn’t raise, and the return value itself is never inspected. Table 22. Django tests flagged non-EFFECTIVE by historical-revert, with qualitative rubric scores. Historic-revert: S = SURVIVED, U = UNREACHABLE. Test (commit) Historic-revert Severity Commentary Dead assertion order (stacked context managers) test_json_response_raises_type_error_with_safe_arg S critical The test wraps two checks in one with (assertRaisesMessage(...), assertWarnsMessage(...)): block. Python exits stacked context managers in reverse order, so the warning check (entered second) exits first, sees the exception still active, and skips its own check; the warning half can never fail. The other half also cannot distinguish old from new code: it passes safe=True explicitly, and both versions raise the same error for that regardless of the fix. Weak-value/loose-bound oracle test_parameter_count_exceeds_variable_or_column_limit S moderate See Table 21. test_aggregate_subquery_annotation S critical Same test as the WEAK-axis table above. test_remove_unique_together_on_pk_field S critical Same test as the WEAK-axis table above. test_db_default_equivalent_sql_noop S critical db_default is not a non_db_attr, so the new guard’s condition genuinely evaluates and its pop() calls genuinely execute here. But Field.__init__ already normalises a bare string db_default="foo" into Value("foo") before deconstruct() ever runs, so old_kwargs["db_default"] and new_kwargs["db_default"] are already equal Python objects whether or not this commit’s pop() calls fire; popping two already-equal values changes nothing. assertNumQueries(0) is precise, but this fixture cannot tell the new code’s effect apart from its absence – live-verified by hand-patching the fix’s comparison and rerunning: still zero queries. No defect test_check_constraints S – The fix changes whether Django loops over all configured databases or just the ones under test when checking constraints; this only changes the warning count on backends that lack table-check-constraint support. When run with SQLite, which supports them, the expected result is identical either way. A backend-coverage gap in the test environment, not a flaw in the test. test_nested_key_transform_on_subquery U – The commit’s one changed line sits on a code path this test’s query never reaches, so the test can’t see it either way. test_check_database_version_supported U – The test’s own code path never reaches the line this commit changes, so it can’t see the difference either way. test_composite_primary_key_not_unique_together U – The commit’s one changed line sits on a code path this test’s query never reaches, so the test can’t see it either way. test_in S – The commit’s change lives entirely in Oracle-specific code for translating JSON lookups; this test runs against SQLite, which never calls that code at all, so nothing the commit touches is exercised. test_order_by_case_when_constant_value S – The commit reorganizes how the SQL compiler builds the fallback value for an ordering expression, but computes exactly the same value both before and after – it’s an internal rewrite, not a behaviour change. The test correctly checks that value, so it passes either way. test_context_copyable S – The commit swaps a manual rebuild of the copied object’s class and attributes for a cleaner, equivalent call to the parent class’s own copy method. Both produce the same copy on the Python version this test runs under, so the test can’t see any difference. test_union_all_none_slice S – The commit changes how an empty half of a union query is built internally, but an empty half contributes no rows to the result either way. The test checks the returned rows, not how the query was built, so it passes regardless. test_union_none_slice S – Same shape as the row above: the commit changes how an empty half of a union is built internally, but empty means zero rows either way. The test checks the returned rows, not the internal SQL, so it can’t tell the two versions apart. test_order_by_expression_ref S – The commit changes how order_by references are resolved, but only for a specific, ambiguous case this test doesn’t set up. For the plain case this test does check, the old and new logic produce the same ordering, so the test passes either way. 10.3.3. Pandas See Tables 23 and 24. Table 23. Pandas tests flagged assertion-strength WEAK by the qualitative rubric, with historical-revert outcome. Historic-revert: E = EFFECTIVE. Test (commit) Historic-revert Severity Commentary No assertion beyond non-crash test_pass_colormap_instance E moderate Calls df.plot.scatter/hexbin with a Colormap instance (rather than a string name) and checks nothing beyond the call not raising. A concrete stricter check exists and is mechanically straightforward: ax.collections[-1].get_cmap() is cmap on the returned Axes. test_interchange_from_corrected_buffer_dtypes E moderate Monkeypatches a string column’s reported buffer dtype to simulate a corrected interchange scenario, then calls pd.api.interchange.from_dataframe(df) without even assigning the return value: the only signal is that the call doesn’t raise. test_xz_compression_level_read E moderate The size-comparison assertion (compressed_size_default < compressed_size_fast) is well-calibrated, but the to_csv branch’s read-back call, pd.read_csv(path, compression="xz"), is never checked against the original data: unlike the file’s sibling round-trip test, which does assert equality. Table 24. Pandas tests flagged non-EFFECTIVE by historical-revert, with qualitative rubric scores. Historic-revert: S = SURVIVED, U = UNREACHED. Test (commit) Historic-revert Severity Commentary Wrong entry point / bypassed wrapper test_fillna_dict_inplace_nonunique_columns S moderate The test’s three duplicate "A" columns sit adjacent, so df.columns.get_loc("A") returns a plain contiguous slice. For a contiguous slice, the old single-write and the new per-location-loop write touch the same positions in the same order and produce identical output. The fix’s real target, non-contiguous duplicate columns, is never constructed here. test_astype_numpy_to_ea U informational For this test’s direction (numpy int64 -> Int64Dtype), the code finds a value before the buggy fallback line runs, so that line is never reached. The typo this commit fixes only manifests in the reverse direction, which this test does not exercise. test_dt_round_nonnano_higher_resolution_no_op S informational The fix is a one-line change inside the array-level _round(), and the test’s assert not np.shares_memory(...) looks like a precise oracle for exactly that line, but it observes memory identity only after the result passes back up through the .dt accessor’s Series-construction wrapper, one layer above the fixed line, which already produces a fresh array regardless of what the inner method returns. test_duration_fillna_numpy S – The test fills with a whole TimedeltaArray, not a scalar Timedelta. The commit’s refactor only changes behaviour for a scalar-Timedelta-with-unit input; the array-shaped branch this test exercises is identical in both old and new code. test_astype_dt64_to_int64 S – M8[ns] -> int64 is a lossless direct cast under both the removed and the new code path for this test’s dtype pair, including the non-NaT-fill-value case this commit adds. The original bug was that old code raised for this dtype combination, not that it computed a wrong value; this test’s pairing does not trigger the raise either way. Weak-value/loose-bound oracle test_private_values_dt64_multiblock S informational The fix’s whole point is that the DataFrame should still have 2 separate internal blocks after df._values is accessed. The test checks the block count only before the call, then checks only the returned values afterwards: old code’s silent re-consolidation does not change the decoded values, so nothing catches the regression the commit targets. test_stack_order_with_unsorted_levels_multi_row_2 S – Reproduced both code paths directly: for this test’s 2-distinct-value column level, the old and new internal code/level pairings decode to the exact same per-row values; only the internal array orientation differs, which the test’s frame-equality check never inspects. 10.4. Low-kill-rate triage detail tables Tables 25 and 26 give the Extractor tests found defective by low-kill-rate triage (§6.2), under the per-commit and coverage-guided mutation evaluations respectively; two of the five (marked †) independently confirm defects already found by the other evaluations rather than being wholly new. Table 25. Extractor tests found defective by per-commit low-kill-rate triage. Historic-revert: S = SURVIVED, E/New = EFFECTIVE (TESTING_NEW_FUNCTION). Test (commit) Historic-revert Kill rate A2 Strength Category Commentary test_varint_tag_shift_is_plain† S 0/14 STRICT Weak-value/loose-bound oracle The test’s input differs from the bad case in a second, unrelated way that alone makes it pass, masking whether the actual safeguard works. test_no_sibling_data_ivar_is_not_phantom E/New 0/4 STRICT Weak-value/loose-bound oracle The fixture’s dict lookup misses regardless of how the surrounding guards are perturbed, so the negative assertion cannot discriminate correct from broken. test_real_vmaddr_is_not_a_bind E/New 0/11 STRICT Weak-value/loose-bound oracle The fixture value sits far from the 1<<62 boundary the guard exists to test, so the guard’s own comparison is never exercised at a discriminating point. test_resolve_image_exact_full_path† E/New 0/8 MODERATE Wrong entry point / bypassed wrapper The single-image fixture only ever exercises the exact-match tier; it cannot tell whether the other two resolution tiers work at all. test_oracle_contradiction_rejected E/New 0/21 STRICT Wrong entry point / bypassed wrapper The code recovers a message’s field numbers, then double-checks them against a second, independent count of the same numbers, rejecting if they do not line up. The test’s example has the two counts share nothing in common at all, such an extreme mismatch that almost any check would catch it, so passing does not prove the actual comparison is correct. †The qualitative evaluation flags test_resolve_image_exact_full_path as the same defect, and as having critical severity (MODERATE assertion strength but axis1=PARTIAL and axis5=DEFECTIVE). test_varint_tag_shift_is_plain is flagged by the historic-revert evaluation. Table 26. Extractor tests found defective by coverage-guided low-kill-rate triage. Historic-revert: S = SURVIVED, E/New = EFFECTIVE (TESTING_NEW_FUNCTION). Test (commit) Historic-revert Kill rate A2 Strength Category Commentary test_varint_tag_shift_is_plain† S 0/38 STRICT Weak-value/loose-bound oracle Same defect as in table above test_no_sibling_data_ivar_is_not_phantom E/New 0/4 STRICT Weak-value/loose-bound oracle Same defect as in table above. test_real_vmaddr_is_not_a_bind E/New 0/3 STRICT Weak-value/loose-bound oracle Same defect as in table above. test_resolve_image_exact_full_path† E/New 12/40 MODERATE Wrong entry point / bypassed wrapper Same defect as in table above. 10.5. Low-kill-rate defect-mining agent instructions See Figure 7. Use the following Category vocabulary: • Weak-value/loose-bound oracle: the changed/relevant code genuinely executes, but the test’s assertion doesn’t distinguish old (buggy) from new (fixed) behavior for the specific input the test supplies. • Wrong entry point / bypassed wrapper: the test’s input/setup routes execution around the specific code path that changed, so the change is never actually exercised by this test (even though a superficially similar path is). • Dead assertion order: a compound with (A, B): block where Python’s exit order means one of the two checks can never actually fail. • Operator-scope gap / short-circuit-masked variant: the surviving mutant sits behind Python and/or short-circuiting, so the mutated sub-expression never actually executes even though coverage.py credits the line – or the mutant is on an AST node type outside the mutation-operator vocabulary used here (comparison/arithmetic/boolean only) to begin with. • Commit-scope contamination: the surviving mutant sits in a commit’s unrelated files, bundled into the same scored diff. • Import-time pseudo-coverage artifact. • No defect: the test is well-written and correctly checks what it claims; a low kill rate does NOT reflect any flaw in the test. A known pattern to watch for specifically: many low-kill-rate tests assert a NEGATIVE/boundary case (e.g. “returns empty”, “returns None”, “is a no-op”, “rejected”). These are frequently legitimate – a mutation applied to a different branch/case than the one the test’s negative-boundary input reaches will trivially survive without that being any flaw in the test. Verify this per-test by actually reading the diff and the mutated code locations, don’t assume either way. For every survived mutant, read the actual test body AND the actual production code, trace whether the mutated line is truly on the path the test’s assertions depend on, and where useful, apply the mutation directly (hand-patch a scratch copy, or use the project’s real mutation machinery) and rerun the test to settle ambiguous cases empirically rather than by inference alone. What to do for each test: 1. Read the test body at its introducing commit, and the production code it exercises. 2. Pull the exact list of mutants for it (operator, file, line, killed/survived) – don’t just trust the aggregate number. 3. For each SURVIVING mutant: explain the mechanism using the category vocabulary above, or propose a new category if genuinely none fits – name it precisely and explain why it doesn’t fit the existing ones. 4. Give a verdict per test: No defect (survivors are a methodology artifact) or Genuine defect (the test’s oracle really is weak/loose and would miss a real regression) – for genuine defects, be precise about what the exact gap is and what input would expose it. Be rigorous – every claim needs to be independently verified against the actual diff and test code, not guessed. Figure 7. Low-kill-rate defect-mining agent instructions (verbatim text given to the triage agents). Table 27. How Extractor test-writing episodes were prompted, by pattern. Pattern n % BUG_FIX_REGRESSION 45 37% OTHER 33 27% FEATURE_IMPLEMENTATION_UNPROMPTED 21 17% EXPLICIT_TEST_REQUEST 16 13% COVERAGE_REVIEW_REQUEST 6 5% 10.6. How was Extractor test writing prompted? Every Claude Code session transcript from the Extractor code development history was mined for episodes where a test file was created or modified. The set of episodes was then analysed to extract common patterns. The following patterns, summarised in Table 27, are observed: BUG_FIX_REGRESSION. A bug was found and fixed, and a regression test was added to pin the fix without a separate request for tests. Bugs were originally surfaced by: (i) the human instruction names a TODO number or points at an existing notes file the coding agent itself maintains e.g. “go ahead with #44”, (i) a short, fresh question or observation from the human user not tied to anything pre-logged, e.g. “why does it stall/fail?”, (i) the agent found the bug itself mid-task. OTHER. Splits roughly evenly between: (i) a refactor’s own stated constraint (“don’t break functionality”) led the agent to write tests, and (i) tests were modified because a file moved, a function was deleted, or a public API changed shape. FEATURE_IMPLEMENTATION_UNPROMPTED. A new feature or function was requested with no mention of tests anywhere in the instruction, and the agent wrote tests while delivering the work. EXPLICIT_TEST_REQUEST. The user directly asked to write tests, add regression coverage, or verify something works e.g. “yes, go ahead with that plan. add tests to ensure good code coverage as you go along.”, “as usual add tests if needed to avoid regression later” COVERAGE_REVIEW_REQUEST. The user asked specifically to review test coverage and improve it, as distinct from asking for a test alongside one feature e.g. “check the test coverage of the new code, add extra tests if needed.” Only 22% of episodes (EXPLICIT_TEST_REQUEST + COVERAGE_REVIEW_REQUEST) trace to a direct ask for tests; the remaining 78% were the agent’s own initiative.