Paper deep dive
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
Douglas Leith
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/26/2026, 2:15:18 AM
Summary
This paper introduces the 'Extractor' dataset, a comprehensive record of a 21,000-line Python tool built entirely by Claude AI without human authorship. The study utilizes custom provenance tracing tools and three new taxonomies (instruction intent, commit provenance, response reliability) to analyze the development process. Key findings include that CLI instructions differ significantly from IDE-chat interactions, code development is predominantly proactive, 14.3% of AI-generated code contained errors caught by tests before committing, and approximately 20-25% of interactive responses contained factual errors.
Entities (9)
Relation Signals (7)
Extractor â generatedby â Claude AI
confidence 99% · built entirely by Claude AI, with no human-authored code or tests
Extractor â contains â Instruction Intent Taxonomy
confidence 95% · application of these [taxonomies] to analyse the dataset... For the behavioural intent behind each user instruction, we extend Tang et al.âs IDE-chat taxonomy
Extractor â contains â Commit Provenance Taxonomy
confidence 95% · For commit provenance, we extend the standard Perfective/Corrective/Adaptive maintenance taxonomy
Extractor â contains â Response Reliability Taxonomy
confidence 95% · For the reliability of the AIâs own claims to the user, we develop a taxonomy with no direct precedent
Claude AI â produces â Error Rate
confidence 94% · 14.3% of Claude AI-generated code contains errors that are caught by the AI-authored test suite
Extractor â demonstrates â Proactive Development
confidence 93% · code development is mainly proactive rather than reactive, with 86-90% of commits... tracing to a proactive trigger
Instruction Intent Taxonomy â extends â Tang et al.
confidence 92% · extend Tang et al.âs IDE-chat taxonomy [3] to a multi-turn CLI setting
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.
Tags
Links
- Source: https://arxiv.org/abs/2609.29744v1
- Canonical: https://arxiv.org/abs/2609.29744v1
Trouble viewing inline? Open PDF directly â
Full Text
71,598 characters extracted from source content.
Expand or collapse full text
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase D. J. Leith Trinity College Dublin Dublin, Ireland AbstractâWe present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (i) two code-provenance tracing tools, (i) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (i) code development is mainly proactive, (i) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AIâs interactive responses contains one or more factual errors. I. INTRODUCTION AI coding agents write code quickly, but make mistakes often enough that their output cannot be trusted at face value [1], [2]. Judging how reliable AI-driven software development actually is requires visibility into the development process itself e.g. how often the AIâs own code and claims turn out to be wrong, and what catches those errors before they ship. To gain that visibility we need full session transcripts and code development artefacts (e.g. AI generated to-do lists) not just commit history. Unfortunately, existing large-scale datasets of AI-authored code consist primarily of commits/PRs with, at best, thin session coverage. Additionally, attributing a piece of code to its origin within an interactive session, recovering AI coding errors that were caught and fixed mid- session, and separating the AIâs reliable claims from its unreliable ones each need tooling that does not yet exist. The main contributions of this paper address these gaps: (i) the Extractor dataset; (i) two reusable tools that trace AI-authored Python code back to its provenance; (i) three taxonomies, each extending or filling a gap in an existing classification scheme, for instruction intent, commit prove- nance, and response reliability; and (iv) application of these to Extractor, to gain insight into the AI code development process. The dataset contains full AI session transcripts, the complete commit history and AI-authored process artefacts (a running to-do list, a heuristics registry, its own limitations note) generated during the development of Extractor. Extractor is 21,000 lines of production code with a comparably sized test suite, built over 210 commits and 25 sessions from 678 user instructions. It was entirely built using Claude AI models over a period of several weeks. No code, test, or commit in its history has human authorship. The Extractor dataset provides a level of completeness no existing large-scale dataset offers (§I). Email: doug.leith@tcd.ie The second contribution comprises two tools that trace AI- authored code back to its provenance. The first is a control- flow-graph block decomposition combined with git log -L history tracing, bridging renames. This traces individual code changes back to a specific user instruction, discovery, or self-correction event. The second is a session-transcript mining technique that recovers self-corrected code-generation errors from tool-use logs. These errors leave no trace in the commit history. The third contribution is a set of three taxonomies. Each extends or fills a gap in an existing classification scheme, and each is of independent interest beyond the specific anal- ysis it supports here. For the behavioural intent behind each user instruction, we extend Tang et al.âs IDE-chat taxonomy [3] to a multi-turn CLI setting. Their taxonomy explicitly excludes CLI-based coding agents. We add two categories that single-session chat has no need for: cross-session coordi- nation, and correction of the AIâs own conduct rather than its code. For commit provenance, we extend the standard Perfective/Corrective/Adaptive maintenance taxonomy [4], [5] to separate why a change was made from how it was worked out. For the reliability of the AIâs own claims to the user, we develop a taxonomy with no direct precedent: to the best of our knowledge, no prior work fact-checks an AI coding agentâs multi-turn conversational claims against the software artefacts they describe. We apply these tools and methods to the Extractor dataset. We find that user CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation. Code development is mainly proactive rather than reactive, with 86-90% of commits and 89-90% of code blocks tracing to a proactive trigger rather than being a reaction to a bug or test failure. We find that 14.3% of Claude AI- generated code contains errors that are caught by the AI- authored test suite before any commit (information that is un- observable by standard commit-based analysis). In interactive coding agent sessions we find that the Claude AIâs responses are relatively reliable when reporting or verifying a fact (94.3% accurate), less reliable when explaining an existing mechanism (91.1%) or diagnosing a root cause (89.3%), and substantially worse when proposing a design fix (79.4%). Overall, we find that roughly 1 in 4-5 of AI interactive responses contain at least one inaccuracy. As far as we are aware, these are the first reported error rates for user-facing responses for a modern coding agent. I. RELATED WORK Datasets. The closest existing dataset is probably SWE- chat [6], linking session transcripts to commit-level human/AI arXiv:2609.29744v1 [cs.SE] 24 Sep 2026 0204060 AI commit share (%) 0 50 100 150 # repositories 163 9 10 4 5 2 5 22 111 10 0 10 1 10 2 10 3 AI commits (log scale) 0 2 4 6 8 10 # repositories Fig. 1: Lifetime AI-authored commit share (a) and absolute count (b) per repository across all 205 repositories in the SWE- chat dataset (on 21st Aug 2026); the 117 repositories with zero AI-authored commits are omitted from (b)âs log-scale axis. attribution across 205 real repositories, but session coverage per repository is thin (median 8 sessions, 17.6% of repositories have only one logged session) and lifetime AI-authored com- mit share is negligible for most (median 0%, maximum 55.6%, 57.1% have no AI-authored commits at all; see Figure 1). AIDev [7] is a larger PR/commit-level dataset (932,791 agent- authored pull requests across 116,211 repositories, five agents including Claude Code), but captures only PR and commit metadata, not session transcripts, and its cutoff (1 August 2025) predates recent improvements in AI code-writing 1 . Tools. Provenance attribution. SZZ, introduced by [8], traces a bug back to its introducing commit via git blame on the fixed lines. It is known to misattribute tangled or reformatting-only commits as the true origin. Recent re- finements add outside evidence to resolve this using bug- discussion text to disambiguate the relevant file [9], or an LLM to judge the correct candidate commit [10] . This paperâs own CFG-block decomposition plus git log -L tracing (§IV-C1) targets a different, broader problem at finer granularity: every code blockâs full origin and churn history, not one bug-fixâs introducing commit. Its own residual gap is different in kind: a rename-detection blind spot on partial-file moves, not commit misattribution, closed mechanically rather than with an LLM, to keep the toolâs output reproducible. Trajectory mining. In [11] and [12], the single step that causes a benchmark-task trajectory to fail overall is localised (on Terminal-Bench and SWE-bench-family tasks respec- tively). In [13] process differences between resolved and failed runs are profiled using fully deterministic anti-pattern detectors, deliberately excluding an LLM judge from its main claims. All three treat failure as the run never reaching a working solution at all, not a single test failing partway through and later being fixed, the gap our session-transcript miner is built to close. Taxonomies. Instruction intent. In [14] 11 developer-AI interaction types (auto-complete, command-driven, conversa- tional, event-triggered, etc.) are proposed, but this is a purely conceptual paper with no underlying usage data. Recently, [3] instead derive an empirical, 7-category/20-subcategory intent taxonomy for IDE-integrated chat. Their dataset excludes CLI- based agents, âprimarily Claude Code,â explicitly leaving that 1 In November 2025 Claude Opus 4.5 was the first model to break 80% on SWE-bench Verified. TABLE I: Extractor summary. Size (src/test LOC)TestsCommitsSessions User instructions 21,000 / 21,8291,46721025678 interaction paradigm, this paperâs own subject, to future work. Commit provenance. Existing commit-taxonomy work clas- sifies why a change was made after the fact, from the message or an issue thread [4], [15], [16], and rationale-mining work does the same from Jira discussion or the message itself [17], [18]. We are not aware of prior work that classifies a commitâs trigger from the agentâs own reasoning and tool use as it happened, rather than reconstructing it afterwards. Response reliability. Prior fact-checking work mostly fo- cusses on journalism, social-media and LLM long-form text fact-checking [19] and verification is via internet search. Our own decompose-classify-verify pipeline follows a similar high-level architecture to ClaimBusterâs four-stage pipeline [20] and SAFEâs fact-decomposition-and-verification approach [21], but checking factual claims about software is rare in the literature and as far as we know, there is no prior work on fact-checking AI coding-session multi-turn conversational transcripts. AI analysis. Code-generation error rate. Prior measure- ments of how often AI-generated code is wrong come from two sources, neither of which can see an error caught and fixed before it is ever committed. Benchmark studies report pass/fail rates against curated problems, not real development e.g. [1], [2]. Repository-level studies instead compare finished, already- committed code: [22] classifies defects and vulnerabilities in shipped AI-authored code using rule-based pattern matching, primarily on the older, ChatGPT-3.5-era HMCorp dataset; [23] and [24] compare committed AI- and human-authored diffs by code metrics and CodeQL-detected vulnerabilities, and by technical debt, respectively; [25] and [26] analyse GitHub review and maintenance patterns from PR metadata. Code provenance. The same repository-level studies also compare AI-authored code against a human-authored baseline. Extractor has none, the whole codebase is AI-authored, so this paper instead traces each code block back to the commit and trigger that produced it (§IV-B), asking how the code came to be written, not how it compares to a human reference. I. EXTRACTOR Extractor is a Claude Sonnet AI-written tool (no human- written code), written from scratch over several weeks of work, see Table I. It is a real tool (for reverse engineering protobuf descriptors from compiled Apple Objective-C/C binaries), not synthetic test or âtoyâ code. Purpose. Many of Appleâs system daemons and frame- works on iOS communicate, and persist data on disk, using Protocol Buffers [27]. Almost none of the .proto schema files that describe those messages are shipped anywhere, public or private, on the device. To read what a given binary actually sends, receives, or stores, that schema has to be recovered directly from the compiled code itself. Extractor (apple_protobuf_extractor) is a tool that automates that recovery: given an Objective-C Mach-O binary or dyld shared cache, extract a complete, correct .proto descriptor for every protobuf message class it defines. Main components. Two things make this challenging. First, almost all of the protobuf traffic on the platform is serialised by PBCodable, a private Apple Objective-C protobuf imple- mentation. Behaviour has to be inferred from how the com- piled code actually behaves. Second, Objective-Câs message- dispatch machinery and Appleâs stripped, private serialisation internals leave a decompiler with almost nothing useful to render, so the schema has to be recovered from raw ARM64 disassembly instead. That is slow and unpleasant work when done by hand: it means tracing individual instructions, one call site and one basic block at a time, across potentially thousands of message classes in a single framework. It is this that motivated the development of the Extractor tool. Dataset.TheExtractordatasetisavailableat https://github.com/doug-leith/extractor-code-development. This contains commits, session history transcripts, AI- generated note files, the taxonomy codebooks (instruction, commit, and response) and scoring rubric, raw LLM categorisation and verdict results, and scripts to regenerate the tables and figures here. IV. ANALYSIS OF CODE DEVELOPMENT PROCESS A. Types of instruction given In this section we characterise how the user directed and collaborated with the AI over Extractorâs development. We classify each of the 678 user instructions 2 given to Claude into behavioural-intent categories, following the taxonomy developed by Tang et al. [3] from single-session IDE chat transcripts. Their scheme captures what kind of thing an instruction is asking for, e.g. new code, a bug report, a request to review output, independently of the specific project. We use it to characterise the division of labour between user and AI, and to test how far a taxonomy built from short IDE chat exchanges carries over to a CLI-based coding agent used across long, multi-session engagements. 1) Decomposing instructions In the Extractor corpus user instructions can be compound instructions, involving multiple steps. This is illustrated, for example, in Figure 2. As Figure 2 shows, the steps can also involve instructions with different intents e.g. runtime moni- toring (âwhen tests passâ), toolchain use (âcommit changesâ), documentation (âupdate to-do listâ). Similarly to Wei et al. [21], we therefore use an LLM to decompose instructions into their component parts before categorising their intent 3 . Figure 2 shows that almost half of instructions are compound instructions, they are not rare. 2 After data cleaning to exclude: 274 background task notifications/wakeups injected as user turns, 19 /compact commands. 3 We found that decomposing instructions into components significantly improves classification consistency. For example, we drew a random sample of 50 instructions that decompose into more than one component, yielding 50 full instructions and 128 components. Using the final codebook, see §IV-A3, we used Codex and Claude to classify the instructions in this sample with/without decomposition, allowing multiple categories to be assigned to each instruction or component instruction. TheÎș was 0.700 with full instructions vs 0.790 for decomposed instructions, with exact match rates of 52% and 76.6% respectively. #ComponentLabel 1when tests pass, (i) com- mit changes 5.2;6.3 2(i) update to-do list to mark #54 and #58 done and remove them from the active list. 6.1 3(i) consider a separate question: proto/extra ctor.py is 1502 lines long, that seems pretty big. do you recommend split- ting it or not? 3.1 4(iv) also check for other large py file. 3.2 (a) Instruction idx289 123456789 components per instruction 0 20 40 60 % of instructions 363 190 69 33 13 4 222 (b) Components per instruction Fig. 2: Instruction decomposition into components. (a) An in- struction decomposed into its components, each independently classified against the taxonomy codebook. (b) Components per instruction, nearly half (46%) bundle two or more. 2) Codebook development Tang et al use an iterative abductive coding method to develop their taxonomy codebook. In this approach a sample of instructions is classified against the current codebook using an LLM, a human reviewer inspects each assignment, and where the fit is poor a new category is proposed. Tang et al. converged over four rounds (9, 5, 1, 0 new or revised subcategories per round). We follow a similar abductive coding approach, modified as follows. We draw the first sample of instructions using stratified sampling rather than uniformly at random 4 . Following Tang et al we categorise these instructions using an LLM instructed using: (i) Tang et alâs codebook plus a temporary Others subcategory for instructions that did not fit an existing definition, and (i) the instruction together with its immediate context (the preceding instruction and the AIâs response). A human reviewer inspects each assignment, where the fit is poor a new category is proposed using corpus evidence and the codebookâs definitions and boundary notes, and this is repeated until no new categories are proposed. In the first round, three additional categories were judged necessary, in the second round one more category was added, in the third round no new categories were proposed. This abductive process yielded four additional categories: one under Context Specification (4.3 Design Principle Specification), one under Delegation (6.3 Version Control), and two un- der Workflow Control (7.6 Compliance Correction, 7.7 Ses- sion/Environment Coordination). Tang et al explicitly exclude CLI agents like Claude Code from their data, and two of the new categories capture differences between CLI agents and the single-session IDE chats studied by Tang et al. Claude Code sessions can run in parallel on the same project, unlike single-session IDE chats, and as a result there is a new category of user instructions handling cross-session co-ordination (7.7 Session/Environment Coordination). A second new category 7.6 Compliance Cor- rection captures corrections to what Claude itself just did or 4 To stratify, we roughly clustered the instructions using an LLM (âwhat themes do you see in the user instructionsâ) and sampled two instructions from each of the resulting eight clusters. The aim of the stratification is to allow us to quickly sample a wide diversity of instructions, so rough clustering is sufficient. TABLE I: Instruction categories identified in Extractor corpus (full definitions in the codebook, dataset package). CategoryExamples 1.1 New Implementationâlook into adding a check for use of google protobufs e.g. for GPBMessage, so we can give the tool user an informative messageâ; âthen look into improving the test coverage so we can properly test later changes for regressionsâ 1.2 Iterative Modificationâthen swap into productionâ; âupdate tool to reflect this changeâ 1.3 Alignment Correctionâthe reference protos are not ground truth, theyâre known to contain errorsâ 2.2 Symptom Descriptionâfix heuristic freq.pyâ; âwith locationd the regex gate fails with garbage.â 3.1 Planning & Consultationâplan out what step 1 would involveâ; âsuggest develop a proof of concept first that can be testedâ 3.2 Project Comprehensionâno field types with swift? is that unavoidable or a tool limitation?â; âwe already talked at length about enum-name-prefix and decided the enum type name was cosmetic, look back to see if you can find that conversation and confirmâ 3.3 General Knowledge Queryâis that open source?â; âwhile we wait, does the c++ libprotobuf-lite allow recovery of field names, numbers and types?â 4.1 Information Injectionâpreviously you advised waiting until after readFrom was updated, but Iâve decide to defer the readFrom updateâ 4.2 Behavior Specificationâwithout modifying production codeâ; âignore the test lines of codeâ 4.3 Design Principle Specification âit should not depend on hardcoded memory addresses or the like - it needs to be portable across different binaries.â; âit should have good test coverage.â 5.1 Code Reviewâcheck that code has appropriate comments linking to each of the entries in the limitations docâ; âthen sample some of the repeated enums that changed and analyse the binary to confirm the changes are correctâ 5.2 Runtime Inspectionâdo this more complete test for the iphone8 data too.â; âwhen its all ok proceed to do step 3 to complete the migrationâ 6.1 Documentationâif not then update itâ; âadd a comment to the code re the architectural constraintâ 6.2 Toolchain Operationâgo ahead with (1)â; âmove the macho extractor apple folder to the project rootâ 6.3 Version Controlâthen commit to gitâ; âcommit to gitâ 7.1 Confirmationâok, understood re proto rulesâ; âok, got it re the enum values.â 7.2 Continuationâkeep goingâ; âcontinueâ 7.5 Sentiment Expressionâok i think i misunderstood.â; âi like the behaviour of this session: youâre careful and doggedâ 7.6 Compliance Correctionâyouâve dropped the links to to-dos, add those back in.â; âthe column ârowâ should still be called âfunctionâ surelyâ 7.7 Session/Environment Coordination âthat sessions commits are now done.â; âsave in a file so i can pick up in another sessionâ displayed, not of the code under development e.g. a failed or ignored instruction. The remaining two additions are not tied to how Claude Code works. 4.3 Design Principle Specification captures standing, codebase-wide quality statements; Tang et al.âs closest category is about interaction style, not code properties, so this appears to fill a gap in their scheme. Tang et alâs 6.2 Toolchain Operation encompasses git commits etc, but we split these out into 6.3 Version Control because git instructions are a frequently occurring category in their own right. See Table I for a summary of the resulting taxonomy. 3) Codebook refinement We refined the codebook to clarify boundary cases using an iterative process. We began by drawing 100 random com- ponent instructions and classifying them using two indepen- dent LLMs (Codex GPT-5.6 Sol and Claude Sonnet 5, from two competing vendors) instructed using the behaviour intent codebook. As Tang et al. note, and their own results support, there is good evidence that LLMs can achieve strong perfor- mance on text classification tasks when guided by structured instruction and schema constraints [3]. Gilardi et al. report ChatGPT outperforming crowd-worker annotators accuracy by around 25% on comparable text-annotation tasks [28], and Zheng et al. find a strong LLM judge can match human-human agreement levels on pairwise preference judgements, while also documenting potential biases (position, verbosity, self- enhancement) [29]. Disagreements in classifications between two independent LLMs therefore provides some insight into ambiguities in the boundaries between categories. For exam- ple, in âGo ahead with Phase Aâ the code for Phase A doesnât exist yet, which sounds like ânewâ work. But if the design was already fully worked out in an earlier conversation (a validated prototype, a detailed plan) and this instruction just says ânow actually build it,â is that creating something new (1.1 New Implementation), or finishing something already decided (1.2 Iterative Modification). A human researcher reviewed disagreements between the two LLMs, looking for patterns in the boundary cases. For commonly occurring patterns the codebook was updated with a clarifying reminder. This was repeated for samples of 200, 300 and 600 components with convergence achieved after three iterations. After this process, Cohenâs Îș = 0.808 between the LLMs (this Îș is for the 512 instructions remaining after excluding the 166 instructions viewed during the codebook development and the iterative refinement process i.e. it is for a held-out sample). This indicates that the LLMs are in good agreement and that the codebook establishes consistent boundaries between categories. It also shows that the results are not sensitive to the choice of LLM used to perform the classification. To validate classifier accuracy, we drew a stratified sample of 55 instruction components, 5 instruction components from each category containing at least 25 instructions. A human researcher independently annotated these 55 instructions using the finalised codebook, yielding Îș = 0.710 with Claude and Îș = 0.718 with Codex. That is, the human categorisation is consistent with both LLM categorisations (Îș in range 0.61â 0.80 indicates âsubstantial agreementâ, and is somewhat higher than Tang et alâs human-human Îș = 0.669). The full codebook, including definitions, key signals, exam- ples, and boundary notes, is provided in our dataset package. 4) Results Our taxonomy extends Tang et al.âs scheme, developed and validated on single-session IDE chat transcripts, to a CLI-based coding agent. Three new subcategories capture observed CLI-specific behaviour (cross-session coordination, standing design-principle statements, and correction of the AIâs own session conduct rather than its code), and boundary- note refinement sharpened category distinctions the original codebook left underspecified in the CLI context. Figure 3 summarises the distribution of instruction cate- gories observed in the Extractor corpus and also shows the 0 20 % 1.22.26.22.14.13.23.11.36.14.21.17.22.33.35.17.17.55.2 25 15 10 9 8 8 8 7 7 6 6 6 4 4 3 3 2 1 Tang et al. 0 10 20 % 3.23.11.26.16.35.14.16.25.24.27.71.12.27.17.23.31.37.52.3 18 16 14 14 9 8 7 7 5 3 2 2 1 1 1 1 0 0 0 Extractor Fig. 3: Behavioral-intent subcategory distribution: Tang et al.âs published data (left, IDE-chat messages) vs. this studyâs Ex- tractor corpus (right). Bars are coloured by top-level category, categories below 1% in both panels are omitted. corresponding distribution for Tang et alâs corpus. It can be seen that the distributions are not the same. In the Extractor cli- based coding agent corpus Project Comprehension, Planning & Consultation, Documentation and Runtime Inspection are much more prominent than in Tang et alâs chat IDE corpus. Symptom Description is much less frequent in the Extrac- tor corpus and Log Paste is absent. Iterative Modification, Information Injection and Toolchain Operation are similarly prominent in both corpora. These differences mean that not all of Tang et al findings carry over unchanged to the Extractor corpus. Finding 1 âConversational programming operates through progressive refinement rather than upfront specificationâ is similar in that a substantial fraction of instructions relate to Code Authoring but that category 1.2 Iterative Modification is much more prevalent than category 1.1 New Implementation (6.0Ă more here, vs. 4.2Ă more in Tang et al.). However, cat- egory 1.3 Alignment Correction is much rarer in the Extractor corpus than in Tang et al.âs corpus, perhaps reflecting recent improvements in coding agent instruction following. Finding 2 âDevelopers delegated error diagnosis to AI by re- porting symptoms and machine outputs rather than articulating code-level causesâ is not supported by the Extractor corpus. Failure Reporting, âthe second most common categoryâ in Tang et al.âs data, appears in only 1.77% of instructions here (vs. 24.0%), and Log Paste never occurs at all. Instead, in the Extractor corpus Validation is considerably higher (18.73% vs 3.99%), reflecting detection of errors by code review and runtime testing rather than by human inspection and input of failures. This appears to reflect a genuine difference in usage between IDE chat sessions and CLI-based coding agents. Finding 3 that developers âqueried AI about behavior, outputs, and feature logic rather than reading code directly,â is more true for Extractor than in Tang et al.âs data (Project Comprehension is 25.7% vs. 8.2%). Finding 4 âDevelopers used AI to evaluate code through both static review and dynamic runtime inspectionâ and Finding 5 âdocumentation to externalize plans, explanations, and progress into persistent ar- tifactsâ show a similar pattern i.e. the qualitative finding holds, but at two-to-five times Tang et al.âs rate (Documentation 2.6Ă, Validation 4.7Ă). Finding 6 that developers âactively managed AI autonomy by shaping its context and constraining or delegating its behaviorâ is similar with Information Injection (11.7% vs. 8.5%) and Behavior Specification (5.2% vs. 6.1%) both close to Tang et al.âs rates. B. Triggers for code development: Commit analysis This section classifies each commit in the Extractor corpus by its development trigger, e.g. whether it was prompted by a test failure or made proactively, using a taxonomy built from the AI coding agentâs own session transcript, not just the commit message. Existing commit-taxonomy work classifies why a change was made after the fact, from the message or an issue thread [4], [15], [16], and rationale-mining work does the same from Jira discussion or the message itself [17], [18]. We are not aware of prior work that classifies a commitâs trigger from the agentâs own reasoning and tool use as it happened, rather than reconstructing it afterwards. We enriched raw code commits in two ways: (i) by finding the command making the commit in the session history, the session context (user instructions, Claude responses and actions) that led to the commit becomes available, (i) often commit messages cite project documents, e.g. TODO.md, heuristics_registry.py, and these also provide ad- ditional context for the commit. Using this enriched context, we used a similar approach to §IV-A2â§IV-A3 to develop a taxonomy for the commits. We began by considering whether to decompose commits. Our analysis showed that 14 of the 210 commits merged multiple different changes. In view of this relatively small number, and the known difficulty of untangling merged commits [30], we chose not to decompose the commits. A commit taxonomy was then developed using an iterative abductive approach: select categories iteratively with convergence after three rounds, then refine boundary cases by examining the differences between the categorisations by two independent LLMs (Codex GPT- 5.6 Sol and Claude Sonnet 5). For the final codebook Cohenâs Îș = 0.786 between Claude and Codex. Table I summarises the final taxonomy. It is primarily structured around proactive and reactive code development. These high-level categories correspond to the Perfective and Corrective categories of Swanson (1976) [4], carried forward into the current maintenance-process standard [5]. Swanson also defines an Adaptive category capturing code changes in response to changes in the softwareâs environment e.g. new OS, new platform, new regulatory requirement, but this has no analogue in the Extractor corpus. Conversely, our Housekeep- ing category, which captures common maintenance activity with no code-behaviour content, has no Swanson analogue. The proactive and reactive categories are sub-divided by method, i.e. changes derived from (i) a specification/document, (i) analysis of artifacts lacking documentation, (i) a design decision, (iv) a code audit/review. In the Extractor corpus reactive code changes are made in response to one of four triggers: cross-decoder differential (two independent decoding paths disagree on the same input e.g. the toolâs own decoder against a reference implementation, or two of the toolâs own internal decode routes), systematic sweep (a broad scan across many binaries surfaces a pattern of anomalies not tied to any single known bug), external validator rejection (an external tool, e.g. the protobuf compiler, rejects the toolâs output as invalid), or unit/regression-test trigger (a test-suite run fails). Refactoring and addition of tests fall under Proactive-Design/Proactive-Audit. The full codebook, including signals, examples, and bound- ary notes, is provided in our dataset package. TABLE I: Definitions of the commit-type categories. CategoryDefinition Proactive-SpecContent directly translated from a documented format spec/ABI (no investigation needed to write it). Proactive- Analysis Content found by directly inspecting a raw artifact, because no documented spec covers it (e.g. disassembling compiled code). Proactive- Design Content is a general software-design/architecture decision. Proactive-AuditFound by systematically re-reviewing already-working code or tests for quality/robustness, via one of two sub-routes: general review (a self-review pass identifies a list of follow-up items) or necessity verification (a candidate mechanism is disabled/relaxed and the output re-diffed; zero diff proves it dead/redundant before retirement). Reactive-SpecFound reactively, the fix was determined from a documented format spec/ABI. Reactive- Analysis Found reactively, the fix required directly inspecting the raw artifact under study to determine. Reactive- Design Found reactively, the fix was a straightforward engineering change (no spec citation or artifact investigation needed to determine it). HousekeepingDocs, notes reorganisation, renames/archiving, build-metadata changes, a mechanical correction of a prior commitâs own incomplete git operation, a commit that only records an empirical finding without changing any code status/behaviour. No code-behaviour content. TABLE IV: Breakdown of the commit population by category and, separately, of CFG code blocks by the category of their originating commit. Category Commits (Codex) Commits (Claude) Code blocks (Codex) Code blocks (Claude) Proactive-Audit71 (33.8%) 76 (36.2%) 437 (23.6%) 673 (36.4%) Proactive-Design35 (16.7%) 31 (14.8%) 513 (27.8%) 476 (25.8%) Proactive-Analysis27 (12.9%) 31 (14.8%) 706 (38.2%) 496 (26.8%) Proactive-Spec0 (0.0%)0 (0.0%)6 (0.3%)6 (0.3%) Housekeeping56 (26.7%) 56 (26.7%) â Reactive-Analysis10 (4.8%)7 (3.3%)111 (6.0%)117 (6.3%) Reactive-Design10 (4.8%)8 (3.8%)52 (2.8%)57 (3.1%) Reactive-Spec1 (0.5%)1 (0.5%)23 (1.2%)23 (1.2%) Total21021018481848 Table IV summarises the breakdown of commits by category by two independent LLMs. It can be seen that two LLMs are in good agreements. They both categorise the bulk of the commits as proactive, with a majority of those arising from audits and a roughly equal split between analysis and design for the rest. The reactive commit share (10.4â13.6% across the two raters) is low by comparison with human baselines: below Lientz, Swanson and Tompkinsâ (1978) survey estimate of 17% [31], and toward the better end of the measured dis- tribution across 7,557 GitHub projects (median 20%, ranging from 6% to 39% across deciles) [32], [33]. C. Triggers for code development: Code block analysis Since the commits vary greatly in size, the commit-level breakdown in Table IV is rather coarse. We therefore also analyse code development using the codebase itself as the reference, giving the Code blocks/% columns of the same table. 750 def _extract_writeto_field_numbers( 751 ctx: BinaryContext, 752 cls: ObjcClass, 753 field_ivars: List[ObjcIvar], 754 _out_helpers: Optional[Dict[str, int]] = None, 755 _out_zero_fields: Optional[Set[str]] = None, 756 ) -> Dict[str, int]: 776 indeg = getattr(ctx, "_writeto_indeg", None) 777 foreign_imps = getattr(ctx, "_writeto_foreign_imps", None) 778 if indeg is None or foreign_imps is None: 779 return 780 try: 781 return _walk_field_numbers(ctx, cls, field_ivars, indeg, foreign_imps, 782 _out_helpers=_out_helpers, 783 _out_zero_fields=_out_zero_fields) 784 except (NameError, ImportError): 785 raise # our own wiring bug â never mask it 786 except Exception as exc: 788 log.debug("writeTo walk failed for %s: %s", getattr(cls, "name", "?"), exc) 789 return origin category (outline) Proactive-AnalysisProactive-Audit CFG block kind (fill) straight-line branch-test (if/elif) handler-arm (except) Fig. 4: Illustrating CFG code blocks and their provenance. PARTITION(function): open a new block for each statement in the function body, in order: if it is a branch point (if/elif, while, for): close the current block; open one block per branch arm; partition each armâs body independently else if it is a try/except: the try body is not itself a branch point; each except clause opens its own block else if it is a nested function or class definition: partition it separately, so its lines are never counted as part of the enclosing function else: extend the current block with this statement close the final block return the blocks: non-overlapping, covering every line exactly once Fig. 5: Pseudo-code for CFG basic-block partitioning of a function. 1) CFG code block decomposition and tracing We built a tool to attribute every code block in Extractor to its originating commit and track its subsequent churn, without manual tracing or AI judgement. It has two parts: cfg_builder.py, which partitions each functionâs abstract syntax tree (AST) into control-flow-graph (CFG) basic blocks, and git_tracer.py, which recovers each blockâs commit history via git log -L. This is a related but different problem from SZZ [8]: SZZ links one known bug-fixing commit back to the single commit that introduced the defect, at file/diff granularity via git blame; we instead attribute every code block, buggy or not, to its full origin and churn history at CFG-basic-block granularity, which no existing SZZ implementation does. Blockpartitioning.Figure5sketches cfg_builder.pyâs partitioning algorithm: it opens a new block at every branch point and every except handler, treats a nested function or class definition as a self-contained sub-problem so its lines are never attributed to the enclosing function, and otherwise extends the current block statement by statement. Provenancetracing.Figure6sketches git_tracer.pyâs algorithm for recovering a blockâs origin commit and churn history. The straightforward case asks git log -L directly for the blockâs line-range history; the fallback case, needed when that returns nothing, is discussed next. The technical challenge. git log -Lâs rename detection is based on whole-file similarity, so it misses a blockâs history TRACE_ORIGIN(block): ask git for the line-range history of the blockâs current location if git returns a history: origin = its earliest commit; churn = every later commit else: # the block may have arrived via a partial-file move that # gitâs own rename detection did not follow if the blockâs file was carved out of a larger file earlier: search that earlier commitâs tree for the blockâs own text, to relocate it before the move if the text is not unique: instead search for the enclosing function by name (a stronger anchor than a short line of code), then confirm the blockâs text really sits inside that functionâs body, and that the commit which appears to have carved it out actually touches both the old and new file, before trusting the match if relocated: repeat the whole trace from the earlier commit if still unresolved: report it as unresolved, not a guess Fig. 6: Pseudo-code for tracing a code blockâs origin commit and churn history. whenever only part of a large file was carved out into a new one, the dominant refactoring pattern in Extractorâs history. We bridge this in two stages: first by relocating the block via its own text in the pre-carve file, and, when that text is too short or repeated to be unique (e.g. a bare return (x, x)), by relocating it via its enclosing functionâs name instead. That second anchor is not always unique either: the search is a bare-name match against every function definition anywhere in the codebase at that point in history, so it fails, deliberately, whenever two different classes or functions happen to share a method name (writeTo and readFrom, for instance, are defined once per message class, so many classes share the same name). Rather than guess between candidates in that case, the tracer reports the block as unresolved. Even when the name is unique, a matching name and matching text alone are still not proof of common ancestry since an unrelated function elsewhere could coincidentally share both, so we additionally require the specific commit that appears to have carved the block out to genuinely touch both the old and new file before trusting the match. Across the full proto/+macho/ sweep, adding the function-anchor stage cut the unresolved rate from 11.4% to 0.4%; every remaining unresolved block is disclosed with the reason it could not be relocated. An LLM could plausibly relocate some of the remaining 1.9% that our mechanical bridge leaves unresolved. We used the mechanical approach instead: the residual miss rate is already small, so there is little left to gain. 2) Results Table IVâs Code blocks columns show the number of code blocks associated with each category of originating commit, separately for each rater. The large majority of code blocks originate proactively for both raters: 1662 (89.9%) for Codex and 1651 (89.3%) for Claude, with the remainder (186, 10.1%, and 197, 10.7%) reactive. Figure 7 shows the Codex proactive/reactive split broken down by code block size (the Claude breakdown is almost identical). It can be seen that the proactive/reactive split is similar regardless of block size (and that larger code blocks are relatively rare). Proactive-Design, Proactive-Analysis and Proactive-Audit together account for the large majority of code blocks (Codex: 1656 of 1848, 89.6%, Claude: 1645 of 1848, 89.0%): proactive 500 1000 1500 1345 146 Proactive Reactive 12345+ Block size (lines) 0 200 194 24 54 4 23 5 46 7 # blocks 500 1000 1500 1447 Untouched Reactive Proactive 01234+ Later commits 0 200 305 83 7 5 #CFG blocks Fig. 7: CFG code blocks broken down (a) by size and proac- tive/reactive provenance, and (b) by number of later commits touching them and why. extractor 6 writeto 111 enums 18 foreign 59 repeated 20 dispatch 5 emitter â write helpers 4 arm64cfg 6 protocore â models â protoclass heuristic â config â proto/ memory â classlist â disasm â arm64 decode â dyld_cache â local symbols â relsel base â cache classlist â strings â objc_abi â macho/ 01â56â1011â3031â7071+ fix-churn events: Fig. 8: Churn by Extractor module; hatched cells are modules with no churn. architecture/integration decisions and proactive investigation of the compiled artifact are both far more common, block- for-block, than any reactively-triggered category. Reactive- Analysis (a reactive trigger whose fix required directly inspect- ing the compiled artifact) is the largest reactive category. D. Code churn Since we can trace code blocks back through historical commits, we can analyse code churn. Figure 7 summarises the frequency of changes made to each code block, broken down by their reactive/proactive provenance (housekeeping does not affect code). It can be seen that the great majority of code blocks undergo no changes. For code blocks that undergo one or two changes, just over half (270 of 400) are proactive. Reactive changes tend to be concentrated as one-off fixes. Code blocks with more than two changes are rare Figure 8 maps code churn onto the modules within Extrac- tor. This shows that most of the churn (79.2%) is concentrated in writeto.py, foreign.py and repeated.py. These three modules do the primary reverse engineering work, and the churn here is consistent with the exploratory nature of that work already noted. Note that while this churn analysis gives insight into the code development process, it does not provide much insight into Claudeâs coding accuracy. This is because obvious coding errors would be caught by tests etc before being committed and so would not show up in the commit history. Instead, we use session history data to analyse Claudeâs coding accuracy below. V. RELIABILITY OF CLAUDE CODE GENERATION As already noted in Section IV-D, commit-based analysis misses code defects that Claude introduces but finds and fixes before the code is committed. We can, nevertheless, gain some insight into this by analysing the Claude session transcripts. These histories include details of tool use, including runs of pytest and the resulting output. A. Session-transcript mining Git history only records what was eventually committed; a bug Claude introduced, caught by a failing test, and fixed be- fore ever committing leaves no trace there. Recent trajectory- mining studies of coding agents donât address this gap: they study benchmark-task trajectories that end in outright task failure, the agent never solving the assigned issue at all [11], [12], not a projectâs history of individual test failures caught and fixed along the way. To measure it, we mine the raw Claude Code session transcripts directly. Unlike the commit log, these preserve every tool call: the exact pytest command run and its pass/fail output. For each sessionâs raw .jsonl, our custom Python tool finds every Bash call containing pytest, classifies it PASS/FAIL by regex against its -q summary line, and flags a FAIL -> (>=1 edit) -> later PASS on an overlapping test target sequence as a candi- date self-correction episode. A companion pass tags every pytest invocation PASS/FAIL to also give a âran cleanâ de- nominator. We found that raw session logs frequently contain duplicates since Claude re-emits earlier events each time a session is compacted, but these can be de-duplicated by using the event UUID. This part of the tool is purely mechanical Python and says nothing about why a pytest run failed. Inspection of the session transcripts reveals deliberate mu- tation testing i.e. mutate the code on purpose, confirm the test now fails, revert. This produces a pytest pattern identical to a genuine bug, i.e. FAIL, then an edit, then PASS. Telling the two apart requires reading the session narration; a mechanical regex pass cannot distinguish them, and neither can trajectory- diagnosis tools built to stay fully rule-based for auditability [13]. We therefore instructed an AI (Claude Sonnet 5) to read each episodeâs transcript and assign it to one of four categories: (i) a production-logic defect, (i) a test/fixture defect, (i) a failure to propagate a change to a dependent test, or (iv) a deliberate mutation-testing cycle. Every episode categorised as a defect was then checked by a human, which caught two mistakes. B. Results We analysed this data to identify cases where a failing run on a given test target is followed, later in the same session, by a passing test run on an overlapping target i.e. candidate instances of code being written, found wrong, and corrected before commit, yielding the results shown in Table VI. Table VI shows that 15 out of 105 code generation events (14.3%) introduced genuine coding errors, later corrected before any commit. Eight of these (7.6% of all code generation events) were errors in the tests and four (3.8%) were errors in the code itself. The remaining three (2.9%) were failures to update tests after a (correct) code change. Note that this analysis covers the 105 events for which a full session transcript is available and which fall within the paperâs audited population window, due to imperfect record keeping 5 not all code generation events could be recovered. Nevertheless, the data is enough to give a rough idea of the code generation error rate. In particular, that it is not small and 5 By default, session transcripts are deleted after 30 days and so some transcripts were lost before we discovered this. TABLE V: Definitions of the categories used in pytest analysis. CategoryDefinition Clean (no failure)A pytest invocation reporting zero failures. The code under test worked as checked, whether on the first attempt or a later confirmation run. Test/fixture defectThe check failed because the newly-written test itself (its assertion or its synthetic fixture data) was wrong. Production-logic defectThe check failed because the reverse-engineering/decoding logic under test was genuinely wrong. Failed to propagate change to existing tests The check failed because a correct change made elsewhere in the same session (a new method, a changed diagnostic message, a build-script fix) had not been carried through to a dependent mock, assertion, or script. TABLE VI: Breakdown of Claude code generation event correctness, based on analysis of session transcripts. CategoryRuns% Clean (no failure)9085.7% Test/fixture defect87.6% Production-logic defect43.8% Failed to propagate change to existing tests 32.9% Total105100% so a good code test-suite is important for protecting against regressions. To the best of our knowledge previous AI code quality analysis has all been based on commit history, and so is blind to bugs fixed before being committed. For the same reason, we are also not aware of a comparable human baseline since standard defect-density measurement (bug trackers, QA test- ing, release history) only observes code after it is committed. The human analogue would be errors caught and fixed within a developerâs own write-test-fix iterations before committing, a rate existing measurement cannot see for the same reason. VI. RELIABILITY OF CLAUDE USER INTERACTIONS We evaluate the reliability of Claudeâs responses when interacting directly with the user. To do this we build a fact checking pipeline. For each response: (i) decompose it into component statements, (i) decide whether each component contains a checkable fact or not, (i) if it does, gather evidence to inform verification, and (iv) evaluate response component accuracy. This decompose-classify-verify pipeline follows the same high-level architecture as existing automated fact-checking systems e.g. ClaimBusterâs four-stage pipeline (monitor, spot, match, check) [20], SAFEâs fact-decomposition-and- verification approach [21]. However, the details differ since checking factual claims about software is rare in the literature. As far as we know, we present the first fact-checking analysis of AI coding-session transcripts. A. Fact-checking pipeline 1) Categorise responses Parts (i) and (i) of the pipeline are developed using a similar approach to §IV-A2â§IV-A3. We use an LLM (Codex GPT- 5.6 Sol) to decompose into component parts each of the 678 TABLE VII: Definitions of the response categories. CategoryDefinitionVerdict categories Checkable-factA single fact about one artifact (a count, a presence/absence, a structural property) e.g. âFull suite green: 1174 passed, 0 failed.â Matches / Contradicts / Unreproducible Mechanism- explanation Relates multiple facts or code locations into an explanation of how existing logic works (data flow, control flow, a multi-step process) e.g. âThe revert at extractor.py:4025-4028 fires when a field was tagged ENUM.â Confirmed / Wrong / Unreproducible Root-cause- diagnosis Diagnoses the cause of a stated or implied symptom/problem (something not working, unexpected, wrong, lost, or failing) e.g. âThe root cause: my scan uses stop_at_ret=False with a 256-insn window, which runs past the wrapperâs end into neighboring functions, picking up other classesâ cores.â Confirmed / Partial / Right-Conclusion-Wrong-Mechanism / Wrong / Unreproducible Design-fix- proposal Describes a proposed or implemented code change/fix and what happened to it e.g. âThe fix: attribute by control-flow block instead.â Adopted-Worked / Adopted-Revised / Adopted-Worked-But-Superseded / Not-Adopted / Failed / Mixed / Unreproducible OtherA statement of intent, an interrogative, an opinion, a hedge, or pure process narration, e.g. âThe exact cause hasnât been pinned down yet.â Not scored responses to the 678 user instructions studied in §IV-A. This yields 9,967 response components. Of these, 6,818 compo- nents are a concluding paragraph, a bolded takeaway or a summary table i.e. the elements of the response that a human would likely pay most attention to. We therefore focus on these 6,818 components in the analysis below. We categorise the response components using an LLM (Codex GPT-5.6 Sol) instructed using: (i) the taxonomy sum- marised in Table ÌVII, and (i) the response component together with its immediate context (the preceding instruction and the AIâs full response). The response taxonomy was developed us- ing an iterative abductive approach: select categories iteratively with convergence after three rounds, then refine boundary cases by examining the differences between two independent LLMs (Claude Sonnet 5 and Codex GPT-5.6 Sol) categorisa- tions. For the final codebook Cohenâs Îș = 0.803 (89.63% raw agreement) between Claude and Codex, and Îș = 0.800/0.746 between Claude/Codex and a human categorising a stratified sample of 5 responses from each category. Prior claim taxonomies come mostly from journalism and social-media fact-checking, and classify by whether a claim is numeric, a quote, an entity/event property, a position statement [19], and verification is by internet search. Our taxonomy also organises claims by the verification method each re- quires, but verification uses software artefacts. Checkable- fact and mechanism-explanation are settled using the commit the response was made against: by executing, searching, or reading code at that one point in time. Root-cause-diagnosis and design-fix-proposal instead require looking forward past that commit: whether a fix targeting the diagnosed cause went on to resolve the symptom, or whether the proposed change was actually implemented and still present at a later, fixed reference commit. The full codebook, including definitions, key signals, exam- ples, and boundary notes, is provided in our dataset package. Figure 9 shows the two independent LLMs distributions of categories. It can be seen that they are in good agreement. Checkable-fact dominates ( Ì64-68%), followed by Design- fix-proposal (16-17%). Root-cause-diagnosis, Mechanism- explanation and Other each make up under 8%. 2) Evidence collection and scoring Table ÌVII lists the possible verdicts for a response com- ponent in each category. To support reasoning about the checkable-fact root-cause mechanism design- fi x other 0 2000 4000 # components Claude Codex Fig. 9: Categories assigned to the response components. appropriate verdict for each response component, an evidence bundle is generated for each as follows. Initially, for each response component a custom python script is used to gather an evidence bundle from the codebase itself (re-running code, static search, git-history tracing, binary parsing etc), then an LLM (Claude Sonnet 5) is used to score the response. The LLM has access to the codebase, session transcripts and project documents and is instructed to gather additional evidence as needed. Any such additional data is added to the evidence bundle and the response is re-scored. This is repeated until the evidence bundle converges (no new data is added): approximately 60% converged immediately, 35% with one extra round and 5% with two extra rounds. Using the converged evidence bundle, a stratified sample of 20 response components (five in each of the four categories) was scored by two independent LLMs (Claude Sonnet 5 and Codex GPT-5.6 Sol). A human researcher reviewed disagree- ments between the two LLMs, looking for patterns in the boundary cases. For commonly occurring patterns the scoring rubric was updated with clarifying text. The stratified sample was then expanded in stages (5%, 40%, 75%, then the full 100% of each category), with the same disagreement review and rubric-refinement cycle repeated at each stage, until every response component in every category had been scored by both LLMs. The full rubric including examples and boundary notes is provided in our dataset package. B. Results Since Checkable-fact dominates the response components, drawing samples uniformly at random tends to lead to rel- atively few samples of the other categories of response and so wide confidence intervals. We therefore use a stratified CategoryScorerAccurate95% CI Checkable- fact Claude 2321/2461 (94.3%)93.3â95.2% Codex 2305/2443 (94.4%)93.4â95.2% Mechanism- explanation Claude381/418 (91.1%)88.0â93.5% Codex376/420 (89.5%)86.2â92.1% Root- cause- diagnosis Claude275/308 (89.3%)85.3â92.3% Codex275/304 (90.5%)86.6â93.3% Design- fix- proposal Claude565/712 (79.4%)76.2â82.2% Codex566/709 (79.8%)76.7â82.6% 0123+ errors in response 0 100 200 300 400 # responses Claude Codex TABLE VIII: Left: evaluation verdicts for each of the response categories, with Wilson 95% confidence intervals. Right: num- ber of responses containing 0, 1, 2, or 3+ inaccurate compo- nents (Checkable-fact, Root-cause-diagnosis and Mechanism- explanation categories only). sampling approach, drawing extra samples from less frequent categories. Table VIII summarises the response accuracy scor- ing by two independent LLMs for the resulting sample of approximately 3,900 response components (out of the 6,818 total population) 6 . To the best of our knowledge these are the first reported error rates for AI responses during multi-turn code development sessions. Table VIII shows that there is good agreement between the two independent LLMs across all categories of re- sponse. Checkable-fact response components are the most accurate with approximately 6% (1 in 18) containing errors. Mechanism-explanation and Root-cause-diagnosis are less ac- curate, with error rates of approximately 10%. Design-fix- proposal has notably lower accuracy than the other categories, but this is also an inherently harder category to get right. Design-fix-proposal is a proposed code fix. Whether it is Adopted or Not-Adopted is decided by the user and involves a judgement call e.g. on the cost/risk of the proposed change, on whether the benefit is sufficient to make the change worthwhile. The Not-Adopted cases are therefore not really response errors, yet the tableâs Accurate column counts them against accuracy since they are neither confirmed working nor Unreproducible. Excluding the 107 Not-Adopted cases (95 for Codex) from the denominator, and counting only Adopted- Revised, Mixed and Failed as errors (each was, at least in part, revised or abandoned because of a deficiency, or in the Failed case reverted outright), gives an accuracy of 565/605 (93.4%) for Claude and 566/614 (92.2%) for Codex. A full response can contain multiple components, and this component error rate translates into 23.0% of responses containing one or more inaccurate components for Claude and 21.6% for Codex, i.e. roughly 1 in 4-5 responses contain errors 7 . Table VIII shows that the majority of responses have either zero or one error, but around 8% of responses containing two or more errors. 6 Note that the Claude verdicts use the Claude categorisations and the Codex verdicts use the Codex categorisations. Responses categorised as Other are not scored. UNREPRODUCIBLE verdicts are not included in the reported figures. 7 These figures exclude Design-fix-proposal components, and responses where that is the only scoreable component. When included the error rate rises to 29.5% for both Claude and Codex. A human researcher with access to all of the project artefacts manually checked 200 of the verdicts reporting deficiencies (Contradicts, Wrong etc) and a stratified sample of 200 clean verdicts (Matches, Confirmed etc), 49-51 per category. The ac- curacy between the AI and human verdicts is: Checkable-fact 98.2%, Mechanism-explanation 98.0%, Root-cause-diagnosis 96.6%, Design-fix-proposal 93.3%. The verdicts with AI- human disagreement were not modified as the aim here is to calibrate the AI results against human judgement. That said, if they were applied then the accuracy figures in Table VIII would change by: Checkable-fact 0% (changes cancel out), Mechanism-explanation -0.48%, Root-cause-diagnosis +0.25%, Design-fix-proposal +0.60%. These shifts are minor, well within the reported confidence intervals. See Additional Material in our dataset package for examples of errors in each category. VII. LIMITATIONS Scope: one dataset. Probably the main limitation of the work is that it is confined to a single dataset, albeit for a reasonably sized, real AI-written tool. On the other hand, full- project datasets, meaning commits plus session transcripts and AI-generated notes rather than just commits, are otherwise largely absent from the public record, so the dataset itself is one of the paperâs primary contributions, which we hope will spur publication of other datasets in future. LLM-based categorisation and scoring. Much of the analysis relies on LLMs to decompose, categorise, and score instructions, commits, and responses. We use two independent LLMs (Claude and Codex, from two competing vendors) throughout, which consistently agree with each other and with human calibration samples (Îș = 0.70â0.81), with no evidence of Claude favouring its own output. A sample of all (both defect/error and agreed-clean) verdicts was additionally checked by a human researcher with only a minor level of disagreement observed. VIII. CONCLUSIONS Our main contributions are: (i) the Extractor dataset, (i) two reusable tools for tracing AI-authored Python code back to its provenance, (i) three taxonomies, extending or filling a gap in an existing classification scheme; and the process, reliability, and communication findings these yield when applied to Extractorâs development. We find that user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consulta- tion, that AI code-generation errors are common (14.3% of checked events introduced one), but the same AI process that made the mistakes also built the test suite that caught them before commit. Claudeâs responses are relatively reliable when reporting or verifying a fact (94.3% accurate), when explaining an existing mechanism (91.1%) or diagnosing a root cause (89.3%) but less so when proposing a design fix (79.4%). Overall roughly 1 in 4-5 responses contain at least one inaccuracy. REFERENCES [1]M. Chen et al., âEvaluating Large Language Models Trained on Code.â arXiv:2107.03374, 2021. [2]J. Liu, C. S. Xia, Y. Wang, and L. Zhang, âIs Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Gen- eration,â in Advances in neural inf. Processing syst. 36 (NeurIPS 2023), 2023. doi: 10.52202/075280-0943. [3]N. Tang et al., âProgramming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions,â in Proc. 41st IEEE/ACM international conference on automated software engineering (ASE 2026), 2026. doi: 10.1145/3832783.3834377. [4]E. B. Swanson, âThe Dimensions of Maintenance,â in Proc. 2nd international conference on software engineering (ICSE), 1976, p. 492â497. [5]ISO/IEC/IEEE, âISO/IEC/IEEE 14764:2022 â Soft- ware Engineering â Software Life Cycle Processes â Maintenance,â International Organization for Standard- ization, 2022. [6]J. Baumann, V. Padmakumar, X. Li, J. Yang, D. Yang, and S. Koyejo, âSWE-chat: Coding Agent Interac- tions From Real Users in the Wild,â arXiv preprint arXiv:2604.20779, 2026. [7]H. Li, H. Zhang, and A. E. Hassan, âAIDev: Studying AI Coding Agents on GitHub,â in Proc. 23rd inter- national conference on mining software repositories (MSR), 2026. [8]J. Ì Sliwerski, T. Zimmermann, and A. Zeller, âWhen Do Changes Induce Fixes?â in Proc. 2005 inter- national workshop on mining software repositories (MSR), 2005. [9]P. Rani, F. Petrulio, and A. Bacchelli, âOn Refining the SZZ Algorithm with Bug Discussion Data,â Empirical Software Engineering, vol. 29, no. 5, p. 115, 2024. [10]L. Tang, J. Liu, Z. Liu, X. Yang, and L. Bao, âLLM4SZZ: Enhancing SZZ Algorithm with Context- Enhanced Assessment on Large Language Models,â Proc. ACM Softw. Eng. 2, ISSTA, p. 343â365, 2025. [11]X. Zhao et al., âFailure as a Process: An Anatomy of CLI Coding Agent Trajectories,â arXiv preprint arXiv:2607.09510, 2026. [12]M. Wang, X. Xie, and Y. Huo, âTrajAudit: Automated Failure Diagnosis for Agentic Coding Systems,â arXiv preprint arXiv:2605.26563, 2026. [13]R. Shu et al., âWhat Resolve Rate Hides: Trajec- tory Structure Diagnostics for Coding Agents,â arXiv preprint arXiv:2607.06184, 2026. [14]C. Treude and M. A. Gerosa, âHow Developers In- teract with AI: A Taxonomy of Human-AI Collabo- ration in Software Engineering,â in Proc. 2nd ACM international conference on AI foundation models and software engineering (FORGE), 2025. [15]A. Mockus and L. G. Votta, âIdentifying Reasons for Software Changes Using Historic Databases,â in Proc. International conference on software mainte- nance (ICSM), 2000, p. 120â130. [16]A. Hindle, D. M. Germ Ì an, and R. C. Holt, âWhat Do Large Commits Tell Us?: A Taxonomical Study of Large Commits,â in Proc. International working conference on mining software repositories (MSR), 2008, p. 99â108. [17]J. Zhao, Z. Yang, L. Zhang, X. Lian, D. Yang, and X. Tan, âDRMiner: Extracting Latent Design Ratio- nale from Jira Issue Logs,â in Proc. 39th IEEE/ACM international conference on automated software engi- neering (ASE), 2024, p. 468â480. [18]M. Dhaouadi, B. J. Oakes, and M. Famelis, âCoMRAT: Commit Message Rationale Analysis Tool,â in Proc. IEEE/ACM 22nd international conference on mining software repositories (MSR 2025), data and tool show- case track, 2025. [19]L. Konstantinovskiy, O. Price, M. Babakar, and A. Zubiaga, âToward Automated Factchecking: Develop- ing an Annotation Schema and Benchmark for Con- sistent Automated Claim Detection,â Digital Threats: Research and Practice, vol. 2, no. 2, p. 1â16, 2021. [20]N. Hassan, F. Arslan, C. Li, and M. Tremayne, âToward Automated Fact-Checking: Detecting Check- worthy Factual Claims by ClaimBuster,â in Proc. 23rd ACM SIGKDD int. Conf. On knowledge discovery and data mining (KDD), 2017, p. 1803â1812. doi: 10.1145/3097983.3098131. [21]J. Wei et al., âLong-Form Factuality in Large Language Models,â in Advances in neural information processing systems 37 (NeurIPS 2024), 2024. [22]D. Cotroneo, C. Improta, and P. Liguori, âHuman- Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity,â in Proc. 36th IEEE international symposium on software relia- bility engineering (ISSRE 2025), 2025. [23]T. Mao, D. Zhao, H. Tang, X. Wang, and H. Zhang, âA Large-Scale Comprehensive Measurement of AI- Generated Code in Real-World Repositories,â arXiv preprint arXiv:2603.27130, 2026. [24]Y. Liu, R. Widyasari, Y. Zhao, I. C. Irsan, J. Chen, and D. Lo, âDebt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild,â arXiv preprint arXiv:2603.28592, 2026. [25]S. Sawada, T. Shirai, Y. Kashiwa, K. Yamaguchi, H. Iwata, and H. Iida, âTo What Extent Does Agent- generated Code Require Maintenance? An Empirical Study,â in Proc. 30th international conference on eval- uation and assessment in software engineering (EASE), 2026. [26]S. T. Cynthia, J. K. Das, and B. Roy, âAre We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developersâ Use of Coding Agents,â in Proc. 23rd international conference on mining software repositories (MSR 2026), 2026. doi: 10.1145/3793302.3793377. [27]Google, âProtocol Buffers.â 2008. Available: https:// protobuf.dev/ [28]F. Gilardi, M. Alizadeh, and M. Kubli, âChatGPT Out- performs Crowd Workers for Text-Annotation Tasks,â Proceedings of the National Academy of Sciences, vol. 120, no. 30, p. e2305016120, 2023. [29]L. Zheng et al., âJudging LLM-as-a-Judge with MT- Bench and Chatbot Arena,â in Advances in neural information processing systems 36 (NeurIPS 2023, datasets and benchmarks track), 2023. [30]Z. Lin, M. Zhou, and L. Li, âAtomicCommit- Bench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?â arXiv preprint arXiv:2607.03332, 2026. [31]B. P. Lientz, E. B. Swanson, and G. E. Tomp- kins, âCharacteristics of Application Software Main- tenance,â Communications of the ACM, vol. 21, no. 6, p. 466â471, 1978, doi: 10.1145/359511.359522. [32]I. Amit and D. G. Feitelson, âWhich Refactorings Reduce Bug Rate?â in Proc. 15th international con- ference on predictive models and data analytics in software engineering (PROMISE), 2019, p. 12â15. doi: 10.1145/3345629.3345631. [33]I. Amit and D. G. Feitelson, âCorrective Commit Probability: A Measure of the Effort Invested in Bug Fixing,â Software Quality Journal, 2021, doi: 10.1007/s11219-021-09564-z. IX. ADDITIONAL MATERIAL A. Examples of response errors Checkable-fact. âThe top of the list is dominated by low- level disassembly plumbing (find_adrp_ldr_target, find_adrp_add_target, _is_valid_selector, decode_ptr), called millions of times during any parsing, not per-output-decision.â This claim is contradicted by the responseâs own work. Thirty-three seconds earlier, the AI had itself computed exact per-function call counts for a benchmark run across three test artifacts: the highest of the four named functions was called 296,681 times, the lowest 40,780. Two and a half minutes later it wrote these same counts into a report file, later committed to the repository. All four counts are well under 300,000, not âmillionsâ. A static count of call sites in the current codebase confirms the structural reason: each function scans a small, fixed-size instruction window and is invoked once per higher- level decision (once per enum case, once per dispatch table), not once per instruction. Mechanism-explanation. âThe golden diff is exactly the9fieldsIpredicted,andnoneofthemare GEOPDComponentValue/GEOPDComponent.Soin the full extract_all run those classes are unchanged (resolved via readFrom dispatch, not writeTo).â The fact half of this claim (which 9 fields changed) is correct, but the mechanism is stated backwards. Thirteen minutes later, in the same session, the AI debugs why four tests are still failing despite this claim, and concludes in its own words that âthe walk itself recovers the wide-oneof field #91â, where âthe walkâ is the writeTo control-flow-graph walk it had just been editing, not readFrom dispatch. The commit implementing this fix is titled âCFG walk is the sole writeTo decoderâ and its message credits the writeTo walk, not readFrom dispatch, with recovering exactly the class and field the claim names. Root-cause-diagnosis. âThatâs because /tmp/c72.txt lives in my tool sandbox, not your shellâs filesystem.â When the user reports that a file the AI had pointed them to could not be found, the AI diagnoses a filesystem boundary between its own tool environment and the userâs shell. Twelve lines later, in the same session, the AI runs the identical command itself, from the very environment it says the file lives in, and gets the identical âfile not foundâ error. It briefly offers a second explanation (that the file âwas wiped during the long waitâ) before working around the problem with an inline heredoc instead. Neither explanation is checked, and neither is correct: earlier in the same session, every previous attempt to create the file had silently failed because of an unrelated tool outage, so the file had never existed anywhere. The AI disproves its own diagnosis without appearing to notice. Design-fix-proposal. âRaising _SPARSE_MAX_INSNS (currently 1000) would recover values for at least some of the larger enums, and itâs the only constant that would.â Asked whether increasing a scan-length constant would help recover values from large enums that a linear scan currently truncates, the AI proposes raising it. The user asks for this to be tested before deciding, and the AI runs the constant at 1000, 4000 and 30,000 instructions against the full test corpus (752 enums, 14,900 values): every enum gains zero additional values at any of the three settings. The constant was not the cause. The actual bug, found and fixed the same day, was an unrelated jump-table decoding gap; a second, separate fix later that day corrects a false âtruncatedâ warning on long methods that were already fully covered.