Paper deep dive
Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair
Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, Pekka Abrahamsson
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Localizing where an LLM-based agent first fails to uphold security during a repair can show which stage of its workflow needs an additional safeguard. This is difficult for silent failures, which are patches that pass syntactic and functional checks but still contain a security vulnerability. Because such patches give no observable failure signal, existing failure attribution methods, which rely on observed task failures and labelled failure steps, are less suited to them. We propose Security Awareness Gap Evaluation (SAGE), a trace-based method that combines an assessment of the security reasoning recorded at each turn with the reconstructed code history to identify the earliest turn at which a repair diverges from the task's security intent. We evaluate SAGE on 95 confirmed silent failures drawn from 3,684 repair traces produced by six agent frameworks and six base models on SecurityEval and CVEfixes. SAGE assigned an origin in 93 cases. Most origins were an unaddressed security requirement or an inadequate defence choice, and only five coincided with the code change itself. When the agent introduced the vulnerable code, the origin preceded the write in 14 of 19 cases. Repeated scoring and a second judge reproduced the origin type more consistently than the exact turn, and agreement was lowest for traces that kept only the final file.
Tags
Links
- Source: https://arxiv.org/abs/2610.06163v1
- Canonical: https://arxiv.org/abs/2610.06163v1
Trouble viewing inline? Open PDF directly →
Full Text
81,798 characters extracted from source content.
Expand or collapse full text
Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair WENJI BAI, Tampere University, Finland MUHAMMAD WASEEM, Tampere University, Finland ZEESHAN RASHEED, Tampere University, Finland JAAKKO PELTONEN, Tampere University, Finland PEKKA ABRAHAMSSON, Tampere University, Finland Localizing where an LLM-based agent first fails to uphold security during a repair can show which stage of its workflow needs an additional safeguard. This is difficult for silent failures, which are patches that pass syntactic and functional checks but still contain a security vulnerability. Because such patches give no observable failure signal, existing failure attribution methods, which rely on observed task failures and labelled failure steps, are less suited to them. We propose Security Awareness Gap Evaluation (SAGE), a trace-based method that combines an assessment of the security reasoning recorded at each turn with the reconstructed code history to identify the earliest turn at which a repair diverges from the task’s security intent. We evaluate SAGE on 95 confirmed silent failures drawn from 3,684 repair traces produced by six agent frameworks and six base models on SecurityEval and CVEfixes. SAGE assigned an origin in 93 cases. Most origins were an unaddressed security requirement or an inadequate defence choice, and only five coincided with the code change itself. When the agent introduced the vulnerable code, the origin preceded the write in 14 of 19 cases. Repeated scoring and a second judge reproduced the origin type more consistently than the exact turn, and agreement was lowest for traces that kept only the final file. Additional Key Words and Phrases: Silent failures, LLM-based agents, Vulnerability repair, Failure localization 1 Introduction Large language model (LLM)-based agents increasingly approach software repair as an iterative process of planning, tool use, code editing, and validation [8,49]. Yet, passing the available tests does not necessarily establish that the relevant security requirements have been met [50]. Recent work has identified silent failures (SFs): patches that pass the available syntactic and functional checks while retaining or introducing a security vulnerability [6]. Confirming a silent failure shows that the delivered repair remains insecure, but not where the repair process first went wrong. The same insecure patch may stem from a plan that omits the relevant security requirement, a defence that does not close the weakness, or an implementation that applies an appropriate defence incorrectly. Such a flaw may then pass through later review stages unnoticed. Each origin may call for a different remedy, ranging from prompt refinement to an additional verification checkpoint. We therefore ask where in the execution trace a divergence from the task’s security intent first becomes observable. We define this position as the origin of the silent failure: the earliest turn at which the recorded reasoning or artefact history shows such a divergence, as opposed to the write that eventually places the vulnerable code in the file. Figure 1 illustrates two ways in which the origin of a silent failure can differ from a code change. In case (a), the planner agent selects AES in ECB mode at turn 1, two turns before the vulnerable code is written. A diff identifies the introducing write, but only the trace reveals the earlier inadequate defence choice. In case (b), the vulnerable code is inherited from the input file and remains untouched, so there is no introducing write to locate. The trace instead records an initial plan that omits the relevant security requirement Authors’ Contact Information: Wenji Bai, wenji.bai@tuni.fi, Tampere University, Tampere, Finland; Muhammad Waseem, muhammad.waseem@tuni.fi, Tampere University, Tampere, Finland; Zeeshan Rasheed, zeeshan.rasheed@tuni.fi, Tampere University, Tampere, Finland; Jaakko Peltonen, jaakko.peltonen@tuni.fi, Tampere University, Tampere, Finland; Pekka Abrahamsson, pekka.abrahamsson@tuni.fi, Tampere University, Tampere, Finland. arXiv:2610.06163v1 [cs.CR] 5 Oct 2026 2Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson Artefact history (file snapshots) Input file cipher = DES.new( secret_key) After write (turn 3) = final patch cipher = AES.new( secret_key, AES.MODE_ECB) ! no file change Execution trace (agent turns) Turn 1 Planner agent “Replace DES with AES; use ECB mode.”The plan fixes the cipher mode before any code exists. Turn 2 Engineer agent Proposes the AES-ECB patch in a chat message. No file is changed. </> Turn 3 Engineer agent Writes the patch to the file and reports completion. </> write Origin inadequate defence Message only no file change Introducing write vulnerable code written (a) Agent-introduced vulnerability Artefact history (file snapshots) Input file origin = ( self.request .headers .get("Origin")) ! After write (turn 2) # four planned fixes # applied; the Origin # line is unchanged ! Final patch # identical to the # turn-2 snapshot ! Execution trace (agent turns) Turn 1 Planner agent Plans four functional fixes; omits the Origin-header checkthat the task requires. Turn 2 Engineer agent Implements the four planned fixes. The vulnerable line is left untouched. </> Turn 3 Reviewer agent “All four planned fixes are correctly applied.”Approves the patch. write Origin unaddressed requirement Write, not introducing weakness already present Missed detection opportunity (b) Inherited vulnerability Fig. 1. Examples of the two cases in silent failure origin localization. and a subsequent review that does not revisit the omitted requirement. Final-patch analysis alone is therefore insufficient; origin localization calls for examining the recorded repair reasoning alongside the artefact history, and for separating the origin from both the introducing write and later missed detection opportunities. Existing research provides useful starting points for analysing agent failures, but does not address the origin of a silent failure in vulnerability repair. Failure taxonomies and attribution methods have mainly been developed and evaluated on executions with an observable task failure [10,46,48], whereas a silent failure passes the available checks and its deficiency is established only through separate confirmation. The prior study of silent failures attributes deficiencies to agent roles and workflow stages, but does not provide a systematic rule for localizing the earliest origin turn [6]. To address this gap, we propose Security Awareness Gap Evaluation (SAGE), a trace-based method for localizing the origin of a silent failure. SAGE first normalizes framework-specific traces into a common sequence of turns. It then uses an LLM judge, given only the trace prefix available at each turn, to score the security reasoning expressed along four dimensions; every non-zero score must cite a verbatim quotation linked to the raw record (see Section 3.3.1). In parallel, it reconstructs the artefact history to determine when the vulnerable code was introduced or left in place. Finally, an explicit decision rule combines the scores and artefact history with the confirmed failure anchor to identify the earliest trace-attributable divergence from the task’s security intent (see Section 3.3.2). We evaluate the resulting origin assignments through repeated scoring, cross- judge comparison, task-disjoint subsets, and human annotation of artefact-derived outputs. Three research questions (RQs) guide the study: RQ1.Where do silent failures originate, and what trace evidence supports the assigned origins? RQ2. How consistently does trace- and artefact-based evidence identify the same origin across repeated runs, task-disjoint data subsets, and independent judges? RQ3.How does localization reliability vary across silent failure categories, agent frame- works, and base models, and under what conditions does the identified origin become less consistent? We evaluate SAGE on 95 confirmed silent failures identified among 3,684 valid repair traces generated by six open-source agent frameworks and six base models on tasks from SecurityEval [33] and CVEfixes [7]. SAGE localizes an origin in 97.89% of cases (93/95). Most identified origins involve unaddressed security requirements or inadequate defence choices. Among the 19 cases in which Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair3 the agent introduced vulnerable code, the origin precedes the introducing write in 14. Repeated scoring and an independent judge model reproduce the same origin type in most cases. Agreement with human annotators on code provenance and the introducing write is moderate to substantial (휅between 0.58 and 0.74) and varies across task subsets. Exact-turn agreement is lowest for the small group of traces that preserve only the final file. To summarize, our main contributions are: • A trace-based localization method SAGE. We formulate silent failure localization as identifying the earliest trace-attributable divergence from the task’s security intent in an execution that passes functional validation. We propose SAGE, which combines quotation- anchored assessment of expressed security reasoning with reconstructed artefact histories. •A multi-source reliability evaluation. We evaluate SAGE through repeated scoring, cross-judge comparison, task-disjoint subsets, and human annotation. •A characterization of localization boundary conditions. We examine how reliabil- ity varies across silent failure categories, agent frameworks, base models, and artefact availability, and identify conditions associated with less consistent localization. 2 Background and Related Work Security of LLM-generated code and agentic vulnerability repair. Empirical studies report security weaknesses in LLM-generated code [4,16,26,30,32] and limitations in LLM vulnerabil- ity detection and reasoning [15,37]. Datasets such as SecurityEval, LLMSecEval, CVEfixes, and BaxBench support security evaluation through vulnerability prompts, historical fixes, and exploit- based tests [7,33,36,38]. Zero-shot and neural repair approaches remain limited on real-world vulnerabilities [31,43], and attention has moved to agents that locate, edit, and validate fixes over several turns [8,49]. Such runs are commonly scored by test passage [22], yet test-passing patches may still be incorrect [34,41,52], and a simple non-agentic pipeline can achieve competitive scores [44]. These studies motivate evaluation beyond final test outcomes, but they do not explain where in an agentic repair trajectory a security failure first becomes attributable. Silent failures in agentic repair. Bai et al. [6] studied silent failures in agentic vulnerability repair and classified confirmed SF cases by category, agent role, and workflow stage. That work shows that security deficiencies can remain hidden behind patches that pass available syntactic and functional checks. The present study addresses the next diagnostic problem: once a silent failure is confirmed, where in the recorded repair trajectory did the process first diverge from the task’s security intent? This requires separating the origin turn from code provenance, the introducing write, and later missed detection opportunities. Prior silent-failure categorization did not provide a systematic method for such turn-level localization or evaluate whether origin assignments are consistent across scoring runs and judges. Failure analysis and attribution in agentic systems. Taxonomies characterize agent failures by system design, inter-agent misalignment, and task verification [10], and analyses of repair- agent trajectories associate failed runs with repetitive, non-adaptive action cycles [9]. AgentDebug classifies errors by agent module and diagnoses root-cause errors to guide recovery [53]. Attribution methods identify responsible agents or steps under different evidence conditions. Zhang et al. [48] apply LLM-based attribution to failed multi-agent runs where task failure is observable and step- level labels are available. AgenTracer [46] trains a tracer on failure trajectories constructed through counterfactual replay and fault injection. FAMAS [18] ranks agents and actions using spectrum analysis over repeated executions, requiring additional trajectories that may be unavailable in a single-trace audit. DuoTrace [47] learns regular execution patterns from normal trajectories and filters anomalous steps before downstream attribution, but its anomaly scores are not designed to 4Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson 1 Phase I: Dataset Construction Task Sampling 150 tasks drawn from 1,236 eligible tasks Agent Framework and Base Model Selection 6 frameworks×6 base models Execution and Verification L0 syntactic validity L1 functional validation L2 static security analysis L3 exploitability-oriented screening Manual Confirmation TP / Partial / FP; Cohen’sκ, percent agreement, Gwet’s AC 1 Trace Normalization logs mapped to canonical turns 2 Phase I: SAGE Origin Localization Turn-Level Scoring Four security-reasoning dimen- sions, scored 0–3 per turn TRDIVR SA 0 12 3 Artefact Reconstruction Snapshot after each write; failure anchor traced back input after write ! final scores history Origin Rule earliest applicable turn becomes the origin; its type is the matching candidate (i)–(i) (i) Inadequate defence selected (i) Unaddressed requirement (i) Defect in the code change Localization Output t 1 t n origin turn and type introducing write missed detection opportunity 3 Phase I: SAGE Evaluation Reliability and Robustness Consistency over repeated LLM runs and a second LLM judge; robustness over task-disjoint subsets Validity Assessment Two annotators; provenance and intro- ducing write Coverage and Boundary Conditions By deficiency, framework, model, arte- fact availability Evaluation Metrics Percent agreement, Cohen’sκ, Krippen- dorff’sα; exact-turn and within-one-turn agreement Fig. 2. Overview of the research design. link a step to a confirmed security deficiency in a functionally passing patch. Benchmark studies further show that attribution accuracy depends on the completeness of the supplied evidence [11] and remains limited on lengthy traces [14,25]. In contrast, SAGE localizes within a single trace whose functional checks passed and asks where the security-intent divergence first becomes trace-attributable. LLM-as-a-judge in software engineering. LLM judges are increasingly used to evaluate gen- erated code and software agents [35,54]. However, their agreement with human evaluators varies across tasks [39], and their verdicts can shift with prompt formulation and answer position [51]. These findings are directly relevant to SAGE because origin localization uses judge-assigned as- sessments of expressed security reasoning. Therefore, we do not rely on a single LLM judgment as ground truth. Instead, SAGE uses prefix-only, quotation-anchored scoring and evaluates reliability through repeated scoring, cross-judge comparison, task-disjoint resampling, and human annotation of artefact-derived outputs. Positioning of this study. Relative to the closest work, SAGE combines three features for localizing confirmed silent failures. First, it analyses a single recorded repair whose patch passed available syntactic and functional checks but was independently confirmed to contain a security deficiency. Second, it combines the failure anchor, prefix-only assessment of expressed security reasoning, and reconstructed artefact history to identify the earliest trace-attributable divergence under a systematic localization rule. This separates the origin turn from code provenance and, where one exists, the introducing write; it also allows origins to be assigned for inherited vulnerabilities. Third, it evaluates reliability from complementary sources: repeated scoring and cross-judge comparison for origin type and origin turn, and human annotation plus task-disjoint resampling for artefact-derived outputs. In this way, the study moves beyond asking whether an agentic repair succeeded or failed, and instead examines where a confirmed security failure first becomes observable in the repair process. 3 Methodology 3.1 Research Design Overview The study consists of three phases (see Figure 2). Phase I constructs the corpus of confirmed silent failures, Phase I applies SAGE to localize their origins, and Phase I assesses the consistency of the localization outputs and the agreement of the artefact-derived outputs with human annotation. Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair5 Table 1. Selected agent frameworks. FrameworkTypeArchitectureSelection rationale CAMEL [24]MultiTwo-role dialoguePlanner–engineer roles with explicit handoffs AG2 [42]MultiGroup chatRound-robin chat with a security reviewer Deep Agents [23]MultiPlanner–subagentHierarchical decomposition and delegation Aider [17]SingleEdit–feedback loopDiff edits with lint and test feedback MiniSWE-Agent [45]SingleReAct loopReasoning and tool use, no inter-agent messages OpenHands [40]SingleEdit–feedback loopSandboxed file editing and shell execution 3.2 Phase I: Silent Failure Dataset Construction 3.2.1 Task Sampling. We use two publicly available security-focused datasets: SecurityEval [33] and CVEfixes [7]. Tasks were selected according to four criteria: security relevance, availability of Common Weakness Enumeration (CWE) annotations, Python implementation, and compatibility with the selected agent frameworks. After language and format filtering, the two datasets provide a pool of 1,236 eligible tasks: 121 from SecurityEval and 1,115 from CVEfixes. From this pool, we draw 150 tasks (75 per dataset), using a fixed random seed and stratification by CWE family with proportional allocation within each dataset. Each SecurityEval task supplies the code context of its prompt as the input file, and each CVEfixes task supplies the vulnerable version of the file together with a repair instruction naming the target CVE and CWE. 3.2.2 Agent Framework and Base Model Selection. We select six open-source frameworks that had active release lines at the time of corpus construction and were applicable to vulnerability repair, directly or through adaptation (see Table 1). Each framework is executed with six base models: GPT-5.5 [28], Claude Opus 4.8 [2], Gemini 3.1 Pro [19], DeepSeek-V4-Pro [13], Kimi K3 [27], and Qwen3.8-Max [1]. These models were selected for coding and reasoning capability, programmatic access, and compatibility with the frameworks’ context-length and tool-use requirements. 3.2.3 Experimental Setup. To limit variation arising from execution settings, all frameworks and models use the same generation parameters where supported (temperature 1.0, 600-second timeout), and each run executes in an isolated environment allocated 5 CPU cores, 16 GB of RAM, and 512 GB of storage. Task content is kept unchanged across frameworks; only its format is adapted to each framework’s input interface. 3.2.4 Execution and Verification. Each task is executed once with each of the 36 framework–model configurations (5,400 runs), and each execution produces a generated patch, a complete agent trace, and any intermediate or auxiliary artefacts created during the repair process. Runs that terminate without producing a valid modification, including crashes, timeouts, and incomplete generations, are recorded as invalid traces and excluded. After this exclusion, 3,684 traces (68.22%) enter verification, of which 331 are flagged as silent failure candidates; Table 2 reports these counts and the excluded runs by framework. Generated patches are evaluated using a multi-level verification framework [6]: •Level 0 (L0): Syntactic validity. The generated code can be parsed without syntax errors. •Level 1 (L1): Functional validation. The generated artefact passes the available functional checks: its imports resolve, the functions and classes defined in the input file are preserved, and the associated tests pass where the task provides them. As SecurityEval provides no test suites, its tasks are assessed with the first two checks. •Level 2 (L2): Static security analysis. Bandit analyses the generated artefacts, with findings mapped to CWE identifiers. Patches containing high-severity findings are flagged. 6Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson Table 2. Silent Failure Dataset construction by framework. FrameworkExcluded runsValid tracesPassed L0–L3Failed L0/L1SF candidates AG23375634594955 Aider1567446245862 CAMEL23266834529231 Deep Agents3785224432950 MiniSWE-Agent3835174242766 OpenHands2306705495467 Total1,7163,6842,844509331 •Level 3 (L3): Exploitability-oriented screening. A rule-based score푅 ∈ [0,100]is computed for each patch by pattern matching. It sums weights for attacker-controllable input sources (2–4) and dangerous sinks such as dynamic code execution or unsafe deserialization (6–12); a sink weight is multiplied by 1.5 when a source occurs in the same patch and by 1.3 when the sink matches the CWE targeted by the task. The score increases by 10 if the targeted weakness pattern persists and decreases by 2 for each detected safeguard category (at most 10). A patch is flagged when푅 ≥15, using the maximum file score for multi-file patches. These settings follow prior work [6]; because every flagged candidate is reviewed manually, the threshold sets screening sensitivity. Patches that pass L0 and L1 but are flagged at L2 or L3 are treated as silent failure candidates. CAMEL accounts for 292 of the 509 L0/L1 failures, of which 241 failed functional validation (L1), mostly because the delivered file omitted functions or classes defined in the input file. These levels provide screening signals. Only manually confirmed candidates are included in the corpus. 3.2.5 Manual Confirmation. For each silent failure candidate, the annotator examines the task description, generated artefact, security findings, and relevant intermediate artefacts. Candidates are classified as follows: •True positive (TP): the patch retains or introduces a clear vulnerability relevant to the task; •Partial: the patch contains a relevant mitigation attempt, but a plausible vulnerability remains; • False positive (FP): the reported finding does not represent a relevant security weakness in context. Both TP and Partial cases are retained in the confirmed silent failure corpus, whereas FP cases are excluded. To assess the consistency of this confirmation process and limit potential individual bias, we conduct an Inter-Rater Reliability (IRR) analysis. The first annotator (R1) reviews all silent failure candidates, while a second annotator (R2) independently reviews a random subset of 60 candidates using the same criteria. Agreement is assessed primarily using Cohen’s휅[12], with percent agreement and Gwet’s AC 1 [20] reported as complementary measures in Table 3. As inclusion in the corpus is a binary decision (TP or Partial versus FP), we also report binary agreement: the annotators agreed on 54 of the 60 candidates (90.00%;휅=0.74). All disagreements were resolved through discussion between the two annotators. The resulting corpus contains 95 confirmed silent failure cases from 19 tasks (87 cases from 15 CVEfixes tasks and 8 from 4 SecurityEval tasks), each with a failure anchor recording the affected artefact and the confirmed deficiency (an absent defence, an insufficient defence, or a vulnerability introduced by the change). The largest task, a CVEfixes repair of a server-side request forgery vulnerability, contributes 16 cases. Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair7 Table 3. Inter-rater reliability on the silent failure candidates subset. (a) Inter-rater agreement MetricValue Observed agreement0.88 Cohen’s 휅0.73 Gwet’s AC 1 0.85 (b) Category confusion matrix (푛= 60) R1\ R2TPPartialFP TP613 Partial061 FP1141 Table 4. The SAGE instrument: the four scoring dimensions and the common anchored scale applied to each of them. (a) Scoring dimensions DimensionInterpretation Threat Recognition (TR)The relevant threat, attack path, or trust boundary is identified Defensive Intent (DI)An appropriate defence and its in- tended security role are specified Verification Rigour (VR)The implemented defence is tested or challenged Scope Awareness (SA)The code and artefacts requiring se- curity consideration are recognized (b) Dimension scale ScoreExpressed reasoning 0None related to security 1Generic security remarks 2Specific to the task 3Task-specific, and addresses completeness or the conditions under which the defence holds 3.2.6 Trace Normalization. The selected frameworks record messages, tool calls, and artefact changes in different formats and at different levels of granularity. To enable consistent analysis across frameworks, we normalize each execution trace into a sequence of canonical turns. A canonical turn consists of an agent-authored message together with the tool calls, tool results, and artefact changes associated with that message. Depending on the framework and the execution step, a canonical turn may include: agent messages; handoff records; planning artefacts; review comments; intermediate code; and auxiliary artefacts. 3.3 Phase I: SAGE Origin Localization In Phase I, SAGE operates on the canonical turns produced by trace normalization. It first measures the security reasoning expressed in every turn and then locates two positions for each confirmed silent failure: the introducing write, derived from the artefact history alone, and the origin turn, derived from the artefact history together with the per-turn scores. 3.3.1 SAGE Turn-Level Scoring. SAGE measures the security reasoning expressed by an agent over the course of an execution. For each canonical turn, SAGE independently scores the four dimensions in Table 4 on a common scale. Alongside these dimensions, each turn is assigned two auxiliary labels: (i) Applicability, which distinguishes substantive planning, implementation, review, and verification turns from administra- tive framework events, such as environment setup or tool polling; and (i) Security opportunity, which indicates whether the turn makes or reviews a decision, code path, or artefact materially connected to the task’s security intent, so that security reasoning at that turn could affect whether this intent is met. The judge assigns this label from the task context and the trace prefix only, without access to the confirmed deficiency; the origin rule links candidate turns to the deficiency afterwards (see Section 3.3.2). We use GPT-5.6 Sol [29] as the primary LLM judge because it lies outside the evaluated base- model set, so that no trace is scored by the model that produced it. The judge scores one canonical 8Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson turn (see Section 3.2.6) at a time in trace order, receiving the task context and the trace up to and including that turn; later turns, multi-level verification results, and confirmation labels are withheld. To ensure auditability, every non-zero score must include a verbatim quotation linked to a canonical item and, through it, to the raw execution trace. All turns involving code planning, implementation, review, or verification are scored. An applicable turn with no expressed security reasoning receives a score of 0, indicating the absence of such reasoning in the recorded trace. Only administrative framework events, such as formatting, file bookkeeping, and framework handshake messages, are excluded. 3.3.2 SAGE Origin Localization. After scoring, SAGE localizes each confirmed silent failure using the failure anchor recorded during manual confirmation, the reconstructed artefact history, and the per-turn scores. The anchor identifies the confirmed deficiency and the affected artefact, but contains no origin label. The origin turn is the earliest canonical turn at which the available evidence indicates a failure to satisfy the task’s security intent and links that failure to the confirmed deficiency under the rule below. This operational definition identifies a trace-attributable divergence. Supporting evidence may come from expressed reasoning or artefact changes; an introducing write is therefore not required to localize the origin of an inherited vulnerability. Localization proceeds in two steps. The first step establishes code provenance. The original vulnerable artefact is treated as the baseline and compared with the snapshot produced after each write. A deficiency is classified as agent-introduced when its anchor-linked fragment is absent from the baseline and appears in a later snapshot. This category includes newly introduced weaknesses, insufficient mitigations, and the removal or weakening of an existing defence. A deficiency is classified as inherited from the input file when it already exists in the original code and is not eliminated by subsequent writes. The second step assigns two trace positions to each case. The first is derived from the artefact history alone. For an agent-introduced vulnerability, the introducing write is the earliest write after which the vulnerability appears and remains present through the final snapshot. If an intermediate write removes the vulnerability and a later write reinstates it, the reinstating write is selected. An inherited vulnerability has no introducing write. The second position, the origin turn, is derived from the artefact history together with the per-turn SAGE scores. A fixed decision rule considers the following candidates and selects the earliest applicable turn: (i)Inadequate defence selected. The earliest turn before the introducing write (or before the delivered repair for an inherited vulnerability) at which the agent expresses a defence linked to the failure anchor, indicated by a non-zero defensive-intent score. The score records only that a defence is expressed; the defence is treated as inadequate because manual confirmation established that the delivered patch still contains the deficiency it addresses. (i) Unaddressed requirement. The first turn, possibly the introducing write itself, at which the agent has an opportunity to address the task’s security intent but expresses neither threat recognition nor defensive intent. (i) Defect in the code change. The introducing write, selected only when no turn qualifies under (i) or (i); in that case, the agent expresses threat recognition or defensive intent at the write itself. This candidate does not apply to inherited vulnerabilities. The origin type therefore records which candidate of the rule was selected, while the origin turn identifies its position in the trace and may precede the introducing write. An auxiliary output flags implementation or review turns after the origin whose verification-rigour or scope-awareness score is at most 1 as missed detection opportunities; these turns are reported separately and are not selected as origins. The threshold follows the scale in Table 4(b): a score of 0 or 1 means that the Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair9 1 Consistency scores at assigned originsorigin typeorigin turn Repeated scoring Tests Run-to-run variation Data 3 runs; 60 of 95 cases Metrics % agreement, Krippendorff’sα Cross-judge comparison Tests Judge dependence Data 2 judges; 60 of 95 cases Metrics % agreement, Krippendorff’sα 2 Agreement with human annotation code provenance introducing write Human annotation Tests Disagreement with humans Data R1: 95 cases; R2: 60 cases Metrics % agreement, Cohen’sκ Task-disjoint resampling Tests Task-sample dependence Data 95 cases; task-level splits Metrics % agreement, Cohen’sκ Fig. 3. Multi-source evaluation design of SAGE. turn records no security-related verification or scope reasoning, or only generic remarks, whereas a score of 2 requires reasoning specific to the task. After applying the rule, each case is assigned one of two outcomes: (i) Localized indicates that an origin turn was selected. (i) Undetermined indicates that no origin turn was selected, for one of two recorded reasons: no candidate turn could be linked to the confirmed deficiency, or the per-turn scores required by the rule are unavailable because scoring failed for the trace. Provenance and the introducing write do not depend on the scores and are reported for both Localized and Undetermined outcomes. All localization results are checked automatically. For an agent-introduced deficiency, the check verifies that the anchor-linked fragment is absent from the baseline, is introduced or reintroduced by the selected write, and remains present thereafter. For a deficiency inherited from the input file, the check verifies that the anchored deficiency is present in the baseline and persists across subsequent snapshots. These checks ensure the fragment history aligns with the assigned provenance and that the origin turn precedes or coincides with the introducing write. Fragment matching only locates code; the vulnerability itself is established through manual confirmation (see Section 3.2.5). 3.4 Phase I: SAGE Evaluation In Phase I, we assess SAGE through a multi-source reliability evaluation (see Figure 3): repeated scoring and cross-judge comparison assess the consistency of judge-dependent outputs, while human annotation and task-disjoint resampling assess agreement on artefact-derived outputs and its sensitivity to task composition. 3.4.1 Reliability and Robustness. We evaluate the reliability and robustness of SAGE through three complementary analyses: (i)Scoring stability. The primary LLM judge (GPT-5.6 Sol) scores the full set of confirmed silent failures once. A stratified random subset (60 of the 95 SF cases, across all frameworks) is scored in two additional runs, giving three scores per case in the subset. The subset covers different frameworks and reconstruction conditions. Across the three runs, we evaluate consistency on the four dimension scores at the origin turn and on the origin type and origin turn re-derived from each run’s scores. (i)Cross-judge consistency. A second LLM judge (Claude Opus 5 [3]) independently scores the same subset using the same rubric, prompt, and prefix-only context. This judge is blind to the primary scores, human annotations, and localization results. We compare the two judges on the same measures. (i)Robustness across data subsets. To assess whether SAGE–human agreement depends on the composition of the evaluation set, we conduct two task-level resampling analyses 10Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson on the 95 confirmed SF cases. First, across 1,000 random partitions, tasks are divided into two approximately equal, task-disjoint subsets, keeping all cases from the same task within the same subset. Agreement is recomputed separately for each resulting subset using the same metrics. Second, a leave-one-task-out analysis recomputes agreement after iteratively excluding each task. 3.4.2 Validity Assessment. We compare SAGE’s code provenance and introducing-write labels with human annotation; the origin type and origin turn are not annotated by humans and are assessed for consistency only (see Section 3.4.1). The first annotator (R1) labels all 95 confirmed cases, and a second annotator (R2) independently labels a stratified subset of 60 cases. Neither annotator is shown the SAGE scores or automated localization outputs. Both use the same failure anchors and reconstructed artefact histories as SAGE to label code provenance and the introducing write (a specific write turn, no introducing write, or cannot determine). Because R1 participated in corpus construction and in the design of the localization rule, R1 may be familiar with the cases and with the rule’s logic; R2 was not involved in either step. 3.4.3 Localization Coverage and Boundary Conditions. We examine localization coverage and its boundary conditions through two analyses: (i)Trace evidence and localizability. For each valid trace, we record the available evidence sources, number of canonical turns, number of applied writes, and reconstruction condition. For each confirmed silent failure, we record its localization outcome, code provenance, and supporting evidence. (i)Boundary conditions. We examine how SAGE localization outcomes vary across failure categories, agent frameworks, base models, reconstruction conditions, and artefact avail- ability. For each condition, we report localizability and the proportion of Undetermined cases. Where data permit, we also compare scoring consistency and cross-judge agreement. Results based on small groups are interpreted with caution. 3.4.4 Evaluation Metrics. To evaluate the reliability and validity of SAGE localization, we report joint localization coverage and, among jointly localized cases, agreement on origin type and exact or within-one-turn origin localization. Krippendorff ’s훼[21] assesses agreement on origin type and turn (cases with at least two ratings) and dimension scores at assigned origins (ordinal). Human comparisons use percent agreement and Cohen’s휅for code provenance and introducing-write labels, comparing annotators with each other and SAGE with each annotator separately. 4 Results 4.1 Trace Evidence and Localized Origins (RQ1) To answer RQ1, we examine how artefact histories and reasoning records support SAGE’s origin assignments, and which subsequent turns are flagged as missed detection opportunities. Table 5 summarizes code provenance, assigned origin types, origin timing relative to the introducing write, and missed detection opportunities. SAGE assigns an origin to 93 of the 95 confirmed cases (97.89%), leaving two cases Undetermined (2.11%). Artefact histories establish code provenance. SAGE classified 73 cases as inherited from the input file (76.84%) and 19 as agent-introduced (20.00%); three cases could not be linked to an identifiable code location (3.16%; see Table 5(a)). These outputs describe the available code-history evidence, while localization of the origin additionally draws on the trace record. For example, in one case, the agent completed the requested repair but introduced an unsafe configuration absent from the input file. The artefact change provides evidence for an introducing write, while identifying any earlier origin requires examining the preceding reasoning. In another case, a vulnerable database Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair11 Table 5. Overview of SAGE localization results (푁= 95 confirmed silent failures). (a) Code provenance in silent failures Outcome푁% Inherited from the input file7376.84 Introduced by the agent1920.00 No identifiable code location33.16 (b) Origin type assigned by SAGE Outcome푁% Unaddressed requirement5153.68 Inadequate defence selected3738.95 Defect in the code change55.26 Undetermined22.11 (c) Origin timing for agent-introduced code (푛= 19) Outcome푛% Before code introduction1473.68 At code introduction526.32 (d) Missed opportunities for later detection Outcome푁% At least one missed opportunity recorded5962.11 No missed opportunity recorded3637.89 Unaddressed requirement Inadequate defence Defect in code change Undetermined No code location (n = 3) Agent-introduced (n = 19) Inherited (n = 73) 413101 8650 2001 (a) Code provenance by origin type 0510152025 Turns from origin to introducing write Unaddressed requirement Inadequate defence Defect in code change (b) Origin-to-write distance (n = 19) Fig. 4. Origin localization outcomes. (a) Code provenance against origin type for all 95 confirmed silent failures; (b) Distance in canonical turns between the origin turn and the introducing write for the 19 cases in which the agent introduced the vulnerable code, one dot per case. operation remained unchanged from the input file to the final patch. Its history supports inherited provenance but provides no introducing write, so origin candidates must be sought in the reasoning record. Beyond code changes, the agents’ repair plans and explanations reveal how vulnerabilities persisted or emerged. SAGE attributed the origin to an unaddressed requirement in 51 cases (53.68%), the selection of an inadequate defence in 37 (38.95%), and a defect in the code change in 5 (5.26%) (see Table 5(b)). Inherited vulnerabilities accounted for most unaddressed-requirement origins (41/51, 80.39%) and inadequate-defence origins (31/37, 83.78%; see Figure 4(a)). In most agent-introduced cases, the assigned origin preceded the introducing write. Among the 19 cases in which the agent introduced the vulnerability (see Table 5(c)), the identified origin turn preceded the introducing write in 14 cases (73.68%). Of these, SAGE assigned eight origins to an unaddressed security requirement and six to the selection of an inadequate defence (see Figure 4(a)). In the remaining five cases (26.32%), the origin coincided with the introducing write; all five were single-message Aider traces in which the patch was generated and written in one 12Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson 0.50.60.70.80.91.0 Ordinal Krippendorff's α Threat recognition (TR) Defensive intent (DI) Verification rigour (VR) Scope awareness (SA) 0.930.76 0.950.86 0.930.68 0.890.68 (a) Agreement on the score at the origin turn Same judge, three runs (n = 57)Two judges (n = 50) 01020304050 Jointly localized cases (n = 50) TR DI VR SA 36 38 41 32 9 9 7 15 5 3 2 3 (b) Score difference between the two judges Same scoreDiffer by one pointDiffer by two or more Fig. 5. Reliability of the judge-assigned profile scores at the origin turn. step, leaving no earlier recorded turn for localization. Because the rule restricts origin candidates to the introducing write or earlier turns, this temporal ordering holds by construction. SAGE also assigned an origin turn in nearly all cases without an identifiable introducing write (74/76, 97.37%). These cases comprise vulnerabilities inherited from the input file and cases for which no specific code location could be identified. SAGE flagged missed detection opportunities in 59 cases. SAGE records a post-origin implementation or review turn as a missed detection opportunity when its verification-rigour or scope-awareness score is at most 1 (see Section 3.3.2). At least one such opportunity was identified in the majority of cases (59/95, 62.11%) (see Table 5(d)). The flags were concentrated in frameworks with multi-turn traces (AG2 29/32, Deep Agents 13/13, MiniSWE-Agent 11/15, OpenHands 4/9, CAMEL 2/4), whereas none of the 22 Aider cases was flagged because their single-message traces contain no turn after the origin. The overall share therefore largely reflects the framework composition of the corpus. Even in AG2, whose group chat includes a dedicated security reviewer, 28 of the 32 cases had a flagged reviewer turn. These flags reflect low recorded verification rigour or scope awareness under the SAGE rule. For example, in one case, the reviewer agent endorsed the completed sorting repair and reported no remaining issues, although an unsafe code-access configuration remained. SAGE left two of the 95 cases undetermined (2.11%; see Figure 4(a)): one with a vulnerability inherited from the input file and one without an identifiable code location. In both, no candidate turn could be linked to the confirmed deficiency. For the other two cases without an identifiable code location, SAGE assigned an origin from the planning and dialogue records alone, in both cases as an unaddressed security requirement. Key Findings from RQ1: We observed that agent traces provide complementary evidence about both failure origins and potential intervention points. Artefact histories support code prove- nance and introducing-write identification, while reasoning records support origin assignments to unaddressed security requirements or inadequate defence choices. Later implementation and review turns with low verification rigour or scope awareness were flagged as missed detection opportunities. Together, these findings suggest that trace-attributable divergence can become observable before later code edits, while subsequent turns provide additional opportunities for intervention. Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair13 Table 6. Reliability of SAGE localization outputs. (a) Origin consistency Repeated scoring Cross judge All four ratings Eligible cases (푛)575048 Type agreement (%)96.4992.0089.58 Exact-turn agreement (%)87.7278.0075.00 Within±1 turn (%)91.2384.0081.25 Origin type 훼0.960.860.92 Origin turn 훼0.820.590.74 (b) Agreement across task subsets Provenance (%) Introducing write (%) All 95 cases84.2181.05 Subset median84.2181.06 Subset central 95%74.42–96.8869.39–93.33 Leave-one-task-out82.56–89.8779.07–87.34 (c) Agreement with human annotations Comparison푛Code provenanceIntroducing-write label Agreement (%)휅Agreement (%)휅 R1 vs. R26090.000.7875.000.58 SAGE vs. R16088.330.7485.000.70 SAGE vs. R26085.000.6678.330.62 SAGE vs. R19584.210.6281.050.58 Notes. (a) Agreement uses cases localized in all compared ratings;훼uses cases with at least two ratings. All four ratings combine three primary-judge runs and one independent judge. (b) Subsets come from 1,000 random task-disjoint partitions; all values are agreement with R1. (c) Introducing-write labels denote a write turn, no introducing write, or cannot determine. 4.2 Reliability of SAGE Localization (RQ2) To answer RQ2, we assess the consistency of SAGE’s origin assignments across repeated scoring and independent judges, as well as the alignment of its artefact-derived outputs with human annotations across different task subsets. The repeated-scoring, cross-judge, and inter-annotator comparisons use the same stratified random subset of 60 cases. Table 6 summarizes these results. Repeated scoring suggests that SAGE reproduces the type of the recorded divergence more consistently than its exact turn. Table 6(a) shows that the primary judge assigned an origin in all three runs for most cases (57/60, 95.00%). Within this shared set, agreement was higher for origin type (55/57, 96.49%) than for the exact origin turn (50/57, 87.72%), suggesting that the nature of the recorded divergence was more consistently reproduced than its precise position. Allowing a one-turn difference increased agreement to 91.23% (52/57), covering two additional cases whose assigned origins fell on neighbouring turns. Krippendorff’s훼followed the same pattern, reaching 0.96 for origin type and 0.82 for origin turn. Figure 5(a) complements these results by showing consistent scoring across all four SAGE dimensions, with ordinal훼ranging from 0.89 to 0.95. Ordinal훼between the two judges was nevertheless lower than within the primary judge: 0.76 for threat recognition, 0.86 for defensive intent, and 0.68 for both verification rigour and scope awareness. The latter two dimensions also determine which turns are flagged as missed detection opportunities, so these flags may likewise be sensitive to the choice of judge. Cross-judge comparison showed the same contrast between origin type and exact turn, with a wider gap than under repeated scoring. Among cases localized by both judges (50/60, 83.33%), origin-type agreement (46/50, 92.00%) exceeded exact-turn agreement (39/50, 78.00%; Table 6(a)). Allowing a one-turn difference increased agreement to 84.00% (42/50), indicating that neighbouring turns accounted for some, but not all, of the temporal disagreements. The same pattern held across 14Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson all four ratings, comprising the primary judge’s three runs and the independent judge:훼was higher for origin type (0.92) than for origin turn (0.74). At the level of individual dimensions, Figure 5(b) shows that scores differed by at most one point in 90.00%–96.00% of jointly localized cases, suggesting broad agreement in the underlying assessments despite variation in exact turn selection. SAGE showed stronger agreement with the annotators on code provenance than on introducing- write labels. On the shared 60-case subset, agreement with each annotator ranged from 85.00% to 88.33% for provenance and from 78.33% to 85.00% for the introducing-write label. Table 6(c) places this alignment alongside the agreement between annotators, which was higher for provenance (54/60, 90.00%;휅=0.78) than for the introducing-write label (45/60, 75.00%;휅=0.58). This suggests that determining whether a vulnerability was inherited or introduced was more reproducible than identifying its introducing write. Where the annotators reached consensus, SAGE matched their labels in most cases, both for provenance (49/54, 90.74%) and for the introducing-write label (44/45, 97.78%). However, although overall alignment with the human reference remains good, agreement rates varied across task subsets, indicating that aggregate estimates may depend on sample composition. As shown in Table 6(b), across 1,000 random task-disjoint partitions, we computed subset-level agreement for code provenance (median 84.21%; central 95% of values: 74.42%–96.88%) and for the introducing write (median 81.06%; central 95% of values: 69.39%–93.33%). In the leave-one-task- out analysis, we recomputed agreement after excluding each task in turn, for code provenance (82.56%–89.87%) and for the introducing write (79.07%–87.34%). The highest values for both labels followed the exclusion of the task with the most cases (16 of 95), which accounted for a large share of the disagreements on code provenance (7 of 15) and on the introducing write (8 of 18). Overall, subset-level estimates appear to vary considerably with task composition, whereas no single task appears to inflate the aggregate estimate. Key Findings from RQ2: We found that SAGE identified the origin type more consistently than the exact origin turn, with lower agreement between judges than across repeated scoring. For the artefact-derived outputs, agreement with human annotators was higher for code provenance than for introducing-write labels and varied across task subsets. Together, these results capture different aspects of SAGE’s localization and are best interpreted jointly. 4.3 Variation and Boundary Conditions (RQ3) To answer RQ3, we analyse how localization reliability varies with the kind of confirmed deficiency, the agent framework, the base model, and the available artefact history. We compare SAGE with R1 on code provenance and the introducing-write label, and use repeated scoring and independent judges to assess the stability of the reasoning-based origin turn. Figure 6 and Table 7 summarize the observed differences. All subgroup comparisons are descriptive because the confirmed SF subsets differ in task, vulnerability, and trace composition. Estimates are particularly limited for CAMEL (푛= 4), OpenHands (푛= 9), Kimi K3 (푛= 5), and Qwen3.8-Max (푛= 1). SAGE aligned most with human references in absent-defence cases, but this advantage did not improve origin-turn stability. Figure 6(a) groups the 95 cases according to manual confirmation of the delivered code: an absent defence, an insufficient defence, or a vulnerability introduced by the change. This classification describes the confirmed silent failure and differs from the origin types reported in Section 4.1. For absent-defence cases, SAGE agreed with R1 on both provenance and the introducing-write label in 93.75% of cases (60/64). Agreement was notably lower for insufficient- defence cases, reaching 61.54% (16/26) for provenance and 53.85% (14/26) for the introducing-write label. The high agreement for absent defences largely reflects their inherited provenance, for which Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair15 Defence absent (n = 64) Defence insufficient (n = 26) Vulnerability introduced (n = 5) 0 25 50 75 100 Agreement with annotator (%) 60/6460/64 16/26 14/26 4/5 3/5 (a) By confirmed deficiency ProvenanceIntroducing writen < 10 Introducing write (vs annotator) Origin turn (three runs) Origin turn (two judges) 0 25 50 75 100 Agreement (%) 57/66 16/22 4/7 36/39 12/14 2/4 30/33 8/13 1/4 (b) By reconstruction condition Single-writePer-writeFinal-only (n < 10) Fig. 6. Agreement by confirmed deficiency and by reconstruction condition. Table 7. Agreement between the localization output and annotator on the 95 confirmed SF cases, by agent framework (a) and by base model (b). ∗ Groups with fewer than ten cases. (a) Agent frameworks Framework푛 Provenance Introducing write CAMEL4 ∗ 4 (100.00%)4 (100.00%) AG232 28 (87.50%)28 (87.50%) Deep Agents139 (69.23%)9 (69.23%) Aider22 18 (81.82%)18 (81.82%) MiniSWE-Agent 15 13 (86.67%)11 (73.33%) OpenHands9 ∗ 8 (88.89%)7 (77.78%) (b) Base models Base model푛 Provenance Introducing write Claude Opus 4.821 16 (76.19%)16 (76.19%) GPT-5.516 10 (62.50%)10 (62.50%) DeepSeek-V4-Pro 24 22 (91.67%)21 (87.50%) Gemini 3.1 Pro28 27 (96.43%)25 (89.29%) Kimi K35 ∗ 5 (100.00%)5 (100.00%) Qwen3.8-Max1 ∗ 0 (0.00%)0 (0.00%) no introducing write exists within the observed trace. However, the repeated scoring reveals a different pattern: all consistently localized insufficient-defence cases retained the same origin turn (16/16, 100.00%), while the assigned turn changed in six absent-defence cases. This contrast suggests that agreement with R1 on the artefact-derived outputs and origin-turn consistency under repeated scoring capture different properties of the localization. For agent frameworks, differences emerge both in human alignment and in the opportunities available to detect vulnerabilities down the trace. Table 7(a) shows that AG2 and Aider maintained identical within-framework agreement for provenance and introducing-write labels (87.50%, 28/32; and 81.82%, 18/22, respectively). However, for MiniSWE-Agent, provenance agreement (13/15, 86.67%) exceeded introducing-write agreement (11/15, 73.33%), indicating that consensus on code origin does not necessarily extend to the specific write operation. Interaction architectures were also associated with differences in the available subsequent evidence: AG2 traces frequently featured reviewer turns marked as missed detection opportunities (28/32, 87.50%), whereas the 22 single- message Aider cases provided no subsequent turns for evaluation. These differences suggest that an agent framework’s trace structure affects what evidence is available for localization. High repeatability in SAGE’s origin assignments did not always coincide with the human reference across base models. Among the larger groups in Table 7(b), SAGE achieved the highest 16Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson human agreement on traces generated by Gemini 3.1 Pro for both code provenance (27/28, 96.43%) and introducing-write labels (25/28, 89.29%), followed by DeepSeek-V4-Pro. Conversely, SAGE showed lower human agreement on GPT-5.5 agent traces (10/16, 62.50% for both labels); yet across all three repeated runs, SAGE assigned the same origin turn to every consistently localized GPT-5.5 case (10/10, 100.00%). This contrast illustrates that high internal consistency can coexist with disagreement over the underlying code evidence. Localization agreement was lowest when intermediate artefact states were unavailable. As shown in Figure 6, final-only traces, which provide no intermediate code states, had the lowest agreement across all three evaluations. In the human comparison, introducing-write agreement was 57.14% (4/7) for final-only traces, compared with 72.73% (16/22) for per-write traces and 86.36% (57/66) for single-write traces. The same pattern appeared for the reasoning-based origin turn. Among jointly localized cases, exact-turn agreement for final-only traces was 50.00% (2/4) across repeated scoring and 25.00% (1/4) across judges, compared with 92.31% (36/39) and 90.91% (30/33), respectively, for single-write traces. Although the final-only groups were small, their consistently lower agreement suggests that the absence of intermediate artefact states limits the evidence available for assigning a precise origin turn, which points to the value of preserving such states. Key Findings from RQ3: We observed that localization reliability varied with trace content, artefact granularity, and the nature of the confirmed silent failure. Final-only traces showed the lowest temporal agreement across all three evaluations, suggesting that intermediate artefact states provide useful evidence for assigning precise origin turns. Repeatability and human alignment also captured complementary aspects of reliability: stable origin assignments could coexist with lower agreement with the human reference. Finally, absent-defence cases showed higher agreement on artefact-derived outputs, while insufficient defences were associated with greater disagreement over code provenance and introducing-write labels. Differences across agent frameworks and base models were also observed, although these estimates reflect the composition of the confirmed-failure subsets within each group. 5 Discussion 5.1 Implications Final-code analysis and origin localization answer different questions (RQ1). The final patch shows that a vulnerability is present, and the artefact history adds its code provenance and, where one exists, the introducing write. Neither shows where the repair process first diverged from the task’s security intent. In 73 of the 95 cases (76.84%), the vulnerable code was inherited from the input file (see Table 5(a)): a diff reports no change at the vulnerable location, so code-level analysis has no position to report. SAGE nevertheless assigned an origin in 72 of these cases from the recorded reasoning (see Figure 4(a)). In the 19 agent-introduced cases, a diff does identify the introducing write, but in 14 of them (73.68%) the origin was an earlier turn, as an unaddressed requirement (8) or an inadequate defence choice (6) (see Table 5(c) and Figure 4). In these cases, the write marks where the vulnerability entered the file, whereas the divergence was already observable at an earlier turn. Code-level analysis would thus report an introducing write in 19 of the 95 cases, and this write was the assigned origin in only five, all from single-message traces with no earlier turn (see Table 5(b)). Origin type and exact origin turn should be evaluated separately (RQ2). Agreement on the origin type exceeded agreement on the exact origin turn under repeated scoring (96.49% against 87.72%) and between judges (92.00% against 78.00%) (see Table 6(a)). The two outputs depend on the scores in different ways. The origin type records which candidate of the rule was selected, whereas the exact turn also depends on the score assigned at each individual turn, so a small Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair17 score difference at one turn may move the origin while leaving its type unchanged. Figure 5(b) is consistent with this reading: the dimension scores of the two judges differed by at most one point in 90.00% to 96.00% of the jointly localized cases, yet the judges selected the same turn in 78.00%. A one-turn tolerance recovered only part of the difference, raising agreement to 91.23% and 84.00%, so some disagreements extend beyond neighbouring turns. A single localization score would hide this contrast. We therefore suggest reporting the origin type and the origin turn separately, and the turn both exactly and within one turn. The origin type may remain informative, for example about whether the divergence concerns requirement coverage or defence selection, when the exact turn is uncertain. These estimates should also be read alongside joint localization coverage, since agreement among jointly localized cases does not describe cases left undetermined in one or more assessments. Human agreement and repeated scoring measure different things (RQ2, RQ3). The two forms of evidence concern different outputs. Human annotation covers code provenance and the introducing write, which are derived from the artefact history and do not depend on the judge scores. Repeated scoring and cross-judge comparison cover the origin type and the origin turn, which do. Agreement with the annotators therefore indicates whether the artefact-derived outputs match a human reading of the code history, and repeated scoring indicates whether the judge-dependent outputs are stable; neither shows that a reasoning-level origin is correct. Our results show that the two can diverge. In the GPT-5.5 group, SAGE assigned the same origin turn in all three runs to each of the ten cases localized in every run, while agreement with R1 was 10/16 on both labels (see Table 7(b) and Section 4.3). Similarly, insufficient-defence cases had the lowest agreement with R1 (16/26 and 14/26; see Figure 6(a)) but an identical origin turn across runs in all 16 cases localized in every run (see Section 4.3). The human reference is also not fixed: the annotators agreed with each other less on the introducing write (휅=0.58) than on provenance (휅=0.78) (see Table 6(c)), and agreement with R1 varied across task-disjoint subsets (central 95% of values: 74.42%–96.88% for provenance; see Table 6(b)). High repeatability should therefore not be read as agreement with human judgement, nor the reverse; we suggest reporting both, each for the outputs it covers. Preserving intermediate artefact states matters for agent framework design (RQ3). Final-only traces had the lowest agreement in all three comparisons (see Figure 6(b)): 4/7 on the introducing write against R1, and 2/4 and 1/4 on the exact origin turn under repeated scoring and between judges. A likely reason is that, without intermediate snapshots, the write that introduced the vulnerable fragment cannot be identified from the artefact history, and the SAGE rule uses this write to bound the origin candidates. Missing states may thus weaken both the artefact-derived outputs and the anchor for the origin turn. Per-write traces, which preserve every intermediate state, still had lower agreement than single-write traces in all three comparisons, so part of the difference may reflect the number of writes in a trace and not only missing states. The final-only group is small (푛=7), but the direction was the same in comparisons that rest on different sources of evidence. For framework design, this suggests recording a snapshot after each write and keeping the boundaries of individual writes, so that reasoning turns can be related to code changes during an audit. Implications for future evaluation of agentic repair systems. Taken together, the results suggest four practices: (i) security evaluation should not stop at the delivered patch, since test passage and final-code checks show whether a repair is insecure, whereas a trace-based analysis adds where the divergence first became observable (see Table 5 and Figure 4); (i) the origin type and the exact origin turn should each be reported with its own consistency estimate (see Table 6(a)); (i) consistency and human agreement should both be reported and kept apart, since they cover different outputs (see Tables 6 and 7); and (iv) results should be reported by reconstruction condition and by subgroup, with small groups given as counts, since aggregate agreement can hide differences 18Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson in the available evidence (see Figure 6 and Table 7). Later turns suggest a further direction. At least one missed detection opportunity was flagged in 59 of the 95 cases (see Table 5(d)), as in the sorting-repair case in which the review did not address the remaining unsafe configuration (see Section 4.1). This is a pattern observed in our corpus and not a general conclusion about reviewer agents; whether an explicit security requirement check in review turns reduces silent failures, and at what cost, remains to be evaluated in future work. SAGE’s origin assignments can thus inform hypotheses about where to intervene, but whether such interventions prevent silent failures requires controlled intervention studies beyond the present observational analysis. 5.2 Threats to Validity Construct validity. The origin turn is an operational construct: the earliest observable, trace- attributable divergence under the SAGE rule. It does not establish that the selected turn caused the vulnerability or that intervening there would have prevented it, and incomplete records may conceal earlier divergence. A zero score means that no relevant reasoning was recorded, not that the agent had none, and a missed detection opportunity marks a low-scoring later turn, not a turn at which detection would have succeeded. Human annotation covers only the artefact-derived outputs; origin type and turn are assessed through repeated scoring and cross-judge comparison, which measure consistency rather than correctness. As the annotators agreed less on the introducing write (휅=0.58) than on provenance (휅=0.78), we report agreement with each annotator separately and treat consensus-subset results as conditional. Internal validity. LLM-judge scoring may introduce run-to-run variation, dependence on the judge model, or hindsight bias. We addressed these risks in five ways: prefix-only scoring; a verbatim quotation for every non-zero score; three scoring runs by the primary judge on a 60-case subset; a second judge from a different model family on the same subset; and comparison of the artefact-derived outputs with two annotators who did not see the SAGE outputs (see Table 6). These measures reduce but do not remove the threat: the two judges jointly localized 50 of the 60 cases with 78.00% exact-turn agreement, and biases shared by both judges, inherited from the rubric, or arising from family overlap with the evaluated base models (the primary judge with GPT-5.5 and the second judge with Claude Opus 4.8) cannot be excluded. The annotators used the same reconstructed histories as SAGE, so shared reconstruction errors may inflate agreement, and R1 took part in corpus construction and rule design; we therefore report agreement with R2 separately. As confirmed cases concentrate in few tasks, we assessed sensitivity to task composition with task-disjoint resampling (see Section 4.2). External validity. The evaluation covers 95 confirmed silent failures from Python repair tasks in two security-focused benchmarks, six agent frameworks, and six base models, and may not generalize to other languages, task types, or deployment settings. The corpus is bounded by candidate screening: vulnerable patches flagged by neither static analysis nor exploitability-oriented screening do not enter manual review. The confirmed-failure subsets differ in task, vulnerability, and trace composition, so cross-framework and cross-model differences are descriptive and do not estimate framework- or model-level effects. Estimates for groups with fewer than ten cases (see Table 7; Figure 6), including final-only traces, are particularly uncertain, which also qualifies the artefact-preservation implication in Section 5.1. 6 Conclusion This paper presented SAGE, a trace-based method for localizing the origin of a confirmed silent failure in agentic vulnerability repair. The results show that a divergence from the task’s security intent can become observable in the agent’s plans and explanations before the vulnerable code is written, and that for inherited vulnerabilities it can be localized even when no code change marks Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair19 it. Final-patch checks therefore show that a repair is insecure, but not where the repair process first diverged. The results also show that the origin type is reproduced more consistently than the exact origin turn, and that judge consistency and agreement with human annotators cover different outputs, so both should be reported. Agreement was lowest for traces that kept only the final file, which suggests that agent frameworks should preserve intermediate code states. Future work will extend the analysis to other programming languages and task types, and will examine through controlled studies whether safeguards placed at the identified origins prevent silent failures. Data Availability The dataset is available at [5]. References [1] Alibaba Group. 2026. Alibaba Unveils Qwen3.8-Max: Its Largest and Most Capable Flagship Model to Date. https: //w.alibabagroup.com/document-2021044032125272064. Published 2026-08-03. Accessed: 2026-09-14. [2]Anthropic. 2026. System Card: Claude Opus 4.8. https://w.anthropic.com/claude-opus-4-8-system-card. Published 2026-05-28. Accessed: 2026-09-14. [3] Anthropic. 2026. System Card: Claude Opus 5. https://w.anthropic.com/claude-opus-5-system-card. Published 2026-07-24. Accessed: 2026-09-14. [4]Owura Asare, Meiyappan Nagappan, and N Asokan. 2023. Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering 28 (2023), 129. [5] Anonymous Authors. 2026. Dataset for the Paper: Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair. doi:10.5281/zenodo.23108850 [6] Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson. 2026. When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems. arXiv preprint arXiv:2609.10548 (2026). [7]Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE). 30–39. [8]Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. RepairAgent: An autonomous, LLM-based agent for program repair. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). 2188–2200. [9] Islem Bouzenia and Michael Pradel. 2025. Understanding software engineering agents: A study of thought-action-result trajectories. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). [10] Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail?. In Proceedings of the 39th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS). [11] Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, and Qing Wang. 2026. Seeing the whole elephant: A benchmark for failure attribution in LLM-based multi-agent systems. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL). [12] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. [13]DeepSeek-AI. 2026. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348 (2026). [14] Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. 2025. TRAIL: Trace reasoning and agentic issue localization. arXiv preprint arXiv:2505.08638 (2025). [15]Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2025. Vulnerability detection with code language models: How far are we?. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). 469–481. [16]Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, and Jinfu Chen. 2025. Security weaknesses of Copilot-generated code in GitHub projects: An empirical study. ACM Transactions on Software Engineering and Methodology 34, 8 (2025), 1–34. [17] Paul Gauthier. 2024. Aider: AI Pair Programming in Your Terminal. https://aider.chat. Accessed: 2026-06-17. [18]Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang. 2026. Spectrum-Based Failure Attribution for Multi-agent Systems. Proceedings of the ACM on Software Engineering 3, FSE (2026), 1888–1909. 20Wenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, and Pekka Abrahamsson [19]Google DeepMind. 2026. Gemini 3.1 Pro Model Card. https://deepmind.google/models/model-cards/gemini-3-1-pro/. Published 2026-02-19. Accessed: 2026-09-14. [20] Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. Brit. J. Math. Statist. Psych. 61, 1 (2008), 29–48. [21]Andrew F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures 1, 1 (2007), 77–89. [22] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world GitHub Issues?. In Proceedings of the 12th International Conference on Learning Representations (ICLR). [23]LangChain. 2026. Deep Agents: An extensible agent harness with planning, file-system context management, and subagent spawning. https://github.com/langchain-ai/deepagents. Version 0.7.12. Accessed: 2026-09-14. [24] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for “mind” exploration of large language model society. Advances in Neural Information Processing Systems 36 (2023), 51991–52008. [25]Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, and Huazheng Wang. 2026. Who&When Pro: Can LLMs really attribute failures in AI agents? arXiv preprint arXiv:2607.09996 (2026). [26]Vahid Majdinasab, Michael Joshua Bishop, Shawn Rasheed, Arghavan Moradidakhel, Amjed Tahir, and Foutse Khomh. 2024. Assessing the security of GitHub Copilot’s generated code: A targeted replication study. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 435–444. [27]Moonshot AI. 2026. Kimi K3: Open Frontier Intelligence. https://github.com/MoonshotAI/Kimi-K3. Published 2026-07-27. Accessed: 2026-09-14. [28] OpenAI. 2026. GPT-5.5 System Card. https://openai.com/index/gpt-5-5-system-card/. Published 2026-04-23. Accessed: 2026-09-14. [29]OpenAI. 2026. GPT-5.6 System Card. https://deploymentsafety.openai.com/gpt-5-6. Published 2026-07-09. Accessed: 2026-09-14. [30]Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768. [31] Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero- shot vulnerability repair with large language models. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356. [32]Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do users write more insecure code with AI assistants?. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS). 2785–2799. [33]Mohammed Latif Siddiq and Joanna CS Santos. 2022. SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security (MSR4P&S). 29–33. [34] Edward K Smith, Earl T Barr, Claire Le Goues, and Yuriy Brun. 2015. Is the cure worse than the disease? Overfitting in automated program repair. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE). 532–543. [35] Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating code generation with large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). 20032–20051. [36]Catherine Tony, Markus Mutas, Nicolás E Díaz Ferreyra, and Riccardo Scandariato. 2023. LLMSecEval: A dataset of natural language prompts for security evaluations. In Proceedings of the 20th IEEE/ACM International Conference on Mining Software Repositories (MSR). 588–592. [37]Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. 2024. LLMs cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 862–880. [38]Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, and Martin Vechev. 2025. BaxBench: Can LLMs generate correct and secure backends?. In Proceedings of the 42nd International Conference on Machine Learning (ICML). [39]Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can LLMs replace human evaluators? An empirical study of LLM-as-a-judge in software engineering. Proceedings of the ACM on Software Engineering 2, ISSTA (2025). [40]Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al.2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In Proceedings of the 13th International Conference on Learning Representations (ICLR). Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair21 [41]You Wang, Michael Pradel, and Zhongxin Liu. 2026. Are “solved issues” in SWE-bench really solved correctly? An empirical study. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE). [42] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al.2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In Proceedings of the 1st Conference on Language Modeling (COLM). [43]Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. 2023. How effective are neural networks for fixing security vulnerabilities. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 1282–1294. [44]Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824. [45]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [46]Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. 2026. AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?. In Proceedings of the 14th International Conference on Learning Representations (ICLR). [47] Jiayi Zhang, Zexin Wang, Degang Sun, Changhua Pei, Fei Sun, Gaogang Xie, and Jingjing Li. 2026. Detect before you attribute: Cascade failure attribution for multi-agent systems. arXiv preprint arXiv:2608.29646 (2026). [48]Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. 2025. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. In Proceedings of the 42nd International Conference on Machine Learning (ICML). 76583–76599. [49]Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 1592–1604. [50]Songwen Zhao, Danqing Wang, Kexun Zhang, Jiaxuan Luo, Zhuo Li, and Lei Li. 2026. Is Vibe Coding Safe? Benchmark- ing Vulnerability of Agent-Generated Code in Real-World Tasks. In Proceedings of the 43rd International Conference on Machine Learning (ICML). [51] Zixiao Zhao, Amirreza Esmaeili, and Fatemeh Fard. 2026. Bias in the loop: Auditing LLM-as-a-judge for software engineering. arXiv preprint arXiv:2604.16790 (2026). [52]Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo. 2024. Leveraging large language model for automatic patch correctness assessment. IEEE Transactions on Software Engineering 50, 11 (2024), 2865–2883. [53]Kunlun Zhu, Zijia Liu, Bingxuan Li, et al.2025. Where LLM agents fail and how they can learn from failures. arXiv preprint arXiv:2509.25370 (2025). [54]Mingchen Zhuge, Changsheng Zhao, Dylan R Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al.2025. Agent-as-a-judge: Evaluate agents with agents. In Proceedings of the 42nd International Conference on Machine Learning (ICML).