Paper deep dive
UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents
Dolly Sah, Tanmay Sah, Harshul Jain, Tanya Sah
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery. We introduce UndoBench, a benchmark spanning 36 base workflows and 36 fault scenarios across 8 enterprise domains, decoupling task competence from recovery capability via counterfactual paired trials under identical seeds alongside wire-level effect-history and environment-state oracles. On 12 held-out TEST workflows across two open-weight models, two frameworks, and three recovery paradigms (5,760 executions / 2,880 paired trials) in the frozen lost-acknowledgment study, nominal competence reached 83.54% while conditional recovery success rate (CRSR) fell to 46.72%, with naive retry producing duplicate external effects in 53.33% of trials. Extensions to commercial API models reproduced this competence-recovery separation. Evaluations across complementary execution boundaries show that recovery is phase-dependent: before mutation, methods perform similarly without duplicate effects among capable trials; during partial mutation, naive retry, per-call idempotency, and zero-privilege journaling collapse on the evaluated composite workflows; after commit but before acknowledgment, verification and server-side idempotency substantially improve safety. These findings demonstrate that evaluating nominal completion alone masks critical, phase-dependent recovery vulnerabilities in autonomous agents.
Tags
Links
- Source: https://arxiv.org/abs/2610.05622v1
- Canonical: https://arxiv.org/abs/2610.05622v1
Trouble viewing inline? Open PDF directly →
Full Text
75,323 characters extracted from source content.
Expand or collapse full text
UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents Dolly Sah* Tanmay Sah* Harshul Jain* Tanya Sah* Affiliation: *Independent Researcher || Code Email: dolly17sah@gmail.com tradertanmay@gmail.com Email: harshuljain1393@gmail.com tanyasah20@gmail.com Abstract Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery. We introduce UndoBench, a benchmark spanning 36 base workflows and 36 fault scenarios across 8 enterprise domains, decoupling task competence from recovery capability via counterfactual paired trials under identical seeds alongside wire-level effect-history and environment-state oracles. On 12 held-out TEST workflows across two open-weight models, two frameworks, and three recovery paradigms (5,760 executions / 2,880 paired trials) in the frozen lost-acknowledgment study, nominal competence reached 83.54% while conditional recovery success rate (CRSR) fell to 46.72%, with naive retry producing duplicate external effects in 53.33% of trials. Extensions to commercial API models reproduced this competence–recovery separation. Evaluations across complementary execution boundaries show that recovery is phase-dependent: before mutation, methods perform similarly without duplicate effects among capable trials; during partial mutation, naive retry, per-call idempotency, and zero-privilege journaling collapse on the evaluated composite workflows; after commit but before acknowledgment, verification and server-side idempotency substantially improve safety. These findings demonstrate that evaluating nominal completion alone masks critical, phase-dependent recovery vulnerabilities in autonomous agents. 1 Introduction Autonomous software agents driven by large language models (LLMs) have advanced from static text generation to interactive tool execution across heterogeneous computing environments. Today, LLM agents inspect source code repositories, execute database migrations, query customer relationship management (CRM) platforms, provision cloud infrastructure, and initiate cross-border wire transfers. Despite rapid advances in agent planning and tool-calling capabilities, a profound discrepancy separates laboratory benchmark evaluations from production reliability. In real-world distributed systems, software environments do not behave as static sandboxes. Distributed networks experience transient latency and dropped packets; upstream microservices return rate limits and unexpected internal server errors; and database writes trigger unique-key conflicts. Crucially, operations frequently suffer from lost acknowledgments—a fundamental distributed systems hazard where an external mutation succeeds on a remote server, but the network connection drops before the client receives the confirmation. When human software engineers encounter these failures, they rely on established fault-tolerance patterns such as idempotency, write-ahead logging, and compensating transactions (Sagas) (Garcia-Molina and Salem, 1987; Helland, 2012; Mohan et al., 1992). In contrast, modern AI agent architectures typically treat tool execution as an uninspected request-response primitive. When a tool raises an exception, the agent runtime either crashes or passes the error string back into the prompt context, relying on naive retry loops or in-context self-reflection mechanisms such as Reflexion (Shinn et al., 2023) to resolve the issue. Existing agent benchmarks—such as SWE-bench (Jimenez et al., 2024), ToolBench (Qin et al., 2024), API-Bank (Li et al., 2023), AgentBench (Liu et al., 2024), and WebArena (Zhou et al., 2024)—fail to surface or quantify these vulnerabilities. These benchmarks evaluate agents almost exclusively under nominal, unperturbed conditions. These task-oriented benchmarks generally do not distinguish nominal planning failures from recovery failures induced by operational execution faults. Current evaluation paradigms provide no mechanism to answer a critical operational question: If an agent possesses the competence to solve a workflow nominally, can it safely and correctly recover when an external system fault occurs mid-execution? Consider a financial agent executing a subscription renewal: process_renewal(sub_id="sub_99", amount=1200). If the remote payment gateway commits the charge but the response packet drops, the agent catches a connection timeout. Lacking knowledge of whether the mutation committed, a standard retry loop immediately re-issues the tool call, charging the account a second time and causing a duplicate billing violation. Traditional benchmarks cannot detect this failure because they never inject acknowledgment loss and never inspect external wire-level side effects. Furthermore, reporting a single end-to-end task success metric conflates two distinct phenomena: (1) Task Competence: whether the model can coordinate tools to accomplish a goal under nominal conditions; and (2) Recovery Capability: whether the agent can diagnose, reconcile, and safely repair execution state when a fault strikes. Research Question. How does recovery behavior change as a fault moves across the external-mutation lifecycle? In distributed architectures, execution proceeds through distinct phases: before remote state modification begins (PRE_MUTATION), mid-execution during active non-atomic state updates (DURING_MUTATION), and after remote commit while awaiting confirmation (POST_MUTATION_PRE_ACK). While UndoBench’s taxonomy formalizes additional execution boundaries, our secondary evaluations prospectively cover these three foundational phases to determine whether agent recovery constitutes a single scalar capability or a phase-dependent systems property. To address these foundational challenges, we introduce UndoBench, a benchmark designed to formally separate multi-turn task competence from recovery capability across the external-mutation lifecycle. This paper provides four core contributions: 1. Multi-Boundary Benchmark Formulation: A suite of 36 base workflows and 36 fault scenarios across 8 enterprise domains, evaluating agent recovery across distinct execution boundaries relative to external state mutation. 2. Decoupled Paired Evaluation & CRSR: A counterfactual evaluation paradigm pairing every faulted run with an identical-seed control execution, defining Conditional Recovery Success Rate (CRSR) to evaluate recovery conditional on paired nominal task success. 3. Dual State and Effect-History Oracles: Programmatic oracles verifying physical environment invariant satisfaction alongside wire-level effect monitors that quantify duplicate (DER), missing (MER), and exactly-once (EOR) execution safety. 4. Phase-Dependent Empirical Characterization: A frozen confirmatory lost-ACK study (5,760 executions / 2,880 paired trials) coupled with prospectively specified post-freeze secondary extensions across contemporary models, complementary execution boundaries, and benchmark-wide idempotency interventions. 2 UndoBench Design Figure 1: UndoBench Decoupled Evaluation Architecture. Every workflow is evaluated as a paired counterfactual execution under identical seeds, separating nominal planning competence (CiC_i) from fault recovery capability (FiF_i). 2.1 Problem Formulation and Decoupling Let T be a distribution over tool-augmented tasks, and let A be an agent system. Under standard benchmarking, the evaluator measures nominal task competence, Comp=P(Success∣Nominal)Comp=P(Success ), operationalized as the baseline Control pass rate. When faults are injected without counterfactual controls, the evaluator measures the unconditional recovery success rate, RSR=P(Success∣Fault)RSR=P(Success ). By the law of total probability, RSR=P(Success∣Fault,Comp)P(Comp)+P(Success∣Fault,¬Comp)P(¬Comp)RSR=P(Success ,Comp)P(Comp)+P(Success , )P( ). When nominal competence is low, unconditional RSR conflates nominal task failure with recovery inability. To isolate recovery conditioned on nominal competence, UndoBench defines: CRSR=P(Success∣Fault,Comp)CRSR=P(Success ,Comp) (1) UndoBench operationalizes this counterfactual separation through paired execution trials under identical seeds (Figure 1). 2.2 Fault and External-Effect Model We formalize a tool-augmented agent as an interactive policy operating over environment state S∈S . Each tool invocation at=tool(k1=v1,…)a_t=tool(k_1=v_1,…) produces an Agent-Visible Return payload rt∈∪⊥r_t ∪\ \ returned to the agent context (where ⊥ denotes an error), alongside an External Side Effect state transition τ:×→τ:S×A mutating remote resources (e.g., database rows, Git trees, balances). In distributed systems, the state transition τ(St,at)→St+1τ(S_t,a_t)→ S_t+1 may commit successfully on the remote service even if the acknowledgment conveying rtr_t is dropped by the network. UndoBench formalizes a multi-dimensional failure taxonomy spanning five canonical execution boundaries: (1) PRE_MUTATION, pre-execution transport failure before remote initiation (e.g., connection timeout, DNS error); (2) DURING_MUTATION, failure mid-execution during active mutation (e.g., unique-key conflict, concurrency abort); (3) POST_MUTATION_PRE_ACK, remote mutation commits, but transport drops before client ACK delivery, creating external state ambiguity; (4) POST_ACK_PRE_CHECKPOINT, client crashes after receiving ACK but before durable checkpoint persistence; and (5) DURING_COMPENSATION, fault occurs during compensatory rollback execution. In the frozen primary confirmatory evaluation, fault injection concentrates on the critical POST_MUTATION_PRE_ACK transport window. Prospectively specified post-freeze secondary extensions subsequently evaluate complementary boundaries, specifically PRE_MUTATION and DURING_MUTATION on qualified workflows. Tool effects are classified by physical reversibility: REVERSIBLE, COMPENSATABLE, and IRREVERSIBLE (telemetry and overhead detailed in Appendix E). 2.3 Physical Invocations, Committed Mutations, and Semantic Effects A primary source of evaluation ambiguity in tool-augmented agents is the conflation of physical transport actions with durable system effects. UndoBench explicitly distinguishes three hierarchical notions: (1) Physical Invocation (ata_t), the wire-level dispatch of an API request or remote procedure call across the network boundary, creating transient transport state; (2) Committed Mutation (τ(St,at)τ(S_t,a_t)), the durable, state-altering execution of the dispatched operation on the remote service (e.g., writing a database record, committing a Git tree, charging a ledger); and (3) Semantic Effect, the higher-level business invariant demanded by the task specification (e.g., reconciling an account balance, deploying a canary release, notifying an on-call engineer). Without execution-level instrumentation, terminal-reward evaluations cannot distinguish whether a fault dropped before, during, or after remote commit, leaving duplicate real-world side effects unobserved. When benchmarks only inspect final string answers, an agent that blindly retries a committed mutation may pass nominal assertions despite inducing duplicate real-world effects, while an agent that safely halts may be marked as failed. By monitoring wire-level physical invocations alongside programmatic state oracles, UndoBench exposes whether recovery preserves exactly-once semantic effects without corrupting external environments. 2.4 Paired Evaluation and Metrics For N paired trials, the oracle records binary indicators Ci∈0,1C_i∈\0,1\ (nominal control success) and Fi∈0,1F_i∈\0,1\ (fault recovery success). With D=∑i=1NCi>0D= _i=1^NC_i>0 competent trials, benchmark metrics are defined as: Ctrl =DN,RSR=1N∑i=1NFi, = DN, = 1N _i=1^NF_i, CRSR =1D∑i=1NCiFi = 1D _i=1^NC_iF_i (2) To evaluate execution safety, programmatic oracles track (dual-oracle instrumentation in Appendix K; safety metrics and validation in Appendix C): • Exactly-Once Semantic Effect Rate (EOR): Fraction of competent trials satisfying task goal invariants with all required semantic effects and zero duplicates: EORcond=1D∑i:Ci=1(Fi=1∧dupi=0∧missi=0)EOR_cond= 1D _i:C_i=1I(F_i=1 _i=0 _i=0). In the frozen TEST regime, the state oracle requires successful goal completion with zero duplicate and zero missing effects; consequently EORcondEOR_cond and CRSR coincide numerically in the reported experiments, although they are conceptually distinct metrics. • Duplicate Effect Rate (DER): External datastore metric measuring trials producing duplicate committed mutations on external services: DER=1N∑i=1N(dupi>0)DER= 1N _i=1^NI(dup_i>0). • Unsafe Retry Rate (URR): The fraction of trials in which retry behavior after an unacknowledged failure results in an unsafe duplicate external effect: URR=1N∑i=1N(outcomei=UNSAFE_RETRY)URR= 1N _i=1^NI(outcome_i= UNSAFE\_RETRY). In the frozen POST_MUTATION_PRE_ACK evaluation, UNSAFE_RETRY is assigned precisely when retry behavior produces a duplicate external effect (outcomei=UNSAFE_RETRY⇔dupi>0outcome_i= UNSAFE\_RETRY _i>0); therefore URR and DER coincide numerically. Deduplicated idempotency-protected redispatches are not classified as unsafe retries. • Missing Effect Rate (MER): Fraction of trials terminating without committing all required semantic effects (missi>0miss_i>0). 3 Experimental Setup Task Suite and Factorial Design. UndoBench comprises 36 base workflows across 8 domains: Cloud, CRM, Database, Git, Messaging, Payments, Storage, and Ticketing (factorial matrix detailed in Table 5 in Appendix A). The benchmark is partitioned into 14 DEV tasks, 10 VALIDATION tasks, and 12 held-out TEST tasks (Appendix G). The primary confirmatory study is frozen on the 12 held-out TEST workflows under POST_MUTATION_PRE_ACK across two open-weight models (Llama-3.1-8B, Llama-3.2-3B), two frameworks (F1,F2F_1,F_2), and three recovery baselines (B0,B2,B5B_0,B_2,B_5) across 20 seeds (2,880 paired trials / 5,760 executions; hardware details in Appendix H). To investigate fault timing and mechanism isolation without altering frozen datasets, prospectively specified post-freeze secondary extensions evaluate commercial API models (Gemini 3.8 Flash, GLM-5.2 MaaS), complementary execution boundaries (PRE_MUTATION, DURING_MUTATION), and benchmark-wide idempotency (B2-KB_2-K). Models and Frameworks. We evaluate two open-weight instruction-tuned language models locally under sampling temperature T=0.2T=0.2, nucleus sampling top-p=1.0p=1.0, and maximum 512 generation tokens per turn: (1) M1: Llama-3.1-8B-Instruct (8.0B parameters, Q4_K_M); and (2) M2: Llama-3.2-3B-Instruct (3.2B parameters, Q4_K_M). We evaluate two runtime agent execution architectures: (1) F1: Direct Tool Calling, lightweight tool dispatch from JSON blocks; and (2) F2: LangGraph Agent, state-graph runtime managing cyclical agent loops. Recovery Baselines. We evaluate three primary recovery paradigms: • B0B_0 Naive Retry: Industry baseline re-issuing tool calls up to 3 times with exponential backoff upon receiving any error. Model provider and transport API retries are handled orthogonally by HTTP middleware and do not consume agent turns or trigger benchmark faults; retry sensitivity is future work. • B2B_2 Idempotency: Mutating calls carry deterministic idempotency keys derived from task context and call arguments (Appendix A). When supported by server endpoints, keys are cached to deduplicate executions on replays. • B5B_5 EvoUndo-RB1: Studies recoverability-constrained self-evolution for LLM agent harnesses (Sah et al., 2026a). EvoUndo-RB1 is the UndoBench-constrained implementation evaluated here, using a client-side mutation journal and zero-privilege reconciliation (no access to sandbox internals, state diffs, or oracle hooks). Implemented in evoundo_adapter, it hashes tool arguments (SHA-256) into a local mutation journal and invokes a crash reconciler on failure. Within UndoBench, EvoUndo-RB1 intentionally receives zero privilege, no remote probes, and no registered inverse handlers or domain schemas; unable to verify whether unacknowledged mutations committed or synthesize compensations, it defaults to bounded retry across the 9 compensatable TEST workflows. Thus, EvoUndo-RB1 reflects these benchmark constraints and should not be interpreted as unrestricted EvoUndo. Statistical Analysis. Confidence intervals are estimated via non-parametric task-clustered bootstrap resampling (B=10,000B=10,000 iterations) alongside Wilson score intervals. Multiple comparisons are adjusted via Holm-Bonferroni correction. Framework equivalence is tested using Two One-Sided Tests (TOST) against an equivalence margin δ=±10.0δ=± 10.0 percentage points (Appendix F). 4 Results 4.1 Competence vs. Recovery Figure 2: Separation of Task Competence from Fault Recovery. Across 2,880 paired trials, nominal competence (83.54%) sharply separates from conditional recovery (46.72% CRSR) and unconditioned recovery (39.03% RSR). Table 1 and Figure 2 demonstrate that baseline task competence and fault recovery capability diverge sharply across all evaluated configurations. This establishes Finding 1: Finding 1 (Task Competence and Recovery Capability Are Distinct Dimensions): Nominal task success reached 83.54% (2,406 / 2,880), compared with 39.03% (1,124 / 2,880) end-to-end recovery success (RSR) and 46.72% (1,124 / 2,406) conditional recovery success (CRSR) under injected faults. Nominal task success alone substantially overstates faulted reliability in tool-using agents. 4.2 Recovery-Method Comparison Table 1: Comparative Recovery Performance and Safety Profiles (TEST Split, N=2,880N=2,880 Paired Trials). Method Control CRSR EORcond_cond DER URR (D/960D/960) (R/DR/D) (/D/D) (/960/960) (/960/960) B0B_0: Naive Retry 802 / 960 42.77% 42.77% 53.33% 53.33% B2B_2: Idempotency 802 / 960 53.49% 53.49% 45.42% 45.42% B5B_5: EvoUndo-RB1 802 / 960 43.89% 43.89% 51.67% 51.67% Contrast B2−B0B_2-B_0 — +10.72 p +10.72 p -7.91 p -7.91 p Contrast B5−B0B_5-B_0 — +1.12 p +1.12 p -1.66 p -1.66 p CRSR and EORcondEOR_cond are conditioned on nominal-control success (D=802D=802); DER and URR are unconditional over all 960 paired trials per method. Table 1 presents the comparative evaluation of recovery baselines across all 2,880 paired trials: • Contrast H1 (B2B_2 vs. B0B_0): Idempotency (B2B_2) produced an observed descriptive advantage of +10.72 p (53.49% vs. 42.77%; bootstrap 95% CI: [0.00 p,+24.23 p][0.00 p,+24.23 p], padj=1.00p_adj=1.00), concentrated in lost-ACK workflows rather than uniform across domains. While B2B_2 attaches keys to all mutating operations, deduplication requires server support: only three TEST endpoints implement idempotency tracking (RB-PAY-004, RB-PAY-005, RB-MSG-005; 100% CRSR, 0.0% DER). Three workflows (RB-CLOUD-004, RB-DB-004, RB-GIT-004) achieved 0.0% DER via inherent state overwrites or datastore constraints. The remaining six endpoints lack deduplication; retrying unacknowledged calls blindly produced duplicates in 100% of trials across five workflows (400 trials) and 45.0% on RB-STOR-005 (36 trials), accounting for all 436 duplicate trials (436/960=45.42%436/960=45.42\% aggregate DER). • Contrast H2 (B5B_5 vs. B0B_0): Under zero privilege, EvoUndo (B5B_5) performed near parity with naive retry (Δ=+1.12 p =+1.12 p, bootstrap 95% CI: [−3.15 p,+6.31 p][-3.15 p,+6.31 p], padj=1.00p_adj=1.00; DER=51.67%DER=51.67\%). Lacking remote unacknowledged state visibility and registered compensating inverse functions, its local journal cannot confirm whether an in-flight mutation committed, leaving it subject to post-commit ambiguity. • Safety Decomposition: Exactly-once rate (EOR) numerically coincides with CRSR (46.72% overall) because the oracle strictly requires goal invariants without duplicates. Under naive retry (B0B_0), the Duplicate Effect Rate reached 53.33% (512 / 960). Furthermore, URR and DER coincide numerically across all evaluated baselines because in the post-commit ACK loss regime, every unsafe retry on an unkeyed endpoint commits a duplicate mutation, and every duplicate mutation originated from an unsafe retry. 4.3 Lost-ACK / UNKNOWN_OUTCOME Analysis Table 2: Representative Task Outcomes under Lost ACK. Task ID Domain / Operation Control B0B_0 CRSR B2B_2 CRSR RB-PAY-004 Subscription Renewal 80 / 80 40.0% 100.0% RB-PAY-005 Wire Transfer 80 / 80 65.0% 100.0% RB-DB-004 Schema Migration 48 / 80 52.1% 72.9% RB-CRM-004 SLA Renewal 80 / 80 0.0% 0.0% RB-STOR-005 Chunk Swap 0 / 80 N/A∗ N/A∗ ∗On RB-STOR-005, nominal control was 0/240 (D=0D=0) due to chunk verification complexity; CRSR is mathematically undefined (0/00/0) and presented as N/A. Retained in aggregate Control and RSR. Table 2 (visualized in Figure 4 in Appendix B) examines performance under post-mutation acknowledgment loss (POST_MUTATION_PRE_ACK). This granular analysis establishes Finding 2 and Finding 3: Finding 2 (Ambiguous Committed Effects Expose Distinct Regimes): When external acknowledgment is lost, naive retry produced duplicate side effects whose severity depends on operation semantics. Under B0B_0, duplicate executions occurred in 60.0% of trials on RB-PAY-004 and 35.0% on RB-PAY-005, but dropped to 0.0% on RB-DB-004 because SQLite’s schema unique constraint blocked the second migration attempt. Across the five workflows lacking idempotency or database constraints, naive retry produced 100.0% (400 / 400) duplicate executions. Finding 3 (Mechanism–Failure Matching): The B2B_2 configuration produced large task-level gains on lost-ACK mutation workflows with deduplication or constraint protection (+60.0 p on RB-PAY-004 and +35.0 p on RB-PAY-005 via application-level idempotency-key deduplication; +20.8 p on RB-DB-004 via a datastore uniqueness constraint; therefore the observed B2−B0B_2-B_0 difference on RB-DB-004 should not be attributed solely to idempotency-key deduplication), while its overall +10.72 p TEST CRSR advantage did not establish benchmark-wide statistical superiority under task-clustered bootstrap uncertainty due to endpoints lacking idempotency support. 4.4 Model and Framework Analysis Model Comparison on Common Support. Across the 10 TEST tasks supported by both models (D≥10D≥ 10), M1 (Llama-3.1-8B) achieved 44.50% CRSR (445 / 1,000), while M2 (Llama-3.2-3B) achieved 54.17% CRSR (547 / 1,010) (Δ=−9.67 p =-9.67 p). Under task-clustered bootstrap analysis, the 95% CI spans [−27.08 p,+7.73 p][-27.08 p,+7.73 p] (p=1.00p=1.00), spanning zero and remaining statistically unresolved. Framework Equivalence. Direct Tool Calling (F1) and LangGraph (F2) achieved near-identical outcomes: Control was 83.54% vs. 83.54% (Δ=0.00 p =0.00 p), RSR was 39.24% vs. 38.82% (Δ=+0.42 p =+0.42 p), and CRSR was 46.97% vs. 46.47% (Δ=+0.50 p =+0.50 p). Two One-Sided Tests (TOST) against the ±10.0± 10.0 p margin confirm statistical equivalence (pTOST<0.001p_TOST<0.001): the two evaluated adapters were empirically equivalent within the ±10± 10 p equivalence margin under the standardized UndoBench contract. 4.5 Commercial API Model Extension To investigate whether recent model architectures alter the observed competence–recovery relationship, we executed a prospectively specified post-freeze secondary extension evaluating two commercial API models: Google’s Gemini 3.8 Flash (gemini-3.8-flash, M3M_3) and Z.ai’s GLM-5.2 MaaS (zai-org/glm-5.2-maas, M4M_4). Both models were interfaced via Google Cloud Vertex AI infrastructure across the identical 12 held-out TEST workflows, two orchestration frameworks (F1F_1 Direct Tool Calling, F2F_2 LangGraph), three recovery baselines (B0B_0 Naive Retry, B2B_2 Idempotency, B5B_5 EvoUndo), and 20 evaluation seeds (2001–2020), yielding N=2,880N=2,880 paired trials (5,760 executions; cell breakdown in Appendix J). Table 3: Commercial API Model Extension Performance. Evaluated on 12 held-out TEST workflows (N=2,880N=2,880 paired trials / 5,760 runs) under identical fault injection across seeds 2001–2020. Model Control B0B_0 CRSR B0B_0 DER B2B_2 CRSR B5B_5 CRSR Gemini 3.8 Flash 80.69% 21.08% 65.62% 33.33% 22.11% GLM-5.2 MaaS 77.71% 21.89% 51.25% 33.07% 21.66% Pooled 79.20% 21.48% 58.44% 33.20% 21.89% Control is nominal pass rate (N=1,440N=1,440 per model). CRSR is conditioned on nominal success (D). DER is unconditional duplicate effect rate under B0B_0 (N=480N=480 per model). Persistence of the Competence–Recovery Gap. Table 3 presents aggregate outcomes for the extension. Both commercial API models exhibited strong nominal planning competence when aggregated across all recovery methods and frameworks (Gemini 3.8 Flash: 80.69%, 1,162 / 1,440, 95% CI: [56.94%, 98.61%]; GLM-5.2: 77.71%, 1,119 / 1,440, 95% CI: [52.71%, 100.0%]). Evaluating method-aligned performance under naive retry (B0B_0), nominal control reached 81.04% (389 / 480) for Gemini and 77.08% (370 / 480) for GLM. Under post-mutation acknowledgment loss, conditional recovery dropped sharply to 21.08% CRSR (82 / 389; 95% CI: [2.73%, 43.87%]) for Gemini and 21.89% CRSR (81 / 370; 95% CI: [0.00%, 50.00%]) for GLM-5.2, establishing substantial B0B_0 competence–recovery gaps of 59.96 p and 55.19 p, respectively. Furthermore, naive retry under ambiguous failure triggered duplicate external mutations in 65.62% (315 / 480) of Gemini trials and 51.25% (246 / 480) of GLM trials, illustrating that advanced general reasoning does not eliminate operational side-effect risks in uncoordinated retry loops. Recovery Mechanism Behavior. Idempotency (B2B_2) increased conditional recovery to 33.33% (128 / 384) on Gemini and 33.07% (124 / 375) on GLM, yielding a pooled descriptive improvement of +11.73 p over naive retry (33.20% vs. 21.48%; task-clustered bootstrap 95% CI: [+0.07 p, +33.25 p], raw p=0.0176p=0.0176). However, after applying Holm-Bonferroni correction across the recovery baseline family, the adjusted p-value is padj=0.0528p_adj=0.0528, failing to achieve statistical significance at the conventional α=0.05α=0.05 threshold. As in the primary study, idempotency eliminated duplicate mutations on endpoints providing deduplication token support, but provided no protection on non-idempotent endpoints. EvoUndo (B5B_5) operated under zero-privilege local journaling and achieved 22.11% (Gemini) and 21.66% (GLM) CRSR (pooled Δ=+0.41 p =+0.41 p vs. B0B_0, Holm padj=0.9180p_adj=0.9180). Lacking remote state visibility and registered compensating transactions, B5B_5 defaulted to bounded retry without achieving semantic reconciliation. Framework Equivalence and Semantic Invariants. Direct Tool Calling (F1F_1: 25.24% CRSR, 366 / 1,450) and LangGraph (F2F_2: 25.80% CRSR, 369 / 1,430) were empirically equivalent within the ±10 p± 10 p equivalence margin in this extension (Δ=−0.56 p =-0.56 p, 90% CI: [−2.35 p,+0.59 p][-2.35 p,+0.59 p], pTOST<0.0001p_TOST<0.0001). Finally, across all six evaluated model × baseline cells, EORcondEOR_cond numerically equaled CRSR because UndoBench’s programmatic oracle evaluates recovery strictly against goal invariants without duplicate committed effects. 5 Multi-Boundary Recovery Dynamics Table 4: Three-Boundary Recovery Performance Across Four Qualified Composite Workflows. Evaluated on commercial API models (Gemini 3.8 Flash, GLM-5.2 MaaS) across 20 seeds (N=320N=320 per cell). Method PRE CRSR DURING CRSR POST CRSR DURING DER B0B_0: Naive Retry 77.17% 0.00% 43.13% 82.50% B2B_2: Idempotency 77.42% 0.00% 45.98% 81.56% B5B_5: EvoUndo-RB1 76.68% 0.00% 45.57% 82.50% B6B_6: Verify-Retry 76.92% 25.89% 75.00% 54.37% Evaluated strictly on the same four qualified composite workflows (RB-CLOUD-004, RB-DB-004, RB-STOR-004, RB-GIT-004). Commercial API models only; prospectively specified post-freeze secondary analysis. To address our central research question—how recovery behavior changes across the external-mutation lifecycle—we conducted prospectively specified secondary evaluations with commercial API models (M3M_3 Gemini 3.8 Flash, M4M_4 GLM-5.2 MaaS) across three distinct fault boundaries. Table 4 summarizes recovery on the four qualified composite workflows supporting faithful multi-boundary semantics. Main Three-Boundary Interpretation. Recovery behavior changes sharply with fault timing. Under PRE_MUTATION, where failure occurs before external state mutation, the four methods exhibit descriptively similar recovery and no duplicate effects among nominally capable trials. Under DURING_MUTATION, the four qualified composite workflows expose intermediate external states: B0B_0, B2B_2, and zero-privilege B5B_5 achieve 0% CRSR, while B6B_6 recovers the publicly inspectable Cloud workflow but not the other three, yielding 25.89% CRSR. Under POST_MUTATION_PRE_ACK, the mutation has already committed and outcome ambiguity makes blind redispatch hazardous; verification and server-side deduplication become substantially more effective. These results indicate that recovery should be evaluated as a phase-dependent systems property rather than a single scalar capability. Pre-Mutation Execution (PRE_MUTATION, Appendix M). Across all 12 TEST workflows evaluated with commercial API models (N=3,840N=3,840 paired trials / 7,680 runs across B0,B2,B5,B6B_0,B_2,B_5,B_6), pre-mutation recovery exhibited near-parity: B0B_0 achieved 78.44% CRSR (593 / 756), B2B_2 achieved 78.86% (597 / 757; Δ=+0.43 p =+0.43 p), B5B_5 achieved 77.78% (588 / 756; Δ=−0.66 p =-0.66 p), and B6B_6 achieved 77.34% (587 / 759; Δ=−1.10 p =-1.10 p). Crucially, zero duplicate effects occurred among 3,028 nominally capable trials (0.00%0.00\% capable duplicate incidence). Canonical unconditional DER was 4.06%–4.17% (39–40 / 960 per method), arising entirely from RB-STOR-005 (D=0D=0). Naive retry (B0B_0) exhibits a massive boundary contrast across the full TEST suite: 78.44% CRSR under PRE_MUTATION vs. 21.48% under POST_MUTATION_PRE_ACK (Δ=+56.96 p =+56.96 p, 95% CI: [+27.97 p,+84.67 p][+27.97 p,+84.67 p]). On common capable support (Dcommon=743D_common=743), the contrast is near-identical: 79.00% (587 / 743) vs. 21.67% (161 / 743; Δ=+57.34 p =+57.34 p), confirming that the gap stems from fault timing rather than conditioning drift. Interrupted Mutation (DURING_MUTATION). Evaluating the four qualified composite workflows admitting faithful partial-mutation semantics (RB-DB-004, RB-STOR-004, RB-CLOUD-004, RB-GIT-004; N=1,280N=1,280 paired trials / 2,560 runs), blind retry and per-call tokens completely collapsed: B0B_0 achieved 0.00% CRSR (0 / 310; DER=82.50%DER=82.50\%), B2B_2 achieved 0.00% (0 / 315; DER=81.56%DER=81.56\%), and B5B_5 achieved 0.00% (0 / 308; DER=82.50%DER=82.50\%). Only B6B_6 achieved recovery, reaching 25.89% aggregate CRSR (80 / 309; DER=54.37%DER=54.37\%). Under Zero Privilege, B6B_6 did not receive bespoke repair logic: on RB-CLOUD-004, its public probe verified the green service entity in the catalog (ProbeStatus.PRESENT), suppressed redundant deployment, and enabled traffic cutover (100% CRSR, 0% DER). Conversely, B6B_6 failed on the other three: on RB-GIT-004, Git failed due to an unremoved lockfile (exit code 128); on RB-STOR-004, multipart upload state was hidden from HeadObject, triggering duplicate PUTs; on RB-DB-004, partial table creation diverged from uncommitted catalog metadata. Thus, observability is insufficient without admissible repair actions (task matrix in Appendix D). 6 Mechanism Isolation: Benchmark-Wide Idempotency To resolve whether B2B_2’s modest +11.73 p advantage in commercial API models stemmed from algorithmic limitations of idempotency or incomplete endpoint support, we executed a hosted-model intervention: B2-KB_2-K (N=960N=960 paired trials / 1,920 runs; Appendix L), deploying benchmark-wide server-side idempotency across all 12 TEST workflows. Benchmark-wide server-side idempotency closed most of the descriptive standard-B2B_2-to-perfect-recovery gap: on full support, standard B2B_2 achieved 33.20% CRSR compared with 97.24% CRSR (741 / 762) for B2-KB_2-K (DER=0.00%DER=0.00\%). On common capable support (Dcommon=738D_common=738), standard B2B_2 achieved 33.33% while B2-KB_2-K reached 98.64% (728 / 738), representing descriptive gap closures of 95.87% and 97.97%. Crucially, 21 / 762 capable failures remained: 17 occurred before any tool call due to empty/invalid hosted responses, and 4 occurred after successful duplicate suppression during multi-turn continuation due to invalid completions. Thus, endpoint deduplication alone does not eliminate all agent failure modes. 7 Discussion and Systems Differentiation Recovery Mechanisms Are Boundary-Specific. Multi-boundary findings show recovery is not a monolithic capability: (1) Pre-Mutation: Redispatch is safe since the external system has suffered no side effects; complex mechanisms add overhead without reliability gains. (2) During-Mutation: The challenge is intermediate-state reconciliation; per-call idempotency cannot restore composite multi-step invariants, and inspection requires admissible repair actions. (3) Post-Mutation Lost-ACK: The challenge is commit uncertainty; server-side deduplication (B2-KB_2-K) and pre-retry verification (B6B_6) directly resolve this ambiguity. Differentiating UndoBench and Related Systems (positioning in Appendix I). Recent benchmarks study execution integrity along distinct dimensions. LIMBO (Li, 2026) asks where exactly-once responsibility resides across model, harness, and contract interventions. AFT-Bench (Wang, 2026) intervenes on interface semantics to estimate mechanism effects. Verified Tool Calls (Mansoor et al., 2026) similarly combines postcondition verification, verify-before-retry, and idempotency middleware for non-atomic tool failures. The Verifier Tax (Sah et al., 2026b) similarly shows that runtime safety intervention can prevent unsafe actions without guaranteeing successful post-intervention recovery; UndoBench studies the complementary problem of recovering from execution faults while preserving externally visible system state. UndoBench emphasizes competence conditioning (CRSR), multi-turn workflow recovery, paired counterfactual control/fault evaluation, dual final-state and effect-history oracles under zero privilege, and phase-dependent recovery dynamics, while B2-KB_2-K provides a targeted secondary contract intervention isolating benchmark-wide idempotency. Reproducibility, Oracle Conformance, and Isolation Audits (Appendix N). Comprehensive audits confirm benchmark integrity: (1) Pair Comparability: Capable-pair prefix equivalence was 100% across PRE_MUTATION (3,028 pairs), B2-KB_2-K (762 pairs), and DURING_MUTATION (1,242 pairs), confirming identical control/fault prompts prior to perturbation. (2) Oracle Conformance: All 72 / 72 adversarial test cases passed across the 12 TEST contracts. (3) Provider Retry Isolation: 12 provider HTTP retries affected 8 executions; zero tool mutations escaped the proxy wrapper. 8 Conclusion Agent evaluations must assess operational recovery alongside nominal task completion. UndoBench establishes a decoupled benchmark isolating agent recovery under distributed failure regimes. By separating task competence from recovery capability, exposing severe duplicate side effects under lost acknowledgments, formalizing dual state and wire oracles, and empirically demonstrating phase-dependent recovery dynamics across the mutation lifecycle, UndoBench provides an empirical foundation for resilient agent systems. Limitations UndoBench has specific methodological boundaries: 1. Model Scope and Provider Determinism: Primary confirmatory evaluations focus on two open-weight instruction models (Llama-3.1-8B and Llama-3.2-3B), supplemented by a prospectively specified secondary extension evaluating Gemini 3.8 Flash and GLM-5.2 MaaS. While benchmark task configurations, fault injection schedules, and environment seeds (2001–2020) were held strictly deterministic across all trials, commercial hosted APIs do not contractually guarantee bitwise generation reproducibility across runs. 2. Task-Cluster Statistical Power: The TEST suite contains 12 independent task clusters (K=12K=12). While 20 seeds yield 5,760 executions, repeated seeds reduce within-task variance but do not increase independent cluster count. 3. Failure Boundary Scope: The frozen confirmatory study concentrates on POST_MUTATION_PRE_ACK, while prospectively specified secondary experiments cover PRE_MUTATION and four workflows supporting faithful DURING_MUTATION partial-state faults. Remaining unstudied boundaries in the UndoBench taxonomy include POST_ACK_PRE_CHECKPOINT (not faithfully realizable in the current synchronous harness), DURING_COMPENSATION, cascading faults, and larger workflow suites. 4. Single-Fault Injection Regime: UndoBench injects exactly one fault event per trajectory. Cascading, concurrent, or Byzantine failures remain for future work. 5. Sandboxed Environments: Environments mock real enterprise APIs (SQLite, local Git, mocked Stripe/AWS). Production systems feature higher concurrency and variable latency distributions. 6. Zero-Privilege Baseline Scope: EvoUndo (B5B_5) operated without domain heuristics or remote probes, functioning as an unspecialized baseline. The B5B_5 result characterizes a zero-privilege UndoBench instantiation rather than the full compensation capability of the broader EvoUndo architecture (Sah et al., 2026a). 7. Post-Hoc Status of Baseline B6B_6: Verify-Before-Retry was evaluated post-hoc and is not part of the confirmatory B0/B2/B5B_0/B_2/B_5 comparison. 8. Forensic EOR Reconstruction: Exactly-Once Rate was reconstructed deterministically from wire-level effect logs due to an initial logging verdict omission. 9. Conditioning on Capable Trials: CRSR is conditioned on D=∑Ci>0D=Σ C_i>0. When nominal competence is zero (e.g., RB-STOR-005), CRSR is mathematically undefined (N/A), requiring joint inspection with unconditional RSR. 10. Retry Budgets and Transport Middleware: B0B_0 uses a fixed 3-retry budget with exponential backoff. Model provider and transport API retries are handled orthogonally by HTTP middleware and do not consume agent turns or trigger benchmark faults; sensitivity sweeps across backoff schedules remain future work. Ethical Considerations UndoBench evaluates fault recovery in strictly simulated, sandboxed software environments. Benchmark evaluations should not be interpreted as a guarantee that an agent is safe for unmonitored production deployment; high benchmark scores under single-fault injection do not guarantee resilience against cascading or adversarial distributed failures. References Chang and Geng (2025) Edward Y. Chang and Longling Geng. 2025. SagaLLM: Context management, validation, and transaction guarantees for multi-agent LLM planning. Proceedings of the VLDB Endowment, 18(12):4874–4886. Garcia-Molina and Salem (1987) Hector Garcia-Molina and Kenneth Salem. 1987. Sagas. ACM SIGMOD Record, 16(3):249–259. Gupta (2026) Aayush Gupta. 2026. ReliabilityBench: Evaluating LLM agent reliability under production-like stress conditions. arXiv preprint arXiv:2601.06112. Helland (2012) Pat Helland. 2012. Idempotence is not a medical condition. Communications of the ACM, 55(5):56–65. Hu et al. (2026) Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng, Yuxuan Liu, Yantao Liu, Dayiheng Liu, and Tsung-Yi Ho. 2026. OccuBench: Evaluating AI agents on real-world professional tasks via language environment simulation. arXiv preprint arXiv:2604.10866. Jimenez et al. (2024) Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). Lan et al. (2026) Wenhao Lan, Shan Li, Meiqi Wu, Xinhua Lai, Junbin Yang, and Haihua Shen. 2026. ContainmentBench: Trace-based evaluation of post-exposure containment in tool-using LLM agents. arXiv preprint arXiv:2607.23999. Li (2026) Jiapeng Li. 2026. Where does exactly-once live? model, harness, and tool-contract effects on duplicate side effects in LLM agents. arXiv preprint arXiv:2609.29095. Li et al. (2023) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3102–3116. Li et al. (2026) Xiaoyang Li, Yiqi Wang, Haohui Lu, Zhi Chen, Mo Li, Pingan Song, Mingkai Zheng, and Taotao Cai. 2026. MemTX: Transactional belief commit for stateful agent memory. arXiv preprint arXiv:2607.23929. Liu et al. (2024) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2024. AgentBench: Evaluating LLMs as agents. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). Mansoor et al. (2026) Isham Kalappurackal Mansoor, Abhishek Phadke, and Pratip Rana. 2026. Verified tool calls improve LLM agent reliability under non-atomic failures. arXiv preprint arXiv:2608.02645. Mohammadi et al. (2026) Bardia Mohammadi, Nearchos Potamitis, Lars Klein, Akhil Arora, and Laurent Bindschaedler. 2026. Atomix: Timely, transactional tool use for reliable agentic workflows. arXiv preprint arXiv:2602.14849. Mohan et al. (1992) C. Mohan, Don Haderle, Bruce Lindsay, Hamid Pirahesh, and Peter Schwarz. 1992. ARIES: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Transactions on Database Systems, 17(1):94–162. Qin et al. (2024) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). Sah et al. (2026a) Tanmay Sah, Dolly Sah, Harshul Jain, and Tanya Sah. 2026a. EvoUndo: Recoverability-constrained self-evolution for LLM agent harnesses. arXiv preprint arXiv:2608.28363. Sah et al. (2026b) Tanmay Sah, Vishal Srivastava, Dolly Sah, and Kayden Jordan. 2026b. The Verifier Tax: Horizon-dependent safety-success tradeoffs in tool-using LLM agents. arXiv preprint arXiv:2603.19328. Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 8634–8652. Sigdel and Baral (2026) Akshey Sigdel and Rista Baral. 2026. ToolMisuseBench: An offline deterministic benchmark for tool misuse and recovery in agentic systems. arXiv preprint arXiv:2604.01508. Tan et al. (2026) Gou Tan, Zhensu Sun, Jieke Shi, Ting Zhang, Zilong He, Qingfu Wu, Shuai Liang, Weifeng Sun, Junda He, Pengfei Chen, Chuanfu Zhang, Lwin Khin Shar, and David Lo. 2026. AgentChaos: Chaos engineering for agent systems via programmatic fault injection. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE). Wang (2026) Zihao Wang. 2026. Callability is not operability: Controlled interface interventions for LLM agents. arXiv preprint arXiv:2608.23628. Zhou et al. (2024) Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A realistic web environment for building autonomous agents. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). Appendix A Benchmark Factorial Design and Task Specifications Table 5 summarizes the overall UndoBench factorial design across development, validation, and test splits. Table 6 details the 12 held-out TEST workflows evaluated in the primary frozen study. Baseline Taxonomy and Indexing. Baseline indices follow the UndoBench architectural recovery taxonomy: B0B_0 (Naive Retry), B1B_1 (Checkpoint/Restore), B2B_2 (Idempotency Keys), B3B_3 (Sagas / Compensating Actions), B4B_4 (LangGraph-Native State Persistence), B5B_5 (EvoUndo-RB1), and B6B_6 (Verify-Before-Retry). Baselines B1B_1, B3B_3, and B4B_4 were excluded from the benchmark-wide factorial evaluation due to domain inapplicability (e.g., inability to snapshot remote cloud or payment APIs for B1B_1, absence of pre-registered inverse actions under zero privilege for B3B_3) or framework coupling (B4B_4 requires LangGraph runtime state, breaking framework-neutral evaluation across F1F_1 and F2F_2). Baseline B6B_6 is evaluated post-hoc as a targeted verification intervention (Section 5 and Appendix D). For baseline B2B_2, idempotency keys are deterministically computed from task_id, tool_name, positional arguments, and filtered keyword arguments using a truncated SHA-256 digest. Identical serialized calls therefore derive the same key across client process restarts without relying on transient process state. Server-side deduplication state in the evaluated sandbox is cached in memory per episode and is not evaluated for persistence across server restarts. Because effect identity is value-based, identical calls within a task map to the same key under the assumption that identical arguments in a single task represent the same logical mutation. Table 5: UndoBench Factorial Evaluation Matrix. Dimension Specification / Scale Base Workflows 36 Workflows across 8 Enterprise Domains Split Allocation DEV (14), VALIDATION (10), TEST (12) Held-Out TEST Tasks 12 Tasks (Table 6) Evaluated Models Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct Orchestration Frameworks Direct Tool Calling (F1), LangGraph Agent (F2) Recovery Paradigms Naive Retry (B0), Idempotency (B2), EvoUndo (B5) Evaluation Scale 12×2×2×3×2×20=5,76012× 2× 2× 3× 2× 20=5,760 Runs Paired Trials (N) 2,880 Paired Counterfactual Trials Evaluation Seeds Seeds 2001–2020 (recorded in commit f11b807) Table 6: Held-Out TEST Benchmark Task Composition, Reversibility, and Idempotency Support. All 12 held-out workflows evaluate the POST_MUTATION_PRE_ACK failure boundary under 20 evaluation seeds. Task ID Domain Baseline Workflow Description Reversibility Idempotency / Deduplication Support RB-CLOUD-004 Cloud Blue-Green Service Deployment Cutover Compensatable Routing state overwrite (Inherent idempotent) RB-CRM-004 CRM Enterprise SLA Tier Renewal Compensatable None (Unkeyed mutable REST API) RB-DB-004 Database Schema Migration with Batch Audit Log Compensatable SQLite schema unique constraint RB-DB-005 Database Read-Modify-Write Bank Dividend Allocation Compensatable None (Unkeyed SQL ledger insert) RB-GIT-004 Git Release Tag Cut and Cherry-Pick Reversible Git tag collision rejection (Inherent unique) RB-MSG-004 Messaging High-Urgency Pager Callout with Delivery ACK Irreversible None (Unkeyed webhook push) RB-MSG-005 Messaging Webhook Fanout with Delivery Confirmation Irreversible Idempotency key tracking (Idempotency-Key) RB-PAY-004 Payments Subscription Renewal with Grace Period Compensatable Idempotency key deduplication (idempotency_key) RB-PAY-005 Payments Multi-Currency Cross-Border Wire Transfer Compensatable Idempotency key deduplication (idempotency_key) RB-STOR-004 Storage Cross-Bucket Object Replication Compensatable None (Unkeyed bucket copy) RB-STOR-005 Storage Atomic Object Swap with Cleanup Compensatable None (Multipart upload chunks) RB-TICK-004 Ticketing SLA Escalation Policy Cascade Compensatable None (Unkeyed escalation ticket creation) Appendix B Full Task × Method Performance Matrix Table 7 and Figure 3 report the complete granular performance matrix and task-level heatmap across all 12 TEST workflows for the primary recovery baselines. Figure 4 plots the impact of mutation observability across representative workflows under lost acknowledgment. Table 7: Granular Task-Level Performance Matrix across 12 Held-Out TEST Tasks (N=2,880N=2,880 Paired Trials). Task ID Domain Workflow Name Pairs Control Pass B0B_0 CRSR B2B_2 CRSR RB-CLOUD-004 Cloud Blue-Green Deployment Cutover 240 100.0% (240 / 240) 100.0% (80 / 80) 100.0% (80 / 80) RB-CRM-004 CRM Enterprise SLA Tier Renewal 240 100.0% (240 / 240) 0.0% (0 / 80) 0.0% (0 / 80) RB-DB-004 Database Schema Migration with Audit Log 240 60.0% (144 / 240) 52.1% (25 / 48) 72.9% (35 / 48) RB-DB-005 Database Bank Dividend Allocation 240 100.0% (240 / 240) 0.0% (0 / 80) 0.0% (0 / 80) RB-GIT-004 Git Release Tag Cut & Cherry-Pick 240 100.0% (240 / 240) 100.0% (80 / 80) 100.0% (80 / 80) RB-MSG-004 Messaging Pager Callout with Delivery ACK 240 100.0% (240 / 240) 0.0% (0 / 80) 0.0% (0 / 80) RB-MSG-005 Messaging Webhook Fanout Confirmation 240 92.5% (222 / 240) 100.0% (74 / 74) 100.0% (74 / 74) RB-PAY-004 Payments Subscription Renewal Grace Period 240 100.0% (240 / 240) 40.0% (32 / 80) 100.0% (80 / 80) RB-PAY-005 Payments Cross-Border Wire Transfer 240 100.0% (240 / 240) 65.0% (52 / 80) 100.0% (80 / 80) RB-STOR-004 Storage Cross-Bucket Object Replication 240 100.0% (240 / 240) 0.0% (0 / 80) 0.0% (0 / 80) RB-STOR-005 Storage Atomic Object Swap with Cleanup 240 0.0% (0 / 240) N/A∗ N/A∗ RB-TICK-004 Ticketing SLA Escalation Policy Cascade 240 50.0% (120 / 240) 0.0% (0 / 40) 0.0% (0 / 40) ∗Control success was 0/240 (D=0D=0) across all configurations due to multipart upload chunk verification; CRSR (C/D=0/0C/D=0/0) is mathematically undefined and presented as N/A. Fully retained in aggregate benchmark metrics (0/240 Control, 0/240 RSR). Figure 3: Held-Out TEST Task × Recovery Method Heatmap (CRSR). Method effectiveness varies sharply across tasks, highlighting mechanism-specific recovery boundaries. Figure 4: The Impact of Mutation Observability on Recovery and Safety. In the evaluated non-idempotent workflows lacking protective constraints, naive retry produced duplicate effects; endpoint-supported idempotency eliminated duplicates in the corresponding payment workflows. Appendix C Safety Metrics and Forensic EOR Validation Figure 6 visualizes safety failure rates across recovery baselines. Figure 5: Failure Safety Rates Across Recovery Baselines (TEST Split). Naive retry (B0B_0) triggers duplicate side effects in 53.33% of trials; idempotency (B2B_2) eliminates duplicates on supported APIs. Figure 6: Cross-Split Descriptive Consistency Across Developmental and Evaluation Splits. Forensic Validation and Conformance Evidence. During initial aggregation audits, an evaluation check identified that Exactly-Once Rate (EOR) initially reported 0.00% across all methods because the field committed_mutation_count was unpopulated on serialized verdicts in StateOracle.evaluate(). We resolved this forensically by deterministically parsing immutable wire-level execution logs (results/rb3c_test_raw.jsonl, SHA-256 digest 1016768449aae019484130b23fda33456861abeb161a5f8dfbeb16e9e5e9f882 at tag f11b807). Rather than re-running LLM trajectories, raw event logs were parsed using our offline forensic script (scripts/reproduce_eor.py) and verified by unit tests (tests/test_eor_reconstruction.py). For each trial, the parser verified whether committed external effects matched task requirements with zero duplicate mutations. Because UndoBench’s state oracle strictly requires invariant satisfaction without duplicates, every successful recovery trajectory (R=1,124/D=2,406R=1,124/D=2,406) satisfied required goal invariants with exactly the intended external mutations (EORcond=46.72%EOR_cond=46.72\%, numerically identical to CRSR). No execution trajectories were re-run or altered. This validation relies strictly on frozen wire-level telemetry and deterministic replay parsing within the evaluation SDK, without claiming unverified third-party implementations. Appendix D Zero-Privilege Observability Matrix for Baseline B6B_6 Table 8 provides the task-level audit for Verify-Before-Retry (B6B_6) across all 12 TEST workflows. Table 8: Zero-Privilege Observability Matrix for Baseline B6B_6 across Held-Out TEST Tasks (N=960N=960 Paired Trials). Task ID Domain / Operation Control (D) Observable? Probe State CRSR Recovery Action / Mechanism RB-CLOUD-004 Cloud Traffic Switch 80 / 80 Yes PRESENT 100.0% Active probe confirms route; suppresses duplicate retry. RB-CRM-004 CRM SLA Tier Renewal 80 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate SLA update. RB-DB-004 DB Schema Migration 48 / 80 Yes PRESENT 100.0% Schema catalog query; suppresses duplicate DDL. RB-DB-005 DB Blind Audit Append 80 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate log insert. RB-GIT-004 Git Release Tagging 80 / 80 Yes PRESENT 100.0% Public Git CLI inspection; suppresses duplicate tag. RB-MSG-004 Slack Channel Alert 80 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate notification. RB-MSG-005 Incident Dispatch Webhook 74 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate broadcast. RB-PAY-004 Payment Capture API 80 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate card charge. RB-PAY-005 Dispute Refund API 77 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate refund debit. RB-STOR-004 S3 Object Copy 79 / 80 Yes PRESENT 100.0% HeadObject existence check; suppresses duplicate PUT. RB-STOR-005 S3 Multipart Chunk Upload 0 / 80 Yes PRESENT N/A Layer A planning failure (D=0D=0, chunk mismatch). RB-TICK-004 Issue State Transition 40 / 80 No UNKNOWN 100.0% Zero-privilege abstention; avoids duplicate transition. Appendix E Physical Reversibility and Operational Overhead Telemetry Table 9 stratifies recovery performance across physical reversibility regimes. Table 10 presents runtime latency and token overhead telemetry across all 5,760 executions. Table 9: Performance Across Physical Reversibility Regimes. Reversibility Regime Tasks (K) Control Pass (D) Naive Retry (B0B_0) Idempotency (B2B_2) Reversible 1 task 100.0% (240 / 240) 100.0% (80 / 80) 100.0% (80 / 80) Compensatable 9 tasks 78.80% (1,702 / 2,160) 33.27% (189 / 568) 48.42% (275 / 568) Irreversible 2 tasks 96.25% (462 / 480) 48.05% (74 / 154) 48.05% (74 / 154) Table 10: Operational Telemetry (N=2,880N=2,880 Pairs / 5,760 Runs). Execution Regime Latency (ms) Prompt Tokens Completion Tokens Tool Calls LLM Invocations Control Runs (N=2,880N=2,880) 2,924 ± 1,482 738.9 ± 112.4 87.2 ± 24.1 1.54 ± 0.62 2.04 ± 0.48 Faulted Runs (N=2,880N=2,880) 3,782 ± 1,694 746.2 ± 115.3 96.4 ± 26.2 2.40 ± 0.71 2.32 ± 0.65 Δ vs. Control +29.3% +1.0% +10.5% +55.8% +13.7% Appendix F Statistical Details and TOST Equivalence Tests Primary confidence intervals are computed via non-parametric task-clustered bootstrap resampling (B=10,000B=10,000 iterations). Multi-comparison contrasts are adjusted using the Holm-Bonferroni procedure. Framework equivalence was tested using Two One-Sided Tests (TOST) against the pre-registered equivalence margin δ=±10.0δ=± 10.0 percentage points (pTOST<0.001p_TOST<0.001). Appendix G Developmental Provenance and Cross-Split Consistency Table 11 and Figure 6 track benchmark properties across the DEV evaluation subset (6 of 14 tasks), VALIDATION (10 tasks), and TEST (12 tasks) splits. The frozen primary TEST raw execution dataset is cryptographically locked with SHA-256 digest 1016768449aae019484130b23fda33456861abeb161a5f8dfbeb16e9e5e9f882 at tag f11b807. Table 11: Cross-Split Consistency Across Developmental and Evaluation Splits. Property DEV Evaluation Subset (6 of 14 tasks) VAL Split (10 Tasks) TEST Split (12 Tasks) Workflows 6 Evaluated Tasks 10 Tasks 12 Tasks Executions (Pairs) 720 (360) 2,400 (1,200) 5,760 (2,880) Layer A Control Pass 63.89% (230 / 360) 74.50% (894 / 1,200) 83.54% (2,406 / 2,880) Unconditional RSR 20.28% (73 / 360) 44.83% (538 / 1,200) 39.03% (1,124 / 2,880) Conditional CRSR 31.74% (73 / 230) 60.18% (538 / 894) 46.72% (1,124 / 2,406) Framework Δ (F1 - F2) 0.00 p 0.00 p +0.50 p Appendix H Post-Study Hardware Interoperability Qualification To verify that the evaluation SDK operates with standard open-weight inference backends without modifying benchmark internals, a post-study compatibility qualification was conducted on a single physical NVIDIA A100-SXM4-40GB GPU host (AMD EPYC 7713, 115 GiB RAM, Ubuntu 24.04, CUDA 13.2, PyTorch 2.13.0, vLLM 0.30.0). Three model families were qualified using downloaded open weights: Qwen3-14B, Mistral-NeMo-12B, and IBM Granite-3.3-8B. All three executed the 8-domain qualification suite using model-appropriate inference/tool-calling configurations without benchmark-semantic modifications under naive retry (B0B_0), with peak VRAM within single-GPU capacity (35.58 to 36.64 GiB). These hardware runs serve exclusively as descriptive engineering qualifications of SDK interoperability and are strictly separated from the primary frozen scientific experiment. Appendix I UndoBench vs. Prior Benchmarks and Verification Frameworks Table 12 positions UndoBench relative to existing agent benchmarks and verification systems across 8 critical dimensions. Table 12: UndoBench vs. Prior Benchmarks and Verification Frameworks. Benchmark / System Multi-Turn Post-Mutation Paired Control/ Competence Ground-Truth Wire-Level Side-Effect Safety Pluggable Workflows Ambiguity (Lost ACK) Fault Design Conditioning (CRSR) State Oracles Effect Logging Metrics (EOR/DER) Substrates ToolBench (Qin et al., 2024) ∼ × × × × × × × SWE-bench (Jimenez et al., 2024) ✓ × × × ∼ × × × WebArena / AgentBench (Liu et al., 2024; Zhou et al., 2024) ✓ × × × ∼ × × × OccuBench (Hu et al., 2026) ✓ × × × ∼ × × × ReliabilityBench (Gupta, 2026) ✓ × × × ∼ × × ∼ AgentChaos (Tan et al., 2026) ✓ × × × ∼ × × ∼ ToolMisuseBench (Sigdel and Baral, 2026) × ∼ (Replayable) × × × × × × Verified Tool Calls (Mansoor et al., 2026) ∼ ✓ × × ∼ × × × Limbo (Li, 2026) × ✓ × × ✓ ✓ ✓(DER) ∼ Atomix (Mohammadi et al., 2026) ✓ ∼ (Epoch buffer) × × ∼ × ∼ ∼ SagaLLM (Chang and Geng, 2025) ✓ ∼ (Saga coords) × × ∼ × ∼ ∼ AFT-Bench (Wang, 2026) ✓ ✓ ∼ (Interface pairs) × ✓ ∼ ✓ ∼ ContainmentBench (Lan et al., 2026) ✓ × × × ∼ × × × MemTX (Li et al., 2026) ✓ × (Internal memory) × × ∼ × × ∼ UndoBench (Ours) ✓ (8 Domains) ✓ ✓ ✓ (R/DR/D) ✓ ✓ ✓ (EOR/DER/URR) ✓ (B0…B6B_0… B_6) Appendix J Commercial API Model Extension: Complete Cell Breakdown and Provenance This appendix provides granular cell breakdowns and cryptographic provenance for the prospectively specified post-freeze commercial API model extension. The extension evaluated two modern architectures—Gemini 3.8 Flash (gemini-3.8-flash, M3M_3) and GLM-5.2 MaaS (zai-org/glm-5.2-maas, M4M_4)—across all 12 held-out TEST workflows, two frameworks (F1F_1 Direct Tool Calling, F2F_2 LangGraph), and three recovery baselines (B0B_0 Naive Retry, B2B_2 Idempotency, B5B_5 EvoUndo) across 20 pre-specified benchmark seeds (2001–2020). Complete Cell Breakdown. Table 13 details the 6-cell model × baseline matrix (2×3=62× 3=6 cells, 480 trials per cell, totaling 2,880 paired trials and 5,760 executions). As established in the forensic audit, canonical EORcond≡CRSREOR_cond across all cells because the strict state oracle requires invariant satisfaction with zero duplicate mutations and zero missing effects. Furthermore, Unsafe Retry Rate (URR) numerically equals Duplicate Effect Rate (DER) across all cells under the post-mutation lost-ACK regime. Cryptographic Data Provenance. The extension was executed using frozen harness code at commit c3108e684f9af105ce72c19737f6077f57fdf59. All raw records in results/contemporary_models/raw/ are verified by SHA-256 digests: • Evaluation Matrix (matrix.csv): SHA-256: 5c6066a380bc9f2615d995f2e55481c4d073965b8778c74de6b2d23623b352f • Execution Trajectories (trajectories.jsonl): SHA-256: b0f989920d517efb3eeeff85b4d382d77a676a3cfa1ba09d389f897af1688a Table 13: Granular Model × Method Evaluation Matrix for Commercial API Models (N=2,880N=2,880 Paired Trials / 5,760 Executions). Model Recovery Method N Control (D) RSR (F/NF/N) CRSR (R/DR/D) 95% CI (CRSR) DER MER Gemini 3.8 Flash B0B_0: Naive Retry 480 81.04% (389) 17.29% (83 / 480) 21.08% (82 / 389) [2.73%, 43.87%] 65.62% 18.33% Gemini 3.8 Flash B2B_2: Idempotency 480 80.00% (384) 27.92% (134 / 480) 33.33% (128 / 384) [8.33%, 61.33%] 53.96% 20.00% Gemini 3.8 Flash B5B_5: EvoUndo-RB1 480 81.04% (389) 18.12% (87 / 480) 22.11% (86 / 389) [2.22%, 45.65%] 63.12% 20.21% GLM-5.2 MaaS B0B_0: Naive Retry 480 77.08% (370) 17.29% (83 / 480) 21.89% (81 / 370) [0.00%, 50.00%] 51.25% 31.46% GLM-5.2 MaaS B2B_2: Idempotency 480 78.12% (375) 26.46% (127 / 480) 33.07% (124 / 375) [3.28%, 65.07%] 41.67% 31.87% GLM-5.2 MaaS B5B_5: EvoUndo-RB1 480 77.92% (374) 16.88% (81 / 480) 21.66% (81 / 374) [0.00%, 50.25%] 51.88% 31.25% All metrics derived from frozen raw trajectories via the audited deterministic parser. N denotes total paired trials; D denotes capable control trials; F denotes all successful fault runs (RSR=F/NRSR=F/N); R=∑CiFiR=Σ C_iF_i denotes fault recoveries among nominally capable paired trials (CRSR=R/DCRSR=R/D). In all six commercial API model–method cells, EORcond≡CRSREOR_cond and URR≡DERURR under the frozen POST_MUTATION_PRE_ACK oracle/taxonomy. Appendix K Effect Identity and Dual-Oracle Instrumentation This appendix provides formal specifications for UndoBench’s effect accounting model, dual-oracle evaluation architecture, and post-hoc secondary analyses. Effect Record Schema and Identity Model. Every tool invocation intercepted by the benchmark proxy generates an immutable WireEffectEvent record: • call_index: Monotonically increasing per-episode call sequence counter. • tool_name: Dispatched tool identifier (e.g., renew_enterprise_sla). • target: Fully qualified resource entity target string (e.g., crm.account.acct_ent_500.sla). • op_type: Operation taxonomy: READ, CREATE, UPDATE, DELETE, or EXECUTE. • arguments: Serialized positional arguments and sanitized keyword parameters. • committed_externally: Boolean flag indicating physical datastore state mutation. • acknowledged_to_agent: Boolean flag indicating successful return delivery to agent client. • error: Intercepted or injected failure token (e.g., FaultInjected:POST_MUTATION_PRE_ACK). • timestamp: Monotonic floating-point epoch timestamp. Dual-Oracle Evaluation Pipeline. UndoBench couples two complementary evaluation substrates: 1. Final-State Oracle: Assesses external sandbox ground truth after episode termination. Queries database schemas, git commit graphs, object stores, and CRM datastores to verify goal invariants and state equivalence. 2. Wire-Level Effect-History Oracle: Audits the complete proxy transaction log to detect out-of-order, uncompensated, duplicate, or missing mutations. Effect Classification Rules. Let ℰiE_i be the set of committed external mutations for episode i. For each target entity k: • Duplicate Mutations: dupi=∑kmax(0,commits(k)−1)dup_i= _k (0,commits(k)-1). • Missing Mutations: missi=∑k(required(k)∧commits(k)=0)miss_i= _kI(required(k) (k)=0). • Exactly-Once Metric: EORcond=1⇔(Fi=1∧dupi=0∧missi=0)EOR_cond=1 (F_i=1 _i=0 _i=0). Deterministic reconstruction and verification of all reported EOR values is automated via scripts/reproduce_eor.py. Appendix L Benchmark-Wide Idempotency Intervention: Hosted-Model Evaluation (B2-KB_2-K) To resolve whether client-side idempotency’s modest improvement over naive retry in commercial API models (+11.73 p under standard B2B_2) stemmed from algorithmic limitations of the idempotency mechanism or incomplete server endpoint deduplication, we executed a hosted-model intervention: B2-KB_2-K (N=960N=960 paired trials / 1,920 runs using Gemini 3.8 Flash and GLM-5.2 MaaS across 2 frameworks, 12 TEST workflows, and 20 seeds), deploying benchmark-wide server-side deduplication across all 12 TEST workflows. Full-Support and Common-Capable Outcomes. Across the full TEST suite (N=960N=960, D=762D=762), B2-KB_2-K achieved 97.24% CRSR (741 / 762) with 0.00% DER (0 / 960), compared with 33.20% CRSR for standard B2B_2. Conditioning strictly on the common-capable support (Dcommon=738D_common=738) between standard B2B_2 and B2-KB_2-K, standard B2B_2 achieved 33.33% CRSR (246 / 738) while B2-KB_2-K achieved 98.64% CRSR (728 / 738). These represent descriptive gap-closure fractions of 95.87% on full support and 97.97% on common capable support. Residual Failure Taxonomy (N=21N=21 Capable Failures). Across all 762 nominally capable trials, exactly 21 failures occurred under B2-KB_2-K, spanning two distinct non-recovery failure modes: 1. Pre-Tool Turn-0 Provider Exceptions (17 trials): In RB-PAY-004 (17 trials), commercial API requests raised INVALID_ARGUMENT or API_RESPONSE_VALIDATION_FAILED before issuing any tool call, preventing the model from initiating the workflow. 2. Post-Suppression Continuation Malformations (4 trials): In RB-DB-004 (4 trials), the first mutating tool call was interrupted, intercepted by B2-KB_2-K, and deduplicated successfully; however, during the subsequent multi-turn continuation, the model generated empty or unparseable tool call arguments, failing the downstream audit log requirement. Zero duplicate side effects occurred across all 960 executions. These residual failures confirm that benchmark-wide idempotency eliminates lost-ACK ambiguity but cannot compensate for non-deterministic model generation errors or downstream multi-turn planning failures. Appendix M Multi-Boundary Experiments with Commercial API Models This appendix provides granular details for the prospectively specified post-freeze boundary extensions. Hosted PRE_MUTATION Extension (12 TEST Workflows). Evaluated across all 12 TEST workflows using Gemini 3.8 Flash and GLM-5.2 MaaS (N=3,840N=3,840 paired trials / 7,680 executions across B0,B2,B5,B6B_0,B_2,B_5,B_6): • B0B_0 Naive Retry: D=756D=756, R=593⟹78.44% CRSRR=593 78.44\% CRSR, DER=4.17%DER=4.17\% (40 / 960). • B2B_2 Idempotency: D=757D=757, R=597⟹78.86% CRSRR=597 78.86\% CRSR (Δ=+0.43 p =+0.43 p vs. B0B_0), DER=4.06%DER=4.06\% (39 / 960). • B5B_5 EvoUndo-RB1: D=756D=756, R=588⟹77.78% CRSRR=588 77.78\% CRSR (Δ=−0.66 p =-0.66 p vs. B0B_0), DER=4.17%DER=4.17\% (40 / 960). • B6B_6 Verify-Before-Retry: D=759D=759, R=587⟹77.34% CRSRR=587 77.34\% CRSR (Δ=−1.10 p =-1.10 p vs. B0B_0), DER=4.06%DER=4.06\% (39 / 960). Across all four baselines, zero duplicate effects occurred among capable trials (0.00%0.00\% capable duplicate incidence). All 39–40 trials per method with duplicate effects occurred in RB-STOR-005 (D=0D=0), where nominal control failure caused unguided tool executions. Common-Support Contrast. Conditioning on common capable support between PRE_MUTATION and POST_MUTATION_PRE_ACK (Dcommon=743D_common=743, matched by task, model, framework, seed) yields: PRE B0 B_0 =79.00%(587/743) =79.00\%\;(587/743) POST B0 B_0 =21.67%(161/743) =21.67\%\;(161/743) (3) representing a descriptive delta of +57.34 p+57.34 p. Accounting for task-level clustering via non-parametric bootstrap (B=10,000,K=10B=10,000,K=10), the estimated boundary contrast is +57.04 p+57.04 p (SE = 14.79 p14.79 p; 95% CI: [+26.99 p,+84.90 p][+26.99 p,+84.90 p]), confirming that the boundary gap is driven by fault timing rather than conditioning drift. Hosted DURING_MUTATION Extension (4 Qualified Workflows). Table 14 details the complete 16-cell task × method matrix (N=1,280N=1,280 paired trials / 2,560 executions) across the four qualified composite workflows admitting faithful partial-mutation semantics. Table 14: Granular Task × Method Matrix under DURING_MUTATION (N=1,280N=1,280 Paired Trials / 2,560 Executions). Task ID Domain Method N Control (D) CRSR (R/DR/D) DER Cap. Dup. (%) MER URR RB-DB-004 Database B0B_0 80 70 0.00% (0 / 70) 97.50% (78) 98.57% 12.50% (10) 97.50% RB-DB-004 Database B2B_2 80 75 0.00% (0 / 75) 97.50% (78) 97.33% 10.00% (8) 97.50% RB-DB-004 Database B5B_5 80 68 0.00% (0 / 68) 95.00% (76) 97.06% 16.25% (13) 95.00% RB-DB-004 Database B6B_6 80 69 0.00% (0 / 69) 91.25% (73) 91.30% 15.00% (12) 91.25% RB-STOR-004 Storage B0B_0 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-STOR-004 Storage B2B_2 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-STOR-004 Storage B5B_5 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-STOR-004 Storage B6B_6 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-CLOUD-004 Cloud B0B_0 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-CLOUD-004 Cloud B2B_2 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-CLOUD-004 Cloud B5B_5 80 80 0.00% (0 / 80) 100.0% (80) 100.0% 0.00% (0) 100.0% RB-CLOUD-004 Cloud B6B_6 80 80 100.0% (80 / 80) 0.00% (0) 0.00% 0.00% (0) 0.00% RB-GIT-004 Git B0B_0 80 80 0.00% (0 / 80) 32.50% (26) 32.50% 0.00% (0) 32.50% RB-GIT-004 Git B2B_2 80 80 0.00% (0 / 80) 28.75% (23) 28.75% 0.00% (0) 28.75% RB-GIT-004 Git B5B_5 80 80 0.00% (0 / 80) 35.00% (28) 35.00% 0.00% (0) 35.00% RB-GIT-004 Git B6B_6 80 80 0.00% (0 / 80) 26.25% (21) 26.25% 0.00% (0) 26.25% Appendix N Verification Audits, Oracle Conformance, and Robustness Prefix Equivalence and Pair Comparability. Across all 3,840 paired trials (7,680 executions) in PRE_MUTATION and 960 paired trials (1,920 executions) in B2-KB_2-K, prefix prompt comparability between paired control and faulted trajectories was audited. Across all 3,028 nominally capable pairs in PRE_MUTATION (B0B_0: 756, B2B_2: 757, B5B_5: 756, B6B_6: 759), capable-pair prefix equivalence was 100.00% (3,028 / 3,028). For B2-KB_2-K, capable-pair prefix equivalence was 100.00% (762 / 762). Under DURING_MUTATION, prefix equivalence was 100.00% across all 1,280 pairs and all 1,242 capable pairs. When evaluated across all pairs (including non-competent runs), pre-fault prompt divergences were confined entirely to unguided exploratory actions in RB-STOR-005 (D=0D=0). While commercial hosted APIs do not contractually guarantee bitwise generation invariance across calls, identical initial states and pre-fault prompts establish rigorous paired comparability. Adversarial Oracle Conformance. To verify the fidelity of the dual-oracle architecture, a dedicated conformance test suite (tests/test_oracle_conformance.py) executed 72 adversarial test cases across all 12 TEST contracts (6 cases per contract: clean success, single duplicate mutation, omitted mutation, out-of-order execution, corrupted final state, and unhandled exception). All 72 / 72 adversarial test cases passed: the oracles correctly detected every synthetic perturbation without false positives or false negatives. Provider Retry Isolation Audit. To ensure that underlying model provider network drops did not contaminate experimental fault injection, HTTP transport middleware logged all provider-level retries. Across all hosted executions, exactly 12 provider retry events occurred across 8 executions (0.28% of executions). Forensic audit confirmed that 0 tool mutations escaped the UndoBench sandbox wrapper during provider retry events: all retries were purely transport-level prompt retries executed prior to tool dispatch. CRSR Support Robustness. To confirm that conditioning on capable control trials does not induce sensitivity to threshold selection, we evaluated method rankings across support thresholds D>0D>0, D≥20D≥ 20, and D≥40D≥ 40, alongside all 12 leave-one-task-out folds. Across all thresholds and folds, recovery-method ordering remained invariant (B2>B5≈B0B_2>B_5≈ B_0). N.1 Illustrative Analysis: Two-Task Scripted EvoUndo Observability Ablation (B5-PB_5-P) On two compensatable workflows (RB-CLOUD-004 and RB-DB-004), a scripted B5-PB_5-P observability ablation with a minimal postcondition query probe confirmed remote commit and suppressed redundant retry in 40 / 40 simulated cases without executing inverse compensation (100% simulated reconciliation success). This illustrates that observability can materially improve reconciliation on these tasks; it does not constitute a benchmark-wide or model-level performance estimate for unrestricted EvoUndo.