Paper deep dive
Outcome Monitors: Recovery Affordances for Silent Tool Failures
Sugam Panthi, Rabab Abdelfattah
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/21/2026, 3:04:08 AM
Summary
The paper introduces Outcome Monitors, a deterministic detection system for language agents that identifies violations of 'outcome contracts' (invariants mined from traces or derived from schemas) in tool results. Unlike traditional guardrails that enforce actions, Outcome Monitors issue nonbinding advisory receipts containing diagnostic witnesses and a list of public recovery tools, leaving the recovery decision to the agent. Evaluations on ToolMaze, tau-bench retail, and AppWorld demonstrate that this approach significantly improves task completion rates under silent tool failures (e.g., raising ToolMaze completion from 10.9% to 28.1%). Controls indicate that the inclusion of recovery tools, rather than diagnostic detail alone, drives these gains.
Entities (11)
Relation Signals (9)
Outcome Monitors → detects → Silent Tool Failures
confidence 95% · Outcome Monitors detect violations of outcome contracts... A cached error page or negative price can instead arrive in the expected format and be consumed as fact.
Outcome Monitors → generates → Advisory Receipt
confidence 95% · On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools.
Outcome Monitors → improves → ToolMaze
confidence 95% · Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models
Outcome Monitors → improves → tau-bench
confidence 95% · In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers.
Outcome Monitors → uses → Outcome Contracts
confidence 95% · We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas.
Advisory Receipt → contains → Recovery Tools
confidence 90% · Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.
Recovery Tools → drives → Performance gains
confidence 90% · In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.
Tags
Links
- Source: https://arxiv.org/abs/2608.19303v1
- Canonical: https://arxiv.org/abs/2608.19303v1
Trouble viewing inline? Open PDF directly →
Full Text
76,790 characters extracted from source content.
Expand or collapse full text
Outcome Monitors: Recovery Affordances for Silent Tool Failures Sugam Panthi Rabab Abdelfattah Abstract When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9%10.9\% to 28.1%28.1\% across four models in two provider families and replicate in a third. In τ-bench retail, completion improves by 14.014.0 and 12.012.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%46\%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open. 1 Introduction Language agents frequently combine reasoning with external tool calls (36; 26). This lets them perform more tasks, but it also makes them rely on information they cannot directly observe. Timeouts are easy to detect. Contradictory, stale, or physically impossible results might instead arrive in the expected format and be taken at face value: in one documented production incident, a cached HTTP error page consumed as content became a confident fabricated analysis, and 70% of that runtime’s silent failures were first caught by a human reading output (32). ToolMaze demonstrates that agents struggle to recover from such implicit failures (39). Silent failure creates two distinct reasoning problems. The system must first recognize that an observation is inconsistent and localize the violated property. Then, it must select a recovery action that advances the task. Traditional agent loops rely on the language model to infer both from the raw observation. At the opposite extreme, a runtime guard prevents actions or selects repairs, obscuring whether improved behavior stems from better reasoning or the guard’s intervention. We introduce a deterministic detector that identifies when a tool result violates a mined or public invariant and signals the violation to the acting agent. We name these invariants outcome contracts, as they specify expected relationships between tool calls and their results, mined from nominal executions or derived from public schemas. On a violation the detector appends an advisory receipt; it never restricts an action, executes a repair, reveals a benchmark fault label, or consults the evaluator. Recent contract and gate systems attach checks before execution and enforce them (19; 24; 5). Our monitor instead checks a returned result and does not enforce a response. ToolMaze contracts use task-disjoint cross-fitting, and injection changes only what the model observes; evaluators therefore score the agent’s chosen recovery rather than a repair encoded by the runtime. The resulting interface separates detection from recovery: the monitor identifies a violated outcome property and lists public recovery tools, while the language agent chooses what to do. We call these tools the signal’s recovery affordances. Three findings support the interface: 1. Detector-triggered receipts raise completion across three provider families. We find paired completion gains with Outcome Monitors on a frozen, task-disjoint ToolMaze confirmation. The effect holds for four models spanning DeepSeek and Qwen. A separately frozen MiniMax M3 replication extends it to a third provider family. 2. Benefits replicate in τ-bench and concentrate where failures block completion. We evaluate 50 τ-bench retail tasks, each subject to both fault types. Categorical violations improve sharply with no paired losses, while conservation violations show no measured effect. Across all studies, the gains concentrate where the fault blocks baseline completion. Delivery is not universally beneficial: held-out AppWorld shows no measured net effect, and clean ToolMaze controls contain both rescues and harms. 3. Recovery tools—not additional diagnostic detail—drive the detectable ToolMaze gain. Neither the baseline’s standing fault-aware prompt nor an extended reasoning budget removes the effect. A generic caution matches the localized witness, and changing placement or salience shows no detectable advantage. A warning stripped of recovery tools performs at baseline, while restoring the list restores the gain. A schema-only detector recovers much of the fault-enriched gain but fires more often; the learned contracts improve detection selectivity. We measure the remaining boundary with an incident-derived study. Outside the mined vocabulary, detection degrades but delivery does not. RAW INTERFACElookup_inventory available = -3detect · localize · recover left to the agentsilent failure value consumed as factDETECT AND ADVISElookup_inventory raw result keptcontract violated mined or public invariantADVISORY RECEIPTwitnessexpected ≥ 0, observed -3recoveryquery_warehouseget_stock_by_skuactive content(Section 5)verify · retry · switch action set intact Figure 1: A raw interface leaves detection, localization, and recovery with the agent. Detect and advise instead flags a contract violation and appends an advisory receipt carrying a diagnostic witness and a list of available recovery tools. It neither removes actions nor selects a repair. Controls (Section 5) identify the recovery-tool list, not the property-specific witness, as the active content in our tested sample. 2 Related Work Tool-using language agents. ReAct interleaves natural-language reasoning with environment actions, while Toolformer trains models to decide when and how to invoke APIs (36; 26). AgentBench and AppWorld broaden evaluation to interactive multi-step environments (18; 29), subsequent work scales tool repertoires (23), and SWE-agent shows agent-computer interface design affects task success (34); tool-hallucination work targets the agent’s own faulty call (38; 33). These lines primarily evaluate planning and tool selection under an assumed interface. We instead modify what the interface communicates after a semantically suspect result. Feedback and self-correction. Reflexion stores linguistic feedback across trials (27), CRITIC uses external tools to critique and revise outputs (10), and Self-Refine iterates with the same model as generator and critic (20); self-correction without external grounding can degrade outputs (14). Reinforced Agent places a separate LLM reviewer before execution and iteratively revises provisional tool calls (28); we compare against a mechanism-faithful port in our experiments. Process supervision trains a learned verifier to score each reasoning step (17); our step-level receipts are instead deterministic, interface-derived checks requiring no trained verifier and no extra model call. PALADIN instead trains on failure-recovery trajectories and retrieves recovery exemplars at inference time (30); ours is training-free and targets implicit inconsistencies in returned results. AgentProp-Bench reports little rank association between rejection and recovery across nine models (13), supporting the distinction between detecting a bad interaction and recovering from it. Runtime monitoring and verification. ToolEmu uses an LM-emulated sandbox to surface consequential agent risks (25). ToolMaze directly perturbs tool environments to study dynamic replanning (39). τ-bench evaluates agents in stateful retail and airline domains (35). Runtime shields enforce safety constraints during execution (2). The closest recent systems make the enforcement contrast concrete: ToolGate admits a tool call only when Hoare-style pre/postconditions verify (19); Reason Less, Verify More rejects policy-violating writes before execution (24); Agent Behavioral Contracts enforce session-level governance rules (5); and further systems supply rule languages, transactional compensation, or state-machine postconditions with recovery owned by the runtime (31; 6; 11). ToolSafe trains a guardrail model to flag unsafe invocations before execution (22). Our monitor acts after a result arrives, makes no enforcement decision, requires no trained verifier, and leaves recovery with the acting agent. Post-trajectory false-success detectors identify unsupported completion claims (1); our checker instead emits a per-call signal while the same trajectory can still recover. Design by contract and runtime assertions. Checking a computed result against a declared expectation is a long-standing software-engineering discipline: design by contract specifies pre- and postconditions a component must honor (21), higher-order contract systems attribute blame on violation (9), and runtime verification monitors execution against formal properties (16). We port the interface-checking idea to language-agent observations and explicitly choose an advisory propagation policy, so the model, not the runtime, decides how to recover. Learned invariants and specification mining. Because hand-written contracts are costly, a parallel line of work infers likely specifications from observed executions. Daikon detects likely invariants from dynamic traces (8), and specification-mining methods generalize this to temporal and API-usage properties (3). We adopt the same non-oracle stance: our contracts are mined from clean agent traces or derived from public response schemas rather than authored against a task solution. Unlike program-analysis settings, the mined invariant is not used to certify code but to explain a suspect observation to a downstream agent at inference time. Contract2Tool likewise infers preconditions and effects from documentation, schemas, and traces, but deploys them for causal tool filtering (4); we cross-fit invariants over returned outcomes and expose violations as advisory observations. Agent safety and guardrails. Recent work proposes guardrail frameworks that filter or constrain agent actions before or after execution (7; 15). A related threat is adversarial payloads carried in tool results themselves (37); our threat model is instead benign-but-inconsistent results; adversarial tools are out of scope. Although nonbinding, false-positive context can still induce the agent to take a harmful action. Strategy Trigger Grounding Localized witness Restricts actions Extra tool reads Generic caution learned detector event external invariant no no no Self-critique model’s own judgment internal uncertain no optional Always verify every advertised read none (fixed policy) only after comparison no yes Hard guard monitor-detected event external rule yes yes no Outcome Monitor learned detector event external invariant yes no no Table 1: Conceptual comparison of recovery interfaces. Shaded cells mark where a strategy differs from the Outcome Monitor row, which serves as the reference; the shading locates design differences and does not rank them. The table characterizes mechanisms, not measured superiority. Our experiments measure the advisory alternatives—generic caution, always-verify, and a reviewer-loop port—under matched scaffolds; hard enforcement remains a design contrast (Table 1). 3 Outcome Monitors The system has two roles: a detector that identifies when a tool result violates a learned or public invariant, and an advisory signal that communicates the violation to the acting agent. The mined invariants (outcome contracts) define the detector; the receipt encodes the signal. These components serve different roles: contracts determine when a result is suspect, while public recovery relations determine which alternative tools the receipt exposes. The controls isolate the receipt’s active content; they do not make contract-based detection dispensable. Interface At step t, an agent selects tool call ata_t from action set AtA_t and receives result yty_t. A contract checker C maps (at,yt)(a_t,y_t) to an ordered set VtV_t of violation records, each a machine-readable property code optionally with a path, expected/observed values, and human-readable detail. If VtV_t is empty, the interface returns yty_t unchanged. Otherwise a versioned encoder EsE_s maps (τ,Vt,Rt)(τ,V_t,R_t) to receipt ρt _t, where RtR_t contains only recovery affordances available from the public tool interface. AtA_t is unchanged. The agent may verify, retry, switch tools, ignore the receipt, or take any other advertised action (Figure 1). Algorithm 1 CheckAndReceipt: per-call contract evaluation. 0: tool τ, arguments a, result y, registry C, visible tools T, encoder EsE_s 0: augmented result; action set A unchanged 1: V←C.Check(τ,a,y)V← C. Check(τ,a,y) 2: if V=∅V= then 3: return y consistent 4: end if 5: R←τ∪s∈Subs(τ):s∈TR←\τ\∪\s∈ Subs(τ):s∈ T\ 6: ρ←Es(τ,V,R)ρ← E_s(τ,V,R) versioned serialization 7: return tool_result:y,outcome_contract:ρ\tool\_result:y,\;outcome\_contract:ρ\ Algorithm 1 formalizes the per-call runtime: deterministic given the call, result, and learned contract, and it never invokes tools, scores candidate actions, or waits for task-level failure. The frozen confirmation uses encoder Ev1E_v1; later controls use Ev2E_v2 with an explicit witness structure. Algorithm 1 shows the ToolMaze envelope; in AppWorld and τ-bench, whose tool results are plain text, the receipt is appended as text (exact envelopes in the supplement). Detector Construction The abstraction permits different non-oracle sources. In ToolMaze, five-fold cross-fitting keeps every evaluated workflow out of the traces used to build its registry. We mine those registries from nominal, unperturbed (P0) trajectories using Algorithm 2. Algorithm 2 LearnContracts: invariant mining with acceptance rules. 0: nominal traces D, K disjoint task folds D1,…,DKD_1,…,D_K 0: per-fold contract registry C1,…,CKC_1,…,C_K 1: for k=1k=1 to K do 2: Dtrain←D∖DkD_train← D D_k 3: for each tool τ observed in DtrainD_train do 4: Freq←⋂F_req← result field sets; Fall←⋃F_all← result field sets 5: types[f]←types[f]← union of kinds observed for f 6: echo: accept f iff f is in ≥2≥ 2 call/result pairs and a[f]=y[f]a[f]\!=\!y[f] in all 7: positivity: accept f∈Freqf∈ F_req iff numeric in ≥3≥ 3 samples and always >0>0 8: domain: accept f∈Freqf∈ F_req iff name matches categorical lexicon, ≥4≥ 4 string samples, ≤8≤ 8 distinct normalized values 9: affine: accept (f,g)∈Freq2(f,g)∈ F_req^2 iff ≥4≥ 4 samples with ≥3≥ 3 distinct values fit one y[g]=αy[f]+βy[g]=α\,y[f]+β (10−810^-8 tolerance) 10: order: accept (f,g)(f,g) among three fixed date pairs iff ≥2≥ 2 parsed samples and f≤gf≤ g in all 11: end for 12: Ck←C_k← registry over all accepted tool specs 13: end for Acceptance is conservative. An invariant must hold on every training sample and meet its class-specific support minimum (the categorical lexicon appears in the supplement). At runtime, the checker evaluates each admitted class and collects its violations in VtV_t; affine relations use a 10−510^-5 tolerance versus the tighter mining-time fit. Recovery relations are constructed separately from detection. We build the static mapping Subs(τ) Subs(τ) programmatically from two public, task-agnostic ToolMaze sources: per-tool substitute fields and the entry tool of each sibling path in the alternatives file. The mapping therefore uses no task descriptions, solutions, gold sequences, perturbations, or outcomes. The indexing code and full mapping ship with the artifact. At runtime, we intersect Subs(τ) Subs(τ) with the workflow’s advertised tools to form RtR_t, so a receipt never names a tool the agent cannot call. A task-disjoint audit of the detector on the experiments’ difficult C3/C4 workflow classes flags 79.6% of implicit faults at a 1.39% clean false-positive rate, with a learned contract available for 94.2% of fault outcomes; rates are per observed tool outcome, not per episode. An earlier schema-only gate reached 55% recall at an 18% false-positive rate. Section 5 measures both detectors end-to-end, so the audit characterizes detection precision rather than a completion ceiling (full audit in the supplement). For AppWorld, contracts derive from released standard API response schemas: eligible read results are validated recursively, and a receipt fires on a missing required field or a scalar type mismatch. For τ-bench retail, contracts are mined from the public database (orders.json) without reading task descriptions, gold sequences, or reward functions. The checker validates categorical domains (order status, transaction type), positivity constraints (item prices), and affine conservation relations (net payment equals item total for non-cancelled orders). These are richer than schema-only contracts but still non-oracle. In all three environments, the injector changes only the text observed by the model; environment state is never mutated, so evaluators score the true state. Construction cost. Construction is deterministic and local with zero model calls: the five cross-fitted ToolMaze registries (15,041 invariants from 400 nominal workflows), the AppWorld schema contracts, and the τ-bench database contracts each build in under two seconds (supplement). Information and Non-Oracle Boundary The runtime is prohibited from reading the task solution, perturbation label, evaluator, gold API sequence, or hidden recovery path. Contract construction uses only public tool metadata and disjoint clean traces. We audit the remaining risk of a de facto oracle by reporting the fraction of receipt events with at least two admissible public recovery actions. This test is necessary but not sufficient: it shows the interface does not usually reduce recovery to a single choice, not that every contract is correct or every action useful. 4 Experimental Design We ask three questions. RQ1: Does detector-triggered advisory feedback improve full task completion under implicit tool failures? RQ2: How does it compare to an inference-time reviewer alternative? RQ3: Does the effect persist across environments and controls, and where does it fail? Task identities, settings, and artifact hashes are frozen before the corresponding outcomes are opened, except where noted; studies frozen or amended after related outcomes were seen are labeled retrospective. ToolMaze Confirmation (RQ1) ToolMaze crosses workflow topology (C1–C4) with explicit/implicit and transient/permanent tool perturbations (39). From the difficult C3/C4 implicit-failure region we sample 80 unique workflow IDs, exactly 20 each from C3/P3, C3/P4, C4/P3, and C4/P4 (transient and persistent implicit faults), each carrying one fault mode. For each model, baseline and advisory episodes are randomized and interleaved, giving 160 episodes per model. We evaluate DeepSeek V4 Flash and V4 Pro via the direct DeepSeek API and Qwen 3.7 Plus and 3.7 Max via OpenRouter, all with identical prompts, temperature zero, and extended reasoning disabled. The primary endpoint within a model is paired full workflow completion, tested with the two-sided exact McNemar/binomial test. Because all four models see the same 80 workflows, joint inference clusters by workflow. We use an exact task-level sign-flip test and a 200,000-resample task-cluster bootstrap. A replication applies the unchanged 80-workflow method to MiniMax M3 through an exact MiniMax-hosted FP8 route on OpenRouter. It is analyzed independently and is not pooled with the four-model primary. Method Comparison (RQ2) A retrospective study compares three arms on 80 additional ToolMaze workflows: contemporaneous baseline, localized receipt, and a mechanism-faithful Reinforced Agent port (28). The port preserves separate pre-execution review, generic v1 feedback, and up to five review loops, but substitutes the matched DeepSeek tier for the source paper’s GPT-4o base and reviewer; it is therefore not an exact reproduction. Both DeepSeek tiers run all three arms. Cross-Environment Studies (RQ3) AppWorld. AppWorld provides state-based tests that recognize alternative valid solutions (29). We deterministically select one variant from each of the 19 generator families in the released development split, and both DeepSeek tiers run fault baseline, fault advisory, and clean advisory, with fault type alternating between a missing field and a scalar type flip. A held-out study then selects 32 generator families from test_normal and runs the same three conditions on DeepSeek V4 Flash. The two samples are analyzed and reported separately. τ-bench retail. τ-bench evaluates tool-calling agents in a stateful retail environment with a simulated user and state-based reward (35). A factorial samples 50 of 115 retail test tasks with seed 20260718 and applies both persistent read faults to every task: categorical status-out-of-set and affine broken-conservation. Each task runs faulted baseline and advisory under both faults plus clean advisory, yielding 250 episodes per tier at temperature zero for each of DeepSeek V4 Flash and V4 Pro on the same 50 tasks. The prespecified primary endpoint is paired completion within each fault class, and the secondary aggregate is clustered by the 50 shared tasks. An earlier exploratory study confounded fault class with task identity and suggested a harmful conservation effect; the same-task design removes that confound. Incident-derived faults. A two-tier study reruns the 80 confirmation workflows unperturbed and injects one of eight fault types. We derive those types from the five failure classes of a published taxonomy of production silent failures (32), under a documented independence protocol: the fault author saw only the taxonomy and fixture shapes, not the contract classes. Each fault lands on the tool the benchmark’s own fault corrupts, and per-cell assignment is frozen among the types expressible on that tool’s nominal output. Every injection’s delivered result and field-level diff are logged. The prespecified primary is intention-to-treat paired completion, and detection is a receipt on the injected tool. Because a post-run audit found imprecise placement, an earlier run was invalidated; the amended protocol was frozen after that run’s outcomes were known and is disclosed as such (supplement). Controls Five control studies probe the receipt’s active components, all using the Ev2E_v2 encoder and paired workflow-clustered analyses. Three run on 57 reserved workflows: a specificity control (generic warning vs. localized receipt), an always-verify control (deterministic substitute after each advertised read, frozen after specificity outcomes and therefore retrospective), and a clean-P0 control (no fault, with the manifest frozen before any P0 outcome on these workflows was opened). Two more run on 57 fresh workflows, with interpretations registered before execution: a four-arm matched-trigger placebo varying only when a fixed generic warning attaches, and a three-arm stripped-envelope rerun that removes the recovery-tools list, because a post-run audit found the always-warn envelope leaks affordances. 5 Results Prespecified, frozen tests are identified as such below; retrospective and post hoc analyses are labeled where they appear. Inferential statistics for the remaining control and boundary analyses are descriptive. Detector-Triggered Feedback Replicates Across Three Families Family Model Base Monitor Δ W/L DeepSeek V4 Flash 14/80 27/80 +16.25 15/2 DeepSeek V4 Pro 13/80 23/80 +12.50 14/4 Qwen 3.7 Plus 3/80 15/80 +15.00 12/0 Qwen 3.7 Max 5/80 25/80 +25.00 21/1 Primary aggregate 35/320 90/320 +17.19 62/7 MiniMax M3 replication 5/80 20/80 +18.75 17/2 Table 2: Frozen paired ToolMaze completion. The four-model primary aggregate clusters shared tasks; MiniMax M3 is a separately frozen third-family replication and is not pooled. Δ is in percentage points; W/L counts discordant pairs. All models run with extended reasoning disabled. Primary task-cluster sign-flip p<.00001p<.00001; 95% bootstrap interval [11.25,23.44][11.25,23.44]. The four-model primary is positive throughout. Table 2 shows a positive, individually significant effect for every model, and all 16 prespecified model-by-stratum point estimates are nonnegative. Baseline succeeds on 35/320 model-workflow pairs (10.9%) and the advisory on 90/320 (28.1%): a clustered mean effect of +17.2+17.2 percentage points (95% bootstrap interval [11.25,23.44][11.25,23.44], sign-flip p<.00001p<.00001). The task remains hard under the advisory, and 230/320 episodes still fail. The separate third-family replication agrees. MiniMax M3 improves from 5/80 to 20/80 completions, +18.75+18.75 points with 17 paired wins and two losses (exact p=.00073p=.00073). All four prespecified stratum estimates are nonnegative. All advisory wins contain at least one detector event. Of 484 events, 443 (91.5%) retain at least two admissible recovery actions. After a receipt, the agent switches to a listed substitute in 48.6% of events, calls another tool in 33.5%, retries in 14.0%, and makes no further tool call in 3.9%. The seven aggregate losses also contain receipts, so localization does not guarantee replanning. A reasoning ablation with 8,192-token thinking budgets shows effects persist on both tiers (95% interval [10.00,25.62][10.00,25.62], combined sign-flip p<.0001p<.0001): neither standing fault awareness nor extended reasoning substitutes for the delivered signal. Both arms receive the same unmodified, fault-aware system prompt, which already instructs the model to inspect unexpected tool outputs; no system instruction mentions contracts or receipts. The intervention is entirely in-band in the returned tool result. Benefits Replicate in a Stateful Retail Environment 0255075100task completion (%)baselineadvisorysameToolMaze implicit (4 models)++17.2τ-bench status++28.0ToolMaze clean P0CLEAN0.0AppWorld type flipDEV++16.7τ-bench conservation0.0AppWorld missing fieldDEV0.0 Figure 2: Paired baseline and advisory completion by study, sorted by baseline rate. Open circles mark baseline and filled circles mark advisory; concentric markers indicate identical rates. Development rows are labeled dev and are not frozen tests. Rows are not pooled; counts appear in the supplement. A same-task factorial establishes a second positive environment. In the τ-bench factorial on DeepSeek V4 Flash, aggregate completion rises from 41/100 to 55/100: +14.0+14.0 points (95% bootstrap interval [3.0,25.0][3.0,25.0], task-clustered sign-flip p=.028p=.028). Categorical status violations improve from 6/50 to 20/50, +28.0+28.0 points with 14/0 W/L (exact p=.00012p=.00012). Broken-conservation violations hold at 35/50 in both arms, an estimate of 00 points, though the 9/9 W/L split shows pairs moving in both directions (p=1.0p=1.0). A matched DeepSeek V4 Pro run on the same 50 tasks confirms the pattern: +12.0+12.0 points overall (p=.023p=.023), categorical +18.0+18.0 points (p=.004p=.004), conservation +6.0+6.0 points (p=.607p=.607). The conservation null is consistent with any effect below the ≈23≈23 points single-tier minimum detectable effect at 80% power; per-tier breakdowns and restriction analyses appear in the supplement. Clean advisory emits one spurious receipt across 100 episodes on the two tiers. Recovery Tools Carry the Detectable Receipt Gain RECOVERY TOOLS ADDEDreceipt −- stripped++11.4receipt −- baseline++12.3RECOVERY TOOLS UNCHANGEDstripped −- baseline++0.9localized −- generic−-2.6on-time −- deferred++0.9−-1001020completion difference (points), 95% interval Figure 3: Contrast estimates with 95% bootstrap intervals on 114 paired workflows (57 per tier). Filled markers clear zero; open markers do not. The three null contrasts have an ≈18≈18 point minimum detectable effect at 80% power, bounding rather than establishing equivalence. The recovery-tool list carries the detectable gain. Five matched contrasts vary one part of the receipt on the same 114 workflow pairs (Figure 3). In the registered three-arm rerun, the stripped receipt performs at baseline; restoring the recovery-tool list adds +11.4+11.4 points (p=.028p=.028). Witness detail and deferred timing produce no detectable differences, though both nulls are power-bounded (≈18≈18 points MDE). The always-warn comparison is confounded because it supplies recovery tools at every step. Targeting the same content with the detector reduces warning volume from ≈600≈600 per tier to 80–91 without a detected completion loss (per-tier estimates in the supplement). Schema-only detector ablation. A contemporaneous three-arm study reruns the 80 confirmation cells on DeepSeek V4 Flash. The learned arm replicates the confirmation (13/8013/80 to 25/8025/80, +15.0+15.0 points, p=.0075p=.0075). The schema-only arm recovers much of the gain (21/8021/80, +10.0+10.0 points, 11/3 W/L, p=.057p=.057); the head-to-head difference is +5.0+5.0 points (9/5 W/L, p=.42p=.42). Detection behavior differs sharply: the schema arm fires 166 receipts to the learned arm’s 111, with only 12.0% listing multiple recovery actions versus 91.9%. A companion clean-P0 study finds no measured completion harm from the schema arm’s extra false triggers (p=.375p=.375). Clean behavior. On clean P0 workflows, advisory and baseline each complete 74/114 pairs, an estimate of 00 points (95% interval [−5.26,5.26][-5.26,5.26]). The advisory emits 16 false-trigger receipts in 14 treated pairs, with five paired rescues and five paired harms (p=1.0p=1.0), so clean traffic is not harm-free. Receipts Outperform the Evaluated Reviewer Port Receipts outperform the reviewer port on these workflows, at a fraction of its cost. In the retrospective 80-workflow comparison, full monitor receipts improve over baseline by +15.0+15.0 points (95% bootstrap interval [6.875,23.75][6.875,23.75], sign-flip p=.0011p=.0011) and over a mechanism-faithful Reinforced Agent port by +11.25+11.25 points ([3.125,19.375][3.125,19.375], p=.0144p=.0144); the reviewer improves over baseline by +3.75+3.75 points ([−2.5,10.0][-2.5,10.0], p=.349p=.349). The receipt adds $0.13 in all-cache-miss list price (+7%+7\% of baseline cost) with no additional model call; the reviewer adds $9.24 (+498%+498\%) through up to five additional calls. The port is not an exact reproduction, so a calibrated reviewer could be more accurate (per-tier counts in the supplement). Where Benefits and Detection Reach Their Limits Gains concentrate where faults block completion. Figure 2 synthesizes the frozen studies alongside two development rows. When faulted-baseline completion is at most 16% (ToolMaze 10.9%, τ-bench categorical 12.0%), net gains are +15.0+15.0 to +28.0+28.0 points. When most baselines complete despite the fault (conservation 70.0%, held-out AppWorld 78.1%), net effects are +6.3+6.3 points or less with rescues offset by harms. Conditional rescue rates are 21.8–34.6% in seven of eight rows; τ-bench Flash conservation is the outlier (60%, 9/15). This synthesis is post hoc and descriptive, not a deployment criterion. Held-out AppWorld: delivery without net gain. In the 19-family development characterization, type-flip faults produce same-sign gains on both DeepSeek tiers, missing-field completion is unchanged, and the schema checker emits zero receipts across 38 clean episodes. In a held-out 32-family test_normal study on DeepSeek V4 Flash, baseline and advisory each complete 25/32 families (p=1.0p=1.0). The checker fires in all 32 faulted advisory episodes, but 23 pairs complete despite the fault, leaving little room for rescue. Per-fault directions and development details appear in the supplement. Incident-derived faults expose the detection boundary. The suite transcribes fault content from real production incidents, authored blind to the contract vocabulary. The unchanged detector flags 46.1% and 45.3% of injected faults on Flash and Pro. Recall is 83% (25/30) for violations expressible over structured values but 22% (6/27) for corruption inside plausible strings. Completion is null in both tiers (p=1.0p=1.0), where faulted baselines already complete 67.5% and 65.0% of episodes. Yet corrupted values propagate in roughly one in five trackable episodes, and about two thirds of those episodes still pass. The contracts’ vocabulary therefore bounds detection, while task completion can miss downstream corruption. Organic traffic evaluates detection, not recovery. On 53,078 recorded StableToolBench responses from 427 public tools, with no agent or counterfactual, the unchanged detector fires on 0.80%, concentrated in 54 tools, versus 1.39% on synthetic clean workflows (12). Flags include HTTP errors inside transport-success responses and empty payloads missing learned fields; not every flag is adjudicated as a true failure. This measures detection, not recovery. 6 Discussion and Limitations Why nonbinding matters. Because the receipt is nonbinding, a false positive can only add context and salience; every action stays available. That does not eliminate harm: the paired losses show an agent can overreact to correct but irrelevant information, so preserving the action space is not a formal safety shield. Deployment value depends on fault prevalence. Every positive study uses a fault-enriched sample, so we bound the deployment mixture by fault prevalence π. At the frozen point estimates the advisory is net-nonnegative at every prevalence; at the unfavorable interval endpoints of both studies a positive net effect requires π≳0.32π 0.32 (supplement). At low prevalence the case rests on the clean effect staying near its observed zero. A deployment should therefore audit both its clean receipt rate and fault prevalence before relying on the observed net benefit. What the signal contributes. The controls point to recovery tools as the active content in these ToolMaze samples; null contrasts for witness detail, timing, and salience remain power-bounded. The schema ablation suggests that learned contracts buy selectivity and better recovery lists rather than a clearly larger completion effect. Detector–injector alignment. We designed the τ-bench and AppWorld faults so that their contract sources could express them, so those studies test whether the signal is actionable, not whether the detector finds arbitrary faults. ToolMaze draws on the benchmark’s own fault taxonomy and on generic Daikon-lineage properties, but the miner and the injector may still be aligned. The incident-derived suite addresses that: on faults authored blind to the vocabulary, detection falls to ≈46%≈46\%, high on violations a value can express and low inside plausible strings. No study here measures recovery from failures that occur naturally. Limitations. No episode here runs against a live deployment. What we inject is placement and prevalence rather than fault content: the incident suite’s eight types come from a published taxonomy of production silent failures (32), authored blind to the contract vocabulary, and the unchanged detector also runs on 53,078 recorded public-API responses. Recovery from organic failures remains open. ToolMaze spans five models from three provider families, but τ-bench covers one family and the signal decomposition only two DeepSeek tiers. Contracts inherit their sources’ blind spots and false positives: schema contracts cover fewer modes, and AppWorld’s development sample leans on two domains. Completion is a blunt measure too. Clean controls record paired harms even where the aggregate effect is zero, and an episode that propagates a corrupted value can still pass. Nothing here establishes safety under adversarial tools (37), privacy failures, or irreversible actions. Reproducibility. The supplement and anonymous artifact report the full 4,683-episode study inventory together with sanitized manifests, aggregate analyses, code, hashes, and invalidation reports. Row-level provider and protected benchmark observations are excluded from the review package. 7 Conclusion Outcome Monitors detect contract violations and expose recovery tools without restricting the agent. Gains replicate across three provider families and two environments but concentrate where faults block completion. The results frame silent-failure recovery as an observability problem: viable actions may already exist, but the interface must make them legible. Detection beyond the contract vocabulary remains open. References Advani (2026) L. Advani From confident closing to silent failure: characterizing false success in LLM agents. Note: Accepted at the ICML 2026 Workshop on Failure Modes in Agentic AI External Links: 2606.09863, Link Cited by: §2. Alshiekh et al. (2018) M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. Cited by: §2. Ammons et al. (2002) G. Ammons, R. Bodík, and J. R. Larus Mining specifications. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), p. 4–16. Cited by: §2. Babu and Iyer (2026) R. S. Babu and L. G. Iyer Contract2Tool: learning preconditions and effects for reliable tool-augmented LLM agents. External Links: 2606.07904, Link Cited by: §2. Bhardwaj (2026) V. P. Bhardwaj Agent behavioral contracts: formal specification and runtime enforcement for reliable autonomous AI agents. External Links: 2602.22302, Link Cited by: §1, §2. Chang and Geng (2025) E. Y. Chang and L. Geng SagaLLM: context management, validation, and transaction guarantees for multi-agent LLM planning. Proceedings of the VLDB Endowment 18, p. 4874–4886. External Links: Document Cited by: §2. Dong et al. (2024) Y. Dong, R. Mu, Y. Zhang, S. Sun, T. Zhang, C. Wu, G. Jin, Y. Qi, J. Hu, J. Meng, S. Bensalem, and X. Huang Safeguarding large language models: a survey. External Links: 2406.02622, Link Cited by: §2. Ernst et al. (2007) M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao The Daikon system for dynamic detection of likely invariants. Science of Computer Programming 69 (1–3), p. 35–45. Cited by: §2. Findler and Felleisen (2002) R. B. Findler and M. Felleisen Contracts for higher-order functions. In Proceedings of the ACM SIGPLAN International Conference on Functional Programming (ICFP), p. 48–59. Cited by: §2. Gou et al. (2024) Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen CRITIC: large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations, External Links: Link Cited by: §2. Guo et al. (2026) L. Guo, W. Liu, Y. W. Heng, T. P. Chen, and Y. Wang Agent-SAMA: state-aware mobile assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 29459–29467. Cited by: §2. Guo et al. (2024) Z. Guo, S. Cheng, H. Wang, S. Liang, Y. Qin, P. Li, Z. Liu, M. Sun, and Y. Liu StableToolBench: towards stable large-scale benchmarking on tool learning of large language models. In Findings of the Association for Computational Linguistics (ACL), p. 11143–11156. Cited by: §5. Gurram (2026) B. Gurram Auditing automated evaluation, error propagation, and runtime mitigation in tool-using language agents. External Links: 2604.16706, Link Cited by: §2. Huang et al. (2024) J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations, Cited by: §2. Inan et al. (2023) H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa Llama guard: LLM-based input-output safeguard for human-AI conversations. External Links: 2312.06674 Cited by: §2. Leucker and Schallhart (2009) M. Leucker and C. Schallhart A brief account of runtime verification. Journal of Logic and Algebraic Programming 78 (5), p. 293–303. Cited by: §2. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Cited by: §2. Liu et al. (2024) X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang AgentBench: evaluating LLMs as agents. In International Conference on Learning Representations, Cited by: §2. Liu et al. (2026) Y. Liu, X. Peng, J. Cao, X. Wang, S. Deng, J. Chen, J. Yin, and X. Zhang ToolGate: contract-grounded and verified tool execution for LLMs. In Findings of the Association for Computational Linguistics: ACL 2026, p. 9653–9684. External Links: Document, Link Cited by: §1, §2. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark Self-refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §2. Meyer (1992) B. Meyer Applying “design by contract”. Computer 25 (10), p. 40–51. External Links: Document Cited by: §2. Mou et al. (2026) Y. Mou, Z. Xue, L. Li, P. Liu, S. Zhang, W. Ye, and J. Shao ToolSafe: enhancing tool invocation safety of LLM-based agents via proactive step-level guardrail and feedback. In Findings of the Association for Computational Linguistics: ACL 2026, p. 37125–37153. External Links: Document Cited by: §2. Qin et al. (2024) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerber, D. Li, Z. Liu, and M. Sun ToolLLM: facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations, Cited by: §2. Reddy et al. (2026) V. Reddy, S. R. Challaram, and A. Basu Reason less, verify more: deterministic gates recover a silent policy-violation failure mode in tool-using LLM agents. Note: Accepted at the KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI External Links: 2607.07405, Link Cited by: §1, §2. Ruan et al. (2024) Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto Identifying the risks of LM agents with an LM-emulated sandbox. In International Conference on Learning Representations, Cited by: §2. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §1, §2. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §2. Ta et al. (2026) A. Ta, J. Zhu, and S. Shayandeh Reinforced agent: inference-time feedback for tool-calling agents. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics, p. 136–147. External Links: Document, Link Cited by: §2, §4. Trivedi et al. (2024) H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian AppWorld: a controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 16022–16076. External Links: Document Cited by: §2, §4. Vuddanti et al. (2025) S. V. Vuddanti, A. Shah, S. K. Chittiprolu, T. Song, S. Dev, K. Zhu, and M. Chaudhary PALADIN: self-correcting language model agents to cure tool-failure cases. Note: Accepted oral paper at the non-archival AAAI 2026 Workshop on LLM-based Multi-Agent Systems External Links: 2509.25238, Link Cited by: §2. Wang et al. (2026) H. Wang, C. M. Poskitt, and J. Sun AgentSpec: customizable runtime enforcement for safe and reliable LLM agents. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), Cited by: §2. Wu (2026) W. Wu When errors become narratives: a longitudinal taxonomy of silent failures in a production LLM agent runtime. External Links: 2606.14589, Link Cited by: §1, §4, §6. Xu et al. (2025) H. Xu, Z. Zhu, L. Pan, Z. Wang, S. Zhu, D. Ma, R. Cao, L. Chen, and K. Yu Reducing tool hallucination via reliability alignment. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Cited by: §2. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §2. Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ-Bench: a benchmark for tool-agent-user interaction in real-world domains. External Links: 2406.12045, Link Cited by: §2, §4. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Zhan et al. (2024) Q. Zhan, Z. Liang, Z. Ying, and D. Kang InjecAgent: benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics (ACL), Cited by: §2, §6. Zhang et al. (2024) Y. Zhang, J. Chen, J. Wang, Y. Liu, C. Yang, C. Shi, X. Zhu, Z. Lin, H. Wan, Y. Yang, T. Sakai, T. Feng, and H. Yamana ToolBeHonest: a multi-level hallucination diagnostic benchmark for tool-augmented large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 11388–11422. Cited by: §2. Zhu et al. (2026) D. Zhu, X. Ma, Y. Shen, X. Li, Y. Zhao, S. Wang, L. Yan, and D. Yin When tools fail: benchmarking dynamic replanning and anomaly recovery in LLM agents. External Links: 2606.05806, Link Cited by: §1, §2, §4. Supplementary Material: Outcome Monitors Outcome Monitors: Recovery Affordances for Silent Tool Failures This supplement accompanies the main paper and provides additional tables, analysis details, and reproducibility information. The main paper is self-contained; this material supports deeper inspection. Appendix A Reasoning Ablation Model Think Base Contract Δ W/L V4 Flash off 14/80 27/80 +16.25 15/2 V4 Flash 8k 13/80 25/80 +15.00 15/3 V4 Pro off 13/80 23/80 +12.50 14/4 V4 Pro 8k 14/80 30/80 +20.00 16/0 Think-on combined 27/160 55/160 +17.50 31/3 Table 3: Contract effects persist in every configuration. Reasoning ablation on DeepSeek V4 with an 8,192-token thinking budget. Combined thinking-on task-cluster sign-flip p<.0001p<.0001; 95% bootstrap interval [10.00,25.62][10.00,25.62]. Appendix B Paired Outcome Decomposition Environment Model Rescue Harm Both pass Both fail ToolMaze DS Flash 15 2 12 51 ToolMaze DS Pro 14 4 9 53 ToolMaze Qw Plus 12 0 3 65 ToolMaze Qw Max 21 1 4 54 ToolMaze repl. MiniMax M3 17 2 3 58 AppWorld dev DS Flash 3 1 13 2 AppWorld dev DS Pro 2 1 14 2 AppWorld test DS Flash 2 2 23 5 Table 4: Paired outcome decomposition. Rescue means advisory-only success; harm means baseline-only success. AppWorld development and held-out test samples remain separate. MiniMax M3 is a separately frozen replication and is not pooled with the four-model primary. Appendix C Post-Receipt Recovery Behavior First action after receipt Events Share Listed recovery substitute 235 48.6% Different tool 162 33.5% Retry receipted tool 68 14.0% No further tool call 19 3.9% Table 5: Descriptive first-action distribution after 484 receipt events in the four frozen ToolMaze confirmations. The categories characterize retained recovery behavior; they do not identify an effect of receipt wording. Appendix D Reviewer-Port Comparison Counts Strategy Flash Pro List cost Baseline 16/80 13/80 $1.853 Localized receipt 27/80 26/80 $1.984 Reinforced Agent port 19/80 16/80 $11.089 Table 6: Matched ToolMaze comparison on 80 separately hashed workflows (main paper, Results). Counts are completed workflows per tier; cost is the two-tier all-cache-miss list price. The port substitutes matched DeepSeek tiers for the source paper’s GPT-4o and is not an exact reproduction. Appendix E Detector Development and Audit ToolMaze detector Impl. recall Clean FPR Coverage Schema-only gate 55.0% 18.0% — Learned C2 gate 80.0% 4.0% — C3/C4 cross-fit audit 79.6% 1.39% 94.2% Table 7: The initial schema-only gate failed its pre-model audit. Outcome-level detector development and task-disjoint audit. Coverage is the share of implicit fault outcomes with a learned contract available; recall given coverage is 84.5% in the C3/C4 audit. The audit rate is per observed outcome rather than per episode, so repeated opportunities accumulate. In the separate clean-P0 control they reach 16 false contract events in 14/114 treated episodes, with five paired harms and five paired rescues. Appendix F Schema-Only Detector Ablation Contrast Completion Δ W/L p Learned vs. baseline 13/80 → 25/80 +15.0 15/3 .0075 Schema-only vs. baseline 13/80 → 21/80 +10.0 11/3 .0574 Learned vs. schema-only 21/80 → 25/80 +5.0 9/5 .4240 Table 8: Frozen three-arm schema ablation (protocol OUTCOME_CONTRACT_SCHEMA_ABLATION_80_V1, manifest hash 592d5a43) on the exact 80 frozen confirmation cells, DeepSeek V4 Flash, all three arms contemporaneous, temperature 0. The learned arm contemporaneously replicates the frozen confirmation effect. Mechanism: the learned arm emits 111 receipt events on 67 cells (91.9% listing ≥2≥ 2 admissible recovery actions); the schema-only arm emits 166 events on 65 cells (12.0%), because the public registry carries no recovery-start relations. Every cell is faulted, so this sample does not exercise the schema detector’s 18% clean false-positive rate. Appendix G Incident-Derived Fault Suite Derivation and independence protocol. Eight fault types were transcribed from the five-class production-incident taxonomy of Wu (2026, arXiv:2606.14589; 22 public postmortems). The fault author was shown only the taxonomy section of that paper, two P0 fixture files, and the transformation API contract, never the contract classes, learned registries, or this paper. They returned deterministic transformation algorithms with per-fault incident provenance. The returned specification was implemented verbatim (two internal example/algorithm inconsistencies were resolved in favor of the algorithm text and logged). Per cell, the fault is injected on the first eligible result from the exact tool(s) the benchmark’s own P3/P4 fault corrupts. Assignment is frozen among the fault types that change the target tool’s nominal fixture output, recomputed by the manifest validator from hashed fixtures. Every injection records the delivered result and a field-level before/after diff. Invalidated first run (disclosure). A first run of this suite used step-count placement. A post-run audit then found that only 58% of its injections hit the benchmark-targeted tool, that its detection count included receipts on non-injected tools, and that delivered content was not recorded. The run was therefore invalidated, and its numbers are not reported. The amended protocol (tool-level targeting, same-tool detection, delivered-content audit trail, eligibility-aware assignment) was frozen after that run’s outcomes were known. Endpoint Flash Pro Primary ITT (base → adv.) 54/80 → 54/80 52/80 → 52/80 paired W/L, exact p 3/3, p=1.0p=1.0 4/4, p=1.0p=1.0 Injected episodes 152/160 149/160 Same-tool detection 35/76 (46.1%) 34/75 (45.3%) Coincidental-receipt episodes 11 10 Contaminated / trackable (base) 12/61 15/61 of which passed 7 10 Table 9: Incident-derived fault suite, protocol OUTCOME_CONTRACT_INCIDENT_FAULT_80_V2 (Flash manifest f4e9924a, Pro manifest d39bd825), 160 episodes per tier on the 80 frozen confirmation workflows with bit-identical per-task fault assignment across tiers. Contamination is a string-match lower bound: a distinctive injected value (≥8≥ 8 characters) appearing in the trace demonstrates propagation into a later real tool call or the final answer, because the engine logs only pre-injection results. Non-injections are attributed: target tool never called (3 episodes per arm on Flash; 5 baseline and 4 advisory episodes on Pro) or live target result ineligible for the assigned transform (1 episode per arm, both tiers). Class Fault type Det. (Flash) Det. (Pro) C vacuous_exec_swallow 10/10 8/8 B positional_field_shift 8/10 8/10 E stale_declared_state 7/10 7/10 C cause_stripped_http_payload 3/9 4/10 C error_page_as_payload 3/10 3/10 D context_pollution_fabrication 3/10 3/10 B ref_id_substitution 1/9 1/9 A env_threshold_truncation 0/8 0/8 Table 10: Same-tool detection by fault type (denominators are changed injections). Classes follow Wu (2026): A environment quirks, B design-assumption mismatches, C error swallowing/dilution, D chained fabrication, E operational omission. The gradient separates value-level violations (caught by domain, positivity, and echo invariants) from content-level corruption inside plausible strings (missed). Appendix H Cross-Environment Counts Env. Fault class Base Adv. Δ Status ToolMaze implicit (4 models) 35/320 90/320 +17.2 frozen τ-bench status (Flash) 6/50 20/50 +28.0 frozen τ-bench conserv. (Flash) 35/50 35/50 0.0 frozen AppWorld dev type flip 12/18 15/18 +16.7 dev AppWorld dev missing field 17/20 17/20 0.0 dev Table 11: Underlying counts for the main paper’s descriptive synthesis. Rows are not pooled. The figure’s 95% intervals are task-cluster bootstrap intervals from the same run ledgers: τ-bench status [16.0,40.0][16.0,40.0], conservation [−16.0,16.0][-16.0,16.0]; AppWorld type flip [0.0,33.3][0.0,33.3], missing field [−20.0,20.0][-20.0,20.0]; ToolMaze uses the frozen confirmation interval [11.25,23.44][11.25,23.44]. The τ-bench intervals resample the 50 tasks shared by both fault classes. Appendix I AppWorld Studies The development characterization and held-out confirmation are distinct studies. The former covers 19 generator families on both DeepSeek tiers and is used for fault-class analysis. The latter freezes 32 families from test_normal and evaluates DeepSeek V4 Flash only. Its manifest listed both DeepSeek tiers as planned, but no Pro episodes were executed or analyzed. Sample Base Adv. W/L Clean Inj. Dev, Flash 14/19 16/19 3/1 15/19 100% Dev, Pro 15/19 16/19 2/1 16/19 100% Held-out test, Flash 25/32 25/32 2/2 26/32 100% Table 12: The held-out test_normal study has zero net effect (McNemar p=1.0p=1.0) despite fault injection in all 64 faulted episodes. AppWorld studies are reported separately. Injection is the fraction of faulted episodes receiving the protocol-specified corruption. Fault-Type Breakdowns Sample Fault Model Base Contract Δ Dev Missing field Flash 8/10 8/10 0 Dev Missing field Pro 9/10 9/10 0 Dev Type flip Flash 6/9 8/9 +22.2 Dev Type flip Pro 6/9 7/9 +11.1 Held-out Missing field Flash 12/16 14/16 +12.5 Held-out Type flip Flash 13/16 11/16 −-12.5 Table 13: The development by-fault direction does not replicate in the held-out sample. AppWorld completion by fault assignment, with development and held-out samples kept separate. Appendix J τ-bench Detailed Results The frozen factorial samples 50 retail test tasks and applies both persistent get_order_details faults to every task, so each task contributes two baseline/advisory pairs plus one clean-advisory episode. The 250-episode DeepSeek V4 Flash run completes without errors: all 200 faulted episodes fire the intended corruption, and none of the 50 clean episodes emits a receipt. Fault class Base Contract Δ W/L Categorical status 6/50 20/50 +28.0 14/0 Conservation 35/50 35/50 0.0 9/9 Aggregate 41/100 55/100 +14.0 23/9 Table 14: Frozen same-task τ-bench factorial. The categorical primary comparison has exact paired p=.00012p=.00012. The secondary aggregate clusters both fault contrasts by task (p=.028p=.028; 95% bootstrap interval [3.0,25.0][3.0,25.0]). An earlier 115-task-per-tier exploratory run assigned fault class by task-index parity. Its harmful conservation estimate is retained as audit history but does not persist in the deconfounded factorial design. Appendix K Control Study Counts and Power Analysis Control Flash Pro Joint Δ [95% CI] Always verify 11→ 12 16→ 18 +2.63 [−2.63-2.63,7.89] Clean P0 advisory 38→ 40 36→ 34 0.00 [−5.26-5.26,5.26] Table 15: Separately hashed ToolMaze controls, each with 57 workflows per tier. Counts are baseline→ completions; joint Δ and its interval are in percentage points; intervals cluster the shared tasks. Always verify is retrospective and has only 19 verification reads. The specificity control’s joint counts (generic caution 43/114, localized 40/114, Δ=−2.63 =-2.63 [−11.40-11.40,6.14]) appear in the main text. The control studies (specificity, always-verify, clean-P0, τ-bench conservation) all report null results, so what matters is how much each null can rule out. The minimum detectable effect (MDE) at 80% power quantifies that. Control study n pairs Observed Δ MDE Specificity (localized vs. generic) 114 −-2.63 p 18 p Always-verify 114 +2.63 p 19 p Clean-P0 114 0.0 p 17 p Placebo timing (true vs. placebo) 114 +0.88 p 18 p Salience (stripped vs. baseline) 114 +0.88 p 18 p τ-bench conservation 50 0.0 p 23 p Table 16: Simulation-based MDE (sign-flip test, 10,000 simulations per delta, α=.05α=.05, seed 42). All controls are underpowered to detect effects below ≈ 17≈\,17–2323 percentage points; null results cannot be interpreted as equivalence. The timing and salience contrasts of the decomposition studies below share the specificity design (114 pairs, sign-flip) and therefore its ≈18≈18 percentage points MDE. Appendix L Warning-Signal Decomposition Controls Matched-trigger placebo. Protocol OUTCOME_CONTRACT_MATCHED_PLACEBO_57_V1 samples 57 fresh C3/C4 workflows from the task IDs appearing in no prior manifest (strata c3/P3 14, c3/P4 15, c4/P3 14, c4/P4 14) and runs four arms per tier (228 episodes per tier). All warned arms deliver the byte-identical generic caution of the specificity control inside the same envelope; the arms differ only in when the warning attaches. The matched placebo suppresses the warning at each detected violation, forwards the violating result raw, and delivers one queued identical warning at the nearest following detector-consistent result, the only online-implementable position match. Of the suppressed violations, 75 of 77 (Flash) and 80 of 82 (Pro) had their warning delivered, and any undelivered pending warning is recorded. Arm Flash Pro Baseline 12/57 14/57 True trigger 16/57 17/57 Matched placebo 17/57 15/57 Always warn 20/57 26/57 Table 17: Matched-trigger placebo completions. Registered primary (true trigger minus matched placebo, two-tier workflow clusters): +0.88+0.88 percentage points, sign-flip p=1.0p=1.0, 95% bootstrap interval [−4.39,6.14][-4.39,6.14]; per-tier W/L 3/4 (Flash, p=1.0p=1.0) and 3/1 (Pro, p=.625p=.625). Secondary arm-minus-baseline estimates: true trigger +7.0+7.0 (p=.344p=.344) and +5.3+5.3 (p=.453p=.453); matched placebo +8.8+8.8 (p=.180p=.180) and +1.8+1.8 (p=1.0p=1.0); always warn +14.0+14.0 (p=.039p=.039) and +21.1+21.1 (p=.004p=.004) for Flash and Pro, respectively. Always-warn confound (audit disclosure). The registered envelope attaches admissible_recovery_tools to every warned result. A post-run audit found that no other channel exposes substitute names (0 of 270 tool descriptions) and that on Pro nine of the fourteen always-warn paired wins first call an envelope-surfaced substitute. Excluding those wins moves the always-warn-versus-baseline p from .004.004 to .45.45 (Flash: .039.039 to .065.065). The exclusion was not registered and is diagnostic only; the stripped-envelope study tests the hypothesis prospectively. The detector is additionally silent in 9 of 57 workflows, where only a constant arm delivers any warning, so part of the always-warn margin is recall coverage rather than timing. Stripped-envelope study. Protocol OUTCOME_CONTRACT_STRIPPED_ALWAYS_WARN_57_V1 reruns the same 57 cells contemporaneously with three arms per tier: baseline, the full always-warn envelope, and the identical envelope with the admissible_recovery_tools key absent. The registered primary is always-warn-stripped minus baseline; the registered secondaries are full minus stripped (the leakage estimate) and full minus baseline (the contemporaneous replication). Interpretations for each outcome were frozen before execution. Arm Flash Pro Baseline 17/57 12/57 Always warn (full) 19/57 24/57 Always warn (stripped) 14/57 16/57 Table 18: Stripped-envelope study completions. Two-tier joint contrasts: stripped minus baseline +0.88+0.88 percentage points (p=1.0p=1.0, [−6.14,7.89][-6.14,7.89]); full minus stripped +11.40+11.40 (p=.028p=.028, [2.63,20.18][2.63,20.18]); full minus baseline +12.28+12.28 (p=.018p=.018, [3.51,21.93][3.51,21.93]). Per-tier full minus stripped: Flash +8.8+8.8 (9/4, p=.267p=.267), Pro +14.0+14.0 (10/2, p=.039p=.039). Pro reproduces its frozen always-warn estimate (12→2412→ 24, 13/1, p=.0018p=.0018); Flash does not (17→1917→ 19, +3.5+3.5, p=.774p=.774), its contemporaneous baseline sitting five completions above the frozen value with a stable always-warn arm. Warned arms fire on every tool result (617/599 full and 550/540 stripped events for Flash/Pro, against 80–91 for the triggered arm of the placebo study). Envelope integrity is verified per event in both ledgers: the full arm carries the affordance key in every event and the stripped arm in none. Both tiers complete with zero infrastructure retries. Appendix M Per-Corruption-Type Detector Recall The cross-fitted audit covers nine corruption types across the C3/C4 region, and recall varies substantially by fault class. Corruption type n faults Recall Coverage Logic violation 408 91.2% 97.5% Physics violation 463 85.3% 93.5% Temporal contradiction 393 82.4% 95.9% Missing required fields 328 80.2% 95.7% Tool leakage 347 79.8% 87.3% Internal contradiction 195 77.9% 94.9% Fact contradiction 596 76.2% 92.3% Irrelevant response 323 70.3% 96.6% Statistical inconsistency 354 70.1% 94.9% Table 19: Per-corruption-type detector recall (C3/C4 cross-fitted audit, sorted by recall). Coverage is the share of fault outcomes with a learned contract available; recall given coverage exceeds 73% for all types. Appendix N Prompts and Receipt Texts This section reproduces the exact texts the models received. Unicode em dashes and one emoji glyph in the original source strings are rendered below as -- and [emoji]; everything else is verbatim. ToolMaze agent system prompt Baseline and advisory arms use the identical, unmodified fault-aware system prompt from the pinned ToolMaze checkout (evaluation/agents/openai_agent.py), shown in Figure 4. No arm receives any instruction mentioning contracts, receipts, or the outcome_contract field: the receipt is purely in-band. ToolMaze agent system prompt (P1–P4, identical in baseline and advisory arms) You are an autonomous AI assistant that executes tool-based workflows. Tools may occasionally fail or return unexpected data. TASK COMPLETION - The user query may describe multiple sub-tasks. You must complete ALL of them -- do NOT skip any sub-tasks. - Every fact, data retrieval, or state change must come from a tool call. Never substitute a tool result with internal knowledge, and never report a step as completed unless you actually called the corresponding tool and received its result in this conversation. - Sub-tasks may be accomplishable through different tool combinations. FAULT HANDLING - When evaluating tool outputs, consider whether the returned data is coherent and usable for the next step -- fields should be present, values should be semantically reasonable, and types should match what the workflow requires. - If you determine that a required step cannot be accomplished by any available tool or approach, stop the workflow and clearly report which step failed and why, rather than proceeding with incomplete results. FINAL ANSWER FORMAT - Be CONCISE and DIRECT. No emojis, markdown formatting (**, [emoji], etc.), or bullet points. - If SUCCESSFUL: State only essential facts and requested results (e.g., confirmation codes). Omit internal technical details. - If HALTED: State clearly that the task was aborted and the reason (e.g., ‘Task aborted: the pricing tool returned corrupted data after retry and no other alternative approaches exist’). Do not apologize excessively. Figure 4: Verbatim ToolMaze system prompt. ToolMaze receipt envelope On a detected violation the interface replaces the tool message content with the JSON serialization of "tool_result": <raw result>, "outcome_contract": <receipt>; the raw result is always preserved. A real receipt from the frozen confirmation ledger (schema v1; task with the corrupted stock quote): ⬇ "status": "inconsistent", "violations": [ "code": "nonpositive_value", "detail": "nominally positive field price_usd is not positive", "code": "unseen_category", "detail": "field currency has unseen categorical value" ], "admissible_recovery_tools": [ "get_stock_yahoo_finance", "get_stock_alpha_vantage" ] The frozen confirmation uses the compact schema v1 above. The ToolMaze control studies use receipt schema v2 (ledger field receipt_schema_version value tool_property_path_expected_observed_first_v2), which carries the paper’s five-tuple witness. A real v2 receipt: ⬇ "status": "inconsistent", "violation": "tool": "get_weatherapi_alert_card_native", "property": "learned_type_mismatch", "path": "celsius_now", "expected": ["integer"], "observed": "string" , "admissible_recovery_tools": [ "get_weatherapi_alert_card_native", "get_visualcrossing_forecast_chip_native" ] Generic-caution text (specificity control). The generic arm shares the detector, envelope, timing, and recovery affordances, and replaces only the violation witness with one frozen string: ⬇ "status": "inconsistent", "violation": "property": "generic_caution", "message": "This result may be unreliable; verify it before relying on it." , "admissible_recovery_tools": ["…"] The decomposition controls (Section L) reuse this envelope. The warned arms of the matched-trigger placebo and the full always-warn arm are byte-identical to it; the stripped arm removes only the affordance key: ⬇ "status": "inconsistent", "violation": "property": "generic_caution", "message": "This result may be unreliable; verify it before relying on it." AppWorld and τ-bench advisory text. Both ports append a plain-text block to the observed result. The τ-bench template (AppWorld differs only in naming its source “its public response contract” and referring to “documented APIs”): τ-bench advisory template (appended to the raw result) [OUTCOME CONTRACT ADVISORY] The observed result from ‘<function>‘ is inconsistent with a known invariant of the retail database. Violation: <code> at ‘<path>‘; expected <expected>, observed <observed>. This advisory is nonbinding. Treat the result as unreliable and verify or recover using the available documented tools and task context before relying on it. All actions remain available. Fired invariant classes. Eight violation classes fire across the frozen ToolMaze ledgers, each shown with a real fired instance: • nonpositive_value: nominally positive field price_usd is not positive. • unseen_category: field currency has unseen categorical value. • learned_type_mismatch: field rate has unexpected type. • learned_echo_mismatch: field flight_number differs from call argument. • missing_learned_field: missing nominal field display_currency. • unexpected_field: field absent from nominal traces: error. • explicit_error: error-bearing field(s): error. • affine_relation_broken: fields cart_subtotal and item_count violate a nominal relation. Appendix O Organic Tool Traffic Every paired-completion study in the main paper injects faults, because that comparison requires a known fault. This section drops the agent and the counterfactual and asks a narrower question that organic data can answer: how often does the unmodified detector fire on real recorded tool responses? Corpus and protocol. We use the response cache released with StableToolBench (Guo et al., 2024; arXiv:2403.07714), which records real RapidAPI responses collected for ToolBench (Qin et al., 2024); both are Apache-2.0. We keep responses the cache marks successful whose payload is a JSON object, and we keep tools with at least 45 such responses. That leaves 427 tools and 53,078 responses. Per tool we mine contracts from 45 responses and hold out 20, repeating over five seeds. The detector, the miner, and the invariant classes are unchanged from the main paper’s method section. Nothing is injected and no fault is authored. Violation class Firings Share unexpected_field 364 61.7% missing_learned_field 78 13.2% explicit_error 69 11.7% learned_type_mismatch 53 9.0% unseen_category 12 2.0% learned_echo_mismatch 6 1.0% nonpositive_value 6 1.0% affine_relation_broken 2 0.3% Table 20: The detector fires on 0.80% of organic responses, and firings concentrate in 54 of 427 tools. Violation classes summed over five seeds on held-out real RapidAPI responses; 6,860 responses scored per seed, 41 to 65 flagged. Field-shape classes dominate; the value and relation classes that carry the main paper’s effects fire rarely on this corpus. Result. The detector flags 0.80% of held-out responses (41 to 65 of 6,860 per seed, range 0.60% to 0.95%). For comparison, the clean-P0 control in the main paper puts the false-positive rate at 1.39% on synthetic clean workflows, so the synthetic estimate is not optimistic relative to this corpus. Firings are concentrated rather than diffuse: 54 of 427 tools account for all of them. What the firings look like. We do not adjudicate every flag, and field-shape deviations on a public API are often benign. Two recurring patterns are worth showing. The first is an error delivered inside a response the transport layer reports as successful, which is the failure mode the paper is about: airportflights → "Error": "invalid ident, please provide a valid ident. You can use the airport search to determine the right one." The second is an expected payload that is simply absent, as when coins_get_markets returns where the mined contract expects a data field. Scope. This is a detection measurement, not an effectiveness one. There is no agent, no receipt, and no counterfactual, so nothing here speaks to recovery or completion in deployment. The responses are cached and filtered by StableToolBench rather than collected live. What the section establishes is narrower. Contracts mine cleanly from organic traffic. The detector’s firing rate there is consistent with the synthetic clean-P0 estimate. Silent failures of the kind the paper targets are present in real tool traffic, without anyone injecting them. Appendix P Construction Cost Detail ToolMaze: 400 unique nominal workflows produce 1,800 workflow-fold inputs across five cross-fitted registries. The build materializes 15,551 candidate invariants and accepts 15,041 (96.7%). Invariant classes include per-field type constraints, echo fields (a[f]=y[f]a[f]=y[f]), positivity (y[f]>0y[f]>0), categorical domains, affine relations, and temporal orderings. AppWorld: 11 released API-documentation files yield 473 candidate read contracts, of which 149 (31.5%) are accepted after recursive schema validation. τ-bench: the public orders.json database yields categorical (status, transaction_type), range (item price >0>0), and affine (net payment conservation) contracts. All builds are deterministic with zero model or API calls. Historical human authoring and inspection time was not contemporaneously recorded and is reported as unknown. Appendix Q Reproducibility Details Models and access. DeepSeek V4 Flash and V4 Pro via the direct DeepSeek API. Qwen 3.7 Plus and 3.7 Max via OpenRouter. The separate MiniMax M3 replication uses OpenRouter pinned to the MiniMax-hosted FP8 route with fallbacks disabled. All ToolMaze runs use temperature zero and extended reasoning disabled. Token usage and cost. The ToolMaze confirmation uses 24.81 million input tokens and 0.906 million output tokens; its all-cache-miss list-price upper bound is $15.46. The AppWorld development study uses 0.539/0.741 million cache-miss input tokens, 7.56/7.40 million cache-hit input tokens, and 0.332/0.328 million output tokens for Flash/Pro, at $1.204 total list price. Its held-out test_normal study uses 0.724 million cache-miss input tokens, 9.754 million cache-hit input tokens, and 0.383 million output tokens, at $0.345 total list price. The method comparison costs $14.926. The always-verify and clean-P0 controls cost $2.730 and $2.372, respectively. The MiniMax M3 replication costs $2.18. The frozen τ-bench factorial costs $0.43. The matched-trigger placebo study costs $5.73 and its stripped-envelope companion $4.36. Artifact mapping. The anonymous artifact’s MANIFEST.md and ARTIFACT_INDEX.md map each paper result to its sanitized manifest, aggregate report, and analysis code. Row-level provider responses and protected benchmark observations are intentionally excluded. Selected study status and analysis units. Study Status Unit and aggregate ToolMaze primary frozen 80 shared tasks; four-model aggregate MiniMax replication frozen separately 80 paired tasks; per-run analysis Reasoning ablation separate hashes 80 shared tasks; paper aggregate Specificity / clean separate hashes 57 shared workflows; per-run joint analyses Matched placebo frozen 57 fresh workflows; two-tier joint Stripped always-warn frozen same 57 workflows; three contemporaneous arms AppWorld dev development 19 families; joint task analysis AppWorld held-out frozen, Flash run 32 families; Flash analysis τ-bench factorial frozen 50 tasks; two fault contrasts per task Table 21: Selected study status and analysis units. The artifact’s STUDY_INVENTORY.csv lists all 22 studies and 4,683 episodes. Separately hashed studies are never pooled. The held-out AppWorld manifest planned two tiers, but only Flash was executed. MiniMax is never pooled into the primary ToolMaze aggregate; the τ-bench secondary aggregate clusters both contrasts by task. Aggregate regeneration. paper_aggregate_analysis.py validates all four frozen ToolMaze ledgers in the research workspace, reconstructs the shared-task effect and bootstrap interval, and independently reproduces the 484-event recovery-action counts. The review artifact packages the resulting aggregate, analysis code, and independent headline-value and cross-study validators, but not row-level provider responses. The aggregate explicitly excludes the retired substitute-availability post-hoc because its classification requires the unavailable external task catalog. For AppWorld, protected API-observation ledgers remain ignored; the packaged Flash aggregate and report contain only family-level counts, injection rates, usage, and hashes. The τ-bench analysis validates the frozen 50×550× 5 factorial grid and records task-clustered inference separately from the prespecified fault-stratified tests. Finally, submission.tex is a flattened, bibliography-inlined source generated from the modular manuscript; the build checks that it has the same page count as main.pdf.