Paper deep dive
Finding Blind Spots in AppWorld and WorkArena Task Verifiers
Richard Abrich
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/8/2026, 6:58:03 AM
Summary
This paper audits the task verifiers for the AppWorld and WorkArena benchmarks using source-informed mutation tests. It identifies specific 'blind spots' where verifiers incorrectly accept failed tasks (false accepts). In AppWorld, duplicating non-idempotent writes creates extra records that pass verification because checks focus on field values rather than record counts. In WorkArena, out-of-scope field writes are accepted because the verification logic relies on page-local flags that are lost upon form submission. The study also performs an intent-swap census, finding that cross-task trajectories are correctly rejected, highlighting the specific nature of the identified false accept mechanisms.
Entities (8)
Relation Signals (6)
Richard Abrich â authored â Finding Blind Spots in AppWorld and WorkArena Task Verifiers
confidence 100% ¡ Richard Abrich Affiliation: OpenAdapt.AI (MLDSAI Inc.) ... title: Finding Blind Spots in AppWorld and WorkArena Task Verifiers
AppWorld â hasblindspot â Record Count Blindness
confidence 95% ¡ AppWorld has a concrete blind spot: duplicating a non-idempotent write preserves every value the evaluator checks while creating an extra record.
WorkArena â hasblindspot â Session-Scoped Evidence Loss
confidence 95% ¡ WorkArena stores the extra-field flag on the page that exists before submit. Submit loads a new page, so a checker that looks only after submit cannot see it.
Mutation Testing â usedin â AppWorld
confidence 95% ¡ We audit the shipped AppWorld and WorkArena verifiers with source-informed, mutation-inspired tests.
WorkArena â runson â ServiceNow
confidence 90% ¡ WorkArenaâs live validators on a hosted ServiceNow instance
AppWorld â uses â State-based evaluators
confidence 90% ¡ We then tested two shipped checker families: AppWorldâs state-based evaluators
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Execution-based task verifiers decide whether an agent succeeded. We audit shipped AppWorld and WorkArena verifiers with source-informed mutation tests. The main audit never modifies a shipped checker. In AppWorld, duplicating a non-idempotent write creates an extra record while preserving every checked field value. The verifier accepts all three task variants from two of five eligible generators: 6/15 constructed effects. A cardinality patch applied to checker copies after the census makes all six cells fail while preserving valid controls. In WorkArena, we prospectively rerun 23 extra-field candidates selected for earlier checker-PASS outcomes. Independent Table API readback confirms nondefault persisted values in 21, while all 23 receive PASS. Two requested strings are aliases of stored defaults. The 21 confirmed wrong effects span three form templates. These selected cases confirm wrong effects under the audit's protocol; they do not estimate a population rate. No other construction produces an independently confirmed false accept. Other checker-PASS cases are effect-correct degeneracies. We report zero-PASS families separately because retained evidence differs. In fixed intent-swap grids, the checkers return no PASS on 2,689 off-diagonal executions. This is a rejection census: 57 WorkArena cells use session-scoped evidence; the other 2,632 lack classified rejection causes and independent target ground truth. Each increment is specified before its own cells are scored. A supplement accompanies the OpenReview submission with the construction grammar, evidence, content-bound stage lineage and count reproducer.
Tags
Links
- Source: https://arxiv.org/abs/2610.09142v1
- Canonical: https://arxiv.org/abs/2610.09142v1
Trouble viewing inline? Open PDF directly â
Full Text
52,004 characters extracted from source content.
Expand or collapse full text
Finding Blind Spots in AppWorld and WorkArena Task VerifiersThanks: Competing interests and funding: see the statement before the bibliography. This work is licensed under a Creative Commons Attribution 4.0 International licence (C BY 4.0). Richard Abrich Affiliation: OpenAdapt.AI (MLDSAI Inc.) Email: richard@openadapt.ai Abstract Execution-based task verifiers decide whether an agent succeeded. Leaderboards, model comparisons, and verifier-reward training inherit their errors. We audit the shipped AppWorld and WorkArena verifiers with source-informed, mutation-inspired tests. The main audit never modified a shipped checker. AppWorld has a concrete blind spot: duplicating a non-idempotent write preserves every value the evaluator checks while creating an extra record. The verifier accepted all three task variants from each of two affected scenario generators, or 2/5 eligible generators and 6/15 constructed effects. On copies specified after that Extra-NI census, a documented len(added_*)==N patch flips all six extra-row cells to Fail. Table 2 still reports the unmodified shipped tests. WorkArena has a second blind spot. We prospectively fixed and reran the 23 earlier extra-field candidates with independent Table API readback before checker evaluation. The Table API confirmed a nondefault persisted value in 21 of the 23 selected reruns, and the shipped checker returned Pass in all 23. Those 23 cells were chosen because they had already passed. Two requested strings were display-label aliases of their stored defaults. The 21 confirmed wrong effects span three form templates. This confirms a mechanism; it does not estimate a rate. No other construction produced an independently confirmed false accept. The remaining checker-Pass cells were effect-correct degeneracies. Retained evidence is not uniform across the zero-Pass families, so we report their verdict vectors separately and do not pool them. In fixed intent-swap grids, the checkers returned no Pass on 2,689 off-diagonal executions. This is a rejection census: 57 WorkArena cells used a session-scoped evidence path, and the other 2,632 remain unclassified by rejection cause and independent target ground truth. Each increment was specified before its own cells were scored. An anonymous supplement accompanies the submission with the construction grammar, primary evidence, content-bound stage lineage, and count reproducer. 1 Introduction Benchmarks for computer-use agents decide success automatically. After an agent runs, an execution-based checker inspects the applicationâs final state, such as database rows, files, or page content, and returns pass or fail (Trivedi et al., 2024; Drouin et al., 2024). Everything downstream inherits the checkerâs errors: leaderboard rankings, model comparisons, and the reward signals that verifier-reward training loops use to shape agent policies (Rad et al., 2026). The measurement literature on these checkers is asymmetric. Published audits often sample FAIL verdicts and estimate false rejects: Dong et al. (2026) audit 150 FAIL trajectories across five benchmarks, find 15.3% were scored incorrectly, and leave the complementary PASS audit open. Judge-focused work estimates false-positive rates for model-based web-agent judges (Rosset et al., 2026), and tool-calling evaluators have been audited in both directions (Bhat et al., 2026). We found no published constructed-fault audit of false accepts for the shipped AppWorld or WorkArena task verifiers. Software-testing research has long treated this as an oracle-adequacy problem and uses mutation analysis to expose tests that miss seeded faults (Jia and Harman, 2011; Barr et al., 2015). We adapt that idea to task effects rather than checker source code. We study both questions in two steps. 1. A two-arm, source-informed audit (§3). We specified and commit-froze each experimental increment before scoring its cells. We then tested two shipped checker families: AppWorldâs state-based evaluators (Trivedi et al., 2024) and WorkArenaâs live validators on a hosted ServiceNow instance (Drouin et al., 2024). The intent-swap arm contains 2,762 scheduled cells: 70 diagonal controls and 2,692 off-diagonal attempts, of which 3 errored and 2,689 returned valid verdicts. It feeds each target checker reference-solution executions for other tasks at graded semantic distances. The arm measures rejection behavior because the retained data do not independently label every target intent as unsatisfied. The fault arm applies mechanical edits to correct reference solutions: omissions, partial effects, wrong values, extra writes, unsubmitted forms, and out-of-scope writes. An edit enters the false-accept analysis only when retained evidence confirms the wrong effect. The intent-swap arm produced no Pass in 2,689 valid off-diagonal cells spanning 239 ordered generator pairs. Fifty-seven WorkArena cells used a session-scoped evidence path; the rejection cause and target ground truth remain unclassified for the other 2,632. We report this arm as a fixed rejection census. The 2,689 cells are not independently proven false-accept opportunities. The fault arm confirms one mechanism: AppWorld accepts duplicated non-idempotent writes because its evaluators check field values but not record counts. The failure occurs in 2/5 eligible scenario generators and all three daily variants of each, or 6/15 constructed effects. We report the remaining registered fault families separately rather than pool a mixture chosen after outcomes were visible. The original WorkArena extra-field increment is also separate: of 76 registered cells, 40 were refused, 13 failed before submit, and 23 reached submit and received Pass. A prospective rerun fixed those 23 cells in advance and read each exact record through the Table API before validation. Twenty-one retained a nondefault out-of-scope value and all 23 again received Pass. Two requested strings were display-label aliases of their stored defaults. The 21 confirmed wrong effects cover three form templates, from 23 cells that had already passed. 2. Evidence requirements for later inference (§5). The benchmark cells and generator pairs form fixed, dependent censuses. Their observed counts are descriptive. A population claim needs an independently sampled holdout, a fixed construction distribution, and an oracle that separates completed wrong effects from refused, incomplete, or uncertain delivery. We state those requirements and preserve the current sealed AppWorld holdout. The paper contributes a constructed-fault audit, confirmed failure mechanisms in both checker families, and a protocol that keeps fixed-census counts distinct from population claims. On copies specified after the Extra-NI census, a documented cardinality patch (len(added_*)==N) sent all six extra-row cells to Fail (§3.6); the main auditâs Table 2 is unchanged. It also identifies a WorkArena session-scoping limitation without classifying it as a false reject outside the benchmarkâs supported execution contract. 2 Threat Model and Fault-Construction Grammar Setting. A checker c is a fixed program attached to a benchmark task t. It receives evidence about a finished trajectory, including terminal application state and, for some families, emitted API calls. It returns Pass, Fail, or, in some configurations, refuses to score. This is a source-informed audit: we inspect checker and driver code to design tests and explain mechanisms, but never modify the checker under test. We make no model calls. AppWorld programs use the benchmark application APIs. WorkArena interactions use the shipped browser task drivers and Playwright. Adversary. The adversary knows the task and controls the executed trajectory, but cannot modify the checker, its evidence channel, or the environment after task start. This models the realistic deployment risk for gated automation and for training pipelines: an agent (or a policy being trained on checker rewards (Rad et al., 2026; Zhang, 2026)) produces plausible-but-wrong work, and the question is whether the checkerâs acceptance boundary separates it from correct work. Construction and ground truth. Both arms start from reference solutions delivered fresh from the task start state. AppWorld uses recorded API programs. WorkArena invokes the shipped live task drivers. The arms differ in what their retained evidence proves: ⢠Intent-swap arm. The reference trajectory of task j is delivered fresh and graded by the checker of task iâ jiâ j. The task definitions differ, but the retained artifacts do not independently prove for every cell that i was initially unsatisfied, remained unsatisfied, and was not satisfied as a collateral effect of j. A Pass is therefore an off-diagonal checker acceptance, not automatically a confirmed false accept. Cells are organized by severity rung: Diag (own-task control), Rung-0 (twin intent one parameter away), Rung-1 (sibling task, same scenario), Rung-2 (cross-scenario, shared effect models), and Rung-3 (cross-scenario, disjoint effect models). ⢠Fault arm. A grammar of mechanical edits is applied to the reference trajectory before delivery. Each edit is designed to make the effect wrong in a named way. A Pass remains a candidate false accept until retained effect evidence confirms that the wrong effect completed. Fault grammar. For the API-native AppWorld family, edits act on calls in the reference program: Claim-Only emits only the final read or claim call; Omit drops the last write; Partial keeps only the first repeated write; Wrong-Value moves one written value; Extra duplicates the last write; and Extra-NI restricts that duplication to endpoints that an idempotence census marks non-idempotent. For WorkArena, perturbations act on the driverâs interaction with the evaluator page: Omit-Field, Omit-Divergent, No-Submit, Extra-Field, and Claim-Only. Appendix A defines them. We commit-froze census-driven refinements before scoring the cells they enabled. Confirmed false accepts. A fault-cell Pass counts as a confirmed false accept only if retained environment evidence proves that the delivered effect differs from the correct effect. The verdict alone is insufficient. This rule excludes 42 degenerate edits: 33 AppWorld Extra passes were idempotent, and 9 WorkArena Omit-Field passes left the formâs default effect correct. It also counts the 15 Extra-NI cells only once because that increment re-ran transformations already present in Extra. The original WorkArena Extra-Field increment stays outside Table 2 because it lacks post-submit field readback; the selected prospective rerun is reported there as a separate row. 3 The Audit 3.1 Protocol discipline Each increment followed a specification-before-scoring discipline. Its protocol, enumeration code, and required census were committed before the first governed cell was scored. This is a prospective commit freeze, not a public preregistration. Interrupted runs resumed cell by cell from recorded ledgers, so no graded cell was scored twice. Two diagonal shakedown cells preceded the earliest WorkArena protocol; the protocol discloses them and the scored increment re-ran both. All raw ledgers, censuses, and derived files are retained. The study contains no model calls, and the main audit never modified a shipped checker; §3.6 regraded copies. 3.2 Suites We audit two shipped families, chosen to cover two distinct execution-checker designs: ⢠AppWorld (Trivedi et al., 2024; Stony Brook Natural Language Processing Group, 2026): state-based unit tests over the application database. We use all 48 state-changing DEV tasks with their shipped evaluators. The sealed HOLDOUT split stays sealed (§5). The fault arm scores up to all 48 tasks per family. The swap grid crosses the 31 tasks with a gate-run reference trajectory against 74 evaluator rows (Appendix C). ⢠WorkArena (Drouin et al., 2024; ServiceNow Research, 2026): live validators executed against a hosted ServiceNow pool instance. We use all 13 pool-permitted state-changing L1 templates (4 form and 9 catalog) at configuration indices 0â2. 3.3 Arm 1 results: an off-diagonal rejection census Table 1 summarizes 2,762 swap cells across both suites. Every diagonal control passed (31/31 and 39/39). Every valid off-diagonal cell was rejected, including Rung-0 twins that differ by one parameter. AppWorld had no evaluator errors. WorkArena had three disclosed error cells: moved trajectories whose reference field has a dependent autocomplete. We exclude those three cells from the denominator and retain them in the ledger. Earlier 12-cell and 6-task pilots are consistent with and subsumed by the full grids. Suite Rung Valid cells Checker Pass AppWorld (2,294 cells) Diag (control) 31/31 pass â Rung-0: twin intent, 1 parameter 43 0 Rung-1: sibling, same scenario 144 0 Rung-2: cross-scenario, shared models 369 0 Rung-3: cross-scenario, disjoint 1,707 0 WorkArena (468 cells; 3 errors) Diag (control) 39/39 pass â Rung-0: twin intent, 1 parameter 75 0 Rung-1: same template, other config 78 0 Rung-2: cross-template, shared tables 216 0 Rung-3: cross-template, disjoint tables 57 0 Generated from the committed grid artifacts analysis/intent_swap/fullgrid_results.json and analysis/intent_swap/workarena_grid_results.json. Table 1: Intent-swap rejection census: cross-task reference trajectories, by severity rung. Off-diagonal Pass counts are checker responses, not confirmed false accepts. The checkers returned no Pass in 2,689 valid off-diagonal cells or in the 239 ordered generator pairs those cells span. This is a census of the constructed grid. Fifty-seven cells used the WorkArena session/provenance gate. The other 2,632 valid rejections remain unclassified by rejection cause and by independent target ground truth. A task definition that differs from the target does not prove that the target was unsatisfied before and after execution. Cells also reuse generators, and generator pairs share endpoints. The result is a rejection baseline for future comparisons. 3.4 A session-scoped evidence limitation The WorkArena form validator stores the created record ID under a browser-local key derived from the Python identity of the validating task object. A submit-time hook writes the key, and validation reads it. The catalog validator binds evidence through a request ID in the current page URL. In disclosed probes, the target record existed in the instance database but a validator outside the originating evidence channel still returned Fail. These form-key and catalog-URL routes collectively account for all 57 WorkArena Rung-3 rejections. This is a session-scoping property of WorkArenaâs supported same-session contract, so those 57 cells cannot be used as semantic rejection evidence. 3.5 Arm 2 results: fault constructions and prospective readback (A) AppWorld Extra-NI Intended: one new row Duplicate write: two rows, same field values Tests check field values, not how many rows Verdict: Pass (B) WorkArena Extra-Field Extra field written; flag stored in this page Submit loads a new page; the flag is gone Checker after submit cannot see the flag Record still has extra value: Pass Figure 1: Two confirmed miss paths. (A) Duplicating a write adds a row; the tests check field values, not how many rows. (B) WorkArena stores an extra-field flag in the page; submit replaces that page, so a checker that looks only after submit cannot see the flag. Panel B uses 21 of 23 reruns chosen from earlier PASS cells. Table 2 reports constructions for which retained evidence proves a no-write path or a state difference. The full flow table in Appendix G retains every other registered construction and checker verdict. We report no pooled fault-family rate. The selected WorkArena row comes from the prospective readback follow-up, not from a random or complete family census. Checker:family Wrong effect Evidence N FA appworld:claim-only claim w/o write runner no-write 48 0 appworld:extra duplicate write (changed state) replay write counters 15 6 workarena:claim-only claim w/o request runner no-write 27 0 workarena:no-submit leave form unsubmitted runner no-write 39 0 workarena:extra-field* out-of-scope write Table API readback 21 21 Generated from the committed bounds and prospective readback artifacts under analysis/intent_swap/. Other registered fault families remain in the appendix flow table as construction-and-verdict evidence. Table 2: Independently supported fault constructions, reported by family. The Evidence column separates retained effect readback from construction-guaranteed no-write. The selected WorkArena row (asterisk) is the prospective follow-up of 23 earlier checker-Pass candidates: Nselected=23N_selected=23, confirmed wrong effect =21=21, checker Pass =23=23, not a sampled family rate. Counts must not be pooled across rows. Record-count blindness in AppWorld. Duplicating a non-idempotent write creates an extra artifact, such as another payment request and notification or a doubled player record. It leaves every field value the evaluator checks intact. The verifier accepted all three daily variants of two affected scenario generators. This is 2/5 eligible generators and 6/15 constructed effects. The later Extra-NI increment re-scores the same 15 transformations already present in Extra, so Table 2 reports them once. AppWorld Omit, Partial, Wrong-Value, and Claim-Only, and WorkArena Claim-Only, No-Submit, and Omit-Divergent, returned Fail in every registered scored cell. WorkArena Omit-Field returned nine Pass verdicts, all on default-preserving degeneracies. The supplement reports these verdict vectors separately. Shipped unit test, generator 22c237 (Venmo payment request). It checks field values and never |added|=N =N:test.case( dict_of(added_payment_requests, "receiver_id", "amount"), "==", private_data.pending_user_id_to_share, ) Generator 396c5a2 checks queue_song_ids by set-union and does not assert uniqueness or |Î| . Duplicating a write adds a row; the tests check field values, not how many rows. Out-of-scope writes in WorkArena. The commit-frozen census selected requested extra-field strings that differed from the recorded default strings. The original flow is: 76 registered cells, 40 refusals because no eligible extra input existed, 36 attempted executions, 13 pre-submit failures, 23 submitted records, 23 checker Pass verdicts, and zero retained field readbacks. We then fixed those 23 source cells in a prospective manifest and reran them without adding or replacing a candidate. Before checker evaluation or cleanup, the runner queried the exact submitted record through the Table API. Twenty-one cells retained a nondefault out-of-scope value, and the shipped checker returned Pass in all 23. In two cells, the requested display label mapped to the same stored value as the default; we classify those rows as degenerate. The 21 confirmed wrong effects cover three form templates. No cell failed before submit, no attempt was retried, and cleanup was clean after every cell. WorkArena tries to catch an extra field with a flag stored in the page. Submitting the form loads a new page, so that flag is gone. A checker that looks only after submit cannot see it. Our cells fill the extra field and submit in one scripted pass, then ask the shipped checker on the page that exists after submit. The record still holds the extra value; the checker still returns Pass. This audit does not test the other protocol, in which the benchmark checks after every step and the extra write and the submit fall in different steps (Appendix E). The six AppWorld duplicate-write cells and 21 of the 23 selected WorkArena extra-field cells are confirmed silent incorrect successes. Their denominators arise from different selection procedures, so we do not combine them. 3.6 Repairs We copied the six extra-row AppWorld tests and added len(added_*)==N. Regrading the same Extra-NI programs sent all six from Pass to Fail. Table 2 is unchanged. The §3.6 copies were specified after the Extra-NI census and are not part of Table 2. The main audit still used the unmodified shipped tests. We did not re-run WorkArena. AppWorldâs generator guide tells authors to write test.case(len(added_*), "==", N) on the changed_records diff, and not to compare whole-table sizes (Stony Brook Natural Language Processing Group, 2025). The example in that guide is a Venmo send. Generator 22c237 checks field values with dict_of and asserts |updated|=0|updated|=0 and |removed|=0|removed|=0, but never |added|=N|added|=N. Generator 396c5a2 compares queue_song_ids by set-union, so a duplicated enqueue is invisible. The copies add that count on the queue. The six controls stayed Pass. AppWorld did-catch. A one-line len(added_*)==N patch, the assertion AppWorld already documents, flipped all six extra-row Extra-NI cells to Fail. The main audit used the unmodified shipped tests. WorkArena stores the extra-field flag on the page that exists before submit. Submit loads a new page, so a checker that looks only after submit cannot see it. Persist that flag in localStorage or on the created record. Or read the row back through the Table API after submit, which is what the audit already did for the 23 selected cells. We did not test BrowserGym per-step validation, and we did not re-run a patched WorkArena checker. WorkArena would-catch. Persist the extra-field flag past submit, or read the submitted row through the Table API. We did not re-run WorkArena with that patch. 4 Rank orders under an assumed Extra-NI budget With no new agent or model calls, a retained-evidence re-read of 8,190 official AppWorld 0.1.0 Task Goal Completion verdicts (14 configurations per test split) asks which shipped pairwise orders would remain identified if Extra-NIâs 2/5 eligible DEV generators were treated as an error budget (Îąmax=0.4 _ =0.4) and 0/31 control variants as false rejects (βmax=0 _ =0). Official TGC is insulated because both generators are DEV-only. Any training or selection loop that rewards DEV Task Goal Completion is not. Duplicate-write successes would look like valid reward. An illustrative generator-level 95% ClopperâPearson envelope is (0.8107,0.2384)(0.8107,0.2384). We exclude Scenario Goal Completion, pooled rates, duplicate rechecks, and all WorkArena evidence. For each shipped score q, the sharp interval is [maxâĄ0,(qâÎąmax)/(1âÎąmax),minâĄ1,q/(1âβmax)][ \0,(q- _ )/(1- _ )\, \1,q/(1- _ )\]. When those intervals overlap, the assumed budget does not identify the order (Table 3). DEV-to-test transfer is assumed, and a confidence reading of the envelope needs an unestablished exchangeability model. Split Budget Îąmax _ βmax _ Pairs surviving Top-1 unique? test_challenge Observed magnitude .4000 .0000 0/91 no Illustrative 95% .8107 .2384 0/91 no test_normal Observed magnitude .4000 .0000 6/91 no Illustrative 95% .8107 .2384 0/91 no From 8,190 official archived AppWorld 0.1.0 verdicts. No task was regraded. Budgets obey the valid observation model. Overlap does not show a rank error. Table 3: AppWorld rank-consequence sensitivity analysis. A pair survives when its intervals do not overlap. Observed magnitude uses 2/5 F-EXTRA generators and 0/31 controls. The illustrative 95% envelope uses 2/5 and 0/11 generators, with exchangeability. 5 What the Audit Can Support Let X=1X=1 when a checker returns Pass on a completed wrong effect. A meaningful rate must name how effects enter the denominator. Our family rows describe fixed constructions conditional on the fault grammar, checker version, and benchmark tasks. They do not estimate the probability that a deployed run is wrongly accepted. Such a probability also depends on the prevalence and composition of wrong effects in deployment. The benchmark cells are fixed census items, not random draws. AppWorldâs 48 state-changing DEV tasks are 16 scenario generators at three daily variants each; WorkArenaâs 13 templates are exercised at three configuration indices. Dependence is visible in the results: 107 of the 109 generator-by-family groups with more than one non-refused cell return the same verdict throughout. The two split groups are WorkArena Extra-Field groups separated by pre-submit failures, not checker disagreement. A binomial interval over cells would therefore treat repeated variants as independent observations when they are not. The intent-swap arm is descriptive for the same reason. It produced no checker Pass in 2,689 valid off-diagonal cells spanning 239 ordered generator pairs. Pairs reuse evaluator and trajectory generators, so their outcomes are dependent. More importantly, the retained artifacts do not independently label every target intent as unsatisfied. We therefore report neither a false-accept rate nor a binomial confidence limit for this arm. In the paired AppWorld duplicate-write census, the verifier accepted duplicate writes from 2/5 eligible scenario generators and 6/15 task variants. The appendix retains every other original family as a construction-and-verdict flow. The prospective WorkArena rerun confirms persisted nondefault out-of-scope writes in 21 of 23 fixed candidate cells, spanning three form templates. The other two cells are default-alias degeneracies that also received Pass. At the generator unit, two AppWorld scenario generators and three selected WorkArena form templates have confirmed false accepts. We do not combine these facts. The AppWorld denominator is the eligible non-idempotent-write census. The WorkArena rerun population was selected from earlier candidate Pass outcomes. A holdout for inference. A later study can support a population claim if it fixes the target population, sampling frame, draw procedure, fault mixture, unit of analysis, and effect oracle before outcomes are visible. It must draw an independent holdout, keep nested refinements from receiving duplicate weight, read each claimed wrong effect back after delivery, and distinguish a completed wrong effect from a refused or incomplete delivery. If it selects a threshold or a family mixture, the selection and evaluation samples must remain separate. The current sealed AppWorld HOLDOUT split is not opened for this paper. Evidence record. A later claim should bind its estimand, sampling design, observed counts, interval and assumptions, corpus hash, checker version, and evaluation date. It should also retain refused, incomplete, and uncertain deliveries as separate outcomes instead of folding them into success or failure. This paper supplies the construction grammar and those reporting requirements. It does not supply a distribution-free certificate. Drift and re-evaluation. A population claim would depend on the declared sampling model and on exchangeability between the evaluation draw and deployment target. A task change, application drift, or checker patch can break that premise. The evidence record should therefore bind the exact checker, corpus, and environment and state when re-evaluation is required. 6 Related Work Test oracles and mutation analysis. A test oracle decides whether an execution is correct; oracle incompleteness is a core testing problem (Barr et al., 2015). Mutation testing evaluates a test suite by introducing small program changes and checking whether the tests detect them (Jia and Harman, 2011). Our unit of mutation is different. The main audit keeps the shipped checker unmodified and perturbs task trajectories or their effects. A checker Pass on a confirmed wrong effect exposes an inadequate task oracle. Edits that leave the effect correct play a role analogous to equivalent mutants and are excluded from the wrong-effect count. Evaluation-integrity audits. Prior work audits benchmarks and judges through best-practice checklists (Zhu et al., 2025), web-agent judge evaluations (LĂš et al., 2025), mobile-agent judge benchmarks (Wang et al., 2026b), compounding-invalidity models (Caban, 2026), FAIL-verdict audits of computer-use scoring (Dong et al., 2026), judge false-positive estimates (Rosset et al., 2026), and audits of tool-calling evaluators (Bhat et al., 2026). We found no constructed-fault false-accept audit of the shipped AppWorld or WorkArena verifiers. Dong et al. (2026) leave the PASS-verdict audit open, while OSWorld 2.0 uses construction-time quality assurance to prevent rather than measure false accepts (Yuan et al., 2026). Work on silent agent failures (Advani, 2026; Dubey, 2026) further motivates direct effect checks. Risk control for agents. Learn-then-test and conformal risk control (Angelopoulos et al., 2025; Angelopoulos et al., 2024) have been applied to execute/abstain boundaries for mobile GUI agents (Feng et al., 2026), grounding-click gates (Wang et al., 2026a), selective acting for RLVR-trained LLMs (Khosravi and Huo, 2026), tool-call intervention rules (Opoku and Banahene, 2026; Rahman et al., 2026), and sequential trajectory monitoring (Sadhuka et al., 2025). Those procedures need a specified sampling model and separate data for selection and evaluation. Our checkers expose a fixed binary verdict, and our benchmark tasks form a selected census. We therefore perform no distribution-free risk-control procedure. The protocol in §5 states how a later independent holdout could support one while keeping uncertain delivery separate from success and failure. Training on verifier rewards. RLVÎľ links reward noise to policy collapse thresholds (Rad et al., 2026), with follow-ups on imperfect verifiers (Cai et al., 2026; Plesner et al., 2026), uncertainty-aware reward modeling (Pan et al., 2026), overoptimization scaling laws (Gao et al., 2023), and preregistered contrasts of natural verifier false positives in RLVR (Zhang, 2026). Interactive environment-state verifiers are entering training loops (Shi et al., 2026; Yuan et al., 2026). These analyses require a justified verifier-error rate. The counts in this paper expose mechanisms and help design that measurement, but the selected benchmark census does not supply a population rate. 7 Future Work Four directions follow. Independent WorkArena holdout: sample extra-field cells before checker outcomes are known and retain the same pre-checker database readback. Holdout-supported reward studies: sample a target verifier population independently, estimate its false-accept rate under a fixed grammar, and measure how that rate maps into a policyâs silent-wrong-action rate. Rank consequence: extend the audit to OSWorld-Verified-class checkers and archived agent trajectories, then measure the before-and-after rank table under a checker change. Adaptive adversaries: replace recorded faults with model-generated candidates and measure selection under optimization pressure. A tunable checker score could also support learn-then-test or conformal risk control, with separate selection and evaluation samples (Angelopoulos et al., 2025; Angelopoulos et al., 2024). 8 Limitations No population sample. The two suites and their task generators are selected benchmark censuses. The observed counts do not estimate a deployment population without a sampling frame and an exchangeability argument. We report no confidence limit for these fixed census counts. Two suites. The counts describe two shipped checker designs: API-native state assertions and live web-form validation. Other execution-checker families, such as OSWorld-Verified-class desktop environments, remain untested. Synthetic-fixture scope. Faults are constructed edits of correct reference solutions delivered through the environmentâs own interface. A row enters Table 2 only with retained effect evidence or a construction-guaranteed no-write path. No live model-generated adversary, checker-aware search, or natural-failure corpus contributes to the results. The counts describe this fixed constructed census, not a sampled fault distribution or all possible wrong effects. Unit of analysis and selection. Cells within a task generator are close to perfectly correlated, and ordered intent-swap pairs reuse endpoints. We report descriptive cell counts and generator facts separately. Two AppWorld generators and three WorkArena form templates have confirmed false accepts. We do not pool the other fault families because their construction mixture and retained evidence types were not sampled uniformly. WorkArena selection. The prospective readback manifest fixed the 23 cells that had reached submit and received Pass in the earlier run. The rerun independently confirms their persisted wrong effects, but this outcome-selected population cannot estimate a false-accept rate for registered, eligible, submitted, or naturally occurring extra-field attempts. Environment scope. One shared hosted instance constrained scheduling to sequential execution; wall-clock costs (531 s for the AppWorld grid, about 71 s per live cell) bound how much replication fits a weekend slice. Intent-swap ground truth. Different task definitions do not by themselves prove that every target intent remained unsatisfied. The swap arm measures off-diagonal checker responses, not confirmed false accepts. Its 57 WorkArena Rung-3 cells are further confounded by session-scoped evidence. Reproducibility statement Each experimental increment was specified and commit-frozen before its cells were scored. The artifact lineage retains protocols, censuses, enumeration code, cell ledgers, and derived counts. Appendix H gives package versions, source pins, execution paths, and reset boundaries. An anonymous, checksum-bound supplement accompanies the submission. The study uses no model calls, and the main audit never modified a shipped checker; §3.6 regraded copies. Competing interests and funding Competing interests: The author is the founder and CEO of MLDSAI Inc. (OpenAdapt.AI), which develops GUI automation software and offers commercial services based on the verification approach this paper describes. Funding: No external funding; experiments were self-funded (cloud compute partly under an AWS Activate credit; model API usage self-paid). References Advani (2026) L. Advani. From confident closing to silent failure: Characterizing false success in LLM agents, 2026. arXiv:2606.09863. Angelopoulos et al. (2024) A. N. Angelopoulos, S. Bates, A. Fisch, et al. Conformal risk control. In International Conference on Learning Representations (ICLR), 2024. arXiv:2208.02814. Angelopoulos et al. (2025) A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2):1641â1662, 2025. doi: 10.1214/24-AOAS1998. arXiv:2110.01052. Barr et al. (2015) E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507â525, 2015. doi: 10.1109/TSE.2014.2372785. Bhat et al. (2026) V. Bhat, J. Vaghasiya, M. A. Mohsin, et al. Benchmarking the benchmarks: A validity audit of tool-calling evaluation, 2026. arXiv:2607.02577. Caban (2026) W. Caban. Measurement without validity: The compounding reliability problem in agentic AI evaluation, 2026. arXiv:2608.00794. Cai et al. (2026) X.-Q. Cai, W. Wang, F. Liu, et al. Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers, 2026. arXiv:2510.00915. Dong et al. (2026) Z. Dong, Z. Ma, Z. Wang, et al. How benchmarks mis-score computer-use agents, 2026. arXiv:2607.28367. Drouin et al. (2024) A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, D. Vazquez, N. Chapados, and A. Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11642â11662, 2024. URL https://proceedings.mlr.press/v235/drouin24a.html. arXiv:2403.07718. Dubey (2026) S. Dubey. Real-time detection and repair of LLM agent failures, 2026. arXiv:2608.02464. Feng et al. (2026) Y. Feng, J. Du, Q. Wang, et al. CORA: Conformal risk-controlled agents for safeguarded mobile GUI automation, 2026. arXiv:2604.09155. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning (ICML), 2023. arXiv:2210.10760. Jia and Harman (2011) Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649â678, 2011. doi: 10.1109/TSE.2010.62. Khosravi and Huo (2026) H. Khosravi and X. Huo. Conformal selective acting: Anytime-valid risk control for RLVR-trained LLMs, 2026. arXiv:2605.20270. LĂš et al. (2025) X. H. LĂš, A. Kazemnejad, N. Meade, et al. AgentRewardBench: Evaluating automatic evaluations of web agent trajectories. In Conference on Language Modeling (COLM), 2025. arXiv:2504.08942. Opoku and Banahene (2026) J. Opoku and D. Banahene. ToolChain-CRC: Conformal risk control for agentic AI under retrieval and tool-use drift, 2026. arXiv:2606.18467. Pan et al. (2026) L. Pan, H. Yang, H. Li, et al. Uncertainty-aware reward modeling for stable RLHF, 2026. arXiv:2606.19818. Plesner et al. (2026) A. Plesner, F. GuzmĂĄn, and A. Athalye. An imperfect verifier is good enough: Learning with noisy rewards, 2026. arXiv:2604.07666. Rad et al. (2026) A. Rad, K. Filom, D. Keivan, P. M. Esfahani, and E. Kamalinejad. Rate or fate? RLVÎľ : Reinforcement learning with verifiable noisy rewards, 2026. arXiv:2601.04411. Rahman et al. (2026) M. A. Rahman, M. A. Rahman, N. H. Samin, et al. Beyond aggregate risk: Role-stratified conformal risk control for LLM tool calls, 2026. arXiv:2607.24343. Rosset et al. (2026) C. Rosset, P. Sharma, A. Zhao, et al. The art of building verifiers for computer use agents, 2026. arXiv:2604.06240. Sadhuka et al. (2025) S. Sadhuka, D. Prinster, C. Fannjiang, et al. E-valuator: Reliable agent verifiers with sequential hypothesis testing, 2025. arXiv:2512.03109. ServiceNow Research (2026) ServiceNow Research. BrowserGym WorkArena software, version 0.5.3. GitHub release, 2026. URL https://github.com/ServiceNow/WorkArena/releases/tag/v0.5.3. Source commit a772230a94cf1caf4166b8ead3983f3b3786455b. Shi et al. (2026) C. Shi, Y. Wu, Y. Liu, et al. Interactive reward agent: GUI task evaluation via environment-state verification, 2026. arXiv:2607.25904. Stony Brook Natural Language Processing Group (2025) Stony Brook Natural Language Processing Group. Developing new task generators. AppWorld repository documentation, 2025. URL https://github.com/StonyBrookNLP/appworld/blob/afc941710b91ab9260b95b7bdebb1179deba90cf/guides/developing_new_task_generators.md. Path guides/developing_new_task_generators.md. Absent from source commit 66ad809 of appworld==0.1.3.post1; first published in AppWorld commit afc94171 on 2025-11-10. Stony Brook Natural Language Processing Group (2026) Stony Brook Natural Language Processing Group. AppWorld software, version 0.1.3.post1. GitHub release, 2026. URL https://github.com/StonyBrookNLP/appworld/releases/tag/v0.1.3.post1. Source commit 66ad8099e12188ece0d3fe45e661dbc01880813b; data version 0.1.0. Trivedi et al. (2024) H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. doi: 10.18653/v1/2024.acl-long.850. arXiv:2407.18901. Wang et al. (2026a) Q. Wang, Y. Fan, and X. E. Wang. SafeGround: Know when to trust GUI grounding models via uncertainty calibration, 2026a. arXiv:2602.02419. Wang et al. (2026b) Z. Wang, L. Gu, Z. Chi, et al. Benchmarking LLM judges for mobile agent evaluation, 2026b. arXiv:2608.11434. Yuan et al. (2026) M. Yuan, Z. Zhou, X. Xiong, et al. OSWorld 2.0: Benchmarking computer use agents on long-horizon real-world tasks, 2026. arXiv:2606.29537. Zhang (2026) C. Zhang. When the reward suite is leaky: A preregistered causal contrast of natural verifier false positives in RLVR, 2026. arXiv:2607.11022. Zhu et al. (2025) Y. Zhu, T. Jin, Y. Pruksachatkun, et al. Establishing best practices for building rigorous agentic benchmarks. In NeurIPS Datasets and Benchmarks Track, 2025. arXiv:2507.02825. Appendix A Fault-Family Grammar: Full Definitions Table 4 defines each family operationally. Census-driven targeting (idempotence for Extra-NI; on-form defaults for Omit-Divergent; template-differing candidates for Extra-Field) was committed before the cells it enables were scored. Table 4: Fault-construction grammar. Each edit is applied before fresh delivery. A cell enters the false-accept analysis only after retained effect evidence confirms that the construction changed the effect. Family Construction claim-only Deliver only the claim/completion step; no write occurs. omit Drop the final write call from the reference program. partial Keep only the first of a repeated write sequence. wrong-value Move one written value to a different legal value. extra Duplicate the final write call. extra-ni Duplicate the final write call restricted to endpoints that a committed idempotence census marks non-idempotent (a duplicated POST creates an extra payment request and notification; a duplicated action doubles a player record). omit-field (GUI) Submit the form without one field. omit-divergent (GUI) Omit a field whose expected value differs from its on-form default (per committed default census), so the omission changes the effect. no-submit (GUI) Leave the form unsubmitted; do not invoke the task driver. extra-field (GUI) Write an additional field the task never mentions, chosen among candidates whose template value differs from the form default (e.g., a choice field on the incident form; an acquisition-method field on the hardware-asset form). claim-only (GUI) Assert task completion without creating the underlying request. Appendix B Prospective Commit Freezes and Interruption Handling Each increment proceeded in three stages: protocol and enumeration code, required censuses, then scored cells. A commit adding a stage precedes the artifacts it governs. These private commit freezes establish chronology but are not public preregistration. External process kills and one session loss interrupted several runs. Resumption skipped each recorded cell file, so no graded cell was scored twice. The ledgers record the interruptions. Two diagonal shakedown cells preceded the earliest WorkArena freeze; the protocol discloses them and the scored increment re-executed both. Refusals are recorded with reasons. Appendix C Full Grid Details for the Intent-Swap Arm The AppWorld grid crosses 74 evaluator rows (31 DEV task evaluators plus 43 admissible moved-intent evaluators from the committed gate manifest) against 31 freshly executed reference trajectories: 2,294 cells, fully sequential, zero evaluator errors or timeouts (531 s of scoring wall clock; a mid-run kill resumed from recorded cells without re-scoring). The WorkArena grid covers 13 pool-permitted state-changing templates at configuration indices 0â2: 468 cells over seven resume passes, with three disclosed error cells (dependent-autocomplete moved references) excluded from denominators. The final resume pass recorded 3,123 s; total elapsed wall time across all passes was about 11 hours. The earlier 6-task pilot contained 30 off-diagonal cells and took 88.4 s. A 12-cell WorkArena pilot took about 71 s per live cell. Both are consistent with and subsumed by the full grids. Appendix D Session-Scoped Evidence Channel: Code Anchors In browsergym-workarena 0.5.3, the form-task validator (i) stores the created recordâs system ID under a localStorage key containing the Python object identity of the validating task instance; (i) uses a submit-time hook to write that key; and (i) reads the key during validation, returning âThe form has not been submittedâ when it is absent. The catalog validator instead reads the request ID from the current page URL. Probe cells confirmed the mechanism: the target record existed in the database while a validator outside the originating evidence channel returned Fail. This is a portability limitation across sessions. We do not classify it as a false reject under the benchmarkâs supported same-session contract. Appendix E Out-of-Scope-Write Monitor: Code Anchors In browsergym-workarena 0.5.3, the form-task setup injects a script the authors introduce with the comment âEnsure that only the expected fields are changedâ. The script enumerates every input, select and textarea then on the page, and for each whose trailing field name is absent from the taskâs protected list it registers a change listener that sets a page-local flag WORKARENA_BAD_FIELD_CHANGED. The validator reads that flag first and, when set, returns reward 0 with âSome fields outside of the task scope have been changed.â Our Extra-Field cells call the shipped cheat() once (fill plus insert), then validate() once on the page after submit. The flag lives on the page and does not survive ServiceNow frame navigation, so that validate() call checks final state after submit. The record still holds the extra value; the shipped checker still returns Pass. This audit does not measure BrowserGym per-step validation when the extra-field write and the submit fall in separate steps. Listeners bound only at injection time may still matter for fields created after inject. That is not the lead cause for the confirmed cells: the extra fields already on the form (category, acquisition_method) were present at injection. Pre-checker Table API readback confirmed a nondefault out-of-scope value in 21 reruns; two requested display labels mapped to their stored defaults. Appendix F Prospective WorkArena Effect Confirmation The first WorkArena run did not retain the extra field after submit, so its 23 Pass cells remain labeled candidate_only_no_post_submit_extra_field_readback. A later protocol fixed exactly those source cells, including task configuration, candidate index, field, default, and requested value, before any rerun. Each rerun used the same shipped browser driver. After submit and before the checker or cleanup, the runner read the exact record ID through the Table API. Twenty-one values differed from their stored defaults. Two requested display labels mapped to the same stored value as the default and are degenerate. The frozen oracle initially counted all 23 because the requested display label matched; a retained-representation audit found two rows whose persisted raw value equaled the stored default, so we conservatively reclassified those rows after scoring. No cell was added or removed. All 23 checker calls returned Pass, and all 23 cleanup checks were clean. No cell was retried or replaced. The run took 1,963.7 s. The rerun used those 23 earlier PASS cells and covers three form templates. Appendix G Complete Fault-Arm Execution Flow Table 5 gives the denominator flow for every distinct fault family. Its categories make the construction and delivery losses visible before any checker outcome is interpreted. Checker:family Sched. Ref. Inc. Deg. Cand. Class. Pass FA appworld:claim-only 48 0 0 0 0 48 0 0 appworld:extra 48 0 0 33 0 15 39 6 appworld:omit 48 0 0 0 0 48 0 0 appworld:partial 48 9 0 0 0 39 0 0 appworld:wrong-value 48 15 0 0 0 33 0 0 workarena:claim-only 52 25 0 0 0 27 0 0 workarena:extra-field 76 40 13 0 23 0 23 0 workarena:no-submit 39 0 0 0 0 39 0 0 workarena:omit-divergent 39 23 0 0 0 16 0 0 workarena:omit-field 13 2 0 9 0 2 9 0 All counts come from analysis/intent_swap/fault_arm_bounds_v3.json. Extra-NI is not a distinct row: 48 cells were scheduled, 33 refused, and 15 completed with effect evidence; six returned Pass. Those 15 are exact re-scores of Extra events, so they add no distinct construction or false accept. The 23 WorkArena Extra-Field candidates lack post-submit field readback in this artifact. Table 5: Complete execution flow for each distinct fault family. Scheduled cells partition into refused or ineligible edits, incomplete deliveries, effect-correct degenerate edits, effect-unconfirmed candidates, and analysis-classified attempts. The Class. column reproduces the frozen reportâs classification; it is not uniformly independent state evidence. Raw checker Pass and confirmed false accepts are outcome columns and are not added to that partition. A zero in FA means that no false accept was independently confirmed, not that none exists. FA=0FA=0 on Extra-Field is the pre-readback increment; the confirmed 21 is Table 2. Appendix H Reproducibility Details Software pins. ⢠AppWorld. Package appworld==0.1.3.post1; data version 0.1.0; source commit 66ad8099e12188ece0d3fe45e661dbc01880813b; wheel SHA-256 db77f8003982502383a50fa2974983894bd1c54f64e2fd3f7e1540d5edd037eb. ⢠WorkArena. Package browsergym-workarena==0.5.3; source commit a772230a94cf1caf4166b8ead3983f3b3786455b; wheel SHA-256 eb0bdb58598f5e8aeb8b9f01ac3a68dc26d477e3067f01b56f640a1016e81a5; Playwright 1.44.0; Chromium 125.0.6422.26 (build 1117). Execution paths. AppWorld calls the shipped evaluator on a fresh execution database for each cell. Faults edit the reference API-call program before execution. The main audit never modified a shipped checker; §3.6 regraded copies. WorkArena executes one audit cell at a time on a Washington DC ServiceNow pool instance. Foreign-fallback swap cells keep the evaluator episode open while a second trajectory episode runs. The runner uses the shipped task driverâs cheat() path through BrowserGym and Playwright. For Extra-Field, the audit adds one source-informed field to the driverâs task_fields. The pinned upstream _fill_fields routine enters its value through Playwright label locators, then the shipped driver clicks the formâs insert button. The audit does not inject the value with JavaScript, write it through the database API, or modify the validator. Swap-grid diagonal controls pass 31/31 for AppWorld and 39/39 for WorkArena. The AppWorld fault arm also retains 48/48 passing reference controls; WorkArena reuses the 39 grid controls rather than repeat them in every fault increment. Isolation and reset boundary. AppWorld resets the execution database for each cell. WorkArena creates a per-episode user, invokes task teardown, and verifies actor-scoped residue. Other users can write to the shared instance, so this is not a global database reset. Cells run sequentially, and interruptions are recorded at run level. WorkArena run context. The grid ran on 23â24 August 2026 and the prospective readback ran on 28 August 2026 against a Washington DC ServiceNow instance (build glide-washingtondc-12-20-2023_01-22-2024_1800.zip). The viewport was 1280Ă720, slow_mo was 1,000 ms, the task timeout was 10 s, and the per-cell limit was 420 s. Every reset used seed 7001 with a fixed configuration index; the per-episode Faker user name was deliberately unseeded. The grid artifacts did not retain the host OS or Python patch version, which remains a replication gap. Appendix I Artifact Availability An anonymous supplement accompanies the submission. It contains the fixed protocols and manifests, source pins, study-stage chronology, public-safe primary evidence, per-family counts, the prospective WorkArena readback, the AppWorld rank-consequence derivative, the Extra-NI copied tests and twelve regrade ledgers, and a standard-library reproducer. A checksum manifest binds every packaged file. The OpenReview ZIP is 56,596 bytes; its SHA-256 is b872c39de4f72ac1521eb1af5ca7681078dedb0969d7b634ef8a72e3b7f4c4. Hosted ServiceNow credentials and instance identifiers are excluded. Both audited suites are open research benchmarks [Trivedi et al., 2024, Drouin et al., 2024].