Paper deep dive
When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation
Avyay M. Casheekar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/18/2026, 4:55:52 AM
Summary
This paper critiques current agent evaluation methodologies by identifying two critical conditions often missing from evaluation boundaries: outcome finality (ensuring delayed operations do not alter the scored result) and cross-unit separation (ensuring runs do not interfere with each other via shared state). Through a controlled replay experiment and a documentation review of ten public benchmarks, the authors demonstrate that endpoint scoring often yields different results than terminal scoring and that shared service state allows runs to affect one another. They propose an 'open-effects record' to document unfinished operations and persistent resources to improve evaluation validity.
Entities (9)
Relation Signals (6)
Documentation Review → analyzed → Ten public protocols
confidence 95% · in a review of ten public protocols, we find that all protocols identify when a run stops...
Open-Effects Record → proposedby → Authors
confidence 95% · Finally, we propose an open-effects record that lists operations or resources that may remain relevant after the endpoint
Outcome Finality → requiresevidence → Completion of delayed operations
confidence 95% · Every relevant operation or event that could change the outcome has been resolved or bounded tightly enough to fix the label.
Cross-Unit Separation → requiresevidence → Isolation of runs
confidence 95% · No relevant operation or shared state lets one run change another’s relevant conditions or outcome.
Shared State → causes → Cross-run interference
confidence 90% · a delayed write changes the next run’s score when service state persists between runs
Controlled Replay → demonstrates → Impact of delayed writes on scores
confidence 90% · a delayed write changes the next run’s score when service state persists between runs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial. However, interpreting the score as a final result would require two conditions that the endpoint does not itself necessarily establish: outcome finality and cross-unit separation. These conditions are independent, since reconciling a delayed outcome can settle the label while runs still share state and isolating runs can prevent carryover while the scored outcome remains unfinished. We develop a completion argument that specifies the evidence needed for each decision and argue that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty. First, in a controlled replay to demonstrate the mechanism where an agent's actions were held fixed, we find that the endpoint and terminal labels differ for every delayed operation, while a delayed write changes the next run's score when service state persists between runs but not after isolation or verified reset. Second, in a review of ten public protocols, we find that all protocols identify when a run stops and what is scored, while unfinished operations and the evidence for treating runs as separate trials are documented less consistently. Finally, we propose an open-effects record that lists operations or resources that may remain relevant after the endpoint, their current status, and whether they could change the scored outcome or affect another run.
Tags
Links
- Source: https://arxiv.org/abs/2608.14940v1
- Canonical: https://arxiv.org/abs/2608.14940v1
Trouble viewing inline? Open PDF directly →
Full Text
52,381 characters extracted from source content.
Expand or collapse full text
When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation Avyay M Casheekar University of Michigan Law School Abstract Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial. However, interpreting the score as a final result would require two conditions that the endpoint does not itself necessarily establish: outcome finality and cross-unit separation. These conditions are independent, since reconciling a delayed outcome can settle the label while runs still share state and isolating runs can prevent carryover while the scored outcome remains unfinished. We develop a completion argument that specifies the evidence needed for each decision and argue that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty. First, in a controlled replay to demonstrate the mechanism where an agent’s actions were held fixed, we find that the endpoint and terminal labels differ for every delayed operation, while a delayed write changes the next run’s score when service state persists between runs but not after isolation or verified reset. Second, in a review of ten public protocols, we find that all protocols identify when a run stops and what is scored, while unfinished operations and the evidence for treating runs as separate trials are documented less consistently. Finally, we propose an open-effects record that lists operations or resources that may remain relevant after the endpoint, their current status, and whether they could change the scored outcome or affect another run. 1 Introduction Recent evaluations give agents tasks on websites, in desktop applications, and in tool-mediated environments [37,10,31,7,34]. During a run, the model acts through the available tools until it reports completion or reaches a success condition, timeout, or action limit. That condition is the stop rule. The evaluator scores the resulting state and usually counts the run as one trial. That procedure supports a final result from a separate trial only when later events cannot change the claimed outcome and earlier runs cannot change the relevant conditions of this one. The endpoint is the moment when the evaluator stops requesting actions. A tool call may still be running at that point, while a file, credential, account, or service value may remain available to later runs [33,13]. In the first case, the outcome can change after it is scored. In the second, an earlier run can change the conditions of a later run. The endpoint can therefore end the interaction without settling the result or the boundary between trials. Recent cyber-testing incidents demonstrate persistence and cross-run exposure. The UK AI Security Institute reported concurrent evaluation samples finding and reusing accounts and artifacts left by other samples [1]. OpenAI and Hugging Face separately described evaluation activity that used external services and continued beyond the original runtime [17,23]. These reports show that effects crossed the nominal evaluation boundary in those settings. They do not show how often the same problem occurs elsewhere. 1 arXiv:2608.14940v1 [cs.AI] 14 Aug 2026 Consider a write requested before the endpoint but applied later. An immediate score can differ from the score after the operation finishes. If the reported outcome is whether the write succeeded, a final label requires evidence of success or failure. This is outcome finality. If the service is shared, the same write may change a later run. Justifying each run as a separate trial requires cross-unit separation. Waiting for the write can establish finality while leaving its state visible to the next run. Isolation can establish separation while the write remains unfinished. The choices about when to stop, when to score, and how to separate runs form the evaluation boundary. We develop a completion argument that states the outcome represented by the score, the period over which it is judged, and what counts as one observation. It then accounts for unfinished operations and shared resources. A final success or failure label requires that no remaining effect can change it. Treating runs as separate trials requires that no relevant route connects them. When a connection remains, the analysis must represent it or group the connected runs. We also propose an open-effects record that lists unfinished operations and persistent resources, what they could affect, and how the protocol handles them. 1.1 What benchmark audits leave unresolved The closest existing work asks whether an agent benchmark supports the claims made from its scores. It is relevant because a task and grader can be sound while the state being scored is unfinished or altered by another run. The Agentic Benchmark Checklist and automated auditing examine benchmark specifications, environments, and scoring logic [38,29]. Zhu et al. also ask whether legacy state is removed when tasks are intended to be independent. Recent work also audits what benchmark papers disclose, tests whether agents can exploit benchmark flaws, and asks whether a protocol keeps the intended capability necessary for success [22,28,25]. Work on parallel computer-use rollouts similarly warns that shared external services can let runs affect one another [38,13]. This work can reveal failures of reset or isolation. It does not distinguish whether the scored outcome has finished from whether any connection between runs is represented in the analysis. A grader can correctly classify a state that was sampled too early or changed by another run. The evidence is often split between implementation details and protocol documents. Declare and Justify and Audit Cards make the assumptions behind a result reportable [3,27]. Interactive Evaluation Requires a Design Science calls for claim-matched reporting of stopping, persistence, and reset [33]. The completion argument applies these ideas to the move from a stopped run to a final result and a separate trial. Gao and Zhou report evidence-supported bounds when stored artifacts do not determine whether a run succeeded [11]. That work asks whether the available evidence supports the scorer’s verdict. Computer-use scoring addresses a different part of the process by asking whether the measurement pipeline returns the right verdict for the state it observes [9]. VIGIL separates an embodied agent’s decision to stop from completion in the environment [5]. Work on temporal interfaces and asynchronous tool use makes delayed feedback explicit [19,26]. The remaining question is what the evaluator may infer after the interaction stops and before any remaining effects have been accounted for. This question supplements broader tests of task, construct, and grader validity. When relevant tool calls finish before returning and the outcome is defined at the endpoint, finality follows directly. Separation follows when each run begins from verified fresh state and no operation from another run can later change its relevant conditions or outcome. Otherwise, the protocol must state what can still change and why the label and analysis unit support the claim. 2 2 Two conditions for completion When a run ends, the evaluator has an observation of the environment at a particular time. A scoring rule assigns a label to that observation. It cannot by itself determine whether a later event will change the outcome or whether an earlier run helped produce the observed state. Treating the score as a final result and treating the run as a separate trial therefore require different evidence. Outcome finality holds when the available evidence fixes the outcome named by the evaluation over the period specified in advance. If uncertainty remains, the result can still be reported, but it should be marked unresolved rather than assigned a final success or failure label. The phrases visible at the endpoint, committed within 24 hours, and eventually committed describe different outcomes. A snapshot can establish that a field had valuevwhen the interaction ended. It cannot establish that a delayed write eventually succeeded unless the protocol observes the operation long enough to learn its result. The rest of the environment need not stop changing. Only events that could change the reported outcome matter. The protocol often uses the endpoint to decide where one reported trial ends and the next begins. The analysis unit is the observation counted separately when scores are aggregated. Cross-unit separation holds when one run cannot change another run’s relevant starting conditions or outcome. This does not require every source of correlation between runs to disappear. It concerns effects carried from one run into another. If the evaluation claims to measure performance from fresh state, inherited state changes the task being measured. If the evaluation instead concerns a continuing assistant, persistence may be part of the task. The connected stream rather than each session is then the appropriate unit. When a route between runs remains, the analysis can model the connection or group the connected runs, but it cannot treat them as separate trials. DecisionEvidence neededDecision supported Response when evidence is missing Outcome finalityEvery relevant operation or event that could change the outcome has been resolved or bounded tightly enough to fix the label. Assigning a final success or failure label. Wait for completion, cancel and confirm, or report the case as unresolved. Cross-unit separation No relevant operation or shared state lets one run change another’s relevant conditions or outcome. Counting runs as separate analysis units. Isolate and verify, or stop counting connected runs separately by modeling the connection or grouping the runs. Table 1. Outcome finality and cross-unit separation require different evidence and support different decisions. The distinction determines what evidence is useful. Following a delayed operation can settle its outcome while leaving the state it created available to another run. Isolating the service can protect later runs while the original operation remains unfinished. Restoring shared state protects later runs only after pending effects can no longer reintroduce it. A general statement that the environment was cleaned up is therefore insufficient. The protocol must identify what was restored, how restoration was checked, and which of the two decisions that evidence supports. Some evaluations satisfy both conditions by construction. A synchronous simulator may finish every tool call before returning and restore all relevant state before the next run begins. An asynchronous evaluation may instead stop the interaction while a service is still applying a write. 3 A snapshot at timetcan validly measure state attwithout supporting a claim that the task was completed. Measurement validity depends on the interpretation attached to an observation [15]. The evaluation boundary must therefore be judged against the outcome and analysis unit that the report actually claims. 3 Local stopping and global completion Distributed systems show why stopping one process does not establish that its work is complete. A process can be idle while a message or operation remains in transit, so termination detection accounts for the wider system [8,4]. An agent evaluation often observes the model’s interaction loop more directly than it observes the services reached through tools. Once work leaves the part watched by the evaluator, the endpoint cannot show that the outcome will no longer change. Causal inference describes the corresponding cross-run problem as interference, which occurs when what happens to one unit changes another’s outcome [14,2]. If run B can use a credential, artifact, message, or stored value produced by run A, then B’s conditions depend on A despite its new run identifier. This changes the interpretation of metrics that assume separate trials. Tau-bench, for example, definespass k as the probability that allkindependent and identically distributed trials succeed [34]. The formula can still be computed when one trial changes the next, but the result no longer has that interpretation. A stop rule can be part of the outcome or merely end observation. Work on reinforcement-learning time limits distinguishes task horizons from cutoffs imposed on observation [24]. A claim of success within 30 actions makes the action limit part of the task. A 30-action observation budget does not establish that work already started has no later consequence. The protocol must state which role its limit serves. Software testing and transaction processing suggest stronger checks. A test whose result changes with execution order reveals shared state [36]. A saga compensates for parts of a long transaction that have already committed [12]. A reset supports separation only for resources that it demonstrably restores. A cancellation settles delayed work only after the service confirms that the operation can no longer commit. For agent runtimes, Cordon stages external effects until a task-level semantic transaction has been validated [6]. An evaluation with access to open systems cannot enumerate every possible consequence. It can state the scope of its claim and examine the routes exposed by its environment, including processes, queues, callbacks, credentials, shared stores, external accounts, public artifacts, and retained memory. For each relevant route, the protocol explains how an effect would be detected and what would show that it can no longer alter the outcome or another run. A synchronous simulator may require a short account. Open internet access requires a broader one. Rule for final labels. Assign a final success or failure label only after every relevant route that could change the outcome has been blocked, followed until its effect is known, or bounded tightly enough to determine the label. Otherwise, report the outcome as unresolved. Rule for separate trials. Count runs separately only after every relevant route between them has been blocked or shown unable to change either run’s relevant conditions or outcome. Otherwise, model the connection or treat the connected runs as a larger unit. Blocking a route requires both preventing the effect and checking that the prevention worked. Following a delayed operation requires observing it until the service reports that it completed, 4 Evaluation endpoint The interaction stops and the current state is scored. ↙↘ Outcome finality Can a later event change the reported outcome? Cross-unit separation Can one run change another? ↓ Wait for the result, cancel and verify, preserve uncertainty, or bound the remaining effect. Isolate and verify, represent the connection, group the runs, or bound carryover. Figure 1. The endpoint can leave either decision open. Waiting for delayed work can establish finality while isolation or grouped analysis addresses connections between runs. failed, or was cancelled and can no longer commit. If neither is possible, the result can preserve the unresolved status, the analysis can represent the connection between runs, or the remaining effect can be bounded. A bound is sufficient only when every value within the justified range leads to the same label or conclusion. A bound can support a conclusion without making connected runs separate. Otherwise, the result remains unresolved. Following a write can settle run A’s result while leaving its state visible to run B. Grouping A and B can account for that connection without treating them as separate repetitions. The protocol must preserve this distinction when it finalizes results and aggregates runs. 4 Boundary policies change the recorded result To isolate the boundary choices, the controlled replay holds the tool calls and operation schedule fixed while varying when the delayed operation is scored and whether service state is shared. The requested operation, delay, and execution order remain fixed. Any difference in the label or the next run’s state therefore comes from the evaluation policy in this constructed system. 4.1 Design The implementation uses AgentDojo 0.1.35 to represent the tasks, tools, environments, and function- call traces [7]. A standalone runner replays the fixed calls without querying a model. Each run receives fresh task state, but a local HTTP service can retain state across runs and apply a write after a chosen delay. The service has no public-network access and creates no external resources. This design lets task state reset while service state remains unchanged. To test outcome finality, the trace schedules a write and ends when the service acknowledges the request. The acknowledgement can arrive before the write is applied. The operation reaches a terminal state once it has completed or been cancelled and can no longer write. Snapshot scoring reads service state at the endpoint. Reconciliation waits for a terminal state and then scores. Verified cancellation cancels a pending write, confirms that the cancellation is terminal, and then scores. A synchronous write with the same successful result serves as a control because it finishes before returning. To test cross-unit separation, run A schedules a service write and run B begins with fresh task state. Cross-run exposure occurs if A’s write changes the value scored in B. Under shared state, both runs use the same service value. Under namespaced state, their values are stored under different keys. Verified reset waits until A’s scheduled operation is terminal, then restores and 5 checks B’s initial service value before B begins. A no-write control tests whether B changes when A schedules nothing. Running B before A tests whether exposure follows A’s write. Grouped analysis leaves the shared state unchanged but counts each connected pair as one unit. Each condition uses five matched schedule identifiers, delays of 0, 25, 100, and 250 ms, and ten repetitions of each combination. Each identifier is reused across policies to match otherwise identical trials. The identifiers do not represent sampled model or benchmark randomness. Median measured delays were 0.34, 29.80, 104.66, and 254.85 ms. All trials completed, all 12 implementation checks passed, and no run was excluded. Repetition checks the constructed timing conditions and does not define a larger population. 4.2 Results At zero delay, endpoint and terminal labels agree in all 50 snapshot trials. At nonzero delays, they disagree in all 150 because scoring precedes the write. Snapshot scoring records 50 of 200 successes, while reconciliation records 200 of 200. Verified cancellation cancels and assigns failure to the 150 writes still pending. The 50 zero-delay writes had already succeeded. The synchronous control produces no disagreement in 600 trials across the three scoring policies. The asynchronous operation scores below that control under snapshot scoring and cancellation and ties it under reconciliation. Under shared state, A changes B in 150 of 200 pairs, covering every nonzero delay. At zero delay, B’s initialization overwrites A’s completed write. Exposure is absent from all 200 namespaced pairs, all 200 verified-reset pairs, and all 200 no-write controls. Running B before A reduces exposure from 150 of 200 pairs to 0 of 200. Grouping keeps the 150 exposures but changes the analysis from 400 run rows to 200 pair-level units. TestPolicy or conditionExact outcomeWhat changes Finality Snapshot0/150 at nonzero de- lays The outcome is unresolved when scored. Finality Reconciliation150/150 at nonzero delays The write’s success is observed. Finality Verified cancellation0/150 at nonzero de- lays Each pending operation reaches a confirmed cancelled state and is scored as failure. Finality Synchronous control0/600 disagreements Endpoint and terminal labels agree. SeparationShared state150/200 exposures B’s mean score is 0.75 lower than under namespacing. SeparationNamespacing, reset, and no-write controls 0/200 in each condi- tion The route between A and B is absent or blocked. SeparationGrouped pairs150/200 exposures and 400 rows reduced to 200 pairs Exposure remains. Each pair is one analysis unit. Table 2. Exact outcomes across the fixed schedule. The counts describe constructed conditions rather than a benchmark population. With fixed traces, the scoring time changes the label and the treatment of service state determines whether run A changes run B. Because the delays and shared state were imposed by design, these counts do not estimate their frequency or effect size in published benchmarks. Taken together, the two tests show that the endpoint alone cannot justify either the final label or the analysis unit in this system. Whether published protocols provide the additional information needed for those decisions is a separate question. The documentation review examines what the public sources for those protocols say about unfinished operations, persistent state, reset, and separate scoring. 6 5 Documentation across ten protocols The review covers the papers and official documentation for WebArena, WorkArena, OSWorld, SWE-bench, tau-bench, ToolSandbox, TheAgentCompany, RE-Bench, Cybench, and AgentCanary [37,10,31,16,34,21,32,30,35,20]. Five of these protocols also appear in the audit by Zhu et al. [38]. The other five extend the sample to settings that include persistent services, long-running computation, and system-level monitoring. The sample covers different ways that effects can outlast a run rather than representing all agent evaluations. The counts therefore apply only to these ten protocols. For each protocol, the review records what counts as one run, what ends the interaction, what state the scorer observes, whether operations can remain active, what state can persist, how it is reset or retained, and what supports counting runs separately. A field is explicit only when a public source states the rule and the resources it covers. A partial code means that the rule is stated but its scope or evidence is incomplete. Not reported means that no statement was found. Not applicable means that the declared execution model excludes the kind of unfinished operation being coded. The review covers sources available on 15 August 2026. Reset or deliberate retention is explicit in eight protocols and partial in two. The treatment of unfinished operations is reported much less often. Six protocols expose shells, browsers, virtual machines, services, or long-running computation without stating whether descendant processes, queued actions, or delayed effects finish, are cancelled, or are checked before scoring. SWE-bench documents timeouts and a cleanup option, while AgentCanary records system activity. Neither gives a general rule establishing that pending actions are terminal before scoring. The field is not applicable to tau-bench and ToolSandbox, whose declared environments use synchronous functions. Three protocols provide explicit evidence for treating runs as separate observations. Seven describe reset, teardown, or fresh provisioning without fully stating which resources are covered or how successful restoration is verified. A reset procedure supports separate scoring only to the extent that its scope and successful completion are documented. The review does not code outcome finality itself. A scoring rule identifies the state used to assign a label. Finality also depends on the outcome that the label is meant to represent and how long relevant effects can change it. The documents did not state those periods consistently, and only one reviewer coded the materials. Missing periods were therefore not inferred. The results describe documentation of unfinished operations rather than counting protocols that satisfy finality. Publicly visible fieldExplicit Partial NR NA Reset or deliberate retention82 0 0 Open or descendant work02 6 2 Evidence for separate scoring37 0 0 Table 3: Documentation reported for the ten reviewed protocols. NR means not reported and NA means not applicable. Appendix A gives the protocol matrix and source for each code. Six protocols do not report how unfinished work is handled, and seven provide only partial evidence for separate scoring. These are findings about public reporting. Undocumented safeguards remain outside the review, which cannot establish benchmark validity or the size of any score change. 7 6 Making completion reviewable A protocol can state when interaction ends and still omit the reasoning needed to treat the resulting label as final or the run as a separate trial. The completion argument states that reasoning. It identifies what the score represents and what the analysis counts separately, then explains how the protocol handles anything that can remain active or pass between runs. An argument that can be checked from the protocol answers the following questions. 1.What outcome does the score represent, and what counts as one observation? State the claimed result and the unit used to aggregate scores and estimate uncertainty. 2.What ends the interaction, and what determines the score? Distinguish the stop rule from the state, event, or observation period used to decide the outcome. 3.What can remain active or reach another run? Consider descendant processes, queues, shared stores, credentials, external resources, and retained memory that the environment makes available. 4.How does the protocol handle each route? State whether it is observed, cancelled, isolated, reset, grouped, bounded, or retained as unresolved. For any claimed cancellation, isolation, or reset, give evidence that it succeeded. The required account depends on the environment. In a coding evaluation with public egress disabled, a task-specific container and repository can form one observation if the test process tree finishes before scoring and relevant state is not reused. A timeout that leaves a descendant process alive leaves the label unresolved unless the process cannot affect the claimed outcome. A transaction may require observing terminal settlement rather than submission. A persistent assistant may deliberately carry state across sessions, making the continuing stream the appropriate unit. Synchronous tools in a closed simulator may establish finality when they return, while fresh namespaces without shared credentials may establish separation. Environments that expose more routes require more evidence. When a route cannot be closed or followed, the protocol can narrow the claim or bound its possible effect. The bound settles the conclusion only when every value within it leads to the same result. 6.1 Open-effects records An ordinary trajectory log can show that the agent requested an operation without showing whether it later succeeded, failed, or changed shared state. We propose an open-effects record for operations and resources that may still matter after the endpoint. An entry is created when an action starts an operation or creates a resource that may remain active, change later, or be visible to another run. It records the action, responsible system, stable observation handle, endpoint status, possible later states, and any route to another run. At the endpoint, the evaluator uses the handle to query the system and records whether the effect is terminal. If the protocol cancels or removes it, the record also retains evidence that this action succeeded. Appendix C gives the full specification. The record makes evidence inspectable but does not establish finality or separation. Tool wrappers can help populate it. Agent self-reports can supply additional entries [18], but either source may be incomplete. System-level checks remain necessary when available. An empty record does not establish that no relevant effect exists. 8 6.2 Score interpretation and unit choice When an effect remains open, the protocol must say whether the reported quantity describes the endpoint or the eventual outcome. If models differ in how often pending operations later succeed, scoring at the endpoint and after completion need not rank them in the same way. If one run can change another, the row count can overstate the number of separate attempts. This matters for metrics such as pass k , whose interpretation assumes independent trials [34]. The analysis should match any connection that remains. A protocol can isolate runs and verify the isolation, model the connection, or count a connected set as one unit. Uncertainty should be estimated at that level [14,2]. This account may be one sentence for a synchronous task defined at the endpoint. More exposed routes or a broader claim require more detail. An endpoint score may still support a narrower claim. A snapshot can measure submission without measuring settlement. Connected runs can measure a stream without representing separate fresh attempts. The completion argument states that interpretation and the evidence needed for a stronger one, following the measurement principle that an operationalization must support its intended interpretation [15]. 7 What the evidence establishes The endpoint determines when scoring occurs, but the meaning of that score depends on what can still change and what can carry over. A final success or failure label requires that later effects cannot change the claimed outcome. When that is not established, the result should remain unresolved. A separate trial requires that carryover is blocked. When a connection remains, the analysis must represent it or use a larger unit. The replay shows that these decisions can diverge under delayed operations and shared state. The review finds gaps in how ten public protocols document unfinished operations and the basis for separate scoring. Neither study estimates how often these mechanisms affect published results. The replay uses a benign local service, fixed traces, scheduled delays, and no model API. The review covers ten selected protocols, was coded by one reviewer, and cannot observe private implementation. An open environment cannot prove that every unknown effect is absent. The protocol can state which effects are in scope, observe the routes its tools expose, and follow those effects for the period required by the claim. Wider boundaries can require more time and monitoring. Grouping connected runs reduces the number of separate observations. Completion also differs from safety. Waiting for a harmful operation to finish can reveal its outcome while allowing the harm to occur. Cancelling it may improve safety while changing the outcome that would otherwise have been observed. The rules in this paper govern what a result can mean. They do not replace incident response or safety controls. Agent evaluations need endpoints, but an endpoint marks only when the evaluator stops requesting actions. A final label and a separate trial require an account of what could still happen, what could carry over, and how each case is handled. The completion argument and open-effects record make that account inspectable. When the evidence is insufficient, the score should be described more narrowly or connected runs should be analyzed as a larger unit. 9 References [1] AI Security Institute. Incident report: Unsanctioned agent behaviour during cy- ber testing.Blog post, August 2026.URLhttps://w.aisi.gov.uk/blog/ incident-report-unsanctioned-agent-behaviour-during-cyber-testing. [2]Peter M. Aronow and Cyrus Samii. Estimating average causal effects under general interference, with application to a social network experiment. The Annals of Applied Statistics, 11(4): 1912–1947, 2017. doi: 10.1214/16-AOAS1005. [3]Peter Barnett and Lisa Thiergart. Declare and justify: Explicit assumptions in AI evaluations are necessary for effective regulation. arXiv preprint arXiv:2411.12820, 2024. URLhttps: //arxiv.org/abs/2411.12820. [4]K. Mani Chandy and Leslie Lamport. Distributed snapshots: Determining global states of distributed systems. ACM Transactions on Computer Systems, 3(1):63–75, 1985. doi: 10.1145/214451.214456. [5] Ying Chen, Lihuang Fang, Rui Jiang, Mingxu Wang, Zhifeng Gu, Lei Yi, and Jie Chen. Done, but not sure: Disentangling world completion from self-termination in embodied agents. arXiv preprint arXiv:2605.08747, 2026. URL https://arxiv.org/abs/2605.08747. [6]Zheng Chen, Hanqing Liu, Duling Xu, Dong Dong, Jialin Li, Bangzheng Pu, and Jidong Zhai. Cordon: Semantic transactions for tool-using LLM agents. arXiv preprint arXiv:2606.17573, 2026. URL https://arxiv.org/abs/2606.17573v1. [7]Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. URL https://arxiv.org/abs/2406.13352. [8] Edsger W. Dijkstra and C. S. Scholten. Termination detection for diffusing computations. Information Processing Letters, 11(1):1–4, 1980. doi: 10.1016/0020-0190(80)90021-6. [9] Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, and Rui Qian. How benchmarks mis-score computer-use agents. arXiv preprint arXiv:2607.28367, 2026. URL https://arxiv.org/abs/2607.28367. [10]Alexandre Drouin, Maxime Gasse, Massimo Caccia, et al. WorkArena: How capable are web agents at solving common knowledge work tasks? In Proceedings of the 41st International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2403.07718. [11]Shanshan Gao and Liyi Zhou. Can agent benchmarks support their scores? evidence-supported bounds for interactive-agent evaluation. arXiv preprint arXiv:2605.10448, 2026. URLhttps: //arxiv.org/abs/2605.10448v1. [12]Hector Garcia-Molina and Kenneth Salem. Sagas. In Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, pages 249–259, 1987. doi: 10.1145/38713. 38742. [13] Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, and Xin Eric Wang. Scaling agents for computer use. arXiv preprint arXiv:2510.02250, 2025. URL https://arxiv.org/abs/2510.02250. 10 [14]Michael G. Hudgens and M. Elizabeth Halloran. Toward causal inference with inter- ference. Journal of the American Statistical Association, 103(482):832–842, 2008. doi: 10.1198/016214508000000292. [15]Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 375–385, 2021. doi: 10.1145/3442188.3445901. URL https://arxiv.org/abs/1912.05511. [16]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/ 2310.06770. [17] Hugo Larcher, Adrien Carreira, Raphaël Gontijo Lopes, and Christophe Rannou. Anatomy of a frontier lab agent intrusion: A technical timeline of the july 2026 inci- dent. Hugging Face technical postmortem, 2026. URLhttps://huggingface.co/blog/ agent-intrusion-technical-timeline. [18]Bruce W. Lee, Yueh-Han Chen, and Tomek Korbak. Training agents to self-report misbehavior. arXiv preprint arXiv:2602.22303, 2026. URL https://arxiv.org/abs/2602.22303. [19] Jialian Li, Yuchen Cao, Junhong Liu, Weiran Guo, Xutao Wang, Jiaming Song, Jiahao Zhang, and Jie Chen. Engagement process: Rethinking the temporal interface of action and observation. arXiv preprint arXiv:2605.11484, 2026. URL https://arxiv.org/abs/2605.11484. [20]Peiyang Li, Songping Wang, Yi Huang, Yanhua Shi, Chenhao Zhang, Qi Li, Yueming Lyu, Caifeng Shan, Fengting Li, Chao Feng, Chuanqun Zhu, and Liang Chen. AgentCanary: A security evaluation framework for autonomous AI agents in real executable environments. arXiv preprint arXiv:2606.10484, 2026. URL https://arxiv.org/abs/2606.10484. [21]Jiarui Lu, Thomas Holleis, Yizhe Zhang, et al. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025. URL https://arxiv.org/abs/2408.04682. [22]Mahdi Naser Moghadasi and Faezeh Ghaderi. What twelve LLM agent benchmark pa- pers disclose about themselves: A pilot audit and an open scoring schema. arXiv preprint arXiv:2605.21404, 2026. URL https://arxiv.org/abs/2605.21404. [23]OpenAI.OpenAI and Hugging Face partner to address security incident dur- ing model evaluation. Security disclosure, 2026. URLhttps://openai.com/index/ hugging-face-model-evaluation-security-incident/. [24]Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 4045–4054. PMLR, 2018. URLhttps://proceedings.mlr.press/v80/pardo18a.html. [25]Jiaqi Shao, Hanck Chen, Wei Zhang, Maxm Pan, and Bing Luo. Do agent benchmarks measure capability? protocol validity in the age of agentic AI. arXiv preprint arXiv:2607.22368, 2026. URL https://arxiv.org/abs/2607.22368. [26] Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, and Feng Zhao. AsyncTool: Evaluating the asynchronous function 11 calling capability under multi-task scenarios. arXiv preprint arXiv:2605.27995, 2026. URL https://arxiv.org/abs/2605.27995. [27]Leon Staufer, Mick Yang, Anka Reuel, and Stephen Casper. Audit cards: Contextualizing AI evaluations. arXiv preprint arXiv:2504.13839, 2025. URLhttps://arxiv.org/abs/2504. 13839. [28] Hao Wang, Hanchen Li, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song. Do androids dream of breaking the game? systematically auditing AI agent benchmarks with BenchJack. arXiv preprint arXiv:2605.12673, 2026. URLhttps://arxiv.org/abs/2605. 12673. [29] Junlin Wang, Federico Bianchi, Shang Zhu, Fan Nie, Yongchan Kwon, Bhuwan Dhingra, and James Zou. Automated benchmark auditing for AI agents and large language models. arXiv preprint arXiv:2605.26079, 2026. URL https://arxiv.org/abs/2605.26079. [30]Hjalmar Wijk, Tao Lin, Joel Becker, et al. RE-Bench: Evaluating frontier AI r&d capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024. URL https://arxiv.org/abs/2411.15114. [31]Tianbao Xie, Danyang Zhang, Jixuan Chen, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2404.07972. [32] Frank F. Xu, Yufan Song, Boxuan Li, et al. TheAgentCompany: Benchmarking LLM agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024. URLhttps: //arxiv.org/abs/2412.14161. [33]Keyang Xuan, Peiyang Song, Pan Lu, Pengrui Han, Wenkai Li, Zhenyu Zhang, Zexue He, Wenyue Hua, Manling Li, Jiaxuan You, Adrian Weller, Yizhong Wang, and Jiaxin Pei. In- teractive evaluation requires a design science. arXiv preprint arXiv:2605.17829, 2026. URL https://arxiv.org/abs/2605.17829. [34]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. URL https://arxiv.org/abs/2406.12045. [35] Andy K. Zhang, Neil Perry, Riya Dulepet, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926, 2024. URL https://arxiv.org/abs/2408.08926. [36]Sai Zhang, Darioush Jalali, Jochen Wuttke, Kivanc Muslu, Wing Lam, Michael D. Ernst, and David Notkin. Empirically revisiting the test independence assumption. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 385–396, 2014. doi: 10.1145/2610384.2610404. [37] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2307.13854. 12 [38]Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang. Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825, 2025. URL https://arxiv.org/abs/2507.02825. 13 A Documentation review matrix Table A.1 gives the protocol-level codes summarized in Table 3. Open work means an operation or descendant process that may remain active after interaction stops. Separate evidence means public evidence that supports counting runs separately. The remaining codes use the rubric stated in Section 5. ProtocolOpen work Reset or retention Separate evidence WebArenaNREP WorkArenaNREP OSWorldNREP SWE-benchPEE τ-benchNAEE ToolSandboxNAEP TheAgentCompanyNRPP RE-BenchNRPP CybenchNREP AgentCanaryPEE Table A.1. Protocol-level documentation codes. E means explicit, P means partial, NR means not reported, and NA means not applicable to the declared task and claim. Each public protocol is one review unit. The codes do not apply to every task or to a private deployment. A not reported code means only that the reviewed public sources do not state the rule. One reviewer assigned each code using the rubric in Section 5. The ledger below gives the source and reason for every decision so that the coding can be checked, but it was not independently recoded. ABC refers to the Agentic Benchmark Checklist report by Zhu et al. [38]. Table A.2. Cell-level source ledger for the three coded fields. ProtocolFieldCode Source locationBasis WebArenaOpen workNR Paper Sec. 3.2 and App. A.9 Stop actions and state checkers are specified, but handling of background work before scoring is not [37]. Reset or retentionE Paper App. A.2 and ABC T.4 The self-hosted sites can be restored, and the ABC review reports that site state is cleared between runs [37, 38]. Separate evidenceP Paper App. A.2 and ABC T.4 Site restoration is described, but the reviewed sources do not document whether state outside those sites is reset [37, 38]. WorkArenaOpen workNR Paper Secs. 4.2 and 5.2 The task lifecycle is specified, but the sources do not give a general rule for descendant or queued work [10]. Reset or retentionE Paper Sec. 4.2Setup and teardown hooks cover resources created for the task [10]. Separate evidenceP Paper Secs. 4.2 and 5.2 Cleanup is described, but the sources do not report a check that every covered resource was restored [10]. OSWorldOpen workNR Paper Secs. 2.1–2.2 Custom evaluators inspect machine state, but handling of descendant processes before scoring is not stated [31]. Continued on next page 14 Cell-level source ledger continued ProtocolFieldCode Source locationBasis Reset or retentionEPaper Sec. 2.2.1 and App. B.5 Each task reverts its virtual machine to a specified snapshot before applying the task setup [31]. Separate evidenceP Paper Sec. 2.2.1, App. B.5, and ABC T.4 Virtual machine restoration is explicit, but the sources do not document coverage of services outside the machine [31, 38]. SWE-bench Open workP Paper App. A.4 and official harness documentation Test execution has a per-instance timeout and the harness exposes a cleanup control, but the sources do not state that descendant work is terminal before grading [16]. Reset or retentionE Paper App. A.4 and official harness documentation Instance-specific Docker images provide reproducible task environments [16]. Separate evidenceE Paper App. A.4 and official harness documentation Each instance is evaluated in a task-specific container, and re-evaluation requires a new run ID because results are cached by run ID and instance ID [16]. τ-benchOpen workNA Paper Sec. 3The declared environment uses synchronous deterministic Python tools over a task database [34]. Reset or retentionE ABC report, τ-bench T.4 The ABC review reports that the database is reinitialized between runs [38]. Separate evidenceE Paper Sec. 3 and ABC T.4 The metric defineskindependent and identically distributed trials, and database reinitialization supplies the stated separation mechanism [34, 38]. ToolSandbox Open workNA Paper Sec. 2 and Apps. A.1–A.2 Tool execution changes an in-memory execution context through Python functions rather than an asynchronous external service [21]. Reset or retentionE Paper Apps. A.1 and B.2 Each scenario specifies the starting world state held in its execution context [21]. Separate evidenceP Paper Apps. A.1 and B.2 Scenario-local state is defined, but the sources do not justify the choice of a separate analysis unit [21]. TheAgent CompanyOpen workNR Paper Secs. 3–4 and 6 The workspace and intranet support extended tasks, but handling of pending processes before scoring is not stated [32]. Reset or retentionP Paper Secs. 3–4Initialization and finalization hooks are described without a complete account of the resources they cover [32]. Separate evidenceP Paper Secs. 3–4The environment is self-contained, but the sources do not report a check that all relevant state was restored [32]. RE-BenchOpen workNR Paper Secs. 3.2 and 5.2, App. A.2.1 Agents can run long computations and background processes, but the sources do not state what happens to live processes when the time budget ends [30]. Reset or retentionP Paper Secs. 4 and 4.2 Runs use secure virtual machines, but the sources do not fully describe what is restored between attempts [30]. Separate evidenceP Paper Secs. 4 and 4.2 The virtual machine limits exposure, but reset scope and successful restoration are not fully documented [30]. CybenchOpen workNR Paper Sec. 2.1 and App. D Agents execute commands in task containers, but handling of descendant work before scoring is not stated [35]. Reset or retentionE ABC report, Cybench T.4 The ABC review reports that Docker state is cleared between runs [38]. Continued on next page 15 Cell-level source ledger continued ProtocolFieldCode Source locationBasis Separate evidenceP Paper App. D and ABC T.4 Container reset is explicit, but the reviewed sources do not give the same evidence for state outside the container [35, 38]. AgentCanary Open workP Paper Sec. 5.3System monitoring records spawned processes, file changes, network connections, and other state changes, but the paper does not state a general rule for bringing pending actions to a final state before scoring [20]. Reset or retentionE Paper Sec. 5.3 Single-instance tasks reset after evaluation, while state is retained only within declared multi-session instances [20]. Separate evidenceE Paper Sec. 5.3Temporary per-task containers isolate state retained within one task from other tasks [20]. The ledger covers the cited papers and official documentation available on 15 August 2026. Paper section labels refer to the cited versions. B Controlled replay details Outcome finality test Fixed trace schedules a delayed write −→ Interaction ends after the request is acknowledged −→ Score at the endpoint, after completion, or after verified cancellation Compare the endpoint label with the terminal result Cross-unit separation test Run A schedules a service write −→ Run A ends and run B begins −→ Use shared state, a namespace, verified reset, or grouped analysis Compare B’s exposure, score, and analysis unit Figure B.1. Controlled replay. The tool-call trace is fixed while the protocol changes when the delayed operation is scored and whether service state is shared, isolated, reset, or represented by a grouped unit. AgentDojo 0.1.35 supplies the task, tool, environment, and function-call structures used in the replay [7]. A standalone runner executes the fixed calls without querying a model. A local HTTP service retains state between runs and applies a write after the scheduled delay. It binds to 127.0.0.1 on an ephemeral port and creates no external resources. The schedule contains identifiers 1–5, write delays of 0, 25, 100, and 250 ms, and ten repetitions for each identifier and delay. The same identifiers match otherwise identical trials across policies, while schedule seed 987654321 fixes their execution order. A separate calibration run produced median measured delays of 0.34, 29.80, 104.66, and 254.85 ms and passed the prespecified timing check. Every scheduled condition completed, all 12 validation tests passed, and there were no failures, exclusions, reconciliation timeouts, or warnings. The reported counts were checked against the frozen schedule, calibration output, run metadata, and 12 implementation tests. The repetitions verify the constructed schedule rather than sample a benchmark population. 16 C Open-effects record specification Section 6.1 proposes an open-effects record because a trajectory can record a request without showing whether its effect later finished or reached another run. An entry is created for an operation or resource that may remain active, change later, or be visible elsewhere. It is updated at the endpoint and after each attempt to observe, cancel, remove, or isolate the effect. A separate field records how the analysis treats any route that remains open. FieldRequired contentUse at the boundary Run and taskRun ID, task ID, and any larger stream or group ID. Connects the effect to the proposed analysis unit. Initiating eventTrace position, action, timestamp, and arguments needed to identify the effect. Records where the effect began. Resource and controller Process, transaction, account, URL, file, queue, job, or service and the system that controls it. Identifies where the effect must be observed or closed. Observation handle Stable process ID, transaction ID, job ID, namespace, or equivalent query handle. Permits a system query without relying on the agent’s description. Effect statusPending, terminal success, terminal failure, cancellation requested, cancelled, persistent, or unknown. Prevents an acknowledgement from being treated as completion. Possible transitionStates still reachable and the period over which they remain relevant to the claimed outcome. Shows whether the label may still change. Cross-run scope Namespace, account, credential, store, or other route through which another run could encounter the effect. Shows whether runs may be counted separately. VerificationMost recent system observation, its time, and the evidence source. Records what was checked rather than what cleanup was intended to do. Analysis disposition Effect closed, connection modeled, runs grouped, remaining effect bounded, or case unresolved. Records how the analysis treats the effect without changing its system status. Table C.1. Fields in an open-effects record. Free text may supplement the fields, but it does not replace a stable handle or system evidence when either is available. At the endpoint, the evaluator queries each handle and records the current system state. It then asks whether a later state could change the claimed outcome or reach another run. A pending or unknown effect remains open. The analysis may report it as unresolved, bound its effect, model the connection, or group exposed runs. None of these choices changes an open system status to closed. Example from the replay. An entry for the delayed write can record run A, the trace position and request time, the local service, the request identifier, the pending status, and the service key visible to run B. A later query records whether the write was applied or cancelled. Until the operation is terminal, run A does not justify a final success or failure label. Run B must use a separate namespace, begin after A’s operation is terminal and the shared state has been reset, or be analyzed with run A as one connected unit. The acknowledgement shows acceptance of the request, not application of the write. Function wrappers, process monitors, transaction logs, and environment instrumentation can populate the record. Agent reports may identify effects first visible in the trajectory, but an empty report does not show that no relevant route exists. 17