Paper deep dive
Copy-on-Write Scoring: Application-Specific Agent Evaluations
Joanna Roy, Sven Hoelzel
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/18/2026, 2:13:10 AM
Summary
Copy-on-Write (CoW) Scoring is a framework for evaluating LLM-based agents directly within application environments. It uses a PostgreSQL-level CoW mechanism to isolate agent writes, preventing interference with production data. By comparing agent executions against ground-truth sessions, it generates session- and operation-level structural and content scores. The framework enables low-cost, application-specific evaluation and iteration, as demonstrated on the Plane project-management platform, where it successfully identified failure modes like vocabulary mismatches and hallucinations, leading to measurable performance improvements after tool surface updates.
Entities (12)
Relation Signals (11)
Copy-on-Write Scoring â uses â PostgreSQL
confidence 98% ¡ This work assumes a PostgreSQL database, although the CoW pattern could similarly be extended to other data stores
Copy-on-Write Scoring â evaluates â LLM-based agents
confidence 97% ¡ Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows
Copy-on-Write Scoring â demonstratedon â Plane
confidence 96% ¡ We demonstrate the framework on Plane, an open-source project-management platform
agent-cow â implements â Copy-on-Write Scoring
confidence 95% ¡ Both components are implemented in the open-source agent-cow library.
Copy-on-Write Scoring â produces â Session-level score
confidence 94% ¡ CoW Scoring produces session- and operation-level scores that highlight where agents' database write operations succeed and fail
Copy-on-Write Scoring â produces â Operation-level score
confidence 94% ¡ CoW Scoring produces session- and operation-level scores that highlight where agents' database write operations succeed and fail
GPT-5 â evaluatedby â Copy-on-Write Scoring
confidence 92% ¡ Five language models were evaluated twice per prompt: GPT-5, GPT-4.1, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, and Gemini-2.5-Pro.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail. Yet existing agent evaluation mechanisms are limited: benchmarks have low construct validity for application-specific workflows and environments, and replica evaluation environments are expensive and prone to drift. We propose Copy-on-Write (CoW) Scoring, a framework that evaluates agent operations directly within application environments using a PostgreSQL-level Copy-on-Write mechanism to isolate agent writes. CoW Scoring produces session- and operation-level scores that highlight where agents' database write operations succeed and fail in a given application environment, enabling inexpensive evaluation and iteration on agent harnesses and tool surfaces. We demonstrate the framework on Plane, an open-source project-management platform, where analysis surfaced specific issues in the tool surface, and corresponding fixes produced measurable improvements on affected models. Python library: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2607.14336v1
- Canonical: https://arxiv.org/abs/2607.14336v1
Trouble viewing inline? Open PDF directly â
Full Text
48,307 characters extracted from source content.
Expand or collapse full text
Copy-on-Write Scoring: Application-Specific Agent Evaluations Joanna Roy 1 Sven H Ě olzel 1 Abstract Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail. Yet existing agent evaluation mecha- nisms are limited: benchmarks have low construct validity for application-specific workflows and en- vironments, and replica evaluation environments are expensive and prone to drift. We propose Copy-on-Write (CoW) Scoring 1 , a framework that evaluates agent operations directly within ap- plication environments using a PostgreSQL-level Copy-on-Write mechanism to isolate agent writes. CoW Scoring produces session- and operation- level scores that highlight where agentsâ database write operations succeed and fail in a given appli- cation environment, enabling inexpensive evalu- ation and iteration on agent harnesses and tool surfaces. We demonstrate the framework on Plane, an open-source project-management plat- form, where analysis surfaced specific issues in the tool surface, and corresponding fixes produced measurable improvements on affected models. 1. Introduction Large Language Model (LLM)-based agents are increas- ingly deployed in production software systems, from en- terprise tooling (e.g., ServiceNow, Salesforce, Notion) to MCP integrations across a wide range of consumer applica- tions. 2 Many of these applications are relied upon by a wide range of organizations and end-users, underpinning core business operations across large segments of the economy. Agent unreliability and errors â such as accidental dele- tion or overwriting of data, misuse of tooling, or incorrect changes to critical records â could have substantial ripple 1 trail-ml, Munich, Germany. Correspondence to: Joanna Roy <joanna@trail-ml.com>, Sven H Ě olzel <sven@trail-ml.com>. Published at the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD) at ICML 2026. Copyright 2026 by the author(s). 1 Python library: agent-cow 2 Gartner (2025); BCG (2025); Deloitte (2026) effects, yet are difficult to identify and prevent in practice, particularly when agent writes are interleaved with the prior database state. Evaluating agents in software contexts remains an under- developed area. Benchmarks are the most common means of assessing performance, but strong benchmark results do not guarantee strong performance in real systems or on real- world tasks (Mohammadi et al., 2025; Raji et al., 2021; Liao & Xiao, 2025; Deng et al., 2023). Existing benchmarks tend to have low construct validity (Bean et al., 2025) for the environments in which software agents are deployed. Some benchmarks have attempted to close this gap by sit- uating agents in replicas of common applications (Drouin et al., 2024; Zhou et al., 2024b; Xu et al., 2025) or in sim- ulated domains with API access (Yao et al., 2024). While these partly address the construct validity gap, two main limitations remain: (a) replicas and simulations are time- and resource-expensive to produce, and can quickly drift from the live state of the application; and (b) evaluating performance on a specific software is still not necessarily predictive of performance in others â for example, one with different tooling, system prompts, and workflows. Since no single benchmark can predict agent performance for arbi- trary applications, teams need complementary evaluation methods that work directly in their application environment, on representative workflows. We propose Copy-on-Write (CoW) Scoring, a framework to develop and safely conduct use-case-specific agent eval- uations in software applications. The CoW mechanism allows agents to operate directly within the application en- vironment, avoiding the cost and drift of replicas while keeping sessions isolated and setup easy to reuse across runs. This work assumes a PostgreSQL database, although the CoW pattern could similarly be extended to other data stores, which would enable scoring a broader range of agent operations. The scoring framework provides session- and operation-level scores that surface where agent writes suc- ceed and fail when interacting with a given tool surface. We demonstrate an evaluation loop on Plane, 3 an open- source project-management platform, where scoring sur- faces multiple failure modes specific to the deployed agents 3 plane.so 1 arXiv:2607.14336v1 [cs.SE] 15 Jul 2026 Copy-on-Write Scoring: Application-Specific Agent Evaluations task_idtitlestatus tasks(view) T-101Fix login bugin_progress T-102Update API docstodo T-103Deploy v2.1done T-106Audit depstodo task_idtitlestatus tasks_base T-101Fix login bugin_progress T-102Update API docstodo T-103Deploy v2.1review T-104Old feature flagblocked task_idtitlestatus session_id operation_id _cow_deleted _cow_updated_at tasks_changes T-103Deploy v2.1done 5a3f...7b2c...false14:02:18 T-104Old feature flagblocked 5a3f...9d1a...true14:02:31 T-106Audit depstodo 5a3f...3f8e...false14:03:07 = + view=base+changes Legend: updated deletedinserted Figure 1. Copy-on-Write (CoW) mechanism in agent evalua- tion. The agent reads and writes through a view, which merges the base table (production data) with the changes table (per-session using the operation/session metadata). Agent writes are intercepted and written to the changes table, leaving base data untouched. and tool surface, and a corresponding fix produces mea- surable improvement on the affected models. The same score-diagnose-fix procedure applies to any application us- ing CoW Scoring, enabling low-cost iteration on the model, prompt, retrieval, or system prompt in a given deployment. 2. Copy-on-Write (CoW) Scoring This section describes the CoW mechanism, which isolates agent database changes, and the CoW Scoring framework, which compares a ground-truth (GT) execution of a given workflow with a corresponding agent execution at the ses- sion and operation level. Both components are implemented in the open-source agent-cow library. 2.1. CoW Mechanism Copy-on-Write (CoW) is a resource-management technique that lets processes share underlying data without creating copies upfront. Rather, reads go to the shared data and writes trigger creation of a copy such that only the writing process observes the change (Silberschatz et al., 2018). Applied to agent operations on databases, CoW prevents agents from writing directly to the underlying production data. To enable CoW on a database, each original table is split into a base table (contains the underlying production data), changes table (contains agent writes), and corresponding view (defined by a query merging base and changes table data). All application operations are sent to the view, since it takes the name of the original table. View triggers redirect CoW write operations to the changes table, with associated CoW metadata columns (sessionid,operationid, cowupdatedat,cowdeleted) appended. Actions from a given session can then be queried using the respective sessionid. The components are visualized in Figure 1, and more details are described in Appendix A. Theagent-cowlibrary handles database setup â creating base tables, changes tables, views, and deploying triggers (Algorithm 1, Appendix A), and the application is respon- sible for passingsessionidandoperationidwith each agent operation. For Plane, CoW integration required âź250 lines of code for CoW functionality, and an addi- tionalâź540 lines specific to our recording infrastructure and portable to a testing harness (Appendix C.1). We have separately integrated CoW Scoring into a closed-source production application, with comparable effort. 2.2. Scoring Scoring takes two CoW sessions as input: a ground-truth (GT) session and an agent session. The GT session is recorded by a human performing the ideal sequence of ac- tions for a given workflow with CoW enabled. The human also writes a prompt describing the workflow such that they would expect an agent to be able to reproduce their actions. The agent is then given this prompt and executes its own sequence of actions with CoW enabled. CoW Scoring happens along two dimensions and at two lev- els of granularity. The two dimensions are structural (which tables, rows, columns, and relationships were written) and content (what values were written). The two levels of gran- ularity are session-level (evaluates the session as a whole) and operation-level (evaluates each operation). Users can supply their own comparators and overall scoring functions; Appendix B.1 describes the library architecture. 2.2.1. SESSION-LEVEL COMPARISON For session-level scoring, we keep the most recent written row state for each primary key. Changed rows are classified as: matched (N matched â rows present in both GT and agent sessions, under UUID mapping), missing (N missing â GT rows the agent never produced), or extra (N extra â agent rows with no GT counterpart). Session-level structural score,s struct : The fraction of matched rows relative to the total number of rows changed. s struct = N matched N matched + N missing + N extra (1) Session-level content score,s content :The mean per-field similarity over all matched rows. s content = 1 N matched X (g,a)âmatched sim(g,a)(2) wheresim(g,a) â [0, 1]is the similarity calculation per row betweeng(GT) anda(agent). Each column has a default similarity comparator according to its data type, but custom comparators can be provided as described in Appendix B.1.1. 2 Copy-on-Write Scoring: Application-Specific Agent Evaluations Notably, CoW Scoring compares final database states rather than action sequences, so the session-level scores are invari- ant to the path the agent took. 2.2.2. OPERATION-LEVEL COMPARISON While the session-level scores evaluate whether the agent reached the desired GT world state, the operation-level scores quantify how useful each operation was in getting there. We first sort operations in topological order, then calculate the following for each operation. Op-level structural utility,u struct (o i ):The change in the session-level structural score before and after o i is applied. u struct (o i ) = s struct (G â¤i )â s struct (G <i )(3) WhereG <i denotes the world state prior to applyingo i , and G â¤i denotes the state after o i is applied. Op-level content utilityu content (o i ):The mean per-field content similarity over operation rows. u content (o i ) = 1 |M i | X (g,a)âM i sim(g,a)(4) whereM i is the subset of matched rows whose agent-session write originated in o i . 3. Methods Because the framework targets application-specific eval- uation rather than a universal benchmark, our empirical validation will not compare against an external ground truth. The preliminary study below aims to show that (a) the CoW mechanism and scoring framework can be implemented in a PostgreSQL application, and (b) resulting scores surface useful information about agent behaviour, which can inform improvements to the tool surface in subsequent iterations. 3.1. Preliminary Study: Application in Plane We apply CoW Scoring to Plane, an open-source project- management platform, as a concrete demonstration of the evaluation workflow. To produce a diverse set of ground- truth (GT) sessions without spending excessive author ef- fort, Claude Opus 4.6 with access to the Plane codebase was prompted to execute realistic workflows against the seeded data, and each workflow manually reviewed and validated by the authors (details in Appendix C). We developed a test- ing harness, which embeds Planeâs OpenAPI specification into a vector store, and exposes them to the agent through two tools: adiscovertool for querying relevant end- points, and anexecutetool for making the corresponding requests. Overall (s overall ) Structural (s struct )Content (s content ) 0.0 0.2 0.4 0.6 0.8 1.0 Score Bars: mean across completed trials. Whiskers: bootstrap 95% CI of the mean. Trials per model gpt-4.1: n=40, gemini-2.5-pro: n=40, gemini-3.1-flash-lite-preview: n=40, gpt-5: n=40, gemini-3.1-pro-preview: n=40 Session-level score by dimension gpt-4.1gemini-2.5-progemini-3.1-flash-lite-previewgpt-5gemini-3.1-pro-preview Figure 2. Score breakdown by dimension per model, for runs 1 and 2. Since CoW sessions are isolated, each GT and agent session started with the same application state. For each GT session, an agent was given the associated prompt and access to the Plane API via the harnessexecuteanddiscovertools, with a 50-operation limit per session. Five language models were evaluated twice per prompt: GPT-5, GPT-4.1, Gemini- 3.1-Pro, Gemini-3.1-Flash-Lite, and Gemini-2.5-Pro. In total,5Ă 2Ă 20 = 200agent sessions were scored against their corresponding GT sessions using CoW Scoring. A third trial per model was run after updating the tool surface (Section 4.3), for a total of 300 sessions. 4. Preliminary Results 4.1. Initial scores across models Figure 2 shows CoW scores along each dimension, averaged between the two runs per (model, workflow) pair, with per- model summaries in Appendix D (Tables 5 and 6). These are early results from a single application and 20 workflows, with two trials per (model, workflow) pair, so broader claims would benefit from more samples. Two initial observations serve as a sanity check on the scoring functions: within each model family, the ordering matches public benchmarks (Gemini-3.1-Pro>Gemini- 3.1-Flash-Lite, GPT-5>GPT-4.1), and per-model overall scores differ by at most 0.03 between trials, suggesting the means are representative rather than single-run artefacts. 4.2. Diagnosing failure modes Vocabulary mismatch. Plane uses âwork itemâ to de- scribe tasks within projects, while the session prompts use the term âissueâ, which is also commonly used to describe these items (the database tables still use âissuesâ, suggesting an artefact of a previous naming convention), so an agent that searched for âissueâ would not immediately find rele- vant operations. Gemini-2.5-Pro repeatedly gave up rather 3 Copy-on-Write Scoring: Application-Specific Agent Evaluations Table 1. Run 3 vs. runs 1 & 2 mean overall score. Run 3 used an updated tool surface based off the analysis of runs 1 and 2. ModelAvg(1,2)Run 3 nâ GPT-4.10.320.7920+0.47 Gemini 2.5 Pro0.430.9820+0.54 Gemini 3.1 Flash-Lite0.800.8320+0.03 GPT-50.950.9720+0.02 Gemini 3.1 Pro0.960.9820+0.01 than rephrasing or reading endpoint descriptions closely enough to recognise the equivalence (Appendix E, Figure 6). GPT-4.1 executed severaldiscovercalls before finding searchworkitems. It executed one search using an incorrect search string, then wrongly concluded no relevant issues existed when several did (Appendix E, Figure 7). It took Gemini-3.1-Flash-Lite fifteendiscovercalls to find the correct operation, causing it to ultimately exceed the 50-operation session limit (Appendix E, Figure 8). Importantly, this vocabulary mismatch challenge is well- known in information-retrieval (Furnas et al., 1987) and not specific to Plane: any application is likely to have multiple terms in circulation for the same concept. Hallucination and extra writes. GPT-4.1 occasionally hallucinated incorrect parameters and operation names, by guessing operations before first calling thediscovertool to ensure accuracy (Appendix E, Figures 9, 10, 11). In these sessions, the hallucinations were benign: the invalid oper- ations failed and left application state unchanged, but the same failure mode could be damaging if the agent invoked a valid operation with an incorrect name or parameters, leading to unintended side effects. Gemini-3.1-Flash-Lite hallucinated extra writes, producing label and assignee up- dates outside the scope of the prompted task (e.g., 47 extra rows in Campaign Push, 44 in Q3 Feature Prep). This is especially easy to miss outside a CoW session: an agent can satisfy the prompt while modifying unrelated application state in incorrect or harmful ways. 4.3. Improving the tool surface We updated the tool surface to address the findings, in partic- ular to ensure âissuesâ was also used to describe operations related to âwork itemsâ. We ran one additional trial per model; score deltas are reported in Table 1. Models most affected by the vocabulary mismatch saw the largest gains: Gemini-2.5-Pro improved by 54%, and GPT-4.1 by 47%. While Gemini-3.1-Flash-Lite improved by just 3%, it had no failed sessions in run 3 compared to at least one per iteration previously. This loop â CoW Scoring, localising the points of failure, and iterating or flagging known weak areas to users â can be applied to analyze any tool surface using CoW Scoring. 5. Limitations GT sessions are created manually.This step takes time, but we believe the effort is comparable to building any application-specific dataset â some investment is needed to address the construct-validity gap that motivates this work. Evaluation assumes prompts that fully specify the in- tended outcome. Real users often send underspecified prompts where multiple final states are reasonable, which CoW Scoring would conflate with failure â custom com- parators, vocabulary variation, and analyzing structural and content scores separately could partially mitigate this. Scoring covers only write operations.Agents also issue reads, and the volume and pattern of those reads carries diagnostic signal â failing models in our runs often issued many redundantdiscovercalls without affecting struc- tural or content scores, but these calls in fact surfaced the work-item/issue mismatch. Incorporating read-operation signals could help diagnose these failure modes directly. 6. Future Work Input to optimization loops.The low computational cost of CoW Scoring suggests it could also serve as a signal in op- timization loops â for example, to iteratively improve tool interfaces via GEPA (Agrawal et al., 2026), or as a dense reward for model fine-tuning via GRPO-style approaches (Shao et al., 2024). Operation-level scores are particularly well-suited to this: they assign credit to partial progress to- wards the final world state, so optimization need not depend on the discovery of complete, successful trajectories. Automated GT generation and coverage. GT sessions are manually generated, which is inherently limited in scale and may not give uniform coverage of the API surface. Au- tomated discovery methods â e.g. MCTS-style (Kocsis & Szepesv Ě ari, 2006; Zhou et al., 2024a) exploration with LLM-guided candidate selection â could generate candi- date sessions, with human review reserved for auditing. Impact Statement CoW Scoring aims to support safer agent deployments by making evaluations possible directly in application envi- ronments. The framework risks giving false confidence if GT sessions are unrepresentative of real usage: strong per- formance on a narrow set of evaluated workflows does not guarantee correct behaviour on others. Though not the focus of this paper, the CoW mechanism can also be used as a general runtime safeguard against unintended agent writes. 4 Copy-on-Write Scoring: Application-Specific Agent Evaluations References Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., Potts, C., Sen, K., Dimakis, A. G., Stoica, I., Klein, D., Zaharia, M., and Khattab, O. GEPA: Reflec- tive Prompt Evolution Can Outperform Reinforcement Learning, February 2026. Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., Mayne, H., Batzner, J., Foroutan, N., Schmitz, C., Korgul, K., Batra, H., Deb, O., Beharry, E., Emde, C., Foster, T., Gausen, A., Grandury, M., Han, S., Hofmann, V., Ibrahim, L., Kim, H., Kirk, H. R., Lin, F., Liu, G. K.-M., Luettgau, L., Magomere, J., Rystrøm, J., Sotnikova, A., Yang, Y., Zhao, Y., Bibi, A., Bosselut, A., Clark, R., Cohan, A., Foerster, J., Gal, Y., Hale, S. A., Raji, I. D., Summerfield, C., Torr, P. H. S., Ududec, C., Rocher, L., and Mahdi, A. Measuring what Matters: Construct Validity in Large Language Model Benchmarks, November 2025. Deng, C., Zhao, Y., Tang, X., Gerstein, M., and Cohan, A.Investigating Data Contamination in Modern Benchmarks for Large Language Models. https://arxiv.org/abs/2311.09783v2, November 2023. Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Verme, M. D., Marty, T., Vazquez, D., Chapados, N., and Lacoste, A. WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? In Proceedings of the 41st International Conference on Machine Learning, p. 11642â11662. PMLR, July 2024. Furnas, G. W., Landauer, T. K., Gomez, L. M., and Dumais, S. T. The vocabulary problem in human-system commu- nication. Commun. ACM, 30(11):964â971, November 1987. ISSN 0001-0782. doi: 10.1145/32206.32212. Garcia-Molina, H., Ullman, J. D., and Widom, J. Database Systems: The Complete Book (2nd Edition). Prentice Hall Press, One Lake Street Upper Saddle River, NJ, United States, 2 edition, June 2008. ISBN 978-0-13-187325-4. Kocsis, L. and Szepesv Ě ari, C. Bandit Based Monte-Carlo Planning. In Hutchison, D., Kanade, T., Kittler, J., Klein- berg, J. M., Mattern, F., Mitchell, J. C., Naor, M., Nier- strasz, O., Pandu Rangan, C., Steffen, B., Sudan, M., Terzopoulos, D., Tygar, D., Vardi, M. Y., Weikum, G., F Ě urnkranz, J., Scheffer, T., and Spiliopoulou, M. (eds.), Machine Learning: ECML 2006, volume 4212, p. 282â 293. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006. ISBN 978-3-540-45375-8 978-3-540-46056-5. doi: 10.1007/1187184229. Liao, Q. V. and Xiao, Z. Rethinking Model Evaluation as Narrowing the Socio-Technical Gap, January 2025. Mohammadi, M., Li, Y., Lo, J., and Yip, W. Evaluation and Benchmarking of LLM Agents: A Survey. In Proceed- ings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, p. 6129â6139, August 2025. doi: 10.1145/3711896.3736570. Raji, D., Denton, E., Bender, E. M., Hanna, A., and Paullada, A. AI and the Everything in the Whole Wide World Benchmark. Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks, 1, December 2021. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models, April 2024. Silberschatz, A., Galvin, P. B., and Gagne, G. Operating System Concepts. Wiley, 10th edition, 2018. ISBN 978- 1-119-32091-3. Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., Yang, M., Lu, H. Y., Martin, A., Su, Z., Maben, L., Mehta, R., Chi, W., Jang, L., Xie, Y., Zhou, S., and Neubig, G. TheAgentCom- pany: Benchmarking LLM Agents on Consequential Real World Tasks. In Advances in Neural Information Process- ing Systems, 2025. doi: 10.48550/ARXIV.2412.14161. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. $Ď $- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024. Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., and Wang, Y.-X. Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models, June 2024a. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024b. 5 Copy-on-Write Scoring: Application-Specific Agent Evaluations A. Copy-on-Write (CoW) Mechanism This appendix expands on the CoW mechanism summarised in Section 2.1, detailing the four database-level components that together intercept and isolate agent writes without modifying production data: 1.Base tables: These tables contain the underlying (base) production data, and follow the schema definitions specified in the application. 2.Changes tables: Each base table has a corresponding changes table. Changes tables are structurally identical to base tables (with four additional columns:sessionid,operationid,cowupdatedat,cowdeleted) but only contain rows that were written to during the agent session. 3.Views (Garcia-Molina et al., 2008): A SQL view is a query stored under a name. A view can be queried in the same way as a table, although no physical table exists â querying a view instead executes the underlying query (the view definition) and returns its results, so it can be read from like a table. In the CoW mechanism, views are used to merge each base table with its corresponding changes table, so that reads during an agent session return rows from the base table, with any rows that the agent has modified in the current session replaced by their changes-table versions. 4.Triggers (Garcia-Molina et al., 2008):INSTEAD OFtriggers can be defined on views, and define how a given operation (in this case, a database write) should be carried out. For CoW, writes should be forwarded to the changes table. i. Create copies of the affected row(s); i. Apply the write to the row(s); i.Append thesessionid,operationid,cowupdatedat(the current time), andcowdeleted (True if the write is a DELETE operation); iv. Write the row(s) to the corresponding changes table. Notably, the handling of complex trigger configurations such as cascade triggers and PostgreSQL-specific triggers should be explored in future work to extend the real-world portability of the CoW mechanism. Enabling CoW on a database creates each of these components, and deploys the necessary trigger functions â following Algorithm 1 below. Algorithm 1 Enabling CoW on a database 1: for each table T in the schema do 2:Rename T â Tbase 3:Create T changes with T âs columns plus CoW metadata 4:Create view T merging Tbase and Tchanges 5:Attach INSTEAD OF triggers to view T 6: end for 7: Deploy CoW trigger functions to the database B. CoW Scoring Figure 3 provides a worked example of the scoring procedure described in Section 2.2, illustrating how the session- and operation-level scores are computed from a paired GT and agent session. On the left, rows are colour-coded as matched, missing, and extra â which is used to computes struct ands content . On the right, each agent operationo i is attributed a structural utilityu struct (o i )from the change in session-level structural score, and a content utilityu content (o i )from the similarity of the rows it wrote. B.1. agent-cow Scoring Library Theagent-cowscoring module (visualized in Figure 4) scores an agentâs CoW session against a ground-truth recording via the API entry point scorecowsessions. The pipeline has five stages: 6 Copy-on-Write Scoring: Application-Specific Agent Evaluations taskidstatusop taskschanges (GT) T-103doneop01 T-106todoop02 T-108todoop04 commentidtaskidop commentschanges (GT) C-201T-106op03 taskidstatusop taskschanges (agent) T-103doneop03 T-106todoop01 T-107todoop04 commentidtaskidop commentschanges (agent) C-201T-106op02 C-202T-107op05 Ground-truth session session start op01 UPDATE tasks T-103 op02 INSERT tasks T-106 op04 INSERT tasks T-108 op03 INSERT comments C-201 FK â T-106 Agent session session start op01 INSERT tasks T-106 u struct =+ 1 4 op03 UPDATE tasks T-103 u struct =+ 1 4 op04 INSERT tasks T-107 u struct =â 3 20 op02 INSERT comments C-201 u struct =+ 1 4 FK â T-106 op05 INSERT comments C-202 u struct =â 1 10 FK â T-107 Session-level â final-state rows compared row-by-rowOp-level â ops as graph nodes; arrows are FK dependencies Session: s struct =0.50 s content =1.00 Overall: s overall =0.71 precision =0.60 recall =0.75 F 1 =0.67 PROMPT âMark task T-103 as done. Create two new backlog tasks â one for the API refactor (T-106) and one for the test suite (T-108) â and leave a comment on the API refactor task.â LEGENDmatchedmissingextra Figure 3. Worked example of CoW scoring. Session-level scores (s struct ,s content ) are calculated by comparing all rows changed for a given GT and agent session pair (left); and operation-level scores (u struct (o i ),u content (o i )) are calculated using theâin session-level structural score, and the content similarity of the rows written by each operation in the agent session (right). 1. Extraction (extraction.py) â rows are read from * changestables, grouped byoperationid, and sorted bycowupdatedat. 2.Matching (matching.py) â both sides are reduced to one row per(table, pk)(last write wins). Each ground- truth entity greedily picks the best matching agent entity in the same table; a UUID mapping bridges the different primary keys created by each side. 3.Fieldcomparison(compare.py)âeachfieldiscomparedbySQLtype:textuses SequenceMatcher.ratio(), JSON is deep-equal, everything else is exact.PKs, FKs (remapped via the UUID mapping), timestamps, and configured ignoredfields are excluded from content scoring. 4. Per-op scoring (scorer.py) â for each agent operation in topological order the cumulative agent rows are re-scored against ground truth, recording the delta inopstructscores.opcontentscoresis computed independently from the final matching. 5.Reduce (scores.py) â registeredscorefnsreduce theScoringResultto scalar entries on result.scores. B.1.1. SCORING CUSTOMIZATION Comparators. The default content similarity uses one comparator per SQL column data type. To override comparison for a specific table, callers can supply either a row-level function â which receives the GT and agent rows as plain dicts (with FK UUIDs already remapped into GT space), returns a bool or float in[0, 1]â or aWriteComparator, which additionally exposes the rowâs CoW metadata, the tableâs column-type metadata, and the cross-session UUID mapping for full control. Additionally, passingcollapse=Truedrops entities the agent created and later deleted within the session before scoring, reflecting only the agentâs net intent. 7 Copy-on-Write Scoring: Application-Specific Agent Evaluations Ground-truth session CoW changes graph Agent session CoW changes graph INPUTS extraction pull rows + metadata matching UUID align; wasted ops compareâ WriteComparator contentscore structscore efficiency scorer per-op and session-level calculations sampleevaluators â default, precision, recall, f1 (ScoreFn protocol) agentcow.scoring ScoringResult session-level scores per-op utilities score-fn outputs OUTPUTS User extension points (â): plug in custom comparators via WriteComparator, or custom score functions via ScoreFn. Figure 4. Copy-on-Write (CoW) scoring library architecture. Score functions. Any callable mapping the raw scoring terms to a float can be registered as anoverallscore. The agent-cowlibrary includes sample scoring functions, each mapping aScoringResultto a single float, which can be passed via the scorefns kwarg, and their outputs are written to result.scores under the names below. Default overall score. An average of the two session-level signals: s overall = 0.5¡ s struct + 0.5¡ s content (5) Precision. Of the rows the agent wrote, the fraction that matched a GT entity: precision = N matched N matched + N extra (6) Recall. Of the rows GT session wrote, the fraction the agent reproduced: recall = N matched N matched + N missing (7) F1. Harmonic mean of precision and recall: F 1 = 2¡ precision¡ recall precision + recall (8) C. Plane Integration and Experimental Setup C.1. CoW Integration in Plane The full implementation of CoW into Plane can be found in the following GitHub repository: Plane (with CoW), which also includes the raw scoring results (results.zip). The breakdown of application-side additions is shown in Table 2. The far-right column indicates whether that portion of code could be ported to the harness if the user wanted to simplify their implementation. Notably, the recording API and CoW scoring metadata (session IDs, scores, API versions, prompts) were included in the Plane application for this experiment, but could be stored in the harness for a simpler, less-invasive implementation. Only the CoW mechanism (using the agent-cow-python library) and associated deployment commands are necessary. 8 Copy-on-Write Scoring: Application-Specific Agent Evaluations CategoryLinesContentsPortable? (a) Core CoW machinery Ě250Middleware that activates a CoW session by reading the sessionid and operation id headers on every agent request. Endpoints to commit or discard a sessionâs writes, check whether CoW is enabled, and return the operation IDs of a session (the endpoint the harness queries to retrieve session state for scoring). Plane-specific list of tables to exclude from the CoW mechanism. No (b) Deployment com- mands Ě180 One-time installation of the SQL functions, views, and triggers described in Section 2 into the application database. Commands that enable, disable, and re-enable CoW. No (c) Recording API â GT & agent session tracking Ě680CRUD endpoints for GT recordings and agent runs (their prompts, model metadata, and links between the two). Plane-specific scoring configuration â which tables and fields to ignore when comparing sessions â together with helpers for writing scoring results to CSV / JSONL. Yes Total Ě790 Table 2. Application-side additions required to integrateagent-cowinto Plane. Category (a) is the minimum any Django project would need to replicate; categories (b) and (c) are specific to our evaluation setup and could move into the eval harness. C.2. Workspace Initial Data Seeding The Plane workspace was seeded with the entities summarised in Table 3 prior to any agent runs. Ground-truth CoW recordings were then captured against this fixed initial state. EntityDescriptionPSPPEGSTotal ProjectsTop-level Plane projects covering distinct organisational domains.â3 MembersWorkspace users (one admin agent plus six personas with role-aligned profiles).â7 StatesPer-project workflow states (Backlog, Todo, In Progress, Done, Cancelled). 55515 LabelsPer-project tag taxonomy (e.g. bug, security, urgent).108826 Work itemsSeeded issues with assignees, priorities, labels, and descriptions.32161563 Table 3. Entities seeded into the Plane workspace before any agent runs. Project-level resources are reported per project (PSP = Paper Streets Press, PE = Platform Engineering, GS = Growth and Subscriptions); workspace-level resources only have a workspace total. C.3. Ground-Truth Session Setup To produce a realistic and diverse set of ground-truth (GT) workflows without spending excessive author effort, an LLM agent (Claude Opus 4.6) under human supervision was used. The agent was given read access to theplane-cowcodebase â including the Plane data model, the seeded workspace dump (Appendix C.2), and the CoW recording API â so it could ground its workflows in the actual entities present in the codebase and database. We provided the agent with short workflow themes (e.g. sprint kickoff, security incident response, stale backlog cleanup) drawn from the kinds of bulk operations someone might be expected to perform in a project management tool. The agent then started a CoW recording, and executed the corresponding sequence of API writes against the seeded workspace, and drafted a natural-language prompt expressing the workflow. Each candidate session was then reviewed. We checked that (i) the prompt was unambiguous and self-contained, (i) every recorded operation was justified by the prompt, (i) no operation was missing relative to a literal reading of the prompt. 9 Copy-on-Write Scoring: Application-Specific Agent Evaluations 567891011121314151617181920212223242526272829303132 Operations per recording 0 1 2 3 4 Number of recordings Figure 5. Distribution of write operations per ground-truth (GT) session across the 20 workflows. In a real application, the product team would likely have some key workflows they would be interested in having the agent execute and perform well on, which would then be recorded manually. Table 4. Ground-truth CoW recordings used in evaluation. Ops is the number of mutating API calls captured in the recording. Projects: PSP = Paper Streets Press, PE = Platform Engineering, GS = Growth and Subscriptions. #Recording nameProjectOpsPrompt 1Sprint KickoffPSP12In project âPaper Streets Pressâ, move every issue in state âTodoâ to state âIn Progressâ. For each that is currently unassigned, assign @alice (user id: 2d02e7e3-e5be-4687-8fbf-32a133b3b271). For each that is currently assigned to @blub (user id: fad82a69-23ed-4260-83f6-cae2caf466a5), reassign to @bob (user id: 617f222e-9879-4436-a1e6-11654e24cffb). 2 Security Incident ResponsePE13In project âPlatform Engineeringâ, find every issue with label âsecurityâ that is in state âBacklogâ or âTodoâ. Move each to state âIn Progressâ. Add label âblockerâ if it does not already have it. Set priority to âurgentâ if not already urgent. Assign @dave (user id: ebdd01a5-b565-4b36-ba28-11f11ecf5dbd) if currently unassigned. 3Bug EscalationGS15In project âGrowth and Subscriptionsâ, find every issue with label âbugâ currently in state âBacklogâ. Move each to state âTodoâ. Add label âblockerâ to each. Assign @frank (user id: 806c8c73-23e7-42f9-9ed4-b9e3cb8c7bfd) to each. 4Stale Backlog CleanupPSP10In project âPaper Streets Pressâ, find every issue in state âBacklogâ that is unassigned and has priority âlowâ or ânoneâ. Add label âstaleâ to each and transition it to state âCancelledâ. 5Triage Unassigned Backlog GS11In project âGrowth and Subscriptionsâ, find every issue in state âBacklogâ with no assignee. Assign @eve (45a64208-112f-4bd4-a284-481cab326106) to issues with priority âhighâ or âurgentâ. Assign @frank (806c8c73-23e7-42f9-9ed4-b9e3cb8c7bfd) to issues with priority âmediumâ. For issues with priority âlowâ or ânoneâ, set priority to âmediumâ and assign @frank. 6Stale Backlog CancelPE10In project âPlatform Engineeringâ, find every issue in state âBacklogâ with no assignee, priority âlowâ or âmediumâ, and without label âsecurityâ. Add label âstaleâ to each and transition it to state âCancelledâ. 7Assign Backlog by Specialty PE14In project âPlatform Engineeringâ, find every issue in state âBacklogâ with no assignee. Assign @dave (ebdd01a5) to issues with label âsecurityâ. Assign @bob (617f222e) to issues with label âbugâ. Assign @carol (71b681c) to all other unassigned Backlog issues. For each newly assigned issue with priority âmediumâ or lower, also set priority to âhighâ. continued on next page 10 Copy-on-Write Scoring: Application-Specific Agent Evaluations Table 4 â continued from previous page #Recording nameProjectOpsPrompt 8Deprioritize Stale Medium Backlog PSP10In project âPaper Streets Pressâ, find every issue in state âBacklogâ with priority âmediumâ and no assignee. Set each to priority âlowâ and add label âstaleâ. 9Escalate High to UrgentPE11In project âPlatform Engineeringâ, find every issue with priority âhighâ in any open state (Backlog, Todo, In Progress). Set each to priority âurgentâ. For those in state âBacklogâ, also add label âurgentâ. 10Assign Backlog by RolePSP15 In project âPaper Streets Pressâ, find every issue in state âBacklogâ with no assignee. Assign @alice (2d02e7e3) to issues with label âbugâ. Assign @carol (71b681c) to issues with label âsecurityâ. Assign @bob (617f222e) to all other unassigned Backlog issues. 11Q3 Feature PrepGS24In project âGrowth and Subscriptionsâ, create a label âq3-target-v2â with color #8b5cf6. Find every issue with label âgrowthâ or âexperimentâ that is not in state Done or Cancelled. Add the âq3-target-v2â label to each (preserving existing labels). Move each from Backlog to Todo if currently in Backlog. Assign @eve (45a64208-112f-4bd4-a284-481cab326106) to any that are unassigned. Set priority to âhighâ for any with priority none, low, or medium. 12Infra Sprint KickoffPE23 In project âPlatform Engineeringâ, create a label âinfra-sprint-v2â with color #0284c7. Find every issue with label âinfraâ that is not in state Done or Cancelled. Add the âinfra-sprint-v2â label to each (preserving existing labels). Move each from Backlog to Todo if currently in Backlog. Assign @dave (ebdd01a5-b565-4b36-ba28-11f11ecf5dbd) to any that are unassigned. Set priority to âhighâ for any with priority none, low, or medium. 13Bug Triage Round 2GS16In project âGrowth and Subscriptionsâ, create a label âbug-sprint-v2â with color #dc2626. Find every issue with label âbugâ that is not in state Done or Cancelled. Add the âbug-sprint-v2â label to each (preserving existing labels). Also add the âblockerâ label if not already present. Set priority to âhighâ for any with priority none, low, or medium. Assign @frank (806c8c73-23e7-42f9-9ed4-b9e3cb8c7bfd) to any that are unassigned. 14Campaign PushGS5In project âGrowth and Subscriptionsâ, create a label âcampaign-active-v2â with color #db2777. Find every issue with label âcampaignâ that is not in state Done or Cancelled. Add the âcampaign-active-v2â label to each (preserving existing labels). Move each from Backlog to Todo if currently in Backlog. For issues with priority âhighâ, set priority to âurgentâ. Assign @eve (45a64208-112f-4bd4-a284-481cab326106) to any that are unassigned. 15In Progress Ownership Audit PSP10In project âPaper Streets Pressâ, create a label âactive-sprint-v2â with color #2563eb. Find every issue currently in state âIn Progressâ. Add the âactive-sprint-v2â label to each (preserving existing labels, but removing the âstaleâ label if present since in-progress work is no longer stale). Assign @alice (2d02e7e3-e5be-4687-8fbf-32a133b3b271) to any that are unassigned. Set priority to âhighâ for any with priority none, low, or medium. 16Editorial SprintPSP13In project âPaper Streets Pressâ, create a label âeditorial-sprint-v2â with color #f59e0b. Find every issue with label âeditorialâ that is not in state Done or Cancelled. Add the âeditorial-sprint-v2â label to each (preserving existing labels). Move each from Backlog to Todo if currently in Backlog. Assign @alice (2d02e7e3-e5be-4687-8fbf-32a133b3b271) to any that are unassigned. Set priority to âhighâ for any with priority none, low, or medium. 17 Performance Backlog PushPE13In project âPlatform Engineeringâ, create a label âperf-sprint-v2â with color #7c3aed. Find every issue with label âperformanceâ that is in state Backlog or Todo. Add the âperf-sprint-v2â label to each (preserving existing labels). Set priority to âhighâ for any with priority none, low, or medium. Assign @carol (71b681c-bace-4f8-834c-9b169cd6bfc9) to any that are unassigned. Move each from Backlog to Todo if currently in Backlog. continued on next page 11 Copy-on-Write Scoring: Application-Specific Agent Evaluations Table 5. Per-model summary across trials per model. Overall (s overall ) score as defined in Appendix B.1.1. Precision, recall, and F1 are computed from row-level matched/missing/extra counts. ModelCompletionOverall (s overall )PrecisionRecallF1 AverageRun 1Run 2 gpt-4.140/400.320.310.330.920.160.28 gemini-2.5-pro40/400.430.450.420.980.210.35 gemini-3.1-flash-lite38/400.800.830.770.780.830.81 gpt-540/400.950.970.920.980.980.98 gemini-3.1-pro40/400.960.960.970.961.000.98 Table 4 â continued from previous page #Recording nameProjectOpsPrompt 18Bug Fix SprintPSP21In project âPaper Streets Pressâ, create a label âbug-fix-sprint-v2â with color #b91c1c. Find every issue with label âbugâ that is not in state Done or Cancelled. Add the âbug-fix-sprint-v2â label to each (preserving existing labels). Set priority to âhighâ for any with priority none, low, or medium. Move each from Backlog to Todo if currently in Backlog. Assign @bob (617f222e-9879-4436-a1e6-11654e24cffb) to any that are unassigned. 19Security Hardening SprintPE14In project âPlatform Engineeringâ, create a label âsecurity-sprint-v2â with color #991b1b. Find every issue with label âsecurityâ that is not in state Done or Cancelled. Add the âsecurity-sprint-v2â label to each (preserving existing labels). Also ensure the âblockerâ label is present on each. Set priority to âurgentâ for any not already urgent. Move each from Backlog to Todo if currently in Backlog. Assign @dave (ebdd01a5-b565-4b36-ba28-11f11ecf5dbd) to any that are unassigned. 20Backlog Ownership Assignment GS32In project âGrowth and Subscriptionsâ, create a label âneeds-ownerâ with color #d97706. Find every issue in state Backlog with no assignee. Add the âneeds-ownerâ label to each (preserving existing labels). Assign @eve (45a64208-112f-4bd4-a284-481cab326106) to those with priority âhighâ or âurgentâ. Assign @frank (806c8c73-23e7-42f9-9ed4-b9e3cb8c7bfd) to those with priority âmediumâ. For issues with priority ânoneâ or âlowâ, set priority to âmediumâ then assign @frank. Move each issue to Todo state. Total292 D. Results Summary This appendix collects per-model summaries from the preliminary study (Section 4). The raw scoring data for all 300 sessions is available atresults.zip. Table 5 reports the overall scores overall (Appendix B.1.1) for each model averaged across the first two trials, alongside per-trial scores and row-level precision, recall, and F1 derived from the matched/missing/extra counts. Table 6 lists the five lowest-scoring (model, workflow) pairs per model; these are the sessions inspected in Section 4 to surface the failure modes discussed in the main text and illustrated in Appendix E. GPT-4.1Gemini 2.5 ProGemini 3.1 Flash LiteGPT-5Gemini 3.1 Pro #Session%Session%Session%Session%Session% 1Bug Escalation0.0Sprint Kickoff0.0Stale Backlog Cleanup56.7Stale Backlog Cancel50.0Security Hardening Sprint87.5 2Stale Backlog Cleanup0.0Bug Escalation0.0Stale Backlog Cancel59.0Security Incident Response84.5Security Incident Response89.1 3Security Incident Response0.0Security Incident Response0.0Q3 Feature Prep63.0Deprioritize Stale Med. Backlog95.8Bug Triage Round 291.5 4Stale Backlog Cancel0.0Assign Backlog by Role0.0Bug Fix Sprint66.7Assign Backlog by Specialty95.8Bug Fix Sprint93.2 5Triage Unassigned Backlog0.0Escalate High to Urgent23.8Editorial Sprint69.6Q3 Feature Prep96.2Performance Backlog Push95.7 Table 6. Five lowest-scoring sessions per model (s overall averaged across two runs). 12 Copy-on-Write Scoring: Application-Specific Agent Evaluations E. Sample Sessions This appendix contains tool-call traces from representative low-scoring sessions identified in Table 6, which illustrate the failure modes discussed in Section 4: vocabulary mismatch between prompt terminology and the API surface (Figures 6, 7, and 8), and operations issued without a priordiscovercall to obtain the correct argument format (Figures 9, 10 and 11). Despite the overall improvement from runs 1 and 2 to run 3, two model-specific failure modes persisted across iterations: ⢠GPT-4.1: Occasionally executes an operation without first callingdiscoverto obtain the correct argument format, producing API errors (Appendix E, Figures 9 and 11) and one failed session in run 3. â˘Gemini-3.1-Flash-Lite: Continued to write extra rows. For example, in the sessions: Sprint Kickoff (8 extra work items and assignees), Deprioritize Stale Medium Backlog (5 extra labels), and Infra Sprint Kickoff (18 extra labels and assignees), none of which were prompted. Figure 6. Gemini-2.5-Pro tool calls for the Sprint Kickoff task. The model gives up after its firstdiscovercall returns no matching endpoints for âissuesâ. Figure 7. GPT-4.1 tool calls for the Security Incident Response task. The model issues severaldiscovercalls before finding searchworkitems, then queries with an unsupported search string format, receives an empty result, and concludes no relevant issues exist. 13 Copy-on-Write Scoring: Application-Specific Agent Evaluations Figure 8. Gemini-3.1-Flash-Lite tool calls for the Bug Triage Round 2 task. The model issues over fifteendiscovercalls before finding the correct endpoint, exhausting the 50-operation limit. Figure 9. GPT-4.1 tool calls for the Assign Backlog by Specialty task. The model executesupdateworkitemwithout first callingdiscover, passingidas an argument; the API rejects it since the valid parameter is pk. 14 Copy-on-Write Scoring: Application-Specific Agent Evaluations Figure 10. GPT-4.1 tool calls for the Security Incident Response task. The model executesupdateworkspaceworkitem without first callingdiscover, guessing anoperationid that does not exist in the API. Figure 11. GPT-4.1 tool calls for the In-Progress Ownership Audit task. The model callsupdateworkitemwith an extraslug field not accepted by the API, resulting in anextraforbidden validation error. 15