Paper deep dive
Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
Yifan Xiong, Jingyi Ge, Zhenpeng Chen, Yiling Lou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-based agentic systems are emerging as a new software paradigm. Modern agents are typically composed of backbone LLMs and a surrounding harness that serves as the operational software infrastructure for agent execution. As agent harnesses grow increasingly complex, agents suffer from diverse harness implementation bugs, raising substantial reliability concerns. In this work, we conduct the first empirical study to systematically investigate the test adequacy of harness in real-world agentic systems. Our analysis reveals that agent harness remains substantially undertested. In particular, LLM-dependent harness (LDH) code, despite its critical role in processing LLM outputs and governing agent behavior, receives limited testing attention, with less than half of its lines and branches covered by existing tests. Motivated by these findings, we further propose HarnessTester, the first harness-oriented test generation technique that incorporates explicit agent-harness contract support to construct contract-faithful test setups and extensively exercise LDH code. Our evaluation shows that HarnessTester substantially outperforms state-of-the-art general-purpose test generation techniques in achieving 75.95%/84.76% larger line/branch coverage gains and 69.89% larger mutation-score gains. Furthermore, HarnessTester detects 122 real-world harness bugs in widely-used agentic systems (e.g., OpenClaw), among which, 88 bugs are previously-unknown bugs and 69 bugs have been confirmed by agent developers. These results highlight the practical effectiveness of HarnessTester in improving test adequacy and assuring the reliability of real-world agentic systems.
Tags
Links
- Source: https://arxiv.org/abs/2610.04921v1
- Canonical: https://arxiv.org/abs/2610.04921v1
Trouble viewing inline? Open PDF directly â
Full Text
91,225 characters extracted from source content.
Expand or collapse full text
Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild YIFAN XIONG, University of Illinois Urbana-Champaign, USA JINGYI GE, University of California, Berkeley, USA ZHENPENG CHEN, Tsinghua University, China YILING LOU, University of Illinois Urbana-Champaign, USA LLM-based agentic systems are emerging as a new software paradigm. Modern agents are typically composed of backbone LLMs and a surrounding harness that serves as the operational software infrastructure for agent execution. As agent harnesses grow increasingly complex, agents suffer from diverse harness implementation bugs, raising substantial reliability concerns. In this work, we conduct the first empirical study to systematically investigate the test adequacy of harness in real-world agentic systems. Our analysis reveals that agent harness remains substantially undertested. In particular, LLM-dependent harness (LDH) code, despite its critical role in processing LLM outputs and governing agent behavior, receives limited testing attention, with less than half of its lines and branches covered by existing tests. Motivated by these findings, we further propose HarnessTester, the first harness-oriented test generation technique that incorporates explicit agent-harness contract support to construct contract-faithful test setups and extensively exercise LDH code. Our evaluation shows that HarnessTester substantially outperforms state-of-the-art general-purpose test generation techniques in achieving 75.95%/84.76% larger line/branch coverage gains and 69.89% larger mutation- score gains. Furthermore, HarnessTester detects 122 real-world harness bugs in widely-used agentic systems (e.g., OpenClaw), among which, 88 bugs are previously-unknown bugs and 69 bugs have been confirmed by agent developers. These results highlight the practical effectiveness of HarnessTester in improving test adequacy and assuring the reliability of real-world agentic systems. 1 Introduction Driven by recent advances in large language models (LLMs), LLM-based agentic systems (also called agents) are emerging as a new software paradigm and have been increasingly adopted across a wide range of domains [4,14,16,27,37,46,48,73]. For example, OpenClaw [49], a recent general-purpose agent, has become one of the fastest-growing open-source software systems in history, attracting 3.2 million monthly active users, more than 346,000 GitHub stars, and over 500,000 active agent instances worldwide within just five months of its release [51, 67]. An agentic system is typically viewed as the composition of probabilistic backbone LLMs and a surrounding harness [33,35,45,60,86]. While LLMs provide reasoning capabilities, harnesses serve as the operational software infrastructure that wraps around the LLM to provide critical functionality spanning execution environment management, context and memory management, guardrail and governance, workflow orchestration, and tool interface. Harnesses in modern agentic systems often comprise a large codebase, e.g., the harness of OpenClaw contains over 196,000 lines of code [49]. Recent studies have shown that substantial improvements in agent performance can be achieved through optimizing harnesses alone while the backbone LLMs remain unchanged [33,89]. Consequently, there has been increasing effort in harness engineering [33,35,60,89], which focuses on building a more capable harness layer to unleash the capabilities of backbone LLMs. Despite the tremendous potential, agentic systems suffer from pervasive reliability issues [6,9,15, 43,70,78,90], leading to severe real-world consequences [1,17,18,20,81]. Particularly, as modern agentic systems have grown increasingly complex, recent studies reveal that agent applications and Authorsâ Contact Information: Yifan Xiong, University of Illinois Urbana-Champaign, Urbana, USA, yifanx14@illinois.edu; Jingyi Ge, University of California, Berkeley, Berkeley, USA, gjy2005@berkeley.edu; Zhenpeng Chen, Tsinghua University, Beijing, China, zpchen@tsinghua.edu.cn; Yiling Lou, University of Illinois Urbana-Champaign, Urbana, USA, yilingl@ illinois.edu. arXiv:2610.04921v1 [cs.SE] 4 Oct 2026 2Xiong et al. frameworks suffer from diverse harness implementation bugs [10,59,80,88,94]. These reliability concerns highlight an urgent need for systematic techniques to assure the agent reliability [55]. Testing, which examines whether a software system exhibits expected runtime behaviors under given inputs, has long been a cornerstone of software reliability assurance. In particular, test adequacy criteria, such as code coverage [93] and mutation scores [25], provide important indicators of test quality by measuring how thoroughly a test suite exercises the program under test and its ability to detect bugs. To date, test adequacy has been extensively studied and widely adopted in traditional software development [25, 77, 93]. Despite the growing adoption of agentic systems and their substantial reliability concerns, how adequately their harness implementations are tested by agent developers during development and maintenance remains largely unexplored. Although numerous benchmarks [26,40,70] have been proposed to evaluate agentic systems, they primarily measure task-level end-to-end success by treating agents as black boxes, without examining how thoroughly their internal harness implementations are exercised. This leaves a fundamental question about the testing practices of real- world agentic systems: How thoroughly do agent developers test the agent harness implementations during development and maintenance, and how effective are these tests at detecting harness bugs? Empirical Study. To fill this knowledge gap, we conduct the first empirical study to systematically analyze the test adequacy of agent harness in the wild. Our key insight is that LLM-dependent harness (LDH), the harness code with explicit data or control dependencies on LLM outputs, deserves particular testing attention, as it is responsible for correctly processing nondeterministic LLM outputs spanning a vast value space. Recent work [10] also shows that many agent failures occur during interactions between LLMs and harness. Therefore, beyond conventional test adequacy analysis over the entire agent codebase, we specifically investigate the test adequacy of LLM dependent harness code. To this end, we study the test adequacy of 10 large-scale and widely- used agentic systems (e.g., OpenClaw [50]) from diverse domains (e.g., coding, general-purpose applications, and scientific discovery), through the following research questions. RQ1: Overall Test Adequacy. Our quantitative analysis shows that real-world agent harness remains substantially undertested. Particularly, LLM-dependent harness code, despite its critical role in processing LLM outputs and governing agent behavior, receives limited testing attention: less than half of LDH lines and branches are covered, with an average mutation score of only 33.60%. Such a gap leaves reliability risks from LLM-harness interactions largely unexposed by current testing practice in agent systems. Moreover, we find that achieving high LDH test adequacy is challenging, as it requires contract-faithful test setups that satisfy the expected data structures, runtime states, and interaction protocols across harness components. RQ2: Semantic Test Adequacy. Beyond the aggregate coverage reported in RQ1, we characterize the semantics of LDH predicates and examine test coverage across semantic categories. Our manual analysis reveals diverse semantic patterns in what LDH predicates control, how they make decisions, and what forms of LLM-derived values they process. Predicates with explicit schemas, bounded value spaces, or directly controllable decisions (e.g., tool action and policy guards) are generally better covered, while larger gaps arise for those involving dynamic execution contexts or open- ended value spaces (e.g., agent orchestration). These results highlight the challenge of adequately testing LDH as the required test input spaces become implicit, dynamic, and difficult to enumerate. Harness-Oriented Test Generation Technique. Given the prevalent test inadequacy of agent harnesses revealed in our study, we further propose HarnessTester, an LLM-driven test generation technique specifically designed to extensively exercise LLM-dependent harness code in agentic systems. Inspired by our finding that reaching target harness requires test setups that satisfy interaction requirements across harness components, HarnessTester enables harness-oriented Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild3 test generation with explicit support for agent harness contracts, which specify the expected data structures, execution preconditions, interaction protocols across harness components. RQ3: Test Adequacy Improvement. Our evaluation results show that the tests generated by Har- nessTester achieve substantially higher coverage and mutation scores than all the four state- of-the-art LLM-based test generation techniques [3,44,58,92]. HarnessTester outperforms the strongest baseline selected for each agent and metric by achieving 75.95%/84.76% larger mean LDH line/branch coverage gains and a 69.89% larger mean mutation-score gain. Our ablation study further demonstrates the substantial contribution of agent-harness contract support in HarnessTester. RQ4: Real-World Harness Bug Detection. We further evaluate the test adequacy of tests generated by HarnessTester through their capability to detect real-world harness bugs. In RQ4.a, among 250 historical harness bugs, tests generated by HarnessTester detect substantially more bugs than the best baseline (43 versus 2). In RQ4.b, we further apply HarnessTester to the latest versions of the studied agents to discover previously unknown bugs. HarnessTester identifies 122 harness bugs, among them, 88 are previously unknown bugs and 69 have been confirmed by developers by the paper submission time. These results demonstrate the practical effectiveness of HarnessTester in uncovering real-world harness bugs and improving the reliability of agentic systems. In summary, this paper makes the following contributions: â˘Study. We conduct the first empirical study of test adequacy in real-world agent harnesses, with a novel focus on LLM-dependent harness code. Our study not only reveals prevalent test inadequacy in agent harness but also identifies key challenges in constructing high-adequacy tests for LDH. â˘Approach. We propose the first harness-oriented test generation technique, HarnessTester, which incorporates harness contract support to construct contract-faithful test execution contexts and exercise diverse LLM-dependent harness code. â˘Practical Impact. Our study reveals important test-adequacy gaps and challenges in current agent testing practices. Moreover, our extensive evaluation demonstrates the practical effectiveness of HarnessTester in improving agent reliability, which not only substantially improves the test adequacy of existing agent tests but also detects harness bugs across real-world agentic systems. 2 Background and Related Work Agentic Systems and Agent Harness. Modern agentic systems (e.g., OpenClaw [49]) are becoming increasingly complex and are typically viewed as the composition of probabilistic backbone LLMs and a surrounding harness [33,35,45,60,86]. The harness surrounding the backbone LLMs serves as the operational software infrastructure that provides critical functionality, including execution environment management, context and memory management, guardrails and governance, workflow orchestration, and tool interfaces. Meanwhile, growing research efforts have focused on harness engineering [33,35,60,89], aiming to develop systematic methodologies for optimizing the agent harness layer. In particular, proper harness design and implementation have been shown to substantially improve the capabilities of agentic systems even when the backbone LLMs remain unchanged [31,33,36,89]. These advances highlight the importance of harness quality, aligning well with our motivation to examine the test adequacy of agent harnesses. Testing for Agentic Systems. Mainstream work focuses on benchmarking agents on end-to-end task success rates [26,34,42,79,84,91]. While useful for comparing task-level capabilities, such evaluation paradigms primarily treat agents as black boxes, without examining how thoroughly their internal harness implementations are exercised. In this work, we provide a complementary perspective to existing agent benchmarks by analyzing the white-box test adequacy of harness. Recent studies have investigated testing practices in emerging agentic systems [22,29,38,56]. While these studies share a similar perspective with our work by studying existing developer tests in agent repositories, they mainly focus on characterizing testing patterns, such as test granularity 4Xiong et al. (e.g., unit- or system-level testing). In contrast, our work focuses on the test adequacy of agent harnesses, with a novel analysis perspective on critical LLM-dependent harness code. In addition, we further propose the first harness-oriented test generation technique, HarnessTester, which not only substantially improves harness test adequacy but also detects real-world harness bugs. LLM-based Test Generation. A growing body of work has explored LLM-based test genera- tion [3,11,32,64,65,75,82,87,92], demonstrating strong effectiveness in generating diverse and semantically meaningful tests. However, existing techniques are primarily designed for general soft- ware systems. As revealed by our study, generating high-adequacy tests for LDH requires contract- faithful test setups that satisfy interaction requirements across harness components. To address this challenge, we propose the first harness-oriented test generation technique HarnessTester that enhances LLM-based test generation with agent harness contract support. Our experiments further confirm the substantial improvements in test coverage and bug detection capabilities from HarnessTester compared to existing general-purpose LLM-based test generators [3,44,58,92]. 3 Empirical Study Setting This work performs the first study to investigate the test adequacy of harness in real-world agentic systems. This section presents the setup of our empirical study. 3.1 LLM-Dependent Harness Code In this work, we refer to the harness code that has data dependency or control dependency on the outputs of LLM invocations as LLM-dependent harness (LDH) code. We consider LLM-dependent harness code a critical testing target because it is responsible for correctly interpreting diverse and nondeterministic LLM outputs to govern subsequent agent execution. Errors in such code can therefore propagate through the agent workflow and lead to incorrect agent behavior. In fact, as revealed by recent work[10], prevalent agent bugs occur between LLM-harness interactions. Definition. Given an agentic programí´, letS LLM â Stmt(í´)denote the set of LLM invocation statements that produce model outputs, and letí â í âdenote that statementâis directly or transitively dependent on statement í . We define the LLM-dependent harness code of í´ as íżíˇíť(í´)= â âH(í´) | âí âS LLM . í â í â , whereH(í´) â Stmt(í´)denotes the set of harness statements. Intuitively,íżíˇíť(í´)captures harness code whose computation depends on values produced by LLM invocations. Extraction. Existing program analysis frameworks provide general machinery for interprocedural data-flow and control-flow analysis [5,62], but they do not directly capture the LLM-specific semantics required by LDH extraction. Particularly in agent harness, LLM outputs can originate from heterogeneous model providers and invocation abstractions, and subsequently flow through agent- specific representations, such as message objects, structured responses, and tool-call structures. Consequently, extracting LDH requires explicitly identifying LLM-output sources and modeling how output-derived values propagate through these abstractions. Therefore, to automate LDH extraction, we implement an interprocedural harness analysis tool that models LLM invocations as sources and propagates the corresponding data- and control dependency facts throughout the program until a fixed point is reached. We include all the explicit dependency through assignments, field or container accesses, parameter and return flows, and other taint-preserving transformations. Implementation details are in our artifact [21]. Listing 1 illustrates an example of LLM-dependent harness code. The code highlighted in red (e.g., Line 15 and Line 18) are the sources of LLM outputs returned from the LiteLLM interface and the provider provenance of the OpenAI client; and the code highlighted (e.g., Line 29) in green are identified as LLM-dependent harness code, to which the LLM outputs propagate. Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild5 Listing 1. Illustration example of LLM-dependent harness code. 1 import json 2 from pathlib import Path 3 from litellm import completion 4 from openai import OpenAI 5 from tools import search_tool 6 7 client = OpenAI(timeout=30) 8 TOOL_REGISTRY = "search": search_tool 9 10 def setup_session(path: Path): 11 ... 12 def teardown_session(path: Path): 13 ... 14 def invoke(messages): 15 returncompletion(model="gpt-5", messages= messages) 16 17 def direct_answer(prompt): 18 response =client.responses.create(input=prompt ) 19 returnresponse.output_text 23 def run_agent_turn(messages, session_path: Path): 24 created_here = False 25 if not session_path.exists(): 26 setup_session(session_path) 27 created_here = True 28 try: 29response =invoke(messages) 30message =response.choices[0].message 31 results = [] 32 forcall inmessage.tool_calls or []: 33tool = TOOL_REGISTRY[ call.function.name] 34args = json.loads( call.function.arguments) 35results.append(tool(session_path= session_path, **args)) 36 returnmessage.content,results 37 finally: 38 if created_here: 39 teardown_session(session_path) 3.2 Studied Agents We include a set of representative open-source agentic systems based on the following inclusion criteria. (1) Literature-backed candidate pool construction: we refer to prior agent research [24,39, 74,76,83] for their studied agentic systems; (2) Open-source quality filtering: we only include the agentic systems that have at least 10,000 GitHub stars as of the data collection date and at least one official release within the preceding 18 months, and an official release or stable revision provides an executable automated test suite that can be reproduced in our environment; (3) Agent diversity: we select agents to cover different domains (e.g., software engineering, auto-research, general purpose) and two programming languages (e.g., Python or TypeScript) widely used to build agents. Table 1 summarizes 10 studied agentic systems with their characteristics. 3.3 Test Adequacy Metrics In this study, we focus on the following test adequacy metrics. Coverage. We include the widely used coverage metrics, i.e., line and branch coverage [93], which calculate the ratio of lines and branches exercised by tests. In particular, we collect both (i) project-wide coverage, which focuses on the coverage over the entire codebase of the studied agent repository; (i) LDH-wide coverage, which focuses on the coverage over LLM-dependent harness code. We collect coverage with the widely-used coverage package [13] for Python, and the coverage providers integrated with Vitest [72] for TypeScript. Mutation Score. While coverage measures how extensively code is exercised, mutation testing assesses the fault detection capabilities of tests by injecting artificial faults into the program under test. A mutant is considered as killed by a test when the test passes on the original program but fails on the mutated program; and the mutation scores calculate the ratio of killed mutants to all the generated mutants. Therefore, to measure how sufficiently existing tests can detect the potential bugs in LLM-harness interaction, we further present the mutation scores over LDH. In particular, we apply the common mutation tools mutmut [23] and StrykerJS [69] to generate mutants on LLM-dependent harness code, and present the proportion of killed mutants. 4 RQ1: Overall Test Adequacy This RQ performs a large-scale quantitative analysis to investigate the overall test adequacy of the existing test suite in studied agentic systems. As shown in Table 2, Column âLDH Shareâ presents the 6Xiong et al. Table 1. Statistics of studied agentic systems. AgentLang. DomainStructure Version KLOC Tests Stars OpenHands [52]Py/TSIssue resolutionMulti-agent1.6.0194.82,64684.1K Aider [2]Python Interactive codingSingle-agent0.86.220.248948.2K SWE-agent [71]Python Issue resolutionSingle-agent1.1.012.79020.1K PR-Agent [57]Python Code reviewSingle-agent0.36.020.947812.6K GPT Researcher [19] Python Research synthesisMulti-agent3.5.030.74729.0K Browser Use [8]Python Browser automation Single-agent0.13.270.9869 109.4K RD-Agent [61]Python Data-driven R&DMulti-agent0.8.044.57514.2K OpenClaw [50]TS/JSGeneral-purposeMulti-agent2026.3.28851.02,352 386.4K Roo Code [63]TS/JSInteractive codingMulti-agent3.54.0161.754924.3K Kimi Code [28]TS/JSInteractive codingSingle-agent0.19.2166.359511.2K ratio of LDH lines/branches to the agent codebase, Column âLDH Coverageâ presents line/branch coverage in LLM-dependent harness code, Column âProject coverageâ presents the project-wide line/branch coverage, and Column âMutation Scoreâ presents the mutation scores over LDH. Table 2. Overall test adequacy. Agent LDH Share LDH Coverage Project Coverage Mutation Score Line Branch Line Branch Line Branch OpenHands16.76% 29.15% 54.91% 53.42% 66.65%49.93%41.66% Aider42.51% 57.57% 53.87% 48.93% 58.65%49.21%35.10% SWE-agent8.78%12.63% 44.81% 44.15% 58.51%40.86%24.11% PR-Agent37.72% 52.30% 27.09% 25.76% 37.88%27.58%16.91% GPT Researcher 32.43% 56.62% 10.67% 11.49% 25.75%10.29%4.47% Browser Use10.82% 16.79% 28.30% 25.48% 37.51%25.75%25.23% RD-Agent18.67% 27.93%5.39%6.73%20.82%3.61%3.58% OpenClaw23.38% 27.45% 56.73% 50.55% 59.74%51.29%53.18% Roo Code17.68% 20.45% 68.14% 62.18% 57.33%48.59%60.86% Kimi Code14.72% 16.18% 92.83% 83.09% 71.98%61.26%70.94% Average22.35% 31.71% 44.27% 41.18% 49.48% 36.84% 33.60% LDH Prevalence. As shown in Column âLDH shareâ, LDH represents a relatively compact portion (i.e., less than 45% of executable lines) of the agent codebase, while containing a comparatively high concentration of decisions. In particular, across all studied agents, the LDH branch share is 1.10-1.75 times its line share. One possible explanation is that LLM outputs are frequently consumed by decision-making logic, such as parsing and validating model responses, dispatching actions, and updating agent states. Meanwhile, it is also as expected that agent codebase contains substantial supporting code that does not directly depend on LLM outputs, e.g., provider initialization, model configuration, and execution environments management, which typically do not consume model- produced values. Overall, the LDH distribution confirms the necessity of further complementing project-wide test adequacy with an additional focus on LLM-dependent harness code. Limited Test Coverage. As shown in Column âLDH Coverageâ and âProject Coverageâ, most studied agents exhibit limited test coverage at both the project and LDH levels. Across the entire codebase, more than half of the executable lines and branches remain uncovered by existing tests on average. In addition, despite the importance of LDH, it does not consistently receive higher test coverage than the project as a whole. Specifically, LDH line coverage is lower than project-level line coverage in eight of the ten agents; for branch coverage, LDH achieves comparable or higher Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild7 coverage in six agents but lower coverage in the remaining four. In summary, these results show that existing agentic systems mostly remain undertested regarding the test coverage, with a limited testing attention drawn to the critical LLM-dependent harness code. Limited Fault Detection Effectiveness. As shown in Column âMutation Scoreâ, existing tests also show limited effectiveness in detecting injected mutants in LLM-dependent harness code. Mutation scores range from 3.58% for RD-Agent to 70.94% for Kimi Code, with an average of only 33.60%. It indicates that, on average, about two-thirds of the injected faults in LDH cannot be detected by any existing tests. These undetected mutants arise for two main reasons: some are not even exercised by any test (e.g., 38.36%) , while others (e.g., 25.92%) are exercised but survive because the tests fail to propagate or capture the behavioral changes introduced by the mutations. Overall, the results further reveal the limited capabilities of existing agent tests in detecting bugs occurring in LLM-dependent harness code, implying that existing testing practices cannot fully mitigate the reliability risks arising from LLMâharness interactions during agent execution. Finding 1: The harnesses of real-world agentic systems remain substantially undertested, with average line and branch coverage below 50% and 40% in agent codebase. More concerningly, LLM-dependent harness code, despite its high concentration of decision logic to process LLM outputs and govern agent behavior, also receives limited testing attention: existing tests only cover less than half of LDH lines and branches and exhibit limited capability to detect bugs in LDH, with only 33.60% mutation scores. The results reveal a substantial gap between the critical role of LLM-dependent harness code and the limited test adequacy it receives, indicating that current testing practice in real-world agentic systems cannot fully mitigate the reliability risks arising from LLMâharness interactions during agent execution. Agent-Wise Comparison. As shown in Table 2, test adequacy can vary substantially across the studied agents. For example, Kimi Code achieves relatively high project- and LDH-wide coverage (e.g., up to 92.83% LDH line coverage) and the highest mutation score (i.e., 70.94%) across all studied agents. We then perform a case analysis on the test implementation in Kimi Code. Although all studied agents, including Kimi Code, extensively use mocks to isolate LLMs and other external components, which is a consistent observation as prior work [56], simply using mocks does not necessarily make deeper harness behavior easy to reach. Model outputs often flow through multiple harness components before reaching deeper LDH logic, where each component may expect specific input structures, runtime states, and interaction sequences. Our inspection shows that Kimi Code provides systematic test support for satisfying these requirements. For example, reusable helpers construct mocked model responses with the required fields for different scenarios, such as turn completion and tool invocation. Fake LLMs and scripted generators reproduce valid interaction sequences, while local HTTP/SSE servers and environment fakes reproduce the runtime interactions required for provider communication and tool execution. Together, these utilities help tests construct valid execution contexts and vary model outputs, interactions, and runtime states to exercise different harness behaviors. These requirements resemble the classic notion of software contracts [7,41], which specify the obligations and guarantees governing interactions between software components. In the context of agentic systems, these requirements form agent-harness contracts, capturing how model-proposed actions map to tool interfaces, how returned observations correspond to the originating calls, and how subsequent decisions depend on state established through preceding interactions. Reusable test utilities in Kimi Code effectively encode many of these contracts, making it easier to construct both contract-faithful and diverse execution contexts to exercise different decisions. 8Xiong et al. In contrast, OpenClaw, which exhibits much lower test adequacy, provides a representative counterexample. Although its tests also contain extensive mocks and fixtures, they are organized in a relatively ad hoc manner, making it difficult to construct contract-faithful setups across diverse test scenarios. At the model-output boundary, parsers, dispatchers, and error handlers often contain branch-rich logic over response structures and content. While a single mocked response may exercise a nominal path, exploring different decisions requires diverse responses that still satisfy the expected provider-response contracts. Reaching deeper LDH logic can be even more challenging, as model-derived values may flow through multiple components, such as agent runners and gateways. Reaching a target path therefore requires tests to configure their callbacks, exchanged data, runtime states, and environments consistently. Without reusable support for constructing such setups, developers must repeatedly identify and assemble the required test context and its variants for different execution paths. Compared with Kimi Code, OpenClaw may therefore face greater difficulty in constructing test setups that are both contract-faithful and sufficiently diverse. Taken together, these cases suggest that adequate LDH testing depends on test-side setups that are both sufficiently diverse and faithful to the relevant agent harness contracts. Constructing such tests, however, can be challenging in practice: agent developers are supposed to identify the component boundaries traversed by model-derived values, understand the data structures and interaction protocols expected at those boundaries, and construct mocks that collectively satisfy these contracts while supporting the diverse required behavioral variations. Finding 2: Achieving high test adequacy on LLM-dependent harness code calls for contract- faithful test setups that satisfy the expected data structures, runtime states, and interaction protocols across harness components. Reusable test utilities can facilitate the construction of such valid execution contexts, while supporting the exploration of diverse inputs. 5 RQ2: Semantic Test Adequacy The quantitative results in RQ1 reveal overall test insufficiency in agent codebase and LLM- dependent harness code. Meanwhile, aggregate metrics alone cannot reveal whether these testing gaps are uniformly distributed in particular types of LLM-dependent harness logic. This distinction is important, as LDH code can contain rich semantics to serve different responsibilities in agent workflows, which may present different testing challenges. In this RQ, we therefore characterize the semantics of LDH branch predicates and examine test coverage across different semantic categories to understand which types of LLM-dependent harness logic are adequately covered and where important testing gaps remain. We focus on branch predicates as they often represent critical decision points that govern agent behavior and encode rich semantics. To this end, we first randomly sample 700 LLM-dependent harness branch predicates from all studied agents. Specifically, we analyze each predicate from three perspectives: (i) component intention, which refers to the responsibility that the predicate fulfills within the agent workflow; (i) judgment pattern, which refers to the decision rule encoded by the predicate when evaluating LLM-derived values; and (i) LLM value form, which refers to the runtime representation of LLM- derived values. As there is no existing taxonomy on semantics of LDH branch predicates, we develop our taxonomy through an open-coding process [66,68]. Two annotators, each with five years of software development experience and substantial experience with agents, first perform a pilot study on another small set of predicates and assign descriptive codes without relying on predefined categories. They then discuss and consolidate the resulting codes, iteratively refining the categories and their definitions until reaching a stable codebook. Using the finalized codebook, the two annotators independently label all the sampled predicates. Cohenâsí on these independent Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild9 Table 3. Taxonomy and distribution of semantic patterns in LDH branch predicates. Component Intention CategoryShare Provider I/O Normalization29.14% Tool Action Handling23.57% Context Memory Management13.57% Agent Orchestration Control14.00% Observation Feedback Handling 12.43% Error Recovery Handling7.29% Judgment Pattern CategoryShare Presence or Shape Check47.14% Value or Pattern Matching 33.29% Size or Threshold Check15.29% Policy Guard4.29% LLM Value Form CategoryShare Generated Content25.00% Provider Response Signal6.57% Tool Action Payload21.71% Agent Context State21.71% Observation Feedback Signal 25.00% annotations before resolving disagreements is 0.96, indicating high inter-annotator agreement [12]. Remaining disagreements are resolved through discussion with a third reviewer. Taxonomy of Semantic Patterns in LDH Branch Predicates. Table 3 presents the taxonomy of semantic patterns in LDH branch predicates along with their distribution across the sample. Given the space limits, the detailed explanation and example of each category are in our artifact [21]. Component Intention. We observe six main categories of component intention across the inspected LDH branch predicates. Among them, provider I/O normalization and tool action handling account for 29.14% and 23.57% of LDH branch predicates, respectively. Together, these two categories comprise more than half of the inspected predicates, showing that LDH branch predicates frequently govern the processing of provider responses and the selection, validation, and gating of model- proposed actions. This concentration highlights the important role of LDH branch predicates in mediating the transition from model-produced information to downstream agent execution. Judgment Pattern. Almost half of LDH branch predicates perform presence or shape checks (e.g., 47.14%), which mainly check the structures of LLM-derived values (e.g., expected type, fields, schema, or container structure). Other recurring patterns enumerate the common properties, such as concrete values, sizes, content patterns, or policies. Overall, LLM-dependent harness branch predicates primarily determine (i) whether model-derived information is available in the expected structure, and (i) which execution path to follow based on its value. Testing LDH therefore requires systematically varying both the structure and values of model-derived inputs to exercise distinct branch outcomes, rather than randomly constructing plausible model responses. LLM Value Form. We observe five forms of LLM-derived values. In addition to immediate model outputs, such as generated content and provider response signals, accounting for 25.00% and 6.57% of LDH branch predicates, LDH branch predicates also process (i) indirect LLM-derived values accumulated throughout agent execution, such as agent context state (21.71%), and (i) heterogeneous runtime values that depend on model outputs, such as tool action payloads (21.71%) and observation feedback signals (25.00%). These results highlight the challenge of LDH testing, as tests need to reproduce not only immediate model outputs but also heterogeneous values and states accumulated throughout long-horizon agent execution. Finding 3: LLM-dependent harness branch predicates exhibit substantial semantic diversity in what they control, how they make decisions, and what forms of LLM-derived values they process. They operate not only on direct model outputs but also on heterogeneous information accumulated throughout agent execution. Adequate LDH testing is therefore challenging: tests need to reproduce different forms of LLM-derived values throughout agent execution and systematically vary their structures and properties, while ensuring that these variations satisfy the relevant agent harness contracts. 10Xiong et al. Test Coverage Across Semantic Patterns. We further examine whether existing tests cover different semantic categories of LDH branch predicates equally well or leave gaps in particular categories. We classify each LDH branch predicate as fully covered, partially covered, or uncovered, depending on whether all, some but not all, or none of its decision outcomes are exercised. We report the coverage by category in Figure 1. 0% 50% 100% Share 24 18 14 38 18 31 22 30 19 50 18 28 43 22 21 28 25 22 23 23 23 25 21 33 13 22 33 20 30 24 49 57 63 39 59 46 52 48 49 37 60 39 36 49 55 Provider I/O Error recovery Agent orchestr. Tool action Obs. feedback Context memory Presence shape Value pattern Size threshold Policy guard Gen. content Provider signal Tool action Agent context Obs. feedback Component Intention Judgment Pattern LLM Value Form Fully coveredPartially coveredUncovered Fig. 1. Test coverage across LDH branch predicates. Overall, among previously sampled 700 LDH branch predicates, only 25.57% are fully covered, 24.57% are partially covered, and 49.86% remain entirely uncovered. This pattern separates two forms of incompleteness: many LLM-dependent predicate decisions are never reached, while many others are reached without all outcomes of the encoded decision boundary being exercised. Coverage Across Component Intention. A clear coverage contrast emerges across different compo- nent intentions. Tool action handling achieves the highest full-coverage rate at 38.18% (63/165), with 61.21% (101/165) of its branch predicates having at least one outcome exercised. In contrast, agent orchestration control predicates exhibit lowest coverage, only 14.29% (14/98) of predicates are fully covered, while 63.27% (62/98) remain uncovered. Observation feedback and error recovery handling show similarly low full-coverage rates of 18.39% (16/87) and 17.65% (9/51), respectively. One possible explanation is that tool actions typically follow explicit schemas, making it easier to construct valid payloads and systematically vary them. In contrast, orchestration, feedback, and error recovery often depend on dynamically evolving execution contexts, including prior interac- tions, accumulated agent states, and cross-component behavior. Reproducing and varying such contexts while satisfying the relevant agent-harness contracts can therefore be more challenging. Coverage Across Judgment Patterns. Coverage also varies substantially across judgment patterns. Policy guards achieve the highest full-coverage rate at 50.00% (15/30). This may be because policies are often implemented as functions or exposed through interfaces, thus providing a clear and controllable decision space: tests can directly mock the corresponding policy interfaces to return different decisions, making alternative branch outcomes relatively easy to exercise. In contrast, such direct guidance is not always available for other judgment patterns. Only 22.12% of presence or shape checks, 18.69% of size or threshold checks, and 30.47% of value or pattern matching predicates are fully covered, with more than 48% of each category remaining uncovered. For these patterns, the values inspected by a predicate may be produced through preceding parsing, or data transformations, making the required inputs less apparent. Constructing tests for different outcomes may therefore require reasoning backward from the target predicate to establish appropriate upstream inputs and states while satisfying the relevant agent-harness contracts. Coverage Across Value Forms. Branch predicates over tool action payloads are the most thoroughly exercised: 43.42% are fully covered, and 63.82% have at least one exercised outcome. Full-coverage rates are substantially lower for generated content (17.71%), observation feedback signals (20.57%), and agent context state (21.71%). Moreover, 60.00% of generated-content predicates and 55.43% of observation-feedback predicates remain uncovered. One possible explanation is that tool action payloads typically have a relatively bounded and explicit value space: available actions and their argument schemas provide clear guidance for constructing valid inputs and systematically varying Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild11 them. In contrast, generated content has a more open-ended value space, making it difficult to enumerate the content and structural variations needed to exercise downstream logic. Agent context and observation feedback can be even more challenging, as their values evolve throughout agent execution based on prior interactions, execution results, and runtime states. Their relevant value spaces are therefore difficult to enumerate in advance, requiring tests to reproduce and vary values along the execution flow while preserving the expected runtime context. Finding 4: Test coverage varies substantially across different semantic categories of LLM- dependent harness branch predicates. Across three semantic dimensions, these predicates are better covered when test inputs and behaviors are explicitly defined and easy to control. For example, tool-action logic benefits from explicit action schemas and bounded value spaces, while policy guards can often be exercised by directly mocking policy decisions. In contrast, larger coverage gaps arise when testing requires reconstructing dynamically evolving execution contexts or exploring open-ended value spaces. In summary, existing tests are less effective at exercising LDH behavior as the required test context and input space become more implicit, dynamic, and difficult to enumerate. 6 HarnessTester: Harness-Oriented Test Generation Our study above reveals prevalent test inadequacy in LLM-dependent harness across real-world agentic systems, motivating the need for effective harness-oriented test generation techniques. However, as shown by our findings, extensively exercising LLM-dependent harness code is chal- lenging: reaching target LDH requires not only test setups that satisfy the interaction contracts across harness components, but also diverse test inputs that represent different model-derived values and execution states within these valid setups. Motivated by our findings, we propose HarnessTester, an LLM-driven test generation technique specifically designed to extensively exercise LLM-dependent harness code in agentic systems. Figure 2 presents the overall workflow of HarnessTester. Following the coverage-driven test generation paradigm, given an agent codebase and its existing test suite, HarnessTester iteratively identifies LDH code that is uncovered by existing tests as targets and generates new tests to cover the target. Different from existing LLM-driven test generation techniques designed for general software, the key insight of HarnessTester is to make harness-oriented test generation explicitly aware of agent harness contracts. As discussed in Section 4, agent harness contracts refer to the requirements that must hold across interactions among harness components for valid execution, such as expected data structures, execution preconditions, and interaction protocols. To this end, HarnessTester augments general LLM-based test generation with agent harness contract support, which helps LLM construct contract-faithful test contexts via prompt-level guidance (i.e., agent-harness contract guidance) and tool-level enhancement (i.e., agent-harness contract context retriever). Agent-Harness Contract Guidance. The guidance consists of a set of high-level, contract-aware principles embedded in the prompts used for iterative test generation and refinement. Its main goal is to guide the model to preserve key agent-harness contracts when constructing tests, in- cluding maintaining the expected runtime representations of provider responses, tool actions, and observations; constructing the agent state or interaction history required to reach the target behav- ior; respecting cross-component invocation contracts, such as synchronous/asynchronous calling conventions and expected return shapes; and installing contract-faithful test doubles at the actual bindings or injection points used by the code under test. These principles are derived from testing challenges identified in our study above. Beyond contract fidelity, the guidance also promotes 12Xiong et al. Agent Codebase Test Pool Generated Test Ta r g e t Code Generation Prompt Coverage Pool Test Generation and Refinement Test Validation and Update Agent-Harness Contract Support Test Output Harness Contract Guidance Harness Contract Context Retriever LLM Request Resolving AST Matching invoke Contract- Faithful Test Doubles Cross- Component Invocation Runtime Representation Fig. 2. Overview of HarnessTester. diversity by encouraging LLM to prioritize distinct behavioral partitions rather than generating additional variants of already exercised behavior. Detailed prompts are in our artifact [21]. Agent-Harness Contract Context Retriever. While the contract guidance above mainly provides prompt-level principles that direct LLM to important contract requirements, generating and refining a concrete harness-oriented test also requires relevant code context to show how these harness contracts are implemented in agent codebase. Therefore, to complement the prompt-level contract guidance, the harness contract context retriever further serves as a callable context collection tool, which retrieves contract-relevant code context on demand during test generation. Such context includes the code needed to construct or control the test setup, such as the definition of a constructor, fixture, collaborator, or mock target. Specifically, when invoking the retriever, the LLM requests a class, function, or method definition by providing its source file and exact symbol name, which can often be inferred from imports in the current file. For example, if the target logic requires an âActionâ input defined elsewhere, an import such as âfrom pkg.actions import Actionâ identifies the module âpkg.actionsâ, corresponding to the source file âpkg/actions.pyâ. The LLM can therefore request the definition of âActionâ from that file. The retrieved definition exposes the required fields and their types, helping the LLM construct a contract-faithful mock action that serves as a valid input for exercising the target logic. Upon receiving the request, the retriever deterministically resolves it using AST-based matching for symbol definitions and bounded extraction for surrounding module context. This demand-driven retrieval complements the high-level guidance with repository-specific context needed to construct contract-faithful test setups and make the target behavior reachable and controllable. 7 RQ3: Test Adequacy Improvement Experiment Setting. This RQ evaluates the effectiveness of HarnessTester by investigating the test adequacy improvement it gains on our studied agentic systems. Baselines. As there are no existing test generation techniques designed for agent harness code, we compare HarnessTester with four state-of-the-art test generation techniques that are designed for general software systems. For the agents implemented in Python, we include CoverUp [3] and TestTailor [92] as baselines, two recent LLM-driven test generation approaches for general Python software; for the agents implemented in TypeScript, we include Qodo Cover [58] and TestPilot 2 [44,65], two recent LLM-driven test generation techniques for general TypeScript software. To isolate the effect of contract-specific design in HarnessTester, we include a contract-agnostic variant of HarnessTester by removing the agent harness contract support. Experimental Process. We include two test generation settings, including (i) general project-wide generation, which does not specify any target but aims at the entire codebase of the studied agentic system; (i) scoped LDH-wide generation, which provides the LDH-containing files as the targets. Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild13 For all baselines, we directly use their released implementation, configuring their target scope for each setting while retaining their native generation workflows. In the scoped setting, the baselines receive the same LDH-containing file as HarnessTester for a fair comparison. We use GPT-5- mini [47] as the backbone LLM for all studied techniques given its cost-effectiveness. Following previous work on application-level automated testing [30,85], all techniques are executed with a 120-minute timed budget per project. 0 153045607590 105120 0.0 2.5 5.0 7.5 10.0 OpenHands 0 153045607590 105120 0 10 20 30 Aider 0 153045607590 105120 0 20 40 SWE-agent 0 153045607590 105120 0 10 20 30 PR-Agent 0 153045607590 105120 0 20 40 60 GPT Researcher 0 153045607590 105120 0 10 20 Browser Use 0 153045607590 105120 0 10 20 30 RD-Agent 0 153045607590 105120 0.0 0.5 1.0 OpenClaw 0 153045607590 105120 0.0 2.5 5.0 7.5 10.0 Roo Code 0 153045607590 105120 0 1 2 3 4 Kimi Code LDH coverage gain (p) Elapsed time (min) Ours Agnostic CoverUp (Scoped) CoverUp (General) Qodo Cover (Scoped) Qodo Cover (General) TestTailor (Scoped) TestTailor (General) TestPilot2 (Scoped) TestPilot2 (General) LineBranch Fig. 3. LDH-wide coverage gains achieved by different test generation techniques over time. 0 153045607590 105120 0 2 4 6 8 OpenHands 0 153045607590 105120 0 10 20 30 Aider 0 153045607590 105120 0 10 20 SWE-agent 0 153045607590 105120 0 10 20 PR-Agent 0 153045607590 105120 0 20 40 60 GPT Researcher 0 153045607590 105120 0.0 2.5 5.0 7.5 10.0 Browser Use 0 153045607590 105120 0 5 10 15 20 RD-Agent 0 153045607590 105120 0.0 0.2 0.4 0.6 0.8 1.0 OpenClaw 0 153045607590 105120 0 1 2 3 Roo Code 0 153045607590 105120 0 1 2 3 Kimi Code Whole-project coverage gain (p) Elapsed time (min) Ours Agnostic CoverUp (Scoped) CoverUp (General) Qodo Cover (Scoped) Qodo Cover (General) TestTailor (Scoped) TestTailor (General) TestPilot2 (Scoped) TestPilot2 (General) LineBranch Fig. 4. Project-wide coverage gains achieved by different test generation techniques over time. Test Coverage Improvement. Figure 3 and Figure 4 present the trend of LDH-level and project- level coverage gains achieved by all studied techniques over time. Consistent LDH Coverage Gains. HarnessTester consistently achieves the highest final LDH- wide line- and branch coverage gains among all studied baselines, indicating its effectiveness in improving test adequacy for LLM-dependent harness code. Moreover, this advantage emerges early in the test-generation process: e.g., LDH-wide branch coverage gains lead across all ten projects from the 30-minute checkpoint onward, and line coverage gains do so from 75 minutes onward. 14Xiong et al. Substantial Gains in Targeted Test Generation. Although restricting test generation to LDH-bearing files substantially benefits baselines (e.g., 25.09% and 23.15% higher gains compared with the general setting), HarnessTester still substantially outperforms all the baselines with the scoped setting. At 120 minutes, HarnessTester achieves macro-averaged LDH line- and branch-coverage gains of 25.84 and 25.41 percentage points, respectively, with the line-coverage gain reaching 68.77 points on GPT Researcher. Compared with the strongest scoped baseline selected separately for each agent and metric, these average gains are 82.27% and 94.36% larger, respectively. Against the strongest general external baselines, the corresponding improvements are 95.29% and 109.22%. These results confirm the advantage of HarnessTester over baselines in the targeted test generation scenario. Contribution from Harness Contract Support. The pronounced performance drop from Har- nessTester to its contract-agnostic variant (the âAgnosticâ in Figure 3) confirms the effectiveness of our key design on harness contract support. In particular, HarnessTester produces 96.51% larger mean line-coverage gains and 96.24% larger branch-coverage gains, outperforming the contract-agnostic variant on both metrics in every project. We further run a fine-grained ablation analysis on OpenHands using variants without harness contract guidance and the harness con- tract context retriever, respectively. We observe consistent decreases in LDH line/branch coverage gains (43.26%/45.98% and 37.38%/41.00%, respectively) compared to HarnessTester, indicating the complementary contribution of each mechanism to harness contract support. Prevalent Improvements Across Semantic Categories. To examine whether HarnessTester im- proves test adequacy across the diverse semantic categories identified in RQ2, we further report category-level coverage improvements in Figure 5. Improvements are observed across all seman- tic categories: 12.12â25.49% of predicates in each category transition from uncovered to fully covered, while another 2.86â10.87% transition from partially to fully covered. Notably, for the challenging category of Agent Orchestration Control, where stateful and cross-component logic requires carefully coordinated test setups, 22.45% of branch predicates move from uncovered to fully covered and another 5.10% from partially to fully covered. These broad improvements suggest that HarnessTester can effectively exercise semantically diverse LDH behaviors, including those requiring complex test setups. The improvement aligns with the design intuition of using contract guidance and retrieved harness context to construct contract-faithful setups with diverse inputs. Consistent Gains in Project-wide Coverage. As shown in Figure 4, besides LDH-wide coverage, HarnessTester also achieves the highest project-level line- and branch-coverage gains among all configurations on eight and nine of the ten agents, respectively. Despite targeting LDH, its macro-averaged gains reach 13.90 and 17.30 percentage points, exceeding the corresponding 10.36 and 11.58 points achieved by the strongest general baselines (CoverUp and Qodo Cover). These results provide a complementary view of HarnessTesterâs effectiveness in generating high-quality tests for agentic systems, with coverage benefits evident not only within LLM-dependent harness code but also within the whole project. Provider I/O Error recovery Agent orchestr. Tool action Obs. feedback Context memory Presence shape Value pattern Size threshold Policy guard Gen. content Provider signal Tool action Agent context Obs. feedback 0 10 20 30 Transition share (%) Component Intention Judgment Pattern LLM Value Form Uncovered â Partially coveredUncovered â Fully coveredPartially covered â Fully covered Fig. 5. Semantic coverage gains achieved by HarnessTester. Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild15 Mutant Detection Effectiveness. Figure 6 presents the mutation score gains achieved by all studied techniques. Overall, HarnessTester achieves the highest mutation-score gain on every studied agent, outperforming all baselines and its contract-agnostic variant. Its largest percentage- point margin over the strongest baseline occurs on SWE-agent, where it gains 43.01 points compared with 22.15 for CoverUp (Scoped). Its advantage on OpenClaw appears smaller in percentage-point terms, partly resulting from the large measurement denominator: HarnessTesterâs 0.39-point gain corresponds to 185 additional kills over a pool of 46,967 mutants, compared with 147 for Qodo Cover (Scoped) and 141 for the contract-agnostic variant. Despite the small percentage- point differences, HarnessTester nevertheless kills 38 and 44 more mutants, respectively. Manual inspection of the generated tests and mutation results shows that they kill both mutants at previously uncovered locations and mutants that survived despite being exercised by the original suite. Overall, the improved capabilities of mutant detection further support HarnessTesterâs effectiveness in improving agent-harness test adequacy. 010203040 OpenHands PR-Agent Aider GPT Researcher SWE-agent 0481216 RD-Agent Browser Use Roo Code Kimi Code OpenClaw Mutation score gain (p) Ours Agnostic CoverUp (Scoped) CoverUp (General) Qodo Cover (Scoped) Qodo Cover (General) TestTailor (Scoped) TestTailor (General) TestPilot2 (Scoped) TestPilot2 (General) Fig. 6. LDH mutation score gains achieved by different test generation techniques. Finding 5: HarnessTester demonstrates strong capability in generating high-adequacy tests for LLM-dependent harness code across all studied agents. It consistently outperforms all evalu- ated test generation baselines in both LDH- and project-wide coverage gains. Our ablation study further demonstrates the substantial contribution of agent-harness contract support. Beyond coverage, tests generated by HarnessTester also exhibit stronger mutant detection capabil- ity. Overall, these results highlight the value of developing harness-oriented test generation techniques to improve the test adequacy of real-world agentic systems. 8 RQ4: Real-World Harness Bug Detection Although mutation scores can reflect the potential fault-detection capability of generated tests, mu- tants are artificially injected faults and may not fully represent real-world harness bugs. Therefore, in this RQ, we further evaluate the test adequacy of tests generated by HarnessTester and the baselines by examining their capability to detect real-world harness bugs. 8.1 RQ 4.a: Detecting Historical Harness Bugs Benchmark and Process. To systematically evaluate the precision and recall of bug detection, we first build H-Bench, a benchmark of 250 real-world harness bugs historically reported and fixed 16Xiong et al. across 10 studied agentic systems, including 140 bugs from seven Python agents and 110 from three TypeScript agents. We construct the benchmark by mining historical bug-fixing commits and retaining bugs whose patches involve LDH code, thereby focusing on bugs in LLM-dependent harness logic, given its importance [10]. To reduce leakage from existing regression tests, we retain only bugs that cannot be exposed by any test in the buggy revision, preventing the LLMs from simply reusing existing bug-revealing tests for test generation. To the best of our knowledge, H-Bench is the first benchmark dedicated to real-world LLM-dependent harness bugs, and we release it in our artifact [21] for future research. Given the large manual effort involved in inspecting test failures, in this RQ, we compare HarnessTester with the two best-performing baselines in RQ3 (i.e., CoverUp for Python and Qodo Cover for TypeScript). To assess the contribution of harness contract support, we also include the contract-agnostic variant of HarnessTester. In particular, for each case, we provide each technique with the buggy location (e.g., the functions or classes where the developer-submitted patch occurs) as the target code for test generation, but without exposing the patch, bug description, or fixed-side implementation. A bug is considered plausibly detected only if the generated test fails on the buggy version but passes on the fixed version. We then manually review all plausible detections to verify whether the test exposes the same bug fixed by developers or instead constitutes a false positive. Table 4. Historical bug revealing. HarnessTester AgnosticBaseline AgentCases TP/FP Cost$ TP/FP Cost$ TP/FP Cost$ OpenHands428/015.002/19.860/03.91 Aider196/15.191/14.510/01.35 SWE-agent72/01.981/01.400/00.71 PR-Agent112/13.281/03.030/02.21 GPT Researcher43/10.663/00.380/00.13 Browser Use418/213.913/015.560/03.85 RD-Agent161/06.770/04.470/01.13 OpenClaw8810/122.724/319.542/06.23 Roo Code162/06.170/04.580/01.42 Kimi Code61/11.890/01.590/00.75 Total250 43/7 77.58 15/5 64.92 2/0 21.68 Results. Table 4 presents the number of de- tected true bugs (TP) and false positives (FP) of all studied techniques on H-Bench with the total generation costs per project. Overall, Har- nessTester achieves the highest bug detec- tion rate (17.20%) by detecting 43 historical bugs, while the best baseline can only detect 2 historical bugs (0.80%). In addition, removing harness-contract support reduces the number of revealed bugs from 43 (17.20%) to 15 (6.00%), while the proportion of false positives among retained differential outcomes increases from 14.00% to 25.00%. Manual review identified typ- ical false positives arising from fixtures that violated production-side agent-harness contracts or assertions that assumed unsupported behavior. For example, one test omitted a required command field from a tool-call payload but expected an undocumented fallback; another passed a brack- eted IP literal to a helper whose caller supplies normalized hostnames. These cases reinforce the need for agent-harness contract fidelity: fixtures and mocks must preserve production-established contracts to help the test reach the target in a valid way, and must not demand behavior beyond existing contracts. In terms of cost, HarnessTester is more expensive overall but substantially more cost-effective for bug revealing. Across 250 cases, HarnessTester reveals 43 bugs at a total cost of $77.58, or $1.80 per revealed bug, compared with $10.84 per bug for the baseline. 8.2 RQ 4.b: Detecting Previously-Unknown Harness Bugs To further investigate whether HarnessTester can generate tests to detect previously-unknown harness bugs, we apply it to the latest releases of agentic systems used in our previous study. In particular, as OpenHands was recently refactored into several components, we therefore select its two core components, Agent Canvas [53] and SDK [54] in this experiment. To control costs, we select 100 functions containing LDH code per agent codebase as test-generation targets. Then, we manually inspect all failures triggered by the generated tests and report new bugs to developers. Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild17 Table 5. Detected real-world bugs. ProjectNew Known Confirmed Fix Cost$ OpenHands Canvas50337.38 OpenHands SDK1031097.68 Aider94996.69 SWE-agent43446.71 PR-Agent52559.16 GPT Researcher61665.80 Browser Use15813127.85 RD-Agent115988.23 OpenClaw96987.15 Roo Code110â7.26 Kimi Code32107.73 Total883469 64 81.64 Results. Table 5 summarizes our detected bugs and their upstream outcomes. In total, HarnessTester identifies 122 true harness bugs, including 34 previously-known and 88 previously-unknown bugs. Among the 88 pre- viously unknown bugs, 69 are confirmed by developers, 64 are fixed or assigned an open fix, and one of them is assigned a CVE number. Roo Codeâs GitHub repository was archived for refactoring during our study period; therefore, new issues could not be created. Overall, these results highlight the practical effectiveness of HarnessTester in generating bug-revealing tests for agent harnesses. Real-World Example. Listing 2 shows a previously-unknown sensitive-data leakage bug in Browser Use detected by HarnessTester and confirmed by developers. The system records model- selected tool calls as typed actions in the agent execution history. Our generated test reveals that a sensitive value nested within a typed action parameter can bypass redaction and remain in the saved history. In the persistence path,dump_actionsserializes the actions, whileredact_dictdelegates list-valued fields toredact_list, which handles strings and dictionaries but leaves nested lists unchanged. Consequently, the synthetic credential nested inrowsremains unredacted in the saved history. Revealing this bug requires more than placing a secret in an arbitrary payload. The test must embed a schema-valid input in a valid chain of action, model-output, and history objects, provide the expected sensitive-data mapping, and inspect the resulting file. This illustrates the role of agent-harness contracts: the test setup must preserve the runtime representations, object relationships, and interactions expected by the target logic. The test must further vary this valid setup with a nested-list input containing the sensitive value. Neither the best-performing Python baseline (CoverUp) nor our contract-agnostic variant revealed this bug. After confirmation, the bug was fixed upstream through recursive redaction. Listing 2. A previously-unknown sensitive-data leakage in Browser Use detected by HarnessTester. # Simplified persistence path class AgentHistoryList: defsave_to_file(self, out, sensitive_data): data = "actions": dump_actions(self.history) write_json(out, redact_dict(data, sensitive_data)) defredact_dict(data, s): return k: mask(v, s) if isinstance(v, str) else redact_dict(v, s) if isinstance(v, dict) elseredact_list(v, s) if isinstance(v, list) else v for k, v in data.items() # Bug: nested lists remain unchanged defredact_list(value, s): return [ mask(x, s) if isinstance(x, str) else redact_dict(x, s) if isinstance(x, dict) else x for x in value ] # Test body; imports and wrapper omitted class InputParams(BaseModel): rows: list[list[str]] class InputAction(ActionModel): input: InputParams secret = "token-123" custom_output =AgentOutput.type_with_custom_actions( InputAction) history =AgentHistoryList(history=[AgentHistory( model_output=custom_output( evaluation_previous_goal="ok", memory="ok", next_goal="done", action=[InputAction( input=InputParams(rows=[[secret]]))]), result=[],state=BrowserStateHistory( url="https://example.com", title="Example", tabs=[], interacted_element=[]))]) out = tmp_path / "history.json" history.save_to_file( out,sensitive_data="api_key": secret) saved = out.read_text(encoding="utf-8") assert secret not in saved # fails assert "<secret>api_key</secret>" in saved 18Xiong et al. Finding 6: HarnessTester effectively detects real-world harness bugs in agentic systems. In particular, tests generated by HarnessTester can detect substantially more historical bugs than the best-performing baseline. Beyond historical bugs, HarnessTester also uncovers 88 previously unknown harness bugs, 69 of which have been confirmed by developers. These results demonstrate the practical effectiveness of HarnessTester in uncovering harness bugs and improving the reliability of agentic systems. 9 Threats to Validity External Validity. The generality of our findings can be scoped to our studied agents. To mitigate this threat, we select diverse and actively developed systems (e.g., OpenClaw) spanning different application domains, implementation languages, and agent architectures. Similarly, our evaluation considers a representative set of existing test generation techniques and configurations, while other techniques or models may yield different results. We therefore release our benchmark and implementation to facilitate replication and evaluation on additional systems and techniques [21]. Internal Validity. The semantic pattern analysis in RQ2 relies on manual labeling and may be subject to human judgment. To reduce this threat, two experienced annotators independently label the sampled branches using an explicitly defined codebook, achieving high inter-annotator agreement (í =0.96), with remaining disagreements resolved through discussion with a third reviewer. Our metric calculation and implementation may be subject to implementation errors. To mitigate this threat, we validate intermediate results, manually inspect representative cases, and use established coverage and mutation-testing tools whenever applicable to reduce potential measurement errors. 10 Conclusion This work presents the first study of test adequacy in real-world agent harness and reveals sub- stantial test-adequacy gaps in exercising the critical LLM-dependent harness code. We find that these gaps are particularly pronounced when testing requires dynamic execution contexts, implicit input spaces, and cross-component interactions. Motivated by these findings, we propose Har- nessTester, the first harness-oriented test generation technique that incorporates agent-harness contract support to construct contract-faithful tests for LLM-dependent harness code. Extensive evaluation demonstrates that HarnessTester substantially outperforms general-purpose test gen- eration in coverage and mutation-score gains, while its contract support contributes significantly to these improvements. Furthermore, HarnessTester detects 122 real-world harness bugs in widely-used agentic systems (e.g., OpenClaw). 88 of them are previously-unknown bugs and 69 of them have been confirmed by developers. Overall, our findings highlight the practical value of building harness-oriented test generation in improving the reliability of real-world agentic systems. Data Availability The data and code of this work are available at [21]. References [1]AI Incident Database. 2025. Incident 1152: LLM-Driven Replit Agent Reportedly Executed Unauthorized Destructive Commands During Code Freeze, Leading to Loss of Production Data. https://incidentdatabase.ai/cite/1152/ [2] Aider Contributors. 2026. Aider. https://github.com/Aider-AI/aider/tree/v0.86.2 [3] Juan Altmayer Pizzorno and Emery D. Berger. 2025. CoverUp: Effective High Coverage Test Generation for Python. Proceedings of the ACM on Software Engineering 2, FSE (2025), 2897â2919. doi:10.1145/3729398 [4] Anthropic. [n. d.]. Claude Code. https://claude.com/product/claude-code [5]Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. 2016. QL: Object-oriented Queries on Relational Data. In 30th European Conference on Object-Oriented Programming (ECOOP 2016) (Leibniz International Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild19 Proceedings in Informatics (LIPIcs), Vol. 56), Shriram Krishnamurthi and Benjamin S. Lerner (Eds.). Schloss Dagstuhl â Leibniz-Zentrum fĂźr Informatik, Dagstuhl, Germany, 2:1â2:25. doi:10.4230/LIPIcs.ECOOP.2016.2 [6] Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, and Jiaxin Pei. 2026. How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks. arXiv:2604.22750 [cs.CL] https://arxiv.org/abs/2604.22750 [7]Antoine Beugnard, Jean-Marc JĂŠzĂŠquel, NoĂŤl Plouzeau, and Damien Watkins. 1999. Making Components Contract Aware. Computer 32, 7 (July 1999), 38â45. doi:10.1109/2.774917 [8] Browser Use Contributors. 2026. Browser Use. https://github.com/browser-use/browser-use/releases/tag/0.13.2 [9]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya G. Parameswaran, Dan Klein, Kannan Ramchandran, Matei A. Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail?. In Advances in Neural Information Processing Systems, Vol. 38. doi:10.52202/085713-4082 [10]Jingyi Chen, Songqiang Chen, Hengcheng Zhu, Jialun Cao, Jiasi Shen, and Shing-Chi Cheung. 2026. Under- standing Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports. arXiv:2607.15684 [cs.SE] doi:10.48550/arXiv.2607.15684 [11]Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. ACM, 572â576. doi:10.1145/3663529.3663801 [12] Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37â46. doi:10.1177/001316446002000104 [13] Coverage.py Contributors. [n. d.]. Coverage.py Documentation. https://coverage.readthedocs.io/en/latest/ [14] Cursor AI. 2026. Cursor. https://cursor.com/home [15] Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. 2025. AI agents under threat: A survey of key security challenges and future pathways. Comput. Surveys 57, 7 (2025), 1â36. doi:10.1145/3716628 [16] Devin. 2026. Devin AI. https://devin.ai/ [17]Aisha Down. 2026. Amazonâs cloud âhit by two outages caused by AI tools last yearâ. The Guardian.https: //w.theguardian.com/technology/2026/feb/20/amazon-cloud-outages-ai-tools-amazon-web-services-aws [18] Aisha Down. 2026.Meta AI agentâs instruction causes large sensitive data leak to employees.The Guardian. https://w.theguardian.com/technology/2026/mar/20/meta-ai-agents-instruction-causes-large-sensitive- data-leak-to-employees [19]GPT Researcher Contributors. 2026. GPT Researcher. https://github.com/assafelovic/gpt-researcher/releases/tag/v3.5.0 [20]Ehud Halamish and Vladimir Tokarev. 2026. Agent-Inflicted Damage: Inside the Real-World Failures of Enterprise AI Systems. Cyera Research. https://w.cyera.com/research/agent-inflicted-damage-inside-the-real-world-failures- of-enterprise-ai-systems [21] HarnessTester 2026. HarnessTester Repository. https://github.com/yifanxiong272/harnesstester_artifact [22]Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E. Hassan. 2026. An empirical study of testing practices in open source AI agent frameworks and agentic applications. Empirical Softw. Engg. 31, 5 (April 2026), 55 pages. doi:10.1007/s10664-026-10857-9 [23] Anders HovmĂśller et al. [n. d.]. mutmut: Python Mutation Tester. https://github.com/boxed/mutmut [24] Ethan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G. K. Dharesan, Alexander Spangher, Shikhar Murty, Tenghao Huang, and Christopher D. Manning. 2026. WebDS: An End-to-End Benchmark for Web-Based Data Science. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=7cHhcrbr6x [25]Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649â678. doi:10.1109/TSE.2010.62 [26]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 54107â54157. https://proceedings.iclr.c/paper_files/paper/2024/file/edac78c3e300629acfe6cbe9ca88fb84-Paper-Conference.pdf [27]Shyam Sundar Kannan, Vishnunandan L. N. Venkatesh, and Byung-Cheol Min. 2024. SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 12140â12147. doi:10.1109/IROS58592.2024.10802322 [28]Kimi Code Contributors. 2026. Kimi Code. https://github.com/MoonshotAI/kimi-code/releases/tag/@moonshot- ai/kimi-code@0.19.2 [29]Jens Kohl, Otto Kruse, Youssef Mostafa, Andre Luckow, Karsten Schroer, Thomas Riedl, Ryan French, David Katz, Manuel P. Luitz, Tanrajbir Takher, Ken E. Friedl, and CĂŠline Laurent-Winter. 2026. Automated structural testing of LLM-based agents: methods, framework, and case studies. arXiv:2601.18827 [cs.SE] https://arxiv.org/abs/2601.18827 20Xiong et al. [30]Yuanhong Lan, Yifei Lu, Zhong Li, Minxue Pan, Wenhua Yang, Tian Zhang, and Xuandong Li. 2024. Deeply Reinforcing Android GUI Testing with Deep Reinforcement Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. doi:10.1145/3597503.3623344 [31]Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. 2026. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052 [cs.AI] doi:10.48550/arXiv.2603.28052 [32]Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 919â931. doi:10.1109/ICSE48619.2023.00085 [33]Junjie Li, Xi Xiao, Yunbei Zhang, Chen Liu, Lin Zhao, Xiaoying Liao, Yingrui Ji, Janet Wang, Yingqiang Ge, Weijie Xu, Xi Fang, Xiang Xu, Tianchen Zhao, Youngeun Kim, Jihun Hamm, Tianyang Wang, and Chandan Reddy. 2026. Agent Harness Engineering: A Survey. https://openreview.net/pdf ?id=eONq7FdiHa [34] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 52989â53046. https://proceedings.iclr.c/paper_files/paper/2024/file/e9df36b21f4e211a8b71e8b7e9f57-Paper-Conference.pdf [35] Ryan Lopopolo. 2026. Harness engineering: leveraging Codex in an agent-first world. https://openai.com/index/ harness-engineering/ [36] Xinghua Lou, Miguel LĂĄzaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P. Murphy. 2026. AutoHarness: improving LLM agents by automatically synthesizing a code harness. arXiv:2603.03329 [cs.CL] https://arxiv.org/abs/2603.03329 [37] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292 [cs.AI] https://arxiv.org/abs/2408.06292 [38]Cleyton Magalhaes, Italo Santos, Brody Stuart-Verner, and Ronnie de Souza Santos. 2025. Testing the Untestable? An Empirical Study on the Testing Process of LLM-Powered Software Systems. arXiv:2508.00198 [cs.SE] https: //arxiv.org/abs/2508.00198 [39] Oorja Majgaonkar, Zhiwei Fei, Xiang Li, Federica Sarro, and He Ye. 2025. Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories. arXiv:2511.00197 [cs.SE] https://arxiv.org/abs/2511.00197 [40] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al.2026. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. In The Fourteenth International Conference on Learning Representations. https: //openreview.net/forum?id=a7Qa4CcHak [41] B. Meyer. 1992. Applying âdesign by contractâ. Computer 25, 10 (1992), 40â51. doi:10.1109/2.161279 [42] GrĂŠgoire Mialon, ClĂŠmentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for General AI Assistants. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 9025â9049. https://proceedings.iclr.c/paper_files/paper/2024/ file/25ae35b5b1738d80f1f03a8713e405ec-Paper-Conference.pdf [43] Microsoft AI Red Team. 2026. Taxonomy of Failure Modes in Agentic AI Systems, Version 2.0. White Paper. Mi- crosoft. https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/bade/documents/products-and- services/en-us/security/Taxonomy-of-Failure-Modes-in-Agentic-AI-Systems-v2-0.pdf [44] neu-se. 2024. TestPilot 2. https://github.com/neu-se/testpilot2/tree/79c3b62 [45] Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, Lingjie Chen, Yanjun Zhao, Ke Yang, Bingxuan Li, Cheng Qian, Gaotang Li, Xiao Lin, Zhichen Zeng, Ruizhong Qiu, Sirui Chen, Yifan Sun, Xiyuan Yang, Ruida Wang, Rui Pan, Chenyuan Yang, Dylan Zhang, Liri Fang, Zikun Cui, Yang Cao, Pan Chen, Dorothy Sun, Ren Chen, Mahesh Srinivasan, Nipun Mathur, Yinglong Xia, Hong Li, Hong Yan, Pan Lu, Lingming Zhang, Tong Zhang, Hanghang Tong, and Jingrui He. 2026. Code as Agent Harness. arXiv:2605.18747 [cs.CL] https://arxiv.org/abs/2605.18747 [46] Alexander Novikov, Ngân V Ě u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv:2506.13131 [cs.AI] https://arxiv.org/abs/2506.13131 [47] OpenAI. 2025. GPT-5 Mini Model. https://developers.openai.com/api/docs/models/gpt-5-mini [48] OpenAI. 2026. Codex. https://openai.com/codex/ [49] OpenClaw. 2026. OpenClaw AI website. https://openclaw.ai/ [50] OpenClaw Contributors. 2026. OpenClaw. https://github.com/openclaw/openclaw/releases/tag/v2026.3.28 [51] openclawvps. 2026. OpenClaw Statistics 2026: Growth, Users, Security, Data. https://openclawvps.io/blog/openclaw- statistics Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild21 [52] OpenHands Contributors. 2026. OpenHands. https://github.com/OpenHands/OpenHands/releases/tag/1.6.0 [53] OpenHands Contributors. 2026. OpenHands Agent Canvas. https://github.com/OpenHands/OpenHands [54] OpenHands Contributors. 2026. OpenHands Software Agent SDK. https://github.com/OpenHands/software-agent-sdk [55]Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Quincy Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, Koushik Sen, Dawn Song, Joseph E. Gonzalez, Ion Stoica, Matei Zaharia, and Marquita Ellis. 2026. Measuring Agents in Production. arXiv:2512.04123 [cs.CY] https://arxiv.org/abs/2512.04123 [56]Rangeet Pan, Tyler Stennett, Divya Sankar, Bridget McGinn, Alessandro Orso, Raju Pavuluri, Saurabh Sinha, and Maja Vukovic. 2026. Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering. https://arxiv.org/abs/2608.08413 [57] PR-Agent Contributors. 2026. PR-Agent. https://github.com/The-PR-Agent/pr-agent/releases/tag/v0.36.0 [58] Qodo AI. 2025. Qodo Cover. https://github.com/qodo-ai/qodo-cover/releases/tag/0.3.10 [59] Alfin Wijaya Rahardja, Junwei Liu, Weitong Chen, Zhenpeng Chen, and Yiling Lou. 2025. Can Agent Fix Agent Issues?. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference. Curran Associates, Inc., 49002â49028. doi:10.52202/085713-1638 [60] Prithvi Rajasekaran. 2026. Harness design for long-running application development. https://w.anthropic.com/ engineering/harness-design-long-running-apps [61] RD-Agent Contributors. 2025. RD-Agent. https://github.com/microsoft/RD-Agent/releases/tag/v0.8.0 [62]Thomas Reps, Susan Horwitz, and Mooly Sagiv. 1995. Precise interprocedural dataflow analysis via graph reachability. In Proceedings of the 22nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (San Francisco, California, USA) (POPL â95). Association for Computing Machinery, New York, NY, USA, 49â61. doi:10.1145/199448. 199462 [63] Roo Code Contributors. 2026. Roo Code. https://github.com/RooCodeInc/Roo-Code/releases/tag/v3.54.0 [64]Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proc. ACM Softw. Eng. 1, FSE, Article 43 (July 2024), 21 pages. doi:10.1145/3643769 [65] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024), 85â105. doi:10. 1109/TSE.2023.3334955 [66]C.B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering 25, 4 (1999), 557â572. doi:10.1109/32.799955 [67] Jakob Steinschaden. 2026. OpenClaw Hits 38 Million Monthly Visitors â Chinese LLMs Dominate Usage. Trending Topics. https://w.trendingtopics.eu/openclaw-numbers/ [68]Anselm L. Strauss and Juliet M. Corbin. 1998. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (2nd ed.). Sage Publications, Thousand Oaks, CA. [69]Stryker Mutator. [n. d.]. StrykerJS: Mutation Testing for JavaScript and Friends. GitHub. https://github.com/stryker- mutator/stryker-js [70] Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu, Haoyun Zhang, Jeffrey Lin, et al.2026. Agentsâ Last Exam. arXiv:2606.05405 [cs.AI] https://arxiv.org/abs/2606.05405 [71] SWE-agent Contributors. 2025. SWE-agent. https://github.com/SWE-agent/SWE-agent/releases/tag/v1.1.0 [72] Vitest Contributors. [n. d.]. Vitest: Coverage Guide. https://vitest.dev/guide/coverage.html [73]Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, et al.2024. Large Language Models for Robotics: Opportunities, Challenges, and Perspectives. arXiv:2401.04334 [cs.RO] https://arxiv.org/abs/2401.04334 [74] Yuhang Wang, Haichang Gao, Zhenxing Niu, Zhaoxiang Liu, Wenjing Zhang, Xiang Wang, and Shiguo Lian. 2026. A Systematic Security Evaluation of OpenClaw and Its Variants. arXiv:2604.03131 [cs.CR] https://arxiv.org/abs/2604. 03131 [75] Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE â24). Association for Computing Machinery, New York, NY, USA, 1258â1268. doi:10.1145/3691620. 3695501 [76]Martin Weiss, Miles Q. Li, Alejandro H. Artiles, Yacine Mkhinini, Chris Pal, Hugo Larochelle, and Nasim Rahaman. 2026. Lacuna: A Research Map for Machine Learning. arXiv:2606.26246 [cs.DL] https://arxiv.org/abs/2606.26246 [77] Elaine J. Weyuker. 1986. Axiomatizing software test data adequacy. IEEE Transactions on Software Engineering SE-12, 12 (1986), 1128â1138. doi:10.1109/TSE.1986.6313008 [78]Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. 2026. Reducing Cost of LLM Agents with Trajectory Reduction. Proc. ACM Softw. Eng. 3, FSE, Article FSE056 (June 2026), 23 pages. doi:10.1145/3797084 22Xiong et al. [79]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems, Vol. 37. doi:10.52202/079017-1650 [80]Ziluo Xue, Yanjie Zhao, Shenao Wang, Kai Chen, and Haoyu Wang. 2025. A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 3369â3380. doi:10.1109/ASE63991.2025.00278 [81]Faouzi El Yagoubi, Godwin Badu-Marfo, and Ranwa Al Mallah. 2026. AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems. IEEE Access 14 (2026), 94960â94978. doi:10.1109/ACCESS.2026.3704541 [82]Chen Yang, Junjie Chen, Bin Lin, Ziqi Wang, and Jianyi Zhou. 2026. Advancing Code Coverage: Incorporating Program Analysis with Large Language Models. ACM Transactions on Software Engineering and Methodology 35, 5 (2026), 1â31. doi:10.1145/3748505 [83] Xu Yang, Xiao Yang, Shikai Fang, Yifei Zhang, Jian Wang, Bowen Xian, Qizheng Li, Jingyuan Li, Minrui Xu, Yuante Li, Haoran Pan, Yuge Zhang, Weiqing Liu, Yelong Shen, Weizhu Chen, and Jiang Bian. 2025. R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science. arXiv:2505.14738 [cs.AI] https://arxiv.org/abs/2505.14738 [84] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025.í-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In The Thirteenth International Conference on Learning Representations. https: //proceedings.iclr.c/paper_files/paper/2025/hash/1b126c38b8638e07bef37e7b2b72bf-Abstract-Conference.html [85]Juyeon Yoon, Robert Feldt, and Shin Yoo. 2024. Intent-Driven Mobile GUI Testing with Autonomous Large Language Model Agents. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 129â139. doi:10.1109/ ICST60714.2024.00020 [86] Justin Young. 2025. Effective harnesses for long-running agents. https://w.anthropic.com/engineering/effective- harnesses-for-long-running-agents [87] Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703â1726. doi:10.1145/3660783 [88] Xiaowen Zhang, Hannuo Zhang, and Shin Hwei Tan. 2026. Understanding Bugs in Modern Agentic Frameworks: A Study of Symptoms, Root Causes, and Triggering Conditions. arXiv:2604.08906 [cs.SE] https://arxiv.org/abs/2604.08906 [89]Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K. Reddy. 2026. Stop Comparing LLM Agents Without Disclosing the Harness. arXiv:2605.23950 [cs.AI] https://arxiv.org/abs/2605.23950 [90]Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2025. Agent- SafetyBench: Evaluating the Safety of LLM Agents. arXiv:2412.14470 [cs.CL] https://arxiv.org/abs/2412.14470 [91] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations. https://proceedings.iclr.c/paper_files/ paper/2024/hash/4410c0711e9154a7a2d26f9b3816d1ef-Abstract-Conference.html [92]Xiaoxuan Zhou, Yiling Lou, Jinhao Dong, and Dan Hao. 2026. TestTailor: Generating High-Coverage Tests via Path- Proximal Tests with LLMs. Proceedings of the ACM on Software Engineering 3, FSE (2026), 277â298. doi:10.1145/3797140 [93]Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software unit test coverage and adequacy. ACM Comput. Surv. 29, 4 (Dec. 1997), 366â427. doi:10.1145/267580.267590 [94] Xinxue Zhu, Jiacong Wu, Xiaoyu Zhang, Tianlin Li, Yanzhou Mu, Juan Zhai, Chao Shen, Chunrong Fang, and Yang Liu. 2026. Bugs in Modern LLM Agent Frameworks: An Empirical Study. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering (Concordia University, Montreal, QC, Canada) (FSE Companion â26). Association for Computing Machinery, New York, NY, USA, 1588â1592. doi:10.1145/3803437.3805536