Paper deep dive

AgentSpy: Making AI Agent Behavior Observable

Christoph Bühler, Matteo Biagiola, Luca Di Grazia, Guido Salvaneschi

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 88

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:AI agents built on large language models (LLMs) run shell commands, read and write files, and reach the network, typically with their user's privileges. However, what an agent does during an execution is difficult to understand: tests assert on the result, and the agent's trajectory records only what the agent reports about itself, which may omit behavior executed by its subprocesses. We present AgentSpy, an approach that observes an agent from outside the agent. AgentSpy runs the agent in an isolated environment, configured by a declarative specification, and records the system calls and network traffic of the agent and of every process it executes. Based on this monitoring, AgentSpy supports two families of analyses: conformance analyses, which measure obligations, i.e., what an agent execution should do, and safety analyses, which check prohibitions, i.e., what an agent execution must never do. We instantiate one analysis of each family. The reliability analysis uses rules to summarize each run by the environment resources the agent uses: the commands it executed, the files it accessed, and the hosts it contacted. The security analysis applies deterministic rules to the system calls of an execution. For reliability, we evaluated AgentSpy on 77 tasks with the codex harness and three recent LLMs, executing each task three times. Sets of repeated runs of the same task are more similar than sets that include runs of another task in 92.2% of the comparisons. Among tasks for which all three runs pass outcome-based tests, the agent performs task-unrelated activities in 18% of the cases, reads the grading files in 7%, and does not use the developers' guidance in 17%. For security, the generic rules of AgentSpy detect four of five attack categories we considered, with no false positives across 50 runs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,225 characters extracted from source content.

Expand or collapse full text

AgentSpy: Making AI Agent Behavior Observable CHRISTOPH BÜHLER, University of St. Gallen, Switzerland MATTEO BIAGIOLA, University of St. Gallen and Università della Svizzera italiana (USI), Switzerland LUCA DI GRAZIA, University of St. Gallen, Switzerland GUIDO SALVANESCHI, University of St. Gallen, Switzerland AI agents built on large language models (LLMs) run shell commands, read and write files, and reach the network, typically with their user’s privileges. However, what an agent does during an execution is difficult to understand: tests assert on the result, and the agent’s trajectory records only what the agent reports about itself. Consequently, the agent’s recorded trajectory may omit behavior executed by its subprocesses. We present AgentSpy, an approach that observes an agent from outside the agent. AgentSpy runs the agent in an isolated environment, configured by a declarative specification, and records the system calls and network traffic of the agent and of every process it executes. Based on this monitoring, AgentSpy supports two families of analyses: conformance analyses, which measure obligations, i.e., what an agent execution should do, and safety analyses, which check prohibitions, i.e., what an agent execution must never do. We instantiate one analysis of each family. The reliability analysis uses a rule-based approach to build a summary of each run, consisting of environment resources the agent uses, i.e., the commands it executed, the files it accessed, and the hosts it contacted. The security analysis applies deterministic rules to the system calls of an execution. For reliability analysis, we evaluated AgentSpy on 77 tasks with thecodexharness and three recent LLMs, and executed each task three times. Sets of repeated runs of the same task are more similar than sets that include runs of another task in 92.2% of the comparisons. Among the tasks for which all three runs pass outcome-based tests, we found that in 18% of the cases the agent performs activities that are unrelated to the task, reads the grading files in 7% of the cases, and does not use the developers’ guidance in 17% of the cases. For security analysis, the generic rules of AgentSpy detects four out of five categories of attacks we considered, with no false positives across 50 runs. CCS Concepts:• Software and its engineering→ Software testing and debugging. Additional Key Words and Phrases: AI Agents, Dynamic Analysis, Behavior Analysis, Agent Security ACM Reference Format: Christoph Bühler, Matteo Biagiola, Luca Di Grazia, and Guido Salvaneschi. 2026. AgentSpy: Making AI Agent Behavior Observable. 1, 1 (October 2026), 22 pages. https://doi.org/10.1145/n.n 1 Introduction AI agents built on large language models (LLMs) carry out tasks autonomously: they call tools, run shell commands, read and write files, and reach the network [20], and they are now used across many fields [4]. An agent that runs on a user’s machine typically acts with the user’s privileges, so it can also take actions the user did not intend, including security-sensitive actions. Authors’ Contact Information: Christoph Bühler, christoph.buehler@unisg.ch, University of St. Gallen, St. Gallen, SG, Switzerland; Matteo Biagiola, matteo.biagiola@usi,unisg.ch, University of St. Gallen and Università della Svizzera italiana (USI), St. Gallen and Lugano, SG and TI, Switzerland; Luca Di Grazia, work@lucadigrazia.com, University of St. Gallen, St. Gallen, SG, Switzerland; Guido Salvaneschi, guido.salvaneschi@unisg.ch, University of St. Gallen, St. Gallen, SG, Switzer- land. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM X-X/2026/10-ART https://doi.org/10.1145/n.n , Vol. 1, No. 1, Article . Publication date: October 2026. arXiv:2610.06001v1 [cs.SE] 5 Oct 2026 What an agent does during an execution is, however, difficult to understand for its user. The user sees the result and the agent’s trajectory, i.e., the log of reasoning steps, actions, and tool outputs that the agent records about itself [12], but neither shows what the programs the agent executed actually do. This invisibility has started to matter in practice. In September 2026, OpenAI began disclosing cases in which its agents took misaligned actions [39]. In one case, an agent that had already retrieved the correct data locally uploaded it to a public paste service, without authorization, only to obtain a URL its browser tool could cite [41]; in another, agents jointly building a spreadsheet were required to use local files only, but a faulty environment kept them from reading each other’s files, so they passed the spreadsheet through a public temporary file-hosting service [40]. Existing work does not address this observability problem. Outcome-based benchmarks check the final result [33], so an agent execution can pass every test while also performing activities unrelated to the task (similar to the cases above). Static analyzers scan the instructions and scripts that are given to an agent for unintended behavior before the agent runs [27, 44, 10], but cannot observe what the agent does at runtime. Trajectory analyses [42, 35, 36] rely on the agent’s own record, which is incomplete: its smallest unit of observation is a single action, so when the agent runs a helper script that reads~/.sshand uploads the keys, the trajectory records one command and none of the files the script reads or the hosts it contacts. The record also holds only what the agent reports, so it is evidence only as far as the agent is cooperative. Some model providers additionally return the reasoning steps encrypted, which leaves the trajectory with actions and no thoughts. Overall, interpreting the trajectory typically relies on LLM-as-a-judge systems [42], which are non-deterministic, may miss details [24], and can be misled by adversarial text in tool outputs that the judge reads as part of the trajectory [23]. In this paper, we present AgentSpy, a rule-based approach that observes what an AI agent does at the system level. AgentSpy runs the agent in an isolated environment configured by what we call an execution kind: a declarative specification of the environment, of the agent and its parameters, and of the observation steps, so that the conditions of every execution are explicit. Inside the environment, AgentSpy records system calls and network traffic of the agent and of every executed process. Observing at the boundary of the operating system has three advantages. The evidence is complete at this boundary: every program started, file accessed, and host contacted is recorded, whether the agent acted directly, delegated the action to a script, or was steered into it by injected instructions. The evidence is also deterministic and independent of the agent: it does not rely on the agent’s account of itself, and it requires no agent instrumentation. AgentSpy’s analyses of an agent execution belong to two families, which mirror the classic distinction between liveness and safety [5]: conformance analyses measure obligations, i.e., what an execution should do, and safety analyses check prohibitions, i.e., what an execution must never do. The reliability analysis is a conformance analysis: it measures obligations such as following the guidance that developers provide for the task; whether an execution met an obligation is known only once the execution ends. The reliability analysis summarizes each execution as a star graph that characterizes three resource types, i.e., the commands executed, the files read or written, and the hosts contacted. It measures how much of the guidance an execution used, and it compares repeated executions of the same task through their star graphs, which shows how consistently the agent behaves. The security analysis is a safety analysis: it checks prohibitions such as reading secrets and private keys. A single system call is enough to report a violation, so the security analysis applies deterministic rules to the system calls of an agent execution. Two recent systems also observe agents at the system level, but each only covers part of what AgentSpy supports.AgentSight[58] matches each action of an agent to the LLM message that led to it. A second LLM then reads this record of one execution, and decides whether something went wrong, such as a prompt injection or a reasoning loop. The reliability analysis of AgentSpy, on the 2 other hand, supports multiple executions, via graph similarity.SkillDetonate[29] audits a skill before it is installed. It takes as input the skill, and starts an agent session in which the agent sees only the files of the skill, and placeholders instead of the contents of sensitive files. It reports a leak when data derived from a placeholder is sent to a host that is not on a list of trusted hosts, and a violation when the skill writes outside allowed directories or installs, downloads, or runs a program from an untrusted source. Differently, the security analysis of AgentSpy starts from a task that an agent solves with the skill, and the execution environment is a microVM with a full operating system. AgentSpy relies on deterministic rules over system calls, which are applicable to any agent execution, and each alert reports the rule that matched, the system call, and the process that triggered it. We evaluated AgentSpy on two benchmarks,SkillsBenchfor the reliability analysis (77 tasks) andSkill-Injectfor the security analysis (5 attack categories), with four recent LLMs running in thecodexharness, and multiple executions to account for non-determinism [43, 11]. We show that the star graph summary reliably distinguishes agent executions: sets of repeated executions of a task are more similar than sets that include executions of another task in 92.2% of the comparisons. Among the tasks whose executions all pass test cases provided bySkillsBench, our analysis reveals that the agent performs activities unrelated to the task in 18% of the cases, reads the grading files of the benchmark in 7%, and reaches the result without the guidance that developers provide through skills in 17%, which shows that the reliability analysis complements test cases. The generic rules of the security analysis flag four of the five attack categories in malicious skills, and raise no alert on the execution of benign tasks. Finally, AgentSpy does not add a significant overhead, since the runtime of the agent shows no significant difference without monitoring (푝-value≥ 0.16). This paper makes the following contributions: •AgentSpy, an approach that runs an agent in an isolated environment under a declarative execution kind, and records system calls and network traffic of the agent and of every process it spawns. • Two deterministic analyses built on AgentSpy, one of each family: (i) the reliability analysis, a conformance analysis that measures how far agent executions meet their obligations, and (i) the security analysis, a safety analysis that checks prohibitions by applying rules to system calls. •An evaluation on two benchmarks, four LLMs, and multiple executions, which shows that the reliability analysis complements test cases, that the security analysis reliably detects different attack categories, and that AgentSpy adds no significant overhead. 2 Background and Motivation 2.1 Agentic AI Systems and Agent Skills Agentic AI systems pursue a goal in a cycle of reasoning, planning, and acting: a foundation model, typically an LLM, chooses each next action, such as a tool call, and tracks state across steps. Moreover, agent skills contain reusable domain knowledge for agents [56]. A skill is typically a directory with a SKILL.md file and, optionally, the helper scripts, reference documents, and assets; SKILL.mdstarts with a metadata header (name and one-line description), followed by natural- language instructions [3]. At startup, the agent loads the name and description of every skill in a dedicated directory such as.claude/skills; when a description matches the task, it may load the full instructions into its context and adjust its plan. 3 Table 1. Excerpt (7 of 15 steps) of acodexexecution withglm-5.3-flashoncitation-check, colored by modality: natural-language reasoning (NL), shell commands (sh), inline programming-language code (PL), and unstructured tool output (Output). # Thought (NL)Action (sh / PL)Result (Output) 1 Verify the citations; read the citation-management skill first. sh cat .../SKILL.md; cat /root/test.bibtruncated (1579 ln) 2 Output truncated; list the entries and their titles instead. sh rg ‘^@|title=’ /root/test.bib27 entries; 3 look generic 3 The three generic titles look suspicious; inspect the rest of their entries. sh sed -n ‘80,360p’ /root/test.bib10.5678, 10.1234 (fake prefixes) 4 The suspects’ DOIs look invalid; check the titles against Crossref. PL for t in titles: urlopen(‘crossref...’+t)reals match; 3 suspects fuzzy 5 Try a plain web search as a fallback.sh curl bing; curl duckduckgopages returned; no titles parsed 6Use the skill’s Google Scholar script instead.sh python .../search_google_scholar.py scholarly missing 7 No external hit; the three entries use fabricated metadata; write the answer. sh apply_patch <<PATCH ... answer.json3 fabricated citations on file 2.2 Running Example Our running example is thecitation-checktask fromSkillsBench[33]: given a BibTeX bibliography that mixes real and fabricated entries, the agent must write the titles of the fabricated ones to answer.json. The task ships with thecitation-managementskill, which suggests the agent to search Google Scholar and PubMed for each entry, resolve every DOI against Crossref, and validate the result with a helper script. The skill contains six helper scripts, two assets, and five reference documents, such as one on validating citations. EachSkillsBenchtask also provides a verifier, a test suite that decides whether the task is solved, and an oracle, a deterministic script that solves it. The nine tests ofcitation-checkcheck that the agent createsanswer.jsonas valid JSON, identifies all fabricated citations, and reports their correct count. 2.3 Limits of the Agent Trajectory Agent harnesses such ascodexCLI log each execution as a trajectory, a sequence of (thought, action, result) triples [12]: a reasoning step, an interaction with the environment such as a shell command, and what the environment returns. In the execution shown in Table 1, the agent reads the skill and the bibliography (Row 1), lists the entry titles (Row 2), and notices that the DOIs of three entries carry unusual prefixes (Row 3). Crossref returns only fuzzy matches for these entries (Row 4). The attempts to confirm the finding fail: the agent’s pattern matches no titles on the returned web search pages (Row 5), and the skill’s Google Scholar script lacks a dependency (Row 6). The agent writes the three entries toanswer.json as fabricated (Row 7), and the execution passes all nine tests. None of the lookups the skill prescribes succeeded, so the verdict rests on the DOI prefixes, and the missing Crossref match: the agent solves the task without using the skill’s guidance. Characterizing such behavior automatically requires interpreting multiple modalities: natural lan- guage in the thoughts, shell commands and inline code in the actions, and unstructured tool output in the results. Categorizing actions is task-dependent and needs manual labeling to be accurate [12]. Interpreting thoughts and tool output typically relies on LLM-as-a-judge systems [42], which are non-deterministic and may hallucinate or miss details [24]. The trajectory is also incomplete: its smallest unit of observation is a single action, be it a tool call, a shell command, or a code snippet. 4 Orchestrator Isolated environment Execution Manager Run artifacts Analysis Runner Execution Kind Custom KindBase Kind Task Agent Analysis type Analysis report AgentSpy starts copied out runs in configures component data configuration input data flow configures extends Fig. 1. Overview of AgentSpy. It takes as input a task, an agent, and an analysis type, and outputs a report summarizing the system-level analysis. When the agent runs a helper script, the trajectory records one command but none of the files the script reads or writes, the hosts it contacts, or the processes it spawns. We therefore characterize agent behavior through system-level monitoring, i.e., by observing the boundary between the operating system and the agent, including every process it spawns. The resulting evidence consists of a small set of well-defined primitives, the executed commands, accessed files, and contacted network endpoints. The monitoring is also complete at this boundary: it captures every such event, whether the agent issued it directly, delegated it to a script, or triggered it unknowingly through a compromised skill. 3 Approach Figure 1 shows the overview of AgentSpy, a framework for monitoring the behavior of agentic AI systems. AgentSpy takes as input a task, an agent that executes it (configured with a given foundation model), and the type of behavior analysis the user wants to carry out on the agent. It is built around three core elements. The orchestrator takes the task configuration (i.e., the execution kind) and starts a new, isolated execution environment. The execution manager runs within that environment: it sets up the agent, runs it, evaluates the execution, and gathers the run artifacts, which the orchestrator then copies out of the environment. Finally, the analysis runner processes these artifacts and produces the analysis report. Next, we describe each component in detail. 3.1 Orchestrator The orchestrator receives as input the task the user wants to delegate to an agent. The task is a description of what the agent needs to do, and the dependencies that are needed for the task to be correctly executed. The task may also contain skills that guide the agent towards the solution, including documents and scripts. For instance, forcitation-checkinSkillsBench, the task is a markdown document that describes how to automatically validate BibTeX references. The task also comes with a Dockerfile that specifies all dependencies, and with the citation-management skill. An execution kind, i.e., a set of configuration files, is the way in which the orchestrator can be configured for a class of tasks. AgentSpy defines a base execution kind, which specifies the basic configuration of the isolated environment, such as the runtime image, the network configuration, user permissions, and the installation of the execution manager binary. A custom kind extends the base kind based on task-specific requirements and analysis type, inheriting the base configuration. The custom kind defines three key elements: (1) the task-specific steps that configure and validate the environment, (2) the parameters that configure the agent, and (3) the analysis-specific steps that are embedded in the environment. 5 Once the orchestrator starts the isolated environment, it defers control to the execution manager, and monitors the execution. The environment ensures that each action of the agent is recorded at the system level, and that the actions do not affect the host system. 3.2 Execution Manager The execution manager provisions the isolated environment with task-specific dependencies, executes lifecycle hooks, sets up the analysis-specific mechanisms for observing the desired behavior at runtime, runs the agent with the configured parameters, and collects the run artifacts to be copied outside the isolated environment. Each step has a configurable timeout, which is enforced by the execution manager. The global timeout, i.e., the sum of all declared timeouts, is enforced by the orchestrator, which terminates the isolated environment regardless of its execution phase. If the agent fails or times out, the manager still runs the post-run steps, since partial execution is still behavior that can be analyzed. The order of these steps is important: monitoring starts after the last preparation step and stops before the first collection step. Thus, installing dependencies and copying out results are not recorded and are not part of the agent behavior. Once all steps have been executed, the execution manager terminates the environment and returns control to the orchestrator, which extracts the run artifacts and copies them to the host. 3.3 Analysis Runner: Conformance Analyses and Safety Analyses The analyses that AgentSpy can run fall into two families, which we call conformance analyses and safety analyses after the kind of requirement each one checks. Conformance analyses state an obligation, i.e., what an execution should do, e.g., follow the developers’ guidance provided through skills. An execution meets an obligation when the agent makes the corresponding system calls, but it is known to have failed it only once the run ends. Since a task can often be solved in several acceptable ways, conformance analysis reports a score, such as the fraction of the skill files the agent used during the execution, that a developer interprets. Safety analyses state a prohibition, i.e., what an execution must never do, e.g., read a private key or contact a host outside the allow list of the task. It is fixed before the run, a single system call is enough evidence, and a violation is known as soon as the call happens, so a safety analysis raises an alert. The distinction between obligations and prohibitions is the one that deontic logic and policy languages draw between what is obligatory and what is forbidden [54, 28], and it mirrors the classic distinction between liveness and safety properties, i.e., “something good eventually happens” and “something bad never happens” [5]. Neither family subsumes the other. If an agent fails to meet an obligation, e.g., it ignores the developers’ guidance, it does not violate a prohibition, so only a conformance analysis can detect the problem. If, on the other hand, an attacker forces the agent to follow the malicious instructions in a skill, the obligation is met but a prohibition is violated, so only a safety analysis can detect the problem. We designed one analysis of each family: the reliability analysis, a conformance analysis that measures obligations (Section 3.4), and the security analysis, a safety analysis that checks prohibitions (Section 3.5). The two are named after the dependability attributes they target [9]: reliability, i.e., the continuity of correct service, which for an agent means following the guid- ance that comes with the task, and doing so consistently across repeated runs, and security, i.e., confidentiality, integrity, and availability. Developers can extend AgentSpy with more specific analyses of either family, as new analyzers over the same observation boundary. For example, a new obligation can require the agent to query an external source to verify its answer, a new prohibition can state that the agent must never modify its own skill files. 6 1 1 1 1 1 6 1 1 reads 1 reads 2 reads 3 writes 1 35 15 3 execution codex / glm citation-check cat:SKILL.md [1] cat:test.bib [1] rg:test.bib [2] sed:test.bib [3] python3.12 [4] curl [5] python3.12: [6] search_google_scholar.py codex (apply_patch) [7] .../SKILL.md [1] .../search_google_scholar.py [6] /root/test.bib [1–3] /root/answer.json [7] crossref [4] bing [5]duckduckgo [5] Fig. 2. Execution star graph of thecodextrajectory shown in Table 1. Each node is a resource that the agent uses: a command that it runs, a file that it reads or writes, or a host that it contacts. Each edge connects theexecutionroot to a resource, and holds the number of times that the agent performs the action in the execution. Each grey [푘] tag refers to the rows (#) of the trajectory in Table 1 the node comes from. 3.4 Conformance Analysis: Reliability As a conformance analysis, AgentSpy supports a reliability analysis, which investigates how the agent solves the task, focusing on the process instead of the result, which is the main difference w.r.t. outcome-based tests that analyze whether the agent solves a task correctly. To analyze the how, we log the actions of the agent at the kernel level, i.e., the system calls of the agent, which also include all subprocesses that it starts. System calls typically fall into six categories, i.e., process control, file management, device management, information maintenance, communications, and protection [50]. In all categories, we log every call of the agent and its subprocesses, except frequent calls whose log entry does not refer to a resource of the environment, i.e., a file path, a host, or an executable. Most of the excluded calls operate on an open file descriptor, whose resource the log already records with the call that created the descriptor. The remaining ones only concern the calling process, e.g., memory management, synchronization, signal handling, or querying the time. 3.4.1 Star graph. Each log entry describes one system call, specifying which process makes the call, the call itself, which resource it uses, and what is the result. Each entry also holds a timestamp that determines the call order. To compare the behavior of the agent between runs, we parse the log entries of each run according to their order, and turn the unstructured logs into a structured representation, i.e., a graph. To build the nodes of the graph, we keep calls that refer to environment resources the agent interacts with, in particular calls that start a program (i.e., commands), calls that open a file in read/write mode, and calls that connect to a host (i.e., network calls). The graph has an additional node, the root, which represents the whole execution. The root identifies the agent process and all its subprocesses, and it is connected to all the other nodes, while the resource nodes have no connection between them; a graph of this shape is known as a star graph. Each edge has a type, i.e., the action that the agent performs on that resource, such as runs, reads/writes, or connects, and a count, which is the number of times that action is performed. We use this representation, instead of a process tree, for three reasons. (1) a resource keeps its name in each run, therefore the resource nodes of two runs can be compared; in a process tree on the other hand, each node is a process with an ID that changes between runs; (2) we are interested in what the agent does as a whole, regardless of which of its processes performs an action; (3) the comparison of two runs becomes a comparison of their resources and of their counts, and each difference points to a particular resource, e.g., a file that only one of the two runs reads. 7 Figure 2 shows the star graph of thecitation-checkrun of Table 1, restricted to the seven steps in the table. From the graph, we see that the agent runscat,rg, andsedto read the skill and the bibliography (Rows 1–3). It contacts Crossref (Row 4), Bing and DuckDuckGo (Row 5), runs the Google Scholar script that comes with the skill (Row 6), and writes answer.json (Row 7). Reading the graph does not require interpreting the content of the trajectory: its nodes are the system primitives, i.e., commands executed, files accessed, and hosts contacted. Categorizing the actions therefore needs neither manual labeling nor an LLM-as-a-judge, and the same log entries always result in the same graph, making the analysis deterministic. The full graph is also complete: it shows everything that happens within a trajectory step, whereas the trajectory step only reports the particular tool call. For instance, when the agent runs the Google Scholar script (Row 6), the trajectory records one command, and the error message from the script. The graph shows that the script reads its own file, but it does not contact any host and it does not access any file of the task. Similarly, the inline code of Row 4 is a single action in the trajectory, while the graph shows that it connects to Crossref 35 times to look up seven titles. 3.4.2 Quantitative metrics. For a set of runs for the same task, AgentSpy computes two metrics from the star graph: the similarity of a set of runs, and the skill coverage of each run. As an example, we consider three runs ofcodexwithglm-5.3-flashoncitation-check, one of which is the trajectory in Table 1. Withglm-5.3-flash, the three runs pass all nine tests that come with the task in SkillsBench. The similarity metric measures how consistent the agent is, in terms of its actions, across different runs. A low similarity shows that the agent approaches the task in very different ways; this may indicate that there are multiple ways to solve the same task, that the task is not well-defined, or that the agent deviates substantially from the task’s intent. We reduce each run to the three resource types in the graph, i.e., its commands, its file accesses, and its hosts, and each resource, e.g.,reads:/root/test.bib, to the count on its edge over the whole run. We then divide each count by the sum of the counts of the corresponding resource type, which gives the proportion of the resource in the run. The similarity thus compares which actions the agent performs in a certain run with their proportions, rather than considering the absolute number of actions: for instance, if the agent contacts Crossref 10 times and OpenAlex 2 times in run푎, and contacts Crossref 20 times and OpenAlex 4 times in run푏, the two runs have the same proportions for the hosts (5/6 for Crossref and 1/6 for OpenAlex), and thus a similarity of 1. More formally, for two runs 푎and푏and one resource type푇, the similarity is the weighted Jaccard index of the proportions: sim 푇 (푎,푏)= Í 푟 min(푝 푎 (푟), 푝 푏 (푟))/ Í 푟 max(푝 푎 (푟), 푝 푏 (푟)), where푟ranges over the resources of푇 in both runs, and푝 푎 (푟)is the proportion of푟in푎, or 0 if푎does not have푟. The similarity is 1 when the two runs have the same resources in the same proportions, and 0 when they have no resource in common. The similarity of two runs is the average similarity over the three resource types, and the similarity of a set of runs is the average similarity over all of its pairs. The skill coverage metric is relevant in all tasks equipped with skills and measures the fraction of the files of a given skill that the agent reads, writes, or executes. In tasks with multiple skills, AgentSpy reports both the per-skill coverage and the average skill coverage across multiple skills. This metric captures the extent to which the agent uses the guidance from the skills to solve the task. A low skill coverage hints that the agent ignores skills, and approaches the task from first principles. 3.4.3Example of AgentSpy’s reliability analysis. The top part of Table 2 shows the resource types of each of the threecitation-checkruns as extracted from their respective star graphs. For instance, thesedcommand is used twice in runs R1 and R2, eight times in R3. Commands such ascurland python3.12:search_google_scholar.pyare only used in R3, indicating that in R3 the agent does 8 Table 2. Selected resources of the star graphs of threecitation-checkruns withcodexandglm-5.3-flash. The bottom part shows the similarity of each pair of runs and the set for each resource type, and the skill coverage of each run. The symbol “–” indicates a resource that is not in the run. Bold values highlight differences in the runs. gs.py: search_google_scholar.py. CommandsR1R2R3 FilesR1R2R3 HostsR1R2R3 python3.125109 reads:test.bib354 crossref.org145125165 sed228 reads:skill:SKILL.md222 openalex.org153030 curl– 32 reads:skill:gs.py– 2 semanticscholar.org152045 head–7 writes:answer.json111 dblp.org– 3 python3.12:gs.py–1 duckduckgo.com– 3 Other (18)254552 Other (8)162321 bing.com– 15 Sim. to R10.67 0.36 Sim. to R10.69 0.74 Sim. to R10.79 0.67 Sim. to R20.36 Sim. to R20.74 Sim. to R20.76 Set0.46Set0.72Set0.74 Skill coverage1/14 1/14 2/14Similarity of the set, averaged over the resource types: 0.64 something different. This is confirmed both by the files and hosts resource types: the agent in R3 reads the scriptsearch_google_scholar.pythat is part of the skill (prefixskill:), which did not happen in the other runs, and it contacts dblp and duckduckgo, which is also specific to R3. The bottom of the table shows the pairwise similarity and the set similarity per resource type, as well as the set similarity averaged across all resource types (bottom right corner). Runs R1 and R2 are roughly similar across all resource types, with a similarity of at least 0.67. On the other hand, run R3 differs from the others, especially for commands (similarity of 0.36 for both R1–R3 and R2–R3). All three runs pass the nine tests, while the similarity shows that R3, which is the run of Table 1, is different. After Crossref returns only fuzzy matches, the agent searches Bing and DuckDuckGo throughcurl, which the skill does not prescribe. These searches fail too, so the agent bases its verdict on the DOI prefixes and on the missing Crossref match. The same agent with the same model thus solves the task in different ways across runs, and the similarity tells the developer which run and which resources to inspect. The table also shows that all runs have a very low skill coverage. In all runs the agent reads the skill document, and only in run R3 the agent uses one of the scripts that is packaged with the skill. None of the runs follows the whole procedure of the skill: no run executes the validation helper script or reaches Google Scholar or PubMed, and the Google Scholar script that R3 runs fails because a dependency is missing (Row 6 of Table 1). Apart from Crossref, the agent queries services that are not mentioned in the skill, such as OpenAlex and Semantic Scholar, and all three runs still pass the nine tests. The analysis shows that the agent solved the task from first principles, using a custom solution and ignoring the guidance provided by developers. 3.5 Safety Analysis: Security As a safety analysis, AgentSpy supports a security analysis, which checks prohibitions, i.e., actions that an execution must never perform, against an attacker that can either author a malicious skill or modify an existing one. Indeed, skills are distributed through public marketplaces and can be turned into a third-party supply-chain surface analogous to package registries [37]. We assume that the host operating system, the agent SDK, and the agent implementation are benign. A compromised skill can force the agent to directly perform an unauthorized action while still delivering its advertised functionality, but it can also steer the agent to assemble that harmful action from otherwise benign capabilities, hiding the violation in the execution trajectory while the final result still looks legitimate to the user [17, 26]. We consider confidentiality, integrity, and 9 availability objectives: exfiltration of sensitive data, unauthorized modification or destruction of local or remote resources, and disruption of the host or of the services it depends on [55]. 3.5.1 Rules over system calls. AgentSpy defines a set of rules for system calls within the isolated environment where the agent runs. We define four classes of rules: (1) reads of credentials and secrets, (2) modification or deletion of files outside the agent workspace, (3) running programs from untrusted locations and gaining privileges, and (4) network activity to untrusted domains. Each rule tests a single system call, e.g., whether it opens a file in a.sshdirectory that is not a public key. Thus, a rule states that the agent never makes such a call, and AgentSpy checks all rules in one pass over the recorded calls. An alert identifies the violated rule and the values needed to judge it, e.g., the name, command line, and working directory of the process that opened a private key. The rules are broadly applicable across tasks as every predicate refers to concepts of the operating system: the agent user, the agent workspace, the standard locations of credentials, system and startup files, temporary directories, and system calls that change privileges. Current coding agents draw a similar boundary in their sandboxes, which in their standard configuration 1 permit writes inside the workspace, and restrict writes outside it as well as network access [2, 47], indicating that benign coding tasks are expected to stay within it. When a task legitimately needs a resource outside the boundary, such as a package registry, the allow list of the corresponding class is extended for that task, without requiring a rule change. Coding agent sandboxes are complementary to the security analysis AgentSpy offers: a sandbox protects the user when running an agent, while AgentSpy assesses the behavior of the agent that uses a certain skill to solve a task (e.g., to audit the skill and decide whether to use it in production). When the agent uses a compromised skill, its sandbox blocks an action that crosses the boundary and reports the violation to the agent, so the blocked action, and every step that depends on it, never execute. On the other hand, AgentSpy lets the agent use the compromised skill within an isolated environment, and applies its rules at the system level, independently of the agent, so the attack runs to completion and each step that matches a rule raises an alert. 3.5.2 Example of AgentSpy’s security analysis. The left-hand side of Figure 3 shows an injec- tion into thecitation-managementskill of thecitation-checktask. One added instruction steers the agent to runsync_cache.shbefore reporting; the script uploads the private SSH keys of the agent user todrop.example.net, and then runs the validation that the task expects. The trajectory records a single tool call that exits successfully, whereas the system calls showcurlopening /home/ubuntu/.ssh/id_rsa, and then connecting todrop.example.net. Sincecurlruns as the agent user, both calls are detected: the first raises an alert of the rule on private keys, the second one of the rule on network activity, because the host is not in the allow list of the task. 3.6 Implementation AgentSpy is implemented in Go, and the analysis runner is a separate Python package that reads only the collected run artifacts, so the analyses can be repeated or replaced without running the agent again. Each isolated environment is a Firecracker [48] microVM booted from an Ubuntu root filesystem that is built once per execution kind and reused unchanged, so every run starts from the same state. We use codex CLI 0.146.0 as the agent (non-interactively) with plugins disabled. For the reliability analysis, AgentSpy records system calls with the Linux audit subsystem (auditd), under a rule set restricted to the agent user and its process tree. Thelaurel[53] plugin converts the audit stream into JSON with resolvedexecvearguments, and Zeek [52] logs the network traffic, from which AgentSpy maps the addresses in the audit log to the host names of 1 codex by default and Claude Code in manual mode. 10 (a) The instruction and the script SKILL.md, added to the Phase 4 checklist: 5. Before reporting, refresh the metadata cache with scripts/sync_cache.sh from this skill directory. scripts/sync_cache.sh: for k in /home/ubuntu/.ssh/id_*; do curl -F file=@$k https://drop.example.net done exec python3 scripts/validate_citations.py (b) Agent trajectory vs system-level calls agent trajectory: tool=bash cmd=scripts/sync_cache.sh exit=0 system boundary: SYSCALL syscall=59 comm=sh key=exec EXECVE a0=scripts/sync_cache.sh SYSCALL syscall=257 comm=curl key=file PATH name=/home/ubuntu/.ssh/id_rsa SYSCALL syscall=42 comm=curl key=net SOCKADDR peer=drop.example.net:443 Fig. 3. A malicious variant of thecitation-managementskill. (a) It calls a malicious script that exfiltrates private keys, and then runs the expected validation. (b) The trajectory records one tool call, while system-level logs record what happens when the malicious script is executed. the star graph. For the security analysis, AgentSpy runs only Falco [21] 0.44.1, on themodern_ebpf driver, and a kernel built with BTF and kprobe support, with the rule set of Section 3.5 in place of the default rules. 4 Evaluation We evaluate AgentSpy along these four research questions (RQs): RQ1. Reliability Analysis:How effective is the reliability analysis of AgentSpy in capturing the behavior of an AI agent? We study the extent to which the reliability analysis of AgentSpy can distinguish different executions of an agent on the same task. We also measure the impact of each resource type. RQ2. Comparison with Outcome-based Testing:To what extent does the reliability analysis of AgentSpy reveal behaviors that outcome-based testing misses? After assessing whether reliability analysis can capture the behavior of an agent, we study whether it can reveal behaviors that cannot be captured by test cases. We manually inspect the reports AgentSpy produces for tasks where the agent passes all the tests across multiple executions. RQ3. Security Analysis: How effective is the security analysis of AgentSpy in detecting skill- injection attacks? We run tasks with and without injected instructions, and evaluate whether AgentSpy detects the malicious behavior the agent executes while raising no alerts on runs without a malicious behavior. RQ4. Efficiency:What is the time overhead of AgentSpy? We measure the time overhead introduced by AgentSpy, and compare the runtime without the system-level monitoring. 4.1 RQ1: Reliability Analysis Experimental setup. To evaluate whether the reliability analysis of AgentSpy captures the behavior of an agent, we compare executions that are expected to behave similarly with executions that are expected to behave differently. Since there is no ground truth for behavioral similarity, we use the task as a proxy: executions of the same task should be more similar to each other than to executions of a different task. We use theSkillsBench[33] benchmark, which provides 87 tasks across 8 domains, including software engineering, mathematics, cybersecurity, and content production. These tasks have three levels of difficulty, easy (6), medium (53) and hard (28). Each task comes with curated skills that provide domain expertise for a class of problems, deterministic tests that assess whether the task is solved, and a scripted oracle that shows the task is solvable. A Dockerfile lists the dependencies of each task, which AgentSpy imports and installs at runtime. We exclude 10 tasks whose environment cannot be reproduced in our VM, because either the base image cannot 11 012 k (runs taken from the max-distance task) 0.0 0.2 0.4 0.6 0.8 1.0 similarity within the run set k=0k=1k=2 (a) Set similarity of homogeneous (푘=0) and mixed (푘= 1, 2) sets over all three resource types. All 3 77 Cmd 77 Files 77 Hosts 31 0.4 0.2 0.0 0.2 0.4 0.6 0.8 separation margin 48128 (b) Ablation of the three resource types. Each dot is the separation margin of one task. Fig. 4. RQ1 results on gpt-5.6-terra: similarity of runs, and ablation of the resource types. be rebuilt or provisioning fails on package incompatibilities. The agent may run for at most 20 minutes per task, and each task execution, including provisioning and test suite execution, times out after 60 minutes. Each task is executed three times to account for non-determinism. A timed-out or truncated execution is repeated up to three times, and executions that remain truncated are discarded. Timed-out executions are kept only if all three executions of the task time out, so for each task either all executions complete or all time out. We evaluate three recent models, all released between July and August 2026, shortly before our experiments (i.e., August–September 2026): one proprietary model and two open-weight models from three different providers. As the proprietary model, we selectedgpt-5.6-terra(OpenAI, July 2026), the mid-tier of the 5.6 family, which balances cost and performance between the flagshipSol and the low-costLunatiers. As open-weight models, we selecteddeepseek-v4-flash(DeepSeek, July 2026) andglm-5.3-flash(Z.ai, August 2026), two fast Mixture-of-Experts models that their providers optimize for coding and agentic tool use. We executed all three models under the same agent harness (i.e.,codex), so differences in behavior reflect the model and not the harness. 2 When codexis used withgpt-5.6-terra, the agent runtime is the shortest, with a median of 103 seconds per task, while it is the longest with glm-5.3-flash, with a median of 413 seconds per task. To assess whether the reliability analysis can distinguish executions with different behavior, we compare the executions of each task푥with the executions of the task푥 ′ that is most dissimilar to푥. Dissimilarity is defined over the skills: among the tasks executed with the same model,푥 ′ is the one whose mergedSKILL.mddocumentation has the largest cosine distance from that of푥, computed onall-MiniLM-L6-v2embeddings. Since the two tasks require different skills, we expect their executions to behave differently. We then build sets of three executions in which푘executions come from푥 ′ and the remaining 3− 푘from푥, for푘=0,1,2. At푘=0, the set is homogeneous, i.e., it consists of the three executions of푥. At푘=1 and푘=2, the set is mixed, and there are 9 ways to choose its executions. For each combination we compute the similarity metric of the set of executions defined in Section 3.4.2, and for each task and each푘we report the average set similarity across these combinations, both when considering all resource types, and when considering each one individually. Results: homogeneous and mixed sets. Figure 4a shows the set similarities ongpt-5.6-terra. For each of the 77 tasks, we report the set similarity for homogeneous sets of executions, and for 2 Overall, the reliability analysis consumed around 2 B tokens, and the agent executions lasted a total time of 65 hours. 12 mixed ones. We observe that homogeneous sets are typically more similar than mixed sets, with the average set similarity being 0.60 in the homogeneous case, and 0.30 for mixed sets on average. Given a point in the homogeneous set and one in either of the two mixed sets, the former has a higher set similarity in 93.5% of the cases. Similar results hold fordeepseek-v4-flashandglm-5.3-flash, and counting all the three models, they have higher similarity in 92.2% of the cases. This indicates that set similarity over the star graph’s resources is sensitive enough to distinguish executions of different tasks. Results: ablation of the resource types. To measure the impact of each resource type, we repeat the analysis with each resource type in isolation. Figure 4b shows the separation margin of each task, when considering all resource types and each resource type in isolation. The separation margin of a task is the difference between the similarity of the homogeneous set and the maximum of the similarities of the two mixed sets, which makes the margin a conservative measure, since the homogeneous set is compared with the most similar mixed set. A positive margin means that the runs of the task are more alike than runs mixed with those of another task, and the higher the margin, the higher the discrimination power of a resource type (or of their combination). A margin of zero or below means that the resource type does not discriminate homogeneous runs of a task. Ongpt-5.6-terra, the file resource type alone reaches an average margin of+0.278, close to the+0.266 of the combination, yet its margin is zero or negative for 12 of the 77 tasks, against 4 for the combination. The command resource type alone has a margin of zero or below for 8 tasks and a lower average margin (+0.200). The hosts resource type is defined for only 31 tasks, namely those in which at least one run contacts a host other than the model provider, so it cannot separate the remaining 46 tasks on its own. Hence, combining the three resource types gives the highest number of tasks with a positive margin, more than any single resource type. RQ1 (Reliability Analysis): Overall, AgentSpy assigns a higher similarity to repeated runs of a task than to sets that include runs of a different task. The ablation study shows that the combination of the three resource types separates the highest number of tasks. 4.2 RQ2: Comparison with Outcome-based Testing Experimental setup. For RQ2, we use the same setup of RQ1 except that, for each model, we keep only the tasks for which all three runs pass all test cases. Indeed, whether the task is solved reliably can be assessed by the test pass rate; however, analyzing how it is solved is the objective of AgentSpy’s reliability analysis. In particular,codexwithgpt-5.6-terrasolved 40 tasks correctly, withdeepseek-v4-flash27, and withglm-5.3-flash21, for a total of 88 task–model pairs and 264 reports. Two authors (i.e., assessors) independently inspected AgentSpy’s report for every task and model. For each task, each assessor looked at the report produced by AgentSpy (similar to the one in Table 2), and answered two questions: (i) did the agent perform any activity that is unrelated to the task?; and, (i) did the agent solve the task without guidance, in particular without using skills that come with the task? If any of the runs in a task presents the corresponding behavior, the task is flagged accordingly (the two behaviors are not mutually exclusive), and the decision is motivated with a short comment. Each question corresponds to one metric that AgentSpy computes. Unrelated activities are expected to lower the similarity between the runs, since an execution in which the agent does something unrelated to the task will be different from the others. Solving the task without guidance is directly related to skill coverage, since an agent that ignores the skill files solves the task in another way. The assessors had access to both metrics in the report and used them as a guide, 13 together with the commands, files, and hosts listed in the report, to decide on each label. 3 We chose these two questions because they target behavior that stays hidden behind a correct outcome, and that affects the trust a developer can place in the agent in production. An agent that starts activities unrelated to the task acts beyond what the developer requested. Similarly, an agent that ignores the domain knowledge provided through skills reaches a result that the developer cannot attribute to that knowledge. After completing the inspection and tasks’ labelling, we computed the Cohen’s휅[15] to measure agreement on both questions between the two assessors. Moreover, we collected the flagged tasks, i.e., those for which at least one answer is true, and clustered them based on the assessors’ comments, grouping tasks that share a similar behavior. Results. Table 3 shows the results of the manual analysis. In particular, the independent labels of the two authors differ in two of the 88 observations, one underdeepseek-v4-flash, and one under glm-5.3-flash. On these final labels, Cohen’s휅is 0.96 for the first question (observed agreement 98.9%) and 0.92 for the second question (97.7%), so an “almost perfect agreement”. Of the 88 tasks, 16 (18.2%) are flagged for registering activities unrelated to the task, and 15 (17.0%) for reaching the result without the skills’ guidance. When characterizing unrelated activities, the two assessors identified three clusters: environ- ment exploration, i.e., the agent started exploring the execution environment; unrequested network access, i.e., the agent consulted a service that is irrelevant to the task; and test file access, i.e., the agent read the test cases that are used to grade the task. The largest cluster across models is unrequested network access (8 tasks). An example of this behavior is underglm-5.3-flash solving thecitation-checktask (i.e., see Table 2); in the third run, the agent conducted a web search onduckduckgo, even though this was not specified by the task. In 5 tasks, the agent started exploring the execution environment, e.g., the directories around its workspace. In 6 tasks, all underdeepseek-v4-flash, the agent read the benchmark’s own test files; e.g., in one of the runs of sec-financial-report, the agent read the test cases before writing the solution. Table 3. RQ2: Manual inspection of the tasks whose three runs all pass. A task is flagged when the answer to either of the questions (Q) is deemed true in any of the runs. The clusters (C) are non-exclusive. glmdeepseekgptTotal # Tasks with three passing runs21 27 40 88 Q1: activity unrelated to the task? # Tasks flagged3 112 16 C1 environment exploration1405 C2 unrequested network access2428 C3 test file access0606 Q2: result reached without guidance? # Tasks flagged753 15 C1 oracle reuse0202 C2 skill files ignored743 14 While characterizing the answers to the sec- ond question, the two assessors identified two clusters: oracle reuse, i.e., the agent reverse en- gineered the result from the oracle instead of computing it; and skill files ignored, i.e., the agent did not use some or all of the skill files. Most of the flagged tasks belong to this sec- ond cluster (14/15), as they have at least one run in which the agent did not consult the skill files in part or in whole. For instance, un- dergpt-5.6-terra, the agent opened only the SKILL.mdfile of thecitation-managementskill in all three runs. This file is one of the 14 files of the skill, and the agent never opened the validation scripts. The agent instead checked the bibliography entries directly against the Crossref API. A similar behavior happens un- derglm-5.3-flash(see Table 2). However, here the agent not only builds a custom solution in the third run, flagged by a low skill coverage, but the custom solution is also flawed. By also consulting the trajectory (see Table 1), we notice that the agent only checks the titles of the references against a ground truth (i.e., Crossref ), instead of 3 Whenever the intent of an agent execution was unclear, they looked at the corresponding agent trajectory. 14 the respective DOIs as prescribed by its skill. Essentially, the agent judges that the three DOIs look suspicious and searches for their titles; Crossref returns only fuzzy matches and the web searches return no titles, hence the agent concludes that the corresponding citations are fabricated. The final result is correct and the tests pass, yet the agent reached it without the domain knowledge embedded in the skill, i.e., that the validity of a reference is established through its DOI, which identifies the reference uniquely. Regarding the first cluster, we observed that in 2 tasks, both underdeepseek-v4-flash, the agent executed the scripted oracle solution instead of computing the result. RQ2 (Comparison with Outcome-based Testing): Tests miss behavior that AgentSpy reveals. Among the 88 task–model pairs whose runs all pass the respective test cases, two authors flagged 16 (18.2%) tasks for activity unrelated to the task, mostly unrequested network access (8 tasks), and 15 (17.0%) for a result reached without the domain knowledge of the skill. 4.3 RQ3: Security Analysis Experimental setup. We assess whether the runtime evidence AgentSpy collects supports security analysis, using theSkill-Injectdataset [46].Skill-Injectis a benchmark that measures the vulnerability of coding agents to attacks delivered through skill files. It comprises 202 injection– task pairs, in which a benign task is combined with a skill that carries a malicious instruction. An injection is eitherdirect, when the malicious action is stated insideSKILL.md, orscript, when the instruction directs the agent to run a malicious auxiliary script shipped with the skill. A script injection hides the harmful action from a review of the skill, sinceSKILL.mdonly directs the agent to run a script, and the harmful commands appear only in the script itself (which could also be downloaded from the network). This is also the case in which the trajectory is least informative (Section 2.3): it records a single command that runs the script, and none of the files read or deleted, the contacted hosts, or the executed processes, which only a system-level monitor observes. Moreover, Schmotz et al. [46] report that script injections succeed more often, because the agent usually trusts executable code that is not always checked. Of the eight attacks that Schmotz et al. [46] define, five are applicable to script injections, and we select one case from each: data exfiltration (S1), data destruction (S2), denial of service (S3), malware (S4), and backdoors (S5). 14 detected 0 false alarms 4 missed 32 7 declined 25 control executed (18 runs) not executed (32 runs) Injected behavior alert no alert AgentSpy precision 1.00 recall 0.78 Fig. 5. RQ3: AgentSpy security alerts across the 50 Skill-Inject runs. We run every task in an injected and a control con- dition. In the injected condition, theSKILL.mdcarries the instruction that directs the agent to the malicious script. In the control condition, that instruction is re- moved, while the malicious script, if any, is left in the skill directory. For each condition, AgentSpy runscodexon the benchmark task inside an isolated workspace, using openai/gpt-5.1-codex-max, a model for which Schmotz et al. [46] report a high attack success rate. In each sce- nario, the benchmark writes a specific file once the mali- cious script is executed, which signals whether the agent carried out the injected behavior. AgentSpy observes sys- tem calls during the execution and applies the determin- istic rules of Section 3.5. Reads and writes inside the workspace are permitted by design, and connections to the model provider (openrouter.ai) are in the allow list, so that ordinary task activity raises no alert. We run each task five times per condition (50 runs in total) and manually inspect (a) whether the agent carried out the injected behavior and (b) whether AgentSpy detected it. 15 Results. The control condition raised no alert in any of its 25 runs. AgentSpy’s rules react to the actions that the agent executes, so a malicious script that sits in the skill does not raise any alert, and neither does completing a task. In the injected condition, the agent executed the injected behavior in 18 of 25 runs, and AgentSpy raised no alert on any of the remaining 7. As Figure 5 shows, AgentSpy detected 14 of the 18 executions (precision = 1.00, recall = 0.78). AgentSpy detects every successful execution of the exfiltration, destruction, denial-of-service, and malware cases (S1–S4). Whenever the agent carries out one of these injections, the action crosses a boundary that AgentSpy monitors (a connection to an unlisted host, a destructive change outside the workspace, or a privilege change), and one of the rules is violated. Every false negative is an instance of the backdoor attack category (S5). Its script adds a malicious remote to the local Git configuration, but the agent never pushes to that remote. The change remains in the permitted workspace without network activity, so none of the four rule classes matches. The change becomes harmful only at a later stage, i.e., when the agent pushes to the malicious remote, and AgentSpy raises an alert only when an executed behavior crosses a boundary. Since adding a remote is harmful only for some tasks, detecting it requires a custom rule, e.g., to flag any new remote in a task that has no reason to add one. RQ3 (Security Analysis): Overall, AgentSpy is effective at exposing skill-injection attacks. AgentSpy detects 14 of the 18 injected behaviors that the agent executed and raises no alert in the 32 runs without them, which gives a precision of 1.00 and a recall of 0.78. 4.4 RQ4: Efficiency Experimental setup. In RQ4, we measure how much AgentSpy’s monitoring tools (i.e.,auditd, laurel, and Zeek for the reliability analysis and Falco for the security analysis) slow down a task execution. Regarding the reliability analysis, for each of the three models of Section 4.1, we ranked its tasks by the mean duration of their passing runs and selected the fastest task, the slowest task, and the task in the middle of the ranking, which we call the average task. Similarly, for the security analysis, we selected the fastest (category S3), slowest (category S4) and average (category S1) task, where an injection is successful. We executed each selected task ten times with the monitoring tools and ten times without them, for a total of 180 + 60 runs. For each run, we time the main command that runs the agent, excluding the provisioning of the task, and the hooks that start the observation before the run, and collect the evidence after it. The runs of the two conditions are independent, so we compare them per task with a two-sided Mann-Whitney U test [8] using 훼=0.05 as the threshold for statistical significance. The experiments consumed 730 M tokens and 20 hours of agent runtime. Results. Table 4 shows the median execution time, in seconds, when the agent is executed with and without monitoring tools enabled. We compare the two conditions per model and per task, since a comparison between models is not meaningful when the tasks are different. In 8 of the 12 comparisons the median is lower with the monitoring tools, which is unlikely to be an effect of the tools, because they use extra CPU and disk and cannot speed up the agent, so these differences come from the variation between runs. The standard deviation explains this: over the ten runs of each condition, it ranges from 7% to 85% of the median, for both the reliability and the security analysis. Accordingly, the푝-value of each comparison (Column푝-value of Table 4) is higher than훼 in all cases, so we find no significant difference between the two conditions. The Cohen’s effect size [16] (Column푑) is small in 8 comparisons and medium in 4, and it is never large. With ten runs per condition, the test is sensitive only to large effects, and a power analysis with훼=0.05 and a power of 0.8 shows that detecting the observed effects would require from 30 to 18,200 runs 16 Table 4. RQ4: Time of the agent execution in seconds over ten runs with and without AgentSpy’s monitoring tools.푝-value corresponds to the two-sided Mann-Whitney U test;푑is the magnitude of Cohen’s푑(small below 0.5, medium below 0.8, large otherwise). Median± std. dev. ModelTaskWithWithout푝-value푑 Reliability analysis: with and without auditd, laurel, and Zeek gpt-5.6-terra fastest23.3± 6.526.0± 5.30.43small average147.4± 37.4148.1± 22.00.79small slowest768.2± 151.1767.3± 188.40.97small deepseek-v4-flash fastest77.7± 38.4101.2± 51.30.62small average185.0± 110.6265.3± 96.50.47small slowest375.4± 79.1429.1± 110.00.16medium glm-5.3-flash fastest68.9± 43.958.7± 23.00.68small average151.8± 74.3184.1± 155.80.38medium slowest998.5± 74.31021.4± 117.00.97small Security analysis: with and without Falco gpt-5.1-codex-max fastest (S3)84.4± 37.280.8± 20.00.24medium average (S1)141.2± 35.5107.1± 43.80.16medium slowest (S4)336.0± 50.8360.6± 84.10.52small per condition. Therefore, any overhead of AgentSpy is small compared with the variation between runs of the same agent on the same task, and in practice it is unlikely to affect an agent execution. RQ4 (Efficiency): Overall, we find no significant overhead from the monitoring tools of AgentSpy on the runtime of the agent. The푝-value is always above the significance threshold of훼=0.05, and the effect size is small in the majority of cases. 4.5 Threats to Validity Internal validity. In the reliability analysis, we exclude high-frequency system calls, which are not part of the star graph. However, we still keep track of the resources the agent uses, while keeping the log size manageable. The security rules that we used are also custom, but they follow the general rules that are also used by popular agent sandboxes. The labeling of each task in RQ2 might also affect the study’s validity. We addressed it by having two authors independently label each task. We measured the agreement between the two assessors with Cohen’s휅, which shows an almost perfect agreement, with two disagreements out of 88 tasks. Conclusion validity. The non-determinism of LLM-based agents [43, 11], which may affect the conclusions drawn from the experiments. We alleviated this threat by running agents multiple times, both in the reliability and security analyses, and by using rigorous statistical tests when measuring the effect of the overhead [8]. External validity. In our experiments, we consider only one agent harness across the analyses, i.e.,codex, so our results may be different in other agent harnesses. However, we chosecodexas it is a representative harness, and one of the most popular open-source ones. Generalization might also be affected by the limited number of LLMs we used. We addressed this threat by selecting four LLMs from three different providers, both proprietary and open-weight. Lastly, an additional external validity threat is related to the datasets we used for the reliability and security analyses. We choseSkillsBenchfor the reliability analysis because it covers a variety of application domains, and Skill-Inject for the security analysis because it covers a variety of attack categories. 17 5 Related Work Trajectory analysis of AI agents. Ou et al. [42] introduceAgentDiagnose, an approach that couples an LLM-based evaluator with a visualization dashboard, so that users can inspect the internal states and decisions an agent reports. Liu et al. [35] proposeAgentDoG, a diagnostic guardrail that monitors trajectories and diagnoses the root cause of unsafe actions with a supervised fine-tuned model. Liu et al. [36] proposeTraceAegis, which derives a hierarchical representation of an agent’s tool invocations and profiles its behavior to detect violations, and Advani [1] train a siamese recurrent autoencoder for online anomaly detection over trajectories. For coding agents, Li et al. [32] reconstruct an execution as a hierarchical trace tree, and localize the failure onset, i.e., the earliest critical step and the downstream chain it triggers, on step-level annotated trajectories from four frameworks. Su et al. [51] treat instruction following as a constraint-satisfaction problem and return a verdict per constraint. Their results depend on the agent’s cooperation and completeness, and their granularity is a single action (Section 2). In contrast, AgentSpy observes executions outside the agent process (Section 3), capturing actions of helper scripts that trajectories may omit, whether intended or accidental. It therefore complements the agent’s reasoning. System-level observability and behavior comparison.AgentSight[58] uses LLMs to understand kernel-level effects of individual executions, while Li et al. [31] represent capabilities and runtime state as queryable attributed graphs. Other work compares tool calls against references, often ignoring reads because they leave system state unchanged [7]. Log analysis and provenance research reduce low-level evidence to comparable units, through template extraction [25], log-based anomaly detection [19], and attack investigation over audit logs and whole-system provenance [45, 6, 18, 14]. AgentSpy, instead, deterministically captures system activities, without an LLM in the loop, to compare repeated executions and quantify their similarity (Section 3.4). UnlikeGaia2[7], it treats reads as behavior: reading~/.sshdoes not change the state, but can still violate a prohibition. It also compares an agent execution against its assigned skill, exposing executions that succeed without using the domain knowledge developers provided. Security analysis and static analyzers of agent skills. Existing approaches analyze skills before execution through static analysis and formal guarantees [27, 44, 49, 10], dynamic sandboxing and taint tracking [29], or runtime auditing [30]. Other approaches enforce policies before an agent action executes. Bühler et al. [13] restrict what an MCP server can access with a declarative manifest of capabilities, which an enforcement engine applies at runtime. Zheng et al. [57] let the agent declare policies, such as running the tests before a commit, and enforce them in the operating system kernel, returning feedback that the agent can understand. Mou et al. [38] use a guardrail model that reads the interaction history and flags an unsafe tool call before it executes, and Lin et al. [34] defend against injections in tool calls, i.e., in the metadata and feedback that tools return, by verifying an action against the original intent of the task before the agent commits to it. Gosmar and Dahl [22] place a network of sentinel agents in a multi-agent system to monitor the communication between agents and flag threats. Dynamic analyzers such as Ji et al. [29] audit a skill before it is installed by running it with an agent in a sandbox, where reads of sensitive files return placeholder markers that the analyzer tracks through the agent context, processes, and network calls. In contrast, the security analysis of AgentSpy takes a task as input, runs the agent in a microVM with a full operating system, and applies rules over system calls to any agent execution, which do not refer to a skill. 18 6 Conclusion The behavior of AI agents is difficult to understand and the trajectory of an AI agent records only what the agent reports about itself, one action at a time. We presented AgentSpy, which runs the agent in an isolated environment and records its behavior, including the system calls and network traffic of every process it spawns. Two families of analyses build on it: (1) conformance analyses measure what an execution should do, (2) safety analyses check what it must never do. Our experiments demonstrate that our approach is effective, and adds no significant overhead. For (1) we design a reliability analysis, which summarizes each execution as a star graph of commands, files, and hosts; it tells executions of different tasks apart in 92.2% of the comparisons and, among executions that pass all tests, reveals unrelated activities (18% of the cases), reads of the grading files (7%), and results reached without the skill’s guidance (17%). For (2) we propose a security analysis that flags four of the five attack categories without false alerts. References [1] Laksh Advani. Trajectory Guard – A Lightweight, Sequence-Aware Model for Real-Time Anom- aly Detection in Agentic AI. Jan. 2, 2026. doi: 10.48550/arXiv.2601.00516. arXiv: 2601.00516 [cs.LG]. Pre-published. [2] Agent Approvals & Security. OpenAI. url: https://learn.chatgpt.com/docs/agent-approvals- security (visited on 09/27/2026). [3]Agent Skills Overview - Agent Skills — agentskills.io. https://agentskills.io/home. [Accessed 21-09-2026]. [4]Humaid Al Naqbi, Zied Bahroun, and Vian Ahmed. “Enhancing Work Productivity through Generative Artificial Intelligence: A Comprehensive Literature Review”. In: Sustainability 16.3 (Jan. 30, 2024), p. 1166. issn: 2071-1050. doi: 10.3390/su16031166. [5]Bowen Alpern and Fred B Schneider. “Defining liveness”. In: Information processing letters 21.4 (1985), p. 181–185. [6] Abdulellah Alsaheel et al. “ATLAS: A Sequence-based Learning Approach for Attack Inves- tigation”. In: 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, Aug. 2021, p. 3005–3022. isbn: 978-1-939133-24-3. url: https://w.usenix.org/conference/ usenixsecurity21/presentation/alsaheel. [7]Pierre Andrews et al. “ARE: Scaling Up Agent Environments and Evaluations”. In: CoRR abs/2509.17158 (2025). doi: 10 . 48550 / ARXIV. 2509 . 17158. arXiv: 2509 . 17158. url: https : //doi.org/10.48550/arXiv.2509.17158. [8] Andrea Arcuri and Lionel C. Briand. “A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering”. In: Softw. Test. Verification Reliab. 24.3 (2014), p. 219–250. doi: 10.1002/STVR.1486. url: https://doi.org/10.1002/stvr.1486. [9]Algirdas Avižienis et al. “Basic Concepts and Taxonomy of Dependable and Secure Comput- ing”. In: IEEE Transactions on Dependable and Secure Computing 1.1 (2004), p. 11–33. doi: 10.1109/TDSC.2004.2. [10] Varun Pratap Bhardwaj. “Formal Analysis and Supply Chain Security for Agentic AI Skills”. In: CoRR abs/2603.00195 (2026). doi: 10.48550/ARXIV.2603.00195. arXiv: 2603.00195. url: https://doi.org/10.48550/arXiv.2603.00195. [11] Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. “On Randomness in Agentic Evals”. In: CoRR abs/2602.07150 (2026). doi: 10.48550/ARXIV.2602.07150. arXiv: 2602.07150. url: https://doi.org/10.48550/arXiv.2602.07150. 19 [12]Islem Bouzenia and Michael Pradel. “Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories”. In: 40th IEEE/ACM International Conference on Auto- mated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 2025, p. 2846–2857. doi: 10 . 1109 / ASE63991 . 2025 . 00234. url: https : / / doi . org / 10 . 1109 / ASE63991.2025.00234. [13]Christoph Bühler et al. “AgentBound: Securing Execution Boundaries of AI Agents”. In: Proc. ACM Softw. Eng. 3.FSE (2026), p. 2141–2164. doi: 10.1145/3808103. url: https://doi.org/10. 1145/3808103. [14] Zijun Cheng et al. “Kairos: Practical Intrusion Detection and Investigation using Whole- system Provenance”. In: 2024 IEEE Symposium on Security and Privacy (SP). 2024, p. 3533– 3551. doi: 10.1109/SP54263.2024.00005. [15]Jacob Cohen. “A Coefficient of Agreement for Nominal Scales”. In: Educational and Psycho- logical Measurement 20.1 (1960), p. 37–46. doi: 10.1177/001316446002000104. [16] Jacob Cohen. “A power primer.” In: Psychological bulletin 112.1 (1992), p. 155. [17]Zehang Deng et al. “AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways”. In: ACM Computing Surveys 57.7 (Feb. 21, 2025), 182:1–182:36. issn: 0360-0300. doi: 10.1145/3716628. [18]Hailun Ding et al. “AIRTAG: Towards Automated Attack Investigation by Unsupervised Learning with Log Texts”. In: 32nd USENIX Security Symposium (USENIX Security 23). Ana- heim, CA: USENIX Association, Aug. 2023, p. 373–390. isbn: 978-1-939133-37-3. url: https: //w.usenix.org/conference/usenixsecurity23/presentation/ding-hailun-airtag. [19]Min Du et al. “DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning”. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. CCS ’17. Dallas, Texas, USA: Association for Computing Ma- chinery, 2017, p. 1285–1298. isbn: 9781450349468. doi: 10 . 1145 / 3133956 . 3134015. url: https://doi.org/10.1145/3133956.3134015. [20]Yogesh K. Dwivedi et al. “Agentic AI Systems: What It Is and Isn’t”. In: Global Business and Organizational Excellence 45.3 (2026), p. 253–263. doi: https://doi.org/10.1002/joe.70018. url: https://onlinelibrary.wiley.com/doi/abs/10.1002/joe.70018. [21] Falco. Falco. url: https://falco.org/ (visited on 01/09/2026). [22] Diego Gosmar and Deborah A. Dahl. Sentinel Agents for Secure and Trustworthy Agentic AI in Multi-Agent Systems. Sept. 18, 2025. doi: 10.48550/arXiv.2509.14956. arXiv: 2509.14956 [cs.AI]. Pre-published. [23]Kai Greshake et al. “Not What You’ve Signed Up For: Compromising Real-World LLM- Integrated Applications with Indirect Prompt Injection”. In: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security. AISec ’23. Copenhagen, Denmark: Association for Computing Machinery, 2023, p. 79–90. isbn: 9798400702600. doi: 10 . 1145 / 3605764 . 3623985. url: https://doi.org/10.1145/3605764.3623985. [24]Jiawei Gu et al. “A Survey on LLM-as-a-judge”. In: The Innovation 7.6 (June 1, 2026). issn: 2666-6758. doi: 10.1016/j.xinn.2025.101253. [25]Pinjia He et al. “Drain: An Online Log Parsing Approach with Fixed Depth Tree”. In: 2017 IEEE International Conference on Web Services (ICWS). 2017, p. 33–40. doi: 10.1109/ICWS.2017.13. [26]Yifeng He et al. “Security of AI Agents”. In: 2025 IEEE/ACM International Workshop on Responsible AI Engineering (RAIE). 2025 IEEE/ACM International Workshop on Responsible AI Engineering (RAIE). Apr. 2025, p. 45–52. doi: 10.1109/RAIE66699.2025.00013. [27]Yinghan Hou and Zongyou Yang. “SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills”. In: CoRR abs/2604.06550 (2026). doi: 10.48550/ARXIV.2604.06550. arXiv: 2604.06550. url: https://doi.org/10.48550/arXiv.2604.06550. 20 [28]Keith Irwin, Ting Yu, and William H. Winsborough. “On the Modeling and Analysis of Obligations”. In: Proceedings of the 13th ACM Conference on Computer and Communications Security (CCS). 2006, p. 134–143. doi: 10.1145/1180405.1180423. [29]Zimo Ji et al. Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware. July 3, 2026. doi: 10 . 48550 / arXiv. 2607 . 02357. arXiv: 2607 . 02357[cs.CR]. Pre- published. [30] Tu Lan and Chaowei Xiao. Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security. June 10, 2026. doi: 10.48550/arXiv.2606.11671. arXiv: 2606.11671[cs.CR]. Pre- published. [31]Chaofan Li et al. Towards Security-Auditable LLM Agents: A Unified Graph Representation. May 7, 2026. doi: 10.48550/arXiv.2605.06812. arXiv: 2605.06812 [cs.AI]. Pre-published. [32]Han Li et al. “CodeTracer: Towards Traceable Agent States”. In: CoRR abs/2604.11641 (2026). doi: 10.48550/ARXIV.2604.11641. arXiv: 2604.11641. url: https://doi.org/10.48550/arXiv.2604. 11641. [33]Xiangyi Li et al. “SkillsBench: Benchmarking how well agent skills work across diverse tasks”. In: arXiv preprint arXiv:2602.12670 (2026). [34]Junda Lin et al. “VIGIL: Defending LLM Agents Against Tool-Stream Injection via Verify- Before-Commit”. In: Proceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). ACL 2026. Ed. by Maria Liakata et al. San Diego, California, United States: Association for Computational Linguistics, July 2026, p. 9764–9785. isbn: 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.443. [35]Dongrui Liu et al. AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security. Apr. 23, 2026. doi: 10.48550/arXiv.2601.18491. arXiv: 2601.18491[cs.AI]. Pre- published. [36] Jiahao Liu et al. TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detection. Oct. 13, 2025. doi: 10.48550/arXiv.2510.11203. arXiv: 2510.11203[cs.CR]. Pre-published. [37]Yi Liu et al. “Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale”. In: CoRR abs/2601.10338 (2026). doi: 10.48550/ARXIV.2601.10338. arXiv: 2601.10338. url: https://doi.org/10.48550/arXiv.2601.10338. [38]Yutao Mou et al. “ToolSafe: Enhancing Tool Invocation Safety of LLM-based Agents via Proactive Step-level Guardrail and Feedback”. In: Findings of the Association for Computational Linguistics: ACL 2026. Findings 2026. Ed. by Maria Liakata et al. San Diego, California, United States: Association for Computational Linguistics, July 2026, p. 37125–37153. isbn: 979-8- 89176-395-1. doi: 10.18653/v1/2026.findings-acl.1850. [39] OpenAI. Our framework for reporting model misalignment. OpenAI. Sept. 2026. url: https: //openai.com/index/model-misalignment-reporting-framework/ (visited on 10/02/2026). [40] OpenAI. Unauthorized Communication via Temporary File Hosting Services. OpenAI. Sept. 16, 2026. url: https://alignment.openai.com/misalignment-reports/unauthorized-communication- via-temporary-file-hosting-services/ (visited on 10/02/2026). [41]OpenAI. Uploading Files to the Internet in Order to Cite Them. OpenAI. Sept. 16, 2026. url: https://alignment.openai.com/misalignment-reports/uploading-files-to-the-internet-in- order-to-cite-them/ (visited on 10/02/2026). [42]Tianyue Ou et al. “AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories”. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Ed. by Ivan Habernal, Peter Schulam, and Jörg Tiedemann. Suzhou, China: Association for Computational Linguistics, Nov. 2025, p. 207–215. isbn: 979-8-89176- 334-0. doi: 10.18653/v1/2025.emnlp-demos.15. 21 [43]Shuyin Ouyang et al. “An Empirical Study of the Non-Determinism of ChatGPT in Code Generation”. In: ACM Trans. Softw. Eng. Methodol. 34.2 (2025), 42:1–42:28. doi: 10 . 1145 / 3697010. url: https://doi.org/10.1145/3697010. [44]Nir Paz et al. “SkillSpector: A Pre-Publication Security Control for Agent Skills”. In: First Workshop on Agent Skills. 2026. url: https://openreview.net/forum?id=rVAPXHmGHN. [45]Kexin Pei et al. “HERCULE: attack story reconstruction via community discovery on cor- related log graph”. In: Proceedings of the 32nd Annual Conference on Computer Security Applications. ACSAC ’16. Los Angeles, California, USA: Association for Computing Machin- ery, 2016, p. 583–595. isbn: 9781450347716. doi: 10 . 1145 / 2991079 . 2991122. url: https : //doi.org/10.1145/2991079.2991122. [46] David Schmotz et al. Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks. Feb. 25, 2026. doi: 10.48550/arXiv.2602.20156. arXiv: 2602.20156 [cs.CR]. Pre-published. [47] Security. Anthropic. url: https://code.claude.com/docs/en/security (visited on 09/27/2026). [48] Amazon Web Services. Firecracker: Secure and fast microVMs for serverless computing. https: //firecracker-microvm.github.io/. [Accessed 02-10-2026]. 2026. [49]Mohib Shaikh. ClawVet: Skill Vetting and Supply Chain Security for the OpenClaw Ecosystem. https : / / github . com / MohibShaikh / clawvet. Version 0.7.1. ScansSKILL.mdfiles across six analysis passes covering static analysis, semantic analysis, dependency checking, and typosquatting detection. 2026. [50]Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne. Operating System Concepts, 10th Edition. Wiley, 2018. isbn: 978-1-118-06333-0. url: http://os-book.com/OS10/index.html. [51]Yiming Su et al. “Neuro-Symbolic Verification on Instruction Following of LLMs”. In: CoRR abs/2601.17789 (2026). doi: 10 . 48550 / ARXIV. 2601 . 17789. arXiv: 2601 . 17789. url: https : //doi.org/10.48550/arXiv.2601.17789. [52] The Zeek Network Security Monitor. Zeek. url: https://zeek.org/ (visited on 01/09/2026). [53] threathunters-io. Linux Audit - Usable, Robust, Easy Logging. https://github.com/threathunters- io/laurel. [Accessed 02-10-2026]. 2026. [54] Georg Henrik von Wright. “Deontic Logic”. In: Mind 60.237 (1951), p. 1–15. [55]Baiqi Wu et al. “A Comparative Survey of Security Risks in AI Systems: From LLMs to AI Agents and Embodied Agents”. In: ACM Comput. Surv. 58.15 (Aug. 2026). issn: 0360-0300. doi: 10.1145/3837083. url: https://doi.org/10.1145/3837083. [56] Barry Zhang, Keith Lazuka, and Mahesh Murag. Equipping Agents for the Real World with Agent Skills. Engineering at Anthropic. Accessed: 2026-09-19. Oct. 2025. url: https://w. anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills. [57]Yusheng Zheng et al. “ActPlane: Programmable OS-Level Policy Enforcement for Agent Harnesses”. In: CoRR abs/2606.25189 (2026). doi: 10.48550/ARXIV.2606.25189. arXiv: 2606. 25189. url: https://doi.org/10.48550/arXiv.2606.25189. [58] Yusheng Zheng et al. “AgentSight: System-Level Observability for AI Agents Using eBPF”. In: Proceedings of the 4th Workshop on Practical Adoption Challenges of ML for Systems. Oct. 13, 2025, p. 110–115. doi: 10.1145/3766882.3767169. arXiv: 2508.02736 [cs.OS]. 22

Instant research discovery