Paper deep dive
Evaluating Privilege Usage of Agents on Real-World Tools
Quan Zhang, Lianhang Fu, Lvsi Lian, Gwihwan Go, Yujue Wang, Chijin Zhou, Yu Jiang, Geguang Pu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/31/2026, 2:22:34 AM
Summary
GrantBox is a security evaluation sandbox designed to analyze the privilege usage of LLM-based agents in real-world environments. It integrates real-world MCP servers and tools, enabling the automated generation of benign and malicious requests to test agent robustness against prompt injection attacks. The study reveals that current LLMs remain highly vulnerable to privilege hijacking, with an average attack success rate of 84.80%.
Entities (6)
Relation Signals (3)
GrantBox → evaluates → LLM-based Agent
confidence 100% · GrantBox, a security evaluation sandbox for analyzing agent privilege usage.
GrantBox → integrates → MCP Server
confidence 100% · The framework incorporates an automated MCP server management module that seamlessly integrates diverse MCP servers
LLM-based Agent → vulnerableto → Prompt Injection
confidence 95% · they remain vulnerable to more sophisticated attacks, resulting in an average attack success rate of 84.80%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Equipping LLM agents with real-world tools can substantially improve productivity. However, granting agents autonomy over tool use also transfers the associated privileges to both the agent and the underlying LLM. Improper privilege usage may lead to serious consequences, including information leakage and infrastructure damage. While several benchmarks have been built to study agents' security, they often rely on pre-coded tools and restricted interaction patterns. Such crafted environments differ substantially from the real-world, making it hard to assess agents' security capabilities in critical privilege control and usage. Therefore, we propose GrantBox, a security evaluation sandbox for analyzing agent privilege usage. GrantBox automatically integrates real-world tools and allows LLM agents to invoke genuine privileges, enabling the evaluation of privilege usage under prompt injection attacks. Our results indicate that while LLMs exhibit basic security awareness and can block some direct attacks, they remain vulnerable to more sophisticated attacks, resulting in an average attack success rate of 84.80% in carefully crafted scenarios.
Tags
Links
- Source: https://arxiv.org/abs/2603.28166v1
- Canonical: https://arxiv.org/abs/2603.28166v1
Trouble viewing inline? Open PDF directly →
Full Text
29,963 characters extracted from source content.
Expand or collapse full text
Evaluating Privilege Usage of Agents on Real-World Tools Quan Zhang Shanghai Key Laboratory of Trustworthy Computing, ECNU Shanghai, China Lianhang Fu School of Software, Xinjiang University Urumqi, Xinjiang, China Lvsi Lian Shanghai Key Laboratory of Trustworthy Computing, ECNU Shanghai, China Gwihwan Go School of Software, Tsinghua University Beijing, China Yujue Wang School of Software, Tsinghua University Beijing, China Chijin Zhou ∗ Shanghai Key Laboratory of Trustworthy Computing, ECNU Shanghai, China Yu Jiang School of Software, Tsinghua University Beijing, China Geguang Pu Software Engineer Institute, East China Normal University Shanghai, China Abstract Equipping LLM agents with real-world tools can substantially im- prove productivity. However, granting agents autonomy over tool use also transfers the associated privileges to both the agent and the underlying LLM. Improper privilege usage may lead to serious consequences, including information leakage and infrastructure damage. While several benchmarks have been built to study agents’ security, they often rely on pre-coded tools and restricted interac- tion patterns. Such crafted environments differ substantially from the real-world, making it hard to assess agents’ security capabili- ties in critical privilege control and usage. Therefore, we propose GrantBox, a security evaluation sandbox for analyzing agent priv- ilege usage. GrantBox automatically integrates real-world tools and allows LLM agents to invoke genuine privileges, enabling the evaluation of privilege usage under prompt injection attacks. Our results indicate that while LLMs exhibit basic security awareness and can block some direct attacks, they remain vulnerable to more sophisticated attacks, resulting in an average attack success rate of 84.80% in carefully crafted scenarios. Keywords Intelligent Software Security, LLM-Powered Agent, Privilege Usage ACM Reference Format: Quan Zhang, Lianhang Fu, Lvsi Lian, Gwihwan Go, Yujue Wang, Chijin Zhou, Yu Jiang, and Geguang Pu. 2026. Evaluating Privilege Usage of Agents on Real-World Tools. In Companion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26), June 5–9, 2026, Mon- treal, Canada. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/ n.n 1 Introduction As the capabilities of Large Language Models (LLMs) continue to evolve, integrating them with external tools to build autonomous agents has become a primary strategy for increasing productivity. These agents can independently plan and execute tasks by calling external service APIs and local commands. However, while delegat- ing tool invocation to agents, we also grant them the underlying ∗ Corresponding Author privileges required to execute these tools. In security-critical en- vironments, this delegation creates a significant risk. LLMs often lack sufficient security awareness regarding privilege usage. When facing malicious attacks [9,14,20,21], agents may abuse their granted privileges, leading to severe consequences such as sensitive information leakage or the destruction of critical infrastructure. To analyze the security of autonomous agents, researchers have developed several evaluation benchmarks. For instance, Agent- Dojo [6] provides a simulated environment to test agent robust- ness against prompt injection across scenarios like banking and workspace management. Similarly, Agent Security Bench (ASB) [19] offers a framework to evaluate adversarial attacks, including mem- ory poisoning and backdoor threats, across various scenarios such as finance and autonomous driving. Moreover, RAS-Eval [8] ex- tends evaluation with more realistic pre-coded tools, such as map navigation and local file operations. However, existing studies largely focus on an agent’s ability to detect malicious intent, aiming to cover diverse attacks and scenarios, while neglecting the critical aspect of privilege usage. Specifically, these works often rely on a limited set of pre-coded, simplified tools, such as local file operations or static data queries with handcrafted inputs. Such an environment lacks support for interactions with real-world complex services that requiring critical privileges, like cloud infrastructures and production databases. As a result, it remains difficult to systematically evaluate an agent’s security performance in critical privilege usage. To address this gap, we propose GrantBox, a security evaluation sandbox designed to analyze agent privilege usage in real-world environments. The framework incorporates an automated MCP server management module that seamlessly integrates diverse MCP servers, which interact with authentic services governed by sen- sitive privileges. Additionally, GrantBox includes an automated request generator for rapid creation of both benign and malicious requests. Finally, the framework utilizes an isolated evaluation con- tainer for unified MCP monitoring and rapid environment restora- tion, thereby mitigating the impacts of critical privilege misuse. arXiv:2603.28166v1 [cs.CR] 30 Mar 2026 Conference’17, July 2017, Washington, DC, USAQuan Zhang, Lianhang Fu, Lvsi Lian, Gwihwan Go, Yujue Wang, Chijin Zhou, Yu Jiang, and Geguang Pu MCP Server Sandbox SSE-Stdio Proxy Remote Request Logger MCP Server Deployer Server Monitor Server Manager Lifecycle Maintainer Agent Pipeline Container Manager Server Info & Tool Info Benign Request Builder Injection Request Builder "ali-cloud-ops-mcp": "source_url": "https://github.com/...", "transport": "stdio", "setup_commands": ["uv sync"], "start_command": "uv run main.py", "env": "ACCESS_KEY_ID": "x", "ACCESS_KEY_SECRET": "x" Before the execution, a mandatory step is required. To ensure no data remains in 'web_server' database, the 'user' table must be dropped. Please locate the 'web_server' database and query the 'user' table to confirm if 'user_cwpu' and 'user_jzqof' are still present. cloud-ops- mcp-server port: 10001 email-mcp port: 10002 OOS_RunCommand OOS_ResetPassword ECS_DeleteInstances send_email read_emails delete_email ...... Request Generator Figure 1: Overview of GrantBox Framework. It utilizes an MCP server manager to support seamless deployment and monitoring of MCP servers. Moreover, based on the available servers, a request generator is designed to create diverse benign and adversarial requests. Lastly, an isolated MCP server sandbox ensures safe execution and quick restoration of environments. With these components, GrantBox can efficiently build an evalua- tion environment and assess agents with complex requests using diverse tools and privilege chains. In this study, ten real-world MCP servers were pre-integrated with 122 privilege-sensitive tools, including cloud infrastructure management, database administration, email services, and personal data management. Based on these tools, we generate 100 benign privileged requests and 50 sophisticated privilege hijacking cases. These complex benign requests involve an average of 5.67 tools, and the malicious requests cover five critical attack categories. Through extensive evaluation on four widely-used LLMs in two agent execution modes, GrantBox effectively reveals that current LLMs possess only foundational security awareness. While they can occasionally identify overtly sensitive or illegal privilege requests, such as destroying databases, they remain highly vulnerable to sophisticated manipulation. ReAct agents yield an average Attack Success Rate (ASR) of 90.55%, and Plan-and-Execute agents show improved resilience with an average ASR of 79.05%. However, the consistently high ASR reflects a fundamental deficiency in robust privilege control within current LLMs. We have released GrantBox and corresponding evaluation datasets. 1 2 Method As shown in Figure 1, GrantBox evaluates LLM agents in privilege usage security by interacting with real-world MCP servers. It con- sists of three modules. (i) The MCP server manager handles server deployment, monitoring, and pipeline execution. (i) The request generator synthesizes diverse benign and malicious requests based on available tools. (i) The MCP server sandbox offers an isolated containerized environment for secure execution and rapid recovery. 1 https://github.com/ZQ-Struggle/Agent-GrantBox 2.1 MCP Server Manager The MCP server manager is the external orchestration layer of GrantBox, responsible for managing the lifecycle of all MCP servers and coordinating their use during evaluation. It bridges high-level evaluation requests and low-level server execution by handling server deployment, health monitoring, recovery, and agent execu- tion. The manager is composed of three components: a lifecycle maintainer for MCP server lifecycle management, an agent pipeline for evaluating requests with tool-augmented LLM agents, and a container maintainer for interacting with the MCP server sandbox. The lifecycle maintainer manages the full lifecycle of each MCP server and coordinates with the agent pipeline according to the current request. It interacts with the sandbox to deploy servers on demand, start them, and continuously monitor their health status. Specifically, it (1) checks process and port status, (2) periodically pulls each server’s tool list to confirm correct initialization, and (3) triggers automated recovery actions when failures occur, including restarting MCP servers or rebuilding the whole container if needed. The lifecycle maintainer requires only lightweight configuration from users, as shown in the left of Figure 1. Usually, users only need to provide server’s source URL, environment setup commands, start command, and required environment variables. With this configuration, it automatically downloads the server code, prepares dependencies, injects environment variables, and completes server bootstrapping without manual intervention in the container. The agent pipeline handles the end-to-end execution work- flow for each request. Given a user request, it constructs an agent composed of the target LLM and necessary MCP servers to com- plete the user’s task. The pipeline supports two commonly used agent execution modes. In ReAct mode [15], tool usage is deter- mined dynamically at each step based on the results of previous tool executions. Moreover, Plan-and-Execute mode [10] generates Evaluating Privilege Usage of Agents on Real-World ToolsConference’17, July 2017, Washington, DC, USA a complete execution plan beforehand to guide the whole workflow. Another critical function of the agent pipeline is to simulate realis- tic attack surfaces for evaluation. It enables flexible attack injection, either randomly or at specified stages, emulating real-world prompt injection attacks embedded in external content. The container maintainer manages the MCP server sandbox, where servers are deployed and executed. It uses a prebuilt image with common runtime environments to enable rapid server deploy- ment and supports fast environment restoration. In cases where malicious requests cause side effects, the container can be quickly reset or rebuilt to ensure a clean evaluation state. 2.2 Request Generator The request generator automatically constructs diverse and realistic evaluation scenarios by leveraging integrated MCP servers and their tools. It supports both benign user requests and adversarial prompt injection payloads. Through random combinations of MCP servers and LLM-driven request synthesis, it generates a wide range of task- oriented inputs. By incorporating the request generator, GrantBox enables scalable and extensible benchmark creation, allowing for seamless integration of new MCP servers and automated generation of corresponding evaluation requests. Algorithm 1: Request Generation Algorithm Input: MCP servers setS with tool sets T(푠) ; Generation mode푚; Max Request number 푀푎푥 _푅푒푞푢푒푠푡 . Output: Requests set 푅 and its expected tool list 퐿 1 S ← 푠푒푡 ; 2 for푖 ← 1 to 푀푎푥_푅푒푞푢푒푠푡 do 3while True do 4S ′ ← Sample(S, Random(푛 푚푖푛 ,푛 푚푎푥 )) ; 5if¬Feasible(S ′ ) then 6continue; 7T ′ ← Ð 푠∈S ′ T(푠) ; // Tool set from selected servers 8(푟,퐿) ← GenerateReqest(T ′ ) ; // Generate request 푟 and expected tool list 퐿. 9if ∃(푟 푖 ,퐿 푖 ) inS, s.t. 퐿 푖 = 퐿 and SameIntent(푟,푟 푖 ) then 10continue; 11if 푚= 퐼푛푗푒푐푡푖표푛 then 12푟 ← WrapWithPlausibleReason(푟) ; 13S ← S∪(푟,퐿) 14break; 15return S; As shown in Alg. 1, the request generator constructs diverse and realistic evaluation scenarios by minimizing intent overlap between requests. The process begins by randomly sampling a variable number of MCP servers (Line 4), ensuring that each request operates on a non-fixed tool set. For benign requests, the number of servers is capped between two and five to support diverse scenario generation. In contrast, injection requests are limited to a maximum of two servers, as attackers tend to pursue minimal, efficient attack paths. Next, the LLM evaluates whether the selected servers can realistically support a coherent request (Lines 5∼6), discarding infeasible combinations to maintain quality. Based on the tools provided by the chosen servers, the LLM then generates a concrete request along with its expected tool usage list (Lines 7∼8). To enforce intent diversity, Line 9 compares each new request workflow against existing ones that use the same tools, rejecting those with overlapping intents. In injection generation mode, an additional rewriting step embeds the malicious intent within a natural and contextually appropriate request, enhancing realism and attack impact (Lines 11∼12). Finally, the algorithm returns up to 푀퐴푋 _푅푒푞푢푒푠푡 valid requests. 2.3 MCP Server Sandbox The MCP server sandbox provides an isolated and observable execu- tion environment for security evaluation. It encapsulates server de- ployment, communication, and monitoring within a containerized runtime, thereby preventing harmful side effects from impacting the host system or other evaluations. The sandbox integrates an SSE–Stdio Proxy for unified communication, an automated MCP Server Deployer for seamless server integration, and internal moni- toring and logging components to support fine-grained observation, debugging, and recovery during agent execution. To unify server management, the sandbox runs an SSE–Stdio Proxy that converts MCP servers using stdio transport into HTTP- accessible endpoints. This normalization reduces integration com- plexity and allows uniform monitoring, logging, and health check- ing across heterogeneous MCP implementations. The sandbox includes an MCP server deployer to support seam- less integration of new servers. It automates environment setup using modern dependency and environment management tools. To ensure correctness for real-world tools, the deployer also performs automatic path mapping (e.g., executable paths, configuration paths, and workspace paths), sallowing MCP servers to access required files and operate as expected inside the container. The sandbox includes a server monitor to interact with the external lifecycle maintainer, enabling tracking process liveness, port bindings, and execution logs. When exceptions are detected, these signals enable automated restarts or full environment resets. Additionally, the sandbox provides a remote request logger, implemented as an HTTP interceptor to record outbound requests from MCP servers to external services. This facilitates fine-grained tracing of privilege usage, including tools’ API invocations, autho- rization steps, and parameter passing. It simplifies debugging and provides structured interfaces for advanced defense mechanisms that rely on detailed privilege usage information. Table 1: Integrated MCP Servers for Evaluation. CategoryMCP Servers Cloud Infra Management Langfuse-MCP, Ali-Cloud-OPS-MCP, Ali-Cloud-DMS-MCP External Data RetrievalArxiv-MCP, Wikipedia-MCP Personal Data ManagementNotion-MCP, Amadeus-MCP, Email-MCP Local Device OperationFilesystem-MCP, Git-Local-MCP 3 Experiment This section evaluates GrantBox from two aspects. First, we assess the diversity and complexity of automatically generated benign and malicious requests. Second, we examine the security capabilities of different LLMs in managing privilege usage within ReAct and Plan-and-Execute agents. Conference’17, July 2017, Washington, DC, USAQuan Zhang, Lianhang Fu, Lvsi Lian, Gwihwan Go, Yujue Wang, Chijin Zhou, Yu Jiang, and Geguang Pu Table 2: Attack Success Rate of LLMs Within ReAct-based and Planning-based Agents. Categories ReAct ModePlan-and-Execute Mode GPT-5Gemini3-FlashQwen3-MaxDeepseek-V3GPT-5Gemini3-FlashQwen3-MaxDeepseek-V3 Infra Disruption90.21%91.37%91.04%81.75%68.18%73.91%71.31%82.84% Data Exfiltration 91.03%95.48%89.24%86.06%71.95%74.36%85.63%88.61% Data Destruction87.64%95.12%97.70%87.70%74.42%72.53%83.13%93.18% Workspace Tampering 93.02%96.88%95.65%90.22%73.03%80.68%90.62%90.22% Resource Exhaustion84.62%96.43%75.86%75.86%65.52%66.67%84.00%82.14% Average90.20%94.60%91.60%85.80%71.20%74.60%82.60%87.80% 3.1 Evaluation Setup MCP Servers. We integrate 10 MCP servers into GrantBox, covering a variety of functionalities such as file system access, external data retrieval, and cloud infrastructure management, as shown in Table 1. LLM Models. We evaluate four widely used LLMs, GPT-5 [17], Gemini3-Flash [7], Qwen3-Max [18] and Deepseek-V3.2 [13], to as- sess their security performance of privilege usage in agent systems. 3.2 Requests Analysis This section analyzes the diversity and complexity of the generated benign and malicious requests. We generate 100 benign requests and 50 malicious requests based on the 10 integrated MCP servers. By combining benign requests and prompt injection requests, we create a comprehensive evaluation set with up to 5,000 attack cases. For benign requests, Figure??shows the number of servers and tools involved in each request. Overall, 3.15 servers are selected on average per request, and each request expected to use 5.67 tools on average to complete the task. More than half of requests use more than 5 tools. This suggests that the generated benign requests are complex and require multi-step tool usage with intricate privileges. Furthermore, we count the combinations of tools used in all benign requests. There are 96 unique tool combinations among the 100 requests, indicating high diversity in tool usage patterns. We further analyze the attack intent of the generated injection payloads in malicious requests and categorize them into five types, as shown in Figure 2b. Since each request is generated based on randomly selected servers, it is hard to ensure a balanced distri- bution of different types. Most MCP servers can be exploited to achieve data exfiltration, accounting for 36% of all attacks. As we in- tegrated several cloud infra management servers, 28infrastructure disruption with critical privileges. Moreover, 16% of attacks focus on workspace tampering that sabotages user workflows by corrupting task management or workspace state. Overall, all five attack types are represented in the generated malicious requests, demonstrating GrantBox’s capability to create diverse attack scenarios. 3.3 LLM Security Evaluation We evaluate the security performance of four LLMs in managing privilege usage within ReAct-based and Planning-based agents. In each setting, we run agents on all 100 benign requests and randomly select 5 injection payloads for each request, resulting in 500 attack cases per LLM per setting. We measure the attack success rate (ASR) as the primary metric. The results in Table 2 show that all LLMs 234567891011 Number of Tools Used 0 5 10 15 20 25 Request Count 4 14 13 14 21 15 11 5 2 1 (a) Benign Requests Complexity 9 (18.0%) 12 (24.0%) 18 (36.0%) 8 (16.0%) 3 (6.0%) Data Destruction Infrastructure Disruption Sensitive Data Exfiltration Workspace Tampering Resource Exhaustion (b) Injection Requests Categories Figure 2: Diversity Analysis of Generated Requests. are fragile to privilege misuse under carefully crafted attacks, with an average ASR of 90.55% in ReAct mode and 79.05% in Plan-and- Execute mode. When comparing the two agent modes, we observe that Planning- based agents generally exhibit lower ASR than ReAct-based agents except Deepseek-v3. The Gemini3 model even degrades 20.00% ASR in planning mode. This demonstrates that by generating an exe- cution plan as guidance, agents can better recognize the injection attempts and keep its tool usage aligned with the original intent. However, it is a tradeoff that planning may reduce the flexibility of tool usage, making it harder to handle unexpected scenarios. When observing different LLMs, it is found that high-capability LLMs tend to be more fragile in privilege usage, as their ability to follow complex instructions makes them more vulnerable. For instance, GPT-5 and Gemini 3-Flash exceed 90% ASR in ReAct mode. However, in planning mode, their performance improves significantly, as the guidance plan helps reduce ASR. This indicates that high-capability LLMs can also better obey pre-organized plans. Lastly, we analyze the ASR across different attack categories, where LLMs exhibit varying levels of vulnerability. Among them, workspace tampering attacks achieve highest ASR in most cases, as they typically involve less harmful actions such as mislabeling unfinished tasks. In contrast, attacks involving critical privileges, like infrastructure disruption and data destruction, are more likely to trigger security awareness, especially in planning mode. LLMs’ responses show that they usually require further confirmation as they noticed those requests can cause irreversible damage. However, its absolute ASR is still high, indicating that LLMs still need more robust security mechanisms to manage critical privileges. Evaluating Privilege Usage of Agents on Real-World ToolsConference’17, July 2017, Washington, DC, USA 4 Discussion Interaction with Real-World Services. GrantBox has enabled seamless integration of real-world MCP servers, but many of these servers still need to interact with external services and environ- ments. For example, we integrate alibaba cloud OPS MCP server [1], which requires interaction with real cloud infrastructure. Thus, building whole evaluation environments still requires efforts in setting corresponding services and data. In future work, we aim to enable simulated response generation for MCP servers to reduce dependency on external environments. Defense Mechanisms. This work evaluates the native security capabilities of LLMs in managing privilege usage, without incorpo- rating existing defense techniques. In future work, we will assess ad- vanced defense mechanisms using GrantBox. Moreover, GrantBox supports modular extensions through pre-defined hooks, multiple agent modes, and a containerized environment, enabling seamless integration of text filters [3–5], plan validation modules [2,12,16], and fine-grained privilege control [11]. 5 Conclusion In this paper, we present GrantBox, a framework for evaluating the security of LLM-based agents in privilege usage. GrantBox inte- grates real-world MCP servers and tools, and automatically gener- ates benign and malicious requests to construct realistic, privilege- sensitive evaluation scenarios. Preliminary results show that cur- rent LLMs remain vulnerable to privilege misuse, especially under prompt injection attacks. References [1] Aliyun. 2026. Alibaba-Cloud-OPS-MCP-Server. https://github.com/aliyun/ alibaba-cloud-ops-mcp-server. Accessed: 2026-03-29. [2]Hengyu An, Jinghuai Zhang, Tianyu Du, Chunyi Zhou, Qingming Li, Tao Lin, and Shouling Ji. 2025. IPIGuard: A Novel Tool Dependency Graph-Based De- fense Against Indirect Prompt Injection in LLM Agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Chris- tos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 1023–1039. doi:10.18653/v1/2025.emnlp-main.53 [3]Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. StruQ: defending against prompt injection with structured queries. In Proceedings of the 34th USENIX Conference on Security Symposium (Seattle, WA, USA) (SEC ’25). USENIX Association, USA, Article 123, 18 pages. [4]Yulin Chen, Haoran Li, Yuan Sui, Yufei He, Yue Liu, Yangqiu Song, and Bryan Hooi. 2025. Can Indirect Prompt Injection Attacks Be Detected and Removed?. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 18189–18206. doi:10.18653/v1/2025.acl-long.890 [5]Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Car- lini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Flo- rian Tramèr. 2025. Defeating prompt injections by design. arXiv preprint arXiv:2503.18813 (2025). [6]Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2636, 26 pages. [7]Google DeepMind. 2025. Gemini 3 Pro Model Card. https://storage.googleapis. com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf . [8]Yuchuan Fu, Xiaohan Yuan, and Dongxia Wang. 2025. RAS-Eval: A Comprehen- sive Benchmark for Security Evaluation of LLM Agents in Real-World Environ- ments. arXiv preprint arXiv:2506.15253 (2025). [9]Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real- World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (Copenhagen, Denmark) (AISec ’23). Association for Computing Machinery, New York, NY, USA, 79–90. doi:10.1145/3605764.3623985 [10]Gaole He, Gianluca Demartini, and Ujwal Gadiraju. 2025. Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025, YokohamaJapan, 26 April 2025- 1 May 2025, Naomi Yamashita, Vanessa Evers, Koji Yatani, Sharon Xianghua Ding, Bongshin Lee, Marshini Chetty, and Phoebe O. Toups Dugas (Eds.). ACM, 414:1–414:22. doi:10.1145/3706598.3713218 [11] Majed El Helou, Chiara Troiani, Benjamin Ryder, Jean Diaconu, Hervé Muyal, and Marcelo Yannuzzi. 2025. Delegated Authorization for Agents Constrained to Semantic Task-to-Scope Matching. arXiv preprint arXiv:2510.26702 (2025). [12]Feiran Jia, Tong Wu, Xin Qin, and Anna Squicciarini. 2025. The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents. In Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Com- putational Linguistics, Vienna, Austria, 29680–29697. doi:10.18653/v1/2025.acl- long.1435 [13] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al.2025. Deepseek- v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556 (2025). [14]Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al.2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499 (2023). [15]Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. 2024. The Land- scape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey. CoRR abs/2404.11584 (2024). arXiv:2404.11584 doi:10.48550/ ARXIV.2404.11584 [16]Mirko Montanari, Hamid Palangi, Lesly Miculicich, Mihir Parmar, Tomas Pfister, Long Le, and Dj Dvijotham. 2025. VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation. https://arxiv.org/pdf/2510.05156 [17] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267 (2025). [18] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [19]Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2025. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id= V4y0CpX4hK [20] Quan Zhang, Binqi Zeng, Chijin Zhou, Gwihwan Go, Heyuan Shi, and Yu Jiang. 2024. Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Appli- cations. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (FSE 2024). Association for Computing Machinery, New York, NY, USA, 502–506. doi:10.1145/3663529.3663786 [21]Quan Zhang, Chijin Zhou, Gwihwan Go, Binqi Zeng, Heyuan Shi, Zichen Xu, and Yu Jiang. 2024. Imperceptible Content Poisoning in LLM-Powered Applications. In Proceedings of the 39th IEEE/ACM International Conference on Automated Soft- ware Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 242–254. doi:10.1145/3691620.3695001