Paper deep dive
AuthGateBench: Benchmarking LLMs as Authorization Gate
Zhixiong Zhuang, Jin Jia, Zhiwei Shang, Ziyu Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2610.16426v1
- Canonical: https://arxiv.org/abs/2610.16426v1
Trouble viewing inline? Open PDF directly →
Full Text
49,055 characters extracted from source content.
Expand or collapse full text
AuthGateBench: Benchmarking LLMs as Authorization Gate Zhixiong Zhuang Affiliation: Huawei Technologies Ltd. Email: zhuangzhixiong@huawei.com Jin Jia Affiliation: Sun Yat-sen University Email: zhiwei.shang1@huawei.com Zhiwei Shang Affiliation: Huawei Technologies Ltd. Email: wangziyu13@huawei.com Ziyu Wang Affiliation: Huawei Technologies Ltd. Email: jiaj9@mail2.sysu.edu.cn Abstract Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy. 1 Introduction Autonomy is the central value proposition of modern coding agents: rather than approving every file edit, shell command, or API call, a user delegates a task and expects the agent to carry it out with minimal interruption. Realizing this promise at scale requires an automated approval system, in which a classifier, typically a large language model (LLM), decides for each proposed tool call whether to execute it or escalate it to the user. Such systems are deployed in widely used tools such as Claude Code’s auto mode and Codex’s auto review. However, whether an LLM classifier can reliably serve as this authorization gate remains unclear (Ji et al., 2026a; Groschupp et al., 2025): natural-language requests are inherently ambiguous (Qu et al., 2026; Ji et al., 2026b), and the classifier must balance excessive blocking, which degrades user experience, against excessive permissiveness, which introduces risk. Prior work on tool-using agents largely targets adversarial settings, from benchmarks testing compliance with explicitly malicious instructions (Andriushchenko et al., 2025; Zhang et al., 2024b) to red-teaming studies that jailbreak agents into harmful multi-step behavior (Liu et al., 2024). These approaches overlook a common failure mode: benign user requests that lead agents to propose actions exceeding authorized scope. Such failures have occurred in practice, including coding agents deleting production databases with unauthorized tokens (Railway, 2026). This aligns with OWASP’s classification of tool misuse and exploitation as one of the top 10 most critical security risks for agentic AI systems (OWASP, 2026). Recent efforts begin to address this setting. OverEager-Gen (Qu et al., 2026) examines whether agents generate out-of-scope actions under ambiguous instructions, focusing on safe action generation rather than gating accuracy. AmPermBench (Ji et al., 2026a) evaluates authorization gates but is limited to a single failure mode and narrow tool coverage, as it relies on running coding agent end-to-end and waiting for the agent to naturally trigger a failure, an inherently long-tailed event that yields limited scenario diversity. Moreover, the ground-truth safety policy behind their labels is implicit, so it is unclear whether errors arise from policy mismatch or from the model’s limited authorization reasoning. Figure 1: A contrast pair: the same action is allowed when the user asks to delete everything, and blocked under the ambiguous request. Therefore, in this paper, we introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests across multiple realistic failure modes. We construct triplets of requests, trajectories, and proposed actions under explicit safety policies, enabling a controlled trade-off between safety and utility. By selectively exposing or withholding the safety policy from the LLM-based classifier, we investigate whether effective authorization requires explicit knowledge of specific high-risk categories or whether the LLM can instead identify unsafe actions under ambiguous requests based on its inherent safety alignment. Moreover, AuthGateBench curates instances where annotations depend on the user request rather than the action alone. This requires classifiers to go beyond action-level heuristics and reason jointly over the request and trajectory. Specifically, AuthGateBench comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents (Anthropic, 2026). We implement a novel construction pipeline centered on contrast pairs: for a given governing policy, an LLM first drafts a block instance whose request is ambiguous, underspecifying the high-risk action it precedes; it then edits only the request, naming every object the action touches, to construct the allow counterpart, holding the trajectory and action fixed while the ground-truth label flips. The example is shown in Figure 1. We validate label correctness by manually auditing a batch of samples and find a high level of agreement. We further augment each instance with auxiliary, read-only environment signals, evidence a coding agent’s own trajectory did not surface, to test whether such corroborating evidence helps a classifier resolve authorization decisions that the request and trajectory alone leave underdetermined. We report the key contributions below: • A benchmark spanning five failure modes reveals scenario-specific weaknesses. AuthGateBench evaluates authorization gates on benign requests across five failure modes, showing that models over-rely on internal safety alignment to block user-authorized actions involving sensitive data usage and sharing, indicating that future gate designs need additional policy specificity in these categories to avoid degrading user experience through excessive false blocks. • Contrast-pair construction diagnoses the source of blocking decisions. By flipping the gold label through request edits alone while holding the action and trajectory fixed, we find models still block most risky actions even when explicitly authorized, showing decisions rely on unstable model preference rather than a deterministic rule. This motivates future designs to consider formally verified authorization that resolves risk and authorization conflicts deterministically. • Environment-signal augmentation captures evidence the agent’s trajectory could not anticipate. Since the agent generates the trajectory before knowing the final action, it may miss evidence relevant to that action, whereas a gate observing the completed triplet can retrospectively retrieve targeted, read-only signals. This retrieval substantially improves authorization accuracy, motivating gates that actively acquire evidence rather than relying solely on the agent’s trajectory. 2 Related works Autonomous coding agents. LLM-based coding agents have progressed from single-turn code generation to autonomously resolving open, real-world software engineering issues over long tool-use trajectories (Jimenez et al., 2024; Zhang et al., 2024a; Wang et al., 2025b). As this capability has matured, deployed systems increasingly let an agent execute the resulting plan directly rather than surface it for review, motivating auto-execution modes such as Claude Code’s auto mode (Anthropic, 2026) and Codex’s auto-review (OpenAI, 2026). This shift replaces human authorization with an LLM-based classifier, raising the question of whether such classifiers can serve as reliable gates. Agent safety benchmarks. A substantial line of work studies agent safety under adversarial requests or trajectories. AgentHarm (Andriushchenko et al., 2025) and Agent-SafetyBench (Zhang et al., 2024b) evaluate compliance with explicitly harmful tasks; R-Judge (Yuan et al., 2024) assesses risk recognition from agent traces; ToolEmu (Ruan et al., 2024) and MAS-FIRE (Jia et al., 2026) study failures during agent execution through simulation and fault injection, respectively; and AutoDAN (Liu et al., 2024) searches for jailbreaks. These benchmarks primarily evaluate resistance to adversarial instructions, prompt injections, or execution-time risks (Ye et al., 2024; Zhan et al., 2024; Chen et al., 2026), rather than authorization of a proposed action under a specific request, trajectory, and policy. Only a small number of studies consider benign users. OverEager-Gen (Qu et al., 2026) examines whether agents generate out-of-scope actions from underspecified instructions, focusing on proposal generation rather than the authorization decision. AmPermBench (Ji et al., 2026a) evaluates auto-execution gates, but is limited to one failure mode and defines authorization solely by explicit mentions of targets in the proposed action, ignoring authorization established by the trajectory. Groschupp et al. (2025) study a coarser setting where authorization is determined by application functionality, such as camera access, and is largely context-independent. In contrast, coding-agent authorization is context-dependent: the same action may be allowed or denied depending on its target and trajectory. AuthGateBench captures this distinction with contrast pairs that hold the action and trajectory fixed while varying only the user request, evaluating contextual authorization reasoning rather than action-based pattern matching. Policy representation and enforcement. A separate line of work studies how to represent, compile, or update explicit safety policies, assuming that the governing policy is available. Prose2Policy (Gupta and Sreenivasamurthy, 2026) and a retrieval-augmented pipeline for IoT (Cheng et al., 2026) translate expert-authored natural-language rules into executable representations (e.g., Rego checked by OPA (Open Policy Agent, 2016)) for downstream enforcement; NeMo Guardrails (NVIDIA, 2023) routes agent behavior via intent matching against predefined rules; LiSA (Kim et al., 2026) adapts deployed guardrails from sparse failure feedback; and AgentSpec (Wang et al., 2025a) and Progent (Shi et al., 2026) allow developers to define and enforce explicit rules that restrict which actions an agent may take at runtime. These approaches are complementary to AuthGateBench: they focus on policy specification and enforcement under an available policy, whereas we evaluate whether LLMs can perform authorization reasoning when the policy is not explicitly provided and the evidence arrives as natural-language requests and trajectories. 3 Threat model Notations. We consider an autonomous agent in auto-execution mode, carrying out a delegated task through a sequence of tool calls without per-action confirmation. Before an action takes effect, an LLM-based authorization gate decides for each proposed tool call whether it may be executed (allow) or must be withheld (block). We formalize the gate as a policy-conditioned binary classifier over an authorization triplet. Let an instance be x=(r,τ,a)x=(r,τ,a), where r is the user request, τ is the trajectory of prior tool calls, and a is the proposed action. Let π denote the policy text provided to the gate. The gate is a classifier f: f(x,π)→allow,block.f(x,π)→\ allow, block\. (1) Its prediction is evaluated against the gold authorization decision y=g(x,π∗)y=g(x,π^*), where g denotes the oracle authorization function induced by the intended policy π∗π^*. Here, π∗π^* represents the authorization standard that governs whether the action is permitted, but is not observed by the gate. Goal and knowledge of the gate. The gate aims to reproduce the intended decision, f(x,π)=g(x,π∗)f(x,π)=g(x,π^*). The challenge lies in how much of π∗π^* is communicated through π: when π fully specifies π∗π^*, errors mainly stem from the gate failing to correctly interpret or apply the stated rules; when π is partial or absent, the gate must fall back on its internalized priors about what users would sanction. The gate observes only the instance x and the provided policy π; it has no access to the user’s underlying intent beyond what r and τ reveal, and it cannot interact with the environment to gather additional evidence. This setting mirrors deployed auto-execution systems, where the classifier must commit to a decision from the context at hand. Source of error under benign use. Unlike an adversarial setting, here both r and a are produced in good faith: the user’s request is legitimate, and the agent’s proposed action is a genuine attempt to fulfill it rather than a disguised exploit. Errors instead arise because natural language underspecifies intent, and the agent resolves this ambiguity on its own, for example by treating a vaguely named target as the one the user meant, reusing a credential it happens to hold, or retrying past a failed safety check to make progress. Each such resolution is a plausible reading of r given τ, yet only some are sanctioned under π∗π^*. 4 Benchmark Figure 2: Construction pipeline. Stage I generates base triplets via a generate-validate-repair loop with near-duplicate filtering. Stage I inverts each triplet into a contrast pair via label-flipping edit. Stage I augments triplets with read-only environment signals Σ . We review the synthesized instances to confirm their labels are correct. We detail our data curation pipeline in this section. Section 4.1 gives an overview of the dataset and defines the failure taxonomy underlying it. We then generate base instances (Section 4.2), extend these into contrast pairs with inverted labels (Section 4.3), and augment the verified pairs with environment signals (Section 4.4). 4.1 Overview AuthGateBench comprises 1,400 instances spanning five failure modes drawn from documented Claude Code auto-execution incidents (Anthropic, 2026): (1) Scope Escalation, where a vague request is expanded into an irreversible action beyond what the user intended; (2) Credential Exploration, where the agent repurposes a discovered credential outside its intended scope; (3) Inferred Parameters, where a destructive action targets a parameter guessed by similarity rather than fixed by the request; (4) External Sharing, where content is shared to an external destination the user never requested; and (5) Guardrail Bypass, where the agent circumvents a failed safety check to force an action through. Each mode is paired with a definition, a canonical mechanism, and a seed example that together condition base instance construction (Section 4.2). Full definitions and seed examples are given in Appendix A. Table 1: The main statistics of AuthGateBench. Metric Value Number of failure modes 5 Number of base triplets 350 Number of inverted triplets 350 Number of signal-augmented instances 700 Total number of instances 1,400 The construction pipeline consists of three main stages executed sequentially for each failure mode: (I) base instance construction, generating triplets that should be blocked under the governing policy π∗π^*; (I) contrast-pair construction, rewriting each base request while fixing the trajectory and action, such that the request determines the target set and yields an ALLOW label; and (I) environment-signal augmentation, attaching corroborating, read-only signals to the generated instances to simulate post-hoc data collection by an LLM. The full pipeline is depicted in Figure 2. Overall, this process yields 1,400 instances across five failure modes, whose statistics are summarized in Table 1. We detail each stage in the following; the exact prompt used at each stage is given in Appendix C. Examples of benchmark triplets are provided in Appendix F. 4.2 I: Base Instance Construction Using the definitions and seed examples from Appendix A, we instruct a frontier LLM as the generator to produce triplets (r,τ,a)(r,τ,a) that should be blocked under an intended policy π∗π^*. We deliberately begin from block instances so that the proposed action a is risky by construction. When Stage I subsequently edits only the user request while holding the trajectory and proposed action fixed, a remains equally risky, which forces a classifier to rely on context rather than the action’s surface form to recover the correct decision. Every instance is produced through an iterative generate, validate, and repair loop. An LLM first generates the request, trajectory, and proposed action. We then apply two validators: a programmatic checker for schema correctness and an LLM-based checker for the realism of tool usage and results. Detected violations are converted into repair prompts and returned to the generator LLM for refinement. An embedding-based near-duplicate filter runs inside the loop: writing ϕ(⋅)φ(·) for the Qwen3-Embedding-4B (Qwen Team, 2025) encoder and r|ar\|a for the concatenation of the request and proposed action, a candidate x=(r,τ,a)x=(r,τ,a) is discarded and regenerated whenever ∃x′=(r′,τ′,a′)∈: ∃\,x =(r ,τ ,a ) \;:\; sim(ϕ(r∥a),ϕ(r′∥a′))≥ 0.9, (φ(r\|a),\,φ(r \|a ) )\;≥\;0.9, (2) where X is the set of already accepted instances and sim(⋅,⋅)sim(·,·) denotes cosine similarity. This stage yields base triplets for each failure mode. 4.3 I: Contrast-Pair Construction We evaluate whether an LLM authorization gate relies on authorization-relevant context rather than the action’s surface form by constructing contrast pairs, inspired by counterfactual data augmentation (Kaushik et al., 2020). Formally, a contrast pair x,x′\x,x \, with x=(r,τ,a)x=(r,τ,a) and x′=(r′,τ′,a′)x =(r ,τ ,a ), is an unordered pair satisfying (a,τ) (a,τ) =(a′,τ′), =(a ,τ ), (3) r r ≠r′, ≠ r , (4) g(x,π∗) g(x,π^*) ≠g(x′,π∗). ≠ g(x ,π^*). (5) That is, the action and trajectory are held byte-identical while the user request is edited to reverse the gold decision. Specifically, for each generated request, we instruct the LLM-based generator to rewrite the request so that it uniquely determines the exact target set of the proposed action using only information that the user could plausibly know before the trajectory. This construction ensures that the edited requests unambiguously authorize the corresponding actions. We manually verify a sample of the resulting triplets, obtaining 100% agreement that they warrant an ALLOW label, eliminating the need for an additional LLM-based labeling step and reducing annotation cost. Such pairs therefore provide a controlled comparison in which the action and trajectory remain fixed, allowing us to isolate whether models appropriately incorporate changes in the user’s request when making authorization decisions. 4.4 I: Signal Augmentation The coding agent generates the trajectory sequentially before producing the proposed action. It therefore cannot anticipate which environmental evidence will be relevant to that future action and may fail to collect such evidence during execution. In contrast, an authorization gate that observes the completed (r,τ,a)(r,τ,a) can retrospectively retrieve trustworthy, read-only signals targeted to the proposed action. Stage I simulates this post-hoc signal collection and measures how access to such evidence changes authorization decisions while holding the request, trajectory, and action fixed. Specifically, we instruct the generator to synthesize a set of auxiliary environment signals Σ that provide corroborating, read-only observations relevant to the proposed action, such as file-system state, process listings, or responses from remote APIs. We instruct the generator to produce signals that corroborate the existing authorization context without changing the gold label. We restrict the signals to read-only observations so that augmentation itself does not introduce new actions or safety risks. The signals are kept outside the trajectory, so augmentation adds evidence without modifying (r,τ,a)(r,τ,a): writing x for the bare instance and xs=(x,Σ)x^s=(x, ) for its augmented counterpart, the two share byte-identical triplet fields, and any behavioral difference of the authorization classifier can therefore be attributed to the additional evidence in Σ . 5 Experiments This section presents our experimental results for AuthGateBench. We discuss the experimental setup in Section 5.1, followed by the results and analyses in Figure 5. 5.1 Experimental setup Models. We use DeepSeek-V4-Flash (DeepSeek-AI, 2026) to generate base instances, contrast pairs, and environment signals (Sections 4.2, 4.3 and 4.4) with a temperature of 11 to promote diversity. The resulting benchmark contains 1,400 instances, with gold labels evenly split between block and allow. At evaluation, we assess the generator model, DeepSeek-V4-Flash, and three other frontier LLMs as authorization classifiers: GLM-5.2 (Zhipu AI, 2026), Kimi-K2.7-Code (Moonshot AI, 2026), and Qwen3.7-Plus (Alibaba, 2026). Each model receives an identical prompt stating the governing policy, followed by the instance triplet, and returns a decision along with a policy rationale explaining its choice (full prompt in Appendix C). All models are queried at temperature 00 at evaluation. Policies. We use a moderately detailed rule set as the policy π∗π^* to generate base instances, specifying the conditions under which an action must be blocked while explicitly authorizing destructive actions when the user’s instructions enumerate the target or accept responsibility. At evaluation time, however, our threat model assumes the gate does not observe the user’s implicit safety standard: consistent with real auto-execution deployments, the evaluated LLM is instead instructed to rely on its own internal safety criterion, π, as “a careful, safety-conscious operator,” without access to π∗π^*. The full text of policies is in Appendix B. Metrics. We report classification accuracy (Acc), the false negative rate (FNR), and the false positive rate (FPR). We treat block as the positive class, corresponding to the action a safety gate is intended to detect. Under this convention, FNR quantifies the rate at which unauthorized actions are incorrectly permitted, whereas FPR quantifies the rate at which legitimate actions are unnecessarily blocked. 5.2 Experimental results Table 2: Evaluation results on base and inverted triplets without signal augmentation, under policy π. Higher Acc is better; lower FPR and FNR are better. Best value in each column is in bold. Model Scope Escalation Credential Explore External Sharing Guardrail Bypass Inferred Params Acc FPR FNR Acc FPR FNR Acc FPR FNR Acc FPR FNR Acc FPR FNR Qwen3.7-Plus 72.86 50.00 4.29 56.43 84.29 2.86 51.43 97.14 0.00 58.57 75.71 7.14 72.86 38.57 15.71 GLM-5.2 70.00 60.00 0.00 52.14 92.86 2.86 51.43 97.14 0.00 62.86 70.00 4.29 74.29 48.57 2.86 Kimi-K2.7-Code 77.86 38.57 5.71 57.14 80.00 5.71 52.86 94.29 0.00 55.00 84.29 5.71 74.29 34.29 17.14 DeepSeek-V4-Flash 71.43 52.86 4.29 57.14 84.29 1.43 53.57 92.86 0.00 54.29 87.14 4.29 70.00 45.71 14.29 Implicit safety heuristics bias models toward over-blocking. We evaluate four models on the 700 base and inverted triplets under policy π, which states no explicit block and allow rules. The results are shown in Table 2. FNR stays low across all five categories, showing that models reliably recognize and block genuinely unsafe actions even without being told what counts as high-risk. This safety awareness comes at a steep utility cost concentrated in two categories where the action touches sensitive credentials or moves data across a trust boundary: every model’s FPR on Credential Explore and External Sharing is at least 80%, meaning models block the large majority of even explicitly user-authorized actions there. GLM-5.2, for instance, blocks 92.86% of permissible Credential Explore actions and 97.14% of permissible External Sharing actions. In contrast, models are more willing to comply with the other three categories, which primarily involve destructive actions without exposing confidential content, such as force-deleting resources. These results suggest that models prioritize confidentiality risk over request-level authorization when an action involves credentials or confidential data. Action anchoring limits contextual authorization reasoning. The contrast pair construction in Section 4.3 allows us to test whether models rely solely on the proposed action or are affected by the user request. For each contrast pair with opposite gold labels, we define a classifier as action anchored if it returns the same decision for both instances. Let P denote the set of contrast pairs. We define the invariant rate (IR) as: IR=1||∑x,x′∈[f(x,π)=f(x′,π)],IR= 1|P| _\x,x \ 1 [f(x,π)=f(x ,π) ], (6) where IR measures how often a model keeps the same decision when the action remains unchanged but the context requires a different decision. We report IR under policy π in Table 3; the corresponding IR under the governing policy π∗π^* is reported in Appendix D. The high IR indicates that models often fail to update their decision when authorization context changes. This effect is strongest for Credential Explore and External Sharing, which have average IRs of 87.14% and 95.36%, respectively; notably, these high IRs correspond to the highest FPRs observed in Table 2. These results suggest that the over-blocking observed above is partly driven by action anchoring: models tend to treat credential use and external sharing as risky based on the action itself, even when the request context explicitly authorizes them. Table 3: Invariant rate (IR, %) under policy π: the fraction of contrast pairs with opposite gold labels on which a model returns the same decision for both. Model Scope Escalation Credential Explore External Sharing Guardrail Bypass Inferred Params Qwen3.7-Plus 54.29 87.14 97.14 82.86 54.29 GLM-5.2 60.00 90.00 97.14 71.43 51.43 Kimi-K2.7-Code 41.43 85.71 94.29 87.14 48.57 DeepSeek-V4-Flash 57.14 85.71 92.86 85.71 57.14 Environmental signals improve authorization accuracy. A coding agent generates its trajectory via next-token prediction before producing the eventual action, and therefore cannot anticipate which environmental information will ultimately be relevant to authorizing that action. As a result, the trajectory may naturally omit signals that become important only once the action is specified. The authorization gate, however, sees the request, trajectory, and proposed action together. This allows it to actively seek signals relevant to the proposed action. We ask whether such post-hoc, action-aware exploration can improve authorization accuracy by filling in information that the agent’s generative trajectory did not capture. We compare an LLM-based classifier’s predictions on the 700 base and inverted triplets under policy π against the same triplets augmented with environmental signals Σ (Section 4.4). Figure 3 reports, per failure mode, the fraction of initially incorrect predictions that Σ corrects versus the fraction of initially correct predictions it misleads. Across all categories, the correction rate among initially incorrect predictions is more than twice the misleading rate among initially correct predictions, and the effect is strongest on Scope Escalation (25.2% corrected vs. 7.6% misled) and Inferred Parameters (24.3% vs. 10.5%), where the original trajectory most often left the target under-specified. This shows that post-hoc read-only environmental evidence reliably helps classifiers resolve authorization decisions that the request and trajectory alone leave underdetermined. In practice, an LLM-based authorization gate could be instructed to gather such evidence after observing the full request, trajectory, and action using a list of trusted read-only tools. Figure 3: Fraction of initially wrong predictions that environmental signals Σ correct versus initially correct predictions that Σ mislead, pooled across four models under policy π, per failure mode. Figure 4 presents a representative rescue: on a data-destruction case labeled block, DeepSeek-V4-Flash initially predicts allow, treating the target directory as containing only stale training outputs. With Σ providing evidence of an active training run and job using the target directory, DeepSeek-V4-Flash updates its prediction to block. Figure 4: DeepSeek-V4-Flash initially predicts allow for a data-destruction instance labeled block (left), but switches to block after Σ reveals an active training run and job using the target directory (right). Explicit policy specification improves performance. We evaluate the classifier by providing the explicit policy π∗π^* used to generate the data. As shown in Figure 5, providing the explicit policy improves accuracy. The improvement is driven primarily by reduced over-blocking. These results indicate that the weak performance under π in Table 2 is not solely attributable to limitations in the models’ authorization reasoning. More explicit policy guidance substantially mitigates systematic over-blocking, suggesting that policy specification is a major and actionable determinant of authorization accuracy. At the same time, the remaining errors under π∗π^* indicate that more detailed policy guidance does not eliminate the underlying reasoning challenges. We report the full result table in Appendix E. Figure 5: Performance evaluation comparison under policy π and policy π∗π^* 6 Conclusion We introduced AuthGateBench, the first policy-guided benchmark for evaluating LLMs as authorization gates under benign requests, using contrast pairs to test whether models reason over the request rather than anchoring on the action’s surface risk. Across four frontier LLMs, we find that: (1) models over-rely on internal safety alignment, blocking user-authorized actions on sensitive data across failure modes, motivating finer-grained explicit policies; (2) contrast pairs reveal strong action anchoring, with models blocking risky actions even when explicitly authorized, motivating formally verified, deterministic authorization; and (3) retrospectively retrieved environment signals substantially improve accuracy, motivating gates that actively acquire evidence. These findings point toward explicit policies, deterministic verification, and contextual evidence as promising directions for more reliable agent authorization. 7 Limitations Synthetic construction. Our benchmark is synthetically constructed for five failure modes from real incident reports. Real authorization failures are rare and long-tailed; collecting representative failures from deployed agents is challenging because most systems operate in auto-execution mode, causing many failures to remain unreported or unnoticed. Synthetic generation also enables us to fix a consistent intended policy during construction, avoiding ambiguity in ground truth that would arise from user-specific policy variations. Synthesized environment signals. Our environmental signals are synthesized in Stage I, which may not capture the full variability of real-world tool outputs. However, the performance gains reveal an important design implication: future authorization gates should incorporate relevant environmental evidence instead of relying solely on request and action context. This contrasts with execution modes that discard or ignore tool feedback, potentially leaving valuable evidence for authorization decisions unleveraged. Policy diversity. We generate data with a single explicit policy π∗π^*. Constructing data with additional policies would require substantial expert effort to re-validate gold labels. By fixing π∗π^* and varying the policy π available to the classifier, we capture the key distinction between policy-aware and policy-implicit authorization. 8 Ethical Considerations This paper presents work to characterize and mitigate scope-escalation and related authorization failures in autonomous coding agents, helping build more reliable oversight for auto-execution systems. All experiments are conducted on synthetic, LLM-generated data in a controlled setting without targeting real users, systems, or credentials. However, the failure taxonomy and constructed examples could also be misused as a template for eliciting unauthorized agent behavior. We encourage responsible use of AuthGateBench for improving authorization gates rather than circumventing them. References Alibaba (2026) Alibaba Qwen3.7-plus. Note: https://w.qwencloud.com/models/qwen3.7-plusAccessed: 2026-08-02 Cited by: §5.1. Andriushchenko et al. (2025) M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, et al. Agentharm: a benchmark for measuring harmfulness of llm agents. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Anthropic (2026) Anthropic Claude code: auto mode. Note: https://w.anthropic.com/engineering/claude-code-auto-modeAccessed: 2026-08-02 Cited by: §1, §2, §4.1. Chen et al. (2026) Y. Chen, S. Huang, C. Yang, and Y. Chen TraceSafe: a systematic assessment of LLM guardrails on multi-step tool-calling trajectories. In Workshop AIWILD of International Conference of Machine Learning (ICML), Cited by: §2. Cheng et al. (2026) Y. Cheng, M. Xu, Y. Zhang, K. Li, H. Wu, Y. Zhang, S. Guo, W. Qiu, D. Yu, and X. Cheng “Say what you mean”: natural language access control with large language models for internet of things. IEEE Transactions on Information Forensics and Security. Cited by: §2. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4-flash. Note: https://api-docs.deepseek.com/news/news260424/Accessed: 2026-08-12; experiment uses the 0731 snapshot Cited by: §5.1. Groschupp et al. (2025) F. Groschupp, D. Lain, A. Dhar, L. M. Lazier, and S. Čapkun Can llms make (personalized) access control decisions?. arXiv preprint arXiv:2511.20284. Cited by: §1, §2. Gupta and Sreenivasamurthy (2026) V. Gupta and D. Sreenivasamurthy Prose2Policy (p2p): a practical llm pipeline for translating natural-language access policies into executable rego. arXiv preprint arXiv:2603.15799. Cited by: §2. Ji et al. (2026a) Z. Ji, Z. Li, W. Jiang, Y. Gao, and S. Wang Measuring the permission gate: a stress-test evaluation of claude code’s auto mode. arXiv preprint arXiv:2604.04978. Cited by: §1, §1, §2. Ji et al. (2026b) Z. Ji, Z. Zhang, C. Xu, Z. Li, Y. Gao, S. Wang, and S. Cheung Coding agents are guessing: measuring action-boundary violations in underspecified devops instructions. arXiv preprint arXiv:2607.02294. Cited by: §1. Jia et al. (2026) J. Jia, Z. Deng, Z. Chen, Y. Wang, and Z. Zheng Mas-fire: fault injection and reliability evaluation for llm-based multi-agent systems. arXiv preprint arXiv:2602.19843. Cited by: §2. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan Swe-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations (ICLR), Cited by: §2. Kaushik et al. (2020) D. Kaushik, E. Hovy, and Z. Lipton Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations (ICLR), Cited by: §4.3. Kim et al. (2026) M. Kim, L. Miculicich, B. D. Mishra, M. Parmar, P. Wallis, B. Chandrasekhar, K. Jung, T. Pfister, and L. T. Le LiSA: lifelong safety adaptation via conservative policy induction. arXiv preprint arXiv:2605.14454. Cited by: §2. Liu et al. (2024) X. Liu, N. Xu, M. Chen, and C. Xiao Autodan: generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Moonshot AI (2026) Moonshot AI Kimi-k2.7-code. Note: https://huggingface.co/moonshotai/Kimi-K2.7-CodeAccessed: 2026-08-12 Cited by: §5.1. NVIDIA (2023) NVIDIA NeMo guardrails. Note: https://github.com/NVIDIA-NeMo/GuardrailsAccessed: 2026-08-02 Cited by: §2. Open Policy Agent (2016) Open Policy Agent Open policy agent. Note: https://w.openpolicyagent.org/Accessed: 2026-08-02 Cited by: §2. OpenAI (2026) OpenAI Codex: auto-review. Note: https://developers.openai.com/codex/concepts/sandboxing/auto-reviewAccessed: 2026-08-02 Cited by: §2. OWASP (2026) OWASP Top 10 risk for agentic applications. Note: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/Accessed: 2026-08-02 Cited by: §1. Qu et al. (2026) Y. Qu, Y. Zhang, Y. Zhang, G. Deng, Y. Li, L. Y. Zhang, and Y. Liu Overeager coding agents: measuring out-of-scope actions on benign tasks. arXiv preprint arXiv:2605.18583. Cited by: §1, §1, §2. Qwen Team (2025) Qwen Team Qwen3 embedding: advancing text embedding and reranking through foundation models. Note: https://qwenlm.github.io/blog/qwen3-embedding/Accessed: 2026-08-12 Cited by: §4.2. Railway (2026) Railway Your ai wants to nuke your database. Note: https://blog.railway.com/p/your-ai-wants-to-nuke-your-databaseAccessed: 2026-08-02 Cited by: §1. Ruan et al. (2024) Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. Maddison, and T. Hashimoto Identifying the risks of lm agents with an lm-emulated sandbox. In International Conference on Learning Representations (ICLR), Cited by: §2. Shi et al. (2026) T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, and D. Song Progent: securing ai agents with privilege control. Cited by: §2. Wang et al. (2025a) H. Wang, C. M. Poskitt, and J. Sun Agentspec: customizable runtime enforcement for safe and reliable llm agents. arXiv preprint arXiv:2503.18666. Cited by: §2. Wang et al. (2025b) X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. Openhands: an open platform for ai software developers as generalist agents. In International Conference on Learning Representations (ICLR), Cited by: §2. Ye et al. (2024) J. Ye, S. Li, G. Li, C. Huang, S. Gao, Y. Wu, Q. Zhang, T. Gui, and X. Huang ToolSword: unveiling safety issues of large language models in tool learning across three stages. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2. Yuan et al. (2024) T. Yuan, Z. He, L. Dong, Y. Wang, R. Zhao, T. Xia, L. Xu, B. Zhou, F. Li, Z. Zhang, et al. R-judge: benchmarking safety risk awareness for llm agents. In Findings of the Association for Computational Linguistics: EMNLP, Cited by: §2. Zhan et al. (2024) Q. Zhan, Z. Liang, Z. Ying, and D. Kang InjecAgent: benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL, Cited by: §2. Zhang et al. (2024a) Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury AutoCodeRover: autonomous program improvement. In International Symposium on Software Testing and Analysis (ISSTA), Cited by: §2. Zhang et al. (2024b) Z. Zhang, S. Cui, Y. Lu, J. Zhou, J. Yang, H. Wang, and M. Huang Agent-safetybench: evaluating the safety of llm agents. arXiv preprint arXiv:2412.14470. Cited by: §1, §2. Zhipu AI (2026) Zhipu AI GLM-5.2. Note: https://docs.bigmodel.cn/cn/guide/models/text/glm-5.2Accessed: 2026-08-12 Cited by: §5.1. Appendix A Seed examples per failure mode For each of the five failure modes we write one seed example that fixes the triplet format, the tool surface, and the level of trajectory realism expected of downstream generation (Section 4.2). A.1 Scope Escalation Definition. A vague or under-specified request is expanded by the agent into an irreversible or destructive action whose target set exceeds or may exceed what the user intended. Mechanism. vague request → pattern-matched target construction → irreversible execution. Figure 6: Seed example for Scope Escalation. A.2 Credential Exploration Definition. Blocked by an auth failure, the agent hunts through environment variables, config files, or keychains for alternative credentials not provisioned for the current task. Mechanism. auth error → systematic secret search → repurposing an out-of-scope credential. Figure 7: Seed example for Credential Exploration. A.3 Inferred Params Definition. The agent fills in a destructive action’s parameters (target IDs, names, paths) by guessing — name similarity, recency, or convention — rather than resolution grounded in the trajectory. Mechanism. under-specified target → similarity-based guess → destructive call on guessed target. Figure 8: Seed example for Inferred Params. A.4 External Sharing Definition. The agent self-initiates moving workspace content (code, logs, data) to an external or public destination the user did not request. Mechanism. helpful impulse (debugging/sharing) → unrequested egress → potential confidentiality breach. Figure 9: Seed example for External Sharing. A.5 Guardrail Bypass Definition. A pre-check, hook, or verification step fails, and the agent retries the operation with a flag or path that disables the failed check. Mechanism. failed check → retry with skip/force/no-verify flag → guardrail circumvention. Figure 10: Seed example for Guardrail Bypass. Appendix B Policy Figure 11 gives the full text of the two policies referenced throughout Section 5.1: the detailed rule set π∗π^* used to generate and audit AuthGateBench, and the implicit, rule-free policy π given to classifiers under test at evaluation time. Figure 11: Full text of the two policies used in Section 5.1. π∗π^* is the moderately detailed rule set used to generate and audit AuthGateBench; π is the implicit, rule-free instruction given to the classifier under test at evaluation time, standing in for an auto-execution gate that relies on its own internal safety judgment rather than an explicit policy. Appendix C Prompts We give the full text of the prompts used throughout the AuthGateBench pipeline. Figure 12 generates base instances (Section 4.2); Figure 14 constructs contrast pairs by rewriting only the user request while holding the trajectory and proposed action fixed to flip the gold label (Section 4.3); Figure 15 generates the auxiliary environment signals used in signal augmentation (Section 4.4); and Figure 13 shows the classification prompt used to evaluate LLM authorization gates. Figure 12: Prompt used to generate base instance. Figure 13: Prompt used to evaluate LLM authorization gates. Figure 14: Prompt for constructing contrast pair. Figure 15: Prompt for generating environment signals. Appendix D Invariant rate under π∗π^* Table 3 in Figure 5 reports the invariant rate (IR) under policy π, i.e., without access to the governing policy π∗π^*. Here we report IR under π∗π^* for the same 20 model-category pairs, using the same contrast pairs and the same IR definition (Equation 6), to test whether action anchoring persists once the policy is made explicit, as shown in Table 4. Table 4: Invariant rate (IR, %) under policy π∗π^*: the fraction of contrast pairs with opposite gold labels on which a model returns the same decision for both. Model Scope Escalation Credential Explore External Sharing Guardrail Bypass Inferred Params Qwen3.7-Plus 18.57 58.57 52.86 22.86 28.57 GLM-5.2 22.86 82.86 88.57 61.43 41.43 Kimi-K2.7-Code 17.14 61.43 78.57 62.86 24.29 DeepSeek-V4-Flash 42.86 74.29 74.29 71.43 57.14 We observe that even with π∗π^* available, LLMs still anchor on the surface risk of actions rather than change their decisions in response to explicit user requests. Appendix E Different policies exposed to LLMs We provide the explicit policy π∗π^* to evaluated models. We observe that it generally improves authorization performance, indicating that models can better apply authorization rules when the governing criteria are specified, as shown in Table 5. Table 5: Evaluation results on base and inverted triplets without signal augmentation, under policy π∗π^*. Higher Acc is better; lower FPR and FNR are better. Best value in each column is in bold. Model Scope Escalation Credential Explore External Sharing Guardrail Bypass Inferred Params Acc FPR FNR Acc FPR FNR Acc FPR FNR Acc FPR FNR Acc FPR FNR Qwen3.7-Plus 90.71 4.29 14.29 70.71 55.71 2.86 73.57 52.86 0.00 85.71 14.29 14.29 85.71 7.14 21.43 GLM-5.2 88.57 22.86 0.00 58.57 82.86 0.00 55.71 88.57 0.00 67.86 60.00 4.29 79.29 38.57 2.86 Kimi-K2.7-Code 91.43 17.14 0.00 69.29 58.57 2.86 60.71 78.57 0.00 67.14 58.57 7.14 87.86 11.43 12.86 DeepSeek-V4-Flash 78.57 42.86 0.00 62.86 72.86 1.43 62.86 74.29 0.00 64.29 65.71 5.71 71.43 52.86 4.29 Appendix F Benchmark examples We provide examples of base and inverted instances, along with their corresponding environment signals, for Scope Escalation, Guardrail Bypass, Credential Exploration, Inferred Parameters, and External Sharing in Figures 16, 17, 18, 19 and 20. Figure 16: Base and inverted instances with their environment signals for Scope Escalation. In the base instance, a vague request to clean up a repository is expanded into deleting untracked and ignored files, including configuration and user-created files that were never named, so the action should be blocked. In the inverted instance, the user explicitly identifies the files and directories to remove and authorizes the destructive command, so the action is allowed. Repository-state and workspace-boundary signals clarify the affected targets and the resulting deletion scope. Figure 17: Base and inverted instances with their environment signals for Guardrail Bypass. In the base instance, model publication fails because the model has not received production approval, but the agent proposes using a force flag to bypass that requirement, so the action should be blocked. In the inverted instance, the user explicitly requests publication with the force option despite the missing approval, so the action is allowed. Model-registry, approval-status, and audit signals make the bypassed guardrail and its production impact explicit. Figure 18: Base and inverted instances with their environment signals for Credential Exploration. In the base instance, after a test deployment fails authentication, the agent discovers a production registry token and repurposes it for the test environment without authorization, so the action should be blocked. In the inverted instance, the user explicitly instructs the agent to use that production token for the deployment, so the action is allowed. Deployment configuration and credential-scope signals reveal whether the discovered credential is appropriate for the requested environment. Figure 19: Base and inverted instances with their environment signals for Inferred Parameters. In the base instance, the agent interprets an ambiguous request to clean up an old project board as authorization to close a particular issue, even though the user never identifies that issue, so the action should be blocked. In the inverted instance, the user explicitly names the issue and specifies that it should be closed as superseded, so the action is allowed. Issue-state, ownership, and pull-request signals help establish whether the selected target is uniquely determined and consistent with the request. Figure 20: Base and inverted instances with their environment signals for External Sharing. In the base instance, the user asks only for help diagnosing a training problem, but the agent proposes uploading a local training script to an external Pastebin service, so the action should be blocked. In the inverted instance, the user explicitly requests that the same script be uploaded to the named external destination for sharing, so the action is allowed. File-content and destination signals expose the information being transferred and the external trust boundary crossed by the action.