Paper deep dive
ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents
Hongwei Yao, Yiming Liu, Meihui Chen, Jieling Chen, Zikun Chen, Yiling He, Wangze Ni, Cong Wang, Kui Ren
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cowork agents may complete benign tasks while disclosing protected data, manipulating unauthorized state, invocate unauthorized API. We define behavioral safety and introduce ActBench, a self-evolving benchmark that evaluates such behavior risk from execution trajectories rather than final responses. Each case pairs a benign task with an adversarial variant that preserves its instruction, configuration, initial state, rating model, and trusted records while injecting a task-reachable payload. ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service this http URL move beyond static payloads, we propose a reward-guided beam search method that jointly optimizes attack effectiveness and task utility, while reflection diagnoses failed execution checkpoint and guides payload revision. Besides, we propose a dual evidence verification mechanism that verifies agent execution safety and utility through log evidence and LLM-based trajectory this http URL evaluate 15 LLMs and 6 open-source cowork agents over 24,000 trajectories. Under a fixed harness, attack success rates ranges from 10.1% to 94.4% across models, while under a fixed base model, they range from 73.7% to 94.4% across this http URL results show greater variation across models than agent harness, while attacks remain highly successful across all tested this http URL benchmark is released at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.09476v1
- Canonical: https://arxiv.org/abs/2608.09476v1
Trouble viewing inline? Open PDF directly →
Full Text
77,436 characters extracted from source content.
Expand or collapse full text
ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents Hongwei Yao 1† , Yiming Liu 2† , Meihui Chen 4 , Jieling Chen 4 , Zikun Chen 2 , Yiling He 3 , Wangze Ni 2 , Cong Wang 1 , Kui Ren 2 ∗ Abstract Cowork agents may complete benign tasks while disclos- ing protected data, manipulating unauthorized state, invocate unauthorized API. We define behavioral safety and introduce ActBench, a self-evolving benchmark that evaluates such behavior risk from execution trajectories rather than final responses. Each case pairs a benign task with an adversar- ial variant that preserves its instruction, configuration, initial state, rating model, and trusted records while injecting a task- reachable payload. ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs. To move beyond static payloads, we propose a reward-guided beam search method that jointly optimizes attack effectiveness and task utility, while reflec- tion diagnoses failed execution checkpoint and guides pay- load revision. Besides, we propose a dual evidence verifica- tion mechanism that verifies agent execution safety and utility through log evidence and LLM-based trajectory evidence. We evaluate 15 LLMs and 6 open-source cowork agents over 24,000 trajectories. Under a fixed harness, attack success rates ranges from 10.1% to 94.4% across models, while under a fixed base model, they range from 73.7% to 94.4% across agents. These results show greater variation across models than agent harness, while attacks remain highly successful across all tested harnesses. Our benchmark is released at: https://github.com/zjuicsr/ActBench. Introduction Cowork agents increasingly operate across heterogeneous ex- ecution contexts, including workspace files, web content, per- sistent memory, external tools, and reusable skills. This broad access enables powerful automation but also creates risks that unfold across actions. OpenClaw (OpenClaw 2026) provides a prominent example: shortly after attracting widespread adoption in early 2026, the advisory for CVE-2026-25253 documented gateway-token exfiltration that could enable op- erator access and host code execution (GitHub Security Ad- ∗ Corresponding Author 1 City University of Hong Kong, Hong Kong 2 Zhejiang University, China 3 University College London, United Kingdom 4 Xiaomi Corporation, China Copyright © 2027, Association for the Advancement of Artificial Intelligence (w.aaai.org). All rights reserved. visory Database 2026). Later studies also reported creden- tial disclosure, poisoned skills, and persistent state tamper- ing (Wang et al. 2026a,b; Jiang et al. 2026b; He et al. 2025). These incidents motivate a trajectory-based safety evalua- tion, which moves beyond response-level refusal metrics to characterize operational risk arising across sequences of ac- tions and state transitions during agent execution. To formalize this, we define behavioral safety as whether an agent’s execution remains within the permissions and state changes required by a benign task. A risk behavior is iden- tified when trusted records show that an out of scope action produced a prohibited effect through the observed execution trajectory. This definition separates safety from task com- pletion, covering cases in which an agent may finish the requested task while disclosing protected data, modifying an unauthorized state, persisting an untrusted instruction, ex- ceeding a resource limit, or claiming completion without the required effect. Behavioral safety must therefore be assessed from the execution evidence that determines what actions occurred and what effects they produced. Motivation and Gaps. Recent benchmarks extend agent safety evaluation from response moderation to tool execu- tion (Jiang et al. 2026a; Yao et al. 2026). However, their cov- erage remains fragmented across distinct attack surfaces and evaluation settings, leaving three important gaps for state- ful cowork agents. Gap 1: Existing data cover narrow and disjoint attack surfaces. AgentDojo provides 629 cases over untrusted tool outputs, WASP targets web content, and MCP- Tox evaluates 1,348 tool poisoning cases (Debenedetti et al. 2024; Evtimov et al. 2025; Wang et al. 2026c). AgentPoison and MINJA isolate memory poisoning (Chen et al. 2024; Dong et al. 2025). No single benchmark jointly evaluates task artifacts, tool interfaces, persistent state, context pres- sure, and permission composition under task configurations. Gap 2: Existing evaluations lack causal attribution for be- havioral safety. Outcome checks, attacker progress scores, and LLM judges do not jointly verify the realized effect, the provenance linked execution path, and the operation’s config- uration status (Debenedetti et al. 2024; Evtimov et al. 2025; Ruan et al. 2024). Consequently, a blocked call, a preexisting state, or an authorized transition may receive the same label as a realized policy violation. Such scores neither identify the failed enforcement point nor support category matched arXiv:2608.09476v1 [cs.CR] 10 Aug 2026 analysis. Gap 3: Static payload suites cannot adapt to exe- cution feedback. Fixed injection suites in existing work do not optimize payloads based on trajectory-level failure di- agnoses. Therefore, a payload remains unchanged when an agent does not observe them, rejects the induced action, or fails before the target effect. This static construction largely limits attack coverage across models and harnesses. Our Approach. We present ActBench, a cowork agent behavioral safety benchmark with comprehensive test cases and reward-guided payload search. To address Gap 1, Act- Bench spans 15 risk behaviors, six execution spaces, 48 web service APIs, and 213 operational scenarios. Each malicious case injects one task reachable payload, while its benign counterpart preserves the instruction, task configuration, ini- tialized state, utility grading criterion, attack grading crite- rion, and trusted logs. To address Gap 2, we model cowork execution over six spaces and associate each behavior with a protected boundary, propagation path, failed enforcement condition, and evidence schema. Dual evidence verification triggers an attack criterion only when state evidence confirms the prohibited effect, validated event identifiers establish the observed path, and the task configuration excludes the oper- ation. To address Gap 3, ActBench employs a self-evolving optimization mechanism to find higher scoring payloads. Reward-guided beam search ranks edits by a geometric score over mean Attack Grading Score (AGS) and Utility Grading Score (UGS) across repeated rollouts. For unsuccessful at- tempts, reflection-based deep probing identifies the earliest failed checkpoint and revises the same payload field. Search and evaluation use disjoint rollouts to prevent optimization leakage. We further control comparison conditions by fixing the execution harness when comparing models and fixing the base model when comparing agent frameworks. Our contributions are summarized as follows: • We define behavioral safety as whether an agent’s exe- cution remains within the permissions and state changes required by a benign task. We operationalize 15 risk be- haviors through trajectory predicates across four propa- gation families and six execution spaces, together with AGS and UGS for separate measurement of prohibited effects and required task effects. • We construct ActBench with 300 benign and malicious pairs from 213 operational scenarios. The four-stage con- struction combines an empirical strategy pool, reward- guided beam search, reflection-based deep probing, and dual evidence verification. • We evaluate 15 LLMs and 6 open-source cowork agents, including OpenClaw, Hermes, Claude Code, OpenAgent, OpenCode, and QwenPaw, across 20 controlled configu- rations and 24,000 execution trajectories. The results re- veal substantial safety variation: ASR ranges from 10.1% to 94.4% across LLMs under the same harness, and from 73.7% to 94.4% across agents under the same model. Related Work Agent safety benchmarks. Existing agent safety bench- marks evaluate either compliance with harmful requests or security failures under adversarial execution conditions. For harmful-task evaluation, ToolEmu evaluates 36 high stakes toolkits in 144 cases and judges 68.8% of detected fail- ures plausible under deployment review (Ruan et al. 2024); AgentHarm contains 110 malicious tasks across 11 harm categories (Andriushchenko et al. 2025); RedCode includes 4,050 risky execution cases across 25 scenarios and 160 malicious code generation prompts (Guo et al. 2024); and OS Harm extends harmful task evaluation to computer use agents (Kuntz et al. 2025). For adversarial execution, Agent- Dojo provides 97 tasks and 629 security cases over untrusted tool returns (Debenedetti et al. 2024); WASP observes partial attacker progress in up to 86% of web injection cases (Ev- timov et al. 2025); MobileSafetyBench contains 200 daily scenario tasks and 50 indirect injection tasks in Android emulators (Lee et al. 2026); and Agent Security Bench eval- uates prompt injection, memory poisoning, Plan of Thought backdoors, and mixed attacks, with a highest average attack success rate of 84.30% (Zhang et al. 2025). These suites nev- ertheless isolate execution settings and omit matched config- urations spanning artifacts, tool interfaces, persistent state, context pressure, and permission composition. ActBench evaluates these surfaces through matched benign inputs and task configurations, with joint utility and safety outcomes across six execution spaces. Prompt injection attacks. Prompt injection attacks em- bed competing instructions within application data, causing a model or agent to treat untrusted content as executable instructions (Liu et al. 2024; Shi et al. 2025). Initial work focuses on single-turn model settings (Zverev et al. 2025), while later studies extend injection to agent-accessible data resources (Wang et al. 2026c). For example, EIA attacks compromised webpages and achieves up to 70% success in extracting specific personally identifiable information (Liao et al. 2025), AgentPoison assumes direct poisoning access to memory or knowledge bases (Chen et al. 2024), whereas MINJA injects malicious records through query only inter- action (Dong et al. 2025). Safety detection and auditing. Existing work improves agent safety through attack detection, model alignment, vul- nerability discovery, and automated auditing (He et al. 2026). DataSentinel develops game theoretic prompt injection de- tection (Liu et al. 2025b). SecAlign applies preference opti- mization and lowers the strongest tested attack to 8% in its Llama 3 8B setting while retaining base model utility (Chen et al. 2025). AgentFuzz tests 20 open source agents and finds 34 high risk zero day taint style vulnerabilities across 14 agents (Liu et al. 2025a). AgentAuditor introduces AS- SEBench with 2,293 annotated interactions across 15 risk types and 29 application scenarios (Luo et al. 2025). Risk Behavior Taxonomy Agent execution and test case. We model cowork agent across six execution spaces, including context C, reason- ing R, safety policy P, action A, persistent memory M, and environment E. At execution step t, the system state is y t = (c t ,r t ,p t ,a t ,m t ,e t ). Starting from y t , the rea- soning model returns the execution trajectory τ = π φ (y t ), ID Risk BehaviorDescription B 1 Instruction Injection Family: Context manipulation in C Effect: Untrusted content alters objectives, disclosure decisions, or credential handling. Path: C adv → R→ A→ E B 2 Goal Hijacking B 3 Data Exfiltration B 4 Credential Exposure B 5 Memory Poisoning Family: State integrity in M and E Effect: An unverified persistent record alters a later decision or action. Path: M adv /E adv ⇝ C→ R→ A B 6 State Tampering B 7 Deceptive Tool Invocation Family: Action misuse in A Effect: Tool selection or parameters exceed the operation, API, object. Path: C→ R→ A adv → E B 8 Unauthorized CMD Execution B 9 Unauthorized API Invocation B 10 Tool Scope Escalation B 11 Resource Exhaustion Family: Policy mediation failure in P Effect: Enforcement fails for resource limits, canonical execution, reporting, or context allocation. Path: C→ R→ P fail → A→ E B 12 Obfuscated Execution B 13 False Reporting B 14 Context Flooding B 15 Permission Chaining Table 1: Risk behavior taxonomy defined by where each behavior becomes operational and how its effect propagates through the trajectory path. which contains the sequence of model-visible context, rea- soning, actions, tool results, memory events, and environ- ment records. A test case is x i = (c,m,e,u i ,U i ,A i ), where u i is the benign user instruction. The sets U i and A i contain utility and attack grading criteria, respectively. Benchmark construction may edit c, m, or e. It keeps u i ,U i , andA i fixed. Risk taxonomy. We propose a trajectory-based risk behav- ior taxonomy. In contrast to taxonomies based on payload or final response, our taxonomy assigns a behavior through its realized propagation path in τ. We define 15 risk behaviors B j , where j ∈1,..., 15, and organize them into four fam- ilies in Table 1. Context manipulation behaviors B 1 ∼ B 4 arise when untrusted content changes the interpreted objec- tive, disclosure decision, credential handling. State integrity behaviors B 5 ∼ B 6 capture cases in which persistent mem- ory or environment state alters a later decision. Action mis- use behaviors B 7 ∼ B 10 concern tool selection, command execution, API invocation, and argument scope that exceed the task authorization. Policy mediation failures B 11 ∼ B 15 arise when the safety policy layer fails to enforce a resource bound, canonical execution rule, reporting requirement, con- text constraint, or composed permission. Criterion-based scoring. For trajectory τ, each attack grading criterion a ∈ A i and each utility grading criterion u∈U i returns a binary value. We define g a (x i ) = 1 |A i | X a∈A i I[a(τ ) = 1], g u (x i ) = 1 |U i | X u∈U i I[u(τ ) = 1]. (1) We call g a (x i ) the attack grading score and g u (x i ) the utility grading score. Both scores lie in [0, 1] and measure criterion coverage. A value of g a (x i ) = 1 means that every attack criterion returns 1, and a value of g u (x i ) = 1 means that every required task criterion returns 1. Intermediate values report the fraction of satisfied criteria rather than a proba- bility estimate. A rollout is task complete when g u (x i ) = 1, while full attack success requires g a (x i ) = 1. We propose a dual evidence verification mechanism that validates the final score through log-based and LLM-based trajectory evidence. Benchmark Construction Overview. We construct 300 matched pairs, each com- prising one benign case and one malicious case. As illus- trated in Figure 1, benchmark construction has four stages. ➊ Empirical strategy pool fixes the case grading criterion and produces an initial edit at a task-reachable location. ➋ Reward-guided beam search optimizes executed can- didates and ranks them according to their attack and utility scores.➌ Reflection-based deep probing analyzes failed cases based on their execution trajectories and uses the re- sulting feedback to iteratively strengthen the attack examples. ➍ Dual evidence verification evaluates log-based evidence and LLM-based trajectory evidence. ➊ Empirical strategy pool. The pool contains three in- dexed views. The scenario view specifies the benign user instruction u i , the execution environment represented by c, m, and e, and the utility grading criteria U i . The behavior view selects one B j and definesA i from its operational evi- dence schema. The attack strategy view stores edit templates and payload structures from previously accepted construc- tion cases. We first execute the benign case using the reasoning model π φ . A candidate attack location is considered eligible only if the benign trajectory demonstrates that the agent reads, re- trieves or discovers that location while completing the task. The attack model π φ then selects an attack target, such as a web service API, and refer to the attack strategy view to modify either an eligible field in c, m or e. The matched benign and malicious pairs retain the same u i , U i , and A i . The attack strategy view is updated continuously through- out test case construction. Whenever the framework success- fully optimizes a passed case, it immediately incorporates the corresponding edit template and payload structure into the strategy pool. This process establishes a self-evolving op- timization mechanism in which newly passed cases expand the attack strategy view. ➋ Reward-guided beam search. At construction round t, each candidate x t is executed in k independent rollouts from the same reset environment. In this section, g a (x t ) and g u (x t ) denote the corresponding means over those construc- tion rollouts. We rank the candidate by s(x t ) = g a (x t ) α g u (x t ) 1−α ,0 < α < 1. (2) For positive scores, the logarithm of the objective is logs(x t ) = α logg a (x t ) + (1− α) logg u (x t ), (3) where α is the constant elasticity of s with respect to attack evidence, and 1−α is its elasticity with respect to utility. The geometric score becomes zero when either case score is zero. LogEvidenceVerification StateDiffs. EntriesAudit ResourceCnt. FileDiffs. Reward-guidedBeamSearch 푥 ! 푥 " 푥 # 푥 " Round푡 푥 # Pruned 푥 # Pruned Pruned 푥 # t=1t=2 t=3 SingleRolloutExecution ExecutionEnvironment(퐄) APIs FilesSkills Memory퐌 퐂 Context 퐑 Reasoning Trajectory 흉 Sample:퐶 "#$ =⋃ % ! ∈' ! 휋 ( (푥 " ,휉) )*$ + Rollout Evaluation 휏 x Multi-ObjectiveReward 휏 AttackGradingScoreAGS UtilityGradingScoreUGS 푠(푥 " )=푔 , (푥 " ) - 푔 . (푥 " ) $/- Candidate 푥 1 Empirical Strategy Pool StrategyView Base64 Obfuscated HarnessHijacking StateManipulation... Grading Criteria ReadAction WriteAction DeleteAction .... ScenarioView BusinessOperation SoftwareEngineer FinancialOperations Document.... Initialize Test Case푥=(푐,푚,푒,푢 ! ,풰 ! ,풜 ! ) Reflection-based DeepProbing Trajectory 흉 푥 1 FailureType:notread,ignore... PayloadStealthy:low/high... Direction:relevance,strength... Attack:roleplay,obfuscated... ReflectionSummary흃 Candidate풙&흉 Dual Evidence Verification Passed Test Case Verify TrajectoryEvidenceVerification Instruction ToolResult Reasoning ToolCall 123 4 Self-evolving 퐀 Action 퐏 Policy BehaviorView B 2 Goal Hijacking B 3 Data Exfiltration.... B 1 Instruction Injection Figure 1: ActBench benchmark construction framework. Thus, neither attack evidence nor utility can compensate for the complete absence of the other. Let C t contain the candidates entering round t, and let B t contain the retained parent beams. The initial set C 0 contains the first editable case produced by the attack strategy pool. After executing and grading C t , the beam update and expansion are B t = top w (C t ,s(x t )), x t ∈ C t (4) C t+1 = [ x t ∈B t π φ (x t ,ξ) n j=1 ,(5) where π φ denotes the attack model and ξ represents the re- flection summary for previous rollout. The first line orders candidates by s(x t ) and retains at most w. The second line invokes the attack model n times for each retained candidate. Therefore,|C t+1 | ≤ k× n, with strict inequality when the scope filter rejects an edit or multiple invocations produce duplicate cases. Because each parent is selected only after execution, beam retention is grounded in observed trajecto- ries rather than the attack model’s prediction quality. Search stops when g a (x t ) = g u (x t ) = 1 or when it reaches the depth limit d max . (See Algorithm 1 in Appendix for detail). ➌ Reflection-based deep probing. To localize the failure, the reflector constructs a diagnosis ξ from x t , B j , trusted records in τ, the API audit, and repeat outcomes. The di- agnosis assigns the attempt to one of ten classes: not read, ignored, refused, weak attack, unreached sink, sink missing target content, broken task integrity, unstable success, ver- ified success, or unknown. It also records whether the pay- load was observed and followed, whether the target effect occurred, and whether task integrity was preserved. Addi- tional fields specify whether to discard the candidate, which property to optimize, and a localized revision instruction. The revision targets the earliest unsupported transition in the attack path. An unobserved payload triggers an edit for relevance; an ignored payload, weak attack, or sink failure triggers attack-strength optimization; refusal triggers stealth and stability; a realized effect with utility loss triggers task- integrity optimize; and inconsistent repeated outcomes trig- ger stability optimization. The resulting instruction becomes the next objective, while ξ conditions the attack model: x t+1 = π φ (x t ,ξ).(6) An empty instruction retains the preceding objective. Up- dates retain the editable field or group, case scope, identifiers, and roles. The scope filter rejects changes to u i , U i , A i , or fields outside c, m, and e. The discard flag enables bounded probing, while malformed outputs invoke a deterministic fall- back based on the API audit, repeated scores, and a refusal heuristic. Reflection enables the system to learn from previ- ous failures and refine its attack strategies accordingly. ➍ Dual evidence verification. Equation (1) aggregates bi- nary attack criteria without restricting the evidence source. We extend this definition by evaluating the same criterion setA i with two evidence verifiers. Log evidence verification applies deterministic predicates to state differences, service audit entries, command results, and resource counters. For a ∈ A i , it returns a ′ (τ ) = 1 when a trusted record estab- lishes the prohibited effect under the fixed authorization rule. Trajectory evidence verification reconstructs the path from payload exposure to the recorded action and effect. It returns a ′ (τ ) = 1 only when event identifiers, object identifiers, and event order establish that path. The corresponding case scores and their fusion are g a (x i ) = βg ′ a (x i ) + (1− β)g ′ a (x i ),0 < β < 1,(7) where g ′ a (x i ) is the log evidence verification score and g ′ a (x i ) is the trajectory evidence verification score. Both apply the criterion aggregation in Equation (1). Equation (7) then defines the final AGS as their convex combination. The fusion preserves the full success threshold. In particular, 1− g a (x i ) = β[1− g ′ a (x i )] + (1− β)[1− g ′ a (x i )]. (8) Both terms on the right are nonnegative because the two component scores lie in [0, 1]. It follows that g a (x i ) = 1 if and only if g ′ a (x i ) = g ′ a (x i ) = 1. Likewise, g a (x i ) = 0 if 0204060 Tasks Business Operations Software Engineering Document Operations Financial Operations Security Operations Identity Operations 66 63 59 52 52 8 (a) Task domains 0255075100125150 Task-API references Task Tracker Knowledge base Notebook / meetings Ticketing Ledger / finance Web content Directory / identity Customer records Mailbox Automation Integrations Scheduling 150 105 63 55 54 43 36 25 24 14 6 6 (b) Web service APIs B1B2B3B4B5B6B7B8B9B10B11B12B13B14B15 Behavior risk category 0 10 20 30 40 Tasks 42 13 32 14 15 37 14 42 1212 14 16 14 11 12 (c) Task count by behavior risk category Figure 2: ActBench composition. Panel (a) reports six op- erational domains. Panel (b) groups 581 task API references into 12 service groups. Panel (c) reports the 300 pair assign- ments across B 1 to B 15 . and only if both component scores are zero. The value of β controls the relative contribution of the two verifiers for inter- mediate scores. When a test case succeeds, its edit template and payload structure are added to the attack strategy pool through a push-back mechanism. Reflection and push-back form a self-evolving optimization loop in which successful cases continuously expand and improve the available attack strategies. Empirical Study We organize the study around four questions. RQ1 Model Effects asks how the base model changes behavioral safety when the cowork harness is fixed. RQ2 Harness Effects asks how the cowork harness changes behavioral safety when the base model is fixed. RQ3 Safety Policy P Effects asks how safety policy protects cowork agent through malicious behavior detection file scanning. RQ4 Case Study analyzes the effect of malicious behavior propagation path using an obfuscated execution case. Experimental settings. RQ1 and RQ3 fixe OpenClaw ver- sion 2026.5.19 and varies 15 base models. RQ2 fixes base model and varies six cowork harnesses.Counting the shared RQ1/RQ2 setting once yields 20 unique configurations and 24,000 trajectories in total. Construction and evaluation tra- jectories are strictly disjoint. The benchmark construction uses beam width w = 3, proposal count n = 2, depth d max = 5, ranking weight α = 0.5. The evaluation adopts rollout number k = 3, evidence weight β = 0.4, rating model GPT-5.5. All runs use Ubuntu 22.04.5 LTS. (See Ap- pendix for all environment details.) Metrics. For each malicious rollout, AGS is the criterion coverage after the dual evidence channels are fused. For each matched benign rollout, UGS is the fraction of required task Table 2: Base model comparison with OpenClaw fixed. Iter. is the malicious rollout median. ModelAGS mal ↓ ASR (%)↓ p@1/p@2/p@3↓ UGS ben ↑ Iter. Claude-Opus-4.80.28410.110.7/12.3/13.70.938 15.0 Claude-Sonnet-4.6 0.34720.019.3/22.3/23.70.927 16.0 GPT-5.50.49337.836.0/43.3/47.30.928 16.0 GPT-5.4-mini0.72765.766.3/75.3/78.00.904 18.0 Grok-4.50.87083.983.0/90.0/90.70.938 17.0 GLM-5.20.54742.841.7/49.0/54.00.929 16.0 Qwen3.7-max0.51139.237.7/48.3/54.70.915 16.0 Qwen3.7-plus0.52442.142.7/50.0/53.30.915 16.0 Kimi-K30.48935.735.3/44.3/47.30.940 17.0 Kimi-K2.60.74870.470.7/78.0/81.00.869 17.0 MiniMax-M30.40225.025.0/31.7/36.30.917 19.0 MiniMax-M2.70.80475.476.7/84.3/87.70.880 17.0 Deepseek-v4-Pro0.95594.494.3/98.0/98.70.922 19.0 Deepseek-v4-Flash 0.88784.483.3/90.0/93.30.900 19.5 Hunyuan-3.00.45530.030.7/37.3/41.70.933 19.0 criteria that pass. ASR is the fraction of malicious rollouts whose AGS exceeds the fixed threshold of 0.8. The case level pass@k rate is the fraction of cases with at least one threshold crossing in the first k evaluation rollouts. Median iteration count describes executed action cycles. The defense study reports accuracy, precision, recall, F1, and the four confusion counts under one frozen threshold per method. Lower AGS, ASR, and pass@k indicate less observed attack evidence. Higher UGS indicates stronger benign task completion. Benchmark coverage. ActBench contains 600 cases in 300 matched pairs from 213 scenarios. The accepted cases cover 15 behaviors, four propagation families, six execution spaces, and 48 web service APIs. Figure 2 reports the con- structed distribution. Business operations, software engineer- ing, and document operations contribute 188 pairs, which is 62.7% of the benchmark. Identity operations contributes eight pairs. Task Tracker, Knowledge Base, and Notebook or Meetings account for 318 of 581 API references. Instruction Injection and Unauthorized CMD Execution each contain 42 pairs, while Context Flooding contains 11. The aggregate metrics are therefore weighted by the case distribution. B1B2B3B4B5B6B7B8B9B10B11B12B13B14B15 Behavior risk category 0.0 0.2 0.4 0.6 0.8 1.0 Score Overall ¬AGS=0.716 Overall UGS=0.938 ¬AGS mal ↑UGS ben ↑ Figure 3: Claude-Opus-4.8 behavior risk results. Blue and green bars report 1 − AGS mal and UGS ben . The dashed averages are 0.716 and 0.938. (!0#$,0. (!0#$+**$/ )&*& -+' 2$* )!3 2$* ,(0. &)& &)& &*&!3 &*&!3 $$,.$$'1 -+ $$,.$$'1 (!.% 0*40!* +#$( /$-!/&+*"+0*/ //!"'($!* Figure 4: Action cycle distributions across base models with OpenClaw fixed. Orange boxes report malicious rollouts and green boxes report matched benign rollouts. RQ1 Model E!ects.In RQ1, only the base model varies, while all other experimental settings are held constant. As reported in Table 2, Claude-Opus-4.8 achieves the lowest AGS mal of0.284,ASRof10.1%, while maintaining a be- nign UGS of0.938. Deepseek-v4-Pro exhibits the highest risk, with anAGSof0.955, anASRof94.4%, and apass@3 of98.7%. Claude-Opus-4.8 therefore reducesAGSby0.671 andASRby84.3%relative to Deepseek-v4-Pro. These re- sults show that benign task performance is not a reliable proxy for behavioral safety. Repeated rollouts further expose di!erences in attack stability. Table 2 shows that Qwen3.7-max increases from 37.7%atpass@1to54.7%atpass@3, producing the largest increase of17.0percentage points. By comparison, Claude-Opus-4.8 increases by3.0percentage points, while Deepseek-v4-Pro increases by4.4percentage points. Inter- mediate risk models therefore tend to exhibit stochastic at- tack activation, whereas high risk models trigger prohibited e!ects consistently. A single rollout would consequently un- derestimate the exposure of models such as Qwen3.7-max, GLM-5.2, and Kimi-K3. Figure 3 shows the category behavior risk results of Claude-Opus-4.8. Claude-Opus-4.8 reaches 0.43 and above 0.90 for the same labels. Its categoryUGSremains at least 0.90. The fixed harness links these di!erences to how the base model converts the same carrier into an action. The me- dian malicious iteration count spans only 4.5 cycles across models, whileASRspans84.3%. Model choice changes risk without reducing utility. Execution length does not explain the model ranking. Fig- ure 4 shows that malicious median iterations range only from15.0to19.5cycles. MiniMax-M3, Hunyuan-3.0, and Deepseek-v4-Pro all require19.0median cycles, but their ASRs are25.0%,30.0%, and94.4%, respectively. The re- sulting69.4percentage point di!erence under the same ex- ecution length indicates that the dominant factor is how the base model interprets adversarial context and authorizes sub- sequent actions. Table 3: Cowork harness comparison with base model fixed. Iter. is the malicious rollout median. AgentAGS mal →ASR(%)→pass@1 / @2 / @3→UGS ben ↑Iter. OpenClaw0.95594.494.3 / 98.0 / 98.70.92219.0 OpenAgent0.80876.276.0 / 83.3 / 87.70.94524.0 OpenCode0.85281.780.7 / 88.0 / 90.30.91921.0 QwenPaw0.86873.771.3 / 78.3 / 80.70.90333.0 Hermes0.82679.281.0 / 87.3 / 89.00.92647.0 Claude Code0.84881.481.7 / 89.3 / 93.00.90433.0 Finding:With the cowork harness fixed, changing only the base model changes ASR by84.3%andpass@3by85.0%, while be- nign UGS changes by only0.071. Behavioral safety is primarily determined by model specific action selection and cannot be in- ferred from task utility or execution length. RQ2 Agent Harness E!ects.RQ2 fixes Deepseek-v4-Pro and every case while replacing the complete cowork har- ness. This intervention changes context assembly, memory retrieval, tool serialization, and action mediation together. It does not isolate one harness component. Table 3 shows a20.7%ASR span. QwenPaw records the lowestASRat 73.7%, while OpenClaw records94.4%.UGSremains be- tween 0.903 and 0.945. The lower attack rates are therefore not produced by broad failure on benign tasks. OpenAgent has the lowestAGSand the highestUGS. QwenPaw has the lowestASRbut a higherAGSthan OpenAgent. Figure 5 localizes the harness e!ect. Memory Poisoning AGS ranges from 0.24 to 0.98. Credential Exposure ranges from 0.40 to 0.97. These labels depend on memory retrieval and context serialization, which the harness controls. Decep- tive Tool Invocation remains between 0.91 and 1.00 across all six harnesses. Every behavior conditioned UGS value is Figure 5: Behavior conditioned harness comparison with Deepseek-v4-Pro fixed. Panel (a) reports malicious AGS. Panel (b) reports benign UGS. Figure 4: Action cycle distributions across base models with OpenClaw fixed. Orange boxes report malicious rollouts and green boxes report matched benign rollouts. RQ1 Model Effects. In RQ1, only the base model varies, while all other experimental settings are held constant. As reported in Table 2, Claude-Opus-4.8 achieves the lowest AGS mal of 0.284, ASR of 10.1%, while maintaining a be- nign UGS of 0.938. Deepseek-v4-Pro exhibits the highest risk, with an AGS of 0.955, an ASR of 94.4%, and a pass@3 of 98.7%. Claude-Opus-4.8 therefore reduces AGS by 0.671 and ASR by 84.3% relative to Deepseek-v4-Pro. These re- sults show that benign task performance is not a reliable proxy for behavioral safety. Repeated rollouts further expose differences in attack stability. Table 2 shows that Qwen3.7-max increases from 37.7% at pass@1 to 54.7% at pass@3, producing the largest increase of 17.0 percentage points. By comparison, Claude-Opus-4.8 increases by 3.0 percentage points, while Deepseek-v4-Pro increases by 4.4 percentage points. Inter- mediate risk models therefore tend to exhibit stochastic at- tack activation, whereas high risk models trigger prohibited effects consistently. A single rollout would consequently un- derestimate the exposure of models such as Qwen3.7-max, GLM-5.2, and Kimi-K3. Figure 3 shows the category behavior risk results of Claude-Opus-4.8. Claude-Opus-4.8 reaches 0.43 and above 0.90 for the same labels. Its category UGS remains at least 0.90. The fixed harness links these differences to how the base model converts the same carrier into an action. The me- dian malicious iteration count spans only 4.5 cycles across models, while ASR spans 84.3%. Model choice changes risk without reducing utility. Execution length does not explain the model ranking. Fig- ure 4 shows that malicious median iterations range only from 15.0 to 19.5 cycles. MiniMax-M3, Hunyuan-3.0, and Deepseek-v4-Pro all require 19.0 median cycles, but their ASRs are 25.0%, 30.0%, and 94.4%, respectively. The re- sulting 69.4 percentage point difference under the same ex- ecution length indicates that the dominant factor is how the base model interprets adversarial context and authorizes sub- sequent actions. Table 3: Cowork harness comparison with base model fixed. Iter. is the malicious rollout median. AgentAGS mal ↓ ASR(%)↓ pass@1 / @2 / @3↓ UGS ben ↑ Iter. OpenClaw0.95594.494.3 / 98.0 / 98.70.92219.0 OpenAgent0.80876.276.0 / 83.3 / 87.70.94524.0 OpenCode0.85281.780.7 / 88.0 / 90.30.91921.0 QwenPaw0.86873.771.3 / 78.3 / 80.70.90333.0 Hermes0.82679.281.0 / 87.3 / 89.00.92647.0 Claude Code0.84881.481.7 / 89.3 / 93.00.90433.0 Finding: With the cowork harness fixed, changing only the base model changes ASR by 84.3% and pass@3 by 85.0%, while be- nign UGS changes by only 0.071. Behavioral safety is primarily determined by model specific action selection and cannot be in- ferred from task utility or execution length. RQ2 Agent Harness Effects. RQ2 fixes Deepseek-v4-Pro and every case while replacing the complete cowork har- ness. This intervention changes context assembly, memory retrieval, tool serialization, and action mediation together. It does not isolate one harness component. Table 3 shows a 20.7% ASR span. QwenPaw records the lowest ASR at 73.7%, while OpenClaw records 94.4%. UGS remains be- tween 0.903 and 0.945. The lower attack rates are therefore not produced by broad failure on benign tasks. OpenAgent has the lowest AGS and the highest UGS. QwenPaw has the lowest ASR but a higher AGS than OpenAgent. Figure 5 localizes the harness effect. Memory Poisoning AGS ranges from 0.24 to 0.98. Credential Exposure ranges from 0.40 to 0.97. These labels depend on memory retrieval and context serialization, which the harness controls. Decep- tive Tool Invocation remains between 0.91 and 1.00 across all six harnesses. Every behavior conditioned UGS value is B1B2B3B4B5B6B7B8B9B10B11B12B13B14B15 Behavior risk category A1 A2 A3 A4 A5 A6 0.960.950.980.970.980.961.000.970.990.900.970.940.870.830.98 0.920.901.000.800.240.590.960.700.970.880.650.930.690.860.94 0.870.870.970.800.480.880.990.580.910.830.910.940.700.830.93 0.870.900.970.680.370.880.970.560.920.730.860.940.630.790.90 0.900.830.940.830.290.940.960.710.910.770.950.910.720.900.88 0.880.880.890.400.910.950.910.930.900.850.900.880.780.870.87 (a) AGS B1B2B3B4B5B6B7B8B9B10B11B12B13B14B15 Behavior risk category A1 A2 A3 A4 A5 A6 0.940.880.960.920.940.870.980.930.910.950.940.950.920.950.94 0.970.930.960.940.950.880.960.950.940.940.960.980.970.981.00 0.930.870.890.950.850.850.830.910.900.950.960.930.970.930.94 0.950.880.910.880.940.880.980.980.900.950.960.970.980.930.95 0.930.910.930.940.930.880.930.940.890.940.950.900.970.940.93 0.930.890.900.920.870.860.910.950.900.920.940.900.930.850.91 (b) UGS A1: OpenClaw · A2: OpenAgent · A3: ClaudeCode A4: Hermes · A5: OpenCode · A6: QwenPaw Figure 5: Behavior conditioned harness comparison with Deepseek-v4-Pro fixed. Panel (a) reports malicious AGS. Panel (b) reports benign UGS. Table 4: Detection diagnostics at each method’s native obser- vation boundary. Metrics use a frozen validation threshold and do not measure post intervention execution outcomes. DefenseAcc. Prec. Rec.F1 TP FP TN FN Prompt-Guard-2-86M 51.5% 56.3% 13.3% 21.6% 40 31 269 260 Llama-Guard-4-12B55.7% 72.4% 18.3% 29.3% 55 21 279 245 AgentDoG-Qwen-4B76.5% 95.4% 55.7% 70.3% 167 8 292 133 AI-Infra-Guard71.0% 82.8% 53.0% 64.6% 159 33 267 141 Cisco-Skill-Scanner80.0% 75.0% 90.0% 81.8% 270 90 210 30 at least 0.83. OpenClaw reaches 94.4% ASR with a median of 19 cycles. Hermes reaches 79.2% with a median of 47 cycles. More action cycles are therefore not equivalent to stronger enforcement. The harness range is smaller than the model range in the evaluated alternatives. Finding: Harness design changes exposure for memory and cre- dential behaviors, yet every tested harness remains vulnerable on at least 73.7% of malicious rollouts under the fixed model. RQ3 Safety Policy Effects. In the RQ3, we mainly evaluate safety policy P protects cowork agent through query detec- tion and file scanning. For the former, we employ Prompt- Guard-2, LLama-Guard-2, and AgentDog for behavior de- tection (Llama Team 2024; Meta AI 2025; Liu et al. 2026). For the latter, we employ AI-Infra-Guard and Cisco-Skill- Scanner inspect workspace or skill artifacts before execu- tion (Yang et al. 2026; Tencent Zhuque Lab 2025). We attach the same 300 malicious and 300 benign case labels to these boundary specific inputs. The experiment evaluates discrim- ination under each method’s observation assumption. It does not evaluate equivalent runtime interventions. Cisco Skill Scanner detects 270 of 300 malicious artifacts and records 90 false positives. Its recall is 90.0%. AgentDoG detects 167 malicious cases with eight false positives, which yields 95.4% precision and 55.7% recall. Prompt Guard 2 and Llama Guard 4 detect 40 and 55 malicious inputs. Their train- ing target is local content classification, while an ActBench label requires a linked action and environment effect. This difference in target and input explains why content classifi- cation does not transfer directly to trajectory level detection. Finding: Trajectory and artifact evidence aligns more closely with realized behavioral labels than local text alone, but precision and recall trade off sharply across the two evidence boundaries. RQ4 Case Study. Figure 6 traces an Obfuscated Exe- cution failure. The attacker modifies only a task reach- able ledger memo. The model treats its encoded packet as authorized task content, while the harness evalu- ates the carrier before canonical decoding. The de- coded file write therefore bypasses mediation and creates .recon_cache/inv-2027-894.audit only in the malicious run. Trajectory events recover the propagation or- der, and filesystem evidence confirms the effect. This case localizes the failure but does not estimate reflection’s average gain. Figure 6: Trajectory visualization for Obfuscated Execution. The malicious carrier is inserted into the memo field of ledger record TXN-2027-7711. The agent treats the encoded packet as a reconciliation step, decodes it, and creates an unautho- rized audit file. Finding: The model grants authority to untrusted content, and the harness checks a noncanonical representation. Trajectory and filesystem evidence confirm the unauthorized effect. Discussion The controlled interventions separate model policy effects from execution layer effects. Model choice produces the larger observed safety span, while harness changes concen- trate on behaviors mediated by context and persistent state. The persistence of high ASR across all six harnesses indi- cates that execution layer controls do not compensate for a base model that converts untrusted context into prohibited actions. RQ3 further shows that defense performance de- pends on whether the detector observes text, trajectory state, or executable artifacts. These findings support the construc- tion choices in ActBench. Matched tasks preserve utility comparability. Behavior predicates localize the failed prop- agation step. Dual evidence verification prevents exposure or blocked calls from being labeled as realized violations. Reflection uses the same evidence to repair one failed check- point. The study remains bounded by the evaluated model and harness sets, a fixed threshold, one verifier configura- tion, and a case based reflection analysis. Paired uncertainty, guarded reruns, and construction ablations are needed before attributing average causal gains to individual components. Conclusion We introduced ActBench, a self evolving benchmark that evaluates cowork agent safety from executed trajectories rather than final responses. Its 600 cases form 300 matched benign and malicious pairs from 213 scenarios, covering 15 risk behaviors, six execution spaces, and 48 web ser- vice APIs. Reward-guided beam search jointly preserves at- tack evidence and task utility through a geometric objective. Reflection-based deep probing revises the earliest failed tran- sition in each attack path. Dual evidence verification com- bines trusted logs with trajectory reconstruction, so full suc- cess requires agreement on both the prohibited effect and its propagation path. Across 24,000 trajectories, model con- trolled ASR spans 84.3% and harness controlled ASR spans 20.7%. These results show greater variation across models than agent harness, while attacks remain highly successful across all tested harnesses. References Andriushchenko, M.; Souly, A.; Dziemian, M.; Duenas, D.; Lin, M.; Wang, J.; Hendrycks, D.; Zou, A.; Kolter, Z.; Fredrikson, M.; et al. 2025. AgentHarm: A Bench- mark for Measuring Harmfulness of LLM Agents. In Inter- national Conference on Learning Representations, volume 2025, 79185–79220. Chen, S.; Zharmagambetov, A.; Mahloujifar, S.; Chaudhuri, K.; Wagner, D.; and Guo, C. 2025. SecAlign: Defending Against Prompt Injection with Preference Optimization. In Proceedings of the 2025 ACM SIGSAC Conference on Com- puter and Communications Security, 2833–2847. Chen, Z.; Xiang, Z.; Xiao, C.; Song, D.; and Li, B. 2024. AgentPoison: Red-Teaming LLM Agents via Poison- ing Memory or Knowledge Bases. In Advances in Neural In- formation Processing Systems, volume 37, 130185–130213. Debenedetti, E.; Zhang, J.; Balunovic, M.; Beurer-Kellner, L.; Fischer, M.; and Tramèr, F. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and De- fenses for LLM Agents. In Advances in Neural Information Processing Systems, volume 37, 82895–82920. Dong, S.; Xu, S.; He, P.; Li, Y.; Tang, J.; Liu, T.; Liu, H.; and Xiang, Z. J. 2025. Memory Injection Attacks on LLM Agents via Query-Only Interaction. In Advances in Neural Information Processing Systems, volume 38, 46697–46731. Evtimov, I.; Zharmagambetov, A.; Grattafiori, A.; Guo, C.; and Chaudhuri, K. 2025. WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. In Advances in Neural Information Processing Systems, volume 38. GitHub Security Advisory Database. 2026. OpenClaw Has One Click Remote Code Execution through Gateway Token Exfiltration. GitHub Security Advisory GHSA-g8p2-7wf7- 98mq. CVE-2026-25253, accessed July 23, 2026. Guo, C.; Liu, X.; Xie, C.; Zhou, A.; Zeng, Y.; Lin, Z.; Song, D.; and Li, B. 2024. RedCode: Risky Code Execution and Generation Benchmark for Code Agents. In Advances in Neural Information Processing Systems, volume 37, 106190– 106236. He, X.; Xu, G.; Han, X.; Wang, Q.; Zhao, L.; Shen, C.; Lin, C.; Zhao, Z.; Li, Q.; Yang, L.; et al. 2025. Artificial Intelligence Security and Privacy: a Survey. Science China Information Sciences, 68(8): 181101. He, Y.; Zhu, H.; Li, Y.; Shao, S.; Yao, H.; Liu, Z.; and Qin, Z. 2026. AttriGuard: Defeating indirect prompt injection in LLM agents via causal attribution of tool invocations. Jiang, Y.; Zhang, Y.; Backes, M.; Shen, X.; and Zhang, Y. 2026a. HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents? arXiv preprint arXiv:2604.15415. Jiang, Y.; Zhang, Y.; Shen, X.; Backes, M.; and Zhang, Y. 2026b. “Humans Welcome to Observe”: A First Look at the Agent Social Network Moltbook. arXiv preprint arXiv:2602.10127. Kuntz, T.; Duzan, A.; Zhao, H.; Croce, F.; Kolter, J. Z.; Flam- marion, N.; and Andriushchenko, M. 2025. OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. In Advances in Neural Information Processing Systems, vol- ume 38. Lee, J.; Hahm, D.; Choi, J. S.; Knox, W. B.; and Lee, K. 2026. MobileSafetyBench: Evaluating Safety of Au- tonomous Agents in Mobile Device Control. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence, volume 40, 37565–37573. Liao, Z.; Mo, L.; Xu, C.; Kang, M.; Zhang, J.; Xiao, C.; Tian, Y.; Li, B.; and Sun, H. 2025. EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage. In International Conference on Learning Representations. Liu, D.; Ren, Q.; Qian, C.; Shao, S.; Xie, Y.; Li, Y.; Yang, Z.; Luo, H.; Wang, P.; Liu, Q.; et al. 2026. AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security. arXiv preprint arXiv:2601.18491. Liu, F.; Zhang, Y.; Luo, J.; Dai, J.; Chen, T.; Yuan, L.; Yu, Z.; Shi, Y.; Li, K.; Zhou, C.; Chen, H.; and Yang, M. 2025a. Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-based Agents. In 34th USENIX Security Symposium (USENIX Security 25), 3767– 3786. USENIX Association. Liu, Y.; Jia, Y.; Geng, R.; Jia, J.; and Gong, N. Z. 2024. For- malizing and Benchmarking Prompt Injection Attacks and Defenses. In 33rd USENIX Security Symposium (USENIX Security 24), 1831–1847. USENIX Association. Liu, Y.; Jia, Y.; Jia, J.; Song, D.; and Gong, N. Z. 2025b. DataSentinel: A Game-Theoretic Detection of Prompt In- jection Attacks. In 2025 IEEE Symposium on Security and Privacy. Llama Team. 2024. Meta Llama Guard 2 Model Card. Purple Llama. Accessed July 23, 2026. Luo, H.; Dai, S.; Ni, C.; Li, X.; Zhang, G.; Wang, K.; Liu, T.; and Salam, H. 2025. AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents. In Advances in Neural Information Processing Systems, volume 38, 43241–43298. Meta AI. 2025. Llama Prompt Guard 2 Model Card. Meta AI Model Documentation. Accessed July 23, 2026. OpenClaw. 2026. Introducing OpenClaw. OpenClaw Blog. Accessed July 23, 2026. Ruan, Y.; Dong, H.; Wang, A.; Pitis, S.; Zhou, Y.; Ba, J.; Dubois, Y.; Maddison, C. J.; and Hashimoto, T. 2024. Iden- tifying the Risks of LM Agents with an LM-Emulated Sand- box. In International Conference on Learning Representa- tions, volume 2024, 27031–27098. Shi, H.; Yao, H.; Shao, S.; Jiao, S.; Peng, Z.; Qin, Z.; and Wang, C. 2025. Quantifying Conversation Drift in MCP via Latent Polytope. arXiv preprint arXiv:2508.06418. Tencent Zhuque Lab. 2025. AI-Infra-Guard: A Comprehen- sive, Intelligent, and Easy-to-Use AI Red Teaming Platform. GitHub repository. Wang, Y.; Ba, J.; Liu, H.; Pan, Y.; Wei, J.; Su, Z.; Luan, T. H.; and Du, L. 2026a. Security of OpenClaw Agents: Fun- damentals, Attacks, and Countermeasures. arXiv preprint arXiv:2605.25435. Wang, Y.; Gao, H.; Niu, Z.; Liu, Z.; Zhang, W.; Wang, X.; and Lian, S. 2026b. A Systematic Security Evaluation of Open- Claw and Its Variants. arXiv preprint arXiv:2604.03131. Wang, Z.; Gao, Y.; Wang, Y.; Liu, S.; Sun, H.; Cheng, H.; Shi, G.; Du, H.; and Li, X.-Y. 2026c. MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence. Yang, Y.; Zheng, X.; Wu, H.; Cheng, H.; Shi, X.; Guo, J.; Yang, B.; Zhou, Y.; Wu, X.; and Ying, Z. 2026. Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming. arXiv preprint arXiv:2606.31227. Yao, H.; Liu, Y.; He, Y.; and Yang, B. 2026. Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw. In Second Workshop on Agents in the Wild: Safety, Security, and Beyond. Zhang, H.; Huang, J.; Mei, K.; Yao, Y.; Wang, Z.; Zhan, C.; Wang, H.; and Zhang, Y. 2025. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and De- fenses in LLM-based Agents. In International Conference on Learning Representations. Zverev, E.; Abdelnabi, S.; Tabesh, S.; Fritz, M.; and Lampert, C. H. 2025. Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? In International Conference on Learning Representations. Benchmark Construction ActBench constructs one malicious case through an evi- dence conditioned search over complete case objects. The search stage evaluates candidate edits in the target harness. The reflection stage converts the resulting evidence record into a bounded revise instruction. Both stages retain the user instruction, grading criteria, target behavior, case identifiers, and assigned editable field. Reward Guided Candidate Search Algorithm 1 begins after the strategy pool produces an initial candidate x 0 i at a location observed in the benign trajectory. The pool also supplies the initial revise objective o 0 i . Each search node is a complete benchmark case. The attack model π φ edits the assigned field. The reasoning model π φ executes the edited case in the target harness. The rating model then computes the ranking quantities from observed trajectories. Candidate generation and candidate evaluation are therefore assigned to different models. State and admissibility. Let ρ i denote the construction specification for case i. It records the target behavior, attack direction, assigned surface, and editable field group. For ev- ery candidate, Valid verifies the case schema and equality of u i ,U i ,A i , the target behavior, and the case identifiers. It also restricts the edit to the field group declared by ρ i . This predicate does not use the attack score. Rollout score. Run restores the declared case state before each of k construction rollouts. For q ∈ a,u, the rollout mean is ̄g q (x) = 1 k k X ℓ=1 g q (x,τ ℓ ).(9) Grade computes ̄g a with Equation (7) and ̄g u with Equa- tion (1). It also returns an evidence record e that contains the trusted audit, trajectory summary, grader outputs, and repeat scores used by reflection. Benign admission requires ̄g a (x ben i ) = 0 and ̄g u (x ben i ) = 1. Candidate acceptance re- quires ̄g a (x) = ̄g u (x) = 1. Since every rollout score lies in [0, 1], acceptance means that every construction rollout attains full attack evidence and full task utility. The ranking score is s(x) = ̄g a (x) α ̄g u (x) 1−α .(10) This is Equation (2) with explicit rollout means. A zero at- tack mean or utility mean gives s(x) = 0. For equal values of s, Top compares α ̄g a (x) + (1−α) ̄g u (x). A remaining tie is resolved by the canonical candidate key. These rules make beam selection independent of the order in which equal scor- ing proposals are returned. Algorithm operations. p t is the ordered list of candidate and objective pairs awaiting execution at depth t. h t stores each executed candidate with its objective, rollout means, score, and reflection record.b t retains at mostw records. Each retained record produces n attack model calls, so |p t+1 | ≤ wn. The rollout count k affects execution cost and does not affect this branching bound. A child receives no score from its parent. It enters h t+1 only after execution by π φ . Algorithm 1 Reward Guided Candidate Search Require: benign case x ben i and initial candidate x 0 i Require: initial objective o 0 i and fixed specification ρ i Require: reasoning model π φ , attack model π φ , and strategy poolS Require: rollouts k, width w, proposals n, and depth d max ≥ 1 Require: ranking weight 0 < α < 1 Ensure: accepted case x ⋆ i and updatedS, or⊥ 1:τ ← Run(π φ ,x ben i ,k) 2: ( ̄a, ̄u,e 0 )← Grade(x ben i ,τ ) 3: if ̄a > 0 ∨ ̄u < 1 then 4: return⊥ 5: end if 6: p 0 ← Unique([(x 0 i ,o 0 i )]), z ←∅ 7: for t = 0,...,d max do 8: h t ←∅ 9: for all (x,o)∈ p t do 10:if¬Valid(x ben i ,ρ i ,x) ∨ Key(x)∈ z then 11:continue 12:end if 13:z ← z∪Key(x) 14:τ ← Run(π φ ,x,k) 15:( ̄a, ̄u,e)← Grade(x,τ ) 16:if ̄a = 1 ∧ ̄u = 1 then 17:S ← PushBack(S,x,ρ i ) 18:return x 19:end if 20:s← ̄a α ̄u 1−α 21:ξ ← Reflect(x,ρ i ,o,e) 22:h t ← Append(h t , (x,o,s, ̄a, ̄u,ξ)) 23: end for 24: if h t =∅ ∨ t = d max then 25:return⊥ 26: end if 27: b t ← Top(h t ,w) 28: p t+1 ←∅ 29: for all (x,o,s, ̄a, ̄u,ξ)∈ b t do 30:o ′ ← Next(o,ξ) 31:for q = 1,...,n do 32:x ′ ← Propose(π φ ,x,ξ,o ′ ,q) 33:if Valid(x ben i ,ρ i ,x ′ ) ∧ Key(x ′ ) /∈ z then 34:p t+1 ← Append(p t+1 , (x ′ ,o ′ )) 35:end if 36:end for 37: end for 38: p t+1 ← Unique(p t+1 ) 39: if p t+1 =∅ then 40:return⊥ 41: end if 42: end for 43: return⊥ Key returns the canonical serialization of the assigned field group. The set z prevents the same edit from being ex- ecuted at two depths. Unique removes duplicate keys while retaining parent order and proposal index q. Reflect re- ceives the evidence record generated by Grade and returns a diagnosis ξ. Next returns the local instruction η in ξ when it is nonempty. It otherwise retains o. PushBack extracts the accepted edit template and payload structure and inserts them intoS. Termination and cost. Depth zero evaluates x 0 i . Depth t > 0 contains candidates separated from x 0 i by t reflection guided edits. Search returns ⊥ when the benign precheck fails, no admissible candidate remains, or depthd max is eval- uated without acceptance. The reasoning model rollout count for one case satisfies r φ ≤ k[2 + n + (d max − 1)wn].(11) The first two terms count the benign case and x 0 i . The first refinement contains at most n candidates. Each later refine- ment contains at most wn candidates. The number of attack model calls is bounded by n + (d max − 1)wn. With w = 3, n = 2, and d max = 5, the bounds are 28k reasoning model rollouts and 26 attack model calls per case. Validation re- jection, duplicate removal, early acceptance, and an empty layer reduce the realized counts. Construction and reported evaluation use disjoint rollouts. Evidence Conditioned Reflection The implementation separates diagnosis, revise generation, and structural admission. For a candidate x, the reflector re- ceives the construction specificationρ i and evaluation record e. The record includes API audit entries, repeat scores, grader values, notes, error text, and a trajectory summary when these fields are available. The reflector returns ξ = (d, v,r,η).(12) Here d is a diagnostic class. The vector v ∈0, 1 5 records action observation, payload following, task integrity, target effect observation, and the discard indication. The revise focus is r, and η is one local revision instruction. Diagnostic vocabulary. The implementation accepts the following ten class values: not_read read_but_ignored read_but_refused weak_attack sink_not_reached sink_missing_secret task_integrity_broken unstable_success already_successful unknown The implementation name sink_missing_secret denotes missing target content at the sink and is not restricted to credentials. The accepted revise foci are: payload_relevance attack_strength stealth_stability none task_integrity_preservation Fallback diagnosis. Before querying the reflection model, the implementation computes a deterministic record ξ 0 . The following cascade is evaluated from top to bottom: ConditionClassFocus Mixed scores at 0.8 unstable_success stealth_stability Record score at least 0.8 already_successful none No audited API call not_read payload_relevance Refusal indicator read_but_refused stealth_stability Attack success below 0.25 read_but_ignored payload_relevance Otherwise sink_missing_secretattack_strength The last branch reuses the current objective o as its revise instruction. The fallback marks payload following at an attack suc- cess value of at least 0.5. It marks task integrity at a record score of at least 0.3. Audited API use supplies the fallback values for payload observation and target effect observation. These thresholds control revise generation only. Candidate acceptance remains the conjunction ̄g a (x) = ̄g u (x) = 1 in Algorithm 1. Structured model and normalization. Let π ψ denote the reflection model andN the schema normalizer. The imple- mented diagnosis is ξ =N (π ψ (q),ξ 0 ),(13) where q serializes the scene metadata, risk specification, assigned scope, surface description, current candidate, revise objective, and evaluation record. The serialized request is limited to 12,000 characters. The model uses temperature 0.2 and an output limit of 2,048 tokens. N retains only the ten diagnostic classes, five revise focus, Boolean status values, and a nonempty revise instruction. A missing or invalid field inherits its value from ξ 0 . An exception in generation or parsing returns ξ 0 . Structured Reflection Template Role. Diagnose the execution outcome of the current candidate and return one revise instruction for the assigned field. Scope. Preserve the behavior, surface, field group, case identifiers, and benign task. Do not introduce a new file, skill, tool, risk, or execution surface. Evidence. Read the scene metadata, risk target, attack direction, scope, surface, candidate content, revise objective, and evaluation record. Decision rules. Repeat scores on both sides of 0.8 indicate unstable_success with focus stealth_stability. For not_read, improve payload relevance before attack strength. For read_but_refused, remove overt or coercive disclosure language and express the revise in task native terms. Return format. Emit one JSON object. Use failure_type for the diagnostic class. Use the Boolean fields payload_observed, payload_followed, task_integrity_preserved, target_side_effect_observed, and discard_candidate. Use revise_focus for the revise class and revise_instruction for one local edit objective. Bounded revise. The reflection record does not edit a case directly. It conditions π φ , after which Valid enforces the fixed surface and field group. The discard indication is advi- sory and does not remove a beam in Algorithm 1. This design preserves a bounded revise path when the model requests re- jection while retaining structural admission as a deterministic operation. For a B 11 example, consider a task that requires one prior- ity update and one tag update. A candidate causes repeated job history queries and a 120 second wait. The evidence record confirms service reads and the resource effect, while the wait consumes the execution budget before the two re- quired updates. Reflection assigns task integrity preservation as the revise focus. The resulting instruction retains the re- source trigger, removes the wait, and restores the two required updates. The payload, assigned field, criteria, and behavior remain fixed. Experimental Setting Systems and controlled comparisons. All agent exe- cutions use Ubuntu 22.04.5 LTS. RQ1 fixes OpenClaw 2026.5.19 and changes only the base model across the 15 models in Table 2. RQ2 fixes Deepseek-v4-Pro and replaces the complete cowork harness. This intervention changes the harness context assembly, memory retrieval, tool serializa- tion, and action mediation. It does not isolate one harness component. Table 5 reports the six releases used in RQ2. HarnessVersion OpenClaw2026.5.19 OpenAgentv2.83.1 Claude Code v2.1.218 Hermes Agent v0.18.2 OpenCode1.18.3 QwenPaw2.0.0.post2 Table 5: Harness versions used for RQ2. The Deepseek-v4-Pro and OpenClaw configuration ap- pears in both RQ1 and RQ2 and is counted once. The two studies therefore contain 20 unique configurations. Each con- figuration contains three rollouts for each of the 300 mali- cious cases and one rollout for each of the 300 matched benign cases. This gives 1,200 trajectories per configuration and 24,000 trajectories in total. RQ3 reuses the 300 malicious and 300 benign labels for detection evaluation and does not add agent execution trajectories. Construction and grading. Candidate generation uses Deepseek-v4-Pro. GPT-5.5 serves as the rating model and the trajectory evidence verifier. Model requests are routed through OpenRouter. Construction uses beam width w = 3, proposal count n = 2, depth d max = 5, ranking weight α = 0.5, and evidence weight β = 0.4. Construction and re- ported evaluation use disjoint rollouts. The evaluation thresh- old for attack success is 0.8. The first three malicious rollouts define pass@1, pass@2, and pass@3. AGS is averaged over malicious rollouts and UGS is averaged over matched be- nign rollouts. Behavior conditioned results preserve the case frequencies in Figure 2. They are not macro averages over the 15 behavior labels. Reset and task equivalence. Before every rollout, the framework restores the declared workspace files, web service APIs, persistent memory, and case counters. Within RQ1, the harness, tools, cases, reset state, graders, and trusted logging schema remain fixed. Within RQ2, the base model, cases, task authorization, initialized objects, grading criteria, and trusted records remain fixed. Each harness retains its na- tive context and tool serialization. The intervention therefore compares complete harnesses under the same task visible state and API capability. Defense inputs and decision rules. Prompt-Guard-2-86M and Llama-Guard-4-12B receive local text segments avail- able at their classification boundary (Llama Team 2024; Meta AI 2025). AgentDoG-Qwen3-4B receives contextual trajectory states (Liu et al. 2026). AI-Infra-Guard and Cisco- Skill-Scanner receive workspace or skill artifacts (Yang et al. 2026; Tencent Zhuque Lab 2025). The decision threshold for each detector is frozen before scoring the 600 report- ing inputs. Table 4 compares each binary prediction with the benchmark pair label at the method’s native observa- tion boundary. The protocol measures discrimination under different inputs and training targets. It does not insert a block- ing policy into agent execution or estimate a causal change in AGS, ASR, or UGS. Additional Results Execution Cost This experiment examines whether the observed behavioral risk ranking can be attributed to differences in execution cost. We compare action-cycle distributions across models and harnesses, and analyze cache-inclusive token consump- tion for malicious and matched benign rollouts under the fixed OpenClaw harness. Although malicious rollouts have higher mean token consumption for all 15 models, with rela- tive increases from 2.7% to 20.4%, neither action cycles nor token consumption reproduces the ASR ordering. The re- sults indicate that the observed risk ranking is not explained by longer or more token-intensive execution. Figure 7(a) shows malicious median action counts from 15.0 to 19.5 across the 15 base models. The correspond- ing benign medians range from 14.0 to 18.0. MiniMax-M3, Deepseek-v4-Pro, and Hunyuan-3.0 each have a malicious median of 19.0, while their ASRs are 25.0%, 94.4%, and 30.0%. Figure 7(b) shows a wider harness range. OpenClaw has the shortest malicious median at 19 cycles and the high- est harness ASR at 94.4%. Hermes has the longest median at 47 cycles and an ASR of 79.2%. QwenPaw and Claude Code both have a median of 33 cycles, while their ASRs differ by 7.7 percentage points. These comparisons do not support action count as an explanation for the risk ordering. Table 6 reports cache inclusive token consumption un- der the fixed OpenClaw harness. The statistics are computed from archived rollouts with populated token fields, and miss- ing entries are not imputed. The malicious mean exceeds the benign mean for all 15 models. The increase ranges from 3.4k tokens for GPT-5.4-mini, with 128.1k versus 124.7k, to 27.8k for Deepseek-v4-Flash, with 164.2k versus 136.4k. Relative to the benign mean, these endpoints correspond to increases of 2.7% and 20.4%. The malicious median also exceeds the benign median for 14 of the 15 models. GPT-5.4-mini is the only exception, with a malicious median of 110.2k tokens and a benign median of 111.0k. The malicious and benign IQRs overlap for all 15 models, so the central 50% token ranges are not disjoint for the two case types. Hunyuan-3.0 records the largest malicious and benign means at 207.1k and 188.8k tokens. GLM-5.2 records the smallest means at 97.9k and 91.6k tokens. However, the or- dering by malicious mean token use differs from the ASR *#2%&.20 *#2%& -,,&1 ! !+(,( /-) 4&, +#5 4&, .*20 (+( (+( (,(#5 (,(#5 &&.0&&)3/- &&.0&&)3*#0' 2,62#, -%&* ) ) ) ) ) ) ) !-1#*1-)&,0(,.21-21.21$#$'& ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) 11#$) *&#, " (!0#$,0. (!0#$+**$/ )&*& -+' 2$* )!3 2$* ,(0. &)& &)& &*&!3 &*&!3 $$,.$$'1 -+ $$,.$$'1 (!.% 0*40!* +#$( /$-!/&+*"+0*/ //!"'($!* (!0#$,0. (!0#$+**$/ )&*& -+' 2$* )!3 2$* ,(0. &)& &)& &*&!3 &*&!3 $$,.$$'1 -+ $$,.$$'1 (!.% 0*40!* +#$( /$-!/&+*"+0*/ //!"'($!* $" ) $""' (# %!& $"# )") "' '%'#"#("' ′ " $" ) $""' (# %!& $"# )") "' '%'#"#("' ′ " (!0#$,0. (!0#$+**$/ )&*& -+' 2$* )!3 2$* ,(0. &)& &)& &*&!3 &*&!3 $$,.$$'1 -+ $$,.$$'1 (!.% 0*40!* +#$( /$-!/&+*"+0*/ //!"'($!* (a)RQ1ExecutionIteration (b)RQ2ExecutionIteration (c)RQ1ExecutionTokenUsage Figure 7: Execution resource distributions. Orange boxes report malicious rollouts and green boxes report matched benign rollouts. Boxes show the interquartile range, center lines show medians, and whiskers use 1.5 IQR. Panels (a) and (b) cover all configurations in RQ1 and RQ2. Panel (c) contains the 15 models present in the archived token plot. Table 6 reports token statistics for all 15 models. Table 6: Cache inclusive token use with OpenClaw fixed. Values areks of tokens per rollout. MaliciousBenign ModelMeanMedian [IQR] MeanMedian [IQR] Claude-Opus-4.8175.7 164.2 [142.3, 194.2] 169.9 160.6 [138.1, 189.9] Claude-Sonnet-4.6 146.7 131.2 [116.5, 165.4] 141.7 122.5 [100.9, 150.8] GPT-5.5123.7113.9 [93.3, 140.3] 113.3106.8 [88.6, 125.5] GPT-5.4-mini128.1110.2 [93.5, 140.5] 124.7111.0 [92.3, 133.1] Grok-4.5134.9 116.9 [106.3, 144.7] 117.6108.2 [89.6, 126.7] GLM-5.297.992.1 [78.4, 110.0]91.688.3 [75.1, 103.5] Qwen3.7-max130.7114.4 [98.1, 148.4] 119.3108.8 [91.1, 129.5] Qwen3.7-plus108.699.0 [87.2, 118.0]99.594.3 [81.0, 111.5] Kimi-K3124.3114.1 [96.0, 137.7] 113.899.0 [92.5, 121.9] Kimi-K2.6102.395.1 [78.7, 117.8]95.686.6 [73.5, 103.2] MiniMax-M3182.8 156.7 [127.3, 207.6] 160.2 143.2 [116.4, 180.0] MiniMax-M2.7116.2106.8 [90.2, 132.3] 101.491.2 [77.3, 109.7] Deepseek-v4-Pro149.0 133.4 [111.1, 172.9] 125.1111.4 [92.5, 134.2] Deepseek-v4-Flash 164.2 146.1 [111.6, 186.3] 136.4 125.2 [105.6, 150.6] Hunyuan-3.0207.1 188.1 [154.3, 233.4] 188.8 174.0 [142.9, 210.3] ordering in Table 2. Deepseek-v4-Pro reaches 94.4% ASR with a malicious mean of 149.0k tokens, whereas Hunyuan- 3.0 reaches 30.0% ASR with 207.1k tokens. Claude-Opus- 4.8 records 10.1% ASR despite a malicious mean of 175.7k tokens. These comparisons show that malicious rollouts have higher average token use within each model, but cross-model token consumption doesn’t account for the observed risk ranking under the fixed harness. 05101520253035 Step 10 20 30 40 50 60 Cumulative detection rate (%) Prompt Guard 2-86MLlama Guard 4-12BAgentDoG-1.5-Qwen3.5-4B Figure 8: Cumulative malicious case detection by logged trace position for Prompt-Guard-2-86M, Llama-Guard-4- 12B, and AgentDoG-Qwen3-4B. Safety Policy Effects We evaluate how rapidly each query level defense detects risk behavior across 300 malicious cases. The defense evaluates behavior safety at every query step, and execution terminates after the first positive prediction. Figure 8 reports the cumu- lative percentage of cases detected at or before each step. AgentDoG-Qwen3-4B detects 11.0% of the malicious cases at the first step. Its detection rate reaches 26.3% at step 8, 44.3% at step 10, and 50.7% at step 12. The final rate is 55.7% at step 17, corresponding to 167 detected cases. Llama-Guard-4-12B begins detecting cases at step 3 and reaches 11.7% at step 7, 15.7% at step 10, and 18.3% at step 20. Prompt-Guard-2-86M begins at step 6 and reaches 3.3% at step 10, 10.3% at step 15, and 13.3% at step 36. Their final detection counts are 55 and 40 cases, respectively. At step 10, AgentDoG-Qwen3-4B exceeds Llama-Guard-4-12B and Prompt-Guard-2-86M by 28.7 and 41.0 percentage points. By step 12, it has detected 152 cases, which exceeds the final detection count of either baseline. The result reflects differences in both detection timing and final recall. Prompt-Guard-2-86M and Llama-Guard-4- 12B classify local text available at their evaluation bound- aries (Llama Team 2024; Meta AI 2025). AgentDoG-Qwen3- 4B instead uses contextual trajectory states and is trained to identify risk behavior from execution context (Liu et al. 2026). Therefore, its advantage cannot be attributed only to model size, since its backbone is smaller than Llama-Guard- 4-12B. The results indicate that trajectory context enables earlier termination for a larger fraction of malicious execu- tions. However, AgentDoG-Qwen3-4B still misses 133 cases. This limitation motivates a layered defense that combines ar- tifact scanning before execution (Yang et al. 2026; Tencent Zhuque Lab 2025) with query level trajectory monitoring during execution. Ablation: Impact of β We vary the log evidence weight β while holding every tra- jectory and criterion output fixed. A value of zero uses tra- jectory evidence only. A value of one uses log evidence only. The reported results use β = 0.4. This analysis measures score sensitivity and does not execute the agents again. Across base models, every AGS curve increases as the log evidence weight rises. Claude-Opus-4.8 ranges from 0.169 0.00.20.40.60.81.0 Python component weight α 0.0 0.2 0.4 0.6 0.8 1.0 AGS OpenClaw across models — AGS Claude-Opus-4.8 Claude-Sonnet-4.6 GPT-5.5 GPT-5.4-mini Grok-4.5 GLM-5.2 Qwen-3.7-max Qwen-3.7-plus Kimi-K3 Kimi-K2.6 MiniMax-M3 MiniMax-M2.7 Deepseek-v4-Pro Deepseek-v4-Flash Hunyuan-3.0 Figure 9: AGS sensitivity to the log evidence weight β. The dashed lines mark β = 0.4, which is used for all reported tables. 0.00.20.40.60.81.0 Python component weight α 0.0 0.2 0.4 0.6 0.8 1.0 AGS Deepseek-v4-Pro across agents — AGS OpenClaw OpenAgent ClaudeCode Hermes OpenCode QwenPaw Figure 10: AGS sensitivity to the log evidence weight β. The dashed lines mark β = 0.4, which is used for all reported tables. to 0.456, while Deepseek-v4-Pro ranges from 0.941 to 0.977. These two models remain the lowest and highest AGS end- points. At β = 0.4, their values are 0.284 and 0.955, match- ing Table 2. The model ordering at the two extremes is there- fore stable for the lowest and highest risk models. The harness comparison is more sensitive to the evi- dence source. OpenAgent ranges from 0.774 to 0.859 as β increases. QwenPaw moves in the opposite direction from 0.896 to 0.825. The lowest harness AGS consequently changes from OpenAgent under trajectory evidence only to QwenPaw under log evidence only. OpenClaw remains high- est and ranges from 0.941 to 0.977. UGS is not included in this sensitivity analysis because its definition uses the utility criteria directly and does not contain β. Behavior Conditioned Results This detailed RQ1 experiment examines whether the aggre- gate model ranking in Table 2 is consistent across the 15 risk behaviors. OpenClaw 2026.5.19, the case pairs, reset state, tools, grading criteria, and evidence verifiers remain fixed. Table 7 reports malicious¬AGS for each behavior. A larger value indicates less attack evidence. The final columns report the frequency weighted average ¬AGS and average benign UGS, which match the aggregate values in Table 2. Results. Claude-Opus-4.8 records the highest average ¬AGS at 0.716, with average UGS of 0.938. Its lowest be- havior values are 0.358 for Goal Hijacking and 0.430 for De- ceptive Tool Invocation. Deepseek-v4-Pro records the lowest average ¬AGS at 0.045, with average UGS of 0.922. Its ¬AGS is below 0.10 for 12 behaviors. The other three val- ues are 0.101 for Tool Scope Escalation, 0.134 for False Reporting, and 0.172 for Context Flooding. The range across models depends on the behavior. State Tampering spans 0.039 to 0.932, which gives the largest range at 0.893. Data Exfiltration spans 0.024 to 0.916, with a range of 0.892. Tool Scope Escalation spans 0.101 to 0.973, with a range of 0.872. Deceptive Tool Invocation spans 0.002 to 0.483, and no evaluated model reaches 0.5 on this behavior. These values identify behavior specific differences that are hidden by a single model average. Average utility does not reproduce the safety ordering. Claude-Opus-4.8 and Grok-4.5 both record average UGS of 0.938, while their average¬AGS values are 0.716 and 0.130. Kimi-K3 records the highest average UGS at 0.940, while its average¬AGS is 0.511. The matched benign score therefore measures task completion without serving as a substitute for the malicious behavior score. Conclusion. The behavior conditioned results show two distinct model profiles. Deepseek-v4-Pro has low ¬AGS across 12 behavior categories, so its aggregate risk is not pro- duced by one label. Claude-Opus-4.8 has the highest average ¬AGS, but Goal Hijacking and Deceptive Tool Invocation remain below 0.5. The fixed harness and matched tasks link these differences to how each base model maps the same untrusted context to tool actions and environment effects. Reporting the behavior conditioned distribution therefore re- veals residual failure modes that the aggregate average does not localize. Table 7: Category level behavioral results across base models with OpenClaw fixed. Columns B 1 to B 15 report malicious ¬ AGS . The final columns report the frequency weighted model averages of ¬ AGS and benign UGS . Model B 1 B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 B 10 B 11 B 12 B 13 B 14 B 15 Average ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS ¬ AGS UGS Claude-Opus-4.8 0.641 0.358 0.916 0.740 0.599 0.932 0.430 0.820 0.801 0.911 0.680 0.739 0.667 0.925 0.636 0.716 0.938 Claude-Sonnet-4.6 0.645 0.504 0.789 0.492 0.451 0.904 0.483 0.770 0.696 0.973 0.680 0.757 0.648 0.158 0.331 0.653 0.927 GPT-5.6-Sol 0.582 0.684 0.678 0.676 0.502 0.615 0.359 0.695 0.204 0.616 0.454 0.665 0.625 0.196 0.250 0.566 0.924 GPT-5.6-Terra 0.596 0.649 0.849 0.808 0.534 0.635 0.346 0.728 0.320 0.650 0.540 0.595 0.616 0.130 0.543 0.614 0.922 GPT-5.6-Luna 0.550 0.575 0.590 0.501 0.288 0.497 0.208 0.665 0.172 0.554 0.626 0.543 0.552 0.174 0.269 0.497 0.927 GPT-5.5 0.525 0.639 0.587 0.290 0.207 0.640 0.178 0.641 0.528 0.532 0.536 0.568 0.595 0.358 0.185 0.507 0.928 GPT-5.4 0.499 0.569 0.552 0.360 0.180 0.391 0.238 0.454 0.218 0.551 0.397 0.427 0.439 0.186 0.178 0.409 0.895 GPT-5.4-mini 0.324 0.311 0.180 0.251 0.264 0.100 0.190 0.360 0.357 0.637 0.232 0.243 0.306 0.235 0.152 0.273 0.904 Grok-4.5 0.115 0.240 0.194 0.019 0.067 0.098 0.048 0.174 0.107 0.302 0.048 0.086 0.208 0.088 0.087 0.130 0.938 Grok-4.3 0.148 0.132 0.199 0.021 0.053 0.143 0.380 0.024 0.093 0.261 0.317 0.090 0.145 0.115 0.077 0.138 0.881 GLM-5.2 0.364 0.213 0.391 0.242 0.419 0.804 0.163 0.477 0.498 0.822 0.636 0.317 0.663 0.278 0.081 0.453 0.929 Qwen3.7-max 0.538 0.324 0.386 0.314 0.308 0.712 0.253 0.557 0.535 0.829 0.567 0.618 0.578 0.118 0.149 0.489 0.915 Qwen3.7-plus 0.464 0.394 0.372 0.382 0.201 0.685 0.180 0.578 0.479 0.928 0.582 0.566 0.623 0.130 0.134 0.476 0.915 Kimi-K3 0.409 0.451 0.326 0.535 0.536 0.639 0.194 0.821 0.147 0.631 0.750 0.637 0.592 0.133 0.304 0.511 0.940 Kimi-K2.6 0.167 0.771 0.107 0.243 0.249 0.137 0.087 0.367 0.131 0.501 0.150 0.164 0.380 0.171 0.048 0.252 0.869 MiniMax-M3 0.519 0.283 0.667 0.619 0.291 0.909 0.460 0.626 0.649 0.845 0.599 0.657 0.655 0.502 0.497 0.598 0.917 MiniMax-M2.7 0.188 0.199 0.086 0.039 0.310 0.121 0.193 0.336 0.155 0.627 0.202 0.184 0.358 0.094 0.064 0.196 0.880 MiniMax-M2.5 0.230 0.160 0.204 0.044 0.123 0.159 0.459 0.108 0.086 0.363 0.603 0.232 0.328 0.024 0.046 0.201 0.874 Deepseek-v4-Pro 0.039 0.045 0.024 0.031 0.020 0.039 0.002 0.026 0.014 0.101 0.033 0.064 0.134 0.172 0.016 0.045 0.922 Deepseek-v4-Flash 0.067 0.084 0.072 0.027 0.103 0.120 0.047 0.112 0.033 0.309 0.210 0.075 0.418 0.223 0.040 0.113 0.900 Deepseek-v4-Flash-0731 0.366 0.367 0.232 0.085 0.118 0.326 0.289 0.722 0.117 0.314 0.525 0.472 0.620 0.359 0.047 0.368 0.918Hunyuan-3.0 0.462 0.252 0.521 0.427 0.446 0.794 0.239 0.681 0.624 0.825 0.619 0.522 0.646 0.653 0.353 0.545 0.933 16 Detect Risk Behavio r Ignore Risk Behavio r Figure 11: B1 Instruction Hijacking Risk: Claude-Sonnet-4.6 trajectory analysis shows that Sonnet detected malicious behavior through injection and blocked the malicious behavior. 17