Paper deep dive
Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis
Da Song, Yuheng Huang, Boqi Chen, Tianshuo Cong, Randy Goebel, Lei Ma, Foutse Khomh
Models: Claude, Gemini, GPT-3.5, GPT-4, Llama, Mistral
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/11/2026, 12:57:40 AM
Summary
The paper introduces LogiSafetyGen, a framework for evaluating implicit regulatory compliance in LLM-based agents. It converts unstructured regulations into Linear Temporal Logic (LTL) oracles and uses logic-guided fuzzing to synthesize safety-critical, executable test traces. The resulting benchmark, LogiSafetyBench, contains 240 human-verified tasks across financial, healthcare, and IoT domains, revealing that state-of-the-art LLMs often prioritize functional task completion over mandatory safety constraints.
Entities (4)
Relation Signals (3)
LogiSafetyGen → generates → LogiSafetyBench
confidence 100% · Building on this framework, we construct LOGISAFETYBENCH
LogiSafetyGen → uses → Linear Temporal Logic
confidence 100% · converts unstructured regulations into Linear Temporal Logic oracles
LogiSafetyBench → evaluates → LLM
confidence 95% · Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models... frequently prioritize task completion over safety
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior.
Tags
Links
- Source: https://arxiv.org/abs/2601.08196
- Canonical: https://arxiv.org/abs/2601.08196
Trouble viewing inline? Open PDF directly →
Full Text
48,764 characters extracted from source content.
Expand or collapse full text
Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis Da Song 1,2 , Yuheng Huang 3 * , Boqi Chen 4 , Tianshuo Cong 1,2 , Randy Goebel 5 , Lei Ma 3,5 , Foutse Khomh 6 Abstract The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously en- force mandatory safety constraints. To fill this gap, we introduce LOGISAFETYGEN, a frame- work that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety- critical traces. Building on this framework, we construct LOGISAFETYBENCH, a benchmark comprising 240 human-verified tasks that re- quire LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of- the-art (SOTA) LLMs reveal that larger mod- els, despite achieving better functional correct- ness, frequently prioritize task completion over safety, which results in non-compliant behav- ior. 1 Introduction Recent advances in large language models (LLMs) have enabled the emergence of LLM-based agents that can interpret complex user instructions, invoke * Corresponding Author 1 School of Cryptologic Science and Engineering, Shandong University, Jinan, Shandong, China 2 Shandong Key Laboratory of Artificial Intelligence Security, Shandong University, Jinan, Shandong, China 3 The University of Tokyo, Tokyo, Japan 4 McGill University, Montreal, Canada 5 University of Alberta, Edmonton, AB, Canada 6 Polytechnique Montreal, Montreal, Canada Emails:song_da,tianshuo.cong@sdu.edu.cn, yuhenghuang42@g.ecc.u-tokyo.ac.jp, boqi.chen@mail.mcgill.ca,rgoebel@ualberta.ca, ma.lei@acm.org, foutse.khomh@polymtl.ca external tools, and interact with the physical or dig- ital world to achieve multi-step goals (Qin et al., 2024). While this capability extends far beyond that of traditional chatbots, it also introduces sub- stantially higher safety and regulatory risks. In high-stakes domains such as financial services, le- gal reasoning, healthcare, and smart home control, functional task completion alone is insufficient; LLMs must consistently satisfy safety and regu- latory constraints throughout the entire decision- making and execution process. Violations may result in severe consequences (Ye et al., 2024; Ra- dosevich and Halloran, 2025), including physical harm and regulatory non-compliance. To mitigate such risks, systematic evaluation and pre-deployment testing are essential for under- standing an LLM’s capability boundaries and for building trust among developers, regulators, and end-users. In particular, evaluation must jointly as- sess whether an LLM can (1) achieve the intended functional goal and also (2) adhere to mandatory safety constraints under realistic interaction sce- narios. However, existing evaluation practices fall short of this requirement. On the one hand, current benchmarks largely rely on static test sets derived from manual cura- tion or web scraping. These datasets are expen- sive to scale (Zhang et al., 2024a), difficult to vali- date (Huang et al., 2025), and prone to data satura- tion (Chen et al., 2025a). More critically, they pri- marily evaluate functional correctness, while treat- ing safety or regulatory compliance as secondary or implicit. On the other hand, real-world safety constraints are often specified in lengthy natural language policy documents, such as regulatory acts (e.g., EU AI Act (EU, 2024)) or internal compli- ance manuals, making it prohibitively costly to involve domain experts in mapping relevant rules to possible user instructions or execution traces. To address these limitations, we propose LO- GISAFETYGEN, an automated framework for syn- 1 arXiv:2601.08196v1 [cs.CL] 13 Jan 2026 thesizing high-quality test cases for safety-critical LLMs. LOGISAFETYGEN takes tool specifications and regulation documents as input and automati- cally generates test scenarios with explicit, verifi- able safety constraints. The framework consists of three stages. (1) Automated Oracle Construction, which translates unstructured policy documents into deterministic Linear Temporal Logic (LTL) over finite traces, establishing a formal “gold stan- dard” for compliance. (2) Logic-Guided Trace Gen- eration, which leverages a fuzzing engine to syn- thesize executable ground-truth traces that are com- pliant with both tool semantics and temporal safety constraints by construction. (3) Safety Masking, which converts these traces into natural language instructions that omit mandatory safety steps, forc- ing LLMs to infer implicit constraints from context that closely reflect real-world user interactions. We instantiate LOGISAFETYGEN in three repre- sentative high-stakes domains—Financial Services, Tele-Healthcare, and Smart Home IoT—and con- struct LOGISAFETYBENCH, a benchmark of 240 manually verified test cases. Each requires LLMs to interleave task-oriented reasoning with manda- tory safety operations under temporal constraints. Our contributions are summarized as follows: • We introduce LOGISAFETYGEN, an automated benchmark generation framework that integrates LTL-based formal verification with logic-guided fuzzing to produce safety-critical test cases. •We propose a Dual-Oracle Evaluation protocol that jointly verifies functional correctness and strict temporal safety compliance, thus enabling precise measurement of the Inference Gap be- tween task success and rule adherence. •We release LOGISAFETYBENCH, a challenging, human-verified benchmark spanning three high- stakes domains, which require LLMs to satisfy implicit regulatory constraints during execution. •We conduct a systematic evaluation of SOTA LLMs, showing that although scaling improves general reasoning ability, it does not resolve fail- ures in implicit regulatory compliance. 2 Related Work Trustworthiness Evaluation of LLM. A grow- ing body of work aims to provide benchmarks on the safety, security, and regulatory compliance of LLM-based systems, which is crucial for their trustworthy deployment in real-world (Moham- madi et al., 2025). Early work, such as Agen- tHarm (Andriushchenko et al., 2025; Mazeika et al., 2024), analyzes LLM behavior under malicious user requests. While important, these scenarios differ from evaluating rule adherence under be- nign conditions and mainly focus on question- answering tasks. More recent frameworks, in- cluding ToolEmu (Ruan et al., 2024), Agent- SafetyBench (Zhang et al., 2024b), and Agent- Bench (Liu et al., 2024), examine general tool-use safety beyond simple QA. However, in these set- tings, safety largely means correct tool manipula- tion rather than interpreting regulations in text and following specific rules. In a related but distinct direction, researchers be- gin to anchor evaluation in explicit regulatory con- texts. EU-Agent-Bench (Lichkovski et al., 2025) and HSE-Bench (Wang et al., 2025a) introduce evaluation scenarios derived from concrete policies and standards, while RuleArena (Zhou et al., 2025) and CNFinBench (Ding et al., 2025) move closer to structured rule reasoning by embedding real- world regulations into their tasks. These bench- marks test agents’ ability to interpret and reason about regulatory constraints expressed in natural language, but they do not require LLMs to generate executable tool-call scripts that satisfy formal com- pliance rules, which can be essential in real-world LLM-based agent deployment. Benchmark Construction Methods. One ma- jor approach to benchmark construction is to col- lect high-quality data with human effort. Hu- manEval (Chen et al., 2021) is one of the classic ex- amples in this category. Some benchmarks also ex- pand their coverage by using web crawlers and then applying manual annotation. In tool-use evaluation, BFCL (Patil et al., 2025) is a representative exam- ple that relies on crawling large-scale repositories. However, since manual benchmarks are both costly and potentially contaminated (Zhu et al., 2024), researchers have explored LLM-in-the-loop bench- mark generation as a more scalable alternative. In these approaches, models help synthesize test instances automatically (Ghazaryan et al., 2025). Notable examples include AutoCodeBench (Chou et al., 2025), which generates code-centric tasks, and MCPBench (Wang et al., 2025c), which pro- duces tool-use test cases with LLM assistance. Al- though these methods reduce annotation effort and broaden evaluation scope, the stochastic nature of LLM generation makes it hard to guarantee strict 2 Masking Filter ... Automated Safety Oracle Construction Policy Extraction & LTL Generation Human Verification Signature Validator Scope Alignment Constraint Deduplication 푅 1 퐷 1 푅 2 푅 푛 퐷 푛 ALWAYS (푷 ퟏ IMPLIES EVENTUALLY(푷 ퟐ )) NOT ((NOT 푷 ퟏ ) UNTIL 푷 ퟐ ) 푅 1 푅 2 푅 3 푅 푛 ... Extracted Policy (푅 푟푎푤 )API Env (퐷 푎푝푖 ) ... ɸ 1 Verified LTL Oracles (ɸ): ɸ 2 ... ɸ 푛 Logic-Guided Trace Generation 푠 0 푠 1 푠 1 푠 0 x API 1 API 1 API 2 푠 0 x API 2 Pass Business Logic Fail Business Logic Precondition Satisfaction 푠 2 푠 2 API 3 API 4 ... API Trace 푠 1 x API 3 푠 1 x API 4 Pass LTL Fail LTL Safety Satisfaction Safety Step (API 4) Business Step (API 1) ... Safety Step (Goal) Ground Truth Trace Safety Masking and Instruction Synthesis Generate Critique/Refine Verified LTL Oracles Input Trace Generator Agent Evaluator Agent Ensure No Safety Related Steps mentioned in Instruction ... Achieve goal X ... ... 1. Perform Action A 2. Perform Action B ... Goal-Oriented Instructions(퐼 푔표푎푙 ） Workflow-Oriented Instructions(퐼 푓푙표푤 ） 1 23 Logical-Guided Fuzzer Dual-Template Mapping Action (푃 푛 ) Figure 1: The LOGISAFETYGEN framework. 1 ⃝ We translate regulations into grounded LTL oracles. 2 ⃝ Logic- Guided Fuzzer synthesizes compliant traces (τ ∗ ). 3 ⃝A safety masking pipeline generates natural language instructions (I ) that force agents to infer implicit constraints. correctness or compliance with specific rules. To support tasks that demand rigorous adherence to complex rules, recent work has proposed sup- plementing deterministic, rule-driven mechanisms with a human/LLM-in-the-loop process. Closest to our setting is ShieldAgent (Chen et al., 2025b), which uses probabilistic rule circuits to constrain agent trajectories; however, it is designed as a run- time defense against adversarial attacks, whereas our framework serves as an offline stress test of compliance. Similarly, StateEval (Huang et al., 2025) introduces state-aware test generation for sequential API calls, focusing on functional cor- rectness rather than regulatory compliance. 3 Methodology 3.1 Overview We formalize the problem of safety-aware LLM evaluation as follows: Given a toolsetTand a regulatory policyP, our goal is to automatically synthesize a test case(I,S ∗ , Φ task )whereIrep- resents user instructions,S ∗ specifies the desired target states, andΦ task encodes the applicable reg- ulatory constraints. The statesS ∗ are guaranteed to be reachable by a valid ground-truth API invoca- tion traceτwhich is generated alongside the test case. To achieve fully automated synthesis, we must overcome three inherent technical challenges: (1) The Ambiguity Gap. Regulations inPare unstructured natural language, and typically lack the required mathematical determinism to support automated verification. (2) The Validity Gap. Naively using LLMs to generate traces often results in hallucinated API calls or invalid arguments, rendering the test cases non-executable. (3) The Inference Gap. Simply stating safety rules in the instructionIis trivial; a rigorous test must ensure the LLM can infer implicit constraints from context, simulating real-world compliance. We address these three critical gaps via our method LOGISAFETYGEN. As illustrated in Fig- ure 1, our framework generates test cases by se- quentially addressing three critical gaps in com- pliance testing: First, 1 ⃝Automated Safety Ora- cle Construction (Section 3.2) addresses the Am- biguity Gap by translating unstructured regula- tions into deterministic Linear Temporal Logic (LTL f ) constraints, establishing a verifiable math- ematical gold standard (Φ). Next, 2 ⃝the Logic- Guided Trace Generation (Section 3.3) compo- nent bridges the Validity Gap by employing a constraint-satisfaction fuzzing engine to synthesize ground truth traces (τ ∗ ) that are guaranteed to be executable. Finally, 3 ⃝Safety Masking and Instruc- tion Synthesis (Section 3.4) targets the Inference Gap using a translation-masking strategy to gener- ate user prompts (I) that explicitly describe busi- ness goals while strictly omitting the safety steps found inτ ∗ . This pipeline enables the rigorous testing of an LLM’s ability to autonomously infer mandatory regulatory constraints from the system context. 3.2 Automated Safety Oracle Construction Establishing a gold standard for compliance from massive, unstructured regulatory texts is non-trivial. Unlike robotics approaches restricted to closed vo- cabularies (Pan et al., 2023), API-based environ- ments are prone to ambiguity and hallucination. To bridge this Ambiguity Gap, we design a pipeline 3 that enforces strict grounding. Rule Extraction. We first leverage LLMs to distill regulations into atomic constraints, applying Scope Alignment and Constraint Deduplication to retain only relevant policies. Furthermore, rather than al- lowing unconstrained free-form logical translation with LLMs, we restrict all extracted constraints to two LTL templates that capture the dominant forms of compliance logic in business, as observed in prior work (Chen et al., 2025b). Specifically, we adopt a dual-template formulation comprising: (1) Operational Restrictions which forbid a sensi- tive actionP 2 until a checkP 1 holds, formalized as¬((¬P 1 )UP 2 ); (2) Instruction Adherence re- quires that a triggerP 1 implies a future outcome P 2 , formalized as□(P 1 →⋄P 2 ). Rule Validation.Template-based genera- tion yields a set of candidate formulasΦ cand , which may still reference semantically plausi- ble but non-existent predicates (e.g., hallucinated verify_uservs. the concretecheck_auth). To guarantee executability, we subjectΦ cand to a deter- ministic Signature Validator that eliminates any for- mula containing predicates not already defined in the API schema. This grounding step ensures that all retained Safety OraclesΦare syntactically valid and computationally solvable by the fuzzer, thus completing the automated logic extraction pipeline. 3.3 Logic-Guided Trace Generation With the formal Safety Oracles (Φ) established, our next goal is to synthesize concrete execution traces (τ). These ordered tool invocation sequences serve as the ground-truth solutions for our benchmark, providing the “gold standard” behavior for how an ideal LLM can interleave functional reasoning with mandatory safety checks to satisfy the constraints inΦ. However, generating the ground-truth se- quences creates the Validity Gap. Naively relying on direct LLM prompting is insufficient, as proba- bilistic models prioritize semantic fluency over exe- cutability, and frequently hallucinate inappropriate parameters or violate temporal dependencies. To bridge this gap, we adopt Fuzzing (Zhu et al., 2022), a dynamic testing paradigm that discovers valid execution paths by systematically exploring the state space under strict constraints. Unlike stan- dard text generation, our fuzzer treats trace con- struction as a bounded search problem: it itera- tively proposes candidate actions and rejects any that fail to meet formal specifications. We imple- ment a safety-constrained fuzzer (with≈3600 lines of code) that constructs tracesτ = [a 1 ,a 2 ,...,a k ] incrementally. The core of the fuzzer is a Dual- Constraint Pruning Mechanism. At each stept, given the current environment stateS t , the fuzzer samples a potential actiona t+1 and validates it against two strict boundaries. Precondition Satisfaction (V schema ): The action must be executable in the current stateS t . This ensures that the trace is functionally sound. Safety Satisfaction (V safe ): The updated traceτ t+1 must satisfy the LTL Safety Oracles. We verify this using a runtime LTL monitor:V safe (τ t+1 ) = True ⇐⇒ ∀φ∈ Φ,τ t+1 |= φ. To operationalize these checks efficiently, we employ a Bottom-Up Depth-First Search strategy Traditional top-down planning requires pre-solving constraints for the entire API call graph, but with complex LTL rules, this becomes computationally intractable. In contrast, our bottom-up approach enables Lazy Evaluation. We compute the valid search space and check compliance only for the immediate next step (S t → S t+1 ). If a candidate action violates either the schema or the safety con- straint, the branch is immediately pruned. When the fuzzer attempts a business action that violates a safety constraint (LTL), the LTL mon- itor triggers a violation (V safe = False), pruning that path. The search is then forced to backtrack and explore alternative branches until it selects the mandatory safety operation. By iterating this pro- cess until the target trace length is reached, the fuzzer produces a ground truth traceτ ∗ where ev- ery user action is wrapped in the necessary safety checks, guaranteeing that the test case is solvable, executable, and strictly compliant by construction. 3.4Safety Masking and Instruction Synthesis Phase 2 ⃝yields a ground truth traceτ ∗ that is exe- cutable and fully compliant. However, real-world LLMs are typically driven by natural language re- quests, rather than code sequences. To simulate authentic user interactions, we must translate these executable traces back into realistic user instruc- tions. Critically, a naive translation would explic- itly list every action, including safety checks, and so would reduce the challenge to simple instruction following. To make the evaluation more challeng- ing and closer to real-world use, we convert explicit safety requirements into implicit ones that require the model to perform its own reasoning. Safety Masking (M).We address this by introducing a masking functionMthat log- 4 Table 1: Benchmark Comparison. LOGISAFETYBENCH uniquely combines logic-guided trace generation, implicit safety testing, and formal LTL oracles within an executable sandbox. BenchmarkPrimary FocusAuto. Valid TraceImplicit SafetyFormal OracleExecutable Env. ToolSword (Ye et al., 2024)Adversarial Injection✗✓ TAI3 (Feng et al., 2025)Intent Integrity✓✗✓ ToolEmu (Ruan et al., 2024)General Risks✗✓✗ Agent-SafetyBench (Zhang et al., 2024b)General Safety✗✓✗✓ LOGISAFETYBENCH(Ours)Regulatory Compliance✓ ically filters out regulatory steps.Given the executable traceτ ∗ , we identify the subset of actionsA safe ⊂ τ ∗ corresponding to safety-critical APIs. The masking function preserves only the functional logic, preserving the order of opera- tions:τ bus = Filter(τ ∗ ,¬A safe ). For example, ifτ ∗ is[CreateUser, Verify, GrantAccess], themaskedsequenceτ bus retainsonly [CreateUser, GrantAccess].The resulting instructionIwill thus be “Create a user and grant them access”, which deliberately hides theVerify step. This forces the LLM-Under-Test to infer the omitted compliance requirement solely from the policy context. Instruction Typology. Real-world users interact with LLMs in diverse ways, ranging from vague goals to rigid commands. To assess robustness under different forms of user input, we derive two types of instructions from the masked trace: •Goal-Oriented (I goal ): Describes only the final desired outcome, which tests the planning ca- pability by requiring the LLM to autonomously decompose an abstract intent and insert necessary safety precautions before execution. • Workflow-Oriented (I flow ): Provides a specific step-by-step procedure for the program logic. This tests the compliance resilience by requiring the LLM to resist the tendency to blindly fol- low user orders and proactively interleave safety checks despite the rigid instructions. Implementation. To transformτ bus into high- quality instructions that adhere to these typologies, we implement a Generator-Evaluator Multi-agent pipeline A generator agent synthesizes candidate instructions, which are then critiqued by an eval- uator agent for unambiguity (accurately reflecting business parameters) and safety masking (ensuring no safety hints leak). Benchmark Evaluation Criteria. Given a gener- ated test caseT = (I,S ∗ , Φ task )as the output of the entire pipeline, an LLM is said to be successful in the task if and only if it passes two oracles: • Functional Oracle (S ∗ ): The LLM is function- ally successful if its final state matches the state derived from the ground truth trace: S mut ≡S ∗ . • Safety Oracle (Φ task ): The LLM is compliant if its execution trace satisfies all hidden LTL con- straints: ∀φ∈ Φ task ,τ mut |= φ. These formulations allow us to quantify the Infer- ence Gap by distinguishing between LLMs that simply fail the task and those that achieve “Unsafe Success”, correctly executing the business logic while violating regulatory constraints. 4 Benchmark Construction We present LOGISAFETYBENCH, the first evalua- tion suite designed to rigorously measure implicit regulatory compliance in LLM tool calling. Com- prising 240 verified tasks, it serves as the concrete instantiation of the LOGISAFETYGEN framework. We use GPT-5-Mini as the backbone engine for policy extraction and instruction synthesis, cho- sen for its balance of reasoning capability and cost efficiency. In this section, we detail the scenario selection and verification protocols that establish the benchmark’s quality, contrasting our approach with prior work. As shown in Table 1, existing benchmarks do not simultaneously satisfy the three requirements of regulatory testing: (1) Implicit Safety: Bench- marks such as ToolSword (Ye et al., 2024) em- phasize adversarial attacks rather than failures caused by passive neglect of safety rules. (2) Formal Oracle: Frameworks such as Agent- SafetyBench (Zhang et al., 2024b) depend on LLM- based judges, which introduce non-determinism and evaluation bias. (3) Executable Environ- ments: Many existing benchmarks (Ruan et al., 2024) lack executable ground truth and only rely on static checking. LOGISAFETYBENCH uniquely integrates Logic-Guided Fuzzing to ensure more robust compliance both by trace validity and Safety Masking to enforce implicit compliance, thus pro- viding an executable sandbox where LLMs are graded against formal LTL constraints. 5 4.1 Scenario Selection We instantiate LOGISAFETYGEN across three high- stakes domains where safety failures carry severe consequences. Tool specifications are adapted from ToolEmu (Ruan et al., 2024), and safety oracles are anchored in authoritative international regulations: Financial Services. Based on the European Union Payment Services Directive 2 (EU, 2015), this do- main tests whether an LLM can enforce safety con- straints during online banking services. Tele-Healthcare. Based on the USA HIPAA Secu- rity Rule (HHS, 1996), a widely accepted standard for protecting health information, LLMs must com- ply with safety constraints while handling patients’ sensitive records. Smart Home IoT. Based on the European Telecom- munications Standards Institute EN 303 645 stan- dard (ETSI, 2024), a leading international policy for consumer IoT security, it tests LLMs’ ability to reason about physical safety risks like door locks. 4.2 Human Verification Protocol Although LOGISAFETYGEN is fully automated, constructing a rigorous benchmark requires expert validation for reliable reference behavior. We em- ploy a human verification protocol that first reviews the regulatory logic and then validates the concrete test cases derived from it. Safety Oracle Verification. First, we examined the LTL formulas generated in Section 3.2 to ensure the fuzzer operates on legally sound constraints. To ensure consistency, all authors independently reviewed every candidate across three rounds, re- taining only those that achieved unanimous agree- ment. The pipeline achieved an aggregate accep- tance rate of 73.9% across the three domains. This high consistency confirms that our Automated Or- acle Construction pipeline can effectively distill unstructured policies into legally sound, determin- istic logic, thereby resolving the Ambiguity Gap. Instruction and Masking Verification. Using the verified oracles, the pipeline generated candidate test tuples(I,τ ∗ ). We then labeled these outputs to curate the final benchmark. Each author was responsible for one scenario, labeling candidates until the target quota of 40 valid paired samples was met. Overall, candidates were accepted at a rate of 70.6%, yielding a final benchmark of 240 evaluation tasks (120 paired samples). 5 Evaluation Our evaluation has two parts, targeting different components of the study. First, we evaluate LO- GISAFETYGEN itself, measuring its ability to pro- duce valid and structurally diverse test cases using coverage metrics (Section 5.2). This analysis fo- cuses on how effectively the framework explores complex tool interactions and state-dependent be- haviors. Second, we use the LOGISAFETYBENCH generated by our framework to evaluate SOTA LLMs. This evaluation measures how well dif- ferent models handle safety requirements that are implicit, revealing their ability to reason about regu- latory constraints in realistic settings (Section 5.3). 5.1 Experimental Setup We evaluate thirteen LLMs, including frontier commercial APIs (e.g., GPT-5 series, Gemini-2.5 series), and a diverse range of open-weight architec- tures, including both general-purpose instruction- tuned models (e.g., Llama-3.1-8B, DeepSeek-R1- Distill-Qwen-14B) and specialized code LLMs (e.g., Qwen-Coder). Due to space limitations, we select only eight representative models (Figure 2) to report in the main text. In all experiments, for every test caseT i , the LLM is initialized with a system prompt containing the Tool Schema (D api ), the Raw Regulatory Doc- ument (P txt ), and the User Instruction (I) where explicit safety requirements are removed. In this case, LLMs must plan their actions under the regu- lations without explicit guidance. 5.2 Evaluation of LOGISAFETYGEN We first assess the effectiveness of our Logic- Guided Fuzzer. In particular, we examine whether it can produce execution traces with sufficient di- versity. Diversity is essential for most benchmarks, as it ensures that the test set covers a broad range of behaviors rather than being biased toward some specific scenarios. Moreover, many regulatory vio- lations arise only under rare combinations of tool states that lie in the tail of the behavior distribution. Capturing such edge cases, therefore, requires a diverse test set. Prior work (Yao et al.; Lee et al., 2025; Wang et al., 2025b) commonly uses coverage criteria to quantify diversity in test generation. When selecting baselines for comparison on the diversity of generated traces, we find that the LLM- based approach is the only scalable solution. Al- though alternative approaches such as symbolic 6 020406080100 % Traces GPT-5 GPT-5-Mini GPT-5-Nano Gemini-Pro Gemini-Flash Gemini-Lite DS-R1-Qwen-14B Llama-8B 100 90 18 48 78 45 35 50 18 65 68 28 15 18 38 15 12 15 Financial Goal 020406080100 % Traces 100 95 75 98 100 40 57 12 15 52 42 78 Financial Workflow 020406080100 % Traces 88 78 45 20 62 20 22 28 22 38 95 20 18 15 20 40 28 55 20 Tele-Healthcare Goal 020406080100 % Traces 95 90 78 80 98 25 35 12 12 22 100 12 62 30 Tele-Healthcare Workflow 020406080100 % Traces 28 32 20 12 25 40 15 60 80 40 82 35 38 30 65 18 12 18 40 18 22 Smart Home Goal 020406080100 % Traces 75 78 70 55 12 12 45 80 80 32 28 18 65 68 12 Smart Home Workflow Unsafe SuccessUnsafe FailureBenign FailureSafe Success Figure 2: Pass@1 Rates and Risk Distribution. Comparing Goal-Oriented vs. Workflow-Oriented prompts. execution can, in principle, produce traces, they require substantial manual adaptation and are there- fore beyond the scope of this study. However, naive LLM-based generation also suffers from the Sampling Bias Problem: they often converge on high-likelihood paths (Yang and Holtzman, 2025; Sivaprasad et al., 2025), failing to traverse the complex edge cases with important safety mech- anisms (Huang et al., 2025; Xia et al., 2024). It remains the case that scalability and sufficient di- versity are critical in the long run. To validate this, we implement a comparative baseline using GPT-5- Mini, prompted to generate diverse API programs using the same constraints as our fuzzer. We assess the quality of the generated test cases using two complementary coverage metrics: Safety Critical API Coverage (S.C. Cov): We begin with a coarse-grained coverage measure that captures whether generated traces exercise the APIs directly involved in safety enforcement. Specif- ically, S.C. Cov measures the proportion of the safety-critical API subset covered by the generated traces. To compute that, we first manually identify the set of safety-critical APIs as ground truth, and then measure how many of these APIs are included across the 40 traces generated by each method. Adjacent Transition Coverage (ATC): While S.C. Cov indicates whether safety-critical APIs are in- voked at all, it does not capture how these APIs are composed into action sequences. Safety risks often arise from specific state transitions triggered by sequences of actions, rather than from isolated API calls. To capture this sequential aspect, we adopt ATC (Huang et al., 2025), which measures the diversity of local execution structures. For a trace τ = [a 1 ,...,a N ], this metric is defined as: ATC = |(a i ,a i+1 )| 1≤ i < N| |A| 2 (1) whereAdenotes the set of unique APIs. This metric aims to capture how APIs are connected within execution sequences rather than how often Table 2: Comparison of Coverage Metrics. We report Adjacent Transition Coverage (ATC) and Safety Crit- ical API Coverage (S.C. Cov) across three scenarios. Subset Size indicates the number of ground-truth safety APIs defined for that domain. Best results are bolded. ScenarioSubset Size GPT-5-miniOurs ATCS.C. CovATCS.C. Cov Tele-Healthcare728.4%85.7%42.6%100% Financial Services423.4%100%85.6%100% Smart Home IoT1030.1%80%63.3%100% they appear. A higher ATC indicates better cover- age of state transitions and greater structural diver- sity in the generated tests. Results. Table 2 presents the comparative results. Our Logic-Guided Fuzzer demonstrates superior effectiveness, achieving 100% Safety Critical API Coverage across all domains. In contrast, the LLM-only baseline shows a significantly lower coverage rate (e.g., missing 20% of safety APIs in Smart Home IoT ), indicating that purely proba- bilistic sampling struggles to incorporate important safety-related operations. Moreover, our method consistently achieves higher ATC than the LLM- based baseline across all domains, with the largest improvement observed in Tele-Healthcare setting (23.4% vs. 85.6%). This result aligns with previous findings (Huang et al., 2025) that LLMs suffer from the Sampling Bias Problem in test case generation. By treating trace generation as a constraint satis- faction problem rather than a text generation task, our framework systematically explores the combi- natorial boundaries of the API schema, ensuring that LOGISAFETYBENCH contains the structural complexity required for safety evaluation. 5.3 Results on LOGISAFETYBENCH We now apply LOGISAFETYBENCH to evaluate the compliance capabilities of eight foundation models across three domains and two instruction typolo- gies. We employ Pass@1 (Chen et al., 2021) (Safe Success Rate) as our primary metric. A test case passes if and only if the LLM achieves the func- tional goal and satisfies all hidden LTL constraints. 7 GPT-5 GPT-5-Mini GPT-5-Nano Gemini-Pro Gemini-Flash Gemini-Lite DS-R1-Qwen-14B Llama-8B Syntax Error Semantic Error Instruction Adherence Operational Restriction Failure Category 00000002 0035500126838 01038218282230 0010051052 Financial Services GPT-5 GPT-5-Mini GPT-5-Nano Gemini-Pro Gemini-Flash Gemini-Lite DS-R1-Qwen-14B Llama-8B 00200200 082520510428 121525573260405 002001000 Tele-Healthcare GPT-5 GPT-5-Mini GPT-5-Nano Gemini-Pro Gemini-Flash Gemini-Lite DS-R1-Qwen-14B Llama-8B 02000000 2838126080522225 45287520828300 20202205718 Smart Home IoT 0 20 40 60 80 100 Figure 3: Failure Mode Analysis (Goal). This heatmap details the failure rates across four distinct categories: (1) Syntax Errors, (2) Semantic Errors, (3) Instruction Adherence violations, and (4) Operational Restriction violations. We classify failures into three modes: Benign Fail- ure (task failed safely), Unsafe Failure (safety vio- lation + task failure), and the most critical Unsafe Success (task completion via rule violation). Performance Analysis. Figure 2 indicates a gen- eral positive correlation between LLMs’ capa- bility and safety compliance. Frontier commer- cial models (e.g., GPT-5) consistently outperform open-weight ones, confirming that strong general- purpose reasoning is the bedrock upon which safety behaviors are built. However, the margin of success is heavily modulated by API Density, defined as the number of distinct safety-critical tools provided in the testing scenario. In Financial Services (4 safety APIs), GPT-5 demonstrates near-perfect per- formance. In contrast, the Smart Home IoT domain (10 safety APIs) induces a combinatorial explosion in the underlying state space. This creates a depth of planning that challenges even the most capable models. It is thus confirmed that by manipulating API density, our framework can effectively stratify model performance, providing a rigorous stress test that distinguishes even between top-tier LLMs. Impact of Instruction Typology. Comparing workflow-oriented vs. goal-oriented instructions re- veals how compliance stability varies with user in- teraction styles. Under workflow guidance, models perform robustly because the planning structure is provided. However, removing this scaffolding trig- gers a collapse. In Smart Home IoT, GPT-5 drops from 75% Pass@1 (Workflow) to 28% (Goal). This decline indicates that while models can execute safety checks when the business logic is explic- itly structured, they struggle to autonomously in- terleave these checks when planning from a high- level intent. Furthermore, this drop reflects a shift towards unsafe behavior rather than mere incom- petence. In Tele-Healthcare, Gemini-Pro’s Unsafe Success Rate rises significantly when switching to Goal-Oriented prompts. This confirms that without rigid workflow scaffolding, LLMs tend to prioritize functional success over safety constraints. Diagnosing Failure Reasons. We conduct a fail- ure analysis (Figure 3) to uncover the specific rea- soning deficits behind functional incorrectness and non-compliance. We focus on Goal-Oriented as it forces autonomous planning, directly exposing the reasoning deficits behind non-compliance. Failures are categorized into four modes: Syntax Errors (in- valid code), Semantic Errors (API hallucinations), Instruction Adherence Violations (omitted safety actions), and Operational Restriction Violations (ordering errors). We observe no universal fail- ure pattern across domains or model families, indi- cating that compliance failures are highly context- dependent and difficult to prevent. Syntax vs. Semantic Grounding. We observe a sharp divergence between code validity and util- ity. Syntax Errors are negligible across all models, indicating strong mastery of Python structure, yet Semantic Errors persist unexpectedly even in fron- tier models. For instance, Gemini-Pro exhibits a 50% semantic failure rate in Financial Services, and Gemini-Flash reaches 80% in Smart Home IoT. This shows that high-level reasoning capabilities do not guarantee robust API grounding. Divergent Scaling Trends. Regarding safety violations (Instruction Adherence and Operational Restrictions), we observe contrasting behaviors be- tween model families. The GPT-5 series exhibits a positive scaling trend, where more capable mod- els (e.g., GPT-5) tend to commit fewer safety vi- olations than their smaller counterparts. In con- trast, the Gemini family defies this logic; the ca- pable Gemini-Pro frequently performs worse than Gemini-Flash (e.g., 57% adherence violations in Tele-Healthcare). This inconsistency suggests that for some architectures, scaling capability does not inherently resolve safety non-compliance. 6 Conclusion We propose LOGISAFETYGEN, an automated framework that uses logic-guided fuzzing and 8 safety masking to synthesize verifiable compliance test cases. By instantiating it, we introduce LO- GISAFETYBENCH, a human-verified suite of 240 tasks anchored in real-world regulations. Our re- sults reveal that while SOTA LLMs possess the rea- soning capability to execute workflows, they lack the autonomy to infer implicit safety constraints, frequently prioritizing functional success over reg- ulatory adherence. This work establishes a vital foundation for measuring and improving the regu- latory compliance of autonomous agents. Limitations Representation Limits of Safety Oracles. Our framework establishes a gold standard by ground- ing regulations in deterministic LTL. To help au- tomate LTL synthesis, we restrict these oracles to two dominant templates—Operational Restriction and Instruction Adherence. This abstraction fo- cuses exclusively on the temporal ordering of API calls, limiting our ability to analyze the safety of specific function arguments (e.g., verifying that a transfer amount is within a safe limit or detecting malicious payloads in valid calls). Consequently, policies requiring deep semantic inspection of pa- rameter values or probabilistic judgment cannot be formalized. Future work could explore designing more sophisticated logic representations to extend our safety oracles to capture fine-grained data-flow and parameter-level constraints. Dependence on Generator Capability. Although LOGISAFETYGEN automates the synthesis of valid traces via fuzzing, the initial diversity of the test scenarios is partially bounded by the creative pri- ors of the backbone LLM used for seeding. While our human verification protocol ensures the final benchmark is high-quality, we observed that purely automated generation can still struggle with ex- treme structural novelty without human guidance. As foundation models continue to scale, we antici- pate that future generators will be better equipped to propose diverse edge cases autonomously, fur- ther reducing the reliance on human-in-the-loop curation for high-stakes benchmarks. Ethical Considerations Given that our paper aims to unveil the implicit compliance risks of LLMs, our publicly available dataset includes test cases that simulate regulatory violations across financial, healthcare, and IoT do- mains. We emphasize that these scenarios are en- tirely synthetic and constructed within a mock envi- ronment, specifically intended for safety evaluation and defensive red-teaming purposes to mitigate real-world deployment risks. References Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. 2025. Agentharm: A benchmark for measuring harmful- ness of LLM agents. In The Thirteenth International Conference on Learning Representations. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. 2025a. Benchmarking large language models under data con- tamination: A survey from static to dynamic evalu- ation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10091–10109, Suzhou, China. Association for Computational Linguistics. Zhaorun Chen, Mintong Kang, and Bo Li. 2025b. Shieldagent: Shielding agents via verifiable safety policy reasoning. arXiv preprint arXiv:2503.22738. Jason Chou, Ao Liu, Yuchi Deng, Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai, Yue Mao, Chenchen Zhang, Lingyun Tan, and 1 others. 2025. Autocodebench: Large language models are auto- matic code benchmark generators. arXiv preprint arXiv:2508.09101. Jinru Ding, Chao Ding, Wenrao Pang, Boyi Xiao, Zhiqiang Liu, Pengcheng Chen, Jiayuan Chen, Tiantian Yuan, Junming Guan, Yidong Jiang, Dawei Cheng, and Jie Xu. 2025. Cnfinbench: A benchmark for safety and compliance of large language models in finance. Preprint, arXiv:2512.09506. ETSI. 2024. Cyber Security for Consumer Internet of Things: Baseline Threats and Mitigation Measures. urlwww.etsi.org. EU. 2015. Payment services in the internal market, amending directives 2002/65/ec, 2009/110/ec and 2013/36/eu and regulation (eu) no 1093/2010, and repealing directive 2007/64/ec. EU. 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (the 9 EU AI Act). Official Journal of the European Union, L 2024/1689. Shiwei Feng, Xiangzhe Xu, Xuan Chen, Kaiyuan Zhang, Syed Yusuf Ahmed, Zian Su, Mingwei Zheng, and Xiangyu Zhang. 2025. TAI3: Testing agent integrity in interpreting user intent. In The Thirty-ninth An- nual Conference on Neural Information Processing Systems. Gayane Ghazaryan, Erik Arakelyan, Isabelle Augen- stein, and Pasquale Minervini. 2025. Syndarin: Syn- thesising datasets for automated reasoning in low- resource languages. In Proceedings of the 31st Inter- national Conference on Computational Linguistics, pages 6459–6466. HHS. 1996. HIPAA. Online atwww.cms.hhs.gov. Public Law 104-191, 110 Stat. 1936. Yuheng Huang, Da Song, Zhenlan Ji, Shuai Wang, and Lei Ma. 2025. Evaluating llms on sequential api call through automated test generation. arXiv preprint arXiv:2507.09481. Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jae- woo Kang. 2025. Ethic: Evaluating large language models on long-context tasks with high information coverage. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pages 5497–5512. Ilija Lichkovski, Alexander Müller, Mariam Ibrahim, and Tiwai Mhundwa. 2025. Eu-agent-bench: Mea- suring illegal behavior of llm agents under eu law. In Proceedings of the Workshop on Regulatable Ma- chine Learning at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025). Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Ao- han Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2024. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: a standardized eval- uation framework for automated red teaming and robust refusal. In Proceedings of the 41st Interna- tional Conference on Machine Learning, ICML’24. JMLR.org. Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and benchmarking of llm agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6129–6139. Jiayi Pan, Glen Chou, and Dmitry Berenson. 2023. Data-efficient learning of natural language to linear temporal logic translators for robot task specification. arXiv preprint arXiv:2303.08006. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Char- lie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Con- ference on Learning Representations. Brandon Radosevich and John Halloran. 2025. Mcp safety audit: Llms with the model context proto- col allow major security exploits. arXiv preprint arXiv:2504.03767. Yangjun Ruan, Honghua Dong, Andrew Wang, Sil- viu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the risks of LM agents with an LM- emulated sandbox. In The Twelfth International Con- ference on Learning Representations. Sarath Sivaprasad, Pramod Kaushik, Sahar Abdelnabi, and Mario Fritz. 2025. A theory of response sam- pling in llms: Part descriptive and part prescriptive. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30091–30135. Jianwei Wang, Mengqi Wang, Yinsi Zhou, Zhenchang Xing, Qing Liu, Xiwei Xu, Wenjie Zhang, and Lim- ing Zhu. 2025a. Llm-based hse compliance assess- ment: Benchmark, performance, and advancements. arXiv preprint arXiv:2505.22959. Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2025b. Testeval: Bench- marking large language models for test case gener- ation. In Findings of the Association for Computa- tional Linguistics: NAACL 2025, pages 3547–3562. Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, and 1 others. 2025c. Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers. arXiv preprint arXiv:2508.20453. Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4all: Universal fuzzing with large language mod- els. In Proceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering, pages 1–13. 10 Chenghao Yang and Ari Holtzman. 2025. How align- ment shrinks the generative horizon. arXiv preprint arXiv:2506.17871. Jihan Yao, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhard- waj, Chang Su, Jialei Wang, Yikai Zhu, Sugam De- vare, Damon Mosk-Aoyama, and 1 others. The mea- sure of all measures: Quantifying llm benchmark quality. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, and Xuan-Jing Huang. 2024. Toolsword: Unveil- ing safety issues of large language models in tool learning across three stages. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2181–2211. Zhehao Zhang, Jiaao Chen, and Diyi Yang. 2024a. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. Advances in Neural Infor- mation Processing Systems, 37:135904–135942. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024b. Agent-safetybench: Evaluating the safety of llm agents. arXiv preprint arXiv:2412.14470. Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, and William Yang Wang. 2025. Rulearena: A benchmark for rule-guided rea- soning with llms in real-world scenarios. In Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 550–572. Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. 2024. Dynamic evaluation of large language models by meta probing agents. In Proceed- ings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022. Fuzzing: a survey for roadmap. ACM Computing Surveys (CSUR), 54(11s):1–36. 11