Paper deep dive
Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin, Khandaker Rifah Tasnia, Sifat Rahman Ahona, Juena Ahmed Noshin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language-model agents act through structured tool calls whose arguments carry different risks. Untrusted content may safely influence an email body but should not determine a recipient, account, command, or credential. Existing statistical methods typically control risk over the entire action, allowing failures in rare, high-risk fields to be obscured by benign arguments. We introduce role-stratified per-field conformal risk control, a calibration layer that wraps any per-field detector and sets separate thresholds and risk budgets for semantic argument roles. For a role with prevalence $p_r$, aggregate-only certification must use an effective budget of $\alpha p_r$ to guarantee role-specific risk $\alpha$, whereas role-stratified calibration certifies each sufficiently sampled role directly with a finite-sample guarantee; rarer roles are handled by pooled certification. Across AgentDojo and InjecAgent with six language models, the empirical utility gap tracks this predicted price of coarseness, and our method achieves the most consistent role-specific budget compliance under model and attack transfer, detector noise, gradual drift, unseen tool suites, and adaptive attacks. It provides formal per-role guarantees under exchangeability or after recalibration, and empirical compliance under frozen distribution shift. These results suggest that structured tool calls should be certified at the semantic-role level, not the whole action.
Tags
Links
- Source: https://arxiv.org/abs/2607.24343v1
- Canonical: https://arxiv.org/abs/2607.24343v1
Trouble viewing inline? Open PDF directly →
Full Text
91,094 characters extracted from source content.
Expand or collapse full text
Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin, Khandaker Rifah Tasnia, Sifat Rahman Ahona, Juena Ahmed Noshin Abstract Language-model agents act through structured tool calls whose arguments carry different risks. Untrusted content may safely influence an email body but should not determine a recipient, account, command, or credential. Existing statistical methods typically control risk over the entire action, allowing failures in rare, high-risk fields to be obscured by benign arguments. We introduce role-stratified per-field conformal risk control, a calibration layer that wraps any per-field detector and sets separate thresholds and risk budgets for semantic argument roles. For a role with prevalence prp_r, aggregate-only certification must use an effective budget of αprα p_r to guarantee role-specific risk α, whereas role-stratified calibration certifies each sufficiently sampled role directly with a finite-sample guarantee; rarer roles are handled by pooled certification. Across AgentDojo and InjecAgent with six language models, the empirical utility gap tracks this predicted price of coarseness, and our method achieves the most consistent role-specific budget compliance under model and attack transfer, detector noise, gradual drift, unseen tool suites, and adaptive attacks. It provides formal per-role guarantees under exchangeability or after recalibration, and empirical compliance under frozen shift. These results suggest that structured tool calls should be certified at the semantic-role level, not the whole action. Introduction Consider an agent asked to pay an invoice. It reads the amount from an email, but hidden text quietly redirects the payment to a new account. Payment calls specify an account and amount, just as a send_email call specifies a recipient, body, and attachments. Untrusted content may safely influence an email body but should not determine a recipient, payment account, or credential. When text from a webpage, email, or retrieved document changes such a high-risk field, the agent may act against the user’s intent (Greshake et al. 2023). Existing defenses follow two main approaches. Security-enforcement systems track high-risk fields and apply hard allow-or-deny rules (Debenedetti et al. 2025; Costa et al. 2025; Fan et al. 2026). These systems can provide strong semantic guarantees, including noninterference, but do not offer tunable risk budgets or statistical bounds on residual violations. Statistical calibration methods, including conformal risk control (CRC), provide tunable finite-sample guarantees but usually control the action as a whole (Angelopoulos et al. 2024; Feng et al. 2026). Action-level control can hide failures in rare high-risk fields: averaging across arguments may keep aggregate loss below budget even when a recipient, account, or credential field exceeds its limit, so the unit being certified differs from the unit that can cause harm. We address this mismatch with role-stratified per-field conformal risk control. Each argument is assigned a semantic role, such as target, credential, command, or content, with its own threshold and risk budget. This applies class-conditional, or Mondrian, conformal control at the role level (Vovk et al. 2003; Ding et al. 2023). Stratification prevents aggregate dilution, while conformal calibration provides finite-sample validity and supports recalibration under changing conditions. The method can wrap any per-field detector and improves directly as the detector improves. Our central thesis is that the appropriate calibration stratum is the semantic role of each argument. Our contributions organize around that claim. • Why role-level control is necessary. Aggregate certification dilutes rare-role failures: a controller that observes only aggregate loss inflates a role-r violation rate by 1/pr1/p_r (Proposition 1) and can certify the role only by shrinking its effective budget to αprα\,p_r (Proposition 2). Theorem 1 orders observation channels by this price of coarseness, specializing Blackwell’s comparison of experiments (Blackwell 1953) to conformal risk certification. • How to obtain it. Role-stratified per-field CRC assigns each role its own budget, a certifiability floor of 1/(nr+1)1/(n_r+1), and a simultaneous high-probability certificate over the final calibration strata, with pooling for rare roles (Theorem 2). Label-conditional calibration further yields a prevalence-invariant attack-conditional guarantee on Vatt(r)V_att(r) (Podkopaev and Ramdas 2021); in the zero-budget limit the method recovers detector-relative noninterference (Proposition 3). • How it behaves in practice. Across six models and two benchmarks, the empirical utility gap follows the predicted price of coarseness. Role-stratified per-field CRC remains compliant under transfer, unseen suites, detector noise, gradual drift, and adaptive attacks. A 2×22×2 ablation shows stratification provides the main robustness gain, while conformal recalibration restores finite-sample validity under the new condition. We use certificate only under exchangeability or after recalibration. Under frozen distribution shift, we report empirical budget compliance rather than a conformal certificate. Figure 1: (a) Utility versus target violation on GPT-4o at a 2%2\% budget using the deployable score over 20 seeds. (b) Empirical compliance across eight shifted conditions: per-field CRC 100%±0%100\%± 0\%, PACT-inspired threshold 61%±33%61\%± 33\%, and FIDES-inspired labels 48%±48%48\%± 48\%. Results in panel (b) report empirical compliance under shift. Related Work Security enforcement for tool-using agents. One line of work protects tool calls through deterministic information-flow control. CaMeL isolates high-risk calls using a dual-LLM, capability-based design (Debenedetti et al. 2025). FIDES tracks integrity and confidentiality labels for individual values (Costa et al. 2025), while PACT traces argument-level provenance using the target, command, credential, and content roles adopted here (Fan et al. 2026). ARM and AgentVisor study causal provenance and semantic isolation (Chinaei 2026; Ying et al. 2026). These systems build on classical information-flow control and noninterference (Denning 1976; Volpano, Irvine, and Smith 1996). They provide strong semantic guarantees, but generally rely on fixed allow-or-deny decisions rather than tunable risk budgets or statistical bounds on residual violations. Statistical calibration and conformal risk control. A second line of work provides tunable, distribution-free guarantees through calibration. CORA applies conformal risk control to mobile GUI agents but defines risk over the action as a whole (Feng et al. 2026). Other methods control hallucination or response-level error through abstention or calibrated refusal (Abbasi-Yadkori et al. 2024; Pang et al. 2025). These approaches also operate at the response or action level rather than the semantic-role level. Our method builds on conformal risk control (Angelopoulos et al. 2024), split conformal prediction (Papadopoulos et al. 2002; Vovk, Gammerman, and Shafer 2005; Lei et al. 2018; Angelopoulos and Bates 2023), Learn-then-Test (Angelopoulos et al. 2025), and risk-controlling prediction sets (Bates et al. 2021). Its structure follows class-conditional, group-conditional, and Mondrian conformal methods (Vovk et al. 2003; Ding et al. 2023; Bairaktari, Wu, and Wu 2025). The key difference is the unit of control: we calibrate risk separately for each semantic argument role. Concurrent conformal defenses and calibration units. Concurrent methods calibrate entire trajectories or streams of agent runs (Opoku and Banahene 2026; Hultberg, Zachariah, and Ribeiro 2026; Khosravi and Huo 2026), whereas we certify semantic argument roles. These temporal and semantic calibration units are complementary. Prompt injection, benchmarks, and positioning. These research directions meet in indirect prompt injection, a practical threat to LLM-integrated systems (Greshake et al. 2023). We evaluate on AgentDojo and InjecAgent, two benchmarks for prompt-injection attacks against tool-using agents (Debenedetti et al. 2024; Zhan et al. 2024). Training- and prompt-level defenses, including StruQ, SecAlign, Spotlighting, the Instruction Hierarchy, and Jatmo, aim to prevent or weaken attacks before calibration and are therefore complementary to our method (Chen et al. 2025a, b; Hines et al. 2024; Wallace et al. 2024; Piet et al. 2024). Broader studies classify and compare prompt-injection attacks and defenses (Ji et al. 2025; Liu et al. 2024). Our contribution is a calibration layer that wraps any per-field detector with role-specific risk budgets, orthogonal to existing defenses rather than a new attack or detector. We compare controlled PACT-, FIDES-, and CaMeL-inspired proxies under a common detector to isolate enforcement granularity; these are not full reproductions of the original systems. Problem Setup and Threat Model Structured actions and per-field violations. A tool call is a structured action A=(op,x1,…,xk),A=(op,\,x_1,…,x_k), (1) where op is an operation, such as send_email, and each xix_i is a named argument. Every argument is assigned a semantic role r(i)∈ℛr(i) , such as target, credential, command, selector, control, or content. These roles carry different risks. A target or credential determines where an action goes or which authority it uses, whereas content is often expected to reflect untrusted input. Fields are the enforcement and sampling units. Semantic roles are the calibration and certification strata. A field is violated when untrusted input changes its value against the user’s intent. It is allowed when the controller permits that value to be emitted. For each role r, we define V(r)=Pr[violated∧allowed|role=r],V(r)= \! [\,violated \ |\ role=r\, ], (2) the probability that an emitted role-r field is both violated and allowed. We certify violation rates over role-specific field populations, not the safety of an individual tool call. We require V(r)≤α(r)V(r)≤α(r), with tighter budgets for higher-risk roles. What V(r)V(r) measures. The denominator of V(r)V(r) includes all emitted role-r fields, both benign and attacked, so it is the quantity optimized by the calibration loss and reported throughout. Because clean fields are included, V(r)V(r) depends on operational attack prevalence and is best read as field-stream integrity under a stated clean-attack mixture, not a prevalence-invariant leakage rate. To isolate attack-conditional behavior, we also report Vatt(r)=Pr[violated∧allowed|role=r,attacked],V_att(r)= \! [\,violated \ |\ role=r,\ attacked\, ], (3) which is prevalence-independent. By construction, an unattacked field cannot receive an attack-induced violation label, so its clean-conditional rate is zero and V(r)=π(r)Vatt(r)V(r)=π(r)\,V_att(r) for field-level attack prevalence π(r)π(r). In our benchmarks the target stream is attack-heavy (π≈85%π≈ 85\%-94%94\%), so the two are similar. They diverge in a low-prevalence deployment (Supplement A). Both differ from trace-level attack success rate (ASR), the fraction of attacked episodes in which any high-risk field is leaked, which we report as a separate security metric, not the certified object. Attacker. The attacker controls untrusted text from webpages, emails, or retrieved documents and attempts to influence one or more high-risk arguments. In the strongest setting, the attacker selects the least detectable successful prompt from several candidates and uses attack families unseen during calibration. The attacker does not control the calibration data or model weights. We also exclude gradient-based prompt optimization (Zou et al. 2023). Per-argument score. Each argument xix_i receives a nonconformity score sis_i that estimates its influence from untrusted input. Our deployable score is annotation-free: si s_i =sim(xi,untrusted context) =sim(x_i,untrusted context) (4) −sim(xi,trusted user prompt). -sim(x_i,trusted user prompt). We shift and rescale the similarity difference to [0,1][0,1]. Larger scores indicate stronger influence from untrusted context, so a field is allowed when si≤τs_i≤τ. The score does not use attacker labels. For analysis, we also evaluate an attacker-literal overlap diagnostic that measures lexical overlap between the emitted field and the injection goal. Because it reads the attacker literal used to define the violation label, its ROC-AUC of 0.930.93 partly reflects access to evaluation-specific attacker text, so we treat it as a label-coupled oracle-aided upper bound rather than the deployable default. This diagnostic is distinct from the oracle-aided counterfactual clean-vs-injected overlap (0.760.76 ROC-AUC) evaluated only in the non-verbatim stress test (Supplement A). Both scores use the same calibration and decision rule. Only the score computation changes, as detailed in Supplement A. The Aggregate-Budget Failure Aggregate control bounds expected harm by averaging risk across all fields in a tool call. This is misleading when structured calls mix fields with very different consequences: if violations concentrate in a rare high-risk role, many benign fields keep the average below budget even when that role exceeds its limit, and raising the aggregate budget only weakens the overall constraint without protecting the risky field. Proposition 1 (Concentration gap). Let pr>0p_r>0 be the fraction of fields with role r, where ∑rpr=1 _rp_r=1. Define aggregate field-averaged harm as V¯=∑rprV(r) V= _rp_rV(r). If aggregate control guarantees V¯≤αagg V≤ _agg, then V(r)≤αagg/prV(r)≤ _agg/p_r for every role r, and this bound is tight. Proof. Because all terms are non-negative, prV(r)≤V¯≤αaggp_rV(r)≤ V≤ _agg, so V(r)≤αagg/prV(r)≤ _agg/p_r. Equality is possible when all violations occur in role r. ∎ Thus, aggregate protection weakens in inverse proportion to role prevalence. In the full clean-attack evaluation population used to compute V¯ V, target fields have prevalence ptarget≈0.12p_ target≈ 0.12 (pooled across the six models, 0.060.06-0.210.21 per model), consistent with the denominator of V(r)V(r) in Equation (2), which likewise includes all role-r fields. Thus, even before distribution shift, a 1%1\% aggregate budget can permit roughly 1%/0.12≈8%1\%/0.12≈ 8\% target violations. The next result shows that aggregate-only certification must reduce its effective budget by the same factor prp_r. Two distributions can have identical scores and aggregate losses while placing every violation in different roles. An aggregate-only controller cannot distinguish them. Definition 1 (Aggregate-measurable controller). Let Φ(A) (A) denote the score information visible to the controller for action A, such as action-level or field-level detector scores. Define the action’s average violation loss as Lagg(A)=1|A|∑i∈Afield i is violated and allowed.L_agg(A)= 1|A| _i∈ A1\field $i$ is violated and allowed\. A policy is aggregate-measurable if it uses calibration data only through (Φ(Aj),Lagg(Aj))j\( (A_j),L_agg(A_j))\_j and makes decisions using only Φ(A) (A). Aggregate control and single-threshold defenses belong to this class. Role-stratified CRC does not, because it also observes the role of the violated field. Proposition 2 (Aggregate control cannot certify a rare role). Fix a role r whose field-level prevalence pr=a/K∈(0,1)p_r=a/K∈(0,1) is realizable in a finite benchmark (integers 1≤a≤K−11≤ a≤ K-1), and a budget α∈(0,1)α∈(0,1). There exist two distributions, P0P_0 and P1P_1, with the same role prevalence prp_r and identical observables (Φ,Lagg)( ,L_agg). Any aggregate-measurable controller π therefore has the same aggregate violation V¯(π) V(π) and benign utility under both distributions. However, V(r)=0V(r)=0 under P0P_0, while V(r)=V¯(π)/prV(r)= V(π)/p_r under P1P_1. To guarantee V(r)≤αV(r)≤α under both distributions, the controller must therefore enforce V¯(π)≤αpr V(π)≤α\,p_r. This shrinks the effective aggregate budget by a factor of prp_r and lowers utility even under P0P_0, where role r is never violated. Role-stratified calibration instead guarantees V(r)≤αV(r)≤α whenever α≥1/(nr+1)α≥ 1/(n_r+1). Proof sketch. Write the field-level prevalence as pr=a/Kp_r=a/K with integers 1≤a≤K−11≤ a≤ K-1; every prevalence realizable in a finite benchmark is rational and arises this way, and the special case a=1a=1 recovers K=1/prK=1/p_r. Give each action K fields, a of which have role r. Mark some actions as bad by violating exactly one field while preserving the same score distribution for bad and clean actions. Under P1P_1, place the violation on a role-r field; under P0P_0, place it on a content field. The controller therefore behaves identically under both distributions, yet under P1P_1 all harm concentrates on role r, giving V(r)=V¯(π)/prV(r)= V(π)/p_r. The full proof appears in Supplement D. Propositions 1 and 2 are endpoints of a broader ordering. We order controllers by observation channel, writing Φ1⪯Φ2 _1 _2 when Φ2 _2 refines Φ1 _1. The hierarchy consists of the aggregate channel Φagg _agg, the role channel Φrole _role, which adds each field’s role, and the field channel Φfield _field. Theorem 1 (Certification-granularity frontier). Let u⋆(β)u (β) be the largest benign utility of any Φagg _agg-measurable controller satisfying aggregate violation V¯≤β V≤β. Then: (i) Achievability. On Φrole _role, Equation (5) certifies every budget vector satisfying α(r)≥1/(nr+1)α(r)≥ 1/(n_r+1). (i) Price of coarseness. Any Φagg _agg-measurable controller that certifies V(r)≤αV(r)≤α must enforce V¯≤αpr V≤α\,p_r, so its utility is at most u⋆(αpr)u (α p_r). (i) Monotonicity. At fixed utility, the set of certifiable budget vectors is nondecreasing along Φagg⪯Φrole⪯Φfield _agg _role _field. Part (i) follows Blackwell’s comparison of experiments (Blackwell 1953). Our contribution is the closed-form price of coarseness in part (i) and the constructive conformal method achieving the bound in part (i). The full proof appears in Supplement D. The empirical results reflect this gap. On AgentDojo with GPT-4o, a 10%10\% aggregate budget allows 10.5%10.5\% target violations, whereas per-field CRC allows 0%0\% (Supplement B, Fig. B.1 and Table B.1). On InjecAgent, under a 10%10\% aggregate budget, aggregate control reaches only 2.9%2.9\% overall violation while allowing 100%100\% of attacked target fields (Supplement B). Reweighting does not remove the problem because any single-threshold action loss can still dilute role-specific failures. Action loss Viol. % Util. % Mean-aggregate CRC 1.1-5.0 23.6-34.3 Risk-weighted 0.6-3.2 14.7-32.6 Inv.-prevalence weighted 0.0-3.2 0.4-32.6 Max-risk (any high-risk) 1.1-21.2 23.3-43.0 Per-field CRC (ours) 0.0-0.3 9.4-14.9 Table 1: Target violation and abstain utility for single-threshold baselines and per-field CRC at a 1%1\% target budget across six models, using the label-coupled diagnostic score. Ranges are across-model mins and maxes of the 2020-split mean violation in this channel-ablation setting; the single-split per-model panel in Supplement B, Table B.1 is a separate protocol, not a numerical expansion of this table. Table 1 compares mean, risk-weighted, inverse-prevalence, and max-risk single-threshold losses calibrated to a 1%1\% target budget. Each either exceeds budget, reaching up to 21.2%21.2\% target violation, or drops utility to 0.4%0.4\% on one model. Among these methods, only per-field CRC remains within budget, with 0.0%0.0\%-0.3%0.3\% violation and 9.4%9.4\%-14.9%14.9\% utility. Protecting a rare high-risk role therefore requires certifying that role directly, motivating the method developed next. Method: Role-Stratified Per-Field CRC We apply conformal risk control separately to each semantic role, assigning each role its own threshold and risk budget. Role-specific calibration. For each role r, the calibration set contains clean and injected episodes with field-level violation labels. For a field x, define Lr(x;τ)=s(x)≤τ∧x is violated,L_r(x;τ)=1\s(x)≤τ\ \ $x$ is violated\, where s(x)s(x) is its nonconformity score and τ is the allow threshold. The loss is monotone in τ: increasing the threshold allows more fields and can only add violations. Let nrn_r be the number of calibration fields with role r. Following conformal risk control (Angelopoulos et al. 2024), we choose the largest threshold whose inflated empirical risk remains within the role-specific budget α(r)α(r): τ^(r)=supτ:1nr+1(∑i=1nrLr(xi;τ)+1)≤α(r). τ(r)= \τ: 1n_r+1 ( _i=1^n_rL_r(x_i;τ)+1 )≤α(r) \. (5) Under exchangeability between calibration and test fields of the same role, [Lr(Xnew;τ^(r))]≤α(r)E[L_r(X_new; τ(r))]≤α(r). The conformal result is standard. The key design choice is to use the semantic role, rather than the full action, as the unit of control. The exchangeable unit is the individual field. Our implemented calibration includes every emitted role-r field. If an episode contains several same-role fields they may be correlated, so nrn_r can overstate the effective sample size; a conservative one-per-episode alternative is discussed in Supplement A (code archive) and is not used for the main results. Runtime enforcement. Runtime enforcement operates independently at the field level, preserving unaffected arguments rather than rejecting the entire call. Let g(r)g(r) map each role to its final stratum: g(r)=rg(r)=r for an individually certifiable role, and g(r)g(r) is the pooled high-risk stratum otherwise. A field of role rjr_j is then compared with the threshold τ^(g(rj)) τ(g(r_j)) of its final stratum. Algorithm 1 Role-Stratified Per-Field CRC 0: Calibration fields (xi,ri,yi)\(x_i,r_i,y_i)\ with scores sis_i, violation labels yiy_i, and per-role budgets α(r)α(r). 1: Calibration: 2: for each role r∈ℛr do 3: nr←|i:ri=r|n_r←|\i:r_i=r\| 4: if 1/(nr+1)>α(r)1/(n_r+1)>α(r) then 5: g(r)←g(r)← pooled high-risk stratum (r is not individually certifiable) 6: else 7: g(r)←rg(r)← r 8: end if 9: end for 10: Form the final calibration set for each stratum in g(ℛ)g(R) 11: Compute one threshold τ^(g) τ(g) per final stratum g using Equation (5) 12: Enforcement: 13: for each argument xjx_j with role rjr_j and score sjs_j do 14: if sj≤τ^(g(rj))s_j≤ τ(g(r_j)) then 15: Allow xjx_j 16: else 17: Revert xjx_j to its trusted value when available, otherwise abstain or escalate 18: end if 19: end for Certifiability and rare roles. A role can be certified only when its calibration set is sufficiently large. From Equation (5), the smallest certifiable budget is 1/(nr+1)1/(n_r+1). If α(r)α(r) lies below this floor, the role cannot be certified separately. For target, the calibration sets contain 145145-531531 fields per model, giving floors of 0.2%0.2\%-0.7%0.7\%. This supports a 1%1\% budget, with empirical violations of 0.0%0.0\%-0.3%0.3\% across all six models, as shown in Table 1. The credential role is much rarer, with only 0-2222 calibration fields per split across the six models. Computing the finite-sample floor 1/(nr+1)1/(n_r+1) on each split and averaging over the 2020 splits gives per-model mean floors of 6.7%6.7\% (GPT-4o) to 63.6%63.6\% (Qwen2.5-7B), all far above 1%1\% (Supplement C, Table C.1). This range is a span of per-model averages, not a span of individual splits. The credential role therefore cannot be certified individually at a 1%1\% budget, so we pool it with the other high-risk roles target,credential,command\ target, credential, command\. The pooled group has a floor of 0.14%0.14\%-0.49%0.49\%, making a 1%1\% certificate possible for every model, with 0%0\% empirical credential violation. The certificate applies to the pooled group, not to credential alone: by Proposition 1 a pooled 1%1\% budget implies only the individual bound 1%/pcred∣pool1\%/p_cred , so we report credential as empirically controlled rather than individually certified. Even pooled across models only about 9191 credential fields exist, versus the ≈300≈ 300 a 1%1\%, δ=0.05δ=0.05 certificate would need. The data supports about a 3%3\% certificate. This is a data, not estimator, limitation (Supplement C, Table C.1). Relation to information-flow control. Assign each role a label in an information-flow lattice. The integrity-critical roles target,credential,command\ target, credential, command\ are sinks that untrusted input must not reach, whereas content is declassifiable. Proposition 3 (Conformal relaxation of noninterference). Under this lattice, as α(r)→0α(r)→ 0 for every integrity-critical role, the rule in Equation (5) converges to a deterministic monitor that blocks every field flagged by the detector. In the zero-budget policy limit, the method therefore recovers detector-relative noninterference. This is a conceptual limit. Finite-sample certification remains subject to the floor 1/(nr+1)1/(n_r+1). For α(r)>0α(r)>0, the method is a tunable relaxation that permits a certified residual-violation budget in exchange for utility. Unlike quantitative information flow, which bounds leakage measures (Smith 2009), this method provides a finite-sample, distribution-free bound on residual violations and can inherit any per-field detector. Simultaneous high-probability certification. Equation (5) controls each role in expectation. Deployment may instead require all final calibration strata to satisfy their budgets simultaneously. Let ℛc=r:α(r)<1R_c=\r:α(r)<1\ and let g:ℛc→g:R_c be the stratum map from Algorithm 1 (g(r)=rg(r)=r when r is individually certifiable; otherwise g(r)g(r) is the pooled high-risk stratum), with =g(ℛc)G=g(R_c). Because the per-stratum loss is Bernoulli, and treating the stratum-g calibration fields as independent draws, let U(kg(τ),ng;δg)U(k_g(τ),n_g; _g) be the exact Clopper-Pearson upper bound, where kg(τ)k_g(τ) counts calibration fields of stratum g that are both allowed and violated. This exactness requires field-level independence within each stratum; when episodes emit several correlated same-role fields, the episode-level construction of Supplement A restores it. We choose τ^g(δg)=supτ:U(kg(τ),ng;δg)≤α(g). τ_g( _g)= \τ:U(k_g(τ),n_g; _g)≤α(g) \. (6) Theorem 2 (Simultaneous stratum certificate). Let δ=∑g∈δgδ= _g _g. Then, with probability at least 1−δ1-δ over the calibration sample, V(g)≤α(g)V(g)≤α(g) holds simultaneously for every g∈g . Individually retained roles therefore receive role-specific certificates; pooled rare roles receive only a pooled-stratum certificate. The proof handles discrete scores and ties by reducing the failure event, via monotonicity, to a Clopper-Pearson underestimate at a single deterministic population boundary τg⋆=infτ:V(g)(τ)>α(g) _g = \τ:V(g)(τ)>α(g)\ rather than at a sample-dependent candidate, then applying a union bound over the final strata. The full proof appears in Supplement D. Prevalence-invariant certification. Conditioning calibration on attacked role-r fields yields, through label-conditional conformal calibration (Podkopaev and Ramdas 2021), a certificate on the attack-conditional rate Vatt(r)V_att(r) that is invariant to test-time attack prevalence π(r)π(r), whereas the mixture rate V(r)=π(r)Vatt(r)V(r)=π(r)\,V_att(r) decreases as attacks become rarer (Supplement A; Table A.1 reports the attack-heavy V≈VattV≈ V_att case). Independent confidence allocation. The confidence allocation δg\ _g\ is selected using an allocation set DallocD_alloc that is independent of the certification set DcertD_cert. Final stratum thresholds and Clopper-Pearson bounds are computed only from DcertD_cert. Conditional on DallocD_alloc, the selected allocation is fixed, so Theorem 2 applies to DcertD_cert. We divide δ uniformly across final strata using the Bonferroni allocation δg=δ/|| _g=δ/|G|. A data-dependent water-filling allocation that equalizes marginal utility across strata is also valid under concavity. However, it provides no measurable gain over the uniform split under finite-sample estimates (Supplement E), so we use uniform allocation as the deployable default. Behavior under distribution shift. A fixed per-role threshold can perform well under stable conditions, but Equation (5) assumes exchangeability between calibration and test fields of the same role, which may fail after transfer to a new model, attack family, or environment. Conformal calibration adds finite-sample validity through Theorem 2 and lets thresholds be recalibrated when the score distribution changes, restoring compliance with a budget that a frozen threshold may exceed. Proposition 4 (Per-role degradation under shift). Fix a role r and threshold τ^(r) τ(r), calibrated on source distribution PrP_r, such that Pr[Lr(X;τ^(r))]≤α(r)E_P_r[L_r(X; τ(r))]≤α(r) and Lr∈[0,1]L_r∈[0,1]. For any test distribution QrQ_r, Qr[Lr(X;τ^(r))]≤α(r)+TV(Pr,Qr)E_Q_r[L_r(X; τ(r))]≤α(r)+TV(P_r,Q_r). Recalibrating on labeled samples from QrQ_r removes the shift term and restores a fresh certificate under exchangeability with QrQ_r. This is the standard bounded-function expectation-difference bound (Barber et al. 2023), not a distribution-free certificate for shifted data. It is useful only when TV(Pr,Qr)TV(P_r,Q_r) is known or tightly bounded, and motivates recalibration. Together with Proposition 1, it shows that aggregate control suffers a 1/pr1/p_r inflation even before shift, whereas per-role control degrades only with the removable role-specific shift TV(Pr,Qr)TV(P_r,Q_r). Experiments Setup. We evaluate on AgentDojo (Debenedetti et al. 2024), which covers banking, workspace, Slack, and travel, and on InjecAgent (Zhan et al. 2024). The six models are GPT-4o and GPT-4o-mini (OpenAI 2024), Gemini 2.5 Pro and Flash (Gemini Team 2025), Llama 3.3 70B (Llama Team 2024), and Qwen2.5-7B (Yang et al. 2024). We run a controlled trace-replay evaluation on real agent traces from six frontier models under live prompt-injection attacks: scores are replayed from fixed trajectories, and calibration/test splits are resampled without additional model calls. Trace replay isolates field-level calibration behavior and makes every number reproducible from the archive. Role budgets are α(r)=1%α(r)=1\% (in-distribution) or 2%2\% (shift) for target/credential. α=0.10α=0.10 is the aggregate budget for whole-action baselines only. We measure utility under conservative abstention and report over-intervention with every safety result. Split sizes, hyperparameters, seeds, and reproduction scripts are provided in the code archive; Supplement F summarizes reproducibility and infrastructure. Notation is summarized in Supplement G. Safety and utility. Abstain utility is trace-replay task success when flagged calls are rejected; revert utility restores flagged fields to trusted values. Because neither intervention re-runs the agent, both estimate utility on frozen trajectories. Over-intervention is the fraction of benign fields modified or blocked, and shift compliance is the fraction of shifted conditions meeting the target budget. We compare these quantities with target violation at matched operating points (Figure 1a). Under shift, per-field CRC reaches 0%0\% ASR at 30.6%30.6\% over-intervention. The trade-off is in-distribution utility: there the PACT-inspired threshold attains higher utility at similar target safety, with 27.2%27.2\%/32.3%32.3\% abstain/revert utility, compared with 9.9%9.9\%/25.2%25.2\% for per-field CRC. Table 2 reports the complete safety-utility comparison; per-model aggregate-failure details appear in Supplement B, Table B.1. Safety with an annotation-free detector. The detector score uses no attacker-text annotations, though conformal calibration still uses field-level violation outcomes. At a 2%2\% target budget, both the label-coupled diagnostic score and the annotation-free provenance score keep worst-case target violation ≤0.3%≤ 0.3\% with at least 95%95\% held-out compliance across all six models, despite the deployable score’s much lower ROC-AUC (0.710.71 versus 0.930.93) (Supplement C, Table C.2). Calibration thus determines validity, while detector quality primarily determines utility. Defense Tun. Cert. Shift % ASR Abs. Rev. Over No defense × × - 39.7 35.1 35.1 0.0 CaMeL-insp. (whole-call) × × - 1.0 22.0 27.2 44.8 FIDES-insp. (IFC) × × 48±4848±48 2.1 22.2 27.5 30.4 PACT-insp. (per-arg.) × × 61±3361±33 2.1 27.2 32.3 6.6 Aggregate CRC ✓ agg. - 6.0 27.4 35.0 16.4 Per-field CRC (ours) ✓ role 100±0100±0 0.0 9.9 25.2 30.6 Table 2: Six-model trace-replay comparison (deployable provenance score). Tun./Cert. = tunable budget / certificate level (agg. or per-role); Shift % = target compliance on eight frozen transfers; Abs./Rev./Over = abstain utility / revert utility / over-intervention. Security rows are controlled proxies, not full PACT/FIDES/CaMeL reproductions. Utility is estimated on frozen trajectories. The price of coarseness: channel ablation. Table 1 confirms Theorem 1: aggregate observation either violates the 1%1\% target budget or sharply reduces utility, whereas role observation satisfies the budget with nonzero utility. As the α→0α→ 0 endpoint of Proposition 3, block-all baselines achieve at most 1%1\% ASR but require at least 44.8%44.8\% over-intervention (Table 2). Distribution shift. We freeze thresholds calibrated on a source distribution and evaluate them on eight held-out attack and model-transfer conditions. Each condition contains 725725-1,3461,346 test fields and uses a 2%2\% target budget. Across 2020 seeds, per-field CRC satisfies the budget in 100%±0%100\%± 0\% of shifted conditions, compared with 61%±33%61\%± 33\% for the PACT-inspired per-argument threshold and 48%±48%48\%± 48\% for the FIDES-inspired labels, as shown in Figure 1b. Its worst-case empirical target violation is 0%0\% in all eight conditions. The PACT-inspired per-argument threshold averages 1.5%1.5\% violation and stays within budget in six of the eight conditions, with paired Wilcoxon p=7.8×10−3p=7.8× 10^-3. What drives the gain? Table 3 separates stratification and conformal calibration through a 2×22×2 ablation on the same shifted fields. With thresholds frozen on the source distribution, stratification is the main source of robustness. Fixed per-role control and per-field CRC both achieve 100%100\% compliance and 0%0\% worst-case violation. Conformal single-threshold aggregate control is 42.542.5 percentage points lower in compliance than per-field CRC, with p=8.2×10−5p=8.2× 10^-5. Method Compl. % Worst viol. % Frozen on source Fixed agg. (PACT), single 65.6±28.265.6±28.2 2.9±2.32.9±2.3 CRC aggregate, single 57.5±18.357.5±18.3 5.2±1.85.2±1.8 Fixed per-role 100.0±0.0100.0 0.0 0.0±0.00.0 0.0 CRC per-role 100.0±0.0100.0 0.0 0.0±0.00.0 0.0 Recalibrated on shifted split Fixed per-role 89.4±12.789.4±12.7 2.9±2.62.9±2.6 CRC per-role 97.5±6.497.5±6.4 0.8±1.60.8±1.6 Table 3: 2×22×2 ablation under shift (20 seeds; same fields as Table 2). Compl. = shift compliance. Fixed agg. (PACT), single is a source-frozen single threshold, not the per-argument proxy in Table 2 (65.6%65.6\% vs. 61%61\%). Stratification drives frozen robustness; conformal recalibration restores a certificate under QrQ_r. After recalibration on a shifted split, conformal correction adds finite-sample value. It raises compliance by 8.18.1 percentage points, with p=2.8×10−2p=2.8× 10^-2, and lowers worst-case violation from 2.9%2.9\% to 0.8%0.8\%. Under detector noise, it adds 6.26.2 points, with p=2.5×10−2p=2.5× 10^-2. Even after both methods are recalibrated, per-field CRC leads by 36.236.2 percentage points, with p=1.2×10−4p=1.2× 10^-4. Unseen tool suites. We calibrate on three AgentDojo suites and evaluate on the held-out fourth, whose tools, argument names, and environment are disjoint. This produces 2424 model-by-suite conditions at a 2%2\% budget over 2020 seeds. Frozen per-field CRC is compliant in all 24/2424/24 conditions, with 1.5%1.5\% worst-case target violation. The PACT-inspired fixed single-threshold proxy is compliant in 14/2414/24, aggregate CRC in 12/2412/24, and fixed per-role control in 19/2419/24 (Supplement E, Table E.1 and Fig. E.6). Deployable reversion narrows the utility gap to fixed baselines (Supplement E). Non-verbatim attacks and gradual drift. When the attacker’s exact text is removed, overlap-based target detection falls from 0.760.76 to 0.390.39 ROC-AUC. A frozen verbatim threshold then exceeds the budget, reaching 4.8%4.8\% mean and up to 60%60\% worst-case violation, whereas recalibrated per-field CRC stays within budget (≤0.06%≤ 0.06\%) regardless of detector quality (Supplement A, Fig. A.1). Under a gradual detector-noise ramp, the frozen PACT-inspired threshold rises from 1.7%1.7\% to 5.7%5.7\% violation and exceeds the budget after the initial stage. The deterministic per-role threshold crosses the 2%2\% budget once TV(P,Q)≳0.41TV(P,Q) 0.41. Online recalibration remains between 0.1%0.1\% and 0.5%0.5\% across all nine stages, consistent with Proposition 4 (Supplement E, Figs. E.4-E.5). Adaptive attacks and confidence allocation. Under adaptive selection of the least detectable successful injection from unseen families, the PACT-inspired proxy reaches 3.6%3.6\%-6.5%6.5\% target violation (up to 3.25×3.25× the 2%2\% limit) while per-field CRC stays at 0.0%0.0\%; under the independent Dalloc/DcertD_alloc/D_cert protocol, uniform Bonferroni matches water-filling on held-out utility. Discussion and Limitations Semantic-role stratification is the main source of robustness in our experiments, while conformal recalibration supplies finite-sample validity when labeled target-condition data are available. Our claims therefore fall into three levels: theoretically certified under exchangeable calibration/test role populations; empirically robust but not freshly certified under frozen transfer and unseen suites; and re-certified after recalibration on labeled target-condition data. These guarantees carry a utility cost: deployable value reversion achieves 25.2%25.2\% utility versus 32.3%32.3\% for the PACT-inspired proxy. Rare roles may require pooled certification; the guarantee covers field-value integrity, not tool selection, omitted actions, call ordering, or leakage through low-risk fields; and abstain/revert utility on recorded traces does not capture live replanning after a block or revert. Conclusion Aggregate action-level control pays a price of coarseness: it inflates a rare role’s violation rate by 1/pr1/p_r and shrinks its effective budget to αprα p_r, at a utility cost. Role-stratified per-field CRC instead certifies risk at the semantic-role level. Across models and stress tests, the utility gap tracks this price while the method delivers the most consistent role-specific compliance, certified under exchangeability or after recalibration and improving with any per-field detector. Future work extends the guarantee to tool selection, omitted actions, and call ordering, and validates the method under interactive deployment. This supplement collects proofs (including the certification-granularity frontier and the conformal IFC relaxation), per-model tables, additional stress tests, and a notation summary for the main paper. Section letters match the references in the main text (Supplement A-G). Unless noted, all numbers are from a controlled trace-replay evaluation on real agent traces: 2020 random calibration/test splits of cached traces with no new model calls and no post-intervention re-execution. The sixth model in the evaluation panel is Qwen2.5-7B. Appendix A Detector Details and Exchangeability Primary per-argument scores. The main comparison uses two scores. Both are shifted and rescaled to [0,1][0,1], with larger values indicating stronger untrusted influence. A field is allowed when si≤τs_i≤τ. The calibration and decision procedure (Equation (5) and Algorithm 1 of the main paper) is identical for each. They differ only in what information they use. The non-verbatim stress test below introduces three further score variants (literal-removed provenance, character-n-gram provenance, and a counterfactual clean-vs-injected overlap); we use one fixed name for each score throughout. Attacker-literal overlap diagnostic (label-coupled). Computed post-hoc by measuring lexical overlap between the emitted field and the injection goal (the attacker literal). Because the violation label is defined by whether the field follows the injection goal, this score reads the same attacker literal used to define the label. Its ROC-AUC of 0.930.93 therefore partly reflects access to evaluation-specific attacker text. We use it only as an oracle-aided diagnostic (an upper bound on detector quality), not as a deployment default. It is distinct from the counterfactual clean-vs-injected overlap below, which does not read the attacker literal. Annotation-free provenance (deployable default). Both oracle-free (no clean counterfactual) and annotation-free (never reads the ground-truth attacker literal): si=sim(xi,u)−sim(xi,t),s_i=sim(x_i,u)-sim(x_i,t), (A.1) where u is the untrusted context and t is the trusted user prompt. The raw similarity difference is shifted and rescaled to [0,1][0,1]. This is the score available at deployment (target-role ROC-AUC 0.710.71). Comparability caveat. ROC-AUCs measured under the standard (verbatim) attack and under the non-verbatim attack use different field populations and are not directly comparable. For example, annotation-free provenance scores 0.710.71 under the standard attack and 0.740.74 under the non-verbatim attack; the two values reflect different conditions, not a change in the score. Verbatim-reuse ablation. A ground-truth-literal ablation attributes most of the target-role signal in all overlap-based scores to verbatim reuse of injected content (target AUC 0.76→0.390.76→ 0.39 once the literal is removed). Consequently, the guarantee we certify is on field-value integrity under this detector, and improving non-verbatim detection, which the calibration layer inherits directly, is the main lever for higher utility. This matches our premise that detector gains are orthogonal to the calibration layer. Non-verbatim adaptive attacker and detector inheritance. Figure A.1 summarizes the non-verbatim stress test. We consider an attacker that places the authority-bearing value in a target field without leaving it verbatim in readable content, corresponding to the literal-removed condition above. Across six models, 20 seeds, and a 2%2\% target budget, we report three findings. (1) Evasion. A PACT-inspired threshold frozen on the verbatim condition does not transfer reliably: mean target violation rises to 4.8%4.8\%, with a worst case of 60%60\%. Per-field CRC recalibrated on the non-verbatim condition restores the budget, reaching 0.04%0.04\% violation. When the score is nearly uninformative, this requires 84%84\% over-intervention. (2) Detector inheritance. Across the tested scores, the same per-field CRC layer keeps target violation at or below 0.06%0.06\%, with at least 99%99\% budget compliance. Detector quality mainly determines utility. Among deployable scores, over-intervention falls from 84%84\% at 0.390.39 ROC-AUC for literal-removed provenance to 77%77\% at 0.740.74 ROC-AUC for annotation-free provenance. The counterfactual clean-vs-injected overlap reaches 0.760.76 ROC-AUC and 38%38\% over-intervention, but is reported only as an oracle-aided upper bound. The attacker-literal overlap diagnostic reaches 0.930.93 ROC-AUC but is excluded because it reads the attacker literal used to define the violation label. (3) Stronger deployable detection. Against a present-but-obfuscated attacker, a character-n-gram provenance score recovers signal lost by exact matching, raising ROC-AUC from 0.390.39 to 0.700.70. With the same calibration and enforcement rule, per-field CRC holds the budget while reducing over-intervention from 84%84\% to 64%64\%. Thus, better non-verbatim detection improves utility, while the calibration layer remains unchanged. Figure A.1: Non-verbatim attacks at a 2%2\% budget across six models and 20 seeds. The PACT-inspired frozen threshold exceeds budget, while per-field CRC holds. Over-intervention decreases as detector AUC improves. Field-level exchangeability. The calibration units are individual fields drawn from multiple episodes. The guarantee therefore requires field-level exchangeability within each role stratum, not episode-level exchangeability. This condition holds when fields of the same role are drawn from i.i.d. episodes. Our implemented calibration includes every emitted role-r field, contributing one calibration point per field rather than one per episode. Because each per-role calibration set is built from role-r fields alone, dependence between fields of different roles within the same episode does not affect any single-role certificate. If an episode contributes several fields of the same role, however, those fields may be correlated, and counting all of them can make nrn_r overstate the effective sample size. This distinction matters for the finite-sample certificate of Theorem 2. Its Clopper-Pearson upper bound is exactly binomial only when the role-r loss observations are independent Bernoulli draws; having independent episodes does not by itself make every individual field independent when several same-role fields come from one episode. We therefore state the assumption explicitly: the all-field certificates reported for the main experiments hold under field-level independence within each role stratum. When multiple same-role fields per episode are correlated, an inflated nrn_r enters the denominator of the bound and can make the interval anti-conservative, so the all-field numbers should be read as field-independence-based rather than as episode-level exact certificates. Two constructions remove this assumption by calibrating over independent episodes: retaining one randomly selected role-r field per episode, or using the episode-level loss Le,r(τ)=maxi∈e:ri=rLr(xi;τ)L_e,r(τ)= _i∈ e:\,r_i=rL_r(x_i;τ), which equals one when at least one role-r field in episode e is both allowed and violated. Because each episode in our datasets contributes at most one attacked (hence at most one violated) field per role, the certificate-relevant successes kr(τ)k_r(τ) are already at most one per episode; the residual within-episode dependence therefore affects only the benign denominator nrn_r. We recommend the episode-level construction whenever deployments emit several same-role fields per episode, and we report the one-per-episode sensitivity check alongside the all-field results in the code archive. Attack-conditional versus unconditional violation. The certified quantity is the unconditional role-specific violation rate V(r)=Pr[violated∧allowed∣role=r],V(r)= [violated =r], whose denominator contains all role-r fields, both benign and attacked. The attack-conditional rate is Vatt(r)=Pr[violated∧allowed∣role=r,attacked],V_att(r)= [violated =r,attacked], By construction, an unattacked field cannot receive an attack-induced violation label, so the clean-conditional rate Vclean(r)=0V_clean(r)=0 and the general decomposition V(r)=π(r)Vatt(r)+(1−π(r))Vclean(r)V(r)=π(r)\,V_att(r)+(1-π(r))\,V_clean(r) reduces to V(r)=π(r)Vatt(r),V(r)=π(r)\,V_att(r), where π(r)π(r) is the field-level attack prevalence for role r. Thus, at fixed Vatt(r)V_att(r), the unconditional rate V(r)V(r) decreases linearly as attacks become rarer. We therefore report Vatt(r)V_att(r) alongside V(r)V(r), so the certificate is not interpreted as a prevalence-invariant guarantee. For the target role, the field stream is attack-heavy across all six models, with π(target)=85%π( target)=85\%-94%94\% and mean 91%91\% (Table A.1). Accordingly, V(target)V( target) and Vatt(target)V_att( target) are close. Per-field CRC keeps both at or below 0.4%0.4\%, whereas aggregate CRC reaches Vatt(target)=5.5%V_att( target)=5.5\%. In a low-prevalence deployment, aggregate control could satisfy an unconditional budget while still allowing a large fraction of attacked fields. Calibrating only on attacked role-r fields instead gives a label-conditional conformal certificate on Vatt(r)V_att(r), which is invariant to test-time attack prevalence π(r)π(r). Aggregate CRC Per-field CRC Model π % V % VattV_att % V % VattV_att % Gemini 2.5 Flash 94.1 2.17 2.31 0.00 0.00 Gemini 2.5 Pro 91.1 3.30 3.61 0.00 0.00 GPT-4o-mini 93.2 3.44 3.70 0.00 0.00 GPT-4o 92.4 5.02 5.45 0.00 0.00 Llama 3.3 70B 89.6 4.24 4.71 0.33 0.38 Qwen2.5-7B 84.9 1.11 1.30 0.00 0.00 Table A.1: Unconditional V(target)V( target) vs. attack-conditional Vatt(target)V_att( target) and field-level attack prevalence π (α=0.10α=0.10, 20 seeds). The attack-heavy field stream (π=85π=85-94%94\%) makes V≈VattV≈ V_att, so per-field CRC’s safety is not a rare-attack artifact. Reproduced by experiments/attack_conditional_analysis.py. Appendix B The Aggregate-Budget Failure: Full Evidence Figure B.1 shows target-role violation as a function of the aggregate budget α on GPT-4o. Aggregate CRC satisfies the global budget while target violation remains above its role-specific limit. Per-field CRC instead controls the target role directly and remains at zero observed target violation across the evaluated budgets. Table B.1 reports the corresponding six-model results using the label-coupled diagnostic score, which serves as an oracle-aided upper bound on detector quality. The same pattern holds across models: aggregate CRC can satisfy its action-level objective while allowing excessive target-role violations, whereas per-field CRC remains within the 1%1\% target budget. Stronger single-threshold action losses (risk-weighted, inverse-prevalence-weighted, and max-risk) are reported in the main paper; they reduce dilution in some settings but do not remove the granularity mismatch. InjecAgent replication. On InjecAgent with GPT-4o-mini under a 10%10\% aggregate budget, aggregate CRC achieves 2.9%2.9\% overall field violation while allowing 100%100\% of attacked target fields. Per-field CRC keeps target violation at 0%0\%. This secondary-benchmark pattern matches the AgentDojo dilution in Figure B.1 and Table B.1. Figure B.1: target violation vs. α: aggregate CRC leaks. Per-field CRC holds (GPT-4o, 20 seeds). No def. target violation % Utility % Model ASR % No def. Agg. CRC PACT† PF CRC PACT† PFab_ab PFrev_rev Gemini 2.5 Flash 49.1 43.8 2.0 1.3 0.0 25.7 7.7 27.9 Gemini 2.5 Pro 23.5 26.3 4.0 0.0 0.0 34.7 14.5 27.2 GPT-4o-mini 53.5 35.6 5.4 3.6 0.0 26.4 7.1 18.8 GPT-4o 57.1 35.7 10.5 2.3 0.0 30.6 10.7 28.9 Llama 3.3 70B 54.0 32.6 8.7 0.0 0.0 20.8 9.4 28.3 Qwen2.5-7B 0.7 0.0 0.0 0.0 0.0 25.1 9.9 19.9 Table B.1: Per-model panel at a 1%1\% target budget using the label-coupled diagnostic score. This table is not a numerical expansion of Main Table 1 (channel ablation); it reports a separate per-model panel under the same score family. The two differ only in split protocol and reporting: Main Table 1 reports the per-field target violation averaged over 2020 random calibration/test splits, so a model with occasional nonzero-violation splits (Llama 3.3 70B) shows a small positive mean (0.00.0-0.3%0.3\%), whereas this panel reports a single held-out split and marks zero observed violations as 0.0%0.0\%. Aggregate CRC leaks; per-field CRC holds (0%0\% = zero observed, Clopper-Pearson bound ≤2.1%≤ 2.1\%). †PACT-inspired fixed single-threshold proxy. Undef. denotes no defense; PF CRC is role-stratified per-field CRC; PFab_ab/PFrev_rev are its abstain/revert utilities. Appendix C Certifiability and Deployable-Detector Results Table C.1 details certifiability for the credential role. Across the 2020 group splits, the number of credential calibration fields is small and varies substantially by model: from 22-66 for Llama 3.3 70B to 88-2222 for GPT-4o, while some Qwen2.5-7B splits contain no credential fields. For each split, we compute the finite-sample floor 1/(nr+1)1/(n_r+1), then average over the 2020 splits. The resulting per-model mean floors range from 6.7%6.7\% to 63.6%63.6\%. This range is across per-model averaged floors, not across individual splits; the Qwen2.5-7B mean is increased by splits with nr=0n_r=0, whose floor is 100%100\%. Because every per-model mean floor exceeds 1%1\%, credential cannot be certified individually at a 1%1\% budget. Pooling it with the other high-risk roles produces a certifiable stratum, with floors of 0.14%0.14\%-0.49%0.49\% and 0%0\% empirical credential violation. The certificate, however, applies to the pooled high-risk average, not to credential alone. An individual high-probability certificate would require approximately 300300 credential fields, compared with approximately 9191 available when pooling across models. Table C.2 compares the label-coupled diagnostic score with the annotation-free provenance score. Although the deployable score has lower ROC-AUC, 0.710.71 versus 0.930.93, it achieves similar safety: worst-case target violation remains at or below 0.3%0.3\%, with at least 95%95\% held-out compliance across all six models. This supports the main distinction that conformal calibration determines risk control, while detector quality primarily affects over-intervention and utility. credential alone Pooled high-risk Model ncaln_cal range Mean floor Floor Cert.@1% (viol) Gemini 2.5 Flash 3-18 10.9% 0.19% ✓ (0%) Gemini 2.5 Pro 2-14 15.1% 0.27% ✓ (0%) GPT-4o-mini 4-18 9.4% 0.14% ✓ (0%) GPT-4o 8-22 6.7% 0.20% ✓ (0%) Llama 3.3 70B 2-6 20.6% 0.27% ✓ (0%) Qwen2.5-7B 0-10 63.6% 0.49% ✓ (0%) Table C.1: credential certifiability over 2020 group splits. ncaln_cal range is the actual integer count of credential calibration fields (minimum-maximum across the 2020 splits). Mean floor is the finite-sample floor 1/(nr+1)1/(n_r+1) computed on each split and then averaged over the 2020 splits. Alone it is ≫1% 1\% for every model, so the 6.7%6.7\%-63.6%63.6\% headline range is a span of per-model averaged floors, not a span of individual splits. The pooled high-risk stratum is certifiable at 1%1\% (stratum average, not individual credential). Label-coupled diag. Deployable prov. Model viol. o.-int. compl. viol. o.-int. compl. Gemini 2.5 Flash 0.0 15.8 100 0.0 16.5 100 Gemini 2.5 Pro 0.0 15.8 100 0.0 15.8 100 GPT-4o-mini 0.0 17.0 100 0.0 16.3 100 GPT-4o 0.0 25.0 100 0.0 26.1 100 Llama 3.3 70B 0.3 15.8 95 0.0 15.6 100 Qwen2.5-7B 0.0 8.8 100 0.2 8.4 95 Table C.2: Per-model deployable vs. label-coupled score (2%2\% budget, 20 seeds): comparable safety despite lower ROC-AUC (0.71 vs. 0.93). Columns: target violation (%), over-intervention (%), and budget compliance (%). Appendix D Proofs Proposition 1 (concentration gap) is proved in the main paper. We give the remaining proofs here. Proposition 2 (Aggregate control cannot certify a rare role) Recall that an aggregate-measurable controller uses calibration data only through the observables (Φ,Lagg)( ,L_agg) and does not observe per-role violation labels. Proof. Because the field-level prevalence in any finite benchmark is rational, write pr=a/Kp_r=a/K with integers 1≤a≤K−11≤ a≤ K-1; the special case a=1a=1 recovers K=1/prK=1/p_r. Consider actions with K fields, of which fields 1,…,a1,…,a have role r and fields a+1,…,Ka+1,…,K have role content, so the field-level prevalence of role r is exactly pr=a/Kp_r=a/K. A fraction b of actions are bad and the remainder are clean. The action score S=Φ(A)S= (A) is drawn from the same fixed distribution G for bad and clean actions, so the detector does not reveal which field is violated. Each bad action contains exactly one violated field, while each clean action contains none. Under P1P_1, the violated field in every bad action is a role-r field (say field 11). Under P0P_0, it is a content field (say field K). The two distributions have the same b, the same score law G, and the same aggregate loss: every bad action that is allowed contributes Lagg=1KL_agg= 1K under both P0P_0 and P1P_1. Therefore, the joint distribution of (Φ,Lagg)( ,L_agg) is identical in the two worlds. Because an aggregate-measurable controller π depends only on these observables, its possibly randomized decision rule has the same distribution under P0P_0 and P1P_1. Its admission decisions, benign utility, and aggregate violation V¯(π) V(π) are therefore identical under both distributions. Let q=Pr[allow∣bad]q= [allow ] under π. A bad allowed action contains exactly one violated-and-allowed field among its K fields, so the aggregate rate is V¯(π)=bqK. V(π)= bqK. Under P1P_1 that violated-and-allowed field is one of the a role-r fields per action, and role-r fields number a per action, so V(r)=bqa=KaV¯(π)=V¯(π)pr.V(r)= bqa= Ka\, V(π)= V(π)p_r. Under P0P_0, no role-r field is ever violated, so V(r)=0V(r)=0. Therefore, guaranteeing V(r)≤αV(r)≤α under P1P_1 requires V¯(π)≤αpr. V(π)≤α\,p_r. Because the controller behaves identically under P0P_0, it incurs the same utility cost there even though role r is never violated. By contrast, role-stratified calibration observes the role labels and applies the standard split-CRC guarantee directly to the nrn_r calibration fields of role r. ∎ Theorem 1 (Certification-granularity frontier) Proof. (i) Achievability. Fix a role r and restrict calibration to its nrn_r fields together with one exchangeable test field. Because Lr(x;τ)∈[0,1]L_r(x;τ)∈[0,1] is nondecreasing in τ, Equation (5) of the main paper is the split conformal risk control selector of Angelopoulos et al. (2024). Therefore, [Lr(Xnew;τ^(r))]≤α(r)E [L_r (X_new; τ(r) ) ]≤α(r) whenever the feasible set is nonempty, equivalently when α(r)≥1nr+1.α(r)≥ 1n_r+1. The selector uses only the field’s role, score, and calibration violation label, so it is measurable with respect to the role-observing channel Φrole _role. Applying the same construction separately to each role certifies the full per-role budget vector. (i) Price of coarseness. Proposition 2 constructs two distributions P0P_0 and P1P_1 with the same joint law of (Φ,Lagg)( ,L_agg), but whose role-r violation rates differ by the factor 1/pr1/p_r. An aggregate-measurable controller cannot distinguish these distributions. Therefore, to certify V(r)≤αV(r)≤α under both, it must enforce V¯≤αpr. V≤α\,p_r. By definition, the greatest benign utility available to any Φagg _agg-measurable controller satisfying this aggregate constraint is u⋆(αpr).u (α p_r). Thus, the factor prp_r is the utility price of certifying a rare role through aggregate observations. (i) Monotonicity. Suppose Φ1⪯Φ2 _1 _2, so that Φ2 _2 refines Φ1 _1. Every policy measurable with respect to Φ1 _1 is also measurable with respect to Φ2 _2. Therefore, at any fixed utility level, the set of certifiable budget vectors cannot shrink as the observation channel becomes more informative: Φagg⪯Φrole⪯Φfield. _agg _role _field. This is Blackwell’s comparison of experiments (Blackwell 1953) applied to the risk-certification decision problem. ∎ Proposition 3 (Conformal relaxation of noninterference) Proof. Fix an integrity-critical role r. As α(r)↓0α(r) 0, the constraint in Equation (5), 1nr+1(∑i=1nrLr(xi;τ)+1)≤α(r), 1n_r+1 ( _i=1^n_rL_r(x_i;τ)+1 )≤α(r), eventually admits no threshold τ that allows any violated calibration field. The selected threshold τ^(r) τ(r) therefore falls below the smallest score assigned to a violated field, so every field flagged by the detector is blocked. In this zero-budget policy limit, the method recovers detector-relative noninterference for role r: untrusted influence cannot reach the integrity-critical field whenever the detector identifies that influence. This limit is conceptual, because finite-sample certification remains subject to the floor 1nr+1. 1n_r+1. For α(r)>0α(r)>0, Equation (5) instead selects the largest threshold whose certified risk remains within α(r)α(r), thereby trading a controlled residual-violation budget for utility. When 1nr+1>α(r), 1n_r+1>α(r), role r is not individually certifiable and must be pooled with other high-risk roles or blocked, consistent with the zero-budget monitor. ∎ Theorem 2 (Simultaneous stratum certificate) Let ℛc=r:α(r)<1R_c=\r:α(r)<1\ and let g:ℛc→g:R_c map each controlled role to its final calibration stratum (g(r)=rg(r)=r when r is individually certifiable; otherwise g(r)g(r) is the pooled high-risk stratum), with =g(ℛc)G=g(R_c). Recall that τ^g(δg)=supτ:U(kg(τ),ng;δg)≤α(g), τ_g( _g)= \τ:U(k_g(τ),n_g; _g)≤α(g) \, where U(k,n;δg)U(k,n; _g) is the one-sided Clopper-Pearson upper confidence limit and kg(τ)k_g(τ) is the number of stratum-g calibration fields that are both allowed and violated at threshold τ. Proof. Fix a final stratum g∈g . Let g=s(1)≤⋯≤s(ng)T_g=\s_(1)≤·s≤ s_(n_g)\ be the sorted unique calibration scores for that stratum, augmented with the sentinels s(0)=−∞s_(0)=-∞ and s(ng+1)=+∞s_(n_g+1)=+∞. Because Lg(x;τ)L_g(x;τ) is nondecreasing and right-continuous in τ, the empirical count kg(τ)k_g(τ) changes only at score values. Hence, τ^g(δg) τ_g( _g) is attained at a candidate in gT_g. Write V(g)(τ)=Pr[violated∧allowed∣stratum=g]V(g)(τ)= [violated =g] for the population violation rate of stratum g at threshold τ. Because a field is allowed exactly when its score satisfies s≤τs≤τ, the map τ↦V(g)(τ)τ V(g)(τ) is nondecreasing and right-continuous, the empirical count kg(τ)k_g(τ) is nondecreasing in τ, and the retained candidate τ^g τ_g satisfies U(kg(τ^g),ng;δg)≤α(g)U(k_g( τ_g),n_g; _g)≤α(g). We do not treat any calibration-derived candidate as fixed; the candidate set gT_g is random because it depends on the calibration sample. Instead we anchor the argument at the deterministic population boundary τg⋆=infτ:V(g)(τ)>α(g), _g = \τ:V(g)(τ)>α(g) \, with τg⋆=+∞ _g =+∞ if no such threshold exists. This quantity depends only on the population law of the scores, not on the calibration sample, so pointwise Clopper-Pearson validity applies at τg⋆ _g . Suppose the selected threshold is unsafe, V(g)(τ^g)>α(g)V(g)( τ_g)>α(g). Since V(g)V(g) is nondecreasing, the point τ^g τ_g lies in τ:V(g)(τ)>α(g)\τ:V(g)(τ)>α(g)\, whose infimum is τg⋆ _g , so τ^g≥τg⋆ τ_g≥ _g and therefore kg(τ^g)≥kg(τg⋆)k_g( τ_g)≥ k_g( _g ). Because U(k,n;δg)U(k,n; _g) is nondecreasing in k, U(kg(τg⋆),ng;δg)≤U(kg(τ^g),ng;δg)≤α(g).U (k_g( _g ),n_g; _g )≤ U (k_g( τ_g),n_g; _g )≤α(g). Thus the unsafe event is contained in the event U(kg(τg⋆),ng;δg)≤α(g)\U(k_g( _g ),n_g; _g)≤α(g)\, which is defined at the fixed threshold τg⋆ _g . If V(g)(τg⋆)>α(g)V(g)( _g )>α(g), this event implies U(kg(τg⋆),ng;δg)<V(g)(τg⋆)U(k_g( _g ),n_g; _g)<V(g)( _g ), a Clopper-Pearson underestimate at a fixed threshold, whose probability is at most δg _g. If right-continuity instead yields the boundary value V(g)(τg⋆)=α(g)V(g)( _g )=α(g), the unsafe event forces τ^g>τg⋆ τ_g> _g ; applying the same containment at fixed thresholds t>τg⋆t> _g with V(g)(t)>α(g)V(g)(t)>α(g) bounds each τ^g≥t\ τ_g≥ t\ by δg _g, and taking the monotone limit t↓τg⋆t _g preserves the bound. In either case, Pr[V(g)(τ^g)>α(g)]≤δg. [V(g)( τ_g)>α(g) ]≤ _g. This argument uses only monotonicity and a fixed population boundary, so it remains valid for discrete scores, ties, and flat regions of V(g)(τ)V(g)(τ). Applying a union bound over the final strata G yields Pr[∃g∈:V(g)>α(g)]≤∑g∈δg=δ. [∃\,g :V(g)>α(g) ]≤ _g _g=δ. Therefore, with probability at least 1−δ1-δ over the calibration sample, V(g)≤α(g)V(g)≤α(g) holds simultaneously for every final stratum g∈g . Individually retained roles (g(r)=rg(r)=r) receive role-specific certificates; pooled rare roles receive only a pooled-stratum certificate. If the confidence allocation δg\ _g\ is selected using an allocation set DallocD_alloc independent of the certification set DcertD_cert, then conditional on DallocD_alloc the allocation is fixed, and the same argument applies to DcertD_cert. ∎ Water-filling allocation (remark) When each stratum utility ug(δg)u_g( _g) is nondecreasing and concave, allocating δg\ _g\ by equalizing marginal weighted utilities is a concave program over the simplex ∑gδg≤δ _g _g≤δ. The uniform Bonferroni split δg=δ/|| _g=δ/|G| is always feasible. Empirically it matches data-dependent water-filling under finite-sample utility estimates (Supplement E, Figure E.3), so we use the uniform split as the deployable default. Proposition 4 (Per-role degradation under shift) Proof. Fix a role r and threshold τ^(r) τ(r) calibrated on the source distribution PrP_r. Because Lr(x;τ)∈[0,1],L_r(x;τ)∈[0,1], the standard expectation-difference bound for bounded functions (Barber et al. 2023) gives |Qr[Lr(X;τ^(r))]−Pr[Lr(X;τ^(r))]|≤TV(Pr,Qr). |E_Q_r [L_r (X; τ(r) ) ]-E_P_r [L_r (X; τ(r) ) ] | (P_r,Q_r). Combining this with the source guarantee Pr[Lr(X;τ^(r))]≤α(r)E_P_r [L_r (X; τ(r) ) ]≤α(r) yields Qr[Lr(X;τ^(r))]≤α(r)+TV(Pr,Qr).E_Q_r [L_r (X; τ(r) ) ]≤α(r)+TV(P_r,Q_r). The same argument applies when τ^(r) τ(r) is data-dependent by also taking expectation over the calibration sample. Recalibrating on labeled samples from QrQ_r replaces PrP_r with QrQ_r, removes the shift term, and restores a fresh certificate under exchangeability with QrQ_r. This bound is not itself a distribution-free certificate under shift unless TV(Pr,Qr)TV(P_r,Q_r) is known and small. Its role is to quantify how a frozen per-role threshold can degrade and to motivate recalibration after distribution shift. ∎ Appendix E Additional Stress Tests and Allocation Detector noise. Figure E.1 varies the detector-noise level σ. As noise increases, the frozen PACT-inspired threshold drifts above the 2%2\% target budget. Per-field CRC instead recalibrates at each noise level and achieves test-set budget compliance on 95%95\%-100%100\% of the sweep, compared with 45%45\%-63%63\% for the PACT-inspired proxy. Even when both methods are recalibrated at every noise level, per-field CRC retains a 28.728.7 percentage-point compliance advantage, with paired Wilcoxon p=4.8×10−8p=4.8× 10^-8. This result shows that recalibration alone does not remove the granularity mismatch: role stratification remains necessary for reliable target-role control under detector noise. Adaptive and unseen attacks. Figure E.2 evaluates an adaptive attacker that selects the least detectable successful injection and uses attack families unseen during calibration. Under this setting, the frozen PACT-inspired threshold exceeds the 2%2\% target budget, reaching 3.6%3.6\%-6.5%6.5\% target violation, or up to 3.25×3.25× the allowed limit. Per-field CRC remains at 0.0%0.0\% target violation. Attack novelty also increases evasion of the fixed threshold. Among violating target fields, 10%10\% evade the threshold for the known attack family, compared with 21%21\%-44%44\% for unseen attack families. These results show that a fixed per-argument threshold is vulnerable to adaptive selection and attack-family shift, whereas role-stratified per-field CRC continues to control the target role. Confidence-budget allocation. Figure E.3 compares uniform and data-dependent confidence allocation. The allocation δg\ _g\ is selected using DallocD_alloc, while thresholds and certificates are computed from the independent set DcertD_cert. Utility is then evaluated on held-out test data. The estimated stratum-utility curves are monotone and at least 99%99\% concave, so the water-filling allocation applies. However, the uniform Bonferroni split already achieves strong held-out utility. Data-dependent water-filling provides no improvement, with pooled utility gaps of -3.1 percentage points at δ=0.10δ=0.10 and -1.8 percentage points at δ=0.05δ=0.05. These results support the uniform split as the deployable default: it preserves the simultaneous certificate, avoids allocation overfitting, and performs competitively under finite-sample utility estimates. Role-label noise and calibration size. Under role-label noise, per-field CRC remains safer than the PACT-inspired proxy. At noise rate η=0.10η=0.10, target violation is 1.9%1.9\% for per-field CRC versus 4.1%4.1\% for the proxy. At η=0.30η=0.30, the corresponding rates are 4.5%4.5\% and 9.7%9.7\%. Although noisy role assignments weaken stratification, per-field CRC continues to reduce target violation. Calibration-size experiments confirm the finite-sample floor 1n+1. 1n+1. For example, when n=5n=5, the smallest certifiable budget is 16.7%16.7\%. This forces rare roles such as credential to be pooled with other high-risk roles rather than certified individually at a 1%1\% budget. Using deployable value reversion, per-field CRC raises utility to 19%19\%-29%29\% while maintaining 0%0\% observed target violation. Figure E.1: Detector noise: PACT-inspired fixed single-threshold drifts. Role-stratified per-field CRC recalibrates (GPT-4o). Figure E.2: Adaptive/unseen attacks: fixed single threshold evaded. Role-stratified per-field CRC holds (GPT-4o-mini). Figure E.3: Uniform vs. data-dependent confidence-budget allocation under independent Dalloc/DcertD_alloc/D_cert (δ=0.10δ=0.10, 20 seeds). Uniform is competitive on held-out utility. Continual shift trajectory. Figure E.4 evaluates deployment as a sequence of shifts rather than a single test condition. We increase detector noise from σ=0σ=0 to 0.400.40 across nine stages, using four models, 20 seeds, and a 2%2\% target budget. The frozen PACT-inspired threshold rises monotonically from 1.7%1.7\% to 5.7%5.7\% target violation and remains within budget at only 11 of 99 stages. Deterministic per-role control remains within budget throughout, with 0.5%0.5\%-1.0%1.0\% violation across all nine stages. Per-field CRC with online recalibration achieves the lowest and most stable worst-case violation, remaining between 0.1%0.1\% and 0.5%0.5\% at every stage. At high noise, online recalibration reduces violation to about half that of the deterministic per-role threshold. These results show that role stratification provides robustness across the full shift trajectory, while conformal recalibration further limits the accumulation of error as detector drift increases. Figure E.4: Detector-noise drift from σ=0σ=0 to 0.400.40 at a 2%2\% budget across four models and 20 seeds. The frozen PACT-inspired threshold rises above budget, while recalibrated per-field CRC remains stable. Figure E.5: When to recalibrate vs. TV(P,Q)TV(P,Q) (four models, 20 seeds). (a) Violation: frozen constants grow with TV. Per-field CRC flat. (b) Utility cost (abstain). Utility-safety crossover versus shift magnitude. Figure E.5 relates performance to the measured target-role distribution shift TV(P,Q)TV(P,Q). As predicted by Proposition 4, violation under frozen thresholds increases with shift magnitude. The Pearson correlation between TV(P,Q)TV(P,Q) and target violation is 0.190.19 for the PACT-inspired proxy and 0.100.10 for the deterministic per-role threshold. The deterministic per-role threshold crosses the 2%2\% target budget once TV(P,Q)≳0.41TV(P,Q) 0.41. Per-field CRC with recalibration remains nearly flat, with correlation approximately zero and target violation near 0.3%0.3\% across the full noise ramp. This crossover provides a practical recalibration rule: estimate TV(P,Q)TV(P,Q) from re-scored traces and recalibrate when the measured shift approaches the point at which the frozen threshold exceeds budget. This avoids the utility cost of recalibration when the shift is small while restoring a fresh certificate when the source threshold is no longer reliable. Leave-one-suite-out transfer. Table E.1 and Figure E.6 evaluate transfer to unseen AgentDojo suites. We calibrate on three suites and test on the held-out fourth, whose tools, argument names, and environment are disjoint from calibration. This produces 2424 model-by-suite conditions at a 2%2\% target budget over 2020 seeds. Frozen role-stratified per-field CRC is compliant in all 2424 conditions, with 1.5%1.5\% worst-case target violation. The frozen PACT-inspired fixed single-threshold proxy is compliant in 14/2414/24 conditions, aggregate CRC in 12/2412/24, and the deterministic per-role threshold in 19/2419/24. These results show that role stratification transfers more reliably than single-threshold control to unseen tool suites. However, the source conformal certificate does not extend to the held-out suite; the reported results are empirical compliance under transfer, not a fresh certificate. Defense (frozen) Compl. Worst viol. Over PACT-insp. (single) 14/24 5.8 9.3 Aggregate CRC 12/24 5.0 38.6 Det. per-role 19/24 3.8 33.0 Per-field CRC 24/24 1.5 42.2 Table E.1: Leave-one-suite-out results across 2424 conditions at a 2%2\% budget. Compl. = test compliance; Worst viol. = worst target violation %; Over = over-intervention %. Role-stratified per-field CRC is compliant in all 2424 conditions. Source certificates do not extend to the unseen suite. PACT-insp. denotes the controlled fixed single-threshold proxy. Figure E.6: Leave-one-suite-out (2424 conditions, 2%2\% budget): per-field CRC 24/2424/24. Frozen thresholds inconsistent. Appendix F Extended Limitations and Reproducibility Because we score interventions on recorded traces, abstain/revert utility does not capture live replanning after a block. PACT-/FIDES-/CaMeL-inspired comparisons isolate enforcement granularity under a shared detector and are not full reproductions. The code archive provides splits, hyperparameters, seeds, versions, raw JSON, and a script-to-experiment map; all reported experiments run from cached traces without API access. Computing infrastructure. Reproduction uses cached traces on CPU (Ubuntu 22.04.5, Python 3.10.18; requirements.txt). Development host: Intel i9-13900K, 62 GiB RAM, RTX 4090 (CUDA 12.2). Trace generation used hosted LLM APIs; regenerating traces is optional and not required to reproduce the paper’s statistics. Appendix G Notations Table G.1 lists the main symbols used in the paper and this supplement (role names such as target are omitted). Symbol Meaning Actions, roles, and scores A=(op,x1,…,xk)A=(op,\,x_1,…,x_k) Structured tool-call action xix_i / x Named argument (field); enforcement unit r(i)r(i), r, ℛR Role of argument i; role set sis_i / s(x)s(x) Nonconformity score (larger ⇒ more untrusted influence) τ Allow threshold (si≤τs_i≤τ) simsim, u, t Similarity; untrusted context; trusted prompt Risks and budgets V(r)V(r) Pr[violated∧allowed∣role=r] [violated =r] Vatt(r)V_att(r), Vclean(r)V_clean(r) Attack-/clean-conditional rates (Vclean=0V_clean=0) π(r)π(r) Attack prevalence (V=πVattV=π V_att) α(r)α(r), α, αagg _agg Role-specific, generic, and aggregate risk budgets prp_r, V¯ V Role prevalence; aggregate harm ∑rprV(r) _rp_rV(r) pcred∣poolp_cred credential prevalence in the pooled high-risk stratum Calibration and CRC Lr(x;τ)L_r(x;τ) s(x)≤τ∧x is violated1\s(x)≤τ $x$ is violated\ Le,r(τ)L_e,r(τ) Episode-level loss maxi∈e:ri=rLr(xi;τ) _i∈ e:\,r_i=rL_r(x_i;τ) nrn_r, 1/(nr+1)1/(n_r+1) Role-r calibration size; certifiability floor τ^(r) τ(r) Largest CRC threshold with inflated risk ≤α(r)≤α(r) XnewX_new, yiy_i Exchangeable test field; violation label g(r)g(r) Final stratum (r if certifiable, else pooled high-risk) Lagg(A)L_agg(A) Action-average violation loss Simultaneous certificate G Final calibration strata =g(ℛc)G=g(R_c) kg(τ)k_g(τ) Allowed-and-violated stratum-g calibration count at τ U(kg(τ),ng;δg)U(k_g(τ),n_g; _g) Exact one-sided Clopper-Pearson upper bound τ^g(δg) τ_g( _g) High-probability threshold with CP bound ≤α(g)≤α(g) δg _g, δ, ℛcR_c Per-stratum / total failure prob.; controlled roles DallocD_alloc, DcertD_cert Allocation and certification splits τg⋆ _g infτ:V(g)(τ)>α(g) \τ:V(g)(τ)>α(g)\ Observation channels and shift Φ(A) (A) Score information visible for action A Φagg _agg, Φrole _role, Φfield _field Aggregate, role, and field observation channels Φ1⪯Φ2 _1 _2, u⋆(β)u (β) Channel refinement; max utility under V¯≤β V≤β PrP_r, QrQ_r, TV(Pr,Qr)TV(P_r,Q_r) Source/test role distributions; total variation Table G.1: Main notations used throughout the paper and this supplement. References Abbasi-Yadkori et al. (2024) Abbasi-Yadkori, Y.; Kuzborskij, I.; Stutz, D.; György, A.; Fisch, A.; Doucet, A.; Beloshapka, I.; Weng, W.-H.; Yang, Y.-Y.; Szepesvári, C.; Cemgil, A. T.; and Tomasev, N. 2024. Mitigating LLM Hallucinations via Conformal Abstention. arXiv:2405.01563. Angelopoulos and Bates (2023) Angelopoulos, A. N.; and Bates, S. 2023. Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning, 16(4): 494-591. Angelopoulos et al. (2025) Angelopoulos, A. N.; Bates, S.; Candès, E. J.; Jordan, M. I.; and Lei, L. 2025. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. The Annals of Applied Statistics, 19(2): 1641-1662. Angelopoulos et al. (2024) Angelopoulos, A. N.; Bates, S.; Fisch, A.; Lei, L.; and Schuster, T. 2024. Conformal Risk Control. In International Conference on Learning Representations (ICLR). Bairaktari, Wu, and Wu (2025) Bairaktari, K.; Wu, J.; and Wu, S. 2025. Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2581-2602. PMLR. Barber et al. (2023) Barber, R. F.; Candès, E. J.; Ramdas, A.; and Tibshirani, R. J. 2023. Conformal Prediction Beyond Exchangeability. The Annals of Statistics, 51(2): 816-845. Bates et al. (2021) Bates, S.; Angelopoulos, A.; Lei, L.; Malik, J.; and Jordan, M. I. 2021. Distribution-Free, Risk-Controlling Prediction Sets. Journal of the ACM, 68(6): 43:1-43:34. Blackwell (1953) Blackwell, D. 1953. Equivalent Comparisons of Experiments. The Annals of Mathematical Statistics, 24(2): 265-272. Chen et al. (2025a) Chen, S.; Piet, J.; Sitawarin, C.; and Wagner, D. 2025a. StruQ: Defending Against Prompt Injection with Structured Queries. In 34th USENIX Security Symposium (USENIX Security 25), 2383-2400. Chen et al. (2025b) Chen, S.; Zharmagambetov, A.; Mahloujifar, S.; Chaudhuri, K.; Wagner, D.; and Guo, C. 2025b. SecAlign: Defending Against Prompt Injection with Preference Optimization. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2833-2847. Chinaei (2026) Chinaei, M. H. 2026. Causality Laundering: Denial-Feedback Leakage in Tool-Calling LLM Agents. arXiv preprint arXiv:2604.04035. Costa et al. (2025) Costa, M.; Köpf, B.; Kolluri, A.; Paverd, A.; Russinovich, M.; Salem, A.; Tople, S.; Wutschitz, L.; and Zanella-Béguelin, S. 2025. Securing AI Agents with Information-Flow Control. arXiv preprint arXiv:2505.23643. Debenedetti et al. (2025) Debenedetti, E.; Shumailov, I.; Fan, T.; Hayes, J.; Carlini, N.; Fabian, D.; Kern, C.; Shi, C.; Terzis, A.; and Tramèr, F. 2025. Defeating Prompt Injections by Design. arXiv preprint arXiv:2503.18813. Debenedetti et al. (2024) Debenedetti, E.; Zhang, J.; Balunović, M.; Beurer-Kellner, L.; Fischer, M.; and Tramèr, F. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. Denning (1976) Denning, D. E. 1976. A Lattice Model of Secure Information Flow. Communications of the ACM, 19(5): 236-243. Ding et al. (2023) Ding, T.; Angelopoulos, A. N.; Bates, S.; Jordan, M. I.; and Tibshirani, R. J. 2023. Class-Conditional Conformal Prediction with Many Classes. In Advances in Neural Information Processing Systems (NeurIPS). Fan et al. (2026) Fan, L.; Li, Z.; Tian, Y.; Wang, Y.; Li, R.; and Wang, X. 2026. The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck. arXiv preprint arXiv:2605.11039. Feng et al. (2026) Feng, Y.; Du, J.; Wang, Q.; Ma, Z.; Niu, Q.; Matsuo, Y.; Feng, L.; and Yu, L. 2026. CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation. arXiv preprint arXiv:2604.09155. Gemini Team (2025) Gemini Team. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261. Greshake et al. (2023) Greshake, K.; Abdelnabi, S.; Mishra, S.; Endres, C.; Holz, T.; and Fritz, M. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec ’23), 79-90. Hines et al. (2024) Hines, K.; Lopez, G.; Hall, M.; Zarfati, F.; Zunger, Y.; and Kiciman, E. 2024. Defending Against Indirect Prompt Injection Attacks With Spotlighting. In Proceedings of the Conference on Applied Machine Learning in Information Security (CAMLIS 2024), volume 3920 of CEUR Workshop Proceedings, 48-62. CEUR-WS.org. Hultberg, Zachariah, and Ribeiro (2026) Hultberg, B.; Zachariah, D.; and Ribeiro, A. H. 2026. Anytime-Valid Conformal Risk Control. arXiv preprint arXiv:2602.04364. Ji et al. (2025) Ji, Z.; Wang, X.; Li, Z.; Ma, P.; Gao, Y.; Wu, D.; Yan, X.; Tian, T.; and Wang, S. 2025. Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks. arXiv preprint arXiv:2511.15203. Khosravi and Huo (2026) Khosravi, H.; and Huo, X. 2026. Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained LLMs. arXiv preprint arXiv:2605.20270. Lei et al. (2018) Lei, J.; G’Sell, M.; Rinaldo, A.; Tibshirani, R. J.; and Wasserman, L. 2018. Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 113(523): 1094-1111. Liu et al. (2024) Liu, Y.; Jia, Y.; Geng, R.; Jia, J.; and Gong, N. Z. 2024. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In 33rd USENIX Security Symposium (USENIX Security 24), 1831-1847. Llama Team (2024) Llama Team. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. OpenAI (2024) OpenAI. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276. Opoku and Banahene (2026) Opoku, J.; and Banahene, D. 2026. ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift. arXiv preprint arXiv:2606.18467. Pang et al. (2025) Pang, L.; Huang, L.; Lin, J.; Wang, T.; Aue, A.; and Priebe, C. E. 2025. Taming Variability: Randomized and Bootstrapped Conformal Risk Control for LLMs. arXiv preprint arXiv:2509.23007. Papadopoulos et al. (2002) Papadopoulos, H.; Proedrou, K.; Vovk, V.; and Gammerman, A. 2002. Inductive Confidence Machines for Regression. In Machine Learning: ECML 2002, volume 2430 of Lecture Notes in Computer Science, 345-356. Springer. Piet et al. (2024) Piet, J.; Alrashed, M.; Sitawarin, C.; Chen, S.; Wei, Z.; Sun, E.; Alomair, B.; and Wagner, D. A. 2024. Jatmo: Prompt Injection Defense by Task-Specific Finetuning. In Computer Security - ESORICS 2024, Part I, volume 14982 of Lecture Notes in Computer Science, 105-124. Springer. Podkopaev and Ramdas (2021) Podkopaev, A.; and Ramdas, A. 2021. Distribution-Free Uncertainty Quantification for Classification Under Label Shift. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161, 844-853. PMLR. Smith (2009) Smith, G. 2009. On the Foundations of Quantitative Information Flow. In Foundations of Software Science and Computational Structures (FoSSaCS), volume 5504 of Lecture Notes in Computer Science, 288-302. Springer. Volpano, Irvine, and Smith (1996) Volpano, D.; Irvine, C.; and Smith, G. 1996. A Sound Type System for Secure Flow Analysis. Journal of Computer Security, 4(2-3): 167-187. Vovk, Gammerman, and Shafer (2005) Vovk, V.; Gammerman, A.; and Shafer, G. 2005. Algorithmic Learning in a Random World. Springer. Vovk et al. (2003) Vovk, V.; Lindsay, D.; Nouretdinov, I.; and Gammerman, A. 2003. Mondrian Confidence Machine. Technical report, Royal Holloway, University of London. On-line Compression Modelling Project, Working Paper #4. Wallace et al. (2024) Wallace, E.; Xiao, K.; Leike, R.; Weng, L.; Heidecke, J.; and Beutel, A. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv preprint arXiv:2404.13208. Yang et al. (2024) Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin, R.; Li, T.; Tang, T.; Xia, T.; Ren, X.; Ren, X.; Fan, Y.; Su, Y.; Zhang, Y.; Wan, Y.; Liu, Y.; Cui, Z.; Zhang, Z.; and Qiu, Z. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. Ying et al. (2026) Ying, Z.; Wang, H.; Liu, J.; Zou, Q.; Liu, A.; Yang, J.; Yang, Y.; and Liu, X. 2026. AgentVisor: Defending LLM Agents Against Prompt Injection via Semantic Virtualization. arXiv preprint arXiv:2604.24118. Zhan et al. (2024) Zhan, Q.; Liang, Z.; Ying, Z.; and Kang, D. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. In Findings of the Association for Computational Linguistics: ACL 2024, 10471-10506. Zou et al. (2023) Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043.