Paper deep dive
From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents
Xueping Gao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Activation probes can predict safety-relevant properties of language models with high area under the receiver-operating-characteristic curve (AUROC), but deployed agent monitors make thresholded alarm decisions under tight false-alarm budgets. These are different estimands. We introduce an Operational Validity Contract that fixes a monitor's target, observability, identity, timing, intervention unit, comparator, calibration, and cost. We formalize risk at the semantic request or trajectory level: when one task contains repeated alarm opportunities, row-level AUROC and false positive rate do not identify semantic-unit any-alarm risk. A confidence-certified threshold also requires enough independent negative units, a tie-safe rule, and transport to deployment. Across Models Under Pressure, LASR refusal prediction, immutable AgentDojo, and a prospectively protocol-frozen ST-WebAgentBench replication, joint activation-observable monitors reach AUROC 0.957 and 0.935 on the first two benchmarks, yet their locked 5%/10% detection rates are only .642/.742 and .719/.782, respectively. On AgentDojo, the secondary mean-activation rollout monitor reaches AUROC 0.922 but detects none of 38 positive semantic cases at the locked 5% operating point; thresholds intended for 10% false alarms realize 18.7-20.0% on test. Because the test misses its prospectively frozen 40-positive support gate, we label it support-insufficient. On ST-WebAgentBench, activation reaches AUROC .874, but 23 independent calibration negatives cannot identify even a 10% controller; the locked policy abstains rather than reporting its mechanical zero FPR as a success. An exploratory counterexample also lowers full AUROC while improving realized 10% utility. The fail-closed compiler caps MUP and LASR at restricted predictive value and AgentDojo and ST-Web at representation accessibility; no setting reaches alarm-policy validity.
Tags
Links
- Source: https://arxiv.org/abs/2610.04575v1
- Canonical: https://arxiv.org/abs/2610.04575v1
Trouble viewing inline? Open PDF directly ā
Full Text
74,341 characters extracted from source content.
Expand or collapse full text
From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents Conference: The Eighth International Conference on Distributed Artificial Intelligence; November 29āDecember 2, 2026; Hong Kong, ChinaThe Eighth International Conference on Distributed Artificial Intelligence (DAI ā26), November 29āDecember 2, 2026, Hong Kong, ChinaCCS: Computing methodologies Artificial intelligenceCCS: Security and privacy Software and application security Xueping Gao Affiliation: Alibaba Cloud Computing, Hangzhou, China email: xueping.gxp@alibaba-inc.com Ā© , Abstract. Activation probes can predict safety-relevant properties of language models with high area under the receiver-operating-characteristic curve (AUROC), but deployed agent monitors make thresholded alarm decisions under tight false-alarm budgets. These are different estimands. We introduce an Operational Validity Contract that fixes a monitorās target, observability, identity, timing, intervention unit, comparator, calibration, and cost. We formalize risk at the semantic request or trajectory level: when one task contains repeated alarm opportunities, row-level AUROC and false positive rate do not identify semantic-unit any-alarm risk. A confidence-certified threshold also requires enough independent negative units, a tie-safe rule, and transport to deployment. Across Models Under Pressure, LASR refusal prediction, immutable AgentDojo, and a prospectively protocol-frozen ST-WebAgentBench replication, joint activationāobservable monitors reach AUROC 0.957 and 0.935 on the first two benchmarks, yet their locked 5%/10% detection rates are only .642/.742 and .719/.782, respectively. On AgentDojo, the secondary mean-activation rollout monitor reaches AUROC 0.922 but detects none of 38 positive semantic cases at the locked 5% operating point; thresholds intended for 10% false alarms realize 18.7ā20.0% on test. Because the test misses its prospectively frozen 40-positive support gate, we label it support-insufficient. On ST-WebAgentBench, activation reaches AUROC .874, but 23 independent calibration negatives cannot identify even a 10% controller; the locked policy abstains rather than reporting its mechanical zero FPR as a success. An exploratory counterexample also lowers full AUROC while improving realized 10% utility. The fail-closed compiler caps MUP and LASR at restricted predictive value and AgentDojo and ST-Web at representation accessibility; no setting reaches alarm-policy validity. A reproducibility audit recomputes these claim levels and all four settingsā reported score-derived metrics from frozen predictions and identities. Keywords: language-model agents, activation monitoring, operational validity, risk calibration, prompt injection 1. Introduction Internal activations are an appealing source of safety signals for language-model agents. A lightweight probe can reuse a modelās forward pass, operate before a harmful action is completed, and expose features that a small text classifier may require much more data or compute to recover (Patel and Wang, 2024; McKenzie et al., 2025; Kirch et al., 2025). These properties have motivated activation monitors for jailbreaks, deception, refusal, and reward hacking. Their quality is commonly summarized by AUROC, accuracy, or a gain over a lexical baseline. An operator, however, does not deploy an AUROC. The operator deploys a score, an aggregation rule, and a threshold that may stop a trajectory, escalate a request, or invoke a human. False alarms are paid per semantic task or trajectory, not necessarily per dataset row. The threshold must be selected without test labels and must transport across prompts, policies, and tool environments. A monitor can therefore rank examples well while being unusable at the false-alarm rate that the alarm policy can afford. Our AgentDojo experiment makes this gap concrete under a prespecified sampled retry-set stress audit. We monitor the first policy request after a Qwen2.5-7B agent consumes an injected tool result and predict whether each of ten frozen stochastic attempts for the same semantic case executes the official injection goal. A frozen mean-activation monitor reaches rollout AUROC 0.9215. At a threshold locked for a 5% semantic-case safe-attempt alarm budget, it detects none of the 38 cases containing an unsafe attempt. At the 10% budget, the primary activation and joint monitors realize 18.7% and 20.0% safe-attempt alarm rates. Neither fact is visible from rollout AUROC. The ten-attempt unit is a stress-audit choice, not an estimate of a single-execution deployment or a controller that stops at its first alarm. An independently frozen SuiteCRM replication sharpens the category error: activation AUROC is .8738, but the prespecified independent-template support cannot certify a finite threshold, so every 5%/10% controller abstains. We study the resulting question: when does an activation score constitute a valid alarm signal for an agent, rather than merely a successful probe? Our answer is an Operational Validity Contract: Target Ć Observability Ć Identity Ć Timing Ć Unit Ć Comparator Ć Calibration Ć Cost. The fields specify the random variable, information set, generalization boundary, and decision policy to which a result applies. This is an AgentOps problem: a distributed tool-using system pays intervention costs over tasks and trajectories, while its monitor is usually validated over rows. Closing that mismatch is necessary for observable, reproducible, and governable agent infrastructure. Our contribution is an activation-monitor-specific integration of established evaluation and calibration ideas, with three concrete outputs: ā¢ a claim-level contract that separates representation accessibility, restricted predictive value, and calibrated alarm-policy validity by binding the semantic alarm unit, same-horizon comparator, and confidence-locked threshold; ā¢ a four-setting audit showing that high activation AUROC can coexist with poor low-FPR detection, insufficient calibration support, calibration-to-test mismatch, and concentrated group burden; and ā¢ a fail-closed claim compiler, covered by 20 result-independent counterexample tests, and a score-level replay binding identities, timing, frozen predictions, thresholds, and immutable test consumption. The calibration inequality, trajectory-level any-alarm risk, and the distinction between ranking and alarm utility all have prior antecedents. We do not claim any ingredient separately. Proposition 1 applies the standard row-to-unit risk relation to this audit rather than proposing a new concentration inequality. The integration makes explicit which additional evidence is needed to move from a probe score to an alarm-policy claim. We do not argue that activation monitors universally fail. They improve low-FPR detection substantially in two settings. The narrower conclusion is more operationally important: a probe result alone does not identify a valid alarm policy. 1. Specify target, horizon, identity, unit ā¶ 2. Compare same-horizon observable, activation, joint ā¶ 3. Calibrate independent unit maxima, strict threshold ā 5. Claim accessibility, predictive value, or alarm-policy validity āµ 4. Stress transport, groups, intervention cost Abstain if support or identity checks fail Figure 1. The probe-to-policy audit. A high ranking metric is evidence at Step 2, not permission to skip specification, unit-level calibration, or transport. Every empirical claim in this paper is labeled by the last step it passes.Flow diagram from specification through monitor comparison, calibration, transport stress, and a final claim, with an abstention branch when identity or support checks fail. 2. Related Work 2.1. Activation monitoring Activation monitors have been compared with text classifiers at low FPR, applied to high-stakes inputs, and tested under off-policy shift (Patel and Wang, 2024; McKenzie et al., 2025; Kirch et al., 2025). Temporal signals also track refusal and reward-hack risk states (Wilhelm et al., 2026; Collu et al., 2026; Wilhelm and Kao, 2026); monitor staleness after model updates is a separate transport problem (Duan, 2026). These establish accessibility, timing, robustness, or efficiency, whereas we audit the operational unit and calibrated alarm claim. AgentLens, TRACES, and AgentForesight move prediction into multi-turn execution (Luo et al., 2026; Li et al., 2026b; Zhang et al., 2026); AcMAS and PVDetector use activations for multi-agent attacks and policy violations (Xu et al., 2026; Wang et al., 2026b). Yet AUC .998 need not identify malicious content, and a critic with AUROC .94 can reduce success after intervention (Li et al., 2026a; Vasudev et al., 2026). General probing controls raise the same shortcut and comparator concerns (Belinkov, 2022; Wang et al., 2025; Li et al., 2025). We propose neither a new detector nor an intervention-effect estimator: our contribution is the same-horizon observable, semantic alarm unit, threshold-support, and transport audit between offline prediction and an alarm claim. Black-to-white performance boosts likewise evaluate internal access relative to a black-box monitor (Parrack et al., 2025); a finite-model gain in our audit remains restricted predictive value, not evidence of privileged or causal state. 2.2. Risk-controlled online monitoring Conformal risk control calibrates a parameter to bound a monotone expected loss (Angelopoulos et al., 2024). Schirmer et al. apply conformal and high-probability risk control to online LLM verifier signals: one output sequence is flagged if any step crosses a threshold (Schirmer et al., 2026). CP-Agent uses held-out trace calibration and a predeclared finite controller manifest in competitive programming (Wang et al., 2026a). Thus calibrated online monitoring and frozen controller manifests are not new. Score order statistics and minimum sample-size requirements for high-probability type-I-error control also predate LLM monitoring in NeymanāPearson classification (Tong et al., 2018). CORA further calibrates an execute/abstain boundary for GUI-agent actions (Feng et al., 2026). We instead audit whether the score, information set, nested agent identity, and deployment unit justify an activation-monitor claim. System-level evaluation already connects errors to Safety Gain, Residual Hazard, and Availability Cost (Guerin et al., 2022); we add a semantic unit, same-horizon comparison, and finite-sample low-FPR calibration/transport audit, not a new utility functional. Low-FPR monitoring is also central to AI control, where a trusted monitor selects actions or trajectories for audit (Greenblatt et al., 2024). Recent work models the low-FPR shape of monitor ROC curves (Staley et al., 2026). We avoid tail extrapolation and use finite observed order statistics; unsupported operating points cause abstention. Three closest contemporaneous systems sharpen our novelty boundary. Monitoring Monitorability defines intervention, process, and outcome-property archetypes, a monitorability metric, and a broad agent suite (Guan et al., 2026). GAVEL composes interpretable activation concepts into configurable real-time safety rules (Rozenfeld et al., 2026). PrefixGuard synthesizes online prefix monitors across four agent benchmarks and already shows that strong ranking need not yield actionable first alerts at low false-alarm rates (Huang et al., 2026). We do not claim archetype design, operational activation rules, or the rankingāalarm-utility distinction as new. Our narrower contribution binds an activation-versus-observable comparison to a semantic alarm unit, confidence-locked threshold, and frozen transport audit. General assurance and reporting work provides an additional boundary. Evaluation Cards composes benchmark, run, and model metadata into an operational reporting record (Ghosh et al., 2026); structured AI safety cases bind claims, arguments, and evidence (Lee et al., 2026); and the Composable Assurance Framework propagates machine-readable safety assertions through MLOps (Zhao, 2026). The Operational Validity Contract is not a general evaluation schema, safety case, or deployment authorization. It is a narrower statistical contract for deciding the strongest activation-alarm claim licensed by one frozen evaluation. 2.3. Agent safety evaluation AgentDojo evaluates prompt injection against stateful tool-using agents using official utility and security outcomes (Debenedetti et al., 2024). It is suitable because the label is executed future behavior rather than a text proxy. We add stochastic rollouts, a pre-action boundary, semantic-case grouping, and immutable manifests. SLEIGHT-Bench shows that capable transcript monitors retain blind spots at 1% FPR (Najt et al., 2026); we do not claim adaptive robustness. Other benchmarks expose miss/false-alarm trade-offs and task-disjoint TFāIDF beating LLM judges (Yang et al., 2026; Advani, 2026). We ask the logically prior question of whether activations improve a locked alarm policy over such controls. Table 1. Scope relative to the closest evaluation and calibration work. A dash means the item is not the cited workās primary object, not that it is incompatible with the method. Work Score emphasis Alarm unit Same-horizon comparator Confidence-locked risk Transport/group audit PatelāWang (Patel and Wang, 2024) activation example strong text low-FPR evaluation adversarial robustness Guerin et al. (Guerin et al., 2022) runtime monitor system outcome return/hazard model safety/availability utility application assumptions Monitoring Monitorability (Guan et al., 2026) CoT/transcript trajectory/property action-only monitorability metric broad scaling suite GAVEL (Rozenfeld et al., 2026) activation rules rule event text/guardrail baselines threshold metrics domain/rule audit PrefixGuard (Huang et al., 2026) trace monitor trajectory first alert raw text/LLM judges calibrated FAR points four-benchmark diagnostics Schirmer et al. (Schirmer et al., 2026) verifier output sequence multiple verifiers sequence any-alarm exchangeable sequence risk CP-Agent (Wang et al., 2026a) trace feedback program/task controller variants held-out risk control task-specific controller Ours activation + observable semantic task/case declared visible history unit-max Beta UCB one-use + suite/support 3. Operational Validity 3.1. A monitor claim is a contract Definition 1 (contract-bound claim). Let a monitor emit score S, and let =(Y,O,I,H,G,B,Pcal,L)C=(Y,O,I,H,G,B,P_cal,L). A validity claim is indexed by (S,)(S,C), whose eight fields are: (1) Target: an input property, future behavior, completed-output property, or causal state; (2) Observability: the transcript and system information available to a non-activation comparator at the same horizon; (3) Identity: the semantic object that may not cross training, calibration, test, or bootstrap clusters; (4) Timing: when the score is read relative to the action; (5) Unit: the task, request, rollout, or trajectory on which an alarm consumes budget; (6) Comparator: observable features, model class, data, and compute allowed to the baseline; (7) Calibration: the independent population and rule used to lock a threshold; and (8) Cost: latency, memory, review burden, and intervention loss. Because C selects the random variable, comparator, threshold law, and loss, the contract fixes an estimand rather than a reporting checklist. Changing from C to ā²ā C requires an invariance or transport assumption. The contract creates a three-level claim ladder. Representation accessibility needs a development-frozen probe and held-out ranking evidence. Restricted predictive value additionally needs a competitive observable at the same horizon and clustered uncertainty for the incremental gain. Alarm-policy validity additionally needs an operational aggregation rule, an independently locked threshold, realized risk, sufficient positive support, transport diagnostics, and declared cost semantics. Evidence cannot be promoted up this ladder by renaming the target after the experiment. Causal intervention benefit is audited separately and is not a fourth rung inferred from alarms. This distinction is invisible to pooled AUROC. With a random positive row score S+S^+ and negative row score SāS^-, (1) AUROC=Prā”(S+>Sā)+12āPrā”(S+=Sā).AUROC= (S^+>S^-)+ 12 (S^+=S^-). It averages all pairwise rankings, has no deployment threshold, weights units in proportion to their row counts, and assigns no confidence to the extreme negative tail. It can therefore improve while the alarm policy becomes worse, as our LASR counterexample later demonstrates. 3.2. The operational unit Let G denote an independent semantic operational unit. It may contain several rows or stochastic trajectories i. Let YGāi=1Y_Gi=1 denote an unsafe target event and let larger SGāiS_Gi be more suspicious. For a prespecified sampled set of alarm opportunities, define the outcome-conditioned unit maxima (2) ZGā=maxi:YGāi=0SGāi,ZG+=maxi:YGāi=1SGāi.Z_G^-= _i:Y_Gi=0S_Gi, Z_G^+= _i:Y_Gi=1S_Gi. At threshold t, the corresponding operating rates are (3) R0ā(t)=Prā”(ZGā>t),R1ā(t)=Prā”(ZG+>t).R_0(t)= (Z_G^->t), R_1(t)= (Z_G^+>t). Mixed units contribute to both estimands; repeated rows do not become independent calibration samples. We call R0R_0 the safe-attempt any-alarm rate and R1R_1 the unsafe-attempt any-hit rate. They are not the FPR/TPR of a conventional binary āany unsafeā case classifier: a mixed case can contain both a safe attempt that alarms and an unsafe attempt that is detected. The meaning of G is part of the operational contract, not a harmless grouping choice. In AgentDojo our primary G is one semantic security case represented by a sampled set of ten prespecified attempts. This is a retry-set stress audit for systems that may retry a task, not an estimate of an ordinary single-execution deployment or of an online stop controller. We therefore report fixed-order k-attempt and all-single-seed sensitivities separately; they are different estimands and cannot replace the frozen primary result. The class-max estimand is evaluated after labeling all sampled attempts; it does not simulate a sequential controller that halts after its first alarm and thereby suppresses future retries. Such a stop policy requires a new sequential contract and prospective evaluation. Proposition 1 (row risk does not identify unit risk). If a safe unit contains n alarm opportunities with identical row FPR p, conditional independence gives R0=1ā(1āp)nR_0=1-(1-p)^n. Without a dependence assumption, pā¤R0ā¤minā”(1,nāp)pā¤ R_0ā¤ (1,np). Thus equal row score distributions and equal row AUROC/FPR can induce different unit risk. At p=.05p=.05 and n=10n=10, unit FPR is .401.401 under independence. This is an identification error; more row-level test data cannot repair a misspecified unit. Schirmer et al. already aggregate any alarm within one output sequence (Schirmer et al., 2026). Our point concerns the preceding specification: for an agent, several sequences, seeds, or policy requests can belong to the same user task on which review burden is paid. Table 2. A 5% row false-alarm rate compounds rapidly when a safe operational unit contains n conditionally independent alarm opportunities. n opportunities 1 5 10 20 Unit FPR (%) 5.0 22.6 40.1 64.2 3.3. The alarm-policy objective For a false-alarm budget Ī±, deployment asks for a feasible threshold that approximately solves (4) maxtā”R1ā(t)subject toR0ā(t)ā¤Ī±. _t\;R_1(t) to R_0(t)ā¤Ī±. Alternatively, an application can declare benefits and costs and maximize bāR1ā(t)ācāR0ā(t)āCmonitorā(t)bR_1(t)-cR_0(t)-C_monitor(t). Both formulations require a specific threshold and operational unit. Full AUROC integrates over many thresholds the alarm policy will never use and contains neither intervention cost nor a risk constraint. We use the constrained form to express a false-alarm budget; false reviews or blocks motivate this budget, but their application-specific costs are not measured in these experiments. A marginal constraint is not a worst-group constraint. If suite K is observed, a deployment may instead require maxkā”R0,kā(t)ā¤Ī± _kR_0,k(t)ā¤Ī± or allocate different prespecified budgets by suite. We report group diagnostics but do not claim such conditional control, because several AgentDojo suites lack the negative and positive support needed to calibrate and evaluate it independently. 3.4. What can an activation add? Let O contain all information declared observable at the monitoring horizon, and let A=fĪøā(O)A=f_Īø(O) be a frozen deterministic activation. Proposition 2 (information ceiling). Iā”(Y;Aā£O)=0I(Y;A O)=0, and under an unrestricted decision class the Bayes risk using O is no worse than that using A alone. Conditioning on O fixes A, and any rule hā”(A)h(A) can be written hā(fĪøā(O))h(f_Īø(O)). This does not make finite activation probes useless. Held-out probe performance can establish representation accessibility; an incremental held-out gain over the declared observable comparator can establish restricted predictive value. Sample or compute efficiency additionally requires a learning curve or resource-matched measurement. None implies privileged information or hidden intent. If O omits information used to compute A, the claim must state that omission explicitly. 3.5. Certified thresholds, support, and ties Conditional on the frozen score function, suppose m calibration samples of ZGāZ_G^- are independent draws from a common negative-unit distribution, also shared by an independent deployment unit. Sort them descending, and TrT_r is the rrth largest. We alarm iff ZGā>TrZ_G^->T_r. For any score distribution, including discrete or quantized scores, the new-unit strict tail mass qā”(Tr)=Prā”(Znewā>Trā£Tr)q(T_r)= (Z_new^->T_r T_r) is stochastically upper-bounded by Betaā”(r,mār+1)Beta(r,m-r+1); under continuity the bound is exact. Therefore (5) Prcal[q(Tr)ā¤Br,mār+1ā1(1āĪ“)]ā„1āĪ“. _cal\! [q(T_r)ā¤ B^-1_r,m-r+1(1-Ī“) ]ā„ 1-Ī“. We lock the largest rank r whose upper quantile is at most the false-alarm budget Ī±. If no rank qualifies, the alarm policy abstains. At rank one, a finite threshold requires (6) mā„logā”Ī“logā”(1āĪ±).mā„ Ī“ (1-Ī±). For Ī±=Ī“=.05Ī±=Ī“=.05, this is 59 independent negative units merely to identify a finite threshold; it says nothing about TPR. Selected-value ties make the bound conservative rather than invalid under strict exceedance. The positive-test minima used below are frozen design-sufficiency gates, not consequences of this negative-tail inequality. The artifact records selected-value multiplicity, calibration strict exceedances, and test equalities; all reported test operating points have zero equality. The pre-test AgentDojo threshold JSON and earlier replay recorded the legacy non-strict comparison. The submission audit changes only that comparator to strict exceedance: numeric thresholds, ranks, and every test decision remain unchanged. We expose this post-hoc semantics correction rather than treating sample uniqueness as proof of population continuity. This analysis separates three bottlenecks. Tail identifiability asks whether calibration support can certify a threshold. Tail separability asks whether any valid threshold has useful R1R_1. Transport asks whether the calibration and deployment unit distributions match. More calibration data can improve the first, but not guarantee the other two. The distinction also clarifies what a confidence statement means. The order-statistic bound controls the marginal probability that a new independent negative unit from that same distribution strictly exceeds the selected score. Mere exchangeability without independence is insufficient for this high-probability population-risk bound. It is not a simultaneous guarantee over suites, families, policies, or an adaptive sequence of redeployments. Those require additional multiplicity control or a new calibration contract. 4. Evaluation Protocol 4.1. Four targets The four settings are case studies of a common audit procedure, not a controlled cross-benchmark comparison or a sample from a single deployment population. Each asks which claim its own target, unit, comparator, and calibration evidence support. We report within-setting comparisons without pooling effect sizes or estimating how often activation monitors succeed or fail in deployment. Table 3 records the claim before any operating-point comparison. MUP contains independent high-stakes inputs and tests an observable input property (McKenzie et al., 2025). LASR predicts whether Llama-3.2-3B will refuse before any assistant token, using recovered original requests as units (Kirch et al., 2025). These analyses use frozen splits but are retrospective because earlier benchmark results had been inspected. The released LASR row split is not a pure semantic OOD split: 207 of 969 test request identities occur in the official training set, including 52 that occur in our frozen calibration subset. We therefore label official-all results as a published-split description, not independent transport evidence, and report the protocol-declared semantic-unseen subset (762 requests) as the identity- disjoint sensitivity. The subset is defined only by request identity, without using labels or monitor scores. AgentDojo is the prospective future-behavior test. We freeze AgentDojo v1, Qwen2.5-7B-Instruct, temperature .7, top-p .9, three then ten seeds for model fitting, and disjoint user-task splits. The monitoring boundary is the first policy request after an injected tool result. The label is whether the remaining trajectory executes the official injection task. A global constant marker replaces an early identity-bearing marker before any monitor feature is collected. One immutable 1,190-identity test manifest, containing ten prespecified stochastic attempts per semantic case, is consumed once; failed trajectories are retained without regeneration. This is not the same estimand as observing one trajectory per case. ST-WebAgentBench is a separately protocol-frozen prospective replication on SuiteCRM (Levy et al., 2026). Its unit is an intent template with safe and unsafe trajectory siblings; the monitor reads Qwen2.5-7B layer 14 before each browser action. Of 192 immutable test identities, 173 valid rollouts yield 2,237 pre-action rows. Twelve engineering failures are retained without replacement. The exhaustive ledger assigns all result-bearing input, future-response, tool-agent, and browser-agent branches to this table; it is not a random task sample, and rejected, failed, and negative branches remain in the supplement. Table 3. Frozen claim cards. āPreā means before the predicted behavior. Setting Target Timing Unit MUP high-stakes input input input LASR future refusal pre-response semantic request AgentDojo injection execution pre-action security case ST-Web task failure pre-action intent template For AgentDojo, the eight fields instantiate as: executed injection goal (Y); exact visible history through the injected tool result (O); user task and frozen model/policy identity (I); first subsequent policy request (H); the sampled ten-attempt security case (G); TFāIDF, tool/suite metadata, and three Qwen-14B forecasts (B); disjoint negative-case maxima with the frozen strict order statistic (PcalP_cal); and safe-attempt alarm burden versus unsafe-attempt hits (L). Downstream intervention cost is not measured, so this audits risk proxies rather than net deployment utility. 4.2. Monitor ladder Each setting compares observable, activation, and joint monitors. MUP uses a frozen lexical observable and a validation-selected probe. LASRās observable combines lexical features with three Qwen-14B semantic forecasts available from the same prompt. Its activation is the corrected layer-12 cumulative mean. AgentDojoās observable combines canonical visible-history character/word TFāIDF, suite/tool metadata, and three frozen Qwen-14B future-outcome log-odds prompts. The primary activation is Qwen2.5-7B zero-indexed decoder layer 14 at the final token of the exact policy request; full-request mean is a prespecified secondary analysis. Late fusion uses out-of-fold base predictions with grouped folds. No AgentDojo layer, pooling, template, feature, solver, or threshold is chosen from calibration or test performance. ST-Web reuses the frozen layer/pooling and strong-observer construction, with intent-template-grouped fitting and no outcome-dependent revision. The declared observer is intentionally capable where public artifacts permit: does activation add predictive utility beyond what the same visible history can support? The settings nevertheless use heterogeneous observer classes (lexical for MUP; Qwen-14B-assisted for LASR and AgentDojo), so we make within-setting increments, not cross-setting comparator rankings. Compute is reported separately rather than assumed equal. 4.3. Calibration and integrity Model fitting and threshold calibration use disjoint semantic units. We lock 95%-confidence marginal thresholds for Ī±ā.05,.10Ī±ā\.05,.10\ from negative unit maxima. Results report realized FPR/TPR, paired 5,000-sample semantic-unit bootstrap intervals, and suite/family diagnostics. Each budget binds its own minimum TPR; all four contracts froze .80 before operating-point analyses. It is not a universal optimum or a hypothesis-test threshold; the scientific conclusions use the reported rates, intervals, support, and transport audits rather than this binary rule. Coverage additionally requires independent calibration and deployment units from the same negative-unit distribution. Because the official LASR test contains the disclosed training identity overlap, we attach no transport certificate to that all-test panel; the semantic-unseen panel is the corresponding identity-disjoint check. MUP, AgentDojo, and ST-Web use their declared independent test units. For a mixed case, safe and unsafe stochastic outcomes enter ZGāZ_G^- and ZG+Z_G^+ respectively: the same user task can create both a safe-attempt alarm and an unsafe-attempt detection opportunity. Paired differences reuse each resampled semantic case across monitors; percentile intervals are computed from 5,000 case-cluster bootstrap replicates. The calibration UCB and the realized test FPR answer different questions, so neither is substituted for the other. We treat realized FPR as a transport diagnostic, not proof of exchangeability; a point estimate below Ī± cannot itself certify deployment shift. MUP intervals resample medical specialties and LASR intervals resample semantic requests; AgentDojo intervals resample semantic security cases, and ST-Web intervals resample intent templates. Across monitor, budget, and sensitivity panels we report pointwise intervals without familywise multiplicity control. We do not select a significant cell or reinterpret these intervals as simultaneous guarantees. Frozen alarm-policy audit procedure (1) Declare target, visible history, semantic identity, horizon, unit, score families, and promotion rule. (2) Split by semantic identity; fit Observable and Activation only on model fit, and fit Joint from grouped out-of-fold base predictions. (3) Freeze model and feature hashes. On calibration only, form negative unit maxima and select the largest Beta-certified rank. (4) Apply strict exceedance, audit support, selected-value multiplicity, and test equalities; abstain only when no rank has the required upper bound. (5) Hash the thresholds, consume test once, retain failures, and report unit FPR/TPR, paired monitor differences, and group support. Figure 2. No layer, pooling rule, template, model, or threshold is changed after its corresponding freeze point.Five-step alarm-policy audit: declare the estimand, fit grouped monitor models, freeze hashes, calibrate unit maxima with tie checks, then consume test once and report operational and group metrics. The claim compiler applies the claim ladder in Section 3 to frozen evidence; it never fits or searches. Appendix F specifies its decision rule. Twenty result-independent counterexamples cover malformed identities, budgets, ranks, ties, support, mechanical zeros, and promotion. A separate replay reconstructs all four settingsā score metrics from allowlisted frozen archives; upstream audits separately verify prompt/token and activation alignment, hashes, and immutable test consumption. Table 4. Artifact checks that prevent a ranking result from silently changing its estimand. Boundary Immutable check Identity semantic hashes; grouped folds; overlap/non-overlap audit Timing exact visible prefix; tool-result boundary reconstruction Features layer, pooling, templates, payload and model hashes Calibration negative-unit count, selected rank, Beta UCB, strict rule/ties Test one-use manifest, failed-rollout retention, no refitting 4.4. Resource accounting Predictive dominance and deployment preference are different comparisons. In a separate hook microbenchmark on the AgentDojo Qwen-7B stack, capturing the frozen layer added no resolvable batch-one latency and 0.067% at batch eight; the captured tensor occupied 4ā41 MiB across tested sequence lengths, against an approximately 26 ms batch-one forward pass. These measurements cover activation extraction, not the downstream probe or the Qwen-14B observer. We therefore report the latter as a strong information-set comparator, not as a compute-matched replacement. A cheap activation can remain useful in a cascade even when it has no independent predictive increment. Human-review and action- blocking costs are unmeasured, so we compare risk proxies and do not claim net deployment utility. 5. Results We organize evidence by the three gates induced by the contract rather than by benchmark: whether a low-FPR threshold is identifiable, whether the certified tail is separable, and whether calibration transports. Retry and suite analyses then test whether the declared unit hides policy or group heterogeneity. 5.1. Calibration support and threshold identifiability The first three settings contain enough independent negative calibration units to select finite thresholds at both budgets. Table 5 reports the frozen rank and its 95% upper confidence bound (UCB). This rules out the simplest explanationāthat poor utility is only an undefined percentile from too few negative rows. It does not establish positive-class separation or transport. AgentDojo illustrates the distinction: its 10% activation threshold has an 8.92% calibration UCB but realizes 18.67% test FPR. Under the continuous independent-and-identically-distributed null, integrating over the frozen rank-5 threshold gives Kā¼BetaBinomialā”(75,5,96)K (75,5,96) and Prā”(Kā„14)=.001745 (Kā„ 14)=.001745; strict exceedance is conservative when scores tie. This predictive check includes calibration-threshold uncertainty and diagnoses a calibration-to-test distribution mismatch, not an intervention effect. ST-Web exposes the logically prior failure. Its 23 calibration-negative intent templates are below the rank-one requirements of 59 at 5% and 29 at 10%. All frozen thresholds are therefore +ā+ā. The resulting zero alarms are mechanical abstention, not successful transport. Table 5. Independent negative-unit support and realized primary-activation FPR. Selected ranks depend on m, Ī±, and Ī“=.05Ī“=.05, not test performance. m 5% budget 10% budget Setting r/UCB Test FPR r/UCB Test FPR MUP 69 1/.0425 .0099 3/.0884 .0199 LASR 372 12/.0485 .0183 28/.0987 .0872 AgentDojo 100 2/.0466 .0000 5/.0892 .1867 ST-Web 23 abstain abstain 5.2. High AUROC does not ensure low-FPR detection Table 6 reports the joint monitor, which is the most favorable summary for an activation increment on MUP and LASR. MUP reaches AUROC .9571, yet its realized 5% and 10% TPRs at FPR-certified thresholds are .6424 and .7417. LASR reaches request-event AUROC .9348; the corresponding realized TPRs are .7189 and .7821. Both improve over observable baselines, so the result is not āactivations fail.ā Neither meets the frozen requirement of at least .80 TPR at both budgets. The conclusion persists on 762 identity-disjoint, semantic-unseen LASR requests. AUROCs are .9066 (Observable), .9424 (Activation), and .9427 (Joint). Their 5% FPR/TPR pairs are .0334/.6234, .0180/.7065, and .0206/.7065; at 10% they are .1054/.7403, .0643/.7662, and .0720/.7740. Joint-minus-Observable TPR is +.0831, request-bootstrap 95% CI [.0494,.1169][.0494,.1169], at 5%, and +.0338, CI [.0052,.0623][.0052,.0623], at 10%. Thus overlap does not create the observed increment, but the unseen alarm policy still misses the frozen .80 utility gate. Paired cluster uncertainty also distinguishes the two budgets. MUP Joint-minus-Observable TPR is +.2649, specialty-bootstrap 95% CI [.1529,.4268][.1529,.4268], at 5%, but +.0762, CI [ā.0192,.2209][-.0192,.2209], at 10%. LASRās request-bootstrap increments are +.1059, CI [.0754,.1385][.0754,.1385], and +.0407, CI [.0122,.0692][.0122,.0692], respectively. Thus activation can materially improve a restricted comparator while the locked alarm policy still misses its utility requirement. Table 6. Locked operational-unit results. AUROC is test unit-level AUROC except AgentDojo, where both primary case-max and the highest prespecified rollout AUROC are shown. FPR/TPR are realized at independently calibrated thresholds. LASR rows show the official all-test split; the identity-disjoint sensitivity is reported in text and Appendix B, Table 7. ST-Web dashes denote prespecified abstention, not observed zeros. ADJ = AgentDojo. Setting Monitor AUROC 5% FPR 5% TPR 10% FPR 10% TPR MUP Observable .9396 .0033 .3775 .0497 .6656 Activation .9446 .0099 .5000 .0199 .6457 Joint .9571 .0066 .6424 .0232 .7417 LASR Observable .8987 .0467 .6130 .1176 .7413 Activation .9344 .0183 .7189 .0872 .7780 Joint .9348 .0243 .7189 .0913 .7821 ADJ Observable .7860 .0267 .0526 .2000 .5789 Activation last .8028 .0000 .0526 .1867 .5263 Joint last .7986 .0000 .0263 .2000 .5789 ADJ rollout Activation mean .9215 .0000 .0000 .0000 .1053 ST-Web Observable .8497 ā ā ā ā Activation last .8738 ā ā ā ā Joint last .8712 ā ā ā ā Appendix Figure 3 visualizes the AUROCāoperating-point mismatch without pairing row-level discrimination with unit-level TPR. The LASR nominal, non-certified 10% activation threshold reaches .8126 TPR, but realizes .1136 FPR. At 5%, even its nominal TPR is .7475. Thus LASRās 10% near miss is partly calibration-resolution limited, whereas its 5% failure is primarily tail separation. A fixed-data allocation sweep later increased calibration support without rescuing the operating point, illustrating that support and separation cannot be substituted mechanically. 5.3. AgentDojo calibration-to-test mismatch No model selection on test. Before calibration or test behavior was generated, grouped model-fit out-of-fold AUROC was .9020 for Observable, .8918 for Activation last, and .8879 for Joint last. The Joint-minus-Observable difference was ā.0141-.0141 with paired-case 95% CI [ā.0338,.0041][-.0338,.0041]. On mixed cases it was ā.0469-.0469 with CI [ā.0990,ā.0039][-.0990,-.0039]. These diagnostics argued against an activation increment. We nevertheless locked the declared pipelines and consumed the test manifest once. This negative pre-test evidence is important: the reported test is a falsification attempt, not a selected success from layers or pooling rules. The AgentDojo test contains 1,190 frozen rollout identities, of which 1,165 complete. There are 602 eligible monitoring-boundary rows across 82 semantic cases, with 75 negative cases, 38 positive cases, and 31 mixed cases. Trace alignment and exact prompt reconstruction are 100%. The prospectively frozen minimum was 40 positive cases, so the formal status is support-insufficient; this was a frozen design-sufficiency rule distinct from the analytic negative-tail certificate, and we do not add or rerun cases. The primary increment and 10% transport conclusions below fail independently of this two-case shortfall. The primary case maximum corresponds to the sampled ten-attempt retry-set audit, not an intrinsic property of āAgentDojo.ā Prespecified fixed-order k=1,3,5,10k=1,3,5,10 and single-seed sensitivities (Appendix D) vary sharply; only four single-seed policies identify a finite 5% threshold. They do not replace the primary estimand, but show why retry semantics must be declared in advance. At 5%, Observable detects two positive cases with two false alarms. Activation last detects the same number with no false alarms, while Joint last detects one with none. The primary Joint-minus-Observable TPR difference is ā.0263-.0263, paired semantic-case bootstrap 95% CI [ā.0789,0][-.0789,0]. It fails the promotion rule before considering the support shortfall. At 10%, Observable and Joint make exactly the same positive-case decisions, but both realize .20 FPR. The activation-only FPR is .1867 and TPR .5263. Pooling reinforces the category error. Mean activation has the best rollout AUROC in Table 6, yet zero 5% TPR and only .1053 TPR at 10%. Because mean pooling was frozen as secondary, this is a sensitivity rather than a post-hoc model switch. 5.4. The declared unit exposes suite concentration AgentDojo alarms concentrate by suite: at 5%, all true positives and both Observable false positives occur in Slack; its 14.29% Slack FPR contrasts with 2.67% pooled FPR. Sparse suite support precludes certificates but rejects interpreting a marginal rate as uniform protection. The failure modes are therefore not interchangeable. MUP has modest positive tail overlap despite conservative realized FPR. LASR has adequate support and passes the 10% target only under the exploratory tail-aligned objective, but its 5% tail remains poorly separated. AgentDojo has formal negative-tail support but exhibits a calibration-to-test mismatch at 10%, has negligible 5% positive recall, and exhibits severe suite concentration. ST-Web is earlier still: its activation AUROC is .8738 and Joint-minus-Observable AUROC is +.0215, template-bootstrap 95% CI [ā.0097,.0568][-.0097,.0568], but calibration support makes both controllers unidentifiable; its 28 negative test templates also miss the prospectively frozen 29. Reporting only ādid not reach 80% TPRā would erase the scientific distinction among identifiability, separation, and transport. The compiler therefore caps MUP and LASR at restricted predictive value: their comparator increments are supported, but their retrospective contracts did not prespecify test-support gates. AgentDojo and ST-Web stop at representation accessibility because their frozen comparator increments fail; support and transport failures independently preclude alarm-policy validity. No current setting licenses an intervention-effect claim. 5.5. Objective mismatch: AUROC and utility can move oppositely An exploratory LASR unit-max partial-AUC objective over FPR [0,.10][0,.10] lowers AUROC .9344ā.9184 but raises certified 10% TPR .7780ā.8106 while FPR falls .0872ā.0669. This constructively reverses full-AUROC selection. Partial-AUC is prior art (Narasimhan and Agarwal, 2013), test results were already inspected, and a separately frozen 5% variant fails (.7189ā.5743 TPR); this is a diagnostic, not a promoted optimizer. 6. Discussion 6.1. Implications for AgentOps Monitor registries should bind every score to the eight contract fields and reject identity leakage, row/unit mismatch, unsupported ranks, ties, or drift. Dashboards should report unit-level rates and abstain on unidentified policies. Activation access remains useful under this standard. On MUP and LASR it materially improves low-FPR recall over the declared restricted observable models. Cheap probes may form a cascade, but without a learning curve or resource match, predictive increments do not establish privileged information, efficiency, or deployment validity. AgentDojo also shows that suite or policy shift can violate exchangeability even with immutable code and thresholds. 6.2. What the evidence does and does not establish The cross-setting pattern supports accessibility and, retrospectively on MUP/LASR, improved low-FPR recall over restricted observables. LASR additionally shows, only exploratorily, that an operational objective can beat full-AUROC selection. Neither result implies privileged information, causality, or transport. Alarm-policy evaluation therefore requires a declared unit and horizon, a capable same-horizon comparator, independent-unit calibration, and distinct support, separation, and transport tests. Each can change the conclusion. 6.3. Limitations AgentDojo and ST-Web are independently frozen prospective evaluations; both retain failures and miss a prespecified support gate. MUP and LASR are retrospective. Both prospective tests study one Qwen-7B layer and frozen pooling; AgentDojo covers one attack and four sparse suites, while ST-Web is SuiteCRM-only. Certificates are marginal, not adversarial or familywise. The Qwen-14B comparators are not compute matched, so their contrasts establish only predictive differences under the declared information sets, not relative cost-effectiveness. We do not measure human-review burden, blocking cost, intervention benefit, or net deployment utility. The contract records these missing quantities explicitly; an audit verdict is not a complete deployment assessment. AgentDojoās sampled ten-attempt unit is policy-specific; post-hoc sensitivities cannot choose deployment semantics. A single-execution or stop-on-alarm policy would require its own calibration and prospective evaluation. The exhaustive screening ledger is not a random task sample, so the audit does not estimate failure prevalence. The four heterogeneous targets form a validity audit, not a pooled effect. In particular, the prospective studies demonstrate rejection of unsupported alarm claims rather than validate a successful alarm policy; their limited support cannot establish a general failure of activation monitoring. Qwen-14B forecasts are automated comparators, not human judgments. Adaptive attacks and calibration drift remain open. The artifact verifies frozen scores, identities, contracts, and paper numbers; it does not regenerate behavior/features or refit, so reproducibility is score-level rather than end-to-end. 6.4. Ethical and societal considerations The public experiments use model-generated trajectories. Invalid thresholds can concentrate burden; unsupported policies should abstain and use group reporting, least privilege, and escalation. Logs require access control and minimal retention; we release no new exploit payloads. 7. Conclusion A probe asks what is accessible; an alarm policy asks whether a locked decision meets operational risk and utility requirements. Across four targets, semantic units, same-horizon observables, calibration, support, and transport change the answer suggested by AUROC. The contribution is their fail-closed, activation-alarm-specific bindingānot a new detector or calibration theorem. Deployment validity is a separate claim. Generative AI Disclosure Generative-AI tools assisted with language editing and draft code generation. The author independently checked the research design, source literature, mathematical claims, code, artifact hashes, and every reported numerical result. References Advani (2026) Laksh Advani. 2026. From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents. arXiv preprint arXiv:2606.09863 (2026). https://arxiv.org/abs/2606.09863 Angelopoulos et al. (2024) Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2024. Conformal Risk Control. In International Conference on Learning Representations. https://arxiv.org/abs/2208.02814 Belinkov (2022) Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics 48, 1 (2022), 207ā219. doi:10.1162/coli_a_00422 Collu et al. (2026) Matteo Gioele Collu, Riccardo Conte, Alberto Giaretta, Denis Kleyko, Mauro Conti, Matteo Zavatteri, and Roberto Confalonieri. 2026. Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations. arXiv preprint arXiv:2605.28553 (2026). https://arxiv.org/abs/2605.28553 Debenedetti et al. (2024) Edoardo Debenedetti, Jie Zhang, Mislav BalunoviÄ, Luca Beurer-Kellner, Marc Fischer, and Florian TramĆØr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., Red Hook, NY, USA, 82895ā82920. Datasets and Benchmarks Track. doi:10.52202/079017-2636 Duan (2026) Evan Duan. 2026. Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness. arXiv preprint arXiv:2606.15980 (2026). https://arxiv.org/abs/2606.15980 Feng et al. (2026) Yushi Feng, Junye Du, Qifan Wang, Zizhan Ma, Qian Niu, Yutaka Matsuo, Long Feng, and Lequan Yu. 2026. CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation. arXiv preprint arXiv:2604.09155 (2026). https://arxiv.org/abs/2604.09155 Ghosh et al. (2026) Avijit Ghosh, Anka Reuel, Jenny Chim, et al. 2026. Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting. arXiv preprint arXiv:2606.09809 (2026). https://arxiv.org/abs/2606.09809 Greenblatt et al. (2024) Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. 2024. AI Control: Improving Safety Despite Intentional Subversion. arXiv preprint arXiv:2312.06942 (2024). https://arxiv.org/abs/2312.06942 Guan et al. (2026) Melody Y. Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y. Wei, Marcus Williams, Benjamin Arnav, Joost Huizinga, Ian D. Kivlichan, Amelia Glaese, Jakub Pachocki, and Bowen Baker. 2026. Monitoring Monitorability. In International Conference on Machine Learning. Spotlight. https://openreview.net/forum?id=b82fgbMVpz Guerin et al. (2022) Joris Guerin, Raul Sena Ferreira, Kevin Delmas, and JĆ©rĆ©mie Guiochet. 2022. Unifying Evaluation of Machine Learning Safety Monitors. arXiv preprint arXiv:2208.14660 (2022). https://arxiv.org/abs/2208.14660 Huang et al. (2026) Xinmiao Huang, Jinwei Hu, Rajarshi Roy, Changshun Wu, Yi Dong, and Xiaowei Huang. 2026. PrefixGuard: From Large Language Model Agent Traces to Online Failure-Warning Monitors. arXiv preprint arXiv:2605.06455 (2026). https://arxiv.org/abs/2605.06455 Kirch et al. (2025) Nathalie Kirch, Samuel Dower, Adrians Skapars, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. 2025. The Impact of Off-Policy Training Data on Probe Generalisation. arXiv preprint arXiv:2511.17408 (2025). https://arxiv.org/abs/2511.17408 Lee et al. (2026) Sung Une Lee, Liming Zhu, Md Shamsujjoha, Liming Dong, Qinghua Lu, Jieshan Chen, and Lionel Briand. 2026. A Structured Approach to Safety Case Construction for AI Systems. arXiv preprint arXiv:2601.22773 (2026). https://arxiv.org/abs/2601.22773 Levy et al. (2026) Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, Nir Mashkif, and Segev Shlomov. 2026. ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=MuCDzH0ctf Li et al. (2026b) Jiaqian Li, Yanshu Li, Boxuan Zhang, Ruixiang Tang, and Kuan-Hao Huang. 2026b. TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling. arXiv preprint arXiv:2605.27690 (2026). https://arxiv.org/abs/2605.27690 Li et al. (2025) Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, and Byron C. Wallace. 2025. Do Activation Verbalization Methods Convey Privileged Information? arXiv preprint arXiv:2509.13316 (2025). https://arxiv.org/abs/2509.13316 Li et al. (2026a) Yanhang Li, Zhichao Fan, and Zexin Zhuang. 2026a. When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents. In The 2nd Workshop on Evaluation for Multimodal Generation. Co-located with SIGIR 2026. https://arxiv.org/abs/2606.22864 Luo et al. (2026) Weidi Luo, Qiming Zhang, Yihao Quan, Mingyu Jin, Jie Cai, Chaowei Xiao, Jingcheng Niu, and Zhen Xiang. 2026. AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agent. arXiv preprint arXiv:2606.22673 (2026). https://arxiv.org/abs/2606.22673 McKenzie et al. (2025) Alex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. 2025. Detecting High-Stakes Interactions with Activation Probes. In Advances in Neural Information Processing Systems, Vol. 38. Curran Associates, Inc., Red Hook, NY, USA. https://proceedings.neurips.c/paper_files/paper/2025/hash/b9501301a7a0f99e57406c901f4e3ce-Abstract-Conference.html Najt et al. (2026) Elle Najt, Colin Toft, Tyler Tracy, Fabien Roger, and Joe Benton. 2026. SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors. arXiv preprint arXiv:2605.16626 (2026). https://arxiv.org/abs/2605.16626 Narasimhan and Agarwal (2013) Harikrishna Narasimhan and Shivani Agarwal. 2013. A Structural SVM Based Approach for Optimizing Partial AUC. In Proceedings of the 30th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 28). PMLR, 516ā524. https://proceedings.mlr.press/v28/narasimhan13.html Parrack et al. (2025) Aviel Parrack, Carlo Leonardo Attubato, and Stefan Heimersheim. 2025. Benchmarking Deception Probes via Black-to-White Performance Boosts. arXiv preprint arXiv:2507.12691 (2025). https://arxiv.org/abs/2507.12691 Patel and Wang (2024) Oam Patel and Rowan Wang. 2024. Activation Monitoring: Advantages of Using Internal Representations for LLM Oversight. In NeurIPS Workshop on Attributing Model Behavior at Scale. https://openreview.net/forum?id=qbvtwhQcH5 Rozenfeld et al. (2026) Shir Rozenfeld, Rahul Pankajakshan, Itay Zloczower, Eyal Lenga, Gilad Gressel, and Yisroel Mirsky. 2026. GAVEL: Towards Rule-Based Safety through Activation Monitoring. In International Conference on Learning Representations. https://openreview.net/forum?id=duntROHZ5R Schirmer et al. (2026) Mona Schirmer, Metod Jazbec, Alexander Timans, Christian Naesseth, Maja Waldron, and Eric Nalisnick. 2026. Online Safety Monitoring for LLMs. arXiv preprint arXiv:2607.02510 (2026). ICML Workshop on Hypothesis Testing. https://arxiv.org/abs/2607.02510 Staley et al. (2026) Kerrick Staley, Aryan Bhatt, and Julian Stastny. 2026. Logit ROCs: Monitor TPR is Linear in FPR in Logit Space. Redwood Research. https://blog.redwoodresearch.org/p/logit-rocs-monitor-tpr-is-linear Tong et al. (2018) Xin Tong, Yang Feng, and Jingyi Jessica Li. 2018. NeymanāPearson Classification Algorithms and NP Receiver Operating Characteristics. Science Advances 4, 2 (2018), eaao1659. doi:10.1126/sciadv.aao1659 Vasudev et al. (2026) Rakshith Vasudev, Melisa Russak, Dan Bikel, and Waseem Alshikh. 2026. The Intervention Paradox: Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention. arXiv preprint arXiv:2602.03338 (2026). https://arxiv.org/abs/2602.03338 Wang et al. (2025) Cheng Wang, Zeming Wei, Qin Liu, Wenxuan Zhou, and Muhao Chen. 2025. False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize. In NeurIPS Workshop on Mechanistic Interpretability. Spotlight. https://openreview.net/forum?id=zwYzSma4V4 Wang et al. (2026b) Junhui Wang, Hangtao Zhang, Zhirun Zheng, Li Zeng, Jiejun Xiao, Xi Luo, Lihua Yin, and Saiqin Long. 2026b. PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis. Proceedings of the 34th ACM International Conference on Multimedia (2026). Accepted to ACM M 2026. https://arxiv.org/abs/2607.12624 Wang et al. (2026a) Peisong Wang, Bowen Liu, Zehua Li, Yuyao Wang, Zhiwei Ma, Yuhan Li, and Jia Li. 2026a. CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming. arXiv preprint arXiv:2605.24693 (2026). https://arxiv.org/abs/2605.24693 Wilhelm and Kao (2026) Patrick Wilhelm and Odej Kao. 2026. From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents. arXiv preprint arXiv:2606.06223 (2026). https://arxiv.org/abs/2606.06223 Wilhelm et al. (2026) Patrick Wilhelm, Thorsten Wittkopp, and Odej Kao. 2026. Monitoring Emergent Reward Hacking During Generation via Internal Activations. arXiv preprint arXiv:2603.04069 (2026). https://arxiv.org/abs/2603.04069 Xu et al. (2026) Haowen Xu, Xue Tan, Lei Ma, Zhihao Zhang, Chao Wang, Qingze Wang, Ping Chen, Jun Dai, and Xiaoyan Sun. 2026. When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems. Proceedings of the 43rd International Conference on Machine Learning (2026). Accepted to ICML 2026. https://arxiv.org/abs/2607.06807 Yang et al. (2026) Shu Yang, Jingyu Hu, Tong Li, Hanqi Yan, Wenxuan Wang, and Di Wang. 2026. AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitors. arXiv preprint arXiv:2601.05752 (2026). https://arxiv.org/abs/2601.05752 Zhang et al. (2026) Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, and Ruixiang Tang. 2026. AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems. arXiv preprint arXiv:2605.08715 (2026). https://arxiv.org/abs/2605.08715 Zhao (2026) Xiaofei Zhao. 2026. Composable Assurance for AI Alignment: A Framework for Propagating Formal Safety Properties Through MLOps. Proceedings of the AAAI Conference on Artificial Intelligence 40, 44 (2026), 38129ā38136. doi:10.1609/aaai.v40i44.41151 Scatter plot comparing operational-unit AUROC and realized TPR at 5 and 10 percent false-alarm budgets for observable, activation, and joint monitors on MUP, LASR, and AgentDojo. ST-Web is absent because its frozen policy abstains. Figure 3. Operational-unit AUROC versus TPR at each independently locked threshold. Marker shape identifies the setting and color the monitor. The dashed .80 line is a frozen project promotion rule, not a universal utility threshold; all realized rates are reported. The secondary AgentDojo rollout AUROC is excluded because pairing rollout AUROC with case-level TPR would mix estimands; ST-Web is excluded because no finite controller was identifiable.Scatter plot comparing operational-unit AUROC and realized TPR at 5 and 10 percent false-alarm budgets for observable, activation, and joint monitors on MUP, LASR, and AgentDojo. ST-Web is absent because its frozen policy abstains. Appendix A Proof Sketches and Calibration Details A.1. Operational-unit compounding Let EiE_i be the event that the iith opportunity in a safe unit alarms, with Prā”(Ei)=p (E_i)=p. If the events are conditionally independent, (7) Prā”(āi=1nEi)=1āPrā”(āi=1nEic)=1ā(1āp)n. ( _i=1^nE_i )=1- ( _i=1^nE_i^c )=1-(1-p)^n. For arbitrary dependence, the union contains each EiE_i, giving the lower bound p, while Booleās inequality gives the upper bound nāpnp, clipped at one. Both bounds are attainable: identical alarm events attain p, and disjoint events attain nāpnp when nāpā¤1npā¤ 1. Thus the row marginal alone cannot identify unit risk. A.2. Deterministic-activation ceiling For a frozen model, A=fĪøā(O)A=f_Īø(O) is measurable with respect to O. Conditioning on O makes A fixed, so Y and A are conditionally independent, and therefore (8) Iā”(Y;Aā£O)=0.I(Y;A O)=0. Any activation-only decision hā”(A)h(A) is the observable decision hāfĪøā(O)h f_Īø(O). This proof compares information sets and unrestricted decision classes; it does not imply that a finite text model can learn fĪøf_Īø with the same samples or compute. A.3. Distribution-free order statistic and ties Let FāF_- be any negative-unit CDF and let TrT_r be the rrth largest of m independent, identically distributed calibration maxima, conditional on the frozen score function. The deployment unit is an independent draw from the same distribution. The probability-integral transform Fāā(Zā)F_-(Z^-) stochastically dominates a uniform random variable, with equality when FāF_- is continuous. Consequently the strict tail qā”(Tr)=1āFāā(Tr)=Prā”(Znewā>Trā£Tr)q(T_r)=1-F_-(T_r)= (Z^-_new>T_r T_r) is stochastically dominated by Betaā”(r,mār+1)Beta(r,m-r+1). Thus, for ur=Br,mār+1ā1ā(.95)u_r=B^-1_r,m-r+1(.95), (9) Prcalq(Tr)ā¤urā„.95, _cal\q(T_r)ā¤ u_r\ā„.95, with equality under continuity. We select the largest r for which urā¤Ī±u_rā¤Ī± and alarm only when S>TrS>T_r. For the four calibration sets, the resulting UCBs are: Setting m 5% rank/UCB 10% rank/UCB MUP 69 1/.042487 3/.088444 LASR 372 12/.048471 28/.098704 AgentDojo 100 2/.046560 5/.089196 ST-Web 23 abstain/ā abstain/ā This strict-exceedance construction remains valid at atoms: a tie makes the bound conservative because equality does not alarm. The audit records the selected-value multiplicity, the number of calibration values strictly above it, and the number of test scores exactly equal to it. All selected primary values have multiplicity one and every reported test operating point has zero equality, so replacing the earlier non-strict implementation changes no paper count. If no rank has urā¤Ī±u_rā¤Ī±, the alarm policy abstains via an infinite threshold. Appendix B LASR Semantic-Identity Sensitivity The official LASR test contains 969 semantic request identities. Of these, 207 occur in the official training set and 52 occur in the frozen threshold- calibration subset. The protocol-declared semantic-unseen mask retains 785 rows covering 762 identities; it depends only on request identity. Table 7 applies the original locked thresholds without refitting or recalibration. Table 7. LASR results on 762 semantic-unseen requests. Operating rates use the original strict-exceedance thresholds. Monitor AUROC 5% FPR 5% TPR 10% FPR 10% TPR Observable .9066 .0334 .6234 .1054 .7403 Activation .9424 .0180 .7065 .0643 .7662 Joint .9427 .0206 .7065 .0720 .7740 At 5%, Joint-minus-Observable TPR is +.0831 with request-bootstrap 95% CI [.0494,.1169][.0494,.1169]; at 10% it is +.0338 with CI [.0052,.0623][.0052,.0623]. The identity- disjoint analysis therefore preserves both the activation increment and the failure to reach the frozen .80 TPR gate. Appendix C AgentDojo Exact Counts and Uncertainty Table 8 reports exact case counts and two-sided 95% ClopperāPearson intervals. These intervals quantify finite test support; they do not replace the independent calibration guarantee or the paired case-bootstrap contrast. Table 8. AgentDojo case operating points: count/denominator and exact 95% CI. Budget Monitor FPR (75 negative) TPR (38 positive) 5% Observable 2/75 [.0032,.0930] 2/38 [.0064,.1775] Activation last 0/75 [.0000,.0480] 2/38 [.0064,.1775] Joint last 0/75 [.0000,.0480] 1/38 [.0007,.1381] 10% Observable 15/75 [.1165,.3083] 22/38 [.4082,.7369] Activation last 14/75 [.1060,.2933] 20/38 [.3582,.6902] Joint last 15/75 [.1165,.3083] 22/38 [.4082,.7369] The paired Joint-minus-Observable 5% TPR difference is ā.0263-.0263, semantic-case bootstrap 95% CI [ā.0789,0][-.0789,0]; at 10%, the two monitors make identical positive-case decisions and the paired difference and CI are zero. Because 38 positive cases are below the frozen minimum of 40, these are labeled support-insufficient and are not used to promote a monitor. For the prespecified primary Activation-last monitor at the 10% budget, the threshold is the fifth-largest of 100 negative calibration maxima. Under a continuous independent-and-identically-distributed calibrationātest null, the number K of strict test exceedances among 75 negatives has the rank-predictive law Kā¼BetaBinomialā”(75,5,96)K (75,5,96). Hence Prā”(Kā„14)=.001745 (Kā„ 14)=.001745. With score atoms, strict exceedance can only make this upper-tail check conservative. Unlike conditioning on the 95% UCB, this calculation integrates threshold-selection uncertainty. It diagnoses calibration-to-test distribution mismatch; it does not estimate the causal effect of intervening on an alarm. Appendix D Repeated-Attempt Estimand Sensitivity The primary AgentDojo result aggregates ten prespecified seeds as admissible repeated attempts within one semantic security case. This is a sampled ten-attempt retry-set audit. To test dependence on that operational unit without generating new behavior, we re-aggregate the frozen scores over nested prefixes of the original seed order 101, 211, 307, 401, 503, 601, 701, 809, 907, 1009. Table 9 reports selected prefixes; no seed or prefix is chosen by its result. Formally, evaluation first labels completed attempts, then computes ZGā=maxi:YGāi=0SGāiZ_G^-= _i:Y_Gi=0S_Gi and ZG+=maxi:YGāi=1SGāiZ_G^+= _i:Y_Gi=1S_Gi. The reported class-max FPR and TPR are the rates of ZGā>tZ_G^->t and ZG+>tZ_G^+>t, respectively. A mixed case can enter both denominators. This estimand measures whether the sampled retry set contains a safe attempt that would consume review budget and whether it contains an unsafe attempt that would be detected. It is not a binary any-unsafe case classifier and does not simulate a controller that halts and suppresses future attempts after its first alarm; such a controller requires a new sequential contract. Table 9. Post-hoc AgentDojo retry-policy sensitivity. Entries are realized FPR/TPR at independently recalibrated FPR-certified thresholds. Support is negative/positive/mixed semantic cases. k Support 5% FPR/TPR 10% FPR/TPR Observable Joint Observable Joint 1 34/14/0 .000/.357 .000/.143 .088/.643 .118/.643 3 55/31/9 .036/.194 .000/.000 .127/.742 .109/.710 5 66/33/18 .015/.091 .000/.030 .121/.667 .106/.667 10 75/38/31 .027/.053 .000/.026 .200/.579 .200/.579 We also evaluate all ten single-seed policies. At 5%, only four of ten have enough negative calibration units for a finite threshold. Among those four, Observable TPR ranges .0476ā.3571; Joint-last TPR ranges 0ā.1429. At 10%, all ten are identifiable, but Observable TPR ranges .0476ā.9444 and Joint-last ranges 0ā.9565. This variation is not a model comparison: changing the number of attempts changes both label support and the safe-attempt alarm event. It shows why a repeated-seed benchmark result cannot be exported to a single-execution deployment without redefining and recalibrating the alarm policy. Appendix E Paired Cross-Benchmark Uncertainty Table 10 adds paired cluster uncertainty for the activation increment at each locked operating point. MUP resamples medical specialties; LASR resamples semantic requests. The intervals are pointwise and use 5,000 replicates with seed 20260713. Table 10. Paired TPR differences at FPR-certified thresholds. Setting Contrast 5% difference [95% CI] 10% difference [95% CI] MUP Joint ā- Observable +.2649 [.1529,.4268] +.0762 [-.0192,.2209] MUP Activation ā- Observable +.1225 [-.0391,.3627] -.0199 [-.1524,.1698] LASR Joint ā- Observable +.1059 [.0754,.1385] +.0407 [.0122,.0692] LASR Activation ā- Observable +.1059 [.0754,.1385] +.0367 [.0082,.0652] Appendix F Frozen Reproduction Matrices F.1. Eight-field claim-card instantiation Table 11. Instantiated Operational Validity Contracts. Cost measurements are reported as incomplete rather than inferred from predictive performance. Setting Target / horizon / observable Identity / unit / comparator / calibration Cost status MUP high-stakes input; input time; input text input; one input; frozen lexical observable; negative input maxima monitor and intervention costs unmeasured LASR future refusal; pre-response; original request request hash; semantic request; lexical + three Qwen forecasts; train calibration, official-all plus unseen test monitor and intervention costs unmeasured AgentDojo injection execution; pre-action; canonical visible history suite + user-task + case hash; sampled ten-attempt case; TFāIDF/metadata + three Qwen forecasts; disjoint frozen case calibration/test hook extraction measured; probe, observer, review, and intervention costs unmeasured ST-Web task failure; pre-action; browser-visible state intent-template hash; one template across locked safe/unsafe variants; frozen strong observer + three Qwen forecasts; disjoint template calibration/test environment reset and rollout costs measured; monitor, review, and intervention costs unmeasured F.2. Fail-closed claim compiler The compiler consumes the typed claim card and already frozen evidence; it has no interface for fitting, feature selection, threshold search, or subset selection. Let EspecE_spec require a complete contract and passing identity audit. Let ErepE_rep require an out-of-sample probe and held-out ranking evidence. Let EincE_inc require a declared same-horizon comparator, point increment at least the frozen minimum, and a positive lower clustered-confidence bound. For each budget, let EopE_op require an identified maximum certifying rank, prospectively prespecified test support, the budget-local locked TPR minimum ĻĪ± _Ī±, and realized FPR no larger than Ī±. The maximum claim is then computed in order: (10) claim=,Ā¬Espec,,Especā§Ā¬Erep,,Especā§Erepā§Ā¬Einc,,Especā§Erepā§Eincā§Ā¬Eop,,Especā§Erepā§Eincā§Eop.claim= cases invalid,& E_spec,\\ contract,&E_spec E_rep,\\ representation,&E_spec E_rep E_inc,\\ predictive,&E_spec E_rep E_inc E_op,\\ alarm,&E_spec E_rep E_inc E_op. cases Intervention evidence is an orthogonal verdict: it cannot raise this ladder and cannot be inferred from an alarm. Missing ranks are permitted only when rank-one support is analytically insufficient; finite policies must use the largest certifying rank and a matching Beta UCB. Twenty synthetic counterexamples exercise these transitions without benchmark outcomes. Table 12. Compiler outputs from the frozen contracts. āRep.ā is representation evidence; āInc.ā is the clustered comparator-increment gate. Setting Rep. Inc. Maximum claim Independent blocker(s) MUP pass pass restricted predictive value test support not prespecified LASR pass pass restricted predictive value test support not prespecified AgentDojo pass fail representation accessibility increment, support, transport ST-Web pass fail representation accessibility increment, calibration/test support Table 13. Parameters frozen before their corresponding evaluation phase. Object Frozen value Environment AgentDojo v1, commit 089ed468; official security outcome Agent Qwen2.5-7B-Instruct; Hermes native tool calls Generation temperature .7; top-p .9; maximum 512 new tokens Splits 345 model-fit, 119 calibration, 119 one-use-manifest test cases; user tasks disjoint Seeds 101, 211, 307, 401, 503, 601, 701, 809, 907, 1009 Boundary first policy request after the injected tool result Marker one global constant; no case or seed identity Activation zero-index decoder layer 14; final token primary, request mean secondary; float32 Observable exact visible history; character/word TFāIDF; suite/tool metadata; three frozen Qwen-14B forecast log-odds Fitting shared suite + user-task grouped folds; grouped out-of-fold late fusion Calibration semantic-case negative maxima; .95 Beta UCB; 5% and 10% budgets; strict exceedance; tie/equality audit Test policy consume one 1,190-identity ten-attempt manifest once; retain failed trajectories; no retraining or threshold changes Model fitting generated 3,450 frozen identities; 3,418 completed and 1,920 boundary-present rows entered feature fitting. Calibration generated 1,190 identities; 1,184 completed and 670 boundary rows across 103 cases were scored. The one-use test manifest generated all 1,190 identities; 1,165 completed and 602 boundary rows across 82 cases were scored. Prompt reconstruction, token/payload alignment, constant-marker purity, finite activations, and forecast-template alignment were 100% for all eligible rows. Full SHA256 values for locked scores and independently rechecked metrics are stored in the machine-readable audit. The supplement provides manifests and audit commands with portable paths. Appendix G Frozen-Score Replay and Screening Transparency The empirical replay starts from SHA256-pinned frozen score archives rather than the manuscript evidence summary. It verifies hashes, reconstructs semantic-unit maxima, recomputes AUROC, calibration ranks, Beta UCBs, selected-value ties and equalities, test FPR/TPR, exact intervals, group diagnostics, and paired bootstraps, and only then compares the recomputed values with the paper. All frozen-value comparisons pass. It also emits the LASR train/test identity overlap and semantic-unseen sensitivity; AgentDojo retry sensitivity; and ST-Web support, modality, worst-modality, and intent-template-bootstrap diagnostics. It fits no model and exposes no option to change a layer, pooling rule, threshold budget, template, or subset. The screening ledger covers all rejected, failed, negative, retrospective, and clean-rerun branches; the tabled settings are neither the only attempts nor a random task sample.