Paper deep dive

What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 143

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/23/2026, 2:59:30 AM

Summary

This paper introduces 'rule blindness,' a phenomenon where compliance detectors (guard models and activation probes) fail to distinguish between different regulatory rules, relying instead on surface features of the scenario. The authors propose the Internal Compliance Score (ICS), a training-free activation readout calibrated on ten labeled pairs, to audit these detectors. While ICS outperforms deployed guards in ranking and generalization within its calibration family, it is also found to be rule-blind, performing similarly to trivial bag-of-words baselines when rules are counterfactually altered. The study highlights that current benchmarks are often degenerate and that step-by-step reasoning is required to escape rule blindness.

Entities (10)

Rule Blindness · concept · 99%Internal Compliance Score · method · 98%GDPR · regulation · 95%Llama-Guard-3 · model · 95%OmniCompliance · dataset · 92%Qwen3Guard · model · 90%WildGuard · model · 90%SIREN · model · 88%LPG · model · 85%TF-IDF · baseline · 85%

Relation Signals (9)

Internal Compliance Score → istrainingfree → true

confidence 98% · ICS is the only one that is training-free, reads the monitored model directly...

Rule Blindness → affects → Llama-Guard-3

confidence 95% · Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard... including Llama Guard 3.

Rule Blindness → affects → Qwen3Guard

confidence 95% · Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard... including Qwen3Guard.

Rule Blindness → affects → WildGuard

confidence 95% · The failure spans the one-pass detector families we test... WildGuard...

OmniCompliance → isdegenerate → true

confidence 92% · Four of seven public compliance benchmarks are lexically degenerate... OmniCompliance... Degenerate

Rule Blindness → affects → SIREN

confidence 90% · An identical pattern has been reported independently for activation probes... SIREN...

Internal Compliance Score → detectsfailureof → Rule Blindness

confidence 90% · ICS is used to audit guards and probes, revealing that they exhibit rule blindness.

Internal Compliance Score → iscalibratedon → OmniCompliance

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy. Such monitoring is meaningful only if a detector's verdict depends on the stated rule rather than on surface features of the scenario. We show this condition fails across the current class of compliance detectors, a failure we call rule blindness. Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard and activation probe we test, including a policy-conditioned guard that correctly cites the governing clause yet barely changes its verdict when that clause is swapped for its permissive counterpart. A purpose-built benchmark crossing two rules with two scenarios, so that neither alone predicts the label, confirms the failure under a design no prior benchmark rules out, and shows that step by step reasoning, not any fast detector we test, is what escapes it. Auditing at scale requires a retraining-free detector, so we introduce the Internal Compliance Score (ICS): a training-free activation readout calibrated from ten labelled pairs and scored by a single projection. We hold ICS to the same scrutiny as the guards it audits: a pre-registered criterion for beating trivial baselines is not met, and a bag-of-words model matches its pooled generalisation exactly. It remains useful because it is inexpensive, letting us audit four deployed guard models, an 8B zero-shot judge, and thirteen benchmarks, and it raises the mechanically verified pass rate when used to rank candidate responses, though an adaptive white-box attack removes this gain. We release the counterfactual protocol and crossed-rule benchmark so rule blindness can be tested in future probe and guard claims.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

139,373 characters extracted from source content.

Expand or collapse full text

Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth Lexsi Labs saisab.sadhu@lexsi.ai Abstract Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy. Such monitoring is meaningful only if a detector’s verdict depends on the stated rule rather than on surface features of the scenario. We show this condition fails across the current class of compliance detectors, a failure we call rule blindness. Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard and activation probe we test, including a policy-conditioned guard that correctly cites the governing clause’s position ninety one to ninety five percent of the time yet barely changes its verdict when that clause is swapped for its permissive counterpart. A purpose-built benchmark crossing two rules with two scenarios, so that neither alone predicts the label, confirms the failure under a design no prior benchmark rules out, and shows that step by step reasoning, not any fast detector we test, is what escapes it. Testing this at scale requires a detector cheap enough to run across many guards and benchmarks without retraining, so we introduce the Internal Compliance Score, a training-free activation readout calibrated from ten labelled pairs and scored by a single projection. We hold it to the same scrutiny as the guards it is compared against, and a pre-registered criterion for beating trivial baselines is not met, and a bag-of-words model matches its pooled generalisation exactly. It remains useful because it is inexpensive, letting us audit four deployed guard models, an 8B zero-shot judge, and thirteen benchmarks with no prior guard comparison, and it raises the mechanically verified pass rate when used to rank candidate responses, though an adaptive white-box attack removes this gain. We release the counterfactual protocol, including the crossed-rule benchmark, so rule blindness can be tested for in future probe and guard claims. Keywords: compliance monitoring, activation probing, guard models, representation engineering, rule blindness 1 Introduction Deploying a language model in a regulated setting requires checking outputs against written rules, and this checking is increasingly performed by a guard model, a classifier fine-tuned on a large labelled corpus and run as an additional forward pass [1,2,3]. Consider the storage-limitation principle in the General Data Protection Regulation (GDPR), which limits retention of personal data to what is necessary for the purpose of its collection [4]. A compliance monitor might operationalise this principle as a concrete rule, deletion within ninety days, and check it against a case in which data was retained for four hundred days. A deployed guard correctly flags the violation, yet flags the identical case even when that rule is replaced with an unrelated one or removed outright, so its verdict tracks the scenario rather than the governing rule. The same requirement recurs across data protection, healthcare privacy, financial regulation, and arXiv:2608.16852v1 [cs.AI] 17 Aug 2026 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models platform content policy alike, and the twenty regulatory domains evaluated in this paper span exactly this range, so the failure is not confined to one jurisdiction or industry. We term this failure rule blindness, and show it is not confined to a single system, since deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard model we test and for a training- free activation probe of our own. The guards divide into two kinds and both fail, for different reasons. Fixed-taxonomy safety classifiers such as Llama Guard 3 and Qwen3Guard never receive the rule, so they cannot track it by construction and their failure is a scope limitation rather than evidence about rule-conditioned systems. Policy-conditioned guards do receive it: LPG reads the rule through its own documented custom-policy channel, correctly cites the governing clause, and still barely changes its verdict when that clause is swapped for its permissive counterpart. The failure therefore spans the one-pass detector families we test, conditioned and unconditioned alike, rather than any particular architecture or training regime. An identical pattern has been reported independently for activation probes applied to safety classification [5,6] (full positioning in the paper’s Extended Related Work section (§W)), suggesting it is not specific to regulatory compliance. This matters because compliance monitoring is increasingly adopted as a legal and audit control across these sectors, and a detector that cannot distinguish a scenario from the rule that governs it offers only the appearance of rule-specific assurance, regardless of which rule is in force. Establishing rule blindness at the scale of an entire field, across many guard models, activation probes, and benchmarks, requires a detector inexpensive enough to be run repeatedly without retraining. Guard models cannot meet this requirement, since fine-tuning demands a large labelled corpus, inference demands a complete additional forward pass, and the resulting classifier remains frozen and cannot track a monitored model that is later fine-tuned. We address this requirement by constructing the Internal Compliance Score (ICS), a difference-of-means readout in the lineage of representation engineering, the refusal direction, and mass-mean probing [7,8,9,10], applied here to written rules rather than safety labels. ICS reads the activations of the monitored model directly rather than querying a separate network, is calibrated from ten labelled pairs, and is scored by a single dot product, allowing the rule-blindness audit to be conducted across deployed guards (Llama Guard 3, WildGuard, Qwen3Guard, HarmBench [11], SIREN [5], GLiGuard, and Latent Policy Guard [12]), activation probes, an 8B zero-shot judge, and thirteen external benchmarks on which no prior guard comparison exists (§D.1); the nearest published neighbour is a compliance-framed whitening score with a different estimator and readout [13], positioned fully in the paper’s Extended Related Work section (§W). ICS is evaluated against the same budget-matched selection null and pre-registered floor criterion applied to the guards it is compared against (§B). Among the methods compared here, ICS is the only one that is training-free, reads the monitored model directly, and recalibrates to a changed model at negligible cost, even though every method under test, ICS included, is rule-blind; the claim advanced for it is accordingly structural rather than one of superior accuracy. Whether such probes and their benchmarks can be trusted at all is a standing concern [14,15], positioned fully, including against dynamic policy-conditioned guards, in the paper’s Extended Related Work section (§W). This paper makes four contributions, in that order of priority. •Method. We introduce the Internal Compliance Score, a training-free activation readout that scores compliance risk directly from the monitored model, built from a difference of class means over a small labelled calibration set with no gradient step and one projection per case. •Evaluation. We compare ICS against deployed guards, internal-representation probes, a gradient-based training-free detector, an 8B zero-shot judge, and lexical baselines, across compliance, safety, and held-out distributions, under budget-matched selection nulls, lexical floors, benign-input false-positive rates, and matched threshold budgets. The same pass audits the public compliance benchmarks themselves: four of seven are solvable by a policy-blind bag-of-words model, so they cannot adjudicate a rule-conditioned claim at all. • Characterisation. Counterfactual rule interventions, and a purpose-built benchmark crossing rules with scenarios so that neither alone predicts the label, show that ICS and every efficient detector we test capture a broad violation signal more reliably than they compose the supplied rule with the scenario. We name this rule blindness, document it across every guard architecture and robustness check we run, and bound what any activation-based compliance readout can claim. • Application. ICS ranks candidate responses well enough to raise the mechanically verified pass rate on IFEval by 5.2 percentage points, while a white-box adaptive attack drives the same mechanism to its floor and so fixes a non-adversarial scope for it. 2 Method: The Internal Compliance Score Figure 1 gives the full pipeline this section formalises. We builddonce from ten pairs, calibrateℓ ∗ andτon two disjoint held-out slices, then score any new case at the cost of one dot product, with every reported number checked 2 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Figure 1 The full pipeline, build to score to decide. Offline calibration (left) fits the direction and calibrates the anchor layer and threshold on two disjoint held-out slices, formalised opposite, before any test case is touched. Online inference (centre) costs one forward pass, one activation extraction, and one dot product. The decision stage (right) thresholds the score or, where useful, ranks or routes on it. Total cost is 2n forward passes to calibrate and one to score, no gradient step anywhere. Omitted for space: every scored arm is also checked against a budget-matched null, a lexical floor, and, where applicable, the pre-registered gate. against a null, a floor, and the pre-registered gate. LetMbe a decoder transformer withLblocks and residual widthd. Leta ℓ (x)∈R d denote the layer-ℓresidual-stream activation at the final token of textx, with the token position and site ablated in §O. Givenncalibration pairs(x + i ,x − i ) n i=1 of adherent and violating texts for a rule familyR, the adherence direction at layer ℓ is the normalized difference of class means, ˆ d ℓ = ̄a + ℓ − ̄a − ℓ ∥ ̄a + ℓ − ̄a − ℓ ∥ + ε , ̄a ± ℓ = 1 n P i a ℓ (x ± i ),(1) and the score is the projection ICS ℓ (x) =⟨a ℓ (x), ˆ d ℓ ⟩. The numerator of(1)points from the violating prototype to the adherent one, so higher ICS means more adherent, and every sign in the paper follows from that choice. Guardrail baselines use the opposite convention, an easy source of silent inversion we guard against explicitly with a 10 −9 reference-implementation check. Since⟨a, ˆ d⟩ =∥a∥ cos(a, ˆ d), the raw score mixes angle with norm, and norm is a live confound (lengthalone scores 0.566), so we report the cosine variant beside it; the two tie, and∥a∥alone is near chance (§O). Equation(1)with a midpoint threshold is nearest-centroid classification, Fisher’s LDA under identity covariance, not parameter-free but requiring no optimizer and no hyperparameter sweep, though the full pipeline still spends labelled data on two validation slices, one to pick the layer and one to set the threshold, and at smallnapproximately the regularized logistic solution (§U), which we treat as the informed ceiling. Each domain’s pairs split40/20/20/20into TRAIN, VAL-LAYER, VAL-THRESH, and TEST (seed 42, grouped so no text straddles a boundary): the direction is fit on TRAIN, the anchor ℓ ∗ = arg max ℓ AUROC val-layer on the second (a single consensus layer costs< 0.01), the thresholdτon the third, and TEST is touched once. We report AUROC as primary, since the equal-variance assumption behindτfails, and FPR@95%TPRas the operating point; every headline finding survives re-expression under both and AUPRC (§P), so none is an AUROC artifact. The hypothesis isI(a(x);Y | S(x)) > 0, that activations carry adherence signal beyond text surfaceS(x), lexical content, sentiment, length. A benchmark that narrates the verdict makesYa function ofS(x)and any AUROC uninformative (§3), so every headline number is on outcome-ablated inputs and the headline statistic is the margin ∆ = AUROC ICS − max b∈B AUROC b over trivial baselines, with paired-bootstrap CIs and BH correction.∆≤ 0is a reportable outcome, not a failure mode. Four statistical rules hold throughout. First, every metric appears with its exact null. Second, ceilings are one-sided Clopper–Pearson bounds, never1.0; raw perfect-separation point estimates are reported conservatively as> 0.999. Third, best-of-kis scored against its exchangeable null,P (arg max ̸= first) = 1− 1/k, the gain over a random-selection arm. Fourth, a permutation-null direction accompanies every detection table. 3 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 1 Four of seven public compliance benchmarks are lexically degenerate. Lex. Floor, strongest policy-blind bag-of-words model. Degenerate, floor is high and the probe does not clear it, even for the human-expert-labelled one (AIReg-Bench). Full 14-benchmark table, Table 16 in §D. BenchmarkLex. FloorICSBudget NullVerdict OmniCompliance0.8960.9520.714Degenerate DynaBench0.9820.7840.576Degenerate SafePyramid0.90+0.8710.673Degenerate AIReg-Bench0.9570.9430.708Degenerate CompliBench0.8790.7000.549Borderline FlexBench0.8020.8320.603Clean IFEval0.5740.6280.530Clean Unless noted, the model is Llama-3.2-1B-Instruct and the corpus is outcome-ablated OmniCompliance [16], 20 domains, with theOutcome:field removed from every scored text (§3), same split above. The full model roster, baseline and floor pool, and statistics are in §A. What ICS is calibrated on, and what it is tested on. Calibration and test are always disjoint, asserted throughout (§A), and only the calibration set’s breadth varies below. Per-distribution calibration draws its pairs from the same distribution as the test, the strong in-domain case. Calibrate-once withholds an entire distribution, building the direction from the others so the held-out one appears only at test time, exactly as a deployed guard meets an input it was never trained on (Figure 3). 3 Evaluation Validity: What Compliance Benchmarks Can Measure This criterion determines which numbers in the paper are meaningful, so we state it before reporting any. OmniCompli- ance [16] structures each case as a rule, a scenario, and an OUTCOME field that states the regulatory result in natural language, for example “fined EUR 725,000 for privacy violations,” and the label is written into the input. On the as-published task the trivial baselines saturate, so no probe result on it is interpretable. The obvious diagnosis, that a probe reads the narrated verdict, is refuted by ablation. Removing the OUTCOME field and refitting barely moves detection (20-domain mean 0.968→0.952): the field is sufficient for the label but not necessary, and the label is encoded redundantly across the scenario. The problem is therefore degeneracy, not contamination. WhenYis near-deterministic in the text surfaceS(x), the conditional termI(a(x);Y | S(x))has no room to be nonzero, bounding what any method on the corpus can show, ours included. It is a property of provenance, not of one dataset: whenever a label is fixed before its prose is written, or recoverable from prose written afterwards, it survives in surface vocabulary. Consequently every detection number we report is on outcome-ablated inputs, carries its trivial-baseline rows and a permutation null, and bolds only the margin over the strongest named baseline. We treat the ablated task as necessary but not sufficient, and the same audit applied to six further public compliance benchmarks leaves only two that can adjudicate a probe at all (Table 1); §D adds benchmarks with mechanical rather than narrated ground truth. Splits are case-level, and we verified the leakage profile rather than assuming it (68% of test cases share a source_rulewith train, yet a rule-disjoint restriction leaves AUROC unchanged, 0.962 vs. 0.956). This is a statement about what such corpora can evaluate, not their construction quality. 4 Generalisation Beyond the Calibration Distribution A guard’s value is that it is calibrated once and then handles whatever arrives, so the first question for a training- free readout is whether it survives leaving its calibration distribution. We answer that here and defer to §6 the separate question of what the resulting score represents. We compare ICS against generative guards (Llama Guard 3, Qwen3Guard-8B, WildGuard-7B, HarmBench), internal-representation probes (SIREN, GLiGuard), an 8B zero-shot LLM judge, and Latent Policy Guard [12], a learned latent reasoner sharing no method overlap with ICS but the same cheap-inference-time-adherence motivation, on the full outcome-ablated OmniCompliance benchmark, under every metric and on benign inputs. We test whether a training-free direction is a calibrate-once guard competitive at near-zero cost: generalisation, external sources, and efficiency support it; cross-family collapse, rule-blindness, and an adaptive attack bound it; in-domain accuracy is the control. Of the five detector types tested, zero read the rule rather than a generic violation signal (§6) or hold across compliance and safety data (Table 12). ICS’s own cost advantage, zero trained parameters and roughly 4 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 2 Calibrate-once, leave-one-distribution-out (AUROC↑). Null, budget-matched random directions; In-Dom., per-distribution ICS within this experiment’s own protocol (an upper bound, not a second measurement of the 0.952 headline figure). Clears its null (0.728 vs. 0.557) but ties TF-IDF in the mean. Held-Out DistributionFamilyICSNullTF-IDFIn-Dom. OmniComplianceCompliance0.7830.5380.5230.976 TRIDENT-financeReg-QA0.7370.5870.8360.837 TRIDENT-lawReg-QA0.5490.5550.8650.777 TRIDENT-medReg-QA0.9170.6270.9370.916 ToxicChatSafety0.7060.5460.6740.846 OpenAI Mod.Safety0.7080.5490.7250.898 Aegis 1.0Safety0.6390.5290.6850.833 Aegis 2.0Safety0.6940.5120.6610.780 WildGuardSafety0.7180.5590.6230.864 PKU-SafeRLHFSafety0.7400.5500.7770.800 BeaverTailsSafety0.8200.5740.7030.810 Mean0.7280.5570.7280.849 ten labelled pairs per source distribution (∼100–160 pooled total, not ten total, §O.1) against every other arm’s 1–8B parameters and 10 3 –10 5 pairs, is in Figure 2. We pool compliant/violating and safe/unsafe pairs from eleven distinct distributions (OmniCompliance, the three TRIDENT domains, and six safety benchmarks, Table 2), fit a single difference-of-means direction, and test it leave-one- distribution-out: calibrate with a distribution held out entirely, then score it. Mean AUROC on distributions never seen is 0.728, 10/11 above 0.60, clearing its budget-matched null (0.557) at a−0.12cost against per-distribution calibration (0.849). The cross-family case is sharpest: with OmniCompliance held out entirely, trained only on TRIDENT and safety, the direction still reads regulatory compliance at 0.783, above the deployed guards on that distribution (Llama Guard 3 0.689, SIREN 0.697). A pooled bag of words matches the pooled mean exactly (TF-IDF LODO 0.728), but that tie is an average: on the outcome-ablated compliance task itself, where lexical shortcuts are removed by design, the internal direction beats TF-IDF by+0.26, and TF-IDF only draws level on distributions with residual surface cues, so ICS’s case where it ties is structural, first-party, zero-parameter, and re-fittable rather than accuracy (§H). Two scope conditions bound the result for a deployer. The direction, not the operating threshold, is what transfers calibrate-once: a fixed cross-domainτblowsFPR@95up to 77–94% (§Q), so the threshold is re-calibrated per domain even as the direction is reused. And the generalisation is within-family: withholding a whole family, all safety or all compliance, collapses cross-family transfer to chance (0.55/0.48, §F). The complete calibrate-on-X, test-on-Y grid (Table 12) confirms that block structure at every cell, and no single calibration ships everywhere: even the best single calibrator collapses on the opposite family. Calibrate on the family you will monitor. A stricter check scores the three pool directions on seven never-pooled external benchmarks sharing no row, benchmark, or collection process with any pool. The readout fires above chance on never-seen sources and the family structure mostly holds, but the compliance half holds only marginally: an external compliance set is read above chance by the shared signal, not the compliance-specialised direction (Table 13). 5 Comparison with Deployed Guards Calibrated on the test distribution directly (Table 21), ICS reaches AUROC 0.952 on OmniCompliance and beats every deployed guard head-to-head (+0.203over Llama Guard 3,+0.255over SIREN, an MLP on the same activations, 0.966, and LPG-4B, 0.943, tie it). This is the in-domain case: ICS gets in-domain pairs while the guards run zero-shot out of their safety-training domain, so the leave-one-out number above, not this one, is the calibrate-once comparison the claim rests on. A guard can screen a compliance case as an input (Scenario in the prompt slot) or judge it as a response (Scenario in the output slot against the Rule); the two are not interchangeable, so we run every generative guard in both modes and report both (§G). Response mode is stronger for all four guards on compliance, decisively for WildGuard, whose prompt-slot head is near-chance on TRIDENT and recovers only in response mode (0.785→0.875 AUROC on OmniCompliance). An earlier draft’s per-metric tables here and in Table 21 mixed a fixed prompt-slot mapping, the weaker of the two modes for Llama Guard 3, WildGuard, and Qwen3Guard, understating them by 0.060, 0.090, and 0.023 AUROC 5 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 3 Detector comparison, in-domain. Native-threshold F1 (Native) vs. ten-pair-budget F1 (Recal.); guard AUROCs now uniformly credit each guard its stronger, response mode (§5, full metrics in Table 21 and Table 22). ArmAUROCF1 (Native)F1 (Recal.) ICS (ours)0.9520.8970.895 Qwen3Guard-Gen (8B)0.9080.1580.804 LLM judge (zero-shot, 8B)0.9010.8310.817 WildGuard (7B)0.8750.1240.715 HarmBench cls (7B)0.7350.0940.656 Llama Guard 3 (1B)0.7490.2650.580 respectively; every headline table now credits each guard its response mode uniformly, recomputed and verified per-domain-then-mean from the raw scores behind Table 20. The comparison the claim actually rests on, calibrate-once generalisation (Table 2), never used these rows either way. ICS sidesteps the mode choice, since it reads the activation over the case text itself. At their shipped threshold, guards rarely flag a regulatory violation (recall 0.08–0.22), so native F1 is low against ICS’s 0.904 (Table 3); this is mostly a threshold artifact, not a capability gap. Given every guard the same target-domain threshold budget ICS gets (native, ten pairs, the full budget, or a Platt/isotonic map, fit held-in and scored on a disjoint fold), the collapse recovers to F1 0.54–0.80 (Table 22). ICS’s advantage is ranking, not threshold access: its AUROC (0.952) clears every guard’s (0.735–0.901) budget-free, and at matched calibration it still leads every threshold metric (F1, FPR@95, TPR@10), so ranking, or the calibrate-once generalisation (Table 2), is the comparison that holds. Two supporting results are summarised here and given in full in §H. The training-free story is not idiosyncratic to difference-of-means: GradSafe, a gradient-based training-free detector, also beats every deployed guard on compliance once re-referenced with sixteen compliance pairs, and collapses to chance under its native safety reference, the same failure the safety-trained guards show. And because ICS reads the monitored model’s own activations, it can be re-fit on that model after fine-tuning, where a shipped guard is a separate frozen network; a fresh ten-pair refit holds across nine models and multiple fine-tunes, AUROC within 0.007 of base (§H). Guards are trained for safety, so we also run every arm on their home domain, the standard seven safety benchmarks, by Macro F1 (Table 11), and on benign inputs (Table 24). There they are strong on detection but pay in over-refusal, WildGuard flagging 75% of adversarially-benign OR-Bench prompts, which is what a comparison run only on regulatory compliance, the guards’ non-native domain, would otherwise obscure. §6 shows the whole class shares a rule-blind spot. 6 What the Score Measures The sections above establish that ICS ranks compliance risk competitively and at low cost. None of it establishes what the score is reading, and that is the question this section answers, for ICS and for the guards it was compared against. The task, as stated, asks whether a given scenario violates that rule, not some other one, so a detector at AUROC 0.952 could be composing the rule with the scenario or could be recognising that the scenario sounds like a violation and ignoring the rule entirely. Conventional accuracy cannot separate the two. We test whether the probe uses the rule at all by refitting ICS from scratch under four corruptions of the rule block and one of the scenario (Table 4). It does not, and the finding is not special to ICS: the same corruptions leave a 4B policy guard with an explicit policy channel unmoved (below), so a first-party probe and a shipped guard that is actually given the rule both read a generic violation signal rather than the rule-scenario relation. Nor is it an artifact of a benchmark whose label ignores the rule, since it survives the sharpest available counterfactual, SafePyramid’s surface-matched flip pairs, where the conversation is byte-identical within a pair and the policy alone flips the label (below). The rule is unnecessary, and it is not an artifact. Deleting the rule moves mean AUROC by+0.003, a slight help, with none of 20 domains showing a BH-significant drop and none a gain, symmetric noise rather than an underpowered near-miss. Shuffling the rule and substituting a different domain’s rule give the same answer, and it is not length or truncation: the three conditions are length-identical, no test text nears the 512-token cap, and the pipeline reproduces the published gate bit-exactly before any condition runs. Nor is it an artifact of one model, one benchmark, one rule source, or our particular readout. Table 6 collects every check we ran against those possibilities; all of them leave the finding intact, and the two positive controls confirm the test has power to detect change. 6 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 4 The rule is optional. Sig. Drops, BH-significant AUROC decreases vs. correct-rule (exact arithmetic, 20 domains). A wrong-domain rule carries zero class information yet costs nothing, and rule-only is the positive control. Replicates on four further models (Table 10). ConditionICS (1B)Null∆ (Cond−Corr)Sig. Drops Correct rule0.95190.714— No rule at all0.95490.719+0.0030/20 Shuffled rule0.95480.713+0.0030/20 Wrong-domain rule0.95540.712+0.0030/20 Rule only, no scenario0.56270.527−0.38919/20 TF-IDF, same conditions (null = 0.5, identical by construction) Correct rule0.89560.500— No rule at all0.93080.500+0.0350/20 (13 gains) Rule only0.58810.500−0.30819/20 Table 5 Confound-free crossed benchmark: every cheap detector is at chance; chain-of-thought is the one configuration that moves. Chance is 6.25% quadruple exact match, 0.500 AUROC. Chain-of-thought is a positive control (40/200 quadruples, one prompt, one seed, not compute-matched against forced-choice; Wilson CI 58.9–85.4%), so it shows the benchmark is solvable, not that reasoning is necessary or sufficient. Full detail in §B.3. DetectorAUROCQuad. Exact (%) TF-IDF (rule, scenario, or both)0.5000.0 Frozen ICS (1B/8B/Qwen3-8B)0.50–0.540.0–3.5 Best refit (Qwen3-8B MLP)0.67011.0 Llama Guard 3 / Qwen3Guard0.50–0.520.0–0.5 LPG-4B (rule as custom policy)0.5351.0 Rachmil whitening (their guards)0.4980.0 Forced-choice judge (1 token)0.5070.0 Same judge, chain-of-thought0.84974.4 A fixed-scenario counterfactual isolates the readout from the model, but the natural design has a confound a reviewer specifically flagged: holding one scenario fixed and flipping only the rule means rule polarity alone, permissive versus prohibitive, determines the label throughout the set, so a detector that has merely learned to read polarity, not the rule-scenario interaction, could pass without doing anything relational. We resolve this with a crossed design that removes the confound by construction rather than by argument. Each of 200 templates across 8 regulatory domains supplies two rules and two scenarios crossing into four labelled cells, rule R0 with scenario Sa a violation, R0 with Sb compliant, R1 reversing both, so neither the rule text nor the scenario text alone predicts the label, each appearing exactly once with each outcome across the 800 rows. Labels come from a numeric threshold comparison, computed mechanically, never judged, and the no-shortcut guarantee is verified rather than assumed: a TF-IDF classifier trained on the rule alone, the scenario alone, or the two concatenated scores exactly chance, 0.500 AUROC, group-disjoint five-fold cross-validation by quadruple (§B.3). On this genuinely unconfounded set, every cheap detector we test sits at or barely above chance, and the two detector classes built specifically to escape rule blindness do not (Table 5). LPG was designed to condition on the active policy rather than surface priors, and its own violated-policy-removal probe already showed two baseline dynamic guards flipping to safe only 64% and 36% of the time when the violated policy is removed rather than reliably, evidence the authors read as guards conditioning on positional and content priors rather than the supplied clauses [12], their figures, not ours. LPG does not escape on our crossed set either. Refitting a probe directly on the crossed data, the strongest possible case for a shallow readout, recovers a real but weak scale-dependent signal, Qwen3-8B topping out at 0.670 AUROC and 11.0% quadruple exact match against a 6.25% chance rate, not a solved task. One configuration does substantially better: the identical local judge, given the same rule and scenario but allowed to reason step by step rather than forced to a single token, reaches 0.849 AUROC and 74.4% quadruple exact match on a seeded 40-quadruple subsample (Wilson CI 58.9–85.4%). That control is neither compute-matched against the forced-choice row nor run over the full set, so it establishes that the benchmark is solvable, not that step-by-step reasoning is necessary or sufficient for it. What it does rule out is a degenerate design, leaving the failure specific to detectors that read a generic violation signal in one forward pass rather than explicitly composing the rule and the scenario. 7 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 6 Rule blindness across every robustness check. Each row refits ICS from scratch under the stated condition against correct-rule; Holds means the no-rule result is unchanged. The two positive controls show the test has power: removing the scenario collapses detection, and the topical rule axis is decodable but label-orthogonal. Full derivations and replication detail throughout this appendix. Robustness checkResultHolds? Is the no-rule result an artifact of one model? Llama-3.2-3B, Llama-3.1-8B0/20 sig. dropsyes Qwen2.5-7B, Qwen3-8B2/20, negligible effectyes LoRA verdict-SFT (0.67→0.93 acc.)0/6 sig. dropsyes Of the benchmark, or of machine-written rules? TRIDENT (real CFA/ABA/AMA codes)∆ < 0.001, 0/3yes SafePyramid surface-matched pairs0.5000 vs. null 0.519yes Of reading the rule at the wrong place? LPG, permissive counterpart swapped inflips 7.1% of casesyes Refusal-orthogonalised contrastive fit0.605, below null 19/20yes Positive controls (these must move, and do) Rule only, no scenario0.56–0.57— On- vs. off-topic rule (topical axis)decodable 0.90, label-orthogonal— A companion, earlier fixed-scenario counterfactual replicates the same qualitative pattern with a caveat the crossed design above resolves (full detail, cross-model replication, and an independent frontier-judge cross-check against Haiku 4.5, GPT-4o, and Gemini 2.5 Flash under the local judge’s exact prompts, in §B.2 and §M, including the frontier-judge validation figure there). The most direct repair also fails: building the direction so that it must attend to the rule, calibrating from badness- matched rule-varying contrasts and orthogonalising against the refusal axis after conditional activation steering [17], lands at 0.605, below its null in 19/20 domains, and deleting the rule still costs nothing. A shipped policy-conditioned guard fares no better: swapping the violated rule for its exact permissive counterpart, scenario and decoy policies fixed, flips LPG’s verdict on only 7.1% of cases even though it cites the governing clause’s list position 91–95% of the time (§B.5). Citing the rule and adjudicating against it are separable, and these detectors do the first. ICS is therefore better interpreted as a broad compliance-risk direction than as a rule-specific adjudication direction. Our geometry section could not decide between these two readings within a single-template benchmark. This measurement does. 7 Score-Guided Candidate Selection The strongest practical result is on substantively different generated responses, not menu labels, and is mechanically verified: an ICS-guided pick of instruction-following responses beats random by +5.2p against a mechanical verifier (541 cases), scaling with within-prompt detection AUROC. A frozen direction, the same one, no re-fitting, also beats random by +11.5p on LLM-judged regulatory advisory responses across six domains, surviving a coherence control that rules out quality-selection on the judged arm (§L). Across a mechanical verifier and a judge, on format and on regulation, the gain is real and modest. It holds in non-adaptive settings only: a white-box suffix optimised against the direction drives the same selection to its floor (§N), so this is not robust enough for standalone adversarial enforcement; the supported use is score-guided ranking under trusted or non-adaptive candidate generation, not adversarial enforcement. As a controlled sanity check, the same mechanism is tested on a classification corpus, OmniCompliance, where best-of-kselection (k=5, pure temperature sampling, ablated inputs) returns the candidate whose ICS-predicted verdict is correct (Table 7), capturing 64–79% of the oracle headroom in every one of six models from 1B to 9B, significant and positive with zero significant negative domains anywhere. This is verdict-selection over sampled menu labels, not content-selection, so the headline headroom number is the more modest generated-response result above, not this one. The absolute benefit is governed by the generator’s blind accuracy: gain over the random arm tracks the available headroom (oracle−random) with slope 0.79, intercept−0.003, andR 2 = 0.94across 120 model-domain points (mean+0.27where blind accuracy< 0.70versus+0.02where> 0.85; Figure 4). ICS-guided selection also beats the same-model zero-shot verdict logit, a readout needing no direction, at 0.872 vs. 0.703 on the 1B, McNemar p = 1.2× 10 −95 , all 20 domains, with candidate diversity verified directly (both verdicts present in 88.9% of cases), confirming the direction itself does the work rather than the sampling. 8 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 7 Verdict-selection sanity check, best-of-5 (outcome-ablated OmniCompliance verdict accuracy). Blind, no selection; Random/ICS-Guided/Oracle, best-of-5 by random, ICS, and true-verdict pick; headroom = (guided−random)/(oracle−random). Absolute gain shrinks on stronger generators as their room shrinks; the recovered fraction (64–79%) does not. GeneratorBlindRandomICS-GuidedOracleHeadroom Llama-3.2-1B0.5690.5410.8720.96178.8% Llama-3.2-3B0.7710.7730.8890.94666.7% Mistral-7B0.7750.7700.7950.80571.8% Llama-3.1-8B0.7830.7810.8890.94566.1% Gemma-2-9B0.7970.7930.8320.84870.4% Qwen2.5-7B0.8430.8420.8620.87363.9% 8 Scope and Limitations Five limits bound the headline claims. ICS requires first-party activation access, an advantage for self-monitoring, not third-party auditing. Its direction transfers within related task families, collapsing across compliance and safety data (0.55/0.48, §F), and its threshold needs local recalibration regardless. The score captures a broad compliance-risk signal rather than reliable rule-specific composition. A trained MLP or logistic probe on the same activations is slightly more accurate (0.966 vs. 0.952) at every calibration size tested, a cost trade rather than an accuracy one (§U). Finally, the readout is serialisation-sensitive, collapsing to 0.668 when the scored text ends on the rule rather than the case, and an adaptive white-box attack defeats it as a standalone enforcement control [14]. Further scope conditions in §V. 9 Conclusion The Internal Compliance Score is a training-free readout of the monitored model’s own activations, a difference of means over a small calibration set scored by one projection per case, no gradient step. It ranks compliance risk competitively in domain, keeps useful signal on distributions withheld from calibration (leave-one-out AUROC 0.728 against a 0.557 null, worst case 0.549), and re-fits after fine-tuning, which a separate guard cannot. Counterfactual testing bounds what the score represents. It reads as a compliance-risk direction rather than a rule-specific adjudication direction, since deleting or substituting the rule leaves it, and every guard and probe tested, unchanged. A bag of words matches its pooled generalisation, transfer between compliance and safety data is weak, and an adaptive attack removes its selection gain. ICS is a low-cost method for monitoring, triage, and candidate ranking, released with the counterfactual protocol that establishes those limits. 9 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models A Experimental Setup Models.Table 8 lists the roster. All extraction isbfloat16, last-token, residual stream. Layers are indexed0,...,L, so the anchor search over the layer-validation slice ranges overL+1positions including the embedding (17 for the 1B), and the pipeline reproduces the stored gate receipts bit-exactly before any new arm runs. Detection and geometry use Llama-3.2-1B-Instruct, and the remaining arms replicate detection and, per the paper’s Score-Guided Candidate Selection section, selection. Compute budget. All detection, gate, and cross-model arms (1B–72B, twelve models) run on a single GPU per arm, the two frontier-scale (70B/72B) rows in int8 (§S); no arm requires multi-GPU sharding. Calibration is ten to two hundred labelled pairs per domain and one forward pass per case, so the full 20-domain, twelve-model detection sweep is on the order of low-thousands of forward passes, not a training run. The adaptive white-box attack (§N) is the single most compute-intensive component, a 40-step GCG [18] search per prompt over 20 stratified prompts; every other reported number is inference-only. Total compute is released with the evaluation protocol. Table 8 Twelve models, four architecture families, 1B to 72B. All extractionbfloat16except the 72B arm (int8; see the 72B Arm methodology note in Cross-Model Replication, §S). † Run in an isolated environment (transformers 5.14). A plain — means not applicable to that row; a separate — in the d column means not required for this paper’s experiments. ModelFamily L+1dRole Llama-3.2-1B-InstructLlama172048primary (detection, geometry, keystone) Llama-3.2-3B-InstructLlama293072detection, rule-blindness Llama-3.1-8B-InstructLlama334096detection, rule-blindness, keystone, geometry Gemma-2-9B-itGemma433584detection Mistral-7B-Instruct-v0.3Mistral334096detection Qwen2.5-7B-InstructQwen293584detection, rule-blindness Qwen3-8BQwen374096detection, rule-blindness, keystone, geometry Llama-3.3-70B-InstructLlama818192detection (cross-model replication, 70B arm) Qwen3.5-4B / 9B † Qwen332560 / 4096detection Qwen3.5-27B † Qwen655120detection Qwen2.5-72B-Instruct (int8)Qwen818192detection (frontier-scale arm) Data.The primary corpus is outcome-ablated OmniCompliance [16], 20 domains, with theOutcome:field removed from every scored text (the paper’s Evaluation Validity section). Detection fits the direction on the 40% train slice of each domain’s 73–500 pairs and reports AUROC on its 16–100-pair test slice. Selection uses its own generation sets of 200 cases per domain (the paper’s Score-Guided Candidate Selection section). We additionally run the gauntlet on six external compliance benchmarks and seven safety benchmarks (Table 16). Splits are 40/20/20/20 (train, layer-val, threshold-val, test), seed 42, grouped so no unit sharing text straddles a boundary. Baselines and the floor pool.The gate (§B) is defined against a pre-registered floor set, TF-IDF, keyword, length, VADER, the same-model Yes/No verdict logit, the SST-2 sentiment direction, and an Arditi-style refusal direction fit with zero compliance supervision [8], fixed before any experiment ran. The 11/20 verdict is the margin over the per-domain maximum of this exact set. Tier-2 estimators (logistic, CCS, shrinkage-LDA, whitened difference-of-means, a one-hidden-layer MLP) and Tier-3 deployed systems (Llama Guard 3, Qwen3Guard-8B, SIREN, GLiGuard, Latent Policy Guard, an 8B LLM judge) are compared in the paper’s Generalisation Beyond the Calibration Distribution section. Every selected direction is additionally scored against a budget-matched selection null, 200 random unit directions pushed through exactly that arm’s hyperparameter search. Statistics.We use paired bootstrap with the pair as the resampling unit (B=2000), Benjamini–Hochberg across the 20 domains with thresholds in exact rational arithmetic (§B reports a verdict that turns on this), and Clopper–Pearson bounds where a cell saturates, with anchor-layer stability read from5×2seed-replicate grids. Every AUROC passes through a wrapper asserting agreement with a reference implementation to10 −9 , a check that caught a sign inversion during development (the paper’s Method section). Provenance-stamped JSONs (model, seed, commit, dtype, pooling, max_length) are released with the code. 10 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models B The Pre-Registered Gate A pre-registered floor test, reported in full. Detection is cheap, but is its absolute margin over trivial baselines real? We committed the rule before any experiment ran. Ablated ICS must beat the maximum trivial floor with a paired-bootstrap 95% CI excluding zero, BH-corrected, in at least 12 of 20 domains. With the full floor pool (this section’s complete, corrected set of trivial baselines), including the corrected SST-2 sentiment direction, the maximum floor in 14/20 domains, ICS clears it in 11/20, just short of the pre-registered bar, an honest limit on the absolute detector we report rather than bury (the deployable claims, namely selection and the guard comparison, do not rest on it). Mean ICS is 0.952 against a mean per-domain maximum floor of 0.925 (Table 9). The strongest single floor arm is the SST-2 sentiment direction (mean 0.914, the maximum in 14/20 domains), with TF-IDF at 0.896. Table 9 The pre-registered gate, outcome-ablated (8 of 20 domains, full table in §C). Floor is the per-domain maximum over trivial baselines, and an SST-2 sentiment direction fit on movie reviews is that maximum in 14/20. The rule, committed before any experiment, needs ∆ > 0 with a BH-corrected paired-bootstrap CI excluding zero in≥12/20. It reaches 11/20. DomainnICSMax. trivial floor(Which floor)∆Gate Data Act1000.9640.894SST2+0.070✓ Policy: Google1000.9700.919SST2+0.051✓ HIPAA1000.9830.941SST2+0.042✓ EU AI Act1000.9860.938SST2+0.048✓ Fin. crypto1000.9970.985SST2+0.012– GDPR1000.8630.862SST2+0.001– Edu: online720.9970.999SST2−0.003– Edu: discrim.160.8750.887TFIDF−0.012– Mean, 20 domains0.9520.925+0.02711/20 The verdict is 11/20 rather than 13/20 for two reasons that are themselves findings about evaluation practice. An earlier computation silently dropped the sentiment floor through a name-prefix filter, and one domain’s pass rested on a floating-point BH threshold (0.03×20/12 = 0.0499...) that fails under exact arithmetic. We report all thresholds exactly, and at meta-analytic boundaries, representation error flips verdicts. Table 31’s cross-model gate column deliberately reuses a cheap floor (the pre-correction, partial baseline set), not this section’s full floor pool, to keep all twelve arms on an identical comparison. Its 13 for this same model is that cheap-floor figure, not a second disagreeing measurement of this section’s full-floor 11/20, and the table’s caption reconciles the two. The failure pattern is structured but not clean, and we state both halves. All eleven passes are full-size (n=100) mid-difficulty domains. Of the nine failures, three are small-n(n≤ 36), two are ceiling-saturated (floor≥ 0.98), and two lose their pass to the corrected sentiment floor closing within 0.002–0.012, but two (GDPR, foundational rights) fail atn=100against mid floors, so low power and high floors do not explain every miss. Where the margin is measurable at all, it is small (mean +0.038 over the sentiment direction). B.1 Full-Detail Tables Referenced from the Main Text Five tables from the paper’s What the Score Measures section, the paper’s Generalisation Beyond the Calibration Distribution section, and the paper’s Comparison with Deployed Guards section in full. Each is summarised in the main text and referenced from there; this section exists so the underlying numbers are checkable in full, not because a reader needs them to follow the argument. B.2 The Fixed-Scenario Counterfactual, in Full The sharpest fixed-scenario test fixes the scenario and flips the label with the rule, which no public benchmark does. We construct 208 such pairs across 8 regulatory domains, one real scenario, byte-identical across the two members, paired with a permissive rule (compliant) and a prohibitive one (violation), condition-based and length-matched so that no lexical polarity survives (scenario-only and cheap rule-cue baselines sit at 0.49 and 0.71). On this set the deployed ICS direction reads the flip at chance (0.502, within-pair 0.587), though it separates OmniCompliance at 0.964 from the same layer. A direction fit on the flips recovers them at 0.736, and a logistic or small MLP probe on the same last-token activation reaches 0.874 and 0.887. As the paper’s What the Score Measures section notes, this design cannot distinguish genuine rule-scenario reading from a probe that has merely learned rule polarity, since rule polarity alone 11 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 10 The no-rule condition (the paper’s rule-is-optional table) replicated on four further models, across two families. Llama holds fully (0/20 significant drops on both arms), Qwen shows the same small hedge on both models, an architecture-consistent pattern, not a single-checkpoint artifact. ModelCorrectNo Rule∆Sig. Drops (Gains) Llama-3.2-1B (main text)0.95190.9549+0.0030/20 Llama-3.2-3B0.95800.9565 −0.00150/20 Llama-3.1-8B0.96240.9604 −0.00200/20 (1) Qwen2.5-7B0.96310.9604 −0.00272/20 (0) Qwen3-8B0.95280.9493 −0.00352/20 (1) predicts the label throughout this set, so these fit-on-the-flips numbers are an upper bound on relational reading, not proof of it, resolved by the crossed benchmark of §B.3. The chance-level readout of the deployed (not fit-on-the-flips) direction replicates across model scale and family regardless of that caveat, since the deployed direction never sees the flips. Llama-3.1-8B-Instruct reads the identical flip set at 0.504 and Qwen3-8B at 0.489, both indistinguishable from the 1B arm’s 0.502, while the same fit-on-the-flips recovery holds on both. Logistic and MLP probes on Llama-3.1-8B reach 0.943 and 0.943, and on Qwen3-8B 0.905 and 0.911. The deployed-detector failure is class-wide. On this set the deployed probe (0.502) and both fixed-taxonomy guards, which never receive the rule, sit at chance (Llama Guard 3 0.497, Qwen3Guard 0.480). Only an LLM judge given the rule reads it, and only partly (0.710, within-pair 0.865). B.3 The Crossed Rule-Scenario Benchmark, in Full Construction. Eight domains (ccpa, cybersecurity_mitre_attack, data_act, eu_ai_act, finance_crypto, gdpr, hipaa, policy_openai) each contribute 25 templates, one per regulatory axis (numeric deadlines, exemption scope, consent basis, jurisdiction, actor type, purpose limitation, and legacy-versus-new-regime conditions), for 200 templates and 800 rows. Each template supplies a rule with two category-conditioned thresholds and two scenarios that differ only in which category applies, at a shared numeric value strictly between the two thresholds, so the crossed 2×2 design (rule R0/R1 by scenario Sa/Sb) is guaranteed by construction, and which of R0/R1 is the “natural” versus “swapped” threshold assignment is independently coin-flipped per template to remove any global authoring bias. Because each rule text and each scenario text appears exactly twice in the set, once with each label, rule-only and scenario-only prediction is at chance not merely empirically but by construction, confirmed by the TF-IDF self-check (rule-only, scenario-only, and rule+scenario concatenated all score 0.500 AUROC, group-disjoint 5-fold CV by quadruple). Statistical protocol and quality control. The quadruple is the resampling and reporting unit throughout, never the individual cell, so every rate above (quadruple exact match, rule-flip accuracy, scenario-flip accuracy) is already a proportion over independent quadruples, and we report Wilson 95% confidence intervals on that proportion directly rather than a Monte Carlo bootstrap over the same units, the two agree in the large-sample limit and the closed form is exact rather than simulated. A predeclared per-domain BH correction (as used for the 20-domain OmniCompliance audit elsewhere in this paper) is not meaningful here, 25 quadruples per domain is too small a cell for a per-domain hypothesis test to be informative, so we report pooled statistics across all 8 domains instead of an underpowered per-domain breakdown. An independent pass manually inspected 45 of the 200 quadruples (180 rows), drawn by an unrelated random seed, for semantic sanity, not label correctness, which is mechanical and not at issue. Forty-one passed cleanly, four had cosmetic issues only, a duplicated article inserted by template substitution in two rule templates and a missing capital letter in two others, all four fixed directly in the released templates, and zero had a genuine construction flaw, no garbled text, no leftover placeholder token, and no scenario stating its own verdict. Detector matrix, primary model (Llama-3.2-1B-Instruct). Frozen ICS (OmniCompliance-calibrated), 0.501 AUROC, 0.0% quadruple exact match. Fit-on-crossed meandiff, logistic, and MLP probes at the same anchor layer, 0.502, 0.507, and 0.501 AUROC, all 0.0% exact match. Forced-choice judge (Llama-3.1-8B, single-token verdict), 0.507 AUROC. Llama Guard 3 and Qwen3Guard, 0.500 and 0.515 AUROC. LPG-4B, given the rule through its own documented custom-policy channel, 0.535 AUROC, 1.0% exact match. Rachmil et al.’s training-free whitening detector, their own precomputed guard categories and thresholds (github.com/FujitsuResearch/LLM-policy-violation-detection), scored zero-shot with no new calibration on our end, 0.498 AUROC, 0.0% exact match. Scale replication. Refitting the probe directly on the crossed data recovers a real, scale-dependent signal well short of solved. Llama-3.1-8B logistic reaches 0.616 AUROC and 7.0% exact match (Wilson 95% CI 4.2–11.4%). Qwen3-8B MLP reaches 0.670 AUROC and 11.0% exact match (CI 7.4–16.1%), the best cheap-detector result obtained, still a weak effect against 6.25% chance. Frozen ICS on both larger models stays near chance (0.510 and 0.540 AUROC). 12 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 11 The guards’ own turf, seven standard safety benchmarks (Macro F1×100, native decision threshold). The calibrate-once leave-one-distribution-out table is the appropriate cross-domain comparison; threshold-free guard AUROC on these safety sets is deferred to future work. Llama Guard 3 (8B), LPG-4B, and GLiGuard were not cached. ArmToxiChatOpenAIModAegis1.0Aegis2.0WildGuardSafeRLHFBeaverTAvg Generative / fine-tuned guard models (Macro F1×100, native threshold) Qwen3Guard-Gen (8B)89.182.578.285.790.087.178.084.4 WildGuard (7B)81.375.687.772.189.979.169.179.3 SIREN56.676.474.869.669.283.276.272.3 HarmBench cls (7B)58.573.045.947.758.084.468.962.3 Llama Guard 3 (1B)48.140.924.232.936.032.529.534.9 Reference (Macro F1×100) LLM judge (zero-shot, 8B)62.076.871.275.568.177.066.271.0 TF-IDF floor75.775.571.975.474.481.168.574.6 Positive control. The identical Llama-3.1-8B judge, given the same rule and scenario but prompted to reason step by step before a final verdict line rather than forced to a single token, is scored on a fixed, seeded 40-quadruple (160-row) subsample for compute reasons. It reaches 0.849 AUROC, 79.5% rule-flip accuracy, 78.2% scenario-flip accuracy, and 74.4% quadruple exact match (Wilson 95% CI 58.9–85.4%), confirming the benchmark is solvable by genuine reasoning and the chance-level results above reflect a readout limitation, not a degenerate design. B.4 A Worked Example The rule-blindness claim above is argued entirely in aggregate numbers. One concrete quadruple, drawn by an independent random seed rather than selected for effect, illustrates what a chance-level detector is actually missing. Domainfinance_crypto, templatet07. Rule R0 states a bank may apply simplified due diligence to a dormant retail account below $1,000 and to a dormant institutional account below $5,000. Rule R1 swaps the two limits. Scenario Sa describes a dormant retail savings account at Kestrel Bank with a $2,500 balance, and scenario Sb describes the same bank’s dormant institutional account, also at $2,500. R0×Sa is a violation ($2,500 exceeds the $1,000 retail limit), R0×Sb is compliant ($2,500 is under the $5,000 institutional limit), and R1 reverses both verdicts. Solving all four cells requires reading which account type the scenario names, finding that type’s specific threshold inside the rule, and comparing it against the one number the scenario states, exactly the composition step every chance-level detector in the paper’s crossed-benchmark table skips. B.5 LPG Violated-Policy-Removal Replication LPG’s own paper runs a violated-policy-removal probe on two OTHER guards (Qwen3-4B, DynaGuard-8B) and finds low, inconsistent flip-to-safe rates when the violated policy is removed, evidence those guards condition on positional and content priors rather than the supplied clauses. We run the analogous protocol on LPG-4B itself, using 200 violating cases from the crossed benchmark, each with its genuine violated rule, a matched permissive-counterpart rule for the identical scenario (the crossed design’s own R1 partner), and four synthetic irrelevant decoy policies drawn from other domains, under six conditions, full policy list, remove all policies, remove only the violated policy but keep the decoys, replace the violated policy with its permissive counterpart, shuffle policy order, and add further irrelevant policies. Of 182 cases LPG flags unsafe under the full policy list, removing the violated policy entirely, or removing it while leaving the decoys in place, flips the verdict to safe in only 44.5% of cases either way, so a majority of verdicts do not track whether the violated rule is even present. Substituting the exact permissive counterpart for the violated rule, the sharpest test since the scenario, decoys, and list position are all held fixed and only the substance of the applicable rule changes, flips the verdict in just 7.1% of cases. Verdict consistency under a pure reordering of the same policy list is 92.5%, and adding further irrelevant policies alongside an untouched violated rule causes a false flip away from unsafe in only 2.7% of cases, so LPG is not simply noise-sensitive to list length or order. It also correctly cites the violated rule’s own list position in its generated verdict 91–95% of the time across the full, shuffled, and irrelevant-added conditions. Read together, LPG reliably points at the right clause when asked which one applies, but the verdict itself barely depends on what that clause actually says, the same dissociation between citation and adjudication the crossed benchmark’s chance-level AUROC (the paper’s What the Score Measures section) already shows from a different angle. As the strictest check, the three pool directions are scored on seven never-pooled external benchmarks, two compliance and five safety, sharing no row, benchmark, or collection process with any pool (Table 13, all verified disjoint by activation- and text-identity). The pool-calibrated readout fires above chance on never-seen sources and the family structure mostly holds. The compliance half of that claim is now tested, and marginally holds: an external 13 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 12 Calibration transfer matrix (ICS AUROC↑). Each row’s direction fit on that distribution’s train, scored on each column’s disjoint test; bold = in-domain diagonal. Block means: diagonal .849, within-family .70, cross-family .53. Transfer holds within a family and collapses across it; no single calibration ships everywhere, and the best single calibrator (openaimod, off-diagonal mean .686) still collapses on the opposite family. Calibration N per Table 19. Compliance (test)Safety (test) Calibrated on↓OmniTfinTlawTmedToxCOAIAeg1Aeg2WildGPKUBvr Compliance-family directions OmniCompliance.975.471.401.595.681.612.696.723.589.630.528 TRIDENT-finance.439.837.689.905.452.572.489.544.380.496.525 TRIDENT-law.598.815.777.935.484.578.466.464.595.572.618 TRIDENT-med.741.788.676.916.498.594.502.551.525.551.607 Safety-family directions ToxicChat.812.257.384.268.845.723.556.584.668.615.698 OpenAIMod.823.654.458.636.826.898.685.737.779.606.641 Aegis1.875.391.394.510.865.769.833.798.660.729.710 Aegis2.828.360.335.600.718.592.700.780.560.684.727 WildGuard.622.253.423.446.771.701.683.622.864.834.749 PKU-SafeRLHF.654.291.346.295.759.658.665.694.864.800.848 BeaverTails.560.578.473.766.732.640.603.771.830.735.810 Table 13 Never-pooled external sets (AUROC↑), pool-calibrated ICS on seven benchmarks sharing no data with any pool. bold = aligned direction; the three row blocks are separate constructs and are not averaged. Prompt-harm sets are read by the safety direction, compliance at chance. Response sets (comply-vs-refuse) are read by the compliance direction along that axis, not regulatory compliance. Unfair-ToS is read above chance for the first time by the shared signal, not the compliance-specialised direction: a compliance direction transfers to unseen scenarios but not to a different compliance genre, and re-fitting on the fine-tuned models of Table 23 still reads these externals at chance (0.44–0.62). † ECtHR inconclusive (512-token truncation). External (never-pooled)ICS direction calibrated on All-11 PoolCompliance-OnlySafety-Only Legal/contractual document labelling (not rule-conditioned compliance) Unfair-ToS (LexGLUE)0.6120.5150.688 ECtHR rulings † 0.4200.4130.518 Is this prompt harmful? WildJailbreak (prompt)0.6870.4990.767 XSTest (prompt)0.8550.5020.931 JailbreakBench0.6240.4860.649 Did the response comply or refuse? LLM-LAT (response)0.9960.9970.980 SORRY-Bench (response)0.6910.7510.536 compliance benchmark is read above chance for the first time, but by the shared signal, not the compliance-specialised direction, since a compliance direction transfers to unseen scenarios but not to a different compliance genre, a ceiling that re-fitting on fine-tuned models does not repair. 14 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 14 The efficiency axis, stated conditionally. ICS’s zero-extra-pass property holds only when the scored text already passed through the monitored model with the anchor-layer activation retained; scoring external text costs one pass, like the other internal-representation probes. Only the generative guards need no hidden-state access at all. The∼0 figure is the readout alone (one dot product), excluding hook overhead, retention memory, and transfer, none of which were measured; treat it as a lower bound. ms/case measured in E-GUARDBENCH. GLiGuard did not run (n/a);∼-marked example counts are order-of-magnitude public estimates. TrainableLabelledExtra pass over the monitored modelReadout ArmParamsExamplesActivations RetainedExternal Textms/case ICS (ours)010 pairs/dist.01∼0 GradSafe016 pairs1 fwd + bwd1 fwd + bwd200 SIREN14M∼10 5 018.4 GLiGuard300M∼10 5 01n/a Llama Guard 3 (8B)8B10 4 –10 5 1 autoregressive1 autoregressive31.9 WildGuard (7B)7B∼87K1 autoregressive1 autoregressive31.8 Qwen3Guard-Gen (8B)8B∼10 6 1 autoregressive1 autoregressive45.3 10 −3 10 −2 10 −1 10 0 10 1 10 2 ms / case (log scale) 0.6 0.7 0.8 0.9 1.0 AUROC (OmniCompliance, in-domain) ICS SIREN WildGuard-7B Llama Guard 3-8B Qwen3Guard-Gen GradSafe A. Inference cost 10 1 10 2 10 3 10 4 10 5 10 6 labelled pairs to build (log scale) 0.6 0.7 0.8 0.9 1.0 ICS GradSafe WildGuard-7B Llama Guard 3-8B Qwen3Guard-Gen SIREN B. Data cost ICS (ours)receipted exactorder-of-magnitude est. Figure 2 Accuracy and estimated scoring cost. Left, inference cost (ms/case); right, data cost (labelled pairs, log scale), both vs. AUROC on outcome-ablated OmniCompliance. Open triangles mark public estimates not receipted; filled points are exact (Table 14). ICS gives competitive accuracy at its cost, beaten only by GradSafe at 200ms and by nothing at its own; a trained probe on the same activations is slightly more accurate still (the paper’s in-domain detector-comparison table, the paper’s Scope and Limitations section), so the advantage plotted here is cost, not a ceiling on accuracy. Cost estimates exclude activation-hook, retention, and transfer overhead (Table 14). 15 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models C Full Per-Domain Gate Table Table 15 gives all 20 domains of the pre-registered gate; the main text shows 8. FOUNDATION_RIGHTS is the one domain that both fails this gate atn=100against a mid-range floor and moves adversely under the rule-disjoint restriction (0.909→0.731, §E). Table 15 The pre-registered gate, all 20 domains. Every passing domain is n=100; failures are heterogeneous: small-n, ceiling-saturated, or the corrected sentiment floor closing to within 0.002–0.012. DomainnICSMax. trivial floor(Which floor)∆Gate Fin. cross-border630.9610.960SST2+0.002– Data Act1000.9640.894SST2+0.070✓ Policy: Google1000.9700.919SST2+0.051✓ SB-351000.9350.885SST2+0.050✓ Policy: OpenAI1000.9260.864SST2+0.063✓ Fin. AML/CTF1000.9650.918SST2+0.047✓ HIPAA1000.9830.941SST2+0.042✓ Policy: GitHub1000.9090.868SST2+0.042✓ Policy: Reddit1000.9210.861SST2+0.060✓ Fin. e-money1000.9950.970SST2+0.025✓ EU AI Act1000.9860.938SST2+0.048✓ Fin. crypto1000.9970.985SST2+0.012– CCPA1000.9860.946TFIDF+0.040✓ GDPR1000.8630.862SST2+0.001– Edu: online720.9970.999SST2−0.003– Policy: X1000.9360.928TFIDF+0.008– Cybersec.360.9970.989TFIDF+0.008– Found. rights1000.9090.902TFIDF+0.006– Edu: discrim.160.8750.887TFIDF−0.012– Edu: integrity360.9630.993TFIDF−0.030– Mean0.9520.925+0.02711/20 D Lexical Floors Across Benchmarks Four of seven public compliance benchmarks cannot adjudicate a probe. If the label is a property of the scenario rather than of the rule–scenario relation (the paper’s What the Score Measures section), it should be recoverable without a model at all. It is. The verdict-narration diagnosis of the paper’s Evaluation Validity section is not one dataset’s quirk: of seven public benchmarks tested, four are lexically degenerate, a fifth is borderline, and only two support measurement at all. On DynaBench [19], the test set is lexically shortcut-able: a policy-blind bag of words over the transcript alone reaches within-test AUROC 0.982 and F1 0.88 under cross-validation. We verified this is a property of the test set’s construction, not a same-split match to their calibrated result: fit on the benchmark’s own published training split and evaluated on their test rows, the same bag of words is near chance (F1 0.51–0.61, verification code released with the evaluation protocol), so the shortcut bounds the benchmark, and does not beat their training-free method head-to-head. ICS numbers on DynaBench are truncation-limited at 512 tokens; the floor-exceeds-probe ordering survives a truncation control. Table 16’s ICS figure for DynaBench (0.784) is measured under this gauntlet’s own uniform within-test protocol (same 5-fold cross-validation as every other benchmark here, for comparability across the fourteen-benchmark table); the stricter head-to-head against Rachmil et al.’s own published train/test split [13] is reported separately in the Related Work comparison against that method (§W). LPG-4B, run on the identical 543-row DynaBench test split, extends this to a policy-reasoning guard rather than an activation probe. Given the correct policy it reaches AUROC 0.681, given no policy at all AUROC 0.913, and given a policy for an unrelated case AUROC 0.687, so the correct policy performs no better than a wrong one and both perform far worse than no policy at all, the reverse of what a rule-conditioned guard should show. This tracks the same degeneracy diagnosis, a transcript-only bag-of-words already reaches 0.982 on this test set, so the policy block mostly displaces useful transcript content inside LPG’s 1024-token window rather than adding signal. On SafePyramid, TF-IDF over the rule text alone, never reading the conversation, reaches 0.90+ at every difficulty level, while the conversation alone (grouped CV) scores exactly 0.500: the label is predictable from which rule was asked about; on a surface-matched flip-pair core (identical rule text, opposite label), every lexical feature collapses to 0.5000. On AIReg-Bench, the one benchmark with human-expert labels, TF-IDF over the document alone reaches 0.957 (grouped 5-fold 0.975): expert annotation does not repair 16 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models outcome-first construction, because the shortcut is the document’s stylistic register, not annotator noise. CompliBench is borderline (0.879), and we additionally found its per-turn guideline metadata leaks the label outright (0.990; 823 of 830 violating turns carry the modified, violation-inducing directive); we report fair reconstructed floors instead. The pattern is not specific to our seven: on the third-party TRIDENT benchmark [20] a policy-blind TF-IDF scores 0.999 on the benchmark’s own texts (a request-vs-advisory form shortcut) and 0.910 on a cleaner response-based contrast, beating ICS (0.802) on 2 of 3 domains; ICS does not clear the floor there either. Prior work reports no lexical floor on any of these. Table 16 Fourteen benchmarks under one gauntlet. “Floor” is the strongest policy-blind lexical model (a bag of words that never reads the rule). A benchmark is degenerate when the floor is high and the probe does not significantly clear it. Compliance floors run 0.57–0.98 and 4/7 are degenerate (including the human-expert-labelled one); safety floors 0.77–0.90, 2/7 degenerate. The diagnosis is specific to compliance benchmarks, not to activation probing: where labels come from a mechanical verifier the floor collapses to 0.574. Safety-suite nulls are the ICS layer-selection budget (never 0.5). BenchmarkLabels fromn Base rate Scoring unitLex. floor ICS (ours) Budget null Verdict Compliance: does this scenario violate this rule? OmniComplianceenforcement text16–100.500 rule–scenario pair0.8960.9520.714 degenerate DynaBenchnarrated verdict543.492 transcript0.9820.7840.576 degenerate SafePyramidrule text alone77,755.344 conversation–rule0.90+0.8710.673 degenerate AIReg-Benchhuman experts120.367 documentation0.9570.9430.708 degenerate CompliBenchnarrated + metadata2,713.306 assistant turn0.8790.7000.549 borderline FlexBenchmoderation labels2,278.375 response0.8020.8320.603 clean IFEvalmechanical verifier541.481 prompt0.5740.6280.530 clean Safety: is this content harmful? (no rule field to ablate) ToxicChathuman annotation9,743.072 prompt0.9030.8840.576 degenerate PKU-SafeRLHFpreference labels10,000.519 prompt+response0.8950.9030.580 degenerate OpenAI Moderation taxonomy1,664.309 prompt0.8500.8930.560 clean Aegis 2.0human annotation10,000.509 prompt0.8360.8090.588 clean WildGuardhuman annotation1,725.437 prompt0.8260.9180.612 clean Aegis 1.0human annotation10,000.681 prompt0.8110.7520.538 clean BeaverTailshuman annotation10,000.582 prompt+response0.7660.7820.574 clean Compliance0.57–0.984/7 degen. Safety0.77–0.902/7 degen. Two benchmarks are clean, and they adjudicate against the probe. On IFEval (TF-IDF 0.574; labels from a mechanical verifier, not narrated verdicts), ICS beats the trivial form-counting floor in 0 of 7 categories, in every configuration tested. On FlexBench (TF-IDF 0.68–0.80), ICS shows a weak, CI-overlapping lead on the response track at strict severity (0.832 vs 0.802), inverts at loose severity, and on the prompt track the zero-supervision refusal direction matches supervised TF-IDF outright. Where the floor is honest but high (DynaBench, SafePyramid, CompliBench), it exceeds the probe, the inverse of the OmniCompliance ordering. The pattern across seven benchmarks: the probe’s apparent advantage is largest exactly where the benchmark is least able to adjudicate it. D.1 Guard Models on the Same External and Degeneracy Benchmarks Tables 13 and 16 report ICS on thirteen benchmarks that, until this pass, had no deployed-guard comparison. We close that gap: the same five guards benchmarked elsewhere in this paper (Llama Guard 3 (1B), WildGuard, HarmBench-cls, Qwen3Guard-Gen (8B), and an 8B zero-shot LLM judge), scored zero-shot (native mode, no threshold fitting) on the identical texts and labels (Tables 17 and 18). Benchmarks whose larger class exceeded 1,500 examples were stratified-capped at 1,500 per class (seed 42) for tractability; capped benchmarks and the number of rows dropped: wildjailbreak (500), llmlat (6,895), sorry_bench (332), unfair_tos (6,716), ecthr_a (8,257), safepyramid (21,000), complibench (383), flexbench (278); jbb, xstest, dynabench, aireg, and ifeval were scored in full. The LLM judge splits along task type, not benchmark family. A pattern spans both tables that neither caption states alone: the 8B zero-shot judge is the strongest or a competitive guard on every classic prompt-harm set (0.864 wildjailbreak, 0.785 jbb, 0.843 xstest) but sits at or below chance on nearly every regulatory or legal-compliance benchmark (0.179 aireg, 0.428 complibench, 0.419 ecthr_a, 0.489 unfair_tos, 0.397 sorry_bench). The judge is prompted with a single fixed question – is the request above harmful or unsafe – which is the right question for a jailbreak benchmark and the wrong one for a rule-adherence benchmark: nothing in that prompt asks whether a specific rule was satisfied. This is not a new, unrelated failure mode; it is the paper’s central rule-blindness finding (the paper’s 17 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 17 Guard models on the never-pooled external OOD set (AUROC↑), closing the gap in Table 13. bold = best guard per row. On the two response-harm sets, HarmBench-cls (scores the full request+response against a harm behaviour) reads correctly (> 0.999, 0.775); WildGuard and the LLM-judge score below chance (0.331/0.255, 0.129/0.397), verified not a label bug, since their native templates ask about a prompt or "the request," not whether a response fulfilled it, the same prompt-vs-response sensitivity as the paper’s Comparison with Deployed Guards section. † ECtHR inconclusive (as Table 13). External (never-pooled)ICS (all-11 pool) Llama Guard 3 (1B) WildGuard HarmBench-cls Qwen3Guard-Gen LLM-judge (8B) Unfair-ToS (LexGLUE)0.6120.6190.6260.5170.3330.489 ECtHR rulings † 0.4200.4790.4870.4850.5260.419 WildJailbreak (prompt)0.6870.7990.9720.6460.9450.864 XSTest (prompt)0.8550.9170.9900.8580.9730.843 LLM-LAT (response)0.996> 0.9990.331> 0.9990.9160.129 SORRY-Bench (response)0.6910.6710.2550.7750.3200.397 JailbreakBench0.6240.9410.9660.8350.9420.785 Table 18 Guard models on the six degeneracy-audit benchmarks without a prior guard comparison (AUROC↑), closing the gap in Table 16. bold = best guard per row. Every guard sits near chance on DynaBench, SafePyramid, and CompliBench – the same three domains where §D finds ICS’s own apparent edge is largest exactly where the honest floor is high, so this is not a case of guards underperforming ICS by an unfair comparison: no detector in this class, first-party or deployed, reads these three domains well. BenchmarkICS (Table 16)Llama Guard 3 (1B)WildGuardHarmBench-clsQwen3Guard-GenLLM-judge (8B) DynaBench0.7840.5710.5540.5740.5270.516 SafePyramid0.8710.5060.5220.5090.5030.491 AIReg-Bench0.9430.6030.5550.4450.4450.179 CompliBench0.7000.5180.5220.4910.5060.428 FlexBench0.8320.8490.7420.8140.8000.677 IFEval0.6280.5590.5600.5490.5360.522 What the Score Measures section) appearing again in a different guise, on benchmarks outside the pool used to establish it. The selection floor is not a compliance quirk, and it is not outgrown. The same budget-matched null inflates random directions on unrelated probing settings (Azaria–Mitchell true/false and SST-2) to 0.50–0.65 against no-selection controls at 0.49–0.51, so the correction applies to layer-selected probes generally. Two qualifications: the magnitude is setting-specific (compliance’s 0.71 is the highest we measured; one topic is essentially uninflated), so a floor must be measured against the arm’s own search, not transferred; and it does not decay with data, since a predicted c/ √ nfit is refuted (r 2 =−23; a plateau fits atr 2 = 0.96, floor 0.09), with SST-2 still at 0.586 at 2000 pairs. Details and the transfer table are in this section. In-domain: calibrate and test on the same distribution calibration pairs from D ICS direction test: held-out D Calibrate-once:D withheld from calibration, met only at test calibration pairs from A,B,C,... (̸= D) one ICS direction test: D (unseen) Figure 3 Calibration vs. test data, and the calibrate-once guard test. ICS is built from a small labelled calibration set (pairs give the direction; a validation slice fixes layer and threshold) and scored on a disjoint test set. Left: per-distribution, with calibration and test from the same D. Right: calibrate-once, where D is withheld from calibration and appears only at test, so the direction meets it unseen, as a deployed guard meets a new input (the paper’s calibrate-once leave-one-distribution-out table). 18 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models 0.00.10.20.30.4 available headroom (oracle − random) 0.0 0.1 0.2 0.3 0.4 selection gain (guided − random) slope 0.79 R 2 =0.94 Llama-1B Llama-3B Llama-8B Gemma-9B Mistral-7B Qwen-7B Figure 4 Selection recovers a near-constant fraction of the available headroom, on every model. Each point is one (model, domain);xis what the oracle could win over random,yis what ICS-guided selection actually wins. The dashed fit has slope 0.79 through the origin (R 2 = 0.94, 120 points); the dotted line is the oracle (y=x). Larger models cluster near the origin because their room is small, not because selection weakens; they sit on the same line. E Split Leakage Audit Splits are case-level; 68% of test cases share asource_rulewith the train slice. Restricting test to rule-disjoint cases leaves AUROC unchanged (0.962 vs. 0.956, restricted≥ full in 13/19 domains), so rule overlap does not inflate the headline. One adverse domain (foundational rights,0.909→ 0.731); power is weakest exactly where leakage is heaviest. The rule-disjoint restriction is computable in 19/20 domains; onlyfoundation_rightsmoves adversely (0.909 to 0.731). This domain’s rule overlap (84.5% of its test cases) is above the dataset mean, so restriction leaves only 31 residual cases (18 violating, 13 compliant) against 200 in the full test set – the smallest surviving residual of any domain whose full-set ICS score is not already near ceiling. Eight other domains have a comparably small residual (n≤ 30); all eight have a full-set ICS score of 0.92 or above and land at a restricted AUROC of exactly 1.0 (a ninth, EDU_DISCRIMINATION_US_EDU_DEPT, restricts to a single case and is not computable at all – the source of this section’s “computable in 19/20 domains”). FOUNDATION_RIGHTS, whose full-set score (0.909) is the third-lowest of all 20 domains (after GDPR’s 0.863 and EDU_DISCRIMINATION_US_EDU_DEPT’s 0.875, the latter itself one of the small-ngate failures, not this section’s concern), is the first domain among these small-residual cases where the signal is weak enough at full scale to show through rather than round up to ceiling. GDPR has a lower full-set score still (0.863) but a far lower rule-overlap fraction (51.5%), so its residual stays large (97 cases) and its restricted AUROC is stable (0.863→0.880). The adverse swing is therefore a small-nconsequence of an already-weak full-set signal meeting unusually high rule overlap, not evidence of a distinct failure mode; it does not explain why the full-set signal is weak on this domain in the first place (the paper’s Scope and Limitations section). F Cross-Family Generalisation (Leave-One-Family-Out) The main leave-one-distribution-out test (the paper’s Generalisation Beyond the Calibration Distribution section) holds out one distribution but keeps its family siblings in the calibration pool (splits, Table 19). The stricter test withholds an entire family. Compliance→Safety (fit on the four compliance distributions only, test on the seven safety benchmarks): mean AUROC 0.552, every target in 0.50–0.60. Safety→Compliance (fit on the seven safety distributions only, test on OmniCompliance+TRIDENT): mean 0.476: OmniCompliance survives (0.784, its generic violation signal is broad) but the three TRIDENT domains invert below chance (finance 0.294, law 0.314). In-domain ceilings stay high (0.78–0.98), so the collapse is purely the cross-family tax, not weak targets. Zero target-family test rows appear in the train-family pool (asserted). The calibrate-once direction is therefore domain-type-specific: it must be calibrated on the family it will monitor. 19 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 19 Cross-validation splits. Per-distribution train/val/test counts (balanced 50/50, seed 42, group-disjoint) pooled for the calibrate-once (the paper’s calibrate-once leave-one-distribution-out table) and transfer-matrix (Table 12) experiments. DistributionFamilyTrainValidationTest OmniCompliancecompliance320320160 TRIDENT-financecompliance240240122 TRIDENT-lawcompliance210208110 TRIDENT-medcompliance16016084 ToxicChatsafety320320160 OpenAIModsafety320320160 Aegis1safety320320157 Aegis2safety320320160 WildGuardsafety320320160 PKU-SafeRLHFsafety320320160 BeaverTailssafety320320160 G Applying Guards on Input vs. Response Table 20 Input vs. response is a real knob, and response wins. Each generative guard applied to the case as an input to screen vs. a response to judge (bold = better mode); response helps almost everywhere, decisively for WildGuard (+0.32 mean AUROC, near-chance on TRIDENT input). † HarmBench is natively a response classifier. ICS has no input/response slot, reading the case activation directly. This table is the reference for guard mode; the headline tables now consistently credit each guard its stronger, response mode recorded here. GuardModeOmniCompliance AUROCAUPRCF1 (native)F1 (recal.)TRIDENT-finTRIDENT-lawTRIDENT-med Llama Guard 3 (1B)input0.6890.6980.3120.6190.8070.7410.903 response0.7490.7580.2450.6810.8370.7780.896 WildGuard (7B)input0.7850.7870.1130.7020.5120.5670.599 response0.8750.8860.1280.8140.8740.7810.967 HarmBench cls (7B) † input0.7200.7240.0720.6580.8010.8080.890 response0.7350.7380.1050.6730.8120.8080.894 Qwen3Guard-Gen (8B)input0.8850.8840.1460.8070.7760.6740.821 response0.9080.9050.1910.8330.8540.7740.946 A compliance case is a Rule plus a Scenario. A generative moderation guard can read it as an input to screen (Scenario in the user slot) or as a response to judge (Scenario in the assistant slot, Rule as policy); the two are distinct tasks for a guard trained on live prompts or outputs. We run each generative guard (Llama Guard 3, WildGuard, HarmBench, Qwen3Guard) in both modes on OmniCompliance and the three TRIDENT domains, scoring each mode’s AUROC/AUPRC and native/recalibrated F1 separately on the same seed-42 test splits; the head-to-head in the paper’s Generalisation Beyond the Calibration Distribution section credits each guard its stronger mode. Table 20 reports the per-benchmark AUROC in both modes. Response mode is the stronger role for all four guards on compliance, and the gap is decisive for WildGuard: its prompt-slot head is near-chance on TRIDENT (0.51–0.60) and only the response head recovers (+0.32AUROC on average). This is exactly why a single fixed mapping (as in the paper’s Generalisation Beyond the Calibration Distribution section’s harness, which sends WildGuard to the prompt slot) can understate a guard, because it may be reading its weaker role. We therefore report each guard’s stronger mode. 20 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 21 All metrics on outcome-ablated OmniCompliance (mean over 20 domains). At native harm-tuned threshold guards rarely fire on compliance (recall 0.08–0.22); recalibrating recovers most of it (WildGuard 0.11→ 0.70, Qwen3Guard 0.15→ 0.81), still below ICS’s val-calibrated F1 of 0.904 (a different protocol from the paper’s in-domain detector-comparison table’s ten-pair-budget 0.897, not a second measurement). ICS is best-or-tied on 5 of 8 metrics (AUROC, F1, Accuracy, Recall, FPR@95) at zero parameters, though the MLP probe on the same activations edges it out; LPG-4B leads on AUPRC (0.947) and Precision (0.923), and Qwen3Guard-Gen on Specificity (0.998). SIREN’s params count its 22MB probe head only, riding the same frozen backbone ICS does. Guard AUROCs credit each guard its stronger, response mode throughout (Table 20), recomputed and verified per-domain-then-mean from the same raw scores, released with the evaluation protocol. ArmParamsAUROCAUPRCF1AccPrecRecFPR@95Spec ICS (ours)0 / 10 pairs0.9520.9400.9040.9030.8950.9150.1640.891 SIREN14M0.6970.6920.2820.5600.6510.2210.7780.900 GLiGuard300M0.7030.6920.2920.5630.7400.1900.7850.935 MLP probe (same activations) ∼0.1M0.9660.9630.9160.9160.9170.9150.1330.917 Llama Guard 3 (1B)1B0.7490.6970.3180.5770.8300.2050.8600.949 Llama Guard 3 (8B)8B0.6430.6800.2000.5530.8130.1220.9030.984 Qwen3Guard-Gen (8B)8B0.9080.8850.1450.5400.8880.0810.4820.998 WildGuard (7B)7B0.8750.7870.1130.5270.8740.0640.6550.990 HarmBench cls (7B)7B0.7350.7380.1050.5300.8370.0630.7630.997 Latent Policy Guard (4B)4B0.9430.9470.8470.8610.9230.7850.2420.936 LLM judge (zero-shot, 8B)8B0.9120.9180.8400.8300.8070.8860.3740.775 TF-IDF floor—0.8960.9000.8220.8210.8210.8260.4260.815 H Matched-Threshold Calibration and Cross-Model Adaptation H.1 Training-free comparators and model-specific recalibration, in full The training-free story generalises, and ICS is cheapest. The result is not idiosyncratic to difference-of-means: GradSafe [21], a gradient-based training-free detector, also beats every deployed guard on compliance (AUROC 0.908) once re-referenced with sixteen compliance pairs, and collapses to chance (0.564) under its native safety reference, the same failure the safety-trained guards show, further evidence that compliance and safety are distinct. ICS is the cheapest of these at the top of the range, reaching 0.952 for one dot product on activations already computed (∼0 ms) where GradSafe needs a forward + backward pass (200 ms) and a deployed guard a full autoregressive one, so on either cost axis it sits at the frontier, beaten only by GradSafe at 200ms and by nothing at its own (Figure 2, Table 14). At a fraction of the cost, and it adapts. ICS adds no trained parameters, and no extra forward pass when the scored text has already gone through the monitored model with its anchor-layer activation retained; scoring externally supplied text costs one pass, like any first-party probe (Table 14). It reaches full accuracy from 20 pairs (§O.1) against the 10 4 –10 5 labelled examples a guard consumes. It is also recalibrable on the monitored model: reading the monitored model’s own activations, it re-fits in seconds on cached activations when that model is fine-tuned, where a shipped guard is a separate, frozen network. A fresh ten-pair refit reads compliance at AUROC 0.948–0.962 on nine models spanning three architectures and multiple fine-tunes, all above the strongest deployed guard (LPG-4B 0.943); a verdict-tuned Llama-3.2-1B likewise keeps AUROC 0.952–0.957, within 0.007 of base across the tested fine-tunes (Table 23). The paper’s Comparison with Deployed Guards section reports two supporting checks in summary; the full tables are here. The first isolates whether ICS’s edge over deployed guards is a ranking advantage or an artifact of guards running at an out-of-domain threshold. Every guard is given ICS’s own target-domain threshold budget (5-fold within-domain, macro over 20 domains, no weight fine-tuning); the native-F1 collapse turns out to be a threshold artifact; a ten-pair re-threshold lifts every guard to F1 0.54–0.80 (Table 22). ICS’s budget-free AUROC still clears every guard, so it leads every threshold metric even after the guards are given the same budget it gets. The second isolates the adaptation claim itself, that ICS re-fits to a fine-tuned model at zero cost while a shipped guard is a separate, frozen network. A fresh ten-pair refit is run on nine models spanning three architectures, five variants of Llama-3.1-8B (base plus medical, code, instruction, and safety fine-tunes) and the Gemma-3-12B and Qwen3-8B bases, each also with a finance fine-tune (Table 23). Every refit numerically exceeds the strongest deployed guard (LPG-4B, AUROC 0.943), which cannot adapt at all, though the margin is small on some arms (detailed below). 21 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 22 Matched threshold calibration, the edge is ranking, not threshold access. Every guard given ICS’s target-domain threshold budget (5-fold within-domain, macro over 20 domains, no weight fine-tuning). The native-F1 collapse is a threshold artifact, a ten-pair re-threshold lifts guards to F1 0.54–0.80, but ICS’s budget-free AUROC clears every guard, so it still leads every threshold metric at matched budget. AUROC for the four generative guards matches Table 20’s response mode, verified independently from the same raw per-case scores; ICS and the LLM judge have no input/response slot and are scored directly. AUROCF1 at calibration budgetFPR@95TPR@10 Arm(rank, budget-free)Native10-pairFull-Data(full-data)(full-data) ICS (ours)0.9520.8970.8710.8950.1240.871 Qwen3Guard-Gen (8B)0.9080.1580.7660.8040.3950.697 LLM-judge (zero-shot, 8B)0.9010.8310.7480.8170.3300.788 WildGuard (7B)0.8750.1240.6910.7150.5930.527 HarmBench cls (7B)0.7350.0940.5910.6560.6820.459 Llama Guard 3 (1B)0.7490.2650.5420.5800.7660.330 Table 23 ICS re-fits across models and families, a guard cannot. Each ICS row is a fresh ten-pair refit on that model’s own activations, anchored and thresholded on disjoint validation; the guard column is identical for every protected model since it classifies the input case, not the activations. Across four Llama-3.1-8B fine-tunes ICS holds within 0.007 of base; both cross-family bases hold within 0.005 on their finance fine-tunes. Frozen-guard AUROCs credit each guard its stronger, response mode (Table 20). Every refit numerically exceeds the strongest deployed guard (LPG-4B 0.943), though the margin is as small as 0.005 without a paired CI on that gap, read as parity-or-better, not a large win. Protected modelFamilyICS AUROC↑∆ vs. base ICS refit on each model’s own activations (0 params, 10 pairs) Llama-3.1-8B-InstructLlama0.962— + medical-SFTLlama0.958 −0.004 + code-SFTLlama0.9620.000 + dolly-SFTLlama0.956 −0.006 + safe-alignLlama0.9620.000 Gemma-3-12B-itGemma0.953— + finance-SFTGemma0.948 −0.005 Qwen3-8BQwen0.953— + finance-SFTQwen0.951 −0.002 Frozen guards (model-independent, cannot refit at all) LPG-4B—0.943fixed LLM judge (zero-shot, 8B)—0.912fixed Qwen3Guard-Gen (8B)—0.908fixed WildGuard (7B)—0.875fixed HarmBench cls (7B)—0.735fixed GLiGuard (300M)—0.703fixed SIREN (14M probe)—0.697fixed Llama Guard 3 (1B)—0.749fixed 22 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models I Rule Ablation, Visualized The rule-ablation result of the paper’s What the Score Measures section, drawn rather than tabulated. Figure 5 puts ICS, the TF-IDF floor, and the budget-matched null on one axis across all five rule conditions, so the flat profile under rule corruption and the single collapse under scenario removal can be read off at a glance. correctno rule shuffled rule wrong-domain rule rule only (no scenario) 0.5 0.6 0.7 0.8 0.9 1.0 test AUROC ICS budget null Deleting the rule costs nothing (0/20 drops); removing the scenario collapses it (19/20) ICS TF-IDF Figure 5 The rule is optional; the scenario carries the label. ICS (blue) is unchanged whether the rule is correct, deleted, shuffled within-domain, or replaced by a different domain’s rule (0/20 significant drops each); it collapses only when the scenario is removed (rule-only, below its own null in 19/20). TF-IDF (grey) behaves the same, improving when the rule is removed. Red bars are the ICS budget-matched null. I.1 Benign Inputs (Over-Flagging) Table 24 Benign inputs (over-flagging). Fraction of clearly-benign inputs flagged as violation, lower is better (N =250/1319/1000). ICS is worse than well-calibrated guards on easy benign (XSTest 0.44, Alpaca 0.18) but lower on adversarially-benign OR-Bench (0.327) than every guard that fires on compliance at all. Three arms score numerically lower still (Llama Guard 3 0.000, HarmBench 0.005, Llama Guard 3 8B 0.193), but as refusal classifiers that essentially never fire on OR-Bench’s style regardless of content, a floor artifact, not better calibration; read this beside detection performance (Table 21). ArmXSTest-safe FPROR-Bench-hard FPRAlpaca/Dolly FPR ICS (ours)0.4370.3270.182 MLP probe (same activations)0.3930.3550.224 Qwen3Guard-Gen (8B)0.0560.5680.001 LLM judge (zero-shot, 8B)0.1760.5330.025 WildGuard (7B)0.0120.7510.003 SIREN0.8400.8360.409 Llama Guard 3 (1B)0.0000.0000.000 HarmBench cls (7B)0.0280.0050.000 GLiGuard0.1720.5600.016 Llama Guard 3 (8B)0.0320.1930.002 TF-IDF floor0.4060.4060.425 23 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models J Selection Nulls: the Budget-Matched Detection Floor Layer selection alone lifts a random direction to 0.71. Layer selection is not free. Shuffling labels within pairs leaves a random-signedO(1/ √ n)component of the true direction in every layer; taking the argmax over layers on a validation slice selects that leak. A random unit direction pushed through ICS’s own pipeline (argmax over 17 layers) scores 0.7135 on average, not 0.5, and a random direction given the refusal direction’s larger budget scores 0.7747; three independent experiments agree on this floor to within 0.004. ICS is accordingly reported against this budget-matched null throughout, not against chance: AUROC 0.952 against a null of 0.714. We do not visualise or tabulate this margin against other arms’ margins, since each arm faces a different search space and a different null construction, and a longer margin does not identify a better method; the full per-arm breakdown is released with the evaluation protocol. Three consequences. First, absolute probe numbers are inflated: “ICS achieves 0.95” is a statement made against a floor of 0.71, not 0.50. Second, the refusal direction, fit with zero compliance supervision, retains a genuine but small margin (+0.089; tail-significant against matched random directions in only 6–8 of 20 domains). Its 0.863 is therefore not evidence that generic harmfulness carries most of the compliance signal: against budget-matched nulls, harmfulness accounts for roughly 37% of ICS’s honest margin, and per-domain calibration for the majority. Third, the zero-selection arms show the largest margins over their own nulls. We resist the obvious reading. Those arms search nothing, so their null is chance, while a layer-selected direction is scored against 0.714; the two margins are differences from different reference points and a larger one does not identify a better method. What the comparison does establish is narrower and still uncomfortable: the activation probe’s advantage over a bag of words is not visible once each is read against its own budget, so the case for ICS rests on cost and first-party access rather than on a margin the null structure cannot support. The surface-residualized score survives the same correction: after projecting out lexical content, sentiment, length, and entropy, ICS scores 0.771 against a measured budget-matched null of 0.587 (residualization removes part, not all, of the selection leak), above it in 18/20 domains. The activations carry some signal beyond the text surface; the practical margin over a surface model is what fails (§O). K Selection-Null Transfer Beyond Compliance The budget-matched null construction used throughout this paper (§J) is not a compliance-specific artifact: the same layer-selection procedure applied to unrelated published probing settings (truthfulness, sentiment) inflates a random direction to 0.50–0.65 against a no-selection control of 0.49–0.51, confirming the null is measuring a real property of layer selection rather than something particular to this task. Compliance’s 0.714 is the highest floor measured of these settings. Full per-setting breakdown, then-decay check, and null-generation detail are released with the evaluation protocol. L Content-Selection: Generated Responses and the Judged Regulatory Task The main text (the paper’s Score-Guided Candidate Selection section) establishes verdict-selection on a classification corpus. Two extensions test whether the gain survives when the candidates are substantively different generated responses rather than menu labels (Table 25). It also works on generated content, not just verdicts.The deployment case (generateksubstantively different responses, return the most compliant) we test on IFEval, whose mechanicalinstruction_followingverifier gives objective per-response ground truth (no judge). Over all 541 prompts, ICS-guided content-selection raises the strictly- verified pass rate by+5.2p over random (CI[+2.4, +7.8],p=1/2000) and+6.1p over first-pick, replicating on a held-out block (+4.5p,p=0.036); on the 237 prompts where candidates actually vary, the gain is+11.8p. The candidates are diverse (mean pairwise word-dissimilarity 0.91), so this is content-selection, not relabelling. The effect is modest, 22.6% of oracle headroom, ceilinged by a within-prompt detection AUROC of 0.62, but positive, powered, and mechanically verified. It is not a 1B artifact: content-selection raises the IFEval verified pass rate on 4 of 5 models (+2.4to+5.2p), each scaling with within-prompt detection AUROC (0.60–0.63); it is null only on Qwen2.5-7B, whose within-prompt detection collapses to chance (0.53). The gain is gated by whether ICS separates a prompt’s compliant candidates from its violating ones, not by headroom, since content detection is the harder problem. And on the regulatory domain itself, judged. IFEval measures format compliance; the paper’s subject is regulatory compliance, where no mechanical verifier exists. We therefore generate advisory responses to violating scenarios (six domains,k=5) and score them with an LLM judge. The selector is the frozen receipted ICS direction, 24 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 25 Content-selection survives on generated text, both mechanically-verified and judged. IFEval, ICS-guided pick of k responses vs. the instruction-following verifier (CI [+2.4, +7.8]). Regulatory, frozen ICS direction on LLM-judged advisory responses (CI [+8.8, +14.2], all six domains significant). TasknVs. RandomVs. FirstOracle Headroom IFEval (mechanical)541+5.2p+6.1p22.6% Regulatory (LLM judge)6 dom., k=5+11.5p—43% never fit on judge labels or generated text, so there is no selector–judge circularity. Guided selection raises the judged compliance rate by+11.5p pooled (CI[+8.8, +14.2],p=0.001, all six domains significant), capturing 43% of oracle headroom (+16.7p on varying prompts). A coherence control rules out quality-selection: judged compliance is uncorrelated with length (−0.06) and ICS is if anything negatively length-correlated, and restricting to coherent candidates leaves the gain unchanged. The judge is a fallible oracle, so this bounds rather than proves; but across a mechanical verifier and a judge, on format and on regulation, content-selection is a real, modest, deployable gain. This claim, and the separate rule-counterfactual claim of the paper’s What the Score Measures section, are both cross-checked against three independent frontier judges in §M. M Independent Frontier-Model Validation of the LLM-Judge Claims Both LLM-judge claims above rest on a single zero-shot judge, Llama-3.1-8B-Instruct, so we cross-check both against three independent judges (Claude Haiku 4.5, GPT-4o, Gemini 2.5 Flash) under exact-prompt replication of the local judge’s own protocol, zero-shot and one-shot (Table 26, Figure 6). All three judges preserve the direction of the content-selection effect (guided>random>first) but its magnitude varies by an order of magnitude across judges. Rule-counterfactual agreement is weaker and should be treated as judge-sensitive: only one configuration (GPT-4o, one-shot) closes most of the gap to the local judge; the other five stay well below it. This rule-flip metric is measured on the fixed-scenario counterfactual set (§B.2), not the confound-free crossed benchmark, and that set has its own known confound, rule polarity alone predicts the label (§B.3); a judge that under-performs the local model here is not shown to be more rule-blind, only that it is more or less exploitable by the same polarity cue. The reading we do not draw is that three low frontier scores make the local judge’s 0.865 the confirmed number: with only four judges and a confounded task, this cross-check establishes that the direction of the content-selection effect is judge-general, and flags rule-counterfactual magnitude as judge-sensitive and requiring the unconfounded crossed benchmark (§B.3), not the local judge alone, as the basis for the paper’s rule-blindness claim. Full per-domain, per-shot breakdown and the earlier flagship-judge round are released with the evaluation protocol. 25 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 26 Judge sensitivity, summarised. Guided−random is the content-selection gap; rule-flip accuracy is within-pair accuracy on the 416-row rule-flip set. Full six-configuration table released with the evaluation protocol. Judge (shot)Guided−randomRule-flip acc.Unparseable Local judge+11.5p0.865— GPT-4o (zero)+14.3p0.462low GPT-4o (one)+9.1p0.58669/416 Claude Haiku (zero)+3.6p0.229low Gemini 2.5 (zero)+2.7p0.263low Claude Haiku 4.5 GPT-4o Gemini 2.5 Flash 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Guided − random (p) local judge +11.5p A. Content-selection Claude Haiku 4.5 GPT-4o Gemini 2.5 Flash 0.0 0.2 0.4 0.6 0.8 1.0 Within-pair accuracy local judge 0.865 B. Rule-counterfactual Zero-shotOne-shot Figure 6 Frontier judges mostly confirm the local judge’s weaknesses, not just its strengths. Left, content-selection guided-random gap (p) vs. the local judge’s +11.5p (dashed). Right, rule-counterfactual within-pair accuracy vs. the local judge’s 0.865 (dashed). N Adversarial Robustness: Verdict- vs. Content-Selection The compliant-register suffix that flips the monitor in §O is applied to the selection pipeline (1B, 20 domains, k=5; the selector scores the case, so a suffix that flips the case verdict also flips the selector). Ground truth is the true verdict, since the cosmetic suffix does not change actual compliance. Table 27 reports captured headroom (guided−random)/(oracle−random) under no attack, a uniform suffix on allkcandidates, and a poison suffix on the truly-violating candidates only (the untrusted-generator / content-filtering threat model). The mechanism is the monitor attack, downstream: the suffix flips the case verdict VIOLATE→COMPLY on 70.8% of violating cases (81.5% among clean-flagged violations), so the selector’s selected-COMPLY rate on violating cases rises0.10→ 0.74and true pass on those cases falls0.90→ 0.26. Selection survives only in the 3/20 domains where the monitor also resists; 11/20 go negative. Because selection inherits the monitor’s case verdict, it has no floor above random once the monitor is defeated. Content-selection resists the same attack.Applied to the deployable content-selection lead (IFEval, 1B,k=5, official mechanical verifier; the poison design excludes the 9.4% of candidates whose strict verdict the suffix genuinely flips, so we measure only ICS being fooled by register into selecting a still-failing response), the attack moves the verified pass rate by−1.1p (CI[−0.033, +0.011], incl. 0,p≈0.36) and retains 79% of headroom (0.226→ 0.177); a uniform register shift does not reorder the ranking. The mechanism is measured: the suffix moves a response’s ICS by only+0.10versus+2.2for the case-text attack (∼22×smaller), because the scored last token follows a diverse ∼1000-token response that a13-token boilerplate tail cannot dominate, and right-truncation removes the suffix on 19% of candidates. Even a selective adversary’s harm is bounded at∼6.4p. The deployable use is robust for the same reason the monitor is fragile, last-token locality, but read the other way: scoring long, diverse content dilutes the local perturbation that dominates a short case text. 26 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 27 The attack defeats verdict -selection. Captured headroom collapses from 78.8% to the random floor under a poison suffix; guided true-pass drops to 0.552, essentially random (0.541). Verdict-selection inherits the monitor’s case verdict and fails wherever the monitor fails, but content-selection does not (below). ConditionHeadroom capturedGuided true-pass∆ vs. baseline (A) baseline (no attack)0.7880.872— (B) uniform suffix (all k)0.0950.581−0.291 (C) poison (violating cases only)0.0270.552−0.320 random-selection floor = 0.541; poison headroom CI [−0.014, +0.067] (includes 0); ∆ true-pass CI [−0.335,−0.305], p = 1/10000 (both B and C). But an adaptive white-box attack breaks it. The fixed suffix is non-adaptive. A GCG [18] suffix optimised directly against the ICS direction (white-box; 12 tokens, 40 steps; 20 stratified attackable IFEval prompts, 1B) is far stronger: it moves a response’s faithful ICS by+1.66versus+0.10for the fixed suffix (∼16×) while the official verifier still fails 52/53 poisoned responses, driving ICS-guided verified pass from 0.70 to 0.00 (captured headroom inverts below random;∆−0.70, CI[−0.90,−0.50],p=1/2000). Content selection’s robustness is therefore bounded to the non-adaptive threat model: against a white-box adversary optimising against the score itself, the ICS content-monitor is not safe as a sole selector. (Subset of 20 stratified attackable prompts; the collapse to 0.00 verified pass on every poisoned response is a floor, not a marginal effect, so the direction of the finding does not depend on sample size.) O Ablations and Robustness O.1 Calibration Sensitivity: which ten pairs, and how many? A training-free guard invites the question: you say ten pairs, but which ten, and must they be in-domain? We vary only the calibration set (pairs feeding the direction), holding the layer anchor on VAL-LAYER and the threshold on VAL-THRESH, and report on the disjoint test split (train∩ test = ∅asserted per draw). Three answers (Figure 8). (1) Which ten is stable: over 50 random 10-pair draws per domain the mean within-domain AUROC standard deviation is 0.014 (F1 0.019); a practitioner drawing ten pairs at random lands within about±0.08AUROC of the typical result. (2) Ten pairs is near enough: AUROC rises0.888(n=2)→ 0.939(n=10)→ 0.947(n=20) and plateaus, reaching within 1% of the full-data value by 20 pairs, against the10 4 –10 5 labelled examples a trained guard consumes. (3) In-domain barely matters: a single generic direction fit from the pooled training pairs scores 0.953, versus 0.952 for per-domain calibration and 0.943 for blind cross-domain transfer, so one direction can ship for all domains at a−0.001cost. (4) The calibrate-once LODO headline needs roughly ten pairs per source distribution, not ten total. The 0.728 mean leave-one-distribution-out AUROC (the paper’s Generalisation Beyond the Calibration Distribution section) is fit on the code’s actual pooled train budget, not a literal ten-pair draw; we re-fit the identical LODO protocol under four explicit training-budget conditions, three random draws each, same held-out val for anchor and threshold throughout. Ten pairs drawn from the entire pool, total, collapses toward chance (mean AUROC 0.556±0.050); twenty pairs total fares little better (0.615±0.041); ten pairs drawn from each source distribution (roughly 100 pooled total across the ten held-in distributions) recovers most of the headline (0.707±0.023); the full pool (the code’s actual, larger budget) reaches the reported 0.728. The near-zero-cost framing is accurate for per-distribution sample size, not for the total labelled-pair count, and we state it that way in the main text accordingly. Template robustness. The single-template concern is answered along the axis that matters. Presenting the same rule and scenario under a paraphrased instruction, a terse one-liner, or the model’s native chat template leaves ICS within 0.947±0.008AUROC and preserves 96% of cross-template transfer, so the direction is not an artifact of OmniCompliance’s exact wording. It is sensitive along one axis, position: reordering so the scored text ends on the (generic, truncated) rule rather than the case narrative collapses detection to 0.668 and breaks transfer, because ICS is a last-token probe of the scenario’s generic violation signal (the paper’s What the Score Measures section). The operating condition is that the scored text end on the case, not that the wording be fixed. The direction is largely one axis, and this is not a 1B artifact. The 20 per-domain directions are substantially one direction, consistent with the calibration finding that a single generic direction generalises (§O.1) and with reading a shared violation-register axis rather than per-rule geometry. Table 28 replicates every quantity on two more models. The collapse itself is real on all three, but its degree is not identical: the 8B-scale arms, Llama and Qwen alike, spread over a few more effective dimensions (participation ratio 3.2–3.5) than the 1B arm (1.47) – consistent with scale, 27 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models not family, and we report the pattern rather than rounding it to “the same everywhere.” The positive cosine to the refusal direction and the near-zero LODO-pooling cost both hold on every model. Table 28 Direction collapse replicates across scale and family. All AUROC gaps are small (collapse); all cosines to the refusal direction are positive and far from the random-vector null; LODO pooling never costs more than 0.01 AUROC on any model. The one real difference: both 8B arms are less extremely collapsed than the 1B arm (participation ratio∼3.2–3.5 vs. 1.47), a scale-consistent pattern across two families, not a single-model quirk. ModelCross-domain AUROC gapParticipation ratioCosine to refusal directionLODO ∆ (diag−off-diag)(random-vector null 18.0)(std. deviations)(vs. per-domain) Llama-3.2-1B+0.00851.47+0.151 (6.8σ)+0.0005 Llama-3.1-8B+0.01203.52+0.119 (7.6σ)+0.0091 Qwen3-8B+0.01953.24+0.09–0.13 (5.8–8.2σ)+0.0066 An OLS residualization against lexical content, sentiment, length and entropy leaves a significant 0.771 (null 0.587, above it in 18/20), so activations do carry signal beyond the surface. We do not claim it exceeds a strong lexical model on this single-template benchmark, which is why the deployable claims rest on selection and the guard comparison, not on an absolute detection margin. Design choices are swept and secondary (§T, Table 32). Raw and cosine scoring tie (∆ ≤ 0.001) and∥a∥ alone is near chance, so no length confound rides the norm; last-token/residual/mean is the weakly-best cell of the pooling/site/estimator grid; the equal-variance threshold assumption is false, so we report AUROC as primary. Three perturbations locate what the probe reads: a prepended opposite-verdict sentence moves nothing (0.012), paraphrase is its best cell (retention 0.994, beating TF-IDF in 18/20), but a single appended compliant-register sentence flips 72.4% of caught violations (TF-IDF 3.4%): the probe reads recent register at the read position, not content (§R). P Metric Robustness Every headline detection finding is re-expressed on the same seed-42 splits (positive=violation) under AUROC, AUPRC,FPR@95%TPR, andTPR@10%FPR; all 100 re-extracted rule conditions match the stored rule-ablation AUROC to< 10 −9 before scoring. Table 29 shows the qualitative conclusions do not depend on the metric. Rule- blindness is the decisive case: deleting, deranging, or swapping the rule causes 0/20 BH-significant degradations under all four metrics (FPR@95moves0.176→ 0.182), while the rule-only positive control collapses 19/20 under all four. AUPRC is a weak stress test here by construction (prevalence0.5makes its baseline0.5, so it tracks AUROC and on 5/20 domains understates ICS); we therefore lead the deployment claims withFPR@95%TPR, where ICS’s advantage over the same lexical floor, now under this metric, is both largest (0.164 vs. 0.426) and operationally meaningful. Table 29 The findings are metric-robust. ICS’s floor-beating margin, the rule-blindness null, and the rule-only positive control all hold under four metrics, so the central results are not AUROC artifacts. Calibration: within-domain Platt ECE 0.023, cross-domain 0.156 (§Q). Finding (domains of 20 supporting it)AUROCAUPRCFPR@95%TPRTPR@10%FPR ICS beats lexical floor18151817 Rule-deletion BH-sig. drops (should be 0)0000 Rule-only positive control collapses19191919 28 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Q Threshold Transfer: Direction vs. Operating Point The same failure shows up as miscalibration: a Platt-scaled ICS is well-calibrated within domain (ECE 0.023) but not across (cross-domain ECE 0.156, p95 0.40), so a fixed cross-domainτinherits exactly this (§P). We fit the ICS direction and thresholdτper domain and evaluate every ordered domain pair (1B, 20 domains, scoring-only over index-identical splits; each domain’s stored receipt AUROC reproduced to< 0.002before transfer). The readout direction is portable; the operating point is not (Table 30). Aτcalibrated to 5% FPR on one domain is heavy-tailed elsewhere (median 7%, but 26–31% at p95), andFPR@95%TPRblows up from 27.7% within-domain to a cross-domain mean of 34.7% and 77–94% at p95, with recall down to 25%. Midpoint-τ accuracy hides this (mean 0.903→ 0.880) but collapses to ≈ chance (0.556) in the tail. The direction that identifies compliance transfers; the threshold that decides it must be re-calibrated per domain. Table 30 The direction transfers; the operating point does not. Cross-domain AUROC nearly matches within-domain (all 380 ordered pairs≥ 0.79), but a fixed threshold’s realized FPR is heavy-tailed and FPR@95%TPR blows up 3–4×. Distribution reported via p95 and a small-n-robust 210-pair subset, not the worst cell. QuantityWithin-domainCross-domain Detection AUROC (the direction)0.9520.943 (all 380 ordered domain pairs)—≥ 0.79 FPR@95%TPR0.2770.347 (p95 0.77–0.94) Realized FPR of a 5%-calibrated τ0.050median 0.07 (p95 0.26–0.31) Accuracy at the midpoint τ0.9030.880 (worst-case tail 0.556) R Perturbation Robustness Three perturbations of the frozen probe. A prepended opposite-verdict sentence moves almost nothing (swap sensitivity 0.012, the lowest of any arm). Under paraphrase, ICS retains 0.994 with zero significant losses, degrading less than TF-IDF in 18/20 domains. But one appended compliant-register sentence flips 72.4% of correctly-flagged violations, while TF-IDF moves 3.4%. S Cross-Model Replication ICS’s margin over TF-IDF replicates across all twelve arms we ran, spanning four families and 1B to 72B, inside a narrow band from+0.053(Qwen3.5-4B) to+0.068(Qwen2.5-7B and Qwen2.5-72B, tied), without a consistent scaling trend: margin rises with scale within the Llama family but the trend does not survive crossing families (Table 31). The 70B-class question is no longer open: Qwen2.5-72B-Instruct (int8, single GPU) lands at margin+0.068, inside the established band, not above it. Layer selection is performed separately per model, since the signal sits at a consistent relative rather than absolute depth; raw and cosine scoring remain similar across every model (∆≤ 0.001). One family (Qwen3.5) is a named exception to the relative-depth pattern with no mechanistic account offered; full anchor-position trends, per-domain profile correlations, and norm-ablation detail are released with the evaluation protocol. The 72B Arm: Methodology Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct were evaluated in int8 (BitsAndBytesConfig(load_in_8bit=True)) because the released bf16 checkpoints did not fit a single 97 GB GPU (≈144 GB of weights); both fit with headroom and needed no multi-GPU sharding, and the same seed-42 split, direction fit, and per-domain layer search were used as every other arm in Table 31, only the quantised forward pass differs. 29 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models Table 31 The finding replicates across four architecture families and two orders of magnitude in scale (1B–72B). Margin=mean ICS raw AUROC−mean TF-IDF-floor AUROC (0.896), outcome-ablated OmniCompliance, 20 domains, identical procedure for every row. An earlier-draft per-model “gate” column has been withdrawn as not the registered gate of §B; provenance released with the evaluation protocol.google/gemma-3-4b-itexcluded: gated repository at submission time. Llama-3.3-70B and Qwen2.5-72B are the two frontier-scale arms (int8); Qwen3.5-27B replicates the margin band but is a named exception to the cross-family anchor-depth band (§S). ModelFamilyParams n layers Mean ICS raw Mean ICS cos Margin vs. TF-IDF floor Llama-3.2-1B-Instruct (reference)Llama1B160.9520.953+0.056 Llama-3.2-3B-InstructLlama3B280.9580.959+0.062 Llama-3.1-8B-InstructLlama8B320.9620.963+0.067 Llama-3.3-70B-InstructLlama70B800.9550.956+0.059 Mistral-7B-Instruct-v0.3Mistral7B320.9610.961+0.066 Gemma-2-9B-itGemma9B420.9590.960+0.063 Qwen2.5-7B-InstructQwen7B280.9630.964+0.068 Qwen3-8BQwen8B360.9530.957+0.057 Qwen3.5-4BQwen4B320.9480.949+0.053 Qwen3.5-9BQwen9B320.9510.951+0.055 Qwen3.5-27BQwen27B640.9520.953+0.056 Qwen2.5-72B-Instruct (int8)Qwen72B800.9640.965+0.068 T Method Axes Pooling×probe site×centroid estimator, swept around the published configuration (paired bootstrap against it, shared resample indices, exact-arithmetic BH). Pooling matters and last-token is right, and not because of truncation, since no OmniCompliance text exceeds 512 tokens: mean-over-all-tokens costs−0.099(15/20 significant losses) and falls into the selection-null zone in 9/20 domains; max-pooling costs−0.067; response-only mean−0.020. Probe site is a tie: attention output trends+0.004(positive in 15/20, never BH-significant), MLP output is a wash. Robust centroid estimators (median, 20% trimmed) change nothing (|∆|≤ 0.0007). Table 32 Method-axis sweep, mean over 20 ablated domains (paired bootstrap vs. the published cell, exact-BH). Last-token/residual/mean is on the Pareto face; attention output is a non-significant +0.004; the centroid estimator is a no-op. Mean-over-all-tokens is the only variant that falls into the selection-null zone (11/20 clear it). AxisConfigurationMean AUROC∆ vs. published Published (used throughout)last-token / residual stream / mean-difference0.9519— poolingmean over all tokens0.8534−0.099 poolingmean over response tokens only0.9318−0.020 poolingmax-pooling0.8847−0.067 probe siteMLP output0.9514−0.001 probe siteattention output0.9561+0.004 centroid estimatormedian0.9512−0.001 centroid estimator20%-trimmed mean0.9512−0.001 U Sample Efficiency Generalisation holds across base-model scale. Re-running the calibrate-once leave-one-distribution-out test (the paper’s Generalisation Beyond the Calibration Distribution section, the paper’s calibrate-once leave-one- distribution-out table) with the pooled direction fit on Llama-3.2-3B and Llama-3.1-8B gives LODO mean AUROC 0.761 and 0.758, versus 0.728 at 1B, a small1B→ 3Blift that then plateaus. The in-domain per-distribution ceiling rises monotonically (0.849→0.859→0.869) but the leave-one-out gap does not shrink, so calibrate-once transfer is a property of the pooled direction, not of base-model capacity. TRIDENT-law remains the weak spot at every scale. The hypothesised small-nadvantage of the closed-form estimator does not exist. A regularised logistic probe ties or beats ICS at every calibration sizen ∈ 2,..., 200(Figure 7; atn=10: 15/20 domains, mean+0.003); shrinkage- LDA and whitening buy nothing (significant in 0–1/20) and shrinkage is catastrophic atn=2. What survives is a fitting-cost claim, not an accuracy claim: ICS is approximately the small-nsolution of the regularised probe, obtained in closed form with no gradient step and no regularisation sweep. It is not free of labelled supervision beyond the calibration pairs; layer and threshold selection each consume a labelled validation slice, exactly as they do for the 30 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models logistic probe. CCS, run zero-supervision, reaches 0.917 with labelled layer selection, collapses to 0.69 under its own unsupervised layer choice. 2510204080200 calibration pairs n 0.5 0.6 0.7 0.8 0.9 test AUROC (mean over domains) selection null (random direction) chance ICS (diff-of-means) logistic probe shrinkage-LDA Figure 7 Sample efficiency (mean over 20 ablated domains,±1 s.e.). The logistic probe ties or beats ICS at every calibration size, there is no small-n crossover, and shrinkage-LDA is catastrophic at n=2. All estimators are read against the selection null (0.71), not chance. V Further Scope Conditions Four additional bounds, each stated briefly in the paper’s Scope and Limitations section and given in full here. First- party access is a cost advantage for self-monitoring, not third-party auditing, and performed compliance is not evaluated adversarially (22,23, the paper’s Comparison with Deployed Guards section). The degeneracy diagnosis is specific to compliance benchmarks, since the safety suite is cleaner at 2/7 degenerate (§D). ICS needs the scored text to end on the case narrative (§O.1). Absolute probe numbers sit on 0.71–0.77 selection floors, not 0.5. 31 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models 0.60.81.0 Test AUROC (n=10 pairs) edu discrimination us ed gdpr policy github foundation rights sb35 policy openai policy reddit policy x data act finance anti laundering policy google finance cross border pay edu academic integrity hipaa ccpa eu ai act finance eletric momey cybersecurity mitre atta finance crypto edu online learning A1 Stability, 50×10-pair draws dot=mean, bar 5-95 pct; σ=0.014 25102050100all # calibration pairs (n) 0.85 0.90 0.95 1.00 Test AUROC (pooled) A2 Calibration-set size curve 17 domains, 20 draws/pt all-train (200 pairs) -1% band within-1% at n=20 random n-pair draws 051015 Layer ℓ (0 = embedding) 0.6 0.8 1.0 Test AUROC A4a Layer sweep (grey = 20 domains) best−anchor gap 0.006; 6.5 within 1% pooled mean val-selected anchor 13510 # calibration domains K (held-out eval) 0.945 0.950 0.955 Held-out AUROC A5a Cross-subset generalisation 10 random K-subsets/pt 0.60.81.0 Test AUROC (n=10 pairs) edu discrimination us ed gdpr policy github foundation rights sb35 policy openai policy reddit policy x data act finance anti laundering policy google finance cross border pay edu academic integrity hipaa ccpa eu ai act finance eletric momey cybersecurity mitre atta finance crypto edu online learning A1 Stability, 50×10-pair draws dot=mean, bar 5-95 pct; σ=0.014 25102050100all # calibration pairs (n) 0.85 0.90 0.95 1.00 Test AUROC (pooled) A2 Calibration-set size curve 17 domains, 20 draws/pt all-train (200 pairs) -1% band within-1% at n=20 random n-pair draws 051015 Layer ℓ (0 = embedding) 0.6 0.8 1.0 Test AUROC A4a Layer sweep (grey = 20 domains) best−anchor gap 0.006; 6.5 within 1% pooled mean val-selected anchor 13510 # calibration domains K (held-out eval) 0.945 0.950 0.955 Held-out AUROC A5a Cross-subset generalisation 10 random K-subsets/pt 0.60.81.0 Test AUROC (n=10 pairs) edu discrimination us ed gdpr policy github foundation rights sb35 policy openai policy reddit policy x data act finance anti laundering policy google finance cross border pay edu academic integrity hipaa ccpa eu ai act finance eletric momey cybersecurity mitre atta finance crypto edu online learning A1 Stability, 50×10-pair draws dot=mean, bar 5-95 pct; σ=0.014 25102050100all # calibration pairs (n) 0.85 0.90 0.95 1.00 Test AUROC (pooled) A2 Calibration-set size curve 17 domains, 20 draws/pt all-train (200 pairs) -1% band within-1% at n=20 random n-pair draws 051015 Layer ℓ (0 = embedding) 0.6 0.8 1.0 Test AUROC A4a Layer sweep (grey = 20 domains) best−anchor gap 0.006; 6.5 within 1% pooled mean val-selected anchor 13510 # calibration domains K (held-out eval) 0.945 0.950 0.955 Held-out AUROC A5a Cross-subset generalisation 10 random K-subsets/pt 0.60.81.0 Test AUROC (n=10 pairs) edu discrimination us ed gdpr policy github foundation rights sb35 policy openai policy reddit policy x data act finance anti laundering policy google finance cross border pay edu academic integrity hipaa ccpa eu ai act finance eletric momey cybersecurity mitre atta finance crypto edu online learning A1 Stability, 50×10-pair draws dot=mean, bar 5-95 pct; σ=0.014 25102050100all # calibration pairs (n) 0.85 0.90 0.95 1.00 Test AUROC (pooled) A2 Calibration-set size curve 17 domains, 20 draws/pt all-train (200 pairs) -1% band within-1% at n=20 random n-pair draws 051015 Layer ℓ (0 = embedding) 0.6 0.8 1.0 Test AUROC A4a Layer sweep (grey = 20 domains) best−anchor gap 0.006; 6.5 within 1% pooled mean val-selected anchor 13510 # calibration domains K (held-out eval) 0.945 0.950 0.955 Held-out AUROC A5a Cross-subset generalisation 10 random K-subsets/pt Figure 8 ICS is insensitive to the calibration set. A1 (top left): test AUROC across 50 random 10-pair calibration draws per domain (dot = mean, bar = 5th–95th percentile; tight across domains,σ ≈ 0.014). A2 (top right): AUROC vs. number of calibration pairsn, within 1% of the full-data value byn=20; calibration on TRAIN only, evaluated on the disjoint test split. A4a (bottom left): test AUROC by candidate anchor layer for all 20 domains (grey), the pooled mean (purple), and each domain’s validation-selected anchor (green); an intermediate anchor-stability variant was consolidated into this panel and is omitted. A5a (bottom right): held-out AUROC vs. the number of calibration domains K pooled before fitting the direction, 10 random K-subsets per point. W Extended Related Work The full positioning against prior work, condensed to a few sentences in the paper’s Introduction section. The independent report of the same rule-blindness pattern for safety-relevant activation probes (the paper’s Introduction section) extends to two further concurrent findings, Legilimens [24] and Gavel [25], on top of the two already named there. Difference-of-means and related linear readouts underpin representation engineering [7], the refusal direction [8], and mass-mean probing [9], with contrast-consistent search [10] offering an unsupervised variant, and safety properties have been shown to occupy multi-dimensional rather than single directions [26,27]. We apply the same estimator as a monitor and ask what it reads when classes are defined by written rules rather than safety labels. Recent systems read policy adherence from hidden states [13,5], including DataShield’s training-sample filter [28], whose inverted valence motivates our term adherence. A parallel line ships dynamic, policy-conditioned guards that re-target to a new policy at inference without retraining [19], and our frozen characterisation (the paper’s Generalisation Beyond the Calibration Distribution section) applies only to the static classifiers benchmarked here, leaving open whether such dynamic systems pass our counterfactual rule test. We benchmark ICS against Llama Guard 3, WildGuard, Qwen3Guard, HarmBench [11], SIREN [5], GLiGuard, and Latent Policy Guard [12], under lexical floors, a selection null, and benign inputs, controls none of them report, extending them to the never-pooled and degeneracy-audit benchmarks of §D.1. A training-free gradient signal [21] and a concurrent activation probe [6] draw the same efficiency argument, the latter independently reaching our shared-blind-spot conclusion for safety harm, that activation scores read a broad risk register rather than the rule-context relation, a result we establish here for regulatory compliance and extend with a 32 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models difference-of-means alternative, a cross-family result (the paper’s Generalisation Beyond the Calibration Distribution section), and a selection deployment (the paper’s Score-Guided Candidate Selection section). Among these, Rachmil et al. [13] are the nearest neighbour, compliance-framed like ours, warranting direct contrast. The methods differ in estimator, our difference of means against their whitened out-of-distribution score, and in readout, our last-token residual against their full-context representation. On their own benchmark our truncation-limited last-token probe lands at chance, AUROC 0.46 with an oracle best layer topping 0.53, since typical cases exceed our context window while their full-context method does not, a result we detail together with a lexical-shortcut audit of that benchmark in §D. The methods are complementary, ICS a zero-cost read for short cases and whitening the appropriate tool for long transcripts, and our rule-blindness finding (the paper’s What the Score Measures section) extends to their detector class too. Whether such probes and the benchmarks used to evaluate them can be trusted at all is a second, related question. Probes degrade under shift [29], pair selection dominates gains [30], models evade probes [31], and obfuscated activations defeat monitors [14], while linear probes more broadly lean on surface evidence rather than the property they claim to read [32]. Linear probes for strategic deception report AUROC 0.96 to 0.999 with neither a lexical floor nor a random-direction control [33], the identical un-floored pattern in a second subfield. The lexical-shortcut check is established practice, not a contribution here. TF-IDF baselines expose construction artifacts in adjacent jailbreak and hallucination benchmarks [34], and are read elsewhere as good news for cheap CPU guardrails rather than as a floor [35], a sign we flip explicitly. Our rule-only and no-rule conditions are premise-only and hypothesis-only baselines [36], and the degeneracy we document is the shortcut-learning failure mode [15] recurring in rule-conditioned compliance, established here for guards as well as probes, and motivating the counterfactual rule-following tests [37] current benchmarks lack. Control tasks [38] audit probe capacity, and our budget-matched nulls extend that logic to the hyperparameter search. Our corpora include OmniCompliance [16], AIReg-Bench [39], CompliBench [40], and FlexBench [41], of which four of seven benchmarks tested cannot support the measurement at all. A convergent audit [42] asks whether benchmark labels are correct from item-response patterns across models, and we ask instead whether correct labels measure the construct claimed, from the item text alone. References [1]Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. [2]Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024. [3] Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, et al. Qwen3guard technical report, 2025. [4] European Parliament and Council of the European Union. Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation). Technical Report L 119/1, Official Journal of the European Union, 2016. [5] Difan Jiao, Yilun Liu, Ye Yuan, Zhenwei Tang, Linfeng Du, Haolun Wu, and Ashton Anderson. Llm safety from within: Detecting harmful content with internal representations, 2026. [6]Dominik Schwarz. The entanglement wall: Activation-space probes as risk detectors, not context adjudicators, 2026. [7]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. [8]Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37:136037–136083, 2024. [9]Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023. [10]Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. [11]Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. [12] Nanxi Li, Zhengyue Zhao, and Chaowei Xiao. Lpg: Balancing efficiency and policy reasoning in latent policy guardrails, 2026. 33 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models [13]Oren Rachmil, Roy Betser, Itay Gershon, Omer Hofman, Nitay Yakoby, Yuval Meron, Idan Yankelev, Asaf Shabtai, Yuval Elovici, and Roman Vainshtein. Training-free policy violation detection via activation-space whitening in llms, 2025. [14]Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated activations bypass llm latent-space defenses, 2024. [15]R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference, 2019. [16]Wenbin Hu, Huihao Jing, Haochen Shi, Changxuan Fan, Haoran Li, and Yangqiu Song. OmniCompliance- 100K: A multi-domain, rule-grounded, real-world safety compliance dataset. In Findings of the Association for Computational Linguistics: ACL 2026, 2026. arXiv:2603.13933. [17]Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 90960– 90985, 2025. [18]Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. [19]Monte Hoover, Vatsal Baherwani, Neel Jain, Khalid Saifullah, Joseph Vincent, Chirag Jain, Melissa Kazemi Rad, C Bayan Bruss, Ashwinee Panda, and Tom Goldstein. Dynaguard: A dynamic guardian model with user-defined policies, 2025. [20] Zheng Hui, Yijiang River Dong, Ehsan Shareghi, and Nigel Collier. Trident: Benchmarking llm safety in finance, medicine, and law. arXiv preprint arXiv:2507.21134, 2025. [21]Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. Gradsafe: Detecting jailbreak prompts for llms via safety-critical gradient analysis, 2024. [22]Alexander Panfilov, Evgenii Kortukov, Kristina Nikoli ́ c, Matthias Bethge, Sebastian Lapuschkin, Wojciech Samek, Ameya Prabhu, Maksym Andriushchenko, and Jonas Geiping. Strategic dishonesty can undermine ai safety evaluations of frontier llms, 2025. [23]Joe Needham, Giles Edkins, Govind Pimpale, Henning Bartsch, and Marius Hobbhahn. Large language models often know when they are being evaluated, 2025. [24] Jialin Wu, Jiangyi Deng, Shengyuan Pang, Yanjiao Chen, Jiayang Xu, Xinfeng Li, and Wenyuan Xu. Legilimens: Practical and unified content moderation for large language model services. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1151–1165, 2024. [25]Shir Rozenfeld, Rahul Pankajakshan, Itay Zloczower, Eyal Lenga, Gilad Gressel, and Yisroel Mirsky. Gavel: Towards rule-based safety through activation monitoring, 2026. [26] Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, and Weiyan Shi. LLMs encode harmfulness and refusal separately. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [27]Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Yu Haining, and Xiaohua Jia. The hidden dimensions of LLM alignment: A multi-dimensional analysis of orthogonal safety directions. In Forty-second International Conference on Machine Learning, 2025. [28]Junbo Zhang, Qianli Zhou, Xinyang Deng, Wen Jiang, Jie Pan, and Jinbiao Zhu. Datashield: Safety-degrading data filtering for llm benign instruction fine-tuning, 2026. [29]Sachin Kumar. Pressure-testing deception probes in llms: Scaling, robustness, and the geometry of deceptive representations, 2026. [30]Vikram Natarajan, Devina Jain, Shivam Arora, Satvik Golechha, and Joseph Isaac Bloom. One probe won’t catch them all: Towards targeted deception detection. In Forty-third International Conference on Machine Learning, 2026. [31] Max McGuinness, Alex Serrano, Luke Bailey, and Scott Emmons. Neural chameleons: Language models can learn to hide their thoughts from unseen activation monitors, 2025. [32]Gerard Boxo, Aman Neelappa, and Shivam Raval. Linear probes rely on textual evidence: Results from leakage mitigation studies in language models, 2025. [33]Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. Detecting strategic deception with linear probes, 2025. [34] Khizar Hussain and Murat Kantarcioglu. Parallax: Separating genuine hallucination detection from benchmark construction artifacts, 2026. [35] Vasudev Majhi, Dhruv Gupta, Advait Singh, Matthew Barker, and Dhruv Kumar. Do you really need a gpu to guard your llm? cpu-class classifiers and multi-stage pipelines for safety enforcement at scale, 2025. 34 What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models [36]Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference, 2018. [37] Wangtao Sun, ChenxiangZhang ChenxiangZhang, XueYou Zhang, Xuanqing Yu, Ziyang Huang, Haotian Xu, Shizhu He, Jun Zhao, and Kang Liu. Beyond instruction following: Evaluating inferential rule following of large language models. In Maosong Sun, Peiyong Duan, Zhiyuan Liu, Ruifeng Xu, and Weiwei Sun, editors, Proceedings of the 24th China National Conference on Computational Linguistics (CCL 2025), pages 1043–1066, Jinan, China, August 2025. Chinese Information Processing Society of China. [38] John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2733–2743, 2019. [39] Bill Marino, Rosco Hunter, Christoph Schnabl, Zubair Jamali, Marinos Emmanouil Kalpakos, Mudra Kashyap, Isaiah Hinton, Alexa Hanson, Maahum Nazir, Felix Steffek, et al. Aireg-bench: Benchmarking language models that assess ai regulation compliance, 2025. [40] Jingbo Yang, Guanyu Yao, Bairu Hou, Xinghan Yang, Nikolai Glushnev, Iwona Bialynicka-Birula, Duo Ding, and Shiyu Chang. Complibench: Benchmarking llm judges for compliance violation detection in dialogue systems, 2026. [41]Zhihao Ding, Jinming Li, Ze Lu, and Jieming Shi. Flexguard: Continuous risk scoring for strictness-adaptive llm content moderation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5825–5851, 2026. [42] Sander Land and Daniel M Bikel. Auditing llm benchmarks with item response theory, 2026. 35

Instant research discovery