Paper deep dive
Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?
Chenglin Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/7/2026, 9:14:32 AM
Summary
This study evaluates the independence of errors in stacked runtime gates for AI agent actions, consisting of one deterministic rule layer and four LLM judges. Using 1,119 labelled actions from three corpora, the authors introduce 'n_mult', a metric for multiplication-equivalent layers, to measure failure correlation. Results show that while a rule layer and an LLM judge compose nearly independently (n_mult ~1.86-2.09), pairs of LLM judges fail more correlatedly (n_mult ~1.2-1.4). Solo accuracy does not predict stack value; for instance, improving a rule layer's solo performance did not increase its joint coverage. The study also highlights that evaluation conventions (PRIMARY vs. STRICT miss definitions) and model version drift can significantly alter conclusions.
Entities (13)
Relation Signals (8)
AgentTrust â contains â Rule Layer
confidence 95% · The rule layer is the default policy engine of the AgentTrust runtime
PRIMARY miss definition â defines â Miss
confidence 95% · Under PRIMARY (escalation scored as caught)
STRICT miss definition â defines â Miss
confidence 95% · Under the STRICT miss definition (escalation to a human scored as not stopped)
GPT-5.5 â isa â LLM Judge
confidence 95% · One, recorded by the operator as GPT-5.5 and called gpt55... The four judges are chat models
n_mult â measures â failure correlation
confidence 95% · We read each stack as a number of multiplication-equivalent layers, n_mult... to measure failure correlation.
Solo Accuracy â doesnotpredict â Stack Value
confidence 92% · Solo accuracy does not predict what a layer adds: a cloud rule pack lowers the rule layer's solo miss rate by 20% and adds no new joint coverage.
Rule Layer â composeswith â LLM Judge
confidence 90% · The rule layer plus one judge composes to 1.86 to 2.09 layers... close to the multiplication prediction.
LLM Judge â composeswith â LLM Judge
confidence 90% · any two judges compose to about 1.2 to 1.4 layers... they miss 5.2%... predicted to miss 1.1% together
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Runtime gates for agent tool calls are stacked on the assumption that their errors multiply. We test it on 1,119 labelled agent actions from three corpora, without an adaptive adversary. The stack has one deterministic rule layer and four LLM judges, three of them re-collected with the served model recorded on every call. We read each stack as a number of multiplication-equivalent layers, n_mult, with its floor under perfect coupling. Under the STRICT miss definition (escalation to a human scored as not stopped), any two judges compose to about 1.2 to 1.4 layers ({\phi} median +0.430, 6 of 6 pairs significant, floors 1.02 to 1.17). The rule layer plus one judge composes to 1.86 to 2.09 layers ({\phi} median +0.014, 0 of 4 significant, floors 1.01 to 1.09). Under PRIMARY (escalation scored as caught) the bands are 1.21 to 1.57 and 1.80 to 2.13. Intervals separate on the pooled data, point estimates split on each corpus, and a third-vendor judge lands in the judge band. Solo accuracy does not predict what a layer adds: a cloud rule pack lowers the rule layer's solo miss rate by 20% and adds no new joint coverage. The difficulty share of judge coupling is not identifiable: 31.8% to 61.8% depending on the probe and the miss definition. One judge tier was served by an unrequested model version in 50 of 112 batches, concentrated on the external corpus. That event overturned a pre-declared analysis rule, and the scoring of review verdicts reversed five conclusions. We report both.
Tags
Links
- Source: https://arxiv.org/abs/2610.07359v1
- Canonical: https://arxiv.org/abs/2610.07359v1
Trouble viewing inline? Open PDF directly â
Full Text
90,219 characters extracted from source content.
Expand or collapse full text
Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently? Chenglin Yang Affiliation: University of Lancashire yangchenglin802@gmail.com Abstract Runtime gates for agent tool calls are stacked on the assumption that their errors multiply. We test it on 1,119 labelled agent actions from three corpora, without an adaptive adversary. The stack has one deterministic rule layer and four LLM judges, three of them re-collected with the served model recorded on every call. We read each stack as a number of multiplication-equivalent layers, nmultn_mult, with its floor under perfect coupling. Under the STRICT miss definition (escalation to a human scored as not stopped), any two judges compose to about 1.2 to 1.4 layers (Ï median +0.430+0.430, 6 of 6 pairs significant, floors 1.02 to 1.17). The rule layer plus one judge composes to 1.86 to 2.09 layers (Ï median +0.014+0.014, 0 of 4 significant, floors 1.01 to 1.09). Under PRIMARY (escalation scored as caught) the bands are 1.21 to 1.57 and 1.80 to 2.13. Intervals separate on the pooled data, point estimates split on each corpus, and a third-vendor judge lands in the judge band. Solo accuracy does not predict what a layer adds: a cloud rule pack lowers the rule layerâs solo miss rate by 20% and adds no new joint coverage. The difficulty share of judge coupling is not identifiable: 31.8% to 61.8% depending on the probe and the miss definition. One judge tier was served by an unrequested model version in 50 of 112 batches, concentrated on the external corpus. That event overturned a pre-declared analysis rule, and the scoring of review verdicts reversed five conclusions. We report both. Index Terms: agent safety, runtime enforcement, defence in depth, LLM-as-judge, correlated failure I Introduction An agent with tool access can delete a production database or push a credential to a public repository in a single call [1]. Deployments therefore gate each proposed action before it executes. A common stack has a deterministic rule layer that matches patterns in the action text, one or more LLM judges that read the action and return a verdict, and sometimes a human reviewer [2, 3]. The stack is justified by a multiplication argument. If the rule layer misses 13.4% of the actions labelled block in our corpora and one judge misses 10.5%, independent errors would give a joint miss rate near 1.4%. The safety question this paper answers is whether stacked action gates have the independence that argument assumes, and what each layer adds once the stack is measured as a whole. The measurement is over labelled corpora of authored actions, without an adaptive adversary; Section I-A states what that covers. On our corpora the argument holds for one kind of pair and fails for another. The rule layer and the judge above miss 1.3% of block cases together (7 of 560), close to the multiplication prediction. Two judges that each miss 10.5% alone would be predicted to miss 1.1% together; they miss 5.2% (29 of 560). These rates use the PRIMARY miss definition of Section I. The difference between the two pairs is the subject of this paper. Failure correlation inside a defence stack has been measured before. The one prior study we found measured it for a seven-layer single-turn jailbreak-defence stack in which every member is a function of one target model [4]. It describes its scope as single-turn jailbreak only, excludes injection-targeted rows from its measurement by design, lists correlation across agentic settings as unmeasured, and collects its correlations under an adaptive adversary. Action gates differ in two ways that bear on correlation. A rule layer over action text consumes no model output, and the judges are black-box models from several vendors that receive the actions directly, in the assume-compromise setting Parallax names [3]. We measure the same statistic, with the same difficulty stratifier, in the setting that study excludes and without its adversary. The paper makes a three-part argument. First, it defines an instrument. The number of multiplication-equivalent layers, nmultn_mult, reads an observed joint miss rate as the number of independent layers that would produce it, each failing at the stackâs geometric-mean miss rate (Section I). Second, it shows that the instrument in current use fails (Section IV). Solo accuracy, the number reported for a gate measured alone, does not predict what the gate adds to a stack. Adding a cloud rule pack cuts the rule layerâs solo miss rate from 0.139 to 0.111. It adds zero new joint coverage on top of three judges under either definition and moves the rule layer toward the judge band on nmultn_mult. A 50-pattern blocklist with a solo miss rate of 0.857 still adds nine cases under STRICT. Third, it reports what the new instrument reads (Section V). Under STRICT, any two judges compose to 1.22 to 1.44 layers above floors of 1.02 to 1.17. The rule layer plus one judge composes to 1.86 to 2.09 layers above floors of 1.01 to 1.09. The two bands separate on the pooled data; on each corpus the point estimates split and the intervals overlap. Four comparison collections give the same structure: the archived panel that preceded this one, a panel collected with a project-memory index in the judgesâ system context, a single-case check against batched judging, and a fifth judge from a third vendor (Section VI). Our data do not separate judge coupling from shared difficulty. We stratify each judge pair by a leave-both-out difficulty score, the construction the prior study uses. The test for a shared surface mechanism runs in the direction opposite to that hypothesis. The share of coupling attributable to difficulty ranges from 31.8% to 61.8% with the choice of difficulty probe and miss definition (Section V-D). Difficulty alone can produce coincident failures between independently built components [5, 6], and the residual cannot be read as evidence of a shared mechanism. Two evaluation conventions changed conclusions during this study. One judge returns the verdict review on 47 of 560 block cases, the rule layer on 3, and the other three judges on 2 to 17. Whether review counts as a catch or as a miss reversed five conclusions, each time in the direction that favoured the deferring judge (Section V-F). Every result is therefore reported under both accounting rules, PRIMARY and STRICT (Section I). Separately, one re-collected judge tier was served by a version other than the one requested in 50 of its 112 batches, 36 of them with no trace except the per-call served-model field. The change fell unevenly across the corpora (Section VI-D). The archived verdicts have no such field. This paper contributes: âą a measurement of failure correlation among action gates on 1,119 labelled actions from three corpora, with one rule layer and four judges under one case-text condition, three judges re-collected with per-call provenance, and both miss definitions reported throughout (Sections I and V); âą the instrument nmultn_mult with its floor and normalised reading, positioned against the beta factor and the Kish effective sample size, and a two-dimensional reading of stack value, mechanism class by in-domain competence, whose empty corner is stated as a falsifiable prediction (Sections I and IV); âą a difficulty-versus-mechanism decomposition with a second implementation, replicating the prior stack measurementâs non-identifiability finding in the setting that study excludes (Section V-D); âą two evaluation conventions that reverse conclusions, a primary judge panel with per-call provenance and system-level blinding, and four comparison collections including a fifth judge from a third vendor (Sections V-F and VI); âą a public artifact with the per-layer verdicts, the per-call provenance records, the analysis scripts, the pre-registration and audit summaries, including the conclusions withdrawn during analysis. Sections VII, VIII and IX give related work, discussion and limitations. I Threat Model and Setup I-A Threat model and scope The adversary controls the text of the proposed action. This is the assume-compromise setting that Parallax names [3]: the agent is treated as already compromised, and the gates are evaluated on the actions it proposes. Each gate sees the action text and nothing else. The measurement is over labelled corpora of authored actions, and no adversary adapts to the gates during it. The only adversarial proxy is a set of obfuscation flags read off the case text (Section V-D), and they were not optimised against any layer. Every Ï and nmultn_mult in this paper is therefore a distributional statement about these corpora. It is not a worst-case statement. An attacker who knows the rules can write an action that none of them matches. The external corpus shows that a distribution shift alone, with no attacker, takes the rule layerâs miss rate to 52.9%. An attacker who controls the action text can also target the judges through it, and adaptive attacks of this kind defeat published defences against jailbreaks and prompt injection [7, 8]. Adaptive evasion is out of scope. The closest prior stack measurement collected its correlations under an adaptive adversary [4]. This paper does not. A deployer facing an adaptive attacker cannot take an nmultn_mult near 2 as protection against an attack aimed at the joint miss set. I-B Layers The stack under test has one deterministic rule layer and four LLM judges. The rule layer is the default policy engine of the AgentTrust runtime [2]: 174 rules, each a regular expression over the text of the proposed action, returning allow, warn, block or review. It reads no model output. The four judges are chat models that receive the action and return the same four-way verdict. One, recorded by the operator as GPT-5.5 and called gpt55, is an archived column with no version string, timestamp or temperature. The other three were re-collected for this paper from one vendor through a coding-assistant harness, in a session with no project memory, with the served model string recorded on every call. Haiku was requested as the dated claude-haiku-4-5-20251001, and sonnet and opus as the aliases claude-sonnet-5 and claude-opus-5. The opus tier was served by claude-opus-4-8 on 50 of its 112 batches (Section VI-D). Section VI compares this panel with the archived columns it replaces. A fifth judge from a third vendor, re-collected with the served model recorded on every call, is reported separately in Section VI-E because it covers two of the three corpora. I-C Elicitation Every judge column in the headline numbers was collected under one case-text condition, which we call no-tell. The judge is prompted in the judge role, shown the action type, tool, content and parameters, asked to rate five risk dimensions, and asked for a verdict. The human-written description field is blanked. The archived gpt55 column was collected in batches of 90 cases in identifier order with no record of wrapper prompt, temperature, model version or date. The three re-collected columns were collected in batches of 10 cases in a seeded random order over all 1,119 cases, one call per batch. Their case blocks are byte-identical to the archived no-tell files; the instruction header is not. The archived collection used one header per corpus. The re-collections share a single header that drops the archived headerâs list of example threat patterns and adds a no-tools clause. In addition, 100 cases per tier were judged again one case per call as a check on batching (Section VI-C). The rule layerâs archived verdicts read the description. Recomputing them with it blanked changes 2 of 560 block verdicts and no headline conclusion (Section IX, item 11). The description matters because on block cases it often names the threat: 126 of the 560 block-case descriptions contain a giveaway word. Two archived columns were elicited with the description shown, which we call faithful, namely a DeepSeek column and a second Opus column. Both are excluded from every headline number, since mixing conditions would confound coupling with elicitation. The exclusion is conservative: including the DeepSeek column raises the significant judge pairs from 6 of 6 to 9 of 10 under PRIMARY. I-D Corpora The 1,119 cases with complete coverage on all five layers come from three corpora with disjoint identifier spaces (Table I). The internal corpus (300 cases) is the runtimeâs own benchmark, six categories of fifty cases each, and served as the rule layerâs tuning set [2]. The independent corpus (630 cases) was authored by the project team of [2] in five batches after the rule set of a frozen release. When a case exposed a missing rule, the rule was added to the production set and the case kept [2]. The rule layer measured here is the current default, after those patches, so that corpus is independently constructed but not zero-shot for the rule layer. Its 6.1% rule miss rate is a post-patch residual. Its categories are verdict tiers such as safe, moderate and dangerous. The external corpus (189 cases) covers cloud identity and secrets, CI and infrastructure-as-code, databases, network egress, package supply chains, shell persistence, file configuration and a deliberately hard benign observability family [9]. Its cases were constructed with an LLM and labelled against a rubric. Every retained case carries a label that an independent human annotator who is not an author of the runtimeâs reports confirmed by blind relabelling. Of the 224 source cases, all relabelled, 193 agreed and 31 disagreed (Îș=0.817Îș=0.817 before filtering). The disagreements were discarded, and the frozen 189 are agreed cases after tell and leakage filtering [9]. The default rule set does not target that domain, and Section V-C shows what that costs. Every case carries a truth label of allow, warn or block. TABLE I: Corpora and truth labels. Misses are measured on block cases and false alarms on allow cases; warn cases belong to neither domain. corpus cases block allow warn internal 300 129 133 38 independent 630 344 188 98 external (held out) 189 87 58 44 all 1,119 560 379 180 I-E Error domains Misses compose conjunctively, since the stack fails only if every layer misses, so the miss domain is the 560 block cases. False alarms compose disjunctively, since any layer that fires raises the alarm, and are reported separately on the 379 allow cases; the 180 warn cases belong to neither domain. I-F Two miss definitions The verdict alphabet contains review, an escalation to a human that is neither an execution nor a stop. On the 560 block cases the layers return it 3 (rules), 47 (gpt55), 17 (haiku), 2 (opus) and 12 (sonnet) times. Two readings of a miss differ only on it. PRIMARY reads a miss as âthe action executedâ and counts allow and warn as misses. STRICT reads a miss as âthe action was not stoppedâ and counts allow, warn and review. Which reading is correct depends on whether a human review layer exists downstream to absorb the escalation, and this study has no data on such a layer. Both definitions are reported everywhere, and where they diverge we say so. The divergence is a finding of its own (Section V-F). I-G Verification and withdrawn conclusions Six headline statistics were recomputed against an independent implementation (Appendix A); one of them, the difficulty share, gained its second implementation after review. Seven candidate conclusions were withdrawn during analysis. The pre-registration and a summary of the audits are in the artifact. I The Instrument: Multiplication-Equivalent Layers For a stack S with per-layer miss rates pLp_L and observed joint miss rate PjointP_joint, we define nmult=lnâĄPjointlnâĄg,g=(âLâSpL)1/|S|.n_mult= P_joint g, g= ( _Lâ Sp_L )^1/|S|. (1) Layers whose errors multiply give nmult=|S|n_mult=|S|. The value under perfect coupling is not 1 in general. Since Pjointâ€minLâĄpLP_joint†_Lp_L, nmultâ„lnâĄminLâĄpLlnâĄg,n_mult\;â„\; _Lp_L g, (2) with equality when the misses of the most accurate layer are a subset of every other layerâs. This floor equals 1 only when the marginals are equal, and it rises with their spread. For the pooled rule pairs under STRICT it is 1.01 to 1.09. On the external corpus, where the rule layer misses half the block cases, it reaches 1.54 to 1.73. We report the floor beside every nmultn_mult and, where the comparison matters, the normalised reading (nmultâfloor)/(|S|âfloor)(n_mult-floor)/(|S|-floor), which is 0 at perfect coupling and 1 under independence. Values above |S||S| indicate negative association and occur for the rule pairs whose Ï is below zero. Confidence intervals propagate the exact ClopperâPearson interval on PjointP_joint [10]. A case-level bootstrap gives narrower intervals for the judge pairs and similar ones for the rule pairs, and Table IV reports both. When a stack has no joint miss at the sample size, nmultn_mult has no point estimate, only a lower bound from the upper limit on PjointP_joint. We mark such a stack as censored. The construction is an effective count: given an observed aggregate, report the number of idealised identical units that would produce it. Hill numbers in ecology use it, with a communityâs diversity read as the size of a reference community of equally abundant species [11, 12, 13, 14, 15]. We apply it to a joint failure probability: given PjointP_joint, how many independent layers, each failing at the geometric-mean rate g, would produce it. The construction is established. We did not find this application in the common-cause-failure family of reliability engineering, which uses ratio parameters, or in the effective-sample-size family of statistics. We claim no new mathematics. If a standard definition of this application exists, we would align with it. Two quantities share the vocabulary. The beta factor [16], the standard treatment of common-cause failure, is the conditional probability that a common cause takes the other channels too. It is a ratio, it moves inversely to nmultn_mult, and it does not say how many layers a stack is worth. The symbol ÎČ also denotes judge miss rates in work we cite [17]. Kishâs effective sample size [18], (âwi)2/âwi2(ÎŁ w_i)^2/ÎŁ w_i^2 over a weighted sampleâs weights, has been applied to LLM judge panels [19]. It is a different quantity about a different object, so we do not reuse that name. The normalisation by g is a deliberate choice. g is the geometric mean of the layersâ own miss rates, so nmultn_mult does not reward a layer for being accurate: a strong layer whose failures coincide with its partnerâs still scores low. Section V-C shows that a stackâs leak rate is the product of independence and in-domain competence. nmultn_mult reads the first factor, though competence still enters it through the marginals and their floor, so the floor is reported with it. Companion statistics are reported per pair. The correlation Ï is the mean-square contingency coefficient on the two binary error indicators, identical to Pearsonâs r on 0/1 data. It is the headline statistic because the closest prior measurement reports it [4]. Ï is bounded by the marginals. For p1â€p2p_1†p_2 the attainable maximum is Ïmax=p1â(1âp2)p2â(1âp1), _ = p_1(1-p_2)p_2(1-p_1), (3) their Eq. 8. Our rule layer and judges have unequal miss rates, so Table I reports Ï/Ïmax / _ beside raw Ï . The coupling ratio c=Pobs/Pindepc=P_obs/P_indep and Cohenâs Îș [20] are computed per pair by the scripts in the artifact. Pairwise significance uses Fisherâs exact test with Holm correction [21]; each table states the family over which the correction runs. IV The Instrument in Use Fails: Solo Accuracy Does Not Predict Stack Value We test whether a gateâs solo accuracy, the number usually reported for it, predicts what the gate adds to a stack. Every increment below is measured the same way. Its new coverage is the number of block cases that a baseline stack of three judges leaks and the increment catches, summed over the four such baselines, under both miss definitions (560 block cases). Table I lists four increments. TABLE I: What an increment adds to a three-judge stack (560 block cases). New coverage is the number of baseline leaks the increment catches, summed over the four three-judge baselines, with the per-baseline breakdown; b1 is a second, weaker rule engine. PRIMARY STRICT increment class solo new (per baseline) solo new (per baseline) rule layer det. lexical 0.134 31 (3/3/3/22) 0.139 86 (17/19/22/28) rule layer + pack det. lexical 0.100 31 (3/3/3/22) 0.111 86 (17/19/22/28) fourth judge LLM semantic 0.099 21 (0/0/0/21) 0.133 22 (0/3/6/13) b1 (50 patterns) det. lexical 0.857 2 (0/0/0/2) 0.857 9 (2/3/2/2) Under STRICT the rule layer adds 86 cases against 22 for a fourth judge. Under PRIMARY the two add 31 and 21, and 22 of the rule layerâs 31 come from the one baseline that lacks gpt55. The reason is the accounting rule for review. Under PRIMARY gpt55âs 47 review verdicts count as caught, so the three baselines that contain it leak only 5 cases each. The fourth judge added to the remaining baseline is gpt55 itself, credited with 8 deferrals among its 21 additions. Under STRICT those baselines leak 19 to 25 cases and the rule layer catches 17 to 22 of them, while a fourth judge catches 0 to 6. The rule layerâs advantage is largest under STRICT, and the fourth judgeâs value is one of the conclusions that the accounting rule moves (Section V-F). Two observations point in opposite directions, and both hold under either definition. First, the cloud pack improves the rule layerâs solo miss rate (0.139 to 0.111 under STRICT, a 20% relative improvement) and leaves the stackâs leak set unchanged: 86 to 86 and 31 to 31, identical case for case. Inspected directly, every block case the pack newly catches on the external corpus (17 under PRIMARY, 15 under STRICT) was already caught by all four judges. On the independent corpus the pack changes three block verdicts: it newly catches two cases, one of them caught by three of the four judges, and it moves one case from block to review. None of the three is leaked by any three-judge baseline, so new joint coverage is exactly zero. The pack was written against separate held-out corpora and did not consult the external corpus scored here, so the solo improvement is itself a held-out result. Second, b1, the layer with the worst solo accuracy, is not the best contributor. It adds 9 cases under STRICT and 2 under PRIMARY, against 86 and 31 for the stronger deterministic layer. The pack also moves the rule layer on the independence axis, and that move should be read together with the zero. On the external corpus, where almost all of the packâs effect falls, the rule layerâs Ï with the four judges rises from +0.10+0.10, â0.01-0.01, +0.10+0.10 and +0.18+0.18 to +0.15+0.15, +0.08+0.08, +0.15+0.15 and +0.27+0.27 (PRIMARY), and its nmultn_mult with each judge falls from 1.75, 2.03, 1.75 and 1.68 to 1.61, 1.81, 1.61 and 1.51; under STRICT it falls from 1.96, 1.76, 2.05 and 1.75 to 1.71, 1.56, 1.88 and 1.57. The joint miss counts do not change (1, 2, 1 and 3 under PRIMARY; 7, 7, 1 and 5 under STRICT). The denominator moves and the numerator does not. The pack removes only rule misses that the judges already caught, which is the difficulty mechanism of Section V-D acting on the rule layer. The rule layerâs independence reading moves toward the judge band as its competence rises on cases the judges find easy. b1 cannot be used to attribute decoupling to the mechanism class. Its solo miss rate of 0.857 gives its pairs with the judges floors of 1.83 to 1.88 (Section I), which leaves only 0.12 to 0.17 up to the value of 2 that independence would give. The observed nmultn_mult of 1.91 to 1.97 cannot be read on that scale, and its Ï/Ïmax / _ with the judges is 0.20 to 0.63, inside the judge band. b1 therefore appears in this paper only as a point about increments. The attribution of decoupling to the mechanism class rests on the default rule layer across three corpora, the comparison panels and the cross-vendor column (Sections V and VI). Solo accuracy therefore over-predicts in one direction and under-predicts in the other. The pattern consistent with all four points is two-dimensional: mechanism class and in-domain competence both matter, and neither alone predicts stack value. Four points on three corners of a 2Ă22Ă 2 form a taxonomy consistent with the observations. They do not fit a two-factor model, since two of the four points carry the caveats above and there are too few points to separate the factors. The reading is an existence statement. The empty corner, same mechanism with low competence, is a falsifiable prediction of about zero added coverage, and we had no such layer to test. Figure 1 (left) plots the four points in the units of Table I. The cloud-pack increment is horizontal on the coverage axis and, on the independence axis of the right panel, a move toward the judge band. Fig. 1: Left: new coverage added to a three-judge stack, summed over the four baselines (STRICT), against the incrementâs in-domain competence (one minus solo miss rate); marker shape gives the mechanism class. The empty corner of the 2Ă22Ă 2, same mechanism with low competence, is a prediction of about zero added coverage, and no layer of that kind was available to test. Four points on three corners form a taxonomy consistent with the observations; they do not fit a two-factor model. Right: nmultn_mult of every two-layer stack by corpus (STRICT) with exact 95% intervals. Short ticks mark each stackâs floor under perfect coupling, which depends on the marginals. Censored stacks, with no joint miss at the sample size, are drawn as arrows from their lower bound; on the internal corpus every rule-plus-judge stack is censored. 560 block cases, no-tell elicitation for every layer. V Read with the New Instrument: Coupling Splits Along Mechanism, Not Count V-A Pairwise correlation Table I gives the pairwise miss correlation on the pooled 560 block cases. The rule layerâs four pairs with judges sit at zero on raw Ï and stay there after marginal normalisation: Ï/Ïmax / _ median +0.002+0.002 under PRIMARY and +0.018+0.018 under STRICT, against +0.697+0.697 and +0.501+0.501 for the six judge pairs. The largest normalised rule pair (+0.134+0.134, rules with gpt55, PRIMARY) is below the smallest normalised judge pair (+0.432+0.432), so the split of point estimates does not depend on unequal marginals. Significance is Fisherâs exact test with Holm correction over the ten pairs of each definition; normalisation rescales Ï and changes no p-value. Intervals add one qualification. Under PRIMARY the 90% intervals of the rule pairs are disjoint from those of the three same-vendor judge pairs, whose lower limits are +0.32+0.32 or above. They overlap those of the pairs that involve gpt55, whose lower limits are +0.04+0.04 to +0.12+0.12 because its review habit holds its Ï down (Section V-F). Under STRICT every judge pair has Ïâ„0.328 â„ 0.328 and the separation is complete, with 6 of 6 judge pairs significant against 0 of 4 rule pairs. TABLE I: Pairwise miss correlation, pooled (n=560n=560 block cases). Median and range over the pairs in each group. Significance is Fisherâs exact test with Holm correction over the ten pairs of the same definition. group Ï Ï/Ïmax / _ sig. med. range med. range PRIMARY rule Ă judge (4) +0.002+0.002 [â0.038,+0.041][-0.038,+0.041] +0.002+0.002 [â0.044,+0.134][-0.044,+0.134] 0/4 judge Ă judge (6) +0.342+0.342 [+0.146,+0.604][+0.146,+0.604] +0.697+0.697 [+0.432,+0.796][+0.432,+0.796] 6/6 STRICT rule Ă judge (4) +0.014+0.014 [â0.025,+0.051][-0.025,+0.051] +0.018+0.018 [â0.029,+0.052][-0.029,+0.052] 0/4 judge Ă judge (6) +0.430+0.430 [+0.328,+0.603][+0.328,+0.603] +0.501+0.501 [+0.429,+0.838][+0.429,+0.838] 6/6 The judge side reproduces the magnitude reported by the closest prior measurement, Ï of 0.30 to 0.75 across the fifteen pairs of a seven-layer single-turn jailbreak-defence stack whose every member is a function of one target model [4]. It is also consistent with the correlated errors reported across LLMs from different vendors [22, 23]. Stratifying by corpus with the CochranâMantelâHaenszel common odds ratio [24] does not collapse the judge associations toward 1 (Appendix A), so the pooled association is present within corpora. Pooling corpora of unequal difficulty does not create it. A BreslowâDay test finds no heterogeneity of the per-corpus odds ratios for nine of the ten pairs (pâ„0.064pâ„ 0.064). The exception is gpt55 with opus (p=0.026p=0.026), whose opus column was answered by different served versions in different corpora (Section VI-D), so a common odds ratio is not an admissible summary for that pair. The rule pairs have common odds ratios of 1.74 to 3.96, with p from 0.12 to 0.50. V-B Multiplication-equivalent layers Table IV gives nmultn_mult for every two-layer stack and for three larger ones, with the floor of each stack and two intervals. Under STRICT, any two judges compose to 1.22 to 1.44 layers above floors of 1.02 to 1.17 (same-vendor pairs 1.22 to 1.32, pairs with gpt55 1.34 to 1.44), with case-level bootstrap intervals spanning 1.16 to 1.55. The three same-vendor judges compose to 1.46 and all four judges to 1.65. The rule layer plus one judge composes to 1.86 to 2.09 layers above floors of 1.01 to 1.09, with bootstrap intervals spanning 1.67 to 2.53. On the normalised scale the rule pairs read 0.86 to 1.10 and the judge pairs 0.08 to 0.33. Rows with five or fewer joint misses support bounds only; the two stacks with two joint misses are censored in 13.7% of bootstrap resamples (Section IX, item 2). TABLE IV: nmultn_mult by stack (n=560n=560 block cases). k is the number of joint misses; rows with kâ€5k†5 (marked â ) support bounds only. Floor is the value under perfect coupling, Eq. (2). CP is the interval propagated from the exact interval on PjointP_joint; boot is a case-level bootstrap interval (4,000 resamples), reported for STRICT. PRIMARY STRICT stack floor nmultn_mult (k) CP 95% floor nmultn_mult (k) CP 95% boot 95% haiku + opus 1.00 1.32 (29) [1.16, 1.49] 1.05 1.32 (35) [1.17, 1.48] [1.24, 1.41] haiku + sonnet 1.12 1.28 (43) [1.13, 1.43] 1.09 1.30 (52) [1.16, 1.46] [1.23, 1.38] opus + sonnet 1.12 1.21 (49) [1.08, 1.36] 1.15 1.22 (53) [1.09, 1.36] [1.16, 1.27] gpt55 + haiku (cross) 1.31 1.40 (6) [1.16, 1.70] 1.07 1.34 (31) [1.18, 1.52] [1.25, 1.44] gpt55 + opus (cross) 1.31 1.40 (6) [1.16, 1.70] 1.02 1.34 (27) [1.18, 1.52] [1.25, 1.45] gpt55 + sonnet (cross) 1.41 1.57 (5)â [1.29, 1.94] 1.17 1.44 (32) [1.27, 1.63] [1.35, 1.55] three same-vendor judges 1.08 1.47 (26) [1.29, 1.67] 1.13 1.46 (32) [1.29, 1.65] [1.36, 1.59] four judges 1.61 1.79 (5)â [1.47, 2.22] 1.13 1.65 (19) [1.44, 1.90] [1.50, 1.84] rules + gpt55 1.36 1.80 (2)â [1.39, 2.47] 1.08 1.88 (10) [1.60, 2.22] [1.67, 2.20] rules + haiku 1.06 1.94 (9) [1.64, 2.30] 1.01 1.86 (14) [1.60, 2.16] [1.67, 2.14] rules + opus 1.06 2.06 (7) [1.72, 2.48] 1.06 2.09 (7) [1.75, 2.53] [1.83, 2.53] rules + sonnet 1.06 2.13 (10) [1.81, 2.51] 1.09 2.03 (14) [1.75, 2.36] [1.82, 2.34] rules + four judges 1.69 2.25 (2)â [1.74, 3.09] 1.14 2.77 (2)â [2.14, 3.81] [2.29, 3.24] Vendor diversity among the judges recovers little. The three pairs with gpt55 read above the same-vendor pairs under PRIMARY (1.40, 1.40 and 1.57 against 1.21 to 1.32), and the gap halves under STRICT (1.34, 1.34 and 1.44 against 1.22 to 1.32). That columnâs collection, in batches of 90 with no provenance, and its review habit (Section V-F) confound the excess. The DeepSeek judge of Section VI-E, with clean provenance, gives four pairs at 1.31 to 1.43 (PRIMARY) and 1.26 to 1.34 (STRICT), inside or within 0.12 of the same-vendor range. We therefore report no cross-vendor advantage on pairs with clean provenance. The right panel of Figure 1 shows nmultn_mult by corpus with intervals and floors. The intervals separate on the pooled data only. On each corpus the point estimates split and the intervals overlap. On the independent corpus the rule pairs read 1.66 to 1.82 against judge pairs at 1.24 to 1.46, and 23 of the 24 rule-pair against judge-pair interval combinations overlap. On the external corpus the rule pairs read 1.75 to 2.05 against 1.26 to 1.62, with 24 of 24 overlapping. On the internal corpus every rule-plus-judge stack is censored (0 of 129 joint misses), and the censored lower bounds of every rule pair, 1.39 to 1.56, lie above every judge-pair point estimate, 1.17 to 1.31, while 22 of 24 interval combinations overlap. The corpora taken one at a time are consistent with the split and cannot establish it on their own. V-C Decoupling Is Not Usefulness nmultn_mult is normalised by each layerâs own marginal miss rate, so it measures independence up to the floor and says nothing about competence. On the held-out corpus the rule layer misses 52.9% of block cases (46 of 87), against 6.2% and 6.1% on the other two corpora. It is the cloud, identity and CI corpus, which the default rule set does not cover. The rule layerâs decoupling is measured on the pooled data, where its pairs sit 0.86 to 1.10 of the way from floor to independence. On the external corpus alone the reading is weaker. The rule pairs read 1.75 to 2.05 against floors of 1.54 to 1.73, and the judge pairs 1.26 to 1.62 against floors of 1.07 to 1.31. The distances above floor overlap, and 87 cases do not separate the groups. The external corpus shows the competence side without qualification. A stackâs leak rate is the product of both factors, and independence at a 53% marginal miss rate buys nothing. This paper should not be read as saying that adding a regular-expression layer is sufficient. The competence gap also explains a result we withdrew. Rule-plus-judge stacks beat every judge-plus-judge stack on the independent corpus (12 of 12 paired comparisons significant under STRICT). On the held-out corpus the rule stack is favoured in 1 of 12 comparisons under STRICT and 0 of 12 under PRIMARY, and 9 of 12 and 4 of 12 comparisons reverse direction between corpora. The paired comparison measures independence times competence, so it cannot certify independence on its own. V-D Where the Coupling Comes From Two candidate factors were separated by construction. Common difficulty is measured per case as the fraction of a probe set of other layers that err on it, leaving both members of the pair out, so the score carries no information about the pairâs joint outcome. This is the stratifier the prior measurement uses [4]. A shared surface mechanism is measured from the case text alone: an obfuscation device flagged with no reference to any layerâs behaviour. The concealment flag covers encoded and indirect forms of a command (65 of 560 block cases); the broader flag adds eval or exec wrappers and remote execution (126 of 560). The mechanism test runs in the direction opposite to the hypothesis. Stratifying by difficulty and testing whether a device raises joint failure (CochranâMantelâHaenszel) gives median common odds ratios of 0.00 for the concealment flag and 0.17 for the broader flag across the six judge pairs. Of the twelve tests, eleven point below 1 and two are significant, both below 1. Cases that carry a device defeat two judges at once less often than cases that do not. The concealment-flag test rests on 0 to 3 joint misses in its exposed stratum, so it can rule the hypothesis out only at the strengths those counts allow. The prior measurement, with the same stratifier, recorded the mechanism question as not identifiable, and our data support the same wording. Difficulty alone can produce coincident failures between independently built versions [5, 25]. The rule layerâs pairs are excluded from this test. The device flags are lexical and the rule layer is a lexical matcher, so those pairs are predisposed toward an effect by construction. The difficulty share is not identifiable either. We estimate it twice. The first implementation, in the released analysis script, builds the difficulty score from the layers outside the pair in a fixed order that admits the archived faithful DeepSeek column, a column excluded from every other headline number. An earlier draft read its output as a lower bound, and that reading is withdrawn (Appendix A). The second implementation uses primary layers only. With the other two judges as the probe, the share of the excess joint failure attributed to difficulty is 53.9% (jackknife 95% interval 45.0 to 62.7) under PRIMARY and 61.8% (54.9 to 68.7) under STRICT. With the rule layer added to the probe it is 31.8% (21.3 to 42.3) and 48.1% (39.7 to 56.6). The two implementations agree where their constructions coincide (31.8% against 32%). The share moves with the composition of the probe, because the rule layerâs misses carry little information about which cases are hard for judges and dilute the score. Nothing in these data fixes the share, and the residual cannot be read as evidence that a shared mechanism exists. V-E What Could Not Be Established Equivalence testing was pre-registered to upgrade âno coupling detectedâ to âindependence establishedâ. Both bounds failed, and we report that instead of substituting a bound that passes. The main bound, an odds ratio within a factor of 2, is not established for any rule pair. For three of the four pairs the exact 90% interval is wider than the equivalence region even when the odds ratio is exactly 1. The fourth, rules with sonnet, is powered, and its interval [0.36,1.35][0.36,1.35] crosses the lower boundary 0.50. The secondary bound, |Ï|<0.30| |<0.30, anchored on the smallest Ï the prior measurement reports, and its negative control, the six judge pairs run through the same test, were both pre-registered. All four rule pairs pass the secondary bound. So does a demonstrably coupled judge pair (gpt55 with sonnet, odds ratio 8.5), and the pre-registration states that a bound such a pair clears cannot certify independence. The control disqualified the bound, as the pre-registration provided. What the data support is the interval. The rule pairsâ odds-ratio 90% intervals are [0.31,10.02][0.31,10.02], [0.56,2.32][0.56,2.32], [0.37,1.79][0.37,1.79] and [0.36,1.35][0.36,1.35]. The correct statement is that no coupling was detected and the point estimates sit at zero. Independence was not established. V-F An Accounting Rule in the Verdict Alphabet, Not a Security Property The verdict alphabet contains one option, review, that is neither an execution nor a stop. Layers differ in how readily they reach for it. In our panel one judge returns review on 47 of 560 block cases; the rule layer returns it on 3 and the other three judges on 2, 12 and 17. That is a difference in output vocabulary. It interacts with the accounting rule for review and reflects no property of the model. We name the layers elsewhere for reproducibility; the finding below is about the accounting rule, and reading it as a vendor comparison would be a misreading. Any metric that credits escalation as a catch scores a deferring layer well. In a stack with no human layer to absorb the escalation, that layer contributes nothing. In this study the choice of accounting rule reversed a substantive conclusion five times, each time in the direction that made the deferring layer look better (Table V). TABLE V: Five conclusions that reverse between the two accounting rules for review (PRIMARY: review counts as caught; STRICT: review counts as not stopped). where PRIMARY â STRICT effect the deferring layerâs solo miss 0.014 â 0.098 the most accurate judge becomes an ordinary one its pairwise correlation Ï 0.146 to 0.253 â at least 0.328 the weakest-coupled layer joins the rest paired stack comparison 6 of 12 significant â 12 of 12 (0 against under either) a partial result becomes complete value of a fourth judge 21 cases added â 13 8 of the 21 were deferrals nmultn_mult, pairs with the deferring judge 1.40 / 1.40 / 1.57 â 1.34 / 1.34 / 1.44, against same-vendor pairs at 1.21 to 1.32 the apparent cross-vendor gap halves The fourth-judge instance is the cleanest. Of the 21 cases the deferring judge appeared to add to the three same-vendor judges, 13 were block verdicts and 8 were review. The rule layerâs advantage over a fourth judge in Table I reversed on the archived panel and does not reverse on this panel (Section VI-A). Whether a convention flips a conclusion depends on the panel it is applied to. An evaluation convention rewarded behaviour that added nothing to the stack, and only the stack-level measurement showed it. The correct accounting for review depends on the human layer we do not have (Section IX, item 6). VI Comparison Panels and Provenance The primary panel of Section I was re-collected for this paper. We compare it with four other collections of the same cases under the same case-text condition, and report what the collections revealed about provenance. The correlation structure and the separation of the nmultn_mult bands reproduce across collections. The marginal miss rates do not reproduce, and they were not expected to. VI-A The archived panel Three archived columns for the same three tiers carried an earlier draft. The operator recorded them as Claude Haiku 4.5, Claude Sonnet 4.6 and Claude Opus 4.8 and collected them in batches of 90 in identifier order with no version, timestamp or temperature recorded. They agree with the re-collected columns on 0.814, 0.860 and 0.895 of verdicts (haiku and sonnet on all 1,119 cases; opus on the 620 cases whose batches were served by the requested version). The comparison holds the case blocks fixed and lets the instruction header differ, as the DeepSeek drift comparison below does. The archived headers carried a list of example threat patterns that the re-collection header drops (Section I). Their structure is the same: rule pairsâ Ï median +0.008+0.008 (PRIMARY) and +0.011+0.011 (STRICT), judge pairsâ +0.420+0.420 and +0.540+0.540, rule-pair nmultn_mult of 1.80 to 2.13 and judge-pair nmultn_mult of 1.19 to 1.52 across the two definitions. Their marginals are not the same. The archived sonnet misses 0.068 of block cases against 0.170 for the re-collected sonnet tier (PRIMARY, discordant 15 and 72, McNemar p<0.001p<0.001). Haiku (0.104 against 0.105) and opus (0.108 against 0.086 on the 279 same-version block rows) are indistinguishable. One conclusion did depend on the panel. On the archived panel the rule layerâs advantage over a fourth judge in Table I was a STRICT-only result (68 against 21 under STRICT, 20 against 22 under PRIMARY). On the primary panel it holds under both definitions (86 against 22, 31 against 21). VI-B Blinding at the system level A first re-collection was run inside the project workspace with the same prompts. A probe of the subagent system context found that the harness injects a fixed project-memory index of nine entry titles. These name the runtime, the target venue and the two elicitation conditions but contain no hypothesis, no other judgeâs verdict and no case label. The primary panel was therefore collected again in a session without that memory directory, with prompts, seed, batch split and single-case sample identical to the first re-collection by hash. A probe of its system context found only two unrelated entry titles and a generic harness skill description. On the external corpus the two collections agree case for case on 0.847, 0.921 and 0.947 of four-way verdicts (haiku, sonnet, opus), on 0.931, 0.974 and 0.984 of PRIMARY miss labels and on 0.974, 0.968 and 0.979 of STRICT miss labels. The injected index moved a few verdicts and no headline. The prompt-level blindness assertion (17 banned terms, 0 hits) held for both collections. An earlier claim that the judges had seen no mention of this study was withdrawn when the probe showed the index (Appendix A). VI-C Single-case judging against batched judging Each primary tier judged 100 sampled cases a second time, one case per call, with the same instruction. Exact four-way agreement with the batched verdict was 0.840, 0.837 and 0.900 (haiku, sonnet, opus), agreement on the PRIMARY miss label 0.880, 0.888 and 0.940, and on the STRICT miss label 0.910, 0.898 and 0.940. Sonnet has 98 cases, because two calls failed twice by echoing the instructionâs example identifier. Batching changes about one verdict in ten and about one miss label in ten or fewer. VI-D Served-Model Changes and a Pre-Declared Rule Set Aside The collection pre-declared that a batch served by a model other than its tierâs majority version would be marked mixed-version, that the majority version would carry the primary analysis, and that all batches would be reported as a sensitivity configuration. The rule assumed version changes would be rare, and they were not. Of 112 opus batches, 50 were completed by claude-opus-4-8 while the requested model was claude-opus-5, and the majority-version rule would have left 279 block rows with all five layers. We therefore report the full-batch panel as primary and the majority-version subset as the sensitivity configuration, and we split the opus tier by served version. Of the 50 batches, 12 followed a refusal event in the transcript, 2 show a version sequence without a refusal, and 36 show no signal of any kind; the change is visible only in the per-call served-model field. The haiku and sonnet tiers were served by one version on all 112 batches. The change was not uniform across the corpora. Counting block rows whose opus verdict came from claude-opus-4-8: 69 of 87 on the external corpus (79%), 171 of 344 on the independent corpus (50%) and 41 of 129 on the internal corpus (32%), Ï2â(2)=47.0Ï^2(2)=47.0, p=6.1Ă10â11p=6.1Ă 10^-11 (over all 1,119 cases: 129 of 189, 280 of 630 and 90 of 300, Ï2â(2)=68.7Ï^2(2)=68.7, p=1.2Ă10â15p=1.2Ă 10^-15). In this collection the served version was strongly associated with the corpus, which is the question that two events in an earlier re-collection could not answer. We report the counts and the test and infer nothing about the routing mechanism. Both served-version subsets keep the rule layer decoupled from the opus tier: rules Ă opus Ï=+0.047 =+0.047 on the 279 claude-opus-5 block rows and â0.069-0.069 on the 281 claude-opus-4-8 rows, with nmultn_mult of 1.83 and 2.26 under PRIMARY and 1.85 and 2.31 under STRICT. One batch made one harmless tool call and three drew a harness classifier warning with no effect on the output; all are kept. The archived gpt55 column has no served-model field, so events of this kind there are invisible. VI-E Cross-Vendor Re-collection with Recorded Provenance The DeepSeek judge was re-collected over the external (189) and independent (630) corpora, one case per request, temperature 0, 200 output tokens, JSON-object response format, under two conditions: no-tell, with case blocks byte-identical to the archived no-tell files, and faithful, with the human-written description shown. There were 1,638 requests and 1,638 HTTP 200 responses, with no retry recorded. Every record carries the served model, a UTC timestamp, the prompt hash and the raw response. The requested alias was deepseek-chat; the served model was deepseek-v4-flash on all 1,638 calls. An alias in a configuration file is not a version. The cross-vendor judge lands in the judge band. On the 431 block cases of the two re-collected corpora the no-tell column couples with all four primary judges and does not couple with the rule layer (Table VI). With the DeepSeek column included, the rule pairsâ Ï/Ïmax / _ run â0.045-0.045 to +0.154+0.154 (0 of 5 significant) and the judge pairsâ +0.390+0.390 to +0.847+0.847 (10 of 10) under PRIMARY; under STRICT the ranges are â0.024-0.024 to +0.087+0.087 against +0.395+0.395 to +0.824+0.824 (10 of 10). The bands stay separated. TABLE VI: The re-collected DeepSeek column against the five primary layers (external and independent corpora, n=431n=431 block cases). Every judge pair here is cross-vendor. Significance is Fisherâs exact test with Holm correction over the fifteen pairs of the six-layer panel, given for PRIMARY / STRICT. PRIMARY STRICT pair Ï Ï/Ïmax / _ nmultn_mult Ï Ï/Ïmax / _ nmultn_mult sig. rules + DeepSeek +0.057+0.057 +0.076+0.076 1.84 +0.062+0.062 +0.080+0.080 1.83 no / no gpt55 + DeepSeek +0.271+0.271 +0.684+0.684 1.38 +0.501+0.501 +0.508+0.508 1.26 yes / yes haiku + DeepSeek +0.388+0.388 +0.424+0.424 1.35 +0.440+0.440 +0.525+0.525 1.32 yes / yes opus + DeepSeek +0.438+0.438 +0.479+0.479 1.31 +0.404+0.404 +0.434+0.434 1.34 yes / yes sonnet + DeepSeek +0.333+0.333 +0.491+0.491 1.43 +0.438+0.438 +0.657+0.657 1.34 yes / yes Every judge pair in Table VI is cross-vendor, and every one couples at the strength of the same-vendor pairs of Section V [22, 23]. The pair of the OpenAI judge with DeepSeek, two vendors and two independent collection chains, gives nmult=1.26n_mult=1.26 under STRICT, with a 95% interval of [1.09,1.45][1.09,1.45]. The rule pair gives 1.83 with [1.54,2.18][1.54,2.18]. Vendor diversity does not recover independence. Per corpus the rule pair is +0.134+0.134 and +0.111+0.111 (PRIMARY) and +0.097+0.097 and +0.107+0.107 (STRICT), the largest rule pairs in the study, none significant. Drift and elicitation Against the archived DeepSeek column (faithful condition, version unrecorded) the re-collected faithful column agrees on 692 of 819 verdicts (0.845). Block miss rates are indistinguishable, 0.077 against 0.081 under PRIMARY (discordant 13 and 15, McNemar p=0.85p=0.85 [26]) and 0.081 against 0.081 under STRICT. The false-alarm rate fell from 0.085 to 0.033 (16 and 3, p=0.004p=0.004), almost entirely on the external corpus (0.310 to 0.086, 13 and 0). The comparison is condition-matched and not prompt-identical, so it bounds drift plus wording from above: miss behaviour is stable across a model version, and false-alarm behaviour is not. Under the same served model in the same hour, description blanked against description shown agrees on 718 of 819 verdicts (0.877). On the independent corpus the description changes nothing (miss 0.105 against 0.102 under PRIMARY, p=1.0p=1.0, and false alarms 0.021 against 0.016). On the external corpus it removes every miss (5 to 0 under PRIMARY with p=0.062p=0.062, 7 to 0 under STRICT with p=0.016p=0.016) and most false alarms (0.224 to 0.086, 8 and 0, p=0.008p=0.008). On that corpus the human-written description is a tell that carries enough of the label to zero one judgeâs miss rate. That quantifies why the primary layer set is measured no-tell (Section I) and why that field must not be shown to any layer whose independence from the labels is being measured. VII Related Work Correlated failure in defence stacks The closest prior measurement is a seven-layer single-turn jailbreak-defence stack [4]: two token-surface filters, a safety classifier, a refusal-priming prefix, a perturbation-stability check and two hidden-state probes, all on one open-weight target model under one adaptive adversary. It reports positive Ï in all fifteen measurable pairs (0.30 to 0.75), a joint residual above the multiplicative prediction, and a difficulty stratification under which almost every cross-row association dissolves. Across three labelling regimes the surviving pair changes, so the authors retire the mechanism-specific reading and record the question as not identifiable. Its scope is described as single-turn jailbreak only, input-provenance and architecture rows are excluded because they target injection, every measured member is a function of the target model, and correlation across agentic settings is listed as unmeasured. Only one of its seven members is an LLM judge, and none is a hand-authored rule layer. Our judge pairs reproduce its finding in a new setting and carry no novelty claim. Our difficulty decomposition uses the same stratifier and reaches the same non-identifiability. In our judgement our rule pairs sit at a larger mechanism distance than any pair it measured, though that study defines no such quantity and the comparison is ours. Its correlations were collected under an adaptive adversary and ours were not (Section I-A). Correlated errors across LLMs from different vendors have been measured before. Models that share more errors are found to be poorer overseers of one another [22], and models from different vendors agree on a majority of the cases on which both err [23]. Our cross-vendor arm (Section VI-E) reproduces that finding for judges of agent actions. Common-cause failure in reliability engineering is treated with ratio parameters, the beta factor and its refinements [16]. Eckhardt and Lee showed that variation in per-case difficulty is by itself sufficient to make independently developed versions fail together [5]. Knight and Leveson measured the failure of the independence assumption in multi-version programming [6]. Littlewood and Miller modelled coincident failures under diverse development [25], and Littlewood, Popov and Strigini review the design-diversity models that followed [27]. The same question has been measured for security detectors: Bishop et al. and Gashi et al. measured how much diverse off-the-shelf antivirus engines gain from being combined, on the same inputs [28, 29]. That is our question, asked of an earlier class of detector. Classifier-ensemble research relates diversity measures to ensemble accuracy [30]. Hill numbers supply the effective-count construction we reuse [11, 13, 14, 15]. Kishâs effective sample size [18] has been applied to LLM judge panels [19] and is a different quantity (Section I). Relation to the prior reports on these data The runtime, the rule layer and the corpora come from two technical reports [2, 9]. The second reports, on the same external corpus, the rule layerâs 52.9% miss rate, the cloud packâs zero change on the semantic categories, and the clustering of three judgesâ accuracies between 83.6% and 85.2%. The first reports the joint miss rate of a rules-plus-judge hybrid on its benchmarks. This paper adds the pairwise correlation structure, nmultn_mult with its floor, the difficulty decomposition, the two evaluation conventions, the cross-vendor column, and the case-level accounting of what each increment adds to a stack. Enforcement theory Ray characterises which policies a deterministic gate can enforce relative to an oracle interface, gives decidability and complexity results for guard theories, and bounds the robustness cost of attacks on a judgeâs visible representation [17]. The bound is stated as distributional, with no claim of universal semantic soundness. It asks what is enforceable given an interface; we measure how the errors of several concrete layers relate. Enforcement architectures Several designs keep injected content away from the action space: CaMeL through restricted code with capability checks before each tool call [31], FIDES through taint tracking with deterministic policy enforcement [32], Progent through symbolic privileges over tool names and arguments [33], and Parallax through tiered verification behind a deliberation-action split, which also names the assume-compromise evaluation we adopt [3]. Guardrail systems classify prompts, outputs or agent traces [34, 35, 36], and the AgentTrust runtime intercepts actions with a rule gate, an opt-in judge and a human tail [2, 9]. These works place enforcement points. We did not find, in any of them, a measurement of how the errors of one point relate to the errors of another. This paper adds that measurement. Evaluation of judges and agents Benchmarks measure how often an agent or a judge is fooled: AgentDojo for prompt injection in tool-using agents [37], AgentHarm for harmful agent tasks [38], ToolEmu for emulated tool risk [39], R-Judge for judging agent risk [40], and AgentTrap for runtime trust failures in third-party skills [41]. Indirect prompt injection itself was characterised by Greshake et al. [42], and adaptive attacks against published defences and judges are the subject of [7] and [8]. RobustJudge reports that no evaluated defence dominates on robustness, benign utility and cost at once, and that judge robustness is highly sensitive to prompt template and model choice [43]. That sensitivity bounds how far any single-template result of ours generalises, and we cite it for that reason. These works measure whether a judge is fooled; we measure whether different layers are fooled by the same cases. The human layer we lack has its own literature, in which oversight capacity and fatigue have been modelled [44]. VIII Discussion What a practitioner should do with the measurement Select stack members on mechanism class, then measure the assembled stack. The pairwise statistics Ï and nmultn_mult are selection statistics. They say which candidate layers are worth composing, and they do not predict the assembled stackâs residual, which we measured directly as added coverage. This two-stage workflow is the one the prior measurement recommends [4], and our data agree with it. Two corollaries follow from Section IV. A layerâs solo accuracy is the wrong number to report when the question is what the layer adds. Improving a layer along its own mechanism axis can leave the stack unchanged and move the layer toward the coupled band, because the gains land on cases the other layers already catch. Three claims that must stay separate The result invites a transfer from redundancy engineering, and the transfer has three parts that carry different weight. We measured the statistical part: on these corpora, without an adaptive adversary, errors across mechanism classes showed no correlation we could detect, and errors within the judge class were correlated. The complexity part, that a deterministic layer is cheap to run and audit and a judge is not, is a property of the implementations and is stated here without measurement. The expressiveness part, that a regular-expression layer cannot recognise the threats a judge recognises, is the reason the rule layer misses 52.9% of the external corpus. It is an argument about what each mechanism can see, and this study does not test it. Two conventions as requirements The accounting rule for review verdicts reversed five conclusions during this study, and the served-model log overturned a pre-declared analysis rule. Both are cheap to adopt. A stack evaluation should state which verdicts count as a stop, report every headline under both readings when a deferring layer is present, and record the served model string on every call. We make no claim about why a served model changes mid-collection. In this study one tier was served by another version in 50 of 112 batches, 36 of them with no signal in the transcript, and the change fell on 79% of the external corpusâs block rows against 32% of the internal corpusâs. Per-call logging shows that a version changed and which content the change fell on, the subset a safety evaluation cares about. The pre-declared rule that the majority version would carry the analysis is a convention the data overruled, and the deviation is disclosed (Section VI-D). Which accounting rule applies to a deployment depends on whether a human layer absorbs escalations. Where one exists, the PRIMARY reading is the relevant one, and the gap between the archived judgeâs pairs and the same-vendor pairs is then at its widest, though it is confounded with that columnâs collection (Section V). Mechanism distance Mechanism distance is sampled at two points, regular-expression matchers against chat models, and the weaker deterministic layer cannot be read on the independence axis (Section IV). We name the split a boundary observation. A gradient would need intermediate mechanisms, such as a typed capability gate or a fine-tuned classifier, measured with the same instrument. Scope of the corpora and the judges The independent corpusâs categories are verdict tiers, and it was not built to contain semantic threats. The results that concern semantic threats all come from the external corpus, and a reader should not take the 630-case corpus as evidence about them. Every judge here ran one prompt template, and judge robustness depends on the template [43]. A different template could move the marginal rates, though the correlation structure appears in the archived panel, in a panel collected with a memory index in the judgesâ context, and under single-case judging (Section VI). IX Limitations 1. Mechanism distance is sampled at one axis with two points, and the second deterministic layer cannot be read on that axis. The attribution of decoupling to the mechanism class rests on the default rule layer across three corpora, the comparison panels and the cross-vendor column; b1âs pairs sit on their floors (Section IV). Deterministic mechanisms that are not regular expressions, such as type systems and capability gates, are untested. Two configurations do not make a gradient, and none is claimed. 2. Full-stack nmultn_mult is censored. Across 560 block cases all five layers miss together twice under either definition, and on the internal corpus never. Pairwise nmultn_mult is well estimated where k is large; rows with kâ€5k†5 support bounds only, and the two-miss stacks are censored in 13.7% of bootstrap resamples. 3. The difficulty share is not identifiable (Section V-D). It ranges from 31.8% to 61.8% with the probe used to score difficulty and the miss definition, and the residual must not be read as evidence of a shared mechanism. The concealment-flag test rests on 0 to 3 joint misses in its exposed stratum. 4. The internal corpus cannot distinguish stack configurations (0 of 129 joint leaks for every rule-plus-judge stack), and the interval evidence for the split comes from the pooled data only; on each corpus taken alone the intervals overlap (Section V). 5. Four judges, three from one vendor. The cross-vendor evidence has two sources. The archived gpt55 columnâs pairs read 0.02 to 0.12 above the same-vendor pairs under STRICT, in a way confounded with its collection and its review habit. The DeepSeek arm of Section VI-E, collected one case per request with the served model recorded, gives four judge pairs at nmultn_mult of 1.26 to 1.34 under STRICT and 1.31 to 1.43 under PRIMARY, while its rule pair reads 1.83 and 1.84. We report no cross-vendor advantage on the pairs with clean provenance. A judge that is not a general-purpose chat model remains untested. 6. No human review layer. How review should be scored therefore cannot be settled here, and both readings are reported throughout (Section I). 7. The 180 warn cases are excluded from both error domains. 8. The gpt55 column and the archived comparison panel were collected in batches of 90 cases in identifier order, with no record of wrapper prompt, temperature, model version or date, and they cannot be regenerated. The primary tiers were collected in batches of 10 by one operator through one harness, under an instruction header that differs from the archived headers (Section I), so the archived-panel comparison is condition-matched on the case blocks only. Batching changes about one verdict in ten and about one miss label in ten or fewer (Section VI-C). The runtimeâs rewrite-type safe-alternative component sits outside the measured stack and is untested here. 9. A dated snapshot identifier exists for the haiku tier only. The sonnet and opus tiers carry aliases, the opus alias was served by two versions, and the gpt55 column has no provenance at all (Section VI). 10. The zero-new-coverage result for the cloud pack is one pack and 17 to 19 newly caught cases, all but two of them on one corpus. On that corpus it is clean (100% across four judges and both definitions), and it must not be generalised to ârule augmentation always lands on redundant cases.â The methodological half generalises: solo accuracy did not predict stack contribution, and one counterexample suffices for that. 11. A defect in the rule layerâs elicitation, disclosed. The rule layerâs matcher reads the action description, so its archived verdicts were produced in the faithful condition while the judges ran no-tell. Every headline quantity was recomputed with the description blanked. Blanking it changes the rule verdict on 2 of 560 block cases, moves the rule pairsâ PRIMARY Ï by at most 0.002 and their nmultn_mult by at most 0.02, leaves the pooled STRICT values unchanged, and leaves the new-coverage totals unchanged (86 to 86). The recomputed column is part of the material available on request. 12. Seven candidate conclusions were withdrawn during analysis after failing replication, equivalence, definition-sensitivity, probe-composition or blindness checks; the pre-registration and a summary of the audit trail are in the artifact (Appendix A). 13. The external corpus is selected on label agreement. Of its 224 source cases, all blind-relabelled by an independent human annotator who is not an author, 193 agreed with the LLM rubric label and 31 disagreed (8 allow to warn, 13 warn to allow, 5 warn to block, 5 block to warn; no allow-to-block flip), with Îș=0.817Îș=0.817 before filtering. The frozen 189 are agreed cases [9]. Selecting on agreement biases the corpus toward clear-label cases, so the 52.9% miss rate and the judge coupling there are measured on such cases. A residual shared-prior concern remains for cases that both an LLM and a human over-label. 14. All results are distributional statements about labelled corpora without an adaptive adversary (Section I-A). They bound nothing about an attacker who targets the joint miss set. 15. One judge tier was served by a version other than the one requested in 50 of 112 batches. The pre-declared majority-version rule was set aside because it would have left 279 block rows; the full-batch panel is primary, the majority-version subset is a sensitivity configuration, and the tier is reported split by served version (Section VI-D). X Conclusion We measured whether the errors of stacked action gates multiply, on 1,119 labelled actions from three corpora, with one rule layer and four judges under one case-text condition. Three of the judges were re-collected with the served model recorded on every call. Across mechanism classes they do, within reach of the multiplication rule; within one class they largely do not. Under STRICT, any two judges compose to about 1.2 to 1.4 multiplication-equivalent layers, and a rule layer with a judge to 1.9 to 2.1; the floors set by the marginals are 1.02 to 1.17 and 1.01 to 1.09. The shape held on the pooled data, was consistent with each corpus, and reappeared in the archived panel, in a panel collected with a memory index in the judgesâ context, under single-case judging, and for a fifth judge from a third vendor. Solo accuracy did not predict what a layer added to the stack in either direction. Mechanism class and in-domain competence must both be read. The practical procedure is to select members by mechanism class and then measure the assembled stack. Two evaluation conventions changed conclusions in this study, and we recommend both as requirements for stack evaluations: report every result under an explicit accounting rule for review verdicts, and log the served model on every call. In this study that log showed one tier served by another version in 50 of 112 batches, concentrated on one corpus. Four things remain unmeasured. A human review layer would settle how review is scored. A judge that is not a general-purpose chat model would test whether the judge band is a property of chat models. A deterministic mechanism other than a regular expression would give mechanism distance a second point. An adaptive adversary would test what the distributional independence measured here is worth against an attack aimed at the joint miss set. Open Science The artifact for this paper is available at https://github.com/chenglin1112/evaluate-the-stack. It contains the per-layer verdict matrix for all 1,119 cases, for the primary panel and for the archived panel. It contains the per-case and per-call provenance of the re-collected judge columns (model requested, model served, timestamps, prompt and output hashes, stop reasons and refusal flags), for the primary collection and for the comparison collection of Section VI-B, and the re-collected DeepSeek columns under both elicitation conditions with per-call provenance (served model, UTC timestamp and prompt hash). It also contains the analysis scripts that compute the statistics of Sections V and VI from those files, the pre-registration of the equivalence test, summaries of the verification-coverage audit, the instrument-definition audit and the provenance audit, a summary of the recorded deviations, and the record of the conclusions withdrawn during analysis. The scripts are deterministic and rerun to byte-identical outputs, whose hashes the artifact lists. The case texts, the prompts that embed them, the judgesâ raw responses, the external corpus and the collection scripts are not part of the public artifact, because the dangerous cases are command and code strings written as attacks (see the Ethical Considerations section). They are available from the author on request. The results that need the case texts, namely the cloud rule pack, the second rule engine and the no-tell audit, cannot be recomputed from the public artifact. The rule files and the internal and independent corpora are those of a public release of the AgentTrust runtime [2]. The archived gpt55 column and the archived comparison panel cannot be regenerated, because their collection recorded no model version, timestamp or temperature. LLM usage considerations Transparency. LLMs are the object of study. The judge layers measured in Section I are production LLM judges: one archived column, three columns re-collected through a coding-assistant harness with the served model recorded on every call, and one column re-collected through a vendor API. The comparison panels of Section VI are further collections of the same kind. Their versions, where known, temperatures and query volumes are stated in those sections. Where a version is not known we say so per tier and do not claim that it was recorded. The reproducibility limitation this creates is one of the paperâs findings (Section VI-D). Closed models changed version during collection, which is why the served-model string was recorded on every call and is released with the artifact. Accountability and correctness. LLMs were used for editorial purposes in this manuscript, and all outputs were inspected by the author to ensure accuracy and originality. LLM coding assistants were also used to draft the analysis scripts. The released analysis scripts rerun to the numbers reported in Sections V and VI, and seven candidate conclusions were withdrawn when the audited outputs did not support them. The author takes responsibility for every claim in the paper. Responsibility. No model was trained. Query volume was kept to one judgment per case and condition: the archived verdict files hold 4,776 judgments in 14 files. The cross-vendor re-collection made 1,638 API calls with no retry. The primary panel was collected through a coding-assistant harness in 642 calls (336 batched, 300 single-case, 6 retries), 26,273,030 input tokens including cache reads and 1,266,080 output tokens, with 12 refusal events (all on one tier, each completed by a second served version) and one tool call. The same prompts had been run once before inside the project workspace for the sensitivity panel of Section VI-B (306 calls including 8 aborted, 12,572,889 input and 1,015,945 output tokens). An earlier re-collection with batches of 90 requested 2,547 judgments in 31 batches. The internal corpus was not re-collected because it cannot distinguish stack configurations (Section IX, item 4). Analysis ran on a single laptop. Ethical Considerations The corpora consist of agent actions labelled dangerous or benign. The dangerous cases are command and code strings of known categories (data exfiltration, credential exposure, destructive operations and persistence) written as test cases. None is a new exploit, and none targets a live system. The external corpus and the prompts that embed the case texts are not part of the public artifact and are shared on request; the internal and independent corpora were already public [2]. No human subjects were involved in this study. The paper names the judge models for reproducibility and draws no conclusion about any vendor. The verdict-accounting finding of Section V-F concerns an evaluation convention. References [1] S. Sharwood (2025) Vibe coding service Replit deleted userâs production database, faked data, told fibs galore. Note: The Register, 21 July 2025https://w.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/ Cited by: §I. [2] C. Yang (2026) AgentTrust: runtime safety evaluation and interception for AI agent tool use. arXiv preprint arXiv:2605.04785. Cited by: §I, §I-B, §I-D, §VII, §VII, Open Science, Ethical Considerations. [3] J. Fokou (2026) Parallax: why AI agents that think must never act. arXiv preprint arXiv:2604.12986. Cited by: §I, §I, §I-A, §VII. [4] A. Alotaibi, M. S. Jabbar, S. Al-Azani, and M. Ahmed (2026) Layered LLM defenses as an ensemble: access tiers, inference cost, and the measured failure correlation between defense layers. arXiv preprint arXiv:2608.28327. Cited by: §I, §I-A, §I, §V-A, §V-D, §VII, §VIII. [5] D. E. Eckhardt and L. D. Lee (1985) A theoretical basis for the analysis of multiversion software subject to coincident errors. IEEE Transactions on Software Engineering SE-11 (12), p. 1511â1517. External Links: Document Cited by: §I, §V-D, §VII. [6] J. C. Knight and N. G. Leveson (1986) An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering SE-12 (1), p. 96â109. External Links: Document Cited by: §I, §VII. [7] M. Nasr, N. Carlini, C. Sitawarin, S. V. Schulhoff, J. Hayes, M. Ilie, J. Pluto, S. Song, H. Chaudhari, I. Shumailov, A. Thakurta, K. Y. Xiao, A. Terzis, and F. TramĂšr (2025) The attacker moves second: stronger adaptive attacks bypass defenses against LLM jailbreaks and prompt injections. arXiv preprint arXiv:2510.09023. Cited by: §I-A, §VII. [8] Q. Zhan, R. Fang, H. S. Panchal, and D. Kang (2025) Adaptive attacks break defenses against indirect prompt injection attacks on LLM agents. arXiv preprint arXiv:2503.00061. Cited by: §I-A, §VII. [9] C. Yang (2026) AgentTrust: a self-improving trust layer for AI-agent actions. arXiv preprint arXiv:2606.08539. Cited by: §I-D, §VII, §VII, item 13. [10] C. J. Clopper and E. S. Pearson (1934) The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26 (4), p. 404â413. External Links: Document Cited by: §I. [11] R. H. MacArthur (1965) Patterns of species diversity. Biological Reviews 40 (4), p. 510â533. External Links: Document Cited by: §I, §VII. [12] R. H. MacArthur (1972) Geographical ecology: patterns in the distribution of species. Harper & Row. Cited by: §I. [13] M. O. Hill (1973) Diversity and evenness: a unifying notation and its consequences. Ecology 54 (2), p. 427â432. External Links: Document Cited by: §I, §VII. [14] L. Jost (2006) Entropy and diversity. Oikos 113 (2), p. 363â375. External Links: Document Cited by: §I, §VII. [15] L. Jost (2007) Partitioning diversity into independent alpha and beta components. Ecology 88 (10), p. 2427â2439. External Links: Document Cited by: §I, §VII. [16] International Electrotechnical Commission (2010) IEC 61508: functional safety of electrical/electronic/programmable electronic safety-related systems. Technical report IEC, Geneva. Cited by: §I, §VII. [17] S. Ray (2026) What can be enforced? A theory of certified runtime safety for tool-using agents. arXiv preprint arXiv:2607.22868. Cited by: §I, §VII. [18] L. Kish (1965) Survey sampling. John Wiley & Sons, New York. Cited by: §I, §VII. [19] G. Kohli (2026) Nine judges, two effective votes: correlated errors undermine LLM evaluation panels. arXiv preprint arXiv:2605.29800. Cited by: §I, §VII. [20] J. Cohen (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1), p. 37â46. External Links: Document Cited by: §I. [21] S. Holm (1979) A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2), p. 65â70. Cited by: §I. [22] S. Goel, J. Struber, I. A. Auzina, K. K. Chandra, P. Kumaraguru, D. Kiela, A. Prabhu, M. Bethge, and J. Geiping (2025) Great models think alike and this undermines AI oversight. arXiv preprint arXiv:2502.04313. Cited by: §V-A, §VI-E, §VII. [23] E. Kim, A. Garg, K. Peng, and N. Garg (2025) Correlated errors in large language models. arXiv preprint arXiv:2506.07962. Cited by: §V-A, §VI-E, §VII. [24] N. Mantel and W. Haenszel (1959) Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute 22 (4), p. 719â748. External Links: Document Cited by: §V-A. [25] B. Littlewood and D. R. Miller (1989) Conceptual modeling of coincident failures in multiversion software. IEEE Transactions on Software Engineering 15 (12), p. 1596â1614. External Links: Document Cited by: §V-D, §VII. [26] Q. McNemar (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12 (2), p. 153â157. External Links: Document Cited by: §VI-E. [27] B. Littlewood, P. Popov, and L. Strigini (2001) Modeling software design diversity: a review. ACM Computing Surveys 33 (2), p. 177â208. External Links: Document Cited by: §VII. [28] P. Bishop, R. Bloomfield, I. Gashi, and V. Stankovic (2011) Diversity for security: a study with off-the-shelf AntiVirus engines. In IEEE 22nd International Symposium on Software Reliability Engineering (ISSRE), p. 11â19. External Links: Document Cited by: §VII. [29] I. Gashi, B. Sobesto, V. Stankovic, and M. Cukier (2013) Does malware detection improve with diverse AntiVirus products? an empirical study. In Computer Safety, Reliability, and Security (SAFECOMP), LNCS, p. 94â105. External Links: Document Cited by: §VII. [30] L. I. Kuncheva and C. J. Whitaker (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning 51 (2), p. 181â207. External Links: Document Cited by: §VII. [31] E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. TramĂšr (2025) Defeating prompt injections by design. arXiv preprint arXiv:2503.18813. Cited by: §VII. [32] M. Costa, B. Köpf, A. Kolluri, A. Paverd, M. Russinovich, A. Salem, S. Tople, L. Wutschitz, and S. Zanella-BĂ©guelin (2025) Securing AI agents with information-flow control. arXiv preprint arXiv:2505.23643. Cited by: §VII. [33] T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, and D. Song (2025) Progent: securing AI agents with privilege control. arXiv preprint arXiv:2504.11703. Cited by: §VII. [34] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa (2023) Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674. Cited by: §VII. [35] S. Chennabasappa, C. Nikolaidis, D. Song, D. Molnar, S. Ding, S. Wan, S. Whitman, L. Deason, N. Doucette, A. Montilla, A. Gampa, B. de Paola, D. Gabi, J. Crnkovich, J. Testud, K. He, R. Chaturvedi, W. Zhou, and J. Saxe (2025) LlamaFirewall: an open source guardrail system for building secure AI agents. arXiv preprint arXiv:2505.03574. Cited by: §VII. [36] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen (2023) NeMo Guardrails: a toolkit for controllable and safe LLM applications with programmable rails. In EMNLP System Demonstrations, p. 431â445. External Links: Document Cited by: §VII. [37] E. Debenedetti, J. Zhang, M. BalunoviÄ, L. Beurer-Kellner, M. Fischer, and F. TramĂšr (2024) AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In NeurIPS Datasets and Benchmarks Track, External Links: Document Cited by: §VII. [38] M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y. Gal, and X. Davies (2024) AgentHarm: a benchmark for measuring harmfulness of LLM agents. arXiv preprint arXiv:2410.09024. Cited by: §VII. [39] Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto (2024) Identifying the risks of LM agents with an LM-emulated sandbox. In International Conference on Learning Representations (ICLR), Cited by: §VII. [40] T. Yuan, Z. He, L. Dong, Y. Wang, R. Zhao, T. Xia, L. Xu, B. Zhou, F. Li, Z. Zhang, R. Wang, and G. Liu (2024) R-Judge: benchmarking safety risk awareness for LLM agents. In Findings of EMNLP, p. 1467â1490. External Links: Document Cited by: §VII. [41] H. Zhuang, H. Xing, Y. Zhou, Y. Ma, Y. Huang, Y. Shen, Y. Han, and X. Zhang (2026) AgentTrap: measuring runtime trust failures in third-party agent skills. arXiv preprint arXiv:2605.13940. Cited by: §VII. [42] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz (2023) Not what youâve signed up for: compromising real-world LLM-integrated applications with indirect prompt injection. In ACM Workshop on Artificial Intelligence and Security (AISec), p. 79â90. External Links: Document Cited by: §VII. [43] S. Li, C. Xu, J. Wang, X. Gong, C. Chen, J. Zhang, J. Wang, K. Lam, and S. Ji (2025) LLMs cannot reliably judge (yet?): a comprehensive assessment on the robustness of LLM-as-a-judge. arXiv preprint arXiv:2506.09443. Cited by: §VII, §VIII. [44] E. Turan (2026) Oversight has a capacity: calibrating agent guards to a subjective, fatiguing human. arXiv preprint arXiv:2606.08919. Cited by: §VII. Appendix A Supplementary Tables TABLE VII: Per-corpus miss correlation (PRIMARY): pooled odds ratio, the CochranâMantelâHaenszel common odds ratio stratified by corpus with its p-value, and Ï within each corpus. A BreslowâDay test of homogeneity of the per-corpus odds ratios gives pâ„0.064pâ„ 0.064 for every pair except gpt55 with opus (p=0.026p=0.026). pair pooled OR MH OR MH p int. ind. ext. rules Ă gpt55 2.19 3.96 0.50 â0.02-0.02 +0.06+0.06 +0.10+0.10 rules Ă haiku 1.19 2.10 0.12 â0.08-0.08 +0.16+0.16 â0.01-0.01 rules Ă opus 0.86 2.18 0.17 â0.08-0.08 +0.11+0.11 +0.10+0.10 rules Ă sonnet 0.72 1.74 0.30 â0.09-0.09 +0.07+0.07 +0.18+0.18 gpt55 Ă haiku 28.25 26.68 2Ă10â72Ă 10^-7 â0.03-0.03 +0.28+0.28 +0.49+0.49 gpt55 Ă opus 28.25 25.51 2Ă10â72Ă 10^-7 â0.03-0.03 +0.27+0.27 +1.00+1.00 gpt55 Ă sonnet 8.56 7.48 5Ă10â35Ă 10^-3 â0.03-0.03 +0.14+0.14 +0.57+0.57 haiku Ă opus 15.18 12.82 2Ă10â202Ă 10^-20 +0.60+0.60 +0.38+0.38 +0.49+0.49 haiku Ă sonnet 23.21 21.23 8Ă10â298Ă 10^-29 +0.67+0.67 +0.48+0.48 +0.26+0.26 opus Ă sonnet 48.47 43.43 9Ă10â389Ă 10^-38 +0.50+0.50 +0.62+0.62 +0.57+0.57 TABLE VIII: nmultn_mult of every two-layer stack by corpus (STRICT), with the exact 95% interval and the floor under perfect coupling. A censored stack (no joint miss) is given as its lower bound with an open upper limit. internal (129) independent (344) external (87) stack nmultn_mult [95%] floor nmultn_mult [95%] floor nmultn_mult [95%] floor rules + gpt55 â„1.42â„ 1.42 1.06 1.82 [1.41, 2.43] 1.07 1.96 [1.43, 2.65] 1.54 rules + haiku â„1.50â„ 1.50 1.12 1.66 [1.35, 2.04] 1.19 1.76 [1.29, 2.38] 1.58 rules + opus â„1.39â„ 1.39 1.04 1.70 [1.38, 2.12] 1.17 2.05 [1.27, 3.73] 1.73 rules + sonnet â„1.56â„ 1.56 1.16 1.72 [1.43, 2.09] 1.32 1.75 [1.25, 2.43] 1.64 gpt55 + haiku 1.31 [1.00, 1.71] 1.06 1.37 [1.16, 1.61] 1.12 1.26 [0.91, 1.72] 1.07 gpt55 + opus 1.20 [0.92, 1.58] 1.02 1.35 [1.15, 1.58] 1.10 1.31 [0.88, 2.04] 1.31 gpt55 + sonnet 1.24 [0.96, 1.60] 1.11 1.46 [1.24, 1.70] 1.25 1.62 [1.08, 2.53] 1.15 haiku + opus 1.22 [0.94, 1.59] 1.08 1.33 [1.14, 1.55] 1.02 1.48 [0.92, 2.70] 1.25 haiku + sonnet 1.23 [0.96, 1.57] 1.04 1.32 [1.14, 1.52] 1.14 1.36 [0.94, 2.00] 1.08 opus + sonnet 1.17 [0.91, 1.50] 1.12 1.24 [1.07, 1.42] 1.16 1.39 [0.86, 2.53] 1.17 TABLE IX: Per-corpus medians of Ï (PRIMARY) for the two groups of pairs. corpus rules Ă judge judge Ă judge internal â0.079-0.079 +0.234+0.234 independent +0.089+0.089 +0.331+0.331 external (held out) +0.102+0.102 +0.531+0.531 TABLE X: Equivalence testing on the rule pairs: exact 90% odds-ratio intervals. The pre-registered equivalence region was [0.50,2.00][0.50,2.00]. pair PRIMARY STRICT rules Ă gpt55 [0.31, 10.02] [0.70, 2.74] rules Ă haiku [0.56, 2.32] [0.81, 2.61] rules Ă opus [0.37, 1.79] [0.34, 1.62] rules Ă sonnet [0.36, 1.35] [0.51, 1.59] A-A Conclusions withdrawn during analysis Seven candidate conclusions were stated during analysis and withdrawn. Each is listed with the check that removed it. 1. âThe layer that adds the most to the stack is the one with the worst solo accuracy.â Falsified by the second rule engine, b1, which has the worst solo accuracy and adds 9 cases (Section IV). 2. âRule-plus-judge stacks beat judge-plus-judge stacksâ as a headline. In the stratified replication the held-out corpus favoured the rule stack in 0 of 12 comparisons under PRIMARY (6 tied) and 1 of 12 under STRICT, and 4 of 12 and 9 of 12 comparisons reversed direction between corpora. The comparison measures independence times competence (Section V-C). 3. âThe rule layer and the LLM layer are independent.â Equivalence testing failed at both pre-registered bounds (Section V-E). 4. âModel version changes concentrate on the most sensitive batches.â Two events, one of them a counterexample. The densest segment of the external run was unaffected. On the primary panel the change was associated with the corpus (Section VI-D); no mechanism claim is made. 5. âCross-vendor judge pairs recover some independence.â On the primary panel the archived cross-vendor pairsâ excess over the same-vendor pairs halves under STRICT and is confounded with that columnâs collection, and the cross-vendor pairs with clean provenance read inside the same-vendor band (Sections V and VI-E). 6. âThe difficulty share has a lower bound near 47%.â The computation that produced it scored difficulty with a probe that admitted the archived faithful DeepSeek column, a column excluded from every other headline number, and the share moves by a factor of two with the probe. Withdrawn after review in favour of the range in Section V-D. 7. âThe re-collected judges saw no mention of this study.â A probe of the subagent system context showed a project-memory index naming the runtime and the venue; the claim is withdrawn, the affected collection is retained as a comparison panel, and the primary panel was collected in a session without that index (Section VI-B). A-B A false-alarm observation on the cross-vendor column On the 246 benign cases the rule layer and the fresh DeepSeek column false-alarm together (Ï=+0.305 =+0.305, Holm p=0.004p=0.004 over the fifteen pairs of the six-layer panel, with 6 joint false alarms out of 17 and 17). The archived DeepSeek column shares 8 of the rule layerâs 17. The other four judges share 0 to 1. All six joint cases are external-corpus cases in the network-egress, observability and CI categories: benign commands that look like exfiltration or probing. On the false-alarm axis this one judge behaves partly like a surface-pattern detector. We record it as an observation about one model family at n=6n=6 and do not build on it. The miss-side split of Section V is unaffected; the observation sits here because it rests on six cases. A-C Verification coverage Six headline statistics were recomputed against an independent implementation. The CochranâMantelâHaenszel common odds ratio was compared with a statistics library on all ten pairs and agreed exactly; Ï with Pearsonâs r on the 0/1 error vectors, agreeing to within 2.6Ă10â152.6Ă 10^-15; Cohenâs Îș with a separately written confusion-matrix formula, to within 1.8Ă10â151.8Ă 10^-15; the exact McNemar tests with a library implementation on all twelve comparisons, exactly; nmultn_mult with a hand calculation from its definition, exactly; and the difficulty share with a second implementation written after review, which agrees with the released script under the same probe construction (31.8% against 32%). Holm correction and the new-coverage counts were also recomputed by a second method with exact agreement. The rule column was regenerated under the archived, description-visible condition and matched the archived column on all 1,119 cases.