Paper deep dive
A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation
Dipankar Sarkar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/17/2026, 5:30:04 AM
Summary
This paper introduces Principle-Bench, a benchmark for evaluating Large Language Models (LLMs) as judges in principle-based financial regulation, specifically focusing on UK FCA principles. It proposes a four-axis evaluation framework: accuracy, paraphrase robustness, adversarial robustness, and calibration. The authors introduce Ceca, a calibrated, auditable assessor with counterfactual attributions. Results show that while LLMs perform well on benign inputs, they suffer significant accuracy drops under adversarial keyword-stuffing ('compliance theatre'), highlighting the need for robust, calibrated, and auditable evaluation methods in regulatory contexts.
Entities (12)
Relation Signals (7)
Ceca → provides → counterfactual attributions
confidence 96% · Ceca... emits exact per-exemplar counterfactual attributions.
Principle-Bench → covers → UK FCA
confidence 95% · 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles
Principle-Bench → evaluates → LLM-as-judge
confidence 95% · Principle-Bench... the first benchmark covering all four axes for principle-based regulation.
Principle-Bench → includessplits → adversarial
confidence 94% · 168 scenarios... with paraphrase, adversarial keyword-stuffing, and boundary perturbations
gpt-oss:120b → suffersfrom → compliance theatre
confidence 93% · A 120B LLM-judge... loses 47 accuracy points... on keyword-stuffed Consumer Duty inputs: 'compliance theatre.'
gpt-oss:120b → comparedwith → Kimi-K2.6
confidence 92% · A second judge from a different model family agrees only at Cohen's kappa = 0.16... localising the failure to the model
Ceca → uses → Platt scaling
confidence 90% · we fit a logistic s = σ(â s+b)(Platt scaling) on a 20-scenario dev split
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Principle-based regulation, with evaluative standards such as "fair, clear, and not misleading" or "deliver good outcomes", cannot be reduced to binary predicates, and LLM-as-judge is increasingly used as the substitute. Our position is that any such judge must be evaluated on four axes: accuracy, paraphrase robustness, adversarial robustness, and calibration. We release Principle-Bench, 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations authored under a pre-registered rubric; the first benchmark covering all four axes for principle-based regulation. We also introduce Ceca (Calibrated Exemplar-Cluster Assessment): a calibrated, auditable assessor that emits exact per-exemplar counterfactual attributions. Across keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, and a calibrated cascade, no method dominates all four axes. A 120B LLM-judge, strongest on benign inputs, loses 47 accuracy points (0.74 to 0.27) on keyword-stuffed Consumer Duty inputs: "compliance theatre." A second judge from a different model family agrees only at Cohen's kappa = 0.16 on that split, localising the failure to the model rather than the corpus. Any deployment-grade LLM-judge for principle-based regulation must report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2608.14329v1
- Canonical: https://arxiv.org/abs/2608.14329v1
Trouble viewing inline? Open PDF directly →
Full Text
35,546 characters extracted from source content.
Expand or collapse full text
A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation Dipankar Sarkar dipankar@skelfresearch.com Independent Researcher Abstract Principle-based regulation, with evaluative standards such as “fair, clear, and not misleading” or “deliver good outcomes”, cannot be reduced to binary predicates, and LLM-as-judge is increasingly used as the substitute. Our position is that any such judge must be evaluated on four axes: accuracy, paraphrase robustness, ad- versarial robustness, and calibration. We release Principle-Bench, 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations authored under a pre-registered rubric; the first benchmark covering all four axes for principle-based regu- lation. We also introduce Ceca (Calibrated Exemplar-Cluster As- sessment): a calibrated, auditable assessor that emits exact per- exemplar counterfactual attributions. Across keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, and a calibrated cascade, no method dominates all four axes. A 120B LLM-judge, strongest on benign inputs, loses 47 accuracy points (0.74→ 0.27) on keyword-stuffed Consumer Duty inputs: “compli- ance theatre.” A second judge from a different model family agrees only at Cohen’s휅=0.16 on that split, localising the failure to the model rather than the corpus. Any deployment-grade LLM-judge for principle-based regulation must report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy. Keywords LLM-as-judge, trustworthy evaluation, calibration, adversarial ro- bustness, AI governance, principle-based regulation, auditability 1 Introduction Automated compliance checking is now a staple of AI governance pipelines in finance, healthcare, and autonomous systems. The dominant paradigm treats every regulation as a Horn clause: a binary predicate that is either satisfied or violated. This rule-based model underpins Datalog verifiers, semantic-graph checkers, and most production reg-tech stacks. However, not every regulatory measure is a rule. The UK Fi- nancial Conduct Authority’s Consumer Duty and COBS 4.5A.3R stipulate that promotions be “fair, clear, and not misleading” and that firms “deliver good outcomes for retail customers” [11]. These are principles: outcome-based, evaluative standards that cannot be verified by checking a single boolean. A promotion can contain every required disclaimer and still be misleading overall. When rule-based systems meet principles, they either silently treat them as rules (false confidence) or ignore them (vacuous compliance). Both are dangerous in high-stakes domains. The EU AI Act [10] classifies credit-scoring and insurance-pricing as high-risk, yet its conformity framework relies on prescriptive checks that do not capture “fairness” or “adequate human oversight.” An obvious alternative is to ask a large language model to judge the principle directly. LLM-as-judge [23] reaches near-human agree- ment on benchmark tasks, but remains opaque, is poorly calibrated by default, and offers no per-example justification a regulator could interrogate. The question for secure and trustworthy deployment is not whether an LLM can evaluate a principle, but whether its evaluations are auditable, reproducible, calibrated, and robust to gaming. This is the four-axis question the rest of the paper makes operational. 2 Contributions (1) Principle-Bench, the first benchmark with a perturbation suite built specifically for principle-based regulatory assessment: 168 sce- narios across two UK FCA principles (100 original + 30 paraphrase + 28 adversarial + 10 boundary), anchored on author-prepared sum- maries of public FCA material and labelled under a pre-registered rubric whose git tag pre-dates all labelling. (2) Ceca, a calibrated exemplar-cluster assessor with exact per-exemplar counterfactual attribution: an auditable, transparency-by-design alternative to an opaque judge (Appendix B). (3) A four-axis comparison of keyword counting, three sentence-transformer embedders, an open-weight LLM-judge (with a second judge from a different model family), and a calibrated cascade. No method dominates. Headline finding: a 120B LLM-judge drops 47 accuracy points (Consumer Duty: 0.74→0.27) on keyword-stuffed inputs, the “compliance theatre” legal schol- arship warns of; a different-family judge confirms the failure is model-side (휅= 0.16). 3 Principle-Bench and Ceca Problem. Let푥be input text and푃= (퐶,휏)a principle with sub-concept clusters퐶= 푐 1 , . . .,푐 푘 and escalation threshold휏. Each cluster푐 푖 =(퐸 + 푖 ,퐸 − 푖 ,푤 푖 )carries positive and negative exem- plars and a weight. An assessment is훼= (푠,푎 푖 ,푒): a calibrated compliance probability푠 ∈ [0,1], per-cluster confidences푎 푖 , and an escalation flag푒=1[푠< 휏]. It is auditable if every component traces to a specific exemplar contribution a regulator could remove or contest. We evaluate assessors on four axes: accuracy, para- phrase robustness (accuracy preserved under exemplar-token- free rewrites), adversarial robustness (accuracy preserved under surface keyword insertion designed to invert the verdict), and cali- bration (low Expected Calibration Error, ECE). Principle-Bench. 168 scenarios across COBS 4.5A.3R (“fair, clear and not misleading”) and PRIN 2A/FG22–5 Consumer Duty (“deliver good outcomes”), in four splits: original (100), paraphrase (30,≤5 shared content tokens with any exemplar, script-verified), adver- sarial (28 keyword-stuffed; 2 of 30 refused by the generator’s safety filter), and boundary (10 near-threshold). Scenarios are generated arXiv:2608.14329v1 [cs.CR] 14 Aug 2026 SeT-LLM @ KDD 2026, August 9–10, 2026, Jeju, South KoreaDipankar Sarkar by an open-weight model (gpt-oss:120b) from one of 22 author- prepared summaries of public FCA material, with a target verdict (the ground truth) and a ban on verbatim exemplar phrases. The per- cluster rubrics were committed and tagged (v0-rubric-prereg) be- fore any scenario was scored, giving a tamper-evident pre-registration record. Construction, limitations (author-only labelling and a cor- pus/judge model-family overlap, partly mitigated in §5), and the reading list are detailed in Appendix C. Ceca. On a pluggable embedder휙, the raw cluster confidence is a sigmoid of the positive-minus-negative mean cosine similarity to exemplars, and the principle-level raw score is their weighted mean (Appendix B, Eq. 1–2); averaging makes the score Lipschitz in any single exemplar, bounding non-adversarial sensitivity. Raw scores occupy a narrow sub-interval of[0,1], so we fit a logistic 푠= 휎(푎 ˆ 푠+푏)(Platt scaling) on a 20-scenario dev split; the calibrated 푠is what a regulator sees. Because the assessor is a closed-form function of the exemplar set, Ceca emits an exact per-exemplar counterfactualΔ 푒 = 푠−푠 \푒 : which exemplar drove the verdict, and what removing it would do. A cascade defers to the LLM-judge only when the primary’s confidence lands in a dev-tuned uncertainty band, bounding judge invocations while keeping attribution for the high-confidence majority. 4 Experimental Setup We compare six methods: Keyword (deterministic bag-of-clusters, the state of practice); three sentence-transformer embedders, MiniLM (all-MiniLM-L6-v2), BGE (bge-base-en-v1.5), and Ollama-mxbai (mxbai-embed-large); LLM-judge (gpt-oss:120b, temperature 0, seed 42, prompted with the principle text, full exemplar set, and input); and a cascade (keyword primary, LLM-judge fallback in band[0.4,0.6]). Calibrators and the cascade band are tuned on a 20-scenario dev subset of original; all metrics use the remaining 80 plus the perturbation splits. For 7–16% of original scenarios the judge returns malformed JSON; those cells are computed on re- tained scenarios (the푛column). Point estimates carry bootstrap 95% CIs (10 4 resamples); paired comparisons use McNemar’s exact test. Full method, metric, and prompt details are in Appendix D. 5 Results No method dominates (Table 1). On original COBS, LLM-judge and cascade are near-indistinguishable (accuracy 0.96 vs. 0.94; AUC 1.00 vs. 0.99) and dominate the embedders, which sit at chance because their raw scores compress around 0.5. On the harder Con- sumer Duty principle the gap widens (LLM-judge 0.74, cascade 0.67). After Platt scaling, the LLM-judge’s deployment-relevant test ECE is 0.04–0.10, down from a raw 0.22–0.27 (Fig. 1); calibration is not optional. Paraphrase robustness favours the LLM-judge and cascade; the embedders, already at chance, have little left to lose (Appendix E, Fig. 3). The judge gets gamed (compliance theatre). The adversarial split inverts the ranking (Fig. 2). The LLM-judge (strongest everywhere else) collapses on Consumer Duty from 0.74 to 0.27, a 47-point drop, while holding 0.92 on COBS. The Consumer Duty perturba- tion inserts short factual assertions (e.g. “Client can absorb a total loss”) verbatim into substantively non-compliant promotions; the 0.00.20.40.60.81.0 Mean predicted escalation probability 0.0 0.2 0.4 0.6 0.8 1.0 Empirical escalation rate CONSUMER_DUTY_FAIR_VALUE Keyword MiniLM BGE-base Ollama (mxbai) LLM-judge CECA Cascade 0.00.20.40.60.81.0 Mean predicted escalation probability 0.0 0.2 0.4 0.6 0.8 1.0 Empirical escalation rate FAIR_CLEAR_NOT_MISLEADING Keyword MiniLM BGE-base Ollama (mxbai) LLM-judge CECA Cascade Figure 1: Reliability diagrams on the original split. Sentence- transformer methods trace near-vertical curves (raw scores compress around 0.5); the LLM-judge over-promises pre- calibration. Post-Platt scaling on a 20-sample dev split, the LLM-judge’s test reliability aligns with the diagonal. Principle MethodAcc. [95% CI] F1 AUC ECE Cons. Duty Keyword0.54 [.40,.68]0.680.810.196 MiniLM0.50 [.36,.64]0.670.610.128 BGE-base0.50 [.36,.64]0.670.540.109 Ollama-mxbai0.50 [.36,.64]0.670.170.106 LLM-judge0.74 [.62,.87]0.800.980.096 Cascade0.67 [.52,.79]0.750.960.113 COBS 4.5A Keyword0.52 [.38,.66]0.670.700.108 MiniLM0.50 [.36,.64]0.670.420.329 BGE-base0.50 [.36,.64]0.670.180.132 Ollama-mxbai0.50 [.36,.64]0.670.200.249 LLM-judge0.96 [.90,1.0]0.961.000.040 Cascade0.94 [.85,1.0]0.940.990.084 Table 1: Primary metrics on the original split (bootstrap 95% CIs, 10 4 resamples). ECE is held-out test ECE after Platt scal- ing fit on a 20-scenario dev split. Embedders sit at chance; the LLM tier dominates on benign inputs; a picture the ad- versarial split overturns. judge over-credits these surface statements and passes the promo- tion. COBS positive exemplars are higher-order qualitative claims (“presents a fair picture”) that resist such injection. This is the com- pliance theatre legal scholarship flags for principle-based regulation: A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based RegulationSeT-LLM @ KDD 2026, August 9–10, 2026, Jeju, South Korea Keyword MiniLM BGE-base Ollama (mxbai) LLM-judge CECA Cascade 0.0 0.2 0.4 0.6 0.8 1.0 Adversarial deception rate Lower is better robustness to keyword-stuffing CONSUMER_DUTY_FAIR_VALUE FAIR_CLEAR_NOT_MISLEADING Figure 2: Adversarial deception rate (verdicts flipped against ground truth on keyword-stuffed inputs; lower is better). Consumer Duty inverts the benign ranking of Table 1: the LLM-judge (the strongest method on every other split) be- comes the weakest (47-point accuracy drop), while uncali- brated sentence-transformer embedders, blind to the injected phrases, become the most robust. SplitCOBS 4.5A.3R Consumer Duty original0.8530.493 paraphrase0.8570.759 boundary0.6001.000 adversarial0.5720.157 Table 2: Inter-judge agreement (Cohen’s휅) between gpt-oss:120band a different-family judgekimi-k2.6. The collapse to휅=0.16 on Consumer Duty adversarial localises the keyword-stuffing vulnerability to the model, not the cor- pus. Boundary휅rests on푛=8 and is a summary, not a significance test. game the assessor by stuffing the right phrases while leaving prac- tice unchanged. The uncalibrated sentence-transformer embedders are the most adversarially robust on Consumer Duty (0.67) precisely because they cannot read the inserted phrases as evidence; keyword fails 100% on COBS adversarial by construction. The failure is model-side, not a corpus artefact (Table 2). Because the corpus generator and one judge share a model family, we ran a second judge from a different family (kimi-k2.6) on all splits. On COBS, inter-judge agreement stays high on original (휅=0.85) and paraphrase (0.86); on Consumer Duty adversarial it collapses to휅=0.16, at chance. Two judges from different families produce materially different verdicts on the same gamed inputs, so the susceptibility to keyword-stuffing is model-side rather than an artefact of who authored the corpus. Auditability and the cascade. Ceca’s counterfactual attribution is faithful: removing the top-attributed exemplar shifts the score by 10×a random exemplar’s effect and flips the verdict on 4% [CI 1–8%] of keyword-backend inputs; enough to let a regulator ask “which exemplar drove this, and what if it were removed?” The cascade (dev-tuned band, 89% deferral) approaches LLM-judge accuracy on benign and paraphrased inputs (gaps 2–14 points) but inherits its primary’s vulnerability: on COBS adversarial it scores 0% because the keyword primary confidently mis-scores stuffed inputs outside the band and never consults the judge. This is a structural hazard for any cascade whose primary is the direct target of the perturbation. Full per-cluster AUC, cascade cost, calibration diagrams, and a worked attribution example are in Appendix E. 6 Discussion Why “just use the LLM” is unsafe. A naive reading of Table 1 says “use the judge.” Three counter-observations: (i) its raw con- fidences are polarised, inflating raw ECE to 0.22–0.27, so without Platt scaling a regulator cannot read the verdict’s confidence as a probability; (i) it offers narrative rationale, not the ranked, exactly- reproducible per-exemplar contributions auditability requires, so by our own criterion the judge is a strong baseline, not a deployable assessor; (i) it is gameable on at least one principle, with the failure confirmed model-side. Recommendation. Any deployment-grade LLM-judge for principle- based assessment should report a per-principle adversarial decep- tion rate and post-hoc calibration alongside aggregate accuracy, and pair the verdict with an auditable, contestable attribution layer. The method and benchmark structure (pre-registered rubric, seed grounding, perturbation suite) transfer to any principle-based do- main: healthcare’s “informed consent,” autonomous vehicles’ “rea- sonable safety,” or the EU AI Act’s evaluative standards, condi- tional on a regulator-led exemplar-curation process. We release prompts, raw responses, calibrators, and the pre-registration tags at https://github.com/sarkar-dipankar/principle-bench (Appendix G). 7 Conclusion Principle-based regulation needs an assessor that is more than “cor- rect on average”: calibrated, paraphrase- and adversary-robust, and counterfactually auditable. Across all four axes no single method wins, and the strongest judge is the one most easily gamed, a trustworthiness failure invisible to headline accuracy. Secure-and- trustworthy LLM evaluation for governance must be grounded in this multi-axis evidence. References [1]Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Ben- netot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, et al.2020. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. Information Fusion 58 (2020), 82–115. doi:10.1016/j.inffus.2019.12.012 [2]Association for Computing Machinery. 2022. Statement on Principles for Re- sponsible Algorithmic Systems. ACM US Public Policy Council and ACM Tech- nology Policy Council. https://w.acm.org/binaries/content/assets/public- policy/final-joint-ai-statement-update.pdf [3]Tara Athan, Harold Boley, Guido Governatori, Monica Palmirani, Adrian Paschke, and Adam Wyner. 2013. OASIS LegalRuleML. In Proceedings of the 14th Inter- national Conference on Artificial Intelligence and Law (ICAIL ’13). ACM, 3–12. doi:10.1145/2514601.2514603 [4] Julia Black. 2008. Forms and Paradoxes of Principles Based Regulation. Capital Markets Law Journal 3, 4 (2008), 425–457. doi:10.2139/ssrn.1267722 [5]Julia Black. 2010. The Rise, Fall and Fate of Principles Based Regulation. LSE Law, Society and Economy Working Papers 17/2010. London School of Economics. https://eprints.lse.ac.uk/38413/ [6]John Braithwaite. 2002. Rules and Principles: A Theory of Legal Certainty. Australian Journal of Legal Philosophy 27 (2002), 47–82. [7]Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The Muppets Straight Out of SeT-LLM @ KDD 2026, August 9–10, 2026, Jeju, South KoreaDipankar Sarkar Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020. ACL, 2898–2904. doi:10.18653/v1/2020.findings-emnlp.261 [8]Chaofan Chen, Oscar Li, Daniel Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. 2019. This Looks Like That: Deep Learning for Interpretable Image Recognition. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, 8930–8941. [9] Ronald M. Dworkin. 1967. The Model of Rules. University of Chicago Law Review 35, 1 (1967), 14–46. [10] European Parliament and Council of the European Union. 2024. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L series, 12 July 2024. https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:32024R1689 [11] Financial Conduct Authority. 2022. Finalised non-Handbook Guidance: The FCA’s Approach to the Consumer Duty. FG22/5, July 2022. https://w.fca.org. uk/publication/finalised-guidance/fg22-5.pdf [12]Guido Governatori and Antonino Rotolo. 2008. An Algorithm for Business Process Compliance. In Legal Knowledge and Information Systems (JURIX 2008). IOS Press, 186–191. doi:10.3233/978-1-58603-900-4-186 [13] Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rock- more, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates. [14] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML ’17). PMLR, 1321–1330. [15]Mustafa Hashmi. 2015. A Methodology for Extracting Legal Norms from Regu- latory Documents. In Proceedings of the 19th IEEE International Enterprise Dis- tributed Object Computing Workshop (EDOCW ’15). IEEE, 41–50. doi:10.1109/ EDOCW.2015.29 [16] Chu Fei Luo, Rohan Bhambhoria, Samuel Dahan, and Xiaodan Zhu. 2023. Prototype-Based Interpretability for Legal Citation Prediction. In Findings of the Association for Computational Linguistics: ACL 2023. ACL, 4883–4898. doi:10.18653/v1/2023.findings-acl.301 [17] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. Comput. Surveys 54, 6 (2021), 1–35. doi:10.1145/3457607 [18] Shira Mitchell, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021. Algorithmic Fairness: Choices, Assumptions, and Definitions. Annual Review of Statistics and Its Application 8 (2021), 141–163. doi:10.1146/annurev- statistics-042720-125902 [19]National Institute of Standards and Technology. 2022. Towards a Standard for Identifying and Managing Bias in Artificial Intelligence. NIST Special Publication 1270. https://doi.org/10.6028/NIST.SP.1270 [20] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP ’19). ACL, 3982–3992. doi:10. 18653/v1/D19-1410 [21] M. J. Sergot, Fariba Sadri, Robert A. Kowalski, Frank Kriwaczek, Peter Hammond, and H. T. Cory. 1986. The British Nationality Act as a Logic Program. Commun. ACM 29, 5 (1986), 370–386. doi:10.1145/5689.5920 [22]Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical Networks for Few-Shot Learning. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, 4077–4087. [23]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, 46595–46623. A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based RegulationSeT-LLM @ KDD 2026, August 9–10, 2026, Jeju, South Korea A Related Work Principles versus rules. Dworkin [9] argues rules operate all-or- nothing whereas principles carry comparative weight; Black [4,5] traces the rise and post-crisis retreat of principles-based financial regulation, and Braithwaite [6] shows principles are needed when rules over- or under-reach. The implication for automated compli- ance is direct: a system limited to encoding rules cannot evaluate principles, however advanced its rule engine. Automated compliance and legal NLP.. Sergot et al. [21] repre- sented the British Nationality Act in Horn clauses; Governatori and Rotolo [12] added defeasible reasoning; Athan et al. [3] for- malised LegalRuleML; Hashmi [15] keys on modal verbs (“must,” “shall”) that principle-based regimes do not use. On the learning side, Reimers and Gurevych [20] introduced Sentence-BERT, Chalkidis et al. [7] specialised BERT for legal text, and LegalBench [13] cu- rates legal-reasoning tasks for LLMs; we depart from this work by targeting evaluative regulation and a purpose-built perturbation suite. Prototypes, calibration, and LLM-as-judge. Ceca’s exemplar clus- ters extend prototypical networks [16,22] and prototype-based explanation [8] to regulatory text. Guo et al. [14] show modern classifiers are overconfident and post-hoc scaling helps; Zheng et al. [23] find LLM judges near human agreement on benchmark tasks but biased. We further find them poorly calibrated and ad- versarially gameable in regulatory settings. Work on auditability and accountability (Mehrabi et al. [17], Arrieta et al. [1], Mitchell et al. [18], and the NIST [19] and ACM [2] frameworks) motivates our transparency-by-design per-exemplar attribution over post-hoc rationalisation. B Ceca Method Details Ceca is built in four layers on a pluggable embedder 휙 : String → R 푑 . Base exemplar-cluster scoring. Withcos(푢,푣)= 푢· 푣/(∥푢∥푣∥) and 휎(푧)= 1/(1+푒 −푧 ), the raw cluster confidence is 휌(푐,푥)= 휎 1 |퐸 + | Í 푒∈퐸 + cos(휙(푥),휙(푒)) − 1 |퐸 − | Í 푒∈퐸 − cos(휙(푥),휙(푒)) (1) and the principle-level raw score is the weighted mean ˆ 푠(푃,푥)= Í 푐∈퐶 푤 푐 휌(푐,푥) Í 푐∈퐶 푤 푐 .(2) Averaging (rather than maximising) makes휌robust to a single spurious exemplar match. Assuming unit-normalised embeddings (∥휙(·)∥=1, the standard sentence-transformer convention),휌is Lipschitz in any single exemplar: replacing휙(푒 푗 ) ∈ 퐸 + by휙(푒 푗 )+훿 with∥훿∥ ≤ 휖shifts the positive-cosine average by at most휖/|퐸 + |, so|Δ휌| ≤ 휎 ′ (·)·휖/|퐸 + | ≤ 휖/(4|퐸 + |)(using휎 ′ ≤1/4); the symmetric bound휖/(4|퐸 − |)holds for negative exemplars, and ˆ 푠inherits the bound through a convex combination. The empirical adversarial- deception rate is the worst-case figure when the perturbation is a deliberate text-level keyword-stuffing edit rather than an infinitesi- mal embedding shift. Platt calibration. Raw scores are sigmoids of cosine differences and occupy a narrow sub-interval of[0,1]. We fit a logistic푠= 휎(푎 ˆ 푠 +푏) on a held-out development split (Newton–Raphson on cross-entropy; coefficients persisted to disk). The calibrated푠is what a regulator sees;휏is then an empirical-probability cut-off, not an arbitrary sigmoid output. We also report isotonic regression as a robustness check. Counterfactual attribution. For an assessment(푠,푎 푖 ), Ceca emits, for each exemplar푒 ∈ 퐸 + 푖 ∪퐸 − 푖 , the counterfactual푠 \푒 obtained if푒were removed, and the signed deltaΔ 푒 = 푠−푠 \푒 . Because the assessor is a closed-form function of the exemplar set, the predicted 푠 \푒 exactly matches re-running the assessor with푒removed. Faith- fulness is stronger than this identity: does the highest-|Δ|exemplar carry the verdict? Empirically (§E), removing the top-attributed exemplar flips the verdict on 4% [95% CI 1–8%] of inputs and shifts the score by 10× the mean shift of a random exemplar. Cascade with LLM-as-judge. Any embedder producing a per- cluster score in[0,1]can be the primary. When the primary’s confidence푠falls in a dev-tuned band[ℓ,ℎ], the cascade defers to an LLM-judge with access to the same principle and exemplars; otherwise it emits the primary’s verdict. It provides per-exemplar attribution for the high-confidence majority and LLM rationale for the low-confidence minority, while bounding judge invocations. A primary whose scores cluster at the band mid-point defers ev- erything (vacuous cascade); one whose scores spread across[0,1] supports meaningful deferral. C Principle-Bench Construction Generation. Each scenario is anchored on one of 22 author- prepared summaries informed by public FCA material (Finalised Guidance, Dear-CEO letters on cryptoasset financial promotions, FOS Ombudsman decisions, and FCA-published exemplars; see the reading list in the supplementary release).gpt-oss:120bis prompted with the seed, a target verdict, and (channel, product, audience) constraints, and is forbidden from using any exemplar phrase verbatim. The target verdict is the ground truth. Paraphrases are LLM rewrites with≤5 shared content tokens with any exem- plar (stop-words removed; script-verified). For the adversarial split we attempted 30 keyword-stuffed variants per polarity-flip target; 28 were retained after the generator’s safety alignment refused 2 prompts framed as “stuff misleading promotions with compliance language”; itself a small finding about open-weight safety filters in compliance-testing contexts. Adversarial variants insert≥3 opposite-polarity exemplar phrases while preserving the substan- tive verdict. Pre-registered rubric. Before any scenario was authored or la- belled, we committed and tagged two assessment rubrics (one per principle) operationalising the regulation handbooks into per- cluster 1–5 Likert criteria and an escalation rule. The rubric commit and tagv0-rubric-preregpre-date every label commit in the repository, a tamper-evident record that the criteria were fixed before scoring. Limitations. Labels are author-assigned via the rubric (no ex- ternal annotator). We mitigate by (a) pre-registering and tagging SeT-LLM @ KDD 2026, August 9–10, 2026, Jeju, South KoreaDipankar Sarkar the rubric before scoring, and (b) anchoring each scenario on a public FCA-material summary so the verdict is constrained by ex- ternal regulatory practice. We disclose that no external annotator validation was performed; a regulator panel for inter-annotator agreement is left to successor work. Corpus/judge family overlap: the corpus is authored bygpt-oss:120band one judge configura- tion is the same model; we partly mitigate with the different-family second judge (kimi-k2.6) and the inter-judge휅analysis (Table 2). Regenerating the corpus with a non-overlapping family is left to a successor benchmark. D Full Experimental Setup Methods. Keyword: bag-of-clusters embedder (one dimension per cluster; entries are L2-normalised pos-minus-neg counts). MiniLM: all-MiniLM-L6-v2(384-dim). BGE:bge-base-en-v1.5(768-dim). Ollama-mxbai:mxbai-embed-large(1024-dim) via local Ollama. LLM-judge:gpt-oss:120bvia Ollama Cloud (temperature 0, seed 42). Cascade: keyword-primary with LLM-judge fallback when푠 ∈ [0.4,0.6]. We use the keyword primary because MiniLM raw scores spanned only[0.476,0.522]on original, making a 20-sample Platt fit degenerate (slope푎 ≈ −773); the keyword embedder’s wider range [0.39, 0.66] supports stable band tuning. Procedure and metrics. Calibrators and the cascade band are tuned on a 20-scenario dev subset of original; metrics use the re- maining 80 plus all perturbation splits. For 7–16% of original sce- narios the judge returns malformed JSON; these are excluded from judge cells (the푛column). Metrics: accuracy, precision, recall, F1, per-cluster AUC, false-compliance rate, ECE [14], Brier score. All point estimates carry bootstrap 95% CIs (10 4 resamples); paired comparisons use McNemar’s exact test. E Full Results Per-cluster AUC (Table 3). Keyword and LLM-judge carry the most per-cluster signal; sentence-transformer embedders are near- random, and several Ollama-mxbai clusters score below 0.5 (0.03– 0.09 on three Consumer Duty clusters), i.e. an inverted similarity ordering, evidence that off-the-shelf embeddings used directly can mislead per-cluster attribution. Counterfactual faithfulness. Tested on all 100 original scenarios with keyword and MiniLM backends. Removing the top-|Δ|exem- plar moves the score by meanΔ푠=0.018 (keyword) and 0.0045 (MiniLM), respectively 10×and 2.3×the random-exemplar control, and flips the verdict on 4% [CI 1–8%] (keyword) and 0% (MiniLM, re- flecting its compressed range). Attribution ranks influence correctly on both; only the keyword primary produces score movement large enough to flip a binary verdict on this corpus. Cascade cost (Table 4). The dev-tuned band[0.40,0.60]defers 89% of inputs. ECE is bounded between components and exceeds LLM-judge ECE here; the calibration cost of mixing poorly cal- ibrated keyword with well-calibrated judge verdicts. Wall-clock latency is not systematically lower because both shared a rate- limited endpoint; under independent scheduling expected latency isdef· 푡 judge +푡 primary ≈0.89·19.7+0.01≈17.6s (∼11% below the judge). The deployable saving is in the count of judge invocations plus the attribution layer on the high-confidence majority. Principle ClusterKw MiniLM BGE mxbai LLM CD Appropriateness0.600.590.550.090.95 Fee fairness0.780.410.180.030.93 Loss capacity0.580.680.640.600.92 Ongoing duty0.680.450.530.370.92 COBS Fee clarity0.720.580.200.060.96 Lang. access.0.360.490.310.250.67 Present. fair.0.770.320.260.350.98 Risk balance0.540.330.490.340.99 Table 3: Per-cluster AUC on original: each cluster’s confi- dence푎 푖 as a standalone classifier of expected-escalate vs. expected-comply. MethodAccuracy ECE Deferral Latency (s) Keyword0.530.152n/a0.007 LLM-judge0.850.068n/a19.743 Cascade0.800.09989%25.308 Table 4: Cascade vs. components on original (averaged over both principles). Latency is same-environment wall- clock under a shared rate-limited endpoint; see text for the independent-scheduling bound. F Ethical Considerations Author-labelled benchmark. Labels are author-assigned per a pre-registered rubric; reproducibility is preserved via the rubric and corpus-freeze tags, but inter-annotator agreement is not re- ported (no external annotator). The path to a regulator-validated successor is mapped above. Dual-use. A principle-assessment tool can triage compliance for regulators or be used to optimise pro- motions against the metric while leaving practice unchanged; our adversarial experiment quantifies this directly. Any method with a non-trivial deception rate cannot safely be the sole primary asses- sor. We recommend periodic human review and public reporting of method-specific deception rates. Synthetic provenance. Scenarios are synthetic but anchored on author-prepared summaries of public FCA material; no verbatim enforcement-document text is used. Bias. Curated exemplars embed the curator’s normative judgements; the pre-registered rubric, open exemplar set, and counterfactual inter- face make these inspectable and contestable, relocating the curation question from a hidden classifier to a public artefact rather than eliminating it. G Reproducibility We release the method code, prompts, raw LLM responses, fit- ted calibrators, the full benchmark, and the pre-registration tags (v0-rubric-prereg,v1-corpus-frozen) at https://github.com/sarkar- dipankar/principle-bench, whose commit ordering establishes that the rubric pre-dates all labelling. The full sweep (6 methods×4 splits×≤2 principles) caches one embedding per (model, exemplar) pair; bootstrap CIs use 10 4 resamples. A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based RegulationSeT-LLM @ KDD 2026, August 9–10, 2026, Jeju, South Korea Keyword MiniLM BGE-base Ollama (mxbai) LLM-judge CECA Cascade 0.10 0.05 0.00 0.05 0.10 Accuracy(original) Accuracy(paraphrase) Paraphrase robustness gap (smaller is better) CONSUMER_DUTY_FAIR_VALUE FAIR_CLEAR_NOT_MISLEADING 0.00.20.40.60.81.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate CONSUMER_DUTY_FAIR_VALUE Keyword (AUC=0.81) MiniLM (AUC=0.61) BGE-base (AUC=0.54) Ollama (mxbai) (AUC=0.17) LLM-judge (AUC=0.98) CECA Cascade (AUC=0.96) 0.00.20.40.60.81.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate FAIR_CLEAR_NOT_MISLEADING Keyword (AUC=0.70) MiniLM (AUC=0.42) BGE-base (AUC=0.18) Ollama (mxbai) (AUC=0.20) LLM-judge (AUC=1.00) CECA Cascade (AUC=0.99) 0.30.40.50.60.70.8 Escalation threshold 0.0 0.2 0.4 0.6 0.8 1.0 Escalation rate on boundary set CONSUMER_DUTY_FAIR_VALUE Keyword MiniLM BGE-base Ollama (mxbai) LLM-judge CECA Cascade 0.30.40.50.60.70.8 Escalation threshold 0.0 0.2 0.4 0.6 0.8 1.0 Escalation rate on boundary set FAIR_CLEAR_NOT_MISLEADING Keyword MiniLM BGE-base Ollama (mxbai) LLM-judge CECA Cascade 0.0030.0020.0010.0000.0010.0020.003 s = s original s e (positive: exemplar pushed score up) [neg] risk_balance: Only upside potential is highlighted [neg] risk_balance: Risks are minimized or omitted [neg] language_accessibility: Dense technical language is used [neg] language_accessibility: Complex terms are not explained [pos] risk_balance: A balanced view of potential downsides is given [pos] risk_balance: Risks and benefits are presented equally Counterfactual attribution for P_FAIR_CLEAR_NOT_MISLEADING_001_escalate (PRINCIPLE_FAIR_CLEAR_NOT_MISLEADING) overall score s = 0.507 Figure 3: Supporting diagnostics (adversarial deception is Fig. 2 and reliability diagrams Fig. 1 in the main text). From left to right: paraphrase robustness gap (smaller is better); per-method ROC; escalation rate vs.휏over the 10 boundary scenarios (all methods cross 50% in휏 ∈ [0.45,0.55];푛=10, a qualitative check); and a worked Ceca attribution example on a Consumer-Duty-adversarial scenario (signedΔ 푒 for the 8 highest-magnitude exemplars, concentrating on “language accessibility”).