Paper deep dive
Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation
Tobias DeuĂer, Abhishek Pillai, Aurelio F. Bariviera, Dhananjay Bhardwaj, Lorenz Sparrenberg, David Berghaus, Christian Bauckhage, Rafet Sifa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/26/2026, 2:42:34 AM
Summary
This paper presents a domain-adapted Retrieval-Augmented Generation (RAG) pipeline for automated regulatory compliance question answering in financial services. The system employs a three-stage retriever built on LegalBERT (entailment tuning, contrastive tuning, and BM25 fusion) and compact generators (2B-12B parameters) adapted via Retrieval-Aware Fine-Tuning (RAFT) with LoRA. Evaluated on the ObliQA benchmark (Abu Dhabi Global Market rulebooks), the staged retriever significantly improves Recall@10 over baselines. While RAFT-LoRA improves answer quality scores (RePASs), the study highlights that compact models can achieve similar scores without grounding, and the adapted models fail to transfer to Australian case law, revealing limitations in current evaluation metrics for verifying factual grounding.
Entities (10)
Relation Signals (10)
LegalBERT â isbasefor â staged retriever
confidence 95% · Our retriever is built in three stages on top of LegalBERT
staged retriever â uses â entailment tuning
confidence 94% · Our retriever is built in three stages on top of LegalBERT: entailment tuning... contrastive tuning... and score-level fusion with BM25.
staged retriever â uses â BM25
confidence 94% · score-level fusion with BM25.
RAFT â isappliedto â compact generators
confidence 93% · generator is a compact model... adapted with retrieval-aware fine-tuning (RAFT) through LoRA.
ObliQA â issourcefor â Abu Dhabi Global Market rulebooks
confidence 92% · ObliQA, a question-answering benchmark built from the Abu Dhabi Global Market rulebooks
staged retriever â outperforms â BM25
confidence 91% · the staged retriever raises Recall@10 from 0.256 to 0.774 and outperforms BM25 (0.678)
staged retriever â outperforms â E5-large-v2
confidence 91% · outperforms BM25 (0.678) and E5-large-v2 (0.758)
RAFT-LoRA â improves â RePASs score
confidence 90% · RAFT-LoRA then improves the composite RePASs answer-quality score for every model we could adapt
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Financial institutions operate under dense, frequently amended rulebooks, and answering a compliance question correctly requires not only fluency but verifiable grounding in the authoritative text. Large language models are attractive for this task, yet the models that firms can realistically deploy on-premise are compact ones, and compact models hallucinate obligations. We study whether a carefully domain-adapted retrieval-augmented generation pipeline closes that gap. Our retriever is built in three stages on top of LegalBERT: entailment tuning that recasts question--passage matching as premise--hypothesis reconstruction, contrastive tuning with in-batch negatives, and score-level fusion with BM25. Our generator is a compact model (2B--12B parameters) served under 4-bit quantization, either prompted or adapted with retrieval-aware fine-tuning (RAFT) through LoRA. On ObliQA, a question-answering benchmark built from the Abu Dhabi Global Market rulebooks, the staged retriever raises Recall@10 from 0.256 to 0.774 and outperforms BM25 (0.678) and E5-large-v2 (0.758), the strongest general-purpose dense encoder we tested. RAFT-LoRA then improves the composite RePASs answer-quality score for every model we could adapt, with the largest gain on the weakest one. However, the adapted models do not transfer to Australian case-law questions, and a closed-book model that receives no passages at all scores within 0.011 RePASs of the full pipeline while producing answers that cite nothing and misstate obligations. The retrieval gain is therefore measured directly, the generation gain is a gain in RePASs rather than demonstrated grounding, and grounding itself requires an evaluation protocol that RePASs does not provide.
Tags
Links
- Source: https://arxiv.org/abs/2609.30009v1
- Canonical: https://arxiv.org/abs/2609.30009v1
Trouble viewing inline? Open PDF directly â
Full Text
69,233 characters extracted from source content.
Expand or collapse full text
Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation Tobias DeuĂer â thanks: tdeusser@uni-bonn.de, ORCID-ID: 0000-0003-4685-0847 Affiliation: University of Bonn, Bonn, Germany Affiliation: Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany Abhishek Pillai Affiliation: University of Bonn, Bonn, Germany Aurelio F. Bariviera Affiliation: Universitat Rovira i Virgili, Reus, Spain Dhananjay Bhardwaj Affiliation: Universitat Rovira i Virgili, Reus, Spain Lorenz Sparrenberg Affiliation: University of Bonn, Bonn, Germany Affiliation: Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany David Berghaus Affiliation: Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany Affiliation: Fraunhofer IAIS, Sankt Augustin, Germany Christian Bauckhage Affiliation: University of Bonn, Bonn, Germany Affiliation: Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany Affiliation: Fraunhofer IAIS, Sankt Augustin, Germany Rafet Sifa Affiliation: University of Bonn, Bonn, Germany Affiliation: Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany Affiliation: Fraunhofer IAIS, Sankt Augustin, Germany Abstract Financial institutions operate under dense, frequently amended rulebooks, and answering a compliance question correctly requires not only fluency but verifiable grounding in the authoritative text. Large language models are attractive for this task, yet the models that firms can realistically deploy on-premise are compact ones, and compact models hallucinate obligations. We study whether a carefully domain-adapted retrieval-augmented generation pipeline closes that gap. Our retriever is built in three stages on top of LegalBERT: entailment tuning that recasts questionâpassage matching as premiseâhypothesis reconstruction, contrastive tuning with in-batch negatives, and score-level fusion with BM25. Our generator is a compact model (2Bâ12B parameters) served under 4-bit quantization, either prompted or adapted with retrieval-aware fine-tuning (RAFT) through LoRA. On ObliQA, a question-answering benchmark built from the Abu Dhabi Global Market rulebooks, the staged retriever raises Recall@10 from 0.256 to 0.774 and outperforms BM25 (0.678) and E5-large-v2 (0.758), the strongest general-purpose dense encoder we tested. RAFT-LoRA then improves the composite RePASs answer-quality score for every model we could adapt, with the largest gain on the weakest one. However, the adapted models do not transfer to Australian case-law questions, and a closed-book model that receives no passages at all scores within 0.011 RePASs of the full pipeline while producing answers that cite nothing and misstate obligations. The retrieval gain is therefore measured directly, the generation gain is a gain in RePASs rather than demonstrated grounding, and grounding itself requires an evaluation protocol that RePASs does not provide. Keywords retrieval-augmented generation â · regulatory compliance â · legal NLP â · hybrid retrieval â · natural language processing 1 Introduction Regulated financial companies operate inside a body of rules that is large, cross-referential, and in constant flux [1]. A single question is typically answered by locating a handful of facts scattered across chapters and reconciling them. These financial companies currently do this through specialist staff and commercial research tools, which makes compliance slow and, when a provision is missed, expensive [2, 3, 4]. Large language models (LLMs) are an obvious candidate for automating parts of this workflow [5, 6, 7, 8]. Used out of the box, however, they are a poor fit. Without access to the source text, an LLM has to rely on what it memorized during training, and its recall of specific rulebook clauses is unreliable: it may paraphrase provisions that do not exist or omit binding conditions, and it sounds equally confident in both cases [9]. In compliance work this is particularly problematic, because such an answer cannot be traced to any provision and therefore cannot be checked. Retrieval-augmented generation (RAG) [10] addresses this problem by retrieving relevant passages from an authoritative corpus and supplying them as context, so that answers can be traced back to the source text. The quality of a RAG system, however, depends heavily on its retriever. Provisions in different chapters of a rulebook often use nearly identical wording while differing in scope, applicability, or exemptions. Lexical overlap is therefore a weak relevance signal, and general-purpose dense encoders trained on web data are not designed to tell which of two similar clauses applies to a given case [11]. A second constraint is a more practical one. Regulatory documents and the questions asked about them are often confidential, so many institutions cannot send queries to a hosted frontier model and have to run models on their own hardware [12]. In practice, this limits the choice to quantized models with up to roughly 12B parameters (depending on the budget of the firm of course), which have the least legal knowledge and the weakest grounding behavior. Improvements therefore have to come from the pipeline around the model rather than from model scale. In this paper, we investigate how far such a pipeline can be improved. We build a RAG system for regulatory question answering in which both halves, the retrieval and the generation, are domain-adapted, and we evaluate it on ObliQA [13], a benchmark derived from the rulebooks of the Abu Dhabi Global Market (ADGM) and on Open Australian LegalQA [14] as an out-of-domain control. Our contributions are: âą A three-stage retriever that starts from LegalBERT [15] and applies entailment tuning, contrastive tuning, and linear score fusion with BM25. Each stage helps, and the composition lifts Recall@10 on ObliQA from 0.256 to 0.774, past BM25 (0.678) and past E5-large-v2 (0.758) despite using a far smaller encoder. âą A leakage-free re-split of ObliQA. The published splits share source documents across train, validation, and test; we rebuild them at document level and quantify what this costs in dataset size. âą A controlled comparison of prompting against retrieval-aware fine-tuning for five compact generators. Prompting effects are model-dependent and often negative, whereas RAFT-LoRA improves the composite RePASs score for all three models we could adapt stably, most for the weakest one. âą Two negative results. Adapters trained on ADGM rulebook data do not transfer to Australian case law, and a closed-book model without access to any passage scores within 0.011 RePASs of the full retrieval pipeline, even though its answers are not grounded. We attribute the latter to a weakness of the metric and discuss what this means for evaluating regulatory QA systems. 2 Related Work 2.1 Regulatory and Legal Question Answering Regulatory NLP applies language technology to statutes, rulebooks, and compliance guidance, where texts are long, heavily structured, and full of normative operators (obligations, permissions, prohibitions) whose scope is set by conditions and exceptions [16]. Answering questions over them is closer to multi-passage aggregation than to span extraction, since the governing conditions for one obligation are routinely stated elsewhere. Progress in this area has been driven by datasets. CUAD [17] annotates contract clauses for review tasks; LexGLUE [18] assembles legal understanding benchmarks; FinQA [19] targets numerical reasoning over financial filings. Closest to our setting is ObliQA [13], which pairs questions with the ADGM provisions that answer them and introduces RePASs, an obligation-aware answer metric; that work also defines the retrieve-then-generate task formulation we adopt. [20] treat regulatory compliance as document-to-document retrieval over European legislation, and [21] study synthetic data for the same domain. [22] generate questionâanswer pairs from German statutes and adapt LLMs with parameter-efficient fine-tuning, reporting gains over unadapted baselines. That strategy is close in spirit to the fine-tuning half of our pipeline, though it is applied to a different jurisdiction and without a retrieval-aware objective. Earlier legal QA systems relied on rules and keyword matching [23, 24, 25], later on neural judgment and retrieval models [26, 27, 28]. A broader and more detailed survey of LLM use in finance is given in [29]. 2.2 Sparse, Dense, and Hybrid Retrieval BM25 [30] remains a strong baseline wherever terminology is fixed, which describes regulatory text well, but it cannot bridge paraphrase. Dense retrieval [31] embeds queries and passages into a shared space and handles paraphrase, and general-purpose encoders such as E5 [32] and BGE [33] transfer well across many domains. Specifically domain-pretrained encoders such as LegalBERT [15] know legal vocabulary but, as we confirm in Section 4.2, are not good retrievers out of the box: masked-language pretraining optimizes token prediction, not the geometry of a similarity space. Two lines of work address this limitation. Contrastive training with in-batch negatives and a multiple-negatives ranking loss [34, 35, 36] can reshape the embedding space directly. Entailment tuning [37] instead reformulates retrieval as an inference task: a passage is relevant if it entails a claim derived from the question, which is a better match for legal text where two passages can be topically identical yet only one legally supports the answer. Combining sparse and dense scores recovers the strengths of both [38, 39]. Our retriever composes all three ideas in sequence rather than choosing among them. 2.3 Retrieval-Augmented Generation RAG [10, 40] conditions generation on retrieved evidence, reducing hallucination and making answers attributable [41, 42]. Extensions target its failure modes: Self-RAG [43] adds self-critique, Robust-RAG [44] defends against corrupted contexts, and RQ-RAG [45] refines ambiguous queries. RAFT [46] takes a different route and trains the generator itself to read a retrieved context: each training instance mixes an oracle passage with distractors, and for a fraction of instances withholds the oracle entirely, so the model learns to use evidence when present and to abstain from inventing it when absent. We adopt RAFT because it specifically targets hallucinations caused by irrelevant retrieved passages, which a high-recall retriever over regulatory text will inevitably return. 2.4 Parameter-Efficient Fine-Tuning Full fine-tuning of a 7B model requires considerably more GPU memory than many compliance teams likely have available. Adapter modules [47] and LoRA [48], which learns a low-rank update to frozen attention projections, reduce the trainable parameter count by orders of magnitude, and QLoRA [49] adds 4-bit quantization of the frozen base so that a 7B model can be adapted on a single commodity GPU. Refinements continue in this direction [50, 51]. We use QLoRA throughout, which allows us to run all RAFT experiments on a single GPU and matches the on-premise setting described in the Introduction section. 2.5 Positioning of this work Each component of our pipeline is established: domain-pretrained legal encoders [15], entailment tuning [37], contrastive tuning with in-batch negatives [34], sparseâdense fusion [39], and retrieval-aware fine-tuning [46]. We claim none of them as new. What is new is their composition into a single pipeline for financial-regulatory QA under an on-premise compute budget, together with a controlled evaluation of that pipeline: a staged retriever ablation on a leakage-free re-split, a like-for-like comparison of prompting against RAFT-LoRA across five compact generators, a transfer test of the resulting adapters to a different legal genre, and a closed-book control that isolates how much of the measured answer quality is attributable to retrieval at all. LLeQA [52] fixes a retrieve-then-read setup and varies the reader. CBR-RAG [53] varies the retrieval representation for a fixed generator. CLERC [54] reports retrieval and generation separately but with off-the-shelf retrievers. LegalBench-RAG [55] isolates retrieval rigorously while stopping short of generation. We instead vary both halves over a single shared retrieval run, and add a closed-book control, so that the contribution of each stage, and the point at which the metric stops tracking grounding, are separately identifiable. 3 Methodology 3.1 Task and Pipeline Let P be a corpus of regulatory passages and q a natural-language compliance question. The system must return an answer a that is entailed by P and that covers the obligations P imposes on the situation in q. We factor this into a retriever R:qâŠPkâR:q P_k and a generator L:(q,Pk)âŠaL:(q,P_k) a, trained and evaluated separately so that the contribution of each is identifiable. Figure 1 shows the arrangement. Regulatory question qqBM25Dense encoderLegalBERT ++ ET ++ CT retrievercorpusshyb=αâs^dense+(1âα)âs^bm25s_hyb=α\, s_dense+(1-α)\, s_bm25, α=0.8α=0.8top-k (k=10k=10) â score & gap filtering â _kBase LLMzero-/few-shot/CoTRAFT-LoRA LLM4-bit, r=16r=16 generatorRePASs: entailment â- contradiction ++ obligations Figure 1: The regulatory QA pipeline. Retrieval and generation are separate modules over a shared top-k context, so a single retrieval run can be reused across every generation strategy. 3.2 Leakage-Free Data Construction A precondition for every result below is that no passage seen at evaluation time was used in training. ObliQA does not satisfy this. It ships with predefined splits, but the sets of source documents behind them intersect: |Dtrainâ©Deval|>0|D_trainâ© D_eval|>0 for every pair. Because ObliQA questions are generated from passages, a model that has seen one question from a document has effectively seen the passage that answers a different question from the same document. This is passage-level trainâtest leakage, and it inflates retrieval scores in particular, since the gold passage is then an item the encoder was optimised on rather than an unseen target. We therefore re-split at document level, which excludes this form of leakage by construction. We therefore pool all questions and re-split at document level. The 40 ADGM documents are shuffled under a fixed seed and assigned 80%, 15%, and 5% to train, validation, and test, respectively. Single-passage questions follow their document. Multi-passage questions are admitted only if all of their passages land in the same split, and are otherwise discarded. This removes 989 questionâpassage pairs, which we consider an acceptable trade-off for a leakage-free evaluation. All numbers in this paper are computed on these re-split sets and are therefore not directly comparable to published ObliQA results measured on the original splits. Table 1 shows the resulting splits. Corpus construction is independent of the split: all 40 documents contribute their 13,705 passages, keyed by documentID-passageID. We then audited the new splits for residual overlap with exact matching, fuzzy matching at 85%, and embedding cosine similarity at 0.85. The first two find nothing. The third flags 139 trainâeval question pairs, but inspection shows these are distinct obligations phrased alike (liquidity-risk stress testing appears under several rules), which is a property of regulatory drafting rather than leakage, so we retain them. Open Australian LegalQA is converted to the same schema. After removing 2 duplicate questions from 2,124 records, we group by legal citation (e.g. Nasr v NRMA Insurance [2006] NSWSC 1018) as the document identifier and split 80/20 by group, giving 1,695 training and 427 evaluation items over a 2,114-passage corpus. The same overlap audit finds 13 fuzzy and 16 semantic near-duplicates, all of which are distinct questions about parallel instruments, such as consecutive tariff concession orders or successive airworthiness directives, and are likewise kept. Table 1: Datasets after leakage-free re-splitting. ObliQA splits are disjoint at document level; Open Australian LegalQA is grouped by legal citation. Dataset Split Documents Questions Corpus passages ObliQA (ADGM) train 32 20,573 13,705 validation 6 3,191 test 2 3,116 Open Australian train â 1,695 2,114 LegalQA evaluation â 427 3.3 Stage 1: Entailment Tuning The base encoder is LegalBERT [15]. Its pretraining objective gives it legal vocabulary but no notion of queryâpassage geometry, so we first retrain it on a task that is structurally closer to retrieval in this domain: deciding whether a passage supports a claim. Each question q is rewritten into a declarative claim c=fâĄ(q)c=f(q) by a rule-based mapping over six question types (when, why, who, where, does, how) with a generic fallback, so that a who question becomes an assertion about the responsible entity. The gold passage p+p^+ becomes the premise and c the hypothesis, giving every training instance the natural-language inference form X="ââšp+â©âentails thatââšHmaskedâ©â".X= "\, p^+ \ entails that\ H_masked \, ". (1) Each hypothesis token is independently replaced by [MASK] with probability ÎČ=0.8ÎČ=0.8, far above conventional MLM rates, and the model is trained to reconstruct only the masked positions: âmlm=ââiâMASKlogP(h^i=hiâŁX).L_mlm=-\!\! _iâ\MASK\\!\! P( h_i=h_i X). (2) The high masking rate is intentional. When most of the hypothesis is masked, the model cannot simply copy tokens and has to use information from the premise to reconstruct it, which is the kind of queryâpassage matching that retrieval requires [37]. Training runs for 5 epochs (4 on Open Australian LegalQA) with AdamW at 2Ă10â52Ă 10^-5, mixed precision, and gradient clipping at 1.0. The tuned encoder is converted into a bi-encoder by mean-pooling token embeddings under the SentenceTransformer interface. 3.4 Stage 2: Contrastive Tuning Entailment tuning teaches the model whether a passage supports a claim, but it does not explicitly separate relevant from irrelevant passages in the embedding space. Stage 2 optimises the embedding space directly with the multiple-negatives ranking loss over in-batch negatives [34]. For a batch of N queryâpassage pairs, â=âlogâĄexpâĄ(simâĄ(qi,pi+)/Ï)âj=1NexpâĄ(simâĄ(qi,pj)/Ï),L=- (sim(q_i,p_i^+)/Ï) _j=1^N (sim(q_i,p_j)/Ï), (3) with simsim the cosine similarity and Ï a temperature. Queries and passages carry the prefixes query: and passage:; the same prefixes are used at inference, since mismatched prefixes degrade retrieval performance. Batch size is 16, learning rate 2Ă10â52Ă 10^-5 with 100 warm-up steps, 2 epochs on ObliQA and 3 on Open Australian LegalQA, selected on validation Recall@10 from schedules up to 5 epochs. We deliberately do not mine hard negatives, although we implemented mining with E5-large-v2 and a BGE reranker. We made this choice for two reasons. First, ObliQA is a multi-passage dataset: a passage that is semantically adjacent to the gold one is frequently a second valid answer, so labeling it negative pushes a relevant passage away. Second, in our development runs mined negatives reduced Recall@10 rather than improving it, consistent with the high terminological overlap of regulatory drafting. Equation (3) already supplies Nâ1N-1 negatives per query at no additional cost and led to more stable training. 3.5 Stage 3: Hybrid Fusion and Context Assembly Dense retrieval alone under-weights the exact identifiers (rule numbers, defined terms, entity names) that regulatory questions often hinge on. We therefore score each query with both retrievers, normalize each score by its per-query maximum, s^â=sâmaxâĄ(sâ)+Ï”,Ï”=10â9, s_ = s_ (s_ )+Δ, Δ=10^-9, (4) and fuse them linearly, as seen in [39]: shybrid=αâs^dense+(1âα)âs^bm25.s_hybrid=α\, s_dense+(1-α)\, s_bm25. (5) We set α=0.8α=0.8, keeping the tuned dense retriever dominant while letting BM25 break ties on exact terminology. The top k=10k=10 passages are filtered before they reach the generator: passages scoring below 0.7 are dropped, and if consecutive ranked scores fall by more than 0.2 the remaining tail is truncated. At least one passage is always retained. This filtering is important because RePASs measures obligation coverage against the supplied context. Weakly relevant passages increase the number of obligations an answer is expected to cover and thus penalize the generator for retrieval errors. 3.6 Generation with Prompting The first generator variant conditions a frozen instruction-tuned model on (q,Pk)(q,P_k). All models receive an identical system instruction, namely to act as a regulatory compliance assistant and synthesize every obligation in the supplied passages into one coherent answer, under three strategies: zero-shot; few-shot with three worked examples, one of which demonstrates the correct response to a question the context does not answer; and few-shot with chain-of-thought [56, 57, 58], adding explicit reasoning instructions and constraints while still emitting only the final answer. Decoding is greedy with a 512-token budget. 3.7 Generation with RAFT-LoRA The second variant adapts the generator. Following RAFT [46], we build training instances from our own hybrid retriever so that the distractors the model learns to resist are the ones it will actually see. For a fraction P=0.7P=0.7 of questions the context contains the oracle passage plus distractors; for the remaining 0.30.3 it contains distractors only: 70%: 70\%: Qi+Diâ+D1+D2+D3âAiâ Q_i+D_i^*+D_1+D_2+D_3â A_i^* (6) 30%: 30\%: Qi+D1+D2+D3+D4âAiâ Q_i+D_1+D_2+D_3+D_4â A_i^* Passage order is shuffled to suppress positional bias. Targets AiâA_i^* are produced by GPT-4o-mini [59] as a teacher, prompted to emit a Reason section followed by an Answer section, to use only the supplied documents, and to state explicitly when the documents do not settle the question. Two properties of this setup should be stated explicitly. The targets AiâA_i^* are teacher outputs from GPT-4o-mini rather than human-written gold answers, so what the adapted models learns is the teacherâs answer behaviour, namely exhaustive enumeration of the obligations present in the supplied passages and explicit abstention when they are absent, over contexts that our own retriever produced. The RAFT training sets are also deliberately small, at 150 to 200 instances per corpus or under 1% of the available training questions. We therefore present this as an efficient adaptation recipe, one teacher pass and a few hundred optimisation steps on a single GPU, and not as evidence that the resulting behaviour is independent of the teacher model or of how the training contexts were assembled. Adaptation uses QLoRA [49]: the base model is frozen in 4-bit NF4, and rank-16 LoRA adapters are injected into the query and value projections. Loss is computed only over the reason and answer spans, with instruction, question, and context tokens masked, so the model is not trained to reproduce its input. Adapters are merged into the base weights for inference and also stored separately for reuse. Table 2 lists the configuration. Table 2: RAFT dataset construction and QLoRA adaptation settings, shared across all adapted models. Component Setting Base models Qwen2.5-7B, Teuken-7B v0.6, Gemma-2-2B Quantisation 4-bit NF4, FP16 compute LoRA rank r / α / dropout 16 / 32 / 0.05 Target modules q_proj, v_proj Optimiser paged AdamW (8-bit) Learning rate / warm-up 2Ă10â42Ă 10^-4 / 0.05 Epochs / batch size 3 / 1 Teacher model (CoT labels) GPT-4o-mini Oracle-passage probability P 0.7 Loss completion-only cross-entropy 3.8 Evaluation Retrieval is measured by Recall@10, MRR@10, and nDCG@10 against the annotated gold passages. Generation is measured with RePASs [13], which scores an answer against the retrieved context rather than against a reference string. For answer sentences aia_i and passage sentences pjp_j, entailment and contradiction take the best-matching passage sentence per answer sentence, Es=1NââimaxjâĄPentâ(pj,ai),Cs=1NââimaxjâĄPconâ(pj,ai),E_s= 1N _i _jP_ent(p_j,a_i), C_s= 1N _i _jP_con(p_j,a_i), (7) while obligation coverage first classifies which context sentences state obligations, using a LegalBERT obligation classifier, and then checks how many are entailed by some answer sentence above a 0.7 threshold: OâCs=1Mââk(maxlâĄPentâ(ok,al)>0.7).OC_s= 1M _k1\! ( _lP_ent(o_k,a_l)>0.7 ). (8) The composite is RePASs=EsâCs+OâCs+13â[0,1].RePASs= E_s-C_s+OC_s+13â[0,1]. (9) Entailment and contradiction come from nli-deberta-v3-xsmall and obligation matching from deberta-large-mnli. We report all three components separately because Equation (9) combines terms that can move in opposite directions, so two systems with the same composite score may behave quite differently. 4 Experiments 4.1 Setup All experiments run on single NVIDIA T4, L4, or A100 GPUs, with A100s reserved for generation and LoRA training. Generators are loaded in 4-bit NF4 quantization via bitsandbytes, so every model in Table 5 fits on one commodity accelerator. We evaluate five compact instruction-tuned models: Qwen2.5-7B [60], Teuken-7B v0.6 [61], Gemma-2-2B [62], Gemma-3-12B [63], and DeepSeek-R1-Distill-Llama-8B [64]. All reported numbers are computed on the validation split of each corpus; the two-document ObliQA test split is held out and untouched. Retrieval is evaluated exhaustively against the complete corpora, on 3,733 ObliQA questionâpassage pairs drawn from 3,191 questions and on 427 Australian queries. Generation is evaluated on a fixed sample of 150 validation questions per configuration, held constant across models and strategies; the sample is a concession to the cost of scoring long-form answers with two NLI models, and we return to it in Section 4.6. 4.2 Retrieval Table 3 reports the staged ablation and Figure 2 plots it. Table 3: Staged retriever ablation. Each stage is initialized from the previous one; the hybrid adds BM25 at α=0.8α=0.8. Retriever stage R@10 MRR@10 nDCG@10 ObliQA (ADGM rulebooks) LegalBERT (baseline) 0.256 0.158 0.181 + entailment tuning (ET) 0.495 0.322 0.363 + contrastive tuning (CT) 0.732 0.518 0.569 + BM25 fusion (hybrid) 0.774 0.594 0.638 Open Australian LegalQA LegalBERT (baseline) 0.190 0.120 0.137 + entailment tuning (ET) 0.726 0.595 0.626 + contrastive tuning (CT) 0.911 0.824 0.845 + BM25 fusion (hybrid) 0.923 0.846 0.865 Figure 2: Retrieval quality after each adaptation stage. The ordering of stages is identical on both corpora, but their relative contributions are not: entailment tuning is decisive on Australian case law, contrastive tuning on the ADGM rulebooks. Values are those of Table 3. âOff-the-shelfâ LegalBERT performs poorly as a retriever: it ranks the gold passage among the top ten for only one in four ObliQA queries and one in five Australian queries. Therefore, Domain-specific pretraining alone does not yield an embedding space suitable for retrieval. Entailment tuning roughly doubles every ObliQA metric (Recall@10 0.256â0.4950.256â 0.495) and has a far larger effect on Australian case law (0.190â0.7260.190â 0.726). We attribute this difference to the question types. Australian questions name a case or instrument and ask what it provides, so a single passage decisively entails the answer and the premiseâhypothesis objective aligns almost perfectly with the task. ADGM questions ask what a class of regulated entity must do, and several provisions bear on that, so entailment alone leaves ambiguity. Contrastive tuning is the larger step on ObliQA (0.495â0.7320.495â 0.732) and consolidates the Australian gain (0.726â0.9110.726â 0.911). We attribute this to the model learning to distinguish between passages with similar regulatory vocabulary, which is the main difficulty in ObliQA. BM25 fusion adds a final +0.042+0.042 on ObliQA and +0.012+0.012 on Australian data, with a disproportionate effect on ranking quality: on ObliQA, Recall@10 rises 5.7% relative while MRR@10 rises 14.7%, i.e. BM25 mostly promotes passages the dense model had already retrieved but ranked too low. That is the expected signature of exact-term evidence, and it is worth more than the recall number suggests, because a generator reads the top of the list most attentively. Table 4: ObliQA retrieval against standard baselines, full validation split (3,733 questionâpassage pairs, 13,705-passage corpus). Retriever Type R@10 MRR@10 nDCG@10 BM25 lexical 0.678 0.504 0.546 BGE-M3 dense 0.711 0.544 0.584 BGE-base-en-v1.5 dense 0.719 0.549 0.590 E5-base-v2 dense 0.721 0.557 0.597 E5-large-v2 dense 0.758 0.595 0.635 Hybrid ET+CT (ours) lexical+dense 0.774 0.594 0.638 Table 4 places the hybrid retriever against standard alternatives on ObliQA. BM25 alone reaches 0.678, within 0.04 Recall@10 of BGE-M3 and BGE-base despite learning nothing about the domain, which underlines the importance of exact terminology in regulatory retrieval. Among dense baselines E5-large-v2 is strongest at 0.758. Our hybrid reaches 0.774 Recall@10 and 0.638 nDCG@10, above every baseline, while its dense component is a BERT-base encoder, roughly a third of E5-largeâs parameters, adapted on in-domain data alone. We note that E5-large-v2 attains a marginally higher MRR@10 (0.595 vs. 0.594); the two systems are equivalent on first-hit rank and differ in the depth of the ranking. 4.3 Generation on ObliQA Table 5 reports RePASs and its components for every model and strategy; Figure 3 summarizes the base-versus-adapted comparison on both corpora. Our four key findings on ObliQA are: Table 5: Answer generation on ObliQA, 150 evaluation questions, hybrid retriever context. Best RePASs per model in bold. Model Strategy EsâE_s CsâC_s OâCsâOC_s RePASsâ Qwen2.5-7B zero-shot 0.894 0.189 0.301 0.668 few-shot 0.971 0.167 0.371 0.725 few-shot + CoT 0.957 0.154 0.359 0.721 RAFT-LoRA 0.926 0.157 0.409 0.725 Teuken-7B zero-shot 0.944 0.256 0.332 0.673 v0.6 few-shot 0.928 0.200 0.259 0.662 few-shot + CoT 0.929 0.225 0.261 0.655 RAFT-LoRA 0.877 0.134 0.397 0.713 Gemma-2-2B zero-shot 0.935 0.317 0.167 0.594 few-shot 0.908 0.342 0.128 0.565 few-shot + CoT 0.900 0.326 0.113 0.562 RAFT-LoRA 0.934 0.212 0.277 0.666 Gemma-3-12B zero-shot 0.961 0.312 0.328 0.659 few-shot 0.930 0.264 0.333 0.666 few-shot + CoT 0.911 0.182 0.353 0.694 RAFT-LoRA unstable adaptation â excluded DeepSeek-R1 zero-shot 0.693 0.365 0.877 0.735 Distill-8B few-shot 0.605 0.335 0.836 0.702 few-shot + CoT 0.629 0.365 0.876 0.713 RAFT-LoRA unstable adapter merge â excluded 1. Prompting does not generalize across models. Qwen2.5 improves from 0.668 zero-shot to 0.725 with few-shot examples. Teuken degrades (0.673â0.6620.673â 0.662), Gemma-2 degrades further (0.594â0.5650.594â 0.565), and DeepSeek-R1 also loses ground (0.735â0.7020.735â 0.702). Adding chain-of-thought on top of few-shot helps only Gemma-3 (0.666â0.6940.666â 0.694) and DeepSeek-R1 (0.702â0.7130.702â 0.713), and is neutral-to-negative elsewhere. The component columns explain why: for Teuken and Gemma-2 the examples cut obligation coverage sharply (Teuken 0.332â0.2590.332â 0.259, Gemma-2 0.167â0.1280.167â 0.128) without a compensating drop in contradiction. Our three demonstrations are short, and models with weaker instruction-following imitate their length and shape rather than their reasoning, truncating the exhaustive obligation enumeration that RePASs rewards. This is consistent with evidence that demonstrations mostly convey format [65], and it means that few-shot prompting is not a safe default in this domain: it must be validated per model. 2. RAFT-LoRA improves every model it can be applied to. Qwen2.5 gains +0.057+0.057 (0.668 â 0.725), Teuken +0.040+0.040 (0.673 â 0.713), and Gemma-2 +0.072+0.072 (0.594 â 0.666). The mechanism is visible in the components and is the same in all three cases: obligation coverage rises substantially (Qwen 0.301â0.4090.301â 0.409, Teuken 0.332â0.3970.332â 0.397, Gemma-2 0.167â0.2770.167â 0.277) while contradiction falls (Qwen 0.189â0.1570.189â 0.157, Teuken 0.256â0.1340.256â 0.134, Gemma-2 0.317â0.2120.317â 0.212). This matches the goal of the RAFT objective, namely covering what the context supports without adding unsupported claims. Interestingly, Teukenâs entailment score drops from 0.944 to 0.877 at the same time. Teukenâs base answers are short and closely paraphrase one passage, which scores well on entailment and poorly on coverage; the adapted model writes longer, multi-provision answers in which some sentences are aggregations rather than restatements. For compliance use the adapted behaviour is preferable, and the composite metric agrees, but the raw entailment column would have suggested a regression. Gemma-2-2B, the smallest model, benefits most from adaptation. In practice, this means that a few hundred training steps of retrieval-aware adaptation can noticeably improve grounding in a 2B model. 3. Adaptation is not universally available. Two of the five models could not be adapted. Merging LoRA adapters into DeepSeek-R1-Distill produced inconsistent generation at inference, and Gemma-3 showed near-zero loss movement across epochs, indicating that the target sequence was recoverable from the input and no useful gradient signal was present. We exclude both models from the comparison. For deployment, however, this shows that LoRA adaptation and adapter merging do not work equally reliably across architectures. 4. The strongest single ObliQA number belongs to an unadapted model. DeepSeek-R1-Distill reaches 0.735 zero-shot, above every RAFT-LoRA result. Its profile is unlike the others: obligation coverage 0.877, far higher than any other system, with entailment of only 0.693 and contradiction of 0.365. A reasoning-distilled model enumerates context obligations exhaustively, which RePASs rewards twice: directly through OâCsOC_s, and indirectly because long enumerations dilute the per-sentence contradiction average. Its lower entailment, meanwhile, shows that much of what it writes is not directly supported by any single passage. The composite score cannot tell whether these answers are more useful for compliance than Qwenâs more conservative ones. Figure 3: RAFT-LoRA against the zero-shot base model. Left: adapters trained and evaluated on ObliQA improve every model. Right: the same ObliQA-trained adapters evaluated on Australian case law, where only Teuken retains a gain. Values are those of Tables 5 and 6. 4.4 Cross-Domain Transfer The Australian results in Table 6 serve two purposes: they test whether the retriever and prompting conclusions hold on a different legal genre, and they test whether the ObliQA-trained adapters transfer. The adapters are reused unchanged; no Australian RAFT data was generated. Table 6: Answer generation on Open Australian LegalQA, 150 evaluation questions. â marks adapters trained on ObliQA and applied without retraining. Model Strategy EsâE_s CsâC_s OâCsâOC_s RePASsâ Qwen2.5-7B zero-shot 0.964 0.414 0.377 0.642 few-shot 0.964 0.402 0.350 0.637 few-shot + CoT 0.952 0.492 0.242 0.567 RAFT-LoRAâ 0.957 0.451 0.238 0.581 Teuken-7B zero-shot 0.943 0.552 0.288 0.559 v0.6 few-shot 0.941 0.578 0.212 0.525 few-shot + CoT 0.945 0.587 0.209 0.522 RAFT-LoRAâ 0.951 0.463 0.271 0.586 Gemma-2-2B zero-shot 0.922 0.639 0.151 0.477 few-shot 0.892 0.676 0.116 0.444 few-shot + CoT 0.881 0.676 0.104 0.436 RAFT-LoRAâ 0.940 0.592 0.013 0.453 Gemma-3-12B zero-shot 0.964 0.515 0.237 0.562 few-shot 0.959 0.545 0.200 0.538 few-shot + CoT 0.955 0.571 0.176 0.520 RAFT-LoRA unstable adaptation â excluded DeepSeek-R1 zero-shot 0.571 0.610 0.850 0.604 Distill-8B few-shot 0.552 0.563 0.844 0.611 few-shot + CoT 0.552 0.573 0.862 0.613 RAFT-LoRA unstable adapter merge â excluded The findings on prompting are even clearer here: zero-shot is the best prompted strategy for Qwen, Teuken, Gemma-2, and Gemma-3, and few-shot and chain-of-thought degrade every one of them. Only DeepSeek-R1 prefers chain-of-thought (0.613), again through obligation coverage. The adapters, however, mostly do not transfer. Qwen drops from 0.642 to 0.581 and Gemma-2 from 0.477 to 0.453; only Teuken improves, 0.559 to 0.586. The component scores show why: Gemma-2âs obligation coverage collapses to 0.013, because the adapted model reproduces the terse rulebook-style answer it was trained to produce, which covers almost nothing in a long case-law passage. Qwenâs coverage falls from 0.377 to 0.238 for the same reason. Teuken is the exception because its adaptation gain came predominantly from contradiction reduction (0.552â0.4630.552â 0.463), and learning not to assert unsupported claims is a genre-independent behaviour, whereas learning the shape of an ADGM obligation answer is not. We attribute this to two structural differences between the corpora. ObliQA passages are short numbered provisions with an explicit normative operator, and the mapping from retrieved passage to answer is close to deterministic. Australian passages are extended excerpts of judicial reasoning in which the answer is distributed across the passage and rarely stated as an obligation at all. Contradiction scores are also consistently higher on the Australian data: every model contradicts its context far more often on Australian data (0.40â0.68) than on ObliQA (0.13â0.37), even though retrieval is much better there (Recall@10 0.923 vs. 0.774). Long argumentative passages contain positions the court ultimately rejects, and an NLI model scoring a summary against every sentence of such a passage will find contradictions that are not errors. This is a limitation of sentence-level NLI metrics on case law, and it means Table 6 should be read within-column rather than against Table 5. The practical implication is that RAFT-LoRA adapters are corpus-specific artifacts. A firm deploying this pipeline across several regulatory regimes should expect to build RAFT data per regime rather than to reuse one adapter. 4.5 Is Retrieval Doing the Work? To check that the retrieval pipeline is responsible for the answer quality we measure, we ran Qwen2.5 and Teuken closed-book: same prompt, same questions, no passages. RePASs is still computed against the retrieved passages the model never saw. These runs cover 132 and 89 validation questions respectively rather than the full 150, so they are indicative rather than precise; the effect below is far larger than that sample difference can explain. Table 7: Closed-book control on ObliQA. Scores are computed against retrieved passages that the closed-book models did not receive. Closed-book runs cover 132 (Qwen) and 89 (Teuken) validation questions. Model Setting EsâE_s CsâC_s OâCsâOC_s RePASsâ Qwen2.5-7B closed-book 0.883 0.081 0.169 0.657 RAG (zero-shot) 0.894 0.189 0.301 0.668 Teuken-7B v0.6 closed-book 0.836 0.096 0.247 0.662 RAG (zero-shot) 0.944 0.256 0.332 0.673 Table 7 shows the results. Closed-book Qwen scores 0.657 compared to 0.668 with retrieval, and closed-book Teuken 0.662 compared to 0.673. A difference of about 0.01 clearly understates the gap between an answer grounded in the ADGM rulebook and one generated from parametric knowledge alone. Inspection of the closed-book outputs confirms that they are plausible-sounding regulatory prose that misstates obligations, omits binding conditions, and cites nothing. The components show how the score is obtained. Closed-book contradiction is very low (0.081 and 0.096, against 0.189 and 0.256 with retrieval), because an answer that never commits to a specific provision has little to contradict; and EsE_s takes a maximum over passage sentences, so generic regulatory statements find some sentence that entails them. Obligation coverage does fall as expected (Qwen 0.301â0.1690.301â 0.169), but it is one of three terms in Equation (9) and the contradiction reduction offsets most of it. We draw two conclusions from this. First, RePASs should not be used on its own to rank regulatory QA systems, and results should be reported together with a closed-book baseline. Second, the main benefit of retrieval in our setting is attributability, i.e., the ability to trace an answer back to a specific provision. Compliance applications require this property, but RePASs does not capture it. Of its three components, only obligation coverage clearly separates the retrieval and closed-book settings. 4.6 Limitations Several constraints bound these results. Generation is scored on 150 questions per configuration, and the closed-book control on fewer still; differences of a few thousandths of RePASs, such as Qwenâs tie between few-shot and RAFT-LoRA, are within noise, and we make no claims at that resolution. The leakage-free re-split discards 989 multi-passage questionâpassage pairs, tilting ObliQA toward single-passage questions and away from the multi-provision reasoning that real compliance work demands. RePASs, as Section 4.5 shows, is not a sufficient measure of grounding; an attribution-sensitive metric or an LLM-as-judge protocol [66] would be a better instrument, and human expert evaluation better still. Finally, the two corpora studied here are both English and both drawn from public sources; internal policy documents, tabular disclosures, and multilingual rulebooks remain untested. 5 Conclusion We investigated how much of the gap between compact language models and reliable regulatory question answering can be closed by adapting the pipeline instead of scaling the model. The retrieval half of that gap closes substantially and is directly measured: a LegalBERT encoder that ranks the correct Abu Dhabi Global Market (ADGM) provision among its top ten for one query in four does so for three queries in four after entailment tuning, contrastive tuning, and BM25 fusion, ahead of E5-large-v2 at a fraction of the parameters. The generation half improves on the metric available to us, in that retrieval-aware LoRA adaptation raises composite RePASs for every compact generator we could adapt, most for the smallest, by increasing obligation coverage and reducing contradictions on a few hundred teacher-labelled examples. We describe this as improved in-domain answer behaviour rather than as improved grounding, because our closed-book control shows that RePASs does not distinguish the two. Three conclusions are supported by these experiments. First, domain-adapted retrieval is effective: staged adaptation of a small in-domain encoder outperforms both a strong lexical baseline and larger general-purpose dense encoders on regulatory text. Second, RAFT-LoRA improves compact generators in-domain, and in-domain only: adapters trained on ADGM rulebook data lost most of their benefit on Australian case law, so deployments spanning several regulatory regimes should plan for regime-specific RAFT data rather than adapter reuse. Third, current evaluation metrics can substantially understate the value of retrieval. A closed-book model that received no passages, cited nothing, and misstated obligations scored within 0.011 RePASs of the full pipeline, so RePASs cannot on its own validate a compliance system. Therefore, Retrieval remains essential for transparency and traceability, but regulatory NLP needs evaluation protocols that can measure these properties. Acknowledgments This research has been partially funded by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence. For this paper, Anthropic Claude Opus 5 [67] was employed to assist in refining and improving the text throughout all sections of this paper. The authors retain full responsibility for the accuracy, integrity, and originality of the work. References [1] D. Jeong, J. Holehouse, J. Yoon, C. P. Kempes, G. B. West, and H. Youn (2026) A dataset showing a century of evolution in the complexity of the united states legal code. Scientific data. Cited by: §1. [2] R. Sifa, A. Ladi, M. Pielka, R. Ramamurthy, L. Hillebrand, B. Kirsch, D. Biesner, R. Stenzel, T. Bell, M. LĂŒbbering, et al. (2019) Towards automated auditing with machine learning. In Proc. DocEng, Cited by: §1. [3] J. Von Solms (2021) Integrating regulatory technology (regtech) into the digital transformation of a bank treasury. Journal of Banking Regulation. Cited by: §1. [4] R. Ramamurthy, M. Pielka, R. Stenzel, C. Bauckhage, R. Sifa, T. D. Khameneh, U. Warning, B. Kliem, and R. Loitz (2021) ALiBERT: improved automated list inspection (ali) with bert. In Proc. DocEng, Cited by: §1. [5] A. Berger, L. Hillebrand, D. Leonhard, T. DeuĂer, T. B. F. De Oliveira, T. Dilmaghani, M. Khaled, B. Kliem, R. Loitz, C. Bauckhage, et al. (2023) Towards automated regulatory compliance verification in financial auditing with large language models. In Proc. BigData, Cited by: §1. [6] T. DeuĂer, D. Leonhard, L. Hillebrand, A. Berger, M. Khaled, S. Heiden, T. Dilmaghani, B. Kliem, R. Loitz, C. Bauckhage, et al. (2023) Uncovering inconsistencies and contradictions in financial reports using large language models. In Proc. BigData, Cited by: §1. [7] H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian (2025) A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology. Cited by: §1. [8] T. DeuĂer, G. Ramien, N. Weber, M. Meidinger, M. HahnbĂŒck, C. Bauckhage, and R. Sifa (2025) Leveraging synthetically generated data for real estate document classification. In Proc. BigData, Cited by: §1. [9] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. RĂ©, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. TramĂšr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang (2022) On the opportunities and risks of foundation models. External Links: 2108.07258, Link Cited by: §1. [10] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. KĂŒttler, M. Lewis, W. Yih, T. RocktĂ€schel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc. NeurIPS, Cited by: §1, §2.3. [11] N. Thakur, N. Reimers, A. RĂŒcklĂ©, A. Srivastava, and I. Gurevych (2021) BEIR: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. NeurIPS Datasets and Benchmarks, Cited by: §1. [12] T. DeuĂer, M. HahnbĂŒck, T. Uelwer, C. Zhao, C. Bauckhage, and R. Sifa (2025) Resource-efficient anonymization of textual data via knowledge distillation from large language models. In Proc. COLING, Cited by: §1. [13] T. Gokhan, K. Wang, I. Gurevych, and T. Briscoe (2024) RIRAG: regulatory information retrieval and answer generation. External Links: 2409.05677, Link Cited by: §1, §2.1, §3.8. [14] U. Butler (2025) Open australian legal corpus. Isaacus. External Links: Document, Link Cited by: §1. [15] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos (2020) LEGAL-BERT: the muppets straight out of law school. In Findings of the ACL: EMNLP 2020, Cited by: 1st item, §2.2, §2.5, §3.3. [16] H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun (2020) How does NLP benefit legal system: a summary of legal artificial intelligence. In Proc. ACL, Cited by: §2.1. [17] D. Hendrycks, C. Burns, A. Chen, and S. Ball (2021) CUAD: an expert-annotated nlp dataset for legal contract review. In Proc. NeurIPS Datasets and Benchmarks, Cited by: §2.1. [18] I. Chalkidis, A. Jana, D. Hartung, M. Bommarito, I. Androutsopoulos, D. Katz, and N. Aletras (2022) LexGLUE: a benchmark dataset for legal language understanding in English. In Proc. ACL, Cited by: §2.1. [19] Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. Routledge, and W. Y. Wang (2021) FinQA: a dataset of numerical reasoning over financial data. In Proc. EMNLP, Cited by: §2.1. [20] I. Chalkidis, M. Fergadiotis, N. Manginas, E. Katakalou, and P. Malakasiotis (2021) Regulatory compliance through Doc2Doc information retrieval: a case study in EU/UK legislation where text similarity has limitations. In Proc. EACL, Cited by: §2.1. [21] Y. Abdullin, D. Molla, B. Ofoghi, J. Yearwood, and Q. Li (2023) Synthetic dialogue dataset generation using LLM agents. Association for Computational Linguistics, Singapore. External Links: Link Cited by: §2.1. [22] A. H. Bashir, M. R. Khalid, K. Cvejoski, J. Birr, J. Berghaus, A. Berger, S. Halscheidt, C. Temath, R. Sifa, and D. Berghaus (2026) Domain-adaptation through synthetic data: fine-tuning large language models for german law. External Links: 2601.14160, Link Cited by: §2.1. [23] M. J. Sergot, F. Sadri, R. A. Kowalski, F. Kriwaczek, P. Hammond, and H. T. Cory (1986) The british nationality act as a logic program. Communications of the ACM. Cited by: §2.1. [24] H. Turtle (1995) Text retrieval in the legal world. Artificial Intelligence and Law. Cited by: §2.1. [25] P. Quaresma and I. Rodrigues (2005) A question-answering system for portuguese juridical documents. In Proc. ICAIL, Cited by: §2.1. [26] I. Chalkidis, I. Androutsopoulos, and N. Aletras (2019) Neural legal judgment prediction in English. In Proc. ACL, Cited by: §2.1. [27] H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun (2020) JEC-qa: a legal-domain question answering dataset. In Proc. AAAI, Cited by: §2.1. [28] L. Zheng, N. Guha, B. R. Anderson, P. Henderson, and D. E. Ho (2021) When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proc. ICAIL, Cited by: §2.1. [29] P. M. S. Choi, S. H. Huang, and Q. Wang (2025) Large language models in finance: an overview. Finance and Large Language Models. Cited by: §2.1. [30] S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford (1994) Okapi at TREC-3. In Proc. TREC, Cited by: §2.2. [31] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering. In Proc. EMNLP, Cited by: §2.2. [32] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei (2024) Multilingual e5 text embeddings: a technical report. External Links: 2402.05672, Link Cited by: §2.2. [33] K. Luo, Z. Liu, S. Xiao, T. Zhou, Y. Chen, J. Zhao, and K. Liu (2024) Landmark embedding: a chunking-free embedding method for retrieval augmented long-context large language models. In Proc. ACL, Cited by: §2.2. [34] N. Reimers and I. Gurevych (2019) Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proc. EMNLP-IJCNLP, Cited by: §2.2, §2.5, §3.4. [35] M. Henderson, R. Al-Rfou, B. Strope, Y. Sung, L. Lukacs, R. Guo, S. Kumar, B. Miklos, and R. Kurzweil (2017) Efficient natural language response suggestion for smart reply. External Links: 1705.00652, Link Cited by: §2.2. [36] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In Proc. ICML, Cited by: §2.2. [37] L. Dai, H. Liu, and H. Xiong (2024) Improve dense passage retrieval with entailment tuning. In Proc. EMNLP, Cited by: §2.2, §2.5, §3.3. [38] M. Luo, S. Jain, A. Gupta, A. Einolghozati, B. Oguz, D. Chatterjee, X. Chen, C. Baral, and P. Heidari (2023) A study on the efficiency and generalization of light hybrid retrievers. In Proc. ACL, Cited by: §2.2. [39] S. Wang, S. Zhuang, and G. Zuccon (2021) BERT-based dense retrievers require interpolation with bm25 for effective passage retrieval. In Proc. ICTIR, Cited by: §2.2, §2.5, §3.5. [40] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang (2024) Retrieval-augmented generation for large language models: a survey. External Links: 2312.10997, Link Cited by: §2.3. [41] G. Izacard and E. Grave (2021) Leveraging passage retrieval with generative models for open domain question answering. In Proc EACL, Cited by: §2.3. [42] L. Hillebrand, A. Berger, D. Uedelhoven, D. Berghaus, U. Warning, T. Dilmaghani, B. Kliem, T. Schmid, R. Loitz, and R. Sifa (2024) Advancing risk and quality assurance: a rag chatbot for improved regulatory compliance. In Proc. BigData, Cited by: §2.3. [43] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi (2024) Self-RAG: learning to retrieve, generate, and critique through self-reflection. In Proc. ICLR, Cited by: §2.3. [44] C. Xiang, T. Wu, Z. Zhong, D. Wagner, D. Chen, and P. Mittal (2024) Certifiably robust rag against retrieval corruption. External Links: 2405.15556, Link Cited by: §2.3. [45] C. Chan, C. Xu, R. Yuan, H. Luo, W. Xue, Y. Guo, and J. Fu (2024) RQ-RAG: learning to refine queries for retrieval augmented generation. In Proc. COLM, Cited by: §2.3. [46] T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez (2024) RAFT: adapting language model to domain specific RAG. In Proc. COLM, Cited by: §2.3, §2.5, §3.7. [47] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for NLP. In Proc. ICML, Cited by: §2.4. [48] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In Proc. ICLR, Cited by: §2.4. [49] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized llms. In Proc. NeurIPS, Cited by: §2.4, §3.7. [50] H. Qin, X. Ma, X. Zheng, X. Li, Y. Zhang, S. Liu, J. Luo, X. Liu, and M. Magno (2024) Accurate LoRA-finetuning quantization of LLMs via information retention. In Proc. ICML, Cited by: §2.4. [51] H. Guo, P. Greengard, E. Xing, and Y. Kim (2024) LQ-loRA: low-rank plus quantized matrix decomposition for efficient language model finetuning. In Proc. ICLR, Cited by: §2.4. [52] A. Louis, G. Van Dijck, and G. Spanakis (2024) Interpretable long-form legal question answering with retrieval-augmented large language models. In Proc. AAAI, Cited by: §2.5. [53] N. Wiratunga, R. Abeyratne, L. Jayawardena, K. Martin, S. Massie, I. Nkisi-Orji, R. Weerasinghe, A. Liret, and B. Fleisch (2024) CBR-rag: case-based reasoning for retrieval augmented generation in llms for legal question answering. In Proc. ICCBR, Cited by: §2.5. [54] A. B. Hou, O. Weller, G. Qin, E. Yang, D. Lawrie, N. Holzenberger, A. Blair-Stanek, and B. Van Durme (2025) CLERC: a dataset for U. S. legal case retrieval and retrieval-augmented analysis generation. In Findings of the ACL: NAACL, Cited by: §2.5. [55] N. Pipitone and G. H. Alami (2024) LegalBench-rag: a benchmark for retrieval-augmented generation in the legal domain. External Links: 2408.10343, Link Cited by: §2.5. [56] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. In Proc. NeurIPS, Cited by: §3.6. [57] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. In Proc. NeurIPS, Cited by: §3.6. [58] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv.. Cited by: §3.6. [59] OpenAI, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. MÄ dry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, B. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. J. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mely, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, D. P. Nguyen, D. Findlay, E. Oiwoh, E. Wong, E. Asdar, E. Proehl, E. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. OâConnell, I. OâConnell, I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider, I. Gulrajani, J. Coxon, J. Menick, J. Pachocki, J. Aung, J. Betker, J. Crooks, J. Lennon, J. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee, J. Shieh, J. Lin, J. Yu, J. Weng, J. Tang, J. Yu, J. Jang, J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman, J. Lachman, J. McKay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, J. Sitkin, J. Kraaijeveld, J. Gross, J. Kaplan, J. Snyder, J. Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi, K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu-Lemberg, K. Button, K. Liu, K. Howe, K. Muthukumar, K. Luther, L. Ahmad, L. Kai, L. Itow, L. Workman, L. Pathak, L. Chen, L. Jing, L. Guy, L. Fedus, L. Zhou, L. Mamitsuka, L. Weng, L. McCallum, L. Held, L. Ouyang, L. Feuvrier, L. Zhang, L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. Gupta, M. Shah, M. Yatbaz, M. J. Yang, M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang, M. Fradin, M. Pokrass, M. Castro, M. O. T. de Castro, M. Pavlov, M. Brundage, M. Wang, M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. Tezak, N. Felix, N. Kudige, N. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Campbell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. Leike, R. Gaubert, R. Zamani, R. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai, R. Ramchandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agarwal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao, S. Wu, S. Jomoto, S. Wu, Shuaiqi, Xia, S. Phene, S. Papay, S. Narayanan, S. Coffey, S. Lee, S. Hall, S. Balaji, T. Broda, T. Stramer, T. Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Cunninghman, T. Degry, T. Dimson, T. Raoux, T. Shadwell, T. Zheng, T. Underwood, T. Markov, T. Sherbakov, T. Rubin, T. Stasi, T. Kaftan, T. Heywood, T. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov (2024) GPT-4o system card. External Links: 2410.21276, Link Cited by: §3.7. [60] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu (2023) Qwen technical report. External Links: 2309.16609, Link Cited by: §4.1. [61] M. Ali, M. Fromm, K. Thellmann, J. Ebert, A. A. Weber, R. Rutmann, C. Jain, M. LĂŒbbering, D. Steinigen, J. Leveling, K. Klug, J. S. Buschhoff, L. Jurkschat, H. Abdelwahab, B. J. Stein, K. Sylla, P. Denisov, N. Brandizzi, Q. Saleem, A. Bhowmick, L. Helmer, C. John, P. O. Suarez, M. Ostendorff, A. Jude, L. Manjunath, S. Weinbach, C. Penke, O. Filatov, F. Barth, P. Mirza, L. Weber, I. Wendler, R. Sifa, F. KĂŒch, A. Herten, R. JĂ€kel, G. Rehm, S. Kesselheim, J. Köhler, and N. Flores-Herr (2025) Teuken-7b-base & teuken-7b-instruct: towards european llms. In Proc. ECAI, Cited by: §4.1. [62] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. RamĂ©, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette-Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. RogoziĆska, D. Herbison, E. Bandy, E. Wang, E. Noland, E. Moreira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak-PluciĆska, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernandez, J. van Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J. Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Görner, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardoliwalla, N. Devanathan, N. Dumai, N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin, S. M. R. Arnold, S. Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Z. Wei, V. Cotruta, P. Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev (2024) Gemma 2: improving open language models at a practical size. External Links: 2408.00118, Link Cited by: §4.1. [63] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. RamĂ©, M. RiviĂšre, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-PluciĆska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. PĂ”der, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot (2025) Gemma 3 technical report. External Links: 2503.19786, Link Cited by: §4.1. [64] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature. Cited by: §4.1. [65] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022) Rethinking the role of demonstrations: what makes in-context learning work?. In Proc. EMNLP, Cited by: §4.3. [66] D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, K. Shu, L. Cheng, and H. Liu (2025) From generation to judgment: opportunities and challenges of LLM-as-a-judge. In Proc. EMNLP, Cited by: §4.6. [67] Anthropic (2026) System card: claude opus 5. External Links: Link Cited by: Acknowledgments.