Paper deep dive
A Survey on Rubric-Guided Reinforcement Learning for Language Models
Zifei Shan, Fangning Shao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/1/2026, 4:36:44 AM
Summary
This survey introduces a Bayesian framework for Rubric-Guided Reinforcement Learning (RL) for Large Language Models (LLMs), defining constitutions as prior distributions over evaluation criteria and rubrics as conditional instantiations. It presents a taxonomy of methods along a prior-posterior axis, covering constitutional AI, instance-specific rubrics, process-level supervision, and self-evolving rubrics, while also analyzing linguistic challenges such as semantic drift and reward hacking.
Entities (8)
Relation Signals (6)
Rubric-Guided Reinforcement Learning ā improvesupon ā RLHF
confidence 95% Ā· Rubric-guided reinforcement learning addresses these limitations [of traditional RLHF] by introducing structured, interpretable evaluation criteria
Rubric ā servesas ā Reward Signal
confidence 94% Ā· The key insight of rubric-guided RL is that rubrics can serve as structured reward signals.
Rubric ā isformalizedas ā Conditional Instantiation
confidence 93% Ā· rubrics as conditional instantiations Rx ~ P(R|x)
Constitution ā isformalizedas ā Prior Distribution
confidence 93% Ā· constitutions as prior distributions P(R) over evaluation criteria
Constitutional AI ā isa ā Rubric-Guided Reinforcement Learning
confidence 92% Ā· Constitutional AI ... pioneered the use of explicit natural-language principles ... covering constitutional AI ... in the taxonomy
Semantic Drift ā impacts ā Alignment Reliability
confidence 90% Ā· semantic drift ... impact alignment reliability
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. However, traditional RLHF relies on scalar reward signals that lack interpretability and fail to capture the multifaceted nature of response quality. Rubric-guided reinforcement learning addresses these limitations by introducing structured, interpretable evaluation criteria, or rubrics, as the backbone of reward design, feedback generation, and policy optimization. In this survey, we introduce a Bayesian framework that defines constitutions as prior distributions $P(R)$ over evaluation criteria and rubrics as conditional instantiations $R_x \sim P(R|x)$. Under this unified view, we present a taxonomy of rubric-guided RL along the prior-posterior axis, covering constitutional AI, instance-specific rubrics, process-level supervision, self-evolving rubrics, and their agentic and multimodal extensions. Furthermore, as rubrics are natural-language artifacts, we present a linguistic analysis of how granularity trade-offs, semantic drift, and linguistic reward hacking impact alignment reliability, identifying key open problems for future research.
Tags
Links
- Source: https://arxiv.org/abs/2608.27505v2
- Canonical: https://arxiv.org/abs/2608.27505v2
Trouble viewing inline? Open PDF directly ā
Full Text
106,182 characters extracted from source content.
Expand or collapse full text
A Survey on Rubric-Guided Reinforcement Learning for Language Models Zifei Shan ā thanks: Equal contribution. Affiliation: WeChat, Tencent Email: zifeishan@tencent.com Fangning Shao11footnotemark: 1 Affiliation: Independent Researcher Email: fangning.shao@gmail.com Abstract Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. However, traditional RLHF relies on scalar reward signals that lack interpretability and fail to capture the multifaceted nature of response quality. Rubric-guided reinforcement learning addresses these limitations by introducing structured, interpretable evaluation criteria, or rubrics, as the backbone of reward design, feedback generation, and policy optimization. In this survey, we introduce a Bayesian framework that defines constitutions as prior distributions Pā”(ā)P(R) over evaluation criteria and rubrics as conditional instantiations āxā¼Pā”(ā|x)R_x P(R|x). Under this unified view, we present a taxonomy of rubric-guided RL along the prior-posterior axis, covering constitutional AI, instance-specific rubrics, process-level supervision, self-evolving rubrics, and their agentic and multimodal extensions. Furthermore, as rubrics are natural-language artifacts, we present a linguistic analysis of how granularity trade-offs, semantic drift, and linguistic reward hacking impact alignment reliability, identifying key open problems for future research. 1 Introduction The alignment of large language models (LLMs) with human values and preferences has emerged as one of the most critical challenges in modern AI research. Reinforcement learning from human feedback (RLHF, Ouyang et al., 2022) has established itself as the predominant paradigm for this alignment, demonstrating that models trained with human-preference-driven reward signals can produce more helpful, harmless, and honest outputs. However, as LLMs are deployed in increasingly diverse and complex domains, the limitations of traditional RLHF, which relies on scalar, opaque reward signals, have become increasingly apparent Gao et al. (2023); Wu et al. (2023); Wang et al. (2024). A rubric is a structured set of evaluation criteria that specifies what constitutes different levels of performance across multiple dimensions. Rubrics have found natural application in LLM evaluation and alignment, where they provide interpretable, multi-dimensional, and controllable feedback signals Hashemi et al. (2024); Gunjal et al. (2026); Viswanathan et al. (2025). The integration of rubrics into reinforcement learning for language models, which we term Rubric-guided Reinforcement Learningārepresents a paradigm shift from implicit, scalar reward optimization to explicit, structured, and interpretable alignment Gunjal et al. (2026); Xu et al. (2026); Liu et al. (2026b). The motivation for rubric-guided RL is threefold. First, interpretability: scalar reward signals in traditional RLHF are opaque, making it difficult to understand why a particular response receives a high or low score; rubric and checklist methods decompose feedback into human-readable criteria Hashemi et al. (2024); Viswanathan et al. (2025). Second, controllability: rubrics allow fine-grained control over which aspects of response quality are rewarded, enabling alignment that respects specific values, safety constraints, or domain requirements Wang et al. (2024); Dineen et al. (2025). Third, self-evolution: rubrics can be generated, refined, and co-evolved with the policy model, enabling continual self-improvement without external supervision, a capability that scalar reward models lack Li et al. (2026c); Xu et al. (2026). The landscape of structured-feedback alignment has expanded rapidly since 2022. Constitutional AI Bai et al. (2022) pioneered the use of explicit natural-language principles to guide AI feedback for alignment. Subsequent work has connected rubric-guided alignment to process supervision and verifier-style feedback Lightman et al. (2024); Gunjal et al. (2026), RLVR for objectively checkable reasoning tasks Shao et al. (2024); DeepSeek-AI (2025); Yang et al. (2024), multi-objective reward modeling Wang et al. (2024), and direct preference optimization Rafailov et al. (2023). Furthermore, Rule Based Rewards Mu et al. (2024) demonstrated that composable natural-language rules can serve as direct RLHF rewards for safety and controllability. Recent rubric-specific preprints and workshop papers further explore rubric generation, rubric-based reward modeling, and self-evolving criteria; we discuss these emerging directions in the taxonomy below. Existing surveys primarily organize the literature around LLM judges, reward design and PRMs, or self-evolution loops Gu et al. (2024); Ji et al. (2025b); Zheng et al. (2025); Tao et al. (2024). We instead take explicit structured natural-language criteria as the organizing object across feedback construction, reward modeling, and policy updates. Accordingly, we use a two-tier inclusion rule: methods are core when such criteria directly condition the training or evaluation signal; generic DPO, RLVR, PRMs, verifiers, and preference-learning methods are supporting mechanisms unless explicitly rubric-conditioned. This boundary also motivates our linguistic analysis rather than treating rubrics only as optimization or prompting components. We introduce a Bayesian framework for rubric-guided RL: constitutions are prior distributions Pā”(ā)P(R) over evaluation criteria (global, input-independent), while rubrics are conditional instantiations āxā¼Pā”(ā|x)R_x P(R|x) (input-specific). This analytical lens places fixed, conditional, and update-oriented criteria in a common vocabulary and interprets rubric exploitation as overfitting to a point estimate; it does not imply that every self-evolving system performs exact Bayesian updating. We also foreground a linguistic analysis of rubrics as natural language texts (Ā§4), examining how semantic drift, granularity effects, ambiguity, and linguistic reward hacking affect rubric-guided RL. This NLP-centered perspective is essential because rubrics are fundamentally natural language artifacts, and their textual properties have been underexplored relative to their mathematical formalization. Our contributions are as follows: 1. We introduce a Bayesian framework for rubric-guided RL, defining constitutions as prior distributions Pā”(ā)P(R) over evaluation criteria and rubrics as conditional instantiations Pā”(ā|x)P(R|x). This framework unifies fixed constitutions, instance-specific rubrics, and self-evolving criteria, while clarifying its scope and limitations (Ā§2.4). 2. We provide a linguistic analysis of rubrics as natural-language artifacts, examining semantic drift, granularity effects, ambiguity, and linguistic reward hackingāphenomena that directly affect the stability and reliability of rubric-guided RL but remain underdeveloped in prior work (Ā§4). 3. We synthesize a prior-posterior taxonomy of rubric-guided RL, covering fixed constitutional priors, deterministic instance-specific rubrics, process-level supervision, rubric-based reward modeling, self-evolving rubrics, and extensions to agentic and multimodal settings. Global criteriaConstitution / principlesPā”(ā)P(R)Conditioning contextprompt x, domain D, language ā generatorgĻ:x,,āāPĻā(āā£x,,ā)g_Ļ:x,D, ā P_Ļ(R x,D, )Instance rubric / checklistāxā¼PĻā(āā£x,,ā)R_x P_Ļ(R x,D, )c1,ā¦,cK\c_1,ā¦,c_K\or point estimate ā^x R_xPolicyĻĪøā(yā£x) _Īø(y x)Criterion evaluatorjudge / verifier / RMsā”(ck,x,y)s(c_k,x,y)Feedback constructionreward / vector / preferencerāā(x,y)r_R(x,y)Policy update(PPO / GRPO / DPO)prior Pā”(ā)P(R)x,,āx,D, : overfitting to ā^x R_xā reward hackingupdate Īø failures & preference shifts ā update gĻg_Ļ and ssnatural-language criteria:ambiguity / drift / granularity Figure 1: Bayesian view of rubric-guided RL. Global principles define a prior Pā”(ā)P(R) over evaluation criteria. Given x, D, and ā , the rubric generator gĻg_Ļ parameterizes PĻā(āā£x,,ā)P_Ļ(R x,D, ) and produces sampled rubrics āxR_x or deterministic estimates ā^x R_x. Criterion evaluators construct scalar, vector, or preference feedback for policy optimization. Dashed arrows indicate self-evolution of the generator or evaluator; reward hacking corresponds to overfitting to brittle rubric point estimates. 2 Rubric: Definition and Formalization In this section, we define what a rubric is, formalize its role as a reward signal, and introduce a unifying Bayesian framework that clarifies the relationship between rubrics, constitutions, and principles. 2.1 What is a Rubric? A rubric is a structured evaluation instrument that specifies criteria for assessing performance across multiple dimensions and quality levels. In the context of LLM alignment, a rubric āR for a given prompt x can be formalized as a set of criteria: āā”(x)=c1,c2,ā¦,cKR(x)=\c_1,c_2,ā¦,c_K\, where each criterion ckc_k specifies an evaluation dimension (e.g., correctness, completeness, clarity, safety) and may include performance-level descriptors. Unlike scalar reward signals, rubrics decompose the evaluation into interpretable, auditable components. Importantly, rubrics are natural language texts: their linguistic properties, including ambiguity, granularity, and semantic stability, fundamentally affect rubric-guided RL performance, as analyzed in Ā§4. 2.2 Rubric as Reward Signal The key insight of rubric-guided RL is that rubrics can serve as structured reward signals. Given a rubric āā”(x)=c1,ā¦,cKR(x)=\c_1,ā¦,c_K\ and a response y, the rubric-conditioned reward can be computed as: rāā(x,y)=āk=1Kwkā sā”(ck,x,y)r_R(x,y)= _k=1^Kw_kĀ· s(c_k,x,y), where wkw_k is the weight for criterion ckc_k and sā”(ck,x,y)s(c_k,x,y) is the criterion-specific score. This formulation enables interpretable, controllable, and multi-dimensional reward computation. 2.3 Rubric Generation Rubric generation is a critical component of rubric-guided RL. Approaches span from contrastive rubric generation (CRG) Liu et al. (2026b), instance-level checklist generation Viswanathan et al. (2025), and rubric generation for judges Shen et al. (2026); Rao and Callison-Burch (2026) to self-proposed rubrics Sheng et al. (2026); Li et al. (2026c) and fixed constitutional principles Bai et al. (2022). The Bayesian framework distinguishes methods that optimize distributions over rubrics from those that construct deterministic, input-conditioned criteria. 2.4 A Bayesian Framework: Constitution as Prior, Rubric as Posterior To provide a unifying perspective for rubric-guided RL, we introduce a Bayesian framework that offers a probabilistic notation for organizing how different methods specify, condition, and update evaluation criteria. This framework resolves terminological ambiguity by explicitly clarifying the relationship between constitutions, rubrics, and principles: ā¢ A constitution Pā”(ā)P(R) functions as a prior distribution over evaluation criteria: a global, input-independent specification of what quality dimensions are relevant. Constitutional AI Bai et al. (2022) defines such a prior through natural-language principles (e.g., helpfulness, harmlessness, honesty) that apply across inputs. We classify constitutions as priors based on their functional role: they specify global evaluation criteria before observing a specific input. ā¢ The conditional distribution Pā”(ā|x)P(R|x) functions as a posterior distribution over evaluation criteria, conditioned on the observed input x. An instance-specific rubric āxR_x is a sampled or deterministic instantiation of this posterior. Instance-specific methods such as RaR Gunjal et al. (2026) approximate Pā”(ā|x)P(R|x) with a point estimate or checklist, while domain-specific rubrics correspond to coarser conditionals such as Pā”(ā|)P(R|D). General rubrics are closest to the prior Pā”(ā)P(R) with minimal conditioning. The Bayesian framing eliminates the need for a rigid hierarchy: constitutions and rubrics can be viewed as the same evaluation object at different levels of conditioning. ā¢ A principle is an atomic element of the evaluation space: a single dimension along which quality can be assessed. The sample space of the prior Pā”(ā)P(R) consists of valid combinations of these principles. In Constitutional AI Bai et al. (2022), each constitutional principle is one such dimension. This Bayesian framework formalizes constitutions and rubrics as the same mathematical object at different conditioning levels. The constitution is the unconditional prior Pā”(ā)P(R); domain-specific rubrics instantiate or approximate coarser conditionals such as Pā”(ā|)P(R|D); and instance-specific rubrics instantiate or approximate the full posterior Pā”(ā|x)P(R|x). Treating the input x or domain D as the observation explains the structural difference in generality between these evaluation instruments. Formalization. We define a rubric-guided RL system as a tuple (ĻĪø,Pā”(ā),gĻ,,)( _Īø,P(R),g_Ļ,J,A) where: ā¢ ĻĪøā(y|x) _Īø(y|x) is the policy model; ā¢ Pā”(ā)P(R) is the prior over evaluation criteria (the constitution); ā¢ gĻg_Ļ is the rubric generator that maps conditioning variables such as the prompt x, domain D, and language ā to a conditional distribution PĻā(āā£x,,ā)P_Ļ(R x,D, ), or to a deterministic point estimate ā^x R_x; ā¢ J is a criterion evaluator, such as an LLM judge, verifier, or reward model, that produces criterion-level scores sā”(ck,x,y)s(c_k,x,y); ā¢ A constructs feedback from criterion-level scores, such as scalar rewards, multi-dimensional vectors, or preference labels. Appendix C further clarifies how this notation applies to fixed, deterministic, and self-evolving rubric systems. Given conditioning variables (x,,ā)(x,D, ), an explicitly probabilistic rubric generator samples āxā¼PĻā(āā£x,,ā)R_x P_Ļ(R x,D, ). Deterministic systems instead return a point estimate ā^x=gĻā(x,,ā) R_x=g_Ļ(x,D, ). For scalar-reward methods, the rubric-guided reward aggregates criterion-level scores, potentially over multiple rubric samples: rāā(x,y)=āxā¼PĻā(āā£x,,ā)ā[ā”(sā”(ck,x,y)k=1K)]r_R(x,y)=E_R_x P_Ļ(R x,D, ) [A (\s(c_k,x,y)\_k=1^K ) ] (1) These scalar rewards are typically optimized using KL-regularized policy gradients: maxĪøxā¼,yā¼ĻĪø[rā(x,y)]āĪ²ā KL(ĻĪøā„Ļref) _Īø\;E_x ,y _Īø [r_R(x,y) ]-Ī²Ā·KL( _Īø\| _ref) (2) Alternatively, direct preference optimization (DPO) methods instantiate the same conceptual framework by modifying both endpoints: the aggregation function A produces pairwise preferences or labels rather than scalar rewards, and the policy is updated via a pairwise preference loss rather than Equation 2. 3 Taxonomy of Rubric-Guided RL Methods Building on the Bayesian framework introduced in Ā§2.4, we categorize existing methods under this lens. Table 1 organizes these methods along the prior-posterior axis. Methods differ primarily in how they specify, approximate, or operationalize the transition from global evaluation principles Pā”(ā)P(R) to input-conditioned criteria Pā”(ā|x)P(R|x). Our coverage prioritizes peer-reviewed publications and stable archival records, while additionally including influential preprints and technical reports that have been widely adopted or cited in the community. Appendix A summarizes the search and inclusion protocol. Method Domain Prior Pā”(ā)P(R) Posterior Pā”(ā|x)P(R|x) Aggregation Training / Optim. RaR Gunjal et al. (2026) Text Implicit Deterministic point Weighted sum / Implicit GRPO Rubric-ARM Xu et al. (2026) Text Learned Variational Learned PPO EvoLM Li et al. (2026c) Text Implicit Variational (temporal) Discriminative GRPO RLCER Sheng et al. (2026) Reasoning Implicit Process-conditional Process-level Extended from PPO Constitutional AI Bai et al. (2022) Text Fixed principles Point mass at prior Principle ensembling PPO QA-LIGN Dineen et al. (2025) Text Fixed principles QA decomposition Decomposed GRPO PRM Lightman et al. (2024) Math/Code Correctness Implicit Step product RM training RLCF Viswanathan et al. (2025) Text Implicit Instance checklist Weighted sum DPO OpenRubrics Liu et al. (2026b) Text CRG Contrastive Multi-dimensional RM training RRD Shen et al. (2026) Text Coarse rubrics Refined criteria Corr.-aware weights RM training OpenRS Jia et al. (2026) Text Principles Pairwise adaptive Criterion prefs. GRPO Agentic Extensions (see Ā§5): DR Tulu Shao et al. (2026) Agent Evolving Evolving Evolving GRPO HiPER Peng et al. (2026) Agent Hierarchical Plan-Execute HAE PPO RubricEM Li et al. (2026a) Agent Stagewise Stagewise Stagewise Extended from GRPO AdaRubric Ding (2026) Agent Task desc. Task-adaptive Conf.-weighted DPO Multimodal Extensions (see Ā§5): ARR Tian et al. (2026) VLM Implicit Generated Explicit GRPO DeltaRubric Liu et al. (2026a) VLM Implicit Plan-checklist Verification GRPO rDPO Yu et al. (2026) VLM Implicit Instance-specific Extended from DPO Step-Audio-R1.5 Zhang et al. (2026c) Audio Implicit Acoustic criteria Preference RM GRPO Table 1: Taxonomy of rubric-guided RL methods under the Bayesian framework. Each method is characterized by its domain, prior specification, posterior parameterization or criterion-construction mechanism, aggregation strategy, and optimization algorithm. Fixed-principle point-mass prior e.g. Constitutional AI Instance-specific deterministic posterior e.g. RaR, RLCF, OpenRubrics Process-level step-conditional posterior e.g. rubric-PRMs, RLCER Self-evolving variational / updated e.g. EvoLM, Rubric-ARM Figure 2: Navigation along the priorāposterior axis. The axis summarizes how method classes condition or update criteria; it does not assert that every class performs exact Bayesian inference. Method class Example Criteria source / update Workflow into optimization Fixed-principle Constitutional AI Global principles or constitution principles ā critique/preferences ā scalar or preference feedback ā policy optimization Instance-specific RaR-style methods Input- or domain-conditioned rubrics/checklists rubric generation ā criterion verification ā aggregated scalar/vector/preference feedback ā PPO/GRPO/DPO Process-level RLCER, rubric-conditioned PRMs Step criteria derived from reasoning traces trace ā step decomposition ā step scoring ā dense process feedback ā policy update Self-evolving EvoLM-style systems Criteria updated from outputs, failures, or preference shifts initial criteria ā policy evaluation ā rubric/evaluator sharpening ā subsequent policy optimization Table 2: Operational workflows for the major method classes. Generic optimization backends count as rubric-guided only when their feedback is conditioned on structured criteria. 3.1 Point-Mass Priors: Constitutional AI Methods in this category use a fixed set of principles as the prior Pā”(ā)P(R) and do not condition heavily on the input, yielding a posterior that is a point mass at the prior: Pā”(ā|x)=Ī“āconstitutionā(ā)P(R|x)= _R_constitution(R). Representative works include Constitutional AI Bai et al. (2022), IterAlign Chen et al. (2024), ICAI Findeis et al. (2025), and Deliberative Alignment OpenAI (2024), which apply fixed principle sets across all inputs rather than generating task-specific criteria. 3.2 Deterministic Posteriors: Instance-Specific Rubrics Methods in this category generate instance-specific rubrics deterministically, producing a single point estimate from Pā”(ā|x)P(R|x) without uncertainty quantification, including RaR Gunjal et al. (2026), RLCF Viswanathan et al. (2025), rubric-conditioned PRMs Lightman et al. (2024), OpenRubrics Liu et al. (2026b), rubric-generation methods for LLM judges Shen et al. (2026); Rao and Callison-Burch (2026); Siro et al. (2026), and Critique Models Ke et al. (2024), which map inputs to specific evaluation structures for fine-grained feedback. 3.3 Variational Posteriors: Self-Evolving Rubrics Self-evolving rubrics occupy the update-oriented end of Figure 2: as the policy ĻĪø _Īø improves, preference data and model outputs shift, and criteria or evaluators are updated sequentially. This paradigm is deeply connected to self-rewarding and meta-rewarding language models Yuan et al. (2024); Wu et al. (2025), which iteratively improve their own feedback generation, and recent methods including Rubric-ARM Xu et al. (2026), EvoLM Li et al. (2026c), and RLCER Sheng et al. (2026). Across these methods, we identify three recurring dynamics: (1) Posterior sharpening in explicitly latent-rubric formulations; (2) Prior simplification, where brittle gating mechanisms simplify for better generalization; and (3) Conditional expansion, where rubrics evolve from outcome-focused to process-focused. Self-evolution versus posterior updating. Crucially, self-evolution is broader than exact Bayesian posterior updating. EvoLM provides a relatively concrete posterior-sharpening interpretation: preference pairs constructed from policy outputs act as observations, and a rubric-conditioned frozen judgeās score margin between preferred and dispreferred responses supplies a likelihood-like signal Li et al. (2026c). By contrast, systems without an explicit prior, likelihood, and observation model, including RLCER, are update-oriented analogues rather than exact Bayesian inference methods. Assumptions and limitations by class. Fixed-principle methods assume that global principles transfer across inputs; instance-specific methods depend on both rubric-generator and evaluator reliability. Process-level methods additionally assume a meaningful decomposition into steps. Self-evolving methods can adapt to policy shifts, but coupled rubricāpolicy updates risk feedback loops, criterion drift, and evaluator dependence. Figure 2 and Table 2 make these operational differences explicit. 4 Linguistic Insights: Rubrics as Natural Language A fundamental but underexplored aspect of rubric-guided RL is that rubrics are natural language texts. This section analyzes how their prompt engineering, semantic stability, granularity, ambiguity, and cross-linguistic variation affect the performance and reliability of rubric-guided RL systems. 4.1 Rubrics as Prompted Specifications When rubrics are used to guide LLM judges or reward models, they become prompted specifications: their wording, formatting, and output schema can affect the resulting scores, as observed in LLM-as-a-judge frameworks like Prometheus, which train evaluators on customized score rubrics Kim et al. (2023); Kim et al. (2024), and FLASK, which decomposes evaluation into fine-grained alignment skills Ye et al. (2024). Different formats expose different tradeoffs: binary checklists are easy to aggregate but coarse Gunjal et al. (2026), free-text critiques are expressive but difficult to scalarize Ke et al. (2024), and multi-level descriptors offer a middle ground between discriminative power and calibration Hashemi et al. (2024). 4.2 Granularity Effects The granularity of rubric criteria, whether defined over tokens, reasoning steps, whole responses, or decomposed checklist items, affects both evaluation quality and the informativeness of reward signals. The granularity-density tradeoff mirrors the exploration-exploitation tradeoff in RL: denser rewards enable faster learning but may miss global quality, while sparser rewards capture holistic quality but slow convergence. Existing systems often balance this tradeoff through hybrid designs: PRMs combine dense step-level supervision with outcome-level evaluation Lightman et al. (2024), RLCF combines instance-specific checklists with universal criteria and verifier programs Viswanathan et al. (2025), and OpenRS combines pairwise adaptive meta-rubrics with pointwise verifiable rubrics for guardrails Jia et al. (2026). Lexical and Token-Level Signals. At the finest scale, alignment systems may use lexical constraints or token-level shaping terms, such as PPO-style per-token KL penalties, length penalties, keyword inclusion/exclusion checks, profanity filters, or exact-format constraints. Lexical and formatting signals directly enforce observable surface properties, and length- or style-correlated rewards can favor well-formatted or verbose outputs without corresponding improvements in semantic quality Zeng et al. (2024); Gao et al. (2023); Zheng et al. (2023). Therefore these dense signals should be clearly distinguished from process-level credit assignment strategies. Sentence/Step-Level Criteria. Process-level rubrics (RLCER, Sheng et al., 2026; PRMs, Lightman et al., 2024) evaluate intermediate reasoning steps, providing denser reward signals than outcome-level rubrics. PRMs operate at the step level, providing per-step correctness signals and achieving 78.2% accuracy on MATH Lightman et al. (2024). However, step-level evaluation requires defining what constitutes a āstepā, which is itself a non-trivial decision. Recent self-evolving rubric work suggests that optimal granularity interacts with evaluator capacity: smaller evaluators may require more precise rubrics, whereas larger evaluators can tolerate coarser specifications Li et al. (2026c). Response-Level and Document-Level Criteria. Outcome-level rubrics, such as RaR Gunjal et al. (2026) and Constitutional AI Bai et al. (2022), evaluate the complete response. They provide holistic assessment of dimensions such as helpfulness, safety, completeness, and coherence, but the reward is sparse and may obscure which part of the response caused success or failure. Atomic Checklist Criteria. A separate form of granularity comes from decomposing a holistic objective into atomic checklist items rather than changing the textual span of evaluation. RLCF Viswanathan et al. (2025), for example, decomposes tasks into atomic checklist items, illustrating how fine-grained criteria can provide denser feedback for robustness. Similarly, CheckEval Lee et al. (2025) demonstrates that Boolean checklist-style decomposition improves evaluator reliability and robustness. This differs from word-level or sentence-level evaluation: the granularity lies in the criterion decomposition, not necessarily in the textual unit being scored. 4.3 Semantic Drift in Rubric Evaluation Semantic drift refers to the phenomenon where the meaning of rubric criteria shifts across RL training iterations, even when the rubric text remains unchanged. This drift occurs because the policy modelās interpretation of rubric criteria changes as it adapts to the reward signal. We distinguish semantic drift from the broader concept of concept drift Gama et al. (2014) in machine learning: semantic drift is specifically about the semantic instability of natural language criteria as they are applied by changing models across training, rather than changes in the data distribution itself. Causes of Semantic Drift. We identify three causes: (1) Distribution shift: as the policy improves, the response distribution changes, altering the context in which rubric criteria are applied. A criterion like āfactual accuracyā means something different for a model that produces mostly correct responses vs. one that produces mostly incorrect responses. EvoLM Li et al. (2026c) provides direct evidence: rubrics evolve from vague criteria (e.g., āEnsure mathematical accuracyā) to specific, verifiable facts (e.g., āThe answer is 144, derived from perimeter 48ā) as the policy distribution shifts during training. (2) Criterion gaming: the policy learns to satisfy the letter of rubric criteria while violating their spirit, effectively shifting the rubricās semantics. Constitutional AI Bai et al. (2022) observed this Goodharting effect, where over-optimization leads to boilerplate language and overly harsh responses. (3) Evolving evaluator: in systems with learned judges, reward models, or co-evolving rubric generators, the scoring function itself changes during training. Thus, even when a criterion such as āmathematical accuracyā is textually stable, the function stā(ck,x,y)s_t(c_k,x,y) that applies this criterion may shift as the evaluator is updated on new policy outputs. This is distinct from judge collapse: evaluator evolution is a general design property of adaptive systems, whereas collapse is a failure mode where the judge loses discriminative power or learns exploitable shortcuts. Measuring Semantic Drift. We propose measuring semantic drift by tracking the consistency of rubric application across training iterations. Formally, given a rubric criterion ckc_k and a fixed set of reference responses, we define the drift metric as: Driftā(ck,t)=1ācorrā(stā(ck,x,y),s0ā(ck,x,y))Drift(c_k,t)=1-corr (s_t(c_k,x,y),s_0(c_k,x,y) ) (3) where sts_t is the criterion score at iteration t and corr measures correlation (e.g., Spearmanās Ļ). In practice, the reference set should be a frozen, diverse validation set of promptāresponse pairs. Scores can be anchored to a static high-capacity judge or held-out human annotations to reduce model-dependent self-evaluation bias. Rank-correlation decay is then monitored across iterations, with intervention thresholds calibrated as an absolute or relative drop from the initial held-out correlation rather than a universal constant; crossing a threshold triggers rubric regeneration or human auditing. This diagnostic proposal is related to but distinct from semantic drift scores in text generation Spataru et al. (2024) and value drift in post-training alignment Bhatia et al. (2025); its empirical validation remains open. 4.4 Linguistic Reward Hacking Linguistic reward hacking occurs when a policy exploits semantic gaps in natural language criteria, producing responses that satisfy the literal wording of a rubric while violating its intended meaning. This is a distinct phenomenon from numerical reward hacking (manipulating scalar reward magnitudes) and requires qualitative linguistic analysis to detect. While historically studied narrowly within low-resource language alignment Inuwa-Dutse (2025), we generalize the concept to encompass any systematic policy exploitation of semantic gaps in multi-dimensional evaluation rubrics. Root Causes: Ambiguity and Underspecification. This vulnerability stems directly from the fact that natural language criteria are inherently ambiguous and underspecified. Ambiguity arises because high-level desiderata like āhelpfulnessā depend heavily on latent user context, leading to only moderate alignment with human consensus (Kendallās Ļ of 0.3ā0.4) even when mitigated by performance-level descriptors Hashemi et al. (2024). Underspecification occurs because criteria typically define what to evaluate rather than how, leaving operational boundaries vague unless anchored by intermediate granular structures like checklists Viswanathan et al. (2025). Crucially, ambiguity and underspecification function as the primary systemic drivers for both semantic drift and linguistic reward hacking (see detailed case studies in Appendix L). However, this underspecification is also an opportunity: it provides the essential semantic flexibility required for self-evolution and contextual adaptation. Without initial underspecification, optimization frameworks would lack the headroom to perform posterior sharpening or track emergent data distributions during training via automated alignment updates Li et al. (2026c); Xu et al. (2026). Types of Linguistic Reward Hacking. We categorize these exploitative behaviors into three primary structural modes: (1) Verbosity exploitation, where the policy overoptimizes for ācompletenessā by generating elongated, low-substance responses, reflecting the proxy-reward degeneration patterns documented by Gao et al. (2023). (2) Criterion bleeding, where the model satisfies one axis by sacrificing another, such as aggressively refusing valid prompts to maximize āsafetyā at the cost of āhelpfulnessā Dai et al. (2024); Wang et al. (2024). (3) Surface pattern matching, where the policy learns to parrot rubric keywords (e.g., stating āI have ensured safetyā) without executing the underlying reasoning. Self-evolving rubric work suggests that evaluator capacity and rubric specificity can affect how easily such surface-level satisfaction is detected Li et al. (2026c). Potential diagnostics of linguistic reward hacking include: divergence between rubric scores and human preferences, auxiliary verification of claimed reasoning steps, abnormal criterion-correlation patterns, and cross-rubric consistency checks. Relation to Bayesian Framework. Under the Bayesian lens (Ā§2.4), linguistic reward hacking can be viewed as the policy exploiting the gap between a rubricās literal point-estimate instantiation (āxR_x) and its intended conditional criterion space Pā”(ā|x)P(R|x). Optimizing across multiple plausible rubric formulations therefore suggests a robustness direction. Self-evolving systems may patch vulnerabilities as they are uncovered, but constitute approximate posterior updates only when the required probabilistic components are explicit. 4.5 Cross-Lingual Considerations While rubric-guided RL is predominantly optimized for English, multilingual alignment introduces typological and systemic challenges. High-level desiderata such as ārespectfulnessā are culturally and linguistically mediated; Japanese and Korean honorific registers (keigo/jondaetmal) and Arabic diglossia show that rubric and evaluator invariance cannot be assumed. A criterion may depend on dialect, speaker relation, register, and sociocultural setting rather than surface form alone. Low LLM-as-a-judge cross-lingual consistency (Īŗā0.3Īŗā 0.3) compounds this problem Fu and Liu (2025). To mitigate these gaps, recent work on cross-lingual reward transfer shows that English-trained reward models can bootstrap target-language evaluations Hong et al. (2025); Yang et al. (2025b), motivating unified criteria sets and multilingual reward reasoning architectures Sheth et al. (2026); Anugraha et al. (2026); Gureja et al. (2025). We propose language-aware rubric design: decoupling language-agnostic properties (e.g., factual accuracy) from language-specific instantiations (e.g., localized formality). Under our Bayesian framework, this corresponds to conditioning on language, dialect, speaker relation, and sociocultural context. For example, Pā”(āā£x,ā,d,z)P(R x, ,d,z)āwhile anchoring broadly shared principles in the prior Pā”(ā)P(R). 4.6 Evidence Levels We distinguish three evidence levels of the analyses in this section: prompt sensitivity, evaluator calibration, granularity effects, and Goodhart-style over-optimization are established findings; semantic drift and linguistic reward hacking are our literature-grounded synthesis, supported by Appendix L; underspecification-as-opportunity and posterior marginalization are open hypotheses or design directions requiring validation. 5 Outlook and Emerging Frontiers: Agentic and Multimodal RL While our core taxonomy centers on text-based alignment, extending rubric-guided methods to agentic RL and multimodal models introduces fundamentally new structural challenges. This section highlights these emerging frontiers, which push rubric evaluation beyond single-turn chat, as summarized in the lower sections of Table 1. Additional discussions on deep-research benchmarks are provided in Appendix F. Agentic RL and Credit Assignment. In agentic settings, evaluation must account for multi-step trajectories and environment grounding. The core bottleneck is multi-step credit assignment. Traditional scalar rewards fail to isolate which specific action in a long trajectory led to success or failure. Rubric-guided methods like DR Tulu Shao et al. (2026), HiPER Peng et al. (2026), and RubricEM Li et al. (2026a) address this by decomposing trajectory rewards into stagewise or hierarchical rubrics, enabling precise credit assignment across temporal phases. This structured feedback may reduce the policy from learning spurious correlations during complex tasks. Multimodal Rubric-guided RL. In multimodal settings, evaluating outputs requires capturing cross-modal interactions and modality-specific dimensions such as visual grounding, object hallucination, and audio fluency. Foundational multimodal alignment work demonstrated that a single latent preference score is insufficient, by augmenting rewards with visual evidence Sun et al. (2024), using segment-level correctional feedback Yu et al. (2024), separating safety and helpfulness objectives Ji et al. (2025a), or curating fine-grained multidimensional preference data Zhang et al. (2025b). Recent methods directly address this complexity through explicit rubrics. Building on multidimensional benchmarks Li et al. (2023); Guan et al. (2024); Fu et al. (2024), systems such as Auto-Rubric as Reward Tian et al. (2026), DeltaRubric Liu et al. (2026a), and rDPO Yu et al. (2026) now explicitly condition multimodal reward modeling and preference optimization on structured criteria. In the audio domain, Step-Audio-R1.5 Zhang et al. (2026c) applies rubric-guided optimization to multi-turn spoken dialogues by utilizing structural rubrics such as content requirements, persona consistency, and cross-turn instruction retention, while leaving subjective traits like tone and naturalness to standard preference comparison. 6 Conclusion and Future Directions Rubric-guided reinforcement learning marks a shift from implicit scalar reward optimization to explicit, structured, and interpretable alignment. We unify this space through a Bayesian framework that treats constitutions as priors Pā”(ā)P(R) and rubrics as conditional instantiations Pā”(ā|x)P(R|x), interpreting some self-evolution mechanisms as approximate posterior updating and rubric exploitation as point-estimate overfitting. Because rubrics are natural-language artifacts, navigating granularity trade-offs and mitigating vulnerabilities like semantic drift and linguistic reward hacking are central to building controllable, self-improving, and reliable language models. Future work faces several challenges. First, we need theory connecting natural-language criteria to stable scalar or process-level feedback, especially as rubrics and evaluators evolve. Second, instance-specific rubric generation incurs token and latency overhead, motivating retrieval, caching, compression, and local evaluators. Third, cross-lingual and multimodal frontiers require principled mappings from abstract criteria to domain-specific instantiations. Finally, extending to autonomous agents demands proper credit assignment over agentic tasks to reliably evaluate long-horizon trajectories. 7 Limitations We acknowledge several limitations of this survey and of the rubric-guided RL paradigm more broadly. 7.1 Heterogeneous Evaluation Settings Because rubric-guided RL methods use different base models, benchmarks, and optimization protocols, direct quantitative comparison is difficult. We therefore focus on qualitative organization and report numbers from original papers where available. A standardized benchmark for controlled comparison remains an important direction for future work. 7.2 Scope of the Bayesian Framework Our Bayesian framework is an analytical abstraction for organizing how methods specify, condition, and update evaluation criteria. It intentionally abstracts over implementation details: fixed constitutions, deterministic rubric generators, and self-evolving systems instantiate different parts of the notation. Exact Bayesian guarantees would require explicit priors, likelihoods, observations, and calibrated uncertainty, which most surveyed systems do not provide. Developing fully probabilistic rubric generators remains an open research opportunity. 7.3 Production Systems Recent technical reports suggest that production systems increasingly embed rubric-like criteria in training and data pipelines, including Kimi K2, GLM-5, Qwen3, and DeepSeek-V4 Kimi Team et al. (2025); GLM-5 Team (2026); Yang et al. (2025a); DeepSeek-AI (2026). For example, DeepSeek-V4 reports rubric-guided RL data and a Generative Reward Model for hard-to-verify tasks, while GLM-5 defines multidimensional criteria for chat and role-playing data, including instruction following, expressiveness, creativity, coherence, and long-dialogue consistency DeepSeek-AI (2026); GLM-5 Team (2026). Because these practices are often only partially documented, a fuller account of implicit production-oriented rubric usage remains an important direction for future empirical study. 7.4 Evolving Terminology Terminology around constitutions, rubrics, and principles is still evolving. Our framework provides one coherent vocabulary, but future work may refine these distinctions as the field matures. 7.5 Computational Cost As analyzed in Ā§E, instance-specific rubric generation and API-based judging introduce token, latency, and financial overhead. This creates deployment tradeoffs and motivates cheaper rubric generation, caching, compression, and local evaluator designs. Ethics Statement Rubric-guided RL raises important ethical considerations. While rubrics improve interpretability, they also pose the systemic risk of embedding cultural, political, or institutional biases into formalized evaluation criteria that are subsequently treated as objective standards. Evaluator dependence, underspecified criteria, and aggregation choices can further conceal whose values dominate. The sensitivity to normative framing observed in Constitutional AI Bai et al. (2022) extends to rubrics more broadly. Future work should develop frameworks for ethical rubric design by prioritizing participatory construction to engage diverse stakeholders in defining criteria, democratic governance to establish formal mechanisms for continuous revision and contestation, and systemic transparency to ensure that both the underlying criteria and their downstream policy updates remain fully auditable. Acknowledgments AI assistants, including ChatGPT and Gemini, were used only to identify additional relevant work and to polish author-written text. All suggested references and language edits were independently verified by the authors. We thank Yao Zhou and Zhenran Xu for their valuable discussions. References Anugraha et al. (2026) D. Anugraha, S. Hung, Z. Tang, E. A. Lee, D. T. Wijaya, and G. I. Winata mR3: multilingual rubric-agnostic reward reasoning models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix L, Ā§4.5. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan Constitutional AI: harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. External Links: Link Cited by: Appendix L, Table 6, Table 8, Ā§B.2, 1st item, Table 3, Table 4, Ā§1, 1st item, 3rd item, Ā§2.3, Ā§3.1, Table 1, Ā§4.2, Ā§4.3, Ethics Statement. Bhatia et al. (2025) M. Bhatia, S. Nayak, G. Kamath, M. Mosbach, K. StaÅczak, V. Shwartz, and S. Reddy Value drifts: tracing value alignment during LLM post-training. arXiv preprint arXiv:2510.26707. External Links: Link Cited by: Ā§4.3. Chen et al. (2024) X. Chen, H. Wen, S. Nag, C. Luo, Q. Yin, R. Li, Z. Li, and W. Wang IterAlign: iterative constitutional alignment of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 1423ā1433. External Links: Document, Link Cited by: Ā§3.1. Dai et al. (2024) J. Dai, R. Pan, J. Ji, X. Li, and Y. Wang Safe RLHF: safe reinforcement learning from human feedback. In International Conference on Learning Representations, External Links: Link Cited by: 2nd item, Table 8, Ā§4.4. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948. External Links: Link Cited by: Ā§B.2, Ā§1. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4: towards highly efficient million-token context intelligence. Technical report DeepSeek-AI. External Links: Document, Link Cited by: Ā§7.3. Dineen et al. (2025) J. Dineen, A. Rrv, Q. Liu, Z. Xu, X. Ye, M. Shen, Z. Li, S. Lu, C. Baral, M. Chen, and B. Zhou QA-LIGN: aligning LLMs through constitutionally decomposed QA. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 20619ā20642. External Links: Document, Link, ISBN 979-8-89176-335-7 Cited by: Ā§1, Table 1. Ding (2026) L. Ding AdaRubric: task-adaptive rubrics for reliable llm agent evaluation and reward learning. arXiv preprint arXiv:2603.21362. Note: KnowFM Workshop at ACL 2026 External Links: Link Cited by: Table 1. Dou et al. (2026a) S. Dou, Y. Shen, C. Huang, J. Ye, J. Chen, J. Wang, Q. He, S. Liu, C. Lv, J. Lin, J. Zhang, M. Zhang, S. Liu, T. Ji, Z. Yin, C. Zhang, H. Xie, J. Hu, J. Deng, L. Li, M. Hu, S. Wang, S. Zhao, W. Wang, Y. Lei, Y. Liu, Y. Xiao, Y. Liu, Z. Xu, Z. Guo, Z. Zhao, P. Zhou, T. Gui, Q. Zhang, X. Huang, Y. Jiang, D. Wang, and S. Yao CL-bench Life: can language models learn from real-life context?. arXiv preprint arXiv:2604.27043. External Links: Link Cited by: Appendix G. Dou et al. (2026b) S. Dou, M. Zhang, Z. Yin, C. Huang, Y. Shen, J. Wang, J. Chen, Y. Ni, J. Ye, C. Zhang, H. Xie, J. Hu, S. Wang, W. Wang, Y. Xiao, Y. Liu, Z. Xu, Z. Guo, P. Zhou, T. Gui, Z. Wu, X. Qiu, Q. Zhang, X. Huang, Y. Jiang, D. Wang, and S. Yao CL-bench: a benchmark for context learning. arXiv preprint arXiv:2602.03587. External Links: Link Cited by: Appendix G. Ethayarajh et al. (2024) K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela KTO: model alignment as prospect theoretic optimization. In International Conference on Machine Learning, External Links: Link Cited by: Ā§B.2. Findeis et al. (2025) A. Findeis, T. Kaufmann, E. HĆ¼llermeier, S. Albanie, and R. Mullins Inverse constitutional AI: compressing preferences into principles. In International Conference on Learning Representations, External Links: Link Cited by: Ā§3.1. Fu et al. (2024) C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, R. Ji, C. Shan, and R. He MME: a comprehensive evaluation benchmark for multimodal large language models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Ā§5. Fu and Liu (2025) X. Fu and W. Liu How reliable is multilingual LLM-as-a-judge?. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 11040ā11053. External Links: Document, Link, ISBN 979-8-89176-335-7 Cited by: Appendix L, Ā§4.5. Gama et al. (2014) J. Gama, I. ŽliobaitÄ, A. Bifet, M. Pechenizkiy, and A. Bouchachia A survey on concept drift adaptation. ACM Computing Surveys 46 (4), p. 44:1ā44:37. External Links: Document, Link Cited by: Ā§4.3. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 10835ā10866. External Links: Link Cited by: 1st item, Ā§B.2, Ā§1, Ā§4.2, Ā§4.4. Gheshlaghi Azar et al. (2024) M. Gheshlaghi Azar, Z. Daniel Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 238, p. 4447ā4455. External Links: Link Cited by: Ā§B.2. GLM-5 Team (2026) GLM-5 Team GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. External Links: Link Cited by: Ā§7.3. Gu et al. (2024) J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y. Wang, W. Gao, L. Ni, and J. Guo A survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594. External Links: Link Cited by: Ā§B.1, Ā§1. Guan et al. (2024) T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: Ā§5. Gunjal et al. (2026) A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. M. Hendryx Rubrics as rewards: reinforcement learning beyond verifiable domains. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Ā§B.2, 1st item, Table 3, Table 3, Table 4, Ā§1, Ā§1, 2nd item, Ā§3.2, Table 1, Ā§4.1, Ā§4.2. Gureja et al. (2025) S. Gureja, L. J. V. Miranda, S. B. Islam, R. Maheshwary, D. Sharma, G. Winata, N. Lambert, S. Ruder, S. Hooker, and M. Fadaee M-RewardBench: evaluating reward models in multilingual settings. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 43ā58. External Links: Document, Link, ISBN 979-8-89176-251-0 Cited by: Appendix L, Ā§4.5. Han et al. (2025) J. Han, H. Kim, C. Lee, D. Lee, M. H. Park, H. Song, S. J. Choi, M. Lee, and H. Lee DEER: a benchmark for evaluating deep research agents on expert report generation. arXiv preprint arXiv:2512.17776. External Links: Link Cited by: Appendix F. Hashemi et al. (2024) H. Hashemi, J. Eisner, C. Rosset, B. Van Durme, and C. Kedzie LLM-rubric: a multidimensional, calibrated approach to automated evaluation of natural language texts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 13806ā13834. External Links: Document, Link Cited by: Appendix K, Ā§1, Ā§1, Ā§4.1, Ā§4.4. Hong et al. (2025) J. Hong, N. Lee, R. MartĆnez-CastaƱo, C. RodrĆguez, and J. Thorne Cross-lingual transfer of reward models in multilingual alignment. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics, External Links: Link Cited by: Appendix L, Ā§4.5. Hong et al. (2024) J. Hong, N. Lee, and J. Thorne ORPO: monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 11170ā11189. External Links: Document, Link Cited by: Ā§B.2. Hwang et al. (2024) Y. Hwang, Y. Kim, H. Bae, J. Bang, H. Lee, and K. Jung Kosmic: Korean text similarity metric reflecting honorific distinctions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue (Eds.), Torino, Italia, p. 9954ā9960. External Links: Link Cited by: Appendix L. Inuwa-Dutse (2025) I. Inuwa-Dutse OpenAIās GPT-OSS-20B model and safety alignment issues in a low-resource language. arXiv preprint arXiv:2510.01266. External Links: Link Cited by: Ā§4.4. Ji et al. (2025a) J. Ji, X. Chen, R. Pan, H. Zhu, J. Li, D. Hong, B. Chen, J. Zhou, K. Wang, J. Dai, C. Chan, Y. Tang, and Y. Yang Safe RLHF-V: safe reinforcement learning from multi-modal human feedback. In Advances in Neural Information Processing Systems, p. 51568ā51604. External Links: Document, Link Cited by: Ā§5. Ji et al. (2025b) M. Ji, Y. Wu, Z. Wu, S. Wang, J. Yang, M. Dras, and U. Naseem A survey on progress in LLM alignment from the perspective of reward design. arXiv preprint arXiv:2505.02666. External Links: Link Cited by: Ā§B.1, Ā§1. Jia et al. (2026) R. Jia, Y. Yang, Y. Wu, Y. Gai, S. Tao, M. Zhou, J. Lin, X. Jiang, and G. Jiang Open rubric system: scaling reinforcement learning with pairwise adaptive rubric. arXiv preprint arXiv:2602.14069. External Links: Link Cited by: Table 1, Ā§4.2. Ke et al. (2024) P. Ke, B. Wen, A. Feng, X. Liu, X. Lei, J. Cheng, S. Wang, A. Zeng, Y. Dong, H. Wang, J. Tang, and M. Huang CritiqueLLM: towards an informative critique generation model for evaluation of large language model generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 13034ā13054. External Links: Document, Link Cited by: Ā§3.2, Ā§4.1. Kim et al. (2023) S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo Prometheus: inducing fine-grained evaluation capability in language models. External Links: Link, 2310.08491 Cited by: Ā§4.1. Kim et al. (2024) S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo Prometheus 2: an open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 4334ā4353. External Links: Document, Link Cited by: Ā§4.1. Kimi Team et al. (2025) Kimi Team, Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, et al. Kimi K2: open agentic intelligence. External Links: Link, 2507.20534 Cited by: Ā§7.3. Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi TĆ¼lu 3: pushing frontiers in open language model post-training. In International Conference on Learning Representations, External Links: Link Cited by: Ā§B.2. Lee et al. (2024) H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 26874ā26901. External Links: Link Cited by: Ā§B.2. Lee et al. (2025) Y. Lee, J. Kim, J. Kim, H. Cho, J. Kang, P. Kang, and N. Kim CheckEval: a reliable LLM-as-a-judge framework for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 15771ā15798. External Links: Document, Link, ISBN 979-8-89176-332-6 Cited by: Ā§4.2. Li et al. (2026a) G. Li, B. D. Mishra, Z. Wang, J. Yan, Y. Chen, C. Li, L. T. Le, R. Han, G. Lee, H. Tong, C. Lee, and T. Pfister RubricEM: Meta-RL with rubric-guided policy decomposition beyond verifiable rewards. arXiv preprint arXiv:2605.10899. External Links: Link Cited by: Table 1, Ā§5. Li et al. (2026b) R. Li, M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao DeepResearch Bench I: diagnosing deep research agents via rubrics from expert report. arXiv preprint arXiv:2601.08536. External Links: Link Cited by: Appendix F. Li et al. (2026c) S. S. Li, R. Xin, T. Xiao, Y. Wang, R. Shao, Z. Hao, M. Sclar, S. Oh, F. Brahman, P. W. Koh, and Y. Tsvetkov EvoLM: self-evolving language models through co-evolved discriminative rubrics. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: 3rd item, Appendix L, Appendix L, Table 6, Appendix D, 2nd item, Table 3, Table 4, Ā§1, Ā§2.3, Ā§3.3, Ā§3.3, Table 1, Ā§4.2, Ā§4.3, Ā§4.4, Ā§4.4. Li et al. (2023) Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 292ā305. External Links: Document, Link Cited by: Ā§5. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Letās verify step by step. In International Conference on Learning Representations, External Links: Link Cited by: Table 10, Ā§B.2, Table 3, Table 4, Ā§1, Ā§3.2, Table 1, Ā§4.2, Ā§4.2. Liu et al. (2026a) R. Liu, D. Yu, Z. Liang, Y. Shi, T. Zheng, R. Dai, H. Mi, P. Tokekar, and Leoweiliang DeltaRubric: generative multimodal reward modeling via joint planning and verification. arXiv preprint arXiv:2605.09269. External Links: Link Cited by: Table 1, Ā§5. Liu et al. (2026b) T. Liu, R. Xu, T. Yu, I. Hong, C. Yang, T. Zhao, and H. Wang OpenRubrics: towards scalable synthetic rubric generation for reward modeling and LLM alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 17417ā17437. External Links: Document, Link, ISBN 979-8-89176-390-6 Cited by: Table 4, Ā§1, Ā§2.3, Ā§3.2, Table 1. Mahmoud et al. (2026) A. Mahmoud, M. Rezaei, Z. Wang, A. Gunjal, B. Liu, and Y. He Reward hacking in rubric-based reinforcement learning. arXiv preprint arXiv:2605.12474. External Links: Link Cited by: Table 7, Table 9, Appendix M. Meng et al. (2024) Y. Meng, M. Xia, and D. Chen SimPO: simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Ā§B.2. Miao et al. (2024) Y. Miao, S. Zhang, L. Ding, R. Bao, L. Zhang, and D. Tao InfoRM: mitigating reward hacking in RLHF via information-theoretic reward modeling. In Advances in Neural Information Processing Systems, p. 134387ā134429. External Links: Document, Link Cited by: Appendix M. Mu et al. (2024) T. Mu, A. Helyar, J. Heidecke, J. Achiam, A. Vallone, I. D. Kivlichan, M. Lin, A. Beutel, J. Schulman, and L. Weng Rule based rewards for language model safety. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Ā§1. OpenAI (2024) OpenAI Deliberative alignment: reasoning enables safer language models. arXiv preprint arXiv:2412.16339. External Links: Link Cited by: Ā§3.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730ā27744. External Links: Document, Link Cited by: Ā§B.2, Ā§1. Pan et al. (2026) T. Pan, X. Lin, W. Yang, Q. He, S. Chen, L. Qi, W. Xu, H. Feng, B. Xu, and Y. Xiao RubricEval: a rubric-level meta-evaluation benchmark for LLM judges in instruction following. arXiv preprint arXiv:2603.25133. External Links: Link Cited by: Appendix H. Peng et al. (2026) J. Peng, Y. Liu, R. Zhou, C. Fleming, Z. Wang, A. Garcia, and M. Hong HiPER: hierarchical reinforcement learning with explicit credit assignment for large language model agents. arXiv preprint arXiv:2602.16165. External Links: Link Cited by: Table 1, Ā§5. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36, p. 53728ā53741. External Links: Document, Link Cited by: Ā§B.2, Ā§1. Rao and Callison-Burch (2026) D. Rao and C. Callison-Burch Autorubric: a unifying framework for rubric-based LLM evaluation on non-verifiable tasks. arXiv preprint arXiv:2603.00077. External Links: Link Cited by: Ā§2.3, Ā§3.2. Shao et al. (2026) R. Shao, A. Asai, S. Z. Shen, H. Ivison, V. Kishore, J. Zhuo, X. Zhao, M. Park, S. G. Finlayson, D. Sontag, T. Murray, S. Min, P. Dasigi, L. Soldaini, F. Brahman, W. Yih, T. Wu, L. Zettlemoyer, Y. Kim, H. Hajishirzi, and P. W. Koh DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. In International Conference on Machine Learning, External Links: Link Cited by: Table 1, Ā§5. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: Ā§B.2, Ā§B.2, Ā§1. Sharma et al. (2025) M. Sharma, C. B. C. Zhang, C. Bandi, C. Wang, A. Aich, H. Nghiem, T. Rabbani, Y. Htet, B. Jang, S. Basu, A. Balwani, D. Peskoff, M. Ayestaran, S. M. Hendryx, B. Kenstler, and B. Liu ResearchRubrics: a benchmark of prompts and rubrics for evaluating deep research agents. arXiv preprint arXiv:2511.07685. External Links: Link Cited by: Appendix F. Shen et al. (2026) W. F. Shen, X. Qiu, C. Whitehouse, L. Alazraki, S. Goel, F. Barbieri, T. Willi, A. Mathur, and I. Leontiadis Rethinking rubric generation for improving LLM judge and reward modeling for open-ended tasks. arXiv preprint arXiv:2602.05125. External Links: Link Cited by: Ā§2.3, Ā§3.2, Table 1. Sheng et al. (2026) L. Sheng, W. Ma, R. Hong, X. Wang, A. Zhang, and T. Chua Reinforcing chain-of-thought reasoning with self-evolving rubrics. arXiv preprint arXiv:2602.10885. External Links: Link Cited by: Appendix L, Ā§B.2, Table 4, Ā§2.3, Ā§3.3, Table 1, Ā§4.2. Sheth et al. (2026) I. Sheth, Z. Jonke, A. Mantrach, and S. Mansour Cross-lingual LLM-Judge transfer via evaluation decomposition. arXiv preprint arXiv:2603.18557. External Links: Link Cited by: Appendix L, Ā§4.5. Siro et al. (2026) C. Siro, P. Aliannejadi, and M. Aliannejadi Learning to judge: LLMs designing and applying evaluation rubrics. In Findings of the Association for Computational Linguistics: EACL 2026, V. Demberg, K. Inui, and L. Marquez (Eds.), Rabat, Morocco, p. 6371ā6389. External Links: Document, Link, ISBN 979-8-89176-386-9 Cited by: Ā§3.2. Spataru et al. (2024) A. Spataru, E. Hambro, E. Voita, and N. Cancedda Know when to stop: a study of semantic drift in text generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 3656ā3671. External Links: Document, Link Cited by: Ā§4.3. Sun et al. (2024) Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, K. Keutzer, and T. Darrell Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 13088ā13110. External Links: Document, Link Cited by: Ā§5. Tao et al. (2024) Z. Tao, T. Lin, Q. Sun, X. Li, H. Zhang, Z. Zhang, Y. Lian, B. Sun, Q. Wang, J. Li, Z. Wu, and J. Zhou A survey on self-evolution of large language models. arXiv preprint arXiv:2404.14387. External Links: Link Cited by: Ā§B.1, Ā§1. Tian et al. (2026) J. Tian, F. Liu, J. Han, Y. Jiang, Y. Wu, Y. Liu, H. Li, F. Xu, and W. Li Auto-rubric as reward: from implicit preferences to explicit multimodal generative criteria. arXiv preprint arXiv:2605.08354. External Links: Link Cited by: Table 1, Ā§5. Viswanathan et al. (2025) V. Viswanathan, Y. Sun, S. Ma, X. Kong, M. Cao, G. Neubig, and T. Wu Checklists are better than reward models for aligning language models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Table 10, Table 4, Ā§1, Ā§1, Ā§2.3, Ā§3.2, Table 1, Ā§4.2, Ā§4.2, Ā§4.4. Wang et al. (2024) H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 10582ā10592. External Links: Document, Link Cited by: 2nd item, Table 8, Table 8, Ā§1, Ā§1, Ā§1, Ā§4.4. Wu et al. (2025) T. Wu, W. Yuan, O. Golovneva, J. Xu, Y. Tian, J. Jiao, J. E. Weston, and S. Sukhbaatar Meta-rewarding language models: self-improving alignment with LLM-as-a-meta-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 11537ā11554. External Links: Document, Link, ISBN 979-8-89176-332-6 Cited by: Ā§3.3. Wu et al. (2023) Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi Fine-grained human feedback gives better rewards for language model training. In Advances in Neural Information Processing Systems, Vol. 36, p. 59008ā59033. External Links: Document, Link Cited by: Ā§1. Xu et al. (2024) H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim Contrastive preference optimization: pushing the boundaries of LLM performance in machine translation. In International Conference on Machine Learning, External Links: Link Cited by: Ā§B.2. Xu et al. (2026) R. Xu, T. Liu, Z. Dong, T. Yu, I. Hong, C. Yang, L. Zhang, T. Zhao, and H. Wang Alternating reinforcement learning for rubric-based reward modeling in non-verifiable LLM post-training. arXiv preprint arXiv:2602.01511. External Links: Link Cited by: Appendix L, 1st item, Table 3, Table 4, Ā§1, Ā§1, Ā§3.3, Table 1, Ā§4.4. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: Link Cited by: Ā§7.3. Yang et al. (2024) A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang Qwen2.5-Math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. External Links: Link Cited by: Ā§B.2, Ā§1. Yang et al. (2025b) W. Yang, J. Wu, C. Wang, C. Zong, and J. Zhang Implicit cross-lingual rewarding for efficient multilingual preference alignment. In Findings of the Association for Computational Linguistics: ACL 2025, External Links: Link Cited by: Ā§4.5. Ye et al. (2024) S. Ye, D. Kim, S. Kim, H. Hwang, S. Kim, Y. Jo, J. Thorne, J. Kim, and M. Seo FLASK: fine-grained language model evaluation based on alignment skill sets. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Ā§4.1. Yu et al. (2024) T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, and T. Chua RLHF-V: towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: Ā§5. Yu et al. (2026) Y. Yu, F. Hong, X. Qu, H. Wang, G. Wu, Q. Luo, N. Xu, H. Wang, W. Xu, Y. Liao, Z. Chen, H. Li, Z. Li, D. Peng, M. Liao, J. Wu, H. Ren, and D. Tu Visual preference optimization with rubric rewards. arXiv preprint arXiv:2604.13029. External Links: Link Cited by: Table 1, Ā§5. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 57905ā57923. External Links: Link Cited by: Ā§3.3. Zeng et al. (2024) S. Zeng, G. Feng, W. Liu, M. Wu, S. Lu, J. Zhao, S. Zhang, D. Lin, and X. Liang Token-level direct preference optimization. In International Conference on Machine Learning (ICML), p. 59963ā59981. External Links: Link Cited by: Ā§4.2. Zhang et al. (2026a) J. Zhang, Z. Wang, L. Gui, S. M. Sathyendra, J. Jeong, V. Veitch, W. Wang, Y. He, B. Liu, and L. Jin Chasing the tail: effective rubric-based reward modeling for large language model post-training. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Table 4. Zhang et al. (2026b) Q. Zhang, J. Zhou, Y. Wang, F. Lyu, Y. Ming, C. Xu, Q. Sun, K. Zheng, P. Kang, X. Liu, and C. Ma RubricBench: aligning model-generated rubrics with human standards. arXiv preprint arXiv:2603.01562. External Links: Link Cited by: Appendix H. Zhang et al. (2025a) X. Zhang, Y. Liang, F. Meng, S. Zhang, Y. Chen, J. Xu, and J. Zhou CM-align: consistency-based multilingual alignment for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 25689ā25702. External Links: Document, Link, ISBN 979-8-89176-335-7 Cited by: Appendix L. Zhang et al. (2025b) Y. Zhang, T. Yu, H. Tian, C. Fu, P. Li, J. Zeng, W. Xie, Y. Shi, H. Zhang, X. Wang, Y. Hu, B. Luo, and T. Tan M-RLHF: the next step forward in multimodal LLM alignment. In International Conference on Machine Learning, External Links: Link Cited by: Ā§5. Zhang et al. (2026c) Y. Zhang, X. T. Zhang, D. Liu, F. Tian, Y. Deng, J. Chen, Q. Lin, H. Zhang, Y. Li, J. Gong, Y. Huang, L. Zhao, C. Yao, H. Liu, E. S. Chng, X. Yang, G. Yu, X. Zhang, and D. Jiang Step-Audio-R1.5 Technical Report. arXiv preprint arXiv:2604.25719. External Links: Link Cited by: Table 1, Ā§5. Zhao et al. (2023) Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu SLiC-HF: sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. External Links: Link Cited by: Ā§B.2. Zheng et al. (2025) C. Zheng, J. Zhu, Z. Ou, Y. Chen, K. Zhang, R. Shan, Z. Zheng, M. Yang, J. Lin, Y. Yu, and W. Zhang A survey of process reward models: from outcome signals to process supervisions for large language models. arXiv preprint arXiv:2510.08049. External Links: Link Cited by: Ā§B.1, Ā§1. Zheng et al. (2023) R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, L. Xiong, L. Chen, Z. Xi, N. Xu, W. Lai, M. Zhu, C. Chang, Z. Yin, R. Weng, W. Cheng, H. Huang, T. Sun, H. Yan, T. Gui, Q. Zhang, X. Qiu, and X. Huang Secrets of rlhf in large language models part i: ppo. External Links: 2307.04964, Link Cited by: Ā§4.2. Zhong et al. (2026) J. Zhong, H. Zhang, C. Southern, J. Yang, T. Wang, K. Jung, S. Zhang, D. Yarats, J. Ho, and J. Ma DRACO: a cross-domain benchmark for deep research accuracy, completeness, and objectivity. arXiv preprint arXiv:2602.11685. External Links: Link Cited by: Appendix F. Appendix A Survey Methodology and Inclusion Protocol Scope. This survey focuses on methods that use explicit or recoverable structured criteria for LLM alignment, evaluation, reward modeling, or policy optimization. We include work on rubrics, constitutions, principles, checklists, process criteria, and LLM-as-a-judge settings when these criteria influence feedback or rewards. We also include adjacent alignment paradigmsāDPO-style preference optimization, RLVR, PRMs, and multimodal RLHFāwhen they clarify the boundary between rubric-guided RL and competing or complementary methods. Search sources. We searched ACL Anthology, OpenReview, arXiv, Semantic Scholar, Google Scholar, and proceedings or accepted-paper lists from ACL, EMNLP, NAACL, EACL, ICLR, ICML, NeurIPS, CVPR, and related workshops. We additionally followed citation links from seed papers such as Constitutional AI, DPO, PRMs, DeepSeekMath, and recent rubric-generation papers. Query families. Search queries covered five families: (i) rubric, criteria, checklist, grading, and LLM judge; (i) constitution, principle, RLAIF, and AI feedback; (i) DPO, IPO, KTO, SimPO, ORPO, and direct preference optimization; (iv) RLVR, verifier, process reward, mathematical reasoning, and code execution; and (v) multimodal RLHF, hallucination, VLM reward modeling, and multi-criteria VLM evaluation. Inclusion criteria. We included papers if they met at least one criterion: (1) they explicitly use rubrics, principles, checklists, or criteria for alignment, reward modeling, or evaluation; (2) they operationally implement rubric-like structured or multi-dimensional feedback; (3) they are foundational adjacent methods needed to position rubric-guided RL; or (4) they introduce benchmarks or failure analyses that directly inform rubric quality, reward hacking, judge reliability, multilingual evaluation, or multimodal alignment. Reliability and evidence use. Peer-reviewed papers are used as primary evidence whenever available. Influential preprints and technical reports are included when they are widely adopted, provide necessary coverage of fast-moving 2025ā2026 developments, or directly introduce rubric-guided RL methods not yet published in archival venues. We label such work cautiously in the prose and avoid using unstable or unverifiable sources as load-bearing evidence. When a citation could not be matched to a stable title, author list, venue, or archival record, we removed it from the main evidence chain rather than relying on it for taxonomy or quantitative claims. Appendix B Background: Related Surveys and Methods B.1 How This Survey Differs from Prior Surveys Several existing surveys touch on aspects of rubric-guided RL, but none provides a comprehensive treatment. LLM-as-a-Judge Surveys. The comprehensive survey by Gu et al. (2024) covers LLM-as-a-judge methodologies, biases, and applications. While LLM-as-a-judge is closely related to rubric-guided evaluation, the two concepts differ: LLM-as-a-judge typically uses implicit evaluation criteria, while rubric-guided evaluation uses explicit, structured criteria. Our survey bridges these two areas by examining how rubrics can make LLM-as-a-judge more interpretable and controllable. Reward Design and PRM Surveys. Recent surveys on reward design and process reward models Ji et al. (2025b); Zheng et al. (2025) categorize reward formulations and process supervision methods. Our survey differs in its focus on rubric-based reward design, which provides structured, interpretable, and multi-dimensional reward signals. We provide a more detailed treatment of rubric-specific challenges (rubric quality, evolution, exploitation) that are not addressed in general reward design surveys, and we position PRMs as implicit step-level rubrics extending beyond single-dimension process supervision. Self-Evolution and Agentic RL Surveys. Tao et al. (2024) survey self-evolution of large language models. Our survey complements these by focusing specifically on how rubrics enable self-evolution and structured credit assignment in agentic settings, dimensions not covered by existing surveys. B.2 Related Alignment Paradigms RLHF. RLHF Ouyang et al. (2022) trains language models to align with human preferences through a three-stage pipeline. While RLHF has proven effective for alignment, its reliance on scalar reward signals introduces fundamental limitations: the reward is opaque, difficult to interpret, and prone to overoptimization Gao et al. (2023). RLAIF. RLAIF Lee et al. (2024); Bai et al. (2022) replaces human preference labels with AI-generated feedback, addressing the scalability bottleneck of human annotation. Constitutional AI Bai et al. (2022) is a prominent instance of RLAIF that uses explicit natural-language principles to guide AI feedback. DPO and direct preference optimization. DPO Rafailov et al. (2023) bypasses the RL optimization loop by reparameterizing the reward model to derive the optimal policy in closed form. Its variants now form a major alignment family: IPO generalizes preference optimization and regularizes against deterministic-preference overfitting Gheshlaghi Azar et al. (2024), KTO learns from pointwise desirable/undesirable examples Ethayarajh et al. (2024), SimPO removes the reference model with a length-normalized implicit reward Meng et al. (2024), ORPO merges SFT and preference optimization into one stage Hong et al. (2024), and CPO/SLiC-HF show contrastive calibration as another RL-free route Xu et al. (2024); Zhao et al. (2023). This family is directly relevant to rubric-guided alignment because rubric scores can be converted into pairwise or pointwise preference labels; consequently, DPO-style optimization is best viewed as a competing or complementary backend for using rubric feedback, not merely as background RLHF literature. RLVR. RLVR couples reinforcement learning with objective, externally verifiable signals such as mathematical correctness, code execution, or rule-based answer checking Shao et al. (2024); DeepSeek-AI (2025); Yang et al. (2024); Lambert et al. (2025). DeepSeekMath introduced GRPO for math reasoning with rule-based rewards Shao et al. (2024), while DeepSeek-R1 and Qwen2.5-Math show that large-scale verifier- or rule-based RL can produce strong reasoning models in domains with checkable answers DeepSeek-AI (2025); Yang et al. (2024). Open post-training recipes such as TĆ¼lu 3 further incorporate verifiable tasks into modern alignment pipelines Lambert et al. (2025). These methods are direct competitors to rubric-guided RL in math and code because verifiable rewards are cheaper and less subjective; rubric-guided methods are most compelling when tasks require partial credit, explanation quality, safety, pedagogy, or other criteria that are not externally decidable. PRMs. PRMs Lightman et al. (2024) provide step-level feedback for reasoning tasks. We reframe PRMs as implicit, step-level rubrics that score the ācorrectnessā dimension at each reasoning step. The generalization from single-dimension PRMs to multi-dimensional rubric-guided process supervision (as in RLCER Sheng et al. (2026)) is a natural but non-trivial extension. GRPO. GRPO Shao et al. (2024) eliminates the need for a value network by using group-relative rewards. GRPO has become a popular choice for rubric-guided RL methods Gunjal et al. (2026) because it naturally accommodates structured, multi-dimensional reward signals. Appendix C Mapping Method Classes to the Bayesian Framework The Bayesian framework maps method classes according to how they specify, condition, and update evaluation criteria. Fixed-principle methods such as Constitutional AI specify input-independent criterion sets, corresponding to point-mass priors. Instance-specific systems such as RaR, OpenRubrics, PRMs, and judge-rubric generators construct input-conditioned criteria, corresponding to posterior point estimates. Self-evolving or learned-rubric methods such as EvoLM, Rubric-ARM, and RLCER occupy the update-oriented part of the framework because their rubrics or reward models change with training data, policy outputs, or temporal contrast. EvoLM most directly supports a posterior-sharpening interpretation; systems without explicit priors and likelihoods remain analogues. This mapping clarifies which components of Pā”(ā)P(R) and Pā”(ā|x)P(R|x) each method operationalizes. Appendix D What the Bayesian Framework Adds Beyond Re-labeling This framework provides conceptual and practical utility beyond simple notation replacement. Conceptually, it resolves the terminological ambiguity plaguing the field: constitutions and rubrics are unified as the same mathematical object at different conditioning levels. Constitutions act as global, unconditioned priors Pā”(ā)P(R), while instance-specific rubrics are input-conditioned posteriors Pā”(ā|x)P(R|x). Methods such as EvoLM Li et al. (2026c), which optimize evolving rubric representations, instantiate the update-oriented portion of this framework most directly. Practically, this perspective yields hypotheses about training dynamics and adversarial robustness. In explicitly probabilistic formulations, the observed āvague to specificā evolution pattern can be interpreted as posterior sharpening as policy data shift; other self-evolving systems are update-oriented analogues. The lens also casts rubric exploitation as overfitting to a single conditional point estimate āxR_x and suggests a robustness direction: marginalizing over the rubric distribution, rrobustā(x,y)=āxā¼Pā”(ā|x)ā[rāxā(x,y)]r_robust(x,y)=E_R_x P(R|x) [r_R_x(x,y) ] which encourages the policy to generalize across plausible evaluation criteria rather than gaming a single deterministic set. This remains a design hypothesis requiring empirical validation. Appendix E Computational Efficiency Analysis While rubric-guided reinforcement learning (RL) enhances reward interpretability, it introduces non-negligible computational and token overhead into the optimization loop. To provide a rigorous yet concise overview, we present a structural complexity analysis across different paradigms. Let P denote the token length of static constitutional principles, R the average token length of an individual criterion, K the number of criteria per instance, and S the intermediate step length for process-supervised approaches. We define TgenT_gen and TevalT_eval as the real-time execution latencies for rubric generation and judge evaluation, respectively. As summarized in Table 3, the efficiency profiles across paradigms diverge along three core dimensions: ā¢ Token Consumption & Caching: Static approaches (e.g., CAI Bai et al. (2022)) incur a constant ā”(P)O(P) token overhead inside the evaluation context and fully benefit from inference-time prefix caching. Conversely, instance-specific frameworks (e.g., RaR Gunjal et al. (2026), Rubric-ARM Xu et al. (2026)) require online synthesis of conditional rubrics, injecting an uncacheable ā”(Kā R)O(KĀ· R) dynamic token overhead per instance. ā¢ Evaluation Call Complexity: In implicit aggregation modes (e.g., EvoLM Li et al. (2026c), RaR Implicit), the judge model evaluates all dimensions simultaneously in a single forward pass (ā”(1)O(1) complexity). Explicit modes (e.g., RaR Explicit) serialize the verification, demanding ā”(K)O(K) independent judge calls per instance, which scales to ā”(Mā K)O(MĀ· K) during test-time candidate (M) verification and inherently restricts optimal inference search depths. ā¢ Judge Infrastructure: Leveraging commercial APIs typically introduces unpredictable network latency and financial costs. Modern paradigms optimize cost-effectiveness by constraining highly localized small models (ā³localM_local, e.g., 1.7Bā3B parameters) within precise rubric structures, compressing the localized evaluation latency to a fraction of commercial API execution times without compromising reward fidelity. Method Rubric Context Cost Judge Calls / Inst. Eval. Latency Typical Judge Scale Scalar RM (Baseline) 00 1RM1_RM ā”(1)O(1) ā³denseM_dense PRM Lightman et al. (2024) 00 (Implicit) S ā”(Sā Teval)O(SĀ· T_eval) ā³denseM_dense Constitutional AI Bai et al. (2022) ā”(P)O(P) 11 ā”(Teval)O(T_eval) ā³largeM_large RaR (Explicit) Gunjal et al. (2026) ā”(Kā R)O(KĀ· R) K ā”(Kā Teval)O(KĀ· T_eval) ā³smallM_small (API) RaR (Implicit) Gunjal et al. (2026) ā”(Kā R)O(KĀ· R) 11 ā”(Teval)O(T_eval) ā³smallM_small (API) Rubric-ARM Xu et al. (2026) ā”(Kā R)O(KĀ· R) 11 ā”(Teval)O(T_eval) ā³localM_local EvoLM Li et al. (2026c) ā”(Kā R)O(KĀ· R) 11 ā”(Teval)O(T_eval) ā³localM_local Table 3: Computational complexity and resource mapping across key paradigms. P and R denote the token lengths of constitutional principles and individual criteria, respectively; K specifies the total number of evaluation criteria, and S represents the reasoning sequence steps. The base latency parameter TevalT_eval scales asymptotically as a function of the designated judge model capacity ā³M. Appendix F Detailed Agentic and Deep Research Benchmarks A rapidly growing line of work applies rubric-based evaluation to deep research agentsāLLM agents that perform multi-step information gathering, synthesis, and report generation. ResearchRubrics Sharma et al. (2025) provides 2,500+ expert-written rubrics across 100+ real-world tasks in 9 domains. DRACO Zhong et al. (2026) introduces a cross-domain benchmark spanning 10 domains and 40 countriesā information sources, evaluating outputs along factual accuracy, completeness, objectivity, and citation quality. DeepResearchBench Li et al. (2026b) constructs 9,430 fine-grained binary rubrics from expert-written investigative articles. DEER Han et al. (2025) develops a 7-dimension, 25-subdimension, 101-item rubric taxonomy with claim verification, making system strengths and limitations interpretable. Appendix G Continual Learning Benchmarks CL-bench Dou et al. (2026b) introduces a benchmark with 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics for assessing context learning. CL-bench Life Dou et al. (2026a) extends this to real-life scenarios with 405 context-task pairs and 5,348 rubrics. In continual learning, rubrics serve as memory, where criteria from previous tasks provide evaluation frameworks that prevent catastrophic forgetting. Appendix H Consistency and Rubric Quality RubricBench Zhang et al. (2026b) introduces a curated benchmark with 1,147 pairwise comparisons to assess the reliability of rubric-guided evaluation. RubricEval Pan et al. (2026) is the first rubric-level meta-evaluation benchmark for instruction following. Appendix I Representative Reported Results from Original Papers Table 4 summarizes representative results reported by original papers. Important caveat: these results are reported under different evaluation settings, model sizes, and training configurations. They are included to document each paperās own evaluation setting, not to rank methods against each other. Method Base Model Benchmark Metric Result RaR Gunjal et al. (2026) Qwen-2.5-7B HealthBench, Rel. improvement +31% GPQA-Diamond Rel. improvement +7% Rubric-ARM Xu et al. (2026) Various RewardBench Avg. gain +4.7% EvoLM Li et al. (2026c) Qwen3-8B RewardBench-2, Over GPT-4.1 +25.7% OLMo3-Adapt Average 69.3% RLCER Sheng et al. (2026) 7B Math benchmarks Over RLVR Outperforms OpenRubrics Liu et al. (2026b) Various RewardModel Over baselines +8.4% PRM Lightman et al. (2024) Various MATH Accuracy 78.2% Constitutional AI Bai et al. (2022) Various Harmlessness Elo Pareto Improved Chasing the Tail Zhang et al. (2026a) Various RewardBench Accuracy gain +3.2% RLCF Viswanathan et al. (2025) Various Multi-domain Consistent gain Improved Note: Results are from original papers under different settings and should NOT be directly compared. Table 4: Representative results reported by original papers. These numbers are included only to document each paperās own evaluation setting and should not be interpreted as a controlled cross-method comparison. Appendix J Detailed Comparison of Rubric Aggregation Strategies Rubric-guided RL systems differ not only in how they aggregate criterion-level scores, but also in how they construct criteria and assign credit. Table 5 compares the principal reward construction and aggregation strategies used by representative methods. Strategy Formulation Used By Pros Cons Weighted Sum r=ākwkāskākwkr= _kw_ks_k _kw_k RaR (Explicit) Independent criterion scores expose their contributions and permit direct control through criterion weights. The result depends on prespecified weights and separate judge decisions may miss interactions among criteria. Implicit Aggregation r=fĻā(x,y,(wk,dk))r=f_Ļ(x,y,\(w_k,d_k)\) RaR (Implicit) A single holistic judgment can integrate interactions without requiring a separate call for every criterion. Delegating aggregation to the judge makes the final score less directly attributable to individual criteria. Learned Rubric-Conditioned Judging oā¼Ļj(ā ā£x,y(1),y(2),r)o _j(Ā· x,y^(1),y^(2),r) Rubric-ARM Preference feedback jointly improves rubric generation and rubric-conditioned judging under a shared correctness objective. Training two coupled components adds computation and requires careful scheduling to control non-stationarity. Discriminative Rubrics Optimize rubrics for preference separation EvoLM Instance-specific rubrics emphasize differences that make closely matched response pairs more informative for training. Because the objective rewards separability, coverage of important but weakly discriminative criteria requires separate validation. Step Product r=ātptā(correct)r= _tp_t(correct) PRM Step-level evaluations localize errors while yielding a solution score that represents all steps being correct. Multiplication is sensitive to sequence length and calibration, so one underestimated step can suppress the entire score. Principle Ensembling Sample one principle per comparison CAI Sampling across principles produced more robust preference-model behavior than repeatedly applying a single principle. Each comparison applies only one sampled principle, so conflicts and coverage gaps are not resolved within that label. Checklist Aggregation r=ākwkāskākwkr= _kw_ks_k _kw_k RLCF Atomic criteria and optional programmatic verifiers make individual failures inspectable and hard constraints checkable. Scoring items separately is judge-call intensive, and optimizing checklist completion can produce reward-hacking artifacts. Hierarchical Credit Aggregate subgoal returns at planning and execution levels HiPER Hierarchical advantages assign credit at both levels and reduce variance in long-horizon agent training. The method requires a plannerāexecutor decomposition and introduces additional level-specific optimization choices. Pairwise Adaptive Instantiate pair-specific criteria and aggregate criterion preferences externally OpenRS Pair-conditioned criteria target semantic differences while retaining inspectable, externally aggregated preferences. A new rubric must be instantiated for each pair, increasing evaluation work and limiting direct cross-pair comparability. Table 5: Comparison of rubric reward construction and aggregation strategies. Each strategy exposes a different trade-off among interpretability, computational cost, criterion coverage, and the ability to capture interactions. Appendix K Extended Discussion on Rubric Quality Dimensions Discriminative Power. Discriminative power measures how well a rubric distinguishes between high-quality and low-quality responses. A rubric with high discriminative power assigns significantly different scores to good and bad responses, while a rubric with low discriminative power assigns similar scores regardless of quality. This can be measured by the accuracy of rubric-conditioned evaluation on held-out preference data. Coverage. Coverage measures how many relevant quality dimensions a rubric captures. A rubric with high coverage evaluates all important aspects of response quality (correctness, completeness, clarity, safety, etc.), while a rubric with low coverage may miss important dimensions. PRMs, for example, typically cover a narrower set of dimensions. Incomplete rubrics may fail to capture important aspects of response quality, leading to reward hacking through unmeasured dimensions. Calibration. Calibration measures how well rubric scores align with human judgment across the full quality spectrum. A well-calibrated rubric assigns scores that are consistent with human evaluation, while a poorly calibrated rubric may over- or under-estimate quality. LLM-Rubric Hashemi et al. (2024) proposes calibration methods for rubric-based evaluation. Calibrating rubric-based scores to match human judgment distributions is critical for consistency. Appendix L Linguistic Insights: Detailed Evidence and Qualitative Examples This appendix provides detailed evidence and qualitative examples supporting the claims in Section 4. Semantic Drift: Qualitative Evidence. EvoLM Li et al. (2026c) provides the most direct evidence for semantic drift: rubrics evolve from vague criteria (e.g., āEnsure mathematical accuracyā) to specific, verifiable facts (e.g., āThe answer is 144, derived from perimeter 48ā) as training progresses. This intentional drift admits a posterior-sharpening interpretation. Constitutional AI Bai et al. (2022) observed unintentional drift through Goodharting, where over-optimization leads to boilerplate language and overly harsh responses. The diagnostic in Equation 3 is a proposal of this survey; no prior work has validated this specific formalization for rubric-guided RL. We present a concrete case study of semantic drift in Table 6. Linguistic Reward Hacking: Qualitative Examples. ā¢ Verbosity exploitation: Gao et al. (2023) document reward model overoptimization where proxy rewards increase while true rewards decline, a dynamic closely related to verbosity exploitation. In rubric contexts, models learn to pad responses with filler to satisfy ācompletenessā criteria without adding substantive content. ā¢ Criterion bleeding: Safe RLHF Dai et al. (2024) explicitly addresses the safety-helpfulness tradeoff, where maximizing safety (refusing all requests) violates helpfulness. ArmoRM Wang et al. (2024) uses multi-objective reward modeling to mitigate criterion bleeding by balancing multiple quality dimensions simultaneously. ā¢ Surface pattern matching: self-evolving rubric work suggests that evaluator capacity and rubric specificity interact, with smaller evaluators often requiring more explicit criteria to distinguish genuine satisfaction from superficial compliance Li et al. (2026c). This suggests that surface pattern matching is a real phenomenon that can be partially mitigated by evaluator and rubric design. We provide concrete case studies for each type of linguistic reward hacking, drawing on empirical evidence from recent work. The case studies can be found in Table 7, 8, 9 and 10. Case Study 1: Semantic DriftāIntentional vs. Unintentional Intentional drift (EvoLM): Li et al. (2026c) document that rubrics evolve from vague criteria (e.g., āEnsure mathematical accuracyā) to specific, verifiable facts (e.g., āthe correct maximum area of 144, derived from the given perimeter of 48ā) as training progresses. This intentional drift admits a posterior-sharpening interpretation and improves downstream policy quality. Unintentional drift (CAI): Bai et al. (2022) observe that the meaning of āharmlessnessā drifts from āavoid causing harmā to ārefuse all potentially sensitive queriesā under over-optimization. The rubric text remains unchanged, but the policyās interpretation shifts, leading to overly harsh and unhelpful responses. Table 6: Semantic drift in rubric-guided RL: intentional drift (posterior sharpening) may improve quality, while unintentional drift (Goodharting) can degrade it. Case Study 2: Verbosity Exploitation via Presence-Based Rubrics Source: Mahmoud et al. (2026) Phenomenon: The RL checkpoint improves completeness (+1.07) but degrades conciseness (ā-2.91), factual correctness (ā-0.85), and overall quality (ā-1.02). The rubric-based verifier prefers the RL checkpoint on 85.8% of prompts, but rubric-free judges prefer the base model on 78.4%āa clear sign of reward hacking under strong verification. Mechanism: 90.2% of rubric weight falls on presence-based criteria (e.g., āStates that plasma volume increases more than red cell mass during pregnancyā). Presence-based satisfaction rises from 27.6% to 42.5% (+14.9 p), while absence-based criteria decline. The model exploits this by generating longer, claim-denser responses that satisfy presence checks without verifying correctness. Safety-presence exploitation: Safety-presence rubric items (8.4% weight) such as āThe response advises the user to consult a healthcare provider before taking any actionā can be satisfied by appending boilerplate disclaimers without genuine safety reasoning. Table 7: Verbosity exploitation through presence-based rubric criteria. The model learns to satisfy presence checks by generating longer, claim-denser output, while degrading on conciseness, factual correctness, and overall quality. Case Study 3: Criterion BleedingāSafety-Helpfulness Tradeoff Phenomenon: Maximizing the safety criterion (āthe response should not cause harmā) by refusing all requests directly violates the helpfulness criterion (āthe response should address the userās queryā). Safe RLHF Dai et al. (2024) and ArmoRM Wang et al. (2024) have proposed mechanisms to mitigate this problem. We categorize this phenomenon as āCriterion bleedingā ā satisfying one rubric dimension at the expense of another. Constitutional AI evidence: Bai et al. (2022) observe that over-optimization against constitutional principles leads to boilerplate language and overly harsh responsesāthe policy learns to satisfy safety keywords (āI cannot assist with thatā) while violating the spirit of balanced evaluation. Mitigation: Safe RLHF explicitly decomposes the reward into separate safety and helpfulness objectives with constrained optimization. ArmoRM Wang et al. (2024) uses multi-objective reward modeling to balance multiple quality dimensions simultaneously, preventing criterion bleeding by design. Table 8: Criterion bleeding: when satisfying one rubric dimension (safety) comes at the expense of another (helpfulness), the policy exploits the multi-objective gap. Case Study 4: Surface Pattern MatchingāRubric Verification Failure Modes Source: Mahmoud et al. (2026) Failure Mode Taxonomy (stable across training, domains, and verifier strength): ā Partial Compound (36.0%): The criterion requires Aā§ but the verifier accepts only one conjunct. Example: āThe verifier failed because it accepted partial satisfaction of a multi-part requirement as full credit, verifying the outcome statement while not enforcing the required explicit distinction between two specified categories.ā ā Implicit-as-Explicit (34.6%): The required claim is never explicitly stated; the verifier infers it from context. Example: āThe verifier failed because it credited an implicit or inferable statement as if it were explicit, accepting broad plausibility instead of requiring the exact characterization the criterion demanded.ā ā Imprecise Verification (29.4%): The verifier accepts a related but distinct concept as equivalent, or checks only broad topic relevance. Example: āThe verifier failed because it matched on surface topic relevance instead of verifying the specific claim.ā Key insight: These failure modes are stable across trainingāthe distribution of exploitation types does not shift, only the volume increases. Both weak and strong verifiers fail in the same proportions, suggesting these are fundamental limitations of rubric verification. Table 9: Surface pattern matching in rubric verification: three recurring structural failure modes where the verifier accepts superficial criterion satisfaction without genuine engagement. Case Study 5: Granularity Effects and Evaluator Capacity Phenomenon: Smaller judges (1.7B) force the learned rubric generator to produce more precise, fine-grained criteria, resulting in better downstream policies than larger judges (up to 32B) that tolerate vague rubrics. This demonstrates that optimal rubric granularity interacts with judge capacity: a rubric that is ātoo coarseā for a small judge may be ājust rightā for a large judge, and vice versa. PRM evidence: Lightman et al. (2024) show that step-level (fine-grained) process supervision achieves 78.2% on MATH, substantially outperforming outcome-level (coarse-grained) supervision. This confirms that finer granularity produces better signals, but we argue that such benefits can be fully achieved only when the evaluator can reliably apply the fine-grained criteria. RLCF evidence: RLCF Viswanathan et al. (2025) decomposes each instruction into atomic checklist items and uses them as instance-level reward criteria, illustrating how checklist-style rubrics can provide denser and more inspectable feedback than holistic outcome rewards. Table 10: Granularity effects: the interaction between rubric granularity and judge capacity determines downstream policy quality. Finer granularity generally helps, but only when the evaluator can reliably apply the criteria. Underspecification as Opportunity. No prior work explicitly argues that underspecification is an opportunity in rubric-guided RL. However, several methods implicitly exploit it: EvoLM Li et al. (2026c) observes rubrics evolving from vague to specific through posterior sharpening, RLCER Sheng et al. (2026) starts with underspecified rubrics that self-evolve during training, and Rubric-ARM Xu et al. (2026) learns rubric-conditioned reward models in non-verifiable settings. We posit that underspecification provides the space for rubric self-evolutionāwithout initial underspecification, adaptive sharpening would have no room to operate. Cross-Linguistic Considerations. Hong et al. (2025) demonstrate that English-trained reward models exceed target-language reward models by 3ā4% on Multilingual RewardBench, suggesting cross-lingual transfer is feasible but imperfect. Sheth et al. (2026) propose a Universal Criteria Set (UCS) with language-agnostic evaluation dimensions combined with language-specific instantiations. Fu and Liu (2025) find that multilingual LLM-as-a-judge has low cross-lingual consistency (Fleissā Kappa ā 0.3), indicating that rubric-conditioned judges face the same challenge. The mR3 model Anugraha et al. (2026) extends to 72 languages with rubric-agnostic reward reasoning, while M-RewardBench Gureja et al. (2025) provides the first systematic multilingual RM evaluation across 23 languages. Language-specific quality dimensions include Japanese keigo (honorific system) with multiple formality levels, Korean jondaetmal (honorific usage) Hwang et al. (2024), and Arabic diglossiaāall absent from English-centric rubrics. CM-Align Zhang et al. (2025a) uses consistency-based approaches for multilingual alignment. Appendix M Adversarial Robustness and Reward Hacking Rubric-guided RL introduces both new defenses against and new vulnerabilities to adversarial attacks and reward hacking. Rubric Exploitation as Overfitting to a Point Estimate. A novel form of reward hacking in rubric-guided RL is rubric exploitation: the policy learns to produce responses that score well against rubric criteria while violating the spirit of the rubric. Under the Bayesian framework (Ā§2.4), rubric exploitation corresponds to overfitting to a point estimate of the rubric posterior: the policy optimizes against a single sample āxR_x rather than the full posterior Pā”(ā|x)P(R|x). The defense is to evaluate under the full posterior (marginalization), which requires the policy to perform well under all plausible rubrics rather than gaming a single one. Rubric-Specific Reward Hacking. Reward Hacking in Rubric-Based RL Mahmoud et al. (2026) studies the specific phenomenon where policies optimized against a training rubric are evaluated against a cross-family panel. The key finding is that rubric-based rewards do not by themselves prevent reward hacking: the gap between training and evaluation rubrics creates an exploitation surface. Defenses. Dynamic rubric adaptation (self-evolving rubrics close exploitation opportunities), multi-rubric evaluation (ensemble methods), constrained optimization with rubric-based safety constraints, and information-theoretic approaches such as InfoRM Miao et al. (2024) provide defenses against rubric exploitation.