Paper deep dive
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
Zhengtao Yao, Runhao Li, Xupeng Chen, Jiayi Cheng, Chenqian Le, Michael Yue, Siheng Wang, Haoyan Xu, Yuqi Li, Chenhao Wei, Zhengdao Li, Rongchao Zhang, Guang Yang, Yidong Wang, Junhao Dong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/1/2026, 11:16:24 AM
Summary
The paper introduces DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), a method that constructs high-confidence preference datasets by generating on-policy responses, evaluating them with rubric-specialized LLMs (helpfulness, factuality, conciseness), applying a process-critic correction, and filtering for consensus. Applied to Mistral-7B, DMAPO retains only 3.45% of candidates (1,871 examples) but achieves superior performance on MT-Bench, AlpacaEval, and IFEval compared to baselines like DPO, KTO, SimPO, and SFT, demonstrating that data quality and consensus filtering are critical for efficient preference optimization.
Entities (16)
Relation Signals (16)
DMAPO → achievesscoreon → MT-Bench
confidence 95% · On Mistral, DMAPO has the highest MT-Bench (7.50)...
DMAPO → trainson → Mistral-7B-Instruct-v0.2
confidence 95% · On Mistral-7B-Instruct-v0.2, DMAPO reaches an MT-Bench score of 7.50...
Qwen3-8B → usedasevaluatorby → DMAPO
confidence 95% · The default pipeline uses Qwen3-8B (Qwen Team, 2025) in /no_think mode.
DMAPO → uses → Qwen3-8b
confidence 95% · Each candidate response y to prompt x is independently evaluated by three Qwen3-8B (Qwen Team, 2025) instances...
DMAPO → uses → KTO
confidence 95% · KTO provides the binary-label objective... KTO receives only the binary labels...
DMAPO → evaluates → Helpfulness
confidence 90% · evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators
DMAPO → evaluates → conciseness
confidence 90% · evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators
DMAPO → evaluates → Factuality
confidence 90% · evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Research on preference optimization often varies the training objective while holding the data fixed. We instead ask whether a small, high-confidence set of on-policy responses can provide a reliable learning signal. Our method, DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators, applies a process-critic correction, and retains only high-consensus desirable or undesirable examples. This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%). KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy. Independent pairwise evaluation also favors DMAPO over SimPO: GPT-4o yields a net win rate of 23.3 points on 129 held-out prompts and 24.0 points on 200 out-of-distribution LMSYS-Chat prompts; Claude Opus 4.7 yields 24.1 points on the held-out set. Changing the evaluator model or rubric alters the selected examples but has little effect on downstream performance. A second-backbone study yields a similar 3.41% acceptance rate, although its performance gains are more modest. Across these experiments, consensus filtering offers a data-efficient route to preference optimization for general instructions, at the cost of additional curation compute and dependence on evaluator judgments.
Tags
Links
- Source: https://arxiv.org/abs/2607.25136v1
- Canonical: https://arxiv.org/abs/2607.25136v1
Trouble viewing inline? Open PDF directly →
Full Text
49,000 characters extracted from source content.
Expand or collapse full text
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization†thanks: Large language models were used to assist with language editing and LaTeX restructuring. The authors verified all technical claims, citations, and reported results. Zhengtao Yao1, Runhao Li1,22footnotemark: 2 Xupeng Chen2 Jiayi Cheng2 Chenqian Le2 Michael Yue3 Siheng Wang4 Haoyan Xu1 Yuqi Li5 Chenhao Wei6 Zhengdao Li4 Rongchao Zhang4 Guang Yang6 Yidong Wang4 Junhao Dong7 1University of Southern California 2New York University 3Columbia University 4University of California, Berkeley 5City College of New York, CUNY 6Stevens Institute of Technology 7Nanyang Technological University Equal contribution. Abstract Research on preference optimization often varies the training objective while holding the data fixed. We instead ask whether a small, high-confidence set of on-policy responses can provide a reliable learning signal. Our method, DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators, applies a process-critic correction, and retains only high-consensus desirable or undesirable examples. This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%). KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy. Independent pairwise evaluation also favors DMAPO over SimPO: GPT-4o yields a net win rate of 23.3 points on 129 held-out prompts and 24.0 points on 200 out-of-distribution LMSYS-Chat prompts; Claude Opus 4.7 yields 24.1 points on the held-out set. Changing the evaluator model or rubric alters the selected examples but has little effect on downstream performance. A second-backbone study yields a similar 3.41% acceptance rate, although its performance gains are more modest. Across these experiments, consensus filtering offers a data-efficient route to preference optimization for general instructions, at the cost of additional curation compute and dependence on evaluator judgments. 1 Introduction Aligning large language models (LLMs) with human preferences is central to making them safe, helpful, and honest (Ouyang et al., 2022; Bai et al., 2022). RLHF (Christiano et al., 2017; Stiennon et al., 2020) and direct objectives such as DPO (Rafailov et al., 2023) have largely framed the problem in terms of the training algorithm: how best to learn from a given preference dataset. Dataset quality, however, can be equally consequential. Annotator disagreement, ambiguous comparisons, and label noise limit what any optimizer can recover. Prior work shows that a small, carefully chosen set can be effective for instruction tuning and preference optimization (Zhou et al., 2023; Deng et al., 2025; Ye et al., 2025). This observation raises a related question: can high-confidence preference feedback be constructed directly from a target policy’s own responses, rather than selected from a fixed preference dataset? DMAPO addresses this question through multi-evaluator consensus gating. It generates responses on-policy and scores them with three rubric-specialized Qwen3-8B evaluator instances. We use “evaluator” rather than “agent” because the method does not require autonomous planning or tool use. A candidate is retained only if its process-critic-adjusted scores meet joint quality and variance criteria. This gate accepts 3.45% of the Mistral candidates and produces nearly balanced desirable and undesirable labels. Trained on 1,871 gated examples, the Mistral model performs well across MT-Bench, AlpacaEval-style evaluation, IFEval, and independent pairwise comparisons. The Llama result is less conclusive: DMAPO yields the strongest diagnostic preference score among the compared methods, but does not improve the stronger base model on MT-Bench. The cross-backbone experiment thus tests the stability of the filtering behavior rather than establishing universal gains. Our contributions are: 1. An end-to-end construction pipeline combining on-policy generation, rubric-specialized evaluation, process criticism, and confidence/variance gating to produce binary preference feedback. 2. A controlled empirical study across two policy backbones, seven training baselines, component ablations, evaluator and prompt variants, and two independent evaluator families. 3. An empirical account of the tradeoffs introduced by aggressive filtering: less training data and shorter optimization, but additional offline curation, moderate variation in the selected set, and slightly lower response diversity. Figure 1: Overview of the DMAPO pipeline and primary MT-Bench result. (A) From a pool of 14,272 prompts, the 13,559-prompt training split yields 54,236 on-policy candidates and 1,871 binary-labeled training examples (3.45% acceptance) after rubric-specialized evaluator scoring, a bounded process critic, and confidence/variance gating. (B) On Mistral-7B-Instruct-v0.2, DMAPO reaches an MT-Bench score of 7.50 using 1,871 rows. This panel reports one benchmark rather than the Qwen-gated diagnostic win rate; independent pairwise results appear in Table 2. 2 Related work Preference optimization. Most preference-optimization research focuses on the learning objective. RLHF trains a reward model and optimizes it with PPO (Ouyang et al., 2022; Schulman et al., 2017); DPO (Rafailov et al., 2023) eliminates the explicit reward model through a closed-form reparameterization; and KTO and ORPO (Ethayarajh et al., 2024; Hong et al., 2024) simplify supervision through binary labels and an odds-ratio objective. SimPO and IPO refine preference losses (Meng et al., 2024; Azar et al., 2024), whereas SPPO introduces iterative self-play (Wu et al., 2024) and REINFORCE++ stabilizes critic-free optimization (Hu et al., 2025). DMAPO is complementary: it studies the construction of a compact, model-specific preference set before optimization. Data-centric alignment. LIMA (Zhou et al., 2023) demonstrated that a small, carefully curated set can support effective instruction tuning, while LIMO (Ye et al., 2025) studied a related phenomenon for mathematical reasoning. WizardLM (Xu et al., 2023) filters by instruction complexity, RLAIF (Lee et al., 2023) uses AI feedback, Self-Instruct (Wang et al., 2023) generates training data, and LESS (Xia et al., 2024) uses influence functions for subset selection. Most directly, Deng et al. (2025) select preference pairs using external and implicit reward margins. UltraFeedback itself supplies multi-aspect AI feedback (Cui et al., 2023). AI feedback, multi-aspect scoring, and data selection are therefore not new in isolation. DMAPO combines them in an on-policy pipeline that converts rubric-specific consensus into binary supervision before preference optimization. LLM-as-Judge. Zheng et al. (2023) introduced MT-Bench and popularized LLM-based evaluation. Such evaluators can exhibit position, verbosity, family, and rubric biases, and agreement within one model family is not equivalent to human validation. Accordingly, we separate Qwen-gated diagnostic metrics from independent evaluation, vary evaluator prompts and model families, and treat evaluator dependence as a limitation. 3 Method DMAPO constructs preference feedback in seven stages. KTO provides the binary-label objective, while most of the additional computation is spent on on-policy generation and evaluation. Three rubric-specialized evaluator instances assess complementary dimensions (helpfulness, factuality, and conciseness), and a candidate is retained only when their adjusted scores satisfy joint confidence and variance criteria. On Mistral-7B this procedure retains 1,871 of 54,236 candidates (3.45%). The accepted desirable and undesirable groups differ by 6.81 points in mean evaluator score. This separation characterizes the labeling procedure; it is not an independent measure of response quality. 3.1 Prompt collection The prompt pool combines UltraFeedback (Cui et al., 2023) (10k diverse instruction prompts) and HelpSteer2 (Wang et al., 2024) (5k helpfulness-focused prompts). Deduplication leaves 14,272 prompts, split 95/5 into training (13,559) and validation (713). 3.2 On-policy candidate generation For each training prompt, Mistral-7B-Instruct-v0.2 (Jiang et al., 2024) generates k=4k=4 candidates with nucleus sampling (temperature 0.8, top-p 0.95). The resulting feedback remains tied to the policy’s current response distribution. 3.3 Multi-evaluator scoring Each candidate response y to prompt x is independently evaluated by three Qwen3-8B (Qwen Team, 2025) instances assessing complementary quality dimensions on a 1–10 scale: helpfulness sh(x,y)s_h(x,y), factuality sf(x,y)s_f(x,y), and conciseness sc(x,y)s_c(x,y). After the process-critic correction (Section 3.4), we define s~j(x,y)=sj(x,y)−δ(x,y),s¯(x,y)=13∑j∈h,f,cs~j(x,y). s_j(x,y)=s_j(x,y)-δ(x,y), s(x,y)= 13 _j∈\h,f,c\ s_j(x,y). (1) Each evaluator follows a structured rubric and returns a numerical score. We use /no_think mode to standardize the output format. The dimensions are complementary: a response may be helpful but factually incorrect, or factual but unnecessarily verbose. 3.4 Process critic A fourth Qwen3-8B instance serves as a process critic. A structured rubric asks it to identify reasoning flaws, factual inconsistencies, and unsupported claims in each prompt–response pair, then assign one of three severity levels: • Minor (stylistic imprecision, redundant phrasing): penalty 0.050.05. • Moderate (incomplete reasoning, weak evidence): penalty 0.100.10. • Severe (logical contradiction, factual error): penalty 0.150.15. Formally, the critic penalty is: δ(x,y)=min(∑i=1|ℱ|pi,αmax),pi∈0.05,0.10,0.15,αmax=0.15δ(x,y)= \! ( _i=1^|F|p_i,\; _ ), p_i∈\0.05,0.10,0.15\, _ =0.15 (2) where ℱF is the set of detected flaws and pip_i is the penalty for flaw i. The same scalar correction is applied to each dimension before gating (Eq. 1). Capping it at 0.15 keeps the critic subordinate to the three primary evaluators: it can alter a decision near the threshold but cannot offset a large deficit on any dimension. Appendix L reports a cap sweep. 3.5 Confidence gating The confidence gate maps multi-evaluator scores to a ternary label ℓ∈des,und,discard ∈\des,und,discard\ via: ℓ(x,y)=desif minjs~j≥τ+∧Var(~)<σ2undif maxjs~j≤τ−∧Var(~)<σ2discardotherwise (x,y)= casesdes&if _j s_j≥ _+\; \;Var( s)<σ^2\\ und&if _j s_j≤ _-\; \;Var( s)<σ^2\\ discard&otherwise cases (3) where ~=(s~h,s~f,s~c) s=( s_h, s_f, s_c), τ+=7 _+=7, τ−=4 _-=4, and σ2=2.5σ^2=2.5. Because the conditions apply jointly, no dimension can compensate for a failure on another; high conciseness, for example, cannot offset low helpfulness or factuality. The gate retains 3.45% of candidates (Table 13), whose mean adjusted scores are 9.23 for desirable and 2.42 for undesirable examples. 3.6 KTO policy training KTO (Ethayarajh et al., 2024) operates directly on binary labels and therefore matches the supervision produced by the gate. Let rθ(x,y)=logπθ(y|x)πref(y|x)r_θ(x,y)= _θ(y|x) _ref(y|x) denote the implicit reward and z0=x′[KL[πθ(⋅|x′)∥πref(⋅|x′)]]z_0=E_x [KL[ _θ(·|x )\| _ref(·|x )] ] a running baseline. The KTO loss is: ℒKTO=(x,y)∼σ(z0−βrθ(x,y))if ℓ(x,y)=desσ(βrθ(x,y)−z0)if ℓ(x,y)=undL_KTO=E_(x,y) casesσ (z_0-β\,r_θ(x,y) )&if (x,y)=des\\ σ (β\,r_θ(x,y)-z_0 )&if (x,y)=und cases (4) where σ is the sigmoid and β=0.1β=0.1 controls the reward scale. The loss raises the relative probability of desirable outputs and lowers that of undesirable outputs. KTO receives only the binary labels, not the evaluator scores or their 6.81-point separation. Its low final training loss (0.037; Appendix G) indicates that these labels are separable, but says nothing by itself about generalization. We use LoRA (Hu et al., 2022) (r=16r=16, α=32α=32) targeting all linear layers (∼ 160M trainable parameters, 2.2% of backbone). 4 Experimental setup 4.1 Models Policy backbones. Mistral-7B-Instruct-v0.2 (Jiang et al., 2024) (7.24B parameters) is the primary backbone; Llama-3.1-8B-Instruct (Grattafiori et al., 2024) (8.03B parameters) provides a second-backbone check. Curation evaluators. The default pipeline uses Qwen3-8B (Qwen Team, 2025) in /no_think mode. Section 6.4 varies both the rubric and evaluator family. Evaluation models. GPT-4o scores MT-Bench and serves as one independent pairwise evaluator; Claude Opus 4.7 provides the second independent pairwise evaluation. 4.2 Baselines The comparison includes seven training-time baselines and four data-filtering ablations. All use the same backbone, LoRA configuration, optimizer, and learning rate. Established baselines. • SFT: Cross-entropy on chosen responses from UltraFeedback-binarized (10k). • DPO (Rafailov et al., 2023): Sigmoid pairwise preference loss (10k pairs). • KTO (Ethayarajh et al., 2024): Binary preference loss (20k rows). • ORPO (Hong et al., 2024): Odds-ratio preference loss, no reference model (10k pairs). • SimPO (Meng et al., 2024): Reference-free reward via average log-probability (10k pairs). Recent 2025 baselines. • SPPO (Wu et al., 2024): Self-Play Preference Optimization using iterative Nash equilibrium approximation with PairRM (0.4B) as the preference model and 60k UltraFeedback prompts. We run 3 iterations of self-play with the same backbone. • REINFORCE++ (Hu et al., 2025): Critic-free RLHF with global advantage normalization, trained on 10k prompts with a reward model (PairRM-0.4B) providing online rewards. Data-filtering ablations. • Random-1.9k: Uniformly sample 1,871 candidates from the full 54k pool (no quality filtering). Trains with KTO using the same recipe as DMAPO. • Single-Evaluator-1.9k: Use only the helpfulness evaluator (shs_h) to select the top 1,871 candidates by score. No multi-evaluator consensus or variance check. • No-Variance-Gate: Apply the desirable/undesirable score thresholds (τ+=7 _+=7, τ−=4 _-=4) but remove inter-evaluator variance constraint (σ2σ^2), yielding ∼ 4,297 examples. Trains with KTO. • NVG-1.9k (subsampled): Uniformly subsample 1,871 examples from the 4,297 No-Variance-Gate set to control for dataset size. Trains with KTO using the same recipe. 4.3 Training details All methods use LoRA (r=16r=16, α=32α=32, dropout 0.05) on the q/k/v/o/gate/up/down projections and AdamW with a learning rate of 5×10−55×10^-5, cosine decay, and 5% warmup. The effective batch size is 16, training uses bfloat16 precision for one epoch, and results are reported as mean± deviation over seeds 42,123,456,789\42,123,456,789\. Experiments run on 8×8× NVIDIA A100 80 GB GPUs with TRL 0.29.0 and PEFT 0.18.1. 4.4 Evaluation MT-Bench (Zheng et al., 2023) comprises 80 multi-turn questions in eight categories, scored from 1–10 by GPT-4o. AlpacaEval-style evaluation uses 805 instructions and a text-davinci-003 reference. Because this is not the standard AlpacaEval 2.0 configuration, we report the setup explicitly and include both raw win rate (WR) and length-controlled win rate (LC) (Dubois et al., 2024). IFEval (Zhou et al., 2023) contains 541 prompts with rule-verifiable constraints. Independent pairwise evaluation compares DMAPO with three strong baselines using GPT-4o and Claude Opus 4.7. It covers 129 held-out prompts and 200 out-of-distribution LMSYS-Chat prompts (Zheng et al., 2023); the latter were not used for generation, filtering, validation, or hyperparameter selection. We report win/tie/loss percentages and net win rate (win minus loss). The Qwen-gated diagnostic WR instead compares log probabilities against the base model on the 129-prompt held-out split. Because its labels come from the evaluator family used for curation, it measures fit to the gated signal rather than independent response quality. 5 Results 5.1 Main results Table 1 summarizes the main benchmark results. Table 1: Main results on Mistral-7B-Instruct-v0.2. All methods use the same LoRA configuration. Dataset sizes count binary rows for KTO and preference pairs for pairwise objectives. The final column is evaluator-dependent and is reported only as a diagnostic. Results are mean± over four seeds. Method |||D| MT-Bench ↑ AE-style WR% ↑ AE-style LC% ↑ IFEval Acc% ↑ Qwen diag. WR% ↑ Base (Mistral-7B-v0.2) — 7.41 96.0 93.2 52.7 — Established baselines + SFT 10,000 6.71± 0.08 86.1± 0.7 81.5± 0.9 52.3± 0.6 32.6± 1.8 + DPO 10,000 7.08± 0.06 95.7± 0.4 92.4± 0.5 54.9± 0.7 43.4± 2.1 + KTO 20,000 7.25± 0.07 95.5± 0.3 91.8± 0.4 54.0± 0.5 36.4± 1.5 + ORPO 10,000 7.42± 0.05 96.3± 0.3 93.5± 0.4 54.3± 0.6 55.8± 2.3 + SimPO 10,000 7.23± 0.09 95.5± 0.5 91.5± 0.6 54.0± 0.8 87.6± 1.4 Recent 2025 baselines + SPPO 60,000 7.28± 0.07 95.9± 0.4 92.6± 0.5 53.5± 0.6 72.8± 2.0 + REINFORCE++ 10,000 7.35± 0.06 96.1± 0.3 93.0± 0.4 53.8± 0.7 76.3± 1.9 Data-filtering ablations (KTO, same recipe as DMAPO) + Random-1.9k 1,871 7.10± 0.11 95.2± 0.5 91.6± 0.7 52.8± 0.8 78.3± 2.5 + Single-Evaluator-1.9k 1,871 7.18± 0.08 95.4± 0.4 92.0± 0.5 53.2± 0.7 86.0± 2.0 + No-Variance-Gate 4,297 7.24± 0.06 95.3± 0.4 92.1± 0.5 53.6± 0.5 77.5± 1.8 + NVG-1.9k (subsamp.) 1,871 7.20± 0.07 95.1± 0.5 91.8± 0.6 53.3± 0.6 80.2± 2.0 + DMAPO (ours) 1,871 7.50± 0.05 98.0± 0.3 95.5± 0.4 57.3± 0.5 90.7± 1.2 On Mistral, DMAPO has the highest MT-Bench (7.50), AE-style LC (95.5), and IFEval (57.3) scores among the compared methods. Its margin over the strongest baseline is 0.08 on MT-Bench, 2.0 points on AE-style LC, and 2.4 points on IFEval. Raw AE-style win rates are uniformly high because the reference is text-davinci-003, making the length-controlled result more informative. The 805 AE-style instructions do not overlap with the curation pool, and IFEval uses a separate prompt set with rule-based scoring. The Qwen-gated diagnostic WR reaches 90.7%, but it is not an independent evaluation because the same evaluator family produced the curation labels. It is useful only for checking whether the model learned the gated signal. Within this diagnostic, removing the variance gate lowers WR despite using more data, and single-dimension filtering underperforms consensus gating. The next section turns to independent evidence. 5.2 Independent pairwise evaluation Table 2 compares model outputs using evaluators that did not participate in data construction. On the 129 held-out prompts, GPT-4o gives DMAPO net-win margins of 23.3 points over SimPO, 27.2 over ORPO, and 31.8 over REINFORCE++. Claude Opus 4.7 produces similar margins. GPT-4o also favors DMAPO on the 200 out-of-distribution LMSYS-Chat prompts, by 24.0–28.0 points. As expected, these independent margins are smaller than the Qwen-gated diagnostic gap, but their direction is consistent across evaluators and prompt sources. Table 2: Independent pairwise evaluation. Evaluators were not used in data construction. Net win is win minus loss; all entries are percentages. Prompt set Evaluator Comparison Win Tie Loss Net win Held-out (129) GPT-4o DMAPO vs. SimPO 50.4 22.5 27.1 +23.3 DMAPO vs. ORPO 54.3 18.6 27.1 +27.2 DMAPO vs. REINFORCE++ 56.6 18.6 24.8 +31.8 Held-out (129) Claude Opus 4.7 DMAPO vs. SimPO 51.2 21.7 27.1 +24.1 DMAPO vs. ORPO 53.5 19.4 27.1 +26.4 DMAPO vs. REINFORCE++ 55.8 18.6 25.6 +30.2 LMSYS-Chat OOD (200) GPT-4o DMAPO vs. SimPO 50.5 23.0 26.5 +24.0 DMAPO vs. ORPO 53.0 19.5 27.5 +25.5 DMAPO vs. REINFORCE++ 55.0 18.0 27.0 +28.0 5.3 MT-Bench category analysis Table 3 breaks MT-Bench down by category. DMAPO attains the highest Reasoning score (7.45) among the compared methods. This may reflect the helpfulness and factuality gates, but the category sample is too small to support a mechanistic conclusion. Table 3: MT-Bench per-category results. 80 multi-turn questions, eight categories. Mean± over four seeds. Best per category in bold. Method Coding Extr. Hum. Math Reason. Role. STEM Writing Avg. Base 5.60 7.90 8.70 6.45 7.15 7.40 8.20 7.90 7.41 + SFT 5.45± 0.15 7.50± 0.18 7.45± 0.20 5.60± 0.22 6.75± 0.18 6.60± 0.15 6.95± 0.17 7.40± 0.14 6.71± 0.08 + DPO 5.70± 0.18 7.45± 0.15 8.55± 0.12 4.70± 0.25 6.80± 0.14 7.60± 0.13 7.65± 0.16 8.15± 0.11 7.08± 0.06 + KTO 6.05± 0.20 6.70± 0.22 8.85± 0.10 6.00± 0.18 6.55± 0.16 7.65± 0.12 8.00± 0.14 8.20± 0.10 7.25± 0.07 + ORPO 6.40± 0.16 7.50± 0.14 8.85± 0.11 5.20± 0.20 7.30± 0.13 7.70± 0.11 8.40± 0.12 8.00± 0.13 7.42± 0.05 + SimPO 5.95± 0.19 6.60± 0.20 8.80± 0.13 5.50± 0.23 7.30± 0.15 7.85± 0.10 7.95± 0.15 7.90± 0.12 7.23± 0.09 + SPPO 5.85± 0.17 7.35± 0.16 8.60± 0.14 5.35± 0.21 7.10± 0.15 7.55± 0.13 8.05± 0.14 7.95± 0.12 7.28± 0.07 + REINFORCE++ 6.10± 0.18 7.55± 0.14 8.65± 0.12 5.55± 0.19 7.25± 0.13 7.60± 0.12 8.15± 0.13 8.05± 0.11 7.35± 0.06 + DMAPO 6.30± 0.16 7.65± 0.14 8.80± 0.11 5.70± 0.19 7.45± 0.12 7.60± 0.14 8.35± 0.13 8.15± 0.11 7.50± 0.05 Math category regression. DMAPO drops by 0.75 points in Math (5.70 vs. 6.45 for the base model). Two factors may contribute. First, the conciseness evaluator may penalize the long derivations rewarded by MT-Bench math questions. Second, mathematical prompts are sparse in the source data and may be further underrepresented after strict gating. DPO (−1.75-1.75) and SimPO (−0.95-0.95) regress as well, so this pattern is not unique to DMAPO. Diagnostic WR and perplexity. DMAPO has the highest Qwen-gated diagnostic WR (90.7± 1.2) while increasing perplexity by 0.68 relative to the base model; SimPO reaches 87.6% with a 2.71 increase. Both quantities describe fit to the gated signal, not independent response quality. Appendices D and E provide the full results and pipeline statistics. 6 Analysis 6.1 Why does quality gating work? The ablations and training dynamics suggest three complementary explanations. Reduced label ambiguity. On the Qwen-gated diagnostic, the No-Variance-Gate ablation performs slightly worse than the size-matched random subset (77.5% vs. 78.3%). Evaluator disagreement may therefore produce less reliable binary labels. Without human ground truth for the full candidate pool, however, this remains an interpretation rather than a demonstrated mechanism. Separable binary supervision. The gated groups differ substantially in evaluator score, and the trained model reaches a low loss and a large log-probability margin (Appendix G). Because KTO never observes the continuous scores, their separation cannot mechanically induce a reward gap. A narrower explanation is that strict gating removes ambiguous labels and thereby simplifies the binary learning problem. Independent evaluation is still needed to distinguish useful separation from memorization. On-policy relevance. Because candidates come from the target policy, the feedback focuses on outputs that the model can already produce. This reduces one form of response-distribution mismatch and exposes model-specific strengths and failures. The procedure aligns behavior within the base model’s support; it does not introduce knowledge or capabilities outside that support. 6.2 Ablation: gating strictness and process critic Table 4 varies the number of evaluators required for consensus and removes the process critic. In these runs, stricter gating retains fewer examples while improving MT-Bench and the Qwen-gated diagnostic. Removing the critic lowers MT-Bench from 7.50 to 7.24 and diagnostic WR by 3.2 points. The cap sweep is not monotonic beyond 0.15, so the evidence supports the critic as a bounded correction in this configuration, not as a universally optimal component. Table 4: Ablation: gating strictness and process critic (same KTO training recipe). Mean± over four seeds. Setting |||D| MT-B Qwen diag. WR% No gating 54,236 7.05± 0.09 72.1± 2.4 1 evaluator 12,847 7.12± 0.08 78.3± 2.1 2 evaluators 5,203 7.22± 0.06 84.5± 1.6 3 evaluators, no critic (α=0α=0) 2,043 7.24± 0.06 87.5± 1.5 3 evaluators + critic (DMAPO) 1,871 7.50± 0.05 90.7± 1.2 The acceptance-threshold sweep (Appendix H) peaks at the default 3.45% and degrades near 1%, indicating that aggressive filtering still requires a sufficient number of examples. Appendices I, F, and N report score distributions, training curves, and cross-dimension agreement. 6.3 Cross-objective use of the curated data To test whether the curated signal is useful beyond KTO, we train DPO, ORPO, and SimPO on within-prompt pairs derived from the same pool. Table 5: Cross-objective comparison using DMAPO-curated supervision. KTO uses 1,871 binary rows; pairwise methods use all 742 valid within-prompt desirable–undesirable pairs. Method |||D| MT-B ↑ AE-style LC% ↑ Qwen diag. WR% ↑ KTO (DMAPO) 1,871 rows 7.50± 0.05 95.5± 0.4 90.7± 1.2 DPO 742 pairs 7.35± 0.06 93.2± 0.5 88.5± 1.4 ORPO 742 pairs 7.32± 0.07 92.8± 0.6 87.9± 1.5 SimPO 742 pairs 7.38± 0.06 93.8± 0.5 88.9± 1.3 The pool contains 951 desirable and 920 undesirable rows, all of which KTO uses. Pairwise methods require both labels for the same prompt. Of the source prompts, 614 meet this condition; all valid within-prompt combinations yield 742 pairs. We do not construct cross-prompt pairs. Table 5 therefore tests portability across objectives, but not optimizer performance at equal sample counts, because both supervision format and dataset size differ. 6.4 Evaluator and rubric sensitivity Changing the rubric or evaluator family moderately changes which examples are selected (Table 6). Overlap with the default set ranges from 71.4% to 84.7%, whereas MT-Bench remains between 7.45 and 7.48, AE-style LC between 94.9 and 95.3, and IFEval between 56.6 and 57.0. Thus, the tested variants preserve downstream performance despite selecting different examples; the incomplete overlap also makes the evaluator dependence explicit. Table 6: Sensitivity to curation evaluator and rubric. Acceptance and overlap are percentages; overlap is measured against the default selected set. Filtering evaluator / rubric Accept. Overlap MT-B AE LC IFEval GPT-4o net vs. SimPO Qwen3-8B, default 3.45 100.0 7.50 95.5 57.3 +23.3 Qwen3-8B, paraphrased rubric 3.31 84.7 7.47 95.2 56.9 +20.2 Qwen3-8B, stricter factuality 3.58 82.1 7.48 95.3 57.0 +21.0 Llama-3.1-70B evaluator 3.82 71.4 7.45 94.9 56.6 +18.5 GPT-4o-mini evaluator 3.64 74.8 7.46 95.0 56.7 +19.4 6.5 Diversity and cross-source behavior Strict filtering narrows the selected set without collapsing it (Table 7). DMAPO retains 184 of 200 semantic clusters, compared with 187 for a same-size random subset; Distinct-2 falls modestly and self-BLEU rises. In a separate LMSYS-Chat pilot, the gate accepts 296 of 8,000 candidates (3.70%), close to the primary rate of 3.45%. Alongside the 200-prompt OOD evaluation, this pilot shows that the filtering rule can operate beyond the UltraFeedback–HelpSteer2 pool, although it does not replace a larger multi-source training study. Table 7: Diversity of the candidate and selected sets. Lower self-BLEU indicates greater diversity. Set Size Clusters Dist.-2 Self-BLEU Med. len. Full pool 54,236 200/200 .842 .382 219 Random-1.9k 1,871 187/200 .827 .401 213 DMAPO 1,871 184/200 .811 .414 201 6.6 Second backbone: Llama-3.1-8B-Instruct We repeat candidate generation and filtering with Llama-3.1-8B-Instruct (Grattafiori et al., 2024). The gate accepts 1,852 of 54,236 candidates (3.41%): 944 desirable and 908 undesirable rows. This nearly matches the Mistral acceptance rate. The performance result is mixed, however. DMAPO reaches 7.80 on MT-Bench, below the Llama base model’s 7.85, although it degrades less than the other preference-optimization methods and has the highest Qwen-gated diagnostic WR. The experiment therefore supports similar filtering behavior across the two backbones, not a general cross-backbone performance claim. Table 8: Results on Llama-3.1-8B-Instruct. Same pipeline and LoRA config. Method |||D| MT-B ↑ AE-style WR ↑ IFE ↑ Qwen diag. WR ↑ Base — 7.85 97.2 58.5 — + DPO 10k 7.55± 0.07 96.8± 0.4 59.2± 0.6 48.7± 2.3 + KTO 20k 7.68± 0.06 96.5± 0.3 58.8± 0.5 41.3± 1.9 + ORPO 10k 7.75± 0.06 97.0± 0.3 59.0± 0.5 58.9± 2.4 + SimPO 10k 7.72± 0.08 96.9± 0.4 59.0± 0.7 85.2± 1.6 + SPPO 60k 7.60± 0.07 96.7± 0.4 58.6± 0.6 70.4± 2.2 + REINFORCE++ 10k 7.70± 0.06 96.8± 0.3 59.1± 0.6 74.8± 2.0 + DMAPO 1,852 7.80± 0.05 96.6± 0.3 59.2± 0.5 89.5± 1.4 7 Limitations Evaluator dependence. Consensus does not guarantee correctness. DMAPO can inherit factual, stylistic, cultural, or verbosity biases shared by its evaluators. The sensitivity study and two independent pairwise evaluators probe this dependence but cannot eliminate it, particularly because the full candidate pool lacks human-verified quality labels. Scope of capabilities and data. On-policy filtering acts within the base model’s behavioral support; it should not be interpreted as adding new knowledge. The primary pool combines UltraFeedback and HelpSteer2, the LMSYS pilot is small, and the experiments do not establish reuse of a curated set across policy backbones. Reasoning regimes. Conciseness is one of three co-vetoing dimensions, and DMAPO obtains the highest MT-Bench Reasoning score in Table 3. Its Math score nevertheless falls from 6.45 to 5.70. The present method targets general instructions with short reasoning traces, not long-reasoning regimes. More representative mathematical data and reasoning-preserving objectives are natural extensions. Compute and training scope. DMAPO does not minimize total compute; it shifts compute from optimization to offline generation and evaluation. The pipeline uses 13.8 A100-hours, of which 0.6 is KTO training (Appendix K). Although the curated set can be reused across objectives, on-policy construction must be repeated for each backbone. The main experiments use LoRA. Appendix M reports a full-parameter SFT sanity check, but full-parameter preference optimization remains untested. Evaluation configuration. The AE-style evaluation uses a text-davinci-003 reference and is not the standard AlpacaEval 2.0 configuration. Training results report variation over four seeds, whereas the pairwise results are aggregate proportions without evaluator-sampling intervals. Preregistered human evaluation and broader benchmark coverage would strengthen the evidence. Reproducibility artifacts. Reproducing the filtering and diversity analyses requires the evaluator rubrics, pairwise-evaluation prompts, output parser, per-example decisions, and semantic-clustering configuration. These artifacts should accompany the submission because aggregate tables alone do not permit an independent audit. 8 Broader impact statement DMAPO can reduce preference-training data and optimization time, but it assigns substantial computational and decision-making responsibility to automated evaluators. Shared factual, cultural, or stylistic biases may then be amplified through high-confidence supervision. Useful safeguards include independent evaluation, evaluator-sensitivity analysis, human audits of selected examples, and publication of the curation prompts. In high-stakes settings, filtered responses still require domain-specific validation; passing the gate does not establish safety or correctness. Data considerations. The experiments use prompts from publicly released UltraFeedback, HelpSteer2, and LMSYS-Chat resources. Public availability does not ensure that every prompt is free of personal, sensitive, or offensive content, nor does it settle consent for downstream reuse. We do not attempt to infer user identities. Any release of selected examples should follow the source licenses, screen for identifying or harmful content, and provide a mechanism for applicable removal requests. 9 Conclusion DMAPO shifts attention from preference-loss design to preference-data construction. It samples responses from the target policy, retains examples that satisfy a multi-evaluator consensus gate, and trains a binary-label objective. On Mistral-7B, 1,871 selected rows improve MT-Bench, AE-style LC, IFEval, and independent pairwise comparisons. Evaluator and rubric variants change the selected set without materially changing downstream performance, and the filtered data retain broad semantic coverage. These results support consensus filtering in the evaluated general-instruction setting, but not the broader claim that less data is always better. The Llama result is mixed, offline curation is expensive, and the labels remain dependent on imperfect evaluators. Broader human validation, cross-source and cross-backbone reuse studies, and evaluation in long-reasoning settings are needed to establish where the approach is most useful. References Azar et al. (2024) Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. PMLR, 2024. Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023. Deng et al. (2025) Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving llm alignment via preference data selection. In Advances in Neural Information Processing Systems, volume 38, p. 161259–161285, 2025. Dubois et al. (2024) Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning, 2024. Grattafiori et al. (2024) Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 11170–11189, 2024. Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. Hu et al. (2025) Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: Stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262, 2025. Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. Meng et al. (2024) Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Qwen Team (2025) Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://qwenlm.github.io/blog/qwen3/. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 13484–13508, 2023. Wang et al. (2024) Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024. Wu et al. (2024) Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024. Xia et al. (2024) Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024. Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. Ye et al. (2025) Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025. Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Appendix A Full experimental setup Table 9: Complete experimental setup. Component Details Policy backbone (primary) Mistral-7B-Instruct-v0.2 (7.24B) Policy backbone (transfer) Llama-3.1-8B-Instruct (8.03B) Curation evaluator Qwen3-8B (/no_think mode) Independent evaluators GPT-4o; Claude Opus 4.7 LoRA rank / alpha 16 / 32 LoRA dropout 0.05 Target modules q, k, v, o, gate, up, down Trainable params ∼ 160M (2.2%) Optimizer AdamW (fused) Learning rate 5×10−55× 10^-5 LR schedule Cosine, 5% warmup Batch size 16 (2 × 8 grad accum) Precision bfloat16 Epochs 1 Seeds 42, 123, 456, 789 Hardware 8× A100 (80 GB) Software TRL 0.29.0, PEFT 0.18.1 Appendix B AlpacaEval-style detailed results Table 10: AlpacaEval-style evaluation: 805 instructions, pairwise against a text-davinci-003 reference. Raw win rate (WR) and length-controlled win rate (LC). Mean± over four seeds. Method WR (%) LC (%) Wins Ties Losses Avg. Len. Base 96.0 93.2 773 2 30 247 + SFT 86.1± 0.7 81.5± 0.9 693 3 109 312 + DPO 95.7± 0.4 92.4± 0.5 770 1 34 271 + KTO 95.5± 0.3 91.8± 0.4 769 4 32 278 + ORPO 96.3± 0.3 93.5± 0.4 775 1 29 255 + SimPO 95.5± 0.5 91.5± 0.6 769 2 34 285 + SPPO 95.9± 0.4 92.6± 0.5 772 1 32 268 + REINFORCE++ 96.1± 0.3 93.0± 0.4 774 1 30 258 + DMAPO 98.0± 0.3 95.5± 0.4 789 1 15 262 The length-controlled (LC) win rate (Dubois et al., 2024) adjusts for the effect of response length on annotator preference. The “Avg. Len.” column reports mean response length in tokens. SFT produces the longest responses (312) and the largest gap between raw and LC win rates (−4.6-4.6 points). DMAPO averages 262 tokens and has one of the smallest gaps (−2.5-2.5 points), so its advantage is unlikely to be explained by verbosity alone. Appendix C IFEval detailed results Table 11: IFEval: 541 prompts with verifiable constraints. Mean± over four seeds. Method Prompt Acc (%) Instr. Acc (%) Base 52.7 65.2 + SFT 52.3± 0.6 63.5± 0.7 + DPO 54.9± 0.7 66.5± 0.6 + KTO 54.0± 0.5 66.0± 0.5 + ORPO 54.3± 0.6 65.8± 0.6 + SimPO 54.0± 0.8 65.6± 0.7 + SPPO 53.5± 0.6 65.3± 0.6 + REINFORCE++ 53.8± 0.7 65.7± 0.5 + DMAPO 57.3± 0.5 67.2± 0.5 Appendix D Qwen-gated diagnostic evaluation and perplexity Table 12: Qwen-gated diagnostic evaluation on 129 validation prompts. Win rate compares log probabilities against the base model; labels use the same evaluator family as curation and are not independent. Perplexity is measured on 63 desirable responses. Method WR (%) ↑ PPL ↓ Δ Base — 6.43 — + SFT 32.6± 1.8 4.89± 0.08 −-1.54 + DPO 43.4± 2.1 7.10± 0.12 +0.67 + KTO 36.4± 1.5 7.50± 0.14 +1.07 + ORPO 55.8± 2.3 7.79± 0.11 +1.36 + SimPO 87.6± 1.4 9.14± 0.18 +2.71 + SPPO 72.8± 2.0 7.62± 0.13 +1.19 + REINFORCE++ 76.3± 1.9 7.45± 0.10 +1.02 + DMAPO 90.7± 1.2 7.11± 0.09 +0.68 Appendix E Pipeline statistics Table 13: Pipeline statistics. Of 14,272 source prompts, 13,559 training prompts produce 54,236 candidates and 1,871 gated examples (3.45% acceptance). The prompt-level train/validation split prevents overlap. Statistic Value Unique source prompts 14,272 Train-split prompts 13,559 (95%) Val-split prompts 713 (5%) Total candidates (train) 54,236 Total candidates (val) 2,852 Acceptance rate 3.45% Gated examples (train) 1,871 Gated examples (val) 129 Desirable / Undesirable (train) 951 / 920 Desirable / Undesirable (val) 63 / 66 Prompt overlap (train ∩ val) 0 Desirable mean score 9.23±1.099.23± 1.09 Undesirable mean score 2.42±1.102.42± 1.10 Quality gap ∼ 6.8 points Appendix F Cross-dimension gate agreement Table 14: Cross-dimension gate agreement on 54,236 candidates. Each evaluator’s score is binarized into pass (≥τ+=7≥ _+=7) versus fail (<τ+< _+); κ is computed on these binary labels. Because evaluators assess different dimensions, κ measures cross-dimension consistency rather than classical inter-rater reliability. Mean κ=0.613κ=0.613. Dimension Pair κ r Agree (%) Help.–Fact. 0.640 0.579 94.0 Help.–Conc. 0.642 0.544 92.4 Fact.–Conc. 0.556 0.461 91.2 Mean 0.613 0.528 92.5 Appendix G Training outcomes Table 15: Training outcomes. DMAPO’s low training loss indicates separation of the gated labels but does not establish generalization. Margin is the average log-probability difference. Mean± over four seeds. Method Final Loss Margin SFT 1.150± 0.012 — DPO 0.473± 0.009 1.88± 0.06 KTO 0.441± 0.011 1.81± 0.07 ORPO 0.413± 0.008 1.54± 0.05 SimPO 1.046± 0.015 40.76± 0.82 SPPO 0.385± 0.010 2.14± 0.08 REINFORCE++ 0.362± 0.009 2.35± 0.09 DMAPO 0.037± 0.004 10.96± 0.31 Appendix H Acceptance threshold ablation Table 16: Ablation: acceptance threshold. Mean± over four seeds. Accept. Rate |||D| MT-B Qwen diag. WR% 100% 54,236 7.05± 0.09 72.1± 2.4 ∼ 20% 10,847 7.15± 0.07 76.8± 2.0 ∼ 10% 5,424 7.20± 0.06 81.2± 1.8 ∼ 5% 2,712 7.25± 0.05 86.9± 1.5 3.45% (ours) 1,871 7.50± 0.05 90.7± 1.2 ∼ 1% 542 7.18± 0.08 85.4± 1.9 Appendix I Score distribution and training dynamics 112233445566778899101000.50.5111.51.5⋅104· 10^4Aggregate Score (1–10)Candidate CountDiscarded (n=52,365n=52,365)Desirable (n=951n=951)Undesirable (n=920n=920) Figure 2: Score distribution across 54,236 candidates. The confidence gate retains only the tails. 010102020303040405050606070708080909010010000.50.511Training Step (%)LossSFTDPOKTOSimPODMAPO Figure 3: Training loss curves. DMAPO’s loss drops rapidly to ∼ 0.037. Appendix J Qualitative examples Table 17: Qualitative examples of gated vs. discarded candidates. H = Helpfulness, F = Factuality, C = Conciseness. Prompt Response (excerpt) H F C s¯ s Var Gate “Explain opportunity cost.” “…is the value of the next-best alternative foregone. If you invest $50k in a car, the opportunity cost is $80k+ you’d earn investing at 5% over 10 years…” 9 9 8 8.67 0.33 Acc. “Health benefits of intermittent fasting?” “Many incredible benefits! Lose weight super fast, cure diabetes, prevent cancer, reverse aging, boost IQ by 20 points…” 6 3 7 4.88 4.33 Disc. “Photosynthesis process?” “Plants eat sunlight. Absorb through leaves, turn into food. That’s basically it.” 3 2 4 2.85 1.00 Unacc. Appendix K Compute cost Table 18: Reported A100 80 GB hours. DMAPO shifts compute to offline generation and scoring; its total is higher than the training-only baselines. Reuse is possible across objectives trained on the same policy-specific set. Method Generation Scoring Training Total Reusable? SFT (10k) — — 1.5 1.5 — DPO (10k) — — 2.0 2.0 — KTO (20k) — — 3.8 3.8 — ORPO (10k) — — 2.0 2.0 — SimPO (10k) — — 2.0 2.0 — SPPO (60k, 3 iter.) — — 12.0 12.0 — REINFORCE++ (10k) — 2.5 4.0 6.5 — DMAPO 6.4 6.8 0.6 13.8 ✓ Appendix L Process-critic cap sweep Table 19 varies the process-critic cap while holding the rest of the pipeline fixed. Performance improves through the default value of 0.15 and changes little at 0.20. The sweep supports 0.15 within the tested range, without implying that it is optimal in other settings. Table 19: Process-critic cap sweep. Diagnostic WR uses Qwen-gated validation labels. Critic cap MT-Bench Qwen diag. WR 0.00 (no critic) 7.24 87.5 0.05 7.37 88.8 0.10 7.46 90.2 0.15 (default) 7.50 90.7 0.20 7.48 90.4 Appendix M Full-parameter SFT sanity check The main comparisons use LoRA preference optimization. As a limited check, we run one epoch of full-parameter SFT on the 951 desirable DMAPO responses. It reaches 7.29 on MT-Bench, 93.7 AE-style LC, and 55.0 on IFEval, below DMAPO-KTO (7.50, 95.5, and 57.3). Because both the objective and parameterization change, this comparison isolates neither factor. It shows only that supervised imitation of the desirable subset does not reproduce the main result. Table 20: Full-parameter SFT sanity check on the desirable subset. Method MT-Bench AE-style LC IFEval Full-parameter SFT (951 rows) 7.29 93.7 55.0 DMAPO-KTO (LoRA, 1,871 rows) 7.50 95.5 57.3 Appendix N Evaluator score distributions Across 54,236 training candidates, the three evaluators produced the following score distributions: • Helpfulness: μ=5.50μ=5.50, σ=0.91σ=0.91 • Factuality: μ=5.49μ=5.49, σ=0.99σ=0.99 • Conciseness: μ=5.52μ=5.52, σ=1.00σ=1.00 • Inter-evaluator variance: mean = 0.45, median = 0.00, max = 27.00 The distributions are centered near the midpoint of the scale, and the gate retains only their tails. These descriptive statistics do not establish calibration against human judgments.