Paper deep dive
Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment
David Tsoi, Esra Dönmez
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 9/24/2026, 4:29:52 AM
Summary
This paper investigates steerable pluralistic alignment using Multi-Objective Direct Preference Optimization (MODPO). It addresses two key questions: predicting when objectives align or conflict, and covering trade-offs without retraining. The study finds that pre-training measurements (preference-pair conflict rate and SFT reference-model score correlation) predict objective compatibility for human-annotated data (HelpSteer) but fail for AI-annotated data (UltraFeedback) due to confounding factors like response length and repetition. For trade-off coverage, nearest-model selection and parameter merging of LoRA adapters are effective but do not consistently match direct training.
Entities (13)
Relation Signals (9)
MODPO → extends → DPO
confidence 95% · It extends Direct Preference Optimization (Rafailov et al., 2023, DPO;) with an objective weight that steers the balance between two objectives
repetition → confounds → reward-model scores
confidence 90% · where response length and repetition confound reward-model scores
response length → confounds → reward-model scores
confidence 90% · where response length and repetition confound reward-model scores
UltraFeedback → usedwith → Alpaca-7B
confidence 90% · We use ... Alpaca-7b (Taori et al., 2023) for UltraFeedback
HelpSteer → usedwith → Llama-2-7b
confidence 90% · We use Llama-2-7b (Touvron et al., 2023) for HelpSteer
preference-pair conflict rate → predicts → objective conflict
confidence 85% · two pre-training measurements predict whether objectives align or conflict for human-annotated data
SFT reference-model score correlation → predicts → objective alignment
confidence 85% · two pre-training measurements predict whether objectives align or conflict for human-annotated data
nearest-model selection → covers → trade-offs
confidence 80% · For broader trade-off coverage, selecting the nearest trained model and merging model parameters both help
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone. Pluralistic alignment therefore calls for steerable models that can balance competing objectives differently. Multi-Objective Direct Preference Optimization (MODPO) does this by using an objective weight to span a continuum of trade-offs. We study two questions: when can one model improve two objectives simultaneously, and how can many trade-offs be covered without training a separate model for each? Across seven objective pairs from HelpSteer and UltraFeedback, two pre-training measurements predict whether objectives align or conflict for human-annotated data, but not for AI-annotated data, where response length and repetition confound reward-model scores. For broader trade-off coverage, selecting the nearest trained model and merging model parameters both help, but neither consistently matches direct training. These findings yield practical guidance for building steerable models that serve diverse preferences.
Tags
Links
- Source: https://arxiv.org/abs/2609.26929v1
- Canonical: https://arxiv.org/abs/2609.26929v1
Trouble viewing inline? Open PDF directly →
Full Text
68,511 characters extracted from source content.
Expand or collapse full text
Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment David Tsoi Affiliation: Department of Computer Science, University of Stuttgart Email: david-alexander.tsoi@ims.uni-stuttgart.de Esra Dönmez Affiliation: Institute for Natural Language Processing, University of Stuttgart Affiliation: Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart Email: esra.doenmez@ims.uni-stuttgart.de Abstract People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone. Pluralistic alignment therefore calls for steerable models that can balance competing objectives differently. Multi-Objective Direct Preference Optimization (MODPO) does this by using an objective weight to span a continuum of trade-offs. We study two questions: when can one model improve two objectives simultaneously, and how can many trade-offs be covered without training a separate model for each? Across seven objective pairs from HelpSteer and UltraFeedback, two pre-training measurements predict whether objectives align or conflict for human-annotated data, but not for AI-annotated data, where response length and repetition confound reward-model scores. For broader trade-off coverage, selecting the nearest trained model and merging model parameters both help, but neither consistently matches direct training. These findings yield practical guidance for building steerable models that serve diverse preferences. 1 Introduction People do not share one set of values. They disagree about what makes a response good, and the qualities they care about trade off against one another: a longer answer can be more helpful but less concise, and a maximally cautious answer can be less useful. When we align a model, these qualities become objectives we optimize, and different stakeholders, holding different values, weight the same objectives differently over the same preferences. Aligning to a single averaged preference hides this diversity and picks winners among objectives that people would weight differently (Bakker et al., 2022; Plank, 2022). Pluralistic alignment asks instead how one system can serve a plurality of stakeholders (Sorensen et al., 2024). One practical route is steerable pluralism: rather than fixing one compromise, build a model that can be steered to weight competing objectives differently (Sorensen et al., 2024). Multi-Objective Direct Preference Optimization (Zhou et al., 2024, MODPO;) is an efficient way to build such a model. It extends Direct Preference Optimization (Rafailov et al., 2023, DPO;) with an objective weight that steers the balance between two objectives, such as helpfulness, honesty, or safety. Training at a range of weights yields a family of models spanning a continuum of trade-offs, so a stakeholder can pick the compromise their context calls for. Two questions decide whether MODPO can be used this way in practice, and neither is answered by reporting scores at a single weight. RQ1: When can one model satisfy two objectives at once? Some objectives reinforce each other and some conflict. If MODPO can improve two objectives together, a practitioner should train one model that serves both; if they conflict, they must instead steer along a trade-off and choose a point on it. We ask whether this can be predicted before training from properties of the preference data and the base model. We study two cheap measurements: the preference-pair conflict rate, how often the two objectives disagree on which response is better (Li et al., 2025), a direct measure of that disagreement in the data (Plank, 2022); and the SFT reference-model score correlation, whether the SFT reference model’s better answers for one objective are also better for the other. RQ2: How can we cover many trade-offs without training a model for each? Serving a continuum of stakeholder preferences should not require training one model per compromise. Given a family of MODPO models already trained at a few weights, we ask how closely two training-free reuse methods match a model trained directly at a new target weight: nearest-model selection, using the trained model whose weight is closest to the target, and parameter merging, blending the LoRA adapters of the two neighbouring trained models (Jang et al., 2024; Rame et al., 2023). Contributions. (i) We frame MODPO as a tool for steerable pluralistic alignment and ask, concretely, when it can satisfy two objectives jointly and when it must trade them off. (i) We test two pre training measurements of objective compatibility across seven objective pairs. They predict the outcome well for the human-annotated HelpSteer data but fail for the AI-annotated UltraFeedback data. (i) We trace that failure to a sociotechnical cause: the apparent conflict is strongly confounded by response length and repetition in the reward-model scores, a warning for how objective behavior is evaluated. (iv) We show that nearest-model selection and parameter merging can cover the trade-off continuum without retraining, but neither reliably matches direct training. We distill each finding into a practical recipe. 2 Related Work Pluralistic alignment. A growing line of work argues that alignment should reflect diverse and potentially conflicting human values rather than collapse them into a single averaged preference (Sorensen et al., 2024; Conitzer et al., 2024; Kirk et al., 2024). This concern is reinforced by evidence that reward-model quality varies across socially consequential domains and that reward models can systematically prefer socially undesirable responses (Ghazaryan and Dönmez, 2026). Sorensen et al. (2024) distinguish several forms of pluralism; we focus on the steerable setting, in which a single model can be adjusted to weight its objectives differently for different stakeholders. Existing approaches include training on population-level agreement (Bakker et al., 2022), optimizing max-min objectives over diverse preferences (Chakraborty et al., 2024), and coordinating multiple LLMs (Feng et al., 2024). We study MODPO as a concrete and efficient approach to steerable pluralism, asking when a single model can satisfy two objectives jointly and when those objectives instead require a trade-off. Steering multiple objectives. DPO (Rafailov et al., 2023) aligns a model to preferences without a separate reward model or reinforcement learning. Several methods extend alignment to several objectives with a controllable balance: MODPO folds the balance into the DPO loss (Zhou et al., 2024); directional preference alignment attaches a preference direction to a multi-objective reward (Wang et al., 2024a); Rewards-in-Context conditions on the balance in the prompt (Yang et al., 2024); and controlled or multi-objective decoding sets it at generation time (Mudgal et al., 2024; Shi et al., 2024). These works report scores at chosen weights; we instead ask whether the outcome at each weight can be predicted before training. Disagreement in preference data. Human annotators disagree, and this variation is signal, not just noise (Plank, 2022; Bakker et al., 2022). For multiple objectives, Li et al. (2025) name preference conflict (the two objectives prefer different responses for the same pair), show it is common, and reduce it by rewriting responses; Xu et al. (2026) study a related reward consistency. We do not try to remove conflict. We measure it, and whether it and a base-model correlation predict which objectives MODPO can jointly satisfy. Reusing and merging models. Weight-space methods combine trained models without retraining: Model Soups (Wortsman et al., 2022), Task Arithmetic (Ilharco et al., 2023), and AdapterSoup (Chronopoulou et al., 2023). For alignment specifically, Rewarded Soups (Rame et al., 2023) and Personalized Soups (Jang et al., 2024) merge models trained on different rewards to reach new preference balances. Our parameter-merging method blends the LoRA adapters (Hu et al., 2022) of the two trained MODPO models nearest a target weight, and we compare it against nearest-model selection and direct training. Evaluating with reward models. Reward models enable scalable, consistent scoring across objectives, and multi-objective models such as ArmoRM (Wang et al., 2024a) let us evaluate both objectives on the same generations. Because scores can be influenced by proxy optimization, response length, category, and repetition (Gao et al., 2023; Singhal et al., 2024; Dubois et al., 2024; Zheng et al., 2023; Lambert et al., 2025; Welleck et al., 2020), we report length and repetition alongside reward scores and re-decode with repetition control. We show that these checks are important for interpreting whether two objectives are genuinely compatible. 3 Background We align to two objectives at a time and always fix the first as helpfulness. Preference data and conflict. A response pair (x,yi,yj)(x,y_i,y_j) is a prompt x with two responses. Each objective o scores responses; the higher-scoring response is preferred, giving a preference pair (x,yw,yl)(x,y_w,y_l) with preferred ywy_w and dispreferred yly_l (ties are dropped). One response pair thus yields one preference pair per objective, and the two need not agree. When two objectives prefer different responses for the same pair, they are in conflict (Li et al., 2025). Over a dataset, the fraction of jointly labelled pairs that conflict is the preference-pair conflict rate. It is a property of the data, measures how much the two objectives disagree, and is computed before any training. SFT reference model. Preference optimization does not start from the pretrained model directly: it starts from a reference model πsft _sft, obtained by supervised fine-tuning (SFT), i.e. plain next-token likelihood training on demonstration responses (here, following Zhou et al. (2024), the preferred responses of the helpfulness preference pairs). SFT uses no preference signal: it never sees the dispreferred response and never sees the second objective. This one model plays three roles in what follows. It initializes every MODPO run; it is the fixed anchor MODPO keeps the trained policy close to (via β below); and it is the model whose generations we score before training in §4.1. Because all trained models share it as a starting point, it is also the baseline that their scores are read against: a model that already answers in the dataset’s domain and format, but has not yet been optimized for either objective of a pair. DPO. DPO treats a model as a policy πθ(y∣x) _θ(y x) and trains it against the reference policy πsft _sft, which stays frozen (Rafailov et al., 2023). Writing Δθ(x,y)=logπθ(y∣x)−logπsft(y∣x) _θ(x,y)= _θ(y x)- _sft(y x) and the preference margin dθ=Δθ(x,yw)−Δθ(x,yl)d_θ= _θ(x,y_w)- _θ(x,y_l), the loss is ℒDPO=−(x,yw,yl)[logσ(βdθ)],L_DPO=-\,E_(x,y_w,y_l) [ σ(β\,d_θ) ], (1) where σ is the logistic function and β limits how far the policy drifts from the reference. DPO optimizes one objective. MODPO. MODPO trains one policy for a weighted mix of two objectives (Zhou et al., 2024). An objective weight w∈(0,1]w∈(0,1] sets the balance: the preference pairs come from the first objective (helpfulness), while the second enters through a margin mϕ=rϕ(x,yw)−rϕ(x,yl)m_φ=r_φ(x,y_w)-r_φ(x,y_l) from a reward model rϕr_φ trained on it. The DPO logit becomes z=1w(βdθw−(1−w)mϕ),z= 1w (β\,d_ _w-(1-w)\,m_φ ), (2) with loss −[logσ(z)]-E[ σ(z)]. Each weight w gives a separate model at a known point on the trade-off, so a set of weights is a steerable family. Flipping the sign of the margin reverses which direction of the second objective is preferred (e.g. shorter vs. longer responses). LoRA. We train with Low-Rank Adaptation (Hu et al., 2022): the pretrained weights W0W_0 are frozen and a low-rank update ΔW=BA W=BA is learned, applied as h=W0u+(α/r)BAuh=W_0u+(α/r)\,BA\,u. A trained model is then the base model plus a small adapter, so storing many objective-weight variants is cheap and their adapters can be merged (§4). 4 Method We run two studies11 1 https://github.com/esradonmez/modpo-objective-trade-off. The trade-off study (RQ1) asks whether pre-training measurements predict which MODPO models improve two objectives together and which trade them off. The coverage study (RQ2) asks how well trained models can be reused to reach new trade-offs without retraining. We report two kinds of score and keep them distinct: dataset scores, which ship with the data and are used only to build preference pairs, and reward-model scores, which the evaluator assigns to generated responses and which we compare. 4.1 Measuring objective compatibility before training For each objective pair we compute two cheap measurements before any MODPO training. Preference-pair conflict rate. Over the training data, we take the response pairs that yield a preference pair for both objectives and measure how often the two prefer different responses (§3). This is a direct count of how often the objectives disagree in the data. A high rate means they pull toward opposite responses, which is what produces the conflicting gradients that can force a trade-off. SFT reference-model score correlation. We generate one response per evaluation prompt from the reference model, score it for both objectives, and take the Pearson correlation of the two score vectors. The correlation is high when the reference model’s higher-scoring responses for one objective are also its higher-scoring responses for the other. We compute it on generated responses with the same prompts and evaluator later used for the MODPO models, so it is comparable to their scores. A correlation of the raw dataset scores would not be, since those come from a different annotator (humans for HelpSteer, GPT-4 for UltraFeedback). A high correlation and a low conflict rate both point to objectives that move together, so we expect them to agree. The interesting cases are where they do not, and testing those is crucial for understanding alignment objectives. 4.2 Probing the conflict rate To test the conflict rate more directly, we split the jointly labelled pairs into a non-conflict set (rate 00) and a conflict set (rate 11), keep the helpfulness preference pairs fixed, and retrain only the second-objective margin reward model on each subset. Because the subsets also differ in size and composition, we read these results as descriptive, not causal. 4.3 Covering the trade-off continuum at inference time A target trade-off is an objective weight w⋆w chosen after training. We compare two training-free reuse methods against a model trained directly at w⋆w . Nearest-model selection. From a set G of trained weights, pick wg=argminw∈G|w−w⋆|w_g= _w∈ G|w-w | (ties broken toward the lower weight) and generate with that model. We compare a fine set (more weights) and a coarse set (fewer). Parameter merging. Using the fine set, take the closest lower and upper trained weights wℓ<w⋆<wuw_ <w <w_u, set λ=(w⋆−wℓ)/(wu−wℓ)λ=(w -w_ )/(w_u-w_ ), and merge their LoRA adapters tensor by tensor, Θmerge=(1−λ)Θℓ+λΘu _merge=(1-λ)\, _ +λ\, _u (Jang et al., 2024; Rame et al., 2023). No new training is done. Merging is an interpolation heuristic, not an optimization procedure for w⋆w : the MODPO objective at w⋆w is never evaluated, so nothing guarantees the merged adapter behaves like one trained there. How closely it does is what we measure. 5 Experimental Setup Datasets. We use two datasets that score more than two objectives so that several objective pairs share one dataset. HelpSteer (Wang et al., 2024c) provides human-assigned scores (00–44) for helpfulness, correctness, coherence, complexity, and verbosity over 10,45910,459 prompts. UltraFeedback (Cui et al., 2024) provides GPT-4-assigned scores (11–55) for helpfulness, honesty, instruction-following, and truthfulness over 63,96763,967 prompts. The coverage study uses PKU-SafeRLHF-10K (Ji et al., 2023; Dai et al., 2024) with the helpfulness/harmlessness pair (an objective pair where the trade-off is real and most consequential) reproducing the MODPO safety setup so our numbers can be checked against Zhou et al. (2024). Objectives. We fix helpfulness as the first objective in every pair and choose the second objectives to span ones that plausibly reinforce helpfulness and ones that plausibly compete with it: from HelpSteer, correctness and coherence should reinforce it, complexity is more independent, and verbosity can compete with it (a concise answer may still be helpful); from UltraFeedback, honesty, instruction-following, and truthfulness are broadly aligned with helpfulness but need not coincide with it response by response. This gives four HelpSteer pairs and three UltraFeedback pairs; for HelpSteer verbosity we train once toward higher and once toward lower verbosity (positive / negative margin sign), for eight settings in all. The spread lets us test the measurements on pairs expected to agree and pairs expected to conflict, rather than on a single case. Models and training. We use Llama-2-7b (Touvron et al., 2023) for HelpSteer and Alpaca-7b (Taori et al., 2023) for UltraFeedback and PKU-SafeRLHF, all trained with LoRA. Following Zhou et al. (2024), each setting trains a margin reward model on the second objective and one MODPO model per objective weight. In the trade-off study we additionally train an SFT reference model on the preferred responses of the helpfulness pairs and use weights w∈0.1,0.2,0.4,0.6,0.8,(0.9)w∈\0.1,0.2,0.4,0.6,0.8,(0.9)\. The coverage study reuses the Alpaca-7b checkpoint as its reference model without further SFT, again following Zhou et al. (2024), and trains the fine weight set 0.0,0.2,0.4,0.6,0.8,1.0\0.0,0.2,0.4,0.6,0.8,1.0\, the coarse set 0.0,0.5,1.0\0.0,0.5,1.0\, and direct models at the target weights w⋆∈0.1,0.3,0.5,0.7,0.9w ∈\0.1,0.3,0.5,0.7,0.9\; at w=0.0w=0.0 all weight is on harmlessness, so the DPO margin adapter is loaded instead of a MODPO adapter. We report the 2048-token training-length limit as primary (it retains almost all pairs) and contrast a 512-token limit that retains far fewer. Training uses learning rate 10−410^-4, β=0.1β=0.1, AdamW with a cosine schedule, and a single seed per model; full hyperparameters are in Appendix A. Evaluation. We generate one greedy response per prompt on 400400 held-out prompts (trade-off study) or 700700 response pairs (coverage study). Trade-off responses are scored per objective with ArmoRM (Wang et al., 2024b); PKU responses are scored with the Beaver reward (helpfulness, higher better) and cost (harmlessness, lower better) models (Dai et al., 2024). We report mean scores, mean response length in words, and the share of responses with a repeated 44-gram, with 95%95\% paired bootstrap confidence intervals (Efron and Tibshirani, 1993). Because greedy decoding can produce long, repetitive text, we additionally re-decode the 2048-token models with a repetition penalty and 44-gram blocking to test whether score changes track length and repetition. Figure 1: The two pre-training measurements for all seven objective pairs (2048-token limit; both verbosity margin signs are plotted, giving eight points). Objectives that move together sit toward the bottom right (high correlation, low conflict); HelpSteer correctness and coherence are the clearest such cases, while HelpSteer verbosity toward concision (HS verb. neg.) has by far the highest conflict rate. The two measurements agree for the three UltraFeedback pairs too: high correlation and a conflict rate below every HelpSteer pair but correctness and coherence. Verbosity is where the two measurements part company; UltraFeedback is where the results will. Objective pair Corr. Conflict Both↑ ? HelpSteer correctness 0.9810.981 0.0270.027 all w coherence 0.9110.911 0.0670.067 all w complexity 0.0950.095 0.2020.202 2 of 5 w verbosity (++) −0.162-0.162 0.2260.226 w=0.9w=0.9 verbosity (−-) 0.1620.162 0.7740.774 none† UltraFeedback honesty 0.8540.854 0.1810.181 none instruction-following 0.8790.879 0.1640.164 none truthfulness 0.8100.810 0.1920.192 none Table 1: Pre-training measurements (SFT reference-model score correlation and preference-pair conflict rate, 2048-token limit) and at which objective weights both objectives improve over the reference under greedy decoding at the same limit. †verbosity(−-) improves both only at w=0.4w=0.4 under the 512-token limit. Appendix B gives the 512-token measurements and Appendix C the underlying scores. HelpSteer follows the measurements; the high-correlation UltraFeedback pairs do not. 6 Results 6.1 When can one model satisfy two objectives? (RQ1) The measurements work for HelpSteer. Table 1 and Figure 1 display the two pre-training measurements; Figure 2 shows the resulting MODPO scores for one pair from each dataset. For HelpSteer the measurements match both intuition and each other. Correctness and coherence have very high correlations (0.980.98, 0.910.91) and low conflict rates (0.030.03, 0.070.07). Under the 2048-token limit, MODPO improves both objectives at every weight (the “all w” entries in Table 1): in Figure 2 (left) every 2048-token model sits up and to the right of the reference, with correctness rising from 0.6360.636 to 0.7060.706 at w=0.1w=0.1 as helpfulness rises from 0.6210.621 to 0.7020.702; the 512-token condition shows weaker and less consistent gains. Complexity has a weak correlation (0.0950.095) and moderate conflict (0.2020.202): it improves at every weight but helpfulness only at two, a partial match. Verbosity toward concision has the highest conflict rate (0.7740.774) and shows a clean trade-off: low weights favor concision, high weights favor helpfulness, exactly as the conflict rate predicts. The measurements fail for UltraFeedback. The three UltraFeedback pairs have high correlations (0.810.81–0.880.88) and only moderate conflict, yet every model degrades performance on the second objective at every weight and never improves both. Figure 2 (right) highlights the contrast: every UltraFeedback model scores below the reference on truthfulness. A high pre-training correlation did not deliver the joint improvement it did for HelpSteer. Figure 2: MODPO scores (ArmoRM mean over 400 prompts) for an objective pair that MODPO can satisfy jointly and one it cannot, at two training-length limits (512, 2048). Empty squares mark the reference model; labels are objective weights w. Left: HelpSteer correctness. Under the 2048-token limit every MODPO model improves both objectives relative to its reference model; the 512-token condition shows weaker and less consistent gains. Right: UltraFeedback truthfulness. Despite a high pre-training correlation, every model scores lower on truthfulness than the reference. The gap between the 512 and 2048 clouds shows how much the training-length limit moves the scores. The apparent conflict is strongly confounded by length and repetition. Almost all MODPO models generate much longer and more repetitive responses than the reference (up to ∼970 970 words with a repeated-44-gram share above 0.950.95 for UltraFeedback). When we re-decode the same 2048-token models with repetition control, mean length roughly halves and the repeated-44-gram share drops by 6060–7070 points (Table 2; 38.738.7–81.681.6 across individual pairs, Appendix D). For HelpSteer this barely moves the scores. For UltraFeedback both objective scores rise by +0.10+0.10 to +0.15+0.15, and the resulting scores are substantially more consistent with the high reference-model correlations. This does not isolate the cause: re-decoding changes length, repetition, wording, and content together, and the same reward model scores both conditions, so we cannot separate a reward-model bias against repetition from genuinely better responses. What it does show is that the apparent UltraFeedback conflict is strongly confounded by generation length and repetition, and was driven at least partly by decoding behavior rather than by objective incompatibility alone; an independent evaluator would be needed to settle this. A plausible contributing factor is that UltraFeedback’s GPT-4 scores track length more strongly than HelpSteer’s human scores, and that HelpSteer separates verbosity into its own objective. The training-length limit adds to this: keeping more pairs shortens HelpSteer generations but lengthens UltraFeedback ones, which is why the same limit helps one dataset and hurts the other (Figure 2). Δ Δ 2 Δ Δ HelpSteer (mean) −0.001-0.001 −0.008-0.008 −48.8%-48.8\% −69.7-69.7 UltraFeedback (mean) +0.137+0.137 +0.125+0.125 −58.9%-58.9\% −62.7-62.7 Table 2: Change in scores, response length, and repeated-44-gram share (percentage points) when the 2048-token models are re-decoded with repetition control. HelpSteer is unmoved; UltraFeedback rises sharply, showing its greedy-decoding scores were confounded by length and repetition. The margin sign steers which direction of an objective wins. By Eq. (2), flipping the sign of the margin reverses which direction of the second objective is preferred. For verbosity this is the difference between steering toward longer and toward shorter responses. This is the explicit, per-objective control that steerable pluralism calls for. Figure 3 shows the positive sign: MODPO reliably raises verbosity but helps helpfulness at a single weight (w=0.9w=0.9, the one setting where this pair improves both), while the negative sign gives the clean trade-off above. The negative sign is also the pair with the highest conflict rate, so its trade-off is what the measurement predicts. Direction is therefore a second control knob, orthogonal to the weight. Figure 3: HelpSteer helpfulness–verbosity under the positive margin sign (steering toward higher verbosity). MODPO raises verbosity at every weight but improves helpfulness only at w=0.9w=0.9, where helpfulness receives the greatest weight and verbosity the least. Flipping the sign steers toward concision and yields the trade-off in the text. Splitting on conflict changes the outcome but does not decide it. Training the margin reward model only on conflicting pairs shifts the results but not all the way to a trade-off. For UltraFeedback instruction-following (the lowest conflict rate of the three UltraFeedback pairs; 512-token limit), the conflict set comes close to a trade-off but stops short, because the pair still has a high correlation. HelpSteer verbosity toward concision points the same way from the other side: the split is computed in the dataset’s score direction, so under the negative margin sign the “conflict” set is the one whose pairs agree with helpfulness, and it duly improves helpfulness at every weight while worsening concision at the higher ones. Conflict alone is thus not enough; the correlation still matters, in line with Li et al. (2025). Full scores are in Appendix E. 6.2 Covering the continuum without retraining (RQ2) Across the trained models, mean reward increases and mean harmlessness worsens as w rises, an empirical trade-off frontier in this single-seed run (Figure 4), consistent with Zhou et al. (2024). No single model dominates, so reaching a new target is non-trivial. Table 3 and Figure 5 report the mean absolute difference from a model trained directly at the target. Parameter merging has the smallest average gap when reward and cost are considered together (combined mean 1.141.14, vs. 1.691.69 for fine selection and 2.402.40 for coarse), although it is not consistently closest at individual target weights and is slightly worse than fine selection on mean reward gap (1.111.11 vs. 1.061.06). Its 95%95\% confidence interval for the paired difference includes zero in 55 of 1010 comparisons, vs. 2/102/10 for fine selection: evidence of no detected difference, not of equivalence. Merging is an interpolation heuristic, not an optimization procedure for the target weight: its score trajectory did not reach the directly trained model at w⋆=0.9w =0.9, producing a large reward gap, while the selection methods instead spike at the w⋆=0.1w =0.1 endpoint. Nearest-model selection is limited by design: it picks on weight distance, not on scores, so the nearest weight is often not nearest in reward or cost. Both methods cover the continuum usefully, but neither reliably reproduces a directly trained model. Figure 4: Reward vs. cost for the fine weight set on PKU-SafeRLHF (labels are objective weights; the w=0w=0 point is the DPO margin adapter, since at w=0w=0 all weight is on harmlessness). Higher reward is more helpful, lower cost is more harmless. Across these trained models mean reward increases and mean cost worsens with w (a single-seed run), so no single model dominates and reaching a new target requires selecting or merging models. Figure 5: Absolute score difference from a directly trained target model at each target weight (lower is closer). Parameter merging (interpolated model, dashed) has the smallest combined gap but is not uniformly best: it is slightly worse than fine selection on mean reward, the selection methods spike at the w⋆=0.1w =0.1 endpoint, and merging shows a large reward gap at w⋆=0.9w =0.9. Merging is an interpolation heuristic, not an optimization procedure for the target weight. Fine Coarse Merge Reward gap (mean∗) 1.061.06 2.012.01 1.111.11 Cost gap (mean∗) 2.322.32 2.792.79 1.181.18 CI incl. 00 2/102/10 1/8†1/8 /5/10 Table 3: Coverage study: mean absolute difference from direct training in Beaver reward and cost, and how many comparisons have a 95%95\% confidence interval for the paired difference that includes zero (no detected difference, not equivalence). Fine selection has the smallest mean reward gap; merging the smallest cost gap and combined gap. ∗mean over target weights excluding w⋆=0.5w =0.5, where the coarse set already contains the direct model. †the coarse count is out of eight for the same reason: its two comparisons at w⋆=0.5w =0.5 are zero by construction. Per-target values are in Appendix F. 7 Discussion Cheap measurements predict compatibility, but only for trustworthy data (RQ1). The conflict rate and the reference-model correlation predict MODPO’s behavior well on human-annotated HelpSteer: aligned objectives (correctness, coherence) improve together, and the most disagreeing pair (concision vs. helpfulness) trades off. They fail on AI-annotated UltraFeedback, but the failure is instructive. Rather than a clean objective incompatibility, the reward-model scores were strongly confounded by length and repetition, and once we controlled for those the results became substantially more consistent with the measurements. We cannot conclude from this alone that the objectives are compatible; that would take an independent evaluator. For steerable pluralism this is a warning: whether two objectives look compatible can depend heavily on how outputs are decoded and scored, so a system that decided which objectives to serve from a cheap measurement alone could draw the wrong line between compatible and conflicting objectives. Recipe for RQ1. To serve two objectives together, pick a pair with high correlation and low conflict, use the training-length limit that keeps the most pairs, and then check whether the models generate very long or repetitive text and whether a different decoding setting changes the scores. To build a steerable trade-off, pick a pair with low correlation and high conflict, and use the margin sign to choose which direction of the objective to steer toward. Coverage is cheap but approximate (RQ2). A continuum of trade-offs can be covered by reusing a few trained models: parameter merging has the smallest combined gap from direct training and nearest-model selection is a simple fallback. Neither matches direct training: selection ignores scores, and merging is an interpolation heuristic rather than an optimization procedure for the target weight. This mirrors the trade-off in the wider merging literature (Jang et al., 2024; Rame et al., 2023): merging buys cheap coverage of the preference space at some cost in fidelity. Recipe for RQ2. Train a family of MODPO models across a range of weights. If it already covers the trade-offs you need, use nearest-model selection; otherwise consider filling the gaps by merging the two nearest models. When a target’s exact behavior matters, train directly at that weight. When is a trade-off the right frame? Not every pair of objectives needs a steerable trade-off. Different stakeholders weight objectives differently, but when the objectives themselves reinforce each other (as correctness and helpfulness do here), a single well-trained model serves both, and offering a dial would be a false choice. A trade-off should be exposed only when the objectives genuinely compete for the same responses, which is what a high conflict rate, read together with the correlation and after controlling for length and repetition confounds, is meant to detect. This suggests a modest but useful role for the measurements in pluralistic systems: not to make the choice automatically, but to flag which pairs are candidates for steerable trade-offs and which are better served jointly, leaving the normative choice of which trade-offs to expose to the people affected (Dönmez and Falenska, 2026). Takeaway. An objective weight is a genuine steering dial, but it is not a promise: the same weight behaves differently depending on the data kept and the decoding used. Steerable pluralism with MODPO therefore needs its models characterized empirically by their generated length, repetition, and scored behavior, not just by their nominal weights. 8 Conclusion We studied MODPO as a tool for steerable pluralistic alignment (one model family that can be steered across a continuum of objective trade-offs to suit different stakeholders) and asked when it can satisfy two objectives at once and how to cover the continuum cheaply. Across seven objective pairs, two cheap pre-training measurements predicted MODPO’s behavior for human-annotated HelpSteer but not for AI-annotated UltraFeedback, where response length and repetition strongly confounded the reward-model scores; controlling for them made the results much more consistent with the measurements, though an independent evaluator would be needed to confirm compatibility. Whether two objectives can be served jointly is therefore not something to read off a single number: it depends on the data, the decoding, and the evaluator together. For covering new trade-offs without retraining, nearest-model selection and parameter merging both help, but neither reliably matches direct training. We hope the resulting recipes help practitioners build models that serve diverse stakeholder preferences, and measure honestly whether they do. Limitations Our study has several limitations. (i) We steer only two objectives at a time (helpfulness and a second objective); behavior with three or more as well as with different pairings may differ, and serving many stakeholders often means balancing many objectives at once. (i) The coverage study reuses models within a fixed trained set and a fixed set of target weights, and parameter merging is tested only at the midpoint λ=0.5λ=0.5; other targets and merge positions are untested. (i) All trained-model results use a single seed, so we cannot quantify seed variation. (iv) All scores come from reward models, a proxy for human judgment. Every comparison we draw is within-evaluator: each MODPO model is read against the SFT reference trained on the same data and scored by the same model, with paired confidence intervals, so a constant evaluator bias shifts the score levels rather than the direction of a comparison. What this does not cover is absolute level and transfer, since ArmoRM is trained on data that includes HelpSteer and UltraFeedback and the Beaver models on the BeaverTails family that includes PKU-SafeRLHF. The one result that turns on the evaluator itself is the re-decoding comparison (§6.1), where generation and scoring change together; we mark it as unsettled there. (v) We use one 7B base model per dataset, so results may transfer differently to other models or sizes. (vi) The conflict-split experiments change the training-set size and composition, so they are suggestive of a conflict-rate effect rather than causal. Promising next steps are a second (human or LLM-judge) evaluator to confirm objective scores independently of length and repetition, which matters most for UltraFeedback; a size-controlled study of the conflict rate; and a selection rule 2that uses reward and cost scores rather than weight distance alone. Ethics Statement This work studies how to balance competing objectives in a single steerable model, using only publicly available datasets and models. Part of our motivation is pluralistic: making the trade-off between objectives (which different stakeholders would weight differently) explicit and inspectable, rather than hiding it inside one aggregate reward. We note two risks. First, steerability cuts both ways: the same objective weight that balances helpfulness against harmlessness can be pushed toward helpfulness to produce less safe outputs (Figure 4), and our PKU-SafeRLHF setup includes adversarial prompts. Deciding which trade-offs a deployed system should expose is a governance question beyond our scope. Second, reward-model scores carry the biases of their training data and, as we show, of response length and repetition, so they should not be read as ground-truth measures of any human value (Ghazaryan and Dönmez, 2026). We report these confounds openly and recommend human or independent evaluation before deployment. Acknowledgments We acknowledge the support of the Ministerium für Wissenschaft, Forschung und Kunst BadenWürttemberg (MWK, Ministry of Science, Research and the Arts Baden-Württemberg under Az. 33-7533-9 19/54/5) in Künstliche Intelligenz & Gesellschaft: Reflecting Intelligent Systems for Diversity, Demography and Democracy (IRIS3D) and the support by the Interchange Forum for Reflecting on Intelligent Systems (IRIS) at the University of Stuttgart. We would like to thank the Institute of Natural Language Processing (IMS) for providing computational support for this project. References Bakker et al. (2022) M. A. Bakker, M. J. Chadwick, H. R. Sheahan, M. H. Tessler, L. Campbell-Gillingham, J. Balaguer, N. McAleese, A. Glaese, J. Aslanides, M. M. Botvinick, and C. Summerfield Fine-tuning language models to find agreement among humans with diverse preferences. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088, Link Cited by: §1, §2, §2. Chakraborty et al. (2024) S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, D. Manocha, F. Huang, A. Bedi, and M. Wang MaxMin-RLHF: alignment with diverse human preferences. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §2. Chronopoulou et al. (2023) A. Chronopoulou, M. Peters, A. Fraser, and J. Dodge AdapterSoup: weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023, A. Vlachos and I. Augenstein (Eds.), Dubrovnik, Croatia, p. 2054–2063. External Links: Link, Document Cited by: §2. Conitzer et al. (2024) V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mosse, E. Pacuit, S. Russell, H. Schoelkopf, E. Tewolde, and W. S. Zwicker Position: social choice should guide AI alignment in dealing with diverse human feedback. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 9346–9360. External Links: Link Cited by: §2. Cui et al. (2024) G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun ULTRAFEEDBACK: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. External Links: Link Cited by: §5. Dai et al. (2024) J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang Safe RLHF: safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §5, §5. Dönmez and Falenska (2026) E. Dönmez and A. Falenska Structuring the space of sociotechnical alignment. External Links: 2607.01250, Link Cited by: §7. Dubois et al. (2024) Y. Dubois, P. Liang, and T. Hashimoto Length-controlled alpacaeval: a simple debiasing of automatic evaluators. In First Conference on Language Modeling, External Links: Link Cited by: §2. Efron and Tibshirani (1993) B. Efron and R. J. Tibshirani An introduction to the bootstrap. Chapman & Hall/CRC, New York. Cited by: Appendix A, §5. Feng et al. (2024) S. Feng, T. Sorensen, Y. Liu, J. Fisher, C. Y. Park, Y. Choi, and Y. Tsvetkov Modular pluralism: pluralistic alignment via multi-LLM collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 4151–4171. External Links: Link, Document Cited by: §2. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. External Links: Link Cited by: §2. Ghazaryan and Dönmez (2026) G. Ghazaryan and E. Dönmez Misaligned by reward: socially undesirable preferences in llms. External Links: 2605.05003, Link Cited by: §2, Ethics Statement. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2, §3. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2. Jang et al. (2024) J. Jang, S. Kim, B. Y. Lin, Y. Wang, J. Hessel, L. Zettlemoyer, H. Hajishirzi, Y. Choi, and P. Ammanabrolu Personalized soups: personalized large language model alignment via post-hoc parameter merging. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, External Links: Link Cited by: §1, §2, §4.3, §7. Ji et al. (2023) J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang BEAVERTAILS: towards improved safety alignment of llm via a human-preference dataset. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. External Links: Link Cited by: §5. Kirk et al. (2024) H. R. Kirk, A. Whitefield, P. Röttger, A. M. Bean, K. Margatina, R. Mosquera, J. M. Ciro, M. Bartolo, A. Williams, H. He, B. Vidgen, and S. A. Hale The PRISM alignment dataset: what participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: §2. Lambert et al. (2025) N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, N. A. Smith, and H. Hajishirzi RewardBench: evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 1755–1797. External Links: Link, Document, ISBN 979-8-89176-195-7 Cited by: §2. Li et al. (2025) M. Li, Y. Zhang, W. Wang, W. Shi, Z. Liu, F. Feng, and T. Chua Self-improvement towards Pareto optimality: mitigating preference conflicts in multi-objective alignment. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 11010–11031. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §1, §2, §3, §6.1. Mudgal et al. (2024) S. Mudgal, J. Lee, H. Ganapathy, Y. Li, T. Wang, Y. Huang, Z. Chen, H. Cheng, M. Collins, T. Strohman, J. Chen, A. Beutel, and A. Beirami Controlled decoding from language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. External Links: Link Cited by: §2. Plank (2022) B. Plank The “problem” of human label variation: on ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), Abu Dhabi, United Arab Emirates, p. 10671–10682. External Links: Link, Document Cited by: §1, §1, §2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 53728–53741. External Links: Document, Link Cited by: §1, §2, §3. Rame et al. (2023) A. Rame, G. Couairon, C. Dancette, J. Gaya, M. Shukor, L. Soulier, and M. Cord Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §2, §4.3, §7. Shi et al. (2024) R. Shi, Y. Chen, Y. Hu, A. Liu, H. Hajishirzi, N. A. Smith, and S. S. Du Decoding-time language model alignment with multiple objectives. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385, Link Cited by: §2. Singhal et al. (2024) P. Singhal, T. Goyal, J. Xu, and G. Durrett A long way to go: investigating length correlations in RLHF. In First Conference on Language Modeling (COLM), External Links: Link Cited by: §2. Sorensen et al. (2024) T. Sorensen, J. Moore, J. Fisher, M. L. Gordon, N. Mireshghallah, C. M. Rytting, A. Ye, L. Jiang, X. Lu, N. Dziri, T. Althoff, and Y. Choi Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 46280–46302. External Links: Link Cited by: §1, §1, §2. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford alpaca: an instruction-following LLaMA model. GitHub. Note: https://github.com/tatsu-lab/stanford_alpaca Cited by: §5. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. External Links: Link Cited by: §5. Wang et al. (2024a) H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang Arithmetic control of LLMs for diverse user preferences: directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 8642–8655. External Links: Link, Document Cited by: §2, §2. Wang et al. (2024b) H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 10582–10592. External Links: Link, Document Cited by: §5. Wang et al. (2024c) Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. Scowcroft, N. Kant, A. Swope, and O. Kuchaiev HelpSteer: multi-attribute helpfulness dataset for SteerLM. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 3371–3384. External Links: Link, Document Cited by: §5. Welleck et al. (2020) S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston Neural text generation with unlikelihood training. In International Conference on Learning Representations, External Links: Link Cited by: §2. Wortsman et al. (2022) M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning (ICML), p. 23965–23998. External Links: Link Cited by: §2. Xu et al. (2026) Z. Xu, Y. Tong, X. Zhang, J. Zhou, and X. Wang Understanding conflicts in multi-objective alignment through reward consistency. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 5450–5472. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: §2. Yang et al. (2024) R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen Rewards-in-context: multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. External Links: Link Cited by: §2. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. External Links: Link Cited by: §2. Zhou et al. (2024) Z. Zhou, J. Liu, J. Shao, X. Yue, C. Yang, W. Ouyang, and Y. Qiao Beyond one-preference-fits-all alignment: multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 10586–10613. External Links: Link, Document Cited by: §1, §2, §3, §3, §5, §5, §6.2. Appendix A Training and Evaluation Details Data splits. HelpSteer uses the official 95/595/5 split (48,88848,888 training / 2,4742,474 validation preference pairs); UltraFeedback has no official validation split, so we use a random 90/1090/10 split with seed 00 (345,414345,414 / 38,38238,382 pairs). PKU-SafeRLHF-10K is split into 9,0009,000 training and 1,0001,000 validation response pairs. Evaluation uses the first 400400 unique validation prompts for the trade-off study and 700700 validation response pairs (665665 unique prompts) for the coverage study. Training. All stages use learning rate 10−410^-4, KL coefficient β=0.1β=0.1, AdamW with a cosine schedule, weight decay 0.050.05, and effective batch size 88 (one example, eight gradient-accumulation steps); the checkpoint with the lowest validation loss is kept. Trade-off-study LoRA uses rank/alpha/dropout 16/32/0.0516/32/0.05 on q_proj,v_proj in bf16; the coverage study uses 64/1/064/1/0 on q,k,v,o_proj in fp16. HelpSteer trains 1,0001,000 steps per stage; UltraFeedback trains 300300 (SFT, margin) and 400400 (MODPO); the coverage study trains 33 epochs per stage with a 10241024-token limit. We use split seed 00, training seed 303303 (trade-off) or 4242 (coverage), and generation seed 00, with a single seed per model. Generation. Primary decoding is greedy with no repetition penalty or n-gram blocking. The trade-off study generates up to 40964096 tokens (15361536 prompt / 25602560 response); the coverage study up to 10241024. The repetition-control setting adds repetition penalty 1.101.10 and no_repeat_ngram_size=4=4. Confidence intervals. We use the percentile bootstrap over paired score differences with 10,00010,000 resamples, seed 10001000, and 95%95\% intervals (Efron and Tibshirani, 1993), paired by validation prompt (trade-off) or response pair (coverage). Compute. Training and evaluation used NVIDIA RTX A6000 GPUs for approximately 1,2521,252 GPU-hours in total (∼985 985 for training, ∼267 267 for generation and scoring). Appendix B Objective Compatibility Measurements Table 4 gives both pre-training measurements at both training-length limits, together with the data they are computed from. The 512-token limit keeps roughly a third of the HelpSteer pairs and half of the UltraFeedback pairs; the 2048-token limit keeps essentially all of them, which is why we report it as primary. Data SFT corr. Conflict rate Pair Limit corr. 512512 20482048 512512 20482048 Joint pairs Retained HS correctness 512512 0.8530.853 0.9820.982 0.9810.981 0.0220.022 0.0270.027 7,0337,033 0.3150.315 20482048 21,18421,184 0.9960.996 HS coherence 512512 0.6350.635 0.9120.912 0.9110.911 0.0600.060 0.0670.067 5,5085,508 0.3170.317 20482048 16,34016,340 0.9960.996 HS complexity 512512 0.2360.236 0.1780.178 0.0950.095 0.1480.148 0.2020.202 4,1684,168 0.3430.343 20482048 11,25611,256 0.9960.996 HS verbosity (++) 512512 0.2560.256 −0.068-0.068 −0.162-0.162 0.1590.159 0.2260.226 5,6545,654 0.3350.335 20482048 15,66315,663 0.9960.996 HS verbosity (−-) 512512 −0.256-0.256 0.0680.068 0.1620.162 0.8410.841 0.7740.774 5,6545,654 0.3350.335 20482048 15,66315,663 0.9960.996 UF honesty 512512 0.6350.635 0.8700.870 0.8540.854 0.1860.186 0.1810.181 93,21893,218 0.5030.503 20482048 182,000182,000 0.9950.995 UF instr.-following 512512 0.6980.698 0.8880.888 0.8790.879 0.1720.172 0.1640.164 112,391112,391 0.5130.513 20482048 216,157216,157 0.9950.995 UF truthfulness 512512 0.6150.615 0.8320.832 0.8100.810 0.1930.193 0.1920.192 87,40287,402 0.5230.523 20482048 164,654164,654 0.9940.994 Table 4: Pre-training measurements and the data behind them, for both training-length limits. Data corr. is the Pearson correlation of the dataset’s own scores; SFT corr. the correlation of ArmoRM scores on reference-model generations (§4.1). Joint pairs counts response pairs that yield a preference pair for both objectives, and Retained the share of valid response pairs the limit keeps. The 2048-token columns are the ones reported in Table 1. Appendix C Per-Model Trade-off Scores Table 5 lists every score behind Figure 2, Figure 3, and the Both↑ ? column of Table 1: all eight settings, both training-length limits, and both decoding conditions. Table 6 gives the matching length and repetition diagnostics. Pair Limit SFT w=.1w=.1 w=.2w=.2 w=.4w=.4 w=.6w=.6 w=.8w=.8 w=.9w=.9 HS correctness 512512 .628.628/.641.641 .677.677/.681.681 .680.680/.683.683 .674.674/.677.677 .621.621/.626.626 .623.623/.627.627 .614.614/.619.619 20482048 .621.621/.636.636 .702.702/.706.706 .699.699/.703.703 .683.683/.688.688 .675.675/.680.680 .668.668/.674.674 – HS coherence 512512 .628.628/.737.737 .684.684/.756.756 .673.673/.749.749 .651.651/.729.729 .633.633/.713.713 .628.628/.710.710 .619.619/.698.698 20482048 .621.621/.730.730 .703.703/.770.770 .692.692/.761.761 .697.697/.763.763 .676.676/.749.749 .676.676/.748.748 – HS complexity 512512 .628.628/.386.386 .575.575/.439.439 .606.606/.446.446 .588.588/.443.443 .609.609/.446.446 .599.599/.436.436 .604.604/.438.438 20482048 .621.621/.382.382 .626.626/.455.455 .626.626/.451.451 .646.646/.456.456 .635.635/.452.452 .653.653/.440.440 – HS verbosity (++) 512512 .628.628/.427.427 .631.631/.632.632 .626.626/.630.630 .605.605/.645.645 .575.575/.653.653 .603.603/.635.635 .616.616/.636.636 20482048 .621.621/.420.420 .618.618/.625.625 .606.606/.634.634 .608.608/.641.641 .602.602/.637.637 .619.619/.622.622 .661.661/.602.602 HS verbosity (−-) 512512 .628.628/.427.427 .599.599/.334.334 .620.620/.359.359 .644.644/.384.384 .660.660/.442.442 .649.649/.597.597 .624.624/.620.620 20482048 .621.621/.420.420 .573.573/.282.282 .582.582/.289.289 .628.628/.343.343 .684.684/.428.428 .696.696/.520.520 .690.690/.547.547 UF honesty 512512 .432.432/.518.518 .438.438/.456.456 .452.452/.471.471 .450.450/.467.467 .440.440/.453.453 .441.441/.456.456 .446.446/.455.455 20482048 .430.430/.532.532 .391.391/.394.394 .388.388/.392.392 .395.395/.398.398 .410.410/.412.412 .428.428/.424.424 – UF instr.-following 512512 .432.432/.509.509 .464.464/.504.504 .470.470/.507.507 .469.469/.497.497 .459.459/.485.485 .441.441/.467.467 .436.436/.459.459 20482048 .430.430/.515.515 .375.375/.395.395 .391.391/.409.409 .418.418/.437.437 .410.410/.432.432 – .413.413/.428.428 UF truthfulness 512512 .432.432/.581.581 .461.461/.511.511 .459.459/.512.512 .462.462/.514.514 .452.452/.498.498 .459.459/.503.503 .441.441/.484.484 20482048 .430.430/.602.602 .463.463/.507.507 .451.451/.480.480 .422.422/.444.444 .421.421/.438.438 .409.409/.419.419 – 2048-token models re-decoded with repetition control HS correctness 20482048 .600.600/.607.607 .678.678/.673.673 .677.677/.671.671 .681.681/.675.675 .678.678/.671.671 .665.665/.661.661 – HS coherence 20482048 .600.600/.722.722 .667.667/.754.754 .669.669/.754.754 .668.668/.752.752 .665.665/.750.750 .668.668/.751.751 – HS complexity 20482048 .600.600/.378.378 .657.657/.463.463 .659.659/.467.467 .660.660/.461.461 .664.664/.462.462 .663.663/.444.444 – HS verbosity (++) 20482048 .600.600/.386.386 .652.652/.602.602 .648.648/.605.605 .653.653/.608.608 .648.648/.593.593 .656.656/.580.580 .670.670/.558.558 HS verbosity (−-) 20482048 .600.600/.386.386 .537.537/.274.274 .547.547/.284.284 .592.592/.330.330 .645.645/.426.426 .668.668/.505.505 .674.674/.522.522 UF honesty 20482048 .442.442/.535.535 .544.544/.553.553 .545.545/.553.553 .548.548/.548.548 .550.550/.551.551 .554.554/.552.552 – UF instr.-following 20482048 .442.442/.504.504 .546.546/.536.536 .548.548/.544.544 .552.552/.546.546 .556.556/.549.549 – .552.552/.545.545 UF truthfulness 20482048 .442.442/.595.595 .543.543/.570.570 .548.548/.574.574 .551.551/.561.561 .553.553/.554.554 .550.550/.543.543 – Table 5: Mean ArmoRM scores over the same 400400 prompts, as helpfulness/second-objective. Leading zeros are dropped; – marks a weight that was not trained. For HS verbosity (−-) a lower second score is better. The lower block re-decodes the 2048-token models with repetition penalty 1.101.10 and 44-gram blocking; its aggregate is Table 2. Pair Limit SFT w=.1w=.1 w=.2w=.2 w=.4w=.4 w=.6w=.6 w=.8w=.8 w=.9w=.9 HS correctness 512512 174174/.438.438 321321/.720.720 338338/.795.795 374374/.825.825 588588/.868.868 550550/.863.863 597597/.865.865 20482048 202202/.398.398 257257/.767.767 263263/.755.755 315315/.790.790 348348/.800.800 317317/.713.713 – HS coherence 512512 174174/.438.438 313313/.838.838 376376/.855.855 438438/.843.843 513513/.865.865 520520/.855.855 560560/.868.868 20482048 202202/.398.398 234234/.728.728 269269/.765.765 272272/.728.728 292292/.740.740 299299/.738.738 – HS complexity 512512 174174/.438.438 788788/.927.927 628628/.890.890 736736/.897.897 622622/.910.910 647647/.915.915 626626/.880.880 20482048 202202/.398.398 544544/.863.863 551551/.870.870 477477/.845.845 505505/.845.845 371371/.713.713 – HS verbosity (++) 512512 174174/.438.438 478478/.895.895 492492/.875.875 589589/.902.902 716716/.915.915 634634/.885.885 592592/.873.873 20482048 202202/.398.398 528528/.845.845 578578/.868.868 610610/.882.882 645645/.858.858 597597/.840.840 429429/.805.805 HS verbosity (−-) 512512 174174/.438.438 8282/.233.233 9090/.263.263 102102/.345.345 182182/.450.450 416416/.790.790 536536/.875.875 20482048 202202/.398.398 4141/.138.138 4646/.145.145 8484/.278.278 125125/.445.445 217217/.640.640 275275/.703.703 UF honesty 512512 329329/.527.527 727727/.892.892 687687/.895.895 677677/.897.897 695695/.917.917 683683/.920.920 670670/.920.920 20482048 234234/.480.480 897897/.965.965 889889/.958.958 862862/.960.960 808808/.960.960 786786/.950.950 – UF instr.-following 512512 329329/.527.527 590590/.823.823 584584/.835.835 607607/.858.858 642642/.900.900 687687/.917.917 706706/.927.927 20482048 234234/.480.480 972972/.973.973 922922/.953.953 824824/.960.960 828828/.960.960 – 830830/.970.970 UF truthfulness 512512 329329/.527.527 640640/.853.853 629629/.855.855 623623/.877.877 656656/.885.885 618618/.905.905 686686/.907.907 20482048 234234/.480.480 599599/.885.885 683683/.915.915 804804/.938.938 787787/.958.958 820820/.968.968 – 2048-token models re-decoded with repetition control HS correctness 20482048 8888/.013.013 164164/.020.020 161161/.010.010 161161/.015.015 164164/.008.008 149149/.010.010 – HS coherence 20482048 8888/.013.013 152152/.023.023 149149/.015.015 154154/.020.020 144144/.010.010 152152/.013.013 – HS complexity 20482048 8888/.013.013 210210/.015.015 204204/.018.018 196196/.033.033 197197/.030.030 156156/.015.015 – HS verbosity (++) 20482048 8888/.013.013 210210/.030.030 223223/.030.030 224224/.048.048 220220/.040.040 198198/.025.025 172172/.028.028 HS verbosity (−-) 20482048 8888/.013.013 3333/.003.003 3535/.000.000 5252/.003.003 9999/.003.003 141141/.008.008 154154/.013.013 UF honesty 20482048 114114/.080.080 329329/.305.305 338338/.313.313 356356/.348.348 345345/.320.320 327327/.298.298 – UF instr.-following 20482048 114114/.080.080 397397/.383.383 383383/.350.350 354354/.385.385 346346/.355.355 – 329329/.338.338 UF truthfulness 20482048 114114/.080.080 270270/.233.233 283283/.265.265 305305/.315.315 331331/.310.310 347347/.353.353 – Table 6: Generation diagnostics for the same models, as mean words/share of responses with a repeated 44-gram. Greedy decoding above, repetition control below. These are the numbers behind the length and repetition confound in §6.1. Appendix D Repetition Control, Per Pair Table 7 breaks Table 2 out by objective pair. The split is clean: no HelpSteer pair moves by more than 0.0360.036 on either objective, no UltraFeedback pair by less than 0.1030.103, while the length and repetition drops are comparable across the two datasets. Δ Δ 2 Δ Δ HS correctness −0.010-0.010 −0.020-0.020 −46.0%-46.0\% −75.3-75.3 HS coherence −0.021-0.021 −0.006-0.006 −44.7%-44.7\% −72.4-72.4 HS complexity +0.023+0.023 +0.009+0.009 −60.5%-60.5\% −80.5-80.5 HS verbosity (++) +0.036+0.036 −0.036-0.036 −62.9%-62.9\% −81.6-81.6 HS verbosity (−-) −0.032-0.032 +0.012+0.012 −30.0%-30.0\% −38.7-38.7 UF honesty +0.145+0.145 +0.147+0.147 −59.9%-59.9\% −64.2-64.2 UF instr.-following +0.149+0.149 +0.124+0.124 −58.7%-58.7\% −60.1-60.1 UF truthfulness +0.116+0.116 +0.103+0.103 −58.2%-58.2\% −63.8-63.8 HelpSteer (mean) −0.001-0.001 −0.008-0.008 −48.8%-48.8\% −69.7-69.7 UltraFeedback (mean) +0.137+0.137 +0.125+0.125 −58.9%-58.9\% −62.7-62.7 Table 7: Per-pair version of Table 2: change in scores, mean words, and repeated-44-gram share (percentage points) when the 2048-token models are re-decoded with repetition control. Positive score changes are in the preferred direction. Appendix E Conflict-Split Results Table 8 gives the scores for the conflict-split experiment of §4.2. The joint and non-conflict sets stay close to the main results, as expected, since non-conflict pairs make up most of the joint set; the conflict set is where the scores move. Setting Set SFT w=0.1w=0.1 w=0.2w=0.2 w=0.4w=0.4 w=0.8w=0.8 UF instr.-following, 512512 joint .432.432/.509.509 .445.445/.482.482 .430.430/.463.463 .456.456/.486.486 .457.457/.483.483 non-conflict .432.432/.509.509 .451.451/.475.475 .458.458/.486.486 .454.454/.480.480 .446.446/.470.470 conflict .432.432/.509.509 .412.412/.511.511 .430.430/.522.522 .427.427/.499.499 .424.424/.453.453 HS verbosity (−-), 512512 joint .628.628/.427.427 .604.604/.348.348 .604.604/.364.364 .631.631/.398.398 .647.647/.588.588 non-conflict .628.628/.427.427 .595.595/.371.371 .600.600/.393.393 .621.621/.422.422 .669.669/.562.562 conflict .628.628/.427.427 .585.585/.653.653 .617.617/.647.647 .594.594/.650.650 .614.614/.626.626 HS verbosity (−-), 20482048 joint .621.621/.420.420 .609.609/.317.317 .608.608/.317.317 .626.626/.345.345 .693.693/.518.518 non-conflict .621.621/.420.420 .581.581/.305.305 .555.555/.277.277 .613.613/.343.343 .670.670/.462.462 conflict .621.621/.420.420 .691.691/.419.419 .694.694/.426.426 .695.695/.441.441 .701.701/.518.518 Table 8: Conflict-split results (§4.2): mean helpfulness/second-objective scores for margin reward models trained on the joint set, the non-conflict set (conflict rate 00), and the conflict set (rate 11). The split is computed in the dataset’s score direction, so under the negative verbosity sign the “conflict” set is the one whose pairs agree with helpfulness. Setting Set SFT w=0.1w=0.1 w=0.2w=0.2 w=0.4w=0.4 w=0.8w=0.8 UF instr.-following, 512512 joint 329329/.527.527 714714/.868.868 738738/.880.880 673673/.887.887 643643/.905.905 non-conflict 329329/.527.527 676676/.917.917 685685/.920.920 667667/.943.943 685685/.915.915 conflict 329329/.527.527 212212/.430.430 270270/.475.475 471471/.650.650 732732/.920.920 HS verbosity (−-), 512512 joint 174174/.438.438 100100/.250.250 119119/.307.307 138138/.338.338 435435/.760.760 non-conflict 174174/.438.438 155155/.292.292 186186/.340.340 184184/.388.388 280280/.705.705 conflict 174174/.438.438 756756/.925.925 606606/.945.945 707707/.900.900 595595/.887.887 HS verbosity (−-), 20482048 joint 202202/.398.398 5858/.217.217 6666/.212.212 9595/.263.263 240240/.598.598 non-conflict 202202/.398.398 6464/.217.217 4646/.155.155 9898/.292.292 192192/.490.490 conflict 202202/.398.398 109109/.390.390 116116/.417.417 135135/.445.445 195195/.605.605 Table 9: Length and repetition diagnostics for the conflict-split models of Table 8, as mean words/share of responses with a repeated 44-gram. Appendix F Coverage Study Table 10 gives the per-target gaps that Table 3 averages, and Table 11 the reward and cost scores they are computed from. Reward gap Cost gap w⋆w Fine Coarse Merge Fine Coarse Merge 0.10.1 2.0182.018 2.0182.018 0.6730.673 4.9004.900 4.9004.900 2.8222.822 0.30.3 0.2530.253 2.2192.219 0.1660.166 3.4443.444 2.4002.400 1.1671.167 0.50.5 1.0381.038 – 0.0510.051 2.1192.119 – 0.1460.146 0.70.7 0.9280.928 0.0840.084 1.5461.546 0.7280.728 2.6452.645 0.2730.273 0.90.9 1.0501.050 3.7323.732 2.0592.059 0.2180.218 1.2261.226 0.4480.448 mean∗ 1.0621.062 2.0132.013 1.1111.111 2.3232.323 2.7932.793 1.1781.178 Table 10: Per-target absolute difference from a model trained directly at w⋆w , the values plotted in Figure 5 and averaged in Table 3. – marks the targets where coarse selection returns the direct model itself. ∗mean excluding w⋆=0.5w =0.5. Direct Lower Merge Upper w⋆w rew. cost rew. cost rew. cost rew. cost 0.10.1 −0.22-0.22 2.532.53 −2.24-2.24 −2.37-2.37 −0.90-0.90 −0.29-0.29 0.990.99 2.102.10 0.30.3 1.241.24 5.545.54 0.990.99 2.102.10 1.411.41 4.374.37 2.422.42 5.825.82 0.50.5 3.463.46 7.947.94 2.422.42 5.825.82 3.513.51 8.098.09 4.474.47 9.869.86 0.70.7 3.543.54 10.5810.58 4.474.47 9.869.86 5.095.09 10.8610.86 5.805.80 11.7611.76 0.90.9 4.754.75 11.9711.97 5.805.80 11.7611.76 6.816.81 12.4212.42 8.488.48 13.2013.20 Table 11: Mean Beaver reward and cost over the 700700 coverage prompts for the directly trained model, the two neighbouring trained models, and their merge. Every merged model lands between its neighbours, which is why merging cannot reach a direct target outside that interval (reward at w⋆=0.7,0.9w =0.7,0.9; cost at w⋆=0.1w =0.1).