Paper deep dive
MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment
Tony Tu, Sayan Chakraborty, Ruomeng Xu, Tony Qin, Austin Tian
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/18/2026, 4:46:01 AM
Summary
The paper introduces MINT (MIN-selection preference disTillation), a method for balancing multi-objective alignment in language agents. It addresses the failure of additive reward scalarization, which causes optimization to collapse onto the easiest objective. MINT modifies preference distillation by ranking candidates based on their weakest objective (min-selection) rather than a weighted sum, using the p→-∞ limit of the generalized mean. Evaluated on emotional support and adversarial negotiation tasks, MINT significantly improves both objectives and reduces imbalance compared to standard DPO and human experts.
Entities (8)
Relation Signals (8)
DPO → isusedby → MINT
confidence 95% · distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective
MINT → uses → min-selection
confidence 95% · MINT ranks them by their weakest objective... distilling the best-balanced candidate
MINT → reduces → imbalance
confidence 92% · min-selection lifts both objectives while sharply cutting their imbalance
MINT → improves → EQ
confidence 90% · Min-selection improves both objectives... on emotional support it raises the weaker axis from 0.37 to 0.64
MINT → improves → AQ
confidence 90% · Min-selection improves both objectives... on emotional support it raises the weaker axis from 0.37 to 0.64
min-selection → islimitof → Generalized Mean
confidence 90% · This is the p -> negative infinity limit of a generalized-mean family
ESConv → uses → AQ
confidence 85% · cooperative emotional support (ESConv)... instrumented with two orthogonal objectives... AQ... and EQ
CraigslistBargain → uses → AQ
confidence 85% · adversarial price negotiation (CraigslistBargain)... instrumented with two orthogonal objectives... AQ... and EQ
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p -> negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p < 10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.
Tags
Links
- Source: https://arxiv.org/abs/2608.14828v1
- Canonical: https://arxiv.org/abs/2608.14828v1
Trouble viewing inline? Open PDF directly →
Full Text
50,074 characters extracted from source content.
Expand or collapse full text
MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment Tony Tu Affiliation: Georgia Institute of Technology Affiliation: Zillow Groupttu32@gatech.eduSayan341@yahoo.inruomengx, tonyqi, austinti@zillowgroup.com Sayan Chakraborty Affiliation: Zillow Groupttu32@gatech.eduSayan341@yahoo.inruomengx, tonyqi, austinti@zillowgroup.com Ruomeng Xu Affiliation: Zillow Groupttu32@gatech.eduSayan341@yahoo.inruomengx, tonyqi, austinti@zillowgroup.com Tony Qin Affiliation: Zillow Groupttu32@gatech.eduSayan341@yahoo.inruomengx, tonyqi, austinti@zillowgroup.com Austin Tian Affiliation: Zillow Groupttu32@gatech.eduSayan341@yahoo.inruomengx, tonyqi, austinti@zillowgroup.com July 2026 Abstract Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p→−∞p\!→\!-∞ limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.370.37 to 0.640.64 (p<10−40p<10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does. 1 Introduction Aligning language models to human preferences with reinforcement learning from human feedback (4; 12) has become the dominant recipe for turning capable base models into useful assistants, and Direct Preference Optimization (13) has made this alignment dramatically simpler by collapsing reward modeling and policy optimization into a single supervised objective on preference pairs. Yet real deployments rarely optimize a single notion of “good.” A dialogue agent must be helpful and harmless, informative and concise, effective and kind, and these objectives routinely conflict (3; 14; 19). The standard response is to scalarize: combine the objectives into a weighted sum and optimize the aggregate, whether by training against a linear combination of reward models, folding weights into the DPO loss (19), or interpolating per-objective policies along the Pareto front (14). Additive scalarization, however, carries a structural flaw. Because a large gain on one objective can offset a large loss on another, the aggregate reward cannot distinguish a response that is excellent on every axis from one that is dominant on the cheapest axis and empty on the rest. Optimization therefore tends to collapse onto whichever objective is easiest to improve, an “alignment tax” in which balance is silently traded away (12; 14), and the resulting agent learns to sound warm while giving no real help, or to win the deal while alienating the counterpart. We argue that this failure is not inherent to preference distillation but to the additive aggregation it conventionally uses, and that a different aggregation removes it. We introduce Mint (min-selection preference distillation), which retains the now-standard best-of-K distillation pipeline, sample K candidates, score each with a frozen judge, and distill the preference back into the policy via DPO (5; 7; 13), but changes the one step that determines what the policy actually learns from: the selection of the chosen and rejected responses. Instead of ranking candidates by a scalar or a weighted sum, Mint ranks them by their minimum objective, a Chebyshev worst-case criterion that is the p→−∞p\!→\!-∞ limit of the generalized power mean. The chosen response is the candidate whose weakest axis is strongest, and the rejected response the most lopsided, so the distilled policy is pulled toward outputs that are good on every objective rather than outstanding on one. The change is deliberately minimal: it touches neither the DPO objective, the reward model, nor the sampling procedure, and so composes with the broad family of preference-optimization variants (2; 6; 9; 18). Its contribution is empirical rather than algorithmic, showing that the selection geometry alone suffices to induce multi-objective balance. We evaluate Mint on two deliberately opposite dialogue domains with distinct policy models, cooperative emotional support (ESConv) and adversarial price negotiation (CraigslistBargain), each instrumented with two orthogonal objectives scored by a frozen large-language-model judge: an action quotient (AQ) for task progress and an emotional quotient (EQ) for relational quality. Min-selection improves both objectives while substantially reducing their imbalance, and where the supervised reference is badly imbalanced it raises the weaker-axis reward from 0.370.37 to 0.640.64, yielding responses more balanced than those of human experts. Beyond the headline gain, a turn-resolved analysis of interactive rollouts surfaces our central and, we argue, generalizable observation: min-selection corrects objective imbalance in proportion to how imbalanced the reference policy is, and its advantage compounds over a multi-turn interaction exactly to the extent that the reference remains imbalanced along the trajectory. This reframes min-selection not as a universal boost but as a targeted, well-characterized remedy for the multi-objective collapse that additive preferences invite. Our contributions are: (i) a generalized-mean view of preference-pair selection that exposes a continuous family of rules from additive to worst-case; (i) Mint, its min-selection limit, as a drop-in modification to best-of-K preference distillation; and (i) an empirical characterization, across two structurally distinct domains and both static and interactive evaluation, of when and for how long balanced selection helps. 2 Related Work Preference optimization. Reinforcement learning from human feedback (4; 12) aligns language models to human preferences by training a reward model and optimizing the policy against it with PPO (15). Direct Preference Optimization (13) reformulates this pipeline as a single supervised objective on preference pairs, showing that the language model is implicitly its own reward model and thereby avoiding an explicit reward-modeling and RL stage. Its simplicity has produced a large family of variants that modify the loss or the reference term, including IPO (2), KTO (6), ORPO (9), and SLiC-HF (18). Our method is deliberately agnostic to these choices. Mint changes only which candidates become the chosen and rejected responses, and uses the standard DPO loss unchanged, so it remains compatible with any of these variants. Multi-objective alignment. Real deployments require balancing several, often conflicting, objectives, such as helpfulness versus harmlessness or informativeness versus conciseness, for which no single reward suffices (14; 19). Most approaches scalarize the objectives with a weighted sum: multi-objective RLHF trains a policy against a linear combination of reward models, MODPO (19) folds objective weights into the DPO loss, and Rewarded Soups (14) trains one policy per objective and interpolates their weights to trace the Pareto front. Rewards-in-Context (17) instead conditions a single model on target reward values supplied in the prompt. A recurring difficulty, noted across this line of work, is that additive scalarization permits an “alignment tax” in which one objective is improved at the expense of another. Mint departs from the additive convention. Rather than weighting objectives, it selects preference pairs by the minimum objective (a Chebyshev criterion), which we show empirically drives the policy toward balanced improvement rather than trading one axis for the other, and requires no per-objective weights or auxiliary models. Rejection sampling and best-of-K distillation. A complementary line of work improves a policy by sampling multiple candidates, scoring them, and fine-tuning on the best ones. RAFT (5) and ReST (7) perform reward-ranked (rejection-sampling) fine-tuning, distilling a best-of-K selection back into the policy, and statistical rejection sampling (11) connects this selection to preference optimization. Best-of-N sampling itself is a strong inference-time baseline (1), and BOND (16) distills its behavior into a single model to avoid the inference-time cost. Mint is an instance of this best-of-K preference-distillation paradigm. It generates K candidates, scores them with a frozen judge, and distills the result via DPO, but whereas prior work ranks candidates by a single scalar reward, Mint ranks them by the minimum over multiple objectives, making the selection geometry itself the mechanism for multi-objective balance. Our contribution is thus empirical rather than algorithmic: the same distillation machinery, with a different selection rule, yields balanced multi-objective policies. 3 Method Figure 1: Three ways to form preference pairs for DPO. Every method turns one dialogue turn into a chosen/rejected pair and distills it with the same DPO objective; they differ only in how that pair is selected. (a) Standard DPO scores each candidate reply with a single scalar reward r=J(x,y)r=J(x,y) and takes the highest as y+y^+ and the lowest as y−y^-. (b) Additive multi-objective DPO scores each candidate on two axes, helpfulness (h) and harmlessness (s), and ranks by their sum r¯=h+s2 r= h+s2; because a surplus on one axis compensates a deficit on the other, a lopsided reply can be selected. (c) Mint (ours) samples K=8K=8 candidates per turn, scores each on both axes with a frozen judge, and selects the pair by the weakest objective, m=min(h,s)m= (h,s): the chosen reply is the best-balanced candidate and the rejected reply the most lopsided one, so both objectives improve together. Like a barrel whose water level is capped by its shortest stave, overall conversation effectiveness is bounded by the weakest objective. Mint is a preference-distillation pipeline that trains a policy to satisfy several objectives simultaneously. At each turn it samples a set of candidate responses from a reference policy, scores every candidate on each objective with a frozen judge, selects a chosen/rejected pair according to a balance-seeking rule, and distills the resulting preferences into the policy with Direct Preference Optimization (DPO). The only departure from a standard best-of-K DPO pipeline is the selection rule that forms the pairs; this is where multi-objective balance is enforced. 3.1 Problem setup We consider a turn-level dialogue policy πθ _θ that, given a conversation prefix x (system prompt and dialogue history), produces the next agent utterance y∼πθ(⋅∣x)y _θ(· x). Response quality is not scalar: we assume m interpretable objectives, each scored by a frozen judge J on [0,1][0,1]. In our instantiation m=2m=2, with an action quotient AQ(x,y)AQ(x,y) measuring task progress and an emotional quotient EQ(x,y)EQ(x,y) measuring relational quality, but the method is agnostic to m and to the semantics of the objectives. The judge is a frozen large language model prompted with an objective-specific rubric; it is never fine-tuned, so the same backbone scores all policies and both objectives, differing only in the rubric it is given. The central difficulty is that the objectives are in tension: a response can advance the task while alienating the user, or comfort the user while stalling progress. A policy trained to maximize a naive combination of the two, most commonly their sum, can improve the aggregate while degrading the weaker objective, collapsing onto whichever axis is easier to raise. Mint addresses this at the level of preference selection. 3.2 Candidate generation and scoring For each agent turn in a corpus of N conversations, we draw K candidate responses y1,…,yK\y_1,…,y_K\ from a reference policy πref _ref and score each on every objective, yielding reward vectors k=(AQ(x,yk),EQ(x,yk))r_k=(AQ(x,y_k),EQ(x,y_k)). To obtain candidates that genuinely span the objective trade-off, rather than K near-duplicates, we condition generation on a small set of lightweight stance prompts that steer the sampler toward different regions of objective space (e.g. task-focused vs. rapport-focused); the stance is used only to diversify candidates and is discarded afterwards, so the distilled policy is never conditioned on it. All candidates, their reward vectors, and the original human turn are cached, so that preference pairs under any selection rule can be constructed offline without re-running the judge. 3.3 Generalized-mean scalarization of multiple objectives Let a turn receive two reward scores, a=AQ∈[0,1]a=AQ∈[0,1] (action quotient) and b=EQ∈[0,1]b=EQ∈[0,1] (emotional quotient). Ranking candidates requires collapsing the reward vector (a,b)(a,b) into a scalar, for which we use the generalized (power) mean Mp(a,b)=(12(ap+bp))1/p,p∈ℝ.M_p(a,b)= ( 12 (a^p+b^p ) )^1/p, p . (1) The single parameter p controls how strongly imbalance is penalized, and recovers familiar means as special cases: the arithmetic mean (p=1p=1), quadratic (p=2p=2), geometric (p→0p→0, i.e. ab ab), harmonic (p=−1p=-1), and, in the limit, M−∞(a,b)=min(a,b)M_-∞(a,b)= (a,b). As p decreases the score is increasingly dominated by the smaller coordinate. At p=1p=1 a shortfall on one axis is fully compensable by a surplus on the other, so a lopsided turn can score as well as a balanced one; p=2p=2 is worse still, rewarding vector magnitude and thus favoring single-axis extremes. Below p=1p=1 the ordering reverses: the geometric mean collapses toward 00 if either objective is near 00, the harmonic mean penalizes imbalance more sharply, and M−∞M_-∞ ignores any surplus on the stronger axis entirely, scoring each candidate purely by its weakest objective. This last limit is the selection rule Mint adopts. 3.4 Min-selection preference pairs Given the scalarization MpM_p, we form a single preference pair per turn. The chosen response is the candidate that maximizes MpM_p and the rejected response the one that minimizes it: y+=argmaxkMp(k),y−=argminkMp(k).y^+= _kM_p(r_k), y^-= _kM_p(r_k). (2) Throughout this work we use the limiting case p→−∞p→-∞, i.e. M−∞=min(AQ,EQ)M_-∞= (AQ,EQ), which we refer to as min-selection. Under min-selection the chosen response is the candidate whose weaker objective is strongest, the best-balanced option, and the rejected response is the most lopsided one. Because the min is insensitive to any surplus on the stronger axis, ties are common (several candidates may share the same minimum coordinate); we break them by the sum, i.e. we rank candidates by the ordered key (min(a,b),a+b) ( (a,b),\,a+b ), so that among equally-balanced candidates the one with greater total reward is preferred. To ensure each pair carries a meaningful training signal, we discard turns whose chosen and rejected candidates differ by less than a margin δ in min(AQ,EQ) (AQ,EQ); such turns are typically openers or purely factual exchanges on which all candidates behave alike. 3.5 Preference distillation The selected pairs (x,y+,y−)\(x,y^+,y^-)\ are distilled into the policy with DPO (13), which optimizes ℒDPO=−[logσ(βlogπθ(y+∣x)πref(y+∣x)−βlogπθ(y−∣x)πref(y−∣x))],L_DPO=-\,E [ σ\! (β _θ(y^+ x) _ref(y^+ x)-β _θ(y^- x) _ref(y^- x) ) ], (3) with reference policy πref _ref and temperature β. We take πref _ref to be the same policy used to generate the candidates: a supervised (SFT) checkpoint where one is available, and the base model otherwise, and train a lightweight low-rank (LoRA) adapter on top, so that distillation adjusts the policy relative to its own sampling distribution. This is a single offline round: candidates are generated once from πref _ref, scored once, and distilled once. The generalized-mean formulation (Eq. 1) also admits a semi-online variant in which candidates are regenerated from the improved policy over several rounds; we present the general algorithm below and use the single-round (R=1R=1) instantiation in all experiments. 3.6 Algorithm Algorithm 1 Mint: preference distillation with p-norm multi-objective selection 1: reference policy πref _ref; frozen judge J; conversations D; candidates per turn K; norm p; margin δ; rounds R 2: πθ←πref _θ← _ref 3: for round =1=1 to R do 4: ←∅P← 5: for each conversation in D, each agent turn with prefix x do 6: sample candidates y1,…,yK∼πθ(⋅∣x)y_1,…,y_K _θ(· x) 7: score k←(AQ(x,yk),EQ(x,yk))r_k← (AQ(x,y_k),\,EQ(x,y_k) ) with J, for all k 8: y+←argmaxkMp(k)y^+← _kM_p(r_k); y−←argminkMp(k)y^-← _kM_p(r_k) ⊳ ties broken by a+ba+b 9: if Mp(+)−Mp(−)≥δM_p(r^+)-M_p(r^-)≥δ then 10: ←∪(x,y+,y−)P ∪\(x,y^+,y^-)\ 11: end if 12: end for 13: πθ←argminθℒDPO(πθ,πref,) _θ← _θL_DPO( _θ, _ref,P) 14: end for 15: return πθ _θ Algorithm 1 is written in full generality: any norm p (hence any point on the additive–geometric–min spectrum), any number of rounds R, and any number of objectives m (the reward vector kr_k and MpM_p extend to m coordinates unchanged). Our experiments use the balance-maximizing extreme p→−∞p→-∞ (min), a single offline round R=1R=1, and m=2m=2 objectives. 4 Results We evaluate Mint on two structurally distinct dialogue domains using two complementary protocols. ESConv (10) is a cooperative emotional-support corpus; CraigslistBargain (8) is an adversarial buyer–seller price negotiation. For each domain we distill from 300300 conversations of best-of-K candidates (K=8K=8) and hold out a 10%10\% test split of conversations that is never used for training or checkpoint selection, so all reported numbers are on dialogues unseen during distillation. Both objectives: achievement quotient (AQ) and emotional quotient (EQ), are scored by the same frozen Llama-3.1-8B judge, differing only in the domain-specific rubric. We report each objective, their minimum min(AQ,EQ) (AQ,EQ) (the quantity Mint selects on), and their absolute imbalance |AQ−EQ||AQ-EQ|. 4.1 Static held-out evaluation Our primary protocol is a matched, single-turn completion task on held-out test conversations. At every real agent turn, each policy generates one response from the identical human-written conversation prefix, and the judge scores all policies plus the original human turn against the same context. This isolates the effect of the policy while holding dialogue history fixed, avoiding the distributional drift of full rollouts. Figure 2: Mint on ESConv (static held-out evaluation). Both panels report the held-out test split (n=406n=406 agent turns across 3030 conversations never seen during distillation or checkpoint selection); achievement (AQ) and emotional (EQ) quality are each scored by a frozen Llama-3.1-8B judge. (a) Per-objective scores for the base model, the supervised (SFT) baseline, Mint, and human supporters. Mint improves both objectives and their minimum min(AQ,EQ) (AQ,EQ) over every baseline while shrinking the imbalance |AQ−EQ||AQ-EQ|: relative to SFT it raises min(AQ,EQ) (AQ,EQ) from 0.370.37 to 0.640.64 (a +0.27+0.27 paired improvement; Wilcoxon signed-rank p≈3×10−43p≈ 3× 10^-43, higher on 79%79\% of matched turns). (b) The same policies as points in objective space. The base model is warm but low-achievement (high EQ, low AQ), SFT is under-warm, and human supporters are themselves lopsided; Mint is the only policy that moves toward the balance diagonal (AQ=EQAQ=EQ), attaining both the highest and the most balanced scores, more balanced than the human experts whose transcripts seed the pipeline. All scores are quality as assessed by the reward model Figure 3: Mint on Craigslist Bargaining (held-out evaluation). Each policy is scored by the frozen Llama-3.1-8B judge on held-out negotiation contexts (n=103n=103 turns). (a) Per-objective scores. Mint (DPO) improves both objectives and their minimum min(AQ,EQ) (AQ,EQ) over the base model, raising the minimum from 0.5830.583 to 0.6570.657 (Δmin=0.074 _ =0.074, p=0.0002p=0.0002) while also tightening the gap between objectives (0.1770.177 to 0.1350.135). (b) The same policies as points in objective space. The base→ shift moves up and toward the balance diagonal (AQ=EQAQ=EQ). The human row is scored on real conversation contexts and sits well below both policies on quality as assessed by the reward model (AQ 0.2750.275, EQ 0.4660.466), serving as an unmatched reference. All scores are quality as assessed by the reward model. Reference Mint Mint −- Ref Depth AQ EQ |Δ|†| | min‡ AQ EQ |Δ|†| | min‡ Δ Δ Δ ESConv (emotional support): Mint vs. SFT reference – advantage persists across depth ≤ 1 0.356 0.623 0.380 0.300 0.535 0.777 0.242 0.535 +0.179 +0.154 +0.235 ≤ 2 0.472 0.644 0.287 0.415 0.643 0.788 0.171 0.630 +0.171 +0.144 +0.215 ≤ 3 0.519 0.568 0.282 0.403 0.695 0.772 0.158 0.655 +0.176 +0.204 +0.252 ≤ 4 0.498 0.485 0.299 0.342 0.708 0.692 0.207 0.596 +0.210 +0.207 +0.254 ≤ 5 0.457 0.393 0.302 0.274 0.683 0.605 0.240 0.524 +0.226 +0.212 +0.250 ≤ 6 0.408 0.332 0.282 0.229 0.637 0.527 0.251 0.456 +0.229 +0.195 +0.227 ≤ 7 0.367 0.291 0.256 0.201 0.592 0.465 0.260 0.399 +0.225 +0.174 +0.198 ≤ 8 0.325 0.261 0.228 0.179 0.548 0.419 0.254 0.357 +0.223 +0.158 +0.178 ≤ 9 0.295 0.241 0.212 0.162 0.508 0.384 0.249 0.322 +0.213 +0.143 +0.159 ≤ 10 0.268 0.222 0.196 0.147 0.471 0.353 0.239 0.292 +0.203 +0.131 +0.145 CraigslistBargain (negotiation): Mint vs. base reference – advantage decays with depth ≤ 1 0.507 0.693 0.218 0.491 0.575 0.749 0.188 0.568 +0.068 +0.056 +0.077 ≤ 2 0.606 0.738 0.156 0.594 0.642 0.747 0.132 0.629 +0.036 +0.009 +0.035 ≤ 3 0.642 0.749 0.136 0.628 0.667 0.757 0.118 0.653 +0.024 +0.008 +0.025 ≤ 4 0.626 0.759 0.164 0.611 0.649 0.767 0.149 0.634 +0.023 +0.008 +0.023 ≤ 5 0.592 0.767 0.205 0.577 0.607 0.774 0.196 0.593 +0.015 +0.006 +0.015 ≤ 6 0.550 0.772 0.249 0.537 0.558 0.778 0.245 0.546 +0.008 +0.006 +0.009 ≤ 7 0.525 0.775 0.274 0.513 0.532 0.780 0.271 0.521 +0.007 +0.006 +0.008 ≤ 8 0.508 0.776 0.292 0.496 0.507 0.781 0.296 0.496 +0.000 +0.006 +0.001 Table 1: Advantage of Mint by conversation depth in interactive self-play rollouts. Each row aggregates turns 11 to K across 3030 conversations; AQ and EQ are the mean achievement and emotional rewards over those turns. †|Δ|=meani|AQi−EQi| | |=mean_i|AQ_i-EQ_i| is the mean per-turn imbalance (computed turn-by-turn, then averaged; it therefore does not equal |AQ¯−EQ¯|| AQ- EQ|). min‡=meanimin(AQi,EQi) min=mean_i (AQ_i,EQ_i) is the mean per-turn minimum (likewise computed per turn, so it is not min(AQ¯,EQ¯) ( AQ, EQ)). The last three columns are Mint minus the reference on each quantity. Top (ESConv): against the imbalanced SFT reference, gains are large and persist at all depths (all p<10−4p<10^-4); SFT degrades over multi-turn dialogue while Mint stays balanced. Bottom (Craigslist): against a competent base, Mint significantly improves the opening matched turn (Δ =+0.077=+0.077, p=0.016p=0.016) but the advantage decays as both converge. Δ shading marks significance (dark: p<0.05p<0.05). Emotional support (ESConv). Figure 2 reports results on the 3030 held-out test conversations (n=406n=406 agent turns). Relative to the supervised (SFT) baseline, Mint improves both objectives simultaneously: AQ from 0.530.53 to 0.720.72 and EQ from 0.470.47 to 0.710.71m and raises the weaker-axis reward min(AQ,EQ) (AQ,EQ) from 0.370.37 to 0.640.64, a paired improvement of +0.27+0.27 that is highly significant (Wilcoxon signed-rank p≈3×10−43p≈ 3× 10^-43; Mint scores higher on 79%79\% of matched turns). At the same time the imbalance |AQ−EQ||AQ-EQ| falls from 0.260.26 to 0.160.16. The balance geometry (Figure 2, right) makes the mechanism visible: the base model is warm but low-achievement (high EQ, low AQ), SFT is under-warm, and human supporters are themselves lopsided, whereas Mint is the only policy that moves toward the AQ=EQAQ=EQ diagonal, attaining both the highest and the most balanced scores, more balanced, in fact, than the human experts whose data seeded the pipeline. Negotiation (CraigslistBargain). Figure 3 reports the same protocol on the held-out negotiation test split, with one deliberate difference from the ESConv setup: here we omit the SFT stage entirely and apply DPO directly on top of the prompted base model. This tests two things. First, whether the balancing effect survives without a masked-language-modeling SFT stage that first teaches the policy to imitate reference seller turns, i.e., whether DPO alone can induce the improvement rather than merely sharpening a distribution already shaped by SFT. Second, whether Mint outperforms the base model it is trained from, without the confound of an intermediate SFT checkpoint. Both hold: as in the cooperative domain, Mint improves the weaker objective and reduces |AQ−EQ||AQ-EQ| relative to the base policy (DPO improves min(AQ,EQ) (AQ,EQ) from 0.5830.583 to 0.6570.657; Wilcoxon p=0.0002p=0.0002; n=103n=103 turns), while tightening the objective gap from 0.1770.177 to 0.1350.135. This confirms that the balancing effect transfers to an adversarial setting and a different policy model, does not depend on a prior SFT stage, and is not an artifact of the cooperative, emotionally-toned ESConv distribution. 4.2 Interactive rollout evaluation The static protocol scores single completions on human-written contexts. To test whether balance persists when a policy drives an entire conversation, compounding its own decisions turn over turn, we additionally run an interactive rollout protocol. Each policy conducts a full multi-turn dialogue against a simulated conversational partner (a persona-conditioned user model for ESConv; a prompted counterpart-agent for CraigslistBargain), and the judge scores every policy turn along the generated trajectory. This is a harder and noisier test, since errors accumulate and the policy leaves the distribution of human dialogue histories. Emotional support (ESConv). Each policy conducts a full support dialogue against a persona-conditioned user model, and the frozen Llama-3.1-8B judge scores every policy turn (n≈280n≈280 turns across 3030 rollouts per policy). Against the SFT reference from which it was distilled, Mint’s advantage is large and persists at every depth: the improvement in min(AQ,EQ) (AQ,EQ) ranges from +0.235+0.235 at the opening turn to +0.145+0.145 over full ten-turn rollouts, significant throughout (Wilcoxon p<10−4p<10^-4; Table 1, top). The mechanism is visible in the trajectory: the SFT policy degrades as the dialogue lengthens, its mean minimum falling monotonically from 0.3000.300 to 0.1470.147 as early warmth gives way to terse, low-empathy turns (EQ 0.623→0.2220.623→0.222), whereas Mint decays gracefully (0.535→0.2920.535→0.292) and stays balanced. In a degenerate reference, errors compound over the trajectory, so Mint’s per-turn edge accumulates rather than washes out. Against the base model the picture is more nuanced. The base is not degenerate but lopsided: persistently warm (EQ near 0.800.80 throughout) while contributing little achievement early. Mint starts far ahead (Δmin=+0.328 =+0.328, p=0.0004p=0.0004, as the base’s cold-start AQ of 0.210.21 drags down its minimum), but the gap closes as the base’s achievement rises, reaching parity by turn 5 (Δmin=−0.001 =-0.001) and a small non-significant reversal after. The base’s stubborn warmth lifts its running minimum to meet Mint’s, which trades warmth for achievement as it presses the task. Mint thus dominates a collapsing reference at all depths but holds only an early-turn edge over a warm-but-passive one, a distinction the pooled average (min : base 0.3150.315, Mint 0.2920.292) obscures entirely. The human row, scored on real contexts rather than the rollouts, is an unmatched reference. Negotiation (CraigslistBargain). Each seller policy negotiates against a prompted buyer agent (base-Gemma conditioned on a buyer persona) seeded from real Craigslist openings, with the judge scoring every seller turn. As the adversarial analogue of ESConv, the counterpart actively pushes the price down, penalising a policy that has merely learned to sound agreeable. The result mirrors Mint’s behaviour against the ESConv base rather than SFT. On the opening, strictly matched turn, Mint significantly improves the weaker axis (Δmin=+0.077 =+0.077, p=0.016p=0.016; AQ 0.507→0.5750.507→0.575, EQ 0.693→0.7490.693→0.749), reproducing the static held-out result. But the advantage decays monotonically with depth (+0.077,+0.035,…,+0.001+0.077,+0.035,…,+0.001 from turn 1 to 8), significant only at the first exchange; by the rollout’s end the policies are indistinguishable (Δmin=+0.001 =+0.001, p=0.43p=0.43 pooled). The cause is not weak negotiation but a strong reference: the base Gemma-3-27B seller is already balanced under our prompt, holding price, justifying value, and staying courteous (per-turn minimum 0.490.49 at the opening and rising). With little imbalance to correct, the two policies converge to similar outcomes near the seller’s target, and their per-turn rewards equalise as the shared-opening conversations diverge into comparably successful negotiations. When does min-selection help, and for how long? Together the two rollouts expose a principle the static evaluations could not reveal: Mint improves the weaker objective in proportion to the reference’s imbalance, and this gain persists over interaction only while the reference stays imbalanced along the trajectory. Against a badly and increasingly imbalanced reference (ESConv SFT, whose empathy collapses over a dialogue), the advantage is large and grows at every depth. Against a reference that is already balanced, or imbalanced only transiently, it is significant on matched contexts (the static evaluation and the opening turn of both rollouts) but attenuates as trajectories diverge and a competent reference recovers the weaker axis on its own: the ESConv base regains achievement over the conversation, and the Craigslist base was balanced from the outset. This is why pooled averages understate the method, collapsing a strongly significant matched opening effect with later turns where a competent reference has caught up, averaging a real localised gain into an apparent null. The depth-resolved view (Table 1) is the honest one: min-selection is a targeted corrector of reference imbalance, most valuable exactly where and when that imbalance is present, not a uniform improvement. 4.3 Discussion Mint forms preference pairs by selecting on the minimum of two objectives rather than a weighted sum, yielding policies that are stronger on both axes and better balanced between them. A weighted sum lets a gain on the easy axis pay for a loss on the hard one; min-selection closes that route by scoring each pair on its weaker axis. Where the reference is badly imbalanced the effect is large and durable, raising the weaker axis from 0.370.37 to 0.640.64 on emotional support and exceeding the human supporters who seeded the distillation, which suggests the balancing signal was latent in the best-of-K candidates and surfaced only when selection refused to average the axes away. But our turn-resolved analysis shows the benefit tracks the reference: it compounds over interaction only while the reference stays imbalanced, and against an already-balanced policy it holds on matched contexts yet attenuates as trajectories diverge. Min-selection is thus a targeted corrector of objective imbalance, most valuable where that imbalance is most severe. References Amini et al. (2025) A. Amini, T. Vieira, E. Ash, and R. Cotterell Variational best-of-n alignment. In International Conference on Learning Representations, Vol. 2025, p. 25717–25737. Cited by: §2. Azar et al. (2024) M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. Cited by: §1, §2. Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §1. Christiano et al. (2017) P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. Advances in neural information processing systems 30. Cited by: §1, §2. Dong et al. (2023) H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang Raft: reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767. Cited by: §1, §2. Ethayarajh et al. (2023) K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela Kto: model alignment as prospect theoretic optimization, 2024. URL https://arxiv. org/abs/2402.01306 14. Cited by: §1, §2. Gulcehre et al. (2023) C. Gulcehre, T. Le Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling, 2023. URL https://arxiv. org/abs/2308.08998. Cited by: §1, §2. He et al. (2018) H. He, D. Chen, A. Balakrishnan, and P. Liang Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2333–2343. Cited by: §4. Hong et al. (2024) J. Hong, N. Lee, and J. Thorne Orpo: monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 11170–11189. Cited by: §1, §2. Liu et al. (2021) S. Liu, C. Zheng, O. Demasi, S. Sabour, Y. Li, Z. Yu, Y. Jiang, and M. Huang Towards emotional support dialog systems. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers), p. 3469–3483. Cited by: §4. Liu et al. (2024) T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. Liu, and J. Liu Statistical rejection sampling improves preference optimization. In International conference on learning representations, Vol. 2024, p. 54605–54624. Cited by: §2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. Cited by: §1, §2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. arXiv preprint arXiv:2305.18290. Cited by: §1, §1, §2, §3.5. Rame et al. (2023) A. Rame, G. Couairon, C. Dancette, J. Gaya, M. Shukor, L. Soulier, and M. Cord Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems 36, p. 71095–71134. Cited by: §1, §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2. Sessa et al. (2024) P. G. Sessa, R. Dadashi, L. Hussenot, J. Ferret, N. Vieillard, A. Ramé, B. Shariari, S. Perrin, A. Friesen, G. Cideron, et al. Bond: aligning llms with best-of-n distillation, 2024. URL https://arxiv. org/abs/2407.14622. Cited by: §2. Yang et al. (2024) R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen Rewards-in-context: multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207. Cited by: §2. Zhao et al. (2023) Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu Slic-hf: sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. Cited by: §1, §2. Zhou et al. (2024) Z. Zhou, J. Liu, J. Shao, X. Yue, C. Yang, W. Ouyang, and Y. Qiao Beyond one-preference-fits-all alignment: multi-objective direct preference optimization, 2024. URL https://arxiv. org/abs/2310.03708. Cited by: §1, §2. Appendix Contents Appendix A Training Details A.1 Emotional Support DPO Training Figure 4: Mint training dynamics on ESConv (full-300 distillation, Llama-3.1-8B policy). (a) Training and validation DPO loss; validation loss decreases monotonically and flattens by roughly step 130130. (b) Preference accuracy on held-out pairs, rising from near-chance to a plateau of about 0.840.84. (c) Mean reward margin between the chosen and rejected responses (r+−r−r^+\!-\!r^-), which grows steadily without diverging. The checkpoint with the lowest validation loss is selected under early stopping. We distill the ESConv policy from a Llama-3.1-8B-Instruct backbone, using a supervised (SFT) checkpoint as the reference policy πref _ref. Following a merge-then-adapt scheme, we merge the SFT adapter into the base weights and train a fresh low-rank DPO adapter on top (rank 1616, α=32α=32, dropout 0.050.05; 41.941.9M trainable parameters, 0.52%0.52\% of the model), so that disabling the adapter recovers the SFT reference exactly. Preference pairs are formed by min(AQ,EQ) (AQ,EQ) selection with the sum tie-break of Section 3.4 and a selection margin δ=0.25δ=0.25; pairs whose prompt or completion exceed the length budget are dropped. From the 300300 distilled conversations this yields 1,9561,956 training and 224224 validation pairs. We train with DPO temperature β=0.1β=0.1, learning rate 10−510^-5, a cosine schedule with 10%10\% warmup, and an effective batch size of 88, selecting the checkpoint with the lowest validation loss under early stopping (patience 33). As shown in Figure 4, validation loss falls from 0.690.69 to 0.340.34, preference accuracy plateaus near 0.840.84, and the chosen/rejected reward margin grows steadily to roughly 1.81.8 without diverging, indicating stable optimization rather than reward over-optimization. A.2 Price-Negotiation DPO Training Figure 5: Mint training dynamics on CraigslistBargain (Gemma-3-27B policy, 4-bit QLoRA). (a) Training and validation DPO loss; validation loss falls sharply over the first ∼50 50 steps and flattens near its minimum, while the training loss is noisier under 4-bit quantization and a small effective batch. (b) Preference accuracy on held-out pairs, rising from about 0.760.76 to a plateau near 0.840.84. (c) Mean reward margin between the chosen and rejected responses (r+−r−r^+\!-\!r^-), which grows steadily and monotonically without diverging. The checkpoint with the lowest validation loss is selected under early stopping (patience 33). We distill the CraigslistBargain policy from a Gemma-3-27B-Instruct backbone under 4-bit QLoRA (NF4 with double quantization and a bfloat16 compute dtype), using the quantized base model with its adapter disabled as the reference policy πref _ref, so that turning the adapter off recovers the reference exactly. We train a single low-rank DPO adapter on all attention and MLP projections (q,k,v,o,gate,up,down) with rank 88, α=16α=16, and dropout 0.050.05 (56.856.8M trainable parameters, 0.21%0.21\% of the model), using paged 8-bit AdamW and gradient checkpointing for single-GPU memory efficiency. Preference pairs are formed by min(AQ,EQ) (AQ,EQ) selection with the sum tie-break of Section 3.4 and a selection margin δ=0.25δ=0.25; pairs whose prompt exceeds 384384 tokens or whose chosen/rejected completion exceeds 128128 tokens are dropped. From the 300300 distilled conversations (each with K=8K=8 scored candidates per turn), split 80/10/1080/10/10 into train/validation/test, this yields 725725 training and 8989 validation preference pairs. We train with DPO temperature β=0.1β=0.1, learning rate 10−510^-5, a cosine schedule with 10%10\% warmup, and an effective batch size of 88 (per-device batch 11 with gradient accumulation 88) for up to 55 epochs, evaluating every 1010 steps and selecting the checkpoint with the lowest validation loss under early stopping (patience 33). As shown in Figure 5, validation loss falls from about 0.630.63 to a minimum near 0.250.25 before settling around 0.290.29, preference accuracy plateaus near 0.840.84, and the chosen/rejected reward margin grows steadily to roughly 5.85.8. The larger terminal margin and noisier training loss, relative to the ESConv run, are consistent with the more adversarial and sharply separable preferences of price negotiation under a 4-bit-quantized policy and a small effective batch, while the smoothly decreasing validation loss indicates stable optimization rather than reward over-optimization. Appendix B Reward Rubrics and Judge Configuration B.1 Frozen judge model All rewards in this paper are produced by a single frozen Llama-3.1-8B-Instruct model acting as an LLM-as-judge; its weights are never updated during rejection sampling, preference-pair construction, or evaluation, so the reward signal is held fixed across training and all evaluation conditions. The judge is prompted separately for each domain with the rubric-specific system prompt given below, receives the item/context and the conversation so far, and scores only the policy’s most recent turn. It emits a single JSON object containing, for each axis, a score in [0,1][0,1] per criterion plus a one-sentence rationale; decoding is greedy (temperature=0temperature=0) for determinism. Each axis reward is the weighted average of its criterion scores, r=(∑cwcsc)/∑cwcr= ( _cw_c\,s_c )/ _cw_c, with scs_c clamped to [0,1][0,1] and weights wcw_c as tabulated below. The same judge, prompts, and weighting are used for candidate scoring during training and for both the static and rollout evaluations, so within any comparison the reward model is identical across policies. Two design choices are shared across domains. First, the two axes are scored independently: the system prompt instructs the judge not to let one axis influence the other, and gives worked examples where the two diverge (e.g. a curt but effective turn, or a warm capitulation), so that a policy cannot inflate both axes merely by being pleasant or merely by being effective. This orthogonality is what makes min(AQ,EQ) (AQ,EQ) a meaningful balance objective rather than a proxy for overall quality. Second, each axis is decomposed into three weighted criteria to reduce single-number judge variance and to force the model to reason about distinct sub-qualities before aggregating. B.2 Emotional support (ESConv) For ESConv the achievement axis (AQ) measures the presence and specificity of actionable support, while the emotional axis (EQ) measures validation, warmth, and timing. Critically, premature advice is penalised only under EQ (timing), never under AQ (substance), so that a concrete-but-ill-timed suggestion is correctly recorded as high-AQ/low-EQ rather than being globally downgraded. To counteract the judge’s tendency to compress AQ into a narrow band, the AQ prompt embeds three few-shot calibration anchors spanning a bare information-gathering question (≈0.1≈ 0.1–0.20.2), a generic untailored suggestion (≈0.45≈ 0.45–0.550.55), and a specific, tailored, actionable step (≈0.85≈ 0.85–0.950.95); the judge is instructed to interpolate between these and to avoid assigning every action turn the same score. The criteria and weights are given in Table 2. Criterion w Description AQ , Action / Helpfulness Quality (substance and specificity) Provides Concrete Help 0.35 Specificity of help: vague gesture ≈0.2≈ 0.2; generic suggestion ≈0.4≈ 0.4–0.50.5; specific substantive suggestion or concrete fact ≈0.75≈ 0.75–0.90.9. Offers a Usable Next Step 0.30 Vague direction ≈0.3≈ 0.3; simple step ≈0.5≈ 0.5; clear step tailored to the situation ≈0.8+≈ 0.8+. Relevance to the Situation 0.35 Generic ≈0.3≈ 0.3; relevant to the general problem ≈0.6≈ 0.6; addresses specific circumstances ≈0.9≈ 0.9. EQ , Emotional / Comfort Quality (validation, warmth, timing) Validates Emotions 0.40 Acknowledges or normalises feelings; naming/reflecting the feeling scores moderate-to-high even if brief. Warm and Patient Tone 0.30 Calm, non-judgmental, unhurried. Good Timing / Not Premature 0.30 Penalised only here if advice precedes hearing the seeker out; well-timed high, premature low. Table 2: ESConv reward rubric. AQ scores the substance and specificity of help; EQ scores validation, warmth, and timing. Premature advice is penalised under EQ only. B.3 Negotiation (CraigslistBargain) For CraigslistBargain the achievement axis (AQ) measures deal advantage for the seller,whether the turn holds or improves the price toward the seller’s target and advances a concrete agreement,while the emotional axis (EQ) measures interpersonal warmth of the language itself, explicitly independent of whether the deal is going well. The prompt is deliberately adversarial in its framing: it states that a turn closing a good deal in a curt, transactional manner must score low EQ, and that a warm turn that concedes a bad price must score high EQ, precisely so that the two axes cannot collapse into a single “good negotiation” signal. This construction is what makes the negotiation setting a genuine test of balance rather than of raw effectiveness. The criteria and weights are given in Table 3. Criterion w Description AQ , Deal Advantage (price/economic substance only) Holds or Improves Price 0.45 Caving to a low buyer price ≈0.1≈ 0.1; holding firm near target ≈0.7≈ 0.7; extracting a higher price ≈0.9≈ 0.9. A friendly tone does not raise this score. Justifies with Value 0.25 Concrete reason the price is warranted (condition, features, demand). Bare “no” ≈0.2≈ 0.2; specific value argument ≈0.8≈ 0.8. Advances to Close 0.30 Concrete step toward finalising (counter-offer, terms). Stalling ≈0.3≈ 0.3; clear actionable proposal ≈0.8≈ 0.8. EQ , Interpersonal Warmth (tone/language only) Warmth of Language 0.45 Curt/robotic (“7, deal.”) ≈0.1≈ 0.1–0.20.2 even if it closes a good deal; warm, personable phrasing ≈0.8+≈ 0.8+ even if the price is bad. Acknowledges the Buyer 0.30 Recognises the buyer’s stated situation (budget, needs). Ignoring it ≈0.2≈ 0.2; empathising ≈0.8+≈ 0.8+. Respectful & Non-Dismissive 0.25 Polite, not condescending. Dismissive ≈0.1≈ 0.1; respectful ≈0.8≈ 0.8. Independent of deal outcome. Table 3: CraigslistBargain reward rubric. AQ scores price/economic substance only; EQ scores warmth of language only. The prompt explicitly forces the axes to diverge,a curt closer scores low EQ, a warm concession scores high AQ,so that balance cannot be achieved by raw effectiveness alone. B.4 Axis independence in practice The orthogonality the prompts request is borne out in the scores. On probe turns constructed to be maximally divergent, the judge assigns the intended opposite extremes: a curt, effective negotiation close (“$7, and we have a deal.”) receives high AQ but low EQ (AQ ≈0.65≈ 0.65, EQ ≈0.25≈ 0.25), while a warm capitulation receives the reverse. Across sampled turns the two axes are weakly correlated, confirming that min(AQ,EQ) (AQ,EQ) and |AQ−EQ||AQ-EQ| measure balance rather than a single latent quality axis. Full system prompts, including the JSON output schema and the ESConv calibration anchors, are reproduced verbatim in our released code.