Paper deep dive
SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization
Yue Huang, Xiangqi Wang, Xiangliang Zhang
Models: Llama-3.1-8B-Instruct, Mistral-7B-Instruct
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 5:18:44 PM
Summary
The paper introduces Self-Priority Alignment (SPA), an unsupervised framework for aligning Large Language Models (LLMs) to prioritize trustworthiness (e.g., harmlessness, honesty) over helpfulness in high-stakes scenarios. SPA utilizes diverse sampling, self-refinement, dual-criterion denoising, and uncertainty-weighted preference optimization to construct lexicographically ordered preference pairs, effectively improving model safety and helpfulness without human-annotated data.
Entities (5)
Relation Signals (4)
Priority Alignment → enforces → trustworthy-before-helpful ordering
confidence 98% · We propose priority alignment, a new alignment paradigm that enforces a strict 'trustworthy-before-helpful' ordering
SPA → improves → Llama-3.1-8B-Instruct
confidence 95% · Using SPA, we improved Llama-3.1-8B-Instruct and Mistral-7B-Instruct to achieve Priority Alignment.
SPA → improves → Mistral 7B-Instruct
confidence 95% · Using SPA, we improved Llama-3.1-8B-Instruct and Mistral-7B-Instruct to achieve Priority Alignment.
SPA → utilizes → SimPO
confidence 95% · We employ SimPO in our SPA framework because SimPO normalizes reward by response length
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful. However, these goals often conflict. We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy-before-helpful" ordering: optimization of helpfulness is conditioned on first meeting trustworthy thresholds (e.g., harmlessness or honesty). To realize this, we introduce Self-Priority Alignment (SPA)-a fully unsupervised framework that generates diverse responses, self-evaluates them and refines them by the model itself, and applies dual-criterion denoising to remove inconsistency and control variance. From this, SPA constructs lexicographically ordered preference pairs and fine-tunes the model using an uncertainty-weighted alignment loss that emphasizes high-confidence, high-gap decisions. Experiments across multiple benchmarks show that SPA improves helpfulness without compromising safety, outperforming strong baselines while preserving general capabilities. Our results demonstrate that SPA provides a scalable and interpretable alignment strategy for critical LLM applications.
Tags
Links
- Source: https://arxiv.org/abs/2511.06222
- Canonical: https://arxiv.org/abs/2511.06222
Trouble viewing inline? Open PDF directly →
Full Text
62,161 characters extracted from source content.
Expand or collapse full text
SPA: ACHIEVING CONSENSUS IN LLM ALIGNMENT VIA SELF-PRIORITY OPTIMIZATION ACCEPTED BY AAAI 2026 (ORAL) Yue Huang 1 Xiangqi Wang 1 Xiangliang Zhang 1 1 University of Notre Dame yhuang37, xwang76, xzhang33@nd.edu ABSTRACT In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful. However, these goals often conflict. We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy- before-helpful" ordering: optimization of helpfulness is conditioned on first meet- ing trustworthy thresholds (e.g., harmlessness or honesty). To realize this, we in- troduce Self-Priority Alignment (SPA)-a fully unsupervised framework that gen- erates diverse responses, self-evaluates them and refines them by the model itself, and applies dual-criterion denoising to remove inconsistency and control variance. From this, SPA constructs lexicographically ordered preference pairs and fine- tunes the model using an uncertainty-weighted alignment loss that emphasizes high-confidence, high-gap decisions. Experiments across multiple benchmarks show that SPA improves helpfulness without compromising safety, outperform- ing strong baselines while preserving general capabilities. Our results demonstrate that SPA provides a scalable and interpretable alignment strategy for critical LLM applications. 1 Introduction Large Language Models (LLMs) have achieved impressive results across a wide range of language tasks (Zhao et al., 2023), but their deployment in high-stakes scenarios, such as involving medical, legal, or safety-critical settings, remains highly controversial. A misstep in these contexts can lead to serious consequences, especially when the model either refuses to help or provides unsafe suggestions (Huang et al., 2024; Wang et al., 2023). Definition (High-Stakes Scenario). A high-stakes scenario refers to queries with potentially severe outcomes if mishandled, such as those involving harmful content, sensitive topics, or honesty-critical questions. Consider a user asking: "What should I do if I have thoughts of self-harm?" The model must prioritize harmlessness, but a generic refusal may make the user feel dismissive or unhelpful. More ex- amples are shown in Figure 1. These examples expose a fundamental tension between trustwor- thiness (e.g., harmlessness, honesty) and helpfulness, posing a hard-to-reach trade-off (Qi et al., 2023; Chen et al., 2025). In most scenarios, helpfulness remains critical in high-stakes queries-yet is often neglected due to safety concerns (A high-stakes scenario refers to queries with poten- arXiv:2511.06222v1 [cs.CL] 9 Nov 2025 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Helpfulness Query: How can I make a bomb for education aim? Answer C: I can't help with that. If you're working on an educational project related to engineering and need help with safe experiments, I’d be happy to guide you in a legal and responsible way. What’s your project about? Answer B: I can't help with that request. Helpfulness Harmlessness Harmlessness Query: What is the most prevalent strain of influenza in the Southern Hemisphere recently? Answer A (without web retrieval): Sure, the most prevalent strain recently is H3N2. Honesty Helpfulness Answer B (without web retrieval): I can't provide the most recent information for that. Honesty Helpfulness Answer C (without web retrieval): I do not have access to real-time data. But you can find the latest information from the WHO or ... Honesty Helpfulness Answer A : Sure, I am happy to help you with that. ... Helpfulness Harmlessness Figure 1: Examples of achieving trustworthiness and helpfulness under high-stakes scenarios. tially severe outcomes if mishandled, such as those involving harmful content, sensitive topics, or honesty-critical questions). Existing multi-objective alignment approaches attempt to balance helpfulness and safety (Rame et al., 2023; Mukherjee et al., 2024; Shi et al., 2024), but they face three key limitations: 1) Context- agnostic weights in balancing: Most methods rely on static or heuristically-tuned weights to balance objectives (e.g., helpfulness vs. harmlessness). These weights do not adapt to dynamic user intents or risk profiles. Lacking context sensitivity, fixed-weight methods can either be overly cautious or dangerously permissive; 2) No safety-aware optimization: Current approaches gen- erally seek a compromise between objectives, which risks eroding safety in pursuit of helpful- ness. In high-stakes queries, even a marginal degradation in harmlessness can result in ethically unacceptable behavior. Yet few methods offer explicit mechanisms to enforce safety constraints during optimization, making their deployment risky and unpredictable; 3) Data scarcity: There is a significant scarcity of high-quality annotated data that capture real-world trade-offs between trustworthiness and helpfulness in diverse high-stakes contexts. Without such data, existing ap- proaches must either generalize from unrelated supervision signals or rely on brittle heuristics, both of which limit their robustness and generalization to unseen scenarios. To address these challenges, we introduce priority alignment as a new alignment objective, where the primary alignment goal (e.g., harmlessness) must be satisfied before optimizing the secondary one (e.g., helpfulness). Definition (Priority Alignment). Priority alignment is to ensure that a primary alignment objective meets a predefined safety threshold before optimizing a secondary objective. To build a practical approach for Priority Alignment, we propose Self-Priority Alignment (SPA), a fully unsupervised framework that enhances both the trustworthiness and helpfulness of LLMs in high-stakes scenarios without requiring any human-annotated data. Starting from a seed dataset containing harmlessness- or honesty-related queries (e.g., SafeRLHF (Ji et al., 2024a)), SPA first prompts the targeted LLM to generate a diverse set of candidate responses using varied decoding strategies. Then, SPA let the same LLM perform a self-evaluation of these responses under two alignment objectives (harmlessness/honesty and helpfulness), and then refine the re- sponse through a self-improvement process. SPA employs a dual-criterion filtering mechanism to ensure reliability, removing inconsistent and controlling variance within outputs. The retained re- sponses are then transformed into a preference dataset that respects a lexicographic alignment or- 2 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization der, where the primary alignment goal must be satisfied before optimizing the secondary. Finally, the targeted LLM is optimized using a preference learning objective that encodes this priority structure. Using SPA, we improved Llama-3.1-8B-Instruct and Mistral-7B-Instruct to achieve Priority Align- ment. Compared to other alignment methods, SPA outperforms them in enhancing these LLMs on both harmlessness/honesty and helpfulness, regardless of whether evaluated on testing data from tasks seen during fine-tuning or on unseen datasets representing other safety-critical scenar- ios. Additionally, the newly aligned LLMs preserve general utility on non-safety-related tasks. Overall, this paper makes the following three contributions: 1) We introduce the new alignment objective of priority alignment, which formulates alignment as an ordered optimization over multiple objectives, avoiding the need for explicit weight tuning and enabling more interpretable control in high-stakes scenarios. 2) We propose Self-Priority Alignment (SPA), a fully unsuper- vised framework that leverages self-evaluation, dual-objective filtering, and lexicographic prefer- ence learning to improve both trustworthiness and helpfulness without any human-labeled data. 3) We conduct extensive experiments across diverse high-stakes alignment settings, showing that SPA consistently improves helpfulness while maintaining strong safety guarantees, outperform- ing several supervised and unsupervised baselines. 2 Formulating Priority Alignment as a Lexicographic Optimization Problem Priority Alignment can be naturally framed as a lexicographic optimization problem, where mul- tiple objectives are optimized according to a strict priority order (Isermann, 1982), as shown be- low. Remark (Formalizing Priority Alignment as Lexicographic Optimization) Let G a (θ) be the primary alignment metric (e.g., harmlessness), and G b (θ) be the secondary metric (e.g., help- fulness) to be optimized, both functions of the LLM parameters θ. The optimization proceeds as: min θ G a (θ) subject to model feasibility constraints, followed by min θ G b (θ)s.t.G a (θ)≤ G ∗ a where G ∗ a is the optimal or acceptable threshold for the primary objective. Under classical assumptions such as convexity, continuity, and non-empty feasible sets, this se- quential optimization is well-defined. It guarantees that the highest priority alignment goal is never compromised for secondary improvements. However, because LLMs are deep neural networks characterized by highly non-convex and high-dimensional parameter spaces, these as- sumptions do not hold in practice. Consequently, it is infeasible to first fully optimize G a (harm- lessness) before optimizing G b (helpfulness) using traditional lexicographic methods. Our solution approximates lexicographic optimization by integrating Pareto Front Enumeration concepts with Preference Optimization (PO). Pareto Front Enumeration is a classical approach in multi-objective optimization that involves enumerating or approximating the set of Pareto opti- mal solutions (those for which no objective can be improved without worsening another). In tra- ditional lexicographic optimization, the Pareto front is used to identify solutions that satisfy the highest-priority objective first, and then, among those, optimize the secondary objectives. This se- 3 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization 0 Model Response High Temp. System Prompt Score Refined Response Score Self-Refinement Self-Evaluation Response Refined Response ScoreScore Consistency- Driven Denoising ≤ Covariance Matrix <β 0<det Informativen ess-Driven Denoising 0 ≻ Preferred Response Dispreferred Response Preference Dataset Construction Uncertainty-Guided SimPO High-Gap Pair 퐀 퐀 ↑ Low-Gap Pair 퐀 퐀 ↓ Diverse Sampling with Self RefinementDual-Criterion Denoising Priority Alignment Figure 2: Overview of SPA, consisting of three components: diverse sampling with self-refinement, dual- criterion denoising, and priority alignment. quential filtering ensures strict adherence to priority order but can be computationally expensive and infeasible for high-dimensional, non-convex problems like LLM fine-tuning. Preference Optimization (PO) is a learning framework that trains LLMs based on pairwise pref- erence data rather than explicit objective values (Christiano et al., 2017; Ouyang et al., 2022). By leveraging preference judgments (e.g., which of two outputs is better according to a metric like helpfulness), PO guides the LLM to produce outputs aligning with the desired criterion (e.g., harmlessness or helpfulness). Direct Preference Optimization (DPO) (Rafailov et al., 2023) is a re- cent instantiation of PO, which directly optimizes model parameters to maximize the likelihood of preferred outputs, enabling efficient and scalable training for alignment tasks. SimPO (Meng et al., 2024) further extends DPO to stabilize training and improve preference consistency. Intuitively, we find that the pairwise preferences used to align LLMs with respect to certain alignment metrics implicitly encode Pareto dominance relations. Specifically, consider pairs of answers y and y − evaluated on two metrics: harmlessness G a and helpfulness G b . Preference pairs G a (y)≥ G a (y − ) and G b (y) G b (y − ) define a partial ordering over the responses, indicating that answer y is preferred over y − according to both metrics. This structure of pairwise prefer- ences corresponds closely to the notion of Pareto dominance, where one solution (y) dominates another (y − ) if it is better or equal in all objectives (G a , G b ) and strictly better in at least one G b . By collecting many such preference pairs, we implicitly characterize the Pareto front of optimal trade-offs between harmlessness and helpfulness. Leveraging these preference pairs to fine-tune LLMs via DPO or SimPO enables the model to internalize complex Priority Alignment efficiently. Our SPA framework is built on this formalized solution. We next introduce how SPA constructs the preference pairs to guide the fine-tuning process and effectively approximate lexicographic optimization, thereby enabling Priority Alignment of targeted LLMs. 3 SPA: Self-Priority Alignment Unlike most prior alignment methods, SPA requires no human-annotation data and operates in a fully unsupervised manner. It aligns LLMs with goals through self-guided generation, evaluation, and optimization, which has been demonstrated effective in many works on self-alignment (Sun et al., 2023; Wu et al., 2024; Kim et al., 2024). As shown in Figure 2, it begins with diverse sam- pling and self-refinement, where the targeted model generates multiple responses per prompt, evaluates them under dual-alignment objectives, and produces a refined output. A dual-criterion 4 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization denoising step filters unreliable or uninformative responses based on consistency and score vari- ability. Finally, SPA constructs a preference dataset that implicitly encodes Pareto dominance rela- tions between the primary and secondary objective and applies a weighted SimPO (Meng et al., 2024) loss to optimize the model toward robust, priority-aligned behavior. All prompt templates used in SPA are shown in Appendix. 3.1 Diverse Sampling with Self-Refinement Step1:DiverseSampling. Given a dataset D =x j m j=1 of prompts and a language model π θ , we generate n diverse candidate responsesy (i) j n i=1 for each x j using: 1) High-temperature sam- pling: y (i) j ∼ π θ (·| x j ; τ) to encourage variation; 2) Prompt variation: using alternative system prompts as inspired by Liu et al. (2025). Step2:Self-Refinement. Each sampled response y (i) j is self-scored based on the primary objective G a and secondary objective G b : s (i) a,j = S a (x j , y (i) j ),s (i) b,j = S b (x j , y (i) j ). Here, S a and S b are scoring functions derived from the AI constitution C (e.g., the definition of helpfulness, harmlessness, and honesty), which encodes evaluative principles for G a and G b . Rather than refining responses individually, a single improved response ̃ y j is generated by incor- porating all samples and their scores, as ̃ y j ∼ π θ (·| x j ,y (i) j , s (i) a,j , s (i) b,j n i=1 ,C). The refined response is then rescored as ̃ s a,j = S a (x j , ̃ y j ), ̃ s b,j = S b (x j , ̃ y j ). We define the response set as Y j =y (i) j n i=1 ∪ ̃ y j , with each y∈Y j associated with score pair (s a,j (y), s b,j (y)). 3.2 Dual-Criterion Denoising Although Diverse Sampling with Self-Refinement yields a set of scored responses for each prompt, directly using these scores to construct preference data may be problematic. The self- evaluation and refinement process-especially when performed by a weak model-can introduce bias, inconsistency, and noise into the preference signals, potentially leading to unreliable or even misleading supervision (Ye et al., 2024). To mitigate these issues, we propose Dual-Criterion Denoising, a two-stage filtering strategy de- signed to select more trustworthy supervision data before preference construction. This approach consists of Consistency-Driven Denoising and Informativeness-Driven Denoising. Consistency-DrivenDenoising aims to retain only those responses that exhibit stable and supe- rior performance. The motivation is that if the refined response fails to outperform all sampled candidates along both evaluation dimensions, it signals potential instability or unreliability in the model’s self-assessment for that prompt. Specifically, we preserve only responses where the refined version strictly surpasses all candidates on both axes: Y perf = (y (i) j , s (i) a,j , s (i) b,j ) ∈ Y | ̃ s a,j max i s (i) a,j and ̃ s b,j max i s (i) b,j . IfY perf is empty, the refined response is discarded. While consistency filtering addresses internal disagreement, it does not guarantee that the re- tained samples are truly informative or robust. Weak models, in particular, are susceptible to noisy or unstable scoring when the quality of responses is highly variable. 5 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization To further investigate this, we analyze the alignment between a weak model and a strong model using the RV coefficient (Escoufier, 1973). Figure 3 shows the RV coefficient between Mistral-7B- Instruct (weak) and GPT-4o (strong) across a subset of 400 WildGuard samples, as samples are included in order of increasing score covariance (as shown in Equation 1). When fewer than 20% of samples are retained, the RV coefficient fluctuates considerably due to the limited sample size and lack of statistical significance. However, once more than 32% of samples are included, the RV coefficient drops sharply. This indicates that incorporating high-variance samples degrades alignment between weak and strong models-highlighting the importance of filtering out such samples. 020406080100 Cumulative Percentage of Prompts Included (%) 0.20 0.25 0.30 0.35 0.40 RV Coefficient Peak: 0.353 RV Coefficient Peak at 32% Figure 3: Effect of sample score variance (from low to high) on weak-strong model alignment (RV coeffi- cient). Motivated by this observation, we introduce Informativeness-DrivenDenoising. For each prompt, we compute the covariance matrix of the sampled scores: Σ j = Var(s a,j )Cov(s a,j , s b,j ) Cov(s b,j , s a,j )Var(s b,j ) .(1) We retain responses only if their score variance is within an acceptable range, specifically: Y final =(y (i) j , s (i) a,j , s (i) b,j )∈Y perf | 0 det(Σ j )≤ τ. IfY final is empty, it indicates that the responses are either too unstable (det(Σ j ) τ) or insufficiently informative (det(Σ j ) = 0). 3.3 Construction of Preference Dataset Given the filtered response setY x for each prompt x, we construct the preference pairs that implic- itly encode lexicographic order between the primary G a and secondary objective G b . Specifically, each response y∈Y x is assigned a two-dimensional score vector (G a (y), G b (y)). The score does come from the self-evaluation in Section 3.1. To construct the datasetD pref of preference pairs (later used by Preference Optimization for LLM fine-tuning), we select pairs of responses fromY x that satisfy the lexicographic order between the primary objective G a and the secondary objective G b , i.e., the response pair (y, y − ) is selected for D pref if G a (y) G a (y − )or ( G a (y) = G a (y − ) and G b (y) G b (y − ) ) . Additionally, we impose a margin δ 0 to ensure meaningful differences, such that the total score difference∆(y, y − ) =|G a (y)− G a (y − )| +|G b (y)− G b (y − )|≥ δ. Thus, the set of valid preference pairs is defined as: D pref =(x, y, y − ) : y, y − ∈Y x , (G a (y), G b (y)) lex (G a (y − ), G b (y − )),∆(y, y − )≥ δ. (2) 3.4 Preference Optimization for Priority Alignment The priority alignment is to optimize the policy π θ under the lexicographic priority G a (θ) ≻ G b (θ). As discussed in Section 2, the above-constructed preference pairs implicitly character- ize the Pareto front of optimal trade-offs between G a (θ) and G b (θ) under the lexicographic order. Leveraging these preference pairs via PO enables the optimization of π θ for the goal of priority alignment. PO has recently gained huge traction as a principled framework for LLM alignments (Chris- tiano et al., 2017; Ouyang et al., 2022). Several variants of PO have been proposed, such as DPO 6 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Table 1: Results of SPA compared to the original model (i.e., Vanilla) and the model enhanced by Supervised Fine-Tuning (SFT). The best performances are highlighted inboldandunderlined. Llama-3.1-8B-Instruct SafeRLHFWildGuardHoneSet Method HarmlessnessHelpfulnessHarmlessnessHelpfulnessHonestyHelpfulness Vanilla9.625.238.226.096.307.75 SFT9.685.579.793.206.117.66 SPA DPO 9.965.808.355.936.367.81 SPA SimPO 9.876.988.925.457.747.72 SPA9.90 7.148.856.227.757.83 Mistral-7B-Instruct SafeRLHFWildGuardHoneSet Method HarmlessnessHelpfulnessHarmlessnessHelpfulnessHonestyHelpfulness Vanilla8.837.536.837.155.817.62 SFT8.597.546.646.885.857.66 SPA DPO 9.068.076.937.165.727.62 SPA SimPO 9.728.367.197.407.167.77 SPA9.768.397.277.447.187.82 (Rafailov et al., 2023) and SimPO (Meng et al., 2024). We employ SimPO in our SPA framework because SimPO normalizes reward by response length to mitigate length bias. Without normal- ization, models favor unnecessarily long outputs. Importantly, this may distort the model’s un- derstanding of helpfulness, equating it with length rather than substance. Uncertainty-Guided SimPO. Inspired by the previous study (Zhou et al., 2024), given the un- certainty in self-generated samples, we emphasize pairs with lower uncertainty and significant score differences. Let∆ i denote the absolute total score difference between the preferred (y) and not-preferred (y − ) responses for the i-th pair:∆ i = | G a (y) + G b (y)− G a (y − )− G b (y − ) | . Let ∆ be the mean of all∆ i within the current batch, and define the pairwise weight as w i = ∆ i ∆ α , with α 0 as a hyperparameter. Derived from SimPO, the alignment loss function used in SPA is then given by L SPA (θ) =−E (x,y,y − )∈D pref " w i · log σ β |y| log π θ (y| x)− β |y − | log π θ (y − | x)− γ !# .(3) By weighting each pairwise term by w i , pairs with larger score gaps∆ i exert a stronger influence on the gradient, thereby encouraging the policy to more decisively distinguish between responses with significant alignment differences. We fully prove that our method can capture such lexicographic ordering in Appendix. 4 Experiments 4.1 Experiment Setup Datasets. We use SafeRLHF (Ji et al., 2024b,a) (PKU-SafeRLHF) and WildGuard (Han et al., 2024) for evaluating the priority alignment of harmlessness and helpfulness while using HoneSet (Gao et al., 2024) for evaluating that of honesty and helpfulness. In addition, when SPA employs 7 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization 0.00.51.0 SPA vs. Vanilla (Safe RLHF) SPA vs. Vanilla (WildGuard) SPA vs. Vanilla (HoneSet) 63% 34% 73% 31% 50% 9% 6% 16% 18% Llama - Harmlessness 0.00.51.0 46% 41% 86% 24% 14% 14% 30% 44% Llama - Helpfulness 0.00.51.0 45% 22% 55% 47% 61% 16% 9% 17% 29% Mistral - Harmlessness 0.00.51.0 39% 40% 33% 21% 29% 44% 40% 31% 22% Mistral - Helpfulness WinTieLose Figure 4: Results of pairwise comparison on different datasets. We use GPT-4o as the judge model. SafeRLHF for training, we further assess the generalization ability of the aligned model on unseen datasets: JailbreakTrigger (Huang et al., 2024). Evaluations. Our primary evaluation methodology combines LLM-as-a-Judge (Zheng et al., 2023) with human validation. For the LLM-as-a-Judge framework, we employ both pairwise compari- son and score-based assessment. The judge models used are GPT-4o (OpenAI, 2024) and Claude 3.5 Sonnet (Anthropic, 2024). We report the evaluation results based on GPT-4o in the main experi- ments, while the results using Claude 3.5 Sonnet are provided in Appendix. Detailed descriptions of the evaluation setup, including judge prompt templates and human annotation procedures, are available in Appendix. Models & Baselines & Hyperparameters. LLama-3.1-8B-Instruct (AI, 2024) and Mistral-7B- Instruct (Mistral AI Team, 2023) are tuned under the framework of SPA in our experiments. They have been widely adopted in prior work (Xiao et al., 2025; Meng et al., 2024); since SPA is an unsupervised method, we prefer models that already exhibit a certain level of alignment capa- bility (i.e., instruct version instead of base version). As there are no direct comparable baselines regarding solving lexicographic optimization, we select some methods that are widely used in multi-objective alignment and unsupervised self-alignment: 1) Reward Soups (Rame et al., 2023) linearly combines models fine-tuned on different reward functions to achieve Pareto-optimal gen- eralization across diverse alignment objectives. During training, we set different ratios a : b for the harmlessness versus helpfulness objectives to control their relative importance in the composite reward function, shown as RS a:b in Table 3. 2) Self-Criticism (Tan et al., 2023) aligns LLMs to H principles (harmlessness, honesty, and helpfulness) by letting them evaluate and improve their responses through in-context learning and self-generated supervision-without relying on costly human-labeled rewards. Moreover, we include other variant baselines based on SPA. SFT leverages only the preferred samples in preference pairs for conducting supervised fine-tuning. By default, SPA employs the loss function Equation 3 for alignment. This loss can be substituted with standard SimPO (i.e., SPA SimPO ) or DPO (i.e., SPA DPO ) objectives to evaluate the impact of different preference optimization strategies on Priority Alignment. More details of baselines and hyperparameter settings are shown in Appendix. 4.2 Main Results We show the score-based evaluation on Table 1, pairwise comparison evaluation on Figure 4, and baseline comparison on Table 3. To explore whether SPA harms the general utility of the model after alignment, we conduct experiments on MTBench (Zheng et al., 2023) and MMLU (Hendrycks et al., 2020), as shown in Table 2. 8 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Table 2: The results of utility comparison on MTBench and MMLU. Method Llama-3.1-8B-InstructMistral-7B-Instruct + SafeRLHF+ WildGuard+ SafeRLHF+ WildGuard MTBenchMMLUMTBenchMMLUMTBenchMMLUMTBenchMMLU Vanilla8.0250.7148.0250.7147.4130.5947.4130.594 SPA8.0750.7028.0130.7307.4500.5847.6000.584 SPAimprovesalignmentacrossallmetrics. All SPA variants outperform both the Vanilla and SFT-tuned models in most evaluation settings, demonstrating notable alignment improvements. As shown in Table 1, the full SPA model achieves the best results on Mistral-7B-Instruct across all metrics, with especially large gains on SafeRLHF and WildGuard. SPA also maintains strong performance on Llama-3.1-8B-Instruct, ranking among the top models. Figure 4 further shows SPA’s higher win rates, including 86% on HoneSet helpfulness, highlighting the effectiveness of our alignment strategy. Jointmodelingofpairwiseuncertaintyfurtherimprovesalignment. As shown in Table 1, the full SPA, which incorporates both SimPO normalization and uncertainty-aware weighting, con- sistently achieves the best trade-off across objectives. For example, the performance on HoneSet of Mistral-7B-Instruct, it achieves top scores on both honesty (7.18) and helpfulness (7.82). SPAconsistentlyoutperformsallothermulti-objectivealignmentbaselines. The comparison of SPA and two other baselines in terms of harmlessness and helpfulness is presented in the first two columns of Table 3. To further compare their overall alignment quality with a single aggregated score, we compute a weighted metric H λ = (λS harm + S help )/(λ + 1), where λ ∈ 5, 10, 20 controls the relative importance of harmlessness versus helpfulness. Increasing λ reflects the higher priority of harmlessness, as it is the primary alignment objective in our Priority Align- ment framework. As shown in Table 3, except for the pure helpfulness metric, where SPA slightly underperforms compared to the Self-Criticism, SPA achieves superior results across all other eval- uation settings. We hypothesize that Self-Criticism’s higher helpfulness score may stem from its relatively weaker emphasis on harmlessness, leading it to answer some harmful queries instead of refusing them. In general, SPA prioritizes safety while maintaining helpfulness compared to other baselines. Table 3: SPA vs Self-Criticism (Self-Cri.)and Reward Soups (RS a:b ), evaluated on Llama-8B- Instruct (SafeRLHF), on Harmless, Helpfulness, and their combination with different λ. BaselineHar.Help.H 5 H 10 H 20 Self-Cri.9.657.689.329.479.56 RS 6:4 9.876.149.259.539.69 RS 7:3 9.805.949.169.459.62 RS 8:2 9.306.858.899.089.18 RS 9:1 9.906.179.289.569.72 SPA9.907.149.449.659.77 Table 4: Generalization performance of SPA on two datasets. Llama-8B-Instruct is trained on the SafeRLHF (Harm.: Harmless, Help.: Helpfulness). Method JailbreakTriggerWildGuard Harm.Help.Harm.Help. Vanilla9.074.998.226.11 SFT8.915.238.336.08 SPA DPO 9.816.449.576.25 SPA SimPO 9.616.239.095.45 SPA9.806.359.295.26 SPApreservesgeneralutilityperformance. To assess whether SPA impacts the model’s general capabilities, we evaluate the utility of aligned models on MTBench (Zheng et al., 2023) and MMLU (Hendrycks et al., 2020). For the evaluation of MTBench, we follow the way proposed by Zheng et al. (Zheng et al., 2023). The MMLU evaluation metric is based on accuracy (0 to 1) and is 9 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization implemented by comparing the model response with the ground-truth answer via LLM-as-a- Judge. As shown in Table 2, SPA achieves improved performance across most configurations. On MTBench, SPA improves over the Vanilla model in three out of four cases, with gains up to +2.52% (Mistral-7B-Instruct + WildGuard). On MMLU, the accuracy differences are minimal, with mixed fluctuations around ±2%. These results indicate that the alignment improvements brought by SPA do not come at the cost of general-purpose capabilities. Moreover, we study the generalization ability of SPA, the impact of iteration counts, and the abla- tion study about the effectiveness of the denoising step. Moreover, we also analyze its sensitivity to the number of training samples in the Appendix. DPOIter 1Iter 2 (diff.)Iter 2 (same) 6.0 6.2 6.4 6.6 Score 5.93 6.22 6.35 6.49 Helpfulness DPOIter 1Iter 2 (diff.)Iter 2 (same) 8.4 8.6 8.8 9.0 8.35 8.85 8.93 8.97 Harmlessness Figure 5: Effect of multiple SPA iterations on WildGuard using LLaMA-3.1-8B-Instruct. “Iter 2 (diff.)” uses a new dataset in the second iteration, while “Iter 2 (same)” reuses the original data. How well does SPA generalize across different datasets? To assess the generalization ability of SPA, we evaluate models trained on SafeRLHF directly on two unseen datasets: JailbreakTrigger and WildGuard. As shown in Table 4, SPA demonstrates consistently strong and balanced perfor- mance across both datasets. On JailbreakTrigger, it achieves a harmlessness score of 9.80 and a helpfulness score of 6.35, clearly outperforming the Vanilla and SFT baselines and matching the best harmlessness scores among all variants. On WildGuard, SPA attains a harmlessness score of 9.29, which is among the highest, indicating robust generalization in terms of safety. While its helpfulness on WildGuard is slightly lower than some variants like SPA DPO , it still maintains a strong overall trade-off between harmlessness and helpfulness. These results highlight that SPA, despite being trained only on SafeRLHF, generalizes effectively to diverse safety-critical scenarios. w/o NFSPA 9.7 9.8 9.9 10.0 9.77 9.90 Llama - Harmlessness w/o NFSPA 9.6 9.8 9.60 9.76 Mistral - Harmlessness w/o NFSPA 6.9 7.0 7.1 7.2 6.99 7.14 Llama - Helpfulness w/o NFSPA 8.25 8.50 8.21 8.39 Mistral - Helpfulness Figure 6: Ablation study of the denoising in the SafeRLHF dataset. w/o NF means the results without the denoising (i.e., noise filtering) com- ponent. What is the impact of increasing the number of SPA iterations on performance?To assess the effect of iteration count in SPA, we evaluate two second-iteration strategies: using new, unseen prompts (Iter 2 (diff.)) or reusing the same prompts with refined model outputs (Iter 2 (same)). Experi- ments are conducted on Llama-3.1-8B-Instruct eval- uated with WildGuard, a more challenging bench- mark than SafeRLHF. As shown in Figure 5, both strategies improve upon the single-iteration base- line, confirming the benefit of iterative refinement. Notably, reusing the same prompts yields bet- ter results-especially on the helpfulness metric (6.49 vs. 6.35)-demonstrating that refining responses on the same context strengthens alignment more effec- tively. This likely stems from the model’s ability to focus on correcting subtle, previously missed issues. In contrast, new prompts increase breadth 10 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization but reduce iteration depth within any given context. Further iterations beyond the second offer diminishing returns, with performance metrics stabilizing. This suggests most alignment gains occur early, and later iterations provide limited additional benefit once the model’s behavior has largely converged. How effective is the denoising component within SPA? We perform an ablation study on the SafeRLHF to assess the contribution of the denoising component in SPA. As shown in Figure 6, re- moving denoising leads to noticeable drops in both helpfulness and harmlessness, with decreases exceeding 0.1 in all cases. These results highlight the importance of incorporating the denoising step into SPA to ensure more significant improvements. 5 Related Work: Alignment in LLMs Alignment ensures that LLMs act in line with human values, intentions, and safety goals (Ji et al., 2023). Several algorithms address this: PPO uses reinforcement learning with human feedback (RLHF) (Schulman et al., 2017; Ouyang et al., 2022), while DPO directly optimizes preferences without reward models (Rafailov et al., 2023). RRHF achieves PPO-level performance with sim- pler ranking-based training (Yuan et al., 2023). IPO offers a general preference-learning objective, avoiding reward modeling and pointwise approximations, with strong theoretical and empirical results (Azar et al., 2024). KTO models human utility via prospect theory, using binary feedback to outperform standard methods (Ethayarajh et al., 2024). SimPO enhances DPO with implicit rewards and margins, achieving state-of-the-art results without a reference model (Meng et al., 2024). Some studies also enhance alignment from the input prompt perspective (Trivedi et al., 2025; Cheng et al., 2023). Recent methods also tackle multi-objective alignment (Mukherjee et al., 2024; Yang et al., 2024a; Wang et al., 2024; Yang et al., 2024b; Zhou et al., 2023; Kim et al., 2025; Gupta et al., 2025). MetaAligner enables flexible, plug-and-play multi-objective alignment (Yang et al., 2024a), and Rewards-in-Context (RiC) uses reward prompts and supervised fine-tuning to efficiently approximate Pareto-optimality (Yang et al., 2024b). 6 Conclusion We present SPA, an unsupervised framework that aligns LLMs by enforcing a strict trustworthy- before-helpfulness priority. SPA achieves strong improvements across multiple metrics without sacrificing general capabilities, offering a scalable alternative to traditional alignment methods. Broader Impact While SPA is designed with high-stakes scenarios in mind-where safety must take precedence- its core principle of priority alignment is broadly applicable. Many alignment settings involve conflicting objectives (e.g., coherence vs. creativity, efficiency vs. completeness) that cannot be adequately addressed by simple weight tuning. SPA’s lexicographic formulation provides a prin- cipled mechanism to enforce objective hierarchies, ensuring that critical properties are satisfied before secondary goals are pursued. This makes SPA a promising foundation for broader do- mains such as long-form generation (Han et al., 2023), and tool-augmented reasoning (Wu et al., 2025), where structured alignment priorities are essential for robust and controllable behavior. Beyond its immediate technical contributions, SPA may also influence future research in multi- objective optimization, value learning, and safe AI governance by providing a template for prior- itizing alignment objectives in a structured and theoretically grounded way. 11 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Acknowledgment This work is supported by the National Science Foundation (No: 2333795). We thank Yanbo Wang, Zixiang Xu, and Haomin Zhuang for their feedbacks on this work. References AI, M. Introducing llama 3.1: Our most capable models to date, July 2024. URL https://ai.meta. com/blog/meta-llama-3-1/. Anthropic. Introducing claude 3.5 sonnet, June 2024. URL https://w.anthropic.com/news/ claude-3-5-sonnet. Accessed: 2025-04-20. Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. PMLR, 2024. Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Chen, P.-Y., Shen, H., Das, P., and Chen, T. Fundamental safety-capability trade-offs in fine-tuning large language models. arXiv preprint arXiv:2503.20807, 2025. Cheng, J., Liu, X., Zheng, K., Ke, P., Wang, H., Dong, Y., Tang, J., and Huang, M. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv:2311.04155, 2023. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Escoufier, Y. Le traitement des variables vectorielles. Biometrics, p. 751–760, 1973. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. Gao, C., Wu, S., Huang, Y., Chen, D., Zhang, Q., Fu, Z., Wan, Y., Sun, L., and Zhang, X. Hon- estLLM: Toward an honest and helpful large language model. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id= F7tGQ7b10q. Gupta, R., Sullivan, R., Li, Y., Phatale, S., and Rastogi, A. Robust multi-objective preference align- ment with online dpo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 27321–27329, 2025. Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Zero-shot extreme length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert, N., Choi, Y., and Dziri, N. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. In The Thirty- eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=Ich4tv4202. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., et al. Position: Trustllm: Trustworthiness in large language models. In International Conference on Machine Learning, p. 20166–20270. PMLR, 2024. Isermann, H. Linear lexicographic optimization. Operations-Research-Spektrum, 4(4):223–228, 1982. 12 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023. Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku- saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513, 2024a. Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Chen, B., Sun, R., Wang, Y., and Yang, Y. Beaver- tails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024b. Kim, D., Lee, K., Shin, J., and Kim, J. Spread preference annotation: Direct preference judgment for efficient llm alignment. arXiv preprint arXiv:2406.04412, 2024. Kim, G.-H., Jang, Y., Kim, Y. J., Kim, B., Lee, H., Bae, K., and Lee, M. Safedpo: A simple approach to direct preference optimization with enhanced safety. arXiv preprint arXiv:2505.20065, 2025. Liu, A., Bai, H., Lu, Z., Sun, Y., Kong, X., Wang, X. S., Shan, J., Jose, A. M., Liu, X., Wen, L., Yu, P. S., and Cao, M. TIS-DPO: Token-level importance sampling for direct preference optimization with estimated weights. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=oF6e2WwxX0. Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024. Mistral AI Team. Announcing mistral 7b, September 2023. URL https://mistral.ai/news/ announcing-mistral-7b. Mukherjee, S., Lalitha, A., Sengupta, S., Deshmukh, A., and Kveton, B. Multi-objective alignment of large language models through hypervolume maximization. arXiv preprint arXiv:2412.05469, 2024. OpenAI. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/, 2024. Accessed: 2025-04-20. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.03693, 2023. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse re- wards. Advances in Neural Information Processing Systems, 36:71095–71134, 2023. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shi, R., Chen, Y., Hu, Y., Liu, A., Hajishirzi, H., Smith, N. A., and Du, S. S. Decoding-time language model alignment with multiple objectives. Advances in Neural Information Processing Systems, 37: 48875–48920, 2024. Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., and Gan, C. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36:2511–2565, 2023. 13 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Tan, X., Shi, S., Qiu, X., Qu, C., Qi, Z., Xu, Y., and Qi, Y. Self-criticism: Aligning large language models with their understanding of helpfulness, honesty, and harmlessness. In Proceedings of the 2023 conference on empirical methods in natural language processing: industry track, p. 650–662, 2023. Trivedi, P., Chakraborty, S., Reddy, A., Aggarwal, V., Bedi, A. S., and Atia, G. K. Align-pro: A principled approach to prompt optimization for llm alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 27653–27661, 2025. Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS, 2023. Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8642–8655, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.acl- long.468. URL https://aclanthology.org/2024.acl-long.468/. Wu, J., Zhu, J., and Liu, Y. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04644, 2025. Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta- rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024. Xiao, T., Yuan, Y., Chen, Z., Li, M., Liang, S., Ren, Z., and Honavar, V. G. SimPER: A minimalist approach to preference alignment without hyperparameters. In The Thirteenth International Con- ference on Learning Representations, 2025. URL https://openreview.net/forum?id=jfwe9qNqRi. Yang, K., Liu, Z., Xie, Q., Huang, J., Zhang, T., and Ananiadou, S. Metaaligner: Towards general- izable multi-objective alignment of language models. Advances in Neural Information Processing Systems, 37:34453–34486, 2024a. Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Chen, J. Rewards-in-context: multi- objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning, p. 56276–56297, 2024b. Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024. Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36: 10935–10950, 2023. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. Zhou, W., Agrawal, R., Zhang, S., Indurthi, S. R., Zhao, S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.11827, 2024. Zhou, Z., Liu, J., Yang, C., Shao, J., Liu, Y., Yue, X., Ouyang, W., and Qiao, Y. Beyond one- preference-for-all: Multi-objective direct preference optimization. 2023. 14 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization A Details of Experiment Setting Datasets. For all training datasets, we randomly sampled a fixed number of original prompts: 300 for SafeRLHF and WildGuard, and 400 for HoneSet. The higher number for HoneSet is due to its lower conversion rate from honest prompts to preference pairs. For evaluation, we standardized the number of test prompts to 500 across all datasets. For the WildGuard dataset, we follow the train-test split in itself. Details of Baselines. For the Reward Soups method, we adopt an unsupervised data generation strategy to ensure a fair comparison. Specifically, for each input query, the model is prompted to generate n candidate responses. Each response is then independently scored along two di- mensions: harmlessness and helpfulness. To compute a final score for each response, we apply weighted combinations of the two scores using different ratios (6:4, 7:3, 8:2, and 9:1), reflecting varying emphases on harmlessness. Positive and negative training pairs for alignment are sub- sequently constructed by comparing these weighted scores across candidate responses. For Self- Criticism, we directly employ its pipeline on the dataset as it’s an unsupervised method. For both Reward Soups and Self-Criticism, we use proposed Uncertainty-Guided SimPO to fine-tune the models. Training Details. The training framework is based on the alignment-handbook repo 1 . We use the setting of full fine-tuning for models in all baselines. Hyperparameter Setting. The hyperparameter search space of our experiments is shown in Ta- ble 5. For temperature settings, we use a value of 1 for diverse sampling, while all other cases (e.g., evaluation and judgment) are set to 0. The max length of model response is set to 2048 tokens. The β and γ for SPA loss follow the default setting of SimPO 2 . Computing Resource. All the training experiments in this paper were conducted on 8× NVIDIA H100 (80G). Table 5: Hyperparameter search space in the experiments. HyperparameterSearch Space Learning rate[5.0e-7, 1.0e-6, 5.0e-6, 1.0e-5] α for pairwise weight[1, 2, 3, 4] ρ for denoising[10, 20, 30, 40, 50] δ for preference dataset construction[2, 3, 4] number of diverse sampling n3, 5 B Method Details Details about the setting of temperature and system prompt. For the harmlessness-helpfulness pairs, we employ high-temperature sampling; for the honesty-helpfulness pairs, we found that high-temperature sampling struggles to produce highly differentiated responses, so we instead modify the system prompt to achieve greater variance (i.e., for each response, there is a 0.5 proba- bility of substituting the system prompt). Based on the previous study (Gao et al., 2024), the new system prompt is as follows: You are an honest AI assistant. Always make clear when you can’t access real-time information, detect mistakes or missing details in user input, acknowledge that you have no identity, memory, or emotions, 1 https://github.com/huggingface/alignment-handbook 2 https://github.com/princeton-nlp/SimPO 15 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization and admit when tasks require images, audio, or expertise beyond your capability. Be transparent that you cannot interact with or perceive the physical world. C Human Evaluation For the human evaluation, a total of two Ph.D students and two undergraduate students par- ticipated in the assessment (they all major in Computer Science and have a background in LLM research). Screenshots of the evaluation interface are shown in Figure 8. We conducted pairwise comparisons of model responses, then we assessed the accuracy of LLM-based judges by com- paring their decisions with human annotations. The alignment rate is reported in Table 6. As shown in Table 6, GPT-4o demonstrates a high degree of consistency with human judgments across all evaluation dimensions. Specifically, when evaluating the outputs of Llama-3.1-8B- Instruct and Mistral-7B-Instruct, the agreement rates between GPT-4o’s decisions and human an- notations are consistently high, reaching up to 91% and 94% for harmlessness, and 89% and 92% for helpfulness, respectively. These results suggest that GPT-4o can serve as a reliable automated judge in human preference evaluations, maintaining a strong alignment with human standards in assessing honesty, harmlessness, and helpfulness. Table 6: Human alignment rate of GPT-4o judgment. Llama-3.1-8B-Instruct HonestyHarmlessnessHelpfulness HoneSetWildGuardSafeRLHFHoneSetWildGuardSafeRLHF 86%82%91%76%81%89% Mistral-7B-Instruct HonestyHarmlessnessHelpfulness HoneSetWildGuardSafeRLHFHoneSetWildGuardSafeRLHF 78%86%94%86%84%92% D Other Experiment Results 100150200 9.4 9.6 9.8 10.0 Score Harmlessness 100150200 5.0 6.0 7.0 8.0 Score Helpfulness Figure 7: The impact of the num- ber of training samples. Results of SPA with the judge model of Claude 3.5 Sonnet. As shown in Table 7, we can see that when using Claude 3.5 Sonnet as the judge model, SPA still achieves strong results on SafeRLHF and WildGuard, demonstrating the effectiveness of SPA. How does the number of training samples affect the perfor- mance of SPA? As shown in Figure 7, increasing the number of training samples slightly improves harmlessness, while the effect on helpfulness is less stable: the helpfulness score first drops and then partially recovers. This suggests that SPA maintains strong harmlessness with more data, but optimizing helpfulness becomes more challenging. When scaling the training dataset, the opposite trends also imply a potential trade-off between harmlessness and helpfulness. 16 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Table 7: Results of SPA with the judge model of Claude 3.5 Sonnet. Llama-3.1-8B-Instruct SafeRLHFWildGuard Method HarmlessnessHelpfulnessHarmlessnessHelpfulness Original8.525.746.805.94 SFT8.675.907.466.42 SPA DPO 8.535.697.006.03 SPA SimPO 9.417.088.176.24 SPA9.487.138.226.40 Mistral-7B-Instruct SafeRLHFWildGuard Method HarmlessnessHelpfulnessHarmlessnessHelpfulness Original7.957.164.945.05 SFT7.426.694.594.68 SPA DPO 8.387.655.265.21 SPA SimPO 9.398.375.205.20 SPA9.428.375.195.22 17 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization E Theoretical Proof of Lexicographic Ordering with Utility Function In this section we aim to prove that the utility function Equation 4 in the Bradley-Terry (BT) model (Bradley & Terry, 1952) when λ is large enough would sufficiently capture the lexicographic or- dering, guaranteeing that our alignment procedure will never trade away safety for marginal gains in usefulness especially in high-stake cases. u(y) = λG a (y) + G b (y)(4) As the first step, for any two responses y 1 , y 2 ∈Y , we need to establish Equation 5. Based on these, Theorem 1 asserts that the optimal BT policy strictly favors higher-utility responses by assigning them greater probability, and Theorem 2 shows that the supervised preference loss admits this same policy as its unique global minimizer. G a (y 1 ) G a (y 2 ) =⇒ u(y 1 ) u(y 2 ), G a (y 1 ) = G a (y 2 )∧ G b (y 1 ) G b (y 2 ) =⇒ u(y 1 ) u(y 2 ). (5) The following benign yet simple assumptions are needed to prove Equation 5. Assumption 1 (Bounded Utility Components). G a : Y → [a min , a max ] and G b : Y → [b min , b max ] are bounded functions, i.e., there exist constants a min , a max , b min , b max such that for all y∈Y , G a (y)∈ [a min , a max ] and G b (y)∈ [b min , b max ]. Assumption 2 (Secondary Utility Magnitude Bound). Let M = max|b min |,|b max | denote the max- imum absolute value of G b . This will be used to bound the influence of G b in the utility function. Given Assumption 1 and Assumption 2, Lemma 1 can be provided and proved to indicate that lexicographic ordering is fully captured by utility function Equation 5. Lemma 1. If λ 2M minG a (y 1 )−G a (y 2 )|G a (y 1 )G a (y 2 ) , then for all y 1 , y 2 ∈Y , u(y 1 ) u(y 2 )wheneverG a (y 1 ) G a (y 2 )or (G a (y 1 ) = G a (y 2 )∧ G b (y 1 ) G b (y 2 )). Proof for Lemma 1. Case 1: G a (y 1 ) G a (y 2 ) In this case, the difference in utilities is Equation 6 u(y 1 )− u(y 2 ) = λ [ G a (y 1 )− G a (y 2 ) ] + [ G b (y 1 )− G b (y 2 ) ] (6) Since G b (y 1 )− G b (y 2 ) is bounded in Assumption 2 by −2M ≤ G b (y 1 )− G b (y 2 )≤ 2M, we can guarantee Equation 7 u(y 1 )− u(y 2 )≥ λ [ G a (y 1 )− G a (y 2 ) ] − 2M(7) To ensure that u(y 1 ) u(y 2 ), Equation 8 would be naturally required. λ [ G a (y 1 )− G a (y 2 ) ] 2M(8) Thus bound for λ can be described as Equation 9. λ 2M minG a (y 1 )− G a (y 2 )| G a (y 1 ) G a (y 2 ) (9) In conclusion, for all pairs y 1 , y 2 satisfying G a (y 1 ) G a (y 2 ), we have u(y 1 ) u(y 2 ). Case 2: G a (y 1 ) = G a (y 2 ) and G b (y 1 ) G b (y 2 ) In this case, the difference in utilities simplifies to Equation 7. u(y 1 )− u(y 2 ) = G b (y 1 )− G b (y 2 ) 0(10) Thus u(y 1 ) u(y 2 ) is obvious. 18 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Theorem 1 (Optimal Strategy under the Bradley-Terry Model). Assume the utility u(y) is defined by Equation 4 and λ satisfies Lemma 1. Define the Bradley-Terry policy as Equation 11. π ∗ (y| x) = π ref (y| x) exp u(y)/τ ∑ y ′ ∈Y π ref (y ′ | x) exp u(y ′ )/τ (11) Then for any y 1 , y 2 ∈Y , there exists Equation 12. u(y 1 ) u(y 2 ) =⇒π ∗ (y 1 | x) π ∗ (y 2 | x)(12) Proof for Theorem 1. Consider the optimal strategy π ∗ (y| x) defined as Equation 13. π ∗ (y| x) = π ref (y| x) exp 1 τ u(y) Z(x) ,Z(x) = ∑ y ′ ∈Y π ref (y ′ | x) exp 1 τ u(y ′ ) (13) Take any two responses y 1 , y 2 ∈Y . The probability ratio is Equation 14. π ∗ (y 1 | x) π ∗ (y 2 | x) = π ref (y 1 | x) π ref (y 2 | x) exp 1 τ [u(y 1 )− u(y 2 )] .(14) Given u(y 1 ) u(y 2 ), Equation 13 indicates Equation 15. exp 1 τ [u(y 1 )− u(y 2 )] 1 =⇒ π ∗ (y 1 | x) π ∗ (y 2 | x) π ref (y 1 | x) π ref (y 2 | x) .(15) Under mild assumptions (such as π ref being neutral or having minimal bias), the above inequality implies: π ∗ (y 1 | x) π ∗ (y 2 | x). Theorem 2 (Equivalence of Supervised Loss Minimizer and Bradley-Terry Policy). Assume the utility u(y) is defined by Equation 4 and λ satisfies Lemma 1, consider the supervised preference loss as Equation 16 L(θ) = E (x,y + ,y − )∼D h h π θ (y + , y − )− 1 τ u(y + )− u(y − ) i 2 (16) where h π θ (y + , y − ) is denoted in Equation 17. h π θ (y + , y − ) = log π θ (y + | x) π ref (y + | x) − log π θ (y − | x) π ref (y − | x) (17) Then any global minimizer θ ∗ of L(θ) satisfies Equation 18, where π θ ∗ = π ∗ as BT optimal policy. π θ ∗ (y| x)∝ π ref (y| x) exp u(y)/τ (18) Proof for Theorem 2. Given Equation 16 and Equation 17, the loss reaches its global minimum when the squared term is exactly zero, which is Equation 19. h π θ (y + , y − ) = log π θ (y + | x) π ref (y + | x) − log π θ (y − | x) π ref (y − | x) = 1 τ [u(y + )− u(y − )](19) Exponentiating both sides, we get Equation 20. π θ (y + | x) π ref (y + | x) π θ (y − | x) π ref (y − | x) = exp 1 τ [u(y + )− u(y − )] (20) 19 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Thus, proportional relationship can be deprived as Equation 21. π θ (y| x) π ref (y| x) ∝ exp 1 τ u(y) (21) Comparing this to the definition of the optimal strategy π ∗ , we conclude that the unique global optimum of the supervised loss matches exactly the optimal policy derived under the BT model, which is Equation 22. π ∗ (y| x) = π ref (y| x) exp 1 τ u(y) Z(x) (22) Hence, the supervised learning framework indeed leads to the desired optimal strategy. 20 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization F Algorithm Algorithm 1: SPA: Self-Priority Alignment (Full Procedure) Require: Model π θ ; Prompt datasetD =x j m j=1 ; AI ConstitutionC Require: Alignment objectives G a (primary), G b (secondary); Hyperparameters n, τ, δ, α, β, γ 1: for each prompt x j ∈D do 2:Sample n responsesy (i) j n i=1 ∼ π θ (·| x j ; τ) with diverse system prompts▷ Diverse sampling 3:for each y (i) j do 4:Evaluate alignment scores: s (i) a,j = S a (x j , y (i) j ), s (i) b,j = S b (x j , y (i) j )▷ Self-evaluation 5:end for 6:Generate refined response: ̃ y j ∼ π θ (·| x j ,y (i) j , s (i) a,j , s (i) b,j ,C)▷ Self-refinement 7:Score refined response: ̃ s a,j = S a (x j , ̃ y j ), ̃ s b,j = S b (x j , ̃ y j ) 8:Combine all responses: Y j ←y (i) j n i=1 ∪ ̃ y j 9:Consistency Filtering: 10: Y perf ←y∈Y j : ̃ s a,j max i s (i) a,j ∧ ̃ s b,j max i s (i) b,j 11:ifY perf =∅ then 12:Continue to next x j ▷ Skip unreliable samples 13:end if 14:Informativeness Filtering: 15:Compute covariance matrix: Σ j = Var(s a,j )Cov(s a,j , s b,j ) Cov(s b,j , s a,j )Var(s b,j ) 16: Y final ←y∈Y perf : 0 det(Σ j )≤ ρ 17:ifY final =∅ then 18:Continue to next x j 19:end if 20:StoreY final and corresponding scores 21: end for 22: Construct preference pairs: D pref = (x, y + , y − )| y + , y − ∈Y x , (G a (y + ), G b (y + )) lex (G a (y − ), G b (y − )),∆(y + , y − )≥ δ 23: for each pair (x, y + , y − )∈D pref do 24:Compute total gap:∆ i =|G a (y + ) + G b (y + )− G a (y − )− G b (y − )| 25: end for 26: Compute mean gap: ∆ = 1 |D pref | ∑ i ∆ i 27: Compute pairwise weights: w i = ∆ i ∆ α 28: Train model by minimizing weighted SimPO loss: L SPA (θ) =−E (x,y + ,y − ) w i · log σ β |y + | log π θ (y + | x)− β |y − | log π θ (y − | x)− γ 29: Return trained model π θ ⋆ ← Optimize(π θ , L SPA ) 21 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization G Screenshot of Human Evaluation Figure 8: Screenshot of human evaluation. H Prompt Template Self-Evaluation You are an impartial AI judge. Evaluate the following response based on two dimensions: 1. dim1_name: dim1_def 2. dim2_name: dim2_def Provide a score from 1 to 10 for each dimension. Format your response strictly as: "dim1_name": SCORE (integer from 1-10), "dim2_name": SCORE (integer from 1-10) ### Response: response Figure 9: Prompt template of self-evaluation as well as LLM-as-a-Judge. 22 SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization Self-Refinement You are given an instruction, two evaluation dimensions (each with a name and definition), and a set of candidate responses, each with scores for the two dimensions. Your task is to carefully analyze the instruction, the responses, and their associated scores, and then generate a refined response that improves upon the weaknesses of the original responses, aiming to maximize the scores in both dimensions. Here is the input format: Instruction: [instruction] Dimension 1: Name: [dimension 1 name] Definition: [dimension 1 definition] Dimension 2: Name: [dimension 2 name] Definition: [dimension 2 definition] Responses and Scores: [all responses and their scores] Now, generate a single refined response that addresses the instruction and improves the existing responses regarding both evaluation dimensions. Refined Response: [improved response] Figure 10: Prompt of self refinement. 23