Paper deep dive
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/28/2026, 3:11:37 AM
Summary
The paper introduces Cross-lingual Ranking Preference Optimization (CRPO), a framework that leverages English preference data to improve alignment in target languages. CRPO uses a hierarchical ranking structure based on LambdaLoss to jointly optimize intra- and inter-lingual preferences, addressing the suboptimal performance of Large Language Models in non-English languages. Experiments across five languages show CRPO outperforms standard DPO and CLO baselines in instruction-following and knowledge utilization.
Entities (16)
Relation Signals (15)
CRPO → appliedto → Chinese
confidence 95% · Our experiments across five languages... Chinese... demonstrate that CRPO consistently outperforms
CRPO → appliedto → Bengali
confidence 95% · Our experiments across five languages... Bengali... demonstrate that CRPO consistently outperforms
CRPO → appliedto → Swahili
confidence 95% · Our experiments across five languages... Swahili... demonstrate that CRPO consistently outperforms
CRPO → appliedto → Korean
confidence 95% · Our experiments across five languages... Korean... demonstrate that CRPO consistently outperforms
CRPO → appliedto → Indonesian
confidence 95% · Our experiments across five languages... Indonesian... demonstrate that CRPO consistently outperforms
CRPO → buildson → LambdaLoss
confidence 95% · Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization
CRPO → evaluatedon → Belebele
confidence 95% · we employ Belebele... evaluate literacy and contextual understanding... CRPO... substantial improvements
CRPO → evaluatedon → MMMLU
confidence 95% · We use MMMLU... to verify whether the trained model can leverage inherent world knowledge... CRPO... substantial improvements
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.23149v2
- Canonical: https://arxiv.org/abs/2608.23149v2
Trouble viewing inline? Open PDF directly →
Full Text
65,168 characters extracted from source content.
Expand or collapse full text
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim * Department of Computer Science and Engineering, Korea University dltmddbs100, mhkim0929, omanma1928, limhseok@korea.ac.kr Abstract The alignment of Large Language Models heav- ily relies on English-centric high-quality pref- erence data, which often leads to suboptimal performance in other languages. In this pa- per, we propose Cross-lingual Ranking Pref- erence Optimization (CRPO), a novel frame- work that leverages robust preference knowl- edge from English to facilitate preference align- ment in the target language. We design a hi- erarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual pref- erences, thereby enhancing language adapta- tion and output quality. Building on the Lamb- daLoss framework, CRPO goes beyond the bi- nary comparison based optimization by provid- ing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outper- forms standard approaches in both instruction- following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further vali- date the empirical effectiveness of our hierarchi- cal design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log- probability of desirable responses, contributing to a more stable preference manifold for cross- lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO. 1 Introduction Integrating human feedback through the align- ment process has emerged as a critical paradigm for optimizing the instruction-following capabil- ities of Large Language Models (LLMs) and grounding their objective functions in human value systems (Stiennon et al., 2020; Askell et al., 2021; Hong et al., 2024; Meng et al., * Corresponding author Query: 你对人工智能的未来有什么看法? (What do you think about the future of AI?) Response: "The future of artificial intelligence is a topic of great debate.Some experts believe..." LLMs: Incorrect Language Language Mismatch Response: "人工智能的未来非常有希望 预计将引领各个行业的创新. 但也伴随着伦理问题和...." (The future of artificial intelligence is very promising, and it is expected to lead innovation across various industries. However, it is also accompanied by ethical issues and...) CRPO Language Match & High Quality Response: "人工智能的未来....嗯, 机器人会取代所有工作, 人类只能休息. 让我们一起想象一下不再需要劳动的未来." (The future of artificial intelligence... well, robots will replace all jobs, and humans will only be able to rest. Let's imagine a future where labor is no longer necessary.) General Approach: Low Quality Language Match & Low Quality Figure 1: Example of model responses to multilingual prompt after alignment tuning. 2024). In particular, Direct Preference Optimiza- tion (DPO) (Rafailov et al., 2023) has stream- lined the complex Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) pipeline, effectively guiding models toward pre- ferred responses while improving helpfulness and utility (Kirk et al., 2023; Pang et al., 2024; Tu et al., 2025). However, these alignment gains re- main largely confined to English-centric contexts, leaving consistency and response accuracy in other languages as a formidable challenge. This dispar- ity often leads to undesirable behavior, including input-output language inconsistency and degraded performance for other language queries (Lai et al., 2023a; Puttaparthi et al., 2023; Huang et al., 2023; Wu et al., 2024a; Marchisio et al., 2024). Existing approaches to multilingual alignment typically rely on expensive human annotation or arXiv:2608.23149v2 [cs.CL] 27 Aug 2026 on integrating target language data into English- dominant corpora (Lai et al., 2023b; Zhao et al., 2024; Ranaldi et al., 2024; Lai et al., 2024). How- ever, these approaches are limited by their inabil- ity to leverage the model’s inherent English ca- pabilities for cross-lingual alignment. By leaving this internalized English preference knowledge de- coupled from the target language, they struggle to achieve effective cross-lingual alignment. In this paper, we propose Cross-lingual Ranking Preference Optimization (CRPO), a novel align- ment framework that aligns the preference distribu- tion of a target language by leveraging the model’s inherent preference knowledge in English. CRPO moves beyond conventional binary comparisons within a single language; instead, we adopt an align- ment strategy based on a hierarchical ranking struc- ture that integrates two core dimensions: language consistency and response quality. As illustrated in Figure 1, while LLMs often suffer from language confusion (e.g., responding in English to a non- English prompt) and existing approaches generate low-quality outputs, CRPO guides the model to- ward generating high-quality responses appropriate for the target language query. To achieve this, we transplant the “Learning-to- Rank” (LTR) (Liu et al., 2009) philosophy, which optimizes relative importance among multiple can- didates, into the cross-lingual alignment training. To be specific, we design a hierarchical signal based on parallel preference pairs between English and the target language. This design facilitates align- ment of target language preferences by referring to well-defined preference distributions, while in- ducing the generation of target language responses rather than English. Our experiments show that CRPO achieves sub- stantial improvements in instruction-following and knowledge-intensive tasks. Furthermore, our com- parative analysis of reward and log-likelihood dis- tributions demonstrates that CRPO leads to a more accurate likelihood preference of winning and los- ing responses on a held-out test set, which in turn translates to a better policy model for the target language. Our contributions are as follows: • We propose CRPO, a novel framework that optimizes the hierarchical priority between language and quality by integrating ranking principles into cross-lingual preference align- ment. •Through an analysis of reward and probability distributions, we find that CRPO more effec- tively steers the preference distribution of the target language toward a desirable alignment compared to existing methodologies. •We demonstrate that our cross-lingual prefer- ence hierarchy structure itself, which lever- ages the relationship between English and the target language, induces significant per- formance gains in target language alignment. 2 Related Work Enhancing the multilingual capabilities of LLMs is a critical endeavor to ensure that global users can benefit from technological advancements without language barriers (Costa-Jussà et al., 2022; Zhao et al., 2024; Huang et al., 2024; Li et al., 2024a). Most research primarily focused on machine trans- lating English datasets or training on heterogeneous mixtures of multilingual corpora (Workshop et al., 2022; Muennighoff et al., 2023; Lai et al., 2023b; Gao et al., 2024; Kew et al., 2024). However, these approaches often require large-scale datasets or suf- fer from persistent language bias (Aggarwal et al., 2024; Li et al., 2024c; Lee et al., 2025). Recent studies seeking to mitigate these limita- tions have increasingly explored extending align- ment algorithms proven in English to multilingual contexts (Yang et al., 2024; Dang et al., 2024a). In prior work, Wu et al. (2024b) facilitates zero- shot cross-lingual transfer in reward model training, aiming to build robust multilingual reward systems. MAPO (She et al., 2024) targets improvement in the multilingual reasoning domain using reward signals derived from external translation models. Similarly, MPO (Zhao et al., 2025) directly opti- mizes the reward margin gap across languages to enhance safety. While effective, these approaches rely on external dependencies, such as online ma- chine translation pipelines, and are limited to spe- cific domains. Another line of work operates di- rectly on the policy model using implicit binary comparisons (Yang et al., 2025; Lee et al., 2025), yet they remain constrained by binary optimization, failing to capture complex preference distributions. Although some works have explored list-based ranking to move beyond binary comparisons, these studies primarily focus on improving quality or safety and are strictly limited to English-centric contexts (Yuan et al., 2023; Song et al., 2024; Liu et al., 2025b). In contrast, CRPO distinguishes it- self by optimizing a cross-lingual hierarchical rank- ing structure that jointly models language and qual- ity. By enabling simultaneous modeling of linguis- tic and quality hierarchies, CRPO leads to meaning- ful improvements in target language capabilities. 3 CRPO Framework Our approach is established on the DPO framework, extending its pairwise alignment capabilities into a ranking paradigm specifically designed for cross- lingual contexts. 3.1 Preliminary: Direct Preference Optimization DPO (Rafailov et al., 2023) is a stable and efficient method to align LLMs with human preferences without the need for training an explicit reward model or using complex reinforcement learning pipelines such as PPO (Schulman et al., 2017): L DPO (π θ ;π ref ) =−E (x,y w ,y l )∼D logσ β log π θ (y w |x) π ref (y w |x) − β log π θ (y l |x) π ref (y l |x) (1) whereπ θ is the policy model,π ref is the reference policy given(x,y w ,y l )are preference pairs which are the prompt, chosen response, and rejected re- sponse from the preference dataset. By minimizing the given loss, the model increases the margin be- tween the preferred response y w relative to y l . 3.2 LambdaLoss Ranking Objective Given that our aim is to encourage cross-lingual adaptation alongside robust preference alignment, the binary formulation of DPO is inherently lim- ited, as it cannot accommodate multiple optimiza- tion spheres. To address this, we draw inspiration from LambdaLoss (Wang et al., 2018), a type of Learning to Rank (LTR) (Liu et al., 2009) frame- work that enables joint optimization of multiple items across a global ranking structure. This approach allows the model to optimize well- founded ranking metrics, such as Discounted Cu- mulative Gain (DCG) (Burges et al., 2006), by weighting pairwise transitions based on their im- pact on the overall list. Specifically, for a promptx and a set of candidate responsesy =y 1 ,...,y n , the training objective of our ranking scheme is de- fined as: L lambda (π θ ;π ref ,β) = E (x,y,ψ)∼D X ψ i >ψ j ∆ i,j log(1 + e −(s i −s j ) ) (2) wheres i = β log π θ (y i |x) π ref (y i |x) represents the implicit reward score of responsey i . The term∆ i,j is the Lambda weight, which determines the importance of the pair(y i ,y j )based on their positions in the ranking: ∆ i,j =|G i − G j |· 1 D(τ (i)) − 1 D(τ (j)) (3) In this formulation,Gis the gain function (typ- icallyG i = 2 ψ i − 1 whereψ i is the relevance label), andDis the rank discount function (typi- callyD(τ (i)) = log(1 +τ (i))). Here,τ (i)denotes the rank position of y i induced by the score s. This objective can optimize the DCG metric by prioritizing the placement of high-relevance items at the top of the list by applying a logarithmic dis- count to lower ranks. Consequently, any rank in- version that displaces a desirable response from its superior position results in a significant reduc- tion of the overall score, which allows the∆ i,j to impose a stringent penalty on misalignment that disrupts the intended preference order. 3.3CRPO: Cross-lingual Ranking Preference Optimization Building upon the LambdaLoss framework, CRPO aims to induce both cross-lingual and intra-lingual alignment by jointly optimizing target language responses alongside semantically equivalent pref- erence knowledge from English. For each training instance, we construct a quadruple of responses consisting of:y t w ,y t l ,y e w , andy e l , which are chosen and rejected responses in the target language and English, respectively. To ensure a consistent pref- erence manifold is maintained across languages, we assign hierarchical relevance labels (ψ) based on both response quality and language type among these four responses: ψ(y t w ) > ψ(y e w ) > ψ(y t l ) > ψ(y e l )(4) Under this hierarchy, the model is trained to priori- tize the preferred response in the target language, y t w , as the highest-ranked candidate. At the same time, CRPO provides comprehensive supervisory signals by aligning all pairs within they t ,y e set for a given target language promptx. The model is compelled to jointly optimize both intra-lingual preferences ((y t w ,y t l ),(y e w ,y e l )) and cross-lingual preference pairs, such as (y t w ,y e l ) and (y e w ,y t l ). Since the optimization of target language in- volves concurrent alignment with semantically equivalent English pairs, the model can leverage these English candidates as informative cues. Fur- thermore, prioritizingy t w provides training signals for maintaining input-output language consistency. Thus, the gain function withψin CRPO enables the model to inherit the logical consistency of En- glish preferences while imposing a stringent quality constraint on responses within the target language. The gain functions for each response in CRPO are detailed in Appendix A. Finally,weemployanegativelog- likelihood (NLL) loss term fory t w in our loss to improve alignment performance during the instruction tuning process. The resulting loss function of CRPO is defined as: L CRPO = (1− α)·L NLL + α·L lambda (π θ ;π ref ,β) (5) whereαis a hyperparameter that balances the ranking-based alignment and the language model- ing objective. While conventional DPO is restricted to binary comparisons of responses within the tar- get language, CRPO promotes the precise mapping of hierarchies defined by both language and quality onto a unified optimization plane. 3.4 Weighting Schemes for CRPO The flexibility of the LambdaLoss framework al- lows CRPO to incorporate various weighting func- tions for each optimizing different bounds of the ranking metric. We explore three schemes for∆ i,j derived from LambdaLoss to investigate the effec- tiveness in cross-lingual alignment. LambdaRank optimizes a coarse upper bound of the nDCG metric. It defines the weight based on the absolute difference between the inverse dis- counts of the two positions, which is presented in Equation 3. This scheme provides a founda- tional gradient signal for ranking by considering the global positions of the responses in the list. nDCG2 introduces a tighter bound for nDCG optimization. Unlike the global position-based weighting of LambdaRank, it focuses on the dis- tance between the two items being compared: ∆ nDCG2 i,j = |G i − G j |· 1 D(|τ (i)− τ (j)|) − 1 D(|τ (i)− τ (j)| + 1) (6) where|τ (i)− τ (j)|represents the rank distance between the two responses. This scheme has been empirically shown to achieve superior performance by providing more localized and precise gradients for rank inversion. nDCG2++is a hybrid scheme that combines the strengths of both nDCG2 and LambdaRank to max- imize ranking performance: ∆ nDCG2++ i,j = μ· ∆ nDCG2 i,j + ∆ i,j (7) whereμis a hyperparameter that controls the contribution of the nDCG2 component. In our experiments,μis set to 10 following the prior study (Wang et al., 2018). While nDCG2++ is known to reach the strongest performance in traditional Information Retrieval tasks, we found that nDCG2 or LambdaRank alone can be sufficient for aligning LLM preference distri- butions across languages. To demonstrate the gener- ality and theoretical robustness of the CRPO frame- work, we employ nDCG2 as our primary weighting scheme for the experiments. 4 Experimental Setup 4.1 Models and Training Models We perform preference optimization with three different models, Llama-2-7B (Touvron et al., 2023), Llama-3-8B (Dubey et al., 2024), and Mistral-7B-v0.1 (Jiang et al., 2023). We em- ploy five languages with varying resource avail- ability: Chinese (zh), Indonesian (id), Korean (ko), Swahili (sw), and Bengali (bn). They were cho- sen to provide a mix of high-, medium-, and low- resource languages, typological and script diversity, while satisfying the practical constraints of avail- able evaluation datasets. Dataset To construct the preference dataset for each language, we sample 3,000 instances from the UltraFeedback dataset (Cui et al., 2024) and translate them into the respective target languages usinggpt-5-chat. This procedure yields fully par- allel target language pairs(x t ,y t w ,y t l )that corre- spond directly to the original English preference pairs(x e ,y e w ,y e l ). We randomly split the resulting dataset, including English pairs, into a 90% training set and a 10% test set. Baseline We compare CRPO against two com- petitive baselines that aim to align cross-lingual preferences via monolithic strategy. We establish SFT+DPO as our strong baseline. In this setup, the model is optimized using response pairs in the language of the input. Since the training dataset encompasses both English and the target language, the model is trained on(y t w ,y t l )forx t and(y e w ,y e l ) for x e , in the same batch. As another method, CLO (Lee et al., 2025) is a variant of DPO that incorporates binary compari- son between English and target language response pairs within the same batch. To be specific, CLO explicitly steers the model toward generating de- sirable and language-consistent responses by des- ignating the target language response as chosen and the English response as rejected. Furthermore, it mitigates inherent English bias and facilitates alignment by restricting the NLL loss exclusively to target language responses. Note that all baselines and models use identical hyperparameters and the same amount of parallel datasets to ensure a fair comparison. Further details about the training can be found in Appendix B. 4.2 Evaluation AlpacaEval We employ AlpacaEval (Li et al., 2023) to evaluate models’ conversational ability across a diverse set of queries. AlpacaEval com- prises 805 questions from 5 datasets and evaluates model performance through qualitative compar- isons of reference responses and candidate model outputs. To support evaluation across multiple lan- guages, we adopt the multilingual version of Al- pacaEval established in prior work (Lee et al., 2025) as our primary evaluation suite. We report the win rate (WR) and length-controlled win rate (LC) against the SFT model. The LC metric is specif- ically designed to be robust against model ver- bosity (Dubois et al., 2024). Also, by including the instruction in the evaluation prompt, the model is penalized for responding in a language other than the target language. We usegpt-5-chatas a judge for assessment and also report multilingual Arena-Hard results in Appendix C. Knowledge & Understanding We further eval- uate the models’ capacity for knowledge uti- lization and linguistic comprehension. We use MMMLU (Multilingual Massive Multitask Lan- guage Understanding) (Hendrycks et al., 2021) to verify whether the trained model can leverage in- herent world knowledge in target language con- texts. MMMLU is a human-translated multilingual extension of the original MMLU benchmark, en- compassing 57 subjects across STEM, humanities, and social science 1 . In addition, we employ Bele- 1 https://huggingface.co/datasets/openai/MMMLU bele (Bandarkar et al., 2024), a benchmark for mul- tilingual reading comprehension. In this task, we can evaluate literacy and contextual understanding, as the model should select the most appropriate answer from four choices based on the short pas- sage and question. Following standard practice, we report accuracy by treating the option with the high- est overall log-likelihood as the model’s predicted answer (Gao et al., 2021). In-depth Analysis To move beyond aggregate benchmark scores and gain deeper insights into the model’s internal mechanics, we conduct an in- trinsic analysis. We investigate the reward distribu- tions and log probabilities of the preference pairs in our test set to verify whether each method en- ables balanced alignment that reinforces desirable generative behaviors, rather than relying solely on suppressing suboptimal outputs. Furthermore, we measure the quality distribution of generated responses using an external reward model. The objective of this analysis is to empir- ically demonstrate that the internal optimization of CRPO translates into measurable improvements in output quality, ensuring that the model’s pref- erence manifold is robustly aligned with objective judgments in the target language. 5 Results and Analysis In this section, we present the main results of our experiments, highlighting CRPO’s superior perfor- mance across various benchmarks in Section 5.1. Then in Section 5.2, we provide an in-depth un- derstanding of the internal reward dynamics and generative behaviors that distinguish CRPO from other alignment methods. In addition, we employ an external reward model to objectively quantify improvements in absolute generation quality in Sec- tion 5.3. Finally, we conduct a study on various ranking schemes to investigate how different con- figurations of the hierarchical preference structure influence the effectiveness of cross-lingual align- ment in Section 5.4. 5.1 Main Results Overall Performance As shown in Table 1, all preference optimization methods provide improve- ments over the SFT model in most cases, with CRPO achieving the best results across languages. SFT+DPO, a standard approach for model align- ment, also serves as a robust baseline in other lan- guages. CLO exhibits comparable or slightly below ModelMethod ChineseIndonesianKoreanSwahiliBengali LC (%)WR (%)LC (%)WR (%)LC (%)WR (%)LC (%)WR (%)LC (%)WR (%) Llama-2-7B SFT+DPO57.8058.5057.9258.8854.2855.4038.6338.0734.9035.68 CLO56.4156.0853.8453.9151.1550.2444.4943.9733.1530.93 CRPO64.1864.3463.8364.4763.0263.9744.0743.7237.5236.77 Llama-3-8B SFT+DPO60.2460.3759.2460.3764.9765.9646.2346.9534.7736.89 CLO58.7058.1352.4252.9653.9453.4143.7844.9335.6733.29 CRPO62.4562.3667.9768.4068.4568.8161.8462.1753.8655.65 Mistral-7B SFT+DPO55.6556.1455.9858.5051.4952.8527.9527.3324.6324.59 CLO55.1855.0351.2951.9848.1448.3235.1535.4019.9818.01 CRPO62.8363.1066.8168.1962.9063.7250.7651.9848.6948.57 Table 1: AlpacaEval across five target languages. LC and WR denote length-controlled and raw win rate, respectively. The best results for each model are highlighted in bold. ModelLang SFT+DPO CLO CRPO MMMLU Llama-2-7B id26.3324.69 27.85 ko26.2028.57 31.07 sw25.0825.6225.12 Llama-3-8B id41.6936.41 45.94 ko39.5840.1939.84 sw36.0227.48 37.03 Mistral-7B id37.3631.38 37.47 ko30.3929.27 35.01 sw30.5728.32 31.78 Belebele Llama-2-7B id33.1136.55 36.66 ko33.0035.66 36.33 sw29.2228.22 29.22 Llama-3-8B id64.2235.11 65.88 ko57.5559.44 68.66 sw47.5528.66 49.22 Mistral-7B id48.3330.4447.44 ko49.6635.77 51.77 sw33.0033.88 34.77 Table 2: MMMLU and Belebele evaluation results. We report the performance of zero-shot in MMMLU and one-shot in Belebele. performance to SFT+DPO, demonstrating the util- ity of a straightforward objective that suppresses the probability of English responses. Furthermore, we include the English performance of each case in Appendix D in terms of alignment tax. Robustness in Low-Resource Languages While most methods outperform SFT in high- and mid-resource languages, a different tendency emerges in low-resource settings. Unlike CRPO, other methods exhibit notable performance degradation in low-resource scenarios. This trend is particularly pronounced for Llama-3-8B, which suffers from a collapse compared to the SFT, as evidenced by notably low WR in Bengali. This stems from optimization instability and ex- cessive parameter volatility during the preference alignment when the model’s linguistic grounding in the target language is insufficient (Kadyrbek et al., 2025; Lee et al., 2025). From this perspective, SFT+DPO, relying on isolated preference pairs in the target language, fails to establish a stable prefer- ence manifold in low-resource scenarios. Similarly, the performance drop in CLO shows that simply enforcing language-consistent responses through binary cross-lingual hierarchies is insufficient for robust alignment. While CRPO also encounters adaptation chal- lenges due to the inherent limitations of mod- els, it consistently surpasses baselines and demon- strates remarkable robustness in most cases. No- tably, CRPO achieves a significant performance gain in Swahili with Llama-3, scoring a WR of 62.17. This gain can be attributed to the use of relatively well-aligned English pairs as a logical an- chor for aligning the target language. By realigning language quality and hierarchy, CRPO exhibits the most robust resistance to this alignment collapse and contributes to overall alignment stabilization. Knowledge Utilization The validity of CRPO extends beyond conversational improvements, as evidenced in the knowledge utilization and linguis- tic comprehension areas. As shown in Table 2, CRPO achieves competitive performance across both benchmarks. Notably, the MMMLU score in Indonesian for Llama-3 exceeds the strongest baseline by over 4 points, while the Korean Bele- bele score reaches 68.66. These results demonstrate -20-1001020304050 0.00 0.02 0.04 0.06 0.08 0.10 Density Chinese SFT CLO SFT+DPO CRPO -20-1001020304050 0.00 0.02 0.04 0.06 0.08 0.10 Indonesian SFT CLO SFT+DPO CRPO -20-1001020304050 0.00 0.02 0.04 0.06 0.08 0.10 Korean SFT CLO SFT+DPO CRPO -20-1001020304050 0.00 0.02 0.04 0.06 0.08 0.10 Swahili SFT CLO SFT+DPO CRPO (a) Reward Difference -150-125-100-75-50-25025 0.000 0.003 0.005 0.007 0.010 0.013 0.015 0.018 0.020 Density Chinese SFT CLO SFT+DPO CRPO -150-125-100-75-50-25025 0.000 0.003 0.005 0.007 0.010 0.013 0.015 0.018 0.020 Indonesian SFT CLO SFT+DPO CRPO -150-125-100-75-50-25025 0.000 0.003 0.005 0.007 0.010 0.013 0.015 0.018 0.020 Korean SFT CLO SFT+DPO CRPO -150-125-100-75-50-25025 0.000 0.003 0.005 0.007 0.010 0.013 0.015 0.018 0.020 Swahili SFT CLO SFT+DPO CRPO (b) Log Probability Distribution Figure 2: (a) Reward difference distribution and (b) Log-likelihood distribution on chosen responses (y w ) of Llama- 3-8B trained with SFT (blue), CLO (orange), SFT+DPO (blue) and CRPO (red) across languages. ModelMethodzhidkosw Llama-2-7B SFT-1.749-1.797-2.870-2.901 SFT+DPO-1.227 -0.904-2.121-2.958 CLO-1.539-1.200-2.855-2.457 CRPO-0.607-0.936-1.601 -2.175 Llama-3-8B SFT0.3440.208-0.904-0.487 SFT+DPO1.5061.5850.869-0.212 CLO0.6131.034-0.119-0.531 CRPO1.738 1.883 0.911 1.122 Mistral-7B SFT-0.706-0.808-1.800-0.667 SFT+DPO0.255-0.037-0.719-1.519 CLO-0.096-0.665-1.237-0.981 CRPO0.508 0.805 -0.344 -0.040 Table 3: Average of reward scores from Skywork- Reward-V2-Qwen3-8B on generation outputs. The best results achieved across the methods are highlighted in bold, while the second-best results are underlined. that the ranking structure in CRPO successfully anchors the model’s inherent knowledge system within the target language space, thereby achieving better alignment for complex reasoning tasks. 5.2 Reward and Probability Shifts To investigate the impact of CRPO on the model’s preference manifold, we examine the internal dy- namics of reward differences and generative log- probabilities. As illustrated in Figure 2, the mag- nitude of distribution shifts differs across methods regarding the SFT model as a default. In Figure 2 (a), we find that regardless of the language, CRPO consistently achieves a large re- ward difference and shifts the distribution in a positive direction compared to the SFT. Likewise, the SFT+DPO approach exhibits a similar trend, though the magnitude of the shift is relatively smaller. In contrast, CLO yields almost identical re- ward difference to SFT, or in the case of Indonesian, even shows signs of degradation. One of the primary risks in preference optimiza- tion is the improper focus on maximizing the re- ward margin, which often results in suppressing the probability of rejected responses rather than in- creasing the probability of chosen responses. This imbalance can undermine the language model’s in- herent stability and fluency by failing to provide an explicit guide for target responses, potentially leading to degeneration (Meng et al., 2024; Hong et al., 2024). As shown in the comparison of Fig- ure 2 (a) and (b), while other methods increase the reward margin, their log-likelihood distributions for chosen responses remain largely congruent with the SFT. This indicates that other approaches pri- marily fail to incentivize the generation of favored responses. On the other hand, CRPO assigns higher val- ues to both the log-likelihood of the chosen re- sponse and the reward difference across all lan- guages than in other cases. This demonstrates that CRPO achieves better alignment in the target lan- guage by increasing the likelihood of the favored response. Further analysis of the reward distribu- tion is provided in Appendix E. Chinese Indonesian KoreanSwahili Bengali 20 40 60 Llama-3-8B Chinese Indonesian KoreanSwahili Bengali 20 40 60 Mistral-7B SFT+DPO uniform nDCG2 nDCG2++ LambdaRank Figure 3: Win rate comparison of different weighting schemes within the CRPO framework. 5.3 External Quality Assessment While win rates provide a measure of relative preference, they may not fully capture the abso- lute quality shift in the model’s generative distri- bution. To provide a more objective assessment of response quality, we use Skywork-Reward-V2- Qwen3-8B (Liu et al., 2025a), an external reward model known to support multilingual environments, to score the outputs of each method on the test set. Table 3 summarizes the average reward scores across diverse models and languages. The results demonstrate that CRPO achieves su- perior reward magnitudes across nearly all config- urations. A notable score escalation is observed in the Mistral-7B; in Indonesian (id), while the SFT+DPO reaches a score of -0.037, CRPO ele- vates the quality to a considerable positive margin of 0.805, outperforming SFT+DPO by a wide gap. This also substantiates our claim that anchoring target language alignment to established English preference structures can guide the model toward higher-quality regions of the generation manifold. 5.4 Study on Ranking Scheme We investigate how the choice of weighting scheme in Equation 2 influences cross-lingual preference alignment. Figure 3 shows AlpacaEval results ob- tained by diverse weighting schemes within the same CRPO framework. As shown in Figure 3, all three schemes consis- tently outperform both SFT+DPO and the uniform scheme, confirming that hierarchical ranking sig- nals provide more informative supervision than pairwise comparisons. We can attribute the success across diverse configurations to CRPO’s hierarchi- cal structural design. As evidence for this, assign- ing a uniform weight to all response pairs results in a notable performance drop. The degradation mainly arises from the lack of alignment between the languages. Removing hierarchical cues triggers an English bias, leading to a failure in linguistic consistency with the given prompt. Moreover, LambdaRank and nDCG2 achieve strong performance across most settings. This sug- gests that global position-based signals and local distance-based signals can complement each other depending on the model and language. Given that our experiments are conducted under the nDCG2 scheme, the additional gains observed with alter- native schemes further indicate that the potential performance ceiling of CRPO may be even higher. 6 Conclusion In this paper, we propose CRPO, a new alignment framework designed to establish robust preference manifolds both within and across languages. By reformulating the target language adaptation and alignment task as a hierarchical ranking problem, CRPO benefits from the transfer of preference knowledge along the language chain. Our empirical results demonstrate that CRPO not only achieves consistent superiority over standard pairwise ap- proaches but also enhances the generative likeli- hood of preferred responses, facilitating the genera- tion of high-fidelity, precise responses in the target language. We believe that our approach shifts the paradigm of cross-lingual alignment from isolated binary comparisons to an integrated cross-lingual ranking perspective. Moving forward, we will in- vestigate a dynamic gain adaptation strategy based on linguistic similarities and training stages. By mitigating the optimization complexity inherent in expansive ranking spaces, we expect to ensure ro- bust cross-lingual alignment across an even wider array of language pairs. Limitations Although we use the same amount of the dataset during the training, our proposed framework em- ploys a hierarchical ranking structure that processes four response candidates in a single step. This in- herently demands more computational resources during training. However, this overhead can be con- sidered an unavoidable cost inherent to methods that leverage item-wise ranking, not only limited to the preference optimization field. Given the limita- tions of standard binary comparisons in optimizing both quality and consistency simultaneously, the performance gains achieved by CRPO demonstrate that this additional cost is a worthwhile trade-off for effective cross-lingual alignment. Furthermore, our main experiments include eval- uation results from the multilingual version of the AlpacaEval dataset, translated using gpt-4o in prior work (Lee et al., 2025), and the MMMLU dataset (Hendrycks et al., 2021), which was trans- lated into the respective languages by professional human translators. Additionally, we employed the Belebele dataset (Bandarkar et al., 2024), which was constructed and reviewed by human experts during translation. While these datasets are suitable for measuring general model performance, they may not fully capture the model’s ability to re- spond appropriately to queries involving language- specific characteristics or cultural contexts. Al- though this limitation may prevent a comprehensive evaluation of the model’s grasp of cultural nuances and localized content, we made every effort to uti- lize the highest-quality benchmarks available. Ethics Statement In this research, we utilized only publicly available resources. All training and evaluation data were ob- tained from publicly authorized open-access repos- itories. Nevertheless, we recognize the potential for residual biases or harmful concepts in the En- glish source data to be propagated across languages. To mitigate this, we carefully adhered to the copy- right guidelines and terms of use for all original works, datasets, and language resources, including translated materials. We confirm that the collection, processing, and application of these resources raise no distinct ethical concerns. Acknowledgments This research was supported by Basic Science Re- search Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021R1A6A1A03045425). This work was supported by Institute for Infor- mation & communications Technology Promo- tion (IITP) grant funded by the Korea govern- ment (MSIT) (RS-2024-00398115, Research on the reliability and coherence of outcomes produced by Generative AI). This research was supported by Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2026 (Project Name: Development of an AI Agent Integrating Korean Language Knowl- edge for Personalized Language Consultation Ser- vices, Project Number: RS-2026-25506607, Con- tribution Rate: 25%). This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) under the Leading Generative AI Human Resources De- velopment (IITP-2026-RS-2026-25545512) grant funded by the Korea government (MSIT). References Divyanshu Aggarwal, Ashutosh Sathe, Ishaan Watts, and Sunayana Sitaram. 2024. MAPLE: Multilingual evaluation of parameter efficient finetuning of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 14824– 14867, Bangkok, Thailand and virtual meeting. Asso- ciation for Computational Linguistics. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for align- ment. arXiv preprint arXiv:2112.00861. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 lan- guage variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics. Christopher Burges, Robert Ragno, and Quoc Le. 2006. Learning to rank with nonsmooth cost functions. Ad- vances in neural information processing systems, 19. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. 2024. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pages 9722–9744. PMLR. John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. 2024a. RLHF can speak many languages: Unlocking mul- tilingual preference optimization for LLMs. In Pro- ceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 13134– 13156, Miami, Florida, USA. Association for Com- putational Linguistics. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Ro- man Castagné, Felipe Cruz-Salinas, Eddie Kim, Lu- cas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. 2024b. Aya expanse: Combining re- search breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Yann Dubois, Balázs Galambosi, Percy Liang, and Tat- sunori B Hashimoto. 2024. Length-controlled al- pacaeval: A simple way to debias automatic evalua- tors. arXiv preprint arXiv:2404.04475. Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024. Multilingual pre- training and instruction tuning improve cross-lingual knowledge alignment, but only shallowly. In Pro- ceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6101–6117, Mexico City, Mexico. Association for Computational Linguistics. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. 2021. A framework for few-shot language model evaluation. Zenodo. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. Preprint, arXiv:2009.03300. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without refer- ence model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 11170–11189. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improv- ing multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Com- putational Linguistics: EMNLP 2023, pages 12365– 12394, Singapore. Association for Computational Linguistics. Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, et al. 2024. A sur- vey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. Nurgali Kadyrbek, Zhanseit Tuimebayev, Madina Mansurova, and Vítor Viegas. 2025. The devel- opment of small-scale language models for low- resource languages, with a focus on kazakh and di- rect preference optimization. Big Data and Cognitive Computing, 9(5):137. Tannon Kew, Florian Schottmann, and Rico Sennrich. 2024. Turning English-centric LLMs into polyglots: How much multilinguality is needed? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13097–13124, Miami, Florida, USA. Association for Computational Linguistics. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023. Understanding the ef- fects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452. Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023a. ChatGPT beyond English: Towards a comprehensive evaluation of large language mod- els in multilingual learning. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023b. Okapi: Instruction-tuned large language mod- els in multiple languages with reinforcement learn- ing from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327. Wen Lai, Mohsen Mesgar, and Alexander Fraser. 2024. LLMs beyond English: Scaling the multilingual capa- bility of LLMs with cross-lingual feedback. In Find- ings of the Association for Computational Linguistics: ACL 2024, pages 8186–8213, Bangkok, Thailand. As- sociation for Computational Linguistics. Jungseob Lee, Seongtae Hong, Hyeonseok Moon, and Heuiseok Lim. 2025. Cross-lingual optimization for language transfer in large language models. In Pro- ceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 15100–15119, Vienna, Austria. Asso- ciation for Computational Linguistics. Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024a.Improving in-context learning of multilingual generative language models with cross- lingual alignment. In Proceedings of the 2024 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pages 8058–8076. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024b. From crowdsourced data to high- quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An au- tomatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. 2024c. Quantifying multilin- gual performance of large language models across languages. arXiv preprint arXiv:2404.11553. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Ji- acai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. 2025a. Skywork-reward- v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352. Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, and Xuanhui Wang. 2025b. LiPO: Listwise prefer- ence optimization through learning-to-rank. In Pro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), pages 2404–2420, Al- buquerque, New Mexico. Association for Computa- tional Linguistics. Tie-Yan Liu et al. 2009. Learning to rank for informa- tion retrieval. Foundations and Trends® in Informa- tion Retrieval, 3(3):225–331. Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in LLMs. In Pro- ceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 6653– 6677, Miami, Florida, USA. Association for Compu- tational Linguistics. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Infor- mation Processing Systems, 37:124198–124235. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generaliza- tion through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference opti- mization. Advances in Neural Information Process- ing Systems, 37:116617–116637. Poorna Chander Reddy Puttaparthi, Soham Sanjay Deo, Hakan Gul, Yiming Tang, Weiyi Shang, and Zhe Yu. 2023. Comprehensive evaluation of chatgpt relia- bility through multilingual inquiries. arXiv preprint arXiv:2312.10524. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741. Leonardo Ranaldi, Giulia Pucci, and Andre Fre- itas. 2024.Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7961–7973, Bangkok, Thailand. Associ- ation for Computational Linguistics. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347. Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. 2024. MAPO: Advancing multilingual reasoning through multilingual-alignment-as-preference optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10015–10027, Bangkok, Thai- land. Association for Computational Linguistics. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024. Pref- erence ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990–18998. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, et al. 2025. Enhancing llm rea- soning with iterative dpo: A comprehensive empirical investigation. arXiv preprint arXiv:2503.12854. Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018. The lambdaloss framework for ranking metric optimization. In Pro- ceedings of the 27th ACM international conference on information and knowledge management, pages 1313–1322. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ́ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luc- cioni, François Yvon, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. Suhang Wu, Jialong Tang, Baosong Yang, Ante Wang, Kaidi Jia, Jiawei Yu, Junfeng Yao, and Jinsong Su. 2024a. Not all languages are equal: Insights into multilingual retrieval-augmented generation. arXiv preprint arXiv:2410.21970. Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami. 2024b. Reuse your rewards: Reward model transfer for zero-shot cross- lingual alignment. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 1332–1353. Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, and Jiajun Zhang. 2024. Language imbalance driven rewarding for multilingual self-improving. arXiv preprint arXiv:2410.08964. Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, and Jiajun Zhang. 2025. Implicit cross-lingual re- warding for efficient multilingual preference align- ment. In Findings of the Association for Computa- tional Linguistics: ACL 2025, pages 21125–21147, Vienna, Austria. Association for Computational Lin- guistics. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feed- back. Advances in Neural Information Processing Systems, 36:10935–10950. Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xu- anjing Huang. 2024. Llama beyond english: An em- pirical study on language capability transfer. arXiv preprint arXiv:2401.01055. Weixiang Zhao, Yulin Hu, Yang Deng, Tongtong Wu, Wenxuan Zhang, Jiahe Guo, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, and Ting Liu. 2025. MPO: Multilingual safety alignment via reward gap opti- mization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 23564–23587, Vienna, Austria. Association for Computational Linguistics. A Gain Formulation in Cross-lingual Ranking Preference HierarchyIn our pilot study, we ob- served that language adaptation is more readily achieved than the fine-grained differentiation of response quality within the same language, which can easily overshadow quality optimization during training. To reflect this, we established the follow- ing preference hierarchy for our ranking-based ob- jective:ψ(y t w ) > ψ(y e w ) > ψ(y t l ) > ψ(y e l ) . A crit- ical design choice in this formulation is the relation ψ(y e w ) > ψ(y t l ), which prioritizes a high-quality response in English over a lower-quality response in the target language. Furthermore, this provides a learning signal that encourages the model to gener- ate desirable responses regardless of the language. Since gain adjustments across items in a ranking structure are interdependent rather than orthogo- nal, penalizing the English winner below the target language loser would otherwise heavily bias the learning signal toward language matching. Balance between Two DimensionsThis philos- ophy also led to our choice of gain values. In stan- dard Information Retrieval, which adopts the Lamb- daLoss framework, the gain function typically fol- Gain values (G) ChineseKoreanIndonesianSwahiliBengali WC (%)LC (%)WC (%)LC (%)WC (%)LC (%)WC (%)LC (%)WC (%)LC (%) 7, 3, 1, 060.5060.7566.4666.0467.2066.8666.4666.3554.1652.54 9, 5, 7, 460.0060.3966.2166.1165.2265.5267.1467.4353.9853.79 9, 7, 5, 462.3662.4668.8268.4668.4167.9862.1761.8555.6553.87 Table 4: WR and LC according to different gain values. ModelMethod ChineseIndonesianKoreanSwahiliBengali LC (%)WR (%)LC (%)WR (%)LC (%)WR (%)LC (%)WR (%)LC (%)WR (%) Llama-2-7B SFT+DPO55.0655.7155.8956.5257.7958.2656.0456.4651.4652.42 CLO57.4657.2753.4952.2455.3054.9153.4151.9353.2252.61 CRPO62.3362.3660.4160.6260.4660.8760.4360.0061.6061.99 Llama-3-8B SFT+DPO 57.7158.4560.3161.3056.2356.8358.0658.0156.9857.76 CLO52.8352.4856.8956.2152.6650.9954.4652.6156.2754.78 CRPO56.7357.2064.2464.3559.1959.0760.5360.0657.4357.64 Mistral-7B SFT+DPO 61.3065.5958.4461.0663.3169.6951.7354.8847.6555.47 CLO51.8454.4752.0551.6158.1961.8050.3749.8147.9745.83 CRPO50.6057.0859.9661.8657.4264.3556.2157.7653.4955.86 Table 5: English performance in AlpacaEval benchmark. The best results for each model are highlighted in bold. lows an exponential discount:G i = 2 ψ i − 1(Wang et al., 2018). However, applying this to our dual- dimensional objective can introduce an inductive bias. Given our hierarchy, the standard formula yields G t w = 7,G e w = 3,G t l = 1 , andG e l = 0 . Although ranking optimization inherently assigns dispropor- tionate weights between items, this exponential spacing exacerbates biased local gradients. To resolve this and provide a more balanced op- timization manifold, we manually set the gains to G t w = 9,G e w = 7,G t l = 5 , andG e l = 4 . In this setup, the primary intra-language quality gaps (G t w −G t l = 4 for target,G e w −G e l = 3 for English) remain strictly greater than the primary language gap (G t w − G e w = 2). This constraint explicitly ensures that the LambdaLoss framework jointly op- timizes both cross-lingual alignment and quality, preventing the simple language-matching signal from overwhelming the fine-grained quality opti- mization. Empirical Validation To validate our gain for- mulation, we compare the performance of Llama- 3-8B across different gain configurations, as pre- sented in Table 4. The theoretical range of possible gains is infinite, but as it is computationally pro- hibitive to explore the entire space, our experiments are restricted to a set of representative fixed values. The standard exponential setting (7, 3, 1, 0) gen- erally yields suboptimal win rates, suggesting that disproportionate local gradients hinder effective intra-lingual alignment. Furthermore, we examine a configuration (9, 5, 7, 4) that explicitly violates our preference hierarchy by penalizing the English winner (G = 5) below the target language loser (G = 7). This configuration also yields slightly lower performance than our hierarchy across most languages, including Chinese, Korean, Indonesian, and Bengali, demonstrating that heavily biasing the signal toward language matching compromises overall response quality. Interestingly, a different tendency appears in Swahili. We attribute this to the low-resource nature of Swahili, where an ag- gressive language-matching penalty might provide a temporary empirical advantage in enforcing the target script. Nevertheless, our proposed hierar- chy configuration demonstrates greater robustness across diverse linguistic environments, justifying our balanced hierarchical design. B Implementation Details TrainingWe observe that hyperparameter tuning is crucial for achieving optimal performance of preference optimization methods. To ensure a fair comparison, we conduct thorough hyperparameter tuning across all methods in our experiments. We conduct preliminary experiments to search for learning rate in[5e− 7, 8e− 6, 1e− 5]and training epochs in[1, 2]. We find that an effective learning rate of 8e-6 and training for two epochs consistently yield stable results across all models -20-1001020304050 0.000 0.020 0.040 0.060 0.080 0.100 Density Chinese SFT CLO SFT+DPO CRPO -20-1001020304050 0.000 0.020 0.040 0.060 0.080 0.100 Indonesian SFT CLO SFT+DPO CRPO -20-1001020304050 0.000 0.020 0.040 0.060 0.080 0.100 Korean SFT CLO SFT+DPO CRPO -20-1001020304050 0.000 0.020 0.040 0.060 0.080 0.100 Swahili SFT CLO SFT+DPO CRPO (a) Reward distribution of y w -30-20-100102030 0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 Density Chinese SFT CLO SFT+DPO CRPO -30-20-100102030 0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 Indonesian SFT CLO SFT+DPO CRPO -30-20-100102030 0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 Korean SFT CLO SFT+DPO CRPO -30-20-100102030 0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 Swahili SFT CLO SFT+DPO CRPO (b) Reward distribution of y l Figure 4: Reward distribution of chosen response (a) and rejected response (b) of Llama-3-8B trained with SFT (blue), CLO (orange), SFT+DPO (blue) and CRPO (red) across languages. and languages. Therefore, we fix these values for all preference optimization experiments and report results using the final checkpoint. Additionally, we set the maximum sequence length to 3072 and apply a linear learning rate schedule with a 10% warmup ratio. To balance the training objective, we use a weighting factor ofα = 0.2for CRPO and SFT+DPO, and 0.5 for CLO following the recom- mended value in the original paper. We setβto 0.1, the default value in DPO. All experiments are conducted under four NVIDIA A100 GPUs. Evaluation For generative evaluation, we main- tain the same maximum sequence length as used during the training phase to ensure consistency. In the generation process, we use temperature as the primary hyperparameter in the decoding step and set it to 0.8 to balance the trade-off between coher- ence and diversity in the outputs. C Arena-Hard To further substantiate our evaluation, we report the multilingual Arena-Hard (m-ArenaHard) 2 (Dang et al., 2024b) evaluation results in Table 6. The m-ArenaHard benchmark was created by trans- lating the prompts from the original Arena-Hard- v0.1 (Li et al., 2024b) test dataset to 22 languages. We conduct supplementary experiments on the three languages that overlap with those covered in our study. Since several queries in the dataset ex- 2 https://huggingface.co/datasets/CohereLabs/ m-ArenaHard ModelMethodChinese Indonesian Korean Llama-3-8B SFT+DPO61.5661.3464.30 CLO54.4954.5354.95 CRPO62.1561.8768.81 Mistral-7B SFT+DPO54.6254.3455.39 CLO55.4953.3453.02 CRPO58.2263.8562.63 Table 6: Win rate in Arena-Hard benchmark. ceed the relatively short context length of Llama-2 (4096 tokens), Llama-2 is excluded from the eval- uation. Similar to AlpacaEval, we report the win rate against the SFT model. D Impact on Alignment Tax In our main experiments, we focused on evaluating the target languages to demonstrate the effective- ness of cross-lingual alignment. However, since our approach leverages the model’s inherent English knowledge as an anchor to induce transfer to the target language, it is imperative to investigate how the alignment training on the target language af- fects the model’s English proficiency. To this end, we conduct an additional English evaluation on AlpacaEval to measure the alignment tax. Table 5 presents the English performance of the models trained under each target language setting. Note that all methods also include English in their training data composition. As shown in the results, CRPO achieves higher English performance com- pared to the baselines in most cases, indicating that it successfully mitigates the alignment tax. To be specific, for the Llama-2-7B and Llama-3-8B mod- els, CRPO achieves the highest WR across almost all target language configurations. This suggests that CRPO preserves the robustness of the model’s English capabilities while effectively performing cross-lingual alignment. E Reward Distribution Analysis In addition to Figure 2, we also provide an analysis of the reward distribution of responses in Figure 4. As shown in Figure 4 (a), CRPO consistently yields higher rewards for favored responses compared to the other methods, whereas the reward distributions for the other methods do not exhibit a significant shift relative to SFT. In Figure 4 (b), SFT+DPO and CRPO demon- strate comparable distributional shifts, which is the nature of DPO training that generally increases the reward of both chosen and rejected responses. Taken together, these findings indicate that the ma- jor reward difference achieved by CRPO, as ob- served in Figure 2, is primarily driven by elevat- ing the rewards of desirable responses rather than solely relying on penalizing those of unfavored ones. This further substantiates our claim in Sec- tion 5.2 that the advantage of CRPO stems from promoting high-quality responses rather than solely from suppressing the probability of disfavored re- sponses. F Comparison with Another Ranked Optimization To investigate whether the effectiveness of CRPO can be attributed simply to the use of ranked pref- erence structure, we further compare CRPO with PRO (Song et al., 2024), which constructs prefer- ences sequentially from a ranked set of responses: L PRO =− log n−1 Y k=1 exp (r π (x,y k )) P n i=k exp (r π (x,y i )) ,(8) which is further combined with an NLL objective. Table 7 reports the AlpacaEval results across three target languages and two backbone models. Despite also performing ranking-based prefer- ence optimization, PRO substantially underper- forms CRPO. An inspection of generations from the PRO-trained models shows that they gener- ally retain the ability to respond in the language of the input. This suggests that PRO can achieve Lang ModelMethod TargetEnglish WRLCWRLC zh Llama-3-8B PRO24.97 24.24 37.27 36.85 CRPO 62.36 62.45 57.20 56.73 Mistral-7B PRO13.42 13.75 36.15 33.67 CRPO 63.10 62.83 57.08 50.60 id Llama-3-8B PRO23.11 24.79 44.47 43.60 CRPO 68.40 67.97 64.35 64.24 Mistral-7B PRO13.29 12.42 29.81 30.52 CRPO 68.19 66.81 61.86 59.96 sw Llama-3-8B PRO11.43 11.78 37.64 36.17 CRPO 62.17 61.84 60.06 60.53 Mistral-7B PRO6.585.11 10.62 12.15 CRPO 51.98 50.76 57.76 56.21 Table 7: AlpacaEval results of PRO and CRPO under the same training budget. the primary objective of language adaptation, but does not translate this adaptation into a compara- ble improvement in response quality. In contrast, CRPO improves both target language and English AlpacaEval performance by a large margin. We attribute this difference to how the ranked preference structure is exploited during optimiza- tion. PRO primarily constructs pairwise compar- isons between a higher-ranked response and lower- ranked alternatives. CRPO instead exploits the rel- ative positions of responses throughout the ranked list and incorporates ranking-aware objectives to place greater emphasis on preference violations that affect the top of the ranking. Consequently, CRPO provides a more fine-grained learning signal for distinguishing responses with different quality levels rather than merely separating preferred from less-preferred responses. This distinction is particularly important in our setting because language adaptation itself can al- ready be encouraged by the NLL objective: the model can learn to generate in the target language without necessarily improving the quality of the generated content. The ranking-aware objective of CRPO complements this signal by explicitly pre- serving and promoting high-quality responses ac- cording to their positions in the ranked list. The considerable gap between PRO and CRPO there- fore indicates that the gains of CRPO do not arise merely from introducing ranked preference opti- mization. Rather, how the ranked preference infor- mation is incorporated into the optimization ob- jective is critical, further supporting the design of CRPO.