Paper deep dive
Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO
Jinquan Zheng, Jia Yuan, Jiacheng Yao, Chenyang Gu, Pujun Zheng, Guoxiu He
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 2:22:34 AM
Summary
The paper introduces Permutation-Aware Group Relative Policy Optimization (PA-GRPO), a training-time debiasing method for LLMs designed to mitigate selection bias in multiple-choice and pairwise evaluation tasks. By organizing prompts into permutation groups and employing a cross-permutation advantage and a consistency-aware reward, the method enforces semantic reasoning consistency across different input permutations, outperforming existing inference-time and training-time baselines.
Entities (5)
Relation Signals (3)
PA-GRPO → mitigates → Selection Bias
confidence 100% · PA-GRPO, which mitigates selection bias by enforcing permutation-consistent semantic reasoning.
PA-GRPO → evaluatedon → MT-Bench
confidence 95% · We evaluated PA-GRPO on seven benchmarks... MT-Bench
PA-GRPO → improvesupon → GRPO
confidence 90% · Standard GRPO and its variants still typically treat different prompts as independent training instances... PA-GRPO explicitly targets permutation invariance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) used for multiple-choice and pairwise evaluation tasks often exhibit selection bias due to non-semantic factors like option positions and label symbols. Existing inference-time debiasing is costly and may harm reasoning, while pointwise training ignores that the same question should yield consistent answers across permutations. To address this issue, we propose Permutation-Aware Group Relative Policy Optimization (PA-GRPO), which mitigates selection bias by enforcing permutation-consistent semantic reasoning. PA-GRPO constructs a permutation group for each instance by generating multiple candidate permutations, and optimizes the model using two complementary mechanisms: (1) cross-permutation advantage, which computes advantages relative to the mean reward over all permutations of the same instance, and (2) consistency-aware reward, which encourages the model to produce consistent decisions across different permutations. Experimental results demonstrate that PA-GRPO outperforms strong baselines across seven benchmarks, substantially reducing selection bias while maintaining high overall performance. The code will be made available on Github (this https URL).
Tags
Links
- Source: https://arxiv.org/abs/2603.21016v1
- Canonical: https://arxiv.org/abs/2603.21016v1
Trouble viewing inline? Open PDF directly →
Full Text
66,943 characters extracted from source content.
Expand or collapse full text
Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO Jinquan Zheng, Jia Yuan, Jiacheng Yao, Chenyang Gu, Pujun Zheng, Guoxiu He School of Economics and Management, East China Normal University jqzheng, yuanjia, jcyao, cygu, pjzheng@stu.ecnu.edu.cn, gxhe@fem.ecnu.edu.cn Corresponding author. Abstract Large language models (LLMs) used for multiple-choice and pairwise evaluation tasks often exhibit selection bias due to non-semantic factors like option positions and label symbols. Existing inference-time debiasing is costly and may harm reasoning, while pointwise training ignores that the same question should yield consistent answers across permutations. To address this issue, we propose Permutation-Aware Group Relative Policy Optimization (PA-GRPO), which mitigates selection bias by enforcing permutation-consistent semantic reasoning. PA-GRPO constructs a permutation group for each instance by generating multiple candidate permutations, and optimizes the model using two complementary mechanisms: (1) cross-permutation advantage, which computes advantages relative to the mean reward over all permutations of the same instance, and (2) consistency-aware reward, which encourages the model to produce consistent decisions across different permutations. Experimental results demonstrate that PA-GRPO outperforms strong baselines across seven benchmarks, substantially reducing selection bias while maintaining high overall performance. The code will be made available on Github. Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO Jinquan Zheng, Jia Yuan, Jiacheng Yao, Chenyang Gu, Pujun Zheng, Guoxiu He†thanks: Corresponding author. School of Economics and Management, East China Normal University jqzheng, yuanjia, jcyao, cygu, pjzheng@stu.ecnu.edu.cn, gxhe@fem.ecnu.edu.cn 1 Introduction Large language models (LLMs) have been widely applied to multiple-choice question (MCQ) reasoning Hendrycks et al. (2020); Achiam et al. (2023) and LLM-as-a-Judge style evaluation tasks Zheng et al. (2023b); Chiang et al. (2024). In these discrete-choice settings, option order and label symbols (e.g., A/B/C/D) are non-semantic factors that, in principle, should not influence the model’s decisions. However, as illustrated in Figure 1, even when two candidate answers are swapped without altering their content, LLMs may still favor the first-listed option (i.e., position bias) or a particular label such as “A” (i.e., label bias), leading to inconsistent decisions and even preference reversals. Such systematic deviations caused by non-semantic perturbations are referred to as selection bias Zheng et al. (2023b, a). Selection bias undermines the reliability and fairness of evaluation because model outputs can depend on the arrangement of options rather than their semantic content Shi et al. (2024); Wei et al. (2024); Wang et al. (2024a). Moreover, these errors may be further amplified in downstream applications that rely on such outputs, such as automated data construction or alignment training Miao et al. (2024). Figure 1: Example of selection bias in discrete-choice settings: swapping the order of answers changes only non-semantic factors. Existing methods for mitigating selection bias can be broadly categorized into two types: inference-time debiasing and training-time debiasing. Inference-time debiasing typically follows two routes. The first performs output-level calibration by estimating and removing priors post-hoc Zhao et al. (2021), such as PriDe for MCQ and CalibraEval for LLM-as-a-Judge Zheng et al. (2023a); Li et al. (2025b). While often effective, such methods merely adjust surface-level probabilities without rectifying the model’s intrinsic misconceptions, and typically incur high computational overhead due to repeated sampling or optimization. The second route applies internal interventions to suppress bias sources, for instance by masking specific attention heads (UniBias) or pruning bias-related parameters (BNP) Zhou et al. (2024); Choi et al. (2025). However, such invasive modifications complicate the inference pipeline and carry the risk of degrading general reasoning capabilities by inadvertently removing useful semantic features. In contrast, training-time debiasing aims to internalize robustness. Yet, existing supervised strategies mostly rely on static augmentation or distillation Liusie et al. (2024); Xue et al. (2024). These methods generally treat permutations as independent static samples, limiting the model to passive imitation of the data distribution rather than actively exploring the policy space to discover robust, permutation-invariant strategies. Fundamentally, selection bias reflects a failure of robust reasoning in discrete-choice prompting: when only non-semantic factors (labels or positions) change, the model should preserve the same semantic decision. Since this is an intrinsic reasoning flaw, a natural solution is to train the model to maintain consistency under such perturbations, rather than relying on post-hoc calibration at inference. In this context, recent reinforcement learning (RL) methods such as Group Relative Policy Optimization (GRPO) Shao et al. (2024) are appealing, as they estimate relative advantages from multiple samples and can directly optimize decision quality. However, standard GRPO and its variants still typically treat different prompts as independent training instances. When multiple prompt variants are derived from the same semantic instance under different permutations, they are optimized separately without an explicit cross-permutation consistency constraint. Consequently, the model may achieve high reward under a “favorable” permutation while performing poorly under others (e.g., reversed order), yet this inconsistency is not systematically penalized. We term this failure mode permutation-blindness. To address this, we propose Permutation-Aware Group Relative Policy Optimization (PA-GRPO), which explicitly targets permutation invariance: for all permuted variants of the same semantic instance, the model should make semantically consistent choices. PA-GRPO organizes multiple permutations of the same instance into a Permutation Group and injects permutation awareness via two mechanisms: (1) Cross-Permutation Advantage: we elevate the advantage baseline from the sample mean within a single prompt to the mean over all samples across the permutation group, so that positive advantages correspond to being globally better across permutations; (2) Consistency-Aware Reward: we explicitly encourage consistent decisions across permutations in the reward, suppressing locally optimal strategies that exploit positional or label shortcuts. We evaluate our approach on seven benchmarks spanning both MCQ and LLM-as-a-Judge tasks. During training, we construct representative permutation sets for each semantic instance, covering all label and position bindings, and validate on Llama3.1-8B-Instruct, Qwen3-8B, and Qwen3-32B. Experimental results show that, compared with strong baselines such as PIF Xue et al. (2024) and CalibraEval Li et al. (2025b), PA-GRPO achieves superior performance in both reducing selection bias and preserving stable performance. Our contributions are threefold: ∙ We identify the permutation-blindness issue in RL alignment under selection-biased settings and propose permutation invariance as a training objective. ∙ We propose PA-GRPO, which incorporates consistency constraints into training through a cross-permutation advantage and a consistency-aware reward. ∙ Extensive experiments on three backbones and seven benchmarks verify that PA-GRPO effectively reduces selection bias. 2 Related Work LLM-as-a-Judge and MCQ. With the rapid evolution of LLMs, efficient and reliable evaluation has become a central challenge. A unifying perspective is that many widely used evaluation protocols can be formulated as discrete-choice decision making, where the model selects from a closed set of candidates. One representative line is LLM-as-a-Judge Zheng et al. (2023b), which utilizes strong LLMs to perform pairwise comparisons of generated responses and has been widely adopted for alignment studies and leaderboard-style benchmarking Li et al. (2025a). Another representative line is Multiple Choice Question (MCQ) evaluation Hendrycks et al. (2020), a standard format for assessing general reasoning by constraining the output space to predefined options. Although their application scenarios differ, both settings ultimately reduce to assigning probabilities over a closed label set (e.g., A/B or A/B/C/D). Crucially, this shared discrete-choice nature makes both settings inherently susceptible to non-semantic interference: even minor perturbations to the input form can trigger drastic fluctuations in decision outcomes, undermining the reliability of such discriminative tasks Pezeshkpour and Hruschka (2024); Holtzman et al. (2021). Selection Bias in LLMs. Models in discrete-choice tasks (e.g., LLM-as-a-Judge, MCQ) are sensitive to non-semantic factors, known as selection bias, including label bias and position bias. Label bias, preferring specific IDs, is addressed by approaches like SDC Wang and Liu (2025), LOOC Reif and Schwartz (2024), and ICC Jang et al. (2024) via content-free prompts or calibration, though often incurring extra inference overhead. Position bias Lu et al. (2022), preferring specific positions, is mitigated by mechanism adjustments: PINE Wang et al. (2024b) and RoToR Yoon et al. (2025) modify attention masks or positional IDs, while fine-tuning strategies Wang et al. (2025) reduce sensitivity during training. Mitigation strategies for selection bias fall into two categories. Inference-time debiasing includes calibration methods like PriDe Zheng et al. (2023a) and CalibraEval Li et al. (2025b), which remove priors post-hoc without fixing the underlying mechanism. Alternatively, internal interventions such as UniBias Zhou et al. (2024) and BNP Choi et al. (2025) prune bias sources but risk unintended side effects. In contrast, training-time debiasing aims to internalize robustness. However, existing supervised strategies Liusie et al. (2024); Xue et al. (2024) typically treat permutations as static samples, limiting models to passive imitation rather than actively exploring the policy space to discover robust, permutation-invariant strategies. Reinforcement Learning for Alignment. Reinforcement Learning (RL) Christiano et al. (2017) has become a core technology for alignment, evolving from RLHF (PPO) Ouyang et al. (2022) to DPO Rafailov et al. (2023), and the efficient GRPO Shao et al. (2024) with its variants Yu et al. (2025); Zheng et al. (2025). While these methods significantly improve generation quality, they fundamentally rely on a sample-independence assumption. Even group-based methods like GRPO typically treat permuted variants of the same semantic input as unrelated, independent events. This leads to permutation-blindness: the optimization focuses on maximizing reward for a specific order in isolation, ignoring consistency across permutations. Consequently, existing RL frameworks fail to enforce global consistency across input variations, a critical limitation that our work seeks to address. Figure 2: Comparison between Standard GRPO (left) and the proposed PA-GRPO (right). Standard GRPO treats permuted prompts as independent samples, suffering from permutation blindness where inconsistency goes unpunished. In contrast, PA-GRPO organizes samples into Permutation Groups. It introduces (1) a Cross-Permutation Advantage (using the permutation group mean as a baseline) and (2) a Consistency-Aware Reward to explicitly enforce semantic invariance across different permutations of the same instance. 3 Methodology We propose PA-GRPO (Permutation-Aware Group Relative Policy Optimization) to mitigate selection bias induced by non-semantic factors in discrete-choice reasoning tasks. As illustrated in Figure 2, unlike standard preference optimization which treats each prompt variation independently, PA-GRPO explicitly models the dependency among permutation variants. Our core objective is to jointly optimize for two goals: (1) maximizing semantic correctness to align with the ground truth, and (2) enforcing permutation invariance to maintain consistent decisions regardless of option order. 3.1 Problem Formulation A base instance x consists of a question q and a candidate set C. We unify MCQ (||=4|C|=4) and Pairwise Judge (||=2|C|=2) as conditional generation tasks. Given a prompt p, a policy πθ(y∣p) _θ(y p) generates a response y. We parse the surface label ℓ(y)∈ℒ (y) (e.g., A/B/C/D) from the output. Ideally, a robust model should satisfy permutation invariance: for any non-semantic permutation τ, the mapped semantic choice should remain invariant. Selection bias is defined as the violation of this property, manifested as systematic preference flips triggered by label or position changes. 3.2 Permutation Group Construction and Sampling To enforce cross-permutation consistency, we organize training data into Permutation Groups. For each base instance x, we apply a set of permutation mappings Π to generate P distinct prompt variants: (x)=p(t)=τt(x)t=1P,τt∈Π.G(x)=\p^(t)= _t(x)\_t=1^P, _t∈ . (1) For each permuted prompt p(t)p^(t), we further sample a response group of N outputs y(t,i)i=1N\y^(t,i)\_i=1^N from the current policy πθ _θ (as in GRPO). Together, this yields a two-level structure: sampling diversity within each permutation and robustness across permutations. In the following sections, we compute rewards and advantage baselines over all P×NP× N samples, enabling explicit cross-permutation learning and reducing reliance on any single “favorable” permutation. Specifically, we design the permutation set Π for different tasks as follows: MCQ (P=5P=5, Structured Subset). The full permutation space (4!=244!=24) is costly in computation. We employ a structured subset strategy consisting of four cyclic shifts and one reverse order: ΠMCQ=ABCD,BCDA,CDAB,DABC,DCBA. _MCQ=\ABCD,BCDA,CDAB,DABC,DCBA\. (2) The four cyclic shifts ensure strictly uniform coverage of each option at every position. We additionally include the reverse order (DCBA) to disrupt relative adjacency patterns (e.g., A is always before B in cyclic shifts), thereby enhancing robustness against complex order-dependent biases. Judge (P=2P=2, Full Set). For pairwise comparison, the space is small (2!=22!=2). We employ the full symmetric set: ΠJudge=AB,BA. _Judge=\AB,BA\. (3) 3.3 Reward Design: Content Alignment and Consistency Awareness Standard RLHF Ouyang et al. (2022) typically relies on a scalar reward signal reflecting response correctness. However, in the context of debiasing, a correct response under a specific permutation is insufficient; the model must demonstrate robust correctness across all variations. Therefore, we design a composite reward mechanism that explicitly decouples the evaluation into two dimensions: (1) Preliminary Reward (rprer_pre), which assesses whether the current response matches the ground truth; (2) Consistency Reward (rconr_con), which penalizes semantic divergence within the permutation group. To implement this, we first establish a mapping from surface labels to semantic identities. 3.3.1 Content Alignment: From Labels to Semantic Choices Consistency must be defined over the selected content, not the surface letters. For each permutation mapping τt _t, we define a mapping function mτtm_ _t that maps a surface label to the index of the original semantic candidate: mτt(ℓ)∈1,…,||.m_ _t( )∈\1,…,|C|\. (4) For the i-th sampled response under permutation t, the semantic choice is: z(t,i)=mτt(ℓ(y(t,i))).z^(t,i)=m_ _t( (y^(t,i))). (5) 3.3.2 Preliminary Reward For each sampled response y(t,i)y^(t,i), the total reward is: r(t,i)=rpre(t,i)+λrcon(t,i).r^(t,i)=r_pre^(t,i)+λ\,r_con^(t,i). (6) The preliminary reward aggregates accuracy, length, and format constraints: rpre(t,i)=racc(t,i)+rlen(t,i)+rfmt(t,i).r_pre^(t,i)=r_acc^(t,i)+r_len^(t,i)+r_fmt^(t,i). (7) In our implementation, we use: correctness reward racc∈+1,−1r_acc∈\+1,-1\ based on whether z(t,i)z^(t,i) matches the ground-truth semantic answer, along with minor regularization terms for length (rlen=±0.1r_len=± 0.1) and formatting (rfmt=±0.3r_fmt=± 0.3). 3.3.3 Consistency-Aware Reward We introduce a consistency reward rconr_con to encourage semantically consistent decisions within each permutation group (x)G(x). Judge (P=2P=2): index-aligned pairwise consistency. We pair the i-th samples across the two permutations: rcon(1,i)=rcon(2,i)=+1,z(1,i)=z(2,i)−1,z(1,i)≠z(2,i).r_con^(1,i)=r_con^(2,i)= cases+1,&z^(1,i)=z^(2,i)\\ -1,&z^(1,i)≠ z^(2,i). cases (8) MCQ (P=5P=5): unique-mode agreement. Let nkn_k be the count of semantic choice k across the entire group, and ℳM be the set of modes: nk n_k =∑t=1P∑i=1N[z(t,i)=k], = _t=1^P _i=1^N1 [z^(t,i)=k ], (9) ℳ =argmaxknk. = _kn_k. If |ℳ|=1|M|=1, denote the unique mode by z⋆z . We explicitly penalize fragmentation (including ties): rcon(t,i)=+1,|ℳ|=1andz(t,i)=z⋆−1,otherwise.r_con^(t,i)= cases+1,&|M|=1\ and\ z^(t,i)=z \\ -1,&otherwise. cases (10) 3.4 Cross-Permutation Advantage Estimation Prompts in a permutation group (x)G(x) are semantically identical but differ in non-semantic surface factors. If advantages are computed independently per prompt variant (as in standard GRPO), the policy can be reinforced by a “favorable” ordering even when it fails under other permutations. PA-GRPO therefore lifts advantage estimation to the permutation-group level. We treat the P×NP× N samples from (x)G(x) as a single comparison set and compute the permutation-group mean and standard deviation: μ _G =1PN∑t=1P∑i=1Nr(t,i), = 1PN _t=1^P _i=1^Nr^(t,i), (11) σ _G =Std(r(t,i)t,i), =Std (\r^(t,i)\_t,i ), where both statistics are computed over all permutations and samples within (x)G(x). The cross-permutation advantage is: APA(t,i)=0,σ<δr(t,i)−μσ+ϵ,otherwise.A_PA^(t,i)= cases0,& _G<δ\\ r^(t,i)- _G _G+ε,&otherwise. cases (12) This yields (i) instance-wise competition across permutations via the permutation-group baseline μ _G: a sample receives a positive advantage only if its reward exceeds the permutation-group mean, and (i) scale-invariant stability via normalization by σ _G. When σ<δ _G<δ, rewards are nearly indistinguishable within the group, so we set APA=0A_PA=0 to avoid amplifying noise. 3.5 Optimization Objective We adopt a clipped objective with KL regularization to stabilize training. Let the importance ratio be defined as: ρ(t,i)(θ)=πθ(y(t,i)∣p(t))πold(y(t,i)∣p(t))ρ^(t,i)(θ)= _θ(y^(t,i) p^(t)) _old(y^(t,i) p^(t)) (13) The clipped surrogate objective is: ℒclip(θ)=[min(ρ(t,i)APA(t,i),clip(ρ(t,i),1±η)APA(t,i))].L_clip(θ)=E [ (ρ^(t,i)A_PA^(t,i),\\ clip(ρ^(t,i),1±η)A_PA^(t,i) ) ]. (14) The final objective maximizes the surrogate reward while penalizing deviation from the reference policy: maxθ(ℒclip(θ)−βp(t)[KL(πθ(⋅|p(t))∥πref(⋅|p(t)))]). _θ\ (L_clip(θ)\ -\\ β\,E_p^(t) [KL\! ( _θ(·|p^(t)) _ref(·|p^(t)) ) ] ). (15) The complete training procedure is provided in Algorithm 1 in Appendix B. 4 Experimental Setup In this section, we present a comprehensive evaluation of PA-GRPO to answer the following research questions (RQs): RQ1: Could PA-GRPO outperform existing inference-time calibration and training-time debiasing methods? RQ2: What are the individual contributions of the Cross-Permutation Advantage (APAA_PA) and the Consistency-Aware Reward (rconr_con)? RQ3: How do the consistency reward coefficient λ and permutation group size P affect model performance, permutation invariance, and training efficiency? RQ4: Can PA-GRPO effectively align reasoning consistency under Chain-of-Thought prompting? RQ5: How does PA-GRPO’s robustness differ between isolated label-only and order-only perturbations and the naturally coupled permutations used in training, and what does this reveal about the remaining sources of bias? 4.1 Datasets and Evaluation Protocol We evaluated PA-GRPO on seven benchmarks spanning two categories: (1) LLM-as-a-Judge: MT-Bench Zheng et al. (2023b), JudgeBench Tan et al. (2024), PreferenceBench Kim et al. (2024), and RewardBench Lambert et al. (2025); and (2) MCQ: ARC-Challenge Clark et al. (2018), TinyMMLU Polo et al. (2024), and GPQA Rein et al. (2024). Brief descriptions of each benchmark and their task formats are provided in Appendix C. For training, we utilized the Chatbot Arena dataset Zheng et al. (2023b) for pairwise tasks and the MMLU training set Hendrycks et al. (2020) for reasoning tasks. Details of data construction are provided in Appendix D. To rigorously quantify permutation invariance during inference, we employed Full Permutation Expansion rather than random sampling. Specifically, we evaluated the complete permutation space (N!N!) for every test instance, covering all 2!=22!=2 possible orders for LLM-as-a-Judge tasks and all 4!=244!=24 possible arrangements for MCQ tasks. 4.2 Metrics We report three core metrics to measure performance and robustness: Accuracy (Acc). The standard performance metric, calculated as the average correctness over all samples in the expanded permutation groups (N!N!). Consistency (Con). It measures decision stability across permutations. For LLM-as-a-Judge, it is defined as the proportion of instance pairs where the model makes the identical semantic choice under both orders. For MCQ, we define it as the majority consensus ratio nmode/24n_mode/24, where nmoden_mode is the count of the most frequent semantic choice within the permutation group. Consistent Accuracy (CA). It measures the ability to be both consistent and correct. For LLM-as-a-Judge, we apply a strict criterion: an instance scores 1 if and only if the model answers correctly under both permutations. For MCQ, we adopt Majority Vote Correctness Wang et al. (2022): an instance scores 1 if the mode of the permutation group matches the ground truth. This reflects whether the consensus converges to the truth after filtering out permutation noise. LLM-as-a-Judge Multiple Choice Questions Model Method MT-Bench JudgeBench PreferenceBench RewardBench GPQA ARC-Challenge TinyMMLU Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Llama-3.1-8B Base 59.6 25.2 22.2 35.0 34.8 6.1 60.8 22.6 22.1 60.5 31.5 26.2 27.3 40.0 27.2 73.0 82.5 78.9 54.2 68.9 57.0 GRPO 75.7 80.6 65.4 48.2 56.1 28.2 82.2 85.1 76.3 70.9 76.9 61.5 29.5 55.6 29.7 89.0 92.1 91.5 67.3 77.9 73.0 UniBias 68.5 59.4 48.2 50.2 23.0 10.9 79.7 65.6 62.5 68.4 64.4 50.6 32.3 56.1 31.8 80.4 90.0 83.6 61.1 76.5 65.0 PriDe 65.6 46.2 38.9 49.1 16.2 7.2 81.6 69.1 66.2 65.1 52.8 41.5 31.0 49.1 27.4 79.1 83.0 79.0 59.7 69.5 58.0 CalibraEval 62.3 42.1 33.4 49.3 15.7 7.1 70.4 44.4 42.7 60.7 34.4 27.8 28.3 48.2 26.5 74.4 71.0 68.3 55.6 72.1 62.0 PIF 76.1 84.6 70.4 53.3 59.2 30.4 54.3 37.1 25.7 73.7 76.7 62.0 29.1 49.7 29.2 73.3 83.0 78.8 56.9 69.1 57.0 PA-GRPO 77.6 (+1.5) 88.0 (+3.4) 71.7 (+1.3) 57.1 (+3.8) 58.3 (-0.9) 32.4 (+2.0) 86.2 (+4.0) 87.2 (+2.1) 79.8 (+3.5) 71.0 (-2.7) 82.7 (+5.8) 62.3 (+0.3) 32.1 (-0.2) 57.2 (+1.1) 33.2 (+1.4) 89.3 (+0.3) 92.9 (+0.8) 92.6 (+1.1) 67.9 (+0.6) 81.9 (+4.0) 75.0 (+2.0) Qwen3-8B Base 71.2 56.3 49.4 43.9 45.5 16.5 82.3 70.2 67.4 79.3 73.8 66.2 32.5 48.2 30.4 90.5 94.7 92.4 67.4 79.6 72.0 GRPO 78.1 82.5 69.3 50.4 62.6 34.8 82.9 77.3 76.2 86.8 86.7 80.2 40.4 63.2 43.8 94.6 97.2 95.9 56.6 83.7 76.0 UniBias 67.0 52.7 43.4 52.2 26.1 14.9 75.8 55.4 53.5 66.6 45.9 39.5 34.6 54.3 34.2 89.9 94.1 91.8 71.3 81.0 74.0 PriDe 72.6 66.7 52.1 51.2 48.8 29.8 81.5 72.6 69.7 80.9 79.1 68.7 34.4 56.9 38.5 91.2 94.9 92.6 68.9 79.9 78.0 CalibraEval 72.4 62.1 50.9 49.7 56.4 31.3 76.5 72.8 70.1 80.4 71.2 69.5 33.5 49.8 32.9 90.1 91.0 89.8 63.5 75.5 68.0 PIF 75.1 83.2 68.9 54.3 59.6 37.4 77.7 64.2 58.9 83.9 80.1 75.8 32.7 68.7 46.6 92.2 95.4 93.5 75.1 81.4 75.0 PA-GRPO 78.8 (+0.7) 86.4 (+3.2) 72.0 (+2.7) 60.1 (+5.8) 70.0 (+7.4) 45.3 (+7.9) 88.9 (+6.0) 89.6 (+12.3) 83.7 (+7.5) 84.4 (-2.4) 88.9 (+2.2) 80.6 (+0.4) 42.3 (+1.9) 72.4 (+3.7) 56.7 (+10.1) 95.0 (+0.4) 97.5 (+0.3) 96.2 (+0.3) 73.2 (-1.9) 88.6 (+4.9) 80.0 (+2.0) Qwen3-32B Base 78.5 83.4 70.3 55.4 62.1 29.7 84.9 85.1 78.7 85.4 84.3 77.5 40.4 59.9 46.7 94.5 97.2 95.7 76.5 86.0 81.0 GRPO 81.0 90.6 73.7 61.4 74.2 45.1 89.1 90.3 83.3 88.2 91.4 83.9 53.2 74.3 59.9 95.3 98.1 97.0 85.2 91.4 86.0 UniBias 76.5 85.6 71.1 56.2 64.0 32.5 87.2 91.1 84.1 85.7 85.9 78.0 46.9 68.9 54.5 94.5 95.8 95.1 81.0 89.3 80.0 PriDe 77.2 86.1 72.1 56.8 63.5 33.1 85.1 86.7 81.2 86.9 88.2 80.3 49.7 73.1 58.5 95.5 97.4 95.8 79.9 84.1 84.0 CalibraEval 78.8 79.1 71.2 52.9 61.0 28.9 83.7 87.2 79.9 85.5 86.7 79.6 48.0 72.2 55.2 93.9 96.8 96.0 80.3 83.3 85.0 PIF 79.4 86.5 72.8 62.2 68.4 36.9 89.3 90.6 85.5 90.8 90.9 84.7 52.1 72.8 60.1 96.1 97.6 96.8 84.3 92.0 88.0 PA-GRPO 81.4 (+0.4) 91.6 (+1.0) 74.8 (+1.1) 59.4 (-2.8) 75.2 (+1.0) 43.4 (-1.7) 90.2 (+0.9) 92.1 (+1.0) 86.2 (+0.7) 90.5 (-0.3) 92.1 (+0.7) 86.6 (+1.9) 54.1 (+0.9) 75.1 (+0.8) 61.2 (+1.1) 96.0 (-0.1) 98.3 (+0.2) 97.2 (+0.2) 86.8 (+1.6) 91.8 (-0.2) 89.0 (+1.0) Table 1: Main results on Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-32B across 7 benchmarks. All scores are rounded to one decimal place. PA-GRPO results show the absolute score (top) and the gain/drop compared to the best baseline (bottom). Bold indicates the best result in each column. 4.3 Baselines We compare PA-GRPO with five strong baselines covering both inference-time debiasing and training-time alignment. Inference-time methods include PriDe Zheng et al. (2023a), which estimates and removes a global prior preference over option-ID tokens; CalibraEval Li et al. (2025b), which learns a label-free calibration mapping to enforce swap-invariant judging; and UniBias Zhou et al. (2024), which mitigates bias by masking identified biased attention heads and FFN vectors. Training-time baselines include PIF Xue et al. (2024), an SFT algorithm that strengthens multiple-choice symbol binding via point-wise feedback on constructed negative instances, and standard GRPO Shao et al. (2024). Due to space constraints, detailed mechanisms and our implementation settings are provided in Appendix E. 4.4 Implementation Details We used Llama-3.1-8B-Instruct Dubey et al. (2024), Qwen3-8B, and Qwen3-32B Yang et al. (2025) as policy models. All models were trained within the verl Sheng et al. (2025) RL framework, and we employed LoRA Hu et al. (2022) for fine-tuning. Detailed training configurations and hyperparameters are provided in Appendix F. 5 Results and Analysis LLM-as-a-Judge Multiple Choice Questions Method MT-Bench JudgeBench PreferenceBench RewardBench GPQA ARC-Challenge TinyMMLU Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Base 59.6 25.2 22.2 35.0 34.8 6.1 60.8 22.6 22.1 60.5 31.5 26.2 27.3 40.0 27.2 73.0 82.5 78.9 54.2 68.9 57.0 GRPO 75.7 80.6 65.4 48.2 56.1 28.2 82.2 85.1 76.3 70.9 76.9 61.5 29.5 55.6 29.7 89.0 92.1 91.5 67.3 77.9 73.0 + rconr_con 75.2 83.1 66.5 48.0 56.5 29.2 82.6 85.9 76.9 70.6 79.0 61.9 29.9 56.4 31.0 87.8 92.3 92.0 67.1 78.8 73.0 + APAA_PA 76.6 84.2 67.8 49.3 55.4 30.5 83.4 86.4 77.8 69.8 78.7 61.1 30.7 57.1 32.2 89.7 92.3 91.9 67.5 79.4 74.0 PA-GRPO 77.6 88.0 71.7 57.1 58.3 32.4 86.2 87.2 79.8 71.0 82.7 62.3 32.1 57.2 33.2 89.3 92.9 92.6 67.9 81.9 75.0 Table 2: Ablation study on Llama-3.1-8B-Instruct across all 7 benchmarks. We group datasets into LLM-as-a-Judge tasks and MCQ tasks. The best results are bolded. Adding both components (APAA_PA and rconr_con) consistently yields the best robustness. LLM-as-a-Judge Multiple Choice Questions Setting MT-Bench JudgeBench PreferenceBench RewardBench GPQA ARC-Challenge TinyMMLU Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Base 59.6 25.2 22.2 35.0 34.8 6.1 60.8 22.6 22.1 60.5 31.5 26.2 27.3 40.0 27.2 73.0 82.5 78.9 54.2 68.9 57.0 λ=0.5λ=0.5 78.0 86.4 69.8 56.5 55.4 31.9 85.3 86.8 78.9 70.7 80.9 60.5 32.1 56.6 32.0 89.4 92.0 92.4 67.1 78.2 74.0 λ=1.0λ=1.0 (Default) 77.6 88.0 71.7 57.1 58.3 32.4 86.2 87.2 79.8 71.0 82.7 62.3 32.1 57.2 33.2 89.3 92.9 92.6 67.9 81.9 75.0 λ=2.0λ=2.0 76.5 88.6 70.5 56.8 55.9 32.0 85.0 87.5 79.1 70.3 82.5 62.1 31.4 58.0 33.6 88.7 91.2 91.1 66.4 80.1 73.0 Table 3: Impact of the consistency reward coefficient (λ) on Llama-3.1-8B-Instruct. We group datasets into LLM-as-a-Judge tasks and MCQ tasks. λ=1.0λ=1.0 achieves the best balance. The best results are bolded. 5.1 Main Results As shown in Table 1, across three different LLMs, PA-GRPO yields consistent gains over the strongest baselines, covering both inference-time calibration and training-time debiasing methods. The improvements are particularly concentrated on permutation-sensitive robustness metrics, including Consistency and Consistent Accuracy, while keeping standard Accuracy competitive. For Llama-3.1-8B-Instruct, PA-GRPO raises MT-Bench accuracy to 77.6% and consistency to 88.0%. This representative case suggests that permutation-aware optimization not only improves the headline score but also mitigates decision instability under reordering. This aligns with our motivation that selection bias often manifests as unstable preferences across equivalent permutations. For Qwen3-8B, the gain becomes more salient in LLM-as-a-Judge evaluations; for instance, JudgeBench accuracy reaches 60.1%, indicating that PA-GRPO regularizes the model more effectively in settings susceptible to selection bias, preventing it from relying on arbitrary presentation factors (e.g., position or label) for pairwise judgments. Meanwhile, the most notable improvement on MCQ tasks is observed on GPQA, where Consistent Accuracy improves from 46.6% to 56.7%, implying that the permutation-group consensus converges more reliably to the ground truth after filtering out permutation noise in complex scientific reasoning. For the larger LLM, Qwen3-32B, although the absolute performance is already high, the gains remain persistent; MT-Bench consistency increases to 91.6%, suggesting that PA-GRPO continues to refine stability even with limited headroom. While a few benchmark-specific metrics show slight regressions, these appear isolated and do not alter the overall trend that enforcing permutation-aware optimization enhances stability and reliability across permutations in both LLM-as-a-Judge and multiple-choice evaluations. 5.2 Ablation Study Table 2 examines the contributions of the Consistency-Aware Reward rconr_con and the Cross-Permutation Advantage APAA_PA on Llama-3.1-8B-Instruct. Adding rconr_con to GRPO explicitly encourages within-group agreement, effectively raising MT-Bench consistency from 80.6% to 83.1% without sacrificing correctness. In contrast, enabling APAA_PA reduces permutation-induced reward variance during advantage estimation, which translates into broader gains, such as improving JudgeBench accuracy from 48.2% to 49.3%. Crucially, the two components are complementary: rconr_con shapes the model toward agreement, while APAA_PA stabilizes the group-level optimization signal. Their combination yields the strongest overall robustness, boosting PreferenceBench consistent accuracy from 76.3% to 79.8% and reaching 75.0% on TinyMMLU. While minor benchmark-specific accuracy trade-offs are observed, the results confirm that both components are essential for reliable cross-permutation behavior. Figure 3: Impact of Permutation Group Size (P). 5.3 Hyperparameter Analysis We analyze two key hyperparameters of PA-GRPO on Llama-3.1-8B-Instruct: the consistency reward coefficient λ and the permutation group size P. Table 3 summarizes the effect of λ across LLM-as-a-judge and MCQ benchmarks, and Figure 3 reports results for different P on MCQ tasks. Consistency reward coefficient λ. We evaluate λ values of 0.5, 1.0, and 2.0 on Llama-3.1-8B-Instruct. We select λ=1.0λ=1.0 as it optimally balances performance and consistency, achieving 79.8% consistent accuracy on PreferenceBench. In contrast, λ=0.5λ=0.5 sacrifices stability for marginal accuracy gains on MT-Bench, while λ=2.0λ=2.0 provides no significant performance benefit to justify the stronger regularization. Permutation group size P. As illustrated in Figure 3, for MCQ tasks, we compare P values of 4, 5, and 24. We find that P=5P=5, which adds a reverse order to cyclic shifts, outperforms P=4P=4 by effectively disrupting adjacency bias. Although expanding to the full set where P=24P=24 increases TinyMMLU consistent accuracy from 75.0% to 77.0%, the marginal gain does not justify the nearly five-fold computational cost. We therefore adopt P=5P=5 for efficiency. 5.4 Chain-of-Thought Analysis We further investigate whether the robustness gains of PA-GRPO depend on explicit chain-of-thought (CoT) generation. Detailed protocols and full results are provided in Appendix G. Overall, our analysis reveals complementary benefits: PA-GRPO internalizes alignment directly into the policy for intrinsic robustness, while utilizing CoT further amplifies these gains to maximize performance. 5.5 Bias Decomposition Analysis We conduct a controlled bias decomposition to isolate label-symbol effects, display-order effects, and their coupled permutation. Full protocol definitions and detailed results are provided in Appendix H. We highlight two observations. First, residual sensitivity to display order is often more pronounced than sensitivity to label symbols, particularly on LLM-as-a-Judge benchmarks. Second, PA-GRPO is most robust under the coupled permutation used in standard evaluation. 6 Conclusion In this paper, we proposed PA-GRPO, a novel RL framework designed to mitigate selection bias in LLMs. By integrating a cross-permutation advantage and a consistency-aware reward, our PA-GRPO explicitly enforces semantic invariance across different option arrangements. Extensive experiments on seven benchmarks demonstrate that PA-GRPO significantly outperforms strong baselines, effectively reducing position and label biases while maintaining high reasoning accuracy. This work highlights the efficacy of internalizing consistency constraints to achieve robust and reliable discrete-choice reasoning. Limitations While PA-GRPO proves effective, we acknowledge specific limitations in our current study scope. First, our method is primarily tailored for discrete-choice tasks (e.g., MCQ and pairwise judgments) where permutation groups are naturally defined. Extending this consistency-based objective to open-ended generation remains a challenge, as semantic equivalence is harder to quantify in long-form text. Second, our evaluation is currently centered on English benchmarks and open-weights models. The interaction between permutation invariance and other systematic artifacts, such as verbosity bias Dubois et al. (2024) or culture-specific biases in multilingual settings, has not yet been fully explored. We view our work as a foundational step toward permutation-robust alignment, leaving these broader generalization tests for future research. Ethics Considerations The research presented in this paper is founded on open-source LLMs and utilizes publicly available datasets. Consequently, we do not anticipate that our study will have any direct adverse effects. However, it is crucial to recognize that any generative AI technology, including the contributions of our research, must be implemented with caution to avert potentially harmful outcomes. References J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, et al. (2024) Chatbot arena: an open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, Cited by: §1. H. K. Choi, W. Xu, C. Xue, S. Eckman, and C. K. Reddy (2025) Mitigating selection bias with node pruning and auxiliary options. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5190–5215. Cited by: §1, §2. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. Advances in neural information processing systems 30. Cited by: §2. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: Appendix C, §4.1. T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré (2022) Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35, p. 16344–16359. Cited by: Appendix F. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.4. Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto (2024) Length-controlled alpacaeval: a simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. Cited by: Limitations. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020) Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: §1, §2, §4.1. A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer (2021) Surface form competition: why the highest probability answer isn’t always right. arXiv preprint arXiv:2104.08315. Cited by: §2. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §4.4. J. Jang, S. Jang, W. Kweon, M. Jeon, and H. Yu (2024) Rectifying demonstration shortcut in in-context learning. arXiv preprint arXiv:2403.09488. Cited by: §2. N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard (2019) Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456. Cited by: Appendix F. S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo (2024) Prometheus 2: an open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535. Cited by: Appendix C, §4.1. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626. Cited by: Appendix F. N. Lambert, V. Pyatkin, J. Morrison, L. J. V. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. (2025) Rewardbench: evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 1755–1797. Cited by: Appendix C, §4.1. D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, et al. (2025a) From generation to judgment: opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 2757–2791. Cited by: §2. H. Li, J. Chen, Q. Ai, Z. Chu, Y. Zhou, Q. Dong, and Y. Liu (2025b) Calibraeval: calibrating prediction distribution to mitigate selection bias in llms-as-judges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 16537–16552. Cited by: Appendix E, Appendix E, §1, §1, §2, §4.3. A. Liusie, Y. Fathullah, and M. Gales (2024) Teacher-student training for debiasing: general permutation debiasing for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, p. 1376–1387. Cited by: §1, §2. I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: Appendix F. Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp (2022) Fantastically ordered prompts and where to find them: overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8086–8098. Cited by: §2. Y. Miao, S. Zhang, L. Ding, R. Bao, L. Zhang, and D. Tao (2024) Inform: mitigating reward hacking in rlhf via information-theoretic reward modeling. Advances in Neural Information Processing Systems 37, p. 134387–134429. Cited by: §1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §2, §3.3. P. Pezeshkpour and E. Hruschka (2024) Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, p. 2006–2017. Cited by: §2. F. M. Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin (2024) TinyBenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992. Cited by: Appendix C, §4.1. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §2. Y. Reif and R. Schwartz (2024) Beyond performance: quantifying and mitigating label bias in llms. arXiv preprint arXiv:2405.02743. Cited by: §2. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2024) Gpqa: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, Cited by: Appendix C, §4.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix E, §1, §2, §4.3. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §4.4. L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi (2024) Judging the judges: a systematic study of position bias in llm-as-a-judge. arXiv preprint arXiv:2406.07791. Cited by: §1. S. Tan, S. Zhuang, K. Montgomery, W. Y. Tang, A. Cuadron, C. Wang, R. A. Popa, and I. Stoica (2024) Judgebench: a benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. Cited by: Appendix C, §4.1. P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, et al. (2024a) Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9440–9450. Cited by: §1. X. Wang and X. Liu (2025) Beyond generation: leveraging llm creativity to overcome label bias in classification. In Findings of the Association for Computational Linguistics: ACL 2025, p. 25500–25506. Cited by: §2. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2022) Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Cited by: §4.2. Y. Wang, F. Xiong, Y. Wang, L. Li, X. Chu, and D. D. Zeng (2025) Position bias mitigates position bias: mitigate position bias through inter-position knowledge distillation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 1495–1512. Cited by: §2. Z. Wang, H. Zhang, X. Li, K. Huang, C. Han, S. Ji, S. M. Kakade, H. Peng, and H. Ji (2024b) Eliminating position bias of language models: a mechanistic approach. arXiv preprint arXiv:2407.01100. Cited by: §2. S. Wei, C. Wu, H. Huang, and H. Chen (2024) Unveiling selection biases: exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics ACL 2024, p. 5598–5621. Cited by: §1. M. Xue, Z. Hu, L. Liu, K. Liao, S. Li, H. Han, M. Zhao, and C. Yin (2024) Strengthened symbol binding makes large language models reliable multiple-choice selectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4331–4344. Cited by: Appendix E, §1, §1, §2, §4.3. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.4. S. Yoon, D. Ahn, Y. Lee, M. Jung, H. Jang, and S. Hwang (2025) RoToR: towards more reliable responses for order-invariant inputs. arXiv preprint arXiv:2502.08662. Cited by: §2. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §2. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh (2021) Calibrate before use: improving few-shot performance of language models. In International conference on machine learning, p. 12697–12706. Cited by: §1. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §2. C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang (2023a) Large language models are not robust multiple choice selectors. arXiv preprint arXiv:2309.03882. Cited by: Appendix E, Appendix E, Appendix E, §1, §1, §2, §4.3. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023b) Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: Appendix C, §1, §2, §4.1. H. Zhou, Z. Feng, Z. Zhu, J. Qian, and K. Mao (2024) Unibias: unveiling and mitigating llm bias through internal attention and ffn manipulation. Advances in Neural Information Processing Systems 37, p. 102173–102196. Cited by: Appendix E, Appendix E, §1, §2, §4.3. Appendix A Statement on the Use of Large Language Models In this study, Large Language Models (LLMs) were employed solely for the purpose of grammatical correction and text refinement to improve readability. They were not involved in the conceptualization of the research, generation of content, or any substantive scientific contribution. The authors retain full responsibility for the content of this manuscript, ensuring its originality and accuracy. Appendix B Training Procedure Algorithm 1 Permutation-Aware GRPO (PA-GRPO) 0: Policy πθ _θ, Ref πref _ref, Set Π , Params P,N,λ,β,ηP,N,λ,β,η 1: for each batch ℬB do 2: Initialize loss ℒbatch←0L_batch← 0 3: for each instance x∈ℬx do 4: // Step 1: Group Construction 5: Construct (x)=p(t)=τt(x)t=1PG(x)=\p^(t)= _t(x)\_t=1^P 6: // Step 2: Sampling & Evaluation 7: Sample N outputs y(t,i)y^(t,i) for all t; map to semantics z(t,i)z^(t,i). 8: Calc rpre(t,i)r_pre^(t,i) (Accuracy + Len + Fmt). 9: // Step 3: Consistency Calculation (rconr_con) 10: if Judge then 11: rcon(t,i)←(z(1,i)==z(2,i))?+1:−1r_con^(t,i)←(z^(1,i)==z^(2,i))?+1:-1 12: else if MCQ then 13: Find modes ℳM of all z(t,i)\z^(t,i)\. 14: if |ℳ|=1|M|=1 with unique mode z⋆z then 15: rcon(t,i)←(z(t,i)==z⋆)?+1:−1r_con^(t,i)←(z^(t,i)==z )?+1:-1 16: else 17: rcon(t,i)←−1r_con^(t,i)←-1 Penalize ties/fragmentation 18: end if 19: end if 20: // Step 4: Advantage Estimation 21: r←rpre+λrconr← r_pre+λ r_con. Compute APAA_PA via group stats (Eq. 12). 22: end for 23: // Step 5: Optimization 24: Update θ via Eq. 15 using ℒbatchL_batch. 25: Update πold←πθ _old← _θ. 26: end for Appendix C Benchmark Descriptions This appendix briefly describes the seven evaluation benchmarks used in Section 4.1. We group them into (i) pairwise judging benchmarks, where each instance contains an instruction (or question) and two candidate responses with a reference preference label, and (i) MCQ benchmarks, where each instance is a 4-option multiple-choice question with a provided answer key. Unless otherwise noted, our permutation-based evaluation protocol (Appendix D) is applied to the canonical input format of each benchmark. MT-Bench. MT-Bench Zheng et al. (2023b) is a controlled, multi-turn instruction-following benchmark designed to evaluate conversational assistants. It consists of 80 manually written, high-quality multi-turn prompts spanning eight common user categories (e.g., writing, roleplay, extraction, reasoning, math, coding, and two knowledge-oriented categories), with 10 prompts per category. Each MT-Bench prompt contains two turns, enabling evaluation of context tracking and multi-step dialogue behaviors. For pairwise judging settings, we use the standard pairwise formulation of MT-Bench prompts where a judge compares two model responses for the same prompt. JudgeBench. JudgeBench Tan et al. (2024) is a benchmark for evaluating the reliability of LLM-based judges under realistic pairwise comparison settings. Each example is a question paired with two candidate responses, accompanied by an objective preference label indicating which response is more correct. The released benchmark includes response pairs generated by strong frontier models (e.g., GPT-4o and Claude-3.5-Sonnet), and is designed to probe common judging failure modes such as order sensitivity and bias. PreferenceBench. PreferenceBench Kim et al. (2024) is an in-domain pairwise ranking benchmark introduced alongside Prometheus 2. It is constructed by pairing two responses from a pool of multiple candidate responses per prompt, producing a set of 200 prompts with 2,000 labeled response pairs. In addition, PreferenceBench includes prompt-specific evaluation criteria intended to make the comparison more fine-grained and to stress judges’ adherence to explicit rubrics. RewardBench. RewardBench Lambert et al. (2025) evaluates reward models (and judge-like scorers) on preference-style comparisons. Each instance is formatted as a prompt with a human-verified chosen completion and a rejected completion. The benchmark aggregates structured comparisons spanning several high-level categories, including chat instruction-following, reasoning-focused comparisons, and safety/refusal behavior, aiming to cover both in-distribution and challenging out-of-distribution preference cases. ARC-Challenge. ARC (AI2 Reasoning Challenge) Clark et al. (2018) is a multiple-choice science QA benchmark, where ARC-Challenge represents the more difficult subset. Questions are grade-school level science problems that typically require non-trivial reasoning beyond superficial retrieval. We use the standard ARC-Challenge configuration with its widely used train/validation/test splits and the provided answer key for evaluation. TinyMMLU. TinyMMLU Polo et al. (2024) is a compact evaluation set derived from MMLU, intended to approximate MMLU-style performance with far fewer questions. It consists of a small subsample of MMLU-format questions spanning diverse subject areas, each presented as a 4-option multiple-choice problem. This benchmark enables faster iteration while retaining broad topical coverage. GPQA. GPQA Rein et al. (2024) is a graduate-level, domain-expert-written multiple-choice QA benchmark designed to be difficult for non-experts even with access to web search. All questions are 4-option MCQ drawn from three high-level domains (biology, physics, and chemistry), with curated subsets that vary in strictness and difficulty (including a higher-quality, harder subset). We follow the standard GPQA evaluation protocol using the released answer key. Appendix D Data Construction and Evaluation Details Training Data Filtering. To ensure training efficiency and explicitly target selection bias, we performed pre-processing filtering on the source datasets (Chatbot Arena and MMLU). We utilized Llama-3.1-8B-Instruct to infer each training instance under both original and reversed orders (or cyclically shifted orders for MCQ). We specifically retained instances where the model yielded inconsistent predictions across permutations, as these samples provide the strongest signal for learning permutation invariance. Full Permutation Protocol. During inference, we strictly evaluated the complete permutation space (N!N!) to calculate consistency metrics: • LLM-as-a-Judge Tasks: For each pairwise comparison, we evaluated all 2!=22!=2 possible orders: standard (AB) and reversed (BA). • MCQ Tasks: For each 4-option question, we evaluated all 4!=244!=24 possible arrangements of the options. All reported metrics (Accuracy, Consistency, Consistent Accuracy) are aggregated results based on these fully expanded permutation groups. Appendix E Baseline Details and Implementation This appendix provides (i) the core idea of each baseline and (i) how we implemented it in our experimental pipeline. PriDe (Debiasing with Prior Estimation). PriDe is a label-free, inference-time debiasing method originally proposed for multiple-choice selection robustness Zheng et al. (2023a). Let q be the question, option-ID tokens be dii=1n\d_i\_i=1^n (e.g., A/B/C/D), and option contents be oii=1n\o_i\_i=1^n. PriDe assumes the observed prediction distribution over option IDs can be decomposed into a prior term over did_i and a debiased term over the paired option content (under a permutation) Zheng et al. (2023a): Pobs(di∣q,xI)∝Pprior(di∣q)Pdeb(ofI(i)∣q,x),P_obs(d_i q,x^I) P_prior(d_i q)\,P_deb(o_f_I(i) q,x), (16) where I indexes an option permutation (we use cyclic permutations as in the paper), and fI(i)f_I(i) maps the i-th ID token to the paired option content under I. With cyclic permutations ℐI, PriDe estimates a sample-specific prior by Zheng et al. (2023a): Pprior(di∣q)=softmax(1|ℐ|∑I∈ℐlogPobs(di∣q,xI)). splitP_prior(d_i q)=softmax (& 1|I| _I \\ & P_obs(d_i q,x^I) ). split (17) It then averages priors computed on a small estimation subset to obtain a global prior P~prior(di) P_prior(d_i), and debiases remaining samples by Zheng et al. (2023a): Pdeb(oi∣q,x)∝Pobs(di∣q,x)P~prior(di).P_deb(o_i q,x)\ \ P_obs(d_i q,x) P_prior(d_i). (18) For each dataset, we select a small subset of evaluation instances as the prior-estimation set. For each instance, we compute Pobs(di∣q,xI)P_obs(d_i q,x^I) under cyclic permutations (for pairwise judging, n=2n=2 reduces to swapping). We estimate per-instance priors using the equation above, average them into a global prior P~prior P_prior, and then apply the debiasing ratio on the remaining instances using only the default prompt once (followed by renormalization and argmax). CalibraEval (Label-free calibration for LLM-as-a-Judge). CalibraEval treats selection bias as a calibration problem and learns a mapping function g(⋅)g(·) that maps observed probabilities to an unbiased distribution Li et al. (2025b): Pdeb(ti∣I,X0)=g(Pobs(ti∣I,X0)),P_deb(t_i I,X_0)=g\! (P_obs(t_i I,X_0) ), (19) where tit_i is an option-ID token (e.g., A/B) and X0X_0 is the default pairing of tokens and contents. It optimizes g(⋅)g(·) so that the calibrated judgments remain consistent under swapping positions and/or swapping ID tokens, and proposes a non-parametric order-preserving algorithm (NOA) to solve the resulting optimization Li et al. (2025b). We follow the paper setup by constructing an estimation set of K unlabeled comparison instances. For each instance, we query the judge under the required swap variants to obtain the observed probabilities (default / swap positions / swap tokens) and fit the order-preserving mapping g(⋅)g(·) with NOA. At test time, we apply g(⋅)g(·) to the judge’s observed probability and renormalize the two-way distribution to make the final decision. UniBias (Internal attention/FFN manipulation). UniBias identifies biased model components (attention heads and FFN vectors) by analyzing their contributions to label prediction, using three criteria: (i) relatedness to label prediction, (i) bias/imbalance across labels, and (i) low variance of the injected label information across samples; thresholds are selected via grid search Zhou et al. (2024). After identifying biased components, UniBias mitigates bias by masking (eliminating) their influence during inference Zhou et al. (2024). We follow the UniBias procedure to (1) build a small support set (labeled or unlabeled, depending on availability), (2) identify biased attention heads / FFN vectors under the paper’s criteria and threshold selection, and (3) apply the corresponding masks during evaluation. This keeps the inference-time complexity essentially identical to the original model once the masks are fixed Zhou et al. (2024). PIF (Point-wise Intelligent Feedback). PIF is an SFT algorithm designed to improve Multiple-Choice Symbol Binding (MCSB). It constructs negative instances by randomly combining incorrect option contents with all candidate symbols and introduces a point-wise loss to provide direct feedback on these negatives, thereby reducing selection bias and improving accuracy Xue et al. (2024). For each training MCQ instance, we construct PIF-style negative pairs by assigning incorrect option contents to candidate symbols, and optimize the point-wise objective described in the paper. We use the same backbone model and training budget as other training-side baselines for a fair comparison. GRPO (Group Relative Policy Optimization). GRPO is a PPO-style reinforcement learning algorithm that samples a group of outputs for the same prompt and uses relative (within-group) reward signals to optimize the policy, avoiding an explicit value model and reducing memory cost compared to standard PPO Shao et al. (2024). We implement GRPO as our direct RL baseline using the same reward design and training budget as PA-GRPO, but treat each permuted prompt as an independent sample and do not enforce cross-permutation consistency. LLM-as-a-Judge Multiple Choice Questions Method MT-Bench JudgeBench PreferenceBench RewardBench GPQA ARC-Challenge TinyMMLU Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Base (Direct) 59.6 25.2 22.2 35.0 34.8 6.1 60.8 22.6 22.1 60.5 31.5 26.2 27.3 40.0 27.2 73.0 82.5 78.9 54.2 68.9 57.0 Base (CoT) 68.7 68.2 58.0 39.0 48.5 17.1 70.3 61.3 58.1 67.5 76.2 54.8 24.5 53.9 26.6 84.6 84.2 81.1 62.4 74.3 64.0 PA-GRPO (Direct) 75.1 85.5 69.3 58.0 54.5 31.8 84.8 82.3 75.9 69.5 80.1 59.8 31.0 54.1 30.8 86.5 89.9 88.7 66.5 77.9 72.0 PA-GRPO (CoT) 77.6 88.0 71.7 57.1 58.3 32.4 86.2 87.2 79.8 71.0 82.7 62.3 32.1 57.2 33.2 89.3 92.9 92.6 67.9 81.9 75.0 Table 4: Performance comparison of Direct vs. Chain-of-Thought (CoT) inference strategies. We report Accuracy, Consistency, and Consistent Accuracy for both the Base model and PA-GRPO across seven benchmarks. The best results are bolded. LLM-as-a-Judge Multiple Choice Questions Bias Setting MT-Bench JudgeBench PreferenceBench RewardBench GPQA ARC-Challenge TinyMMLU Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Acc Con CA Label Bias (Label-only) 76.7 92.2 72.8 56.3 79.0 45.8 74.9 62.8 56.2 70.7 87.5 64.4 22.0 67.2 20.8 80.7 95.6 81.3 63.5 83.8 66.0 Position Bias (Order-only) 77.4 85.1 69.9 55.0 45.5 27.7 74.2 66.9 57.7 71.6 82.0 62.6 27.9 65.0 27.7 80.9 91.9 82.8 64.3 80.5 72.0 Selection Bias (Standard) 77.6 88.0 71.7 57.1 58.3 32.4 86.2 87.2 79.8 71.0 82.7 62.3 29.7 62.3 31.0 81.6 93.2 83.1 69.3 82.8 73.0 Table 5: Bias decomposition via controlled protocols. We report Accuracy, Consistency, and Consistent Accuracy under Label Bias, which rotates option-ID symbols while keeping option content and display order fixed, Position Bias, which permutes the display order while keeping symbols fixed, and Selection Bias, which applies the standard coupled permutation of symbols and content. The results reflect residual sensitivity of PA-GRPO to each factor. Appendix F Training Hyperparameters and Configurations We implemented PA-GRPO based on the VeRL framework. Our experiments utilized three backbone models: Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-32B. These models were evaluated across both MCQ and Judge tasks. To ensure computational efficiency, we leveraged FlashAttention Dao et al. (2022) for accelerated attention computation and employed Low-Rank Adaptation (LoRA) on all linear layers with rank r=32r=32 and α=64α=64. For Qwen3-8B and Qwen3-32B, we used the non-thinking mode to ensure a consistent decoding setting. The models were optimized using AdamW Loshchilov and Hutter (2017) with a learning rate of 1e-5 for 2 epochs. We set the KL regularization Jaques et al. (2019) coefficient β=0.001β=0.001 and the entropy coefficient to 0.001 to stabilize training. Rollout generation was accelerated using vLLM Kwon et al. (2023) with a sampling size of N=8N=8 per prompt variant. Regarding batch configurations, we set the global batch size to 40 for MCQ tasks and 32 for Judge tasks to accommodate varying memory constraints. To ensure statistical reliability, all reported results are averaged over multiple independent runs. Appendix G Detailed Chain-of-Thought Analysis We investigate the role of reasoning in mitigating selection bias by comparing four settings: Base (Direct), Base (CoT), PA-GRPO (Direct), and PA-GRPO (CoT). Table 4 summarizes the results, revealing that PA-GRPO enhances robustness through two complementary mechanisms. Internalized alignment establishes intrinsic robustness. First, PA-GRPO (Direct) consistently outperforms the Base model, demonstrating that the training effectively internalizes permutation invariance into the model’s direct policy. For example, on MT-Bench, consistent accuracy improves from 58.0% with Base (CoT) to 69.3% with PA-GRPO (Direct). This indicates that the model parameters have been fundamentally optimized to resist bias, allowing for robust decision-making even without the computational cost of explicit reasoning. Explicit reasoning unlocks further potential for debiasing. Second, leveraging CoT acts as a powerful amplifier for robustness. PA-GRPO (CoT) achieves the strongest overall performance, particularly on complex tasks like GPQA, where consistent accuracy further rises to 33.2%. This confirms that explicit reasoning chains allow the model to deliberate on content semantics and override subtle positional priors that the direct policy might miss. Together, these results show that PA-GRPO benefits from both parameter-level alignment and inference-time reasoning. Appendix H Detailed Bias Decomposition Analysis This section provides full results for the bias decomposition analysis discussed in Section 5.5. Experimental protocols. To disentangle bias sources, we evaluate three controlled protocols: Label Bias: Rotates option-ID symbols while keeping option content and display order fixed. Position Bias: Permutes the display order while keeping option-ID symbols fixed. Selection Bias: Applies the coupled permutation used in standard evaluation, where option symbols and display order change together. Table 5 reports Accuracy, Consistency, and Consistent Accuracy under these protocols. Residual position sensitivity is prominent on LLM-as-a-Judge benchmarks. Changing the display order can lead to larger consistency degradation than changing label symbols alone. This effect is most evident on JudgeBench, where consistency is 79.0% under label-only changes but drops to 45.5% under order-only changes. On MCQ tasks, label-symbol effects are generally weaker, for example ARC-Challenge reaches 95.6% consistency under label-only changes. However, the relative strength of factors varies across datasets, indicating that residual bias is dataset-dependent rather than dominated by a single factor in all cases. Coupled invariance aligns with the training objective. On PreferenceBench, the coupled protocol yields higher consistency at 87.2% than label-only at 62.8% or order-only at 66.9%. This pattern suggests that PA-GRPO primarily learns invariance to the coupled permutation that matches standard evaluation and the training-time consistency objective, while factor-isolated protocols probe sensitivities that are less directly optimized. As a result, the model is most robust under natural selection bias even though residual label-only or order-only effects may remain.