Paper deep dive
BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
Antonio Ferrara, Alberto Rumi, Francesco Bonchi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator's bias up to a shared constant. The remaining constant is what fixes the aligned model's attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability 0.96 and BA-DPO removes 81 to 95\% of that shift; on MultiPref with real annotators it removes about half of DPO's lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality.
Tags
Links
- Source: https://arxiv.org/abs/2609.35044v1
- Canonical: https://arxiv.org/abs/2609.35044v1
Trouble viewing inline? Open PDF directly →
Full Text
136,751 characters extracted from source content.
Expand or collapse full text
BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment Antonio Ferrara †thanks: Equal contribution. Affiliation: Intesa Sanpaolo AI Research Alberto Rumi11footnotemark: 1 Affiliation: Intesa Sanpaolo AI Research Francesco Bonchi Affiliation: Intesa Sanpaolo AI Research Abstract Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator’s bias up to a shared constant. The remaining constant is what fixes the aligned model’s attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability 0.960.96 and BA-DPO removes 8181 to 95%95\% of that shift; on MultiPref with real annotators it removes about half of DPO’s lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality. 1 Introduction Preference-based alignment has become the dominant paradigm for shaping language model behavior from pairwise preference labels produced by human annotators. Standard techniques, whether they rely on reinforcement learning with a reward model (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022) or optimize a cross-entropy loss over preference pairs directly, as in Direct Preference Optimization (DPO) (Rafailov et al., 2023), inherit the foundational assumption of the Bradley-Terry model (BT) (Bradley and Terry, 1952): that every human-annotated pairwise comparison is a noisy but unbiased measurement of the quality gap between two responses. This assumption rarely holds. Human evaluators carry systematic preferences for attributes unrelated to quality, such as a name that signals a gender or an ethnicity (Bertrand and Mullainathan, 2004), a persona, a language variety, a formatting convention, or response length (Park et al., 2024; Lu et al., 2024; Liu et al., 2024). Because standard preference optimization treats every label as evidence of quality, systematic biases shared across annotators do not average out; instead, they are absorbed and amplified by the aligned policy as a preference for the attribute itself. In practice, preference optimization cannot discern why a label was assigned. For instance, if annotators consistently favor answers signed with names signaling a specific demographic group, the objective simply treats the signature as a sign of response quality. Under standard DPO the effect is large: a policy that initially signs across demographics with equal probability shifts to placing probability 0.960.96 on the favored marker. The same happens with real annotators who favor longer answers or Markdown styling (Section 5). In this paper, we address the following alignment problem: given preference judgments from annotators and a declared attribute of the responses, how can we train a policy that does not inherit the annotators’ shared bias toward that attribute? We introduce Bias-Adjusted DPO (BA-DPO), which generalizes DPO to possibly biased annotators, where each annotator carries a parameter for how much they favour the declared attribute. The parameter is added to the reward and does not enter the partition function, so the reparameterization that lets DPO train without a reward model goes through unchanged (Proposition 2). We then characterise what the labels determine. The votes fix every annotator’s bias relative to the others. What they leave open is one direction: raising all the biases together while lowering how much the policy itself favours the attribute changes no judgment (Theorem 3, Corollary 4). That direction is what sets the attribute rate of the trained policy. Training from the reference leaves the rate the reference had, and stating a target rate moves it instead. BA-DPO fundamentally differs from existing alignment corrections in its underlying formulation. Single-attribute heuristics like R-DPO (Park et al., 2024) and SamPO (Lu et al., 2024) modify or regularize length distributions; the attribute is fixed in the objective, and as our experiments show, correcting it leaves another one in place or substitutes one spurious shortcut for another (Lamparth et al., 2026). Heterogeneity techniques like Group-DRO (Sagawa et al., 2020) and EM-DPO (Chidambaram et al., 2026) represent or reweight evaluator disagreement, which addresses how annotators differ rather than what they hold in common. Methods like ADPO (Wang, 2025) remove unindexed response-level offsets, while BiasDPO (Allam, 2024) addresses the converse problem of cleansing biased model outputs using clean preference data. By contrast, BA-DPO models label bias directly, as additive per-annotator scalars on declared attributes, which separates the bias the annotators share from the quality signal and removes it rather than accommodating it. A more detailed discussion of prior related literature is provided in Appendix A. The primary contributions of this paper are summarized as follows: 1. Problem formulation and method: We formalize the challenge of removing shared, attribute-directed annotator bias from preference optimization and propose BA-DPO, an efficient objective requiring one parameter per attribute when annotators are pooled and one per annotator and attribute otherwise, with no reward model and no dedicated network (Section 3). 2. Theoretical characterization: We prove that the objective is convex in the bias parameters and that the votes fix every annotator’s bias relative to the others, leaving one direction that sets the trained policy’s attribute rate (Section 4). 3. Empirical validation: On a corpus with planted biases and on MultiPref (Miranda et al., 2025) with real annotators, at 0.5B with full fine-tuning and at 8B with LoRA, we show that BA-DPO removes 8181 to 95%95\% of DPO’s attribute shift on the planted corpus and about half of its lengthening on MultiPref, at no higher KL than DPO and no loss in judged quality (Section 5). 2 Background Given a prompt x and two responses y1,y2y_1,y_2, the Bradley-Terry (BT) model expresses the probability that y1y_1 is preferred over y2y_2 (denoted y1≻y2y_1 y_2) in terms of the difference in intrinsic quality between y1y_1 and y2y_2: p(y1≻y2∣x)=σ(r⋆(x,y1)−r⋆(x,y2))p(y_1 y_2 x)=σ (r (x,y_1)-r (x,y_2) ) (1) where r⋆r is the latent reward function and σ is the logistic sigmoid. Reinforcement Learning with Human Feedback (RLHF) fits a reward model to pairwise comparisons and subsequently maximizes expected reward under a KL-divergence penalty of strength β>0β>0 relative to a reference policy πref _ref: maxπθx∼,y∼πθ(⋅∣x)[r(x,y)]−βKL[πθ(y∣x)∥πref(y∣x)]. _ _θE_x ,y _θ(· x) [r(x,y) ]- _KL [ _θ(y x)\,\|\, _ref(y x) ]. (2) Rafailov et al. (2023) observe that this constrained optimization problem has an exact closed-form solution πθ(y∣x)=1Z(x)πref(y∣x)exp(1βr(x,y)) _θ(y x)= 1Z(x) _ref(y x) ( 1βr(x,y) ) (3) where Z(x)=∑yπref(y∣x)exp(1βr(x,y))Z(x)= _y _ref(y x) ( 1βr(x,y) ) is the partition function. Rearranging terms yields the implicit reward: r(x,y)=βlogπθ(y∣x)πref(y∣x)+βlogZ(x).r(x,y)=β _θ(y x) _ref(y x)+β Z(x). (4) When substituted back into the BT preference model, the partition function Z(x)Z(x) cancels out in the reward difference. Maximum likelihood estimation over preference pairs thus simplifies to a direct loss over the policy parameters, bypassing explicit reward modeling and reinforcement learning: ℒDPO(πθ,πref)=−(x,yw,yl)∼[logσ(βlogπθ(yw∣x)πref(yw∣x)−βlogπθ(yl∣x)πref(yl∣x))].L_DPO( _θ; _ref)=-E_(x,y_w,y_l) [ σ (β _θ(y_w x) _ref(y_w x)-β _θ(y_l x) _ref(y_l x) ) ]. (5) The BARP model (Ferrara et al., 2024) extends Bradley-Terry to account for evaluator bias in pairwise ranking. Item i has a true quality score sis_i and a group attribute indicator δgi∈0,1 _g_i∈\0,1\. Evaluator k carries a scalar bias θk _k and perceives the score as si+θkδgis_i+ _k _g_i, so that: pk(i≻j)=σ(si+θkδgi−sj−θkδgj).p_k(i j)=σ (s_i+ _k _g_i-s_j- _k _g_j ). (6) Scores and bias parameters are estimated jointly via maximum likelihood. The bias terms vanish for same-group comparisons (δgi=δgj _g_i= _g_j), which isolate true item quality, whereas cross-group comparisons identify evaluator bias. We adopt this formulation as our foundation for developing BA-DPO. 3 Bias-Adjusted DPO Given a prompt x, an item (a response to the prompt by the language model) y has a latent reward r⋆(x,y)r (x,y) and a group indicator δg(x,y)∈0,1 _g(x,y)∈\0,1\ recording whether it exhibits the attribute whose influence we wish to remove. Annotator k∈1,…,mk∈\1,…,m\ carries a bias parameter θk∈ℝ _k . Under the BARP model (Ferrara et al., 2024), annotator k prefers y1y_1 to y2y_2 with probability pk(y1≻y2∣x)=σ(r⋆(x,y1)+θkδg(x,y1)−r⋆(x,y2)−θkδg(x,y2)),p_k(y_1 y_2 x)=σ\! (r (x,y_1)+ _k\, _g(x,y_1)-r (x,y_2)- _k\, _g(x,y_2) ), (7) so that the annotator acts on a perceived reward r~k(x,y)=r⋆(x,y)+θkδg(x,y) r_k(x,y)=r (x,y)+ _k\, _g(x,y) rather than on r⋆r itself. Substituting the DPO reparameterization (4) into (7), the two copies of βlogZ(x)β Z(x) cancel exactly as they do in standard DPO, because the bias terms are additive to the reward and independent of the partition function. Given a dataset =(k(i),x(i),yw(i),yl(i))i=1ND=\(k^(i),x^(i),y_w^(i),y_l^(i))\_i=1^N in which each comparison is attributed to its annotator; by applying maximum likelihood over D we obtain the BA-DPO loss: ℒBA-DPO(πθ,θk,πref)=−(k,x,yw,yl)∼[logσ(u+bk)],L_BA-DPO( _θ,\ _k\; _ref)=-E_(k,x,y_w,y_l) [ σ (u+b_k ) ], (8) where we write u≔βlogπθ(yw∣x)πref(yw∣x)−βlogπθ(yl∣x)πref(yl∣x),bk≔θk(δg(x,yw)−δg(x,yl)).u β _θ(y_w x) _ref(y_w x)-β _θ(y_l x) _ref(y_l x), b_k _k ( _g(x,y_w)- _g(x,y_l) ). (9) We write Δδ≔δg(x,yw)−δg(x,yl)∈−1,0,1 δ _g(x,y_w)- _g(x,y_l)∈\-1,0,1\ for the group difference of a comparison, so that bk=θkΔδb_k= _k\, δ. The loss is optimised jointly over the policy parameters and the m scalar bias parameters. Remark 1 (BA-DPO reduces to DPO for unbiased annotators). Setting θk=0 _k=0 for every annotator in (8) recovers the DPO loss (5) exactly. DPO is therefore the special case of BA-DPO in which the annotators are assumed unbiased, which is the assumption of the Bradley-Terry model. Gradient analysis. The gradient of (8) with respect to the policy parameters is the DPO gradient with the importance weight σ(−u)σ(-u) replaced by σ(−u−bk)σ(-u-b_k), ∇ϕℒBA-DPO=−β(k,x,yw,yl)∼[σ(−u−bk)⋅(∇ϕlogπ(yw∣x)−∇ϕlogπ(yl∣x))]. _φL_BA-DPO=-β\,E_(k,x,y_w,y_l) [σ(-u-b_k)· ( _φ π(y_w x)- _φ π(y_l x) ) ]. (10) When the annotator’s bias favours the winner (bk>0b_k>0) the weight shrinks and the comparison teaches less, since part of its outcome is explained by bias; when the winner was preferred against the bias (bk<0b_k<0) the weight grows; when the two responses share a group, bk=0b_k=0 and the update is DPO’s. The gradient with respect to a bias parameter is the scalar ∂ℒBA-DPO∂θk=−nkN(x,yw,yl)∼k[σ(−u−bk)⋅(δg(x,yw)−δg(x,yl))], _BA-DPO∂ _k=- n_kN\,E_(x,y_w,y_l) _k [σ(-u-b_k)· ( _g(x,y_w)- _g(x,y_l) ) ], (11) where kD_k are the comparisons provided by annotator k and nk/Nn_k/N their share of the data. The bias channel adds m scalar parameters, a negligible cost next to the policy; the factor nk/Nn_k/N is decisive for the optimisation dynamics (see Proposition 6 in Appendix B.5). Per-annotator, pooled, and shared-mean. The model above requires annotator identities. When they are unavailable the bias channel collapses to a single population-level scalar θ∈ℝθ shared by every comparison (the pooled variant). When they are available there are two ways to use them: the naive per-annotator variant with free parameters θk\ _k\, and the shared-mean variant θk=θ¯+εk _k= θ+ _k in which the mean is a single parameter updated on every cross-group comparison and the deviations εk _k carry the individual structure. When the annotators belong to known classes, a class offset can be inserted between the two, θk=θ¯+δc+εk _k= θ+ _c+ _k. All variants span the same model class where they overlap; they differ in optimisation dynamics, and Section 5 and Appendix F show that this difference, not the identity information itself, decides how much bias is removed. Multiple attributes. For q binary or continuous attributes with group vectors g→y∈ℝq g_y ^q, the bias term generalises to bk=θ→k⊤(g→yw−g→yl)b_k= θ_k ( g_y_w- g_y_l) with θ→k∈ℝq θ_k ^q per annotator. The Signed-UltraFeedback corpus of Section 5 uses q=2q=2 binary attributes, and Appendix F.1 shows that declaring only one of the two leaves the other in the policy and removes only part of the declared one. 4 Theoretical Characterization This section presents the theoretical characterization of BA-DPO (proofs in Appendix B). The first result states precisely the substitution already presented in Section 3. Proposition 2 (Compatibility with the DPO reparameterization). Let r(x,y)r(x,y) be any reward function for which Z(x)=∑yπref(y∣x)exp(r(x,y)/β)Z(x)= _y _ref(y x) (r(x,y)/β) is finite for every prompt, and let θkk=1m\ _k\_k=1^m be arbitrary evaluator bias parameters. Then the bias-aware preference model (7) can be written equivalently as pk(y1≻y2∣x)=σ(βlogπ(y1∣x)πref(y1∣x)−βlogπ(y2∣x)πref(y2∣x)+θk(δg(x,y1)−δg(x,y2)))p_k(y_1 y_2 x)=σ\! (β π(y_1 x) _ref(y_1 x)-β π(y_2 x) _ref(y_2 x)+ _k ( _g(x,y_1)- _g(x,y_2) ) ) with π(y∣x)=πref(y∣x)exp(r(x,y)/β)/Z(x)π(y x)= _ref(y x) (r(x,y)/β)/Z(x). Proposition 2 is what keeps the construction at one policy: the bias term is added to the reward and does not involve the partition function Z(x)Z(x), so it passes through the reparameterization unchanged and stays outside the policy. Methods for annotator disagreement differ in how much freedom they give each annotator’s reward. DPO gives none: one reward is fit for everyone, so a bias the annotators share is represented nowhere except in the policy. Mixture methods give each of K annotator types its own reward, and since each reward has its own optimal policy, they train K policies (Chidambaram et al., 2026). BA-DPO lets annotators differ only in the coefficient θk _k on a declared direction δg _g, a fixed function of (x,y)(x,y) with no parameters of its own. A single policy then carries the reward the annotators share, the annotator-specific part enters the logit as the scalar θk _k, and modelling annotator bias costs m scalars, not K policies. We now characterise what the observed votes determine about the bias parameters. Part of what they leave undetermined is inherited from DPO, which cannot see a reward shift that is constant within a compared pair, because rewards enter the likelihood only through the difference between the two responses of a pair. The rest is what the bias channel adds, and the theorem holds for an otherwise arbitrary reward function. Consider the graph whose vertices are the annotators, with an edge between two annotators whenever they judged at least one cross-group comparison in common. We say that the annotator pool is connected if this graph is connected. Theorem 3 (Identifiability of the bias parameters). The loss ℒL of (8), viewed as a function of the reward values r(x,y)r(x,y) and of θ, is jointly convex in (r,θ)(r,θ), and all its minimisers assign the same probability to every judgment. If the annotator pool is connected and (r^,θ^)( r, θ) is a minimiser, the minimisers are exactly the pairs (r^+cδg+h,θ^−c),c∈ℝ, ( r+c\, _g+h,\ θ-c1 ), c , with h any function constant on the two responses of every compared pair. Every difference θk−θk′ _k- _k therefore takes the same value at every minimiser, while their mean θ¯≔1m∑kθk θ 1m _k _k is not determined by the judgments. Connectedness requires that every annotator judged at least one cross-group comparison, since the votes of an annotator who never saw the attribute on both sides say nothing about their θk _k. It holds whenever annotators are assigned to comparisons at random, as in every corpus here; if it fails, the theorem applies within each connected group of annotators, each with its own offset. For a fixed policy, a minimiser in θ exists unless some annotator chose the attribute side in every one of their cross-group comparisons, in which case their θk _k diverges. Theorem 3 is stated in terms of the individual θk _k; it can be restated by decomposing the annotators’ biases into a shared mean and deviations. Corollary 4 (Reward-bias degeneracy). Write θk=θ¯+εk _k= θ+ _k with ∑kεk=0 _k _k=0. Shifting the mean by −c-c and adding cδgc\, _g to the reward leaves every judgment probability unchanged, so θ¯ θ is not determined by the votes. If the annotator pool is connected, the deviations εk _k take the same value at every minimiser. The degeneracy is intrinsic rather than an artefact of the parameterisation: the likelihood cannot distinguish between the attribute genuinely carrying reward and every annotator being uniformly biased toward it. The free constant c is therefore a choice, and what it decides is how much the trained policy favours the attribute: by Corollary 4, moving c from θ¯ θ into the reward multiplies the policy’s odds for a response over its counterfactual without the attribute by ec/βe^c/β and changes no judgment probability. Two anchors are natural. Anchoring to the reference takes c such that the policy gives a response and its counterfactual the same odds as the reference does, so the shared level of the labels sits in θ¯ θ and the policy keeps the reference’s attribute rate; every experiment in the paper starts from πref _ref with θ=0θ=0 and ends there (Appendix C.3). Anchoring to a target takes c such that the outputs meet a stated criterion, for instance statistical parity, an attribute rate of one half: the required c is found by a one-dimensional search on the rate and applied as a fixed offset (Appendix C.4; Appendix C.5 uses it to bring a biased DPO policy to parity). Which anchor is right is a normative decision, like the attribute partition itself; the labels do not make it. Proposition 5 (Representability of the bias components). Decompose θk=θ¯+εk _k= θ+ _k with ∑kεk=0 _k _k=0. The shared component θ¯δg(x,y) θ\, _g(x,y) is expressible as a function of (x,y)(x,y) and can therefore be represented inside the policy’s implicit reward, whereas the deviation term εkδg(x,y) _k\, _g(x,y) depends on k, which the policy never observes, and is representable by no policy π(y∣x)π(y x) unless every εk=0 _k=0, that is, unless all annotators have the same bias. Proposition 5 says what the policy could express, not what training makes it express. Had the policy taken up the shared component, the bias parameters would be left with the deviations alone, and the naive per-annotator and shared-mean variants would then behave alike, since they write the deviations the same way. In our experiments the shared component goes to the bias parameter instead: a mean started at zero and a mean started at the offline estimate converge to the same interior value (Appendix F.2). The two variants therefore differ only in how they write the same m parameters, as m free θk _k or as a mean plus deviations, and that decides how fast the mean moves. The shared parameter is updated by every cross-group judgment, while each free θk _k is updated only by the judgments of its own annotator. Under gradient descent the naive mean is therefore exactly m times slower (Proposition 6). Together, these results explain the pattern of the experiments. The deviations are the only component the votes determine on their own (Theorem 3) and the only one the policy cannot express (Proposition 5). Debiasing the policy, however, requires the mean, which is the component the naive per-annotator variant learns most slowly and the shared-mean variant learns first. 5 Experiments In this section, we test whether DPO absorbs a bias shared by the annotators and whether BA-DPO removes it, on two corpora: Signed-UltraFeedback (built from UltraFeedback (Cui et al., 2024)), where the annotators’ biases are planted and the amount to remove is therefore known, and MultiPref (Miranda et al., 2025), with real annotators, under length and under formatting. DPO takes up the bias in every setting and model tested, and BA-DPO removes most of it without moving further from the reference than DPO does. 5.1 Experimental settings Datasets. The method needs an attribute that can be read from each response and, for the per-annotator variants, the identity of the annotator behind each judgment. We evaluate four attributes on two corpora (Appendix D has the full details). Length and formatting can be read from any preference corpus. We use MultiPref (Miranda et al., 2025), one of the few public corpora with annotator identities: after removing ties, 30,84730,847 disaggregated judgments on 10,46110,461 comparisons, each judged by four of 227227 annotators. A response is long when it has at least 1.51.5 times the words of its partner and formatted when it contains Markdown. We know of no public preference corpus with identified annotators in which the responses carry a gender- or race-coded marker. We therefore introduce Signed-UltraFeedback, built from the prompts and response pairs of UltraFeedback (Cui et al., 2024) by ending each response with a signature whose first name comes from published audit lists (Bertrand and Mullainathan, 2004; Caliskan et al., 2017) and is both gender-coded (woman or man) and race-coded (white or black), so that every response carries two binary attributes at once. Sixty annotators in three classes of 2020 vote on each pair, four per pair, through the biased Bradley-Terry likelihood (7), each with a bias vector θk∼(μc,0.82I) _k ( _c,0.8^2I). The class means are planted so that both attributes have a population mean near 1.01.0 in log-odds, the classes agree on the gender-coded attribute, and on the race-coded one a class of 2020 leans the other way. The θk _k are used only to generate the labels and to score the outcome, never by a training method. The reference policy signs with each name group equally often, so any bias in a trained policy comes from the preference stage. We say “gender-coded” and “race-coded” because the corpus tracks how a text marker propagates through alignment, not human prejudice. Models and methods. We run every experiment on two models, Qwen2.5-0.5B-Instruct (Yang et al., 2024) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024), which differ in scale and in family. The 0.5B model is fully fine-tuned; the 8B model is trained with LoRA (Hu et al., 2022) on all linear layers. Each model and corpus has its own SFT reference, fine-tuned on the majority-chosen response of each comparison and shared by every method and seed, so that implicit rewards are comparable (Appendix E). A third model, Mistral-7B-Instruct-v0.3 (Jiang et al., 2023), of a different family and with a different tokenizer, repeats the comparison between DPO and BA-DPO as a robustness check (Appendix F.5). We compare the reference, DPO, and the two BA-DPO variants of Section 3: pooled, one scalar per attribute shared by every annotator, and θ¯+εk θ+ _k, a shared mean per attribute plus one deviation per annotator. The remaining variants of the bias model are controls for the mechanism, reported at 0.5B in Appendix F. r3 Baselines. To the best of our knowledge, no method for preference optimisation takes the attribute as an input that the user supplies and can change between runs. We therefore compare with the two families that come closest: the corrections built for length bias, which fix that attribute in the objective, and the methods built for annotator disagreement. Each runs on the corpus of the bias it was built for and, for the length methods, also under formatting, by a rule fixed before any result was read (Appendix E.3). R-DPO (Park et al., 2024) and SamPO (Lu et al., 2024) target length bias: they are trained on MultiPref and re-scored under formatting without retraining. R-DPO uses α=0.005α=0.005, because the published 0.020.02 collapses the policy on this corpus (Appendix F.2). Group-DRO (Sagawa et al., 2020) and EM-DPO with MinMax-DPO (Chidambaram et al., 2026) target annotator disagreement. Both are our own implementations (Appendix E.3). Group-DRO runs on both corpora, with the annotator classes as groups on Signed-UltraFeedback and the annotators on MultiPref; EM-DPO runs on Signed-UltraFeedback and at 0.5B only, since its four rounds over three type policies cost five times the steps of any other method. Metrics. We evaluate each policy on 300300 held-out prompts, computing five metrics from the generations and held-out judgments (formal definitions in Appendix E.4). Each configuration is trained at three seeds, and we report the mean alongside a 95%95\% confidence interval; the reference policy is trained once and therefore has no interval. • Attribute rate: How often the generations carry the declared attribute. On Signed-UltraFeedback, we compute this exactly from the policy’s output probabilities rather than from sampling: the probability mass placed on woman-coded (p(woman)p(woman)) or black-coded (p(black)p(black)) names at the signature position. On MultiPref, this is the mean token count for length bias, and the markdown rate reweighted onto the reference’s length distribution for formatting bias, so that a policy cannot lower the measured rate by producing shorter answers alone. • Bias removed: The proportion of the gap in attribute rate between standard DPO and the reference policy closed by the method. This is computed per seed against the corresponding DPO seed, scaled such that 0%0\% is DPO and 100%100\% is the reference. • Held-out gap: The difference in the policy’s implicit reward accuracy between held-out judgment pairs that differ on the attribute, and pairs that do not. A biased policy predicts annotator labels better exactly when the attribute differs; a gap approaching zero, while same-group accuracy holds, shows that the bias has left the implicit reward. • RM and RM-: The judge’s score, from Skywork-Reward-V2-Qwen3-1.7B (Liu et al., 2026). RM- is the score after uniformly stripping the attribute from all generations (that is, removing signatures in Signed-UltraFeedback or markup in formatting). No length-invariant transform exists for length bias, so there we report the raw RM score and read it with caution. • KL: The per-token Kullback-Leibler divergence from the reference, estimated on the policy’s own generations. This distinguishes methods that remove the bias by changing the policy from those that stay close to the reference distribution. 5.2 Signed-UltraFeedback DPO absorbs the bias on both attributes and on both models (Table 1), moving from the near-balanced reference to 0.960.96 to 0.990.99. BA-DPO removes 8181 to 95%95\% of that shift, with both variants and on both models, at no cost in judge score and at about two thirds of DPO’s KL: the policy still differs from the reference, but not along the attribute. The two BA-DPO variants give the same policy to within about 0.010.01 everywhere, although pooled has one parameter per attribute and θ¯+εk θ+ _k has a mean and sixty deviations. Proposition 5 predicts this: a deviation depends on who voted, which the policy never sees, so only the shared mean can reach it. Appendix F.1 tests this prediction with the annotator identities shuffled and with the annotator classes given. The race-coded attribute is removed less than the gender-coded one at 0.5B, 8181 against 89%89\%, although both were planted at the same mean. A shared scalar is fitted to how the population votes, not to how it is biased, and here the classes disagree, so the votes partly cancel: on the pairs that differ only in the name the attribute-carrying side wins at log-odds 0.700.70 against a realised mean bias of 1.131.13, and the learned scalar settles at 0.620.62 to 0.680.68. The part of the bias the scalar does not absorb stays in the policy. The deviations do not change the policy, but they do estimate each annotator’s own bias. The learned θk=θ¯+εk _k= θ+ _k correlate with the planted θk _k at r=0.95r=0.95 on the gender-coded attribute and 0.980.98 on the race-coded one, on both models. When we use them to predict each annotator’s held-out votes, they are as accurate as the planted θk _k (both as 0.7520.752 at 0.5B). The pooled scalar gives every annotator the same bias, so for the class that leans against the race-coded attribute its predictions are no better than chance (Appendix G). Neither baseline removes the bias. Group-DRO removes 44 to 18%18\%: it reweights the groups it is given, and a bias that every group shares is not disagreement between groups. EM-DPO does not recover the classes, one type ending up with 5454 to 5858 of the 6060 annotators, and leaves the gender-coded attribute at DPO’s level; its 61%61\% on the race-coded one is averaging rather than removal, since part of the mixture sits on a type policy carrying the opposite bias. Table 1: Signed-UltraFeedback, two attributes and three annotator classes. Columns as defined in Section 5.1; mean and 95%95\% interval over three seeds. RM- is comparable within a model only. Method p(woman)p(woman) Removed p(black)p(black) Removed RM- KL Qwen2.5-0.5B-Instruct, full fine-tuning Reference (SFT) 0.498 — 0.517 — −3.30-3.30 — DPO 0.964±0.0050.964± 0.005 — 0.991±0.0010.991± 0.001 — −2.63±0.28-2.63± 0.28 0.038±0.0010.038± 0.001 BA-DPO (pooled) 0.549±0.0080.549± 0.008 89±2%89± 2\% 0.608±0.0170.608± 0.017 81±4%81± 4\% −2.66±0.29-2.66± 0.29 0.026±0.0010.026± 0.001 BA-DPO (θ¯+εk θ+ _k) 0.542±0.0110.542± 0.011 91±2%91± 2\% 0.595±0.0180.595± 0.018 83±4%83± 4\% −2.76±0.35-2.76± 0.35 0.026±0.0010.026± 0.001 Group-DRO DPO 0.927±0.0160.927± 0.016 8±3%8± 3\% 0.941±0.0180.941± 0.018 11±4%11± 4\% −2.62±0.08-2.62± 0.08 0.035±0.0010.035± 0.001 EM-DPO + MinMax-DPO 0.923±0.1090.923± 0.109 9±24%9± 24\% 0.702±0.1570.702± 0.157 61±33%61± 33\% −2.63±0.10-2.63± 0.10 0.041±0.0070.041± 0.007 Llama-3.1-8B-Instruct, LoRA Reference (SFT) 0.465 — 0.521 — 0.410.41 — DPO 0.967±0.0120.967± 0.012 — 0.986±0.0050.986± 0.005 — 1.43±0.171.43± 0.17 0.021±0.0020.021± 0.002 BA-DPO (pooled) 0.492±0.0020.492± 0.002 95±<1%95±<1\% 0.555±0.0020.555± 0.002 93±<1%93±<1\% 1.41±0.261.41± 0.26 0.013±0.0030.013± 0.003 BA-DPO (θ¯+εk θ+ _k) 0.491±0.0020.491± 0.002 95±<1%95±<1\% 0.552±0.0020.552± 0.002 93±<1%93±<1\% 1.49±0.121.49± 0.12 0.014±0.0020.014± 0.002 Group-DRO DPO 0.945±0.0180.945± 0.018 4±2%4± 2\% 0.904±0.0200.904± 0.020 18±3%18± 3\% 1.44±0.251.44± 0.25 0.020±0.0030.020± 0.003 5.3 MultiPref: length bias On MultiPref the annotators are real and length has no natural target, since a longer answer is sometimes the better one. The token count therefore says how far the policy moved, not whether it stopped in the right place. We read the held-out gap for that: it compares how well the policy’s implicit reward predicts these annotators’ labels on held-out pairs that differ on the declared attribute with how well it predicts them on pairs that do not. Table 2 reports the results. Under DPO the policy produces answers 9696 tokens longer than the reference at 0.5B and 6868 tokens longer at 8B, with held-out gaps of 0.130.13 and 0.090.09. BA-DPO removes 4545 to 51%51\% of that increase and reduces the gap to 0.020.02 and 0.000.00 at 0.5B and to 0.030.03 and 0.050.05 at 8B. Its learned scalar converges to 0.770.77 to 0.810.81 at 0.5B and to 0.690.69 to 0.710.71 at 8B, below the offline estimate of 0.990.99, which counts every win of the longer response as bias. Each baseline misses the bias in a different way. R-DPO removes more tokens than BA-DPO does, 7777 to 89%89\% of DPO’s increase, but its gap becomes negative, −0.04-0.04 at both scales: the policy now prefers the shorter response on the pairs that carry the attribute, which inverts the bias rather than removing it. Under SamPO the policy is longer than under DPO (−38%-38\% and −7%-7\%), and Group-DRO changes neither the length nor the gap, since a bias that every annotator shares is not disagreement between annotators. At 8B the judge ranks the methods in order of their length, from 4.94.9 for the reference to 6.76.7 for SamPO, so BA-DPO’s 6.16.1 against DPO’s 6.56.5 cannot be separated from the tokens removed. The held-out gap does not use the judge, which is why we report it here. Table 2: MultiPref with length declared. Columns as in Section 5.1; mean and 95%95\% interval over three seeds; RM is the raw judge score, which rises with answer length (Section 5.3). Method Tokens Removed Gap RM KL Qwen2.5-0.5B-Instruct, full fine-tuning Reference (SFT) 288.6 — — −1.34-1.34 — DPO 384.5±3.7384.5± 3.7 — 0.126±0.0220.126± 0.022 −0.66±0.16-0.66± 0.16 0.033±<0.0010.033±<0.001 BA-DPO (pooled) 341.3±8.7341.3± 8.7 45±10%45± 10\% 0.019±0.0450.019± 0.045 −0.66±0.20-0.66± 0.20 0.033±0.0010.033± 0.001 BA-DPO (θ¯+εk θ+ _k) 335.3±7.4335.3± 7.4 51±8%51± 8\% 0.001±0.0320.001± 0.032 −0.78±0.30-0.78± 0.30 0.032±0.0010.032± 0.001 R-DPO (α=0.005α=0.005) 310.8±10.3310.8± 10.3 77±10%77± 10\% −0.040±0.007-0.040± 0.007 −0.74±0.38-0.74± 0.38 0.035±0.0020.035± 0.002 SamPO 420.9±4.4420.9± 4.4 −38±8%-38± 8\% 0.088±0.0230.088± 0.023 −0.76±0.07-0.76± 0.07 0.044±0.0020.044± 0.002 Group-DRO DPO 389.9±4.3389.9± 4.3 −6±3%-6± 3\% 0.122±0.0320.122± 0.032 −0.53±0.24-0.53± 0.24 0.032±0.0020.032± 0.002 Llama-3.1-8B-Instruct, LoRA Reference (SFT) 309.7 — — 4.854.85 — DPO 378.2±4.7378.2± 4.7 — 0.095±0.0080.095± 0.008 6.52±0.116.52± 0.11 0.018±0.0010.018± 0.001 BA-DPO (pooled) 343.9±2.6343.9± 2.6 50±4%50± 4\% 0.034±0.0320.034± 0.032 6.10±0.026.10± 0.02 0.018±<0.0010.018±<0.001 BA-DPO (θ¯+εk θ+ _k) 346.9±6.5346.9± 6.5 46±6%46± 6\% 0.046±0.0130.046± 0.013 6.10±0.096.10± 0.09 0.018±<0.0010.018±<0.001 R-DPO (α=0.005α=0.005) 317.4±2.9317.4± 2.9 89±4%89± 4\% −0.038±0.004-0.038± 0.004 5.73±0.175.73± 0.17 0.020±0.0010.020± 0.001 SamPO 382.8±3.1382.8± 3.1 −7±9%-7± 9\% 0.061±0.0240.061± 0.024 6.70±0.546.70± 0.54 0.022±<0.0010.022±<0.001 Group-DRO DPO 381.4±12.1381.4± 12.1 −5±11%-5± 11\% 0.097±0.0160.097± 0.016 6.49±0.346.49± 0.34 0.018±0.0010.018± 0.001 5.4 MultiPref: formatting bias The data is the same as in Section 5.3, with formatting declared in place of length: the same annotators and the same judgments. The two attributes are favoured about equally in the labels, markdown winning at log-odds 0.980.98 and length at 0.990.99, but DPO amplifies only one of them. Under length it adds 9696 tokens, many times the spread across seeds, while under formatting it raises the markdown rate by 0.0350.035, less than the ±0.053± 0.053 interval on that rate. The rate is therefore a weak readout here, since DPO barely moves it, and we read the held-out gap instead. DPO opens a gap of 0.0660.066 (Table 3). BA-DPO closes it to 0.0040.004 and 0.0000.000 while leaving the markdown rate where the reference had it. At 8B DPO does not raise the rate beyond seed noise (0.567±0.0340.567± 0.034 against 0.5600.560), and still opens a gap of 0.0670.067, which both BA-DPO variants halve. Neither length method can be given another attribute, and correcting length does not correct formatting: R-DPO turns the gap negative again (−0.039-0.039) and SamPO leaves it near DPO’s level (0.0530.053). At its published strength, α=0.02α=0.02, R-DPO raises the markdown rate to 0.700.70, above DPO, in answers collapsed to 138138 tokens (Appendix F.2): the pressure DPO had put on length has moved to markdown. On two attributes of one corpus this is the substitution pattern of Lamparth et al. (2026), and it is what declaring the attribute avoids. Appendix F.1 shows the same pattern inside Signed-UltraFeedback when only one of its two biased attributes is declared. Table 3: MultiPref with formatting declared, on the judgments and annotators of Table 2. Columns as in Table 2. No removal share is given because DPO moves the markdown rate by less than its seed interval (Section 5.4). The length baselines are their Table 2 checkpoints scored under formatting without retraining. Method Rate Gap RM- KL Qwen2.5-0.5B-Instruct, full fine-tuning Reference (SFT) 0.583 — −1.51-1.51 — DPO 0.618±0.0530.618± 0.053 0.066±0.0250.066± 0.025 −0.98±0.14-0.98± 0.14 0.033±<0.0010.033±<0.001 BA-DPO (pooled) 0.592±0.0270.592± 0.027 0.004±0.0130.004± 0.013 −0.89±0.12-0.89± 0.12 0.032±0.0020.032± 0.002 BA-DPO (θ¯+εk θ+ _k) 0.581±0.0340.581± 0.034 0.000±0.0510.000± 0.051 −0.86±0.19-0.86± 0.19 0.032±0.0020.032± 0.002 R-DPO (α=0.005α=0.005, re-scored) 0.634±0.0030.634± 0.003 −0.039±0.039-0.039± 0.039 −0.98±0.34-0.98± 0.34 0.035±0.0020.035± 0.002 SamPO (re-scored) 0.636±0.0490.636± 0.049 0.053±0.0240.053± 0.024 −1.16±0.15-1.16± 0.15 0.044±0.0020.044± 0.002 Llama-3.1-8B-Instruct, LoRA Reference (SFT) 0.560 — 4.674.67 — DPO 0.567±0.0340.567± 0.034 0.067±0.0160.067± 0.016 6.18±0.126.18± 0.12 0.018±0.0010.018± 0.001 BA-DPO (pooled) 0.536±0.0180.536± 0.018 0.033±0.0330.033± 0.033 6.05±0.036.05± 0.03 0.018±0.0010.018± 0.001 BA-DPO (θ¯+εk θ+ _k) 0.539±0.0220.539± 0.022 0.033±0.0460.033± 0.046 6.08±0.096.08± 0.09 0.018±<0.0010.018±<0.001 R-DPO (α=0.005α=0.005, re-scored) 0.568±0.0410.568± 0.041 −0.031±0.029-0.031± 0.029 5.50±0.155.50± 0.15 0.020±0.0010.020± 0.001 SamPO (re-scored) 0.569±0.0130.569± 0.013 0.040±0.0070.040± 0.007 6.34±0.546.34± 0.54 0.022±<0.0010.022±<0.001 6 Conclusions We addressed the vulnerability of preference optimization to a bias that annotators share by introducing Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator for each declared attribute. We proved that the objective is convex in these parameters and that the votes fix every annotator’s bias relative to the others. How much of the shared bias counts as quality is left to one free choice, which sets the aligned policy’s attribute rate. On three models, BA-DPO removed 8181 to 95%95\% of the shift DPO absorbs on planted biases and about half of the length DPO adds on MultiPref, with no higher KL to the reference than DPO and no loss in judged quality wherever the judge can be read independently of the attribute. Limitations and future work. BA-DPO removes only the bias it is told about: the attribute must be declared and computable from the response, and an attribute that is not declared stays in the policy (Appendix F.1). Because the labels do not fix the shared level of the bias, the default anchor keeps the reference’s attribute rate, including any bias the reference already has; a target rate removes it, but the target must be chosen. That level could instead be estimated from a small set of judgments by annotators known to be unbiased. Our evidence comes from one reward-model judge, one corpus with real annotators whose two attributes are both surface features, and LoRA at 7B and 8B; a corpus with annotator identities and a social attribute, such as a persona or a language variety, is the natural next test. Since the bias term is additive to the reward, it also applies to reward-model training and to other pairwise objectives such as IPO (Azar et al., 2024), and the per-annotator estimates could flag biased annotators while a pool is still being labelled. AI use statement In this work, we used generative AI tools (Claude Code with Fable 5.1 and Opus 5) as a coding assistant and to polish the writing of the manuscript. We have reviewed all AI-assisted code and text, and we take full responsibility for the final content of this work. Ethics statement This work aims to reduce the influence of systematic annotator bias on aligned language models. Two cautions apply. First, the partition of response attributes into bias to be removed and quality to be preserved is a declared input, since preference data determine the biases only up to a common offset; the choice is normative and should involve the stakeholders affected. Second, per-annotator modelling requires annotator identifiers and produces per-person bias estimates; we recommend treating both as personal data, and we use only datasets whose identifiers were pseudonymised by their publishers. Reproducibility statement All models (Qwen2.5-0.5B-Instruct, Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3, with Skywork-Reward-V2-Qwen3-1.7B as the judge) and datasets (ultrafeedback_binarized and allenai/multipref) are public. Every reported number derives from a logged run keyed by (method, parameters, seed); every cell of the main tables is a mean over seeds 42, 123 and 456 with a 95% interval, and single-seed appendix rows are labelled as controls. Complete data-generation procedures and hyperparameters are in Appendices D.1 to E. All runs used a single NVIDIA A100 80GB GPU; the peak memory and duration of each kind of run are given in Appendix E. The source code, with the configuration of every experiment and the commands that reproduce each table, is available at https://github.com/Ambress92/BA-DPO. References Allam (2024) A. Allam BiasDPO: mitigating bias in language models through direct preference optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), p. 42–50. Cited by: Appendix A, §1. Azar et al. (2024) M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, Vol. 238, p. 4447–4455. Cited by: §6. Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §1. Bertrand and Mullainathan (2004) M. Bertrand and S. Mullainathan Are Emily and Greg more employable than Lakisha and Jamal? a field experiment on labor market discrimination. American Economic Review 94 (4), p. 991–1013. Cited by: §D.2, §1, §5.1. Bradley and Terry (1952) R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §1. Caliskan et al. (2017) A. Caliskan, J. J. Bryson, and A. Narayanan Semantics derived automatically from language corpora contain human-like biases. Science 356 (6334), p. 183–186. Cited by: §D.2, §5.1. Chen et al. (2025) H. Chen, H. Zhao, H. Lam, D. Yao, and W. Tang MallowsPO: fine-tune your LLM with preference dispersions. In International Conference on Learning Representations (ICLR), Cited by: Appendix A. Chen et al. (2024) L. Chen, C. Zhu, J. Chen, D. Soselia, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro ODIN: disentangled reward mitigates hacking in RLHF. In Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 235, p. 7935–7952. Cited by: Appendix A. Chhan et al. (2024) D. Chhan, E. Novoseller, and V. J. Lawhern Crowd-PrefRL: preference-based reward learning from crowds. arXiv preprint arXiv:2401.10941. Cited by: Appendix A, §E.3, §F.2. Chidambaram et al. (2026) K. Chidambaram, K. V. Seetharaman, and V. Syrgkanis Direct preference optimization with unobserved preference heterogeneity: the necessity of ternary preferences. In International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, Vol. 300, p. 4555–4563. Cited by: Appendix A, §E.3, §1, §4, §5.1. Christiano et al. (2017) P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. In NeurIPS, Cited by: §1. Clark et al. (2019) C. Clark, M. Yatskar, and L. Zettlemoyer Don’t take the easy way out: ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 4069–4082. External Links: Document Cited by: Appendix A. Cui et al. (2024) G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun UltraFeedback: boosting language models with scaled AI feedback. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 235, p. 9722–9744. Cited by: §D.2, §5.1, §5. Ferrara et al. (2024) A. Ferrara, F. Bonchi, F. Fabbri, F. Karimi, and C. Wagner Bias-aware ranking from pairwise comparisons. Data Mining and Knowledge Discovery 38, p. 2062–2086. Cited by: §2, §3. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1. He et al. (2019) H. He, S. Zha, and H. Wang Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo), p. 132–142. External Links: Document Cited by: Appendix A. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §5.1. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: §F.5, §5.1. Karimi Mahabadi et al. (2020) R. Karimi Mahabadi, Y. Belinkov, and J. Henderson End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 8706–8716. External Links: Document Cited by: Appendix A. Lamparth et al. (2026) M. Lamparth, D. Fein, A. Haupt, M. Hussing, and M. J. Kochenderfer Reward bias substitution: single-axis bias mitigations redirect optimization pressure. arXiv preprint arXiv:2605.27996. Cited by: §1, §5.4. Liu et al. (2026) C. Y. Liu, L. Zeng, Y. Xiao, J. He, J. Liu, C. Wang, R. Yan, W. Shen, F. Zhang, J. Xu, Y. Liu, and Y. Zhou Skywork-Reward-V2: scaling preference data curation via human-AI synergy. In International Conference on Learning Representations (ICLR), Cited by: §E.4, 4th item. Liu et al. (2025) S. Liu, W. Fang, Z. Hu, J. Zhang, Y. Zhou, K. Zhang, R. Tu, T. Lin, F. Huang, M. Song, Y. Li, and D. Tao A survey of direct preference optimization. arXiv preprint arXiv:2503.11701. Cited by: Appendix A. Liu et al. (2024) W. Liu, Y. Bai, C. Han, R. Weng, J. Xu, X. Cao, J. Wang, and X. Cai Length desensitization in direct preference optimization. arXiv preprint arXiv:2409.06411. Cited by: Appendix A, §1. Lu et al. (2024) J. Lu, J. Li, S. An, M. Zhao, Y. He, D. Yin, and X. Sun Eliminating biased length reliance of direct preference optimization via down-sampled KL divergence. In EMNLP, Cited by: Appendix A, §E.3, §1, §1, §5.1. Miranda et al. (2025) L. J. V. Miranda, Y. Wang, Y. Elazar, S. Kumar, V. Pyatkin, F. Brahman, N. A. Smith, H. Hajishirzi, and P. Dasigi Hybrid preferences: learning to route instances for human vs. AI feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), p. 7162–7200. Cited by: Table 5, item 3, §5.1, §5. Ng et al. (2025) I. Ng, P. Blöbaum, S. Bhandari, K. Zhang, and S. Kasiviswanathan Debiasing reward models by representation learning with guarantees. arXiv preprint arXiv:2510.23751. Cited by: Appendix A. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In NeurIPS, Cited by: §1. Park et al. (2024) R. Park, R. Rafailov, S. Ermon, and C. Finn Disentangling length from quality in direct preference optimization. In Findings of ACL, Cited by: Appendix A, §E.3, §1, §1, §5.1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. In NeurIPS, Cited by: §B.1, §1, §2. Sagawa et al. (2020) S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang Distributionally robust neural networks for group shifts: on the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), Cited by: §E.3, §1, §5.1. Shen et al. (2023) W. Shen, R. Zheng, W. Zhan, J. Zhao, S. Dou, T. Gui, Q. Zhang, and X. Huang Loose lips sink ships: mitigating length bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 2859–2873. External Links: Document Cited by: Appendix A. Wang et al. (2025a) C. Wang, H. Shi, L. Tian, B. An, and S. Yan Removing prompt-template bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: ACL 2025, p. 24110–24122. External Links: Document Cited by: Appendix A. Wang et al. (2025b) C. Wang, Z. Zhao, Y. Jiang, Z. Chen, C. Zhu, Y. Chen, J. Liu, L. Zhang, X. Fan, H. Ma, and S. Wang Beyond reward hacking: causal rewards for large language model alignment. arXiv preprint arXiv:2501.09620. Cited by: Appendix A. Wang (2025) Z. Wang ADPO: anchored direct preference optimization. arXiv preprint arXiv:2510.18913. Cited by: Appendix A, §1. Xiao et al. (2025) J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su On the algorithmic bias of aligning large language models with RLHF: preference collapse and matching regularization. Journal of the American Statistical Association 120 (552), p. 2154–2164. External Links: Document Cited by: Appendix A. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §5.1. Zhang et al. (2025) X. Zhang, W. Xiong, L. Chen, T. Zhou, H. Huang, and T. Zhang From lists to emojis: how format bias affects model alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), p. 26940–26961. External Links: Document Cited by: Appendix A. Zhao et al. (2026) K. Zhao, J. Cai, J. Zhu, R. Sun, D. Xue, W. Zhou, L. Li, and H. Li Bias fitting to mitigate length bias of reward model in RLHF. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), p. 2912–2927. External Links: Document Cited by: Appendix A. Appendix contents 1 Introduction 2 Background 3 Bias-Adjusted DPO 4 Theoretical Characterization 5 Experiments 5.1 Experimental settings 5.2 Signed-UltraFeedback 5.3 MultiPref: length bias 5.4 MultiPref: formatting bias 6 Conclusions References A Related Work B Proofs B.1 Proof of Proposition B.2 Proof of Theorem B.3 Proof of Corollary B.4 Proof of Proposition B.5 The speed of the mean under the two parameterisations C The common offset and the choice of anchor C.1 The policy’s tilt toward the attribute C.2 What the labels determine C.3 Reference anchoring C.4 Target anchoring C.5 Experiment: a fixed offset cures the DPO policy D Datasets D.1 The MultiPref corpus and the length attribute D.2 The names corpus (Signed-UltraFeedback) E Experimental details E.1 Models and training E.2 Arms E.3 Baselines E.4 Readouts F Ablations and controls F.1 Variants of the bias model on Signed-UltraFeedback F.2 MultiPref: variants, warm starts and the remaining baselines F.3 Reading the signature from probabilities, not samples F.4 LoRA calibration at 0.5B F.5 Robustness to the model family: Mistral-7B G Vote prediction with the learned bias parameters Appendix A Related Work Length bias in RLHF and DPO. A substantial line of work identifies the tendency of preference-trained models to exploit response length as a proxy for quality. Park et al. (2024) study length exploitation in DPO specifically and propose a regularisation strategy, Lu et al. (2024) eliminate length reliance through down-sampled KL divergence, and Liu et al. (2024) desensitise DPO to length by decoupling explicit length preference from implicit quality signals. Wang et al. (2025a) show that length is not the only such bias: reward models also favour the response format that the prompt template invites, and correcting length alone leaves that preference in place. On the reward-model side, Zhao et al. (2026) learn and correct length-bias patterns before preference optimisation, while Shen et al. (2023) and Chen et al. (2024) give the reward model a separate head for length and discard it when the policy is trained. All of them build length into the objective; BA-DPO treats length as one instance of a per-annotator additive bias on an attribute the user declares, and our experiments show that correcting length leaves a second attribute in place. Format and other spurious features. Beyond length, Zhang et al. (2025) show that format features such as lists, markdown and emojis bias alignment in the same way, and Wang et al. (2025b) propose causal reward models addressing several bias types jointly. Ng et al. (2025) learn a representation that separates the factors the reward should depend on from the spurious ones, with identifiability guarantees. These operate at the reward-model level, whereas BA-DPO corrects at the preference-likelihood level and so remains compatible with the reward-model-free setting DPO defines. Allam (2024) address a different problem under a similar name: they use DPO to make a model produce less biased text, with a curated dataset in which the less biased of two completions is always the preferred one. There the bias is in the model’s outputs and the labels are the correction; here the bias is in the annotators’ labels and the outputs are what it corrupts. Their preferred completions carry the quality signal to be learned, whereas our declared attribute carries the bias to be removed. Bias-only models. Training a separate term on a declared biased feature alongside the main model and dropping it at test time is the standard way to keep a classifier from relying on a known dataset bias (Clark et al., 2019; He et al., 2019; Karimi Mahabadi et al., 2020). BA-DPO is that construction inside preference optimisation: the bias term is additive in the reward, so it survives the DPO reparameterisation and needs no second network, and its coefficient is per annotator, which those methods have no analogue for because their corpora carry no annotator identity. Annotator heterogeneity. Standard RLHF treats all annotators as sampling from one latent reward. Xiao et al. (2025) analyse the algorithmic bias this induces and show that RLHF can exhibit preference collapse under annotator disagreement. In the taxonomy of Liu et al. (2025), BA-DPO belongs to the data-quality heterogeneity branch, alongside methods that model disagreement as a mixture over latent preference types (Chidambaram et al., 2026) or as preference dispersion (Chen et al., 2025); Chhan et al. (2024) use annotator identifiers to infer per-user reliability in preference-based reward learning, though not within DPO. ADPO (Wang, 2025) anchors the policy’s logits to the reference so that a response’s prior popularity is separated from its quality; the offset it removes is common to every response and carries no annotator identity, so it does not address a preference for an attribute that only some responses carry. Those methods represent or balance disagreement, whereas BA-DPO models it as an additive bias on a declared attribute, which is what makes removal rather than accommodation possible, and separates what its parameters do: the mean removes the bias and the deviations describe the annotators. Appendix B Proofs Throughout, judgments are indexed by j=1,…,Nj=1,…,N; judgment j was made by annotator k(j)k(j) on prompt xjx_j, with winner yw,jy_w,j and loser yl,jy_l,j, and Δδj≔δg(xj,yw,j)−δg(xj,yl,j)∈−1,0,1 _j _g(x_j,y_w,j)- _g(x_j,y_l,j)∈\-1,0,1\. We write ϕ(t)≔−logσ(t)=log(1+e−t),ϕ′(t)=−σ(−t),ϕ′(t)=σ(t)σ(−t)>0,φ(t) - σ(t)= (1+e^-t ), φ (t)=-σ(-t), φ (t)=σ(t)\,σ(-t)>0, so that the negative log-likelihood of any model whose logit on judgment j is LjL_j equals ∑jϕ(Lj) _jφ(L_j). B.1 Proof of Proposition 2 Proposition 2 (Compatibility with the DPO reparameterization). Let r(x,y)r(x,y) be any reward function for which Z(x)=∑yπref(y∣x)exp(r(x,y)/β)Z(x)= _y _ref(y x) (r(x,y)/β) is finite for every prompt, and let θkk=1m\ _k\_k=1^m be arbitrary evaluator bias parameters. Then the bias-aware preference model (7) can be written equivalently as pk(y1≻y2∣x)=σ(βlogπ(y1∣x)πref(y1∣x)−βlogπ(y2∣x)πref(y2∣x)+θk(δg(x,y1)−δg(x,y2)))p_k(y_1 y_2 x)=σ\! (β π(y_1 x) _ref(y_1 x)-β π(y_2 x) _ref(y_2 x)+ _k ( _g(x,y_1)- _g(x,y_2) ) ) with π(y∣x)=πref(y∣x)exp(r(x,y)/β)/Z(x)π(y x)= _ref(y x) (r(x,y)/β)/Z(x). Since Z(x)Z(x) is finite and positive, π(y∣x)=πref(y∣x)exp(r(x,y)/β)/Z(x)π(y x)= _ref(y x) (r(x,y)/β)/Z(x) is a probability distribution over responses for every prompt. Taking logarithms and solving for the reward, r(x,y)=βlogπ(y∣x)πref(y∣x)+βlogZ(x),r(x,y)=β π(y x) _ref(y x)+β Z(x), which is the reparameterisation of Equation 5 of Rafailov et al. (2023). Now substitute this expression for r into the logit of the biased Bradley-Terry model (7): r(x,y1)+θkδg(x,y1)−r(x,y2)−θkδg(x,y2) r(x,y_1)+ _k\, _g(x,y_1)-r(x,y_2)- _k\, _g(x,y_2) =βlogπ(y1∣x)πref(y1∣x)+βlogZ(x)−βlogπ(y2∣x)πref(y2∣x)−βlogZ(x)+θk(δg(x,y1)−δg(x,y2)) =β π(y_1 x) _ref(y_1 x)+β Z(x)-β π(y_2 x) _ref(y_2 x)-β Z(x)+ _k ( _g(x,y_1)- _g(x,y_2) ) =βlogπ(y1∣x)πref(y1∣x)−βlogπ(y2∣x)πref(y2∣x)+θk(δg(x,y1)−δg(x,y2)). =β π(y_1 x) _ref(y_1 x)-β π(y_2 x) _ref(y_2 x)+ _k ( _g(x,y_1)- _g(x,y_2) ). The two copies of βlogZ(x)β Z(x) cancel because both responses share the prompt, and the bias terms pass through untouched because they are added to the reward and do not involve Z(x)Z(x). Applying σ to the last line gives the stated form. A bias that entered multiplicatively, or through the partition function, would not cancel in this way. ∎ B.2 Proof of Theorem 3 Theorem 3 (Identifiability of the bias parameters). The loss ℒL of (8), viewed as a function of the reward values r(x,y)r(x,y) and of θ, is jointly convex in (r,θ)(r,θ), and all its minimisers assign the same probability to every judgment. If the annotator pool is connected and (r^,θ^)( r, θ) is a minimiser, the minimisers are exactly the pairs (r^+cδg+h,θ^−c),c∈ℝ, ( r+c\, _g+h,\ θ-c1 ), c , with h any function constant on the two responses of every compared pair. Every difference θk−θk′ _k- _k therefore takes the same value at every minimiser, while their mean θ¯≔1m∑kθk θ 1m _k _k is not determined by the judgments. We first write the logit. For judgment j the logit of the bias-aware model is Lj(r,θ)=r(xj,yw,j)−r(xj,yl,j)+θk(j)Δδj.L_j(r,θ)=r(x_j,y_w,j)-r(x_j,y_l,j)+ _k(j)\, _j. As a function of the reward values and the bias vector together, LjL_j is a sum of parameters with coefficients +1+1, −1-1 and Δδj _j, hence affine. For a fixed policy the reward difference is a constant uju_j and Lj=uj+θk(j)ΔδjL_j=u_j+ _k(j) _j is affine in θ alone. The negative log-likelihood is ℒ=∑j=1Nϕ(Lj).L= _j=1^Nφ (L_j ). Convexity follows. ϕφ is convex because ϕ′>0φ >0. A convex function composed with an affine map is convex, and a sum of convex functions is convex. Hence ℒL is convex in (r,θ)(r,θ) jointly for an arbitrary reward, and in particular convex in θ for a fixed policy. All minimisers give the same probabilities. Let (r,θ)(r,θ) and (r′,θ′)(r ,θ ) both minimise ℒL, with logits LjL_j and Lj′L_j . Their midpoint has logits 12(Lj+Lj′) 12(L_j+L_j ), because LjL_j is affine, and by convexity it minimises ℒL as well. Since ϕφ is strictly convex (ϕ′>0φ >0), ∑jϕ(Lj+Lj′2)≤12∑jϕ(Lj)+12∑jϕ(Lj′), _jφ ( L_j+L_j 2 )\;≤\; 12 _jφ(L_j)+ 12 _jφ(L_j ), with equality if and only if Lj=Lj′L_j=L_j for every j. All three quantities equal the minimum of ℒL, so equality holds, the logits agree, and so do the probabilities σ(Lj)σ(L_j). For the strictness of the dependence on a single θk _k, take the direction that moves only θk _k. Since ∂Lj/∂θk=Δδj∂ L_j/∂ _k= _j when k(j)=k(j)=k and 00 otherwise, the second derivative of ℒL along this direction is ∂2ℒ∂θk2=∑j:k(j)=kϕ′(Lj)Δδj2. ∂^2L∂ _k^2= _j:\,k(j)=kφ (L_j)\, _j^2. Every term is non-negative, and a term is positive exactly when Δδj≠0 _j≠ 0, that is, when judgment j is a cross-group comparison. So the second derivative is positive at every point as soon as annotator k has one cross-group judgment, which is strict convexity in θk _k. For the characterisation of the minimisers, let (r,θ)(r,θ) and (r′,θ′)(r ,θ ) be two parameter vectors and write dr≔r′−rdr r -r and dθ≔θ′−θdθ θ -θ for their difference. The probability assigned to judgment j is σ(Lj)σ(L_j), and σ is strictly increasing, so the two vectors assign the same probability to every judgment if and only if Lj(r′,θ′)=Lj(r,θ)L_j(r ,θ )=L_j(r,θ) for every j. Because LjL_j is affine, this is the linear system dr(xj,yw,j)−dr(xj,yl,j)+dθk(j)Δδj=0for all j=1,…,N.dr(x_j,y_w,j)-dr(x_j,y_l,j)+d _k(j)\, _j=0 all j=1,…,N. (12) It remains to show that the solutions of (12) are exactly the pairs (dr,dθ)(dr,dθ) with dθ=−cdθ=-c1 and dr−cδgdr-c\, _g constant on the two responses of every compared pair, for some c∈ℝc . We solve the system one compared pair at a time. Fix a compared pair (y1,y2)(y_1,y_2) on prompt x and let K be the set of annotators who judged it. If the pair is same-group, then Δδj=0 _j=0 for each of these judgments and (12) reads dr(x,y1)=dr(x,y2).dr(x,y_1)=dr(x,y_2). If the pair is cross-group, then for each annotator k∈Kk∈ K, whichever response that annotator chose, (12) reads dr(x,y1)−dr(x,y2)=−dθk(δg(x,y1)−δg(x,y2)).dr(x,y_1)-dr(x,y_2)=-\,d _k\, ( _g(x,y_1)- _g(x,y_2) ). (13) (If the annotator chose y2y_2, both sides of (12) are the negatives of those in (13), so the equation is the same.) The left side of (13) does not depend on k, and the bracket on the right is ±1± 1, so dθkd _k takes the same value for every k∈Kk∈ K: annotators who judged the same cross-group pair have the same dθkd _k. Connectedness spreads this common value. Two annotators joined by an edge of the graph, that is, who judged a cross-group pair in common, share the same dθdθ. When the pool is connected, any two annotators are joined by a chain of such edges, so the value is the same along the chain and hence for the whole pool: there is a single constant c with dθk=−cfor every k,that is,dθ=−c.d _k=-c every k, is, dθ=-c1. Next the reward part. Substitute dθk=−cd _k=-c back into (13): every cross-group pair satisfies dr(x,y1)−dr(x,y2)=c(δg(x,y1)−δg(x,y2)),that is,(dr−cδg)(x,y1)=(dr−cδg)(x,y2).dr(x,y_1)-dr(x,y_2)=c\, ( _g(x,y_1)- _g(x,y_2) ), is, (dr-c\, _g )(x,y_1)= (dr-c\, _g )(x,y_2). Every same-group pair satisfies the same equality, because there dr(x,y1)=dr(x,y2)dr(x,y_1)=dr(x,y_2) and δg(x,y1)=δg(x,y2) _g(x,y_1)= _g(x,y_2). So dr−cδgdr-c\, _g is constant on the two responses of every compared pair. This proves that every solution of (12) has the stated form. Conversely, take any c, any dθ=−cdθ=-c1, and any drdr such that dr−cδgdr-c\, _g is constant within every compared pair. For judgment j, dr(xj,yw,j)−dr(xj,yl,j)=c(δg(xj,yw,j)−δg(xj,yl,j))=cΔδj=−dθk(j)Δδj,dr(x_j,y_w,j)-dr(x_j,y_l,j)=c\, ( _g(x_j,y_w,j)- _g(x_j,y_l,j) )=c\, _j=-\,d _k(j)\, _j, so (12) holds, and the solutions are exactly as claimed. It remains to put the two halves together. Any two minimisers have the same logits, so their difference solves (12), and conversely every solution leaves all logits and hence ℒL unchanged. The minimisers are therefore exactly the stated family. Any two of them have dθ=−cdθ=-c1, so dθk−dθk′=0d _k-d _k =0: they agree on every difference θk−θk′ _k- _k , while the common level of θ shifts by c. ∎ B.3 Proof of Corollary 4 Corollary 4 (Reward-bias degeneracy). Write θk=θ¯+εk _k= θ+ _k with ∑kεk=0 _k _k=0. Shifting the mean by −c-c and adding cδgc\, _g to the reward leaves every judgment probability unchanged, so θ¯ θ is not determined by the votes. If the annotator pool is connected, the deviations εk _k take the same value at every minimiser. The transformation r↦r+cδgr r+c\, _g, θ↦θ−cθ θ-c1 is the case of Theorem 3 in which the within-pair-constant part of the reward shift is zero, so the invariance follows from the theorem. The direct check is short. The logit for a comparison (yw,yl)(y_w,y_l) by annotator k is L=r(x,yw)−r(x,yl)+θk(δg(x,yw)−δg(x,yl)).L=r(x,y_w)-r(x,y_l)+ _k ( _g(x,y_w)- _g(x,y_l) ). After the transformation it becomes L′ L =(r(x,yw)+cδg(x,yw))−(r(x,yl)+cδg(x,yl))+(θk−c)(δg(x,yw)−δg(x,yl)) = (r(x,y_w)+c\, _g(x,y_w) )- (r(x,y_l)+c\, _g(x,y_l) )+( _k-c) ( _g(x,y_w)- _g(x,y_l) ) =r(x,yw)−r(x,yl)+c(δg(x,yw)−δg(x,yl))−c(δg(x,yw)−δg(x,yl))+θk(δg(x,yw)−δg(x,yl)) =r(x,y_w)-r(x,y_l)+c ( _g(x,y_w)- _g(x,y_l) )-c ( _g(x,y_w)- _g(x,y_l) )+ _k ( _g(x,y_w)- _g(x,y_l) ) =L. =L. Every logit, and so every likelihood term, is unchanged, so the data cannot distinguish (r,θ¯)(r, θ) from (r+cδg,θ¯−c)(r+c\, _g, θ-c). For the deviations, Theorem 3 shows that every difference θk−θk′ _k- _k takes the same value at every minimiser. Each deviation is an average of such differences, θk−θ¯=1m∑k′=1m(θk−θk′), _k- θ= 1m _k =1^m ( _k- _k ), so each εk=θk−θ¯ _k= _k- θ does too. ∎ B.4 Proof of Proposition 5 Proposition 5 (Representability of the bias components). Decompose θk=θ¯+εk _k= θ+ _k with ∑kεk=0 _k _k=0. The shared component θ¯δg(x,y) θ\, _g(x,y) is expressible as a function of (x,y)(x,y) and can therefore be represented inside the policy’s implicit reward, whereas the deviation term εkδg(x,y) _k\, _g(x,y) depends on k, which the policy never observes, and is representable by no policy π(y∣x)π(y x) unless every εk=0 _k=0, that is, unless all annotators have the same bias. Write θk=θ¯+εk _k= θ+ _k with ∑kεk=0 _k _k=0. The bias contribution to the logit of a comparison splits into two terms, θk(δg(x,yw)−δg(x,yl))=θ¯(δg(x,yw)−δg(x,yl))⏟does not depend on k+εk(δg(x,yw)−δg(x,yl))⏟depends on k. _k ( _g(x,y_w)- _g(x,y_l) )= θ ( _g(x,y_w)- _g(x,y_l) )_does not depend on k+ _k ( _g(x,y_w)- _g(x,y_l) )_depends on k. The first term can be moved into the reward. Define r~(x,y)≔r(x,y)+θ¯δg(x,y). r(x,y) r(x,y)+ θ\, _g(x,y). Since δg(x,y) _g(x,y) is a function of the prompt and the response only, r~ r is a reward function on (x,y)(x,y) like any other, and the reward difference of r~ r on a comparison equals the reward difference of r plus the first term. By Proposition 2, r~ r admits a policy representation, so the shared component of the bias can be carried by the policy’s implicit reward. The second term cannot. Suppose there were a function h(x,y)h(x,y) with εkδg(x,y)=h(x,y)for all k and all (x,y). _k\, _g(x,y)=h(x,y) all k and all (x,y). Pick any (x,y)(x,y) with δg(x,y)=1 _g(x,y)=1; one exists whenever the attribute occurs in the corpus. Then εk=h(x,y) _k=h(x,y) for every k, so all deviations are equal to one number, and since they sum to zero that number is 00. Hence the second term is a function of (x,y)(x,y) only when every εk=0 _k=0, that is, when all annotators have the same bias. Otherwise no policy πθ(y∣x) _θ(y x), whose implicit reward is a function of (x,y)(x,y), can represent it. ∎ B.5 The speed of the mean under the two parameterisations Proposition 6 (The mean learns m times more slowly). Assume annotators contribute equal data shares and gradient descent with a common learning rate on the bias parameters. Under the shared-mean parameterisation θk=θ¯+εk _k= θ+ _k, one step moves the shared parameter θ¯ θ exactly m times further than the same step moves the mean of the θk _k under the naive per-annotator model, and moves the deviations from the mean identically. By (11), the gradient of the loss with respect to θk _k is ∂ℒ∂θk=nkNGk,Gk≔−(x,yw,yl)∼k[σ(−u−bk)Δδ], ∂ _k= n_kN\,G_k, G_k -E_(x,y_w,y_l) _k [σ(-u-b_k)\, δ ], and with equal shares nk=N/mn_k=N/m this is Gk/mG_k/m. In the naive per-annotator model each θk _k is its own parameter, so one gradient-descent step with learning rate η moves it by −ηGk/m-η\,G_k/m, and the mean of the parameters moves by Δθ¯=1m∑k=1m(−ηGkm)=−ηmG¯,G¯≔1m∑k=1mGk. θ= 1m _k=1^m (-η\, G_km )=- ηm\, G, G 1m _k=1^mG_k. In the shared-mean model θk=θ¯+εk _k= θ+ _k, the parameter θ¯ θ enters every θk _k with coefficient 11, so by the chain rule ∂ℒ∂θ¯=∑k=1m∂ℒ∂θk=∑k=1mGkm=G¯, ∂ θ= _k=1^m ∂ _k= _k=1^m G_km= G, and one step moves θ¯ θ by −ηG¯-η\, G, which is m times the motion of the naive mean. For the deviations, the gradient on εk _k equals the gradient on θk _k, namely Gk/mG_k/m, because εk _k enters only θk _k and with coefficient 11. In the naive model the deviation of θk _k from the mean of the θ’s therefore moves by −ηGkm−Δθ¯=−ηm(Gk−G¯).-η\, G_km- θ=- ηm (G_k- G ). In the shared-mean model, θk _k moves by −ηG¯−ηGk/m-η\, G-η\,G_k/m and the mean of the θ’s by −ηG¯−ηG¯/m-η\, G-η\, G/m, so the deviation moves by the same amount, −ηm(Gk−G¯)- ηm(G_k- G). The constraint ∑kεk=0 _k _k=0 of Proposition 5 is a decomposition, not a constraint enforced in training; the mean of the εk _k drifts by −ηG¯/m-η\, G/m per step, the same m-times smaller motion as the naive mean. ∎ A rough estimate under Adam. The bias parameters are trained with Adam, which divides each coordinate’s step by a running estimate of the magnitude of its gradient. Consider a coordinate whose gradient in a step equals g≠0g≠ 0 with probability p and 00 otherwise, independently across steps, and Adam with learning rate η, moment parameters β1,β2 _1, _2 and a negligible stabiliser ϵε. The first-moment estimate mtm_t is an exponential average of past gradients, so at stationarity [mt]=pgE[m_t]=p\,g. The second-moment estimate vtv_t averages past squared gradients over a window of about 1/(1−β2)1/(1- _2) steps, which for the default β2=0.999 _2=0.999 makes it close to its mean pg2p\,g^2. The expected step is therefore −η[mt]pg2=−ηpsign(g),-η\, E[m_t] p\,g^2=-η\, p\,sign(g), against −ηpg-η\,p\,g under gradient descent: a coordinate updated in a fraction p of the steps moves at a rate proportional to p p rather than p. In a step of B judgments of which b are cross-group, a free θk _k receives a gradient only from its own annotator’s cross-group judgments, so pk≈b/mp_k≈ b/m when the judgments are spread evenly over the m annotators, whereas θ¯ θ receives a gradient whenever the step contains any cross-group judgment, so p≈1p≈ 1. The shared parameter therefore moves about m/b m/b times further per step than each free θk _k, and than their mean, in place of the factor m of Proposition 6. In our runs B=32B=32; on MultiPref, with 227227 annotators and 59%59\% of the judgments cross-group, the factor is about 3.53.5 against 227227 under gradient descent. This is an estimate, not a proven result: it takes the gradient as constant when present and does not model the policy’s own motion, so it orders the two parameterisations and does not predict where the naive mean stops. The measured runs answer that: the free mean ends at 0.500.50 of θ¯ θ under formatting and at 0.610.61 under length (Appendix F.2). Appendix C The common offset and the choice of anchor Theorem 3 and Corollary 4 state that the labels determine the annotators’ biases up to a common offset. This appendix says what that offset is in terms of the policy, why the reference policy fixes it in every experiment of the paper, and how a requirement on the outputs can fix it instead. Nothing here adds an assumption: the results are those of Section 4, read in a coordinate in which the offset is visible. We write the argument for one binary attribute; with several attributes it applies to each coordinate of θk _k separately, and the swap pairs of Signed-UltraFeedback, which flip exactly one attribute, are the pairs it refers to. C.1 The policy’s tilt toward the attribute By Proposition 2, the policy enters the likelihood only through its implicit reward r(x,y)≔βlogπ(y∣x)πref(y∣x),r(x,y) β π(y x) _ref(y x), a score the policy assigns to every response relative to the reference: positive where the policy has made the response more likely than the reference did, negative where less. The logit of a judgment by annotator k is r(x,yw)−r(x,yl)+θkΔδr(x,y_w)-r(x,y_l)+ _k\, δ with Δδ=δg(x,yw)−δg(x,yl) δ= _g(x,y_w)- _g(x,y_l). For a response y that carries the attribute, δg(x,y)=1 _g(x,y)=1, let y¯ y denote its counterfactual: the same response with the attribute removed or exchanged, so that δg(x,y¯)=0 _g(x, y)=0. The tilt of the policy toward the attribute at (x,y)(x,y) is the extra score it gives to the version that carries the attribute, beyond what the reference gives it: a(x,y)≔r(x,y)−r(x,y¯)=β[logπ(y∣x)π(y¯∣x)−logπref(y∣x)πref(y¯∣x)].a(x,y) r(x,y)-r(x, y)=β [ π(y x)π( y x)- _ref(y x) _ref( y x) ]. Every score then splits into the score of the attribute-free version and the tilt, r(x,y)=q(x,y)+a(x,y)δg(x,y),r(x,y)=q(x,y)+a(x,y)\, _g(x,y), where q(x,y)q(x,y) is r(x,y¯)r(x, y) when y carries the attribute and r(x,y)r(x,y) otherwise. This is a change of variables, not a model: q is the part of the score that concerns the answer itself, and a is the part that concerns the attribute. The tilt is a function of (x,y)(x,y), which the policy can represent (Proposition 5); the case to keep in mind is the one in which it takes a single value a for every response, since that is the part of the tilt on which the degeneracy acts. The two kinds of comparison in the data now read differently. On a pair whose two responses differ only in the attribute, q cancels and the logit of annotator k is a+θk,a+ _k, the policy’s tilt plus the annotator’s bias and nothing else. On a pair whose two responses carry the same attribute value, the tilt cancels and the logit is q(x,yw)−q(x,yl)q(x,y_w)-q(x,y_l), quality alone. Mixed pairs combine the two. C.2 What the labels determine The transformation of Corollary 4, r↦r+cδgr r+c\, _g and θk↦θk−c _k _k-c for every k, adds c to the tilt of every response and subtracts c from every bias, and every logit is unchanged. In the coordinates above, Theorem 3 says three things. 1. The quality part q is determined up to the shifts that standard DPO cannot see either, a constant within each compared pair. 2. For every annotator the sum a+θka+ _k is determined. Hence every difference θk−θk′ _k- _k is determined, and in the parameterisation θk=θ¯+εk _k= θ+ _k every deviation εk _k is determined. 3. The split of a+θ¯a+ θ into a and θ¯ θ is not determined. The common offset is therefore not a parameter but a direction: the line a+θ¯=consta+ θ=const in the (a,θ¯)(a, θ) plane, along which the likelihood is flat. Fixing the model means choosing a point on this line, and since every point fits every judgment equally well, the choice is an input, not an estimate. The statistical content is the one stated after Corollary 4: a bias shared by every annotator cannot be told apart from the attribute genuinely carrying reward. Declaring the attribute states that the shared preference for it is bias; the anchor states from which zero that bias is counted. C.3 Reference anchoring Every run in this paper starts from the reference, π=πrefπ= _ref, so at initialisation r≡0r≡ 0, a=0a=0 and θ=0θ=0. Training moves a+θ¯a+ θ to the value the labels demand. Which of the two moves is decided by the dynamics, not by the objective: the scalar has a convex path to its conditional optimum (Theorem 3) and its own learning rate, whereas the policy moves through a non-convex landscape under the KL anchoring, so the level goes into θ¯ θ and a stays near zero (Section 5; Proposition 6 explains why the naive per-annotator model gets there m times more slowly). The learned θ¯ θ is therefore the annotators’ shared tilt toward the attribute measured relative to the reference, and the debiased policy’s attribute rate is the reference’s rate. Three consequences follow. First, removal is measured against the reference, which is why the tables report the share of DPO’s increase over the reference that is removed, and why the 0.5B reference of Signed-UltraFeedback is balanced on both attributes by construction (Appendix D.2): the bias then arrives through the preference stage alone. Second, a reference that is itself tilted stays tilted by its own amount: at 8B the reference signs with a woman-coded name at 0.4650.465, and BA-DPO returns to 0.490.49, not to 0.500.50 (Table 1). Third, the estimates of the annotators do not depend on the reference’s tilt: r≡0r≡ 0 at the start whatever the reference prefers, so θ¯ θ measures the tilt of the labels beyond zero implicit reward, and the deviations εk _k are the same at every point of the line by item (i) above. The audit of the annotators is anchor-independent; only the policy’s rate depends on the anchor. C.4 Target anchoring The same free direction can be fixed by a requirement on the outputs instead of by the reference. Suppose the attribute has a fair rate t, one half for the name in a signature, and write logitp=logp1−plogitp= p1-p. Let p be the reference’s rate in the pairwise sense, logπref(y∣x)/πref(y¯∣x)=logitp _ref(y x)/ _ref( y x)=logitp. A policy with tilt a has log-odds logitp+a/βlogitp+a/β, so the tilt that reaches the target is c(t)=β[logitt−logitp].c(t)=β\, [logitt-logitp ]. (14) When the reference’s log-odds vary across prompts, the rate of the tilted policy is still continuous and strictly increasing in c, so the target is reached at exactly one value, found by a one-dimensional search; (14) is that value for constant log-odds. Corollary 7 (Target anchoring). Let (r,θ)(r,θ) be any parameter vector and c∈ℝc . The parameter vector (r+cδg,θ−c)(r+c\, _g,\;θ-c1) assigns the same probability to every judgment, has the same deviations εk _k, has shared mean θ¯−c θ-c, and corresponds to the policy πc(y∣x)∝π(y∣x)exp(cδg(x,y)/β) _c(y x) π(y x) (c\, _g(x,y)/β ), whose log-odds between a response and its counterfactual exceed those of π by c/βc/β. Proof. The first three claims are Corollary 4. The policy follows from π∝πrefexp(r/β)π _ref (r/β) (Proposition 2) with r+cδgr+c\, _g in place of r, and the log-odds shift is c/βc/β because δg _g differs by one between y and y¯ y. ∎ Target anchoring therefore changes exactly one thing. The fit to every judgment is the same, so nothing is paid in likelihood. The deviations are the same, so the audit of the annotators is the same. The shared mean shifts by −c(t)-c(t), which means that the reference’s own tilt is now counted as bias rather than as quality. What changes is the policy, which is moved along the attribute by c(t)/βc(t)/β in log-odds, and with it the KL to the reference. The requirement spends the one degree of freedom the data do not use, and nothing else moves. How the anchor is applied. Three procedures produce the tilted policy. At sampling time, in closed form: the responses that carry the attribute are reweighted by exp(c/β) (c/β) per prompt, which for a signature is a shift on the choice of the name and for an attribute of the whole text is a reweighting or a rejection step. In training, with a fixed offset: on pairs that differ only in the attribute, each taken in both orders, the loss of the two orders is −logσ(u+b)−logσ(−u−b)- σ(u+b)- σ(-u-b) with u the policy’s margin, minimised at u=−bu=-b, so a fixed offset b=−c(t)b=-c(t) in the logit drives the policy to carry the tilt c(t)c(t) itself. This removes a tilt a policy already has, from its own generations and with no annotator labels, and is the natural cure for a model trained without the bias term; Section C.5 tests it. Through the reference: a reference balanced on the attribute makes the two anchors coincide, which is the construction used at 0.5B. Limits. Target anchoring needs an attribute with a fair value. A name has one; length and formatting do not, and for them the reference remains the only sensible anchor, which is why Table 2 reads the held-out gap rather than a target rate. The target is a normative choice, as the attribute partition is, but it is one that can be audited by measuring the rate, whereas the reference’s rate is an accident of pretraining. Moving the policy changes which responses it produces, so judged quality has to be checked after the tilt. Finally, the level can also be pinned from data rather than from a target: a small set of judgments by annotators known to be unbiased breaks the flat direction, since for them θk=0 _k=0 is known and a is then identified. We leave this to future work (Section 6). Worked example. Take a response signed Emily Miller and its counterfactual signed Greg Miller, the same text otherwise. The policy’s tilt a is β times the difference between the policy’s and the reference’s log-odds for the first version over the second. When annotator k judges this pair, the logit is a+θka+ _k: the labels of the swap pairs fix, for every annotator, the sum of the policy’s tilt and the annotator’s bias, but not how it splits. At initialisation a=0a=0, and after training under reference anchoring a is still near zero and θ¯ θ holds the shared level. For the numbers, take β=0.1β=0.1. The 8B reference signs with a woman-coded name at 0.4650.465, so logitp=−0.14logitp=-0.14 and reaching t=0.5t=0.5 needs c=0.1×(0+0.14)=0.014c=0.1×(0+0.14)=0.014, a shift of 0.140.14 in log-odds: for a nearly balanced reference the two anchors almost coincide. The 8B DPO policy trained on the same labels signs with a woman-coded name at 0.9670.967, so logitp=3.38logitp=3.38 and curing it needs c=−0.34c=-0.34, a shift of −3.4-3.4 in log-odds, which the training-time procedure obtains with the offset b=0.34b=0.34 on the gender coordinate. C.5 Experiment: a fixed offset cures the DPO policy Setting. The source is the DPO policy of Signed-UltraFeedback at 0.5B (Table 1), one per seed, which signs with a woman-coded name with probability 0.960.96 and with a black-coded name with probability 0.990.99. The training pairs come from the policy itself: 3,0003,000 fresh UltraFeedback prompts (the ones after the 8,0008,000 of the corpus) with the signing instruction, answered by the source; the answers signed with a pool name are kept (2,6592,659 at seed 42); each signature is replaced by a random cell and then by the cell that differs from it in one attribute, with the same surname, which gives one pair per attribute per answer, taken in both orders: 10,63610,636 rows, of which 9,5769,576 are used for training and 1,0601,060 held out. No annotator and no label enter this stage. Since every pair is present in both orders, the loss on a pair is −logσ(u+b)−logσ(−u−b)- σ(u+b)- σ(-u-b) with u the policy’s margin, minimised at u=−bu=-b, so the offset b in the logit (the term b(gw−gl)b\,(g_w-g_l) of Section C.4) is the only thing the stage teaches. Training is DPO with the source as reference and the hyperparameters of Table 7 (β=0.1β=0.1, learning rate 5×10−75× 10^-7, 1,0001,000 steps of 3232 pairs). The offset for a target rate t is b=−βs∗(t)b=-β s (t), where s∗(t)s (t) is the log-odds shift that brings the source’s signature probabilities to t, found by bisection on the 300300 evaluation prompts: for parity at seed 42, s∗=3.39s =3.39 on the gender coordinate and 5.065.06 on the race coordinate. Calibration. The exact tilt alone stops short: at seed 42 it reaches 0.7360.736 and 0.7530.753 instead of 0.500.50, two thirds of the intended shift in log-odds. Three offsets at seed 42 (3.39/5.063.39/5.06, 4.42/6.184.42/6.18 and 5.13/6.925.13/6.92 in log-odds, gender/race) give achieved shifts of 2.24/3.682.24/3.68, 3.13/4.763.13/4.76 and 3.79/5.513.79/5.51, which lie on a line with residuals below 0.0150.015: achieved =0.888intended−0.777=0.888\,intended-0.777 on gender and 0.980intended−1.2850.980\,intended-1.285 on race. The policy therefore follows the offset one for one after a fixed deficit of 0.80.8 and 1.31.3 log-odds, which the 1,0001,000 steps leave. Every other run below inverts this line for the exact tilt of its target; the line is fitted at seed 42 and applied unchanged to the other two source seeds and to the targets 0.30.3 and 0.70.7. Table 4: A fixed offset applied to the DPO policy of Signed-UltraFeedback at 0.5B, trained on the policy’s own generations with no annotator. Rates are the probability readout of Table 1; RM- is the judge with the signature removed; KL is per token, to the source (the DPO policy) and to the SFT reference; <0.01<0.01 marks an estimate below the noise of the Monte Carlo estimate, which can come out negative; the held-out gap is defined in Section 5.1; t is the target rate; Signed is the share of answers that carry a signature. Seed 42 unless stated; the parity row over three seeds gives the mean and the 95% interval over the source seeds. Policy t p(woman)p(woman) p(black)p(black) Signed Tokens RM- KL src / SFT Gap DPO (source) – 0.963 0.992 0.84 213 −2.75-2.75 – / 0.037 +0.095+0.095 + offset, exact tilt 0.5 0.736 0.753 0.86 210 −2.79-2.79 <0.01<0.01 / 0.028 +0.029+0.029 + offset, calibrated 0.5 0.533 0.509 0.86 214 −2.71-2.71 <0.01<0.01 / 0.023 −0.046-0.046 + offset, calibrated, 3 seeds 0.5 0.50±0.070.50± 0.07 0.47±0.080.47± 0.08 0.86 213 −2.72-2.72 0.003 / 0.022 −0.053-0.053 + offset, calibrated 0.3 0.262 0.233 0.84 217 −2.77-2.77 0.034 / 0.024 −0.144-0.144 + offset, calibrated 0.7 0.681 0.659 0.85 206 −2.90-2.90 <0.01<0.01 / 0.027 +0.019+0.019 BA-DPO pooled, Table 1 – 0.545 0.616 0.86 213 −2.53-2.53 – / 0.025 +0.005+0.005 Results. Table 4 reports the runs. With the calibrated offset the DPO policy signs with a woman-coded name at 0.50±0.070.50± 0.07 and with a black-coded name at 0.47±0.080.47± 0.08 over the three source seeds, from 0.960.96 and 0.990.99; the targets 0.30.3 and 0.70.7 are reached within 0.070.07 and 0.040.04, and the rate is monotone in the offset. Nothing else moves: the signing rate, the length and the judge score with the signature removed stay within their seed spread, the KL to the source is at the noise floor of the estimate except at the largest offset (0.0340.034), and the KL to the SFT reference falls from DPO’s 0.0370.037 to 0.0220.022 at parity, below the 0.0250.025 of BA-DPO, since the part of DPO’s divergence that the offset removes is the name shift itself. The held-out gap, which marks a policy that has internalised the bias, falls from DPO’s 0.0950.095 to 0.030.03 after the exact tilt and to −0.05-0.05 at parity, where the cross-group accuracy (0.500.50 to 0.530.53) no longer exceeds the same-group one; at the target 0.30.3 it reaches −0.14-0.14, the policy now siding against the annotators on the pairs that differ in the attribute. The one remaining deficit is the constant one of the calibration paragraph: a property of the training budget, not of the objective, which a longer run or the second calibration point removes. Two remarks. First, the sampled signatures lag the probability readout at mild offsets and swing past it at strong ones (at parity, 69%69\% of the sampled signatures at seed 42 still carry a black-woman name while the pool probabilities are balanced; at the target 0.30.3, 92%92\% carry a white-man name), the concentration of the decoder on a few names described in Appendix F; the probability readout is the quantity the offset controls. Second, the offset is the same quantity as the sampling-time tilt of Section C.4 up to the fixed deficit, so a policy can be cured either by one training stage on its own generations or by reweighting at sampling time, with the same target. Appendix D Datasets Table 5 lists the two corpora, their sizes and the modifications made to each; the subsections give the construction in full. The 8B runs use the same data as the 0.5B runs. Table 5: The corpora. Judgments are training / held-out rows; the held-out rows are 10%10\% of prompts, split by prompt id. On MultiPref the 300300 evaluation prompts come from the held-out side; on Signed-UltraFeedback they come from UltraFeedback’s test split. Cross-group is the share of judgments in which exactly one response carries the attribute. Corpus Size Annotators Attributes (cross-group) Modifications MultiPref (Miranda et al., 2025) 10,46110,461 comparisons over 5,3235,323 prompts; 27,81227,812 / 3,0353,035 judgments 227227 real evaluators, four per comparison length, ratio ≥1.5≥ 1.5 (58.7%58.7\%); formatting, markdown present (26.4%26.4\%) ties dropped (26%26\% of rows); judgments kept disaggregated with the evaluator id; attributes computed from the response texts Signed-UltraFeedback 7,1777,177 UltraFeedback pairs (from 8,0008,000, responses under five words dropped) in three types: 2,8872,887 quality, 1,4621,462 swap, 2,8282,828 mixed; 25,84025,840 / 2,8682,868 judgments 6060 with planted biases in three classes of 2020, θk∈ℝ2 _k ^2 woman-coded and black-coded first name in the signature (36.4%36.4\% each) signature line --- First Last appended, names from an 89-name pool; swap pairs duplicate one response under two names; labels re-drawn; SFT answers re-signed at random so the reference starts balanced D.1 The MultiPref corpus and the length attribute allenai/multipref contains 10,46110,461 comparisons, each judged by exactly four evaluators (two crowd workers and two experts; 227227 distinct evaluators, no overlap between the two pools), over 5,3235,323 distinct prompts. We keep every judgment as its own training row, so that annotator identity is available to the per-annotator model, and drop the 26%26\% of judgments marked as ties, leaving 30,84730,847 rows. Ten percent of prompts are held out by prompt id (27,81227,812 train, 3,0353,035 held-out rows); the 300300 evaluation prompts are drawn from the held-out prompts. The reference policy is fine-tuned on the majority-chosen response of each comparison and is shared by every method and seed. Length as a binary attribute. A response is assigned δg=1 _g=1 when it is markedly longer than its partner, defined as a word-count ratio of at least 1.51.5; pairs closer in length than that are same-group: both responses have δg=0 _g=0. Because this attribute is defined on the pair, it is not a function of (x,y)(x,y): Proposition 2 and Theorem 3 need only a δg _g per judgment and are unaffected, whereas the representability of the shared component inside the policy’s implicit reward (Proposition 5) holds for length only through a per-response proxy such as the token count. The threshold is part of the attribute’s definition and not a tuning choice: assigning δg=1 _g=1 to the longer side of every pair makes 99.5%99.5\% of MultiPref comparisons cross-group and leaves no same-group comparisons to identify the reward. At ratio 1.51.5, 58.7%58.7\% of judgments are cross-group, the longer response wins 73%73\% of those, and the corpus-level offline bias estimate is θ^=0.99 θ=0.99 in log-odds. Ratios of 1.251.25, 22 and 33 give cross-group fractions of 73%73\%, 40%40\% and 21%21\% with θ θ of 0.910.91, 1.041.04 and 1.051.05; the estimate is stable across thresholds. Formatting as a binary attribute. A response is assigned δg=1 _g=1 when it contains markdown markup. The attribute is declared on the same 27,81227,812 training and 3,0353,035 held-out judgments as length; 26.4%26.4\% of judgments are cross-group and the offline estimate is θ^=0.98 θ=0.98. Table 6 characterises both attributes from the preference data before any training. Table 6: The declared attributes on MultiPref, characterised from the preference data before any training. Cross-group is the fraction of judgments in which exactly one response carries the attribute, which are the only judgments that identify θ. θ θ is the log-odds that the attribute-carrying side wins such a judgment, restated as odds in the next column. The last column is DPO’s measured amplification over the reference at 0.5B, in units of its own per-seed standard deviation. Attribute Cross-group θ θ Odds DPO amplification Length 58.7% 0.99 2.7 : 1 longer +96+96 tokens (65σ65σ) Formatting 26.4% 0.98 2.7 : 1 markdown +0.034+0.034 (1.6σ1.6σ) D.2 The names corpus (Signed-UltraFeedback) The names corpus is built rather than collected: the responses come from a public preference corpus, a name signature is added to them, and annotators whose biases are planted relabel every pair, so the bias of every annotator is known. Two attributes are planted at once, and the annotators are organised in classes whose biases differ. The main text calls it Signed-UltraFeedback (Section 5.2). We stress that the annotators are constructed to prefer a name marker. It does not measure prejudice, and we refer to the attributes as gender-coded and race-coded names rather than as gender or race bias. Prompts and responses. We draw 8,0008,000 preference pairs from the train_prefs split of ultrafeedback_binarized (Cui et al., 2024) and discard pairs in which either response has fewer than five words, which leaves 7,1777,177. Each pair carries a prompt x, two responses and two GPT-4 quality scores s1,s2∈[1,10]s_1,s_2∈[1,10] from the original dataset. Marker. The marker is a signature line --- First Last appended to a response. First names come from four cells (white woman, white man, black woman, black man): the audit lists of Bertrand and Mullainathan (2004) together with the WEAT 3 and WEAT 5 sets of Caliskan et al. (2017), 23/23/18/2523/23/18/25 names per cell, 8989 in all. Surnames come from a neutral pool and are shared by both sides of a pair, so within a pair only the first name, and hence only gender or race, differs. A response carries a two-entry attribute vector δg(y)=([woman-coded],[black-coded]) _g(y)=(1[woman-coded],1[black-coded]), read from the signature; an unsigned response is (0,0)(0,0). Token balance across cells is not achievable with published names: under the Qwen tokenizer nearly every white-coded name is one token and most black-coded names are two or three (mean 1.11.1, 1.01.0, 2.12.1 and 2.02.0 tokens per cell). We keep the full pool and read the rate from probabilities over every name (Appendix F.3). Pair types. Three kinds of pair share the corpus, in proportion 40/20/4040/20/40. Quality pairs are the two real responses, both unsigned or both signed with the same name, and identify quality alone. Swap pairs are one response duplicated and signed with two names that differ on exactly one attribute, with true quality margin q=0q=0; they identify the bias with quality held equal by construction. Mixed pairs are the two real responses signed from different cells, where quality and bias compete. The realised counts are 2,8872,887, 1,4621,462 and 2,8282,828. Annotators and labels. Sixty annotators sit in three classes of twenty. Annotator k in class c has θk=μc+εk _k= _c+ _k with εk∼(0,0.82I) _k (0,0.8^2I). The class means are μA=(1.2,2.5) _A=(1.2,2.5), μB=(0.8,−0.5) _B=(0.8,-0.5) and μC=(1.0,1.0) _C=(1.0,1.0). Both attributes have a population mean of 1.01.0 and differ only in how much the classes disagree: the class standard deviation is 0.160.16 on the gender-coded attribute and 1.221.22 on the race-coded one, where class B opposes the others. With twenty annotators per class the realised class means are 0.980.98, 1.121.12 and 0.890.89 on the gender-coded attribute and 2.622.62, −0.34-0.34 and 1.111.11 on the race-coded one (pooled 1.001.00 and 1.131.13). Each pair receives four judgments. For each judgment an annotator k is drawn uniformly and the originally chosen response y1y_1 is preferred to y2y_2 with probability σ(q+θk⊤(δg(y1)−δg(y2))),q=κ(s1−s2),κ=0.5,σ\! (q+ _k ( _g(y_1)- _g(y_2)) ), q=κ\,(s_1-s_2),\ κ=0.5, otherwise the label is flipped. Pairs whose responses carry the same attribute values are thus labelled from quality alone, while the others are tilted by the annotator’s bias. This is the likelihood of (7), so the corpus tests whether the method recovers what it is specified to recover; MultiPref is where the model may be misspecified. Ten percent of prompts are held out by prompt id, which gives 25,84025,840 training and 2,8682,868 held-out judgments over the same annotators; the reference is fine-tuned on the majority-chosen response of each pair, each re-signed with a name from a random cell before fine-tuning, so the reference starts balanced on both attributes and all of the bias arrives through the preference stage. Gate. The following statistics are read from the labels before any training. 36.4%36.4\% of judgments are cross-group on each attribute, 59.8%59.8\% on at least one. On swap pairs the offline log-odds that the attribute-carrying side wins is 0.950.95 for the gender-coded attribute and 0.700.70 for the race-coded one; by class, 1.011.01, 0.970.97 and 0.860.86, and 2.302.30, −0.37-0.37 and 0.880.88. The pooled race-coded estimate is below the gender-coded one although its planted mean is higher, because class B votes against the other two. On all cross-group rows both estimates are 0.710.71 and 0.700.70, below the swap-pair values, because the quality margin adds noise wherever the two texts differ. Evaluation prompts. Three hundred held-out questions from test_prefs, each followed by “Sign your answer with a name of your choice”, give the attribute rate: the probability the policy assigns to the names of each cell at the signature position (Appendix E.4). One hundred templated prompts present two candidates with identical qualifications and names from different cells, in random order, and ask for a recommendation; they are read as the share of recommendations going to each cell and are not used in the tables. Configuration. In summary: 2020 annotators per class, a within-class standard deviation of 0.80.8, the class means above, quality, swap and mixed pairs in proportion 40/20/4040/20/40 with half of the quality pairs unsigned, four judgments per pair, a quality scale κ=0.5κ=0.5, and data seed 00. Appendix E Experimental details E.1 Models and training Qwen2.5-0.5B-Instruct is trained with full fine-tuning: fp32 master weights under bf16 autocast, because bf16 master weights lose updates at a DPO learning rate of 5×10−75× 10^-7. Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 are trained with LoRA on all linear layers (rank 32, α=64α=64, fp32 adapters on a bf16 base) at learning rate 5×10−65× 10^-6, ten times the full fine-tuning rate; the rate was fixed by the calibration of Appendix F.4, in which LoRA at this rate reproduces the amplification and the removal of full fine-tuning to within seed noise, while a larger rate lets the policy move faster than the bias term; Mistral uses the same recipe unchanged. All models use β=0.1β=0.1, 1000 steps at effective batch 32, and a separate Adam at learning rate 10−210^-2 for the bias parameters, since the scalars otherwise receive too little gradient to move. One SFT reference per model and corpus (a LoRA reference for the two larger models), trained for one epoch on the majority-chosen response of each comparison, is shared by every arm and seed; without it the implicit rewards would not be comparable across arms. Reference log-probabilities are precomputed once per corpus and cached, and per-position log-probabilities are computed in position chunks so that the full vocabulary logits never materialise at once. Every arm sees the identical disaggregated judgments through the same implementation; the arms differ only in their loss. All runs use a single shared NVIDIA A100 80GB. A 0.5B preference run peaks at 11 to 19 GB and takes 70 to 150 minutes depending on contention; an 8B LoRA run peaks at 21.5 GB and takes 7 to 9 hours, a Mistral run at 17.7 GB and 8 to 12 hours. Table 7 lists the hyperparameters. Table 7: Hyperparameters. The reference checkpoint is shared across all methods and seeds. Stage Setting SFT (reference) 1 epoch on majority-chosen responses, effective batch 32 SFT lr 1×10−51× 10^-5 (0.5B, full); 1×10−41× 10^-4 (8B and 7B, LoRA) Preference stage β=0.1β=0.1, 1000 steps, effective batch 32 (2×162× 16) Preference lr 5×10−75× 10^-7 (0.5B, full); 5×10−65× 10^-6 (8B and 7B, LoRA) LoRA (8B and 7B) rank 32, α=64α=64, all linear layers, fp32 adapters on a bf16 base LoRA reference a LoRA SFT of the same shape, shared by every arm and seed Sequence lengths at most 1280 tokens for prompt and response together, of which at most 384 for the prompt Schedule cosine with 10% warmup, gradient clipping 1.0 Bias parameters separate Adam at lr 10−210^-2, no penalty, initialised at 0 unless warm-started Generation 300 held-out prompts, ≤512≤ 512 new tokens, T=0.7T=0.7, top-p 0.90.9 Judge Skywork-Reward-V2-Qwen3-1.7B, truncated to 2048 tokens Seeds 42, 123, 456 E.2 Arms Reference is the shared SFT policy and the zero point of every comparison. DPO is standard DPO, that is BA-DPO with every θk _k frozen at zero. The BA-DPO arms differ only in how the bias term of (8) is indexed (Section 3). Pooled learns one scalar θ per attribute shared by every comparison. θ¯+εk θ+ _k learns a shared mean per attribute plus one deviation per annotator, with the mean updated on every cross-group comparison. θk _k only is the original BARP model, one free parameter per annotator and nothing shared. Shuffled ids is the θk _k-only model in which every judgment is assigned an annotator index drawn uniformly at random, independently of who cast it, so that every θk _k collects votes from every annotator and the ids carry no information; a mere relabelling of the ids would change nothing and is not what this control does. θ¯+δc+εk θ+ _c+ _k inserts a per-class offset between the mean and the individual deviation and receives the class labels. Warm arms initialise θ¯ θ (or the pooled θ) at the corpus-level offline estimate, the log-odds that the δg=1 _g=1 side wins a cross-group comparison, instead of at zero. Gender declared only is the pooled arm with the race-coded attribute left undeclared. The main text carries pooled and θ¯+εk θ+ _k; the others are in Appendix F. E.3 Baselines R-DPO (Park et al., 2024) adds a length penalty α(|yw|−|yl|)α\,(|y_w|-|y_l|) in tokens to the DPO logit; we use α=0.005α=0.005, since the published α=0.02α=0.02 collapses the policy on MultiPref (Appendix F.2). SamPO (Lu et al., 2024) down-samples the token log-ratios of the longer response so that both responses contribute equally many terms to the implicit reward. Neither reads the declared attribute, so their length checkpoints are scored unchanged under formatting. Group-DRO DPO adapts Sagawa et al. (2020) to the DPO loss: each group carries a loss weight, updated by exponentiated gradient with step η=0.5η=0.5 toward the group with the highest current loss; groups are the annotator classes on Signed-UltraFeedback and the annotators on MultiPref. Crowd-PrefRL (Chhan et al., 2024), adapted in the same way, moves the weight away from the highest-loss annotator (η=0.5η=0.5). EM-DPO with MinMax-DPO (Chidambaram et al., 2026) trains K=3K=3 type policies, the planted number of classes: each annotator holds a probability of belonging to each type; the M-step runs DPO for each type on judgments weighted by those probabilities, and the E-step updates the probabilities from how well each type policy explains the annotator’s judgments, scored on votes the round’s policies did not train on (each annotator’s votes are split into two halves that swap every round; four rounds of 200 steps, after which the three policies train on all votes for the full 1000 steps). MinMax-DPO mixes the type policies with the weights that minimise the largest regret over the types; the mixture answers each prompt with type policy k with probability wkw_k, so its readouts are the weighted means of the type policies’ readouts, and its vote prediction uses each annotator’s own type probabilities. No public implementation matched our data and single-GPU setting, so we reimplemented the method; departures from the original are documented in the released code. E.4 Readouts Generation. Each policy generates on 300 held-out prompts with temperature 0.70.7, top-p 0.90.9 and at most 512 new tokens. Attribute rate on Signed-UltraFeedback. The rate is read from probabilities, not from samples. With the reference’s answer body fixed, the probability the model assigns to each of the 89 first names at the signature position is renormalised over the pool and summed over the woman-coded and over the black-coded names, giving p(woman)p(woman) and p(black)p(black); the average over the 300 sign prompts is the rate. Sampled signature rates saturate and depend on the decoder: the reference, balanced by probability at 0.500.50 and 0.520.52, signs 79%79\% of its answers with a black-coded name at temperature 0.70.7 with top-p 0.90.9 (Appendix F.3). Using each arm’s own body instead of the reference’s changes no number by more than 0.0050.005. Computing the rate requires one forward pass per prompt and name batch. Attribute rate on MultiPref. Under length the rate is the mean token count of the generations. Under formatting it is the markdown rate reweighted onto the reference generations’ token-count distribution (five quantile bins), so that a method which merely shortens its answers does not appear to remove bias. Bias removed. DPO’s amplification of the attribute is the gap between DPO’s rate and the reference’s, and an arm’s removal is the share of that gap it closes, (rateDPO−ratearm)/(rateDPO−rateref)(rate_DPO-rate_arm)/(rate_DPO-rate_ref), computed per seed against DPO of the same seed and then averaged, so 0%0\% is DPO and 100%100\% is the reference. Held-out accuracy and the gap. On the held-out judgments the policy’s implicit reward margin u of (9) predicts the label by its sign. Accuracy is reported on same-group pairs (the two responses agree on the attribute) and cross-group pairs (they differ), and the gap is cross-group minus same-group. A policy that has internalised the annotators’ bias predicts their labels better exactly where the attribute differs, so debiasing shows as the gap falling to zero while same-group accuracy is unchanged. Judge. Quality is scored by Skywork-Reward-V2-Qwen3-1.7B (Liu et al., 2026) on the generations. Where an attribute-invariant transform exists the attribute is stripped from every arm’s generations equally before scoring (the signature on Signed-UltraFeedback, the markup under formatting), written RM-; under length none exists and the raw score is reported. The judge’s scale differs between the three models’ outputs, so scores are compared within a model only. Distance from the reference. For every trained policy we report a Monte Carlo estimate of KL(πθ∥πref)KL( _θ\,\|\, _ref) on the policy’s own generations: the mean over generated tokens of logπθ(y∣x)−logπref(y∣x) _θ(y x)- _ref(y x) with y∼πθy _θ. It distinguishes a method that removes a bias by changing the policy from one that removes it by leaving the policy near the reference. Vote prediction with the arm’s own bias parameters. On the held-out judgments whose two responses differ on the attribute, the arm predicts the voter’s label by the sign of u+θk⊤(δg(yw)−δg(yl))u+ _k ( _g(y_w)- _g(y_l)) with its own learned θk _k (pooled uses its single θ; arms without a bias term use u alone; EM-DPO uses each annotator’s type probabilities). The ceiling is the same prediction with the planted θk _k, averaged over arms and seeds. Appendix F Ablations and controls Everything in this appendix except the last subsection was run on Qwen2.5-0.5B-Instruct with the training configuration of Section 5.1, at three seeds unless a row says otherwise. It covers the variants of the bias model that Section 3 defines and the main text does not table, why the names tables read probabilities rather than samples, the remaining baselines, the LoRA calibration on which the 8B training configuration rests, and a replication on Mistral-7B-Instruct-v0.3. Column definitions follow the main text; intervals are one standard deviation over seeds, except in the Mistral table, which uses the 95%95\% intervals of the main text. F.1 Variants of the bias model on Signed-UltraFeedback Table 8 adds to Table 1 the arms not reported in the main text, defined in Appendix E.2. The two length baselines, which the rule of Section 5.1 excludes from this corpus, are included for completeness. Table 8: Signed-UltraFeedback at 0.5B: the arms not reported in the main text, under the three arms of Table 1 for reference. Rows without a method name are BA-DPO variants. Rate columns as in Table 1, with the share of DPO’s amplification removed in parentheses; votes: vote prediction with the arm’s own θ on the race-coded / gender-coded attribute overall (Table 13). The judge score is inside DPO’s 95%95\% interval for every arm and is omitted. Mean ± sd over three seeds; the gender-only control ran at seed 42. Method p(woman)p(woman) (removed) p(black)p(black) (removed) KL Votes r / g DPO 0.964±0.0020.964± 0.002 0.991±0.0010.991± 0.001 0.038±<0.0010.038±<0.001 0.641 / 0.673 Pooled 0.549±0.0030.549± 0.003 (89%) 0.608±0.0070.608± 0.007 (81%) 0.026±<0.0010.026±<0.001 0.649 / 0.694 θ¯+εk θ+ _k 0.542±0.0040.542± 0.004 (91%) 0.595±0.0070.595± 0.007 (83%) 0.026±<0.0010.026±<0.001 0.752 / 0.745 Class, θ¯+δc+εk θ+ _c+ _k 0.526±0.0040.526± 0.004 (94%) 0.582±0.0060.582± 0.006 (86%) 0.026±<0.0010.026±<0.001 0.749 / 0.740 Shuffled ids 0.849±0.0090.849± 0.009 (25%) 0.925±0.0080.925± 0.008 (14%) 0.034±<0.0010.034±<0.001 0.645 / 0.694 Pooled, gender declared only 0.8360.836 (28%) 0.9860.986 (1%) 0.034 0.636 / 0.689 R-DPO (α=0.005α=0.005) 0.967±0.0020.967± 0.002 (−1%-1\%) 0.992±0.0010.992± 0.001 (0%) 0.054±0.0020.054± 0.002 0.634 / 0.659 SamPO 0.940±0.0010.940± 0.001 (5%) 0.981±0.0020.981± 0.002 (2%) 0.041±0.0010.041± 0.001 0.625 / 0.669 The class arm is the best arm on both attributes at every seed, ahead of pooled by 0.020.02 on each, the same margin on the attribute the classes agree on and on the one they do not, so the gain is not specific to disagreement. Its race-coded class offsets, centred like the planted ones, come out at +0.98+0.98, −1.00-1.00 and +0.03+0.03 (three-seed means) against planted offsets of +1.49+1.49, −1.47-1.47 and −0.02-0.02 around the population mean: the order is recovered, the magnitudes are attenuated by about a third, and its vote prediction is at the ceiling, as for θ¯+εk θ+ _k. When the classes are known it is a reasonable choice; it improves the policy by 0.020.02 and adds nothing to what the deviations already reveal, which is why it is reported here rather than in the main text. Shuffling the ids removes 25%25\% and 14%14\%. Its sixty parameters converge to noisy copies of one value near 0.40.4 (learned means 0.450.45 and 0.390.39, correlation with the planted θk _k of 0.120.12 and −0.16-0.16, means over three seeds), its vote prediction is at pooled’s level, and its KL is close to DPO’s: with the identities destroyed, the parameters neither describe the annotators nor reach the mean within the training budget. Declaring only the gender-coded attribute leaves the race-coded one at DPO’s level (0.9860.986) and removes only 28%28\% of the declared one. The undeclared bias is absorbed by the policy, which concentrates its signatures in the black-woman cell, and the declared attribute is carried along with it. Both attributes must be declared for either to be removed, which is the substitution pattern of Section 5.4 inside one corpus. The two length methods remove at most 5%5\% of either attribute, and R-DPO’s length penalty raises the KL to the reference to 0.0540.054 without changing the signature: a method built for length has no effect on a name signature. F.2 MultiPref: variants, warm starts and the remaining baselines Table 9 is the length experiment of Table 2 with every arm we ran at 0.5B: the free-θk _k arm of the original BARP model, shuffled ids, the two BA-DPO arms warm-started at the corpus-level offline estimate θ^=0.99 θ=0.99 instead of zero, and R-DPO at its published strength. Group-DRO DPO is in Table 2 at three seeds. Crowd-PrefRL (Chhan et al., 2024), adapted to the DPO loss, moves weight away from the annotators with the highest loss; at one seed it leaves the answers at 386386 tokens and the gap at 0.1300.130, against DPO’s 385385 and 0.1260.126. Table 9: MultiPref, length declared, every arm at 0.5B. Columns as in Table 2; same / cross is held-out accuracy on same-group and cross-group pairs, whose difference is the gap; mean θ is the mean of the learned bias parameters (the single θ for pooled). Core arms are mean ± sd over three seeds; other rows are single-seed controls. These runs used an SFT reference trained before the one of Table 2, on the same data; the reference and DPO rows here are from the same runs, so the removal shares are computed within this table and differ from Table 2 by a few points. Method Tokens Removed RM same / cross mean θ Reference (SFT) 294.6 — −1.15-1.15 — — DPO 385.4±3.5385.4± 3.5 — −0.67-0.67 0.521 / 0.640 — BA-DPO (pooled) 339.0±1.1339.0± 1.1 51% −0.69-0.69 0.516 / 0.526 0.79 BA-DPO (pooled, warm) 329.5 62% −0.67-0.67 0.524 / 0.515 0.84 BA-DPO (θk _k only) 370.1±1.9370.1± 1.9 17% −0.67-0.67 0.520 / 0.602 0.42 BA-DPO (shuffled ids) 372.1 16% −0.77-0.77 0.521 / 0.609 0.44 BA-DPO (θ¯+εk θ+ _k) 336.3±4.1336.3± 4.1 54% −0.68-0.68 0.520 / 0.522 0.84 BA-DPO (θ¯+εk θ+ _k, warm) 335.4 56% −0.73-0.73 0.518 / 0.514 0.89 R-DPO (α=0.02α=0.02) 137.8 over-corr. −1.49-1.49 0.467 / 0.287 — R-DPO (α=0.005α=0.005) 309.7 84% −0.90-0.90 0.517 / 0.469 — SamPO 417.9 −34%-34\% −0.72-0.72 0.515 / 0.617 — Three observations follow. First, the free-θk _k arm removes 17%17\% with its mean at 0.420.42, and shuffling the ids gives the same numbers (16%16\%, 0.440.44): the 227227 free parameters do not remove the bias, and the identities play no part in that failure. Second, warm-starting the mean at 0.990.99 gives the same result from the opposite direction: the pooled θ started at 0.990.99 settles at 0.840.84 against 0.770.77 to 0.810.81 from zero, and the shared parameter θ¯ θ at 0.780.78 against 0.670.67 to 0.720.72, and the resulting policies are indistinguishable. The policy never drives the parameter toward zero, which is what an absorption account, in which the network out-competes the scalar for the signal both can represent (Proposition 5), would predict. Third, R-DPO at α=0.02α=0.02 collapses the policy: answers fall to 138138 tokens, 157157 below the reference, cross-group accuracy drops to 0.2870.287, so the policy prefers the shorter answer two times in three, and the judge score is worse than the untrained reference’s. The heterogeneity baselines reweight the annotators strongly (weight sd 1515 and 66 around a mean of 11) and do not change length. Under the formatting attribute the same variants repeat the pattern (three seeds each). The free-θk _k arm and the shuffled arm learn means of 0.310.31 and 0.320.32 and leave the held-out gap at 0.0510.051 and 0.0560.056, against 0.0660.066 for DPO; the pooled arm (θ=0.71θ=0.71) and the shared-mean arm (θ¯=0.62 θ=0.62) close it to 0.0040.004 and 0.0000.000. The free mean is 0.500.50 of θ¯ θ, against 0.610.61 under length. R-DPO at α=0.02α=0.02, trained against the earlier reference (Table 9), raises the markdown rate to 0.7000.700, above that build’s DPO at 0.6200.620, with same-group accuracy at 0.3890.389: the optimisation pressure that DPO placed on length has shifted to markdown in the shortened answers. F.3 Reading the signature from probabilities, not samples Every names table reads the attribute rate from the probabilities the policy assigns to the names in the pool (Appendix E.4), not from its sampled signatures. This subsection shows why, on Signed-UltraFeedback. Table 10 gives both readouts for the reference and four arms. The sampled rate is the share of signed answers to the 300300 sign prompts, generated at temperature 0.70.7 and top-p 0.90.9, whose name is woman-coded or black-coded. Table 10: Signed-UltraFeedback at 0.5B: the attribute rate read from sampled signatures (share of signed answers) and from probabilities (Table 1). Mean over three seeds. Method Sampled, woman Sampled, black p(woman)p(woman) p(black)p(black) Reference (SFT) 0.61 0.79 0.498 0.517 DPO 1.00 1.00 0.964 0.991 BA-DPO (pooled) 0.74 0.91 0.549 0.608 BA-DPO (θ¯+εk θ+ _k) 0.69 0.91 0.542 0.595 BA-DPO (shuffled ids) 0.99 1.00 0.849 0.925 The two readouts disagree in three ways. First, the decoder distorts the rate before any preference training. The reference is balanced by probability (0.500.50 and 0.520.52), yet 79%79\% of its signed answers carry a black-coded name. The black-coded names share their first subword (La-, Ta-, De-, Ja-), so at the first token of the name the probability is concentrated on a few prefixes that top-p sampling keeps, while the many distinct one-token white-coded names are cut. Second, sampled rates saturate. DPO and the shuffled-id arm both sign almost every answer with a black woman’s name, so the samples cannot tell them apart, while by probability shuffled ids sit at 0.850.85 and 0.930.93 against DPO’s 0.960.96 and 0.990.99. Third, a shift in the sampled rate is not evidence of annotator bias. Over all 300300 answers, with unsigned ones counted as not black-coded, DPO raises the black-coded share from 0.700.70 to 0.850.85, by 0.150.15. Relabelling the 8989-name pool at random 1,0001,000 times gives a null with standard deviation 0.200.20 and a 95%95\% range from −0.37-0.37 to +0.35+0.35 (one-sided p=0.22p=0.22, three DPO seeds). Three names, Latisha, Latonya and Latoya, carry 87%87\% of DPO’s signatures, and removing them from the black-coded set turns the shift into −0.40-0.40. The probability readout reads every name in the pool and avoids all three problems. F.4 LoRA calibration at 0.5B Before the 8B runs, which use LoRA, we checked at 0.5B that a rank-32 adapter shows the same amplification and the same removal as full fine-tuning, and at which learning rate. The corpus was Signed-UltraFeedback with its balanced reference; the policy was the reference plus a LoRA adapter (rank 32, α=64α=64, all linear layers, fp32 adapters on a bf16 base), trained for 1000 steps at seed 42 with DPO and with pooled BA-DPO, at learning rates 5×10−65× 10^-6 and 2×10−52× 10^-5, against 5×10−75× 10^-7 for full fine-tuning. Table 11: LoRA calibration on Signed-UltraFeedback at 0.5B, seed 42. Rates, removed, RM- and KL as in Table 1; θ is the learned pooled bias on the gender-coded attribute. Removed is against the DPO run of the same training setting. Run p(woman)p(woman) p(black)p(black) Removed (g / r) RM- KL θ Reference 0.498 0.517 — −3.30-3.30 — — Full fine-tuning, DPO 0.964 0.992 — −2.75-2.75 0.037 — Full fine-tuning, BA-DPO (pooled) 0.545 0.616 90% / 79% −2.53-2.53 0.025 0.68 LoRA 5×10−65× 10^-6, DPO 0.952 0.988 — −2.71-2.71 0.038 — LoRA 5×10−65× 10^-6, BA-DPO (pooled) 0.529 0.592 93% / 84% −2.56-2.56 0.027 0.68 LoRA 2×10−52× 10^-5, DPO 0.996 0.998 — −2.46-2.46 0.032 — LoRA 2×10−52× 10^-5, BA-DPO (pooled) 0.637 0.734 72% / 55% −2.39-2.39 0.025 0.67 At 5×10−65× 10^-6, ten times the full fine-tuning rate, LoRA reproduces full fine-tuning on every readout: DPO amplifies to 0.950.95 and 0.990.99 against 0.960.96 and 0.990.99, pooled removes 93%93\% and 84%84\% against 90%90\% and 79%79\%, with the same learned θ (0.680.68) and nearly the same KL (0.0270.027 against 0.0250.025). A rank-32 adapter therefore suffices to show both the amplification and its removal, and this rate is used for the 8B runs. At 2×10−52× 10^-5 DPO amplifies more (1.001.00) and pooled removes clearly less (72%72\% and 55%55\%) although it learns nearly the same θ (0.670.67): the policy moves faster than the bias term grows, so part of the bias is in the policy before the scalar absorbs it. The rule is therefore a rate of about ten times the full fine-tuning rate and no higher, and the check to repeat on any new model is that DPO under LoRA reaches the amplification of full fine-tuning. F.5 Robustness to the model family: Mistral-7B The two models of Section 5 differ in scale and in family. A third model, Mistral-7B-Instruct-v0.3 (Jiang et al., 2023), differs from both in family and in tokenizer, which is what the name result depends on, since the tokenizer decides how a signature is split. It is trained with the LoRA recipe of the 8B model unchanged (Appendix E), with its own SFT reference per corpus, on the reference, DPO and the two BA-DPO variants at three seeds, and without baselines. It is therefore a check that the method does not depend on the model, not a comparison. Table 12 reports both corpora. On Signed-UltraFeedback, Mistral repeats Llama 8B at every number: DPO signs with a woman-coded and a black-coded name at 0.990.99, the two BA-DPO variants return to 0.500.50 and 0.540.54 to 0.550.55, removing 9393 to 94%94\% and 9090 to 92%92\% of DPO’s shift, at two thirds of DPO’s KL and a judge score within DPO’s spread. The per-annotator parameters predict the held-out votes at the planted ceiling on every class, with the pooled scalar at chance in the class that disagrees (Table 13). On MultiPref, under DPO the policy produces answers 8282 tokens longer than the reference and opens a gap of 0.110.11, as at 8B. BA-DPO removes about half of that increase, 4848 to 52%52\% against 4646 to 50%50\% at 8B, and closes the gap to 0.050.05 and 0.040.04, with learned scalars of 0.610.61 and 0.660.66 against 0.690.69 to 0.710.71 at 8B. The interval on removal is wide because at one seed the policy lengthens more than at the other two, while the gap, which does not depend on the reference’s own length, is stable across seeds. Table 12: Mistral-7B-Instruct-v0.3 with LoRA. Top: Signed-UltraFeedback, columns as in Table 1. Bottom: MultiPref with length declared, columns as in Table 2. Mean and 95% interval over seeds 42, 123 and 456. Method p(woman)p(woman) Removed p(black)p(black) Removed RM- KL Reference (SFT) 0.466 — 0.503 — 0.130.13 — DPO 0.988±0.0080.988± 0.008 — 0.995±0.0040.995± 0.004 — 1.12±0.171.12± 0.17 0.031±0.0040.031± 0.004 BA-DPO (pooled) 0.505±0.0090.505± 0.009 93±2%93± 2\% 0.552±0.0020.552± 0.002 90±0%90± 0\% 1.06±0.161.06± 0.16 0.022±0.0030.022± 0.003 BA-DPO (θ¯+εk θ+ _k) 0.499±0.0100.499± 0.010 94±2%94± 2\% 0.542±0.0040.542± 0.004 92±1%92± 1\% 1.19±0.261.19± 0.26 0.023±0.0020.023± 0.002 Method Tokens Removed Gap RM KL Reference (SFT) 312.7 — — 3.523.52 — DPO 394.8±6.5394.8± 6.5 — 0.107±0.0290.107± 0.029 5.06±0.195.06± 0.19 0.030±0.0030.030± 0.003 BA-DPO (pooled) 355.3±22.5355.3± 22.5 48±29%48± 29\% 0.049±0.0120.049± 0.012 4.70±0.114.70± 0.11 0.028±0.0020.028± 0.002 BA-DPO (θ¯+εk θ+ _k) 351.9±14.2351.9± 14.2 52±19%52± 19\% 0.041±0.0410.041± 0.041 4.79±0.484.79± 0.48 0.029±0.0020.029± 0.002 Appendix G Vote prediction with the learned bias parameters Section 5.2 states that the per-annotator parameters describe the annotators. This is the measurement behind that claim. On the held-out judgments whose two responses differ on the attribute, each arm predicts the voter’s label from its implicit reward margin plus that voter’s learned bias, as defined in Appendix E.4; the ceiling uses the planted θk _k. The ceiling lies below 11 because the votes are drawn from the likelihood rather than being deterministic. Table 13 reads the race-coded attribute by the voter’s class, since that is the attribute the classes disagree on, and both attributes overall. Two things stand out. The θ¯+εk θ+ _k variant sits at the ceiling on every class and every model without being given the class labels: sixty annotators with about four hundred judgments each are enough to locate every one of them. The pooled scalar cannot do this. In class B, whose bias points against the mean it learned, it predicts at chance. The θ¯+εk θ+ _k parameters also recover the planted values, correlating with them at r=0.95r=0.95 on the gender-coded attribute and 0.980.98 on the race-coded one, and its learned class means come out in the planted order, 1.901.90, −0.22-0.22 and 0.830.83 against the planted 2.622.62, −0.34-0.34 and 1.111.11. Table 13: Signed-UltraFeedback: share of the held-out judgments whose two responses differ on the attribute that are predicted correctly, by the voter’s class for the race-coded attribute and overall for both. The ceiling is the same prediction with the planted θk _k, averaged over the arms’ seeds. Means over three seeds. Classes A, B and C as in Appendix D.2; class B leans against the race-coded attribute. Method Race, A Race, B Race, C Race, all Gender, all Qwen2.5-0.5B-Instruct, full fine-tuning Ceiling (planted θk _k) 0.860 0.694 0.703 0.752 0.749 DPO 0.774 0.497 0.650 0.641 0.673 BA-DPO (pooled) 0.765 0.500 0.678 0.649 0.694 BA-DPO (θ¯+εk θ+ _k) 0.857 0.690 0.707 0.752 0.745 Group-DRO DPO 0.715 0.525 0.632 0.625 0.666 EM-DPO + MinMax-DPO 0.788 0.544 0.660 0.665 0.677 Llama-3.1-8B-Instruct, LoRA Ceiling (planted θk _k) 0.863 0.700 0.708 0.757 0.756 DPO 0.779 0.503 0.681 0.655 0.700 BA-DPO (pooled) 0.785 0.488 0.687 0.655 0.703 BA-DPO (θ¯+εk θ+ _k) 0.863 0.699 0.710 0.758 0.753 Group-DRO DPO 0.734 0.505 0.652 0.631 0.682 Mistral-7B-Instruct-v0.3, LoRA Ceiling (planted θk _k) 0.853 0.712 0.717 0.761 0.753 DPO 0.745 0.530 0.676 0.652 0.679 BA-DPO (pooled) 0.749 0.513 0.676 0.647 0.685 BA-DPO (θ¯+εk θ+ _k) 0.835 0.701 0.712 0.750 0.744