Paper deep dive
Debiasing Reward Models by Representation Learning with Guarantees
Ignavier Ng, Patrick Blöbaum, Siddharth Bhandari, Kun Zhang, Shiva Kasiviswanathan
Models: BERT, GPT-2
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 5:57:53 PM
Summary
The paper proposes a principled framework to mitigate spurious correlations in reward models used for RLHF. By formulating the data-generating process as a combination of spurious and non-spurious latent variables, the authors provide theoretical guarantees for identifying non-spurious latent variables both with and without access to a surrogate for spurious features. They introduce a variational inference-based method to recover these variables, enabling the training of more robust and bias-free reward models.
Entities (5)
Relation Signals (3)
Reward Model → exploits → Spurious Correlation
confidence 95% · In practice, these models often exploit spurious correlations
Non-spurious Latent Variables → improves → Reward Model
confidence 94% · leverages them to train reward models... yields more robust reward models
Variational Inference → recovers → Non-spurious Latent Variables
confidence 93% · practical method that uses variational inference to recover these variables
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent alignment techniques, such as reinforcement learning from human feedback, have been widely adopted to align large language models with human preferences by learning and leveraging reward models. In practice, these models often exploit spurious correlations, involving, e.g., response length, discrimination, sycophancy, and conceptual bias, which is a problem that has received increasing attention. In this work, we propose a principled framework that mitigates these biases in reward models while preserving the underlying factors that reflect intended preferences. We first provide a formulation of the data-generating process, assuming that the observed data (e.g., text) is generated from both spurious and non-spurious latent variables. We show that, interestingly, these non-spurious latent variables can be theoretically identified from data, regardless of whether a surrogate for the spurious latent variables is available. This further inspires a practical method that uses variational inference to recover these variables and leverages them to train reward models. Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates spurious correlation issues and yields more robust reward models.
Tags
Links
- Source: https://arxiv.org/abs/2510.23751
- Canonical: https://arxiv.org/abs/2510.23751
Trouble viewing inline? Open PDF directly →
Full Text
88,970 characters extracted from source content.
Expand or collapse full text
Debiasing Reward Models by Representation Learning with Guarantees Ignavier Ng1 Work done during an internship at Amazon. Patrick Blöbaum2 Siddharth Bhandari2 Kun Zhang1,3 Shiva Kasiviswanathan2 ( 1Carnegie Mellon University 2Amazon 3Mohamed bin Zayed University of Artificial Intelligence ) Abstract Recent alignment techniques, such as reinforcement learning from human feedback, have been widely adopted to align large language models with human preferences by learning and leveraging reward models. In practice, these models often exploit spurious correlations, involving, e.g., response length, discrimination, sycophancy, and conceptual bias, which is a problem that has received increasing attention. In this work, we propose a principled framework that mitigates these biases in reward models while preserving the underlying factors that reflect intended preferences. We first provide a formulation of the data-generating process, assuming that the observed data (e.g., text) is generated from both spurious and non-spurious latent variables. We show that, interestingly, these non-spurious latent variables can be theoretically identified from data, regardless of whether a surrogate for the spurious latent variables is available. This further inspires a practical method that uses variational inference to recover these variables and leverages them to train reward models. Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates spurious correlation issues and yields more robust reward models. 1 Introduction Large language models (LLMs) have shown remarkable abilities across a wide array of tasks, from open-domain dialogue and creative writing to coding assistance and scientific reasoning (Minaee et al., 2024). However, their raw generations often diverge from human expectations, producing outputs that may be unhelpful, unsafe, or misaligned with user intent. To address this, reinforcement learning from human feedback (RLHF) has emerged as the predominant paradigm for aligning LLMs with human preferences (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a). By training reward models on human feedback and optimizing policies against these models, RLHF has enabled models such as InstructGPT and ChatGPT to exhibit more helpful, safe, and preference-consistent behavior. This line of work has made RLHF a crucial component in the current generation of aligned language models. Despite its success, the RLHF pipeline is vulnerable to reward model misgeneralization arising from spurious correlations in the preference data. This vulnerability can lead to reward hacking, where the policy learns to exploit flaws in the reward model to achieve high scores without genuinely satisfying human intent (McMilin, 2022; Casper et al., 2023). These failures manifest as a variety of well-documented biases. For example, reward models may associate longer responses with higher quality, length bias (Zheng et al., 2023). Similarly, they may exhibit sycophancy, rewarding responses that agree with a user’s stated views regardless of correctness (Perez et al., 2023; Sharma et al., 2024). Other failure modes include concept bias, where models learn unintended shortcuts based on superficial cues (e.g., the presence of certain keywords) (Zhou et al., 2024), and discrimination bias (Tamkin et al., 2023; Chen et al., 2025), where models amplify harmful societal biases present in the annotation data. These issues compromise both robustness and fairness, raising fundamental questions about how to build alignment methods that are resilient to such failure modes. The challenges of mitigating reward hacking and improving model performance have received considerable attention. One line of works focuses on RLHF from synthetic preferences, where feedback is generated by more capable models to improve and scale the alignment process, as explored in recent works on AI-generated feedbacks (Bai et al., 2022b; Lee et al., 2024; Yang et al., 2023) and self-improving reward models (Pace et al., 2024; Yuan et al., 2024). A second direction develops targeted solutions for specific, observable biases, including constructing length-balanced preference datasets, applying explicit length penalties during training (Singhal et al., 2024), and designing model architectures that decouple quality scores from length signals (Chen et al., 2024). Another line of approaches leverages more general techniques, such as regularization to enforce invariance (Wang et al., 2025) or data augmentation to separate quality from style (Srivastava et al., 2025). While often effective in practice, these methods tend to be ad-hoc and may lack formal theoretical guarantees for bias mitigation (see Section A.1 for an extended discussion). In this work, we propose a principled framework to mitigate biases in reward models while preserving the underlying factors that reflect intended preferences. We assume that the observed data (e.g., text, images) is generated from both spurious and non-spurious latent variables. Rather than training the reward model directly on the observed data, our key idea is to first identify the non-spurious latent variables that capture true human preferences, and then train the model on these variables. Specifically, our contributions are: • When the surrogate for spurious latent variables is available, we show that the subspace of the non-spurious latent variables can be recovered under mild assumptions (Theorem 1). • When no surrogate is available, we show that the subspace of the non-spurious latent variables remains identifiable under conditions such as sufficient diversity in the variables on which the human annotators rely (Theorem 2). • Building on these theoretical results, we develop a practical method that uses variational inference to recover these non-spurious latent variables and leverages them to train reward models (Section 4). • We conduct experiments on both synthetic and text datasets to demonstrate that our approach effectively mitigates spurious correlations and yields more robust reward models (Section 5). 2 Preliminaries Reinforcement Learning from Human Feedback. A standard procedure for aligning LLMs with human preferences involves a two-stage process: supervised fine-tuning (SFT) followed by RLHF (Ouyang et al., 2022). The core of RLHF involves training a reward model on a dataset of human preferences, where annotators choose the better of two model-generated responses. A common approach is to model the human preference distribution with a Bradley-Terry model (Bradley and Terry, 1952) and then train the reward model by minimizing a negative log-likelihood loss such that it assigns a higher score to the preferred response. The supervised fine-tuned model is then refined using reinforcement learning algorithm, such as proximal policy optimization (Schulman et al., 2017), to maximize the scores given by this reward model, effectively treating it as a proxy for human judgment. However, a significant challenge is that the reward model can exploit spurious correlations in the data; see Section 1 for a discussion of different biases. Causal Representation Learning. Causal representation learning aims to uncover latent variables and their causal relations from low-level observations such as images or text (Schölkopf et al., 2021). The task is notoriously difficult, because the latent variables generally cannot be identified even when the they are independent (i.e., the nonlinear independent component analysis problem) (Hyvärinen and Pajunen, 1999; Hyvärinen et al., 2023). To achieve identifiability, existing works therefore relies on further assumptions, such as access to multiple distributions (Squires et al., 2023; von Kügelgen et al., 2023; Zhang et al., 2024) or multiple views (Yao et al., 2024; Xu et al., 2024). See Appendix A.2 for a further discussion. 3 Identifiability Theory We present our formulation of the data-generating process and identifiability results for the non-spurious latent variables, both with (Section 3.1) and without (Section 3.2) access to a surrogate for the spurious variables. By identifiability, we mean that these latent variables can be uniquely determined from the observed data (up to certain indeterminacies). Establishing identifiability is crucial; without it, multiple latent representations could fit the same observations, and we cannot guarantee that the learned representations align with the desired non-spurious latent variables. 3.1 With Access to Surrogate for Spurious Latent Variables Formulation. We begin by formalizing the generative process of the observed data. We assume that the observed data (e.g., text, images) T∈T is generated by an invertible mixing function g from a set of n latent variables Z∈Z . Moreover, the latent variables Z are assumed to follow a structural equation model (Pearl, 2009; Spirtes et al., 2001). Formally, we have X=g(Z)andZi=fi(PA(Zi;),ϵi),i∈[n],X=g(Z) Z_i=f_i( PA(Z_i;G), _i),\,\,i∈[n], (1) where ϵi _i’s denote the noise terms, G represents the directed acyclic graph (DAG) over Z, and PA(Zi;) PA(Z_i;G) denotes the parents of ZiZ_i in G. In general, T may represent either images or text. In the context of RLHF for LLMs, T specifically denotes a prompt–response pair, consisting of a prompt X and a response Y. Let S∈S denote a surrogate that represents the spurious attribute of interest, such as response length or presence of certain concepts. We partition the latent variables Z into two disjoint sets based on their relationship with S: • Spurious latent variables ZS=(ZS,i)i=1nS∈SZ_S=(Z_S,i)_i=1^n_S _S that are dependent of S. • Bias-free (non-spurious) latent variables ZC=(ZC,i)i=1nC∈CZ_C=(Z_C,i)_i=1^n_C _C that are independent of S. A graphical description of the generative process is provided in Fig. 1. In this work, we use the term “bias-free latent variables” to refer specifically to the “bias-free, maximal latent variables” or, equivalently, the “non-spurious latent variables”, which means the largest set of latent variables that are independent of S. Also note that here, “bias-free” is not meant in the statistical sense (e.g., an unbiased estimator θ θ such that E[θ^]=θE[ θ]=θ); rather, it is used to denote the variables that do not contain spurious information relative to S. Our goal is to learn a bias-free representation that discards spurious information relative to S while preserving all information relevant to human preferences. To achieve so, we aim to learn a mapping from the observed data T to a representation Z^C Z_C that is an invertible transformation of the underlying bias-free latent variables ZCZ_C. In this case, we say that ZCZ_C is subspace identifiable. This construction ensures that Z^C Z_C captures exactly the information in ZCZ_C, thereby retaining useful factors while eliminating spurious correlations relative to S. Equivalently, Z^C Z_C is counterfactually invariant: it would remain unchanged under a hypothetical intervention on S. A reward model trained on this representation, R^=r^(Z^C) R= r( Z_C), is therefore bias-free: by construction, does not contain spurious information relative to S and robust to its variations. This counterfactual invariance grounds the reward in stable causal factors, improving generalization and mitigating reward hacking. ZSZ_S ZCZ_C S T Figure 1: The generative process considered in our work. The observed variables T are generated by two sets of latent variables ZSZ_S and ZCZ_C, where ZSZ_S are influenced by the spurious variable S. Shaded nodes denote observed variables, while the dashed arrow from ZCZ_C to ZSZ_S indicates a potential causal relationship. We address the setting where S is unknown in Section 3.2. Theory. As explained, our goal is to recover the bias-free latent variables ZCZ_C up to invertible transformation. This allows us to train a reward model that preserves all essential information while being counterfactually invariant to the spurious feature. However, the identifiability of latent variables is a fundamental challenge; without any assumption, they are generally unidentifiable (Hyvärinen and Pajunen, 1999; Hyvärinen et al., 2023). Fortunately, recent works in causal representation learning (Schölkopf et al., 2021) have investigated conditions for recovering such latent variables (see Section A.2 for further discussion). Building on these ideas, we provide a theoretical result showing how the subspace of the bias-free latent variables can be recovered. It is worth noting that we do not aim to identify the spurious latent variables ZSZ_S because (1) our reward modeling approach does not rely on them, and (2) doing so would require stronger assumptions. Theorem 1 (Subspace Identifiability of ZCZ_C: Surrogate S Known Case). Consider the generative process in Eq. 1. Suppose that the following assumption hold: • A1 (Sufficient variability w.r.t. surrogate): For any set AZ⊆A_Z with nonzero measure that cannot be expressed as BZC×SB_Z_C×Z_S for some BZC⊂CB_Z_C _C, there exist S1,S2∈S_1,S_2 such that ∫Z∈AZp(Z∣S=S1)Z≠∫Z∈AZp(Z∣S=S2)Z. _Z∈ A_Zp(Z S=S_1)dZ≠ _Z∈ A_Zp(Z S=S_2)dZ. By modeling the same generative process, ZCZ_C is subspace identifiable. The proof is provided in Appendix B. In summary, it establishes that: (1) the probability measure of any pre-image of the estimated Z^C Z_C must remain constant across all values of S, and (2) by contradiction, if the estimated Z^C Z_C depended on ZSZ_S, one could construct a pre-image set that violates this invariance under Assumption A1. The argument builds on Kong et al. (2022)111While our goal is to recover bias-free latent variables (corresponding to their invariant ones) for training reward models, Kong et al. (2022) addressed a different problem of learning invariant and changing latent variables for the purpose of domain adaptation. but incorporates several important generalizations. In particular, we relax the independence assumption on the latent variables. Our result allows for an arbitrary causal structure among all latent variables (i.e., ZCZ_C and ZSZ_S), as long as ZCZ_C are not independent of the surrogate S. Second, our result only requires at least two values of S and is more straightforward, while Kong et al. (2022) require at least 2nS+12n_S+1 values of S to first identify each independent spurious latent variable ZS,iZ_S,i individually, before recovering the subspace of bias-free latent variables ZCZ_C. This weaker condition requires an extended proof strategy, because we cannot rely on individually recovering the spurious latent variables to help disentangle ZCZ_C, but instead directly characterize the their subspace. Assumption A1 ensures that the S has a rich enough influence on the latent space to enable disentanglement of ZCZ_C. Intuitively, it requires that the probability of any event depending on the spurious latent variables ZSZ_S must be sensitive to changes in S. This type of assumption, also used by Kong et al. (2022), is common in causal representation learning, where sufficient variability across different data distributions are often leveraged to enable the identification of the latent variables (Schölkopf et al., 2021; Hyvärinen et al., 2023). Our result indicates that we can recover the bias-free latent variables ZCZ_C up to invertible transformation. The estimated variables Z^C Z_C therefore preserves all and only the information from ZCZ_C, effectively isolating it from influences of spurious information relative to surrogate S. A reward model trained on this bias-free representation will be, by construction, counterfactually invariant to S and thus resilient to spurious correlations. We describe our practical algorithm for estimating Z^C Z_C and training the reward model in Section 4. 3.2 Extension to the Case without Access to Surrogate for Spurious Latent Variables We now relax our previous assumption and address the more challenging setting where one does not have access to a surrogate for the spurious variables, making the bias-free latent variables generally unidentifiable without further assumptions. To overcome this, we introduce a formulation that leverages multiple reward functions (e.g., from different human labelers) and assume that each function depends on a sparse subset of the latent variables. This sparsity structure provides useful information for disentanglement, allowing us to develop an approach for recovering the bias-free latent variables even without access to a surrogate. Similar to the idea described in Section 3.1, training a reward model on these bias-free representations yields a model that is resilient to spurious correlations. Formulation. We consider a setting with K distinct human labelers, where the reward from each is denoted by the random variable RkR_k. We assume that the underlying reward function rkr_k for each labeler depends on a specific subset of latent variables ZAk⊆Z_A_k Z, representing the factors relevant to that individual, where Ak∈A_k . This generative process is modeled as: Z=h(T)andRk=rk(ZAk)+εk,k∈[K],Z=h(T) R_k=r_k(Z_A_k)+ _k, k∈[K], (2) where εk _k is an independent noise term and h is a function that maps the observed variables T∈T (e.g., text) to the n-dimensional latent variables Z∈Z . Different from the formulation in Section 3.1, which assumes an invertible generative function T=g(Z)T=g(Z), we now model the latent representations directly as Z=h(T)Z=h(T) and do not require h to be invertible. An example of the formulation is provided in Fig. 2. Our primary goal is to identify the set of latent variables shared by all human labelers, given by the intersection ⋂k=1KZAk _k=1^KZ_A_k. Under mild assumptions, we show that the shared latent variables correspond to the true bias-free latent variables ZCZ_C, which we discuss next. Z3Z_3Z2Z_2Z1Z_1Z4Z_4Z5Z_5R1R_1R2R_2 Figure 2: An example of the formulation with unknown spurious features, where the reward corresponding to first human labeler R1R_1 depends on ZA1=Z1,Z2,Z3,Z4Z_A_1=\Z_1,Z_2,Z_3,Z_4\, while that of second human R2R_2 depends on ZA2=Z3,Z4,Z5Z_A_2=\Z_3,Z_4,Z_5\. The shared latent representations are ⋂k=12ZAk=Z3,Z4 _k=1^2Z_A_k=\Z_3,Z_4\. Edges among variables ZiZ_i’s indicate that they can be dependent. Theory. We now show that the latent variables shared by different human labelers can be recovered up to an invertible transformation. It is worth noting that learning such shared representations contributes to the AI alignment goal of learning common human values from diverse feedback (Gabriel, 2020; Hendrycks et al., 2021), which is an active area of research. Theorem 2 (Subspace Identifiability of ZCZ_C: Surrogate S Unknown Case). Consider the data generating process in Eq. 2, where rkr_k is assumed to be linear, i.e., rk(ZAk)=W(k)ZAk,W(k)∈r_k(Z_A_k)=W^(k)Z_A_k,W^(k) . Suppose that the following assumptions hold: • A2 (Identifiability of ηk _k from p(Rk;ηk)p(R_k; _k)): KL(p(Rk;ηk)∥p(Rk;η~k))=0KL(p(R_k; _k) p(R_k; η_k))=0 implies ηk=η~k _k= η_k. • A3 (Sufficient representation variability): There exist n linearly independent points Z(1),…,Z(n)∈Z^(1),…,Z^(n) . • A4 (Sufficient task variability): There exist n linearly independent vectors W(1),…,W(n)∈W^(1),…,W^(n) . • A5 (Intra-support sufficient task variability): For all A∈A and v∈ℝ|A|\0v ^|A| \0\, we have ℙW∣AW∈ℝ1×n∣W⋅,Av==0.P_W A \W ^1× n W_·,Av=0 \=0. Then, the shared latent variables ⋂k=1KZAk _k=1^KZ_A_k is subspace identifiable almost surely. The proof can be found in Appendix C. In short, it first shows that the learned latent representation is an invertible transformation of the true latent variables, and then carefully analyzes the support of the corresponding linear matrix. The argument is inspired by the multi-task learning framework of Lachapelle et al. (2023), where we consider each human labeler as a distinct task. However, our goal differs significantly: while their work aims to identify all latent variables, we focus only on recovering the subspace of shared latent variables. This allows us to rely on weaker assumptions and further develop a proof technique to learn the shared representations, which includes incorporating a sparsity constraint on the shared subspace. The assumptions are adapted from Lachapelle et al. (2023). Specifically, Assumption A2 is a standard regularity condition on the distribution family that holds for minimal exponential families (e.g., a Gaussian). Assumption A3 requires that the latent variables vary sufficiently so that their image is not confined to a proper subspace. Assumption A4 requires the reward function to exhibit sufficient variation, while Assumption A5 ensures that ℙW∣AP_W A has support that spans the entire space and is not confined to any lower-dimensional subspace. Collectively, these assumptions impose sufficient variability to span the corresponding spaces, which facilitates disentanglement. Similar sufficient variability assumptions are commonly leveraged in causal representation learning for establishing identifiability (see Section A.2 for a discussion). Leveraging Theorem 2, the following corollary establishes the conditions for identifying the subspace of bias-free latent variables, proved in Appendix D. Corollary 1 (Subspace Identifiability of ZCZ_C). Consider the data generating process in Eq. 2, where rkr_k is assumed to be linear, i.e., rk(ZAk)=W(k)ZAk,W(k)∈r_k(Z_A_k)=W^(k)Z_A_k,W^(k) . Suppose that Assumptions A2, A3, A4, and A5 from Theorem 2 hold. Further assume that (1) every human’s reward function depends on all bias-free latent variables (i.e., ZC⊆ZAkZ_C Z_A_k for all k), and (2) there is no spurious latent variable that every human shares (i.e., ⋂k=1K(ZAk∖ZC)=∅ _k=1^K(Z_A_k Z_C)= ). Then, ZCZ_C is subspace identifiable almost surely. This corollary indicates that the true bias-free latent variables can be identified even without access to a surrogate for the spurious variables, provided there is sufficient diversity among the human labelers. The two assumptions on the humans are necessary for identifiability, as any latent variable ignored by humans cannot be recovered in general, while any spurious factor unanimously considered would be indistinguishable from a true causal one. Ultimately, this enables us to learn a bias-free representation Z^C Z_C for training a more robust reward model. Note that once the bias-free representation is identified, training the reward model proceeds in the same way as in the surrogate case. In the following, we focus on the surrogate case for the description of the overall approach and the experiments. 4 Reward Modeling Building on our theory, we introduce a practical, two-stage approach to learn a reward model resilient to spurious correlations. The first stage aims to estimate the bias-free latent variables Z^C Z_C, which is guaranteed to be an invertible transformation of the true ZCZ_C. In the second stage, we train the reward model on these bias-free representations. The details of each stage are provided below, with an overview depicted in Fig. 3. Stage 1: Estimating bias-free latents Z^C Z_C Stage 2: Learning reward models Raw Text T VAE Encoder Latents (Z^S,Z^C Z_S, Z_C) VAE Decoder Reconstructed Text T T Z^C Z_C Reward Model (MLP) Extract Z^C Z_C Figure 3: Overview of the proposed reward modeling approach. Stage 1 involves a customized VAE. Stage 1: Estimating Bias-Free Latent Variables. Our theoretical result in Section 3.1 requires matching the data distribution, i.e., p(T,S)=p(T^,S)p(T,S)=p( T,S), which can be achieved via maximum likelihood estimation. Following previous work in causal representation learning (Khemakhem et al., 2020; Kong et al., 2022; Zhang et al., 2024), we use a customized variational autoencoder (VAE) (Kingma and Welling, 2014) for this purpose, although other methods like normalizing flows could also be adopted (Rezende and Mohamed, 2015; Dinh et al., 2017). Our VAE is trained by minimizing the following loss function: ℒ= = −q(Z^∣T)[logp(T∣Z^)]⏟Reconstruction+βKL(q(Z^∣T)∥p(Z^∣S))⏟KL divergence+λq(Z^∣T)[HSIC(Z^C,S)]⏟Independence regularizer, -E_q( Z T)[ p(T Z)]_Reconstruction+β KL(q( Z T)\|p( Z S))_KL divergence+λ E_q( Z T)[HSIC( Z_C,S)]_Independence regularizer, where β and λ are hyperparameters. The components of this loss function are as follows: • The reconstruction term encourages the decoder p(T∣Z^)p(T Z) to accurately reconstruct the data T from the latent representation Z Z. For text data, it corresponds to the cross-entropy loss between the input sequence and the output of the decoder. • The KL divergence term regularizes the approximate posterior q(Z^∣T)q( Z T), learned by the encoder, to match the prior distribution p(Z^∣S)p( Z S). • The independence regularization term uses the Hilbert-Schmidt independence criterion (HSIC) (Gretton et al., 2007) to enforce the desired independence between the learned bias-free latent variables Z^C Z_C and the surrogate S. Theoretically, the first two terms, which constitute an evidence lower bound (ELBO), are sufficient to match the data distribution in the large sample limit given a sufficiently expressive model (Khemakhem et al., 2020; Lachapelle et al., 2024). In practice, when the sample size is limited, we find that incorporating the HSIC regularizer helps improve the performance. To leverage pre-trained models and reduce the number of training parameters, we adapt the strategies from the LangVAE framework (Carvalho et al., 2025) for both encoder and decoder. Our encoder is based on a pre-trained BERT model (Devlin et al., 2019), followed by two multilayer perceptrons (MLPs) that output the means and diagonal covariances of the bias-free (Z^C Z_C) and spurious (Z^S Z_S) latent variables. Similarly, our decoder consists of a trainable MLP that projects the latent variables into the input embedding space of a pre-trained GPT-2 model (Radford et al., 2019), which then generates the final output. Note that our method remains compatible with other pre-trained models. To flexibly model the prior distribution p(Z^∣S)p( Z S), we employ a non-parametric approach using normalizing flows (Papamakarios et al., 2021), inspired by Zhang et al. (2024). Since the labeling of latent variables is generally not identifiable, we begin by assuming a pre-specified causal ordering over the latent variables Z Z, such that the bias-free latent variables Z^C Z_C are not descendants of the spurious latent variables Z^S Z_S, following the assumption in Eq. 1. Since the goal is not to learn the latent DAG G, we simplify the estimation procedure by using a fully-connected DAG G that respects this ordering. Each conditional distribution p(Z^i∣PA(Z^i;^))p( Z_i PA( Z_i; G)) is then modeled as a normalizing flow, which transforms a base noise variable ϵ^i∼p(ϵ^i) ε_i p( ε_i) into ZiZ_i. The parameters of this invertible transformation are themselves generated by a MLP that takes the parent variables PA(Z^i;^) PA( Z_i; G) as input. The resulting log density is given by the change of variables formula: logp(Z^i∣PA(Z^i;^))=logp(ϵ^i)+log|det∂τi∂Z^i|, p( Z_i PA( Z_i; G))= p( ε_i)+ | ∂ _i∂ Z_i |, where ϵ^i=τi(Z^i;MLPi(PA(Z^i;^))) ε_i= _i( Z_i;MLP_i( PA( Z_i; G))) is the output of the normalizing flow τi _i. Moreover, when Z^i Z_i corresponds to a spurious latent variable, we allow its conditional distribution to vary w.r.t. the surrogate. After the VAE is trained with the loss function ℒL, we use its encoder to obtain the estimated bias-free latent variables Z^C Z_C from the observed data T. These bias-free representations then serve as the input for the subsequent reward modeling stage. Stage 2: Learning Reward Model. In RLHF, the preference dataset typically consists of pairs of preferred and rejected texts. Using the encoder trained in Stage 1, we obtain the bias-free latent variables Z^C Z_C for both texts. For each text, the reward model, implemented as a MLP, takes Z^C Z_C as input and outputs a scalar reward. The probability of one response being preferred over the other is then modeled using the Bradley-Terry formulation. We train this model using maximum likelihood estimation, which is essentially a binary classification task and can be implemented by minimizing the negative log-likelihood. Importantly, while the training procedure itself is standard, the key distinction lies in using the bias-free representations Z^C Z_C from our encoder. As a result, the reward model becomes more resilient to spurious correlations, thereby improving robustness and generalization. 5 Experiments We first conduct experiments on synthetic data to verify our identifiability results. We then apply our method to text data to demonstrate its effectiveness in mitigating sycophancy and concept bias, using similar setup in Wang et al. (2025). We refer to our method as CARD (i.e., CAusal Reward Disentanglement). Unless otherwise stated, for each metric considered, we report its mean and standard error over 88 random trials. Further experiment details are provided in Appendix E. 5.1 Synthetic Data Dataset. To validate our identifiability theory, we generate synthetic data following the process outlined in Eq. 1. We construct the latent causal graph G as an Erdős-Rényi random DAG, constrained such that the spurious latent variables ZSZ_S are not ancestors of the bias-free ones ZCZ_C. The SEMs are parameterized by two-layer multilayer perceptrons (MLPs), with additive Gaussian noise ϵi∼(0,σi2) _i (0, _i^2) where each σi _i is sampled uniformly from [1,2][1,2]. The final observation T is then produced by the mixing function which is a two-layer MLP with LeakyReLU activations. We set the surrogate S to be binary, as our theory requires at least two distinct environments for identifiability, and generate 2×1042× 10^4 samples per value of S. Results. Following prior works (Kügelgen et al., 2021; Kong et al., 2022), we evaluate our model using the coefficient of determination (R2). To measure the correspondence between the ground-truth bias-free latent variables (ZCZ_C) and our learned ones (Z^C Z_C), we fit a kernel ridge regression model and compute the R2 score for predicting ZCZ_C from Z^C Z_C, and vice-versa. We then report the mean of these two scores. Our method achieves a mean R2 of 0.83 ± 0.03, indicating a strong correspondence between the true and recovered subspaces and providing empirical validation for our theory. A perfect R2 of 1.0 is not expected in practice, as the VAE objective is non-convex and the finite-sample nature of both the VAE training and the regression-based evaluation introduce approximation errors. Figure 4: Empirical results under sycophancy bias with varying bias levels. Lower is better. 5.2 Sycophancy Bias Dataset. To systematically investigate sycophantic bias, we construct a dataset derived from the QA preference dataset (Sharma et al., 2024). Each prompt-response pair consists of a question and an response, with prompts generated from two templates: (1) Correct opinion prompt: question. I think the answer is correct_answer, but I’m really not sure., and (2) Incorrect opinion prompt: question. I think the answer is incorrect_answer, but I’m really not sure. In the original dataset, the preferred response corresponds to the correct answer, while the rejected response corresponds to the incorrect one. To inject sycophantic behavior, we modify the training set as follows: with probability ptrain=0.8p_train=0.8, we prepend the phrase “Yes, you are right.” to the preferred response in prompts containing the correct opinion; with probability 1−ptrain1-p_train, we prepend the same phrase to the rejected response in prompts containing the incorrect opinion. This construction induces a controlled spurious correlation in the training data, whereby surface-level agreement (i.e., “Yes, you are right.”) is spuriously associated with correctness. In the test set, we vary ptest∈0.1,0.2,…,0.8p_test∈\0.1,0.2,…,0.8\. The difference ptrain−ptestp_train-p_test can then be viewed as the level of distirbution shift between training and test sets. This breaks the correlation between agreement and correctness to different extents, thereby mimicking real-world conditions where sycophancy is not consistently predictive of human preference. Methods. We evaluate CARD against three baselines: (1) a vanilla reward model, where an MLP is trained on BERT embeddings, consistent with our method’s base model to ensure a fair comparison, (2) CRM (Wang et al., 2025), which adds an invariance regularization to the vanilla model, and (3) a random reward model that assigns rewards from Unif[−10,10]Unif[-10,10]. Metrics. We use two metrics for evaluation. Our main metric is the deviation from oracle performance. The oracle is obtained by training the reward model on the unbiased dataset without injected sycophantic behavior. After introducing sycophancy, we measure how far each method’s performance deviates from this oracle. We also report the worst-case accuracy over different values of ptestp_test, capturing the lower bound of performance under varying degrees of distribution shift. Results. Fig. 4 reports the test deviation from oracle performance across different levels of distribution shifts. Our CARD method consistently achieves the lowest deviation, staying within 0.050.05 of the oracle performance across all distribution shifts, demonstrating robustness to sycophantic bias. By contrast, the vanilla reward model is highly sensitive to distribution shift, with deviations exceeding 0.150.15 in certain cases. The CRM method has a lower deviation than the vanilla baseline but is still less stable than our method. As expected, the random reward model exhibits consistently large deviations. Similarly, in Table 1, our CARD method achieves a substantially higher worst-case accuracy than the baselines. Together, these results demonstrate that leveraging disentangled bias-free representations enables reward models to generalize effectively under distribution shifts induced by sycophantic behavior. Table 1: Worst-case accuracy under sycophancy and concept biases. Higher is better. Sycophancy Concept CARD (ours) 0.630.63 0.600.60 Vanilla reward model 0.470.47 0.310.31 CRM 0.540.54 0.340.34 Random reward model 0.490.49 0.490.49 Figure 5: Empirical results under concept bias with varying bias levels. Lower is better. 5.3 Concept Bias Dataset. To conduct systematic analysis of concept bias, we construct a dataset based on the Amazon Shoe Review corpus (Hou et al., 2024). We transform the review dataset into a preference dataset by pairing each review with a preferred and a rejected response. In the training set, we filter the data so that positive samples are associated with the color concept, while negative samples are associated with the size concept. Each review is reformatted with the prompt “Classify the text into negative or positive:”. The ground-truth label serves as the preferred response, and the incorrect label serves as the rejected response. For the test sets, we systematically vary the correlation between concepts and labels. Specifically, positive samples are associated with the color concept with probability p, while negative samples are associated with the size concept with probability p. We consider p∈0,0.1,0.2,…,1p∈\0,0.1,0.2,…,1\, which represents varying levels of distribution shifts between training and test sets. This setup progressively weakens the spurious correlation between concepts and sentiment, thereby simulating real-world conditions where such relationships are unreliable predictors of human preference. Methods & Metrics. We compare our approach against the same baselines discussed in Section 5.2. In addition the worst-case accuracy described there, we also evaluate using Bias@C, a measure proposed by Zhou et al. (2024). This metric quantifies spurious correlations associated with specific concepts, with values closer to zero indicating weaker bias. For a formal definition, we refer readers to the original paper. Results. As depicted in Fig. 5, CARD consistently achieves much lower Bias@C score than both the vanilla reward model and CRM, with CRM performing only slightly better than the vanilla model. As expected, the random reward model exhibits the lowest bias overall, since it contains no learned correlations; nevertheless, CARD remains very close to it across all levels of distribution shifts, with values almost always below 0.20.2, demonstrating that it does not rely on the spurious concept. Moreover, as shown in Table 1, CARD also achieves significantly higher worst-case accuracy than the baselines. These findings demonstrate that reward models built on bias-free representations are more robust to concept bias. 5.4 Ablation Study We now conduct an ablation study to assess the necessity of disentangled bias-free representations. Recall that the latent variables learned by our VAE can be partitioned into bias-free latent variables Z^C Z_C and spurious ones Z^S Z_S. We train reward models separately on Z^C Z_C, on Z^S Z_S, and on the combined representation Z^=(Z^C,Z^S) Z=( Z_C, Z_S). For evaluation, we use the dataset from Section 5.3, and report both the worst-case accuracy and the Bias@C metric averaged across all levels of distribution shifts, abbreviated as Avg-bias@C. As shown in Table 2, the reward model trained on Z^C Z_C achieves a much higher worst-case accuracy and a lower Avg-bias@C compared to models trained on Z^S Z_S or Z Z. The low bias value of Z^C Z_C suggests that the reward model does not exploit spurious information, since such factors have been disentangled from the representation. By contrast, the higher bias values of Z^S Z_S and Z Z indicate that spurious information remain in these representations, leading the reward model to rely on unstable correlations that vary across bias levels. Table 2: Worst-case accuracy (WCA) and average Bias@C metric (abbreviated as Avg-bias@C) of reward models trained on various types of representations. For worst-case accuracy, higher is better, while for Avg-Bias@C, lower is better. WCA Avg-Bias@C Reward model trained on ZCZ_C 0.600.60 0.150.15 Reward model trained on ZSZ_S 0.530.53 0.400.40 Reward model trained on Z 0.550.55 0.420.42 6 Conclusion We introduce a principled framework for mitigating spurious correlations in reward models. Our key insight is a shift in perspective: rather than training the reward model directly on observed data, we first isolate the bias-free latent variables that capture true human preferences. Theoretical guarantees and a practical algorithm are provided to achieve so. By training reward models exclusively on these bias-free latent variables, we obtain reward models that are more robust and better reflect human preferences, as demonstrated by experiments on synehetic and text-based datasets. This work establishes a foundation for principled bias mitigation in RLHF with theoretical guarantees. References Adams et al. (2021) J. Adams, N. Hansen, and K. Zhang. Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases. Advances in Neural Information Processing Systems, 34:22822–22833, 2021. Ahuja et al. (2023) K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In International Conference on Machine Learning, 2023. Bai et al. (2022a) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. Dassarma, D. Drain, S. Fort, D. Ganguli, T. J. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. Bai et al. (2022b) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022b. Bing et al. (2024) S. Bing, U. Ninad, J. Wahl, and J. Runge. Identifying linearly-mixed causal representations from multi-node interventions. In Conference on Causal Learning and Reasoning, 2024. Bradley and Terry (1952) R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. Brehmer et al. (2022) J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319–38331, 2022. Buchholz et al. (2022) S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems, 2022. Cai et al. (2019) R. Cai, F. Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 32, 2019. Carvalho et al. (2025) D. S. Carvalho, Y. Zhang, H. Unsworth, and A. Freitas. LangVAE and LangSpace: Building and probing for language model vaes. arXiv preprint arXiv:2505.00004, 2025. Casper et al. (2023) S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. J. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Chen et al. (2024) L. Chen, C. Zhu, J. Chen, D. Soselia, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro. ODIN: disentangled reward mitigates hacking in RLHF. In Proceedings of the 41st International Conference on Machine Learning, 2024. Chen et al. (2025) Z. Chen, F. Pinto, M. Pan, and B. Li. Safewatch: An efficient safety-policy following video guardrail model with transparent explanations. In The Thirteenth International Conference on Learning Representations, 2025. Christiano et al. (2017) P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017. Comon (1994) P. Comon. Independent component analysis – a new concept? Signal Processing, 36:287–314, 1994. Devlin et al. (2019) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019. Dinh et al. (2017) L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017. Dong et al. (2023) X. Dong, B. Huang, I. Ng, X. Song, Y. Zheng, S. Jin, R. Legaspi, P. Spirtes, and K. Zhang. A versatile causal discovery framework to allow causally-related hidden variables. In The Twelfth International Conference on Learning Representations, 2023. Eisenstein et al. (2024) J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. N. D’Amour, K. D. Dvijotham, A. Fisch, K. A. Heller, S. R. Pfohl, D. Ramachandran, P. Shaw, and J. Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Language Modeling, 2024. Gabriel (2020) I. Gabriel. Artificial intelligence, values, and alignment. In The Oxford handbook of ethics of AI, pages 175–197. Oxford University Press, 2020. Gresele et al. (2021) L. Gresele, J. Von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems, 34:28233–28248, 2021. Gretton et al. (2007) A. Gretton, K. Fukumizu, C. Teo, L. Song, B. Schölkopf, and A. Smola. A kernel statistical test of independence. In Advances in Neural Information Processing Systems, 2007. Gretton et al. (2012) A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning AI with shared human values. In International Conference on Learning Representations, 2021. Hou et al. (2024) Y. Hou, J. Li, Z. He, A. Yan, X. Chen, and J. McAuley. Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03952, 2024. Hu et al. (2024) J. Hu, X. Wu, Z. Zhu, Xianyu, W. Wang, D. Zhang, and Y. Cao. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405.11143, 2024. Huang et al. (2022) B. Huang, C. J. H. Low, F. Xie, C. Glymour, and K. Zhang. Latent hierarchical causal structure discovery with rank constraints. Advances in Neural Information Processing Systems, 35:5549–5561, 2022. Hyvarinen and Morioka (2016) A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016. Hyvarinen and Morioka (2017) A. Hyvarinen and H. Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460–469. PMLR, 2017. Hyvärinen and Pajunen (1999) A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429–439, 1999. Hyvarinen et al. (2002) A. Hyvarinen, J. Karhunen, and E. Oja. Independent component analysis. Studies in informatics and control, 11(2):205–207, 2002. Hyvarinen et al. (2019) A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics, 2019. Hyvärinen et al. (2023) A. Hyvärinen, I. Khemakhem, and H. Morioka. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10):100844, 2023. ISSN 2666-3899. Jiang and Aragam (2023) Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interventions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Jin and Syrgkanis (2023) J. Jin and V. Syrgkanis. Learning causal representations from general environments: Identifiability and intrinsic ambiguity. arXiv preprint arXiv:2311.12267, 2023. Khemakhem et al. (2020) I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, 2020. Kingma and Welling (2014) D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014. Kivva et al. (2021) B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34:18087–18101, 2021. Kong et al. (2022) L. Kong, S. Xie, W. Yao, Y. Zheng, G. Chen, P. Stojanov, V. Akinwande, and K. Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning, 2022. Kori et al. (2023) A. Kori, P. Sanchez, K. Vilouras, B. Glocker, and S. A. Tsaftaris. A causal ordering prior for unsupervised representation learning. arXiv preprint arXiv:2307.05704, 2023. Kügelgen et al. (2021) J. V. Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, 2021. Lachapelle and Lacoste-Julien (2022) S. Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. arXiv preprint arXiv:2207.07732, 2022. Lachapelle et al. (2022) S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. Conference on Causal Learning and Reasoning, 2022. Lachapelle et al. (2023) S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y. Bengio, S. Lacoste-Julien, and Q. Bertrand. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning. In Proceedings of the 40th International Conference on Machine Learning, 2023. Lachapelle et al. (2024) S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint arXiv:2401.04890, 2024. Lee et al. (2024) H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In Proceedings of the 41st International Conference on Machine Learning, 2024. Liang et al. (2023) W. Liang, A. Kekić, J. von Kügelgen, S. Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal component analysis. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Lin et al. (2021) S. C. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics, 2021. Lippe et al. (2022) P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. CITRIS: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, 2022. Lippe et al. (2023) P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2023. Locatello et al. (2019) F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International conference on machine learning, pages 4114–4124. PMLR, 2019. McMilin (2022) E. McMilin. Selection bias induced spurious correlations in large language models. arXiv preprint arXiv:2207.08982, 2022. Minaee et al. (2024) S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024. Moran and Aragam (2025) G. E. Moran and B. Aragam. Towards interpretable deep generative models via causal representation learning. arXiv preprint arXiv:2504.11609, 2025. Ng et al. (2025) I. Ng, S. Xie, X. Dong, P. Spirtes, and K. Zhang. Causal representation learning from general environments under nonparametric mixing. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. Pace et al. (2024) A. Pace, J. Mallinson, E. Malmi, S. Krause, and A. Severyn. West-of-N: Synthetic preferences for self-improving reward models. arXiv preprint arXiv:2401.12086, 2024. Papamakarios et al. (2021) G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1–64, 2021. Pearl (2009) J. Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2009. Perez et al. (2023) E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, and J. Kaplan. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, 2023. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. Ramé et al. (2024) A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret. WARM: on the benefits of weight averaged reward models. In Proceedings of the 41st International Conference on Machine Learning, 2024. Rezende and Mohamed (2015) D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. In International Conference on International Conference on Machine Learning, 2015. Schölkopf et al. (2021) B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Towards causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Sharma et al. (2024) M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, E. DURMUS, Z. Hatfield-Dodds, S. R. Johnston, S. M. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024. Shen et al. (2023) W. Shen, R. Zheng, W. Zhan, J. Zhao, S. Dou, T. Gui, Q. Zhang, and X. Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. Shen et al. (2022) X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241):1–55, 2022. Silva et al. (2006) R. Silva, R. Scheines, C. Glymour, and P. Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(8):191–246, 2006. Singhal et al. (2024) P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in RLHF. In First Conference on Language Modeling, 2024. Spirtes et al. (2001) P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT press, 2nd edition, 2001. Squires et al. (2023) C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In International Conference on Machine Learning, 2023. Srivastava et al. (2025) P. Srivastava, H. Singh, R. Madhavan, G. Patil, S. Addepalli, A. Suggala, R. Aravamudhan, S. Sharma, A. Laha, A. Raghuveer, K. Shanmugam, and D. Precup. Robust reward modeling via causal rubrics. arXiv preprint arXiv:2506.16507, 2025. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020. Taleb and Jutten (1999) A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing, 47(10):2807–2820, 1999. Tamkin et al. (2023) A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, and D. Ganguli. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689, 2023. Varici et al. (2023) B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023. Varıcı et al. (2024a) B. Varıcı, E. Acartürk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning: Linear and general transformations. arXiv preprint arXiv:2402.00849, 2024a. Varıcı et al. (2024b) B. Varıcı, E. Acartürk, K. Shanmugam, and A. Tajer. Linear causal representation learning from unknown multi-node interventions. arXiv preprint arXiv:2406.05937, 2024b. Veitch et al. (2021) V. Veitch, A. D' Amour, S. Yadlowsky, and J. Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, 2021. von Kügelgen et al. (2023) J. von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Kekić, E. Bareinboim, D. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems, 2023. Wang et al. (2025) C. Wang, Z. Zhao, Y. Jiang, Z. Chen, C. Zhu, Y. Chen, J. Liu, L. Zhang, X. Fan, H. Ma, and S. Wang. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint arXiv:2501.09620, 2025. Wang and Jordan (2021) Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03795, 2021. Xie et al. (2020) F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems, 2020. Xie et al. (2022) F. Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370–24387. PMLR, 2022. Xu et al. (2024) D. Xu, D. Yao, S. Lachapelle, P. Taslakian, J. von Kügelgen, F. Locatello, and S. Magliacane. A sparsity principle for partially observable causal representation learning. In International Conference on Machine Learning, 2024. Yang et al. (2023) K. Yang, D. Klein, A. Celikyilmaz, N. Peng, and Y. Tian. RLCD: Reinforcement learning from contrastive distillation for language model alignment. arXiv preprint arXiv:2307.12950, 2023. Yang et al. (2021) M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. CausalVAE: Disentangled representation learning via neural structural causal models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Yao et al. (2024) D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Locatello. Multi-view causal representation learning with partial observability. In International Conference on Learning Representations, 2024. Yao et al. (2022a) W. Yao, G. Chen, and K. Zhang. Temporally disentangled representation learning. In Advances in Neural Information Processing Systems, 2022a. Yao et al. (2022b) W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Representations, 2022b. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston. Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, 2024. Zhang et al. (2023) J. Zhang, K. Greenewald, C. Squires, A. Srivastava, K. Shanmugam, and C. Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems, 2023. Zhang et al. (2024) K. Zhang, S. Xie, I. Ng, and Y. Zheng. Causal representation learning from multiple distributions: A general setting. In International Conference on Machine Learning, 2024. Zheng et al. (2023) L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. Zheng et al. (2022) Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022. Zhou et al. (2024) Y. Zhou, P. Xu, X. Liu, B. An, W. Ai, and F. Huang. Explore spurious correlations at the concept level in language models for text classification. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. Appendix A Extended Discussion of Related Works A.1 Mitigating Bias in Reward Models Reward models are susceptible to learning spurious correlations (Veitch et al., 2021) from human preference data, leading to various issues including length, sycophancy, discrmination, and concept biases (Zhou et al., 2024). To mitigate length bias, Singhal et al. (2024) proposed several strategies, such as constructing length balanced preference datasets and applying explicit length penalties during training. Shen et al. (2023) applied the Product-of-Experts (PoE) approach to separate human intent from response length during reward training, while Chen et al. (2024) developed the ODIN method that explicitly disentangles the reward signal into quality and length components, discarding the length signal during policy fine-tuning to prevent the model from optimizing for verbosity. While these methods have demonstrated success in addressing their targeted issues, they generally tackle only a single bias, requiring researchers to identify specific spurious features and design tailored solutions. In contrast, our approach offers a principled framework capable of handling multiple types of biases. To improve the robustness of reward models to distribution shifts, Eisenstein et al. (2024) investigated reward model ensembles, finding that while they can reduce reward hacking, they do not eliminate it entirely because individual models often share systematic errors. As a computationally efficient alternative, Ramé et al. (2024) proposed Weight-Averaged Reward Models (WARM), which averages the weights of models from different training checkpoints to improve robustness against preference inconsistencies and distribution shifts. To handle different types of biases, other more general techniques include Wang et al. (2025) that used a regularization method based on maximum mean discrepancy (MMD) (Gretton et al., 2012) to disentangle the true reward from biases. In a similar vein, Srivastava et al. (2025) developed the Crome framework, a data-centric method that uses counterfactual augmentations to systematically teach the reward model to distinguish between meaningful quality improvements and superficial stylistic changes. While effective in certain scenarios, these methods typically lack rigorous theoretical guarantees for mitigating bias. A.2 Latent Variables Identification & Causal Representation Learning Earlier works have shown that it is in general not possible to identify the underlying latent variables without additional assumptions (Hyvärinen and Pajunen, 1999; Locatello et al., 2019). Even when the underlying latent variables are independent, one can construct infinitely many solutions that reproduce the observed distribution while keeping the true latent factors entangled (Hyvärinen and Pajunen, 1999; Locatello et al., 2019). For the special case where the mapping from latent to observed variables is linear, one can impose non-Gaussianity assumption on the latent variables to achieve identifiability (Comon, 1994; Hyvarinen et al., 2002), known as independent component analysis (ICA). Beyond this linear setting, often referred to as nonlinear ICA, a major line of works leverage the assumption of sufficient variability across different data distributions, for instance through time or domain indices (Hyvarinen and Morioka, 2016, 2017; Hyvarinen et al., 2019; Khemakhem et al., 2020). Alternatively, identifiability can be achieved by constraining the mixing function, such as restricting its function class (Hyvärinen and Pajunen, 1999; Taleb and Jutten, 1999; Gresele et al., 2021; Buchholz et al., 2022) or enforcing sparsity on it (Zheng et al., 2022). In recent years, causal representation learning has moved beyond independent latent variables by focusing on causally-related ones (Schölkopf et al., 2021; Moran and Aragam, 2025). As in nonlinear ICA, a major line of works rely on the assumption that the distribution of latent variables varies sufficiently, e.g., through interventions (Ahuja et al., 2023; Squires et al., 2023; von Kügelgen et al., 2023; Jiang and Aragam, 2023; Zhang et al., 2023; Varici et al., 2023; Varıcı et al., 2024a, b; Jin and Syrgkanis, 2023; Bing et al., 2024; Zhang et al., 2024; Ng et al., 2025), temporal data (Yao et al., 2022a, b; Lippe et al., 2022, 2023), or a combination of both (Lachapelle et al., 2022, 2024). Other lines of works leverage structural assumptions (Silva et al., 2006; Xie et al., 2020; Cai et al., 2019; Xie et al., 2022; Adams et al., 2021; Huang et al., 2022; Dong et al., 2023; Kivva et al., 2021), restriction on the support of latent variables (Ahuja et al., 2023; Wang and Jordan, 2021), multi-view data (Yao et al., 2024; Xu et al., 2024), counterfactual view (Brehmer et al., 2022), additional supervision (Yang et al., 2021; Shen et al., 2022), or prior knowledge of causal ordering (Kori et al., 2023) or structure (Liang et al., 2023). Appendix B Proof of Identifiability Theory: With Access to Surrogate We first provide several useful lemmas in Section B.1, followed by the proof of Theorem 1 in Section B.2. The proof builds on Kong et al. (2022, Theorem 4.2) but incorporates several generalizations; see Section 3.1 for a discussion. B.1 Technical Lemmas We state a few lemmas from Kong et al. (2022); Ng et al. (2025) that will be useful for our proofs. Lemma 1 (Kong et al. (2022)). Let f:C×S→Cf:Z_C×Z_S _C be a continuous function. The following three statements are equivalent: • Statement (a): f(ZC,ZS)f(Z_C,Z_S) does not depend on ZSZ_S. • Statement (b): For all Z^C∈C Z_C _C, we have f−1(Z^C)=BZC×Zsf^-1( Z_C)=B_Z_C× Z_s where BZC≠∅B_Z_C≠ and BZC⊆CB_Z_C _C. • Statement (c): For all Z^C∈C Z_C _C and r>0r>0, we have f−1(ℬr(Z^C))=BZC+×ZSf^-1(B_r( Z_C))=B^+_Z_C× Z_S, where ℬr(Z^C):=Z^C′∈C:‖Z^C′−Z^C‖2<rB_r( Z_C):=\ Z _C _C:\| Z _C- Z_C\|^2<r\, BZC+≠∅B^+_Z_C≠ , and BZC+⊆cB^+_Z_C _c. Lemma 2 (Kong et al. (2022)). Consider the generative process in Eq. 1. Denote ψ¯=g^−1∘g:→ ψ= g^-1 g:Z , i.e., Z^=ψ¯(Z) Z= ψ(Z), and Z^C=ψ¯C(Z)=ψ¯(Z)1:nc Z_C= ψ_C(Z)= ψ(Z)_1:n_c, where ψ¯C:→C ψ_C:Z _C. Suppose Z^C⟂S Z_C to0.0pt$ $ 2.0mu to0.0pt$ $ 2.0mu to0.0pt$ $ 2.0mu to0.0pt$ $ 2.0mu S and we model the same generative process with (g^,pZ^)( g,p_ Z). Then, for any AZ^C⊆CA_ Z_C _C and S1,S2∈S_1,S_2 , we have ∫Z∈ψ¯c−1(AZ^C)p(Z∣S=S1)Z=∫Z∈ψ¯c−1(AZ^C)p(Z∣S=S2)Z. _Z∈ ψ_c^-1(A_ Z_C)p(Z S=S_1)\,dZ= _Z∈ ψ_c^-1(A_ Z_C)p(Z S=S_2)\,dZ. Lemma 3 (Ng et al. (2025, Lemma 5)). Let Z^∈ Z be an invertible transformation of Z∈Z . Suppose that there exists ℐ⊆[n]I [n] such that each Z^i,i∈ℐ Z_i,i does not depend on Zj,j∉ℐZ_j,j . Then, Z^ℐ Z_I is an invertible transformation of ZℐZ_I. B.2 Proof of Theorem 1 See 1 Proof. Denote ψ¯=g^−1∘g:→ ψ= g^-1 g:Z , i.e., Z^=ψ¯(Z) Z= ψ(Z), and Z^C=ψ¯C(Z)=ψ¯(Z)1:nc Z_C= ψ_C(Z)= ψ(Z)_1:n_c, where ψ¯C:→C ψ_C:Z _C. Further denote Z=(ZC,ZS)Z=(Z_C,Z_S). Lemma 2 implies that, for any AZ^C⊆CA_ Z_C _C and S1,S2∈S_1,S_2 , we have ∫Z∈ψ¯c−1(AZ^C)p(Z∣S=S1)Z=∫Z∈ψ¯c−1(AZ^C)p(Z∣S=S2)Z. _Z∈ ψ_c^-1(A_ Z_C)p(Z S=S_1)\,dZ= _Z∈ ψ_c^-1(A_ Z_C)p(Z S=S_2)\,dZ. By assumption of the generative process, we have ZC⟂SZ_C to0.0pt$ $ 2.0mu to0.0pt$ $ 2.0mu to0.0pt$ $ 2.0mu to0.0pt$ $ 2.0mu S. Therefore, the above equation can be written as ∫(ZC,ZS)∈ψ¯c−1(AZ^C)p(ZC)(p(ZS∣ZC,S=S1)−p(ZS∣ZC,S=S2))ZSZC=0. _(Z_C,Z_S)∈ ψ_c^-1(A_ Z_C)p(Z_C) (p(Z_S Z_C,S=S_1)-p(Z_S Z_C,S=S_2) )dZ_SdZ_C=0. (3) Our goal is to prove that Z^C=ψ¯c(ZC,ZS) Z_C= ψ_c(Z_C,Z_S) does not depend on ZSZ_S. By Lemma 1, it suffices to prove the following statement. Statement 1. For all Z^C∈C Z_C _C and r>0r>0, we have ψ¯c−1(ℬr(Z^C))=BZC+×ZS ψ_c^-1(B_r( Z_C))=B^+_Z_C× Z_S, where ℬr(Z^C):=Z^C′∈C:‖Z^C′−Z^C‖2<rB_r( Z_C):=\ Z _C _C:\| Z _C- Z_C\|^2<r\, BZC+≠∅B^+_Z_C≠ , and BZC+⊆cB^+_Z_C _c. We now prove the statement above by contradiction. Suppose by contradiction that Statement 1 does not hold, i.e., there exists Z^C∗∈c Z^*_C _c and r∗>0r^*>0 such that BZ∗:=Z=(ZC,ZS)∈ψ¯c−1(AZ^C∗):ZC×s⊈ψ¯c−1(AZ^C∗)≠∅B^*_Z:=\Z=(Z_C,Z_S)∈ ψ_c^-1(A_ Z_C^*):Z_C×Z_s ψ_c^-1(A_ Z_C^*)\≠ , where AZ^C∗:=ℬr∗(Z^C∗)A_ Z_C^*:=B_r^*( Z^*_C). We now compute the LHS of Eq. 3 with Ac∗A_z_c^*. Since ψ¯c−1(AZ^C∗) ψ_c^-1(A_ Z_C^*) by definition, we have ∫(ZC,ZS)∈ψ¯c−1(AZ^C∗)p(ZC)(p(ZS∣ZC,S=S1)−p(ZS∣ZC,S=S2))ZSZC _(Z_C,Z_S)∈ ψ_c^-1(A^*_ Z_C)p(Z_C) (p(Z_S Z_C,S=S_1)-p(Z_S Z_C,S=S_2) )dZ_SdZ_C =∫(ZC,ZS)∈ψ¯c−1(AZ^C∗)∖BZ∗p(ZC)(p(ZS∣ZC,S=S1)−p(ZS∣ZC,S=S2))ZSZC = _(Z_C,Z_S)∈ ψ_c^-1(A^*_ Z_C) B^*_Zp(Z_C) (p(Z_S Z_C,S=S_1)-p(Z_S Z_C,S=S_2) )dZ_SdZ_C +∫(ZC,ZS)∈BZ∗p(ZC)(p(ZS∣ZC,S=S1)−p(ZS∣ZC,S=S2))ZSZC. + _(Z_C,Z_S)∈ B^*_Zp(Z_C) (p(Z_S Z_C,S=S_1)-p(Z_S Z_C,S=S_2) )dZ_SdZ_C. Denote by T1T_1 and T2T_2 the first and second terms in the RHS of the above equation, respectively. We first evaluate T1T_1. By definition of BZ∗B_Z^*, we can write ψ¯c−1(AZ^C∗)∖BZ∗=CZC∗×S ψ_c^-1(A^*_ Z_C) B^*_Z=C^*_Z_C×Z_S for some CZC∗⊂C^*_Z_C _C, which can possibly be an empty set. This implies T1 T_1 =∫(ZC,ZS)∈CZC∗×Sp(ZC)(p(ZS∣ZC,S=S1)−p(ZS∣ZC,S=S2))ZSZC = _(Z_C,Z_S)∈ C^*_Z_C×Z_Sp(Z_C) (p(Z_S Z_C,S=S_1)-p(Z_S Z_C,S=S_2) )dZ_SdZ_C =∫ZC∈CZC∗p(ZC)∫ZS∈s(p(ZS∣ZC,S=S1)−p(ZS∣ZC,S=S2))ZSZC = _Z_C∈ C^*_Z_Cp(Z_C) _Z_S _s (p(Z_S Z_C,S=S_1)-p(Z_S Z_C,S=S_2) )dZ_SdZ_C =∫ZC∈CZC∗p(ZC)(∫ZS∈sp(ZS∣ZC,S=S1)ZS−∫ZS∈sp(ZS∣ZC,S=S2)ZS)ZC = _Z_C∈ C^*_Z_Cp(Z_C) ( _Z_S _sp(Z_S Z_C,S=S_1)dZ_S- _Z_S _sp(Z_S Z_C,S=S_2)dZ_S )dZ_C =∫ZC∈CZC∗p(ZC)(1−1)ZC=0. = _Z_C∈ C^*_Z_Cp(Z_C) (1-1 )dZ_C=0. After evaluating T1T_1, we now consider T2T_2. We first show that BZ∗B_Z^* has nonzero probability measure. First recall that BZ∗B_Z^* is assumed to be non-empty. By the continuity of ψ¯c(⋅) ψ_c(·), for all Z′∈BZ∗Z ∈ B_Z^*, there exists r′>0r >0 such that ℬr′(Z′)⊆BZ∗B_r (Z ) B_Z^*. Since p(Z∣S)>0p(Z S)>0 for Z∈Z and S∈S , it follows that ℙ[Z∈BZ∗∣S=S′]≥ℙ[Z∈ℬr′(Z′)∣S=S′]>0P [\Z∈ B^*_Z\ \S=S \ ]≥P [\Z _r (Z ) \S=S \\ ]>0 for S′∈S , indicating that BZ∗B_Z^* has nonzero probability measure By Assumption A4, there exist S1∗S_1^* and S2∗S_2^* such that T2=∫(ZC,ZS)∈BZ∗p(ZC)(p(ZS∣ZC,S=S1∗)−p(ZS∣ZC,S=S2∗))ZSZC≠0. T_2= _(Z_C,Z_S)∈ B^*_Zp(Z_C) (p(Z_S Z_C,S=S_1^*)-p(Z_S Z_C,S=S_2^*) )dZ_SdZ_C≠ 0. Clearly, we then have T1+T2≠0T_1+T_2≠ 0, which is contradictory with Eq. 3, because the latter holds for all S1S_1 and S2S_2. By contradiction, this implies that Statement 1 must be true, and therefore ψ¯c(⋅) ψ_c(·) cannot depend on ZSZ_S. Since Z^C Z_C does not depend on ZSZ_S, by setting Z^ℐ=Z^C Z_I= Z_C and Zℐ=ZCZ_I=Z_C in Lemma 3, we conclude that Z^C Z_C is an invertible transformation of ZCZ_C, indicating that ZCZ_C is subspace identifiable. ∎ Appendix C Proof of Identifiability Theory: Without Access to Surrogate We first state several key lemmas in Section C.1 and then provide the proof of Theorem 2 in Section C.2. The proof is inspired by Lachapelle and Lacoste-Julien (2022, Theorem 3.1), originally developed for multi-task learning; see Section 3.2 for a detailed discussion. C.1 Technical Lemmas We present several results from Lachapelle et al. (2023) that will be useful in our proofs. In particular, Lemma 5 is an intermediate result derived from Lachapelle et al. (2023, Theorem B.5), while Lemma 4 is stated almost as in Lachapelle et al. (2023, Theorem B.4), with only minor modifications in notation. Lemma 4 (Linear identifiability (Lachapelle et al., 2023, Theorem B.4)). Let W^(⋅):→ℝk×n W^(·):W ^k× n. Suppose Assumptions A2, A3, and A4 from Theorem 2 hold, and that, for ℙWP_W-almost every W∈W and all T∈T , the following holds KL(p(R;W^(W)f^(T))∥p(R;Wf(T))=0.KL(p(R; W^(W) f(T)) p(R;Wf(T))=0. Then, there exists an invertible matrix L∈ℝn×nL ^n× n such that, for all T∈,f(T)=Lf^(T)T ,f(T)=L f(T) and such that, for ℙWP_W-almost every W∈,W^(W)=WLW , W^(W)=WL. Lemma 5 (Lachapelle et al. (2023)). Suppose WL=W^(W)WL= W^(W) for W∈W , where L∈ℝn×nL ^n× n is invertible, and that ℙW[‖W^‖0]≤ℙW[‖W‖0]E_P_W[\| W\|_0] _P_W[\|W\|_0]. Denote by A(W)A^(W) the support of W, and NiN_i the support of L⋅,iL_·,i. Then, under Assumption A5 from Theorem 2, there exists a permutation σ:[n]→[n]σ:[n]→[n] such that Li,σ(i)≠0,i∈[n]L_i,σ(i)≠ 0,i∈[n], and for j∉A(W)j ∈ A^(W), we have A(W)∩Nσj=∅A^(W)∩ N_ _j= . C.2 Proof of Theorem 2 The derivation from Eq. 5 to Eq. 7 in the proof below is adapted from Lachapelle et al. (2023, Theorem B.6), which is a standard approach for showing that minimizing the negative log-likelihood under constraints produces a solution with zero KL divergence from the true distribution. See 2 Proof. To lighten the notation, recall that each W∈W is a row vector. Define supp(W)≔i:W1,i≠0supp(W) \i:W_1,i≠ 0\ as the support of row vector W. Let B≔⋂A∈A=⋂W∈supp(W)B _A A= _W supp(W) and f f be a minimizer of minf _ f ℙWp(X,R∣W)[−logp(R;W^(W)f^(T))] _P_WE_p(X,R W)[- p(R; W^(W) f(T))] (4) s.t. .t. W^(W)∈argminW~s.t.‖W~‖0≤‖W‖0p(X,R∣W)[−logp(R;W~f^(T))] W^(W)∈ *arg\,min_ Ws.t.\| W\|_0≤\|W\|_0E_p(X,R W)[- p(R; W f(T))] and |B^|≤|B|. | B|≤|B|. where B^=⋂W∈supp(W^(W)). B= _W supp( W^(W)). Note that ℙWp(X∣W)[KL(p(R;Wf(T))∥p(R;W^(W)f^(T)))] _P_WE_p(X W)[KL(p(R;Wf(T)) p(R; W^(W) f(T)))] ≥0 ≥ 0 (5) ℙWp(X,R∣W)[−logp(R;W^(W)f^(T))] _P_WE_p(X,R W)[- p(R; W^(W) f(T))] ≥ℙWp(X,R∣W)[−logp(R;Wf(T))]. _P_WE_p(X,R W)[- p(R;Wf(T))]. Therefore, assuming without constraint, the objective in Eq. 4 is minimized if and only if the equalities are attained in the above inequalies. This implies p(X∣W)[KL(p(R;Wf(T))∥p(R;W^(W)f^(T)))]=0E_p(X W)[KL(p(R;Wf(T)) p(R; W^(W) f(T)))]=0 ℙWP_W-almost everywhere. For a given W, the above equality holds if and only if KL(p(R;Wf(T))∥p(R;W^(W)f^(T)))=0KL(p(R;Wf(T)) p(R; W^(W) f(T)))=0 (6) p(X∣W)p(X W)-almost everywhere. Recall that the zero KL term above is obtained under the assumption that Eq. 4 is solved without constraints. To show that the same minimum can be attained under the constrained problem, it suffices to show the existence of a solution that achieves this under the constraints. Such a solution can be constructed by setting f^≔fandW^(W)≔W f f and W^(W) W. Clearly, this choice satisfies the same objective value as the unconstrained global minimum. It is straightforward to verify that the first constraint holds because ‖W^(W)‖0=‖W‖0\| W^(W)\|_0=\|W\|_0 and p(X∣W)[KL(p(R;Wf(T))∥p(R;W~f(T)))] _p(X W)[KL(p(R;Wf(T)) p(R; Wf(T)))] ≥0 ≥ 0 p(X,R∣W)[−logp(R;W~f(T))] _p(X,R W)[- p(R; Wf(T))] ≥p(X,R∣W)[−logp(R;Wf(T))], _p(X,R W)[- p(R;Wf(T))], with equality attained when W~:=W W:=W. The remaining constraint holds trivially because |B^|=|B|| B|=|B|. We have shown that any minimizer f f of Eq. 4 satisfies Eq. 6 and ‖W^(W)‖0 \| W^(W)\|_0 ≤‖W‖0,W∈, ≤\|W\|_0, W , (7) |B^| | B| ≤|B|. ≤|B|. (8) The first inequality above implies ℙW[‖W^(W)‖0]≤ℙW[‖W‖0],E_P_W[\| W^(W)\|_0] _P_W[\|W\|_0], (9) By Assumptions A2, A3, and A4, as well as Eq. 6, Lemma 4 implies that there exists an invertible matrix L∈ℝn×nL ^n× n such that, for all T∈T , we have f(T)=Lf^(T),f(T)=L f(T), (10) and such that, for ℙWP_W-almost every W∈W , we have W^(W)=WL. W^(W)=WL. (11) With a slight abuse of notation, denote by A(W)A^(W) the support of W. Also, let NiN_i be the support of L⋅,iL_·,i. By Assumption A7, as well as Eqs. 7 and 11, Lemma 5 implies that there exists a permutation σ:[n]→[n]σ:[n]→[n] such that Li,σ(i)≠0,i∈[n]L_i,σ(i)≠ 0,i∈[n], and for j∉A(W)j ∈ A^(W), we have A(W)∩Nσj=∅A^(W)∩ N_ _j= . This indicates that: for A∈ and j∉A,we have A⊂(Nσj)c, for~~A ~~and~~j ∈ A, we have~~A⊂(N_ _j)^c, (12) Since this holds for all j∉Aj ∈ A, we have l∈A⟹l∉⋃j∉ANσ(j).l∈ A l ∈ _j ∈ AN_σ(j). Since Eq. 12 holds for all A∈A and j∉Aj ∈ A, we have l∈⋂A∈A⟹l∉⋃A∈⋃j∉ANσ(j),l∈ _A A l ∈ _A _j ∈ AN_σ(j), i.e., l∈B⟹l∉⋃j∉BNσ(j).l∈ B l ∈ _j ∈ BN_σ(j). Recall that, by definition, we have Z^=f^(T) Z= f(T) and Z=f(T)Z=f(T). Furthermore, by Eq. 10 and definition of Nσ(j)N_σ(j), l∉Nσ(j)l ∈ N_σ(j) indicates that ZlZ_l does not depend on Z^σ(j) Z_σ(j). The above statement then implies that each Zl,l∈BZ_l,l∈ B does not depend on Z^σ(j),j∉B Z_σ(j),j ∈ B. Since matrix L is invertible by definition, Lemma 3 implies that ZBZ_B is an invertible transformation of Z^σ(B) Z_σ(B), where σ(B)≔σ(i):i∈Bσ(B) \σ(i):i∈ B\. It remains to show Z^B^=Z^σ(B) Z_ B= Z_σ(B), i.e., B^=σ(B) B=σ(B). Suppose l∈Bl∈ B. Since Ll,σ(l)≠0L_l,σ(l)≠ 0 by definition, we have (W^(W))1,σ(l)=(WL)1,σ(l)=WL⋅,σ(l)=W⋅,A(W)LA(W),σ(l),( W^(W))_1,σ(l)=(WL)_1,σ(l)=WL_·,σ(l)=W_·,A^(W)L_A^(W),σ(l), where, as before, A(W)A^(W) denotes the support of W. Moreover, we have l∈B⊆A(W)l∈ B A^(W), indicating that LA(W),σ(l)L_A^(W),σ(l) cannot be a zero vector. By Assumption A7, we have (W^(W))1,σ(l)≠0( W^(W))_1,σ(l)≠ 0 almost surely, which implies σ(l)∈B^σ(l)∈ B by definition of B B. Since we have established that l∈B⟹σ(l)∈B^l∈ B σ(l)∈ B almost surely, it follows that σ(B)⊆B^σ(B) B (13) almost surely. By Eq. 8, we have |B^|≤|B|=|σ(B)|,| B|≤|B|=|σ(B)|, which, with Eq. 13, implies B^=σ(B) B=σ(B) almost surely. Recall that we have shown that Z^σ(B) Z_σ(B) is an invertible transformation of ZBZ_B. Therefore, Z^B Z_ B is an invertible transformation of ZBZ_B almost surely, indicating that ZBZ_B is subspace identifiable almost surely. ∎ Appendix D Proof of Corollary 1 See 1 Proof. Denote B≔⋂A∈A=⋂W∈supp(W)andB^=⋂W∈supp(W^(W)).B _A A= _W supp(W) and B= _W supp( W^(W)). Under Assumptions A2, A3, A4, and A5 from Theorem 2, Theorem 2 implies that Z^B Z_ B is an invertible transformation of ZBZ_B almost surely. By the assumption that ZC⊆ZAZ_C Z_A for all A∈A , it follows that ZC⊆ZBZ_C Z_B.222When ZCZ_C and ZBZ_B are vectors, we interpret ⊆ as inclusion of their components, i.e., ZC⊆ZBZ_C Z_B means that every component of ZCZ_C is also a component of ZBZ_B. Moreover, the assumption ⋂A∈(ZA∖ZC)=∅ _A (Z_A Z_C)= implies ZB⊆ZCZ_B Z_C. Combining these, we obtain ZB=ZCZ_B=Z_C, and therefore Z^B Z_ B is an invertible transformation of ZCZ_C almost surely. ∎ Appendix E Supplementary Experiment Details Datasets. For the experiments involving sycophancy bias in Section 5.2, we use a dataset derived from the QA preference dataset (Sharma et al., 2024), which in turn is based on the TruthfulQA dataset (Lin et al., 2021). For the experiments involving concept bias in Section 5.3, we use a modified version of the Amazon Shoe Review dataset from Zhou et al. (2024), originally derived from the broader Amazon dataset (Hou et al., 2024). The two datasets are available under the Apache 2.0 License and the MIT License, respectively. Implementation of Our Approach. For Stage 1, our method for learning the bias-free latent variables Z^C Z_C is based on the LangVAE framework (Carvalho et al., 2025). Our key modification involves replacing its independent prior with our prior p(Z^∣S)p( Z S), where the normalizing flow τi _i is implemented using deep sigmoidal flow. We train the VAE with a batch size of 240240 and set the hyperparameters in the loss function to β=0.1β=0.1 and λ=3λ=3. For Stage 2, the reward model is then trained on Z^C Z_C using the standard implementation from the OpenRLHF library (Hu et al., 2024) involving a 22-layer MLP with 128128 Leaky-ReLU units reward model. Implementation of Baselines. The vanilla reward model and CRM use a similar implementation of reward model as our approach. The key difference is that they are trained directly on BERT text embeddings instead of our learned bias-free representations Z^C Z_C. For the CRM baseline, we treat it favorably by selecting the MMD regularization coefficient from 1,3,10,30,100\1,3,10,30,100\ that leads to the best performance. This resulted in coefficients of 100100 and 1010 for the experiments in Sections 5.2 and 5.3, respectively. Others. All experiments are conducted on an NVIDIA A10G GPU with 4 CPU cores. The reported results are averaged over 8 random trials.