Paper deep dive

Interpretable Reward Modeling with Active Concept Bottlenecks

Sonia Laguna, Katarzyna Kobalczyk, Julia E. Vogt, Mihaela van der Schaar

Year: 2025Venue: arXiv preprintArea: Alignment TrainingType: EmpiricalEmbeddings: 48

Models: LLaMA-3-8B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 5:20:19 PM

Summary

The paper introduces Concept Bottleneck Reward Models (CB-RM), a framework for interpretable reward modeling in RLHF. It decomposes reward prediction into human-interpretable concepts and employs an active learning strategy using Expected Information Gain (EIG) to selectively acquire concept labels, improving sample efficiency and transparency in reward model training.

Entities (5)

CB-RM · framework · 100%Concept Bottleneck Model · model-architecture · 100%RLHF · methodology · 100%UltraFeedback · dataset · 100%Expected Information Gain · acquisition-function · 95%

Relation Signals (3)

CB-RM → evaluatedon → UltraFeedback

confidence 100% · Evaluated on the UltraFeedback dataset, our method outperforms baselines

CB-RM → uses → Active Learning

confidence 95% · We introduce CB-RM, a novel approach... We model an AL CB-RM framework that selectively acquires concept annotations

Expected Information Gain → improves → CB-RM

confidence 90% · We propose an acquisition strategy based on Expected Information Gain (EIG), which significantly improves the baselines

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,222 characters extracted from source content.

Expand or collapse full text

Interpretable Reward Modeling with Active Concept Bottlenecks Sonia Laguna Katarzyna Kobalczyk Julia E. Vogt Mihaela van der Schaar Abstract We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models. Machine Learning, ICML 1 Introduction Motivation. A key challenge in aligning machine learning (ML) systems, particularly Large Language Models (LLM), with human preferences lies in the interpretability of the reward models used for their alignment with human values. While significant progress has been made in learning reward functions from human feedback (Christiano ., 2017; Ouyang ., 2022), most existing approaches rely on black-box reward models, making it difficult to understand which factors drive human preferences. This lack of interpretability limits the ability to diagnose, refine, and trust these models in real-world applications (Doshi-Velez Kim, 2017). To address this, we propose a framework for learning interpretable reward functions that explicitly identify and leverage latent dimensions of human preferences with minimal annotation costs, building on the Concept Bottleneck Model (CBM) paradigm (Koh ., 2020). By actively querying human feedback for annotations of interpretable latent dimensions, we aim to uncover the underlying concepts that influence user decisions. Our proposed approach not only enhances the transparency of reward models but also ensures robust alignment with human values. Integrating uncertainty estimation and bayesian experimental design (Melo ., 2024), we optimize the human feedback, scaling the annotation process. This work advances the development of interpretable and trustworthy systems, with more transparent human-machine interactions. Background Reward modeling is central to Reinforcement Learning from Human Feedback (RLHF), a framework where ML systems learn to align with human preferences by optimizing behavior based on feedback—typically in the form of pairwise comparisons between model outputs. Instead of handcrafting reward functions, RLHF uses human preference feedback to train proxy reward models, which are then used to guide policy optimization. This has been effective in fine-tuning LLMs for instruction following and safe interaction tasks (Christiano ., 2017; Ouyang ., 2022). However, the reward models used in RLHF are often opaque and monolithic, making it difficult to understand, debug, or adapt their behavior (Bai ., 2022). Moreover, they require large quantities of labeled data, which is expensive and generally noisy (Casper ., 2023; Sharma ., 2024). To faithfully capture the spectrum of human intent, reward models should support interpretability, personalization, and uncertainty estimation. These properties are not only useful for model introspection but are critical for scalable and trustworthy alignment. CBMs (Koh ., 2020) offer an interpretable alternative to black-box function approximators by explicitly modeling the intermediate concepts that drive model decisions. These structured neural models decompose predictions into two stages: (i) predicting human-interpretable concepts from raw inputs, and (i) predicting the final task label based solely on these concepts. This structure enables inspection, intervention, and debugging of the model’s behavior. CBMs have been extended to interactive settings with different interaction policies (Chauhan ., 2023), stochastic variants considering correlations to propagate through predictions (Vandenhirtz ., 2024), and adapting to external interventions from pretrained models (Laguna ., 2024). Although previous work explores how to intervene most effectively at test time, the strategies assume full access to concept annotations during training. This is, however, not the case in typical data collection setups for preference learning. In this work, we introduce an Active Learning (AL) framework on Concept Bottleneck Reward Models (CB-RM). Our AL algorithm must decide which concept labels to query during training for maximal utility. This setting poses new challenges: the model must identify which missing concept labels will have the most influence on generalization and interpretability. Prioritizing such queries effectively is crucial for building scalable, data-efficient, and trustworthy reward models. A recent attempt at interpretable reward modeling, ArmoRM (Wang ., 2024), learns multi-objective concept scores in a similar fashion to CBMs and combines them via a mixture-of-experts gating mechanism. However, we identify several potential concerns regarding this work and address them here. Contributions In this work, we overcome the above challenges with the following contributions: (i) We introduce CB-RM, a novel approach for modeling reward functions with underlying representations of human understandable concepts; (i) We model an AL CB-RM framework that selectively acquires concept annotations during training episodes for improved interpretability and enables effective learning in low-data regimes—we introduce this formalization for the first time in the CBM context, which itself has extensive potential applications; (i) We demonstrate that active concept acquisition is critical in preference learning settings, where annotating every concept for every pairwise example is expensive and infeasible. To this end, we introduce an acquisition strategy based on Expected Information Gain (EIG), which significantly improves the baselines on concept learning efficiency across training episodes without compromising downstream preference prediction accuracy. Here, we define concepts as interpretable response attributes—such as helpfulness, correctness, and others—that reflect human evaluative preferences. Figure 1: Overview of CB-RM with Active Learning. In CB-RM, each prompt-response pair is encoded by an LLM encoder and an MLP to predict Gaussian-distributed concept representations Δ⁢cΔ cΔ c, to generate a reward prediction using context-conditioned weights ww from gϕ⁢(x)subscriptitalic-ϕg_φ(x)gitalic_ϕ ( x ). During Active Learning, concept-label pairs are selectively acquired using an Activation Function (AF), the best performing being EIG, to maximize concept learning across episodes. Human-annotated concept scores from an LLM judge supervise the pipeline. 2 Problem Formalism Setup Let p⁢o⁢o⁢l=(xi,yi′,yi)subscriptsubscriptsuperscriptsubscript′subscriptD_pool=\(x_i,y_i ,y_i)\Ditalic_p o o l = ( xitalic_i , yitalic_i′ , yitalic_i ) be an unlabeled pool of pairwise preference data conventionally used in reward modeling for RLHF. We denote by xi∈⊂Σ∗subscriptsuperscriptΣx_i ⊂ ^*xitalic_i ∈ X ⊂ Σ∗ a prompt, and yi′,yi∈⊂Σ∗superscriptsubscript′subscriptsuperscriptΣy_i ,y_i ⊂ ^*yitalic_i′ , yitalic_i ∈ Y ⊂ Σ∗ two candidate responses. We use Σ∗superscriptΣ ^*Σ∗ to denote the space of natural language and XX, YY the subsets of all plausible human prompts and LLM responses, respectively. We assume the existence of a ground-truth human reward function r:×→ℝ:→ℝr:X×Y : X × Y → blackboard_R that determines the preference choice between two candidate responses based on the standard BTL (Bradley Terry, 1952) model: p⁢(y≻y′|x)=σ⁢(r⁢(x,y)−r⁢(x,y′)).succeedsconditionalsuperscript′p(y y |x)=σ (r(x,y)-r(x,y ) ).p ( y ≻ y′ | x ) = σ ( r ( x , y ) - r ( x , y′ ) ) . (1) In this work, we assume that the rewards assigned to candidate responses depend on a set of underlying latent concepts CC extractable from the texts (i.e. helpfulness, correctness, coherence, …). Thus, we model r via a context-aware bottleneck composed of two functions: fθsubscriptf_θfitalic_θ and gϕsubscriptitalic-ϕg_φgitalic_ϕ. fθ:×→:subscript→f_θ:X×Y _θ : X × Y → C, maps a prompt-response pair (x,y)(x,y)( x , y ) to a vector fθ⁢(x,y)≡∈subscriptf_θ(x,y)≡ c _θ ( x , y ) ≡ italic_c ∈ C representing the set of concepts present in (x,y)(x,y)( x , y ) and gϕ:→ℝK:subscriptitalic-ϕ→superscriptℝg_φ:X ^Kgitalic_ϕ : X → blackboard_RK, where dim()=Kdimension (C)=Kdim ( C ) = K, maps the prompt x to weight vector gϕ⁢(x)≡(w1,w2,…,wK)∈ℝKsubscriptitalic-ϕsubscript1subscript2…subscriptsuperscriptℝg_φ(x)≡ w≡(w_1,w_2,…,w_K) ^Kgitalic_ϕ ( x ) ≡ italic_w ≡ ( w1 , w2 , … , witalic_K ) ∈ blackboard_RK, with wksubscriptw_kwitalic_k representing the importance of the k-th concept in the context of the topic or task implicitly defined in x. The final reward function can be described as: rθ,ϕ⁢(x,y)=gϕ⁢(x)T⁢fθ⁢(x,y)=T⁢.subscriptitalic-ϕsubscriptitalic-ϕsuperscriptsubscriptsuperscriptr_θ,φ(x,y)=g_φ(x)^Tf_θ(x,y)= w^T c.ritalic_θ , ϕ ( x , y ) = gitalic_ϕ ( x )T fitalic_θ ( x , y ) = italic_witalic_T italic_c . (2) To account for uncertainty in the concept predictions, we model the concept encoder fθsubscriptf_θfitalic_θ as a probabilistic function. Specifically, for each input (x,y)(x,y)( x , y ), it predicts a Gaussian distribution over concept scores: fθ⁢(x,y)∼⁢(⁢(x,y),diag⁢(2⁢(x,y))),similar-tosubscriptdiagsuperscript2f_θ(x,y) ( μ(x,y),diag( σ^2(x,y% ))),fitalic_θ ( x , y ) ∼ N ( italic_μ ( x , y ) , diag ( italic_σ2 ( x , y ) ) ) , (3) where ∈ℝKsuperscriptℝ μ ^Kitalic_μ ∈ blackboard_RK is the predicted mean vector and 2∈ℝKsuperscript2superscriptℝ σ^2 ^Kitalic_σ2 ∈ blackboard_RK is the vector of variances for each concept dimension. Given two responses y and y′ to the same prompt x, we define the concept difference distribution as the distribution of the vector Δ⁢=fθ⁢(x,y)−fθ⁢(x,y′)Δsubscriptsubscriptsuperscript′ c=f_θ(x,y)-f_θ(x,y )Δ italic_c = fitalic_θ ( x , y ) - fitalic_θ ( x , y′ ). Since both fθ⁢(x,y)subscriptf_θ(x,y)fitalic_θ ( x , y ) and fθ⁢(x,y′)subscriptsuperscript′f_θ(x,y )fitalic_θ ( x , y′ ) are modeled as independent Gaussians we have: Δ⁢∼⁢(⁢(x,y)−⁢(x,y′),diag⁢(2⁢(x,y)+2⁢(x,y′))),similar-toΔsuperscript′diagsuperscript2superscript2superscript′ c ( μ(x,y)- μ(x,y ),\ % diag( σ^2(x,y)+ σ^2(x,y )) ),Δ italic_c ∼ N ( italic_μ ( x , y ) - italic_μ ( x , y′ ) , diag ( italic_σ2 ( x , y ) + italic_σ2 ( x , y′ ) ) ) , (4) which enables downstream acquisition strategies to leverage both mean and variance for uncertainty-aware querying. The CB-RM model is trained jointly on the task and concept objectives, with equal weighting by default. While this framework can be generalized, we focus on classification tasks, using cross-entropy loss for both objectives. Goal To learn rθ,ϕsubscriptitalic-ϕr_θ,φritalic_θ , ϕ we must obtain preference labels ℓi∈0,1subscriptℓ01 _i∈\0,1\ℓitalic_i ∈ 0 , 1 determining if for a given sample (xi,yi,yi′)subscriptsubscriptsuperscriptsubscript′(x_i,y_i,y_i )( xitalic_i , yitalic_i , yitalic_i′ ) the first (ℓi=0subscriptℓ0 _i=0ℓitalic_i = 0) or the second (ℓi=1subscriptℓ1 _i=1ℓitalic_i = 1) response is preferred. In addition, to learn the human-interpretable concept bottleneck, we also require a learning signal for grounding the latent representations isubscript c_iitalic_citalic_i in a human-interpretable space of concepts. In this work, we choose to collect relative concept labels sik∈0,1superscriptsubscript01s_i^k∈\0,1\sitalic_iitalic_k ∈ 0 , 1 , where i∈1,…,|p⁢o⁢o⁢l|1…subscripti∈\1,…,|D_pool|\i ∈ 1 , … , | Ditalic_p o o l | and k∈1,…⁢K1…k∈\1,… K\k ∈ 1 , … K . That is, for a tuple (xi,yi,yi′)subscriptsubscriptsuperscriptsubscript′(x_i,y_i,y_i )( xitalic_i , yitalic_i , yitalic_i′ ), we query binary concept labels indicating which of the two responses yisubscripty_iyitalic_i or yi′subscript′y_i yitalic_i′ performs better with respect to the k-th concept e.g., which response is more helpful or more coherent. Obtaining both preference labels and concept annotations across the entire dataset p⁢o⁢o⁢lsubscriptD_poolDitalic_p o o l—which must be large for robust reward learning—is expensive. Therefore, we propose an active learning strategy to efficiently select queries, reducing labeling costs. 3 Method: Active Learning in CB-RM Our method builds on CBMs using probabilistic concept prediction to produce reward scores, and selectively acquires concept labels during training via active learning. In what follows, we formalize this framework and introduce acquisition strategies designed to improve supervision efficiency. We explore selective acquisition strategies that prioritize concept labels expected to be most beneficial for training. Figure 1 shows a general overview of the proposed AL method in CB-RM. To manage training stability and incorporate recent feedback efficiently, we adopt a FIFO (First-In-First-Out) (Dwaracherla ., 2024) replay buffer strategy. After each training episode, newly acquired preference and concept annotations are appended to the buffer, while the oldest entries are discarded once capacity is reached. This approach maintains a bounded memory footprint and ensures the model trains predominantly on up-to-date and relevant samples. Empirically, this strategy supports rapid adaptation to informative queries without overfitting to stale data, aligning with best practices in active exploration under feedback constraints. Algorithm 1, Appendix A includes a comprehensive description of the AL pipeline. We propose a set of acquisition functions to guide the selective annotation of concept labels during training. Let the unlabeled pool be defined as ⊂[|pool|]×[K]delimited-[]subscriptpooldelimited-[]U⊂[|D_pool|]×[K]U ⊂ [ | Dpool | ] × [ K ], representing the Cartesian product of instance indices and concept indices. Each instance index i∈[1,…,|pool|]1…subscriptpooli∈[1,…,|D_pool|]i ∈ [ 1 , … , | Dpool | ] corresponds to a tuple (xi,yi,yi′)subscriptsubscriptsuperscriptsubscript′(x_i,y_i,y_i )( xitalic_i , yitalic_i , yitalic_i′ ), and each concept index k∈[1,…,K]1…k∈[1,…,K]k ∈ [ 1 , … , K ] corresponds to the k-th concept. Thus, each pair (i,k)∈(i,k) ( i , k ) ∈ U identifies which label siksuperscriptsubscripts_i^ksitalic_iitalic_k to query. During each learning episode, we select a batch of B pairs (i,k)(i,k)( i , k ) from UU according to an acquisition function ∙⁢(i,k)subscript∙A_ (i,k)A∙ ( i , k ), annotate the selected concept labels, and retrain the model. We consider the following acquisition strategies: Random Selection As a baseline acquisition strategy, we uniformly sample concept-query tuples: random=UniformSample⁢().subscriptrandomUniformSampleA_random=UniformSample(U).Arandom = UniformSample ( U ) . (5) Concept Variance We select the concept-query tuples with the highest predictive variance in concept differences: Var⁢(i,k)=Var⁢[Δ⁢cik]=σk2⁢(xi,yi)+σk2⁢(xi,yi′),subscriptVarVardelimited-[]Δsubscriptsuperscriptsuperscriptsubscript2subscriptsubscriptsuperscriptsubscript2subscriptsuperscriptsubscript′A_Var(i,k)=Var [ c^k_i ]= _% k^2(x_i,y_i)+ _k^2(x_i,y_i ),AVar ( i , k ) = Var [ Δ citalic_kitalic_i ] = σitalic_k2 ( xitalic_i , yitalic_i ) + σitalic_k2 ( xitalic_i , yitalic_i′ ) , (6) where Δ⁢cikΔsubscriptsuperscript c^k_iΔ citalic_kitalic_i is the k-th dimension of the concept difference. Concept-weighted Influence Score (CwIS) We select concept-query tuples based on their influence on the reward difference, weighted by how uncertain the model is about the prediction itself: CwIS⁢(i,k)=subscriptCwISabsent _CwIS(i,k)=\ ACwIS ( i , k ) = |r⁢(xi,yi)−r⁢(xi,yi′)−r(k)⁢(xi,yi)+r(k)⁢(xi,yi′)|subscriptsubscriptsubscriptsuperscriptsubscript′subscriptsubscriptsuperscriptsubscriptsuperscriptsubscript′ |r(x_i,y_i)-r(x_i,y_i )-r^(k)(x_i,y_i)% +r^(k)(x_i,y_i ) || r ( xitalic_i , yitalic_i ) - r ( xitalic_i , yitalic_i′ ) - r( k ) ( xitalic_i , yitalic_i ) + r( k ) ( xitalic_i , yitalic_i′ ) | +λ⋅Var⁢[Δ⁢cik],⋅Vardelimited-[]Δsubscriptsuperscript \!+λ·Var[ c^k_i],+ λ ⋅ Var [ Δ citalic_kitalic_i ] , (7) where r(k)⁢(⋅)superscript⋅r^(k)(·)r( k ) ( ⋅ ) denotes the reward computed after intervening on the k-th concept (i.e., setting its logit to a fixed high/low value), and we set λ to 0.1. This acquisition function targets concept annotations that are both highly influential for reward prediction and uncertain. Our CwIS strategy is inspired by the CooP policy of Chauhan . ( 2023), which combines concept uncertainty and influence for test-time interventions. While they target inference-time interaction, CwIS is the closest adaptation of existing complementary literature, despite differing goals. Expected Information Gain (EIG) EIG maximizes the expected reduction in uncertainty about the model’s predictions after observing a concept label, which can be approximated by computing the difference between the expected entropy and the entropy of the expected prediction after sampling (Houlsby ., 2011). Concretely, we have: EIG⁢(i,k)subscriptEIG _EIG(i,k)AEIG ( i , k ) =θ,ϕ⁢[ℋ⁢[p⁢(sik∣xi,yi,yi′,θ,ϕ)]]absentsubscriptitalic-ϕdelimited-[]ℋdelimited-[]conditionalsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscript′italic-ϕ =E_θ,φ [H [p(s^k_i x% _i,y_i,y_i ,θ,φ) ] ]= blackboard_Eθ , ϕ [ H [ p ( sitalic_kitalic_i ∣ xitalic_i , yitalic_i , yitalic_i′ , θ , ϕ ) ] ] −ℋ⁢[θ,ϕ⁢[p⁢(sik∣xi,yi,yi′,θ,ϕ)]].ℋdelimited-[]subscriptitalic-ϕdelimited-[]conditionalsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscript′italic-ϕ -H [E_θ,φ [p(s^k_i x% _i,y_i,y_i ,θ,φ) ] ].- H [ blackboard_Eθ , ϕ [ p ( sitalic_kitalic_i ∣ xitalic_i , yitalic_i , yitalic_i′ , θ , ϕ ) ] ] . (8) 4 Experimental Setup We evaluate our method on the UltraFeedback dataset (Cui ., 2024), a large-scale resource of diverse prompt-response pairs from 17 language models, designed to support alignment research without relying on explicit human preference labels. Further explanation of the dataset is included in Appendix B.1. To encode the prompts and responses, we leverage the representations predicted for an LLM encoder (See Figure 1). In particular, to avoid data leakage, all embeddings are computed using LLaMA-2 7B (AI, 2023), whose weights were released prior to the UltraFeedback dataset. This ensures no overlap between model pretraining and evaluation data. To acquire the ground truth concept annotations resembling the human concept-preferences, we annotate each sample using an LLM judge, OpenAI GPT-4o-based, with ten interpretable and broadly applicable concepts: helpfulness, correctness, coherence, complexity, verbosity, instruction following, truthfulness, honesty, safety, and readability. These were chosen for their relevance to human evaluative reasoning. With our approach, we circumvent some of the potential issues present in prior work on interpretable reward models (Wang ., 2024). These include: repeated concept definitions, unevenly distributed annotations across datasets, potentially introducing bias (Kobalczyk van der Schaar, 2025), the use of LLM encoders already trained on the same preference data, raising concerns of information leakage and confounding effects, the lack of concept-level evaluation, and finally, the assumed access to all concept labels. In contrast, our approach generates interpretable labels that are uniformly available across all data points, ensuring greater robustness for downstream learning. Further implementation details are found in Appendix B.2 and the code111https://github.com/sonialagunac/cb-rm-workshop is publicly available. 5 Results To assess the effectiveness of the proposed acquisition functions, we track the improvement in concept accuracy across AL episodes, aiming to enhance reward model interpretability in a cost-efficient manner. In Figure 2 (Top), we see how EIG consistently achieves the fastest gains in concept accuracy when compared with the random baseline. In Figure 2 (Bottom), we include a comparison with the remaining baselines (not in (Top) for clarity), showing that CwIS performs closer to EIG, while concept variance does not significantly affect. Each plot is evaluated over five random seeds. At the same time, preference accuracy remains comparable across all acquisition methods. Figure 2: Top: Concept and preference accuracy in the EIG and random acquisition strategies. Bottom: Concept and preference accuracy in all studied acquisition strategies. Shaded areas denote standard deviation across 5 seeds. 6 Discussion and Conclusion Although preference accuracy remains comparable across methods, models trained with EIG are notably more robust and interpretable in terms of concept performance (Figure 2). Among baselines, CwIS ranks second after EIG. However, unlike in test-time intervention methods (Chauhan ., 2023; Shin ., 2023), concept uncertainty alone does not ensure optimal active learning. This underscores a key distinction: intervention targets immediate fixes, while active learning must prioritize long-term representation quality and generalization. The instability in preference performance likely stems from noisy or biased feedback (Casper ., 2023; Sharma ., 2024), especially for complex responses, leading to misaligned or saturated reward signals. We also observe that embeddings from LLMs that have been likely exposed to the UltraFeedback dataset during their pretraining, exhibit strong concept correlations at the representation level, and the additional concept-level supervision does not improve performance (more details in Appendix C). This observation suggests a potential information leakage in the LLM’s representations. Our findings caution against blind reliance on encoder outputs and suggest a direction for future work on identifying the impact of data leakage in robust reward modeling. Conclusions We propose CB-RM, a concept-based reward modeling framework that enables interpretable preference learning through selective concept annotation. By introducing an active learning formulation and an EIG-based acquisition strategy, we show that it is possible to efficiently improve concept accuracy in low-supervision settings without sacrificing preference performance. Our results highlight EIG’s effectiveness in aligning latent concepts with human preferences, paving the way for more transparent and auditable reward models. While challenges and open questions remain, our approach—combining CBMs, information-theoretic acquisition, and structured training—presents a step toward programmatic reward models aligned with human reasoning. Acknowledgments This work was supported by Azure sponsorship credits granted by Microsoft’s AI for Good Research Lab. SL is supported by the Swiss State Secretariat for Education, Research, and Innovation (SERI) under contract number MB22.00047. K is supported by funding from Eedi. References AI ( 2023) 2023llama2APACrefauthorsAI, M. 2023. 2 7B model. Llama 2 7b model. ://huggingface.co/meta-llama/Llama-2-7b. Bai . ( 2022) 2022trainingAPACrefauthorsBai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N. 2022. a helpful and harmless assistant with reinforcement learning from human feedback Training a helpful and harmless assistant with reinforcement learning from human feedback. preprint arXiv:2204.05862. Bradley Terry ( 1952) 1952rankAPACrefauthorsBradley, R . Terry, M . 1952. analysis of incomplete block designs: I. The method of paired comparisons Rank analysis of incomplete block designs: I. the method of paired comparisons. 393/4324–345. Casper . ( 2023) 2023openAPACrefauthorsCasper, S., Davies, X., Shi, C., Gilbert, T ., Scheurer, J. . 2023. problems and fundamental limitations of reinforcement learning from human feedback Open problems and fundamental limitations of reinforcement learning from human feedback. on Machine Learning Research. Chauhan . ( 2023) 2023interactiveAPACrefauthorsChauhan, K., Tiwari, R., Freyberg, J., Shenoy, P. Dvijotham, K. 2023. concept bottleneck models Interactive concept bottleneck models. of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence ( 37, 5948–5955). Christiano . ( 2017) 2017deepAPACrefauthorsChristiano, P ., Leike, J., Brown, T., Martic, M., Legg, S. Amodei, D. 2017. reinforcement learning from human preferences Deep reinforcement learning from human preferences. in neural information processing systems30. Cui . ( 2024) 2024ultrafeedbackAPACrefauthorsCui, G., Yuan, L., Ding, N., Yao, G. . 2024. : boosting language models with scaled AI feedback Ultrafeedback: boosting language models with scaled ai feedback. of the 41st International Conference on Machine Learning Proceedings of the 41st international conference on machine learning ( 9722–9744). Doshi-Velez Kim ( 2017) 2017towardsAPACrefauthorsDoshi-Velez, F. Kim, B. 2017. a rigorous science of interpretable machine learning Towards a rigorous science of interpretable machine learning. preprint arXiv:1702.08608. Dwaracherla . ( 2024) 2024efficientAPACrefauthorsDwaracherla, V., Asghari, S ., Hao, B. Van Roy, B. 2024. exploration for LLMs Efficient exploration for llms. of the 41st International Conference on Machine Learning Proceedings of the 41st international conference on machine learning ( 12215–12227). Houlsby . ( 2011) 2011bayesianAPACrefauthorsHoulsby, N., Huszár, F., Ghahramani, Z. Lengyel, M. 2011. active learning for classification and preference learning Bayesian active learning for classification and preference learning. preprint arXiv:1112.5745. Kobalczyk van der Schaar ( 2025) 2025preferenceAPACrefauthorsKobalczyk, K. van der Schaar, M. 2025. Learning for AI Alignment: a Causal Perspective Preference learning for AI alignment: a causal perspective. -second International Conference on Machine Learning. Forty-second international conference on machine learning. APACrefURL https://openreview.net/forum?id=iuD649wPAw Koh . ( 2020) 2020conceptAPACrefauthorsKoh, P ., Nguyen, T., Tang, Y ., Mussmann, S., Pierson, E., Kim, B. Liang, P. 2020. bottleneck models Concept bottleneck models. of the 37th International Conference on Machine Learning Proceedings of the 37th international conference on machine learning ( 5338–5348). Laguna . ( 2024) 2024beyondAPACrefauthorsLaguna, S., Marcinkevičs, R., Vandenhirtz, M. Vogt, J. 2024. concept bottleneck models: How to make black boxes intervenable? Beyond concept bottleneck models: How to make black boxes intervenable? in neural information processing systems3785006–85044. Makonnen . ( 2025) 2025measuringAPACrefauthorsMakonnen, M., Vandenhirtz, M., Laguna, S. Vogt, J . 2025. leakage in concept-based methods: An information theoretic approach Measuring leakage in concept-based methods: An information theoretic approach. 2025 Workshop: XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge. Melo . ( 2024) APACrefauthorsMelo, L ., Tigas, P., Abate, A. Gal, Y. 2024. Bayesian Active Learning for Preference Modeling in Large Language Models Deep bayesian active learning for preference modeling in large language models. Thirty-eighth Annual Conference on Neural Information Processing Systems. The thirty-eighth annual conference on neural information processing systems. Ouyang . ( 2022) 2022trainingAPACrefauthorsOuyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C ., Mishkin, P. 2022. language models to follow instructions with human feedback Training language models to follow instructions with human feedback. of the 36th International Conference on Neural Information Processing Systems Proceedings of the 36th international conference on neural information processing systems ( 27730–27744). Sharma . ( 2024) 2024towardsAPACrefauthorsSharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A. . 2024. UNDERSTANDING SYCOPHANCY IN LANGUAGE MODELS Towards understanding sycophancy in language models. 12th International Conference on Learning Representations, ICLR 2024. 12th international conference on learning representations, iclr 2024. Shin . ( 2023) 2023closerAPACrefauthorsShin, S., Jo, Y., Ahn, S. Lee, N. 2023. closer look at the intervention procedure of concept bottleneck models A closer look at the intervention procedure of concept bottleneck models. Conference on Machine Learning International conference on machine learning ( 31504–31520). Vandenhirtz . ( 2024) 2024stochasticAPACrefauthorsVandenhirtz, M., Laguna, S., Marcinkevičs, R. Vogt, J. 2024. concept bottleneck models Stochastic concept bottleneck models. in Neural Information Processing Systems3751787–51810. Wang . ( 2024) 2024interpretableAPACrefauthorsWang, H., Xiong, W., Xie, T., Zhao, H. Zhang, T. 2024. Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Interpretable preferences via multi-objective reward modeling and mixture-of-experts. of the Association for Computational Linguistics: EMNLP 2024 Findings of the association for computational linguistics: Emnlp 2024 ( 10582–10592). Appendix A Active Learning in CB-RM Algorithm 1 outlines the active learning loop used in CB-RM, including acquisition, labeling, and model retraining steps. Algorithm 1 Active Learning with CB-RM 0: Pool dataset UU, labeled set ℒLL, number of episodes T, number of labels to query at each episode B. 1: Train model on initial labeled set ℒLL 2: for episode t=11t=1t = 1 to T do 3: Compute acquisition scores for all candidate (x,y)∈(x,y) ( x , y ) ∈ U using an acquisition function 4: Select B indices (i,k)(i,k)( i , k ) to query the concept annotations. 5: For each pair (i,k)(i,k)( i , k ), query the k-th concept label of the i-th instance (xi,yi,yi′)subscriptsubscriptsuperscriptsubscript′(x_i,y_i,y_i )( xitalic_i , yitalic_i , yitalic_i′ ) and update the labeled set ℒLL. 6: Remove selected indices from the pool UU. 7: Retrain CB-RM model on the updated labeled set ℒLL. 8: end for Appendix B Experimental Details B.1 Ultrafeedback Dataset We use the UltraFeedback dataset (Cui ., 2024), a large-scale resource of 63,967 prompts and 255,864 responses designed for training and evaluating alignment methods in large language models. Prompts are sourced from high-quality datasets like TruthfulQA, UltraChat, FLAN, and ShareGPT, and span a broad range of tasks including question answering, instruction following, and factual verification. Each prompt is paired with four responses sampled from a diverse pool of 17 open-source and commercial models (i.e., LLaMA2, GPT-4, Vicuna), ensuring broad stylistic and qualitative coverage. Responses were generated using varied decoding strategies and model sizes to capture natural variation in assistant behavior. The instructions are primarily single-turn queries but include both simple factual and more complex creative or ethical tasks. This diversity makes UltraFeedback a robust testbed for preference modeling without relying on explicit human labels, offering rich signal for evaluating interpretable, concept-based reward models. B.2 Extended Implementation Details Regarding the dataset, we use a train-validation-test split of 70-10-20 of Ultrafeedback. In order to generate the concept annotations using LLM judge, for each example, the LLM was presented with a system prompt and a user query, followed by two alternative assistant responses. The model was then instructed to rate which response was better with respect to ten predefined concepts. Each concept was scored independently on a scale from 0 to 1, where a score of 0 indicates that the first response is clearly better, and 1 that the second response is clearly better. We generate the final preference labels as a linear combination of the mentioned concepts to ensure the ultimate reward model remains interpretable. Moreover, the implemented FIFO has a buffer capacity of 32000 and a number of acquired samples of 320. Both the CBM and gating mechanism use a one-layer perceptron to extract concept logits in a probabilistic variant and weights, and were trained for 1 epoch. This architectural modeling ensures the use of programmatic representations: by structuring reward models through interpretable components, we enable modular, debuggable learning that goes beyond black-box supervision. Appendix C Active Learning Results on Large LLMs with Information Leakage Figure 3: Concept and preference accuracy in all studied acquisition strategies using embeddings generated from Llama3-8B. Shaded areas denote standard deviation across 5 seeds. Figure 3 illustrates the limitations of applying active learning with concept-based acquisition strategies on large LLMs such as LLaMA-3-8B, whose training cutoff date is after the release of the UltraFeedback dataset and likely have been exposed to this dataset during their pre-training. In this setting, embeddings exhibit strong linear correlations with target concepts, and additional supervision yields little to no improvement. This highlights a critical risk of information leakage, where pretrained representations already encode the target signals, undermining the benefits of active acquisition. We include this analysis to caution practitioners: when evaluating interpretability or data efficiency methods, care must be taken to avoid confounded setups that mask true learning dynamics. Furthermore, the CBM paradigm itself has been shown to be vulnerable to information leakage within the bottleneck, with recent works proposing methods to quantify this effect (Makonnen ., 2025). Therefore, it is crucial to avoid additional sources of leakage in encoder representations, which would only compound the problem and obscure the model’s reasoning further.

Instant research discovery