Paper deep dive
A Method for Learning Value Systems in Generative AI
Andrés Holgado-Sánchez, Holger Billhardt, Sascha Ossowski
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 4:14:45 AM
Summary
This paper proposes a method for learning value systems in generative AI by simultaneously inferring a multi-objective reward model (grounding) for specific values and a weighted linear scalarization representing an agent's value system. The approach uses pairwise prompt-response preference data to learn coherent value representations and their aggregation, aiming to improve explainability and alignment over traditional RLHF methods that lack explicit value decomposition.
Entities (9)
Relation Signals (7)
Proposed Method → learns → Value System
confidence 96% · simultaneously learns: ... ii) a value system representation in the form of a weighted linear scalarization
Proposed Method → learns → Grounding
confidence 96% · simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model
Grounding → implementedby → Multi-objective Reward Model
confidence 95% · A grounding of the set of values V is a set of weak orders... approximating them using separate utility models... represented by a multi-objective reward model
Generative AI → requires → Value System
confidence 95% · Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours.
Value System → representedas → Linear Scalarization
confidence 94% · value system representation in the form of a weighted linear scalarization of the previous grounding model
Helpfulness → exampleof → Value
confidence 90% · Two principles arise as the most studied, namely helpfulness and harmlessness/safety
RLHF → critique → Generative AI
confidence 85% · A fundamental limitation of this approach is its inability to recognise the pluralistic nature of alignment... they do not decompose the value alignment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours. As such representations are difficult to elicit, value learning seeks to infer them by observing human behaviour. This work addresses the lack of grounded value learning methods in generative AI: existing approaches typically replicate human preferences without awareness of the multidimensional structure of value alignment, or lack principled value system elicitation methods. To address these gaps, we adapt a previously validated value system learning method to the generative AI setting, which, based on pairwise prompt-response preference data, simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model, and ii) a value system representation in the form of a weighted linear scalarization of the previous grounding model. To ensure that the learned value systems are based on coherent value representations, our algorithm dynamically prioritizes the grounding learning process. We evaluate the method against baselines and a contemporary method on prompt-response preference datasets. Results show competitive performance and minimal trade-offs against the baselines, while improving explainability.
Tags
Links
- Source: https://arxiv.org/abs/2607.16903v1
- Canonical: https://arxiv.org/abs/2607.16903v1
Trouble viewing inline? Open PDF directly →
Full Text
93,724 characters extracted from source content.
Expand or collapse full text
A Method for Learning Value Systems in Generative AI Andr ́es Holgado S ́anchez 1 Holger Billhardt 1∗ , Sascha Ossowski 1∗ 1 CETINIA, University Rey Juan Carlos Unnumbered Tulip ́an Street, M ́ostoles, 28933 Madrid, Spain Abstract Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours. As such representations are diffi- cult to elicit, value learning seeks to infer them by ob- serving human behaviour. This work addresses the lack of grounded value learning methods in generative AI: existing approaches typically replicate human prefer- ences without awareness of the multidimensional struc- ture of value alignment, or lack principled value system elicitation methods. To address these gaps, we adapt a previously validated value system learning method to the generative AI setting, which, based on pair- wise prompt-response preference data, simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model, and i) a value system representation in the form of a weighted linear scalarization of the previous grounding model. To ensure that the learned value systems are based on coherent value representations, our algorithm dynam- ically prioritizes the grounding learning process. We evaluate the method against baselines and a contempo- rary method on prompt-response preference datasets. Results show competitive performance and minimal trade-offs against the baselines, while improving ex- plainability. 1 Introduction Aligning AI systems with human intentions has long been recognized as a central challenge in artificial intelligence (Anderson, Anderson, and Armen 2006). With the widespread deployment of generative AI mod- els –predominantly Large Language Models (LLMs)– the need for reliable mechanisms to steer their be- haviour has become increasingly pressing. The domi- nant paradigm for this purpose is Reinforcement Learn- ing from Human Feedback (RLHF) (Christiano et al. 2017), which aligns a generative model with human preferences by (i) learning a reward model that esti- mates the alignment of its generations with human de- ∗ These authors contributed equally. Copyright© 2026, Association for the Advancement of Ar- tificial Intelligence (w.aaai.org). All rights reserved. Value system, implemented by a linear scalarization of social groundings Social grounding of values, implemented by one reward model per value Helpfulness: Prompt Trufthfuless: Safety: x0.8 x0.1 x0.1 Value alignment preferences ? ? ? Value-based preferences ? ? MLP MLP MLP LLM (Frozen) Response Figure 1: Diagram of the proposed value system learn- ing solution. Based on pairwise preference data reflect- ing value alignment with multiple human values and the value-based preferences of multiple agents, we learn both a grounding for these values implemented by a multi-objective reward model and a value system rep- resentation in the form of linear scalarization weights. sires, and (i) fine-tuning the generative model to max- imize this reward using reinforcement learning. A fundamental limitation of this approach is its inability to recognise the pluralistic nature of align- ment (Sorensen et al. 2024), i.e. they do not decompose the value alignment (Russell 2022) of actions in terms of different human values. The field of value awareness engineering argues that AI systems should explicitly represent both values and the value-based preferences of agents (here called value systems) in order to be able to reason about them (Montes et al. 2023). With regard to values, the manual design of value representations is difficult and prone to misspecification (Sumers et al. 2022), especially if these representations are based on reward models (Hadfield-Menell et al. 2017). As an al- ternative, authors have proposed value learning (Soares 2018) as a means to acquire value representations – hereafter, groundings (Holgado-S ́anchez et al. 2026)– automatically from stated preferences or demonstra- tions of behaviour. This idea has been explored in some arXiv:2607.16903v1 [cs.CY] 18 Jul 2026 decision-making contexts, where utility or reward func- tions represent morally relevant concepts and are subse- quently used as decision objectives (Reddy et al. 2020; Rodr ́ıguez-Soto et al. 2026). Following this idea, some works in the generative AI domain (e.g. (Dong et al. 2023; Wang et al. 2024a, 2025)) model alignment with multiple values using inde- pendent reward functions, employing a multi-objective alignment (Vamplew et al. 2018) framework. With re- gard to modelling the value systems of users, in those works it is assumed that users explicitly state their value systems directly (in the form of value weights or constraints). However, there is evidence that users are often unable to communicate their value system reli- ably (Siebert et al. 2022). Other methods approximate a Pareto frontier of aligned models for users to choose from (Zhou et al. 2024; Gupta et al. 2025), avoiding directly asking for the users’ value systems. However, this approach incurs in additional computational effort and user interactions. Finally, there are approaches that infer value systems from user data. Still, they tend to propose latent (Li et al. 2025) or opaque value system representations (Wang et al. 2024b) that have limited interpretability in terms of objectives/values. To address these shortcomings, we propose a method for learning an approximation of a complete value system representation suited for preference modelling in generative AI applications, by means of a multi- objective decision making framework. Specifically, we learn a grounding of value labels implemented us- ing reward models, and approximate value systems as weighted linear scalarizations over the previous value representations. Our algorithm jointly learns both rep- resentations from prompt–response preference pairs ex- pressed in terms of the alignment of each response with each value, as well as of the alignment with the value system of a user, as illustrated in Figure 1. Notably, our optimization dynamically prioritizes the grounding learning process over the value system one, as careful value recognition is a precondition for value system es- timation (Liscio et al. 2023). This paper is organized as follows. In Section 2 we survey related works. In Section 3.2 we propose our value alignment and value system representation ap- proach for generative AI models. In Section 4 we formal- ize our value system learning problem, and propose a theoretically sound algorithm that approximates a solu- tion to it. Lastly, in Section 5, we evaluate our proposal and in Section 6 we provide conclusions, limitations and avenues for future work. 2 Related Works 2.1 Values in Generative AI Recent work on AI alignment acknowledges that users demand different behavioural properties from genera- tive AI assistants. Two principles arise as the most stud- ied, namely helpfulness and harmlessness/safety (Bai et al. 2022b). Lately, other principles or goals such as co- herence (Wang et al. 2024c), objectivity (Ji et al. 2024), honesty (Cui et al. 2023), or even humor (K ̈opf et al. 2023) have been explored and aligned for. These have often been considered as alignment targets or subgoals that describe different dimensions of the task of aligning assistant AI models with human values (Dognin et al. 2025; Kirk et al. 2024). These alignment targets can be interpreted as domain-dependent “concepts” that real- ize broader human values (Osman and d’Inverno 2024). For instance, the helpfulness of a model may be consid- ered a particularization of the more abstract value of achievement (Schwartz 1992), i.e. the value that fosters the pursuit of personal goals and aims. 2.2 Alignment With Given Value Systems Authors typically introduce alignment targets/at- tributes/criteria (hereafter also referred to simply as values) 1 with the explicit goal of steering the behaviour of LLMs towards different value-based preferences, or value systems (Serramia et al. 2018), specified by the user. In this process, most methods learn or rely on an existing computable representation of the values, i.e. a grounding (Holgado-S ́anchez et al. 2026), that mea- sures the alignment or misalignment of responses with the values considered. This grounding notion is to some extent inherent in recent large language models (LLMs) and can be extracted using careful prompting strate- gies (Adams et al. 2025). However, for novel application contexts or previously unexplored values, it is gener- ally necessary to train grounding models (normally us- ing reward models) from human or AI annotations (Bai et al. 2022a). These value-alignment rewards can then be used to fine-tune a response generator (LLM-based), for instance through RLHF (Christiano et al. 2017). Notable examples follow the previous paradigm. In (Dong et al. 2023), the authors learn a multi-attribute reward model that estimates the degree of alignment of prompt–response pairs with multiple values or at- tributes. They then fine-tune LLMs through supervised fine-tuning (SFT) to generate responses that adhere to user-specified desired alignment levels over these at- tributes. Similarly, (Wang et al. 2024a) learn a multi- objective reward model via linear regression on quanti- fied value-annotated examples, and subsequently train an LLM to combine these value-specific rewards accord- ing to given reward weightings. Other approaches aim to control the trade-offs between alignment with dif- ferent values, such as Controllable DPO (Guo et al. 2024) and MAP (Wang et al. 2025), by enforcing manu- ally specified value-alignment constraints. Finally, some authors focus on obtaining a Pareto front of LLMs aligned for multiple value combinations (Yang et al. 2024; Gupta et al. 2025). These approaches, however, 1 To simplify the reading, we will refer to all alignment targets or related goals (such as helpfulness) as full-fledged “values”, assuming they have already been identified as rele- vant alignment targets in the application context considered; see value identification (Liscio et al. 2021). may expend computational resources on approaching regions of the Pareto front that are ultimately undesir- able for the user. 2.3 Estimating Value Systems The previous works proposed value-steerable models that learn value representations (groundings), however, to adequately steer their models according to particular users, they expect them to convey their preferences in the form required by their respective architectures (e.g. weights). However, eliciting the value system of a user in terms of context-dependent values, is, in general, a hard problem that has received the name of value system es- timation (Siebert et al. 2022; Liscio et al. 2023). Classi- cal solutions for this problem include surveys (Schwartz 1992) and participatory design methodologies (Ziegfeld et al. 2025). However, it has been shown that value sys- tems elicited in surveys may change depending on the task the user is trying to perform (Kirk et al. 2024), which makes translating these value system estimations into actionable model behaviour a challenging task. Therefore, authors tend to learn value system repre- sentations directly from application-oriented data. For example, if a set of models aligned with known value system representations is available, selecting an appro- priate model by asking an user to judge responses gener- ated by each model is a natural approach to indirectly estimate her value system (Ram ́e et al. 2023; Gupta et al. 2025), albeit a costly one. A more scalable solution is proposed by Wang et al. (Wang et al. 2024b), who learn value-specific reward models and infer prompt- dependent value system representations in the form of scalarization weights via a dedicated deep neural net- work. While effective, this design lacks interpretabil- ity, as the relationship between prompt features and the inferred value system weights is not directly trace- able. Relatedly, works such as LoRe (Bose et al. 2025) and PAL (Chen et al. 2025) learn latent preference em- beddings to efficiently capture personal user preferences (i.e. value systems), but the dimensions of these embed- dings do not directly correspond to identifiable values. Unlike these approaches, our method aims to learn a transparent value system representation based on rep- resented values, which allows an agent to reason explic- itly on the values and how they combine into a value system. 3 Value Alignment in Generative AI Following common practice, we model the generative task of any given generative model (e.g. LLM) as a con- textual bandit problem (Tennant, Hailes, and Musolesi 2025) (or MDP, in general), where there is a set of states or instructions (from those of a possibly infinite set S, e.g. prompts) for which different actions (from a set A, e.g. responses) can be generated or selected. The model is described by a policy π(a| s) that outputs the most appropriate action a ∈ A given a state s ∈ S. We de- fine the pair τ = (s,a) as a generation and the set of generations as T ⊂ S× A. We assume that appropriateness of a generation τ = (s,a) can be modelled through a quantitative reward function (or reward model in the generative AI jargon) R(s,a). Such reward model can be learned from datasets of human annotated examples of prefer- ences (Christiano et al. 2017) or, directly, from scored outputs to example user inputs (Cui et al. 2023). The goal of a model π that is aware of this reward model is to generate actions that maximize the expected reward. In the following, we will be interested in measuring and learning the appropriateness of generations solely in terms of their alignment with human values and, based on that, their appropriateness in terms of the value systems of particular agents, using several reward models. In this paper, we leave out of scope the fine- tuning process of policies with these reward models, that may be carried out using existing methods such as RLHF (Christiano et al. 2017; Bai et al. 2022b). 3.1 Representing Value Alignment To properly introduce our reward learning objectives, we will adopt the theoretical value alignment represen- tation assumptions previously proposed in (Holgado- S ́anchez et al. 2025) and adapt these to the generative AI context. Definition 1 (Value Alignment) The alignment of a set of generations T with a value v i (in general, the alignment preferences with v i ) is represented by a weak order ≼ v i over T , where τ ≼ v i τ ′ means that the gen- eration τ ′ is at least as aligned with value v i as τ . Then, to jointly specify the alignment with a set of val- ues, we formalize the notion of grounding. Definition 2 (Grounding) A grounding of the set of values V is a set of weak orders ≼ V =≼ v i m i=1 . To facilitate the computational representation of these value alignment preferences, we propose approx- imating them using separate utility models for differ- ent values. This view has been paramount in works that model human/moral values in general (Rodr ́ıguez- Soto et al. 2022; Serramia et al. 2018; Montes and Sierra 2022; Karanik et al. 2024). Thus, we propose a value alignment function,A v i , that quantifies the value alignment of a generation with a value v i ∈ V , i.e. that represents the relation ≼ v i : i.e., for all τ,τ ′ ∈ T : τ ≼ v i τ ′ ⇐⇒ A v i (τ )≤A v i (τ ′ ). In the generative AI setting, an alignment function can be directly implemented through a reward model: R v i (τ ) ≡ A v i (τ ). Thus, gathering alignment functions for each preference relation, we can represent a ground- ing ≼ V with a multi-objective reward model, or reward vector R R R(s,a) such thatR R R(τ ) = (R v 1 (τ ),...,R v m (τ ))≡ (A v 1 (τ ),...,A v m (τ )). Given a reward vector, we de- fine the particular grounding that it implements as ≼ R R R =≼ R v i m i=1 , i.e. τ ≼ R v i τ ′ ⇐⇒ R v i (τ )≤ R v i (τ ′ ). In occasions, we will refer to this reward vector also as the grounding function/model to emphasize its role as the implementation of a grounding. Importantly, our reward functions do not aim to cap- ture the whole meaning of the given values, rather, we are estimating the alignment of actions with these value based on the proposed multi-objective model. 3.2 Representing Value Systems A value system expresses the general importance as- signed to each value by a certain person or group (in general, an “agent”) in a specific context. Given a grounding, it represents the individual (value-based) preferences over generations of an agent. Definition 3 (Value system) The value system of an agent j, based on a grounding ≼ V , is determined by a weak order ≼ j V over T . If τ ≼ j V τ ′ , we say that τ is equal ly or more aligned than τ with j’s value system. The idea is that the value system preferences of an agent are based on the individual value preferences. Fol- lowing other works in the field (Serramia et al. 2018; Rodr ́ıguez-Soto et al. 2026), we assume that the value system of an agent can be approximated by a linear combination of the alignment of a generation with the values. Thus, our value system model is welfarist util- itarian (Sinnott-Armstrong 2023), i.e. values are con- ceived as different sources of good, and each agent’s value system weighs their relative importance. Definition 4 (Value System Function) Let j be an agent with a value system ≼ j V and a grounding repre- sented by a reward vector R R R. The function A W j ,R R R (τ ) = W j · (R v 1 (τ ),...,R v m (τ )) T is a value system func- tion for j if it represents ≼ j V over T , i.e., for al l τ,τ ′ ∈T : A W j ,R R R (τ )≤A W j ,R R R (τ ′ ) ⇐⇒ τ ≼ j V τ ′ where W j = (w v 1 j ,...,w v m j ) are called value system weights. These are bounded in the unit (m−1)-simplex 2 : W j ∈ ∆ m−1 , i.e. W j ∈ (0, 1) m , P m i=1 w v i j = 1. We can interpret the previous definition as having a value system reward for each agent that is a linear scalarization (Van Moffaert, Drugan, and Now ́e 2013) of the multi-objective reward vector R R R with weights W j : R j (τ ) = W j ·R R R(τ ) T Finally, we denote the value system implemented through the value system alignment function A W j ,R R R 2 As we treat values as sources of good, we do not consider value systems that willingly demote values by using nega- tive weights. If demoting a value is expected to occur, the designer should instead include another value in the model as the negated version of the original. An example of such case is the technical value of response complexity (Wang et al. 2024c), which in some situations might be undesirable (when a response is for a audience of children, for instance). Instead of using a negative value system weight, we suggest adding the contrary value of simplicity, to represent the ab- stract goal of reducing complexity. (with weights W j and reward vector R R R) by ≼ W j R R R , i.e. τ ≼ W j R R R τ ′ ⇐⇒ W j ·R R R(τ )≤ W j ·R R R(τ ′ ). We emphasize that our value system representation is a mere approximation of the true value systems of agents, which might be much more complex and context-dependent. We chose a linear representation that eases the interpretability of our model. 4 Learning Value Systems In this section, we present our proposal: a model and algorithm for learning simultaneously a reward vector R R R and a weight vector W , which approximate a ground- ing of the values considered and the value system of a certain agent, respectively. Our learning approach relies on a dataset of annotated preferences for both, values and the value system of an agent. 4.1 Data Assumptions and Scope Within the field of generative AI, even though datasets that identify individual annotators or agents exist (K ̈opf et al. 2023; Kirk et al. 2024), the majority of datasets that label examples along multiple alignment dimen- sions or values agglomerate the opinions of distinct in- dividuals with potentially divergent preferences (Wang et al. 2024c; Ji et al. 2023, 2024). Our work seeks to be applicable in the latter cases as well, assuming that the dataset preferences are those of an hypothetical ab- stract agent representing the collective. This assumption may raise concerns about the use- fulness of the proposed approach, as it might attempt to reconcile potentially divergent opinions into a single representation. However, with regard to value recogni- tion in particular, we argue this view might not be prob- lematic. Namely, although we consider agents may hold divergent interpretations of the values under consider- ation (for example, the meaning of “freedom”), within delimited domains we can reasonably assume that most agents understand values in broadly similar manner. That is, there exists a degree of consensus regarding the meaning of values, which we refer to as a social grounding (Holgado-S ́anchez et al. 2025). In the generative AI domain, the existence of such social groundings is often implicitly assumed. We iden- tify two main reasons for this. First, most alignment- related datasets justify the inclusion of multiple align- ment criteria as universal quality dimensions and pro- vide explicit definitions for each, which condition the preference annotation process (Bai et al. 2022b). Sec- ond, some works explicitly average different opinions over the same utterances to improve “robustness” of the annotations (Wang et al. 2024c), to then train a shared “grounding-function-like” model. With regard to value systems, it is clear that even in the same application domain, agents may have diverse value-based preferences (value systems). Our work, however, performs an implicit aggregation (Lera- Leri et al. 2024) of potentially divergent value sys- tems. Doing this has been shown to have limitations in terms of the final representativeness of value-based preferences, specially in datasets with heterogeneous agents (Holgado-S ́anchez et al. 2025; Chakraborty et al. 2024). We acknowledge potential limitations of the rep- resentativeness that we achieve in these cases, but argue that our approach is still valuable, as the learned value system can be interpreted as a transparent representa- tion of the overall, empirical, importance given to each value according to the dataset annotators. In conclusion, in this paper, we consider that the value alignment preferences observed in our dataset cor- respond to a social grounding in a delimited context (e.g. text generation) and the value system preferences pertain to some abstract agent. When data annotations originate from a group of agents with potentially het- erogeneous value systems, then our learning approach would try to identify the aggregated value system that best represents the collective preferences of that group. Importantly, due to the limitations mentioned above, if our model is learned from heterogeneous datasets, the resulting value system reward (given by W ·R R R) should not be intended for use as a stand-alone objective in RLHF fine-tuning. In such cases, it would likely bias the generative model toward majority opinions. Instead, we recommend using the learned grounding function R R R for multi-objective fine-tuning. Specifically, generative models should be trained in a way that allows their im- mediate alignment with respect diverse value systems, following approaches similar to prior work (Gupta et al. 2025; Wang et al. 2024c; Jang et al. 2024). Our ap- proach offers an additional advantage: it enables the ex- ploration of value systems in the vicinity of the learned one, that acts as the representation of the preferences of the majority. 4.2 Problem Definition Considering the above, our value system learning prob- lem consists of learning a grounding implementation together with a representation of the value system of a certain (possibly abstract) agent j that best approx- imates the social grounding ≼ V and value system ≼ j V enacted in a observed preference dataset. In particu- lar, we assume access to examples of value alignment and value system preferences over pairs of generations. Formally, we assume a dataset, DS, composed by en- tries of the type (τ,τ ′ ,y j V ,y v 1 ,...,y v m ), where each la- bel y ∈ 0, 0.5, 1 indicates the preference of τ over τ ′ (1), the contrary case (0) or indifference (0.5). Specifi- cally, label y j V indicates the preference according to the agent’s value system, and labels y v i , i = 1,...,m repre- sent the preference regarding the alignment of each gen- eration pair with respect to each value v i . As mentioned above, we assume that the latter labels correspond to the social grounding of the values. Given the analysis in Section 3.2, we propose reduc- ing our learning problem into two tasks i) learn a quan- titative reward vector R R R such that the grounding ≼ R R R implemented by R R R approximates the real social ground- ing≼ V and i) learn a vector of value system weights W such that the value system ≼ W R R R approximates the real value system ≼ j V . We now provide ways to quantitatively measure the appropriateness of potential candidate solutions. For that, we quantify the alignment preference differences between our models and the given data. To measure preference differences in general, we propose the dis- cordance between preference relations (≼ 1 , ≼ 2 ) as the proportion of ordered pairs of generations in a set S ⊆T ×T that are ranked differently: d S ≼ 1 ,≼ 2 = 1 |S| X (τ,τ ′ )∈S 1 τ ≼ 1 τ ′ ̸≡ τ ≼ 2 τ ′ (1) In our settings, the aim is to learn grounding func- tions and value system weights with minimal discor- dance to the original data (represented in the dataset DS). Thus, we can define our value system learning problem as a nested optimization problem: (W ∗ ,R R R ∗ )∈arg min W,R R R ′ =(R v 1 ,...,R v m ) d DS ≼ W R R R ′ ,≼ j V subject to R ′ v i ∈ arg min R v i d DS ≼ R v i ,≼ v i , ∀i∈1,...,m(2) The formulation prioritizes the minimization of the av- erage discordance on the learned grounding functions, before improving the value system estimation. This en- sures that the learned value system weights are based on a coherent social grounding approximation. It should be noted that in Problem 6, the discordance can be directly calculated by comparing the preference predictions of the learned models with the original pref- erence relations on the generation pairs included in the dataset DS. To do so, we discretize the score differences predicted by our model for each pair of generations to predict strict preference (τ ≺ τ ′ /τ ≻ τ ′ ) or indifference (τ ≃ τ ′ ), which is treated as predicting both τ ′ ≼ τ and τ ′ ≽ τ . Then, we check the agreement of the dis- cretized predictions with the corresponding dataset la- bel y ∈0, 1, 0.5. Details on how the predictions of the model are discretized and the data-based discordance is calculated are given in supplementary material, Section A.3. 4.3 Algorithm Our algorithm is an approximate solution to Problem 2 that versions the classical deep reward learning method based on the Bradley-Terry (BT) model (Bradley and Terry 1952). The solution is represented by several neural networks. First, a reward vector network R R R θ with parameters θ implements an approximation the social grounding in the data. Second, a linear layer given by certain value system weights W ω that are parametrized with ω ∈R m through a softmax calcula- tion: W ω = (w v 1 ,...,w v m ) = expω P exp(ω) , so W ω remains in the simplex. For learning purposes, we approximate the value alignment and value system preferences in a differen- tiable manner with the BT model. Namely, we use the sigmoid (σ) of the reward difference between two gen- erations that share the same initial prompt: p(τ ≻ τ ′ | R) = σ(R(τ )− R(τ ′ )), as an approximation of the de- gree of preference (given as a probability between 0 and 1) of preferring the generation τ over τ ′ . To learn with this model, we modify the parameters of R so that the values of p(τ ≻ τ ′ | R) are closer to the preference la- bels y ∈0, 0.5, 1 by minimizing a binary cross-entropy loss (Christiano et al. 2017). Minimizing this loss, in Eq. 3, discordance between the model R and the dataset will also be decreased. We add a modification to the loss using the centering mechanism proposed in (Eisenstein et al. 2024), which reduces the magnitude of the estimated rewards and helps ensure that reward scales are comparable across values. This mechanism introduces a hyperparameter r, which we set to the recommended value of r = 0.01. L(τ,τ ′ ,y | R) =− y logp(τ ≻ τ ′ | R) − (1− y) log (p(τ ≺ τ ′ | R)) + r (R(τ ) + R(τ ′ )) 2 (3) Then, we propose a loss associated to the discordance of our model with the alignment preference examples for each value v i ∈ V in our dataset (Eq. 4). L θ v i (DS) = X (τ,τ ′ ,y v i )∈DS L(τ,τ ′ ,y v i | R R R θ ) |DS| (4) We consider all the previous losses together to de- fine a grounding (discordance) loss given by:L θ V (DS) = L θ v 1 (DS),...,L θ v m (DS) . And we define a loss associated to value system dis- cordance (Eq. 5). L ω,θ VS (DS) = X (τ,τ ′ ,y j V )∈DS L(τ,τ ′ ,y j V | W ω ·R R R θ ) |DS| (5) A na ̈ıve approach to use these losses to solve Prob- lem 2 would be first minimizing the average of the grounding discordance losses L θ V , and then, with θ frozen, learn the weights that minimize the value sys- tem discordance loss L ω,θ VS . However, it has been shown that multiple grounding functions are compatible with a low discordance while only a subset of those can also be effectively combined linearly to approximate value system (Holgado-S ́anchez et al. 2025) . Therefore, we propose a simultaneous learning ap- proach that balances both goals. The idea is treat- ing Problem 2 as a constrained optimization problem, where the constraint is minimizing the inner grounding discordance objective. In this formulation (Problem 6), we introduce the constraints that the grounding discor- dance losses in Eq. 4 should reach an (unknown, ap- proximated) per-value loss target L ∗ v i that depends on the capabilities of the available model. These targets L ∗ v i m i=1 are estimated during training by an “expo- nentially weighted minimum” of observed losses during training (see step 3 of the Algorithm description). (ω ∗ ,θ ∗ )∈ arg min W,R R R ′ L ω,θ ′ VS (DS) subject to θ ′ ∈θ |L θ v i (DS)≤L ∗ v i ∀i∈1,...,m(6) To approach Problem 6, we obtain its Lagrangian (Eq. 7). It introduces Lagrange multipliers λ λ λ = (λ 1 ,...,λ m ) ∈R + . Then, the optimization consists of finding a saddle point for the min-max goal in Eq. 8. L ω,θ λ λ λ (DS) =L ω,θ VS (DS) + m X i=1 λ i L θ v i (DS)−L ∗ v i (7) min θ,ω max λ λ λ>0 h L ω,θ λ λ λ (DS) i (8) Assuming that the reward vector R R R θ is an affine transformation (e.g. obtained by training only an ad- ditional single-layer perceptron over embeddings ex- tracted by a “base” LLM), Problem 6 is convex, and with additional minor assumptions, strong duality holds (Boyd and Vandenberghe 2004) , which justifies the theoretical use of dual ascent (Boyd et al. 2011) to find an optimal solution to Problem 8 (for fixed loss targets). Instead of dual ascent, which requires the ex- act minimization of the Lagrangian at every training iteration with respect to ω and θ, we approximate a so- lution with a modified gradient descent-ascent (GDA) algorithm (Zamani, Abbaszadehpeivasti, and de Klerk 2022), which converges to a local Nash equilibrium and even the global optimum with certain learning rate choices (Fallah, Ozdaglar, and Pattathil 2020). See sup- plementary material, Section A.2 for proofs of the pre- vious theoretical aspects. Before introducing the algorithm, we add some tech- nical modifications. First, to guarantee that the learn- ing step sizes on the Lagrangian remain constant, we add an auxiliar “value system multiplier”, λ VS , which is calculated so that the sum of the multipliers including this one have an average value of 1 3 . Second, to avoid any multiplier (including λ VS ) from vanishing, we force them to be over a certain user-defined threshold λ 0 > 0. 3 A similar multiplier normalization process was proposed for Safe RLHF methods (Dai et al. 2024). To satisfy the previous constraints, we use auxiliary pa- rameters ̄ λ λ λ = ( ̄ λ 1 ,..., ̄ λ m , ̄ λ VS )∈R m+1 to parametrize the “grounding multipliers” λ λ λ = (λ 1 ,...,λ m ) and the value system one λ VS , using: (λ 1 ,...,λ m ,λ VS ) = (m+1) exp ̄ λ λ λ P ̄ λ∈ ̄ λ λ λ exp ̄ λ (1−λ 0 ) +λ 0 . Finally, for theoretical reasons, we add ℓ 2 regularization to the Lagrangian, affecting all parameters ω,θ, ̄ λ λ λ. The Lagrangian used in our al- gorithm with the previous modifications is in Eq. 9. ̄ L ω,θ ̄ λ λ λ (DS) = λ VS L ω,θ VS (DS) + γ ω ∥ω∥ 2 + γ λ ∥ ̄ λ λ λ∥ 2 + m X i=1 λ i L θ v i (DS)−L ∗ v i + γ θ ∥θ∥ 2 (9) Algorithm description. After random parameter initialization, our algorithm proceeds by repeatedly ex- ecuting the following three steps over all batches in the dataset, for N epochs 4 : 1. Gradient descent. Calculate gradients of Eq. 9 with respect to the model parameters (ω and θ) for several batches (accumulating them). Then, perform gradient descent step to update ω and θ . 2. Gradient ascent. We perform a gradient ascent step on the Lagrange multiplier parameters ̄ λ λ λ based on the Lagrangian. In practice, for this update, we compute the gradient of a modified version of the La- grangian in Eq. 7 with grounding discordance losses measured on previous batches (BS): ∇ ̄ λ λ λ " m X i=1 λ i max(L θ v i (BS)−L ∗ v i , 0) + γ λ ∥ ̄ λ∥ 2 # With the “max” projection, this gradient is approx- imately zero (up to the effect of ℓ 2 regularization) for those multipliers whose associated grounding dis- cordance losses, computed over recent batches, have already met their target values. This design ensures that the primary objective remains reducing ground- ing discordance losses as much as possible, even be- yond the target estimations. The multipliers can still may still decrease slightly due to the ℓ 2 regulariza- tion term, which helps correct potential overestima- tions. It is noticeable that we do not update the value system multiplier λ VS based on the value sys- tem discordance loss. However, when the gradients of the grounding multipliers (λ λ λ = (λ 1 ,...,λ m ) van- ish, the ℓ 2 regularization progressively reduces the gap between λ VS and λ λ λ. Thus, only once the ground- ing discordance losses have reached their targets, the relative priority of minimizing the value system dis- cordance loss is subject to a slight increase. 3. Loss target update. After repeating steps 1-2 for a number of steps and batches (let this number be u > 0), we re-estimate the grounding discor- dance loss targets that are attainable with our re- ward model R R R θ . To do so, we employ a “exponen- tially weighted minimum” with coefficient η ∈ (0, 1) 4 An epoch here is a pass over the complete dataset. (close to 1) of the averages of the losses obtained during each period of u steps. Namely, given the calculation of the loss associated with each value v i across u different batches BS k u k=1 , and its cur- rent loss target estimationL ∗ v i , the update is:L ∗ v i ← ηL ∗ v i + (1− η) minL ∗ v i , 1 u P u k=1 L θ v i (BS k ). We set the initial value of the target losses as the average of the losses obtained in the first u batches. We supply a fully-detailed algorithm pseudocode in the supplementary material. 5 Evaluation We evaluate our proposal by comparing its performance against two baselines and the Armo-RM method (Wang et al. 2024b), a competitive reward model with the most similar architecture to our approach we found in the lit- erature. We analyse 2 datasets: UltraFeedback (Cui et al. 2023) and the text-to-text task in PKU-Align- Anything (Ji et al. 2024). We chose these datasets since they have been widely used for LLM alignment and because their structure is close to our dataset as- sumptions: indeed, in PKU-Align-Anything the an- notators are not identified and in UltraFeedback, the annotator is a single GPT-4 agent. Methods. Here we detail each method characteristics and training setting: • Armo-RM. A pretrained multi-objective reward model that predicts alignment with 19 attributes (that we treat as values) from six datasets. It uses a frozen Llama3-8B-v0.1 backbone and a linear head over the last hidden state acting similar to our re- ward vector R R R (grounding function). A scalar reward is obtained by linearly combining these outputs with weights W arm (p)∈ [−1, 1] 19 , produced by a prompt- conditioned three-layer MLP trained with MSE. • VSL-RM (ours). We use the same reward vector architecture as Armo-RM (a linear layer over the frozen Llama3-8B-v0.1 hidden state). We train it with our algorithm from scratch on both datasets. • BT-RM. A single-objective model that minimizes only the value system discordance loss (Eq. 5), with- out observing grounding discordance losses. It shares the same architecture and parameters as VSL-RM, and we also train one version per dataset. It serves as a lower bound on attainable value system discor- dance with the used architecture. • SEQ-RM. Shares the same architecture as VSL- RM but is trained sequentially: first to solely mini- mize grounding discordance (updating θ), then min- imizing the value system discordance (updating ω with θ frozen) for an additional 50% of epochs and higher learning rate. It serves both as a control base- line for comparing sequential vs. simultaneous train- ing and as oracle for minimum attainable grounding discordance with the used architecture. Training details. The datasets are split into train- ing, validation, and test sets. The validation data is used for hyperparameter selection, while test data is used only for final evaluation. All models are trained with four random seeds (shared across methods), except Armo-RM, which is used as provided on HuggingFace 5 . We trained the models (except Armo-RM) using ap- proximately the same set of common hyperparameters selected with a standard search method. The base- line models rely primarily on standard hyperparame- ters (e.g., batch size), which have limited impact on performance in this context, so we keep them aligned with those used for VSL-RM without further tuning. The only exception is SEQ-RM, for which we increase the value system learning rate to compensate for the smaller number of value system training iterations. All experiments were repeated with four random seeds. We observed little variation across runs, and therefore decided not to increase the number of training seeds. As optimizer, we used a paged, 32 bit “AdamW” one. Further details on hyperparameters, dataset pre- processing, and hardware are provided in supplemen- tary material, Section A.4. Metrics. For an easier interpretation of results, we report grounding and value system accuracies, which we define as the complement of the grounding and value system discordances, respectively: acc DS (R v i ,≼ v i ) = 1− d DS (≼ R v i ,≼ v i ), acc DS ((W,R R R),≼ j V ) = 1− d DS (≼ W R R R ,≼ j V ). Accuracy tends to 1 when each model perfectly repro- duces the preferences of the data (corresponding to a discordance of 0), and tends to 0 when discordance raises to its maximum (1). 5.1 Results Figure 2 presents training curves for the aver- age grounding accuracy (denoted by AGA= P m i=1 1 m acc DS (Rv i ,≼ v i )) and the value system accu- racy (denoted by VSA = acc DS ((W,R R R),≼ j V )). The curves depict the evolution of these accuracies through- out training, measured on the validation splits of each dataset. We do not present the curves for Armo-RM, as it was originally pretrained. As expected, BT-RM achieves highest VSA, as it is optimized solely for this objective, while VSL-RM clearly outperforms it in AGA due to the inclusion of the grounding discordance loss. Interestingly, AGA in- creases rapidly in both methods. Likely, this occurs be- cause of our architecture where the value system re- ward is a positive combination of a reward vector (cor- responding to the different values). Thus, a better pre- diction of the preferences of the value system correlates with a better value alignment estimation. Also in the case of BT-RM a reward vector is learned. However, 5 Armo RM model: https://huggingface.co/RLHFlow/ ArmoRM-Llama3-8B-v0.1, version from May 11th, 2026. 02468101214 Training Epoch 0.60 0.65 0.70 0.75 0.80 VSL-RM Value System Accuracy (VSA) VSL-RM Avg. Grounding Accuracy (AGA) BT-RM Value System Accuracy (VSA) BT-RM Avg. Grounding Accuracy (AGA) SEQ-RM Value System Accuracy (VSA) SEQ-RM Avg. Grounding Accuracy (AGA) 020406080100120140 Training Epoch 0.50 0.55 0.60 0.65 0.70 0.75 0.80 VSL-RM Value System Accuracy (VSA) VSL-RM Avg. Grounding Accuracy (AGA) BT-RM Value System Accuracy (VSA) BT-RM Avg. Grounding Accuracy (AGA) SEQ-RM Value System Accuracy (VSA) SEQ-RM Avg. Grounding Accuracy (AGA) Figure 2: Average grounding and value system accuracy per training method (VSL-RM, BT-RM, SEQ-RM) in the validation splits of UltraFeedback (Top) and PKU-Align-Anything (Bottom). It might be diffi- cult to appreciate, but AGA by VSL-RM and SEQ-RM almost coincide. The vertical line indicates the training moment where VSL-RM an BT-RM end their training and SEQ-RM starts its second phase to maximize VSA. this vector would not directly correspond to the under- lying values. Still, the learned vector is indirectly cor- related with the original values. This explains the fact that AGA increases rapidly for BT-RM (due to the pos- itive correlations) and then drops (since the reward vec- tor does not actually represent the original values). In VSL-RM, AGA still improves over the learning epochs. For SEQ-RM, both AGA and VSA improve during the first phase in both datasets, with VSA computed using uniform value system weights (1/m) 6 . Despite its simultaneous learning approach, VSL-RM matches the AGA achieved by SEQ-RM, which is optimized solely for grounding discordance. This indicates that VSL-RM effectively prioritizes learning grounding functions with the lowest discordance, as intended in Problem 2. In the second phase of SEQ-RM, VSA increases further, 6 VSA increases here due to the architectural effect de- scribed before: a good estimation of value alignemnt pref- erences, combined with positive weights, leads to a better estimation of value-based preferences. MethodHelpfulnessHonestyTruthfulnessInstruction FollowingAGAVSA Armo-RM 0.8260.8140.8400.8240.8260.754 BT-RM— 0.768 ±0.0002 VSL-RM 0.812 ±0.0005 0.793 ±0.0002 0.786 ±0.0007 0.815 ±0.0003 0.802 ±0.0002 0.760 ±0.0003 SEQ-RM 0.813 ±0.0003 0.793 ±0.0003 0.788 ±0.0008 0.815 ±0.0003 0.802 ±0.0002 0.758 ±0.0002 Table 1: Average grounding accuracy and value system accuracy (UltraFeedback). almost matching the one achieved with VSL-RM. We will show the gap is still statistically significative, by evaluating the test datasets. Table 1 (for UltraFeedback) and Table 2 (for PKU-Align-Anything) present the performance of the models on the test datasets. We omit the presen- tation of grounding accuracies in the case of BT-RM, as the model predictions in value alignment are latent constructs used to minimize value system discordance. All methods achieve comparable VSA across datasets, with BT-RM consistently performing best and Armo-RM exhibiting the weakest results. In the PKU- Align-Anything dataset, this could be expected, be- cause Armo-RM was not trained on it. However, Armo- RM was trained on UltraFeedback, and we expected that Armo-RM’s gating mechanism should, in prin- ciple, outperform our linear value system representa- tion in predicting overall preferences. This inaccuracy might be due to the fact that its training shared fo- cus with other 5 datasets with possibly divergent over- all preference criteria. As observed in Figure 2, BT- RM slightly outperforms VSL-RM regarding VSA in UltraFeedback and more substantially in PKU- Align-Anything, suggesting that, in the latter, value systems may not easily be approximated as linear func- tions of the dataset value alignment scores. This indi- cates that more expressive value system models will be necessary to improve performance. Comparing VSL-RM with SEQ-RM, the former con- sistently achieves marginally higher VSA, yet with sta- tistically significant differences (one-sided Welch t-tests: p = 0.0004 in PKU-Align-Anything and p < 0.0001 in UltraFeedback). This highlights a limitation of sequential optimization: although multiple grounding functions can achieve similar AGA, some are more suit- able for linear aggregation when minimizing value sys- tem discordance, as have been shown (Holgado-S ́anchez et al. 2025). The relatively small gap here is likely due to the regularization imposed on the grounding function implementation, given by the use of both ℓ 2 regulariza- tion and the centering parameter r. Regarding grounding accuracy, VSL-RM and SEQ- RM achieve comparable results, consistent with the training curves (differences are not statistically signif- icant 7 ). Notably, Armo-RM outperforms all methods in UltraFeedback, which may be attributed to its per-value training and the use of a MSE loss over quan- titative annotations. In contrast, our methods are re- stricted to less informative qualitative labels, and the use of cross-entropy losses that are prone to overfitting. This limitation could be mitigated in future work, e.g. by label smoothing (Zhu, Jordan, and Jiao 2024). Lastly, in Table 3, we analyze the value system weights predicted by VSL-RM and SEQ-RM 8 . Both models learned broadly similar value system represen- tations, assigning comparable importance to the values. This indicates that the simultaneous learning procedure in VSL-RM does not substantially distort the learned value system, compared to a grounding implemented by a reward vector learned without a bias to maximize VSA. Notably, however, VSL-RM yields less “radical” weight distributions than SEQ-RM: it maintains a min- imum weight of approximately 0.082 across all values, whereas SEQ-RM assigns some weights below 0.05. We consider that learning a value system model for a whole dataset that does not neglect any value achieves a better alignment with the hypotheses of the dataset creators, as they would expect that all values have a meaningful contribution towards explaining the overall preferences based on their proposed values. Our learned grounding functions and value system weights with VSL-RM, are, in combination, qualitatively better in this regard. As a conclusion, we have shown the advantages of an approach that defends an “intentional” interpretation of value alignment, by learning not only accurate ap- proximations of value groundings but also ones which facilitate their combination through value systems as a means for an accurate estimation of value-based prefer- ences. This is a fundamental advantage of our approach that is neglected by most existing methods that learn reward components and user preferences separately or sequentially, an approach that often leads to reward hacking in the fine-tuning phase (Zhou et al. 2024). 7 In supplementary material, we performed one-sided Welch t-tests to compare the metrics achieved by both meth- ods. 8 We omit the weights learned by BT-RM, as they are not based on valid value interpretations. MethodPrompt FollowingObjectivityClarityInformation RichnessSafetyAGAVSA Armo-RM 0.7150.7980.6090.6470.8370.7210.685 BT-RM— 0.773 ±0.0008 VSL-RM 0.765 ±0.0029 0.798 ±0.0030 0.720 ±0.0022 0.819 ±0.0029 0.890 ±0.0024 0.799 ±0.0005 0.714 ±0.0009 SEQ-RM 0.762 ±0.0014 0.806 ±0.0031 0.721 ±0.0024 0.818 ±0.0042 0.896 ±0.0014 0.801 ±0.0015 0.707 ±0.0015 Table 2: Average grounding and value system accuracies (PKU-Align-Anything). The accuracies for each PKU value are estimated by the scores given by its most related Armo-RM attribute (detailed in supplementary material). MethodHFHOTFIFPFOBCLIRSF VSL-RM 0.096 ±0.002 0.347 ±0.008 0.408 ±0.007 0.147 ±0.003 0.223 ±0.003 0.125 ±0.005 0.368 ±0.005 0.203 ±0.003 0.082 ±0.002 SEQ-RM 0.050 ±0.009 0.477 ±0.010 0.316 ±0.006 0.158 ±0.006 0.265 ±0.003 0.110 ±0.006 0.410 ±0.004 0.168 ±0.001 0.047 ±0.001 Table 3: Average and standard deviation of value system weights of VSL-RM and SEQ-RM in UltraFeedback (left) and PKU-Align-Anything (right). HF = Helpfulness, HO = Honesty, TF = Truthfulness, IF = Instruction Following, PF = Prompt Following, OB = Objectivity, CL = Clarity, IR = Information Richness, SF = Safety. 6 Conclusions and Future Work In this paper, we have put forward a principled ap- proach to value system learning for generative AI ap- plications. Specifically, we have introduced a learn- ing algorithm that, based on preference data, simul- taneously learns representations of the value align- ment (groundings) of prompt-response pairs (training a multi-objective reward model), and of their alignment with a certain value system (approximated by value im- portance weights). We have tested our solution against a single-objective reward learning baseline, a sequential variant of our algorithm, and a state-of-the art related algorithm. We have shown that our algorithm priori- tizes learning accurate value grounding models to serve as a correct basis for value system estimation, and thus fosters interpretability, with minor losses in capturing overall value-based preferences (value systems). Despite these positive results, our work also presents some limitations. First, even though our approach al- lows for explaining outcomes in terms of grounding functions and value system weights, interpretability is still limited by the fact that our algorithm performs an implicit aggregation of preferences regarding values and value systems of possibly divergent opinions, poten- tially obscuring minorities. Moreover, our current value system representation uses static linear weights, which in certain settings may put limits to the accuracy of modelling real value-based preferences. In future work, we will address these potential short- comings without sacrificing interpretability. For this purpose, we will introduce context-dependency into the value system model. A promising option is to dy- namically switch between different linear value systems based on contextual conditions. Another major line of work will address methods to learn diverse value sys- tems for heterogenous societies in datasets with known annotation sources. Alternative datasets that we will explore for either task are OpenAssistant (K ̈opf et al. 2023) and PRISM (Kirk et al. 2024). Still, the former involves tree-like conversations, and thus requires an extension of our present model which is geared towards single prompt-response preferences. The latter appears particularly suitable for studying the alignment of our approach with sociological findings. 7 Acknowledgments Thisworkhasbeensupportedbygrant COSASS:PID2021-123673OB-C32fundedby MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe”, and by project grant EVASAI: PID2024-158227NB-C32 funded by MICI- U/AEI/10.13039/501100011033/FEDER, UE. Andr ́es Holgado-S ́anchez has received funding by grant “Con- tratos Predoctorales de Personal Investigador en Formaci ́on en Departamentos de la Universidad Rey Juan Carlos (C1 PREDOC 2025)”, funded by Univer- sidad Rey Juan Carlos. Special thanks to the faculty ETSII at Rey Juan Carlos University for providing the computational resources used in the experiments. References Adams, J.; Hu, B.; Veenhuis, E.; Joy, D.; Ravichan- dran, B.; Bray, A.; Hoogs, A.; and Basharat, A. 2025. Steerable Pluralism: Pluralistic Alignment via Few- Shot Comparative Regression.Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(1): 15–25. Anderson, M.; Anderson, S. L.; and Armen, C. 2006. An approach to computing ethics. IEEE Intel ligent Sys- tems, 21: 56–63. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.; Amodei, D.; Brown, T.; Clark, J.; McCandlish, S.; Olah, C.; Mann, B.; and Kaplan, J. 2022a. Training a Helpful and Harmless As- sistant with Reinforcement Learning from Human Feed- back. arXiv:2204.05862. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; Chen, C.; Olsson, C.; Olah, C.; Hernan- dez, D.; Drain, D.; Ganguli, D.; Li, D.; Tran-Johnson, E.; Perez, E.; Kerr, J.; Mueller, J.; Ladish, J.; Lan- dau, J.; Ndousse, K.; Lukosuite, K.; Lovitt, L.; Sellitto, M.; Elhage, N.; Schiefer, N.; Mercado, N.; DasSarma, N.; Lasenby, R.; Larson, R.; Ringer, S.; Johnston, S.; Kravec, S.; Showk, S. E.; Fort, S.; Lanham, T.; Telleen- Lawton, T.; Conerly, T.; Henighan, T.; Hume, T.; Bow- man, S. R.; Hatfield-Dodds, Z.; Mann, B.; Amodei, D.; Joseph, N.; McCandlish, S.; Brown, T.; and Kaplan, J. 2022b. Constitutional AI: Harmlessness from AI Feed- back. arXiv:2212.08073. Beck, A. 2017. First-Order Methods in Optimization. Philadelphia, PA: Society for Industrial and Applied Mathematics. Bose, A.; Xiong, Z.; Chi, Y.; Du, S. S.; Xiao, L.; and Fazel, M. 2025. LoRe: Personalizing LLMs via Low- Rank Reward Modeling. In Second Conference on Lan- guage Modeling. Boyd, S.; Parikh, N.; Chu, E.; Peleato, B.; and Eck- stein, J. 2011. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Mul- tipliers. Found. Trends Mach. Learn., 3(1): 1–122. Boyd, S.; and Vandenberghe, L. 2004. Convex optimiza- tion. Cambridge university press. Bradley, R. A.; and Terry, M. E. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4): 324–345. Chakraborty, S.; Qiu, J.; Yuan, H.; Koppel, A.; Manocha, D.; Huang, F.; Bedi, A.; and Wang, M. 2024. MaxMin-RLHF: Alignment with Diverse Human Pref- erences. In Proc. 41st Int. Conf. on Machine Learn- ing, volume 235 of Proceedings of Machine Learning Re- search, 6116–6135. https://w.jmlr.org: JMLR.org. Chen, D.; Chen, Y.; Rege, A.; Wang, Z.; and Vinayak, R. K. 2025. PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment. In The Thirteenth International Conference on Learning Representations. Christiano, P. F.; Leike, J.; Brown, T. B.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from human preferences. In NIPS’17: Proceed- ings of the 31st International Conference on Neural In- formation Processing Systems, 4302–4310. Red Hook, NY, USA: Curran Associates Inc. Cui, G.; Yuan, L.; Ding, N.; Yao, G.; Zhu, W.; Ni, Y.; Xie, G.; Liu, Z.; and Sun, M. 2023. UltraFeedback: Boosting Language Models with High-quality Feed- back. arXiv:2310.01377. Dai, J.; Pan, X.; Sun, R.; Ji, J.; Xu, X.; Liu, M.; Wang, Y.; and Yang, Y. 2024. SAFE RLHF: SAFE REIN- FORCEMENT LEARNING FROM HUMAN FEED- BACK. In 12th International Conference on Learn- ing Representations, ICLR 2024. Red Hook, NY, USA: Curran Associates Inc. Publisher Copyright:© 2024 12th International Conference on Learning Represen- tations, ICLR 2024. All rights reserved.; 12th Interna- tional Conference on Learning Representations, ICLR 2024 ; Conference date: 07-05-2024 Through 11-05- 2024. Deng, K.; Wang, R.; Zhu, Z.; Zhang, J.; and Wen, Z. 2025. The Augmented Lagrangian Methods: Overview and Recent Advances. arXiv:2510.16827. Dognin, P.; Rios, J.; Luss, R.; Sattigeri, P.; Liu, M.; Padhi, I.; Riemer, M.; Nagireddy, M.; Varshney, K.; and Bouneffouf, D. 2025. Contextual Value Alignment. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1–5. Dong, Y.; Wang, Z.; Sreedhar, M.; Wu, X.; and Kuchaiev, O. 2023. SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 11275–11288. Singapore: Association for Compu- tational Linguistics. Eisenstein, J.; Nagpal, C.; Agarwal, A.; Beirami, A.; D’Amour, A.; Dvijotham, D.; Fisch, A.; Heller, K.; Pfohl, S.; Ramachandran, D.; Shaw, P.; and Berant, J. 2024. Helping or Herding? Reward Model Ensem- bles Mitigate but do not Eliminate Reward Hacking. arXiv:2312.09244. Fallah, A.; Ozdaglar, A.; and Pattathil, S. 2020. An Op- timal Multistage Stochastic Gradient Method for Min- imax Problems. In 2020 59th IEEE Conference on De- cision and Control (CDC), 3573–3579. Guo, Y.; Cui, G.; Yuan, L.; Ding, N.; Sun, Z.; Sun, B.; Chen, H.; Xie, R.; Zhou, J.; Lin, Y.; Liu, Z.; and Sun, M. 2024. Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. In Al- Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Pro- ceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, 1437–1454. Miami, Florida, USA: Association for Computational Linguis- tics. Gupta, R.; Sullivan, R.; Li, Y.; Phatale, S.; and Rastogi, A. 2025. Robust Multi-Objective Preference Alignment with Online DPO. Proceedings of the AAAI Conference on Artificial Intel ligence, 39(26): 27321–27329. Hadfield-Menell, D.; Milli, S.; Abbeel, P.; Russell, S. J.; and Dragan, A. 2017.Inverse Reward Design.In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fer- gus, R.; Vishwanathan, S.; and Garnett, R., eds., Ad- vances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc. Holgado-S ́anchez, A.; Billhardt, H.; Ossowski, S.; and Degli-Esposti, S. 2025. Learning the Value Systems of Societies from Preferences. In Proceedings of the 28th European Conference on Artificial Intel ligence (ECAI 2025), volume 413 of Frontiers in Artificial Intelligence and Applications, 1123–1130. IOS Press. Holgado-S ́anchez, A.; Billhardt, H.; Fern ́andez, A.; and Ossowski, S. 2026. Learning the value systems of agents with preference-based and inverse reinforcement learn- ing.Autonomous Agents and Multi-Agent Systems, 40(4): 4. Jang, J.; Kim, S.; Lin, B. Y.; Wang, Y.; Hessel, J.; Zettlemoyer, L.; Hajishirzi, H.; Choi, Y.; and Am- manabrolu, P. 2024.Personalized Soups: Personal- ized Large Language Model Alignment via Post-hoc Parameter Merging. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning. https://openreview.net: OpenReview. Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Chen, B.; Sun, R.; Wang, Y.; and Yang, Y. 2023. BEAVERTAILS: towards improved safety alignment of llm via a human-preference dataset. In Proceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS ’23. Red Hook, NY, USA: Curran Associates Inc. Ji, J.; Zhou, J.; Lou, H.; Chen, B.; Hong, D.; Wang, X.; Chen, W.; Wang, K.; Pan, R.; Li, J.; Wang, M.; Dai, J.; Qiu, T.; Xu, H.; Li, D.; Chen, W.; Song, J.; Zheng, B.; and Yang, Y. 2024. Align Anything: Training All- Modality Models to Follow Instructions with Language Feedback. arXiv:2412.15838. Karanik, M.; Billhardt, H.; Fern ́andez, A.; and Os- sowski, S. 2024.On the relevance of value sys- tem structure for automated value-aligned decision- making.In Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, 679–686. New York, NY, USA: Association for Computing Machinery. ISBN 9798400702433. Kirk, H. R.; Whitefield, A.; R ̈ottger, P.; Bean, A.; Margatina, K.; Ciro, J.; Mosquera, R.; Bartolo, M.; Williams, A.; He, H.; Vidgen, B.; and Hale, S. A. 2024. The PRISM alignment dataset: what participatory, rep- resentative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. In Proceedings of the 38th Inter- national Conference on Neural Information Processing Systems, NIPS ’24. Red Hook, NY, USA: Curran Asso- ciates Inc. ISBN 9798331314385. K ̈opf, A.; Kilcher, Y.; von R ̈utte, D.; Anagnostidis, S.; Tam, Z.-R.; Stevens, K.; Barhoum, A.; Duc, N. M.; Stanley, O.; Nagyfi, R.; ES, S.; Suri, S.; Glushkov, D.; Dantuluri, A.; Maguire, A.; Schuhmann, C.; Nguyen, H.; and Mattick, A. 2023. OpenAssistant conversa- tions - democratizing large language model alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23. Red Hook, NY, USA: Curran Associates Inc. Lera-Leri, R. X.; Liscio, E.; Bistaffa, F.; Jonker, C. M.; L ́opez-S ́anchez, M.; Murukannaiah, P. K.; Rodr ́ıguez- Aguilar, J. A.; and Salas-Molina, F. 2024. Aggregating value systems for decision support. Knowledge-Based Systems, 287: 111453. Li, C.; Zhang, H.; Xu, Y.; Xue, H.; Ao, X.; and He, Q. 2025. Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models. In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), 11214–11232. Vienna, Austria: Association for Computational Linguistics. ISBN 979- 8-89176-251-0. Liscio, E.; Lera-Leri, R.; Bistaffa, F.; Dobbe, R. I. J.; Jonker, C. M.; L ́opez-S ́anchez, M.; Rodr ́ıguez-Aguilar, J. A.; and Murukannaiah, P. K. 2023. Value Inference in Sociotechnical Systems. In Proc. AAMAS, Blue Sky Ideas Track, 1774 – 1780. Liscio, E.; van der Meer, M.; Siebert, L. C.; Jonker, C. M.; Mouter, N.; and Murukannaiah, P. K. 2021. Ax- ies: Identifying and Evaluating Context-Specific Values. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’21, 799–808. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450383073. Montes, N.; Osman, N.; Sierra, C.; and Slavkovik, M. 2023. Value Engineering for Autonomous Agents. arxiv:2302.08759. Montes, N.; and Sierra, C. 2022. Synthesis and proper- ties of optimally value-aligned normative systems. Jour- nal of Artificial Intel ligence Research, 74: 1739–1774. Osman, N.; and d’Inverno, M. 2024. A Computational Framework of Human Values. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’24, 1531–1539. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864. Ram ́e, A.; Couairon, G.; Shukor, M.; Dancette, C.; Gaya, J.-B.; Soulier, L.; and Cord, M. 2023. Rewarded soups: towards Pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23. Red Hook, NY, USA: Curran Associates Inc. Reddy, S.; Dragan, A.; Levine, S.; Legg, S.; and Leike, J. 2020. Learning Human Objectives by Evaluating Hypo- thetical Behavior. In I, H. D.; and Singh, A., eds., Pro- ceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, 8020–8029. https://w.jmlr.org: JMLR.org. Rodr ́ıguez-Soto, M.; R ̆adulescu, R.; Bistaffa, F.; Ricart, O.; Mayoral-Macau, A.; L ́opez-S ́anchez, M.; Rodr ́ıguez- Aguilar, J. A.; and Now ́e, A. 2026. Multi-objective rein- forcement learning for provably incentivising alignment with value systems. Artificial Intel ligence, 351: 104460. Rodr ́ıguez-Soto, M.; Serramia, M.; L ́opez-S ́anchez, M.; and Rodr ́ıguez-Aguilar, J. A. 2022. Instilling moral value alignment by means of multi-objective reinforce- ment learning. Ethics and Information Technology, 24: 9. Russell, S. 2022. Artificial Intelligence and the Problem of Control. In Werthner, H.; Prem, E.; Lee, E. A.; and Ghezzi, C., eds., Perspectives on Digital Humanism, 19– 24. Springer. Schwartz, S. H. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, 1–65. Elsevier. Serramia, M.; L ́opez-S ́anchez, M.; Rodr ́ıguez-Aguilar, J. A.; Rodr ́ıguez, M.; Wooldridge, M.; Morales, J.; and Ans ́otegui, C. 2018. Moral Values in Norm Decision Making. In Proceedings of the 17th International Con- ference on Autonomous Agents and MultiAgent Sys- tems, AAMAS ’18, 1294–1302. Richland, SC: Interna- tional Foundation for Autonomous Agents and Multia- gent Systems. Siebert, L. C.; Liscio, E.; Murukannaiah, P. K.; Kaptein, L.; Spruit, S.; Hoven, J. V. D.; and Jonker, C. 2022. Estimating Value Preferences in a Hybrid Partic- ipatory System. Frontiers in Artificial Intel ligence and Applications, 354: 114 – 127. Sinnott-Armstrong, W. 2023. Consequentialism. In Zalta, E. N.; and Nodelman, U., eds., The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2023 edition. Soares, N. 2018. The Value Learning Problem, 89–97. Chapman and Hall/CRC. Sorensen, T.; Jiang, L.; Hwang, J. D.; Levine, S.; Py- atkin, V.; West, P.; Dziri, N.; Lu, X.; Rao, K.; Bhaga- vatula, C.; Sap, M.; Tasioulas, J.; and Choi, Y. 2024. Value kaleidoscope: engaging AI with pluralistic hu- man values, rights, and duties. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intel li- gence and Thirty-Sixth Conference on Innovative Appli- cations of Artificial Intelligence and Fourteenth Sympo- sium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. AAAI Press. ISBN 978-1- 57735-887-9. Sumers, T.; Hawkins, R.; Ho, M. K.; Griffiths, T.; and Hadfield-Menell, D. 2022. How to talk so AI will learn: Instructions, descriptions, and autonomy. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Proceedings of the 36th International Conference on Neural Information Processing Systems, volume 35 of NIPS ’22, 34762–34775. Red Hook, NY, USA: Curran Associates, Inc. Tennant, E.; Hailes, S.; and Musolesi, M. 2025. Moral Alignment for LLM Agents. In The Thirteenth Inter- national Conference on Learning Representations. Vamplew, P.; Dazeley, R.; Foale, C.; Firmin, S.; and Mummery, J. 2018. Human-aligned artificial intelli- gence is a multiobjective problem. Ethics and Infor- mation Technology, 20(1): 27–40. Van Moffaert, K.; Drugan, M.; and Now ́e, A. 2013. Scalarized multi-objective reinforcement learning: novel design techniques. In 2013 IEEE Symposium on Adap- tive Dynamic Programming and Reinforcement Learn- ing (ADPRL), 191–199. IEEE. Wang, H.; Lin, Y.; Xiong, W.; Yang, R.; Diao, S.; Qiu, S.; Zhao, H.; and Zhang, T. 2024a. Arithmetic Control of LLMs for Diverse User Preferences: Directional Pref- erence Alignment with Multi-Objective Rewards. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), 8642–8655. Bangkok, Thailand: Association for Computational Linguistics. Wang, H.; Xiong, W.; Xie, T.; Zhao, H.; and Zhang, T. 2024b. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts.In Al- Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Find- ings of the Association for Computational Linguistics: EMNLP 2024, 10582–10592. Miami, Florida, USA: As- sociation for Computational Linguistics. Wang, X.; Le, Q.; Ahmed, A.; a Enmao Diao; Zhou, Y.; Baracaldo, N.; Ding, J.; and Anwar, A. 2025. MAP: Multi-Human-Value Alignment Palette. In The Thir- teenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025, 81284–81313. Red Hook, NY, USA: Curran Associates Inc. Wang, Z.; Dong, Y.; Delalleau, O.; Zeng, J.; Shen, G.; Egert, D.; Zhang, J. J.; Sreedhar, M. N.; and Kuchaiev, O. 2024c. HelpSteer2: open-source dataset for training top-performing reward models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24. Red Hook, NY, USA: Curran Associates Inc. ISBN 9798331314385. Yang, R.; Pan, X.; Luo, F.; Qiu, S.; Zhong, H.; Yu, D.; and Chen, J. 2024. Rewards-in-context: multi-objective alignment of foundation models with dynamic prefer- ence adjustment.In Proceedings of the 41st Inter- national Conference on Machine Learning, ICML’24. https://w.jmlr.org: JMLR.org. Zamani, M.; Abbaszadehpeivasti, H.; and de Klerk, E. 2022. Convergence rate analysis of the gradient descent- ascent method for convex-concave saddle-point prob- lems. arXiv:2209.01272. Zhou, X. 2018.On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradient. arXiv:1803.06573. Zhou, Z.; Liu, J.; Shao, J.; Yue, X.; Yang, C.; Ouyang, W.; and Qiao, Y. 2024. Beyond One-Preference-Fits- All Alignment: Multi-Objective Direct Preference Op- timization. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 10586–10613. Bangkok, Thai- land: Association for Computational Linguistics. Zhu, B.; Jordan, M. I.; and Jiao, J. 2024. Iterative data smoothing: mitigating reward overfitting and overopti- mization in RLHF. In Proceedings of the 41st Inter- national Conference on Machine Learning, ICML’24. JMLR.org. Ziegfeld, L.; Kox, E.; Akrum, I.; and Heijnen, M. 2025. Aligning AI Systems with Human Values: A Method for Identifying and Specifying Values. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(3): 2821–2831. A Appendix A.1 Source Code Source code is available at the following Github repos- itory and branch: https://github.com/andresh26-uam/ ValueLearningInGenAI/tree/AIES. A.2 Theoretical considerations This section provides the theoretical justification and key properties of the solution adopted in our problem formulation, recalled in Problem 10. The scope of the following is to show the soundness of the basic problem formulation, and provide justification for the decisions on our final algorithm. Our algorithm adds some details such as the normalization of the multipliers, that we omit for now for clarity. (ω ∗ ,θ ∗ )∈ arg min ω,θ ′ L ω,θ ′ VS (DS) subject to θ ′ ∈θ |L θ v i (DS)≤L ∗ v i (for i∈1,...,m)(10) We also recall the theoretical Lagrangian in Eq. 11, which employs the coefficient vector λ λ λ = (λ 1 ,...,λ m ) satisfying λ λ λ > λ 0 and P m i=1 λ i = m + 1. In addition, we explicitly incorporate the previously mentioned ℓ 2 regularization terms. L L L ω,θ λ λ λ (DS) =L VS (DS) + γ ω ||ω|| 2 + m X i=1 λ i L θ v i (DS)−L ∗ v i + γ θ ||θ|| 2 + γ λ λ λ ||λ λ λ|| 2 (11) Proposition 1 (Strong duality) Let L ∗ v i m i=1 de- note the exact attainable grounding discordance loss val- ues over the admissible parameter space Θ, i.e. L ∗ v i = min θ∈Θ L θ v i (DS). Assume that the parameter space Θ is dense and that R R R θ is affine in θ. Further assume that the loss targets L ∗ v i m i=1 in Problem 10 constitute a strict approxima- tion of the ideal loss values, i.e., L ∗ v i <L ∗ v i , ∀i∈1,...,m. Then, strong duality holds: the solutions of the min– max problem (12) and the max–min (dual) problem (13) coincide. min θ,ω max λ λ λ>0 h L L L ω,θ λ λ λ (DS) i (12) = max λ λ λ>0 min θ,ω h L L L ω,θ λ λ λ (DS) i (13) Proof. We observe that the loss functions are convex with respect to the difference in predicted rewards. In- deed, the log-sigmoid function is strictly concave, and the cross-entropy loss is a negative weighted combina- tion of concave functions; therefore, their composition is convex. Since the reward vector R R R θ is assumed to be affine in θ, the grounding discordance loss functions are, in particular, convex with respect to θ 9 . The value sys- tem discordance loss is also convex with respect to its parameters, since the reward differences on which it is computed are linear combinations of affine functions. As a consequence, Problem 10 is convex (Boyd and Van- denberghe 2004). Given that we assumed that the parameter space Θ is dense and the loss functions are, of course, continuous, we can consider the existence of arbitrary points θ for which L θ v i <L ∗ v i given any L ∗ v i >L ∗ v i . Thus, the Slater condition holds, i.e. ∃θ ∈ relint(Θ) s.t. L θ v i (DS) < L ∗ v i . Then, applying what is explained in Section 5.2.3 from (Boyd and Vandenberghe 2004), strong duality holds for any fixed combination of loss targetsL ∗ v i m i=1 . □ The preceding result motivates the use of dual as- cent methods to solve the dual problem (Boyd et al. 2011), whose solution coincides with that of the pri- mal (Problem 10). However, such methods require exact minimization of the Lagrangian with respect to θ and ω for a fixed λ λ λ at each iteration. We instead approximate the minimization step using gradient descent updates, resulting in an algorithm commonly referred to in game theory as gradient descent–ascent (GDA) (Zamani, Ab- baszadehpeivasti, and de Klerk 2022). To further justify our approach, we first observe that the ℓ 2 -regularized Lagrangian is strongly convex with 9 For deep networks, strong duality might not hold. An augmented Lagrangian formulation (Deng et al. 2025) could be used to overcome this issue, at the expense of solution complexity. respect to θ, ω, and λ λ λ. This follows from the fact that ℓ 2 regularization is itself strongly convex and, when added to convex loss terms, yields a strongly convex objective (Zhou 2018). Secondly, the logistic loss is L- smooth (Fallah, Ozdaglar, and Pattathil 2020), since its second derivative is bounded (see Theorem 5.12 in (Beck 2017)). Then, the Lagrangian is also L ′ - smooth, given that it is a linear combination of smooth functions. If gradient estimates computed on mini- batches are unbiased, i.e., they preserve the expected gradients of the full dataset, then not only gradient descent–ascent (GDA) (Zamani, Abbaszadehpeivasti, and de Klerk 2022), but also its stochastic counter- part (SGDA), with appropriately chosen step sizes, will converge to an optimal solution (Fallah, Ozdaglar, and Pattathil 2020). Our algorithm can be interpreted as a modified SGDA scheme in which the optimization prob- lem is updated at fixed intervals by updating the loss target estimates L ∗ v i m i=1 . We argue that this modifi- cation does not materially affect convergence, provided that the multipliers are initialized and updated so that the loss targets remain strictly decreasing toward the ideal optimal values L ∗ v i m i=1 (which can be achieved by experimentally choosing both an adequate ℓ 2 reg- ularization of multipliers, γ λ , and a sufficient learning rate α λ ). Our final algorithm has a slightly different La- grangian expression that adds the normalization of the multipliers and other changes. These changes do not invalidate the convexity requirements from before: they are only introduced to regulate the practical implemen- tation of the training algorithm. Namely, this makes learning step sizes similar across iterations and effec- tively avoid the multipliers from reaching λ 0 > 0, which would automatically make impossible to continually de- crease the loss target estimates L ∗ v i m i=1 . Of course, to completely prove these and the previous arguments, we still rely on the experimental validation shown in the main paper. A.3 Implementation details Transforming quantitative datasets into qualita- tive ones. The considered datasets provide quantita- tive annotations, namely a score S(τ ) that measures the degree of alignment of each generation τ with a given value or value system. However, both training and eval- uation in our framework require only qualitative super- vision in the form of pairwise preferences 10 . A qualitative label encoding the preference between a pair of generations (τ,τ ′ ) takes values y ∈1, 0.5, 0, corresponding to τ ≻ τ ′ , τ ≃ τ ′ , and τ ≺ τ ′ , respec- tively. These labels are derived directly from the avail- 10 Armo-RM was trained directly on quantitative annota- tions and therefore benefits from stronger supervision; we discuss this advantage explicitly in the evaluation section. able scores by comparing their differences: y = 1if S(τ ) > S(τ ′ ), 0if S(τ ) < S(τ ′ ), 0.5 otherwise. Qualitative interpretation of model outputs. Although the datasets consist of qualitative preferences, our models are quantitative: they assign a scalar re- ward R to each generation and value or value system to estimate the preference data. During learning, we employ the already mentioned Bradley-Terry probabil- ity measure of the preference between two responses, p(τ ≻ τ ′ | R). To obtain predicted qualitative la- bels, one could in principle apply the same rule as above by replacing S with R, or just estimating y with p(τ ≻ τ ′ | R). However, due to the intrinsic impre- cision of neural network outputs, exact equality cases R(τ ) = R(τ ′ ) (or p(τ ≻ τ ′ | R) = 0.5) are unlikely to occur in practice. To account for this, we introduce a tolerance thresh- old ε > 0. Whenever the absolute reward difference is smaller than ε, the two generations are considered in- distinguishable and the predicted label is set to ˆy = 0.5. Only when the difference exceeds this threshold do we predict a strict preference: ˆy ε (τ,τ ′ | R) = 1 if R(τ ) > R(τ ′ ) + ε 0 if R(τ ) < R(τ ′ )− ε 0.5 else (|R(τ )− R(τ ′ )|≤ ε) This ε introduces a counterpart threshold in the BT probability space, given by p ε = σ(ε)−0.5. This way, we can alternatively estimate the model output preference label ˆy ε (τ,τ ′ | R) using the BT model p(τ ≻ τ ′ | R), as follows: ˆy ε (τ,τ ′ | R) = 1 if p(τ ≻ τ ′ | R)− σ(ε) > 0.5 0 if p(τ ≻ τ ′ | R) + σ(ε) < 0.5 0.5 else (|p(τ ≻ τ ′ | R)− 0.5| < σ(ε)) We propose to select the threshold ε by analyzing the label granularity of the available quantitative datasets. Specifically, ε is chosen as half of the minimum non- zero difference between distinct score values. With this choice, adding or subtracting any perturbation ε ′ < ε to a score does not alter the induced qualitative preference after discretization. For instance, when the dataset uses an integer-valued scoring scheme, we set ε = 0.5. In this case, any score difference smaller than ε does not affect the resulting qualitative preference comparison between any pair of generations. This criterion was adopted throughout our experi- ments. Concretely, for UltraFeedback, we set ε = 0.25, as the overall value-alignment scores increase in in- crements of 0.5 over the range [0, 10]. For PKU-Align- Anything, we set ε = 0.5, since all annotations are provided on a Likert (1–5) scale. Chosen ε for calculating metrics in Table 1. As explained above, we also used ε = 0.25 in the Ultra- Feedback dataset and ε = 0.5 in the PKU-Align- Anything dataset for calculating the model predic- tions of VSL-RM and the newly-trained baselines. Re- garding Armo-RM, which was neither trained nor eval- uated using the proposed qualitative framework, we se- lected the values of ε that yielded maximum average grounding accuracy (AGA) and value system accuracy (VSA) over a predefined set of feasible values: ε∈0, 0.001, 0.01, 0.02, 0.05, 0.1, 0.25, 0.5. Empirically, we found that setting ε = 0.1 to esti- mate grounding accuracies and ε = 0.01 for value system accuracy resulted in the best overall perfor- mance in UltraFeedback. Regarding PKU-Align- Anything, these thresholds were set at ε = 0.1 and ε = 0.0, respectively. The choice of ε was observed to have a substantial impact on accuracy: for instance, in UltraFeedback, using the same threshold as in our proposed algorithms (ε = 0.25) led to a test AGA of only 0.7672, while VSA dropped to 0.586. Effects on the training process The introduction of the margin ε alters the relationship between mini- mizing the loss in Eq. 3 and minimizing discordances. In particular, a non-zero margin implies that small dif- ferences in alignment scores should not induce strict preferences (labels 0 or 1). To make the loss function explicitly aware of this tolerance—i.e., that the difference between alignment scores must exceed ε in order to predict labels 1 or 0—we modify the computation of the predicted pref- erence probability p(τ ≻ τ ′ | R) in Eq. 3 by introducing a label-dependent margin, following an approach simi- lar to that adopted in UltraRM (Cui et al. 2023): p(τ ≻ τ ′ | R)= △ Eq. (3) σ(R(τ )− R(τ ′ )− m(y)), where σ(·) denotes the logistic sigmoid and the margin term m(y) is defined as m(y) = +ε if y = 1, −ε if y = 0, 0if y = 0.5. Precise calculation of discordance. The theoreti- cal definition of discordance introduced in Section 4 do not explicitly specify how to handle the indifference la- bel y = 0.5, corresponding to τ ≃ τ ′ . To account for this case in a manner consistent with Eq. 1, we com- pute the discordance of a model R over a collection of labelled generation pairs XS⊂T ×T ×0, 0.5, 1, using the following empirical formulation: d XS (R) = 1 |XS| X (τ,τ ′ ,y)∈XS |y− ˆy ε (τ,τ ′ | R)|(14) MethodTrain. StepsWall-clock time VSL-RM (PKU)230006h 35m 14 ±55s BT-RM (PKU) 230006h 35m 8s ±2m SEQ-RM (PKU)345009h 33m 43s ±1m VSL-RM (UF)2645012h 1m 37s ±9m BT-RM (UF)2645011h 59m 30s ±58s SEQ-RM (UF) 3967515h 7m 29s ±3m Table 4: Wall-clock times per method and dataset across the 4 seeds (UF: UltraFeedback dataset, PKU: PKU-Align-Anything dataset). This definition assigns, for each element of XS, a dis- cordance of 1 when y = 0 and ˆy ε = 1 (or vice versa); a discordance of 0 when the label and its correspond- ing prediction coincide; and a discordance of 0.5 when y = 0.5 but ˆy ε ∈ 0, 1 (or vice versa). The latter case follows from the interpretation of the indifference rela- tion τ ≃ τ ′ (y = 0.5) as the simultaneous satisfaction of τ ≼ τ ′ and τ ≽ τ ′ . Consequently, predicting a strict inequality correctly satisfies one of the two conditions and is therefore penalized by half a unit (0.5) rather than a full discordance of 1. We must note a special case in the calculation of dis- cordances with Armo-RM. First, one of the six datasets used during the training of Armo-RM is the Ultra- Feedback dataset. Accordingly, Armo-RM predicts alignment scores for the values in UltraFeedback (along with 13 additional values), which we use to esti- mate grounding discordances in this dataset. However, Armo-RM does not provide predictions for the values in the PKU-Align-Anything dataset; so we predicted the PKU values with the most related one we found among the Armo-RM’s attributes (see the last subsec- tion in Appendix A.4 for details). Value system dis- cordance/accuracy is derived from the model’s overall score, which aggregates performance across all 19 val- ues. This metric can be computed for both datasets, as Armo-RM is designed for general-purpose reward benchmarking. A.4 Experiment details Hardware and Wall-clock times All experiments were carried out in two different GPU clusters. One used an AMD EPYC 9654 96-Core Processor with both NVIDIA L40S (48GB) and H100 (96GB). The other uses 2 CPU AMD 7742 64C/128T (256 threads total) and NVIDIA A100 GPUS (80GB). For each training run we employed a single GPU (of any type). Of course, we used the same hyperparameter and library settings across machines. Per trained model, we show the approximate wall- clock times in Table 4 (using the L40S graphics cards). Times may vary due to cluster congestion. Runs exe- cuted with the second cluster with A100 graphics cards were ∼40% slower. Attribute in Armo-RMValuein PKU- Align-Anything Instruction Fol lowing from UltraFeedback(Cui et al. 2023) Prompt Fol lowing Truthfulness from Ultra- Feedback (Cui et al. 2023) Objectivity Coherencefrom Help- steer (Wang et al. 2024c) Clarity Complexity from Help- steer (Wang et al. 2024c) Information Richness Safety from Beavertails (Ji et al. 2023) Safety Table 5: Mapping from the attibutes predicted by the multi-objective Armo-RM model to the values in the PKU dataset. We used the attributes in the first column to predict the alignment with the respective values in the PKU-Align-Anything dataset. The wall clock times apparently indicate that our novel algorithm (VSL-RM) has no significant overhead in training time with regard the baselines. However, we deduce the fact that all baselines and VSL-RM share the same codebase affects these results. Interestingly, yet no surprisingly, the first learning phase in SEQ-RM was about ∼17% faster than the full training of BT-RM’s or VSL-RM’s, despite these training settings all required the same number of iter- ations. We understand this ocurred because SEQ-RM avoided calculating the value system linear layer, which for backpropagation is a expensive process due to the softmax weight calculation. As a final note, we increased training epochs much beyond the necessary, to check convergence and possi- ble overfitting problems. Reasonably good models could be learned with less than 20% of the training epochs utilized in the experiments. Datasetpreprocessingandsplitting. The datasets are divided into training, validation, and test splits. Validation splits are used to monitor training and select hyperparameters, while test splits are used exclusively for performance evaluation and are never observed during training. • UltraFeedback. We use the version available in HuggingFace 11 . The dataset contains 383,796 feed- back annotations corresponding to 64,000 prompts. For each prompt, approximately four responses are provided, each annotated across four goals (that we treat as values). We reorganize the data by treat- ing each prompt–response pair as an independent 11 UltraFeedback paper (Cui et al. 2023), source: https://huggingface.co/datasets/openbmb/UltraFeedback. We observed the snapshot from May 11, 2026. instance, yielding 383,796 instances in the format of our dataset DS. Some value labels are missing for specific instances, so they were excluded from the computation of the metrics. UltraFeedback does not provide official training, validation, and test splits. We therefore construct them as follows: 10% of the data (38,380 instances) is reserved for testing. From the remaining data, 2% (6,909 instances) is used for validation, and the re- maining 98% (338,507 instances) is used for training. • PKU-Align-Anything (text-to-text). We use the text-to-text subset of the PKU dataset 12 , which contains 31,430 annotated prompt–response pairs. The annotations include quantitative value align- ment scores, qualitative value alignment preferences, and an overall qualitative preference label indicating the preferred response. We use the overall preference label as the value-system alignment label y j V (with a fixed j, as annotator identities are not provided). We use the quantitative alignment scores to generate re- spective value alignment preference labels between responses, as we found the qualitative annotations to be inconsistent with the quantitative ones and, wrongly, tend to align with the overall preference la- bel by taking into account other criteria apart from the values. The original dataset provides a training split (“train”) and a validation split (“val”). We treat “train” as a combined training–validation pool and “val” as the test dataset. From the combined pool, we select the first 29,430 instances for training and use the remaining 1,000 instances for validation. Approximating PKU values with Armo-RM at- tributes. Because Armo-RM was not trained on PKU, we decided to predict the alignment with the val- ues of the PKU dataset with the scores given to the most similar attributes in the Armo-RM reward model. These proxy scores are used as a measure of ground- ing accuracy in Table 2. The chosen correspondence be- tween PKU values and Armo-RM attributes is given in Table 5. Three of the mappings can be considered as a perfect equivalence, yet with care, as the distributions of the data are different. First, prompt fol lowing is certainly similar if not equal to instruction fol lowing from Ul- traFeedback. Second, objectivity is included in the defintion of truthfulness from UltraFeedback, as it includes being “faithful to factual knowledge”. Third, Safety is clearly already predicted by Armo-RM via the safety parallel in the Beavertails dataset (Ji et al. 2023). Hyperparameters. In Table 7, we report the full set of hyperparameters used in our experiments for each 12 PKU-Align-Anything paper (Ji et al. 2024), Hug- ging Face source: https://huggingface.co/datasets/PKU- Alignment/align-anything/viewer/text-to-text/. We ob- served the snapshot from May 11, 2026. Symbol Description NTraining epochs: Number of training passes (stepping through the necessary batches) across full training dataset. bBatch size. gr acc Gradient accumulation steps α θ ,α ω Learning rates for model parameters (θ, ω). α λ Learning rate for Lagrange multipliers. γ θ ,γ ω Weight decay factors for ℓ 2 regularization of the main model parameters θ and ω. γ λ Weight decay factor for the norm of the Lagrange multiplier parameters (ℓ 2 reg- ularization). ηUpdate ratio of the “exponentially weighted minimum” formula that calcu- lates the target losses L ∗ VS and L ∗ V . uNumber of passes through step 1-2 before updating the target losses L ∗ VS and L ∗ V . rReward centering coefficient (Eq. 3). Table 6: Glossary of hyperparameters used in the ex- periments. method. Explanations of individual hyperparameters are provided in the glossary tables (Table 6). These parameters were selected based on configurations that tended to minimize the average total discordance on the PKU-Align-Anything dataset when using VSL-RM. The values were obtained through a standard Bayesian search, with the number of training epochs restricted to a 10% of the epochs used in the final experiments. A.5 Algorithm pseudocode In Algorithm 1 we provide the pseudocode of the de- scribed algorithm in Section 4. A.6 Statistical significance tests In the main paper, we report several claims regarding the statistical significance of the differences between the test metrics obtained by VSL-RM and SEQ-RM. Here, we provide the detailed results of these statistical tests in Table 8 for the UltraFeedback dataset, and in Table 9 for the PKU-Align-Anything dataset. HyperparameterVSL-RMBT-RMSEQ-RM b323232 gr acc 444 α θ 0.00010.00010.0001 α ω 0.00010.00010.001 α λ 0.05-- γ θ 0.0030.0030.003 γ ω 0.0030.0030.003 γ λ 0.005-- η 0.9-- u20-- r 0.010.010.01 Epochs (N ) (UF|PKU)10| 10010| 10010 + 5| 100 + 50 Training steps (T ) (UF|PKU)26450| 2300026450| 2300026450 + 13225| 23000 + 11500 Table 7: Hyperparameters used in each environment per algorithm. Below, the number of epochs in each dataset (UF: UltraFeedback, PKU: PKU-Align-Anything. Method AMethod BMetricTestStatistic tOne-tailed p-valueSig. VSL-RMSEQ-RMHelpfulnessWelch t-test (A < B)-1.780.0695ns VSL-RMSEQ-RMHonestyWelch t-test (A < B)-0.600.2857ns VSL-RMSEQ-RMTruthfulnessWelch t-test (A < B)-2.870.0142* VSL-RMSEQ-RMInstruct. Follow.Welch t-test (A < B)-1.190.1406ns VSL-RMSEQ-RMAVG.Welch t-test (A < B)-4.060.0046** VSL-RMSEQ-RMVSAWelch t-test (A > B)13.150.0000*** Table 8: Pairwise one-tailed Welch t-tests comparing metrics achieved by VSL-RM and SEQ-RM in the Ultra- Feedback dataset: we test H 1 : μ A > μ B when Method A’s sample mean exceeds Method B’s, otherwise we test H 1 : μ A < μ B . Method AMethod BMetricTestStatistic tOne-tailed p-valueSig. VSL-RMSEQ-RMPrompt Follow.Welch t-test (A > B)1.470.1050ns VSL-RMSEQ-RMObject.Welch t-test (A < B)-3.640.0055** VSL-RMSEQ-RMClarityWelch t-test (A < B)-0.380.3580ns VSL-RMSEQ-RMInf. Rich.Welch t-test (A > B)0.200.4263ns VSL-RMSEQ-RMSafetyWelch t-test (A < B)-4.090.0053** VSL-RMSEQ-RMAvg. Chr.Welch t-test (A < B)-2.850.0254* VSL-RMSEQ-RMRepresent.Welch t-test (A > B)7.290.0004*** Table 9: Pairwise one-tailed Welch t-tests comparing metrics achieved by VSL-RM and SEQ-RM in the PKU-Align- Anything dataset: we test H 1 : μ A > μ B when Method A’s sample mean exceeds Method B’s, otherwise we test H 1 : μ A < μ B . Algorithm 1: Value System Learning with Reward Models via Stochastic Gradient Descent-Ascent (VSL-RM) Let DS a training dataset with the structure described in the paper. Set a desired λ 0 > 0 and number of training epochs N . Choose hyperparameters: batch size b > 0, gradient accumulation steps gr acc , learning rates α θ ,α ω ,α λ , ℓ 2 regularization weights γ θ ,γ ω ,γ λ , and exponential weighted minumum update coefficient η < 1 (e.g. approx 0.9). Initialize trainable parameters θ,ω, ̄ λ λ λ∈R m+1 and gradient accumulator variables g θ ,g ω = 0. 1: for e = 1 to N do Divide DS into batches of size b: BS k , k ∈1,...,K. 2:for k = 1,...,K do 3:Calculate and save value system discordance lossL ω,θ VS (BS k ), and grounding discordance losses: ,L θ v i (BS k ), for each i∈1,...,m. 4:if e = 1∧ k ≤ u then 5:if k mod u = 0 then 6:L ∗ v i ← P u k ′ =0 L θ v i (BS k ′ )▷ Initialization of loss targets 7:end if 8:continue 9:end if 10:Calculate normalized multipliers: (λ 1 ,...,λ m ,λ VS ) = (m + 1) exp( ̄ λ λ λ) P ̄ λ∈ ̄ λ λ λ exp( ̄ λ) (1− λ 0 ) + λ 0 11:(1) Gradient descent ̄ L ω,θ ̄ λ λ λ (BS k ) = λ VS L ω,θ VS (BS k ) + m X i=1 λ i L θ v i (BS k )−L ∗ v i + γ θ ∥θ∥ 2 + γ ω ∥ω∥ 2 + γ λ ∥ ̄ λ λ λ∥ 2 12:Accumulate gradients: g θ ← g θ +∇ θ ̄ L ω,θ ̄ λ λ λ (BS k ); g ω ← g ω +∇ ω ̄ L ω,θ ̄ λ λ λ (BS k ) 13:if k mod gr acc = 0 then 14:θ ← θ− α θ gr acc · g θ 15:ω ← ω− α ω gr acc · g ω 16:Reset gradient accumulators g θ ← 0, g ω ← 0 17:(2) Gradient ascent on multipliers ̄ λ λ λ← ̄ λ λ λ + α λ ∇ ̄ λ λ λ m X i=1 λ i max k X k ′ =k−gr acc +1 1 gr acc L θ v i (BS k ′ ) −L ∗ v i , 0 + γ λ ∥ ̄ λ λ λ∥ 2 18:end if 19:if k mod u = 0 then 20:(3) Target loss update 21:for i = 1,...,m do 22:L ∗ v i ← ηL ∗ v i + (1− η) min L ∗ v i , 1 u P k k ′ =k−u+1 L θ v i (BS k ′ ) 23:end for 24:end if 25:end for 26: end for