Paper deep dive
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, Matthieu Cord
Models: BLIP-2, LLaMA-7B, Stable Diffusion 2.2B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 8:08:22 PM
Summary
The paper introduces 'Rewarded Soups' (RS), a multi-policy strategy for aligning foundation models with diverse human preferences. By independently fine-tuning networks on different proxy rewards and then linearly interpolating their weights, the authors demonstrate that they can achieve Pareto-optimal generalization across multiple objectives without the computational overhead of traditional Multi-Objective Reinforcement Learning (MORL). The approach is validated across text-to-text, text-to-image, and control tasks, confirming linear mode connectivity and Pareto-optimality.
Entities (5)
Relation Signals (3)
Llama → finetunedvia → Reinforcement Learning from Human Feedback
confidence 95% · We begin our experiments with text-to-text generation. Our pre-trained network is LLaMA-7b, instruction fine-tuned on Alpaca.
Rewarded Soups → utilizes → Linear Mode Connectivity
confidence 95% · This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards
Rewarded Soups → improvesupon → Multi-objective Reinforcement Learning
confidence 90% · RS alleviates MORL’s scaling issue as it requires only M=N trainings while being flexible and transparent.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfections in the proxy reward may hinder the training and lead to suboptimal results; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather than focusing on a single a priori reward, we aim for Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards from a shared pre-trained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
132,189 characters extracted from source content.
Expand or collapse full text
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards Alexandre Rame 1∗ , Guillaume Couairon 1,2† , Mustafa Shukor 1† , Corentin Dancette 1† ,Jean-Baptiste Gaya 1,2† ,Laure Soulier 1 ,Matthieu Cord 1,3 1 Sorbonne Université, CNRS, ISIR, Paris, France 2 Meta AI 3 Valeo.ai Abstract Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfec- tions in the proxy reward may hinder the training and lead to suboptimalresults; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather than focusing on a single a priori reward, we aim for Pareto-optimal generalization across the entire space of preferences. To this end, we proposerewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards from a shared pre-trained initialization. We demon- strate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity. 1 Introduction Foundation models [1] have emerged as the standard paradigm to learn neural networks’ weights. They are typically first pre-trained through self-supervision [2,3,4,5] and then fine-tuned [6,7] via supervised learning [8]. Yet, collecting labels is expensive, and thus supervision may not cover all possibilities and fail to perfectly align [9,10,11] the trained network with the intended applications. Recent works [12,13,14] showed that deep reinforcement learning (DRL) helps by learning from various types of rewards. A prominent example is reinforcement learning from human feedback (RLHF) [12,15,16,17], which appears as the current go-to strategy to refine large language models (LLMs) into powerful conversational agents such as ChatGPT [13,18]. After pre-training on next token prediction [19] using Web data, the LLMs are fine-tuned to follow instructions [20,21,22] before reward maximization. This RL strategy enhances alignment by evaluating the entire generated sentence instead of each token independently, handling the diversity of correct answers and allowing for negative feedback [23]. Similar strategies have been useful in computer vision (CV) [14,24], for instance to integrate human aesthetics into image generation [25, 26, 27]. Diversity of proxy rewards.RL is usually seen as more challenging than supervised training [28], notably because the real reward—ideally reflecting the users’ preferences—is often not specified at training time. Proxy rewards are therefore developed to guide the learning, either as hand-engineered metrics [29,30,31] or more recently in RLHF as models trained to reflect human preferences ∗ Project lead, main contributor, correspondence to alexandre.rame@isir.upmc.fr. † Equal experimental contribution, order determined at random. Further information and resources related to this project can be found on this website. 37th Conference on Neural Information Processing Systems (NeurIPS 2023). arXiv:2306.04488v2 [cs.LG] 16 Oct 2023 Rewarded soups by weight interpolation Selection selected by the users according to their preferences RL fine-tuning of proxy reward RL fine-tuning of proxy reward Unsupervised pre-training on large dataset Supervised fine-tuning on target task (a) Illustration of our proposed rewarded soup (RS). 1.01.21.41.61.8 R 1 −1.8 −1.6 −1.4 −1.2 −1.0 −0.8 R 2 LLaMA init R 2 rewarded (μ=λ= 1) RS forλ= 0.5 MORL forμ= 0.5 R 1 rewarded (μ=λ= 0) RS front: ( 1−λ ) ·θ 1 +λ·θ 2 λ MORL front: θ (1−μ)×R 1 +μ×R 2 μ (b) LLaMA RLHF for summarization. Figure 1: Figure 1(a) details the different steps in rewarded soup. After unsupervised pre-training and supervised fine-tuning, we launchNindependent RL fine-tunings on the proxy rewardsR i N i=1 . Then we combine the trained networks by interpolation in the weight space. The final weights are adapted at test time by selecting the coefficientλ. Figure 1(b) shows our results (extended in Figure 2(a)) with LLaMA-7b [44] instruct fine-tuned on Alpaca [22], when RL fine-tuning for news summarization [12] withN= 2reward models assessing diverse preferences of summaries. With only two trainings (R 1 andR 2 rewarded on Figure 1(b)), theλ-interpolation (0≤λ≤1) reveals the green front of Pareto-optimal solutions, i.e., that cannot be improved for one reward without sacrificing the other. RS matches the costly yellow front of multi-objective (MORL) [45,46] requiring multiple trainings on different linear weightings over the rewards(1−μ)×R 1 +μ×R 2 with0≤μ≤1. [15,32,33]. Nonetheless, designing reliable proxy rewards for evaluation is difficult. Thisreward misspecification[9,34] between the proxy reward and the users’ actual rewards can lead to unforeseen consequences [35]. Moreover, the diversity of objectives in real-world applications complicates the challenge. In particular, human opinions can vary significantly [36,37,38] on subjects such as aesthetics [39], politics or fairness [40]. Humans have also different expectations from machines: for example, while [41] stressed aligning LLMs towards harmless feedback, [42] requested helpful non-evasive responses, and others’ [43] interests are to make LLMs engaging and enjoyable. Even hand-engineered metrics can be in tension: generating shorter descriptions with higher precision can increase the BLEU [29] score but decrease the ROUGE [30] score due to reduced recall. Towards multi-policy strategies.Considering these challenges, a single model cannot be aligned with everyone’s preferences [13]. Existing works align towards a consensus-based user [47,48], relying on the “wisdom of the crowd” [49], inherently prioritizing certain principles [42,50], resulting in unfair representations of marginalized groups [51,52]. The trade-offs [53] are decided a priori before training, shifting the responsibility to the engineers, reducing transparency and explainability [54], and actually aligning towards the “researchers designing the study” [13,55]. These limitations, discussed in Appendix A.1, highlight the inability of single-policy alignment strategies to handle human diversity. Yet, “human-aligned artificial intelligence is a multi-objective problem” [56]. Thus, we draw inspiration from the multi-objective reinforcement learning (MORL) literature [45,46,57, 58,59,60,61,62] and [54]; they argue that tackling diverse rewards requires shifting from single- policy to multi-policy approaches. As optimality depends on the relative preferences across those rewards, the goal is not to learn a single network but rather aset of Pareto-optimal networks[63]. In this paper, we proposerewarded soup(RS), an efficient and flexible multi-policy strategy to fine-tune any foundation model. As shown in Figure 1(a), we first use RL to learn one network for each proxy reward; then, we combine these expert networks according to user preferences. This a posteriori selection allows for better-informed trade-offs, improved transparency and increased fairness [54,64]. The method to combine those networks is our main contribution: we do this through linear interpolation in the weight space, despite the non-linearities in the network. This is in line with recent findings on linear mode connectivity (LMC) [65,66]: weights fine-tuned from a shared pre-trained initialization remain linearly connected and thus can be interpolated. This LMC inspired a plethora of weight interpolation (WI) strategies [67,68,69,70,71,72], discussed in Section 4. Actually, the namerewarded soupsfollows the terminology ofmodel soups[67], as we combine variousingredientseach rewarded differently. Unlike previous works, which focused on supervised learning, we explore LMC in RL, in a challenging setup where each training run uses a different reward. Perhaps surprisingly, we show that we can trade off the capabilities of multiple weights in a 2 single final model, thus without any computational overhead. This enables the creation of custom weights for any preference over the diverse rewards. We summarize our contributions as follows: •We advocate a multi-policy paradigm to align deep generative models with human prefer- ences and reduce reward misspecification. •We then propose a new multi-policy strategy, rewarded soup, possible when fine-tuning foundation models with diverse rewards. By weight interpolation, it defines a continuous set of (close to) Pareto-optimal solutions, approximating more costly multi-policy strategies. In Section 3, we consistently validate the linear mode connectivity and thus the effectiveness of RS across a variety of tasks and rewards: RLHF fine-tuning of LLaMA, multimodal tasks such as image captioning or text-to-image generation with diffusion models, as well as locomotion tasks. 2 Rewarded soups 2.1 RL fine-tuning with diverse rewards We consider a deep neural networkfof a fixed non-linear architecture (e.g., with batch normalization [73], ReLU layers [74] or self-attention [75]). It defines a policy by mapping inputsxtof(x,θ) when parametrized byθ. For a reward ˆ R (evaluating the correctness of the prediction according to some preferences) and a test distributionTof deployment, our goal is to maximize R x∈T ˆ R(f(x,θ)). For example, withfa LLM,xwould be textual prompts, ˆ Rwould evaluate if the generated text is harmless [76], andTwould be the distribution of users’ prompts. Learning the weightsθis now commonly a three-step process: unsupervised pre-training, supervised fine-tuning, and reward optimization. Yet ˆ R is usually not specified before test time, meaning we can only optimize a proxy rewardRduring training. Thisreward misspecificationbetweenRand ˆ R may hinder the alignment of the network with ˆ R . Moreover, thediversity of human preferencescomplicates the design ofR. Rather than optimizing one single proxy reward, our paper’s first key idea is to consider a family ofNdiverse proxy rewardsR i N i=1 . Each of these rewards evaluates the prediction according to different (potentially conflicting) criteria. The goal then becomes obtaining a coverage set of policies that trade-off between these rewards. To this end, we first introduce the costly MORL baseline. Its inefficiency motivates our rewarded soups, which leverages our second key idea: weight interpolation. MORL baseline.The standard MORL scalarization strategy [45,46] (recently used in [62] to align LLMs) linearizes the problem by interpolating the proxy rewards usingMdifferent weightings. Specifically, during thetraining phase,Mtrainings are launched, with thej-th optimizing the reward P N i=1 μ j i R i , where∀j∈1,...,M,μ j i N i=1 ∈∆ N theN-simplex s.t. P N i=1 μ j i = 1 and 0≤μ j i ≤1. Then, during theselection phase, the user’s reward ˆ Rbecomes known and thej-th policy that maximizes ˆ R on some validation dataset is selected. We typically expect to selectjsuch that P N i=1 μ j i R i ≈ ˆ R linearly approximates the user’s reward. Finally, thisj-th weight is used during theinference phaseon test samples. Yet, a critical issue is that “minor [preference] variations may result in significant changes in the solution” [77]. Thus, a high level of granularity in the mesh of∆ N is necessary. This requires explicitly maintaining a large set ofM≫Nnetworks, practically one for each possible preference. Ultimately, this MORL strategy is unscalable in deep learning due to the computational, memory, and engineering costsinvolved (see further discussion in Appendix A.2). Rewarded soup (RS).In this paper, we draw inspiration from the weight interpolation literature. The idea is to learn expert weights and interpolate them linearly to combine their abilities. Specifically, we propose RS, illustrated in Figure 1(a) and whose recipe is described below. RS alleviates MORL’s scaling issue as it requires onlyM=Ntrainings while being flexible and transparent. 1.During thetraining phase, we optimize a set ofNexpert weightsθ i N i=1 , each correspond- ing to one of theNproxy rewardsR i N i=1 , and all from a shared pre-trained initialization. 2.For theselection phase, we linearly interpolate those weights to define a continuous set of rewarded soups policies: P N i=1 λ i ·θ i λ i N i=1 ∈∆ N . Practically, we uniformly sampleM 3 interpolating coefficientsλ j i N i=1 M j=1 from theN-simplex∆ N and select thej-th that maximizes the user’s reward ˆ Ron validation samples, i.e.,argmax M j=1 ˆ R P N i=1 λ j i θ i . 3. For theinference phase, we predict using the networkfparameterized by P N i=1 λ j i θ i . While MORL interpolates the rewards, RS interpolates the weights.This is a considerable advantage as the appropriate weightingλ, which depends on the desired trade-off, can be selecteda posteriori; the selection is achieved without additional training, only via inference on some samples. In the next Section 2.2 we explicitly state the Hypotheses 1 and 2 underlying in RS. These are consideredWorking Hypothesesas they enabled the development of our RS strategy. Their empirical verification will be the main motivation for our experiments on various tasks in Section 3. 2.2 Exploring the properties of the rewarded soups set of solutions 2.2.1 Linear mode connectivity of weights fine-tuned on diverse rewards We considerθ i N i=1 (orθ i i for brevity) fine-tuned onR i i from a shared pre-trained initialization. Previous works [65,66,67,72] defined linear mode connectivity (LMC) w.r.t. a single performance measure (e.g., accuracy or loss) in supervised learning. We extend this notion in RL withNrewards, and define that the LMC holds if all rewards for the interpolated weights exceed the interpolated rewards. It follows that the LMC condition which underpins RS’s viability is the Hypothesis 1 below. Working Hypothesis 1(LMC).∀λ i i ∈∆ N andk∈1,...,N,R k ( P i λ i ·θ i )≥ P i λ i R k (θ i ). 2.2.2 Pareto optimality of rewarded soups The Pareto front (PF) is the set of undominated weights, for which no other weights can improve a reward without sacrificing another, i.e.,θ|∄θ ′ ∈Θs.t.R i (θ ′ ) i > N R i (θ) i where> N is the dominance relation inR N . In practice, we only need to retain one policy for each possible value vector, i.e., a Pareto coverage set (PCS). We now introduce the key Hypothesis 2, that state the Pareto-optimality of the solutions uncovered by weight interpolation in RS. Working Hypothesis 2(Pareto optimality).The set P i λ i ·θ i |λ i i ∈∆ N is a PCS ofR i i . Empirically, in Section 3, we consistently validate Hypotheses 1 and 2. Theoretically, in Appendix C.2, we prove they approximately hold, in a simplified setup (quadratic rewards with co-diagonalizable Hessians) justifiable when weights remain close. Remark 1.Hypotheses 1 and 2 rely on a good pre-trained initialization, making RS particularly well- suited to fine-tune foundation models. This is because pre-training prevents the weights from diverging during training [66]. When the weights remain close, we can theoretically justify Hypotheses 1 and 2 (see Appendix C.2) and, more broadly, demonstrate that WI approximates ensembling [78,79] (see Lemma 4). In contrast, the LMC does not hold when training from scratch [66]. Neuron permutations strategies [80,81] tried to enforce connectivity by aligning the weights, though (so far) with moderate empirical results: their complementarity with RS is a promising research avenue. Remark 2.Pareto-optimality in Hypothesis 2 is defined w.r.t. a set of possible weightsΘ. Yet, in full generality, improvements in initialization, RL algorithms, data, or specific hyperparameters could enhance performances. In other words, for real-world applications, the true PF is unknown and needs to be defined w.r.t. a training procedure. In this case,Θrepresents the set of weights attainable by fine-tuning within a shared procedure. As such, in Section 3 we analyze Hypothesis 2 by comparing the fronts obtained by RS and scalarized MORL while keeping everything else constant. 2.2.3 Consequences of Pareto optimality if the user’s reward is linear in the proxy rewards Lemma 1(Reduced reward misspecification in the linear case).If Hypothesis 2 holds, and for linear reward ˆ R= P i ˆμ i R i withˆμ i i ∈∆ N , then∃λ i i ∈∆ N such that P i λ i ·θ i is optimal for ˆ R. The proof outlined in Appendix C.1 directly follows the definition of Pareto optimality. In simpler terms, Lemma 1 implies that if Hypothesis 2 holds, RS mitigates reward misspecification for linear rewards: for any preferenceˆμ, there exists aλsuch that theλ-interpolation over weights maximizes theˆμ-interpolation over rewards. In practice, as we see in Figure 5(a), we can setλ= ˆμ, or cross-validateλon other samples. 4 3 Experiments In this section we implement RS across a variety of standard learning tasks: text-to-text generation, image captioning, image generation, visual grounding, visual question answering, and locomotion. We use either model or statistical rewards. We follow a systematic procedure. First, we independently optimize diverse rewards on training samples. For all tasks, we employ the default architecture, hyperparameters and RL algorithm; the only variation being the reward used across runs. Second, we evaluate the rewards on the test samples: the results are visually represented in series of plots. Third, we verify Hypothesis 1 by examining whether RS’s rewards exceed the interpolated rewards. Lastly, as the true Pareto front is unknown in real-world applications, we present empirical support for Hypothesis 2 by comparing the front defined by RS (slidingλbetween0and1) to the MORL’s solutions optimizing theμ-weighted rewards (sometimes onlyμ= 0.5for computational reasons). Implementations are released on github, and this website provides additional qualitative results. 3.1 Text-to-text: LLaMA with diverse RLHFs Given the importance of RLHF to train LLMs, we begin our experiments with text-to-text generation. Our pre-trained network is LLaMA-7b [44], instruction fine-tuned [20,83] on Alpaca [22]. For RL training with PPO [84], we employ the trl package [85] and the setup from [86] with low-rank adapters (LoRA) [87] for efficiency. We first consider summarization [12,17] tasks on two datasets: Reuter news [88] in Figures 1(b) and 2(a) and Reddit TL;DR [89] in Figure 2(b). We also consider answering Stack Exchange questions [90] in Figure 2(c), movie review generation in Figure 2(d), and helpfulness as a conversational assistant [49] in Figures 2(e) and 2(f). To evaluate the generation in 0.40.60.81.01.21.41.61.8 R 1 −1.8 −1.6 −1.4 −1.2 −1.0 −0.8 R 2 LLaMA init R 2 rewarded (μ=λ= 1.0) λ= 0.9λ= 0.8 λ= 0.7 λ= 0.6 λ= 0.5 λ= 0.4 λ= 0.3 λ= 0.2 λ= 0.1 R 1 rewarded (μ=λ= 0.0) RLR 1 :μ= 0.0 MORL:μ= 0.1 MORL:μ= 0.2 MORL:μ= 0.3 MORL:μ= 0.4 MORL:μ= 0.5 MORL:μ= 0.6 MORL:μ= 0.7 MORL:μ= 0.8 MORL:μ= 0.9 RLR 2 :μ= 1.0 MORL front RS front (a) News summary. −0.20.00.20.40.60.81.01.2 R 1 −0.6 −0.4 −0.2 0.0 0.2 0.4 R 2 LLaMA init R 2 rewarded (μ=λ= 1.0) λ= 0.9λ= 0.8λ= 0.7λ= 0.6 λ= 0.5 λ= 0.4 λ= 0.3 λ= 0.2 λ= 0.1 R 1 rewarded (μ=λ= 0.0) RLR 1 :μ= 0.0 MORL:μ= 0.1 MORL:μ= 0.2 MORL:μ= 0.3 MORL:μ= 0.4 MORL:μ= 0.5 MORL:μ= 0.6 MORL:μ= 0.7 MORL:μ= 0.8 MORL:μ= 0.9 RLR 2 :μ= 1.0 MORL front RS front (b) Reddit TL;DR summary. −2.5−2.0−1.5−1.0−0.50.0 R 1 −2.25 −2.00 −1.75 −1.50 −1.25 −1.00 −0.75 −0.50 R 2 LLaMA init R 1 rewarded R 2 rewarded R 1 +R 2 2 rewarded λ= 0.5 RL:R 1 RL:R 2 MORL: R 1 +R 2 2 RS: (1−λ)·θ 1 +λ·θ 2 (c) Stack Exchange Q&A. 0.51.01.52.02.5 R 1 −1.0 −0.5 0.0 0.5 1.0 1.5 R 2 LLaMA init R 1 rewarded R 2 rewarded R 1 +R 2 2 rewarded λ= 0.5 RL:R 1 RL:R 2 MORL: R 1 +R 2 2 RS: (1−λ)·θ 1 +λ·θ 2 (d) Useful movie review. −2.9−2.8−2.7−2.6−2.5−2.4−2.3−2.2 R 1 −3.25 −3.00 −2.75 −2.50 −2.25 −2.00 −1.75 −1.50 R 2 LLaMA init R 1 rewarded R 2 rewarded R 1 +R 2 2 rewarded λ= 0.5 RL:R 1 RL:R 2 MORL: R 1 +R 2 2 RS: (1−λ)·θ 1 +λ·θ 2 (e) Helpful assistant. R 1 R 2 R 3 R 4 -2.626 -2.451 -2.276 -2.604-2.079-1.555 -0.937 -0.712 -0.487 -2.409-1.736-1.063 LLaMA init RL:R 1 RL:R 2 RL:R 3 RL:R 4 MORL: ∑ N=4 i=1 R i /4 RS: ∑ N=4 i=1 θ i /4 (f) Helpful assistant: spider map. Figure 2: RLHF results in NLP with LLaMA-7b [44] and reward modelsR i from HuggingFace [82]. The blue line reports checkpoints’ results along the training trajectory ofθ 1 rewardingR 1 , the red lineθ 2 rewarding R 2 , and the purple line the MORL rewarding R 1 +R 2 2 . Our rewarded soup (RS) linearly interpolates between the weightsθ 1 andθ 2 ; sliding the interpolation coefficientλfrom0to1reveals the green solid front of rewarded soups solutions. In Figures 2(a) and 2(b), we additionally show the multiple MORL runs rewarding (1−μ)×R 1 +μ×R 2 with preferences0≤μ≤1. It reveals a similar yellow front, yet more costly. In Figure 2(f), we uniformly (λ i = 1 4 ) average the weights fine-tuned for the assistant task onN= 4reward models. 5 the absence of supervision, we utilizedN= 2different reward models (RMs) for each task, except in Figure 2(f) whereN= 4. These RMs were trained on human preferences datasets [15] and all open-sourced on HuggingFace [82]. For example in summarization,R 1 follows the “Summarize from Human Feedback” paper [12] and focuses on completeness, whileR 2 leverages “contrast candidate generation” [91] to evaluate factuality. For other tasks, we rely on diverse RMs from OpenAssistant [92]; though they all assess if the answer is adequate, they differ by their architectures and procedures. Table 1 details the experiments. The results are reported in Figure 2. The green front, defined by RS between the two weights special- ized onR 1 andR 2 , is above the straight line connecting those two points, validating Hypothesis 1. Second, the front passes through the point obtained by MORL fine-tuning on the average of the two rewards, supporting Hypothesis 2. Moreover, when comparing both full fronts, they have qualitatively the same shape; quantitatively in hypervolume [93] (lower is better, the area over the curve w.r.t. an optimal point), RS’s hypervolume is 0.367 vs. 0.340 for MORL in Figure 2(a), while it is 1.176 vs. 1.186 in Figure 2(b). Finally, in Figure 2(f), we useN= 4RMs for the assistant task and uniformly average theN= 4weights, confirming that RS can scale and trade-off between more rewards. 3.2 Image-to-text: captioning with diverse statistical rewards RL is also effective for multimodal tasks [14] such as in image captioning [24], to generate textual descriptions of images. Precisely evaluating the quality of a prediction w.r.t. a set of human-written 0.8250.8300.8350.8400.8450.850 R 1 :BLEU1 0.603 0.604 0.605 0.606 0.607 0.608 0.609 0.610 0.611 R 2 : ROUGE pre-trained init μ=λ= 0 μ=λ= 1 λ= 0.1 λ= 0.2 λ= 0.3 λ= 0.4 λ= 0.5λ= 0.6λ= 0.7λ= 0.8λ= 0.9 RLBLEU1:μ= 0 MORL:μ= 0.1 MORL:μ= 0.2 MORL:μ= 0.3 MORL:μ= 0.4 MORL:μ= 0.5 MORL:μ= 0.6 MORL:μ= 0.7 MORL:μ= 0.8 MORL:μ= 0.9 RLROUGE:μ= 1 MORL front RS front (a) BLEU1 and ROUGE. 0.8250.8300.8350.8400.8450.850 R 1 :BLEU1 0.408 0.410 0.412 0.414 0.416 0.418 0.420 0.422 0.424 R 2 : BLEU 4 pre-trained init BLEU1 rewarded BLEU4 rewarded BLEU1+BLEU4 2 rewarded λ= 0.5 RL:BLEU1 RL:BLEU4 MORL: BLEU1+BLEU4 2 RS: (1−λ)·θ BLEU1 +λ·θ BLEU4 (b) BLEU1 and BLEU4. BLEU1 BLEU4 ROUGEMETEOR CIDEr 0.806 0.828 0.85 0.386 0.405 0.424 0.603 0.607 0.61 0.303 0.307 0.312 1.343 1.37 1.396 Pre-trained init RL:BLEU1 RL:BLEU4 RL:ROUGE RL:METEOR MORL: ∑ N=5 i=1 R i /5 RS: ∑ N=5 i=1 θ i /5 (c) Captioning: spider map. Figure 3: Results in image captioning on COCO [94]. As rewardsR 1 (blue stars every epoch) andR 2 (red stars), we consider standard statistical metrics: BLEU1 (1-gram overlap), BLEU4 (4-grams overlap), ROUGE, METEOR and CIDEr. Figure 3(a) include the MORL training trajectories optimizing(1−μ)×BLEU1 + μ×ROUGE, uncovering a yellow front similar to RS’s green front. In Figure 3(c), RS uniformly averages the 5weights (one for each reward), resulting in the largest area and the best trade-off between the5rewards. BLEU1 BLEU4 ROUGEMETEOR CIDEr 0.806 0.828 0.85 0.386 0.405 0.424 0.603 0.607 0.61 0.303 0.307 0.312 1.343 1.37 1.396 ∑ M=1 i=1 θ i /1: B1 ∑ M=2 i=1 θ i /2: B1, B4 ∑ M=3 i=1 θ i /3: B1, B4, R ∑ M=4 i=1 θ i /4: B1, B4, R, M ∑ M=5 i=1 θ i /5: B1, B4, R, M, C (a) BLEU1 first. BLEU1 BLEU4 ROUGEMETEOR CIDEr 0.806 0.828 0.85 0.386 0.405 0.424 0.603 0.607 0.61 0.303 0.307 0.312 1.343 1.37 1.396 ∑ M=2 i=2 θ i /1: B4 ∑ M=3 i=2 θ i /2: B4, R ∑ M=4 i=2 θ i /3: B4, R, M ∑ M=5 i=2 θ i /4: B4, R, M, C ∑ M=5 i=1 θ i /5: B4, R, M, C, B1 (b) BLEU4 first. BLEU1 BLEU4 ROUGEMETEOR CIDEr 0.806 0.828 0.85 0.386 0.405 0.424 0.603 0.607 0.61 0.303 0.307 0.312 1.343 1.37 1.396 ∑ M=3 i=3 θ i /1: R ∑ M=4 i=3 θ i /2: R, M ∑ M=5 i=3 θ i /3: R, M, C ∑ M=5 i6=2 θ i /4: R, M, C, B1 ∑ M=5 i=1 θ i /5: R, M, C, B1, B4 (c) ROUGE first. Figure 4: Those spider maps uniformly average1≤M≤5weights for captioning, whereθ 1 is fine-tuned on BLEU1 (B1),θ 2 on BLEU4 (B4),θ 3 on ROUGE (R),θ 4 on METEOR (M) andθ 5 on CIDEr (C). To show different combinations among the 5 M possible, we iterate in a clockwise direction starting in Figure 4(a) from i= 1(always includingθ 1 optimized on BLEU1), in Figure 4(b) fromi= 2(always includingθ 2 optimized on BLEU4), and in Figure 4(c) fromi= 3(always includingθ 3 optimized on ROUGE). 6 0.00.20.40.60.81.0 ˆ μ 0.6 0.8 1.0 1.2 1.4 1.6 (1 − ˆ μ ) × BLEU 1 Norm + ˆ μ × ROUGE Norm RSλ:MORLμ: 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ˆμ CVλ 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ˆμ CVμ (a) Analysis of Lemma 1. 0.00.20.40.60.81.0 λ 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Normalized rewards R 1 :BLEU1 BLEU2 BLEU3 BLEU4 R 2 :ROUGE METEOR CIDEr (b) Performances w.r.t.λ. 0.8250.8300.8350.8400.8450.850 R 1 :BLEU1 0.603 0.604 0.605 0.606 0.607 0.608 0.609 0.610 0.611 R 2 : ROUGE BLEU1 rewarded ROUGErewarded λ= 0.5 RS: (1−λ)·θ BLEU1 +λ·θ ROUGE ENS: (1−λ)·f(·,θ BLEU1 ) +λ·f(·,θ ROUGE ) (c) RS vs. prediction ensembling. Figure 5: Results in captioning forR 1 =BLEU1andR 2 =ROUGE. When normalized, rewards are set to 1 for the init and 0 for the worst model. Figure 5(a) validates Lemma 1 by reporting results of RS (for varying λ) and of MORL (for varyingμ) for varying user’s preferenceˆμ. Figure 5(b) evaluates different rewards as a function of the interpolating coefficient. Figure 5(c) reports ensembling scores when interpolating predictions. captions is challenging, thus the literature relies on various non-differentiable metrics: e.g., the precision-focused BLEU [29], the recall-focused ROUGE [30], METEOR [95] handling synonyms and CIDEr [31] using TF-IDF. As these metrics are proxies for human preferences, good trade-offs are desirable. We conduct our experiments on COCO [94], with an ExpansionNetv2 [96] network and a Swin Transformer [97] visual encoder, initialized from the state-of-the-art weights of [96] optimized on CIDEr. We then utilize the code of [96] and their self-critical [24] procedure (a variant of REINFORCE [98]) to reward the network on BLEU1, BLEU4, ROUGE or METEOR. More details and results can be found in Appendix E. We observe in Figure 3 that tuning solely BLEU1 sacrifices some points on ROUGE or BLEU4. Yet interpolating betweenθ 1 andθ 2 uncovers a convex set of solutions approximating the ones obtained through scalarization of the rewards in MORL. When comparing both full fronts in Figure 3(a), they qualitatively have the same shape, and quantitatively the same hypervolume [93] of 0.140. One of the strengths of RS is its ability to scale to any number of rewards. In Figure 3(c), we uniformly (λ i = 1 5 ) averageN= 5weights fine-tuned independently. It improves upon the initialization [96] and current state-of-the-art on all metrics, except for CIDEr, on which [96] was explicitly optimized. We confirm in Figure 4 that RS can handle more than2rewards through additional spider maps. Specifically, we compare the performances across allN= 5metrics when averaging1≤M≤Nnetworks (each fine-tuned on one of theNrewards, thus leaving outN−Mrewards at training) and sequentially adding more networks to the weight average. We consistently observe that adding one additional network specialized on one additional reward extends the scope of the possible rewards that RS can tackle Pareto-optimally. Figure 5 refines our analysis of RS. Figure 5(a) validates Lemma 1: for any linear preferenceˆμover the proxy rewards, there exists an optimal solution in the set described by RS. Two empirical strategies to set the value ofλare close to optimal: selectingλ= ˆμifˆμis known, or cross-validating (CV) λif a different data split [99] is available. Moreover, Figure 5(b) (and Appendix E) investigate all metrics as evaluation. Excluding results’ variance, we observe monotonicity in both training rewards, linear in BLEU1 and quadratic in ROUGE. For other evaluation rewards thatcannot be linearly expressedover the training rewards, the curves’ concavity shows that RS consistently improves the endpoints, thereby mitigating reward misspecification. The optimalλdepends on the similarity between the evaluation and training rewards: e.g., best BLEU2 are with smallλ. Lastly, as per [100] and Lemma 4, Figure 5(c) suggests that RS succeeds because WI approximatesprediction ensembling [78,79] when weights remain close, interpolating the predictions rather than the weights. Actually, ensembling performs better, but it cannot be fairly compared as its inference cost is doubled. 7 3.3 Text-to-image: diffusion models with diverse RLHFs Beyond text generation, we now apply RS to align text-to-image generation with human feedbacks [25,26,33]. Our network is a diffusion model [101] with 2.2B parameters, pre-trained on an internal dataset of 300M images; it reaches similar quality as Stable Diffusion [102], which was not used for copyright reasons. To represent the subjectivity of human aesthetics, we employN= 2open-source reward models:ava, trained on the AVA dataset [103], andcafe, trained on a mix of real-life and manga images. We first generate 10000 images; then, for each reward, we remove half of the images with the lowest reward’s score and fine-tune 10% of the parameters [104] on the reward-weighted negative log-likelihood [25]. Details and generations for visual inspection are in Appendix F. The results displayed in Figure 6(a) validate Hypothesis 1, as the front described by RS when sliding λfrom0and1is convex. Moreover, RS gives a better front than MORL, validating Hypothesis 2. Interestingly, theavareward model seems to be more general-purpose thancafe, as RL training on avaalso enhances the scores ofcafe. In contrast, the modelθ cafe performs poorly in terms ofava in Figure 6(a). Nonetheless, RS with(1−λ)·θ ava +λ·θ cafe outperformsθ ava alone, not only in terms ofcafe, but also ofavawhenλ∈0.1,0.2. These findings confirm that RS can better align text-to-image models with a variety of aesthetic preferences. This ability to adapt at test time paves the way for a new form of user interaction with text-to-image models, beyond prompt engineering. 5.225.245.265.285.30 R 1 :ava 0.780 0.785 0.790 0.795 0.800 0.805 0.810 0.815 0.820 R 2 : cafe pre-trained init ava rewarded cafe rewarded μ= 0.5 λ= 0.5 λ= 0.1 λ= 0.2 MORL front: θ (1−μ)×R ava +μ×R cafe μ RS front: ( 1−λ ) ·θ ava +λ·θ cafe λ (a) Image generation:avaandcafe. 21.621.822.022.222.422.6 R 1 :Small 12.4 12.5 12.6 12.7 12.8 R 3 : Large pre-trained init Small rewarded Large rewarded Small+Large 2 rewarded RS: (1−λ)·θ Small +λ·θ Large (b) VG: Small and Large. Figure 6: Figure 6(a) reports our RLHF experiments on text-to-image generation with diffusion models. From the pre-trained initialization, we learnθ ava andθ cafe by optimizing the two reward modelsavaandcafe. Interpolation between them reveals the green Pareto-optimal front, above the yellow MORL front. Figure 6(b) report our results in visual grounding (VG) on RefCOCO+ [105], where we optimize to predict boxes with IoU>0.5w.r.t. the ground-truth, for objects of either small, medium or large size. 3.4 Text-to-box: visual grounding of objects with diverse sizes We now consider visual grounding (VG) [105]: the task is to predict the bounding box of the region described by an input text. We use UnIVAL [106], a seq-to-seq model that predicts the box as a sequence of location tokens [107]. This model is pre-trained on a large image-text dataset, then fine-tuned with cross-entropy for VG; finally, we use a weighted loss between the cross-entropy and REINFORCE in the RL stage. As the main evaluation metric for VG is the accuracy (i.e., intersection over union (IoU)>0.5), we consider 3 non-differentiable rewards: the accuracy on small, medium, and large objects. We design this experiment because improving results on all sizes simultaneously is challenging, as shown in Figure 19(c), where MORL performs similarly to the initialization. The results in Figure 6(b) confirm that optimizing for small objects degrades performance on large ones; fortunately, interpolating can trade-off. In conclusion, we can adapt to users’ preferences at test time by adjustingλ, which in turn changes the object sizes that the model effectively handles. On the one hand, if focusing on distant and small objects, a large coefficient should be assigned toθ Small . On the other hand, to perform well across all sizes, we can recover initialization’s performances by averaging uniformly (in Figure 19(c)). More details are in Appendix G. 8 3.5 Text&image-to-text: VQA with diverse statistical rewards We explore visual question answering (VQA), where the task is to answer questions about images. The models are usually trained with cross-entropy, as a classification or text generation task, and evaluated using the VQA accuracy: it compares the answer to ten ground truth answers provided by different annotators and assigns a score depending on the number of identical labels. Here, we explore the fine-tuning of models using the BLEU (1-gram) and METEOR metrics: in contrast with accuracy, these metrics enable assigning partial credit if the ground truth and predicted answers are not identical but still have some words in common. In practice, we use the OFA model [107] (generating the answers token-by-token), on the VQA v2 dataset, pre-trained with cross-entropy, and fine-tuned with REINFORCE during the RL stage. More details can be found in Appendix H. Our results in Figure 7(a) validate the observations already made in previous experiments: RL is efficient to optimize those two rewards, and RS reveals a Pareto-optimal front. 52.052.252.452.652.853.053.253.453.6 R 1 :BLEU 2.45 2.50 2.55 2.60 2.65 2.70 2.75 2.80 R 2 : METEOR pre-trained init BLEU rewarded METEOR rewarded BLEU+METEOR 2 rewarded λ= 0.5 RS: (1−λ)·θ BLEU +λ·θ METEOR (a) VQA: BLEU and METEOR. 3500400045005000550060006500 R 1 :velocity 0 1000 2000 3000 4000 R 2 : velocity − ∑ t a 2t pre-trained init risky rewarded cautious rewarded risky+cautious 2 rewarded λ= 0.5 RL:velocity RL:velocity− ∑ t a 2 t MORL:velocity− 1 2 ∑ t a 2 t RS: (1−λ)·θ risky +λ·θ cautious (b) Locomotion: risky and cautious. Figure 7: Figure 7(a) report our results for visual question answering. Figure 7(b) report our results from Section 3.6 for the locomotion task with humanoids. 3.6 Locomotion with diverse engineered rewards Teaching humanoids to walk in a human-like manner [108] serves as a benchmark to evaluate RL strategies [109] for continuous control. One of the main challenges is to shape a suitable proxy reward [110,111], given the intricate coordination and balance involved in human locomotion. It is standard [112] to consider dense rewards of the formR=velocity−α× P t a 2 t , controlling the agent’s velocity while regularizing the actionsa t t taken over time. Yet, the penalty coefficientαis challenging to set. To address this, we devised two rewards in the Brax physics engine [113]: a risky R 1 withα= 0, and a more cautiousR 2 withα= 1. Like in all previous tasks, RS’s front in Figure 7(b) exceeds the interpolated rewards, as per Hy- pothesis 1. Moreover, the front defined by RS indicates an effective balance between risk-taking and cautiousness, providing empirical support for Hypothesis 2, although MORL withμ= 0.5(i.e., α= 0.5) slightly surpasses RS’s front. We provide animations of our RL agent’s locomotion on our website, and more details are in Appendix I. 3.7 Efficiency gain of RS over MORL The efficiency gain of RS versus MORL is by design; when considering2rewards, RS only requires 2fine-tunings, while MORL actually requires an infinite number of fine-tunings to reveal the entire front of preferences. To end this experimental section, we quantify this efficiency gain by introducing in Figure 8 the expected rewardE ˆμ∼Unif(0,1) ˆ R ˆμ where ˆ R ˆμ = (1−ˆμ)×R 1 + ˆμ×R 2 and the expectation is over all the possible user’s preferencesˆμ. We then measure the difference between the expected rewards for RS (with2runs) and MORL (withMruns). Plotting this expected reward advantage for different values ofMshows that MORL needsM≫2to match RS. 9 246810 Number of MORL trainings. 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 RS expected advantage over MORL (a) News summary. 246810 Number of MORL trainings. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 RS expected advantage over MORL (b) Reddit summary. 246810 Number of MORL trainings. 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 0.0030 0.0035 0.0040 RS expected advantage over MORL (c) Captioning. Figure 8:Expected reward advantage of RS (always requiring only2trainings) over MORL (withMtrainings), defined asE ˆμ∼Unif(0,1) h max λ∈Λ ˆ R ˆμ (θ RS λ )−E Λ M h max μ∈Λ M ˆ R ˆμ (θ MORL μ ) i , where ˆ R ˆμ = (1−ˆμ)×R 1 + ˆμ×R 2 is the user reward for user linear preferenceˆμsampled uniformly between0 and1,Λ =0,0.1,...,1.0is the set of the11possible values forλ, and where the expectation for the MORL term is over the 11 M possible combinationsΛ M ofMelements fromΛ(representing theMlinear weightings μused for MORL training). We observe that MORL matches RS only forMsufficiently big. 4 Related work Our RS approach leans on two key components from traditional DRL. The first isproxy rewards, whose design is challenging. Statistical metrics (the standard in captioning [24]) are not practical to measure human concepts [32] such as helpfulness [49,76]. Thus recent RLHF works [12,13,15] leverage human comparison of prediction to learn a reward model. Second, RS relies on existing RL algorithmsto maximize the given rewards. RS succeeds with variants of two of the most common, REINFORCE [98] and PPO [84], suggesting it could be applied to others [114,115]. When dealing with multiple objectives in deep learning, the common strategy is to combine them into a single reward [59,60]. For example, [116] sum the predictions of a preference RM (as a proxy for helpfulness) and a rule RM (detecting rules breaking); [62] assign different weightings to the relevance/factuality/completeness rewards, thereby customizing how detailed and lengthy the LLMs responses should be. Yet, thosesingle-policyapproaches (optimizing over a single set of linear preferences) force a priori and uncertain decisions about the required trade-offs [52,54], as further detailed in Appendix A.1. Themulti-policyalternatives [45,46,57,58,61] are not suitable because of the computational costs required to learn set of policies. To reduce the cost, [117,118,119,120] build experts and then train a new model to combine them; [121,122,123] share weights across experts; [124,125,126,127] directly train a single model; the recent and more similar work [128] learns one linear embedding per (locomotion) task. Yet, all those works were developed for academic benchmarks [112,129]; moreover, in terms of Pareto-optimality, they perform equal or worse than the linearized MORL. As far as we know, the only approaches that might improve performances are those inspired from the multitask literature [130], tackling gradients conflicts [131,132] or different variance scales [133,134] across tasks. Though they succeed for games such as ATARI [135], our attempts to apply [131] in our setups failed. Overall, as previous MORL works modify the training procedure and usually introduce specific hyperparameters, adapting them to RLHF for foundation models with PPO is complex; in contrast, RS can be used on top of any RLHF system. Finally, performance and simplicity are not the only advantages of RS over other MORL approaches; in brief, and as discussed in Appendix A.2, RS is compatible with the iterative alignment process. Recent works extended thelinear mode connectivitywhen fine-tuning on different tasks [70,71,72, 136], modalities [106] or losses [68,137], while [138] highlighted some failures in text classification. In contrast, we investigate the LMC in RL. The most similar works are for control system tasks: [139] averaging decision transformers and [140] explicitly enforcing connectivity in subspaces of policies trained from scratch on a single reward. When the LMC holds, combining networks in weights combines their abilities [141,142]; e.g., averaging an English summarizer and an English-to-French translator can summarize in French [143]. In domain generalization, [67,68,144] showed that WI reduces model misspecification [145]; by analogy, we show that RS reduces reward misspecification. 10 5 Discussion: limitations and societal impacts The recent and rapid scaling of networks presents both opportunities and major concerns [9,146,147]. Our approach is a step towards betterempirical alignment[10,11]. Yet, many challenges remain untackled. First, proxy rewards may lack robustness [148] or be hacked [149] via adversarial exploita- tion, making them unreliable. Second, overfitting during training may lead to poor generalization, with a risk of goal misgeneralization [150,151]. RS could alleviate the impact of some badly shaped proxy rewards and some failed optimizations, as well as tackling Goodhart’s law [152]. Yet, without constraint on the test distribution, complete alignment may be impossible [153], for example for LLMs with prompts of arbitrary (long) length. Theoretical guaranteesfor alignment are also needed [154]. Yet, RS (as all weight interpolation strategies) relies on an empirical finding: the LMC [65], which currently lacks full theoretical guarantees, even in the simplest case of moving averages [100]. That’s why we state explicitly our Working Hypotheses1 and 2 in Section 2.2. Nonetheless, we want to point out that in Appendix C.2 we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards; specifically, in Lemma 3, we bound the reward difference between the optimal policy and our interpolated policy. A remaining limitation is that we theoretically fix issues only for ˆ R linear over the proxy rewards. Suchlinearizationfollows thelinear utility functionssetup from the MORL literature [60], that cannot encapsulate all types of (human) preferences [56,77]. Nonetheless, we showed in Figures 5(b) and 13 that RS improves results even when ˆ Ris not linear. We may further improve results by continually training on new and diverse proxy rewards, to capture the essential aspects of all possible rewards, such that their linear mixtures have increasingly good coverage. Finally, our a posteriori alignment with users facilitatespersonalization[155] of models. As dis- cussed in Appendix A.1 and in [52], this could increase usefulness by providing tailored generation, notably to under-represented groups. Moreover, the distributed nature of RS makes it paralleliz- able thus practical in a federated learning setup [156] where data must remain private. Yet, this personalization comes with risks for individuals of “reinforcing their biases [...] and narrowing their information diet”[52]. This may worsen the polarization of the public sphere. Under these concerns, we concur with the notion of “personalization within bounds” [52], with these boundaries potentially set by weights fine-tuned on diverse and carefully inspected rewards. 6 Conclusion As AI systems are increasingly applied to crucial real-world tasks, there is a pressing issue to align them to our specific and diverse needs, while making the process more transparent and limiting the cultural hegemony of a few individuals. In this paper, we proposed rewarded soup, a strategy that efficiently yields Pareto-optimal solutions through weight interpolation after training. Our experiments have consistently validated our working hypotheses for various significant large-scale learning tasks, demonstrating that rewarded soup can mitigate reward misspecification. We hope to inspire further research in exploring how the generalization literature in deep learning can help for alignment, to create AIs handling the diversity of opinions, and benefit society as a whole. Acknowledgments This work was granted access to the HPC resources of IDRIS under the allocations AD011011953R1 and A0100612449 made by GENCI. Sorbonne Université acknowledges the financial support by the ANR agency in the chair VISA-DEEP (ANR-20-CHIA-0022-01). 11 References [1]Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint, 2021.(p. 1) [2]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL, 2019.(p. 1) [3]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. InNeurIPS, 2020. (p. 1) [4]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InICCV, 2021.(p. 1) [5]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021.(p. 1) [6]Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. InCVPR, 2014.(p. 1) [7]Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks?NeuriPS, 2014.(p. 1) [8] Vladimir N Vapnik. An overview of statistical learning theory. InTNN, 1999.(p. 1) [9]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety.arXiv preprint, 2016.(p. 1, 2, and 11) [10]Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems.Ethics of AI, 2016.(p. 1 and 11) [11] Richard Ngo, Lawrence Chan, and Soren Mindermann. The alignment problem from a deep learning perspective.arXiv preprint, 2022.(p. 1 and 11) [12]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, 2020.(p. 1, 2, 5, 6, 10, 31, and 32) [13]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.NeurIPS, 2022.(p. 1, 2, and 10) [14]André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards.arXiv preprint, 2023.(p. 1 and 6) [15] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InNeurIPS, 2017.(p. 1, 2, 6, and 10) [16]Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint, 2019.(p. 1) [17] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback.arXiv preprint, 2021.(p. 1 and 5) [18] OpenAI. Gpt-4 technical report.arXiv preprint, 2023.(p. 1) 12 [19]Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.(p. 1) [20]Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. InICLR, 2022.(p. 1 and 5) [21] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Morad- shahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. InACL, 2022.(p. 1) [22]Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model.https://github.com/tatsu-lab/stanford_alpaca, 2023.(p. 1, 2, 5, 31, and 32) [23]Yoav Goldberg. Reinforcement learning for language models.https://gist.github.com/ yoavg/6bff0fecd65950898eba1b321cfbd81, 2023.(p. 1) [24]Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. InCVPR, 2017.(p. 1, 6, 7, 10, and 34) [25]Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.arXiv preprint, 2023.(p. 1, 8, and 36) [26]Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference.arXiv preprint, 2023.(p. 1, 8, and 36) [27]Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. HIVE: Harnessing human feedback for instructional visual editing.arXiv preprint, 2023.(p. 1) [28]Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, bench- marks and analysis.Machine Learning, 2021.(p. 1) [29]Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InACL, 2002.(p. 1, 2, 7, and 34) [30] Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co- occurrence statistics. InNAACL, 2003.(p. 1, 2, 7, and 34) [31]Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Consensus-based image description evaluation. InICCV, 2015.(p. 1, 7, and 34) [32]Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. InICLR, 2023.(p. 2 and 10) [33]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint, 2023.(p. 2, 8, and 36) [34] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. InICLR, 2022.(p. 2) [35] Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint, 2020.(p. 2) 13 [36]Aaron Wildavsky. Choosing preferences by constructing institutions: A cultural theory of preference formation.American political science review, 1987.(p. 2) [37]CA Coello. Handling preferences in evolutionary multiobjective optimization: A survey. In CEC, 2000.(p. 2) [38]Shalom H Schwartz et al. An overview of the schwartz theory of basic values.Online readings in Psychology and Culture, 2012.(p. 2) [39]Marcos Nadal and Anjan Chatterjee. Neuroaesthetics and art’s diversity and universality.Wiley Interdisciplinary Reviews: Cognitive Science, 2019.(p. 2) [40]David Lopez-Paz, Diane Bouchacourt, Levent Sagun, and Nicolas Usunier. Measuring and signing fairness as performance under multiple stakeholder distributions.arXiv preprint, 2022. (p. 2) [41]Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint, 2022.(p. 2) [42] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint, 2022.(p. 2) [43]Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Vineet Mudupalli, Aliaksei Kor- shuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, et al. Rewarding chatbots for real-world engagement with millions of users.arXiv preprint, 2023.(p. 2) [44]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient founda- tion language models.arXiv preprint arXiv:2302.13971, 2023.(p. 2, 5, 31, and 32) [45]Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In ICML, 2008.(p. 2, 3, and 10) [46]Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization.IEEE-T-CYBERNETICS, 2020.(p. 2, 3, and 10) [47]Michiel A. Bakker, Martin J Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick, and Christopher Summerfield. Fine-tuning language models to find agreement among humans with diverse preferences. InNeurIPS, 2022.(p. 2) [48]Aviv Ovadya. Generative CI through collective response systems.arXiv preprint, 2023.(p. 2) [49]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint, 2022.(p. 2, 5, 10, and 32) [50]Grgur Kova ˇ c, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. Large language models as superpositions of cultural perspectives.arXiv preprint, 2023.(p. 2) [51]Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models.arXiv preprint, 2021.(p. 2) [52]Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback.arXiv preprint, 2023.(p. 2, 10, 11, and 23) 14 [53]Alexander Pan, Chan Jun Shern, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the MACHIAVELLI benchmark. InICML, 2023.(p. 2) [54]Conor F Hayes, Roxana R ̆ adulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfar- lane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. JAAMAS, 2022.(p. 2, 10, and 23) [55]Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect?arXiv preprint, 2023.(p. 2) [56] Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Human- aligned artificial intelligence is a multiobjective problem.Ethics and Information Technology, 2018.(p. 2, 11, and 23) [57]Fumihide Tanaka and Masayuki Yamamura. Multitask reinforcement learning on the distribu- tion of mdps. InCIRA, 2003.(p. 2 and 10) [58]Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies.JMLR, 2014.(p. 2 and 10) [59]Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making.JAIR, 2013.(p. 2 and 10) [60] Roxana R ̆ adulescu, Patrick Mannion, Diederik M Roijers, and Ann Nowé. Multi-objective multi-agent decision making: a utility-based analysis and survey.AAMAS, 2020.(p. 2, 10, and 11) [61]Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Iolanda Leite. Aligning human preferences with baseline objectives in reinforcement learning. InICRA, 2023.(p. 2 and 10) [62] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. InNeuriPS, 2023.(p. 2, 3, and 10) [63] Vilfredo Pareto.Cours d’économie politique. Librairie Droz, 1964.(p. 2) [64] Patrick Mannion, Fredrik Heintz, Thommen George Karimpanal, and Peter Vamplew. Multi- objective decision making for trustworthy ai. InMODeM Workshop, 2021.(p. 2) [65]Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. InICML, 2020.(p. 2, 4, and 11) [66]Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? InNeurIPS, 2020.(p. 2, 4, 24, and 25) [67]Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. InICML, 2022.(p. 2, 4, 10, 24, 26, 31, and 35) [68]Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. InNeurIPS, 2022.(p. 2, 10, 31, and 35) [69]Michael Matena and Colin Raffel. Merging models with Fisher-weighted averaging. In NeurIPS, 2022.(p. 2 and 29) [70] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. InNeurIPS, 2022.(p. 2 and 10) 15 [71]Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning.arXiv preprint, 2022.(p. 2 and 10) [72] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez- Paz. Model Ratatouille: Recycling diverse models for out-of-distribution generalization. In ICML, 2023.(p. 2, 4, 10, and 24) [73]Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. InICML, 2015.(p. 3) [74] Abien Fred Agarap. Deep learning using rectified linear units (relu).arXiv preprint, 2018. (p. 3) [75]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017.(p. 3 and 32) [76]Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment.arXiv preprint, 2021.(p. 3 and 10) [77]Peter Vamplew, John Yearwood, Richard Dazeley, and Adam Berry. On the limitations of scalarisation for multi-objective reinforcement learning of pareto fronts. InAJCAIA, 2008. (p. 3 and 11) [78] Lars Kai Hansen and Peter Salamon. Neural network ensembles.TPAMI, 1990.(p. 4 and 7) [79]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. InNeurIPS, 2017.(p. 4 and 7) [80]Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. InICLR, 2022.(p. 4) [81]Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging models modulo permutation symmetries. InICLR, 2023.(p. 4) [82]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. InEMNLP, 2020.(p. 5, 6, and 31) [83]Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions.arXiv preprint, 2022.(p. 5) [84]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint, 2017.(p. 5, 10, 32, and 40) [85]Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. TRL: Transformer reinforcement learning.https://github.com/lvwer ra/trl, 2020.(p. 5 and 32) [86]Edward Beeching, Younes Belkada, Leandro von Werra, Sourab Mangrulkar, Lewis Tunstall, and Kashif Rasul. Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU.https: //huggingface.co/blog/trl-peft, 2023.(p. 5, 31, and 32) [87] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InICLR, 2022.(p. 5, 31, and 32) [88] Hadeer Ahmed.Detecting opinion spam and fake news using n-gram analysis and semantic similarity. PhD thesis, 2017.(p. 5 and 32) 16 [89]Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. InACL Workshop, 2017.(p. 5, 31, and 32) [90] Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023.(p. 5 and 32) [91]Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection. InNAACL, 2021.(p. 6, 31, and 32) [92]Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Ope- nassistant conversations–democratizing large language model alignment.arXiv preprint, 2023. (p. 6 and 31) [93] Gary G Yen and Zhenan He. Performance metric ensemble for multiobjective evolutionary algorithms.TEVC, 2013.(p. 6 and 7) [94]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InECCV, 2014.(p. 6, 7, and 34) [95]Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. InACL Workshop, 2005.(p. 7 and 34) [96]Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. ExpansionNet v2: Block static expansion in fast end to end training for image captioning.arXiv preprint, 2022.(p. 7 and 34) [97]Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer V2: Scaling up capacity and resolution. InCVPR, 2022.(p. 7 and 34) [98]Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning.Reinforcement learning, 1992.(p. 7, 10, 34, 38, and 39) [99] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. InCVPR, 2015.(p. 7 and 34) [100]Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. InUAI, 2018. (p. 7, 11, 26, and 31) [101]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.NeurIPS, 2020.(p. 8) [102] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, 2022.(p. 8 and 36) [103]Naila Murray, Luca Marchesotti, and Florent Perronnin. AVA: A large-scale database for aesthetic visual analysis. InCVPR, 2012.(p. 8 and 36) [104] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter- efficient fine-tuning.arXiv preprint arXiv:2304.06648, 2023.(p. 8 and 36) [105]Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. InECCV, 2016.(p. 8, 38, and 39) [106]Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unified model for image, video, audio and language tasks.arXiv preprint arXiv:2307.16184, 2023.(p. 8, 10, 38, and 39) [107]Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.CoRR, 2022.(p. 8, 9, 39, and 40) 17 [108]Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. InICML, 2016.(p. 9) [109] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. InICML, 1999.(p. 9) [110]Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning.Artificial intelligence, 1994.(p. 9) [111]Dan Dewey. Reinforcement learning and the reward engineering principle. InAAAI, 2014. (p. 9) [112]Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. InIROS, 2012.(p. 9 and 10) [113]C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax–a differentiable physics engine for large scale rigid body simulation.arXiv preprint, 2021.(p. 9 and 40) [114] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint, 2023.(p. 10) [115] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback without tears.arXiv preprint, 2023.(p. 10) [116] Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements.arXiv preprint, 2022.(p. 10) [117]Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters.TOG, 2020.(p. 10) [118]Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. Multi-expert learning of adaptive legged locomotion.Science Robotics, 2020.(p. 10) [119]Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. InICML, 2020.(p. 10) [120]Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi-objective optimization. InNeuriPS, 2022.(p. 10) [121]Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi- objective deep reinforcement learning.arXiv preprint, 2016.(p. 10) [122]Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. InICML, 2007.(p. 10) [123] Thanh Thi Nguyen, Ngoc Duy Nguyen, Peter Vamplew, Saeid Nahavandi, Richard Dazeley, and Chee Peng Lim. A multi-objective deep reinforcement learning framework.EAAI, 2020. (p. 10) [124]Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run.Water Resources Research, 2013.(p. 10) [125]Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. InNeurIPS, 2019.(p. 10) [126] Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. InICML, 2019.(p. 10) 18 [127]Markus Peschl, Arkady Zgonnikov, Frans A Oliehoek, and Luciano C Siebert. Moral: Aligning ai with human norms through multi-objective reinforced active learning.arXiv preprint, 2021. (p. 10) [128] Pu Hua, Yubei Chen, and Huazhe Xu. Simple emergent action representations from multi-task policy training. InICLR, 2023.(p. 10) [129]Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms.Deakin University, 2011.(p. 10) [130] Rich Caruana. Multitask learning.Machine learning, 1997.(p. 10 and 24) [131]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. InNeurIPS, 2020.(p. 10 and 24) [132] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning.NeurIPS, 2021.(p. 10 and 24) [133]Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. InICML, 2018.(p. 10 and 24) [134] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Had- sell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. NeurIPS, 2017.(p. 10 and 24) [135]M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade learning environment: An evaluation platform for general agents.JAIR, 2013.(p. 10 and 24) [136] Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold learning: Tack- ling multiple tasks via ensembles of single-task models.arXiv preprint, 2022.(p. 10) [137] Francesco Croce, Sylvestre-Alvise Rebuffi, Evan Shelhamer, and Sven Gowal. Seasoning model soups for robustness to adversarial and natural distribution shifts. InCVPR, 2023.(p. 10) [138]Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. InICLR, 2023.(p. 10) [139] Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for forming multi-task policies. InICLR RRL Workshop, 2023.(p. 10) [140]Jean-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. InICLR, 2022.(p. 10) [141] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. InICLR, 2023.(p. 10, 24, and 35) [142]Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo M Ponti. Elastic weight removal for faithful and abstractive dialogue generation.arXiv preprint, 2023.(p. 10) [143]Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning.arXiv preprint, 2023.(p. 10) [144]Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. InNeurIPS, 2021.(p. 10 and 26) [145] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Un- derspecification presents challenges for credibility in modern machine learning.JMLR, 2020. (p. 10) 19 [146]Dan Hendrycks and Mantas Mazeika. X-risk analysis for AI research.arXiv preprint, 2022. (p. 11) [147] Dan Hendrycks. Natural selection favors AIs over humans.arXiv preprint, 2023.(p. 11) [148] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint, 2022.(p. 11) [149]Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. InNeurIPS, 2022.(p. 11) [150]Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals.arXiv preprint, 2022.(p. 11) [151] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. InICML, 2022.(p. 11) [152]Ben Smith. A brief review of the reasons multi-objective RL could be important in AI Safety Research.https://w.alignmentforum.org/posts/i5dLfi6m6FCexReK9/a-brief -review-of-the-reasons-multi-objective-rl-could-be, 2021.(p. 11) [153]Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models.arXiv preprint, 2023.(p. 11) [154]Manel Rodriguez-Soto, Maite Lopez-Sanchez, and Juan A Rodríguez-Aguilar. Guaranteeing the learning of ethical behaviour through multi-objective reinforcement learning. InAAMAS, 2021.(p. 11) [155] Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization.arXiv preprint, 2023.(p. 11 and 23) [156]Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InAISTATS, 2017.(p. 11 and 24) [157] Philip E Tetlock. A value pluralism model of ideological reasoning.JPSP, 1986.(p. 23) [158]Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. InICML, 2020.(p. 23) [159]Iason Gabriel and Vafa Ghazavi. The challenge of value alignment: From fairer algorithms to AI safety.arXiv preprint, 2021.(p. 23) [160] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InACM SIGSAC, 2016.(p. 23) [161]Kristof Van Moffaert, Tim Brys, Arjun Chandra, Lukas Esterle, Peter R Lewis, and Ann Nowé. A novel adaptive weight selection algorithm for multi-objective multi-agent reinforcement learning. InIJCNN, 2014.(p. 24) [162]Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum-based weight interpolation of strong zero-shot models for continual learning. InNeurIPS Interpolate Workshop, 2022.(p. 24) [163]Steven Vander Eeckt et al. Weight averaging: A simple yet effective method to overcome catastrophic forgetting in automatic speech recognition.arXiv preprint, 2022.(p. 24) [164]Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-Train-Merge: Embarrassingly parallel training of expert language models.arXiv preprint, 2022.(p. 24) [165]Colin Raffel. A Call to Build Models Like We Build Open-Source Software.https: //colinraffel.com/blog/a-call-to-build-models-like-we-build-open-sou rce-software.html, 2021.(p. 24) 20 [166]Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. InNeural Networks. 2012.(p. 28) [167]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InICLR, 2015.(p. 28, 32, and 36) [168]Yann LeCun, J. S. Denker, Sara A. Solla, R. E. Howard, and L.D. Jackel. Optimal brain damage. InNeurIPS, 1990.(p. 28) [169] Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. InICML, 2022.(p. 28) [170]Sue Becker and Yann Le Cun. Improving the convergence of back-propagation learning with second order methods. InConnectionist models summer school, 1988.(p. 28) [171]Ronald A Fisher. On the mathematical foundations of theoretical statistics.Philosophical Transactions of the Royal Society of London., 1922.(p. 29) [172] Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. InNeural computation, 2002.(p. 29) [173] Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. InAISTATS, 2020.(p. 29) [174]Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. InNeurIPS, 2019.(p. 29) [175] Eric J. Wang. Alpaca-LoRA.https://github.com/tloen/alpaca-lora, 2023.(p. 32) [176] Hadeer Ahmed, Issa Traore, and Sherif Saad. Detecting opinion spams and fake news using text classification.Security and Privacy, 2018.(p. 32) [177]Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Rajani, and Nathan Lambert. StackLLaMA: An RL Fine-tuned LLaMA Model for Stack Exchange Question and Answering, 2023.(p. 32) [178]Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christo- pher Potts. Learning word vectors for sentiment analysis. InACL, 2011.(p. 32) [179] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring.arXiv preprint, 2019.(p. 33) [180]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 2019.(p. 33) [181] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. InCVPR, 2009.(p. 34) [182] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. InICLR, 2020.(p. 34) [183] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Hanna Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, 2022.(p. 35) [184]Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint, 2021.(p. 36) [185]Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment.arXiv preprint, 2023.(p. 36) 21 [186]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.NeurIPS, 2017.(p. 37 and 38) [187]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. InEMNLP, 2021.(p. 37 and 38) 22 Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards Supplementary material This supplementary material is organized as follows: • Appendix A further discusses the practical benefits of rewarded soups. • Appendix B anticipates questions that might arise from readers. • Appendix C details some theoretical guarantees. • Appendix D details our text-to-text generation experiments. • Appendix E enriches our image captioning experiments. • Appendix F enriches our image generation experiments. • Appendix G enriches our visual grounding experiments. • Appendix H enriches our visual question answering experiments. • Appendix I enriches our locomotion experiments. The shareable code is released on github. Moreover, you can find additional qualitative results of our experiments on this website. A Discussion In this section we discuss the benefits of our rewarded soup (RS) approach with respect to the two families of strategies: thesingle-policyand themulti-policyapproaches. A.1 Compared to single-policy approaches The main reason why single-policy approaches are not suitable is because they optimize over a single set of preferences. In contrast, we build a coverage set of Pareto-optimal policies. This is important for the following reasons, mostly first discussed in Kirket al.[52] and in Hayeset al.[54]. Indeed, the user’s true reward is highly uncertain before training. This “semi-blind” [54] manual process forces a priori and uncertain decisions about the required trade-offs. Itshifts the respon- sibilityfrom the problem stakeholders to the system engineers, who need to anticipate the impact of their choices on the final performance. Critically, the RLHF process may cause the “tyranny of the crowdworker” [52], as models are “tailored to meet the expectations of [...] a small number of crowdworkers primarily based in the US, with little to no representation of broader human cultures, geographies or languages.” [52]. Moreover, biased are caused by chaotic engineering choices, and “are exacerbated by a lack of [...] documentation” [52]. In contrast, our approach makespersonal- ization explicit, as argued by [52]. Moreover, we couldsupport decision-makingto find a good balance between (potentially conflicting) parties’ interests. This value pluralism [157] can lead to fairerand more equitable outcomes [56,158]. Single-policy cannot adapt to test time requirements; in contrast, RS facilitates personalized assistances [155]. This is all the more important as human preferences change from time to time. In thisdynamic utility functionscenario, RS can quickly adapt with fewer data, by simply adjusting theλto match new preferences (rather than the full network). Finally, RS could also improve theinterpretabilityandexplainabilityof the decisions. Letting the users decide would make the process moretransparent[159], which is essential to ensure that the development process is fair, unbiased, and inclusive [160]. A.2 Compared to multi-policy approaches The main reason why existing multi-policy approaches through multitasking are not suitable is because of theircomputational costsrequired to learn a dense set of policies. In contrast, RS only 23 trains the proxy rewards independently and enables the selection of the interpolating coefficient a posteriori. This is especially useful with large number of rewards and thus growing number of combinations. Second, multitask [130] is challenging; for example, even if the true reward is actually a linear weighted sum of some proxy rewards and those coefficients are known, using those preferences during training can lead to suboptimal results [161], because of conflicting gradients [131,132] or different variance scales [133,134]. This has been tackled in RL, but so far mostly for games such as ATARI [135]. Third, our strategy is compatible with the inherentiterative engineering processof alignment. Indeed, RS can continually include adjusted opinions while preventing forgetting of the old behaviours. This relates to thecontinual learningchallenge, and the empirical observations that weight averaging can reduce catastrophic forgetting [162,163]. Moreover, as shown in [141] and confirmed in Figure 14(c), negative editing by weight interpolation can fix and force the removal of some behaviours. Finally, RS is computationally effective, requiringno communication across servers, thus enabling “embarrassingly simple parallelization” [164]. This facilitates its use infederated learningscenario [156] where the data should remain private. Actually, RS follows theupdatable machine learning paradigm[165], “allowing for the collaborative creation of increasingly sophisticated AI system” [72]. In the future, we may develop open-source personalized models, rewarded on decentralized private datasets, and combine them continuously. B FAQs We addressed below questions that might arise from readers. B.1 What is the difference between rewarded soups and model soups? Rewarded soups (RS) and model soups (MS) [67] both average weights of models fine-tuned from a shared pre-trained initialization. That’s why we chose the same terminology as “model soups” and named our method “rewarded soups”. Yet, we want to clarify that RS and MS tackle different problems, have different goals, leading to different methods and implementations. •RS challenges single-policy approaches to improve alignment in reinforcement learning, and aims at reducing reward misspecification by revealing a Pareto front of solutions across the entire space of preferences: thus RS considers different training objectives for fixed hyperparameters across runs, and non-uniform interpolating coefficientsλset a posteriori. •MS challenges the standard model selection after a grid search to improve generalization in supervised learning, and aims at reducing model underspecification and reducing variance by combining all fine-tuned models: thus MS considers different hyperparameters for a fixed training objective across runs, and (usually) uniform interpolating coefficientsλ= 1 M . These differences mean that MS cannot be applied to reduce reward misspecification, as validated empirically in Figure 14(b) for the captioning task. This Figure 14(b) also shows that RS and MS are actually complementary and can combine their benefits; specifically, reward misspecification and variance reduction. B.2 Limitations for the LMC? B.2.1 Limitations for the design of networks for the LMC? In our experiments, we consider different network architectures (transformers, CNNs, and MLPs). We also investigate different training procedures: with low-rank adapters, partial or end-to-end fine-tunings. We do so for many different tasks and modalities: text generation, image captioning, image-to-test generation, visual grounding, etc. Our empirical observation is that, across those setups, the LMC is architecture-agnostic, procedure-agnostic, task-agnostic and modality-agnostic. The main condition we require is the shared pre-trained initialization [66], so that the weights remain close (as detailed in Remark 1). As a side note, there is another condition suggested by the literature [164,141]: the LMC would work better when the architecture has enough trainable parameters. For example, according to [141], larger networks may facilitate the orthogonality of the fine-tuned updates; then [141] "speculate that this [orthogonality] enables the combination of task vectors via addition with minimal interference". 24 B.2.2 Limitations for the number of training steps for the LMC? As argued above, good performances are guaranteed when weights remain close; thus longer trainings may be worrisome, as the models may potentially diverge in the weight space. We investigate this question in Figure 9, for the news summarization and the captioning task; we double the number of training steps, and report multiple RS fronts over the course of fine-tuning. Fortunately, we consistently observe good performances for RS along fine-tuning. This confirms that the only condition for the LMC is the shared pre-trained initialization [66]. 0.81.01.21.41.61.8 R 1 −1.8 −1.6 −1.4 −1.2 −1.0 −0.8 −0.6 R 2 LLaMA init R 2 rewarded R 1 rewarded λ= 0.5 Epoch: 0.22 0.44 0.67 0.89 1.11 1.33 1.56 1.78 2.00 2.22 2.44 2.67 2.89 3.11 3.33 3.56 3.78 4.00 (a) News summary. 0.8250.8300.8350.8400.8450.8500.855 R 1 :BLEU1 0.602 0.604 0.606 0.608 0.610 R 2 : ROUGE pre-trained init ROUGErewarded BLEU1 rewarded λ= 0.5 Epoch: 1 2 3 4 5 6 7 8 9 10 11 12 (b) Captioning. Figure 9: Those figures show how RS’s fronts evolve over the course of fine-tuning, and confirms the LMC even when doubling the number of training epochs (previously2for news summarization and6for image captioning). B.2.3 How does the number of rewards (and networks) affects the LMC? For visualization clarity, the fronts were mostly shown forN= 2rewards, one of thex-axis, the other on they-axis. Yet, RS can scale and trade-off between more rewards. We validated this empirically in the spider maps from Figure 2(f) (for text generation), from Figures 3(c) and 4 (for image captioning), and from Figure 19(c) (for visual grounding), where we respectively consider up toN= 4,N= 5 andN= 3networks fine-tuned onNdifferent rewards, one reward each. B.3 Comparison of MORL and RS B.3.1 How to evaluate Pareto-optimality? Given a fixed preferenceˆμbetween two rewardsR 1 andR 2 , we would like to compare our RS policy to an oracle policy maximizing(1−ˆμ)×R 1 +μ×R 2 in test. Yet, this oracle policy (and the true Pareto front) is unknown in real-world applications. That’s why, in practice, and as argued in Remark 2, we presented empirical support for Hypothesis 2 by considering the MORL’s solutions fine-tuned to optimize(1−ˆμ)×R 1 +μ×R 2 in train, for 0≤μ≤1. In other words, the linearized MORL is our reference to evaluate Pareto optimality. Overall, in Section 3, MORL and RS usually perform similarly (with small differences further discussed below in Appendices B.3.2 and B.3.3). Our conclusion is that rewarded soup is an empirical solutiontowardsPareto-optimality, with indeed an experimental limitation highlighted in the paper’s name. B.3.2 How does reward diversity affect the effectiveness of RS? Our experiments in captioning and image generation provide empirical evidence that the more similar the rewards, the higher the gains of RS versus MORL. In the captioning experiment, by analyzing the transfer abilities across rewards in the spider maps from Figure 3(c), we can deduce that BLEU4 and ROUGE are more similar than BLEU1 and ROUGE, while METEOR is an outlier (fine-tuning on METEOR worsens the results for the other rewards). Then, we can observe that the gains of RS versus MORL are consistent with these similarities across rewards. Specifically, when consideringR 2 =ROUGE, the RS green front is more convex and significantly above the MORL yellow front in Figure 12(a) (withR 1 =BLEU4) than in Figure 3(a) (withR 1 =BLEU1). In Figure 13(b), withR 2 =METEOR, MORL performs better than RS. 25 Similarly, in the image generation experiment, when we consider two (arguably similar) aesthetic rewards in Figure 6(a) to fine-tune a diffusion model, RS’s front is to the right and above MORL’s front. In contrast, performances get worse in Figure 15 where we also include annsfwreward inversely correlated with image quality. In conclusion, despite using diverse and heterogeneous rewards that are in tension, we consistently obtain positive results. Yet, in the case where rewards are fully antagonist, we acknowledge that RS is likely to produce less favorable results. This empirical limitation of weight interpolation can be explained in two different ways. (i) Intuitively from a loss landscape perspective: weights fine-tuned on antagonist rewards will be more distant, thus potentially breaking the linear mode connectivity. (i) Theoretically thanks to Lemma 3, where we bound the difference between the optimal reward and RS’s reward by a RHS term growing the maximum of eigenvalues ratio for rewards’ Hessians: if the rewards are more diverse, their Hessians would have more different eigenvalues, thus maximum of eigenvalues ratio would grow, the RHS term would grow in Lemma 3, and our guarantees for the optimality of RS would get loose. As a final note, to tackle this limitation under antagonist rewards, the complementarity of MORL and RS appears as a promising research direction; this is further discussed in the legend of Figure 14(a) for the captioning task and in Appendix F.2 for the image generation task. B.3.3 Why RS is sometimes superior to MORL? We observe a few times that the RS solutions are actually above the linearized MORL solutions. We speculate this is related to the multiple benefits of weight interpolation. The main benefit that we discuss in our paper is the ability to interpolate between different policies: from this benefit, we would expect RS to perform similarly to MORL. The second benefit from weight averaging is the implicit regularization, causing variance reduction and stabilizing performances [100,144]. This is the main focus of the traditional weight averaging literature, for example in model soups [67]. In conclusion, we speculate that this second benefit (combined with the first) can explain why RS sometimes outperforms MORL. 26 C Theoretical insights C.1 Proof of Lemma 1 Proof. Consideringθmaximizing ˆ R, we first show thatθis on the PF ofR i i . Otherwise, consid- eringθ ′ > N θand as∀i,ˆμ i ≥0, we have P i ˆμ i R i (θ ′ )> P i ˆμ i R i (θ) . This implies thatθ ′ would produce a better policy thanθfor ˆ R= P i ˆμ i R i and thus the contradiction. Finally, asθis on the PF and by definition of a PCS, there existsλs.t.∀k,R k ( P i λ i ·θ i ) =R k (θ). C.2 Theoretical guarantees with quadratic rewards In this section, we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards. This simplification amounts to replacing the rewards by their second-order Taylor approximation, which is a realistic assumption when the weights remain within a small neighborhood. C.2.1 Simple case with Hessians proportional to the Identity matrix For the first Lemma 2, we make the following simplifying Assumption 1. Assumption 1(Hessians proportional to the Identity matrix.).Every rewardR i is quadratic, with Hessians proportional toI d . Specifically, letΘ⊂R d be the set of possible weights, and letR i N i=1 be theNrewards, we can write fori∈1,...,N: ∀θ∈Θ, R i (θ) =R i (θ i )−η i ∥θ−θ i ∥ 2 (1) whereη i ∈R ∗ + andθ i is the global maximum for rewardR i . Lemma 2.Letˆμ= (ˆμ 1 ,...,ˆμ N )∈∆ N . Then, under Assumption 1, the rewardR ˆμ = P i ˆμ i ×R i is maximized on the convex hull ofθ 1 ,...,θ N . Proof. The functionR ˆμ is quadratic thus has an unique global maximum ˆ θ , that we find analytically: ∇ θ R ˆμ ( ˆ θ) = 0 =⇒ N X i=1 μ i η i ·( ˆ θ−θ i ) = 0 =⇒ ˆ θ= P N i=1 ˆμ i η i ·θ i P N i=1 ˆμ i η i Since all theˆμ i η i are positive or zero, and at least one is greater than zero, ˆ θ is indeed in the convex hull ofθ 1 ,...,θ N . Remark 3.Under Assumption 1, the reward functions are concave; thus we can reasonably assume that each fine-tuning procedure forR i reaches its global optimumθ i fori∈ 1,...,N. Then, Lemma 2 tells us that the maximum value for linear user’s rewardR ˆμ is obtainable by weight interpolation between theθ i N i=1 : the interpolating coefficients in∆ N such thatλ i ∝ˆμ i η i make rewarded soups optimal. C.2.2 Advanced case with diagonal Hessians We now consider the more complex case with the relaxed Assumption 2. For simplicity, we only considerN= 2rewardsR 1 andR 2 . Assumption 2(Diagonal Hessians).The rewards are quadratic, with Hessians diagonal negative definite. Specifically, we can write fori∈1,2: ∀θ= (θ 1 ,...,θ d )∈Θ, R i (θ) =R i (θ i )− d X j=1 η j i (θ j −θ j i ) 2 ,(2) where(η 1 i ,...η d i )∈R ∗ + d andθ i = (θ 1 i ,...,θ d i )is the global maximum for rewardR i . 27 Remark 4.This diagonal Assumption 2 of the Hessian is common: for example in optimization [166,167], to prune networks [168] or in out-of-distribution generalization [169]. This strong assumption is supported by the empirical observation [170] that Hessians are diagonally dominant, in particular at the end of training. Also, we note that our findings remain valid assuming only that the Hessians are co-diagonalizable. Lemma 3.We consider the user’s rewardR ˆμ = (1−ˆμ)×R 1 + ˆμ×R 2 withˆμ∈[0,1], and ∆R ˆμ = max θ∈Θ R ˆμ (θ)−max λ∈[0,1] R ˆμ ((1−λ)·θ 1 +λ·θ 2 ).(3) ∆R ˆμ corresponds to the difference in terms ofR ˆμ between the global maximum and the maximum reachable by weight interpolation through rewarded soups (with a single interpolating coefficient for all dimensions). Then, under Assumption 2, we have: ∆R ˆμ ≤ ˆμ 2 (1−ˆμ) 2 (M∆ 1 −∆ 2 )(M∆ 2 −∆ 1 ) (ˆμ(1−ˆμ)(M−1) 2 +M)((1−ˆμ)∆ 1 + ˆμ∆ 2 ) ,(4) whereM= max j∈1,...,d max η j 1 η j 2 , η j 2 η j 1 is the maximum of eigenvalues ratio,∆ 1 =R 1 (θ 1 )− R 1 (θ 2 )and∆ 2 =R 2 (θ 2 )−R 2 (θ 1 ). When∆ 1 = ∆ 2 , the bound simplifies into: ∆R ˆμ ≤ ˆμ 2 (1−ˆμ) 2 (M−1) 2 ˆμ(1−ˆμ)(M−1) 2 +M ∆ 1 (5) Furthermore, when the Hessians are equal, thenM= 1and∆R ˆμ = 0: RS is optimal . Proof.This novel proof is in three steps. First, we find ˆ θmaximizingR ˆμ (θ)forθon the full set of weightsΘ. Second, we find ̄ λmaximizingR ˆμ ((1−λ)·θ 1 +λ·θ 2 )forλ∈[0,1]and thus defining the best interpolation between the expert weights. Finally, we bound∆R ˆμ , the differences between their rewards, by applying the Bhatia-Davis inequality. First step.Let’s first find the maximum ofR ˆμ onΘ. DenotingS= (1−ˆμ)×R 1 (θ 1 ) + ˆμ×R 2 (θ 2 ), we have for allθ∈Θ: R ˆμ (θ) =S− d X j=1 (1−ˆμ)η j 1 θ j −θ j 1 2 + ˆμη j 2 θ j −θ j 2 2 (6) SinceR ˆμ is a sum of concave quadratic functions, it has a unique global maximum reached at a point we note ˆ θ= ˆ θ 1 ,..., ˆ θ d . The global maximum can be computed by differentiatingR ˆμ with respect to each variableθ j , which gives: ˆ θ j = 1− ˆ λ j ·θ j 1 + ˆ λ j ·θ j 2 where the interpolating coefficients per dimension ˆ λ j are defined forj∈1,...,das: ˆ λ j = ˆμη j 2 (1−ˆμ)η j 1 + ˆμη j 2 ∈[0,1].(7) Second step.Withλ∈[0,1]andθ= (1−λ)·θ 1 +λ·θ 2 , we can writeR ˆμ (θ)as a function ofλ: R ˆμ (θ) =S− d X j=1 (1−ˆμ)η j 1 + ˆμη j 2 λ− ˆ λ j 2 + ˆμ(1−ˆμ)η j 1 η j 2 (1−ˆμ)η j 1 + ˆμη j 2 ! θ j 1 −θ j 2 2 =R ˆμ ( ˆ θ)− d X j=1 p j λ− ˆ λ j 2 (8) wherep j is defined asp j = (1−ˆμ)η j 1 + ˆμη j 2 θ j 1 −θ j 2 2 . 28 From Equation (8), we can compute the maximum reward obtainable for weight averaging max λ∈[0,1] R ˆμ ((1−λ)·θ 1 +λ·θ 2 ). Since the functionλ7→R ˆμ ((1−λ)·θ 1 +λ·θ 2 )is a con- cave quadratic function, there is a unique value ̄ λmaximizingR ˆμ equal to ̄ λ= P d j=1 p j ˆ λ j P d j=1 p j .(9) Since allp j are positive and all ˆ λ j are between0and1, ̄ λis also between0and1. Therefore, R ˆμ (1− ̄ λ)·θ 1 + ̄ λ·θ 2 is indeed the maximum reward for rewarded soups. Third step.Applying Equation (8) to ̄ λgives: ∆R ˆμ =R ˆμ ( ˆ θ)−R ˆμ (1− ̄ λ)·θ 1 + ̄ λ·θ 2 (10) = d X j=1 p j ̄ λ− ˆ λ j 2 (11) = d X j=1 p j P n i=1 p i ̄ λ− ˆ λ j 2 n X j=1 p j (12) The second term in Equation (12) can be simplified as: d X j=1 p j = (1−ˆμ)∆ 1 + ˆμ∆ 2 .(13) The core component of this proof is the upper bounding of the first term in Equation (12). The key idea is to recognize the variance of a discrete random variableΛwithP(Λ = ˆ λ i ) = p i P n j=1 p j ; then, ̄ λ from Equation (9) is actually the expectation ofΛ. Then, we can apply theBhatia-Davis inequality, as recalled in Equation (14), on the variance of a bounded random variablea≤Λ≤b: V ar(Λ)≤(b−E(Λ))(E(Λ)−a)(14) Therefore Equation (12) is bounded by: ∆R ˆμ ≤ max 1≤j≤d ˆ λ j − ̄ λ ̄ λ−min 1≤j≤d ˆ λ j ((1−ˆμ)∆ 1 + ˆμ∆ 2 ).(15) Now, we bound the variables ˆ λ j , since1/M≤η j 1 /η j 2 ≤M. Then for alljwe have: ˆμ (1−ˆμ)M+ ˆμ ≤ ˆ λ j ≤ ˆμM (1−ˆμ) + ˆμM ,(16) and thus: ∆R ˆμ ≤ ˆμM 1 + ˆμ(M−1) − ̄ λ ̄ λ− ˆμ M−ˆμ(M−1) ((1−ˆμ)∆ 1 + ˆμ∆ 2 ).(17) Finally, noting that∆ i = P d j=1 η j i θ j 2 −θ j 1 2 , we deduce from Equation (9) that ̄ λ= ˆμ∆ 2 (1−ˆμ)∆ 1 +ˆμ∆ 2 . Replacing this in the previous Equation (17) gives the final Equation (4), concluding the proof. Remark 5.As a final remark, please note that the suboptimality of RS comes from the need of having one single interpolating coefficient ̄ λfor alldparameters(θ 1 ,...,θ d )of the network. Yet, the advanced merging operations in [69] remove this constraint, with interpolating coefficients propor- tional to the eigenvalues of the Fisher matrices [171], which actually approximate the eigenvalues of the Hessian [172,173]. Combining [69] and our RS is a promising research direction, the key issue being the computation of the Fisher matrices [174] for networks with billions of parameters. 29 C.2.3 Bound visualization We visualize in Figure 10 the bound given by Lemma 3. We show that for small values ofMlike M= 2, the value ofR ˆμ for RS is quite close to the global optimum. Also, recall that RS theoretically matches this upper bound whenM= 1. For larger values likeM= 10, the bound is less tight, and we note that the maximum value ofR ˆμ approaches the constant function 1 asM→∞. 0.00.20.40.60.81.0 ˆμ 0.4 0.5 0.6 0.7 0.8 0.9 1.0 R ˆ μ = (1 − ˆ μ ) × R 1 + ˆ μ × R 2 Upper bound, M=10 Upper bound, M=2 Rewarded soups LMC lower bound Figure 10: Illustration of the bound given by Lemma 3 under Assumption 2. For simplicity, we showcase the case whereR 1 (θ 1 ) =R 2 (θ 2 ) = 1,R 1 (θ 2 ) =R 2 (θ 1 ) = 0, thus∆ 1 = ∆ 2 = 1. In green, we plot the rewards obtained with rewarded soups for the optimal ̄ λ , i.e.,R ˆμ (1− ̄ λ)·θ 1 + ̄ λ·θ 2 , whose value is independent of Min this case. In blues, we plot the maximum value ofR ˆμ given by Equation (5) in Lemma 3, forM= 2and M= 10. For reference, we also plot the values for the lower bound in the LMC Hypothesis 1, i.e., equal to (1−ˆμ)(1− ̄ λ)R 1 (θ 1 ) + ˆμ ̄ λR 2 (θ 2 ). As RS outperforms this lower bound, it validates Hypothesis 1 in this case. 30 C.3 Similarity between weight interpolation and functional ensembling Lemma 4(λ-interpolation of weights approximates theλ-ensembling of predictions. Adapted from [67,68,100].).Givenθ 1 andθ 2 optimized forR 1 andR 2 s.t. they remain close, i.e.,∥θ 1 −θ 2 ∥ 2 ≈0. Denotingθ λ the interpolated weightsθ λ = (1−λ)·θ 1 +λ·θ 2 andf λ the ensembling of predictions f λ (·) = (1−λ)·f(·,θ 1 ) +λ·f(·,θ 2 ): f(·,θ λ )≈f λ (·) and fork∈1,2: R k (f(·,θ λ ))≈R k (f λ (·)) Proof.This proof follows [68] and has two components. Functional approximation.First, we perform a Taylor expansion at the first order of the models’ predictions w.r.t. parametersθforx∈T: f(x,θ 1 ) =f(x,θ λ ) +∇ θ f(x,θ λ ) ⊺ (θ 1 −θ λ ) +O ∥θ 1 −θ λ ∥ 2 2 =f(x,θ λ ) +∇ θ f(x,θ λ ) ⊺ (λ·θ 1 −λ·θ 2 ) +O ∥θ 1 −θ 2 ∥ 2 2 and similarly: f(x,θ 2 ) =f(x,θ λ ) +∇ θ f(x,θ λ ) ⊺ ((λ−1)·θ 1 + (1−λ)·θ 2 ) +O ∥θ 1 −θ 2 ∥ 2 2 Then byλ-weighted sum overi, the term multiplying∇ θ f(x,θ λ ) ⊺ cancels out and we obtain: f λ (x) = (1−λ)·f(x,θ 1 ) +λ·f(x,θ 2 ) =f(x,θ λ ) +O ∥θ 1 −θ 2 ∥ 2 2 .(18) Reward approximation.Second, we obtain the reward approximation with a Taylor expansion at the zeroth order of the rewardR k fork∈1,2and injecting Equation (18): R k (f λ (x)) =R k (f(x,θ λ )(x)) +O(∥f λ (x)−f(x,θ λ )∥ 2 ) =R k (f(x,θ λ )(x)) +O ∥θ 1 −θ 2 ∥ 2 2 . We obtain the results whenθ 1 andθ 2 remain close, i.e., when we can ignore theOterm. D Text-to-text: LLaMA with diverse RLHFs D.1 Experimental details We summarize the key implementation details of our text-to-text generation experiments in Table 1. The pre-trained network is LLaMA-7b [44]; then low-rank adapters [87] were fine-tuned on Alpaca [22] to follow instructions. We eventually fine-tune via PPO on the different considered tasks. Our code is adapted from [86]; we kept most of their hyperparameter values, only dividing by 2 the batch size to fit in our GPU and extending the output length. For each task, we consider existing open-source datasets 3 and available reward models, that we download from HuggingFace. Regarding the reward models, in summarization tasks,R 1 was open-sourced in an effort to reproduce the Summarize from Human Feedback paper [12], whileR 2 [91] aimed at improved “faithfulness in abstractive summarization with contrast candidate generation”. For other dialog tasks, we mostly rely on different reward models from OpenAssistant [92]; though they all aim at evaluating whether an answer is adequate given a question, they differ in their predictions due to differences in their architecture and training procedures. In practice, we leverage these reward models as block-box classification pipelines, implemented in the transformers library [82]. 3 For example, the TL;DR dataset is a previously existing dataset extracted and obtained by [89] that contains preprocessed comments posted on the social network Reddit and hosted on HuggingFace. 31 Table 1: LLaMA with RLHF experiments: key implementation details. Model ArchitectureTransformer [75] Pre-trainingLLaMA-7b [44] Instruction FTAlpaca [22] RL procedure Fine-tuning strategyLoRA [87] following Alpaca-LoRA [175] LoRA alpha16 LoRA dropout0.05 following trl-peft [85, 86] OptimizerAdam [167] Learning rate1.41e-5 Batch size128 Output lengthUniformly sampled between 16 and 32 RL algorithmPPO [84] KL PPO0.05 for summary tasks else 0.2 Epochs2 for Reuter summary else 1 HardwareNVIDIA RTX A6000 49 Go Compute budget4000 GPUh Task nameReuter summary DescriptionGenerate a concise and clear summary of newspaper articles from Reuters. Prompt“Generate a one-sentence summary of this post.” DatasetReuter news from [88, 176] from news-summary R 1 gpt2-reward-summarization trained here. R 2 bart-faithful-summary-detector [91] FigureFigures 1(b) and 2(a) Task nameReddit TL;DR summary DescriptionGenerate a concise and clear summary of posts from Reddit across a variety of topics (subreddits). Prompt“Generate a one-sentence summary of this post.” DatasetReddit crawl from the TL;DR dataset [89] from summarize-from-feedback [12] R 1 gpt2-reward-summarization trained here. R 2 bart-faithful-summary-detector [91] FigureFigure 2(b) Task nameStack Exchange DescriptionAnswer accurately to technical questions from Stack Exchange. PromptNo prompt, only users’ questions. DatasetQ&A from Stack Exchange [90, 177] from stack-exchange-preferences R 1 reward-model-deberta-v3-base R 2 reward-model-electra-large-discriminator FigureFigure 2(c) Task nameMovie review DescriptionGenerate movie reviews that accurately describe a movie. Prompt“Generate a movie review.” DatasetIMDB reviews [178] from IMDB R 1 reward-model-deberta-v3-base R 2 reward-model-electra-large-discriminator FigureFigure 2(d) Task nameHelpful assistant DescriptionProvide helpful and harmless answers to potentially complex and sensitive questions. PromptNo prompt, only users’ questions. DatasetHelpfulness and harmlessness datasets [49] from h-rlhf R 1 reward-model-deberta-v3-large-v2 R 2 reward-model-electra-large-discriminator R 3 reward-model-deberta-v3-base-v2 R 4 reward-model-deberta-v3-base FigureFigures 2(e) and 2(f) 32 D.2 Inspection of the predictions by the interpolated models This supplementary material includes multiple visualizations of predictions generated by models with interpolated weights. For example, Table 2 provides generated summaries, that remain grammatically coherent. We include more generated samples on this website. To quantitatively validate that samples generated by interpolated models do not suffer from reduced quality, we provide evaluations with general-purpose quality metrics. Specifically, in Figure 11, we measure the perplexity (of MLMS [179] or of GPT2 [180]) for summaries generated when λ-interpolating between two LLMs fine-tuned on the two summary rewards. We observe that, by interpolating the weights, we also interpolate the metrics. In conclusion, we confirm quantitatively that RS does not deteriorate quality. Table 2: Examples of generated summaries about vampire vigilantes in Malawi news, for(1−λ)·θ 1 +λ·θ 2 interpolating betweenθ 1 andθ 2 fine-tuned onR 1 (evaluating completeness) andR 2 (evaluating faithfulness). λGeneration 0.0Malawi vigilante mobs have been killing people suspected of being vampires, prompting the UN and US embassy to declare no-go zones in the affected’ 0.1Malawi vigilante mobs have been killing people suspected of being vampires, prompting the UN and US embassy to declare no-go zones in the affected 0.2Malawi vigilante mobs have killed nine people since mid-September, prompting the United Nations and the US embassy to declare some parts of the country no- 0.3Malawi vigilante mobs have killed nine people, prompting the UN and US embassy to declare parts of the country no-go zones due to widespread 0.4Malawi vigilante mobs have killed nine people, prompting the UN and US embassy to declare some parts of the country no-go zones. 0.5Malawi vigilante mobs have arrested and killed suspected vampires, prompting the UN and US embassy to declare no-go zones and President Peter Muth 0.6Malawi vigilante mobs have arrested and killed suspected vampires, prompting the UN and US embassy to declare no-go zones. 0.7Malawi vigilante mobs have arrested suspected vampires, resulting in deaths and prompting the UN and US embassy to declare no-go zones. 0.8Malawi vigilante mobs have arrested suspected vampires, resulting in deaths and violence. 0.9Malawi vigilante violence has caused widespread panic and death, prompting authorities to arrest suspected members and investigate the belief in vampirism. 1.0Malawi vigilante violence has caused widespread panic and death. 0.00.20.40.60.81.0 λ 50 100 150 200 250 300 350 Perplexity ( ↓ ) Perplexity (↓) MLMS GPT2 0.19 0.20 0.21 0.22 0.23 Quality ( ↑ ) Quality (↑) News quality Figure 11: News summaries generated byλ-interpolatedLLMs evaluated in terms of perplexity (by MLMS [179] or by GPT2 [180]) or news quality by this classifier. 33 E Image-to-text: captioning with diverse statistical rewards E.1 Experimental details We summarize the key implementation details of our captioning experiments in Table 3. In short, we took the state-of-the-art network [96] for captioning on COCO, fine-tuned with their code and only changed the reward. In more details, since theself-criticalpaper [24] (a variant of REINFORCE [98] with a specific estimation of the baseline score) it is now common in captioning to optimize the CIDEr reward [31] after a first step of supervised fine-training. The recent ExpansionNetv2 [96] follows this strategy to reach state-of-the-art results, with a Swin Transformer [97] visual encoder and a block static expansion for efficiency. We investigate whether additional RL trainings on the other traditional statistical metrics can help. We use the code from [96] and their hyperparameters, only reducing the batch size from 24 to 18 to fit in our GPUs and consequently adapting the learning rate. Table 3: Captioning experiments: key implementation details. Model ArchitectureExpansionNetv2 [96] Visual encoderSwin Transformer [97] Visual encoder pre-trainingImageNet 22k [181] Fine-tuningCross-entropy then CIDEr RL [24] on COCO [94] RL procedure Fine-tuning strategyUsually frozen visual backbone, but end-to-end in Figure 14(d) RL algorithmSelf-critical [24], a variant of REINFORCE [98] OptimizerRadam [182] DatasetCOCO [94] and Karpathy split [99] RewardsBLEU [29] (with 1-gram or 4-grams), ROUGE [30], METEOR [95], CIDEr [31] Learning rate1e-5 Batch size18 Gradient accumulation2 WarmupAnneal 0.8 during 1 epoch Epochs6 HardwareGPU V100 32G Compute budget1500 GPUh E.2 Additional results 0.4170.4180.4190.4200.4210.4220.4230.424 R 1 :BLEU4 0.606 0.607 0.608 0.609 0.610 R 2 : ROUGE pre-trained init BLEU4 rewarded ROUGErewarded BLEU4+ROUGE 2 rewarded λ= 0.5 RL:BLEU4 RL:ROUGE MORL: BLEU4+ROUGE 2 RS: (1−λ)·θ BLEU4 +λ·θ ROUGE (a) BLEU4 and ROUGE. 0.780.790.800.810.820.830.840.85 R 1 :BLEU1 0.300 0.302 0.304 0.306 0.308 0.310 0.312 R 2 : METEOR pre-trained init BLEU1 rewarded METEORrewarded BLEU1+METEOR 2 rewarded λ= 0.5 RL:BLEU1 RL:METEOR MORL: BLEU1+METEOR 2 RS: (1−λ)·θ BLEU1 +λ·θ METEOR (b) BLEU1 and METEOR. 0.83000.83250.83500.83750.84000.84250.84500.84750.8500 R 1 :BLEU1 1.370 1.375 1.380 1.385 1.390 1.395 R 2 : CIDEr pre-trained init BLEU1 rewarded CIDErrewarded BLEU1+CIDEr 2 rewarded λ= 0.5 RL:BLEU1 RL:CIDEr MORL: BLEU1+CIDEr 2 RS: (1−λ)·θ BLEU1 +λ·θ CIDEr (c) BLEU1 and CIDEr. Figure 12: Additional results in captioning with more rewards, complementing Figure 3. Specifically, Figure 12(a) usesR 1 =BLEU4andR 2 =ROUGE; then, withR 1 =BLEU1, Figure 12(b) usesR 2 =METEOR and Figure 12(c) usesR 2 =CIDEr. In particular, the latter shows the failure when optimizing CIDEr; indeed, let’s recall that the pre-trained initialization [96] has already been trained by optimizing CIDEr [24]. Thus optimizing CIDEr a second time does not help, neither in CIDEr nor in other rewards. That’s why in Figure 3(c) we consider the initialization as the network parametrization optimized for CIDEr. 34 0.00.20.40.60.81.0 λ 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Normalized rewards RS: (1−λ)·θ BLEU1 +λ·θ BLEU4 R 1 :BLEU1 BLEU2 BLEU3 R 2 :BLEU4 ROUGE METEOR CIDEr (a)R 2 =BLEU4. 0.00.20.40.60.81.0 λ 0.0 0.5 1.0 1.5 2.0 2.5 Normalized rewards RS: (1−λ)·θ BLEU1 +λ·θ METEOR R 1 :BLEU1 BLEU2 BLEU3 BLEU4 ROUGE R 2 :METEOR CIDEr (b)R 2 =METEOR. 0.00.20.40.60.81.0 λ 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Normalized rewards RS: (1−λ)·θ BLEU1 +λ·θ CIDEr R 1 :BLEU1 BLEU2 BLEU3 BLEU4 ROUGE METEOR R 2 :CIDEr (c)R 2 =CIDEr. 0.00.20.40.60.81.0 μ 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Normalized rewards MORL: (1−μ)×BLEU1 +μ×ROUGE R 1 :BLEU1 BLEU2 BLEU3 BLEU4 R 2 :ROUGE METEOR CIDEr (d) MORL. Figure 13: Additional results in captioning when measuring performances on all rewards and varying the interpolating coefficients, complementing Figure 5(b). In Figures 13(a) to 13(c), we extend the results for RS withR 1 =BLEU1and for varyingR 2 ; the optimalλdepends on the similarity between the evaluation metric andR 1 andR 2 . We also see in Figure 13(c) that all rewards are normalized to1for the CIDEr-initialization. In Figure 13(d), we perform the same analysis for MORL while varying the weightingμover the proxy rewards R 1 =BLEU1andR 2 =ROUGE; we recover similar curves than in Figure 5(b) for RS. 0.8250.8300.8350.8400.8450.850 R 1 :BLEU1 0.603 0.604 0.605 0.606 0.607 0.608 0.609 0.610 0.611 R 2 : ROUGE BLEU1 rewarded ROUGErewarded BLEU1+ROUGE 2 rewarded λ= 0.5 WiSE: (1−λ)·θ init +λ·θ BLEU1 WiSE: (1−λ)·θ init +λ·θ ROUGE RS: (1−λ)·θ BLEU1 +λ·θ ROUGE RS: (1−λ)·θ BLEU1 +λ·θ MORL RS: (1−λ)·θ ROUGE +λ·θ MORL (a) Exploring new WI strategies. 0.8250.8300.8350.8400.8450.850 R 1 :BLEU1 0.603 0.604 0.605 0.606 0.607 0.608 0.609 0.610 0.611 R 2 : ROUGE pre-trained init BLEU1 rewarded κ= 0.5 between twoBLEU1 ROUGE rewarded κ= 0.5 between twoROUGE λ= 0.5 between BLEU1 andROUGE RL:BLEU1 v1 RL:BLEU1 v2 MS: (1−κ)·θ v1 B +κ·θ v2 B RL:ROUGEv1 RL:ROUGEv2 MS: (1−κ)·θ v1 R +κ·θ v2 R RS fronts: (1−λ)·θ v1 orv2 B +λ·θ v1 orv2 R RS of MS: 1−λ 2 ·(θ v1 B +θ v2 B ) + λ 2 ·(θ v1 R +θ v2 R ) (b) Results variances and model soups (MS). 0.810.820.830.840.850.86 R 1 :BLEU1 0.5900 0.5925 0.5950 0.5975 0.6000 0.6025 0.6050 0.6075 0.6100 R 2 : ROUGE pre-trained init λ= 0.0 λ= 1.0 λ= 0.5 λ= 1.1 λ= 1.3 λ= 1.5 λ=−0.1 λ=−0.3 λ=−0.5 RS: (1−λ)·θ BLEU1 +λ·θ ROUGE (c) Extrapolation withλoutside of[0,1]. 0.8200.8250.8300.8350.8400.8450.8500.855 R 1 :BLEU1 0.598 0.600 0.602 0.604 0.606 0.608 0.610 R 2 : ROUGE pre-trained init BLEU1 rewarded ROUGErewarded BLEU1+ROUGE 2 rewarded λ= 0.5 RL:BLEU1 RL:ROUGE MORL: BLEU1+ROUGE 2 RS: (1−λ)·θ BLEU1 +λ·θ ROUGE (d) End-to-end training. Figure 14: Additional results in captioning withR 1 =BLEU1andR 2 =ROUGE. In Figure 14(a), we investigate interpolating the fine-tuned networks with the pre-trained initialization as in WiSE [183]; this only reveals a small portion of the front. In contrast, the interpolation withθ MORL (μ= 0.5) solution improves RS’s front: this highlights some limitations in Hypothesis 2 and strict Pareto optimality of RS. Adding the MORL solutions asintermediateweights may help interpolate between two weights too distant. This suggests some practical complementarity between RS and MORL; given a training budget larger than the number of rewards, one may learn a few MORL for varying0≤μ≤1, and then interpolate the obtained solutions. Figure 14(b) shows results’ variance with two RL trainings for BLEU1, and two for ROUGE, each time with a different seed defining the data ordering and augmentations. Though we observe some randomness, the Hypothesis 1 is consistently validated. Moreover, it presents the fronts described when we interpolate weights fine-tuned on a shared reward, as in model soups (MS) [67,68]; it mostly reduces variance and reveals only a small portion of the spectrum of preferences, validating the need to fine-tune on different rewards (as proposed in RS) to reveal the front across the entire space of preferences. Finally, the orange line shows that RS and MS can be complementary, byλ-interpolating the MS for BLEU1 and the MS for ROUGE withκ= 0.5. Figure 14(c) presents the extrapolation results whenλgoes outside of[0,1]. This suggests that we can artificially reduce a reward with negative coefficients, as studied in [141]. Finally, Figure 14(d) shows the results when the networks are trained end-to-end, rather than keeping the backbone frozen. This validates the efficiency of rewarded soups in a new more general setting where all layers are trainable. 35 F Text-to-image: diffusion models with diverse RLHFs F.1 Experimental details Task description.Several works have studied the problem of aligning the output of diffusion models with human feedbacks [25,26,33]. Notably, diffusion models can be fine-tuned to match human aesthetic perception. As for any subjective metric, there is a variety of reward models capturing different aesthetics. In our experiments, the two first reward models were trained in a supervised setting to match human quality ratings collected on large image datasets. Specifically, the firstR 1 is theavaaesthetic model, available here, trained on 250.000 images from the AVA dataset [103], based on CLIP features. The secondR 2 is thecafeaesthetic model, available here, trained on 3500 real-life and anime/manga images. Moreover, in Figure 15, we also consider ansfwdetector, estimating the probability of an image beingsafeby computing the cosine similarity with the CLIP embeddings of a set ofunsafewords, as already done to filter the LAION dataset [184]. Implementation details.We use a 2.2B parameters diffusion model trained on an internal dataset of 300M images, which reaches similar generation quality as Stable Diffusion [102] in terms of CLIP alignment and FID scores on prompts from the 5000 images of the COCO test dataset (CLIPScore 30.0 vs 30.2 for Stable Diffusion, FID 19.0 vs 19.1 for Stable Diffusion). Given a reward modelR, we first generate 10000 images with the pre-trained diffusion model on prompts from the COCO dataset, and compute the rewards for every generated image. For computational efficiency, we keep only a datasetD ′ containing the 50% images with the best scores, and rescale rewardsRlinearly into rso thatmin x 0 ∈D ′ r(x 0 ) = 0and 1 |D ′ | P x 0 ∈D ′ r(x 0 ) = 1 . Then, wefine-tune the diffusion model on the reward-weighted negative log-likelihood [25]: L=E (x 0 ,Q)∈D,ε∼N(0,1),t∼Uniform(0,T) r(x 0 )×∥ε θ (x t ,t,Q)−ε∥ 2 ,(19) whereε θ is the noise estimation network,Tis the total number of training steps,r(x 0 )is the rescaled reward of imagex 0 andQis the text associated to imagex 0 . As a side note, on-policy RL would require performing loops of image generations and model fine-tunings [185], but we only perform a singleofflineiteration for simplicity. Moreover, for efficiency, we only fine-tune 10% of the diffusion model’s weights [104] corresponding to the cross-attention layers and the bias/scaling parameters. As further described in Table 4, we apply the Adam [167] optimizer for 4000 steps with a batch size of 64 and a learning rate of 5e-6. To report results for each model (fine-tuned or interpolated via RS), we generate 1000 images from a held-out set of COCO prompts and then we average the scores given by the reward models. To reduce the variance in image generation, each prompt has a unique seed for all models, so that the input noise given to the diffusion model only depends on the text prompt. Table 4: Image generation experiments: key implementation details. Model ArchitectureGLIDE (2.2B parameters) Pre-trainingInternal dataset of 300M captioned images RL Procedure Fine-tuning objectiveReward-weighted diffusion loss Fine-tuned parametersCross-attention layers and bias/scale OptimizerAdam [167] DatasetGenerated with COCO prompts Rewardsava[103] andcafeandnsfw Learning rate5e-6 Batch size64 Epochs25 HardwareSingle GPU V100 32G Compute budget500 GPUh F.2 Additional results RS can trade-off between the two aesthetic rewards in Figure 6(a), allowing adaptation to the user’s preferences at test time. Yet, we show some limitations in the spider map of Figure 15, when 36 computing MORL and RS on all three rewards:ava,cafeand also thensfw. In this case, MORL has higher scores than RS. We speculate this is because thensfwis very different from aesthetic preferences. Actually, thensfwis inversely correlated with image quality: lower quality images result are less flagged asunsafe. This shows some limitations of weight interpolation when combining antagonist rewards. An improved strategy would first learn the MORL of theN= 3rewards, and then optimize each reward independently from this improved initialization, before applying RS. ava cafensfw 5.245 5.271 5.296 0.795 0.808 0.82 0.934 0.942 0.951 Init RL:ava RL:cafe RL:nsfw MORL: ∑ N=3 i=1 R i /3 RS: ∑ N=3 i=1 θ i /3 Figure 15: Image generation: spider map, withava,cafeandnsfwreward models. F.3 Visualization of generated images from interpolated models We show in Appendix F.3 images generated by rewarded soups when varying the interpolation coefficientλbetween the two models fine-tuned for theavaand thecafeaesthetic rewards. You can find additional qualitative results for this experiment on this website. Moreover, in Appendix F.3, we measure the FID [186] and the CLIPScore [187] of the images generated by the same interpolated models. This confirms quantitatively that images generated by interpolated models remain coherent. Promptλ= 0.0λ= 0.2λ= 0.4λ= 0.6λ= 0.8λ= 1.0 a dog stands inside of a boat as it stares at a camera A family room with woodfloorand beige walls and a mattressleaning againstastone wall. A man sitting on top of a chair hold- ing up a cell phone. Amansitting underneathan umbrella and other structures. Figure 16: Visualization of images generated with rewarded soups for a varying interpolation coefficientλ between the two models fine-tuned for theava(corresponding toλ= 0) andcafe(corresponding toλ= 1) reward models. We can see that all interpolated models produce images of similar quality compared to fine-tuned models, demonstrating linear mode connectivity between the two fine-tuned models. 37 0.00.20.40.60.81.0 λ 32.3 32.4 32.5 32.6 32.7 32.8 32.9 33.0 FID ( ↓ ) FID (↓) 31.025 31.050 31.075 31.100 31.125 31.150 31.175 31.200 31.225 CLIPScore ( ↑ ) CLIPScore (↑) Figure 17: Images generated byλ-interpolated diffusion models evaluated in terms of realism by FID [186] or text alignment by CLIPScore [187]. G Text-to-box: visual grounding of objects with diverse sizes G.1 Experimental details We show the implementation details in Table 5. We use UnIVAL [106], a model pre-trained solely on public benchmarks, to solve a variety of multimodal tasks such as VQA, visual grounding and image captioning. It is then fine-tuned on RefCOCO+ dataset for visual grounding. During the last fine-tuning phase, we complement the cross-entropy loss with an additional REINFORCE [98] term rewarding accuracy when the object is of the considered size. This means that the loss forθ Small is− log(ˆy) + 5×1 area(ˆy)is small ×1 AUC(y,ˆy)>0.5 ×log(y) for an object with ground-truth box ˆyand predictiony. The image is discretized into1000×1000bins before calculating the box areas. The task is illustrated in Figure 18. Figure 18: Illustration of the visual grounding task. The RS model results from the average ofN= 3weights specialized to detect respectively small, medium and large objects. The model takes a text (one description at a time) as input and outputs the bounding box in the corresponding region of the image. We show an example of small, medium and large predictions, and the associated ground truths in green. These texts and image are from the validation set of RefCOCO+ [105]. 38 Table 5: Visual grounding experiments: key implementation details. Model ArchitectureUnIVAL [106] Visual encoderResNet-101 Pre-trainingCross-Entropy on Public datasets (VQA, VG, Captioning) Supervised fine-tuningCross-Entropy on RefCOCO+ [105] RL procedure Fine-tuning strategyend-to-end DatasetRefCOCO+ [105] RL algorithmCross-entropy +5×REINFORCE Reward SmallIoU>0.5 for object with area<30000 Reward MediumIoU>0.5 for object with30000≤area<100000 Reward LargeIoU>0.5 for object with100000≤area OptimizerAdam Learning rate3e-5 Batch size256 Epochs10 Hardware8 GPU 60GB Compute budget800 GPUh G.2 Additional results 21.221.421.621.822.022.222.422.622.8 R 1 :Small 44.00 44.25 44.50 44.75 45.00 45.25 45.50 45.75 R 2 : Medium pre-trained init Small rewarded Medium rewarded Small+Medium 2 rewarded RS: (1−λ)·θ Small +λ·θ Medium (a) Small and Medium. 43.544.044.545.045.5 R 2 :Medium 12.0 12.2 12.4 12.6 12.8 R 3 : Large pre-trained init Medium rewarded Large rewarded Medium+Large 2 rewarded RS: (1−λ)·θ Medium +λ·θ Large (b) Medium and Large. Small MediumLarge 21.57 22.14 22.71 43.917 44.833 45.75 11.95 12.4 12.85 Pre-trained init RL:Small RL:Medium RL:Large MORL: ∑ N=3 i=1 R i /3 RS: ∑ N=3 i=1 θ i /3 (c) Spider map. Small MediumLarge 21.46 21.82 22.18 44.277 44.553 44.83 12.343 12.487 12.63 Pre-trained init RS of CE: ∑ N=3 i=1 θ CE i /3 RS of RL: ∑ N=3 i=1 θ RL i /3 (d) CE vs. RL. Figure 19: Results in visual grounding on RefCOCO+ [105]. We use REINFORCE [98] to improve directly the non-differentiable accuracy, i.e., predict boxes with IoU>0.5w.r.t. the ground-truth. Fine-tunings are specialized on either small, medium, or large objects. These experiments complement Figures 6(b) and 19(c). Figure 19(c) shows that improving results on all sizes simultaneously is challenging, as MORL performs similarly to the initialization. Finally, Figure 19(d) motivates the use of RL to fine-tune on different sizes. Indeed, the results for (the proposed) RS of RL are significantly better than the results for RS of CE, where we average weights specialized on different sizes by fine-tuning with cross-entropy (rather than with REINFORCE). H Text&image-to-text: VQA with diverse statistical rewards We detail our VQA experiments in Table 6, where the goal is to answer a question w.r.t. an image. Our pre-trained model is OFA [107], which was trained on a variety of multimodal tasks such as VQA, visual grounding and image captioning. We then fine-tune it only on the VQA v2 dataset using the cross-entropy as the loss. Finally, we fine-tune with REINFORCE [98] on the different rewards. We use a held-out set for the RL fine-tuning that was not used to train the main model. The rewards are BLEU1 and METEOR: as there are 10 ground-truth answers for each VQA example, the final reward is the average score over all those answers. 39 Table 6: Visual question answering experiments: key implementation details. Model ArchitectureOFA Medium [107] Pre-trainingPublic datasets (multimodal, text-only, image-only) [107] Supervised fine-tuningCross-Entropy fine-tuning on VQA v2 [?] RL procedure Fine-tuning strategyend-to-end DatasetVQA v2 [?] RL algorithmREINFORCE RewardBLEU1 and METEOR OptimizerAdam Learning rate1e-5 Batch size32 Epochs5 Hardware4 GPU 32G Compute budget20GPUh I Locomotion with diverse engineered rewards Task description.This experiment takes on the intricate challenge of controlling a running humanoid in the Brax [113] physics engine. The complexities involved in achieving natural or fast movement in continuous control environments serve as a testament to the robustness of our approach. The fine-tuning procedure is carried out on two distinct reward functions, with the aim of refining the running behavior of the humanoid, potentially resulting in smoother motion patterns. You can find qualitative results of this experiment on this website. Pre-training.According to Remark 1, the LMC requires pre-training the base policy before fine- tuning. Thus, as the pre-training task, we use the default dense reward implemented in Brax: R=velocity−0.1× P t a 2 t . This pre-training phase also serves to collect statistics about observa- tions and normalize them before inputting to the model (as it facilitates training). We used the Brax implementation of PPO [84]. The pre-trained policy is saved while the value function is discarded. Fine-tuning.We keep the same environment as in pre-training. We also use the normalization procedure inherited from pre-training but freeze the statistics. Two reward functions are designed: ariskyone forR 1 =velocityand acautiousone whereR 2 =velocity− P t a 2 t . We tried a few hyperparameters (see the values in brackets in Table 7) but results (see Figure 20) remain close and consistently validate our working hypotheses. 40 Table 7: Locomotion experiments: key implementation details. PPO Pre-training Interactions5e8 Reward Scaling1.0 Episode Length1000 Unroll Length10 Discounting0.99 Learning Rate5e-5 Entropy Cost1e-3 Number of environments in parallel4096 Batch Size1024 Hardware1GPU Tesla V100-SXM2-16GB Runtime per experiment80min PPO Fine-tuning Interactions1e8 Reward Scaling1. Normalize observationsTrue Unroll Length10 Discounting0.97, 0.99, 0.999 Learning Rate1e-5, 3e-5, 1e-4 Entropy Cost1e-3, 3e-3, 1e-2 Number of environments in parallel4096 Batch Size1024 Hardware1GPU Tesla V100-SXM2-16GB Runtime per experiment20min Model architecture Policy ArchitectureMLP Nb of Layers6 Hidden Size512 Value ArchitectureMLP Nb of Layers5 Hidden Size256 400050006000 0 1000 2000 3000 4000 R 2 : cautious 4000500060007000 0 1000 2000 3000 4000 400050006000 0 1000 2000 3000 4000 400050006000 0 1000 2000 3000 4000 R 2 : cautious 4000500060007000 0 1000 2000 3000 4000 400050006000 0 1000 2000 3000 4000 400050006000 R 1 :risky 0 1000 2000 3000 4000 R 2 : cautious 400050006000 R 1 :risky 0 1000 2000 3000 4000 400050006000 R 1 :risky 0 1000 2000 3000 4000 Pre-trained init RS: (1−λ)·θ risky +λ·θ cautious Figure 20: Analysis of results’ variance for the locomotion task when varying the hyperparameters. Each column icorresponds to thei-thθ risky , interpolated in case(i,j)towards thej-thθ cautious . The Figure 7(b) is actually the plot from case(1,1). 41