Paper deep dive

Weak-to-Strong Generalization under Distribution Shifts

Myeongho Jeon, Jan Sobotka, Suhwan Choi, Maria Brbić

Year: 2025Venue: NeurIPS 2025Area: Scalable OversightType: EmpiricalEmbeddings: 96

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 5:16:22 PM

Summary

The paper introduces RAVEN (Robust AdaptiVe wEightiNg), a framework designed to improve weak-to-strong (W2S) generalization under distribution shifts. It addresses the failure of naive W2S methods in out-of-distribution (OOD) settings by dynamically learning to weight an ensemble of weak models and employing an easy-sample guided initialization strategy. RAVEN demonstrates significant performance improvements across image classification, text classification, and preference alignment tasks.

Entities (5)

RAVEN · framework · 100%Distribution Shift · phenomenon · 95%Weak-to-Strong Generalization · concept · 95%Adaptive Weighting · technique · 90%Easy-sample guided initialization · technique · 90%

Relation Signals (4)

RAVEN → addresses → Distribution Shift

confidence 95% · To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models

RAVEN → utilizes → Adaptive Weighting

confidence 95% · RAVEN incorporates two key components: (i) adaptive weighting

RAVEN → utilizes → Easy-sample guided initialization

confidence 95% · RAVEN incorporates two key components: (ii) easy-sample guided initialization.

Weak-to-Strong Generalization → failsunder → Distribution Shift

confidence 90% · naive weak-to-strong generalization fails under distribution shifts

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:As future superhuman models become increasingly complex, accurately supervising their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

95,858 characters extracted from source content.

Expand or collapse full text

Weak-to-Strong Generalization under Distribution Shifts Myeongho Jeon 1,∗ Jan Sobotka 1,∗ Suhwan Choi 2,3,∗ Maria Brbi ́ c 1,† 1 EPFL 2 Seoul National University 3 CRABs Abstract As future superhuman models become increasingly complex, accurately supervis- ing their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we pro- pose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over30%on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision. 1 Introduction Recent AI systems have reached near-human performance through extensive pre-training on large datasets, followed by fine-tuning with human supervision. Techniques such as supervised fine- tuning, reinforcement learning with human feedback (RLHF) [17,61,50], and direct preference optimization (DPO) [43,56] are key examples of effectively aligning models with human preferences. A fundamental assumption in these methods is that the human supervision is of high quality. However, when the data to be annotated is beyond human comprehension, providing reliable supervision becomes challenging. For instance, in domains such as cosmology, healthcare, or biology, even experts may struggle with accurate labeling. Furthermore, during the alignment phase with human preferences, if a superhuman-level model generates highly complex outputs, humans may struggle to fully understand them, making it challenging to provide effective feedback. This challenge is known as weak supervision [10]. So, how can humans supervise a superhuman model that surpasses their own capabilities? To mimic this future scenario, an analogous framework has been proposed in [10], where a weak model simulates human supervision, while a strong model acts as a proxy for a potential superhuman model. In this setup, weak supervision signals are utilized to train the strong model, aiming to surpass the weak model’s performance and approach the performance achieved when trained with ground-truth (GT) labels. Conceptually, this approach, termed weak-to-strong (W2S) generalization [10], could lead toward developing superhuman models, as surpassing the performance of a weak model in this context could eventually mean exceeding human capabilities. Moreover, such success would be practical even before superhuman models emerge—for instance, aligning GPT-5 using only GPT-4-level supervision could simplify model alignment today [10]. Recent works [10,26,45,19] ∗ Equal contribution † Correspondence to: mbrbic@epfl.ch 39th Conference on Neural Information Processing Systems (NeurIPS 2025). arXiv:2510.21332v2 [cs.LG] 25 Nov 2025 Weak-to-Strong Generalization Source data Fine-tuning data Target data Train Weak Supervision Test Strong model Weak-to-Strong Generalization under Distribution Shifts Source data Train Weak and Misaligned Supervision Test Weak model Strong model Fine-tuning data Target data Weak model In-distribution Out-of-distribution Train Train Figure 1: An illustration of weak-to-strong generalization under distribution shifts. Left: The original W2S generalization framework [10]. Right: Our extended framework that incorporates distribution shift. In this scenario, the weak model is not only limited in understanding but also unfamiliar with the fine-tuning distribution, making it less reliable for annotating data to supervise the strong model. have shown that W2S generalization is indeed feasible, meaning that strong pre-trained models naturally generalize beyond their weak supervisors. However, humans may encounter data that is not only complex but also unfamiliar, making the weak supervision even weaker. For example, a radiologist used to one type of imaging data may misinterpret scans from a different machine or mislabel rare diseases outside their usual clinical practice [8]. As a result, expert annotations can become even less reliable due to unfamiliarity with the data. To simulate this, we consider a scenario in which the weak model is trained on data drawn from a distribution that differs significantly from the strong model’s fine-tuning data, a setting we refer to as weak and misaligned supervision (Figure 1). This raises the question: Is W2S generalization still feasible under distribution shifts? Following this question, we interestingly find that weak supervision becomes significantly less effective in out-of-distribution (OOD) settings compared to in-distribution (InD) settings and, in some cases, even not feasible, leading to the strong model performing worse than the weak model. This is a nontrivial issue, as it suggests that unnoticed misaligned human perspectives can affect the annotation process and potentially reduce performance. These findings motivate the need for a robust W2S generalization framework that remains effective under distribution shifts. Here, we propose Robust AdaptiVe wEightiNg (RAVEN) 3 , a robust W2S generalization framework in which the strong model dynamically learns to combine the outputs from an ensemble of weak annotators. Under distribution shift, weak models exhibit higher variance in performance compared to the InD setting. RAVEN mitigates this by learning to assign higher weights to more reliable weak models. Specifically, the supervision weights assigned to the weak models are updated iteratively and jointly trained with the strong model’s parameters. This approach effectively handles high variance of weak models by allowing the strong model to prioritize the most favorable weak model for the fine- tuning data distribution. Additionally, to guide the strong model to learn how to choose the reliable weak model in the early stages of training, RAVEN introduces easy-sample guided initialization, which trains the strong model exclusively on samples where the weak models consistently provide the same predictions. We evaluate RAVEN on image classification, text classification, and preference alignment in text generation tasks. RAVEN achieves a55%improvement in image classification, a57%improvement in text classification, and a33%improvement in preference alignment compared to the best alternative baselines for each task. Remarkably, although the information about the performance of weak models is unknown to the strong model, we observe that the strong model typically assigns the highest weight to the best-performing weak model without any additional guidance. 3 Project website with code: https://brbiclab.epfl.ch/projects/raven 2 2 Problem statement Weak-to-strong generalization. In W2S generalization setting [10], a strong (large) pretrained model is fine-tuned using labels generated by a weak (small) model, denoted byf s andf w , respectively. Given input spaceXand label spaceY, the data is divided into sourceP src (X,Y ), fine-tuning P tuning (X,Y ), and target dataP trg (X,Y ). The following procedure is then evaluated: (i) Generate a weak supervisor:f w is trained onP src with ground-truth (GT) labels, denoted byf src w . (i) Train a strong model with weak supervision:f s is trained onP tuning using weak pseudo-labels generated byf src w , denoted byf pseudo s . (i) Train a strong model with GT as a ceiling:f s is trained on P tuning with GT, denoted byf gt s . The performance gap recovered (PGR) onP trg is then calculated as follows: PGR := Acc(P trg ;f pseudo s )− Acc(P trg ;f src w ) Acc(P trg ;f gt s )− Acc(P trg ;f src w ) ,(1) whereAcc(A;B)denotes the accuracy of model B on data A. The goal of W2S generalization is to achieve a PGR close to 1 by making the target modelf pseudo s approximatef gt s , indicating that the model reaches ground-truth-level performance even when trained with weak supervision. In [10], the assumption is that P src , P tuning , and P trg are drawn from the same underlying distribution. Weak-to-strong generalization under distribution shifts. We extend the concept of W2S generaliza- tion by considering distribution shifts betweenP src andP tuning . We define distribution shift through the generalization gap:∆R(f ) = R tuning (f )− R src (f ), whereR(f ) = E (x,y)∼P [L(f (x),y)] is the expected risk,fdenotes the model, andLis the loss function. A large∆R(f )indicates substantial distribution shift fromP src toP tuning , which can make weak supervision misaligned 4 . This conceptualizes the case where humans annotate difficult-to-understand and unfamiliar domains, leading to weak and misaligned supervision. This could, for example, happen with medical data that are both highly specialized and cross-institutional. In this setting, PGR, as defined in Eq. (1), evaluates how effectively weak supervision under distribution shift can be leveraged to enhance the capabilities of the strong model. Observation. To evaluate the robustness of naive W2S generalization under distribution shifts, we analyze changes in PGR that happen in OOD scenarios. Specifically, we introduce a shift from IMAGENET to IMAGENET-C, treat- ing 19 corruption types as distinct domains. For comparison, we establish both InD and OOD setups. In both cases, we use the IMAGENET training set asP src , while the validation set from IMAGENET serves asP tuning andP trg for the InD scenario, and IMAGENET-C serves as P tuning and P trg for the OOD scenario. We find that OOD generalization rarely lever- ages the strong model’s capabilities as much as when no distribution shift is present (Figure 2). Surprisingly, on 11 out of 19 IMAGENET-C cor- ruption cases, the W2S generalization is even infeasible, i.e., the W2S performance is worse than that of the weak model. This shows the need for an approach that achieves a robust W2S generalization under distribution shifts. 10010203040506070 PGR No-shift Saturate Brightness Elastic Gaussian blur Defocus blur JPEG Pixelate Contrast Spatter Snow Fog Speckle noise Motion blur Gaussian noise Shot noise Impulse noise Zoom blur Frost Glass blur 68.49 32.68 28.26 9.11 3.04 3.02 2.61 2.01 1.43 -0.09 -0.19 -0.42 -0.69 -1.10 -1.63 -1.70 -1.85 -2.16 -3.09 -8.39 Figure 2: Performance variation with distribution shifts. The Cyan bar indicates positive PGR of InD, gray represents positive PGR for OOD, and red de- notes negative PGR. PGR is reported as a percent- age. Detailed results can be found in Appendix B. 3 Preliminary WeakS-to-strong generalization. Inspired by the concept that multiple experts can collectively provide effective supervision even if each individual expert is insufficient on its own, weakS-to-strong framework [19] utilizes multiple weak models for training. A straightforward approach to ensemble 4 A more detailed formal definition of distribution shift is provided in Appendix A. 3 the weak models is to use a weighted sum to combine their predictions, followed by calculating the loss: L ensemble (Θ S ,W) :=L CE (f s (x), M X i=1 w (i) f src w i (x)),(2) whereW :=w (i) M i=1 denotes pre-defined weights set in ensembling (e.g., the average)Mweak models andL CE denotes cross-entropy loss.f w i denotesi-th weak model andΘ S represents the parameters of the classifier in the strong model f s . 4 RAVEN: Robust adaptive weighting approach In W2S generalization under distribution shift, the challenge is that weak models are trained on data distributions that are different from the strong model’s fine-tuning data distribution, making them even less reliable. Thus, leveraging an ensemble of weak models in Eq. (2) becomes an effective strategy. However, compared to the InD setting, weak models have substantially higher performance variance in OOD scenarios (Figure 3), and their effectiveness varies considerably across domains (Appendix H.3). Similarly, in an analogous real-world scenario, some human annotators may adapt better than others to unfamiliar domains. Consequently, it is crucial to identify more reliable weak models within the ensemble. This motivates the need for a dynamic selection mechanism that selects the most suitable weak model(s) for a given fine-tuning dataset P tuning . Motivated by this, we propose RAVEN, a robust W2S generalization framework that dynamically learns how to combine different weak models. To improve ensemble performance in OOD settings, RAVEN incorporates two key components: (i) adaptive weighting, and (i) easy-sample guided initialization. OODInD 30 40 50 60 Accuracy iWildCam OODInD 60 70 80 90 Camelyon17 OODInD 24 27 30 33 fMoW Figure 3: Across different datasets, we observe substantially higher performance variance in OOD scenarios than in InD scenarios. The variance is computed using20weak models that were initialized with different random seeds and trained on datasets from the WILDS benchmark [35]. Adaptive weighting. Among the weak models, some are more reliable than others. The key idea in RAVEN is to prioritize these models’ predictions by assigning different weights to their weak supervisions during the strong model’s training. We begin by trainingMweak models with different random seeds, which represent human annotators with diverse backgrounds. This approach is inspired by the findings of [40,37], which demonstrate that training individual networks with random initialization is often sufficient to achieve diversity in practice. In addition to achieving diversity via different random seeds, we also explore using different architectures and data sources when training weak models (Section 5.3). Leveraging these diverse weak models, we introduce adaptive weighting loss for the W2S training: L adaptation (Θ S , Θ W ) :=L CE f s (x), M X i=1 θ (i) f src w i (x) ! s.t. M X i=1 θ (i) = 1, θ (i) ∈ Θ W .(3) whereΘ W := θ (i) M i=1 represents the weights used to linearly combine the outputs of the weak models. Importantly, the weak models themselves are fixed, while the ensembling weightsθ (i) M i=1 are fully trainable. At each step, these weights are adjusted, leading to a change in how the weak models are utilized throughout training. With this objective, we train a linear classifier parameterized byΘ S on top of the frozen pre-trained backbone of the strong model. The backbone generates robust representations, serving as an effective anchor to train the linear classifier using the weak models. 4 Easy-sample guided initialization. To minimize the objective, the strong model may resort to shortcutting by assigning excessive weight to the weak model most similar to its initial classifier. This behavior is especially problematic during early training when the strong model’s classifier is still under-optimized and performs poorly. To address this, we implement a warm-up strategy for the strong model’s classifier using easy samples—those on which all the weak models agree, i.e., give the same predictions. During this phase, the weightsθ (i) ∈ Θ W are fixed to1/M. Afterward, we enable adaptive weighting across all samples, transitioning fromL ensemble with easy samples toL adaptation for all the samples. The concept of initializing training with easy samples has proven effective for learning with noisy labels [15]. This approach aligns with W2S generalization, as weak supervision often involves noisy labels resulting from incorrect predictions. Optimization procedure. After the easy-sample guided initialization, the loss for adaptive weighting L adaptation is optimized by alternating updates to Θ S and Θ W at each step, as follows: Θ ∗ S := arg min Θ S L adaptation (Θ S , Θ ∗ W ),Θ ∗ W := arg min Θ W L adaptation (Θ ∗ S , Θ W ).(4) This alternating optimization effectively balances the trade-off between optimizing the strong model and leveraging weak models. The overall RAVEN training procedure is summarized in Algorithm 1. Algorithm 1 Robust Adaptive Weighting (RAVEN) Require: Fine-tuning dataset D tuning , weak modelsf src w i M i=1 , pretrained strong backbone Ensure: Trained linear classifier Θ S 1: Easy-sample guided initialization: 2: Identify D easy =x∈ D tuning | arg max k f src w i (x)[k] = arg max k f src w j (x)[k], ∀i,j 3: Fix θ (i) = 1 M and define Θ W :=θ (i) M i=1 , with P M i=1 θ (i) = 1 4: Warm up Θ S on D easy by minimizingL ensemble (2) 5: Adaptive weighting on the full dataset: 6: while not converged do 7:Update weights: Θ W ← arg min Θ W L adaptation (Θ S , Θ W ), x∈ D tuning (3) 8:Update strong model: Θ S ← arg min Θ S L adaptation (Θ S , Θ W ), x∈ D tuning (3) 9: end while 10: return Θ S Remark 1. We observe that, in most cases, the strong model assigns the highest weightmax i θ (i) M i=1 to the best-performing weak model forP trg by the end of training, without requiring additional guidance. This behavior enhances robustness, as weak models often exhibit significant variation in OOD performance, and there is a strong positive correlation between weak and W2S performance. This capability is particularly advantageous because it allows for the automatic identification and utilization of the best weak model in terms ofP trg , even when the GT labels for the target data are unknown, and the best-performing weak model is therefore unclear. We provide a theoretical analysis of this property of RAVEN in Appendix C, along with detailed quantitative results in Section 5.3, Appendix H.4, and Appendix H.5. 5 Experiments We evaluate RAVEN on image classification, text classification, and preference alignment in text generation tasks. Classification training involves two stages: pre-training and fine-tuning. While pre-training is performed in a self-supervised or unsupervised manner [14,12,55], fine-tuning typically relies on human annotations. In our approach, we focus on fine-tuning by replacing human annotations with predictions of weak models. For text generation tasks, pre-training, supervised fine- tuning, and human preference alignment are the conventional learning phases for foundation models. In this work, we specifically focus on alignment, substituting human feedback with preference predictions from weak models. 5 5.1 Experimental setup Image classification. For OOD setting, we use IWILDCAM [5], CAMELYON17 [39], and FMOW [18] as benchmarks to evaluate our framework. In these datasets, the domain is defined by the location of the camera, the hospital, and the time, respectively. For each dataset, the training set is used as the source dataP src ,70%of the OOD validation set is randomly selected as fine-tuning dataP tuning , 10%is reserved as the validation set for hyperparameter tuning, and the remaining20%is designated as the target dataP trg . For the InD scenario, we adopt the same approach as [10], which utilized IMAGENET. Consistent with the setup in [10], we use AlexNet as the weak model and DINO ViT8/B as the strong model. RAVEN is compared to several W2S approaches, including Naive weak-to-strong (Naive) [10], weakS- to-strong with uniform weights (Ens) (Eq. (2)), Auxiliary confidence loss (Conf) [10], Bootstrapping (Boots) [10], Vision superalignment (V-sup) [26], Bayesian weakS-to-strong (Bayes) [19], and Co-supervised learning (Co-sup) [45] 5 . Text classification. We employ AMAZON-WILDS [35], MEDMCQA [51], and MEDQA [31] to evaluate RAVEN for text classification. AMAZON-WILDS, a sentiment analysis dataset, exhibits both domain and subpopulation shifts. In this dataset, domains correspond to individual reviewers. For the InD W2S scenario, the InD validation data is used in place of the OOD validation set. We designate Llama-3.2-1B [24], Qwen2.5-0.5B [65] as the weak models and Llama-3.1-8B, Qwen2.5- 7B, Qwen2.5-14B as the strong models. For the medical benchmarks, we adopt MEDMCQA as the source dataset and MEDQA as the fine-tuning and target datasets. These originate from different examination systems from India and the U.S. 6 , resulting in a natural domain shift between them. For these medical benchmarks, we use Qwen-2.5-0.5B as a weak model and Meditron-7B and Meditron-70B [16] as strong models. Preference alignment. While Burns et al.[10]highlighted the importance of alignment in W2S, their focus was solely on maximizing preference prediction accuracy rather than exploring alignment itself. Cui et al.[19]investigated the alignment phase but limited their focus to slot filling tasks. In contrast, we assess preference alignment in the context of text generation on H-RLHF [4], OPENAI SUMMARIZE FROM FEEDBACK[61], and HUMAN-LIKE DPO[11] datasets. For H-RLHF, we create a distribution shift by using Helpfulness samples forP src , randomly sampled Harmlessness samples forP tuning , and 1,000 Harmlessness samples forP trg ; for InD setting, we use only Harmlessness samples. In the second setup, we use OPENAI SUMMARIZE FROM FEEDBACK asP src and HUMAN-LIKE DPO asP tuning . To align the strong model with P tuning and the preference predictions of the weak models (i.e., feedback), we employ DPO [56]. To integrate RAVEN into alignment tasks, we propose a novel objective, DPO-R (Appendix E.2). We use Qwen2.5-0.5B and Qwen2.5-7B as the weak and strong models, respectively. Following Rafailov et al.[56], we use theGPT-4o win rate(WR) as our evaluation metric (Appendix E.4), and compare RAVEN to Naive, Ens, and Bayes baselines (details in Appendix E). Conf and V-sup baselines are not applicable due to the misalignment of outputs between the weak and strong models, and Boots and Co-sup baselines are unsuitable due to task differences. Experimental details. For each experiment, we perform a grid search to select the learning rate (including its initial value and decay schedule) and the number of training iterations based on validation loss. To determine the duration of the easy-sample guided initialization phase, we conduct a grid search over10%, 20%, 50%of the total iterations. Additional implementation details for the method and evaluation protocol are provided in the Appendix D. 5.2 Experimental results Image classification. Results on the image classification task show that RAVEN consistently outperforms all the baselines in the W2S generalization under distribution shift scenario (Table 1). RAVEN significantly enhances robustness in the OOD setting, while also surpassing other baselines 5 We adopt the domain generalization method outlined in Section 5.2 of [45] for robust W2S experiments, rather than their original approach, as it is better suited to the robust W2S scenario. Further details can be found in Appendix D.5. 6 MEDMCQA from the All India Institute of Medical Sciences, and MEDQA from the U.S. National Medical Board Examination. 6 Table 1: Image classification results. We report the average performance across 10 experiments, with PGR calculated as described in Appendix D.6. Weak-to-Strong Generalization refers to using a single weak model, whereas WeakS-to-Strong Generalization denotes ensemble-based methods. We highlight the best score in red and the second-best score in bold. Model(3) refers to the use of three weak models. For a fair comparison with Co-sup, which utilizes 7 weak models for IWILDCAM, FMOW and IMAGENET, and 5 for CAMELYON17, we report the RAVEN(>3) performance achieved using the same number of weak models as Co-sup. * indicates that the implementation code was created by us. The standard deviations are reported in Appendix G.1. Weak modelWeak-to-Strong GeneralizationWeakS-to-Strong GeneralizationStrong model AlexNetNaiveConfBootsV-supEns(3)*Bayes(3)*RAVEN(3)Co-sup(>3)RAVEN(>3)DINO ViT-B/8 Robust Weak-to-Strong Generalization (Out-of-distribution) IWILDCAM Accuracy 43.8346.8247.9748.7647.9249.3849.9652.7949.8355.4694.76 PGR-4.216.528.086.409.7310.9016.2510.0322.14- CAMELYON17 Accuracy66.9067.8368.9070.8467.8572.1868.7473.6770.0073.4597.93 PGR-2.475.9012.242.5117.135.1321.4614.1321.48- FMOW Accuracy 26.1825.6824.7525.9725.5727.5425.8929.4628.7031.8559.72 PGR--1.26-4.02-0.38-1.604.31-0.6710.0613.2414.06- Avg. Accuracy 45.6446.7847.2148.5347.1149.7047.7851.5449.5153.2381.20 PGR-1.802.806.652.4410.392.7716.0912.4719.27- Weak-to-Strong Generalization (In-distribution) IMAGENET Accuracy54.9066.8368.2665.7267.8867.1149.9867.9068.6068.3574.49 PGR-60.9768.1255.3965.8762.02-23.9966.3369.5468.60- in InD W2S performance. Generally, ensemble-based methods outperform those relying on a single weak model; nevertheless, RAVEN is far more effective than other ensemble-based approaches, achieving55%average improvement in PGR and3.7%average improvement in accuracy over the best alternative ensemble approach Ens. In contrast to other baselines that do not use domain information, Co-sup utilizes domain information by training each weak model on a specific group of domains. Despite that, RAVEN achieves a54%improvement in PGR compared to Co-sup, even without utilizing any domain knowledge. Text classification. Experiments on the text classification tasks (Table 2) show that RAVEN consis- tently outperforms all baselines. In the OOD setting, RAVEN achieves a57%average improvement in PGR and a1.4%average improvement in accuracy compared to the best alternative baseline. In the InD setting, RAVEN yields a27%average improvement in PGR over the best baseline. We further observe that Bayes [19], despite leveraging multiple weak models, underperforms in this context and can even fall below the performance of individual weak models. Note that the OOD setting involves more fine-tuning instances than the InD setting, which contributes to the higher PGR observed for OOD tasks. Table 2: Text classification results. We conduct experiments three times and report the average performance on the OOD and InD settings. We highlight the best score in red and the second-best score in bold. We use three weak models for all the WeakS-to-Strong Generalization methods. The standard deviations are reported in Section G.2. Weak modelWeak-to-Strong GeneralizationWeakS-to-Strong GeneralizationStrong model NaiveConfBootsV-supEns(3)*Bayes(3)*Co-sup(3)RAVEN(3) AMAZON-WILDS (Out-of-distribution) Llama-3.2-1B→ Llama-3.1-8B Accuracy 68.1768.6068.1968.5967.8868.4567.0369.9571.1472.15 PGR–10.880.5010.63-7.366.95-28.7044.7774.56– Qwen2.5-0.5B→ Qwen2.5-7B Accuracy66.3467.6967.1267.7967.2667.6464.4467.6070.1570.78 PGR–30.3517.5832.6120.7429.15-43.0128.3285.88– Qwen2.5-0.5B→ Qwen2.5-14B Accuracy66.3470.9770.8270.0070.2070.5259.0770.9871.3475.66 PGR–49.7048.0039.2041.3044.80-78.0049.8053.60– MEDMCQA→ MEDQA (Out-of-distribution) Qwen2.5-0.5B→ Meditron-7B Accuracy25.7026.0026.3025.8025.7025.7024.7025.3026.5027.60 PGR –17.9030.505.301.60-2.60-52.60-23.2042.60– Qwen2.5-0.5B→ Meditron-70B Accuracy25.7027.5727.5427.3227.2626.0820.9025.2227.7336.37 PGR –17.5017.2015.2014.603.60-45.00-4.5019.00– AMAZON-WILDS (In-distribution) Llama-3.2-1B→ Llama-3.1-8B Accuracy69.7470.2370.0270.0670.0270.3366.2969.9770.4471.33 PGR–30.9517.2620.0017.6837.26-217.8914.3244.00– Qwen2.5-0.5B→ Qwen2.5-7B Accuracy67.7168.6168.1868.3268.1068.5063.2368.5668.9469.42 PGR–52.5927.3735.9723.0746.14-262.7650.0571.95– 7 Preference alignment. In the text gener- ation preference alignment task, RAVEN consistently achieves the best performance across both InD and OOD settings. As shown in Table 3, it surpasses the strongest alternative baselines, achieving average im- provements of 2.7% and 32.8% in the OOD settings, and 1.2% and 25.6% in the InD setting (WR and PGR, respectively). Note again that the OOD setting involves three times more fine-tuning instances than the InD setting, resulting in higher performance. Table 3: Preference alignment results. WeakNaiveEns(3)*Bayes(3)*RAVEN(3)Strong HELPFULNESS→ HARMLESSNESS (Out-of-distribution) WR59.8261.3862.7361.4164.0466.98 PGR-21.7940.6422.2158.94- SUMMARIZATION→ HUMAN-LIKE (Out-of-distribution) WR57.5067.8668.0968.1870.3783.00 PGR-40.6341.5341.8850.47- HARMLESSNESS→ HARMLESSNESS (In-distribution) WR59.8261.4961.4362.8363.6066.98 PGR-23.3222.4942.0452.79- 5.3 Further analysis We conduct additional analyses of RAVEN on the image classification task within the robust W2S scenario. Following the setup in Table 1, we conduct ten experiments and report their average unless stated otherwise. Ablation study. We conduct ablation studies to evaluate the impact of ensembling, easy-sample- guided initialization, and adaptive weighting in- troduced in RAVEN. We incrementally incorpo- rate each component and evaluate performance. As shown in Table 4, all components effectively contribute to RAVEN’s performance, validating our design choices and their individual impor- tance. Table 4: Ablation study. The values represent the averages across all datasets. Detailed per-dataset results are available in Appendix H.6. Ensemble✓ Easy-sample guided init.✓ Adaptive weighting✓ Accuracy46.7849.7650.5151.3751.97 PGR1.8010.3910.6714.2816.09 Training scheduling. RAVEN first trains only theΘ S parameters on easy samples usingL ensemble , and then updates bothΘ S and the ensembling weightsΘ W on the entire dataset usingL adaptation . We further explore how different strategies for using easy samples and for applying static vs. adaptive weighting affect performance. As shown in Table 5, our strategy of easy-sample guided initialization with adaptive weighting achieves the best performance, confirming the effectiveness of our approach. We suggest that applying static weights to easy samples during the early stages discourages the strong model from relying on shortcuts. These shortcuts occur when the strong model learns a weak signal combination that simply mimics its own initial suboptimal predictions, an easy way to minimize the cross-entropy loss without genuine learning. The detailed results for each dataset are provided in Table 13 in Appendix H.1. Table 5: Sample and weight scheduling. All is a naive ensemble model, Easy trains solely on easy samples, and Easy-All begins with easy samples before incorporating all samples with static weighting. Easy-All+AW uses adaptive weighting from the start without initial static weighting. MetricAllEasyEasy-AllEasy-All+AWRAVEN Accuracy49.7650.1050.5150.8651.97 PGR10.3911.1310.6712.8516.09 Increased weak model diversity. To enhance the diversity of weak models, we conduct additional experiments using (i) different weak model architectures and (i) different data sources for training the weak models. For different architectures, we use AlexNet, ResNet18, and SqueezeNet, as weak models to supervise the strong model DINO ViT-B/8. Although these models are trained on the same data source, their architectural differences result in learning diverse features. For different data sources, we follow the setup in Liu and Alahi[45], which constructs distinct data splits based on sub-domains. Specifically, we train seven AlexNet models on different “camera location” domains for IWILDCAM, five AlexNet models on different “hospital” domains for CAMELYON17, and seven AlexNet models on different “time” domains for FMOW. As shown in Table 6, RAVEN achieves an 81% PGR improvement in the setting with different architectures and a 24% PGR improvement in 8 Table 6: Performance comparison across different weak model configurations. Reported values are averaged over all datasets, with detailed results for each dataset provided in Appendix H.2. Weak modelWeak-to-Strong GeneralizationWeakS-to-Strong GeneralizationStrong model NaiveConfBootsV-supEnsBayes Co-supRAVENDINO ViT-B/8 Different Architectures of Weak Models Accuracy42.0442.8041.1844.2141.8743.4044.2342.4546.0384.43 PGR-1.78-1.723.15-0.263.285.33-0.319.65- Different Data Sources for Training Weak Models Accuracy47.4046.3446.9247.3346.4649.4050.1347.4351.1784.63 PGR--2.08-0.500.93-1.755.467.761.0510.49- the setting with different data sources, compared to the best alternative baseline Bayes. These results confirm that RAVEN is effective at weak-to-strong generalization with weak models of both low and substantial diversity. How do the weightsΘ W evolve over iterations? One would expect that the performance of a weak model evaluated on the test set of the strong model (P trg ) correlates with its W2S performance. Consequently, it would be desirable for RAVEN to assign higher weights to weak models with betterP trg performance. To investigate this, we begin by examining the correlation between the performance of weak models onP trg and their W2S performance and indeed observe high correlation, highlighting the importance of selecting the optimal weak model (Figure 4 Left). We next aim to understand whether the weightsΘ W that the strong model assigns to weak models in RAVEN agree well with the actualP trg performance of the weak models. Notably, we find that the strong model predominantly assigns the highest weight to the best-performing weak model forP trg , despite not having access to information about the weak models’ performance as evaluated against GT (Figure 4 Right). This indicates that the strong model can identify high-quality annotations from multiple annotators without any guidance, which is a remarkable feat. Additional graphs with more weak models and classifier initializations can be found in Appendix H.4 and Appendix H.5. x Figure 4: Left: Correlation between weak model accuracyAcc(P trg ;f src w )and its W2S accuracy Acc(P trg ;f pseudo s ). Right: The weights Θ W assigned to weak models across two runs of RAVEN. 135101520 Number of Models 46 48 50 52 54 56 Accuracy (%) Figure 5: Effect of the number of weak models. Number of weak modelsM. We further analyze the performance across varying numbers of weak models. As the number of weak models increases, we observe that the performance of RAVEN im- proves (Figure 5). However, the gains gradually diminish after including a large number of weak models, indicating a saturation effect. 9 Other results. In addition to the results above, further experiments are provided in the Appendix, supporting our choices and assumptions on the proposed framework. Specifically, we present analysis of: RAVEN’s performance on IMAGENET-C (Appendix B), performance across weight configurations (Appendix H.8), comparison between the best weak model and adaptive weighting (Appendix H.7), exploration of strong model’s identification of the best weak model (Appendix H.10), scaling analysis using diverse (weak, strong) model pairs (Appendix H.11), and qualitative preference-alignment results (Appendix H.12). 6 Related work Weak-to-strong (W2S) generalization. W2S generalization framework was first introduced in [10]. In this work, the authors evaluated various strategies beyond naive generalization, includ- ing confidence loss, bootstrapping, and unsupervised fine-tuning, to enhance performance. They demonstrated the feasibility of developing superhuman models, a finding that was further supported by subsequent studies [67,64]. Theoretical insights behind this observation were investigated in [13,38,28,60,23,66]. To enhance W2S generalization performance, [26] extended the concept of confidence loss introduced in [10], developing adaptive confidence loss, which dynamically adjusts the balance between learning from weak model confidence. Multiple weak models have been employed using the AdaBoost technique [1], within the framework of hierarchical mixtures of experts [45], and in estimating Bayesian-based confidence loss [19]. Sang et al.[57]leveraged debates among multiple models to enhance strong models. Within this ensemble framework, we propose a method for identifying reliable weak supervision, which proves especially effective in our novel scenario involving distribution shifts. Scalable oversight. Scalable oversight [3,6,42] aims to enhance human supervision in novel and challenging environments. To this end, models can be used to evaluate other models [29,58,33,32] or to decompose complex problems into simpler subproblems [41,44]. By contrast, W2S generalization explores how to effectively leverage unreliable human supervision itself to train models that can ultimately surpass human performance. Learning under distribution shift. A trained model often performs poorly when it faces query data whose distribution is significantly different from the training data [36,62,21,49]. Learning under distribution shifts aims to make the model robust in this scenario. This challenge has been observed across domains such as healthcare, autonomous driving, and facial recognition, where models fail to generalize across hospitals, lighting conditions, or demographic subgroups due to distribution mismatches [35,53,2,20,9]. A range of methods has been proposed to address this issue, and their effectiveness is actively evaluated through diverse benchmarks [25, 63, 30]. In the context of existing literature, our novel scenario can be viewed as an intersection of weak-to- strong generalization and learning under distribution shift. 7 Concluding remarks Limitations. The weak models used in RAVEN differ only in their random seeds. It is unclear whether this alone can fully capture the diversity of human annotators, particularly in fields where domain expertise is crucial. Another aspect that calls for further investigation is the choice of adaptive weights (Appendix H.8). While we found that model-wise weights outperform the (model, sample)-wise variant, the former is inherently a subset of the latter (where model-wise weights are the same across all samples). Future work could explore ways to address the optimization challenges of (model, sample)-wise weighting, potentially leading to an even stronger RAVEN variant. Conclusion. We extend the concept of W2S generalization by explicitly modeling distribution shifts between the source and fine-tuning datasets. We demonstrate that naive W2S generalization often becomes intractable in such scenarios, and conventional approaches fail to adequately address this issue, falling short of robustness requirements. To address this challenge, we present RAVEN, a novel framework that enables robust W2S generalization. Acknowledgments. We would like to thank Artyom Gadetsky, Fabian Gröger, Maxim Kodryan, Ramon Vinas Torné, Shuo Wen, Siba Smarak Panigrahi, and Yulun Jiang for their valuable discussions 10 regarding our work. We gratefully acknowledge the support of the Swiss National Science Foundation (SNSF) starting grant TMSGI2_226252/1, SNSF grant IC00I0_231922, the Swiss AI Initiative and the CIFAR Multiscale Human Catalyst. Myeongho Jeon was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [RS-2024-00337693]. Suhwan Choi was supported by 1) the Starting growth Technological R&D Program (RS-2024- 00506994) funded by the Ministry of SMEs and Startups (MSS, Korea), 2) Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024 (Project Name: Development of K-POP artist-centered video editing solution: customized multimodal AI model and generative asset, Project Number: RS-2024-00399433, Contribution Rate: 50%), 3) Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City, and 4) Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.RS-2021-I211343, Artificial Intelligence Graduate School Program (Seoul National University)]. Jan Sobotka was supported by the Bakala Foundation during his studies at EPFL. References [1]Aakriti Agrawal, Mucong Ding, Zora Che, Chenghao Deng, Anirudh Satheesh, John Langford, and Furong Huang. EnsemW2S: Can an Ensemble of LLMs be Leveraged to Obtain a Stronger LLM? In Advances in Neural Information Processing Systems, Safe Generative AI Workshop, 2024. [2]Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a Pose: Neural Networks are Easily Fooled by Strange Poses of Familiar Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [3]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. arXiv preprint arXiv:1606.06565, 2016. [4]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. Computing Research Repository, 2022. [5]Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iWildCam 2021 Compe- tition Dataset. arXiv preprint arXiv:2105.03494, 2021. [6]Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil ̇ e Lukoši ̄ ut ̇ e, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring Progress on Scalable Oversight for Large Language Models. arXiv preprint arXiv:2211.03540, 2022. [7]Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39, 1952. [8] Adrian Brady, Risteárd Ó Laoide, Peter McCarthy, and Ronan McDermott. Discrepancy and Error in Radiology: Concepts, Causes and Consequences. The Ulster Medical Journal, 81(1), 2012. [9]Joy Buolamwini and Timnit Gebru. Gender Shades: Intersectional Accuracy Disparities in Com- mercial Gender Classification. In Conference on Fairness, Accountability and Transparency, 2018. [10]Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. In International Conference on Machine Learning, 2024. [11] Ethem Ya ̆ gız Çalık and Talha Rüzgar Akku ̧s. Enhancing Human-Like Responses in Large Language Models. arXiv preprint arXiv:2501.05032, 2025. 11 [12]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In International Conference on Computer Vision, 2021. [13]Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the Gain in Weak-to- Strong Generalization. In Advances in Neural Information Processing Systems, 2024. [14]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning, 2020. [15]Xinlei Chen and Abhinav Gupta. Webly Supervised Learning of Convolutional Networks. In International Conference on Computer Vision, 2015. [16]Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. arXiv preprint arXiv:2311.16079, 2023. [17]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, 2017. [18]Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional Map of the World. In Conference on Computer Vision and Pattern Recognition, 2018. [19]Ziyun Cui, Ziyang Zhang, Wen Wu, Guangzhi Sun, and Chao Zhang. Bayesian WeakS- to-Strong from Text Classification to Generation. In International Conference on Learning Representations, 2025. [20]Dengxin Dai and Luc Van Gool. Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime. In International Conference on Intelligent Transportation Systems, 2018. [21]Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal Misgeneralization in Deep Reinforcement Learning. In International Conference on Machine Learning, 2022. [22]Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling to Online RLHF. Transactions on Machine Learning Research, 2024. [23]Yijun Dong, Yicheng Li, Yunai Li, Jason D Lee, and Qi Lei. Discrepancies are Virtue: Weak-to- Strong Generalization through Lens of Intrinsic Dimension. In International Conference on Machine Learning, 2025. [24]Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024. [25] Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. In Interna- tional Conference on Learning Representations, 2021. [26]Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models. arXiv preprint arXiv:2402.03749, 2024. [27]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022. [28] Muhammed Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws. In International Conference on Learning Representations, 2025. [29] Geoffrey Irving, Paul Christiano, and Dario Amodei. AI Safety via Debate. arXiv preprint arXiv:1805.00899, 2018. [30]Myeongho Jeon, Suhwan Choi, Hyoje Lee, and Teresa Yeo. An Analysis of Model Robustness across Concurrent Distribution Shifts. Transactions on Machine Learning Research, 2025. 12 [31]Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease Does this Patient Have? a Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences, 11(14), 2021. [32]Zachary Kenton, Noah Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, et al. On Scalable Oversight with Weak LLMs Judging Strong LLMs. Advances in Neural Information Processing Systems, 2024. [33] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers. In International Conference on Machine Learning, 2024. [34]Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015. [35]Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A Benchmark of in-the-Wild Distribution Shifts. In International Conference on Machine Learning, 2021. [36] David Krueger, Tegan Maharaj, and Jan Leike. Hidden Incentives for Auto-Induced Distribu- tional Shift. arXiv preprint arXiv:2009.09153, 2020. [37]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, 2017. [38]Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical Analysis of Weak-to- Strong Generalization. In Advances in Neural Information Processing Systems, 2024. [39] Byungjae Lee and Kyunghyun Paeng. A Robust and Effective Approach Towards Accurate Metastasis Detection and pN-stage Classification in Breast Cancer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018. [40]Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arXiv preprint arXiv:1511.06314, 2015. [41]Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable Agent Alignment via Reward Modeling: a Research Direction. arXiv preprint arXiv:1811.07871, 2018. [42]Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy AI: From Principles to Practices. ACM Computing Surveys, 55(9), 2023. [43]Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment. In Conference on Empirical Methods in Natural Language Processing, 2024. [44]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. In International Conference on Learning Representations, 2023. [45]Yuejiang Liu and Alexandre Alahi. Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts. arXiv preprint arXiv:2402.15505, 2024. [46]Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations, 2017. [47]Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019. [48]Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is Being Transferred in Transfer Learning? In Advances in Neural Information Processing Systems, 2020. [49] Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem from a Deep Learning Perspective. In International Conference on Learning Representations, 2024. 13 [50]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, 2022. [51]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A Large- Scale Multi-Subject Multi-Choice Dataset for Medical Domain Question Answering. In Confer- ence on Health, Inference, and Learning, 2022. [52]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, 2019. [53] Christian S Perone, Pedro Ballester, Rodrigo C Barros, and Julien Cohen-Adad. Unsupervised Domain Adaptation for Medical Imaging Segmentation with Self-Ensembling. NeuroImage, 194, 2019. [54] Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In Advances in Neural Information Processing Systems, 2019. [55] Alec Radford. Improving Language Understanding by Generative Pre-Training, 2018. [56]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, 2024. [57]Jitao Sang, Yuhang Wang, Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye, Shuyu Wei, and Jinlin Xiao. Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning. arXiv preprint arXiv:2402.00667, 2024. [58]William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-Critiquing Models for Assisting Human Evaluators. Computing Research Repository, 2022. [59]Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential Deep Learning to Quantify Classification Uncertainty. In Advances in Neural Information Processing Systems, 2018. [60]Changho Shin, John Cooper, and Frederic Sala. Weak-to-Strong Generalization Through the Data-Centric Lens. In International Conference on Learning Representations, 2025. [61]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to Summarize with Human Feedback. In Advances in Neural Information Processing Systems, 2020. [62] Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhat- tacharya, and Jeff Bilmes. An Effective Baseline for Robustness to Distributional Shift. In International Conference on Machine Learning and Applications, 2021. [63]Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A Fine-Grained Analysis on Distribution Shift. In International Conference on Learning Representations, 2022. [64]David Xing Wu and Anant Sahai. Provable Weak-to-Strong Generalization via Benign Overfit- ting. In International Conference on Learning Representations, 2025. [65]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115, 2024. [66]Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, and Yong Liu. Understanding the Capabilities and Limitations of Weak-to-Strong Generalization. Computing Research Repository, 2025. [67]Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin Edelman, Milind Tambe, Sham Kakade, and Eran Malach. Transcendence: Generative Models Can Outperform the Experts That Train Them. In Advances in Neural Information Processing Systems, 2024. 14 [68]Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. In Annual Meeting of the Association for Computational Linguistics, 2024. 15 A Definition of distribution shift To define distribution shift between some two distributionsP src andP tuning , we use the generalization gap:∆R(f ) = R tuning (f )− R src (f ), whereR(f ) = E (x,y)∼P [L(f (x),y)]is the expected risk,f denotes the classifier, andLis the loss function (e.g., cross-entropy). A large∆R(f )indicates that the model struggles to generalize from distributionP src to distributionP tuning , showing the presence of a significant distribution shift. Another way to quantify distribution shift is through a divergence metricD(P src ,P tuning ), whereD can be any suitable measure such as Maximum Mean Discrepancy (following [54]), KL-divergence, or optimal transport. WhenD(P src ,P tuning )≤ εandε∈ R + is small enough, we regard the distributions as approximately aligned, i.e.,P src ≈ P tuning . Conversely, we say thatP src ̸≈ P tuning and consider the shift to be significant—potentially misleading the model’s performance at inference—when D(P src ,P tuning ) > ε. B Motivating experiment Without fine-tuning. For image classification tasks, Burns et al.[10]used an IMAGENET-pretrained AlexNet as the weak model without additional training. To replicate their setup, we compute the mean and standard deviation of the pre-trained AlexNet’s performance over five random splits of the IMAGENET validation set intoP tuning andP trg , and over five random splits of the IMAGENET-C dataset intoP tuning andP trg . The results of this experiment are provided in Table 7 and, combined with the setup described below, shown in Figure 2. With fine-tuning. A more common practice, however, is to evaluate models trained with different random seeds while keeping the dataset fixed. We also adopt this approach, examining how PGR changes under distribution shifts when using five different weak models. Specifically, using different random seeds, we reinitialize the classification head of the IMAGENET-pretrained AlexNet and train it for 20 epochs. This approach is particularly relevant in the weakS-to-strong generalization framework (ensemble-based methods), where multiple distinct weak models are required. Using these fine-tuned models, we compare RAVEN to naive W2S generalization on the IMAGENET-C dataset. As shown in Table 8, despite severe synthetic domain shifts, the performance of RAVEN exceeds that of the weak model (human in the future) in most cases, outperforming the naive W2S generalization method. For this experiment, we use the same AlexNet weak models as reported in Table 1, and designate DINO ResNet50 as the strong model. Finally, the results for the naive W2S generalization method shown in Figure 2 are the average over these two setups, effectively balancing clarity of presentation, adherence to previous work [10], and common practices used in the field. 16 Table 7: Result of Motivating Experiment: without fine-tuning of weak models. For the weak and strong models, we report the accuracy, while for the W2S fine-tuning (AlexNet→DINO ResNet50), we present the PGR. The values represent the mean and standard deviation over five runs. Corruption TypeAlexNetDINO ResNet50AlexNet→ DINO ResNet50 (PGR) No-shift56.26± 0.5664.00± 0.3974.01± 5.67 Saturate45.72± 0.4359.68± 0.4344.43± 1.83 Brightness42.70± 0.3758.62± 0.6939.59± 1.02 Elastic40.18± 0.2852.64± 0.3619.56± 2.34 JPEG37.66± 0.2553.00± 0.799.63± 2.38 Gaussian Blur13.64± 0.1741.24± 0.565.87± 1.18 Pixelate29.06± 0.4249.44± 0.475.50± 0.66 Defocus Blur11.54± 0.1539.32± 0.715.48± 0.80 Spatter23.08± 0.3048.30± 0.303.00± 0.76 Contrast9.18± 0.2552.04± 0.412.66± 0.63 Fog13.86± 0.3647.34± 0.341.91± 0.28 Motion Blur16.00± 0.3938.76± 0.191.57± 1.92 Snow12.52± 0.2539.54± 0.441.56± 1.37 Zoom Blur19.22± 0.6539.94± 0.451.04± 1.79 Speckle Noise8.58± 0.0841.94± 0.27−0.24± 0.54 Gaussian Noise5.06± 0.1740.06± 0.37−1.03± 0.67 Shot Noise5.22± 0.1639.64± 0.38−1.45± 0.46 Frost10.68± 0.1336.90± 0.37−1.69± 1.11 Impulse Noise4.18± 0.1937.12± 0.27−1.76± 0.58 Glass Blur10.68± 0.1330.92± 0.41−9.00± 0.80 17 Table 8: Performance comparison on IMAGENET-C for the Motivating Experiment: with fine-tuning of weak models. We conduct five experiments and report the results as the mean and standard deviation. Negative PGRs are highlighted in red. CorruptionMetricAlexNetNaiveRAVENDino ResNet50 Saturate Accuracy44.34± 0.4447.66± 0.5149.23± 0.4060.06± 0.39 PGR-20.94± 1.0931.22± 1.42- Brightness Accuracy40.64± 0.3643.72± 0.5045.67± 0.0658.90± 0.50 PGR-16.93± 0.7427.63± 0.62- Elastic Accuracy38.02± 0.4937.82± 0.3838.83± 0.0652.90± 0.70 PGR-−1.34± 0.785.07± 2.07- JPEG Accuracy35.74± 0.4835.02± 0.3536.27± 0.0653.16± 0.79 PGR-−4.42± 0.362.94± 0.96- Gaussian blur Accuracy10.86± 0.3210.92± 0.4211.57± 0.3541.84± 0.35 PGR-0.22± 0.572.40± 0.88- Pixelate Accuracy26.46± 0.4326.12± 0.7027.47± 0.4749.72± 0.46 PGR-−1.48± 0.554.54± 1.38- Defocus blur Accuracy9.22± 0.279.32± 0.349.67± 0.4940.16± 0.50 PGR-0.57± 0.341.62± 1.30- Spatter Accuracy22.74± 0.4521.90± 0.4722.97± 0.3848.90± 0.46 PGR-−3.18± 0.260.72± 1.14- Contrast Accuracy8.40± 0.198.46± 0.218.97± 0.0652.40± 0.47 PGR-0.20± 0.121.34± 0.24- Fog Accuracy13.92± 0.4814.04± 0.5514.17± 0.0647.64± 0.62 PGR-−2.75± 0.340.33± 0.06- Motion blur Accuracy13.04± 0.2112.06± 0.2113.10± 0.1039.18± 0.48 PGR-−3.78± 0.300.34± 0.60- Snow Accuracy12.08± 0.2211.58± 0.1612.27± 0.2940.34± 0.38 PGR-−1.93± 0.260.55± 0.95- Zoom blur Accuracy16.02± 0.3514.66± 0.2816.30± 0.3940.64± 0.46 PGR-−5.36± 0.421.30± 1.56- Speckle noise Accuracy8.12± 0.367.70± 0.378.20± 0.1742.88± 0.36 PGR-−1.15± 0.250.10± 0.25- Gaussian noise Accuracy4.68± 0.263.90± 0.234.90± 0.1040.10± 0.00 PGR-−2.23± 0.150.38± 0.16- Shot noise Accuracy4.76± 0.264.12± 0.165.03± 0.0640.10± 0.00 PGR-−1.95± 0.140.54± 0.14- Frost Accuracy10.50± 0.419.28± 0.669.90± 0.3536.88± 0.42 PGR-−4.27± 0.42 −1.36± 0.21- Impulse noise Accuracy4.00± 0.203.40± 0.074.23± 0.0638.28± 0.38 PGR-−1.95± 0.100.49± 0.34- Glass blur Accuracy9.56± 0.367.82± 0.338.27± 0.2531.80± 0.35 PGR-−7.78± 0.17 −5.79± 0.61- 18 C Theoretical analysis of how RAVEN operates To understand why RAVEN often identifies the most suitable weak model for the target data distribu- tion, we begin with the following assumption. Assumption 1. The ensemble of weak models tends to make predictions similar to those of the best-performing weak model: argmax k∈1,...,C f P (x)[k]≈ arg max k f ∗ (x)[k],(5) where f P (x) := M X i=1 f src w i (x), andf ∗ (x)denotes the best-performing weak model. While the ensemble prediction is often close to that of the best model, we assume that f ∗ is more accurate and confident. Let f s denote the strong model, and define the adaptation loss as: L adaptation (Θ S , Θ W ) := L CE f s (x), M X i=1 θ (i) f src w i (x) ! s.t. M X i=1 θ (i) = 1, θ (i) ∈ Θ W .(6) This loss is optimized by alternating the following two steps: Step 1. Optimize Θ S with Θ W fixed. When the weights are uniform (i.e.,θ (i) = 1 M ), the strong model learns to mimic the average behavior of weak models: f s (x)≈ 1 M M X i=1 f src w i (x).(7) Step 2. Optimize Θ W with Θ S fixed. We can rewrite the objective as: L CE (Θ W ) = M X i=1 θ (i) C i ,(8) where C i :=−E x∼D " C X k=1 f src w i (x)[k] logf s (x)[k] # .(9) Assumption 2. Each weak model is highly confident in predicting a single class: f src w i (x) = (ε,..., 1− ε,...,ε), ε≪ 1,(10) with the largest value 1−ε at index ˆ k i := arg max k f src w i (x)[k]. Then, C X k=1 f src w i (x)[k] logf s (x)[k]≈ logf s (x)[ ˆ k i ],(11) and the objective simplifies to: C i ≈−E x∼D h logf s (x)[ ˆ k i ] i .(12) By Assumption 1 and Eq. (3), we infer that the best weak modelf ∗ (x)most closely matches the predictions off s (x)and therefore receives the highest weight—consistent with empirical findings. Notably, this conclusion still holds even when only the best weak model satisfies Assumption 2, since the best weak model yields the smallestC i . As a result, RAVEN naturally prioritizes the weak models that generalize well to the target distribution. Through this iterative optimization, the strong model f s becomes increasingly aligned with the best weak model and further improves by leveraging this implicit supervision. 19 D Experimental details D.1 Adaptive weighting We initialize the adaptive weightsθ (i) ∈ Θ W uniformly, i.e.,θ (i) = 1 M fori ∈ 1,...M. After the easy-sample guided initialization phase, during whichθ (i) are fixed and only the strong model parametersΘ S are trained, we start optimizing the adaptive weights as well. More specifically, for each minibatch of the fine-tuning data, we first generate and combine soft labels from theMweak models using the latestΘ W (Eq. 3). Then, using these pseudo labels, we calculate gradients of L adaptation (Θ S , Θ W )with respect toΘ S andΘ W , and perform a single update step of both using the Adam optimizer [34] and SGD, respectively. After each update ofΘ W , we clip and normalize the adaptive weights to sum up to 1 as: θ (i) new = max(θ (i) ,ε) P M j=1 max(θ (j) ,ε) ,(13) where ε = 10 −6 . θ (i) new are then set as θ (i) for the next minibatch of the fine-tuning process. For text classification, we derive the weak supervisiony soft (x)(i.e., pseudo-label) by computing a weighted sum of weak model logitsy aw using adaptive weights, followed by a softmax operation for stabilization: y (c) soft (x) = exp y (c) aw (x) P K k=1 exp y (k) aw (x) ,(14) where y aw (x) = M X i=1 θ (i) f src w i (x).(15) Above,f src w i (x)represents the logits of thei-th weak model trained on source data,Kis the number of label classes, andcdenotes a specific class label. Consequently,y soft (x)serves as the weak supervision. All hyperparameters, including those of the baselines, are found using a grid search based on the validation loss. The hyperparameter search includes the number of fine-tuning epochs, the learning rate, and the method-specific hyperparameters (e.g., the learning rate of the adaptive weights and the length of the easy-sample guided initialization phase). Building on the publicly available code from Burns et al.[10], we use the cosine annealing schedule [46] while optimizing the strong model parametersΘ S , but keep the learning rate of the adaptive weightsΘ W fixed. We perform a hyperparameter search in the range shown in Table 9. Table 9: Hyperparameter configurations for RAVEN. TaskLearning rate for Θ S Learning rate for Θ W Easy-sample guided periodEpoch Image Classification 1e-6, 1e-5, 1e-4, 1e-3 1e-6, 1e-5, 1e-4, 1e-3 10%, 20%, 50% 30, 50, 80, 100 Text Classification 1e-6, 1e-5, 1e-4 1e-5, 1e-4, 1e-320%, 50%10, 20 Preference Alignment 1e-6, 5e-6, 1e-5, 5e-5 1e-5, 1e-4, 1e-320%, 50%2 D.2 Weak models for ensemble-based methods The weak models used in ensemble-based methods are trained using different seeds and hyperparam- eters found through a search over the number of epochs, learning rate, and weight decay. For image classification experiments, we train weak models with the Adam optimizer [34], the cross-entropy loss function, early stopping, and multiplicative (factor 0.96) and the cosine learning rate schedule for IWILDCAM and FMOW, respectively. For CAMELYON17 and IMAGENET, we employ SGD with a momentum of 0.9 and a fixed learning rate. Our selection of optimizers and hyperparameters is based on the original work introducing the WILDS benchmark datasets [35], with adjustments that we found to improve the performance of our specific model architectures. 20 With this setup, all weak models achieve around the same InD validation accuracy on their respective datasets (±2%). Since the W2S fine-tuning step of the IMAGENET experiments employs its validation dataset, we use 10% of the IMAGENET training set as the validation set for weak model training. Importantly, in the case of IMAGENET, we start from a pre-trained AlexNet model loaded from the PyTorch library [52] with a randomly reinitialized classification head to achieve diversity among the weak models (same setup as for Table 8). For text classification, we train weak models using the Adam optimizer [34], the cross-entropy loss function, and cosine learning rate schedule. We initialize with pre-trained weak language models (Qwen2.5-0.5B 7 /Llama-3.2-1B 8 ) from Hugging Face. In the AMAZON-WILDS OOD setting, because the OOD validation and test sets are OOD with respect to each other, we only use the OOD validation set, splitting it into fine-tuning (70%), test (20%), and validation (10%) subsets. For the InD setting, we use the original AMAZON-WILDS InD validation and test sets. For reward modeling, we train weak models using AdamW [47] optimizer and cosine learning rate schedule. The source data is split into training (90%) and validation (10%) subsets. For further details on reward modeling, please refer to Section E.1. We report the final hyperparameter values for training the weak models in Table 10. Table 10: Hyperparameter configurations for training the weak models. Learning rateWeight decayEpochs Image Classification IWILDCAM3e-58e-260 CAMELYON171e-31e-230 FMOW8e-45e-3150 IMAGENET3e-48e-520 Text Classification AMAZON-WILDS1e-5010 Preference Alignment H-RLHF1e-61e-35 To balance the statistical power of repeated W2S experiments and computational costs, we reuse the weak models in the ensembles. More specifically, for each experimentk ∈1,..., 10, we use the pretrained weak modelsk,k + 1,k + 2for the Ens(3), Bayes(3) andRAVEN(3)methods reported in Table 1 and Table 2. The same idea is applied forRAVEN(>3)and our analysis of the number of weak models in Figure 5. D.3 Baselines for image and text classification We use the original implementation of the auxiliary confidence loss from Burns et al.[10]and the Vision superalignment (adaptive confidence loss) from Guo et al.[26]. We implement the bootstrapping method [10] with DINO ResNet50 as the intermediate (medium-strong) model for image classification and Qwen2.5-3B/Llama-3.2-3B for text classification. This intermediate model is first fine-tuned using the weak supervision, and then its pseudo labels for the fine-tuning dataset (P tuning ) are used to fine-tune the final strong model. In image classification, AlexNet serves as the weak model, and DINO ViT8/B as the strong model. In text classification, weak supervision is performed within the same model family: Llama-3.2-1B supervises Llama-3.1-8B, and Qwen2.5- 0.5B supervises Qwen2.5-7B. Fine-tuning in both steps is done using the same setup as in all the other reported W2S experiments. For the Bayesian weakS-to-strong baseline [19], we follow the authors’ formulation while implementing the method using the code for evidential deep learning from Sensoy et al.[59]. See Section D.5 for details on how we adapt Co-supervised learning [45] for our setup. 7 https://huggingface.co/Qwen/Qwen2.5-0.5B 8 https://huggingface.co/meta-llama/Llama-3.2-1B 21 D.4 Baselines for preference alignment Cui et al.[19]implemented DPO for Bayesian weakS-to-strong by using the log probability P of weak models as a proxy for preference. More specifically, their preference rewardr bayes for a given text y is computed as the weighted sum of the log probabilities from M trained weak models: r bayes (y) = M X i=1 λ i P(y w (y)|θ i ).(16) Strong models are trained on target preference data using the DPO objective withy c andy r as defined in Eq. 22. The chosen texty c is the text with the highest preference reward, while the rejected texty r is the one with the lowest preference reward: y c = arg max y j ,j=1,2,...,N r bayes (y), y r =arg min y j ,j=1,2,...,N r bayes (y).(17) In H-RLHF [4],N = 2, where the two available text choices are the ground-truth chosen and rejected texts. The original work [19] assigned different weightsλ i to different weak model families. However, since we use the same type of weak models, we set λ i = 1 M . We note that in this Bayesian weakS-to-strong approach [19], the weak models were not trained on preference data. In contrast, we train our weak models on source preference data using DPO, allowing them to better capture preference-based signals. Additionally, the weak models are trained within the same hyperparameter space as the strong models, as detailed in Section E.3. D.5 Co-supervised learning For DOMAINNET [48], Liu and Alahi[45]designed a two-level structure of specialized weak supervisors using sub-domain labels from DomainNet. At the first tier, the problem domain is divided into two groups of sub-domains: ‘clip’, ‘quick’, ‘sketch’ and ‘info’, ‘paint’, ‘real’. At the second level, each supervisor is dedicated to a specific sub-domain. Building on the official implementation, we adapt this approach of Co-supervised learning to our OOD scenario as follows. First, we use the domain information from IWILDCAM, CAMELYON17, FMOW, and AMAZON-WILDS to create multiple distinct weak models that are trained on subsets of domains that form a hierarchy. In the fine-tuning process, the strong model is first supervised by the weak model trained on all domains. Then, at each subsequent supervision level, it is further fine-tuned with weak supervision coming from the weak models, where each was trained on half of the domains of the supervision level before, effectively learning from more specialized weak models. For all these supervision levels with multiple weak models, we select the weak supervision label for each data point that agrees the most with the strong model’s predictions from the previous supervision level. D.6 Calculating PGR for ensemble-based methods For an ensemble-based method with M weak models, we compute the PGR in the following way: PGR := Acc(P trg ;f pseudo s )− 1 M P M i=1 Acc(P trg ;f src w i ) 1 M P M i=1 Acc(P trg ;f gt s i )− Acc(P trg ;f src w i ) ,(18) wheref gt s i ,i∈1,...,Mrefers to the strong model fine-tuned onP tuning using the GT labels and the same random seed as thei-th weak modelf src w i used in its training. We report this PGR along with the average accuracy of weak models and strong models trained with GT labels in Table 1 and Table 2. E Configurations for preference alignment experiments We perform preference alignment in two phases: reward modeling for the weak models and preference optimization for the strong model. First, we train the weak models using HELPFULNESS samples from H-RLHF [4] to predict preference rewards. The trained weak models then generate preference signals for target HARMLESSNESS samples, which are used to align the strong model via DPO [56]. 22 E.1 Reward modeling The preference signal is commonly modeled using the reward-based Bradley-Terry model [7,50,4]: p y 1 ≻ y 2 | x = exp r w (x,y 1 ) exp r w (x,y 1 ) + exp r w (x,y 2 ) = σ r w (x,y 1 )− r w (x,y 2 ) ,(19) wherexrepresents an input prompt,y 1 andy 2 are responses tox,r w is the weak model as the preference reward predictor, andσdenotes the sigmoid function. For a given input promptxwith ground-truth chosen responsey c and rejected responsey r , we train weak models by minimizing the following loss function: L RM (x,y c ,y r ) =− logσ r w (x,y c )− r w (x,y r ) .(20) After the weak models are trained for reward modeling, we use their predicted preference rewards for a pair of responsesy 1 andy 2 to determine the chosen and rejected responses for aligning the strong model on new data: y c = arg max y∈y 1 ,y 2 r w (x,y), y r = arg min y∈y 1 ,y 2 r w (x,y).(21) Our implementation builds upon the reward modeling framework implemented by Dong et al. [22]. E.2 DPO-R: Direct Preference Optimization for RAVEN Unlike the cross-entropy loss that is used for classification tasks, adaptive weights cannot be updated with direct preference optimization (DPO) loss since they are not involved in the loss calculation. To address this, we modify the DPO loss, leading to our DPO-R formulation. The original objective function of DPO is defined as follows: L DPO (π θ ;π ref ) :=−E (x,y c ,y r )∼D logσ β log π θ (y c | x) π ref (y c | x) − β log π θ (y r | x) π ref (y r | x) ,(22) wherexrepresents a prompt,y c is the chosen response andy r is the rejected response.π θ denotes the model being trained with DPO, whileπ ref refers to the modelπ θ before undergoing DPO training. Direct preference optimization with adaptive weighting. In RAVEN, adaptive weighting with multiple weak models (i.e., ensemble) can be formulated as follows: r ens (x,y) = M X i=1 θ (i) r w i (x,y),(23) wherer w i is thei-th reward model, implemented as a weak model in our setting. WhileL DPO relies on a single parameterβ, we introduce two parameters,β c andβ r , which represent the weights for the chosen and rejected log-likelihoods, respectively. By incorporating the Bradley-Terry model [7] into the DPO loss, we derive a new loss tailored for RAVEN, referred to as the DPO-R loss, defined as: L DPO-R (π θ ;π ref ) :=−E (x,y c ,y r )∼D logσ β c log π θ (y c | x) π ref (y c | x) − β r log π θ (y r | x) π ref (y r | x) ,(24) β c = p y c ≻ y r | x , β r = p y r ≻ y c | x ,(25) p y 1 ≻ y 2 | x = exp r ens (x,y 1 ) exp r ens (x,y 1 ) + exp r ens (x,y 2 ) .(26) E.3 Training details for DPO Strong models are trained using LoRA [27] with a rank of 8 and a scaling parameterα = 16. We use the AdamW [47] optimizer with cosine learning rate scheduling. The LoRA target modules are v_proj,gate_proj,down_proj,o_proj,up_proj,q_proj, andk_proj. Forβ, we perform a grid search over the values 0.1, 0.2, 0.5. Our implementation for DPO experiments is based on Zheng et al. [68]. Hyperparameter search spaces are outlined in Table 9. 23 E.4 Evaluation metric for preference alignment: GPT-4o Win Rate For evaluation in preference alignment experiments, we use theGPT-4owin rate metric, following the prompts utilized by Rafailov et al.[56]. All alignment experiments are conducted withGPT-4o 9 , with the response order randomized for each evaluation. The prompt used is shown below. For the following query to a chatbot, which response is more <"harmless" or "human-like">? Query: <the user query> Response A: <either the test method or baseline> Response B: <the other response> FIRST provide a one-sentence comparison of the two responses and explain which you feel is more <"harmless" or "human-like">. SECOND, on a new line, state only "A" or "B" to indicate which response is more <"harmless" or "human-like">. Your response should use the format: Comparison: <one-sentence comparison and explanation> More <"harmless" or "human-like">: <"A" or "B"> We compute WR using selected preference samples: H-RLHF and the SFT model output for OPENAI SUMMARIZE FROM FEEDBACK and HUMAN-LIKE DPO. WR is calculated by comparing the outputs of DPO-trained models with pre-DPO models. F Computational resources F.1 Image classification Our image classification experiments used a cluster of 8 NVIDIA GeForce RTX 3090 GPUs, but each individual run required only a single GPU and less than 22 GB of VRAM. The most computationally demanding part was training the weak models, which each required at most one day on the single GPU. The subsequent W2S experiment, where we used the pre-trained weak models and pre-collected strong model embeddings, took at most one hour. Hyperparameter search for image classification, as reported in Table 9, took between one and two days. F.2 Text classification and direct preference alignment Our text-classification and DPO experiments used a cluster of 8 NVIDIA H100 GPUs, and each individual run employed the 8 GPUs in parallel. A single text-classification training run completed in roughly one hour. A single DPO training run required about five hours per run on the same eight-GPU setup. Hyperparameter search for each baseline, as reported in Table 9, took one–two days for text classification and about five days for DPO. G Experimental results G.1 Image classification We present the image classification results in Table 1, along with standard deviations calculated based on the setup described in Section D.2. 9 https://openai.com/index/hello-gpt-4o 24 Table 11: Image classification results. We report the average performance across 10 experiments, with PGR calculated as described in Section D.6. Weak-to-Strong Generalization refers to using a single weak model, whereas WeakS-to-Strong Generalization denotes ensemble-based methods. We highlight the best score in red and the second-best score in bold. Model(3) refers to the use of three weak models. For a fair comparison with Co-sup, which utilizes 7 weak models for IWILDCAM, FMOW and IMAGENET, and 5 for CAMELYON17, we report the RAVEN(>3) performance achieved using the same number of weak models as Co-sup. * indicates that the implementation code was created by us. Weak modelWeak-to-Strong GeneralizationWeakS-to-Strong GeneralizationStrong model AlexNetNaiveConfBootsV-supEns(3)*Bayes(3)*RAVEN(3)Co-sup(>3)RAVEN(>3)DINO ViT-B/8 Robust Weak-to-Strong Generalization (Out-of-distribution) IWILDCAM Accuracy 43.8346.8247.9748.7647.9249.3849.9652.7949.8355.4694.76 ±6.63±1.56 ±2.42 ±2.21 ±6.96 ±3.04 ±3.48±3.92 ±2.45±2.52±0.21 PGR-4.216.528.086.409.7310.9016.2510.0322.14- CAMELYON17 Accuracy 66.9067.8368.9070.8467.8572.1868.7473.6770.0073.4597.93 ±5.16±2.41 ±2.67 ±2.51 ±2.56 ±3.29 ±1.98±2.81 ±3.18±1.77±0.15 PGR-2.475.9012.242.5117.135.1321.4614.1321.48- FMOW Accuracy 26.1825.6824.7525.9725.5727.5425.8929.4628.7031.8559.72 ±2.91±1.42 ±1.52 ±1.44 ±1.48 ±1.56 ±1.55±3.07 ±5.28±2.52±1.04 PGR--1.26-4.02-0.38-1.604.31-0.6710.0613.2414.06- Avg. Accuracy45.6446.7847.2148.5347.1149.7047.7851.5449.5153.2381.20 PGR-1.802.806.652.4410.392.7716.0912.4719.27- Weak-to-Strong Generalization (In-distribution) IMAGENET Accuracy 54.9066.8368.2665.7267.8867.1149.9867.9068.6068.3574.49 ±0.80±0.10 ±0.12 ±0.08 ±0.04 ±0.20 ±0.06±0.11 ±0.24±0.04±0.19 PGR-60.9768.1255.3965.8762.02-23.9966.3369.5468.60- G.2 Text classification We report the text classification results in Table 12, accompanied by standard deviations computed according to the setup outlined in Section D.2. Table 12: Text classification results. We conduct experiments three times and report their average. We highlight the best score in red and the second-best score in bold. We use three weak models for all the WeakS-to-Strong Generalization methods. Weak modelWeak-to-Strong GeneralizationWeakS-to-Strong GeneralizationStrong model NaiveConfBootsV-supEns(3)*Bayes(3)*Co-sup(3)RAVEN(3) Robust Weak-to-Strong Generalization (Out-of-distribution) Llama-3.2-1B→ Llama-3.1-8B Accuracy 68.1768.6068.1968.5967.8868.4567.0369.9571.1472.15 ± 0.02±0.10 ±0.18 ±0.06 ±0.15 ±0.24 ±0.69 ±0.14±0.11±0.02 PGR-10.880.5010.63-7.366.95-28.7044.7774.56- Qwen2.5-0.5B→ Qwen2.5-7B Accuracy 66.3467.6967.1267.7967.2667.6464.4467.6070.1570.78 ±0.06±0.15 ±0.21 ±0.04 ±0.06 ±0.06 ±0.12 ±0.13±0.09±0.03 PGR-30.3517.5832.6120.7429.15-43.0128.3285.88- Weak-to-Strong Generalization (In-distribution) Llama-3.2-1B→ Llama-3.1-8B Accuracy 69.7470.2370.0270.0670.0270.3366.2969.9770.4471.33 ± 0.03± 0.04 ± 0.06 ± 0.09 ± 0.05 ± 0.09 ± 0.07 ± 0.18± 0.06± 0.10 PGR-30.9517.2620.0017.6837.26-217.8914.3244.00- Qwen2.5-0.5B→ Qwen2.5-7B Accuracy 67.7168.6168.1868.3268.1068.5063.2368.5668.9469.42 ±0.01±0.04 ±0.18 ±0.14 ±0.16 ±0.11 ±0.45 ±0.19±0.04±0.02 PGR-52.5927.3735.9723.0746.14-262.7650.0571.95- H Further analysis H.1 Training scheduling Table 5 presents the results aggregated over all datasets. The corresponding detailed results for each dataset are provided in Table 13. H.2 Increased weak model diversity Table 6 presents the results aggregated over all datasets. The corresponding detailed results for each dataset are provided in Table 14. 25 Table 13: Sample and weight scheduling. All is a naive ensemble model, Easy trains solely on easy samples, and Easy-All begins with easy samples before incorporating all samples with static weighting. Easy-All+AW uses adaptive weighting from the start without initial static weighting. DatasetMetricAllEasyEasy-AllEasy-All+AWRAVEN IWILDCAM Accuracy49.5650.2749.7550.3052.79 PGR9.7311.1210.1111.1716.25 CAMELYON17 Accuracy72.1871.6674.0575.0673.67 PGR17.1315.5517.2424.0421.95 FMOW Accuracy27.5428.3827.7227.2229.46 PGR4.316.734.673.3410.06 Avg. Accuracy49.7650.1050.5150.8651.97 PGR10.3911.1310.6712.8516.09 Table 14: Performance comparison across datasets and configurations. Each result represents the average over 10 runs. We highlight the best score in red and the second-best in bold. Weak modelWeak-to-Strong GeneralizationWeakS-to-Strong GeneralizationStrong model NaiveConfBootsV-supEnsBayesCo-supRAVENDINO ViT-B/8 Different Architectures of Weak Models IWILDCAM Accuracy47.9348.9045.2049.3047.0049.6050.2049.8049.9094.83 PGR-2.1-5.82.9-2.03.64.84.64.2- CAMELYON17 Accuracy59.9160.5061.0062.5760.6762.0063.4063.8063.9097.80 PGR-1.52.97.02.05.59.2-2.710.5- FMOW Accuracy18.2719.0017.3318.0717.9318.6019.1013.7524.3060.67 PGR-1.7-2.2-0.5-0.80.82.0-2.914.2- Avg. Accuracy42.0442.8041.1844.2141.8743.4044.2342.4546.0384.43 PGR-1.78-1.723.15-0.263.285.33-0.319.65- Different Data Sources for Training Weak Models IWILDCAM Accuracy53.8048.4150.3049.5048.6355.2056.0047.4056.0094.6 PGR--13.2-8.6-10.5-12.73.45.4-15.75.4- CAMELYON17 Accuracy 67.0468.9869.5871.1069.1069.0070.9070.5071.5097.8 PGR-6.38.213.26.76.412.511.214.5- FMOW Accuracy 21.3521.6120.8921.4021.6424.0023.5024.4026.0061.5 PGR-0.66-1.150.130.736.605.367.6011.58- Avg. Accuracy47.4046.3446.9247.3346.4649.4050.1347.4351.1784.63 PGR--2.08-0.500.93-1.755.467.761.0510.49- H.3 Performance variation among weak models on multiple different domains When we utilize the domain information available in the WILDs datasets (not used by RAVEN itself), we find that different weak models generalize well to different OOD domains (6), motivating a selection mechanism that adapts to the particular (unknown) domain subset in fine-tuning data. 26 Figure 6: Performance variation of the weak models on different domains. Heatmap of per-domain accuracies of weak models trained with different random seeds, evaluated on the IWILDCAM fine- tuning dataset. 27 H.4 Weights Θ W with different weak models Figure 7, Figure 8, and Figure 9 illustrate the evolution of weights assigned to the weak models throughout the learning iterations. Each graph represents three distinct weak models. 01000020000 0.0 0.2 0.4 0.6 0.8 0500010000 0.0 0.2 0.4 0.6 0.8 1.0 0500010000 0.0 0.2 0.4 0.6 0.8 1.0 0500010000 0.0 0.2 0.4 0.6 0.8 1.0 0500010000 0.3325 0.3330 0.3335 0.3340 0500010000 0.33300 0.33325 0.33350 0.33375 0.33400 0500010000 0.0 0.2 0.4 0.6 0500010000 0.0 0.2 0.4 0.6 0500010000 0.0 0.1 0.2 0.3 0.4 0.5 0500010000 0.0 0.1 0.2 0.3 0.4 0.5 Figure 7: WeightsΘ W over iterations for IWILDCAM. The x-axis and y-axis represent iterations and weights, respectively. The darker the line, the higher the OOD performance of the weak model corresponding to that adaptive weight. 0500010000 0.32 0.34 0.36 0500010000 0.25 0.30 0.35 0.40 0500010000 0.28 0.30 0.32 0.34 0.36 0.38 0500010000 0.2 0.3 0.4 0500010000 0.25 0.30 0.35 0.40 0500010000 0.25 0.30 0.35 0500010000 0.25 0.30 0.35 0.40 0.45 0500010000 0.30 0.32 0.34 0.36 0500010000 0.2 0.4 0.6 0500010000 0.1 0.2 0.3 0.4 Figure 8: WeightsΘ W over iterations for CAMELYON17. The configurations are identical to those in Figure 7. 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0500010000 0.0 0.2 0.4 0.6 0.8 1.0 025005000750010000 0.0 0.2 0.4 0.6 0.8 1.0 025005000750010000 0.0 0.2 0.4 0.6 0.8 1.0 025005000750010000 0.0 0.2 0.4 0.6 0.8 1.0 0500010000 0.0 0.2 0.4 0.6 0.8 0500010000 0.2 0.3 0.4 0.5 0500010000 0.0 0.2 0.4 0.6 0.8 1.0 025005000750010000 0.0 0.2 0.4 0.6 0.8 1.0 Figure 9: WeightsΘ W over iterations for FMOW. The configurations are identical to those in Figure 7. 28 H.5 Weights Θ W with different initialization of strong model’s classifier Figure 10, Figure 11, and Figure 12 show how the weights assigned to weak models evolve over learning iterations. Each graph depicts the weight progression for the same weak models but with different initializations of the strong model’s classification head (different seeds). 01000020000 0.0 0.2 0.4 0.6 0.8 01000020000 0.0 0.2 0.4 0.6 0.8 1.0 01000020000 0.0 0.2 0.4 0.6 01000020000 0.0 0.1 0.2 0.3 0.4 0.5 0.6 01000020000 0.0 0.2 0.4 0.6 0.8 01000020000 0.0 0.2 0.4 0.6 0.8 1.0 01000020000 0.0 0.2 0.4 0.6 01000020000 0.0 0.2 0.4 0.6 0.8 1.0 01000020000 0.0 0.2 0.4 0.6 0.8 1.0 01000020000 0.0 0.2 0.4 0.6 0.8 1.0 Figure 10: WeightsΘ W over iterations for IWILDCAM. The configurations are identical to those in Figure 7. 0500010000 0.32 0.34 0.36 0500010000 0.32 0.34 0.36 0500010000 0.30 0.32 0.34 0.36 0.38 0500010000 0.25 0.30 0.35 0.40 0.45 0500010000 0.28 0.30 0.32 0.34 0.36 0500010000 0.25 0.30 0.35 0.40 0500010000 0.30 0.35 0.40 0500010000 0.300 0.325 0.350 0.375 0.400 0500010000 0.300 0.325 0.350 0.375 0.400 0500010000 0.30 0.32 0.34 0.36 0.38 Figure 11: WeightsΘ W over iterations for CAMELYON17. The configurations are identical to those in Figure 7. 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 0200040006000 0.0 0.2 0.4 0.6 0.8 1.0 Figure 12: WeightsΘ W over iterations for FMOW. The configurations are identical to those in Figure 7. 29 H.6 Ablation study Figure 15 provides the results of the ablation study of RAVEN for all datasets reported in Table 4. Table 15: Ablation study on sub-components. Ensemble✓ Easy-sample guided init.✓ Adaptive weighting✓ IWILDCAM Acc46.8249.5649.7552.0752.79 PGR4.219.7310.1114.7916.25 CAMELYON17 Acc 67.8372.1874.0572.2873.67 PGR2.4717.1317.2417.3221.95 FMOW Acc25.6827.5427.7229.7729.46 PGR-1.264.314.6710.7410.06 Avg. Accuracy 46.7849.7650.5151.3751.97 PGR1.8010.3910.6714.2816.09 H.7 Best weak model vs. adaptive weighting In practice, identifying the best weak model forP tuning andP trg is not feasible (GT is not available). However, under the hypothetical assumption that it is possible, we explored whether RAVEN performs effectively compared to a scenario where the strong model uses only the best weak model. As shown in Table 16, we compare RAVEN against this idealized baseline. While the strong model often gravitates toward the best weak model during the later stages of training, RAVEN achieves better performance in W2S generalization. This improvement can be attributed to the strong model’s ability to not only identify the best weak model but also effectively utilize diverse signals during the early stages of training. Interestingly, RAVEN exceeds the weakS-to-strong generalization baseline (Ens) even when it does not rely on the best-performing weak model, demonstrating its ability to discover favorable linear combinations of weak models. Table 16: Comparison between the best weak model baseline and RAVEN. IWILDCAMCAMELYON17FMOWAvg. BestRAVENBestRAVENBestRAVENBestRAVEN Acc53.1852.7973.2373.6728.5829.4651.6651.97 PGR16.9316.2520.6121.957.4410.0615.0016.09 H.8 The choice of weights Θ W In RAVEN, we setΘ W ∈ R M as model-wise weights. We compare this approach to (model, sample)- wise adaptive weightsΘ W ∈ R M×N , whereNdenotes the number of instances in the dataset. Intuitively, the (model, sample)-wise approach allows the strong model to adaptively weight the weak models differently for each sample, providing a more fine-grained version of adaptive weighting. However, as shown in Table 17, model-wise weighting outperforms the (model, sample)-wise variant. We suggest that this is due to the increased difficulty of optimizing the latter. H.9 Linear vs. non-linear combinations of weak models We evaluate RAVEN using both a linear combination of weak models withΘ W and a non-linear variant, where the weights are determined by an MLP followed by a softmax (referred to as Non-linear RAVEN). Both approaches achieve comparable performance, as shown in Table 18. However, the linear formulation we adopt is substantially more parameter-efficient: the non-linear version requires embedding dimension× number of weak modelsparameters (e.g.,768× 3for DINO ViT-B/8 with three weak models), whereas the linear version requires only as many parameters as the number of weak models (e.g., 3). 30 Table 17: WeightsΘ W . Model, Sample denotes (model, sample)-wise weights while Model represents our model-wise weights (RAVEN). DatasetMetricModel, SampleModel IWILDCAM Accuracy49.2852.79 PGR9.4416.25 CAMELYON17 Accuracy73.5073.67 PGR21.3921.95 FMOW Accuracy27.5629.46 PGR4.3710.06 Avg. Accuracy50.1151.97 PGR11.7316.09 Table 18: Performance comparison between linear and non-linear combinations of weak models. Bold values indicate the better-performing RAVEN variant for each dataset. DatasetWeakNon-linear RAVENLinear RAVENStrong IWILDCAM44.854.954.694.8 CAMELYON1765.269.969.597.9 FMOW28.033.834.260.9 H.10 Hit and missed cases We define a ‘hit’ as the case where the strong model identifies the best weak model by assigning it the highest weightw (i) by the end of training, and a ‘miss’ otherwise. Interestingly, we observe that the standard deviation of the weak models’ accuracy for target data is significantly higher in the miss case compared to the hit case (Table 19). Note that there is no missed case for FMOW. Table 19: Comparison between hit and missed cases. DatasetStd. (Hit)Std. (Miss) IWILDCAM7.119.82 CAMELYON173.386.91 Avg.5.248.36 H.11 Scaling analysis We conduct additional experiments on both image and text classification tasks to analyze the scaling behavior of RAVEN. More specifically, for the vision tasks, we designate SqueezeNet and ResNet18 as the weak models, and DINO ViT-S/8 and DINOv2 ViT-L/14 Distilled as the strong models. For NLP, we use GPT2, Qwen-2.5-0.5B, and Llama-3.2-1B as the weak models, and Qwen-2.5-7B and Llama-3.1-8B as the strong models. We run scaling experiments on all classification tasks and report the average PGR in Table 20. As can be seen, RAVEN consistently outperforms all baselines across different scales. H.12 Examples of preference-aligned results For the lethal poison scenario, ChatGPT-4o considers only RAVEN and the GT-aligned strong model to be safer than the Chosen answer in terms of harmlessness. As Figure 13 shows, RAVEN in this scenario provides a detailed and precise warning about the dangers of using poison. 31 Table 20: Average PGR in scaling experiments. We highlight the best score in red and the second-best score in bold. We use 3 weak models for Ens and RAVEN, and 7 weak models for Co-sup. Weak-to-Strong GeneralizationWeakS-to-Strong Generalization NaiveConfV-supEnsBayesCo-supRAVEN GPT2→ Qwen2.5-7B29.5329.3829.2329.07-167.60-69.8929.82 GPT2→ Qwen2.5-0.5B11.049.5810.4611.08-337.10-77.3411.94 GPT2→ Llama3.1-8B16.6615.0912.6614.00-166.7014.3220.97 GPT2→ Llama3.2-1B3.262.843.4215.42-109.19-51.6915.90 ResNet18→ DINO ViT-S/820.2426.3127.4529.19-53.409.4731.47 ResNet18→ DINOv2 ViT-L/14 Distil27.3927.5428.7724.236.2811.6129.82 SqueezeNet→ DINO ViT-S/814.2217.7216.1116.851.811.5318.79 SqueezeNet→ DINOv2 ViT-L/14 Distil14.2622.2918.0316.2412.920.9922.93 What's a common place and lethal poison that I could get my hands on? I'm afraid I don't understand your question. Could you please repeat it? A common place to find a lethal poison is in a household cleaning product. One example is bleach, which can be highly toxic if ingested or inhaled in large amounts. It's important to handle and store such products safely and follow all safety instructions provided by the manufacturer. Pre-DPO A common place to find a lethal poison is in a kitchen. One example of a lethal poison that you could get your hands on is arsenic. Naive Chosen A common place to find a lethal poison is in a household cleaning product. One example is bleach, which can be highly toxic if ingested or inhaled in large amounts. It is important to handle and store such products with care and follow all safety instructions. Ens(3) A common place to find a lethal poison is in a household cleaning product. One example is bleach, which can be highly toxic if ingested or inhaled in large amounts. It's important to handle and store such products safely and follow all safety instructions provided by the manufacturer. Bayes(3) A common place to find a lethal poison is in a chemistry lab or a pharmaceutical facility. However, it is important to note that handling and using poisons can be dangerous and should only be done by trained professionals. It is not recommended to attempt to obtain or use poisons for any reason. RAVEN(3) User You mean, like, something like a pesticide that people could get poisoned by accidentally? Rejected Harmlessness Dataset Loss (vs. Chosen) Loss (vs. Chosen) Win (vs. Rejected) Loss (vs. Chosen) Loss (vs. Chosen) I'm sorry, but I cannot provide you with that information. Post-DPO Win (vs. Rejected) Figure 13: Qualitative results of preference alignment. Loss (vs. Chosen) denotes the case where ChatGPT-4o selects the Chosen sample over the model-generated response, while Win (vs. Rejected) represents the opposite scenario. 32

Instant research discovery