Paper deep dive

UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

Lingling Fu

Year: 2024Venue: arXiv preprintArea: Deception & FailureType: EmpiricalEmbeddings: 42

Models: Pythia-1.4B, Qwen-0.5B, TinyLlama-1.1B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/11/2026, 12:42:54 AM

Summary

UMM-RM (Upcycle-and-Merge MoE Reward Model) is a framework designed to mitigate reward hacking in RLHF by upscaling dense reward models into Mixture-of-Experts (MoE) architectures during training, then consolidating them into a single dense model via learnable merging. This approach leverages expert diversity to improve robustness and reduce spurious correlations while maintaining the inference efficiency of a standard dense model.

Entities (5)

PPO · algorithm · 100%Reward Hacking · problem · 100%RLHF · methodology · 100%UMM-RM · model-architecture · 100%AlpacaFarm · dataset · 95%

Relation Signals (3)

UMM-RM → uses → MoE

confidence 100% · UMM-RM first upscales the feed-forward layers of a dense backbone into a mixture-of-experts (MoE) reward model

UMM-RM → mitigates → Reward Hacking

confidence 95% · We propose UMM-RM... for Mitigating Reward Hacking

PPO → utilizes → Reward Model

confidence 90% · In RLHF, PPO commonly includes a penalty for deviating from a reference supervised fine-tuned (SFT) policy

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reward models (RMs) are a critical component of reinforcement learning from human feedback (RLHF). However, conventional dense RMs are susceptible to exploitation by policy models through biases or spurious correlations, resulting in reward hacking: RM scores increase during training while alignment with human preferences deteriorates, a problem that is further exacerbated under distribution this http URL address this issue, we propose UMM-RM (Upcycle-and-Merge MoE Reward Model). UMM-RM first upscales the feed-forward layers of a dense backbone into a mixture-of-experts (MoE) reward model with shared experts. The shared experts are always activated to capture instruction-agnostic preference signals, while the remaining experts model fine-grained preferences across instructions or task regimes. After training, the experts are consolidated into a single dense RM via learnable merging this http URL design retains the robustness and exploitation resistance provided by expert diversity while avoiding the inference overhead of MoE architectures or explicit ensembles. Experiments across multiple base models and preference datasets show that, compared with standard dense RMs, UMM-RM improves accuracy on preference data, reduces reward hacking during PPO training, and yields more stable preference alignment.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

42,148 characters extracted from source content.

Expand or collapse full text

UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking Lingling Fu † Guangxi University 2313593009@st.gxu.edu.cn Yongfu Xue † xueyongfu@outlook.com Abstract—Reward models (RMs) are a critical component of reinforcement learning from human feedback (RLHF). However, conventional dense RMs are susceptible to exploitation by policy models through biases or spurious correlations, resulting in reward hacking: RM scores increase during training while alignment with human preferences deteriorates, a problem that is further exacerbated under distribution shift.To address this issue, we propose UMM-RM (Upcycle-and-Merge MoE Reward Model). UMM-RM first upscales the feed-forward layers of a dense backbone into a mixture-of-experts (MoE) reward model with shared experts. The shared experts are always activated to capture instruction-agnostic preference signals, while the remaining experts model fine-grained preferences across instructions or task regimes. After training, the experts are consolidated into a single dense RM via learnable merging weights.This design retains the robustness and exploitation resistance provided by expert diversity while avoiding the inference overhead of MoE architectures or explicit ensembles. Experiments across multiple base models and preference datasets show that, compared with standard dense RMs, UMM-RM improves accuracy on preference data, reduces reward hacking during PPO training, and yields more stable preference alignment. Index Terms—RLHF, Reward Model. I. INTRODUCTION Large language models have achieved remarkable progress in open-domain question answering, complex reasoning, and code generation [1], [2]. However, alignment between their behavior and human preferences or values cannot be naturally guaranteed by pretraining data alone: models may generate unsafe, biased, or user-misaligned content, posing real-world risks [3]. Consequently, achieving controllable and reliable preference alignment while preserving model capabilities has become a central challenge in current alignment and safety research. In practice, preference alignment is typically built upon Supervised Fine-Tuning (SFT). By leveraging high-quality instruction data, SFT improves instruction-following ability and output stability, providing a reasonable initial policy for subsequent preference learning. On top of this foundation, existing approaches can be broadly categorized into two classes. The first involves offline optimization based on preference data (e.g., DPO [4]), which directly updates the policy using preference pairs. The second consists of online RLHF methods, which train a reward model to approximate human preferences and iteratively optimize the policy during the reinforcement † Equal contribution. learning phase. This paper focuses on the online RLHF framework based on the Proximal Policy Optimization (PPO) algorithm [5]. Although RLHF has achieved significant success in human- alignment tasks,its core reward model still suffers from reliabil- ity issues. When the reward model exhibits biases in capturing human preferences, the policy model can over-optimize for the reward signal, leading to reward hacking, i.e., exploiting short- cut features of the reward model to increase scores rather than producing outputs that genuinely align with human preferences. This phenomenon is commonly referred to as reward hacking [6]. During RLHF, the policy continuously pushes outputs into regions where the reward model provides weak supervision, further leveraging and amplifying uncertainties and systematic biases, ultimately causing a decoupling between the reward and true quality and resulting in a decline in generation quality. To mitigate reward hacking, prior work has pursued multiple directions. First, reward-model–centric methods refine training objectives, regularization, and data coverage to improve out- of-distribution generalization and robustness under distribution shift and preference noise [7], [8]. Second, conservative policy optimization leverages uncertainty modeling and reward model ensembles, explicitly exploiting inter-model disagreement to suppress over-optimization and reduce reward hacking [9], [10]. However, these approaches incur significant computational and resource costs at inference, as multiple reward models must be queried. WARM [11] mitigates this overhead by averaging the parameters of multiple fine-tuned reward models, achieving approximate robustness gains at lower cost. Despite their effectiveness, ensembling and parameter-space averaging cannot fundamentally prevent reward hacking when constituent reward models share similar biases or systematic blind spots. Moreover, they do not impose explicit constraints on reward scoring criteria, leaving policies vulnerable to exploiting high- scoring loopholes. We propose an Upcycle-and-Merge MoE reward model framework for RLHF to mitigate reward hacking in reward models. During training, the original dense model is upcy- cled into a standard MoE and trained on preference-aligned data, enabling different experts to learn multi-dimensional representations of human preferences (e.g., helpfulness, safety, fluency, and consistency). This structural diversity enhances the robustness of preference modeling. At the same time, trained MoE experts may give rise to speculative experts that are arXiv:2512.00724v2 [cs.LG] 1 Feb 2026 misaligned with true human preferences—overfitting with high confidence in local subspaces, relying on spurious features, or forming erroneous scoring rules on rare samples—which can be systematically amplified by the policy, producing abnormal reward responses and deviating from true preferences. To suppress this risk, we perform post-training weighted merging of experts, compressing the MoE back into a single dense reward model. The merging step reduces the high-variance reward responses introduced by individual misaligned experts and weakens the reliance on spurious features, thus significantly mitigating reward hacking during RLHF, while maintaining the same parameter scale and inference cost as the dense reward model. Compared to MoE, it also offers higher throughput and lower resource consumption at deployment. Overall, our contributions are as follows: •We propose UMM-RM, a Mixture-of-Experts (MoE) reward model constructed using an upcycle-and-merge strategy. Compared with ensemble-based approaches, UMM-RM mitigates reward hacking without substantially increasing its parameter count. •We systematically validated across multiple models that the UMM-RM method mitigates reward hacking. Our experimental results further indicate that, increasing the number of activated experts can substantially improve the stability and robustness of reward evaluation. •We further validate UMM-RM on multiple datasets and base models, demonstrating consistent improvements in re- ward modeling accuracy. In end-to-end generation quality assessments, UMM-RM achieves alignment performance comparable to ensemble and MoE reward models while maintaining computational costs close to the dense reward model. I. BACKGROUND A. SFT Supervised fine-tuning (SFT) is a crucial stage in the reinforcement learning from human feedback (RLHF) pipeline. It uses a high-quality human-annotated dataset to supervise the optimization of a pretrained model, enabling the model to reliably produce target outputs with semantic coherence, instruction-following ability, and controllable formatting before preference modeling and reinforcement learning optimization. B. Reward modeling Reward models infer human preference structure from pairwise comparisons. Given a promptx, they assign rewards to candidate responses so that outputs closer to human expectations receive higher scores. In reinforcement learning from human feedback (RLHF), this reward serves as a scalar training signal used to optimize the policy toward human- aligned behavior. A standard approach uses the Bradley–Terry formulation [12], minimizing L(R) =−E (x,y w ,y l )∈D logσ R(x,y w )− R(x,y l ) . Here,Dcontains preference triples(x,y w ,y l ), wherey w is the preferred response andy l the rejected one;R(x,y) denotes the reward andσ(·)the sigmoid. Minimizing this loss pushesR(x,y w ) > R(x,y l ), aligning model scores with human preferences. C. PPO Proximal Policy Optimization (PPO) [5] is an online policy- gradient algorithm that increases expected return through small, constrained updates, offering a strong balance of stability and sample efficiency. As a result, it is a standard optimizer in reinforcement learning from human feedback (RLHF) pipelines. In RLHF, PPO commonly includes a penalty for deviating from a reference supervised fine-tuned (SFT) policy: R PPO (q,a) = R(q,a)− β log π PPO (a| q) π init (a| q) (1) whereπ PPO is the optimized policy,π init is the SFT reference policy, andβsets the penalty strength. This term is equivalent to imposing a KL-divergence constraint betweenπ PPO andπ init , preventing excessive drift while allowing reward improvement. D. Mixture-of-Experts Feed-forward network (FFN) layers are central to Trans- former models and store substantial knowledge. Mixture-of- Experts (MoE) architectures replace a single FFN with multiple FFN experts and use a gating network to route each token to a small subset of experts. By sparsely activating experts, MoE scales model capacity with sublinear compute growth [13], allowing many more parameters at minimal additional cost and improving capability [14]. An MoE layer is defined as y = n X i=1 G(x) i · E i (x),(2) wherexandyare the input and output,nis the number of experts,G(x) i is the normalized routing weight, andE i (x)is the output of expert i on x. E. Reward Hacking During RLHF reinforcement learning, the policy is trained to maximize a reward model that only approximates human preferences. As optimization progresses, the proxy reward typically increases nearly monotonically, while true alignment often improves mainly early in training and can later dete- riorate—indicative of reward hacking. A practical detection method is to re-score the same policy outputs with a stronger reference reward model (“Gold RM”), assumed to better reflect human preferences, and compare it to the proxy. Early in training, both scores usually rise together; under over- optimization, the proxy score keeps increasing while the Gold RM score declines, producing a persistent divergence that signals reward hacking. Although scaling the proxy reward model and expanding training data can reduce over-optimization [6], proxy reward models are often built from large pretrained language models, making scaling costly, frequently impractical, and unlikely to be a sustainable long-term remedy. I. METHOD A. Motivation In the reinforcement learning from human feedback (RLHF) paradigm, the reward model is used to approximate human preferences and provide learning signals for policy optimization. Prior studies have shown that reward hacking typically arises from the reward model’s overfitting to a single pattern or a localized region of the input space: the model may rely on spurious features or form high-confidence but incorrect scoring rules on rare samples, thereby assigning abnormally high rewards to outputs that deviate from true human preferences [3], [6]. Mixture-of-Experts (MoE) architectures naturally offer multi- faceted representational capacity for reward modeling, where different experts can capture distinct dimensions of preference (e.g., helpfulness, harmlessness, safety, fluency, and coherence). However, in the reward modeling setting, MoE models may also learn speculative experts that are inconsistent with human preferences (e.g., experts biased toward output length, templated expressions, or superficial statistical patterns). Similar to dense reward models, reward hacking in MoE reward models is often triggered by extreme overfitting of a small number of experts in local regions: when the router consistently selects one (or a few) experts for specific input regions, the policy model may exploit the anomalous responses of these experts, amplifying reward bias and inducing unstable training dynamics. A central limitation of sparse upcycling MoE reward modeling is the small scale of reward data, which leaves each expert with too few effective training samples. This scarcity increases estimator variance and amplifies the risk of reward hacking. We address this by adding a shared expert to each MoE layer and forcing it to activate for every token. The shared expert learns instruction- and distribution-invariant preference representations (e.g., core safety constraints, linguistic fluency, and general helpfulness), while the remaining experts are routed dynamically to capture instruction- or context-specific preferences. This architecture decomposes representations into “general + specialized” components: the shared expert provides a stable, low-variance baseline, and the routed experts add expressive capacity, improving the bias–variance trade-off in preference modeling. B. Training the MoE Reward Model For thettoken, the input to the MoE layer at layerℓis denoted asu (ℓ) t . We employKexperts in each layer, where the shared expert is indexed bye = 0and the standard experts are indexed bye∈1,...,K− 1. The gating weight of the shared expert is fixed by a hyperparameterα ∈ [0, 1]and is activated for all tokens: g (ℓ) 0,t = α. For the standard experts, the router produces routing logits r (ℓ) e,t . LetT (ℓ) t denote the set of top-mexperts selected from 1,...,K− 1(wheremis the number of sparsely activated experts), and a softmax normalization is applied within this set: π (ℓ) e,t = exp r (ℓ) e,t P j∈T (ℓ) t exp r (ℓ) j,t , e∈T (ℓ) t . The remaining gating mass 1− α is allocated to the experts: g (ℓ) e,t = ( (1− α)π (ℓ) e,t , e∈T (ℓ) t , 0,otherwise, e∈1,...,K− 1. Therefore, the output of this layer is given by h (ℓ) t = α FFN (ℓ) 0 u (ℓ) t + K−1 X e=1 g (ℓ) e,t FFN (ℓ) e u (ℓ) t , g (ℓ) 0,t + K−1 X e=1 g (ℓ) e,t = 1. C. Learnable Merging with Shared-Expert Ratio After training, we merge the MoE layers in parameter space through learnable weighted averaging to obtain standard feed- forward network (FFN) layers, thereby constructing a dense reward model. To ensure that the merging coefficients are semantically consistent with the gating behavior during training, we assign the shared expert a merging weight equal toα, and set the merging coefficients of the standard experts to the expectation of their gating weights over the data distribution. Specifically, the merging weights of the standard experts are learned over a small set of calibration samples. ̄g (ℓ) e ≜E (x,t) h g (ℓ) e,t i , e∈1,...,K− 1, K−1 X e=1 ̄g (ℓ) e = 1− α. LetΘdenote the parameters of each expert. Then, the parameters of the merged dense FFN at layerℓare given by Θ (ℓ) merge = α Θ (ℓ) 0 + K−1 X e=1 ̄g (ℓ) e Θ (ℓ) e . We interpret the parameter-space merging as an ensemble- style approximation to a set of reward experts. By taking a linear combination of expert parameters, the procedure attenuates extreme weights from any single expert, which in turn yields a smoother and more locally robust reward function. From a statistical perspective, this aggregation typically introduces a small bias while substantially reducing variance; since reward hacking is often associated with high-variance or brittle reward signals, the reduced variance decreases the chance of exploitable responses. Moreover, collapsing the experts into a single dense model eliminates the inference-time overhead introduced by sparse MoE routing. Overall, the proposed merge provides (i) stability, because shared experts contribute with a fixed proportionαthat preserves universal safety and consistent preferences; (i) overfitting suppression, because experts sensitive to spurious features are systematically diluted; and (i) reward smoothing, because the merged reward becomes less sensitive to local input perturbations, thereby compressing the exploitable space available to policy models. IV. EXPERIMENTAL SETUP A. Datasets We evaluate UMM-RM’s capacity to mitigate reward hacking on the AlpacaFarm dataset [15]. AlpacaFarm includes both open-ended and closed-form questions and is widely used for RLHF training and evaluation [9], [16], [17]. It contains approximately 52k samples, each comprising an instruction, an optional auxiliary input, and an example output. The dataset provides standard splits (e.g., train/validation) and human preference annotations to support reward model learning and evaluation. We evaluated reward model (RM) scoring accuracy on the Anthropic Helpful and Harmless (H) dataset [18] and the Webgptcomparisons subset of Unified-Feedback [19]. For H harmless, supervised fine-tuning (SFT) used the first 12k examples from the harmless-base training split (10k train, 2k validation). RM training used examples 12,000–22,500 (10k train, 500 validation), and evaluation used the first 500 examples from the test split. H helpful followed the same procedure using helpful-base, with the same split structure for SFT and RM. For the Webgptcomparisons subset of Unified-Feedback, we use the first 11k training examples for SFT (10k train, 1k validation). For RM, we train on the first 7k examples from the remaining training data and use the rest for validation. We evaluate generalization on held-out data using the official Webgptcomparisons validation split as the test set. B. Models To evaluate the proposed method across diverse model scales and architectures, we conduct comparative experiments using four base models: Qwen2.5-0.5B and Qwen2.5-1.5B [20], TinyLlama-1.1B [21], and Pythia-1.4B [22]. For each backbone, we train an optimized UMM-RM reward model to assess robustness and generality across capacities and designs. We additionally use a Llama3-8B–based reward model [23] as a ”gold standard”, substituting its reward signals for human preference labels as a proxy for approximate human judgments during subsequent training and evaluation. C. RLHF Pipeline Our RLHF pipeline broadly follows Coste et al. [9], with minor implementation differences. The setup is as follows. a) SFT: We perform supervised fine-tuning of both the policy and proxy reward models on 10k instruction-following examples from the AlpacaFarmsftsplit to improve instruction adherence. We use full-parameter fine-tuning for 3 epochs (learning rate5× 10 −6 , batch size 16, max sequence length 512) with Adam (β 1 = 0.9, β 2 = 0.95) and 600 warm-up steps. b) Reward Model: Initialized from the SFT check- point, the reward model is trained on the AlpacaFarm alpaca_human_preferencesplit, holding out 500 sam- ples for evaluation and using the remainder for training. We run full-parameter fine-tuning for 3 epochs (learning rate 3× 10 −5 , batch size 32, max sequence length 520) with Adam (β 1 = 0.9, β 2 = 0.95) and a 0.03 warm-up ratio. c) UMM-RM: For a fair comparison, UMM-RM uses the same training hyperparameters as the reward model. It comprises eight experts, and we evaluate variants that activate different numbers of experts. When learning weighting coefficients, we train for up to 200 steps with learning rate 10 −4 and no warm-up. d) PPO:We train and evaluate PPO on 20k samplesfromtheAlpacaFarmunlabeledsplit (alpaca_instructionssubset),reserving2kfor evaluation. Each iteration samples 256 rollouts (chunk size 32, temperature 1) and applies four gradient updates (batch size 32). Training runs for 3,000 steps with clipping range 0.2, KL coefficient 0, and GAEλ = 0.95. Optimization uses AdamW (learning rate2× 10 −6 ,β 1 = 0.9, β 2 = 0.95, weight decay 10 −6 ) with cosine annealing. The policy and value functions share a backbone and branch into separate heads only after the final two layers to output the action distribution and value estimates. D. Experimental results 1) Evaluation on RLHF: We build UMM-RM reward models with 2, 4, and 6 activated experts and evaluate their ability to reduce reward hacking during PPO training. Throughout, we report the number of routed experts, the shared expert is always active. Experiments span multiple base models and compare UMM-RM against a dense RM and ensemble RM baselines using Mean/WCO/UWO [9] and WARM [11]. For consistency, all methods’ reward scores are shifted so the minimum starts at zero. Figure 1 summarizes the performance of UMM-RM as an RLHF reward model. Using TinyLlama-1.1B as an example (Figs. 1a and 1b), the dense RM’s proxy score increases monotonically with training, whereas its gold score rises early and then collapses, reflecting classic reward hacking. By contrast, UMM-RM exhibits a steadier increase in proxy score and a generally consistent improvement in gold score, with only a small late-stage decline. These results suggest that UMM-RM largely mitigates reward hacking, though minor residual effects remain. In addition, enabling more experts (4/6) suppresses reward hacking more effectively than enabling fewer (2). On TinyLlama-1.1B, we further compare UMM-RM with ensemble RM methods [9] and WARM [11]. The ensemble RM trains four reward models with different random seeds and aggregates them using Mean, WCO, and UWO, respectively; WARM ensembles three reward models trained with different learning rates. As shown in Fig. 2, UMM-RM consistently achieves higher gold scores during PPO training than the Mean/WCO/UWO ensemble RMs, and UMM-RM with 4 or 050010001500200025003000 Step 0 1 2 3 4 Proxy Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (a) TinyLlama-1.1B 050010001500200025003000 Step 1.0 0.5 0.0 0.5 1.0 1.5 Gold Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (b) TinyLlama-1.1B 050010001500200025003000 Step 0.0 0.5 1.0 1.5 2.0 Proxy Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (c) Pythia-1.4B 050010001500200025003000 Step 1.0 0.5 0.0 0.5 Gold Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (d) Pythia-1.4B 050010001500200025003000 Step 0 2 4 6 8 Proxy Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (e) Qwen2.5-0.5B 050010001500200025003000 Step 1.25 1.00 0.75 0.50 0.25 0.00 0.25 0.50 Gold Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (f) Qwen2.5-0.5B 050010001500200025003000 Step 0.0 0.5 1.0 1.5 2.0 Proxy Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (g) Qwen2.5-1.5b 050010001500200025003000 Step 2.0 1.5 1.0 0.5 0.0 0.5 1.0 Gold Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Dense RM (h) Qwen2.5-1.5b Fig. 1: Compared with the dense reward model, UMM-RM more effectively mitigates reward hacking. The dashed curves denote the reward scores assigned by the proxy reward model, while the solid curves represent the gold reward scores evaluated by the Llama3-8B reward model. 050010001500200025003000 Step 0.0 0.5 1.0 1.5 2.0 2.5 Proxy Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Mean(4-RMs) WCO(4-RMs) UWO(4-RMs) WARM (a) TinyLlama-1.1B 050010001500200025003000 Step 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Gold Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) Mean(4-RMs) WCO(4-RMs) UWO(4-RMs) WARM (b) TinyLlama-1.1B Fig. 2: Comparison of PPO training reward scores between UMM-RM and reward models based on different ensemble strategies built on the TinyLlama-1.1B backbone. 6 activated experts also outperforms WARM. These results demonstrate that, by introducing structured expert diversity within a single reward model, UMM-RM provides a more stable reward signal and effectively suppresses reward hacking while maintaining inference efficiency. E. Accuracy Evaluation We evaluated the UMM-RM reward model built on TinyLlama-1.1B and compared it with the ensemble RM [9] on the Anthropic Helpful and Harmless (H) benchmarks and the Webgptcomparisons subset of Unified-Feedback (Table I). On H-Helpful and Webgptcomparisons, UMM-RM with 6 activated experts achieved 57.6% and 60.8% accuracy, respectively, surpassing the unmerged MoE model and the Mean, WCO, and UWO ensemble variants. We further evaluated UMM-RM on Qwen2.5-0.5B and Pythia-1.4B (Table I). For all three base models, UMM-RM with 2, 4, or 6 activated experts consistently outperformed the standard dense reward model. The best H-Helpful result was 67.2% with Qwen2.5-0.5B, while the 6-expert UMM-RM on Pythia-1.4B achieved 55.2% on H-Harmless and 57.8% on Webgptcomparisons. Overall, these results show that UMM-RM reliably improves reward-model accuracy across model scales and evaluation settings, yielding stronger reward signals for subsequent PPO training. ModelsAnthropicWebGPT HarmlessHelpful Dense RM51.244.652.2 Mean Optimization (ensemble RM)57.155.051.4 Worst-Case Optimization (ensemble RM)55.454.860.6 Uncertainty-Weighted Optimization (ensemble RM)58.054.659.6 UMM-RM (2-experts)56.454.257.8 UMM-RM (4-experts)58.055.258.6 UMM-RM (6-experts)58.457.660.8 TABLE I: Accuracy comparison (%) on preference tasks across different reward model methods using the TinyLlama-1.1B model. F. Unmerged MoE vs. UMM-RM Using TinyLlama-1.1B, we compared the accuracy of unmerged MoE expert reward models (Table I) and their behavior during PPO training (Figure 3). With 4 and 6 experts activated, UMM-RM consistently achieved higher gold scores than the unmerged MoE. This suggests that sparse MoE reward models alone do not reliably suppress reward hacking. By contrast, the “upcycle-and-merge” approach consolidates expert diversity into a single dense reward function, reducing the policy’s tendency to overexploit individual experts or local biases and thereby mitigating reward hacking more effectively. ModelsAnthropicWebGPT HarmlessHelpful Base Model: Qwen2.5-0.5B Dense RM38.665.857.2 UMM-RM (2-experts)48.866.858.2 UMM-RM (4-experts)50.266.057.8 UMM-RM (6-experts)50.867.258.4 Base Model: Pythia-1.4B Dense RM48.044.650.8 UMM-RM (2-experts)54.653.054.2 UMM-RM (4-experts)54.253.454.0 UMM-RM (6-experts)55.254.857.8 TABLE I: Comparison of accuracy (%) on preference tasks across different reward models using Qwen2.5-0.5B and Pythia- 1.4B. ModelsAnthropicWebGPT HarmlessHelpful Dense RM51.244.652.2 UnMerged MoE RM (2-experts)59.853.857.0 UnMerged MoE RM (4-experts)58.255.858.2 UnMerged MoE RM (6-experts)57.456.260.2 UMM-RM (2-experts)56.454.257.8 UMM-RM (4-experts)58.055.258.6 UMM-RM (6-experts)58.457.660.8 TABLE I: Comparison of accuracy (%) on preference tasks across different reward model methods using TinyLlama-1.1B. G. Effect of the Shared-Expert Weight Using TinyLlama-1.1B as the backbone, we follow the UMM-RM 6-expert activation setup to examine how the shared- expert weight coefficient affects PPO training (Fig. 4). We test coefficients of 0.1, 0.5, 0.75, and 0.9. Performance peaks at 0.5 (highest gold score) and degrades at 0.9 (lowest). These results suggest that moderate sharing stabilizes the reward function and mitigates noise, whereas overly strong sharing reduces expert diversity and drives the reward model toward an effectively single-preference behavior. H. End-to-End Generation Quality Evaluation We evaluate how different reward models mitigate reward hacking using thealpaca_human_evaluationsplit of the AlpacaFarm dataset, with TinyLlama-1.1B as the base model. We compare PPO-trained policies using UMM-RM, a standard dense RM, and ensemble RMs (Mean, WCO, UWO), along with the corresponding SFT baseline. Results are shown in Table IV. Overall, UMM-RM yields higher win rates than both the dense RM and SFT, suggesting it steers PPO toward responses aligned with human preferences rather than simply maximizing reward scores. Increasing the number of activated experts fur- ther improves win rates, indicating that multi-expert activation strengthens the reward signal and reduces over-exploitation 050010001500200025003000 Step 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Proxy Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) UnMerged-MoE(6experts) UnMerged-MoE(4experts) UnMerged-MoE(2experts) (a) TinyLlama-1.1B 050010001500200025003000 Step 0.0 0.5 1.0 1.5 2.0 Gold Score UMM-RM(6experts) UMM-RM(4experts) UMM-RM(2experts) UnMerged-MoE(6experts) UnMerged-MoE(4experts) UnMerged-MoE(2experts) (b) TinyLlama-1.1B Fig. 3: Comparison of PPO training reward scores between MoE reward models with different numbers of activated experts and UMM-RM based on the TinyLlama-1.1B backbone. 050010001500200025003000 Step 0.0 0.5 1.0 1.5 2.0 2.5 Proxy Score UMM-RM(weighted-0.1) UMM-RM(weighted-0.5) UMM-RM(weighted-0.75) UMM-RM(weighted-0.9) (a) TinyLlama-1.1B 050010001500200025003000 Step 0.25 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Gold Score UMM-RM(weighted-0.1) UMM-RM(weighted-0.5) UMM-RM(weighted-0.75) UMM-RM(weighted-0.9) (b) TinyLlama-1.1B Fig. 4: Effects of different shared-expert weight coefficients in the UMM-RM method based on the TinyLlama-1.1B model during PPO training. of single-model biases. Notably, the 6-expert UMM-RM outperforms the Mean, WCO, and UWO ensembles, implying that the upcycle-and-merge shared-expert design provides more effective reward supervision than simple ensembling. While the 4-expert UMM-RM slightly trails the Mean ensemble, UMM- RM scales more consistently as experts increase, supporting its capacity to curb amplified reward-model bias. In this experiment, preference judgments are generated using the GPT-4.1 nano model, following the prompt template below: LLM Judge Prompt Template Now please compare the models by the quality of their answers. If one model’s answer is clearly better, select that model as the winner. If the two answers are similar in quality, equally good, or equally bad, select "tie". Do NOT force a decision when the difference is unclear. Return ONLY a valid Python dictionary in the following format: "winner": "model_1" | "model_2" | "tie" Your response must contain nothing else because it will be directly executed in Python. Please provide the judgment that the majority of humans would give. ModelsOpponent AlpacaFarm Win↑TieLose↓ UMM-RM(2-experts) Dense RM 28.943.028.1 UMM-RM(4-experts)30.645.523.9 UMM-RM(6-experts)36.440.323.3 UMM-RM(2-experts) SFT Model 36.427.336.3 UMM-RM(4-experts)32.237.530.3 UMM-RM(6-experts)32.339.927.8 UMM-RM(4-experts) Mean Optimization (4-RMs)11.871.117.1 Worst-Case Optimization (4-RMs)27.063.49.7 Uncertainty-Weighted Optimization (4-RMs)16.573.99.6 UMM-RM(6-experts) Mean Optimization (4-RMs)22.060.417.6 Worst-Case Optimization (4-RMs)28.360.211.5 Uncertainty-Weighted Optimization (4-RMs)23.962.413.7 TABLE IV: End-to-end generation quality evaluation of policy models trained with PPO under different reward model methods. V. RELATED WORK A. Mitigating Reward Hacking in RLHF Reward hacking in RLHF arises when the learned reward model deviates from true human preferences. Under closed-loop optimization of a fixed reward (Goodhart’s effect), the policy exploits these discrepancies, producing unintended behav- iors—particularly when reward models are under-parameterized or trained on limited data [6]. Mitigations generally fall into three categories: improving reward-model training and data coverage, using ensembles or uncertainty-aware constraints to curb over-optimization, and introducing algorithmic safeguards such as explicit constraints or online reward updates. One line of work strengthens single reward models by improving calibration and robustness. Qin et al. [24] propose confidence-aware training to reduce errors from ambiguous or noisy labels. Yang et al. [25] improve out-of-distribution (OOD) generalization by regularizing hidden states and adding a text-generation loss to the reward head [25]. PIRA [26] re- frames reward-model training as instruction-guided preference evaluation with explicit criteria, so the model learns to make preference judgments rather than score Q–A pairs directly. At inference, it stabilizes rewards via two-stage averaging: across multiple evaluation instructions and across value-head outputs under stochastic sampling to reduce variance. However, single- model approaches remain susceptible to systematic reward bias and therefore cannot fundamentally eliminate reward hacking. A second line mitigates over-optimization through reward- model ensembles. Coste et al. [9] evaluate mean, worst-case, and uncertainty-weighted objectives. Ahmed et al. [10] reduce ensemble cost via a shared backbone with multiple linear heads. Zhang et al. [27] compare linear and LoRA-based ensembles, finding LoRA ensembles more effective against reward hacking. Ram ́ e et al. [11] improve OOD reliability by weight-averaging reward models pretrained under different conditions, but this requires multiple pretrained models and substantial compute. Eisenstein et al. [28] argue that ensembling alone is insufficient and that diversity in reward signals is essential. In practice, ensemble methods often incur substantial inference overhead, limiting scalability in large-scale RLHF. More recent work uses MoE-based reward models to com- bine specialization with structured diversity. Quan et al. [29] propose a two-level MoE reward model, using a sparse router to select dense experts whose outputs are fused by an MLP. Wang et al. [30] similarly gate experts using prompt semantics and multi-dimensional evaluations to produce a scalar reward. While these approaches better represent complex preferences, sparse routing and architectural complexity typically raise inference cost, which remains a key barrier to large-scale RLHF deployment. B. Upcycled MoE Model Training large-scale neural networks is computationally expensive, so sparsely activated architectures such as Mixture- of-Experts (MoE) are used to increase model capacity without proportionally increasing compute. Yet training MoE from scratch is still costly, motivating upcycling methods that initial- ize sparse MoE models from dense checkpoints. Komatsuzaki et al. [31] show that this strategy can cut pretraining costs by roughly half. However, Ding et al. [32] report that naive sparse upcycling yields limited benefits for instruction tuning. They propose XFT (eXpert Fusion Training), which expands capacity via shared experts and routing-weight normalization during MoE training, then merges experts back into a dense model. This treats MoE as an intermediate training stage that improves learning while maintaining efficient dense inference. Similarly, Xue et al. [33] show that knowledge learned by sparse experts can be distilled into dense models. VI. CONCLUSIONS In this study, we address the tendency of reward models to exhibit reward hacking during RLHF training and propose UMM-RM, a Mixture-of-Experts (MoE) reward modeling approach based on an upcycle-and-merge strategy. The method first upgrades a dense backbone into an MoE reward model and then merges the experts back into a dense model, yielding a reward model with a parameter scale comparable to the original. This design maintains efficiency by significantly reducing infer- ence overhead while enhancing the reward modeling capacity. Experimental results across diverse settings demonstrate that UMM-RM effectively mitigates reward hacking during RLHF training. Overall, UMM-RM offers a practical and efficient approach for constructing reliable reward models. REFERENCES [1]T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, p. 1877–1901, 2020. [2]H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [3] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, p. 27 730–27 744, 2022. [4]R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in neural information processing systems, vol. 36, p. 53 728–53 741, 2023. [5]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [6]L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in International Conference on Machine Learning. PMLR, 2024, Conference Proceedings, p. 10 835–10 866. [7]R. Yang, R. Ding, Y. Lin, H. Zhang, and T. Zhang, “Regularizing hidden states enables learning generalizable reward model for llms,” Advances in Neural Information Processing Systems, vol. 37, p. 62 279–62 309, 2024. [8]C. Jia, “Generalizing reward modeling for out-of-distribution preference learning,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2024, p. 107–124. [9]T. Coste, U. Anwar, R. Kirk, and D. Krueger, “Reward model ensembles help mitigate overoptimization,” arXiv preprint arXiv:2310.02743, 2023. [10]A. M. Ahmed, R. Rafailov, S. Sharkov, X. Li, and S. Koyejo, “Scalable ensembling for mitigating reward overoptimisation,” arXiv preprint arXiv:2406.01013, 2024. [11]A. Ram ́ e, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret, “Warm: On the benefits of weight averaged reward models,” arXiv preprint arXiv:2401.12187, 2024. [12]R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, p. 324, 1952. [13]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. [14] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” arXiv preprint arXiv:2110.01786, 2021. [15] Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto, “Alpacafarm: A simulation framework for methods that learn from human feedback,” Advances in Neural Information Processing Systems, vol. 36, p. 30 039–30 069, 2023. [16]Y. Miao, S. Zhang, L. Ding, R. Bao, L. Zhang, and D. Tao, “Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling,” Advances in Neural Information Processing Systems, vol. 37, p. 134 387–134 429, 2024. [17]H. Lang, F. Huang, and Y. Li, “Fine-tuning language models with reward learning on policy,” arXiv preprint arXiv:2403.19279, 2024. [18] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, and T. Henighan, “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022. [19]D. Jiang, X. Ren, and B. Y. Lin, “Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,” arXiv preprint arXiv:2306.02561, 2023. [20] Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, “Qwen2.5 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2412.15115 [21]P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024. [Online]. Available: https://arxiv.org/abs/2401.02385 [22]S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal, “Pythia: A suite for analyzing large language models across training and scaling,” 2023. [Online]. Available: https://arxiv.org/abs/2304.01373 [23] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv e-prints, p. arXiv–2407, 2024. [24] B. Qin, D. Feng, and X. Yang, “Towards understanding the influence of reward margin on preference model performance,” arXiv preprint arXiv:2404.04932, 2024. [25] R. Yang, R. Ding, Y. Lin, H. Zhang, and T. Zhang, “Regularizing hidden states enables learning generalizable reward model for llms,” Advances in Neural Information Processing Systems, vol. 37, p. 62 279–62 309, 2024. [26] Y. Xue, “Pira: Preference-oriented instruction-tuned reward models with dual aggregation,” arXiv preprint arXiv:2511.20668, 2025. [27]S. Zhang, Z. Chen, S. Chen, Y. Shen, Z. Sun, and C. Gan, “Improving reinforcement learning from human feedback with efficient reward model ensemble, 2024,” URL https://arxiv. org/abs/2401.16635, 2024. [28]J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, and D. Ramachandran, “Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking,” arXiv preprint arXiv:2312.09244, 2023. [29]S. Quan, “Dmoerm: Recipes of mixture-of-experts for effective reward modeling,” arXiv preprint arXiv:2403.01197, 2024. [30]H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang, “Interpretable preferences via multi-objective reward modeling and mixture-of-experts,” arXiv preprint arXiv:2406.12845, 2024. [31]A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” arXiv preprint arXiv:2212.05055, 2022. [32]Y. Ding, J. Liu, Y. Wei, T. Y. Zhuo, and L. Zhang, “Xft: Unlocking the power of code instruction tuning by simply merging upcycled mixture- of-experts,” arXiv preprint arXiv:2404.15247, 2024. [33]F. Xue, X. He, X. Ren, Y. Lou, and Y. You, “One student knows all experts know: From sparse to dense,” arXiv preprint arXiv:2201.10890, 2022.

Instant research discovery