Paper deep dive
Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan
Models: 7B language model (unspecified base)
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:31:01 PM
Summary
The paper introduces 'easy-to-hard generalization' as a scalable alignment strategy, where AI models are trained on easy tasks (e.g., level 1-3 MATH problems) and then used to evaluate or supervise performance on harder tasks (e.g., level 4-5 MATH problems). The authors demonstrate that reward models (evaluators) trained on easy tasks can effectively guide generators on hard tasks through re-ranking or reinforcement learning, achieving significant performance improvements on the MATH500 benchmark.
Entities (5)
Relation Signals (3)
Easy-to-Hard Generalization → appliedto → MATH
confidence 95% · Specifically, we explore the efficacy and scalability of various easy-to-hard methodologies on competition-level mathematical problem-solving problems (MATH)
OPRM → combines → Process Reward Model (PRM)
confidence 95% · OPRM is trained on the mixed data of ORMs and PRMs.
Process Reward Model (PRM) → facilitates → Easy-to-Hard Generalization
confidence 92% · We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., level 1-3 MATH problems), which we term as easy-to-hard generalization. Our key insight is that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the (process-supervised) reward models on easy problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators either through re-ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL model and 34b model (reranking@1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests a promising path toward AI systems that advance beyond the frontier of human supervision.
Tags
Links
- Source: https://arxiv.org/abs/2403.09472
- Canonical: https://arxiv.org/abs/2403.09472
Trouble viewing inline? Open PDF directly →
Full Text
159,461 characters extracted from source content.
Expand or collapse full text
Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision Zhiqing Sun 1∗ , Longhui Yu 2∗ , Yikang Shen 3 , Weiyang Liu 4,5 , Yiming Yang 1† , Sean Welleck 1† , Chuang Gan 3,6† 1 Carnegie Mellon University, 2 Peking University, 3 MIT-IBM Watson AI Lab 4 University of Cambridge, 5 Max Planck Institute for Intelligent Systems, 6 UMass Amherst Code:Edward-Sun/easy-to-hard Abstract Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., level 1-3 MATH problems), which we term aseasy-to-hard generalization. Our key insight is that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the (process-supervised) reward models on easy problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that sucheasy-to-hard generalization from evaluatorscan enableeasy-to-hard generalizations in generatorseither through re-ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL model and 34b model (reranking@1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests a promising path toward AI systems that advance beyond the frontier of human supervision. 1 Introduction Rapid advancements in large language models (LLMs) indicate that in the near future, highly sophisticated AI systems could surpass human capabilities in certain areas, significantly enhancing our capabilities in solving harder problems beyond the levels we can currently solve [47,49]. Since the current AI alignment methods mostly rely on either supervised fine-tuning (SFT) with human- provided demonstrations [59,78,14] or reinforcement learning from human feedback (RLHF) [97,68,50], their capabilities would be inherently limited as humans cannot always provide helpful demonstrations or supervision on the hard tasks beyond their expertise [64]. In order to build future AI systems for tackling complex challenges, such as advancing scientific knowledge, it is crucial to develop new approaches forscalable oversightchallenge, i.e., to supervise the AI systems that can potentially outperform humans in most skills [9]. The key question is: •Can we limit human supervision to easier tasks, yet enable the model to excel in harder tasks? * Equal contributions as leading authors. † Equal contributions as senior authors. 38th Conference on Neural Information Processing Systems (NeurIPS 2024). arXiv:2403.09472v2 [cs.LG] 10 Dec 2024 Traditional Alignment 2x2=4 humans supervise strong models on hard tasks Scalable Alignment (Superalignment) 3^3^3 humans cannot reliably supervise superhuman models on the hardest tasks 3^3=? 3x3=? Burns’ Analogy on Weak-to-Strong Generalization 2x2=3? weak models unreliably supervise strong models on hard tasks that humans can evaluate Our Analogy on Easy-to-Hard Generalization 1+1=2 humans reliably supervise strong models on easy tasks and evaluate them on hard tasks 3x3=? 3x3=? 5^5=? Figure 1: Illustration of different alignment scenarios:traditional alignmentrelies on human demon- strations or judgements [50];scalable alignment[9] assumes that humans cannot reliably supervise smarter-than-human models;weak-to-strong generalization[11] focuses on using weak models with unreliable labels to supervise strong models; Our proposedeasier-to-general generalization focuses on the transfer of rewarding policies from weak models to harder tasks. We refer to this scenario asEasy-to-Hard Generalization[63,95,11,29]. This setting requires no human supervision on the harder tasks, which differs from existing work that either enhances humans’ ability to verify the outputs of AI systems [81,60,9,57] or enables weak-to-strong generalization via a teacher that only offers unreliable or noisy supervision [11]. The most basic form of easy-to-hard generalization can be achieved by training the policy models (i.e., generator) using supervised fine-tuning (SFT) or in-context learning (ICL) on easy tasks [55,10], and expect this will unlock the ability to perform well on hard tasks. However, it has been observed that SFT or ICL training of generators on easy tasks often fails to generalize to hard tasks [71,24,95]. We hypothesize and show that methods beyond these can enable stronger degrees of easy-to-hard generalization. Our intuition is guided by the observation thatevaluation is easier than generation[34,46], so an evaluator may offer a degree of easy-to-hard generalization that is useful for improving a generator. If that is true, we can first train a verifier on easy tasks, then make use of its generalization ability to supervise the generator on hard tasks. Complex tasks can often be broken down into smaller steps [95] and verified by validating the individual steps – a strategy that is commonly employed in solving mathematical problems [74,40,73]. Inspired by this, we train outcome-supervised and process-supervised reward models [74,85,75,40] as our easy-to-hard evaluators. The training dataset is often comprised of a set of labeled easy tasks, each with a question and a high-quality solution 1 , paired with a set of unlabeled hard tasks that are represented only by their questions. This simulates the practical setting of having numerous problems with known solutions, as well as significant unresolved challenges, such as the Millennium Prize Problems [12], which present challenging open problems. The pivotal aspect of easy-to-hard generalization thus lies in how we effectively leverage the capabilities of easier-level models in solving harder problems. Our investigation includes to training policy and reward models on the easy (i.e., level 1-3) portion of the PRM800K [40] dataset, and comparing the performance of majority voting with the policy model 1 We assume that human supervision is of high quality on the easy tasks in general. 2 Easy-to-Hard Generators Easy-to-Hard Evaluators Easy (train) : Domain of Human Supervision Hard (test): Domain Beyond Human Supervision Sampling Solutions Verifying Solutions trained on process supervision optimized against easy-to-hard evaluators Successful Generalization in Easy-to-Hard Evaluation Failed Generalization in Easy-to-Hard Generation Figure 2: We first train the evaluator with process supervision or outcome supervision (which simulates the process supervision) to enable easy-to-hard evaluation, and then use it to facilitate easy-to-hard generation via re-ranking or RL. only and weighted majority voting with the policy model and PRMs (Process-supervised Reward Models). We also introduce theOutcome & Process Reward Model (OPRM), which harnesses the complementary strengths of outcome reward models (ORMs) and process reward models (PRMs): judging if each step in reasoning is correct (like PRMs do) and deciding if the final answer is right (like ORMs do). Our findings reveal a marked performance improvement with the inclusion of reward models, especially on the hard (i.e., level 4-5) portion of the MATH500 test set. This improvement indicates that easier-level evaluators can maintain their effectiveness on harder tasks. We have similar observations in our experiments on the MetaMath dataset [86] and the Math-Shepherd dataset [75]. We further investigate the use of the easy-to-hard evaluator as a reward model in reinforcement learning, where the evaluator provides targeted, step-by-step guidance in solving hard problems. We have an intriguing finding thattraining with human supervision only on the easy tasks (i.e., training with Level 1-3 problems and answers) can outperform both SFT and Final-Answer RL training on the full dataset (Level 1-5). This finding underscores the potential of using easy-to-hard evaluation to improve easy-to-hard generators, particularly when dealing with varied levels of task complexity. 2 Related Work 2.1 Scalable Oversight While present-day models operate within the scope of human assessment, future, more advanced models may engage in tasks that are beyond human evaluation capabilities. This raises a concern that such models might prioritize objectives other than maintaining accuracy (Andreas3, Perez et al. 53, Sharma et al.64, Wei et al.80). To address this, a branch of research develops techniques to enhance the human capacity to supervise such models, such as via using AI to evaluate the work of other AIs [1,38,60,9]. Our setting differs from enhancing human oversight; instead, we focus on enabling models to excel in hard tasks where human supervision may not be available. This also differs from weak-to-strong generalization [11], where human supervision may be available, but not reliable, on hard tasks. However, our framework aligns with the “sandwiching” concept proposed for measuring progress in scalable oversight, which involves domain experts evaluating the outputs of AI-assisted non-experts [18, 9, 57]. 3 MATH-Easy (Level 1-3) Problems and Final Answers MATH-Hard (Level 4-5) Problems and Final Answers* Base Language Model Easy-to-Hard SFT Model In-Context Learning (ICL) Model Solution Samples MATH-Easy (Level 1-3) Solutions MATH-Easy (Level 1-3) Solution Samples and Process Labels Easy-to-Hard Reward Model Majority Voting Reranking: Weighted Voting, Best-of-N Reinforcement Learning: ReST-EM, DPO, PPO Figure 3: The overview diagram of our methods: the different components of modeling and training and how they are interconnected. 2.2 Compositional Generalization Compositional generalization is a fundamental aspect of how language works [13]. It refers to the ability to understand and utilize novel combinations based on the understanding of basic concepts and a limited number of their combinations [23]. Recently, least-to-most prompting [95,20] teaches language models how to solve a complex problem by reducing it to a series of easier sub-problems, achieving easy-to-hard generalization on semantic parsing tasks like SCAN [37] and CFQ [35] with perfect generalization accuracy. In addition, least-to-most prompting has also been successful in mathematical reasoning tasks, specifically in datasets like GSM8K [16] and DROP [21], by teaching language models to solve problems more difficult than those seen in the prompts. This success not only underscores the capacity of language models to effectively break down complex tasks into simpler sub-tasks Perez et al.[51], but also demonstrates their generalization capability in solving these sub-problems. 2.3 Easy-to-Hard Generalization Past work has evaluated easy-to-hard generalization by training easy-to-hard generators on easy tasks using supervised finetune-tuning (SFT) or in-context learning (ICL) [55,10]. Nevertheless, Swayamdipta et al.[71]showed that the BERT model performs poorly on common-sense reasoning when only trained on easy data. Fu et al.[24]showed similar results for ICL on reasoning tasks like GSM8K [17]. In concurrent work, Hase et al.[29]evaluate the performance of easy-to-hard generators on more datasets and models, and find that ICL or SFT on easy tasks is a strong baseline for multiple-choice tasks like ARC [15] and MMLU [30]. In contrast, we evaluate the easy-to-hard generation performance on the more challenging MATH dataset [32], and show that easy-to-hard evaluation can improve a generator’s easy-to-hard generalization beyond ICL and SFT. Iterative machine teaching [43] gives theoretical justification to show that training classifiers from easy to hard examples yield better generalization. 3 Methodology We study the easy-to-hard generalization problem: how can we enable capabilities beyond human su- pervision? Specifically, we explore the efficacy and scalability of various easy-to-hard methodologies on competition-level mathematical problem-solving problems (MATH; Hendrycks et al.32). This dataset is suitable for our study since it explicitly categorizes problems across five difficulty levels. We consider levels 1-3 as “easy” tasks, encompassing both the problems and their respective solution demonstrations, along with the correct answers. Conversely, levels 4-5, characterized by their more complex nature, are treated as “hard” tasks and are represented solely by their questions. The MATH 4 dataset’s difficulty distribution roughly follows a1 : 2 : 2 : 3 : 3ratio across levels 1 to 5. So our division maintains a balanced number of easy and hard tasks. The remainder of the paper aims to answer following research questions: RQ1:How do generators generalize from easy to hard? RQ2:How do evaluators generalize from easy to hard? RQ3:If evaluators generalize better than generators, how can we take advantage of this to enable stronger easy-to-hard generalization in generators? 3.1 Setup DatasetMATH [32] is a dataset of 12,500 challenging competition mathematics problems, where 7,500 of them are training problems and 5,000 are originally used for testing. Following Lightman et al. [40], Wang et al.[75], we use the identical subset of 500 representative problems (i.e., MATH500) as our test set, uniformly sample another 500 problems for validation, across all five difficulty levels, and leave the rest 4,000 MATH test split problems combined with the original 7,500 MATH training split problems as our training set. Simulated Human DemonstrationsWhile the original MATH dataset provides full step-by-step solutions, these solutions typically skip many chain-of-thought steps [79], which can be hard for language models to directly imitate 2 . Instead, we consider filtered PRM800K [40] and MetaMATH [86] as our SFT training data: the former is generated by a Minerva-style base GPT-4 model using few-shot prompting after filtering the correct answers [39,48], while the latter is generated by ChatGPT [47]. We keep all the GSM8K data in the MetaMATH dataset since they are typically easier than the problems in MATH. PRM800K comes with human annotated process labels, while for MetaMath, we use Math-Shepherd as the corresponding process labels [75]. 3.2 Generators For a given dataset (e.g., a variant of MATH), we consider the following generator models: Full & Hard ICLFull in-context learning (ICL) is a base model prompted with exemplars sampled from all difficulty levels, or only from the level 5 [24]. Easy-to-Hard ICLThis model is prompted with exemplars from easy problems. This baseline evaluates the degree to which a model can solve problems more difficult than those seen in the prompts [95]. Full SFTAs prior work suggests that finetuning should outperform prompting alone [68,52,50], the full supervised fine-tuning (SFT) model is typically considered as a ceiling that a model can achieve on a type of task [11, 29]. Easy-to-Hard SFTThis generator model is trained only on the easy tasks. Prior work suggests that it can generalize to hard tasks but with some degeneration in performance [71]. The generator models are evaluated in greedy decoding and self-consistency (also known as majority voting) settings [76]. 3.3 Evaluators Similarly, we consider the following evaluator models that can be trained either on the easy tasks only, or on the full dataset. Notably, unlike final-answer rewards, reward models trained on easy tasks can be applied to evaluate solutions to hard problems. Final-Answer Rewardis a symbolic reward that provides a binary reward based on the accuracy of the model’s final answer. The matching is performed after normalization 3 . 2 Hendrycks et al.[32]found that having models generate MATH-style step-by-step solutions before producing an answer actually decreased accuracy. 3 https://github.com/openai/prm800k/blob/main/prm800k/grading/grader.py 5 Table 1: Easy-to-hard generalization of generators. We compare generator performance under various decoding settings.PRM800KandMETAMATHindicate the SFT training data and ICL exemplars. Evaluations are performed on the same MATH500 test set. PRM800KMETAMATH GREEDYMAJ@16MAJ@256GREEDYMAJ@16MAJ@256 LLEMMA-7B FULLICL12.815.620.816.418.425.6 HARDICL12.618.027.016.619.027.0 EASY-TO-HARDICL14.017.624.414.217.426.8 FULLSFT20.632.036.231.440.241.6 EASY-TO-HARDSFT19.831.636.030.038.642.4 LLEMMA-34B FULLICL18.623.636.020.628.839.2 HARDICL15.821.434.221.826.438.6 EASY-TO-HARDICL18.225.236.819.826.837.2 FULLSFT25.641.846.435.444.245.6 EASY-TO-HARDSFT24.840.846.032.242.643.4 Outcome Reward Model (ORM)is trained on the Final-Answer rewards. Following Cobbe et al. [16], Uesato et al.[74], Lightman et al.[40], we train the reward head to predict on every token whether the solution is correct, in a similar sense to a value model [85]. At inference time, we use the ORM’s prediction at the final token as the reward of the solution. Process Reward Model (PRM)is trained to predict whether each step (delimited by newlines) in the chain-of-thought reasoning path is correct. The labels are usually labeled by humans [74,40] or estimated with rollouts [65, 75]. Outcome & Process Reward Model (OPRM)Building on the distinct advantages of ORMs and PRMs, we introduce theOutcome & Process Reward Model (OPRM), which harnesses the comple- mentary strengths of both. OPRM is trained on the mixed data of ORMs and PRMs. Specifically, it evaluates the correctness of each intermediate reasoning step, akin to PRMs, while also assesses the overall solution’s accuracy at the final answer stage, mirroring the functionality of ORMs. 3.4 Optimizing Generators Against Evaluators Finally, given a generator model (i.e., policy model) and a evaluator model (i.e., reward model; RM), we optimize the generator against the evaluator using either re-ranking or reinforcement learning. Best-of-n(BoN), also known as rejection sampling, is a reranking approach that sample multiple solutions from the generator and selects one with the highest RM score. Weighted Votingis similar to majority voting or self-consistency [76], but weights each solution according to its RM score [74]. Reinforcement Learning (RL)We consider three online/offline RL variants, Reinforced Self- Training (ReST) [28,67], Direct Policy Optimization (DPO) [56], and Proximal Policy Optimization (PPO) [62]. Due to the space limit, please find their detailed description in Appendix B. 3.5 Evaluation Metrics In this study, we have chosen not to establish terms analogous to the weak-to-strong performance gap recovery (PGR) as discussed in Burns et al.[11]or the easy-to-hard supervision gap recovery (SGR) highlighted by Hase et al.[29]. This decision is based on our observations that sometimes, models trained exclusively on simpler tasks—particularly when employing RL training—can outperform those trained across the entire spectrum of problem difficulties. Therefore, we mainly focus on the absolute and relative performance of generators (optionally optimized by the evaluator) on the MATH500 test set [40]. 3.6 Implementation Details Base Language ModelLlemma is a large language model for mathematics [6], which is continue pre- trained from Code Llama [58] / LlaMA-2 [72]. We use both 7b and 34b variants in our experiments. 6 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 27.84 29.02 36.19 43.88 49.84 53.21 55.36 56.58 57.03 57.17 56.76 28.23 35.31 41.63 47.07 51.72 55.36 58.29 60.63 62.07 62.80 63.20 27.87 35.66 41.69 46.07 49.23 51.38 52.86 53.87 54.65 55.03 55.06 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 % Problems Solved 6.84 7.56 9.20 11.60 14.14 15.75 17.03 17.94 18.38 18.92 19.19 6.93 9.32 11.41 13.69 15.69 17.70 19.25 20.89 22.39 23.33 24.12 6.80 9.48 11.84 13.92 15.26 16.50 17.66 18.70 19.38 19.95 19.91 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 16.87 17.78 22.07 26.96 31.09 33.56 35.27 36.33 36.78 37.12 37.10 17.07 21.67 25.84 29.57 32.85 35.60 37.86 39.75 41.27 42.09 42.74 16.81 21.93 26.04 29.20 31.41 33.08 34.42 35.43 36.17 36.62 36.66 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.52 38.93 47.59 56.06 61.96 65.69 67.52 68.56 68.89 68.85 68.51 37.47 46.88 54.25 60.14 64.96 68.36 71.00 72.64 73.61 74.08 74.37 37.79 46.97 54.16 59.83 63.40 65.88 67.57 68.19 68.24 68.04 67.44 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.19 10.86 13.69 17.25 20.59 23.09 24.75 25.60 26.09 26.37 26.41 10.34 13.87 17.18 20.27 23.04 25.42 27.34 29.16 30.50 31.61 32.58 10.31 14.11 17.40 20.31 22.30 24.04 24.97 25.73 26.16 26.68 27.02 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.21 24.22 29.80 35.71 40.28 43.34 45.16 46.07 46.52 46.62 46.53 23.26 29.58 34.83 39.21 42.99 45.84 48.17 49.87 50.98 51.81 52.45 23.41 29.76 34.88 39.10 41.86 43.94 45.26 45.97 46.25 46.37 46.31 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on All (Level 1-5) Problems Figure 4: Easy-to-hard generalization of 7b (upper) and 34b (lower) evaluators. Both SFTs and RMs are trained on the easy data. We found that PRMs trained on easy tasks can significantly improve the re-ranking (i.e., weighted voting) performance on hard tasks. The shaded margin of the curve plot in this paper represents the performance variance. SFT / RL / Reward ModelWe fine-tune all models in full fine-tuning with frozen input-output embedding layers and normalization layers. RMs are initialized from the base model, and have an added scalar head to output the reward. In PPO training, we initialize the value model from the reward model. Hyper-parametersDue to the space limit, our training hyper-parameters can be found in Ap- pendix. C. 4 Main Results 4.1 Easy-to-Hard Generalization of Generators In Table 1, we compare the easy-to-hard generalization performance of the generators under various decoding settings: Supervised Fine-Tuning (SFT) outperforms In-Context Learning (ICL):This is consistent with prior work [68,50,74]. We also find that the performance of ICL has larger variance than SFT with respect to data ordering (or random seeds) [19, 93]. SFT data quality impacts easy-to-hard generalization:PRM800K data is generated by a base (unaligned) GPT-4 model through few-shot prompting and is thus of lower quality than well-aligned ChatGPT-generated MetaMATH data. We find that only MetaMath-trained models have certain easy-to-hard gaps (e.g., 16.6 v.s. 14.2 in MetaMath-7b-ICL), while such gaps in PRM800K-trained models are very small (less than 1%), or even inverted in the ICL setting. We hypothesize that low-quality SFT data may only teach the model the format of the task [59,78,76], while high-quality (imitation) SFT data can teach the model the principles of solving the task [70,27]. Nevertheless, the strongest performance is achieved by full SFT on the high-quality MetaMath data (35.4), showing an unignorable difference, with a gap of up to 3.2, compared to its easy-to-hard SFT counterpart (32.2). 4.2 Easy-to-Hard Generalization of Evaluators The primary metric we use to assess the effectiveness of our process reward model is not the average accuracy of verifying each step in a solution but rather the overall performance achieved through re-ranking methods (See discussion in Sec. 3.5). We first use re-ranking to evaluate the easy-to-hard generalization performance of evaluators. 7 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 70 % Problems Solved 27.84 29.02 36.19 43.88 49.84 53.21 55.36 56.58 57.03 57.17 56.76 28.22 36.18 43.20 49.09 54.09 57.74 60.77 63.07 64.65 66.05 66.54 27.97 37.00 44.01 49.80 54.47 57.62 59.90 61.68 62.66 63.52 64.38 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 % Problems Solved 6.84 7.56 9.20 11.60 14.14 15.75 17.03 17.94 18.38 18.92 19.19 6.83 9.52 12.22 14.82 17.35 19.40 21.50 23.54 25.25 26.83 27.81 6.84 9.83 12.90 15.65 17.91 19.89 21.50 23.19 24.45 25.42 26.00 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 50 % Problems Solved 16.87 17.78 22.07 26.96 31.09 33.56 35.27 36.33 36.78 37.12 37.10 17.03 22.20 26.97 31.13 34.83 37.67 40.16 42.32 44.04 45.49 46.27 16.93 22.74 27.70 31.86 35.34 37.84 39.77 41.50 42.63 43.55 44.27 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.52 38.93 47.59 56.06 61.96 65.69 67.52 68.56 68.89 68.85 68.51 37.72 45.99 52.58 58.04 62.63 66.03 68.57 70.42 71.42 72.28 72.68 37.50 46.31 52.58 56.84 59.29 60.92 62.13 62.87 63.27 63.60 63.46 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.19 10.86 13.69 17.25 20.59 23.09 24.75 25.60 26.09 26.37 26.41 10.23 13.33 16.33 18.99 21.44 23.58 25.64 27.41 28.80 30.06 30.68 10.39 13.59 16.61 18.71 20.17 21.44 22.59 23.32 24.04 24.33 24.38 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.21 24.22 29.80 35.71 40.28 43.34 45.16 46.07 46.52 46.62 46.53 23.31 28.86 33.58 37.59 41.04 43.77 46.12 47.93 49.10 50.15 50.67 23.27 29.16 33.69 36.86 38.75 40.20 41.41 42.11 42.69 43.02 42.96 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on All (Level 1-5) Problems Figure 5: Easy-to-hard generalization of evaluators applied to generators of different sizes. We evaluated 7b generator + 34b evaluator (upper) and 34b generator + 7b evaluator (lower). Both SFTs and RMs are trained on the easy data. 4.2.1 Re-ranking We consider two re-ranking strategies: Best-of-n(or rejection sampling) and Weighted Voting. In our easy-to-hard generalization setting, both SFT models and Reward Models (RMs) are trained on easier tasks (levels 1-3), but evaluated on all difficulty levels (1-5). We compare the performance between majority voting (SFT only) and re-ranking (SFT + OPRM) on the PRM800K dataset in Figure 4-5, and the performance of different reward models (PRMs, ORMs, & OPRMs) on the PRM800K dataset in Figure 8-9. Specifically, we useminas the reward aggregation function for best-of-nandprodfor weighted voting 4 . The figures illustrate the performance of different decoding strategies or reward models under the same number of sampled solutions per problem. We have the following findings: OPRMs outperforms ORMs and PRMsThis confirms our hypothesis that Process Reward Models (PRMs) and Outcome Reward Models (ORMs) capture different aspects of task-solving processes. By integrating the strengths of both PRMs and ORMs, Outcome & Process Reward Models (OPRMs) demonstrate superior performance. However, follow-up experiments conducted on the MetaMath/Math-Shepherd datasets do not demonstrate significant improvements from incorporating additional ORM training examples. This lack of enhancement may be attributed to the fact that Math-Shepherd is already generated from final-answer rewards. This suggests that there remains a substantial difference between process rewards labeled by humans (e.g., PRM800K) and those generated automatically (e.g., Math-Shepherd). Weighted voting outshines Best-of-nThis finding diverges from past research where minimal performance differences were observed between weighted voting and Best-of-n[40,74]. Our hypothesis is that this discrepancy arises from our specific experiment, which involves training a less powerful base model (Llemma; Azerbayev et al.6) on more difficult tasks (MATH; Hendrycks et al.32). This setup might diminish the effectiveness of the reward model, potentially leading to an over-optimization of rewards [25]. Given these insights, weighted voting is preferred as the primary re-ranking method for further discussions. Nevertheless, Best-of-nstill achieves competitive performance to majority voting when producing only one full solution. In Figure 5, we also find that the 34b evaluator can significantly improve the 7b generator, while the 7b evaluator can still improve the performance of the 34b generator. Greater effectiveness of re-ranking on harder tasks:Weighted voting not only consistently surpasses majority voting but also shows a more pronounced advantage on harder tasks. This 4 See more detailed analysis of reward aggregation functions in Appendix. L. 8 Table 2: Comparing reinforcement learning (RL) approaches for easy-to-hard generalization. All methods are of 7b size and evaluated with greedy decoding. RL DATA REWARDACCURACY FINAL-ANSWERPROCESSRMEASY(LEVEL1-3)HARD(LEVEL4-5)ALL (SFT / PRM trained on level 1-3 of PRM800K) SFT28.212.219.8 REST-EMEASYEASY×33.212.622.4 ITERATIVEDPOEASYEASY √ 42.012.226.4 PPOEASYEASY×42.014.127.4 PPOALLEASY √ 45.414.929.4 (SFT / PRM trained on level 1-5 of MetaMath / Math-Shepherd) LLEMMA-BASEDSFT SOTA (OURS)51.713.731.4 PREVIOUSRL SOTA [75]--33.0 (SFT / PRM trained on level 1-3 of MetaMath / Math-Shepherd) SFT44.114.928.8 REST-EMEASYEASY×50.414.531.6 ITERATIVEDPOEASYEASY √ 53.816.034.0 ITERATIVEDPOALLEASY √ 49.610.729.2 PPOEASYEASY×50.8 15.332.2 PPOALLEASY √ 53.816.034.0 observation leads to the conclusion thatevaluators demonstrate better easy-to-hard generalization capabilities in comparison to generators. This motivates us to explore RL approaches that optimize the generator against the evaluator to further improve the performance of easy-to-hard generation. 4.2.2 Reinforcement Learning (RL) Given the conclusion above, an important question arises: how can evaluators once again assist generators in achieving enhanced easy-to-hard generalization capabilities? We further investigate the enhancement of policy models through RL, utilizing easy-to-hard evaluators as reward models. Similar to re-ranking, SFT and PRM are only trained on easy data. For a fair comparison between PRM800K and MetaMath, we only use vanilla PRMs in the RL training. All the RL methods use the validation accuracy for selecting the best checkpoint 5 . Our comparison spans offline (ReST & DPO) and online (PPO) RL algorithms under two training conditions: Easy Questions & Easy Final Answers.The SFT model samples from easy questions and receives the corresponding Final-Answer and optional PRM rewards. All Questions & Easy Final Answers.This assumes access to a range of easy and hard problems for RL training, with rewards for hard tasks solely provided by the easy-to-hard evaluator. Based on the results reported in Table 2, we have the following findings: DPO and PPO excel over ReST.Among the RL algorithms trained on the PRM800K dataset, PPO emerges as the most effective, significantly surpassing both ReST and DPO. On the MetaMATH dataset, PPO and DPO achieve top performance, while ReST shows only marginal improvements over the SFT baseline. The comparative analysis between DPO and PPO across the PRM800K and MetaMATH datasets indicates that while DPO’s efficacy is on par with PPO given a high-quality SFT model as initialization, PPO’s effectiveness is less contingent on the quality of the underlying SFT model [50, 56]. PRM rewards are more beneficial than Final-Answer rewards for hard tasks.Notably, models trained with PRM rewards with human supervision on the easy tasks (achieving a top performance of 34.0) outperform the previous state-of-the-art model trained across all task levels (33.0). This highlights the effectiveness of leveraging easy-to-hard evaluations to improve generator performance across varying task difficulties. 5 This includes stopping iterations in ReST-EM and iterative DPO, and stopping online steps in PPO. 9 Table 3: Easy-to-hard generalization of evaluators on coding problems (APPS). Both SFTs and RMs are trained on the easy (Introductory) data. We found that ORMs trained on easy tasks can improve the re-ranking (Best-of-N) performance on hard (Interview & Competition) coding problems. SFT / ORM DECODING AVERAGEACCURACY(%)STRICTACCURACY(%) TRAINDATAINTRO.INTER.COMP.ALLINTRO.INTER.COMP.ALL CODELLAMA- 7B ALLGREEDY31.415.512.218.017.02.32.05.2 EASYGREEDY26.814.19.515.711.03.00.04.0 EASYBEST-OF-125.412.00.113.516.02.70.04.8 EASYBEST-OF-427.113.88.115.314.04.00.05.2 EASYBEST-OF-1629.716.311.318.019.05.03.07.4 CODELLAMA- 34B ALLGREEDY37.619.911.321.722.05.02.07.8 EASYGREEDY33.919.48.520.121.06.01.08.0 EASYBEST-OF-128.514.54.415.321.03.30.06.2 EASYBEST-OF-436.321.310.522.124.08.71.010.2 EASYBEST-OF-1645.925.810.026.630.010.73.013.0 4.3 Easy-to-Hard Generalization on the Coding Domain We conduct further experiments in the coding domain with the APPS dataset [31]. Similarly to Lightman et al.[40], we sub-sampled 500 questions from the original test set of APPS as our test set. Specifically, we sub-sampled 100 Introductory questions, 300 Interview questions, and 100 Competition questions, following the original distribution in the test set. In Table 3, we compare the performance of SFT-trained Code Llama [58] (7b & 34b) with greedy decoding and best-of-N approach. In the latter, an Outcome Reward Model (ORM) of the same model size is trained to select the best coding one from N sampled solutions. We found that while the reward model is only trained on the outcome supervision of easy (Introduc- tory) data, it significantly improves the model performance on hard (Interview & Competition) data. These findings extend the premise of easy-to-hard generalization beyond the confines of mathematical reasoning, suggesting its applicability across diverse domains. 5 Conclusion Our study advances the field of AI alignment by demonstrating the potential of easy-to-hard gen- eralization, where models trained on simpler tasks can be guided to solve more complex problems without direct human supervision on these harder tasks. Through the use of (process-supervised) reward models for evaluating and enhancing policy models, we show that evaluators can facilitate this form of generalization, outperforming traditional training methods. Our findings highlight the effectiveness of re-ranking strategies and reinforcement learning (RL) in leveraging evaluators for performance gains on difficult tasks. This approach presents a promising direction for developing AI systems capable of surpassing human problem-solving capabilities, suggesting a scalable alignment method that could enable AI to independently advance knowledge in complex domains. While our study provides valuable insights into easy-to-hard generalization and the potential of process-supervised reward models, there are limitations to consider. These include the focus on specific model sizes and datasets, the domain specificity of reasoning tasks, and the need for further research on the long-term implications and robustness of the method. 6 Acknowledgement This work is supported by OpenAI Superalignment Fast Grants and Microsoft Accelerate Foundation Models Research (AFMR) Initiative. Additionally, ZS thanks Google PhD Fellowship; SW thanks NSF SCALE (NSF DMS 2134012) and Convergent Research. References [1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016. 3 10 [2]Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes llm better reasoner.arXiv preprint arXiv:2310.20689, 2023. 18 [3]Jacob Andreas. Language models as agent models. InFindings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 5769–5779, 2022. 3 [4]Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search.Advances in neural information processing systems, 30, 2017. 18 [5]Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences.arXiv preprint arXiv:2310.12036, 2023. 18 [6]Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics.arXiv preprint arXiv:2310.10631, 2023. 6, 8, 18, 24 [7]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022. 18 [8]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. 18 [9]Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models.arXiv preprint arXiv:2211.03540, 2022. 1, 2, 3 [10]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020. 2, 4 [11]Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023. 2, 3, 5, 6 [12]James A Carlson, Arthur Jaffe, and Andrew Wiles.The millennium prize problems. American Mathematical Soc., 2006. 2 [13] Noam Chomsky. On the representation of form and function. 1981. 4 [14]Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models.arXiv preprint arXiv:2210.11416, 2022. 1 [15]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018. 4 11 [16]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 4, 6, 18 [17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 4 [18] Ajeya Cotra. The case for aligning narrowly superhuman models. InAI Alignment Forum, 2021. 3 [19] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. 7 [20] Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. InThe Eleventh International Conference on Learning Representations, 2022. 4 [21]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1246. URLhttps://aclanthology. org/N19-1246. 4 [22]Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback.arXiv preprint arXiv:2305.14387, 2023. 20 [23]Jerry A Fodor and Ernest Lepore.The compositionality papers. Oxford University Press, 2002. 4 [24]Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. InThe Eleventh International Conference on Learning Representations, 2022. 2, 4, 5, 18 [25]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. InInternational Conference on Machine Learning, pages 10835–10866. PMLR, 2023. 8 [26]Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving.arXiv preprint arXiv:2309.17452, 2023. 18 [27]Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms.arXiv preprint arXiv:2305.15717, 2023. 7 [28]Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998, 2023. 6, 18 [29]Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks.arXiv preprint arXiv:2401.06751, 2024. 2, 4, 5, 6 [30]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2020. 4 [31]Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938, 2021. 10 12 [32]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 4, 5, 8, 18, 24 [33]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023. 18 [34]Richard M Karp. On the computational complexity of combinatorial problems.Networks, 5(1): 45–68, 1975. 2 [35]Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashu- bin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generalization: A comprehensive method on realistic data. InInternational Conference on Learning Representations, 2019. 4 [36]Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.arXiv preprint arXiv:2205.11916, 2022. 18 [37]Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. InInternational conference on machine learning, pages 2873–2882. PMLR, 2018. 4 [38]Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction.arXiv preprint arXiv:1811.07871, 2018. 3 [39]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models.arXiv preprint arXiv:2206.14858, 2022. 5, 18 [40]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023. 2, 5, 6, 8, 10, 18, 19, 30 [41] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, 2017. 18 [42]Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241, 2023. 18 [43]Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative machine teaching. InInternational Conference on Machine Learning, pages 2149–2158. PMLR, 2017. 4 [44] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023. 18 [45] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback.arXiv preprint arXiv:2312.00886, 2023. 18 [46] Moni Naor. Evaluation may be easier than generation. InProceedings of the twenty-eighth annual ACM symposium on Theory of computing, pages 74–83, 1996. 2 [47] OpenAI. OpenAI: Introducing ChatGPT, 2022. URLhttps://openai.com/blog/chatgpt. 1, 5 13 [48] OpenAI. Gpt-4 technical report, 2023. 5, 18 [49] OpenAI. OpenAI: GPT-4, 2023. URLhttps://openai.com/research/gpt-4. 1 [50]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.arXiv preprint arXiv:2203.02155, 2022. 1, 2, 5, 7, 9, 18, 20 [51]Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864–8880, 2020. 4 [52]Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:11054–11070, 2021. 5 [53]Ethan Perez, Sam Ringer, Kamil ̇ e Lukoši ̄ ut ̇ e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations.arXiv preprint arXiv:2212.09251, 2022. 3 [54] Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020. 18 [55]Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. 2, 4 [56]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. 6, 9, 18, 20 [57]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023. 2, 3 [58]Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023. 6, 10 [59]Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. InInternational Conference on Learning Representations, 2021. 1, 7 [60] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators.arXiv preprint arXiv:2206.05802, 2022. 2, 3 [61]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015. 20 [62]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. 6, 18 [63]Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks.Advances in Neural Information Processing Systems, 34:6695–6706, 2021. 2 [64]Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models.arXiv preprint arXiv:2310.13548, 2023. 1, 3 14 [65]David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search.Nature, 529(7587):484–489, 2016. 6 [66] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017. 18 [67] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models.arXiv preprint arXiv:2312.06585, 2023. 6, 18, 19, 20 [68]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. 1, 5, 7, 18 [69]Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with principle-following reward models.arXiv preprint arXiv:2310.05910, 2023. 18 [70]Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision.arXiv preprint arXiv:2305.03047, 2023. 7 [71]Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293, 2020. 2, 4, 5 [72] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 6, 18 [73] Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations.Nature, 625(7995):476–482, 2024. 2 [74] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022. 2, 6, 7, 8, 18, 29 [75]Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations.CoRR, abs/2312.08935, 2023. 2, 3, 5, 6, 9, 19, 20, 26 [76] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions.arXiv preprint arXiv:2212.10560, 2022. 5, 6, 7 [77] Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision.arXiv preprint arXiv:2402.02658, 2024. 30 [78] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021. 1, 7 [79]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.NeurIPS, 2022. 5, 18 15 [80]Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. Simple synthetic data reduces sycophancy in large language models.arXiv preprint arXiv:2308.03958, 2023. 3 [81] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback.arXiv preprint arXiv:2109.10862, 2021. 2 [82]Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss.arXiv preprint arXiv:2312.16682, 2023. 18 [83] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022. 18 [84]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.arXiv preprint arXiv:2305.10601, 2023. 18 [85] Fei Yu, Anningzhe Gao, and Benyou Wang. Outcome-supervised verifiers for planning in mathematical reasoning.arXiv preprint arXiv:2311.09724, 2023. 2, 6 [86]Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023. 3, 5, 18, 19, 20, 26 [87] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023. 18 [88]Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023. 18 [89]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022. 18, 19 [90]Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models.arXiv preprint arXiv:2210.03493, 2022. 18 [91]Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. InThe Eleventh International Conference on Learning Representations, 2022. 18 [92] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023. 18 [93]Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. InInternational Conference on Machine Learning, pages 12697–12706. PMLR, 2021. 7 [94]Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompt- ing improves reasoning in large language models.arXiv preprint arXiv:2304.09797, 2023. 18 [95]Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most prompting enables complex reasoning in large language models. InThe Eleventh International Conference on Learning Representations, 2022. 2, 4, 5, 18 [96]Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers.arXiv preprint arXiv:2211.01910, 2022. 18 16 [97]Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 1 17 A Additional Related Work A.1 Rationale-Augmented (Mathematical) Reasoning Ling et al.[41]pioneer the work of solving math word problems by generating step-by-step solutions before the final answer. Cobbe et al.[16]extend this work by constructing a much larger dataset to finetune a pre-trained large language model to solve math word problems, and a outcome-supervised verifier to rank candidate solutions. Wei et al.[79]demonstrate that the reasoning ability of a language model can be elicited through the use of prefixed rationales. Subsequent research [36,83,39,96, 84] in tasks requiring human-level reasoning skills has also highlighted the efficacy of rationale augmentation. Among all the reasoning tasks, we select mathematical reasoning to evaluate easy-to-hard generaliza- tion ability, given that mathematical reasoning serves as a valuable assessment for complex reasoning abilities and features a clear delineation of difficulty levels. Recent research efforts focus on prompt design [79,95,24,90,94] to elicit the intrinsic reasoning capabilities of models, or data engineering for fine-tuning [44,87,88,86,26,42,2,6], which draws on experts to provide high-quality training datasets. Our work is categorized as fine-tuning based work. However, unlike previous work, our focus lies in exploring how to generalize to more challenging mathematical problems when only provided with easy mathematical data. A.2 Outcome Reward Models & Process Reward Models For some multi-step complex reasoning tasks, such as generating highly complex code, it may be challenging for humans to fully grasp the outputs produced by an advanced AI system. In such scenarios, process-supervised reward models (PRMs) present a promising solution [74,40]. These models operate by supervising each step in the reasoning or generation process, rather than focusing solely on the end result. They are particularly effective in tasks where the reasoning process itself is as important as the final outcome [32, 33]. Uesato et al.[74]find that process-supervised reward models (PRMs) achieve better performance than outcome-supervised reward models (ORMs) when re-ranking sampled solutions from the policy model, but their performance is similar during reinforcement learning (RL) via expert iteration [66,4,54,89,28,67]. Lightman et al.[40]compare ORMs and PRMs with a more capable base model [48] and significantly more human-labeled process feedback on the more challenging MATH dataset, and also find that PRMs significantly outperform ORMs in the reranking setting. In contrast to these works, which only study the effectivenss of PRM in an independent and identically distributed (IID) domain, we study the utilization of PRMs in the easy-to-hard generalization scenario, and show that easy-to-hard evaluators instantiated by PRMs can enable easy-to-hard generation of policy models. B Reinforcement Learning Algorithms Reinforced Self-Training (ReST)is an offline RL algorithm, which alternates between generating samples from the policy, which are then used to improve the LLM policy with RM-weighted SFT [28, 67]. Its variants include expert iteration [4] and rejection sampling fine-tuning [72, 87]. Direct Policy Optimization (DPO)is a class of offline RL algorithms [56] that consider both positive and negative gradient updates. It fine-tunes the policy model on a preference dataset consisting of paired positive and negative samples. The variants include NLHF [45], IPO [5], and SLiC [91,92]. Recent work shows that iteratively applying DPO leads to improved performance [82]. Proximal Policy Optimization (PPO)is an online RL algorithm which samples from the policy during fine-tuning [62]. It is widely used in RLHF [68, 7, 50] and RLAIF [8, 69]. 18 C Hyper-parameters C.1 Supervised Fine-Tuning & Reward Modeling For the PRM800K dataset [40], the SFT model is trained using steps that are labeled as correct. For the MetaMath dataset [86], given that the original dataset can contain upwards of ten solutions for the same question, potentially leading to over-fitting, we implement a filtering process. This process ensures that, during any given epoch, no more than three solutions per question are retained, thereby mitigating the risk of over-fitting. The PRMs are trained on the corresponding released dataset [40,75]. For generating solutions to train ORMs, we sample 32 solutions for each question from the language model using top-K sampling with K=20 and temperature of 0.7. We also ensure that the ratio between positive and negative samples for each question is between 1:3 to 3:1. See Table 4 for a list of training hyper-parameters used in the training jobs. We use full fine-tuning for all SFT/RM training. Table 4: Hyper-parameters in our SFT/RM training jobs PRM800KMETAMATH SFTPRMORMOPRMSFTPRM LLEMMA-7B LEARNINGRATE2E-52E-52E-52E-58E-62E-5 EPOCHS322232 BATCHSIZE128128128128128128 MAXSEQLEN768768102410241024768 DTYPEBF16BF16BF16BF16FP32BF16 LLEMMA-34B LEARNINGRATE1E-51E-51E-51E-55E-6- EPOCHS32223- BATCHSIZE128128128128128- MAXSEQLEN76876810241024768- DTYPEBF16BF16BF16BF16FP32- C.2 Re-Ranking For majority voting, weighted voting, and best-of-n, we sample from the language model using top-K sampling with K=20 and temperature of 0.7. At test time, we use the ORM’s prediction at the final token as the overall score for the solution, and use the PRM’s prediction at each intermediate step (denoted by the new line symbol) and the final token as the process reward scores. C.3 Reinforcement Learning We use full fine-tuning during the RL stage. ReST-EMFollowing Singh et al.[67], we sample 32 solutions for each question from the language model using top-K sampling with K=40. We also used a cut-off threshold of 10 for the maximum number of solutions per problem [89,67]. We performed iterative ReST training for two epochs, and observed performance degeneration starting from the third epoch. For PRM800K, we used a temperature of 1.0, while for MetaMath, we used a temperature of 1.2. The rest training hyper- parameters are the same as in SFT training. Iterative DPOWe sample 8 solutions for each question from the language model using top-K sampling with K=20 and temperature of 1.0. We use the process reward model to assign a score between 0 and 1 to each solution, and use final-answer reward to assign an additional 0/1 score to each solution. A preference training pair is constructed only when the score difference between positive and negative solutions is greater than 1.0. We used a cut-off threshold of 3 for the maximum number of preference pairs per problem. 19 Table 5: Full results of comparing reinforcement learning (RL) approaches for easy-to-hard gener- alization. All methods are of 7b size and evaluated with greedy decoding.† indicates the model is trained with additional final-answer labels on hard tasks (similar to Singh et al.[67]), which is not strictly a easy-to-hard generalization setup. RL DATA REWARDACCURACY FINAL-ANSWERPROCESSRMEASY(LEVEL1-3)HARD(LEVEL4-5)ALL (SFT / PRM trained on level 1-3 of PRM800K) SFT28.212.219.8 REST-EMEASYEASY×33.212.622.4 REST-EMHARDHARD×31.98.019.4 REST-EM†ALLALL×35.78.821.6 ITERATIVEDPOEASYEASY √ 42.012.226.4 ITERATIVEDPO†ALLALL √ 38.211.524.2 PPOEASYEASY×42.0 14.127.4 PPOHARDHARD×34.09.221.0 PPO†ALLALL×42.010.725.6 PPOALLEASY √ 45.414.929.4 (SFT / PRM trained on level 1-5 of MetaMath / Math-Shepherd) LLEMMA-BASEDSFT SOTA (OURS)51.713.731.4 PREVIOUSRL SOTA [75]--33.0 (SFT / PRM trained on level 1-3 of MetaMath / Math-Shepherd) SFT44.114.928.8 REST-EMEASYEASY×50.414.531.6 ITERATIVEDPOEASYEASY √ 53.816.034.0 ITERATIVEDPOALLEASY √ 49.610.729.2 ITERATIVEDPO†ALLALL √ 47.912.229.2 PPOEASYEASY×50.8 15.332.2 PPO†ALLALL×50.813.431.2 PPOALLEASY √ 53.816.034.0 For all DPO training [56], we used a learning rate of2×10 −6 , a batch size of 64, and a DPO training epoch of 1. We setβ= 0.1for all DPO experiments, and performed at most 5 DPO iterations (i.e., sampling new solutions and performing one DPO epoch). PPOWe follow Dubois et al.[22]on the implementation of the PPO algorithm, which is a variant of [50] 6 . Specifically, we normalize the advantage across the entire batch of rollouts obtained for each PPO step and initialize the value model from the reward model. We clipped the gradient by its Euclidean norm at a limit of1. Our training spanned500PPO steps on the RL data (MATH questions except MATH500 and our 500 validation questions). For generalized advantage estimation (GAE; Schulman et al. [61]), bothλandγwere set at 1. For PRM800K, we used a batch size of 512 for each PPO step. This comprised 8 epochs of gradient steps, each having 64 rollouts. We applied a peak learning rate of2×10 −5 with cosine decay. We opted for a constant KL regularizer coefficient of0.01, and a sampling temperature of0.7. For MetaMath/Math-Shepherd, we used a batch size of 512 for each PPO step. This comprised 2 epochs of gradient steps, each having 256 rollouts. We applied a peak learning rate of5×10 −6 with cosine decay. We opted for a constant KL regularizer coefficient of0.002, and a sampling temperature of1.2. D Re-ranking Results on MetaMath Similar to Sec. 4.2.1, we assess the effectiveness of process reward models on the MetaMath/Math- Shepherd dataset [86,75]. From Figure 6, we can see that PRMs are also more effective on harder tasks when trained on MetaMath/Math-Shepherd. 6 https://github.com/openai/lm-human-preferences 20 Table 6: Pass@N scores (upper bound of Best-of-N) on coding problems (APPS). SFT / ORM DECODING ACCURACY(%) TRAINDATAINTRO.INTER.COMP.ALL CODELLAMA- 7B EASYPASS@111.01.60.03.2 EASYPASS@420.04.72.07.2 EASYPASS@1639.09.36.014.6 CODELLAMA- 34B EASYPASS@119.05.32.07.3 EASYPASS@432.010.73.013.4 EASYPASS@1644.018.38.021.4 12481632641282565121024 N = number of solutions per problem 45 50 55 60 65 % Problems Solved 44.84 44.84 51.52 56.45 59.60 61.78 63.08 63.83 64.11 64.21 64.17 44.62 50.01 54.79 58.63 61.38 63.14 64.56 65.17 65.49 65.56 65.63 44.75 50.18 53.18 54.73 55.92 56.69 57.09 57.38 57.43 57.13 56.93 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-7b Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved 12.11 12.26 14.57 17.20 19.06 20.20 20.80 21.25 21.72 21.94 22.32 12.14 14.50 16.51 18.59 20.30 21.45 22.05 22.57 22.72 23.09 23.37 12.12 14.34 16.12 17.17 18.08 18.69 19.29 19.61 19.66 19.66 20.54 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-7b Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 27.5 30.0 32.5 35.0 37.5 40.0 42.5 % Problems Solved 27.68 27.77 32.11 35.87 38.37 40.00 40.90 41.53 41.88 42.09 42.19 27.57 31.39 34.70 37.65 39.82 41.25 42.27 42.78 43.01 43.23 43.39 27.65 31.38 33.75 35.05 36.14 36.79 37.32 37.60 37.66 37.52 37.89 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-7b Accuracy on All (Level 1-5) Problems Figure 6: Easy-to-hard generalization of evaluators on MetaMath. Both SFTs and RMs are trained on the easy data. 21 E Qualitative Comparison QUESTIONThe value ofyvaries inversely as √ xand whenx= 24,y= 15. What isxwhen y= 3? METAMATH-SFT Ifyvaries inversely as √ x, theny= k √ x for some constantk. Plugging in the given values, we have15 = k √ 24 , ork= 15 √ 24 24 = 15 √ 6 4 . Now we can findxwheny= 3by plugging in these values into the equationy= k √ x . We have3 = 15 √ 6 4 √ x , or3 = 15 √ 6 4 √ x . Cross-multiplying, we get3·4 √ x= 15 √ 6, or12 √ x= 15 √ 6. Dividing both sides by12, we find √ x= 15 √ 6 12 √ 6 = 5 2 . Squaring both sides, we getx= 5 2 2 = 6.25. METAMATH- PPO Ifyvaries inversely as √ x, then we can write the equation asy= k √ x for some constantk. We are given that whenx= 24,y= 15, so we can substitute these values into the equation to solve fork:15 = k √ 24 Multiplying both sides by √ 24gives15 √ 24 =k Simplifying the left side gives15·2 √ 6 =k, which further simplifies to30 √ 6 =k Now we can use this value ofkin the equation to findxwheny= 3: 3 = 30 √ 6 √ x Multiplying both sides by √ xgives3 √ x= 30 √ 6 Dividing both sides by3gives √ x= 10 √ 6 Squaring both sides givesx= (10 √ 6) 2 = 100(6) = 600 Table 7: Qualitative comparison between MetaMath-7b-SFT and MetaMath-7b-PPO, as well as the first error detected in MetaMath-SFT greedy decoding by Math-Shepherd. Examples are cherry- picked for better illustration. F Performance of DPO Iterations Figure 7: Test performance as a function of DPO iterations. 22 QUESTIONWhat is the total surface area, in square inches, of a cube with a volume of1 cubic foot? PRM800K-SFT The volume of a cube is given byV=s 3 , wheresis the length of one side. We are given thatV= 1cubic foot, sos 3 = 1. Taking the cube root of both sides, we find thats= 1. The surface area of a cube is given byA= 6s 2 , so the total surface area of this cube isA= 6(1 2 ) = 6 square inches. # Answer 6 PRM800K-PPO So the volume of a cube iss 3 . And we know the volume is1. Sos 3 = 1. Sos= 1. The surface area of a cube is6s 2 . So the surface area is6·1 2 = 6. So the answer is6. # Answer 6 Table 8: The distribution shift before and after PPO training of a Llemma-7B model trained on PRM800K. We believe such a big shift could explain why does PPO work exceptionally well on PRM800K, where the policy is initialized from a weak SFT model. 23 G Re-ranking Performance Analysis of PRMs, ORMs & OPRMs We compare the re-ranking performance of Process Reward Models (PRMs), Outcome Reward Models (ORMs), and our proposed Outcome & Process Reward Models (OPRMs). Figure 8 shows the results on 7b models and Figure 9 is on 34b models. We find that in our setting of Llemma [6] + MATH [32], PRMs and ORMs perform similarly, with PRMs slightly outperforming ORMs on hard tasks. But the OPRMs that trained on the mixed data of PRMs and ORMs significantly outperforms both of them. 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 27.84 29.02 36.19 43.88 49.84 53.21 55.36 56.58 57.03 57.17 56.76 28.02 34.59 40.48 45.77 50.42 53.81 56.40 58.11 59.14 59.82 60.19 28.02 34.26 38.85 42.25 45.54 47.61 49.06 49.88 50.21 49.81 49.09 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + PRM-7b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 27.84 29.02 36.19 43.88 49.84 53.21 55.36 56.58 57.03 57.17 56.76 28.00 36.08 42.89 48.70 53.16 56.08 58.04 59.55 60.25 60.77 60.81 28.21 36.16 41.76 46.07 48.51 50.22 50.90 51.16 51.29 50.92 50.91 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + ORM-7b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 27.84 29.02 36.19 43.88 49.84 53.21 55.36 56.58 57.03 57.17 56.76 28.23 35.31 41.63 47.07 51.72 55.36 58.29 60.63 62.07 62.80 63.20 27.87 35.66 41.69 46.07 49.23 51.38 52.86 53.87 54.65 55.03 55.06 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 % Problems Solved 6.84 7.56 9.20 11.60 14.14 15.75 17.03 17.94 18.38 18.92 19.19 6.90 8.82 10.79 12.93 14.95 16.66 18.12 19.62 20.89 21.98 22.77 6.93 8.93 10.35 11.82 12.88 14.05 14.70 15.28 15.67 15.79 15.66 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + PRM-7b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 % Problems Solved 6.84 7.56 9.20 11.60 14.14 15.75 17.03 17.94 18.38 18.92 19.19 6.92 9.62 11.95 14.19 15.87 17.41 18.69 19.86 20.51 21.01 21.23 6.88 9.44 11.50 13.09 14.06 14.85 15.60 15.96 16.25 16.38 16.34 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + ORM-7b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 % Problems Solved 6.84 7.56 9.20 11.60 14.14 15.75 17.03 17.94 18.38 18.92 19.19 6.93 9.32 11.41 13.69 15.69 17.70 19.25 20.89 22.39 23.33 24.12 6.80 9.48 11.84 13.92 15.26 16.50 17.66 18.70 19.38 19.95 19.91 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 16.87 17.78 22.07 26.96 31.09 33.56 35.27 36.33 36.78 37.12 37.10 16.99 21.10 24.95 28.59 31.83 34.32 36.35 37.93 39.05 39.93 40.47 17.00 20.95 23.95 26.31 28.40 29.99 31.05 31.72 32.09 31.95 31.55 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + PRM-7b (on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 16.87 17.78 22.07 26.96 31.09 33.56 35.27 36.33 36.78 37.12 37.10 16.97 22.23 26.66 30.64 33.61 35.80 37.41 38.74 39.42 39.90 39.98 17.06 22.17 25.91 28.75 30.44 31.66 32.38 32.67 32.88 32.80 32.77 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + ORM-7b (on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 16.87 17.78 22.07 26.96 31.09 33.56 35.27 36.33 36.78 37.12 37.10 17.07 21.67 25.84 29.57 32.85 35.60 37.86 39.75 41.27 42.09 42.74 16.81 21.93 26.04 29.20 31.41 33.08 34.42 35.43 36.17 36.62 36.66 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on All (Level 1-5) Problems Figure 8: Comparing process reward models (PRMs, left), outcome reward models (ORMs, middle), and outcome & process reward models (OPRMs, right) on 7b models trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data. 24 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.52 38.93 47.59 56.06 61.96 65.69 67.52 68.56 68.89 68.85 68.51 37.65 45.44 52.37 57.84 62.71 66.31 68.41 69.77 70.59 70.69 70.48 37.60 45.20 50.74 55.06 58.48 60.82 62.40 63.64 64.09 64.51 64.57 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + PRM-34b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.52 38.93 47.59 56.06 61.96 65.69 67.52 68.56 68.89 68.85 68.51 37.84 46.71 54.42 60.66 65.29 68.38 70.28 71.18 71.54 71.71 71.53 37.65 46.72 53.41 57.51 60.25 62.05 62.83 63.50 64.21 64.53 64.99 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + ORM-34b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.52 38.93 47.59 56.06 61.96 65.69 67.52 68.56 68.89 68.85 68.51 37.47 46.88 54.25 60.14 64.96 68.36 71.00 72.64 73.61 74.08 74.37 37.79 46.97 54.16 59.83 63.40 65.88 67.57 68.19 68.24 68.04 67.44 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.19 10.86 13.69 17.25 20.59 23.09 24.75 25.60 26.09 26.37 26.41 10.36 13.32 16.14 18.88 21.20 23.40 25.49 27.07 27.93 28.70 29.38 10.16 13.11 15.51 17.52 18.86 20.16 21.35 22.40 23.09 23.43 23.67 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + PRM-34b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.19 10.86 13.69 17.25 20.59 23.09 24.75 25.60 26.09 26.37 26.41 10.27 13.73 17.03 19.90 22.56 24.88 26.62 27.87 29.01 29.78 30.19 10.23 13.75 16.56 18.53 20.17 21.24 21.95 22.45 22.74 23.00 23.70 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + ORM-34b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.19 10.86 13.69 17.25 20.59 23.09 24.75 25.60 26.09 26.37 26.41 10.34 13.87 17.18 20.27 23.04 25.42 27.34 29.16 30.50 31.61 32.58 10.31 14.11 17.40 20.31 22.30 24.04 24.97 25.73 26.16 26.68 27.02 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.21 24.22 29.80 35.71 40.28 43.34 45.16 46.07 46.52 46.62 46.53 23.38 28.58 33.37 37.44 40.95 43.80 45.98 47.43 48.24 48.72 48.91 23.24 28.41 32.27 35.39 37.75 39.51 40.89 42.02 42.61 42.96 43.12 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + PRM-34b (on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.21 24.22 29.80 35.71 40.28 43.34 45.16 46.07 46.52 46.62 46.53 23.38 29.42 34.82 39.30 42.90 45.58 47.45 48.49 49.28 49.73 49.88 23.26 29.45 34.09 37.10 39.24 40.64 41.38 41.96 42.45 42.73 43.32 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + ORM-34b (on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.21 24.22 29.80 35.71 40.28 43.34 45.16 46.07 46.52 46.62 46.53 23.26 29.58 34.83 39.21 42.99 45.84 48.17 49.87 50.98 51.81 52.45 23.41 29.76 34.88 39.10 41.86 43.94 45.26 45.97 46.25 46.37 46.31 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on All (Level 1-5) Problems Figure 9: Comparing process reward models (PRMs, left), outcome reward models (ORMs, middle), and outcome & process reward models (OPRMs, right) on 34b models trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data. 25 H Re-ranking Results on MetaMath Similar to Sec. 4.2.1, we assess the effectiveness of process reward models on the MetaMath/Math- Shepherd dataset [86, 75]. From Figure 10, we can see that PRMs are also more effective on harder tasks when trained on MetaMath/Math-Shepherd. 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved 44.84 44.84 51.52 56.45 59.60 61.78 63.08 63.83 64.11 64.21 64.17 44.75 50.22 54.64 58.53 61.22 63.22 64.55 64.98 65.16 65.17 65.27 44.69 50.33 53.71 55.95 57.79 58.97 59.73 60.39 60.62 60.78 60.80 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-7b Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved 44.84 44.84 51.52 56.45 59.60 61.78 63.08 63.83 64.11 64.21 64.17 44.41 49.87 54.35 57.98 60.84 62.92 64.10 64.82 65.40 65.61 65.72 44.72 49.66 52.64 54.48 55.59 56.26 56.89 57.39 57.85 57.77 57.45 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + ORM-7b Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved 44.84 44.84 51.52 56.45 59.60 61.78 63.08 63.83 64.11 64.21 64.17 44.76 50.18 55.18 59.20 62.11 64.29 66.07 67.13 67.67 67.87 67.80 44.78 50.35 53.46 55.61 56.46 57.20 57.21 56.92 57.06 56.92 56.58 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-OPRM-7b Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved 12.11 12.26 14.57 17.20 19.06 20.20 20.80 21.25 21.72 21.94 22.32 12.08 14.30 16.68 18.55 20.38 21.41 22.11 22.51 22.72 22.88 23.09 11.98 14.48 16.09 17.30 17.92 18.28 18.64 18.80 18.82 18.33 17.54 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-7b Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved 12.11 12.26 14.57 17.20 19.06 20.20 20.80 21.25 21.72 21.94 22.32 12.05 14.37 16.37 18.07 19.58 20.76 21.25 21.5921.60 21.71 21.77 11.95 14.25 15.91 17.31 18.55 19.08 19.23 19.33 19.43 19.53 19.40 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + ORM-7b Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved 12.11 12.26 14.57 17.20 19.06 20.20 20.80 21.25 21.72 21.94 22.32 12.17 14.42 16.42 18.30 19.83 20.91 21.52 21.83 22.05 22.20 22.31 12.02 14.39 16.03 17.07 17.62 17.85 17.91 18.32 19.04 19.73 19.48 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-OPRM-7b Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25.0 27.5 30.0 32.5 35.0 37.5 40.0 42.5 45.0 % Problems Solved 27.68 27.77 32.11 35.87 38.37 40.00 40.90 41.53 41.88 42.09 42.19 27.63 31.38 34.74 37.59 39.81 41.32 42.29 42.67 42.89 42.93 43.08 27.57 31.52 33.94 35.69 36.89 37.62 38.20 38.60 38.73 38.49 38.10 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-7b Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 25.0 27.5 30.0 32.5 35.0 37.5 40.0 42.5 45.0 % Problems Solved 27.68 27.77 32.11 35.87 38.37 40.00 40.90 41.53 41.88 42.09 42.19 27.46 31.25 34.47 37.07 39.22 40.87 41.66 42.14 42.41 42.56 42.63 27.52 31.12 33.38 35.00 36.18 36.75 37.17 37.44 37.71 37.75 37.50 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + ORM-7b Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 25.0 27.5 30.0 32.5 35.0 37.5 40.0 42.5 45.0 % Problems Solved 27.68 27.77 32.11 35.87 38.37 40.00 40.90 41.53 41.88 42.09 42.19 27.66 31.44 34.83 37.75 39.98 41.54 42.69 43.40 43.76 43.95 43.90 27.60 31.46 33.82 35.41 36.11 36.58 36.63 36.71 37.16 37.43 37.16 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM MetaMath-7b + Math-Shepherd-OPRM-7b Accuracy on All (Level 1-5) Problems Figure 10: Comparing process reward models (PRMs, left, trained on Meth-Shepherd), outcome reward models (ORMs, middle), and outcome & process reward models (OPRMs, right) on 7b models trained on the MetaMath dataset. Both SFTs and RMs are trained on the easy data. 26 I More Comparisons 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 28.02 34.59 40.48 45.77 50.42 53.81 56.40 58.11 59.14 59.82 60.19 28.00 36.08 42.89 48.70 53.16 56.08 58.04 59.55 60.25 60.77 60.81 28.23 35.31 41.63 47.07 51.72 55.36 58.29 60.63 62.07 62.80 63.20 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Weighted Voting on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 % Problems Solved 6.90 8.82 10.79 12.93 14.95 16.66 18.12 19.62 20.89 21.98 22.77 6.92 9.62 11.95 14.19 15.87 17.41 18.69 19.86 20.51 21.01 21.23 6.93 9.32 11.41 13.69 15.69 17.70 19.25 20.89 22.39 23.33 24.12 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Weighted Voting on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 16.99 21.10 24.95 28.59 31.83 34.32 36.35 37.93 39.05 39.93 40.47 16.97 22.23 26.66 30.64 33.61 35.80 37.41 38.74 39.42 39.90 39.98 17.07 21.67 25.84 29.57 32.85 35.60 37.86 39.75 41.27 42.09 42.74 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Weighted Voting on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 28.02 34.26 38.85 42.25 45.54 47.61 49.06 49.88 50.21 49.81 49.09 28.21 36.16 41.76 46.07 48.51 50.22 50.90 51.16 51.29 50.92 50.91 27.87 35.66 41.69 46.07 49.23 51.38 52.86 53.87 54.65 55.03 55.06 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Best-of-N on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 % Problems Solved 6.93 8.93 10.35 11.82 12.88 14.05 14.70 15.28 15.67 15.79 15.66 6.88 9.44 11.50 13.09 14.06 14.85 15.60 15.96 16.25 16.38 16.34 6.80 9.48 11.84 13.92 15.26 16.50 17.66 18.70 19.38 19.95 19.91 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Best-of-N on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 17.00 20.95 23.95 26.31 28.40 29.99 31.05 31.72 32.09 31.95 31.55 17.06 22.17 25.91 28.75 30.44 31.66 32.38 32.67 32.88 32.80 32.77 16.81 21.93 26.04 29.20 31.41 33.08 34.42 35.43 36.17 36.62 36.66 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Best-of-N on PRM800K) Accuracy on All (Level 1-5) Problems Figure 11: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 7b models trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data. 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.65 45.44 52.37 57.84 62.71 66.31 68.41 69.77 70.59 70.69 70.48 37.84 46.71 54.42 60.66 65.29 68.38 70.28 71.18 71.54 71.71 71.53 37.47 46.88 54.25 60.14 64.96 68.36 71.00 72.64 73.61 74.08 74.37 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-34b (Weighted Voting on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.36 13.32 16.14 18.88 21.20 23.40 25.49 27.07 27.93 28.70 29.38 10.27 13.73 17.03 19.90 22.56 24.88 26.62 27.87 29.01 29.78 30.19 10.34 13.87 17.18 20.27 23.04 25.42 27.34 29.16 30.50 31.61 32.58 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-34b (Weighted Voting on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.38 28.58 33.37 37.44 40.95 43.80 45.98 47.43 48.24 48.72 48.91 23.38 29.42 34.82 39.30 42.90 45.58 47.45 48.49 49.28 49.73 49.88 23.26 29.58 34.83 39.21 42.99 45.84 48.17 49.87 50.98 51.81 52.45 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-34b (Weighted Voting on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.60 45.20 50.74 55.06 58.48 60.82 62.40 63.64 64.09 64.51 64.57 37.65 46.72 53.41 57.51 60.25 62.05 62.83 63.50 64.21 64.53 64.99 37.79 46.97 54.16 59.83 63.40 65.88 67.57 68.19 68.24 68.04 67.44 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-34b (Best-of-N on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.16 13.11 15.51 17.52 18.86 20.16 21.35 22.40 23.09 23.43 23.67 10.23 13.75 16.56 18.53 20.17 21.24 21.95 22.45 22.74 23.00 23.70 10.31 14.11 17.40 20.31 22.30 24.04 24.97 25.73 26.16 26.68 27.02 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-34b (Best-of-N on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.24 28.41 32.27 35.39 37.75 39.51 40.89 42.02 42.61 42.96 43.12 23.26 29.45 34.09 37.10 39.24 40.64 41.38 41.96 42.45 42.73 43.32 23.41 29.76 34.88 39.10 41.86 43.94 45.26 45.97 46.25 46.37 46.31 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-34b (Best-of-N on PRM800K) Accuracy on All (Level 1-5) Problems Figure 12: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 34b models trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data. 27 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.61 45.01 51.38 56.68 61.51 64.99 67.32 68.98 69.85 70.32 70.41 37.62 46.62 54.00 60.19 64.77 67.98 69.71 70.41 70.72 70.88 71.01 37.72 45.99 52.58 58.04 62.63 66.03 68.57 70.42 71.42 72.28 72.68 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-7b (Weighted Voting on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.26 12.96 15.75 18.25 20.53 22.69 24.63 26.11 27.23 27.75 28.12 10.35 13.48 16.53 19.60 22.09 24.13 25.96 27.06 27.88 28.60 29.09 10.23 13.33 16.33 18.99 21.44 23.58 25.64 27.41 28.80 30.06 30.68 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-7b (Weighted Voting on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.29 28.19 32.64 36.54 40.01 42.81 44.99 46.56 47.53 48.02 48.25 23.34 29.25 34.35 38.96 42.38 45.05 46.79 47.72 48.33 48.76 49.06 23.31 28.86 33.58 37.59 41.04 43.77 46.12 47.93 49.10 50.15 50.67 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-7b (Weighted Voting on PRM800K) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 75 % Problems Solved 37.52 44.79 49.93 53.80 56.52 58.58 60.01 61.35 62.41 63.31 63.89 37.71 46.31 52.57 56.59 58.42 59.45 59.66 59.74 59.70 59.09 57.72 37.50 46.31 52.58 56.84 59.29 60.92 62.13 62.87 63.27 63.60 63.46 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-7b (Best-of-N on PRM800K) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 35 % Problems Solved 10.29 13.26 15.16 16.91 18.39 19.53 20.41 20.78 21.00 21.11 21.16 10.31 13.49 15.77 16.96 18.07 18.52 19.05 19.00 18.97 18.90 18.86 10.39 13.59 16.61 18.71 20.17 21.44 22.59 23.32 24.04 24.33 24.38 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-7b (Best-of-N on PRM800K) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 45 50 55 % Problems Solved 23.24 28.22 31.67 34.46 36.55 38.11 39.26 40.09 40.72 41.20 41.51 23.37 29.13 33.28 35.81 37.27 38.00 38.38 38.37 38.33 38.02 37.33 23.27 29.16 33.69 36.86 38.75 40.20 41.41 42.11 42.69 43.02 42.96 Process RM Outcome RM Outcome \& Process RM SFT-34b + RM-7b (Best-of-N on PRM800K) Accuracy on All (Level 1-5) Problems Figure 13: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 34b SFT model and 7b reward model trained on the PRM800K dataset. Both SFTs and RMs are trained on the easy data. 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved 44.75 50.22 54.64 58.53 61.22 63.22 64.55 64.98 65.16 65.17 65.27 44.41 49.87 54.35 57.98 60.84 62.92 64.10 64.82 65.40 65.61 65.72 44.76 50.18 55.18 59.20 62.11 64.29 66.07 67.13 67.67 67.87 67.80 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Weighted Voting on MetaMath) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved 12.08 14.30 16.68 18.55 20.38 21.41 22.11 22.51 22.72 22.88 23.09 12.05 14.37 16.37 18.07 19.58 20.76 21.25 21.5921.60 21.71 21.77 12.17 14.42 16.42 18.30 19.83 20.91 21.52 21.83 22.05 22.20 22.31 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Weighted Voting on MetaMath) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25.0 27.5 30.0 32.5 35.0 37.5 40.0 42.5 45.0 % Problems Solved 27.63 31.38 34.74 37.59 39.81 41.32 42.29 42.67 42.89 42.93 43.08 27.46 31.25 34.47 37.07 39.22 40.87 41.66 42.14 42.41 42.56 42.63 27.66 31.44 34.83 37.75 39.98 41.54 42.69 43.40 43.76 43.95 43.90 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Weighted Voting on MetaMath) Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved 44.69 50.33 53.71 55.95 57.79 58.97 59.73 60.39 60.62 60.78 60.80 44.72 49.66 52.64 54.48 55.59 56.26 56.89 57.39 57.85 57.77 57.45 44.78 50.35 53.46 55.61 56.46 57.20 57.21 56.92 57.06 56.92 56.58 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Best-of-N on MetaMath) Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved 11.98 14.48 16.09 17.30 17.92 18.28 18.64 18.80 18.82 18.33 17.54 11.95 14.25 15.91 17.31 18.55 19.08 19.23 19.33 19.43 19.53 19.40 12.02 14.39 16.03 17.07 17.62 17.85 17.91 18.32 19.04 19.73 19.48 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Best-of-N on MetaMath) Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25.0 27.5 30.0 32.5 35.0 37.5 40.0 42.5 45.0 % Problems Solved 27.57 31.52 33.94 35.69 36.89 37.62 38.20 38.60 38.73 38.49 38.10 27.52 31.12 33.38 35.00 36.18 36.75 37.17 37.44 37.71 37.75 37.50 27.60 31.46 33.82 35.41 36.11 36.58 36.63 36.71 37.16 37.43 37.16 Process RM Outcome RM Outcome \& Process RM SFT-7b + RM-7b (Best-of-N on MetaMath) Accuracy on All (Level 1-5) Problems Figure 14: Comparing different reward models with Weighted Voting (upper) and Best-of-N (lower) on 7b models trained on the MetaMath dataset. Both SFTs and RMs are trained on the easy data. 28 Table 9: Results of Full, Easy-to-Hard, & Hard-to-Easy SFT training of the Llemma-7b model TRAININGDATA PRM800KMETAMATH ALLHARDALLHARD FULLSFTALL20.69.931.413.7 EASY-TO-HARDSFTEASY19.812.230.014.9 HARD-TO-EASYSFTHARD18.413.030.415.3 Figure 15: The agreement between the prediction from the Llemma-7b-based reward model when trained on ORM and PRM data, and their agreement to ground-truth final-answer labels. J Hard-to-Easy Generalization From Table 5, it is evident that reinforcement learning training on hard tasks alone significantly underperforms compared to training the model on easy tasks or on all tasks. This difference is especially pronounced for PPO on the PRM800K dataset. This raises a crucial question: does training on hard tasks only generalize to easy tasks? To address this, we fine-tuned the Llemma-7b model using all data (easy and hard), only easy data, and only hard data. As shown in Table 9, we found that training on all data consistently yields the best performance. Conversely, the generator’s performance deteriorates when transitioning from easy- to-hard and hard-to-easy tasks. This suggests that language models face difficulties in generalizing in both directions. It is also worth noting that while Full SFT underperforms Easy-to-Hard SFT and Hard-to-Easy SFT on hard test questions, it eventually outperforms Easy-to-Hard SFT and Hard-to-Easy SFT when evaluated on all test questions. We believe that this is because by exposing the model to a wider variety of unique questions and difficulties, it gains a better understanding of the problem space in general, as measured by the accuracy on the full distribution. K On ORM’s Approximation of PRM Labels From Sec. G, we observe that in PRM800K, PRMs and ORMs exhibit similar performance levels, with OPRMs outperforming both. This raises the question of why ORMs also demonstrate strong easy-to-hard generalization ability. A straightforward explanation is that ORMs are trained to approximate PRM labels [74]. Specifically, ORMs are trained to predict the correctness of the entire solution through value estimation. As Uesato et al.[74]state, “it is simpler for the ORM to learn to recognize when steps are correct than it is to check the answer by internally computing the final answer itself.” Nevertheless, people may argue that the conclusion from Uesato et al.[74]is based on GSM8K’s experimental results, so the conclusion may not transfer to the more challenging Hendrick’s MATH dataset. To show the universal existence of “ORM’s approximation of PRM labels”, we further conduct evaluation of agreement between different rewards on two variants of the MATH dataset: PRM800K and MetaMath. The results are shown in Figure 15. Similarly to the findings from Uesato et al.[74], we see that the ORM has higher agreement with the PRM, despite being trained to predict the Final-Answer rewards. 29 Thus, “this result indicates that the ORM tends more towards predicting whether the full trace is correct, and not just whether the final answer is correct.” Overall, this shows easy-to-hard generalization is not exclusively linked to reward models trained on explicit step-wise annotations. It also applies to ORMs that are trained to perform value estimation and practically evaluates each solution step. We also perform DPO training on a MetaMath-initialized Llemma-7b model. We find that in this RL setting, re-ranking the output pairs with ORM also gives similar performance to re-ranking with PRM (29.2 v.s. 30.4 & 31.2 v.s. 34.0). L Analysis of Aggregation Functions in PRMs & OPRMs We explored different methods to consolidate step-wise prediction scores into a single score value, a process we describe as employing an aggregation function, during the use of the evaluator. Lightman et al.[40]report comparable performance when usingmin(minimum) andprod(product) as the aggregation function to reduce multiple scores into a single solution-level score. Note that when training PRMs on PRM800K [40], we have already considered neutral steps to be positive as training labels. Following Wang et al.[77], givenp 1 ,p 2 ,...,p n as a list of predicted correctness probability of each step (including the final answer), we considered the following aggregation functions: min= minp 1 ,p 2 ,...,p n (1) max= maxp 1 ,p 2 ,...,p n (2) prod= Y i p i (3) mean= P i p i n (4) mean_logit=σ P i log p i 1−p i n ! (5) mean_odd= ReLU P i p i 1−p i n ! (6) last=p n (7) In Figure 16-18, we perform analysis of aggregation functions on PRM800K and Math-Shepherd (from MetaMath) datasets with weighted voting and best-of-ndecoding and PRMs or OPRMs. In general, we findprodworks universally well in weighted voting andminworks well in best-of-n. So we adopt these two strategies in our main experiments. One interesting finding is that for reward models trained on the human annotated process reward (e.g., PRM800K), thelaststrategy does not perform very well, butlastworks much better on OPRMs and pseudo PRMs (e.g., Math-Shepherd). This could partially explain why OPRMs does not further improve the performance on the Math-Shepherd dataset. 30 12481632641282565121024 N = number of solutions per problem 30 35 40 45 50 55 60 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + PRM-7b + Weighted Voting Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 6 8 10 12 14 16 18 20 22 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + PRM-7b + Weighted Voting Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + PRM-7b + Weighted Voting Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 30 35 40 45 50 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + PRM-7b + Best-of-N Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 8 10 12 14 16 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + PRM-7b + Best-of-N Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 18 20 22 24 26 28 30 32 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + PRM-7b + Best-of-N Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + PRM-34b + Weighted Voting Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10.0 12.5 15.0 17.5 20.0 22.5 25.0 27.5 30.0 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + PRM-34b + Weighted Voting Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + PRM-34b + Weighted Voting Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + PRM-34b + Best-of-N Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10 12 14 16 18 20 22 24 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + PRM-34b + Best-of-N Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + PRM-34b + Best-of-N Accuracy on All (Level 1-5) Problems Figure 16: Analysis of aggregation functions in process reward models (PRMs) on the PRM800K dataset with Weighted Voting and Best-of-N. Both SFTs and RMs are trained on the easy data. 31 12481632641282565121024 N = number of solutions per problem 30 35 40 45 50 55 60 65 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + OPRM-7b + Weighted Voting Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 7.5 10.0 12.5 15.0 17.5 20.0 22.5 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + OPRM-7b + Weighted Voting Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 20 25 30 35 40 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + OPRM-7b + Weighted Voting Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 30 35 40 45 50 55 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + OPRM-7b + Best-of-N Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 8 10 12 14 16 18 20 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + OPRM-7b + Best-of-N Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 17.5 20.0 22.5 25.0 27.5 30.0 32.5 35.0 37.5 % Problems Solved prod min max mean mean_logit mean_odd last SFT-7b + OPRM-7b + Best-of-N Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 75 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + OPRM-34b + Weighted Voting Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + OPRM-34b + Weighted Voting Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + OPRM-34b + Weighted Voting Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + OPRM-34b + Best-of-N Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 10.0 12.5 15.0 17.5 20.0 22.5 25.0 27.5 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + OPRM-34b + Best-of-N Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 % Problems Solved prod min max mean mean_logit mean_odd last SFT-34b + OPRM-34b + Best-of-N Accuracy on All (Level 1-5) Problems Figure 17: Analysis of aggregation functions in outcome & process reward models (OPRMs) on the PRM800K dataset with Weighted Voting and Best-of-N. Both SFTs and RMs are trained on the easy data. 32 12481632641282565121024 N = number of solutions per problem 45 50 55 60 65 % Problems Solved prod min mean last MetaMath-7b + Math-Shepherd-7b + Weighted Voting Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 12 14 16 18 20 22 % Problems Solved prod min mean last MetaMath-7b + Math-Shepherd-7b + Weighted Voting Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 28 30 32 34 36 38 40 42 44 % Problems Solved prod min mean last MetaMath-7b + Math-Shepherd-7b + Weighted Voting Accuracy on All (Level 1-5) Problems 12481632641282565121024 N = number of solutions per problem 44 46 48 50 52 54 56 58 % Problems Solved prod min mean last MetaMath-7b + Math-Shepherd-7b + Best-of-N Accuracy on Easy (Level 1-3) Problems 12481632641282565121024 N = number of solutions per problem 12 14 16 18 20 % Problems Solved prod min mean last MetaMath-7b + Math-Shepherd-7b + Best-of-N Accuracy on Hard (Level 4-5) Problems 12481632641282565121024 N = number of solutions per problem 28 30 32 34 36 38 % Problems Solved prod min mean last MetaMath-7b + Math-Shepherd-7b + Best-of-N Accuracy on All (Level 1-5) Problems Figure 18: Analysis of aggregation functions in psuedo process reward models (PRMs) on the Math-Shepherd (from MetaMath) dataset with Weighted Voting and Best-of-N. Both SFTs and RMs are trained on the easy data. M Societal Impact Our work on easy-to-hard generalization has the potential for both positive and negative societal impacts. On the positive side, this approach could enable AI systems to tackle increasingly complex problems in domains such as scientific discovery, healthcare, and education, potentially leading to groundbreaking advancements that benefit society. However, the development of AI systems that can operate beyond human supervision also raises concerns about the transparency, accountability, and potential misuse of such systems. It is crucial to carefully consider the ethical implications and establish robust safeguards to mitigate the risks of unintended consequences or malicious applications. Ongoing research and public discourse on the responsible development and deployment of these technologies will be essential to ensure that their societal benefits outweigh the potential drawbacks. 33 N Fine-Grained Analysis of OPRMs’ Re-ranking strategies 12481632641282565121024 N = number of solutions per problem 40 50 60 70 80 % Problems Solved 41.75 42.65 52.27 61.07 66.67 69.98 71.53 72.55 72.47 72.42 72.13 41.34 50.92 59.31 65.49 70.49 74.56 78.04 80.65 82.21 82.95 83.45 41.48 52.08 59.37 65.16 70.28 72.96 75.06 75.69 75.97 75.30 73.19 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Level1 Problems 12481632641282565121024 N = number of solutions per problem 40 50 60 70 80 % Problems Solved 41.75 42.65 52.27 61.07 66.67 69.98 71.53 72.55 72.47 72.42 72.13 41.67 52.09 60.79 66.64 70.42 73.28 75.46 77.03 78.42 79.52 80.02 41.60 53.09 62.06 68.52 73.91 76.97 79.45 81.17 82.02 82.68 84.05 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Level1 Problems 12481632641282565121024 N = number of solutions per problem 50 60 70 80 90 % Problems Solved 53.06 54.32 65.29 74.04 80.41 84.66 87.42 89.02 89.67 89.67 89.75 53.09 62.64 70.44 76.28 80.36 83.56 85.22 86.32 86.91 87.12 87.36 52.94 63.89 69.84 74.41 77.57 79.61 80.84 81.39 81.97 82.75 83.75 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Level1 Problems 12481632641282565121024 N = number of solutions per problem 50 60 70 80 90 % Problems Solved 53.06 54.32 65.29 74.04 80.41 84.66 87.42 89.02 89.67 89.67 89.75 53.26 63.81 72.67 78.18 82.11 84.50 86.18 86.66 86.64 86.35 86.03 53.15 64.99 72.11 77.28 80.46 82.69 83.94 83.98 83.94 83.32 82.90 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Level1 Problems 12481632641282565121024 N = number of solutions per problem 30 35 40 45 50 55 60 65 70 % Problems Solved 33.00 33.99 41.98 50.23 56.12 59.79 61.98 63.71 64.74 65.07 64.58 33.14 41.02 47.72 52.88 57.59 60.97 63.19 65.15 66.49 67.40 67.91 33.49 41.44 47.06 50.53 53.65 55.28 57.23 58.78 60.03 61.11 62.45 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Level2 Problems 12481632641282565121024 N = number of solutions per problem 30 40 50 60 70 80 % Problems Solved 33.00 33.99 41.98 50.23 56.12 59.79 61.98 63.71 64.74 65.07 64.58 33.33 41.77 49.35 55.35 60.69 64.92 68.23 70.94 73.01 74.64 76.06 33.13 42.95 50.08 55.69 60.14 63.22 65.71 67.14 68.06 68.84 69.38 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Level2 Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 75 80 % Problems Solved 43.24 43.77 53.93 62.41 68.52 71.53 73.17 74.10 74.83 74.88 74.75 43.45 51.62 58.60 63.93 68.33 71.55 74.07 75.97 77.34 78.04 78.25 43.02 52.23 58.59 61.61 63.21 64.30 64.81 65.40 65.45 65.25 64.28 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Level2 Problems 12481632641282565121024 N = number of solutions per problem 40 45 50 55 60 65 70 75 80 % Problems Solved 43.24 43.77 53.93 62.41 68.52 71.53 73.17 74.10 74.83 74.88 74.75 43.59 53.34 60.56 67.02 71.31 74.63 77.16 78.38 78.88 78.99 78.83 43.24 53.75 60.81 66.06 69.54 71.86 72.98 73.50 73.22 72.52 71.62 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Level2 Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 50 % Problems Solved 18.27 18.88 24.45 31.37 37.02 41.05 43.20 43.75 44.13 44.29 44.07 18.08 23.60 29.40 33.97 39.00 43.36 46.35 48.55 49.91 50.83 50.99 17.80 24.22 30.00 33.83 36.93 38.79 40.01 40.64 41.23 41.24 41.16 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Level3 Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 50 55 % Problems Solved 18.27 18.88 24.45 31.37 37.02 41.05 43.20 43.75 44.13 44.29 44.07 18.18 24.43 30.73 36.07 41.28 45.33 48.27 50.39 51.58 52.37 52.59 17.99 25.84 31.48 37.24 41.58 45.10 47.19 48.89 50.25 51.15 51.52 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Level3 Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 26.77 27.85 34.44 42.76 49.30 52.80 54.47 54.92 55.13 55.06 54.59 26.52 33.37 39.74 45.74 50.30 54.22 56.90 58.96 60.42 61.17 62.03 26.66 33.87 40.28 44.84 48.16 50.23 51.97 52.84 53.37 54.13 53.96 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Level3 Problems 12481632641282565121024 N = number of solutions per problem 30 40 50 60 70 % Problems Solved 26.77 27.85 34.44 42.76 49.30 52.80 54.47 54.92 55.13 55.06 54.59 26.74 33.92 41.23 47.28 52.25 56.56 60.05 62.22 63.76 65.17 66.19 26.58 34.49 41.31 47.05 51.32 53.56 55.70 57.35 57.73 58.23 57.12 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Level3 Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 % Problems Solved 10.32 10.93 13.70 17.88 21.62 23.82 25.47 26.39 26.83 27.21 27.56 10.39 14.01 17.54 20.56 23.76 26.89 29.07 31.03 32.88 33.84 34.49 10.64 14.30 17.80 20.57 22.72 24.67 26.34 27.60 28.31 28.80 29.52 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Level4 Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 40 % Problems Solved 10.32 10.93 13.70 17.88 21.62 23.82 25.47 26.39 26.83 27.21 27.56 10.35 14.62 18.75 22.57 25.66 28.75 31.66 34.44 36.41 38.70 40.51 10.44 14.82 19.55 23.39 26.55 29.46 31.86 33.94 36.07 37.10 37.59 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Level4 Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 15.40 16.26 20.22 25.92 30.00 33.43 35.94 37.83 38.54 38.81 38.69 15.37 19.95 23.84 27.96 31.69 35.09 38.25 40.61 42.69 44.39 45.61 15.29 20.01 24.19 27.32 29.60 31.95 33.05 34.38 35.50 36.50 36.41 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Level4 Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 15.40 16.26 20.22 25.92 30.00 33.43 35.94 37.83 38.54 38.81 38.69 15.17 20.31 25.30 29.53 33.30 36.81 39.58 42.25 43.91 45.31 46.08 15.18 20.77 25.63 29.15 32.50 34.48 36.50 37.95 38.22 39.30 40.37 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Level4 Problems 12481632641282565121024 N = number of solutions per problem 2 4 6 8 10 12 14 % Problems Solved 3.45 3.85 4.61 5.65 6.90 8.12 8.83 9.66 10.49 10.79 11.19 3.47 4.64 5.77 6.84 7.94 8.78 9.95 11.02 12.23 13.26 14.10 3.45 4.80 6.25 7.34 8.02 8.84 9.50 10.02 10.74 11.00 10.75 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Level5 Problems 12481632641282565121024 N = number of solutions per problem 2 4 6 8 10 12 14 16 % Problems Solved 3.45 3.85 4.61 5.65 6.90 8.12 8.83 9.66 10.49 10.79 11.19 3.40 4.66 6.08 7.37 8.90 10.33 11.89 12.90 14.54 15.61 15.78 3.43 4.99 6.61 8.11 9.32 10.66 11.71 12.52 13.42 14.27 15.02 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Level5 Problems 12481632641282565121024 N = number of solutions per problem 4 6 8 10 12 14 16 18 % Problems Solved 5.27 6.04 7.13 9.37 11.43 12.89 13.85 14.22 14.32 14.46 14.73 5.36 7.10 8.94 10.40 11.66 12.83 13.84 14.85 15.66 16.19 16.36 5.22 7.34 9.09 10.29 11.35 11.94 12.23 12.61 12.92 13.01 12.74 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Level5 Problems 12481632641282565121024 N = number of solutions per problem 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 % Problems Solved 5.27 6.04 7.13 9.37 11.43 12.89 13.85 14.22 14.32 14.46 14.73 5.14 7.34 9.53 11.16 12.92 14.37 15.38 16.48 17.55 18.45 19.77 5.41 7.62 9.82 11.67 12.88 13.83 14.18 14.32 14.61 14.58 14.47 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Level5 Problems Figure 19: Easy-to-hard generalization for different difficulty levels’ data. Both SFTs and OPRMs are trained on the level1-3 data. Each row compares the performance of different OPRMs’ reranking strategies across different levels’ data. As shown in Figure 19, both the Best-of-N and Weighted Voting strategies demonstrate strong perfor- mance across all levels, which leverage the advantages of OPRM methods and thereby underscoring OPRM’s effectiveness. Furthermore, despite the SFT models and OPRM models being trained on level 1-3 data, re-ranking strategies enhanced by OPRMs continue to perform well on the unseen and more challenging level 4-5 data. This indicates the feasibility of generalizing from easier to harder tasks using OPRMs. 34 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 60 65 % Problems Solved 27.67 28.15 34.82 42.20 47.75 51.42 53.86 55.23 56.35 57.07 57.10 27.52 34.30 40.31 45.19 49.80 53.38 56.42 58.85 61.05 62.78 63.57 27.62 34.82 40.02 43.84 46.59 48.52 50.30 51.58 52.76 53.28 54.02 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Algebra Problems 12481632641282565121024 N = number of solutions per problem 30 40 50 60 70 % Problems Solved 27.67 28.15 34.82 42.20 47.75 51.42 53.86 55.23 56.35 57.07 57.10 27.74 35.29 42.09 47.61 52.17 55.96 59.39 61.69 64.19 66.31 67.36 27.68 36.12 42.66 48.30 52.06 55.33 57.68 59.33 61.00 62.03 63.11 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Algebra Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 % Problems Solved 35.80 36.66 43.86 51.28 56.31 59.50 61.31 61.98 62.45 62.45 62.54 35.92 43.10 49.14 54.44 58.62 62.16 64.84 66.61 67.74 68.75 69.46 35.80 43.57 49.14 53.34 55.71 57.68 58.87 59.84 60.71 61.26 60.84 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Algebra Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 % Problems Solved 35.80 36.66 43.86 51.28 56.31 59.50 61.31 61.98 62.45 62.45 62.54 35.87 43.62 50.42 55.84 60.40 63.85 66.35 68.16 69.44 70.38 71.46 35.90 44.48 50.56 55.49 59.21 61.32 62.73 63.68 63.78 63.93 63.78 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Algebra Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 % Problems Solved 13.66 13.85 17.40 20.96 23.53 25.39 26.23 27.44 28.89 30.69 32.53 13.26 16.42 19.26 21.43 22.74 23.64 24.21 25.30 25.59 25.29 24.93 13.48 16.48 19.12 20.52 21.66 22.17 22.89 23.58 24.10 24.78 24.24 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Counting & Probability Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 % Problems Solved 13.66 13.85 17.40 20.96 23.53 25.39 26.23 27.44 28.89 30.69 32.53 13.80 17.72 20.71 23.28 25.01 26.49 27.55 28.97 30.42 31.48 34.21 13.43 17.82 22.07 24.48 25.69 27.36 28.54 28.44 29.28 28.22 26.94 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Counting & Probability Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 19.60 20.04 24.80 28.88 32.01 34.45 36.58 38.51 39.08 39.67 39.97 19.72 23.24 26.59 30.18 32.91 35.66 38.39 41.03 43.36 44.48 45.29 19.51 23.66 26.49 27.74 28.48 29.11 29.67 30.27 30.38 31.41 31.75 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Counting & Probability Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 % Problems Solved 19.60 20.04 24.80 28.88 32.01 34.45 36.58 38.51 39.08 39.67 39.97 19.71 24.70 28.36 31.67 34.30 36.12 37.97 39.86 41.93 43.28 44.31 19.86 24.83 27.81 30.07 31.95 32.73 33.50 33.84 33.89 34.75 35.44 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Counting & Probability Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 40 45 % Problems Solved 13.46 14.26 17.63 22.22 26.02 29.12 31.16 32.49 33.22 32.61 31.92 13.73 18.29 22.80 25.41 29.66 33.39 36.84 39.91 42.37 44.22 45.22 13.73 18.54 22.96 25.96 28.49 30.96 32.92 34.85 36.58 37.25 39.34 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Geometry Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 40 45 50 % Problems Solved 13.46 14.26 17.63 22.22 26.02 29.12 31.16 32.49 33.22 32.61 31.92 13.81 18.56 22.50 26.69 30.44 34.52 37.73 40.82 43.39 44.92 45.58 13.31 19.36 24.25 27.23 30.90 33.55 35.94 38.77 40.40 42.05 42.91 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Geometry Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 50 % Problems Solved 19.20 20.35 24.77 31.63 37.46 41.92 44.53 46.17 46.20 45.73 45.24 19.58 25.36 30.38 34.86 38.81 42.47 45.21 46.97 47.65 47.60 47.48 18.91 25.30 31.35 34.36 37.00 39.04 39.70 39.96 40.71 41.68 41.96 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Geometry Problems 12481632641282565121024 N = number of solutions per problem 20 30 40 50 % Problems Solved 19.20 20.35 24.77 31.63 37.46 41.92 44.53 46.17 46.20 45.73 45.24 19.38 25.22 31.19 35.71 40.09 43.35 47.08 50.17 51.76 52.80 53.25 19.39 25.48 30.73 35.27 38.09 40.90 42.39 42.63 42.93 42.17 43.00 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Geometry Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 % Problems Solved 6.60 7.43 9.33 12.18 14.69 17.44 19.20 20.30 21.04 21.56 21.83 6.43 8.92 11.31 13.75 16.23 18.94 20.34 21.96 22.93 23.40 23.76 6.44 9.02 11.79 14.25 16.50 18.27 19.39 20.52 21.32 21.67 21.63 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Intermediate Algebra Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 % Problems Solved 6.60 7.43 9.33 12.18 14.69 17.44 19.20 20.30 21.04 21.56 21.83 6.45 8.96 11.26 14.12 16.43 18.83 20.71 22.50 23.83 24.95 25.28 6.62 9.37 12.19 15.06 17.90 20.28 21.86 23.35 24.58 25.83 27.07 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Intermediate Algebra Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 % Problems Solved 8.14 9.65 12.10 15.48 19.40 21.51 22.87 23.27 23.65 23.78 23.45 8.41 11.15 14.09 17.12 19.88 22.17 24.32 26.26 27.90 29.04 29.64 8.38 11.60 14.81 17.62 19.93 21.38 22.79 24.12 25.13 26.15 27.15 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Intermediate Algebra Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 % Problems Solved 8.14 9.65 12.10 15.48 19.40 21.51 22.87 23.27 23.65 23.78 23.45 8.44 11.58 14.79 18.21 21.36 23.71 25.84 27.52 29.06 30.51 31.47 8.31 11.78 15.41 18.81 21.97 24.39 26.06 28.02 29.30 30.27 30.52 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Intermediate Algebra Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 % Problems Solved 11.33 12.41 15.25 18.76 21.70 23.76 24.55 24.65 24.43 24.23 24.12 11.54 14.27 17.69 20.62 23.72 26.58 29.07 31.00 32.80 33.59 34.65 11.67 14.98 17.67 19.73 22.45 23.56 24.80 25.66 26.74 27.12 28.14 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Number Theory Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 40 % Problems Solved 11.33 12.41 15.25 18.76 21.70 23.76 24.55 24.65 24.43 24.23 24.12 11.37 14.83 19.17 22.28 25.63 28.06 31.26 33.68 34.86 36.75 38.91 11.61 15.86 19.32 23.12 25.60 27.85 30.41 32.67 34.59 36.18 36.87 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Number Theory Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 50 % Problems Solved 19.03 20.03 25.11 30.95 36.23 39.92 42.63 44.77 45.92 46.63 46.85 19.02 23.32 27.59 31.55 35.25 38.47 40.32 42.38 43.76 44.81 46.53 18.83 23.62 27.43 29.95 32.23 33.52 34.36 35.19 35.57 36.42 37.31 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Number Theory Problems 12481632641282565121024 N = number of solutions per problem 15 20 25 30 35 40 45 50 % Problems Solved 19.03 20.03 25.11 30.95 36.23 39.92 42.63 44.77 45.92 46.63 46.85 18.89 24.18 29.44 33.70 37.72 41.88 44.30 46.44 47.06 47.72 48.01 18.86 24.24 29.31 32.96 36.38 38.25 39.97 41.49 40.73 40.80 38.78 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Number Theory Problems 12481632641282565121024 N = number of solutions per problem 25 30 35 40 45 50 55 % Problems Solved 27.16 27.74 34.94 43.45 49.72 52.39 53.73 54.36 54.19 53.36 52.80 27.17 34.62 41.12 46.17 50.05 53.07 54.86 55.41 55.83 56.10 56.00 27.47 35.28 41.86 45.63 47.86 48.99 49.55 49.39 49.06 48.79 47.50 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Prealgebra Problems 12481632641282565121024 N = number of solutions per problem 30 40 50 60 % Problems Solved 27.16 27.74 34.94 43.45 49.72 52.39 53.73 54.36 54.19 53.36 52.80 26.97 35.14 43.42 49.15 54.10 58.26 60.73 62.57 63.69 64.36 64.27 26.88 36.27 43.88 49.84 54.50 57.98 59.77 60.97 61.90 62.39 62.81 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Prealgebra Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 % Problems Solved 37.83 37.88 47.33 56.75 62.55 66.14 67.41 67.79 67.65 67.34 67.23 36.94 45.87 52.31 57.53 61.11 64.10 65.74 66.68 67.92 68.56 68.68 37.01 46.56 52.33 55.50 57.28 58.54 58.40 58.06 57.51 56.70 55.27 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Prealgebra Problems 12481632641282565121024 N = number of solutions per problem 35 40 45 50 55 60 65 70 % Problems Solved 37.83 37.88 47.33 56.75 62.55 66.14 67.41 67.79 67.65 67.34 67.23 37.04 46.88 55.02 60.77 64.55 67.41 69.11 69.96 70.25 70.45 70.98 36.90 47.87 55.53 60.18 62.76 63.80 64.19 63.64 63.10 62.36 61.47 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Prealgebra Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 % Problems Solved 7.13 7.58 9.34 11.38 13.65 15.12 16.30 17.36 17.91 18.38 18.50 7.32 9.59 12.44 15.10 18.28 20.64 22.96 25.26 26.91 28.54 29.95 7.22 10.04 12.91 16.09 18.72 21.00 23.30 24.48 24.87 25.13 24.81 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-7b (on PRM800K) Accuracy on Precalculus Problems 12481632641282565121024 N = number of solutions per problem 5 10 15 20 25 30 % Problems Solved 7.13 7.58 9.34 11.38 13.65 15.12 16.30 17.36 17.91 18.38 18.50 7.43 9.93 12.67 15.25 18.06 20.12 22.00 23.97 25.80 27.05 27.09 7.30 10.43 13.71 16.80 20.00 22.03 23.82 24.70 25.57 25.91 25.90 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-7b + OPRM-34b (on PRM800K) Accuracy on Precalculus Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 % Problems Solved 10.53 11.37 14.52 18.30 21.27 23.31 24.61 25.66 26.32 26.57 26.27 10.80 14.19 17.78 20.42 23.07 25.07 26.99 28.23 29.33 29.89 29.81 10.85 14.38 17.73 20.38 22.99 25.01 26.75 28.23 29.00 29.00 26.91 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-7b (on PRM800K) Accuracy on Precalculus Problems 12481632641282565121024 N = number of solutions per problem 10 15 20 25 30 35 % Problems Solved 10.53 11.37 14.52 18.30 21.27 23.31 24.61 25.66 26.32 26.57 26.27 10.86 15.03 18.34 21.41 23.93 26.62 28.44 29.24 29.81 30.23 30.35 10.86 15.33 18.80 22.01 24.73 26.48 28.15 28.90 29.34 30.07 30.94 Majority Voting Weighted Voting w/ RM Best-of-N w/ RM SFT-34b + OPRM-34b (on PRM800K) Accuracy on Precalculus Problems Figure 20: Easy-to-hard generalization for different type’s data. Both SFTs and OPRMs are trained on the easy data. Each row compares the performance of different OPRMs’ reranking strategies across different types’ data. To determine which types of data benefit more from OPRM’s easy-to-hard generalization and which types still struggle with this challenging generalization, we compare OPRM’s generalization abilities on different problem types in Figure 20. Among Algebra, Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Prealgebra, and Precalculus problems, OPRMs generalize best on Algebra, Intermediate Algebra, and Precalculus problems. Conversely, OPRMs generalize worst on Counting & Probability problems. These findings are highly valuable for practical system design, allowing us to decide when to use OPRMs to enhance performance based on the downstream data type. 35 O Pass@N Analysis for Different Math Questions 12481632641282565121024 N = number of solutions per problem 20 40 60 80 100 % Problems Solved 41.84 57.89 72.52 83.93 91.40 95.10 97.20 98.36 98.85 99.44 99.88 33.41 47.21 59.60 71.71 80.58 87.51 91.27 93.78 95.92 97.14 98.05 18.06 28.84 41.93 55.34 68.44 78.24 85.59 90.08 92.90 95.02 96.20 10.51 17.20 26.75 37.17 47.75 57.85 67.03 74.22 79.75 83.99 87.17 3.48 6.48 11.18 18.32 27.88 38.83 49.89 60.26 69.64 76.69 81.75 Level 1 Level 2 Level 3 Level 4 Level 5 SFT-7b (on PRM800K) Pass@N on Problems by Level 12481632641282565121024 N = number of solutions per problem 20 40 60 80 100 % Problems Solved 52.99 69.65 82.44 90.38 94.77 97.38 98.66 99.41 99.89 100.00100.00 43.49 58.38 70.79 80.10 86.29 89.84 92.46 94.91 96.93 98.26 98.90 26.52 39.22 52.99 66.06 76.00 83.81 88.99 91.55 93.66 95.47 96.95 15.20 24.18 34.28 44.61 55.39 65.38 73.53 80.03 85.02 88.14 90.32 5.34 9.64 16.09 24.97 34.92 45.66 55.66 65.08 72.48 78.36 82.71 Level 1 Level 2 Level 3 Level 4 Level 5 SFT-34b (on PRM800K) Pass@N on Problems by Level 12481632641282565121024 N = number of solutions per problem 20 40 60 80 100 % Problems Solved 27.74 40.01 52.36 64.81 74.92 82.64 88.07 91.42 93.92 95.55 96.73 27.13 40.18 55.07 68.59 80.69 89.50 94.80 97.59 98.76 99.18 99.50 13.52 21.33 30.19 39.09 48.93 60.61 71.41 81.00 88.81 95.26 98.69 11.65 19.12 29.63 41.80 53.21 64.49 74.17 82.30 88.41 93.11 96.41 6.62 10.73 17.05 24.71 33.12 42.34 52.09 61.04 68.58 74.42 78.49 7.19 11.64 16.78 24.01 31.90 39.55 47.04 55.29 62.59 68.74 73.79 14.02 21.05 29.69 40.59 52.15 62.13 71.95 78.48 83.88 86.94 89.50 Prealgebra Counting & Probability Algebra Number Theory Geometry Intermediate Algebra Precalculus SFT-7b (on PRM800K) Pass@N on Problems by Category 12481632641282565121024 N = number of solutions per problem 20 40 60 80 100 % Problems Solved 35.51 48.80 60.90 72.02 81.04 87.67 92.11 94.59 96.12 97.09 97.70 37.55 52.53 66.93 79.39 88.49 94.34 96.73 98.22 99.10 99.76 99.98 19.48 28.68 37.48 48.68 62.27 75.67 86.16 92.65 96.36 98.46 99.75 18.57 29.64 42.96 54.33 65.77 74.57 82.99 89.56 93.66 95.96 96.92 8.38 13.46 20.64 28.16 35.89 44.39 54.51 63.19 70.32 75.19 78.43 11.12 17.08 23.64 29.73 35.48 42.47 49.87 57.90 65.87 73.34 80.53 19.54 28.88 39.51 49.26 58.03 66.07 72.26 78.72 83.79 88.58 92.11 Prealgebra Counting & Probability Algebra Number Theory Geometry Precalculus Intermediate Algebra SFT-34b (on PRM800K) Pass@N on Problems by Category Figure 21: The increase of Pass@N metrics with the number of samples, divided by level and category. As the number of samples increases, the Pass@N rate tends to improve. However, not all problems will eventually yield a correct solution, meaning that even as N approaches infinity, the Pass@N metric may not reach 100%. Figure 21 shows the changes in Pass@N as the number of samples increases. As shown in the figure, not all problems reach a 100% Pass@N rate, even with a larger N. Specifically, three key observations can be made: First, Pass@N is strongly correlated with problem difficulty. As depicted in Figure 21, for Level 1 problems, the Pass@N rate nearly saturates with more samples, while for Level 5 problems, many instances still do not achieve a correct solution. Second, larger models (e.g., 34b) perform significantly better on Pass@1 compared to smaller models (e.g., 7b). However, as N increases, the advantage of the larger model diminishes on metrics like Pass@1024. Third, the variations in Pass@N across different categories of math questions are different, which is expected, as the difficulty of different types of math questions varies for large language models. 36 P Case Studies for OPRM Case Study 1: A robot moving forward at a constant speed takes 2.5 hours to travel 1 kilometer. Moving forward at this same constant speed, it takes the robot 90 seconds to travel the length of a particular hallway. How many meters long is the hallway? Level: 5 GT: 10 Majority Solution:OPRM Solution: Voting V.S. Reward: Answer: 36, Voting times: 139, OPRM reward: 11.3, Answer: 10, Voting times: 99, OPRM reward: 23.7 Let's call the number of meters in the hallway x. 0.97 We know the robot moved 1 km in 2.5 hours. 0.97 That means it moved x meters in 90 seconds. 0.28 We can write that as the equation $\ 12.5=\ x90$. 0.11 To solve for x, we can multiply both sides by 90 to get $36=x$. 0.07 So the hallway is 36 meters long. 0.03 # Answer 36 To solve this problem, I need to find the relationship between the robot's speed, distance, and time. 0.98 I can use the formula distance = speed * time to write an equation for each case. 0.98 Let s be the robot's speed in meters per hour. 0.98 Then the first case gives me 1000 = s * 2.5, or s = 1000 / 2.5 = 400 meters per hour. 0.98 The second case gives me the length of the hallway = s * 90 / 3600, or length = 400 * 90 / 3600 = 10 meters. 0.79 # Answer 10 #Reward#Reward #Final Reward: 6.7e-5 #Final Reward: 0.74 Figure 22: Case Study 1. The evaluator can help identify and reduce the confidence of hallucinations caused by misleading information. Case Study 2: In a regular polygon, the measure of an interior angle is 6.5 times the measure of an exterior angle. How many sides does the polygon have? Level: 4 GT: 15 Majority Solution:OPRM Solution: Voting V.S. Reward: Answer: 0, Voting times: 132, OPRM reward: 0.03, Answer: 15, Voting times: 94, OPRM reward: 17.3 So we know that the measure of an interior angle is 6.5 times the measure of an exterior angle. 0.98 And we know that the sum of the measures of the interior angles of a polygon is given by the formula (n-2)180, where n is the number of sides. 0.89 Right. So we can say that 6.5(n-2)=(n-2)180. 0.22 And that simplifies to 6.5=180, which is false. 0.12 So there is no such polygon. 0.03 # Answer 0 I know that the sum of the interior angles of any polygon with n sides is (n-2) times 180 degrees, and the sum of the exterior angles is 360 degrees. 0.98 I can write an equation relating the interior and exterior angles of a regular polygon: (n-2) * 180 = 6.5 * 360, where n is the number of sides. 0.84 I can simplify this equation by dividing everything by 180: n - 2 = 6.5 * 2, which gives me n - 2 = 13. 0.97 I can add 2 to both sides to isolate n: n = 13 + 2, which gives me n = 15. 0.95 # Answer 15 #Reward#Reward #Final Reward: 6e-4 #Final Reward: 0.75 Figure 23: Case Study 2. The evaluator can assist in reducing the confidence of solutions that misuse mathematical theorems or formulas. We have included more case studies in Figures 22 and 23. We find that evaluator can help generalize to harder ones in the following ways: • The evaluator can help identify and reduce the confidence of hallucinations caused by misleading information in problems. As demonstrated in Case Study 1, the solution selected by majority voting with an answer of 36 is misled by the different units of measurement in the problem (2.5 hours and 90 seconds), resulting in an incorrect solution. Then, the ORPM model successfully gives this solution a low score. •The evaluator can assist in reducing the confidence of solutions that misuse mathematical theorems. In Case Study 2, the majority solution incorrectly applies the theorem "the sum of the exterior angles of a polygon is 360°", leading to erroneous reasoning, and low confidence by the ORPM model. 37 Q Step and Outcome ROC Curve ModelStep ACC (%) Outcome ACC (%) ORM-PRM800K-7B64.371.7 ORM-PRM800K-7B80.463.5 OPRM-PRM800K-7B79.874.4 Table 10: The accuracy of the reasoning steps for differ- ent models. To assess the accuracy of reasoning step judgments of different reward models, we conducted additional experiments using the PRM800K-test data, which includes correctness annotations for each step, to test our model’s ability to distinguish cor- rect reasoning steps. We randomly se- lected a portion of PRM800K-test data to balance positive and negative samples. The accuracy of the reasoning steps for the three models is shown in Table 10. This table demonstrates the effectiveness of our trained PRM, showing that PRM has a significantly greater ability to distinguish steps compared to ORM. Additionally, in Figure 24, we present the Step ROC curves of three models, where PRM and OPRM exhibit better step discrimination abilities compared to ORM. However, it is important to note that a stronger ability to distinguish steps does not necessarily indicate that the evaluator is more helpful for generation. We then also present the Outcome ROC curves of three models on discriminating the final outcome. We collect data generated on MATH500 test set from our 7B policy model. According to the final outcome and groundtruth, we label each data and select a positive-negative balanced set to plot the Outcome ROC curves, where OPRM exhibits better outcome discrimination abilities compared to ORM and PRM. The above table also shows the effectiveness of OPRM on Outcome discrimination ability. 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Step ROC Curve Random Guess (AUC = 0.50) ORM-PRM800K (AUC = 0.55) PRM-PRM800K (AUC = 0.79) OPRM-PRM800K (AUC = 0.76) 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Outcome ROC Curve Random Guess (AUC = 0.50) ORM-PRM800K (AUC = 0.67) PRM-PRM800K (AUC = 0.62) OPRM-PRM800K (AUC = 0.71) Figure 24: The Step and Outcome ROC Curve show the ability of discriminate the reasoning step and the whole reasoning solution, illustrating OPRM’s effectiveness on identifying both steps and outcomes. 38 R Few-Shot Prompt in In-Context Learning We sample with temperature T = 0.9 for ICL-related experiments. We list our few-shot sample for the In-context learning experiments in Table 1. 4-shot example for PRM800K Full ICL: Problem: In right triangle $PQR$, we have $\ Q = \ R$ and $PR = 6\ 2$. What is the area of $\ PQR$? Solution: I see that this is an isosceles right triangle, since $\ Q = \ R = 45^\ $. means that $PQ = PR = 6\ 2$. find the area of a triangle, I need to multiply the base and the height and divide by 2. this case, the base and the height are both $6\ 2$, so the area is $(6\\ sqrt2)(6\ 2)/2$. can simplify this expression by using the property that $\ a\ b = \ ab$. , the area is $(\ 2\ 2)(6\· 6)/2 = 2\· 36/2 = 36$. Final Answer: The final answer is $36$. I hope it is correct. Problem: The length of the longer side of rectangle $R$ is $10$ percent more than the length of a side of square $S.$ The length of the shorter side of rectangle $R$ is $10$ percent less than the length of a side of square $S.$ What is the ratio of the area of rectangle $R$ to the area of square $S?$ Express your answer as a common fraction. Solution: Let the side of the square be $s$. the longer side of the rectangle is $1.1 s$. the shorter side of the rectangle is $0.9s$. area of the square is $s^2$. area of the rectangle is $1.1s * 0.9s = 0.99s^2$. the ratio of the area of rectangle $R$ to the area of square $S$ is $\ 0.99s^2s^2 = 0.99$. , the answer is $\ \ 99100$. Final Answer: The final answer is $\ 99100$. I hope it is correct. Problem: Compute $\ (-60^\ )$. Solution: I know the sine function is periodic with period $360^\ $, so $\ (-60^\ ) = \ (-60^\ + 360^\ ) = \ (300^\ )$. also know that the sine function is symmetrical about the origin, so $\ (-x) = -\ (x)$ for any angle $x$. , $\ (-60^\ ) = -\ (60^\ )$. find $\ (60^\ )$, I can use the special right triangle with angles $30 ^\ $, $60^\ $, and $90^\ $ and sides in the ratio $1:\ 3:2$.\ n sine of an angle is the ratio of the opposite side to the hypotenuse, so $\ (60^\ ) = \ \ 32$. , $\ (-60^\ ) = -\ \ 32$. Final Answer: The final answer is $-\ \ 32$. I hope it is correct. Problem: Simplify $\ (x - y) \ y + \ (x - y) \ y.$ Solution: ’Ok, so we have $\ (x - y) \ y + \ (x - y) \ y$ that $\\ sin (x - y) \ y + \ (x - y) \ y$ is in the form of $\ (\α) \ (\β)+\ (\α)\ (\β)$. , and we know that $\ (\\ alpha)\ (\β)+\ (\α)\ (\β)=\ (\α+\β)$ , $ \ (x - y) \ y + \ (x - y) \ y=\ ((x-y)+y)$ $\ ((x-y)+y )=\ (x)$ Final Answer: The final answer is $\ $. I hope it is correct. 39 4-shot example for PRM800K Hard ICL: Problem: In right triangle $PQR$, we have $\ Q = \ R$ and $PR = 6\ 2$. What is the area of $\ PQR$? Solution: I see that this is an isosceles right triangle, since $\ Q = \ R = 45^\ $. means that $PQ = PR = 6\ 2$. find the area of a triangle, I need to multiply the base and the height and divide by 2. this case, the base and the height are both $6\ 2$, so the area is $(6\\ sqrt2)(6\ 2)/2$. can simplify this expression by using the property that $\ a\ b = \ ab$. , the area is $(\ 2\ 2)(6\· 6)/2 = 2\· 36/2 = 36$. Final Answer: The final answer is $36$. I hope it is correct. Problem: The length of the longer side of rectangle $R$ is $10$ percent more than the length of a side of square $S.$ The length of the shorter side of rectangle $R$ is $10$ percent less than the length of a side of square $S.$ What is the ratio of the area of rectangle $R$ to the area of square $S?$ Express your answer as a common fraction. Solution: Let the side of the square be $s$. the longer side of the rectangle is $1.1 s$. the shorter side of the rectangle is $0.9s$. area of the square is $s^2$. area of the rectangle is $1.1s * 0.9s = 0.99s^2$. the ratio of the area of rectangle $R$ to the area of square $S$ is $\ 0.99s^2s^2 = 0.99$. , the answer is $\ \ 99100$. Final Answer: The final answer is $\ 99100$. I hope it is correct. Problem: Suppose that $y^3$ varies inversely with $\ [3]z$. If $y=2$ when $z=1$, find the value of $z$ when $y=4$. Express your answer in simplest fractional form. Solution: I know that inverse variation means that the product of the two quantities is constant, so I can write an equation of the form $y^3\·\ [3]z=k$, where $k$ is some constant. find $k$, I can plug in the given values of $y$ and $z$: $2^3\·\ [3]1=k$, which simplifies to $8=k$. I can use this equation to find $z$ when $y=4$: $4^3\·\ [3]z=8$, which implies that $\ [3]z=\ 864=\ 18$. get rid of the cube root, I can cube both sides: $z=\ (\ 18\ )^3=\ 1512$. Final Answer: The final answer is $\ 1512$. I hope it is correct. Problem: Let $d$ be a positive number such that when $109$ is divided by $d$, the remainder is $4.$ Compute the sum of all possible two-digit values of $d$. Solution: This problem involves finding the divisors of a given number, as well as using the concept of remainders. way to approach this is to write the division as a quotient and a remainder, like this: $109 = qd + 4$, where $q$ is the quotient. , I can rearrange this equation to get $105 = qd$, which means that $d$ is a divisor of $105$. , I need to find all the two-digit divisors of $105$. can use prime factorization to help me do this. notice that $105 = 3 \× 5 \× 7$, which are all prime numbers. \ nTherefore, any divisor of $105$ must be a product of some combination of these three factors. possible products are $1, 3, 5, 7, 15, 21, 35, 105$. \ nHowever, not all of these are two-digit numbers. Only $15, 21, 35$ are. \ nThese are the only possible values of $d$ that satisfy the given condition. \ nTo find their sum, I just add them up: $15 + 21 + 35 = 71$. Final Answer: The final answer is $71$. I hope it is correct. 40 4-shot example for PRM800K Easy-to-Hard ICL: Problem: How many 4-letter words with at least one consonant can be constructed from the letters $A$, $B$, $C$, $D$, and $E$? (Note that $B$, $C$, and $D$ are consonants, any word is valid, not just English language words, and letters may be used more than once.) Solution: To count the number of 4-letter words with at least one consonant, I can use the complement principle and subtract the number of 4-letter words with no consonants from the total number of 4-letter words. total number of 4- letter words is simply $5^4$, since each letter has 5 choices and the order matters. number of 4-letter words with no consonants is just the number of ways to choose 4 letters from the 2 vowels $A$ and $E$, which is $2^4$. \ nTherefore, the number of 4-letter words with at least one consonant is $5^4 - 2^4$. 625 - 16 = 609 Final Answer: The final answer is $609$. I hope it is correct. Problem: Compute the integer $k > 2$ for which \\[\ _10 (k - 2)! + \ _10 (k - 1)! + 2 = 2 \ _10 k!.\\] Solution: I recognize that this equation involves logarithms of factorials, which are products of consecutive integers. also know that logarithms have some useful properties, such as $\ _10 a + \ _10 b = \ _10 (ab)$ and $c \ _10 d = \ _10 d^c$. these properties, I can simplify the equation as follows: \\[\ _10 (k - 2)! + \ _10 (k - 1)! + 2 = 2 \ _10 k! \ \ _10 \ [ (k - 2)! (k - 1)! 100 \ ] = \\ log_10 (k!)^2.\\] the bases of the logarithms are equal, I can conclude that the arguments must also be equal, i.e., \\[(k - 2)! (k - 1)! 100 = (k!)^2.\\] I have a simpler equation to solve for $k$. notice that the left-hand side has a factor of $(k - 2)!$, which is also a factor of $ (k - 1)!$ and $(k!)^2$. , I can divide both sides by $(k - 2)!$ to get \\[(k - 1)! 100 = k! (k - 1)!\\] further, I get \\[100 = k !.\\] means that $k$ is the smallest integer such that its factorial is greater than or equal to 100. can try some values of $k$ to find the right one. $k = 3$, then $k! = 3! = 6 < 100$. $k = 4$, then $k! = 4! = 24 < 100$. $k = 5$, then $k! = 5! = 120 > 100$. , $k = 5 $ is the smallest integer that satisfies the equation. Final Answer: The final answer is $5$. I hope it is correct. Problem: What is the largest perfect square factor of 1512? Solution: I know that 1512 is divisible by 2, so I can try to factor out powers of 2 until I get an odd number. by 2 once, I get 756. by 2 again, I get 378. by 2 one more time, I get 189. 189 is odd, so I can ’t divide by 2 anymore. notice that 189 is divisible by 3, so I can try to factor out powers of 3. by 3 once, I get 63. by 3 again, I get 21. by 3 one more time, I get 7. 7 is prime, so I can’t factor it anymore. have factored 1512 as 2^3 * 3^3 * 7. find the largest perfect square factor, I need to use the highest even powers of each prime factor. highest even power of 2 that I have is 2^2. highest even power of 3 that I have is 3^2. highest even power of 7 that I have is 7^0, which is just 1. the largest perfect square factor is 2^2 * 3^2 * 7^0, which is 36. Final Answer: The final answer is $36$. I hope it is correct. Problem: If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to? Solution: 41 I notice that the coefficients of $p$ and $q$ are reversed in the two equations, so I wonder if I can use elimination to get rid of one of them. try multiplying the first equation by 4 and the second equation by 3, and then subtracting them to eliminate $p$. get $12p+16q-12p-9q=32-39$, which simplifies to $7q=-7$. both sides by 7, I get $q=-1$. Final Answer: The final answer is $-1$. I hope it is correct. 42 4-shot example for MetaMath Full ICL: Problem: In right triangle $PQR$, we have $\ Q = \ R$ and $PR = 6\ 2$. What is the area of $\ PQR$? Solution: Since $\ Q = \ R$, we have $\ PQR$ is an isosceles right triangle. , $PQ = PR = 6\ 2$. area of $\ PQR$ is $\ 12 \· PQ \· PR = \ 12 \· 6\ 2 \· 6\\ sqrt2 = \ 36$. Final Answer: The final answer is $36$. I hope it is correct. Problem: The length of the longer side of rectangle $R$ is $10$ percent more than the length of a side of square $S.$ The length of the shorter side of rectangle $R$ is $10$ percent less than the length of a side of square $S.$ What is the ratio of the area of rectangle $R$ to the area of square $S?$ Express your answer as a common fraction. Solution: Let the side length of square $S$ be $x$. the longer side of rectangle $R$ is $1.1x$ and the shorter side is $0.9x$. area of square $S$ is $x^2$ and the area of rectangle $R$ is $(1.1x)(0.9x) = 0.99x^2$. , the ratio of the area of rectangle $R$ to the area of square $S$ is $\ 0.99x^2x^2 = \ \ 99100$. Final Answer: The final answer is $\ 99100$. I hope it is correct. Problem: Compute the integer $k > 2$ for which \\[\ _10 (k - 2)! + \ _10 (k - 1)! + 2 = 2 \ _10 k!.\\] Solution: Using the properties of logarithms, we can rewrite the equation as \\[\ _10 ((k-2)! \· (k-1)!) + 2 = \ _10 (k!)^2.\\] , using the fact that $ \ _a b + \ _a c = \ _a (bc)$, we have \\[\ _10 ((k-2)! \\ cdot (k-1)! \· 100) = \ _10 (k!)^2.\\] , using the fact that $ \ _a b = \ _a c$ if and only if $b = c$, we have \\[(k-2)! \· (k -1)! \· 100 = (k!)^2.\\] , we have \\[(k-2)! \· (k-1)! \\ cdot 100 = (k \· (k-1) \· (k-2))!.\\] out the common factors, we have \\[100 = k \· (k-1) \· (k-2).\\] , we find that $100 = 5 \· 5 \· 4 = 2^2 \· 5^2$. $k > 2$, we see that $k = \ 5$. Final Answer: The final answer is $5$. I hope it is correct. Problem: If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to? Solution: We can solve this system of equations using the method of elimination. the first equation by 3 and the second equation by 4 gives us: $9p+12q=24$\ n$16p+12q=52$ can subtract the first equation from the second equation to eliminate the q term: $16p+12q-(9p+12q)=52-24$ $7p=28$ both sides by 7 gives us: $p=4$ this value of p into either of the original equations gives us: $4(4)+3q=13$ $16+3q=13$ $3q=-3$ $q=\ -1$ Final Answer: The final answer is $-1$. I hope it is correct. 43 4-shot example for MetaMath Hard ICL: Problem: In right triangle $PQR$, we have $\ Q = \ R$ and $PR = 6\ 2$. What is the area of $\ PQR$? Solution: Since $\ Q = \ R$, we have $\ PQR$ is an isosceles right triangle. , $PQ = PR = 6\ 2$. area of $\ PQR$ is $\ 12 \· PQ \· PR = \ 12 \· 6\ 2 \· 6\\ sqrt2 = \ 36$. Final Answer: The final answer is $36$. I hope it is correct. Problem: The length of the longer side of rectangle $R$ is $10$ percent more than the length of a side of square $S.$ The length of the shorter side of rectangle $R$ is $10$ percent less than the length of a side of square $S.$ What is the ratio of the area of rectangle $R$ to the area of square $S?$ Express your answer as a common fraction. Solution: Let the side length of square $S$ be $x$. the longer side of rectangle $R$ is $1.1x$ and the shorter side is $0.9x$. area of square $S$ is $x^2$ and the area of rectangle $R$ is $(1.1x)(0.9x) = 0.99x^2$. , the ratio of the area of rectangle $R$ to the area of square $S$ is $\ 0.99x^2x^2 = \ \ 99100$. Final Answer: The final answer is $\ 99100$. I hope it is correct. Problem: Suppose that $y^3$ varies inversely with $\ [3]z$. If $y=2$ when $z=1$, find the value of $z$ when $y=4$. Express your answer in simplest fractional form. Solution: Since $y^3$ varies inversely with $\ [3]z$, we can write the equation as $y ^3\ [3]z=k$, where $k$ is a constant. are given that $y=2$ when $z=1$ , so we can substitute these values into the equation to solve for $k$: $2^3\\ sqrt[3]1=k$ $8\· 1=k$ $k=8$ we can use this value of $k$ to find $z$ when $y=4$: $4^3\ [3]z=8$ $64\ [3]z=8$ $\ [3]z=\ 864=\ 18$ $z=\ (\ 18\ )^3=\ 1512$ when $y=4$, $z=\ \ 1512$. Final Answer: The final answer is $\ 1512$. I hope it is correct. Problem: Let $d$ be a positive number such that when $109$ is divided by $d$, the remainder is $4.$ Compute the sum of all possible two-digit values of $d$. Solution: If $109$ is divided by $d$ and the remainder is $4$, then $d$ divides $109-4=105$.\ nThe prime factorization of $105$ is $3\·5\·7$. $d$ must be a factor of $105$, the possible values of $d$ are $1, 3, 5, 7, 15, 21, 35, 105$.\ nOut of these, only the two-digit values are $15, 21, 35$. sum of these values is $15+21+35=\ 71$. Final Answer: The final answer is $71$. I hope it is correct. 44 4-shot example for MetaMath Easy-to-Hard ICL: Problem: How many 4-letter words with at least one consonant can be constructed from the letters $A$, $B$, $C$, $D$, and $E$? (Note that $B$, $C$, and $D$ are consonants, any word is valid, not just English language words, and letters may be used more than once.) Solution: There are a total of $5^4 = 625$ possible 4-letter words that can be constructed from the given letters. count the number of words with no consonants, we can use the principle of complementary counting. there are 2 vowels ($A$ and $E$) and 3 consonants ($B$, $C$, and $D$), there are $2^4 = 16$ words with only vowels. , the number of words with at least one consonant is $625 - 16 = \ 609$. Final Answer: The final answer is $609$. I hope it is correct. Problem: Compute the integer $k > 2$ for which \\[\ _10 (k - 2)! + \ _10 (k - 1)! + 2 = 2 \ _10 k!.\\] Solution: Using the properties of logarithms, we can rewrite the equation as \\[\ _10 ((k-2)! \· (k-1)!) + 2 = \ _10 (k!)^2.\\] , using the fact that $ \ _a b + \ _a c = \ _a (bc)$, we have \\[\ _10 ((k-2)! \\ cdot (k-1)! \· 100) = \ _10 (k!)^2.\\] , using the fact that $ \ _a b = \ _a c$ if and only if $b = c$, we have \\[(k-2)! \· (k -1)! \· 100 = (k!)^2.\\] , we have \\[(k-2)! \· (k-1)! \\ cdot 100 = (k \· (k-1) \· (k-2))!.\\] out the common factors, we have \\[100 = k \· (k-1) \· (k-2).\\] , we find that $100 = 5 \· 5 \· 4 = 2^2 \· 5^2$. $k > 2$, we see that $k = \ 5$. Final Answer: The final answer is $5$. I hope it is correct. Problem: If $n$ is $1$ less than a multiple of $50$, then what is the remainder when $n^2+2n +3$ is divided by $50$? Solution: Since $n$ is $1$ less than a multiple of $50$, we can write $n=50k-1$ for some integer $k$. \\[n^2+2n+3=(50k-1)^2+2(50k-1)+3=2500k^2-96k+2,\\] so the remainder when $n^2+2n+3$ is divided by $50$ is $\ 2$. Final Answer: The final answer is $2$. I hope it is correct. Problem: If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to? Solution: We can solve this system of equations using the method of elimination. the first equation by 3 and the second equation by 4 gives us: $9p+12q=24$\ n$16p+12q=52$ can subtract the first equation from the second equation to eliminate the q term: $16p+12q-(9p+12q)=52-24$ $7p=28$ both sides by 7 gives us: $p=4$ this value of p into either of the original equations gives us: $4(4)+3q=13$ $16+3q=13$ $3q=-3$ $q=\ -1$ Final Answer: The final answer is $-1$. I hope it is correct. 45