Paper deep dive
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
Hoang Phan, Minh Pham, Chau Pham, Chinmay Hegde, Trung Le, Qi Lei
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/7/2026, 9:10:53 AM
Summary
The paper introduces Rationale-Guided Policy Optimization (RGPO), a framework that mitigates reward sparsity in on-policy reinforcement learning for large language models. RGPO uses ground-truth rationales as temporary scaffolds rather than fixed imitation targets. It adaptively adjusts the amount of rationale guidance based on the model's current capability: if the model fails, more hints are revealed; if it succeeds, hints are removed. This approach improves reasoning performance in both language-only and vision-language settings compared to baselines like RLVR.
Entities (6)
Relation Signals (5)
RGPO → mitigates → reward sparsity
confidence 95% · These results suggest that RGPO offers a practical and general approach for reducing reward sparsity...
RGPO → uses → Scaffolding
confidence 93% · Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds...
Adaptive Hinting → iscomponentof → RGPO
confidence 92% · we introduce an adaptive hint mechanism... This adaptive mechanism allows RGPO to concentrate guidance on difficult examples...
RGPO → outperforms → RLVR
confidence 90% · Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines...
GRPO → iscompatiblewith → RGPO
confidence 85% · RGPO is compatible with either PPO or GRPO: it leaves the first-round RL update intact and adds a second-round reward-gated distillation term.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate. Existing approaches mitigate this issue by incorporating off-policy demonstrations, expert traces, or model-generated solutions, but they typically require the auxiliary data to match the format of the reinforcement-learning task, often relying on rejection sampling from stronger models to obtain suitable training trajectories. We introduce Rationale-Guided Policy Optimization (RGPO), a framework that adaptively leverages ground-truth rationale information according to the model's current capability while preserving its freedom to explore. Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds: rationales help the model generate improved responses, after which only higher-reward, model-generated solutions are transferred back to the original unguided setting. This design allows training to exploit available ground-truth information without requiring off-policy data to follow the same format as the RL task. Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines, and ablation studies show that adaptive rationale guidance is a key contributor to these gains. These results suggest that RGPO offers a practical and general approach for reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning performance in both text-only and multimodal models.
Tags
Links
- Source: https://arxiv.org/abs/2610.07342v1
- Canonical: https://arxiv.org/abs/2610.07342v1
Trouble viewing inline? Open PDF directly →
Full Text
137,903 characters extracted from source content.
Expand or collapse full text
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding Hoang Phan Affiliation: New York University Email: hvp2011@nyu.edu Minh Pham Affiliation: New York University Email: mnpham@nyu.edu Chau Pham Affiliation: University at Buffalo Email: haichaup@buffalo.edu Chinmay Hegde Affiliation: New York University Email: chinmay.h@nyu.edu Trung Le Affiliation: Monash University Email: trunglm@monash.edu Qi Lei Affiliation: New York University Email: ql518@nyu.edu Abstract On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate. Existing approaches mitigate this issue by incorporating off-policy demonstrations, expert traces, or model-generated solutions, but they typically require the auxiliary data to match the format of the reinforcement-learning task, often relying on rejection sampling from stronger models to obtain suitable training trajectories. We introduce Rationale-Guided Policy Optimization (RGPO), a framework that adaptively leverages ground-truth rationale information according to the model’s current capability while preserving its freedom to explore. Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds: rationales help the model generate improved responses, after which only higher-reward, model-generated solutions are transferred back to the original unguided setting. This design allows training to exploit available ground-truth information without requiring off-policy data to follow the same format as the RL task. Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines, and ablation studies show that adaptive rationale guidance is a key contributor to these gains. These results suggest that RGPO offers a practical and general approach for reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning performance in both text-only and multimodal models. Our implementation is available at https://github.com/VietHoang1512/rgpo. 1 Introduction Recent advances in large language model reasoning have shown that reinforcement learning, particularly reinforcement learning with verifiable rewards (RLVR), can elicit long-horizon reasoning behaviors from pretrained models. Current frontier models, including GPT-6 Astra (OpenAI, 2026), Gemini 4 Argon (Google DeepMind, 2026), Claude Opus 5.5 (Anthropic, 2026), DeepSeek-V4.1-Flash (DeepSeek-AI, 2026), and Kimi K3 (Team et al., 2026), have continued to advance performance in mathematical reasoning, coding, tool use, and long-horizon agentic tasks. Reinforcement learning has become an important component of post-training for frontier reasoning systems, enabling models to improve beyond supervised demonstrations through task-level feedback (Schulman et al., 2017; Shao et al., 2024; OpenAI, 2026; DeepSeek-AI, 2026). Despite its effectiveness, on-policy reinforcement learning remains fundamentally constrained by the current capability of the policy model. When the model is trained on problems beyond its evolving reasoning ability, all sampled rollouts for a prompt may be incorrect. In such cases, rewards become sparse or uniform, and the resulting learning signal provides little information about how to reach a correct solution. This capacity–difficulty mismatch is especially severe for weak or small models, where standard on-policy RL may spend substantial computation sampling failed trajectories without obtaining positive examples for improvement (Yan et al., 2025; Liu et al., 2025; Zhang et al., 2026). Consequently, the training process can stagnate precisely on the hard examples that are most important for improving reasoning. A natural direction for mitigating reward sparsity is to incorporate external guidance, such as reference solutions or expert traces (Yan et al., 2025; Wang et al., 2025). Prior work has explored several forms of such guidance. However, directly using external rationales also introduces important challenges. Maximizing the likelihood of those reference solutions may force the model to imitate trajectories that are far from its own policy distribution, causing distribution mismatch, memorization, and limited generalization Chu et al. (2025). Off-policy expert traces can provide successful solutions, but they may not reflect the reasoning style or token-level decisions that the current model can naturally reproduce. Human learning research suggests that effective reasoning instruction requires a careful balance between independent problem solving and guided assistance. Since initial struggle can prepare learners to better recognize, organize, and internalize later guidance (Kapur, 2008; Schwartz and Bransford, 1998), they can benefit from attempting a problem before receiving explicit instruction. At the same time, intelligent tutoring research identifies an “assistance dilemma”: too little help may leave learners stuck, whereas too much help may reduce effort, encourage shallow processing, and weaken transfer (Koedinger and Aleven, 2007). Motivated by this perspective, we propose Rationale-Guided Policy Optimization (RGPO), a simple yet effective framework that builds on the standard RLVR training loop for improving reasoning by using reference rationales as temporary scaffolds for exploration rather than as direct imitation targets. For each prompt, the model first generates ordinary on-policy rollouts and receives rewards from the verifier. Then, for examples where guidance may be useful, RGPO constructs a second-round refinement prompt that includes the original problem together with a rationale, solution hint, or feedback derived from the training example. The model is asked to generate a new standalone solution to the original problem. If this yields a better response than the original rollout, RGPO applies a supervised update to maximize the likelihood of the refined response conditioned only on the original prompt, rather than on the hint-augmented prompt. Figure 1: Overview of the proposed method: Inspired by how humans approach a problem, the model first tries to solve each problem without access to hints. When it succeeds, the correct response is directly reinforced. When it fails, progressively stronger rationales are introduced as temporary scaffolds to guide the model toward a correct solution. To further balance guidance and exploration, we introduce an adaptive hinting mechanism. Rather than always revealing the full reference solution, the scheduler adjusts the amount of guidance according to the model’s current success on each problem (see Figure 1). If the model repeatedly fails to solve a problem, RGPO reveals more of the rationale to help it reach a successful trajectory. If the model can already solve the problem, the scheduler reduces or removes the hint, encouraging independent reasoning and avoiding unnecessary simplification. This adaptive mechanism allows RGPO to concentrate guidance on difficult examples while preserving on-policy exploration for problems within the model’s current capability. Overall, our contributions are summarized as follows: • We propose RGPO, a rationale-guided optimization framework that augments RLVR with hint-assisted refinement. In particular, we show how reference rationales can be used as temporary reasoning scaffolds rather than direct imitation targets, reducing reward sparsity without requiring the rationale to match the final-answer format used in RL rollouts. • We introduce an adaptive hint mechanism that dynamically increases or decreases the amount of revealed solution guidance according to the model’s current ability. • We conduct experiments on both language models and vision-language models, where RGPO consistently shows superior training sample efficiency over standard RLVR and outperforms other hybrid baselines that combine both on-policy and off-policy training. 2 Related Work RL with verifiable rewards. PPO optimizes a clipped policy-gradient objective and is widely used for language-model post-training (Schulman et al., 2017). GRPO replaces value estimation with group-relative reward normalization, reducing training complexity for mathematical reasoning (Shao et al., 2024). DeepSeek-R1 further demonstrated that large-scale RLVR can elicit sophisticated reasoning behaviors without relying entirely on supervised reasoning traces (DeepSeek-AI, 2025). RGPO is compatible with either PPO or GRPO: it leaves the first-round RL update intact and adds a second-round reward-gated distillation term. Off-policy guidance and mixed SFT–RL training. A major limitation of pure on-policy RL is that the policy can only learn from trajectories it already samples. LUFFY addresses this by incorporating off-policy reasoning traces while using policy shaping to avoid rigid imitation (Yan et al., 2025). CHORD reframes SFT as a dynamically weighted auxiliary objective within RL (Zhang et al., 2025b). Minimalist rejection-sampling and reinforce-style methods also show that filtering positive trajectories can be a strong baseline (Xiong et al., 2025). RGPO differs in that its auxiliary examples are generated online by the current policy under rationale-guided refinement, then trained under the original prompt distribution. GHPO adjusts the amount of guidance according to the model’s ability to solve the problem (Liu et al., 2025). QuestA augments hard questions with partial solutions (Li et al., 2025b). BREAD inserts expert prefixes and branches rollouts from them (Zhang et al., 2026). HINT supplies heuristic hints and argues that external guidance should preserve training affinity (Wang et al., 2025). Failure-conditioned guidance. The design of RGPO is primarily motivated by cognitive studies on when and how guidance should be provided. Effective reasoning instruction depends on both the timing and form of guidance, as learners often benefit from attempting problems before receiving instruction, because initial struggle can help them notice relevant structures, activate prior knowledge, and better internalize subsequent explanations (Kapur, 2008; Schwartz and Bransford, 1998). Besides, active recall and testing improve long-term retention more effectively than passive study, motivating an initial phase of independent answer generation (Roediger I and Karpicke, 2006). The “assistance dilemma” (Koedinger and Aleven, 2007) states that too little help may leave learners stuck, while too much help may reduce effort, encourage shallow processing, and weaken transfer. Studies of help seeking also show that learners may engage in hint abuse, using assistance to obtain answers rather than to understand the underlying reasoning process (Aleven et al., 2004). Complementary work on worked examples demonstrates that examples can support schema acquisition, but their benefits depend on active engagement, such as self-explanation, rather than copying solution steps (Sweller and Cooper, 1985; Chi et al., 1989). Finally, fading worked-out steps helps learners transition from guided examples to independent problem solving, suggesting that explanations and hints are most useful as temporary scaffolds rather than permanent supports (Atkinson et al., 2003). 3 Background 3.1 Supervised Fine-Tuning Supervised fine-tuning (SFT) is one of the most widely used post-training procedures for adapting a pretrained language model to a target task, domain, or response format. Given a dataset of prompt–response pairs sft=(xi,yi)i=1ND_sft=\(x_i,y_i)\_i=1^N, SFT optimizes the model by maximizing the conditional likelihood of the reference response: ℒsft(θ)=−(x,y)∼sft[∑t=1|y|logπθ(yt∣x,y<t)].L_sft(θ)=-E_(x,y) _sft [ _t=1^|y| _θ(y_t x,y_<t) ]. (1) In instruction-following pipelines, SFT is commonly used to initialize the policy before preference-based or reward-based optimization. For example, Ouyang et al. (Ouyang et al., 2022) first fine-tuned a language model on human-written demonstrations and then further optimized it using reinforcement learning from human feedback. In reasoning tasks, SFT can teach the model answer formatting, step-by-step reasoning style, and task-specific conventions. However, human-written or oracle solutions may be concise or stylistically different from model-generated reasoning (i.e. outside the model’s current reasoning distribution). As a result, simply maximizing the likelihood of such demonstrations can lead to superficial imitation or memorization (Chu et al., 2025), rather than improving the model’s ability to independently discover correct reasoning trajectories. This limitation can be particularly pronounced for smaller models when expert traces impose a reasoning or output format that is difficult for the current policy to reproduce. Appendix D.5 shows that the 2B model struggles to optimize under the Mulberry-style (Yao et al., 2024) template. 3.2 Policy Optimization for Language Model Reasoning Reinforcement learning (RL) provides an alternative post-training paradigm in which the model is optimized according to a reward signal rather than fixed demonstrations. For reasoning tasks, the reward can often be automatically verified, for example by checking whether the final answer matches the ground truth. Given a prompt x, the policy πθ _θ samples a response y, and a reward function R(x,y)R(x,y) assigns a scalar score. The goal is to maximize expected reward: J(θ)=x∼,y∼πθ(⋅∣x)[R(x,y)].J(θ)=E_x ,\,y _θ(· x) [R(x,y) ]. (2) Group Relative Policy Optimization (GRPO) Shao et al. (2024) estimates the baseline using a group of responses sampled for the same prompt. For each prompt x, the old policy samples a group of G responses: y1,…,yG∼πθold(⋅∣x).\y_1,…,y_G\ _ _old(· x). (3) Each response receives a reward ri=R(x,yi)r_i=R(x,y_i). The group-relative advantage is then computed by normalizing rewards within the group: A^i=ri−mean(rjj=1G)std(rjj=1G). A_i= r_i-mean(\r_j\_j=1^G)std(\r_j\_j=1^G). (4) Using this relative advantage, GRPO optimizes a PPO-style clipped objective: −[1G∑i=1G1|yi|∑t=1|yi|min(ρi,t(θ)A^i,clip(ρi,t(θ),1−ϵ,1+ϵ)A^i)]+βDKL(πθ∥πref),-E [ 1G _i=1^G 1|y_i| _t=1^|y_i| ( _i,t(θ) A_i,\,clip( _i,t(θ),1-ε,1+ε) A_i ) ]+β D_KL ( _θ\,\|\, _ref ), (5) where we have defined ρi,t(θ)=πθ(yi,t∣x,yi,<t)πθold(yi,t∣x,yi,<t). _i,t(θ)= _θ(y_i,t x,y_i,<t) _ _old(y_i,t x,y_i,<t). (6) REINFORCE++ emphasizes global advantage normalization rather than prompt-local normalization (Hu et al., 2025a). If ℬB denotes a global batch of sampled responses, the normalized advantage can be written as A^(x,y)=R(x,y)−μℬσℬ, A(x,y)= R(x,y)- _B _B, (7) where μℬ _B and σℬ _B are the mean and standard deviation of rewards or advantages over the global batch. This design aims to produce more stable advantage estimates than small prompt-level groups. The resulting objective can be viewed as a critic-free policy-gradient objective: −(x,y)∼ℬ[∑t=1|y|A^(x,y)logπθ(yt∣x,y<t)]+βDKL(πθ∥πref).-E_(x,y) [ _t=1^|y| A(x,y) _θ(y_t x,y_<t) ]+β D_KL ( _θ\,\|\, _ref ). (8) We follow recent reasoning-focused RL pipelines that reduce or remove KL (β=0β=0) both to encourage exploration (Hu et al., 2025b; Hao et al., 2025) and for simplicity. 4 Methodology: Rationale-Guided Policy Optimization 4.1 Problem setup Let =(xi,si,ai)i=1MD=\(x_i,s_i,a_i)\_i=1^M denote the training set, where xix_i is the original prompt, sis_i is a reference rationale, and aia_i is the is the ground-truth answer. We assume access to a verifiable reward function r(x,y,a)∈ℝr(x,y,a) , which assigns reward to a model response y for prompt x given answer a. We train the model for N+1N+1 epochs with the first epoch fully unguided to estimate the model’s initial capability on each training example. The following N epochs adaptively reveal or hide increments of size 1/N1/N of the reference solution according to whether the current model can already solve the example. For each example, we partition the reference solution into N ordered segments, si=[si,1;si,2;…;si,N],s_i=[s_i,1;s_i,2;…;s_i,N], where each segment contains approximately 1/N1/N of the solution. The partition may be defined by token count, sentence boundaries, reasoning steps, or another deterministic segmentation rule. We employ raw character counts for simplicity in our experiments. Let gi(t)∈0,1,…,Ng_i^(t)∈\0,1,…,N\ denote the number of revealed segments for example i at epoch t. The corresponding hint is the prefix hi(t)=[si,1;…;si,gi(t)],h_i^(t)=[s_i,1;…;s_i,g_i^(t)], with hi(t)=∅h_i^(t)= when gi(t)=0g_i^(t)=0. We initialize gi(0)=0g_i^(0)=0 for all examples, so the first epoch uses no hint. After each epoch, the hint level is updated adaptively: if the model produces no correct rollout for an example, then the next epoch reveals one additional segment; if the model produces at least one correct rollout, then the next epoch removes one segment. Formally, if Ci(t)C_i^(t) indicates whether any rollout for example i is correct at epoch t, then gi(t+1)=min(gi(t)+1,N),if Ci(t)=0,max(gi(t)−1,0),if Ci(t)=1.g_i^(t+1)= cases (g_i^(t)+1,N),&if C_i^(t)=0,\\ (g_i^(t)-1,0),&if C_i^(t)=1. cases This update rule creates a curriculum that automatically moves each example toward the smallest amount of hinting needed to keep the problem learnable. Hard examples gradually receive more of the reference solution until the model can discover a correct trajectory, while easier examples are annealed back toward the original no-hint setting. In this way, RGPO avoids a fixed hint schedule that may over-assist easy problems or under-assist hard ones. 4.2 Adaptive rationale-guided rollout The first epoch is a standard on-policy RL epoch without any solution hint. For each prompt xix_i, we sample K rollouts from the current policy, yi,1(0),…,yi,K(0)∼πθ(⋅∣xi),y_i,1^(0),…,y_i,K^(0) _θ(· x_i), and compute their rewards ri,k(0)=r(xi,yi,k(0),ai)r_i,k^(0)=r(x_i,y_i,k^(0),a_i). We estimate the initial solvability of each example under the current policy. We define Ci(0)=[maxkri,k(0)≥τcorrect]C_i^(0)=I[ _kr_i,k^(0)≥ _correct] If Ci(0)=0C_i^(0)=0, then none of the sampled unguided responses solved the problem, and RGPO reveals the first 1/N1/N of the reference solution for this example in the next epoch. If Ci(0)=1C_i^(0)=1, then the example remains unguided in the next epoch because the model has already demonstrated that the problem is learnable without assistance. For epoch t∈1,…,Nt∈\1,…,N\, each example is assigned a hint prefix hi(t)h_i^(t) according to the previous epoch’s correctness. We then perform two rollout passes. The first pass is the ordinary unguided rollout from the original prompt, yi,k(t)∼πθ(⋅∣xi),y_i,k^(t) _θ(· x_i), which is used for the main on-policy RL objective. The second pass is a guided revision rollout, used only when a nonempty hint is available or when feedback from the first pass is included. The guided prompt is constructed as x~i(t)=Reprompt(xi,hi(t),fi(t)), x_i^(t)=Reprompt(x_i,h_i^(t),f_i^(t)), where fi(t)f_i^(t) includes verifier feedback and answer-level correctness, … Note that x~i(t) x_i^(t) is used only to generate candidate improved trajectories. The guided rollout is sampled as y~i,k(t)∼πθ(⋅∣x~i(t)), y_i,k^(t) _θ(· x_i^(t)), and evaluated using the same reward function, r~i,k(t)=r(xi,y~i,k(t),ai). r_i,k^(t)=r(x_i, y_i,k^(t),a_i). We compare the guided candidate to the unguided rollout distribution from the same epoch. A guided response is accepted into an auxiliary buffer only if it improves over the unguided attempts: y~i,k(t)∈ℬRGPO⟺r~i,k(t)>maxjri,j(t)+δ, y_i,k^(t) _RGPO r_i,k^(t)> _jr_i,j^(t)+δ, where δ≥0δ≥ 0 is a margin. In the simplest exact-reward setting, δ=0δ=0, so a guided response is accepted whenever it achieves a higher verifier reward than all unguided rollouts for the same example. This reward gate is crucial because the reference solution prefix may not always help the model reason correctly. Instead of assuming that every hint-conditioned response is useful, RGPO treats the guided pass as a proposal distribution and keeps only proposals that improve measured task reward. 4.3 Policy optimization objective RGPO combines a main on-policy RL objective with an auxiliary reward-gated supervised objective. The RL objective is computed on the unguided rollouts sampled from πθ(⋅∣xi) _θ(· x_i), so the policy is directly optimized for the deployment condition. The auxiliary supervised objective is computed on accepted guided responses, but conditioned on the original prompt: ℒSFT(θ)=−(xi,y~)∈ℬRGPO[logπθ(y~∣xi)].L_SFT(θ)=-E_(x_i, y) _RGPO [ _θ( y x_i) ]. This objective intentionally maximizes pθ(y~∣xi)p_θ( y x_i) rather than pθ(y~∣x~i(t))p_θ( y x_i^(t)). The distinction is central to RGPO, because training on x~i(t) x_i^(t) would teach the model to rely on rationales at inference time. Training on xix_i instead transfers the benefit of the guided revision back into the original task distribution. The full objective for each epoch is ℒtotal(θ)=ℒRL(θ)+λtℒSFT(θ)L_total(θ)=L_RL(θ)+ _tL_SFT(θ) where ℒRLL_RL is the main RLVR objective (e.g. with GRPO or REINFORCE++), and λt _t controls the weight of the auxiliary term. Algorithm 1 gives the complete training procedure. 4.4 Why RGPO helps under sparse rewards Consider binary verifier reward and let p∈[0,1]p∈[0,1] be the success probability of one unguided rollout. RGPO draws K unguided responses and, when guidance is active, KgK_g guided revisions. If all unguided responses fail, let s denote the success probability of one guided revision. Proposition 1. At equal response budget K+KgK+K_g, the probabilities of discovering at least one successful trajectory are PRGPO=1−(1−p)K(1−s)Kg,Pplain=1−(1−p)K+Kg.P_ RGPO=1-(1-p)^K(1-s)^K_g, P_ plain=1-(1-p)^K+K_g. (9) Hence PRGPO>PplainP_ RGPO>P_ plain iff s>ps>p. In particular, on the all-failure event—where group-centered binary-reward GRPO has zero reward-advantage signal—RGPO obtains an accepted positive target with probability (1−p)K[1−(1−s)Kg](1-p)^K[1-(1-s)^K_g]. Hence, at a fixed rollout budget, guidance improves the probability of finding a successful trajectory exactly when the guided success probability exceeds the unguided one. The scheduler reduces the hint level as the model becomes more likely to solve the problem without guidance. For 0<gt<N0<g_t<N, [gt+1−gt∣ℋt]=2(1−pt)K−1,E[g_t+1-g_t _t]=2(1-p_t)^K-1, (10) so hints have negative drift once pt>p⋆:=1−2−1/Kp_t>p_ :=1-2^-1/K Theorem 2. Under the assumptions of the two-response model, suppose sufficiently strong hints produce a guided success with probability bounded below by a constant c>0c>0 while pt<τp_t<τ. Holding the model and hint-quality constants fixed, as p0↓0p_0 0, [TτRGPO]=O(log1p0),[Tτplain]=Ω(1p0),E[T_τ RGPO]=O\! ( 1p_0 ), [T_τ plain]= \! ( 1p_0 ), (11) where plain GRPO uses the same total number of responses per visit. 5 Experiments We begin with a controlled simulation designed to isolate two challenges that motivate RGPO: learning from hard problems for which on-policy sampling rarely discovers a correct solution, and using external rationales that are helpful but not perfectly reliable. We then evaluate RGPO in language-model and vision-language-model post-training to study whether these advantages translate to realistic reasoning tasks. All language- and vision-language-model experiments are conducted using the verl framework (Sheng et al., 2025). 5.1 Controlled Simulation Setup. Each synthetic problem requires the policy to make a short sequence of reasoning decisions before producing the final answer. Most decision types are easy: the initial policy already assigns high probability to a correct choice. A smaller set is hard: the initial policy has no preference for the correct choice. Easy problems contain only easy decisions, whereas hard problems contain four difficult decisions. The same decision types recur across different problems, so learning how to handle a difficult decision on one problem can transfer to other problems that require it. Each problem is paired with a reference rationale that can be revealed progressively. To model imperfect supervision, the reference is systematically incorrect for three of the eight hard decision types. Revealing the full rationale also exposes the final answer, creating a potential shortcut. We compare RGPO with unguided GRPO, RL performed in the hinted context, RL with supervised learning on the reference, and always-on full-rationale training, together with ablations of RGPO’s reward gate and adaptive hinting. All methods are evaluated on the original problems without hints. Figure 2: Controlled simulation. Accuracy without hints on easy and hard problems, evaluated on both training and held-out problems (mean ± s.e. over 5 seeds). Unguided GRPO readily learns the easy problems but makes no progress on the hard ones. Figure 2 highlights three aspects of how guidance affects learning. First, guidance is most useful when ordinary on-policy exploration provides little signal: GRPO reaches 99.1%99.1\% test accuracy on easy problems but remains at 0.0%0.0\% on hard problems after three epochs. On these problems, almost every group of sampled trajectories is incorrect, leaving little useful reward variation from which to learn. RGPO instead reaches 99.5%99.5\% hard test accuracy while preserving performance on easy problems. Second, more guidance is not always better. Directly training on the imperfect reference plateaus at 12.8%12.8\% hard test accuracy because its systematic errors are reinforced. Removing RGPO’s reward gate produces essentially the same behavior, because incorrect guided trajectories are then distilled together with correct ones. At the opposite extreme, training with the full rationale available allows the model to exploit the exposed final answer, yet performance falls to the 1%1\% chance level when the rationale is removed at evaluation. Third, RGPO uses the rationale as a search aid rather than as a training target. Guided responses are distilled only when they are correct and improve over the unguided samples. In the main setting, only 7.47.4 guided responses are accepted per run on average, yet these few verified trajectories are sufficient to bootstrap learning because the same hard decisions recur across problems. 5.2 RGPO on Language Models For the language-model experiments, we follow the training protocol of HINT (Wang et al., 2025) and use Qwen2.5-3B (Team, 2024) as the base model. To ensure a consistent comparison across methods, we train on the DAPO-Math-27K split used by HINT, which is derived from the DAPO-Math170K dataset (Yu et al., 2025). We optimize the model with GRPO (Shao et al., 2024). Unlike HINT, which trains for 10 epochs, we train for 3 epochs, as we observe that the model already outperforms the compared baselines under this setting. We use a training batch size of 256, a maximum prompt length of 1024 tokens, and a maximum response length of 4096 tokens. For each prompt, the rollout engine samples eight responses, four with hints and four without hints to ensure a fair comparison. The base model is optimized with AdamW (Loshchilov and Hutter, 2019) using a learning rate of 1×10−61× 10^-6. For efficient large-model training, we use FSDP2 (Zhao et al., 2023), together with bfloat16 precision (Kalamkar et al., 2019), fused kernels, and gradient checkpointing (Chen et al., 2016). We do not enable parameter or optimizer offloading. Table 1: Overall Performance Comparison of RGPO vs. Baselines. RGPO obtains the highest in-distribution average, improving over HINT by 5.0 points (26.8 vs. 21.8), and improves the out-of-distribution average by 2.3 points (32.2 vs. 29.9). Methods In-Distribution Avg Out-of-Distribution Avg AIME Math Olympiad Minerva ARC GPQA-D MMLU-Pro Vanilla 2.9 39.8 12.0 9.8 16.1 44.8 11.4 28.8 28.3 GRPO 4.3 44.0 18.2 12.2 19.7 45.0 11.8 28.0 28.3 CHORD 4.5 46.6 20.2 13.0 21.1 40.0 11.0 26.4 25.8 LUFFY 3.3 40.0 18.0 13.2 18.6 40.8 11.2 24.0 25.3 GHPO 4.0 42.2 19.6 12.8 19.7 45.5 12.0 28.2 28.6 QuestA 3.9 42.0 19.6 12.4 19.5 44.8 12.0 29.0 28.6 BREAD 4.1 44.4 20.4 13.4 20.6 45.5 11.8 29.2 28.8 HINT 4.9 48.6 20.2 13.4 21.8 48.8 11.8 30.2 29.9 RGPO 5.1 55.2 24.9 22.1 26.8 51.2 14.7 30.8 32.2 We benchmark against the base model, vanilla GRPO, PPO where applicable, and representative approaches for improving rollout quality or efficiency in reinforcement learning with verifiable rewards such as LUFFY (Yan et al., 2025), CHORD (Zhang et al., 2025b), GHPO (Liu et al., 2025), QuestA (Li et al., 2025b), BREAD (Zhang et al., 2026), and HINT (Wang et al., 2025). Evaluation is conducted without providing hints at test time. We consider both in-distribution and out-of-distribution settings. For mathematical reasoning, we use AIME24, MATH-500 (Hendrycks et al., 2021), OlympiadBench (He et al., 2024) and Minerva Math (Lewkowycz et al., 2022). Because AIME24 contains a relatively small number of test items, we report avg@32 on this benchmark, while pass@1 is used for the remaining datasets. To examine broader reasoning transfer, we also evaluate on ARC-Challenge (Clark et al., 2018), GPQA-Diamond (Rein et al., 2023), and MMLU-Pro (Wang et al., 2024d). Table 1 reports the overall performance of RGPO against the vanilla model, standard GRPO, and other hybrid RL training baselines. Overall, RGPO achieves the best result on every in-distribution benchmark, improving the average score from 19.7 with GRPO to 26.8, and outperforming the strongest prior baseline, HINT, by 5.0 points. The gains are especially pronounced on MATH-500 and Minerva Math, where RGPO improves over HINT by 6.6 and 8.7 points, respectively, suggesting that rationale-guided optimization is particularly effective on problems requiring longer or more structured reasoning. RGPO also generalizes well beyond the training distribution, achieving the highest scores on ARC, GPQA, and MMLU, with an out-of-distribution average of 32.2 compared with 29.9 for the best baseline. 5.3 RGPO on Vision-Language Models For vision-language experiments, we follow the experimental protocol of MINT-CoT (Chen et al., 2025b) and use Qwen2-VL-7B-Instruct (Wang et al., 2024a) as the base multimodal model. We additionally conduct experiments on Qwen2-VL-2B-Instruct as a lower-capacity base model to test whether the benefit persists at smaller scale. Similar to the language model experiment, we train each model for 3 epochs with adaptive hinting. We use a training batch size of 512, a maximum prompt length of 768 tokens, and a maximum response length of 1024 tokens. Evaluation. We follow MINT-CoT and evaluate on multimodal mathematical reasoning benchmarks: MathVista-Math (Lu et al., 2024) and MMStar-Math (Chen et al., 2024a). For reference, we report results from contemporary VLMs, including R1-VL-7B (Chen et al., 2025a; Zhang et al., 2025a), Open-R1-Multimodal (EvolvingLMMs-Lab, 2025), Mulberry (Yao et al., 2024), M-Eureka (Meng et al., 2025), LLaVA-OneVision-Qwen2-7B-OV (Li et al., 2025a), InternVL2-8B (Chen et al., 2024b), InternVL2-8B-MPO (Wang et al., 2024c), DeepSeek-VL2 (Wu et al., 2024), and Qwen2.5-VL-7B-Instruct (Bai et al., 2025). We compare RGPO primarily with contemporary VLMs evaluated under comparable conditions. We report later-released state-of-the-art VLMs only as reference points because they often use stronger post-training pipelines. Our goal is to evaluate whether RGPO improves reasoning when applied to a fixed backbone and compared with contemporaneous models and controlled training baselines. Figure 3: RGPO accelerates and improves RLVR training. Across Qwen2-VL-2B and Qwen2-VL-7B models, RGPO reaches the final performance of RLVR baselines substantially faster and achieves higher final accuracy. The gains are especially pronounced for weaker models and less stable RLVR algorithms (e.g. REINFORCE++). Figure 3 compares RGPO against standard RLVR algorithms, including GRPO and REINFORCE++, on Qwen2-VL-2B-Instruct and Qwen2-VL-7B-Instruct. We report accuracy over training steps to evaluate both final reasoning performance and learning efficiency. Across all settings, RGPO improves the data efficiency over the corresponding RLVR baseline. On Qwen2-VL-2B with GRPO, RGPO reaches the baseline performance in 2.95× fewer training steps and achieves a higher final accuracy. The improvement is even larger when compared with REINFORCE++, where the baseline struggles to learn effectively, while RGPO continues to improve and obtains a substantial final gain. On the stronger Qwen2-VL-7B model, both methods improve during training, but RGPO still converges more efficiently and reaches a higher final accuracy. Table 2: Performance comparison on MMStar-Math and the mathematical subset of MathVista We compare RGPO with the base Qwen2-VL-7B-Instruct model, CoT supervised fine-tuning variants, and representative open-source multimodal reasoning models. Dashes indicate unavailable results. Model MMStar-Math MathVista-Math All GEO ALG GPS TQA LLaVA-OneVision-Qwen2-7b-ov Li et al. (2025a) – 67.04 69.34 67.04 69.71 58.06 InternVL2-8B Chen et al. (2024b) 66.8 62.59 62.26 62.92 62.50 62.90 InternVL2-8B-MPO Wang et al. (2024c) – 68.52 68.87 68.91 69.71 64.52 DeepSeek-VL2 Wu et al. (2024) – 65.56 63.68 65.54 63.94 70.97 Qwen2.5-VL-7B-Instruct Bai et al. (2025) 66.8 66.66 65.56 66.29 65.87 69.35 Open-R1-Multimodal EvolvingLMMs-Lab (2025) 59.2 54.81 52.36 54.68 53.37 59.68 R1-VL-7B Zhang et al. (2025a) 68.4 69.63 68.87 69.66 69.71 69.35 Mulberry Yao et al. (2024) 66.8 68.52 67.92 68.54 68.75 67.74 M-Eureka Meng et al. (2025) – 72.59 71.22 72.66 72.60 72.58 Qwen2-VL-7B-Instruct Wang et al. (2024b) 46.4 41.11 35.85 41.57 36.54 56.45 Original Image CoT SFT – 40.37 38.68 40.82 39.42 43.54 Bounding Box CoT SFT – 65.56 63.21 65.54 63.94 70.97 Text-only CoT SFT 67.6 64.07 64.15 64.04 64.42 62.90 RGPO 70.1 73.65 73.47 73.62 73.89 70.35 As shown in Table 2, RGPO delivers the strongest overall performance among the compared methods. On MMStar-Math, RGPO reaches 70.1, outperforming the base Qwen2-VL-7B-Instruct model by 23.7 points and surpassing other open-source multimodal reasoning models such as R1-VL-7B and Mulberry. On MathVista-Math, RGPO achieves the highest overall score of 73.65, improving substantially over the base model’s 41.11 and also outperforming strong reasoning-oriented baselines such as M-Eureka. The gains are especially clear in the GEO, ALG, and GPS subsets, where RGPO obtains the best scores of 73.47, 73.62, and 73.89, respectively. In contrast, the gain on the Textbook Question Answering subset is relatively modest. This may be due to a small degree of domain overlap between the MINT-CoT (Chen et al., 2025b) training set and this subset of MathVista, which leaves less room for additional improvement on this category. Figure 4: Effect of failure-conditioned hinting versus always-on full hints: Training (solid lines) and validation (dashed lines) rewards. Full hints rapidly inflate training reward but generalize poorly, while RGPO learns more gradually and achieves stronger no-hint validation performance. Figure 4 compares RGPO with a full-hint setting, where the model is given the complete hint during training. Although the full-hint model quickly obtains high training reward, this improvement is misleading: the model can simply use the provided hint to output the correct answer with only a few tokens, without learning to perform the reasoning process independently. This behavior indicates shortcut behavior, where the policy exploits the availability of the hint rather than developing robust reasoning ability. The full-hint model initially produces extremely short outputs while still receiving high reward, indicating that it can bypass reasoning and directly answer from the hint. Moreover, the policy-gradient loss and gradient-norm curves show that full-hint training is less stable, with large early spikes and sharp fluctuations. In contrast, RGPO achieves more gradual reward improvement, maintains longer reasoning responses, and exhibits smoother optimization dynamics. These results suggest that always providing hints can create a shortcut that produces less stable optimization and poorer no-hint validation performance, while RGPO’s failure-conditioned hinting mechanism provides guidance only when needed and avoids encouraging the model to rely on hints as a substitute for reasoning. Figure 5: Training dynamics under different reasoning prompt formats. Left: using the concise DeepSeek-R1-style format, RGPO achieves higher reward than GRPO while maintaining a comparable entropy trajectory throughout training. Right: under the original Mulberry-style template, AHPO eventually improves reward after learning the required output structure, but this improvement is accompanied by a sharp entropy collapse, indicating reduced exploration and more deterministic generation behavior. Figure 5 highlights how RGPO can yield a strong reasoning model even when the rationale format is poorly matched to the policy. Unlike methods that directly optimize likelihood on the reference rationale, RGPO does not require the policy to reproduce the rationale format and is therefore compatible with different response templates. With the concise DeepSeek-R1-style prompt, both RGPO and GRPO begin improving rapidly once the model discovers the correct answer format, but RGPO reaches a consistently higher reward plateau. At the same time, its entropy decreases smoothly and remains close to GRPO, suggesting that the performance gain does not come from simply collapsing the policy into a narrow set of outputs. In contrast, when using the more structured Mulberry-style prompt, AHPO (Zhao et al., 2025), which combines off-policy expert SFT loss with RLVR, shows a delayed reward increase but undergoes a sharp entropy collapse shortly afterward. This suggests that AHPO learns to satisfy the complex template by concentrating probability mass on a limited response pattern, reducing exploration once the formatting objective is acquired. We therefore use the simpler DeepSeek-R1-style format in our main experiments: it makes the reward signal easier to optimize while allowing RGPO to improve reasoning performance without prematurely sacrificing policy diversity. 6 Conclusion In this paper, we introduced Rationale-Guided Policy Optimization, an RL framework that improves reasoning by using rationales as temporary scaffolds rather than direct imitation targets. RGPO’s design reduces hint dependence by preventing the model from relying on rationales for every example, and reduces rationale memorization by avoiding direct training on the rationale text itself. Motivated by cognitive principles, RGPO treats rationales as instructional interventions that help convert failed attempts into useful learning signals. Empirically, RGPO consistently shows superior performance on post-training of both language and vision-language models. Acknowledgments QL acknowledges support of NSF DMS-2523382 and DOE Office of Science under Award #DESC0024721. References [1] OpenAI (2026) GPT-6 astra: a new generation of intelligence. Note: https://openai.com/index/gpt-6-astra/Accessed: 2026-10-04 Cited by: §1. [2] Google DeepMind (2026) Gemini 4 argon: our next era of frontier intelligence. Note: Company announcement and technical overview accessed October 2026 External Links: Link Cited by: §1. [3] Anthropic (2026) Claude opus 5.5 system card. Note: Technical report / system card released September 22, 2026 External Links: Link Cited by: §1. [4] DeepSeek-AI (2026) DeepSeek-v4.1-flash: pushing the limits of kv cache compression. Cited by: §1. [5] K. Team, T. Bai, Y. Bai, Y. Bao, J. Cai, X. Cai, P. Cao, Y. Cao, Z. Chai, Y. Charles, et al. (2026) Kimi k3: open frontier intelligence. Cited by: §1. [6] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. CoRR abs/1707.06347. External Links: Link, 1707.06347 Cited by: §1, §2. [7] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. CoRR abs/2402.03300. External Links: Link, Document, 2402.03300 Cited by: §C.3, §1, §2, §3.2, §5.2. [8] J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang (2025) Learning to reason under off-policy guidance. CoRR abs/2504.14945. External Links: Link, Document, 2504.14945 Cited by: §C.3, §1, §2, §5.2. [9] Z. Liu, C. Gong, X. Fu, Y. Liu, R. Chen, S. Hu, S. Zhang, R. Liu, Q. Zhang, and D. Tu (2025) GHPO: adaptive guidance for stable and efficient LLM reinforcement learning. CoRR abs/2507.10628. External Links: Link, Document, 2507.10628 Cited by: §C.3, §1, §2, §5.2. [10] X. Zhang, Z. Huang, Y. Li, C. Ni, J. Chen, and S. Oymak (2026) Bread: branched rollouts from expert anchors bridge sft & rl for reasoning. Vol. 38, p. 96726–96752. Cited by: §C.3, §1, §2, §5.2. [11] X. Wang, J. Han, Z. Jiang, T. Li, J. Liang, S. Jiang, Z. Dai, S. Ma, F. Yu, and Y. Xiao (2025) HINT: helping ineffective rollouts navigate towards effectiveness. CoRR abs/2510.09388. External Links: Link, Document, 2510.09388 Cited by: §C.3, §C.6, §1, §2, §5.2, §5.2. [12] T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma (2025) SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §1, §3.1. [13] M. Kapur (2008) Productive failure. Cognition and instruction 26 (3), p. 379–424. Cited by: §1, §2. [14] D. L. Schwartz and J. D. Bransford (1998) A time for telling. Cognition and instruction 16 (4), p. 475–5223. Cited by: §1, §2. [15] K. R. Koedinger and V. Aleven (2007) Exploring the assistance dilemma in experiments with cognitive tutors. Educational psychology review 19 (3), p. 239–264. Cited by: §1, §2. [16] DeepSeek-AI (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. CoRR abs/2501.12948. External Links: Link, Document, 2501.12948 Cited by: §2. [17] W. Zhang, Y. Xie, Y. Sun, Y. Chen, G. Wang, Y. Li, B. Ding, and J. Zhou (2025) On-policy RL meets off-policy experts: harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. CoRR abs/2508.11408. External Links: Link, Document, 2508.11408 Cited by: §C.3, §2, §5.2. [18] W. Xiong, J. Yao, Y. Xu, B. Pang, L. Wang, D. Sahoo, J. Li, N. Jiang, T. Zhang, C. Xiong, and H. Dong (2025) A minimalist approach to LLM reasoning: from rejection sampling to reinforce. CoRR abs/2504.11343. External Links: Link, Document, 2504.11343 Cited by: §2. [19] J. Li, H. Lu, K. Wen, Z. Yang, J. Gao, H. Lin, Y. Wu, and J. Zhang (2025) QuestA: expanding reasoning capacity in llms via question augmentation. CoRR abs/2507.13266. External Links: Link, Document, 2507.13266 Cited by: §C.3, §2, §5.2. [20] H. L. Roediger I and J. D. Karpicke (2006) Test-enhanced learning: taking memory tests improves long-term retention. Psychological science 17 (3), p. 249–255. Cited by: §2. [21] V. Aleven, B. McLaren, I. Roll, and K. Koedinger (2004) Toward tutoring help seeking: applying cognitive modeling to meta-cognitive skills. In International conference on intelligent tutoring systems, p. 227–239. Cited by: §2. [22] J. Sweller and G. A. Cooper (1985) The use of worked examples as a substitute for problem solving in learning algebra. Cognition and instruction 2 (1), p. 59–89. Cited by: §2. [23] M. T. Chi, M. Bassok, M. W. Lewis, P. Reimann, and R. Glaser (1989) Self-explanations: how students study and use examples in learning to solve problems. Cognitive science 13 (2), p. 145–182. Cited by: §2. [24] R. K. Atkinson, A. Renkl, and M. M. Merrill (2003) Transitioning from studying examples to solving problems: effects of self-explanation prompts and fading worked-out steps.. Journal of educational psychology 95 (4), p. 774. Cited by: §2. [25] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. CoRR abs/2203.02155. External Links: Link, Document, 2203.02155 Cited by: §3.1. [26] H. Yao, J. Huang, W. Wu, J. Zhang, Y. Wang, S. Liu, Y. Wang, Y. Song, H. Feng, L. Shen, and D. Tao (2024) Mulberry: empowering MLLM with o1-like reasoning and reflection via collective monte carlo tree search. CoRR abs/2412.18319. External Links: Link, Document, 2412.18319 Cited by: §C.3, §3.1, §5.3, Table 2. [27] J. Hu, J. K. Liu, H. Xu, and W. Shen (2025) REINFORCE++: stabilizing critic-free policy optimization with global advantage normalization. CoRR abs/2501.03262. External Links: 2501.03262 Cited by: §C.3, §3.2. [28] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum (2025) Open-reasoner-zero: an open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290. Cited by: §3.2. [29] Z. Hao, H. Wang, H. Liu, J. Luo, J. Yu, H. Dong, Q. Lin, C. Wang, and J. Chen (2025) Rethinking entropy interventions in rlvr: an entropy change perspective. arXiv preprint arXiv:2510.10150. Cited by: §3.2. [30] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, External Links: Link, Document Cited by: §5. [31] Q. Team (2024) Qwen2.5 technical report. CoRR abs/2412.15115. External Links: Link, Document, 2412.15115 Cited by: §5.2. [32] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source LLM reinforcement learning system at scale. CoRR abs/2503.14476. External Links: Link, Document, 2503.14476 Cited by: §5.2. [33] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §5.2. [34] Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li (2023) PyTorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow. 16 (12), p. 3848–3860. External Links: Link, Document Cited by: §5.2. [35] D. D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey (2019) A study of BFLOAT16 for deep learning training. CoRR abs/1905.12322. External Links: Link, 1905.12322 Cited by: §5.2. [36] T. Chen, B. Xu, C. Zhang, and C. Guestrin (2016) Training deep nets with sublinear memory cost. CoRR abs/1604.06174. External Links: Link, 1604.06174 Cited by: §5.2. [37] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, External Links: Link Cited by: §5.2. [38] C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun (2024) OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, External Links: Link, Document Cited by: §5.2. [39] A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra (2022) Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §5.2. [40] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR abs/1803.05457. External Links: Link, 1803.05457 Cited by: §5.2. [41] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2023) GPQA: A graduate-level google-proof q&a benchmark. CoRR abs/2311.12022. External Links: Link, Document, 2311.12022 Cited by: §5.2. [42] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen (2024) MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §5.2. [43] X. Chen, R. Zhang, D. Jiang, A. Zhou, S. Yan, W. Lin, and H. Li (2025) MINT-cot: enabling interleaved visual tokens in mathematical chain-of-thought reasoning. CoRR abs/2506.05331. External Links: Link, Document, 2506.05331 Cited by: §C.3, §5.3, §5.3. [44] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. CoRR abs/2409.12191. External Links: Link, Document, 2409.12191 Cited by: §5.3. [45] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao (2024) MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), Cited by: §5.3. [46] L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, and F. Zhao (2024) Are we on the right way for evaluating large vision-language models?. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §5.3. [47] L. Chen, L. Li, H. Zhao, Y. Song, and Vinci (2025) R1-v: reinforcing super generalization ability in vision-language models with less than $3. Note: https://github.com/Deep-Agent/R1-VAccessed: 2026-05-02 Cited by: §5.3. [48] J. Zhang, J. Huang, H. Yao, S. Liu, X. Zhang, S. Lu, and D. Tao (2025) R1-VL: learning to reason with multimodal large language models via step-wise group relative policy optimization. CoRR abs/2503.12937. External Links: Link, Document, 2503.12937 Cited by: §C.3, §5.3, Table 2. [49] EvolvingLMMs-Lab (2025) Open-r1-multimodal: a fork to add multimodal model training to open-r1. Note: https://github.com/EvolvingLMMs-Lab/open-r1-multimodalAccessed: 2026-05-2 Cited by: §C.3, §5.3, Table 2. [50] F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, B. Shi, W. Wang, J. He, K. Zhang, P. Luo, Y. Qiao, Q. Zhang, and W. Shao (2025) M-eureka: exploring visual aha moment with rule-based large-scale reinforcement learning. CoRR abs/2503.07365. External Links: Link, Document, 2503.07365 Cited by: §C.3, §5.3, Table 2. [51] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li (2025) LLaVA-onevision: easy visual task transfer. Trans. Mach. Learn. Res. 2025. External Links: Link Cited by: §C.3, §5.3, Table 2. [52] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai (2024) Intern VL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Conference on Computer Vision and Pattern Recognition, External Links: Link, Document Cited by: §C.3, §5.3, Table 2. [53] W. Wang, Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, J. Zhu, X. Zhu, L. Lu, Y. Qiao, and J. Dai (2024) Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. CoRR abs/2411.10442. External Links: Link, Document, 2411.10442 Cited by: §C.3, §5.3, Table 2. [54] Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, Z. Xie, Y. Wu, K. Hu, J. Wang, Y. Sun, Y. Li, Y. Piao, K. Guan, A. Liu, X. Xie, Y. You, K. Dong, X. Yu, H. Zhang, L. Zhao, Y. Wang, and C. Ruan (2024) DeepSeek-vl2: mixture-of-experts vision-language models for advanced multimodal understanding. CoRR abs/2412.10302. External Links: Link, Document, 2412.10302 Cited by: §C.3, §5.3, Table 2. [55] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. CoRR abs/2502.13923. External Links: Link, Document, 2502.13923 Cited by: §C.3, §5.3, Table 2. [56] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. CoRR abs/2409.12191. External Links: Link, Document, 2409.12191 Cited by: Table 2. [57] X. Zhao, J. Lin, T. Liang, Y. Zhou, W. Chai, Y. Gu, W. Wang, K. Chen, G. Luo, W. Zhang, et al. (2025) M-helix: boosting multimodal long-chain reflective reasoning with holistic platform and adaptive hybrid policy optimization. arXiv preprint arXiv:2510.08540. Cited by: §C.3, §C.3, §D.4, §5.3. [58] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §A.1. [59] S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §A.1. [60] I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal (2026) Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897. Cited by: §A.1. [61] K. Sun, Y. Bai, J. Qi, L. Hou, and J. Li (2024) Mm-math: advancing multimodal math evaluation with process evaluation and fine-grained classification. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1358–1375. Cited by: §D.6. [62] C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan (2026) Self-distilled rlvr. arXiv preprint arXiv:2604.03128. Cited by: §D.6. [63] S. Li, C. Yu, H. Wang, W. Yang, R. Rossi, F. Dernoncourt, X. Hu, P. Yu, C. Xiao, H. Zhang, et al. (2026) FORTIS: benchmarking over-privilege in agent skills. arXiv preprint arXiv:2605.09163. Cited by: §D.6. Due to space constraints, some implementation details, additional analyses, and supplementary results are omitted from the main paper. In this appendix, we provide a more complete description of the RGPO training procedure, including the prompt templates, algorithmic details, and hyperparameter settings. We also include additional experimental results and ablation studies to better understand the effects of rationale guidance, prompt format, and failure-conditioned hinting on model learning and final performance. Appendix A Method Details This section provides implementation details for Rationale-Guided Policy Optimization (RGPO) that are omitted from the main text. We first give the complete training procedure and then clarify several design choices that are important in practice. A.1 Training Procedure Algorithm 1 summarizes Rationale-Guided Policy Optimization. In our experiments, we set the correctness threshold to τcorrect=1.0 _correct=1.0, requiring an unguided rollout to be perfectly correct in order to be considered solved. We set the acceptance margin to δ=0δ=0, so a rationale-guided revision is added to the auxiliary buffer whenever it achieves a strictly higher verifier reward than the best unguided rollout. The rationale-guided loss coefficient λt _t is tuned over 10−2,10−3,10−410^-2,10^-3,10^-4. Algorithm 1 Rationale-Guided Policy Optimization 1: Training set =(xi,si,ai)i=1MD=\(x_i,s_i,a_i)\_i=1^M, policy πθ _θ, verifier reward r, number of hint segments N, rollouts per prompt K, acceptance margin δ 2: Partition each solution sis_i into N segments [si,1;…;si,N][s_i,1;…;s_i,N] 3: Initialize hint level gi(0)←0g_i^(0)← 0 for all examples 4: for t=0t=0 to N do 5: Clear auxiliary buffer ℬRGPOB_RGPO 6: for each minibatch of examples do 7: Sample unguided rollouts yi,1(t),…,yi,K(t)∼πθ(⋅∣xi)y_i,1^(t),…,y_i,K^(t) _θ(· x_i) 8: Compute rewards ri,k(t)=r(xi,yi,k(t),ai)r_i,k^(t)=r(x_i,y_i,k^(t),a_i) and advantages for the main RL objective 9: if t>0t>0 and gi(t)>0g_i^(t)>0 then 10: Construct hint prefix hi(t)=[si,1;…;si,gi(t)]h_i^(t)=[s_i,1;…;s_i,g_i^(t)] 11: Construct guided prompt x~i(t)=Reprompt(xi,hi(t),fi(t)) x_i^(t)=Reprompt(x_i,h_i^(t),f_i^(t)) 12: Sample guided revisions y~i,1(t),…,y~i,K(t)∼πθ(⋅∣x~i(t)) y_i,1^(t),…, y_i,K^(t) _θ(· x_i^(t)) 13: Compute guided rewards r~i,k(t)=r(xi,y~i,k(t),ai) r_i,k^(t)=r(x_i, y_i,k^(t),a_i) 14: Add (xi,y~i,k(t))(x_i, y_i,k^(t)) to ℬRGPOB_RGPO if r~i,k(t)>maxjri,j(t)+δ r_i,k^(t)> _jr_i,j^(t)+δ 15: end if 16: Update πθ _θ using ℒRL+λtℒSFTL_RL+ _tL_SFT 17: end for 18: for each example i do 19: Set Ci(t)=[maxkri,k(t)≥τcorrect]C_i^(t)=I[ _kr_i,k^(t)≥ _correct] using unguided rollouts 20: if Ci(t)=0C_i^(t)=0 then 21: gi(t+1)←min(gi(t)+1,N)g_i^(t+1)← (g_i^(t)+1,N) 22: else 23: gi(t+1)←max(gi(t)−1,0)g_i^(t+1)← (g_i^(t)-1,0) 24: end if 25: end for 26: end for On-Policy Self-Distillation (OPSD) Compared to RGPO, exact on-policy distillation [58, 59, 60] would require access to the teacher’s token-level probabilities or logits for the actions taken under the student’s rollout distribution. Even when the teacher is open, computing this KL term introduces substantial memory and compute overhead because training must run an additional teacher forward pass and retain token-level probability information, often over large vocabularies and long reasoning trajectories. Treating teacher-sampled completions as ordinary SFT targets avoids the logit requirement, but becomes only an off-policy approximation and can reintroduce the distribution-mismatch and format-imitation issues discussed in the main paper. RGPO avoids these dependencies by using external rationales only as prompt-level scaffolds: the refined candidate is generated by the trainable policy itself, accepted only when it improves verifier reward, and then distilled back under the original no-hint prompt. Consequently, RGPO can exploit black-box, human-written, or dataset-provided rationales without requiring teacher logits or an additional teacher model during training, while still optimizing the deployed policy for the original inference condition. Below, we provide three qualitative examples illustrating how partial rationales, together with the model’s verifier-judged response, can guide the model toward the correct solution. In these examples, attempting to answer the question directly can lead to an incorrect answer; after receiving auxiliary information, the model is steered toward a correct refined response. The model is then trained to reproduce such corrected response without relying on the auxiliary context. Example 1: The question for generating the refined answer consists of the following sections, indicated by their corresponding colors: original question, partial solution, previous response, previous judge feedback, and further instructions. System prompt: You are a helpful AI assistant. A conversation takes place between the User and the Assistant. The User asks a question, and the Assistant solves it. Please help me solve this question. Wrap only the final answer in . Question: Human beings have discovered a habitable planet and soon after, they find 10 more habitable planets. Of these 11, only 5 are deemed “Earth-like” in their resources and the rest are deemed “Mars-like” since they lack many important resources. Assume that planets like Earth take up 2 units of colonization, while those like Mars take up only 1. If humanity mobilizes 12 total units of colonies, how many different combinations of planets can be occupied if the planets are all different from each other? This is the (partial) solution without formatting, you can leverage it to check your answer: Think about combinations and constraints. How might you use the fact that Earth-like planets take up 2 units and Mars-like planets take up 1 unit to de Previous response: To solve this problem, we need to consider the constraints and find combinations that meet them. Here’s a step-by-step breakdown: 1. **Earth-like Planets**: Each Earth-like planet requires 2 units of colonization. 2. **Mars-like Planets**: Each Mars-like planet requires 1 unit of colonization. 3. **Total Units of Colonization**: There are 12 units available. We need to find combinations of Earth-like and Mars-like planets such that their total units add up to 12. Let’s denote the number of Earth-like planets by e and the number of Mars-like planets by m. We then have: 2e+m=122e+m=12 We also need to consider that the planets are distinct, so we need to count the number of ways to choose these combinations. Specifically, we need to count the valid pairs (e,m)(e,m) where both e and m are integers between 0 and a maximum that keeps the total units within the limit. Let’s solve for different values of m: * If m=0m=0: 2e=12⇒e=62e=12 e=6 (This means 6 Earth-like planets and 0 Mars-like planets). The number of ways to choose 6 Earth-like planets out of 5 is: (56)=0 56=0 * If m=1m=1: 2e+1=12⇒2e=11⇒e=5.52e+1=12 2e=11 e=5.5 (This is not an integer, so not feasible). * If m=2m=2: 2e+2=12⇒2e=10⇒e=52e+2=12 2e=10 e=5 (This means 5 Earth-like planets and 2 Mars-like planets). The number of ways to choose 5 Earth-like planets out of 5 is: (55)⋅(62)=1⋅(62)=1⋅15=15 55· 62=1· 62=1· 15=15 * If m=3m=3: 2e+3=12⇒2e=9⇒e=4.52e+3=12 2e=9 e=4.5 (This is not an integer, so not feasible). * If m=4m=4: 2e+4=12⇒2e=8⇒e=42e+4=12 2e=8 e=4 (This means 4 Earth-like planets and 4 Mars-like planets). The number of ways to choose 4 Earth-like planets out of 5 is: (54)⋅(64)=5⋅15=75 54· 64=5· 15=75 * If m=5m=5: 2e+5=12⇒2e=7⇒e=3.52e+5=12 2e=7 e=3.5 (This is not an integer, so not feasible). * If m=6m=6: 2e+6=12⇒2e=6⇒e=32e+6=12 2e=6 e=3 (This means 3 Earth-like planets and 6 Mars-like planets). The number of ways to choose 3 Earth-like planets out of 5 is: (53)⋅(66)=10⋅1=10 53· 66=10· 1=10 * If m=7m=7: 2e+7=12⇒2e=5⇒e=2.52e+7=12 2e=5 e=2.5 (This is not an integer, so not feasible). * If m=8m=8: 2e+8=12⇒2e=4⇒e=22e+8=12 2e=4 e=2 (This means 2 Earth-like planets and 8 Mars-like planets). The number of ways to choose 2 Earth-like planets out of 5 is: (52)⋅(68)=10⋅1=10 52· 68=10· 1=10 * If m=9m=9: 2e+9=12⇒2e=3⇒e=1.52e+9=12 2e=3 e=1.5 (This is not an integer, so not feasible). * If m=10m=10: 2e+10=12⇒2e=2⇒e=12e+10=12 2e=2 e=1 (This means 1 Earth-like planet and 10 Mars-like planets). The number of ways to choose 1 Earth-like planet out of 5 is: (51)⋅(610)=5⋅0=0 51· 610=5· 0=0 * If m=11m=11: 2e+11=12⇒2e=1⇒e=0.52e+11=12 2e=1 e=0.5 (This is not an integer, so not feasible). Adding up all feasible cases, we get: 0+15+75+10+10+0=1100+15+75+10+10+0=110 Thus, the number of different combinations of planets that can be occupied is 110. Feedback for your previous response: Format score: 1.0/1.0 Accuracy score: 0/1.0 Overall score: 0.1/1.0 You should maximize the overall score by improving both the format and accuracy of your answer. Write a new standalone solution to the original question without mentioning the previous responses or feedback. You first think through the reasoning process as an internal monologue. Then, provide your final answer enclosed within . Please strictly follow the format. Answer: We are trying to find different combinations of Earth-like (2 units) and Mars-like (1 unit) planets. There are 12 units of colonization available, and we have 5 Earth-like and 6 Mars-like planets. Here’s a step-by-step approach: 1. **Define Variables**: Let e be the number of Earth-like planets and m be the number of Mars-like planets. The total units of colonization constraint can be written as 2e+m=122e+m=12. 2. **Constraints on e and m**: Since we have 5 Earth-like planets and 6 Mars-like planets, the values of e and m must satisfy 0≤e≤50≤ e≤ 5 and 0≤m≤60≤ m≤ 6. 3. **Possible Solutions**: We need to find all possible valid pairs (e,m)(e,m) that satisfy the equation 2e+m=122e+m=12 within the given constraints. Let’s enumerate these pairs: - For e=0e=0: 2(0)+m=12⇒m=122(0)+m=12 m=12 (not possible since m≤6m≤ 6). - For e=1e=1: 2(1)+m=12⇒m=102(1)+m=12 m=10 (not possible since m≤6m≤ 6). - For e=2e=2: 2(2)+m=12⇒m=82(2)+m=12 m=8 (not possible since m≤6m≤ 6). - For e=3e=3: 2(3)+m=12⇒m=62(3)+m=12 m=6 (possible). - For e=4e=4: 2(4)+m=12⇒m=42(4)+m=12 m=4 (possible). - For e=5e=5: 2(5)+m=12⇒m=22(5)+m=12 m=2 (possible). From these valid pairs, we count the number of ways to choose the required number of Earth-like and Mars-like planets. 4. **Calculate Combinations**: - For (3,6)(3,6): We need to choose 3 out of 5 Earth-like planets and 6 out of 6 Mars-like planets. The number of ways is (53)×(66)=10×1=10 53× 66=10× 1=10. - For (4,4)(4,4): We need to choose 4 out of 5 Earth-like planets and 4 out of 6 Mars-like planets. The number of ways is (54)×(64)=5×15=75 54× 64=5× 15=75. - For (5,2)(5,2): We need to choose 5 out of 5 Earth-like planets and 2 out of 6 Mars-like planets. The number of ways is (55)×(62)=1×15=15 55× 62=1× 15=15. Adding these together gives us the total number of combinations: 10+75+15=10010+75+15=100. Therefore, the number of different combinations of planets that can be occupied is 100. Judge: Solution: 100 Ground truth: 100 Format score: 1.0 Accuracy score: 1.0 Overall score: 1.0 In Example A.1, the model leverages the partial solution to identify the correct counting strategy and formulate the constraint (2e+m=12). It then uses the previous response to retain the valid setup while correcting its main error: allowing more than the six available Mars-like planets. By enforcing the bounds (e≤5)(e≤ 5) and (m≤6)(m≤ 6), the model considers only the valid cases ((3,6)), ((4,4)), and ((5,2)), counts the distinct planet selections with combinations, and obtains the correct total of (100). Example 2: The question for generating the refined answer consists of the following sections, indicated by their corresponding colors: original question, partial solution, previous response, previous judge feedback, and further instructions. System prompt: You are a helpful AI assistant. A conversation takes place between the User and the Assistant. The User asks a question, and the Assistant solves it. Please help me solve this question. Wrap only the final answer in . Question: Each corner of a rectangular prism is cut off. Two (of the eight) cuts are shown. How many edges does the new figure have? This is the (partial) solution without formatting, you can leverage it to check your answer: Think about the original number of edges on a rectangular prism. How does cutting of Previous response: acing its corners, we need to determine how many vertices it has. Since each cut off a vertex adds one new edge and removes two, the net increase in edges is 2×4=82× 4=8. So, the number of edges is 24. Feedback for your previous response: Format score: 1.0/1.0 Accuracy score: 0/1.0 Overall score: 0.1/1.0 You should maximize the overall score by improving both the format and accuracy of your answer. Write a new standalone solution to the original question without mentioning the previous responses or feedback. You first think through the reasoning process as an internal monologue. Then, provide your final answer enclosed within . Please strictly follow the format. Answer: To determine the number of edges in the new figure after cutting off the corners of a rectangular prism, we’l first understand the initial properties and then consider the effects of cutting off the corners. ### Initial Number of Edges A rectangular prism has 8 vertices, 12 edges, and 6 faces. However, since we are asked to count the edges created by cutting off the corners, it’s more efficient to consider the total vertices and faces first. ### Basic Structure Before Cutting 1. **Vertices**: 8 2. **Edges**: 12 3. **Faces**: 6 (2 pairs of identical faces) Now, let’s think about the effect of cutting off corners: - Each corner cut off removes one vertex and creates 3 new edges (one for each edge of the faces that used to meet at the vertex). ### Analysis After Cutting Off Corners When we cut off each corner, we effectively: - Remove 1 vertex from the original 8 vertices. - Add 3 new edges. Since we are cutting off all 8 corners of the rectangular prism: - **Total number of edges added by cutting off corners**: 8×3=248× 3=24. ### Calculation of Final Edge Count Initially, the rectangular prism has 12 edges. After cutting off the corners, these initial 12 edges plus the additional 24 edges give us: 12+24=3612+24=36 Thus, the new figure has 36. Judge: Solution: 36 Ground truth: 36 Format score: 1.0 Accuracy score: 1.0 Overall score: 1.0 Leveraging the previous response and the partial solution in Example A.1 helped the model identify the correct structure of the problem: begin with the rectangular prism’s 12 original edges and then account for the effect of cutting off all 8 corners. The earlier incorrect response highlighted that the edge change needed to be reconsidered, while the partial solution directed attention to how each cut modifies the prism. Using these cues, the model correctly determined that each corner cut creates 3 new edges, resulting in (12+8⋅3=36)(12+8· 3=36) edges. Example 3: The question for generating the refined answer consists of the following sections, indicated by their corresponding colors: original question, partial solution, previous response, previous judge feedback, and further instructions. System prompt: You are a helpful AI assistant. A conversation takes place between the User and the Assistant. The User asks a question, and the Assistant solves it. Please help me solve this question. Wrap only the final answer in . Question: Each corner of a rectangular prism is cut off. Two (of the eight) cuts are shown. How many edges does the new figure have? This is the (partial) solution without formatting, you can leverage it to check your answer: Think about the original number of edges on a rectangular prism. How does cutting of To find the smallest perfect square larger than 1 with a perfect square number of positive integer factors, we can consider perfect squares of factors of perfect numbers. Here’s an explanation: A perfect square number of positive integer factors means that the number of factors of the perfect square is itself a perfect square. We can start by considering the perfect numbers, which are integers equal to the sum of their proper positive factors. Examples include 6 (1 + 2 + 3) and 28 (1 + 2 + 4 + 7 + 14). The smallest relative prime pair (ignoring multiples of 2) that gives a perfect number is (4, 9). When we multiply these, we get a perfect square, 36, which has (1, 4, 9, and 36) perfect factors, all of which are perfect squares. Therefore, the smallest perfect square larger than 1 with a perfect square number of positive integer factors is 36. Feedback for your previous response: Format score: 0.0/1.0 Accuracy score: 0/1.0 Overall score: 0.0/1.0 You should maximize the overall score by improving both the format and accuracy of your answer. Write a new standalone solution to the original question without mentioning the previous responses or feedback. You first think through the reasoning process as an internal monologue. Then, provide your final answer enclosed within . Please strictly follow the format. Answer: To find the smallest perfect square larger than 1 with a perfect square number of positive integer factors, we can start by considering the prime factorization of perfect squares. A perfect square can be expressed as the product of pairs of prime factors. If we denote a perfect square as n=p12a1⋅p22a2⋯pk2akn=p_1^2a_1· p_2^2a_2·s p_k^2a_k, where pip_i are distinct prime numbers and aia_i are integers, the number of factors is given by (2a1+1)(2a2+1)⋯(2ak+1)(2a_1+1)(2a_2+1)·s(2a_k+1). For the number of factors to be a perfect square, each term in the product (2a1+1)(2a2+1)⋯(2ak+1)(2a_1+1)(2a_2+1)·s(2a_k+1) must be a perfect square or a product of factors that can be paired to form a perfect square. Let’s consider the smallest perfect square that appears in the set of perfect numbers. The smallest perfect number is 6, whose prime factorization is 2⋅32· 3. When we consider the smallest relative prime pair (ignoring multiples of 2), which is (4, 9), their product is 3636. Here, 36 has the prime factorization 22⋅322^2· 3^2, which has (2a1+1)(2a2+1)=3⋅3=9(2a_1+1)(2a_2+1)=3· 3=9, a perfect square. Therefore, 36 is a perfect square with a perfect square number of positive integer factors. Thus, the smallest perfect square larger than 1 with a perfect square number of positive integer factors is 36. Judge: Solution: 36 Ground truth: 36 Format score: 1.0 Accuracy score: 1.0 Overall score: 1.0 In Example A.1, the model uses the judge’s feedback to correct the formatting by placing only the final answer inside and to ensure that the answer is parsed properly. Although some unnecessary reasoning from the earlier response remains, these changes allow the refined response to receive full accuracy and format scores. Appendix B Theoretical Analysis of Rationale-Guided Learning For a prompt x, let R(x,y)R(x,y) denote the verifier reward and πθ(y∣x) _θ(y x) the policy likelihood of a complete response. We distinguish πθ _θ from the decoding distribution ρθ(⋅∣x) _θ(· x) used to generate rollouts. These coincide under ancestral sampling, but need not coincide under temperature scaling or truncation. Sampling and scheduler quantities are defined with respect to ρθ _θ, whereas likelihood-based quantities are defined with respect to πθ _θ. At a fixed policy θ, sample K unguided responses independently as Y1,…,YK∼ρθ(⋅∣x),B=(Y1,…,YK).Y_1,…,Y_K _θ(· x), B=(Y_1,…,Y_K). When guidance is applied, KgK_g additional candidates Y~1,…,Y~Kg Y_1,…, Y_K_g are generated conditional on the unguided group B, the current rationale prefix, and verifier feedback. A candidate is accepted if R(x,Y~m)>max1≤j≤KR(x,Yj)+δ,δ≥0.R(x, Y_m)> _1≤ j≤ KR(x,Y_j)+δ, δ≥ 0. (12) Let A denote the event that at least one guided candidate is accepted, and let NaccN_ acc denote the number of accepted candidates. For binary rewards, R∈0,1R∈\0,1\, define the unguided success probability p=PrY∼ρθ(⋅∣x)(R(x,Y)=1),p= _Y _θ(· x) (R(x,Y)=1 ), and the all-failure event F=R(x,Yj)=0,∀j=1,…,K.F=\R(x,Y_j)=0,\ ∀ j=1,…,K\.. When ρθ=πθ _θ= _θ, we write pθ=PrY∼πθ(⋅∣x)(R(x,Y)=1),p_θ= _Y _θ(· x) (R(x,Y)=1 ), and denote the policy conditioned on success by πθ+(y∣x)=πθ(y∣x,R(x,Y)=1). _θ^+(y x)= _θ\! (y x,\ R(x,Y)=1 ). B.1 Recovering supervision from failed unguided groups Proposition 3 (Guided supervision recovery). Fix x and a policy snapshot. Suppose R∈0,1R∈\0,1\, 0≤δ<10≤δ<1, and p<1p<1. Conditional on B, suppose the KgK_g guided candidates are independent and have common success probability sBs_B. Then Pr(A) (A) =(1−p)K[1−(1−sB)Kg∣F], =(1-p)^K\,E\! [1-(1-s_B)^K_g F ], (13) [Nacc] [N_ acc] =Kg(1−p)K[sB∣F]. =K_g(1-p)^K\,E[s_B F]. (14) If p=1p=1, both quantities are zero. Proof. If any unguided response succeeds, the right-hand side of the gate in Eq. (12) is at least 11 and no binary-reward candidate can be accepted. On F, the gate threshold is δ<1δ<1, so a candidate is accepted if and only if it succeeds. Conditional on B∈FB∈ F, therefore, Nacc∼Binomial(Kg,sB)N_ acc (K_g,s_B). Since Pr(F)=(1−p)K (F)=(1-p)^K, conditioning on F gives both identities. ∎ For standard group-centered GRPO with binary reward, an all-failure group has zero within-group reward variance and hence zero reward-advantage vector. The proposition therefore identifies a source of positive supervision precisely on an event on which this GRPO signal vanishes. The following proposition also quantifies how the expected correctness signal shrinks as success becomes rare. Proposition 4 (GRPO’s correctness signal vanishes linearly in the pass rate). Assume ρθ=πθ _θ= _θ, 0<p<10<p<1, and define the summed first-step GRPO score update g^GRPO=∑k=1KA^k∇θlogπθ(Yk∣x), g_ GRPO= _k=1^K A_k _θ _θ(Y_k x), where the binary rewards are normalized by their group mean and population standard deviation and A^k=0 A_k=0 when the group reward variance is zero. Let M∼Binomial(K,p)M (K,p). Then [g^GRPO]=cK(p)∇logp,cK(p)=M(K−M)1−p≤K−1p.E[ g_ GRPO]=c_K(p)\,∇ p, c_K(p)= E M(K-M)1-p≤ K K-1\,p. (15) In particular cK(p)=O(p)c_K(p)=O(p) as p↓0p 0, while the update is exactly zero on the all-failure event, whose probability is (1−p)K(1-p)^K. Proof. Let πθ+ _θ^+ and πθ− _θ^- be the policy conditioned on success and failure, and set μ±=πθ±∇logπθ(Y∣x)μ^±=E_ _θ^±∇ _θ(Y x). Conditioned on M=mM=m with 0<m<K0<m<K, a successful sample has normalized advantage (K−m)/m (K-m)/m and a failed sample has normalized advantage −m/(K−m)- m/(K-m). Therefore [g^GRPO∣M=m]=m(K−m)(μ+−μ−).E[ g_ GRPO M=m]= m(K-m)(μ^+-μ^-). Since ∇p=pμ+=−(1−p)μ−∇ p=pμ^+=-(1-p)μ^-, we have μ+−μ−=∇p/[p(1−p)]μ^+-μ^-=∇ p/[p(1-p)]. Averaging over M∼Binomial(K,p)M (K,p) gives the equality in Eq. (15). For 1≤m≤K−11≤ m≤ K-1, m(K−m)≤m(K−m)/K−1 m(K-m)≤ m(K-m)/ K-1, while both sides vanish for m∈0,Km∈\0,K\. Hence M(K−M)≤[M(K−M)]K−1=KK−1p(1−p),E M(K-M)≤ E[M(K-M)] K-1=K K-1\,p(1-p), which proves the bound. ∎ Corollary 5 (Equal response budget). Suppose that, conditional on an all-failure unguided group, the KgK_g guided candidates are conditionally independent and each succeeds with probability s. Then the probability of discovering at least one successful response using K unguided and KgK_g guided samples is PRGPO=1−(1−p)K(1−s)Kg.P_ RGPO=1-(1-p)^K(1-s)^K_g. (16) If the same total budget of K+KgK+K_g responses is instead used entirely for unguided sampling, then PGRPO=1−(1−p)K+Kg.P_ GRPO=1-(1-p)^K+K_g. (17) For Kg≥1K_g≥ 1, PRGPO>PGRPO⟺s>p.P_ RGPO>P_ GRPO s>p. (18) Proof. Using Eqs. (16)–(17), PRGPO>PGRPO⇔(1−s)Kg<(1−p)Kg.P_ RGPO>P_ GRPO (1-s)^K_g<(1-p)^K_g. Since Kg≥1K_g≥ 1, this is equivalent to s>ps>p. ∎ Feedback-dependent proposals. When sBs_B varies with the failed group, the exact equal-budget condition is [(1−sB)Kg∣F]<(1−p)Kg.E[(1-s_B)^K_g F]<(1-p)^K_g. (19) Thus, for Kg>1K_g>1, the comparison depends on the conditional failure probability of the guided group, rather than only on [sB∣F]E[s_B F]. General rewards. For an arbitrary reward and possibly heterogeneous guided proposals, define am(B)=Pr(R(x,Y~m)>maxjR(x,Yj)+δ∣B).a_m(B)= \! (R(x, Y_m)> _jR(x,Y_j)+δ B ). If the guided candidates are conditionally independent given B, then Pr(A)=B[1−∏m=1Kg(1−am(B))],[Nacc]=B∑m=1Kgam(B). (A)=E_B\! [1- _m=1^K_g(1-a_m(B)) ], [N_ acc]=E_B _m=1^K_ga_m(B). (20) The second identity follows by linearity even without candidate independence. B.2 Transfer to the original prompt Theorem 6 (Binary-reward transfer with target mismatch). Suppose 0<pθ<10<p_θ<1. Let μ be a fixed distribution supported on successful responses, and define Lμ(θ)=−Y∼μlogπθ(Y∣x),Dθ=KL(μ∥πθ+(⋅∣x)).L_μ(θ)=-E_Y μ _θ(Y x), D_θ=KL\! (μ\,\|\, _θ^+(· x) ). (21) For any θ0,θ1 _0, _1, let ΔL=Lμ(θ0)−Lμ(θ1). L=L_μ( _0)-L_μ( _1). Then logpθ1pθ0=ΔL+Dθ1−Dθ0≥ΔL−Dθ0. p_ _1p_ _0= L+D_ _1-D_ _0≥ L-D_ _0. (22) In particular, pθ1>pθ0p_ _1>p_ _0 whenever ΔL>Dθ0 L>D_ _0. If μ=πθ0+(⋅∣x)μ= _ _0^+(· x), any strict decrease in LμL_μ implies pθ1>pθ0p_ _1>p_ _0. Proof. On the support of μ, πθ(y∣x)=pθπθ+(y∣x). _θ(y x)=p_θ\, _θ^+(y x). Hence Lμ(θ) L_μ(θ) =−logpθ−Y∼μlogπθ+(Y∣x) =- p_θ-E_Y μ _θ^+(Y x) (23) =−logpθ+H(μ)+Dθ, =- p_θ+H(μ)+D_θ, (24) where H(μ)=−Y∼μlogμ(Y)H(μ)=-E_Y μ μ(Y). Thus logpθ=H(μ)+Dθ−Lμ(θ). p_θ=H(μ)+D_θ-L_μ(θ). Taking the difference between θ1 _1 and θ0 _0 gives the equality in (22); the inequality follows from Dθ1≥0D_ _1≥ 0. ∎ Proposition 7 (Transfer under nonnegative rewards). Fix a prompt x and let R(x,y)≥0R(x,y)≥ 0. Define Jx(θ)=Y∼πθ(⋅∣x)[R(x,Y)],J_x(θ)=E_Y _θ(· x)[R(x,Y)], and assume Jx(θi)>0J_x( _i)>0 for i∈0,1i∈\0,1\. Let μ be a fixed distribution such that, for every y in its support, πθi(y∣x)R(x,y)>0,i∈0,1. _ _i(y x)R(x,y)>0, i∈\0,1\. Define the reward-weighted response distribution QθR(y∣x)=πθ(y∣x)R(x,y)Jx(θ),DθR=KL(μ∥QθR(⋅∣x)).Q_θ^R(y x)= _θ(y x)R(x,y)J_x(θ), D_θ^R=KL\! (μ\,\|\,Q_θ^R(· x) ). (25) For ΔL=Lμ(θ0)−Lμ(θ1), L=L_μ( _0)-L_μ( _1), we have logJx(θ1)Jx(θ0)=ΔL+Dθ1R−Dθ0R≥ΔL−Dθ0R. J_x( _1)J_x( _0)= L+D_ _1^R-D_ _0^R≥ L-D_ _0^R. (26) Proof. Expanding the KL divergence, DθR D_θ^R =Y∼μ[logμ(Y)Jx(θ)πθ(Y∣x)R(x,Y)] =E_Y μ [ μ(Y)J_x(θ) _θ(Y x)R(x,Y) ] (27) =−H(μ)+Lμ(θ)−Y∼μlogR(x,Y)+logJx(θ). =-H(μ)+L_μ(θ)-E_Y μ R(x,Y)+ J_x(θ). (28) The first and third terms are independent of θ. Taking the difference between θ1 _1 and θ0 _0 therefore gives the equality in (26). The inequality follows from Dθ1R≥0D_ _1^R≥ 0. ∎ Corollary 8 (Local alignment when the mismatch is small). Let fθ(y)=∇θlogπθ(y∣x)f_θ(y)= _θ _θ(y x) and suppose ‖fθ(y)‖≤G\|f_θ(y)\|≤ G on the supports of μ and QθRQ_θ^R. Define u=∇logJx(θ)u=∇ J_x(θ) and v=Y∼μfθ(Y)=−∇Lμ(θ)v=E_Y μf_θ(Y)=-∇ L_μ(θ). Then ‖v−u‖≤G2DθR.\|v-u\|≤ G 2D_θ^R. (29) In particular, small reward-weighted proposal mismatch makes the auxiliary SFT direction close to the gradient of log expected reward. Proof. Differentiating JxJ_x gives u=QθRfθu=E_Q_θ^Rf_θ. Hence ‖v−u‖≤2GTV(μ,QθR)≤G2KL(μ∥QθR),\|v-u\|≤ 2G\,TV(μ,Q_θ^R)≤ G 2\,KL(μ\|Q_θ^R), where the last step is Pinsker’s inequality. ∎ B.3 Adaptive assistance from unguided competence With index repeated visits to one prompt by t, let ℋtH_t contain the history before its unguided samples, including the current policy and hint level. Let ptp_t be its correctness probability under the unguided decoding law, and let CtC_t indicate at least one correct response among K conditionally independent unguided samples. The scheduler is gt+1=clip(gt+1−2Ct,0,N).g_t+1=clip(g_t+1-2C_t,0,N). (30) Proposition 9 (Conditional drift of the hint level). For 0<gt<N0<g_t<N, [gt+1−gt∣ℋt]=2(1−pt)K−1.E[g_t+1-g_t _t]=2(1-p_t)^K-1. (31) At gt=0g_t=0 the drift equals (1−pt)K(1-p_t)^K, and at gt=Ng_t=N it equals −[1−(1−pt)K]-[1-(1-p_t)^K]. Every positive hint level has negative drift whenever pt>p⋆:=1−2−1/K.p_t>p_ :=1-2^-1/K. (32) Proof. The unguided group contains no successful rollout with probability (1−pt)K(1-p_t)^K. For 0<gt<N0<g_t<N, this event increases gtg_t by one, while its complement decreases gtg_t by one, giving (31). The boundary expressions follow from clipping. The threshold p⋆p_ is obtained by solving 2(1−pt)K−1<02(1-p_t)^K-1<0. ∎ Corollary 10 (Expected time to remove assistance). Starting from g0=g_0=g, let T0=inft≥0:gt=0T_0= \t≥ 0:g_t=0\. If pt≥τ>p⋆p_t≥τ>p_ almost surely for all t<T0t<T_0, then [T0∣ℋ0]≤g1−2(1−τ)K.E[T_0 _0]≤ g1-2(1-τ)^K. (33) Proof. Set d=1−2(1−τ)K>0.d=1-2(1-τ)^K>0. By Proposition 9, for all t<T0t<T_0, [gt+1−gt∣ℋt]≤−d.E[g_t+1-g_t _t]≤-d. Applying this bound to the stopped process gives 0≤[gT0∧n∣ℋ0]≤g−d[T0∧n∣ℋ0].0 [g_T_0 n _0]≤ g-d\,E[T_0 n _0]. Letting n→∞n→∞ and using monotone convergence yields (33). ∎ B.4 Learning time in a two-response model Policy and update. This section analyzes a stylized two-response model intended to isolate the sparse-reward mechanism. Given one prompt with exactly two responses, one correct and one incorrect, the policy is parameterized by a scalar logit ztz_t with pt=σ(zt)p_t=σ(z_t). The rollout and likelihood laws coincide. At each visit, we draw nt∼Binomial(K,pt),K≥2,n_t (K,p_t), K≥ 2, where ntn_t is the number of correct unguided responses. Sampled advantages and accepted targets are held fixed during one fresh SGD step. For a population-standard-deviation group normalization with stabilizer ϵ≥0ε≥ 0, define VK(n)=nK(1−nK),bK,ϵ(n)=VK(n)VK(n)+ϵ,0<n<K,0,n∈0,K.V_K(n)= nK (1- nK ), b_K,ε(n)= cases V_K(n) V_K(n)+ε,&0<n<K,\\[6.0pt] 0,&n∈\0,K\. cases (34) With learning rate η>0η>0, no KL penalty or momentum, and a mean auxiliary loss over accepted responses from the current visit, the scalar update is zt+1=zt+η[bK,ϵ(nt)+λtAt(1−pt)],λt≥λmin>0.z_t+1=z_t+η [b_K,ε(n_t)+ _t1_A_t(1-p_t) ], _t≥ _ >0. (35) For binary reward and 0≤δ<10≤δ<1, AtA_t occurs exactly when nt=0n_t=0, gt>0g_t>0, and at least one guided response succeeds. We use the scheduler in Eq. (30) with Ct=nt>0C_t=1\n_t>0\. Every increment in Eq. (35) is nonnegative, so ptp_t is nondecreasing in this model. Useful-hint condition. Fix 0<p0<τ<10<p_0<τ<1. There exist a hint level g∗∈1,…,Ng_*∈\1,…,N\ and a constant c>0c>0 such that, whenever pt<τp_t<τ and gt≥g∗g_t≥ g_*, conditional on an all-failure unguided group, Pr(At∣ℋt,all-failure)≥c. (A_t _t,all-failure)≥ c. (36) If the KgK_g guided candidates are conditionally independent and each succeeds with probability at least smin>0s_ >0, then one may take c=1−(1−smin)Kgc=1-(1-s_ )^K_g. No condition is imposed for gt<g∗g_t<g_*. In particular, g∗=Ng_*=N allows the condition to hold only when the full rationale is revealed. Let Tτ=inft≥0:pt≥τ,T_τ= \t≥ 0:p_t≥τ\, and define ℓ=g∗+1,β=(1−τ)Kℓc,m=⌈logit(τ)−logit(p0)ηλmin(1−τ)⌉. =g_*+1, β=(1-τ)^K c, m= logit(τ)-logit(p_0)η _ (1-τ) . (37) Lemma 11 (Block progress). From any history prior to TτT_τ, a block of ℓ consecutive visits contains either an accepted guided response or a visit at which pt≥τp_t≥τ with conditional probability at least β. Proof. While t<Tτt<T_τ, an unguided group fails with conditional probability at least f:=(1−τ)K.f:=(1-τ)^K. Consider a block in which the target is not reached and no guided response is accepted during its first g∗g_* visits. If all of these unguided groups fail, the scheduler increases the hint level at each visit, so that the next visit has gt≥g∗g_t≥ g_*. An additional all-failure group then yields an accepted guided response with conditional probability at least c by Assumption B.4. Hence Pr(progress within the block∣ℋt)≥fg∗+1c=β. (progress within the block _t)≥ f^g_*+1c=β. ∎ Theorem 12 (Rare-success scaling). Under Eq. (35) and Assumption B.4, [Tτ] [T_τ] ≤ℓmβ, ≤ mβ, (38) Pr(Tτ>ℓn) (T_τ> n) ≤PrBinomial(n,β)<m ≤ \! \Binomial(n,β)<m \ (39) ≤exp(−(nβ−m)22nβ),nβ>m. ≤ \! (- (nβ-m)^22nβ ), nβ>m. For group-centered GRPO using the same response budget b=K+Kgb=K+K_g per visit and no auxiliary update, [TτGRPO]≥11−(1−p0)b−p0b≥1bp0.E[T_τ^GRPO]≥ 11-(1-p_0)^b-p_0^b≥ 1bp_0. (40) Therefore, for fixed K,Kg,g∗,τ,η,λminK,K_g,g_*,τ,η, _ and c, [Tτ]=O(log1p0),[TτGRPO]=Ω(1p0)as p0↓0.E[T_τ]=O\! ( 1p_0 ), [T_τ^GRPO]= \! ( 1p_0 ) p_0 0. (41) Proof. For t<Tτt<T_τ, an accepted guided response increases the logit by at least d:=ηλmin(1−τ),d:=η _ (1-τ), while all other terms in Eq. (35) are nonnegative. Thus m accepted responses suffice to increase the logit from logit(p0)logit(p_0) to logit(τ)logit(τ). Partition the visits into blocks of length ℓ . By Lemma 11, before TτT_τ each block has conditional probability at least β of producing an accepted response or reaching the target. The resulting block indicators stochastically dominate i.i.d. Bernoulli(β)(β) variables. On the event Tτ>ℓn\T_τ> n\, no block can reach the target, and therefore fewer than m blocks can contain an acceptance. This gives (39). The expectation bound follows by comparison with a negative-binomial random variable with parameters (m,β)(m,β). For GRPO, the policy does not change until the first group containing both a successful and an unsuccessful response. Prior to this event, the success probability remains p0p_0. Its waiting time is geometric with parameter q=1−(1−p0)b−p0b.q=1-(1-p_0)^b-p_0^b. Since reaching τ>p0τ>p_0 requires at least one nonzero update, [TτGRPO]≥q−1E[T_τ^GRPO]≥ q^-1. Moreover, q≤bp0q≤ bp_0, which gives (40). Finally, logit(τ)−logit(p0)=log1p0+O(1),logit(τ)-logit(p_0)= 1p_0+O(1), and hence m=O(log(1/p0))m=O( (1/p_0)) as p0↓0p_0 0. ∎ Corollary 13 (Time to competence without guidance). Under the conditions of Theorem 12, suppose τ>p⋆=1−2−1/Kτ>p_ =1-2^-1/K, and define Tzero=inft≥0:pt≥τ,gt=0.T_ zero= \t≥ 0:p_t≥τ,\ g_t=0\. Then [Tzero]≤ℓmβ+N1−2(1−τ)K.E[T_ zero]≤ mβ+ N1-2(1-τ)^K. (42) Proof. Equation (35) implies that ptp_t is nondecreasing, so pt≥τp_t≥τ for all t≥Tτt≥ T_τ. Applying Corollary 10 at TτT_τ and using gTτ≤Ng_T_τ≤ N gives the result. ∎ Appendix C Experimental Setup C.1 Language-Model Experiments We implement all language-model post-training experiments in verl using the GRPO advantage estimator and vLLM for rollout generation. Training uses a global batch size of 256 prompts. For each prompt, we sample four on-policy responses. Responses are allowed up to 4,096 tokens, while reference hints are truncated to at most 512 tokens. Because guided generation appends additional context to the original problem, we allow a maximum re-prompted sequence length of 6,000 tokens. Prompts longer than the configured 1,024-token input limit are filtered before training rather than silently truncated. Table 3: Training configuration for the language-model and vision-language-model experiments. We report the optimization and sequence-length settings shared across RGPO and its corresponding baselines. Hyperparameter LM VLM Training epochs 3 3 Batch size 256 512 Rollouts per prompt 4 4 Learning rate 1×10−61× 10^-6 1×10−61× 10^-6 Max. prompt length 1,024 768 Max. response length 4,096 1,024 Max. hint length 512 512 Max. guided prompt length 6,000 2,560 Entropy coefficient 1×10−31× 10^-3 0 KL regularization None None Precision BF16 BF16 Rollout samples K 4 + 4 4 + 4 We optimize the actor with a learning rate of 1×10−61× 10^-6 for three epochs. The PPO minibatch size is 256, corresponding to one optimizer minibatch per global training batch, with gradient accumulation through smaller per-GPU microbatches. Training uses bfloat16 parameters, FSDP2 sharding, gradient checkpointing, and fused model kernels. We do not use a KL penalty either in the actor objective or as part of the reward. We use a small entropy coefficient of 10−310^-3 for the language-model experiments. Rollouts are generated with tensor-parallel size 1 and four samples per prompt. We use an actor microbatch size of 8 and a rollout log-probability microbatch size of 16 per GPU. C.2 Vision-Language Training The vision-language experiments use the same GRPO/RGPO implementation, with multimodal inputs enabled throughout rollout and optimization. As in the language-model experiments, we disable a learned reward model and use a programmatic verifier to determine response correctness. We use a larger global batch of 512 prompts and sample four responses per prompt. The maximum text prompt length is 768 tokens, the generated response length is 1,024 tokens, and at most 512 tokens of the reference rationale are exposed as guidance. Guided re-prompting is allowed up to 2,560 tokens to accommodate the original multimodal prompt, partial rationale, and refinement instruction. Over-length input prompts are filtered rather than truncated. The actor is optimized for three epochs with learning rate 1×10−61× 10^-6 and PPO minibatch size 512. Training uses FSDP2, bfloat16 precision, gradient checkpointing, and fused kernels. We use neither KL regularization nor an entropy bonus in the vision-language experiments. Rollouts are generated with vLLM using tensor-parallel size 1 and four samples per prompt. Chunked prefill is enabled for the VLM rollout engine. The actor microbatch size is 32 per GPU and the rollout log-probability microbatch size is 64 per GPU. C.3 Baselines On-policy RLVR. Our primary on-policy baseline is GRPO [7], which estimates advantages by comparing rewards among multiple responses sampled for the same prompt and avoids training a separate value model. We also evaluate REINFORCE++ [27] in the learning-efficiency experiments. REINFORCE++ is a critic-free REINFORCE variant that uses global advantage normalization together with PPO-style stabilization techniques. Comparing RGPO against both objectives tests whether its gains arise from rationale-guided exploration rather than from a particular underlying RLVR optimizer. Off-policy supervision and guided exploration. We compare against several recent methods that address sparse or uninformative on-policy rewards by introducing external reasoning signals. LUFFY [8] mixes off-policy expert reasoning traces with on-policy rollouts and uses policy shaping to control the influence of demonstrations. CHORD [17] incorporates expert solutions as a dynamically weighted supervised objective within on-policy RL, gradually balancing imitation and exploration. These methods directly optimize toward externally provided expert trajectories. A second family instead uses external information to make difficult rollouts more tractable. GHPO [9] adaptively adjusts the amount of guidance according to the model’s current ability and combines imitation-oriented learning on difficult examples with RL on more accessible ones. QuestA [19] augments difficult questions with partial solutions during RL, effectively reducing their difficulty and increasing the probability of obtaining informative rollouts. BREAD [10] introduces short expert prefixes when self-generated trajectories fail and branches new rollouts from these expert-provided intermediate states. HINT [11] supplies heuristic guidance to ineffective rollouts while explicitly seeking to keep the guided trajectories close to the current policy. For the additional prompt-format analysis, we also compare with AHPO from M-HELIX [57], which combines adaptive off-policy supervision with online RL. Multimodal reasoning baselines. For controlled multimodal comparisons, we include the supervised reasoning variants from MINT-CoT [43]: Original Image CoT SFT, Bounding-Box CoT SFT, and Text-only CoT SFT. These baselines use the same Qwen2-VL backbone family and directly supervise the model with different forms of multimodal reasoning traces, providing a comparison between rationale supervision and rationale-guided RL. We further report representative open multimodal reasoning systems. R1-VL [48] improves multimodal reasoning through StepGRPO with step-wise rule-based rewards; Open-R1-Multimodal [49] provides an open implementation of GRPO-style multimodal RL; Mulberry [26] constructs explicit multimodal reasoning trajectories using collective Monte Carlo tree search and trains on them with supervised learning; and M-Eureka [50] applies large-scale rule-based reinforcement learning to multimodal reasoning. Finally, we include strong general-purpose VLMs as reference points: LLaVA-OneVision-Qwen2-7B-OV [51], InternVL2-8B [52], InternVL2-8B-MPO [53], DeepSeek-VL2 [54], and Qwen2.5-VL-7B-Instruct [55]. We report these systems to place RGPO’s multimodal reasoning performance in the broader landscape rather than as strictly matched baselines, since they use different backbones, training corpora, and post-training recipes. M-HELIX experiment. To examine whether RGPO requires a high-quality expert chain-of-thought, we additionally train Qwen2.5-VL-7B-Instruct on the 24-Points subset of M-HELIX-100K [57]. M-HELIX constructs these problems using a deterministic solver and converts intermediate solver states into programmatically generated reasoning traces. In our RGPO experiment, we use only these solver-derived traces as rationale scaffolds; we do not use the longer natural-language rationales rewritten by Qwen3-235B. We further consider a feedback-only variant, RGPO w/o CoT, in which no solution rationale is provided. Instead, unsuccessful responses are refined using diagnostic feedback from the deterministic 24-Points evaluator, which checks output format, use of the required numbers, arithmetic correctness, and evaluability. This setting tests whether RGPO can benefit from weaker forms of structured guidance that do not require an expert-generated reasoning trajectory. C.4 Controlled Simulation Setup Synthetic reasoning problems. Each problem consists of a short sequence of reasoning decisions followed by a final answer. Decision types are shared across problems, so improving the policy on a particular type of decision can transfer to other problems in which the same decision recurs. The simulator contains 40 such decision types and 16 possible actions for each decision. We divide the decision types into 32 easy and 8 hard types. For an easy decision, one or two actions are valid and the policy is initialized to assign substantially higher probability to these actions. Its initial probability of making a valid easy decision is therefore approximately 0.780.78–0.890.89. For a hard decision, only one of the 16 actions is valid and the policy is initialized uniformly, giving an initial success probability of 1/161/16. Easy problems contain two to five easy decisions. Hard problems contain five to seven decisions, four of which are hard. Because a reasoned response is correct only when every decision is valid, obtaining a complete solution to a hard problem by unguided sampling is unlikely at initialization. Hard decision types do not occur in easy problems, preventing the easy subset from implicitly teaching the difficult decisions. We independently sample 512 training problems and 256 held-out test problems for each random seed, with hard problems comprising 25%25\% of both splits. Responses and reward. The policy can either produce a reasoning trajectory or answer the problem directly. In reasoning mode, it samples one action for each decision in the problem, and the response receives reward 1 only if every sampled action is valid. In direct-answer mode, it chooses among 100 possible answers and receives reward 1 only for the gold answer. The initial probability of choosing the direct-answer mode is 0.20.2; the answer distribution itself is initially uniform. The direct-answer path serves two purposes. First, it provides a weak alternative to multi-step reasoning, analogous to guessing an answer without constructing a valid solution. Second, when the complete reference is revealed, the gold answer becomes visible, allowing us to study whether a method learns to rely on the additional information rather than improving its unguided reasoning policy. Reference rationales and hints. Each problem is paired with a reference rationale containing one recommended action for every reasoning decision. To represent an informative but imperfect rationale, we systematically corrupt the reference for three of the eight hard decision types: whenever one of these types occurs, the reference recommends the same fixed invalid action. The remaining reference steps are correct. The reference can be revealed at three levels, g∈0,1,2g∈\0,1,2\. At g=0g=0, no rationale is provided. At g=1g=1, the first half of the reference reasoning trajectory is available, and at g=2g=2, the complete trajectory and final answer are visible. At a revealed reasoning step, the generated trajectory follows the reference action with probability ρ=0.7ρ=0.7; otherwise, the policy generates the step from its own action distribution. This construction allows guidance to influence generation without completely determining the resulting trajectory. Table 4: Controlled simulation configuration. We construct the environment so that hard problems rarely yield a correct unguided trajectory at initialization, while the reference rationale provides useful but imperfect guidance. Parameter Value Decision types 40 total: 32 easy and 8 hard Actions per decision 16 Problem composition Easy: 2–5 easy decisions; hard: 5–7 decisions with 4 hard decisions Train / test problems 512 / 256, with 25% hard problems Reference corruption 3 of the 8 hard decision types have systematically incorrect reference actions Hint levels g∈0,1,2g∈\0,1,2\: no hint, partial rationale, or full rationale Hint-following probability ρ=0.7ρ=0.7 Unguided / guided rollouts K=4K=4 / K′=4K =4 Initial direct-answer probability 0.20.2 Acceptance margin δ=0δ=0 Auxiliary-loss weight λ=1.0λ=1.0 Batch size 32 problems Training 3 epochs (16 updates per epoch) Learning rate 64 Evaluation 5 random seeds; all methods evaluated without hints Training strategies. All methods use the same policy, training problems, reward function, and optimization budget as described in Table 4. Training proceeds for three epochs with minibatches of 32 problems, corresponding to 16 parameter updates per epoch. We use plain gradient ascent with learning rate 64 and no KL regularization. Each unguided group contains K=4K=4 responses. GRPO uses only the four unguided responses. Advantages are computed by normalizing rewards within each group; when every response receives the same reward, the group contributes no policy-gradient update. RGPO begins with the same unguided GRPO update. All problems start with g=0g=0. After each epoch, the hint level of a problem is increased by one if none of its unguided responses was correct during that epoch and decreased by one otherwise, subject to 0≤g≤20≤ g≤ 2. For a problem with g>0g>0, we sample an additional K′=4K =4 guided responses. A guided response is retained only if it is correct while all corresponding unguided responses are incorrect. Accepted responses are then optimized under the original problem without the hint. The auxiliary supervised loss is averaged over accepted guided responses in the current minibatch and weighted by λ=1λ=1. We include several variants to isolate how the same reference information is used. RGPO w/o gate distills every guided response rather than retaining only verified improvements. RGPO, full hint always follows the RGPO procedure but provides the complete reference for every problem throughout training. RL + SFT(ref) combines unguided GRPO with supervised learning directly on the reference rationale for every problem, while RL + SFT(ref) on failure applies the reference-SFT objective only when all unguided responses for a problem fail. Hint-conditioned RL applies an additional GRPO update to responses generated in the adaptively hinted context rather than distilling them under the original problem. Finally, Full hint performs GRPO entirely on responses generated with the complete reference available. All methods are evaluated on the original problems without hints. Thus, performance measures what has transferred to the unguided policy rather than what the model can accomplish while the reference remains available. pθ(x)=P(reason)∏j=1Lxπθ(A(k(x,j)))+P(direct)softmax(ψx)ax,p_θ(x)=P(reason) _j=1^L_x _θ (A(k(x,j)) )+P(direct)\,softmax( _x)_a_x, where A(k)A(k) denotes the set of valid actions for decision type k and axa_x is the gold answer. We report the mean of pθ(x)p_θ(x) over a split as accuracy and the mean of −logpθ(x)- p_θ(x) as negative log-likelihood (NLL). This removes sampling noise from evaluation and makes it possible to examine changes in the policy even when the probability of completing an entire hard problem remains small. C.5 Hardware configuration All experiments were conducted on high-performance machines equipped with Intel Xeon CPUs and NVIDIA GPUs. Specifically, we utilized two machine configurations: (1) INTEL(R) XEON(R) PLATINUM 8592+ with 2.0Ti RAM and H200 SXM 141GB GPUs and (2) Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 503Gi RAM and NVIDIA A100 SXM4 80GB GPUs. Although different GPU types were used to balance workload priorities, we ensured that all comparisons across methods were performed on the same hardware configuration for a given model and dataset to eliminate hardware-induced variability and maintain consistency and fairness in evaluation. C.6 Prompt templates RGPO uses rationales as temporary scaffolds to elicit an improved response. To implement this idea, we design a refinement prompt that conditions the model on three pieces of information: the original problem, a partial solution prefix, and feedback on its previous attempt. The partial solution provides a rationale guide, while the feedback signal indicates whether the previous answer failed due to formatting, accuracy, or both. Importantly, the model is instructed to generate a new standalone solution to the original question, rather than to comment on or revise the previous response. This prevents the model from learning a conversational repair pattern and instead encourages it to internalize the reasoning needed to solve the original task. Refinement prompt prompt This is the (partial) solution without formatting, you can leverage it to check your answer: solution Previous response: response Feedback for your previous response: Format score: format_score/1.0 Accuracy score: accuracy_score/1.0 Overall score: score/1.0 You should maximize the overall score by improving both the format and accuracy of your answer. Write a new standalone solution to the original question without mentioning the previous responses or feedback. You first think through the reasoning process as an internal monologue, enclosed within <think> </think> tags. Then, provide your final answer enclosed within . Please strictly follow the format. The refinement prompt is shown above. Here, prompt denotes the original question, solution denotes the currently revealed rationale segment, response denotes the model’s previous unguided response, and the feedback fields are obtained from the verifier reward. The instruction explicitly asks the model to maximize both format and accuracy, while enforcing the same answer format used during training and evaluation. This is because we found in practice that the model can sometimes respond with the correct option but in the wrong format due to the long refinement-instruction prompt. For the language-only experiments, we use the following prompt, closely following [11]. DAPO’s system prompt You are a helpful AI assistant. A conversation takes place between the User and the Assistant. The User asks a question, and the Assistant solves it. Please help me solve this question. Wrap only the final answer in . For training vision-language models, we adopt a concise reasoning format inspired by DeepSeek-R1. The model is instructed to place its reasoning inside <think> and </think> tags and to wrap the final answer in . Compared with the Mulberry prompt used in MINT-CoT, this format is simpler and imposes fewer structural constraints on the model output. In the appendix, we further analyze how this prompt design affects learning dynamics and final performance. DeepSeek-R1’s system prompt You first think through the reasoning process as an internal monologue, enclosed within <think> </think> tags. Then, provide your final answer enclosed within . This prompt is more concise and straightforward than the Mulberry-style prompt used in the MINT-CoT dataset, which decomposes the response into multiple explicit sections, including image description, rationale, step-by-step reasoning, and final answer. While such a structured format may encourage detailed multimodal reasoning, it also imposes a more complex output schema. Mulberry’s system prompt Generate an image description based on the question. Then, provide a rationale to analyze the question. Next, generate a step-by-step reasoning process to solve the problem. Ensure the steps are logical and concise. Finally, provide a concise summary of the final answer in the following format: The final answer is: x. Format your response with the following sections, separated by ###: ### Image Description: ### Rationales: ### Let’s think step by step. ### Step 1: ### Step 2: … ### The final answer is: Appendix D Additional experimental results D.1 Controlled Simulation Figure 6 shows both no-hint accuracy and NLL on the complete training and test sets and separately on easy and hard problems. GRPO rapidly learns the easy subset, reaching 99.1%99.1\% test accuracy, but remains at 0.0%0.0\% on hard problems after three epochs. The NLL curves in Figure 6 make the underlying behavior more visible. Once direct answering is suppressed, solving a hard problem requires producing all four difficult decisions correctly in the same trajectory. Almost all hard-problem groups therefore contain only failures, yielding no relative reward signal for GRPO. The easy decisions continue to improve, but the hard decisions remain close to their initialization. Hint-conditioned RL receives more information than GRPO, but transfers it inefficiently in this short training horizon. Its hard-problem NLL decreases even though final hard accuracy remains only 0.4%0.4\%, indicating that the policy is beginning to improve individual decisions without yet assigning enough probability to a completely correct multi-step trajectory. Figure 6: Full learning dynamics in the controlled simulation. Accuracy without hints and corresponding negative log-likelihood on training and held-out problems, shown for all, hard, and easy examples. Curves report mean ± s.e. over five seeds. All methods are evaluated on the original problems without access to the reference rationale. Both reference-SFT variants reach 12.8%12.8\% hard test accuracy. Their improvement over GRPO comes from directly supervising the difficult decisions for which the reference is correct. However, the same objective repeatedly reinforces an invalid action for the three hard decision types whose references are systematically corrupted. Problems involving these decisions therefore remain unsolved. The same behavior appears in RGPO w/o gate. Although its guided trajectories are generated by the policy rather than copied directly from the reference, removing the verifier-based acceptance criterion allows incorrect guided responses to enter the supervised objective. Its hard test accuracy consequently matches the reference-SFT baselines at 12.8%12.8\%. This comparison isolates the role of verification: guidance is useful only when the resulting trajectory is checked before it becomes a training target. Table 5: Final no-hint accuracy (%) in the controlled simulation after three training epochs. Values are mean ± standard deviation over five seeds. The hard test split is the primary diagnostic: it contains problems for which successful unguided reasoning trajectories are initially extremely unlikely. Method Test Test (hard) Test (easy) Train (hard) GRPO 74.2±1.574.2± 1.5 0.0±0.00.0± 0.0 99.1±0.199.1± 0.1 0.0±0.00.0± 0.0 Full hint 1.0±0.01.0± 0.0 1.0±0.01.0± 0.0 1.0±0.01.0± 0.0 1.0±0.01.0± 0.0 Hint-conditioned RL 74.4±1.474.4± 1.4 0.4±0.80.4± 0.8 99.3±0.199.3± 0.1 0.4±0.90.4± 0.9 RL + SFT(ref) 77.9±0.777.9± 0.7 12.8±5.612.8± 5.6 99.7±0.099.7± 0.0 18.7±1.618.7± 1.6 RL + SFT(ref) on failure 77.8±0.877.8± 0.8 12.8±5.612.8± 5.6 99.6±0.199.6± 0.1 18.7±1.618.7± 1.6 RGPO w/o gate 77.8±0.877.8± 0.8 12.8±5.612.8± 5.6 99.5±0.199.5± 0.1 18.7±1.618.7± 1.6 RGPO, full hint always 1.0±0.01.0± 0.0 1.0±0.01.0± 0.0 1.0±0.01.0± 0.0 71.2±0.571.2± 0.5 RGPO 99.4±0.199.4± 0.1 99.5±0.299.5± 0.2 99.3±0.199.3± 0.1 99.5±0.199.5± 0.1 Table 5 reports the corresponding final accuracies after three epochs. The Full hint baseline illustrates a different failure mode. Because the complete reference exposes the gold answer, the policy can obtain perfect reward in the guided context by relying on that information. When the reference is removed for evaluation, however, accuracy falls to 1%1\%, exactly the chance rate of selecting among 100 answers. A related effect appears for RGPO, full hint always. Distilling full-hint responses allows the policy to fit many of the training answers, reaching 71.2%71.2\% accuracy on hard training problems, but this behavior does not transfer to held-out problems: hard test accuracy remains at 1%1\%. Thus, high reward under guidance does not by itself imply that the policy has learned a reasoning strategy that can be executed from the original input. RGPO behaves differently because the rationale determines where the policy searches, but not which trajectory is ultimately trusted. A guided response contributes to the auxiliary objective only when it passes the verifier and improves over all unguided responses for the same problem. The accepted response is then optimized under the original problem without the rationale. This produces a rapid change in the hard-problem learning dynamics. Once a verified guided trajectory improves one of the difficult decision types, that improvement transfers to other problems containing the same type. These problems in turn become more likely to produce successful unguided rollouts, allowing the ordinary RL objective to begin contributing useful signal. After three epochs, RGPO reaches 99.5%99.5\% hard test accuracy and 99.4%99.4\% overall test accuracy while retaining 99.3%99.3\% performance on easy problems. D.2 Fine-Grained Benchmark Results Figure 7 presents a category-level comparison of the base model, GRPO, and RGPO on MathVista. Figure 7: Category-wise performance comparison of Qwen2-VL-2B on MathVista. We compare the base model, GRPO, and RGPO across different mathematical reasoning categories. RGPO achieves the best overall accuracy and consistently improves over both the base model and GRPO on most categories, with the largest gain observed on textbook question answering. RGPO achieves the highest overall accuracy, improving from 36.7 for the base model and 39.6 for GRPO to 40.7. The gains are consistent across several reasoning categories, including geometry reasoning, algebraic reasoning, geometry problem solving, and textbook question answering. The improvement is especially pronounced on textbook question answering, where RGPO reaches 53.2, outperforming GRPO by 6.4 points and the base model by 8.0 points. This suggests that rationale-guided refinement is particularly helpful for problems requiring structured interpretation and multi-step reasoning. On statistical reasoning, all methods achieve perfect accuracy, while on arithmetic reasoning both GRPO and RGPO improve over the base model. Overall, these results show that RGPO provides a stronger optimization signal than standard RLVR-style training, leading to better no-hint reasoning performance across most MathVista categories. Figure 8: Category-wise performance comparison of Qwen2-VL-2B on MMStar: We compare the base model, GRPO, and RGPO across the mathematical reasoning categories in MMStar. RGPO achieves the highest overall accuracy and improves over both baselines on geometry and numeric commonsense/calculation tasks, while matching GRPO on statistical reasoning. Figure 8 reports the category-wise performance of the base model, GRPO, and RGPO on MMStar. RGPO achieves the best overall accuracy, improving from 36.8 for the base model and 41.2 for GRPO to 45.2. The largest gain appears in geometry, where RGPO reaches 34.5, outperforming GRPO by 7.8 points and the base model by 10.4 points. RGPO also improves numeric commonsense and calculation accuracy to 58.3, compared with 56.2 for GRPO and 50.0 for the base model. On statistical reasoning, RGPO matches GRPO at 52.3, with both methods outperforming the base model. These results show that RGPO provides consistent benefits on MMStar, especially for visually grounded and calculation-heavy reasoning tasks where rationale-guided refinement can help convert failed attempts into more informative training signals. D.3 Dynamics of Adaptive Rationale Guidance Figure 9 illustrates how RGPO adaptively controls the use of rationale guidance during training. In the first epoch, no hints are provided, allowing the algorithm to estimate the model’s unguided problem-solving ability. Once rationale-guided refinement is activated in the second epoch, the non-empty hint fraction rises to around 20%, indicating that only a minority of examples require scaffolded assistance. By the third epoch, this fraction drops to roughly 15%, suggesting that many examples that previously required guidance can now be solved without hints. This behavior supports the central design of RGPO: rationales are not used as always-on supervision, but as temporary scaffolds for difficult examples. As training progresses, the model becomes less dependent on external rationale guidance, making full-solution exposure for every sample unnecessary. Figure 9: Fraction of training examples receiving rationale guidance: The hint rate remains zero during the probing epoch, rises when RGPO begins guided refinement, and decreases as the model learns to solve more examples independently. D.4 Beyond Expert Rationale Scaffolds The experiments above use reference solutions as a convenient source of rationale scaffolds. We next ask whether RGPO fundamentally depends on such expert-written reasoning traces. To separate the benefit of scaffolding from the quality of the rationale source, we evaluate RGPO on the 24-Points task from M-HELIX [57], where guidance can be obtained directly from the underlying programmatic solver. Samples from this dataset are visually presented in Figure 10. Figure 10: Representative 24-Points examples used in our M-HELIX experiment. Each problem presents four numbers and requires constructing an arithmetic expression that uses every number exactly once and evaluates to 24. For RGPO-7B, we use the mechanical solver traces provided by the M-HELIX data-generation pipeline as rationale scaffolds. These traces encode valid intermediate reasoning states but are substantially shorter and less natural than the Qwen3-235B-rewritten trajectories used by M-HELIX-7B. We therefore deliberately give RGPO the less polished source of supervision rather than the expert-rewritten chain-of-thought. We also evaluate RGPO-7B w/o CoT, which removes the solution trace entirely. When an unguided response fails, this variant receives only diagnostic feedback from the deterministic task verifier and uses that feedback to refine its response. Table 6: Programmatic feedback used by RGPO-7B (w/o CoT). The judge supplies diagnostic information about the model’s current attempt without revealing a correct solution. The example uses the input numbers [2,5,10,12][2,5,10,12]. # Error category Trigger condition Feedback returned to the model (example, numbers [2, 5, 10, 12]) 1 No expression / format No arithmetic operator found in the extracted answer “No arithmetic expression was found in your answer. Combine [2, 5, 10, 12], using each number exactly once, with +, -, ×, ÷ and parentheses to make 24 …” 2 Number-usage (missing) Used numbers ≠ given numbers (multiset) “Number-usage error: you did not use 2. You must use each of [2, 5, 10, 12] exactly once and use no other numbers.” 3 Number-usage (wrong count / extra) Same, with per-number diagnosis “you used 2 2x but only 1 is available; you used 9, which is not one of the given numbers.” 4 Wrong value — too small Correct numbers, result <24<24 “You used the right numbers, but your expression (12−10)×5+2(12-10)× 5+2 equals 12, which is 12 short of 24 — make the result larger.” 5 Wrong value — too large Correct numbers, result >24>24 “…equals 48, which is 24 above 24 — make the result smaller.” 6 Division by zero Expression divides by zero “Your expression divides by zero. Rearrange it so that no division by zero occurs. …” 7 Unevaluable Malformed / invalid characters / unbalanced parens “Your expression could not be evaluated. Use only [2, 5, 10, 12] with +, -, ×, ÷ and balanced parentheses, and nothing else.” 8 Correct Right numbers and value =24=24 “Correct — your expression equals 24 using each of [2, 5, 10, 12] exactly once.” We further remove solution rationales entirely in RGPO-7B (w/o CoT). In this setting, failed responses receive only diagnostic feedback from the deterministic task judge. The judge does not reveal a correct expression or a solution trajectory. Instead, it identifies why the current attempt fails, and the policy uses this feedback to generate a revised candidate response. As in standard RGPO, the resulting candidate must satisfy the task verifier before it can provide a learning signal. For 24-Points, the judge first extracts and normalizes the arithmetic expression in the model response and then applies a deterministic sequence of checks. Rather than returning only a binary correctness signal, it produces a short diagnosis of the first detected failure. Table 6 summarizes the feedback used by RGPO-7B (w/o CoT). This feedback is intentionally weaker than a solution rationale. For example, when an expression evaluates to 12, the judge tells the model that the result is 12 below the target and should be increased; it does not specify which operation to change or provide a correct expression. Similarly, number-usage errors identify the violated constraint without suggesting a solution. The refinement step must therefore remain policy-generated. Table 7: RGPO without expert solution rationales on the M-HELIX 24-Points task. RGPO-7B uses only programmatically generated solver traces, whereas RGPO-7B (w/o CoT) receives no solution rationale and refines failed attempts using only deterministic verifier feedback. Model Accuracy Response length Qwen2.5-VL-32B-Instruct 36.67 2226.5 WeThink-7B 16.67 3054.0 VLAA-Thinker-7B 10.00 3431.1 Llama-3.2-11B-Vision 3.33 1349.7 M-HELIX-7B 33.33 12667.1 RGPO-7B 40.00 9584.9 RGPO-7B (w/o CoT) 36.67 10592.8 Despite receiving no expert-written rationale, RGPO-7B achieves the highest accuracy in this comparison at 40.00%40.00\% (Table 7) using only the mechanical solver traces, outperforming M-HELIX-7B (33.33%33.33\%). More strikingly, RGPO-7B (w/o CoT) reaches 36.67%36.67\% without any solution rationale, outperforming M-HELIX-7B and matching the substantially larger Qwen2.5-VL-32B-Instruct model in this evaluation. These results suggest that the useful abstraction in RGPO is broader than rationale-prefix supervision. What the refinement stage requires is a scaffold that makes an improved trajectory easier to discover; that scaffold can be a partial solution, a mechanically generated trace, or structured feedback describing why the current attempt failed. The final training signal still comes from the policy’s own verifier-approved response rather than from directly imitating the guidance source. D.5 Effect of Prompt Format In Figure 11, we show that prompt format has a significant effect on both optimization and efficiency. The Mulberry-style prompt requires image descriptions, rationales, step-by-step sections, additional delimiters, and line breaks, produces much longer responses throughout training, typically around 380–400 tokens. Despite this higher generation cost, its reward remains nearly flat, suggesting that the complex output schema makes the format difficult to learn and does not translate into better task performance without additional supervision (e.g. from SFT loss in the AHPO baseline in the main paper). In contrast, the DeepSeek-R1-style prompt shows a clear reward increase after training begins to improve, while its response length drops sharply to below 100 tokens, indicating that the model learns a more concise and effective reasoning format. This reduction in response length also lowers the runtime per step, making training more efficient. These results motivate our use of the simpler DeepSeek-R1-style prompt: it reduces formatting burden, avoids unnecessary verbosity, and enables more stable learning without requiring extra supervised fine-tuning to enforce the response structure. Figure 11: Effect of prompt format on training dynamics and efficiency. We compare the Mulberry-style structured prompt with the simpler DeepSeek-R1-style prompt in terms of reward, response length, and runtime per training step. The DeepSeek-R1-style prompt leads to substantially better learning while producing shorter responses and reducing training time. D.6 Robustness to Rationale Quality Figure 12 evaluates whether training with rationale guidance leads to improved no-hint performance or merely teaches the model to rely on hints. On the left, both methods are evaluated with the correct CoT appended to the question as a rationale hint. The full-hint baseline achieves the highest accuracy, 95.2, slightly outperforming RGPO at 93.8, which is expected because the baseline is optimized to maximally exploit the provided hint. However, this advantage disappears in the original no-hint setting, where RGPO improves over the full-hint baseline from 33.6 to 38.4. The difference becomes more pronounced under noisy hints, where the CoT is shuffled and appended with an instruction that the model may use it if helpful. In this setting, the full-hint baseline drops to 20.4, suggesting that it tends to trust and follow the misleading rationale rather than solve the problem independently. RGPO achieves a higher score of 25.5, indicating better resistance to corrupted guidance. These results support the motivation of RGPO: rationales should serve as temporary scaffolds during training, not as permanent dependencies. Figure 12: Robustness to helpful and noisy rationale hints on MMMath: We compare RGPO with a full-hint baseline under three evaluation settings: correct rationale hints, the original no-hint benchmark, and shuffled noisy hints. While the full-hint baseline performs slightly better when the correct CoT is provided, RGPO achieves higher accuracy in the original and noisy-hint settings, indicating stronger robustness and less dependence on external rationales. By selectively using hints only after failure and optimizing for refined solutions, RGPO improves no-hint reasoning while reducing over-reliance on potentially noisy rationales from MMMath [61]. This aligns with recent concerns about over-privileging where they can leak rationales and degrade validation performance, while trained models may overuse unnecessary rationales [62, 63]. Always exposing a full or noisy CoT creates a similar shortcut, encouraging the model to rely on the rationale rather than solve independently. RGPO mitigates this risk by providing hints only after unguided failure, verifier-filtering guided outputs, and distilling accepted responses back to the no-hint prompt. D.7 Qualitative Analysis Figure 13: Qualitative comparison of model responses: Example outputs from the base model, GRPO, and RGPO. We provide qualitative examples comparing the responses generated by the base model, GRPO, and RGPO in Figure 13 and Figure 14. Compared with the base model, GRPO and RGPO often yield more focused reasoning processes. Moreover, by using rationales as temporary scaffolds, RGPO is able to identify the relevant solution path and answer questions more concisely. Figure 14: Qualitative comparison of model responses: Example outputs from the base model, GRPO, and RGPO. Figure 15: Qualitative comparison of model responses: Example outputs from the base model, GRPO, and RGPO. Appendix E Limitations In its standard form, RGPO assumes a reference rationale or another informative guidance source, and its effectiveness depends on the quality and granularity of that guidance. The feedback-only M-HELIX variant relaxes the need for a solution rationale, but requires task-specific diagnostic verifier feedback. If the revealed prefix contains too much answer-specific information, the method may encourage shortcut learning or memorization rather than transferable reasoning; if it contains too little or is stylistically far from the model’s own reasoning distribution, it may fail to rescue hard examples from sparse-reward collapse. Moreover, the reward-gated acceptance rule only guarantees improvement under the available verifier, not genuine reasoning quality, so RGPO can still inherit verifier bias, reward hacking, or spurious formatting gains. The adaptive schedule also introduces additional compute from guided rollouts and auxiliary SFT updates, making the method more expensive than a single-pass PPO or GRPO baseline. However, we keep the number of rollouts per prompt the same as in the GRPO baseline to ensure a fair comparison. Appendix F Broader Impacts RGPO aims to make reasoning-oriented post-training more sample-efficient by using reference rationales as adaptive training-time scaffolds, which may help smaller language and vision-language models learn from sparse verifiable rewards without relying as heavily on large teacher models, rejection sampling, or fully formatted supervised traces; this could benefit education, scientific analysis, software assistance, mathematical problem solving, and multimodal question answering. However, stronger reasoning ability is inherently dual-use: the same improvements that help benign problem solving may also assist harmful planning, automated persuasion, misinformation generation, cyber misuse, or other strategic tasks. RGPO also improves verifier reward rather than guaranteeing faithful reasoning, so generated rationales may remain incomplete, misleading, or optimized for the reward function rather than for truth, especially when verifiers or reference solutions contain biases, shortcuts, or annotation artifacts. These risks suggest that RGPO should be paired with careful verifier design, dataset auditing, robustness and out-of-distribution evaluation, and appropriate deployment safeguards for high-capability models. The method also adds compute overhead through guided rollouts and auxiliary distillation, although this cost may be offset in some settings if adaptive hinting improves sample efficiency or enables smaller models to reach competitive reasoning performance.