Paper deep dive

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Ken Ding

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 41

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 8/1/2026, 2:13:19 AM

Summary

The paper introduces LoRA Scaffolded Policy Optimization (LSPO), a method to recover reinforcement learning gradients on 'cliff' prompts where standard Group Relative Policy Optimization (GRPO) fails due to zero reward variance. LSPO uses a transient low-rank adapter (LoRA) to generate successful rollouts for these hard prompts, which are then spliced back into the training batch with importance-sampling corrections. The adapter is discarded after use, yielding a base-only model. Experiments on DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B show LSPO outperforms the DAPO baseline across multiple benchmarks.

Entities (9)

LSPO · method · 98%DeepSeek-R1-Distill-Qwen-1.5B · model · 97%DEEPMATH-103K · dataset · 95%GRPO · algorithm · 95%LoRA · technique · 95%Cliff Prompt · concept · 93%DAPO · algorithm · 92%AIME24 · benchmark · 90%MATH500 · benchmark · 90%

Relation Signals (6)

LSPO → appliedto → DeepSeek-R1-Distill-Qwen-1.5B

confidence 95% · On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B... LSPO’s 5-seed mean matches or beats a DAPO baseline

LSPO → evaluatedon → DEEPMATH-103K

confidence 95% · On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B... LSPO’s 5-seed mean matches or beats a DAPO baseline

LSPO → uses → LoRA

confidence 95% · LSPO exploits exactly this property: it uses a small, short-lived LoRA adapter purely as a sampling-time mechanism

LSPO → targets → Cliff Prompt

confidence 93% · LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter... re-rolls the cliffs

LSPO → outperforms → DAPO

confidence 92% · LSPO’s 5-seed mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells

LSPO → improves → GRPO

confidence 90% · LSPO is a sampling-time mechanism that wraps an existing policy-gradient RL loop (here GRPO...)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated over n=5 paired seeds per arm at a matched 1000-step reporting horizon, LSPO's 5-seed mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells (15 strict wins and one exact tie), with gains of up to +10.7 points on AIME24/pass@4, +6.7 points on AIME24 and AIME26 at pass@16, and +2.4 points on MATH500/pass@1; averaged over the 16 cells the improvement is +3.8 points.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

39,249 characters extracted from source content.

Expand or collapse full text

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts Ken Ding NVIDIA kennethd@nvidia.com Abstract Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on “cliff” prompts—those on which every sampled rollout in a group fails—the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model’s capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated over n=5n=5 paired seeds per arm at a matched 10001000-step reporting horizon, LSPO’s 55-seed mean matches or beats a DAPO baseline on all 1616 (benchmark, pass@k) cells (1515 strict wins and one exact tie), with gains of up to +10.7+10.7 points on AIME24/pass@4, +6.7+6.7 points on AIME24 and AIME26 at pass@16, and +2.4+2.4 points on MATH500/pass@1; averaged over the 1616 cells the improvement is +3.8+3.8 points. 1 Introduction Reinforcement learning from verifiable rewards (RLVR) has become the dominant recipe for eliciting mathematical reasoning from large language models (Shao et al., 2024; DeepSeek-AI et al., 2025). Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and its scaled-up successor DAPO (Yu et al., 2025) dispense with a learned critic by sampling a group of K rollouts per prompt and normalizing each rollout’s reward against the group mean to form an advantage. This group-relative construction has a structural blind spot. On a cliff prompt—one where all K sampled rollouts are wrong, so the group’s total reward is zero—every rollout carries the same reward, the within-group advantage is identically zero, and the policy-gradient contribution of that prompt vanishes exactly. The model therefore receives no learning signal from precisely the prompts at the frontier of its capability: the problems it cannot yet solve are the ones GRPO is structurally unable to learn from. As training proceeds and the easy prompts are mastered, the unsolved tail accumulates at this frontier, and an increasing share of the batch produces no gradient. Several lines of work attack this regime, and they cleave naturally into two groups. The first targets the cliff directly, recovering the missing signal from ground-truth solutions (Section 2.4). The second group combines RL with a low-rank adapter, though not aimed at the cliff problem specifically. Tina (Wang et al., 2025) trains a LoRA adapter (Hu et al., 2021) by RL and ships the adapted weights as the final model. BRIDGE (Chen et al., 2025) fuses supervised and RL gradients in the base-model update and updates a LoRA teacher using a cooperative-gain meta-objective derived from a bilevel formulation. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that wraps an existing policy-gradient RL loop and recovers gradient on cliff prompts without modifying the loss. On each step, LSPO detects the all-zero-reward cliffs, attaches a small low-rank adapter and runs a brief supervised step on the cliff prompts’ ground-truth solutions (updating the adapter only, base frozen), re-samples the cliffs with the adapter active to restore within-group reward variance, then splices the now-successful rollouts back into the RL batch and backpropagates the policy-gradient loss into the base alone. What distinguishes LSPO from prior adapter-RL methods is three design choices acting together: a sampling-time low-rank scaffold — the adapter is a transient proposal distribution (never the deliverable), kept low-rank and fine-tuned for only a handful of steps, the intent being to keep the importance-sampling correction well-conditioned rather than in the high-variance regime of further-off-policy proposals; strict gradient routing via a two-optimizer split that sends the supervised gradient to the adapter only and the RL gradient to the base only (no bilevel or meta-objective coupling); and splice-and-discard, in which the adapter is reused only as an importance-sampling proposal for the spliced cliff rows and is stripped at checkpoint. The supervised signal therefore reaches the base only indirectly, through adapter-elicited rollouts, and the deliverable is a base-only model that carries no adapter weights. Figure 1 summarises one training step. Our contributions are as follows: • Method. We introduce LSPO, a sampling-time low-rank scaffold that recovers RL gradient on zero-reward cliff prompts via a two-optimizer split (adapter trained by supervision only, base trained by RL only) and a splice-and-discard pipeline with an importance-sampling correction, yielding a base-only deliverable (Section 3). • Headline empirical result. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI et al., 2025), over n=5n=5 paired seeds per arm at a matched 10001000-step reporting horizon, the 55-seed mean of LSPO (per-iteration variant) matches or beats a DAPO baseline on all 1616 (benchmark, pass@k) cells—1515 strict wins and one exact tie—across MATH500 and AIME24/25/26. The largest gains are +10.7+10.7 points on AIME24/pass@4 and +6.7+6.7 points on both AIME24 and AIME26 at pass@16; averaged over the 1616 cells the improvement is +3.8+3.8 points. Notably LSPO improves pass@1 on every benchmark (+1.3+1.3 to +4.7+4.7 points), so the gain is not merely a broadening of the sampled distribution (Section 4). • Cliff-conversion efficacy. We measure the mechanism directly: across the 55 runs the scaffold converts ∼43% 43\% of otherwise-zero-gradient cliff groups into groups carrying usable policy-gradient signal, with a tight per-seed spread (Section 4.3). 2 Related Work and Background 2.1 RL with verifiable rewards: GRPO and the DAPO recipe Reinforcement learning with verifiable rewards (RLVR) trains a policy πθ _θ to generate a solution y to a prompt x and assigns a binary outcome reward r(x,y)∈0,1r(x,y)∈\0,1\ that is 11 iff a verifier accepts the final answer (DeepSeek-AI et al., 2025). Group Relative Policy Optimization (GRPO) (Shao et al., 2024) removes the value network used by PPO (Schulman et al., 2017) by drawing a group of G rollouts y1,…,yG\y_1,…,y_G\ per prompt and normalizing rewards within the group to form the advantage Ai^=(ri−μ)/σ A_i=(r_i-μ)/σ, where μ and σ are the group reward mean and standard deviation. The policy is then updated with the PPO-style clipped surrogate LGRPO=−[min⁡(ρtA^,clip(ρt, 1−εlow, 1+εhigh)A^)],ρt=πθ(at∣st)πold(at∣st),L_GRPO=-\,E\! [ \! ( _t A,\;clip( _t,\,1- _low,\,1+ _high)\, A ) ], _t= _θ(a_t s_t) _old(a_t s_t), (1) where ρt _t is the importance ratio between the current policy and the sampling (behavior) policy. 2.2 The cliff problem The group-normalized advantage in Eq. (1) creates a sharp failure mode. For a given prompt x the outcome falls into one of three cases: (i) all G rollouts succeed, so every reward equals the group mean and all advantages are zero; (i) the rollouts are mixed, yielding positive advantages for the successes and negative for the failures — the standard learning regime; or (i) all G rollouts fail, so the group reward sum is zero, σ is zero, and every within-group advantage is identically zero. Case (i) — a cliff prompt, the “learning cliff” of Zhang et al. (2026) — contributes no gradient to the policy update. These are precisely the prompts at the frontier of the model’s capability: too hard for any of the G samples to solve, yet exactly where a learning signal is most needed. Under standard GRPO the cliff boundary can therefore only advance indirectly, through weight sharing as the model learns on nearby intermediate-difficulty prompts; there is no direct gradient on the cliffs themselves. 2.3 Low-rank adapters Low-Rank Adaptation (LoRA) (Hu et al., 2021) freezes the pretrained weight matrix W0W_0 and learns a low-rank update ΔW=BA W=BA with B∈ℝd×rB ^d× r, A∈ℝr×kA ^r× k and rank r≪min⁡(d,k)r (d,k), so that the adapted forward pass computes (W0+BA)x(W_0+BA)x. Because only A and B are trainable, adaptation is parameter-efficient and the adapter can be attached, detached, or reset cheaply. LSPO exploits exactly this property: it uses a small, short-lived LoRA adapter purely as a sampling-time mechanism, never shipping it. 2.4 Related work Rescuing cliffs with a scaffold. A cliff group can be repaired in place, by synthesising a non-zero advantage through entropy-modulated shaping or a confidence-weighted penalty on wrong answers (Le et al., 2026; Feng et al., 2025); this is far cheaper than LSPO, but on an all-fail group it can only push probability mass away from the observed failures, never toward a success. Methods that instead rescue the cliff—LSPO among them—re-sample the failed prompt under privileged guidance and return the successes to the batch. The move predates RLVR: STaR (Zelikman et al., 2022) recovered failed problems by conditioning on the ground-truth answer, within a generate–filter–finetune lineage that consumes the recovered trajectory as a cross-entropy target rather than an RL sample. The closest of these hints in the context and corrects the importance ratio back to the hint-free prompt (Nath et al., 2025). LSPO’s scaffold is a weight perturbation rather than a prompt, so its ratio compares two parameter settings; and it splices only the verifier-passing rows, leaving the rest of the group on-policy, where that method replaces the group wholesale. The others differ mainly in how the hint is constructed—tiered from abstract concepts to concrete steps (Zhang et al., 2026), taken as a prefix of the ground-truth trace (Liu et al., 2025; Zhang et al., 2025), generated by the policy itself (Chen et al., 2026; Liao et al., 2026), or produced by a separate hinter policy (Xia et al., 2026); all place the scaffold in the prompt, so answer leakage into the sampled trajectory is a live concern, which for LSPO it structurally is not. LatentRevise also fits its scaffold rather than writing it: a soft prefix in input-embedding space, optimised against the gold answer and then discarded (Guo et al., 2026). The prefix is still an input, and refitting it per prompt against that prompt’s own answer is a more direct privileged channel than LSPO’s single low-rank update shared across the whole cliff set. Off-policy rows and privileged supervision. POPO substitutes a variance-bearing group drawn from a prioritised replay buffer under a decoupled importance correction (Mao et al., 2026); unlike replay, LSPO can help on prompts the policy has never solved. ZPPO argues that injecting a teacher’s response into the policy gradient breaks the on-policy assumption, and so places that response in the prompt instead, as an anonymised candidate the student must pick out (Lee et al., 2026). LSPO reaches the same conclusion by another route: the ground-truth solution trains only the adapter, and what enters the base’s gradient is the model’s own rollout, importance-corrected. LUFFY imports off-policy teacher traces into RLVR directly (Yan et al., 2025)—the further-off-policy regime that Section 3 argues against on variance grounds. Where the supervision lands. Hybrid distillation / privileged self-distillation (HDPO) adds a student–teacher divergence term to the RL objective, so that cliff prompts contribute a supervised distillation signal optimised jointly with the policy gradient on the model’s own parameters (Ding, 2026); ReLIFT alternates phases of full-model supervised fine-tuning on hard-prompt ground-truth solutions with phases of RL, supplying cliff signal by overwriting the base weights with expert traces (Ma et al., 2025). In both, a supervised loss is applied to the base’s own parameters—jointly with the policy gradient in the first case, in alternating phases in the second. LSPO applies none: the RL objective is left exactly as it was, the intervention acts only on the proposal distribution, and the supervision terminates in an adapter that is discarded. In BRIDGE neither signal is confined to one weight set: the base takes a fused SFT-plus-RL gradient, and the adapter is updated under a bilevel cooperative-gain objective (Chen et al., 2025). LSPO routes the two to disjoint parameter sets with no meta-objective, maintains two weight sets rather than three, and activates only on cliff prompts. Where Tina’s adapter is trained by RL and shipped as the final model (Wang et al., 2025), LSPO’s receives only supervised gradient and is discarded. 3 Method: LoRA Scaffolded Policy Optimization Figure 1: The LSPO cliff-rescue mechanism. (1) On a cliff prompt p, all K rollouts from the base policy π fail (red); the group reward sum is zero and standard GRPO has no learning signal. (2) A trainable LoRA adapter ϕLoRA _LoRA is composed with the (frozen) base π and fitted via SFT on the dataset’s ground-truth trajectory T. (3) Under the composed policy π+ϕLoRAπ+ _LoRA, some rollouts on the same prompt now pass the verifier (green): the cliff is surpassed. Successful rollouts are spliced into the GRPO batch as positive examples. (4) The GRPO step runs on the same batch with the LoRA branch scale set to 0 (dashed grey), so the RL gradient (purple, dotted) lands on π only; ϕLoRA _LoRA receives no RL gradient and is reset before the next step. LSPO is a sampling-time mechanism that wraps an existing policy-gradient RL loop (here GRPO (Shao et al., 2024) with a DAPO-style recipe (Yu et al., 2025)) and recovers gradient on cliff prompts—prompts on which every sampled rollout in the group scores zero reward, so the group-relative advantage, and hence the policy gradient on that prompt, is identically zero. The core idea is to attach a small low-rank adapter (Hu et al., 2021) as a transient scaffold: it is supervised-fine-tuned on the cliff prompts’ own ground-truth solutions, used only to re-sample those prompts, and then discarded. Successful adapter-elicited rollouts are spliced back into the RL batch so that the within-group reward variance—and the RL gradient—is restored, while the gradient that actually updates the deliverable flows into the base model alone. 3.1 Algorithm Figure 1 illustrates the mechanism and Algorithm 1 states the per-step loop. Each RL step: (A) sample K completions per prompt from the base and flag as cliffs those whose group scores zero total reward; (B) attach a LoRA adapter and run a brief supervised step on the cliff prompts’ ground-truth solutions, base frozen; (C) re-sample the cliff prompts with the adapter on; (D) swap each successful re-rollout into its failed counterpart’s slot, leaving every other row untouched; (E) apply the RL update to the base alone and discard the adapter. The swap in (D) is deliberately minimal: a failed re-rollout carries no signal the group does not already have from its base counterpart, so admitting it would add off-policy mass without adding information. Advantages are recomputed on the spliced batch, and each spliced row’s base-plus-adapter probability is retained for the importance correction (Section 3.3). Algorithm 1 LoRA Scaffolded Policy Optimization (per-step loop) 0: Base policy πθ _θ, prompt set X, ground truth y∗\y^*\, reward R, rollouts per prompt K, clip bounds (εlow,εhigh)( _low, _high), RL optimizer OptRLOpt_RL (updates θ), SFT optimizer OptSFTOpt_SFT (updates adapter ϕφ) 1: for each RL training step do 2: Sample batch B⊂B ; for each x∈Bx∈ B draw K rollouts y(k)∼πθ(⋅∣x)y^(k) _θ(· x) and record per-row sampler density πsamp←πθ _samp← _θ 3: Score every rollout with R; ←x∈B:∑kR(x,y(k))=0C←\x∈ B: _kR(x,y^(k))=0\ // cliffs: zero RL gradient 4: if ≠∅C≠ then 5: Attach low-rank adapter ϕφ to πθ _θ (base frozen); ϕ←OptSFTφ _SFT on (x,y∗):x∈\(x,y^*):x \ // supervised gradient to ϕφ only 6: Re-sample C with adapter on: y~∼πθ+ϕ(⋅∣x) y _θ+φ(· x); ←(x,y~):R(x,y~)=1S←\(x, y):R(x, y)=1\ 7: Splice S into B in place of the failed cliff rollouts; for each (x,y~)∈(x, y) overwrite πsamp←πθ+ϕ _samp← _θ+φ // IS denominator switches to base+adapter 8: end if 9: Compute group-normalized advantages A^y A_y on the (possibly spliced) batch B 10: Per-token IS ratio: ρt(θ)=πθ(yt∣x,y<t)πsamp(yt∣x,y<t) _t(θ)\;=\; _θ(y_t x,y_<t) _samp(y_t x,y_<t) // πsamp _samp equals πθ _θ for on-policy rows, πθ+ϕ _θ+φ for spliced cliffs 11: θ←OptRLθ _RL on the clipped surrogate (RL gradient to base only; adapter receives none): ℒGRPO=−1T∑(x,y)∈B∑tmin⁡(ρtA^y,clip(ρt, 1−εlow, 1+εhigh)A^y)L_GRPO\,=\,- 1T _(x,y)∈ B _t \! ( _t\, A_y,\,clip( _t,\,1- _low,\,1+ _high)\, A_y ) where T=∑(x,y)∈B|y|T= _(x,y)∈ B|y| is the total tokens in the batch 12: Discard / reset adapter ϕφ 13: end for 14: return base model πθ _θ // adapter never shipped 3.2 Gradient routing The defining architectural choice of LSPO is that gradient routing is strictly disjoint. We maintain two optimizers. The supervised optimizer OptSFTOpt_SFT updates the low-rank adapter only, on the cross-entropy of the cliff ground-truth solutions (stage B). The RL optimizer OptRLOpt_RL updates the base only, on the GRPO surrogate over the spliced batch (stage E). No parameter is touched by both objectives: the base never receives supervised gradient, and the adapter never receives RL gradient. The supervised signal reaches the base only indirectly—through the adapter-elicited rollouts that are spliced into the RL batch and then learned by the ordinary policy gradient. Because a fresh adapter is fitted at every step, that signal always measures the gap from the current base to the cliff solutions, with no adapter state carried across steps. 3.3 Importance-sampling correction Spliced rows are off-policy: they were drawn from πμ:=πθ+ϕ _μ:= _θ+φ, not from the base πθ _θ whose parameters the update modifies. The correction is a change of measure. The quantity we want is the base’s policy gradient, and y∼πθ[A(y)∇θlog⁡πθ(y)]=y∼πμ[πθ(y)πμ(y)A(y)∇θlog⁡πθ(y)],E_y _θ\! [A(y)\, _θ _θ(y) ]=\;E_y _μ\! [ _θ(y) _μ(y)\,A(y)\, _θ _θ(y) ], (2) so each spliced row must carry the weight πθ(y)/πμ(y) _θ(y)/ _μ(y), which factorises per token into ∏tπθ(yt∣y<t)/πμ(yt∣y<t) _t _θ(y_t y_<t)/ _μ(y_t y_<t). The implementation instead uses the corresponding per-token ratio in the clipped surrogate, ρt=πθ(yt∣y<t)/πprev(yt∣y<t) _t= _θ(y_t y_<t)/ _prev(y_t y_<t) (Schulman et al., 2017; Shao et al., 2024). For ordinary rows πprev _prev is the base; for spliced rows we set πprev=πμ _prev= _μ, using the base-plus-adapter log-probabilities recorded at stage (D). No new estimator is introduced — the denominator is simply the distribution that generated the row. Keeping the adapter small keeps that ratio well-conditioned. This is a variance motivation rather than an exactness guarantee. Because the implementation selects verifier-passing proposal rows and applies clipped per-token ratios rather than an unclipped sequence-product weight, we do not claim that its update is an unbiased sequence-level policy-gradient estimator or provide a convergence guarantee; formal characterization is deferred to future work. 4 Experiments 4.1 Setup We evaluate LSPO on mathematical reasoning. The base model is DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI et al., 2025), a distilled reasoning model. Training uses DeepMath-103K (He et al., 2025), a public collection of competition-style mathematics problems with verifiable final answers (HuggingFace zwhe99/DeepMath-103K). We validate on four held-out benchmarks: MATH500 (Hendrycks et al., 2021; Lightman et al., 2024) and the AIME 2024, 2025, and 2026 competition sets (AIME24/25/26),111We use the publicly released problem sets from the Mathematical Association of America’s American Invitational Mathematics Examination for 2024, 2025, and 2026. Problems and official answer keys are available via the MAA archive and community wikis such as https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions. reporting best pass@k for k∈1,4,8,16k∈\1,4,8,16\. Validation runs every 10 training steps. pass@k is computed by the first-k-slice estimator (whether any of the first k of the 1616 validation rollouts passes the verifier); it is unbiased for pass@k under exchangeability of the rollouts, though higher-variance than the combinatorial estimator of Chen et al. (2021) which averages over all size-k subsets. The estimator is applied identically to both arms. Seeds and reporting horizon. Both arms are trained with the same seed set 42,43,44,45,46\42,43,44,45,46\, and all analysis is paired seed-by-seed. We fix a 1000-step reporting horizon: runs are launched with a 1500-step budget, and every peak, comparison, and aggregate reported in this paper is computed over validation events at step ≤1000≤1000, identically for LSPO and DAPO. We do not characterize behavior beyond this horizon in this version. All training is run on 8×8×H100 GPUs. The base RL algorithm is GRPO (Shao et al., 2024) configured with the DAPO recipe (Yu et al., 2025): Clip-Higher with asymmetric ratio clipping (εlow=0.20 _low=0.20, εhigh=0.28 _high=0.28), token-level policy-gradient loss, and overlong-reward shaping (a soft length penalty that engages past 7168 generated tokens up to the 8192-token generation limit). Dynamic sampling is disabled. Each step samples 32 prompts with 16 generations per prompt (512 rollouts/step). The base model is updated with AdamW (Loshchilov and Hutter, 2019) at learning rate 1e−61e-6; we apply no reference-policy KL penalty. The LSPO scaffold uses a LoRA (Hu et al., 2021) adapter of rank 16 (α=64α=64) fine-tuned on cliff ground-truth solutions for sft_steps_per_new_cliff=4=4 steps with a separate AdamW optimizer at learning rate 2.5e−52.5e-5. Cliffs are groups whose total reward is zero; only successful adapter-elicited rollouts replace their failed counterparts (partial swap), and spliced rows reuse the base-plus-adapter log-probabilities as the importance-sampling denominator. The adapter receives supervised gradient only and the base receives RL gradient only, per Section 3. Unless stated otherwise, LSPO denotes the per-iteration variant (a fresh adapter each step). Our baseline is identical in every respect—data, recipe, optimizer, hardware, and step budget—with the LSPO scaffold disabled, i.e. plain DAPO-style GRPO. This isolates the contribution of the scaffold. 4.2 Main result: peak-vs-peak comparison Table 1 reports the full 16-cell peak comparison at the matched reporting horizon (step ≤1000≤1000) with n=5n=5 paired seeds for both arms. For each (run, benchmark, k) we take the peak value over that run’s validation trace, then average across the five seeds. LSPO’s mean matches or beats DAPO’s on every cell, with 15 strict wins and one exact tie (AIME25/pass@8: 46.0046.00 vs. 46.0046.00; at integer counts both arms recover 6969 of 150150 seed-problem pairs). The advantage is largest on AIME24 (mean +4.7+4.7 to +10.7+10.7 points across k, peaking at +10.66+10.66 points at pass@4) and on AIME26/pass@16 (+6.67+6.67 points). MATH500 gains are smaller in absolute terms (+0.4+0.4 to +2.4+2.4 points) because both arms approach the benchmark ceiling. Notably, LSPO improves pass@1 on every benchmark (+1.3+1.3 to +4.7+4.7 points): the improvement is not merely a broadening of the sampled distribution but a shift in single-sample accuracy. Averaged over all 1616 cells, LSPO scores 59.9059.90 against DAPO’s 56.0856.08, a +3.82+3.82-point improvement. Because peaks are taken per cell independently, the peak steps differ across cells, so Table 1 reports best-achievable-per-metric during training rather than the performance of any single checkpoint. Both arms are measured under identical protocol, so the paired comparison is unaffected. Benchmark pass@k LSPO mean LSPO range DAPO mean DAPO range Δ (pts) (5 seeds) (5 seeds) MATH500 pass@1 86.28 84.80–87.20 83.88 83.20–84.60 +2.40+2.40 MATH500 pass@4 94.80 94.60–95.20 93.36 93.00–93.60 +1.44+1.44 MATH500 pass@8 96.44 96.00–96.60 95.56 95.40–96.00 +0.88+0.88 MATH500 pass@16 97.52 97.20–97.60 97.08 96.80–97.20 +0.44+0.44 AIME24 pass@1 37.33 33.33–40.00 32.66 30.00–33.33 +4.67+4.67 AIME24 pass@4 59.33 53.33–63.33 48.67 46.67–53.33 +10.66+10.66 AIME24 pass@8 65.33 63.33–70.00 56.67 56.67–56.67 +8.66+8.66 AIME24 pass@16 72.00 70.00–73.33 65.33 63.33–70.00 +6.67+6.67 AIME25 pass@1 28.67 26.67–30.00 27.34 26.67–30.00 +1.33+1.33 AIME25 pass@4 42.00 40.00–43.33 38.00 36.67–43.33 +4.00+4.00 AIME25 pass@8 46.00 43.33–50.00 46.00 43.33–46.67 +0.00+0.00 AIME25 pass@16 54.00 50.00–56.67 51.33 50.00–53.33 +2.67+2.67 AIME26 pass@1 30.00 26.67–33.33 25.33 23.33–26.67 +4.67+4.67 AIME26 pass@4 42.66 40.00–43.33 40.00 36.67–43.33 +2.66+2.66 AIME26 pass@8 49.33 46.67–53.33 46.00 43.33–50.00 +3.33+3.33 AIME26 pass@16 56.67 53.33–60.00 50.00 46.67–53.33 +6.67+6.67 Mean over 16 cells 59.90 — 56.08 — +3.82+3.82 Table 1: Peak pass@k at the matched 10001000-step reporting horizon, averaged across n=5n=5 paired seeds per arm, on DeepSeek-R1-Distill-Qwen-1.5B trained with DeepMath-103K (8×8×H100). LSPO is the per-iteration variant; DAPO is the identical recipe with the scaffold disabled. LSPO’s mean matches or beats DAPO’s on all 1616 cells (1515 strict wins ++ 11 exact tie at AIME25/pass@8). Δ entries ≥+4≥+4 points are highlighted in the final column. Peak steps vary by cell. AIME sets have N=30N=30 problems (one problem ≈3.33≈ 3.33 points), so single-cell deltas are coarse; the pattern across cells is the reliable signal. 4.3 Cliff-conversion efficacy Table 1 measures the outcome; this section measures the mechanism. Across the 55 LSPO runs at step ≤1000≤1000, 10,86410,864 cliff groups were detected in total. The scaffold — four iterations of LoRA fitting on the cliff ground-truth solutions followed by re-sampling — surpassed approximately 43%43\% of them, i.e. produced at least one verifier-passing rollout on 4,6804,680 groups that were originally 0-of-K failures and therefore carried exactly zero policy gradient. The per-seed surpass rate is tight (41.341.3–45.5%45.5\%, standard deviation 1.5%1.5\%). Two details sharpen the interpretation. First, the per-rollout post-scaffold pass rate is only 5.8%5.8\% (cliff-weighted mean across seeds), so the group-level 43%43\% predominantly reflects one of the K=16K=16 re-sampled rollouts succeeding rather than confident solving — which is all GRPO requires, since a single success restores within-group reward variance. Second, the per-step surpass rate is bimodal (31%31\% of cliff-bearing steps have rate 0.00.0; 21%21\% have rate 1.01.0), so the aggregate 43%43\% is a mean of extremes rather than a typical value. By construction the DAPO baseline converts 0%0\% of its cliffs: the leave-one-out advantage zeroes those groups’ gradients outright. 5 Limitations Scope. We evaluate one model (DeepSeek-R1-Distill-Qwen-1.5B), one dataset (DeepMath-103K), one recipe family (GRPO with the DAPO configuration), and one scaffold configuration (per-iteration variant, LoRA rank 1616, four SFT steps). The scaffold’s behavior plausibly depends on adapter rank, SFT step count, splice variant, and the size of the dataset’s answer space — a small-integer answer space makes verifier-accepted guessing more likely than a large one — and we do not establish how the results transfer across these axes. Ground-truth requirement. LSPO requires ground-truth solutions (not merely final answers) for the cliff prompts, since the scaffold is fit by supervised learning on those derivations. This is available in DeepMath-103K but restricts applicability to datasets that ship worked solutions. 6 Conclusion We introduced LSPO (LoRA Scaffolded Policy Optimization), a sampling-time mechanism that recovers reinforcement-learning gradient on zero-reward “cliff” prompts—those frontier prompts where every sampled rollout fails, the within-group advantage is identically zero, and standard GRPO therefore provides no learning signal. LSPO attaches a small low-rank adapter, fine-tunes it only on the cliff prompts’ ground-truth solutions, re-samples the cliffs with the adapter active to restore within-group reward variance, and splices the resulting successes back into the RL batch. Two design choices keep the deliverable a clean base-only model: strictly disjoint gradient routing via a two-optimizer split (the supervised signal reaches the adapter only and the RL gradient reaches the base only, reusing the standard policy-ratio term as the importance-sampling correction for the spliced rows), and a splice-and-discard policy that strips the adapter at checkpoint so it is never shipped. Over n=5n=5 paired seeds at a matched 10001000-step reporting horizon, LSPO’s mean matches or beats a DAPO baseline on all 1616 peak pass@k cells (1515 strict wins and one exact tie), with gains up to +10.7+10.7 points on AIME24/pass@4 and +3.8+3.8 points averaged across the 1616 cells, and with improvements on pass@1 for every benchmark. The mechanism is directly measurable: roughly 43%43\% of otherwise-zero-gradient cliff groups are converted into groups that carry usable policy-gradient signal. We view LSPO as evidence that the cliff problem can be addressed at sampling time, with the supervision localized in transient low-rank parameters rather than written irreversibly into the base. LLM Usage Statement The research idea underlying LSPO—using a transient, sampling-time low-rank scaffold to recover reinforcement-learning gradient on zero-reward cliff prompts—originated with the author. However, an AI language model (Claude, Anthropic) was used extensively throughout this project in ways that go beyond minor writing assistance. Specifically: (1) the paper text, including the related-work survey, the method exposition, and the mechanism-analysis and discussion sections, was substantially drafted and edited with LLM assistance; (2) the LLM was used as a research collaborator to stress-test explanations for the observed behavior, to assemble and check the related-work survey, and to position LSPO against prior methods; a second model (OpenAI Codex, GPT) was used as an independent verifier, recomputing every reported number from the raw training logs and auditing citations; and (3) the importance-sampling correction described in Section 3.3 — reusing the standard PPO/GRPO policy-ratio term with the base-plus-adapter sampling log-probabilities as the ratio’s denominator on spliced cliff rows — was proposed by the LLM during the method-design phase and adopted by the author. All experimental results (training runs, metric measurements, and rollout dumps) were produced by the author without LLM involvement. References J. C. Chen, B. X. Peng, P. K. Choubey, K. Huang, J. Zhang, M. Bansal, and C. Wu (2026) Nudging the boundaries of LLM reasoning. In International Conference on Learning Representations (ICLR), Note: arXiv:2509.25666 External Links: Link Cited by: §2.4. L. Chen, X. Han, L. Shen, J. Bai, and K. Wong (2025) Beyond two-stage training: cooperative SFT and RL for LLM reasoning. arXiv preprint arXiv:2509.06948. External Links: Link Cited by: §1, §2.4. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. External Links: Link Cited by: §4.1. DeepSeek-AI, D. Guo, D. Yang, et al. (2025) DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948. External Links: Link Cited by: 2nd item, §1, §2.1, §4.1. K. Ding (2026) HDPO: hybrid distillation policy optimization via privileged self-distillation. arXiv preprint arXiv:2603.23871. External Links: Link Cited by: §2.4. Y. Feng, P. Jain, A. Hartshorn, Y. Duan, and J. Kempe (2025) Don’t waste mistakes: leveraging negative RL-groups via confidence reweighting. arXiv preprint arXiv:2510.08696. External Links: Link Cited by: §2.4. Y. Guo, X. Han, Q. Jia, G. Zhai, and J. Bai (2026) LatentRevise: learning from zero-hit reasoning. arXiv preprint arXiv:2606.29938. External Links: Link Cited by: §2.4. Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu (2025) DeepMath-103K: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. External Links: Link Cited by: §4.1. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the MATH dataset. In Neural Information Processing Systems, External Links: Link Cited by: §4.1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. External Links: Link Cited by: §1, §2.3, §3, §4.1. T. V. Le, M. Jeon, K. Vu, V. Lai, and E. Yang (2026) No prompt left behind: exploiting zero-variance prompts in LLM reinforcement learning via entropy-guided advantage shaping. In International Conference on Learning Representations (ICLR), Note: arXiv:2509.21880 External Links: Link Cited by: §2.4. B. Lee, X. Lu, S. Diao, M. Kang, S. Muralidharan, K. Sapra, A. Tao, P. Molchanov, Y. Choi, Y. F. Wang, and R. Hachiuma (2026) Zone of proximal policy optimization: teacher in prompts, not gradients. arXiv preprint arXiv:2606.18216. External Links: Link Cited by: §2.4. B. Liao, H. Dong, X. Xu, C. Monz, and J. Bian (2026) Self-hinting language models enhance reinforcement learning. arXiv preprint arXiv:2602.03143. External Links: Link Cited by: §2.4. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2024) Let’s verify step by step. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Z. Liu, C. Gong, X. Fu, Y. Liu, R. Chen, S. Hu, S. Zhang, R. Liu, Q. Zhang, and D. Tu (2025) GHPO: adaptive guidance for stable and efficient LLM reinforcement learning. arXiv preprint arXiv:2507.10628. External Links: Link Cited by: §2.4. I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. L. Ma, H. Liang, M. Qiang, L. Tang, X. Ma, Z. H. Wong, J. Niu, C. Shen, R. He, Y. Li, B. Cui, and W. Zhang (2025) Learning what reinforcement learning can’t: interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527. External Links: Link Cited by: §2.4. Y. Mao, Y. Qu, Q. Wang, H. Zou, and X. Ji (2026) RLVR without ineffective samples: group prioritized off-policy optimization for LLM reasoning. arXiv preprint arXiv:2606.01281. External Links: Link Cited by: §2.4. V. Nath, E. Lau, A. Gunjal, M. Sharma, N. Baharte, and S. Hendryx (2025) Adaptive guidance accelerates reinforcement learning of reasoning models. arXiv preprint arXiv:2506.13923. External Links: Link Cited by: §2.4. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: Link Cited by: §2.1, §3.3. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y.K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §1, §2.1, §3.3, §3, §4.1. S. Wang, J. Asilis, Ö. F. Akgül, E. B. Bilgin, O. Liu, and W. Neiswanger (2025) Tina: tiny reasoning models via LoRA. arXiv preprint arXiv:2504.15777. External Links: Link Cited by: §1, §2.4. Y. Xia, C. Xu, Z. Yao, J. McAuley, and Y. He (2026) Learning to hint for reinforcement learning. arXiv preprint arXiv:2604.00698. External Links: Link Cited by: §2.4. J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang (2025) Learning to reason under off-policy guidance. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2504.14945 External Links: Link Cited by: §2.4. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, et al. (2025) DAPO: an open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. External Links: Link Cited by: §1, §3, §4.1. E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2022) STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2203.14465 External Links: Link Cited by: §2.4. X. Zhang, S. Wu, Y. Zhu, H. Tan, S. Yu, Z. He, and J. Jia (2026) Scaf-GRPO: scaffolded group relative policy optimization for enhancing LLM reasoning. In International Conference on Learning Representations (ICLR), Note: arXiv:2510.19807 External Links: Link Cited by: §2.2, §2.4. X. Zhang, Z. Huang, Y. Li, C. Ni, J. Chen, and S. Oymak (2025) BREAD: branched rollouts from expert anchors bridge SFT & RL for reasoning. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2506.17211 External Links: Link Cited by: §2.4.

Instant research discovery