Paper deep dive

Decoupling Exploration from Optimization in RLVR

Saif Punjwani, Micah Goldblum

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 61

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/13/2026, 3:25:47 AM

Summary

The paper introduces Exploration-Distillation (ExpDis), a framework that decouples exploration from optimization in Reinforcement Learning with Verifiable Rewards (RLVR). By training separate explorer policies with novelty bonuses and distilling their filtered, correct trajectories into a student policy, ExpDis avoids the degradation of prior knowledge often caused by aggressive exploration. The method outperforms baselines like DAPO in mathematical reasoning benchmarks and improves pass@k scaling, indicating greater diversity in correct solutions.

Entities (8)

ExpDis · method · 98%RLVR · technique · 95%DAPO · baseline · 92%Explorer Policy · model-component · 90%Qwen3-4B · model · 90%Student Policy · model-component · 90%Novelty Bonus · component · 88%RND · algorithm · 85%

Relation Signals (6)

ExpDis → decouples → exploration

confidence 95% · we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis).

ExpDis → outperforms → DAPO

confidence 95% · ExpDis outperforms DAPO at the same wall-clock budget.

ExpDis → employs → Distillation

confidence 92% · filter their trajectories for correctness and quality, and distill them into a separate student policy.

ExpDis → evaluatedon → Qwen3-4B

confidence 90% · We evaluate ExpDis on mathematical reasoning benchmarks with Qwen3-1.7B, Qwen3-4B... as base models.

ExpDis → uses → Novelty Bonus

confidence 90% · We train one or more explorer policies with a novelty bonus in the reward

RND → istypeof → Novelty Bonus

confidence 85% · we choose a Random Network Distillation (RND) bonus

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,912 characters extracted from source content.

Expand or collapse full text

Decoupling Exploration from Optimization in RLVR Saif Punjwani 1,∗ Micah Goldblum 1,2,∗ Columbia University Abstract Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model’s knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@kscaling, indicating that ExpDis produces models that generate more diverse correct solutions. 3 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become the dominant recipe for im- proving the reasoning abilities of language models [19,15,33,26,44]. RLVR samples solutions to problems, scores each solution with a programmatic verifier, and shifts probability toward those that pass. By iteratively sampling new solutions and reinforcing successful ones, RLVR can in principle discover and then reinforce new solutions that were absent from the model’s training data. In practice, however, it can only reinforce what the model samples, so discovery depends on exploration. A natural way to boost exploration is to add an explicit novelty bonus to the reward, following a long line of intrinsic-motivation methods in RL [30,2,3,10]. When the model explores new solution strategies, some of them may succeed but often most of them fail, leading to a temporarily degraded model as we now are left with a model that mostly generates failed solutions. Though our initial policy is a pretrained LLM with vast knowledge including virtually all of Wikipedia, the reward supervises only a tiny slice of its knowledge. This means an LLM could forget most of Wikipedia and still achieve high reward. The reward therefore cannot detect damage that exploration causes outside that slice, let alone repair such degradations. We argue that this failure is not a property of novelty as a signal, but of where the novelty bonus is applied. We decouple exploration and optimization by giving them separate model instances, both trained from the same base checkpoint. An explorer policy explores aggressively under a novelty bonus whose weightλsets how strongly it is pushed toward unfamiliar solutions relative to correctness. A separate student policy never sees a novelty reward. It learns from the explorer only through filtered trajectories, and is then trained with standard RLVR. The two models communicate through filtered trajectories rather than parameters, so the student policy absorbs the explorer’s discoveries without inheriting its degradations induced by exploration. We call this framework Exploration-Distillation. * Corresponding authors: saif.punjwani@columbia.edu, micah.g@columbia.edu. 1 Department of Computer Science, Columbia University. 2 Department of Electrical Engineering, Columbia University. 3 We release all code and checkpoints. arXiv:2610.10536v1 [cs.LG] 7 Oct 2026 Base model Explorer Rejection sampling Student RL (correctness + λ⋅ novelty) Distill + RL (correctness only) correct region correctincorrectfiltered out π 0 μ π 1 1248163264 k (samples per problem) 60 70 80 90 mean pass@ k (%) Base DAPO DAPO (4× steps) DAPO (+novelty) ExpDis Qwen3-1.7BQwen3-4BMinistral-3B −2 −1 0 +1 +2 Prior capabilities Δ (%) μ π 1 μ π 1 μ π 1 ExpDis (explorer) (a) Math reasoning performance (Qwen3-4B) (b) Knowledge not supervised by the reward can degrade Figure 1: Decoupling exploration from optimization improves math accuracy without degrading prior capabilities. Left: Rather than coupling exploration and optimization in a single RLVR run, ExpDis assigns these roles to separate policies. We first train an explorerμwith a novelty bonus in its reward, then distill a diverse pool of its reasoning traces to cold start a student, and finally train the student with standard RLVR using only a correctness reward. As depicted in the figure, this allows us to explore a diverse set of reasoning strategies and preserve this diversity in our student policy. (a) ExpDis outperforms DAPO in both average accuracy and pass@k, even when DAPO is trained for4×as long (Qwen3-4B, mean pass@kover five math benchmarks). (b) Adding a novelty bonus (RND) directly to the DAPO reward degrades performance on prior capabilities relative to the base model. Because the reward supervises only a narrow slice of the model’s knowledge (here, mathematical reasoning), achieving high reward does not prevent general capability degradation. By decoupling the explorer from the student policy, we avoid this degradation. Decoupling makes exploration safe to scale aggressively, since the student policy never receives the novelty bonus and the filter keeps failed trajectories out of its training data. We can scale breadth by running several explorers in parallel and distilling their pooled trajectories into one student policy. We can also scale depth by repeating the pipeline for multiple rounds that each start from the previous student. We scale both of these axes independently and find that their composition enables sustained performance gains even when all experiments are wall-clock matched. We evaluate ExpDis on mathematical reasoning benchmarks with Qwen3-1.7B, Qwen3-4B, and Ministral-3-3B-Instruct-2512 as base models. At the same amount of compute, ExpDis outperforms DAPO in both pass@1 and pass@64. Furthermore, spreading this fixed budget over multiple parallel explorers and several rounds of the ExpDis procedure yields further gains. Adding a novelty bonus directly to DAPO, in contrast, increases diversity but degrades performance on general knowledge benchmarks. In ExpDis, the explorer can be pushed to far higher diversity without harming the student, which improves in both diversity and accuracy and avoids degradation. In principle, ExpDis is compatible with any exploration mechanism, since the student learns only from the explorer’s filtered trajectories. Since the explorer is never deployed, it can also use mechanisms too aggressive to apply to the deployed model directly. Exploration methods that would destabilize standard RLVR may therefore become viable under ExpDis. 2 Related Work Exploration in LLM reasoning. A number of works ask whether standard RLVR algorithms discover new solutions or only sharpen a model’s pre-RL generative distribution. Shao et al.[33] observe that RL improves majority-vote accuracy but not pass@k, and Chen et al.[5]find that base models overtake their RLVR-trained counterparts at largek, suggesting that RLVR narrows the set of problems a model can solve. Others find that RL can expand this set through prolonged 2 training [25], or that the model learns new skills by composing existing ones [45]. A related line of work studies entropy collapse: policy entropy falls sharply early in training [7], and the gains from RLVR rely on a small set of high-entropy tokens to allow sampling diverse solutions [40]. Many methods counteract this entropy collapse during training. For example, DAPO [44] raises the upper clipping bound so that low-probability tokens with positive advantage can grow faster, whereas He et al.[16]directly regulate the policy’s entropy. Other approaches add exploration or novelty bonuses to the reward [36,11,38,21]. These methods draw on a long line of exploration bonuses in classical RL, including count-based bonuses [2], intrinsic curiosity [30], and random network distillation [3]. ExpDis is compatible with a broad spectrum of exploration bonuses, and we try plugging several example bonuses into our framework, observing benefits of exploration in all cases. Reasoning trace distillation.A classical precursor that trains a policy to imitate a separate process that finds better solutions is expert iteration [1], in which a tree-search expert generates improved moves and a neural apprentice learns to imitate them. Today, distilling reasoning traces from a stronger model is standard practice: DeepSeek-R1 [15] is distilled into smaller and faster open models, and several works study how to curate reasoning traces to instill reasoning in fine-tuned models [43,13]. When no stronger model is available, a model can instead learn from its own filtered samples, as in STaR [46], ReST [14], and ReST EM [35]. DeepSeek-R1 fine-tunes its base model on correct traces rejection-sampled from an RL checkpoint before running RL again [15], and multi-teacher on-policy distillation methods [28] distill separately trained specialist models into one model. We similarly distill trajectories from one model into another, but we train our explorer policies from the same base model as our student with the distinct goal of exploration. Decoupling exploration in RL.Separating exploration from exploitation has a long history in RL: E 3 [18] keeps separate exploration and exploitation policies and explicitly switches between them, ̧Sim ̧sek and Barto[34]explore with a policy rewarded for improving a separate task value function that never sees this reward, and Go-Explore [9] first finds high-reward trajectories and then turns them into a robust policy via imitation learning. In reinforcement learning for LLMs, R 2 PO [39] trains a residual logit head to diversify within-group outcomes, uses it to sample rollouts for the policy’s GRPO updates, and discards it at inference. In concurrent work, ICE [8] explores with instruction-conditioned prompts and self-distills correct rollouts into the unconditioned policy. In both, the exploring policy and the student share weights. We instead train the explorer as a separate network and pass only filtered trajectories to the student. This design choice insulates the student from exploration pressures and thus avoids degrading the student. 3 Method Exploration-Distillation trains an explorer policyμwith a novelty bonus to generate diverse trajecto- ries, filters them for quality and correctness, and distills the resulting trajectories into a student policy π 1 . We present a single iteration below and describe how we scale it in Section 3.1 and Section 3.2. We study the importance of each stage in Appendix E. Explorer policy training. We initialize an explorer policyμfrom the base policyπ 0 with DAPO [44] and augment its reward with a novelty reward r novelty ≥ 0. r explorer (τ ) = r correct (τ ) + λ· r novelty (τ )· 1[r correct (τ ) = +1] + r overlong (τ ),(1) wherer correct ∈−1, +1andλ≥ 0. As introduced in DAPO,r overlong ∈ [−1, 0]is a soft penalty on completions that approach the length budget. Apart from the novelty term, the explorer is identical to our DAPO baseline. This reward formulation ensures that every correct rollout receives a higher reward than any incorrect one for any value ofλ. We find that applying the bonus to all rollouts leads to training instability as a model can achieve increased reward simply by generating novel garbage. Rejection sampling. We now need a method for collecting trajectories generated by the explorer policy to be distilled into the student. We remove any incorrect trajectories, since the point of this rejection sampling step is to insulate the student from any degradations caused to the explorer by novelty pressure. A naive choice would be to take all correct explorer trajectories or a random subset and hand them off, but we find that filtering not only for correctness but also for quality strongly 3 RL (correctness + λ r ⋅ novelty) Distill + RL (correctness only) Round 1 (shard 1 / 4 ) Round 2 (shard 2 / 4 ) Round 3 (shard 3 / 4 ) Round 4 (shard 4 / 4 ) correct region correctincorrectfiltered out μ Explorers ×3 from π 0 , λ 1 = 0.75 Rejection sampling π 1 Student π 1 initializes round 2 μ Explorers ×3 from π 1 , λ 2 = 0.50 Rejection sampling π 2 Student π 2 initializes round 3 μ Explorers ×3 from π 2 , λ 3 = 0.35 Rejection sampling π 3 Student π 3 initializes round 4 μ Explorers ×3 from π 3 , λ 4 = 0.25 Rejection sampling π 4 Student Figure 2: ExpDis scales across parallel explorers and iterations. Within a round, several explorers are trained in parallel with a novelty bonus, and their traces are pooled to distill a single student. Across rounds, that student initializes both the next round’s explorers and the next round’s student, so every round starts from a stronger model. Each round trains on its own shard of the training prompts, and the novelty weight anneals fromλ 1 =0.75 to λ 4 =0.25. We study how to allocate a fixed RL budget across explorers and rounds in Section 5. improves performance. A trajectory passes the filter only if (1) its final answer is correct, (2) it terminates within a fixed token budget, and (3) it contains no repetition loops. After this filtering, we keep only a single shortest resulting trajectory per problem. Problems that the explorer solves correctly many times therefore do not dominate the student’s training data. We cap the resulting set at 500 trajectories per round. In Appendix E, we ablate our quality filtering procedure. Student policy training.We distill the base policy on the resulting trajectories simply by perform- ing supervised finetuning (SFT) on those trajectories with the standard cross-entropy loss. Following SFT, we train the distilled model with DAPO and the standard correctness-only reward (i.e., no nov- elty bonus) to obtain the student policy π 1 , which inherits none of the degradation from exploration pressure. The student both receives the benefit of exploration via distillation, which is off-policy, and also the benefit of on-policy training during its DAPO stage. 3.1 Parallel Explorer Policies Rather than train a single explorer policyμ, we can trainKindependent explorer policies and pool their trajectories before filtering and performing distillation. Compared to the single-explorer variant, training parallel explorers can increase the diversity of trajectories we curate for the student. This diversity increases coverage over solutions and boosts our probability of discovering successful ones. We set the novelty bonus weightλidentically for all peer explorer policies, as we find little benefit from varying them. In our experiments, we divide compute among explorers, requiring no additional wall-clock time over a single explorer. The more compute a training run uses, the more parallel explorers we expect to be optimal. 3.2 Iterative Exploration-Distillation We can scale the above method by performing many rounds of alternating exploration and distillation. After producing a student policy, we can use this student as a strong initialization for both the next explorer policy and subsequent student policy. Formally: to train studentπ t , we first train one or more explorersμfromπ t−1 , filter the trajectories, distill those trajectories ontoπ t−1 via SFT and then perform standard RLVR on the distilled checkpoint without an exploration bonus to produceπ t . 4 We partition the training prompts intoRnon-overlapping shardsD 1 ,...,D R of equal size. As shown in Figure 2, roundrdraws its training prompts fromD r . Hence, each round pairs a stronger initialization with prompts it has not yet trained on. Annealing novelty bonus.Because exploration is split across several rounds, we can set its strength per round through the novelty-bonus weightλ, which we anneal on a fixed schedule from0.75to 0.25over four rounds. This schedule follows the common practice of decreasing exploration pressure as training progresses [37]. Early rounds, where correct trajectories are still undiscovered, get strong novelty pressure whereas later rounds get lower novelty pressure to consolidate what has been found. 4 Experimental Setup Base models.We evaluate ExpDis on Qwen3-1.7B, Qwen3-4B [42], and Ministral-3-3B-Instruct- 2512 [24]. The Qwen3 models are distilled from larger RL-trained Qwen3 models rather than trained with RL directly [42], whereas Ministral-3-3B-Instruct is instruction-tuned and undergoes online preference optimization [24]. We describe all sampling parameters in Appendix A. Novelty bonus. In principle, ExpDis is compatible with any novelty bonus. In our primary experiments, we choose a Random Network Distillation (RND) bonus [3] as we find that this formulation is especially effective. Following Burda et al.[3], we train one small predictor MLPf θ,ℓ per layer to match a frozen random target MLPf ⋆ ℓ on features extracted from the language model’s own hidden states, and use the prediction error as the novelty signal. We extract features from a fixed set of intermediate layersL, mean-pool over completion tokens (the completion is encoded without the prompt), and average across layers: r novelty (τ ) = 1 |L| X ℓ∈L 1 √ d f ⋆ ℓ h ℓ (τ ) − f θ,ℓ h ℓ (τ ) 2 .(2) Hereh ℓ (τ )is the pooled layer-ℓfeature of trajectoryτanddis the output dimension of the predictor and target networks, so each layer contributes a root-mean-square prediction error. The predictor updates continuously on the same features that drive the bonus, while the target stays frozen for the entire run. In preliminary experiments, features from several intermediate layers gave a cleaner signal than last-layer features alone (layers and network sizes in Appendix A). We ablate the choice of r novelty in Appendix C.1 and observe a consistent improvement over DAPO across bonuses. Reward. We use a standard math verifier forr correct . In Appendix C, we sweep the weightλon r novelty and find that even very large weights leave the student model outperforming the base model. We find that performance roughly peaks at λ=0.5 and conduct all experiments with this value of λ. Training dataset. We sample random prompts without replacement from DAPO-Math-17K [44], a collection of 17K mathematical reasoning problems, until reaching a cap of 300 steps (discussed below). We perform additional experiments on the DeepScaleR [27] dataset in Appendix E. Evaluation. We report performance on seven mathematical reasoning benchmarks: AIME24, AIME25, AIME26 [29], MATH500 [17,22], AMC23 [29], Minerva-Math [20], and GSM8K [6]. Following the Qwen3 technical report [42], sample64completions per problem (32 for AMC23 and 8 for GSM8K) and report the unbiased pass@kestimator fork ∈1, 2, 4,..., 64. GSM8K is close to saturation for all three models, and AMC23 is close to saturation for the Qwen3 models, which leaves little room to compare methods. We therefore focus on the other five benchmarks and report their mean unless noted otherwise. We additionally study the model’s degradation on a suite of general capability benchmarks: MMLU- Pro [41], MMLU-Redux [12], and GPQA-Diamond [32] for knowledge, ZebraLogic [23] for logical reasoning, and IFEval [47] for instruction following. This serves as a proxy for capabilities not directly supervised by our training dataset and reward to study the extent to which exploration pressures can degrade a model’s knowledge over time. Baselines. We compare against three RLVR baselines with a standard correctness-only reward: GRPO [33], Dr. GRPO [26], and DAPO [44]. We find empirically that DAPO is the strongest of 5 1248163264 k (samples per problem) 50 60 70 80 Mean pass@ k (%) Qwen3-1.7B 1248163264 k (samples per problem) 60 65 70 75 80 85 Qwen3-4B 1248163264 k (samples per problem) 30 40 50 60 70 Ministral-3-3B BaseDAPO DAPO (4× steps) DAPO (+novelty)ExpDis (single-round)ExpDis Figure 3: ExpDis outperforms all baselines under the same RL compute budget. By partitioning this budget into alternating rounds of exploration and optimization (as described in Section 3.2), we can surpass DAPO and even a single round of ExpDis. We report full benchmark results in Appendix B. these baselines and design two extensions: (1) adding a novelty bonus to the reward withλ=0.5to isolate the effect of decoupling exploration, and (2) running DAPO for 4× as many steps. Compute matching. We perform 300 steps of RL with 4 prompts per step, each having a group size of 16 and a 32,768 max generation limit. We also study the effect of training DAPO for4×the number of steps to determine if prolonged training allows baselines to catch up [25]. ExpDis splits these updates between the two policies: 200 for the explorer and 100 for the student. When we scale across explorers or rounds, each share is divided evenly among them, while we keep the SFT stage of the student policy two epochs on at most 500 filtered explorer trajectories per round. 5 Results Across all three models, ExpDis outperforms the DAPO baseline in mean accuracy on mathematical reasoning benchmarks (Figure 3). A single round of ExpDis already outperforms DAPO trained for 4×as many steps. By spreading the same compute over several rounds of alternating exploration and optimization (Section 3.2), we observe further gains. All ExpDis runs outperform this prolonged DAPO run in both average performance and pass@k up to 64 generations per problem. 12345 ExpDis rounds 46 48 50 52 Mean accuracy (%) DAPO (4× steps) DAPO 1 explorer 3 explorers Breadth (3 explorers) Depth (4 rounds) Both 0 2 4 6 AIME24 gain over ExpDis (1 explorer, 1 round) (%) +0.7 +3.3 +3.1 +3.3 +4.3 +6.7 pass@1 pass@64 1248163264 k 50 60 70 80 90 AIME24 pass@ k (%) Base DAPO DAPO (4× steps) 1 explorer, 1 round 3 explorers, 1 round 1 explorer, 4 rounds 3 explorers, 4 rounds Figure 4: Under a fixed RL compute budget, how we allocate compute impacts downstream student policy performance. We vary the number of explorers and rounds under a fixed compute budget, and find complementary impacts of each scaling axis. Under a fixed budget, adding rounds improves pass@1 more than adding explorers, and combining both gives the largest gains in pass@1 and pass@64. All results are reported on Qwen3-1.7B. 6 Scaling exploration.Decoupling lets us scale exploration along two axes: running several explorers in parallel within a round and repeating the ExpDis procedure over several rounds. In Figure 4, we vary both on Qwen3-1.7B while holding the total budget fixed at 300 RL updates. We observe that depth contributes more to pass@1than breadth, though a combination of both improves pass@kover either. One interpretation is that the two axes play different roles: each additional round reinforces a progressively stronger student, whereas additional explorers widen the set of problems with correct trajectories and the variety of trajectories that get distilled into the student. At our budget, gains saturate and begin to show signs of regression after three explorers and four rounds. We expect this optimum to shift as the budget grows: with a higher compute budget, longer rounds and larger batch sizes per explorer will have diminishing marginal returns, and we expect the optimal number of rounds and explorers to increase accordingly. Note that we vary breadth and depth at a fixed compute budget; explorers run in parallel, and spreading compute across more explorers adds no wall-clock time. In principle, this parallelization allows practitioners with vast compute access to ramp up the number of explorers at no time cost. Diversity analysis. RLVR is known to collapse entropy and sharpen the sampling distribution as training proceeds [7,5]. Counteracting this collapse within a single model requires a delicate balance, since the same model must both explore diverse reasoning strategies but must simultaneously be optimized for correctness and avoiding incorrect solutions. Adding a novelty bonus to DAPO illustrates this trade-off: it increases both lexical and answer diversity relative to standard DAPO, yet slightly reduces mathematical reasoning performance. 0.30.40.5 Explorer lexical diversity 0.3 0.4 0.5 Student lexical diversity ExpDis (single-explorer), λ 0.25 0.5 0.75 1 2 0.30.40.5 Explorer lexical diversity 0 5 10 Student gain over Base (%) pass@1 pass@64 0.250.300.350.40 Explorer token entropy R1 R2 R3 R4 0.050.100.150.20 Semantic diversity BaseDAPODAPO (4× steps) DAPO (+novelty, λ=0.5) ExpDis (single-explorer)ExpDis Figure 5: Explorer policy can be driven to more diversity without harming the student. Left: We study how explorer policy diversity impacts the downstream student in ExpDis. In general, we find that we can push the diversity of the explorer policy without corrupting the downstream student policy. Since we filter for correctness and quality before distilling, a moderate increase in explorer diversity (for example, lexical diversity) brought on by tuning the novelty bonus weightλdoes not negatively impact the student policy. Similarly, we can increase the explorer token entropy far beyond that of DAPO (+novelty) without incurring downstream reasoning degradation. Right: We find that ExpDis generations contain more semantically diverse mathematical reasoning strategies. We define each diversity metric in Appendix D. Diversity and performance are not always complementary; a model may, for example, learn to rephrase the same solution strategy, thereby increasing lexical diversity or entropy without improving pass@k. As shown in Figure 5, DAPO + novelty exhibits higher lexical diversity and token entropy, but regresses on mathematical reasoning compared to the DAPO model. Broadly, we find that we can push the explorer toward far greater diversity than a single model could tolerate while maintaining or improving on downstream accuracy. Stage ablations.We run ablations on Qwen3-1.7B to understand whether ExpDis reduces to either (1) distilling unfiltered explorer trajectories and then running RL, or (2) distilling filtered trajectories without running RL afterward. We find that both variants score slightly below DAPO on AIME24, whereas the full procedure scores above it (Appendix E). Furthermore, keeping only the shortest valid trajectory per problem (i.e., quality filtering) outperforms keeping every correct trajectory. Finally, we replace RND with a kNN-based or an elliptical novelty bonus (the latter adapted from Tuyls et al. [38]) and find that mean accuracy drops by at most1.6points, and both variants still outperform DAPO (Appendix C.1). 7 02468 Math gain over Base (%) −2 −1 0 +1 +2 Prior capabilities change (%) DAPO DAPO (4× steps) ExpDis DAPO (+novelty) Qwen3-1.7B Qwen3-4B Ministral-3B Figure 6: Adding RND novelty bonus to DAPO degrades prior capabilities. Novelty bonus degradation. A verifiable reward, typically applied to math and code tasks, directly supervises only a nar- row slice of a model’s capabilities. In our experiments, DAPO with the RND novelty bonus degrades below the base model on prior capability benchmarks (Section 4) despite improving on math (right). Conversely, ExpDis gets the benefits of explo- ration without degradation. We report results for each general capability benchmark in Figure 12. While the drops we observe are modest at our scale, we specu- late that larger-scale RLVR runs that apply exploration bonuses directly to the trained model may accumulate substantially larger degradations over time, whereas decoupling avoids this risk of corrupting the deployed model. Policy evolution.We study how the explorer and student policies evolve across several iterations of ExpDis. Figure 7 suggests a simple picture: the explorers repeatedly expand the space of reasoning strategies available to the student, and the student keeps a growing share of that expansion while leaving its costs (reduced pass@kand reasoning faithfulness) behind. Within each round, the novelty bonus pushes the explorer entropy and semantic diversity well above the previous student, and subsequent distillation on the shortest correct trajectories contracts them again. We find that both the student’s diversity and accuracy ratchet upward together across subsequent rounds. 50 60 70 80 pass@1 pass@8 pass@64 BaseRound 1Round 2Round 3Round 4 Accuracy 0.25 0.30 0.35 R1R2R3R4 (a) Token Entropy 0.100 0.125 R1R2R3R4 (b) Semantic Diversity 0.28 0.32 0.36 R1R2R3R4 (c) Lexical Diversity 0.80 0.84 0.88 R1R2R3R4 (d) Reasoning Faithfulness explorer RL (+ novelty) filter & distillstudent RL student π r−1 explorers μstudent after SFT student π r DAPODAPO (4× steps) DAPO (+novelty)ExpDis (single-explorer) Figure 7: In successive ExpDis iterations, the student captures more of the explorer’s diversity while avoiding its degradations. We track the explorers and the student as they evolve during several ExpDis rounds and report the final baseline policies to the right of each panel. (a) Explorer RL causes a rise in token entropy often at the expense of accuracy, whereas the student’s entropy generally rises together with its accuracy across rounds. (b) The explorer policy semantic diversity stays roughly constant, and the student closes most of the gap to it by the final round. (c) We find that the student does not retain the lexical diversity of its explorers: it declines over rounds even as the student’s accuracy improves. Together with Figure 5, where largeλharms the student, this suggests that lexical diversity is not a useful target for exploration. (d) Reasoning faithfulness, as defined by Rahman et al.[31], measures whether a response’s reasoning supports its final answer. We find that novelty bonus typically causes faithfulness to decline, though the impact is mild by the final round. Results are averaged over three models (breakdown in Figure 14). Furthermore, explorer policies reach roughly the same semantic diversity in every round, but the gap between them and the student they produce nearly closes by the fourth. One explanation is that each round’s explorers start from a stronger student and therefore solve more of the training problems, so 8 more problems contribute a correct trajectory to the distillation set and the student inherits a broader range of the explorer policy’s solutions. 6 Discussion In standard RLVR, exploration is limited by the need to remain correct. ExpDis removes this constraint by moving exploration into a separate policy. Because the explorer is never deployed, it can be rewarded for novelty, pushed to extremes, or discarded after a single round, with only filtered reasoning traces passing to the student. Our work is inspired by humans who experiment with bold new ideas when failure is permitted. We hypothesize that models too need a space to explore, try new ideas, and be allowed to fail. Our results suggest that decoupling exploration from optimization improves RLVR even at a small budget, and we expect its importance and the value of exploration to grow as RL is scaled. As models improve, they become proficient at employing the solution strategies they have seen before, and the problems worth training on are increasingly those where correct solutions are rare or depart from the solution strategies the model has already mastered. A model optimized only for correctness may avoid sampling such novel strategies. This problem may be especially true beyond mathematical reasoning, for example in long-horizon agentic tasks, theorem proving, and open-ended scientific problems. Decoupling lets explorers be aggressive, and this aggressive exploration might be necessary in order for models to discover completely new ideas. In our work, we focus our effort primarily on a single novelty bonus and explorers that share the student’s architecture. Under ExpDis, explorers could use different novelty signals or sampling strategies to specialize in different parts of the solution distribution. Because an explorer only has to produce useful traces, the best exploration method under ExpDis may look quite different from the best one under standard RLVR. We speculate that exploration methods previously too destabilizing to apply directly to a trained model may become usable in this setting. ExpDis may unlock new and more aggressive exploration mechanisms. 7 Acknowledgments This project was generously supported by the Google TPU Builders Program, by IBM, and by the NVIDIA Academic Grant Program. We thank Ang Li, Rohun Agrawal, Kaiwen Zha, Andrea Zanette, and He He for helpful discussions. 9 References [1]Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017. [2] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016. [3]Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018. [4]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [5]Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? Advances in Neural Information Processing Systems, 38:57654–57689, 2026. [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [7]Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025. [8]Jim Dilkes, Vahid Yazdanpanah, and Sebastian Stein. Instruction-conditioned exploration with asymmetric reinforcement learning and self-distillation. arXiv preprint arXiv:2608.02087, 2026. [9] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580–586, 2021. [10]Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018. [11]Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Maolin Wang, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration. arXiv preprint arXiv:2505.17621, 2025. [12]Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5069–5096, 2025. [13] Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. In International Conference on Learning Representations, volume 2026, pages 108059–108130, 2026. [14]Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. [15]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [16] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series.https://capricious-hydrogen-41c.notion.site/ Skywork-Open-Reaonser-Series-1d0bc9ae823a80459b46c149e4f51680, 2025. Notion Blog. [17] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 10 [18]Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2):209–232, 2002. [19]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. [20]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857, 2022. [21]Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations. arXiv preprint arXiv:2509.02534, 2025. [22]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pages 39578–39601, 2024. [23]Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. ZebraLogic: On the scaling limits of LLMs for logical reasoning. In International conference on machine learning, pages 37889–37905. PMLR, 2025. [24] Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3. arXiv preprint arXiv:2601.08584, 2026. [25]Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. Advances in Neural Information Processing Systems, 38:17998–18031, 2026. [26]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025. URL https://arxiv. org/abs/2503.20783, 1, 2024. [27]Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing O1-Preview with a 1.5B model by scaling RL, 2025. Notion Blog. [28] Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406, 2026. [29] Mathematical Association of America. American invitational mathematics examination (aime), 2026. [30]Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017. [31] Salman Rahman, Jingyan Shen, Anna Mordvina, Hamid Palangi, Saadia Gabriel, and Pavel Izmailov. When can LLMs learn to reason with weak supervision? arXiv preprint arXiv:2604.18574, 2026. [32]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. [33]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [34] Özgür ̧Sim ̧sek and Andrew G Barto. An intrinsic reward mechanism for efficient exploration. In Proceed- ings of the 23rd international conference on Machine learning, pages 833–840, 2006. [35]Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023. [36]Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for llm reasoning. arXiv preprint arXiv:2509.06941, 2025. 11 [37]Richard S Sutton and Andrew Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. [38]Jens Tuyls, Dylan Foster, Akshay Krishnamurthy, and Jordan Ash. Representation-based exploration for language models: From test-time to post-training. In International Conference on Learning Representations, volume 2026, pages 120621–120645, 2026. [39]Jingchu Wang, Bingbing Xu, Yige Yuan, Bin Xie, Xiaoqian Sun, and Huawei Shen. R 2 PO: Decoupling training trajectories from inference responses for llm reasoning. arXiv preprint arXiv:2601.11960, 2026. [40]Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. Advances in Neural Information Processing Systems, 38: 115452–115486, 2026. [41]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. [42]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [43] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025. [44]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. [45] Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. From f (x) and g (x) to f (g (x)): Llms learn new skills in rl by composing old ones. In International Conference on Learning Representations, volume 2026, pages 147547–147574, 2026. [46]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. [47]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 12 A Training & Hyperparameters We report all hyperparameters shared by all methods in Table 1. As in the standard DAPO [44] algorithm, we perform dynamic sampling, overlong filtering, clip-higher, and do not include a KL penalty term. We use the Dr. GRPO [26] loss normalization, which divides the token-summed loss by a fixed constant instead of the number of tokens. Table 1: Hyperparameters. Shared by all models and all runs unless noted. RL (explorer, student, baselines) ObjectiveDAPO clipped objective, ε low =0.2, ε high =0.28, KL β=0 Prompts per step, group size B=4, G=16 Stepsexplorer 200, student 100, baselines 300, DAPO (4× steps) 1,200 total Max completion length32,768 Soft overlong penaltylinear over the final 6,554 tokens OptimizerAdamW (β=(0.9, 0.95), no weight decay) Learning rate (constant)explorer and DAPO baselines 5×10 −6 ; student 1×10 −6 SFT (student) Lossmasked cross-entropy loss (completion-only) Epochs, max examples2 epochs, 500 per round OptimizerAdamW Learning rate5×10 −6 (constant) Novelty bonus (RND) Layers⌊L/4⌋,⌊L/2⌋,⌊3L/4⌋ Predictor and target3-layer MLP (Linear–ReLU–Linear–ReLU–Linear), width and output 512 Predictor optimizerAdam, learning rate10 −4 , one step per policy update; predictor and target re- initialized for every explorer and round Featureslive policy’s hidden states, mean-pooled over completion tokens Bonusraw (no normalization), credited on verifier-correct completions only RL update.All RL stages use entirely on-policy steps: vLLM servers reload the trainer’s weights after every step and each batch of4prompts× 16samples contributes to the following gradient step. 0255075100125150175200 Explorer step 0.0 0.2 0.4 0.6 Correctness reward (a) Explorer RL ExpDis (λ=0.5) DAPO 020406080100 Student RL step 0.45 0.50 0.55 0.60 0.65 Correctness reward (b) Student RL 0255075100125150175200 Explorer step (within round) 0.30 0.35 0.40 0.45 0.50 0.55 0.60 Novelty reward (c) Explorer novelty reward R1 R2 R3 R4 051015202530 SFT step (within round) 1.0 1.2 1.4 1.6 1.8 SFT loss (d) Student SFT Figure 8: Training dynamics of continued training on Qwen3-1.7B with our ExpDis setup (λ=0.5). 13 Table 2: Update allocation per configuration. MR-ME (multi-round, multi-explorer) combines both axes: Rrounds withKexplorers per round. Every ExpDis run uses 200 explorer and 100 student updates (19,200 selected rollouts); single-model baselines spend all 300 on one model. Parallel explorers run concurrently, so wall-clock time is equal across rows. DAPO (4× steps) is the only exception. ConfigurationR× KExplorer updatesStudent updates per round Single-Explorer1× 1 200100 Breadth K=21× 2 100 each100 Breadth K=31× 3 67/67/66100 Breadth K=51× 5 40 each100 Breadth K=71× 7 29/29/29/29/28/28/28100 Depth R=44× 1 50 per round25 Depth R=55× 1 40 per round20 MR-ME R=4, K=3 4× 3 17/17/16 per round25 MR-ME R=5, K=3 5× 3 14/13/13 per round20 MR-ME R=4, K=5 4× 5 10 per explorer per round25 A.1 Compute budget Hardware. Each run uses one TPU v5litepod-64 slice (16 hosts): 4 hosts train the policy with FSDP and 12 serve rollouts with vLLM in bf16 with a 40,960-token context. Table 3: Sampling parameters. ∗ 32 for AMC23 and 8 for GSM8K. Training rolloutsEvaluation Temperature1.00.6 top-p / top-k / min-p0.95 / 20 / –0.95 / 20 / 0 Max completion tokens32,76832,768 Samples per problem1664 ∗ A.2 pass@k estimator Withnsamples per problem, of whichc i are correct on problemi, we use the unbiased estimator of Chen et al. [4] averaged over N problems: pass@k = 1 N N X i=1 " 1− n−c i k n k # , k ≤ n.(3) B Extended Results We report pass@k for all models and methods in Figure 9 and Table 4. 14 60 80 AIME24 pass@ k (%) Qwen3-1.7B 40 60 80 AIME25 pass@ k (%) 40 60 80 AIME26 pass@ k (%) 75 80 85 90 MATH500 pass@ k (%) 1248163264 k 30 40 50 Minerva-Math pass@ k (%) 80 90 Qwen3-4B 70 80 90 60 70 80 90 80 85 90 95 1248163264 k 40 50 60 20 40 60 Ministral-3-3B 20 40 60 20 40 60 70 80 90 1248163264 k 30 40 50 BaseDAPO DAPO (4× steps) DAPO (+novelty)ExpDis (single-round)ExpDis Figure 9: Full pass@k results. C Novelty Bonus Analysis Impact on DAPO (+novelty). In the DAPO (+novelty) baseline, we augment the standard correctness-only reward with a novelty bonus as in(1). We study how the method performs with various novelty bonus weightsλfrom0.25to 2. We perform three training runs over different seeds and find that, with the RND novelty bonus, the mean mathematical reasoning performance degrades in comparison to DAPO. 00.250.50.7512 Novelty weight λ on a single model 43.0 43.5 44.0 44.5 45.0 45.5 46.0 Five-benchmark mean (%) λ=0 control Figure 10: Adding a novelty bonus directly to the reward in DAPO degrades accuracy on math reasoning. 15 Table 4: Mean pass@k over the five primary benchmarks. ModelMethod@1@2@4@8@16@32@64 Qwen3-1.7BBase43.42 52.18 58.26 63.02 66.95 70.63 73.82 Qwen3-1.7BDAPO45.63 54.31 60.67 65.54 69.16 72.14 74.64 Qwen3-1.7BDAPO (4× steps)47.28 56.44 62.83 67.41 70.80 73.62 76.24 Qwen3-1.7BDAPO (+novelty)45.08 53.78 60.07 64.91 68.61 71.77 74.43 Qwen3-1.7BExpDis (single-round) 48.93 58.55 64.98 69.27 72.43 75.10 77.83 Qwen3-1.7BExpDis51.81 61.40 68.03 72.94 76.97 80.29 83.04 Qwen3-4BBase61.06 68.90 72.83 75.43 77.41 78.90 79.86 Qwen3-4BDAPO62.58 69.97 73.84 76.46 78.41 79.88 81.14 Qwen3-4BDAPO (4× steps)63.64 71.15 75.05 77.65 79.62 81.20 82.82 Qwen3-4BDAPO (+novelty)62.20 69.70 73.59 76.20 78.16 79.64 80.82 Qwen3-4BExpDis (single-round) 64.69 72.32 76.24 78.83 80.82 82.51 84.50 Qwen3-4BExpDis66.50 74.80 79.13 81.82 83.82 85.53 87.63 Ministral-3-3B Base27.95 34.54 40.52 45.34 49.36 53.00 56.50 Ministral-3-3B DAPO29.63 36.63 43.21 48.79 53.45 57.19 60.12 Ministral-3-3B DAPO (4× steps)30.90 38.19 45.05 50.86 55.74 59.63 62.36 Ministral-3-3B DAPO (+novelty)29.21 36.11 42.54 47.93 52.43 56.15 59.22 Ministral-3-3B ExpDis (single-round) 32.16 39.74 46.87 52.93 58.01 62.05 64.59 Ministral-3-3B ExpDis34.76 43.22 51.31 58.35 64.37 69.24 72.56 Impact on student. We additionally study the impact that sweeping the novelty bonus weight (λ) on the explorer policy has on the downstream student policy in a single-round ExpDis setup. Though any amount of novelty bonus hurts the explorer policy, even at high weights (such asλ=2) where pass@k regresses, the downstream student policy does not inherit this degradation. 0.25.5.7512 λ 48 50 52 avg@ n (%) AIME24 0.25.5.7512 λ 35 36 37 38 39 AIME25 0.25.5.7512 λ 35 40 45 AIME26 0.25.5.7512 λ 73 74 75 76 MATH500 0.25.5.7512 λ 84 85 86 avg@ n (%) AMC23 0.25.5.7512 λ 28 29 30 Minerva-Math 0.25.5.7512 λ 89.5 90.0 90.5 GSM8K Qwen3-1.7B d) DAPO DAPO (+novelty) Base ExpDis (single-round) Figure 11: On Qwen3-1.7B, we find that a novelty bonus weight ofλ=0.5 on the explorer policy roughly maximizes downstream student policy performance, though several values ofλoutperform the DAPO baseline. C.1 Novelty metric (r novelty ) To test whether our gains depend on RND, we train two single-round explorers on Qwen3-1.7B with a different novelty term based on the mean-pooled, L2-normalized final-layer hidden states of each trajectory. 1.kNN novelty scores each correct completion by its mean cosine distance to itsk=16nearest neighbors in a replay buffer of the 4,096 most recent correct completions. 16 048 Math gain (%) −2 0 +2 Change from Base (%) MMLU-Pro 048 Math gain (%) MMLU-Redux 048 Math gain (%) GPQA-Diamond 048 Math gain (%) IFEval 048 Math gain (%) ZebraLogic DAPODAPO (4× steps)ExpDisDAPO (+novelty)Qwen3-1.7BQwen3-4BMinistral-3B Figure 12: We report the degradation from adding a novelty bonus directly to the DAPO reward. 2.Elliptical novelty, adapted from the elliptical bonus of Tuyls et al.[38], uses the same featuresφand the bonus p φ ⊤ Σ −1 φ, whereΣis a ridge-regularized running covariance, Σ = I + P i φ i φ ⊤ i over past correct completions. We use λ=0.5 for both and report results in Table 5. Table 5: We ablate the choice of novelty metric used as our novelty bonus. Methodr novelty AIME24 AIME25 AIME26 MATH500 Minerva Mean DAPO–50.0536.9337.6674.2429.2745.63 DAPO (4× steps) –51.4138.0541.9675.0229.9847.28 ExpDis (ours)RND52.7639.1746.2575.8030.6948.93 ExpDiskNN50.8337.5043.7575.0029.7547.37 ExpDisElliptical51.5638.2344.7975.3030.4048.06 12345 Round r 54 56 avg@ n (%) AIME24 12345 Round r 39 40 41 42 AIME25 12345 Round r 46.0 46.5 47.0 47.5 AIME26 12345 Round r 76 77 78 MATH500 12345 Round r 31 32 33 34 Minerva-Math Qwen3-1.7B, annealed λ: K=3 explorersK=1 explorer Figure 13: Impact of increasing rounds given a fixed number of explorers per round. D Diversity Analysis We define the diversity metrics used in the paper. InterDistinct-4. Givenngenerationsg 1 ,...,g n for a single problem, let4grams(g i )be the multiset of 4-grams in g i and G 4 = S i 4grams(g i ) the set of distinct 4-grams in the pool. Then InterDistinct-4 = |G 4 | P i |4grams(g i )| , the ratio of distinct 4-grams to total 4-grams in the pool. We average across problems. AnswerDistinct@n.For each problem, count the number of distinct extracted final answers across the n generations, then average across problems. 17 Token entropy. For each model we sample 8 rollouts for each of the 30 AIME24 problems with the evaluation sampler and record the mean per-token entropy of the sampled tokens. Semantic diversity. We follow Li et al.[21], who finetune Qwen3-Embedding-4B as a classifier of whether two mathematical reasoning traces use the same approach. We use their checkpoint to produce pairwise similarity scores between all generations for a single prompt. We then cluster the generations and report the resulting number of clusters C normalized by the number of generations. Table 6: We report diversity metrics for Qwen3-1.7B models on AIME24 generations. ModelEntropy Semantic Ans. entropy Distinct ans. InterDistinct-4 Base0.2370.1001.428.940.311 DAPO0.2510.0961.378.500.308 DAPO (4× steps)0.2650.0911.358.460.302 ExpDis (single-explorer) 0.3060.1201.419.930.357 ExpDis0.3910.1431.309.440.339 Table 7: We report InterDistinct-4 (Inter4) and AnswerDistinct@n(Ans@n) for the explorer and the student on AIME24 across explorer weights λ. ExplorerStudent ModelMethodInter4 Ans@n Inter4 Ans@n Qwen3-1.7B Base– 0.3118.94 Qwen3-1.7B DAPO– 0.3088.50 Qwen3-1.7B DAPO (+novelty, λ=0.5)– 0.3429.39 Qwen3-1.7B ExpDis (λ=0.25)0.40311.60 0.3469.63 Qwen3-1.7B ExpDis (λ=0.5)0.42212.61 0.3579.93 Qwen3-1.7B ExpDis (λ=0.75)0.47814.44 0.36310.02 Qwen3-1.7B ExpDis (λ=1.0)0.53416.28 0.3359.30 Qwen3-1.7B ExpDis (λ=2.0)0.55023.61 0.2837.87 Qwen3-4BBase– 0.2904.02 Qwen3-4BDAPO– 0.2873.82 Qwen3-4BDAPO (+novelty, λ=0.5)– 0.3194.22 Qwen3-4BExpDis (λ=0.5)0.3945.67 0.3334.47 E Ablations Quality filtering.With three explorers, keeping every correct trajectory from the pooled explorers (NaivePool) gives a mean of48.95%, no better than a single explorer (48.93%). Applying the quality filter increases performance to 49.47%. Student distillation vs student RL.On Qwen3-1.7B, distilling unfiltered explorer traces and then running RL reaches49.79% on AIME24, and distilling filtered traces without RL reaches49.84%. Both variants underperform DAPO (at50.05%). Only filtering followed by RL reaches52.76% (Table 8). Annealingλ. We find that annealingλduring multiple rounds (0.75, 0.5, 0.35, 0.25) outperforms a fixed λ=0.5 at every round (see Figure 15). Training data.Every run samples the same number of prompts (about1,200), so these runs differ only in the pool they sample from (Figure 16, Table 9). Sampling from DAPO-Math-17K or a 17K subset of DeepScaleR gives similar results, with neither better on every benchmark. Sampling ExpDis prompts from the full 40K DeepScaleR pool is better on every benchmark. All main results use DAPO-Math-17K. 18 60 80 Accuracy (%) R1R2R3R4 Qwen3-1.7B 60 75 90 R1R2R3R4 Qwen3-4B 40 60 R1R2R3R4 Ministral-3-3B 0.24 0.32 0.40 Token entropy R1R2R3R4 0.25 0.30 R1R2R3R4 0.32 0.36 R1R2R3R4 0.100 0.125 Semantic diversity R1R2R3R4 0.100 0.125 0.150 R1R2R3R4 0.100 0.125 R1R2R3R4 0.30 0.35 0.40 Lexical diversity R1R2R3R4 0.25 0.30 0.35 R1R2R3R4 0.30 0.33 R1R2R3R4 0.80 0.84 0.88 Reasoning faithfulness R1R2R3R4 0.88 0.92 0.96 R1R2R3R4 0.75 0.80 0.85 R1R2R3R4 explorer RL (+ novelty) filter & distillstudent RL student π r−1 explorers μ student after SFT student π r DAPODAPO (4× steps) DAPO (+novelty)ExpDis (single-explorer) Figure 14: We study the evolution of student and teacher policies during several stages of ExpDis. 19 Table 8: Ablating student distillation, student RL, or both. ConfigurationAIME24 DAPO50.05 Unfiltered SFT + RL49.79 Filtered SFT, no RL49.84 Filtered SFT + RL (ExpDis)52.76 1234 Round r 48.5 49.0 49.5 50.0 50.5 51.0 51.5 Mean accuracy (%) λ=0.75 λ=0.5 λ=0.35 λ=0.25 annealed λ fixed λ=0.5 Figure 15: Fixed versus annealedλacross four rounds (Qwen3-1.7B, single-explorer). The an- nealed schedule uses λ=0.75, 0.5, 0.35, and 0.25. Table 9: We ablate the choice of training dataset. We see roughly the same performance across all datasets. MethodDatasetAIME24 AIME25 AIME26 MATH500 AMC23 Minerva GSM8K GRPODAPO-Math-17K47.8135.3635.6872.8483.7528.0289.89 GRPODeepScaleR-17K47.9635.5235.6872.7184.0927.8689.81 Dr. GRPODAPO-Math-17K48.4935.9936.4673.3484.3828.4590.04 Dr. GRPODeepScaleR-17K48.8336.2636.5173.1884.7828.2089.95 DAPODAPO-Math-17K50.0536.9337.6674.2485.1629.2790.25 DAPODeepScaleR-17K49.6837.2137.7674.0885.6228.9090.14 ExpDis λ=0.5 DAPO-Math-17K52.7639.1746.2575.8086.1730.6990.68 ExpDis λ=0.5 DeepScaleR-17K51.7738.3344.8475.1686.4830.4490.59 ExpDis λ=0.5 DeepScaleR-full52.8939.2447.6676.0886.8431.3790.78 DAPO- 17K DSR- 17K DSR- full 48 50 52 avg@ n (%) AIME24 DAPO- 17K DSR- 17K DSR- full 36 38 AIME25 DAPO- 17K DSR- 17K DSR- full 35 40 45 AIME26 DAPO- 17K DSR- 17K DSR- full 72 74 76 MATH500 DAPO- 17K DSR- 17K DSR- full 84 86 avg@ n (%) AMC23 DAPO- 17K DSR- 17K DSR- full 28 30 Minerva-Math DAPO- 17K DSR- 17K DSR- full 90 91 GSM8K Qwen3-1.7B GRPO Dr.GRPO DAPO ExpDis λ=0.5 Figure 16: We ablate the choice of training dataset, sampling a fixed prompt budget from each and training with a single-explorer ExpDis approach. We find a consistent gain over several RLVR baselines. 20

Instant research discovery