Paper deep dive
BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?
Soumadeep Saha, Krish Sharma, Akshay Chaturvedi, Nicholas Asher
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/5/2026, 4:36:28 AM
Summary
This paper investigates whether Reinforcement Learning with Verifiable Rewards (RLVR) expands the reasoning capabilities of Large Language Models (LLMs) or merely improves sampling efficiency by reducing exploration. Using 'BODHI-Trees' to map semantic equivalence in mathematical reasoning and controlled maze-solving experiments, the authors find that RLVR-trained models exhibit a significant collapse in policy entropy. This collapse is not just syntactic (stylistic variations) but semantic, meaning the models constrict the space of valid, distinct reasoning continuations. While RLVR improves adherence to constraints and backtracking, it reduces genuine rollout diversity, suggesting that sample efficiency gains come at the cost of exploratory capability.
Entities (10)
Relation Signals (7)
RLVR → causes → Policy Entropy Collapse
confidence 95% · Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic...
BODHI-Tree → usedfor → Analyzing Semantic Equivalence
confidence 95% · extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence.
RLVR → reduces → Semantic Branching Entropy
confidence 92% · accompanied by a significant reduction in semantic branching entropy.
RLVR → constricts → Space of Continuations
confidence 90% · it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR...
RLVR → improves → Adherence to Environmental Constraints
confidence 90% · While RLVR improves adherence to environmental constraints and backtracking capabilities...
Policy Entropy Collapse → isnotjust → Syntactic Variation
confidence 90% · the policy entropy collapse observed in RLVR models is not merely syntactic...
Maze Solving → usedfor → Isolating Routing and Planning Capabilities
confidence 85% · maze solving serves as a rigorous, noise-free proxy for multi-step logical reasoning... to isolate the routing and planning capabilities...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity.
Tags
Links
- Source: https://arxiv.org/abs/2608.02867v1
- Canonical: https://arxiv.org/abs/2608.02867v1
Trouble viewing inline? Open PDF directly →
Full Text
60,409 characters extracted from source content.
Expand or collapse full text
BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences? Soumadeep Saha† , Krish Sharma† , Akshay Chaturvedi*, Nicholas Asher† † , Université de Toulouse; *LINAGORA Labs Correspondence: soumadeep.saha97@gmail.com Abstract Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity. BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences? Soumadeep Saha† , Krish Sharma† , Akshay Chaturvedi*, Nicholas Asher† † , Université de Toulouse; *LINAGORA Labs Correspondence: soumadeep.saha97@gmail.com 1 Introduction Figure 1: Construction of mathematical reasoning trees. We first segment reasoning traces into a chain of reasoning nodes, followed by merging said nodes based on semantic similarity. Further details are provided in Appendix A. 111All associated artifacts can be found at espressovi.github.io/BODHI. Several recent papers have pointed to a curious problem in reinforcement learning with verifiable rewards (RLVR)-trained large language models (LLMs). These models do not seem to “discover” novel abilities beyond the base model, are less exploratory, and are reliant on a minority of high-entropy tokens to drive reasoning diversity (Chen et al., 2026a; Saha et al., 2026; Wang et al., 2025; Yue et al., 2025). An exploratory model with the ability to access a large number of valid, semantically distinct continuations is evidently desirable—it allows for the realisation of novel solutions, increased diversity, and even performance scaling with techniques such as self-consistency (Wang et al., 2023), tree-of-thought (Yao et al., 2023), etc. However, it is also clear that not all exploration is desirable. For instance, an LLM with the capability to produce a large set of valid continuations, i.e., correct mathematical reasoning traces, which differ in superficial ways such as the naming of variables, order of commutative operations, or minor syntactic variations in its natural language (NL) framing does not constitute meaningful exploration, as it fails to traverse fundamentally distinct reasoning pathways or yield novel solution strategies. This brings us to the core tenet motivating this study: do LLMs Branch Out and Discover Heterogeneous Inferences, or is the exploration largely decorative? To this end, we pose three research questions in this study: ⋄ RQ1: Does RLVR limit test-time exploration and demonstrate stronger preferences for certain trajectories? ⋄ RQ2: If so, is the policy concentration merely syntactic/stylistic? ⋄ RQ3: And how is the RLVR-induced policy shift connected to performance? Surprisingly, despite many recent papers making fundamental improvements to our understanding of the dynamics of RL training, especially pertaining to preventing policy collapse, encouraging exploration, etc. (Yu et al., 2026; Liu et al., 2025; Li et al., 2025; Yang et al., 2025b; Zhao et al., 2026), the fundamental question of how RLVR changes preferences among semantically distinct reasoning continuations remains under-explored. We investigate exploration in LLMs through two modalities: mazes, where the notion of exploration is rather straightforward, and mathematical reasoning, where it is much less straightforward due to the complexities of the mathematical and linguistic expressions used. In order to investigate exploration in mathematical reasoning, we concretize the notion of exploration by collapsing a large number of alternate traces corresponding to a problem into a tree structure (BODHI-Tree, see Figure 1), wherein each node represents several semantically identical reasoning traces and its distinct children represent conceptually different continuations, thus allowing for fine-grained analysis of the nature of exploration. The core contributions of our work are as follows: (i) Using our BODHI-Tree dataset and mazes, we demonstrate that the policy resulting from RL-training shows a collapse in branching behavior, (i) the collapse in entropy is not just syntactic, and RLVR-trained models also demonstrate a significant collapse in the semantic-branch preference entropy, and (i) further, we provide evidence suggesting that RLVR drives performance by constricting the accessible state space of both invalid continuations and valid, semantically distinct ones. Additionally, in the course of this study we created and open-sourced several fully-reproducible intermediate post-training checkpoints (in addition to datasets and code), to aid future work into RLVR dynamics. 2 Background RLVR reframes next-token prediction as a Markov Decision Process, learning a policy πθ _θ for predicting next tokens oio_i, that leads to trajectories of states [Input]→[Input,o1]→…→[Input,o1,…oT][ Input]→[ Input,o_1]→…→[ Input,o_1,… o_T], that maximize expected reward from deterministic verifiers. This paradigm has significantly improved overall performance on several reasoning and planning benchmarks, and has pushed the frontiers of capabilities such as integration of tools and agentic applications. It also enables an ancillary technique, called Long-CoT distillation or simply distillation (Guo et al., 2025; Shao et al., 2024), wherein an LLM is fine-tuned (SFT) on traces generated from a more capable RL-trained “teacher” LLM. Given a state sts_t, the set of continuations—every finite string with sts_t as a prefix—can be segmented into various classes which are “verifier equivalent”. However, since the verifier often only evaluates the last few tokens of the terminal state (Shao et al., 2024; Guo et al., 2025; Yu et al., 2026; Liu et al., 2025), verifier equivalence does not cleanly map to semantic equivalence. For instance, a specious continuation which happens to present the correct answer has the same utility as a semantically correct continuation, and there is no a priori guarantee that verifier-equivalent but semantically diverse continuations have an equal propensity of realization (Anschel et al., 2025). This serves as the core object of analysis in our study: the accessibility of different verifier-equivalent trajectories222i.e., the probabilities of different (finite) continuations that are verifier-equivalent. at test time, a property we refer to as exploration. This is different from inter-trace exploration (Jiang et al., 2025), and asks the counterfactual question: what other continuations were likely? Test-time exploration elicits the “reasoning capability” boundary (Yue et al., 2025), since, given enough samples, a model capable of accessing diverse trajectories will arrive at novel solution approaches or techniques. Thus, a litmus test for the efficacy of RL-training is its ability to explore. Additionally, exploration can amplify performance with the aid of accessory ensembling techniques such as tree-of-thought (Yao et al., 2023). With the proliferation of RL-trained LLMs, it is therefore critical to understand the impact of the reward signal on the model’s generative policy, especially since the assumption that RLVR consistently incentivizes semantically diverse and meaningful exploration has been widely challenged (Saha et al., 2026; Wang et al., 2025; Yue et al., 2025; Anschel et al., 2025). Yue et al. (2025) claimed that RL-trained LLMs have degraded pass@k pass@k scaling and do not discover fundamentally novel reasoning pathways beyond the base LLM. Saha et al. (2026) posited that RL-trained models are “over-confident” compared to distilled models: they have a lower entropy policy which is correlated with a drop in performance. Similarly, Wang et al. (2025) demonstrated that the RL-policy is reliant on a minority (∼20% 20\%) of high-entropy “forking tokens” to drive meaningful exploration at test time (Cheng et al., 2026), and Jang et al. (2026) attributed this collapse to “over-confidence at a small number of structurally critical decision points”. Yuan et al. (2026), however, oppose the view that decline of high-k pass@k pass@k performance is indicative of a decline in reasoning diversity. They liken RLVR exploration collapse to “overtraining” and proffer re-weighting policy updates to favor low-success rollout groups. Cai et al. (2026) make a similar diagnosis, and advance “boundary-aware” curriculum learning to enhance the frontier of reasoning capabilities. Huang et al. (2026) argue that token-level statistics do not reflect how reasoning progresses over multi-token semantic structures. This debate underscores a critical gap: existing performance-based and token-level metrics do not indicate if entropy collapse implies a lack of reasoning diversity. The collapse of policy entropy during RL-training has received significant attention, and several recent papers propose modifications to the standard RL-training regimen. Monte Carlo Tree Search-based algorithms (Li et al., 2025; Yang et al., 2025b; Zheng et al., 2025) that sample rollouts during training in a tree-like structure and assign segment-level credits have been proposed. Other techniques explicitly include cues to enhance exploration based on properties of the rollout group (Chen et al., 2026a). Techniques such as those based on occurrence frequency (Anschel et al., 2025), embedding similarity (Zhao et al., 2026), feedback from auxiliary models (Mishra et al., 2026; Hu et al., 2026a), and many others (Li and Li, 2026; He et al., 2026; Cai et al., 2026; Hu et al., 2026b; Jang et al., 2026; Yuan et al., 2026; Huang et al., 2026; Chen et al., 2026b) have been proposed, with varying degrees of success (Zhao et al., 2026; Hao et al., 2026). Hao et al. (2026) study factors influencing entropy collapse during training and highlight that recent approaches rely on heuristic adjustments to some of these factors, thus limiting their efficacy. In related studies, Jin et al. (2026) studied the relationship between model performance and training entropy, finding a negative but task-dependent correlation. Wen et al. (2026) used an LLM-as-a-judge framework to evaluate intermediate trace correctness (CoT-pass@k pass@k) alongside the answer, demonstrating that RLVR-trained models have enhanced trace validity compared to the base model. The results for CoT-pass@k pass@k scaling in mathematical reasoning tasks are mixed, and the authors speculate that the “distilled LLM may already master major reasoning capabilities that can be learned with RLVR”, which further underscores the need to isolate the effect of RLVR post-training on test-time exploration. Jiang et al. (2025) introduce LCoT2Tree, which parses Long CoT traces into trees by mapping segments of reasoning traces to an extracted high-level summary. By classifying segment functions (e.g., verification, exploration, backtracking), they identify structural error patterns such as over-branching. Our work, instead of focusing on token-level metrics or performance alone, attempts to measure how RLVR changes preferences among semantically distinct reasoning continuations, and tries to ascertain if the entropy collapse is simply an artifact of stronger policy preferences for syntax and style or if the model also has calcified preferences when it comes to semantically diverse verifier-equivalent continuations. 3 Methodology: Measuring Exploration In this section, we outline the construction of the probing datasets that enable us to distinguish differences between the policies of various LLMs with regard to exploration. While NL mathematical reasoning directly measures an LLM’s logical capabilities, it is inherently noisy; the vast linguistic state space allows models to generate superficial restatements or varying syntactic formulations that mimic diversity without representing genuine inferential branching. Thus, we employ two modalities: (i) maze solving, to isolate the routing and planning capabilities and completely eliminate effects of superficial restatements; and (i) mathematical reasoning. 3.1 Maze Exploration Figure 2: RL-trained models are less exploratory. We plot how often a node is visited by the distilled and RLVR-trained Qwen3-8B. As a probe, maze solving serves as a rigorous, noise-free proxy for multi-step logical reasoning. Maze traversal preserves the fundamental structure of step-by-step problem-solving—the model must sequentially execute valid, discrete actions to reach a goal state—while stripping away the linguistic confounders that arise from NL reasoning tasks. Because the action space is strictly constrained, alternate trace realizations reflect a shift in the model’s policy. Furthermore, because local trace validity (e.g., avoiding walls) and distance to the goal are trivial to formalize, mazes provide an ideal, controlled environment to investigate how RL-based post-training alters an LLM’s policy. In this study, we employed the Maze Dataset (Ivanitskiy et al., 2023) library to instantiate various deterministic graph traversal tasks, constraining the model’s action space to four directional moves: <left>,<right>,<up>,<down>\ <left>, <right>, <up>, <down>\. 3.2 Exploration in Mathematical Reasoning To study exploration in NL mathematical reasoning, given a question Q, we first generate several reasoning traces by sampling from a diverse set of frontier LLMs, followed by filtering based on ground truth to get “correct” reasoning responses A(1),A(2),…,A(n)A^(1),A^(2),…,A^(n).333n≈90n≈ 90 per question, total API budget of USD 3000. These are then segmented (with the help of an LLM) into individual reasoning steps/nodes aj(i)a^(i)_j, such that A(i)=concat([a1(i),a2(i),…,ani(i)])A^(i)= concat([a^(i)_1,a^(i)_2,…,a^(i)_n_i]). With this set of reasoning nodes, and a notion of semantic similarity sim(aα(i),aβ(j)) sim(a^(i)_α,a^(j)_β), we can create a prefix tree T=(V,E)T=(V,E) of mathematical reasoning traces, called a BODHI-Tree, where each node v=aα(i)∈Vv=\a^(i)_α\∈ V represents a set of semantically equivalent mathematical statements, and an edge u↝v∈Eu v∈ E if every statement in v comes after some statement in u in some reasoning trace A(i)A^(i) (see Figure 1, Algorithm 1). Thus, every distinct path from the root to a leaf in this tree represents an alternate solution to the mathematical reasoning question Q. Creating such a structure allows us to explicitly analyze differences in exploration between different kinds of LLMs. 235235 questions from AIME, alongside ∼20K 20K responses and their corresponding trees constitute our probe dataset BODHI-Tree. Semantic similarity of two segments is gauged using GPT-oss-120b in an LLM-as-a-judge framework. Further details on the construction of BODHI-Trees, like prompts, settings, validation of the LLM-as-a-judge framework, etc., can be found in Appendix A. 3.3 Metrics Since this work focuses on the probability of different continuations, the quantity of interest is candidate preference entropy (CPE). A branch node from BODHI-Tree (or the tree implicit in maze exploration) gives us a prefix a, and at least two alternative completions s(1)s^(1) and s(2)s^(2). Given an LLM PmodelP_ model we can compute: Smodel(s(i)|a)=1|s(i)|∑k=1|s(i)|log(Pmodel(sk(i)|s<k(i),a)) split&S_ model (s^(i)|a )\\ &= 1|s^(i)| _k=1^|s^(i)| (P_model(s^(i)_k|s^(i)_<k,a) )\\ split (1) Which we can normalize to obtain P(s(i)|a)=exp(Smodel(s(i)|a))∑i=1,2exp(Smodel(s(i)|a)),P(s^(i)|a)= (S_ model(s^(i)|a)) _i=1,2 (S_ model(s^(i)|a)), (2) and compute: CPEmodel(s(1),s(2);a)=HmodelBranch(s(1),s(2))=−∑i=1,2P(s(i)|a)log(P(s(i)|a)) split& CPE_ model(s^(1),s^(2);a)=H_ model Branch(s^(1),s^(2))\\ &=- _i=1,2P(s^(i)|a) (P(s^(i)|a) ) split (3) where HmodelBranch(s(1),s(2))H_ model Branch(s^(1),s^(2)) is the conditional entropy in the choice of s(1)s^(1), s(2)s^(2) (or CPE), given prefix a, and sk(i)s^(i)_k is the kkth token of s(i)s^(i). Further, since we want to study differences resulting from post-training, we compute: ΔCPEmodel(s(1),s(2);a)=ΔHmodelBranch(s(1),s(2))=HmodelBranch(s(1),s(2))−HbaseBranch(s(1),s(2)) split& CPE_ model(s^(1),s^(2);a)= H_ model Branch(s^(1),s^(2))\\ &=H_ model Branch(s^(1),s^(2))-H_ base Branch(s^(1),s^(2)) split (4) ΔHmodelBranch H_ model Branch is the difference in conditional preference entropy (or ΔCPE CPE) and measures changes in preferences of branches arising from a certain post-training scheme. 3.4 Models A notable challenge in analyzing differences arising out of RLVR is the dearth of intermediate post-training checkpoints. Popular model vendors usually do not release both the pre-trained and RLVR checkpoints, and even if these were available, the opacity of the training data makes it hard to discern the effects of RLVR. Thus, most of our experiments use models where post-training (distillation and RLVR) was performed by us with controlled matched datasets starting from publicly available pre-training checkpoints. For mazes, we distilled (SFT) HuggingFace’s SmolLM3-3B-Base (Bakouch et al., 2025) and Qwen3-8B-Base (Yang et al., 2025a) on 3030K oracle-generated solutions, followed by RLVR on 1212K samples. To construct (LLMBase LLM_ Base, LLMDistil LLM_ Distil, LLMRLVR LLM_ RLVR) triplets for mathematical reasoning, we selected Google’s gemma-3-12b-pt (Team et al., 2025) alongside three Qwen foundation models: Qwen3-8B-Base, Qwen3.5-9B-Base (Team, 2026), and Qwen2.5-32B-Base (Qwen et al., 2024). These underwent Long-CoT distillation using 5050K samples from the OpenThoughts-114k-math dataset, and RLVR employing the DAPO-Math-17k dataset (Yu et al., 2026). We directly adopted DAPO-Qwen2.5-32B (Yu et al., 2026) as the RL-trained counterpart for Qwen2.5-32B-Base since it is also trained on DAPO-Math-17k. Further experimental details are deferred to Appendix B. 4 Results 4.1 Measuring Test-time Exploration Figure 2 plots the number of times each node in the maze was visited by the RLVR and distillation policies (in 11K generations per maze), and clearly shows that RL-training results in a much less exploratory policy, i.e., fewer states are accessible at test time. We empirically compute the probability of visiting a node using 11K generations for 5050 mazes, and compute its entropy to find out how spread out each distribution is. We find that for Qwen3-8B, average HDistil=2.3380H_Distil=2.3380 (95% CI [2.1815,2.4899]95\% CI [2.1815,2.4899])444Bootstrapped 95%95\% confidence intervals. and HRLVR=1.8138H_RLVR=1.8138 (95% CI [1.6477,1.9755]95\% CI [1.6477,1.9755]). Exploration is significantly reduced in the RL-trained model at test time (ΔH=−0.5242 H=-0.5242, 95% CI [−0.7005,−0.3435]95\% CI [-0.7005,-0.3435], p<0.0001p<0.0001). Results with SmolLM3-3B are similar, and we have: HDistil=1.9288H_Distil=1.9288 (95% CI [1.7938,2.0632]95\% CI [1.7938,2.0632]), HRLVR=1.6092H_RLVR=1.6092 (95% CI [1.4344,1.7756]95\% CI [1.4344,1.7756]), and ΔH=−0.3196 H=-0.3196 (95% CI [−0.4252,−0.2085]95\% CI [-0.4252,-0.2085], p<0.0001p<0.0001). 4.2 Trajectory Preference in RLVR Model ΔCPE CPE Difference % with reduced ΔCPE CPE Qwen3-8B −0.0061-0.0061 [−0.0062,−0.0060][-0.0062,-0.0060] 95.4995.49 [95.07,95.89][95.07,95.89] Qwen3.5-9B −0.0069-0.0069 [−0.0070,−0.0068][-0.0070,-0.0068] 100.0100.0 [100.0,100.0][100.0,100.0] Gemma3-12B −0.0225-0.0225 [−0.0227,−0.0223][-0.0227,-0.0223] 99.9899.98 [99.95,100.0][99.95,100.0] Qwen2.5-32B −0.0345-0.0345 [−0.0349,−0.0340][-0.0349,-0.0340] 99.4699.46 [99.31,99.60][99.31,99.60] Table 1: Effect of post-training on LLM branching. The table shows the change in ΔCPE CPE (ΔCPERLVR−ΔCPEDistil CPE_RLVR- CPE_Distil) between distilled and RLVR models, and the percentage of samples where ΔCPE CPE reduced in the RL-trained model compared to the distilled one. There is a significant drop in ΔCPE CPE (p<0.0001p<0.0001) for almost all samples, suggesting that RL-trained models have significantly stronger trajectory preferences at branch points. 95%95\% CIs with paired bootstrap tests are reported. BODHI-Trees allow for controlled tests to investigate if there are differences in exploration patterns between different policies, and in particular, we can check whether models trained with Long-CoT distillation have different trajectory preferences compared to RLVR-trained models. To this end, we compute ΔCPE(s(1) CPE(s^(1), s(2);a)s^(2);a) (Eq. 4) for 1010K tuples from BODHI-Trees. Our experiment (see Table 1) shows that there is a significant drop in ΔCPE CPE for RL-trained models, i.e., they have lower branching entropy, and this effect is observed in nearly every sample. This indicates that RL-trained models have stronger preferences when choosing between alternate completions at branch points, which reinforces test-time compute scaling results. The distribution is plotted in Figure 9 in Appendix C. 4.3 Exploring Invalid Continuations Figure 3: RL-trained models successfully learn the semantics of mazes. We plot the entropy of the models at hall nodes (degree 2) to see if different training modalities produce models which correctly model the lack of ambiguity at halls. A part of the observed entropy collapse could be explained by the model assigning less probability mass to invalid continuations. To investigate this possibility, we turn to the maze task. We choose 55K “hall” nodes (nodes with degree 2) along maze solution paths, and since there is no ambiguity in the choice of next-token, a semantically faithful policy should have lower entropy at these nodes. We find that (see Figure 3) the RL-trained models demonstrate statistically significant entropy collapse in halls, suggesting that the RLVR policy better delineates between different verifier-equivalence classes. A related notion for mathematical reasoning is “local trace coherence”, i.e., the ability of a model to avoid logical errors within sub-segments of traces, and there is preliminary evidence suggesting that RLVR-trained models perform better in this regard (Samineni et al., 2025). 4.4 Is Entropy Collapse Stylistic? Figure 4: Is the entropy collapse just syntactic? We observe that RLVR-trained models usually have lower entropy (CPE) both when choosing between two syntactic variants of a mathematical statement and two semantically distinct continuations. The entropy collapse in choosing between continuations with different semantics is stronger. The dashed lines show the means of the distributions. To analyze whether the entropy collapse in the RLVR policy arises due to the model developing stronger preferences for trivial mathematical representational choices (such as how variables are named or the order of commutative operations) alone, or also encompasses lack of semantic diversity such as the choice of the mathematical approach to use, we select a branching node α (with at least 2 children), and its two children β and γ from a BODHI-Tree. We pick an anchor trace A from the node α, two continuations a,ba,b from child β that are semantically similar but have different syntax, and a continuation g from child γ which has different semantics from a,ba,b. Then, as in Eq. 4, we can compute ΔCPEmodelSyntax:=ΔCPEmodel(a,b;A) CPE_ model Syntax:= CPE_ model(a,b;A) and ΔCPEmodelSemantics:=ΔCPEmodel(a,g;A) CPE_ model Semantics:= CPE_ model(a,g;A). Since we are interested in the difference, we plot the distribution of the differences (see Figure 4), i.e., ΔCPERLVR∗−ΔCPEDistil∗ _ RLVR^*- _ Distil^*, for 2.52.5K sampled nodes from BODHI-Trees. We observe a drop in ΔCPESyntax CPE Syntax for all RL-trained models compared to their distilled counterparts except for Qwen3.5-9B, where the drop is not statistically significant. However, all RL-trained models in the study have a statistically significant drop in ΔCPESemantics CPE Semantics compared to their distilled counterparts. The difference between them is statistically significant (p<0.02p<0.02 except for Qwen3-8B, p=0.11p=0.11). This suggests that the policy distribution is more collapsed when it comes to making choices between semantically distinct continuations. Thus, the policy collapse cannot be explained by just syntactic/stylistic artifacts, and RL-trained models have a lower propensity for semantically meaningful exploration. This is important because this underlines the fact that RLVR doesn’t just eliminate spurious variations but also legitimate exploration. 5 Discussion 5.1 Constraints, Diversity and RLVR The evidence presented in this paper thus far suggests that RLVR restricts the space of continuations accessible to the model (§4.1), with reduced probability of branching (§4.2) and invalid continuations (§4.3). More importantly, the observed reduction in diversity cannot be attributed to calcification of stylistic choices alone, as there is a greater calcification in the choice of alternate semantically distinct continuations (§4.4). This leads us to ask: does RLVR achieve better sample efficiency (pass@1 pass@1 performance) by learning environmental constraints, but it does so at the cost of trajectory diversity? To establish a more compelling connection between these, we design the following interventions on the decoding strategy in the mazes case: (i) “Legal Only” where invalid next tokens (e.g., moves that result in wall collisions) are masked out, and (i) “Avoid Short Path” where the model is steered away from the shortest path. The Legal Only intervention sets the probability of invalid continuations to zero, and thus if a particular policy already had very low likelihood of selecting these continuations, it will be affected less. Similarly, the Avoid Short Path intervention sets the probability of the optimal continuation to zero (another verifier-equivalent continuation is guaranteed to exist), and thus should have a larger effect on a policy which has stronger trajectory preferences. Table 2 presents our results with the distilled and RL-trained variants of Qwen3-8B (T=1.0) with 500500 mazes. Decoding Policy Training Method Distil RLVR Baseline 7.6±1.27.6± 1.2 13.0±1.513.0± 1.5 Legal Only 62.6±2.262.6± 2.2 39.2±2.239.2± 2.2 Avoid Short Path 3.8±0.93.8± 0.9 4.6±0.94.6± 0.9 No Reversal 8.0±1.28.0± 1.2 13.8±1.513.8± 1.5 Table 2: RLVR restricts accessible states. Figure 5: Environmental constraint-adherence improves during RL-training. We plot the baseline performance (dashed line) of different RL-training checkpoints of Qwen3-8B, and the ratio of performance with the Legal Only decoding strategy to baseline. As predicted by the hypothesis, we see that the Legal Only intervention has only a ∼3× 3× improvement on the RL-trained model, which is dwarfed by the ∼8× 8× improvement on the distilled model. The distilled model with the Legal Only constraint shows a ∼60% 60\% improvement over the RL-trained model, illustrating that a policy which has a flatter distribution amongst verifier-equivalent continuations is more performant. Figure 5 plots the ratio of the performance with the Legal Only intervention and the baseline through the course of RL training. As this measures the magnitude of the effect the Legal Only intervention has, it shows that the policy continually evolves to assign lower probabilities to invalid continuations as training progresses. The Avoid Short Path intervention reduces performance by ∼65% 65\% for the RL-trained model but only by ∼50% 50\% for the distilled model, showing that RL-trained models have ossified preferences, i.e., other valid continuations are less likely to be explored. The RL policy deprioritizes invalid moves to achieve a higher baseline (13%13\%), but suffers significantly owing to a lack of trajectory diversity. 5.2 Backtracking Model Method Original (%)(\%) Distracted (%)(\%) Unchanged (%)(\%) Δ(%) (\%) Acc. [Pass@3] Acc. [Pass@3] Qwen3-8B Distilled 28.76±1.8528.76± 1.85 10.54±1.2610.54± 1.26 7.69±1.097.69± 1.09 +5.52±1.76+5.52± 1.76 RLVR 33.61±1.9333.61± 1.93 17.73±1.5617.73± 1.56 13.21±1.3813.21± 1.38 Qwen3.5-9B Distilled 34.78±1.9534.78± 1.95 9.20±1.189.20± 1.18 7.02±1.047.02± 1.04 +7.36±1.77+7.36± 1.77 RLVR 32.11±1.9132.11± 1.91 25.08±1.7725.08± 1.77 14.38±1.4314.38± 1.43 Gemma3-12B Distilled 23.58±1.7423.58± 1.74 11.54±1.3111.54± 1.31 9.03±1.179.03± 1.17 +4.18±1.81+4.18± 1.81 RLVR 26.42±1.8026.42± 1.80 16.22±1.5116.22± 1.51 13.21±1.3813.21± 1.38 Qwen2.5-32B Distilled 70.40±1.8770.40± 1.87 60.03±2.0060.03± 2.00 52.68±2.0452.68± 2.04 +29.09±2.58+29.09± 2.58 RLVR 85.95±1.4285.95± 1.42 86.06±1.3886.06± 1.38 81.77±1.5881.77± 1.58 Table 3: RLVR models have improved ability to recover from dead-ends. The Unchanged column lists the proportion of samples for which the distraction intervention did not change the final answer, and the Δ column shows the change between the two policies. During reasoning, a model may reach a state where further progress is impossible, and a sound reasoning model must be able to backtrack out of these “dead-ends” to a valid state and explore alternate trajectories. To test this in mazes, as in §5.1, we design the “No Reversal” intervention, where two consecutive moves that are opposites of each other (e.g., <up> followed by <down>) are masked out. Our results show that (see Table 2) the No Reversal intervention does not have a statistically significant effect on either variant. However, this does not imply that the models cannot backtrack, as it could also be explained by the fact that the policies do not reach dead-ends in the first place. To investigate this further, we resort to prefix steering to evaluate policies, and ask: what trajectories are available to the model at dead-ends? For the maze case, we sampled 2.52.5K dead-end states—defined as states where the goal is only reachable if the last move is reversed—and measured the probability of the backtracking move (pbp_b). Our results indicate that RLVR training significantly enhances backtracking capabilities compared to distillation. For SmolLM3-3B, the average backtracking probability increases from [pb]=0.0647±0.0008E[p_b]=0.0647± 0.0008 (Distil) to 0.1687±0.00350.1687± 0.0035 (RLVR) (Δ=0.1040, 95% CI [0.0978,0.1101],p<0.0001 =0.1040,\,95\% CI [0.0978,0.1101],\,p~<~0.0001). Qwen3-8B demonstrates a similar improvement, rising from 0.0068±0.00100.0068± 0.0010 to 0.1537±0.00620.1537± 0.0062 (Δ=0.1469, 95% CI [0.1348,0.1589],p<0.0001 =0.1469,\,95\% CI [0.1348,0.1589],\,p~<~0.0001). Furthermore, RLVR reduces absolute failures (pb=0p_b=0), with zero-probability instances dropping from 2422→18242422→ 1824 for Qwen-3-8B and 476→357476→ 357 for SmolLM-3-3B. The distribution is plotted in Figure 10 in Appendix C. Analyzing dead-ends in mathematical reasoning is less straightforward, because unlike mazes, the valid action space is unconstrained. To simulate this, we utilize a partial mathematical reasoning trajectory (a non-leaf node from a BODHI-Tree) and append a “distractor”—a spurious reasoning step sampled from a different tree—intended to derail generation (see Figure 8). We then evaluate whether the model can still arrive at the correct final answer, which measures the model’s ability to recover from missteps in reasoning. As shown in Table 3, RLVR-trained models exhibit a significantly enhanced capability to backtrack compared to their distilled counterparts. Since trajectories that do not involve backtracking are verifier-invalid, these results also suggest that the RLVR policy better delineates between valid and invalid continuations. 6 Conclusions We investigated the nature of the policy shift arising from RLVR training through controlled experiments using maze traversal and our mathematical exploration trees (BODHI-Trees). We establish that RL-trained models have stronger trajectory preferences at branch points, and that this is not merely an artifact of stylistic or syntactic preferences. RLVR policies show a significant collapse in candidate preference entropy, suggesting that they prune semantically distinct, verifier-equivalent trajectories. While the policy concentration restricts trajectory diversity, it also enhances the model’s adherence to environmental constraints and ability to backtrack. The improved trace validity is likely responsible for RLVR’s success, but it comes at the cost of genuine trajectory diversity. Future work must address this limitation to push the boundaries of reasoning capabilities further. Limitations Mathematical Reasoning Trees The BODHI-Tree construction uses an LLM-as-a-judge framework with GPT-oss-120b to gauge semantic equivalence, which might introduce some errors. We performed spot checks and inter-annotator agreement tests with more capable frontier LLMs such as deepseek-v4-pro and OpenAI’s gpt-5.6-luna, finding near-perfect agreement (see Appendix A); however, some errors may remain. We could not perform rigorous human evaluations owing to resource constraints. Domain Specificity The presented empirical evidence is limited to deterministic environments (mazes) and mathematical reasoning (AIME). Since we study exploration, this involves sampling a large number of continuations from frontier LLMs which is prohibitively expensive for some domains (USD 3000 for collecting AIME traces). Future work that extends this to domains such as code generation would be compelling. Ethics Statement In keeping with ACL ethical guidelines, all scientific artifacts generated for this study—including code, prompts, data, and raw model outputs—are made freely available as open source under the MIT license. Only public datasets available on the Huggingface platform were used in the study. AI assistants were not used in ideation, coding, or writing involved in this work, and their usage was limited to copyediting tasks (e.g., checking spelling, grammar, tone). We do not foresee any potential ethical, societal, or environmental risks from this work. References Anschel et al. (2025) Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, and Gerard Medioni. 2025. Group-aware reinforcement learning for output diversity in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32394–32415, Suzhou, China. Association for Computational Linguistics. Bakouch et al. (2025) Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, and 4 others. 2025. SmolLM3: smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3. Cai et al. (2026) Pengxiang Cai, Tianchen Fang, Xiaohan Li, Qingyuan Zeng, Guocong Li, and Jintai Chen. 2026. Curriculum reinforcement learning can incentivize reasoning capacity in llms beyond the base model. Preprint, arXiv:2606.22317. Chen et al. (2026a) Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aris Sotiras, Yalin Wang, and Abolfazl Razi. 2026a. DRA-GRPO: Your GRPO needs to know diverse reasoning paths for mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, pages 13995–14019, San Diego, California, United States. Association for Computational Linguistics. Chen et al. (2026b) Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Xin Zhao, and Guang Shi. 2026b. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models. Cheng et al. (2026) Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. 2026. Reasoning with exploration: An entropy perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30377–30385. DeepSeek-AI et al. (2026) DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, and 300 others. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence. Preprint, arXiv:2606.19348. Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638. Hao et al. (2026) Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. 2026. Rethinking entropy interventions in RLVR: An entropy change perspective. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31105–31133, San Diego, California, United States. Association for Computational Linguistics. He et al. (2026) Xu He, Jialiang Guo, Fucheng Xiong, Haodong Zhao, Xingyang Li, Ke Zeng, and Xunliang Cai. 2026. VANE: Guiding high-value exploration in RLVR via outcome-process novelty shaping. In Findings of the Association for Computational Linguistics: ACL 2026, pages 28721–28739, San Diego, California, United States. Association for Computational Linguistics. Hu et al. (2025) Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Hu et al. (2026a) Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, and Bryan Hooi. 2026a. Rewarding the rare: Uniqueness-aware RL for creative problem solving in LLMs. In Findings of the Association for Computational Linguistics: ACL 2026, pages 39765–39790, San Diego, California, United States. Association for Computational Linguistics. Hu et al. (2026b) Zican Hu, Shilin Zhang, Yafu Li, Jianhao Yan, Xuyang Hu, Leyang Cui, Xiaoye Qu, Chunlin Chen, Yu Cheng, and Zhi Wang. 2026b. Diversity-incentivized exploration for versatile reasoning. In The Fourteenth International Conference on Learning Representations. Huang et al. (2026) Fanding Huang, Guanbo Huang, Xiao Fan, Yi He, Xiao Liang, Xiao Chen, Qinting Jiang, Faisal Nadeem Khan, Jingyan Jiang, and Zhi Wang. 2026. Semantic-space exploration and exploitation in RLVR for LLM reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, pages 38402–38449, San Diego, California, United States. Association for Computational Linguistics. Ivanitskiy et al. (2023) Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F. Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung. 2023. A configurable library for generating and manipulating maze datasets. Preprint, arXiv:2309.10498. Jang et al. (2026) Jaeeun Jang, Hansle Lee, and Sangmin Kim. 2026. A few bad apples spoil the bunch: Preventing global entropy collapse driven by a small set of tokens in LLM reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, pages 13134–13154, San Diego, California, United States. Association for Computational Linguistics. Jiang et al. (2025) Gangwei Jiang, Yahui Liu, Zhaoyi Li, Wei Bi, Fuzheng Zhang, Linqi Song, Ying Wei, and Defu Lian. 2025. What makes a good reasoning chain? uncovering structural patterns in long chain-of-thought reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6490–6514, Suzhou, China. Association for Computational Linguistics. Jin et al. (2026) Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, and Deyi Xiong. 2026. Revisiting entropy in reinforcement learning for large reasoning models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25300–25322, San Diego, California, United States. Association for Computational Linguistics. Li and Li (2026) Wendi Li and Sharon Li. 2026. LAD: Learning advantage distribution for reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, pages 14327–14348, San Diego, California, United States. Association for Computational Linguistics. Li et al. (2025) Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang. 2025. Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling. Preprint, arXiv:2508.17445. Liu et al. (2025) Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. Preprint, arXiv:2503.20783. Mishra et al. (2026) Kshitij Mishra, Nils Lukas, and Salem Lahlou. 2026. SD-e2: Semantic exploration for reasoning under token budgets. In Findings of the Association for Computational Linguistics: EACL 2026, pages 6144–6157, Rabat, Morocco. Association for Computational Linguistics. OpenAI et al. (2025) OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. OpenAI (2025) OpenAI. 2025. OpenAI o3 and o4-mini system card. OpenAI (2026) OpenAI. 2026. GPT-5.6: Frontier intelligence that scales with your ambition. Qwen et al. (2024) Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. Qwen2.5 technical report. ArXiv preprint, abs/2412.15115. Saha et al. (2026) Soumadeep Saha, Akshay Chaturvedi, Saptarshi Saha, Utpal Garain, and Nicholas Asher. 2026. KisMATH: Do LLMs have Knowledge of Implicit Structures in mathematical reasoning? Transactions of the Association for Computational Linguistics, 14:1308–1328. Samineni et al. (2025) Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, and Subbarao Kambhampati. 2025. Local coherence or global validity? investigating rlvr traces in math domains. Preprint, arXiv:2510.18176. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300. Team et al. (2025) Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. Team (2026) Qwen Team. 2026. Qwen3.5: Accelerating productivity with native multimodal agents. von Werra et al. (2020) Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. TRL: Transformers Reinforcement Learning. Wang et al. (2025) Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. Wen et al. (2026) Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. 2026. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs. In The Fourteenth International Conference on Learning Representations. Yang et al. (2025a) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388. Yang et al. (2025b) Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. 2025b. Treerpo: Tree relative policy optimization. Preprint, arXiv:2506.05183. Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems. Yu et al. (2026) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others. 2026. DAPO: An open-source LLM reinforcement learning system at scale. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Yuan et al. (2026) Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li, Lei Feng, Dadong Wang, Tao Xiang, Tongliang Liu, and Bo An. 2026. Understanding diversity collapse in rlvr via the lens of overtraining. Preprint, arXiv:2606.15455. Yue et al. (2025) Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In 2nd AI for Math Workshop @ ICML 2025. Zhao et al. (2026) Ziqi Zhao, Zhaochun Ren, Jiahong Zou, Liu Yang, Zhiwei Xu, Xuri Ge, Zhumin Chen, Xinyu Ma, Daiting Shi, Shuaiqiang Wang, Dawei Yin, and Xin Xin. 2026. Reinforced efficient reasoning via semantically diverse exploration. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47994–48007, San Diego, California, United States. Association for Computational Linguistics. Zheng et al. (2025) Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. 2025. First return, entropy-eliciting explore. Preprint, arXiv:2507.07017. Appendix A BODHI-Tree Construction To construct BODHI-Trees, we start with 304304 questions from AIME as in Saha et al. (2026) and sample responses from a diverse set of LLMs, namely OpenAI o3-2025-04-16, OpenAI o4-mini-2025-04-16 (OpenAI, 2025), DeepSeek r1-0528 (Guo et al., 2025), and Qwen’s qwen3-235b-a22b-07-25 (Yang et al., 2025a), with T=1.0T=1.0 and reasoning effort set to the highest available setting. The number of samples was decided by the API budget, which in this case totaled USD 3000. All responses with incorrect answers were removed and this resulted in ∼20 20K responses to 235235 questions, with an average of ∼85 85 responses per question. These were then split into reasoning segments using OpenAI o4-mini-2025-04-16. Given a question Q, a set of “correct” reasoning traces A(1),A(2),…,A(n)A^(1),A^(2),…,A^(n), and their corresponding segmentations aj(i)a^(i)_j, such that A(i)=concat([a1(i),a2(i),…,ani(i)])A^(i)= concat([a^(i)_1,a^(i)_2,…,a^(i)_n_i]), we employ Algorithm 1 to create BODHI-Trees. Algorithm 1 BODHI-Tree Construction 1:Given a question Q. 2:Given A(i)=concat([a1(i),a2(i),…,ani(i)])A^(i)~=~ concat([a^(i)_1,a^(i)_2,…,a^(i)_n_i]) for i∈1,…ni∈\1,… n\. 3:Given M(x,y)M(x,y) an LLM-based matching function, true if y is equivalent to x, or equivalent to a part of x (subsumed in x). 4:Initialize T with a single root node vrootv_ root 5:for i∈1…ni∈ 1… n do 6: z[i]←vrootz[i]← v_ root 7:end for 8: 9:while ∃A(i)∃ A^(i) with remaining segments do 10:# Try to consume a segment into a node. 11: for Each A(i)A^(i) with remaining segments do 12: shead←dequeue(A(i));v←z[i]s_ head← dequeue(A^(i));v← z[i] 13: if M(v,shead)M(v,s_ head) is true then 14: Consume sheads_ head in v 15: else 16: requeue(shead,A(i)) requeue(s_ head,A^(i)) 17: end if 18: end for 19: 20:# Move trace to child node or fork. 21: for Each A(i)A^(i) with remaining segments do 22: shead←dequeue(A(i));v←z[i]s_ head← dequeue(A^(i));v← z[i] 23: matched←false matched← false 24: for Each child vchildv_ child of v in T do 25: if M(vchild,shead)M(v_ child,s_ head) is true then 26: z[i]←vchildz[i]← v_ child 27: Consume sheads_ head in vchildv_ child 28: matched←true matched← true 29: break 30: end if 31: end for 32: if matched is false then 33: Create new node vnewv_new 34: Add edge v↝vnewv v_new 35: z[i]←vnewz[i]← v_ new 36: Consume sheads_ head in vnewv_ new 37: end if 38: end for 39:end while 40:return T Algorithm 1 starts with a root node, and each trace is assigned to this root node (z[i]←vrootz[i]← v_ root). At each iteration, for every trace we try to consume its first available segment into its assigned node, failing which we either try to match it with one of its children or spawn a new branch. If the head of the trace matches any of the children, we consume it at the child, and assign the rest of the trace to the child. If no such match is found, a new child is spawned, consuming the head segment, and the rest of the trace is assigned here. The algorithm terminates when no trace has any unassigned segments. The resulting tree (BODHI-Tree) has the property that the lowest common ancestor of two traces represents their longest shared equivalent reasoning prefix. The matching function M(x,y)M(x,y) is instantiated with OpenAI GPT-oss-120b (OpenAI et al., 2025) and checks if x is mathematically equivalent to y or y is mathematically equivalent to a part of x. This helps align traces with differing levels of granularity. In practice, when consuming a segment into a node, we pick a representative of the node to match against the segment. The LLM is sampled at T=1T=1, topp=0.99 top_p=0.99, reasoning_effort = medium, using 4-shot prompts. A total of 470~470K LLM calls are made for the construction of the BODHI-Trees (∼2 2K calls per BODHI-Tree). We also measured the inter-annotator agreement between GPT-oss-120b and more capable frontier LLMs, namely deepseek-v4-pro (DeepSeek-AI et al., 2026) and gpt-5.6-luna (OpenAI, 2026), and the Cohen’s κ scores are 0.8150.815 and 0.8370.837, respectively, suggesting that there is near-perfect agreement. The prompt is given in Figure 6, and some example trees are given in Figure 7. <system> # Identity You are an expert mathematician who gauges similarity of mathematical statements. Help the user detect similarity in statements by answering yes/no. # Instructions * ONLY answer with ’yes’ or ’no’. Do not produce any extra text. * The user will provide two statements Statement A and Statement B. * Answer ’yes’ if and only if Statement B is completely equivalent to Statement A or Statement B is equivalent to a part of Statement A. * Answer ’no’ if the statements are not equivalent, if Statement A is contained in Statement B, or if they have different mathematical meaning. * Ignore superficial elements like names of variables, whitespace, synonyms, phrasing, etc. * The only important aspect is the mathematical meaning of the statements. # Examples … </system> <user> # Statement A: The equation of the circle is given by x2+y2=5x^2+y^2=5. # Statement B: The circle’s equation is z2+y2=5z^2+y^2=5. </user> <assistant> yes </assistant> Figure 6: System prompt to instantiate the M(x,y)M(x,y) function in Algorithm 1. Figure 7: Examples of extracted BODHI-Trees. Appendix B Experimental Details B.1 Training For math, we performed Long-CoT distillation employing 5050K randomly chosen samples from the OpenThoughts-114k-math dataset. We perform Q-LoRA (r=64,α=128r=64,α=128; Dettmers et al. (2023)) fine-tuning for 11 epoch (lr =5e−4=5e-4) with max_tokens =16=16K, batch size of 6464, a cosine learning rate scheduler with warm-up (5%5\%), and AdamW optimizer. RL-training was performed with the TRL library (von Werra et al., 2020) using the DAPO-Math-17k dataset (Yu et al., 2026) for 11K steps with lr =1e−6=1e-6, group size of 88, batch size of 4848, and max_tokens =4=4K. Following suggestions by Yu et al. (2026), we set ϵlow=0.2,ϵhigh=0.28 _low=0.2, _high=0.28, and following suggestions by Liu et al. (2025); Yu et al. (2026); Hu et al. (2025), we set β=0β=0. We employ the Dr. GRPO loss (Liu et al., 2025) to avoid length-based biases. We use the Math-Verify library to compute accuracy rewards, and use a format reward which ensures that <think> tags are present and there is exactly one containing the answer. Length-based rewards are not employed. Qwen3.5-9B, like DAPO-Qwen2.5-32B, was trained with Zero-RL, whereas for Qwen3-8B and gemma3-12b-pt we perform distillation followed by RLVR. The chat templates for the distilled and RL models are matched, and are distributed alongside the model. All training datasets are de-duplicated against the probing set. For mazes, we performed SFT with oracle-generated traces employing the Maze Dataset by Ivanitskiy et al. (2023). We used 3030K mazes with sizes 5×5, 7×7, 9×95× 5,\>7× 7,\>9× 9 and similar settings as described above (except max_tokens =2=2K). For RL-training, 1212K mazes were used with similar settings as described above (except group size 1616, and max_tokens=1.5=1.5K). We used a custom verifier to check correctness, and a format reward to ensure that the predicted moves were parsable. Five tokens were added to the tokenizer before the distillation stage: they are <left>, <right>, <up>, <down> (for predicting a path), and <--> (to denote an edge while express the maze as an adjacency list). RL-training was performed with 4×44× 4 H100 9393GB GPUs, and all other experiments used 1×41× 4 H100s. B.2 Evaluation For results in §4.1, we used 5050 mazes and sampled 11K generations from the models at T=1.0T=1.0 and top_k =10=10. For results in §4.2, §4.3, and §4.4, we directly employ the logits, and we truncate candidate continuations to 2020 tokens for CPE calculation (see Eq. 4). The results in §5.1 and §5.2 are sampled at T=1.0T=1.0. Unless explicitly mentioned, all sampling parameters are at default values. An example distractor for the experiment in §5.2 is given in Figure 8. <system> You are a helpful AI assistant. Assist the user with their math query. The final answer must be in . </system> <user> Question: For non-negative integers a and b with a+b≤6a+b≤ 6, let T(a,b)=(6a)(6b)(6a+b)T(a,b)= 6a 6b 6a+b. Let S denote the sum of all T(a,b)T(a,b), where a and b are non-negative integers with a+b≤6a+b≤ 6. Find the remainder when S is divided by 10001000. </user> <assistant> # Added prefix, BODHI-Tree segments. Let S=∑a=06∑b=06−a(6a)(6b)(6a+b)S= _a=0^6 _b=0^6-a 6a 6b 6a+b. Rewrite with k=a+bk=a+b. S=∑k=06(6k)∑a=0k(6a)(6k−a)S= _k=0^6 6k _a=0^k 6a 6k-a ∑a=0k(6a)(6k−a) _a=0^k 6a 6k-a is the coefficient of xkx^k in (1+x)6(1+x)6=(1+x)12(1+x)^6(1+x)^6=(1+x)^12, so it equals (12k) 12k. S=∑k=06(6k)(12k)S= _k=0^6 6k 12k Compute each term: k=0: 1k=0:\,1 k=1: 6⋅12=72k=1:\,6· 12=72 k=2: 15⋅66=990k=2:\,15· 66=990 k=3: 20⋅220=4400k=3:\,20· 220=4400 k=4: 15⋅495=7425k=4:\,15· 495=7425 k=5: 6⋅792=4752k=5:\,6· 792=4752 Solving gives c=4±16+2406=4±166c= 4± 16+2406= 4± 166, so c=103c= 103 or c=−2c=-2. # Model continues... k=6:(66)(126)=1⋅924=924…k=6:\, 66 126=1· 924=924… Figure 8: Example distractor (red) for experiment in §5.2. Appendix C Additional Results Figure 9: Effect of post-training on LLM branching. (top) We plot the distribution of ΔCPE CPE as described in §4.2 for distilled and RLVR-trained models. There is a significant leftward shift in the distribution of ΔCPE CPE suggesting that RL-trained models have enhanced preference at branch points. The (bottom) row shows a zoomed-in view of the left 20-%ile tail of each distribution. Figure 10: RL-trained models are more likely to backtrack. We plot the distribution of probability of backtracking (pbp_b) as described in §5.2.