Paper deep dive
Contrastive Branch Policy Optimization
Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/29/2026, 4:19:41 AM
Summary
The paper introduces Contrastive Branch Policy Optimization (CBPO), a reinforcement learning method for fine-grained credit assignment in tool-integrated language models. CBPO disentangles branch selection from credit assignment by using generation entropy to screen candidate branch positions and applying path/node-level decay to distribute a fixed rollout budget. It defines Contrastive Branch Value (CBV) based on reward variation within exact-prefix groups to rescale continuation advantages, ensuring non-overlapping credit segments and avoiding duplicated gradients. Experiments on ten benchmarks demonstrate that CBPO outperforms state-of-the-art methods in mathematical reasoning and knowledge-intensive search.
Entities (7)
Relation Signals (6)
Ying Wang → affiliatedwith → Alibaba Group
confidence 95% · Ying Wang Affiliation: Alibaba Group, Hangzhou, China
Changlin Qiu → affiliatedwith → Harbin Institute of Technology
confidence 95% · Changlin Qiu ... Affiliation: Harbin Institute of Technology, Harbin, China
CBPO → defines → CBV
confidence 95% · the reward variation within this controlled group defines the Contrastive Branch Value (CBV)
CBPO → uses → Generation Entropy
confidence 95% · Generation entropy screens candidate branch positions across the entire response
CBPO → appliesto → RLVR
confidence 90% · CBPO provides a practical solution for fine-grained credit assignment in tool-integrated agent training... Reinforcement learning with verifiable rewards (RLVR) enables language models
CBPO → outperforms → GRPO
confidence 90% · CBPO consistently outperforms state-of-the-art policy-optimization and branch-based methods
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning with verifiable rewards (RLVR) enables language models to learn multi-turn interaction with external tools, yet its sparse outcome rewards provide no signal for identifying which intermediate decisions are responsible for success. Branch sampling induces local comparisons among alternative continuations, but existing methods tend to conflate two distinct problems: allocating a fixed rollout budget and translating branch outcomes into token-level credit. We introduce Contrastive Branch Policy Optimization (CBPO), which disentangles these two problems and assigns a dedicated mechanism to each. Generation entropy screens candidate branch positions across the entire response, while path-level and node-level decay distribute a fixed budget across trajectories and positions to prevent exploration from collapsing onto a few paths or adjacent tokens. A parent trajectory together with the branches that share an identical token prefix forms an exact-prefix group, and the reward variation within this controlled group defines the Contrastive Branch Value (CBV), an outcome-based estimate of local decision sensitivity that rescales continuation advantages without altering their sign. When multiple nodes are selected along the same trajectory, CBPO partitions it into non-overlapping credit segments, thereby avoiding duplicated gradients on shared tokens. Requiring only outcome rewards and no process-level annotation, CBPO provides a practical solution for fine-grained credit assignment in tool-integrated agent training. Extensive experiments on ten benchmarks, including five for mathematical reasoning and five for knowledge-intensive search, show that CBPO consistently outperforms state-of-the-art policy-optimization and branch-based methods, attaining the highest macro-average accuracy in both domains and across two model scales.
Tags
Links
- Source: https://arxiv.org/abs/2608.24300v1
- Canonical: https://arxiv.org/abs/2608.24300v1
Trouble viewing inline? Open PDF directly →
Full Text
60,805 characters extracted from source content.
Expand or collapse full text
@authorgroups =1 Contrastive Branch Policy Optimization CCS: Computing methodologies Reinforcement learningCCS: Computing methodologies Intelligent agents Ying Wang Affiliation: Alibaba Group, Hangzhou, China Affiliation: Harbin Institute of Technology, Harbin, China email: 24s005080@stu.hit.edu.cn , Changlin Qiu Note: Corresponding author: Changlin Qiu (qiuchanglin.qcl@taobao.com). Affiliation: Alibaba Group, Hangzhou, China email: qiuchanglin.qcl@taobao.com , Bang Lin Affiliation: Alibaba Group, Hangzhou, China email: linbang.lb@taobao.com , Linbo Jin Affiliation: Alibaba Group, Hangzhou, China email: yuyi.jlb@alibaba-inc.com , Wen Jiang Affiliation: Alibaba Group, Hangzhou, China email: wen.jiangw@alibaba-inc.com , Zhe Sun Affiliation: Alibaba Group, Hangzhou, China email: sz481403@alibaba-inc.com and Jingli Yang Affiliation: Harbin Institute of Technology, Harbin, China email: jinglidg@hit.edu.cn Abstract. Reinforcement learning with verifiable rewards (RLVR) enables language models to learn multi-turn interaction with external tools, yet its sparse outcome rewards provide no signal for identifying which intermediate decisions are responsible for success. Branch sampling induces local comparisons among alternative continuations, but existing methods tend to conflate two distinct problems: allocating a fixed rollout budget and translating branch outcomes into token-level credit. We introduce Contrastive Branch Policy Optimization (CBPO), which disentangles these two problems and assigns a dedicated mechanism to each. Generation entropy screens candidate branch positions across the entire response, while path-level and node-level decay distribute a fixed budget across trajectories and positions to prevent exploration from collapsing onto a few paths or adjacent tokens. A parent trajectory together with the branches that share an identical token prefix forms an exact-prefix group, and the reward variation within this controlled group defines the Contrastive Branch Value (CBV), an outcome-based estimate of local decision sensitivity that rescales continuation advantages without altering their sign. When multiple nodes are selected along the same trajectory, CBPO partitions it into non-overlapping credit segments, thereby avoiding duplicated gradients on shared tokens. Requiring only outcome rewards and no process-level annotation, CBPO provides a practical solution for fine-grained credit assignment in tool-integrated agent training. Extensive experiments on ten benchmarks—five for mathematical reasoning and five for knowledge-intensive search—show that CBPO consistently outperforms state-of-the-art policy-optimization and branch-based methods, attaining the highest macro-average accuracy in both domains and across two model scales. Keywords: reinforcement learning, large language models, tool-integrated reasoning, credit assignment 1. Introduction Chain-of-thought prompting (Wei and others, 2022) and reinforcement learning with verifiable rewards (RLVR) (Schulman and others, 2017; Shao and others, 2024) have substantially improved the reasoning capabilities of large language models (LLMs). Many tasks, however, require current information, exact computation, or feedback from an external environment. Addressing such tasks entails multi-turn interaction with search engines, code interpreters, or both (Yao and others, 2023; Gou and others, 2024). Recent systems apply reinforcement learning to these interactions (Feng and others, 2026; Li et al., 2025), including settings that require coordinated use of multiple tools (Dong and others, 2025b). The resulting training signal nevertheless remains coarse. Most RLVR and agentic reinforcement-learning methods assign a single terminal reward to every generated token in a trajectory. Such uniform credit cannot distinguish a decisive tool request or correction from text that merely precedes a successful answer. Process supervision (Lightman and others, 2024) and step-wise preference optimization (Lai and others, 2024) provide finer signals but require intermediate labels or preferences. Tree-based sampling instead compares continuations from a shared history using only outcome rewards (Hou et al., 2025). This approach, however, still requires a principled policy for locating branches under a limited budget and mapping their outcomes to local policy updates. Existing methods address these requirements only partially. GIGPO and Tree-GRPO construct local advantages from shared or tree-structured histories (Feng et al., 2025; Ji et al., 2026), whereas ARPO uses generation entropy to select branch points after tool feedback (Dong and others, 2025a). Entropy provides an inexpensive proxy for token-level uncertainty (Lin et al., 2024; Cheng et al., 2025; Wang and others, 2025), but does not measure outcome sensitivity. Allocation based solely on entropy can concentrate branches at adjacent positions, overlook consequential decisions outside tool boundaries, or emphasize variation that leaves the final answer unchanged. Contrastive Branch Policy Optimization (CBPO) separates candidate discovery from credit assignment. CBPO scans local entropy throughout the response and applies path-level and node-level decay to distribute a fixed branch budget. A parent trajectory and branches sharing an identical token prefix form an exact-prefix group. Reward variation within this controlled group defines Contrastive Branch Value (CBV), an outcome-based estimate of local decision sensitivity. A bounded form of CBV changes the magnitude, but not the sign, of continuation advantages. Copied prefixes receive no branch gradient, and multiple branch nodes partition a parent trajectory into non-overlapping credit segments. The two signals therefore serve distinct functions: entropy identifies alternative continuations, whereas observed outcome variation estimates their task relevance. The evaluation spans two model scales and includes five tool-augmented mathematical benchmarks and five knowledge-intensive search benchmarks. CBPO consistently outperforms state-of-the-art policy-optimization and branch-based methods, attaining the highest macro average in both domains. The main contributions are as follows: • Branch selection is formulated as budgeted exploration over the entire response. Fixed-interval entropy windows expose candidates beyond tool boundaries, while path-level and node-level decay prevent allocation from collapsing onto a few trajectories or adjacent positions. • CBV estimates local decision sensitivity from reward variation within exact-prefix groups. Standardized and bounded CBV modulates continuation advantages without changing their signs. Prefix masking and non-overlapping segmentation prevent repeated credit on shared tokens. • Experiments across ten benchmarks and two model scales show that CBPO consistently outperforms strong policy-optimization and branch-based baselines. Cross-scale ablations further attribute these gains to balanced branch allocation and outcome-contrastive credit assignment. 2. Related Work 2.1. Fine-Grained Credit Assignment PPO optimizes a policy from trajectory-level returns (Schulman and others, 2017). GRPO-based reasoning systems likewise derive one advantage from each completed response (Shao and others, 2024; Guo and others, 2025). Subsequent variants improve optimization stability or scaling (Yu and others, 2025; Hu and others, 2025), while GSPO moves the optimization unit from tokens to sequences (Zheng et al., 2025). These methods train directly from outcome rewards but do not distinguish decisive intermediate choices from weakly related transitional text. Problem decomposition, computational correction, tool selection, and answer verification can contribute differently to a multi-step solution. Assigning a single advantage may therefore reinforce effective decisions and incidental behavior together. Process rewards (Lightman and others, 2024) and step-wise preference optimization (Lai and others, 2024) supervise intermediate reasoning more directly. Preference objectives such as DPO (Rafailov et al., 2023) and token-entropy methods (He et al., 2026) offer alternative local signals. Process labels and learned value estimates can provide dense feedback but entail annotation costs or estimation error. Entropy requires neither annotation nor learned value estimation, yet measures predictive uncertainty rather than a position’s empirical association with the final reward. Tree-based methods reuse prefixes and sample several continuations from intermediate states, enabling local comparisons at controlled cost. Online variants derive advantages from current-policy descendants, within-tree contrasts, or recurring histories (Hou et al., 2025; Ji et al., 2026). Entropy can identify uncertain candidates at low computational cost (Lin et al., 2024; Cheng et al., 2025). Moreover, a small fraction of high-entropy tokens can dominate effective updates (Wang and others, 2025). EAPO combines reward polarity with token entropy. AEPO balances entropy during rollout and optimization while limiting repeated branches at consecutive high-entropy positions (He et al., 2026; Dong et al., 2025). These approaches use uncertainty to guide optimization, but divergent token distributions need not yield divergent outcomes. CBPO instead restricts entropy to candidate screening and estimates local credit from outcome contrasts among continuations sharing an identical prefix. 2.2. Tool-Integrated Reasoning External tools allow language models to access current information, perform exact computation, and act on an environment (Yao and others, 2023; Lin and Xu, 2025). Tool-integrated reasoning extends beyond the binary decision to invoke a tool. A model must select a tool, construct the request, determine when to invoke it, verify the response, integrate the result, and terminate appropriately. Search provides open-world evidence, whereas code interpreters support calculation and programmatic verification. Coordinating both requires repeated transitions between internal reasoning and external actions as new observations arrive. Work in this area has progressed from prompted reasoning and acting (Yao and others, 2023; Gou and others, 2024) to reinforcement learning for search (Jin et al., 2025; Sun et al., 2025), code interpreters (Feng and others, 2026; Li et al., 2025), and multi-tool coordination (Dong and others, 2025b). Recent WSDM studies train autonomous programmatic agents and optimize tool selection with reinforcement learning (Jiang et al., 2026; Zhang et al., 2026a). Unlike prompting or supervised imitation, reinforcement learning can optimize the complete interaction from environmental feedback and terminal outcomes. In long trajectories, however, success may depend on pre-call reasoning, request construction, feedback interpretation, or post-call synthesis. Tool-return boundaries therefore capture only a subset of potentially decisive positions. Knowledge-intensive tasks also depend on retrieval quality and timing. Adaptive retrieval can extend beyond an initial ranking when the first-stage candidate pool has insufficient recall (Rathee et al., 2025). In LLM-based fact-checking, adaptive evidence retrieval allows a model to determine when internal knowledge is inadequate or conflicts with external evidence (Zhang et al., 2026b). These WSDM studies concern retrieval and verification rather than token-level credit assignment, but reveal the same structural challenge. An agent must determine where additional computation or evidence can alter the final outcome. Long-horizon tool learning often improves planning or attribution through reward shaping, process supervision, or step-level optimization. Intermediate supervision is costly, while predefined states and step boundaries constrain the granularity of credit assignment. GIGPO derives step-level advantages from shared histories. ARPO branches after tool feedback and assigns the resulting advantage to the suffix (Feng et al., 2025; Dong and others, 2025a). The former depends on repeated history states, and the latter treats tool returns as fixed branch boundaries. CBPO searches the full response for branch candidates, then attributes local credit by comparing outcomes under an identical prefix. 3. Preliminaries Tool-integrated rollout. Given a problem x drawn from dataset D and a tool set T, policy πθ _θ generates an interaction trajectory τ that interleaves model tokens, tool requests, and environmental observations. Let z=(z1,…,z|z|)z=(z_1,…,z_|z|) collect the model-generated tokens of τ, and let o<to_<t denote the tool observations available before position t. Trajectory generation then factorizes autoregressively over model tokens only: (1) Pθ(τ∣x;)=∏t=1|z|πθ(zt∣x,z<t,o<t;).P_θ(τ x;T)= _t=1^|z| _θ\! (z_t x,z_<t,o_<t;T ). Observations are inserted by the environment upon each tool call; they condition subsequent generation but contribute neither likelihood terms nor policy gradients. Upon termination, a verifier assigns a bounded outcome reward R(τ)R(τ). Following agentic reinforcement learning formulations (Dong and others, 2025a; Dong et al., 2025), training maximizes the KL-regularized expected reward: (2) maxπθx∼,τ∼πθ(⋅∣x;)[R(τ)]−βDKL(πθ∥πref), _ _θ\;E_x ,\,τ _θ(· x;T) [R(τ) ]-β\,D_KL\! ( _θ\,\|\, _ref ), where πref _ref is a frozen reference policy and β controls the regularization strength. Shared-history, suffix-based, and tree-search agent RL methods adopt this trajectory-level reward setting (Feng et al., 2025; Ji et al., 2026; Dong and others, 2025a). Group-relative optimization. For a given problem, Group Relative Policy Optimization (GRPO) samples G trajectories and estimates the advantage of trajectory k as (Shao and others, 2024): (3) Ak=R(τk)−μRσR+ϵ,μR=1G∑j=1GR(τj),A_k= R( _k)- _R _R+ε, _R= 1G _j=1^GR( _j), Here, μR _R and σR _R are the within-group reward mean and standard deviation, and ϵε provides numerical stability. Standard GRPO assigns AkA_k to every model-generated token in trajectory k, so the update magnitude is uniform along the trajectory regardless of which intermediate decisions determine the outcome. Generation uncertainty. At generation position t, let the context be ht=(x,z<t)h_t=(x,z_<t) and the next-token distribution be pt=πθ(⋅∣ht)p_t= _θ(· h_t). Its Shannon entropy is (4) Ht=−∑v∈pt,vlogpt,v,H_t=- _v p_t,v p_t,v, where V denotes the vocabulary. Larger HtH_t indicates greater uncertainty over the next token. This quantity reflects the dispersion of the generation distribution at position t rather than the confidence of any particular sampled token. Prior work uses token entropy to identify critical decisions (Lin et al., 2024) and characterize exploration in language-model reasoning (Cheng et al., 2025; Wang and others, 2025). 4. Contrastive Branch Policy Optimization 4.1. Overview CBPO addresses two coupled limitations of branch-based RL: concentrated exploration and imprecise local credit. The method uses generation uncertainty for candidate selection and observed outcome variation to modulate update magnitude. Figure 1 summarizes the resulting training framework. Starting from complete interaction trajectories, CBPO scans each response for uncertain positions and allocates a fixed branch budget through path-level and node-level decay. Rewards are then compared among continuations sharing an exact prefix to construct CBV-aware local credit. Shared-prefix deduplication and non-overlapping segmentation convert this credit into token-level advantages. Section 4.2 constructs balanced exact-prefix groups, and Section 4.3 converts their outcomes into policy updates. The design maintains three invariants. First, every problem receives the same total rollout budget because unused branch slots are reallocated to independent complete trajectories. Second, each local comparison conditions on an identical token history, so reward variation arises only from resampled continuations. Third, copied prefixes are excluded from branch losses, and parent trajectories are partitioned into non-overlapping intervals when several nodes are selected. In tool-integrated trajectories, environmental observations condition later actions but are neither scanned nor assigned policy gradients, since they are not model-generated. These constraints separate allocation from attribution and prevent additional branches from duplicating shared-token gradients. Figure 1. CBPO training framework. The model first performs interactive reasoning with Python and Search tools and samples complete initial trajectories. Candidate nodes are identified across the full response, and path-level and node-level decay allocate the branch budget. Outcome rewards estimate CBV within exact-prefix groups. Shared-prefix deduplication and non-overlapping hierarchical segmentation then construct token-level advantages for the policy update.End-to-end CBPO pipeline from tool-integrated trajectory sampling through balanced branching, CBV estimation, and policy optimization. 4.2. Entropy-Guided Balanced Branch Exploration Exhaustive branching at every token is computationally infeasible, whereas branching only at tool-return boundaries assumes that consequential decisions coincide with predefined interaction events. Entropy provides an inexpensive candidate signal, but an unregularized ranking can allocate most of a fixed budget to a few trajectories or neighboring positions. This stage therefore combines full-response candidate discovery with explicit coverage control. The procedure first samples N0N_0 complete trajectories from the current policy. Figure 2 illustrates how fixed-interval scanning identifies high-entropy decisions beyond tool-call boundaries. Figure 2. Full-response candidate discovery. Fixed-interval entropy scans identify high-entropy decisions beyond predefined tool-call boundaries.Line plot of token entropy across a full response, with fixed-interval windows, tool-call positions, and selected candidate branch points. Full-trajectory candidate identification. For trajectory i, CBPO places candidate boundaries throughout the response at intervals of dmind_ : (5) ℬi=bi,q=qdminq=1Qi,B_i=\b_i,q=qd_ \_q=1^Q_i, where QiQ_i is the number of valid candidates. Let i,bW_i,b denote the window of w model-generated tokens beginning at boundary b∈ℬib _i. Its normalized entropy is (6) Hi,b=−1|i,b|log||∑t∈i,b∑v∈Ki,tpi,t,vlogpi,t,v,H_i,b=- 1|W_i,b| |V| _t _i,b _v _K^i,tp_i,t,v p_i,t,v, where Ki,tV_K^i,t is the top-K candidate set at token t, and pi,t,vp_i,t,v is the probability of candidate v. The retained probability mass is not renormalized. Thus, Hi,bH_i,b is a truncated entropy proxy used only to rank candidate locations, not the entropy of a top-K distribution. The first window of each trajectory provides a path-specific reference: (7) Hiroot=Hi,0.H_i^root=H_i,0. Local uncertainty at boundary b is measured by the entropy increase relative to this reference: (8) ΔHi,b=Hi,b−Hiroot. H_i,b=H_i,b-H_i^root. The corresponding raw priority is (9) Pi,braw=clip(α+γΔHi,b,0,1),P_i,b^raw=clip (α+γ H_i,b,0,1 ), where α sets the base priority, γ scales the entropy increase, and clipclip restricts the score to [0,1][0,1]. Path-node balanced budgeting. An entropy ranking alone can allocate most branches to a few trajectories or neighboring positions. Let LiL_i count branches assigned to path i, and let li,bl_i,b count branches at node (i,b)(i,b). CBPO updates the priority as (10) Pi,bbal=Pi,braw(1+Li)ρpath(1+li,b)ρnode,P_i,b^bal= P_i,b^raw(1+L_i) _path(1+l_i,b) _node, where ρpath,ρnode≥0 _path, _node≥ 0 control path-level and node-level decay. Before allocation, CBPO retains at most JmaxJ_ candidates from each parent. A candidate remains eligible while li,b<Bnodel_i,b<B_node and Li<BpathL_i<B_path. The hard caps are necessary because power decay never reduces a priority exactly to zero. Updating both counts after each sampled branch progressively shifts allocation toward less-explored paths and nodes. At each allocation step, CBPO selects the eligible candidate with the largest PbalP^bal. A branch is sampled only if this score exceeds κ. Otherwise, independent complete trajectories fill the unused rollout slots, preserving the fixed budget. At a selected node, the trajectory is decomposed at token boundary b as (11) τ=(pb,cb),pb=τ<b,cb=τ≥b,τ=(p_b,c_b), p_b= _<b, c_b= _≥ b, Branch generation holds prefix pbp_b fixed and resamples only continuation cbc_b. The parent and all branches sharing pbp_b form an exact-prefix group (pb)G(p_b). Comparing their outcomes controls for the preceding token history and isolates variation among the sampled continuations. The resulting balanced exact-prefix groups serve as input to the outcome-contrastive credit-assignment stage. 4.3. Hierarchical Policy Optimization with CBV The first stage identifies positions at which the policy admits alternatives, but entropy alone cannot establish whether those alternatives affect the reward. The second stage therefore measures reward variation within each exact-prefix group and translates it into local credit. This formulation emphasizes continuations sampled at outcome-sensitive nodes while suppressing redundant gradients on their shared histories. Figure 3 illustrates the distinction between shared-prefix credit and CBV-aware continuation credit. Tree schematic in which three continuations share a prefix, receive different outcome rewards, and obtain CBV-aware continuation credit alongside deduplicated shared-prefix credit. Figure 3. Outcome-contrastive credit assignment. Exact-prefix continuations are compared by outcome to construct CBV-aware credit while deduplicating shared-prefix credit.Tree schematic in which three continuations share a prefix, receive different outcome rewards, and obtain CBV-aware continuation credit alongside deduplicated shared-prefix credit. CBPO requires only bounded outcome rewards. Following ReTool and ToRL (Feng and others, 2026; Li et al., 2025), mathematical tasks use a binary correctness reward that compares trajectory answer a^(τ) a(τ) with reference answer a⋆a : (12) R(τ)=1,a^(τ) is symbolically equivalent to a⋆,0,otherwise.R(τ)= cases1,& a(τ) is symbolically equivalent to a ,\\ 0,&otherwise. cases Search tasks use normalized token-level F1 or an LLM-as-a-Judge score. Every reward satisfies R(τ)∈[0,1]R(τ)∈[0,1]; Section 5.1 gives the benchmark-specific metrics. Contrastive Branch Value. Consider exact-prefix group i=τi,0,…,τi,KiG_i=\ _i,0,…, _i,K_i\, where τi,0 _i,0 is the parent and the remaining trajectories are branches. Contrastive Branch Value (CBV) is defined as the population standard deviation of group rewards Ri,0,…,Ri,KiR_i,0,…,R_i,K_i: (13) CBVi=1Ki+1∑k=0Ki(Ri,k−R¯i)2,R¯i=1Ki+1∑k=0KiRi,k.CBV_i= 1K_i+1 _k=0^K_i (R_i,k- R_i )^2, R_i= 1K_i+1 _k=0^K_iR_i,k. Larger CBV indicates greater reward variation among continuations sampled after the same prefix. Because raw CBV is nonnegative, direct addition would only increase advantage values. Standardization across valid nodes in the batch produces a comparable signed signal: (14) ZiCBV=CBVi−μCBVσCBV+ϵ.Z_i^CBV= CBV_i- _CBV _CBV+ε. Here, ZiCBV>0Z_i^CBV>0 denotes reward variation above the batch mean. If σCBV _CBV falls below a numerical threshold, every ZiCBVZ_i^CBV is set to zero and the base advantages are retained. Credit assignment. CBV identifies relative outcome sensitivity at the group level, but the optimizer still requires token-level advantages. Problem-level GRPO normalization first yields base advantage Ai,kA_i,k for each trajectory. The common prefix at node i receives the group-mean advantage: (15) Aishared=1Ki+1∑k=0KiAi,k.A_i^shared= 1K_i+1 _k=0^K_iA_i,k. Following bounded advantage modulation (He et al., 2026), CBV changes only the magnitude of continuation advantage k. The standardized credit is bounded as (16) Ci,kCBV=clip(ZiCBV,−|Ai,k|ϕ,|Ai,k|ϕ).C_i,k^CBV=clip (Z_i^CBV,- |A_i,k|φ, |A_i,k|φ ). The CBV-aware advantage is (17) Ai,kfinal=Ai,k+ηsign(Ai,k)stopgrad(Ci,kCBV),A_i,k^final=A_i,k+η\,sign(A_i,k)stopgrad (C_i,k^CBV ), where η controls modulation strength and ϕφ limits its relative magnitude. For 0≤η<ϕ0≤η<φ, the modulated advantage retains the original sign and therefore preserves the local optimization direction. Hierarchical segmentation for multiple nodes. Let parent trajectory i contain JiJ_i selected boundaries (18) bi,1<bi,2<⋯<bi,Ji.b_i,1<b_i,2<·s<b_i,J_i. Multiple selected ancestors can otherwise assign credit repeatedly to the same suffix. CBPO therefore partitions each parent trajectory into non-overlapping intervals, with token advantage (19) A~t=Ai,1shared,0≤t<bi,1,12(Ai,j,0final+Ai,j+1shared),bi,j≤t<bi,j+1,Ai,Ji,0final,t≥bi,Ji. A_t= casesA_i,1^shared,&0≤ t<b_i,1,\\[3.0pt] 12 (A_i,j,0^final+A_i,j+1^shared ),&b_i,j≤ t<b_i,j+1,\\[3.0pt] A_i,J_i,0^final,&t≥ b_i,J_i. cases where Ai,j,0finalA_i,j,0^final is the parent’s continuation advantage at its j-th selected node. Each intermediate interval is simultaneously a suffix of the preceding node and a shared prefix of the following node. The two associated terms therefore receive equal weight. CBPO retains the GRPO objective, replacing only trajectory-level advantage AkA_k with segmented token advantage A~k,t A_k,t. For model-generated token yk,ty_k,t, the importance ratio is (20) rk,t(θ)=πθ(yk,t∣x,yk,<t)πold(yk,t∣x,yk,<t),r_k,t(θ)= _θ(y_k,t x,y_k,<t) _old(y_k,t x,y_k,<t), and its clipped form (21) r¯k,t(θ)=clip(rk,t(θ),1−ε,1+ε), r_k,t(θ)=clip (r_k,t(θ),1- ,1+ ), (22) ℓk,t(θ)=min(rk,t(θ)A~k,t,r¯k,t(θ)A~k,t). _k,t(θ)= \! (r_k,t(θ) A_k,t, r_k,t(θ) A_k,t ). The CBPO objective is (23) CBPO(θ)=[1M∑k=1M1|yk|∑t=1|yk|(ℓk,t(θ)−βDKL(πθ∥πref))].J_CBPO(θ)=E\! [ 1M _k=1^M 1|y_k| _t=1^|y_k| ( _k,t(θ)-β D_KL ( _θ\,\|\, _ref ) ) ]. The objective connects the two stages: balanced exploration defines the controlled comparisons, and CBV determines their contribution to non-overlapping continuation updates. Algorithm 1 summarizes the end-to-end training procedure. Algorithm 1 Contrastive Branch Policy Optimization 1: initial policy πθinit _ _init, reference policy πref _ref, data D, tools T 2: training steps S, rollout budget M, initial count N0N_0, caps Jmax,Bnode,BpathJ_ ,B_node,B_path 3: α,γ,κ,ρpath,ρnode,dmin,w,K,η,ϕα,γ,κ, _path, _node,d_ ,w,K,η,φ 4: trained policy πθ _θ 5: πθ←πθinit _θ← _ _init 6: for training step s=1,…,Ss=1,…,S do 7: πold←πθ _old← _θ; sample minibatch b⊂D_b 8: for all problems x∈bx _b do 9: sample N0N_0 parent trajectories; initialize rollout set ℛxR_x 10: for all parent trajectories τi∈ℛx _i _x do 11: scan dmind_ -spaced boundaries and compute Hi,b,ΔHi,b,Pi,brawH_i,b, H_i,b,P_i,b^raw 12: retain the top JmaxJ_ candidates; set Li←0L_i← 0 and li,b←0l_i,b← 0 13: while |ℛx|<M|R_x|<M do 14: form eligible set x=(i,b):Li<Bpath,li,b<BnodeC_x=\(i,b):L_i<B_path,\ l_i,b<B_node\ 15: if x=∅C_x= then 16: sample M−|ℛx|M-|R_x| independent complete trajectories; break 17: compute Pi,bbalP_i,b^bal and choose (i∗,b∗)=argmax(i,b)∈xPi,bbal(i^*,b^*)= _(i,b) _xP_i,b^bal 18: if Pi∗,b∗bal>κP_i^*,b^*^bal>κ then 19: fix prefix τi∗,<b∗ _i^*,<b^*, resample one continuation, and add it to ℛxR_x 20: Li∗←Li∗+1L_i^*← L_i^*+1; li∗,b∗←li∗,b∗+1l_i^*,b^*← l_i^*,b^*+1 21: else 22: sample M−|ℛx|M-|R_x| independent complete trajectories; break 23: evaluate R(τ)R(τ) and compute prompt-level base advantages Ai,kA_i,k over ℛxR_x 24: build exact-prefix groups; compute CBViCBV_i, ZiCBVZ_i^CBV, AisharedA_i^shared, and Ai,kfinalA_i,k^final 25: mask each copied branch prefix; assign Ai,kfinalA_i,k^final only to its sampled suffix 26: segment each parent by Eq. (19); keep base advantages for fallback rollouts 27: maximize Eq. (23) and update θ 4.4. Theoretical Analysis Two properties clarify the roles of the two signals. Conditional entropy bounds the outcome information available in sampled continuations, which supports its use for candidate screening. CBV estimates conditional outcome variance under a fixed prefix, which supports its use for local credit. The bounded modulation additionally preserves the advantage sign and local PPO gradient direction. Property 1: generation entropy bounds attainable outcome information. Given exact prefix p, let continuation C=(Y1,…,YL)∼πθ(⋅∣p)C=(Y_1,…,Y_L) _θ(· p) produce final outcome reward R. The chain rule for conditional entropy and the upper bound on conditional mutual information give (24) H(C∣p) H(C p) =∑t=1LY<t|p[H(Yt∣p,Y<t)], = _t=1^LE_Y_<t p [H(Y_t p,Y_<t) ], I(C;R∣p) I(C;R p) ≤H(C∣p). ≤ H(C p). For deterministic outcome verification, a nearly deterministic continuation can carry little information about alternative outcomes. High conditional entropy is not sufficient, however, because variation in wording, formatting, or inconsequential reasoning may leave the answer unchanged. Thus, entropy provides an upper bound rather than a direct estimate of I(C;R∣p)I(C;R p). CBPO uses it to screen for positions where meaningful alternatives may exist, then relies on observed branch rewards for credit assignment. Property 2: CBV estimates conditional outcome variance under a shared prefix. Suppose n continuations are sampled independently from prefix p, yielding rewards R1,…,RnR_1,…,R_n, and let R¯=n−1∑kRk R=n^-1 _kR_k. Then (25) CBV2(p)=1n∑k=1n(Rk−R¯)2=12n2∑i=1n∑j=1n(Ri−Rj)2,CBV^2(p)= 1n _k=1^n(R_k- R)^2= 12n^2 _i=1^n _j=1^n(R_i-R_j)^2, and (26) [CBV2(p)∣p]=n−1nVar(R∣p).E\! [CBV^2(p) p ]= n-1nVar(R p). The pairwise identity follows by expanding the squared differences: (27) ∑i=1n∑j=1n(Ri−Rj)2=2n∑i=1nRi2−2(∑i=1nRi)2. _i=1^n _j=1^n(R_i-R_j)^2=2n _i=1^nR_i^2-2 ( _i=1^nR_i )^2. For i≠ji≠ j, conditional independence gives [(Ri−Rj)2∣p]=2Var(R∣p)E[(R_i-R_j)^2 p]=2Var(R p). The n diagonal terms are zero, leaving n(n−1)n(n-1) nonzero ordered pairs. Substitution into Eq. (25) yields Eq. (26). Equations (25) and (26) show that n−1CBV2(p) nn-1CBV^2(p) is an unbiased estimator of conditional outcome variance. Its pairwise form measures reward divergence among continuations sharing the same token history. For binary correctness with qp=Pr(R=1∣p)q_p= (R=1 p), this variance is qp(1−qp)q_p(1-q_p) and is maximal at qp=1/2q_p=1/2. Consequently, an uncertain node with consistent outcomes receives little CBV, whereas a node separating success from failure receives more. Bounded modulation changes update magnitude within a controlled interval while preserving the advantage sign and local PPO gradient direction. Property 3: bounded CBV modulation preserves the update direction. Let C=CCBVC=C^CBV satisfy the clipping constraint |C|≤|A|/ϕ|C|≤|A|/φ. For any nonzero base advantage A, Eq. (17) can be written as (28) Afinal=wA,w=1+ηC|A|∈[1−ηϕ,1+ηϕ].A^final=wA, w=1+η C|A|∈ [1- ηφ,1+ ηφ ]. When 0≤η<ϕ0≤η<φ, the lower endpoint is positive. Therefore, AfinalA^final has the same sign as A. The PPO surrogate is positively homogeneous in its advantage argument, so ℓ(r,wA)=wℓ(r,A) (r,wA)=w (r,A) for w>0w>0. Because the modulation is stop-gradient, its local policy gradient is also scaled by the same positive w. CBV consequently reallocates update magnitude without reversing the preference induced by the outcome reward. When A=0A=0, the clipping constraint forces C=0C=0, and the statement holds trivially. 5. Experiments 5.1. Experimental Setup Datasets and evaluation. The evaluation covers mathematical reasoning and knowledge-intensive search, two forms of tool-integrated reasoning. Each task provides access to Web Search and Python, allowing the policy to select tools without dataset-specific routing. Mathematical evaluation uses AIME 2024, AIME 2025, MATH-500, MATH (Hendrycks and others, 2021), and the complete GSM8K test set (Cobbe and others, 2021). Reported metrics include Pass@1, the five-task macro average, and Pass@3/5 computed from five samples per problem. Knowledge-intensive search evaluation uses WebWalker (Wu et al., 2025), HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). WebWalker is evaluated with LLM-as-a-Judge Pass@1, whereas the other four benchmarks use normalized token-level F1. Domain-specific averages are reported because these metrics are not directly comparable. Table 1. Measured results (%) for Qwen3-1.7B and Qwen3-4B under a unified protocol. Mathematical tasks report Pass@1; WebWalker reports LLM-as-a-Judge Pass@1; the other search tasks report normalized token-level F1. Math Avg. and Search Avg. are unweighted five-task means. Green subscripts show improvement over size-matched GIGPO. Method Mathematical Reasoning Knowledge-Intensive Search AIME24 AIME25 MATH500 GSM8K MATH Math Avg. WebWalker HotpotQA 2Wiki MuSiQue Bamboogle Search Avg. Backbone: Qwen3-1.7B Training-Free Method Base 13.3 16.7 82.2 90.8 86.7 57.9 5.0 22.0 25.0 9.5 34.0 19.1 TIR Prompting 16.7 23.3 81.4 88.4 85.1 59.0 14.5 36.0 41.0 16.0 48.0 31.1 [3pt/3pt] Supervised and Distillation Methods SFT 16.7 23.3 82.8 91.0 87.6 60.3 16.5 48.0 58.0 22.5 58.0 40.6 OPD 20.0 26.7 84.4 92.0 88.7 62.4 18.5 52.0 66.5 25.0 60.5 44.5 [3pt/3pt] Classic RL Method GRPO 16.7 30.0 83.6 91.3 89.0 62.1 18.0 53.5 68.5 26.5 62.0 45.7 REINFORCE++ 20.0 26.7 83.2 91.9 88.3 62.0 18.5 51.0 65.5 24.0 61.0 44.0 DAPO 13.3 26.7 84.8 91.5 88.8 61.0 17.5 52.0 65.0 25.0 60.5 44.0 GSPO 20.0 26.7 84.6 91.0 88.7 62.2 18.5 54.0 68.0 26.0 63.0 45.9 EAPO 23.3 30.0 85.6 91.8 90.0 64.1 20.5 55.0 69.5 32.0 65.0 48.4 OC-GRPO 20.0 33.3 85.2 92.4 89.5 64.1 19.5 54.5 69.0 27.0 64.0 46.8 [3pt/3pt] Agentic RL Method GIGPO 23.3 26.7 84.2 92.4 89.1 63.1Δbase63.1_ _base 23.5 58.0 73.0 29.5 67.0 50.2Δbase50.2_ _base Tree-GRPO 23.3 30.0 85.0 92.0 88.8 63.8+1.1%63.8_ [rgb]0,0.88,0+1.1\% 24.0 58.5 73.5 30.0 68.0 50.8+1.2%50.8_ [rgb]0,0.88,0+1.2\% ARPO 26.7 30.0 85.0 91.7 88.3 64.3¯+1.9% 64.3_ [rgb]0,0.88,0+1.9\% 24.5 59.0 74.0 30.5 69.0 51.4¯+2.4% 51.4_ [rgb]0,0.88,0+2.4\% CBPO 26.7 33.3 86.4 93.2 90.4 66.0+4.6%66.0_ [rgb]0,0.88,0+4.6\% 26.0 61.0 75.5 31.5 72.0 53.2+6.0%53.2_ [rgb]0,0.88,0+6.0\% Backbone: Qwen3-4B Training-Free Method Base 13.3 20.0 84.6 90.0 89.8 59.5 7.0 27.0 31.0 12.0 40.0 23.4 TIR Prompting 16.7 26.7 85.1 92.7 89.3 62.1 17.5 41.5 47.0 19.5 53.5 35.8 [3pt/3pt] Supervised and Distillation Methods SFT 16.7 26.7 85.7 93.2 90.5 62.6 20.0 52.0 63.0 25.5 63.0 44.7 OPD 23.3 33.3 86.5 94.1 89.6 65.4 21.5 56.0 71.5 28.0 65.5 48.5 [3pt/3pt] Classic RL Method GRPO 16.7 40.0 86.0 93.8 91.3 65.6 21.5 57.5 73.0 29.5 66.5 49.6 REINFORCE++ 26.7 33.3 85.8 94.2 91.4 66.3 22.0 55.0 70.0 27.0 65.5 47.9 DAPO 13.3 30.0 87.8 93.5 91.5 63.2 21.0 56.0 69.0 28.0 65.0 47.8 GSPO 20.0 33.3 87.1 93.4 91.2 65.0 21.5 57.0 72.0 29.0 66.0 49.1 EAPO 23.3 36.7 87.2 94.4 91.8 66.7 24.0 59.5 74.5 35.0 69.0 52.4 OC-GRPO 23.3 33.3 87.0 94.2 90.6 65.7 22.5 58.5 73.5 29.5 67.5 50.3 [3pt/3pt] Agentic RL Method GIGPO 26.7 33.3 86.7 94.6 91.0 66.5Δbase66.5_ _base 27.0 62.0 77.0 32.5 73.0 54.3Δbase54.3_ _base Tree-GRPO 26.7 33.3 87.5 94.0 91.9 66.7+0.3%66.7_ [rgb]0,0.88,0+0.3\% 28.0 62.5 77.5 33.0 74.0 55.0+1.3%55.0_ [rgb]0,0.88,0+1.3\% ARPO 30.0 33.3 87.4 94.1 90.7 67.1¯+0.9% 67.1_ [rgb]0,0.88,0+0.9\% 29.0 63.0 78.0 33.5 75.0 55.7¯+2.6% 55.7_ [rgb]0,0.88,0+2.6\% CBPO 30.0 36.7 89.5 96.3 94.0 69.3+4.2%69.3_ [rgb]0,0.88,0+4.2\% 31.0 65.5 80.0 34.5 76.5 57.5+5.9%57.5_ [rgb]0,0.88,0+5.9\% Evaluation controls. All methods use the same system prompt, tool interfaces, decoding settings, and answer-extraction procedure. Search calls return the top ten snippets and share an evaluation cache, preventing method-specific retrieval variation from confounding policy comparisons. Mathematical answers are checked with the same symbolic-equivalence verifier. Pass@3/5 uses five independent samples for every method and problem. AIME 2024 and AIME 2025 each contain 30 problems, so one additional correct answer changes accuracy by 3.3 percentage points. Accordingly, the analysis emphasizes cross-task averages, both model scales, and component ablations rather than small AIME differences in isolation. Training setup. The experiments use Qwen3-1.7B and Qwen3-4B backbones (Yang and others, 2025). Training data are drawn from Tool-Star (Dong and others, 2025b). Mathematical experiments use 5,000 mathematical samples, whereas search experiments use search and dual-tool trajectories from 10,000 open-domain RL samples. All reward-based methods share the SFT initialization, training data, update count, outcome rewards, tool environment, and 16 rollout slots. OPD checkpoints follow the same benchmark and decoding protocol during evaluation. CBPO allocates six slots to initial trajectories and ten to a second branching stage. Independent complete trajectories fill any unused branch slots. Candidate scanning uses w=20w=20, K=10K=10, dmin=64d_ =64, and Jmax=3J_ =3. Allocation uses α=0.2α=0.2, γ=2.0γ=2.0, κ=0.25κ=0.25, ρpath=ρnode=0.2 _path= _node=0.2, Bnode=3B_node=3, and Bpath=4B_path=4. Credit modulation uses η=0.2η=0.2, ϕ=2.0φ=2.0, and a CBV standard-deviation threshold of 10−610^-6. Baselines. At both model scales, the comparison includes Base, TIR prompting (Lin and Xu, 2025), SFT, and on-policy distillation (OPD) (Li and others, 2026). Policy-optimization baselines are GRPO (Shao and others, 2024), REINFORCE++ (Hu and others, 2025), DAPO (Yu and others, 2025), GSPO (Zheng et al., 2025), EAPO (He et al., 2026), and OC-GRPO (Agrawal et al., 2026). Agentic and branch-based baselines are GIGPO (Feng et al., 2025), Tree-GRPO (Ji et al., 2026), and ARPO (Dong and others, 2025a). Grouped bar charts comparing ARPO and CBPO at Pass@1, Pass@3, and Pass@5 on five mathematical reasoning benchmarks and two model scales. Figure 4. Pass@K comparison between ARPO and CBPO. The top and bottom rows show Qwen3-1.7B and Qwen3-4B. Pass@3/5 is computed from five samples per problem. Each benchmark uses an independent vertical axis; consequently, bar heights should not be compared across benchmarks.Grouped bar charts comparing ARPO and CBPO at Pass@1, Pass@3, and Pass@5 on five mathematical reasoning benchmarks and two model scales. 5.2. Main Results The primary comparison evaluates whether the complete CBPO pipeline improves closed-form reasoning and open-world retrieval under matched rollout budgets. Table 5.1 reports mathematical macro-average Pass@1 scores of 66.0% and 69.3% with the 1.7B and 4B backbones. These scores exceed those of ARPO, the strongest size-matched baseline by macro average, by 1.7 and 2.2 percentage points. The corresponding search averages are 53.2 and 57.5, exceeding ARPO by 1.8 points at both scales. Relative to OPD, CBPO gains 3.6 and 3.9 points on the mathematical average and 8.7 and 9.0 points on the search average. CBPO ranks first or ties for first on all five mathematical benchmarks at 1.7B and on four of five at 4B. At 4B, GRPO leads AIME 2025 by one problem. CBPO also leads four of five search benchmarks at each scale, trailing EAPO on MuSiQue by 0.5 points. This consistency across domains and model scales supports the overall pipeline. The following analyses examine the component-specific hypotheses. Pass@K extension. Figure 4 evaluates whether the gain persists when several candidate answers are available. From five independent samples per problem, Qwen3-1.7B with CBPO reaches macro-average Pass@1, Pass@3, and Pass@5 scores of 66.0%, 74.6%, and 77.9%. The corresponding Qwen3-4B scores are 69.3%, 76.9%, and 80.2%. CBPO exceeds ARPO at each reported value, indicating that its advantage is not confined to single-sample decoding. Training dynamics and branch selection. Figure 5 compares reward, tool use, and selected branch positions. CBPO attains a higher final mean reward than GIGPO and ARPO while averaging fewer tool calls. The performance gain therefore cannot be attributed to more frequent tool invocation. The qualitative examples show that decay can reorder high-entropy candidates before allocation, consistent with separating candidate discovery from budget control. 5.3. Full-Response Candidate Analysis Three Qwen3-1.7B trajectories from the MATH test set were inspected to locate decisions associated with their outcomes. In one successful trajectory, the model enumerated all 16 one- and two-digit candidates before calling Python. The interpreter then correctly identified the ten primes. In a failed trajectory, the submitted code also executed correctly, but the preceding reasoning listed only six parenthesizations of an arithmetic expression. Python evaluated this incomplete list and returned two values instead of the correct four. The decisive error therefore occurred in the reasoning before tool invocation, not at the tool boundary. The third trajectory exhibited the converse pattern. Python raised the same execution exception twice, yet the model recovered the correct answer from an algebraic derivation completed before either call. A tool-return position was therefore salient but not decisive for the final reward. Together, these cases illustrate why candidate discovery should cover the full response rather than treat tool observations as universal branch boundaries. These examples provide qualitative mechanism checks, not population-level or causal evidence. A controlled tool-boundary-only ablation remains necessary to quantify this design choice independently. Figure 5. Training dynamics and branch selection. (a) Mean reward and (b) mean tool calls per trajectory; thick lines denote smoothed means, with light traces and shading showing variation. In the qualitative word clouds, (c) token size increases monotonically with candidate entropy, whereas (d) token size increases monotonically with post-decay branching priority among retained candidates. Entropy therefore favors selection, but path- and node-level decay can change the ranking and retain some positions below the entropy-only cutoff; sizes do not encode token frequency.A single-column two-by-two panel grid with mean reward at upper left and mean tool calls at lower left. The upper-right qualitative word cloud sizes candidate tokens by entropy. The lower-right cloud shows retained candidates sized by post-decay branching priority, preserving an overall entropy relationship while introducing rank reversals from path- and node-level decay. 5.4. Ablation Study The method motivates two quantitative predictions. Path-level and node-level decay should reduce complementary forms of allocation concentration, whereas CBV should improve learning beyond branch exploration alone. Table 2 tests these predictions under matched training, rollout, and evaluation protocols. Removing either decay level reduces the mathematical macro average by 1.5 points at both model scales. Removing both reduces it by 2.2 points, more than either individual ablation. The two decay terms therefore regulate distinct sources of allocation concentration. Removing CBV produces the largest losses, 3.0 points for Qwen3-1.7B and 2.9 points for Qwen3-4B. Branch exploration alone is therefore insufficient; branch outcomes must also inform local credit. Most of this reduction occurs on AIME 2024 and AIME 2025, the two benchmarks with the lowest absolute accuracy. Table 2. Component ablations for CBPO. Values are Pass@1 (%) and the five-task macro average. Shading marks the full method, and bold indicates the best value in each column. Method AIME24 AIME25 MATH500 GSM8K MATH Avg. Backbone: Qwen3-1.7B CBPO 26.7 33.3 86.4 93.2 90.4 66.0 w/o Path-level 23.3 30.0 86.2 92.8 90.2 64.5 w/o Node-level 23.3 30.0 86.1 93.0 90.3 64.5 w/o Dual decay 20.0 30.0 86.3 92.5 90.0 63.8 w/o CBV 20.0 26.7 85.5 92.9 90.1 63.0 Backbone: Qwen3-4B CBPO 30.0 36.7 89.5 96.3 94.0 69.3 w/o Path-level 26.7 33.3 89.3 95.9 93.8 67.8 w/o Node-level 26.7 33.3 89.0 96.0 93.9 67.8 w/o Dual decay 23.3 33.3 89.4 95.7 93.9 67.1 w/o CBV 23.3 30.0 89.0 96.0 93.7 66.4 Table 3. CBPO Pass@1 (%) under different rollout configurations. M=N+BM=N+B. Shading marks the default configuration, and bold indicates the best value in each column. Rollout allocation Pass@1 accuracy (%) Total M Initial N Branch B AIME24 AIME25 MATH500 GSM8K MATH Avg. Backbone: Qwen3-1.7B 4 1 3 13.3 23.3 82.2 89.9 86.0 58.9 4 2 2 16.7 23.3 82.8 90.5 86.8 60.0 [3pt/3pt] 8 2 6 16.7 26.7 83.7 91.0 87.8 61.2 8 4 4 20.0 26.7 84.0 91.5 88.3 62.1 [3pt/3pt] 16 4 12 23.3 36.7 85.8 92.9 89.8 65.7 16 6 10 26.7 33.3 86.4 93.2 90.4 66.0 16 8 8 26.7 30.0 86.1 93.0 90.2 65.2 16 12 4 23.3 30.0 85.8 92.6 89.8 64.3 Backbone: Qwen3-4B 4 1 3 16.7 26.7 84.4 91.8 89.0 61.7 4 2 2 20.0 26.7 85.0 92.3 89.7 62.7 [3pt/3pt] 8 2 6 20.0 30.0 86.6 93.8 91.0 64.3 8 4 4 23.3 30.0 87.0 94.2 91.5 65.2 [3pt/3pt] 16 4 12 26.7 40.0 88.9 95.8 93.3 68.9 16 6 10 30.0 36.7 89.5 96.3 94.0 69.3 16 8 8 30.0 33.3 89.2 96.0 94.4 68.6 16 12 4 26.7 33.3 88.8 95.6 93.2 67.5 5.5. Branching Configuration Balanced branching requires sufficient independent paths for global coverage and sufficient shared-prefix continuations for local comparison. This trade-off is evaluated by varying the allocation between initial trajectories N and dynamic branches B. Total budgets are M=N+B∈4,8,16M=N+B∈\4,8,16\, with all other settings held fixed. Increasing M from 4 to 16 improves the macro average at both model scales. Within a fixed budget, too few initial trajectories restrict path coverage, whereas too few branches weaken exact-prefix comparisons. The highest macro average occurs at N=6,B=10N=6,B=10 for both backbones. This shared optimum favors branch sampling while retaining sufficient independent trajectories for global coverage. 6. Conclusion CBPO addresses fine-grained credit assignment by separating two operations that branch-based RL often conflates. Full-response entropy scanning with path-level and node-level decay allocates a fixed rollout budget. Reward variation within exact-prefix groups then modulates the contribution of each sampled continuation. Prefix masking and hierarchical segmentation prevent shared tokens from receiving duplicated credit. With Qwen3-1.7B and Qwen3-4B, CBPO attains mathematical macro averages of 66.0% and 69.3% and search averages of 53.2 and 57.5. Cross-scale ablations attribute distinct gains to balanced allocation and CBV. However, the independent contribution of full-response scanning still requires a controlled boundary-only ablation. Within the evaluated models, tasks, and budgets, the evidence supports a bounded design principle: uncertainty identifies alternatives, whereas observed outcome variation provides a stronger signal for local credit assignment. Ethical Considerations The study uses public benchmarks and collects no new personal or human-subject data. Open-web retrieval may nevertheless expose a model to inaccurate, biased, offensive, or privacy-sensitive material. Benchmark accuracy does not establish the reliability or social neutrality of retrieved sources. Generated Python code runs in a sandbox, and tool calls are capped to limit security and resource risks. Like other branch-based methods, CBPO samples several continuations and therefore incurs additional generation cost. A fixed rollout budget bounds this cost, and all methods are compared using matched rollout slots. The experiments evaluate benchmark accuracy rather than deployment safety. High-stakes applications would require domain-specific testing, content filtering, access controls, and human oversight. References Agrawal et al. (2026) P. Agrawal, A. Samanta, S. Ghasemlou, B. Vidolov, J. Bhandari, K. Asadi, D. Jiang, and A. Modi Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information. Note: arXiv preprint arXiv:2607.19313 External Links: Document, Link Cited by: §5.1. Cheng et al. (2025) D. Cheng, S. Huang, X. Zhu, B. Dai, W. X. Zhao, Z. Zhang, and F. Wei Reasoning with Exploration: An Entropy Perspective. Note: arXiv preprint arXiv:2506.14758 External Links: Document, Link Cited by: §1, §2.1, §3. Cobbe et al. (2021) K. Cobbe et al. Training Verifiers to Solve Math Word Problems. Note: arXiv preprint arXiv:2110.14168 External Links: Document, Link Cited by: §5.1. Dong et al. (2025) G. Dong, L. Bao, Z. Wang, K. Zhao, X. Li, J. Jin, J. Yang, H. Mao, F. Zhang, K. Gai, G. Zhou, Y. Zhu, J.-R. Wen, and Z. Dou Agentic Entropy-Balanced Policy Optimization. Note: arXiv preprint arXiv:2510.14545 External Links: Document, Link Cited by: §2.1, §3. Dong et al. (2025a) G. Dong et al. Agentic Reinforced Policy Optimization. Note: arXiv preprint arXiv:2507.19849 External Links: Document, Link Cited by: §1, §2.2, §3, §3, §5.1. Dong et al. (2025b) G. Dong et al. Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning. Note: arXiv preprint arXiv:2505.16410 External Links: Document, Link Cited by: §1, §2.2, §5.1. Feng et al. (2026) J. Feng et al. ReTool: Reinforcement Learning for Strategic Tool Use in LLMs. In International Conference on Learning Representations, Cited by: §1, §2.2, §4.3. Feng et al. (2025) L. Feng, Z. Xue, T. Liu, and B. An Group-in-Group Policy Optimization for LLM Agent Training. Note: arXiv preprint arXiv:2505.10978 External Links: Document, Link Cited by: §1, §2.2, §3, §5.1. Gou et al. (2024) Z. Gou et al. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. In International Conference on Learning Representations, Cited by: §1, §2.2. Guo et al. (2025) D. Guo et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Note: arXiv preprint arXiv:2501.12948 External Links: Document, Link Cited by: §2.1. He et al. (2026) Y. He, H. Wu, S. Liu, H. Ge, H. Zhou, K. Wu, Z. Zheng, Q. Lin, Z. Zhong, and Y. Zhang Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis. Note: arXiv preprint arXiv:2604.11056 External Links: Document, Link Cited by: §2.1, §2.1, §4.3, §5.1. Hendrycks et al. (2021) D. Hendrycks et al. Measuring Mathematical Problem Solving with the MATH Dataset. Note: arXiv preprint arXiv:2103.03874 External Links: Document, Link Cited by: §5.1. Ho et al. (2020) X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), p. 6609–6625. External Links: Document, Link Cited by: §5.1. Hou et al. (2025) Z. Hou, Z. Hu, Y. Li, R. Lu, J. Tang, and Y. Dong TreeRL: LLM Reinforcement Learning with On-Policy Tree Search. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Cited by: §1, §2.1. Hu et al. (2025) J. Hu et al. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization. Note: arXiv preprint arXiv:2501.03262 External Links: Document, Link Cited by: §2.1, §5.1. Ji et al. (2026) Y. Ji, Z. Ma, Y. Wang, G. Chen, X. Chu, and L. Wu Tree Search for LLM Agent Reinforcement Learning. In International Conference on Learning Representations, Cited by: §1, §2.1, §3, §5.1. Jiang et al. (2026) C. Jiang, M. Cheng, X. Tao, Q. Mao, J. Ouyang, and Q. Liu TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoning. In Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining, Boise, ID, USA, p. 260–270. External Links: Document, Link Cited by: §2.2. Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. O. Arik, D. Wang, H. Zamani, and J. Han Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. In Conference on Language Modeling, Cited by: §2.2. Lai et al. (2024) X. Lai et al. Step-DPO: Step-Wise Preference Optimization for Long-Chain Reasoning of LLMs. Note: arXiv preprint arXiv:2406.18629 External Links: Document, Link Cited by: §1, §2.1. Li et al. (2025) X. Li, H. Zou, and P. Liu ToRL: Scaling Tool-Integrated Reinforcement Learning. Note: arXiv preprint arXiv:2503.23383 External Links: Document, Link Cited by: §1, §2.2, §4.3. Li et al. (2026) Y. Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. Note: arXiv preprint arXiv:2604.13016 External Links: Document, Link Cited by: §5.1. Lightman et al. (2024) H. Lightman et al. Let’s Verify Step by Step. In International Conference on Learning Representations, Cited by: §1, §2.1. Lin and Xu (2025) H. Lin and Z. Xu Understanding Tool-Integrated Reasoning. Note: arXiv preprint arXiv:2508.19201 External Links: Document, Link Cited by: §2.2, §5.1. Lin et al. (2024) Z. Lin, T. Liang, J. Xu, X. Wang, R. Luo, C. Shi, S. Li, Y. Yang, and Z. Tu Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability. Note: arXiv preprint arXiv:2411.19943 External Links: Document, Link Cited by: §1, §2.1, §3. Press et al. (2023) O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, p. 5687–5711. External Links: Document, Link Cited by: §5.1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. In Advances in Neural Information Processing Systems, Cited by: §2.1. Rathee et al. (2025) M. Rathee, S. MacAvaney, and A. Anand Quam: Adaptive Retrieval through Query Affinity Modelling. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, Hannover, Germany, p. 954–962. External Links: Document, Link Cited by: §2.2. Schulman et al. (2017) J. Schulman et al. Proximal Policy Optimization Algorithms. Note: arXiv preprint arXiv:1707.06347 External Links: Document, Link Cited by: §1, §2.1. Shao et al. (2024) Z. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Note: arXiv preprint arXiv:2402.03300 External Links: Document, Link Cited by: §1, §2.1, §3, §5.1. Sun et al. (2025) H. Sun, Z. Qiao, J. Guo, X. Fan, Y. Hou, Y. Jiang, P. Xie, Y. Zhang, F. Huang, and J. Zhou ZeroSearch: Incentivize the Search Capability of LLMs without Searching. Note: arXiv preprint arXiv:2505.04588 External Links: Document, Link Cited by: §2.2. Trivedi et al. (2022) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics 10, p. 539–554. External Links: Document, Link Cited by: §5.1. Wang et al. (2025) S. Wang et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. Note: arXiv preprint arXiv:2506.01939 External Links: Document, Link Cited by: §1, §2.1, §3. Wei et al. (2022) J. Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, Cited by: §1. Wu et al. (2025) J. Wu, W. Yin, Y. Jiang, Z. Wang, Z. Xi, R. Fang, L. Zhang, Y. He, D. Zhou, P. Xie, and F. Huang WebWalker: Benchmarking LLMs in Web Traversal. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 10290–10305. External Links: Document, Link Cited by: §5.1. Yang et al. (2025) A. Yang et al. Qwen3 Technical Report. Note: arXiv preprint arXiv:2505.09388 External Links: Document, Link Cited by: §5.1. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, p. 2369–2380. External Links: Document, Link Cited by: §5.1. Yao et al. (2023) S. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations, Cited by: §1, §2.2, §2.2. Yu et al. (2025) Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Note: arXiv preprint arXiv:2503.14476 External Links: Document, Link Cited by: §2.1, §5.1. Zhang et al. (2026a) J. Zhang, D. Bi, T. Sun, M. Yang, J. Wang, and Y. Wang TOOL-CURE: Tool Selection via Curriculum-Enhanced Reinforcement Learning with Sample Screening for LLMs. In Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining, Boise, ID, USA, p. 946–954. External Links: Document, Link Cited by: §2.2. Zhang et al. (2026b) Y. Zhang, S. Zhou, X. Li, Z. Tian, Y. Gao, S. Zhang, W. Hou, Y. Liu, and B. Zhou KnowFC: Navigating Knowledge Conflicts in Large Language Model-based Fact-Checking. In Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining, Boise, ID, USA, p. 996–1006. External Links: Document, Link Cited by: §2.2. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X.-H. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin Group Sequence Policy Optimization. Note: arXiv preprint arXiv:2507.18071 External Links: Document, Link Cited by: §2.1, §5.1.