Paper deep dive
Multi-Branch Policy Optimization for Multimodal Large Language Models
Shuai Lyu, Yuning Gong, Ruiling Gao, Xiaoran Shang, Zhonghong Ou, Ping Zong, Yifan Zhu, Yuan Sun, Yang Qin, Peng Hu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/11/2026, 3:19:07 AM
Summary
The paper introduces Multi-Branch Policy Optimization (MBPO), a tree-based reinforcement learning framework for Multimodal Large Language Models (MLLMs). It addresses the inadequacy of trajectory-level credit assignment in multimodal reasoning by constructing reasoning trees at vision-language decision boundaries. MBPO assigns segment-level credit through branch-relative advantages and employs a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments demonstrate that MBPO outperforms baselines like GRPO in learning signal quality and optimization efficiency.
Entities (8)
Relation Signals (6)
Multi-Branch Policy Optimization โ evaluatedon โ Geometry3K
confidence 95% ยท Evaluated on Geometry3K and MMK12, MBPO generalizes to six out-of-domain benchmarks
Multi-Branch Policy Optimization โ evaluatedon โ MMK12
confidence 95% ยท Evaluated on Geometry3K and MMK12, MBPO generalizes to six out-of-domain benchmarks
Multi-Branch Policy Optimization โ outperforms โ GRPO
confidence 95% ยท Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines
Multi-Branch Policy Optimization โ uses โ Temporal Replay Buffer
confidence 95% ยท We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness.
Multi-Branch Policy Optimization โ addresses โ Relative Advantage Degeneration
confidence 90% ยท MBPO maintains a more stable non-zero advantage proportion than GRPO throughout training.
Multi-Branch Policy Optimization โ appliedto โ Qwen2.5-VL-3B-Instruct
confidence 85% ยท MBPO remains stable, while GRPO shows a decreasing trend over training steps. throughout training. Evaluated on Geometry3K and MMK12... with Qwen2.5-VL-3B-Instruct.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.07581v1
- Canonical: https://arxiv.org/abs/2608.07581v1
Trouble viewing inline? Open PDF directly โ
Full Text
49,112 characters extracted from source content.
Expand or collapse full text
Multi-Branch Policy Optimization for Multimodal Large Language Models Shuai Lyu โ Beijing University of Posts and Telecommunications Beijing, China Lxb_savior@bupt.edu.cn Yuning Gong โ Sichuan University Chengdu, China 2021323040003@stu.scu.edu.cn Ruiling Gao Shanghai University Shanghai, China ruilinggao@shu.edu.cn Xiaoran Shang Beijing University of Posts and Telecommunications Beijing, China sxr15@bupt.edu.cn Zhonghong Ou Beijing University of Posts and Telecommunications Beijing, China zhonghong.ou@bupt.edu.cn Ping Zong Beijing University of Posts and Telecommunications Beijing, China 2010919530@bupt.cn Yifan Zhu Beijing University of Posts and Telecommunications Beijing, China yifan_zhu@bupt.edu.cn Yuan Sun Sichuan University Chengdu, China sunyuan_work@163.com Yang Qin โ Huawei Technologies Ltd. Shenzhen, China qinyang.gm@gmail.com Peng Hu โ Sichuan University Chengdu, China penghu.ml@gmail.com Abstract Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. How- ever, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, the model must repeatedly re- examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inade- quate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that con- structs reasoning trees at vision-language decision boundaries, en- abling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informa- tive segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO out- performs representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO. CCS Concepts โข Computing methodologiesโComputer vision;โข Informa- tion systemsโ Multimedia information systems. โ These authors contributed equally. โ Corresponding authors. This work is licensed under a Creative Commons Attribution 4.0 International License. M โ26, Rio de Janeiro, Brazil ยฉ 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3836564 Keywords VLM, Credit Assignment, Multimodal Reasoning ACM Reference Format: Shuai Lyu, Yuning Gong, Ruiling Gao, Xiaoran Shang, Zhonghong Ou, Ping Zong, Yifan Zhu, Yuan Sun, Yang Qin, and Peng Hu. 2026. Multi-Branch Policy Optimization for Multimodal Large Language Models. In Proceedings of the 34th ACM International Conference on Multimedia (M โ26), November 10โ14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3836564 1 Introduction With the recent success of reinforcement learning (RL) [8], train- ing paradigms have gradually shifted from purely supervised fine- tuning to outcome-driven optimization for Multimodal Large Lan- guage Models (MLLMs). Group Relative Policy Optimization [37] (GRPO), as a representative RL method, assigns outcome-based advantages to generated reasoning trajectories, providing a simple, verifiable training signal for reasoning in MLLMs. However, existing GRPO-style methods treat multimodal reasoning as a single linear token sequence and apply uniform trajectory-level credit assign- ment, ignoring that only a few perceptual or reasoning segments are causally decisive. Compared with text-only reasoning, multi- modal tasks introduce additional decision points: visual grounding, cross-modal alignment, and perceptual integration, that increase both the number of branching opportunities and the overall chain length, amplifying credit ambiguity and making trajectory-level credit assignment particularly inadequate. A distinctive challenge of multimodal reasoning is the inherent visual uncertainty of images: a single image region can admit multiple plausible interpretations, and the downstream reasoning chain is highly sensitive to whichever interpretation the model commits to at each step. The model must therefore re- peatedly re-examine visual information to verify its perceptual understanding, and different re-examinations can lead to divergent arXiv:2608.07581v1 [cs.CV] 5 Aug 2026 M โ26, November 10โ14, 2026, Rio de Janeiro, BrazilLyu, Gong, et al. Question: Determine the size of angle RST in degrees, using the information given in the diagram and the stated angle relationships. Ground Truth: 55 ยฐ GRPO MBPO BranchAdv. โํ.ํ TrajAdv โํ.ํ BranchAdv. +ํ.ํ Question ... ... +ํ reward: ํ.ํํ ํ+ํ+ํ โํ.ํ ํ.ํ ํ ํ.ํํ ํ ํ+ํ ... ... Question adv.: โํ.ํ ํ.ํ GRPO To solve for the measure of angle RST, we need to understand the properties of an equilateral triangle. An equilateral triangle is a triangle in which all three sides are equal in length, and consequently, all three angles are equal in measure. Given that triangle RST is an equilateral triangle, each angle in the triangle must be the same. The sum of the angles in any triangle is always 180 degrees. We can find the measure of each angle by dividing 180 degrees by 3. Letโs perform the calculation: โ ํ ํํ=180 ยฐ /3=60 ยฐ In a triangle, the sum of the interior angles is always 180 ยฐ . Given that triangle RST is an equilateral triangle, all three angles are equal. Let each angle be x. Then x + x + x =180 ยฐ , so 3x=180 ยฐ and x=60 ยฐ . However, the problem states that one of the angles is 70 ยฐ . This suggests that the triangle is not equilateral but rather has one angle of 70 ยฐ . So 70 ยฐ +2x =180 ยฐ ,thus 2x =110 ยฐ and x=55 ยฐ . Therefore, the measure of โ RST is 55 ยฐ . MBPO vision- language boundary Figure 1: The difference between MBPO and GRPO in terms of reward and advantage propagation. Unlike the one-size- fits-all strategy that uses a single trajectory-level advantage for the full response, our branch-level MBPO assigns a local negative advantage to the incorrect branch and a positive one to the correct branch, enabling more accurate global advantage assignment. This allows for fine-grained reward while mitigating relative advantage collapse. yet individually coherent reasoning paths. This perceptual diversity calls for a structured exploration mechanism that can systemati- cally branch at vision-language decision points and compare sibling hypotheses grounded in different visual interpretations. This property also exposes a limitation of trajectory-level policy optimization. As illustrated in Figure 1, a response may contain an incorrect intermediate segment followed by a correction, yet GRPO assigns the same trajectory-level advantage to all tokens, blurring the learning signal at critical decision points. Over time, this credit confusion triggers relative advantage degeneration, where group- relative advantages collapse toward zero as rollouts become more similar. We track this degeneration using the valid advantage ratio (VAR; Equation (1)) in Figure 2, following R1-ShareVL [32] and Shuffle-R1 [39], and observe a significant and consistent decline. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), which improves credit assignment through branch-level advantage estimation. Tree search is a natural fit for exploring the perceptual uncertainty of multimodal reasoning: sib- ling branches pursue distinct visual hypotheses from a common prefix, allowing the optimizer to directly compare alternative visual groundings rather than averaging over a single trajectory (Figure 1). We further introduce a temporal replay buffer that retains informa- tive segments to reduce policy drift. As shown in Figure 2, MBPO maintains a more stable non-zero advantage proportion than GRPO 0100200300400500 Training Step 0.990 0.992 0.994 0.996 0.998 1.000 VAR(MBPO) MBPO maintains ratio close to 1.0 with rare sharp drops Non-Zero Advantage Ratio Trend MBPO GRPO 0.5 0.6 0.7 0.8 0.9 VAR(GRPO) Figure 2: Relative advantage degeneration during training on Geo3K with Qwen2.5-VL-3B-Instruct. MBPO remains stable, while GRPO shows a decreasing trend over training steps. throughout training. Evaluated on Geometry3K and MMK12, MBPO generalizes to six out-of-domain benchmarks and consistently out- performs RL baselines including GRPO and DAPO [33], as well as Shuffle-R1 and M-Eureka-Qwen-7B when trained on MMRL18K. Our main contributions are summarized as follows: โขWe propose Multi-Branch Policy Optimization (MBPO), a tree-structured RL framework that leverages the inherent visual diversity in multimodal reasoning to construct reason- ing trees at vision-language decision boundaries, enabling branch-level credit assignment via sibling-relative advan- tages. โขWe introduce a temporal replay buffer with question- balanced sampling to reuse informative segments while lim- iting policy staleness. โข We demonstrate consistent gains over strong RL baselines on Geometry3K and MMK12, and improved out-of-domain generalization across six multimodal reasoning benchmarks. 2 Related Work 2.1 RL for MLLMs Reasoning With the development of multimodal communities [10,25,26], RL has become an important tool for improving the reasoning ability of MLLMs. Recent studies extend RL from text-only reasoning to MLLMs and downstream vision tasks. For example, R1-VL [34] adapts GRPO to MLLMs and introduces StepGRPO with stepwise reasoning rewards, while R1-ShareVL [32] proposes Share-GRPO with question expansion, trajectory sharing, and hierarchical advan- tage estimation to alleviate sparse rewards and advantage vanishing. Shuffle-R1 [39] studies the training dynamics of GRPO and uses pairwise trajectory sampling and advantage-based batch shuffle to reduce advantage collapsing and rollout silencing. DAPO [33] introduces a token-level policy gradient loss that modifies the loss attribution at the token dimension of a trajectory to stabilize op- timization. Unlike these methods, we represent each candidate solution as a multimodal reasoning tree and assign relative advan- tages among sibling branches, providing a principled way to exploit tree structure in RL for multimodal reasoning. Multi-Branch Policy Optimization for Multimodal Large Language ModelsMM โ26, November 10โ14, 2026, Rio de Janeiro, Brazil The Pipeline of MBPO Temporal Replay Buffer & Policy Optimization Replay buffer Question 1: [S1, S2, S4, ...] Question N: [S1, S2, S3, ...] ... Replay buffer Question 2: [S1, S2, ...] Parallel Reasoning Tree Construction Branch-level Advantage Assignment Illustrative Tree ํ Multi-branch reasoning tree Branch Factor [2, 2, 2] โ(ํฃ)= 1 ํถโ(ํฃ) เท ํขโํถโ(ํฃ) โ(ํข) Reward Propagation ํ ํ ํ 3 ํ ํ 1 ํ 2 ํ 1,1 ํ 1,2 ํ 1,3 ํ 2,1 ํ 2,2 0+1 +1+1000 0 Sibling Groups per Parent Node Node-level Advantage ํ ํํ ํ ํ 1 ํ 3 0.407 ํ 1 ํ 2 ํ 1,1 ํ 1,2 ํ 1,3 ํ 0.222 0.667 ํ 0-0.407โ 0 0.667-0.222โ 0 ํ 1 1-0.667โ 0 ํ 1,1 ํ 1,2 Parallel BFS & Reward Propagation Advantage Normalization and Whitening Temporal Replay Buffer Workflow Training Step Time-window Filtering Replay buffer ... Mini-batch ... Policy Optimization Loop Frequency Normalized Advantage Distribution -4 -20 2 4 -50 0 50 100 150 Negative Subsequence index Traj. 1 Traj. 2 Traj. 3 Traj. 4 Subsequence-wise Advantage Heatmap (whitened) 0123456789101112 Store Sample Policy ํ ํ ํก+1 Update actor Replay-based PPO Optimization โขSample segments โขCompute policy / Value Losses with KL Regularization โขUpdate actor & critic ํ Depth 0: Root Depth 1: vLLM #1 Scoring ํ 1 ํ 2 ํ 1 ํ 2 ํ 1 ํ 2 ํ 3 ํ 4 ํ 5 ํ 3 ํ 4 ํ 3 ํ 4 1ร vLLM.generate 2ร vLLM.generate ํ 1 Positive Figure 3: The illustration of MBPO. The figure illustrates how MBPO uses tree-structured rollouts to share reasoning branches and compute branch-level advantages. It shows the training with advantage normalization and a temporal replay buffer. 2.2 Tree-based Policy Optimization Several concurrent works also integrate tree structures into RL training for LLMs. TreePO [16] focuses on heuristic tree rollout with dynamic divergence and KV-cache reuse to improve sam- pling efficiency. TreeRPO [31] estimates step-level reward expec- tations through tree sampling to construct dense process rewards. TreeRL [12] introduces uncertainty-driven on-policy tree search that branches from high-uncertainty steps for intermediate supervi- sion. Notably, existing tree-based methods operate in the text-only domain and define branches solely through textual cues. In contrast, MBPO extends tree-structured policy optimization to multimodal reasoning through vision-language-driven branching and assigns sibling-relative advantages under shared prefixes for segment-level credit assignment. Unlike TreePO, TreeRPO, and DAPO, which respectively target sampling efficiency, step-level reward estima- tion, and token-level loss aggregation, MBPO addresses both credit ambiguity and relative advantage degeneration. 3 Methodology This section introduces three core components of MBPO: (1) rea- soning tree construction via breadth-first search (BFS) for efficient exploration, (2) branch-level advantage assignment and reward propagation for precise credit assignment, and (3) a temporal re- play buffer to limit the reuse of outdated reasoning segments. 3.1 Preliminaries Relative Advantage Degeneration. We characterize a phenom- enon referred to as relative advantage degeneration: as training progresses, relative advantages among model responses gradually flatten, leaving fewer responses with meaningful learning signals. To measure its severity, we define the valid advantage ratio as: VAR= 1 ํพ ํพ โ๏ธ ํ=1 I(|ํด ํ |> 0),(1) whereI(ยท)is the indicator function.VARmeasures the proportion of non-zero advantages. Under relative advantage degeneration,VAR declines, resulting in fewer informative updates. To address this, MBPO prioritizes updates on branches where|ํด|>0, maintaining a high valid advantage ratio during reasoning. Problem Formulation. A multimodal reasoning task is defined as follows: Given inputํฅ= (ํผ,ํ) โ VรT, whereํผis an image or a set of images from the visual spaceVandํis a text query from the text spaceT. The response space is denoted asY, and the responseํฆshould correctly answer the question and provide a clear reasoning process. 3.2 Parallel Reasoning Tree Construction We model reasoning as a tree of parallel rollouts to enable branch- level advantage estimation among sibling branches. To align branch- ing with reasoning boundaries, MBPO uses a fixed token budget with adaptive detection of vision-language decision points marked by the special token<look>. When<look>is detected, the preced- ing segment becomes a branching node where sibling branches ex- plore different visually grounded continuations. Otherwise, MBPO uses fixed-length segmentation. This balances semantic branching with stable sibling comparison (Section 4.4). Reasoning Tree Representation. We model the reasoning process as a treeT=(ํ,ํธ), whereํis the set of nodes representing partial reasoning states, andํธis the set of directed edges connecting them. Each edge corresponds to a continuous reasoning segment from a parent state to a child state. When a<look>marker is detected, a new node is created from the resulting partial reasoning state at that boundary. The reasoning tree begins with the root node, initialized with the input promptํฅ 0 and an empty responseํฆ 0 , whose state is defined as(ํฅ 0 ,ํฆ 0 ). BFS Parallel Expansion. MBPO follows BFS to expand the reason- ing tree from the root node, while generating all child branches in parallel at each node. For a nodeํ, if its depth satisfiesํ ํ โฅD, the M โ26, November 10โ14, 2026, Rio de Janeiro, BrazilLyu, Gong, et al. Algorithm 1 Parallel BFS Tree Construction for MBPO 1:Input: promptsํฅ ํ ํต ํ=1 , policyํ ํ , branch factors[ํพ 1 ,...,ํพ ํท ], step length ํ , context limit ํฟ. 2: Output: tree T and training segments S. 3: L โ โ , Currentโ ํฅ 1 ,...,ํฅ ํต 4: forํ= 1 to ํท do 5:Split Current into Expand and Done; update L โ Lโช Done 6: // Set generation budget at depthํ 7: for each ํ โ Expand do 8: ํ ํ โ ํฟโ|context(ํ)| ifํ= ํท, elsemin(ํ, ํฟโ|context(ํ)|) 9: end for 10: // Expand all current nodes in one parallel call 11:Outputs โ BatchGenerate(Expand ;ํ ํ , ํ= ํพ ํ , ํ ํ ), Nextโ โ 12: for each (ํ,ํ) โ zip(Expand, Outputs) do 13:for each sample ํ โ ํ do 14:rโ ํ .tokens, finโ (ํ .finishโ length) 15:// Truncate at vision-language boundary if detected 16: ifํ< ํทand a vision-language boundary marker appears at position ํ inr then 17:rโr 1:ํ , finโ false 18:end if 19:Create child ํ and add edge (ํ,ํ) 20:Add ํ to L ifํ= ํท or fin, else to Next 21:end for 22: end for 23:Currentโ Next 24: end for 25: // Score leaves, propagate rewards and compute advantage (Eqs. (5)โ(7)) 26: Score allโ โ Lby Eq. (5); propagateํ bottom-up by Eq. (6); compute normalized sibling-relative advantages by Eq. (7). 27: return T and S node is treated as a leaf node, and its reward is computed as: ํ (ํ)= ํ(ํฅ ํ ,ํฆ ํ ),(2) whereํฅ ํ = ํฅ 0 โํฆ 1 โยทโํฆ ํโ1 is the accumulated input sequence from the root to nodeํ, andํ(ํฅ ํ ,ํฆ ํ )is a score that measures whether the reasoning resultํฆ ํ matches the ground truth. If the remaining context length is insufficient to generate valid tokens, node ํ is also treated as a leaf node. Nodeํis expandable whenํ ํ < D. We generateํพ ํ ํ child branchesํง (1) , . . .,ํง (ํพ ํ ํ ) by sampling fromํ ํ (ยท | ํฅ ํ ,ํฆ ํ )in one batched forward pass. ํง (ํ) โผ ํ ํ (ยท | ํฅ ํ ,ํฆ ํ ), ํ= 1, . . .,ํพ ํ ํ .(3) For each ํง (ํ) , child nodeํ is instantiated with state: ํ ํ =(ํฅ ํ โ ํฆ ํ ,ํง (ํ) ).(4) The policy conditions on the full contextํฅ ํ โํฆ ํ , with token-level log-probabilities stored for optimization. Reward Back-propagation. After BFS tree construction, MBPO evaluates all leaf nodes in a single batched decoding pass and assigns each leaf nodeํ a scalar reward: ํ (ํ)= ํ(ํ ํ ).(5) The rewards are then propagated bottom-up through the tree. For each internal nodeํ, its reward is defined as the mean reward of its child nodes: ํ (ํ)= 1 |Cโจ(ํ)| โ๏ธ ํโCโจ(ํ) ํ (ํ),(6) whereCโจ(ํ)denotes the set of children of nodeํ. Based on the propagated rewards, we first compute a local parent-child value difference for each non-root node: ํด(ํ)= ํ (ํ)โ ํ (parent(ํ)).(7) This provides an intermediate reward signal for branch evaluation. 3.3 Branch-level Advantage Assignment Based on the parallel reasoning tree construction described in Sec- tion 3.2, we represent the sampling process as a tree. Each node ํin the tree corresponds to a reasoning subsequence, denoted as Sํขํํํํ(ํ). This subsequence is generated by extending the se- quence of its parent nodePํ(ํ) with ํ newly sampled tokens: Sํขํํํํ(ํ)= ํฆ (ํ) 1 , . . .,ํฆ (ํ) ํ , ํฆ (ํ) ํก โผ ํ ํ ยท | [ traj(Pํ(ํ)), ํฆ (ํ) <ํก ] . (8) wheretraj(Pํ(ํ))denotes the complete reasoning trajectory from the root node to the parent nodePํ(ํ). Each nodeํhas a set of child nodesCโ(ํ)and a set of sibling nodesSํํ(ํ). These sibling nodes share the same prompt prefix and sequence length. To enable fair comparison under the same token budget, we estimate the value of the nodeํin a recursive manner, denoted as ห ํ(ํ): ห ํ(ํ)= ( ํ (ํฅ ํ ,ํฆ ํ ),if ํ is a leaf node, 1 |Cโ(ํ)| ร ํ โฒ โCโ(ํ) ห ํ(ํ โฒ ),otherwise. (9) The branch-level advantage of nodeํis denoted asํด(ํ)and is defined through relative comparison with its sibling nodes. To compare competing continuations under the same parent, we define the branch-level sibling-relative advantage of node ํ as: ํด sib (ํ)= ห ํ(ํ)โ mean ํ โฒ โSํํ(ํ) ห ํ(ํ โฒ ) std ํ โฒ โSํํ(ํ) ห ํ(ํ โฒ )+ํ .(10) This normalized sibling-relative advantage is used in PPO. 3.4 Temporal Replay Buffer During training, we generate a large number of reasoning subse- quences, while each parameter update only consumes a limited minibatch. As a result, some high-value subsequences generated earlier may not be fully reused after the model parameters are updated, which can lead to training instability and performance degradation. To address this issue, MBPO introduces a temporal replay buffer that reuses previously generated reasoning subse- quences while controlling policy staleness. An illustration of the replay buffer is shown in Figure 4. The replay buffer is defined as: B ํํํํํํฆ =(Sํขํํํํ(ํ),ํก ํ ) ํ ํ=1 ,(11) whereํdenotes the number of subsequences stored in the buffer, andํก ํ denotes the training iteration at which subsequenceSํขํํํํ(ํ) is inserted into the buffer. Multi-Branch Policy Optimization for Multimodal Large Language ModelsMM โ26, November 10โ14, 2026, Rio de Janeiro, Brazil q 2 (Count:2) t 2 =8 All N kept Selected Selected q 3 (Count:1) t 4 =11 All N kept Selected q 4 (Count:1) t 6 =13 All N kept Selected S max =32 t -t i <T max๏ผ T max =8 Active Window: t=15 0 5 10 t=15 20 Training Iterations Expired Expired Active Active Active Active Active Keep last- T max samples Expired Step 1:Union ! t 3 =10t 7 =14 Step 2:Shuffle Take B balanced samples. Step 3:Extract Mini-Batch Each question contributes at most S max subsequences for balanced sampling. Question-balanced Sampling Time-Window Filtering Reuses high-quality subsequences for efficient sampling, stabilized by age-filtering and question balancing. Mini-Batch Assembly t 3 =10 t 4 =11 t 5 =12 t 6 =13 t 2 =8 t 1 =7 (1) Question IDs: (2) Time states: (3) Operations: Active Expired ! Union Shuffle q 3 q 4 q 1 q 2 Stores subsequences from previous iterations. q 1 q 1 q 2 q 2 q 3 q 4 q 1 (Count:2) t 1 =5 t 3 =10 t 3 =10 t 5 =12 t 8 =15 Replay Buffer B replay Figure 4: Time-window replay buffer with question-balanced mini-batch sampling, designed to limit stale trajectories and improve sample diversity during training. Time-Window Filtering. At iterationํก, we restrict the replay buffer to a fixed time window to limit the reuse of outdated subse- quences. The active replay set is defined as: B (ํก) ํํํํํํฆ =(Sํขํํํํ(ํ),ํก ํ ) โ B replay | ํก โํก ํ < ํ max ,(12) whereํ max is a predefined time horizon, andํกโํก ํ can be interpreted as the age of subsequenceSํขํํํํ(ํ). This time-window constraint limits the memory length of the replay buffer and avoids reusing outdated subsequences after significant policy changes. Question-balanced Sampling. Based on the active replay setB (ํก) replay , we construct a question-balanced mini-batch to reduce data imbal- ance across questions. For each questionํ, letํ ํ denote the set of subsequences inB (ํก) replay that are associated with questionํ. To limit the number of subsequences retained for each question, we truncate ํ ํ as: ํ trunc ํ = ( Rํํํํํํํํํํํ(ํ ํ ,ํ max ),if|ํ ํ |> ํ max , ํ ํ ,otherwise, (13) whereํ max denotes the maximum number of subsequences allowed for each question, which is typically set to 32. To construct a mini- batch of sizeํต, we combine the truncated sets from all questions and randomly shuffle the result: B batch = Shuffle ร ํ ํ trunc ํ ! [: ํต].(14) This question-balanced sampling strategy prevents the model from overfitting to a small set of questions that produce many subsequences and improves coverage across training questions. 4 Experiments 4.1 Experimental Setup for MBPO Datasets and Benchmarks. To evaluate MBPO across data scales, we conduct experiments on multimodal reasoning datasets of differ- ent sizes. For small-scale training, we use Geometry3K (Geo3K) [19] and a size-matched subset of MMK12 (K12) [22]. For large-scale training, we construct MMRL-18K using 7.2K filtered samples from M-Eureka [22] and 11K samples from ThinkLite-VL [29]. We evaluate on the in-domain Geo3K and K12 test sets and six out-of- domain benchmarks: MathVerse [35], MathVision [28], WeMath [24], MathVista [18], HallusionBench [11], and ChartQA [21]. These benchmarks cover mathematical reasoning, visual perception, and chart understanding. We use MathRuler and Gemini-2.0-Flash to evaluate free-form and multiple-choice responses, respectively. 4.2 Main Experimental Results Comparison with strong baselines. We evaluate on the large-scale MMRL18K dataset. Our MBPO-Qwen-7B model achieves 52.6% on MathVerse and 30.6% on MathVision, outperforming the M- Eureka-Qwen-7B model, which uses only RL, by 1.0% and 2.5%, respectively. It also outperforms Vision-R1-7B, and achieves this without cold-start initialization. Additionally, the model scores 87.7% on ChartQA and 71.4% on HallBench, which further demon- strates its strong performance in cross-domain reasoning tasks. Moreover, our method shows competitive performance when com- pared to closed-source models. 0100200300400500 Step 30 35 40 45 50 55 60 65 Test Accuracy (%) Test Accuracy on K12 Dataset GSPO DAPO GRPO RF++ RLOO MBPO Figure 5: Test accuracy during the training process of MBPO and representative RL algorithms on K12 dataset. Performance Comparison with RL Methods. As shown in Figure 5, MBPO converges faster and achieves higher final accuracy than RLOO, REINFORCE++, GRPO, and DAPO on K12. MBPO also con- sistently outperforms GRPO, DAPO, and GSPO [36] across all re- ported metrics on both datasets, as reported in Tables 2 and 3. On Geo3K, the Qwen-3B model improves the Math Average by 2.89% over GRPO and 1.63% over GSPO, while achieving a 2.00% gain on HallusionBench. The Qwen-7B model exhibits similarly consis- tent improvements. These results demonstrate the effectiveness of M โ26, November 10โ14, 2026, Rio de Janeiro, BrazilLyu, Gong, et al. Table 1: Performance comparison on representative multimodal reasoning benchmarks. The models marked with โ indicate results obtained from re-implementation following the original papers. To ensure comparability, all RL methods are grouped into a single category and retrained on MMRL18K, with cold-start methods marked byโ . The best results among RL-based models are highlighted in bold, and the second-best results are underlined. ModelsMathVerseMathVisionMathVistaWeMathHallBenchChartQAAvg. Closed-Source Models GPT-4o [1]50.830.463.868.855.0โ- o1 [14]57.060.373.9โ- Gemini-2.0 Pro [6]67.348.171.3โ49.8โ- Claude-3.7-Sonnet [2]52.041.366.872.655.4โ- Open-Source SFT Models InternVL-2.5-8B [5]39.517.064.5โ50.179.1- InternVL-3-8B [38]โ29.371.6โ49.986.6- Qwen2.5-VL-3B โ [3]32.322.260.853.959.874.550.5 Qwen2.5-VL-7B โ [3]43.322.867.468.659.278.156.5 Reinforcement Learning R1-VL-7B โ โ [34]40.523.861.658.759.277.553.5 Vision-R1-7B โ โ [13]46.3โ69.9โ59.181.6- R1-OneVision-7B โ โ [30]46.425.963.761.665.379.857.1 OpenVLThinker-7B โ โ [7]47.727.069.767.559.478.858.4 VLAA-Thinker-7B โ โ [4]48.526.169.667.367.181.059.9 M-Eureka-Qwen-7B โ [22]51.628.171.567.466.779.060.7 MMR1-Math-7B โ [15]38.130.2 70.268.367.783.059.6 ThinkLite-VL-7B โ [29]45.428.271.769.569.182.661.1 VL-Rethinker-7B โ [27]50.328.270.769.868.980.161.3 NoisyRollout-7B-K12 โ [17]49.527.471.070.068.382.261.5 Shuffle-R1-Qwen-7B โ [39]51.8 30.274.870.369.984.063.4 MBPO-Qwen-VL-7B52.630.674.472.671.487.764.9 Table 2: Performance of MBPO on Geo3K dataset compared with GRPO, DAPO and GSPO. Method Geo3K Math Avg. HallBench ChartQA Qwen-3B27.8342.3559.8374.54 + GRPO42.64(+14.81)46.46(+4.11)61.60(+1.77)80.20(+5.66) + DAPO45.09(+17.26)48.57(+6.22)62.34(+2.51)79.70(+5.16) + GSPO44.53(+16.70)47.68(+5.33)63.12(+3.29)79.40(+4.86) + Ours 49.91(+22.08) 49.35(+7.00) 63.60(+3.77) 83.52(+8.98) Qwen-7B42.0948.6461.0978.13 + GRPO51.31(+9.22)53.41(+4.77)66.31(+5.22)79.96(+1.83) + DAPO53.26(+11.17)53.62(+4.98)67.10(+6.01)81.33(+3.20) + GSPO52.73(+10.64) 53.89(+5.25)67.33(+6.24)79.71(+1.58) + Ours 56.07(+13.98) 53.79(+5.15) 68.52(+7.43) 85.15(+7.02) MBPO across datasets and model scales, particularly on challenging multimodal reasoning tasks. 4.3 Why Does MBPO Work? Advantage Density Analysis. To analyze the evolution of the advantage distribution during training, we estimate the per-step advantage density by decomposing it into positive, negative and zero components, as shown in Figure 6. With GRPO, the distribution is largely dominated by zero-advantage values throughout training, while the positive and negative components exhibit substantial fluctuations. This indicates that many sampled updates provide Table 3: Performance of MBPO on K12 dataset compared with GRPO, DAPO and GSPO. MethodK12 Math Avg. HallBench ChartQA Qwen-3B43.1042.3559.8374.54 + GRPO55.82(+12.72)48.04(+5.69)63.84(+4.01)78.31(+3.77) + DAPO58.83(+15.73)48.24(+5.89) 64.88(+5.05)78.42(+3.88) + GSPO57.63(+14.53)48.29(+5.94)64.48(+4.65)78.20(+3.66) + Ours 60.14(+17.04) 50.19(+7.84)64.77(+4.94) 83.44(+8.90) Qwen-7B52.0648.6461.0978.13 + GRPO65.18(+13.12)53.53(+4.89)65.80(+4.71)82.08(+3.95) + DAPO67.65(+15.59)54.52(+5.88)68.71(+7.62)81.76(+3.63) + GSPO67.23(+15.17)54.24(+5.60)69.02(+7.93)81.16(+3.03) + Ours 67.68(+15.62) 55.51(+6.87) 69.84(+8.75) 86.36(+8.23) limited learning signal and result in a sparse advantage distribution. In contrast, MBPO shows a different profile, with little mass at zero and density concentrated on non-zero advantages. As training progresses, the proportion of positive advantages steadily increases, accompanied by a gradual decline in negative ones. These results suggest that MBPO promotes a more informative and consistently positive advantage distribution during optimization. Diversity in the Replay Buffer. To better understand the role of the replay buffer in MBPO, we compute statistics on the types of questions in the buffer during training and analyze the changes in Multi-Branch Policy Optimization for Multimodal Large Language ModelsMM โ26, November 10โ14, 2026, Rio de Janeiro, Brazil 020040060080010001200 Step 0.0 0.2 0.4 0.6 0.8 1.0 Estimated Advantage Density Estimated Advantage Density over Training (GRPO) Positive Negative Zero 0100200300400500 Step 0.0 0.2 0.4 0.6 0.8 1.0 Estimated Advantage Density Estimated Advantage Density over Training (MBPO) Positive Negative Zero Figure 6: Estimated advantage distribution across training steps for GRPO and MBPO on Geo3K dataset. 0100200300400500 Training Step 20 40 60 80 100 120 Distinct Questions Distinct Questions in Replay Buffer MBPO (4-4-4) (EMA) MBPO (6-6-6) (EMA) MBPO (8-8-8) (EMA) Figure 7: Replay buffer diversity under different MBPO tree configurations, measured by the number of distinct questions stored in the replay buffer. sample diversity by comparing different tree structure configura- tions (such as 4-4-4, 6-6-6, and 8-8-8). As shown in Figure 7, MBPO (8-8-8) rapidly increases question diversity in the early stages and maintains a high level of diversity throughout training, with only a slight decrease in the final stage. In contrast, MBPO (6-6-6) reaches a moderate plateau but fluctuates more, with several sharp drops in diversity. MBPO (4-4-4) saturates at a lower diversity level and remains roughly flat, indicating lower exploration and a narrower replay buffer. These results suggest that larger tree structures help support broader and more stable replay diversity, potentially pro- viding richer learning signals for the optimization process. Self-Correction Behavior. For each generated response, we ana- lyze the text following the<Look>marker and identify self-correction steps via explicit revision cues at the token level (e.g.,however,but, actually,wait,reconsider) and phrase level (e.g.,on second thought,that is incorrect), and report the rate at which such revisions lead to correct final answers. As shown in Tables 4 and 5, MBPO exhibits increasingly frequent and earlier self-corrections as training progresses, with the successful correction rate rising from 4.7% to 29.8%, whereas GRPO, DAPO, and GSPO show limited self- correction and rarely perform multi-step revisions. These results confirm that MBPO provides clear learning signals that encourage the model to revisit visual inputs and perform corrections at critical reasoning nodes. Table 4: Self-correction statistics during MBPO training. Re- sponse length is measured in tokens from the first token after<think>. SC: self-correction rate; Succ. Rate: successful correction rate; Median 1st Corr. Pos.: median token position of the first correction. Training Steps Median 1st Corr. Pos.โ Avg. Response Length (tokens)โ SC Rate (%)โ Succ. Rate (%)โ 0129.0170.513.34.7 250115.0236.345.320.5 524108.5239.260.929.8 Table 5: Comparison of self-correction behaviors among GRPO, DAPO, MBPO, and GSPO. Method Avg. Response Length (tokens)โ SC Rate (%)โ โฅ2C Rate (%)โ Succ. Rate (%)โ GRPO196.220.17.77.8 DAPO198.512.33.87.4 GSPO243.829.916.2119.3 MBPO239.260.944.629.8 Compute-Controlled Analysis. A key concern for tree-based pol- icy optimization is whether the observed gains simply reflect in- creased rollout expenditure. To address this, we compare MBPO with GRPO under matched cumulative compute budgets, keeping the number of training segments per update fixed at 1,024 for both methods. As shown in Table 6 and Figure 8(a), MBPO achieves higher accuracy at every aligned compute checkpoint and under the same wall-clock training time, with improvements ranging from +2.0 to +8.7 points. These results indicate that the gains stem from more effective use of computation rather than a larger rollout budget. Although tree construction introduces additional cost per optimization step, the BFS batching strategy groups all same-depth nodes into a single vLLM call withํ= ํพparallel sampling, keeping the per-epoch GPU cost comparable: 19.48 GPU-hours for MBPO versus 19.32 for GRPO on 4 GPUs (less than 1% overhead). MBPO thus maintains a favorable accuracyโtime trade-off and consistently outperforms GRPO and DAPO throughout training. M โ26, November 10โ14, 2026, Rio de Janeiro, BrazilLyu, Gong, et al. 0.0481216 Training Time (Hours) 25 35 45 55 Test Accuracy (%) Test Accuracy vs. Training Time GRPO DAPO MBPO (a) 0100200300400500 Step 25 35 45 55 Test Accuracy (%) Test Accuracy on Geo3K MBPO (4-4-4) MBPO (6-6-6) MBPO (8-8-8) (b) 0.0481216202428323640 Relative Time (Hours) 25 35 45 55 Test Accuracy (%) Accuracy vs. Hours on Geo3K MBPO (4-4-4) MBPO (6-6-6) MBPO (8-8-8) MBPO (8-8-8) gradually reaches MBPO (6-6-6) with extended training (c) Figure 8: (a) Test accuracy over training time (hours) for MBPO, GRPO, and DAPO on Geo3K. (b) Test accuracy over training steps for different tree structures. (c) Test accuracy over training time (hours) for different tree structures. Table 6: Compute-matched performance comparison be- tween GRPO and MBPO on Geo3K. Each row fixes the cumu- lative training TFLOPs and reports the optimization steps and test accuracy achieved by each method under the same compute budget. Bold numbers indicate the best result. Cumul. TFLOPs GRPO Steps GRPO Acc MBPO Steps MBPO Acc ฮ 0.469M5029.85033.8+4.0 0.917M10032.69540.6+8.0 1.387M15037.814039.8+2.0 1.892M20039.919041.9+2.0 2.354M25037.923542.4+4.5 2.832M30039.828045.3+5.5 3.309M35040.332545.6+5.3 3.788M40037.137045.8+8.7 4.242M45041.441548.4+7.0 4.688M50044.145547.1+3.0 4.871M52043.447548.1+4.7 4.4 Ablation Study Ablation on Tree Structure. We compare three tree structure con- figurations: MBPO (4-4-4), MBPO (6-6-6), and MBPO (8-8-8). As shown in Figure 8(b), larger tree structures yield better performance in later training stages, with MBPO (6-6-6) and MBPO (8-8-8) both outperforming MBPO (4-4-4). Figure 8(c) shows that MBPO (8-8- 8) gradually approaches the accuracy of MBPO (6-6-6) given ex- tended training time, indicating that MBPO (6-6-6) offers a better efficiencyโaccuracy trade-off. Effect of Branching Strategy and Segment Length. As shown in Ta- ble 7, all MBPO variants outperform GRPO (42.64), which confirms the effectiveness of tree-structured exploration. Fixed-ํbranching achieves 48.35 with an average segment length of 100.0 tokens, offer- ing stable and frequent splits, but it may divide the reasoning path at positions that are not well aligned with semantic boundaries. Vi- sual branching via<look>obtains 45.17, but produces much longer segments (avg. 287.4 tokens), suggesting that<look>is triggered too infrequently and therefore leads to shallow trees with limited branching. The hybrid strategy combines both mechanisms, reduc- ing the average segment length to 83.6 tokens while achieving the best accuracy of 49.91, which indicates that<look>helps refine Table 7: Ablation of branching strategies and segment granu- larity on Geo3K. Bold numbers indicate the best result. Setting Branching Geo3K Acc. Avg. Seg. Len. GRPOflat42.64โ MBPOhybrid (ํ=100 + <look>) 49.9183.6 ํ only (fixed ํ=100)48.35100.0 <look> only45.17287.4 hybrid (ํ=50 + <look>)46.8243.2 hybrid (ํ=200 + <look>)48.53163.5 segment boundaries while fixed-ํmaintains sufficient branching density. For segment length,ํ=50 over-fragments the reasoning process, resulting in shorter segments (avg. 43.2 tokens) but lower accuracy (46.82), whereasํ=200 produces overly long segments (avg. 163.5 tokens, 48.53) that restrict branching diversity. By com- parison,ํ=100 provides the best balance between branching frequency and segment coherence. 5 Conclusion In this work, we propose Multi-Branch Policy Optimization (MBPO), a reinforcement learning framework for MLLMs that improves credit assignment through branch-level advantage estimation on a reasoning tree. With the temporal replay buffer, MBPO reuses useful segments and makes learning more stable for long reason- ing tasks. Experiments on multiple multimodal reasoning datasets demonstrate the effectiveness and training stability of MBPO. 6 Limitations and Future Work However, MBPO has limitations, including high training costs, chal- lenges in adapting to broader multimodal settings, and difficulty in large-scale transfer. A key future direction is to predict whether a partial reasoning path is worth continuing, so we can stop low- value branches early and focus on more promising ones. We also plan to build stronger datasets and benchmarks, and to scale MBPO to longer contexts and larger models. Multi-Branch Policy Optimization for Multimodal Large Language ModelsMM โ26, November 10โ14, 2026, Rio de Janeiro, Brazil Acknowledgments This work was supported by the National Natural Science Founda- tion of China under Grants U25B6003, U25A20534, 62472295, and 62406036; the National Social Science Fund of China under Grant 24BGL131; the National Key Research and Development Program of China under Grant 2024YFC3308500; the Beijing Municipal Natural Science Foundation under Grant L251042; the China Postdoctoral Science Foundation under Grant 2025M781457; the Key Laboratory of Knowledge Mining and Services for Integrated Publishing in Education under Grant KT20250804; the State Key Laboratory of Networking and Switching Technology under Grant NST20250110; the Chengdu Science and Technology Bureau Project under Grant 2025-YF05-00391-SN; the Sichuan Science and Technology Planning Project under Grant 2024NSFSC0521; and the Luzhou City School- Local-Enterprise-Academy Science and Technology Cooperation Project under Grant 2024XDY200. References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [2] Ibar Federico Anderson. 2025. Comparative Analysis Between Industrial Design Methodologies Versus the Scientific Method: AI: Claude 3.7 Sonnet. Preprints.org (2025). [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025). [4]Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. 2025. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468 (2025). [5]Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024). [6]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025). [7]Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. 2025. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement. arXiv preprint arXiv:2503.17352 (2025). [8] Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wen- wen Xiong, Yuqin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang, et al.2025. Atom-searcher: Enhancing agentic deep research via fine-grained atomic thought reward. arXiv preprint arXiv:2508.12800 (2025). [9]Lihuang Fang, Yuchen Zou, Jin kebing, and Jinghui Qin. 2026. EmoAgent- R1:Towards Multimodal Emotion Understanding with Reinforcement Learning- based Dynamic Agent Specialization. In Proceedings of the 34th ACM International Conference on Multimedia. [10]Yanglin Feng, Yang Qin, Dezhong Peng, Rui Wang, Xiaomin Song, and Peng Hu. [n. d.]. Multimodal Nested Learning for Decoupled and Coordinated Optimization. In Forty-third International Conference on Machine Learning. [11]Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. arXiv:2310.14566 [cs.CV] https://arxiv.org/abs/2310.14566 [12]Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. 2025. Treerl: Llm reinforcement learning with on-policy tree search. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 12355โ12369. [13]Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. 2025. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749 (2025). [14]Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.2024. Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024). [15]Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Hang Zhang, Yuming Jiang, Xin Li, Deli Zhao, et al.2025. Mmr1: Advancing the frontiers of multimodal reasoning. [16]Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, et al.2025. Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling. arXiv preprint arXiv:2508.17445 (2025). [17]Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. 2025. Noisyrollout: Reinforcing visual reasoning with data augmentation, 2025a. URL https://arxiv. org/abs/2504.13055 (2025). [18] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. arXiv:2310.02255 [cs.CV] https://arxiv.org/abs/2310.02255 [19] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165 (2021). [20]Shuai Lyu, Zijing Tian, Zhonghong Ou, Yifan Zhu, Xiao Zhang, Qiankun Ha, Haoran Luo, and Meina Song. 2025. TSVC: Tripartite Learning with Semantic Variation Consistency for Robust Image-Text Retrieval. Proceedings of the AAAI Conference on Artificial Intelligence 39, 18 (April 2025), 19269โ19277. doi:10.1609/ aaai.v39i18.34121 [21]Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. arXiv:2203.10244 [cs.CL] https://arxiv.org/abs/2203.10244 [22]Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al.2025. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365 (2025). [23] Siwei Meng, Yawei Luo, Shu Zhang, and Ping Liu. 2026. When Physical Pref- erences Meet Semantic Constraints: Physical and Semantic Direct Preference Optimization for Text-to-Video Generation. arXiv preprint arXiv:2607.16947 (2026). [24] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. 2024. We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? arXiv:2407.01284 [cs.AI] https://arxiv.org/abs/2407. 01284 [25]Yang Qin, Chao Chen, Zhihang Fu, Dezhong Peng, Xi Peng, and Peng Hu. 2025. Human-centered interactive learning via mllms for text-to-image person re- identification. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). IEEE, 14390โ14399. [26]Yang Qin, Yuan Sun, Xi Peng, Dezhong Peng, Joey Tianyi Zhou, Xiaomin Song, and Peng Hu. 2026. Robust semi-paired multimodal learning for cross-modal retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 24964โ24972. [27] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. 2025. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837 (2025). [28] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems 37 (2024), 95095โ95169. [29]Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. 2025. Sota with less: Mcts- guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934 (2025). [30]Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al.2025. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615 (2025). [31]Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. 2025. Treerpo: Tree relative policy optimization. arXiv preprint arXiv:2506.05183 (2025). [32]Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, et al.2025. R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO. arXiv preprint arXiv:2505.16673 (2025). [33]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.2025. Dapo: An open- source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 (2025). [34]Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. 2025. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937 (2025). [35]Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al.2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. In European M โ26, November 10โ14, 2026, Rio de Janeiro, BrazilLyu, Gong, et al. Conference on Computer Vision. Springer, 169โ186. [36]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. Group Sequence Policy Optimization. arXiv:2507.18071 [cs.LG] https: //arxiv.org/abs/2507.18071 [37]Qihao Zhu. Zhihong Shao, Peiyi Wang. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/ 2402.03300 [38] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al.2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025). [39]Linghao Zhu, Yiran Guan, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Bin Qin, Jian Luan, Yuliang Liu, and Xiang Bai. 2025. Shuffle-r1: Efficient rl framework for multimodal large language models via data-centric dynamic shuffle. arXiv preprint arXiv:2508.05612 (2025).