Paper deep dive
Collaborative Learning and Reasoning over Model Slices
Yuheng Wang, Runde Yang, Zhuohan Bao, Qiran Zhang, Huatao Li, Yiyang Zheng, Lin Wu, Jingru Fan, Shu Yao, Yihan Li, Zhi Li, Chen Qian
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/12/2026, 6:34:12 AM
Summary
The paper introduces 'Collaborative Learning and Reasoning over Model Slices', a framework that replaces a single large language model (LLM) with multiple smaller, heterogeneous models under a comparable parameter budget. This approach, termed 'model slicing', enables reasoning-state isolation and cross-model reassessment. The framework employs 'pyramid supervision', combining protocol distillation and step-level supervision, to train these models for effective collaboration. Experiments across four task categories (multi-hop QA, math reasoning, code generation, open-ended generation) show that the trained heterogeneous models match or outperform the original large model and multi-agent baselines in performance and efficiency.
Entities (11)
Relation Signals (10)
Model Slicing → replaces → Qwen3-14B
confidence 98% · Model slicing replaces Qwen3-14B (ML_L) with Qwen3-1.7B, Qwen3-4B, and Qwen3-8B
Collaborative Learning → comprises → Pyramid Supervision
confidence 95% · Collaborative learning encompasses both the construction of heterogeneous collaborators and the learning of cross-model collaboration... Pyramid supervision guides collaborative learning
Collaborative Learning → comprises → Model Slicing
confidence 95% · Collaborative learning encompasses both the construction of heterogeneous collaborators and the learning of cross-model collaboration... Model slicing constructs heterogeneous collaborators
Pyramid Supervision → comprises → Protocol Distillation
confidence 93% · Pyramid supervision combines protocol distillation with step-level supervision
Pyramid Supervision → comprises → Step-Level Supervision
confidence 93% · Pyramid supervision combines protocol distillation with step-level supervision
Collaborative Learning → evaluatedon → Conifer
confidence 90% · We evaluate our approach across four task categories: open-ended constrained generation... Conifer
Collaborative Learning → evaluatedon → MuSiQue
confidence 90% · We evaluate our approach across four task categories: multi-hop question answering... MuSiQue
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) excel across diverse tasks, benefiting from knowledge priors, in-context reasoning, and test-time computation. Despite these strengths, single-model reasoning remains vulnerable to error propagation, as later steps depend on the model's earlier outputs, leaving reasoning blind spots difficult to independently reassess. To address this limitation, we develop \textit{collaborative learning}, encompassing the construction of heterogeneous collaborators and the learning of effective cross-model collaboration. Specifically, \textit{model slicing} replaces a large model with smaller, heterogeneous models under a comparable parameter budget, enabling prior reasoning to be reassessed across distinct model parameterizations. \textit{Pyramid supervision} combines protocol distillation with step-level supervision to refine reasoning and interaction decisions throughout collaboration. We evaluate our approach across four task categories: multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation. The trained heterogeneous models achieve comparable or better task performance than the original large model and established multi-agent baselines, including reinforcement-learning-based methods, while also achieving higher inference efficiency than the multi-agent baselines and exhibiting differentiated collaboration patterns across tasks. These findings show that learned collaboration turns model heterogeneity into complementary reasoning capabilities.
Tags
Links
- Source: https://arxiv.org/abs/2610.12927v1
- Canonical: https://arxiv.org/abs/2610.12927v1
Trouble viewing inline? Open PDF directly →
Full Text
66,930 characters extracted from source content.
Expand or collapse full text
Collaborative Learning and Reasoning over Model Slices Yuheng Wang†Runde Yang†Zhuohan BaoQiran ZhangHuatao LiYiyang ZhengLin WuJingru FanShu YaoYihan LiZhi Li Chen QianSchool of Artificial Intelligence, Shanghai Jiao Tong University Abstract Large language models (LLMs) excel across diverse tasks, benefiting from knowledge priors, in-context reasoning, and test-time computation. Despite these strengths, single-model reasoning remains vulnerable to error propagation, as later steps depend on the model’s earlier outputs, leaving reasoning blind spots difficult to independently reassess. To address this limitation, we develop collaborative learning, encompassing the construction of heterogeneous collaborators and the learning of effective cross-model collaboration. Specifically, model slicing replaces a large model with smaller, heterogeneous models under a comparable parameter budget, enabling prior reasoning to be reassessed across distinct model parameterizations. Pyramid supervision combines protocol distillation with step-level supervision to refine reasoning and interaction decisions throughout collaboration. We evaluate our approach across four task categories: multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation. The trained heterogeneous models achieve comparable or better task performance than the original large model and established multi-agent baselines, including reinforcement-learning-based methods, while also achieving higher inference efficiency than the multi-agent baselines and exhibiting differentiated collaboration patterns across tasks. These findings show that learned collaboration turns model heterogeneity into complementary reasoning capabilities. Code is available at: github.com/SJTUFreshman/collaborative-learning. †footnotetext: † Equal contribution.†footnotetext: 🖂 Corresponding author. 1 Introduction Large language models (LLMs) have demonstrated capabilities across tasks (Wang et al., 2024b). Advances highlight the benefits of scaling test-time computation (Snell et al., 2025; Muennighoff et al., 2025; Wu et al., 2025). Many approaches seek to improve task performance by allowing a single model to structure its reasoning process (Besta et al., 2024) and generate, inspect, and revise intermediate steps (Zhou et al., 2024; Zhang et al., 2024; Kumar et al., 2025). However, when the same model both generates and evaluates a solution, it may exhibit self-bias, favoring its own outputs and limiting its ability to identify and correct reasoning errors (Huang et al., 2024; Xu et al., 2024). Consequently, errors in intermediate steps may remain undetected and influence subsequent reasoning (Zheng et al., 2025). This motivates reasoning-state isolation across models, allowing prior reasoning to be independently reassessed under distinct model parameterizations rather than repeatedly evaluated within the same model’s reasoning space (Du et al., 2024; Liang et al., 2024). Accordingly, we develop a collaborative learning framework for reasoning and learning across heterogeneous models, as illustrated in Figure 2. Model slicing replaces a single large model with several smaller, heterogeneous models under a comparable total parameter budget (Liu et al., 2026b). Through explicit handoffs of task state between models (Chen et al., 2024c), prior reasoning can be independently reassessed under distinct model parameterizations, thereby enabling model-specific reasoning-state isolation while producing interaction trajectories that capture reasoning and interaction decisions. Together, this isolation and the resulting trajectories provide the structural basis for training heterogeneous models to leverage complementary capabilities (Wang et al., 2025; Abdulaal et al., 2025; Chen et al., 2025) when questioning, extending, or revising prior conclusions. Figure 1: Collaborative Learning and Reasoning over Model Slices. Model slicing reallocates a single large model into multiple smaller, heterogeneous models under a comparable parameter budget, enabling reasoning to be reassessed across model parameterizations. Building on this, collaborative learning trains heterogeneous models through pyramid supervision, combining protocol distillation for interaction initialization with step-level supervision over collaboration trajectories22 2 Short linear, long linear, short cyclic, and long cyclic trajectories are defined in Section 3.5.. Although this structure provides a foundation for cross-model collaboration, outcome-level supervision alone provides insufficient guidance for intermediate reasoning and interaction decisions throughout the collaborative process (Lightman et al., 2024; Wang et al., 2024a; Setlur et al., 2025). To address this limitation, we propose pyramid supervision, which uses supervisory signals from larger models to guide the heterogeneous models produced by model slicing toward effective cross-model collaboration. It combines interaction initialization with fine-grained supervision over reasoning and interaction decisions along model-generated collaboration trajectories, while integrating task-level rewards to jointly optimize intermediate decision quality and overall task performance. Through this training mechanism, heterogeneous models learn effective collaborative reasoning and develop differentiated interaction patterns across tasks. We evaluate our approach across four task categories: multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation. With a total parameter budget comparable to that of the original large model, the trained heterogeneous models collectively match or outperform the original model as well as established multi-agent system (MAS) and multi-agent reinforcement learning (MARL) baselines. Analysis of interaction trajectories further reveals reduced reasoning redundancy and differentiated collaboration patterns characterized by questioning, complementing, and revising prior reasoning. Together, these results suggest that collaborative reasoning across heterogeneous models can improve task performance while enabling complementary contributions and the reassessment of prior reasoning across distinct model parameterizations. Our key contributions are summarized in three points: • We propose a trainable paradigm for collaborative reasoning and learning across heterogeneous models, enabling independent reassessment across distinct model parameterizations and progressive learning of effective cross-model collaboration from interaction trajectories. • Collaborative learning comprises model slicing and pyramid supervision. Model slicing constructs heterogeneous models under a comparable budget and enables reasoning-state isolation through task-state handoffs. Pyramid supervision combines interaction initialization with fine-grained supervision from the original large model over reasoning and interaction decisions. • We validate collaborative learning across four task categories. The trained heterogeneous models achieve comparable or better performance than the original large model, established MAS baselines, and MARL methods, while demonstrating higher inference efficiency than MAS and MARL baselines and exhibiting emergent task-adaptive cross-model collaboration patterns. 2 Method Figure 2: Overview of collaborative learning. (a) Model slicing constructs smaller, heterogeneous models under a comparable total parameter budget. (b) Pyramid supervision initializes collaboration through protocol distillation and refines it through step-level supervision. The original large model evaluates each step’s reasoning and interaction decisions within the complete trajectory. Step-level scores and final task rewards jointly guide collaborative policy refinement. Collaborative reasoning characterizes an iterative reasoning process among heterogeneous models, where explicit task-state handoffs under a unified interaction protocol allow prior reasoning to be continually reassessed by models with different parameters (⊳ Section 2.1). To realize this process, collaborative learning constructs a heterogeneous model set through model slicing and learns effective cross-model collaboration through pyramid supervision (⊳ Section 2.2). 2.1 Collaborative Reasoning Collaborative reasoning refers to a process in which multiple models jointly construct a solution through successive contributions (Yin et al., 2023; Chen et al., 2024a). We define a unified interaction protocol that specifies how heterogeneous models generate reasoning and retain, transfer, or terminate control. For a task input q, each interaction step t is represented by zt=(it,xt,yt,at)z_t=(i_t,x_t,y_t,a_t) (1) where iti_t denotes the active model MitM_i_t, xtx_t contains the task input and preceding interaction history, yty_t is the current reasoning, and ata_t is the interaction action. The action space is defined as at∈continue,terminate∪handoff(j)∣j∈1,…,N,j≠ita_t∈\ continue, terminate\∪\ handoff(j) j∈\1,…,N\,\ j≠ i_t\ (2) where continue keeps control with the model, handoff(j) transfers the task state and control to MjM_j, and terminate ends collaborative reasoning. A complete trajectory is represented as τ=(z1,z2,…,zT)τ=(z_1,z_2,…,z_T) (3) where T denotes the total number of interaction steps recorded in the complete trajectory. Through handoff, models use distinct parameters to extend, question, or revise prior conclusions, enabling cross-model reassessment across the evolving reasoning process. The resulting trajectories record reasoning and interaction decisions for subsequent collaborative learning. 2.2 Collaborative Learning Collaborative learning encompasses both the construction of heterogeneous collaborators and the learning of cross-model collaboration, with the pipeline illustrated in Figure 2. Model slicing constructs heterogeneous collaborators under a comparable parameter budget, providing the structural basis for cross-model reasoning. Pyramid supervision guides collaborative learning through protocol distillation and step-level supervision, progressively refining models’ reasoning and interaction decisions while integrating task-level rewards throughout the collaborative reasoning process. 2.2.1 Model Slicing Model slicing replaces a large model with smaller, independently parameterized models under a comparable parameter budget, each operating as a standalone model and interacting through explicit task-state handoffs. Let MLM_L denote the large model with parameters θL _L. We have ℳ=M1,…,MN,∑i=1N|θi|≈|θL|M=\M_1,…,M_N\, _i=1^N| _i|≈| _L| (4) where θi _i denotes the parameters of MiM_i, and |θ||θ| denotes the parameter count. These models differ in size and parameters and are not obtained by directly partitioning the weights of MLM_L. Together with task-state handoffs, this heterogeneous model set enables model-specific reasoning-state isolation across distinct model parameterizations during collaboration. 2.2.2 Pyramid Supervision We introduce pyramid supervision, which refines collaboration through supervision across model scales. Protocol distillation uses the largest model slice to initialize cross-model interaction, while step-level supervision uses the original large model to evaluate collaboration trajectories. Protocol Distillation. Although the interaction protocol specifies how models communicate and transfer control, heterogeneous models do not naturally learn to use it effectively as reasoning evolves. In particular, they must learn when to retain control, hand off the task state, or terminate based on the evolving reasoning context (Zeng et al., 2024). We therefore use protocol distillation to initialize cross-model interaction from teacher-generated trajectories (Chen et al., 2024b). We select the largest model in ℳM by parameter count as the teacher M⋆M and instantiate N homogeneous copies to form the teacher system: ℳteacher=M1⋆,…,MN⋆,Mj⋆≡M⋆,j=1,…,NM_teacher=\M_1 ,…,M_N \, M_j ≡ M , j=1,…,N (5) Following the same interaction protocol as the target models, the teacher system generates multi-step trajectories for distillation inputs distill=q(m)m=1KQ_distill=\q^(m)\_m=1^K: τ(m)∼Pℳteacher(τ∣q(m)),m=1,…,Kτ^(m) P_M_teacher (τ q^(m) ), m=1,…,K (6) We retain qualified trajectories as demonstrations for subsequent distillation. To avoid a fixed correspondence between teacher slots and target models, we sample a uniform permutation for each trajectory: π(m)∼Uniform(SN),i~t(m)=π(m)(it(m)),t=1,…,Tmπ^(m) (S_N), i_t^(m)=π^(m) (i_t^(m) ), t=1,…,T_m (7) where SNS_N denotes the set of permutations over 1,…,N\1,…,N\. Within each trajectory, the same permutation is applied consistently to all occurrences associated with each teacher slot, including model indices, handoff targets, and corresponding references in the interaction history. This operation changes only the correspondence between teacher slots and target models across trajectories, while preserving the reasoning content, step order, and handoff structure of the original trajectory. Let x~t(m) x_t^(m) and o~t(m) o_t^(m) denote the reassigned context and response. Supervision covers the complete response, including reasoning and interaction decisions (Chen et al., 2023). For each model MiM_i, the reassigned context–response pairs satisfying i~t(m)=i i_t^(m)=i form its training set iproD_i^pro. Following sequence-level knowledge distillation (Kim and Rush, 2016), each model minimizes ℒipro=−(x,o)∼ipro[logpθi(o∣x)]L_i^pro=-E_(x,o) _i^pro [ p_ _i(o x) ] (8) This objective initializes cross-model interaction capabilities in the heterogeneous models, providing a stable starting point for trajectory-level refinement of reasoning and interaction decisions. Step-Level Supervision. After protocol distillation, the heterogeneous models perform collaborative reasoning under the shared interaction protocol, generating trajectories for subsequent optimization. For a given training task q, a collaborative trajectory is sampled as τ=(z1,…,zT)∼Pℳ(τ∣q)τ=(z_1,…,z_T) P_M(τ q) (9) where PℳP_M is induced by the current models and the shared interaction protocol. Learning effective collaboration from these trajectories requires identifying the contribution of each reasoning and interaction decision to the overall process (Foerster et al., 2018). The value of an individual step depends not only on its local correctness but also on how it influences subsequent collaboration. We therefore retain the original large model MLM_L as an evaluator J, which assesses each step in the context of the complete trajectory. Given the task q and trajectory τ, the evaluator produces two complementary scores: (strea,stact)t=1T=(q,τ),strea,stact∈[−1,1] \ (s_t^rea,s_t^act ) \_t=1^T=J(q,τ), s_t^rea,s_t^act∈[-1,1] (10) The reasoning score streas_t^rea evaluates the model’s reasoning contribution in terms of correctness, informativeness, and relevance to the task state and interaction history. The interaction score stacts_t^act evaluates whether its decision to continue, hand off the task state, or terminate appropriately advances the collaborative process. Separating these dimensions avoids conflating reasoning quality with interaction quality. We average the two scores to obtain a step-level score: stdel=12(strea+stact)s_t^del= 12 (s_t^rea+s_t^act ) (11) Reward integration. Step-level scores provide fine-grained supervision for intermediate reasoning and interaction decisions, but local decision quality does not necessarily translate into successful task completion. To provide decision-specific feedback while also accounting for the overall task objective, we integrate each step-level score with the final task reward. Let m(q,y^)∈[0,1]m(q, y)∈[0,1] denote the task-specific evaluation metric for the final prediction y y. We map it to the same [−1,1][-1,1] range as the step-level score and define rtask=2m(q,y^)−1,rt=αrtask+(1−α)stdelr_task=2m(q, y)-1, r_t=α r_task+(1-α)s_t^del (12) where α∈[0,1]α∈[0,1] controls the trade-off between the task-level reward and the step-level score. The resulting reward distinguishes the contributions of intermediate steps across the complete collaborative trajectory while aligning their optimization with the final task objective. Collaborative Policy Refinement. Using the combined rewards, we extract the steps at which each model was active from the collaborative trajectories and construct model-specific training data. For model MiM_i, the corresponding training examples are collected as i=(xt(m),ot(m),rt(m))∣it(m)=iD_i= \(x_t^(m),o_t^(m),r_t^(m)) i_t^(m)=i \ (13) where ot(m)o_t^(m) denotes the complete response at step t of trajectory m, serialized under the shared interaction protocol and including its reasoning, answer, and interaction fields. Following the score-function policy-gradient form, we optimize each model with the reward-weighted objective ℒicol(θi)=−(x,o,r)∼i[rlogpθi(o∣x)]L_i^col( _i)=-E_(x,o,r) _i [r p_ _i(o x) ] (14) This objective increases the likelihood of responses assigned positive rewards and decreases the likelihood of those assigned negative rewards. Because each reward combines step-level evaluation conditioned on the complete trajectory with the final task outcome, each model can adapt its policy according to how its own decisions contribute to the collaborative result, allowing differentiated collaboration patterns to progressively emerge through training (Cui et al., 2026). 3 Experiments 3.1 Experimental Setup Datasets. We use five benchmarks across four task categories: multi-hop question answering (MuSiQue (Trivedi et al., 2022)), mathematical reasoning (GSM-Hard (Gao et al., 2023) and MATH (Hendrycks et al., 2021)), code generation (an eight-language MultiPL-E subset (Cassano et al., 2023))33 3 Following the language selection in the Qwen3 technical report (Yang et al., 2025a), we restrict code-generation evaluation to Python, C++, Java, PHP, TypeScript, C#, Bash, and JavaScript., and open-ended constrained generation (Conifer (Sun et al., 2024)). Baselines. (1) Single-agent baselines: Qwen3-14B (Yang et al., 2025a) and its self-evaluated RL variant (SE-RL)44 4 SE-RL uses Qwen3-14B self-feedback for comparison with cross-model feedback.. (2) A homogeneous MAS of three Qwen3-8B models (24B total) without collaborative training, serving as a higher-capacity multi-model baseline. (3) A homogeneous collaborative-learning system (Homo. CL) of three Qwen3-4B models (12B total), trained with the same procedure as our heterogeneous system to assess the effect of model heterogeneity in collaborative reasoning. (4) Established multi-agent frameworks: MAD (Du et al., 2024), AgentVerse (Chen et al., 2024c), AFlow (Zhang et al., 2025), and GPTSwarm (Zhuge et al., 2024). (5) MARL methods: MAGRPO (Liu et al., 2026b), MAPoRL (Park et al., 2025), and AT-GRPO (Zhao et al., 2026b). Metrics. We report accuracy and F1 on MuSiQue (Trivedi et al., 2022), GSM-Hard (Gao et al., 2023), and MATH (Hendrycks et al., 2021). We use weighted pass@1 and macro-language pass@1 for MultiPL-E (Cassano et al., 2023), and Coverage and Explicit for Conifer (Sun et al., 2024). Beyond task performance, we analyze handoffs, anomaly rates, reasoning redundancy, and model-wise behavioral distributions to characterize training-induced changes in collaboration patterns. Implementation Details. Model slicing replaces Qwen3-14B (MLM_L) (Yang et al., 2025a) with Qwen3-1.7B, Qwen3-4B, and Qwen3-8B (13.7B total). The three collaborating models share the same prompt and action space. Three Qwen3-8B instances generate protocol distillation demonstrations, while Qwen3-14B evaluates reasoning and interaction decisions within complete trajectories for step-level supervision. Experiments run on 8×8× NVIDIA A800-SXM4-80GB GPUs. Each model is trained with a rank-64 LoRA adapter (Hu et al., 2022) while its base parameters remain frozen. 3.2 Does Collaborative Learning Lead to Improved Performance? Table 1: Performance comparison with baseline methods. Scores are percentages (%); higher is better. Acc. denotes accuracy; Wtd. and Mac. denote weighted and macro-language pass@1, respectively; Cov. and Expl. denote Coverage and Explicit, respectively. CL. denotes collaborative learning. Best and second-best results are shown in bold and underline, respectively. ↑ [rgb]0.9453,0.375,0.2031 and ↓ [rgb]0.0039,0.7031,0.7266 indicate scores above and below the corresponding baseline mean for each metric in the table. † indicates that the corresponding baseline differs significantly from CL. (p≤0.05p≤ 0.05). !201-12-11 4,6,8,10,12,14 [l]2-1Method MuSiQue GSM-Hard MATH MultiPL-E Conifer Acc. F1 Acc. F1 Acc. F1 Wtd. Mac. Cov. Expl. Qwen3-14B 39.55†↑0.87_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.87$ 49.70†↑0.47_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.47$ 68.94↑1.65_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.65$ 80.69†↑3.19_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.19$ 74.60†↑4.40_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.40$ 82.86†↑3.25_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.25$ 55.18†↑1.08_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.08$ 57.34†↑2.59_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.59$ 88.00†↑0.17_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.17$ 95.72↑0.62_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.62$ SE-RL 41.75↑3.07_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.07$ 52.71†↑3.48_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.48$ 68.18↑0.89_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.89$ 79.67†↑2.17_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.17$ 76.00↑5.80_ [rgb]0.9453,0.375,0.2031 1.1$ \!5.80$ 80.70↑1.09_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.09$ 60.43†↑6.33_ [rgb]0.9453,0.375,0.2031 1.1$ \!6.33$ 62.55†↑7.80_ [rgb]0.9453,0.375,0.2031 1.1$ \!7.80$ 88.00↑0.17_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.17$ 95.81↑0.71_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.71$ Homo. MAS 26.40†↓12.28_ [rgb]0.0039,0.7031,0.7266 1.1$ \!12.28$ 32.82↓16.41_ [rgb]0.0039,0.7031,0.7266 1.1$ \!16.41$ 71.21↑3.92_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.92$ 79.93†↑2.43_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.43$ 68.20†↓2.00_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.00$ 76.41†↓3.20_ [rgb]0.0039,0.7031,0.7266 1.1$ \!3.20$ 56.36†↑2.26_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.26$ 57.20↑2.45_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.45$ 88.02†↑0.19_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.19$ 94.27†↓0.83_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.83$ Homo. CL. 36.00†↓2.68_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.68$ 47.19†↓2.04_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.04$ 60.61†↓6.68_ [rgb]0.0039,0.7031,0.7266 1.1$ \!6.68$ 76.51↓0.99_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.99$ 68.60†↓1.60_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.60$ 77.95↓1.66_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.66$ 53.33†↓0.77_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.77$ 52.65†↓2.10_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.10$ 88.15↑0.32_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.32$ 95.22†↑0.12_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.12$ MAD 39.51†↑0.83_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.83$ 50.59†↑1.36_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.36$ 66.67†↓0.62_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.62$ 74.18↓3.32_ [rgb]0.0039,0.7031,0.7266 1.1$ \!3.32$ 72.40†↑2.20_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.20$ 81.32↑1.71_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.71$ 55.03†↑0.93_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.93$ 55.35†↑0.60_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.60$ 88.55↑0.72_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.72$ 95.84↑0.74_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.74$ AgentVerse 41.99↑3.31_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.31$ 52.85†↑3.62_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.62$ 71.97↑4.68_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.68$ 78.44†↑0.94_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.94$ 68.20†↓2.00_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.00$ 79.42↓0.19_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.19$ 59.25↑5.15_ [rgb]0.9453,0.375,0.2031 1.1$ \!5.15$ 59.83†↑5.08_ [rgb]0.9453,0.375,0.2031 1.1$ \!5.08$ 87.74†↓0.09_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.09$ 95.86†↑0.76_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.76$ AFlow 41.83†↑3.15_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.15$ 50.36†↑1.13_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.13$ 68.18↑0.89_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.89$ 76.04↓1.46_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.46$ 65.00†↓5.20_ [rgb]0.0039,0.7031,0.7266 1.1$ \!5.20$ 76.53†↓3.08_ [rgb]0.0039,0.7031,0.7266 1.1$ \!3.08$ 42.83↓11.27_ [rgb]0.0039,0.7031,0.7266 1.1$ \!11.27$ 42.46†↓12.29_ [rgb]0.0039,0.7031,0.7266 1.1$ \!12.29$ 87.92†↑0.09_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.09$ 96.51†↑1.41_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.41$ GPTSwarm 36.20†↓2.48_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.48$ 47.37†↓1.86_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.86$ 70.45↑3.16_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.16$ 78.39†↑0.89_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.89$ 69.20†↓1.00_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.00$ 79.90↑0.29_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.29$ 51.63†↓2.47_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.47$ 52.43↓2.32_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.32$ 88.17†↑0.34_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.34$ 95.96†↑0.86_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.86$ MAGRPO 40.13†↑1.45_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.45$ 51.74↑2.51_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.51$ 65.91↓1.38_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.38$ 76.44↓1.06_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.06$ 65.00†↓5.20_ [rgb]0.0039,0.7031,0.7266 1.1$ \!5.20$ 76.01†↓3.60_ [rgb]0.0039,0.7031,0.7266 1.1$ \!3.60$ 49.93↓4.17_ [rgb]0.0039,0.7031,0.7266 1.1$ \!4.17$ 50.48†↓4.27_ [rgb]0.0039,0.7031,0.7266 1.1$ \!4.27$ 88.58†↑0.75_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.75$ 95.82†↑0.72_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.72$ MAPoRL 40.59†↑1.91_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.91$ 52.74†↑3.51_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.51$ 63.64†↓3.65_ [rgb]0.0039,0.7031,0.7266 1.1$ \!3.65$ 75.34†↓2.16_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.16$ 70.80†↑0.60_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.60$ 80.87†↑1.26_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.26$ 53.25†↓0.85_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.85$ 53.27†↓1.48_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.48$ 86.37†↓1.46_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.46$ 89.19†↓5.91_ [rgb]0.0039,0.7031,0.7266 1.1$ \!5.91$ AT-GRPO 41.54↑2.86_ [rgb]0.9453,0.375,0.2031 1.1$ \!2.86$ 53.51†↑4.28_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.28$ 64.39†↓2.90_ [rgb]0.0039,0.7031,0.7266 1.1$ \!2.90$ 76.86↓0.64_ [rgb]0.0039,0.7031,0.7266 1.1$ \!0.64$ 74.20†↑4.00_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.00$ 83.70†↑4.09_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.09$ 57.91†↑3.81_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.81$ 58.74†↑3.99_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.99$ 86.60†↓1.23_ [rgb]0.0039,0.7031,0.7266 1.1$ \!1.23$ 95.85†↑0.75_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.75$ CL. 42.37↑3.69_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.69$ 53.43↑4.20_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.20$ 71.97↑4.68_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.68$ 81.15↑3.65_ [rgb]0.9453,0.375,0.2031 1.1$ \!3.65$ 78.40↑8.20_ [rgb]0.9453,0.375,0.2031 1.1$ \!8.20$ 85.25↑5.64_ [rgb]0.9453,0.375,0.2031 1.1$ \!5.64$ 59.39↑5.29_ [rgb]0.9453,0.375,0.2031 1.1$ \!5.29$ 59.63↑4.88_ [rgb]0.9453,0.375,0.2031 1.1$ \!4.88$ 88.90↑1.07_ [rgb]0.9453,0.375,0.2031 1.1$ \!1.07$ 95.90↑0.80_ [rgb]0.9453,0.375,0.2031 1.1$ \!0.80$ As shown in Table 1, collaborative learning achieves the highest average performance across five benchmarks, with best or joint-best results on most metrics. With a total parameter budget of 13.7B, it outperforms the original Qwen3-14B on all ten metrics, with an average absolute gain of 2.38%, and surpasses SE-RL on eight. It also outperforms the 24B homogeneous MAS without collaborative training on all ten metrics, despite using fewer parameters. Compared with Homo. CL, which trains homogeneous models using the same procedure, the heterogeneous collaborative system achieves higher scores across all ten metrics, with an average absolute gain of 6.02%. Its average absolute gains over established MAS and MARL baselines range from 2.08% to 6.87% across the ten metrics, demonstrating competitive performance against both classes. Trajectory analysis reveals task-specific characteristics of collaboration and how intermediate behaviors translate into final outcomes. On MuSiQue, subsequent interactions correct initial errors, while on MATH, the trained heterogeneous models better preserve correct intermediate answers than MAPoRL. On MuSiQue and GSM-Hard, differences in partial-match scores on questions missed by both the collaborative system and the baseline help explain why accuracy and F1 rankings do not fully align. Code-generation performance varies across task subsets: the collaborative system solves more MBPP tasks but fewer HumanEval tasks than SE-RL. On most tasks solved only by SE-RL, it produces no passing intermediate program, suggesting that this gap primarily reflects failure to generate correct solutions rather than the loss of correct candidates during subsequent interactions. On Conifer, the collaborative system achieves the highest Coverage and almost always maintains compliance in the final output when an intermediate answer already satisfies all constraints. 3.3 How Much Does Each Mechanism Contribute to Performance? As shown in Figure 3, we compare three ablation variants: without collaborative training (w/o CL.), without protocol distillation (w/o PD.), and without the step-level score (w/o SLS.). Averaged over the ten reported metrics, these variants incur absolute score drops of 9.37, 8.85, and 5.41 points, respectively, and each underperforms the full method on every metric. These results indicate that heterogeneity alone is insufficient to achieve the full performance gains, and that both protocol distillation and step-level scoring contribute meaningfully to learned collaboration. Figure 3: Ablation study of the mechanism across five benchmarks. We compare a zero-shot variant that receives no training (w/o CL.), a variant without Protocol Distillation (w/o PD.), and a variant without the step-level score (w/o SLS.). Arrows indicate the gain of collaborative learning over the arithmetic mean of the available variants. For each benchmark, denotes Acc., Wtd., and Cov., while denotes the corresponding second metric (F1, Mac., or Expl.). Beyond aggregate performance, trajectory-level behavior provides a more direct view of how the two training components shape collaboration throughout collaborative reasoning. For example, without protocol distillation, first-answer EM on MuSiQue increases from 21.02% to 25.24%, yet the number of initially incorrect answers corrected by the final response falls from 594 to 171. Meanwhile, the proportion of Conifer trajectories that initially satisfy all constraints but fail at termination increases, and protocol-following failures on MATH also become more frequent. These observations suggest that protocol distillation primarily supports effective downstream interaction rather than simply improving the quality of the initial response. Similarly, removing the step-level score increases the average MuSiQue trajectory length from 3.4 to 3.8 steps while reducing the number of corrected answers by 18.5%, indicating that step-level feedback helps models use interaction steps more productively and achieve more corrections with fewer interactions on this task. 3.4 How Efficient and Effective Is Collaborative Reasoning? Alongside improvements in task performance, collaborative learning leads to more efficient inference. As shown in Figure 4, the collaborative system requires fewer model invocations than MAD, AgentVerse, AFlow, and GPTSwarm across all five benchmarks, with reductions ranging from 39.1% to 84.8%. Specifically, it requires an average of 4.0, 4.1, 1.8, 3.0, and 3.3 invocations per question on MuSiQue, GSM-Hard, MultiPL-E, MATH, and Conifer, respectively. This efficiency advantage also extends to the evaluated MARL baselines, with invocation reductions ranging from 46.7% to 86.0%. In terms of parameter-weighted output tokens, the collaborative system achieves the lowest count among the compared methods on four of five benchmarks. Compared with the MARL baselines, it further reduces parameter-weighted output tokens by 8.2%–89.4%. We further compare reasoning-overlap rates before and after collaborative learning to examine redundancy across successive reasoning steps. On MuSiQue, the overlap rates of the three collaborating models decrease from 48.3%, 24.1%, and 55.2% to 0.2%, 0.3%, and 0.5%, respectively. On MultiPL-E, they decrease from 34.8%, 2.3%, and 17.6% to 7.4%, 1.8%, and 10.7%. These results suggest that collaborative learning substantially reduces repetitive reasoning across model interactions, yielding less redundant subsequent contributions throughout the process. To further assess collaborative effectiveness, we measure correct-answer retention, defined as the proportion of trajectories whose final answer remains correct among those containing a correct intermediate candidate. On MuSiQue, GSM-Hard, MultiPL-E, and MATH, the trained heterogeneous models achieve 80.42%, 97.94%, 98.26%, and 98.00% retention, respectively, ranking first on the latter two tasks, second on GSM-Hard, and third on MuSiQue among comparison methods with recoverable intermediate candidates55 5 Conifer is excluded because its open-ended outputs lack an intermediate-candidate correctness criterion.. These results show that the trained models generally preserve available correct intermediate answers throughout subsequent collaborative interactions. Figure 4: Computation cost versus task performance. The center of each circle indicates task performance and parameter-weighted output tokens for the collaborative system and four multi-agent baselines across five benchmarks; its area is proportional to the number of model invocations. 3.5 What Collaboration Patterns Emerge from Collaborative Learning? Figure 5: Post-training interaction topologies across tasks. , , , and denote short linear, long linear, short cyclic, and long cyclic trajectories, respectively. Bar height indicates the proportion of each trajectory type, while depth indicates the average number of interaction steps. To examine how collaborative learning reorganizes model interactions across tasks, we compare trajectory topologies and interaction lengths before and after training. We categorize trajectories as short linear, long linear, short cyclic, or long cyclic. Linear trajectories involve each model at most once, with at most two interaction steps considered short; cyclic trajectories revisit a previously participating model, with at most four steps considered short. On MuSiQue, the proportion of long linear trajectories increases from 9.18% to 60.12%, showing a shift toward longer linear interactions. In contrast, this proportion decreases from 10.95% to 8.58% on MultiPL-E. Short cyclic trajectories evolve differently across tasks: their average interaction length decreases from 4.00 to 3.67 steps on GSM-Hard, but increases from 3.33 to 4.00 on MATH. These results show that collaborative learning does not uniformly increase interaction complexity or length, but instead induces task-dependent collaboration topologies rather than converging to a single universal interaction pattern. Task-dependent functional specialization also emerges across models during collaborative learning. On MuSiQue, Qwen3-4B’s answer-revision rate increases from 13% to 69%, its handoff rate from 48% to 96%, and its termination rate decreases from 50% to 4%, indicating a shift toward intermediate revision and handoff. Meanwhile, Qwen3-8B increasingly provides the final answer, with its share rising from 13.0% to 58.4%. This division of labor changes across tasks: on GSM-Hard, Qwen3-4B instead becomes the more frequent final-answer provider, with its share increasing from 34.09% to 64.39%. Thus, the same model can assume different functions across tasks, revealing task-dependent divisions of labor rather than a fixed collaboration structure. 4 Related Work Reasoning in LLMs. Existing work improves LLM reasoning by eliciting existing capabilities or strengthening them through training (Zhao et al., 2026a; Li et al., 2026c; Liu et al., 2026a). Inference-time methods use chain-of-thought prompting (Wei et al., 2022), systematic exploration (Hariri et al., 2026; Li et al., 2026a), evaluation (Wu and Raghunathan, 2026), and revision (Yang et al., 2026a); training-based methods learn from bootstrapped demonstrations (Zelikman et al., 2022; Dong et al., 2026), distill teacher reasoning (Yang et al., 2025b; Liu et al., 2026c; Fu et al., 2026), or use outcome- and process-level supervision. Yet these methods remain bounded by a single model’s knowledge and reasoning biases, leaving blind spots unaddressed and allowing intermediate errors to persist through subsequent reasoning (Xie et al., 2026). From Single-Model Reasoning to Multi-Model Composition. These limitations motivate moving beyond single-model reasoning toward multi-model composition, which exploits complementary knowledge and task-specific strengths across models (Yang et al., 2026b; Zhang et al., 2026c; Huot et al., 2026). Existing approaches select models suited to an input (Ong et al., 2025; Cao et al., 2026; Feng et al., 2026b; Fisch et al., 2026), distribute tasks or subtasks (Agashe et al., 2025; Chen et al., 2026), organize sequential model invocations (Dekoninck et al., 2025; Li et al., 2026b), determine when additional models are needed (Rajib et al., 2026; Qin et al., 2026), and aggregate candidate outputs (Zhang et al., 2026b). However, system-level composition does not necessarily constitute collaborative reasoning in which models jointly develop, evaluate, and revise a shared solution. Multi-Model Collaborative Reasoning. Collaborative reasoning directly addresses how models jointly develop solutions through interaction. Existing approaches range from hand-designed roles and protocols to optimized workflows (Hu et al., 2025; Zhou et al., 2026; Lee et al., 2026; Koh et al., 2026) and learned collaborative policies (Xue et al., 2026; Zhang et al., 2026a). Representative systems, however, often retain an explicitly prescribed division of labor (Zhao et al., 2026b; Feng et al., 2026a; Ding et al., 2026). In contrast, we train heterogeneous small models to collaborate without predefined roles under a total parameter budget comparable to that of a single large model. 5 Conclusion We introduced a collaborative learning framework for reasoning across heterogeneous models. Model slicing replaces a large model with smaller heterogeneous models under a comparable parameter budget, enabling reasoning-state isolation through task-state handoffs. Pyramid supervision enables cross-model collaboration by combining protocol distillation for interaction initialization with step-level supervision over model-generated trajectories. Across four task categories, the trained heterogeneous models match or outperform the large model and MAS and MARL baselines, while requiring fewer model invocations than multi-agent frameworks. Analysis reveals reduced reasoning redundancy, stronger correct-answer retention, and task-dependent interaction topologies and functional specialization. These results suggest that collaborative learning can transform model heterogeneity into complementary reasoning capabilities across distinct model parameterizations. Our study covers three model sizes and five benchmarks, leaving generalization to larger model pools, longer interaction horizons, and unseen task distributions unexplored. Pyramid supervision depends on inspector and evaluator quality, whose biases may affect step-level feedback. Comparable parameter counts do not imply equal computational cost, as collaborative inference adds latency and memory overhead beyond our parameter-weighted output-token metric. Future work will study generalization and robust supervision through multiple inspectors or uncertainty-aware aggregation. References Abdulaal et al. (2025) A. Abdulaal, C. Jin, N. Montaña-Brown, A. P. Gema, D. Castro, D. Alexander, P. Teare, T. Diethe, D. Oglic, and A. Saseendran Balancing act: diversity and consistency in large language model ensembles. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Agashe et al. (2025) S. Agashe, K. Wong, V. Tu, J. Yang, A. Li, and X. E. Wang Agent S2: a compositional generalist-specialist framework for computer use agents. In Conference on Language Modeling (COLM), External Links: Link Cited by: §4. Besta et al. (2024) M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler Graph of thoughts: solving elaborate problems with large language models. In AAAI Conference on Artificial Intelligence (AAAI), External Links: Link Cited by: §1. Cao et al. (2026) Q. Cao, S. Zhang, R. Zhou, R. Zhang, P. Qin, and P. Xie Models under SCOPE: scalable and controllable routing via pre-hoc reasoning. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §4. Cassano et al. (2023) F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation. In IEEE Transactions on Software Engineering (TSE), External Links: Link Cited by: §3.1, §3.1. Chen et al. (2023) B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao FireAct: toward language agent fine-tuning. In arXiv preprint arXiv:2310.05915, External Links: Link Cited by: §2.2.2. Chen et al. (2026) G. Chen, S. Yang, C. Li, W. Liu, J. Luan, and Z. Xu End-to-end optimization of LLM-driven multi-agent search systems via heterogeneous-group-based reinforcement learning. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §4. Chen et al. (2024a) J. Chen, S. Saha, and M. Bansal ReConcile: round-table conference improves reasoning via consensus among diverse LLMs. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §2.1. Chen et al. (2024b) J. Chen, S. Saha, E. Stengel-Eskin, and M. Bansal MAGDi: structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.2.2. Chen et al. (2024c) W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Chan, H. Yu, Y. Lu, Y. Hung, C. Qian, Y. Qin, X. Cong, R. Xie, Z. Liu, M. Sun, and J. Zhou AgentVerse: facilitating multi-agent collaboration and exploring emergent behaviors. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §3.1. Chen et al. (2025) W. Chen, Z. You, R. Li, Y. Guan, C. Qian, C. Zhao, C. Yang, R. Xie, Z. Liu, and M. Sun Internet of agents: weaving a web of heterogeneous agents for collaborative intelligence. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Cui et al. (2026) G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, J. Yuan, H. Chen, K. Zhang, X. Lv, S. Wang, Y. Yao, X. Han, H. Peng, Y. Cheng, Z. Liu, M. Sun, B. Zhou, and N. Ding Process reinforcement through implicit rewards. In Transactions on Machine Learning Research (TMLR), External Links: Link Cited by: §2.2.2. Dekoninck et al. (2025) J. Dekoninck, M. Baader, and M. Vechev A unified approach to routing and cascading for LLMs. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §4. Ding et al. (2026) J. Ding, R. Sun, X. Zhang, Z. Zhang, and X. Liu APEx: distillation of agent procedural experience for adaptive deep research question answering. In arXiv preprint arXiv:2609.02253, External Links: Link Cited by: §4. Dong et al. (2026) L. Dong, D. Zhang, Y. Sun, Q. Wang, Y. Wang, L. Chen, and J. Zhu Better starts, better ends: bootstrapped iterative self-reasoning distillation for compressed reasoning. In arXiv preprint arXiv:2607.15736, External Links: Link Cited by: §4. Du et al. (2024) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §1, §3.1. Feng et al. (2026a) L. Feng, L. Zheng, S. He, F. Zhang, and B. An Dr. MAS: stable reinforcement learning for multi-agent LLM systems. In arXiv preprint arXiv:2602.08847, External Links: Link Cited by: §4. Feng et al. (2026b) T. Feng, F. Yu, H. Zhang, Z. Dai, L. Yuan, Z. Lei, W. Zhang, K. Zhu, H. Yue, K. Xuan, G. Liu, and J. You LLMRouter: unified infrastructure for developing, evaluating, and deploying LLM routers. In arXiv preprint arXiv:2608.06867, External Links: Link Cited by: §4. Fisch et al. (2026) A. Fisch, S. Trivedi, F. Huot, W. W. Cohen, M. Kaisers, M. Lapata, K. Larson, and J. Eisenstein Pandora’s AI model routing box: efficient allocation with costly value estimation. In arXiv preprint arXiv:2608.20316, External Links: Link Cited by: §4. Foerster et al. (2018) J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), External Links: Link Cited by: §2.2.2. Fu et al. (2026) Z. Fu, B. He, Y. Zuo, H. Huang, J. Zhang, R. Xiao, C. Qian, Q. Luo, H. Gao, Y. Wang, Z. Liu, N. Ding, and C. Xiao Rethinking on-policy distillation of large language models I: one training example. In arXiv preprint arXiv:2609.04172, External Links: Link Cited by: §4. Gao et al. (2023) L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig PAL: program-aided language models. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §3.1, §3.1. Hariri et al. (2026) M. Hariri, W. Chen, N. Shahini, V. Singh, K. Ye, A. Samandar, D. Ganguly, S. Sankar, Y. Zhang, S. Wang, J. Peng, B. Zhang, M. Hinczewski, and V. Chaudhary Test-time scaling in reasoning LLMs: inference regimes, evaluation, and reproducibility. In arXiv preprint arXiv:2608.04001, External Links: Link Cited by: §4. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In Conference on Neural Information Processing Systems, Datasets and Benchmarks Track (NeurIPS), External Links: Link Cited by: §3.1, §3.1. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §3.1. Hu et al. (2025) S. Hu, C. Lu, and J. Clune Automated design of agentic systems. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Huang et al. (2024) J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. Yu, X. Song, and D. Zhou Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Huot et al. (2026) F. Huot, M. Kaisers, and M. Lapata When is routing meaningful? diversity and robustness in language model societies. In arXiv preprint arXiv:2607.09197, External Links: Link Cited by: §4. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §2.2.2. Koh et al. (2026) D. Koh, G. Kim, J. Bak, S. Na, T. Yang, H. Jang, and C. Park Global optimization and inference-time region grafting for agentic workflows. In arXiv preprint arXiv:2608.02353, External Links: Link Cited by: §4. Kumar et al. (2025) A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, L. Zhang, K. McKinney, D. Shrivastava, C. Paduraru, G. Tucker, D. Precup, F. Behbahani, and A. Faust Training language models to self-correct via reinforcement learning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Lee et al. (2026) J. H. Lee, K. H. Lee, and G. H. Yoo MEGA: self-evolving agent optimization infrastructure via wisdom graph. In arXiv preprint arXiv:2608.10504, External Links: Link Cited by: §4. Li et al. (2026a) M. Li, J. Qian, and W. Mou Predicting and improving test-time scaling laws via reward tail-guided search. In arXiv preprint arXiv:2602.01485, External Links: Link Cited by: §4. Li et al. (2026b) S. Li, A. M. Abdelmoniem, and S. Wang ProgRouter: online progress-guided orchestration for multi-agent LLM workflows under quality-cost tradeoffs. In Findings of the Association for Computational Linguistics: Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §4. Li et al. (2026c) Y. Li, S. Yavuz, and S. Joty RISE: recursive improvement via self-extrapolating policy distillation. In arXiv preprint arXiv:2609.05295, External Links: Link Cited by: §4. Liang et al. (2024) T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, and Z. Tu Encouraging divergent thinking in large language models through multi-agent debate. In Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §1. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Liu et al. (2026a) B. Liu, S. Yu, Y. Jiang, A. Qu, A. Zhao, Z. Liu, J. Kim, Z. Zhou, S. Kim, T. Ren, M. Liu, H. Yu, Z. Chen, W. Shi, P. P. Liang, L. Zettlemoyer, Y. Choi, and N. Jaques SPADE: self-play in adaptive synthetic executable environments. In arXiv preprint arXiv:2608.19197, External Links: Link Cited by: §4. Liu et al. (2026b) S. Liu, Z. Liang, X. Lyu, and C. Amato LLM collaboration with multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), External Links: Link Cited by: §1, §3.1. Liu et al. (2026c) Z. Liu, X. Xu, M. S. Seyfioglu, P. Xu, and K. Bouyarmane Extremely sparse supervision incentivizes reasoning ability. In arXiv preprint arXiv:2609.04565, External Links: Link Cited by: §4. Muennighoff et al. (2025) N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto s1: simple test-time scaling. In Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §1. Ong et al. (2025) I. Ong, A. Almahairi, V. Wu, W. Chiang, T. Wu, J. E. Gonzalez, M. Kadous, and I. Stoica RouteLLM: learning to route LLMs from preference data. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Park et al. (2025) C. Park, S. Han, X. Guo, A. E. Ozdaglar, K. Zhang, and J. Kim MAPoRL: multi-agent post-co-training for collaborative large language models with reinforcement learning. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §3.1. Qin et al. (2026) P. Qin, Q. Cao, and P. Xie ATLAS: agentic test-time learning-to-allocate scaling. In arXiv preprint arXiv:2606.01667, External Links: Link Cited by: §4. Rajib et al. (2026) R. H. Rajib, M. Zheng, and Q. Lou Learning what to retain: gated-memory routing for efficient collaboration in multi-agent LLM systems. In Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §4. Setlur et al. (2025) A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar Rewarding progress: scaling automated process verifiers for LLM reasoning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Snell et al. (2025) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Sun et al. (2024) H. Sun, L. Liu, J. Li, F. Wang, B. Dong, R. Lin, and R. Huang Conifer: improving complex constrained instruction-following ability of large language models. In arXiv preprint arXiv:2404.02823, External Links: Link Cited by: §3.1, §3.1. Trivedi et al. (2022) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal MuSiQue: multihop questions via single-hop question composition. In Transactions of the Association for Computational Linguistics (TACL), External Links: Link Cited by: §3.1, §3.1. Wang et al. (2025) J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Y. Zou Mixture-of-agents enhances large language model capabilities. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Wang et al. (2024a) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-Shepherd: verify and reinforce LLMs step-by-step without human annotations. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §1. Wang et al. (2024b) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Conference on Neural Information Processing Systems, Datasets and Benchmarks Track (NeurIPS), External Links: Link Cited by: §1. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §4. Wu and Raghunathan (2026) C. H. Wu and A. Raghunathan Self-trained verification for training- and test-time self-improvement. In arXiv preprint arXiv:2605.30290, External Links: Link Cited by: §4. Wu et al. (2025) Y. Wu, Z. Sun, S. Li, S. Welleck, and Y. Yang Inference scaling laws: an empirical analysis of compute-optimal inference for LLM problem-solving. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Xie et al. (2026) S. Xie, J. Liu, B. Yang, W. Yuan, F. Yang, T. Gao, and M. Wu From reasoning depth to reasoning breadth: evaluating multi-point associative reasoning in large language models. In arXiv preprint arXiv:2608.10444, External Links: Link Cited by: §4. Xu et al. (2024) W. Xu, G. Zhu, X. Zhao, L. Pan, L. Li, and W. Wang Pride and prejudice: LLM amplifies self-bias in self-refinement. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §1. Xue et al. (2026) X. Xue, Y. Zhou, G. Zhang, Z. Zhang, Y. Li, C. Zhang, Z. Yin, P. Torr, W. Ouyang, and L. Bai CoMAS: co-evolving multi-agent systems via interaction rewards. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Yang et al. (2026a) A. Yang, Q. Guo, Z. Huang, Y. Chen, S. Qian, and J. Cao Self-correcting long-horizon search agents via tree-structured memory. In arXiv preprint arXiv:2608.10676, External Links: Link Cited by: §4. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. In arXiv preprint arXiv:2505.09388, External Links: Link Cited by: §3.1, §3.1, footnote 3. Yang et al. (2025b) L. Yang, Z. Yu, T. Zhang, M. Xu, J. E. Gonzalez, B. Cui, and S. Yan SuperCorrect: advancing small LLM reasoning with thought template distillation and self-correction. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Yang et al. (2026b) Y. Yang, Y. Bian, Y. Tian, D. Fu, T. Huang, Y. Shi, Z. Xiao, N. Vasconcelos, and Y. Li Co-RL: unsupervised reasoning emerges from diverse cohort in multi-agent RL. In arXiv preprint arXiv:2608.17253, External Links: Link Cited by: §4. Yin et al. (2023) Z. Yin, Q. Sun, C. Chang, Q. Guo, J. Dai, X. Huang, and X. Qiu Exchange-of-thought: enhancing large language model capabilities through cross-model communication. In Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §2.1. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning. In Conference on Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §4. Zeng et al. (2024) A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang AgentTuning: enabling generalized agent abilities for LLMs. In Findings of the Association for Computational Linguistics: Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §2.2.2. Zhang et al. (2024) D. Zhang, S. Zhoubian, Z. Hu, Y. Yue, Y. Dong, and J. Tang ReST-MCTS*: LLM self-training via process reward guided tree search. In Conference on Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1. Zhang et al. (2025) J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. Wu AFlow: automating agentic workflow generation. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §3.1. Zhang et al. (2026a) Y. Zhang, Y. Zhao, N. Wang, Y. Wu, J. Chang, Y. Chen, Q. Wu, J. Zhao, and H. Wang MetaAgent-X: breaking the ceiling of automatic multi-agent systems via end-to-end reinforcement learning. In arXiv preprint arXiv:2605.14212, External Links: Link Cited by: §4. Zhang et al. (2026b) Z. Zhang, K. Shi, Z. Yuan, Z. Wang, T. Ma, K. Murugesan, V. Galassi, C. Zhang, and Y. Ye AgentRouter: a knowledge-graph-guided LLM router for collaborative multi-agent question answering. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §4. Zhang et al. (2026c) Z. Zhang, Z. Huang, G. Li, H. Wang, C. Yuan, X. Xia, D. Wang, F. Zhuang, S. Ma, N. Ding, Y. Yang, J. Li, and Y. Ban Heterogeneous agent collaborative reinforcement learning. In arXiv preprint arXiv:2603.02604, External Links: Link Cited by: §4. Zhao et al. (2026a) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §4. Zhao et al. (2026b) Y. Zhao, L. Hu, Y. Wang, M. Hou, H. Zhang, K. Ding, and J. Zhao Stronger-MAS: multi-agent reinforcement learning for collaborative LLMs. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §3.1, §4. Zheng et al. (2025) C. Zheng, Z. Zhang, B. Zhang, R. Lin, K. Lu, B. Yu, D. Liu, J. Zhou, and J. Lin ProcessBench: identifying process errors in mathematical reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §1. Zhou et al. (2026) H. Zhou, X. Wan, R. Sun, H. Palangi, S. Iqbal, I. Vulić, A. Korhonen, and S. Arik Multi-agent design: optimizing agents with better prompts and topologies. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Zhou et al. (2024) P. Zhou, J. Pujara, X. Ren, X. Chen, H. Cheng, Q. V. Le, E. H. Chi, D. Zhou, S. Mishra, and H. S. Zheng SELF-DISCOVER: large language models self-compose reasoning structures. In Conference on Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1. Zhuge et al. (2024) M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber GPTSwarm: language agents as optimizable graphs. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §3.1.