Paper deep dive
SAGE: Multi-Agent Self-Evolution for LLM Reasoning
Yulin Peng, Xinxin Zhu, Chenxing Wei, Nianbo Zeng, Leilei Wang, Ying Tiffany He, F. Richard Yu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/22/2026, 5:19:08 AM
Summary
SAGE (Self-evolving Agents for Generalized reasoning Evolution) is a closed-loop multi-agent framework that enables LLMs to co-evolve reasoning capabilities in math and coding domains. It utilizes four specialized agentsāChallenger, Planner, Solver, and Criticātrained from a shared LLM backbone using a small seed set and verifiable rewards. SAGE employs Task-Relative REINFORCE++ for stable, end-to-end training, demonstrating consistent performance gains across model scales and superior out-of-distribution generalization compared to existing self-play and multi-agent methods.
Entities (7)
Relation Signals (5)
SAGE ā includesagent ā Challenger
confidence 100% Ā· SAGE instantiates four specialized agents: a Challenger for task generation
SAGE ā includesagent ā Planner
confidence 100% Ā· a Planner for strategy outlining
SAGE ā includesagent ā Solver
confidence 100% Ā· a Solver for solution execution
SAGE ā includesagent ā Critic
confidence 100% Ā· a Critic for quality assessment and format calibration
SAGE ā optimizesmodel ā Qwen-2.5-7B
confidence 95% Ā· SAGE delivers consistent gains across model scales, improving the Qwen-2.5-7B model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning with verifiable rewards improves reasoning in large language models (LLMs), but many methods still rely on large human-labeled datasets. While self-play reduces this dependency, it often lacks explicit planning and strong quality control, limiting stability in long-horizon multi-step reasoning. We present SAGE (Self-evolving Agents for Generalized reasoning Evolution), a closed-loop framework where four agents: Challenger, Planner, Solver, and Critic, co-evolve from a shared LLM backbone using only a small seed set. The Challenger continuously generates increasingly difficult tasks; the Planner converts each task into a structured multi-step plan; and the Solver follows the plan to produce an answer, whose correctness is determined by external verifiers. The Critic scores and filters both generated questions and plans to prevent curriculum drift and maintain training signal quality, enabling stable self-training. Across mathematics and code-generation benchmarks, SAGE delivers consistent gains across model scales, improving the Qwen-2.5-7B model by 8.9% on LiveCodeBench and 10.7% on OlympiadBench.
Tags
Links
- Source: https://arxiv.org/abs/2603.15255v2
- Canonical: https://arxiv.org/abs/2603.15255v2
Trouble viewing inline? Open PDF directly ā
Full Text
49,702 characters extracted from source content.
Expand or collapse full text
SAGE: Multi-Agent Self-Evolution for LLM Reasoning Yulin Peng 1 , Xinxin Zhu 1, 2 , Chenxing Wei 1, 2 , Nianbo Zeng 1, 2 , Leilei Wang 1, 2 , Ying Tiffany He 1 , F. Richard Yu 3 1 College of Computer Science and Software Engineering, Shenzhen University, China 2 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China 3 School of Information Technology, Carleton University, Canada Abstract Reinforcement learning with verifiable rewards improves reasoning in large language models (LLMs), but many methods still rely on large human-labeled datasets. While self-play re- duces this dependency, it often lacks explicit planning and strong quality control, limiting stability in long-horizon multi-step reasoning. We present SAGE (Self-evolving Agents for Generalized reasoning Evolution), a closed- loop framework where four agents: Challenger, Planner, Solver, and Critic, co-evolve from a shared LLM backbone using only a small seed set. The Challenger continuously generates in- creasingly difficult tasks; the Planner converts each task into a structured multi-step plan; and the Solver follows the plan to produce an an- swer, whose correctness is determined by ex- ternal verifiers. The Critic scores and filters both generated questions and plans to prevent curriculum drift and maintain training signal quality, enabling stable self-training. Across mathematics and code-generation benchmarks, SAGE delivers consistent gains across model scales, improving the Qwen-2.5-7B model by 8.9% on LiveCodeBench and 10.7% on OlympiadBench. 1 Introduction Large language models (LLMs) have achieved re- markable advancements in reasoning tasks such as mathematics and coding through reinforcement learning (RL) techniques (Guo et al., 2025; Sheng et al., 2025; Sun et al., 2024). However, these meth- ods often depend on large-scale human-curated datasets for verifiable rewards, posing scalability challenges and limiting autonomous adaptation as models approach superhuman capabilities (Zhao et al., 2025a; Chen et al., 2025). Recent efforts have explored self-play and multi- agent frameworks to enable self-evolution with- out extensive external data. For instance, self-play paradigms like SPIRAL (Liu et al., 2025) and Ab- Proposed Questions Format Check Pass Rate Quality Check Quality Check Questions + Plans Format Check Format Check Challenger Critic Solver Planner Figure 1: Overview of the SAGE framework. Four specialized agentsāChallenger, Planner, Solver, and Criticāinteract through quality filtering and format val- idation to enable closed-loop self-evolution. solute Zero (Zhao et al., 2025a) leverage verifiable environments for autonomous improvement, while multi-agent systems such as MARS (Yuan et al., 2025) and MAE (Chen et al., 2025) facilitate col- laborative reasoning through role specialization. Despite these advances, existing approaches strug- gle with open-ended domains lacking robust ver- ification and often fail to integrate planning for complex, multi-step tasks (Huang et al., 2025; Gao et al., 2025; Yue et al., 2025). To address these gaps, we propose SAGE (Self-evolving Agents for Generalized reasoning Evolution), a closed-loop multi-agent framework that enables LLMs to co-evolve in verifiable do- mains like math and coding using only minimal seed examples. As illustrated in Figure 1, SAGE in- stantiates four specialized agents: a Challenger for task generation, a Planner for strategy outlining, a Solver for solution execution, and a Critic for qual- ity assessment and format calibration. These agents interact adversarially, with the Challenger rewarded arXiv:2603.15255v2 [cs.AI] 17 Mar 2026 for difficulty and the Solver optimized via verifier- based correctness, forming a self-rewarding cycle trained end-to-end using task-relative policy gradi- ents. Through experiments on mathematics and cod- ing benchmarks, SAGE demonstrates signifi- cant performance gains, outperforming baselines trained on human-curated datasets in sample effi- ciency and generalization. We outline our contribu- tion as follows: ā¢We design a scalable multi-agent framework for self-evolving LLMs in reasoning tasks. ā¢We propose a dual-role Critic mechanism en- suring task quality and solution verification. ā¢We conduct empirical evidence of effective co-evolution in math and code domains under few-example settings. 2 Related Work Reinforcement Learning for LLM Reasoning. Early work applied RL (e.g., PPO (Schulman et al., 2017)) to language tasks, but recent research fo- cuses on reinforcement learning with verifiable re- wards (RLVR) for reasoning (Wan et al., 2025). For example, DeepSeek-R1 (Guo et al., 2025) shows that RLVR can extend an LLMās reasoning capabil- ities on math by training from correctness signals. WebAgent-R1 (Wei et al., 2025) is an end-to-end multi-turn RL framework that significantly boosts web navigation success using binary success re- wards. Critic-free RL variants (e.g., GRPO (Guo et al., 2025)) reduce training overhead, but typically still rely on human-curated or grounded environ- ments. Recent work has systematically character- ized agentic RL for LLMs, emphasizing capabil- ities like planning and self-improvement (Zhang et al., 2025; Wen et al., 2025; Wu et al., 2025). In contrast, SAGE learns from self-generated, verifi- able tasks with little external data. Multi-Agent LLM Systems. LLM-based multi- agent frameworks facilitate complex tasks via role specialization. MetaGPT (Hong et al., 2024) en- codes human-like workflows into a multi-agent as- sembly line, breaking down large tasks into sub- tasks among collaborating agents. CAMEL (Li et al., 2023) uses inception prompting to guide a society of role-playing agents, enabling study of cooperative behaviors in instruction-following tasks. MARS (Yuan et al., 2025) introduces a rein- forcement learning framework where multi-agent self-play enhances strategic reasoning capabilities across cooperative and competitive tasks. These systems demonstrate that coordinating multiple LLM agents can enhance performance on com- plex tasks (Zhao et al., 2025b; Zhu et al., 2025). MARFT (Liao et al., 2025) applies multi-agent rein- forcement fine-tuning to optimize LLM-based sys- tems, and MALT (Motwani et al., 2025), which di- vides reasoning into generation, verification, and re- finement steps using heterogeneous agents. SAGE extends this line by instantiating distinct agents (Challenger, Planner, Solver, Critic) within one LLM and jointly training them with shared feed- back. Self-Play and Self-Evolving Agents.Recent works explore self-play and self-evolution to im- prove LLMs autonomously. The SPIRAL (Liu et al., 2025) framework shows that self-play on zero-sum games can automatically induce gener- alizable reasoning strategies without human data. Absolute Zero (Zhao et al., 2025a) generates its own coding problems and uses a code executor as a verifier to self-critique and solve them, achieving strong math and coding reasoning without exter- nal data. Agentic Self-Learning (Sun et al., 2025) is a closed-loop framework unifying task genera- tion, policy execution, and reward modelling for LLM agents in search environments. Additional ap- proaches include AgentEvolver (Zhai et al., 2025) enables efficient self-evolving through curiosity- driven task generation and experience reuse, and Agent0 (Xia et al., 2025), which unleashes self- evolving agents via tool-integrated reasoning in a co-evolutionary curriculum-executor loop. While prior work has explored various components of self-evolving agents such as planning and task gen- eration (Gao et al., 2025; Fang et al., 2025; Belle et al., 2025), SAGE is distinguished by integrating planning and critic roles to decompose reasoning and jointly train all agents for improved stability and depth in math and code domains. 3 Preliminaries Multi-Agent Reasoning in Verifiable Domains. LetM Īø denote an LLM parameterized byĪø. In role-based multi-agent reasoning, multiple agents share a backbone model. Still, they are conditioned on different role instructions (e.g., proposer, plan- ner, solver, evaluator) to enhance robustness via Reference questions Optimized by Task-Relative REINFORCE++ Add to Dataset Generated Questions Verified Questions Generated Plans Question-Plan Pairs Sample from Dataset Challenger Solver Planner ķ„ 1 ķ„ 2 āÆ ķ„ ķ ķ„ 1 ķ„ 2 āÆ ķ„ ķ ķ 1 ķ 2 āÆ ķ ķ ķ„ 1 ,ķ 1 ķ„ 2 ,ķ 2 āÆ ķ„ ķ ,ķ ķ ķ¦ 1 ķ¦ 2 āÆ ķ¦ ķ Optimized by Task-Relative REINFORCE++ Optimized by Task-Relative REINFORCE++ Critic Critic Figure 2: The SAGE training pipeline. (1) The Challenger generates questions from reference examples, filtered by the Critic for quality; (2) verified questions expand the dataset; (3) sampled questions are processed by the Planner and Solver to produce solutions; (4) all agents are jointly updated using Task-Relative REINFORCE++ with per-role advantage normalization. collaboration and decomposition (Du et al., 2023; Liang et al., 2024). For a questionq, agents pro- duce structured answersa. In verifiable domains (mathematics, programming), a domain-specific verifierV gt (q,a,v) ā [0, 1]evaluates answer cor- rectness given a referencev(ground-truth or unit tests), enabling automatic reward computation with- out human annotation. Policy Gradient Optimization.To enable self-evolution, we frame agent optimization as reinforcement learning, maximizingJ (Īø) = E qā¼D,oā¼Ļ Īø [R(q,o)]whereDis the task distribu- tion,Ris the reward signal, andois the output. REINFORCE++ (Hu et al., 2025) is a critic-free method that computes the advantage asA t q,o = r(q,o)ā Ī² kl P T i=t KL(Ļ Īø ā„Ļ ref ) i with KL penalty to a reference policy, and applies global-batch nor- malization:A norm = (Aā Ī¼ B )/(Ļ B + Īµ). This stabilizes training and improves robustness across prompt distributions. To coordinate multiple agents with heterogeneous objectives, we adopt Task- Relative REINFORCE++ (Huang et al., 2025), which applies per-role advantage normalization: A role norm = rā Ī¼ role Ļ role + Īµ ,(1) whereĪ¼ role andĻ role are the mean and standard deviation computed over the corresponding role- specific batch. 4 The SAGE Framework SAGE is a fully automated, self-iterative evolution framework requiring only a small seed set with automatic verification signals. SAGE instantiates four agents from a shared LLM backboneM Īø : (1) Challenger generates challenging tasks with verifiers; (2) Planner produces solution plans; (3) Solver outputs final answers; and (4) Critic evalu- ates quality and format compliance. These agents engage in continuous co-evolution, with the train- ing workflow illustrated in Figure 2. In verifiable domains such as mathematics and programming, SAGE forms a closed-loop pipeline (challengeāplanāsolveācriticize) that com- bines multi-agent interactions with verifier-based reward signals. The Challenger and Solver co- evolve adversarially: the Solver is rewarded for verified correctness, while the Challenger receives difficulty rewards when the Solver fails under veri- fication, pushing the curriculum toward harder yet still solvable tasks. Quality filtering and verifier val- idation are applied to prevent dataset degradation and improve training stability. 4.1 Reward Design and Normalization Format reward. Across phases, SAGE applies a format rewardr f ā [0, 1]to stabilize self-training by enforcing required tags (e.g.,<question>, <answer>,<type>,<score>). In practice,r f is a soft score (not strictly binary): missing tags yield low reward, redundant tags may receive partial credit, and empty outputs fall back to a neutral value (e.g., 0.5). Score normalization.The Critic outputs scalar scores typically on a 1ā10 scale inside <score></score>, which are normalized to[0, 1] by Norm(s) = ļ£± ļ£“ ļ£² ļ£“ ļ£³ s,0ā¤ sā¤ 1, sā1 9 , 1 < sā¤ 10, 0.5,otherwise. (2) 4.2 Challenger Agent Training The Challenger proposes verifiable tasks to drive the Solverās learning. During training, the Chal- lenger policyĻ c is prompted with reference prob- lems sampled from a small human-curated seed setD(about 500 examples across datasets), where each seed item includes a problem statement and its verifier (ground-truth answer or executable tests). Given a reference item(q ref ,v ref ), the Challenger generates a new problemqand an associated veri- fier v in a constrained format: (q,v)ā¼ Ļ c (Ā·| q ref ,v ref ;Īø),(3) where Īø represents the shared LLM parameters. Composite reward. The Challenger receives (i) a quality scores q ā [0, 1]from the Critic (clarity, relevance, well-formedness), (i) a difficulty reward computed from the Solverās verified success rate, and (i) a format reward. Concretely, we estimate the Solver success by samplingN s answers and verifying them with V gt : a j ā¼ Ļ s (Ā·| q;Īø), j = 1,...,N s , Ģs gt (q,v) = 1 N s N s X j=1 V gt (q,a j ,v), r d (q,v) = 1ā Ģs gt (q,v). (4) Here,V gt (q,a,v) ā [0, 1]denotes the domain- specific verifier (e.g., exact-match/symbolic grad- ing for math or test pass rate for code),Ļ s denotes the Solver policy (formally introduced in Section 4.4). The Challenger reward is computed as r c (q,v) = 1 3 s q (q) + 1 3 r d (q,v) + 1 3 r f (o c ),(5) whereo c (resp.o p ,o s ,o cr ) denotes the raw textual output of the Challenger (resp. Planner, Solver, Critic). Algorithm 1 Training Process of SAGE Require: Base LLM Ļ base , iterations T , thresholds Ī±,Ī², sample size N s 1: Init agents Ļ c ,Ļ p ,Ļ s ,Ļ cr from Ļ base 2: Init datasetD āD 0 (each item has verifier) 3: for t = 1 to T do 4:Sample (q ref ,v ref )ā¼D ā· (1) Challenge Phase 5:(q t ,v t )ā Ļ c (Ā·| q ref ,v ref ) 6: s q ā Norm(Ļ cr (q t )); validate v t 7:Sample a j ā¼ Ļ s (Ā·| q t ) for j = 1,...,N s 8: Ģs gt ā 1 N s P N s j=1 V gt (q t ,a j ,v t ); r d ā 1ā Ģs gt 9:if s q ā„ Ī± and v t valid then 10: D āDāŖ(q t ,v t ) ;r c ā 1 3 s q + 1 3 r d + 1 3 r f (o c ) 11:else 12:r c ā 1 2 s q + 1 2 r f (o c ) 13:end if 14:ā· (2) PlanāSolve Phase 15:Sample (q,v)ā¼D; p t ā Ļ p (Ā·| q) 16: s p ā Norm(Ļ cr (q,p t )) 17:if s p ā„ Ī² then 18:a t ā Ļ s (Ā·| q,p t ;Īø); Ģs p ā s p 19:else 20:a t ā Ļ s (Ā·| q,ā ;Īø); Ģs p ā 0 21:end if 22: s gt ā V gt (q,a t ,v) 23: r p ā Ī» plan s p + Ī» f r f (o p ); 24: r s ā w p Ģs p + w c s gt + w f r f (o s ) 25: r cr ā r f (o cr )ā· (3) Joint Update 26:Update Ļ c ,Ļ p ,Ļ s ,Ļ cr using r c ,r p ,r s ,r cr 27: end for Quality filtering and difficulty suppression. To prevent dataset degradation, we filter low-quality questions with a thresholdĪ±(in this paper,Ī± = 0.7), and also validate the generated verifier (e.g., parsable and executable for code tests). Only can- didates that satisfy both criteria are added toD. Moreover, fors q < Ī±, we suppress the difficulty term to avoid rewarding āhardā but ill-posed tasks and use r c (q,v) = 1 2 s q (q) + 1 2 r f (o c ).(6) This stabilizes long-horizon self-training and miti- gates reward collapse. 4.3 Planner Agent Training The PlannerĻ p generates a structured planpfor a given questionq, encapsulated in<plan></plan> tags. The Critic evaluates the plan quality to pro- duce a normalized score s p ā [0, 1]. pā¼ Ļ p (Ā·| q;Īø), s p = Norm Critic(q,p) . (7) Ifs p meets a gating threshold (in this paper,Ī² = 0.3 ), the plan is provided to the Solver; otherwise, the Solver answers directly. For optimizing the Planner, we use a compos- ite reward that combines plan quality and format compliance: r p = Ī» plan s p + Ī» f r f (o p ),(8) whereĪ» plan andĪ» f are weighting coefficients (we use Ī» plan = Ī» f = 0.5 by default). 4.4 Solver Agent Training The Solver agent is tasked with generating final answers based on the given questionqand the plan p(if the plan passes Critic gating). The Solver policyĻ s produces an answera, typically wrapped in<answer></answer>tags or Markdown blocks: aā¼ Ļ s (Ā·| q, Ģp;Īø), Ģp = ( p, s p ā„ Ī², ā , s p < Ī². (9) Verifier-based composite reward (plan, cor- rectness, format). Solver correctness is com- puted by automatic verification in the target do- main (symbolic/metric-based grading for math, or execution-/test-based validation for code), yielding s gt ā [0, 1]. We combine plan quality, verified correctness, and format adherence as Ģs p = ( s p , s p ā„ Ī², 0, s p < Ī², (10) r s = w p Ģs p + w c s gt + w f r f (o s ), w p + w c + w f = 1. (11) In this paper, we use(w p ,w c ,w f ) = (0.2, 0.6, 0.2) as the default setting. If the plan score is unavail- able (e.g., when the planning module is disabled), we fall back to a simpler mixture of verified cor- rectness and format (e.g., 1 2 s gt + 1 2 r f ) to maintain robustness. In adversarial interaction with the Challenger, Solver failures under ground-truth verification con- tribute to the Challengerās difficulty reward (Eq. 5), forming a co-evolutionary loop that progressively pushes the curriculum toward harder yet solvable problems. 4.5 Critic: Scoring and Format Calibration The Critic provides two types of signals: (1) soft format rewardsr f ā [0, 1]by checking required tags, and (2) quality scores for Challenger ques- tions (s q ) and Planner plans (s p ), normalized via Eq. 2. Importantly, in the verifiable setting, cor- rectness is determined by the external verifierV gt rather than the Critic. The Critic policyĻ cr outputs a scalar score de- terministically: sā¼ Ļ cr (Ā·| x;Īø),(12) wherex ā (q,Ā·), (q,p)denotes the evaluation context (either a question alone or a question-plan pair). Optionally, we calibrate the Critic with a lightweight format-consistency objective r cr = r f (o cr ),(13) which reduces parsing failures and improves stabil- ity of downstream reward computation. 4.6 Multi-Agent Co-Training A training step in SAGE comprises: (1) Challenger Phase to generate verifiable candidate tasks and expandDwith quality-and-verifier filtering; (2) PlanāSolve Phase where the Planner generates a single plan scored by the Critic and the Solver is optimized using the verifier-based reward in Eq. 10; (3) Critic Phase (optional) for format calibration; and (4) Synchronized Update that jointly updates the shared backbone using Task-Relative REIN- FORCE++ with per-role advantage normalization (see Section 3). 5 Experiments 5.1 Experimental Setup Training details Our framework is implemented based on VeRL(Sheng et al., 2025), and we evalu- ate it using the Qwen2.5-3B-Instruct, Qwen2.5-7B- Instruct, and Qwen3-4B-Base models(Yang et al., 2025b,a). All agents are initialized from their corre- sponding base models. We apply LoRA (Hu et al., 2021) with rank 128 and a learning rate of 3e-6. Additional hyperparameter settings are provided in Table 4. Baseline Methods. To comprehensively assess the effectiveness of the proposed SAGE frame- work, we conduct experiments on several repre- sentative foundation models, including Qwen2.5- 3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3- 4B-Base. For each model, we report results for both the original checkpoint and the correspond- ing variant fine-tuned with SAGE. In addition, we include Absolute-Zero-Reasoning (AZR) (Zhao et al., 2025a) and Multi-Agent Evolve (MAE) (Chen et al., 2025) as alternative training baselines. Specifically, each model is trained for 200 steps under AZR. For MAE, we adopt the half-reference setting and train each model for 200 steps. MethodHEval+ MBPP+ LCB v1ā5 GSM8K Math AI24 AI25 AMC OlympiadC Avg. M Avg. O Avg. Qwen-2.5-3B-Instruct Base Model68.360.612.084.660.43.36.740.028.046.937.240.4 AZR 68.961.415.081.262.43.33.335.028.948.435.739.9 MAE68.361.115.982.265.83.33.332.532.548.436.640.5 SAGE68.962.416.985.566.26.76.735.029.849.438.342.0 Qwen-2.5-7B-Instruct Base Model73.265.317.591.775.1 13.36.757.528.052.045.447.6 AZR71.369.125.392.876.2 10.0 13.3 50.038.555.246.849.6 MAE76.265.323.391.776.2 13.3 13.3 42.532.754.945.048.3 SAGE76.264.026.492.274.7 13.3 13.3 52.538.755.547.550.1 Qwen-3-4B-Base Base Model76.865.321.594.587.0 16.7 13.3 77.549.054.556.355.7 AZR74.465.026.189.376.2 10.0 13.3 50.041.555.246.749.5 MAE 76.265.324.294.592.0 13.3 10.0 70.043.755.253.954.4 SAGE75.662.430.694.391.0 16.7 10.0 75.047.956.255.855.9 Table 1: Main results on reasoning benchmarks. Comparison of post-training methods across three model scales. We report pass@1 accuracy (%) on code generation (HumanEval+, MBPP+, LiveCodeBench) and mathematical reasoning (GSM8K, MATH, AIME 2024, AIME 2025, AMC, and OlympiadBench). C Avg., M Avg., and O Avg. denote the mean scores over code, math, and all benchmarks. SAGE achieves the best overall performance across all three model backbones. Bold indicates best per LLM backbone. Training and Evaluation Datasets. Our training set comprises 500 instances sampled from MATH (Hendrycks et al., 2021a), GSM8K (Cobbe et al., 2021), HumanEval (Chen et al., 2021), and MBPP (Austin et al., 2021), with detailed statistics in Appendix B. We evaluate on two domains: (1) Mathematical Reasoning: GSM8K and MATH (in- distribution, ID), along with four competition-level benchmarksāAIMEā24, AIMEā25, Olympiad- Bench (He et al., 2024), and AMCā23 (Hendrycks et al., 2021b)āas out-of-distribution (OOD) tests. (2) Code Generation: HumanEval+ and MBPP+ evaluated via Evalplus (Liu et al., 2023) (ID), and LiveCodeBench (Jain et al., 2024) v1āv5 (May 2023āFebruary 2025) for OOD assessment. We report the accuracy (pass@1) based on greedy de- coding across all benchmarks. 5.2 Main Results Table 1 presents the performance of SAGE and baseline methods across code generation and math- ematical reasoning benchmarks on three model backbones. Consistent Improvements Across Model Scales. SAGE achieves the highest Overall Avg. on both Qwen-2.5-3B-Instruct (42.0%) and Qwen-2.5-7B- Instruct (50.1%), outperforming all baselines in- cluding AZR and MAE. On the 3B model, SAGE improves upon the base model by 1.6% overall, with notable gains on in-distribution benchmarks BackboneMethodID Avg.OOD Avg. Qwen-2.5-3B Base Model68.418.0 AZR68.517.1 MAE69.417.5 SAGE70.819.0 Qwen-2.5-7B Base Model76.324.6 AZR77.427.4 MAE76.825.0 SAGE77.428.8 Qwen-3-4B Base Model80.935.6 AZR76.228.2 MAE82.032.2 SAGE80.836.0 Table 2: ID and OOD generalization comparison. SAGE consistently improves OOD performance (+4.2% on 7B) without sacrificing in-distribution accuracy. (GSM8K: 84.6%ā85.5%; MATH: 60.4%ā 66.2%). Similarly, on the 7B model, SAGE yields a 2.5% improvement over the base model in Overall Avg., demonstrating consistent effectiveness across model scales. Strong Out-of-Distribution Generalization. A key strength of SAGE lies in its generalization to out-of-distribution benchmarks. As shown in Table 2, SAGE achieves the best or near-best OOD Avg. across all three backbones (19.0%, 28.8%, and 36.0% respectively), while maintain- ing competitive ID Avg. scores. This balanced improvement is particularly evident on Qwen-2.5- 7B, where SAGE improves OOD Avg. by 4.2% MethodHEval+ MBPP+ LCB v1ā5 GSM8K Math AI24 AI25 AMC OlympiadC Avg. M Avg. O Avg. SAGE (full implementation) 68.962.416.985.566.26.76.735.029.849.438.342.0 SAGE (w/o challenger training) 66.561.39.086.765.50.03.335.028.045.636.439.5 SAGE (w/o solver training) 67.764.39.081.260.43.30.030.028.047.033.838.2 SAGE (w/o critic training) 66.553.714.186.065.93.36.740.027.444.838.240.4 Table 3: Ablation study of SAGE components on Qwen-2.5-3B. We evaluate the impact of removing individual agent training while keeping other components active. over the base model while preserving strong in- distribution performance.On LiveCodeBench specifically, SAGE achieves the best performance across all three backbones (16.9%, 26.4%, and 30.6%), substantially outperforming both base models and other post-training methods.For mathematical reasoning, SAGE maintains competi- tive performance on competition-level benchmarks such as OlympiadBench, where it achieves 38.7% (+10.7% over base) on Qwen-2.5-7B. Comparison with Baselines. While AZR and MAE show improvements on certain individual benchmarks, they exhibit inconsistent gains and occasional performance degradation. For instance, AZR on Qwen-3-4B-Base leads to a significant drop in Math Avg. (56.3%ā46.7%). In con- trast, SAGE maintains more balanced improve- ments across both domains without sacrificing per- formance on any benchmark group. Results on Qwen-3-4B. On this stronger back- bone, the base model already achieves high per- formance (Overall Avg. 55.7%). Nevertheless, SAGE attains the highest Code Avg. (56.2%) and remains competitive overall (55.9%), with partic- ularly strong gains on LiveCodeBench (21.5%ā 30.6%, +9.1%). This suggests that SAGE continues to provide meaningful improvements even when applied to capable base models. 5.3 Ablations Studies and Analyses Ablation Study. To understand the contribution of each agent, we conduct ablation experiments by selectively disabling the training of individual roles while keeping the remaining components active. As shown in Table 3, the full SAGE implementation achieves the highest overall average (42.0%), and removing any single agent leads to performance degradation. Disabling Challenger training results in a no- table drop in code benchmarks, particularly on LiveCodeBench (16.9%ā9.0%), indicating that curriculum generation is essential for out-of- distribution generalization. Similarly, removing Solver training causes the largest overall decline (O Avg. 38.2%), with substantial drops on both GSM8K (85.5%ā81.2%) and MATH (66.2% ā60.4%), confirming that the Solver is the pri- mary driver of reasoning capability. Interestingly, excluding Critic training yields competitive math performance (M Avg. 38.2%) but degrades code benchmarks (C Avg. 44.8%), suggesting that the Criticās quality filtering is more critical for code generation where output format and correctness are tightly coupled. These results validate that all three trainable agents contribute complementarily to SAGEās over- all effectiveness, with the ChallengerāSolver inter- action forming the core co-evolutionary loop and the Critic providing essential quality control. 4080120160200240 Step 0 5000 10000 15000 20000 Valid Questions 0.3 0.4 0.5 0.6 0.7 Validation Accuracy (%) Valid Questions Validation Accuracy (%) Figure 3: Training dynamics on Qwen-2.5-3B. The Challenger steadily expands the question pool (bars) throughout training, while validation accuracy (line) reaches peak performance around step 100ā120 before gradual decline, suggesting potential over-specialization on the self-generated curriculum. Training Dynamics Analysis. To gain deeper in- sights into the self-evolution process, we analyze the training dynamics of SAGE on Qwen-2.5-3B- Instruct, as shown in Figure 3. The validation accuracy (line) exhibits a charac- Question Judged by Critic Agent Challenger Solver Planner Thehospitalhas11doctorsand18nurses.If5doctorsand2nursesquit, howmanydoctorsandnursesareleft? 1.Determinetheinitialnumberofdoctorsandnurses. 2.Calculatetheremainingdoctorsafterthedoctorsquit. 3.Calculatetheremainingnursesafterthenursesquit. 4.Sumtheremainingdoctorsandnursestogetthetotalnumberleft. 1.Let'sdeterminetheinitialnumbers: -Initialnumberofdoctors:11 -Initialnumberofnurses:18 2.Now,let'saccountforthedoctorswhoquit: -Numberofdoctorswhoquit:5 -Remainingdoctors:11-5=6 3.Next,let'saccountforthenurseswhoquit: -Numberofnurseswhoquit:2 -Remainingnurses:18-2=16 4.Finally,weneedtofindthetotalnumberremaining: -Totalremainingdoctorsandnurses:6doctors+16nurses=22 people.So,after5doctorsand2nursesquit,thehospitalhas22 peopleleft(6doctorsand16nurses). Sotheansweris 22. <score>7</score> Let'sevaluatetheproposedplanstep-by-stepagainstthegivenrubric: 1.**Clarity**:Theplanisclearandwell-structured.Eachstepis straightforwardandeasytounderstand. 2.**Completeness**:Theplancoversallnecessarystepstosolvethe problem.Itcorrectlyidentifiestheinitialnumbers,calculatesthe reductions,andsumstheremainingpersonnel. 3.**Correctness**:Thereasoningandapproacharelogicallysound. Subtractingthenumberofdoctorsandnurseswhoquitfromtheir initialcountswillindeedgivethecorrectremainingnumbers. 4.**Feasibility**:Followingthisplanwillleadtoacorrectsolution. Thecalculationsarestraightforwardanddonotinvolveanylogical errors. 5.**Efficiency**:Theapproachisreasonablyefficientasitavoids unnecessarysteps.Onlytherelevantarithmeticoperationsare performed. Giventheseevaluations,theplanmeetsallthecriteriaeffectivelyand efficiently. <score>8</score> Question Generated by Challenger Agent Plan Generated by Planner Agent Answer Output by Solver Agent Plan Judged by Critic Agent Figure 4: Qualitative case study. The Challenger generates a math word problem, the Planner decomposes it into structured steps, the Solver executes the plan to produce the final answer, and the Critic provides quality scores for both the question and the plan. teristic learning curve. During the initial phase (steps 0ā80), the model demonstrates rapid im- provement from 29.1% to 65.8%, reflecting effi- cient knowledge acquisition from the multi-agent co-evolutionary training. The accuracy reaches its peak of 69.5% around step 100ā140, repre- senting the optimal balance between task diffi- culty and model capability. Beyond this point, we observe a gradual decline to 61.6% by step 240, suggesting that prolonged training may lead to over-specialization on the self-generated curricu- lum. This motivates our choice of reporting results around step 100 in the main experiments. Meanwhile, the cumulative number of valid questions (bars) grows steadily throughout training, expanding from 1,136 to 20,532 by step 250, an 18-fold increase from the seed set. Notably, the growth rate accelerates around step 120ā130, co- inciding with peak validation accuracy, suggesting that a well-trained Challenger produces questions that pass the quality thresholdĪ± = 0.7at an in- creasing rate. The continued growth of the question pool despite declining accuracy after step 120 sug- gests that increased quantity alone does not ensure better performance, highlighting the importance of curriculum diversity and difficulty calibration. Nevertheless, this trend demonstrates SAGEās abil- ity to autonomously scale its training data without human intervention. Qualitative Analysis. Figure 4 illustrates the col- laborative reasoning process of SAGE. The Chal- lenger generates a well-formed arithmetic problem involving subtraction across two categories. The Planner decomposes this into four sequential steps, progressing from initial value identification to final summation. Guided by this structured plan, the Solver executes each step systematically and ar- rives at the correct answer. The Critic evaluates both outputs, assigning scores of 7 and 8 based on clarity, completeness, and logical soundness. This example highlights how role specialization enables effective division of labor: task generation, strategic planning, solution execution, and qual- ity assessment operate as distinct yet coordinated functions within a unified training loop. 6 Conclusion We introduce SAGE, a multi-agent self-evolution framework where four specialized agents: Chal- lenger, Planner, Solver, and Critic, co-evolve through adversarial yet collaborative dynamics. Starting from minimal seed examples, SAGE au- tonomously expands its training curriculum while maintaining quality via critic-based filtering. Ex- periments demonstrate consistent improvements across model scales, with strong out-of-distribution generalization on competition-level benchmarks. These results highlight a scalable and effective path- way for evolving capable reasoning agents while reducing dependency on human-curated supervi- sion. 7 Limitations Among the limitations of our work, firstly, SAGE operates in verifiable domains where correctness can be automatically determined through ground- truth answers or executable tests. Extending the framework to open-ended tasks with subjective evaluation criteria, potentially through learned re- ward models, remains an interesting direction for future work. Secondly, although SAGE signifi- cantly reduces reliance on large-scale annotations, it still requires a small seed set (500 examples) to bootstrap the self-evolution process. Investi- gating strategies to further minimize seed require- ments could broaden applicability to extremely low- resource scenarios. Thirdly, our evaluation focuses on mathematical reasoning and code generation benchmarks. Future exploration of other structured reasoning domains, such as logical reasoning or scientific problem solving, could offer valuable in- sights and validate the generalizability of our multi- agent architecture. Additionally, as with standard self-training approaches, monitoring training dy- namics and applying early stopping is advisable to ensure optimal performance. References Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. Preprint, arXiv:2108.07732. Nikolas Belle, Dakota Barnes, Alfonso Amayuelas, Ivan Bercovich, Xin Eric Wang, and William Wang. 2025. Agents of change: Self-evolving llm agents for strate- gic planning. Preprint, arXiv:2506.04651. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating Large Language Models Trained on Code. Preprint, arXiv:2107.03374. Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, and Jiaxuan You. 2025. Multi-Agent Evolve: LLM Self-Improve through Co-evolution. Preprint, arXiv:2510.23595. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Prob- lems. Preprint, arXiv:2110.14168. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023.Im- proving Factuality and Reasoning in Language Models through Multiagent Debate. Preprint, arXiv:2305.14325. Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, Zhaochun Ren, Nikos Aletras, Xi Wang, Han Zhou, and Zaiqiao Meng. 2025. A Compre- hensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems. Preprint, arXiv:2508.07407. Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, and 8 others. 2025. A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence. Preprint, arXiv:2507.21046. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhu- oshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645:633ā638. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yu- jie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 3828ā3850, Bangkok, Thailand. Association for Computational Linguistics. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring Massive Multitask Language Un- derstanding. Preprint, arXiv:2009.03300. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. In Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and JĆ¼rgen Schmidhuber. 2024. MetaGPT: Meta Pro- gramming for A Multi-Agent Collaborative Frame- work. Preprint, arXiv:2308.00352. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adap- tation of Large Language Models. Preprint, arXiv:2106.09685. Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. 2025. REINFORCE++: Stabilizing Critic-Free Pol- icy Optimization. Preprint, arXiv:2501.03262. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hong- ming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. 2025.R-zero: Self- evolving reasoning llm from zero data. Preprint, arXiv:2508.05004. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- CodeBench: Holistic and Contamination Free Evalu- ation of Large Language Models for Code. Preprint, arXiv:2403.07974. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for "mind" exploration of large language model society. In Advances in Neural Information Processing Systems, volume 36, pages 51991ā52008. Curran Associates, Inc. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shum- ing Shi. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent De- bate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18335ā18345, Miami, Florida. Association for Computational Linguistics. Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. 2025. MARFT: Multi-Agent Reinforcement Fine-Tuning. Preprint, arXiv:2504.16129. Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. 2025. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning. Preprint, arXiv:2506.24119. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. Is Your Code Gen- erated by ChatGPT Really Correct? Rigorous Eval- uation of Large Language Models for Code Genera- tion. In Advances in Neural Information Processing Systems, volume 36, pages 21558ā21572. Curran As- sociates, Inc. Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jy- oti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. 2025. MALT: Improving Rea- soning with Multi-Agent LLM Training. Preprint, arXiv:2412.01928. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.Proxi- mal Policy Optimization Algorithms. Preprint, arXiv:1707.06347. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexi- ble and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279ā1297, Rotterdam, The Nether- lands. ACM. Chuanneng Sun, Songjun Huang, and Dario Pom- pili. 2024. Llm-based multi-agent reinforcement learning: Current and future directions. Preprint, arXiv:2405.11106. Wangtao Sun, Xiang Cheng, Jialin Fan, Yao Xu, Xing Yu, Shizhu He, Jun Zhao, and Kang Liu. 2025. To- wards Agentic Self-Learning LLMs in Search Envi- ronment. Preprint, arXiv:2510.14253. Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, and Ying Wen. 2025.Rema: Learning to meta-think for llms with multi-agent reinforcement learning. Preprint, arXiv:2503.09501. Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. 2025. WebAgent-R1: Training Web Agents via End-to- End Multi-Turn Reinforcement Learning. Preprint, arXiv:2505.16421. Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhi- rong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. 2025. Reinforcement learning with verifiable rewards im- plicitly incentivizes correct reasoning in base llms. Preprint, arXiv:2506.14245. Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xue- meng Yang, Yufan Shen, Yuxin Wang, and Botian Shi. 2025. EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle. Preprint, arXiv:2510.16079. Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. 2025. Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning. Preprint, arXiv:2511.16043. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Day- iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jian- hong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 oth- ers. 2025b. Qwen2.5 Technical Report. Preprint, arXiv:2412.15115. Huining Yuan, Zelai Xu, Zheyue Tan, Xiangmin Yi, Mo Guang, Kaiwen Long, Haojia Hui, Boxun Li, Xinlei Chen, Bo Zhao, Xiao-Ping Zhang, Chao Yu, and Yu Wang. 2025. MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs. Preprint, arXiv:2510.15414. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize rea- soning capacity in llms beyond the base model? Preprint, arXiv:2504.13837. Yunpeng Zhai, Shuchang Tao, Cheng Chen, Anni Zou, Ziqian Chen, Qingxu Fu, Shinji Mai, Li Yu, Jiaji Deng, Zouying Cao, Zhaoyang Liu, Bolin Ding, and Jingren Zhou. 2025. AgentEvolver: Towards Efficient Self-Evolving Agent System. Preprint, arXiv:2511.10395. Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Song- tao Huang, Yue Liao, Hongru Wang, Mengyue Yang, and 6 others. 2025. The Landscape of Agentic Rein- forcement Learning for LLMs: A Survey. Preprint, arXiv:2509.02547. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. 2025a. Absolute zero: Reinforced self-play reasoning with zero data. Preprint, arXiv:2505.03335. Yujie Zhao, Lanxiang Hu, Yang Wang, Minmin Hou, Hao Zhang, Ke Ding, and Jishen Zhao. 2025b. Stronger-mas: Multi-agent reinforcement learning for collaborative llms. Preprint, arXiv:2510.11062. Guobin Zhu, Rui Zhou, Wenkang Ji, and Shiyu Zhao. 2025. Lamarl: Llm-aided multi-agent reinforcement learning for cooperative policy generation. Preprint, arXiv:2506.01538. A Hyperparameter Settings Table 4: Training Hyperparameters of our experiments. HyperparameterValue Training Configuration Batch Size128 Learning Rate3Ć 10 ā6 Training Steps200 Generation Settings Maximum Prompt Length8192 Maximum Response Length8192 Challenger Temperature0.6 Planner Temperature0.6 Solver Temperature0.6 Critic Temperature0.1 Algorithm Settings Learning AlgorithmTask-Relative REINFORCE++ KL RegularizationDisabled LoRA Configuration LoRA Rank128 LoRA Alpha256 LoRA Dropout0.95 Target Modulesq proj , k proj , v proj , o proj , gate proj , up proj , down proj B Training Data Composition Table 5 presents the composition of the 500 training instances sampled from four benchmark datasets. These samples are drawn from the official training splits and serve as the foundation for our training procedure. Table 5: Distribution of Training Samples Across Bench- marks BenchmarkCount MATH (Hendrycks et al., 2021a)156 GSM8K (Cobbe et al., 2021)148 HumanEval (Chen et al., 2021)87 MBPP (Austin et al., 2021)109 Total500 C Prompts for Agents Here, we list the prompt of each agent as follows. Challenger Agent Prompt Role:TaskDesignerAgent Description: Youareataskgenerationspecialist.Yourgoalistocreateasingle,high-qualityevaluationtaskthatchallengescomplexreasoningabilities. DesignConstraints: -Self-containedwithclearproblemstatement -Non-trivial:requiresmultiplereasoningstepsorconstraintsatisfaction -Deterministicortightlybounded(avoidsubjectivejudgment) -Culturallyneutral,noreal-timedatadependency -Difficultbutsolvable Avoid: -Triviaoropinion-basedprompts -Ambiguoussuccesscriteria -Web-dependentortime-sensitivecontent -Unsolvableorill-definedproblems Respondusing: <question> [Yourgeneratedtaskhere] </question> Figure 5: The prompt of the Challenger Agent. Planner Agent Prompt Role:PlannerAgent Description: Youwillreviewtheuserproblemandproposeaconciseplanthatasolvercanfollow. Problem:question Respondusing: <plan> 1.... 2.... </plan> Figure 6: The prompt of the planner Agent. Solver Agent Prompt Role:SolverAgent Description: Youwillsolvetheproblembyfollowingtheverifiedplanandprioritizingcorrect,well-reasonedcontentoverformattingtricks. Input: -Problem:question -VerifiedPlan:plan Instructions: -Explainthekeyreasoningstepsclearly -Followtheanswer-formatinstructionintheproblemstatementexactly -Donotintroduceadditionalwrappers/tagsunlessexplicitlyrequired Figure 7: The prompt of the Solver Agent. Critic Agent Prompt(Question) Role:QuestionQualityCriticAgent Description: Youareanexpertevaluator.Yourtaskistoassessthequalityofageneratedquestionforreasoningbenchmarks. Input: -Questiontoevaluate:question Evaluatethequestionbasedonthefollowingcriteria: <think> EvaluationCriteria: -Solvability:Isthequestionsolvablewithsufficientinformation?Nointernalcontradictions? -LogicalSoundness:Isthequestionlogicallycoherentanddoesnotviolatecommonsense? -Clarity:Isthewordingunambiguouswithclearobjectivesandconstraints? -Appropriateness:Isitsafe,relevant,andactuallyintheformofaquestion? -Conciseness:Isitfreefromredundantrepetitionorunnecessarycomplexity? ScoringGuidelines: -8-10:Excellentquestion-fullyclear,logicallysound,solvable,self-contained,andconcise.Appropriateforevaluationpurposes. -4-7:Acceptablequestion-hassomeambiguityormissingdetailsbutnofatalflawsinsolvabilityorlogic. -1-3:Poorquestion-unsolvable,contradictory,violatescommonsense,unsafe,tooopen-ended,ornotavalidquestion. CriticalRule: Ifanyunsolvabilityorcommonsenseviolationexists,scoremustbe[1-3]. [Writeyourdetailedanalysishere,addressingeachcriterion.] < > Provideyourfinalscore. Important: Outputonlyonetaglike<score>7</score>(replace7withyourintegerscore1-10). Figure 8: The prompt of the Critic Agent(question). Critic Agent Prompt(Plan) Role:PlanCriticAgent Description: Youareanexpertevaluator.Yourtaskistoassessthequalityofaproposedplanforsolvingaproblem. Input: -Problem:question -ProposedPlan:plan Evaluatetheplanbasedonthefollowingcriteria: <think> EvaluationCriteria: -Clarity:Istheplanclear,structured,andeasytofollow? -Completeness:Doesitcoverallnecessarystepstosolvetheproblem? -Correctness:Isthereasoningandapproachlogicallysound? -Feasibility:Canfollowingthisplanleadtoacorrectsolution? -Efficiency:Istheapproachreasonablyefficient,avoidingunnecessarysteps? ScoringGuidelines: -8-10:Excellentplan-clear,complete,logicallysound,andfeasible.Followingitshouldleadtoacorrectsolution. -4-7:Acceptableplan-hassomegapsorminorissuesbutthegeneraldirectioniscorrect. -1-3:Poorplan-unclear,incomplete,logicallyflawed,orunlikelytoleadtoacorrectsolution. [Writeyourdetailedanalysishere,addressingeachcriterion.] </think> Provideyourfinalscore. Important: Outputonlyonetaglike<score>7</score>(replace7withyourintegerscore1-10). Figure 9: The prompt of the Critic Agent(plan). Critic Agent Prompt(Answer) Role:SolutionQualityCriticAgent Description: Youareanexpertevaluator.Yourtaskistoassessthequalityofageneratedsolutiontoagivenquestionorproblem. Input: -Question:question -GeneratedSolution:answer Evaluatethesolutionbasedonthefollowingcriteria: <think> EvaluationCriteria: -Accuracy:Isthesolutionfactuallycorrectwithnoerrorsinreasoning,arithmetic,units,orassumptions? -Completeness:Doesitfullyaddressthequestionwithallnecessarystepsandderivations? -Coherence:Isthereasoninglogicalandfreefromcontradictionsorhallucinations? -Conciseness:Istheanswerdirectwithoutmeaninglessrepetition,rambling,orfiller? -InstructionFollowing:Doesthesolutionfollowanyexplicitformattingorstructuralrequirements? ScoringGuidelines: -8-10:Excellentsolution-entirelycorrect,complete,logicallysound,concise,andfollowsallinstructions. -4-7:Acceptablesolution-generallyon-topicandpartiallycorrect,buthasomissionsorclarityissues(nofactualerrors). -1-3:Poorsolution-containsanyfactual/logic/calculationerror,hallucinatedcontent,excessiverepetition,orsevereirrelevance. CriticalRules: -Anyfactualerror(arithmetic,reasoning,commonsense,units,invalidassumptions)āscoremustbe[1-3] -Hallucinatedreferences,fabricateddata,orunsupportedclaimsāscoremustbe[1-3] -Meaninglessrepetitionorexcessiveramblingāscoremustbe[1-3] [Writeyourdetailedanalysishere,addressingeachcriterion.Ifanycriticalissueexists,notethatthescoremustbe[1-3].] < > Provideyourfinalscore. Important: Outputonlyonetaglike<score>7</score>(replace7withyourintegerscore1-10). Figure 10: The prompt of the Critic Agent(answer).