Paper deep dive

MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov, Zifeng Ding, Volker Tresp, Yunpu Ma

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 65

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 7/21/2026, 6:09:38 AM

Summary

The paper introduces MADA-RL, a parameter-efficient post-training framework for compact large language models (≤4B parameters) that specializes agents into generator and critic roles. It employs a counterfactual critic advantage signal within a multi-agent debate protocol to improve reasoning accuracy on mathematical benchmarks while significantly reducing trainable parameters via LoRA adapters.

Entities (9)

MADA-RL · method · 98%Counterfactual Critic Advantage · concept · 95%DeepSeek-R1-Distill-Qwen-1.5B · model · 92%LoRA · technique · 90%GRPO · algorithm · 88%AIME 2024 · dataset · 85%MATH-500 · dataset · 85%DeepScaleR · model · 80%Still-3 · model · 80%

Relation Signals (6)

MADA-RL → uses → Counterfactual Critic Advantage

confidence 95% · Our central contribution is a counterfactual critic advantage... This explicitly optimizes critics to improve over generator consensus

MADA-RL → specializes → DeepSeek-R1-Distill-Qwen-1.5B

confidence 92% · We introduce MADA-RL... specializes compact models into generator and critic roles... on DeepSeek-R1-Distill-Qwen-1.5B

MADA-RL → employs → LoRA

confidence 90% · fine-tuning only a small subset of parameters via LoRA adapters

MADA-RL → outperforms → DeepSeek-R1-Distill-Qwen-1.5B

confidence 90% · raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from 39.9 % to 41.9 %

Counterfactual Critic Advantage → improves → Critic Performance

confidence 88% · the counterfactual advantage produces the highest critic improvement rate of any model evaluated

MADA-RL → evaluatedon → MATH-500

confidence 85% · Across five mathematical reasoning benchmarks... Math-500

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 \, \%$ to $41.9 \, \%$ ($+2.0$ points, $p < 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.

PDF

Open source PDF →

PDF not stored locally. Use the link above to view on the source site.

Full Text

61,393 characters extracted from source content.

Expand or collapse full text

MADA-RL: Multi-Agent Debate-Aware Reinforcement Learn- ing for Parameter-Efficient Reasoning in Compact Models Martino M. L. Pulicimartino.pulici@de.bosch.com Bosch Center for Artificial Intelligence, Germany LMU Munich, Germany Cuong Xuan Chucuongxuan.chu@de.bosch.com Bosch Center for Artificial Intelligence, Germany Evgeny Kharlamovevgeny.kharlamov@de.bosch.com Bosch Center for Artificial Intelligence, Germany University of Oslo, Norway Zifeng Dingzd320@cam.ac.uk University of Cambridge, United Kingdom Volker Trespvolker.tresp@lmu.de LMU Munich, Germany Munich Center for Machine Learning, Germany Yunpu Macognitive.yunpu@gmail.com LMU Munich, Germany Munich Center for Machine Learning, Germany Abstract Large language models achieve strong reasoning performance, but often at prohibitive training cost—a challenge that is especially acute for compact models (≤4 B parameters) trained under limited budgets. We introduce mada-rl, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic’s advantage as its reward minus the generator ensemble’s per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean- reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, mada-rl raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from 39.9 % to 41.9 % (+2.0 points, p <0.001) using 16 times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy–trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, Still-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of mada-rl’s gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them. 1 Introduction Large language models (LLMs) have achieved impressive performance across diverse language tasks, including translation, dialogue, reading comprehension, question answering, and open-ended text generation (OpenAI, 1 arXiv:2607.18006v1 [cs.LG] 20 Jul 2026 2024a; Gemini Team, 2025; Qwen Team, 2025a;b). Yet, multi-step logical reasoning remains a core challenge, especially for resource-constrained models with fewer than 4 B parameters (Xu et al., 2025). While increasing model scale often yields better reasoning (Brown et al., 2020), it is prohibitively expensive in both training and inference (Hoffmann et al., 2022; Arora & Zanette, 2025). To bridge this gap, two complementary strategies have recently emerged. The first leverages reinforcement learning (RL) to fine-tune models for reasoning. Notable systems include OpenAI’s o-series (OpenAI, 2024b; 2025) and DeepSeek-R1 (DeepSeek-AI, 2025), which use RL policy optimization (Shao et al., 2024; Mei et al., 2025) and preference learning (Christiano et al., 2017; Ouyang et al., 2022) to align model behaviours. Despite impressive gains, RL approaches often incur high computational cost (Hu et al., 2023; Wang et al., 2025b), risk training instability and model collapse (Cui et al., 2025; Yuan et al., 2025), and depend heavily on reward-design quality. The second strategy comprises methods that increase computational cost per query, a phenomenon commonly referred to as test-time scaling. Among them, chain-of-thought prompting (Wei et al., 2023), self-consistency (Wang et al., 2023), and tree-of-thought search (Yao et al., 2023) provide explicit multi-step reasoning, sampling diversity, and lookahead search, respectively. Reflection and self-refinement methods further allow a single model to critique and improve its own outputs (Madaan et al., 2023). Multi-agent debate frameworks take this further: by pitting agents against one another, they promote answer robustness and reduce superficial or shortcut reasoning (Irving et al., 2018; Du et al., 2024; Subramaniam et al., 2025). Prior attempts to integrate RL with test-time prompting or debate (Park et al., 2025; Subramaniam et al., 2025) often involve significant computational overhead or architectural complexity. We instead study how to combine these strands in a lightweight pipeline, addressing three limitations of prior RL-based reasoning methods: they often require full-model fine-tuning, suffer from unstable credit assignment, and under-utilize test-time deliberation. To this end, we propose multi-agent debate-aware reinforcement learning (mada-rl), a lightweight post-training framework that combines structured multi-agent RL with debate, while fine-tuning only a small fraction of parameters via low-rank adaptation (LoRA; Hu et al., 2022) and group relative policy optimization (grpo; Shao et al., 2024). We also introduce counterfactual advantage to stabilize learning and sharpen credit assignment among debating agents. We demonstrate our approach on DeepSeek-R1-Distill-Qwen-1.5B, a 1.5 B-parameter, distillation-based model widely used in small-scale reasoning research (Chen et al., 2025; Dang & Ngo, 2025; Luo et al., 2025; Wang et al., 2025b). Beyond our own fine-tuned agents, we evaluate a collection of community-released variants, assessing performance under single-agent and multi-agent debate conditions. We focus on standard mathematical reasoning benchmarks and report rigorous statistical metrics, enabling more robust comparisons. We also provide a detailed analysis of the training and inference cost trade-offs involved, and we isolate the source of mada-rl’s gains to improved critic behaviour rather than to additional test-time deliberation alone. To summarize, our contributions include: •a counterfactual critic advantage for role-specialized RL, constituting a dynamic, role-conditioned baseline that rewards critics for improving over the generator ensemble’s per-instance accuracy, sharpening credit assignment without value models, replay buffers, or external verifiers • mada-rl, a parameter-efficient post-training method that applies this signal to compact LLMs, together with a suite of LoRA-fine-tuned agents built on DeepSeek-R1-Distill-Qwen-1.5B •a controlled analysis that traces mada-rl’s gains to learned corrective behaviour rather than to additional test-time deliberation alone • a comprehensive empirical study of existing 1.5 B fine-tunings under a common role-specialized pro- tocol, together with an accounting of the training-parameter and inference-tokentrade-offs involved. The remainder of the paper is organized as follows: section 2 introduces the test-time debate algorithm, section 3 presents the mada-rl training framework and advantage design, section 4 describes the experimental setup and results, section 5 reviews related work, and section 6 concludes. 2 2 Test-time debate To contextualize our training objective, we must first define the test-time debate algorithm that our agents will operate within. Our RL framework relies on this procedure to generate the counterfactual advantage used to train critic agents. Following Du et al. (2024) and Subramaniam et al. (2025), we define the framework as: • multi-agent, as multiple agents provide their answers in parallel • multi-round, as agents debate in sequence using previous round answers •multi-role, as agents that generate answers (generators) need different expertise from those who assess previous responses (critics). We use the term debate in the deliberative sense of Du et al. (2024)—multiple model instances exposing their answers to one another and revising over rounds— rather than the adversarial, judge-arbitrated sense of Irving et al. (2018) and Liang et al. (2024). Concretely, our protocol is parallel generation followed by sequential critic revision conditioned on the other agents’ outputs; we make no claim of explicit argumentation or persuasion between agents. This deliberately simple interaction is what makes the protocol cheap to deploy on compact models, and our contribution lies not in the protocol itself but in the training signal that shapes how critics behave within it (section 3.3). Figure 1 and algorithm 1 detail the procedure: during the first debate round, each generator agentG i generates an answer to a questionx; then, for all successive debate rounds, each critic agentC i produces an updated guess based on the original question and concatenated answers from the previous round; at the end, accuracy is computed using only answers from the final debate round. Input x® Generator G i Generator G 1 Generator G N . . . . . . ˆy 1,i Ú ˆy 1,1 Ú ˆy 1,N Ú . . . . . . Input x 2 Critic C i u Critic C 1 u Critic C M u . . . . . . ˆy 2,i Ú ˆy 2,1 Ú ˆy 2,M Ú . . . . . . Input x 3 Critic C i u Critic C 1 u Critic C M u . . . . . . ˆy R,i Ú ˆy R,1 Ú ˆy R,M Ú · · · . . . . . . Figure 1: Multi-agent debate, adapted from Subramaniam et al. (2025). While we adopt the role specialization of agents into generators and critics from Subramaniam et al. (2025), we intentionally omit another enhancement proposed in the same work: replacing response concatenation between debate rounds with summarization. This choice reflects our aim to keep the method simple and memory efficient. Although summarization has been shown to yield modest performance improvements, it would require an additional model and introduce non-trivial computational and memory overhead. Given that our setting involves only a few agents, whose responses comfortably fit within the model’s context window, we found this trade-off unnecessary. However, in scenarios with a larger number of agents, incorporating a summarizer could represent a valuable extension. 3 Multi-agent debate-aware reinforcement learning In this section, we introduce mada-rl, our lightweight RL framework that employs multiple agents with distinct roles: generators and critics. The training process follows a two-stage design, where each stage uses the same underlying RL algorithm but employs different datasets and advantage computations, tailored to the specific objectives of the two agent types. We begin by outlining the overall framework, then describe the training procedures for generator and critic agents in detail. 3.1 The MADA-RL framework Our framework builds on grpo (Shao et al., 2024), a value-model-free policy optimization method that has been effectively employed to train the DeepSeek-R1 model family (DeepSeek-AI, 2025). Inspired by 3 Algorithm 1 Multi-agent debate Input: ModelA,Ngenerator LoRA adaptersL G,i ,Mcritic LoRA adaptersL C,i , datasetD=(x i ,y i ) Parameter: Number of debate rounds R Output: Final round accuracy 1: success← 0▷ Initialize success counter 2: G i N i=1 ←L G,i (A) N i=1 ▷ Instantiate generator agents 3: C i M i=1 ←L C,i (A) M i=1 ▷ Instantiate critic agents 4: for all (x,y)∈D do 5:y 1,i N i=1 ←G i (x) N i=1 ▷ Generate generators’ responses 6:for r ∈2,...,R do 7:x r ← [x;y r−1,i ]▷ Concatenate question and previous round responses 8:y r,i M i=1 ←C i (x r ) M i=1 ▷ Generate critics’ responses 9:success← success + P M i=1 1(y R,i ≡ y)▷Update success counter 10: accuracy← success M|D| ▷Compute accuracy 11: return accuracy Subramaniam et al. (2025), we train each agent on a disjoint subset of the data to foster role specialization, an approach shown to enhance multi-agent debate outcomes. To maintain efficiency and scalability, mada-rl uses LoRA adapters (Hu et al., 2022), allowing fine-tuning of only a small fraction of the model parameters while keeping the base weights frozen. This significantly reduces computational and memory overhead, making our method accessible even in resource-constrained environments. Figure 2 presents the complete mada-rl training pipeline, and algorithm 2 goes into further details: the first block (lines 1–5) trains the generator agents; the second block (lines 6–12) constructs the debate-aware datasetD ′ used to train the critic agents; the final block (lines 13–7) mirrors the generator training loop, but applies it to the critic agents. DatasetDõ SubsetD i i SubsetD 1 i SubsetD N i . . . . . . Generator G i Generator G 1 Generator G N . . . . . . Grpo Grpo Grpo DatasetD ′ õ SubsetD ′ i i SubsetD ′ 1 i SubsetD ′ M i . . . . . . Critic C i u Critic C 1 u Critic C M u . . . . . . Grpo Grpo Grpo Figure 2: The mada-rl training pipeline. 3.2 Training generator agents Generator agents are trained independently as single-agent models. This setup is appropriate because they are only responsible for producing the initial responses during inference and do not participate in or receive feedback from subsequent debate rounds. Treating them as isolated agents simplifies the training process and reflects their role in the system architecture. A similar strategy is adopted by Subramaniam et al. (2025), who report that independently trained generators can effectively support multi-agent reasoning when paired with collaborative or evaluative agents. To incentivize high-quality outputs, each generator is trained using a composite reward that balances two core aspects of answer quality: correctness and brevity. Specifically, the reward function is a weighted sum of an accuracy reward and a length reward, with a 2:1 weighting in favor of accuracy. This prioritization reflects the primary goal of producing correct answers, while also encouraging conciseness to avoid verbosity. 4 Algorithm 2 Mada-rl training Input: Model A, LoRA adapter L, dataset D =(x i ,y i ) Parameters: Number of generators N, number of critics M Output: Fine-tuned LoRA adapters L G,i N i=1 and L C,i M i=1 1: D i N i=1 ←D▷ Partition dataset into disjoint subsets 2: for i∈1,...,N do 3:G i ← L(A)▷Instantiate generator from base model and blank LoRA adapter 4:train G i on D i ▷ Train generator with grpo advantage 5:save L G,i ▷Save the best LoRA adapter 6: G i N i=1 ←L G,i (A) N i=1 ▷ Instantiate generator agents 7: D ′ ←∅▷ Initialize empty critic dataset 8:for all (x,y)∈D do 9:ˆy i N i=1 ←G i (x) N i=1 ▷ Generate generators’ responses 10:x ′ ← h x;ˆy i N i=1 i ▷ Concatenate question and generators’ responses 11:acc G ← mean 1(ˆy i ≡ y) N i=1 ▷Compute mean generator accuracy 12:D ′ ←D ′ ∪(x ′ ,y, acc G )▷ Update critic dataset 13: D ′ i M i=1 ←D ′ ▷ Partition critic dataset into disjoint subsets 14:for i∈1,...,M do 15:C i ← L(A)▷Instantiate critic from base model and blank LoRA adapter 16:train C i on D ′ i ▷ Train critic with counterfactual advantage 17:save L C,i ▷Save the best LoRA adapter 18: return L G,i N i=1 and L C,i M i=1 ▷ Return generators’ and critics’ LoRA adapters 5 This design is informed by prior RL studies such as those by Chen et al. (2025) and Wang et al. (2025b), which demonstrate the benefits of multi-objective reward shaping in language model fine-tuning. Accuracy reward. Given that our datasets consist of mathematically grounded problems, we define the reward signal as the binary function R acc (ˆy,y) = 1(ˆy ≡ y) based on symbolic equivalence between the model’s predictionˆyand the ground truth answery. We use the equivalence symbol≡instead of the equality symbol = to reflect the method of evaluation: answers are parsed and normalized using a L A T E X-to-SymPy converter (Kydlíček, 2025a) and verified for symbolic equality using a mathematical reasoning engine (Kydlíček, 2025b). This approach enables reliable reward computation even in the presence of minor formatting differences or expression reordering. Length reward. As language models are prone to verbosity, especially during RL, we introduce a length- based reward to promote concise outputs. Following the method proposed by Kimi Team (2025), we penalize unnecessarily long completions based on token count, defining a length reward as R len (ˆy,y) = ( λˆy ≡ y min(0,λ) ˆy ̸≡ y λ = 0.5− len(ˆy)− min i len(ˆy i ) max i len(ˆy i )− min i len(ˆy i ) wherelen(ˆy) is the number of tokens in the generated answer andˆy i represents the set of sampled outputs for the same prompt within a single grpo sampling group. The length reward ranges from−0.5 (for the longest correct output) to +0.5 (for the shortest correct output in the group), with incorrect outputs capped at zero to avoid reward hacking through trivially short but incorrect completions. Final generator reward. The overall generator reward combines both objectives into a single scalar signal used for policy gradient updates, resulting in the final reward function R G (ˆy,y) = 2R acc (ˆy,y) + R len (ˆy,y) ensuring that accurate answers are strongly rewarded, while still giving a slight preference to brevity among correct outputs. It also provides a safeguard against degenerate solutions, such as repetitive or vacuous responses. 3.3 Training critic agents The critic training process reuses much of the generator training pipeline but incorporates two key modifications to account for the critic’s distinct role. First, the input dataset is enriched to include not only the original question but also the set of generator responses from the initial debate round. Second, the policy update is guided by a counterfactual advantage signal, which sharpens credit assignment by measuring the critic’s performance relative to its generator peers. Counterfactual advantage. The critic’s objective is not merely to produce a correct answer, but to do so especially in situations where the generators fail. To formalize this, we define the advantage signal for the critic by first calculating the critic’s total reward as R C (ˆy,y) = 2R acc (ˆy,y) + R len (ˆy,y) using the same composite function as the generators. We then compute the advantage by subtracting a dynamic, role-conditioned baseline from this total reward. This baseline is derived from the per-instance average generator accuracyacc G , computed at the time of the critic’s dataset creation. To ensure the baseline is scaled consistently with the primary component of the critic’s reward, we multiply this average accuracy by two. The derivation proceeds as A C (ˆy,y, acc G ) = R C (ˆy,y)− 2 acc G = 2 (R acc (ˆy,y)− acc G ) + R len (ˆy,y) 6 providing a more informative learning signal. The term in parentheses (R acc (ˆy,y)− acc G ) directly compares the critic’s correctness to the generator average, and scaling it by two ensures this comparison is weighted appropriately. A positive advantage strongly rewards the critic for outperforming the generator consensus on correctness, while a negative advantage penalizes it for underperforming. 4 Experiments 4.1 Experimental setup For our main experiment, we trained a total of six agents—three generators and three critics—on random splits from the Still-3-Preview-RL-Data dataset (Ruc-aibox, 2025a), which contains 29 925 mathematical reasoning problems. To adapt the dataset to our multi-agent setting, we created three disjoint training subsets of 7500 problems each, along with a shared validation set of 750 problems for checkpoint selection. All agents were fine-tuned starting from the DeepSeek-R1-Distill-Qwen-1.5B model (DeepSeek, 2025b). Training was performed using the grpo algorithm (Shao et al., 2024), in combination with the AdamW optimizer (Loshchilov & Hutter, 2019) and LoRA adapters (Hu et al., 2022). This setup allowed for efficient fine-tuning with a minimal memory footprint. Training and inference hyperparameters are reported in the appendix, together with details about the computing infrastructure. Simple accuracy is used for all experiments and all results are computed averaging the results of all agent instances. In situations where only a subset of agents are used (as in single-agent, two-agent, and homogeneous settings of section 4.4), multiple runs with all agent combinations are averaged. To assess whether differences between models are meaningful, we compare per-seed average accuracies (across the five benchmarks, ten seeds per model) using Welch’s two-samplet-test; results are reported in table 2 and, in full pairwise form, in section A.4. Answer correctness was evaluated by parsing model generations and ground truth answers into symbolic form as described in section 3.2. 4.1.1 Baselines We compare our agents against seven baselines built on the same base model, DeepSeek-R1-Distill-Qwen-1.5B, ensuring controlled and fair comparisons across different fine-tuning strategies. These baselines are: •DeepSeek-R1 (DeepSeek, 2025b), a reasoning-oriented fine-tuning of Qwen2-1.5B (Qwen, 2024), distilled from the flagship DeepSeek-R1 model (DeepSeek, 2025a) • Still-3(Ruc-aibox, 2025b), inspired by slow-thinking paradigms such as o1 (OpenAI, 2024b), focusing on deliberate multi-step reasoning •DeepScaleR (Agentica, 2025), which applies grpo with an accuracy-based reward, emphasizing longer-context reasoning •Open-RS1, Open-RS2, and Open-RS3 (Knovel Engineering, 2025c;d;e); lightweight fine-tunings that prioritize low-resource adaptability while preserving reasoning performance • Tina (Tina, 2025), a LoRA-based variant from the Tina family. For a fairer comparison, we also re-trained the best-performing baselines (DeepScaleR and Still-3) using the original datasets and methods but with LoRA, yielding DeepScaleR-LoRA and Still-3-LoRA. 4.1.2 Benchmarks We evaluate all models on five established benchmarks covering competition-style and academic mathemat- ical reasoning: • Math-500(Hugging Face H4, 2024), a representative subset of the Math benchmark (Lightman et al., 2024), featuring challenging competition-level problems across diverse topics 7 Table 1: Results of the main experiment, with accuracy values shown in base 100 ModelMath-500 Aime 2024 Aime 2025 AMC-23 Minerva-Math Average DeepSeek-R169.9 ± 0.223.0± 1.0 19.6± 0.8 65.3± 0.921.6± 0.339.9± 0.3 Still-372.5 ± 0.228.4± 1.3 20.9± 0.670.7± 1.023.2± 0.443.1± 0.4 DeepScaleR72.81± 0.1928.8± 0.822.4± 0.871.4± 0.926.0± 0.344.3± 0.3 Open-RS169.8 ± 0.325.2± 1.1 19.9± 0.7 66.0± 1.321.2± 0.440.4± 0.4 Open-RS269.7 ± 0.223.3± 0.9 17.2± 0.8 64.8± 1.121.9± 0.339.4± 0.3 Open-RS369.5 ± 0.324.8± 0.9 21.2± 0.9 64.4± 0.722.3± 0.340.4± 0.3 Tina69.57± 0.19 25.2± 1.2 19.0± 0.8 63.4± 0.821.9± 0.339.8± 0.3 Still-3-LoRA72.1 ± 0.324.3± 1.3 21.0± 0.7 67.3± 1.121.9± 0.441.3± 0.4 DeepScaleR-LoRA 70.7 ± 0.423.6± 1.0 20.3± 0.9 66.7± 1.221.4± 0.240.5± 0.4 Mada72.0 ± 0.226.2± 0.7 20.0± 0.968.7± 1.222.5± 0.341.9± 0.3 Note: standard errors of the mean are reported throughout; cell colours indicate performance relative to the baseline DeepSeek- R1 model (red for worse, green for better), colour intensity scales with the magnitude of the difference, and white marks non-significant differences. • Aime 2024andAime 2025(Hugging Face H4, 2025; Lin, 2025), consisting of full American Invitational Mathematics Examination (Aime) problem sets, known for their structured, multi-step algebraic reasoning •AMC-23 (Knovel Engineering, 2025a), including problems from the American Mathematics Com- petition, generally shorter than Aime but still requiring careful reasoning •Minerva-Math (Knovel Engineering, 2025b) contains university-level problems spanning mathem- atics and related stem fields, often involving symbolic manipulation and domain-specific modelling. 4.2 Main results We applied the inference procedure of section 2 across all benchmarks. Our models (Mada) use the pipeline directly; for the single-agent baselines we instantiate identical copies in every generator and critic role, keeping the same agent count and rounds for a controlled comparison. Mirroring the training setting, we use three generators, three critics, and two debate rounds. Table 1 reports average accuracy and standard error for all models, with the full pairwise Welch’st-tests in section A.4. The two models fine-tuned on the largest datasets (DeepScaleR and Still-3) clearly perform best, with DeepScaleR significantly ahead of Still-3 (p= 0.025); the other community fine-tunes (Open-RS1, Open-RS2, Open-RS3, Tina) are statistically indistinguishable from the base DeepSeek-R1 (all p > 0.20). Aside from the two data-heavy baselines (DeepScaleR and Still-3), the Mada agents are the only ones to improve clearly and consistently over the base model with non-overlapping error bars, and the only ones to outperform it on every benchmark. We state the gap plainly: Mada does not match DeepScaleR or Still-3. Both are trained on substantially larger corpora with full-model fine-tuning—DeepScaleR and Still-3 update all 1.78×10 9 parameters, against 110×10 6 trainable parameters in our setup—so the comparison most relevant to our claim is not raw accuracy but accuracy gain per unit of training cost, which we analyse in section 4.3. Notably, when DeepScaleR and Still-3 are re-trained with LoRA on the same method and data (DeepScaleR-LoRA and Still-3-LoRA), their accuracy drops to 40.5 and 41.3 respectively: DeepScaleR-LoRA falls significantly below Mada (p= 0.015), while Still-3-LoRA is statistically indistinguishable from it (p= 0.280). The full fine-tuning advantage of these stronger baselines therefore rests largely on data scale rather than on a mechanism that LoRA preserves, and at an equal trainable-parameter budget Mada matches or exceeds plain LoRA re-training. Table 2 reports the statistical significance of Mada against every other model. The +2.0 point gain over the base model is significant (p <0.001), confirming that the improvement is not an artifact of decoding variance. 8 Table 2: Mada accuracy against each baseline BaselineAccuracy DifferenceCI 95 % p DeepSeek-R139.9± 0.3+2.0± 0.5 [+1.0, +3.0] <0.001 ∗ Still-343.1± 0.4−1.3± 0.5 [−2.3,−0.2] 0.020 ∗ DeepScaleR44.3± 0.3−2.4± 0.5 [−3.4,−1.5] <0.001 ∗ Open-RS140.4± 0.4+1.5± 0.5 [+0.4, +2.5] 0.011 ∗ Open-RS2 39.4± 0.3+2.5± 0.5 [+1.5, +3.5] <0.001 ∗ Open-RS340.4± 0.3+1.4± 0.5 [+0.5, +2.4] 0.006 ∗ Tina39.8± 0.3+2.1± 0.5 [+1.0, +3.1] <0.001 ∗ Still-3-LoRA41.3± 0.4 +0.6± 0.5 [−0.5, +1.7] 0.280 DeepScaleR-LoRA 40.5± 0.4+1.4± 0.5 [+0.3, +2.4] 0.015 ∗ Mada41.9± 0.3— Note: standard errors of the mean are reported throughout;‘accuracy’ is the mean accuracy across all benchmarks; ‘difference’ is the accuracy difference (Mada minus baseline) in points, shown with its 95 % confidence interval; green favours Mada, red favours the baseline, colour intensity scales with the magnitude of the difference, and white marks non-significant differences;p-values are computed using Welch’s two-sample t-test on per-seed average accuracy across the five benchmarks (ten seeds per model); ∗ p < 0.05, ∗ p < 0.01, ∗ p < 0.001. Mada also significantly outperforms every parameter-efficient community fine-tune (Open-RS1, Open-RS2, Open-RS3, and Tina), and no matched-budget method significantly exceeds it. Consistent with our framing, it remains significantly below the two data-heavy, fully fine-tuned baselines (DeepScaleR and Still-3). The full pairwise matrix of p-values is given in section A.4. 4.3 Computational costs and performance To evaluate our method’s efficiency, we assess the computational costs of mada-rl, considering both training-time and inference-time expenses. Training costs. To estimate training costs, we consider the number of trainable parameters per model. The base model, DeepSeek-R1-Distill-Qwen-1.5B, contains approximately 1.78×10 9 parameters in total, and fully fine-tuned baselines update all of these. In contrast, LoRA adapters add only 36.9×10 6 trainable parameters in the Tina model, 1.48×10 8 in DeepScaleR-LoRA and Still-3-LoRA, and 18.5×10 6 per agent in our setup, totaling 110×10 6 . Figure 3 and table 3 report trainable parameters and performance gains, showing that our method achieves the largest improvement relative to trainable parameters. While DeepScaleR achieves roughly twice the accuracy gain, it requires 16 times more trainable parameters, underscoring the advantage of mada-rl in enhancing reasoning performance under constrained training budgets. In addition, DeepScaleR-LoRA and Still-3-LoRA have more trainable parameters than our setup yet perform worse, highlighting the training-parameter efficiency of our approach. Inference costs. The training-parameter savings come at a real inference-time price, which we report in table 3. The protocol issues six agent calls per question (three generators, then three critics) across two rounds, and Mada generates 33 818 tokens per question—among the highest of any model and roughly an order of magnitude more than a single forward pass. For latency-sensitive deployment this is the method’s main practical limitation, and any efficiency claim concerns training parameters, not inference compute. Yet the token count alone misses a structural benefit: Mada achieves the highest critic improvement rate (19.6 %), correcting wrong generator answers more often than any other model—a direct consequence of the counterfactual advantage, which rewards critics for being correct when the generators are not. Baselines that place standard single-agent models in the critic role show lower improvement rates even when they generate comparable numbers of tokens, indicating that Mada’s gains stem from the critic’s learned corrective behaviour rather than from deliberation volume alone. 9 00.511.5 ·10 9 39 40 41 42 43 44 45 DeepSeek-R1 Mada DeepScaleR-LoRA Still-3-LoRA Open-RS2 Open-RS1 Open-RS3 Still-3 DeepScaleR Tina Trainable parameters Accuracy Figure 3: Trainable parameters and accuracy. Darker points connected with a solid line represent the Pareto front. The star highlights our method, which achieves the highest gain per trainable parameter. Table 3: Computational costs and performance ModelTrainable parameters Gain/parameter Tokens/question Critic improv. rate DeepSeek-R1—30 22118.5 Still-31.78× 10 9 +1826 62917.5 DeepScaleR 1.78× 10 9 +2525 52216.9 Open-RS11.78× 10 9 +329 91918.2 Open-RS21.78× 10 9 −330 57219.3 Open-RS31.78× 10 9 +330 22818.1 Tina 3.69× 10 7 −1727 46318.6 DeepScaleR-LoRA1.48× 10 8 +4230 76318.3 Still-3-LoRA1.48× 10 8 +9633 86518.8 Mada1.10× 10 8 +18233 81819.6 Note: ‘gain/parameter’ is the accuracy improvement over the base model divided by the number of trainable parameters (scaled by 10 12 ), a proxy for training-cost efficiency; ‘tokens/question’ is the mean number of tokens generated per question under the full three-generator, two-round protocol, a proxy for inference cost; ‘critic improv. rate’ is the percentage of questions on which the critics’ final answer corrects the generators’ consensus; relative to the base DeepSeek-R1 model, green cells indicate better performance and red cells indicate worse performance, with intensity scaling with the magnitude of the difference; grey encodes magnitude only, where better or worse does not apply. 4.4 Ablation studies To assess the effectiveness of our design choices, we conducted several ablation studies, summarized in table 4. Number of rounds. We compared the accuracy of the main experiment with both one-round and three- round debate variants. A single round is effectively equivalent to generating multiple answers independently and evaluating them without any interaction. As expected, this results in significantly worse performance compared to the standard two-round setup. Adding a third round yields a further improvement, although the gain is smaller. This suggests that iterative interaction between agents is an important driver of performance, while additional rounds provide diminishing returns. Number of agents. We compared the main configuration with smaller debate ensembles consisting of one and two generator–critic pairs. The three-agent configuration outperforms smaller ensembles, suggesting that 10 Table 4: Ablation studies SettingAccuracy DifferenceCI 95 % p 2 rounds 3 generators + 3 critics 41.9± 0.3— Counterfactual 1 round36.7± 0.3−5.2± 0.5 [−6.2,−4.2] <0.001 ∗ 3 rounds 43.3± 0.4+1.4± 0.5 [+0.3, +2.5] 0.016 ∗ 1 generator + 1 critic39.3± 0.3−2.6± 0.5 [−3.6,−1.6] <0.001 ∗ 2 generators + 2 critics41.0± 0.2−0.9± 0.4 [−1.8, 0.0] 0.050 ∗ Homogeneous41.3± 0.2 −0.6± 0.4 [−1.5, +0.3] 0.166 No counterfactual41.1± 0.3 −0.8± 0.5 [−1.8, +0.2] 0.095 Note: standard errors of the mean are reported throughout; ‘difference’ is the mean accuracy difference (main experiment minus ablation setting) in points, shown with its 95 % confidence interval; green favours the ablation setting, red favours the main experiment, colour intensity scales with the magnitude of the difference, and white marks non-significant differences;p-values are computed using Welch’s two-samplet-test on per-seed average accuracy across the five benchmarks (ten seeds per model); ∗ p < 0.05, ∗ p < 0.01, ∗ p < 0.001. further scaling could yield additional gains; however, adding agents would require retraining the full pipeline from scratch due to the disjoint data partitioning scheme, making this a non-trivial extension that we leave for future work. Agent diversity. To assess the role of diversity, we evaluated a homogeneous configuration in which all generators share a single generator checkpoint and all critics share a single critic checkpoint, as opposed to the three independently trained checkpoints used in the main experiment. This configuration is 0.6 points below the main experiment, but the difference does not reach significance (p= 0.166;CI 95 % = [−1.5,+0.3], which includes zero). We therefore read agent diversity as a plausibly helpful but unconfirmed factor at this sample size: the point estimate favours diverse checkpoints, yet ten seeds do not provide the power to distinguish it from no effect. Advantage computation. We trained an additional set of critics using the standard grpo advantage, removing the counterfactual component introduced in section 3.3. This variant is 0.8 points below the full mada-rl configuration; the drop is consistent in direction but does not reach significance at ten seeds (p= 0.095;CI 95 % = [−1.8,+0.2]). We therefore do not rest the case for the counterfactual advantage on this ablation alone. Its more direct signature is the critic improvement rate analysed in section 4.3: the counterfactual-trained critics correct wrong generator answers more often (19.6 %) than any baseline placed in the critic role, a mechanism-level effect that does not depend on this accuracy difference clearing a significance threshold. Read together, the directional accuracy drop and the corrective-behaviour evidence support the counterfactual component, while its marginal contribution to end-task accuracy remains within the noise at this sample size. Debate contribution. Table 5 shows the models’ performance in a single-agent setting, without any debate mechanism. As expected, all models experience a drop in performance when debate is removed. However, the drop is largest for Mada (−5.2 points): this indicates that Mada relies more strongly on the debate process than the baseline models, suggesting that the training procedure encourages specialization of generator and critic roles rather than simply improving single-agent reasoning. Ablation takeaways. The debate structure is the dominant and statistically robust driver: removing rounds or shrinking the ensemble produces large, significant drops. Agent diversity and the counterfactual advantage contribute smaller, directionally consistent effects that do not individually reach significance at ten seeds; for the counterfactual component the accuracy ablation is corroborated by its distinct mechanism-level 11 Table 5: Single-agent performance ModelSingle agent DifferenceCI 95 % p DeepSeek-R136.1± 0.3−3.8± 0.5 [−4.7,−2.8] <0.001 ∗ Still-339.5± 0.4−3.7± 0.5 [−4.7,−2.6] <0.001 ∗ DeepScaleR39.3± 0.3−5.0± 0.4 [−5.9,−2.4] <0.001 ∗ Open-RS135.9± 0.4−4.6± 0.5 [−5.7,−3.4] <0.001 ∗ Open-RS235.3± 0.3 −4.1± 0.5 [−5.1,−3.1] <0.001 ∗ Open-RS336.2± 0.3−4.2± 0.4 [−5.2,−3.3] <0.001 ∗ Tina35.2± 0.3−4.7± 0.5 [−5.7,−3.7] <0.001 ∗ DeepScaleR-LoRA 36.9± 0.4−3.6± 0.5 [−4.7,−2.5] <0.001 ∗ Still-3-LoRA36.3± 0.4 −5.0± 0.5 [−6.2,−3.9] <0.001 ∗ Mada36.7± 0.3−5.2± 0.5 [−6.2,−4.2] <0.001 ∗ Note: standard errors of the mean are reported throughout; for Mada, the score is the average performance of the fine-tuned generator agents; relative to the base DeepSeek-R1 model, green cells indicate better performance, intensity scales with the magnitude of the difference, and white marks non-significant differences; ‘difference’ is the mean accuracy difference (full debate minus single-agent setting) in points; grey encodes magnitude only, where better or worse does not apply;p-values are computed using Welch’s two-samplet-test on per-seed average accuracy across the five benchmarks (ten seeds per model); ∗ p <0.05, ∗ p < 0.01, ∗ p < 0.001. signature, the highest critic improvement rate among all models. We thus frame the debate structure as established and the training-objective refinements as supported but more lightly powered. 5 Related work Reinforcement learning for LLM reasoning. RL has become a key tool for enhancing reasoning in language models, particularly in low-resource settings. Grpo (Shao et al., 2024) removes the value model and uses group-normalized rewards for stability in long-form tasks, while decoupled clip and dynamic sampling policy optimization (dapo) (Yu et al., 2025) adds dynamic sampling and structured reward shaping. We draw on both, but unlike dapo or value-model-based token-level methods (Yue et al., 2025), we derive answer- level correctness and counterfactual signals from multi-agent debate, yielding a lighter, fully interpretable reward pipeline. Multi-agent debate and adjudication. Composing several model instances and letting them deliberate is an established route to better reasoning. Du et al. (2024) have multiple instances propose and revise answers over rounds to converge on a consensus, while Liang et al. (2024) cast the interaction as an adversarial debate arbitrated by a judge, and Irving et al. (2018) motivate debate as a mechanism for scalable oversight. A parallel line aggregates rather than argues—self-consistency (Wang et al., 2023) by majority vote, LLM-as-judge by a separate adjudicator—but these methods are training-free: they improve a fixed model at inference time without adapting the agents. Closer to us, Park et al. (2025) proposed multi-agent post-co-training for collaborative LLMs (MAPoRL), which guides RL with a learned verifier, and Subramaniam et al. (2025) use debate to generate data for supervised fine-tuning. We differ in what we train: rather than learning a verifier or distilling transcripts, we fold one debate-derived statistic—the generator ensemble’s per-instance accuracy—directly into the critic’s advantage, so critics are optimized online to correct consensus errors without any auxiliary model. Compact models and efficient multi-agent RL. Other approaches explore multi-agent RL, such as multi-agent reinforcement fine-tuning (marft; Liao et al., 2025), which jointly trains agents with shared gradients but adds coordination overhead. We instead keep agents independent and assign credit locally via debate, enabling efficient LoRA training (Hu et al., 2022). Our work also connects to compact-model reasoning: tiny reasoning models via LoRA (Tina; Wang et al., 2025b) shows small LLMs are competitive 12 with LoRA and minimal RL, and offline reasoning optimization (oreo; Wang et al., 2025a) uses offline RL with learned critics; unlike these, we train online with interpretable, multi-agent feedback. 6 Conclusion We introduce mada-rl, a parameter-efficient post-training framework that specializes a compact LLM into generator and critic roles and trains the critics with a counterfactual advantage that rewards them for correcting generator consensus. Using only LoRA adapters and grpo, it improves a 1.5 B model by 2.0 points across five mathematical reasoning benchmarks while updating 16 times fewer parameters than fully fine-tuned baselines—the largest accuracy gain per trainable parameter of any model we evaluate. It approaches but does not surpass the strongest data-heavy baselines, whose matched-parameter LoRA re-trainings do not exceed mada-rl, indicating that their advantage rests on data scale rather than on a component LoRA retains. A controlled analysis ties these gains to learned corrective behaviour rather than deliberation volume, positioning mada-rl as a practical recipe for reasoning gains on compact models under tight training budgets. Several limitations remain and sharpen the agenda. First, the most informative comparison is still open: we have not measured mada-rl against training-free debate, voting, or LLM-as-judge baselines at matched inference budget, so we cannot yet claim it beats spending the same test-time compute on a strong single model. Second, we characterize the counterfactual advantage empirically but do not analyse its convergence or effect on gradient variance relative to standard grpo. Third, our Welch tests use per-seed average accuracy (table 2); per-benchmark and paired tests, bootstrap intervals, and multiple-comparison correction remain future work. Fourth, we evaluate only a single 1.5 B model on mathematical benchmarks; scaling toward 4 B and transfer to scientific reasoning or planning are untested. Finally, the multi-round protocol raises inference latency (section 4.3), which distilling the debate into a single model could mitigate. We will release our code, trained LoRA adapters, and evaluation scripts. References Agentica. DeepScaleR-1.5B-Preview, 2025. URLhttps://huggingface.co/agentica-org/DeepScaleR-1 .5B-Preview. Accessed 13 July 2026. Daman Arora and Andrea Zanette. Training language models to reason efficiently. Computing Research Repository, arXiv:2502.04463, 2025. URL https://arxiv.org/abs/2502.04463. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. URLhttps: //papers.nips.c/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving R1-like reasoning models. Computing Research Repository, arXiv:2503.04548, 2025. URL https://arxiv.org/abs/2503.04548. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30: 4302–4310, 2017. URLhttps://papers.nips.c/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4 e49-Abstract.html. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process 13 reinforcement through implicit rewards. Computing Research Repository, arXiv:2502.01456, 2025. URL https://arxiv.org/abs/2502.01456. Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small LLMs: What works and what doesn’t. Computing Research Repository, arXiv:2503.16219, 2025. URLhttps://arxiv.org/abs/2503.1 6219. DeepSeek. DeepSeek-R1, 2025a. URLhttps://huggingface.co/deepseek-ai/DeepSeek-R1. Accessed 13 July 2026. DeepSeek. DeepSeek-R1-Distill-Qwen-1.5B, 2025b. URLhttps://huggingface.co/deepseek-ai/DeepSe ek-R1-Distill-Qwen-1.5B. Accessed 13 July 2026. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Computing Research Repository, arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501.12948. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. Proceedings of Machine Learning Research, 235: 11733–11763, 2024. URL https://proceedings.mlr.press/v235/du24e.html. Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Computing Research Repository, arXiv:2507.06261, 2025. URL https://arxiv.org/abs/2507.06261. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:30016–30030, 2022. URLhttps://papers.nips.c/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e 5-Abstract-Conference.html. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. Proceedings of the 10th International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Jian Hu, Li Tao, June Yang, and Chandler Zhou. Aligning language models with offline learning from human feedback. Computing Research Repository, arXiv:2308.12050, 2023. URLhttps://arxiv.org/abs/2308 .12050. Hugging Face H4. Math-500, 2024. URLhttps://huggingface.co/datasets/HuggingFaceH4/MATH-500. Accessed 13 July 2026. Hugging Face H4. aime_2024, 2025. URLhttps://huggingface.co/datasets/HuggingFaceH4/aime_2024. Accessed 13 July 2026. Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. Computing Research Repository, arXiv:1805.00899, 2018. URL https://arxiv.org/abs/1805.00899. Kimi Team. Kimi k1.5: Scaling reinforcement learning with LLMs. Computing Research Repository, arXiv:2501.12599, 2025. URL https://arxiv.org/abs/2501.12599. Knovel Engineering. AMC-23, 2025a. URLhttps://huggingface.co/datasets/knoveleng/AMC-23. Accessed 13 July 2026. Knovel Engineering. Minerva-Math, 2025b. URLhttps://huggingface.co/datasets/knoveleng/Minerv a-Math. Accessed 13 July 2026. Knovel Engineering. Open-RS1, 2025c. URLhttps://huggingface.co/knoveleng/Open-RS1. Accessed 13 July 2026. 14 Knovel Engineering. Open-RS2, 2025d. URLhttps://huggingface.co/knoveleng/Open-RS2. Accessed 13 July 2026. Knovel Engineering. Open-RS3, 2025e. URLhttps://huggingface.co/knoveleng/Open-RS3. Accessed 13 July 2026. Hynek Kydlíček. latex2sympy2-extended (version 1.10.2), 2025a. URLhttps://pypi.org/project/latex 2sympy2-extended/1.10.2/. Accessed 13 July 2026. Hynek Kydlíček. math-verify (version 0.8.0), 2025b. URLhttps://pypi.org/project/math-verify/0.8. 0/. Accessed 13 July 2026. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 17889–17904, 2024. URL https://aclanthology.org/2024.emnlp-main.992/. Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. Marft: Multi-agent reinforcement fine-tuning. Computing Research Repository, arXiv:2504.16129, 2025. URL https://arxiv.org/abs/2504.16129. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. International Conference on Learning Representations, 2024:39578–39601, 2024. URLhttps://proceedings.iclr.c/paper_files/ paper/2024/hash/aca97732e30bcf1303bc22ac3924fd16-Abstract-Conference.html. Yen-Ting Lin. aime_2025, 2025. URLhttps://huggingface.co/datasets/yentinglin/aime_2025. Accessed 13 July 2026. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Proceedings of the 7th International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing o1-Preview with a 1.5B model by scaling RL, 2025. URLhttps://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-P review-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2 . Accessed 13 July 2026. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023. URL https://proceedings.neurips.c/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6 c0f3-Abstract-Conference.html. Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. ReaL: Efficient rlhf training of large language models with parameter reallocation. Computing Research Repository, arXiv:2406.14088, 2025. URL https://arxiv.org/abs/2406.14088. OpenAI. GPT-4 technical report. Computing Research Repository, arXiv:2303.08774, 2024a. URLhttps: //arxiv.org/abs/2303.08774. OpenAI. Learning to reason with LLMs, 2024b. URLhttps://openai.com/index/learning-to-reaso n-with-llms. Accessed 13 July 2026. OpenAI. Introducing OpenAI o3 and o4-mini, 2025. URLhttps://openai.com/index/introducing-o3-a nd-o4-mini. Accessed 13 July 2026. 15 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. URLhttps://proceedings.neurips.c/paper_files/paper/2022/ha sh/b1efde53be364a73914f58805a001731-Abstract.html. Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. MAPoRL: Multi-agent post-co-training for collaborative large language models with reinforcement learning. Computing Research Repository, arXiv:2502.18439, 2025. URL https://arxiv.org/abs/2502.18439. Qwen. Qwen2-1.5B, 2024. URL https://huggingface.co/Qwen/Qwen2-1.5B. Accessed 13 July 2026. Qwen Team. Qwen2.5 technical report. Computing Research Repository, arXiv:2412.15115, 2025a. URL https://arxiv.org/abs/2412.15115. Qwen Team. Qwen3 technical report. Computing Research Repository, arXiv:2505.09388, 2025b. URL https://arxiv.org/abs/2505.09388. Ruc-aibox. Still-3-Preview-RL-Data, 2025a. URLhttps://huggingface.co/datasets/RUC-AIBOX/STI L-3-Preview-RL-Data. Accessed 13 July 2026. Ruc-aibox. Still-3-1.5B-preview, 2025b. URLhttps://huggingface.co/RUC-AIBOX/STILL-3-1.5B-pre view. Accessed 13 July 2026. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. Computing Research Repository, arXiv:2402.03300, 2024. URLhttps://arxiv.org/ab s/2402.03300. Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. Computing Research Repository, arXiv:2501.05707, 2025. URL https://arxiv.org/abs/2501.05707. Tina. R1-Distill-Qwen-1.5B-Open-RS2, 2025. URLhttps://huggingface.co/Tina-Yi/R1-Distill-Qwe n-1.5B-Open-RS2. Accessed 13 July 2026. Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for LLM multi-step reasoning. Findings of the Association for Computational Linguistics: ACL 2025, p. 8881–8893, 2025a. URL https://aclanthology.org/2025.findings-acl.464/. Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. Tina: Tiny reasoning models via LoRA. Computing Research Repository, arXiv:2504.15777, 2025b. URL https://arxiv.org/abs/2504.15777. Xuezhi Wang, Jason Wei andF Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. Proceedings of the 11th International Conference on Learning Representations, 2023. URLhttps://open review.net/forum?id=1PL1NIMMrw. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Computing Research Repository, arXiv:2201.11903, 2023. URL https://arxiv.org/abs/2201.11903. Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models. Computing Research Repository, arXiv:2501.09686, 2025. URL https://arxiv.org/abs/2501.09686. 16 Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Computing Research Repository, arXiv:2305.10601, 2023. URL https://arxiv.org/abs/2305.10601. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source LLM reinforcement learning system at scale. Computing Research Repository, arXiv:2503.14476, 2025. URLhttps://arxiv.org/abs/2503.1 4476. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind PPO’s collapse in Long-CoT? Value optimization holds the secret. Computing Research Repository, arXiv:2503.01491, 2025. URL https://arxiv.org/abs/2503.01491. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks. Computing Research Repository, arXiv:2504.05118, 2025. URL https://arxiv.org/abs/2504.05118. A Technical appendix A.1 Prompts This section reports the various prompts used for text generation. Math reasoning. This is the text that is prepended to all queries to the models: Solve the following math problem efficiently and clearly. The last line of your ,→ response should be of the following format: ’Therefore, the final answer is: ,→ $\ ANSWER$. I hope it is correct’ (without quotes) where ANSWER is ,→ just the final number or expression that solves the problem. Think step by step ,→ before answering. question Multi-agent debate. When concatenating outputs during the multi-agent debate, this was inserted first: These are the recent opinions from other agents: followed by this text for each agent: One agent’s response: ‘ response ‘ 17 and ending with: Using each response as additional advice, can you give an updated answer to the ,→ question? A.2 Hyperparameters Training hyperparameters are detailed in table 6. These values were chosen based on empirical validation and informed by the tuning strategies proposed by previous works. Table 7 reports the hyperparameters used for text generation at inference. A.3 Computing infrastructure Experiments were run on the Red Hat Enterprise Linux 9.5 operating system with 32GBof RAM. Both training and inference were run on a single H200 GPU with 141 GB of video RAM. A.4 Full pairwise significance Table 8 reports all pairwise mean accuracy differences (with standard errors) as an effect-size matrix, and table 9 the corresponding Welcht-testp-values. With ten models there are 45 pairwise comparisons and we apply no multiple-comparison correction, so we treat these as exploratory; the pre-specified comparisons of Mada against each baseline in table 2 carry the formal claims. 18 Table 6: Training hyperparameters Batch size16 Learning rate1× 10 −6 Warmup ratio0.1 AdamW weight decay0 AdamW β 1 0.9 AdamW β 2 0.999 AdamW ε1× 10 −8 Grpo group size4 Grpo sampling temperature0.7 Grpo β0.04 Grpo ε0.2 LoRA rank16 LoRA α128 LoRA dropout0.05 Table 7: Inference hyperparameters Temperature1.0 Top p1.0 Maximum generated tokens 32 768 Number of generators3 Number of critics3 Number of rounds2 Samples per experiment10 19 Table 8: Pairwise mean accuracy differences between models DS-R1 Still-3 DSR Open-RS1 Open-RS2 Open-RS3 Tina S3-LoRA DSR-LoRA Mada DS-R1 — − 3 . 3 ± 0 . 5 − 4 . 4 ± 0 . 4 − 0 . 5 ± 0 . 5 +0 . 5 ± 0 . 5 − 0 . 6 ± 0 . 4 +0 . 1 ± 0 . 5 − 1 . 4 ± 0 . 5 − 0 . 6 ± 0 . 5 − 2 . 0 ± 0 . 5 Still-3 +3 . 3 ± 0 . 5 — − 1 . 1 ± 0 . 5 +2 . 7 ± 0 . 5 +3 . 8 ± 0 . 5 +2 . 7 ± 0 . 5 +3 . 3 ± 0 . 5 +1 . 8 ± 0 . 5 +2 . 6 ± 0 . 5 +1 . 3 ± 0 . 5 DSR +4 . 4 ± 0 . 4 +1 . 1 ± 0 . 5 — +3 . 9 ± 0 . 5 +4 . 9 ± 0 . 4 +3 . 9 ± 0 . 4 +4 . 5 ± 0 . 4 +3 . 0 ± 0 . 5 +3 . 8 ± 0 . 5 +2 . 4 ± 0 . 5 Open-RS1 +0 . 5 ± 0 . 5 − 2 . 7 ± 0 . 5 − 3 . 9 ± 0 . 5 — +1 . 0 ± 0 . 5 0 . 0 ± 0 . 5 +0 . 6 ± 0 . 5 − 0 . 9 ± 0 . 5 − 0 . 1 ± 0 . 5 − 1 . 5 ± 0 . 5 Open-RS2 − 0 . 5 ± 0 . 5 − 3 . 8 ± 0 . 5 − 4 . 9 ± 0 . 4 − 1 . 0 ± 0 . 5 — − 1 . 1 ± 0 . 5 − 0 . 4 ± 0 . 5 − 1 . 9 ± 0 . 5 − 1 . 1 ± 0 . 5 − 2 . 5 ± 0 . 5 Open-RS3 +0 . 6 ± 0 . 4 − 2 . 7 ± 0 . 5 − 3 . 9 ± 0 . 4 0 . 0 ± 0 . 5 +1 . 1 ± 0 . 5 — +0 . 6 ± 0 . 5 − 0 . 9 ± 0 . 5 − 0 . 1 ± 0 . 5 − 1 . 4 ± 0 . 5 Tina − 0 . 1 ± 0 . 5 − 3 . 3 ± 0 . 5 − 4 . 5 ± 0 . 4 − 0 . 6 ± 0 . 5 +0 . 4 ± 0 . 5 − 0 . 6 ± 0 . 5 — − 1 . 5 ± 0 . 5 − 0 . 7 ± 0 . 5 − 2 . 1 ± 0 . 5 S3-LoRA +1 . 4 ± 0 . 5 − 1 . 8 ± 0 . 5 − 3 . 0 ± 0 . 5 +0 . 9 ± 0 . 5 +1 . 9 ± 0 . 5 +0 . 9 ± 0 . 5 +1 . 5 ± 0 . 5 — +0 . 8 ± 0 . 5 − 0 . 6 ± 0 . 5 DSR-LoRA +0 . 6 ± 0 . 5 − 2 . 6 ± 0 . 5 − 3 . 8 ± 0 . 5 +0 . 1 ± 0 . 5 +1 . 1 ± 0 . 5 +0 . 1 ± 0 . 5 +0 . 7 ± 0 . 5 − 0 . 8 ± 0 . 5 — − 1 . 4 ± 0 . 5 Mada +2 . 0 ± 0 . 5 − 1 . 3 ± 0 . 5 − 2 . 4 ± 0 . 5 +1 . 5 ± 0 . 5 +2 . 5 ± 0 . 5 +1 . 4 ± 0 . 5 +2 . 1 ± 0 . 5 +0 . 6 ± 0 . 5 +1 . 4 ± 0 . 5 — Note : each cell reports the mean accuracy difference (row model minus column model) in points, with its standard error of the mean; green indicates the row model is significantly more accurate and red significantly less accurate (Welch’s two-sample t -test, p < 0 . 05 ), with intensity scaling with the magnitude of the difference, while white marks non-significant pairs; ‘DS-R1’ stands for DeepSeek-R1, ‘DSR’ for DeepScaleR, and ‘S3’ for Still-3 . Table 9: Pairwise statistical significance of mean accuracy differences between models DS-R1 Still-3 DSR Open-RS1 Open-RS2 Open-RS3 Tina S3-LoRA DSR-LoRA Mada DS-R1 — < 0 . 001 < 0 . 001 0 . 296 0 . 293 0 . 233 0 . 895 0 . 011 0 . 217 < 0 . 001 Still-3 < 0 . 001 — 0 . 025 < 0 . 001 < 0 . 001 < 0 . 001 < 0 . 001 0 . 003 < 0 . 001 0 . 020 DSR < 0 . 001 0 . 025 — < 0 . 001 < 0 . 001 < 0 . 001 < 0 . 001 < 0 . 001 < 0 . 001 < 0 . 001 Open-RS1 0 . 296 < 0 . 001 < 0 . 001 — 0 . 055 0 . 973 0 . 254 0 . 115 0 . 863 0 . 011 Open-RS2 0 . 293 < 0 . 001 < 0 . 001 0 . 055 — 0 . 033 0 . 365 0 . 001 0 . 037 < 0 . 001 Open-RS3 0 . 233 < 0 . 001 < 0 . 001 0 . 973 0 . 033 — 0 . 197 0 . 093 0 . 877 0 . 006 Tina 0 . 895 < 0 . 001 < 0 . 001 0 . 254 0 . 365 0 . 197 — 0 . 009 0 . 185 < 0 . 001 S3-LoRA 0 . 011 0 . 003 < 0 . 001 0 . 115 0 . 001 0 . 093 0 . 009 — 0 . 151 0 . 280 DSR-LoRA 0 . 217 < 0 . 001 < 0 . 001 0 . 863 0 . 037 0 . 877 0 . 185 0 . 151 — 0 . 015 Mada < 0 . 001 0 . 020 < 0 . 001 0 . 011 < 0 . 001 0 . 006 < 0 . 001 0 . 280 0 . 015 — Note : each cell reports the two-sided p -value of the Welch two-sample t -test for the difference between the row and column models (per-seed average accuracy, ten seeds per model); grey intensity reflects the significance level ( p < 0 . 05 , p < 0 . 01 , p < 0 . 001 ), and white indicates non-significance ( p ≥ 0 . 05 ); ‘DS-R1’ stands for DeepSeek-R1, ‘DSR’ for DeepScaleR, and ‘S3’ for Still-3 . 20

Instant research discovery