Paper deep dive

Harness-RL: Black-Box Reinforcement Learning with Action-Args Decoupling for Central-Agent Multi-Agent Harnesses

Xinke Jiang, Zhixin Zhang, Zhibang Yang, Jiaran Gao, Rihong Qiu, Shijin Chen, Xu Chu, Junfeng Zhao, Yasha Wang

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 67

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/1/2026, 4:25:52 AM

Summary

The paper introduces Harness-RL, a structured reinforcement learning framework designed to train central-agent multi-agent harnesses. It addresses two main challenges: conflicting gradients between low-cardinality action labels and high-dimensional argument sequences, and the complexity of dynamic, non-linear multi-agent trajectories. The framework combines Conflict-Aware Policy Optimization (CAPO), which decouples gradients for actions and arguments using activation-based parameter partitioning, with a black-box RL mechanism that captures interface call records and constructs per-session prefix trees for reward alignment. Experiments on multi-hop QA and retrieval benchmarks show improved performance with Qwen2.5 models.

Entities (10)

Qwen2.5-1.5B · model · 99%Qwen2.5 3B · model · 99%Harness-RL · methodframework · 98%Conflict-Aware Policy Optimization · algorithm · 96%Central Agent · component · 95%Sub-agent · component · 95%Action · concept · 92%Args · concept · 92%DAPO · baseline-method · 90%GRPO · baseline-method · 90%

Relation Signals (7)

Harness-RL → evaluatedon → Qwen2.5-1.5B

confidence 95% · it reaches average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, respectively

Harness-RL → evaluatedon → Qwen2.5 3B

confidence 95% · it reaches average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, respectively

Harness-RL → uses → Conflict-Aware Policy Optimization

confidence 95% · We introduce Harness-RL, a structured reinforcement learning framework that combines Conflict-Aware Policy Optimization (CAPO) with interface-level black-box trajectory construction.

Conflict-Aware Policy Optimization → decouples → Action

confidence 90% · CAPO uses forward activations to identify parameter partitions associated with action and args tokens, then routes their policy gradients to the corresponding subspaces.

Conflict-Aware Policy Optimization → decouples → Args

confidence 90% · CAPO uses forward activations to identify parameter partitions associated with action and args tokens, then routes their policy gradients to the corresponding subspaces.

Harness-RL → outperforms → GRPO

confidence 85% · Experiments on multiple multi-agent reasoning and collaboration benchmarks show that Harness-RL improves planning, dispatch, and coordination while outperforming strong baselines.

Harness-RL → outperforms → DAPO

confidence 85% · Experiments on multiple multi-agent reasoning and collaboration benchmarks show that Harness-RL improves planning, dispatch, and coordination while outperforming strong baselines.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language model agents increasingly solve long-horizon tasks through multi-agent harnesses in which a central agent coordinates specialized sub-agents, tools, and environments. Training the central policy in such a harness raises two challenges. First, an action label is a low-cardinality decision, whereas its args form a high-dimensional conditional sequence; optimizing both with a shared sequence-level signal can produce conflicting gradients. Second, dynamic scheduling creates interdependent sessions with branches, parallel calls, and rewritten contexts, which cannot be faithfully reduced to one flat token sequence. We introduce Harness-RL, a structured reinforcement learning framework that combines Conflict-Aware Policy Optimization (CAPO) with interface-level black-box trajectory construction. The black-box component captures Interface Call Records, builds per-session prefix trees, and aligns outcome and process rewards with trainable tokens. CAPO uses forward activations to identify parameter partitions associated with action and args tokens, then routes their policy gradients to the corresponding subspaces. Harness-RL supports both central-only and joint multi-agent training. Across seven multi-hop question answering and agentic retrieval benchmarks, it reaches average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, respectively, while ablations validate the contribution of CAPO and favor central-only optimization in the evaluated setting. Our code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

63,947 characters extracted from source content.

Expand or collapse full text

Harness-RL: Black-Box Reinforcement Learning with Action–Args Decoupling for Central-Agent Multi-Agent Harnesses Xinke Jiang †thanks: Equal contribution. Affiliation: National Engineering Research Center of Software Engineering, Peking University, Beijing, China Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Zhixin Zhang Affiliation: National Engineering Research Center of Software Engineering, Peking University, Beijing, China Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Zhibang Yang Affiliation: National Engineering Research Center of Software Engineering, Peking University, Beijing, China Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Jiaran Gao Affiliation: National Engineering Research Center of Software Engineering, Peking University, Beijing, China Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Rihong Qiu Affiliation: National Engineering Research Center of Software Engineering, Peking University, Beijing, China Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Shijin Chen Affiliation: Guangxi Land and Resources Planning and Design Group Co., Ltd, Guangxi, China Xu Chu †thanks: Corresponding authors. Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Affiliation: Center on Frontiers of Computing Studies, Peking University, Beijing, China Junfeng Zhao Affiliation: School of Computer Science, Peking University, Beijing, China Affiliation: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China Yasha Wang Affiliation: National Engineering Research Center of Software Engineering, Peking University, Beijing, China Affiliation: Peking University Information Technology Institute (Tianjin Binhai), Tianjin, Chinaxinkejiang, yangzb@stu.pku.edu.cn, chu_xu, zhaojf, wangyasha@pku.edu.cn Abstract Large language model agents increasingly solve long-horizon tasks through multi-agent harnesses in which a central agent coordinates specialized sub-agents, tools, and environments. Training the central policy in such a harness raises two challenges. First, an action label is a low-cardinality decision, whereas its args form a high-dimensional conditional sequence; optimizing both with a shared sequence-level signal can produce conflicting gradients. Second, dynamic scheduling creates interdependent sessions with branches, parallel calls, and rewritten contexts, which cannot be faithfully reduced to one flat token sequence. We introduce Harness-RL, a structured reinforcement learning framework that combines Conflict-Aware Policy Optimization (CAPO) with interface-level black-box trajectory construction. The black-box component captures Interface Call Records, builds per-session prefix trees, and aligns outcome and process rewards with trainable tokens. CAPO uses forward activations to identify parameter partitions associated with action and args tokens, then routes their policy gradients to the corresponding subspaces. Harness-RL supports both central-only and joint multi-agent training. Across seven multi-hop question answering and agentic retrieval benchmarks, it reaches average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, respectively, while ablations validate the contribution of CAPO and favor central-only optimization in the evaluated setting. Our code is available at https://github.com/jiangxinke/Harness-RL. 1 Introduction Large Language Models (LLMs) continue to advance Guo et al. (2025); OpenAI (2025); Yang et al. (2024), their applications are expanding beyond single-turn text generation toward agent systems capable of planning Yao et al. (2022b), tool use Jiang et al. (2024), and interaction with external environments. For complex reasoning, long-horizon task execution, and decision making in open environments, a single agent that generates a linear sequence is often insufficient. Researchers have therefore developed sophisticated agent harnesses that organize a central agent Hou et al. (2024); Yue et al. (2025); Zhang et al. (2026), specialized sub-agents Wu et al. (2023); Hong et al. (2024); Qian et al. (2024), external tools, memory modules, and an execution environment to decompose, execute, and coordinate complex tasks Guo et al. (2024); Chen et al. (2025b). Figure 1: Two challenges addressed by Harness-RL. Left: unified sequence-level optimization entangles Action and Args gradients. Right: dynamic multi-agent execution produces context-dependent trajectory trees that cannot be directly concatenated. Within such systems, the harness organizes the overall task workflow, while the central agent generally serves as the core policy model. The central agent is responsible for global task decomposition, sub-task dispatch, context maintenance, and multi-round decision coordination. Specialized sub-agents, by contrast, perform concrete operations within their respective capability domains, such as retrieval Jin et al. (2025); Trivedi and others (2023), code execution Wu et al. (2023); Qian et al. (2024), and tool invocation Jiang et al. (2024). Every decision made by the central agent affects the subsequent execution path and environment state. An incorrect action or a low-quality args can therefore have a cumulative effect over a long interaction. Consequently, this raises a key question: how can we effectively train the central agent’s policy within a multi-agent harness to improve overall system performance? Existing studies have made substantial progress in multi-agent system architectures Wu et al. (2023); Hong et al. (2024); Qian et al. (2024), communication mechanisms Du et al. (2024); Yang et al. (2025), workflow design Qian et al. (2025a); Zhang et al. (2025); Hu et al. (2025), and memory management Zhang et al. (2026). Meanwhile, recent work such as Search-R1 Jin et al. (2025) has demonstrated the effectiveness of reinforcement learning (RL) for tool use and agentic task execution. However, existing agentic RL methods generally make two implicit assumptions. At the policy-optimization level, different types of generated tokens can be jointly updated under a unified sequence-level objective. At the trajectory-modeling level, interactions within an agent harness can be reconstructed as a single linear token sequence according to a predefined workflow. As illustrated in Figure 1, these assumptions no longer hold in agent harnesses with dynamic sub-agent scheduling, conditional branches, loops, concurrency, backtracking, and context compression, giving rise to two challenges in policy optimization and trajectory modeling: ❶ Challenge 1: optimization conflicts between heterogeneous Action and Args decisions. Agents at different levels make decisions through Action–Args pairs. A central agent selects an operation or dispatches a sub-agent through an Action, while a sub-agent uses an Action to execute a task or invoke a tool. Args describe the corresponding sub-task content or invocation args. An Action is a low-cardinality discrete decision, whereas Args constitute high-flexibility sequence generation conditioned on that Action. Although both occur in the same autoregressive sequence, they differ in decision characteristics and optimization objectives. For example, the same Dispatch Action may be paired with either a high-quality search query or an irrelevant or erroneous one. In this case, the difference in trajectory reward is primarily caused by the quality of Args. Standard GRPO Shao et al. (2024) or DAPO Yu et al. (2025), however, computes policy gradients uniformly over the full sequence, allowing the negative update induced by poor Args to interfere with an otherwise correct Action decision. As a result, the model may exhibit both unstable action planning and inadequate args generation. ❷ Challenge 2: multi-tree trajectory modeling and context alignment in a multi-agent harness. The central agent dynamically schedules multiple sub-agents, whose executions may themselves contain conditional branches, loops, concurrency, backtracking, and context compression. The complete harness interaction therefore no longer corresponds to a single trajectory tree; instead, it forms multiple related agent interaction trees. Contexts associated with different agents and branches must remain isolated, while retaining the cross-agent dependencies created by task dispatch, message passing, and shared environment state. These trajectories cannot be concatenated directly. Manually implementing rollout scripts, trajectory concatenation, and token-mask logic for each harness incurs substantial training-system overhead and is difficult to reuse. To address these challenges, we propose Harness-RL, a black-box reinforcement learning framework with action–args decoupling for central-agent-driven multi-agent harnesses. The framework consists of two connected components: Conflict-Aware Policy Optimization (CAPO) and a black-box RL mechanism for multi-agent harnesses. CAPO constructs functional partitions for Action and Args and decouples their gradients. The black-box RL mechanism converts actual harness interactions into context-aligned training trajectories with attributable rewards. Together, the two components provide a unified solution spanning heterogeneous-decision optimization and complex multi-agent trajectory modeling. ❶ For C1, we introduce CAPO. It uses activation signals from model forward passes to measure the response strength of Action Tokens and Args Tokens across layers and structural units, and uses these statistics to construct the corresponding functional parameter partitions. During RL optimization, policy gradients produced by Action Tokens and Args Tokens primarily update their respective partitions. This design decouples action selection from args generation and reduces parameter interference between the two heterogeneous decision types. ❷ For C2, we introduce a black-box RL mechanism for multi-agent harnesses. The mechanism treats the harness as a black box and captures the input and output of each agent interaction. It reconstructs multiple related interaction trees according to inter-agent invocation relationships, thereby representing dynamic scheduling, branched execution, and cross-agent contextual dependencies in a unified form. It further groups contiguous generated content into action-level decision units and attributes global outcome rewards and local process rewards to the relevant agents and decisions. This enables trajectory modeling, context alignment, reward attribution, and policy optimization for complex multi-agent interactions. Our main contributions are as follows: • We formulate policy training for central-agent-driven multi-agent harnesses and propose Harness-RL, which converts long-horizon, multi-round interactions among agents, tools, and the environment into structured RL trajectories and supports both central-only and joint multi-agent training. • We combine CAPO, which uses forward activations and token-specific gradient routing to decouple Action and Args optimization, with a black-box RL mechanism that reconstructs multi-session trajectories and aligns outcome and process rewards with trainable decisions. • Experiments on multiple multi-agent reasoning and collaboration benchmarks show that Harness-RL improves planning, dispatch, and coordination while outperforming strong baselines. Ablations further validate CAPO and the choice of training scope. 2 Preliminaries 2.1 Central-Agent Multi-Agent Harnesses We consider a structured multi-agent harness whose policy layer contains a central agent and a set of specialized sub-agents. Let ℛ=∪,=1,…,N,R=\ c\ ,A=\ a_1,…, a_N\, where c denotes the central agent and A the sub-agent set. The harness ℋH is an execution and control layer outside the policies. It manages messages, workflow control, dispatch, tool invocation, state updates, and termination. Tools, search engines, and task environments are execution resources exposed through ℋH. Given a task q, the harness coordinates multiple rounds of interaction until termination. We call the complete interaction a harness rollout. The e-th rollout is τe=(re,j,se,j,ae,j(re,j),oe,j+1)j=0Te−1, _e= \ (r_e,j,s_e,j,a_e,j^(r_e,j),o_e,j+1 ) \_j=0^T_e-1, (1) where j indexes interaction steps, re,j∈ℛr_e,j is the scheduled role, se,js_e,j is that role’s visible state, ae,j(re,j)a_e,j^(r_e,j) is its action, and oe,j+1o_e,j+1 is the resulting observation, such as a sub-agent response, tool result, or environment feedback. Local contexts may be isolated across roles, while dispatch, messaging, and execution state connect them at the rollout level. Central agent. The central agent interprets the task, constructs and revises a plan, dispatches sub-tasks, and maintains global state. For long-horizon execution, it maintains a dynamic memory Mj()M_j^( c): Mj+1()=Update⁡(Mj(),aj(),oj+1()).M_j+1^( c)=Update_ c (M_j^( c),a_j^( c),o_j+1^( c) ). (2) We represent its structured decision as aj()=(uj(),zj()),a_j^( c)= (u_j^( c),z_j^( c) ), (3) where uj()∈Reason,Dispatch,Summary,u_j^( c)∈\ Reason, Dispatch, Summary\, is the action and zj()z_j^( c) is args sequence. Reason represents planning and decision analysis; Dispatch specifies a sub-agent task and its args; and Summary consolidates intermediate results and may support revision. Sub-agents. A sub-agent i a_i executes an assigned task in a isolated context and may invoke tools, search engines, or the task environment. Its result is returned to the harness, serialized as an observation for the central agent, and incorporated into subsequent state. In the central-only setting, sub-agents are part of the environment and receive no policy gradient. In joint training, sub-agents are trainable policies with their own sessions. 2.2 Policy Optimization in a Multi-Agent Harness Let ℛtrain⊆ℛR_train denote the trainable roles. The central-only setting uses ℛtrain=R_train=\ c\; joint training may include sub-agents. For role r, let θr _r be its policy parameters and define Θtrain=θr∣r∈ℛtrain. _train=\ _r r _train\. At interaction step j, the scheduled role rjr_j samples an action aj(rj)a_j^(r_j) from its policy conditioned on the visible state sjs_j. The harness then executes the action, returns observation oj+1o_j+1, and updates the interaction state to sj+1s_j+1. Together, the trainable role policies and the harness induce the rollout distribution ΠΘtrain,ℋ(⋅∣q) _ _train,H(· q). With rollout reward R⁡(τ)R(τ), the learning objective is Θtrain∗=argmaxΘtrainq∼,τ∼ΠΘtrain,ℋ(⋅∣q)[R(τ)]. _train^*= _ _trainE_q ,\,τ _ _train,H(· q) [R(τ) ]. (4) Harness rollouts differ from ordinary prompt–response optimization in three ways: they are long-horizon, combine heterogeneous decisions, and continually condition on dynamic observations. Standard GRPO or DAPO typically broadcasts one rollout advantage to all trainable response tokens. This can couple action selection with args generation. Moreover, multiple model calls and role switches do not naturally form one trainable token sequence. Section 3 addresses these optimization and trajectory-construction problems jointly. 3 Harness-RL This section presents the core methodology of Harness-RL. Section 3.1 provides an overview of the framework; Section 3.2 introduces Conflict-Aware Policy Optimization (CAPO); and Section 3.3 presents a black-box reinforcement learning method for multi-agent harnesses. 3.1 Framework Overview Figure 2: Overall Framework of Harness-RL. Harness-RL couples CAPO with an interface-level black-box RL mechanism. The black-box mechanism converts harness interactions into token-aligned training data, while CAPO routes the gradients induced by Action and Args tokens to their respective functional parameter subsets. For each query q, the two components operate on a shared structured training set ℋ(q)D_H(q), abbreviated as ℋD_H when the query is clear. On this dataset, CAPO decomposes the policy objective into Action and Args terms: JHarness-RL(ℋ)=Jaction(ℋ)+Jargs(ℋ).J_Harness-RL(D_H)=J_action(D_H)+J_args(D_H). (5) Here, JactionJ_action and JargsJ_args denote the contributions from the corresponding token types; Section 3.2 details their optimization. The construction of ℋD_H proceeds in three stages. First, at call k of session h in rollout τe _e, the black-box mechanism captures an Interface Call Record ξe,h,k _e,h,k containing the exact input tokens, output tokens, and aligned sampling log-probabilities. Second, prefix matching organizes the records within each session into a token-prefix tree, from which captured output spans directly yield base samples in ~ℋ(τe) D_H( _e). Third, rollout-group outcome and process rewards are converted into token-aligned advantages, producing the structured training set ℋD_H. Section 3.3 formalizes these three stages. 3.2 Conflict-Aware Policy Optimization Conflict between Heterogeneous Decisions Let ℐactionI_action and ℐargsI_args denote action- and args-token positions. For a trainable role r∈ℛtrainr _train, we write θ=θrθ= _r when the role is clear; CAPO is applied independently to each trainable role. Standard sequence-level optimization can be written as J⁡(θ)=Jaction(θ)+Jargs(θ),J(θ)=J_action(θ)+J_args(θ), (6) ∇θJ=∇θJaction+∇θJargs. _θJ= _θJ_action+ _θJ_args. (7) The two decisions have different functions: action tokens select high-level operations, while args tokens specify executable content. Their updates conflict when ⟨∇θJaction,∇θJargs⟩<0. _θJ_action, _θJ_args <0. Applying both gradients to the full parameter space may therefore impair action selection, args quality, or both. CAPO constructs the functional partitions and applies gradient routing in three steps. ❶High-Quality Trajectory Construction. For every training query, CAPO samples multiple agentic rollouts and retains successful ones as a probing set. This rejection-sampling procedure reduces contamination from activation patterns associated with failed behavior. ❷Activation Statistics and Functional Partitioning. Assume the policy has L structural units and let hl,j,t(n)h_l,j,t^(n) denote the activation of unit j in layer l at token t for probing example n. We estimate decision-specific importance by Iaction(l,j) I_action(l,j) =1N∑n=1N1|ℐaction(n)|∑t∈ℐaction(n)ϕ⁡(hl,j,t(n)), = 1N _n=1^N 1|I_action^(n)| _t _action^(n)φ (h_l,j,t^(n) ), (8) Iargs(l,j) I_args(l,j) =1N∑n=1N1|ℐargs(n)|∑t∈ℐargs(n)ϕ⁡(hl,j,t(n)), = 1N _n=1^N 1|I_args^(n)| _t _args^(n)φ (h_l,j,t^(n) ), (9) where ϕ⁡(⋅)φ(·) measures positive activation strength. We select the most responsive units for each behavior: action=TopK⁡(Iaction,Kaction),U_action=TopK (I_action,K_action ), and args=TopK⁡(Iargs,Kargs).U_args=TopK (I_args,K_args ). These unit sets induce parameter- or unit-level gradient masks. Units selected by both sets are treated as shared and follow the implementation’s overlap rule. For the role under optimization, θaction,θargs⊆θr _action, _args _r denote the parameters associated with the selected units. ❸Token-Specific Gradient Routing. For role r, let e(r)⊆~ℋ(τe)S_e^(r) D_H( _e) be its linear samples from rollout e. For sample SnS_n, let ℐaction,nI_action,n and ℐargs,nI_args,n be disjoint trainable response positions ℐaction,n∩ℐargs,n=∅I_action,n _args,n= and ℐaction,n∪ℐargs,n⊆ℐout,n.I_action,n _args,n _out,n. Action spans are identified from structured action markers; args spans are identified from the corresponding fields. Inputs, observations, replayed context, and frozen-role outputs are excluded. Aggregate positions as ℐ¯b(r)=(n,t)∣∃e,Sn∈e(r),t∈ℐb,n. I_b^(r)=\(n,t) ∃ e,\;S_n _e^(r),\;t _b,n\. (10) Where b∈action,argsb∈\action,args\. Let PnP_n be the prompt length and yn,t=xn,Pn+ty_n,t=x_n,P_n+t the t-th response token. Define its policy log-probability as ℓn,t(θ)=log⁡πθ(yn,t∣xn,≤Pn+t−1). _n,t(θ)= _θ(y_n,t x_n,≤ P_n+t-1). (11) For b∈action,argsb∈\action,args\, and omitting clipping for clarity, the routed gradient of the corresponding term in Eq. (5) is ∇θbJb=∑(n,t)∈ℐ¯b(r)An,t∇θbℓn,t(θ). _ _bJ_b= _(n,t)∈ I_b^(r)A_n,t _ _b _n,t(θ). (12) By construction, ℐ¯b(r) I_b^(r) contains only trainable positions; An,tA_n,t is the token-aligned advantage defined in Eq. (22). 3.3 Interface-Level Black-Box RL Because role-specific calls maintain isolated contexts and interact through harness messages and observations, a harness rollout is not a single linear token sequence. We construct trainable RL data in three steps: Interface Call Record capture, session-wise token-prefix tree construction, and rollout-group reward attribution. ❶Interface Call Records. For session h in rollout e, let r⁡(h)r(h) be its role. Before call k, the harness provides messages e,h,kC_e,h,k and tool specifications Ue,h,kU_e,h,k. Applying the model’s chat template gives the exact Token-In: e,h,k=Template⁡(e,h,k,Ue,h,k).X_e,h,k=Template(C_e,h,k,U_e,h,k). (13) The role policy samples Token-Out and stores aligned rollout log-probabilities: e,h,k _e,h,k ∼πθr⁡(h)(r⁡(h))(⋅∣e,h,k), _ _r(h)^(r(h))(· _e,h,k), (14) ℓe,h,kroll _e,h,k^roll =(ℓe,h,k,1roll,…,ℓe,h,k,|e,h,k|roll). =( _e,h,k,1^roll,…, _e,h,k,|Y_e,h,k|^roll). (15) Thus one call record is ξe,h,k=(e,h,k,e,h,k,ℓe,h,kroll), _e,h,k= (X_e,h,k,Y_e,h,k, _e,h,k^roll ), with rollout, session, role, and call identifiers stored as metadata. This record contains only model-side training information. The harness executes the output and writes tool, sub-agent, or environment feedback into the next message context:(σe,h,k+1,e,h,k+1)=Transitionℋ⁡(σe,h,k,e,h,k,e,h,k).( _e,h,k+1,C_e,h,k+1)=Transition_H( _e,h,k,C_e,h,k,Y_e,h,k). Here, σe,h,k _e,h,k is the harness execution state. The transition connects the captured segment to the context of subsequent calls. ❷Prefix Matching and Tree Construction. Let eU_e be the sessions in rollout e. For each captured call, concatenate its exact input and output tokens as e,h,k=e,h,k⊕e,h,kZ_e,h,k=X_e,h,k _e,h,k. Calls in each session form a prefix tree, and the session trees form the rollout-level tree collection: e,h _e,h =PrefixTree⁡(e,h,kk=1Ke,h), =PrefixTree (\Z_e,h,k\_k=1^K_e,h ), (16) e _e =e,hh∈e. =\T_e,h\_h _e. Shared prefixes are merged and unmatched suffixes form separate branches. A rewritten context is attached at its longest shared prefix, while the pre-rewrite output remains on the original branch. Thus eT_e encodes token visibility rather than causal execution. Each output-span node v⁡(n)v(n) in eT_e corresponds to one Interface Call Record ξn _n and defines one base sample: Sn S_n =(ξn,n,ηn), =( _n,m_n, _n), (17) ~ℋ(τe) D_H( _e) =Sn∣v⁡(n)∈out(e). =\S_n v(n) _out(T_e)\. Here out(e)V_out(T_e) denotes captured output-span nodes. The record ξn _n supplies the exact input tokens, output tokens, and aligned sampling log-probabilities. The mask satisfies mn,t=1m_n,t=1 only for new Token-Out positions from trainable roles and is zero elsewhere, while ηn _n stores call and tree-node metadata. For n=(e,h,k)n=(e,h,k), the response length is Lnresp=|e,h,k|L_n^resp=|Y_e,h,k|. Reusing the captured token IDs avoids retokenization mismatch. ❸Segment-Wise Reward Attribution. For query q, sample a rollout group G⁡(q)=τee=1KG(q)=\ _e\_e=1^K. The outcome reward and its group-normalized advantage are Reout R_e^out =F1token⁡(ye,ye∗)Ieartifact, =F1_token(y_e,y_e^*)I_e^artifact, (18) Aeout A_e^out =Reout−μG⁡(q)outσG⁡(q)out+δ, = R_e^out- _G(q)^out _G(q)^out+δ, (19) where IeartifactI_e^artifact indicates whether the required output constraints are satisfied. We parse trainable Action–Args spans into decision units eQ_e. For unit d, let ℐe,dI_e,d contain its trainable token positions. Its process reward and normalized advantage are Re,dproc R_e,d^proc =∑v=1VλvRv(e,d), = _v=1^V _vR_v(e,d), (20) Ae,dproc A_e,d^proc =Re,dproc−μGd(q)procσGd(q)proc+δ, = R_e,d^proc- _G_d(q)^proc _G_d(q)^proc+δ, (21) where Rv(e,d)R_v(e,d) is a process signal with weight λv _v, and Gd(q)G_d(q) contains units with the same role and action type. We align rollout- and decision-level advantages to tokens by An,t A_n,t =mn,t[Ae⁡(n)out+λproc∑d∈e⁡(n) =m_n,t [A_e(n)^out+ _proc _d _e(n) (22) [(n,t)∈ℐe⁡(n),d]Ae⁡(n),dproc]. [(n,t) _e(n),d]A_e(n),d^proc ]. The final task-level training set is ~ℋ(q) D_H(q) =⋃τe∈G⁡(q)~ℋ(τe), = _ _e∈ G(q) D_H( _e), (23) ℋ(q) _H(q) =(Sn,n)∣Sn∈~ℋ(q). =\(S_n,A_n) S_n∈ D_H(q)\. where n=(An,1,…,An,Lnresp)A_n=(A_n,1,…,A_n,L_n^resp). For each trainable token, let ρn,t=exp⁡(ℓn,t(θr⁡(n))−ℓn,troll) _n,t= ( _n,t( _r(n))- _n,t^roll) and ρ¯n,t=clip⁡(ρn,t,1−ϵ,1+ϵ) ρ_n,t=clip( _n,t,1-ε,1+ε). Let ℐetrainI_e^train collect positions with mn,t=1m_n,t=1 in ~ℋ(τe) D_H( _e). The rollout-balanced clipped objective is JHarness-RL J_Harness-RL =1K∑e=1K1max⁡(|ℐetrain|,1) = 1K _e=1^K 1 (|I_e^train|,1) (24) ∑(n,t)∈ℐetrainmin⁡(ρn,tAn,t,ρ¯n,tAn,t). _(n,t) _e^train ( _n,tA_n,t, ρ_n,tA_n,t ). The per-rollout normalization prevents branch-rich rollouts from dominating the update. Partitioning the sum into Action and Args positions gives Eq. (5), to which CAPO applies the routed gradients in Eq. (12). 4 Experiments We systematically evaluate Harness-RL along the following four dimensions: • RQ1: Does Harness-RL consistently outperform existing baselines on multi-agent reasoning and collaboration benchmarks? • RQ2: How do CAPO and its conflict-aware gradient-routing mechanism contribute to overall performance? • RQ3: How does the training scope, central-only training versus joint training of all agents, affect system performance? • RQ4: How do the planning and dispatch behaviors of Harness-RL qualitatively differ from those of the baselines? 4.1 Experimental Setup ❶Training Data. We construct RL queries from the training splits of 2WikiMultiHopQA Ho et al. (2020) and HotpotQA Yang et al. (2018). We remove queries that require no retrieval or only a trivial single retrieval, retaining examples that require multi-step reasoning and sub-agent coordination. ❷ Evaluation Benchmarks. We report token-level F1 Yacouby and Axman (2020) (%) on seven benchmarks: 2WikiMultiHopQA Ho et al. (2020), HotpotQA Yang et al. (2018), Bamboogle Press et al. (2022), FRAMES Krishna et al. (2025), MuSiQue Trivedi et al. (2022), Natural Questions Kwiatkowski and others (2019), and TriviaQA Joshi and others (2017). 2WikiMultiHopQA and HotpotQA are in-domain; the remaining five are out-of-domain. This collection measures multi-hop reasoning, fact retrieval, and transfer to unseen question distributions. ❸ Backbones and agents. We use Qwen2.5-1.5B and Qwen2.5-3B Yang et al. (2024) as policy backbones. The harness contains a Search Agent equipped with Wikipedia and knowledge-graph retrieval tools and a Conclusion Agent equipped with an answer-synthesis tool. Unless otherwise stated, these sub-agents are frozen environment components and only the central agent is optimized. The agent configuration follows STACKPLANNER Zhang et al. (2026). ❹ Baselines. We compare four families: (i) no-retrieval methods (Base and chain-of-thought Wei et al. (2022)); (i) naive RAG (FS-RAG and FL-RAG); (i) agentic RAG (ReAct Yao et al. (2022a), IRCoT Trivedi and others (2023), and TC-RAG Jiang et al. (2024)); and (iv) RL-based agents (ReSearch Chen et al. (2025a), Search-R1 Jin et al. (2025), AEPO Dong et al. (2025a), ARPO Dong et al. (2025b), MEM1 Zhou et al. (2025), and AgenticRAG-R1 Jiang et al. (2025)). ❺ Training details. Each query produces K=4K=4 rollouts. We set the process-reward coefficient to λproc=0.1 _proc=0.1. And we train for 100 optimizer updates using four NVIDIA A100-SXM4-80GB GPUs for Qwen2.5-1.5B-Instruct and eight for Qwen2.5-3B-Instruct. Complete hyperparameters are provided in Appendix B. 4.2 Main Result Analysis Method In-Domain Out-of-Domain Paradigm Approach 2Wiki HotpotQA Bamboogle FRAMES MuSiQue NQ TriviaQA Avg. Qwen2.5-1.5B No RAG Base 21.95 20.20 5.87 9.51 8.26 11.31 32.81 15.70 CoT 16.75 17.90 19.74 8.43 7.22 9.82 25.17 15.00 Naive RAG FS-RAG 21.37 26.55 15.31 11.40 9.50 17.74 44.73 20.94 FL-RAG 26.08 28.14 14.84 12.58 10.30 21.94 48.59 23.21 Agentic RAG ReAct 12.43 23.48 12.54 8.63 7.99 21.06 33.91 17.15 IRCoT 21.79 26.68 17.49 9.32 8.73 22.93 44.19 21.59 TC-RAG 25.23 22.29 11.63 10.27 7.90 14.21 34.05 17.94 RL-based ReSearch 25.55 30.45 22.16 9.21 9.54 29.07 44.65 24.38 Search-R1 26.51 20.66 13.80 9.65 8.36 11.93 25.76 16.67 AEPO 17.34 14.75 11.58 8.98 7.11 9.55 23.02 13.19 ARPO 20.02 15.48 12.38 9.11 6.16 9.19 23.96 13.76 MEM1 14.43 14.34 12.03 8.18 6.08 18.41 22.06 13.65 AgenticRAG-R1 29.53 30.88 20.00 13.46 16.48 31.35 51.83 27.65 Ours Harness-RL 41.63 44.44 28.63 37.71 27.83 62.04 58.22 42.93 Qwen2.5-3B No RAG Base 23.98 24.08 9.45 8.01 9.70 14.27 38.84 18.33 CoT 18.90 23.82 20.80 7.16 10.47 16.53 39.62 19.61 Naive RAG FS-RAG 15.47 25.85 10.48 10.42 7.64 19.84 45.38 19.30 FL-RAG 16.80 26.78 11.05 9.19 7.29 21.93 48.50 20.22 Agentic RAG ReAct 25.09 34.37 24.86 10.53 13.92 27.19 46.04 26.00 IRCoT 15.89 24.50 25.27 6.79 12.43 27.86 49.19 23.13 TC-RAG 28.47 21.94 17.59 7.69 8.99 20.74 50.46 22.27 RL-based ReSearch 27.23 33.96 15.09 10.00 9.47 34.61 53.93 26.33 Search-R1 29.90 37.24 29.90 10.76 13.53 34.73 55.08 30.16 AEPO 23.01 28.71 22.09 12.26 11.70 26.76 47.78 24.62 ARPO 29.55 36.48 27.32 13.49 13.38 33.29 53.66 29.60 MEM1 18.06 20.15 5.19 4.99 4.47 19.18 33.09 15.02 AgenticRAG-R1 32.92 44.00 31.48 16.23 16.48 37.15 56.62 33.55 Ours Harness-RL 45.78 57.35 40.37 34.90 24.33 63.70 68.07 47.79 Table 1: Performance comparison (token-level F1, %) on in-domain and out-of-domain benchmarks. Bold denotes the best result and underlining denotes the strongest baseline for each backbone. Backbone Method 2Wiki HotpotQA Bamboogle FRAMES MuSiQue NQ TriviaQA Avg. Qwen2.5-1.5B Harness-RL 41.63 44.44 28.63 37.71 27.83 62.04 58.22 42.93 w/o CAPO 41.33 43.70 30.74 32.68 23.02 59.44 56.37 41.04 Qwen2.5-3B Harness-RL 45.78 57.35 40.37 34.90 24.33 63.70 68.07 47.79 w/o CAPO 47.56 53.70 38.89 32.67 25.02 60.37 65.31 46.22 Table 2: Ablation of CAPO measured by token-level F1 (%). Blue rows denote the complete method, and bold indicates the better result within each backbone. Metric Training Scope 2Wiki HotpotQA Bamboogle FRAMES MuSiQue NQ TriviaQA Avg. Accuracy Central only 42.22 55.56 26.67 28.89 17.78 61.67 67.34 42.88 All agents 45.34 53.33 22.22 22.56 15.56 55.56 68.89 40.49 F1 Central only 45.78 57.35 40.37 34.90 24.33 63.70 68.07 47.79 All agents 43.24 52.59 38.15 33.54 24.86 58.78 66.38 45.36 Table 3: Effect of training scope with Qwen2.5-3B. Blue rows denote central-only training, and bold indicates the better result under each metric. Figure 3: Layer-wise Action/Args functional partitions derived from forward activations. Blue and orange denote Action and Args importance and their selected layers, respectively. Left: Qwen2.5-1.5B (28 layers). Right: Qwen2.5-3B (36 layers). Table 1 compares Harness-RL with the baselines on the multi-hop QA and agentic retrieval benchmarks. With Qwen2.5-1.5B, Harness-RL achieves an average F1 of 42.93, outperforming AgenticRAG-R1, the strongest baseline overall, by 15.28 points and the strong RL-based ReSearch baseline by 18.55 points. Specifically, Harness-RL obtains F1 scores of 41.63, 44.44, 28.63, 37.71, 27.83, 62.04, and 58.22 on 2Wiki, HotpotQA, Bamboogle, FRAMES, MuSiQue, NQ, and TriviaQA, respectively, achieving the best result on all seven benchmarks. With Qwen2.5-3B, Harness-RL reaches an average F1 of 47.79, exceeding the strongest baseline, AgenticRAG-R1, by 14.24 points. It also outperforms Search-R1 and ARPO by 17.63 and 18.19 points, respectively. On 2Wiki, HotpotQA, Bamboogle, FRAMES, MuSiQue, NQ, and TriviaQA, Harness-RL achieves F1 scores of 45.78, 57.35, 40.37, 34.90, 24.33, 63.70, and 68.07, respectively, again ranking first on every benchmark. These results show that Harness-RL consistently improves the reasoning, dispatch, and coordination capabilities of the central agent across both model scales, providing an affirmative answer to RQ1. 4.3 Component Analysis To answer RQ2–RQ3, we compare CAPO with standard GRPO and central-only training with joint multi-agent training. ❶Effectiveness of CAPO. As shown in Table 2, Harness-RL achieves an average F1 of 42.93 with Qwen2.5-1.5B, exceeding w/o CAPO (41.04) by 1.89 points. With Qwen2.5-3B, it improves the average F1 from 46.22 to 47.79, a gain of 1.57 points. CAPO produces clear improvements on FRAMES, NQ, and TriviaQA at both model scales. Its effect nevertheless varies across datasets: the complete method underperforms w/o CAPO on Bamboogle with Qwen2.5-1.5B and on 2Wiki and MuSiQue with Qwen2.5-3B. One possible explanation is that the training data are primarily drawn from 2WikiMultiHopQA, whereas the evaluation benchmarks differ in question decomposition, reasoning-path length, and the difficulty of args generation. Consequently, the effectiveness of decoupled optimization may vary across data distributions. Overall, the improvement in average F1 demonstrates the contribution of CAPO and its conflict-aware gradient-routing mechanism, answering RQ2. ❷Functional Partition Analysis. To further examine the behavior of CAPO, Figure 3 visualizes the normalized layer-wise importance used to construct the Action and Args functional partitions. For both Qwen2.5-1.5B and Qwen2.5-3B, Action Tokens and Args Tokens exhibit different importance distributions across layers, resulting in distinct selected partitions. This observation supports the use of activation-based functional partitioning to separate the two heterogeneous decision types. ❸Effect of Training Scope. Table 3 compares Central only, which trains only the central agent, with All Agents, which jointly trains the central agent and sub-agents. Both configurations use decoupled policy optimization with Qwen2.5-3B. In terms of accuracy, their average scores are 42.88 and 40.49, respectively, giving central-only training a 2.38-point advantage. In terms of F1, central-only and all-agent training achieve 47.79 and 45.36, respectively, a difference of 2.42 points. Although joint training is better on a small number of individual benchmarks, it does not improve overall performance. These results suggest that, in the evaluated central-coordination tasks, optimizing only the central agent better preserves the consistency of the global planning objective and reduces interference among policy updates from different agents, answering RQ3. 5 Conclusion We introduced Harness-RL, a reinforcement learning framework for central-agent policies in structured multi-agent harnesses. Its interface-level black-box mechanism reconstructs multi-round, multi-session, and branching interactions as aligned training samples, then combines rollout-level outcome rewards with decision-level process rewards. CAPO complements this representation by using forward activations to identify action- and args-related parameter subsets and routing their policy gradients accordingly. Across seven multi-hop QA and agentic retrieval benchmarks, Harness-RL achieves average F1 scores of 42.93 with Qwen2.5-1.5B and 47.79 with Qwen2.5-3B, outperforming the strongest non-ours row in the reported comparison by 15.28 and 14.24 points, respectively. Removing CAPO reduces average F1 by 1.89 and 1.57 points, and central-only training outperforms joint training in the evaluated 3B harness. Future work should test larger policies, more diverse harness topologies, dynamic functional partitions, multimodal actions, and long-term memory, while measuring the robustness and computational cost of partition estimation. Ethical considerations To evaluate the proposed method, we conducted experiments exclusively on publicly available benchmark datasets, including 2WikiMultiHopQA, HotpotQA, Bamboogle, FRAMES, MuSiQue, Natural Questions, and TriviaQA, in accordance with their respective licenses and usage policies. We did not intentionally collect or use any personally identifiable information, and no human or animal subjects were involved in this research. References Chen et al. (2025a) M. Chen, L. Sun, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470. Cited by: §4.1. Chen et al. (2025b) S. Chen, Y. Liu, W. Han, W. Zhang, and T. Liu A survey on llm-based multi-agent system: recent advances and new frontiers in application. External Links: 2412.17481, Link Cited by: §A.1, §1. [3] X. Chu, X. Jiang, R. Qiu, J. Gao, and J. Zhao MODEL shapley: find your ideal parameter player via one gradient backpropagation. In NeurIPS 2025, Cited by: §A.3. Cui et al. (2026) Y. Cui, H. Fu, H. Zhang, L. Wang, and C. Zuo Free-mad: consensus-free multi-agent debate. In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), p. 31977–31997. External Links: Link, Document Cited by: §A.1. Dai et al. (2021) D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696. Cited by: §A.3. Dong et al. (2025a) G. Dong, L. Bao, Z. Wang, K. Zhao, X. Li, J. Jin, J. Yang, H. Mao, F. Zhang, K. Gai, et al. Agentic entropy-balanced policy optimization. arXiv preprint arXiv:2510.14545. Cited by: §A.2, §4.1. Dong et al. (2025b) G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. Cited by: §A.2, §4.1. Du et al. (2024) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, R. Salakhutdinov, Z. Kolter, K. A. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 11733–11763. External Links: Link Cited by: §A.1, §1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §A.2, §1. Guo et al. (2021) D. Guo, A. M. Rush, and Y. Kim Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), p. 4884–4896. External Links: Link, Document Cited by: §A.3. Guo et al. (2024) T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, p. 8048–8057. External Links: Link Cited by: §A.1, §1. Ho et al. (2020) X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), p. 6609–6625. External Links: Link Cited by: §4.1. Hong et al. (2024) S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber MetaGPT: meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: Link Cited by: §A.1, §1, §1. Hou et al. (2024) X. Hou, M. Yang, W. Jiao, X. Wang, Z. Tu, and W. X. Zhao CoAct: A global-local hierarchy for autonomous agent collaboration. CoRR abs/2406.13381. External Links: Link, Document, 2406.13381 Cited by: §A.1, §1. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §A.3. Hu et al. (2025) M. Hu, Y. Zhou, W. Fan, Y. Nie, Z. Ye, B. Xia, T. Sun, Z. Jin, Y. Li, Z. Zhang, Y. Wang, Q. Ye, B. Ghanem, P. Luo, and G. Li OWL: optimized workforce learning for general multi-agent assistance in real-world task automation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. Ruíz, and A. Loaiza-Bonilla (Eds.), External Links: Link Cited by: §A.1, §1. Jiang et al. (2024) X. Jiang, Y. Fang, R. Qiu, H. Zhang, Y. Xu, H. Chen, W. Zhang, R. Zhang, Y. Fang, X. Chu, et al. TC-rag: turing-complete rag’s case study on medical llm systems. arXiv preprint arXiv:2408.09199. Cited by: §1, §1, §4.1. Jiang et al. (2025) X. Jiang, J. Gao, R. Qiu, Z. Zhang, W. Zhang, Y. Fang, and H. Ding Agentic rag-r1: enhance agentic rag reasoning capacity via reinforcement learning. Note: https://github.com/jiangxinke/Agentic-RAG-R1GitHub repository Cited by: §4.1. Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-R1: training LLMs to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Cited by: §A.2, §1, §1, §4.1. Joshi et al. (2017) M. Joshi et al. TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In ACL, Cited by: §4.1. Krishna et al. (2025) S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui Fact, fetch, and reason: a unified evaluation of retrieval-augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4745–4759. Cited by: §4.1. Kwiatkowski et al. (2019) T. Kwiatkowski et al. Natural questions: a benchmark for question answering research. TACL. Cited by: §4.1. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), External Links: Link Cited by: §A.3. Meng et al. (2023) K. Meng, A. S. Sharma, A. J. Andonian, Y. Belinkov, and D. Bau Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: §A.3. OpenAI (2025) OpenAIOpenAI-o1(Website) Note: Accessed: 2025-05-16 External Links: Link Cited by: §A.2, §1. Press et al. (2022) O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. Cited by: §4.1. Qian et al. (2024) C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun ChatDev: communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), p. 15174–15186. External Links: Link, Document Cited by: §A.1, §1, §1, §1. Qian et al. (2025a) C. Qian, Z. Xie, Y. Wang, W. Liu, K. Zhu, H. Xia, Y. Dang, Z. Du, W. Chen, C. Yang, Z. Liu, and M. Sun Scaling large language model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: Link Cited by: §A.1, §1. Qian et al. (2025b) C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-Tür, G. Tur, and H. Ji ToolRL: reward is all tool learning needs. External Links: 2504.13958, Link Cited by: §A.2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §A.2, §1. Tang et al. (2024) T. Tang, W. Luo, H. Huang, D. Zhang, X. Wang, X. Zhao, F. Wei, and J. Wen Language-specific neurons: the key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 5701–5715. External Links: Link, Document Cited by: §A.3. Todd et al. (2023) E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau Function vectors in large language models. arXiv preprint arXiv:2310.15213. Cited by: §A.3. Trivedi et al. (2022) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal MuSiQue: multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics 10, p. 539–554. Cited by: §4.1. Trivedi et al. (2023) H. Trivedi et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In ACL, Cited by: §1, §4.1. Wang et al. (2025) Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, E. Gottlieb, Y. Lu, K. Cho, J. Wu, L. Fei-Fei, L. Wang, Y. Choi, and M. Li RAGEN: understanding self-evolution in llm agents via multi-turn reinforcement learning. External Links: 2504.20073, Link Cited by: §A.2. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: §4.1. Wu et al. (2023) Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang AutoGen: enabling next-gen LLM applications via multi-agent conversation framework. CoRR abs/2308.08155. External Links: Link, Document, 2308.08155 Cited by: §A.1, §1, §1, §1. Yacouby and Axman (2020) R. Yacouby and D. Axman Probabilistic extension of precision, recall, and f1 score for more thorough evaluation of classification models. In Proceedings of the first workshop on evaluation and comparison of NLP systems, p. 79–91. Cited by: §4.1. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §1, §4.1. Yang et al. (2025) Y. Yang, H. Chai, S. Shao, Y. Song, S. Qi, R. Rui, and W. Zhang AgentNet: decentralized evolutionary coordination for llm-based multi-agent systems. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. Ruíz, and A. Loaiza-Bonilla (Eds.), External Links: Link Cited by: §A.1, §1. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, et al. HotpotQA: a dataset for diverse, explainable multi-hop question answering. In EMNLP, Cited by: §4.1. Yao et al. (2022a) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §4.1. Yao et al. (2022b) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, J. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang DAPO: an open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. Ruíz, and A. Loaiza-Bonilla (Eds.), External Links: Link Cited by: §A.2, §1. Yue et al. (2025) Y. Yue, G. Zhang, B. Liu, G. Wan, K. Wang, D. Cheng, and Y. Qi MasRouter: learning to route llms for multi-agent systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), p. 15549–15572. External Links: Link, Document Cited by: §A.1, §1. Zhang et al. (2025) J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. Wu AFlow: automating agentic workflow generation. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: Link Cited by: §A.1, §1. Zhang et al. (2026) R. Zhang, X. Jiang, Z. Yang, Z. Zhang, J. Gao, Y. Xiao, H. Lai, X. Chu, J. Zhao, and Y. Wang StackPlanner: a centralized hierarchical multi-agent system with task-experience memory management. arXiv preprint arXiv:2601.05890. Cited by: §A.1, §1, §1, §4.1. Zhou et al. (2025) Z. Zhou, A. Qu, Z. Wu, S. Kim, A. Prakash, D. Rus, J. Zhao, B. K. H. Low, and P. P. Liang MEM1: learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841. Cited by: §4.1. Appendix Contents 1 Introduction 2 Preliminaries 2.1 Central-Agent Multi-Agent Harnesses 2.2 Policy Optimization in a Multi-Agent Harness 3 Harness-RL 3.1 Framework Overview 3.2 Conflict-Aware Policy Optimization 3.3 Interface-Level Black-Box RL 4 Experiments 4.1 Experimental Setup 4.2 Main Result Analysis 4.3 Component Analysis 5 Conclusion References A Related Work A.1 LLM-Based Multi-Agent Systems A.2 Reinforcement Learning for Agentic Systems A.3 Functional Specialization in Language Models B Training Configuration C Computational Resources and Software Environment D Detailed Case Study E Use of Large Language Models Appendix A Related Work A.1 LLM-Based Multi-Agent Systems LLM-based multi-agent systems have emerged as an important paradigm for solving complex tasks Guo et al. (2024); Chen et al. (2025b). By decomposing a complex problem into sub-tasks and assigning specialized roles to different agents, these systems can improve task planning and collaborative decision making. Representative systems such as AutoGen Wu et al. (2023), MetaGPT Hong et al. (2024), and ChatDev Qian et al. (2024) employ explicit role assignment and structured communication in software development and collaborative decision-making scenarios. Beyond role-based collaboration, decentralized and debate-based systems use dynamic inter-agent interaction to increase reasoning diversity and mitigate single-agent decision bias Du et al. (2024); Yang et al. (2025); Cui et al. (2026), although they may introduce redundant communication and complicate global-state consistency. For complex long-horizon tasks, centralized coordination is a widely adopted alternative: a central planner or coordinator performs global task planning, sub-task dispatch, and cross-agent state management. CoAct Hou et al. (2024) develops a global–local hierarchical architecture, MasRouter Yue et al. (2025) learns dynamic task routing, and MacNet, OWL, and AFlow improve collaboration structures and workflow optimization Qian et al. (2025a); Hu et al. (2025); Zhang et al. (2025). STACKPLANNER Zhang et al. (2026) further introduces RL-based active memory management, using a Task Memory Stack and a learnable Revise action to retrieve and reuse cross-task experience. Although these studies demonstrate the effectiveness of centralized coordination, they primarily focus on system architecture, communication, or workflow design, with limited attention to learning and optimizing the central agent itself. Our work focuses on this underexplored policy-optimization problem. A.2 Reinforcement Learning for Agentic Systems RL has shown substantial potential for improving the complex reasoning capabilities of LLMs OpenAI (2025). Representative work such as DeepSeek-R1 Guo et al. (2025) uses Group Relative Policy Optimization (GRPO) to train long-chain reasoning, achieving substantial advances in mathematical and code reasoning Shao et al. (2024), while DAPO Yu et al. (2025) further improves training stability. Motivated by these advances, agentic RL methods learn strategies for tool use and task execution. Search-R1 Jin et al. (2025) trains a model to decide when to invoke a search engine; RAGEN Wang et al. (2025) models multi-step agent interaction as an RL problem; and ToolRL Qian et al. (2025b) develops reward mechanisms for tool learning. AEPO Dong et al. (2025a) and ARPO Dong et al. (2025b) improve optimization and exploration in high-entropy agentic settings. However, these methods primarily target single-agent settings or homogeneous action spaces, implicitly treating tokens in a sequence as having similar semantic functions and optimization objectives. A central-agent output instead contains two fundamentally different decisions: high-level meta-action selection (Action Tokens) and concrete args generation (Args Tokens), whose gradient statistics and optimization dynamics can differ substantially. Applying a shared policy objective can therefore cause gradient conflicts and policy degradation, motivating our decoupled optimization framework. A.3 Functional Specialization in Language Models Researchers have increasingly investigated whether LLMs develop internal functional specialization during pretraining. Attention heads, feed-forward network layers, and individual neurons can exhibit specialized responses to particular tasks or types of knowledge Dai et al. (2021); Tang et al. (2024); Todd et al. (2023). Model Shapley Chu et al. () quantifies parameter-level importance from a cooperative-game-theoretic perspective, providing a principled basis for functional-partition analysis. At the level of parameter localization, ROME Meng et al. (2022) and MEMIT Meng et al. (2023) identify specific MLP components associated with factual knowledge, providing evidence of functionally differentiated parameter structures. Diff Pruning Guo et al. (2021) and LoRA Hu et al. (2021) further constrain updates to compact parameter subspaces, enabling efficient and targeted adaptation. Inspired by these findings, we use forward activation statistics to identify functional partitions associated with Action selection and Args generation, and route their policy gradients to the corresponding subspaces to reduce interference between the two decision types. Appendix B Training Configuration Table 4 records the settings explicitly available in the current experimental report. Hyperparameter Value Rollouts per query K 4 GPU configuration 4×4× A100-SXM4-80GB (1.5B); 8×8× A100-SXM4-80GB (3B) Optimizer Adam Learning rate 2×10−72× 10^-7 Global/per-device batch size 32 / dynamic (mean 8 for 1.5B; 4 for 3B) Maximum harness steps 12 Maximum input/output length 16,384 / 1,024 tokens Sampling temperature / top-p 0.8 / 1.0 Clipping coefficient ϵε 0.2 Table 4: Harness-RL training configuration for the central-capability runs. Appendix C Computational Resources and Software Environment Experiments were conducted on a server with two Intel Xeon Platinum 8488C processors, providing 96 physical cores and 192 hardware threads, and 503 GB of system memory. Harness-RL training used NVIDIA A100-SXM4 GPUs with 80 GB of memory per GPU: four GPUs for Qwen2.5-1.5B-Instruct and eight GPUs for Qwen2.5-3B-Instruct. The operating system was Ubuntu 22.04.5 LTS. The software environment used Python 3.11.11, Conda 23.5.2, PyTorch 2.6.0 with CUDA support, Hugging Face Transformers 4.51.3, and spaCy 3.8.4. Unless otherwise specified, all experiments used the same hardware and software environment. Appendix D Detailed Case Study We present a representative trajectory to illustrate how Harness-RL separates high-level dispatch decisions from their args and refines the latter using sub-agent feedback. As shown in Table 5, the central agent preserves the appropriate Dispatch action across two retrieval rounds while changing its args from broad entity identification to precise timeline verification. This behavior allows the agent to resolve an ambiguous temporal clue before returning the correct answer. Question Which Yuri served as USSR president for approximately two years after heading the KGB for 15 years? Ground Truth Yuri Andropov Harness-RL Action: Dispatch → Search Agent Args: Identify the Yuri who became a Soviet leader after serving as KGB chairman for approximately 15 years, and verify the relevant timeline. Observation: Yuri Vladimirovich Andropov chaired the KGB from 1967 to 1982 and subsequently became the Soviet leader; the exact duration of his later tenure remains uncertain. Reason: Andropov served as General Secretary for roughly 15 months, whereas the question says “approximately two years.” The dates and possible alternative candidates should therefore be verified before answering. Action: Dispatch → Search Agent Args: Verify Andropov’s exact tenure dates and determine whether any other person named Yuri satisfies both the KGB-service and presidency clues. Observation: The evidence confirms Andropov as the unique candidate; the phrase “approximately two years” is an approximate description of his tenure. Final Answer: Yuri Andropov Table 5: Case study of Harness-RL on an ambiguous multi-hop question. Blue and green boxes denote Actions and Args, respectively; gray boxes show sub-agent observations, violet shows central-agent reasoning, and yellow shows the final answer. Appendix E Use of Large Language Models Large language models were used only as auxiliary tools for language refinement and programming-related assistance. They were used to improve grammatical accuracy, readability, and presentation, and to provide general coding or debugging suggestions. All model-assisted outputs were reviewed and validated by the authors before inclusion. The authors were responsible for the research conception, methodological design, experimental execution, interpretation of results, and final conclusions.

Instant research discovery