Paper deep dive
CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning
Dongming Wang, Jie Xu, Yanyu Zhang, Wei Ren
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/23/2026, 1:55:23 AM
Summary
The paper introduces CHMAS, a Coupled Hierarchical Multi-Agent System framework for Multi-Agent Reinforcement Learning (MARL). It addresses the challenge of balancing global coordination and local execution by decomposing decision-making into a centralized strategic layer and distributed tactical layer. Key features include bidirectional information flow, where strategic guidance influences tactical execution and accumulated tactical rewards influence strategic objectives via a coupling coefficient. The framework employs an Asynchronous Hierarchical Policy Gradient (AHPG) algorithm with an asynchronous update protocol to handle non-stationarity, achieving provable convergence rates. Experimental validation in a multi-agent foraging domain demonstrates stable learning of spatially partitioned exploration strategies.
Entities (7)
Relation Signals (7)
CHMAS → uses → AHPG
confidence 95% · Second, we develop the Asynchronous Hierarchical Policy Gradient (AHPG) algorithm... Experimental validation... demonstrates successful learning
CHMAS → consistsof → Strategic Layer
confidence 92% · decomposes multi-agent decision-making into centralized strategic planning and distributed tactical execution
CHMAS → consistsof → Tactical Layer
confidence 92% · decomposes multi-agent decision-making into centralized strategic planning and distributed tactical execution
AHPG → achieves → O(log K/sqrt(K)) convergence
confidence 90% · Theoretical analysis proves that this asynchronous scheme achieves O(log K/sqrt(K)) convergence for the strategic layer
Strategic Layer → influences → Tactical Layer
confidence 90% · strategic guidance, generated every T timesteps, broadcasts to tactical agents
Tactical Layer → influences → Strategic Layer
confidence 90% · accumulated tactical rewards influence strategic objectives through a coupling coefficient lambda
CHMAS → validatedin → Multi-Agent Foraging
confidence 88% · Experimental validation in a multi-agent foraging domain demonstrates successful learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent reinforcement learning (MARL) systems face fundamental challenges in balancing global coordination with local execution across different temporal scales. This paper introduces the Coupled Hierarchical Multi-Agent System (CHMAS), a novel framework that decomposes multi-agent decision-making into centralized strategic planning and distributed tactical execution with bidirectional information flow. The strategic layer integrates all agents' states with an exclusive global environmental state to generate guidance actions every $T$ timesteps, while tactical agents execute distributed policies augmented by strategic guidance and local neighborhood observations. Unlike existing hierarchical approaches with unidirectional control, CHMAS establishes a feedback mechanism where accumulated tactical rewards influence strategic objectives through a coupling coefficient $\lambda$, ensuring strategic plans remain grounded in tactical feasibility. To address the non-stationarity inherent in hierarchical learning, we propose an asynchronous update protocol where strategic parameters update every $N_f$ tactical episodes, allowing tactical policies to converge to quasi-stationary points between strategic changes. We present both a general bi-level formulation capturing full system dynamics and a tractable additive approximation enabling rigorous analysis. Theoretical analysis proves that this asynchronous scheme achieves $\mathcal{O}(\log K/\sqrt{K})$ convergence for the strategic layer after $K$ strategic updates under standard assumptions. Experimental validation in a multi-agent foraging domain demonstrates successful learning of spatially partitioned exploration strategies, with both layers converging stably despite hierarchical coupling.
Tags
Links
- Source: https://arxiv.org/abs/2607.19555v1
- Canonical: https://arxiv.org/abs/2607.19555v1
Trouble viewing inline? Open PDF directly →
Full Text
36,650 characters extracted from source content.
Expand or collapse full text
CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning Dongming Wang, Jie Xu, Yanyu Zhang, and Wei Ren This work was supported by the National Science Foundation under Grant ECCS-2129949.The authors are with the Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA. wdong025, jxu150, yzhan831, wei.ren@ucr.edu. Abstract Multi-agent reinforcement learning (MARL) systems face fundamental challenges in balancing global coordination with local execution across different temporal scales. This paper introduces the Coupled Hierarchical Multi-Agent System (CHMAS), a novel framework that decomposes multi-agent decision-making into centralized strategic planning and distributed tactical execution with bidirectional information flow. The strategic layer integrates all agents’ states with an exclusive global environmental state to generate guidance actions every T timesteps, while tactical agents execute distributed policies augmented by strategic guidance and local neighborhood observations. Unlike existing hierarchical approaches with unidirectional control, CHMAS establishes a feedback mechanism where accumulated tactical rewards influence strategic objectives through a coupling coefficient λ, ensuring strategic plans remain grounded in tactical feasibility. To address the non-stationarity inherent in hierarchical learning, we propose an asynchronous update protocol where strategic parameters update every NfN_f tactical episodes, allowing tactical policies to converge to quasi-stationary points between strategic changes. We present both a general bi-level formulation capturing full system dynamics and a tractable additive approximation enabling rigorous analysis. Theoretical analysis proves that this asynchronous scheme achieves (logK/K)O( K/ K) convergence for the strategic layer after K strategic updates under standard assumptions. Experimental validation in a multi-agent foraging domain demonstrates successful learning of spatially partitioned exploration strategies, with both layers converging stably despite hierarchical coupling. I INTRODUCTION Multi-agent reinforcement learning (MARL) has emerged as a fundamental paradigm for autonomous coordination in complex environments, with applications spanning robotics [12], traffic management [22], and resource allocation [15]. Despite significant progress, real-world deployment faces a critical challenge: coordinating distributed agents across multiple temporal and spatial scales while maintaining computational tractability. The curse of dimensionality renders purely centralized approaches intractable. For N agents with state spaces iS_i and action spaces iA_i, the joint space grows as O(∏i=1N|i|⋅|i|)O( _i=1^N|S_i|·|A_i|) [23]. While distributed methods address scalability through local decision-making [24], they sacrifice coordination quality for long-horizon objectives. Traditional approaches like QMIX [16] and MADDPG [8] leverage centralized training with decentralized execution, but assume all agents access relevant global information during training, which often fails in practice due to partial observability and communication constraints [5]. Furthermore, multi-agent systems inherently operate across multiple temporal scales. Strategic decisions (formation control, task allocation) require global information including environmental states inaccessible to individual agents, operating on extended timescales. Tactical responses (collision avoidance, local adaptation) demand rapid reactions based on immediate observations. While hierarchical reinforcement learning offers a natural framework for multi-scale decision-making [19, 2], existing hierarchical MARL approaches [14, 21] employ unidirectional control where high-level policies dictate low-level actions, missing opportunities for tactical feedback to inform strategic planning. The interaction between hierarchical levels creates complex learning dynamics. Strategic changes fundamentally alter the environment perceived by tactical agents, violating stationarity assumptions [5]. Conversely, evolving tactical policies change execution characteristics that strategic planners rely upon. Previous hierarchical MARL work [7, 9] addresses this through reward engineering or option learning but requires domain expertise and often fails to generalize. This paper introduces CHMAS, which addresses these limitations through bidirectional coupling between strategic and tactical layers at different temporal scales. The strategic level maintains centralized access to both agents’ states and an exclusive global environmental state envS^env capturing system-wide properties essential for coordination. Strategic guidance, generated every T timesteps, broadcasts to tactical agents who execute distributed policies based on augmented local observations. Tactical rewards accumulate and influence strategic objectives through coupling coefficient λ, creating feedback that grounds strategic planning in tactical realities. An asynchronous protocol with strategic updates every NfN_f tactical episodes manages non-stationary dynamics, allowing tactical convergence between strategic changes. Our contributions are threefold. First, we formalize the hierarchical MARL problem with bidirectional coupling and information asymmetry, providing both a general bi-level formulation and tractable additive approximation. Second, we develop the Asynchronous Hierarchical Policy Gradient (AHPG) algorithm managing non-stationarity through temporal separation while maintaining (logK/K)O( K/ K) convergence guarantees for the strategic layer. Third, we validate the framework empirically, demonstrating stable bidirectional learning in a cooperative multi-agent foraging task. I PRELIMINARIES We consider a cooperative multi-agent system with N agents operating in a shared environment. The system employs a hierarchical architecture with a strategic layer that accesses global information at low temporal frequency and a tactical layer that operates distributedly at high frequency using only local information. I-A Multi-Agent Communication Graph The tactical agents interact through a time-invariant communication graph =(,ℰ)G=(V,E) where =1,2,…,NV=\1,2,…,N\ represents the agent set and ℰ⊆×E ×V denotes the edge set [13, 10]. For each agent i, the neighborhood i=j∈:(i,j)∈ℰN_i=\j :(i,j) \ is the set of agents that can directly communicate with agent i. This graph constrains information flow at the tactical level, ensuring distributed execution remains scalable. I-B Strategic MDP Definition 1 (Strategic MDP) The strategic layer evolves as ℳstr=(str,str,Pstr,Rstr,γstr)M^str=(S^str,A^str,P^str,R^str, _str), where: (i) the state str=env×∏i=1NiS^str=S^env× _i=1^NS_i combines the exclusive global environmental state envS^env (resource distributions, global objectives) with all agent states, so that at epoch k the strategic state is skstr=(skenv,s1,k,…,sN,k)s^str_k=(s^env_k,s_1,k,…,s_N,k); (i) actions str=∏i=1NistrA^str= _i=1^NA^str_i provide personalized guidance ai,kstra^str_i,k to each agent, persisting for the interval [kT,(k+1)T)[kT,(k+1)T); and (i) the reward integrates global objectives with tactical feedback: Rstr(skstr,kstr)=Rglobal(skenv,kstr)+λ∑t=kT(k+1)T−1∑i=1Nri,ttac,R^str(s^str_k,a^str_k)=R^global(s^env_k,a^str_k)+λ _t=kT^(k+1)T-1 _i=1^Nr^tac_i,t, (1) where λ∈[0,1]λ∈[0,1] controls the influence of tactical execution quality on strategic evaluation. The strategic policy πstr:str→(str)π^str:S^str (A^str) maps complete state observations to guidance distributions, encoding high-level coordination logic that individual agents cannot compute from local information alone. I-C Tactical Multi-Agent MDP Definition 2 (Tactical MAMDP) Given fixed strategic guidance kstra^str_k, the tactical layer evolves as ℳtac=(i,itac,itac,Ptac,Ritac,γtac)M^tac=(\S_i\,\O^tac_i\,\A^tac_i\,P^tac,\R^tac_i\, _tac), where: (i) each agent i observes oi,ttac=ϕi(si,t,sj,tj∈i,ai,kstr)∈itac,o^tac_i,t= _i(s_i,t,\s_j,t\_j _i,a^str_i,k) ^tac_i, (2) combining local state, neighbor states, and strategic guidance (k=⌊t/T⌋k= t/T ); (i) the joint transition factorizes over the communication graph: Ptac(t+1|t,ttac)=∏i=1NPi(si,t+1|si,t,sj,t,aj,ttacj∈i∪i)P^tac(s_t+1|s_t,a^tac_t)= _i=1^NP_i(s_i,t+1|s_i,t,\s_j,t,a^tac_j,t\_j _i∪\i\), maintaining tractability as N grows [23]; and (i) the per-agent reward Ritac(si,t,ai,ttac,sj,t,aj,ttacj∈i,ai,kstr)R^tac_i(s_i,t,a^tac_i,t,\s_j,t,a^tac_j,t\_j _i,a^str_i,k) (3) enables local optimization and neighbor coordination while incorporating strategic objectives. I-D Bidirectional Coupling Unlike traditional top-down hierarchies [14], CHMAS establishes feedback in both directions. Downward coupling (strategic → tactical): guidance ai,kstra^str_i,k shapes tactical behavior through observation augmentation (2), reward modulation (3), and temporal persistence over T timesteps. Upward coupling (tactical → strategic): accumulated rewards Rktac,cum=∑t=kT(k+1)T−1∑i=1Nri,ttacR^tac,cum_k= _t=kT^(k+1)T-1 _i=1^Nr^tac_i,t influence strategic rewards via λ in (1), so that strategic decisions account for tactical feasibility. Setting λ=0λ=0 recovers a purely top-down hierarchy. The framework maintains asymmetric information access: the strategic layer observes complete state skstrs^str_k while tactical agents access only iN_i, with strategic guidance serving as an information bottleneck [20]. I PROBLEM FORMULATION I-A General Coupled Formulation The bidirectional coupling creates a bi-level optimization structure [4, 18]. The strategic objective: Jstr(θstr|θitac)=[∑k=0∞γstrkRstr(skstr,kstr)]J^str(θ^str|\θ^tac_i\)=E\! [ _k=0^∞γ^k_strR^str(s^str_k,a^str_k) ] (4) depends on tactical policies through (1). Each tactical objective: Jitac(θitac|θstr)=[∑t=0∞γtactri,ttac]J^tac_i(θ^tac_i|θ^str)=E\! [ _t=0^∞γ^t_tacr^tac_i,t ] (5) depends on strategic parameters through (2) and (3). Definition 3 (Coupled Hierarchical Problem) The coupled optimization problem is the bi-level program: maxθstr _θ^str Jstr(θstr|θitac∗(θstr)) J^str(θ^str|\θ^tac*_i(θ^str)\) s.t. θitac∗(θstr)=argmaxθitacJitac(θitac|θstr),∀i∈, θ^tac*_i(θ^str)= _θ^tac_iJ^tac_i(θ^tac_i|θ^str), ∀ i , where the strategic optimization accounts for tactical best responses. Remark 1 We operate in the cooperative MARL setting where agents share a common team objective. Accordingly, the per-agent best responses θitac∗θ^tac*_i are aligned with the joint team optimum, and no equilibrium-seeking across agents is required at the tactical level [23]. This Stackelberg structure is computationally intractable in general [6], motivating the approximation below. I-B Additive Approximation for Tractable Optimization Definition 4 (Additive Hierarchical Problem) Under the additive approximation, the global objective is: Jglobal(θstr,θitac)=Jstr(θstr)+∑i=1NJitac(θitac),J^global(θ^str,\θ^tac_i\)=J^str(θ^str)+ _i=1^NJ^tac_i(θ^tac_i), (6) where each term is evaluated with fixed parameters of the other layer. Remark 2 Equation (6) is an approximation of the true coupled objective: the cross-layer dependencies in (4)–(5) are not eliminated but handled implicitly through the asynchronous update schedule. The approximation error is controlled by the tactical optimality gap δk _k which will be defined in Assumption 2 at each epoch, which vanishes as learning progresses. The separable gradient structure, ∇θJglobal=[∇θstrJstr,∇θtacJtac]T, _θJ^global= [ _θ^strJ^str,\; _θ^tacJ^tac ]^T, (7) allows updates to be treated as coordinate steps in the joint parameter space, enabling rigorous convergence analysis via two-timescale stochastic approximation [3, 11]. IV ASYNCHRONOUS LEARNING ALGORITHM IV-A Algorithm Design Principles AHPG leverages three principles for stable hierarchical learning. Temporal separation. Strategic decisions occur every T timesteps and persist throughout the interval. Strategic parameters follow a decaying step size schedule ηkstr=α/kη^str_k=α/ k and update only every NfN_f tactical episodes, operating at a slower effective timescale than tactical learning. This allows tactical policies to converge to near-optimal responses between strategic changes, transforming the non-stationary hierarchical problem into alternating stationary subproblems. Structured information flow. Downward flow provides coordination signals through strategic guidance without requiring centralized control at each timestep. Upward flow aggregates tactical performance Rktac=∑t,iri,ttacR^tac_k= _t,ir^tac_i,t into strategic rewards, keeping high-level plans grounded in execution realities. The strategic layer maintains exclusive access to skenvs^env_k, preserving the information asymmetry that enables scalable distributed execution. Distributed tactical learning. Each tactical agent maintains its own policy πitacπ^tac_i and experience buffer itacD^tac_i, updating parameters using only local gradients without inter-agent communication during learning, so that tactical computation scales linearly with N. IV-B Algorithm Specification Algorithm 1 presents the complete AHPG procedure. Strategic guidance is generated at the start of each new strategic epoch (every NfN_f episodes), and strategic parameters are updated at the end of each epoch using the accumulated tactical performance. Tactical parameters update every episode. Algorithm 1 Asynchronous Hierarchical Policy Gradient (AHPG) 0: Initial parameters θ1strθ^str_1, θi,0tac\θ^tac_i,0\ 0: Hyperparameters T, NfN_f, α, ηtacη^tac, λ, c 1: Initialize episode counter e←0e← 0, strategic epoch k←1k← 1 2: for episode e=1,2,…e=1,2,… do 3: // Strategic Planning Phase 4: if emodNf=1e N_f=1 then 5: Observe full state skstr=(skenv,k)s^str_k=(s^env_k,s_k) 6: Generate kstr∼πstr(skstr;θkstr)a^str_k π^str(s^str_k;\,θ^str_k) 7: Broadcast ai,kstra^str_i,k to each agent i 8: end if 9: // Tactical Execution Phase 10: for t=0t=0 to T−1T-1 do 11: for agent i∈i in parallel do 12: oi,ttac=ϕi(si,t,sj,tj∈i,ai,kstr)o^tac_i,t= _i(s_i,t,\s_j,t\_j _i,a^str_i,k) 13: ai,ttac∼πitac(oi,ttac;θi,etac)a^tac_i,t π^tac_i(o^tac_i,t;\,θ^tac_i,e) 14: Store (oi,ttac,ai,ttac,ri,ttac)(o^tac_i,t,a^tac_i,t,r^tac_i,t) in itacD^tac_i 15: end for 16: end for 17: // Tactical Policy Update 18: for agent i∈i in parallel do 19: gitac=∇θitacJitacg^tac_i= _θ^tac_iJ^tac_i using itacD^tac_i 20: θi,e+1tac=θi,etac+ηtacgitacθ^tac_i,e+1=θ^tac_i,e+η^tacg^tac_i 21: end for 22: // Strategic Update 23: if emodNf=0e N_f=0 then 24: Rktac=∑t=kT(k+1)T−1∑i=1Nri,ttacR^tac_k= _t=kT^(k+1)T-1 _i=1^Nr^tac_i,t 25: rkstr=Rglobal(skenv,kstr)+λRktacr^str_k=R^global(s^env_k,a^str_k)+λ R^tac_k 26: gstr=∇θstrJstrg^str= _θ^strJ^str using strD^str 27: θk+1str=θkstr+(α/k)gstrθ^str_k+1=θ^str_k+(α/ k)\,g^str 28: k←k+1k← k+1 29: end if 30: end for During the NfN_f episodes between strategic updates, tactical agents operate in a stationary environment with fixed guidance, converging toward near-optimal policies. This ensures strategic updates observe stable tactical performance that accurately estimates the tactical contribution to strategic rewards. For learning components, the strategic layer employs centralized policy gradient algorithms (e.g., PPO [17]) leveraging complete state access, while tactical agents use distributed policy gradient or actor-critic variants operating on local observations. The framework naturally extends to continuous control through continuous guidance vectors and to partially observable settings through recurrent policies. V CONVERGENCE ANALYSIS We establish convergence for AHPG under the additive approximation Jglobal(θ)=Jstr(θstr)+∑i=1NJitac(θitac)J^global(θ)=J^str(θ^str)+ _i=1^NJ^tac_i(θ^tac_i). Assumption 1 1. Smoothness: JstrJ^str and each JitacJ^tac_i are L-smooth with L-Lipschitz gradients. 2. Boundedness: Jstr≤Jstr∗J^str≤ J^str* and Jitac≤Jitac∗J^tac_i≤ J^tac*_i for all i. 3. Bounded variance: Stochastic gradient estimates satisfy [‖gkstr−∇Jkstr‖2]≤σstr2E[\|g^str_k-∇ J^str_k\|^2]≤σ^2_str and [‖gi,etac−∇Jitac‖2]≤σtac2E[\|g^tac_i,e-∇ J^tac_i\|^2]≤σ^2_tac. 4. Biased strategic gradient: [gkstr∣θkstr]=∇Jstr(θkstr)+bkE[g^str_k θ^str_k]=∇ J^str(θ^str_k)+b_k, where bkb_k captures the error from evaluating strategic gradients with non-converged tactical parameters. Assumption 2 1. PL condition: Each JitacJ^tac_i satisfies the μ-PL inequality: ‖∇Jitac(θ)‖2≥2μ(Jitac∗−Jitac(θ))\|∇ J^tac_i(θ)\|^2≥ 2μ(J^tac*_i-J^tac_i(θ)). 2. Warm-start: At the beginning of each strategic epoch k, tactical agents are initialized from the converged state of the previous epoch. The initial optimality gap satisfies [Jitac∗−Jitac(θi,kNftac)]≤δkE[J^tac*_i-J^tac_i(θ^tac_i,kN_f)]≤ _k, where δk=(ηkstr)=(1/k) _k=O(η^str_k)=O(1/ k), reflecting the magnitude of the strategic parameter change between successive epochs. 3. Round-robin updates: Within each tactical phase, agents update sequentially with others’ parameters held fixed, so that the PL condition applies to each per-agent landscape independently. Remark 3 The PL condition in Assumption 2 holds for log-linear (softmax) policies in tabular settings [1] and locally for over-parameterized neural networks near stationary points. The round-robin assumption is a standard device to decouple per-agent landscapes; it can be relaxed to simultaneous updates at the cost of a more involved coupling analysis. Assumption 3 (Coupling structure) The strategic gradient is LbL_b-Lipschitz in the tactical parameters: ‖∇Jstr(θstr,tac)−∇Jstr(θstr,tac∗)‖≤Lb‖tac−tac∗‖.\|∇ J^str(θ^str, θ^tac)-∇ J^str(θ^str, θ^tac*)\|≤ L_b\| θ^tac- θ^tac*\|. Theorem 1 (AHPG Convergence) Under Assumptions 1–3, with ηkstr=α/kη^str_k=α/ k where α≤1/(8L)α≤ 1/(8L), ηtac=β/Kη^tac=β/ K where β<1/(2L)β<1/(2L), and Nf=cKN_f=c K for a constant c>0c>0, after K strategic updates AHPG achieves: mink∈1,…,K[‖∇Jstr(θkstr)‖2]=(logK), _k∈\1,…,K\E[\|∇ J^str(θ^str_k)\|^2]=O\! ( K K ), 1KNf∑k=1K∑e=kNf(k+1)Nf−1∑i=1N[‖∇Jitac(θi,etac)‖2]=(1K). 1KN_f _k=1^K _e=kN_f^(k+1)N_f-1 _i=1^NE[\|∇ J^tac_i(θ^tac_i,e)\|^2]=O\! ( 1 K ). Proof: Strategic layer. By L-smoothness of JstrJ^str and the update θk+1str=θkstr+ηkstrgkstrθ^str_k+1=θ^str_k+η^str_kg^str_k: Jk+1str≥Jkstr+ηkstr⟨∇Jkstr,gkstr⟩−L2(ηkstr)2‖gkstr‖2.J^str_k+1≥ J^str_k+η^str_k ∇ J^str_k,\,g^str_k - L2(η^str_k)^2\|g^str_k\|^2. Taking expectation with [gkstr]=∇Jkstr+bkE[g^str_k]=∇ J^str_k+b_k, we bound the two terms. For the inner product, applying Young’s inequality ⟨a,b⟩≥−14‖a‖2−‖b‖2 a,b ≥- 14\|a\|^2-\|b\|^2: [⟨∇Jkstr,gkstr⟩]≥34‖∇Jkstr‖2−‖bk‖2.E[ ∇ J^str_k,g^str_k ]≥ 34\|∇ J^str_k\|^2-\|b_k\|^2. For the squared norm, using [‖gkstr‖2]≤‖∇Jkstr+bk‖2+σstr2≤2‖∇Jkstr‖2+2‖bk‖2+σstr2E[\|g^str_k\|^2]≤\|∇ J^str_k+b_k\|^2+σ^2_str≤ 2\|∇ J^str_k\|^2+2\|b_k\|^2+σ^2_str: [Jk+1str] [J^str_k+1] ≥Jkstr+(34ηkstr−L(ηkstr)2)‖∇Jkstr‖2 ≥ J^str_k+\! ( 34η^str_k-L(η^str_k)^2 )\|∇ J^str_k\|^2 −(ηkstr+L(ηkstr)2)‖bk‖2−L2(ηkstr)2σstr2. -\! (η^str_k+L(η^str_k)^2 )\|b_k\|^2- L2(η^str_k)^2σ^2_str. Since ηkstr≤1/(8L)η^str_k≤ 1/(8L), we have L(ηkstr)2≤ηkstr/8L(η^str_k)^2≤η^str_k/8, so 34ηkstr−L(ηkstr)2≥58ηkstr≥18ηkstr 34η^str_k-L(η^str_k)^2≥ 58η^str_k≥ 18η^str_k and ηkstr+L(ηkstr)2≤98ηkstr≤2ηkstrη^str_k+L(η^str_k)^2≤ 98η^str_k≤ 2η^str_k. Rearranging and telescoping over k=1,…,Kk=1,…,K: 18∑k=1Kηkstr[‖∇Jkstr‖2] 18 _k=1^Kη^str_kE[\|∇ J^str_k\|^2] ≤ΔJstr+2∑k=1Kηkstr‖bk‖2 ≤ J^str+2 _k=1^Kη^str_k\|b_k\|^2 +Lσstr22∑k=1K(ηkstr)2, + Lσ^2_str2 _k=1^K(η^str_k)^2, (8) where ΔJstr=Jstr∗−J1str J^str=J^str*-J^str_1. Bias bound. By Assumption 3, ‖bk‖2≤Lb2‖(k+1)Nftac−tac∗‖2\|b_k\|^2≤ L_b^2\| θ^tac_(k+1)N_f- θ^tac*\|^2. Running Nf=cKN_f=c K SGD steps under the PL condition (Assumption 2) with step size β/Kβ/ K and warm-start gap δk _k yields contraction factor ρ=(1−2μβ/K)cK→e−2μβc<1ρ=(1-2μβ/ K)^c K→ e^-2μβ c<1, so the residual tactical deviation satisfies ‖bk‖2=(δk+1/K)=(1/k)\|b_k\|^2=O( _k+1/ K)=O(1/ k). Completing the strategic bound. With ηkstr=α/kη^str_k=α/ k: ∑k=1Kηkstr‖bk‖2 _k=1^Kη^str_k\|b_k\|^2 =(α∑k=1K1k⋅1k)=(logK), =O\! (α _k=1^K 1 k· 1 k )=O( K), ∑k=1K(ηkstr)2 _k=1^K(η^str_k)^2 =α2∑k=1K1k=(logK). =α^2 _k=1^K 1k=O( K). Dividing (V) by ∑k=1Kηkstr=(αK) _k=1^Kη^str_k=O(α K) gives mink[‖∇Jkstr‖2]=(logK/K) _kE[\|∇ J^str_k\|^2]=O( K/ K). Tactical layer. Within epoch k, θkstrθ^str_k is fixed, so JitacJ^tac_i is stationary. Standard SGD under the μ-PL condition over NfN_f steps gives per-epoch average gradient norm (δk+1/K)O( _k+1/ K). Averaging over K epochs and using K−1∑k=1Kδk=K−1∑k=1K(k−1/2)=(K−1/2)K^-1 _k=1^K _k=K^-1 _k=1^KO(k^-1/2)=O(K^-1/2), the overall tactical average is (1/K)O(1/ K), completing the proof. ∎ Remark 4 Theorem 1 is established for policy gradient implementations. The DQN-based experiments in Section VI serve as an empirical proof of concept; extending the convergence analysis to Q-learning variants is left for future work. VI EXPERIMENTAL VALIDATION VI-A Experiment Setting We evaluate CHMAS on a 25×25 GridWorld environment with N=4N=4 agents performing cooperative foraging, instantiating the key features of hierarchical multi-agent coordination: information asymmetry, temporal abstraction, and bidirectional coupling. Figure 1: The 25×25 GridWorld environment. Four agents (A0-blue, A1-orange, A2-purple, A3-green) operate within their assigned 9×9 regions (dashed boxes) determined by strategic guidance. Colored areas indicate visited cells tracked in the global state senvs^env, while agents pursue local resources (red apple). The global environmental state senv∈0,1625s^env∈\0,1\^625 tracks cell visitation, representing system-wide coverage that individual agents cannot observe. The strategic layer accesses the complete state sstr=(senv,s1,s2,s3,s4)s^str=(s^env,s_1,s_2,s_3,s_4), while each tactical agent i observes only local information oitac∈ℝ20o^tac_i ^20 comprising its position, nearby resources, and strategic guidance. Strategic actions assign each agent a 9×9 region center aistr=(gix,giy)∈[4,20]2a^str_i=(g^x_i,g^y_i)∈[4,20]^2 at the start of each epoch (T=5T=5 timesteps). Tactical agents select discrete actions aitac∈0,1,2,3,4,5a^tac_i∈\0,1,2,3,4,5\ (movement, collection, wait). The reward structure implements bidirectional coupling: tactical agents receive ri,ttac=ri,tcollect−0.01−0.01⋅[outside region]r^tac_i,t=r^collect_i,t-0.01-0.01·1[outside region], while the strategic layer exclusively receives global coverage reward Rkglobal=0.01×|newly visited cells|R^global_k=0.01×|newly visited cells| computed from senvs^env. Strategic rewards integrate both: Rkstr=Rkglobal+λ∑t,iri,ttacR^str_k=R^global_k+λ _t,ir^tac_i,t with λ=0.5λ=0.5. The implementation is publicly available at https://github.com/EricDmWang/Hierarchy_RL.git. VI-B Results and Analysis We implement the tactical layer using Multi-Agent DQN (MA-DQN) with parameter sharing, dueling architecture, double Q-learning, and prioritized experience replay. The strategic layer employs a centralized DQN on the global state. Training runs for 400 episodes with 5 independent seeds. (a) Tactical Q-values (b) Tactical returns (c) Tactical critic loss (d) Tactical policy loss Figure 2: Tactical layer learning dynamics. Figure 2 shows the tactical layer’s learning progression. Q-values (a) grow rapidly before stabilizing near episode 300. Returns (b) improve from approximately −80-80 to −10-10, showing agents learn efficient resource collection while respecting region constraints. Critic loss (c) and policy loss (d) peak early during exploration and then decrease as policies stabilize. (a) Strategic Q-values (b) Strategic returns (c) Strategic critic loss (d) Strategic actor loss Figure 3: Strategic layer learning dynamics with Nf=5N_f=5. Figure 3 shows the strategic layer’s asynchronous learning with updates every Nf=5N_f=5 episodes. Strategic Q-values (a) improve from −8-8 to near 0, and strategic returns (b) increase from −10-10 to approximately 1515, demonstrating successful optimization of global coverage objectives. The delayed onset of learning (first 10–20 episodes) reflects the asynchronous protocol allowing tactical policies to stabilize before strategic updates commence. Low variance across runs confirms algorithmic stability. VI-C Qualitative Analysis Figure 4 shows the execution trajectory of a trained AHPG model over a complete episode. The strategic layer learns to assign non-overlapping regions that collectively tile the grid, initially placing agents in corners and then adapting assignments to unexplored areas. Agents balance region compliance with opportunistic resource collection. By frame (e), full resource collection with maintained spatial coordination validates the effectiveness of bidirectional hierarchical control. (a) t=0t=0 (b) t=6t=6 (c) t=9t=9 (d) t=16t=16 (e) t=19t=19 Figure 4: Evolution of hierarchical coordination during execution. Agents operate within their assigned 9×9 regions (dashed boxes) while collectively achieving grid coverage. Red apples represent resources collected opportunistically. VII CONCLUSION This paper introduced CHMAS, a hierarchical MARL framework based on bidirectional coupling between strategic and tactical layers. By enabling strategic planning with exclusive global state access while maintaining distributed tactical execution, CHMAS provides a new architectural paradigm for coordinating agents across multiple temporal scales. The asynchronous update protocol, with a decaying strategic step size schedule, manages the non-stationarity inherent in hierarchical learning and admits (logK/K)O( K/ K) convergence guarantees for the strategic layer. The bidirectional coupling mechanism offers a principled approach to constrained reinforcement learning: strategic guidance encodes global constraints while tactical feedback ensures feasibility, with λ providing tunable constraint satisfaction. References [1] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan (2020) Optimality and approximation with policy gradient methods in markov decision processes. In Conference on learning theory, p. 64–66. Cited by: Remark 3. [2] P. Bacon, J. Harb, and D. Precup (2017) The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31. Cited by: §I. [3] V. S. Borkar (1997) Stochastic approximation with two time scales. Systems & Control Letters 29 (5), p. 291–294. Cited by: §I-B. [4] B. Colson, P. Marcotte, and G. Savard (2007) An overview of bilevel optimization. Annals of Operations Research 153 (1), p. 235–256. Cited by: §I-A. [5] P. Hernandez-Leal, B. Kartal, and M. E. Taylor (2019) A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems 33 (6), p. 750–797. Cited by: §I, §I. [6] M. Hong, H. Wai, Z. Wang, and Z. Yang (2023) A two-timescale stochastic algorithm framework for bilevel optimization: complexity analysis and application to actor-critic. SIAM Journal on Optimization 33 (1), p. 147–180. Cited by: §I-A. [7] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum (2016) Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: §I. [8] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §I. [9] O. Nachum, S. S. Gu, H. Lee, and S. Levine (2018) Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems 31. Cited by: §I. [10] A. Nedic, A. Ozdaglar, and P. A. Parrilo (2010) Constrained consensus and optimization in multi-agent networks. IEEE Transactions on Automatic Control 55 (4), p. 922–938. Cited by: §I-A. [11] Y. Nesterov (2012) Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization 22 (2), p. 341–362. Cited by: §I-B. [12] T. T. Nguyen, N. D. Nguyen, and S. Nahavandi (2020) Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications. IEEE transactions on cybernetics 50 (9), p. 3826–3839. Cited by: §I. [13] R. Olfati-Saber, J. A. Fax, and R. M. Murray (2007) Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE 95 (1), p. 215–233. Cited by: §I-A. [14] S. Pateria, B. Subagdja, A. Tan, and C. Quek (2021) Hierarchical reinforcement learning: a comprehensive survey. ACM Computing Surveys 54 (5), p. 1–35. Cited by: §I, §I-D. [15] G. Qu, Y. Lin, A. Wierman, and N. Li (2020) Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing Systems 33, p. 2074–2086. Cited by: §I. [16] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21 (178), p. 1–51. Cited by: §I. [17] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §IV-B. [18] A. Sinha, P. Malo, and K. Deb (2018) A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation 22 (2), p. 276–295. Cited by: §I-A. [19] R. S. Sutton, D. Precup, and S. Singh (1999) Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112 (1-2), p. 181–211. Cited by: §I. [20] N. Tishby, F. C. Pereira, and W. Bialek (2000) The information bottleneck method. arXiv preprint physics/0004057. Cited by: §I-D. [21] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu (2017) Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, p. 3540–3549. Cited by: §I. [22] M. A. Wiering et al. (2000) Multi-agent reinforcement learning for traffic light control. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML’2000), p. 1151–1158. Cited by: §I. [23] K. Zhang, Z. Yang, and T. Başar (2021) Multi-agent reinforcement learning: a selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, p. 321–384. Cited by: §I, Definition 2, Remark 1. [24] K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Basar (2018) Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning, p. 5872–5881. Cited by: §I.