Paper deep dive
Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections
Gil Lifshits, Igal Bilik, Gilad Katz
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/25/2026, 1:13:18 AM
Summary
The paper introduces the Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning architecture for coordinating autonomous vehicles at unsignalized intersections. A centralized Master agent generates a continuous 'proto-plan' embedding that encodes global coordination strategies, which decentralized Worker agents use alongside local observations to execute vehicle-specific control. This approach decouples strategic intent from tactical execution, reducing combinatorial action space complexity and enabling robust zero-shot generalization from smaller to larger agent populations.
Entities (7)
Relation Signals (7)
MAPS → uses → Master Agent
confidence 95% · MAPS is a hierarchical architecture in which a centralized Master agent generates...
MAPS → uses → Worker Agent
confidence 95% · Decentralized Worker agents integrate this embedding with local observations...
Worker Agent → consumes → Proto-plan
confidence 92% · Decentralized Worker agents integrate this embedding with local observations
Master Agent → generates → Proto-plan
confidence 92% · a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan
MAPS → evaluatedin → HighwayEnv
confidence 90% · we test MAPS across 72 intersection configurations in HighwayEnv
MAPS → trainedwith → PPO
confidence 88% · Both Master and Worker agents are trained using Proximal Policy Optimization (PPO)
MAPS → addresses → Multi-Agent Reinforcement Learning
confidence 85% · Coordinating autonomous vehicles... remains a critical challenge for multi-agent reinforcement learning (MARL) systems
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination.
Tags
Links
- Source: https://arxiv.org/abs/2607.21488v1
- Canonical: https://arxiv.org/abs/2607.21488v1
Trouble viewing inline? Open PDF directly →
Full Text
49,031 characters extracted from source content.
Expand or collapse full text
Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections Gil Lifshits Ben-Gurion University of the Negev gillif@post.bgu.ac.il Igal Bilik Ben-Gurion University of the Negev bilik@bgu.ac.il Gilad Katz Ben-Gurion University of the Negev giladkz@bgu.ac.il Abstract Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination. 1 Introduction Coordinating multiple autonomous vehicles (AVs) at unsignalized intersections is a central challenge for intelligent transportation: vehicles must jointly navigate complex, dynamic scenarios while guaranteeing collision-free operation and maintaining throughput [Wang et al.(2024)]. The difficulty of this multi-agent coordination problem has driven extensive research in multi-agent deep reinforcement learning (MADRL) [Chen et al.(2024)], spanning value-decomposition methods [Huang et al.(2023)], graph-based representations [Cai et al.(2022)], policy optimization [Peng et al.(2023), Xu et al.(2022)], and hierarchical frameworks [Al-Sharman et al.(2022), Liu et al.(2025), Zhao et al.(2024)]. Despite notable progress, existing approaches share several limitations. First, many exhibit limited generalization, as they are trained and evaluated on simplified layouts that do not reflect the real-world diversity of intersections. Second, state-of-the-art methods often depend on privileged information, such as future trajectories, game-theoretic priors, rule-based safety layers, or expert demonstrations, that may be unavailable in practical deployments. Third, most methods employ fixed, discrete action spaces consisting of small sets of predefined maneuvers, which scale combinatorially with the number of agents and require redesign when vehicle capabilities change. We address these limitations with master-agent proto-plan system (MAPS), a hierarchical DRL architecture that replaces explicit, per-vehicle action assignments with proto-plans: learned, continuous embedding vectors that encode high-level coordination strategies. A centralized Master agent observes the global traffic state and produces a single proto-plan; decentralized Worker agents each combine this proto-plan with their own local observations to select vehicle-specific actions. This decomposition decouples coordination from control, keeps communication overhead constant at O(d)O(d) regardless of fleet size, and allows each module to be updated independently. Experiments across 72 intersection configurations show that MAPS achieves zero collisions during evaluation while reducing average travel time to 7.8 steps, which is a 38% improvement over the best baseline. The architecture further demonstrates robust zero-shot transfer: trained with only three active agents, it attains a 94% success rate when deployed to five-agent scenarios without fine-tuning. Our contributions are as follows: • We introduce MAPS, a hierarchical DRL architecture that replaces rigid coordination commands with continuous proto-plan embeddings, keeping action and communication complexity at O(d)O(d) as fleet size grows. • We demonstrate zero-shot generalization through incremental training: a model trained on three agents transfers directly to five-agent deployment, confirming that the learned proto-plans capture transferable coordination strategies. • We show that effective multi-vehicle coordination requires only readily available kinematic state (positions and velocities), with no dependence on privileged information or expert demonstrations. Scope. This work introduces the proto-plan coordination mechanism as a foundational advancement in hierarchical multi-agent systems. Our primary contribution is the architectural demonstration that continuous, learned embeddings provide a more robust and scalable coordination interface than traditional discrete command structures. By isolating the multi-agent coordination challenge from the complexities of raw perception and low-level vehicle dynamics, we provide a rigorous validation of the proto-plan’s efficacy under controlled simulation conditions. This design choice allows us to confirm that the observed performance gains, including collision-free navigation and zero-shot transfer, are intrinsic to the hierarchical architecture itself. While this work serves as a definitive proof-of-concept for the MAPS framework, the architecture is designed for modular extensibility to high-fidelity environments and complex sensor suites, as discussed in Section 5.4. 2 Related Work 2.1 Multi-Agent Reinforcement Learning MARL extends single-agent RL to settings with multiple decision-makers facing partial observability, non-stationary dynamics, and cooperation–competition tradeoffs. Recent work targets scalability, safety, and coordination in large-scale systems [Liu et al.(2024)]. Model-based approaches include Ma et al. [Ma et al.(2024)], who topologically decouple global dynamics for local model learning while approximating global value information, improving sample efficiency at scale. For safety-critical settings, Zhang et al. [Zhang et al.(2024b)] enforce joint constraints via local trust-region updates over κ-hop neighborhoods, enabling decentralized training without centralized critics, while Hsu and Pajic [Hsu and Pajic(2025)] provide regret guarantees for safe cooperative MARL with function approximation. Graph-based representations encode structural priors effectively. GNN-based methods support resilient multi-robot coordination under agent failures and communication disturbances [Weil et al.(2024)]. In traffic management, decentralized graph-based MARL with traffic digital twins improves signal timing in large networks [Wang et al.(2025a)], and EECG [Peng et al.(2025)] integrates GNNs with curiosity-driven exploration and evolutionary optimization for improved credit assignment in partially observable tasks. These approaches address important coordination aspects but typically operate within fixed action-space structures requiring agents to reason over peer-scaling representations. Our work is complementary: rather than improving the learning algorithm, we introduce a hierarchical communication mechanism via the proto-plan embedding, compressing coordination into a constant-size signal that allows more effective application of existing RL algorithms. 2.2 MARL for Autonomous Driving Applying MARL to autonomous driving demands safety, efficiency, and real-time responsiveness. Xu et al. [Xu et al.(2022)] propose DDPG-based algorithms with meta-exploration and twin-delayed variants for intersection navigation, highlighting reward engineering for multi-objective optimization. In value decomposition, QMIXwD [Huang et al.(2023)] integrates self-generated demonstrations for improved early exploration, while VN-MADDPG [Zhang et al.(2024a)] extends MADDPG with variable-noise and importance-sampling for more efficient multi-vehicle learning. Graph-based models effectively capture vehicle interactions. DQ-GAT [Cai et al.(2022)] leverages bird’s-eye-view maps and graph attention for complex spatial relationships, and Spatharis and Blekas [Spatharis and Blekas(2024)] propose collaborative frameworks with route agents and collision terms for scalable SUMO coordination. For policy optimization, Peng et al. [Peng et al.(2023)] introduce curriculum PPO with stage-decaying clipping across difficulty levels, and Xu et al. [Xu et al.(2022)] combine TD3 with LSTM-based motion prediction for smoother trajectories. Hierarchical architectures, particularly relevant to our work, decouple strategic from tactical decision-making. Al-Sharman et al. [Al-Sharman et al.(2022)] separate high-level behavioral planning from low-level control via SAC-MPC hierarchy. MA-GA-DDPG [Liu et al.(2025)] augments MADDPG with multi-head attention and level-k game priors, incorporating a safety inspector for cooperative CAV behavior. SafeR-ADAIM [Zhao et al.(2024)] demonstrates risk-aware constrained optimization for dense intersection safety. These hierarchical methods typically issue discrete commands or explicit sub-goals, yielding action spaces that grow combinatorially with fleet size and require redesign for new behaviors. Similarly, peer-to-peer learned communication methods like CommNet [Sukhbaatar et al.(2016)] and TarMAC [Das et al.(2019)] require each agent to send and receive messages, with per-agent aggregation scaling with peer count. MAPS departs from both paradigms: the Master produces a continuous proto-plan embedding encoding coordination intent in fixed-dimensional latent space. This avoids combinatorial scaling, requires no predefined command vocabulary, keeps Worker input size constant regardless of fleet size, and, as shown in Section 5.1, enables zero-shot transfer to larger agent populations. 3 Proposed Approach 3.1 Overview We formulate multi-vehicle intersection coordination as a hierarchical Markov decision process comprising two levels (Figure 1): a centralized Master agent that observes the global traffic state and generates a proto-plan embedding zt∈ℝdz_t ^d representing the desired coordination strategy, and decentralized Worker agents, each controlling an individual vehicle, that combine the proto-plan with local observations to select vehicle-specific actions. The Master’s action space is a continuous latent space rather than an enumeration of per-vehicle commands, avoiding the ||N|A|^N combinatorial scaling of joint action spaces while keeping communication overhead at O(d)O(d) regardless of fleet size. Figure 1: Hierarchical MARL framework. The Master agent observes global state stMs_t^M and generates a proto-plan embedding ztz_t. Each Worker i receives ztz_t along with its local observation to produce vehicle-specific actions atia_t^i. 3.2 Problem Formulation We model the problem as a decentralized partially observable Markov decision process (Dec-POMDP) with hierarchical control, defined by the tuple ⟨,,i,i,,Ri,γ⟩ ,S,\O_i\,\A_i\,T,\R_i\,γ , where =1,…,NN=\1,…,N\ is the set of vehicles, S is the global state space (positions and velocities), iO_i and iA_i are the local observation and action spaces for Worker i, :×1×⋯×N→Δ()T:S×A_1×·s×A_N→ (S) is the transition function, Ri:×i→ℝR_i:S×A_i is the reward for agent i, and γ∈[0,1)γ∈[0,1) is the discount factor. The objective is to find policies πii=1N\ _i\_i=1^N maximizing the expected cumulative discounted reward: J=[∑t=0Tγt∑i=1NRi(st,ati)],J=E [ _t=0^Tγ^t _i=1^NR_i(s_t,a_t^i) ], (1) subject to each Worker i accessing only its local observation oti∈io_t^i _i rather than the full state st∈s_t . The Master bridges this information gap by observing the global state and communicating coordination intent through the proto-plan, realizing a centralized-training-with-decentralized-execution (CTDE) paradigm [Lowe et al.(2017), Wang et al.(2025b)]. 3.3 State Space Representation 3.3.1 Master State Space The Master observes a global state stMs_t^M formed by concatenating the kinematic information of all N vehicles: stM=[x1,y1,vx1,vy1,…,xN,yN,vxN,vyN]∈ℝ4N,s_t^M= [x_1,y_1,v_x_1,v_y_1,…,x_N,y_N,v_x_N,v_y_N ] ^4N, (2) where (xi,yi)(x_i,y_i) and (vxi,vyi)(v_x_i,v_y_i) denote the position and velocity of vehicle i in a global frame centered at the intersection. Vehicles are ordered by approach direction (N, E, S, W) and then by descending proximity to the intersection center. The input layer accommodates a fixed maximum of NmaxN_ vehicles. When only k<Nmaxk<N_ vehicles are active, the remaining slots are zero-padded: stM=[x1,y1,vx1,vy1,…,xk,yk,vxk,vyk,4(Nmax−k)].s_t^M= [x_1,y_1,v_x_1,v_y_1,…,x_k,y_k,v_x_k,v_y_k,0_4(N_ -k) ]. (3) 3.3.2 Worker State Space Each Worker i observes a local state combining its own kinematics with the proto-plan: stWi=[xi,yi,vxi,vyi,zt]∈ℝ4+d,s_t^W_i= [x_i,y_i,v_x_i,v_y_i,z_t ] ^4+d, (4) where zt∈ℝdz_t ^d is the proto-plan embedding (d=4d=4 in our experiments). Workers receive no direct observations of other vehicles; they rely entirely on the proto-plan for coordination information. This keeps each Worker’s input size constant at 4+d4+d regardless of fleet size, improving both scalability and privacy. 3.4 Action Space and Proto-Plan Mechanism 3.4.1 Master Action Space The Master’s action space is defined as M=ℝdA_M=R^d. Given the global state stMs_t^M, the Master policy outputs a proto-plan embedding: zt=πMθM(stM),z_t= _M _M(s_t^M), (5) where πMθM _M _M is parameterized by neural network weights θM _M, with a tanh output layer bounding zt∈(−1,1)dz_t∈(-1,1)^d. Unlike conventional hierarchical RL that issues discrete high-level commands such as “vehicle 1 yields, vehicle 2 proceeds”, the proto-plan encodes coordination strategies as dense vectors in a continuous space. The semantics of individual dimensions emerge from end-to-end training rather than manual specification. While this mechanism shares surface similarity with learned communication protocols [Sukhbaatar et al.(2016), Das et al.(2019)], the design differs fundamentally. First, proto-plan communication is asymmetric and one-to-many: a single Master broadcasts a fixed-size vector to all Workers, whereas CommNet and TarMAC employ symmetric peer-to-peer messaging with message complexity scaling with fleet size. Second, the proto-plan is not a message about any individual agent; it is a compressed global coordination strategy, conceptually closer to a learned option or subgoal in the hierarchical RL sense. Third, Workers are fully decoupled from fleet composition: they receive the same d-dimensional input regardless of active agents, whereas peer-to-peer protocols require each agent to aggregate messages from a variable number of peers. This architectural separation means the Master absorbs all coordination complexity, allowing Worker policies to remain fleet-size-agnostic and enabling the zero-shot transfer demonstrated in Section 5.3. 3.4.2 Worker Action Space Each Worker operates with a discrete action space W=accelerate,decelerateA_W=\ accelerate, decelerate\, where each action modifies the target speed by Δvtarget=±5m/s v_target=± 5\,m/s. A low-level controller then computes the physical acceleration required to reach the new target speed. The control frequency is 1Hz1\,Hz; no “maintain speed” option is provided. At each time step, the Worker policy selects an action based on its local state: ati=πWθW(stWi).a_t^i= _W _W(s_t^W_i). (6) 3.5 Reward Structure 3.5.1 Worker Reward Function Each Worker receives a reward encouraging safe, efficient traversal: rtWi=+Rsuccessif vehicle i crosses successfully−Rcollisionif vehicle i is involved in a collision−Rstepotherwise (per time step)r_t^W_i= cases+R_success&if vehicle i crosses successfully\\ -R_collision&if vehicle i is involved in a collision\\ -R_step&otherwise (per time step) cases (7) with Rsuccess=50R_success=50, Rcollision=300R_collision=300, and Rstep=5R_step=5. The asymmetric magnitudes (Rcollision≫Rsuccess>RstepR_collision R_success>R_step) enforce a strict priority hierarchy: safety first, then efficiency, then throughput [Hua et al.(2025), Zheng and Gu(2025)]. We forego reward normalization to maintain a steep value gradient, ensuring the collision penalty strictly dominates cumulative step costs. 3.5.2 Master Reward Function The Master’s reward aggregates individual Worker rewards via a minimum operator: RtM=mini∈active(t)rtWiif active(t)≠∅0otherwiseR_t^M= cases _i _active(t)r_t^W_i&if W_active(t)≠ \\ 0&otherwise cases (8) where active(t)=i∈∣fi(t)=0W_active(t)=\i f_i(t)=0\ is the set of vehicles that have not yet reached their destination. This maximin objective forces the Master to maximize the worst-case individual outcome, preventing coordination strategies that sacrifice any single vehicle. Replacing min with mean aggregation reduces SR to 84% during training and 80% during evaluation. 3.5.3 Episode Termination An episode terminates when (i) all vehicles reach their destinations, (i) a collision occurs, or (i) the 50-step limit is reached. 3.6 Network Architecture Both Master and Worker agents are trained using Proximal Policy Optimization (PPO) [Schulman et al.(2017)]. Each Worker module consists of separate policy and value networks with four fully-connected layers (64, 32, 16, 8 neurons; ReLU activations). The policy network outputs a categorical distribution over the two actions via softmax; the value network outputs a scalar estimate. The Master uses three-layer networks (128, 256, 128 neurons) to accommodate the higher-dimensional global input (4Nmax4N_ features), with a tanh output layer producing the d-dimensional proto-plan. All Worker agents share parameters θW _W, reducing learnable parameters and encouraging generalizable control strategies. Since Workers operate on local, ego-centric observations, a single policy can map diverse situational contexts to unified driving behaviors. 3.7 Training Process Training proceeds through alternating optimization of the Master and Worker modules, as formalized in Algorithm 1. In the initial cycle (cycle 0), both policies are updated jointly to establish preliminary coordination. Subsequent cycles alternate between freezing the Master (Workers adapt to current proto-plan representations) and freezing the Workers (the Master learns to generate more effective proto-plans given Workers’ current behavior). This reduces the non-stationarity inherent in multi-agent optimization: each module trains against a fixed counterpart, yielding more reliable gradient estimates and stable convergence. Experience is collected into a dedicated rollout buffer for the Master and individual buffers for each Worker. The complete set of training hyperparameters is provided in Table LABEL:tab:hyperparameters. 3.7.1 Incremental Training for Generalization To evaluate the transferability of learned proto-plans, we employ an incremental training protocol. The Master’s input layer is sized for Nmax=5N_ =5 agents from the outset, with unused slots zero-padded. Training proceeds in two phases: 1. Phase 1: Train with 1 learning agent until convergence. 2. Phase 2: Introduce 2 additional agents (3 total) and continue training. The resulting model is then evaluated on 5-agent configurations without fine-tuning; results are reported in Section 5. Algorithm 1 Alternating Master-Worker Training Input: Configuration ℰE, environment E, Master policy πM _M, Worker policy πW _W Output: Trained policies πM _M, πW _W; collision count c; training statistics 1: N←N← number of vehicles in E 2: Initialize rollout buffers ℬMB_M for Master and ℬWB_W for Workers 3: Initialize collision count c←0c← 0 4: Initialize statistics container S 5: for cycle=0cycle=0 to ℰ.NUM_CYCLES−1E.NUM\_CYCLES-1 do 6: for episode=1episode=1 to ℰ.EPISODES_PER_CYCLEE.EPISODES\_PER\_CYCLE do 7: (rep,τ)←ExecuteEpisode(πM,πW,E)(r_ep,τ)← ExecuteEpisode( _M, _W,E) 8: Store trajectory τ in appropriate buffers 9: if collision detected in τ then 10: c←c+1c← c+1 11: end if 12: Record episode statistics in S 13: if cycle=0cycle=0 then 14: UpdatePPO(πM,ℬM _M,B_M); UpdatePPO(πW,ℬW _W,B_W) Joint 15: else if cyclemod2=0cycle 2=0 then 16: UpdatePPO(πM,ℬM _M,B_M) Master only 17: else 18: UpdatePPO(πW,ℬW _W,B_W) Workers only 19: end if 20: E.Reset()E. Reset() 21: end for 22: end for 23: return πM _M, πW _W, c, S 4 Experimental Setup 4.1 Simulation Environment We evaluate our approach using the HighwayEnv simulator [Leurent(2018)], an open-source reinforcement learning environment for autonomous driving research. Its modular architecture permits customization of state representations, action spaces, and reward functions, making it well-suited to our hierarchical framework. We deliberately employ HighwayEnv’s simplified kinematics as a feature: by abstracting away perception pipelines and detailed vehicle dynamics, the simulator isolates the multi-agent coordination challenge, providing a controlled testbed where performance differences can be attributed to coordination architecture rather than confounding factors. 4.2 Scenario Configuration 4.2.1 Intersection Scenarios We constructed 18 base scenarios varying in approach direction (N, S, E, W), initial distance from the intersection, and turning intention (straight, left, right). Each base scenario is rotated through all four cardinal orientations (0°, 90°, 180°, 270°), yielding 72 unique configurations. This rotation augmentation prevents overfitting to specific approach directions and ensures generalization across symmetric traffic patterns. 4.2.2 Scenario Design Principles Scenarios were hand-designed to satisfy two properties. First, solvability: each scenario admits at least one collision-free coordination strategy, validated by executing rule-based policies. Second, non-triviality: all scenarios feature inherently conflicting trajectories that cannot be resolved by simple reactive behaviors such as fixed yielding. The high collision rates of both baselines on these scenarios (Table 1) empirically confirm that non-trivial coordination is required. 4.2.3 Vehicle Initialization and Agent Assignment At the start of each episode, five vehicles are positioned along their respective approach lanes at distances ranging from 0 to 75 m from the intersection entry point, all initialized at 20 m/s. Each vehicle’s approach direction and intended trajectory are specified by the scenario definition. Of the five vehicles, three are controlled by MAPS learning agents, and two follow constant-speed trajectories (20 m/s) to simulate non-cooperative background traffic. Assignments are fixed per scenario. Figure 2 illustrates an example initial configuration and the resulting conflict zone. 4.2.4 Training and Evaluation Protocol Training is conducted over 900 episodes using the alternating optimization procedure of Algorithm 1, with scenarios sampled uniformly at random from the 72 configurations. Evaluation is performed deterministically, removing all exploration noise across 100 episodes drawn uniformly from the same scenario distribution. Deterministic evaluation tests whether the learned policy has converged to a robust coordination strategy, and uniform sampling ensures coverage across the full range of traffic patterns. All methods use a fixed random seed for deterministic reproducibility; we discuss implications in Section 5.4. 4.3 Baseline Methods Table LABEL:tab:hyperparameters (see Appendix) summarizes the training configuration used by our approach in all experiments. We compare MAPS against two recent methods representing distinct paradigms for multi-agent intersection coordination: value decomposition with demonstrations and actor-critic with adaptive exploration. (a) Initial configuration: five vehicles approach from all directions with varying distances to the intersection. (b) Conflict zone: vehicles converge at the intersection center, requiring coordinated timing to avoid collisions. Figure 2: Example scenarios from the HighwayEnv environment, illustrating initial vehicle placement and the resulting high-density conflict zone. QMIXwD [Huang et al.(2023)]. This baseline addresses exploration challenges by integrating learning from demonstrations into the QMIX value-decomposition framework. A pre-training stage leverages both expert demonstrations and self-generated interaction data to mitigate distributional shift. Following the original methodology, our implementation employs a loss combining a supervised margin term, a TD(λ)TD(λ) loss, and L2L_2 regularization. The demonstration dataset consists of 10% expert trajectories from a pre-trained greedy policy and 90% self-generated data. VN-MADDPG [Zhang et al.(2024a)]. This baseline extends the MADDPG actor-critic framework with variable-noise exploration and importance-sampling mechanisms to improve learning efficiency in continuous multi-agent action spaces. Fair comparison. All methods were implemented following the original papers, received identical observations, and were trained for the same 900 episodes on the same scenario distribution. Baseline hyperparameters follow the original publications. Baseline selection rationale. We selected baselines representing two dominant paradigms: value decomposition with demonstrations (QMIXwD) and actor-critic with adaptive exploration (VN-MADDPG). Other hierarchical methods from Section 2, such as MA-GA-DDPG [Liu et al.(2025)] and SafeR-ADAIM [Zhao et al.(2024)], were not included because they rely on additional privileged mechanisms—level-k game-theoretic priors and safety inspector modules, or risk-aware constrained optimization with domain-specific safety layers—that are external to the core learning architecture. Since our central claim concerns the proto-plan coordination mechanism itself, we compare against methods that, like MAPS, rely solely on learned coordination from kinematic observations, ensuring performance differences are attributable to architectural design rather than supplementary safety modules. 4.4 Evaluation Metrics We assess performance using four complementary metrics capturing safety, efficiency, and learning stability: • Success Rate (SRSR): The percentage of episodes in which all vehicles cross the intersection without collision: SR=Nsuccess/Ntotal×100%SR=N_success/N_total× 100\%. The complementary collision rate is CR=100%−SRCR=100\%-SR. Episodes reaching the 50-step time limit without all vehicles completing traversal are counted as failures. • Training Collision Count (CC): The total number of collision events across all training episodes, CC=∑k=1KckCC= _k=1^Kc_k where ck∈0,1c_k∈\0,1\. This tracks learning safety, where lower values indicate fewer dangerous experiences during training. • Cumulative Episode Reward (R): The total reward accumulated over an episode, R=∑t=1TrtR= _t=1^Tr_t. Higher values indicate better combined safety and efficiency. • Average Travel Time (ATTATT): The mean number of simulation steps for active learning agents to traverse the intersection: ATT=1|active|∑i=1|active|τiATT= 1|W_active| _i=1^|W_active| _i, where τi _i is the travel time for vehicle i. This reflects traversal efficiency: inefficient coordination leads to longer episodes, while early collisions artificially reduce step counts. 5 Evaluation Results 5.1 Main Results Table 1 summarizes the comparative results, and Figure 3 illustrates the learning dynamics during training. MAPS is the only approach to achieve collision-free evaluation while recording the lowest average travel time. Figure 3: Training dynamics comparing MAPS and baselines: (a) success rate, (b) collision rate, (c) cumulative episode reward, and (d) average travel time. Moving average window size: 20. Table 1: Comparison of collision counts and efficiency across methods. Approach Collisions (Train - 900) Collisions (Eval - 100) Avg Steps (Evaluation) MAPS (ours) 21 0 7.8 VN-MADDPG 132 35 12.7 QMIXwD 147 31 13.2 5.1.1 Safety Performance MAPS achieves zero collisions during evaluation (100% success rate), compared to 35 collisions (65% SR) for VN-MADDPG and 31 collisions (69% SR) for QMIXwD. The safety advantage extends to training. MAPS incurs only 21 collisions during 900 training episodes, an 84–85% reduction relative to VN-MADDPG (132) and QMIXwD (147), indicating that the hierarchical architecture learns safer behaviors more rapidly. A two-proportion z-test confirms these reductions are statistically significant (p<0.001p<0.001) for both baselines. 5.1.2 Efficiency Performance MAPS demonstrates superior traversal efficiency: vehicles require an average of 7.8 simulation steps to cross the intersection, compared to 12.7 for VN-MADDPG and 13.2 for QMIXwD, a 38% reduction over the best baseline. This gain stems from MAPS learning proactive coordination strategies that minimize unnecessary yielding, with the proto-plan mechanism enabling Workers to anticipate coordination requirements rather than react to imminent conflicts. 5.1.3 Learning Dynamics Figure 3 reveals several notable patterns. MAPS converges to near-optimal performance within approximately 180 episodes, after which its learning curves exhibit markedly lower variance than either baseline. VN-MADDPG and QMIXwD show persistent oscillations in both success rate and reward throughout training, suggesting difficulty maintaining consistent coordination. Despite sharing the same 900-episode training budget, MAPS achieves substantially better final outcomes, indicating that the hierarchical decomposition yields a more tractable learning problem than flat multi-agent architectures. 5.2 Ablation Studies 5.2.1 Contribution of the Master Agent To isolate the contribution of the learned proto-plan, we hold Worker policies fixed and manipulate only the Master’s output during inference. We evaluate three conditions: zttrained z_t^trained =πM(stM), = _M(s_t^M), (9) ztrandom z_t^random ∼(−1,1)d, (-1,1)^d, (10) ztzero z_t^zero =d, =0_d, (11) where πM _M is the trained Master policy. Under all conditions, Workers use the same trained policy πW _W, with stWis_t^W_i constructed by concatenating kinematic observations with the manipulated ztz_t. Figure 4 reports cumulative episode rewards over 400 evaluation episodes. The trained Master achieves the highest returns (mean reward: 56.6), random proto-plans yield intermediate performance (25.7), and the zero-output condition collapses to strongly negative rewards (−304.7-304.7), a difference of over 360 reward points. These results demonstrate that the proto-plan channel is essential for coordination and that Workers have learned to condition their behavior on its content. 5.2.2 Proto-Plan Embedding Analysis To interpret the information encoded by the Master, we project the d-dimensional proto-plan vectors collected during evaluation to two dimensions using PCA and label each timestep by a safety predicate (dangerous if any inter-vehicle distance falls below 5 m). Figure 4: Cumulative episode reward under three proto-plan conditions with Workers fixed. The trained Master significantly outperforms random and zero baselines, confirming that learned proto-plans encode essential coordination information. Figure 5 reveals that proto-plans occupy distinct regions for safe versus dangerous states, indicating that the Master has learned to compress global interaction risk into a compact representation. The first two principal components capture 76% of the total variance (PC1: 63%, PC2: 13%). Figure 5: PCA projection of proto-plan vectors, colored by safety predicate (dangerous: any inter-vehicle distance <5<5 m). Proto-plans encode safety-relevant context that Workers use for coordination decisions. 5.2.3 Sensitivity to Embedding Dimension We evaluated the architecture across embedding dimensions d∈2,4,8,16d∈\2,4,8,16\. As shown in Figure 6, d=4d=4 yields the best performance (80.0% SR), compared to 70.0% for d=2d=2, 78.0% for d=8d=8, and 72.0% for d=16d=16. Note on evaluation protocols. The success rates reported in this ablation study differ from those in Table 1 because the two experiments use different training configurations. Figure 6 uses a single-phase training setup to isolate the effect of embedding dimension, whereas Table 2 reports the final incrementally trained model (Phase 1 → Phase 2). Analysis of training dynamics confirms that d=4d=4 yields the most stable reward distribution. Figures 7 and 8 provide direct optimization and reward evidence. Figure 7 shows that d=4d=4 achieves the most balanced convergence across both Master and Worker value losses, with low final losses and reduced late-stage oscillations. Figure 8 shows that d=4d=4 also maximizes reward quality with the strongest central tendency and robust dispersion, while higher-dimensional settings (especially d=16d=16) exhibit degraded central tendency and frequent outliers. Together, these results indicate that a 4-dimensional proto-plan is the optimal operating point: it is stable to optimize, converges reliably to low-loss solutions, and yields the strongest policy-level returns. Figure 6: Impact of proto-plan embedding dimension on success rate. Performance peaks at d=4d=4 (80.0%), indicating that a compact 4-dimensional vector provides the optimal balance between representational capacity and optimization stability. Figure 7: Master and Worker value-loss trajectories across embedding dimensions. The d=4d=4 setting shows the most balanced and stable joint convergence across modules, with low final losses and reduced late-stage oscillations. This pattern reflects a trade-off between representational capacity and learnability: d=2d=2 provides insufficient capacity to encode necessary coordination context, while d≥8d≥ 8 introduces optimization difficulty, leading to noisier policies. The result supports the central thesis that effective fleet coordination can be compressed into a very low-dimensional signal, making the proto-plan mechanism practical even under bandwidth-constrained conditions. Figure 8: Reward distribution by embedding dimension. The d=4d=4 policy exhibits the strongest central reward profile and robust dispersion, while higher-dimensional settings, especially d=16d=16, show degraded central tendency and increased outlier behavior. 5.3 Zero-Shot Transfer to Unseen Fleet Sizes To evaluate whether the learned proto-plans capture transferable coordination strategies, we test the model trained with three active agents (Section 3.7.1) on five-agent configurations without fine-tuning. Across 100 test episodes, the system achieves a 94% success rate, demonstrating robust generalization to larger fleet sizes. The 6 failed episodes result from collisions in high-density or turning scenarios, with no significant pattern. This result can be attributed to three architectural properties: (1) the proto-plan embedding conveys coordination information in a format agnostic to the specific number of agents; (2) Worker parameter sharing ensures that control policies generalize across vehicle instances; and (3) the maximin reward structure encourages strategies that do not depend on particular agent configurations. Together, these properties indicate that proto-plan-based coordination generalizes beyond the training regime without requiring additional learning. 5.4 Limitations and Design Scope We identify several limitations that reflect deliberate scoping decisions and directions for future work. Simulation fidelity. Our evaluation uses HighwayEnv, a kinematic simulator that abstracts away perception noise, sensor latency, and detailed vehicle dynamics. This choice is intentional: it isolates the multi-agent coordination mechanism from confounding factors, providing a controlled testbed for validating the proto-plan architecture. However, validation in higher-fidelity environments (e.g., SUMO or CARLA) is needed to confirm that coordination benefits transfer when richer dynamics and sensor noise are present. Action space. The current Worker action space is limited to binary longitudinal speed control (accelerate/decelerate). While this is a simplification, the architecture is agnostic to the Worker action space: continuous acceleration, lateral maneuvers, or richer discrete action sets can be substituted by modifying only the Worker policy head, without changes to the Master or the proto-plan mechanism. Evaluating such extensions is a priority for future work. Fleet size. While the architecture demonstrates zero-shot transfer from three to five agents, the current Master uses a fixed-size concatenated input (4Nmax4N_ features), which imposes a hard maximum. Scaling to substantially larger fleets would benefit from permutation-invariant input processing, such as attention-based or set-based architectures for the Master, which is a natural extension that preserves the proto-plan broadcast mechanism. The present study validates the core coordination mechanism; large-scale deployment remains an open question. Baseline scope. We compare against two representative flat MARL methods. Hierarchical baselines that incorporate privileged mechanisms (game priors, safety inspectors) were excluded to ensure that performance differences reflect the coordination architecture rather than supplementary modules (see Section 4.3 for details). 6 Conclusions and Future Work We presented MAPS, a hierarchical DRL architecture for multi-vehicle coordination at unsignalized intersections. The architecture uses a continuous proto-plan embedding generated by a centralized Master and consumed by decentralized Workers, decoupling strategic coordination from tactical control while avoiding combinatorial action space scaling. Evaluations across 72 intersection configurations showed collision-free navigation with 38% reduced travel time versus the best baseline, and ablation studies confirmed that proto-plans encode meaningful coordination information. The architecture achieved 94% success in zero-shot generalization from three to five agents. The key insight is that multi-vehicle coordination compresses into a 4-dimensional continuous signal, enabling deployment in communication-constrained and bandwidth-limited V2X systems. Future work includes: (1) extending proto-plans to encode long-horizon strategies, amortizing communication cost across multiple time steps; (2) exploring hierarchical scaling through mid-level masters that relay refined plans from higher nodes to local Workers; and (3) investigating heterogeneous fleets via vehicle-class-specific Worker modules, leveraging modularity to integrate new agent types without retraining. References [1] [Al-Sharman et al.(2022)] Mohammad Al-Sharman, Rowan Dempster, Mohamed A. Daoud, Mahmoud Nasr, Derek Rayside, and William Melek. 2022. Self-Learned Autonomous Driving at Unsignalized Intersections: A Hierarchical Reinforced Learning Approach for Feasible Decision-Making. doi:10.36227/techrxiv.20770486.v1 Preprint. [Cai et al.(2022)] Peide Cai, Hengli Wang, Yuxiang Sun, and Ming Liu. 2022. DQ-GAT: Towards Safe and Efficient Autonomous Driving With Deep Q-Learning and Graph Attention Networks. IEEE Transactions on Intelligent Transportation Systems 23, 11 (2022), 21102–21112. doi:10.1109/TITS.2022.3189917 [Chen et al.(2024)] Kaixin Chen, Bing Li, Rongqing Zhang, and Xiang Cheng. 2024. Autonomous Intersection Management with Heterogeneous Vehicles: A Multi-Agent Reinforcement Learning Approach. In 2024 IEEE Intelligent Vehicles Symposium (IV). 2255–2260. doi:10.1109/IV55156.2024.10588863 [Das et al.(2019)] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. 2019. TarMAC: Targeted Multi-Agent Communication. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 1538–1546. https://proceedings.mlr.press/v97/das19a.html [Hsu and Pajic(2025)] Hao-Lun Hsu and Miroslav Pajic. 2025. Safe Cooperative Multi-Agent Reinforcement Learning with Function Approximation. In Proceedings of the 7th Annual Learning for Dynamics & Control Conference (Proceedings of Machine Learning Research, Vol. 283). PMLR, 1353–1364. https://proceedings.mlr.press/v283/hsu25a.html [Hua et al.(2025)] Min Hua, Xinda Qi, and Dong Chen. 2025. Multi-Agent Reinforcement Learning for Connected and Automated Vehicles Control: Recent Advancements and Future Prospects. IEEE Transactions on Automation Science and Engineering 22 (2025). [Huang et al.(2023)] Chang Huang, Junqiao Zhao, Hongtu Zhou, Hai Zhang, Xiao Zhang, and Chen Ye. 2023. Multi-agent Decision-making at Unsignalized Intersections with Reinforcement Learning from Demonstrations. In 2023 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1–8. doi:10.1109/IV55152.2023.10186792 [Leurent(2018)] Edouard Leurent. 2018. An Environment for Autonomous Driving Decision-Making. https://github.com/eleurent/highway-env. [Liu et al.(2024)] Dingbang Liu, Fenghui Ren, Jun Yan, Guoxin Su, Wen Gu, and Shohei Kato. 2024. Scaling Up Multi-Agent Reinforcement Learning: An Extensive Survey on Scalability Issues. IEEE Access 12 (2024), 94610–94631. doi:10.1109/ACCESS.2024.3410318 [Liu et al.(2025)] Jiaqi Liu, Peng Hang, Xiaoxiang Na, Chao Huang, and Jian Sun. 2025. Cooperative Decision-Making for CAVs at Unsignalized Intersections: A MARL Approach With Attention and Hierarchical Game Priors. IEEE Transactions on Intelligent Transportation Systems 26, 1 (Jan. 2025), 443–455. doi:10.1109/TITS.2024.3503092 [Lowe et al.(2017)] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017). [Ma et al.(2024)] Chengdong Ma, Aming Li, Yali Du, Hao Dong, and Yaodong Yang. 2024. Efficient and Scalable Reinforcement Learning for Large-Scale Network Control. Nature Machine Intelligence 6, 9 (2024), 1006–1020. doi:10.1038/s42256-024-00879-7 [Peng et al.(2025)] Kexing Peng, Pengyi Li, and Jianye Hao. 2025. Enhancing Graph-based Coordination with Evolutionary Algorithms for Episodic Multi-agent Reinforcement Learning. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025). International Foundation for Autonomous Agents and Multiagent Systems, Detroit, MI, USA, 1623–1631. https://w.ifaamas.org/Proceedings/aamas2025/pdfs/p1623.pdf [Peng et al.(2023)] Zengqi Peng, Xiao Zhou, Yubin Wang, Lei Zheng, Ming Liu, and Jun Ma. 2023. Curriculum Proximal Policy Optimization with Stage-Decaying Clipping for Self-Driving at Unsignalized Intersections. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 5027–5033. doi:10.1109/ITSC57777.2023.10422594 [Schulman et al.(2017)] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). [Spatharis and Blekas(2024)] Christos Spatharis and Konstantinos Blekas. 2024. Multiagent reinforcement learning for autonomous driving in traffic zones with unsignalized intersections. Journal of Intelligent Transportation Systems 28, 1 (2024), 103–119. doi:10.1080/15472450.2022.2109416 [Sukhbaatar et al.(2016)] Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. 2016. Learning Multiagent Communication with Backpropagation. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc. https://proceedings.neurips.c/paper_files/paper/2016/file/55b1927fdafef39c48e5b73b5d61ea60-Paper.pdf [Wang et al.(2024)] Bowen Wang, Xinle Gong, Yafei Wang, Peiyuan Lyu, and Sheng Liang. 2024. Coordination for Connected and Autonomous Vehicles at Unsignalized Intersections: An Iterative Learning-Based Collision-Free Motion Planning Method. IEEE Internet of Things Journal 11, 3 (2024), 5439–5454. doi:10.1109/JIOT.2023.3306572 [Wang et al.(2025a)] Kang Wang, Zhishu Shen, Zhen Lei, Xianhui Liu, and Tiehua Zhang. 2025a. Toward Multi-Agent Reinforcement Learning Based Traffic Signal Control Through Spatio-Temporal Hypergraphs. IEEE Transactions on Mobile Computing 24, 9 (2025), 8258–8271. doi:10.1109/TMC.2025.3556243 [Wang et al.(2025b)] Ye Wang, Jingjing Wang, Ruijie Zhu, Hang Fu, Jianrui Chen, and C. L. Philip Chen. 2025b. Facilitating Multiagent Coordination Relying on Graph Information Representation. IEEE Transactions on Neural Networks and Learning Systems 36, 10 (2025), 17929–17940. doi:10.1109/TNNLS.2025.3575196 [Weil et al.(2024)] Jannis Weil, Zhenghua Bao, Osama Abboud, and Tobias Meuser. 2024. Towards Generalizability of Multi-Agent Reinforcement Learning in Graphs with Recurrent Message Passing. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS ’24). 1919–1927. [Xu et al.(2022)] Shu-yuan Xu, Xue-mei Chen, Zi-jia Wang, Yu-hui Hu, and Xin-tong Han. 2022. Decision-Making Models for Autonomous Vehicles at Unsignalized Intersections Based on Deep Reinforcement Learning. In 2022 7th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). IEEE, 672–677. doi:10.1109/ICARM54641.2022.9959664 [Zhang et al.(2024a)] Hao Zhang, Yu Du, Shixin Zhao, Ying Yuan, and Qiuqi Gao. 2024a. VN-MADDPG: A Variable-Noise-Based Multi-Agent Reinforcement Learning Algorithm for Autonomous Vehicles at Unsignalized Intersections. Electronics 13, 16 (2024), 3180. doi:10.3390/electronics13163180 [Zhang et al.(2024b)] Lijun Zhang, Lin Li, Wei Wei, Huizhong Song, Yaodong Yang, and Jiye Liang. 2024b. Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 37. doi:10.52202/079017-4400 NeurIPS. [Zhao et al.(2024)] Rui Zhao, Yun Li, Kui Wang, Yuze Fan, Fei Gao, and Zhenhai Gao. 2024. Centralized Cooperation for Connected Autonomous Vehicles at Intersections by Safe Deep Reinforcement Learning. IEEE Transactions on Mobile Computing 23, 12 (2024), 12830–12847. doi:10.1109/TMC.2024.3417441 [Zheng and Gu(2025)] Zhi Zheng and Shangding Gu. 2025. Safe Multiagent Reinforcement Learning With Bilevel Optimization in Autonomous Driving. IEEE Transactions on Artificial Intelligence 6, 4 (2025), 829–840. doi:10.1109/TAI.2025.10752922 Appendix A The Hyperparameters of Our Proposed Approach Table 2: Training Hyperparameters Parameter Value Algorithm PPO Optimizer Adam Loss Function MSE Discount Factor (γ) 0.99 Learning Rate (Master & Agent) 0.005 Batch Size 32 Rollout Buffer Size (N_STEPSN\_STEPS) 64 Clip Range (PPO) 0.2 Additional Configuration Episodes Per Cycle 300 Total Cycles 3 Gradient Update Frequency Every 2 Episodes Max Episode Time 50 Steps Agent State Input Size 8 proto-plan Embedding Dimension 4 Action Space Size 2 Number of Agents 5 Seed 42 Deep learning framework PyTorch RL library Stable-Baselines3 Hardware CPU Approximate training time 20m Reward Structure Target Reached Reward 50 Collision Reward -300 Starvation Reward -5