Paper deep dive
Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality
Ye Han, Lijun Zhang, Dejian Meng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating Multi-Agent Reinforcement Learning (MARL) policies in autonomous driving fundamentally relies on extrinsic statistical indicators (e.g., reward curves and success rates), which often mask intrinsic policy degradation and algorithmic blind spots. To break this black-box evaluation, this letter proposes a novel information-theoretic diagnostic framework. By leveraging a fully converged Monte Carlo Tree Search (MCTS) as an asymptotic oracle, we establish a theoretical ground-truth baseline distribution. We formulate a bounded policy optimality score ($\mathcal{M}_{opt}$) using the forward KL divergence to rigorously penalize fatal collaborative omissions. Crucially, we semantically decouple this metric into lateral and longitudinal dimensions, creating a granular "semantic microscope". Extensive spatial and temporal diagnostics on state-of-the-art MARL architectures and exploration mechanisms demonstrate that our framework conclusively exposes hidden directional biases, identifies temporal average-policy traps, and transforms heuristic hyperparameter tuning into a visually trackable trajectory optimization. This framework establishes a rigorous, model-agnostic standard for benchmarking intrinsic multi-agent policy quality.
Tags
Links
- Source: https://arxiv.org/abs/2608.01133v1
- Canonical: https://arxiv.org/abs/2608.01133v1
Trouble viewing inline? Open PDF directly →
Full Text
48,201 characters extracted from source content.
Expand or collapse full text
Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality Ye Han, Lijun Zhang∗, Dejian Meng Ye Han, Lijun Zhang, Dejian Meng are with the School of Automotive Studies, Tongji University, Shanghai 201804, China. hanye_leohancnjs, tjedu_zhanglijun, mengdejian@tongji.edu.cn∗Corresponding author: Lijun Zhang Abstract Evaluating Multi-Agent Reinforcement Learning (MARL) policies in autonomous driving fundamentally relies on extrinsic statistical indicators (e.g., reward curves and success rates), which often mask intrinsic policy degradation and algorithmic blind spots. To break this black-box evaluation, this letter proposes a novel information-theoretic diagnostic framework. By leveraging a fully converged Monte Carlo Tree Search (MCTS) as an asymptotic oracle, we establish a theoretical ground-truth baseline distribution. We formulate a bounded policy optimality score (ℳoptM_opt) using the forward KL divergence to rigorously penalize fatal collaborative omissions. Crucially, we semantically decouple this metric into lateral and longitudinal dimensions, creating a granular “semantic microscope”. Extensive spatial and temporal diagnostics on state-of-the-art MARL architectures and exploration mechanisms demonstrate that our framework conclusively exposes hidden directional biases, identifies temporal average-policy traps, and transforms heuristic hyperparameter tuning into a visually trackable trajectory optimization. This framework establishes a rigorous, model-agnostic standard for benchmarking intrinsic multi-agent policy quality. I Introduction The coordination of Connected and Automated Vehicles (CAVs) in dense, mixed-traffic environments is fundamentally formulated as a partially observable Markov Game [16, 30]. Due to the continuous state spaces, high-dimensional joint action spaces, and the dynamic number of interacting vehicles, traditional rule-based and centralized optimization methods often fall short [4, 31]. Consequently, Multi-Agent Reinforcement Learning (MARL), particularly under the Centralized Training with Decentralized Execution (CTDE) paradigm, has emerged as the frontier solution for multi-vehicle collaborative decision-making [16, 7]. To resolve the complex non-Euclidean interactive topologies among CAVs and human-driven vehicles (HDVs), recent studies have intensely focused on architectural innovations. Advanced spatial-temporal representation networks, such as Graph Neural Networks (GNNs) [4, 35] and Transformer-based attention mechanisms [31], have been heavily investigated. Concurrently, a plethora of specialized mechanisms—ranging from heuristic action masking [9, 3], control barrier functions (CBF) [10, 35], to multi-objective reward shaping [23, 29]—have been proposed to enhance policy convergence and empirical safety. Despite the proliferation of MARL architectures, the field has hit a severe bottleneck in policy evaluation [36]. The vast majority of current autonomous driving MARL research fundamentally relies on extrinsic statistical indicators to benchmark algorithmic superiority. These metrics predominantly consist of macro-level reward curves and post-simulation statistics, such as task success rates, collision rates, travel times, and traffic throughput [7, 31, 23, 8]. However, these superficial metrics mask the intrinsic degradation of the learned policies, leading to three critical diagnostic blind spots. First, the reliance on aggregated scalar rewards is intrinsically flawed for measuring multi-agent synergy, often leading to reward hacking [32, 20]. Hand-crafted weighted summations of safety, efficiency, and comfort inevitably induce agents to exploit environmental artifacts rather than learning genuine collaborative intelligence [21, 1]. Consequently, a converged cumulative return curve merely proves that the algorithm has overfitted to a specific heuristic, rather than achieving policy optimality [18]. Second, the widespread tune-and-report paradigm creates an illusion of algorithmic progress [15, 18]. The black-box nature of deep reinforcement learning [6] means that averaging extrinsic metrics over multiple episodes obscures the severe fragility and variance of the underlying probability distributions. An algorithm might occasionally yield high macro-returns but completely fail to guarantee structural verifiability in edge cases. Furthermore, high task success rates or collision-free statistics do not necessarily equate to advanced collaborative intelligence [36]. Empirical evaluations show that a 100% success rate can sometimes be achieved by algorithms that degenerate into overly conservative, static average behaviors (e.g., refusing to negotiate and simply waiting), severely degrading system-wide efficiency [4]. Even under high-density benchmarks, models boasting high throughput may still harbor significant collision risks due to fatal spatial misjudgments [8]. Extrinsic metrics fail to explain why a policy succeeded or mathematically pinpoint its algorithmic blindness—whether the model is specifically failing in lateral lane-change planning or longitudinal velocity coordination [29, 3]. To break this black box of MARL evaluation, a transition from extrinsic indicators to intrinsic information-theoretic metrics is urgently required. To evaluate the intrinsic quality of MARL probability distributions rather than their empirical outcomes, we need a mathematically rigorous baseline. Since both MARL and heuristic tree search fundamentally approximate the same Bellman optimality objective, Monte Carlo Tree Search (MCTS) can serve as an asymptotic oracle [27]. Utilizing MCTS as an expert demonstrator to evaluate and guide neural network policies has achieved profound success in complex zero-sum and multi-agent games [26, 14]. By extracting the optimal value landscape from a fully converged MCTS and applying a Boltzmann mapping, we can project the absolute value advantages into a theoretical baseline probability distribution [27]. With the theoretical baseline established, we transition from scalar reward evaluations to information-theoretic distribution comparisons. Information theory, particularly the Kullback-Leibler (KL) divergence, has long provided the mathematical orthodoxy for bounding policy evolution and quantifying distribution shifts in foundational RL algorithms [24, 25]. Recent advancements further validate the use of policy divergence to constrain vast exploration spaces [28] and explicitly quantify multi-agent behavioral heterogeneity [5]. Crucially, by adopting the forward KL divergence, our framework leverages its inherent mode-covering asymmetry [2]. This heavily penalizes the MARL policy if it assigns near-zero probabilities to critical collaborative maneuvers highlighted by the MCTS oracle. Consequently, it functions as a rigorous mathematical detector for fatal omissions [2] in dynamic traffic negotiations, effectively bypassing the deceptive nature of extrinsic rewards. In this paper, we propose an information-theoretic diagnostic framework that transitions MARL evaluation in autonomous driving from extrinsic statistical indicators to intrinsic policy quality. The primary contributions are summarized as follows: • Information-Theoretic Optimality Metric Formulation: We establish a theoretical baseline via an MCTS oracle and reconstruct the MARL joint policy under the CTDE paradigm. By leveraging the forward KL divergence, we formulate a bounded optimality score (ℳoptM_opt) that rigorously quantifies the intrinsic structural discrepancy of multi-agent policies. • Spatial Diagnostics via Semantic Decoupling: We semantically decouple the holistic joint policy into distinct lateral and longitudinal optimality metrics (ℳopt,latM_opt,lat, ℳopt,lonM_opt,lon). This allows the framework to act as a semantic microscope, accurately diagnosing spatial reasoning blind spots in representation network architectures and exposing the severe directional biases inherent in SOTA unconstrained exploration algorithms. • Temporal Diagnostics and Guided Optimization: We visualize the temporal evolution of the decoupled metrics to expose average-policy and collusion traps hidden by high task success rates. Furthermore, we transform hyperparameter tuning from high-variance reward-hacking into a visually trackable trajectory optimization process in a 2D metric space, ensuring balanced collaborative exploration. I Formulation of the Policy Optimality Metric To establish a rigorous diagnostic framework for multi-agent policies, we must first define the shared physical and mathematical space in which both heuristic search and reinforcement learning operate. The overall workflow for generating this metric is illustrated in Fig. 1. Figure 1: Overall framework for generating the policy optimality metric. The theoretical baseline πref _ref derived from the MCTS oracle and the evaluated policy πθ _θ from the MARL algorithm are projected into a unified probability measure space for structural comparison. I-A Multi-Vehicle Collaboration as a Markov Game We formulate the multi-vehicle collaborative decision-making process in a mixed traffic environment as a fully cooperative Markov Game. This process is defined by the tuple ⟨,,,,ℛ,γ⟩ ,S,A,P,R,γ , =1,2,…,NN=\1,2,…,N\ represents the dynamic set of controlled Connected and Automated Vehicles (CAVs) within the coordination region. S is the continuous state space capturing the global traffic kinematics and topology. ≜∏i=1N(i)A _i=1^NA^(i) denotes the vast joint action space, where (i)A^(i) is the discrete lateral and longitudinal action space of an individual CAV i. :×→[0,1]P:S×A×S→[0,1] represents the state transition dynamics, which encapsulate the physical constraints and the highly uncertain behaviors of background Human-Driven Vehicles (HDVs). ℛ:×→ℝR:S×A is the global reward function that evaluates the overall team performance (e.g., safety, throughput efficiency, and smoothness) resulting from the joint action =(a(1),…,a(N)) a=(a^(1),…,a^(N)). γ∈(0,1)γ∈(0,1) is the discount factor. The ultimate objective of this multi-vehicle system is to discover an optimal joint policy ∗ π^* that maximizes the expected cumulative discounted return from any given state. This objective is strictly governed by a unique optimal joint action-value function, Q∗(s,)Q^*(s, a). Definition 1 (Bellman Optimality Objective). For the cooperative Markov Game, the optimal joint action-value function Q∗(s,)Q^*(s, a) represents the maximum achievable expected return when taking joint action a in state s, and subsequently following the optimal policy. It satisfies the Bellman optimality equation: Q∗(s,)=ℛ(s,)+γ∑s′∈(s′|s,)max′∈Q∗(s′,′).Q^*(s, a)=R(s, a)+γ _s P(s |s, a) _ a Q^*(s , a ). (1) Remark (Theoretical Equivalence): Equation (1) serves as the theoretical bridge between MCTS and MARL. Both approaches are essentially distinct numerical solvers striving to approximate this exact same mathematical target. MCTS attempts to derive Q∗(s,)Q^*(s, a) online through asymmetric tree expansion and episodic rollouts, whereas MARL seeks to parameterize it offline using deep neural networks (e.g., via the CTDE paradigm). The existence of this shared Bellman optimality objective mathematically justifies utilizing the value estimates from a fully converged MCTS as a reliable ground-truth baseline to measure the intrinsic parameterization flaws of MARL policies. I-B Establishing the Theoretical Baseline via Bellman Equivalence Since directly solving the Bellman optimality equation in high-dimensional continuous traffic flows is mathematically intractable, we require a robust numerical proxy. I-B1 Asymptotic Optimality of MCTS Monte Carlo tree search, specifically when employing Upper Confidence Bounds applied to Trees (UCT), provides a theoretically guaranteed pathway to this proxy. Given a sufficient computational budget, the vast exploration of the joint action space allows the node action-value estimates Q^MCTS∗(s,) Q_MCTS^*(s, a) to asymptotically converge to the true Bayesian optimal value Q∗(s,)Q^*(s, a). By deploying a highly optimized MCTS solver (e.g., with parallel update pruning mechanisms[12]) as an oracle, we can extract the ground-truth value landscape of the cooperative game for any critical traffic state s. I-B2 Boltzmann Distribution Mapping While MCTS yields absolute value estimates, MARL algorithms typically output stochastic policy distributions (probabilities). To enable a direct mathematical comparison using information-theoretic metrics, the absolute value domain of MCTS must be rigorously mapped into a probability measure space. We achieve this by projecting the MCTS value landscape into a target probability distribution using a Boltzmann (softmax) formulation. Definition 2 (Theoretical Baseline Policy). For any given traffic state s, the theoretical baseline policy distribution πref(⋅|s) _ref(·|s) over the joint action space A is defined as: πref(|s)≜exp(Q^MCTS∗(s,)/κ)∑′∈exp(Q^MCTS∗(s,′)/κ), _ref( a|s) ( Q_MCTS^*(s, a)/κ ) _ a ( Q_MCTS^*(s, a )/κ ), (2) where κ>0κ>0 is the temperature parameter. Remark (Engineering Semantics of Temperature): The mapping in Equation (2) transforms the absolute advantage relationships of the Q-values into relative probabilities. The temperature parameter κ controls the smoothness of this projection. Instead of applying a hard argmax that collapses the baseline into a one-hot distribution, an appropriate κ preserves the probabilities of high-value, secondary collaborative actions. This is crucial in dynamic multi-vehicle environments, where multiple near-optimal joint maneuvers (e.g., yielding vs. accelerating) might safely resolve a traffic conflict, and punishing a MARL agent for choosing a valid secondary strategy would be unreasonable. I-C Joint Policy Reconstruction for the CTDE Paradigm To rigorously evaluate a multi-agent reinforcement learning algorithm, its decision output must be aligned with the probability measure space of the theoretical baseline. However, prevailing MARL algorithms typically adopt the Centralized Training with Decentralized Execution (CTDE) paradigm to bypass the curse of dimensionality. Depending on the specific MARL architecture, we provide two mathematically consistent pathways to reconstruct the joint policy πθ(|s) _θ( a|s). I-C1 General Formulation: Independent Marginalization In standard CTDE frameworks (e.g., independent learners or actor-critic methods like MAPPO), agents do not explicitly output a joint policy. Instead, each CAV i maintains a local utility network producing a decentralized marginal probability distribution πθ(i)(a(i)|τ(i)) _θ^(i)(a^(i)|τ^(i)) based solely on its local history τ(i)τ^(i). Assumption 1 (Conditional Independence of Execution): During the decentralized execution phase, the action selection of each individual CAV is conditionally independent of others, given its own local observation. Definition 3 (General Reconstructed Joint Policy). Under Assumption 1, the evaluated joint policy distribution for a global traffic state s is reconstructed as the product of the individual marginal policies: πθ(|s)≜∏i=1Nπθ(i)(a(i)|τ(i)), _θ( a|s) _i=1^N _θ^(i)(a^(i)|τ^(i)), (3) where =(a(1),…,a(N)) a=(a^(1),…,a^(N)) represents the sampled joint maneuver. Figure 2: Synergistic joint policy reconstruction for value decomposition networks. By iterating through the joint action space A and leveraging the centralized mixing network, the global action-value landscape QtotQ_tot is extracted to formulate a high-fidelity joint probability distribution. I-C2 Advanced Formulation for Value Decomposition While Equation (3) guarantees universality, it may inadvertently discard the complex, non-linear collaborative values learned during centralized training for algorithms explicitly utilizing a mixing network (e.g., QMIX, VDN). As illustrated in Fig. 2, for such value decomposition architectures, we can bypass the independence assumption to achieve a higher-fidelity reconstruction. By iterating through the entire joint action space A, we extract the corresponding local utilities and feed them, alongside the global state s, into the trained mixing network to reconstruct the complete global action-value landscape Qtot(s,)Q_tot(s, a). Definition 4 (Synergistic Reconstructed Joint Policy). For value decomposition networks, the joint policy distribution is derived directly by applying the Boltzmann projection to the holistic mixing network output: πθ(|s)≜exp(Qtot(s,)/κ)∑′∈exp(Qtot(s,′)/κ). _θ( a|s) (Q_tot(s, a)/κ ) _ a (Q_tot(s, a )/κ ). (4) Remark (Universality and Fidelity): By providing both the general marginalization pathway and the advanced synergistic pathway, our diagnostic framework guarantees universal applicability across any MARL architecture. It effectively lifts isolated, decentralized decisions back into the global joint probability space, ensuring strict commensurability with the MCTS baseline πref(|s) _ref( a|s) over the domain A. I-D Information-Theoretic Optimality Formulation With the theoretical baseline πref(⋅|s) _ref(·|s) and the reconstructed MARL policy πθ(⋅|s) _θ(·|s) strictly aligned within the same joint probability space, we can now mathematically quantify their structural discrepancy. Information theory, specifically the Kullback-Leibler (KL) divergence, provides a rigorous tool to measure the information loss incurred when using a parameterized policy to approximate the true optimal distribution. I-D1 Policy Divergence via Kullback-Leibler Definition 5 (Policy Optimality Distance). For a given traffic state s, the policy optimality distance optD_opt of the evaluated MARL policy πθ _θ relative to the theoretical baseline πref _ref is defined as the forward KL divergence from πθ _θ to πref _ref: opt(πθ|s) _opt( _θ|s) ≜DKL(πref(⋅|s)∥πθ(⋅|s)) D_KL ( _ref(·|s) _θ(·|s) ) (5) =∑∈πref(|s)logπref(|s)πθ(|s). = _ a _ref( a|s) _ref( a|s) _θ( a|s). Remark (Engineering Semantics of Asymmetry): The choice of the forward KL divergence, DKL(πref||πθ)D_KL( _ref|| _θ), rather than its reverse, is highly deliberate and carries profound engineering significance. Due to its inherent asymmetry, Equation (5) heavily penalizes the MARL algorithm if it assigns a near-zero probability to a joint maneuver that the MCTS baseline deems critical (πref()>0 _ref( a)>0 while πθ()≈0 _θ( a)≈ 0). In the context of multi-vehicle collaboration, this perfectly models the risk of ”fatal omissions”—where the neural network completely ignores a vital, highly collaborative, or safety-critical action required to resolve a complex bottleneck. Thus, optD_opt acts as a strict detector for missing critical optimal decisions. I-D2 The Bounded Optimality Score While optD_opt is theoretically rigorous, its domain is [0,∞)[0,∞). An unbounded metric poses practical challenges for cross-scenario aggregation, training dynamic visualization, and hyperparameter tuning. To construct an intuitive diagnostic tool, we map this divergence into a bounded scalar. Definition 6 (Bounded Optimality Score). The policy optimality score ℳoptM_opt is formulated by applying a negative exponential mapping to the policy optimality distance: ℳopt(πθ|s) _opt( _θ|s) ≜exp(−opt(πθ|s)) (-D_opt( _θ|s) ) (6) =exp(−DKL(πref(⋅|s)∥πθ(⋅|s))). = (-D_KL ( _ref(·|s) _θ(·|s) ) ). Remark (Interpretability for Diagnostics): The exponential mapping elegantly projects the divergence into a normalized, strictly bounded range ℳopt∈(0,1]M_opt∈(0,1]. A score of ℳopt=1M_opt=1 indicates that the MARL policy πθ _θ perfectly replicates the optimal synergy and action preferences of the MCTS baseline. Conversely, a score approaching 0 signifies a catastrophic deviation from rational collaboration. This bounded nature enables researchers to robustly track the intrinsic quality of a policy as it evolves across thousands of training episodes, unaffected by the noise of extrinsic reward signals. I-E Semantic Decoupling of Policy Dimensions While the global optimality score ℳoptM_opt provides a macroscopic evaluation of the joint policy, multi-vehicle collaboration inherently consists of two orthogonal physical tasks: spatial routing and velocity regulation. To achieve fine-grained diagnostics and pinpoint specific algorithmic deficiencies, we semantically decouple the holistic metric into distinct lateral and longitudinal dimensions. I-E1 Action Space Decomposition and Marginalization In structured traffic environments, the individual action a(i)a^(i) of any CAV i can be factorized into a tuple a(i)=(alat(i),alon(i))a^(i)=(a_lat^(i),a_lon^(i)). Here, alat(i)∈lat=−1,0,1a_lat^(i) _lat=\-1,0,1\ represents the lateral decisions (left lane-change, lane-keeping, right lane-change), and alon(i)∈lon=−1,0,1a_lon^(i) _lon=\-1,0,1\ denotes the longitudinal maneuvers (decelerate, maintain, accelerate)[cite: 1, 2]. To evaluate a specific dimension independently, we extract the marginal probability distributions by integrating (summing) out the orthogonal dimension. For the theoretical baseline πref _ref, the marginal policy for agent i along dimension d∈lat,lond∈\lat,lon\ is derived as: πref,d(i)(ad(i)|s)=∑ad′∈d′∑−i∈−iπref(ad(i),ad′,−i|s), _ref,d^(i)(a_d^(i)|s)= _a_d _d _ a^-i ^-i _ref(a_d^(i),a_d , a^-i|s), (7) where d∈lat,lon,d∈\lat,lon\, d′∈lat,lon∖dd ∈\lat,lon\ \d\ denotes the complementary action dimension, and −i a^-i represents the joint action of all other agents except i. The marginal distributions for the evaluated MARL policy, πθ,d(i) _θ,d^(i), are obtained analogously. I-E2 Decoupled Optimality Metrics Using the extracted marginal distributions, we independently compute the KL divergence for each dimension and apply the bounded exponential mapping. Definition 7 (Decoupled Optimality Scores). The lateral policy optimality score ℳopt,latM_opt,lat and the longitudinal policy optimality score ℳopt,lonM_opt,lon for the multi-vehicle system are defined as the mean decoupled scores across all agents: ℳopt,d=exp(−1N∑i=1NDKL(πref,d(i)(⋅|s)∥πθ,d(i)(⋅|s))),M_opt,d= (- 1N _i=1^ND_KL ( _ref,d^(i)(·|s) _θ,d^(i)(·|s) ) ), (8) where d∈lat,lond∈\lat,lon\. Remark (Diagnostic Granularity for ”Algorithmic Blindness”): The decoupled metrics ℳopt,latM_opt,lat and ℳopt,lonM_opt,lon equip the diagnostic framework with a powerful “microscope.” While a MARL algorithm might exhibit a seemingly acceptable global ℳoptM_opt, the decoupled scores can expose severe “partial blindness” in specific physical tasks. For instance, it can mathematically diagnose whether a structurally-driven exploration mechanism overly biases the network towards complex lateral lane-changes at the expense of fundamental longitudinal velocity coordination. This granular visibility is indispensable for evaluating specialized network architectures and targeted exploration rewards. I Experimental Setup for Policy Diagnostics To systematically validate the proposed information-theoretic diagnostic framework, we construct a comprehensive evaluation matrix that stress-tests the multi-agent policies across varying degrees of environmental stochasticity and coordination complexity. I-A Continuous Traffic Flow Simulation The diagnostic evaluation is conducted within a high-fidelity continuous traffic flow environment, modeled after a 250-meter segment of a four-lane urban arterial road with a design speed of 60 km/h. Unlike simplified grid-worlds or closed-loop tracks, the continuous injection and departure of vehicles mandate that the MARL policies robustly handle a time-varying number of agents and non-Euclidean interactive topologies. To rigorously diagnose policy degradation boundaries, the evaluation matrix incorporates orthogonal combinations of two critical variables: Traffic Flow Rates: We evaluate the algorithms under three representative flow rates: 400, 600, and 700 pcu/h/ln. These correspond to the Level of Service (LOS) C (stable flow), LOS D (high-density stable flow), and the LOS D/E boundary (critical near-congestion flow), respectively. This variation allows us to diagnose how algorithms compromise safety for efficiency under extreme congestion. CAV Penetration Rates: For each flow rate, the policies are tested under 25%, 50%, 75%, and 100% CAV penetration rates. Lower penetration rates introduce severe partial observability and uncooperative uncertainties from background Human-Driven Vehicles (HDVs), thereby stress-testing the robustness of the MARL algorithms’ spatial reasoning. I-B The Oracle Baseline and Evaluated MARL Configurations I-B1 The MCTS Oracle for Baseline Generation The theoretical ground-truth baseline policy, πref(⋅|s) _ref(·|s), is generated dynamically for a vast dataset of sampled traffic states using a highly optimized Parallel Evaluation MCTS (PE-MCTS) solver. Equipped with a parallel update pruning mechanism and experiential action preferences, the PE-MCTS acts as an oracle (nrollout=1200n_rollout=1200, cpuct=21c_puct=21), generating a near-optimal value landscape QMCTS∗(s,)Q_MCTS^*(s, a) that guarantees strict asymptotic convergence. I-B2 Evaluated MARL Algorithms To demonstrate the diagnostic versatility of the proposed metrics ℳoptM_opt, ℳopt,latM_opt,lat, and ℳopt,lonM_opt,lon, we apply the framework to evaluate a wide spectrum of MARL architectures and exploration mechanisms. The evaluations are divided into two distinct diagnostic applications: Spatial Representation Diagnostics: To evaluate the capacity of different neural architectures in capturing dynamic multi-vehicle topologies, we compare the QMIX baseline integrated with four distinct agent networks: a Multi-Layer Perceptron (MLP), a Convolutional Neural Network (CNN), a Graph Attention Network (GAT), and our developed Representation Network (RepNet), which utilizes a Vision Transformer (ViT) backbone equipped with physical positional encoding. Temporal Exploration Diagnostics: To diagnose how different intrinsic reward mechanisms bias the learned policies, we evaluate state-of-the-art exploration algorithms, including CDS (diversity-driven)[19], SI2E (structural information-driven)[34], MASER (subgoal-driven)[17], SPIE (retrospective information-driven)[33], and our Topology-Enhanced (TPE)[13] algorithm, which utilizes game-theoretic topology and mutual information for guided collaboration. Remark (Standardization of Assessment): All evaluated MARL algorithms adhere to the CTDE paradigm and are trained until convergence under identical environmental configurations. By holding the extrinsic reward structure constant across all experiments, any observed divergence in ℳoptM_opt strictly isolates and quantifies the intrinsic architectural or exploratory deficiencies of the evaluated algorithms. IV Spatial Diagnostics: Uncovering Policy Biases Traditional extrinsic metrics (e.g., success rate, collision rate) merely indicate macro-level task failures, offering zero transparency into the underlying causal mechanisms. By projecting the joint policy into the decoupled optimality space (ℳopt,lat,ℳopt,lon)(M_opt,lat,M_opt,lon), our diagnostic framework acts as a semantic filter, mathematically exposing algorithmic blind spots in spatial reasoning. (a) MLP (b) GAT (c) RepNet Figure 3: Decoupled optimality distribution (ℳopt,lat,ℳopt,lon)(M_opt,lat,M_opt,lon) for different multi-agent representation architectures. Red dots indicate the distribution mean center. IV-A Diagnosing Representation Network Architectures The capability of an agent network to encode non-Euclidean vehicle interactions inherently dictates its policy ceiling. As illustrated in Fig. 3, we diagnose three distinct architectures integrated within the QMIX framework: Multi-Layer Perceptron (MLP)[22], Graph Attention Network (GAT), and ViT-based Representation Network (RepNet)[11]. The Failure of MLP: As shown in Fig. 3LABEL:sub@sfig:diag_mlp, the MLP architecture exhibits severe degradation in both dimensions, with its optimality mean center collapsing to (0.66,0.77)(0.66,0.77). This mathematically proves that simple feature concatenation is insufficient for resolving dynamic spatial interactions. Topological Limits of Graph Networks: The GAT architecture demonstrates marginal improvements (Fig. 3LABEL:sub@sfig:diag_gat), achieving a mean center of (0.74,0.75)(0.74,0.75). While explicitly modeling multi-vehicle interactions as graph edges aids in cross-lane relationship extraction (lateral planning), its longitudinal optimality remains highly constrained. Attention-Based Efficacy: The RepNet architecture (Fig. 3LABEL:sub@sfig:diag_repnet) yields a significantly superior and balanced optimality distribution with a mean center of (0.76,0.84)(0.76,0.84). This confirms that a transformer-based attention mechanism effectively captures both spatial and temporal dependencies required for robust cooperative planning. (a) CDS (b) MASER (c) SI2E (d) SPIE (e) TPE Figure 4: Decoupled optimality diagnosis for SOTA exploration mechanisms versus the proposed Topology-Enhanced (TPE) algorithm. The decoupled metric vividly uncovers the lateral or longitudinal biases inherent in unconstrained exploration paradigms. IV-B Identifying Biases in SOTA Exploration Algorithms Beyond network architectures, the design of intrinsic exploration rewards intrinsically biases the learned policy distribution. We apply the decoupled metric to diagnose four state-of-the-art (SOTA) exploration algorithms against our TPE baseline, as visualized in Fig. 4. The diagnostics reveal profound directional biases: Lateral Bias in Structural Information (SI2E): As shown in Fig. 4(c), SI2E yields a highly asymmetric optimality distribution. It achieves a high lateral score (ℳopt,lat≈0.93M_opt,lat≈ 0.93), yet its longitudinal performance remains dispersed (ℳopt,lon≈0.84M_opt,lon≈ 0.84). This diagnoses a clear algorithmic bias: structurally-driven exploration heavily incentivizes discrete lateral lane-changes but fails to adequately regularize continuous longitudinal velocity coordination. Longitudinal Bias in Retrospective Exploration (SPIE): Conversely, SPIE exhibits the exact inverse pathology (Fig. 4LABEL:sub@sfig:diag_spie). It achieves near-optimal longitudinal execution (ℳopt,lon≈0.95M_opt,lon≈ 0.95) but struggles laterally (ℳopt,lat≈0.84M_opt,lat≈ 0.84). This mathematically isolates the limitation of retrospective information tracking: it excels in optimizing temporally coherent car-following behavior but lacks the spatial horizon necessary for complex lateral negotiation. Lateral Failure in Subgoal-Driven Methods (MASER): MASER exhibits a highly dispersed distribution with a mean center of (0.55,0.81)(0.55,0.81) (Fig. 4LABEL:sub@sfig:diag_maser). While subgoal generation provides some longitudinal guidance, the low lateral score explicitly correlates with its empirically observed low success rates, proving its inadequacy in multi-lane bottleneck resolution. The Pitfall of Unconstrained Diversity (CDS): The CDS algorithm collapses entirely, yielding a mean center of (0.59,0.57)(0.59,0.57) (Fig. 4LABEL:sub@sfig:diag_cds). The metric proves that unconstrained diversity-driven exploration disperses computational resources indiscriminately, preventing convergence to any optimal collaborative policy in either dimension. Remark (Diagnostic Conclusion): In stark contrast to the SOTA baselines, the proposed TPE mechanism achieves a concentrated optimality mean of (0.97,0.99)(0.97,0.99) (Fig. 4LABEL:sub@sfig:diag_tpe). The visual and numerical evidence provided by the decoupled metrics conclusively proves that only a balanced, topology-aware exploration strategy can eliminate directional biases and converge to the true theoretical baseline. V Temporal Diagnostics: Evaluating Dynamic Adaptation Macro-level reward convergence often masks temporal policy degradation. MARL agents can collapse into static, average behaviors that perform adequately in trivial states but fail during complex negotiations. We utilize time-series action heatmaps to diagnose this dynamic adaptation deficiency against the MCTS oracle. (a) RepNet-QMIX (b) TPE Figure 5: Temporal evolution of single-agent policies. The top rows represent the dynamic MCTS baseline, while the bottom rows show the evaluated MARL policy. V-A Identifying the “Average Policy” Trap Visualizing the single-agent policy evolution (Fig. 5) exposes the behavioral rigidity of standard representation networks. The RepNet Collapse: As shown in Fig. 5LABEL:sub@sfig:temp_single_repnet, the RepNet-QMIX policy stagnates into an average policy trap. Its probability mass anchors continuously on Action 4 (maintain speed) and Action 5 (accelerate). It completely fails to track the dynamic state-dependent switching of the MCTS baseline, yielding a mere 33.94%33.94\% single-action match rate. TPE Dynamic Tracking: Conversely, the TPE algorithm (Fig. 5LABEL:sub@sfig:temp_single_tpe) dynamically shifts its probability hotspots in strict alignment with the MCTS oracle. The topology-enhanced exploration effectively prevents behavioral stagnation, elevating the single-action match rate significantly to 69.49%69.49\%. (a) RepNet-QMIX (Joint Policy) (b) TPE (Joint Policy) Figure 6: Temporal evolution of joint policies alongside the continuous tracking of the global optimality score ℳoptM_opt. V-B Quantifying Joint Synergy Fluctuation The temporal tracking of the joint policy (Fig. 6) provides a rigorous assessment of multi-agent synergy. Volatility in Standard MARL: Fig. 6LABEL:sub@sfig:temp_joint_repnet reveals severe structural discrepancies in the RepNet-QMIX joint policy. Its optimality score ℳoptM_opt fluctuates violently between 0.25440.2544 and 0.95690.9569, proving that its occasional high returns are coincidental rather than systematically synergistic. Consequently, its optimal joint action match rate is an abysmal 6.74%6.74\%. Synergistic Stability in TPE: The TPE joint policy (Fig. 6LABEL:sub@sfig:temp_joint_tpe) exhibits high structural fidelity to the MCTS baseline. Its ℳoptM_opt strictly stabilizes above 0.910.91 throughout the entire episodic horizon. The optimal joint action match rate improves significantly to 15.73%15.73\%, confirming that the intrinsic mutual information reward successfully binds individual decisions into coherent, long-horizon collaborative strategies. VI Optimality-Guided Hyperparameter Optimization Traditional hyperparameter tuning relies on extrinsic reward curves, which suffer from high variance and sparse signals, often masking the structural deterioration of policies. By mapping the policy quality into a bounded 2D metric space, our framework transforms hyperparameter tuning into a visual, vector-based trajectory tracking process. (a) Trajectory of β1 _1 (b) Trajectory of β2 _2 Figure 7: Trajectory tracking of the optimality mean centers under varying intrinsic reward weights. The optimal configurations mathematically converge toward the (1,1)(1,1) coordinate. VI-A Tracking the Trajectory of Metric Distribution Centers We apply the decoupled metric to diagnose the sensitivity of the TPE algorithm to its two intrinsic reward weights: topological novelty (β1 _1) and mutual information-based collaboration (β2 _2). Balancing Exploration Breadth (β1 _1): As shown in Fig. 7LABEL:sub@sfig:param_beta1, increasing β1 _1 from 0 to 0.010.01 pulls the optimality mean center drastically from (0.8,0.9)(0.8,0.9) to (0.97,0.99)(0.97,0.99), achieving peak policy quality. However, excessive weight (β1→2.0 _1→ 2.0) induces a severe regression toward the lower-left quadrant. The metric mathematically diagnoses the over-exploration collapse: the algorithm abandons fundamental driving tasks, becoming entirely distracted by the pursuit of novel but meaningless topological states. Balancing Collaborative Exploitation (β2 _2): The trajectory of β2 _2 (Fig. 7LABEL:sub@sfig:param_beta2) provides even deeper diagnostic insights. Increasing β2 _2 from 0 to 0.020.02 successfully shifts the center from (0.65,0.70)(0.65,0.70) to (0.93,0.84)(0.93,0.84). Yet, an excessive weight (β2≥0.2 _2≥ 0.2) triggers a sharp downward drift in longitudinal optimality. This visually captures a collusion trap: agents learn to minimize speed and remain static to artificially stabilize their mutual topologies (maximizing predictability and mutual information), completely sacrificing the extrinsic throughput objective. VII Conclusion This paper introduces an information-theoretic diagnostic framework that transitions MARL evaluation in autonomous driving from extrinsic statistical indicators to intrinsic policy quality. By establishing a rigorous MCTS theoretical baseline and decoupling the KL divergence into lateral and longitudinal dimensions, the proposed metrics (ℳopt,latM_opt,lat, ℳopt,lonM_opt,lon) function as a semantic microscope. The extensive spatial and temporal diagnostics conclusively prove the framework’s capability to expose algorithmic blind spots, identify average-policy traps, and guide optimal hyperparameter configurations. This framework offers a robust, model-agnostic standard for benchmarking future multi-agent robotic systems. References [1] A. Abouelazm, J. Michel, and J. M. Zollner (2024) A review of reward functions for reinforcement learning in the context of autonomous driving. arXiv preprint arXiv:2404.18520. Cited by: §I. [2] A. Chan, H. Silva, S. Lim, T. Kozuno, A. R. Mahmood, and M. White (2022) Greedification operators for policy optimization: investigating forward and reverse kl divergences. J. Mach. Learn. Res. 23, p. 1–61. Cited by: §I. [3] D. Chen, M. R. Hajidavalloo, Z. Li, K. Chen, Y. Wang, L. Jiang, and Y. Wang (2023) Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic. IEEE Trans. Intell. Transp. Syst.. Cited by: §I, §I. [4] S. Chen, J. Dong, P. Y. J. Ha, Y. Li, and S. Labi (2021) Graph neural network and reinforcement learning for multi-agent cooperative control of connected autonomous vehicles. Comput. Aided Civ. Inf. Eng. 36 (9), p. 1183–1197. Cited by: §I, §I. [5] H. Dou, L. Dang, Z. Luan, and B. Chen (2024) Measuring mutual policy divergence for multi-agent sequential exploration. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), Cited by: §I. [6] C. Glanois, P. Weng, M. Zimmer, D. Li, T. Yang, J. Hao, and W. Liu (2024) A survey on interpretable reinforcement learning. Mach. Learn. 113 (8), p. 5229–5288. Cited by: §I. [7] A. Guillen-Perez and M. Cano (2022) Multi-agent deep reinforcement learning to manage connected autonomous vehicles at tomorrow’s intersections. IEEE Trans. Intell. Transp. Syst.. Cited by: §I, §I. [8] L. Guo, L. Liu, J. Tang, B. Liu, and S. Cao (2025) OpenCDA-marl: a unified benchmarking framework for cooperative autonomous intersection management with multi-agent reinforcement learning. IEEE Trans. Intell. Transp. Syst.. Cited by: §I, §I. [9] Z. Guo, Y. Wu, L. Wang, and J. Zhang (2024) Heuristic-based multi-agent deep reinforcement learning approach for coordinating connected and automated vehicles at non-signalized intersection. IEEE Trans. Veh. Technol.. Cited by: §I. [10] S. Han, S. Zhou, J. Wang, L. Pepin, C. Ding, J. Fu, and F. Miao (2022) A multi-agent reinforcement learning approach for safe and efficient behavior planning of connected autonomous vehicles. IEEE Trans. Intell. Transp. Syst.. Cited by: §I. [11] Y. Han, L. Zhang, D. Meng, X. Hu, and Y. Lu (2024) SPformer: a transformer based drl decision making method for connected automated vehicles. In Proc. IEEE Int. Conf. Intell. Transp. Syst., Vol. , p. 1223–1230. Cited by: §IV-A. [12] Y. Han, L. Zhang, D. Meng, Z. Zhang, X. Hu, and S. Weng (2026) A value-based parallel update mcts method for multi-agent cooperative decision-making of connected and automated vehicles. IEEE Trans. Intell. Transp. Syst. 27 (1), p. 1400–1415. Cited by: §I-B1. [13] Y. Han, L. Zhang, D. Meng, and Z. Zhang (2025) Topology enhanced marl for multi-agent cooperative decision-making of cavs. arXiv preprint arXiv:2507.12110. Cited by: §I-B2. [14] F. Helfenstein, J. Czech, J. Blüml, M. Eisel, and K. Kersting (2024) Checkmating one, by using many: combining mixture of experts with mcts to improve in chess. IEEE Trans. Games. Cited by: §I. [15] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger (2018) Deep reinforcement learning that matters. In Proc. AAAI Conf. Artif. Intell. (AAAI), Vol. 32. Cited by: §I. [16] M. Hua, X. Qi, D. Chen, K. Jiang, Z. E. Liu, H. Sun, Q. Zhou, and H. Xu (2024) Multi-agent reinforcement learning for connected and automated vehicles control: recent advancements and future prospects. IEEE Trans. Intell. Transp. Syst.. Cited by: §I. [17] J. Jeon, W. Kim, W. Jung, and Y. Sung (2022-17–23 Jul) MASER: multi-agent reinforcement learning with subgoals generated from experience replay buffer. In Proc. 39th Int. Conf. Mach. Learn., Vol. 162, p. 10041–10052. Cited by: §I-B2. [18] S. M. Jordan, Y. Chandak, D. Cohen, M. Zhang, and P. S. Thomas (2020) Evaluating the performance of reinforcement learning algorithms. J. Mach. Learn. Res. (JMLR) 21, p. 1–34. Cited by: §I, §I. [19] C. Li, T. Wang, C. Wu, Q. Zhao, J. Yang, and C. Zhang (2021) Celebrating diversity in shared multi-agent reinforcement learning. In Adv. Neural Inf. Process. Syst., Vol. 34, p. 3991–4002. Cited by: §I-B2. [20] Y. Li (2025) Hierarchical reward function design for autonomous driving based on reinforcement learning. In Proc. Int. Symp. Machine Learning and Social Computing (MLSC), Cited by: §I. [21] T. Liu, W. Xiong, J. Ren, L. Chen, J. Wu, R. Joshi, Y. Gao, J. Shen, Z. Qin, T. Yu, D. Sohn, A. Makarova, J. Liu, Y. Liu, B. Piot, A. Ittycheriah, A. Kumar, and M. Saleh (2025) RRM: robust reward model training mitigates reward hacking. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I. [22] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson (2018) QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Proc. 35th Int. Conf. Mach. Learn., Vol. 80, p. 4295–4304. Cited by: §IV-A. [23] M. A. Saleem, S. Zhou, A. Quraishi, M. Shabaz, I. Javed, S. Basheer, A. F. Alenezi, and H. Aldawsari (2024) Multi-agent reinforcement learning for real-time adaptive lane control in mixed cav-hdv freeways traffic. IEEE Trans. Consum. Electron.. Cited by: §I, §I. [24] J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel (2015) Trust region policy optimization. In Proc. Int. Conf. Mach. Learn. (ICML), p. 1889–1897. Cited by: §I. [25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §I. [26] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis (2017) Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. Cited by: §I. [27] M. Świechowski, K. Godlewski, B. Sawicki, and J. Mańdziuk (2023) Monte carlo tree search: a review of recent modifications and applications. Artif. Intell. Rev. 56 (3), p. 2497–2562. Cited by: §I. [28] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver (2019) Grandmaster level in starcraft i using multi-agent reinforcement learning. Nature 575 (7782), p. 350–354. Cited by: §I. [29] S. Wang, Z. Wang, R. Jiang, F. Zhu, R. Yan, and Y. Shang (2023) A multi-agent reinforcement learning-based longitudinal and lateral control of cavs to improve traffic efficiency in a mandatory lane change scenario. Transp. Res. Part C Emerg. Technol.. Cited by: §I, §I. [30] Z. Wang, Y. Xue, L. Liu, H. Zhang, C. Qu, and C. Fang (2024) Multi-agent drl-controlled connected and automated vehicles in mixed traffic with time delays. IEEE Trans. Intell. Transp. Syst.. Cited by: §I. [31] Y. Xu, Y. Shi, X. Tong, S. Chen, and Y. Ge (2024) A multi-agent reinforcement learning based control method for cavs in a mixed platoon. IEEE Trans. Intell. Veh.. Cited by: §I, §I. [32] R. Yan, X. Duan, Z. Shi, Y. Zhong, J. R. Marden, and F. Bullo (2024) Policy evaluation and seeking for multiagent reinforcement learning via best response. IEEE Trans. Autom. Control. Cited by: §I. [33] C. Yu, N. Burgess, M. Sahani, and S. J. Gershman (2023) Successor-predecessor intrinsic exploration. In Adv. Neural Inf. Process. Syst., Cited by: §I-B2. [34] X. Zeng, H. Peng, and A. Li (2024) Effective exploration based on the structural information principles. In Adv. Neural Inf. Process. Syst., Cited by: §I-B2. [35] Z. Zhang, S. Han, J. Wang, and F. Miao (2023) Spatial-temporal-aware safe multi-agent reinforcement learning of connected autonomous vehicles in challenging scenarios. IEEE Trans. Intell. Transp. Syst.. Cited by: §I. [36] M. Zhou, J. Luo, J. Villella, Y. Yang, D. Rusu, J. Miao, W. Zhang, M. Alban, I. Fadakar, Z. Chen, et al. (2020) SMARTS: scalable multi-agent reinforcement learning training school for autonomous driving. In Proc. Conf. Robot Learn. (CoRL), p. 264–285. Cited by: §I, §I.