Paper deep dive
Offline Multi-Agent Reinforcement Learning with a Physics-Informed World Model for Cooperative Mixed Traffic Control
Lu Liu, Chi Xie, Xi Xiong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/19/2026, 3:48:37 AM
Summary
This paper proposes a physics-informed world model-based offline multi-agent reinforcement learning (MARL) framework for cooperative control of connected and automated vehicles (CAVs) in mixed traffic at highway bottlenecks. The framework addresses partial observability by reconstructing a global traffic state from local CAV observations using a Transformer-based encoder and macroscopic-microscopic traffic dynamics as physics-based supervision. A probabilistic ensemble world model learns state transitions and rewards, quantifying epistemic uncertainty to enable safe offline policy learning via pessimistic imagined rollouts.
Entities (10)
Relation Signals (7)
Physics-Informed World Model → appliedto → Connected and Automated Vehicles
confidence 95% · cooperative control of connected and automated vehicles (CAVs)
Physics-Informed World Model → uses → Offline Multi-Agent Reinforcement Learning
confidence 95% · We propose a physics-informed world model-based offline multi-agent reinforcement learning framework
Physics-Informed World Model → incorporates → Lighthill-Whitham-Richards Model
confidence 92% · coupled macroscopic-microscopic traffic dynamics providing physics-based supervision... Lighthill-Whitham-Richards (LWR) conservation law
Physics-Informed World Model → incorporates → Cell Transmission Model
confidence 92% · discretized over the cell partition using the cell transmission model (CTM)
Physics-Informed World Model → handles → Epistemic Uncertainty
confidence 90% · model disagreement quantifies epistemic uncertainty... uncertainty-driven truncation
SUMO → usedforsimulationof → Mixed Traffic Bottleneck
confidence 90% · Experiments in a SUMO-based on-ramp bottleneck
Transformer → usedin → Physics-Informed World Model
confidence 88% · a Transformer-based history encoder Enc_phi is employed to extract the local belief representation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This study investigates cooperative control of connected and automated vehicles (CAVs) at partially observable highway bottlenecks in mixed traffic, aiming to mitigate congestion without relying on complete global traffic states or online trial-and-error. We propose a physics-informed world model-based offline multi-agent reinforcement learning framework that reconstructs a physically interpretable global traffic state from local CAV observation-action histories, with coupled macroscopic-microscopic traffic dynamics providing physics-based supervision. A probabilistic ensemble world model learns traffic-state transitions and system rewards, while model disagreement quantifies epistemic uncertainty. Multi-step imagined rollouts with pessimistic rewards and uncertainty-driven truncation are then used for offline policy learning. Experiments in a SUMO-based on-ramp bottleneck using approximately $1\times10^6$ offline transitions show that physics supervision improves state reconstruction and world-model prediction accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2608.17739v1
- Canonical: https://arxiv.org/abs/2608.17739v1
Trouble viewing inline? Open PDF directly →
Full Text
58,447 characters extracted from source content.
Expand or collapse full text
Lu LiuCollege of Transportation, Tongji Universityluliu0720@tongji.edu.cn[Shanghai, China, 201804] Chi XieCollege of Transportation, Tongji Universitychi.xie@tongji.edu.cn[Shanghai, China, 201804] *Xi XiongCollege of Transportation, Tongji Universityxi_xiong@tongji.edu.cn[Shanghai, China, 201804] Lu Liu, Chi Xie, and Xi Xiong Offline Multi-Agent Reinforcement Learning with a Physics-Informed World Model for Cooperative Mixed Traffic Control 1 Abstract This study investigates cooperative control of connected and automated vehicles (CAVs) at partially observable highway bottlenecks in mixed traffic, aiming to mitigate congestion without relying on complete global traffic states or online trial-and-error. We propose a physics-informed world model-based offline multi-agent reinforcement learning framework that reconstructs a physically interpretable global traffic state from local CAV observation-action histories, with coupled macroscopic-microscopic traffic dynamics providing physics-based supervision. A probabilistic ensemble world model learns traffic-state transitions and system rewards, while model disagreement quantifies epistemic uncertainty. Multi-step imagined rollouts with pessimistic rewards and uncertainty-driven truncation are then used for offline policy learning. Experiments in a SUMO-based on-ramp bottleneck using approximately 1×1061× 10^6 offline transitions show that physics supervision improves state reconstruction and world-model prediction accuracy. Introduction With the rapid development of automated driving and vehicular networking technologies, connected and automated vehicles (CAVs) are evolving from passive traffic participants into active control agents. Unlike conventional traffic control strategies that rely on roadside infrastructure, CAVs can directly regulate surrounding driving behavior and local traffic evolution through their own control actions, providing a new approach for link-level traffic flow control 17. This capability is particularly important for highway bottlenecks, such as on-ramp merging and lane-drop areas, where intensified vehicle interactions and capacity variations can amplify local speed disturbances, leading to queue formation and capacity degradation 4. Although recent studies have explored cooperative CAV control to improve traffic efficiency, most of them assume that the global traffic state is available or that the observations of all CAVs can collectively cover the entire traffic system. This assumption is difficult to satisfy at highway bottlenecks, where CAVs are constrained by limited sensing and communication ranges and can only observe partial surrounding traffic conditions. To enable effective coordination among multiple CAVs for mitigating traffic congestion, this paper proposes a physics-informed world model-based offline multi-agent reinforcement learning framework for mixed traffic at bottlenecks with limited global visibility. As shown in Fig. 1, we consider a typical on-ramp merging scenario where human-driven vehicles (HDVs) and CAVs coexist. The blue shaded area denotes the CAV control zone upstream of the merging point. Once entering this zone, each CAV obtains local traffic information within a limited range through onboard sensors and vehicle-to-everything (V2X) communication. Based on these partial observations and historical interactions, CAVs execute longitudinal acceleration control to actively regulate upstream traffic flow. In contrast to fully automated traffic, mixed traffic involves highly stochastic car-following and merging behaviors of HDVs, making the traffic response to CAV control difficult to predict 27. Therefore, online trial-and-error learning on real roads may introduce potential safety risks. To address this issue, this paper focuses on learning a world model from historical observation-action data of CAVs to capture bottleneck traffic evolution and support reliable CAVs cooperative control. Figure 1: Mixed-traffic on-ramp bottleneck with the CAV control zone Early efforts in CAV control were dominated by rule-based approaches, which designed control policies based on vehicle spacing, relative speed, and leading-vehicle motion information 16. To extend CAV control from individual vehicle regulation to traffic-level coordination, many studies have combined macroscopic-microscopic coupled models with model predictive control 19. However, the performance of these methods depends on the modeling accuracy of driving behaviors and traffic demand variations 14. Moreover, as the number of CAVs increases, the optimization scale grows rapidly, leading to substantial communication and computational burdens 8. Data-driven methods have therefore attracted increasing attention, among which reinforcement learning (RL) has shown strong potential for cooperative CAV control by learning from interaction feedback 25; 15. Online policy exploration in real traffic systems may induce unsafe driving behaviors and incur prohibitive costs 3. Offline RL, which learns from pre-collected datasets without continual environment interaction during training, has therefore emerged as a promising direction. Nevertheless, offline RL suffers from distribution shift, as poorly covered states or actions may lead to suboptimal or unsafe decisions 20. For example, Fang et al. deployed offline RL using real-world driving data and found that limited data coverage led to unsatisfactory CAV control performance 7. Although policy constraints and value regularization have been widely used to improve policy reliability 10, CAVs coordination introduces additional challenges: even when each agent selects an in-distribution action, the resulting joint action may still be out of distribution. Moreover, without interactive feedback, effective multi-agent coordination is difficult to guarantee 29. As a result, offline RL has seen limited application in CAV control. World model-based offline RL offers a promising way to address these limitations. By learning environment dynamics and generating imagined experience, these methods can improve sample efficiency and enhance lookahead prediction capability 23. To mitigate model exploitation in offline settings, existing studies often incorporate uncertainty estimation, pessimistic rewards, or penalties for uncertain regions 26. However, most world model methods are designed for general control tasks or single-vehicle decision-making and typically rely on relatively complete environment states 12. Such an assumption is often difficult to satisfy in practice because of limited sensing and communication capabilities. Therefore, some studies directly concatenate the local observations of multiple CAVs to construct an approximate global state for centralized training 21. However, such representations may introduce substantial information redundancy, and their dimensionality increases with the number of CAVs. Other studies learn low-dimensional latent states to obtain more compact system representations 9. Although latent representations can reduce the modeling complexity, they typically lack explicit traffic-physical semantics, potentially leading to predictions that are inconsistent with actual traffic evolution. In addition, multi-agent systems involve larger joint state-action spaces, making it difficult for finite offline datasets to sufficiently cover the state-action regions that may be visited by the learned policy 2. Under such distribution shifts, the policy may exploit prediction errors of the world model in poorly covered regions, thereby generating unreliable imagined experiences 13. Based on the above analysis, applying world models to cooperative multi-CAV control in partially observable mixed traffic still faces three interrelated challenges: how to construct an effective global state that captures overall traffic evolution from distributed local observations, how to improve the consistency of the learned world model with traffic-flow physics, and how to mitigate the impact of model prediction errors on imagined policy learning under limited offline data coverage. This paper proposes a physics-informed world model-based offline multi-agent reinforcement learning framework for mixed-traffic bottlenecks. First, temporal traffic information is extracted from the local observation-action histories of multiple CAVs, and a structured global traffic state consisting of traffic density, average speed, and microscopic CAV states is reconstructed through traffic-cell-based feature aggregation. Unlike directly concatenating local observations or learning latent states without explicit physical semantics, the proposed method incorporates a coupled macroscopic-microscopic traffic model to provide physics-based consistency supervision for state reconstruction, enabling the reconstructed state to explain local observations while better reflecting traffic-flow evolution. Based on this reconstructed state, a probabilistic ensemble world model is developed to learn traffic-state transitions and system rewards under joint CAV control, while epistemic uncertainty is quantified through prediction disagreement among ensemble members. Multi-step imagined rollouts are then performed in the learned traffic world model, together with uncertainty-aware pessimistic rewards and rollout truncation, to reduce the risk of exploiting unreliable model predictions in poorly covered regions. Finally, cooperative multi-CAV policies are learned under the centralized training with decentralized execution (CTDE) paradigm, allowing each CAV to make decisions based solely on its own local observation history during execution, without online trial-and-error or access to the true global traffic state. A mixed-traffic on-ramp scenario is constructed using simulation of urban mobility (SUMO) to evaluate the proposed method in terms of world-model prediction accuracy, cooperative control performance, and sensitivity to different traffic demands and CAV penetration rates. The remainder of this paper is organized as follows. Section 2 introduces the coupled macroscopic-microscopic traffic model and the Dec-POMDP formulation. Section 3 presents the proposed physics-informed world model-based offline multi-agent reinforcement learning method. Section 4 describes the experimental setup and analyzes the results. Finally, Section 5 concludes the paper. 2 Preliminaries 2.1 Decentralized POMDP Formulation The cooperative control problem of multiple CAVs in mixed-autonomy traffic is formulated as a decentralized partially observable Markov decision process (Dec-POMDP) 1, represented by the tuple ⟨,S,A,P,R,Ω,O,γ⟩ ,S,A,P,R, ,O,γ . =1,2,…,NN=\1,2,…,N\ denotes the set of CAVs operating within the coordination region. The road segment within the coordination region is partitioned into M homogeneous cells with length Δx x, indexed by J=1,2,…,M\J\=\1,2,…,M\. The global state space is denoted by S, and at time t, the system state s(t)∈Ss(t)∈ S characterizes the underlying traffic state of the mixed-autonomy system. Specifically, the global state is defined as s(t)=(ρ(t),v¯(t),e(t))s(t)= (ρ(t), v(t),e(t) ), where ρ(t)=ρ1(t),ρ2(t),…,ρM(t)ρ(t)=\ _1(t), _2(t),…, _M(t)\ and v¯(t)=v¯1(t),v¯2(t),…,v¯M(t) v(t)=\ v_1(t), v_2(t),…, v_M(t)\ denote the traffic density and average speed of the M road cells, respectively. The term e(t)e(t) contains the microscopic states of all CAVs, including their positions and velocities. The joint action space is defined as A=∏i∈AiA= _i A_i, where AiA_i is the action space of CAV i. In this study, the action of each CAV is represented by a continuous longitudinal acceleration command ai(t)∈Ai=[amin,amax]a_i(t)∈ A_i=[a_ ,a_ ], where amina_ and amaxa_ denote the minimum and maximum admissible accelerations. The actions of all CAVs jointly form the action vector a(t)=ai(t)i∈a(t)=\a_i(t)\_i . The stochastic evolution of the traffic system is characterized by the transition function P:S×A×S→[0,1]P:S× A× S→[0,1], which captures the combined effects of traffic-flow propagation, CAV control actions, and stochastic human-driving behaviors. Let Ω=∏i∈Ωi = _i _i denote the joint observation space, and let O=Oii∈O=\O_i\_i denote the set of local observation functions. For each CAV i, the observation function maps the global traffic state to its local observation, oi(t)=Oi(s(t))o_i(t)=O_i(s(t)). The local observation of CAV i is defined as oi(t)=⟨xi(t),vi(t),εisurr(t)⟩o_i(t)= x_i(t),v_i(t), _i^surr(t) , where xi(t)x_i(t) and vi(t)v_i(t) denote the position and velocity of CAV i, respectively, and εisurr(t) _i^surr(t) represents the surrounding traffic information perceived within its sensing or communication range. The observations of all CAVs constitute the joint observation vector o(t)=oi(t)i∈o(t)=\o_i(t)\_i . The parameter γ∈[0,1)γ∈[0,1) is the discount factor. All CAVs share a common team reward function R:S×A→ℝR:S× A . The immediate reward is designed to improve system-level traffic efficiency while suppressing excessive control actions, r(t)=R(s(t),a(t))=w1∑j=1Mρj(t)v¯j(t)−w2∑i∈ai2(t), r(t)=R(s(t),a(t))=w_1 _j=1^M _j(t) v_j(t)-w_2 _i a_i^2(t), (1) where the first term approximates the aggregated traffic flow over all road cells, and the second term penalizes aggressive acceleration or deceleration. The parameters w1>0w_1>0 and w2>0w_2>0 balance traffic efficiency and driving smoothness. At each decision step t, CAV i does not have access to the complete global state s(t)s(t). Instead, it selects its control action based on its local interaction history τi(t)=(oi(t−H:t),ai(t−H:t−1)) _i(t)=(o_i(t-H:t),a_i(t-H:t-1)), according to a decentralized policy, ai(t)∼πi(⋅∣τi(t))a_i(t) _i(· _i(t)). After the joint action a(t)a(t) is executed, the traffic system evolves to the next state s(t+1)s(t+1) according to P(s(t+1)∣s(t),a(t))P(s(t+1) s(t),a(t)), and a team reward r(t)=R(s(t),a(t))r(t)=R(s(t),a(t)) is generated. The objective is to learn the optimal decentralized policies π∗=πi∗i∈π^*=\ _i^*\_i that maximize the expected cumulative discounted return, π∗=argmaxππ[∑t=0∞γtr(t)]. π^*= _πE_π [ _t=0^∞γ^tr(t) ]. (2) 2.2 Traffic Flow Dynamics Prior To ensure the physical consistency of state inference and traffic evolution prediction, a macroscopic-microscopic coupled traffic flow model is introduced as a physics-informed prior. At the macroscopic level, the mainstream traffic dynamics are described by the Lighthill-Whitham-Richards (LWR) conservation law 22, ∂tρ(x,t)+∂xq(x,t)=0, _tρ(x,t)+ _xq(x,t)=0, (3) q(x,t)=Q(ρ(x,t))=ρ(x,t)v(x,t), q(x,t)=Q(ρ(x,t))=ρ(x,t)v(x,t), (4) where ρ(x,t)ρ(x,t), v(x,t)v(x,t), and q(x,t)q(x,t) denote the traffic density, traffic speed, and traffic flow at location x and time t, respectively. The function Q(ρ)Q(ρ) denotes the traffic flow fundamental diagram. In this study, a triangular fundamental diagram 18 is adopted to characterize the relationship between traffic density and flow. The corresponding equilibrium speed is given by, v(x,t)=V(ρ(x,t))=Q(ρ(x,t))ρ(x,t). v(x,t)=V(ρ(x,t))= Q(ρ(x,t))ρ(x,t). (5) For computational implementation, the continuous traffic-flow model is discretized over the cell partition J using the cell transmission model (CTM) 6. For each cell j∈j , the density evolution is given by, ρj(t+1)=ρj(t)+ΔtΔx(qj−1(t)−qj(t)+rjon(t)−rjoff(t)), _j(t+1)= _j(t)+ t x (q_j-1(t)-q_j(t)+r_j^on(t)-r_j^off(t) ), (6) where qj−1(t)q_j-1(t) and qj(t)q_j(t) denote the upstream inflow and downstream outflow of cell j, respectively. The terms rjon(t)r_j^on(t) and rjoff(t)r_j^off(t) represent the on-ramp inflow and off-ramp outflow. The boundary flow between two adjacent cells is determined by the demand and supply functions 11, qj(t)=minDj(t),Sj+1(t), q_j(t)= \D_j(t),S_j+1(t)\, (7) Dj(t)=minVfρj(t),qmax, D_j(t)= \V_f _j(t),q_ \, (8) Sj+1(t)=minw(ρmax−ρj+1(t)),qmax, S_j+1(t)= \w( _ - _j+1(t)),q_ \, (9) where, VfV_f denotes the free-flow speed, w denotes the backward wave speed, ρmax _ denotes the jam density, and qmaxq_ denotes the maximum flow capacity. Accordingly, the cell physical speed prior is given by v¯j(t+1)=V(ρj(t+1)). v_j(t+1)=V( _j(t+1)). (10) At the microscopic level, the motion of each CAV is described by a discrete-time kinematic model, vi(t+1)=clip(vi(t)+ai(t)Δt,0,Vmax), v_i(t+1)=clip (v_i(t)+a_i(t) t,0,V_ ), (11) xi(t+1)=xi(t)+vi(t)Δt, x_i(t+1)=x_i(t)+v_i(t) t, (12) where xi(t)x_i(t), vi(t)v_i(t), and ai(t)a_i(t) denote the position, velocity, and acceleration command of CAV i, respectively. The operator clip(⋅)clip(·) ensures that the updated speed remains within the admissible speed range. To capture the impact of CAV control on local traffic flow dynamics, CAVs are modeled as controllable moving disturbances embedded in the macroscopic traffic stream. Their longitudinal control actions affect local car-following behavior, gap formation, and traffic flow propagation. Accordingly, a CAV-induced flow correction term is introduced into the CTM boundary flow, qj(t)=minDj(t),Sj+1(t)+ΔqjCAV(t), q_j(t)= \D_j(t),S_j+1(t)\+ q_j^CAV(t), (13) where ΔqjCAV(t) q_j^CAV(t) represents the aggregate impact of nearby CAVs. A positive value of ΔqjCAV(t) q_j^CAV(t) indicates an improvement in local throughput induced by cooperative CAV control, whereas a negative value indicates reduced local traffic throughput caused by traffic disturbances. The flow correction term is computed according to the moving-bottleneck Riemann formulation in the literature 5. By combining the macroscopic traffic evolution and microscopic CAV dynamics, the physics-based state transition can be compactly written as s~(t+1)=Fphy(s(t),a(t)). s(t+1)=F_phy(s(t),a(t)). (14) This physics-based transition serves as a structural prior rather than a complete replacement for the learned world model. It provides physical guidance for global state inference and future traffic evolution prediction, ensuring that the learned model remains consistent with traffic conservation principles. 3 Methodology 3.1 Framework Overview In this section, we propose a physics-informed world model-based offline multi-agent reinforcement learning framework for cooperative control of multiple CAVs in mixed traffic. As illustrated in Fig. 2, the proposed framework consists of two main stages. In the first stage, a traffic world model is trained using offline interaction data. By integrating traffic flow physics priors and epistemic uncertainty, the learned world model recovers physically meaningful global traffic states from local observations and predicts both traffic state evolution and system-level rewards. In the second stage, world model-based imagination rollout and policy learning are conducted. Starting from historical states sampled from the offline dataset, the policy performs multi-step virtual interactions within the learned world model to generate additional imagined trajectories. These trajectories are then used to optimize cooperative multi-agent control policies under the centralized training with decentralized execution (CTDE) paradigm. Figure 2: A traffic world model learning framework integrating traffic flow physics priors and epistemic uncertainty 3.2 Physics-Informed Global Traffic State Inference In this subsection, we develop a physically meaningful global traffic state inference module that recovers global traffic states from multi-vehicle historical observations and actions, thereby providing a state foundation for world model learning and centralized policy training. For each CAV i located within the cooperative control region at time t, the local interaction history τi(t) _i(t) is used to characterize its recent motion, surrounding traffic evolution, and control response information. Then, a Transformer-based history encoder EncϕEnc_φ is employed to extract the local belief representation of each CAV, hi(t)=Encϕ(τi(t)), h_i(t)=Enc_φ ( _i(t) ), (15) where ϕφ denotes the learnable parameters. In this paper, all CAVs share the same encoder parameters, which enables the model to accommodate dynamic traffic scenarios where vehicles may enter or leave the control region and improves its generalization capability under different traffic demands and CAV penetration rates. By aggregating the local beliefs of all CAVs, the proposed module further recovers a global traffic representation according to the spatial structure of the road. Specifically, each road cell is regarded as a spatial query unit. For each cell j∈j , a cross-attention mechanism is used to adaptively aggregate the information relevant to that cell from the local beliefs of all CAVs, αi,j(t)=exp((ηjWQ)(hi(t)WK)⊤d)∑i′=1Nexp((ηjWQ)(hi′(t)WK)⊤d), _i,j(t)= ( ( _jW_Q)(h_i(t)W_K) d ) _i =1^N ( ( _jW_Q)(h_i (t)W_K) d ), (16) zj(t)=∑i=1Nαi,j(t)(hi(t)WV), z_j(t)= _i=1^N _i,j(t) (h_i(t)W_V ), (17) where ηj _j is the learnable spatial query vector associated with cell j, and WQW_Q, WKW_K, and WVW_V are learnable projection matrices. The attention weight αi,j(t) _i,j(t) quantifies the degree to which cell j extracts information from CAV i, while zj(t)z_j(t) represents the inferred state of cell j. The global traffic representation is denoted as Z(t)=[z1(t),…,zM(t)]Z(t)= [z_1(t),…,z_M(t) ]. To enable the world model to learn traffic evolution dynamics in a state space that is consistent with traffic flow mechanisms, we further employ multilayer perceptrons (MLPs) to map the traffic representations into macroscopic traffic variables with clear physical meanings. These variables include cell density and average speed, which are essential for characterizing congestion propagation and flow variations, ρ^j(t)=MLPρ(zj(t)),v^j(t)=MLPv(zj(t)). ρ_j(t)=MLP_ρ (z_j(t) ), v_j(t)=MLP_v (z_j(t) ). (18) The recovered density and average speed over the entire road segment are represented as ρ^(t)=(ρ^1(t),…,ρ^M(t)) ρ(t)= ( ρ_1(t),…, ρ_M(t) ) and v^(t)=(v^1(t),…,v^M(t)) v(t)= ( v_1(t),…, v_M(t) ), respectively. The density prediction head uses a Softplus activation function to ensure non-negativity, while the speed prediction head uses a Sigmoid mapping scaled to the free-flow speed range [0,V][0,V]. In addition to macroscopic traffic states, we extract the individual state representation ei(t)e_i(t) of each CAV from its local observation oi(t)o_i(t), and denote the individual states of all CAVs as e(t)=(e1(t),…,eN(t))e(t)= (e_1(t),…,e_N(t) ). The recovered global traffic state is then constructed as s^(t)=[ρ^(t),v^(t),e(t)] s(t)= [ ρ(t), v(t),e(t) ]. To ensure the validity and interpretability of the inference module, an observation reconstruction constraint is introduced. Specifically, an observation decoder OψO_ψ is constructed to map the recovered global state back to the corresponding local observation of each CAV i, o^i(t)=Oψ(s^(t),i), o_i(t)=O_ψ ( s(t),i ), (19) where ψ denotes the parameters of the observation decoder. The observation reconstruction loss is defined as, ℒobs=∑i=1N‖o^i(t)−oi(t)‖2, _obs= _i=1^N \| o_i(t)-o_i(t) \|^2, (20) where oi(t)o_i(t) is the truth observation from the offline dataset. This loss encourages the recovered global state to explain the local observations of all CAVs. Moreover, after training, the observation decoder is used in the subsequent imagination rollout process, providing inputs for decentralized policy execution. Relying solely on observation reconstruction may allow the recovered state to numerically explain local observations, but it does not necessarily guarantee consistency with real traffic flow evolution. Therefore, a traffic flow physics model is introduced as a soft constraint. Given the recovered state s^(t) s(t) and the joint control action a(t)a(t), the traffic flow physics model Fphy(⋅)F_phy(·) provides a physics-based reference state for the next time step, sphy(t+1)=Fphy(s^(t),a(t)). s^phy(t+1)=F_phy ( s(t),a(t) ). (21) The physical consistency loss is defined as, ℒphy=‖s^(t+1)−sphy(t+1)‖2. _phy= \| s(t+1)-s^phy(t+1) \|^2. (22) It should be noted that the physics model is not treated as the state transition function in this paper. Instead, it is used as a traffic flow prior to regularize the state inference process. This design constrains the learned state within a physically reasonable range defined by traffic conservation laws and fundamental diagram relationships, thereby improving the interpretability of state recovery and the stability of the constructed world model. Overall, the training objective of the global traffic state inference module is formulated as, ℒstate=λ1ℒobs+λ2ℒphy, _state= _1L_obs+ _2L_phy, (23) where λ1>0 _1>0 and λ2>0 _2>0 are the weighting coefficients. 3.3 Probabilistic Ensemble Traffic World Model This subsection focuses on learning the evolution dynamics of the traffic system in the state–action space. In mixed traffic, future traffic states are uncertain due to the stochastic behavior of human-driven vehicles (HDVs) and the control actions of CAVs. Meanwhile, offline datasets cannot cover all possible state–action combinations, and a single predictive model may suffer from large errors in sparsely sampled regions. To address these issues, we develop a probabilistic ensemble traffic world model to jointly predict state transitions, estimate system-level rewards, and quantify epistemic uncertainty. Specifically, we initialize K MLPs with the same network architecture but independent parameters, denoted as ℳθkk=1KM _kk=1^K. Each MLP k takes the recovered global traffic state s^(t) s(t) and the joint action a(t)a(t) as inputs, and predicts the distribution of the next global traffic state, s^(t+1)∼(μθk(s^(t),a(t)),diag(σθk2(s^(t),a(t)))), s(t+1) ( _ _k ( s(t),a(t) ),diag ( _ _k^2 ( s(t),a(t) ) ) ), (24) where μθk(s^(t),a(t))∈ℝds _ _k ( s(t),a(t) ) ^d_s and σθk(s^(t),a(t))∈ℝ>0ds _ _k ( s(t),a(t) ) _>0^d_s denote the mean vector and standard deviation vector of the predicted state transition distribution, respectively. dsd_s is the dimension of the global traffic state vector. Meanwhile, the MLP k also outputs the corresponding immediate reward prediction, Rθk(s^(t),a(t),s^(t+1)), R_ _k ( s(t),a(t), s(t+1) ), (25) where RθkR_ _k denotes the reward prediction head. To enhance the diversity among different MLPs and prevent all models from converging to similar solutions, which may invalidate uncertainty estimation, we introduce a bootstrap-based data perturbation mechanism. At each parameter update, a mini-batch of size B, denoted as (o(b)(t),a(b)(t),o(b)(t+1))b=1B \ (o^(b)(t),a^(b)(t),o^(b)(t+1) ) \_b=1^B, is randomly sampled from the offline dataset D, and the corresponding global traffic states are inferred using the state inference module, where b denotes the sample index in the mini-batch. For the MLP k and the training sample b, an independent Bernoulli mask is sampled as, mk(b)∼Bern(p),b=1,…,B,k=1,…,K, m_k^(b) (p), b=1,…,B, k=1,…,K, (26) where p denotes the sample retention probability. When mk(b)=1m_k^(b)=1, the MLP k uses this sample for parameter updating; otherwise, when mk(b)=0m_k^(b)=0, this sample is excluded from gradient computation for the MLP k. This mechanism allows different MLPs to observe different subsets of data during each training iteration, thereby inducing diverse generalization behaviors in the learned state transition functions. For learning the probabilistic state transition distribution of each MLP, we adopt the negative log-likelihood (NLL) loss as the training objective. Let the target state be y=sg(s^(t+1))y=sg ( s(t+1) ), where sg(⋅)sg(·) denotes the stop-gradient operation. The state prediction loss for a single sample is defined as, ℒdyn=12∑d=1ds[(yd−μθkd)2(σθkd)2+log(σθkd)2]+ds2log(2π), _dyn= 12 _d=1^d_s [ (y^d- _ _k^d )^2 ( _ _k^d )^2+ ( _ _k^d )^2 ]+ d_s2 (2π), (27) where μθkd _ _k^d and σθkd _ _k^d denote the d-th elements of the predicted mean and standard deviation vectors, respectively. For reward prediction, we use the mean squared error loss, ℒrew=‖r^θk(t)−r(t)‖2, _rew= \| r_ _k(t)-r(t) \|^2, (28) where r(t)r(t) is the truth reward from the offline dataset. The training loss of the MLP k is then defined as ℒk=ℒdyn+λrℒrew, _k=L_dyn+ _rL_rew, (29) where λr _r is the weighting coefficient for the reward prediction loss. For a given state–action pair, the ensemble transition loss is formulated as, ℒtransition=∑k=1K∑b=1B(mk(b)ℒk(b))∑k=1K∑b=1Bmk(b). _transition= _k=1^K _b=1^B (m_k^(b)L_k^(b) ) _k=1^K _b=1^Bm_k^(b). (30) We integrate the global traffic state inference module and the probabilistic dynamics model into a unified traffic world model. The overall training objective is defined as, ℒWM=γ1ℒstate+γ2ℒtransition, _WM= _1L_state+ _2L_transition, (31) where γ1>0 _1>0 and γ2>0 _2>0 are weighting coefficients. In addition, we characterize model epistemic uncertainty using the dispersion among the predicted means of different ensemble members, χ(s^(t),a(t))=1K∑k=1K‖μθk(s^(t),a(t))−μ¯(s^(t),a(t))‖2, χ ( s(t),a(t) )= 1K _k=1^K\| _ _k ( s(t),a(t) )- μ ( s(t),a(t) )\|^2, (32) μ¯(s^(t),a(t))=1K∑k=1Kμθk(s^(t),a(t)). μ ( s(t),a(t) )= 1K _k=1^K _ _k ( s(t),a(t) ). (33) When a region is sufficiently covered by the offline dataset, the predictions of different ensemble members are generally consistent, resulting in low uncertainty. In contrast, when the policy visits state–action regions that are rarely covered by the training data, the predictions of different members may diverge significantly, leading to increased uncertainty. Therefore, χ(s^(t),a(t))χ ( s(t),a(t) ) can serve as an important indicator of model extrapolation risk and is used in subsequent policy learning to restrict unreliable imagined interactions. 3.4 World Model-Based Offline Multi-Agent Policy Learning After training, the world model is fixed and used to perform multi-step imagination rollouts to generate virtual trajectories for policy optimization. We adopt the CTDE paradigm and use multi-agent proximal policy optimization (MAPPO) 28 to optimize the cooperative control policy for multiple CAVs. Specifically, during training, the recovered global traffic state and joint actions are used to evaluate the system-level return, thereby capturing the coupling effects among different CAV control behaviors. During execution, each CAV makes decisions independently based only on its local observations, satisfying the requirement for decentralized control in practical traffic scenarios. However, policy optimization in world model-based offline learning is prone to model exploitation, where the policy exploits poorly covered state–action regions with overestimated returns, leading to spurious improvement. To mitigate this issue, we construct a pessimistic reward based on the epistemic uncertainty estimated by the ensemble model, r~(s^(t),a(t))=r^(s^(t),a(t))−βχ(s^(t),a(t)), r( s(t),a(t))= r( s(t),a(t))-βχ( s(t),a(t)), (34) where β is the pessimistic penalty coefficient. When the ensemble models exhibit large prediction disagreement for a given state–action pair, the pessimistic reward automatically reduces the estimated return in that region. During imagination rollouts, we start from the recovered state s^(t) s(t) sampled from the offline dataset and first use the observation decoder OψO_ψ to obtain the local observation o^i(t) o_i(t) for each CAV. Each CAV then selects its control action according to its decentralized execution policy πνi(ai∣τi(t)) _ _i(a_i _i(t)), and the individual actions are combined into the joint action a(t)a(t). subsequently, the world model predicts the next state s^(t+1) s(t+1) and outputs the pessimistic reward r~(s^(t),a(t)) r( s(t),a(t)) and epistemic uncertainty χ(s^(t),a(t))χ( s(t),a(t)). This procedure is repeated recursively to construct multi-step virtual trajectories. To prevent error accumulation in long-horizon imagination rollouts and avoid policy learning from highly uncertain virtual experiences, we further introduce an epistemic uncertainty-based rollout truncation mechanism. Let χmax _ denote the threshold. If χ(s^(t),a(t))≥χmaxχ( s(t),a(t))≥ _ during rollouts, the current virtual trajectory is immediately terminated. 3.5 Solution Algorithm Algorithm 1 summarizes the overall training procedure of the proposed framework. The world model is parameterized by Θ=ϕ,ψ,ξ,θ1:K =\φ,ψ,ξ, _1:K\, where ϕφ, ψ, ξ, and θ1:K _1:K denote the learnable parameters of the history encoder, observation decoder, cell-wise Cross-Attention module, and K ensemble transition-and-reward models, respectively. 4 Numerical Results 4.1 Experimental Setup To evaluate the effectiveness of the proposed method, we conduct numerical experiments using the simulation of urban mobility (SUMO) platform in an on-ramp merging bottleneck scenario based on a section of the Hujin expressway, as illustrated in Fig. 3. The CAV cooperative control zones on the mainline and on-ramp are 30003000 m and 600600 m long, respectively. Both are discretized with a spatial step of Δx=300m x=300~m, resulting in 1010 traffic cells on the mainline and 22 cells on the on-ramp. The traffic demands on the mainline and on-ramp are set to 25002500 veh/h and 18001800 veh/h, respectively. The baseline CAV penetration rate is set to 15%15\%, and the free-flow speed of all vehicles is 3030 m/s. The longitudinal acceleration of each CAV is constrained within ai∈[−3m/s2,3m/s2]a_i∈[-3~m/s^2,3~m/s^2]. Each simulation lasts for 12001200 time steps, with the first 200200 time steps used as a warm-up period. Algorithm 1 Physics-Informed World Model-Based Offline MARL 1: Offline dataset D; ensemble size K; uncertainty threshold χmax _ ; pessimism coefficient β. 2: World model Θ∗ ^* and decentralized policies Π∗ ^*. 3: Initialize world model Θ=ϕ,ψ,ξ,θ1:K =\φ,ψ,ξ, _1:K\ 4: Initialize actors πνii∈\ _ _i\_i and critic QςQ_ 5: for iter=1,…,NWMiter=1,…,N_WM do 6: Sample mini-batch from D 7: Build histories τi(t),τi(t+1)i∈\ _i(t), _i(t+1)\_i 8: Infer states s^(t) s(t) and s^(t+1) s(t+1) 9: Compute physical prior sphy(t+1)=Fphy(s^(t),a(t))s^phy(t+1)=F_phy( s(t),a(t)) 10: Predict (s^θk(t+1),r^θk(t))=MLPθk(s^(t),a(t)),k=1,…,K( s_ _k(t+1), r_ _k(t))=MLP_ _k( s(t),a(t)), k=1,…,K 11: Estimate uncertainty χ(t)χ(t) 12: Update Θ by minimizing LWML_WM 13: end for 14: Set Θ∗←Θ ^*← 15: for iter=1,…,NRLiter=1,…,N_RL do 16: Sample history from D and infer initial state s^(t0) s(t_0) 17: for h=0,…,Himg−1h=0,…,H_img-1 do 18: Decode observations and build τ^i(t0+h)i∈\ τ_i(t_0+h)\_i 19: Sample actions ai(t0+h)∼πνi(⋅∣τ^i(t0+h))a_i(t_0+h) _ _i(· τ_i(t_0+h)) 20: Form joint action a(t0+h)=ai(t0+h)i∈a(t_0+h)=\a_i(t_0+h)\_i 21: Predict (s^(t0+h+1),r^(t0+h),χ(t0+h))( s(t_0+h+1), r(t_0+h),χ(t_0+h)) using Θ∗ ^* 22: if χ(t0+h)≥χmaxχ(t_0+h)≥ _ then 23: break 24: end if 25: Store transition with r~(t0+h) r(t_0+h) 26: end for 27: Update QςQ_ and πνii∈\ _ _i\_i 28: end for 29: return Θ∗ ^* and Π∗=πνi∗i∈ ^*=\ _ _i^*\_i The offline dataset is generated using predefined behavior policies with different exploration levels under randomized traffic conditions. A total of 10001000 independent traffic trajectories are collected, resulting in approximately 1×1061× 10^6 state-transition samples. To avoid information leakage caused by temporal correlations between consecutive samples, the dataset is split at the trajectory level into training, validation, and test sets with a ratio of 70%/15%/15%70\%/15\%/15\%. All offline learning methods use the same training data, local observations, and action spaces to ensure a fair comparison. The world model consists of 77 independently initialized ensemble members. The historical observation window is set to H=10H=10, and the imagination horizon is set to Himg=5H_ img=5. The learning rates of the world model and policy networks are set to 1×10−31× 10^-3 and 1×10−41× 10^-4, respectively. The pessimistic reward coefficient is set to β=0.5β=0.5, while the uncertainty truncation threshold χmax _ is determined as the 9595th percentile of the epistemic uncertainty measured on the validation set. All control experiments are independently conducted using 55 random seeds, and the mean and standard deviation are reported. Figure 3: Simulation scenario of the on-ramp merging bottleneck 4.2 Baselines and Evaluation Metrics In addition to the without control case and the online multi-agent proximal policy optimization (MAPPO) 28 baseline, we consider eight methods to systematically investigate the effects of physics supervision, global state representation, and world model-based learning on offline multi-agent cooperative control. • M1: Proposed. The proposed framework. • M2: WM-Physical. This variant adopts the same physically interpretable state representation as M1 but removes the physics supervision from the state reconstruction process. Both the state reconstruction module and the world model are therefore trained in a fully data-driven manner. • M3: WM-Latent. This variant learns a global latent state without explicit physical interpretation from the local historical information of multiple CAVs and trains a purely data-driven world model in the resulting latent space. • M4: WM-Concat. This variant directly concatenates the local observations of all CAVs to form the global information representation and trains a purely data-driven world model based on this representation. • M5-M8: Model-Free Offline Multi-agent Reinforcement Learning. These methods directly learn control policies from the offline dataset without using a world model to generate imagined trajectories. To ensure a fair comparison, all four methods adopt the same multi-agent implicit q-learning (MAIQL) 24 policy learning framework and differ only in their global state representations. Specifically, M5 uses the physics-supervised global physical state, M6 uses the global physical state reconstructed without physics supervision, M7 uses the learned latent state, and M8 uses the concatenation of all CAV local observations. These baselines enable the contribution of each component to be examined from complementary perspectives. The comparison between M1 and M2 evaluates the effect of physics supervision. The paired comparisons M1/M5, M2/M6, M3/M7, and M4/M8 assess the benefit of introducing a world model under comparable global state representations. In addition, comparisons among M1-M4 and among M5-M8 are used to examine the influence of different global state representations. The methods are evaluated from two aspects: world model prediction accuracy and traffic control performance. For the world model, the mean absolute error (MAE) of state prediction and reward prediction is used to evaluate predictive accuracy. For traffic control, average travel time (ATT), average waiting time (AWT), and average time loss (ATL) are adopted to evaluate the system-level traffic control performance of different methods. 4.3 World-Model Prediction Accuracy We first evaluate the ability of different world models to predict traffic-state evolution and system rewards. As shown in Table 1, M1-M4 achieve good next-state prediction accuracy within their respective representation spaces. In particular, M4 uses the concatenation of all CAV local observations as the state representation, without requiring additional state reconstruction or latent feature mapping. Its prediction task is therefore relatively direct within its own representation space, resulting in a comparatively low state prediction error. Table 1: World-model prediction accuracy under different state representations Method Recovered-State MAE Recovered-Reward MAE True-State MAE True-Reward MAE M1: Proposed 0.01633 0.02031 0.09543 0.02065 M2: WM-Physical 0.01912 0.02377 0.41197 0.02340 M3: WM-Latent 0.06158 0.03221 – – M4: WM-Concat 0.00704 0.03427 – – For M1 and M2, whose state representations have explicit physical interpretations, we further compare their predictions with the ground-truth traffic states obtained from SUMO, as illustrated in Fig. 4. The true-state MAE of M1 is 0.095430.09543, compared with 0.411970.41197 for M2, corresponding to a reduction of 76.84%76.84\%. Similarly, the true-reward MAE decreases from 0.023400.02340 for M2 to 0.020650.02065 for M1, representing an improvement of 11.75%11.75\%. These results indicate that traffic-flow physics guidance not only improves the predictive consistency within the reconstructed state space, but, more importantly, strengthens the correspondence between the reconstructed states and the actual traffic evolution. (a) M1: State (b) M1: Reward (c) M2: State (d) M2: Reward Figure 4: World-model prediction results of M1 and M2 against the SUMO ground truth 4.4 Cooperative Control Performance Table 2 reports the traffic control performance of different methods in the on-ramp merging scenario shown in Fig. 1. The proposed M1 achieves the best overall control performance among all offline learning methods, with the lowest ATT and ATL. Compared with the without control case, M1 reduces ATT, AWT, and ATL by 19.79%19.79\%, 61.62%61.62\%, and 36.10%36.10\%, respectively. Moreover, its performance approaches that of Online MAPPO, indicating that the proposed method can learn effective cooperative control policies for multiple CAVs without requiring online interaction with the environment. Table 2: Traffic control performance of different methods Metric Without Control Online MAPPO M1 M2 M3 M4 M5 M6 M7 M8 ATT (s) 145.81 107.93 116.96 122.92 127.59 132.14 120.69 127.40 129.23 135.90 AWT (s) 0.99 0.14 0.38 0.32 0.37 0.34 0.38 0.53 0.51 0.49 ATL (s) 85.44 47.51 54.60 65.43 68.33 76.32 57.10 70.37 72.80 81.21 By incorporating physics supervision, M1 reduces ATT and ATL by 4.85%4.85\% and 16.55%16.55\%, respectively, compared with M2, although its AWT increases slightly. Combined with the world-model prediction results in Section 4.3, these results further suggest that physics supervision improves the reliability of global state reconstruction and world-model prediction, which in turn leads to better overall system-level traffic control performance. In addition, the paired comparisons between M1-M4 and their corresponding model-free counterparts M5-M8 show that world model-based policy learning generally achieves better traffic control performance under comparable state representations. This indicates that imagined rollouts generated by the learned world model can effectively expand the coverage of the fixed offline dataset and provide additional training experience for offline policy optimization, thereby improving the learned cooperative control policies. 4.5 Sensitivity Analysis To further evaluate the sensitivity of the proposed method under different traffic conditions, we examine the effects of traffic demand and CAV penetration rate on control performance. (a) Average travel time (b) Average waiting time (c) Average time loss Figure 5: Sensitivity of traffic control performance to traffic demand Fig. 5 compares the traffic performance of the proposed method with the without control case under different traffic demand levels. Under low traffic demand, the bottleneck operates under relatively uncongested conditions, resulting in only a small difference in ATT between the two cases. As traffic demand increases and bottleneck congestion intensifies, the performance gap progressively widens. In particular, when traffic demand exceeds approximately 2000veh/h2000~veh/h, ATT, AWT, and ATL increase rapidly in the without control case, whereas the proposed method effectively suppresses their growth and maintains effective congestion mitigation even under high-demand conditions. (a) Average travel time (b) Average waiting time (c) Average time loss Figure 6: Sensitivity of traffic control performance to CAV penetration rate Fig. 6 further investigates the effect of CAV penetration rate. Overall, ATT, AWT, and ATL exhibit decreasing trends as the CAV penetration rate increases, with the most pronounced improvements occurring at relatively low penetration levels. When the penetration rate reaches approximately 30%30\%-35%35\%, the marginal benefits of further increasing the number of CAVs gradually diminish. This indicates that once a sufficient number of CAVs are available to effectively regulate upstream traffic flow, additional controllable vehicles provide progressively smaller performance gains. 5 Conclusions This paper proposes a physics-informed world model-based offline multi-agent reinforcement learning framework for cooperative CAV control at partially observable mixed-traffic bottlenecks. The proposed method reconstructs a structured global traffic state with explicit physical meaning from the local observation-action histories of multiple CAVs and enhances its consistency with traffic evolution through a coupled macroscopic-microscopic traffic model. Based on the reconstructed state, a probabilistic ensemble world model is developed to learn traffic state transitions and system rewards. Pessimistic rewards and uncertainty-driven rollout truncation are further incorporated to generate reliable imagined trajectories for offline cooperative policy learning. Simulation experiments based on a real-world on-ramp bottleneck demonstrate that physics supervision improves the reliability of state reconstruction and world-model prediction, leading to better cooperative control performance. Sensitivity analysis further shows that the proposed method remains effective under varying traffic demands and CAV penetration rates and provides consistent congestion mitigation under congested conditions. Overall, the proposed framework enables effective cooperative CAV control without online trial-and-error interactions or access to complete global traffic states. Future work will focus on validating the proposed framework using field-collected trajectory and CAV observation data, investigating its robustness to sensing and communication uncertainties as well as traffic-model mismatch, and extending the framework to more complex bottleneck configurations and larger-scale traffic networks. These extensions will further assess the practical transferability and generalization capability of the proposed method under realistic traffic conditions. 6 Acknowledgments The authors used OpenAI ChatGPT to assist with language editing and the identification of potentially relevant literature. All technical content and research results were independently developed by the authors, and all references were independently verified. The authors take full responsibility for the content of this manuscript. AUTHOR CONTRIBUTIONS The authors confirm contribution to the paper as follows: study conception and methodology: Lu Liu, Xi Xiong; data collection, experiments, and manuscript preparation: Lu Liu; manuscript review and revision: Lu Liu, Chi Xie, Xi Xiong. All authors reviewed and approved the final version of the manuscript. DECLARATION OF CONFLICTING INTERESTS The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. FUNDING The authors disclosed no financial support for the research, authorship, and/or publication of this article. References Bernstein et al. (2002) D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein The complexity of decentralized control of Markov decision processes. Mathematics of operations research 27 (4), p. 819–840. Cited by: §2.1. Bui et al. (2024) T. V. Bui, T. H. Nguyen, and T. Mai ComaDICE: offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. arXiv preprint arXiv:2410.01954. Cited by: Introduction. Cang et al. (2021) C. Cang, A. Rajeswaran, P. Abbeel, and M. Laskin Behavioral priors and dynamics models: improving performance and domain transfer in offline RL. arXiv preprint arXiv:2106.09119. Cited by: Introduction. Chung et al. (2007) K. Chung, J. Rudjanakanoknad, and M. J. Cassidy Relation between traffic density and capacity drop at three freeway bottlenecks. Transportation Research Part B: Methodological 41 (1), p. 82–95. Cited by: Introduction. Čičić and Johansson (2018) M. Čičić and K. H. Johansson Traffic regulation via individually controlled automated vehicles: A cell transmission model approach. In 2018 21st International Conference on Intelligent Transportation Systems, Vol. , p. 766–771. Cited by: §2.2. Daganzo (1994) C. F. Daganzo The cell transmission model: a dynamic representation of highway traffic consistent with the hydrodynamic theory. Transportation Research Part B: Methodological 28 (4), p. 269–287. Cited by: §2.2. Fang et al. (2022) X. Fang, Q. Zhang, Y. Gao, and D. Zhao Offline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems, Vol. , p. 3417–3422. Cited by: Introduction. Feng et al. (2021) S. Feng, Z. Song, Z. Li, Y. Zhang, and L. Li Robust platoon control in mixed traffic flow based on tube model predictive control. IEEE Transactions on Intelligent Vehicles 6 (4), p. 711–722. Cited by: Introduction. Hafner et al. (2019) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi Dream to control: learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. Cited by: Introduction. Huang et al. (2025) X. Huang, X. Wang, and Y. Cheng Uncertainty-based alternative diffusion policy for safe autonomous driving. IEEE Transactions on Intelligent Transportation Systems 26, p. 18854–18863. Cited by: Introduction. Jin (2012) W. Jin A kinematic wave theory of multi-commodity network traffic flow. Transportation Research Part B: Methodological 46 (8), p. 1000–1022. Cited by: §2.2. Kang et al. (2025) S. Kang, Y. Lee, G. Kim, S. Chong, and S. Yun MA2A^2E: Addressing partial observability in multi-agent reinforcement learning with masked auto-encoder. In International Conference on Learning Representations, Vol. 2025, p. 20145–20165. Cited by: Introduction. Kidambi et al. (2020) R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims MOReL: Model-Based offline reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 33, p. 21810–21823. Cited by: Introduction. Li et al. (2020) L. Li, R. Jiang, Z. He, X. Chen, and X. Zhou Trajectory data-based traffic flow studies: A revisit. Transportation Research Part C: Emerging Technologies 114, p. 225–240. External Links: ISSN 0968-090X Cited by: Introduction. Li et al. (2023) Y. Li, B. Pan, Z. Chen, and L. Xing Developing a dynamic speed control system for mixed traffic flow to reduce collision risks near freeway bottlenecks. IEEE Transactions on Intelligent Transportation Systems 24 (11), p. 12560–12581. Cited by: Introduction. Liao et al. (2022) X. Liao, Z. Wang, X. Zhao, K. Han, P. Tiwari, M. J. Barth, and G. Wu Cooperative ramp merging design and field implementation: A digital twin approach based on vehicle-to-cloud communication. IEEE Transactions on Intelligent Transportation Systems 23 (5), p. 4490–4500. Cited by: Introduction. Lu et al. (2022) G. Lu, Z. Shen, X. Liu, Y. Nie, and Z. Xiong Are autonomous vehicles better off without signals at intersections? A comparative computational study. Transportation Research Part B: Methodological 155, p. 26–46. Cited by: Introduction. Newell (1993) G. F. Newell A simplified theory of kinematic waves in highway traffic, part i: Queueing at freeway bottlenecks. Transportation Research Part B: Methodological 27 (4), p. 289–303. Cited by: §2.2. Qiu and Du (2023) J. Qiu and L. Du Cooperative trajectory control for synchronizing the movement of two connected and autonomous vehicles separated in a mixed traffic flow. Transportation Research Part B: Methodological 174, p. 102769. Cited by: Introduction. Ran et al. (2023) Y. Ran, Y. Li, F. Zhang, Z. Zhang, and Y. Yu Policy regularization with dataset constraint for offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, Vol. 202, p. 28701–28717. Cited by: Introduction. Rauch et al. (2024) R. Rauch, Z. Becvar, P. Mach, and J. Gazda Cooperative multi-agent deep reinforcement learning for dynamic task execution and resource allocation in vehicular edge computing. IEEE Transactions on Vehicular Technology 74 (4), p. 5741–5756. Cited by: Introduction. Richards (1956) P. I. Richards Shock waves on the highway. Operations Research 4 (1), p. 42–51. Cited by: §2.2. Ross and Bagnell (2012) S. Ross and J. A. Bagnell Agnostic system identification for model-based reinforcement learning. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Vol. 2, p. . Cited by: Introduction. Tampuu et al. (2015) A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente MultiAgent cooperation and competition with deep reinforcement learning. arXiv preprint arXiv:1511.08779. Cited by: 5th item. Wang et al. (2023) J. Wang, M. Yuan, Y. Li, and Z. Zhao Hierarchical attention master–slave for heterogeneous multi-agent reinforcement learning. Neural Networks 162, p. 359–368. External Links: ISSN 0893-6080 Cited by: Introduction. Wiesemann et al. (2013) W. Wiesemann, D. Kuhn, and B. Rustem Robust markov decision processes. Mathematics of Operations Research 38 (1), p. 153–183. Cited by: Introduction. Xiong et al. (2024) X. Xiong, M. Wang, D. Sun, and L. Jin An approximate dynamic programming approach to vehicle platooning coordination in networks. IEEE Transactions on Intelligent Transportation Systems 25 (11), p. 16536–16547. Cited by: Introduction. Yu et al. (2022) C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu The surprising effectiveness of PPO in cooperative multi-agent games. Advances in neural information processing systems 35, p. 24611–24624. Cited by: §3.4, §4.2. Zhou et al. (2025) Y. Zhou, Y. Zheng, Y. Hu, K. Chen, T. Zheng, J. Song, M. Song, and S. Liu Cooperative policy agreement: Learning diverse policy for offline marl. Proceedings of the AAAI Conference on Artificial Intelligence 39, p. 23018–23026. External Links: Document Cited by: Introduction.