Paper deep dive
Reinforcement Learning-Based Control of CAV Platoon Joining Maneuvers in Mixed Traffic
Biao Yin, Abderrahmane Kasmi, Nadir Farhi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/28/2026, 4:19:03 AM
Summary
This paper proposes a reinforcement learning-based control framework for Connected and Automated Vehicle (CAV) platoon joining maneuvers in mixed traffic environments. Using an agent-based model coupled with the SUMO simulator, the study evaluates Deep Q-Network (DQN), Double DQN (DDQN), and Proximal Policy Optimization (PPO) algorithms. The results indicate that PPO outperforms DQN and DDQN, achieving a 98% joining success rate and <1% collision rate by incorporating risk-related penalties. The study highlights a trade-off between safety, efficiency, and decision steps, and compares internal penalty mechanisms with external safety controllers.
Entities (8)
Relation Signals (6)
CAV → coexistswith → Human-Driven Vehicles
confidence 95% · mixed-traffic environments, where CAVs coexist with human-driven vehicles exhibiting heterogeneous longitudinal and lateral behaviors.
PPO → outperforms → DQN
confidence 95% · Results show that PPO outperforms DQN and DDQN, achieving a joining success rate of approximately 98 %
PPO → outperforms → DDQN
confidence 95% · Results show that PPO outperforms DQN and DDQN, achieving a joining success rate of approximately 98 %
External safety controller → prevents → Collisions
confidence 93% · An external safety controller effectively prevents collisions, although its interventions may reduce joining efficiency.
Risk-related penalties → improves → joining success rate
confidence 92% · achieving a joining success rate of approximately 98 % and a collision rate below 1 %, largely due to risk-related penalties incorporated into the reward function.
SUMO → usedwith → PLEXE
confidence 90% · the environment includes a CAV control module – PLEXE [22], which is a cooperative driving framework extended in SUMO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Connected and automated vehicle (CAV) platooning offers a promising approach to improving road safety and traffic capacity. However, platoon control in real-world traffic is challenging due to uncertainty and heterogeneous driving behaviors. Reinforcement learning (RL) has strong potential for addressing such control problems, but its practical deployment raises challenges related to safety and learning efficiency. This paper proposes a generic modeling and simulation framework for investigating CAV platoon joining maneuvers and comparing deep reinforcement learning (DRL)-based control algorithms. The problem is particularly challenging in mixed-traffic environments, where CAVs coexist with human-driven vehicles exhibiting heterogeneous longitudinal and lateral behaviors. The objective is to achieve safe and efficient joining maneuvers by either incorporating penalties for risky behaviors into the learning process or using an external safety controller to constrain the learned policy. An agent-based modeling framework coupled with the Simulation of Urban MObility (SUMO) simulator is used to evaluate Deep Q-Network (DQN), Double Deep Q-Network (DDQN), and Proximal Policy Optimization (PPO). Results show that PPO outperforms DQN and DDQN, achieving a joining success rate of approximately 98 % and a collision rate below 1 %, largely due to risk-related penalties incorporated into the reward function. However, this improved performance requires more decision steps to complete the maneuver, revealing a trade-off between safety, joining effectiveness, and decision efficiency. An external safety controller effectively prevents collisions, although its interventions may reduce joining efficiency. The results highlight the importance of jointly considering safety and efficiency when designing RL-based controllers for CAV platoon joining in mixed traffic.
Tags
Links
- Source: https://arxiv.org/abs/2608.26860v1
- Canonical: https://arxiv.org/abs/2608.26860v1
Trouble viewing inline? Open PDF directly →
Full Text
49,902 characters extracted from source content.
Expand or collapse full text
Reinforcement Learning-Based Control of CAV Platoon Joining Maneuvers in Mixed Traffic Biao Yin Email: biao.yin@univ-eiffel.fr Corresponding author: Corresponding author. Address: Université Gustave Eiffel, SATIE, Gif-sur-Yvette, 91190, France. Abderrahmane Kasmi Address: Université Paris Dauphine-PSL, Paris, 75016, France. Nadir Farhi Address: Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France. Abstract Connected and automated vehicle (CAV) platooning offers a promising approach to improving road safety and traffic capacity. However, platoon control in real-world traffic is challenging due to uncertainty and heterogeneous driving behaviors. Reinforcement learning (RL) has strong potential for addressing such control problems, but its practical deployment raises challenges related to safety and learning efficiency. This paper proposes a generic modeling and simulation framework for investigating CAV platoon joining maneuvers and comparing deep reinforcement learning (DRL)-based control algorithms. The problem is particularly challenging in mixed-traffic environments, where CAVs coexist with human-driven vehicles exhibiting heterogeneous longitudinal and lateral behaviors. The objective is to achieve safe and efficient joining maneuvers by either incorporating penalties for risky behaviors into the learning process or using an external safety controller to constrain the learned policy. An agent-based modeling framework coupled with the Simulation of Urban MObility (SUMO) simulator is used to evaluate Deep Q-Network (DQN), Double Deep Q-Network (DDQN), and Proximal Policy Optimization (PPO). Results show that PPO outperforms DQN and DDQN, achieving a joining success rate of approximately 98 % and a collision rate below 1 %, largely due to risk-related penalties incorporated into the reward function. However, this improved performance requires more decision steps to complete the maneuver, revealing a trade-off between safety, joining effectiveness, and decision efficiency. An external safety controller effectively prevents collisions, although its interventions may reduce joining efficiency. The results highlight the importance of jointly considering safety and efficiency when designing RL-based controllers for CAV platoon joining in mixed traffic. The source code is available at: https://github.com/biaoyin/platoon-drl/tree/platoon_join Keywords: Vehicle platooning , Deep reinforcement learning , CAVs , Mixed traffic , Platoon joining , Collision avoidance. 1 Introduction CAVs enable vehicle-to-vehicle and vehicle-to-infrastructure communications to achieve cooperative control and mitigate undesirable human driving behavior (e.g., aggression and stop-and-go instability) [1][2]. The benefits of CAVs extend to traffic flow, yielding improvements in safety and efficiency, as well as contributing to environmental sustainability. Abundant studies have focused on longitudinal speed coordination control of CAVs within a platoon, in which a group of CAVs moves at a consensual speed while maintaining a small, nearly constant distance between adjacent vehicles, such as cooperative adaptive cruise control (CACC). It has been demonstrated that CAV platoons have enormous potential to enhance road capacity [3], traffic stability [4], and energy efficiency [5]. Platooning optimization typically involves several key operational phases in the lifecycle of a platoon. These phases structure how vehicles form, operate within, and leave a platoon. Each phase requires sophisticated mechanisms to ensure both safety and efficiency. Although car-following dynamics (such as CACC) have been extensively studied, the initial platoon joining phase is fundamentally important, enabling a CAV to merge into a platoon at a designated position. This requires precise speed control and well-timed lane-changing maneuvers. A key challenge lies in managing longitudinal and lateral control that avoids traffic disturbances, such as congestion or collisions. This challenge is further intensified in mixed traffic, where CAVs coexist with human-driven vehicles in the near future. In previous work, conventional approaches for platoon joining have been mainly explored based on rules, optimization models, or conceptualized techniques such as virtual platoons [6],[7],[8]. Rule-based approaches enable coordinated merging behavior with low computational complexity, but their performance may be limited in highly dynamic environments [6]. Optimization-based approaches, such as model predictive control [7], can improve flexibility and performance by formulating the problem as a constrained optimization problem, while requiring higher computational effort and more accurate modeling. Virtual platooning introduces a higher level of cooperation by organizing vehicles into a logical platoon before physical merging occurs. This enables early synchronization of speeds, precise position adjustment, and proactive gap creation with the target platoon. The technique is mostly used in on-ramp joining scenarios, where the merging vehicle is virtually integrated into the main-road platoon in advance, as demonstrated in [5] and [8]. Recently, reinforcement learning (RL) algorithms have been increasingly adopted to tackle the complexities of platoon-joining tasks. Early foundational work primarily focused on single-agent settings. In the value-based RL domain, [9] introduced a deep Q-network (DQN) solution for platoon merging within a discrete-cell road environment. The results show that the proposed DQN approach requires less merging travel time and fewer vehicle lane-change times than the rule-based approach. Concurrently, policy-based RL methods were explored in [10], which applied proximal policy optimization (PPO) to automated lane-changing maneuvers and achieved a 95% success rate in dense traffic. To address the inherently multi-agent nature of platoon coordination, subsequent research extended RL frameworks to handle inter-agent competition and cooperation. A series of multi-agent RL algorithms have been applied to CAV platoon formation, as demonstrated across studies [11], [12], [13], [14], [15]. However, a critical step toward real-world deployment lies in navigating mixed-traffic environments, where CAVs and human-driven vehicles (HDVs) coexist. In this context, [16] proposed a two-stage framework: the first stage enumerates all feasible platoon formations given a specific vehicle sequence, while the second stage employs a multi-agent policy to guide each vehicle—both automated and human-driven—into its designated position according to the formation plan. To further accelerate convergence, hybrid approaches have also been investigated, such as combining genetic algorithms with deep RL for smart platooning [17]. Beyond formation planning and convergence speed, collision during the joining maneuver remains a persistent and fundamental challenge. The literature reveals two predominant safety strategies. The first appends an independent safety controller—an external, rule-based module—to override potentially hazardous decisions and ensure collision-free execution [18]. The second incorporates internal risky-joining penalties directly within the reward function, thereby discouraging unsafe behaviors intrinsically during the training process [10]. A recent study integrates a control barrier function (CBF) into an RL framework to guarantee a safe joining process in the formation of mixed platoons (including CAVs and HDVs), demonstrating promising results for longitudinal and lateral control [25]. Despite these advances, existing studies indicate that most frameworks and simulations rely on oversimplified or abstracted environments that fail to capture realistic microscopic traffic dynamics. The transition from mixed-traffic lanes to dedicated CAV lanes—a highly practical scenario—has received limited systematic investigation, which was deemed efficient for flow organization with mixed traffic [19]. Most critically, while safety modules (external shields) and penalty-based rewards (internal costs) have been individually explored, there is a lack of comprehensive comparative analysis that evaluates their distinct roles, trade-offs, and combined effectiveness under diverse and dynamic traffic conditions. To bridge these gaps, this paper proposes a generic RL-based modeling framework built within the SUMO simulation environment to systematically study CAV platoon joining maneuvers. Our main contributions therefore are threefold: 1) developing a system framework that enables CAVs to join a platoon when operating from a mixed-traffic lane to a dedicated CAV lane on the highway; 2) implementing and comparing three distinct types of deep RL algorithms to learn optimal joining policies; and 3) conducting comprehensive performance analyses under diverse traffic conditions, with a specific emphasis on assessing the roles of risky penalties during training versus the independent safety shield for collision avoidance, thereby providing critical insights into the design of robust and reliable platoon-joining controllers. The rest of the paper is organized as follows. We introduce the fundamentals of RL and typical deep RL algorithms in Section 2. The detailed methodology, including the simulation system architecture, platooning task modeling, and algorithm implementation, is presented in Section 3. Afterwards, we present the simulation experiments and results analysis. The conclusion is drawn in the last section. 2 Background Knowledge 2.1 Fundamentals of RL RL is designed for an agent to learn by interacting with the environment. The agent learns how to make decisions by taking actions and receiving feedback from the environment. Specifically, the decision-making process is modeled as a Markov decision process (MDP), defined by the tuple (,,P,r,γ)(S,A,P,r,γ), where S and A denote the state and action spaces, respectively, P(s′|s,a)P(s |s,a) represents the state transition probability, r(s,a,s′)r(s,a,s ) is the reward function, and γ∈(0,1]γ∈ (0,1 ] is the discount factor for future reward. At each time step t, the agent selects action ata_t based on the current state of the environment sts_t. The environment then responds by transitioning to a new state st+1∼P(⋅|st,at)s_t+1 P(·|s_t,a_t) and returns a reward rt+1=r(st,at,st+1)r_t+1=r(s_t,a_t,s_t+1). The goal of RL is to learn a policy π:→π:S in the form of a condition probability π(at|st)π(a_t|s_t) to maximize the expected cumulative discounted reward in infinite or over a finite horizon T, defined as J(π)=π[∑t=0T−1γtrt+1]J(π)=E_π [ _t=0^T-1γ^tr_t+1 ]. Q-learning is a classical RL method based on the Bellman Equation and seeks to maximize the objective J(π)J(π) by updating the estimate of Q(s,a)Q(s,a) iteratively. 2.2 DQN and DDQN When deep neural networks are used to approximate the value function (e.g., the Q-function) or the policy, the approach is known as deep RL. As one of the deep RL algorithms, DQN uses the same Q-value maximization for both action selection and evaluation, although the online and target networks are maintained separately. The target value function is given by, yDQN=r+γmaxa′Q(s′,a′,θ−)y^DQN=r+γ a maxQ(s ,a ;θ^-) (1) where θ−θ^- are the weights of the target network. The goal is to find θ in the Q-network that estimates the best Q function, i.e., Q(s,a,θ)≈Q∗(s,a)Q(s,a;θ)≈ Q^*(s,a) by minimizing the following loss function: ℒ(θ)=[(yDQN−Q(s,a,θ))2]L(θ)=E [(y^DQN-Q(s,a;θ))^2 ] (2) where θ can be optimized using gradiant decent and θ−θ^- is periodically cloned from θ (e.g., a soft update strategy) to maintain stability. Due to noise in the estimated Q(s,a,θ)Q(s,a;θ), the max operator can lead to a systematic overestimation bias. Therefore, DDQN [21] is proposed to use two different networks (with the same architecture), where the online network picks the best action, and the target network evaluates its value, as shown in Eq.(3). yDDQN=r+γQ(s′,argmaxa′Q(s′,a′,θ),θ−)y^DDQN=r+γ Q(s ,arg a maxQ(s ,a ;θ);θ^-) (3) where argmax uses the online weight θ and the Q value uses the target weights θ−θ^-. The loss function in Eq. (2) then uses yDDQNy^DDQN instead of yDQNy^DQN. Normally, the ϵε-greedy policy is used for action selection. The best action is chosen with probability 1−ϵ1-ε and the random action selection with probability ϵε. Specifically, we adopt the time-dependent ϵ(t)ε(t) determined by an exponential decay function as shown in Eq. (4). This ϵ(t)=ϵminε(t)= _ is satisfied after a large number of simulation steps of ϵdecay _decay, which means the best actions are mostly chosen in the final convergence. ϵ(t)=exp(log(ϵstart)+tϵdecay(log(ϵmin)−log(ϵstart)))ε(t)= ( ( _start)+ t _decay ( ( _ )- ( _start) ) ) (4) For both algorithms, a replay buffer is used, in which a batch of stored experience quadruplets (state, action, reward, new state) is randomly sampled for θ updates. 2.3 PPO PPO does not use a Q-target network. It updates its policy using an advantage function in Eq.(5) (usually computed via GAE - Generalized Advantage Estimation) to tell the agent how much better an action is compared to the average. At^=δt+(γλ)δt+1+…+(γλ)T−t+1δT−1 A_t= _t+(γλ) _t+1+...+(γλ)^T-t+1 _T-1 (5) where δt=rt+γV(st+1)−V(st) _t=r_t+γ V(s_t+1)-V(s_t) and λ is the GAE parameter. The objective in PPO is to maximize a surrogate objective function (see Eq.(6)) by using importance sampling to safely reuse old data to update the neural network’s weights θ. LPPO(θ)=t[min(rt(θ)At^,clip(rt(θ),1−ϵ,1+ϵ)At^)]L^PPO(θ)=E_t [min(r_t(θ) A_t,clip(r_t(θ),1-ε,1+ε) A_t) ] (6) The related PPO algorithm can be seen in [10]. 3 Methodology 3.1 System architecture To realize the agent-based learning process in a traffic dynamic simulation environment, we illustrate the built system architecture in Fig. 1. The system consists of two main components: the environment and the agent. For the environment, two-lane highway traffic is modeled in SUMO, along with vehicle characteristics and traffic load variations over time. In the network configuration, the inner lane is dedicated exclusively to CAV platoons, and the outer lane is a mixed lane that includes both HDVs and CAVs. The speed limits of these two lanes can be different. Besides the embedded longitudinal and lateral control models in SUMO for HDVs, the environment includes a CAV control module – PLEXE [22], which is a cooperative driving framework extended in SUMO to enable automated driving behavior simulation, such as ACC and CACC. To facilitate communication with the learning agent, the environment is encapsulated as a Gym environment, which is a standard open-source library used for many RL applications. The current environmental information around the ego CAV is transmitted via the SUMO Traci API to construct the state representation, which is used as input for the agent-based learning model. The model predicts values of possible actions in terms of longitudinal or lateral driving control, and the selected action is sent back to the ego CAV for maneuver execution. Notably, the built system architecture is supposed to be transferable for other agent-based traffic control problems, such as traffic signal control [23] and on-ramp merging [24]. Figure 1: System architecture for the RL-based agent modeling and simulation 3.2 CAV platoon formation problem In this study, we assume that the joining CAV enters at the rear of the target platoon and leaves from the rear position. All members in the platoon from the head to the last are organized in increasing order according to their destinations (i.e., exits of the highway). This strategy reduces the need for internal reordering and mitigates traffic disturbances. Focusing on our CAV platoon formation, the control procedure consists of three phases: 1) CAV joiner and platoon selection, 2) joining maneuver, and 3) gap closing. Fig. 2 illustrates the workflow of platoon formation. Regarding the life cycle of the platoon, an additional process for exiting platoons should be integrated. This process is not in the scope of this paper. The above three main phases for the platoon formation are introduced below. Joiner and target platoon selection. A CAV is randomly selected from the set of joiner candidates on the mixed-traffic lane (i.e., outer lane). The set of joiner candidates consists of CAVs that must have the possibility to choose one of the existing platoons within their perception range. Here, the perception range is not the same as the communication zone, in which joining requests can be reached as far as possible. We define the perception range as follows: if the joiner is at position x meters (m), the last member of the target platoon should be located in the range of [x−δ1,x+δ2][x- _1,x+ _2]. δ1 _1 is the distance behind the joiner vehicle to ensure the maximum platooning size, and it must be less than the radius of the communication zone R. δ2 _2 is a short distance ahead of the joiner vehicle for following the target platoon in a short time. In our study, we set δ1=150 _1=150 m, δ2=5 _2=5 m and R=200R=200 m. When platoons receive the joining request from the selected CAV joiner, they may accept or refuse the joining request, subject to their availability. The platoon’s availability depends on two conditions: 1) no ongoing joining/splitting process by another CAV, and 2) satisfying the ordered destinations regarding the existing CAV members in the platoon. Under these constraints, a joiner vehicle will be finally confirmed to join one of the available platoons in its neighborhood. If there is no platoon in the zone, a CAV can move to the inner lane under the default lane-change model to form an initial platoon containing only one vehicle. Figure 2: Workflow of platooning control Joining maneuver. Once the CAV joiner and the target platoon are selected, the joining maneuver (either lane change or speed adjustment) starts under control by the agent learning model. This phase mainly focuses on implementing the deep RL algorithms (i.e., DQN, Double DQN, and PPO). The CAV joiner will learn from the surrounding traffic conditions (i.e., state), intending to join the target platoon by executing an appropriate driving maneuver (i.e., action), through either adjusting longitudinal speed or switching to the inner lane, under the assessment of the cumulative reward. The configuration of each component of the deep RL algorithms will be detailed in Section 3.3. To avoid collisions, related safety constraints such as time-to-collision (TTC) can be applied during the algorithm training or before the execution of lane changes. We will initially take TTC-related collision risks into account during the model training for the purpose of assessing the learning performance. A TTC-related safety shield will be considered for a comparison as needed. Gap closing. After the ego vehicle VeV_e arrives at the target platoon lane, it should approach the target platoon and minimize the coordinated distance dcrd_crd (e.g., 5 m) to its predecessor. To achieve this, we use the embedded PLEXE package for the CACC implementation, which allows the ego vehicle to synchronize its behavior to the target platoon members and complete its formation. Since the inner lane is dedicated exclusively to platoons composed of CAVs and the ego vehicle is correctly positioned after the lane-change maneuver, this simple mechanism is sufficient to maintain the desired headway among platoon members. 3.3 RL for CAV agent modeling State representation. For a CAV joining a platoon at the right moment, it should have a vision of the surrounding vehicles. This is crucial for the ego vehicle to adapt speed and know when to change lanes without causing collisions. Fig. 3 illustrates the CAV joiner and its surrounding vehicles as follows: 1. VeV_e: the CAV ego vehicle, i.e., the joiner. 2. VpmV_pm: the predecessor of the ego vehicle on the mixed lane m. 3. VfmV_fm: the follower of the ego vehicle on the mixed lane m. 4. VppV_p: the predecessor of the ego vehicle on the platoon lane p. 5. VfpV_fp: the follower of the ego vehicle on the platoon lane p. 6. VltV_lt: the leader of the target platoon. 7. VtargetV_target: the last member of the target platoon. It can be the same as VltV_lt if the platoon initially consists of a single vehicle. 8. VlfV_lf: the leader of the following platoon. Figure 3: Ego vehicle and surrounding vehicles Here, the state is a vector of 1515 variables given by Eq. (7). s= s=\\, ve,vpm,vfm,vpp,vfp,vtarget,vlf,dpm,dfm,dpp,dfp,dtarget,dint,le,lp v_e,v_pm,v_fm,v_p,v_fp,v_target,v_lf,d_pm,d_fm,d_p,d_fp,d_target,d_int,l_e,l_p\,\ (7) where ve,vpm,vfm,vpp,vfp,vtargetv_e,v_pm,v_fm,v_p,v_fp,v_target, and vlfv_lf are respectively the speeds of the ego vehicle VeV_e, its predecessor and follower vehicles on the mixed lane (i.e, Vfm,VfmV_fm,V_fm), its predecessor and follower vehicles on the platoon lane (i.e., Vpp,VfpV_p,V_fp), and the predecessor VltV_lt and follower VlfV_lf of the target joining position; dpm,dfm,dppd_pm, d_fm, d_p, and dfpd_fp are the absolute distances between the ego vehicle and the predecessor/follower vehicles on the two lanes; dtargetd_target is the directional distance between the ego vehicle and the last vehicle in the target platoon (notice that dtargetd_target is negative when VtargetV_target is behind of VeV_e, otherwise.); dintd_int is the absolute value of the distance between the rear of VtargetV_target and the head of VlfV_lf. Except for dintd_int, for simplification, all distances are calculated as head distances between the ego vehicle and the related surrounding vehicles. lel_e and lpl_p are the indices of the ego vehicle’s located lane and the platoon lane. We assume that related vehicle information can be acquired in time within the communication zone. In a low-density traffic situation, the predecessor and follower vehicles may be partially missing. We adopt the default limit speed on the lane and the communication range R as substitutes, e.g.,vpmv_pm = 100 km/h and dpmd_pm = 200 m when no vehicle is ahead of VeV_e. Action space. Our action space consists of six discrete actions, including the lane change action a0a_0 with a current speed and five actions for speed adjustements without lane changes, which are defined as follows: a1a_1 = 2m/s22\,m/s^2, a2a_2 = 1m/s21\,m/s^2, a3a_3 = 0m/s20\,m/s^2, a4a_4 = −1m/s2-1\,m/s^2, and a5a_5 = −2m/s2-2\,m/s^2. To simulate an action in a reasonable implementation, we set a decision-action interval Δt t consisting of five simulation steps (i.e., a total of 0.5 s due to the simulation time step of 0.1 s), meaning that once an action is chosen, the next action can only be taken after five simulation steps. The decision-action interval should ensure the time delay from the decision-making process and the communication between vehicles. Reward function. To achieve an efficient and safe joining maneuver, we propose a reward function with sub-rewards in Eq. (8), implying a bonus of a successful joining and penalties in other cases such as a failure joining, a joining with a collision, a delayed joining, and unstable joinings (either being risky as too close to the last member of the target platoon or being inefficient as farway behind). The reward formulation is presented as follows: r=αs(rs+r+ru)+αf⋅rf+αc⋅rc+αk(r+rj) array[]lcl r= _s(r_s+r_r+r_u)+ _f· r_f+ _c· r_c+ _k(r_r+r_j) array (8) where: 1. αs _s, αf _f, αc _c, and αk _k are dummy variables. For each variable, a value of 1 indicates that the corresponding status (success, failure, collision, or lane keeping) occurs, while 0 indicates otherwise. 2. rsr_s is a success bonus. A successful joining maneuver leads the ego vehicle VeV_e to the target position behind the predecessor VtargetV_target in the target platoon. The success bonus is set to 100100. 3. r_r is the risky maneuver penalty, although the joiner gets a successful join. It is related to two alternative penalties for a risky situation of either changing to the platoon lane (αs=1 _s=1) or staying on the mixed lane with speed adjustments (αk=1 _k=1), according to the measures of rear and front TTC. We set it as: r=−100,ifmin(TTC(Ve,Vlf),TTC(Ve,Vtarget))<2andαs=1,−50,ifmin(TTC(Ve,Vfm),TTC(Ve,Vpm))<1andαk=1,0,otherwise. array[]lcl r_r= cases-100,&if\ min(TTC_(V_e,V_lf),TTC_(V_e,V_target))<2\\ &and\ _s=1,\\[6.0pt] -50,&if\ min(TTC_(V_e,V_fm),TTC_(V_e,V_pm))<1\\ &and\ _k=1,\\[6.0pt] 0,&otherwise. cases array (9) 4. rur_u is the unstable maneuver penalty. This is related to the distance (either too close or too far) to the target platoon after joining. We consider the range of safe positions between 10 m and 20 m without any penalty. Eq.(10) represents how this sub-reward is formulated: ru(d)=−2(10−d),ifd<10,0,if 10≤d≤20,−0.5(d−20),otherwise. array[]lcl r_u(d)= cases-2(10-d),&if\ d<10,\\[6.0pt] 0,&if\ 10≤ d≤ 20,\\[6.0pt] -0.5(d-20),&otherwise. cases array (10) where d is the immediate distance between the joiner VeV_e and the vehicle VtargetV_target after a joining maneuver (i.e., only completing a lane change before the gap closing). 5. rfr_f is the failure penalty. Any joining maneuver by which the joiner VeV_e is not placed after the predecessor VtargetV_target is considered a failure. A failure also occurs when the joiner exits the network before performing the joining maneuver. We set the failure penalty to −50-50. 6. rcr_c is the collision penalty. To avoid collisions, we set a heavy penalty of -10,000 in a collision case that occurs either on the platoon lane or on the mixed lane. 7. rjr_j is the jerk penalty. This penalty relates to excessive longitudinal acceleration and jerk when actions are taken to keep the current lane with the speed adjustment. We penalize the jerk if it satisfies |a˙|>4m/s3 | a |>4\,m/s^3 (when αk=1 _k=1), reflecting an uncomfortable driving experience. In our study, the penalty is set as: rj=−|a˙|,if|a˙|>4,0,otherwise. array[]lcl r_j= cases- | a |,&if | a |>4,\\[6.0pt] 0,&otherwise. cases array (11) 4 Simulation and Results 4.1 Experimental setup To build a mixed CAV traffic environment in SUMO, we adopt a CAV penetration ratio of 50% on the mixed lane coexisting with HDVs and set 100% CAVs on the platoon lane. The default car-following model (IDM) and lane-change model (LC2013) are applied only for HDVs. Thanks to the package PLEXE embedded in SUMO, for CAVs not in platoons (i.e., on the mixed lane) and platoon leaders (i.e., on the platoon lane), the ACC car-following model is adopted; on the platoon lane, the CAV ego vehicle adopts the CACC car-following model to catch up to the target platoon. Once a CAV is selected as a platoon joiner, its ACC or CACC is deactivated, and lane changes and speed adjustments are governed by the deep RL algorithms. Table A.1 and Table A.2 show the hyperparameter settings for training/testing the value-based models of DQN and DDQN, and the policy-based PPO model, respectively. Table A.3 gives the configuration for simulating traffic scenarios. All experiments were performed on a server featuring a 2.4 GHz CPU with 64 processors and 512 GB of RAM. 4.2 Model training performance Before training, a warm-up phase of 1,200 simulation steps is launched in SUMO to allow vehicles to spread out in the network, and platoons are also initialized during the phase. Specifically, in our configuration, we set event as the unit of executed joining status (i.e., success, failure, or collision). Each training episode consists of 1,000 decision-making steps, thus resulting in fewer than 1,000 events. That is to say, one event could occur within multiple decision steps, as a selected joiner can either stay on its current lane due to an adopted action for just speed adjustment, or give up its platoon joining if potential collision risks are detected (see safety shield tests in Section 4.4). We train the model by 10610^6 decision steps with incremental levels of random traffic. The training time costs are: 46.8 hours for the DQN, 18.9 hours for the DDQN, and 12.1 hours for the PPO. Figure 4: Reward evolutions The generated average rewards per recording log iteration of the last 100 decision steps are shown in Fig. 4. To make the comparison observable, we smoothed the rewards to represent their evolution, which increases over iterations. Overall, PPO outperforms the other two algorithms, especially during the two periods before 4,000 iterations and after 8,000 iterations, which almost correspond to low and high traffic load scenarios. Compared to DQN, the PPO and DDQN algorithms have relatively more stable and larger mean rewards (<<-50) for the last periods. Small negative rewards indicate that collisions, failures, or risky joining occur very occasionally. We present the operational performances of the three algorithms in Fig. 5. The evolutions of success, collision, and failure rates are shown respectively in Fig. 5(a), (b), and (c). By using a smoothing spline fit, they exhibit similar tendencies where the success rates increase, while the failure and collision rates decrease over time. Compared to DQN and DDQN, obviously, PPO achieves superior performance, which has not only the fastest convergence (around 3,000 iterations) but also the most efficiency with the highest success rate and lowest failure/collision rates. DDQN has relatively lower collision rates than DQN. On the contrary, its performance in terms of success rates and failure rates is a bit worse than DQN. Fig. 5(d) represents the convergence of decision steps per event that occurred. PPO decreases the number of decision steps for convergence, while DQN and DDQN are opposites. They all fall in the range of 10 ∼ 15 decision steps, namely between 5 and 7.5 seconds (as mentioned, one decision step is 0.5 seconds). Among them, PPO seems more conservative for platoon joining by taking a few more steps. This can also be seen in Fig. 5(e), where its cumulative number of events is less than the other two algorithms entirely. From this, it explains well why PPO consumed so much less time to complete the training. Regarding the value-based DQN and DDQN algorithms, initially, the agent randomly selects actions to explore the environment with a strong possibility of a high exploration rate ϵε. In cases where the selected vehicle’s target position is behind, for instance, any early lane-change manoeuvre will lead to either a collision or a join in the wrong position. This is why, at the beginning of training, the failure rate is very high (>70%>70\%), and the collision rate reaches 10%10\%. Successful join maneuvers occur occasionally (20%20\%). With ϵε decreasing exponentially over time, the probability (i.e., 1−ϵ1-ε) of selecting a greedy action gets higher. The agent learns to successfully join the platoon, reaching approximately 98%98\%, and the failure and collision rates tend to be smaller near zero. We should mention that at the end of training, the success rate still oscillates because we keep a small exploration rate (ϵmin _min = 0.01) to ensure continuous learning. This means the agent selects random actions with a probability of 1%1\%, which may lead to collisions or failures occasionally. (a) Success rate (b) Failure rate (c) Collision rate (d) Decision steps per event (e) Events Figure 5: Comparisons of training performance. 4.3 Model testing analysis 4.3.1 Joining manoeuvre and speed synchronization We analyze the joining maneuver based on the speed profiles of the joiner samples, as illustrated in Fig. 6. The solid lines represent the joiners staying on the mixed lane or starting the lane change (controlled by the deep RL algorithm), and the dotted lines represent ongoing lane changes and catching up with the last members of the platoons (controlled by CACC within the PLEXE). In most cases, CAVs on the mixed lane are ahead of the target platoons when they are selected as joiners. That is why the joiner briefly decelerates to allow the target platoon to pass. Once the last platoon member has moved ahead, the joiner initiates a lane change and accelerates to catch up with the platoon. The joiner slows down again to synchronize with the platoon speed by adopting the CACC mechanism. Figure 6: Samples of joiners’ speed evolutions 4.3.2 Joining quality We first investigate joining quality regarding the average number of decision steps until completing the joining, as shown in Fig. 7. Boxplots use the standard 1.5×IQR whisker definition. Outlier markers are omitted for clarity. As previously shown in Fig. 5(d) PPO has more decision steps per event than the other two algorithms in convergence, we show in detail the distributions for different traffic load levels. With higher symmetric traffic loads in Fig. 7 (a), the number of decision steps gets smaller, especially for the DDQN. This may be explained by the decreased vehicle speeds due to high traffic density on both lanes. In cases with asymmetric loads (see Fig. 7 (b)), PPO takes many more decision steps for platoon joining than DQN and DDQN do when the traffic on the platoon lane is much more than that on the mixed lane, i.e., (500, 2000) and (1000, 2000). This difference mitigates a lot for the opposite demand cases. This implies that PPO’s control of platoon joining is more conservative and dynamic when the platoon lane has much higher traffic density than the mixed lane. To some extent, DQN and DDQN both reflect their joining possibilities within stable decision steps when meeting largely asymmetric traffic loads on highways. We then check the joining distance between the joiner and its vehicle ahead (i.e., the last member of the target platoon), as shown in Fig. 8. As a whole, the distributions of average joining distances seem very stable in symmetric loads, and they are located between 10 and 20 meters. In asymmetric loads, all three algorithms generate a bit longer distances for the cases of low traffic on the platoon lane, i.e., (1500, 500) and (2000, 500). No matter what, the observations are aligned with our previous stable-maneuver setting, where no penalty is valued in this range during the training procedure (see Eq. (10)). (a) Symmetric loads (b) Asymmetric loads Figure 7: Number of decision steps per event. (a) Symmetric loads (b) Asymmetric loads Figure 8: Joining distance between VeV_e and VtargetV_target 4.3.3 Robustness analysis The success, failure, and collision percentages of platoon joining under different traffic loads were recorded to assess the robustness of the three types of deep RL approaches, as shown in Fig. 9. The solid lines refer to the performance of the designed algorithms, whereas the dashed lines are those without taking the TTC penalty (i.e., r_r in Eq. (9)) into account. Obviously, PPO presents the best performance with more than 97.5% success rate (see Fig. 9(a)) and less than 1.0% collision rate (see Fig. 9(c)) for all levels of traffic loads. Unexpectedly, the success rates of DDQN and DQN drop a lot, even under 90.0% in the four traffic demand pairs of (500, 2000), (1000, 2000), (1500, 2000), and (2000, 2000). This can be explained by the large rates of failed joinings (including wrong joining positions and uncompleted joinings) due to the platoon lane with the highest traffic of 2000 veh/h (see Fig. 9(b)). Compared to the variants of the three algorithms without TTC penalty, there are no surprisingly large failures because of the removed TTC constraints. However, this causes most of their collision rates to be relatively larger than those of the designed algorithms with the TTC risky penalty, as shown in Fig. 9(c). Based on these results, the policy-based PPO algorithm seems more robust than the value-based DDQN and DQN algorithms, and PPO is even better when the TTC risky penalty is included in the reward function. (a) Success rate (b) Failure rate (c) Collision rate Figure 9: Model tests of algorithms with or without TTC penalty (r_r) in different traffic loads. 4.4 Safety shield for lane changes The importance of the risky maneuver penalty designed in the reward function was demonstrated above. Differently regarding this "inner" safety mode, we also intend to investigate external protection when the algorithms are designed without the inner TTC conditions, causing relatively high collision rates for the high traffic demands on the platoon lane (see Fig. 9(c)). A safety shield (noted as "S") beyond the lane-changing decisions made by the deep RL algorithms was implemented. The rear and front time-to-collision (TTC) conditions are both applied before platoon joining or speed adjustment on the mixed lane, following a decision by the learning algorithm, with the same settings as in the reward function design. Specifically, they are min(TTC(Ve,Vlf),TTC(Ve,Vtarget))>=2min(TTC_(V_e,V_lf), TTC_(V_e,V_target))>=2 for the lane-change execution; and 2) min(TTC(Ve,Vfm),TTC(Ve,Vpm))>=1min(TTC_(V_e,V_fm), TTC_(V_e,V_pm)) >=1 for speed adjustement actions. The obtained results of metrics are given in Table 1. It shows that collision rates are almost reduced to zero (<< 0.001), and they are transferred as abandonment. Notice that very small parts of previous successful joinings are also abandoned. The abandoned joinings are related to previous dangerous but successful joinings with a small TTC (i.e., << 2 s). Again, PPO shows relatively higher success joining rates (>=>= 96.8%) with higher density traffic, followed by DDQN, which even demonstrates the best joining efficiency reaching 97.1% in the asymmetric traffic case of (500, 2000). Mention that the safe PPO approaches proposed in [18] only achieved lane-changing goals of 75% to 88% with free collisions. Table 1: Joining performance of algorithms after applying a safety shield (S) Metrics (%) (500, 2000) (1000, 2000) (1500, 2000) (2000, 2000) DQN (S instead of TTC penalty) Success 93.7 93.3 92.8 91.6 Failure 0.2 0.2 0.2 0.1 Collision 0.0 0.0 0.0 0.0 Abandonment 6.1 6.5 7.0 8.3 DDQN (S instead of TTC penalty) Success 97.1 96.3 96.3 95.5 Failure 0.1 0.3 0.1 0.2 Collision 0.0 0.0 0.0 0.0 Abandonment 2.8 3.4 3.6 4.3 PPO (S instead of TTC penalty) Success 94.4 96.1 96.9 96.8 Failure 0.4 0.4 0.3 0.3 Collision 0.0 0.0 0.0 0.1 Abandonment 5.2 3.5 2.8 2.8 5 Conclusion In this paper, we proposed a deep RL-based framework for optimizing platoon joining on highways with mixed traffic. We primarily implemented the typical value-based DQN and DDQN algorithms and the policy-based PPO algorithm for this control. Compared to the DQN and DDQN, the PPO achieves superior performance on model training and testing, although it takes a few more decision steps (about +10%) for platoon joining and longer distances (+ 0.9 ∼ 1.2 m.) to target platoons on average for the model tests. Concretely, the PPO performs excellently in all cases of traffic load testing, with about 98% of successful joinings and less than 1% for both failures and collisions. Regarding some quite degraded results of DQN and DDQN when the platoon load is set to 2,000 veh/h where a very short traffic head time is generated, we adopted strict conditions set by the rear/front TTC constraints before implementing the joining decisions that were learned by our approaches. The success rates under this safety shield for those specific cases are eventually over 92% for DQN and 96% for DDQN with guarantees of free collisions. Despite the generality of the proposed deep RL modeling framework for CAV platoon joining and results comparison by the proposed deep RL algorithms, a limitation of this study is related to the definite discrete action space for speed adjustments and its lack of optimization. Although driving comfort was considered through the inclusion of a jerk penalty in the reward function, joining efficiency is not confirmed as the ego vehicle’s speed trajectory is not optimized explicitly. Future work will focus on enhancing the reward design by incorporating speed trajectory optimization objectives, such as energy efficiency and time-efficient platoon joining globally. Furthermore, the proposed framework will be extended by developing an efficient strategy for CAVs to safely and effectively exit platoons. References [1] Ahmed, H. U., Huang, Y., Lu, P., Bridgelall, R., 2022. Technology developments and impacts of connected and autonomous vehicles: An overview. Smart Cities, 5(1), p. 382-404. [2] Jiang, L., Xie, Y., Wen, X., Chen, D., Li, T., Evans, N. G., 2021. Dampen the stop-and-go traffic with connected and automated vehicles–a deep reinforcement learning approach. In: 2021 IEEE 7th International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS), p. 1-6. [3] Sala, M., Soriguera, F., 2021. Capacity of a freeway lane with platoons of autonomous vehicles mixed with regular traffic. Transportation research part B: methodological, 147, p. 116-131. [4] Yang, B., Chen, J., Zhang, J., Zhou, B., Zhang, J., Ji, H., 2025. The effect of local platoon control strategy on the stability of mixed traffic flow. Transportmetrica B: Transport Dynamics, 13(1), p. 2496828. [5] Li, W., Ding, H., Xu, N., Song, Z., Zhang, J., 2024. A time and energy efficient merging control for platoon formation of connected and automated electric vehicles at on-ramps. Nonlinear Dynamics, 112, p. 1–24. DOI: 10.1007/s11071-023-09238-4. [6] Ding, J., Li, L., Peng, H., Zhang, Y., 2019. A rule-based cooperative merging strategy for connected and automated vehicles. IEEE Transactions on Intelligent Transportation Systems, 21(8), p. 3436-3446. DOI: 10.1109/TITS.2019.2928969 [7] Liu, P., Kurt, A., Ozguner, U., 2018. Distributed model predictive control for cooperative and flexible vehicle platooning. IEEE Transactions on Control Systems Technology, 27(3), p. 1115-1128. DOI: 10.1109/TCST.2018.2808911 [8] Huang, Z., Zhuang, W., Yin, G., Xu, L., Luo, K., 2019. Cooperative merging for multiple connected and automated vehicles at highway on-ramps via virtual platoon formation. In: Proceedings of the 2019 Chinese Control Conference (C), p. 6709–6714. DOI: 10.23919/ChiCC.2019.8866378. [9] Wang, J., Hu, C., Zhao, J., Zhang, L., Han, Y., 2024. Deep Q-Network-enabled platoon merging approach for autonomous vehicles. Transportation Research Record, 2678(7), p. 17–31. DOI: 10.1177/03611981231203229. [10] Ye, F., Cheng, X., Wang, P., Chan, C.-Y., 2020. Automated lane change strategy using proximal policy optimization-based deep reinforcement learning. CoRR, abs/2002.02667. Available at: https://arxiv.org/abs/2002.02667. [11] Kolat, M., Bécsi, T., 2024. Cooperative MARL-PPO approach for automated highway platoon merging. Electronics, 13(15). DOI: 10.3390/electronics13153102. [12] Zhou, W., Chen, D., Yan, J., Li, Z., Yin, H., Ge, W., 2022. Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic. Autonomous Intelligent Systems, 2(1), 5. [13] Zhang, J., Chang, C., Zeng, X., Li, L., 2022. Multi-agent DRL-based lane change with right-of-way collaboration awareness. IEEE Transactions on Intelligent Transportation Systems, 24(1), p. 854–869. [14] Wang, S., Wang, Z., Jiang, R., Zhu, F., Yan, R., Shang, Y., 2024. A multi-agent reinforcement learning-based longitudinal and lateral control of CAVs to improve traffic efficiency in a mandatory lane change scenario. Transportation Research Part C: Emerging Technologies, 158, p. 104445. [15] Zhou, W., Chen, D., Yan, J., Li, Z., Yin, H., Ge, W., 2021. Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic. CoRR, abs/2111.06318. Available at: https://arxiv.org/abs/2111.06318. [16] Shi, Y., Dong, H., He, C. R., Chen, Y., Song, Z., 2025. Mixed vehicle platoon forming: A multi-agent reinforcement learning approach, 12(11), p. 16886-16898. IEEE Internet of Things Journal. [17] Prathiba, S. B., Raja, G., Dev, K., Kumar, N., Guizani, M., 2021. A hybrid deep reinforcement learning for autonomous vehicles smart-platooning. IEEE Transactions on Vehicular Technology, 70(12), p. 13340–13350. [18] Krasowski, H., Wang, X., Althoff, M., 2020. Safe reinforcement learning for autonomous lane changing using set-based prediction. IEEE 23rd international conference on Intelligent Transportation Systems, p. 1-7. [19] Kim, J., Lim, D., Seo, Y., So, J., Kim, H., 2023. Influence of dedicated lanes for connected and automated vehicles on highway traffic flow. IET Intelligent Transport Systems, 17(4), p. 678-690. [20] Zhang, X., Wu, L., Liu, H., Wang, Y., Li, H., Xu, B., 2023. High-speed ramp merging behavior decision for autonomous vehicles based on multi-agent reinforcement learning. IEEE Internet of Things Journal, 10(24), p. 22664–22672. [21] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A., 2013. Playing Atari with Deep Reinforcement Learning. CoRR, abs/1312.5602. Available at: http://arxiv.org/abs/1312.5602. [22] Segata, M., Cigno, R. L., Hardes, T., Heinovski, J., Schettler, M., Bloessl, B., Dressler, F., 2022. Multi-technology cooperative driving: An analysis based on PLEXE. IEEE Transactions on Mobile Computing, 22(8), p. 4792-4806. [23] Ducrocq, R., Farhi, N., 2023. Deep reinforcement Q-learning for intelligent traffic signal control with partial detection. International journal of intelligent transportation systems research, 21(1), p. 192-206. [24] Dinneweth, J., Boubezoul, A., Mandiau, R., Espié, S., 2026. Archicool: Driver model based on the selective empathy. IEEE Transactions on Intelligent Transportation Systems, 27(3), p. 3357-3368. [25] Zhou, J., Yan, L., Yang, K., 2024. Enhancing system-level safety in mixed-autonomy platoon via safe reinforcement learning. IEEE Transactions on Intelligent Vehicles, p. 1-13. DOI: 10.1109/TIV.2024.3373512 Appendix A Configuration Table A.1: Configuration for the DQN and DDQN algorithms Hyperparameter Value/Definition Model type Fully connected online/target neural networks (MLP) Hidden layers 4 Hidden units [[32, 64, 64, 32]] Activation function LeakyReLU Output layer Linear Input dimension 15 Output dimension 6 Optimizer Adam Loss function SmoothL1Loss Learning rate α 0.0002 Discount factor γ 0.9 Soft update coefficient τ 0.001 Soft update frequency C 10310^3 Decay(ϵε) Exponential decay ϵstart _start 1 ϵmin _min 0.01 ϵdecay _decay 6×1056× 10^5 Mini-batch size M 64 Replay buffer size 2×1042× 10^4 Training warm-up steps 1.2×1031.2× 10^3 Maximum training steps 10610^6 Maximum steps per episode 10310^3 Decision step duration 0.5s (5 simulation steps) Table A.2: Configuration for the PPO algorithm Hyperparameter Value/Definition Model type Fully connected actor/critic neural networks (MLP) Hidden layers 4 Hidden units [[32, 64, 64, 32]] Activation function LeakyReLU Output layer Linear Input dimension (actor network) 15 Output dimension (actor network) 6 Input dimension (critic network) 15 Output dimension (critic network) 1 Optimizer Adam Actor learning rate αactor _actor 0.0001 critic learning rate αcritic _critic 0.001 Discount factor γ 0.9 Clipping coefficient ϵclip _clip 0.1 GAE coefficient λ 0.95 Critic loss coefficient c1c_1 0.5 Entropy coefficient c2c_2 0.01 Epochs K 5 Mini-batch size M 256 Rollout size 2048 Training warm-up steps 1.2×1031.2× 10^3 Maximum training steps 10610^6 Maximum steps per episode 10310^3 Decision step duration 0.5s (5 simulation steps) Table A.3: Configuration for traffic scenarios Hyperparameter Value/Definition Traffic load per lane Varying from 500 veh/h to 2,000 veh/h, with an increment of 500 veh/h CAV penetration rate (on the mixed lane) 50% Communication radius R 200 m Human driver imperfection 0.5 Rear distance search δ1 _1 150 m Front distance search δ2 _2 5 m Intervehicle distance (CACC) 5 m Speed limits on lane vmv_m = 100 km/h; vpv_p = 120 km/h