Paper deep dive
Learning-Based Behavior Planning for Automated Driving: Real-World Integration and Deployment
Jean-Pierre Busch, Guido Linden, Jan Bergmann, Lutz Eckstein
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent research in machine and deep learning has shown the potential of learningbased motion planning approaches to improve the driving behavior of automated vehicles, especially in complex environments. However, their complex nature and lack of transparency can hinder explainability and trustworthiness and complicate safety assurance. Motivated by these challenges, we propose a hybrid planning architecture that combines the advantages of machine learning with the verifiability and the determinism of classical approaches. Specifically, we developed a deep neural network to interpret complex traffic scenes and propose driving behavior, while an optimization-based supervision layer validates this proposal and enforces explicit drivability and safety constraints. We evaluate the learned planner's driving behavior in open-loop studies on real-world urban data, discuss system integration aspects for stable closed-loop operation, and report results from real-world deployment on our research vehicle karl..
Tags
Links
- Source: https://arxiv.org/abs/2608.12198v1
- Canonical: https://arxiv.org/abs/2608.12198v1
Trouble viewing inline? Open PDF directly →
Full Text
52,531 characters extracted from source content.
Expand or collapse full text
Learning-Based Behavior Planning for Automated Driving: Real-World Integration and Deployment Jean-Pierre Busch Thanks: Both authors are doctoral researchers at the Institute for Automotive Engineering (ika), RWTH Aachen University, Germany. firstname.lastname@ika.rwth-aachen.de Guido Linden Jan Bergmann Thanks: Jan Bergmann contributed to this work during his master’s thesis at ika and is now with the Chair of Automotive Technology (FTM), Technical University of Munich. Lutz Eckstein Thanks: Lutz Eckstein is head of the Institute for Automotive Engineering (ika). Abstract Recent research in machine and deep learning has shown the potential of learning-based motion planning approaches to improve the driving behavior of automated vehicles, especially in complex environments. However, their complex nature and lack of transparency can hinder explainability and trustworthiness and complicate safety assurance. Motivated by these challenges, we propose a hybrid planning architecture that combines the advantages of machine learning with the verifiability and the determinism of classical approaches. Specifically, we developed a deep neural network to interpret complex traffic scenes and propose driving behavior, while an optimization-based supervision layer validates this proposal and enforces explicit drivability and safety constraints. We evaluate the learned planner’s driving behavior in open-loop studies on real-world urban data, discuss system integration aspects for stable closed-loop operation, and report results from real-world deployment on our research vehicle karl.. Keywords: automated driving, hybrid planning architecture, behavior planning, deep learning ATC Aldenhoven Testing Center ADS automated driving system ADE average displacement error BEV bird’s-eye view DNN deep neural network GRU gated recurrent unit HD high-definition ML machine learning MLP multilayer perceptron MLOps machine learning operations MPC model predictive control MCTS Monte Carlo Tree Search OCP optimal control problem PID proportional-integral-derivative POMDP partially observable Markov decision process PRM probabilistic roadmap RRT rapidly-exploring random tree V2X vehicle-to-everything VRU vulnerable road user 1 Introduction Although companies such as Waymo [26] and Zoox [33] are already operating automated vehicles in U.S. cities, motion planning remains one of the central research topics for urban automated driving. According to Donges [11], the vehicle guidance task can be divided into the subtasks of navigation, guidance, and stabilization. Nowadays, the most significant challenges are faced at the guidance level: incorporating the vehicle’s environment – in particular, other traffic participants – into behavior and trajectory planning makes the task highly variable and considerably enlarges the relevant solution space. Traditionally, behavior and trajectory planning algorithms are realized using search-, sampling-, or optimization-based methods, which offer deterministic behavior and allow the system to enforce clear and transparent constraints. However, in highly interactive urban traffic, purely rule-based behavior design quickly becomes difficult to scale: the long tail of corner cases, complex interactions, and the need for continuously evolving driving behavior require substantial engineering effort. This is a key reason why data-driven planning approaches for automated vehicles are becoming increasingly important [6]. In particular, ML (ML) models can capture implicit interaction patterns between the vehicle and its environment that are difficult to encode in rule- or optimization-based approaches. Additionally, these models can be continuously improved via MLOps (MLOps) pipelines. The way in which ML models are integrated into vehicle guidance ranges from modular system architectures that integrate ML components for specific functions (e.g., tactical decision making) to end-to-end approaches that operate directly on raw sensor data and jointly address perception, situation understanding and planning [30, 15]. Regardless of whether ML is integrated as modular components or as an end-to-end model, all approaches face the same core challenge: they must satisfy system-level requirements such as trustworthiness, explainability, accountability, drivability, traffic-rule compliance, and safety. This has also sparked standardization efforts (e.g., ISO/TS 5083:2025) that emphasize architectural measures and safety layers around ML components [16]. Building on this motivation, this paper proposes a hybrid planning architecture that combines deep learning-based behavior planning with an optimization-based trajectory supervision, aiming to exploit the generalization capabilities of learned behavior generation while retaining deterministic, constraint-based safeguarding of the resulting trajectories. Beyond proposing the architectural concept, this work details its concrete implementation, demonstrates its integration and operation on the research vehicle karl. [4], and releases substantial parts of the resulting planning framework as open source software within the OpenADS11 1 https://github.com/openads-project ecosystem. 2 Related Work This section reviews related work on motion planning for automated vehicles, starting with classical (search-, sampling-, and optimization-based) approaches, moving on to learning-based planners, and concluding with hybrid architectures that combine ML-driven approaches with deterministic supervision. Rule-based motion planners have been used since the early days of automated driving, including the DARPA Grand- and Urban Challenges. For example, the winning system of the DARPA Grand Challenge 2005 used a search procedure over dynamically feasible trajectory candidates and selected solutions via cost aggregation [27]. Besides search-based planning on discretized state lattices, sampling-based motion planning methods (e.g., RRT [19] and RRT* [17]) generate candidate trajectories by sampling the state/control space and selecting the best candidate under a cost function and constraints. For tactical decision making under uncertainty, some ADS formulate behavior planning as a POMDP (POMDP) and solve it via MCTS (MCTS), followed by trajectory optimization to ensure feasibility [18]. The main advantages of these approaches are explicit constraint handling and a traceable decision process, supporting verification and safety argumentation. On the other hand, learning-based planners aim to overcome the limited scalability of manually encoded rule sets in complex urban environments. Over the past years, a wide range of model architectures and data representations has been explored. Early and widely used approaches represent the scene as a top-down raster image and apply convolutional backbones. ChauffeurNet, for instance, learns trajectories from rasterized scene representations and uses synthetic perturbations and additional loss terms to mitigate compounding errors in closed-loop operation [2]. More recent approaches use vectorized scene encodings and transformer backbones. For example, PlanT encodes a scene as a set of planar object tokens and predicts future waypoints with a transformer encoder and recurrent decoder [24]. Vectorized representations that encode agents and map geometry as polylines and lane graphs have become a de facto standard for scene encoding [12, 20]. Building on them, imitation-learning planners combine attention-based scene fusion [21] with transformer backbones and report strong results on large-scale planning benchmarks [9, 8], with some approaches modeling multi-agent interaction explicitly during decoding [14]. Besides such modular approaches, end-to-end driving policies aim to map raw sensor inputs to driving behavior and may expose intermediate reasoning traces. EMMA is a representative multimodal end-to-end approach that outputs driving-task-specific predictions from camera inputs [15]. NVIDIA’s Alpamayo is a recent large-scale foundation model that also uses camera images as input, but predicts trajectories rather than direct vehicle control commands alongside textual reasoning traces [30, 23]. To combine the strengths of rule- and learning-based planners, recent works increasingly explore hybrid planning architectures in which learning-based behavior is validated, refined, or replaced using deterministic checks or optimization [3, 7, 29, 13]. SafetyNet [29], for example, combines a learned planner with an explicit safety layer that evaluates feasibility, legality, and collision risk and, upon violations, switches to a non-learning-based fallback trajectory generator. In practice, the learning-based planner is based on a structured scene representation (e.g., graph-/point-based encoders with transformer-style decoding), while the fallback is realized through a trajectory generation method based on [31]. Gariboldi et al. propose a hybrid design in which a lightweight MLP (MLP) predicts a trajectory proposal that is then consistently refined by a downstream model-predictive trajectory optimization to enforce feasibility and collision avoidance [13]. This line of work also connects to recent guidance on the safe integration of ML in ADS. For example, ISO/TS 5083:2025 provides a state-of-the-art reference for integrating ML components into ADS via well-defined safety layers and architectural measures [16]. This work follows a similar hybrid principle by employing a learning-based behavior planner and an optimization-based supervision layer within a modular ADS. 3 Planning Architecture The main objective of this work is to develop a planning architecture that combines the performance potential of learning-based behavior planning with the verifiability and deterministic constraint enforcement of classical planning methods. In particular, we address the system-level challenges of learning-based approaches highlighted in Section 1 and translate them into architectural measures that enable stable closed-loop behavior in a real-world vehicle. The architecture is designed in a modular manner with clearly separated responsibilities, allowing the learning-based behavior planner to be exchanged and updated without modifying the safety-critical supervision and fallback modules. This separation also facilitates an MLOps workflow in which operational data can be collected, incorporated into subsequent training and validation cycles, and used for the continuous refinement of the behavior planner. In this section, the overall hybrid architecture and its interfaces are described, while Section 4 then details the learning-based module. Fig. 1 shows a simplified view of our proposed automated driving stack with a focus on the planning architecture. Upstream modules such as localization, perception, map processing, and V2X (V2X) communication provide the sensing inputs that are aggregated in an environment model. Based on this scene representation, the learning-based behavior planning module proposes a reference trajectory that is subsequently refined by the trajectory supervision module under explicit drivability and safety constraints. In the context of this paper, a reference trajectory refers to a tactical decision that is not necessarily directly executable by the vehicle, but is intended to achieve the desired behavior, such as stopping behind another vehicle or performing a lane change. Parallel to the learning-based planning branch, a non-learning-based module reliably plans a drivable trajectory, acting as safety fallback. A switch selects the reasonable trajectory that is then tracked by the trajectory controller and executed by the vehicle actuation. In the following, the central rule-based modules of the architecture are described in more detail. All presented modules are implemented as C++ ROS 2 nodes and individually containerized using Docker for reproducible deployment [5]. Substantial parts of the proposed hybrid planning architecture are released as open source software within OpenADS, where they serve as an initial planning baseline. Figure 1: Simplified automated driving stack focusing on the hybrid planning architecture combining learning-based and rule-based planning modules. Trajectory Supervision The trajectory supervision module safeguards the learning-based reference trajectory before it is passed to downstream modules. Its role in the architecture is to transform the reference provided by the behavior planning module into a drivable trajectory while enforcing explicit safety, traffic-rule compliance, drivability, and vehicle-dynamic constraints. For this purpose, we developed a trajectory optimization framework22 2 https://github.com/openads-project/trajectory_optimization that formulates and solves an OCP (OCP). To ensure general drivability, a kinematic single-track model is used. The state vector x, control input vector u, and system dynamics f(,)f(x,u) are defined as follows: =[xysvaψδ],=[jα],˙=f(,)=[vcos(ψ)vsin(ψ)vajvLtan(δ)α].x= bmatrixx\\ y\\ s\\ v\\ a\\ ψ\\ δ bmatrix, = bmatrixj\\ α bmatrix, x=f(x,u)= bmatrixv (ψ)\\ v (ψ)\\ v\\ a\\ j\\ vL (δ)\\ α bmatrix. (1) Here, (x,y)(x,y) denotes the position, s the distance along the trajectory, v, a, and j the longitudinal velocity, acceleration, and jerk, respectively, ψ the heading angle, δ the steering angle, α the steering rate, and L the wheelbase. For the discretized OCP, the cost function J is defined as: J J =∑k=0N−1ℓk(k,k)+ℓN(N) = _k=0^N-1 _k(x_k,u_k)+ _N(x_N) (2) ℓk _k =qk(wlat⋅elat,k2+wv⋅ev,k2+wa+⋅ea+,k2+wa−⋅ea−,k2+walat⋅ealat,k2) =q_k (w_lat· e_lat,k^2+w_v· e_v,k^2+w_a^+· e_a^+,k^2+w_a^-· e_a^-,k^2+w_a_lat· e_a_lat,k^2 ) +wj+⋅ej+,k2+wj−⋅ej−,k2+wjlat⋅ejlat,k2+wα⋅eα,k2 +w_j^+· e_j^+,k^2+w_j^-· e_j^-,k^2+w_j_lat· e_j_lat,k^2+w_α· e_α,k^2 ℓN _N =wψ,N⋅eψ,N2 =w_ψ,N· e_ψ,N^2 The stage cost ℓk _k is evaluated at each discretization step k along the prediction horizon. The residuals elate_lat and eve_v penalize lateral path and velocity deviation with respect to the spatially matched reference trajectory. The terms ea+e_a^+ and ea−e_a^- distinguish between positive and negative longitudinal acceleration, while ealate_a_lat penalizes lateral acceleration. Similarly, ej+e_j^+ and ej−e_j^- distinguish between positive and negative longitudinal jerk, ejlate_j_lat penalizes lateral jerk, and eαe_α penalizes steering-rate actuation. The coefficients wiw_i define the relative importance of the cost terms. The factor qkq_k denotes a dynamic weighting factor along the prediction horizon and is applied to the reference-tracking and acceleration-related terms. The terminal cost ℓN _N penalizes the final heading-angle deviation from the reference path. Besides the system dynamics and cost function, the OCP incorporates constraints on states, controls, obstacle avoidance, drivable space, and vehicle-dynamic feasibility. State and control constraints impose bounds on velocity, longitudinal acceleration, steering angle, jerk, and steering rate. Drivability constraints ensure the vehicle remains within the drivable space, while obstacle constraints guarantee collision-free trajectories with respect to dynamically predicted objects. Additional nonlinear vehicle-dynamics constraints, such as limits on absolute acceleration, are included to ensure physically feasible motion. The optimization framework is built on acados [28], which is used to formulate the nonlinear OCP and generate solver libraries for online execution within a ROS 2 node. Through this formulation, the trajectory supervision module transforms the reference trajectory into a drivable and collision-free trajectory with respect to the modeled constraints and predicted environment. Safety Fallback In addition to the learning-based planning path, the proposed architecture includes a deterministic safety fallback. Its role is to provide a reliable, rule-based planning branch that continuously produces a drivable trajectory. If a regular fallback trajectory cannot be generated due to invalid inputs, unavailable information, or unknown traffic situations, the module initializes a safe-stop trajectory. The fallback is therefore a complementary safety layer for degraded operation and is not intended to replace the learned behavior planning during nominal operation. For the fallback path, we developed a simple centerline-based planner33 3 https://github.com/openads-project/simple_planner. Initially, the planner constructs a reference trajectory along a given map-based centerline, including encoded speed limits as reference velocity. Based on specified traffic situations, such as approaching traffic lights or conflicts with surrounding objects, the longitudinal behavior is adapted. This is achieved by reducing the velocity profile down to a complete standstill, while the lateral reference remains tied to the centerline. The presented planner is intentionally designed as a deterministic, interpretable, and computationally efficient fallback, providing a robust and drivable trajectory in, e.g., degraded situations. The module that determines which trajectory is to be tracked by the downstream trajectory controller is represented by the switch shown in Fig. 1. In principle, it is designed to compare, evaluate, and select between multiple trajectories based on a set of arbitrary rules, for example, based on planner competence or geofences. In the presented setup, the supervised output of the learning-based planner is forwarded to the trajectory controller by default. If the reference trajectory is of insufficient quality and the supervision module is therefore unable to ensure drivability and safety, the switch forwards the output of the deterministic fallback path. In these cases, it is also possible to collect the necessary data for generating new training sets so that the behavior planning can handle these scenarios in the future as well. This is a central point for the continuous improvement of the system. Trajectory Controller The trajectory controller is the final module of the presented architecture and converts the selected drivable trajectory into actuator commands for vehicle execution. It compensates tracking errors, model inaccuracies, and external disturbances while preserving the trajectory selected by the upstream planning modules as closely as possible. To achieve this, we developed a cascaded PID (PID)-based Ackermann trajectory controller44 4 https://github.com/openads-project/ackermann_trajectory_control. It is structured into a longitudinal and a lateral control path, both combining feed-forward terms from drivable trajectories provided by trajectory supervision or safety fallback with feedback PID control. The required longitudinal and lateral target quantities are obtained by linear interpolation using dedicated look-ahead times. Furthermore, PID gains are scheduled over the operating velocity range, allowing the controller behavior to be adapted to varying vehicle dynamics. The longitudinal path tracks the target velocity by means of a velocity PID controller. In addition, the target acceleration is used as a feed-forward contribution. The resulting acceleration command is constrained by constant longitudinal acceleration and jerk limits. The lateral path follows a cascaded feedback structure. First, a displacement PID controller converts the lateral trajectory deviation into a desired heading correction. A subsequent heading PID controller combines this correction with the trajectory’s target heading and generates a desired yaw rate, which is mapped to a feedback curvature command using an inverse single-track model. Feedback and feed-forward curvature terms are combined and subsequently constrained by curvature, curvature rate, and curvature acceleration limits and finally converted into Ackermann steering angle commands. To improve overall robustness, the planning architecture follows a bi-level stabilization concept inspired by Werling et al. [32]. During regular operation, trajectory supervision uses the interpolated state from the previously generated trajectory as the initial state for the next optimization. Whenever predefined thresholds in longitudinal or lateral tracking are exceeded, replanning from the actual vehicle state is initiated. In this way, small deviations are handled by the feedback controller, whereas larger deviations initiate a new trajectory from the actual vehicle state. 4 Learning-based Behavior Planning After presenting the overall architecture and its rule-based modules, this section focuses on the developed behavior planning module. By learning from behavioral data, it can capture interaction patterns and driving styles in dense, complex urban environments, where manually defined rules and heuristics often struggle to cover the multitude of scenarios. The presented behavior planner is implemented as a DNN (DNN) in PyTorch [22] and predicts a reference trajectory based on a policy learned through supervised learning from selected data. The neural network does not operate on raw sensor data, but on the processed output of the upstream environment model, including HD (HD) map, navigation, tracked object list, and ego state information. In the following, the ego vehicle and the surrounding dynamic objects, such as other vehicles or pedestrians, are referred to as agents. The network architecture builds on a vectorized scene representation derived from the previously described data [12, 20] and adopts design principles from recent imitation-based transformer planners [9, 8], including attention-based scene fusion mechanisms [21], adapted from motion forecasting to behavior planning and to the interfaces of the hybrid architecture. Fig. 2 illustrates an abstract representation of the implemented network architecture while the following paragraphs detail the input and output representation, the training objective, and the used datasets. The model evaluation then follows in Section 5. Figure 2: Abstract architecture of the DNN: Agent history and map / route information are encoded separately, fused through an attention-based fusion block, and decoded into a kinematically constrained reference trajectory. Additional outputs are object predictions, interpretable attention weights and map-grounding outputs (not illustrated). Input & Output Representation The input scene representation is described in an ego-centric BEV (BEV), where all elements are expressed relative to the current ego pose and the x-axis is aligned with the driving direction. Each agent is represented not by a single state but by a short state history, including position, heading, velocity, and bounding box size over the last five temporal steps at a spacing of Δt=0.5 s t=$0.5\,s$, i.e. the past 2 s2\,s including the current step. The object classification is kept as a separate categorical attribute. In addition to the agent information, the map and route information are also defined in the ego-centric frame. The vectorized map comprises lane centerlines with their width and type, lane boundaries with their type and whether they may be crossed, and regulatory elements such as traffic lights, yield lines and right-of-way rules together with their current state. In addition, the map information contains the relations between centerlines, boundaries, and regulatory elements. The route is given as an ordered reference of the lanes the vehicle is meant to follow. The number of agents and map elements varies between scenes, which is handled by masking. Due to this object-level scene representation, the domain gap between data sources stays comparatively small, which makes it possible to train on various real-world or even simulation datasets. Given this representation, the planner primarily outputs a reference trajectory over an 8 s8\,s horizon at the same 0.5 s0.5\,s spacing, represented as a sequence of positions and velocities. Rather than regressing independent waypoints, the decoder builds the trajectory stepwise. At each step it predicts a velocity and a heading change and integrates them using a simple kinematic model [10]. Per-step increments are bounded to physically reasonable ranges. Additionally, two auxiliary tasks are learned jointly with the trajectory: a short-horizon prediction of the surrounding objects, produced by a separate decoder, and a set of map-grounding quantities read from the ego decoder’s per-step state, namely the lateral offset to the route and the distance to the drivable corridor. Both encourage the model to encode the interaction and the map structure. The attention weights of the fusion and the decoder remain accessible as an interpretability output. Fig. 3 visualizes the scene representation and the predicted trajectory for an exemplary validation sample. Figure 3: Example of the scene representation in the ego-centric frame, shown together with the predicted reference trajectory and the corresponding velocity graph on the right. Route, Map, Ego (history & future), Objects (history), Predicted Reference Trajectory. The solid green line represents the reference line of a green traffic light. Training Objective As already mentioned, the planner is trained by imitation: for each recorded scene, the network is trained to predict the corresponding expert trajectory, typically originating from human driving behavior. Let tp_t, vtv_t, and ψt _t be the position, velocity, and heading that the decoder produces for step t=1,…,Tt=1,…,T, and let ^t p_t, v^t v_t, ψ^t ψ_t be the corresponding targets, obtained from the expert trajectory. The loss combines four imitation terms with an auxiliary multi-task term, ℒ=wpℒpos+wvℒvel+wψℒψ+wΔψℒΔψ+ℒaux,L=w_p\,L_pos+w_v\,L_vel+w_ψ\,L_ψ+w_ ψ\,L_ ψ+L_aux, (3) with the trajectory terms ℒpos _pos =1T∑t=1T∥t−^t∥1, = 1T _t=1^T _t- p_t _1, ℒvel _vel =1T∑t=1T|vt−v^t|, = 1T _t=1^T v_t- v_t , (4) ℒψ _ψ =1|S|∑t∈S|wrap(ψt−ψ^t)|, = 1|S| _t∈ S ( _t- ψ_t) , ℒΔψ _ ψ =1|S|∑t∈S|wrap(Δψt−Δψ^t)|, = 1|S| _t∈ S ( _t- ψ_t) , (5) where Δψt=wrap(ψt−ψt−1) _t=wrap( _t- _t-1) is the per-step heading change, wrap(⋅)wrap(·) maps an angle to (−π,π](-π,π], and the heading terms are evaluated only on the moving steps S=t:v^t>vminS=\t: v_t>v_ \ with vmin=0.5 m/sv_ =$0.5\,m/s$, where the heading is well defined. The velocity and heading-change terms are weighted more strongly than position and absolute heading, since a faithful velocity profile and turn-in behavior are key to reproducing the reference trajectory, while the heading-change term acts directly on the decoder’s control increments. The auxiliary loss, ℒaux=waℒagent+wrℒroute+wcℒcorrL_aux=w_a\,L_agent+w_r\,L_route+w_c\,L_corr, consists of three lightly weighted masked regression terms supervising the auxiliary prediction heads. Their purpose is to shape the learned representation while keeping the primary focus on the imitation objective. Datasets and Trained Models Building on the previously described objective, three models on different datasets are trained. The DrivIng dataset [25] serves as the primary data source for training. It comprises three recordings of the same route, approximately 18 km18\,km long, captured at three different times (day, dusk, and night) with a total duration of approximately 105 min105\,min. The route passes through Ingolstadt in Germany, resulting in data that is characterized by urban traffic including diverse object interactions. Prior to training, object tracks were enriched with velocity estimates, and the route centerline and the relevant traffic lights were extracted from an HD map. Traffic light states were manually annotated. Finally, samples inconsistent with the intended urban driving context or lacking relevant future information were filtered out to reduce noise. Although the dataset is comparatively small, it enables careful curation of the training data, including detailed analysis and filtering unwanted behavior. Such a level of quality control is difficult to achieve with benchmark-scale datasets. At the same time, the limited dataset size naturally constrains the diversity of observed driving situations and therefore does not allow training a behavior planner that generalizes perfectly to arbitrary scenarios. To still compensate the reduced amount of data and further improve closed-loop performance, the data is augmented by adding noise to selected state variables of the ego vehicle and other dynamic objects, as well as applying dropout to objects that are not directly interacting with the ego vehicle. All three models are trained on the day and night recordings of the DrivIng dataset, while the dusk recording was held out for validation. This isolates generalization across varying traffic conditions while the route geometry stays fixed. This results in 34 43834\,438 training samples and 16 51816\,518 validation samples, corresponding to approximately 57 min57\,min of driving data for training and 27 min27\,min for validation. To study transferability to a different domain, two additional datasets were recorded on ika’s test track using our research vehicle karl. [4]. The first dataset, referred to as test track, captures driving without dynamic objects and represents the road geometry of the test track, while the second dataset (test track interaction) includes dedicated interaction scenarios with an additional vehicle. Separate recordings were used for training and validation, yielding 30833083/11361136 training/validation samples for the test track dataset and 66686668/23342334 samples for the test track interaction dataset. The first model is trained exclusively on DrivIng, while the second and third models additionally incorporate the test track and test track interaction datasets, respectively. This setup allows evaluating the overall quality of the learned behavior, its transferability to a new domain, and the effect of incrementally adding domain-specific data for road-geometry adaptation and interaction behavior. 5 Evaluation and Results This section evaluates the proposed learning-based planning approach, from isolated model performance to full vehicle integration. First, the trained models are evaluated in isolation to assess their performance independently of the automated driving stack and to select the model used for vehicle integration. Second, the integrated model is demonstrated and qualitatively evaluated within the overall hybrid architecture. 5.1 Model Evaluation As described in Section 4, the planner is trained by imitation to predict an 8 s8\,s reference trajectory in 0.5 s0.5\,s steps, together with the auxiliary outputs. Generalization is assessed in an open-loop manner by comparing the predicted reference trajectories with the recorded ego trajectories in the validation data. Tab. 1 reports the trajectory accuracy as the ADE (ADE) of position, velocity, and heading. On the DrivIng validation split, the errors remain small: over the 8 s8\,s horizon the position ADE is 1.83 m1.83\,m, the velocity ADE 0.56 m/s0.56\,m/s, and the heading ADE 2.6 °2.6\, . Since the decoder determines the position by integrating the predicted velocity and heading, small but persistent errors in these quantities accumulate over the horizon, so that a velocity offset of only 2 km/h2\,km/h already amounts to more than 4 m4\,m after 8 s8\,s. This is expected, as human drivers do not maintain a perfectly constant velocity. Table 1: Open-loop trajectory accuracy given as ADE and trajectory collision ratio. All values are evaluated on distinct temporal horizons (each cell: 4 s4\,s /8 s8\,s horizon). Rows are grouped by validation domain; within each group, the three trained models are listed based on their training data. The collision ratio is omitted for validation on test track due to absence of objects in this data. Training data Pos. ADE [mm] Vel. ADE [m/sm/s] Head. ADE [° ] Coll. [%\%] Validation: DrivIng DrivIng only 0.74 / 1.83 0.37 / 0.56 1.9 / 2.6 0.3 / 2.3 +test track 0.73 / 1.81 0.37 / 0.55 1.9 / 2.6 0.3 / 1.7 +test track interact. 0.71 / 1.78 0.35 / 0.53 2.1 / 2.9 0.3 / 1.2 Validation: test track DrivIng only 2.26 / 6.96 1.14 / 1.77 5.0 / 11.0 – +test track 0.66 / 1.31 0.32 / 0.41 1.7 / 2.0 – +test track interact. 0.72 / 1.62 0.35 / 0.49 2.0 / 2.6 – Validation: test track interaction DrivIng only 2.54 / 7.83 1.33 / 2.15 5.3 / 10.0 6.1 / 26.7 +test track 1.02 / 2.40 0.53 / 0.74 2.2 / 2.8 0.0 / 5.9 +test track interact. 0.78 / 1.70 0.39 / 0.51 2.3 / 2.9 0.0 / 0.0 Beyond trajectory accuracy, the learned behavior is analyzed, focusing specifically on traffic lights and their interaction with other agents: Across the entire DrivIng validation set, the model never proposes to cross a red light, and 2.3 %2.3\,\% of the predicted 8 s8\,s trajectories would have led to a collision, assuming that the ego vehicle followed the predicted trajectory and all other agents followed their recorded future trajectories (see Tab. 1). Given the continuous replanning in receding-horizon operation, the full trajectory horizon is never executed. Restricting the evaluation to the first 4 s4\,s therefore provides a more meaningful assessment in practice, with a lower corresponding collision rate of 0.3 %0.3\,\%. Fig. 4 visualizes the learned behavior on two validation frames, each paired with a minimally modified counterfactual version. In the first pair, the originally green traffic light is switched to red. In the second pair, a stopped lead vehicle is inserted ahead of the ego vehicle. All other scene elements remain unchanged. The inserted lead vehicle is placed 50 m50\,m ahead of the ego vehicle to ensure that collision avoidance is physically feasible. In both cases the planner adapts its prediction accordingly, stopping at the now-red traffic light and slowing down behind the inserted lead vehicle. Figure 4: Counterfactual validation examples. Top: changing the traffic light state from green to red causes the predicted trajectory to stop before the stop line. Bottom: inserting a stopped lead vehicle causes the planner to slow down behind the object. Route, Map, Ego (history), Objects (history), Predicted Reference Trajectory. Traffic light reference lines are colored according to their corresponding state. Transfer to a different domain is assessed on the test track recordings, whose road geometry is unseen by a model trained exclusively on DrivIng. Zero-shot, performance is clearly degraded because the training data does not sufficiently cover the new road geometry, which shows in the much larger heading and velocity errors (see Tab. 1; the heading ADE rises from about 2.6 °2.6\, to roughly 11 °11\, ), and the model collides frequently in the interaction scenarios (see Tab. 1; 26.7 %26.7\,\% at 8 s8\,s). When additionally trained on test track data, however, the model is able to follow the test track geometry, while the interaction behavior learned from DrivIng transfers effectively to the new domain. The resulting model remains collision-free over the 4 s4\,s horizon, with only a few number of residual collisions occurring over 8 s8\,s. A small number of test track interaction samples removes these residual collisions as well. The auxiliary tasks contribute to this transfer. Trained without them, the same +test track model still collides in 20.6 %20.6\,\% of the 8 s8\,s interaction trajectories, which the auxiliary objectives reduce to the 5.9 %5.9\,\% reported in Tab. 1. Conversely, incorporating the limited set of test track interaction examples does not degrade performance on DrivIng; instead, it yields a slight improvement, reducing the collision rate on the validation set from 2.3 %2.3\,\% to 1.2 %1.2\,\%. This result suggests that even small amounts of diverse training data can improve interaction behavior without measurable catastrophic forgetting, highlighting the potential for continuous, data-driven model improvement. For the vehicle integration described in Section 5.2, the third model is used, which is trained on all available data and exhibits the most consistent generalization performance across the domains. 5.2 Integration into a Real-World Demonstrator We deployed the proposed hybrid planning architecture on our research vehicle karl. (cf. Fig. 5). In contrast to purely simulation-based studies, deployment on a real-world demonstrator exposes the integrated system to realistic sensor artifacts, actuation constraints, and timing effects. It therefore enables a qualitative assessment of whether the individual modules can operate jointly under real-world conditions. At the current stage of development, vehicle operation is restricted to a proving ground. Accordingly, all real-world experiments reported in this paper were conducted on ika’s test track. Detailed descriptions of the research vehicle platform and the underlying software architecture are provided in [4]. Figure 5: ika’s research vehicle karl. used for the integration experiments. [4] To evaluate the integration of the proposed planning architecture, a route was defined on the test track that covers the main behaviors considered in this work. Since the test track is not equipped with traffic lights, traffic light-related behavior is not included in the evaluation. Instead, the defined scenario focuses on lane following, lane changing, turning, and interaction with another road user. Following an initial straight segment that includes a lane change, karl. performs two left turns. During the second turn, a challenger vehicle approaches from the right and is therefore given priority by karl.. Subsequently, karl. follows the challenger vehicle along the remaining route. The proposed planning architecture has been successfully integrated into karl.. During the conducted test runs, the behavior planning module generated trajectory predictions at a frequency of 10 Hz10\,Hz and provided references that followed the predefined route and did not cause collisions. Since the scenario requires reactions to another road user, the integration comprises not only the proposed planning modules but also the required sensor drivers and perception components. Existing software developed at the institute was used for this purpose. Fig. 6 illustrates exemplary trajectories generated during the initial interaction with the challenger vehicle at three consecutive time steps. Each snapshot shows the learning-based reference trajectory, the supervised trajectory, and the trajectory generated by the safety fallback. The differences in trajectory length primarily result from the different planning horizons of the corresponding modules: the trajectory supervision and safety fallback generate trajectories with a horizon of 5 s5\,s, whereas the behavior planning module predicts over a horizon of 8 s8\,s. Figure 6: Consecutive BEV snapshots of the integration experiment during the left-turn and right-of-way interaction. The orange area shows the learning-based reference, the blue line visualizes the trajectory obtained after trajectory supervision, and the red line shows the safety fallback trajectory computed in parallel. Qualitatively, the snapshots indicate that the supervised trajectory largely preserves the maneuver represented by the learning-based reference while converting it into a feasible trajectory for vehicle execution. In comparison, the safety fallback exhibits more conservative behavior, remaining closer to the lane center and decelerating more strongly when approaching the interaction area. While this behavior may provide an additional safety margin, it also results in more centerline-bound trajectories, particularly in curved road segments. The learning-based reference, by contrast, follows a smoother and less centerline-bound path through the curve, which is largely retained by the trajectory supervision. Overall, the test track demonstration serves as a proof of concept for the real-world integration of the proposed hybrid planning architecture. It provides qualitative evidence that the individual modules can operate jointly on the research vehicle under real-world sensing, actuation, and timing conditions. Within the considered scenario, the learning-based planner provides maneuver-level reference trajectories, the supervision layer transforms these references into feasible trajectories for vehicle execution, and the conservative fallback remains available as an additional safety mechanism. In addition to the results reported in this paper, the integrated system was demonstrated in real-vehicle operation at the autotech.agil final event [1], where the proposed planning architecture was operated in front of invited guests at the ATC (ATC). This public demonstration provides additional qualitative evidence that the full stack – from perception through learning-based behavior planning and trajectory supervision to actuation – can operate stably in a real-world demonstrator setting. 6 Conclusion This paper presented a hybrid planning architecture for automated driving that combines learning-based behavior planning with optimization-based supervision to provide drivable and safeguarded vehicle guidance. Beyond the architectural concept, the paper provides detailed information on the implemented software modules and their integration into an operational automated driving stack. To support reproducibility and reuse, selected components of the proposed planning framework are released as open source software within the OpenADS ecosystem. In addition, the developed ML model and its training pipeline are described, and an open-loop evaluation on real-world urban driving data as well as findings from its integration into the research vehicle karl. are reported. While the open-loop evaluation provides quantitative results on prediction accuracy, interaction behavior, and domain transfer, the test track experiments serve as a qualitative proof of concept for the integration of the proposed modules under real-world sensing, actuation, and timing conditions. Overall, the results highlight the potential of hybrid architectures to couple data-driven performance with deterministic supervision and safeguarding, while additionally enabling continuous, data-driven improvement of the driving behavior. Future work will focus on a broader quantitative evaluation of the integrated system in more diverse and complex traffic scenarios. This includes evaluating the planning architecture in public road traffic once the required approval is granted. 7 Acknowledgements We acknowledge the financial support by the German Federal Ministry of Research, Technology and Space (BMFTR) for autotech.agil (FKZ 1IS22088A) and by the European Union’s Horizon Europe Research and Innovation Programme for AIthena (Grant Agreement No. 101076754) and AIGGREGATE (Grant Agreement No. 101202457). References [1] Abschlussveranstaltung | autotech.agil. Note: Website External Links: Link Cited by: §5.2. [2] M. Bansal, A. Krizhevsky, and A. Ogale (2018) ChauffeurNet: learning to drive by imitating the best and synthesizing the worst. arXiv:1812.03079. Cited by: §2. [3] B. Brito, A. Agarwal, and J. Alonso-Mora (2022) Learning interaction-aware guidance for trajectory optimization in dense traffic scenarios. IEEE Transactions on Intelligent Transportation Systems 23 (10), p. 18808–18821. Cited by: §2. [4] J. Busch, L. Ostendorf, G. Linden, L. Reiher, T. Beemelmanns, B. Lampe, T. Woopen, and L. Eckstein (2026) Karl. – a research vehicle for automated and connected driving. In 2026 IEEE Intelligent Vehicles Symposium (IV), Cited by: §1, §4, Figure 5, Figure 5, §5.2. [5] J. Busch, L. Reiher, and L. Eckstein (2024) Enabling the Deployment of Any-Scale Robotic Applications in Microservice-Based Service-Oriented Architectures through Automated Containerization. 2024 IEEE International Conference on Robotics and Automation (ICRA). External Links: Document, Link Cited by: §3. [6] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li (2024) End-to-end autonomous driving: challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12), p. 10164–10183. External Links: Document Cited by: §1. [7] S. Chen, Y. Sun, D. Li, Q. Wang, Q. Hao, and J. Sifakis (2022) Runtime safety assurance for learning-enabled control of autonomous driving vehicles. In 2022 International Conference on Robotics and Automation (ICRA), p. 8978–8984. Cited by: §2. [8] J. Cheng, Y. Chen, and Q. Chen (2024) PLUTO: pushing the limit of imitation learning-based planning for autonomous driving. arXiv:2404.14327. External Links: Document Cited by: §2, §4. [9] J. Cheng, Y. Chen, X. Mei, B. Yang, B. Li, and M. Liu (2024) Rethinking imitation-based planners for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 14123–14130. External Links: Document Cited by: §2, §4. [10] H. Cui, T. Nguyen, F. Chou, T. Lin, J. Schneider, D. Bradley, and N. Djuric (2020) Deep kinematic models for kinematically feasible vehicle trajectory predictions. In 2020 IEEE International Conference on Robotics and Automation (ICRA), p. 10563–10569. Cited by: §4. [11] E. Donges (1982) Aspekte der aktiven sicherheit bei der führung von personenkraftwagen. Automobil-Industrie 27 (2). Cited by: §1. [12] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid (2020) VectorNet: encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11525–11533. Cited by: §2, §4. [13] C. Gariboldi, M. Corno, and B. Jin (2024) Hybrid imitation-learning motion planner for urban driving. In 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), p. 2578–2583. Cited by: §2. [14] Z. Huang, H. Liu, and C. Lv (2023) GameFormer: game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3903–3913. External Links: Document Cited by: §2. [15] J. Hwang, R. Xu, H. Lin, W. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp, Y. Zhou, J. Guo, D. Anguelov, and M. Tan (2024) EMMA: end-to-end multimodal model for autonomous driving. arXiv:2410.23262. External Links: Document Cited by: §1, §2. [16] (2025) Road vehicles – Safety for automated driving systems – Design, verification and validation. Standard International Organization for Standardization, Geneva, CH. Cited by: §1, §2. [17] S. Karaman and E. Frazzoli (2011) Sampling-based algorithms for optimal motion planning. The international journal of robotics research 30 (7), p. 846–894. Cited by: §2. [18] G. Kueppers, L. Kloeker, J. Busch, A. Kloeker, P. Peltzer, F. Thomsen, S. Klaudt, and L. Eckstein (2022) Future mobility applications in the komodnext and accord digital test fields. In 31st Aachen Colloquium Sustainable Mobility, Cited by: §2. [19] S. M. LaValle (1998) Rapidly-exploring random trees: a new tool for path planning. Research Report 9811. Cited by: §2. [20] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun (2020) Learning lane graph representations for motion forecasting. In European Conference on Computer Vision (ECCV), p. 541–556. Cited by: §2, §4. [21] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp (2023) Wayformer: motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 2980–2987. Cited by: §2, §4. [22] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, Cited by: §4. [23] M. Pavone (2026) Building autonomous vehicles that reason with nvidia alpamayo. Note: NVIDIA Technical Blog External Links: Link Cited by: §2. [24] K. Renz, K. Chitta, O. Mercea, A. S. Koepke, Z. Akata, and A. Geiger (2022) PlanT: explainable planning transformers via object-level representations. In Conference on Robotic Learning (CoRL), Proceedings of Machine Learning Research, Vol. 205, p. 459–470. Cited by: §2. [25] D. Rößle, X. Xie, A. Mohan, V. T. Sambandham, D. Cremers, and T. Schön (2026) DrivIng: a large-scale multimodal driving dataset with full digital twin integration. arXiv:2601.15260. Cited by: §4. [26] The Waymo Team (2025) Bringing waymo to more people, sooner. Note: Waymo blog External Links: Link Cited by: §1. [27] S. Thrun, M. Montemerlo, H. Dahlkamp, D. Stavens, A. Aron, J. Diebel, P. Fong, J. Gale, M. Halpenny, G. Hoffmann, et al. (2006) Stanley: the robot that won the darpa grand challenge. Journal of field Robotics 23 (9), p. 661–692. Cited by: §2. [28] R. Verschueren, G. Frison, D. Kouzoupis, J. Frey, N. v. Duijkeren, A. Zanelli, B. Novoselnik, T. Albin, R. Quirynen, and M. Diehl (2022) Acados—a modular open-source framework for fast embedded optimal control. Mathematical Programming Computation 14 (1), p. 147–183. Cited by: §3. [29] M. Vitelli, Y. Chang, Y. Ye, A. Ferreira, M. Wołczyk, B. Osiński, M. Niendorf, H. Grimmett, Q. Huang, A. Jain, et al. (2022) SafetyNet: safe planning for real-world self-driving vehicles using machine-learned policies. In 2022 International Conference on Robotics and Automation (ICRA), p. 897–904. Cited by: §2. [30] Y. Wang, W. Luo, J. Bai, Y. Cao, T. Che, K. Chen, Y. Chen, J. Diamond, Y. Ding, W. Ding, et al. (2025) Alpamayo-r1: bridging reasoning and action prediction for generalizable autonomous driving in the long tail. arXiv:2511.00088. Cited by: §1, §2. [31] M. Werling, S. Kammel, J. Ziegler, and L. Gröl (2012) Optimal trajectories for time-critical street scenarios using discretized terminal manifolds. The International Journal of Robotics Research 31 (3), p. 346–359. Cited by: §2. [32] M. Werling (2014) Ein neues konzept für die trajektoriengenerierung und -stabilisierung in zeitkritischen verkehrsszenarien. Vol. 34, KIT Scientific Publishing. Cited by: §3. [33] Zoox (2025) Zoox robotaxi launches in las vegas. Note: Zoox journal web page External Links: Link Cited by: §1.