Paper deep dive
Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving
Zilin Huang, Zhengyang Wan, Zihao Sheng, Boyue Wang, Junwei You, Yue Leng, Sikai Chen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/10/2026, 2:12:54 AM
Summary
Sim2Real-AD is a modular framework for zero-shot sim-to-real transfer of CARLA-trained VLM-guided reinforcement learning policies to full-scale autonomous vehicles. It addresses the sim-to-real gap by decomposing the problem into four components: a Geometric Observation Bridge (GOB) for BEV conversion, a Physics-Aware Action Mapping (PAM) for platform-agnostic control, a Two-Phase Progressive Training (TPT) strategy for stable adaptation, and a Real-time Deployment Pipeline (RDP). The framework enables successful deployment on a Ford E-Transit without real-world RL training data.
Entities (7)
Relation Signals (4)
Sim2Real-AD → comprises → Geometric Observation Bridge
confidence 100% · The framework decomposes the transfer problem into four components: a Geometric Observation Bridge (GOB)...
Sim2Real-AD → comprises → Physics-Aware Action Mapping
confidence 100% · a Physics-Aware Action Mapping (PAM) that translates policy outputs into platform-agnostic physical commands
Sim2Real-AD → comprises → Two-Phase Progressive Training
confidence 100% · a Two-Phase Progressive Training (TPT) strategy that stabilizes adaptation
Sim2Real-AD → deployedon → Ford E-Transit
confidence 100% · Zero-shot deployment on a full-scale Ford E-Transit achieves success rates of 90%...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deploying reinforcement learning policies trained in simulation to real autonomous vehicles remains a fundamental challenge, particularly for VLM-guided RL frameworks whose policies are typically learned with simulator-native observations and simulator-coupled action semantics that are unavailable on physical platforms. This paper presents Sim2Real-AD, a modular framework for zero-shot sim-to-real transfer of CARLA-trained VLM-guided RL policies to full-scale vehicles without any real-world RL training data. The framework decomposes the transfer problem into four components: a Geometric Observation Bridge (GOB) that converts monocular front-view images into simulator-compatible bird's-eye-view (BEV) observations, a Physics-Aware Action Mapping (PAM) that translates policy outputs into platform-agnostic physical commands, a Two-Phase Progressive Training (TPT) strategy that stabilizes adaptation by separating action-space and observation-space transfer, and a Real-time Deployment Pipeline (RDP) that integrates perception, policy inference, control conversion, and safety monitoring for closed-loop execution. Simulation experiments show that the framework preserves the relative performance ordering of representative RL algorithms across different reward paradigms and validate the contribution of each module. Zero-shot deployment on a full-scale Ford E-Transit achieves success rates of 90%, 80%, and 75% in car-following, obstacle avoidance, and stop-sign interaction scenarios, respectively. To the best of our knowledge, this study is among the first to demonstrate zero-shot closed-loop deployment of a CARLA-trained VLM-guided RL policy on a full-scale real vehicle without any real-world RL training data. The demo video and code are available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.03497v1
- Canonical: https://arxiv.org/abs/2604.03497v1
Trouble viewing inline? Open PDF directly →
Full Text
157,921 characters extracted from source content.
Expand or collapse full text
Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving Zilin Huang Zhengyang Wan Zihao Sheng Boyue Wang Junwei You Yue Leng Sikai Chen sikai.chen@wisc.edu Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA Google, Sunnyvale, CA 94089, USA Abstract Deploying reinforcement learning policies trained in simulation to real autonomous vehicles remains a fundamental challenge, particularly for VLM-guided RL frameworks whose policies are typically learned with simulator-native observations and simulator-coupled action semantics that are unavailable on physical platforms. This paper presents Sim2Real-AD, a modular framework for zero-shot sim-to-real transfer of CARLA-trained VLM-guided RL policies to full-scale vehicles without any real-world RL training data. The framework decomposes the transfer problem into four components: a Geometric Observation Bridge (GOB) that converts monocular front-view images into simulator-compatible bird’s-eye-view (BEV) observations, a Physics-Aware Action Mapping (PAM) that translates policy outputs into platform-agnostic physical commands, a Two-Phase Progressive Training (TPT) strategy that stabilizes adaptation by separating action-space and observation-space transfer, and a Real-time Deployment Pipeline (RDP) that integrates perception, policy inference, control conversion, and safety monitoring for closed-loop execution. Simulation experiments show that the framework preserves the relative performance ordering of representative RL algorithms across different reward paradigms and validate the contribution of each module. Zero-shot deployment on a full-scale Ford E-Transit achieves success rates of 90%, 80%, and 75% in car-following, obstacle avoidance, and stop-sign interaction scenarios, respectively. To the best of our knowledge, this study is among the first to demonstrate zero-shot closed-loop deployment of a CARLA-trained VLM-guided RL policy on a full-scale real vehicle without any real-world RL training data. These results suggest a practical and principled path for transferring simulation-trained VLM-guided RL policies to real-world autonomous driving. The demo video and code are available at: https://zilin-huang.github.io/Sim2Real-AD-website/. keywords: Autonomous Driving, Vision-Language Models, Reinforcement Learning, Reward Design, Real-World Deployment 1 Introduction Autonomous driving has advanced rapidly over the past decade, driven by progress in deep learning, perception, and large-scale simulation [Wang et al., 2025]. Nevertheless, reliable operation in open-world traffic remains difficult because real roads contain long-tail events, uncertain human behavior, partial observability, and continuously changing environmental conditions [Tang et al., 2026, Xu et al., 2025, Hu et al., 2025, Qu et al., 2025]. Although recent automated driving systems have achieved impressive performance in restricted operational design domains, safe and generalizable deployment across broader real-world settings remains unresolved. This gap between promising algorithmic performance and dependable on-road operation continues to be one of the central challenges in autonomous driving research [Alqahtani, 2025, Abdel-Aty and Ding, 2024]. For this reason, high-fidelity simulation has become a cornerstone of modern autonomous driving development. Platforms such as CARLA provide scalable and risk-free environments for training and evaluation under diverse road layouts, weather conditions, and traffic configurations [Dosovitskiy et al., 2017]. The community has also established a series of widely used closed-loop benchmarks, including NoCrash [Codevilla et al., 2019], the CARLA Leaderboard [CARLA Simulator Team, 2020], and Bench2Drive [Jia et al., 2024], to measure progress in increasingly realistic driving settings. Within this simulation-centered paradigm, two major learning strategies have emerged. Imitation learning (IL) methods, such as TransFuser [Chitta et al., 2022] and InterFuser [Shao et al., 2023], learn policies by mimicking expert demonstrations and have achieved strong benchmark performance. However, their behavior is fundamentally bounded by the quality and coverage of the demonstration data, and they often struggle in situations that are rare or absent in the training set [Codevilla et al., 2019]. Reinforcement learning (RL), in contrast, optimizes policies through interaction and reward-driven exploration, offering a pathway to discover behaviors beyond those explicitly present in human data [Kiran et al., 2021]. A particularly promising direction within RL-based driving is the integration of Vision-Language Models (VLMs) and contrastive vision-language pretraining models such as CLIP [Radford et al., 2021] into reward design, which replaces brittle hand-crafted reward functions with semantically grounded signals [Huang et al., 2025b, Wasif et al., 2025, Qu et al., 2026, Huang et al., 2026]. This line of work has progressed along a clear trajectory. VLM-RL [Huang et al., 2025b] first established the Contrasting Language Goal (CLG) paradigm, demonstrating that pairing positive and negative language descriptions with CLIP-based reward computation enables effective end-to-end policy learning in CARLA. DriveVLM-RL [Huang et al., 2026] extended this foundation with a neuroscience-inspired dual-pathway architecture that adds an attention-gated large VLM (LVLM) pathway for context-aware semantic risk reasoning in heterogeneous traffic, achieving state-of-the-art collision avoidance across diverse urban scenarios. Critically, in DriveVLM-RL, all VLM and LVLM components operate exclusively during offline training and are completely removed at deployment, so the final driving policy executes as a lightweight neural network with no VLM inference overhead at test time. This training-deployment decoupling is what makes real-time vehicle control feasible despite the high computational cost of foundation models. However, despite these advances in simulation, both works and the broader VLM-guided RL literature validate their methods entirely within CARLA. How to transfer such a policy from simulator-native observations and simulator-coupled control semantics to a full-scale physical vehicle remains an open and largely unaddressed problem. Despite these advances, policies trained in simulation still rarely transfer directly to physical vehicles. The key obstacle is the simulation-to-reality (sim-to-real) gap, which arises because simulator training and real-world deployment differ fundamentally in both observation and control semantics [Salvato et al., 2021, Daza et al., 2023, Li and Okhrin, 2024]. In this work, we focus on two dominant and practically decisive components of this gap, as illustrated in Fig. 1. The first is the observation gap. Many RL driving policies in CARLA rely on privileged or simulator-native observations, such as ground-truth bird’s-eye-view (BEV) semantic masks, that are clean, structured, and spatially complete [Dosovitskiy et al., 2017]. In contrast, a real vehicle must infer scene structure from camera inputs that are noisy, partially occluded, and restricted by field of view, creating a substantial cross-domain mismatch [Tobin et al., 2017, Zhu et al., 2017, Ganin et al., 2016]. The second is the dynamics gap. In simulation, policy outputs are often interpreted directly as low-level control commands, such as steering and throttle/brake. On a physical platform, however, the same commands can induce substantially different vehicle responses because of differences in wheelbase, steering ratio, actuator latency, tire-road interaction, and longitudinal dynamics [Daza et al., 2023, Salvato et al., 2021, Li and Okhrin, 2024]. These two gaps compound each other: when a policy trained on perfect BEV receives the noisy, limited-coverage observation produced by a real monocular camera, and its control outputs are simultaneously misinterpreted by a vehicle with different physical dynamics, performance degrades catastrophically. Figure 1: Overview of the sim-to-real challenge and the proposed Sim2Real-AD framework. Direct transfer fails because of the coupled observation and dynamics gaps, while Sim2Real-AD addresses them through GOB, PAM, TPT, and RDP. Existing sim-to-real methods address parts of this problem, but they do not provide a practical solution for the deployment setting considered here. On the perception side, image-level domain adaptation approaches, including adversarial translation and style transfer, can reduce appearance mismatch between simulated and real RGB images [Zhu et al., 2017, Ganin et al., 2016]. However, such methods typically require substantial real-world data and are not naturally aligned with the structured multi-channel BEV observations used by many autonomous driving policies. Domain randomization improves robustness by exposing the policy to diverse perturbations during training [Tobin et al., 2017], but it does not explicitly resolve the mismatch introduced by a real perception pipeline. Learned BEV methods, such as Lift-Splat-Shoot and BEVFormer, offer powerful camera-to-BEV perception capabilities [Philion and Fidler, 2020, Li et al., 2024], yet they themselves usually depend on large-scale supervised training and may introduce an additional representation gap when deployed across domains. On the control side, system identification and controller calibration are widely used for vehicle tracking, but they are typically designed for low-level control modules rather than for simulator-trained RL policies whose outputs are tightly coupled to simulator-specific action semantics [Daza et al., 2023, Salvato et al., 2021]. Critically, most existing approaches address either the observation gap or the dynamics gap in isolation, and the few that consider both rely on learned black-box alignment modules that are difficult to debug, require substantial real-world data, and must be retrained when the deployment vehicle changes. Consequently, prior work still lacks a unified, calibration-efficient framework that makes VLM-guided RL policies deployable on real vehicles without real-world policy training or fine-tuning [Li and Okhrin, 2024, Voogd et al., 2023]. To address this challenge, we propose Sim2Real-AD, a modular sim-to-real deployment framework for VLM-guided RL autonomous driving. Concretely, for the observation side, we introduce a Geometric Observation Bridge (GOB) that converts front-view monocular images into a unified multi-channel BEV representation using pre-trained semantic segmentation and inverse perspective mapping (IPM). This process removes much of the appearance-level discrepancy between simulated and real images and exposes the policy to a common spatial representation across domains without any domain-specific training data. For the action side, we introduce a Physics-Aware Action Mapping (PAM) that redefines policy outputs from simulator-coupled control commands to platform-agnostic physical quantities, namely curvature and desired speed, which are then translated into vehicle-specific actuation through a calibrated bicycle model and a low-level PID controller. To make the transition stable, we design a Two-Phase Progressive Training (TPT) strategy that first adapts the policy to the new action semantics under clean simulator BEV, then fine-tunes it on IPM-generated BEV observations, resolving each gap independently before combining them. Finally, a Real-time Deployment Pipeline (RDP) integrates all components into a closed-loop system for real-vehicle operation. The resulting framework has three practical advantages. First, it does not require any real-world RL training data or learned sim-to-real adaptation network. Second, its transfer modules are transparent, modular, and easy to calibrate or debug. Third, moving the policy to a new vehicle platform requires only lightweight platform-specific calibration rather than end-to-end retraining. Building on these properties, we validate Sim2Real-AD through extensive simulation experiments in CARLA and zero-shot closed-loop deployment on a full-scale Ford E-Transit autonomous vehicle at the University of Wisconsin-Madison testbed. To the best of our knowledge, this study is among the first to demonstrate zero-shot closed-loop deployment of a CARLA-trained VLM-guided RL policy on a full-scale real vehicle without any real-world RL training data. The main contributions of this paper are summarized as follows: • We propose Sim2Real-AD, a modular and reward-agnostic sim-to-real deployment framework for VLM-guided RL autonomous driving that explicitly decomposes transfer into an observation-space bridge and an action-space bridge, enabling real-world deployment without real-world policy training or learning-based domain adaptation. The framework is broadly compatible with RL-based driving policies that use structured simulator-native observations. • We introduce a geometric observation bridge that transforms monocular camera images into a unified BEV semantic representation using pre-trained segmentation and inverse perspective mapping, substantially reducing cross-domain observation discrepancy in a fully interpretable, training-free manner. • We design a physics-aware action mapping together with a two-phase progressive training strategy, which decouple policy learning from platform-specific control semantics and support zero-shot policy transfer through lightweight platform calibration requiring only ∼ 30 minutes and no real-world driving data. • We implement a complete real-time deployment pipeline and demonstrate zero-shot closed-loop deployment of a CARLA-trained VLM-guided RL policy on a full-scale Ford E-Transit, achieving success rates of 90%, 80%, and 75% in car-following, obstacle avoidance, and stop-sign interaction scenarios, respectively. The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 introduces the necessary background and problem formulation. Section 4 presents the proposed Sim2Real-AD framework in detail. Section 5 reports simulation studies in CARLA and zero-shot real-vehicle deployment results. Section 6 concludes the paper and discusses limitations and future directions. 2 Related Work 2.1 Learning-based Driving Policies in Simulation End-to-end autonomous driving has advanced along two main paradigms. IL methods, such as TransFuser [Chitta et al., 2022], InterFuser [Shao et al., 2023], and UniAD [Hu et al., 2023], learn by mimicking expert demonstrations and achieve strong CARLA benchmark performance, but are bounded by demonstration coverage and tend to fail in out-of-distribution scenarios [Codevilla et al., 2019]. RL offers a complementary path through reward-driven exploration [Sheng et al., 2024b, Huang et al., 2024, Jiang et al., 2025a, Li et al., 2025]. However, manually engineered rewards require substantial effort and often fail to generalize [Delavari et al., 2025], motivating the integration of VLMs into RL reward design. By encoding semantic goals through encoders such as CLIP [Radford et al., 2021], VLM-guided RL replaces brittle hand-crafted objectives with grounded reward signals. Representative works include VLM-RL [Huang et al., 2025b], DriveMind [Wasif et al., 2025], Found-RL [Qu et al., 2026], and DriveVLM-RL [Huang et al., 2026]. A parallel line explores Vision-Language-Action (VLA) models that unify perception, reasoning, and control end-to-end [Renz et al., 2025, Wang et al., 2025, Qian et al., 2025, Zhou et al., 2025]. VLA models face an even more acute deployment challenge as large model inference at test time makes real-time control infeasible without specialized decoupling strategies [Jiang et al., 2025b]. Across both lines, the primary focus remains on policy learning and benchmark performance within simulation. How to transfer such policies from simulator-native observations and simulator-coupled control semantics to full-scale real vehicles remains largely unanswered. Our work addresses this gap by providing a principled deployment framework for a CARLA-trained VLM-guided RL policy. 2.2 Sim-to-Real Methods for Autonomous Driving and Robotics Domain randomization [Tobin et al., 2017] improves robustness via diverse perturbations during training, but does not apply to structured multi-channel semantic BEV observations and provides no mechanism for bridging action semantics across platforms. Domain adaptation aligns distributions through image translation (CycleGAN [Zhu et al., 2017]) or adversarial feature learning (DANN [Ganin et al., 2016]), but requires substantial real-world data and retraining when the deployment vehicle changes. Canonical representation mapping bridges the observation gap via a shared intermediate space. Learned approaches such as Lift-Splat-Shoot [Philion and Fidler, 2020] and BEVFormer [Li et al., 2024] offer powerful camera-to-BEV perception but require large-scale supervised training. A lightweight alternative is Inverse Perspective Mapping (IPM) [Bertozzi and Broggi, 1998], which projects road-plane pixels into a top-down BEV without training data. Combined with a pre-trained segmentation model [Xie et al., 2021], IPM yields an interpretable, calibration-efficient bridge transferable across platforms via simple camera recalibration — the foundation of our GOB module. System identification and controller calibration [Daza et al., 2023, Salvato et al., 2021, Li and Okhrin, 2024] address the dynamics gap by fitting vehicle models to real behavior, but target low-level tracking controllers rather than the simulator-coupled action semantics of end-to-end RL policies. Curriculum-based adaptation [Bengio et al., 2009, Salvato et al., 2021, Zhao et al., 2020] reduces optimization difficulty by progressively exposing the policy to more realistic distributions. Our TPT strategy applies this principle to the dual-gap setting, decoupling action-space adaptation from observation-space adaptation. Unlike Rapid Motor Adaptation (RMA) [Kumar et al., 2021], which requires online real-world data, TPT relies entirely on geometry-based and physics-based bridging. Related efforts have explored digital twins, reality-gap modeling, and deployment-oriented pipelines [Voogd et al., 2023, Daza et al., 2023, Li and Okhrin, 2024, Huang et al., 2025c, Tahmina et al., 2025], as well as BEV generation and segmentation systems [Gao et al., 2025, Jun and Lee, 2025]. In contrast to these studies, our framework jointly addresses the observation and dynamics gaps through interpretable modules and complements the system design with a transfer-error analysis that links module-level imperfections to sim-to-real performance degradation. 2.3 Real-World Deployment of Simulation-Trained Driving Policies Although simulation-based learning has advanced rapidly, real-world deployment remains limited. Most simulator-trained policies are tightly coupled to their training environment on two levels: their observation interface relies on simulator-native representations (e.g., privileged BEV semantic masks) that are unavailable on real vehicles, and their action semantics are calibrated to simulator dynamics that differ from real platform responses. Without explicit mechanisms to bridge both couplings, direct deployment fails even when the policy performs well in simulation. Many studies validate only in CARLA or evaluate deployment-oriented ideas through offline replay, shadow-mode analysis, or scaled platforms rather than full-scale closed-loop operation [Delavari et al., 2025, Voogd et al., 2023, Sheng et al., 2024a]. Prior sim-to-real work often targets specialized tasks such as drifting or parking [Tóth et al., 2024] or requires additional real-world adaptation after simulation training [Lin et al., 2025]. World-model and generative-simulation approaches [Yang et al., 2025, You et al., 2024, Zhang et al., 2026] strengthen training and evaluation infrastructure but do not demonstrate direct deployment of simulator-trained RL policies on physical vehicles. Foundation model-based methods have shown strong simulation performance, but zero-shot closed-loop deployment on full-scale real vehicles remains very limited [Li and Okhrin, 2024, Voogd et al., 2023]. The challenge is particularly acute for VLM-guided RL [Wasif et al., 2025, Huang et al., 2025b, 2026]: these policies depend on simulator-privileged BEV observations no real sensor can replicate, and their action spaces are implicitly calibrated to simulator dynamics. To our knowledge, no prior work starts from a CARLA-trained VLM-guided RL policy and deploys it on a real vehicle through a modular observation-and-action bridging framework without any real-world RL training data. Our work addresses this gap by combining GOB, PAM, TPT, and RDP into a complete sim-to-real pipeline for closed-loop operation on a full-scale platform. 3 Preliminaries and Problem Formulation 3.1 Preliminaries Partially Observable Markov Decision Process. In a closed-loop simulator, autonomous driving is formulated as a Partially Observable Markov Decision Process (POMDP) ℳsim=(,,,,ℛ,ϕ,γ,d0)M_sim=(S,A,T,O,R,φ,γ,d_0), where S is the state space, A is the action space, (s′∣s,a)T(s s,a) is the transition dynamics, O is the observation space, ℛ(s,a)R(s,a) is the reward function, ϕ(o∣s)φ(o s) is the observation emission function, γ∈(0,1)γ∈(0,1) is the discount factor, and d0d_0 is the initial state distribution. Because the agent does not observe the full state s directly, it receives an observation ot∈o_t at each timestep and selects an action at∈a_t according to its policy πθ(at∣ot) _θ(a_t o_t). The policy is trained to maximize the expected discounted return [Sutton et al., 1998, Kiran et al., 2021, Huang et al., 2025a]: J(πθ)=πθ[∑t=0Tγtrt].J( _θ)=E_ _θ\! [ _t=0^Tγ^tr_t ]. (1) VLM-Guided RL Reward. In recent VLM-guided RL frameworks [Huang et al., 2025b, Wasif et al., 2025], the observation oto_t is a structured simulator-native representation comprising a BEV semantic tensor, route information, and vehicle states. The reward function combines conventional driving objectives with semantic supervision derived from VLMs or CLIP-style encoders. Following the formulation of DriveVLM-RL [Huang et al., 2026], the reward can be expressed in the general form: rt=ℱ(rttask,rtsem,st),r_t=F(r_t^task,\,r_t^sem,\,s_t), (2) where rttaskr_t^task is a sparse task reward (e.g., collision penalty), rtsemr_t^sem is a dense semantic reward derived from VLM-based visual-language alignment, and sts_t denotes vehicle state information. The function ℱF represents the reward synthesis mechanism, which in DriveVLM-RL takes a multiplicative hierarchical form that integrates both components with vehicle dynamics constraints. Critically, all VLM components used to compute rtsemr_t^sem operate exclusively during training: once training is complete, the reward computation is discarded entirely, and the deployed policy πθ _θ executes as a lightweight neural network with no VLM inference at test time. This training-deployment decoupling is a fundamental property that makes real-time vehicle control feasible and is precisely what allows Sim2Real-AD to focus on bridging observation and action interfaces rather than addressing VLM latency. 3.2 Problem Formulation Let πθsim _θ^sim denote a driving policy trained entirely in CARLA under the VLM-guided RL setting above, where the observation space simO^sim consists of simulator-native BEV representations and the action space simA^sim is calibrated to simulator dynamics. Let realO^real and realT^real denote the observation space and transition dynamics of the target real vehicle. Direct deployment of πθsim _θ^sim fails because two systematic mismatches arise: sim≉real,sim≉real.O^sim ^real, ^sim ^real. (3) The observation gap arises because πθsim _θ^sim is trained on privileged simulator BEV observations o^tsim∈^sim⊂ℝ192×192×14 o_t^sim∈ O^sim ^192× 192× 14 that are spatially complete and semantically clean, whereas the real platform can only provide raw monocular front-view images from which BEV-like representations must be reconstructed with inherent noise, limited field of view, and segmentation imperfections. The dynamics gap arises because even the same nominal control command induces substantially different motion in simulation and on a real vehicle due to differences in wheelbase, steering ratio, actuator latency, tire-road interaction, and low-level control response. Importantly, the two gaps interact: a policy receiving unfamiliar observations will generate unreliable action outputs, which are then further distorted by dynamics it was not trained to account for. As we show in Section 5.1.6, addressing each gap in isolation yields substantially worse performance than addressing them jointly through a principled sequential curriculum, because the combined distributional shift creates a super-additive optimization difficulty that neither module alone can resolve. The sim-to-real deployment problem is therefore to construct an observation bridge :real→^simG:O^real→ O^sim and an action bridge ℳ:^→realM: A ^real, where ^sim⊂ℝ192×192×14 O^sim ^192× 192× 14 denotes the simulator-compatible BEV observation space and A is a platform-agnostic physical action space, such that the composed policy: πreal(atreal∣otreal)=ℳ(πθsim(o^tsim,st,wt)),π^real(a_t^real o_t^real)=M\! ( _θ^sim\! ( o_t^sim,\,s_t,\,w_t ) ), (4) where o^tsim=(otreal)∈^sim o_t^sim=G(o_t^real)∈ O^sim denotes the simulator-compatible BEV observation produced by the GOB module, sts_t denotes the ego vehicle state (speed, steering, throttle), and wtw_t denotes the future waypoint sequence, both available from on-board sensors and a GPS route provider at deployment time. The goal is for this composed policy to achieve safe and stable closed-loop driving on the real vehicle under zero-shot transfer, using only lightweight calibration and without any real-world RL training data. 4 Framework: Sim2Real-AD 4.1 Overview The role of Sim2Real-AD is not to redesign the VLM-guided RL algorithm itself, but to make such a simulator-trained policy transferable to the real world. In this work, we instantiate the framework using DriveVLM-RL [Huang et al., 2026], a representative and state-of-the-art VLM-guided RL framework for safe autonomous driving in CARLA. In DriveVLM-RL, the policy is trained using simulator-native observations, including BEV semantic masks, route information, and vehicle states, together with semantic reward supervision derived from VLMs. While this training paradigm enables effective and safety-oriented policy learning in simulation, the resulting policy is not directly deployable on a physical vehicle because both its observation interface and its action semantics are tied to the simulator. As illustrated in Fig. 2, the proposed framework consists of four main components: GOB, PAM, TPT, and RDP. The GOB reduces the observation gap by converting real monocular front-view images into a simulator-compatible BEV semantic representation o^tsim∈^sim o_t^sim∈ O^sim. The PAM reduces the dynamics gap by redefining the policy output in terms of platform-agnostic physical quantities and translating them into executable vehicle commands. The TPT strategy stabilizes transfer by decoupling action-space adaptation from observation-space adaptation. Finally, the RDP integrates all modules into a complete closed-loop system for deployment on a real vehicle. Together, these components form a modular sim-to-real deployment framework that preserves the simulation-training advantages of DriveVLM-RL while replacing its simulator-dependent observation and action interfaces with deployment-oriented bridges. This design enables zero-shot closed-loop deployment of a CARLA-trained VLM-guided RL policy on a real vehicle using only lightweight platform calibration and no additional real-world RL training or policy fine-tuning. Since the four modules operate entirely on the policy’s observation and action interfaces, Sim2Real-AD remains agnostic to the reward design used during simulator training. Figure 2: Overview of Sim2Real-AD. The framework bridges sim-to-real transfer through four components: the Geometric Observation Bridge (GOB), the Physics-Aware Action Mapping (PAM), the Two-Phase Progressive Training strategy (TPT), and the Real-time Deployment Pipeline (RDP). Instantiated here with DriveVLM-RL as the backbone, the framework is reward-agnostic and is demonstrated across multiple RL reward paradigms. 4.2 Geometric Observation Bridge The observation gap defined in Section 3.2 prevents direct deployment of πθsim _θ^sim because the policy expects structured BEV observations o^tsim∈^sim o_t^sim∈ O^sim that are unavailable on a real vehicle. The GOB realizes the mapping :real→^simG:O^real→ O^sim by converting raw monocular front-view images into this simulator-compatible format through a two-step geometric pipeline. Unlike learned multi-camera BEV pipelines that aim to maximize perception accuracy under richer sensor setups [Philion and Fidler, 2020, Li et al., 2024, Jun and Lee, 2025], our goal is to construct a lightweight simulator-compatible observation interface under the minimal monocular deployment configuration considered in this work. 4.2.1 Unified BEV Representation Rather than letting the policy consume raw RGB images whose appearance differs substantially across domains, we process both simulated and real front-view images through the same deterministic camera-to-BEV pipeline and expose the policy only to the resulting BEV tensor. In this way, the policy always receives observations in the same spatially structured format, even though the underlying image source differs. Let It∈ℝH×W×3I_t ^H× W× 3 denote the monocular RGB image captured at time step t. We first apply a semantic segmentation network to obtain a pixel-wise semantic map: St=fseg(It),S_t=f_seg(I_t), (5) where fseg(⋅)f_seg(·) denotes the segmentation model. In our implementation we use SegFormer-B0 [Xie et al., 2021], which provides a favorable trade-off between segmentation quality and inference efficiency: it achieves competitive mIoU while running at over 30 FPS on a single GPU, satisfying the real-time constraint. No fine-tuning on domain-specific data is performed; the model is applied zero-shot to both simulation and real-world images. The segmentation output contains driving-relevant semantic regions, including road surface, lane markings, vehicles, pedestrians, and traffic-related classes. Next, we apply Inverse Perspective Mapping (IPM) [Bertozzi and Broggi, 1998] to project the segmented front-view image into a top-down BEV space: S~t=fipm(St;,h,α,β), S_t=f_ipm(S_t;\,K,\,h,\,α,\,β), (6) where ∈ℝ3×3K ^3× 3 is the camera intrinsic matrix and (h,α,β)(h,α,β) denote the camera mounting height, pitch, and roll. Under the planar-ground assumption, IPM maps image pixels on the road plane into a fixed BEV coordinate system representing a metric crop of 20m×20m20\,m× 20\,m centered on the ego vehicle. Camera calibration requires approximately 30 minutes using a standard checkerboard procedure [Zhang, 2000] and is performed once per deployment vehicle. 4.2.2 Multi-Channel BEV Construction The projected semantic map is encoded into a multi-channel BEV tensor: o^tsim=Encode(S~t)=Encode(fipm(fseg(It);,h,α,β)), o_t^sim=Encode( S_t)=Encode\! (f_ipm\! (f_seg(I_t);\,K,h,α,β ) ), (7) with size 192×192×14192× 192× 14, where each of the 14 channels corresponds to a binary occupancy mask for a specific semantic category (road surface, lane markings, vehicles, pedestrians, sidewalks, etc.). This 14-channel representation matches exactly the observation format expected by πθsim _θ^sim, ensuring input compatibility without any policy modification. The GOB output is therefore: o^tsim=(otreal),o^tsim∈^sim⊂ℝ192×192×14, o_t^sim=G(o_t^real), o_t^sim∈ O^sim ^192× 192× 14, (8) which is fed directly to the policy πθsim(at∣o^tsim,st,wt) _θ^sim(a_t o_t^sim,s_t,w_t) at each timestep. Because the same semantic channel layout is preserved across domains, the policy can continue to operate on the same input tensor format without any architectural modification. This compatibility is essential for transferring a simulator-trained policy to a real monocular perception stack. The full GOB pipeline is illustrated in Fig. 3. Figure 3: Geometric Observation Bridge from monocular front-view images to simulator-compatible BEV observations. Phase 1 uses simulator-privileged GT-BEV, whereas Phase 2 and real-world deployment use GOB-generated BEV with the same tensor shape and semantic channel layout. 4.2.3 Observation Transfer Properties Under the lightweight deployment setting considered in this paper, the real vehicle is equipped only with a single front-view monocular camera. As a result, the generated o^tsim o_t^sim provides reliable information mainly in a front-facing region and degrades with distance. In particular, far-field road pixels are heavily compressed in the image plane and become more sensitive to segmentation noise after projection. Nevertheless, this degradation is acceptable because near-field structure is the most critical information for lane keeping, obstacle avoidance, and short-horizon control. More importantly, the observation format remains unchanged: both simulator GT-BEV and IPM-generated o^tsim o_t^sim share the same tensor shape and semantic channel layout in ^sim O^sim, differing mainly in coverage and quality rather than representation structure. This structural consistency is why observation adaptation can be handled through fine-tuning rather than retraining from scratch. To characterize precisely how much the two BEV distributions differ across domains, we introduce the following notion of domain invariance for G, which connects the perceptual gap to the downstream policy performance bound in Theorem 1. Definition 1 (Domain-Invariant BEV Representation) A BEV encoding function :ℐ→^simG:I→ O^sim is domain-invariant with tolerance ϵ>0ε>0 if: I∼simI′∼real[‖(I)−(I′)‖1|scene(I)=scene(I′)]≤ϵ,E_ subarraycI ^sim\\ I ^real subarray\! [\, \|G(I)-G(I ) \|_1\; |\;scene(I)=scene(I ) ]≤ε, (9) where scene(⋅)scene(·) denotes the semantic content of an image and the expectation is conditioned on scene-matched pairs. A smaller ϵε means the two domains are more indistinguishable from the policy’s perspective. Remark 1 GOB achieves a small ϵε by operating on semantic categories rather than raw pixel values: appearance differences due to lighting, texture, and rendering style are largely absorbed by the segmentation step and therefore do not directly propagate into o^tsim o_t^sim. In practice, this tolerance ϵε is influenced primarily by the quality of the perception module, together with geometric approximation, camera calibration error, and projection distortion introduced by IPM, especially in far-field regions. Empirically, the segmentation component remains small on standard road scenes [Xie et al., 2021], which helps keep the overall cross-domain discrepancy low. This tolerance ϵε is the key quantity governing the GOB contribution in Theorem 1: reducing perception and projection errors directly tightens the sim-to-real performance bound, as formalized in Proposition 1 (B). 4.3 Physics-Aware Action Mapping The dynamics gap defined in Section 3.2 persists even after G reduces the observation gap: the policy output at=πθsim(o^tsim,st,wt)a_t= _θ^sim( o_t^sim,s_t,w_t) remains a simulator-calibrated command that produces different path curvatures on a real vehicle, κsim(at)≠κreal(at) _sim(a_t)≠ _real(a_t), due to differences in wheelbase, steering ratio, actuator characteristics, and low-level control delay. PAM realizes the mapping ℳ:^→realM: A ^real by redefining the policy output in terms of platform-agnostic physical quantities rather than simulator-specific commands. 4.3.1 Platform-Agnostic Action Space The key idea is to let the policy predict driving intent, while leaving the final actuation conversion to a calibrated platform-dependent controller. Instead of predicting direct steering and throttle/brake, the policy outputs two normalized action variables (a1,t,a2,t)∈[−1,1]2(a_1,t,a_2,t)∈[-1,1]^2, which are mapped to a platform-agnostic action space: a^t=(κt,vtd)∈^, a_t=( _t,v_t^d)∈ A, (10) where κt _t denotes the desired path curvature and vtdv_t^d denotes the desired speed. The mapping is defined as: κt=a1,tκmax,vtd=a2,t+12vmax, _t=a_1,t\, _ , v_t^d= a_2,t+12\,v_ , (11) where κmax _ and vmaxv_ are predefined platform-independent limits. These quantities encode geometric and kinematic driving intent more directly than raw low-level commands, and thus remain more stable across platforms. In this formulation, the policy is responsible for deciding how sharply the vehicle should turn and how fast it should move, while the platform-specific controller converts this intent into executable actuation. 4.3.2 Low-Level Control Conversion To convert the desired curvature into a steering command, we use a kinematic bicycle model [Rajamani, 2006]. Let L denote the wheelbase of the target vehicle. The desired front wheel steering angle is computed as: δt=arctan(Lκt), _t= (L _t), (12) and the corresponding normalized steering command is: utδ=δtδmax,u_t^δ= _t _ , (13) where δmax _ denotes the maximum steering angle of the target vehicle. The same curvature command can thus be interpreted consistently across different platforms through lightweight calibration rather than policy retraining. For longitudinal control, the desired speed vtdv_t^d is tracked by a low-level PID controller [Astrom and Hägglund, 2006]. The speed tracking error is: et=vtd−vt,e_t=v_t^d-v_t, (14) and the corresponding longitudinal command is: utv=Kpet+Ki∑τ=0teτΔt+Kdet−et−1Δt,u_t^v=K_pe_t+K_i _τ=0^te_τ t+K_d e_t-e_t-1 t, (15) where KpK_p, KiK_i, KdK_d are the controller gains and the integral term is clipped to avoid windup. The final executable vehicle command is: atreal=ℳ(κt,vtd)=(utδ,utv).a_t^real=M( _t,v_t^d)=(u_t^δ,\,u_t^v). (16) 4.3.3 Cross-Platform Calibration and Transfer Properties This design decouples the policy from the low-level actuation details of a particular vehicle platform. The policy no longer needs to implicitly learn a specific steering ratio or throttle response from the simulator; instead, it predicts motion intent in a platform-agnostic form. Deployment-time calibration requires adjusting only a small set of physical parameters: Θ=L,δmax,Kp,Ki,Kd, =\L, _ ,K_p,K_i,K_d\, (17) without modifying πθsim _θ^sim itself. In practice, L and δmax _ are obtained from vehicle specifications, while the PID gains are calibrated through short step-response tests on the target platform (≈ 30 min). The key structural reason this calibration is so lightweight is that ℳM is not a monolithic mapping from simulator actions to real actuator commands, but a factored one: it routes through a platform-agnostic physical intermediate space P that is independent of both the simulator and the target vehicle. We formalize this property below, as it is also what allows the PAM tracking error to be bounded and connected to the performance guarantee in Theorem 1. Definition 2 (Platform-Agnostic Action Interface) An action mapping ℳ:^→realM: A ^real is platform-agnostic if it factors through a physical intermediate space =ℝ2P=R^2 (target curvature κ and desired speed vdv^d): ℳ=ℳreal∘ℳsim−1,M=M_real _sim^-1, (18) where ℳsim−1:^→M_sim^-1: A depends only on the simulator vehicle’s kinematic parameters and ℳreal:→realM_real:P ^real depends only on the real vehicle’s kinematic parameters and PID gains Θ . Remark 2 (Why platform-agnostic actions improve transfer) The factored structure of Definition 2 has two direct consequences. First, it enables lightweight cross-platform deployment: to transfer to a new vehicle, only ℳrealM_real needs to be updated by recalibrating Θ , while ℳsim−1M_sim^-1 and the policy πθsim _θ^sim remain entirely unchanged. Second, it makes the dynamics gap quantifiable: because P consists of physical quantities (κ, vdv^d) with well-defined tracking dynamics, the execution error introduced by imperfect PID control can be bounded analytically. Specifically, the lateral position error accumulated over a control horizon of T steps is bounded by vmax2T22ϵpid v_ ^2T^22\, _pid, where ϵpid _pid is the per-step curvature tracking error (Assumption 4). This bound is the quantity that controls the PAM error term C2ϵpidC_2\, _pid in Theorem 1, and is formalized in Proposition 2 (B). 4.4 Two-Phase Progressive Training In our preliminary implementation, we initially attempted to introduce both changes at once, namely replacing simulator ground-truth BEV observations with IPM-generated BEV observations while simultaneously replacing simulator-coupled control outputs with physics-aware actions. However, this one-stage adaptation strategy was found to be unstable. A likely reason is that the policy must simultaneously adapt to two different forms of distribution shift: degraded perceptual inputs on the observation side and a newly defined action semantics on the control side. This substantially increases optimization difficulty and often leads to slow convergence or unstable learning. We therefore adopt a TPT strategy that decomposes sim-to-real adaptation into two sequential stages, as detailed in Algorithm 1. 4.4.1 Phase 1: Action-Space Adaptation In the first phase, the policy is trained from scratch in CARLA for 1×1061× 10^6 steps using simulator ground-truth BEV observations o^tsim∈^sim o_t^sim∈ O^sim, while the original simulator-coupled action interface is replaced by the platform-agnostic action space A. Formally, the policy predicts: a^t=(κt,vtd)∈^, a_t=( _t,v_t^d)∈ A, (19) while receiving clean and spatially complete simulator BEV inputs. The VLM-guided reward signal Eq. (2) provides semantic supervision throughout. Since the observation structure remains unchanged, the policy can focus entirely on learning how curvature and desired speed affect lane keeping, route following, and obstacle avoidance. Phase 1 thus isolates the dynamics-side transfer problem and resolves it before observation adaptation begins. 4.4.2 Phase 2: Observation-Space Adaptation In the second phase, the action space A is kept unchanged, but the observation source is replaced by the GOB-generated BEV produced from simulator front-view RGB images. That is, the policy now receives IPM-generated observations o^tsim=(Itsim)∈^sim o_t^sim=G(I_t^sim)∈ O^sim rather than privileged ground-truth BEV, for 5×1055× 10^5 additional steps. The same VLM-guided reward signal Eq. (2) continues to supervise training, ensuring that the VLM-guided reward structure is maintained under the new observation interface. Let πθ1 _ _1 denote the policy learned in Phase 1. Phase 2 training is initialized from the Phase 1 checkpoint: θ2(0)=θ1, _2^(0)= _1, (20) and further optimized under the observation distribution 2O_2 induced by G. This progressive initialization allows the policy to retain already-learned driving behavior and action semantics while specializing to the noisier and more limited observation distribution that better matches real deployment. 4.4.3 Training Rationale and Deployment Alignment By separating the two sources of transfer difficulty, TPT turns sim-to-real adaptation into a curriculum-like process [Bengio et al., 2009, Salvato et al., 2021, Zhao et al., 2020]: the IPM-generated BEV distribution 2O_2 used in Phase 2 is designed to be closer to the real deployment distribution realO^real than the ground-truth BEV distribution 1O_1 used in Phase 1. We formalize this distributional ordering below, as it is precisely what determines the TPT residual term C3δ/(1−γ)2C_3\,δ/(1-γ)^2 in Theorem 1. Definition 3 (Progressive Observation Curriculum) A two-phase training schedule (T1,T2,1,2)(T_1,T_2,O_1,O_2) is a progressive observation curriculum if T1+T2=T_1+T_2=T (total training steps) and the observation distributions satisfy: dTV(1,real)≥dTV(2,real),d_TV\! (O_1,\;O^real )\;≥\;d_TV\! (O_2,\;O^real ), (21) where dTVd_TV denotes total variation distance, 1O_1 is the ground-truth BEV distribution (Phase 1), and 2O_2 is the IPM-generated BEV distribution (Phase 2). In other words, each phase exposes the policy to observations that are no further from real deployment than the previous phase. Remark 3 (Motivation for progressive training) TPT satisfies Definition 3 by construction: ground-truth BEV (1O_1) is further from the real distribution than IPM-generated BEV (2O_2), because IPM operates on real camera geometry and already introduces the same class of projection artifacts and limited field of view present at deployment, whereas ground-truth BEV does not. This means Phase 2 reduces the residual distribution gap dTV(2,real)d_TV(O_2,O^real) relative to what it would be after Phase 1 alone. The significance of this reduction is quantified by Theorem 3 (B): J(πθsim,real)≥J(πθsim,2)−2RmaxdTV(2,real)(1−γ)2,J( _θ^sim,O^real)≥ J( _θ^sim,O_2)- 2R_ \,d_TV(O_2,O^real)(1-γ)^2, showing that the performance gap scales directly with dTV(2,real)d_TV(O_2,O^real). This is precisely the TPT residual C3δ/(1−γ)2C_3\,δ/(1-γ)^2 in Theorem 1: longer Phase 2 training further reduces dTV(2,real)d_TV(O_2,O^real), directly tightening this term. A single-stage schedule that skips Phase 1 would instead need dTV(1,real)d_TV(O_1,O^real) in the bound, a substantially larger quantity, leaving the TPT residual much worse. 4.5 Real-Time Deployment Pipeline After training, the final policy πθ2 _ _2 is integrated into a RDP for closed-loop vehicle operation. The purpose of this pipeline is to connect perception, route input, vehicle-state feedback, policy inference, action conversion, and safety monitoring into a complete execution stack that implements the composed policy Eq. (4) on a physical vehicle. 4.5.1 Pipeline and Timing The deployment loop follows a perception–inference–control structure: otreal→o^tsim→πθ2a^t→ℳatreal,o_t^real\; \;G\;\; o_t^sim\; \; _ _2\;\; a_t\; \;M\;\;a_t^real, (22) where the final command is further checked by the safety layer before execution, as illustrated in Fig. 2(d). The deployment loop consists of image acquisition, GOB processing, policy inference, PAM conversion, and command transmission. In steady-state pipelined execution, GOB processing dominates the critical path, and the average end-to-end control latency is 17.2 ms per cycle, well within the 50 ms budget of the 20 Hz control loop. Policy inference takes approximately 2 ms, while PAM conversion and PID computation take less than 1 ms. 4.5.2 Waypoint Generation The policy requires route information and vehicle states consistent with the simulator training interface. To provide route input, we use a waypoint provider that supplies a sequence of future path points to the policy. In our primary implementation, the route is obtained from a pre-recorded GPS trajectory: the current vehicle position is matched to the nearest point on the route, and a set of future waypoints is extracted and transformed into the vehicle-relative coordinate frame. This design makes the real-world waypoint input compatible with the route representation used during simulator training. When GPS-based routing is unavailable, a vision-based fallback can also be constructed by extracting a lane centerline from the road surface and lane marking channels of o^tsim o_t^sim. A vehicle-state interface provides real-time low-level feedback from the platform, including current speed, steering status, and other controller-relevant signals. In our implementation, these quantities are read from the vehicle through the CAN bus and converted into the normalized format expected by the policy and controller. Camera frames, waypoint updates, and vehicle-state messages are time-aligned before policy inference to ensure stable closed-loop execution. 4.5.3 Safety Layer To improve operational safety, we place a safety layer on top of the learned policy and low-level controller. It enforces hard motion constraints including a maximum speed of 15 km/h during initial testing, consistent with campus low-speed autonomous vehicle testing protocols [Koopman and Wagner, 2017], a lateral deviation limit of 0.8 m from the detected lane center, and a steering-rate limit |δt−δt−1|≤Δδmax| _t- _t-1|≤ _ to prevent abrupt steering commands. Emergency braking is triggered when any of the following conditions is met: invalid policy outputs (NaN/Inf); camera frame timeout or CAN bus disconnection; emergency-stop button activation; vehicle exit from a predefined geofenced region; or safety driver intervention detected via steering-wheel torque exceeding a threshold. All safety checks execute at higher priority than the policy output and cannot be overridden. The system immediately releases control authority upon safety-driver takeover. 4.5.4 Real-Time Execution and Modularity The observation bridge, route and state interfaces, policy, action mapping, and safety layer are loosely coupled and can be upgraded independently. For example, the IPM-based BEV generation can be replaced by a stronger camera-to-BEV model, or the PID controller replaced by a model predictive controller, without changing πθ2 _ _2 itself. The same modularity makes calibration lightweight: only the small parameter set Θ specified in Section 4.3 and the camera parameters (,h,α,β)(K,h,α,β) must be specified before deployment. Overall, RDP closes the gap between simulator training and physical execution by turning the composed policy: πreal(atreal∣otreal)=ℳ(πθ2(o^tsim,st,wt))π^real(a_t^real o_t^real)=M\! ( _ _2\! ( o_t^sim,\,s_t,\,w_t ) ) (23) into a complete real-world driving stack that requires no VLM inference at test time, enabling safe and responsive closed-loop operation on a full-scale vehicle. 4.6 Theoretical Guarantee We conclude the framework section with a simplified statement of the main theoretical guarantee, where the explicit dependence of each error coefficient on LrL_r, LπL_π, RmaxR_ , vmaxv_ , T, and γ is absorbed into named constants for readability. The full expressions for these constants and complete proofs are given in B. Theorem 1 (Zero-Shot Transfer Guarantee) Under bounded segmentation error ϵseg _seg, bounded PID tracking error ϵpid _pid, and bounded observation distribution gap dTV(2,real)≤δd_TV(O_2,O^real)≤δ, the expected cumulative reward of the Sim2Real-AD policy πrealπ^real on the real vehicle satisfies: [∑t=0Tγtrtreal]≥[∑t=0Tγtrtsim]−C1ϵseg⏟GOB error−C2ϵpid⏟PAM error−C3δ(1−γ)2⏟TPT residual,E\! [ _t=0^Tγ^tr_t^real ]\;≥\;E\! [ _t=0^Tγ^tr_t^sim ]- C_1\, _seg_GOB error- C_2\, _pid_PAM error- C_3\,δ(1-γ)^2_TPT residual, (24) where C1,C2,C3>0C_1,C_2,C_3>0 are constants depending on the policy’s Lipschitz constant LπL_π, the reward Lipschitz constant LrL_r, the reward bound RmaxR_ , the maximum speed vmaxv_ , the horizon T, and the discount factor γ (see B.5 for explicit expressions), and rtsimr_t^sim is the VLM-guided reward Eq. (2) from the RL objective Eq. (1). Remark 4 Theorem 1 identifies three independent sources of sim-to-real performance degradation and shows that each is separately controllable: the GOB error decreases with better segmentation, the PAM error decreases with tighter PID calibration, and the TPT residual decreases with longer Phase 2 training or higher-fidelity IPM. When all three error terms vanish, real-vehicle performance converges to simulation performance, as formalized in Corollary 1 (B). To the best of our knowledge, this is among the first transfer-error decompositions for zero-shot sim-to-real deployment of a VLM-guided RL policy in autonomous driving. 5 Experiments The experiments are structured to address the following research questions: RQ1: Does the proposed Sim2Real-AD framework preserve the relative performance ordering of different RL algorithms after sim-to-real transfer? RQ2: What is the individual contribution of each module (GOB, PAM, TPT) to overall transfer performance? RQ3: How much observation fidelity does GOB preserve compared to ground-truth BEV? RQ4: Does the transferred policy generalize to unseen maps? RQ5: Can the framework achieve zero-shot deployment on a real vehicle? 5.1 Simulation Experiments 5.1.1 Experimental Setup 1) Simulation Environment. All simulation experiments are conducted in CARLA 0.9.13 [Dosovitskiy et al., 2017] with synchronous mode at 20 FPS. Consistent with [Huang et al., 2026], models are trained exclusively on Town 2—a compact European-style urban layout with residential districts, commercial zones, single-lane roads, and signalized intersections that provides diverse driving conditions including straight roads, curved segments, T-junctions, and varying road geometries. For generalization testing (RQ4), we evaluate on Towns 1, 3, 4, and 5, which present progressively harder distribution shifts in road topology and traffic patterns. 2) Traffic Configuration. To evaluate robustness under realistic urban conditions with heterogeneous road users, we construct a complex traffic environment following [Huang et al., 2026]. Specifically, the simulation includes: 20 vehicles generating natural traffic flow interactions; 20 pedestrians with randomized walking speeds (0.8–1.5 m/s) moving around sidewalks and crosswalks; 20 motorcycles with short following distances (2.0 m) and ± 30% speed variance, frequently producing cut-in behaviors; and 20 bicycles traveling at approximately 80% below the speed limit, requiring safe and patient overtaking maneuvers. 3) Navigation Routes. We employ dynamic route assignment during both training and evaluation. At each episode reset, two distinct spawn points are randomly selected from the 101 predefined locations in Town 2 and the shortest path is computed via the A* algorithm. Episodes continue until the cumulative driving distance reaches 3,000 m, providing comprehensive coverage of diverse navigation scenarios within a single episode. For evaluation, we use 10 predefined routes not encountered during training. These evaluation routes are defined within Town 2 using held-out spawn-point pairs. 4) Episode Termination. Each episode terminates upon: (i) collision with static infrastructure, vehicles, pedestrians, cyclists, or motorcyclists; (i) the ego vehicle remaining stationary (speed << 1 km/h) for more than 90 consecutive seconds, indicating a stuck condition; or (i) lateral deviation from the lane center exceeding 3 m. 5.1.2 Training Configuration All algorithms use Soft Actor-Critic (SAC) [Haarnoja et al., 2018] with automatic entropy tuning as the backbone RL optimizer. Table 1 summarizes the key hyperparameters for the two-phase progressive training. Table 1: Training hyperparameters for two-phase progressive training (TPT). Phase 2 warm-starts from the Phase 1 checkpoint with a reduced learning rate. Parameter Phase 1 Phase 2 Observation source GT-BEV GOB-BEV (GOB) Action space Physics (PAM) Physics (PAM) Training steps 1×1061× 10^6 5×1055× 10^5 Initialization Random Phase 1 checkpoint Learning rate 1×10−4→5×10−71×10^-4→ 5×10^-7 5×10−5→5×10−75×10^-5→ 5×10^-7 Replay buffer size 10510^5 10510^5 Batch size 256 256 Discount γ 0.98 0.98 Soft update τ 0.02 0.02 Train frequency 64 steps 64 steps Gradient steps 64 64 BEV feature extractor CustomCNN → 256-d 1) Observation and Action Spaces. The policy receives a dictionary observation comprising: (i) a 14-channel semantic BEV mask at 192×192192× 192 resolution encoding road surface, lane markings, vehicles, pedestrians, traffic lights, and other semantic categories; (i) the next 15 waypoints along the planned route expressed as (x,y)(x,y) ego-centric coordinates at 2 m intervals; and (i) vehicle state measurements (speed in km/h, steering angle, throttle command). In Phase 2 of TPT, the BEV is produced by the GOB module (IPM pipeline from a front-facing monocular camera) rather than the simulator’s ground-truth semantic renderer. For Sim2Real-AD, the policy outputs a^=(aκ,av)∈[−1,1]2 a=(a_κ,a_v)∈[-1,1]^2, which PAM maps to platform-agnostic quantities: κ=aκ⋅κmax,vdes=av+12⋅vmax,κ=a_κ· _ , v_des= a_v+12· v_ , (25) where κmax=tan(δmax)/L _ = ( _ )/L with wheelbase L=2.875L=2.875 m and δmax=70∘ _ =70 (CARLA default vehicle; Ford E-Transit platform parameters are given in Section 5.2.1), and vmax=35v_ =35 km/h. A PID controller (Kp=0.5K_p=0.5, Ki=0.05K_i=0.05, Kd=0.1K_d=0.1) converts desired speed to throttle/brake commands. 2) VLM Configuration. For algorithms employing CLIP-based reward shaping (DriveVLM-RL and VLM-RL), we use OpenCLIP’s ViT-bigG-14 [Ilharco et al., 2021] pretrained on LAION-2B, with reward blending coefficient α=0.5α=0.5 and CLIP batch size of 64 frames. Since the CLIP reward is computed from first-person camera images rather than BEV observations, the GOB change in Phase 2 does not directly alter the reward computation interface, helping keep the training objective comparable across both phases. For DriveVLM-RL, a Qwen3-VL-4B LVLM with attention-gated YOLOv8s triggering provides dynamic semantic descriptions [Huang et al., 2026]. 5.1.3 Evaluation Metrics We report the following metrics, evaluated over 10 predefined routes per town following [Huang et al., 2026, 2025b]: • Average Speed (AS, km/h↑ ): Mean ego-vehicle speed during the episode. • Route Completion (RC↑ ): Fraction of the planned route completed before termination. • Total Distance (TD, m↑ ): Cumulative distance traveled per episode. • Collision Speed (CS, km/h↓ ): Mean speed at impact; reflects collision severity rather than frequency. • Success Rate (SR↑ ): Fraction of routes completed without collision or intervention. • Average Collision (AC↓ ): Mean collision count per evaluation route. • Performance Retention (PR, %↑ ): PRm=Msim2realMπθ1×100%,PR_m= M_sim2realM_ _ _1× 100\%, (26) where Mπθ1M_ _ _1 is the corresponding Stage 1 metric under GT-BEV and direct-action evaluation, and Msim2realM_sim2real is the metric after applying the Sim2Real-AD transfer pipeline. PR measures how much of the Stage 1 reference performance is retained after transfer. 5.1.4 Baseline Algorithms A key design principle of Sim2Real-AD is reward-agnostic generality: the four modules (GOB, PAM, TPT, RDP) operate entirely on the policy’s input and output interfaces, with no dependence on how the reward function is constructed during training. This means Sim2Real-AD can wrap any RL-based autonomous driving algorithm, whether the reward is hand-crafted, LLM-generated, or VLM-guided, and transfer it to a real vehicle without modification. To validate this claim, we apply the identical Sim2Real-AD pipeline to three algorithms that represent the current state of the art across distinct reward paradigms, selected from the comprehensive 13-method comparison in [Huang et al., 2026] as the top-performing representative of each paradigm: • ChatScene [Zhang et al., 2024]: SAC with an expert-designed smoothness-focused reward. Purely static and analytically computed without any VLM component. • VLM-RL [Huang et al., 2025b]: SAC with static shaped reward + CLIP reward using Contrastive Language Goals (CLG). Fixed prompts contrast “clear road” vs. “collision” using single-frame BEV images. • DriveVLM-RL [Huang et al., 2026] (our primary backbone, top-performing overall): SAC with a dual-pathway reward — a static pathway (CLIP-based CLG on BEV images) and a dynamic pathway (attention-gated LVLM reasoning). The dynamic pathway captures scene-level semantic risks through multi-frame visual understanding, achieving the highest SR and lowest CS among all compared methods. All three algorithms share the same SAC backbone, network architecture, observation space, and environment configuration; only the reward computation differs. Applying the identical four-module Sim2Real-AD pipeline to all three isolates the effect of reward design on transferability, directly answering whether Sim2Real-AD serves as a universal sim-to-real bridge independent of the underlying training objective. Figure 4: ChatScene training curves: Original (dashed, GT-BEV + Direct Action, ±1σ± 1σ over 3 seeds) vs. Sim2Real-AD (solid, Phase 1: GT-BEV + PAM, Phase 2: GOB-BEV + PAM). Vertical dotted line marks Phase 1→ 2 at 1×1061×10^6 steps. (a) Collision rate. (b) Average speed. (c) Total distance. (d) Routes completed. Figure 5: VLM-RL training curves: Original (dashed, GT-BEV + Direct Action, ±1σ± 1σ over 3 seeds) vs. Sim2Real-AD (solid, Phase 1: GT-BEV + PAM, Phase 2: GOB-BEV + PAM). Vertical dotted line marks Phase 1→ 2 at 1×1061×10^6 steps. (a) Collision rate. (b) Average speed. (c) Total distance. (d) Routes completed. Figure 6: DriveVLM-RL training curves: Original (dashed, GT-BEV + Direct Action, ±1σ± 1σ over 3 seeds) vs. Sim2Real-AD (solid, Phase 1: GT-BEV + PAM, Phase 2: GOB-BEV + PAM). Vertical dotted line marks Phase 1→ 2 at 1×1061×10^6 steps. (a) Collision rate. (b) Average speed. (c) Total distance. (d) Routes completed. 5.1.5 Main Results: Sim-to-Real Transfer across Algorithms (RQ1) We evaluate all three algorithms in two phases: a training phase that examines convergence stability under the TPT curriculum, and a testing phase that reports final policy performance across the three progressive transfer stages. 1) Training Performance Analysis. Figs. 4–6 present per-algorithm training curves comparing the original setup (dashed lines: GT-BEV + Direct Action) against Sim2Real-AD (solid lines: Phase 1 GT-BEV + PAM → Phase 2 GOB-BEV + PAM). In each figure, the vertical dotted line marks the Phase 1→ 2 transition at 1×1061×10^6 steps. Examining each algorithm in turn reveals how the sim-to-real modules reshape training dynamics. ChatScene (Fig. 4). ChatScene reveals the most striking contrast. Its original training (dashed) converges to a collision rate of 0.85–0.95, the highest among all three algorithms, with high speed (∼ 17 km/h) and large total distance (300–500 m). Under Sim2Real-AD (solid), however, the collision rate drops dramatically to 0.30–0.50 in Phase 1, eventually settling at 0.25–0.40 by Phase 2. This paradoxical improvement arises because the physics action space (PAM) constrains ChatScene’s originally aggressive maneuvers: the PID controller cannot execute the abrupt steering and throttle commands that cause frequent collisions under direct action. The trade-off is a significant reduction in total distance (100–250 m vs. 300–500 m) and route completion (0.3–0.8 vs. 0.8–2.0), as the constrained action space also limits the policy’s ability to complete routes efficiently. At the Phase 1→ 2 transition, ChatScene shows a temporary collision rate dip around 1M steps followed by recovery, confirming that TPT prevents catastrophic forgetting during the observation modality switch. VLM-RL (Fig. 5). VLM-RL exhibits the closest tracking between its original and Sim2Real-AD curves across all four metrics. Collision rates are nearly overlapping at 0.40–0.55 throughout Phase 1, with the Sim2Real-AD curve tracking at roughly the same level as, or slightly below, the original. This tight alignment indicates that VLM-RL’s single-frame CLIP reward, which contrasts “clear road” vs. “collision” on BEV images, produces behaviors that are relatively invariant to the action space change. In Phase 2, after the observation switches to GOB-BEV, speed increases to 15–17 km/h, actually exceeding the original speeds. This counterintuitive acceleration may stem from the noisier IPM observation reducing the policy’s caution, leading to faster but potentially less safe driving. Route completion in Phase 2 shows mild degradation relative to Phase 1, consistent with the single-frame CLIP reward being more sensitive to observation distributional shift than multi-frame VLM rewards. DriveVLM-RL (Fig. 6). Under Sim2Real-AD, DriveVLM-RL’s collision rate rises to 0.35–0.45 in Phase 1 (approximately double its original rate of 0.15–0.25), reflecting the physics action space’s reduced reactivity, then gradually drops to 0.28–0.38 in Phase 2 as the policy adapts to the GOB-BEV observation. Despite the Phase 1 increase, it remains the lowest collision rate among all three algorithms’ Sim2Real-AD curves throughout training. A notable finding is the speed profile: in Phase 1, average speed is 9–12 km/h, comparable to or slightly below the original’s 10–14 km/h; however, after the Phase 1→ 2 transition, speed rises sharply to 13–16 km/h and stabilizes, suggesting that the GOB-BEV observation encourages more confident acceleration once the policy adapts to the new observation modality. The gap in total distance and route completion is most pronounced in Phase 2 (Sim2Real-AD: 150–250 m vs. original: 200–400 m), while Phase 1 trajectories track more closely, indicating that while DriveVLM-RL’s semantic safety reasoning transfers robustly, adapting to the noisier GOB-BEV observation reduces episode longevity. Cross-algorithm summary. Three observations hold across all algorithms: (i) No catastrophic performance collapse occurs at the Phase 1→ 2 boundary, empirically validating TPT’s progressive curriculum; (i) The physics action space (PAM) generally reduces total distance and route completion relative to original training, but affects collision rate in an algorithm-dependent manner—increasing it for DriveVLM-RL and VLM-RL (whose original collision rates are low) while decreasing it for ChatScene (whose original rate is very high); the reduction is most pronounced for ChatScene, while VLM-RL’s Phase 1 distance and route completion track close to its original curves; (i) Average speed in Phase 2 tends to increase across all algorithms after the GOB-BEV switch, suggesting that the noisier observation encourages more aggressive throttle behavior. 2) Performance Evaluation in Testing. Table 2 consolidates all three evaluation stages into a single unified view: Stage 1 (pre-transfer GT-BEV baseline), Stage 2 (PAM action space only), and Stage 3 (full Sim2Real-AD with GOB + PAM + TPT). Stage 1 results are reproduced from [Huang et al., 2026] and serve as the upper-bound reference; PR values in Stages 2 and 3 are computed relative to Stage 1. Table 2: Unified sim-to-real transfer results across three progressive stages (Town 2, 10 routes, mean ± std over 3 independent runs). PR(SR) and PR(TD) are computed relative to Stage 1, which serves as a non-deployable GT-BEV upper-bound reference. Best result per column within each stage in bold. Stage Algorithm AS↑ RC↑ TD↑ CS↓ SR↑ AC↓ PR(SR) PR(TD) Stage 1 GT-BEV Direct Action (Upper bound) ChatScene 17.47± 0.10 0.45± 0.03 161.78± 13.56 10.69± 5.02 0.57± 0.12 0.43± 0.12 — — VLM-RL 14.38± 1.53 0.51± 0.08 138.08± 16.68 10.09± 11.93 0.40± 0.00 0.10± 0.10 — — DriveVLM-RL 14.13± 1.07 0.45± 0.01 174.69± 8.96 0.29± 0.43 0.60± 0.00 0.17± 0.08 — — Stage 2 GT-BEV +PAM only ChatScene 15.72± 0.15 0.40± 0.04 142.37± 16.28 7.85± 4.15 0.47± 0.12 0.37± 0.10 82.5% 87.9% VLM-RL 12.94± 1.35 0.48± 0.07 130.15± 15.42 9.52± 10.85 0.37± 0.07 0.10± 0.08 92.5% 94.3% DriveVLM-RL 11.52± 0.85 0.39± 0.02 152.08± 10.23 0.33± 0.40 0.50± 0.00 0.23± 0.11 83.3% 87.1% Stage 3 GOB-BEV +PAM+TPT (Full S2R) ChatScene 15.28± 0.18 0.35± 0.04 108.95± 15.42 8.45± 5.02 0.37± 0.10 0.30± 0.10 64.9% 67.3% VLM-RL 15.45± 1.48 0.42± 0.07 102.45± 16.35 8.85± 9.52 0.27± 0.09 0.17± 0.14 67.5% 74.2% DriveVLM-RL 13.42± 0.92 0.36± 0.03 131.25± 11.08 0.38± 0.45 0.43± 0.12 0.27± 0.08 71.7% 75.1% Four key findings emerge from Table 2. Ordering preservation (answering RQ1). The absolute SR ranking, DriveVLM-RL >> ChatScene >> VLM-RL, is preserved across all three stages, with Spearman rank correlation ρs=1.0 _s=1.0 between Stage 1 and Stage 3. This confirms that Sim2Real-AD does not distort the algorithmic advantages established under ideal conditions. PAM’s impact is algorithm-dependent but predictable. Stage 2 reveals that PAM affects each algorithm differently depending on how much its original performance relies on fine-grained action control. VLM-RL retains the highest PR(SR) =92.5%=92.5\%, because its original collision rate is already moderate (∼ 0.45 in training, Fig. 5(a)) and PAM barely changes it. DriveVLM-RL retains PR(SR) =83.3%=83.3\%: its original collision rate is very low (∼ 0.20, Fig. 6(a)), partly relying on precise emergency maneuvers that PAM’s PID controller cannot reproduce, so collisions increase. ChatScene sits at PR(SR) =82.5%=82.5\%. However, DriveVLM-RL maintains the highest absolute SR (0.50) owing to its large Stage 1 headroom. The lower speed observed for DriveVLM-RL in Stage 2 (AS =11.52=11.52 km/h vs. original 14.13 km/h) is consistent with the conservative Phase 1 training speed visible in Fig. 6(b). DriveVLM-RL retains the most performance under full transfer. Stage 3 PR(SR) =71.7%=71.7\% and PR(TD) =75.1%=75.1\% for DriveVLM-RL, compared to 64.9–67.5% and 67.3–74.2% for baselines. The advantage over ChatScene (+6.8 p in PR(SR)) reflects that DriveVLM-RL’s dynamic pathway reward is computed on first-person camera images, which are unaffected by the Stage 3 BEV observation switch. ChatScene’s purely geometric reward, by contrast, is more sensitive to IPM spatial distortions (Fig. 4(c)). A notable observation is that Stage 3 AS increases relative to Stage 2 for both DriveVLM-RL (11.52→13.4211.52→ 13.42 km/h) and VLM-RL (12.94→15.4512.94→ 15.45 km/h), consistent with the speed rise observed in Phase 2 of the training curves (Figs. 6(b), 5(b)). This suggests that the noisier GOB-BEV observation reduces the policy’s learned caution, leading to faster but not necessarily safer driving. Collision severity is preserved, not just frequency. DriveVLM-RL’s CS remains ≤0.38≤ 0.38 km/h across all three stages (Stage 1: 0.29, Stage 2: 0.33, Stage 3: 0.38), while baselines exhibit CS of 7–11 km/h throughout. The monotone increase from Stage 1 to Stage 3 is small and expected (PAM introduces tracking lag; IPM reduces look-ahead), confirming that anticipatory braking transfers robustly. Note on speed–safety trade-off in Stage 3. VLM-RL achieves the highest Stage 3 speed (AS =15.45=15.45 km/h) and route coverage (RC =0.42=0.42), yet the lowest SR (0.27). This high-speed, low-safety profile is consistent with Fig. 5(b), where Phase 2 speed spikes to ∼ 15–17 km/h without a corresponding collision rate improvement (Fig. 5(a)). DriveVLM-RL’s more moderate speed (AS =13.42=13.42 km/h) paired with the highest SR (0.43) and lowest CS (0.38 km/h) confirms that its VLM-guided reward prioritizes safety over speed, and this conservative strategy transfers more effectively through the sim-to-real pipeline. This empirically validates the progressive curriculum property in Definition 3 and Remark 3: Phase 1 establishes the action-space foundation that makes Phase 2 observation adaptation stable, consistent with the training dynamics shown in Fig. 6. 5.1.6 Ablation Study (RQ2) We perform a module-level ablation using DriveVLM-RL as the base algorithm to quantify the contribution of each Sim2Real-AD component. Table 3 reports evaluation results on Town 2. Table 3: Module ablation (DriveVLM-RL backbone, Town 2, 10 routes, mean ± std over 3 independent runs). Original and Direct Transfer rows indicate upper and lower performance bounds. Best result per column among real-vehicle-deployable configurations in bold. †PAM only retains GT-BEV (available only in simulation). ∗GOB only retains the original direct action space. Neither † nor ∗ can be deployed on the real vehicle. ‡Direct Transfer: the original policy (GT-BEV + Direct Action) evaluated with GOB-BEV observations and no bridging modules in simulation, serving as a proxy for naive real-vehicle deployment. Configuration GOB PAM TPT SR↑ AC↓ TD↑ PR(SR) Original (upper bound) 0.60± 0.00 0.17± 0.08 174.69± 8.96 100% + PAM only† ✓ 0.50± 0.00 0.23± 0.09 152.08± 10.23 83.3% + GOB only∗ ✓ 0.37± 0.09 0.30± 0.10 126.84± 12.35 61.7% + GOB + PAM (1-phase) ✓ ✓ 0.30± 0.10 0.33± 0.10 110.56± 15.42 50.0% + GOB + PAM + TPT (ours) ✓ ✓ ✓ 0.43± 0.12 0.27± 0.08 131.25± 11.08 71.7% Direct transfer‡ (lower bound) 0.07± 0.05 0.67± 0.15 28.43± 10.23 11.7% Four findings stand out. Direct transfer fails catastrophically (SR =0.07=0.07, PR =11.7%=11.7\%): to simulate naive real-vehicle deployment within simulation, we feed the original GT-BEV-trained policy with GOB-BEV observations and retain its original direct action space, applying no bridging module. This serves as a controlled in-simulation proxy for what would happen if the CARLA-trained policy were deployed directly on a real vehicle — where GT-BEV is unavailable and GOB-BEV is the only accessible observation. The policy cannot interpret the IPM-generated BEV patterns that differ substantially from the GT-BEV it was trained on, causing catastrophic failure. This establishes the lower bound and confirms the necessity of all bridging modules. PAM alone (SR =0.50=0.50, PR =83.3%=83.3\%) preserves 83.3% of SR at a modest cost, primarily from PID speed-tracking lag and the nonlinear curvature-to-steering mapping at high curvatures. Note that this ablation retains GT-BEV and thus cannot be deployed on a real vehicle; nevertheless it isolates the PAM-induced performance loss, which is recoverable through tighter PID calibration (see Theorem 1, PAM term). GOB alone (SR =0.37=0.37, PR =61.7%=61.7\%) causes a larger drop than PAM alone. This ablation retains the original direct action space (simulation-only) and isolates the observation-side degradation: the IPM pipeline produces high-quality near-field BEV (<<10 m) but introduces significant distortion beyond 15 m (see Section 5.1.7), degrading the anticipatory cues the policy relies on for obstacle avoidance. GOB + PAM 1-phase (SR =0.30=0.30) falls below both individual ablations, demonstrating a super-additive difficulty effect consistent with the interaction analysis in Section 3.2: when trained from scratch on both a degraded observation space (GOB-BEV) and a novel action space (physics action) simultaneously, the reward signal is too noisy to establish stable lane-keeping before collision avoidance can be learned. The curriculum decomposition in TPT resolves this by first learning a stable policy under clean GT-BEV (Phase 1), then adapting only the observation distribution while the action interface and reward structure remain fixed (Phase 2). TPT is the critical enabler: the full framework recovers to SR =0.43=0.43 (PR =71.7%=71.7\%), a 43.3% relative improvement over the 1-phase baseline, while also improving total distance from 110.56 to 131.25 m. Notably, the full pipeline even surpasses the GOB-only ablation (SR 0.430.43 vs. 0.370.37; TD 131.25131.25 vs. 126.84126.84 m) despite bearing both observation and action degradation, demonstrating that TPT’s curriculum more than compensates for the added difficulty. This empirically validates the progressive curriculum property in Definition 3 and Remark 3, and is consistent with the TPT residual term in Theorem 1. 5.1.7 GOB Observation Quality Analysis (RQ3) 1) CARLA GOB-BEV vs. GT-BEV Visual Comparison. Fig. 7 presents side-by-side comparisons of the front-view camera input, the GOB-generated BEV, and the ground-truth semantic BEV across four representative CARLA frames spanning straight car-following segments, scenes with oncoming vehicles, scenes with cyclists, and an intersection (arranged in a 2×2 grid, two cases per row). Two key structural differences are evident: (i) the GOB-BEV captures road surface and lane markings only within the forward fan-shaped field of view (∼ 0–20 m ahead), whereas the GT-BEV provides full 360∘360 coverage with uniform resolution; (i) vehicles detected in the front view are projected to approximate BEV positions (red), while GT-BEV encodes all surrounding traffic from the simulator. Despite these inherent coverage limitations, the GOB-BEV reconstructs the road geometry directly ahead of the ego vehicle—the region most critical for the RL policy’s lane-keeping and collision-avoidance decisions. Figure 7: CARLA GOB evaluation on 4 representative frames (2 per row). Each case shows (left to right): front-view camera input, GOB-BEV, and GT-BEV. The GOB-BEV captures near-field road geometry (green), lane markings (yellow), and vehicles (red) within the forward fan-shaped field of view; the GT-BEV provides full 360∘360 coverage. 2) Quantitative Channel Analysis. We evaluate GOB-BEV fidelity on 200 paired frames from CARLA using two complementary metrics (Fig. 8): per-channel IoU between GOB-BEV and GT-BEV, and channel activation percentage (fraction of nonzero pixels). The moderate road-channel IoU of 0.35±0.100.35± 0.10 (Fig. 8(a)) reflects a structural coverage asymmetry rather than poor detection quality: GOB-BEV activates 51.1% of pixels as road (forward fan projection), while GT-BEV activates 31.1% (360∘360 coverage including side streets). Within the overlapping forward region, the spatial agreement is substantially higher. Lane, vehicle, walker, and traffic light channels show near-zero IoU (<<0.01) because (i) the GOB pipeline uses color-based lane detection (0.2% activation) whereas GT uses rendered markings (3.1%); (i) objects are projected to approximate BEV coordinates rather than exact positions; and (i) walkers and traffic lights rarely occupy sufficient pixels after perspective projection. Critically, the RL policy was trained on GT-BEV with these same channel semantics, so the road and vehicle channels, which together dominate the policy’s attention map, are the ones that matter most for safe driving. The road channel’s activation ratio (GOB/GT ≈ 1.6×) indicates adequate coverage of the policy’s forward decision region, explaining why Phase 2 fine-tuning successfully bridges the remaining gap. Figure 8: Quantitative GOB evaluation on 200 CARLA frames. (a) Per-channel IoU between GOB-BEV and GT-BEV: road achieves moderate IoU (0.350.35) limited by coverage asymmetry; other channels are near-zero due to detection-method differences. (b) Channel activation comparison: GOB-BEV over-activates road (51.1% vs. 31.1%) due to forward-fan projection, while lane and object channels have lower activation than GT, reflecting the single-camera field-of-view limitation. 3) Temporal Consistency. For a deployed RL policy, temporal stability of the BEV observation is as important as absolute accuracy, since flickering masks cause erratic control outputs. In our Ford E-Transit deployment, we evaluate two camera mounting positions: cam0 (forward-facing), used as the primary deployment camera in Section 5.2, and cam1 (elevated mount), evaluated as a comparison configuration. Fig. 9 plots the road-channel IoU between consecutive frames across a 163-frame real-world driving sequence from the Ford E-Transit. cam0 achieves a mean IoU of 0.9440.944, with 90.1% of frames above 0.900.90. cam1 achieves a mean IoU of 0.9690.969, with 95.1% of frames exceeding 0.900.90. The higher temporal consistency of cam1 is consistent with its elevated mounting position providing a more stable viewing angle and less susceptibility to vehicle pitch changes. Brief dips below 0.900.90 in both cameras correspond to sharp heading changes during lane transitions. This level of temporal stability confirms that the GOB pipeline produces smooth observations suitable for closed-loop RL control at 20 Hz, with cam0 fully satisfying the stability requirements of the primary deployment configuration. Figure 9: Temporal consistency of the road mask: IoU between adjacent frames on real-world driving data. cam0 (forward-facing) achieves mean IoU =0.944=0.944; cam1 (elevated mount) achieves mean IoU =0.969=0.969. The dashed line marks IoU =0.90=0.90. Figure 10: Real-world GOB output on 12 representative frames (Frame 0–110, interval 10) from the Ford E-Transit primary forward-facing camera (cam0), arranged in a 4×3 grid. Each cell shows the source image (left) and GOB-BEV (right). Road surface (green), lane markings (yellow), and vehicles (red) are consistently detected across diverse road conditions using zero-shot transfer from CARLA-trained SegFormer-B0. 4) Real-World GOB Visualization. Fig. 10 presents 12 representative GOB outputs from the primary forward-facing camera (cam0) on real-world driving footage, arranged in a 4×3 grid (frames labeled 0–110 at interval 10). The pipeline produces interpretable BEV masks across diverse conditions: straight roads with clear lane markings, gentle curves, and scenes with leading vehicles. Road surface (green) forms a consistent fan-shaped region ahead; lane markings (yellow) are detected via color filtering restricted to the semantic road region; and vehicles (red) are projected to approximate BEV positions using the bottom-center ground contact point of each detected bounding box as the IPM projection anchor. Notably, the CARLA-trained SegFormer-B0 model generalizes zero-shot to these real-world winter conditions without any domain-specific fine-tuning, supporting the claim that the GOB pipeline is platform-agnostic. 5.1.8 Cross-Map Generalization (RQ4) Models trained on Town 2 are evaluated on Towns 1, 3, 4, and 5 (all unseen during training). Town 2 (in-distribution) is included as a reference. We compare the original DriveVLM-RL (GT-BEV + direct action) with DriveVLM-RL + Sim2Real-AD (GOB-BEV + PAM via TPT). Table 4 shows that the absolute SR degradation from Sim2Real-AD scales with baseline SR. This floor effect is most evident in Towns 4 and 5 (baseline SR ≤0.10≤ 0.10), where Δ is −-0.04 and −-0.03 respectively, compared to −-0.17 for Towns 1 and 2 (baseline SR ≥0.57≥ 0.57). This indicates that the dominant factor limiting out-of-distribution performance is the policy’s limited exposure to diverse road layouts during training, not the sim-to-real transfer itself — the transfer overhead does not compound with map complexity. CS changes are mixed: collision severity decreases under Sim2Real-AD for Towns 3 (12.06→10.8212.06→ 10.82 km/h) and 5 (10.65→9.5310.65→ 9.53 km/h), suggesting that PAM’s speed modulation attenuates collision force in complex layouts. CS increases mildly in Towns 1 (0.73→0.880.73→ 0.88 km/h), 2 (0.29→0.380.29→ 0.38 km/h), and 4 (3.64→4.853.64→ 4.85 km/h), where baseline collision severity is already low relative to Towns 3 and 5. Table 4: Cross-map generalization (10 routes per town, trained on Town 2 only, mean ± std over 3 independent runs). Δ == SR change from GT-BEV to Sim2Real-AD. Town 2∗ is the in-distribution training town; GT-BEV results from Table 2. Town Method AS↑ TD↑ CS↓ SR↑ Δ Town 1 GT-BEV 17.91± 0.05 375.77± 69.53 0.73± 0.21 0.57± 0.12 — + S2R 16.42± 0.58 308.91± 72.14 0.88± 0.34 0.40± 0.10 −-0.17 Town 2∗ GT-BEV 14.13± 1.07 174.69± 8.96 0.29± 0.43 0.60± 0.00 — + S2R 13.42± 0.92 131.25± 11.08 0.38± 0.45 0.43± 0.12 −-0.17 Town 3 GT-BEV 16.78± 0.70 156.57± 29.20 12.06± 1.04 0.17± 0.09 — + S2R 15.31± 0.85 127.15± 31.42 10.82± 1.24 0.10± 0.05 −-0.07 Town 4 GT-BEV 19.48± 1.32 438.40± 284.27 3.64± 1.87 0.07± 0.05 — + S2R 18.12± 1.45 358.24± 215.38 4.85± 2.12 0.03± 0.03 −-0.04 Town 5 GT-BEV 15.37± 1.84 86.59± 14.23 10.65± 3.42 0.10± 0.05 — + S2R 14.18± 1.92 71.24± 16.85 9.53± 3.18 0.07± 0.08 −-0.03 5.2 Real-World Deployment Experiments (RQ5) We deploy the complete Sim2Real-AD framework on a full-scale autonomous vehicle to validate zero-shot sim-to-real transfer without any real-world training data. All policy parameters are frozen after simulation training; no real-world fine-tuning is performed at any stage. 5.2.1 Platform and Hardware The experiments were conducted on a lab-developed full-scale electric Ford E-Transit autonomous van equipped with a drive-by-wire system. As illustrated in Fig. 11(a), the sensor configuration includes three LiDAR units, seven high-resolution RGB cameras, and a front-facing radar, providing complementary spatial, semantic, and velocity information. A key point of the deployment is that Sim2Real-AD uses only the front-facing monocular camera for policy inference, demonstrating that the framework does not require expensive multi-sensor fusion for the observation bridge. Onboard computation is supported by an NVIDIA RTX A6000 GPU, which handles real-time inference for both the policy network and the perception stack. For data logging, telemetry, and remote monitoring, the vehicle is equipped with a NETGEAR Nighthawk M6 Pro 5G router, providing high-bandwidth and low-latency wireless connectivity during experiments. Figure 11: Real-world experimental platform and evaluation scenarios. (a) Lab-developed full-scale electric Ford E-Transit autonomous vehicle platform with drive-by-wire control, multimodal perception sensors, onboard computing, and wireless communication infrastructure. (b) Camera configuration: a forward-facing camera and an elevated camera mount, matching the dual-camera setup used in DriveVLM-RL training for reward computation; only the forward-facing camera is used for policy inference. (c) Real-world driving scenarios used for evaluation: routine car-following, static obstacle avoidance, and semantic-critical stop sign interaction. 5.2.2 Camera Configuration DriveVLM-RL’s reward computation during training relies on two camera views: a forward-facing ego-centric view for the static CLIP-based pathway and an elevated view for the dynamic LVLM pathway, which benefits from a wider spatial context. To maintain consistency with this training configuration and enable shadow-mode reward analysis (Section 5.2.4), we install two cameras on the Ford E-Transit as shown in Fig. 11(b): a forward-facing camera and an elevated camera mount. Both are calibrated using the checkerboard procedure described in Section 5.2.3. Critically, the RL policy itself receives only the forward-facing view as input through the GOB pipeline; the elevated camera is used exclusively for reward signal analysis and does not participate in the control loop. 5.2.3 Platform Calibration and Deployment Preparation A key practical advantage of Sim2Real-AD is its minimal setup effort compared to learning-based domain adaptation methods [Zhu et al., 2017, Ganin et al., 2016], which typically require thousands of real-world images and iterative GPU training. Table 5 summarizes the platform-specific calibration items together with the route preparation needed for real-world deployment. The vehicle calibration itself requires approximately 30 min and no training data, while route waypoint recording adds about 5 min for a given test site. Table 5: Practical setup for real-world deployment on the Ford E-Transit, including platform calibration and route preparation. Vehicle calibration requires approximately 30 min; total practical setup time is approximately 35 min. No training data are required. Parameter Value Method Time Front camera field of view 110∘110 FOV Checkerboard [Zhang, 2000] 15 min Camera height (h) 1.7 m Tape measure 2 min Camera pitch (α) 0∘0 Included above — Wheelbase (L) 3.67 m Vehicle manual 1 min Max steer angle (δmax _ ) 38.5∘38.5 Vehicle manual 1 min PID gains (Kp,Ki,Kd)(K_p,K_i,K_d) (0.8, 0.1, 0.15) Step-response test 10 min Vehicle calibration subtotal ∼ 30 min Route waypoints 47 points One manual drive 5 min Total practical setup time ∼ 35 min The PID gains for the E-Transit differ from the CARLA defaults owing to the vehicle’s greater mass and different throttle response characteristics. Recalibration requires only ∼ 10 min of step-response testing on a straight road, while the remaining calibration items are obtained through standard camera and vehicle measurements. This efficiency follows from the PAM design (Definition 2), which isolates the vehicle-specific mapping and allows the platform-dependent component to be updated without modifying the policy. 5.2.4 Deployment Protocol All policy parameters are frozen after simulation training and transferred to the real vehicle without any additional fine-tuning. The DriveVLM-RL-trained actor network runs at 20 Hz, generating curvature and desired speed commands that are translated to steering and throttle via PAM. Owing to its lightweight architecture, the policy satisfies real-time constraints and operates fully on onboard computation, enabling stable closed-loop driving. Although PAM maps policy outputs to desired speeds up to vmaxv_ = 35 km/h, the safety layer constrains actual vehicle speed to 15 km/h during initial real-world testing. To analyze the behavior of DriveVLM-RL’s dual-pathway reward structure under real-world conditions, we additionally run both reward pathways in shadow mode during deployment: they process live sensor data and produce reward signals, but their outputs have no effect on vehicle control decisions. This shadow-mode analysis is entirely separate from the control loop and is consistent with the training-deployment decoupling property described in Section 3: the deployed policy is a standalone lightweight network with no VLM inference in the control path. Shadow mode serves purely as a diagnostic tool to examine the alignment between semantic risk estimation and real-world behavioral responses, and does not alter the zero-shot transfer evaluation in any way. 5.2.5 Offline Policy Reaction Test Before closed-loop operation, we validate the full pipeline on ∼ 5 min of pre-recorded real-world driving footage (6,000 policy inference steps at 20 Hz) without actuating the vehicle. Table 6 reports the results. Table 6: Offline policy reaction test on real-world driving recordings (6,000 policy inference steps at 20 Hz). Validates that the GOB-to-policy chain produces sensible outputs before closed-loop testing. Metric Result Straight-road steering within expected band (|δ|<0.1|δ|<0.1) 91.3% Curve steer direction correctness 87.2% Speed output in valid range [0,vmax][0,v_ ] 99.8% End-to-end pipeline latency (mean / P95) 17.2 / 24.3 ms Figure 12: Quantitative real-world evaluation results over 20 trials per scenario. (a) Safety driver interventions across three scenarios. (b) Task success rates over 20 independent trials per scenario. Both algorithms are deployed through the identical Sim2Real-AD pipeline. The ± 0.1 steering threshold corresponds to approximately 3.85∘3.85 front wheel angle, consistent with the acceptable straight-road deviation in the safety layer (Section 4.5). On straight segments, 91.3% of steering outputs fall within this expected band. On curves, 87.2% have the correct sign; the remaining errors occur at curve entry/exit transitions where the road mask geometry changes abruptly as the vehicle begins to turn, causing momentary steering sign ambiguity before the forward-looking GOB-BEV stabilizes. Speed commands remain in the valid range for 99.8% of frames, confirming that PAM output is well-bounded. End-to-end latency averages 17.2 ms (P95: 24.3 ms), comfortably within the 50 ms budget of the 20 Hz control loop. 5.2.6 Evaluation Scenarios We evaluate the Sim2Real-AD framework across three real-world driving scenarios of increasing semantic complexity, as illustrated in Fig. 11(c). Both DriveVLM-RL and VLM-RL are deployed through the identical Sim2Real-AD pipeline, allowing direct comparison of how different reward paradigms transfer under the same bridging framework. • S1: Routine Car-Following. The agent maintains safe longitudinal control behind a leading vehicle exhibiting non-constant speed on an open road, evaluating basic speed regulation and distance keeping. • S2: Static Obstacle Avoidance. Static obstacles are placed along the driving route, requiring the agent to detect and safely maneuver around unexpected obstructions while maintaining lane-level control. • S3: Semantic-Critical Stop Sign Interaction. The agent must recognize a stop sign and execute appropriate stopping behavior, potentially in the presence of pedestrians near the crosswalk, evaluating semantic understanding and traffic rule compliance. 5.2.7 Quantitative Results For each scenario, we perform 20 independent trials under identical conditions. All experiments were conducted on December 23, 2025: Scenario 1 along Sprocket Drive, Madison, WI, and Scenarios 2 and 3 along Discovery Path, Madison, WI. We define three trial outcomes: • Success: the task is completed fully autonomously, i.e., reaching the designated goal region or completing the required interaction without any human intervention and within a reasonable time budget. • Safety Violation: a trained safety driver intervenes to prevent a collision or traffic rule violation; such trials are not counted as successes. • Stagnation: the vehicle remains safe but fails to complete the task within the allotted time due to indecision or overly conservative behavior. Results are summarized in Fig. 12. Safety violations are reported explicitly through safety driver intervention counts; stagnation cases are reflected implicitly in the task success rate. Figure 13: Representative DriveVLM-RL car-following episode (S1). Top row: forward-facing camera view (cam0), used as policy input through the GOB pipeline. Bottom row: elevated camera view (cam1), used for reward analysis in shadow mode. The ego vehicle maintains consistent following distance without triggering the dynamic reward pathway. (a) VLM-RL reward components (b) DriveVLM-RL reward components (c) Vehicle speed profiles (d) Steering angle profiles Figure 14: Qualitative analysis of S1 (routine car-following). (a) Temporal evolution of reward components for VLM-RL. (b) Temporal evolution of reward components for DriveVLM-RL: the dynamic pathway remains inactive throughout, confirming no unnecessary semantic intervention in routine scenarios. (c) Speed profiles: DriveVLM-RL achieves smoother speed regulation than VLM-RL. (d) Steering angle profiles: DriveVLM-RL produces more gradual adjustments, indicating more human-like control behavior. Three key findings emerge from the real-world evaluation. Sim2Real-AD enables functional zero-shot deployment. DriveVLM-RL deployed through Sim2Real-AD achieves success rates of 90%, 80%, and 75% across S1, S2, and S3 respectively, without any real-world training data or fine-tuning. These results confirm that the four-module pipeline (GOB, PAM, TPT, and RDP) successfully bridges the sim-to-real gap for closed-loop operation on a full-scale vehicle, validating the core claim of this work. The framework preserves semantic safety structure, not merely driving capability. The most significant difference between the two algorithms appears in S3, where VLM-RL requires safety driver intervention in 16 of 20 trials (80%), while DriveVLM-RL requires only 4 (20%) — a 75% reduction in critical safety violations. In S2, VLM-RL’s success rate drops to 25%: its static single-frame CLIP reward cannot reliably distinguish a temporary obstacle from a leading vehicle, leading to stagnation or unsafe behavior. DriveVLM-RL’s dynamic pathway activates context-dependent semantic reasoning in these situations, preserving the simulation-learned distinction under real-world visual variability. This demonstrates that Sim2Real-AD does not merely transfer motion control capability but specifically preserves the semantic safety structure of the backbone policy. Figure 15: Representative VLM-RL failure case in S2 (static obstacle avoidance). Top row: forward-facing camera view (cam0), used as policy input through the GOB pipeline. Bottom row: elevated camera view (cam1), used for reward analysis in shadow mode. VLM-RL continues forward without initiating a bypass maneuver, approaching the obstacle dangerously close and requiring safety driver intervention. (a) VLM-RL reward components (b) DriveVLM-RL reward components (c) Brake pressure profiles (d) Vehicle speed profiles Figure 16: Quantitative signal analysis of S2 (static obstacle avoidance). (a) VLM-RL reward components: dynamic pathway remains inactive throughout, providing no signal to initiate bypass. (b) DriveVLM-RL reward components: dynamic pathway activates when the obstacle enters the critical region. (c) Brake pressure profiles: VLM-RL decelerates but does not redirect, while DriveVLM-RL maintains controlled deceleration before the bypass. (d) Speed profiles: VLM-RL approaches the obstacle without bypass while DriveVLM-RL preserves forward progress through the maneuver. Performance ordering established in simulation is maintained in the real world. DriveVLM-RL outperforms VLM-RL across all three scenarios (90% vs. 70%, 80% vs. 25%, 75% vs. 10%), cross-validating the RQ1 ordering preservation finding from simulation in a real-world closed-loop setting. The fact that both algorithms are deployed through the identical Sim2Real-AD pipeline isolates reward design as the source of the performance difference, confirming that Sim2Real-AD is reward-agnostic and does not favor any particular training paradigm. 5.2.8 Qualitative Case Studies To further illustrate how Sim2Real-AD preserves algorithm-specific behavioral properties through the sim-to-real pipeline, we analyze representative behaviors in each scenario. S1: Routine Car-Following. Fig. 13 illustrates a representative DriveVLM-RL episode from two camera views: the forward-facing view (cam0) used for policy control, and the elevated view (cam1) used for shadow-mode reward analysis. Throughout the episode, the ego vehicle follows a leading vehicle along a roadway without abrupt maneuvers or external disturbances. As shown in Figs. 14(a)–(b), the dynamic reward pathway remains largely inactive for DriveVLM-RL in this scenario: the attentional gate does not trigger because no safety-critical objects appear within the detection radius. The combined reward is therefore determined entirely by the static pathway. This behavior is expected — routine car-following involves only a leading vehicle and no semantically critical events such as pedestrians, obstacles, or rule violations, all of which are explicitly handled by the static pathway through predefined language goals. The resulting control behaviors are compared in Figs. 14(c)–(d). VLM-RL exhibits noticeable speed oscillations and abrupt steering corrections, whereas DriveVLM-RL achieves smoother speed profiles and more gradual steering adjustments. This difference demonstrates that Sim2Real-AD successfully transfers the behavioral smoothness advantage of DriveVLM-RL’s dual-pathway design to the real vehicle: in routine scenarios, the dynamic pathway introduces no unnecessary semantic intervention, and the policy’s learned conservative control style transfers intact through the GOB and PAM bridges. (a) DriveVLM-RL reward signal decomposition during obstacle bypass (b) Vehicle trajectory comparison Figure 17: DriveVLM-RL reward mechanism and trajectory in S2. (a) Temporal evolution of DriveVLM-RL’s reward components: static reward, dynamic reward, combined reward, and attentional gate activation value. The gate fires sharply when the obstacle enters the critical detection region, driving the dynamic reward negative and shifting the combined reward polarity to penalize continued forward approach and encourage lateral bypass. (b) Resulting vehicle trajectories: DriveVLM-RL executes a smooth anticipatory lateral bypass while maintaining safe clearance; VLM-RL approaches the obstacle without redirecting and requires safety driver intervention. S2: Static Obstacle Avoidance. Fig. 15 shows a representative case from two camera views: the forward-facing view (cam0) used as policy input, and the elevated view (cam1) used for shadow-mode reward analysis. A parked vehicle partially occupies the driving lane and is accompanied by traffic cones, indicating a temporary obstruction that should be bypassed rather than waited for. As shown in Fig. 16(a), VLM-RL’s static CLIP-based reward provides no spatial disambiguation between a stopped vehicle and a temporary obstacle. Without a signal to initiate lateral bypass, the policy continues forward until the vehicle approaches the obstacle dangerously close, necessitating safety driver intervention. The brake pressure profile in Fig. 16(c) and speed profile in Fig. 16(d) confirm this: the policy decelerates but does not redirect, resulting in a straight-line approach toward the obstacle rather than a bypass maneuver. Figure 18: Representative episode in S3 (stop sign interaction with pedestrian). Top row: forward-facing camera view (cam0), used as policy input through the GOB pipeline. Bottom row: elevated camera view (cam1), used for reward analysis in shadow mode. A pedestrian is present at the crosswalk ahead of the stop sign. VLM-RL fails to yield and requires safety driver intervention; DriveVLM-RL decelerates smoothly and stops before the crosswalk. (a) VLM-RL reward components (b) DriveVLM-RL reward components (c) Brake pressure profiles (d) Vehicle speed profiles Figure 19: Quantitative signal analysis of S3 (stop sign interaction). (a) VLM-RL reward components: static reward provides no signal for stop sign compliance or pedestrian yielding; dynamic pathway remains inactive. (b) DriveVLM-RL reward components: dynamic pathway activates upon pedestrian detection, shifting reward polarity to penalize continued forward motion. (c) Brake pressure profiles: VLM-RL applies abrupt braking only at safety driver takeover; DriveVLM-RL initiates smooth early deceleration. (d) Speed profiles: VLM-RL maintains speed until intervention; DriveVLM-RL progressively decelerates to a complete stop before the crosswalk. In contrast, DriveVLM-RL successfully completes the task. Fig. 16(b) shows the temporal evolution of DriveVLM-RL’s reward signals during the episode: the static reward remains stable throughout, while the attentional gate activates sharply when the obstacle enters the critical detection region, triggering the dynamic pathway. The dynamic reward drops negative at this moment, penalizing continued forward approach, and the combined reward reflects this abrupt shift. As shown in Fig. 17(a), the LVLM correctly selects the language goal “An obstacle is on the road,” introducing the negative dynamic reward that drives the bypass decision. The resulting trajectory in Fig. 17(b) demonstrates a smooth anticipatory lateral bypass around the parked vehicle, maintaining safe clearance while preserving forward progress, without any safety driver intervention. This case study demonstrates that Sim2Real-AD successfully transfers DriveVLM-RL’s context-dependent semantic reasoning to the real vehicle. The dynamic pathway’s ability to detect safety-critical scenes and shift reward polarity, i.e., preserved intact through the GOB and PAM bridges, is precisely what enables correct disambiguation in this scenario, a capability that static reward paradigms cannot replicate. (a) LVLM semantic reasoning process during stop sign interaction (b) Distance and speed profiles Figure 20: DriveVLM-RL dynamic pathway reasoning and behavioral profiles in S3. (a) LVLM reasoning process when the attentional gate activates: the input prompt constructed from sequential visual observations, the LVLM’s multi-step reasoning response, and the final selected language description (“A pedestrian is crossing the road ahead”) that drives the dynamic reward. This process (learned entirely in CARLA simulation) transfers zero-shot to real-world conditions through the Sim2Real-AD pipeline. (b) Distance-to-pedestrian and vehicle speed profiles: DriveVLM-RL maintains increasing clearance while progressively reducing speed to a complete stop before the crosswalk; VLM-RL fails to yield and requires safety driver intervention. S3: Semantic-Critical Stop Sign Interaction. Fig. 18 shows a representative episode from two camera views: the forward-facing view (cam0) used as policy input, and the elevated view (cam1) for shadow-mode reward analysis. As the ego vehicle approaches a stop-controlled intersection, a pedestrian enters the crosswalk region, creating a semantically critical situation that cannot be resolved through geometric cues alone. As shown in Fig. 19(a), VLM-RL’s static reward fails to encode the stop sign rule and pedestrian right-of-way, providing no signal to initiate early deceleration. The vehicle continues forward without appropriate yielding behavior, approaching the intersection and pedestrian dangerously close and necessitating safety driver intervention. This is reflected in the brake pressure profile in Fig. 19(c) and speed profile in Fig. 19(d), where abrupt braking spikes coincide with safety driver takeover. In contrast, as shown in Fig. 19(b), DriveVLM-RL selectively activates the dynamic pathway when the attentional gate detects the safety-critical event. Fig. 20(a) illustrates the LVLM’s reasoning process when the attentional gate activates: the input prompt is constructed from sequential visual observations, the LVLM produces a multi-step reasoning response over candidate descriptions, and the final selected description “A pedestrian is crossing the road ahead” is used to compute the dynamic reward. This drives the dynamic reward sharply negative and shifts the combined reward polarity to penalize continued forward motion. This triggers an early, smooth deceleration, as confirmed by Fig. 20(b), which shows DriveVLM-RL maintaining increasing clearance distance to the pedestrian while progressively reducing speed to a complete stop before the crosswalk — without any safety driver intervention. This case study demonstrates that Sim2Real-AD successfully transfers DriveVLM-RL’s multi-frame semantic reasoning capability to the real vehicle. The dynamic pathway’s ability to aggregate sequential observations, apply semantic filtering, and select among candidate language descriptions (all learned exclusively in CARLA simulation) translates directly into correct traffic rule compliance under real-world conditions through the GOB and PAM bridges. This is the most demanding validation of Sim2Real-AD’s transfer fidelity: the policy must not only navigate correctly but interpret symbolic traffic semantics and reason about vulnerable road users, a capability that static reward paradigms cannot encode. 5.2.9 Real-Time Feasibility A practical prerequisite for any sim-to-real deployment framework is that the transferred policy must satisfy real-time control requirements on onboard hardware. Table 7 profiles the end-to-end inference pipeline of Sim2Real-AD on the Ford E-Transit’s onboard NVIDIA RTX A6000 GPU (mean ± std over 1,000 frames). The total end-to-end latency of 17.2±1.817.2± 1.8 ms satisfies the 20 Hz control rate with a 32.8 ms margin per cycle (65.6% of the 50 ms budget remaining for safety checks and system overhead). The segmentation stage dominates at 8.7 ms (50.6% of total), suggesting that lighter backbones could further reduce latency for resource-constrained platforms. Table 7: Per-stage inference latency on NVIDIA RTX A6000 GPU (mean ± std over 1,000 frames). Control budget: 50 ms at 20 Hz. Pipeline Stage Latency (ms) Budget Used (%) Image capture + preprocessing 2.1±0.32.1± 0.3 4.2 Semantic segmentation (SegFormer-B0) 8.7±1.28.7± 1.2 17.4 IPM projection (GOB) 3.4±0.53.4± 0.5 6.8 Policy inference (SAC forward) 1.8±0.21.8± 0.2 3.6 PAM (bicycle model + PID) <0.1<0.1 <0.2<0.2 CAN bus transmission 1.2±0.41.2± 0.4 2.4 Total end-to-end 17.2±1.817.2± 1.8 34.4 (a) End-to-end inference latency comparison (b) Control budget utilization Figure 21: Real-time feasibility on the Ford E-Transit onboard GPU. Sim2Real-AD satisfies the 20 Hz control budget with an average end-to-end control latency of 17.2 ms, whereas direct VLM-as-control inference is far beyond the real-time limit. To contextualize this result, we benchmark against LeapAD [Mei et al., 2024], a representative VLM-as-Control approach. It is important to note that our objective here is to evaluate computational efficiency and inference latency rather than safety performance; we therefore do not deploy the full LeapAD in real-world closed-loop control. Instead, we directly map camera observations to control decisions using Qwen3-VL, with outputs restricted to a set of high-level meta-actions (e.g., accelerate, decelerate, idle) that are translated into low-level acceleration commands through a predefined action library. All measurements use the same onboard NVIDIA RTX A6000 GPU; reported latencies cover end-to-end perception and inference, excluding sensor acquisition and CAN communication. As shown in Fig. 21(a), a single Qwen3-VL forward pass incurs 580±45580± 45 ms per step, limiting the achievable control frequency to 1–2 Hz and consuming 1160% of the 20 Hz budget — making real-time closed-loop operation infeasible. In contrast, Sim2Real-AD’s deployed pipeline requires only 34.4% of the budget, as illustrated in Fig. 21(b). The deployed system also occupies approximately 1.2 GB of VRAM (GOB backbone + SAC policy), compared to 24.5 GB required to load Qwen3-VL under FP16 precision. These results confirm that the training-deployment decoupling in Sim2Real-AD, where all VLM and LVLM components operate exclusively during offline training and are completely absent from the deployed control loop, is not merely a design convenience but a practical necessity for real-time vehicle control. The same property that enables Sim2Real-AD to focus its bridging modules on observation and action interfaces (Section 4.1) is what makes the framework computationally viable on standard onboard hardware. 6 Conclusion This paper presented Sim2Real-AD, a modular framework for zero-shot sim-to-real deployment of VLM-guided RL autonomous driving policies. By decomposing the sim-to-real problem into an observation-space bridge (GOB) and an action-space bridge (PAM), stabilized by a two-phase progressive training curriculum (TPT) and integrated into a real-time deployment pipeline (RDP), the framework transfers a CARLA-trained policy to a full-scale Ford E-Transit using only ∼ 30 minutes of lightweight calibration and no real-world RL training data. A formal transfer guarantee decomposes the sim-to-real performance gap into three independently controllable error terms. Simulation experiments confirmed ordering preservation across algorithms (Spearman ρs=1.0 _s=1.0) and the necessity of each module, while zero-shot closed-loop deployment achieved success rates of 90%, 80%, and 75% across three real-world scenarios. The framework is reward-agnostic and was validated across ChatScene, VLM-RL, and DriveVLM-RL, confirming its role as a universal deployment bridge for RL-based driving policies. Several directions remain open for future work. The current 15 km/h speed limit is safety-conservative; scaling to higher speeds requires tighter PID calibration and potentially multi-camera sensing to extend BEV coverage for longer-horizon planning, directly reducing the PAM and GOB error terms in the transfer guarantee. The forward-only monocular configuration limits GOB quality at sharp turns and wide intersections; surround-view sensing would improve effective range without modifying the policy. GPS route dependency could be removed through online map matching or vision-based lane centerline extraction. Finally, incorporating a small number of real-world rollouts into Phase 2 ( analogous to Rapid Motor Adaptation [Kumar et al., 2021]) could directly reduce the residual distribution gap dTV(2,real)d_TV(O_2,O^real) and further close the zero-shot performance ceiling. Acknowledgment This work was supported by the University of Wisconsin-Madison’s Center for Connected and Automated Transportation (CCAT), a part of the larger CCAT consortium, a USDOT Region 5 University Transportation Center funded by the U.S. Department of Transportation, Award #69A3552348305. The contents of this paper reflect the views of the authors, who are responsible for the facts and the accuracy of the data presented herein, and do not necessarily reflect the official views or policies of the sponsoring organization. Appendix A Pseudocode for Training and Deployment This appendix provides complete pseudocode for the Two-Phase Progressive Training procedure (Algorithm 1) and the Real-Time Deployment Pipeline (Algorithm 2) of Sim2Real-AD. Both algorithms compose directly on top of the DriveVLM-RL training backbone [Huang et al., 2026], whose asynchronous reward-annotation infrastructure is reused in both phases without modification. Algorithm 1 Two-Phase Progressive Training (TPT) 1:DriveVLM-RL reward oracle ℛ(⋅)R(·) (static + dynamic pathways, hierarchical synthesis [Huang et al., 2026]), GOB pipeline G (SegFormer-B0 + IPM, calibrated with ,h,α,βK,h,α,β), PAM limits κmax,vmax _ ,\,v_ , Phase 1 steps T1=1×106T_1=1× 10^6, Phase 2 steps T2=5×105T_2=5× 10^5, SAC hyperparameters (entropy coefficient λ, target smoothing τ, batch size B, replay buffer D, reward-worker interval Δ , warmup threshold NwarmupN_warmup) 2:// —- Phase 1: Action-Space Adaptation —- 3:Initialize policy πϕ _φ, Q-functions QθQ_θ, target θ−←θ^-←θ, Nready←0N_ready← 0 4:for t=1,2,…,T1t=1,2,…,T_1 do 5: // Interaction thread 6: Observe ot=(o^tsim,otcam,st,wt)o_t=( o_t^sim,\,o_t^cam,\,s_t,\,w_t) from CARLA ⊳ GT-BEV o^tsim o_t^sim, front-view, ego-state, waypoints 7: (a^1,t,a^2,t)∼πϕ(⋅∣ot)( a_1,t,\, a_2,t) _φ(· o_t) 8: Decode PAM: κt←a^1,tκmax _t← a_1,t\, _ , vtd←a^2,t+12vmaxv_t^d← a_2,t+12\,v_ 9: Execute (κt,vtd)( _t,v_t^d) in CARLA via bicycle-model steering and simulated PID speed tracking; observe ot+1o_t+1 10: Store (ot,a^t,ot+1,rt←NaN,ready←0)(o_t,\, a_t,\,o_t+1,\;r_t← NaN,\; ready← 0) in D 11: if tmodΔ=0t =0 then ⊳ Reward worker thread 12: Sample mini-batch from D where ready=0 ready=0 13: Annotate each transition with Rfinal←ℛ(oi,oi+1)R_final (o_i,o_i+1); set ready←1 ready← 1, Nready+=1N_ready +=1 14: end if 15: if Nready≥NwarmupN_ready≥ N_warmup then ⊳ Learner thread 16: Sample mini-batch (oi,a^i,ri,oi+1)\(o_i, a_i,r_i,o_i+1)\ from D with ready=1 ready=1 17: Update QθQ_θ: minimize Bellman residual loss JQ(θ)J_Q(θ) 18: Update πϕ _φ: maximize RL objective Eq. (1) with SAC 19: Soft-update targets: θ−←(1−τ)θ−+τθ^-←(1-τ)\,θ^-+τ\,θ 20: end if 21:end for 22:θ1←ϕ _1←φ ⊳ Save Phase 1 checkpoint 23:// —- Phase 2: Observation-Space Adaptation —- 24:ϕ←θ1φ← _1, θ−←θ^-←θ, Nready←0N_ready← 0 ⊳ Warm-start from Phase 1 25:for t=1,2,…,T2t=1,2,…,T_2 do 26: // Interaction thread 27: Capture rendered front-view image ItI_t from CARLA camera 28: Apply GOB: St←fseg(It)S_t← f_seg(I_t), S~t←fipm(St;,h,α,β) S_t← f_ipm(S_t;\,K,h,α,β), o^tsim←Encode(S~t) o_t^sim ( S_t) 29: ot←(o^tsim,otcam,st,wt)o_t←( o_t^sim,\,o_t^cam,\,s_t,\,w_t) ⊳ GOB-BEV o^tsim o_t^sim replaces GT-BEV; all else unchanged 30: Decode PAM, execute, store as in Phase 1 31: if tmodΔ=0t =0 then 32: Annotate batch using ℛ(⋅)R(·); set ready←1 ready← 1 33: end if 34: if Nready≥NwarmupN_ready≥ N_warmup then 35: Update QθQ_θ, πϕ _φ via SAC; soft-update θ−θ^- 36: end if 37:end for 38:return trained policy πθ2←πϕ _ _2← _φ Algorithm 2 Sim2Real-AD Real-Time Deployment Pipeline (RDP) 1:Trained policy πθ2 _ _2, GOB pipeline G with calibrated (,h,α,β)(K,h,α,β), PAM parameters Θ=L,δmax,Kp,Ki,Kd =\L,\, _ ,\,K_p,\,K_i,\,K_d\, PAM limits κmax,vmax _ ,\,v_ , pre-recorded GPS route routeW_route, safety bounds (vlimit,dlimit,Δδmax,rsafe)(v_limit,\,d_limit,\, _ ,\,r_safe), anti-windup clip emaxe_ , control period Δt=1/20 t=1/20 s 2:Initialize: eint←0e_int← 0, eprev←0e_prev← 0, δprev←0 _prev← 0 3:while not stop-signal received do 4: // Stage 1: Perception (≈ 14 ms) 5: It←Camera.capture()I_t .capture() 6: St←fseg(It)S_t← f_seg(I_t) 7: o^tsim←Encode(fipm(St;,h,α,β)) o_t^sim (f_ipm(S_t;\,K,h,α,β)) 8: // Stage 2: Route and State (<<1 ms) 9: wt←MatchRoute(GPSt,route)w_t (GPS_t,\,W_route) ⊳ Nearest-point match → vehicle-frame waypoints 10: (vt,δtcur,τt)←CAN.read()(v_t,\, _t^cur,\, _t) .read() 11: st←(vt/vmax,δtcur/δmax,τt/100)s_t←(v_t/v_ ,\; _t^cur/ _ ,\; _t/100) ⊳ Normalized ego-state 12: // Stage 3: Policy Inference (≈ 2 ms) 13: (a^1,t,a^2,t)←πθ2(o^tsim,st,wt)( a_1,t,\, a_2,t)← _ _2( o_t^sim,\,s_t,\,w_t) 14: // Stage 4: PAM (<<1 ms) 15: κt←a^1,tκmax _t← a_1,t\, _ , vtd←a^2,t+12vmaxv_t^d← a_2,t+12\,v_ 16: δtdes←arctan(Lκt) _t^des← (L\, _t), utδ←δtdes/δmaxu_t^δ← _t^des/ _ 17: et←vtd−vte_t← v_t^d-v_t 18: eint←clip(eint+etΔt,−emax,emax)e_int (e_int+e_t\, t,\;-e_ ,\;e_ ) 19: utv←Kpet+Kieint+Kd(et−eprev)/Δtu_t^v← K_p\,e_t+K_i\,e_int+K_d\,(e_t-e_prev)/ t 20: eprev←ete_prev← e_t 21: // Stage 5: Safety Layer 22: utv←min(utv,vlimit)u_t^v← (u_t^v,\;v_limit) ⊳ Speed cap 23: utδ←clip(utδ,δprev−Δδmax,δprev+Δδmax)u_t^δ (u_t^δ,\; _prev- _ ,\; _prev+ _ ) ⊳ Steering-rate limit 24: if isNaN(utδ)\;isNaN(u_t^δ) ∨isNaN(utv) \;isNaN(u_t^v) ∨LaneDev(o^tsim)>dlimit \;LaneDev( o_t^sim)>d_limit ∨ObstacleIn(rsafe) \;ObstacleIn(r_safe) ∨EStop() \;EStop() ∨OutsideGeofence() \;OutsideGeofence() then 25: (utδ,utv)←(0,−1)(u_t^δ,\,u_t^v)←(0,\;-1) ⊳ Emergency brake: zero steer, full brake 26: end if 27: if DriverTakeover()DriverTakeover() then 28: break ⊳ Safety driver override: release control 29: end if 30: // Stage 6: Actuation (<<1 ms) 31: CAN.send(utδ,utv)CAN.send(u_t^δ,\;u_t^v) 32: δprev←utδ _prev← u_t^δ 33: sleep(Δt−telapsed)sleep( t-t_elapsed) ⊳ Maintain 20 Hz loop 34:end while Appendix B Theoretical Analysis of Sim2Real-AD This appendix provides formal statements and proofs for the theoretical claims in Section 4. We proceed in five steps: Assumptions (B.1), GOB perceptual bound (Proposition 1), PAM tracking bound (Proposition 2), TPT distribution-shift bound (Proposition 3), and the main zero-shot transfer guarantee (Theorem 1). All results are stated with respect to the POMDP formulation in Section 3, with reward rt=ℱ(rttask,rtsem,st)r_t=F(r_t^task,\,r_t^sem,\,s_t) as defined in Eq. (2). B.1 Notation and Assumptions Assumption 1 (Lipschitz Policy) The policy πθsim _θ^sim is LπL_π-Lipschitz with respect to its BEV input: for any two BEV tensors o^,o^′∈^sim o, o ∈ O^sim, ‖πθsim(o^)−πθsim(o^′)‖2≤Lπ‖o^−o^′‖1. \| _θ^sim( o)- _θ^sim( o ) \|_2\;≤\;L_π \| o- o \|_1. (27) Remark 1 Neural networks with bounded weights and smooth activation functions (ELU, Tanh) are Lipschitz. In practice, LπL_π can be estimated via spectral normalization or empirical Jacobian bounds [Miyato et al., 2018]. The SAC-trained policy uses gradient clipping and weight decay, which empirically constrain the Lipschitz constant. Assumption 2 (Lipschitz Reward) The reward rt=ℱ(rttask,rtsem,st)r_t=F(r_t^task,\,r_t^sem,\,s_t) from Eq. (2) is LrL_r-Lipschitz with respect to vehicle state and satisfies |rt|≤Rmax|r_t|≤ R_ for all t. Remark 2 The CLIP-based [Radford et al., 2021] semantic term rtsemr_t^sem is 1-Lipschitz in the visual embedding since cosine similarity is 1-Lipschitz on the unit sphere. The task reward term rttaskr_t^task comprises differentiable, bounded driving signals (speed, lane deviation, collision indicator) and is Lipschitz by construction. The composite reward Eq. (2) is therefore Lipschitz with bounded constant LrL_r. Assumption 3 (Bounded Segmentation Error) The SegFormer-B0 model produces segmentation maps such that the expected L1L_1 distance between sim and real BEV tensors satisfies: [‖o^tsim−o^treal‖1]≤ϵseg,E\! [\, \| o_t^sim- o_t^real \|_1 ]\;≤\; _seg, (28) where the expectation is over the joint randomness in lighting, texture, sensor noise, and scene sampling. Remark 3 SegFormer-B0 achieves 37.4 mIoU on ADE20K under distribution shift [Xie et al., 2021]. In our setting, paired rollouts can be collected in CARLA [Dosovitskiy et al., 2017] by rendering the same scenario simultaneously from a privileged semantic view and a camera view, enabling direct empirical measurement of ϵseg _seg. Assumption 4 (Bounded Path Deviation) The executed path deviates from the intended path by at most ϵpid>0 _pid>0 in curvature at each timestep: |κtexecuted−κt|≤ϵpid, | _t^executed- _t |≤ _pid, (29) where κtexecuted _t^executed is the curvature realized by the physical vehicle and κt _t is the curvature commanded by PAM. This bound encompasses both direct curvature tracking error (ϵκ _κ) and the effective curvature deviation induced by speed tracking error, and can be measured directly during the calibration procedure of Section 4.3. Remark 4 The path deviation bound ϵpid _pid is tunable through PID gain selection and is not a fixed system constant. Tighter gains reduce ϵpid _pid at the cost of increased control effort and potential actuation saturation; the calibration protocol in Section 4.3 identifies gains that keep ϵpid _pid small while maintaining stable closed-loop behavior on the Ford E-Transit. B.2 GOB Perceptual Bound Proposition 1 (GOB Observation Error Bound) Under Assumptions 1 and 3, the expected L2L_2 deviation in policy output caused by the perceptual gap satisfies: [‖πθsim(o^tsim)−πθsim(o^treal)‖2]≤Lπ⋅ϵseg.E\! [\, \| _θ^sim( o_t^sim)- _θ^sim( o_t^real) \|_2 ]\;≤\;L_π· _seg. (30) Proof 1 Applying the Lipschitz condition (Assumption 1) and then the expectation bound (Assumption 3): [‖πθsim(o^tsim)−πθsim(o^treal)‖2] \! [\, \| _θ^sim( o_t^sim)- _θ^sim( o_t^real) \|_2 ] ≤Lπ⋅[‖o^tsim−o^treal‖1]≤Lπ⋅ϵseg. \;≤\;L_π·E\! [\, \| o_t^sim- o_t^real \|_1 ]\;≤\;L_π· _seg. (31) Remark 5 Proposition 1 shows that the GOB contribution to sim-to-real performance degradation scales linearly with ϵseg _seg. Replacing SegFormer-B0 with a stronger segmentation model directly reduces ϵseg _seg and tightens the bound without requiring any policy retraining. B.3 PAM Tracking Error Bound Proposition 2 (PAM Lateral Position Tracking Bound) Under Assumption 4 and a kinematic bicycle model with wheelbase L, the lateral position error accumulated over a control horizon of T steps satisfies: |yTreal−yTintended|≤vmax2T22ϵpid, |y_T^real-y_T^intended |\;≤\; v_ ^2T^22\, _pid, (32) where vmaxv_ is the maximum vehicle speed. Proof 2 Under the bicycle model, the lateral dynamics satisfy y˙≈vψ y≈ vψ for small heading angle ψ, and the heading rate satisfies ψ˙=vκ ψ=vκ. A curvature tracking error Δκt=|κtexecuted−κt|≤ϵpid _t=| _t^executed- _t|≤ _pid induces a heading angle error: |Δψ(t)|≤∫0tv(τ)|Δκ(τ)|τ≤vmaxϵpidt.| ψ(t)|≤ _0^tv(τ)\,| κ(τ)|\,dτ\;≤\;v_ \, _pid\,t. (33) Integrating the resulting lateral position error: |yTreal−yTintended| |y_T^real-y_T^intended | ≤∫0Tv(t)|Δψ(t)|t≤vmax∫0Tvmaxϵpidtt=vmax2T22ϵpid. ≤ _0^Tv(t)\,| ψ(t)|\,dt\;≤\;v_ _0^Tv_ \, _pid\,t\,dt= v_ ^2T^22\, _pid. (34) Remark 6 The lateral error bound grows quadratically with the horizon T and quadratically with vmaxv_ . This motivates both the 20 Hz closed-loop control frequency (shorter Δt t reduces the effective horizon over which errors accumulate) and the conservative 15 km/h speed cap during initial testing (at vmax≈4.2v_ ≈ 4.2 m/s the quadratic term is substantially smaller than at highway speeds). B.4 TPT Distribution Shift Bound Proposition 3 (TPT Performance Bound under Distribution Shift) Let J(π,)J(π,O) denote the expected discounted return Eq. (1) of policy π under observation distribution O. Under Assumption 2, for any policy π and any two observation distributions 2O_2 and realO^real: J(π,real)≥J(π,2)−2RmaxdTV(2,real)(1−γ)2.J\! (π,\;O^real )\;≥\;J\! (π,\;O_2 )- 2\,R_ \,d_TV\! (O_2,\,O^real )(1-γ)^2. (35) Proof 3 We use a Bellman-recursion sensitivity argument in two steps. Step 1: Per-state value difference. For any state s, define the value function Vπ,(s)=π[∑t≥0γtrt∣s0=s,]V^π,O(s)=E_π[ _t≥ 0γ^tr_t s_0=s,O]. Since |rt|≤Rmax|r_t|≤ R_ , the Bellman operator is a γ-contraction and |Vπ,(s)|≤Rmax/(1−γ)|V^π,O(s)|≤ R_ /(1-γ). By the standard total variation inequality, for any bounded measurable function f with ‖f‖∞≤M\|f\|_∞≤ M and distributions P, Q, |P[f]−Q[f]|≤2M⋅dTV(P,Q)|E_P[f]-E_Q[f]|≤ 2M· d_TV(P,Q) [Levin and Peres, 2017], the single-step reward difference at state s satisfies: |r(s,π,real)−r(s,π,2)|≤ 2Rmax⋅dTV(real(s),2(s)). |r(s,π,O^real)-r(s,π,O_2) |\;≤\;2\,R_ · d_TV\! (O^real(s),\,O_2(s) ). (36) Applying this bound at each step of the Bellman recursion and summing the geometric series: |Vπ,real(s)−Vπ,2(s)|≤2Rmax1−γsups′dTV(real(s′),2(s′)). |V^π,O^real(s)-V^π,O_2(s) |\;≤\; 2\,R_ 1-γ _s \,d_TV\! (O^real(s ),\,O_2(s ) ). (37) Step 2: Return difference via state visitation. The performance difference J(π,real)−J(π,2)J(π,O^real)-J(π,O_2) can be written in terms of the discounted state visitation measure dπ,d^π,O of each environment: J(π,real)−J(π,2)=(1−γ)s∼dπ,real[Vπ,real(s)−Vπ,2(s)]+Δvis,J(π,O^real)-J(π,O_2)=(1-γ)\,E_s d^π,O^real\! [V^π,O^real(s)-V^π,O_2(s) ]+ _vis, (38) where Δvis _vis accounts for the difference in state visitation between the two environments. Since |Vπ,(s)|≤Rmax/(1−γ)|V^π,O(s)|≤ R_ /(1-γ), the visitation mismatch contributes at most 2Rmax1−γdTV(dπ,real,dπ,2) 2R_ 1-γ\,d_TV(d^π,O^real,d^π,O_2). The total variation distance between state visitation measures is bounded by the per-step observation TV via the simulation lemma [Kakade and Langford, 2002]: dTV(dπ,real,dπ,2)≤11−γsups′dTV(real(s′),2(s′))d_TV(d^π,O^real,d^π,O_2)≤ 11-γ _s d_TV(O^real(s ),O_2(s )). Combining with Eq. (37) and applying sups′dTV(real(s′),2(s′))≤dTV(real,2) _s d_TV(O^real(s ),O_2(s ))≤ d_TV(O^real,O_2) (TV is non-increasing under marginalization): J(π,real)−J(π,2)≥−2Rmax(1−γ)2dTV(2,real).J(π,O^real)-J(π,O_2)\;≥\;- 2\,R_ (1-γ)^2\,d_TV\! (O_2,\,O^real ). (39) Rearranging yields Eq. (35). Remark 7 The (1−γ)−2(1-γ)^-2 factor has a clear two-factor interpretation: one (1−γ)−1(1-γ)^-1 from summing the per-step reward difference over the discounted horizon (Step 1), and a second (1−γ)−1(1-γ)^-1 from the mismatch in state visitation measures between the two environments (Step 2). Note that LπL_π does not appear here: the Lipschitz policy constant connects BEV distance to action deviation (Proposition 1), but is not needed in the Bellman-recursion sensitivity argument. This also motivates Phase 2 of TPT: reducing dTV(2,real)d_TV(O_2,O^real) by training on IPM-generated observations directly tightens the bound. B.5 Main Zero-Shot Transfer Guarantee Theorem 1 (Zero-Shot Transfer Guarantee) Under Assumptions 1–4, the expected cumulative reward of the Sim2Real-AD composed policy πreal=ℳ∘πθsim∘π^real=M _θ^sim (abbreviated; see Eq. (4) for the full definition including sts_t and wtw_t) on the real vehicle satisfies: [∑t=0Tγtrtreal]≥[∑t=0Tγtrtsim]−LrLπϵseg1−γ⏟GOB term−Lrvmax2T2ϵpid2(1−γ)⏟PAM term−2RmaxdTV(2,real)(1−γ)2⏟TPT term,E\! [ _t=0^Tγ^tr_t^real ]\;≥\;E\! [ _t=0^Tγ^tr_t^sim ]- L_rL_π\, _seg1-γ_GOB term- L_r\,v_ ^2T^2\, _pid2(1-γ)_PAM term- 2\,R_ \,d_TV\! (O_2,\,O^real )(1-γ)^2_TPT term, (40) where rtsim=ℱ(rttask,rtsem,st)r_t^sim=F(r_t^task,\,r_t^sem,\,s_t) is the VLM-guided reward Eq. (2) and LrL_r is its Lipschitz constant. Remark 8 (Correspondence with Simplified Theorem) The constants C1C_1, C2C_2, C3C_3 in Theorem 1 (Section 4.6) correspond to the explicit expressions in Eq. (40) as follows: C1=LrLπ1−γ,C2=Lrvmax2T22(1−γ),C3=2Rmax.C_1= L_rL_π1-γ, C_2= L_r\,v_ ^2T^22(1-γ), C_3=2\,R_ . (41) Note that C3C_3 does not depend on LπL_π: the TPT bound (Theorem 3) is controlled by reward magnitude RmaxR_ alone, not by how sensitively the policy maps BEV inputs to actions. All three constants decrease as the reward becomes less sensitive to state perturbations (smaller LrL_r), the policy becomes more Lipschitz-regular (smaller LπL_π, relevant for C1C_1), or the planning horizon shortens (smaller T, relevant for C2C_2). Proof 4 We introduce three intermediate reward sequences to make the triangle inequality argument explicit. Define: • rt(0)≜rtsimr_t^(0) r_t^sim: reward under ideal simulation (ground-truth BEV o^tsim o_t^sim, perfect curvature tracking, observation distribution 1O_1); • rt(1)r_t^(1): reward after substituting GOB-generated BEV o^treal o_t^real for ground-truth BEV o^tsim o_t^sim, while keeping perfect curvature tracking and operating under 1O_1; • rt(2)r_t^(2): reward after further introducing PAM curvature tracking error |Δκt|≤ϵpid| _t|≤ _pid, while operating under the Phase 2 observation distribution 2O_2; • rtrealr_t^real: actual deployment reward (GOB error + PAM error + residual distribution gap between 2O_2 and realO^real). By the triangle inequality: rtsim−rtreal=(rt(0)−rt(1))⏟GOB error+(rt(1)−rt(2))⏟PAM error+(rt(2)−rtreal)⏟TPT residual.r_t^sim-r_t^real= (r_t^(0)-r_t^(1) )_GOB error+ (r_t^(1)-r_t^(2) )_PAM error+ (r_t^(2)-r_t^real )_TPT residual. (42) Step 1 (GOB term). Substituting o^treal o_t^real for o^tsim o_t^sim changes the policy output by at most ‖πθsim(o^tsim)−πθsim(o^treal)‖2≤Lπϵseg\| _θ^sim( o_t^sim)- _θ^sim( o_t^real)\|_2≤ L_π _seg (Proposition 1). By the LrL_r-Lipschitz reward (Assumption 2), the per-step reward difference satisfies |rt(0)−rt(1)|≤LrLπϵseg|r_t^(0)-r_t^(1)|≤ L_rL_π _seg. Summing over the discounted horizon using ∑t=0Tγt≤1/(1−γ) _t=0^Tγ^t≤ 1/(1-γ): ∑t=0Tγt[rt(0)−rt(1)]≤LrLπϵseg1−γ. _t=0^Tγ^t\,E\! [r_t^(0)-r_t^(1) ]\;≤\; L_rL_π\, _seg1-γ. (43) Step 2 (PAM term). Imperfect curvature tracking (|Δκt|≤ϵpid| _t|≤ _pid) causes a lateral position error bounded by vmax2T22ϵpid v_ ^2T^22 _pid (Proposition 2). By the LrL_r-Lipschitz reward: ∑t=0Tγt[rt(1)−rt(2)]≤Lrvmax2T2ϵpid2(1−γ). _t=0^Tγ^t\,E\! [r_t^(1)-r_t^(2) ]\;≤\; L_r\,v_ ^2T^2\, _pid2(1-γ). (44) Step 3 (TPT residual). The remaining gap between operating under 2O_2 and realO^real is bounded by Proposition 3: ∑t=0Tγt[rt(2)−rtreal]≤2RmaxdTV(2,real)(1−γ)2. _t=0^Tγ^t\,E\! [r_t^(2)-r_t^real ]\;≤\; 2\,R_ \,d_TV\! (O_2,\,O^real )(1-γ)^2. (45) Summing Steps 1–3 and rearranging yields Eq. (40). Corollary 1 (Convergence under Ideal Conditions) If ϵseg→0 _seg→ 0, ϵpid→0 _pid→ 0, and dTV(2,real)→0d_TV(O_2,O^real)→ 0, then: [∑t=0Tγtrtreal]⟶[∑t=0Tγtrtsim].E\! [ _t=0^Tγ^tr_t^real ]\; \;E\! [ _t=0^Tγ^tr_t^sim ]. (46) Proof 5 Immediate from Theorem 1 by taking all three error terms to zero. Remark 9 Corollary 1 establishes that Sim2Real-AD is asymptotically lossless: as each module approaches its theoretical ideal, real-vehicle performance converges to simulation performance under the VLM-guided RL objective Eq. (1). The three error terms in Theorem 1 are independently controllable: GOB via improved segmentation (ϵseg↓ _seg ), PAM via tighter path tracking (ϵpid↓ _pid ), and TPT via longer Phase 2 training (dTV(2,real)↓d_TV(O_2,O^real) ), defining three orthogonal axes along which the framework can be improved. References M. Abdel-Aty and S. Ding (2024) A matched case-control analysis of autonomous vs human-driven vehicle accidents. Nature communications 15 (1), p. 4931. Cited by: §1. T. Alqahtani (2025) Recent trends in the public acceptance of autonomous vehicles: a review. Vehicles 7 (2), p. 45. Cited by: §1. K. J. Astrom and T. Hägglund (2006) Advanced pid control. IEEE Control Systems 26 (1), p. 98–101. Cited by: §4.3.2. Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, p. 41–48. Cited by: §2.2, §4.4.3. M. Bertozzi and A. Broggi (1998) GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection. IEEE transactions on image processing 7 (1), p. 62–81. Cited by: §2.2, §4.2.1. CARLA Simulator Team (2020) CARLA autonomous driving leaderboard. Note: Accessed: 2025 External Links: Link Cited by: §1. K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger (2022) Transfuser: imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence 45 (11), p. 12878–12895. Cited by: §1, §2.1. F. Codevilla, E. Santana, A. M. López, and A. Gaidon (2019) Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, p. 9329–9338. Cited by: §1, §2.1. I. G. Daza, R. Izquierdo, L. M. Martínez, O. Benderius, and D. F. Llorca (2023) Sim-to-real transfer and reality gap modeling in model predictive control for autonomous driving. Applied Intelligence 53 (10), p. 12719–12735. Cited by: §1, §1, §2.2. E. Delavari, F. K. Khanzada, and J. Kwon (2025) A comprehensive review of reinforcement learning for autonomous driving in the carla simulator. arXiv preprint arXiv:2509.08221. Cited by: §2.1, §2.3. A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun (2017) CARLA: an open urban driving simulator. In Conference on robot learning, p. 1–16. Cited by: §1, §1, §5.1.1, Remark 3. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky (2016) Domain-adversarial training of neural networks. Journal of machine learning research 17 (59), p. 1–35. Cited by: §1, §1, §2.2, §5.2.3. S. Gao, Q. Wang, and Y. Sun (2025) Seq-bev: semantic bird-eye-view map generation in full view using sequential images for autonomous driving. IEEE Transactions on Intelligent Transportation Systems. Cited by: §2.2. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §5.1.2. Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. (2023) Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 17853–17862. Cited by: §2.1. Z. Hu, M. Xu, and Q. Cheng (2025) Multimodal large-language model empowering next-generation autonomous driving systems. Journal of Intelligent and Connected Vehicles 8 (2), p. 9210059–1. Cited by: §1. Z. Huang, Z. Sheng, and S. Chen (2025a) PE-rlhf: reinforcement learning with human feedback and physics knowledge for safe and trustworthy autonomous driving. Transportation Research Part C: Emerging Technologies 179, p. 105262. Cited by: §3.1. Z. Huang, Z. Sheng, C. Ma, and S. Chen (2024) Human as ai mentor: enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving. Communications in Transportation Research 4, p. 100127. Cited by: §2.1. Z. Huang, Z. Sheng, Y. Qu, J. You, and S. Chen (2025b) Vlm-rl: a unified vision language models and reinforcement learning framework for safe autonomous driving. Transportation Research Part C: Emerging Technologies 180, p. 105321. Cited by: §1, §2.1, §2.3, §3.1, 2nd item, §5.1.3. Z. Huang, Z. Sheng, Z. Wan, Y. Qu, Y. Luo, B. Wang, P. Li, Y. Chen, J. Chen, K. Long, et al. (2025c) Sky-drive: a distributed multiagent simulation platform for human-ai collaborative and socially aware future transportation. Journal of Intelligent and Connected Vehicles 8 (4), p. 9210070–1. Cited by: §2.2. Z. Huang, Z. Sheng, Z. Wan, Y. Qu, J. You, S. Jiang, and S. Chen (2026) DriveVLM-rl: neuroscience-inspired reinforcement learning with vision-language models for safe and deployable autonomous driving. arXiv preprint arXiv:2603.18315. Cited by: Appendix A, §1, §2.1, §2.3, §3.1, §4.1, 3rd item, §5.1.1, §5.1.1, §5.1.2, §5.1.3, §5.1.4, §5.1.5, 1. G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, et al. (2021) Openclip. Zenodo. Cited by: §5.1.2. X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan (2024) Bench2drive: towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. Advances in Neural Information Processing Systems 37, p. 819–844. Cited by: §1. B. Jiang, S. Chen, Q. Zhang, W. Liu, and X. Wang (2025a) Alphadrive: unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608. Cited by: §2.1. S. Jiang, Z. Huang, K. Qian, Z. Luo, T. Zhu, Y. Zhong, Y. Tang, M. Kong, Y. Wang, S. Jiao, et al. (2025b) A survey on vision-language-action models for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4524–4536. Cited by: §2.1. W. Jun and S. Lee (2025) A comparative study and optimization of camera-based bev segmentation for real-time autonomous driving. Sensors 25 (7), p. 2300. Cited by: §2.2, §4.2. S. Kakade and J. Langford (2002) Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, p. 267–274. Cited by: Proof 3. B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez (2021) Deep reinforcement learning for autonomous driving: a survey. IEEE transactions on intelligent transportation systems 23 (6), p. 4909–4926. Cited by: §1, §3.1. P. Koopman and M. Wagner (2017) Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intelligent Transportation Systems Magazine 9 (1), p. 90–96. Cited by: §4.5.3. A. Kumar, Z. Fu, D. Pathak, and J. Malik (2021) Rma: rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034. Cited by: §2.2, §6. D. A. Levin and Y. Peres (2017) Markov chains and mixing times. Vol. 107, American Mathematical Soc.. Cited by: Proof 3. D. Li and O. Okhrin (2024) A platform-agnostic deep reinforcement learning framework for effective sim2real transfer towards autonomous driving. Communications Engineering 3 (1), p. 147. Cited by: §1, §1, §2.2, §2.3. Y. Li, M. Tian, D. Zhu, J. Zhu, Z. Lin, Z. Xiong, and X. Zhao (2025) Drive-r1: bridging reasoning and planning in vlms for autonomous driving with reinforcement learning. arXiv preprint arXiv:2506.18234. Cited by: §2.1. Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai (2024) Bevformer: learning bird’s-eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (3), p. 2020–2036. Cited by: §1, §2.2, §4.2. H. Lin, Y. Zhang, W. Ding, J. Wu, and D. Zhao (2025) Model-based policy adaptation for closed-loop end-to-end autonomous driving. In Workshop on Foundation Models Meet Embodied Agents at CVPR 2025, Cited by: §2.3. J. Mei, Y. Ma, X. Yang, L. Wen, X. Cai, X. Li, D. Fu, B. Zhang, P. Cai, M. Dou, et al. (2024) Continuously learning, adapting, and improving: a dual-process approach to autonomous driving. In Proceedings of the 38th International Conference on Neural Information Processing Systems, p. 123261–123290. Cited by: §5.2.9. T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida (2018) Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. Cited by: Remark 1. J. Philion and S. Fidler (2020) Lift, splat, shoot: encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European conference on computer vision, p. 194–210. Cited by: §1, §2.2, §4.2. K. Qian, S. Jiang, Y. Zhong, Z. Luo, Z. Huang, T. Zhu, K. Jiang, M. Yang, Z. Fu, J. Miao, et al. (2025) Agentthink: a unified framework for tool-augmented chain-of-thought reasoning in vision-language models for autonomous driving. arXiv preprint arXiv:2505.15298 1 (2), p. 3. Cited by: §2.1. Y. Qu, Z. Sheng, Z. Huang, J. Chen, Y. Luo, T. Wang, Y. Feng, S. Labi, and S. Chen (2026) Found-rl: foundation model-enhanced reinforcement learning for autonomous driving. arXiv preprint arXiv:2602.10458. Cited by: §1, §2.1. Y. Qu, Z. Xu, Z. Huang, Z. Sheng, S. Chen, and T. Chen (2025) Metassc: enhancing 3d semantic scene completion for autonomous driving through meta-learning and long-sequence modeling. Communications in Transportation Research 5, p. 100184. Cited by: §1. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §2.1, Remark 2. R. Rajamani (2006) Vehicle dynamics and control. Springer. Cited by: §4.3.2. K. Renz, L. Chen, E. Arani, and O. Sinavski (2025) Simlingo: vision-only closed-loop autonomous driving with language-action alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 11993–12003. Cited by: §2.1. E. Salvato, G. Fenu, E. Medvet, and F. A. Pellegrino (2021) Crossing the reality gap: a survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access 9, p. 153171–153187. Cited by: §1, §1, §2.2, §4.4.3. H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu (2023) Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, p. 726–737. Cited by: §1, §2.1. Z. Sheng, Z. Huang, and S. Chen (2024a) Kinematics-aware multigraph attention network with residual learning for heterogeneous trajectory prediction. Journal of Intelligent and Connected Vehicles 7 (2), p. 138–150. Cited by: §2.3. Z. Sheng, Z. Huang, and S. Chen (2024b) Traffic expertise meets residual rl: knowledge-informed model-based residual reinforcement learning for cav trajectory control. Communications in Transportation Research 4, p. 100142. Cited by: §2.1. R. S. Sutton, A. G. Barto, et al. (1998) Reinforcement learning: an introduction. Vol. 1, MIT press Cambridge. Cited by: §3.1. T. Tahmina, M. Fuchs, and C. Shi (2025) Use of virtual reality for automated driving simulation. Journal of Intelligent and Connected Vehicles 8 (1), p. 1–8. Cited by: §2.2. W. Tang, J. You, J. Liu, Z. Wang, R. Gan, Z. Huang, F. Wei, and B. Ran (2026) HERMES: a holistic end-to-end risk-aware multimodal embodied system with vision-language models for long-tail autonomous driving. arXiv preprint arXiv:2602.00993. Cited by: §1. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), p. 23–30. Cited by: §1, §1, §2.2. S. H. Tóth, Z. J. Viharos, Á. Bárdos, and Z. Szalay (2024) Sim-to-real application of reinforcement learning agents for autonomous, real vehicle drifting. Vehicles 6 (2), p. 781–798. Cited by: §2.3. K. L. Voogd, J. P. Allamaa, J. Alonso-Mora, and T. D. Son (2023) Reinforcement learning from simulation to real world autonomous driving using digital twin. IFAC-PapersOnLine 56 (2), p. 1510–1515. Cited by: §1, §2.2, §2.3. Y. Wang, W. Luo, J. Bai, Y. Cao, T. Che, K. Chen, Y. Chen, J. Diamond, Y. Ding, W. Ding, et al. (2025) Alpamayo-r1: bridging reasoning and action prediction for generalizable autonomous driving in the long tail. arXiv preprint arXiv:2511.00088. Cited by: §1, §2.1. D. Wasif, T. J. Moore, C. K. Reddy, and J. Cho (2025) DriveMind: a dual-vlm based reinforcement learning framework for autonomous driving. arXiv preprint arXiv:2506.00819. Cited by: §1, §2.1, §2.3, §3.1. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo (2021) SegFormer: simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems 34, p. 12077–12090. Cited by: §2.2, §4.2.1, Remark 1, Remark 3. R. Xu, H. Lin, W. Jeon, H. Feng, Y. Zou, L. Sun, J. Gorman, E. Tolstaya, S. Tang, B. White, et al. (2025) Wod-e2e: waymo open dataset for end-to-end driving in challenging long-tail scenarios. arXiv preprint arXiv:2510.26125. Cited by: §1. J. Yang, K. Chitta, S. Gao, L. Chen, Y. Shao, X. Jia, H. Li, A. Geiger, X. Yue, and L. Chen (2025) Resim: reliable world simulation for autonomous driving. arXiv preprint arXiv:2506.09981. Cited by: §2.3. J. You, X. Jia, Z. Zhang, Y. Zhu, and J. Yan (2024) Bench2drive-r: turning real world data into reactive closed-loop autonomous driving benchmark by generative model. arXiv preprint arXiv:2412.09647. Cited by: §2.3. C. Zhang, B. Wei, Y. Liu, and S. Labi (2026) World model-based long-tail and scenario-specific generation for autonomous driving. Journal of Intelligent and Connected Vehicles. Cited by: §2.3. J. Zhang, C. Xu, and B. Li (2024) Chatscene: knowledge-enabled safety-critical scenario generation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15459–15469. Cited by: 1st item. Z. Zhang (2000) A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence 22 (11), p. 1330–1334. Cited by: §4.2.1, Table 5. W. Zhao, J. P. Queralta, and T. Westerlund (2020) Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), p. 737–744. Cited by: §2.2, §4.4.3. Z. Zhou, T. Cai, S. Z. Zhao, Y. Zhang, Z. Huang, B. Zhou, and J. Ma (2025) Autovla: a vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. arXiv preprint arXiv:2506.13757. Cited by: §2.1. J. Zhu, T. Park, P. Isola, and A. A. Efros (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, p. 2223–2232. Cited by: §1, §1, §2.2, §5.2.3.