Paper deep dive
D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models
Heesang Han, A. Lynn Abbott, Abhijit Sarkar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/23/2026, 2:23:29 AM
Summary
The paper introduces D3VL, a novel Multimodal Large Language Model (MLLM) framework designed to understand driving scenes by integrating 2D video and 3D time-series data (LiDAR/stereo). D3VL addresses the challenges of data sparsity and lack of grid structure in LiDAR data by projecting 3D points into depth images, which are then fused with camera images in a unified architecture. The framework demonstrates an 11% improvement over baselines on the KITTI QA dataset and introduces the WaymoQA dataset extension to assess performance under diverse driving conditions.
Entities (7)
Relation Signals (6)
D3VL → improvesaccuracyon → KITTI QA
confidence 95% · D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods
D3VL → introduces → WaymoQA
confidence 95% · This paper further introduces the Waymo QA dataset extension
D3VL → uses → LiDAR
confidence 95% · this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras.
WaymoQA → isextensionof → Waymo Open Dataset
confidence 90% · WaymoQA, a Video Question-Answering (VideoQA) extension for the Waymo Open Dataset (WOD)
D3VL → uses → Stereo Camera
confidence 90% · this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras.
D3VL → isbasedon → Qwen-2.5-VL
confidence 85% · D3VL-Qwen ... Qwen-2.5-VL ... backbone
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in Multimodal Large Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA dataset extension, which assesses models' capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2607.19528v1
- Canonical: https://arxiv.org/abs/2607.19528v1
Trouble viewing inline? Open PDF directly →
Full Text
32,478 characters extracted from source content.
Expand or collapse full text
D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models Heesang Han1, 3, A. Lynn Abbott1,3, Abhijit Sarkar1,2,3 1Bradley Department of Electrical and Computer Engineering, Virginia Tech, USA 2Virginia Tech Transportation Institute, USA 3Sanghani Center for Artificial Intelligence and Data Analytics, USA heesang, abbott, asarkar1@vt.edu Abstract Recent advances in Multimodal Large Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA dataset extension, which assesses models’ capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: https://automotivesafety-lvlm.github.io. I Introduction Figure 1: Examples in processing 2D and 3D time-series data. D3VL framework can effectively utilize 3D data when it is difficult to find smaller cyclists from 2D image, compared to baseline Qwen 2.5 model. More examples are available in our website. Recent advances in Multimodal Large Language Models (MLLM) have revolutionized Intelligent Transportation Systems (ITS) [31], excelling in complex reasoning tasks essential for ITS operations. Further, researchers have shown promising results in designing MLLMs for autonomous driving [14, 29, 42]. MLLMs have partially addressed the long-tailed object detection problem with their vast pretrained knowledge [22, 11], showing potential to help AV systems detect rare objects such as emergency vehicles and traffic cones. MLLMs also excel in adapting to different domains, unlike traditional systems, which falter when deployed in different countries [17]. However, it is unknown whether current MLLMs could answer safety related questions related to 3-Dimensional (3D) data scene perception. MLLMs’ cannot perceive 3D temporal data direct from LiDAR and stereo cameras, hence it cannot infer the dynamics of the scene from 3D data. One main reason for this limitation is practical: constructing and annotating large traffic datasets is difficult, and processing 3D and temporal data generally requires specialized models. In addition, MLLMs require extensive testing with multiple tools, such as benchmarking or visualizers, before being deployed to critical applications such as Autonomous Vehicles (AV). (a) (b) (c) Figure 2: Previous MLLM systems for AV applications have used either a) a single LiDAR frame with additional learnable LiDAR processing units M ([35, 26, 41]), or b) multiple image frames ([14, 33, 23, 29]). The P module denotes LiDAR-to-camera projection, ϵε represents a vision encoder, and LMLM refers to Language Model. Proposed D3VL method incorporates both temporal LiDAR data and camera data, and does not require an additional learnable module for LiDAR processing. Prepending trainable modules to pretrained LLM is a popular way to add a new modality. Recent image-centric MLLMs [21, 2, 19] integrate vision encoders and vision-language aligners with pretrained LLMs. MLLMs with LiDAR modalities [35, 13] were built by adding pointcloud encoder and pointcloud-text aligner. MLLM’s video understanding capabilities could be enhanced by adding video processing modules to image-centric MLLMs. [38, 5, 39, 40]. These methods benefit from efficient finetuning thanks to pretrained LLMs, enabling smaller models to perform comparably to large, task-agnostic MLLMs. However, such trainable modules often require complex finetuning strategies and increase model size. Multimodal Time-series data processing is also essential for development and understanding of autonomous driving; it requires the tracking and motion prediction of vehicles, pedestrians, and other moving objects near the ego vehicle, and it cannot be achieved without temporal data processing. Jain et al. [15] has shown that providing tracking information to MLLMs improves performance in question-answering in the context of AV. Furthermore, numerous studies have shown that sensor fusion of 2D Cameras and 3D LiDAR improves the performance of AV perception models [20] in various tasks such as 3D object detection; yet, MLLMs’ capability in processing temporal LiDAR and camera data has not been explored. To enable End-to-End (E2E) MLLMs in autonomous driving, they must be compact and simple for onboard units while ensuring top safety performance. This paper investigates the ability of MLLMs in processing temporal LiDAR and camera data and introduces D3VL - Driving Scene Understanding with 3D Time-Series Data and Video with Language Model - a novel framework for existing MLLMs to perform better in understanding 2D and 3D time-series driving scenes. Experiments show that the proposed D3VL framework improves the baseline MLLMs’ accuracy by 11% when tested with the KITTI QA dataset [8, 15]. Key contributions include: • This paper proposes D3VL, a novel framework to enhance the performance of smaller MLLMs in processing LiDAR and camera time series data through fine-tuning. • This paper introduces WaymoQA, a Video Question-Answering (VideoQA) extension for the Waymo Open Dataset (WOD) [30], focusing on scene understanding and AV safety. I Related Work I-A Foundational Language Models and MLLMs Recent advances in MLLMs began with contrastive learning on text-image pairs [27, 18, 1, 37], learning the relationship between them. Inspired by early LLMs [7], MaskVLM [16] added masked vision-language learning to the contrastive learning. Recent MLLMs [19, 2, 21] use a pretrained LLM backbone, a vision encoder, and a vision-language aligner; then they are finetuned in multiple stages. Instead of an aligner, LLaMA 3.2-VL [9] alternates self-attention layers from the LLM backbone with cross-modal attention layers. Newer models [24] adopt Mixture-of-Experts with early fusion to pretrain on large unlabeled data. State-of-the-art closed-source MLLMs [25, 6] often match or surpass open-source models across many tasks. Figure 3: Overall architecture for D3VL. P is LiDAR-to-Camera projection as in algorithm 1. ε is a transformer vision encoder in an MLLM, projector is vision-language projector, T is text tokenizer, and LLM is foundational LLM decoder, such as Qwen-2.5. I-B MLLMs for Pointcloud and Video Researchers have been enhancing foundational MLLMs to incorporate additional modalities. Inspired by CLIP [27], CLIP2 [36] uses contrastive text-image-point pretraining. Chat-3D [32] combines a point cloud encoder, projector, relation modules, and an LLM to understand 3D relations in a scene. PointLLM [34] adds a 3D encoder and projector to support LiDAR, while LiDAR-LLM [35] uses a bird’s-eye-view-aware transformer as a projector instead. LSceneLLM [41] applies adaptive scaling for larger scenes, and SpatialBot [4] is built on prompt engineering with depth images. Although these methods integrate LiDAR with foundational LLMs, they require specialized modules for 3D input that increase model size and complexity. Also, they mostly operate on single LiDAR frames, leaving temporal sequence integration largely unexplored. Recent studies [39, 5, 38, 40] highlight the potential of integrating video into foundational LLMs. These models concatenate tokens from video frames and feed them to pretrained LLMs, which are then finetuned with instruction or reinforcement tuning so that the model can interpret tokens from a sequence of frames. I-C MLLMs for Autonomous Driving Recent advances in MLLMs have unlocked the potential of E2E MLLMs in AV. EMMA and OpenEMMA [14, 33] explore the potential of Chain-of-Thought prompting in E2E MLLMs for autonomous driving. AutoVLA [42] explores adaptive reasoning and reinforcement tuning in MLLM-based autonomous driving. These models only take camera images, historical ego poses, and high level text instructions; the training process is self-supervised. Question-Answering (QA) datasets and methods have been developed to facilitate research on E2E autonomous driving models. DriveLM-NuScenes [29] provides high-quality QA pairs on perception, prediction, and planning for AV safety. LingoQA [23] is targeted for training E2E autonomous driving models. ScVLM [28] presents a MLLM-based approach to answer QA in safety-critical events. However, all these datasets lack 3D modality. NuScenesQA [26] contains a large number of QA pairs with camera and LiDAR data, but its questions mainly target spatial relations. In addition, the dataset lacks temporal information, as all scenearios are 0.5 seconds long. The KITTI QA dataset [8, 15] features high-quality QA pairs on AV scene understanding with 3D object tracking but shares the limitations of the KITTI dataset[8]: the dataset is small and was collected in ideal condition - clear weather, daytime, and ego vehicle moving slow. Table I summarizes these findings. TABLE I: QA Datasets for Autonomous Driving Dataset WaymoQA KITTI QA NuScenesQA LingoQA (ours) [30] [15] [26] [23] ‖Train‖\|Train\| 768 21 28K 28K ‖Test‖\|Test\| 150 29 6K 100 ‖VQA‖\|VQA\| 60K 2K 460K 421K Scene length 20s 3s-1min 0.5s 4s Modality 2D & 3D 2D 2D & 3D 2D ∥\|Train∥\| and ∥\|Test∥\| are the number of scenes in each dataset. ∥\|VQA∥\| is the total number of Video-Question-Answer triplets in the dataset. I Methods I-A D3VL Framework All sensors are assumed to be temporally synchronized. 3D data is obtained from a LiDAR sensor or a stereo camera. The proposed framework first produces a depth image from the 3D data. Given a LiDAR point cloud X=x1,…,xnX=x_1,…,x_n where xi=[xyz 1]Tx_i=[x\ y\ z\ 1]^T, LiDAR-to-Camera transformation matrix E∈ℝ4×4E ^4× 4, and a camera matrix P∈ℝ3×4P ^3× 4, a depth image X′∈ℝW×HX ^W× H can be obtained using LiDAR-to-Camera projections [12] as in algorithm 1. First, all pixels in a depth image X′X are initialized to dmaxd_max, the maximum effective range of a sensor used in a dataset. For every point xix_i in LiDAR point cloud X, E is applied to transform the LiDAR coordinates into the camera coordinates, and then P is applied to project onto the camera plane. If the projected coordinate (x,y)(x,y) is within the camera frame, a pixel X’(x, y) in depth image is encoded with the depth value ‖xi‖\|x_i\|. Algorithm 1 LiDAR to Camera Projection 1:X′(i,j)←dmax∀(i,j)∈[0,W]×[0,H]X (i,j)← d_max\ ∀\ (i,j)∈[0,W]×[0,H] 2:for k=1,…Nk=1,… N do 3: p=PEXkp=PEX_k // p=[x′,y′,1]∈R3p=[x ,y ,1]∈ R^3 4: x←p1/p3x← p_1/p_3, y←p2/p3y← p_2/p_3 5: if (x,y)∈[0,W]×[0,H](x,y)∈[0,W]×[0,H] then 6: X′(x,y)=min(‖Xk‖2,dmax)X (x,y)=min(\|X_k\|_2,d_max) 7: end if 8:end for Figure 4: Example Video-Question-Answer triplets from Waymo QA dataset extension, showcasing diverse driving conditions and temporal 3D-aware questions. More examples are available in our website. Given a stereo image pair, the framework uses BridgeDepth [10] for stereo depth estimation to generate a depth map. Then, a linear colormap is applied to improve visibility. At a given time t, the framework receives a sequence of t+1t+1 camera images C0,…,CtC_0,…,C_t and a corresponding sequence of depth images D0,…,DtD_0,…,D_t. Each camera image CiC_i and depth image DiD_i are tokenized separately to CTiCT_i and DTiDT_i using an image encoder ε in MLLM. Each token from the 2D image sequence and 3D depth sequence is then projected into 1-D language space with a vision-language projector in MLLM, which is often a multi-layer perceptrons. Text instructions and questions are also converted to tokens TtextT_text at this stage. A complete sequence of input tokens ITIT IT=Concat(CT0,…,CTt,DT0,…DTt,Ttext)IT=Concat(CT_0,…,CT_t,\ DT_0,… DT_t,\ T_text) is then created and fed into the LLM to retrieve the final output, which is the answer to the question in this case. D3VL framework includes a finetuning process to learn rare, AV-specific scenes. During the process, the vision encoder ε and vision-language projector learn to better encode depth images, while the backbone LLM learns to better decode depth images and rare scenes. Leveraging the pretrained knowledge of foundational LLMs, D3VL only requires at most 3 training epochs on the KITTI QA dataset; no performance improvement has been observed after the 3 epochs. Key training hyperparameters are: learning rate 1×10−41× 10^-4, temperature 1.01.0, and top-p 1.01.0. I-B Dataset To the best of our knowledge, this is the first study to explore MLLMs’ ability to process timeseries LiDAR and camera data using QA datasets for AV, and no existing QA dataset satisfies all requirements of this study. As shown in Table I, some datasets [23, 29] lack 3D modalities; another [26, 3] does not contain meaningful temporal contexts due to short scene length; the other [8, 15] only contains driving scenes captured during clear weather in the daytime. Therefore, this paper introduces WaymoQA, a QA extension for the Waymo Open Dataset (WOD) [30], which includes 60K Video-Question-Answer triplets on autonomous driving. WOD features 768 training scenes and 150 testing scenes, each 20 seconds long, showcasing various driving conditions, such as adverse weather, nighttime, rush hours, and construction. Each scene is annotated with 65 QA pairs to assess AV safety in four categories: Environment, Traffic Signals and Signs, Moving Objects, and Planning. These questions are extended from Jain et al.’s work [15], adding various QA pairs that require 3D perception and temporal perception, including questions that require object tracking or speed estimation. The dataset also includes rural scenes with higher vehicle speeds, which are absent from KITTI [8] and NuScenes-based [3] datasets. All QA pairs use multiple-choice questions to reduce output token generation, which is more expensive than input token processing. Example questions from each area are listed below and in Figure 4: • Environment - Are there any potholes on the road? - Has ego vehicle passed through a pedestrian crossing? • Traffic Signals and Signs - What is the last observed speed limit? - Which light was on when the ego vehicle passed the traffic light? • Moving Objects - Are there any pedestrians on a road, excluding crosswalk? - Are there any moving vehicles ahead within two car lengths of ego vehicle? • Safety & Planning - Is it safe for the ego vehicle to change lanes? - Are there any vehicles accelerating on the same road as ego vehicle? Human annotation of a dataset is costly. Therefore, WaymoQA is annotated in a semi-supervised fashion, inspired by NuScenesQA[26]. Human experts annotate testing scenes, and finetuned MLLM annotates the rest. Google Gemini 2.5 Flash [6] underwent supervised and Low-Rank Adapter (LoRA) finetuning with more than 10K of 2D and 3D Video-QA triplets from human-annotated test scenes over 1 epoch using a Rank 8 LoRA. Annotations for training scenes are generated using the finetuned model with Temperature of 0.5 and Top-p of 1. TABLE I: D3VL and baseline MLLMs benchmark on KITTI QA dataset Methods 2D (Camera) 3D (Stereo) + 2D (Camera) RI VRU HF V2VI OEF Overall RI VRU HF V2VI OEF Overall Baseline methods: Qwen-2.5-VL (3B) [2] 62.17 55.56 75.00 74.81 81.48 68.70 37.57 51.85 52.31 61.48 67.72 50.93 Qwen-2.5-VL (7B) [2] 68.25 64.20 51.39 77.78 91.53 69.54 70.63 42.59 71.30 70.37 70.9 66.57 Qwen-2.5-VL (32B) [2] 70.63 42.59 71.30 70.37 70.90 66.57 68.25 48.15 74.07 58.52 63.49 64.35 VideoLLaMA3 (2B) [38] 38.10 38.89 42.59 40.74 47.62 41.11 42.33 36.42 40.74 37.04 49.74 41.76 Gemini 2.5 (Flash) [6] - - - - - 71.94 - - - - - 74.26 Existing State-of-the-Art methods: Jain et al. [15] 81.56 80.43 85.60 77.78 87.11 81.75 76.63 71.53 89.32 81.25 80.65 79.69 D3VL (ours): D3VL-Qwen 85.19 64.20 89.81 83.70 96.83 84.81 86.77 64.81 89.35 83.70 97.35 85.46 Blue highlights overall best performance, and Red highlights best performance among baseline methods. RI, VRU, HFDB, V2VI, and OEF denotes Road Infrastructure, Vulnerable Road User, Human Factors & Driving Behavior, Vehicle-to-Vehicle Interaction, anb Other Environment Factors, respectively IV Experiments IV-A Setup We extensively evaluate popular MLLMs on processing time-series 3D and 2D data from KITTI QA datasets [8, 15] and Waymo QA datasets [30]. As baselines, we test Qwen 2.5 VL (3B, 7B, 32B) [2], VideoLLaMA3 (2B) [38], and Gemini 2.5 (Flash) [6]. We also evaluate and compare our D3VL framework with the baselines and existing state-of-the-art methods. Each baseline method is evaluated zero-shot with (1) 2D time-series input only and (2) 2D and 3D time-series inputs from the testing dataset. The D3VL framework with Qwen-2.5-VL-3B backbone is finetuned on 2D and 3D time-series inputs from training dataset and evaluated on testing dataset, which remains unseen during tuning. To assess the impact of 3D data, another D3VL variant is finetuned and tested without 3D inputs. All metrics are Top-1 accuracy. D3VL is finetuned for 1-3 epochs, as more epochs do not significantly improve performance. IV-B Results Table I summarizes the results on KITTI QA dataset; results on the Waymo QA dataset are available on our supplemental website. Most baselines perform better without 3D input, likely because their pretraining data contain very few 3D samples. Even for larger models, performance on 2D input alone remains substantially better than on 2D and 3D input. The D3VL framework outperforms all baselines across all five areas in KITTI QA, achieving an 11% gain in overall accuracy. It also surpasses Jain et al.[15] in overall performance and in all areas except VRU. Compared to D3VL finetuned only on 2D inputs, fine-tuning on both 2D and 3D inputs notably improves RI question processing, with particularly large gains for the following questions: • RI: Are there any visible road dividers or medians? +7% • RI: Are there proper road signs indicating turns and intersections? +11% • VRU: Are there any cyclists on the road? +7% • VRU: Are there any pedestrians present on the road? +7% On the other hand, D3VL finetuned on 2D inputs alone performs noticeably better in processing one question: • VRU: Is there a designated pedestrian crossing? +7% V Discussions In general, D3VL significantly improves MLLMs’ capability to process 2D and 3D time series data by 11% to 27%, across different areas in KITTI QA dataset, compared to baselines with both 2D and 3D inputs. Table I shows that adding 3D input to the baseline model without D3VL hinders their ability to process driving scenes and produce correct answers. It is noticeable that Jain et al.’s work [15] with and without 3D data performs substantially better in answering VRU questions than any other baseline methods and D3VL framework. VRU questions assess models’ ability to process information regarding pedestrians and cyclists, which are relatively common concepts. Based on the GPT-4 model and trained on a large proprietary dataset, it effectively processes common concepts using only images. Notably, it performs better without 3D data, indicating its heavy reliance on camera images when processing VRU questions. Its performance remains unknown in poor weather or at night, which is not included in the KITTI QA dataset. Compared to D3VL with 2D input alone, D3VL with 3D input shows superior performance in perceiving cyclists, pedestrians, and road dividers, all of which contain rich 3D features that differentiate them from other types of objects that exist in driving scenes. Also, as shown in figure 1, all of those objects are usually very small in 2D images. The only question D3VL finetuned without 3D input includes perceiving pedestrian crossings, which is hard to differentiate from regular roads without pedestrian crossings. The D3VL framework effectively helps MLLMs learn prominent 3D features. VI Conclusion This paper is the first to investigate the capability of MLLMs in processing Camera (2D) and LiDAR/Stereo (3D) time-series data in the context of autonomous driving. While baseline MLLMs achieve 51-74% accuracy in processing such data in the KITTI QA dataset, the proposed D3VL framework improves the ability to handle 2D and 3D time-series data, reaching an accuracy of 85% and surpassing all existing methods. Especially, the D3VL framework helps MLLMs learn prominent 3D features, enhancing MLLMs’ capability to perceive information regarding pedestrians and cyclists in driving scenes. Additionally, the paper further identifies limitations in existing AV QA datasets and presents Waymo QA dataset extensions that address those problems. Acknowledgment The authors sincerely thank Joe Bekiranov and Jin Woo Baik for their help with the human annotation process of the WaymoQA test dataset. References [1] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a Visual Language Model for Few-Shot Learning. Advances in Neural Information Processing Systems (NeurIPS) 35, p. 23716–23736. Cited by: §I-A. [2] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2025) Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923. Cited by: §I, §I-A, TABLE I, TABLE I, TABLE I, §IV-A. [3] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom (2020) nuScenes: A Multimodal Dataset for Autonomous Driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11618–11628. Cited by: §I-B, §I-B. [4] W. Cai, I. Ponomarenko, J. Yuan, X. Li, W. Yang, H. Dong, and B. Zhao (2025) SpatialBot: Precise Spatial Understanding with Vision Language Models. In IEEE International Conference on Robotics and Automation (ICRA), p. 9490–9498. External Links: Document Cited by: §I-B. [5] Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao, and L. Bing (2024) VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. arXiv preprint arXiv:2406.07476. Cited by: §I, §I-B. [6] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, et al. (2025) Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261. Cited by: §I-A, §I-B, TABLE I, §IV-A. [7] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Vol. 1, p. 4171–4186. Cited by: §I-A. [8] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun (2013) Vision meets Robotics: The KITTI Dataset. International Journal of Robotics Research (IJRR). Cited by: §I, §I-C, §I-B, §I-B, §IV-A. [9] A. Grattafiori et al. (2024) The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Cited by: §I-A. [10] T. Guan, J. Guo, C. Wang, and Y. Liu (2025) BridgeDepth: Bridging Monocular and Stereo Reasoning with Latent Alignment. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 27681–27691. Cited by: §I-A. [11] G. Han and S. Lim (2024) Few-Shot Object Detection with Foundation Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 28608–28618. Cited by: §I. [12] R. Hartley and A. Zisserman (2004) Multiple View Geometry in Computer Vision. 2nd Edition edition, Cambridge University Press, Cambridge. External Links: Link Cited by: §I-A. [13] H. Huang, Y. Chen, Z. Wang, R. Huang, R. Xu, T. Wang, L. Liu, X. Cheng, Y. Zhao, J. Pang, et al. (2024) Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. Advances in Neural Information Processing Systems (NeurIPS) 37, p. 113991–114017. Cited by: §I. [14] J. Hwang, R. Xu, H. Lin, W. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp, Y. Zhou, J. Guo, D. Anguelov, and M. Tan (2025) EMMA: End-to-End Multimodal Model for Autonomous Driving. Transactions on Machine Learning Research (TMLR). External Links: ISSN 2835-8856, Link Cited by: Figure 2, Figure 2, §I, §I-C. [15] S. Jain, S. Thapa, K. Chen, A. L. Abbott, and A. Sarkar (2024) Semantic Understanding of Traffic Scenes with Large Vision Language Models. In IEEE Intelligent Vehicles Symposium (IV), p. 1580–1587. External Links: Document Cited by: §I, §I, §I-C, TABLE I, §I-B, §I-B, TABLE I, §IV-A, §IV-B, §V. [16] G. Kwon, Z. Cai, A. Ravichandran, E. Bas, R. Bhotika, and S. Soatto (2023) Masked Vision and Language Modeling for Multi-modal Representation Learning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §I-A. [17] B. Li, Y. Wang, J. Mao, B. Ivanovic, S. Veer, K. Leung, and M. Pavone (2024) Driving Everywhere with Large Language Model Policy Adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14948–14957. Cited by: §I. [18] J. Li, D. Li, C. Xiong, and S. Hoi (2022) BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In International Conference on Machine Learning (ICML), p. 12888–12900. Cited by: §I-A. [19] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual Instruction Tuning. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 34892–34916. Cited by: §I, §I-A. [20] Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. Rus, and S. Han (2023) BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation. In IEEE International Conference on Robotics and Automation (ICRA), Cited by: §I. [21] H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, Y. Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan (2024) DeepSeek-VL: Towards Real-World Vision-Language Understanding. arxiv preprint arXiv:2403.05525. Cited by: §I, §I-A. [22] A. Madan, N. Peri, S. Kong, and D. Ramanan (2024) Revisiting Few-Shot Object Detection with Vision-Language Models. Advances in Neural Information Processing Systems (NeurIPS) 37, p. 19547–19560. Cited by: §I. [23] A. Marcu, L. Chen, J. Hünermann, A. Karnsund, B. Hanotte, P. Chidananda, S. Nair, V. Badrinarayanan, A. Kendall, J. Shotton, et al. (2024) LingoQA: Visual Question Answering for Autonomous Driving. In European Conference on Computer Vision (ECCV), p. 252–269. Cited by: Figure 2, Figure 2, §I-C, TABLE I, §I-B. [24] MetaAI et al. (2024) Introducing LLaMA 4: Advancing Multimodal Intelligence. External Links: Link Cited by: §I-A. [25] OpenAI et al. (2024) GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. Cited by: §I-A. [26] T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y. Jiang (2024) NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario. In AAAI Conference on Artificial Intelligence (AAAI), Vol. 38, p. 4542–4550. Cited by: Figure 2, Figure 2, §I-C, TABLE I, §I-B, §I-B. [27] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), p. 8748–8763. Cited by: §I-A, §I-B. [28] L. Shi, B. Jiang, T. Zeng, and F. Guo (2025) ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding. In Proceedings of the Winter Conference on Applications of Computer Vision, p. 1061–1071. Cited by: §I-C. [29] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li (2024) DriveLM: Driving with Graph Visual Question Answering. In European Conference on Computer Vision (ECCV), p. 256–274. Cited by: Figure 2, Figure 2, §I, §I-C, §I-B. [30] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. (2020) Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2446–2454. Cited by: 2nd item, TABLE I, §I-B, §IV-A. [31] S. Wandelt, C. Zheng, S. Wang, Y. Liu, and X. Sun (2024) Large Language Models for Intelligent Transportation: A Review of the State of the Art and Challenges. Applied Sciences 14 (17), p. 7455. Cited by: §I. [32] Z. Wang, H. Huang, Y. Zhao, Z. Zhang, T. Jin, and Z. Zhao (2025) Data-Efficiently Learn Large Language Model for Universal 3D Scene Perception. In Findings of the Association for Computational Linguistics, p. 313–333. Cited by: §I-B. [33] S. Xing, C. Qian, Y. Wang, H. Hua, K. Tian, Y. Zhou, and Z. Tu (2025) OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving. In IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), p. 911–919. Cited by: Figure 2, Figure 2, §I-C. [34] R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang, and D. Lin (2024) PointLLM: Empowering Large Language Models to Understand Point Clouds. In European Conference on Computer Vision (ECCV), p. 131–147. Cited by: §I-B. [35] S. Yang, J. Liu, R. Zhang, M. Pan, Z. Guo, X. Li, Z. Chen, P. Gao, H. Li, Y. Guo, et al. (2025) LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding. In AAAI Conference on Artificial Intelligence (AAAI), Vol. 39, p. 9247–9255. Cited by: Figure 2, Figure 2, §I, §I-B. [36] Y. Zeng, C. Jiang, J. Mao, J. Han, C. Ye, Q. Huang, D. Yeung, Z. Yang, X. Liang, and H. Xu (2023) CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15244–15253. Cited by: §I-B. [37] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid Loss for Language Image Pre-Training. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 11975–11986. Cited by: §I-A. [38] B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Li, et al. (2025) VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv preprint arXiv:2501.13106. Cited by: §I, §I-B, TABLE I, §IV-A. [39] H. Zhang, X. Li, and L. Bing (2023) Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, p. 543–553. External Links: Document Cited by: §I, §I-B. [40] Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li (2025) LLaVA-Video: Video Instruction Tuning With Synthetic Data. Transactions on Machine Learning Research (TMLR). External Links: Link Cited by: §I, §I-B. [41] H. Zhi, P. Chen, J. Li, S. Ma, X. Sun, T. Xiang, Y. Lei, M. Tan, and C. Gan (2025) LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 3761–3771. Cited by: Figure 2, Figure 2, §I-B. [42] Z. Zhou, T. Cai, S. Z. Zhao, Y. Zhang, Z. Huang, B. Zhou, and J. Ma (2025) AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §I, §I-C.