Paper deep dive
Bi-Level Routing and Sparse Spatial Attention based Multi-View BEV 3D Object Detection for Autonomous Driving
Jing Zhang, Jiaqi Liu, Zibo Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 9/21/2026, 4:58:19 AM
Summary
The paper proposes Sparse-BEVNet, an efficient multi-view BEV 3D object detection framework for autonomous driving that addresses computational complexity and feature extraction challenges. It integrates Bi-Level Routing Attention (BRA) in the backbone to reduce computational burden, Cascaded Group Attention (CGA) for multi-scale feature fusion, and Sparse Spatial Cross-Attention (SSCA) to replace dense view projection. Experiments on the nuScenes dataset show improvements in mAP (45.2%) and NDS (54.5%) over the baseline.
Entities (11)
Relation Signals (8)
Zibo Wang → authored → Sparse-BEVNet
confidence 100% · 3rdZibo Wang ... proposes an improved BEV 3D object detection algorithm Sparse-BEVNet
Jing Zhang → authored → Sparse-BEVNet
confidence 100% · 1st Jing Zhang* ... proposes an improved BEV 3D object detection algorithm Sparse-BEVNet
Jiaqi Liu → authored → Sparse-BEVNet
confidence 100% · 2ndJiaqi Liu ... proposes an improved BEV 3D object detection algorithm Sparse-BEVNet
Sparse-BEVNet → evaluatedon → nuScenes
confidence 100% · Experimental results on the public nuScenes dataset demonstrate
Sparse-BEVNet → usesmodule → Bi-Level Routing Attention
confidence 95% · Bi-Level Routing Attention (BRA) mechanism is introduced into the image feature extraction network
Sparse-BEVNet → usesmodule → Cascaded Group Attention
confidence 95% · Cascaded Group Attention (CGA) is employed in the feature fusion module
Sparse-BEVNet → usesmodule → Sparse Spatial Cross-Attention
confidence 95% · Sparse Spatial Cross-Attention mechanism is adopted to replace the conventional dense view projection pipeline
Sparse-BEVNet → outperforms → BEVFormer
confidence 90% · surpassing the BEVFormer-S baseline by 3.6% and 2.8%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Bird's Eye View (BEV)-based multi-view 3D object detection suffers from challenges of computational complexity, multi-scale feature extraction, and efficiency of dense 2D-to-BEV view transformation. To address these problems, this paper proposes an improved BEV 3D object detection algorithm Sparse-BEVNet. Firstly, a Bi-Level Routing Attention (BRA) mechanism is introduced into the image feature extraction network to reduce the computational burden of the backbone. Second, Cascaded Group Attention (CGA) is employed in the feature fusion module, which enhances deep interaction across features of different hierarchical levels without introducing additional computational overhead. Furthermore, a Sparse Spatial Cross-Attention mechanism is adopted to replace the conventional dense view projection pipeline. Experimental results on the public nuScenes dataset demonstrate that the proposed method achieves a mean Average Precision (mAP) of 45.2% and a nuScenes Detection Score (NDS) of 54.5%, corresponding to 3.6% and 2.8% improvements relative to the baseline model, respectively.
Tags
Links
- Source: https://arxiv.org/abs/2609.14185v1
- Canonical: https://arxiv.org/abs/2609.14185v1
Trouble viewing inline? Open PDF directly →
Full Text
23,254 characters extracted from source content.
Expand or collapse full text
Bi-Level Routing and Sparse Spatial Attention based Multi-View BEV 3D Object Detection for Autonomous Driving 1st Jing Zhang* Affiliation: Stuart Weitzman School of Design University of Pennsylvania Philadelphia, USA jing321@alumni.upenn.edu *Corresponding author 2ndJiaqi Liu Affiliation: Courant Institute of Mathematical Science New York University New York, USA jl8456@nyu.edu 3rdZibo Wang Affiliation: Courant Institute of Mathematical Science New York University New York, USA zibo.wang@nyu.edu Abstract Bird’s Eye View (BEV)-based multi-view 3D object detection suffer from challenges of computational complexity , multi-scale feature extraction, and efficiency of dense 2D-to-BEV view transformation. To address these problems, this paper proposes an improved BEV 3D object detection algorithm Sparse-BEVNet. Firstly, a Bi-Level Routing Attention (BRA) mechanism is introduced into the image feature extraction network to reduce the computational burden of the backbone. Second, Cascaded Group Attention (CGA) is employed in the feature fusion module, which enhances deep interaction across features of different hierarchical levels without introducing additional computational overhead. Furthermore, a Sparse Spatial Cross-Attention mechanism is adopted to replace the conventional dense view projection pipeline. Experimental results on the public nuScenes dataset demonstrate that the proposed method achieves a mean Average Precision (mAP) of 45.2% and a nuScenes Detection Score (NDS) of 54.5%, corresponding to 3.6% and 2.8% improvements relative to the baseline model, respectively. Index Terms: 3D Object Detection; Bird’s Eye View (BEV); Bi-Level Routing Attention; Sparse Transformer I Introduction Compared with costly LiDAR-based solutions, vision based multi-view 3D object detection has garnered extensive attention due to its low hardware deployment cost and rich semantic information [1, 2]. In particular, the Bird’s Eye View (BEV) [3] perception enables the unified transformation of 2D image features from multiple on-board cameras into an ego-vehicle-centric 3D space [4]. With the comprehensive breakthrough of the Transformer architecture, attention-based BEV perception algorithms have been proposed successively. Despite the accuracy improvements achieved by existing Transformer-based multi-view perception algorithms, they still face severe challenges when deployed on practical on-board edge devices [5]. Firstly, the computational complexity of global attention generally grows quadratically, resulting in computational overhead. Secondly, existing multi-scale feature fusion modules often suffer from severe computational redundancy [6, 7], and exhibit insufficient capability in preserving features of distant small-scale objects. More critically, mainstream 2D-to-3D view projection mechanisms universally adopt dense spatial grid queries and cross-attention computation. Such dense projection not only consumes massive computing resources, but also causes the model to perform ineffective interactions in meaningless background regions, which constrains the real-time inference performance [8]. To address the limitations, this paper proposes an efficient 3D object detection framework Sparse-BEVNet for autonomous driving. Specifically, Bi-Level Routing Attention (BRA) [9] is introduced into the image feature extraction stage. BRA dynamically filters irrelevant backgrounds at the coarse-grained region level to reduce the computational burden while enhancing the feature capture capability for small objects. In the multi-scale feature fusion stage, a feature neck based on Cascaded Group Attention (CGA) is constructed. CGA divides features of different scales into multiple groups for cascaded deep interaction to improve the feature consistency of multi-view overlapping areas and multi-scale objects. To break through the computational bottleneck of view transformation [10], this paper further proposes a Sparse Spatial Cross-Attention mechanism SSCA. It abandons traditional dense BEV grid queries and adopts instance-based sparse 3D reference points. To verify the effectiveness of Sparse-BEVNet, experimental evaluations are conducted on the nuScenes public dataset [11]. Extensive quantitative comparisons show that the proposed method achieves a mean Average Precision (mAP) of 45.2% and a nuScenes Detection Score (NDS) of 54.5%, representing significant improvements of 3.6% and 2.8% respectively over the baseline algorithm. I Methodology I-A Overall Network Architecture The proposed Sparse-BEVNet is designed to address the problems of high computational cost and insufficient multi-scale feature capture in conventional multi-view single-frame perception algorithms. This network is s developed via modular reconstruction based on the well-established Sparse4D and BEVFormer [12]. As illustrated in Fig. 1, the overall architecture consists of four modules: feature extraction backbone based on BiFormer (Backbone), multi-scale feature fusion neck based on cascaded group attention (CGA), sparse spatial cross-attention mechanism, and a detection head. Fig. 1: Overall architecture of the proposed Sparse-BEVNet. I-B Backbone Feature Extraction Based on Bi-Level Routing Vanilla baseline methods generally adopt ResNet-101 or Swin Transformer as the feature extraction backbone. However, the local convolution operations of ResNet struggle to capture long-range dependencies, while the fixed-window attention mechanism of Swin not only fragments spatial context, but also assigns equal computational weights to large-area low-information-entropy backgrounds (e.g., sky and road surfaces) in high-resolution surround-view images, resulting in severe waste of computing resources. To address this issue, this paper reconstructs the original backbone network into a lightweight feature extractor based on the BiFormer architecture, and introduces the Bi-Level Routing Attention (BRA) mechanism. Specifically, given a 2D feature map output from the shallow network X∈ℝH×W×CX ^H× W× C, the network first uniformly divides it into S×S× S non-overlapping macro regions along the spatial dimension. To evaluate the semantic correlations between different regions with extremely low computational overhead, the algorithm generates dimension-reduced region-level queries QrQ^r and region-level keys Kr∈ℝS2×CK^r ^S^2× C by performing average pooling projection on tokens within each region. Subsequently, the network computes the coarse-grained semantic affinity matrix between macro regions, and applies an asymmetric truncation operation to extract the top-k most relevant region indices, thereby constructing a directed sparse region dependency graph: Ir=TopK(Qr(Kr)⊤,k)I^r=TopK (Q^r(K^r) ,k ) where Ir∈ℕS2×kI^r ^S^2× k denotes the generated routing index matrix. Based on this dynamic routing decision, for the i-th specific region, the network no longer blindly interacts with pixels in the global scope or a fixed window. Instead, it only aggregates corresponding key and value tokens from the k highly relevant regions pointed to by IirI^r_i via the Gather operation, and reconstructs the dynamic context tensor: Kig=Gather(K,Iir),Vig=Gather(V,Iir)K^g_i=Gather (K,I^r_i ), V^g_i=Gather (V,I^r_i ) After completing the above tensor reconstruction, fine-grained attention computation is only performed between query tokens of the current region and the dynamically collected context tensors (Kig,Vig)(K^g_i,V^g_i). I-C Feature Fusion Based on Cascaded Group Attention After multi-level feature extraction, baseline models generally adopt the standard Feature Pyramid Network (FPN) for simple feature concatenation and fusion. This approach suffers from redundant attention computation and weak cross-scale interaction, making it difficult to cope with the drastic scale variation of objects at different distances in autonomous driving scenarios. To this end, this paper replaces the original FPN structure with a feature fusion neck based on Cascaded Group Attention (CGA), which aims to significantly enhance the deep interaction and representation capability of multi-scale features without introducing additional parameter overhead. Specifically, the CGA module first uniformly divides the input multi-level feature maps into M independent group subsets X=[X1,X2,…,XM]X=[X_1,X_2,…,X_M] along the channel dimension, and allocates a separate attention head to each group for computation. To break the limitation of information isolation among groups, CGA introduces an explicit cascaded connection mechanism, which injects the attention output features of the preceding group as additional contextual information into the input of the current group. The mathematical formulation of this cascaded fusion process can be formalized as: Yj=Attnj(Xj+Yj−1),j∈1,2,…,MY_j=Attn_j (X_j+Y_j-1 ), j∈\1,2,…,M\ where AttnjAttn_j denotes the attention operation applied to the j-th group, YjY_j represents the output feature of the j-th group, and the initial boundary condition is set as Y0=0Y_0=0. Through this progressively accumulated serial computation path, each subsequent group in the network can not only focus on its own exclusive feature subspace, but also continuously absorb and integrate the receptive field information of all preceding groups. Finally, the output features YjY_j of all groups are re-concatenated along the channel dimension, and processed by linear projection to generate an enhanced high-quality multi-scale 2D feature set F. I-D Sparse Spatial Cross-Attention Existing BEV perception algorithms typically construct dense grid queries in 3D space to accomplish the 2D-to-3D view transformation. However, the vast majority of the 3D physical space in autonomous driving scenarios is vacant. Dense grid projection not only consumes massive computing resources, but also inevitably introduces a large amount of ineffective background noise interactions. To address this issue, this paper proposes a pure and efficient sparse spatial cross-attention mechanism SSCA, which replaces implicit dense grid search with deterministic geometric physical mapping. Specifically, the network first initializes a set of sparse instance query features Q and their corresponding 3D reference points P=(x,y,z)P=(x,y,z) in the unified 3D coordinate system. To establish precise associations between these 3D anchors and the previously generated multi-scale 2D image features F, the proposed method performs geometric projection using the actual physical calibration parameters of on-board cameras. For the i-th camera with intrinsic matrix Ki∈ℝ3×3K_i ^3× 3 and extrinsic transformation matrix Ti∈ℝ3×4T_i ^3× 4, the projection coordinate pi=(u,v)p_i=(u,v) of a 3D reference point onto the corresponding 2D pixel plane can be rigorously computed via homogeneous coordinate mapping. After obtaining the accurate 2D pixel coordinates, the network no longer performs time-consuming global feature matching. To adapt to object scale variations and projection errors, this paper formalizes the Sparse Spatial Cross-Attention (SSCA) as a process of local deformable sampling and feature aggregation based on projection coordinates: SSCA(Q,P)=∑i∈(P)∑l=1L∑k=1KWi,l,k⋅Fi,l(pi+Δpi,l,k)SSCA(Q,P)= _i (P) _l=1^L _k=1^KW_i,l,k· F_i,l (p_i+ p_i,l,k ) In this formulation, (P)V(P) represents the set of visible camera views where the 3D reference point P can be effectively projected, i.e., without exceeding the image boundary or being occluded by the ego-vehicle. L denotes the number of multi-scale feature levels, and K is the number of local reference points sampled at each level. Δpi,l,k p_i,l,k and Wi,l,kW_i,l,k are the 2D spatial sampling offsets and normalized attention weights respectively, both predicted from the query feature Q via linear layers. Fi,lF_i,l refers to the 2D feature map of the corresponding camera and level output by the CGA neck. After completing the 2D-to-3D feature lifting, the sparse query sequence enriched with abundant visual features is directly fed into the task detection head. The detection head consists of a set of parallel multi-layer perceptrons (MLPs), which decode and regress each sparse instance in parallel to output the final 3D bounding box parameter matrix. The state vector of each predicted object is expressed as y^=(x,y,z,w,l,h,θ,c) y=(x,y,z,w,l,h,θ,c), corresponding to the 3D center coordinates, physical dimensions (width, length, height), yaw angle, and confidence distribution of object categories in the ego-vehicle coordinate system, respectively. In this way, an end-to-end closed loop of single-frame 3D environmental perception is established. I Experiments I-A Experimental Settings The experiments are conducted on nuScenes, a highly authoritative multi-view perception dataset in the autonomous driving field. This dataset covers diverse complex environments including sunny days, rainy days, and nighttime, and provides synchronized images from six surround-view cameras mounted around the ego-vehicle. Considering the computational constraints of experimental hardware and the efficiency of model iterative validation, the complete full dataset is not employed in this work. Instead, a highly representative subset is selected via uniform sampling for both training and evaluation. This subset consists of 200 real road scenes, which is divided into a training set and a validation set following the 8:2 ratio. The development and testing of the proposed algorithm are implemented based on mmdetection3d, an open-source autonomous driving perception framework, and the PyTorch deep learning engine. All model training and inference evaluations are conducted on a high-performance server cluster equipped with 4 NVIDIA A100 Tensor Core GPUs (80 GB VRAM per GPU), which fully meets the parallel computing demands of multi-view high-resolution images.The experiments adopt mean Average Precision (mAP) and nuScenes Detection Score (NDS) as evaluation metrics.In terms of inference efficiency, Frames Per Second (FPS) is adopted as the sole metric to measure the real-time performance of the algorithm. In the training phase, the input images from the 6 surround-view cameras are uniformly resized and cropped to a resolution of 704×256704× 256 pixels. The AdamW optimizer is adopted for model optimization, with an initial learning rate of 2×10−42× 10^-4 and a cosine annealing strategy for dynamic learning rate decay. The weight decay coefficient is set to 1×10−21× 10^-2. The entire network is trained for 100 epochs with a global batch size of 8. I-B Comparative Analysis of Main Experiments To verify the performance of the proposed Sparse-BEVNet in single-frame 3D object detection, quantitative comparisons with mainstream multi-view detection algorithms are conducted on the nuScenes validation subset, with detailed results presented in Table I. TABLE I: Performance comparison of different multi-view 3D object detection algorithms on the nuScenes validation set Method Backbone mAP (%) NDS (%) FPS FCOS3D ResNet-101 29.5 37.2 12 DETR3D ResNet-101 30.3 37.4 14 BEVFormer-S ResNet-101 41.6 51.7 15 Ours BiFormer 45.2 54.5 22 As shown in Table I, Sparse-BEVNet achieves 45.2% mAPmAP and 54.5% NDSNDS, surpassing the BEVFormer-S baseline by 3.6% and 2.8% absolute improvements respectively. The accuracy gain stems from two core designs: bi-level routing attention filters invalid backgrounds early in feature extraction to boost the signal-to-noise ratio of key instances, and cascaded group attention integrates cross-view truncated features to improve the recall of hard samples like distant pedestrians. For inference efficiency, limited by the cross-attention computation of global dense grids, the conventional BEVFormer algorithm only achieves a single-frame inference speed of 15 FPS, which is difficult to cope with unexpected road conditions under high-speed driving. In contrast, the proposed Sparse-BEVNet completely abandons the dense 2D-to-3D view projection mechanism, and directly maps 3D sparse anchors to the 2D plane for local sampling via camera intrinsic and extrinsic matrices. This pure spatial sparse interaction paradigm based on geometric priors not only successfully avoids the massive computational waste caused by vast sky areas and static road surfaces, but also greatly boosts the inference frame rate of the model to 22 FPS. I-C Ablation Experiments and Analysis To quantify the individual contributions of the three core components ablation experiments are conducted on the nuScenes validation subset under consistent experimental settings. The baseline is built with ResNet-101 backbone, standard FPN neck and dense BEV grid projection. TABLE I: Ablation results of core components in Sparse-BEVNet BRA CGA SSCA mAP (%) NDS (%) FPS 41.6 51.7 15 ✓ 42.8 52.6 17 ✓ ✓ 43.7 53.4 16 ✓ ✓ ✓ 45.2 54.5 22 As shown in Table I, the three modules bring complementary performance gains when added sequentially. First, introducing BRA alone improves mAP by 1.2% and raises inference speed to 17 FPS. Specifically, compared to the heavy ResNet-101 baseline (comprising approx. 44.5M parameters and 7.8 GFLOPs), the proposed BRA-based backbone significantly curtails the computational overhead to roughly 26.0M parameters and 4.5 GFLOPs. Second, the cross-scale interaction of CGA compensates for feature truncation in multi-view overlapping areas and feature loss of distant small objects. Adding CGA further brings an extra 0.9% mAP improvement. Finally, replacing dense grid queries with SSCA achieves optimal accuracy and lifts inference speed to 22 FPS, demonstrating that geometry-guided sparse projection cuts redundant computation in empty regions. Furthermore, to investigate the impact of different multi-scale merging paths in the feature fusion module, an additional ablation study was conducted on the input resolution tiers of the CGA neck. As shown in Table I, fusing only the deep semantic layers (C4,C5\C_4,C_5\) yields sub-optimal detection scores due to the severe loss of geometric details. While adding the intermediate layer (C3,C4,C5\C_3,C_4,C_5\) improves the performance, the proposed full hierarchical cascade (C2,C3,C4,C5\C_2,C_3,C_4,C_5\) achieves the highest mAP and NDS. This confirms that integrating the high-resolution shallow features is crucial for detecting truncated and small targets, representing the optimal trade-off in the architectural design. TABLE I: Comparison of alternative multi-scale merging paths in the CGA neck Fusion Resolution Tiers mAP (%) NDS (%) Deep layers only C4,C5\C_4,C_5\ 40.8 51.0 Partial cascade C3,C4,C5\C_3,C_4,C_5\ 43.1 52.8 Full cascade C2,C3,C4,C5\C_2,C_3,C_4,C_5\ (Proposed) 45.2 54.5 IV Conclusion To address computational redundancy and insufficient multi-scale feature extraction in multi-view 3D object detection for autonomous driving, this paper proposes Sparse-BEVNet. With bi-level routing attention (BRA) for backbone reconstruction and cascaded group attention (CGA) as the feature neck, it filters background noise effectively and enhances feature representation for cross-scale and truncated objects. The sparse spatial cross-attention (SSCA) further breaks the computational bottleneck of conventional dense view projection in 2D-to-3D geometric mapping. Experiments on the nuScenes dataset show that the proposed method outperforms the baseline in both mAP and NDS. References [1] T. Wang, X. Zhu, J. Pang, and D. Lin, “Fcos3d: Fully convolutional one-stage monocular 3d object detection,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, p. 913–922. [Online]. Available: https://arxiv.org/abs/2104.10956 [2] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon, “Detr3d: 3d object detection from multi-view images via 3d-to-2d queries,” in Conference on robot learning. PMLR, 2022, p. 180–191. [Online]. Available: https://arxiv.org/abs/2110.06922 [3] Y. Wang, S. Wang, Y. Li, and M. Liu, “Developments in 3d object detection for autonomous driving: A review,” IEEE Sensors Journal, 2025. [Online]. Available: 10.1109/JSEN.2025.3562284 [4] J. Huang, G. Huang, Z. Zhu, Y. Ye, and D. Du, “Bevdet: High-performance multi-camera 3d object detection in bird-eye-view,” arXiv preprint arXiv:2112.11790, 2021. [Online]. Available: https://arxiv.org/abs/2112.11790 [5] J. Wang, F. Li, S. Lv, L. He, and C. Shen, “Physically realizable adversarial creating attack against vision-based bev space 3d object detection,” IEEE Transactions on Image Processing, vol. 34, p. 538–551, 2025. [Online]. Available: https://ieeexplore.ieee.org/document/10838314 [6] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, p. 2117–2125. [Online]. Available: https://arxiv.org/abs/1612.03144 [7] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” arXiv preprint arXiv:2010.04159, 2020. [Online]. Available: https://arxiv.org/abs/2010.04159 [8] Y. Li, B. Huang, Z. Chen, Y. Cui, F. Liang, M. Shen, F. Liu, E. Xie, L. Sheng, W. Ouyang et al., “Fast-bev: A fast and strong bird’s-eye view perception baseline,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, p. 8665–8679, 2024. [Online]. Available: https://ieeexplore.ieee.org/document/10557672 [9] L. Zhu, X. Wang, Z. Ke, W. Zhang, and R. W. Lau, “Biformer: Vision transformer with bi-level routing attention,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, p. 10 323–10 333. [Online]. Available: https://arxiv.org/abs/2303.08810 [10] X. Lin, Z. Pei, K. Li, T. Lin, L. Huang, C. Yu, J. See, Z. Su, and C. Yang, “Sparse4d: Sparse-based end-to-end multi-sensor temporal perception,” IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 1–18, 2026. [Online]. Available: https://arxiv.org/abs/2303.08810 [11] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, p. 11 621–11 631. [Online]. Available: https://arxiv.org/abs/1903.11027 [12] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai, “Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,” arXiv preprint arXiv:2203.17270, 2022. [Online]. Available: https://arxiv.org/abs/2203.17270