Paper deep dive
Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking
Maximilian Luz, Rohit Mohan, Thomas Nürnberg, Yakov Miron, Daniele Cattaneo, Abhinav Valada
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/20/2026, 9:11:22 AM
Summary
The paper introduces Latent Gaussian Splatting (LaGS), a novel framework for 4D Panoptic Occupancy Tracking (4D-POT). It addresses the limitations of existing methods by using 3D Gaussians as a sparse, dynamic, volume-oriented latent representation to aggregate multi-view features. These Gaussians act as keypoints that enable flexible, long-range spatial interactions before being splatted into a voxel grid for decoding. The approach combines hierarchical Gaussian streams (fine and coarse) and a Serialized Multi-Stream Attention mechanism to improve efficiency and accuracy. LaGS achieves state-of-the-art performance on the Occ3D nuScenes and Waymo datasets.
Entities (15)
Relation Signals (12)
LaGS → solves → 4D-POT
confidence 95% · In this work, we present Latent Gaussian Splatting (LaGS) for 4D Panoptic Occupancy Tracking (4D-POT).
Thomas Nürnberg → affiliatedwith → Bosch Research
confidence 90% · Thomas Nürnberg2... 2Bosch Research, Robert Bosch GmbH, Germany
Maximilian Luz → affiliatedwith → University of Freiburg
confidence 90% · Maximilian Luz1... 1Department of Computer Science, University of Freiburg, Germany
LaGS → containscomponent → Serialized Multi-Stream Attention
confidence 90% · we devise a novel cross-stream attention operation: Serialized Multi-Stream Attention (SMSA)
LaGS → evaluatedon → Waymo
confidence 90% · We evaluate LaGS on the Occ3D nuScenes and Waymo datasets
LaGS → evaluatedon → nuScenes
confidence 90% · We evaluate LaGS on the Occ3D nuScenes and Waymo datasets
LaGS → evaluatedon → Occ3D
confidence 90% · Extensive experiments on Occ3D nuScenes and Waymo demonstrate state-of-the-art performance
LaGS → uses → 3D Gaussians
confidence 90% · We revisit the underlying representation and model 3D features as a sparse set of feature-bearing Gaussians.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Capturing 4D spatiotemporal scene structure is crucial for the safe and reliable operation of robots in dynamic environments. However, existing approaches typically address only part of the problem: they either provide coarse geometric tracking via bounding boxes or detailed 3D occupancy estimates that lack explicit temporal association and instance-level reasoning. In this work, we present Latent Gaussian Splatting (LaGS) for 4D Panoptic Occupancy Tracking (4D-POT). We revisit the underlying representation and model 3D features as a sparse set of feature-bearing Gaussians. These act as dynamic, volume-oriented keypoints that enable spatially continuous, distance-weighted aggregation of multi-view features before being splatted into a voxel grid for decoding. This point-centric formulation enables flexible, data-dependent receptive fields and long-range spatial interactions that are difficult to capture with local and dense voxel-based operators. A hierarchical Gaussian representation further enables multi-scale reasoning by combining global context from coarse super-points with fine-grained detail from higher-resolution streams. Extensive experiments on Occ3D nuScenes and Waymo demonstrate state-of-the-art performance for 4D-POT. We provide code and models at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.23172v2
- Canonical: https://arxiv.org/abs/2602.23172v2
Trouble viewing inline? Open PDF directly →
Full Text
50,937 characters extracted from source content.
Expand or collapse full text
Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking Maximilian Luz1, Rohit Mohan1, Thomas Nürnberg2, Yakov Miron2,3, Daniele Cattaneo1, Abhinav Valada1 1Department of Computer Science, University of Freiburg, Germany.2Bosch Research, Robert Bosch GmbH, Germany.3University of Haifa, Israel Abstract Capturing 4D spatiotemporal surroundings is crucial for the safe and reliable operation of robots in dynamic environments. However, most existing methods address only one side of the problem: they either provide coarse geometric tracking via bounding boxes, or detailed 3D structures like voxel-based occupancy that lack explicit temporal association. In this work, we present Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking (LaGS) that advances spatiotemporal scene understanding in a holistic direction. Our approach incorporates camera-based end-to-end tracking with mask-based multi-view panoptic occupancy prediction, and addresses the key challenge of efficiently aggregating multi-view information into 3D voxel grids via a novel latent Gaussian splatting approach. Specifically, we first fuse observations into 3D Gaussians that serve as a sparse point-centric latent representation of the 3D scene, and then splat the aggregated features onto a 3D voxel grid that is decoded by a mask-based segmentation head. We evaluate LaGS on the Occ3D nuScenes and Waymo datasets, achieving state-of-the-art performance for 4D panoptic occupancy tracking. We make our code available at https://lags.cs.uni-freiburg.de/. I Introduction Camera-based 4D panoptic occupancy tracking (4D-POT) [6] combines dense geometric reconstruction, semantic understanding, and temporal consistency into a unified framework. Through this holistic formulation, it promises a comprehensive representation of dynamic environments, addressing key shortcomings of prior paradigms [23]. Specifically, classical box-based tracking methods model scenes only with coarse cuboids [3, 18, 16], lacking fine-grained geometry and volumetric semantics. Conversely, standard 3D occupancy prediction [42, 35, 24] typically operates per frame, producing dense voxel grids but without instance identities or temporal association. In contrast, 4D-POT assigns a semantic class to every voxel in the 3D scene while simultaneously distinguishing and tracking individual object instances across time. By bridging the gap between geometric fidelity and instance-level awareness, 4D-POT has the potential to serve as the central perception backbone for autonomous agents. Figure 1: Illustration of our latent Gaussian representation. Bottom left: panoptic voxel predictions. Center: latent Gaussian features colorized via principal component analysis, splatted to 2D, and overlaid over voxel predictions. Top right: latent Gaussians. Despite the relative novelty of 4D-POT, existing approaches largely follow a direct composition of mask-based 3D occupancy prediction with query-based end-to-end 3D multi-object tracking (3D MOT) [6], primarily by adapting state-of-the-art occupancy prediction approaches. While this is a viable starting point, we argue that the additional temporal and instance-level requirements motivate a reconsideration of the underlying representation and encoder design. In particular, we explore 3D Gaussians as an intermediate sparse latent representation that replaces classical dense 3D voxel feature volumes. Prior work has leveraged Gaussians in 3D occupancy prediction either at the output level [15, 13] or for 2D self-supervision [10]. Here, we instead use Gaussians as a feature-bearing representation that shifts dense voxel-centric architectures towards sparse point-centric ones. Our key idea is to treat 3D Gaussians as dynamic volume-oriented keypoints in space. We first efficiently aggregate features for each Gaussian before splatting them back to a 3D voxel grid for final refinement and decoding. This design yields a sparse latent representation that preserves dense scene context while improving encoder efficiency and effectiveness. We term this concept as latent Gaussian splatting (LaGS), illustrated in Figure 1. This point-centric reformulation enables larger and more flexible data-dependent attention neighborhoods, improving information exchange and scaling behavior compared to more classical dense voxel encoders such as COTR [24], which tend to have smaller and more limited neighborhood relations. By exploiting hierarchical representations, we further improve our encoder and aggregate features into compressed Gaussian superpoints that aid in both 2D-to-3D lifting and mask decoding, making full quadratic-complexity attention operations feasible. Beyond representation, we identify two major challenges in mask-based 4D-POT. First, moving from semantic occupancy to the panoptic setting requires predicting masks not only for global semantic classes but also for individual local instances. This leads to a considerable imbalance between the broad and global (non-instance stuff) masks and the locally constrained instance (thing) masks. While it may be conceptually appealing to treat both semantic and instance queries in a unified manner [6], we find that this imbalance must be handled explicitly. Notably, we observe that instead of jointly aggregating instance and semantic masks, aggregating them separately before merging both aggregated volumes enhances instance segmentation quality considerably. Second, this shift also drastically increases memory requirements, as the number of instance queries typically needs to be many times higher than the maximum number of expected instances [25, 40, 41, 28]. This is further aggravated when integrating classical query-based end-to-end tracking approaches (such as MUTR3D [41]) directly, which generally backpropagate gradients across multiple frames, scaling resource requirements linearly with the number of processed frames. However, our approach indicates that this is not required. Instead, we find that propagating detached queries forward and optimizing each frame in the multi-frame training sequence independently frees significant resources that are better spent in the decoder transformer. In summary, our primary contributions are six-fold: (1) We introduce Gaussians as a sparse intermediate feature representation for dense 3D/4D prediction, extending Gaussian-to-voxel splatting [13] from semantics to features, yielding a more effective and scalable 3D voxel feature encoder. (2) We streamline the integration of query-based end-to-end tracking and mask-based panoptic occupancy prediction, resulting a new state-of-the-art 4D-POT approach. (3) We re-evaluate previously proposed metrics for 4D-POT [6] and address inaccuracies in current implementations. (4) We extend 4D-POT to the nuScenes dataset [9] more commonly used for 3D occupancy prediction, reproduce baselines previously reported only on Waymo [29], and provide groundtruth 4D panoptic occupancy annotations. (5) Through extensive evaluations, we demonstrate state-of-the-art results on both datasets by up to 18.9 p.p.18.9\, p.p. in occupancy segmentation and tracking quality (STQ) [6]. (6) We make the code and models publicly available upon acceptance. I Related Work In this section, we first briefly review methods for 3D occupancy prediction, followed by a discussion of sparse representations addressing the dense 3D nature of the task, and finally cover 4D panoptic occupancy tracking. 3D Occupancy Prediction: While the majority of 3D perception in autonomous driving still relies on bounding-box representations [27], the promise of finer geometric details has led to significant advances in 3D semantic occupancy prediction in recent years [24, 26]. Notably, Occ3D [31] extends the widely adopted nuScenes [9] and Waymo [29] datasets to semantic occupancy prediction, providing more challenging dynamic scenes. Current state-of-the-art thereon [42, 35, 24] largely follows MaskFormer [8], employing a mask-based transformer decoder and posing the task as 3D segmentation. SparseOcc [30] and PaSCo [4] show that, akin to 2D panoptic image segmentation, such a decoder can be adapted straightforwardly to predict 3D panoptic occupancy. However, while this can be conceptually easy, we observe that treating (local) instance and (global) semantic masks as equals (as done in these prior approaches) can lead to underconfident instance segmentation due to the larger semantic masks generally dominating, requiring careful handling. Sparse 3D Occupancy Prediction: A major practical challenge in 3D semantic occupancy prediction is the dense 3D nature of the task. Several works explore more compressed representations [14, 24] or the intrinsic sparsity of the task by focusing on occupied regions only [30, 33], yet retain largely voxel-aligned representations. Notably deviating from this, GaussianFormer [15] proposes 3D Gaussians as a representation for both occupied and free space, followed by GaussianFormer-2 [13] modeling only occupied space for a truly sparse object-centric approach. Both methods splat the predicted 3D Gaussians to the final 3D voxel grid for the task output. We believe that this represents an opportunity to integrate concepts from recent point-based 3D perception methods [22, 37]. Specifically, it allows treating Gaussians as superpoints in space, yielding a sparse latent representation, alleviating costly dense 3D processing, that can be converted into a dense representation at will via splatting. Our novelty, therefore, lies in treating splatting as an intermediate step in the encoder, splatting features instead of appearance or semantics. Moreover, the sparse representation allows our approach to reason more effectively, across larger and more flexible neighborhoods, improving exchange and aggregation of information, 2D-to-3D lifting, and scalability. While sparsity has also been explored in classical object detection [5], the prevalent architectures remain dense networks. 4D Panoptic Occupancy Tracking: By extending 3D panoptic occupancy prediction temporally, TrackOcc [6] introduces the task of 4D panoptic occupancy tracking (4D-POT). While it can be conceptually understood as a 3D extension to the task of video panoptic segmentation (VPS) [17], its dense 3D nature requires specifically tailored approaches due to its inherent computational complexity. To this end, TrackOcc incorporates principles previously explored in end-to-end 3D multi-object tracking [25, 40, 41, 28], ensuring temporally consistent instance assignments by propagating instance-mask-queries forward across frames. While TrackOcc relies on MUTR3D [41] for query-based tracking, we base our approach on PF-Track [28] to incorporate gains from their comparatively lightweight temporal query refinement and further streamline our approach to allow for more than one decoder layer, significantly improving tracking performance. I Method Multi-View Images DepthFeatures D FImage Encoder lift ⨂ pool D F0 V_02 V_2Voxel BackboneSACAFFNWSASCAFFNSMSAFFNFFNN×N× Fsamplesample2 V_20 V_02 G_20 G_0splat features to voxelsmerge V_0 V_2 VLatent Gaussian EncoderSACAVCACAFFN F V2 G_2M×M×Sem.DetectTrack⨂ V Occupancy & IDsPanoptic Mask DecoderMemoryrefineto t+1t+1Query Propagation Figure 2: Architecture overview (left to right). An image encoder ( ) produces image features F and depth D. Features are lifted via depth to a 3D volume and further encoded into a 3D voxel feature pyramid (0 V_0, 2 V_2, ). Our latent Gaussian encoder ( ) samples points from the pyramid volumes and processes them in a coarse (left) and fine (right) stream, using self-attention (SA), windowed self-attention (WSA), cross-attention (CA), spatial cross-attention (SCA), and feed-forward networks (FFN). Our novel Serialized Multi-Stream Attention (SMSA) facilitates information exchange between streams. Refined points are decoded as Gaussians (0 G_0, 2 G_2) and splatted back to a 3D feature volume, which is then refined to the final voxel volume V. Our transformer decoder ( ) then decodes this volume into semantic and instance masks using volume cross-attention (VCA) for efficient query-to-3D-volume attention. Tracking is facilitated by the tracking-by-attention paradigm. We refine track queries by spatio-temporal reasoning before passing them onto the next frame ( ). 4D panoptic occupancy tracking is a highly challenging task, involving close interaction of several components (Section I-A). At its core, our approach (illustrated in Fig. 2) combines concepts from mask-based segmentation [8, 7, 32] with transformer-based end-to-end tracking [25, 41, 28], treating each object instance as a separate tracked query. It can be divided into two major parts: a 3D occupancy volume encoder (Sections I-B and I-C) and a 4D panoptic mask decoder (Sections I-D and I-E). Notably, our occupancy encoder is founded on the same principles as COTR [24], combining explicit (Section I-B) and implicit (Section I-C) view transforms, adhering to the same general structure. Our overall approach begins by lifting and aggregating image features into 3D volume features (Section I-B), which are further refined by our novel latent Gaussian encoder (Section I-C). Given the refined 3D feature volume, a panoptic mask decoder (Section I-D) then uses a query-centric approach to decode semantic and instance masks. Tracking (Section I-E) is facilitated via the same queries following the tracking-by-attention paradigm [25, 40] and enhanced by an additional spatio-temporal query refinement module based on PF-Track [28]. Sections I-F and I-G provide further details for training and inference, respectively. I-A Task Definition Given a consecutive sequence of multi-view images =Itj,It−1j,…It−Tjj=1N I=\I^j_t,I^j_t-1,… I^j_t-T\_j=1^N, where t is the current timestep, T the sequence length, and N the number of camera views, camera-based 4D panoptic occupancy tracking requires joint prediction of occupancy, semantics, and instance associations for each 3D voxel surrounding the ego-vehicle [6]. Specifically, the task requires predicting (cp,t,ip,t)(c_p,t,i_p,t) for each voxel position p, where cp,tc_p,t represents the semantic class and occupancy state (i.e., category including unknown/other and free) and ip,ti_p,t the associated instance ID at timestep t. Notably, semantic predictions incorporate both tracked thing and non-tracked stuff classes, while instance IDs are only assigned for tracked thing classes. Intrinsic and extrinsic parameters of all cameras as well as ego-motion of the vehicle are assumed to be known. I-B Image Encoder, Explicit Lifting, and Voxel Backbone We begin by extracting multi-view image features F with an off-the-shelf image encoder (Fig. 2, ). A small head is trained to predict a binned depth distribution D independently for each image, which is then used to lift the image features (explicitly) to 3D via the outer product =⊗ P= F D, creating a pseudo point cloud. This is subsequently pooled to a 3D voxel grid representation and further refined into a voxel feature pyramid (0 V_0, 2 V_2) using standard 3D convolutions (Fig. 2, ). I-C Latent Gaussian Occupancy Feature Encoder Instead of further refining the dense voxel representation, as done in COTR [24], we explore a novel way of representing 3D features as latent Gaussians, treating them as volumetric keypoints. This point-centric representation allows us to leverage insights from recent point transformer networks [37]. By serializing points via space-filling curves, we can achieve larger and more flexible receptive fields than dedicated dense 3D operations (such as COTR’s voxel self-attention [24]), improving scalability, and replacing fixed neighborhood relations with learned ones through positional encodings. Our final encoder (Fig. 2, ) further extends this hierarchically and employs two parallel streams: a higher-resolution (fine) stream (0 G_0) with more points, capturing details, and a coarse stream (2 G_2), allowing us to aggregate a smaller set of super-points that can subsequently aid query refinement and decoding. Point-Based Refinement (Fine Stream): Given a voxel feature volume 0 V_0, we use multinomial sampling with probability scores based on the feature magnitude to obtain a set of k voxels. These voxels act as seed points for our latent Gaussian representation (0 G_0) and allow us to shift from the dense domain to sparse point-based transformer architectures. Specifically, we follow Point Transformer V3 [37] and first serialize our seed points via a space-filling curve before enriching them with a multi-layer transformer architecture. We employ sliding-window self-attention (WSA) for efficiency, ensuring required memory scales linearly with the number of points k. To efficiently integrate image features for a large number of points, we rely on spatial cross-attention (SCA) [20], projecting the points into the images and thereon using 2D deformable attention. Hierarchical Extension (Coarse Stream): We further enhance our encoder by introducing the concept of hierarchical streams: instead of sampling k points from a single voxel feature volume, we rely on the feature pyramid (0 V_0, 2 V_2), sampling from multiple scales, serializing and processing streams (0 G_0, 2 G_2) independently. This allows us to create a coarser stream (2 G_2) with fewer points, derived from a coarser scale (2 V_2), where full point-wise self-attention and full image-point cross-attention are feasible, while, at the same time, retaining details via a higher-resolution finer stream (0)( G_0). To facilitate efficient stream-to-stream communication, we devise a novel cross-stream attention operation: Serialized Multi-Stream Attention (SMSA). SMSA first merges all streams before re-serializing all points with a single space-filling curve to obtain a single linearized stream. Through this, it can then jointly refine features and exchange information via windowed self-attention. Finally, it splits the unified stream back up into the original hierarchical streams. Notably, SMSA handles varying densities of points and streams intrinsically, without special consideration. Gaussian Feature Aggregation: Our downstream architecture depends on a 3D feature volume. To this end, we adapt 3D Gaussian occupancy splatting [13] for voxel feature aggregation. Specifically, for each point j, we predict centers j μ_j, covariances Σj _j (composed from scale and rotation), opacities αj _j, and feature embeddings j∈ℝC e_j ^C. From this, occupancy o and voxel features f are computed as o() o( x) =1−∏j(1−exp(−12‖−j‖Σj−12)), =1- _j (1- (- 12\| x- μ_j\|^2_ _j^-1 ) ), (1) () f( x) =o()⋅∑jαjj()j∑jαjj(), =o( x)· _j _jG_j( x) e_j _j _jG_j( x), (2) where j()=(∣j,Σj)G_j( x)=N( x μ_j, _j) is the 3D Gaussian PDF, and ‖M2=⊺M\| v\|_M^2= v M v denotes Mahalanobis distance. Note that the decoded Gaussians only represent occupied space. Hierarchical streams are concatenated before aggregation. Subsequently, we merge the aggregated feature volume with the initial feature pyramid volumes, essentially introducing skip connections and creating a U-Net-like structure. I-D Panoptic Mask Decoder We use a mask transformer decoder (Fig. 2, ) with detection queries for instance (thing) segmentation and semantic queries for global instance-less (stuff) semantic masks. As this aligns conceptually well with detection transformers, we adapt PETR [21]. PETR uses cross-attention between multi-view images and queries, which we further extend by letting queries attend to the voxel features via 3D deformable cross-attention and, in the case of hierarchical Gaussian encoding, also to the refined coarse encoder point features. After refinement by the transformer, queries are decoded into a mask embedding and semantic class scores. Similar to MaskFormer [8], binary occupancy masks are computed via a dot product between the mask embedding and the voxel features. I-E Tracking, Query Propagation, and Refinement To facilitate tracking, we propagate successfully decoded detection queries from the decoder forward to the next frame, following the established tracking-by-attention paradigm [25, 40]. These forwarded track queries are then used together with newly initialized detection as well as (non-temporal) semantic/stuff queries as input to the decoder in this subsequent frame. Newly instantiated and successfully decoded detection queries introduce new tracks, while successfully decoded track queries continue old ones. To further improve tracking performance, we integrate the spatio-temporal refinement module of PF-Track [28] (Fig. 2, ), refining queries based on memory (past) and trajectory prediction (future). Predicted trajectories are also used to fill gaps for intermittently missed or low-confidence detections. I-F Training and Supervision We supervise our approach on multiple levels. Following COTR [24], we supervise the depth prediction for explicit feature lifting in the encoder via sparse depth from LiDAR. Further, we add a small head to decode semantic scores for each Gaussian, splatting them to a semantic voxel grid for direct supervision via a cross-entropy loss, akin to GaussianFormer-2 [13]. The decoder is supervised via both semantic and instance masks as well as box predictions (akin to center supervision in TrackOcc [6]). For detection queries, we use bipartite matching, considering both predicted boxes and masks to assign ground truth instances. Once a ground-truth instance has been assigned, it is kept across all subsequent training frames. For semantic queries, we use bipartite matching with masks only. Supervision of the spatio-temporal refinement module follows PF-Track [28]. To enable temporal supervision, we train on short multi-frame sequences. We prevent linear scaling of gradient buffers by detaching track and detection queries after decoding and before refinement, meaning gradients from subsequent frames still flow back to the refinement stage, but not the decoder itself, decoupling individual frames. I-G Inference The raw predictions of our approach are class scores q c_q and occupancy mask scores mq,m_q, x for both instance and stuff queries. We score queries via the maximum class score, i.e., sq=‖q‖∞s_q=\| c_q\|_∞, filter out inactive ones via a threshold, and compute the dominant query q^=argmaxqsq⋅mq, q_ x=arg\,max_q\s_q· m_q, x\ for each voxel x. In contrast to previous methods [6, 8, 7], we compute dominant queries independently for instance and stuff classes, merging both afterward by overriding the stuff predictions with instance ones. IV Experiments The effectiveness of our proposed approach is demonstrated by extensive evaluation. To this end, we first describe the datasets used for benchmarking in Section IV-A and the evaluation metric in Section IV-B, including necessary revisions to correct existing inaccuracies. Brief descriptions of the baselines follow in Section IV-C, along with implementation and training details in Section IV-D. Benchmarking results are presented in Section IV-E, followed by a detailed ablation study of the architectural components (Section IV-F), concluding with qualitative comparisons (Section IV-G). All evaluations are performed on the respective validation splits. IV-A Datasets We evaluate our approach on both nuScenes [9] and Waymo [29], with 3D occupancy ground-truth provided by Occ3D [31]. For both datasets, the spatial range is bounded from −40 m-40\,m to 40 m40\,m for x and y and from −1 m-1\,m to 5.4 m5.4\,m for z, with a voxel size of 0.4 m0.4\,m in each axis, resulting in a voxel grid resolution of 200200×200200×1616. For Waymo, we follow TrackOcc [6] and subsample the dataset at every 5th frame, yielding 789789 training scenes and 202202 validation scenes with 4040 samples each. For nuScenes, we train and evaluate on the full dataset with 700700 training and 150150 validation scenes, with each scene containing around 4040 samples. To obtain 4D panoptic occupancy labels for nuScenes, we assign instance IDs to all thing-class voxels of the Occ3D semantic occupancy data by using the ground-truth box labels. Specifically, instance IDs are assigned based on the intersecting box of the same class. Ambiguities, such as voxels being intersected by none or multiple boxes, are resolved by choosing the closest instance. Notably, the assigned instance IDs are consistent with the nuScenes box instance IDs, facilitating direct box-to-voxel correspondence. We make this data preprocessing available alongside our code. For Waymo, we rely on the data provided by TrackOcc [6]. IV-B Evaluation Metrics Following TrackOcc [6], we adapt the Segmentation and Tracking Quality (STQ), originally introduced for video panoptic segmentation [34], to 4D occupancy prediction and tracking. STQ is defined as the geometric mean of Segmentation Quality (SQ) and Association Quality (AQ), STQ=SQ⋅AQ,STQ= SQ·AQ, (3) where SQ is the classical mean intersection over union (mIoU) of semantic occupancy prediction [2, 31]. AQ represents the mean IoU between each ground-truth and predicted 4D tube, where each individual IoU is weighted by the respective intersected fraction to facilitate a soft assignment and bound the AQ score by one. Mathematically, we define the 4D panoptic occupancy predictions P=(p,t,i,c)P=\(p,t,i,c)\ as the set over tuples of 3D voxel position p, time step t, instance ID i, and class c. Equivalently, we define G as the set of ground-truth tuples. From this, we derive Pi=(p,t,i)∣(p,t,i,c)∈PP_i=\(p,t,i) (p,t,i,c)∈ P\ and equivalently GiG_i as the instance prediction and ground-truth for instance i. Finally, we define AQ =1|IG|∑i∈IG1|Gi|∑j∈IP|Gi∩Pj|⋅|Gi∩Pj||Gi∪Pj|, = 1|I_G| _i∈ I_G 1|G_i| _j∈ I_P |G_i∩ P_j |· |G_i∩ P_j||G_i∪ P_j|, (4) where IG=i∣(p,t,i,c)∈GI_G=\i (p,t,i,c)∈ G\ is the set of ground-truth instance IDs and IPI_P is the set of predicted instance IDs. Extending over TrackOcc, we further propose the AQ1_1 metric for single-frame panoptic assessment. AQ1_1 is constructed analogously to Eq. 4, with the exception that we do not consider tracking tubes but only single-frame instances for matching, i.e., enforce ∀(p1,t1,i1,c1),(p2,t2,i2,c2)∈G:t1≠t2⇒i1≠i2∀(p_1,t_1,i_1,c_1),(p_2,t_2,i_2,c_2)∈ G:t_1≠ t_2 i_1≠ i_2. STQ1_1 follows analogously again as the geometric mean over the already non-temporal mIoU and AQ1_1. This avoids the well-documented shortcomings of the panoptic quality (PQ) metric [34]. In addition to STQ, AQ, and mIoU/SQ, we also use the binary IoU to assess the quality of the binary free/non-free occupancy prediction. Following Occ3D [31], we evaluate only on visible regions (using the “camera” mask). Notably, AQ does not depend on any class assignments, and SQ does not depend on any instance information, strictly separating semantic and instance segmentation between SQ and AQ. While mathematically sound, however, we find that the metric implementations of TrackOcc [6] are flawed: they solely consider areas occupied in the ground-truth data and ignore regions marked as free space. This skews the metric significantly, as any false positives in known free space are disregarded, essentially only counting true positives and false negatives. Mathematically, this is equivalent to applying Eq. 4 to Pi′=Pi∩MP_i =P_i∩ M and Gi′=Gi∩MG_i =G_i∩ M, where M is a mask indicating occupied space in G. We therefore reconstruct and re-evaluate the baselines presented by TrackOcc, as well as TrackOcc itself, and provide corrected implementations alongside our code. IV-C Baselines To fairly evaluate our approach under the revised metrics, we reconstruct the baselines proposed by TrackOcc [6]: MinVIS-inspired bipartite matching of queries across subsequent frames by cosine similarity [11], additionally extended to a CTVIS-based approach where embeddings used for matching are trained contrastively end-to-end [39], heuristic box extraction from instance occupancy data and tracking via AB3DMOT [36], and 4D-LiDAR-panoptic-segmentation-inspired IoU-matching of subsequent predictions [1]. To evaluate the actual effectiveness of the tracking approaches, we propose an additional baseline: assigning new and independent instance IDs each frame (Per-Frame). Through this, we create a minimal metric target for the tracking approaches, below which tracking is measurably ineffective and instead hinders per-frame instance segmentation. IV-D Implementation and Training Details Following PF-Track [28], we choose VoVNetV2 [19] as our image backbone with input resolution 800×320800×320 for nuScenes and 704×256704×256 for Waymo. Image-to-3D lifting follows the BEVDet4D variant of COTR [24] and TrackOcc [6], with memory-intensive stereo and temporal multi-frame aggregation of input frames disabled. All approaches and baselines are trained on 8 NVIDIA L40s for 24 epochs with a batch size of 1. All baselines are based on a single-frame-adapted version of TrackOcc and follow its training procedure [6]. Training of our method is split into 12 epochs of single-frame pre-training and 12 epochs of tracking training, akin to PF-Track [28]. We use a two-stream latent Gaussian encoder with 512512 (coarse) and 81928192 (fine) points and a window size of 10241024 for all window-based attention operations. The encoder and decoder both employ 4 transformer layers. IV-E Comparison with State-of-the-Art TABLE I: 4D-POT performance on Occ3D-nuScenes. mIoU Approach STQ AQ STQ1_1 AQ1_1 all things stuff IoU Per-Frame 9.09.0 2.52.5 21.821.8 14.714.7 32.532.5 26.426.4 41.241.2 59.259.2 MinVIS† [11] 11.811.8 4.34.3 21.821.8 14.714.7 32.532.5 26.426.4 41.241.2 59.259.2 CTVIS† [39] 11.411.4 3.93.9 22.522.5 15.415.4 33.033.0 27.027.0 41.541.5 59.959.9 4D-LCA† [1] 12.512.5 4.84.8 21.821.8 14.714.7 32.532.5 26.426.4 41.241.2 59.259.2 AB3DMOT† [36] 13.113.1 5.35.3 21.821.8 14.714.7 32.532.5 26.426.4 41.241.2 59.259.2 TrackOcc‡ [6] 12.212.2 4.74.7 19.719.7 12.112.1 32.132.1 25.325.3 41.841.8 59.859.8 LaGS-2s (Ours) 31.231.2 24.624.6 35.035.0 31.031.0 39.539.5 35.135.1 45.645.6 64.164.1 † : Baselines reproduced by us. ‡ : Official code adapted for nuScenes. TABLE I: 4D-POT performance on Occ3D-Waymo. mIoU Approach STQ AQ STQ1_1 AQ1_1 all things stuff IoU Per-Frame 11.911.9 4.74.7 — — 30.030.0 32.732.7 29.329.3 — MinVIS† [11] 15.015.0 7.57.5 — — 30.030.0 32.732.7 29.329.3 — CTVIS† [39] 16.416.4 9.39.3 — — 28.928.9 31.931.9 28.128.1 — 4D-LCA† [1] 16.216.2 8.78.7 — — 30.030.0 32.732.7 29.329.3 — AB3DMOT† [36] 18.018.0 10.810.8 — — 30.030.0 32.732.7 29.329.3 — TrackOcc‡ [6] 20.220.2 13.813.8 — — 29.429.4 29.729.7 29.429.4 — Per-Frame 9.19.1 4.04.0 18.118.1 15.615.6 20.920.9 21.921.9 20.620.6 55.555.5 MinVIS† [11] 11.011.0 5.85.8 18.118.1 15.615.6 20.920.9 21.921.9 20.620.6 55.555.5 CTVIS† [39] 12.512.5 7.37.3 18.718.7 16.516.5 21.221.2 22.322.3 20.920.9 56.956.9 4D-LCA† [1] 12.112.1 7.07.0 18.118.1 15.615.6 20.920.9 21.921.9 20.620.6 55.555.5 AB3DMOT† [36] 13.313.3 8.58.5 18.118.1 15.615.6 20.920.9 21.921.9 20.620.6 55.555.5 TrackOcc‡ [6] 15.215.2 10.710.7 18.118.1 15.215.2 21.621.6 21.921.9 21.521.5 57.557.5 LaGS-2s (Ours) 20.320.3 18.618.6 22.222.2 22.222.2 22.222.2 26.126.1 21.321.3 60.260.2 † : Baselines reproduced by us. ‡ : Results reproduced with official code and weights. Gray text: metrics as implemented by TrackOcc [6], ignoring false positives. TABLE I: 3D occupancy prediction performance on Occ3D-nuScenes. Approach mIoU IoU TPVFormer [14] 34.234.2 66.866.8 SurroundOcc [35] 34.634.6 65.565.5 OccFormer [42] 37.437.4 70.170.1 BEVDet4D [12] 39.339.3 73.873.8 BEVDet4D + COTR [24] 44.544.5 75.075.0 BEVDet4D + COTR (w/o longterm, stereo)∗ [24] 34.634.6 66.366.3 TrackOcc‡ [6] 32.132.1 59.859.8 LaGS-2s (Ours) 39.539.5 64.164.1 ‡ : Official code adapted for nuScenes. ∗*: Methodologically closest baseline. Front Cam Ground Truth TrackOcc[6] LaGS-2s (Ours)timeq213334 Figure 3: Qualitative results on the Occ3D-nuScenes validation split. Our approach shows clear improvements in (1) instance separation, (2) instance association, (3) missing detections, and (4) underconfident detections. Front Cam Ground Truth TrackOcc[6] LaGS-2s (Ours)timeq11213 Figure 4: Qualitative results on the Occ3D-Waymo validation split. Our approach shows clear improvements in (1) instance association, (2) instance separation, and (3) ID switches. 4D panoptic occupancy tracking results for nuScenes and Waymo are presented in Tables I and I, respectively. LaGS achieves significant improvements in both overall (STQ) and pure tracking performance (AQ), with up to 18.9 p.p.18.9\, p.p. STQ and 19.8 p.p.19.8\, p.p. AQ for nuScenes, and 5.1 p.p.5.1\, p.p. STQ and 7.9 p.p.7.9\, p.p. AQ for Waymo. The larger increases on nuScenes can be attributed to a more diverse set of tracked (thing) classes, on which our method excels specifically (cf. mIoU-things in Tables I and I). Qualitative evaluation (Fig. 3, Fig. 4) shows a similar picture: The instance segmentation of LaGS is significantly clearer, whereas instance masks in TrackOcc [6] often seem comparatively underconfident, or instance assignments are mixed. An adverse effect of this can be seen on Waymo, where stuff classes are more diverse than on nuScenes and gains in thing-class segmentation come at a minor expense of stuff-class performance. Notably, LaGS also makes a significant step towards closing the gap in semantic occupancy performance (mIoU) between single-frame and 4D-POT methods (see Table I). We outperform the non-temporal non-stereo BEVDet4D+COTR baseline (Table I, gray shade) used as a starting point for TrackOcc [6] and our 4D-POT implementation by 4.9 p.p.4.9\, p.p. mIoU, achieving scores close to and higher than prior state-of-the-art methods like SurroundOcc [35], OccFormer [42], and BEVDet4D [12]. Bringing back temporal aggregation and stereo-based lifting, dropped by TrackOcc [6] and our method due to resource constraints, would likely bridge this gap entirely. IV-F Ablation Studies TABLE IV: Ablation study on the latent Gaussian encoder. pre-training tracking Type Gaussians Layers AQ1_1 mIoU IoU AQ mIoU IoU COTR N/A 1 29.429.4 36.236.2 61.461.4 23.623.6 38.638.6 61.861.8 LaGS 8192 1 29.729.7 35.935.9 61.561.5 23.423.4 38.238.2 61.561.5 COTR N/A 2 28.528.5 36.036.0 61.161.1 23.423.4 38.438.4 61.661.6 LaGS 8192 2 30.130.1 36.236.2 61.761.7 23.923.9 38.538.5 61.961.9 COTR N/A 4 28.728.7 36.236.2 61.261.2 23.223.2 38.338.3 61.761.7 LaGS 2048 4 30.230.2 36.636.6 62.362.3 24.224.2 39.239.2 63.163.1 LaGS 4096 4 30.530.5 36.736.7 62.062.0 24.124.1 39.039.0 62.462.4 LaGS 8192 4 30.430.4 36.436.4 62.062.0 24.124.1 38.938.9 62.062.0 LaGS-2s 512, 2048 4 30.930.9 37.037.0 62.762.7 24.624.6 39.439.4 63.863.8 LaGS-2s 512, 4096 4 31.031.0 37.437.4 63.263.2 24.424.4 39.139.1 63.963.9 LaGS-2s 512, 8192 4 31.131.1 37.237.2 63.063.0 24.624.6 39.539.5 64.164.1 Performance reported after both single-frame pre-training (12 epochs) and full tracking training (12 epochs pre-training + 12 epochs tracking training). Shaded gray background indicates the original COTR encoder. Shaded blue background indicates our chosen configuration. TABLE V: Ablation study on the decoder transformer. mIoU Layers Refinement STQ AQ all things stuff IoU 1 ✓ 26.126.1 18.518.5 36.836.8 31.131.1 44.944.9 61.561.5 4 ✗ 29.129.1 22.722.7 37.237.2 32.232.2 44.344.3 61.761.7 4 ✓ 30.230.2 23.623.6 38.638.6 34.634.6 44.344.3 61.861.8 Performance reported using the COTR encoder with 1 encoder layer. Shaded background indicates our chosen configuration. Latent Gaussian Encoder: Effective aggregation of information from images and spatial neighborhoods is crucial for both 3D panoptic occupancy prediction and 4D panoptic occupancy tracking. Hence, we validate the efficacy of our proposed latent Gaussian encoder in both single-frame 3D panoptic occupancy (pre-training) and 4D tracking scenarios (Table IV). While our proposed encoder performs largely similarly to the COTR [24] encoder with a single transformer layer, its benefits lie in scaling, outperforming the COTR encoder with both 2 and 4 layers. We believe that this is due to the more dynamic neighborhoods of our approach. Both our encoder and COTR rely on spatial cross-attention [20] for efficiency and feasibility, incurring the drawback that all points on a given view ray will be projected onto the same image coordinate. This leads to image information being erroneously attributed not just to the front surface but to the full ray, especially when working with dense grids like COTR. To reason about this, finding the correct attribution and deduplicating the information, networks likely benefit from exchanging information across larger data-dependent regions. Therein, COTR is limited again: its use of deformable attention [38] for voxel self-attention restricts queries to attend to only a small (kp=8k_p=8) set of query-dependent points around the query location. Through its point-based design, our approach, however, allows for neighborhoods of kw=1024k_w=1024, in which queries can freely interact. This is further improved upon in our dual-stream encoder, where the additional coarse stream a) extends the neighborhood hierarchically and b) facilitates the use of full cross-attention between coarse points and voxels, leading to gains in binary occupancy of up to 2.3 p.p.2.3\, p.p. IoU and tracking quality of up to 1.0 p.p.1.0\, p.p. AQ. Decoder Transformer: The decoder transformer (Sections I-D and I-E) plays a major role in both instance segmentation and tracking. Results of our ablation studies are shown in Table V. While previous works [24, 6] employ only a single decoder layer, we find that using multiple layers yields a significant increase in performance for semantic segmentation of tracked classes (mIoU-things) as well as instance-track associations (AQ). Further, spatio-temporal refinement of tracked queries enables considerable improvements in thing-class semantic segmentation with up to 2.4 p.p.2.4\, p.p. mIoU-things, showing that temporal refinement can effectively aggregate useful semantic occupancy and instance information across time steps. TABLE VI: Ablation study on mask aggregation. mIoU Approach STQ AQ AQ1_1 all things stuff IoU Unified 29.529.5 22.322.3 28.828.8 39.239.2 34.634.6 45.7 64.1 Split 31.1 24.5 31.0 39.5 35.1 45.645.6 64.1 +/−+/- +1.6 +2.2 +2.2 +0.3 +0.5 -0.1 0.0 Shaded background indicates our chosen configuration. Mask Aggregation: Moving from pure semantic to panoptic masks introduces a discrepancy: stuff-class semantic masks remain global, whereas thing-class masks are now locally constrained, comparatively small, and act independently for each instance. Visual inspection indicates that instance masks are likely less confident at boundary regions; hence, when treating stuff- and thing-class masks jointly, stuff-class masks tend to dominate. We can counteract this by computing dominant masks separately for both types (cf. Section I-G), leading to improved instance segmentation (AQ1_1) and tracking (AQ) metrics, as shown in Table VI. IV-G Qualitative Evaluation Qualitative results are presented in Fig. 3 (nuScenes) and Fig. 4 (Waymo). We demonstrate improvements across five categories: (1) instance separation, correctly separating nearby instances, (2) instance association, concisely assigning an object to a single instance, (3) missing detections, leading to incorrect free-space predictions, (4) underconfident mask predictions, leading to incomplete instance segmentation, and (5) ID switches, where the same instance ID is wrongly assigned to different objects in subsequent frames. V Conclusion We proposed a novel Gaussian-driven architecture, outperforming the state-of-the-art in 4D panoptic occupancy tracking. Inspired by recent advancements in Gaussian splatting and point-transformer methods, we designed a novel 3D occupancy feature encoder. Using Gaussians as a 3D representation allows us to convert the classically dense voxel-grid-based encoders of occupancy prediction tasks into a sparse, point-wise format, suitable for standard transformer-based architectures. This allows for more flexible, data-driven information aggregation, effective 2D-to-3D lifting, and improved scalability. By utilizing splatting to convert the sparse point-wise representation back into a voxel grid, our encoder can replace any classical voxel feature encoder. Extensive evaluations on both Occ3D nuScenes and Waymo datasets demonstrate the effectiveness of our approach. We hope that this paves the way for further exploration into more dynamic, effective, and saliency-driven 3D representations. Acknowledgments This work was funded by the Bosch Research collaboration on AI-driven automated driving. A.V. was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH_2698644), and the state of Baden-Württemberg. References [1] M. Aygün, A. Osep, M. Weber, M. Maximov, C. Stachniss, J. Behley, and L. Leal-Taixé (2021) 4D panoptic lidar segmentation. In CVPR, p. 5527–5537. Cited by: §IV-C, TABLE I, TABLE I, TABLE I. [2] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall (2019) SemanticKITTI: a dataset for semantic scene understanding of lidar sequences. In ICCV, p. 9297–9307. Cited by: §IV-B. [3] M. Büchner and A. Valada (2022) 3d multi-object tracking using graph neural networks with cross-edge modality attention. IEEE Robotics and Automation Letters 7 (4), p. 9707–9714. Cited by: §I. [4] A. Cao, A. Dai, and R. de Charette (2024) PaSCo: urban 3D panoptic scene completion with uncertainty awareness. In CVPR, p. 14554–14564. Cited by: §I. [5] Y. Chen, J. Liu, X. Zhang, X. Qi, and J. Jia (2023) VoxelNeXt: fully sparse voxelnet for 3D object detection and tracking. In CVPR, p. 21674–21683. Cited by: §I. [6] Z. Chen, K. Li, X. Yang, T. Jiang, Y. Li, and H. Zhao (2025) TrackOcc: camera-based 4d panoptic occupancy tracking. In ICRA, Cited by: §I, §I, §I, §I, §I, §I-A, §I-F, §I-G, Figure 3, Figure 4, §IV-A, §IV-A, §IV-B, §IV-B, §IV-C, §IV-D, §IV-E, §IV-E, §IV-F, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I. [7] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-attention mask transformer for universal image segmentation. In CVPR, p. 1290–1299. Cited by: §I-G, §I. [8] B. Cheng, A. Schwing, and A. Kirillov (2021) Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, Vol. 34, p. 17864–17875. Cited by: §I, §I-D, §I-G, §I. [9] W. K. Fong, R. Mohan, J. V. Hurtado, L. Zhou, H. Caesar, O. Beijbom, and A. Valada (2022) Panoptic nuscenes: a large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters 7 (2), p. 3795–3802. Cited by: §I, §I, §IV-A. [10] W. Gan, F. Liu, H. Xu, N. Mo, and N. Yokoya (2025) GaussianOcc: fully self-supervised and efficient 3D occupancy estimation with gaussian splatting. In ICCV, p. 28980–28990. Cited by: §I. [11] D. Huang, Z. Yu, and A. Anandkumar (2022) MinVIS: a minimal video instance segmentation framework without video-based training. In NeurIPS, Vol. 35, p. 31265–31277. Cited by: §IV-C, TABLE I, TABLE I, TABLE I. [12] J. Huang and G. Huang (2022) BEVDet4D: exploit temporal cues in multi-camera 3D object detection. Note: arXiv preprint, arXiv:2203.17054 Cited by: §IV-E, TABLE I. [13] Y. Huang, A. Thammatadatrakoon, W. Zheng, Y. Zhang, D. Du, and J. Lu (2025) GaussianFormer-2: probabilistic gaussian superposition for efficient 3D occupancy prediction. In CVPR, p. 27477–27486. Cited by: §I, §I, §I, §I-C, §I-F. [14] Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and J. Lu (2023) Tri-perspective view for vision-based 3D semantic occupancy prediction. In CVPR, p. 9223–9232. Cited by: §I, TABLE I. [15] Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and J. Lu (2025) GaussianFormer: scene as gaussians for vision-based 3D semantic occupancy prediction. In ECCV, p. 376–393. Cited by: §I, §I. [16] M. Käppeler, Ö. Çiçek, D. Cattaneo, C. Gläser, Y. Miron, and A. Valada (2025) Bridging perspectives: foundation model guided bev maps for 3d object detection and tracking. arXiv preprint arXiv:2510.10287. Cited by: §I. [17] D. Kim, S. Woo, J. Lee, and I. S. Kweon (2020) Video Panoptic Segmentation. In CVPR, p. 9859–9868. Cited by: §I. [18] C. Lang, A. Braun, L. Schillingmann, and A. Valada (2023) Self-supervised multi-object tracking for autonomous driving from consistency across timescales. IEEE Robotics and Automation Letters 8 (11), p. 7711–7718. Cited by: §I. [19] Y. Lee, J. Hwang, S. Lee, Y. Bae, and J. Park (2019) An energy and GPU-computation efficient backbone network for real-time object detection. In CVPRW, p. 752–760. Cited by: §IV-D. [20] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai (2025) BEVFormer: learning bird’s-eye-view representation from LiDAR-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (3), p. 2020–2036. Cited by: §I-C, §IV-F. [21] Y. Liu, T. Wang, X. Zhang, and J. Sun (2022) PETR: position embedding transformation for multi-view 3D object detection. In ECCV, p. 531–548. Cited by: §I-D. [22] Z. Liu, J. Hou, X. Wang, X. Ye, J. Wang, H. Zhao, and X. Bai (2024) LION: linear group RNN for 3D object detection in point clouds. In NeurIPS, Vol. 37, p. 13601–13626. Cited by: §I. [23] M. Luz, R. Mohan, A. R. Sekkat, O. Sawade, E. Matthes, T. Brox, and A. Valada (2024) Amodal optical flow. In ICRA, Cited by: §I. [24] Q. Ma, X. Tan, Y. Qu, L. Ma, Z. Zhang, and Y. Xie (2024) COTR: compact occupancy transformer for vision-based 3D occupancy prediction. In CVPR, p. 19936–19945. Cited by: §I, §I, §I, §I, §I-C, §I-F, §I, §IV-D, §IV-F, §IV-F, TABLE I, TABLE I. [25] T. Meinhardt, A. Kirillov, L. Leal-Taixé, and C. Feichtenhofer (2022) TrackFormer: multi-object tracking with transformers. In CVPR, p. 8844–8854. Cited by: §I, §I, §I-E, §I, §I. [26] R. Mohan, J. V. Hurtado, R. Mohan, and A. Valada (2026) ForecastOcc: vision-based semantic occupancy forecasting. arXiv preprint arXiv:2602.08006. Cited by: §I. [27] R. Mohan, D. Cattaneo, F. Drews, and A. Valada (2024) Progressive multi-modal fusion for robust 3d object detection. In CoRL, Cited by: §I. [28] Z. Pang, J. Li, P. Tokmakov, D. Chen, S. Zagoruyko, and Y. Wang (2023) Standing between past and future: spatio-temporal modeling for multi-camera 3D multi-object tracking. In CVPR, p. 17928–17938. Cited by: §I, §I, §I-E, §I-F, §I, §I, §IV-D. [29] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. (2020) Scalability in perception for autonomous driving: waymo open dataset. In CVPR, p. 2446–2454. Cited by: §I, §I, §IV-A. [30] P. Tang, Z. Wang, G. Wang, J. Zheng, X. Ren, B. Feng, and C. Ma (2024) SparseOcc: rethinking sparse latent representation for vision-based semantic occupancy prediction. In CVPR, p. 15035–15044. Cited by: §I, §I. [31] X. Tian, T. Jiang, L. Yun, Y. Mao, H. Yang, Y. Wang, Y. Wang, and H. Zhao (2023) Occ3d: a large-scale 3d occupancy prediction benchmark for autonomous driving. In NeurIPS, Cited by: §I, §IV-A, §IV-B, §IV-B. [32] N. Vödisch, K. Petek, W. Burgard, and A. Valada (2023) CoDEPS: online continual learning for depth estimation and panoptic segmentation. RSS. Cited by: §I. [33] J. Wang, Z. Liu, Q. Meng, L. Yan, K. Wang, J. Yang, W. Liu, Q. Hou, and M. Cheng (2024) OPUS: occupancy prediction using a sparse set. In NeurIPS, p. 119861–119885. Cited by: §I. [34] M. Weber, J. Xie, M. D. Collins, Y. Zhu, P. Voigtlaender, H. Adam, B. Green, A. Geiger, B. Leibe, D. Cremers, et al. (2021) STEP: segmenting and tracking every pixel. In NeurIPS, Cited by: §IV-B, §IV-B. [35] Y. Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu (2023) SurroundOcc: multi-camera 3D occupancy prediction for autonomous driving. In ICCV, p. 21729–21740. Cited by: §I, §I, §IV-E, TABLE I. [36] X. Weng, J. Wang, D. Held, and K. Kitani (2020) 3D multi-object tracking: a baseline and new evaluation metrics. In IROS, Cited by: §IV-C, TABLE I, TABLE I, TABLE I. [37] X. Wu, L. Jiang, P. Wang, Z. Liu, X. Liu, Y. Qiao, W. Ouyang, T. He, and H. Zhao (2024) Point transformer v3: simpler faster stronger. In CVPR, p. 4840–4851. Cited by: §I, §I-C, §I-C. [38] Z. Xia, X. Pan, S. Song, L. E. Li, and G. Huang (2022) Vision transformer with deformable attention. In CVPR, p. 4794–4803. Cited by: §IV-F. [39] K. Ying, Q. Zhong, W. Mao, Z. Wang, H. Chen, L. Y. Wu, Y. Liu, C. Fan, Y. Zhuge, and C. Shen (2023) CTVIS: consistent training for online video instance segmentation. In CVPR, p. 899–908. Cited by: §IV-C, TABLE I, TABLE I, TABLE I. [40] F. Zeng, B. Dong, Y. Zhang, T. Wang, X. Zhang, and Y. Wei (2022) MOTR: end-to-end multiple-object tracking with transformer. In ECCV, Cited by: §I, §I, §I-E, §I. [41] T. Zhang, X. Chen, Y. Wang, Y. Wang, and H. Zhao (2022) MUTR3D: a multi-camera tracking framework via 3D-to-2D queries. In CVPRW, Cited by: §I, §I, §I. [42] Y. Zhang, Z. Zhu, and D. Du (2023) OccFormer: dual-path transformer for vision-based 3D semantic occupancy prediction. In ICCV, Cited by: §I, §I, §IV-E, TABLE I.