Paper deep dive
Failure Modes for Deep Learning-Based Online Mapping: How to Measure and Address Them
Michael Hubbertz, Qi Han, Tobias Meisen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 92%
Last extracted: 3/23/2026, 12:09:24 PM
Summary
The paper addresses failure modes in deep learning-based online mapping for autonomous driving, specifically identifying localization overfitting (memorization of input features) and geometric overfitting (overfitting to known map geometries). The authors propose a framework using Fréchet distance-based metrics to quantify these failure modes and introduce a minimum-spanning-tree (MST) based dataset sparsification strategy to improve model generalization and performance.
Entities (7)
Relation Signals (4)
Online Mapping → exhibits → Localization Overfitting
confidence 90% · Deep learning-based online mapping models still suffer from significant failure modes... localization overfitting
Online Mapping → exhibits → Geometric Overfitting
confidence 90% · overfitting to known map geometries
Fréchet Distance → measures → Geometric Overfitting
confidence 90% · We introduce Fréchet distance-based reconstruction statistics that capture per-element shape fidelity
MST-based sparsification → mitigates → Geometric Overfitting
confidence 85% · we formulate an MST-based sparsification strategy that reduces redundancy and improves balancing
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep learning-based online mapping has emerged as a cornerstone of autonomous driving, yet these models frequently fail to generalize beyond familiar environments. We propose a framework to identify and measure the underlying failure modes by disentangling two effects: Memorization of input features and overfitting to known map geometries. We propose measures based on evaluation subsets that control for geographical proximity and geometric similarity between training and validation scenes. We introduce Fréchet distance-based reconstruction statistics that capture per-element shape fidelity without threshold tuning, and define complementary failure-mode scores: a localization overfitting score quantifying the performance drop when geographic cues disappear, and a map geometry overfitting score measuring degradation as scenes become geometrically novel. Beyond models, we analyze dataset biases and contribute map geometry-aware diagnostics: A minimum-spanning-tree (MST) diversity measure for training sets and a symmetric coverage measure to quantify geometric similarity between splits. Leveraging these, we formulate an MST-based sparsification strategy that reduces redundancy and improves balancing and performance while shrinking training size. Experiments on nuScenes and Argoverse 2 across multiple state-of-the-art models yield more trustworthy assessment of generalization and show that map geometry-diverse and balanced training sets lead to improved performance. Our results motivate failure-mode-aware protocols and map geometry-centric dataset design for deployable online mapping.
Tags
Links
- Source: https://arxiv.org/abs/2603.19852v1
- Canonical: https://arxiv.org/abs/2603.19852v1
Trouble viewing inline? Open PDF directly →
Full Text
72,984 characters extracted from source content.
Expand or collapse full text
Failure Modes for Deep Learning–Based Online Mapping: How to Measure and Address Them Michael Hubbertz1,2 Qi Han2 Tobias Meisen1 1University of Wuppertal, <lastname>@uni-wuppertal.de 2Aptiv Services Deutschland GmbH, <firstname>.<lastname>@aptiv.com Abstract Deep learning-based online mapping has emerged as a cornerstone of autonomous driving, yet these models frequently fail to generalize beyond familiar environments. We propose a framework to identify and measure the underlying failure modes by disentangling two effects: Memorization of input features and overfitting to known map geometries. We propose measures based on evaluation subsets that control for geographical proximity and geometric similarity between training and validation scenes. We introduce Fréchet distance–based reconstruction statistics that capture per-element shape fidelity without threshold tuning, and define complementary failure-mode scores: a localization overfitting score quantifying the performance drop when geographic cues disappear, and a map geometry overfitting score measuring degradation as scenes become geometrically novel. Beyond models, we analyze dataset biases and contribute map geometry-aware diagnostics: A minimum-spanning-tree (MST) diversity measure for training sets and a symmetric coverage measure to quantify geometric similarity between splits. Leveraging these, we formulate an MST-based sparsification strategy that reduces redundancy and improves balancing and performance while shrinking training size. Experiments on nuScenes and Argoverse 2 across multiple state-of-the-art models yield more trustworthy assessment of generalization and show that map geometry-diverse and balanced training sets lead to improved performance. Our results motivate failure-mode-aware protocols and map geometry-centric dataset design for deployable online mapping. GitHub Page 1 Introduction Maps are essential navigation tools, offering critical spatial information that helps drivers safely and efficiently navigate intricate road systems. Standard definition (SD) maps typically provide enough detail for human drivers to understand general road layouts. However, high-definition (HD) maps have become vital for autonomous vehicles, delivering detailed information including precise lane structures which complement onboard sensor perception of the dynamic environment and ensure safe navigation. While traditional HD maps are constructed offline, autonomous driving systems require the ability to generate and update maps in real-time as the vehicle traverses new environments. This task, commonly referred to as online mapping, leverages multimodal sensor inputs such as lidar, radar, and camera data. Contemporary research has shifted toward deep learning-based methods, as conventional rule-based approaches are unable to capture the complex spatial and semantic relationships necessary for accurate HD map generation. Figure 1: Validation set performance of different state-of-the-art online mapping models on the nuScenes dataset [2] with original and two different geographically disjoint [24, 42] splits. Despite remarkable progress, deep learning-based online mapping models still suffer from significant failure modes. Lilja et al. have highlighted that models trained on widely used datasets such as nuScenes and Argoverse 2 exhibit strong geographic memorization effects rather than true generalization [24]. When evaluated on geographically overlapping splits, models show inflated performance, whereas performance drops drastically under geographically disjoint splits [24, 42, 31, 33] (cf. Fig. 1). Furthermore, dataset biases amplify the observed generalization issues through skewed training and evaluation, as the data may lack sufficient diversity in road structures and layouts. In this work, we aim to examine these model and dataset related failure modes in detail. We propose new evaluation criteria to disentangle model memorization from generalization and provide dataset analysis and modification tools to quantify and minimize geometric biases. Our key contributions can be summarized as follows: • We propose new evaluation measures based on discrete Fréchet distance that better capture geometric reconstruction quality compared to traditional average precision metrics based on Chamfer distance. • We introduce a systematic framework to categorize and measure model failure modes in online mapping, disentangling localization overfitting from geometric overfitting. • We conduct extensive experiments across multiple state-of-the-art online mapping models, examining their failure modes in more detail. • We analyze biases in online mapping datasets, introducing new measures of geometric diversity and geometric similarity between splits and propose new splits with higher dissimilarity regarding map geometries. • We propose a minimum spanning tree-based dataset sparsification strategy that reduces redundancy, enhances geometric balancing, and improves model generalization. 2 Related Work 2.1 Online Mapping Model Architectures HDMapNet [18] pioneered deep learning-based online mapping as the first method to directly generate vectorized map elements from sensor data. Building on this, more recent approaches adopt Detection Transformer (DETR) inspired designs [3], where sensor features are projected into bird’s-eye view space and processed by transformer decoders with learnable object queries, followed by classification and regression heads to identify map categories and shapes. Key examples include MapTR [22], VectorMapNet [26], and MapTRv2 [23]. Further enhancements have optimized this foundational architecture. Examples include incorporating mask guidance to improve map object predictions [25], refining query designs within transformer decoders [27], integrating prior and additional information [41, 20, 17, 14, 39, 34, 37], and employing temporal fusion techniques to mitigate inconsistencies caused by occlusions and complex environments [19, 42, 5, 29]. Building on the generalization challenges outlined in Sec. 1, we employ several state-of-the-art online mapping architectures as experimental testbeds to systematically analyze failure modes. We deliberately include diverse architectures to reduce model-specific bias and to draw conclusions about localization and map geometry overfitting that are robust across different design choices. 2.2 Overfitting Countermeasures and Metrics Deep learning-based online mapping models are particularly prone to overfitting. Lilja et al. demonstrated in their study that rather than learning generalizable structures, they tend to memorize location-specific patterns during training [24]. This highlights a form of structural memorization that differs from conventional overfitting. Consequently, standard countermeasures such as early stopping, weight decay, and dropout [6, 10, 35] are insufficient, motivating the need for more domain-specific strategies. To measure how much a deep learning model overfits, Aburass and Rumman [1] introduced the Overfitting Index, combining accuracy and loss to track generalization across training epochs. Margin-based distributions can also predict the generalization gap [11, 13, 12], while reviews by Valle-Pérez and Louis [38] and critiques by Gastpar et al. [9] draw attention to the theoretical limits of generalization bounds. In this work, we aim to investigate and quantify overfitting specifically for deep learning-based online mapping models. We aim to provide deeper insights into the core problems that lead to performance decrease for geographically disjoint splits, quantifying intuitive expectations and proposing improved performance scores and additional evaluation measures. We also propose countermeasures through pruning samples with redundant map geometries in training data, leading to improved balance during training. 2.3 Online Mapping Datasets and Biases Recent advances in online mapping for automated driving have predominantly relied on supervised learning, requiring datasets that pair sensor inputs capturing the vehicle’s environment with corresponding HD ground-truth maps. Publicly available datasets that satisfy this requirement are Argoverse 1 and 2 [4, 40], nuScenes [2], and Waymo [36], with nuScenes and Argoverse 2 emerging as predominantly used datasets in the domain of online mapping [18, 22, 26, 23, 27, 41, 20, 17, 14, 39, 34, 37, 19, 42, 5, 29]. Both datasets provide predefined splits that partition the data into training and validation sets. Thereby, inherent biases in these datasets and splits can skew training and evaluation, leading to inflated benchmark performance while hindering real-world generalization. For example, Koe et al. and Marathe et al. showed that autonomous-driving datasets are biased toward clear-weather conditions, leading to poor robustness under adverse weather [28, 16]. Similar issues have been explored in object detection, including studies on class imbalance [15, 30] and fairness in pedestrian detection [8, 21, 32]. Research on dataset biases in online mapping has focused on geographic bias for dataset splits [33, 31, 24, 42]. Geographic bias occurs when training and evaluation data come from overlapping regions, allowing models to rely on location-specific cues rather than learn generalizable representations, thereby inflating benchmark performance. The original nuScenes and Argoverse 2 splits were originally designed for object detection as well as motion forecasting tasks and follow a temporal division, resulting in partially large geographical overlap [24]. For the original nuScenes and Argoverse 2 splits, approximately 80 % and 45 % of validation samples, respectively, are within five meters of a sample used during training. To mitigate such a geographic bias, several geographical splits that minimize overlap for nuScenes [33, 31, 24, 42] and Argoverse 2 [24, 42] have been proposed recently. Building on these observations, we revisit dataset bias for online mapping from a broader perspective. Beyond the well-studied issue of geographical sampling bias, we identify geometric similarity between train and validation scenes as a critical yet underexplored factor. We further quantify geometric diversity and similarity across existing splits and show their correlation with model performance, motivating the design of map geometry-aware splits for more reliable evaluation of generalization. 3 Failure Modes for Online Mapping Models We hypothesize that performance drops between geographically overlapping and disjoint dataset splits reveal two recurring issues: Reliance on memorized input features (localization overfitting) and overfitting to known map geometries (geometric overfitting). To study these effects systematically, we first derive evaluation sets from the dataset splits that enable inference of models’ tendencies toward distinct modes of overfitting. Second, we introduce improved performance measures. Based on both, we introduce measures that systematically capture and quantify both model-based failure modes. In the following, the training and validation sets are denoted as T and V, respectively, and the set of map object classes is defined as CmapC_map. 3.1 Evaluation Set Derivation As a first step, we introduce two criteria to further specify each validation sample. The first criterion is the geographical distance d from each validation sample to its closest training sample, capturing potential leakage due to spatial overlap and concrete feature memorization. d(v):=mint∈Tdist(v,t)d(v):= _t∈ Tdist(v,t) (1) where dist(v,t)dist(v,t) denotes the geographical Euclidean distance (or a different suitable distance measure) between the validation sample v∈Vv∈ V and the training sample t∈Tt∈ T. However, two scenes may be geographically distant yet geometrically identical, or adjacent yet structurally novel. Consequently, we also assess how closely the map geometries align across different scenes. Therefore, we introduce the geometric similarity s between the ground truth map of each validation sample and its most geometrically similar training sample. s(v):=mint∈Tsim(v,t)s(v):= _t∈ Tsim(v,t) (2) where sim(v,t)sim(v,t) denotes the geometric similarity cost (low = similar, high = dissimilar) between the validation sample v∈Vv∈ V and the training sample t∈Tt∈ T. For computing the geometry similarity cost sim(v,t)sim(v,t), each class ci∈Cmapc_i∈ C_map is handled separately. We compare the two samples’ map elements ℳvM_v and ℳtM_t by forming the assignment-cost matrix Av,t∈ℝ|ℳv|×|ℳt|A_v,t ^|M_v|×|M_t|. The cost between two elements (each a polygon or polyline with a variable number of points) is the minimum discrete Fréchet distance [7] over all point orderings: For polylines, original and reversed; for polygons, all cyclic permutations in both orientations (original and reversed). A mathematical definition and details on computing the discrete Fréchet distance are provided in the supplementary material. Element assignments are obtained via minimum-cost bipartite matching on Av,tA_v,t. Let amatcheda_matched be the sum of costs for matched pairs across classes, nmatchedn_matched their count, and nunmatchedn_unmatched the number of unmatched elements, each incurring a fixed penalty δ. The geometry similarity cost is then defined as sim(v,t):=amatched+nunmatched⋅δnmatched+nunmatched.sim(v,t):= a_matched+n_unmatched·δn_matched+n_unmatched. (3) Normalizing by nmatched+nunmatchedn_matched+n_unmatched prevents samples with few elements from being overemphasized. We provide an instructive visualization of sim(v,t)sim(v,t) for an exemplary sample pair in the supplementary material. To check whether the inductive biases of current DETR-based mapping decoders correlates more with rotation- and translation-invariant topological features rather than geometrical features, we also provide an ablation study with a topological similarity measure in the supplementary material. Figure 2: Correlation between d(v)d(v) and s(v)s(v) for the nuScenes original split (Pearson correlation coefficient r=0.724r=0.724). Three representative pairs of s(v)s(v) are presented, showing each validation sample alongside its closest geometric match from the training set. As we intend to derive disentangled failure mode measures based on s(v)s(v) and d(v)d(v) later on, it is important to examine how these two quantities correlate. As seen in Fig. 2, both measures are strongly positively correlated on the nuScenes original split: Validation samples in close proximity to the training set tend to have more similar map geometries (as expected), whereas more distant samples generally exhibit more dissimilar map geometries. Next, we can separate the validation split V into sample sets that provide information about the model failure modes. First, we divide V into two subsets Vclose,Vfar⊂V_close,\>V_far⊂ V based on d(v)d(v) with the distance threshold TdistT_dist: Vclose V_close :=v∈V∣d(v)≤Tdist, :=\v∈ V d(v)≤ T_dist\, (4) Vfar V_far :=v∈V∣d(v)>Tdist. :=\v∈ V d(v)>T_dist\. (5) The separation indicates whether the deep learning model has already been trained with a sample in close vicinity to the validation sample or not and would be able to utilize knowledge about memorized location-specific features. We therefore expect the performance of validation samples in VcloseV_close to be superior compared to VfarV_far. However, to draw valid conclusions about the performance drop between VcloseV_close and VfarV_far, one must also account for s(v)s(v), given its strong correlation with d(v)d(v) and to avoid conflating localization overfitting with geometric overfitting. To quantify localization overfitting, we refine the VcloseV_close and VfarV_far sets by alignment of the distributions of s(v)s(v) across both sets. The empirical cumulative distribution Fs,VsubF_s,V_sub of s(vsub)s(v_sub) for a subset Vsub⊂V_sub⊂ V is described by Fs,Vsub(t):=1|Vsub||vsub∈Vsub∣s(vsub)≤t|.F_s,V_sub(t)\;:=\; 1|V_sub|\, |\\,v_sub∈ V_sub s(v_sub)≤ t\ |. (6) To match the distributions w.r.t. s(v)s(v) between VcloseV_close and VfarV_far, we sample the subsets Vclose*⊂VcloseV_close*⊂ V_close and Vfar*⊂VfarV_far*⊂ V_far by approximating Fs,Vclose*(t)≈Fs,Vfar*(t)∀t.F_s,V_close*(t)≈ F_s,V_far*(t)\;∀\,t. (7) To obtain Vclose*V_close* and Vfar*V_far*, we perform bipartite matching of s(v)s(v) across both sets, followed by filtering based on a predefined threshold for the absolute value of the differences between s(v)s(v) for the matched pairs. To ensure that the geometric similarity distributions of s(v)s(v) for VcloseV_close and Vfar*V_far* are closely aligned, we choose the threshold such that their Kullback–Leibler divergence remains below 0.01. We further examine geometric overfitting by stratifying the geographically distant subset VfarV_far. Let τ0≤τ1≤…≤τb _0≤ _1≤…≤ _b partition [minv∈Vfars(v),maxv∈Vfars(v)][ _v∈ V_fars(v),\, _v∈ V_fars(v)] into b equal-width intervals. For i=1,…,bi=1,…,b, we define Bi:=v∈Vfar∣τi−1<s(v)≤τi.B_i:=\\,v∈ V_far _i-1<s(v)≤ _i\,\. (8) Because VfarV_far enforces no geographic overlap with training data, variation across Bii=1b\B_i\_i=1^b isolates the effect of map geometry alone: Low-index bins (low s(v)s(v)) group geometrically similar scenes, whereas high-index bins (high s(v)s(v)) group geometrically dissimilar scenes. Evaluating the performance per bin yields a geometry–performance curve that quantifies sensitivity to geometric novelty. 3.2 Performance Scores To introduce measurements for the previously defined failure modes, we first aim to quantify the performance of the examined online mapping model on the evaluation sets. To ensure comparability with recent online mapping works [18, 22, 26, 23, 27, 41, 17, 14, 39, 34, 37, 19, 42, 5, 29] we adopt the standard average precision (AP) metric with the Chamfer distance as the matching criterion, details and definitions are provided in the supplementary material. For each class ci∈Cmapc_i∈ C_map, we compute the AP across all samples in the set by declaring a prediction a true positive if its Chamfer distance DChamferD_Chamfer to the ground truth is below a threshold τ. We evaluate the AP separately for each threshold τ∈Tτ∈ T, where T=0.5,1.0,1.5T=\0.5,1.0,1.5\, and then take the average across all thresholds T and all classes CmapC_map to obtain the final mean average precision (mAP) score used for comparison: mAP=1|Cmap||T|∑ci∈Cmap∑τ∈TAPci,τ.mAP= 1|C_map||T| _c_i∈ C_map _τ∈ TAP_c_i,τ. (9) While the Chamfer distance-based AP suffices for performance estimates for large sets of samples, it struggles to capture the performance for sets with low amounts of samples due to its sensitivity to discrete outcomes and lack of granularity in small sample scenarios. Because matches are discretized into hits or misses, the metric can be sensitive to individual outcomes and lacks granularity regarding how close a near-miss is, failing to capture the exact degree of similarity between prediction and ground truth. Moreover, the Chamfer distance is permutation-invariant and only enforces proximity of points, not their global arrangement, which limits its ability to assess whether the reconstructed map element preserves the correct shape. Figure 3: Visualization of two exemplary prediction and ground truth map element pairs. The Chamfer distance (red) and Fréchet distance (green) are shown as performance metrics for both cases. In example (a), both metrics produce meaningful results. In example (b), the Chamfer distance remains nearly unchanged compared to (a) because it ignores point ordering, whereas the Fréchet distance yields a much higher value, capturing the larger geometric deviation. To address these limitations, we introduce an intuitive measure of the quality of ground-truth map geometry reconstruction. This new performance measure is based on the discrete Fréchet distance. It takes into account the order of points within each map element, while being independent of their individual lengths. In contrast to the Chamfer distance, this yields a more accurate assessment of per-element similarity (cf. Fig. 3). Analogous to sim(⋅,⋅)sim(·,·), we compute class-wise bipartite matching between the prediction P and the ground truth G and collect the matched costs into a distribution D. To assess how well the prediction reconstructs the ground-truth map, we use the median M and the interquartile range IQRIQR: M=median(D)andIQR=Q3(D)−Q1(D)M=median(D) IQR=Q_3(D)-Q_1(D) (10) where Q1(D)Q_1(D) and Q3(D)Q_3(D) denote the first and third quartiles of D. Unlike mAP, they avoid threshold calibration and remain informative for sets of varying sizes, being less sensitive to discrete outcomes and enabling comparisons at the single-sample level. Additional details for M can be found in the supplementary material. 3.3 Measures for Failure Modes As stated in Sec. 3.1, the performance drop between Vclose*V_close* and Vfar*V_far* describes the model’s extent of localization overfitting. We utilize the proposed Fréchet distance-based measure M across all classes for quantifying the performance drop, to improve the significance of the results for sets with low amounts of samples compared to mAP. We propose the following localization overfitting score locO_loc for measuring this property of a model: loc:=Mfar*−Mclose*Mclose*O_loc:= M_far*-M_close*M_close* (11) This score thus measures the relative performance drop. A value near zero indicates that the model generalizes well to novel feature combinations (Mfar*≈Mclose*M_far*≈ M_close*), whereas a large value reveals a strong reliance on memorizing concrete location-specific features (Mfar*≫Mclose*M_far* M_close*). To quantify geometric overfitting, we measure the performance decay of Mfar,iM_far,i across bins BiB_i. This is estimated via linear regression, where the slope of the fitted line represents the rate of performance drop. To enable an arbitrary number of bins b, we use the mean value of s(v)s(v) within each interval [τi−1,τi][ _i-1, _i], denoted as μs,far,i _s,far,i, together with the corresponding Mfar,iM_far,i value for the linear regression. To account for differing bin sizes, we weight the regression data points by their sample counts wiw_i. Formally, we propose the following geometry overfitting score geomO_geom: geom=∑i=1bpi(μs,far,i−μx)(Mfar,i−μy)∑i=1bpi(μs,far,i−μx)2wherepi=wi∑j=1bwj,μx=∑i=1bpiμs,far,i,μy=∑i=1bpiMfar,i. split&O_geom= _i=1^bp_i ( _s,far,i- _x ) (M_far,i- _y ) _i=1^bp_i ( _s,far,i- _x )^2 \\[4.0pt] &p_i= w_i _j=1^bw_j,\ \> _x= _i=1^bp_i\, _s,far,i,\ \> _y= _i=1^bp_i\,M_far,i. split (12) A value close to zero for geomO_geom indicates great generalization properties regarding map geometry (Mfar,1≈Mfar,bM_far,1~≈~M_far,b). The higher the value of geomO_geom, the more the performance decreases across bins and the model tends to overfit on the learned training map geometries. 4 Dataset Biases in Online Mapping Beyond model-specific failure modes, dataset bias can distort training and evaluation in online mapping. Building on Sec. 3.3, we consider two relevant biases: geographical bias (spurious performance improvements from geographical overlap and resulting feature memorization from T to V) and geometric bias (performance gains from the similarity of map geometries across T and V). Geographical bias in widely used online mapping datasets such as nuScenes [2] and Argoverse 2 [40] has been extensively studied, as discussed, and geographically disjoint splits are available to mitigate this effect [24, 31, 33, 42]. Building on these efforts, we turn our attention to geometric biases in these datasets, which remain largely unexplored. 4.1 Geometric Diversity of Sets We hypothesize that online mapping models achieve better performance when the training data exhibits high geometric diversity and a well-balanced representation across different map geometries, as opposed to datasets dominated by a few or highly similar geometries. To confirm this, we introduce a dataset-level measure of geometric diversity. For a given sample set D=s1,…,sMD=\s_1,...,s_M\, we construct a fully connected weighted graph simG_sim where each node ni∈n_i corresponds to a sample sis_i and each edge eij∈ℰe_ij is assigned the corresponding similarity cost sim(si,sj)sim(s_i,s_j). From simG_sim, we extract the minimum spanning tree (MST) simT_sim. We take the sum of all edge weights eij=sim(si,sj)e_ij=sim(s_i,s_j) in simT_sim as a measure for the geometric diversity of the sample set D and define it as geomdiv(D)geomdiv(D): geomdiv(D):=∑(i,j)∈ℰ(sim)sim(si,sj).geomdiv(D):= _(i,j) (T_sim)sim(s_i,s_j). (13) High values of geomdiv(D)geomdiv(D) indicate a set with diverse and varied map geometries, while low values suggest redundancy and limited geometric variety. This approach offers an interpretable method for comparing splits, monitoring geometric coverage as data volume increases, and selecting subsets that maintain structural diversity. 4.2 Geometric Similarity between Sets Besides geometric diversity, geometric similarity between training and validation splits could also influence evaluation performance. We expect prediction performance to improve when the map geometry distributions of the training and evaluation sets are similar, compared to cases where these distributions differ significantly. To capture geometric similarity between two sets D1D_1 and D2D_2, we define it as their symmetric coverage-based similarity. The directed cover cov(D1→D2)cov(D_1→ D_2) measures how well D2D_2 covers the map geometries in D1D_1 by averaging the nearest-neighbor map geometry similarity cost from each sample s1,i∈D1s_1,i∈ D_1 into D2D_2: cov(D1→D2):=1|D1|∑s1,i∈D1mins2,j∈D2sim(s1,i,s2,j).cov(D_1\!→\!D_2):= 1|D_1| _s_1,i∈ D_1 _s_2,j∈ D_2\,sim(s_1,i,s_2,j). (14) Because coverage is asymmetric, we define the geometric similarity between both sets geomsim(D1,D2)geomsim(D_1,D_2) as the mean of both directions, yielding an order-agnostic score that attains low values only when the sets mutually cover each other’s map geometries: geomsim(D1,D2):=12(cov(D1→D2)+cov(D2→D1)).geomsim(D_1,D_2):= 12 (cov(D_1\!→\!D_2)+cov(D_2\!→\!D_1) ). (15) 5 Experiments 5.1 Experimental Setup Models Our primary analyses use MapTRv2 [23] as a representative state-of-the-art online mapping architecture to validate the proposed evaluation protocol and failure-mode measures. To assess whether the observations generalize across design choices, we additionally examine a basic model (MapTR [22]) and more advanced variants with different transformer decoder query architecture (MapQR [27]) and auxiliary masking task (MGMap [25]). All models were trained by us based on the publicly available codebases and configurations. Datasets and Splits We use the nuScenes and Argoverse 2 datasets. Following prior work, we compare the original dataset splits with geographically disjoint splits (near extrapolation split by [24] and geographical split by [42]). Furthermore, we derive geometric dataset splits for the whole dataset that maximize dissimilarity between training and evaluation splits. All splits partition the dataset into 70/15/15% for the training, validation, and test sets, respectively. More details about the derivation of geometric dataset splits are provided in the supplementary material. 5.2 Measure Verification and Results Figure 4: Sample-wise performance of MapTRv2 measured by M for the nuScenes original split, plotted over d(v)d(v) (left) and s(v)s(v) (right). In the left plot, an exemplary Tdist=5mT_dist=5\,m is displayed, separating the validation set V into VcloseV_close and VfarV_far. Performance is positively correlated with both d(v)d(v) and s(v)s(v), with the correlation being stronger for s(v)s(v) (Pearson correlation coefficient r=0.568>0.379r=0.568>0.379). Dataset and split geomdiv(T)geomdiv(T) geomdiv(V)geomdiv(V) geomsim(T,V)geomsim(T,V) d(v)<5md(v)<5\,m mAP↑mAP M±IQR↓M^± IQR loc↓O_loc geom↓O_geom nuScenes original 96.8 km 30.1 km 8.32 m 79.47 % 60.95 1.94±3.051.94^± 3.05 24.73 21.22 geo. [24] 80.6 km 22.5 km 14.66 m 0.95 % 24.96 4.07±6.144.07^± 6.14 n.a. 9.75 geo. [42] 90.2 km 22.4 km 13.85 m 0 % 28.53 3.24±5.503.24^± 5.50 n.a. 13.84 geometric 91.3 km 10.7 km 21.08 m 8.53 % 28.37 4.17±6.084.17^± 6.08 4.40 10.49 Argoverse 2 original* 91.0 km 23.7 km 8.98 m 44.89 % 63.97 1.77±2.991.77^± 2.99 7.29 11.17 geo. [24]* 87.3 km 19.8 km 11.37 m 0 % 49.53 2.40±4.182.40^± 4.18 n.a. 11.97 geo. [42]* 97.2 km 18.6 km 10.47 m 0 % 57.61 2.13±3.692.13^± 3.69 n.a. 11.15 geometric* 90.7 km 18.6 km 26.51 m 4.73 % 34.13 2.59±4.302.59^± 4.30 9.37 7.98 Table 1: Comparison of original, geographically disjoint [24, 42] and geometric splits for nuScenes and Argoverse 2. Dataset split geometric diversities (geomdiv(geomdiv) and similarities (geomsimgeomsim) are listed for training set T and validation set V as well as their geographical overlap (d(v)<5md(v)<5\,m). For the dataset properties, the Argoverse 2 splits (*) were subsampled from 10 Hz to 2 Hz to enable comparison with nuScenes. Performance and overfitting measures are provided for MapTRv2 [23]. Localization Overfitting Score locO_loc (%) ↓ Geometric Overfitting Score geomO_geom (%) ↓ nuScenes Argoverse 2 nuScenes splits Argoverse 2 splits Model original split original split original geo. [24] geo. [42] original geo. [24] geo. [42] MapTR [22] 24.42 22.06 18.66 13.55 18.66 10.69 22.43 13.86 MapTRv2 [23] 24.73 7.29 21.22 9.75 13.84 11.17 11.97 11.15 MapQR [27] 57.07 13.91 21.03 10.87 19.48 10.73 11.37 9.08 MGMap [25] 33.19 22.06 24.12 14.06 17.42 10.69 23.50 24.47 Table 2: locO_loc and geomO_geom for different state-of-the-art online mapping models on nuScenes and Argoverse 2 datasets with original and geographically disjoint splits [24, 42] and Tdist=5mT_dist=5\;m to separate between VcloseV_close and VfarV_far. Model Failure Modes First, we aim to analyze our proposed measures and verify our preceding hypotheses. For the following experiments, we employ MapTRv2 [23] as a representative online mapping model. In Fig. 4, we evaluate per-sample performance on all validation samples from the nuScenes original split and plot the newly proposed measure M versus d(v)d(v) (left) and versus s(v)s(v) (right). The performance drop between VcloseV_close and VfarV_far can be clearly observed with the new performance measure, as well as degrading performance for decreasing sample similarity with even stronger correlation in comparison. The same effects can be observed for all other examined splits, since the failure mode measures locO_loc and geomO_geom in Tab. 1 yield positive results across all datasets and splits. Furthermore, the performance results for mAP and M in Tab. 1 are coherent, suggesting that M±IQRM^± IQR is a suitable performance measure. Tab. 2 compares localization overfitting (locO_loc) and geometric overfitting (geomO_geom) across the previously selected models on all datasets and splits. The results indicate that all models exhibit noticeable overfitting, but the extent varies by architecture and dataset. MapQR, for instance, shows the strongest localization overfitting on nuScenes. In contrast, MapTRv2 achieves comparatively lower localization overfitting scores on Argoverse 2, suggesting stronger generalization capabilities. The results indicate that model performance is shaped by both input memorization and geometric similarity, underscoring the need for evaluation protocols that separate these effects. Dataset Bias In Tab. 1, we list dataset split measures for geometric diversity (geomdiv(T)geomdiv(T) and geomdiv(V)geomdiv(V)), geometric similarity (geomsim(T,V)geomsim(T,V)) and geographical bias (d(v)<5md(v)<5\,m). The original splits of both datasets demonstrate significant geographical overlap, in addition to substantial similarity across the training and validation sets, suggesting the presence of a bias. The geographical splits proposed in [24] and [42] exhibit minimal geographical overlap and low geometric similarity between training and validation sets, leading to a significant performance drop. A detailed comparison of the two geographical splits reveals that a high level of diversity in the training set (high geomdiv(T)geomdiv(T)) is advantageous for the model. Both splits demonstrate only minor deviations for both geomdiv(V)geomdiv(V) and geomsim(T,V)geomsim(T,V). In contrast, the geographical training splits from [42] exhibit a higher level of diversity compared to those from [24], resulting in enhanced performance. Evaluation on the geometric splits shows that the geometric MST-based split derivation leads to much larger geomsim(T,V)geomsim(T,V) compared to the previous splits. The low geometric similarity between the training and validation split also leads to low geographical overlap, due to the correlation of d(v)d(v) and s(v)s(v) (cf. Fig. 2). As a consequence, the performance measure M suggests a decrease in performance compared to other splits for both datasets, while the mAP result for nuScenes is similar to the geographical splits. For Argoverse 2, both mAP and M indicate a significant performance drop to original and geographical splits. Figure 5: Effect of MST-based training set sparsification on sample size and diversity (Top) and model performance for MapTRv2 on the validation set (Bottom). Besides examining the effect across all examined nuScenes and Argoverse 2 splits, we randomly sample from the training sets of the original splits for comparison. 5.3 Map Geometry-Based Training Set Sparsification We also conduct ablation studies for the dataset diversity measure, to verify our hypothesis that a model trained on a geometrically diverse and well-balanced dataset outperforms one trained on a more homogeneous or imbalanced dataset. To demonstrate this, the MST for the training set is taken across all datasets and splits, and sparsification is performed based on a geometric similarity threshold. All samples that are connected with edges below the threshold belong to one cluster, for which a representative sample is chosen by the lowest average neighbor weight. In Fig. 5 we show the remaining sample amounts, geomdiv(T)geomdiv(T) and validation set performance (mAP) for the sparsified training sets. For low thresholds (0.1-1), the cumulative MST edge length geomdiv(T)geomdiv(T) remains practically unchanged, while the sample amounts drop significantly. For higher thresholds (2-10), both quantities decline. Interestingly, the performance seems to increase for most splits between sparsification thresholds 0.1 and 1, suggesting that multiple samples with very similar map geometry do not benefit the model and removing each but one leads to a more well-balanced training of the model, without overemphasizing specific map geometries. For higher thresholds (2-10) the performance decreases, because more samples are being removed leading to significant loss of information in the training data. To confirm that our MST-based sparsification strategy is effective, we compared the sparsified training sets from the original splits of nuScenes and Argoverse 2 with randomly sampled ones with the same remaining sample amounts. In contrast to our strategy, the geometric diversity decreases significantly even for low thresholds (0.1-1) and random sampling never leads to any performance gains. Exact numbers for Fig. 5 are provided in the supplementary material. 6 Conclusion We introduced a failure-mode–aware evaluation for online mapping that separates dependence on location-specific features from sensitivity to map geometry. Using distance to training data and a geometric similarity measure, we built stratified subsets to reveal whether models are memorizing features or geometric structures or truly generalizing. A Fréchet distance-based, order-aware performance measure complemented established metrics on small and imbalanced splits. We also quantified dataset geometric diversity with a MST approach and showed how it can guide pruning of redundant samples. Across two benchmarks, the analysis revealed clear signs of feature memorization and consistent performance drop with lower geometric similarity, and linked higher diversity in training data to better performance. In future work, we aim to transfer the concept of diversity measurements through feature-based MSTs to other domains and investigate its general applicability. We also plan to explore map geometry-aware training objectives as a means to reduce geometric overfitting and to leverage diversity and similarity measures for guiding active data selection. Furthermore, we see potential in scaling similarity estimation by employing faster or learned approximations, enabling our methods to operate effectively on larger datasets. Acknowledgements This work is a result of the joint research project STADT:up (19A22006B). The project is supported by the German Federal Ministry for Economic Affairs and Climate Action (BMWK), based on a decision of the German Bundestag. The authors are solely responsible for the content of this publication. References [1] S. Aburass and M. A. Rumman (2024-07) Quantifying Overfitting: Introducing the Overfitting Index. In 2024 International Conference on Electrical, Computer and Energy Technologies (ICECET, Sydney, Australia, p. 1–7 (en). External Links: ISBN 9798350395914, Link, Document Cited by: §2.2. [2] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom (2020-05) nuScenes: A multimodal dataset for autonomous driving. arXiv (en). Note: arXiv:1903.11027 [cs]Comment: CVPR 2020 camera ready incl. supplementary material External Links: Link, Document Cited by: Figure 1, Figure 1, §2.3, §4. [3] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020-05) End-to-End Object Detection with Transformers. arXiv (en). Note: arXiv:2005.12872 [cs] External Links: Link Cited by: §2.1. [4] M. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays (2019-11) Argoverse: 3D Tracking and Forecasting with Rich Maps. arXiv (en). Note: arXiv:1911.02620 [cs]Comment: CVPR 2019 External Links: Link, Document Cited by: §2.3. [5] J. Chen, Y. Wu, J. Tan, H. Ma, and Y. Furukawa (2024-10) MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping. arXiv (en). Note: arXiv:2403.15951 [cs]Comment: ECCV 2024 (Oral). Project page: https://map-tracker.github.io External Links: Link Cited by: §2.1, §2.3, §3.2. [6] T. Dietterich (1995-09) Overfitting and undercomputing in machine learning. ACM Computing Surveys 27 (3), p. 326–327 (en). External Links: ISSN 0360-0300, 1557-7341, Link, Document Cited by: §2.2. [7] T. Eiter and H. Mannila (1994) Computing Discrete Fréchet Distance. (en). Cited by: §3.1, §8. [8] D. Fernández Llorca, P. Frau, I. Parra, R. Izquierdo, and E. Gómez (2024-09) Attribute annotation and bias evaluation in visual datasets for autonomous driving. Journal of Big Data 11 (1), p. 137 (en). External Links: ISSN 2196-1115, Link, Document Cited by: §2.3. [9] M. Gastpar, I. Nachum, J. Shafer, and T. Weinberger (2023-11) Fantastic Generalization Measures are Nowhere to be Found. arXiv (en). Note: arXiv:2309.13658 [cs]Comment: 34 pages, 1 figure. Minor fix: subsection 6.2 -> section 7 External Links: Link, Document Cited by: §2.2. [10] D. M. Hawkins (2004-01) The Problem of Overfitting. Journal of Chemical Information and Computer Sciences 44 (1), p. 1–12 (en). External Links: ISSN 0095-2338, Link, Document Cited by: §2.2. [11] M. Huang, H. Yu, and J. Zhang (2025-03) A practical generalization metric for deep networks benchmarking. Scientific Reports 15 (1), p. 9747 (en). External Links: ISSN 2045-2322, Link, Document Cited by: §2.2. [12] Y. Jiang, P. Foret, S. Yak, D. M. Roy, H. Mobahi, G. K. Dziugaite, S. Bengio, S. Gunasekar, I. Guyon, and B. Neyshabur (2020-12) NeurIPS 2020 Competition: Predicting Generalization in Deep Learning. arXiv (en). Note: arXiv:2012.07976 [cs]Comment: 20 pages, 2 figures. Accepted for NeurIPS 2020 Competitions Track. Lead organizer: Yiding Jiang External Links: Link, Document Cited by: §2.2. [13] Y. Jiang, D. Krishnan, H. Mobahi, and S. Bengio (2019-06) Predicting the Generalization Gap in Deep Networks with Margin Distributions. arXiv (en). Note: arXiv:1810.00113 [stat]Comment: Published in ICLR 2019 External Links: Link, Document Cited by: §2.2. [14] Z. Jiang, Z. Zhu, P. Li, H. Gao, T. Yuan, Y. Shi, H. Zhao, and H. Zhao (2024-03) P-MapNet: Far-seeing Map Generator Enhanced by both SDMap and HDMap Priors. arXiv (en). Note: arXiv:2403.10521 [cs]Comment: Code: https://jike5.github.io/P-MapNet External Links: Link, Document Cited by: §2.1, §2.3, §3.2. [15] D. Katare, N. Kourtellis, S. Park, D. Perino, M. Janssen, and A. Y. Ding (2022-12) Bias Detection and Generalization in AI Algorithms on Edge for Autonomous Driving. In 2022 IEEE/ACM 7th Symposium on Edge Computing (SEC), Seattle, WA, USA, p. 342–348 (en). External Links: ISBN 978-1-66548-611-8, Link, Document Cited by: §2.3. [16] W. Kou, G. Zhu, R. Ye, Q. Lin, Z. Ren, M. Tang, and Y. Wu (2025-01) Generalizable Autonomous Driving System across Diverse Adverse Weather Conditions. arXiv (en). Note: arXiv:2409.14737 [cs]Comment: 16 Pages External Links: Link, Document Cited by: §2.3. [17] H. Li, Z. Huang, Z. Wang, W. Rong, N. Wang, and S. Liu (2024-06) Enhancing 3D Lane Detection and Topology Reasoning with 2D Lane Priors. arXiv (en). Note: arXiv:2406.03105 [cs]Comment: 20 pages, 9 figures, 6 tables External Links: Link, Document Cited by: §2.1, §2.3, §3.2. [18] Q. Li, Y. Wang, Y. Wang, and H. Zhao (2022-03) HDMapNet: An Online HD Map Construction and Evaluation Framework. arXiv (en). Note: arXiv:2107.06307 [cs] External Links: Link Cited by: §2.1, §2.3, §3.2. [19] S. Li, J. Lin, H. Shi, J. Zhang, S. Wang, Y. Yao, Z. Li, and K. Yang (2024-05) DTCLMapper: Dual Temporal Consistent Learning for Vectorized HD Map Construction. arXiv (en). Note: arXiv:2405.05518 [cs, eess]Comment: The source code will be made publicly available at https://github.com/lynn-yu/DTCLMapper External Links: Link Cited by: §2.1, §2.3, §3.2. [20] S. Li, K. Yang, H. Shi, J. Zhang, J. Lin, Z. Teng, and Z. Li (2023-09) Bi-Mapper: Holistic BEV Semantic Mapping for Autonomous Driving. arXiv (en). Note: arXiv:2305.04205 [cs]Comment: Accepted to IEEE Robotics and Automation Letters (RA-L). The source code is publicly available at https://github.com/lynn-yu/Bi-Mapper External Links: Link, Document Cited by: §2.1, §2.3. [21] X. Li, Z. Chen, J. M. Zhang, F. Sarro, Y. Zhang, and X. Liu (2025-03) Bias behind the Wheel: Fairness Testing of Autonomous Driving Systems. ACM Transactions on Software Engineering and Methodology 34 (3), p. 1–24 (en). External Links: ISSN 1049-331X, 1557-7392, Link, Document Cited by: §2.3. [22] B. Liao, S. Chen, X. Wang, T. Cheng, Q. Zhang, W. Liu, and C. Huang (2023-01) MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction. arXiv (en). Note: arXiv:2208.14437 [cs]Comment: Accepted to ICLR 2023 as Spotlight Presentation. Code&demos: https://github.com/hustvl/MapTR External Links: Link Cited by: §10, §2.1, §2.3, §3.2, §5.1, Table 2. [23] B. Liao, S. Chen, Y. Zhang, B. Jiang, Q. Zhang, W. Liu, C. Huang, and X. Wang (2023-08) MapTRv2: An End-to-End Framework for Online Vectorized HD Map Construction. arXiv (en). Note: arXiv:2308.05736 [cs]Comment: Code available at https://github.com/hustvl/MapTR . arXiv admin note: substantial text overlap with arXiv:2208.14437 External Links: Link Cited by: §10, §2.1, §2.3, §3.2, §5.1, §5.2, Table 1, Table 1, Table 2. [24] A. Lilja, J. Fu, E. Stenborg, and L. Hammarstrand (2024-06) Localization is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix it. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, p. 22150–22159 (en). External Links: ISBN 9798350353006, Link, Document Cited by: Figure 1, Figure 1, §1, Table 5, Table 5, Table 8, Table 8, §2.2, §2.3, §2.3, §4, §5.1, §5.2, Table 1, Table 1, Table 1, Table 1, Table 2, Table 2, Table 2, Table 2, §7, Table 3, Table 3. [25] X. Liu, S. Wang, W. Li, R. Yang, J. Chen, and J. Zhu (2024-06) MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, p. 14812–14821 (en). External Links: Link, Document Cited by: §10, §2.1, §5.1, Table 2. [26] Y. Liu, T. Yuan, Y. Wang, Y. Wang, and H. Zhao (2023-06) VectorMapNet: End-to-end Vectorized HD Map Learning. arXiv (en). Note: arXiv:2206.08920 [cs]Comment: Accepted by ICML 2023 External Links: Link Cited by: §2.1, §2.3, §3.2. [27] Z. Liu, X. Zhang, G. Liu, J. Zhao, and N. Xu (2024-07) Leveraging Enhanced Queries of Point Sets for Vectorized Map Construction. arXiv (en). Note: arXiv:2402.17430 [cs]Comment: Accepted to European Conference on Computer Vision (ECCV), 2024.Code can be found at https://github.com/HXMap/MapQR External Links: Link Cited by: §10, §2.1, §2.3, §3.2, §5.1, Table 2. [28] A. Marathe, R. Walambe, and K. Kotecha (2023-01) In Rain or Shine: Understanding and Overcoming Dataset Bias for Improving Robustness Against Weather Corruptions for Autonomous Vehicles. arXiv (en). Note: arXiv:2204.01062 [cs]Comment: Under review External Links: Link, Document Cited by: §2.3. [29] N. Peng, X. Zhou, M. Wang, X. Yang, S. Chen, and G. Chen (2024-07) PrevPredMap: Exploring Temporal Modeling with Previous Predictions for Online Vectorized HD Map Construction. arXiv (en). Note: arXiv:2407.17378 [cs] External Links: Link Cited by: §2.1, §2.3, §3.2. [30] N. Peri, A. Dave, D. Ramanan, and S. Kong (2022) Towards Long-Tailed 3D Detection. In 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand (en). Cited by: §2.3. [31] Z. Qin, J. Chen, C. Chen, X. Chen, and X. Li (2023-03) UniFusion: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird’s-Eye-View. arXiv (en). Note: arXiv:2207.08536 [cs] External Links: Link, Document Cited by: §1, §2.3, §2.3, §4. [32] S. S. Rajan, E. Soremekun, Y. Le Traon, and S. Chattopadhyay (2024-09) Distribution-aware fairness test generation. Journal of Systems and Software 215, p. 112090 (en). External Links: ISSN 01641212, Link, Document Cited by: §2.3. [33] T. Roddick and R. Cipolla (2020-03) Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks. arXiv (en). Note: arXiv:2003.13402 [cs] External Links: Link, Document Cited by: §1, §2.3, §2.3, §4. [34] J. Song, X. Chen, L. Lu, J. Li, and K. A. Skinner (2024-11) MemFusionMap: Working Memory Fusion for Online Vectorized HD Map Construction. arXiv (en). Note: arXiv:2409.18737 [cs]Comment: Accepted to WACV 2025 External Links: Link, Document Cited by: §2.1, §2.3, §3.2. [35] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov (2014-06) Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15, p. 1929–1958 (en). Cited by: §2.2. [36] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, S. Zhao, S. Cheng, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov (2020-05) Scalability in Perception for Autonomous Driving: Waymo Open Dataset. arXiv (en). Note: arXiv:1912.04838 [cs]Comment: CVPR 2020 External Links: Link, Document Cited by: §2.3. [37] R. Sun, L. Yang, D. Lingrand, and F. Precioso (2024-03) Mind the map! Accounting for existing map information when estimating online HDMaps from sensor. arXiv (en). Note: arXiv:2311.10517 [cs]Comment: 23 pages, 4 figures, 6 tables External Links: Link, Document Cited by: §2.1, §2.3, §3.2. [38] G. Valle-Pérez and A. A. Louis (2020-12) Generalization bounds for deep learning. arXiv (en). Note: arXiv:2012.04115 [stat] External Links: Link, Document Cited by: §2.2. [39] R. Wang, X. Lu, X. Liu, X. Zou, T. Cao, and Y. Li (2024-08) PriorMapNet: Enhancing Online Vectorized HD Map Construction with Priors. arXiv (en). Note: arXiv:2408.08802 [cs] External Links: Link, Document Cited by: §2.1, §2.3, §3.2. [40] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays (2023-01) Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. arXiv (en). Note: arXiv:2301.00493 [cs]Comment: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks External Links: Link, Document Cited by: §2.3, §4. [41] X. Xiong, Y. Liu, T. Yuan, Y. Wang, Y. Wang, and H. Zhao (2023-06) Neural Map Prior for Autonomous Driving. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, p. 17535–17544 (en). External Links: ISBN 9798350301298, Link, Document Cited by: §2.1, §2.3, §3.2. [42] T. Yuan, Y. Liu, Y. Wang, Y. Wang, and H. Zhao (2024-01) StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Construction. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, p. 7341–7350 (en). External Links: ISBN 9798350318920, Link, Document Cited by: Figure 1, Figure 1, §1, Table 6, Table 6, Table 9, Table 9, §2.1, §2.3, §2.3, §2.3, §3.2, §4, §5.1, §5.2, Table 1, Table 1, Table 1, Table 1, Table 2, Table 2, Table 2, Table 2, Table 3, Table 3. Supplementary Material 7 Visual Example of Sample Similarity Computation Following the definition of sim(v,t)sim(v,t) in Sec. 3.1, LABEL:fig:frechet_metric_vis visualizes its computation for an exemplary sample pair from the geographically disjoint near extrapolation nuScenes split introduced in [24]. 8 Metric Definitions In the following, we provide the mathematical definitions for the Chamfer distance used in the AP metric in Sec. 3.2 and the discrete Fréchet distance [7] used in sim(⋅,⋅)sim(·,·) in Sec. 3.1 and M in Sec. 3.2. 8.1 Chamfer Distance Let P,Q⊂ℝdP,Q ^d be two finite point sets, each representing one map element (polygon or polyline) after uniformly resampling it to NptsN_pts vertices. The Chamfer distance dchd_ch between P and Q is defined as dch(P,Q)=12(1|P|∑p∈Pminq∈Q‖p−q‖2+1|Q|∑q∈Qminp∈P‖q−p‖2). splitd_ch&(P,Q)=\\ & 12 ( 1|P| _p∈ P _q∈ Q\|p-q\|_2+ 1|Q| _q∈ Q _p∈ P\|q-p\|_2 ). split (16) 8.2 Discrete Fréchet Distance Let P and Q be two map elements represented as polygons or polylines in ℝdR^d, and let Figure 7: Visualization of the topology-based similarity measure simtopo(v,t)sim_topo(v,t) and its computation steps. Candidate rigid transformations are derived via class and map element-wise Procrustes analysis and discrete Fréchet distance matching. For each transformed training sample, similarity is evaluated within the overlapping field-of-view and penalized for low overlap. The final similarity is obtained by selecting the minimum across all candidates. σ(P)=(p1,…,pn),σ(Q)=(q1,…,qm)σ(P)=(p_1,…,p_n), 28.80008ptσ(Q)=(q_1,…,q_m) (17) be the corresponding sequences of uniformly resampled vertices, with pi,qj∈ℝdp_i,q_j ^d. A coupling L between P and Q is a sequence of distinct pairs between σ(P)σ(P) and σ(Q)σ(Q), L=((pa1,qb1),…,(paK,qbK)),L= ((p_a_1,q_b_1),…,(p_a_K,q_b_K) ), (18) where (ak)k=1K(a_k)_k=1^K and (bk)k=1K(b_k)_k=1^K are nondecreasing surjective index sequences, i.e. a1=1,aK=n,b1=1,bK=m, a_1=1,\;a_K=n,\;b_1=1,\;b_K=m, (19) a1,…,aK=1,…,n,b1,…,bK=1,…,m, \a_1,…,a_K\=\1,…,n\, \b_1,…,b_K\=\1,…,m\, (20) and for all r<sr<s, ar≤as,br≤bs.a_r≤ a_s, 28.80008ptb_r≤ b_s. (21) The norm ‖L‖\|L\| of a coupling L is the length of its longest pair, ‖L‖=maxk=1,…,K‖pak−qbk‖2.\|L\|= _k=1,…,K\|p_a_k-q_b_k\|_2. (22) The discrete Fréchet distance dfrd_fr between P and Q is then defined as dfr(P,Q)=min‖L‖|L is a coupling between P and Q.d_fr(P,Q)= \\|L\|\; |\;L is a coupling between P and Q \. (23) Figure 8: Correlation between the topology-based similarity stopo(v)s_topo(v) and the geographical distance d(v)d(v) for the nuScenes original split. Compared to the geometric similarity s(v)s(v), the topology-based score exhibits a stronger correlation and a less cluttered distribution (Pearson correlation coefficient r=0.785>0.724r=0.785>0.724, cf. Fig. 2). 9 Topology-Based Similarity Measure Motivation Besides examining the effect of geometrical similarity, we argue that online mapping models are capable of learning translation- and rotation-invariant features. We therefore introduce the similarity measure simtopo(v,t)sim_topo(v,t) to measure translation- and rotation-invariant similarity between samples, comparing their topological structures rather than the geometrical patterns in the field of view as in sim(v,t)sim(v,t). We base simtopo(v,t)sim_topo(v,t) on sim(v,t)sim(v,t) with a preceding alignment step. A visualization of the process to derive the topology-based similarity measure for an exemplary validation sample is shown in Fig. 7, we advise to follow the steps in the figure along with the mathematical definition below. Definition Let Ωv⊂ℝ2 _v ^2 and Ωt⊂ℝ2 _t ^2 denote the FOV of the validation sample v and the training sample t, respectively. We first obtain a finite set of candidate rigid transformations (v,t)=Tk(x)=Rkx+τk∣k=1,…,K,T(v,t)=\T_k(x)=R_kx+ _k k=1,…,K\, (24) by comparing the shapes of all map elements within the same class using Procrustes analysis (without uniform scaling) followed by measuring similarity using discrete Fréchet distance (cf. Sec. 8.2) and selecting the k top matches. Each transformation TkT_k is defined by a rotation matrix Rk∈SO(2)R_k (2) and a translation vector τk∈ℝ2 _k ^2. For a given candidate TkT_k, we transform the training sample tk:=Tk(t),t_k:=T_k(t), (25) and consider only the part of the scene that lies in the overlapping FOV Ωk:=Ωv∩Tk(Ωt). _k:= _v∩ T_k( _t). (26) We denote by sim(v,tk;Ωk)sim(v,t_k; _k) the original similarity simsim evaluated only on those map elements of v and tkt_k whose geometry lies inside or intersects Ωk _k. All elements outside Ωk _k are discarded and intersecting elements are clipped, since elements outside of Ωk _k cannot occur in the opposing sample. To avoid degenerate alignments through trivial solutions where the overlap between FOVs becomes very small and no map elements lie inside or intersect with Ωk _k, we penalize low-overlap candidates. We define the overlap ratio rk:=|Ωk||Ωv|∈[0,1]r_k\;:=\; _k _v ∈[0,1] (27) and add a penalty term that increases linearly as the overlap decreases. With a weight parameter λ≥0λ≥ 0, we define the topology-based similarity for a candidate transform TkT_k as simtopo∗(v,t;Tk):=sim(v,tk;Ωk)+λ(1−rk).sim _topo(v,t;T_k)\;:=\;sim(v,t_k; _k)\;+\;λ (1-r_k ). (28) Finally, the topology-based similarity between v and t is obtained by minimizing over all candidate transforms: simtopo(v,t):=minTk∈(v,t)simtopo∗(v,t;Tk).sim_topo(v,t)\;:=\; _T_k (v,t)sim _topo(v,t;T_k). (29) Analogous to s(v)s(v), we define stopo(v)s_topo(v) as the lowest similarity cost between v and any training sample t∈Tt∈ T with the new topological similarity measure across all candidate transforms stopo(v):=mint∈Tsimtopo(v,t).s_topo(v):= _t∈ Tsim_topo(v,t). (30) Results For this ablation study, we aim to validate our translation- and rotation-invariant similarity measure stopos_topo and compare it against s in terms of their correlation with performance. This allows us to determine whether overfitting is more strongly driven by geometric patterns or by topological structure. Figure 9: Comparison of geometric similarity s(v)s(v) and topology-based similarity stopo(v)s_topo(v) for the nuScenes original split. Samples deviating from the diagonal most often satisfy s(v)>stopo(v)s(v)>s_topo(v), demonstrating that stopo(v)s_topo(v) identifies topologically similar scenes that differ due to translation or rotation. Two exemplary validation samples are displayed along their best matches from the training set for s(v)s(v) and stopo(v)s_topo(v). We begin by examining the correlation between stopo(v)s_topo(v) and d(v)d(v) in the nuScenes original split in Fig. 8. As expected, the correlation is stronger compared to s(v)s(v), and the plot is visibly less cluttered (Pearson correlation coefficient r=0.785>0.724r=0.785>0.724, cf. Fig. 2). This is because samples that lie close to each other tend to share similar topological structure, whereas their geometric structure, which is sensitive to translation and rotation, differs more significantly. This suggests that the metric accurately captures topological structure rather than pure geometric alignment. To further substantiate this claim, we compare s(v)s(v) and stopo(v)s_topo(v) in Fig. 9. Several samples deviate from the bisector of the two axes, most often with s(v)>stopo(v)s(v)>s_topo(v), indicating that a closer topological match has been identified (cf. examples in Fig. 9). Figure 10: Correlation between the topology-based similarity stopo(v)s_topo(v) and the per-sample performance M(v)M(v) on the nuScenes original split. The slightly weaker correlation compared to s(v)s(v) (Pearson correlation coefficient r=0.552<0.568r=0.552<0.568, cf. Fig. 4) indicates that current models rely more strongly on geometric patterns than on topological structure. Dataset and split r(s(v),M(v))r(s(v),M(v)) r(stopo(v),M(v))r(s_topo(v),M(v)) nuScenes original 0.568 0.552 geo. [24] 0.226 0.211 geo. [42] 0.275 0.270 geometric 0.137 0.171 Argoverse 2 original 0.392 0.298 geo. [24] 0.433 0.366 geo. [42] 0.330 0.202 geometric -0.071 0.209 Table 3: Pearson correlation between M(v)M(v) and geometric similarity s(v)s(v) versus topology-based similarity stopo(v)s_topo(v) across all examined dataset splits. While s(v)s(v) correlates more strongly with performance for original and geographical splits, stopo(v)s_topo(v) shows mildly higher correlation for the geometric splits, where geometric similarity is intentionally minimized. We also reexamine the correlation between stopo(v)s_topo(v) and the per-sample performance M(v)M(v) on the original nuScenes split in Fig. 10. The Pearson correlation is slightly lower for stopo(v)s_topo(v) than for s(v)s(v) (r=0.552<0.568r=0.552<0.568, cf. Fig. 4), suggesting that the online mapping model relies more on rotation- and translation-dependent geometric features than on invariant topological features. To support this claim, we examine the correlation between s(v)s(v) and M(v)M(v) against the correlation between stopo(v)s_topo(v) and M(v)M(v) for all splits that are examined in Sec. 5. The results are shown in Tab. 3. While the Pearson correlation is in a similar range per split, we see slightly stronger correlation for the geometrical similarity measure s(v)s(v) across all original and geographical splits. However, in the geometrical splits we introduced, the correlation for stopos_topo is higher, even though both values indicate minimal correlation to M. This indicates, that for these validation sets where geometries are contained that are minimize similarity to the training split, topological alignment is still beneficial for model performance. Due to the lack of significant differences in the correlation, we expect similar results for any measures derived from stopo(v)s_topo(v) in place of s(v)s(v). Since the correlation with performance for all original and geographical splits is lower for stopo(v)s_topo(v) compared to s(v)s(v) and the effect of benefiting from topological alignment only seems to show for sets with very dissimilar geometries, we refrain from additional experiments. 10 Details on the Geometric Similarity-Based Performance Measure In the following, we provide additional details on M which is defined in Sec. 3.2. Our objective is to offer a clearer intuition for M and justify why it provides a more faithful assessment of geometric reconstruction quality than traditional AP-style metrics. We obtain class-wise bipartite matching based on the discrete Fréchet distance (cf. Sec. 8.2) between the prediction map elements pc,i∈Pcp_c,i∈ P_c and ground truth gc,i∈Gcg_c,i∈ G_c to find the closest prediction for each ground truth element. In contrast to the Chamfer distance, the discrete Fréchet distance is sensitive to point ordering (cf. Fig. 3). For all examined online mapping architectures in the experiments [23, 22, 27, 25], the predicted map elements have the same number of points compared to the ground truth. If the number of vertices per map element do not match, we advise to subsample or simplify to predicted map elements w.r.t. to the number of vertices in the ground truth, since the discrete Fréchet distance comparison between prediction and ground truth relies on discrete points and large distances between points could lead to inaccurate results. Note that similarly to the Chamfer-based AP metric, multiple predictions for the same ground truth element are not penalized in order to not introduce overwhelming complexity to the metric. In practice, we examine that the predicted map elements that are matched to a ground truth map element yield significantly higher confidence scores in comparison to the unmatched predictions, so the proposed measure could also be used to determine a suitable confidence threshold for deployment of a model. The visualization in LABEL:fig:frechet_metric_vis illustrates how the matching between two samples works for sim(v,t)sim(v,t), the same concept also applies to the measure M but instead of comparing two samples, we compare the sets of predictions and ground truth. Furthermore, instead of summing the matched costs resulting in amatcheda_matched, we aggregate all matched costs into a distribution D that represents the map-element-wise geometric reconstruction quality. We then take the median M and interquartile range IQRIQR to characterize D in a comparable way, which remains informative for sets of varying sizes, being less sensitive to discrete outcomes in comparison to the Chamfer distance-based AP metric and enabling comparisons at the single-sample level. 11 Geometrical Dataset Split Derivation Besides the original and geographical training splits, we want to examine model performance on a geometric dataset split, focusing on maximum geometric dissimilarity between training and evaluation sets. To be coherent with the other splits, the geometric split should also partition the data into 70/15/15% for the training, validation, and test sets, respectively. We base our geometric split on the geometric similarity MST for the whole dataset. At first, we identify the edges in the MST with highest similarity costs, suggesting highest geometric dissimilarity. We then consider these edges as first-cut candidates and retain those that yield a subset of size near the desired training set size (70% of the dataset size). For each retained first cut, we search for a second cut in the remaining opposite subset (30% of the dataset size) to separate it into the validation and test set (each 15% of the dataset size). We then evaluate the resulting three subsets against size tolerances, and score candidates by a balance–separation criterion (small size deviation, large cut edge weights). Finally, we remove the chosen two edges and assign the three connected subtrees to train/val/test sets. 12 Exact Numbers for Dataset Sparsification In the following, we list the exact numbers for Fig. 5 for the reader to reference. For each split, we report the sparsification threshold, remaining samples, MST length, and mAP values on the validation set corresponding to the plotted results. Sparsification Threshold Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) – 28130 (100.00 %) 96828.00 m (100.00 %) 60.95 (± 0.00 %) 0.1 23084 (82.06 %) 96815.33 m (99.99 %) 61.47 (+0.85 %) 0.2 22752 (80.88 %) 96805.15 m (99.98 %) 61.56 (+1.00 %) 0.5 21417 (76.14 %) 96613.71 m (99.78 %) 61.48 (+0.87 %) 1 18397 (65.40 %) 95585.00 m (98.72 %) 59.71 (-2.03 %) 2 13969 (49.66 %) 92051.27 m (95.07 %) 56.71 (-6.96 %) 5 6512 (23.15 %) 72116.01 m (74.48 %) 48.11 (-21.07 %) 10 2184 (7.76 %) 37363.45 m (38.59 %) 28.50 (-53.24 %) Table 4: Original nuScenes split Sparsification Threshold Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) – 27840 (100.00 %) 80572.21 m (100.00 %) 24.96 (± 0.00 %) 0.1 22815 (81.95 %) 80559.35 m (99.98 %) 25.73 (+3.08 %) 0.2 22427 (80.56 %) 80543.07 m (99.96 %) 25.48 (+2.08 %) 0.5 20381 (73.21 %) 80197.95 m (99.54 %) 25.64 (+2.72 %) 1 16515 (59.32 %) 78843.58 m (97.85 %) 26.46 (+6.01 %) 2 11970 (43.00 %) 74909.77 m (92.97 %) 25.07 (+0.44 %) 5 5575 (20.03 %) 57577.21 m (71.46 %) 23.62 (-5.37 %) 10 1697 (6.10 %) 26597.42 m (33.01 %) 15.59 (-37.54 %) Table 5: Geographically disjoint nuScenes split from [24] (Near Extrapolation Split) Sparsification Threshold Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) – 28008 (100.00 %) 90172.57 m (100.00 %) 28.53 (± 0.00 %) 0.1 22801 (81.41 %) 90159.43 m (99.99 %) 27.56 (-3.40 %) 0.2 22451 (80.16 %) 90148.35 m (99.97 %) 28.44 (-0.32 %) 0.5 20840 (74.41 %) 89897.52 m (99.69 %) 27.19 (-4.70 %) 1 17414 (62.18 %) 88718.11 m (98.39 %) 27.95 (-2.03 %) 2 12825 (45.79 %) 85090.37 m (94.36 %) 28.19 (-1.19 %) 5 6051 (21.60 %) 67000.17 m (74.30 %) 25.57 (-10.38 %) 10 1964 (7.01 %) 34123.62 m (37.84 %) 18.10 (-36.56 %) Table 6: Geographically disjoint nuScenes split from [42] Sparsification Threshold Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) – 22279 (100.00 %) 91003.65 m (100.00 %) 63.97 (± 0.00 %) 0.1 19892 (89.29 %) 90995.80 m (99.99 %) 64.32 (+0.55 %) 0.2 19580 (87.89 %) 90991.00 m (99.99 %) 64.97 (+1.56 %) 0.5 18633 (83.63 %) 90909.49 m (99.90 %) 63.71 (-0.41 %) 1 17159 (77.02 %) 90555.71 m (99.51 %) 64.22 (+0.39 %) 2 14440 (64.81 %) 88289.72 m (97.02 %) 63.40 (-0.89 %) 5 6882 (30.89 %) 66745.42 m (73.34 %) 59.42 (-7.11 %) 10 2116 (9.50 %) 31739.04 m (34.88 %) 45.72 (-28.53 %) Table 7: Original Argoverse 2 split, remaining samples in T and geomdiv(T)geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz. Sparsification Threshold Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) – 22223 (100.00 %) 87302.62 m (100.00 %) 49.53 (± 0.00 %) 0.1 19777 (88.99 %) 87294.12 m (99.99 %) 50.47 (+1.90 %) 0.2 19454 (87.54 %) 87289.24 m (99.98 %) 50.13 (+1.21 %) 0.5 18445 (83.00 %) 87188.20 m (99.87 %) 49.50 (-0.06 %) 1 16731 (75.29 %) 86747.95 m (99.36 %) 50.18 (+1.31 %) 2 13827 (62.22 %) 84389.69 m (96.66 %) 49.78 (+0.50 %) 5 6636 (29.86 %) 63978.80 m (73.28 %) 46.85 (-5.41 %) 10 1919 (8.64 %) 28972.13 m (33.19 %) 36.35 (-26.61 %) Table 8: Geographically disjoint Argoverse 2 split from [24] (Near Extrapolation Split), remaining samples in T and geomdiv(T)geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz. Sparsification Threshold Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) – 23954 (100.00 %) 97153.16 m (100.00 %) 57.61 (± 0.00 %) 0.1 21391 (89.30 %) 97144.47 m (99.99 %) 57.55 (-0.10 %) 0.2 21077 (87.99 %) 97138.62 m (99.99 %) 57.74 (+0.23 %) 0.5 20106 (83.94 %) 97050.71 m (99.89 %) 58.36 (+1.30 %) 1 18403 (76.83 %) 96625.41 m (99.46 %) 57.88 (+0.47 %) 2 15348 (64.07 %) 93994.73 m (96.75 %) 56.75 (-1.49 %) 5 7353 (30.70 %) 71206.04 m (73.29 %) 54.08 (-6.13 %) 10 2251 (9.40 %) 33621.14 m (34.61 %) 45.45 (-21.11 %) Table 9: Geographically disjoint Argoverse 2 split from [42], remaining samples in T and geomdiv(T)geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz. Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) 28130 (100.00 %) 96828.00 m (100.00 %) 60.95 (± 0.00 %) 23084 (82.06 %) 87906.66 m (90.79 %) 59.79 (-1.90 %) 22752 (80.88 %) 87715.78 m (90.59 %) 59.50 (-2.38 %) 21417 (76.14 %) 85104.96 m (87.89 %) 59.36 (-2.61 %) 18397 (65.40 %) 78757.05 m (81.34 %) 58.48 (-4.05 %) 13969 (49.66 %) 68450.78 m (70.69 %) 56.79 (-6.83 %) 6512 (23.15 %) 44951.67 m (46.42 %) 48.79 (-19.95 %) 2184 (7.76 %) 22131.76 m (22.86 %) 31.70 (-47.99 %) Table 10: Random sampling for remaining sample amounts from Tab. 4 from the original nuScenes split Remaining Samples in T geomdiv(T)geomdiv(T) Performance on V (mAP) 22279 (100.00 %) 91003.65 m (100.00 %) 63.97 (± 0.00 %) 19892 (89.29 %) 86989.97 m (95.59 %) 63.79 (-0.28 %) 19580 (87.89 %) 86184.47 m (94.70 %) 63.67 (-0.47 %) 18633 (83.63 %) 84021.93 m (92.33 %) 63.06 (-1.42 %) 17159 (77.02 %) 80187.60 m (88.11 %) 63.95 (-0.03 %) 14440 (64.81 %) 73394.62 m (80.65 %) 62.34 (-2.55 %) 6882 (30.89 %) 46629.68 m (51.24 %) 62.04 (-3.02 %) 2116 (9.50 %) 21827.41 m (23.98 %) 57.18 (-10.61 %) Table 11: Random sampling for remaining sample amounts from Tab. 7 from the original Argoverse 2 split, remaining samples in T and geomdiv(T)geomdiv(T) are computed for 2 Hz to be comparable to nuScenes. Performance results are for 10 Hz.