Paper deep dive

Exposing Blind Spots in Deep Imbalanced Regression Evaluation

Noah C. Puetz, Jens U. Brandt, Marc Hilbert, Elena Raponi, Thomas Bäck, Thomas Bartz-Beielstein

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 99

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/23/2026, 3:57:19 AM

Summary

This paper addresses three blind spots in Deep Imbalanced Regression (DIR) evaluation: domain bias towards images, lack of decision-complete metrics, and instability in tail regions. The authors introduce the MUVIS benchmark for multimodal time-series regression, propose balanced metrics (bMAE and bMASE) for robust evaluation, and demonstrate that DIR methods suffer from high sensitivity to random seeds in rare target regimes.

Entities (10)

Deep Imbalanced Regression · problem-domain · 98%MUVIS · datasetbenchmark · 97%bMAE · metric · 96%bMASE · metric · 96%RESNET1D · model-architecture · 92%Focal-L1 · algorithm · 90%LDS · algorithm · 90%RnC · algorithm · 90%UVote · algorithm · 90%PM10 · dataset-task · 88%

Relation Signals (8)

Deep Imbalanced Regression → hasblindspot → Data Blind Spot

confidence 95% · First, the Data Blind Spot: While DIR claims to be data domain agnostic its empirical ecosystem is still overwhelmingly concentrated in computer vision

Deep Imbalanced Regression → hasblindspot → Metric Blind Spot

confidence 95% · Second, the Metric Blind Spot: the established reporting practice in DIR is to partition the target space into many-, medium-, and few-shot regions

Deep Imbalanced Regression → hasblindspot → Stability Blind Spot

confidence 95% · Third, the Stability Blind Spot: current DIR evaluation pays limited attention to experimental stability.

bMAE → isproposedtoaddress → Metric Blind Spot

confidence 94% · Addressing the Metric Blind Spot, we formalize a macro-averaged, distribution-invariant view of DIR evaluation through bMAE

bMASE → isproposedtoaddress → Metric Blind Spot

confidence 94% · introduce the novel, scale-invariant bMASE... that is not doable with the current evaluation design.

MUVIS → isusedfor → Deep Imbalanced Regression

confidence 93% · we broaden the data domain by evaluating DIR on a multimodal virtual sensing benchmark (MUVIS)

DIR Methods → suffersfrom → Tail-Region Instability

confidence 92% · we show that the tail regions targeted by DIR exhibit particularly high sensitivity to seed-level variability.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Deep Imbalanced Regression (DIR) addresses a common failure mode of regression models: target distributions are highly non-uniform, causing models to perform best in densely populated target regions even when reliable performance is required across the full target range. Despite rapid methodological progress, DIR evaluation remains constrained by three blind spots: it is dominated by image-based benchmarks, its standard many-/medium-/few-shot protocol is diagnostic but not decision-complete, and tail-region stability across random seeds has not been systematically evaluated. We revisit DIR evaluation along these three axes. First, we broaden the data domain by evaluating DIR on a multimodal virtual sensing benchmark (\textsc{MuViS}) with nine time-series extrinsic regression tasks across six physical domains, where rare target values often correspond to operationally meaningful regimes. Second, we adopt balanced MAE (\emph{bMAE}) and introduce balanced Mean Absolute Scaled Error (\emph{bMASE}), a scale-normalized metric for decision-complete comparison across methods and datasets. Third, through a repeated reevaluation of six representative DIR methods across multiple random seeds, we show that the tail regions targeted by DIR exhibit particularly high sensitivity to seed-level variability. Our results show that standard virtual-sensing models exhibit substantial tail degradation hidden by global MAE, that existing DIR methods can improve balanced performance but transfer unevenly to multimodal time-series data, and that tail-region instability remains a largely hidden failure mode under current DIR evaluation practice. Together, these findings and our publicly available code provide a reproducible basis for future DIR research toward regression systems that capture rare target regimes as reliably as common ones.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

95,517 characters extracted from source content.

Expand or collapse full text

Exposing Blind Spots in Deep Imbalanced Regression Evaluation Noah C. Puetz 1,2 ∗ Jens U. Brandt 1,2 Marc Hilbert 2,3 Elena Raponi 2 Thomas Bäck 2 Thomas Bartz-Beielstein 1 1 TH Köln 2 Leiden University 3 Toyota Racing Abstract Deep Imbalanced Regression (DIR) addresses a common failure mode of regression models: target distributions are highly non-uniform, causing models to perform best in densely populated target regions even when reliable performance is required across the full target range. Despite rapid methodological progress, DIR evalu- ation remains constrained by three blind spots: it is dominated by image-based benchmarks, its standard many-/medium-/few-shot protocol is diagnostic but not decision-complete, and tail-region stability across random seeds has not been sys- tematically evaluated. We revisit DIR evaluation along these three axes. First, we broaden the data domain by evaluating DIR on a multimodal virtual sensing bench- mark (MUVIS) with nine time-series extrinsic regression tasks across six physical domains, where rare target values often correspond to operationally meaningful regimes. Second, we adopt balanced MAE (bMAE) and introduce balanced Mean Absolute Scaled Error (bMASE), a scale-normalized metric for decision-complete comparison across methods and datasets. Third, through a repeated reevaluation of six representative DIR methods across multiple random seeds, we show that the tail regions targeted by DIR exhibit particularly high sensitivity to seed-level vari- ability. Our results show that standard virtual-sensing models exhibit substantial tail degradation hidden by global MAE, that existing DIR methods can improve balanced performance but transfer unevenly to multimodal time-series data, and that tail-region instability remains a largely hidden failure mode under current DIR evaluation practice. Together, these findings and our publicly available code provide a reproducible basis for future DIR research toward regression systems that capture rare target regimes as reliably as common ones. Imbalanced Train Distribution Failure in Rare and Critical Regimes Silent Failure due to Global Metrics Heart Rate Estimation Figure 1: Motivating Example. In heart-rate estimation [1], the training distribution is strongly imbalanced (left), prediction quality degrades in rare but critical regimes (middle), and this failure is only weakly reflected by the global MAE (right). ∗ Contact: noah_christoph.puetz@th-koeln.de Preprint. arXiv:2609.25152v1 [cs.LG] 21 Sep 2026 1 Introduction In many real-world regression problems, the target variable is highly imbalanced. Some regions of the target space are densely populated with training samples, whereas others, often the most extreme and practically most consequential values, are observed only rarely [2,3]. When trained on such data, standard deep learning models naturally achieve their best predictive accuracy in well-supported regions of the training distribution and degrade in underrepresented ones [2,4,5]. In many applications, however, predictive performance is required not only where data are abundant, but across the entire target range [6]. Figure 1 provides such an example: In heart rate estimation most of the data is collected during normal operation of the human body [1], generating an imbalanced train distribution and leading to poor performance in the heart rate spikes, which ultimately results in worst performance in the most critical regimes. In such a scenario, the imbalanced training distribution becomes misaligned with the desired evaluation objective: rare outcomes receive little influence during optimization, despite being equally important at test time. Deep Imbalanced Regression (DIR) [2] addresses this mismatch and characterizes it as an implicit target distribution shift [3]. Motivated by this problem, recent years have seen a growing body of DIR methods that aim to improve performance in under-represented regions of the target space [3]. However, despite this methodological progress, the existing DIR evaluation design still suffers from three important blind spots. First, the Data Blind Spot: While DIR claims to be data domain agnostic its empirical ecosystem is still overwhelmingly concentrated in computer vision, while other domains like time-series settings remain largely absent (see Appendix C) [3]. Second, the Metric Blind Spot: the established reporting practice in DIR is to partition the target space into many-, medium-, and few-shot regions and report separate error values for each [2,3]. However, these partitions are arbitrary in continuous target spaces, and are difficult to interpret consistently across datasets [7]. More importantly, they do not produce a single aggregate quantity for ranking methods. As a result, comparisons remain partly subjective: a method may improve in underrepresented areas at the expense of well-represented ones, yet the field lacks a principled way to assess whether this trade-off is beneficial. Third, the Stability Blind Spot: current DIR evaluation pays limited attention to experimental stability. Methods are frequently compared using single runs or only a small number of repetitions, even though sparse supervision in the distribution tails potentially makes these regions especially vulnerable to variance from random initialization and optimization noise [8]. If DIR’s central claim is improved performance in underrepresented regions, then stability in those regions is part of what needs to be evaluated. In this work, we argue that meaningful DIR evaluation must go beyond the evaluation and reporting conventions that currently dominate the field. Therefore aligned with the three blind spots and supported by our motivating example in Figure 1 we revisit DIR from three complementary angles: 1.Addressing the Data Blind Spot, we broaden the DIR data domain to MUVIS (Multimodal Virtual Sensing Benchmark) [9], a collection of multimodal time-series extrinsic regression tasks specialized for virtual sensing and therefore characterized by a clear domain-level requirement for reliable performance across the full target range. 2.Addressing the Metric Blind Spot, we formalize a macro-averaged, distribution-invariant view of DIR evaluation through bMAE [6] and introduce the novel, scale-invariant bMASE. We contrast the prevailing many-/medium-/few-shot reporting with standard global regres- sion metrics such as mean absolute error (MAE), as well as with these balanced alternatives, and show that the bMASE creates an objective cross-method and cross-dataset ranking, that is not doable with the current evaluation design. 3.Addressing the Stability Blind Spot, we run repeated experiments to probe seed-level performance variability in DIR methods, focusing on whether underrepresented target regions exhibit disproportionately high performance variance. 2 Related Work The problem of imbalanced training data is not new in deep learning, but most imbalance-aware methods were developed for classification, where the output space is categorical and finite [10]. Extending these ideas to continuous and potential infinite target spaces is non-trivial. Yang et al.[2] introduced DIR as a distinct problem setting and showed that reweighting and resampling strategies transferred from classification perform poorly once the notion of a class boundary disappears. Since 2 Figure 2: Empirical target distributions of the nine MUVIS-DIR tasks, illustrating task-specific target imbalance across continuous sensor-like quantities. Legends report sequence lengthT, feature channels D, sample size n, skewness, and kurtosis. Domain icons follow Brandt et al. [9]. then, DIR has developed into a substantial methodological literature, with existing methods broadly grouped into three families [3]: algorithm-level methods that modify the loss or sample weighting (LDS [2], SqInv [2], Focal-L 1 [2], Balanced MSE [6], DenseLoss [11], VIR [12], Dist Loss [13]); representation-learning methods that reshape the latent space (FDS [2], RankSim [14], Ordinal Entropy [15], ConR [16], RnC [17], Geometric Representation Constraints [18], PRIME [19]); and ar- chitectural methods that modify the predictor itself (UVote [20], HCA [21], Multi-Classification [22], IM-Context [23]). Additionally, these methods can be divided into orthogonal approaches that compose with standard regression pipelines and non-orthogonal approaches that require specific architectures or substantially restructured training [3]. Despite DIR’s methodological progress, its empirical basis remains narrow. Across the fifteen DIR method papers surveyed by Puetz et al.[3], fourteen paper evaluate on at least one image-based benchmark, predominantly AGEDB-DIR [2,24] or IMDB-WIKI-DIR [2,25]; eight additionally evaluate on text regression (STS-B-DIR [2,26]); four on synthetic or tabular data; and only three on time-series data (full table shown in Appendix C). Moreover, each of these three evaluations use a different dataset with no overlap between papers (SHHS-DIR [27] in Yang et al.[2], TUAB [28] in Zha et al.[17], and ECG-K-DIR [29] in Nie et al.[13]) which prevents direct cross-method comparison. None of the three releases a public training pipeline for its respective time-series task, and two are difficult to reconcile with the continuous-target, deep-regression premise that motivates DIR: SHHS-DIR provides with1,892a limited amount of training samples with only21discrete target values, and TUAB provides1,246samples with integer targets in[0, 95]. As a result, current DIR evidence says far more about image benchmarks than about the broader class of imbalanced continuous prediction problems the field claims to address. A more complete evaluation landscape should include data domains that differ structurally from images while preserving the core DIR challenge: continuous targets with sparsely supported but practically relevant regions. 3 Extending the Evaluation Framework of DIR We extend DIR evaluation along the three blind spots identified in Section 1. First, in Subsection 3.1 we broaden the data domain by evaluating on multimodal time-series regression. Second, in Subsec- tion 3.2 we replace many-/medium-/few-shot reporting with balanced scalar metrics that are aligned with continuous targets and enable direct comparison across datasets. Third, we treat stability in underrepresented regions as an evaluation criterion and analyze it empirically in Subsection 4.3. 3.1 Benchmark Extension: MUVIS-DIR To contribute to closing the Data Blind Spot, we use MUVIS [9], a multimodal virtual sensing benchmark with nine continuous regression tasks across six physical domains. Virtual sensing denotes the data-driven inference of hard-to-measure physical quantities from available sensor measurements [30,31]. The task is sequence-to-value regression, at the intersection of virtual sensing 3 and multimodal dynamic time-series learning [32], with an interface to time-series extrinsic regression (TSER) [33]. Combining DIR and MUVIS yields the following problem setting for MUVIS-DIR: Problem setting of MUVIS-DIR: We consider a datasetD = (X i ,y i ) N i=1 , where each input X i = (x 1 i ,...,x M i )consists ofMmodality-specific time series withx j i ∈ R D j ×T j . Modalityj providesD j feature channels overT j time steps, andy i ∈ Rdenotes the continuous sensor-like target valuey i (t 0 )at reference timet 0 . A deep modelf θ maps the multimodal sequential input to an estimateˆy i = f θ (X i ). In the DIR setting, the empirical target distributionp train (y)is highly non-uniform: some target intervals are well-represented, whereas others contain few or no samples. The evaluation objective, however, is to generalize to a more balanced desired evaluation distribution p eval (y). This mismatch induces an implicit target-distribution shift between training and evaluation, weakening the standard empirical-risk-minimization (ERM) assumption that training and evaluation samples are drawn from the same target distribution. MUVIS-DIR is a useful DIR testbed because it combines continuous targets, heterogeneous sequen- tial inputs, and domain-level reasons to care about rare target values. In many virtual sensing tasks, underrepresented regions correspond to safety-critical [34,35], clinically relevant [1], or physically extreme operating [36,5] regimes rather than merely rare labels. Poor tail performance is therefore a practical failure mode of the trained model. Figure 2 visualizes the diverse target distributions used in our study; full dataset statistics, domain motivations for reliable performance across the target range, and imbalance summaries are reported in Appendix C. 3.2 Metric Extension: Balanced Scalar Evaluation 20406080 True value y 0.000 0.005 0.010 0.015 0.020 Density (a) Distribution & per-bin MAE All Many MedianFew (b) MAE per shot regions 1.1 4.3 1.1 4.9 1.0 10.6 0.8 14.7 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Per-bin MAE Mod. A Mod. B Mod. C Mod. D Many Median Few Figure 3: Failure mode of many-/medium- /few-shot reporting. (a) Four synthetic pre- dictors exhibit different per-bin error pro- files over the same imbalanced target distribu- tion. (b) Region-wise MAE produces differ- ent rankings depending on the selected shot re- gion and therefore does not provide a unique model comparison criterion. The dominant DIR protocol of Yang et al.[2]re- ports MAE separately over many-, medium-, and few-shot regions, which creates two problems for con- tinuous regression. First, the thresholds are inherited from long-tailed classification [7] (few-shot:< 20 samples, many-shot:> 100samples, medium-shot: otherwise) and are arbitrary once the target space is continuous. Second, the protocol produces three region-specific scores but no scalar ranking criterion. A model may improve rare regions while degrading dense ones, yet the protocol provides no principled rule for deciding whether this trade-off is favorable. The following motivating example illustrates this am- biguity. Motivating example.Figure 3 (a) shows four syn- thetic predictors with different error profiles over the same imbalanced target distribution. Model A re- sembles the typical ERM failure mode [3]: low error in the dense head and increasing error in the tails. Model C is nearly uniform across the target range and is visually the most robust predictor under a balanced evaluation objective. Model D exposes the ambiguity most clearly: it performs worst in the dense head but best in the medium- and few-shot regions, precisely the regimes DIR is meant to improve. Depending on the evaluation preference, Model D can therefore be viewed either as a poor model that sacrifices common cases or as a strong DIR model that improves rare regimes (Figure 3 (b)). The shot-based protocol re- ports this trade-off but does not resolve it. A naive average over the three shot scores would introduce another arbitrary decision rule, ranking Model D ahead of Model B (mean MAE: 2.13 vs. 2.88) without justifying how tail gains should be weighted against head degradation. Thus, the protocol can be diagnostic but not decision-complete: it reveals (arbitrary) regional behavior, yet it does not define an objective comparison criterion. 4 Motivated by this limitation, and by the premise that DIR evaluation should reward models that perform consistently well across an idealized balancedp eval (y)[3], we adopt balanced scalar metrics that weight target regions uniformly rather than by sample density. Sinceyis continuous, we follow Ren et al.[6]and approximate uniform weighting over the target range by discretizing [a,b]intoKequal-width bins; Subsection D.2 verifies the agreement between the continuous and discretized formulations for sufficiently large test sets. Balanced MAE. We macro-average the per-bin MAE over the K equal-width bins [6]: bMAE = 1 K K X k=1 1 n k X i∈bin k |y i − ˆy i |,(1) wheren k is the number of test samples in bink. Every bin receives equal weight regardless of sample count, reducing the dependence on the test-set density while preserving the interpretability of MAE in the original target units. Balanced MASE.While bMAE provides an interpretable absolute error, it is scale-dependent and cannot be compared across datasets. Because the nine datasets in our benchmark span fundamentally different physical quantities and scales, we introduce the balanced Mean Absolute Scaled Error (bMASE, based on the mean absolute scaled error by Hyndman and Koehler[37]), which normalises bMAE by the error of a trivial constant predictor: bMASE = 1 K P K k=1 1 n k P i∈bin k |y i − ˆy i | 1 K P K k=1 1 n k P i∈bin k |y i − c ref | = bMAE(f θ ) bMAE(c ref ) (2) wherec ref = median(y train )is theL 1 -optimal constant predictor. AbMASE < 1indicates improvement over the trivial predictor; values closer to0indicate stronger performance. Becausec ref normalises out the dataset-specific error scale, bMASE values are directly comparable across datasets and can be aggregated via the geometric mean: bMASE = D Y d=1 bMAE d (f θ ) bMAE d (c ref ) ! 1 D (3) The geometric mean is the only aggregation of normalised ratios that is invariant to the choice of reference, i.e. the ranking is identical whether one normalises by the trivial predictor, the best model, or any other baseline [38]. We useL 1 -based metrics because balanced evaluation gives equal weight to every target bin, including bins with very small sample counts. In such bins, squared-error metrics are especially unstable: per-bin MSE depends on fourth-order error moments, whereas MAE depends only on second-order moments through the variance of|e i |, withe i = y i − ˆy i [39]. Since metric stability in rare target regions is central to our setting, we use theL 1 -based bMASE rather than balanced variants of RMSE or R 2 . Both Equation 1 and Equation 2 require a target range[a,b]and a number of binsK. We set a = min(y test ) and b = max(y test ), and choose K using the Freedman–Diaconis rule [40], w FD = 2 IQR(y test )N −1/3 test , K = b− a w FD . Thus, the binning is deterministic, dataset-specific, shared by bMAE and bMASE, and not treated as a tunable hyperparameter. 4 Experiments We organize the experiments around the three evaluation blind spots introduced in Section 1. First, we test whether the DIR failure mode appears in MUVIS (Subsection 4.1). Second, we evaluate whether established DIR methods improve balanced performance under bMAE and bMASE (Subsection 4.2). Third, we analyze whether model performance in the same low-density regions is also less stable across random seeds (Subsection 4.3). Code for reproducing the experiments is available atwww. github.com/noah-puetz/muvis-dir. 5 Figure 4: Per-bin MAE of RESNET1D across three representative tasks from MUVIS [9]. Error increases sharply in sparse target regions; on PM10, the extreme upper tail exceeds the median- predictor baseline. Table 1: Contrasting classical MAE, many-/median-/few-shot MAE, and bMAE for each dataset with its corresponding RESNET1D configuration. Mean±95% bootstrap CI half-width on the mean over 10 seeds. The MAE vs. bMAE ratio quantifies how strongly global MAE underestimates balanced target-range error. Larger values indicate stronger imbalance-induced masking. MetricMAEMany-MAEMedi.-MAEFew-MAEbMAEMAE vs. bMAE Dataset Battery SoC0.0052± 0.00010.0052± 0.0001–0.0056± 0.00011.0636 Chem. Conc.0.0556± 0.00580.0554± 0.00590.1245± 0.01110.1480± 0.01640.0931± 0.00961.6747 Heart Rate4.1265± 0.11353.9351± 0.10846.3535± 0.19975.7599± 0.43165.1623± 0.15491.2510 Monterey0.1289± 0.03200.1113± 0.02440.2060± 0.06110.4445± 0.19580.4377± 0.18203.3955 PM1054.6721± 1.707938.0200± 0.572083.9907± 2.3227291.9299± 36.5234242.0897± 16.91074.4280 PM2.538.3933± 2.110028.5859± 1.372284.5718± 5.8982159.7854± 13.3461185.3354± 9.56534.8273 Targa ’130.0505± 0.00050.0468± 0.00040.0639± 0.00150.1109± 0.00480.0950± 0.00391.8812 Targa ’140.0725± 0.01340.0717± 0.01190.0788± 0.02410.0837± 0.02560.0777± 0.02141.0720 Tire Temp.3.1058± 0.2498–2.9534± 0.23424.8940± 0.45103.4308± 0.27481.1046 4.1 Imbalance Bias in Standard Virtual Sensors We first examine whether MUVIS-DIR exhibits the imbalance bias that motivates DIR [2]. For each dataset, we train the dataset-specific RESNET1D architectures provided by Brandt et al.[9]with anL 1 loss across ten random seeds. This provides a baseline for the later DIR comparison while avoiding architectural confounds. Results for the remaining MUVIS baselines under bMAE and bMASE are reported in Subsection G.1; among them, RESNET1D obtains the best aggregate bMASE and is therefore used as the basis for the main analysis. Figure 4 shows that the core DIR failure mode is present in MUVIS. Across the representative tasks, per-bin MAE increases in sparsely supported target regions, with the single exception of the per bin MAE on BATTERY SOC. BATTERY SOC has the least imbalanced target support among the shown tasks, and its per-bin MAE remains comparatively stable across the target range. Table 1 shows that the failure mode on PM10 is largely hidden by global MAE. Only when target regions are weighted equally by the bMAE, the RESNET1D obtains a4.42×increase. This illustrates why balanced metrics across target range are practically important: predictions on PM10 in the extreme upper tail, y > 600approach the performance of a naive median predictor, but these rare target ranges are also corresponding to critical air-quality regimes which would trigger public health advisories [35]. The MAE let the model appear competitive, yet it fails in the operating region where reliability is most consequential (similar masking effect appears on PM2.5 and MONTEREY, see Subsection G.2). The many-/median-/few-shot protocol is less reliable in this setting. Some datasets contain no bins satisfying a given shot criterion, producing undefined entries, and the fixed thresholds can still overweight denser subregions inside the nominal few-shot range. On PM2.5, the few-shot region spans approximatelyy = 300toy = 1000, but most samples within that region are concentrated near its lower end. Consequently, few-shot MAE underweights the most severe upper-tail errors, while bMAE exposes them. Thus, the shot-based reporting, meant for reporting models sensitivity due to data imbalance, can itself remain sensitive to within-region imbalance. 4.2 DIR Methods under Balanced Evaluation We next ask whether established DIR methods can reduce this imbalance bias. We evaluate six representative and state-of-the-art methods (based on the extrapolation benchmark from Puetz et al. 6 Table 2: Mean bMASE±95% bootstrap CI half-width over 10 seeds for each method–dataset pair. Below VANILLA are italicized; Best value in each row is bold. GMean aggregates across datasets via geometric mean. Lower is better; bMASE < 1 improves over the trivial median predictor. MethodVanillaConRFocal-L 1 LDSRnCSQInvUVote Dataset Battery SoC0.0229± 0.00030.0306± 0.00890.0280± 0.0051 0.0224± 0.00040.2536± 0.09770.0262± 0.00530.0251± 0.0008 Chem. Conc.0.6910± 0.07090.6485± 0.04830.6343± 0.0134 0.5826± 0.03250.6349± 0.01970.6371± 0.06730.6056± 0.0028 Heart Rate0.1277± 0.00380.1925± 0.0921 0.0748± 0.00280.1130± 0.00740.0923± 0.00520.1232± 0.02340.0782± 0.0039 Monterey0.2511± 0.10280.2695± 0.11620.2215± 0.07500.2401± 0.09620.3622± 0.26320.2260± 0.0907 0.0853± 0.0016 PM100.6869± 0.04830.6793± 0.02580.6171± 0.0094 0.4765± 0.02540.6663± 0.00850.6230± 0.02140.6792± 0.0353 PM2.50.5705± 0.02860.5507± 0.02890.5242± 0.01230.5489± 0.05410.5675± 0.02270.5457± 0.0287 0.5142± 0.0080 Targa ’130.1116± 0.00470.1136± 0.00250.1933± 0.00910.1212± 0.00390.3452± 0.18470.1132± 0.00290.1136± 0.0051 Targa ’140.1276± 0.03490.1319± 0.04090.1326± 0.00990.1273± 0.02830.1746± 0.03320.1183± 0.0195 0.1055± 0.0133 Tire Temp.0.3662± 0.02960.3618± 0.02300.3777± 0.02840.3534± 0.02610.3933± 0.03580.3505± 0.0221 0.3313± 0.0254 GMean0.2163± 0.01580.2338± 0.02720.2135± 0.01170.1997± 0.01480.3314± 0.03640.2084± 0.0202 0.1732± 0.0057 [3]) covering the main DIR families from Section 2: algorithm-level methods, LDS, SQInv, and Focal-L 1 ; representation-learning methods, ConR and RnC; and the architectural method UVote. LDS, SQInv, Focal-L 1 , and ConR are orthogonal interventions. All methods are trained on top of the same RESNET1D from Subsection 4.1, and each method–dataset pair is selected by a hyperparameter sweep (explained in detail in Subsection F.4). (a) (b) (c) Figure 5:Per-bin improvements over VANILLA. Top row compares per-bin MAE; bottom row shows the signed difference to the baseline. (Full Overview: Figure 16) Table 2 demonstrates the advantage of bMASE as a single scalar comparison criterion: it provides a cross-method and cross-dataset ranking that many-/medium-/few-shot reporting cannot supply. Four out of six methods improve the vanilla baseline, showing that imbalance correction transfers to mul- timodal virtual sensing. However, the gains are less regular than in common computer-vision DIR benchmarks. UVote achieves the best aggregate score, reducing GMean bMASE from0.2163to0.1732, and is best on four datasets. LDS is the strongest evaluated orthogonal method, reaching0.1997 GMean and producing the largest single improvement on PM10, from0.6869to0.4765. SQInv and Focal-L 1 also im- prove the aggregate score, whereas RnC substantially degrades performance. Figure 5 illustrates how these aggregate gains arise. Focal-L 1 improves high-heart-rate regions on PPGDALIA and UVote substantially reduces tail error on MONTEREY with both achieving smaller improvements across the whole respec- tive target range. In contrast LDS compromises in the well- represented region for better tail-performance on PM10. These results support two conclusions. First, the relationship between MUVIS and DIR is bidirectional: MUVIS exposes how well DIR methods transfer to multimodal time-series regression, while DIR methods yield meaningful gains in un- derrepresented sensor regimes. Second, direct transfer from existing DIR methods remains incomplete. Strong methods in image-based DIR do not necessarily dominate multimodal time-series regression, making MUVIS-DIR a realistic benchmark for future method development. 4.3 Tail Instability Across Seeds Finally, we evaluate the Stability Blind Spot. Since DIR methods are designed to improve low-density regions, their performance should be reproducible precisely in those regions. Motivated by prior work on random-seed variability in deep learning [41,42], we compute the standard deviation of per-bin MAE across ten seeds for every method–dataset pair. Rather than treating ten seeds as a sufficiency guarantee, we use this repeated-run setting to test whether instability is already visible in the target regimes DIR aims to improve. 7 Figure 6: Relationship between local target support and seed-level instability. Each point is one bin; thex-axis shows bin sample count (←less data) and they-axis shows the standard deviation of per-bin MAE across ten seeds (↑higher instability). Both axes are log-scaled. Gray denotes VANILLA; red denotes the best-performing DIR method according to Table 2 per dataset. Figure 6 shows, that across datasets, vanilla RESNET1D becomes less stable as local target support decreases, with Spearman correlations between bin count and per-bin standard deviation ranging from−0.57to−0.95. The same trend persists for the best-performing DIR methods: each method either shows an even stronger density–instability relation or differs from vanilla by less than0.05 correlation points. Figure 7: Seed-level instability for UVote on MONTEREY. Each point corresponds to one seed’s performance in one target bin; sparse tar- get regions exhibit visibly larger dispersion. This confirms that the regions central to DIR are also the least stable under repeated training. Im- portantly, improved mean performance does not appear to remove this instability. On MONTEREY, UVote substantially improves bMASE but retains a similar density-dependent variance trend as the vanilla model: the trend starts lower, but still rises as target support decreases. Figure 7 visualizes this effect directly, showing that seed-level vari- ability remains compact in dense target regions but widens in sparse regions. Additionally inter- esting: many method–dataset pairs exhibit an ap- proximately linear relationship on log–log axes, suggesting an inverse power-law-like scaling be- tween local target support and seed-level variance. We do not propose a new stability remedy here. Instead, we show that stability is a first-order evaluation concern for DIR. If the goal is to improve performance in underrepresented target regions, then reporting only average variance or from a single seed, is insufficient. Tail accuracy and tail stability should be evaluated jointly. 4.4 Discussion The experiments address the three blind spots identified in this paper. Subsection 4.1 establishes MUVIS-DIR as a natural extension of the DIR evaluation ecosystem: it provides multimodal time- series tasks with physically meaningful tail regimes, publicly available code, and a benchmark structure suitable for evaluating future DIR methods. Subsection 4.2 further shows that tail perfor- mance is improvable with existing DIR methods, but not consistently across the evaluated method set. This positions MUVIS-DIR not as a solved application of DIR, but as an open challenge broadening the current data landscape and by this addressing the Data Blind Spot. 8 Subsection 4.2 also demonstrates the practical value of bMASE. A single table is sufficient to compare methods across datasets, physical units, and imbalance regimes, while still preserving dataset-level detail. Most importantly, bMASE provides a scalar basis for ranking methods, addressing the ambiguity of many-/median-/few-shot reporting and therefore the Metric Blind Spot. Finally, Subsection 4.3 directly addresses the Stability Blind Spot in the current DIR evaluation ecosystem and confirms that performance instability across seeds in underrepresented regions is not only present but amplified. We do not claim this as a new optimization phenomenon; rather, we show that it has been largely absent from DIR evaluation despite being central to the problem. 5 Limitations and Future Work This study has several limitations. First, while MUVIS-DIR broadens DIR evaluation beyond image- centric benchmarks, it is not intended to cover the full space of time-series regression problems. We deliberately focus on virtual sensing as a multimodal, sensor-based sequence-to-value setting because it isolates the transfer of DIR methods to time-series inputs while retaining a single continuous target. Forecasting and sequence-to-sequence prediction introduce additional structure: imbalance may depend not only on the target value, but also on where rare regimes occur along the prediction horizon. Extending the analysis to these settings is therefore an important direction for future work. Second, our empirical comparison is necessarily selective. We evaluate six representative DIR methods spanning algorithm-level, representation-learning, and architectural approaches, but the DIR literature is growing rapidly. Our results therefore should not be read as a definitive ranking of all DIR methods, but as evidence that existing methods transfer unevenly to multimodal time-series data. Future work should expand the benchmark to newer methods and combinations of orthogonal interventions. Third, bMAE and bMASE depend on a discretization of the target range. We mitigate this choice by using a shared data-driven binning rule and show in Subsection D.2 that the resulting estimates agree closely with a continuous kernel-based alternative. Still, balanced evaluation necessarily requires a choice about local target resolution. Developing equally interpretable continuous alternatives with explicit uncertainty estimates remains an open direction. Finally, our stability analysis is diagnostic rather than corrective. We show that low-density target regions exhibit higher seed-level variability, but our scope was not to propose a method that explicitly optimizes for stable tail performance. This opens a promising direction for future DIR methods: improving performance in underrepresented regions while also reducing variance across random initialization, data order, and optimization trajectories. 6 Conclusion We revisited DIR from an evaluation perspective and identified three blind spots in the current empirical ecosystem. The first is a Data Blind Spot: DIR evaluation remains concentrated in image- based benchmarks, leaving structurally different regression domains underexplored. The second is a Metric Blind Spot: many-/medium-/few-shot reporting is useful diagnostically, but does not provide a decision-complete scalar criterion for comparing methods. The third is a Stability Blind Spot: the rare target regions that motivate DIR are rarely evaluated for reproducibility across random seeds. To address these issues, we introduced MUVIS-DIR, a multimodal virtual sensing benchmark with continuous time-series regression tasks and practically meaningful tail regimes and we introduced bMASE, a scale-normalized balanced metric that enables cross-method and cross-dataset comparison. Finally, we analyzed DIR methods performance stability as a function of local target support. Our experiments show that standard deep learning models can exhibit substantial tail degradation that is masked by global MAE, that existing DIR methods can improve balanced performance but transfer unevenly to multimodal time-series data, and that tail regions are also the least stable across seeds. These findings do not suggest that any single method solves MUVIS-DIR. Rather, they show that evaluating DIR requires broader data domains, balanced scalar metrics, and explicit attention to tail reliability. 9 References [1] Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. Deep PPG: Large- Scale Heart Rate Estimation with Convolutional Neural Networks. Sensors, 19(14):3079, January 2019. ISSN 1424-8220. doi: 10.3390/s19143079. URLhttps://w.mdpi.com/ 1424-8220/19/14/3079. [2]Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into Deep Imbalanced Regression, May 2021. URLhttp://arxiv.org/abs/2102.09554. arXiv:2102.09554 [cs]. [3]Noah C. Puetz, Jens U. Brandt, Marc Hilbert, Elena Raponi, Thomas Bäck, and Thomas Bartz- Beielstein. Deconstructing deep imbalanced regression: a comprehensive review and experi- mental evaluation. Artificial Intelligence Review, 59(6):135, April 2026. ISSN 1573-7462. doi: 10.1007/s10462-026-11570-1. URL https://doi.org/10.1007/s10462-026-11570-1. [4] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, May 2014. ISBN 978-1-107-05713-5. [5] Zhongwei Zhang, Erich Fischer, Jakob Zscheischler, and Sebastian Engelke. Physics-based models outperform AI weather forecasts of record-breaking extremes. Science Advances, 12 (18):eaec1433, April 2026. doi: 10.1126/sciadv.aec1433. URLhttps://w.science.org/ doi/10.1126/sciadv.aec1433. [6]Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced MSE for Imbalanced Visual Regression. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7916–7925, New Orleans, LA, USA, June 2022. IEEE. ISBN 978-1-6654- 6946-3. doi: 10.1109/CVPR52688.2022.00777. URLhttps://ieeexplore.ieee.org/ document/9879350/. [7]Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-Scale Long-Tailed Recognition in an Open World, April 2019. URLhttp://arxiv. org/abs/1904.05160. arXiv:1904.05160 [cs]. [8]Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. Accounting for Variance in Machine Learning Benchmarks. March 2021. doi: 10.48550/arXiv. 2103.03098. [9] Jens U. Brandt, Noah C. Puetz, Jobel Jose George, Niharika Vinay Kumar, Elena Raponi, Marc Hilbert, Thomas Bäck, and Thomas Bartz-Beielstein. MuViS: Multimodal Virtual Sensing Benchmark, March 2026. URLhttp://arxiv.org/abs/2603.24602. arXiv:2603.24602 [eess]. [10]Haibo He and Edwardo A. Garcia. Learning from Imbalanced Data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284, September 2009. ISSN 1558-2191. doi: 10.1109/TKDE.2008.239.URLhttps://ieeexplore.ieee.org/document/5128907. Conference Name: IEEE Transactions on Knowledge and Data Engineering. [11]Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density-based weighting for imbalanced regression. Machine Learning, 110(8):2187–2211, August 2021. ISSN 1573-0565. doi: 10.1007/s10994-021-06023-5. URLhttps://doi.org/ 10.1007/s10994-021-06023-5. [12]Ziyan Wang and Hao Wang. Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing, October 2023. URLhttp://arxiv.org/abs/2306.06599. arXiv:2306.06599 [cs, stat]. [13]Guangkun Nie, Gongzheng Tang, and Shenda Hong. Dist Loss: Enhancing Regression in Few-Shot Region through Distribution Distance Constraint, March 2025. URLhttp://arxiv. org/abs/2411.15216. arXiv:2411.15216 [cs]. 10 [14]Yu Gong, Greg Mori, and Frederick Tung. RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, June 2022. URLhttp://arxiv.org/abs/2205.15236. arXiv:2205.15236 [cs]. [15]Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving Deep Regression with Ordinal Entropy, February 2023. URLhttp://arxiv.org/abs/2301. 08915. arXiv:2301.08915 [cs]. [16] Mahsa Keramati, Lili Meng, and R. David Evans. ConR: Contrastive Regularizer for Deep Im- balanced Regression, 2023. URLhttps://arxiv.org/abs/2309.06651. Version Number: 4. [17]Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-N-Contrast: Learning Continuous Representations for Regression, October 2023. URLhttp://arxiv.org/abs/ 2210.01189. arXiv:2210.01189 [cs]. [18]Zijian Dong, Yilei Wu, Chongyao Chen, Yingtian Zou, Yichi Zhang, and Juan Helen Zhou. Improve Representation for Imbalanced Regression through Geometric Constraints, March 2025. URL http://arxiv.org/abs/2503.00876. arXiv:2503.00876 [cs]. [19] Jongin Lim, Sucheol Lee, Daeho Um, Sung-Un Park, and Jinwoo Shin. PRIME: Deep Im- balanced Regression with Proxies. In Proceedings of the 42nd International Conference on Machine Learning, pages 37705–37729, Vancouver, Canada, October 2025. PMLR. URL https://proceedings.mlr.press/v267/lim25a.html. [20]Yuchang Jiang, Vivien Sainte Fare Garnot, Konrad Schindler, and Jan Dirk Wegner. Uncertainty Voting Ensemble for Imbalanced Deep Regression, October 2024. URLhttp://arxiv.org/ abs/2305.15178. arXiv:2305.15178 [cs]. [21]Haipeng Xiong and Angela Yao. Deep Imbalanced Regression via Hierarchical Classification Adjustment. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23721–23730, June 2024. doi: 10.1109/CVPR52733.2024.02239. URLhttps: //ieeexplore.ieee.org/document/10655108. [22]Dekun Lin, Tailai Peng, Rui Chen, Xinran Xie, and Zhe Cui. Let Multi-classification Help Deep Imbalanced Regression. In Michael Wand, Kristína Malinovská, Jürgen Schmidhuber, and Igor V. Tetko, editors, Artificial Neural Networks and Machine Learning – ICANN 2024, pages 430–447, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-72338-4. doi: 10.1007/978-3-031-72338-4_29. [23]Ismail Nejjar, Faez Ahmed, and Olga Fink. IM-Context: In-Context Learning for Im- balanced Regression Tasks, October 2024. URLhttp://arxiv.org/abs/2405.18202. arXiv:2405.18202 [cs]. [24]Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. AgeDB: The First Manually Collected, In-the-Wild Age Database. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1997–2005, July 2017. doi: 10.1109/CVPRW.2017.250. URLhttps: //ieeexplore.ieee.org/document/8014984. [25]Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks. International Journal of Computer Vision, 126(2):144–157, April 2018. ISSN 1573-1405. doi: 10.1007/s11263-016-0940-3. URL https://doi.org/10.1007/s11263-016-0940-3. [26]Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://aclanthology.org/S17-2001/. 11 [27]S. F. Quan, B. V. Howard, C. Iber, J. P. Kiley, F. J. Nieto, G. T. O’Connor, D. M. Rapoport, S. Redline, J. Robbins, J. M. Samet, and P. W. Wahl. The Sleep Heart Health Study: design, rationale, and methods. Sleep, 20(12):1077–1085, December 1997. ISSN 0161-8105. [28]Denis A. Engemann, Apolline Mellot, Richard Höchenberger, Hubert Banville, David Sabbagh, Lukas Gemein, Tonio Ball, and Alexandre Gramfort. A reusable benchmark of brain-age prediction from M/EEG resting-state signals. NeuroImage, 262:119521, November 2022. ISSN 1095-9572. doi: 10.1016/j.neuroimage.2022.119521. [29]Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-Wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, January 2023. ISSN 2052-4463. doi: 10.1038/s41597-022-01899-x. [30]P.F. Muir. A virtual sensor approach to robot kinematic identification: theory and experimental implementation. September 1990. ISBN 978-0-7803-0173-3. doi: 10.1109/ICSYSE.1990. 203189. Pages: 445. [31]Dominik Martin, Niklas Kühl, and Gerhard Satzger. Virtual Sensors. Business & Infor- mation Systems Engineering, 63(3):315–323, June 2021. ISSN 1867-0202. doi: 10.1007/ s12599-021-00689-w. URL https://doi.org/10.1007/s12599-021-00689-w. [32]Payal Mohapatra, Yueyuan Sui, Akash Pandey, Stephen Xia, and Qi Zhu. MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series, September 2025. URL http://arxiv.org/abs/2509.25278. arXiv:2509.25278 [cs]. [33]Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Monash University, UEA, UCR Time Series Extrinsic Regression Archive, October 2020. URLhttp: //arxiv.org/abs/2006.10996. arXiv:2006.10996 [cs]. [34]Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):20170457, September 2017. ISSN 1364-5021. doi: 10.1098/rspa.2017.0457. URL https://doi.org/10.1098/rspa.2017.0457. [35]Yuyu Chen, Avraham Ebenstein, Michael Greenstone, and Hongbin Li. Evidence on the impact of sustained exposure to air pollution on life expectancy from China’s Huai River policy. Proceedings of the National Academy of Sciences, 110(32):12936–12941, August 2013. doi: 10. 1073/pnas.1300018110. URL https://w.pnas.org/doi/10.1073/pnas.1300018110. [36] J. J. Downs and E. F. Vogel. A plant-wide industrial process control problem. Computers & Chemical Engineering, 17(3):245–255, March 1993. ISSN 0098-1354. doi: 10.1016/ 0098-1354(93)80018-I.URLhttps://w.sciencedirect.com/science/article/ pii/009813549380018I. [37]Rob J. Hyndman and Anne B. Koehler. Another look at measures of forecast accuracy. Inter- national journal of forecasting, 22(4):679–688, 2006. URLhttps://w.sciencedirect. com/science/article/pii/S0169207006000239. [38]Philip J. Fleming and John J. Wallace. How not to lie with statistics: the correct way to summarize benchmark results. Commun. ACM, 29(3):218–221, March 1986. ISSN 0001-0782. doi: 10.1145/5666.5673. URL https://dl.acm.org/doi/10.1145/5666.5673. [39]Peter J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1):73–101, March 1964. ISSN 0003-4851. doi: 10.1214/aoms/1177703732. URL http://projecteuclid.org/euclid.aoms/1177703732. [40]David Freedman and Persi Diaconis. On the histogram as a density estimator:L2 theory. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 57(4):453–476, December 1981. ISSN 1432-2064. doi: 10.1007/BF01025868. URLhttps://doi.org/10.1007/ BF01025868. 12 [41]Nils Reimers and Iryna Gurevych. Reporting Score Distributions Makes a Difference: Per- formance Study of LSTM-networks for Sequence Tagging. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1035. URL https://aclanthology.org/D17-1035/. [42] Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments, July 2018. URLhttp:// arxiv.org/abs/1806.08295. arXiv:1806.08295 [cs]. [43]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection, February 2018. URLhttp://arxiv.org/abs/1708.02002. arXiv:1708.02002 [cs]. [44]Jens U. Brandt, Noah C. Pütz, Marcus Greiff, Thomas Jonathan Lew, John Subosits, Marc Hilbert, and Thomas Bartz-Beielstein. From Faults to Features: Pretraining to Learn Robust Representations against Sensor Failures. October 2025. URLhttps://openreview.net/ forum?id=9aElHWiZ72. [45]John C. Kegelman, Lene K. Harbott, and J. Christian Gerdes. Insights into vehicle tra- jectories at the handling limits: analysing open data from race car drivers. Vehicle Sys- tem Dynamics, 55(2):191–207, February 2017. ISSN 0042-3114. doi: 10.1080/00423114. 2016.1249893.URLhttps://doi.org/10.1080/00423114.2016.1249893._eprint: https://doi.org/10.1080/00423114.2016.1249893. [46]D. Mori, R. K. Aggarwal, N.D. Broadbent, T. Kobayashi, and J. Christian Gerdes. Vehicle Dynamics Dataset for Highly Dynamic Automated Driving, 2025. [47] Phillip Kollmeyer. Panasonic 18650PF Li-ion Battery Data. 1, June 2018. doi: 10.17632/ wykht8y7tg.1. URL https://data.mendeley.com/datasets/wykht8y7tg/1. [48]Pratik Mondal, Divyakumar Bhavsar, Kanupriya Mittal, and Mayank Mittal. Estimating State-of-Charge in Lithium-Ion Batteries Through Deep Learning Techniques: A Comparative Evaluation. IEEE Access, P:1–1, January 2024. doi: 10.1109/ACCESS.2024.3408220. [49]Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset Shift in Machine Learning. January 2009. ISSN 0262170051, 9780262170055. [50]Rita Ribeiro and Nuno Moniz. Imbalanced regression and extreme value prediction. Machine Learning, 109:1–33, September 2020. doi: 10.1007/s10994-020-05900-9. [51] Bernard W. Silverman. Density Estimation for Statistics and Data Analysis. Routledge, Boca Raton, February 2018. ISBN 978-1-315-14091-9. doi: 10.1201/9781315140919. A Appendix This appendix is organized as follows. Appendix B lists the mathematical notation used throughout the paper. Appendix C, Appendix D, and Appendix E provide additional analysis for the data-, metric-, and stability blind spots, respectively. Appendix F provides instructions for reproducing the experi- ments from Section 4 and Appendix G contains supplementary results and figures for Section 4. The publicly accessible code for the paper can be found atwww.github.com/noah-puetz/muvis-dir. 13 B Mathematical Notations Table 3: Mathematical notation used throughout the paper. SymbolMeaning Problem setting DDataset of input–target pairs. NNumber of samples in a dataset. When referring specifically to the test set for binning, we use N test . iSample index, with i∈1,...,N. X i Multimodal sequential input for sample i. x j i Time series of modality j for sample i. MNumber of input modalities. jModality index, with j ∈1,...,M. D j Number of feature channels in modality j. T j Number of time steps in modality j. y i Continuous scalar target value for sample i. y i (t 0 )Target value at reference time t 0 . t 0 Reference time at which the virtual sensor target is defined. f θ Deep regression model with parameters θ. θTrainable model parameters. ˆy i Model prediction for sample i, i.e. ˆy i = f θ (X i ). e i Prediction error for sample i, defined as e i = y i − ˆy i . p train (y)Empirical target distribution in the training set. p eval (y)Desired evaluation distribution over the target space, typically more balanced than p train (y). Balanced evaluation metrics [a,b]Target range used for balanced evaluation. In this work, a = min(y test ) and b = max(y test ). aLower endpoint of the test target range. bUpper endpoint of the test target range. KNumber of equal-width target bins used for bMAE and bMASE. bin k Set of test-sample indices whose target values fall into bin k. kTarget-bin index, with k ∈1,...,K. n k Number of test samples in bin k, i.e. n k =|bin k |. y train Collection of training target values. y test Collection of test target values. c ref Constant reference predictor used for scaling bMASE; here c ref = median(y train ). MAEMean absolute error under the empirical test distribution. bMAE Balanced mean absolute error, obtained by macro-averaging per-bin MAE across the target range. MASEMean absolute scaled error. bMASEBalanced mean absolute scaled error, defined as bMAE(f θ )/bMAE(c ref ). bMAE d (·)bMAE on dataset d. dDataset index used when aggregating results across datasets. DNumber of benchmark datasets included in the aggregate score. bMASEGeometric mean of bMASE values across datasets. Binning rule w FD Bin width given by the Freedman–Diaconis rule. IQR(y test )Interquartile range of the test targets. ⌈·⌉Ceiling operator. Stability analysis sRandom seed index. SNumber of random seeds. In the experiments, S = 10. σ k Standard deviation of per-bin MAE across seeds for bin k. ρ Spearman Spearman rank correlation between bin sample count and seed-level per-bin instability. C Data Blind Spot Appendix This appendix supplements the data-blind-spot discussion in Section 1 along two complementary axes. Subsection C.1 provides a comprehensive snapshot of the DIR methodological landscape 14 and architectures they were originally evaluated on, and indicating which combinations come with public training code. The purpose is to make explicit how narrow the empirical evidence behind current DIR claims actually is, and why a domain-extended benchmark such as MUVIS is needed. Subsection C.2 then characterises MUVIS itself, both statistically (target imbalance, effective support of the training distribution) and operationally (why uniform performance across the target range is a domain requirement, not just a statistical preference). C.1 DIR Methods and Dataset Landscape Table 4 makes the data blind spot quantitatively visible: 18 of the 19 surveyed methods include AgeDB-DIR or IMDB-WIKI-DIR as their primary evaluation, and only a small subset of works ventures beyond facial-image regression. Table 4: Overview of DIR methods, datasets, and backbone architectures. Datasets and models in bold indicate that public training code is available for the corre- sponding combination. “Algo.” stands for algorithm-level methods, “Repr.” for representation-learning methods, and “Archit.” for architectural methods. The Orth. column indicates whether the method is orthogonal to the backbone archi- tecture and can be combined with other methods. TypeMethodVenueDatasetsModelsOrth.Repo Algo.LDS [2]ICML 2021IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR NYUD2-DIR SHHS-DIR ResNet-50 BiLSTM+GloVe CNN-RNN ✓GitHub Balanced MSE [6] CVPR 2022IMDB-WIKI-DIR NYUD2-DIR IHMR ResNet-50 SPIN ✓ GitHub VIR [12]NeurIPS 2023IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR NYUD2-DIR ResNet-50 BiLSTM+GloVe ✗ GitHub DenseLoss [11]Mach. Learn. 2021 Synthetic DataMLP✓ GitHub Focal-R [2, 43]ICML 2021IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR NYUD2-DIR SHHS-DIR ResNet-50 BiLSTM+GloVe CNN-RNN ✓GitHub INV& SQINV [2] ICML 2021IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR NYUD2-DIR SHHS-DIR ResNet-50 BiLSTM+GloVe CNN-RNN ✓ GitHub RRT [2]ICML 2021IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR NYUD2-DIR SHHS-DIR ResNet-50 BiLSTM+GloVe CNN-RNN ✓ GitHub Dist Loss [13]ICLR 2025IMDB-WIKI-DIR AgeDB-DIR ECG-K-DIR ResNet-50 Net1D ✓GitHub Repr.FDS [2]ICML 2021IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR NYUD2-DIR SHHS-DIR ResNet-50 BiLSTM+GloVe CNN-RNN ✓ GitHub RankSim [14]ICML 2022IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR ResNet-50 BiLSTM+GloVe ✓GitHub Continued on next page 15 Table 4 – continued from previous page TypeMethodVenueDatasetsModelsOrth.Repo ConR [16]ICLR 2024IMDB-WIKI-DIR AgeDB-DIR NYUD2-DIR MPIIGaze-DIR ResNet-50 LeNet ✓ GitHub RnC [17]NeurIPS 2023IMDB-WIKI-DIR AgeDB-DIR TUAB MPIIGaze-DIR SkyFinder ResNet-18 ResNet-50 ✗ (✓) GitHub Geom. Rep. [18]CVPR 2025UCI-DIR AgeDB-DIR IMDB-WIKI-DIR STS-B-DIR ResNet-50 BiLSTM+GloVe MLP ✓ GitHub Ordinal Ent. [15] ICLR 2023Synthetic data NYU-Depth-v2 AgeDB-DIR SHTech MLP NeW-CRFs ResNet-50 DeepONet CSRNet ✓ GitHub PRIME [19]ICML 2025AgeDB-DIR IMDB-WIKI-DIR NYUD2-DIR STS-B-DIR ResNet-50 BiLSTM+GloVe ✓— Archit. UVOTE [20]GCPR 2024IMDB-WIKI-DIR AgeDB-DIR Wind STS-B-DIR ResNet-50 ResNet-18 BiLSTM ✗ GitHub HCA [21]CVPR 2024IMDB-WIKI-DIR AgeDB-DIR NYUDv2-DIR SHTech ResNet-50 VGG16 ✗ GitHub Multi-Class. [22] ICANN 2024IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR ResNet-50 BiLSTM+GloVe ✗— Im-Context [23]TMLR 2024IMDB-WIKI-DIR AgeDB-DIR STS-B-DIR Boston Concrete Abalone Communities Kin8nm Airfoil GPT2 PFN ✗GitHub C.2MUVIS Dataset Overview We characterise the MUVIS datasets along two complementary dimensions. The imbalance ratio (IR) summarises the contrast between dense head and sparse tails of the training target distribution, while the effective supportS eff quantifies how much of the available target range the training distribution actually uses. Together, the two metrics distinguish distributions that are similarly skewed but differ in coverage, and are reported alongside conventional moments (skew, excess kurtosis) in Table 5. Figure 2 additionally visualises all nine target distributions. Imbalance ratio. The imbalance ratio is defined as the ratio of sample density in the head of the training distribution to sample density in its tails: IR = d head d tail ,(4) 16 where, d head = i : Q 25 ≤ y i ≤ Q 75 Q 75 − Q 25 , d tail = i : y i < Q 10 + i : y i > Q 90 (Q 10 − y min ) + (y max − Q 90 ) ,(5) whereQ p denotes thep-th percentile of the training targets. The quantile thresholds determine which samples count as head or tail, by construction the head always contains50%of training samples and the two tails together20%, while the densitiesd head andd tail are measured in target-space units, matching the equal-width binning thatbMAEandbMASEapply at evaluation time. A uniform target distribution yieldsIR = 1. For sharply peaked distributions the20%of tail samples are spread over a wider target range, driving d tail down and IR up. Effective support. While IR captures the head-to-tail contrast, two distributions with identical IR can still differ substantially in how much of the target range they cover (e.g. a heavily skewed distribution versus a bimodal one). The effective supportS eff provides a single global measure of this coverage. We fit a Gaussian KDE with Silverman bandwidth to the training targets, evaluate it on a grid ofGequally spaced points spanning the target range, normalise to a discrete distributionq, and define: S eff = exp H(q) G , H(q) = − G X g=1 q g logq g ,(6) whereH(q)is the Shannon entropy ofq. Intuitively,S eff answers the question what fraction of the target range is effectively occupied by the training distribution? A value of1corresponds to a perfectly uniform distribution; lower values indicate that the mass is concentrated in a smaller fraction of the range.S eff is the continuous analogue of the “effective number of classes” notion from imbalanced classification. Because the smoothing is controlled by the KDE bandwidth rather than by G, S eff is stable across discretisation choices (< 0.6% variation over a 20× range of G). Dataset statistics.Table 5 reports both metrics together with sample sizes, sequence lengthT, input channelsD, and conventional distributional moments. The nine MUVIS tasks span more than an order of magnitude in IR (from 2.0 on Battery SoC to 41.2 on Monterey) and a corresponding range inS eff (from0.92down to0.25). This range is a deliberate property of the benchmark: it allows DIR methods to be tested across the full spectrum from near-uniform to extremely peaked target distributions, rather than within the narrow band covered by the predominantly Gaussian facial-age benchmarks. Table 5: MUVIS dataset statistics and imbalance characterisation.T: sequence length;D: input channels; IR: imbalance ratio (↑= more imbalanced);S eff : effective support (↓= more concentrated). Tasks ordered by decreasing IR. Datasetn train n test T DSkewKurt.IR↑ S eff ↓ Monterey120,77721,3572022 +0.203.2741.20.25 Targa ’1333,52011,1092022 +0.351.9626.00.32 Targa ’1446,7399,0362022 −0.180.1616.80.43 PM 2.5 11,9185,048249 +2.096.8615.60.42 PM 10 11,9185,048249 +2.216.8712.90.44 Chem. Conc.240,500470,5002033 −0.01 −0.0512.10.45 Tire Temp.1,3842805011 +0.18 −0.608.90.55 Heart Rate51,75712,9405126 +0.900.858.00.60 Battery SoC199,827158,1261207 −0.20 −1.012.00.92 C.2.1 Domain motivation for tail-region reliability The statistical imbalance summarised above is necessary but not sufficient to motivate balanced evaluation. What makes MUVIS a meaningful DIR benchmark is that, for every task, the distributional tails correspond to operationally critical regimes where predictive failures carry disproportionate consequences. 1. PM 2.5 / PM 10 . The Beijing Multi-Site Air Quality data [34] pose a virtual sensing task in which the target is particulate-matter concentration and the inputs combine pollutant and 17 meteorological measurements. Extreme concentrations exceed air-quality-index thresholds that trigger public-health advisories and emergency traffic restrictions [35]. 2.Monterey / Targa ’13 / Targa ’14. The Revs Program Vehicle Dynamics Database integrates multimodal sensing sources (driver inputs, wheel and chassis measurements, GNSS-aided inertial navigation). Following Brandt et al.[44], we define a virtual sensing task of estimating the vehicle’s lateral velocityv y , a key state for stability assessment and control that is typically not directly available from low-cost on-board sensing. Extreme values correspond to near-limit handling regimes marking the onset of tire-grip loss and vehicle instability. These are precisely the safety-critical conditions that electronic stability control systems are designed to navigate [45]. 3.Tire Temp. MUVIS additionally includes a high-performance autonomous-driving dataset that records vehicle state from RTK-GPS alongside control inputs, actuator states, and the target real-time tire temperature [46]. Extreme surface temperatures indicate accelerated tread degradation and elevated blowout risk; accurate tail prediction is therefore essential for high-performance autonomous applications [9]. 4.Chem. Conc. Tails flag abnormal operating states corresponding to the 21 benchmarked fault scenarios of the Tennessee Eastman process [36]. A model that fails to predict these extremes effectively fails to detect process faults. 5.Battery SoC. Extremes represent overcharge and deep-discharge regimes where cell degra- dation accelerates and safety margins narrow. Prediction bias at these boundaries can trigger premature or delayed charge termination [47, 48]. 6. Heart Rate. Extremes indicate exercise-induced tachycardia or arrhythmia events that are clinically significant yet inherently rare in ambulatory recordings [1]. In each case, an evaluation protocol that down-weights tail performance in proportion to its empirical frequency, as standard MAE andR 2 implicitly do, would systematically reward models that fail in exactly the regimes the underlying application cares most about. D Metric Blind Spot Appendix This appendix provides the empirical analysis underlying the metric choices in Subsection 3.2. We study (i) the distribution invariance of standard and balanced regression metrics, and (i) the discrete-versus-continuous estimation question for the bMAE numerator that underlies bMASE. D.1 Distribution invariance of standard and balanced metrics In this section we analyse the sensitivity of different evaluation metrics to the test distribution and show why several widely used metrics are not aligned with the evaluation objective of DIR. Puetz et al.[3]describes the fundamental challenge in DIR as a “mismatch between the training objective and the evaluation goal”: the very act of addressing imbalance implies that we value uniform performance across all target regions more than their representation in the training data would suggest, which is equivalent to a distribution shift in the target space and therefore breaks the standard i.i.d. assumption [3,4,49]. Building on this line of argument, our goal is a single scalar metric that ranks models consistently regardless of the test-set composition. The experiment below shows how a metric’s sensitivity to the test distribution can shift both model rankings and performance estimates, and demonstrates that bMAE and bMASE are robust against this effect. D.1.1 Experimental setup To isolate metric properties from model properties, we construct four deterministic synthetic models whose error profile is a fixed function of the target value. Given a test sample with true targety, we define d =|y− μ train |/(0.4 (b− a)) and s = sign(y− μ train ), and study: • A (tail collapse): ˆy = y + (0.5 + 10d 2 )s • B (uniformly mediocre): ˆy = y + (2.5 + 0.3d 2 )s • C (uniformly good): ˆy = y + (1.0 + 0.3d 2 )s 18 20406080 y 0.00 0.01 0.02 0.03 0.04 0.05 Density Laplace (symmetric) 20406080 y Gaussian (symmetric) 20406080 y Weibull (right-skewed) 0 5 10 15 20 25 30 Per-bin MAE Model A Model B Model C Model D 0.0 0.2 0.4 0.6 0.8 Figure 8: Experimental Setup wiht all three distributions and the corresponding error profiles of the four synthetic models overlaid • D (tail specialist): ˆy = y + (0.5 + 10e −3d )s The four models cover the failure modes most relevant to DIR: a typical empirical-risk-minimisation profile that collapses in the tails (A), a uniformly mediocre baseline (B), a uniformly strong predictor (C), and a tail-specialist that sacrifices head accuracy in exchange for better tail behaviour (D). Because the errors are fully deterministic functions ofy, any variation in a metric across test conditions is attributable purely to the metric’s sensitivity to the test-set distribution rather than to randomness in the predictions. For each experiment, test sets are parametrised byα∈ [0, 0.95]: atα = 0the test set is drawn uni- formly over the target range, while atα = 0.95it concentrates around the corresponding distribution. We test three target distribution families: Gaussian (symmetric), Laplace (symmetric, heavy-tailed), and Weibull (right-skewed). The cap atα = 0.95guarantees a minimum of5%uniform samples, which prevents empty edge bins and avoids the discontinuities that occur when a test set has zero support in parts of the target range. All three distributions and the corresponding error profiles can be seen in Figure 8. For each metric we report (i) the coefficient of variationCV = std/|mean|across theαsweep (lower=more stable) and (i) ranking stability: the number of distinct model orderings observed and Kendall’s τ between the rankings at α = 0 and α = 0.95. D.1.2 Results Figure 9 shows the values of seven candidate metrics across theαsweep on the Weibull distribution the most challenging setting due to its heavy right skew. Standard MAE, MSE,R 2 , the geometric mean of absolute errors (GM; Yang et al.[2]), and SERA [50] all drift withαfor at least one model, indicating that the reported metric value depends on the empirical test-set composition rather than only on the model itself. The balanced metrics (bMAE and bMASE) remain flat across α. Table 6 quantifies these observations across all three distribution families by computing the per-model CV across theαsweep and the resulting ranking stability. SERA, despite its threshold-free design, is among the least stable metrics (per-model CV up to0.515on Weibull and0.452on Gaussian). This arises because SERA computes an unnormalised sum of squared errors at each relevance threshold; when the test distribution shifts, the sample count above each threshold changes, conflating test-set composition with model quality. The proposed balanced metrics achieve near-perfect invariance: per-model CV≤ 0.005across all distributions and exactly one unique ranking across allαvalues, withτ (0↔ 0.95) = +1.00. Figure 10 additionally visualises theα-sweep curves for Laplace and Gaussian targets to confirm that the qualitative pattern of Figure 9 holds in the symmetric setting. We emphasise that the analysis in Figure 9, Figure 10, and Table 6 is not intended as a blanket judgement against MAE, MSE,R 2 , GM, or SERA. Each of these metrics has well-established uses and remains appropriate in many regression settings. What we report is specifically the behaviour that becomes problematic under the uniform-performance objective of DIR: when the goal is to characterise model quality consistently across the entire target range, sensitivity to test-set composition 19 (0 = uniform, 0.95 = peaked) 2 4 6 8 10 MAE MAE (0 = uniform, 0.95 = peaked) 0 50 100 150 200 MSE MSE (0 = uniform, 0.95 = peaked) 0.70 0.75 0.80 0.85 0.90 0.95 1.00 R² R² (0 = uniform, 0.95 = peaked) 1 2 3 4 GM GM (0 = uniform, 0.95 = peaked) 0 100000 200000 300000 400000 SERA SERA (0 = uniform, 0.95 = peaked) 2 4 6 8 bMAE bMAE 0.000.250.500.75 (0 = uniform, 0.95 = peaked) 0.05 0.10 0.15 0.20 0.25 0.30 0.35 bMASE bMASE Model A Model B Model C Model D Figure 9: Metric values across test-distribution shifts (α = 0: uniform;α = 0.95: Weibull-peaked) for the four deterministic models. Standard metrics (top) and SERA drift substantially withα, while the balanced metrics (bMAE,bMASE) remain flat and produce a single stable ranking. is undesirable, and our experiments show that the balanced metrics are the only candidates that meet this criterion. D.2 Theoretical grounding and continuous estimation of bMAE The bMAE used in both Equation 1 and in bMASE (Equation 2) is a discrete estimator of the conditional absolute error integrated under the uniform measure over the target range: bMAE(f θ ) ≈ 1 b− a Z b a E |Y − ˆ Y| Y =y dy .(7) The uniform measure 1 b−a dyensures that every point in the target range contributes equally to the integral, independent of the empirical test-set density. The binned estimator in Equation 1 approximates this integral via the midpoint quadrature rule, with each bin’s per-sample MAE serving as the function value at that bin’s centre. A continuous alternative is to estimate the integrand with Nadaraya–Watson (NW) kernel regression and integrate over a uniform grid: \ bMAE NW = 1 G G X g=1 P N i=1 K h (y g − y i )|y i − ˆy i | P N i=1 K h (y g − y i ) ,(8) whereK h is a Gaussian kernel with bandwidthhchosen by Silverman’s rule [51] andG = 500 uniformly spaced grid points. The grid densityGis a numerical integration parameter rather than a statistical one; results are stable from G≈ 100 onward. Figure 11 compares the two estimators across seven test-set sizes and three distribution families at α = 0.95. ForN test ≥ 2000, the discrete and continuous estimators agree to within1–5%on bMAE and to within0.01on bMASE across all three families. The agreement tightens monotonically with sample size, reaching|∆ bMASE| < 0.004atN test = 10,000. The bMASE agreement is somewhat tighter than the raw bMAE agreement because numerator and denominator share the same binning, which partially cancelsK-induced bias in the ratio. The right column of Figure 11 confirms this 20 (0 = uniform, 0.95 = peaked) 1 2 3 4 5 6 MAE MAE (0 = uniform, 0.95 = peaked) 0 10 20 30 40 50 MSE MSE (0 = uniform, 0.95 = peaked) 0.70 0.75 0.80 0.85 0.90 0.95 1.00 R² R² (0 = uniform, 0.95 = peaked) 1 2 3 4 5 GM GM (0 = uniform, 0.95 = peaked) 0 20000 40000 60000 80000 100000 SERA SERA (0 = uniform, 0.95 = peaked) 1 2 3 4 5 bMAE bMAE 0.000.250.500.75 (0 = uniform, 0.95 = peaked) 0.05 0.10 0.15 0.20 0.25 bMASE bMASE Model A Model B Model C Model D (0 = uniform, 0.95 = peaked) 1 2 3 4 5 6 MAE MAE (0 = uniform, 0.95 = peaked) 0 10 20 30 40 50 MSE MSE (0 = uniform, 0.95 = peaked) 0.80 0.85 0.90 0.95 1.00 R² R² (0 = uniform, 0.95 = peaked) 1 2 3 4 GM GM (0 = uniform, 0.95 = peaked) 0 20000 40000 60000 80000 100000 SERA SERA (0 = uniform, 0.95 = peaked) 1 2 3 4 5 bMAE bMAE 0.000.250.500.75 (0 = uniform, 0.95 = peaked) 0.05 0.10 0.15 0.20 0.25 bMASE bMASE Model A Model B Model C Model D Figure 10: Same analysis as Figure 9 for the Laplace (top) and Gaussian (bottom) target distributions. The stability pattern of the balanced metrics is consistent across distribution families. 21 Table 6: Distribution invariance of evaluation metrics across the three target distribution families. We report the per-model CV across theαsweep (lower=more stable), the number of distinct model rankings observed acrossα∈ [0, 0.95], and Kendall’sτbetween the rankings at the endpoints. The proposed balanced metrics (bMAE, bMASE) maintain near-perfect invariance across all three settings. CV across α↓ # Unique↓ τ(0↔0.95)↑ Distribution MetricABCD Gaussian MAE0.285 0.013 0.161 0.0303+0.60 MSE0.424 0.026 0.225 0.0642+0.80 R 2 0.013 0.008 0.058 0.0012+0.80 GM0.293 0.013 0.213 0.0294+0.40 SERA0.452 0.221 0.133 0.2542+0.80 bMAE0.002 0.000 0.002 0.0001+1.00 bMASE 0.001 0.002 0.004 0.0021+1.00 Laplace MAE0.305 0.014 0.195 0.0323+0.60 MSE0.408 0.028 0.279 0.0662+0.80 R 2 0.008 0.009 0.087 0.0012+0.80 GM0.343 0.013 0.251 0.0314+0.40 SERA0.425 0.259 0.130 0.2872+0.80 bMAE0.003 0.000 0.003 0.0011+1.00 bMASE 0.001 0.002 0.005 0.0021+1.00 Weibull MAE0.379 0.026 0.173 0.0583+0.60 MSE0.506 0.054 0.229 0.1292+0.80 R 2 0.078 0.005 0.040 0.0012+0.80 GM0.363 0.024 0.260 0.0523+0.40 SERA0.515 0.270 0.153 0.3332+0.80 bMAE0.002 0.000 0.002 0.0001+1.00 bMASE 0.001 0.001 0.003 0.0011+1.00 visually: at the largest sample size, the discrete bin averages track the smooth NW estimate of the local MAE function across the entire target range for all three distributions. We therefore adopt the discrete estimator as the default throughout this work. It is simpler to implement, transparent in its assumptions, and produces identical model rankings to the continuous estimator on all settings considered. WhyL 1 rather than squared-error balanced metrics. Balanced metrics macro-average errors across target bins, so bins with few samples contribute as much as densely populated bins. This makes the stability of each per-bin estimate important. For a sample-mean estimator 1 n P i Z i , the variance isVar(Z i )/n. For MAE,Z i = |e i |, whose variance depends onE[e 2 i ]. For MSE,Z i = e 2 i , whose variance depends onE[e 4 i ]. Squared-error estimates are therefore more sensitive to heavy-tailed errors and outliers, especially whenn k is small [39]. This motivates our use of bMASE as anL 1 -based balanced metric. E Stability Blind Spot Appendix In addition to the stability analysis based on target imbalance, we characterise the label noise present in each MUVIS task, because label noise and target imbalance interact and confound evaluation. In low-density regions the empirical conditional expectationE[y | x]is estimated with higher variance. In sensor-based domains this interaction is compounded by physics: extreme operating conditions often coincide with higher measurement uncertainty due to sensor saturation or nonlinear transduction (e.g. IEC 60584-1 thermocouple tolerances scale with the measured value). Characterising the noise landscape is therefore a prerequisite for interpreting DIR results: methods that up-weight rare samples simultaneously amplify the unreliable gradients those samples carry [3]. 22 10 2 10 3 10 4 N test 0 5 10 15 20 25 30 35 40 Relative error (%) bMAE agreement Gaussian Max rel. error Mean rel. error 1% threshold 10 2 10 3 10 4 N test 10 2 10 1 | bMASE| bMASE agreement Gaussian Max | bMASE| Mean | bMASE| 0.005 threshold 20406080 Target value y 0 2 4 6 8 10 12 14 16 Local MAE( y ) Model A Gaussian (N=10000) Continuous (NW) Discrete (K=53) 10 2 10 3 10 4 N test 0 5 10 15 20 Relative error (%) bMAE agreement Laplace Max rel. error Mean rel. error 1% threshold 10 2 10 3 10 4 N test 10 3 10 2 | bMASE| bMASE agreement Laplace Max | bMASE| Mean | bMASE| 0.005 threshold 20406080 Target value y 0 2 4 6 8 10 12 14 16 Local MAE( y ) Model A Laplace (N=10000) Continuous (NW) Discrete (K=74) 10 2 10 3 10 4 N test 0 10 20 30 40 50 60 Relative error (%) bMAE agreement Weibull Max rel. error Mean rel. error 1% threshold 10 2 10 3 10 4 N test 10 3 10 2 10 1 | bMASE| bMASE agreement Weibull Max | bMASE| Mean | bMASE| 0.005 threshold 20406080 Target value y 0 5 10 15 20 25 30 35 Local MAE( y ) Model A Weibull (N=10000) Continuous (NW) Discrete (K=52) Continuous vs discrete bMAE estimation (= 0.95) Figure 11: Agreement between the continuous (Nadaraya–Watson) and discrete (binned, Freedman– Diaconis) bMAE estimators across test-set sizes and distribution families (α = 0.95). Left: maximum relative error of the discrete estimator on bMAE. Centre: maximum absolute|∆ bMASE|. Right: local MAE curves for Model A atN test = 10,000, with the smooth NW estimate (blue line) overlaid on discrete bin averages (green bars). Method.For each training sample(x i ,y i )we identify itsk = 20nearest neighbours in PCA-reduced feature space and compute the normalised noise ratio r i = Var y j : j ∈N k (i) σ 2 (y) ,(9) whereN k (i)is the index set of theknearest neighbours ofx i andσ 2 (y)is the global variance of the training targets. A ratior≈ 0indicates a clean, locally deterministic mapping;r≈ 1means that local target disagreement matches the global spread;r > 1signals a multi-valued or strongly heteroscedastic regime. We bin per-sample ratios across the target range and report the median and IQR per bin (Figure 12). 23 0200400600800 0 500 1000 1500 Count PM2.5 02004006008001000 0 250 500 750 1000 1250 1500 PM10 42024 0 10000 20000 30000 Monterey 21012 0 2000 4000 6000 8000 10000 Count Targa '13 210123 0 2000 4000 6000 8000 10000 Targa '14 5060708090100110 0 25 50 75 100 125 Tyre Temp. 4.64.74.84.95.05.1 Target value 0 5000 10000 15000 20000 25000 30000 Count Chem. Conc. 0.00.20.40.60.81.0 Target value 0 2000 4000 6000 8000 10000 Battery SoC 406080100120140160180 Target value 0 1000 2000 3000 4000 5000 6000 Heart Rate 0.0 0.5 1.0 1.5 0 2 4 6 0 1 2 3 Noise ratio Sample count IQR Median noise ratio Baseline (ratio=1) 0 1 2 0 1 2 3 0.5 1.0 1.5 Noise ratio 0.6 0.8 1.0 0.00 0.25 0.50 0.75 1.00 0 1 2 3 4 Noise ratio Figure 12: Normalisedk-N noise ratio (Equation 9) as a function of target value for all datasets in MUVIS. Blue bars: sample count per bin (left axis). Orange line and band: median noise ratio with IQR (right axis). Dashed line: random baseline (r = 1). Results. The noise profiles reveal systematic co-occurrence of imbalance and noise (Figure 12). The severely imbalanced tasks exhibit the most pronounced tail noise: Monterey shows a U-shaped profile rising fromr≈ 0.05in the centre tor> 1in both tails, reflecting the multi-valued physics of near-limit handling; PM 10 displays an asymmetric pattern with clean lower values (r ≈ 0.1) but extreme upper-tail noise (r> 3.5), consistent with sporadic, meteorologically driven pollution episodes; the Chem. Conc. shows a milder U-shape that remains belowr = 1. The moderately imbalanced tasks (Hart Rate, Tire Temp.) exhibit more uniform profiles, while battery SoC maintains r< 0.02 throughout, confirming an almost deterministic mapping. This pattern carries a direct methodological consequence: DIR methods that up-weight the tails of the severely imbalanced tasks must contend with amplified gradient noise alongside data scarcity, whereas on near-uniform tasks like battery SoC any tail error is purely a capacity problem, which could be the ideal scenario for DIR corrections. The noise profiles thus refine the imbalance-only view of Table 5: imbalance ratio and effective support quantify how skewed the target distribution is, while Figure 12 quantifies how much of the residual tail-region error a model can in principle eliminate. Together, the two characterisations explain why uniform improvements across all MUVIS tasks are difficult to achieve: the tasks for which DIR methods are most needed are also the tasks on which their corrections are most likely to amplify noise rather than correct bias. F Implementation and Reproducibility This section summarizes the implementation choices required to reproduce the experiments in Section 4 alongside with the public available code atwww.github.com/noah-puetz/muvis-dir. All methods use the same datasets, train/test splits, ResNet1D backbone family, evaluation metrics, and seed protocol unless stated otherwise. All experiments were implemented in Python 3.13.7 and executed on a single NVIDIA H100 (80 GB) GPU with CUDA 13.2. F.1 Data, Splits, and Preprocessing All experiments use the nine MUVIS-DIR datasets introduced in the main text. The preprocessed train/test files are inherited from Brandt et al.[9]; the DIR experiments do not introduce additional upstream preprocessing or target transformations. Targets are predicted in their original physical units. 24 Table 7: Dataset-specific ResNet1D backbone used in the Section 4 experiments. The architecture parameters are inherited from the original MUVIS ResNet1D configuration. Here,Tdenotes sequence length and C the number of input channels. DatasetFiltersFC unitsBlocksDropout 1Dropout 2 TTarget PM1015651240.1540.02824PM10 PM2.52566440.0300.06024PM2.5 Heart Rate25651240.0090.004512heart rate Battery SoC25612820.1650.00005120state of charge Chem. Conc.25625640.0930.09520chemical concentration Tire Temp.3251240.0600.01450front-right tire temperature Monterey2563240.0360.0005320lateral velocity Targa ’132563240.0360.0005320lateral velocity Targa ’142563240.0360.0005320lateral velocity For each random seed, the provided training split is further divided into a training and validation subset using a stratified random 90/10 split. The test set remains fixed across all seeds and is used only for final evaluation. Input features are standardized channel-wise using statistics estimated on the training subset only; the same transformation is then applied to validation and test inputs. Targets are not standardized. F.2 Backbone and Training Protocol All methods are built on the ResNet1D architecture selected in the original MUVIS hyperparameter optimization. We keep the dataset-specific architecture parameters fixed and vary only the DIR method and its associated hyperparameters. The retained architecture parameters are shown in Table 7. Unless stated otherwise, models are trained with anL 1 regression loss, Adam optimization, no weight decay, no gradient clipping, and a stepwise learning-rate decay by a factor of0.1at epochs 60 and 80. Training runs for 120 epochs, and the checkpoint with the lowest validation MAE is restored before test evaluation. This shared protocol applies to Vanilla, LDS, SQInv, Focal-L 1 , and ConR. UVote and RnC use method-specific schedules, described below. F.3 DIR Method Configurations The evaluated DIR methods follow the formulations described in the main text and are adapted to multimodal time-series regression. LDS, SQInv, and Focal-L 1 modify the regression loss through target-density- or residual-dependent weighting. ConR and RnC add contrastive objectives on the learned representation. UVote replaces the single regression head with an ensemble of expert heads trained with an uncertainty-aware objective. Table 8 summarizes the method-specific hyperparameters considered during configuration selection. F.4 Hyperparameter Selection Hyperparameters are selected separately for each dataset–method pair. The selection is performed at seed 42 before the final multi-seed evaluation. For methods with a small number of predefined variants, both variants are trained and compared. For Focal-L 1 , ConR, and RnC, we additionally run the method-specific sweeps listed in Table 8. The selected configuration is the one with the best validation balanced MAE at seed 42. After this selection step, the chosen configuration for each dataset–method pair is fixed and retrained across all ten seeds. After hyperparameter selection, one configuration is fixed for each dataset–method pair and reused for all ten seeds in the final evaluation. Table 9 and Table 10 report the effective method-specific parameters used in these final runs. Parameters that are shared across all datasets, such as the ResNet1D backbone, optimizer, training length, and validation-based checkpoint selection, are described in Subsection F.2 and are not repeated here. Vanilla uses the shared training protocol without additional method-specific parameters. 25 Table 8: Method-specific hyperparameters used for configuration selection. Parameters not listed here follow the shared training protocol from Subsection F.2. MethodHyperparameterValues considered LDSnumber of bins50, 100 smoothing kernelGaussian kernel size5, 9 smoothing bandwidth2.0 reweighting ruleinverse or square-root inverse frequency SQInvnumber of bins50, 100 Focal-L 1 β0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0 γ1, 2 ConRcontrastive weight 0.5, 1.0, 2.0, 4.0 relative label window 0.01, 0.02, 0.05, 0.10 of the training target range temperature0.2 RnCcontrastive temperature 0.1, 2.0 stage-1 learning rate 0.05, 0.1, 0.3, 0.5 stage-1 epochs400 stage-2 epochs90 UVotenumber of experts 2, 3 training schedule90 epochs, batch size 64 Table 9: Selected configurations for the loss- and reweighting-based DIR methods. For LDS,B denotes the number of target bins,kthe Gaussian smoothing kernel size, andrthe reweighting rule. LDS uses Gaussian smoothing with bandwidth σ = 2.0 in all cases. For SQInv, B denotes the number of target bins. For Focal-L 1 , β and γ denote the focal-loss parameters. DatasetLDSSQInvFocal-L 1 PM10B = 50, k = 9, r = inverse B = 50 β = 0.01, γ = 1 PM2.5B = 50, k = 9, r = inverse B = 50 β = 0.5, γ = 1 Heart Rate B = 100, k = 5, r = sqrt-inv B = 100 β = 0.05, γ = 1 Battery SoC B = 50, k = 9, r = inverse B = 100 β = 0.2, γ = 2 MontereyB = 50, k = 9, r = inverse B = 100 β = 0.2, γ = 2 Targa ’13B = 50, k = 9, r = inverse B = 50 β = 0.2, γ = 2 Targa ’14B = 100, k = 5, r = sqrt-inv B = 100 β = 5.0, γ = 1 Chem. Conc. B = 100, k = 5, r = sqrt-inv B = 100 β = 0.5, γ = 1 Tire Temp. B = 50, k = 9, r = inverse B = 100 β = 0.5, γ = 1 F.5 Seeds and Statistical Reporting The final experiments are run with ten seeds: 42, 43, 44, 45, 46, 47, 48, 49, 50, 51. Each seed controls the train/validation split, model initialization, data shuffling, dropout, and worker- level randomness in the data-loading pipeline. We also set deterministic backend options where available. However, strict bitwise determinism is not enforced, since some GPU operations may remain implementation-dependent. The reported results should therefore be interpreted as statistically reproducible rather than guaranteed bitwise identical across all hardware and software stacks. All tables in Section 4 report the mean across the ten seeds. Confidence intervals are computed from the seed-wise results using the procedure described in the main text. The final multi-seed evaluation consists of 9 datasets× 7 methods× 10 seeds = 630 26 Table 10: Selected configurations for the contrastive and architectural DIR methods. For ConR, wdenotes the target-distance window in raw target units andβthe contrastive-loss weight; the temperature is fixed toτ = 0.2and the hard-negative coefficient toe = 0.01. For RnC,τdenotes the contrastive temperature andη s1 the stage-1 learning rate; all selected RnC configurations use 400 contrastive pretraining epochs followed by 90 linear-probing epochs. For UVote,Kdenotes the number of expert heads. DatasetConRRnCUVote PM10w = 1.0, β = 4.0 τ = 2.0, η s1 = 0.1 K = 3 PM2.5w = 2.0, β = 2.0 τ = 2.0, η s1 = 0.5 K = 3 Heart Rate w = 1.0, β = 2.0 τ = 2.0, η s1 = 0.1 K = 2 Battery SoC w = 1.0, β = 4.0 τ = 2.0, η s1 = 0.3 K = 2 Montereyw = 1.0, β = 0.5 τ = 2.0, η s1 = 0.5 K = 3 Targa ’13w = 1.0, β = 4.0 τ = 2.0, η s1 = 0.5 K = 3 Targa ’14w = 1.0, β = 2.0 τ = 2.0, η s1 = 0.1 K = 3 Chem. Conc. w = 1.0, β = 4.0 τ = 2.0, η s1 = 0.3 K = 3 Tire Temp. w = 2.0, β = 2.0 τ = 2.0, η s1 = 0.05 K = 3 training runs. G Experiments Appendix G.1 Re-evaluation of MUVIS Models To contextualize the proposed MUVIS-DIR results, we re-evaluate the six original MUVIS baselines under the balanced metrics introduced in Subsection 3.2. All results are averaged across ten random seeds and use the original training protocol of Brandt et al.[9]. In contrast to the DIR experiments in Section 4, these models were trained with an MSE loss rather than anL 1 loss. Differences between the ResNet1D results in Table 11 and Table 12 and the corresponding results in Table 2 and Table 1 therefore reflect the change in training loss. This comparison provides additional context on how the choice of optimization objective affects balanced and tail-sensitive performance. Table 11: Balanced Mean Absolute Error (bMAE)±standard deviation across seeds per dataset and model. DatasetCatBoostLSTMMLPResNet1DTransformerXGBoost PM10230.412 ± 2.653210.981 ± 11.933210.117 ± 4.624226.208 ± 6.264238.499 ± 5.879229.785 ± 1.310 PM2.5172.316 ± 2.501158.011 ± 8.883164.419 ± 10.582170.355 ± 6.330177.748 ± 3.705167.499 ± 3.115 Heart Rate11.120 ± 0.0809.182 ± 1.36811.348 ± 0.3213.279 ± 0.2793.542 ± 0.16412.437 ± 0.083 Battery SoC0.008 ± 0.0000.006 ± 0.0000.006 ± 0.0000.006 ± 0.0000.007 ± 0.0000.017 ± 0.000 Monterey0.194 ± 0.0050.157 ± 0.0040.185 ± 0.0050.158 ± 0.0070.227 ± 0.0100.230 ± 0.004 Targa ’130.150 ± 0.0030.089 ± 0.0030.126 ± 0.0040.096 ± 0.0090.198 ± 0.0160.182 ± 0.003 Targa ’140.063 ± 0.0010.065 ± 0.0020.070 ± 0.0020.062 ± 0.0030.081 ± 0.0040.061 ± 0.001 Chem. Conc.0.082 ± 0.0000.083 ± 0.0010.082 ± 0.0020.082 ± 0.0010.088 ± 0.0020.082 ± 0.000 Tire Temp.3.817 ± 0.0912.600 ± 0.6552.617 ± 0.2673.225 ± 0.4993.735 ± 0.6153.618 ± 0.106 G.2 Performance comparison The following figures complement the aggregate results in the main text by showing per-bin MAE curves for all methods across the nine MUVIS-DIR datasets. These plots make the regional structure of the errors visible and show where individual methods improve over, match, or degrade relative to the vanilla baseline. They also illustrate that improvements in aggregate bMAE or bMASE can arise from different error profiles across the target range, motivating the use of both scalar balanced metrics and per-bin diagnostic plots. 27 Table 12: Balanced Mean Absolute Scaled Error (bMASE)±standard deviation across seeds per dataset and model. DatasetCatBoostLSTMMLPResNet1DTransformerXGBoostGMean (DATA) PM100.654 ± 0.0080.599 ± 0.0340.596 ± 0.0130.642 ± 0.0180.677 ± 0.0170.652 ± 0.0040.636 PM2.50.530 ± 0.0080.486 ± 0.0270.506 ± 0.0330.524 ± 0.0190.547 ± 0.0110.516 ± 0.0100.518 Heart Rate0.275 ± 0.0020.227 ± 0.0340.281 ± 0.0080.081 ± 0.0070.088 ± 0.0040.308 ± 0.0020.184 Battery SoC0.032 ± 0.0010.024 ± 0.0010.026 ± 0.0010.024 ± 0.0010.029 ± 0.0010.071 ± 0.0010.032 Monterey0.111 ± 0.0030.090 ± 0.0030.106 ± 0.0030.091 ± 0.0040.130 ± 0.0060.132 ± 0.0020.109 Targa ’130.176 ± 0.0040.105 ± 0.0030.148 ± 0.0050.113 ± 0.0110.233 ± 0.0190.214 ± 0.0040.158 Targa ’140.103 ± 0.0010.107 ± 0.0040.114 ± 0.0030.102 ± 0.0040.134 ± 0.0060.100 ± 0.0020.109 Chem. Conc.0.606 ± 0.0010.618 ± 0.0090.611 ± 0.0150.606 ± 0.0060.653 ± 0.0120.609 ± 0.0010.617 Tire Temp.0.407 ± 0.0090.278 ± 0.0700.279 ± 0.0290.344 ± 0.0530.399 ± 0.0660.386 ± 0.0110.345 GMean (MODEL)0.2260.1870.2060.1740.2150.257 Table 13: Mean bMASE [CI low, CI high] across ten seeds for each method–dataset pair. Values below VANILLA are italicized; the best value in each row is bold. GMean aggregates across datasets via geometric mean. Lower is better; bMASE < 1 improves over the trivial median predictor. DatasetVanillaConRFocal-L 1 LDSRnCSQInvUVote Battery SoC0.0229 [0.0226, 0.0233]0.0306 [0.0231, 0.0410]0.0280 [0.0238, 0.0339] 0.0224 [0.0221, 0.0228]0.2536 [0.1614, 0.3568]0.0262 [0.0226, 0.0331]0.0251 [0.0242, 0.0259] Chem. Conc.0.6910 [0.6303, 0.7721]0.6485 [0.6154, 0.7120]0.6343 [0.6230, 0.6498] 0.5826 [0.5547, 0.6197]0.6349 [0.6165, 0.6558]0.6371 [0.5861, 0.7207]0.6056 [0.6026, 0.6082] Heart Rate0.1277 [0.1237, 0.1313]0.1925 [0.1210, 0.3051] 0.0748 [0.0723, 0.0778]0.1130 [0.1074, 0.1222]0.0923 [0.0871, 0.0975]0.1232 [0.1067, 0.1535]0.0782 [0.0745, 0.0823] Monterey0.2511 [0.1502, 0.3557]0.2695 [0.1536, 0.3860]0.2215 [0.1472, 0.2972]0.2401 [0.1444, 0.3368]0.3622 [0.1422, 0.6685]0.2260 [0.1390, 0.3204] 0.0853 [0.0838, 0.0870] PM100.6869 [0.6460, 0.7425]0.6793 [0.6562, 0.7079]0.6171 [0.6080, 0.6268] 0.4765 [0.4537, 0.5046]0.6663 [0.6576, 0.6746]0.6230 [0.6047, 0.6475]0.6792 [0.6444, 0.7150] PM2.50.5705 [0.5418, 0.5991]0.5507 [0.5242, 0.5820]0.5242 [0.5129, 0.5376]0.5489 [0.4967, 0.6049]0.5675 [0.5482, 0.5937]0.5457 [0.5177, 0.5751] 0.5142 [0.5065, 0.5225] Targa ’130.1116 [0.1064, 0.1158]0.1136 [0.1111, 0.1161]0.1933 [0.1844, 0.2027]0.1212 [0.1175, 0.1253]0.3452 [0.1867, 0.5560]0.1132 [0.1106, 0.1163]0.1136 [0.1081, 0.1182] Targa ’140.1276 [0.0988, 0.1686]0.1319 [0.0974, 0.1793]0.1326 [0.1239, 0.1437]0.1273 [0.1038, 0.1604]0.1746 [0.1442, 0.2107]0.1183 [0.1044, 0.1434] 0.1055 [0.0957, 0.1222] Tire Temp.0.3662 [0.3357, 0.3949]0.3618 [0.3389, 0.3849]0.3777 [0.3482, 0.4051]0.3534 [0.3283, 0.3805]0.3933 [0.3578, 0.4295]0.3505 [0.3289, 0.3731] 0.3313 [0.3057, 0.3565] GMean0.2163 [0.1903, 0.2423]0.2338 [0.1907, 0.2822]0.2135 [0.1941, 0.2330]0.1997 [0.1771, 0.2229]0.3314 [0.2532, 0.4071]0.2084 [0.1833, 0.2407] 0.1732 [0.1657, 0.1818] Finally, Figure 16 compares the best-performing DIR method for each dataset directly against the vanilla baseline. The signed difference plots highlight whether improvements are concentrated in sparse target regions or distributed more uniformly across the target range. 28 02004006008001000 Target value 0 50 100 150 200 250 300 350 400 Sample count PM10 (mean ± std, 10 seeds) 0 100 200 300 400 500 600 700 800 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote 02004006008001000 Target value 0 50 100 150 200 250 300 350 400 Sample count PM2.5 (mean ± std, 10 seeds) 0 200 400 600 800 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote 0.20.40.60.81.0 Target value 0 500 1000 1500 2000 2500 3000 3500 4000 Sample count Battery SoC (mean ± std, 10 seeds) 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote Figure 13: Per-bin MAE of the six representative DIR methods and the vanilla baseline across three MUVIS tasks [9]. From top to bottom, rows show PM10, Beijing PM25, and Battery SoC prediction. The plots illustrate how method performance varies across the target range. 29 406080100120140160180 Target value 0 100 200 300 400 500 600 700 Sample count Heart Rate (mean ± std, 10 seeds) 0 25 50 75 100 125 150 175 200 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote 5055606570758085 Target value 0 10 20 30 40 Sample count Tyre Temp. (mean ± std, 10 seeds) 0 1 2 3 4 5 6 7 8 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote 4.64.74.84.95.05.1 Target value 0 1000 2000 3000 4000 5000 6000 7000 Sample count Chem. Conc. (mean ± std, 10 seeds) 0.0 0.1 0.2 0.3 0.4 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote Figure 14: Per-bin MAE of the six representative DIR methods and the vanilla baseline across three MUVIS tasks [9]. From top to bottom, rows show PPG, Tire. Temp, and Chem. Conc. prediction. The plots illustrate how method performance varies across the target range. 30 32101234 Target value 0 250 500 750 1000 1250 1500 1750 2000 Sample count Monterey (mean ± std, 10 seeds) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote 1.51.00.50.00.51.01.52.0 Target value 0 200 400 600 800 1000 1200 Sample count Targa '13 (mean ± std, 10 seeds) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote 1.00.50.00.51.0 Target value 0 200 400 600 800 1000 Sample count Targa '14 (mean ± std, 10 seeds) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Per-bin MAE (mean ± std) Vanilla LDS SQInv Focal-L1 ConR RnC UVote Figure 15: Per-bin MAE of the six representative DIR methods and the vanilla baseline across three MUVIS tasks [9]. From top to bottom, rows show Monterey, Targa ’13, and Targa ’14 prediction. The plots illustrate how method performance varies across the target range. 31 0 150 300 Test sample count 02004006008001000 Target value 0 200 400 MAE difference (vanilla - lds) 0 200 400 600 800 Per-bin MAE Vanilla LDS PM10 Vanilla vs LDS 0 150 300 Test sample count 02004006008001000 Target value 0 50 100 MAE difference (vanilla - uvote) 0 200 400 600 800 Per-bin MAE Vanilla UVote PM2.5 Vanilla vs UVote 0 1500 3000 Test sample count 0.20.40.60.81.0 Target value 0.000 0.001 0.002 MAE difference (vanilla - lds) 0.0000 0.0025 0.0050 0.0075 0.0100 Per-bin MAE Vanilla LDS Battery SoC Vanilla vs LDS 0 200 400 600 Test sample count 406080100120140160180 Target value 0.0 2.5 5.0 7.5 10.0 MAE difference (vanilla - focal_l1) 0 5 10 15 Per-bin MAE Vanilla Focal-L1 Heart Rate Vanilla vs Focal-L1 0 600 1200 1800 Test sample count 32101234 Target value 0.0 0.5 1.0 1.5 MAE difference (vanilla - uvote) 0.0 0.5 1.0 1.5 2.0 Per-bin MAE Vanilla UVote Monterey Vanilla vs UVote 0 400 800 1200 Test sample count 1.51.00.50.00.51.01.52.0 Target value 0.04 0.02 0.00 0.02 0.04 MAE difference (vanilla - sqinv) 0.0 0.2 0.4 0.6 0.8 Per-bin MAE Vanilla SQInv Targa '13 Vanilla vs SQInv 0 300 600 900 Test sample count 1.00.50.00.51.0 Target value 0.04 0.02 0.00 0.02 0.04 MAE difference (vanilla - uvote) 0.00 0.05 0.10 0.15 Per-bin MAE Vanilla UVote Targa '14 Vanilla vs UVote 0 2000 4000 6000 Test sample count 4.64.74.84.95.05.1 Target value 0.00 0.05 MAE difference (vanilla - lds) 0.0 0.1 0.2 Per-bin MAE Vanilla LDS Chem. Conc. Vanilla vs LDS 0 15 30 45 Test sample count 5055606570758085 Target value 0.00 0.25 0.50 0.75 MAE difference (vanilla - uvote) 0 2 4 6 Per-bin MAE Vanilla UVote Tyre Temp. Vanilla vs UVote Figure 16: Comparison of the respective best performing DIR methods against the vanilla baseline across all nine MUVIS-DIR datasets. For every figure top row compares per-bin MAE; bottom row shows the signed difference to the baseline. 32

Instant research discovery