Paper deep dive
Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping
Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev, Ivan Rubachev, Daniil Krasylnikov, Artem Babenko
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/13/2026, 3:12:22 AM
Summary
This paper systematically re-evaluates regression objectives for tabular deep learning, finding that quantile loss consistently outperforms Mean Squared Error (MSE) across various backbones. It identifies a limitation in global early stopping where regions with different noise levels reach optimal performance at different epochs. To address this, the authors propose Adaptive Quantile Regression (AdaQR), which uses input-dependent label smoothing to mitigate stopping conflicts and improve predictive performance.
Entities (14)
Relation Signals (7)
Quantile loss → outperforms → Mean Squared Error
confidence 95% · quantile loss yields consistent aggregate improvements over MSE across the tested backbones
Global Early Stopping → suffersfrom → Epoch Misalignment
confidence 93% · regions with different levels of target noise can reach their best predictive performance at different training epochs
AdaQR → improvesover → Quantile loss
confidence 92% · AdaQR ... improves over quantile loss on several backbones
AdaQR → mitigates → Global Early Stopping
confidence 90% · AdaQR is designed as one potential remedy for the mismatch induced by a shared stopping point
HL-Gaussian → outperforms → Mean Squared Error
confidence 88% · HL-Gaussian and quantile loss improve over MSE in aggregate
Soft Binning → outperforms → Mean Squared Error
confidence 88% · Soft Binning ... stand out in the aggregate results, improving upon MSE
Muon → usedwith → Quantile loss
confidence 85% · The Muon configurations achieve larger median gains over the common MSE++AdamW reference
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.13909v1
- Canonical: https://arxiv.org/abs/2610.13909v1
Trouble viewing inline? Open PDF directly →
Full Text
175,933 characters extracted from source content.
Expand or collapse full text
Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping https://github.com/dholzmueller/pytabkit Danil Rudenko Georgii Bulgakov Nikolay Kartashev Affiliation: Yandex Research, BRAInLab, HSE University Ivan Rubachev Affiliation: Yandex Research, BRAInLab, HSE University Daniil Krasylnikov Artem Babenko Affiliation: Yandex Research, BRAInLab, HSE University Abstract While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at https://github.com/yandex-research/tabular-regression. 11footnotetext: Equal contribution.22footnotetext: Work done while at Yandex Research. 1. Introduction Despite the growing adoption of deep learning models for tabular data, in practice, it is not entirely clear what objective function should be used to solve regression tasks with DNNs. While the design of advanced regression objectives is an active research direction [1, 2, 3, 4, 5, 6, 7, 8], the traditional Mean Squared Error (MSE) remains widely used. Existing studies evaluate alternative objectives on different datasets, backbones, and tuning protocols, making it difficult to establish which alternatives provide reliable improvements in practice. We revisit traditional and modern regression objectives on 41 established tabular datasets with extensive hyperparameter tuning, focusing on conventional DNNs (pretrained PFNs are outside our scope). We find that most objectives do not consistently improve over a carefully tuned MSE baseline. Two approaches stand out on MLP: Gaussian-smoothed binning objectives [1, 2], which represent targets as distributions over discrete bins, and quantile loss [9], which predicts conditional quantiles of p(y∣x)p(y x) via the pinball objective. Of these, quantile loss yields more consistent gains across different backbones. Related evidence comes from the PFN literature: TabICLv2 [10] adopts quantile loss for regression and reports benefits over MSE and binning; our findings complement this with a systematic comparison in conventional tabular models. Having identified quantile loss as a strong regression objective, we investigate a limitation that remains under standard training: regions with different levels of data uncertainty can reach their best predictive performance at different epochs. Consequently, a single early-stopping checkpoint can impose a compromise between uncertainty regions. Such input-dependent variation in target noise is common in tabular data [11]. We examine this conflict through controlled synthetic experiments and uncertainty-based checkpoint selection on real datasets. Motivated by this limitation, we propose Adaptive Quantile Regression (AdaQR), which incorporates jointly learned, input-dependent smoothing into quantile training. Effectively, by reducing the sensitivity of quantile updates to individual label realizations in high-uncertainty regions, adaptive smoothing aims to delay overfitting there while allowing learning to continue in lower-uncertainty regions. AdaQR is designed as one potential remedy for the mismatch induced by a shared stopping point, which improves upon standard quantile loss on several DNN backbones as we show in our experiments. Our main contributions are: • We benchmark 18 regression objectives on MLP across 41 datasets and evaluate selected objectives on three advanced tabular DNN backbones, using up to 100 hyperparameter tuning trials and the Muon optimizer [12, 13]. • We show that most objectives do not consistently beat tuned MSE. Gaussian-smoothed binning objectives and quantile loss stand out on MLPs, while quantile loss provides more consistent gains across the tested backbones. • We identify a limitation of global early stopping in tabular regression: synthetic experiments reveal different preferred stopping epochs across noise levels, while real-data experiments show that selecting checkpoints separately for cohorts defined by estimated uncertainty can improve predictive performance over global selection. • We propose a new AdaQR objective that provides a jointly learned smoothing mechanism for investigating how uncertainty-aware training can mitigate stopping conflicts and evaluate this objective on established tabular backbones. 2. Related Work Regression objectives. Alternatives to MSE include robust losses [14], target discretization [1, 2, 3], conditional distribution modeling [15, 9, 4, 16], and representation regularization [7, 8, 6]. We compare representative methods under a common tabular evaluation protocol and describe the individual objectives in the following sections. Early stopping and overfitting. Different parts of a model or subsets of the data can reach their best generalization performance at different training epochs. Heckel and Yilmaz [17] analyze different bias–variance tradeoffs across feature directions and network layers, showing that adjusting learning rates can improve early-stopping performance. Bao et al. [18] observe that some token groups are already overfitting while others remain underfit at the selected checkpoint in sequence-to-sequence models. Berta et al. [19] show that calibration and refinement errors are minimized at different stages of training, including in tabular classification. We extend the uncertainty-based analysis of tabular models [11] to training trajectories and checkpoint selection, examining how input-dependent noise relates to the epochs at which different regions achieve their best predictive performance. Heteroskedasticity-aware objectives and adaptive smoothing. A large body of work models input-dependent uncertainty in regression [20, 21]. However, joint mean and variance estimation can introduce optimization difficulties [22, 23], motivating methods that preserve the mean-prediction accuracy of a corresponding MSE-trained model [24]. Related work on quantile regression investigates smoothing and local scale adaptation. Tan et al. [25] study convolution smoothing of the pinball loss to improve optimization and statistical estimation. Fasiolo et al. [26] derive covariate-dependent smoothing scales from a preliminary location–scale fit, while Jung et al. [27] estimate local scales using preliminary quantile or absolute-residual regressions. In distributional reinforcement learning, Malekzadeh et al. [28] propose a noise-motivated generalization of quantile-Huber loss with adaptive parameter adjustment. AdaQR combines an energy-distance objective with an input-dependent smoothing head learned jointly with the neural quantile predictor. We investigate this formulation through its effects on uncertainty-dependent training dynamics and tabular point prediction. 3. Revisiting regression objectives Notation. We consider supervised regression on tabular data. Let D=(xi,yi)i=1nD=\(x_i,y_i)\_i=1^n denote a dataset, where xi∈ℝdx_i ^d is the feature vector and yi∈ℝy_i is the target. We denote the neural model by fθ:ℝd→ℝd′f_θ:R^d ^d , where the output dimension d′d depends on the objective and need not equal one. For pointwise objectives, the loss on an observation is l(fθ(xi),yi)l(f_θ(x_i),y_i). 3.1 Losses Figure 1: Performance comparison of different objective functions using an MLP backbone across 41 BeyondArena datasets. (Left) Average rank performance across all datasets. (Right) Relative RMSE improvement over the standard MSE baseline, visualizing the magnitude of performance gains or degradation. The box plots visualize the 10th, 25th, 50th, 75th, and 90th percentiles of the jitter plots. Notation details are provided in the beginning of Section . We compare the following regression objectives. Traditional losses. MSE penalizes squared errors, MAE penalizes absolute errors, and Huber loss [29] combines quadratic penalties for small errors with linear penalties for large ones. Binning-based losses. These methods discretize the target range into bins and train a categorical prediction head to match a target distribution. Hard Binning assigns each target to its nearest bin, while Soft Binning distributes probability mass according to Gaussian weights evaluated at bin centers [2]. HL-Gaussian instead integrates a truncated Gaussian density over each bin [1]. Joint end-to-end encoding learns the target distribution using an auxiliary encoder and a reconstruction objective [2]. All four methods return the predicted expectation over bin centers. Softmax-MSE uses the same prediction head but directly minimizes the squared error of this expectation [2]. Quantile regression. The model predicts a grid of conditional quantiles using the pinball loss [9]; their average serves as the scalar prediction. Likelihood-based objectives. Gaussian NLL jointly learns a conditional mean and variance. β-NLL reweights this objective by a detached power of the predicted variance to address optimization difficulties [22]. Mixture Density Networks model the conditional distribution as a Gaussian mixture [15]. The autoregressive approach of Song and Bahri [4] instead predicts tokenized targets using next-token cross-entropy. We evaluate both its normalized and unnormalized target encodings. Additional objectives and regularizers. GAR combines pointwise and pairwise terms to capture target values and their differences [5]. Ordinal Entropy [6] and ROT-REG [8] augment regression training with representation regularization, using entropy-based and optimal-transport-based terms, respectively. We provide a detailed description of all objectives in Appendix . 3.2 Experiments Datasets. We evaluate a subset of 41 regression datasets from BeyondArena [30], excluding three computationally demanding datasets to keep hyperparameter tuning feasible within our 24-hour budget. We use the first predefined split of each retained dataset. Further details are provided in Appendix . 3.2.1 Implementation details Model backbone. We first compare regression objectives on the simplest MLP backbone, then evaluate selected objectives on additional advanced architectures. Depending on the objective, the prediction head outputs either a scalar, y^i=fθ(xi)∈ℝ y_i=f_θ(x_i) , or a vector parameterizing the corresponding predictive output. Experimental setup. We follow the evaluation protocol of Gorishniy et al. [31], detailed in Appendix . Unless stated otherwise, we use Muon-based training [12], following recent evidence of its effectiveness on tabular data [13]. For each dataset and objective, we tune the model, objective, and optimizer hyperparameters on the validation set. We then retrain the selected configuration from scratch across 15 random seeds and report test RMSE. Method comparisons use the mean and standard deviation across these seeds, as described in Appendix . 3.2.2 Results We evaluate all methods following the protocol above and report the results in Figure . For readability, Decoder[U] and Decoder[N] denote the unnormalized and normalized decoders from Song and Bahri [4], respectively; GAR[MSE] and GAR[MAE] denote GAR variants with MSE- and MAE-based formulations [5], respectively. Key takeaways. 1. Most objectives do not provide consistent improvements over a carefully tuned MSE. 2. Soft Binning [2], HL-Gaussian [1], and quantile loss [9] stand out in the aggregate results, improving upon MSE in terms of both ranks and relative RMSE improvement. These findings also hold under AdamW [32]. Table shows that HL-Gaussian and quantile loss improve over MSE in aggregate with this optimizer. The Muon configurations achieve larger median gains over the common MSE++AdamW reference, highlighting the importance of optimizer choice when evaluating regression objectives. Table 1: Objective and optimizer comparison on MLP across 41 datasets. Gain is the median relative RMSE improvement over the common MSE++AdamW baseline; positive values indicate improvement. W/T/L counts wins, ties, and losses under the standard-deviation-based comparison rule in Appendix . Bold denotes the largest median gain within each optimizer. AdamW Muon Objective Gain (%) W / T / L Gain (%) W / T / L MSE — — +1.50 18 / 19 / 4 HL-Gaussian +0.78 19 / 19 / 3 +2.34 26 / 11 / 4 quantile loss +0.67 15 / 23 / 3 +2.18 24 / 13 / 4 3.2.3 Advanced backbones Having identified promising objectives on MLP, we examine whether their benefits transfer to other state-of-the-art backbone architectures. We retain HL-Gaussian as a representative binning-based objective and compare it and quantile loss against MSE. Following the same evaluation protocol, we train TabPack with periodic feature embeddings [33], TabM-mini with PLE-Q embeddings [31], and RealMLP [34]. Implementation details are provided in Appendix . Table shows that quantile loss provides more consistent gains across the tested backbones than HL-Gaussian. On TabPack, HL-Gaussian has more losses than wins against MSE, despite a small positive median improvement; on RealMLP, its median gain is also limited. In contrast, quantile loss achieves positive median improvements and more wins than losses on every tested backbone. These results support quantile loss as a practical alternative to MSE, while showing that its benefits remain dataset-dependent. Table 2: Quantile loss yields consistent aggregate improvements across the tested backbones. Each objective is compared against MSE on the same backbone. Gain is the median relative RMSE improvement; positive values indicate improvement. W/T/L follows the comparison rule in Appendix . Bold denotes the larger median gain within each backbone, with both values highlighted when equal at the reported precision. HL-Gaussian quantile loss Backbone Gain (%) W / T / L Gain (%) W / T / L MLP +0.65 19 / 17 / 5 +0.55 21 / 15 / 5 TabPack +0.06 8 / 20 / 13 +0.46 18 / 21 / 2 RealMLP +0.12 13 / 17 / 11 +0.40 20 / 16 / 5 TabM +0.54 17 / 19 / 5 +0.58 20 / 16 / 5 4. Quantile regression under heteroskedasticity The benchmark in Section identifies several regression objectives that consistently outperform the MSE baseline. While HL-Gaussian and quantile regression (QR) perform comparably on a plain MLP, QR is more robust across backbones (Table ). Therefore, we focus on QR for the remainder of our analysis. QR predicts conditional quantiles without assuming a parametric family for p(y∣x)p(y x), allowing the predicted distribution’s shape and scale to vary across inputs. This flexibility is particularly beneficial for tabular data, where the conditional variance Var[y∣x]Var[y x], often termed data uncertainty, changes substantially with x (i.e., the data is heteroskedastic). Kartashev et al. [11] identify regions of low and high data uncertainty in several established tabular datasets, and demonstrate that dataset-wise aggregate scores often obscure how methods perform across these different regions. However, modeling input-dependent distributions does not guarantee that regions with different noise levels achieve their optimal predictive performance at the same epoch. Standard global early stopping selects a single checkpoint based on average validation error, applying it uniformly to all inputs. We first examine how prediction errors evolve during training across different uncertainty levels (Section ). We then test whether selecting checkpoints separately for different uncertainty cohorts improves upon doing so for random cohorts of the same size (Section ). Together, these experiments reveal an optimization challenge that distributional flexibility alone cannot resolve, motivating the adaptive objective introduced in Section . Both experiments in this section require a per-point estimate of data uncertainty and are most interpretable when this estimate is exact and test targets are noise-free. To demonstrate the claims of this section in a controlled environment that has these traits, we construct a synthetic dataset y=f(x)+εy=f(x)+ , with noise added only to training targets, so test RMSE directly measures recovery of f=[y∣x]f=E[y x]. We use two variants sharing exactly the same f and x: heteroskedastic, with ε∼(0,σ(x)2) (0,σ(x)^2), and homoskedastic, with ε∼(0,σ2) (0,σ^2), serving as a control. Importantly, for this dataset, levels of true data uncertainty σ2σ^2 are known to us exactly. See Appendix for details. To estimate uncertainty on real data we use a CatBoost model [35] trained via Gaussian negative log-likelihood on the real datasets, utilizing the predicted σ(x)σ(x) as our uncertainty estimate. To prevent data leakage, this estimator is fit exclusively on the training split and applied to validation and test points during inference. 4.1 Optimal stopping epochs vary by uncertainty region In this subsection we investigate whether models require different stopping epochs for regions with varying data uncertainty. This requires analyzing the test error disaggregated by uncertainty level across epochs. The uncertainty plot [11] provides this perspective. Figure (a) shows an MLP trained with the quantile loss on the heteroskedastic synthetic dataset. Each curve corresponds to one epoch and displays the point-wise change in squared error relative to a reference epoch (the one with the best aggregate validation MSE), plotted against data uncertainty. Before the reference epoch, the model performs better on high-uncertainty points and worse on low-uncertainty ones; afterward, this trend reverses. In other words, the model starts overfitting high-uncertainty regions before it finishes fitting low-uncertainty ones, and the reference epoch is merely a compromise between these competing dynamics. To verify that this pattern stems from varying noise levels rather than intrinsic properties of x or f(x)f(x), Figure (c) repeats the experiment on the homoskedastic variant (same f, constant noise scale, with samples sorted as in the heteroskedastic version). The effect disappears: performance changes across the displayed epochs are largely unrelated to the uncertainty ordering inherited from the heteroskedastic variant. Ideally, a model would retain its early-epoch error on high-uncertainty points while achieving its late-epoch error on low-uncertainty points, improving upon the QR baseline. However, Figure (a) only illustrates this misalignment on synthetic data and does not quantify the resulting performance loss. Section closes both gaps. Figure 2: In this synthetic experiment, heteroskedastic noise creates a mismatch between the stopping epochs preferred by low- and high-uncertainty regions, while AdaQR mitigates this effect. Uncertainty plots [11] for an MLP trained with (a) quantile loss and (b) AdaQR on the heteroskedastic dataset, and (c) quantile loss on the homoskedastic counterpart. Each curve shows the smoothed point-wise difference in squared test error relative to the reference epoch with the best aggregate validation MSE; negative values indicate improvement. Test samples are ordered by increasing uncertainty in the heteroskedastic dataset, with the same ordering used in all panels. Under heteroskedastic noise, earlier epochs perform relatively better on high-uncertainty points and later epochs on low-uncertainty points, indicating that different uncertainty regions favor different stopping times. This separation is reduced with AdaQR and disappears when the noise is homoskedastic. 4.2 Measuring the impact of epoch misalignment To quantify the performance penalty implied by the epoch misalignment in Figure (a), we evaluate a hypothetical strategy of per-cohort early stopping. This approach requires dividing the validation and test sets into groups of comparable data uncertainty, termed cohorts. To construct these, we compute the K−1K-1 empirical kK kK-quantiles of the estimated uncertainty σ^(x) σ(x) on the validation set. These thresholds are used to split both the validation and test sets into K cohorts, ordered from lowest (cohort 11) to highest (cohort K) uncertainty. Deriving these thresholds strictly from the validation set prevents data leakage, yielding equal-sized validation cohorts and test cohorts that are balanced up to sampling noise. With the cohorts established, we save a model checkpoint at every epoch during training. For each uncertainty cohort k, we select the checkpoint ek∗e_k^* that minimizes the validation MSE restricted specifically to that group. During inference, each test point is routed to the checkpoint ek∗e_k^* corresponding to its uncertainty cohort. Because this approach assigns exactly one checkpoint to each test example, the resulting aggregated test RMSE reflects the empirical benefit of tailoring stopping times to local uncertainty levels. To accurately isolate this benefit, we cannot simply compare the resulting RMSE to standard global early stopping. While per-cohort selection allows the model to choose K different epochs, each choice relies on only a fraction (1/K1/K) of the validation data. To create a fair baseline, we apply the exact same procedure to random cohorts of matching size, created by randomly permuting the points before splitting. These random groups suffer the same penalty of smaller validation sets but contain no uncertainty information. Therefore, comparing uncertainty cohorts directly to random cohorts isolates the advantage of grouping by local noise levels. Table presents the relative improvements in test RMSE when using uncertainty-based (U) and random (R) cohorts compared to standard global (G) early stopping. To minimize the negative effects of smaller validation sets, we restrict this experiment to the seven largest datasets from our benchmark, reporting results for K=5K=5 and K=10K=10. The results demonstrate that uncertainty-based selection improves median RMSE over global selection by 0.18% and 0.114% for five and ten cohorts, respectively. Under our comparison rule, both configurations record three wins, four ties, and no losses across the seven datasets. Furthermore, the drop in overall performance when increasing from K=5K=5 to K=10K=10 cohorts is consistent with increased checkpoint-selection noise from smaller validation subsets. The table also includes the homoskedastic variant of our synthetic dataset, which shares the same features x and noiseless target function f(x)f(x), but features a constant noise scale σ. This allows us to apply the exact same cohort splits used for the heteroskedastic variant. If σ(x)σ(x) were merely acting as a proxy for partitioning the feature space into regions of similar inputs, both variants would show similar gains. Instead, performance on the homoskedastic version worsens when applying uncertainty-based splits, while the opposite is true for the heteroskedastic variant. Thus, the benefit stems specifically from adapting to how varying label noise alters training dynamics, not from the spatial structure of the inputs. Per-cohort early stopping, however, is not a practical solution: it requires storing numerous checkpoints and assigning test points to cohorts during inference. These results motivate adapting the objective to delay overfitting in high-uncertainty regions while allowing learning to continue in low-uncertainty regions. Section introduces AdaQR as a potential way to reduce this mismatch during training. Table 3: Cohort-based vs. global checkpoint selection for an MLP trained with quantile loss. Improv. (%) is the relative reduction in mean test RMSE using uncertainty (U/G) or random (R/G) cohorts compared with global selection. All checkpoints are selected on validation data. Bottom rows summarize available comparisons among the seven largest benchmark datasets, excluding synthetic data; W/T/L follows Appendix . Results are computed from mean test RMSEs over 15 random seeds. K=5K=5 K=10K=10 Dataset / summary U/G R/G U/G R/G Synthetic (heteroskedastic) +0.318+0.318 −0.041-0.041 +0.286+0.286 −0.118-0.118 Synthetic (homoskedastic) −0.019-0.019 −0.021-0.021 −0.048-0.048 −0.078-0.078 Seven largest benchmark datasets Median Improv. (%) +0.180+0.180 −0.029-0.029 +0.114+0.114 −0.003-0.003 W / T / L 3/4/0 0/7/0 3/4/0 0/7/0 5. Adaptive quantile regression To address the stopping-time mismatch identified in Section , we introduce AdaQR, a quantile objective with jointly learned, input-dependent smoothing inspired by Adaptive Label Smoothing [36]. By applying stronger smoothing in high-uncertainty regions, we aim to postpone overfitting there. Up to quantile discretization, quantile loss is proportional to the squared Energy Distance between the predicted distribution and a Dirac δ distribution at the observed target y. Appendix details this connection through the Continuous Ranked Probability Score (CRPS). To incorporate adaptive smoothing, we replace this point mass with a Gaussian distribution (y,σl2)N(y, _l^2) centered around the target, where σl2 _l^2 is a learned variance predicted from the input x. We design our loss function such that a larger conditional variance Var[y∣x]Var[y x] yields a larger predicted σl2 _l^2. Specifically, the loss function we use is ℒ(F,y,σl)=122(F,(y,σl2))+λσl,L(F,y, _l)= 12D^2 (F,N(y, _l^2) )+λ _l, where D denotes the Energy Distance [37]. The first term is half of the squared Energy Distance, where the predicted distribution F is estimated through the predicted quantiles qiq_i. The second term acts as a regularization penalty, scaling the predicted smoothing parameter σl _l by a tunable hyperparameter λ. As shown in Appendix , when expressed in terms of specific predicted quantiles qiq_i, our loss becomes: 1N∑i=1N((qi−y)(2Φ(qi−yσl)−1)+2σl2πe−(qi−y)22σl2)−1N∑i=1N(2τi−1)qi−σlπ+λσl, 1N _i=1^N ((q_i-y) (2 \! ( q_i-y _l )-1 )+ 2 _l 2πe^- (q_i-y)^22 _l^2 )- 1N _i=1^N(2 _i-1)q_i- _l π+λ _l, (1) where N is the number of predicted quantiles, Φ is the CDF of the standard normal distribution, and τi _i represents the probability level of the i-th quantile. Assuming Gaussian label noise, Gaussian model predictions, sufficient quantile resolution, and convergence to the Bayes optimum, Proposition shows that the optimal smoothing variance satisfies σl2∝σt2 _l^2 _t^2, where σt2=Var[y∣x] _t^2=Var[y x]. The full statement and proof are provided in Appendix . Table 4: Predictive performance on 28 BeyondArena datasets with at least 1,000 training samples. Gain is the median relative RMSE improvement over MSE on the same backbone; positive values indicate improvement. W/T/L summarizes comparisons with MSE using the rule in Appendix . Bold denotes the largest median gain within each backbone. AdaQR† includes auxiliary scale regularization (Appendix ). MLP Objective Gain (%) W / T / L NLL +0.02 9 / 12 / 7 β-NLL +0.25 10 / 11 / 7 quantile loss +0.47 16 / 10 / 2 AdaQR +0.66 16 / 10 / 2 Backbone Objective Gain (%) W / T / L TabPack quantile loss +0.22 12 / 14 / 2 AdaQR +0.25 14 / 14 / 0 RealMLP quantile loss +0.35 15 / 11 / 2 AdaQR† +0.82 19 / 6 / 3 TabM quantile loss +0.39 12 / 12 / 4 AdaQR +0.97 14 / 11 / 3 These assumptions provide a theoretical framework for AdaQR’s uncertainty adaptation but are not strict prerequisites for empirical success. In practice, AdaQR adds a scalar output that parameterizes σl _l alongside the predicted quantiles. This output is used only during training; at inference, the point prediction is obtained by averaging the quantiles, as in standard quantile regression. Thus, AdaQR introduces no additional test-time cost. Implementation details are provided in Appendix . 5.1 AdaQR results We first evaluate AdaQR’s predictive accuracy. In the following sections, we analyze the role of adaptive smoothing, the quality of the learned uncertainty estimates, and AdaQR’s effect on the global stopping compromise. Our main comparison covers 28 datasets with at least 1,000 training samples, excluding the Tiny category of BeyondArena [30]. Since AdaQR trains an additional σl _l logit jointly with the main predictions and requires tuning λ, which can complicate optimization on very small datasets, we exclude such datasets from the main comparison. We follow our standard evaluation protocol with Muon-based training on MLP, TabPack [33], RealMLP [34], and TabM-mini with PLE-Q embeddings [31]. On MLP, we additionally evaluate Gaussian NLL and β-NLL as heteroskedastic baselines. For RealMLP, we use a scale-regularized variant, denoted by AdaQR†, motivated by the uncertainty-estimation analysis and described in Appendix . Table shows that AdaQR achieves positive median RMSE improvements over MSE on all four backbones. Its median gains over MSE are also larger than those of quantile loss, with the clearest differences on TabM and RealMLP and a small difference on TabPack. These results support AdaQR as a useful training alternative, while showing that its benefits vary across backbones. 5.2 Importance of Adaptive Smoothing Table 5: Ablation of adaptive scale prediction on RealMLP. Gain is the median relative RMSE improvement over MSE. W/T/L follows the comparison rule in Appendix . Objective Gain (%) W / T / L quantile loss +0.35 15 / 11 / 2 AdaQR† +0.82 19 / 6 / 3 Global σl _l +0.14 11 / 10 / 7 To highlight the importance of adaptivity in AdaQR, we conduct an ablation experiment in which the per-sample σl(x) _l(x) is replaced with a single global σl _l shared across all instances. Rather than being learned jointly with the model, the global σl _l is fixed and tuned as a hyperparameter. We perform this ablation on RealMLP, a strong-performing tabular model. We exclude the Tiny subset as described in Section . All other settings follow the standard experimental protocol described in Section . The substantial degradation with a global σl _l highlights the importance of instance-wise adaptivity in AdaQR. 5.3 Uncertainty Estimation Figure 3: AdaQR versus CatBoost data uncertainty on three real-world datasets from the Beyond benchmark (test splits; logarithmic axes). AdaQR estimates are averaged across TabM branches after conversion to variances using (). Dashed lines indicate y=xy=x; ρS _S denotes Spearman’s rank correlation computed over all test inputs for each dataset. At most 8,000 randomly sampled inputs are displayed per dataset. We examine whether AdaQR learns meaningful data uncertainty on real-world tasks. Since ground-truth uncertainty is unavailable, we use CatBoost [35] as a trusted reference. Figure shows strong correlations between their uncertainty estimates, suggesting AdaQR captures meaningful input-dependent variation. See Appendix for additional examples and implementation details. 5.4 Effect on the Global Stopping Compromise The synthetic results in Figure suggest that AdaQR alleviates the mismatch between optimal stopping epochs across uncertainty regions. To examine whether this also holds on real-world data, we repeat the cohort-based experiment from Section for AdaQR on the same seven largest benchmark datasets, using the MLP backbone. We compare the benefit of cohort-based over global checkpoint selection under AdaQR and quantile loss. Table 6: Cohort-based vs. global checkpoint selection for an MLP trained with quantile loss (QR) or AdaQR. Improv. (%) is the relative reduction in the mean test RMSE compared with global selection under the same objective. Results summarize seven real-world datasets, using 15 random seeds. All checkpoints are selected on validation data; W/T/L follows Appendix . K=5K=5 K=10K=10 Summary quantile loss AdaQR quantile loss AdaQR Median Improv. (%) +0.180+0.180 +0.092+0.092 +0.114+0.114 +0.085+0.085 W / T / L 3/4/ 0 1/6/0 3/4/0 2/5/0 Table shows a lower benefit from cohort-based selection under AdaQR for both values of K. This benefit decreases on five of the seven datasets in both settings (see extended results in Appendix ), consistent with our findings on synthetic data. 6. Conclusion In this paper, we systematically revisit regression objectives for tabular deep learning, comparing 18 objectives on 41 datasets and evaluating leading approaches across multiple backbones. Quantile loss emerges as a consistently strong alternative to carefully tuned MSE. Yet a strong objective does not eliminate the limitations of global early stopping: regions with different noise levels can peak at different training epochs, making a shared checkpoint a compromise. Motivated by this, we introduce AdaQR, which uses jointly learned, input-dependent smoothing as one possible remedy and improves predictions in several settings, with benefits varying across backbones and datasets. Our findings highlight that progress in tabular regression requires attention not only to what models optimize, but also to how different parts of the data are learned over time. 7. Limitations Our study identifies uncertainty-dependent stopping conflicts as an opportunity to improve tabular regression, but our understanding of the underlying training dynamics remains incomplete. AdaQR helps mitigate these stopping conflicts: on real datasets, cohort-based checkpoint selection yields smaller improvements for AdaQR than for standard quantile regression (Section ), yet still provides additional gains. This residual benefit suggests that adaptive smoothing does not fully resolve differences in preferred stopping times across uncertainty regions. Further work is needed to understand when these conflicts arise and to explore alternative objectives, training procedures, or checkpoint-selection strategies that can address them more effectively. AdaQR’s predictive gains also vary in magnitude across backbones, and its main evaluation excludes Tiny datasets, although results including them are reported in Appendix . AI use statement We used generative AI tools to assist with implementing experimental code, identifying relevant literature, discussing research methodology and experimental design, and editing the manuscript. The authors reviewed all AI-assisted material and verified AI-generated code. All final methodological decisions and scientific conclusions were made by the authors, who take responsibility for the manuscript, code, and reported results. Reproducibility statement The appendix describes the datasets, preprocessing, hyperparameter search spaces, and evaluation protocol. Code and experiment configurations are available in our repository: https://github.com/yandex-research/tabular-regression. The repository README provides instructions for downloading the datasets and reproducing the experiments. References [1] Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International conference on machine learning, pages 2157–2166. PMLR, 2018. [2] Lawrence Stewart, Francis Bach, and Quentin Berthet. Building bridges between regression, clustering, and classification. arXiv preprint arXiv:2502.02996, 2025. [3] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=dVpFKfqF3R. [4] Xingyou Song and Dara Bahri. Decoding-based regression. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=avUQ8jguxg. [5] Dixian Zhu, Tianbao Yang, and Livnat Jerby. Gradient aligned regression via pairwise losses. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=1bnH3Q3zL6. [6] Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=raU07GpP0P. [7] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: learning continuous representations for regression. Advances in Neural Information Processing Systems, 36:17882–17903, 2023. [8] Shihao Zhang, Yuguang Yan, and Angela Yao. Improving deep regression with tightness. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=dkoiAGjZV9. [9] Roger Koenker and Gilbert Bassett, Jr. Regression quantiles. Econometrica, 46(1):33–50, 1978. ISSN 00129682. doi: 10.2307/1913643. URL http://w.jstor.org/stable/1913643. [10] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabiclv2: A better, faster, scalable, and open tabular foundation model. In Forty-third International Conference on Machine Learning, 2026. [11] Nikolay Kartashev, Ivan Rubachev, and Artem Babenko. Unveiling the role of data uncertainty in tabular deep learning. arXiv preprint arXiv:2509.04430, 2025. [12] Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/. [13] Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov, and Artem Babenko. Benchmarking optimizers for mlps in tabular deep learning. arXiv preprint arXiv:2604.15297, 2026a. [14] Peter J Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101, 1964. [15] Christopher M Bishop. Mixture density networks. Technical Report NCRG/94/004, Neural Computing Research Group, Aston University, 1994. [16] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35:18100–18115, 2022. [17] Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how to eliminate it. arXiv preprint arXiv:2007.10099, 2020. [18] Guangsheng Bao, Zhiyang Teng, and Yue Zhang. Token-level fitting issues of seq2seq models. In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pages 276–288, 2023. [19] Eugène Berta, David Holzmüller, Michael I Jordan, and Francis Bach. Rethinking early stopping: Refine, then calibrate. arXiv preprint arXiv:2501.19195, 2025. [20] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017. [21] Alexander Immer, Emanuele Palumbo, Alexander Marx, and Julia Vogt. Effective bayesian heteroscedastic regression with deep neural networks. Advances in Neural Information Processing Systems, 36:53996–54019, 2023. [22] Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=aPOpXlnV1T. [23] Eliot Wong-Toi, Alex Boyd, Vincent Fortuin, and Stephan Mandt. Understanding pathologies of deep heteroskedastic regression. arXiv preprint arXiv:2306.16717, 2023. [24] Andrew Stirn, Harm Wessels, Megan Schertzer, Laura Pereira, Neville Sanjana, and David Knowles. Faithful heteroscedastic regression with neural networks. In International Conference on Artificial Intelligence and Statistics, pages 5593–5613. PMLR, 2023. [25] Kean Ming Tan, Lan Wang, and Wen-Xin Zhou. High-dimensional quantile regression: Convolution smoothing and concave regularization. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):205–233, 2022. [26] Matteo Fasiolo, Simon N Wood, Margaux Zaffran, Raphaël Nedellec, and Yannig Goude. Fast calibrated additive quantile regression. Journal of the American Statistical Association, 116(535):1402–1412, 2021. [27] Yoonsuh Jung, Yoonkyung Lee, and Steven N MacEachern. Efficient quantile regression for heteroscedastic models. Journal of Statistical Computation and Simulation, 85(13):2548–2568, 2015. [28] Parvin Malekzadeh, Konstantinos N Plataniotis, Zissis Poulos, and Zeyu Wang. A robust quantile huber loss with interpretable parameter adjustment in distributional reinforcement learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6120–6124. IEEE, 2024. [29] Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pages 492–518. Springer, 1992. [30] Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Gaël Varoquaux, Frank Hutter, et al. Beyond iid: How general are tabular foundation models, really? arXiv preprint arXiv:2606.30410, 2026. [31] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Sd4wYYOhmY. [32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. [33] Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev, and Artem Babenko. Tabpack: Efficient hyperparameter ensembles for tabular deep learning. In Forty-third International Conference on Machine Learning, 2026b. URL https://openreview.net/forum?id=8qHjrpxrQn. [34] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned MLPs and boosted trees on tabular data. Advances in Neural Information Processing Systems, 37:26577–26658, 2024. [35] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018. [36] Ujwal Krothapalli and A Lynn Abbott. Adaptive label smoothing. arXiv preprint arXiv:2009.06432, 2020. [37] Gábor J Székely. E-statistics: The energy of statistical samples. Bowling Green State University, Department of Mathematics and Statistics Technical Report, 3(05):1–18, 2003. [38] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in python, 2018. URL https://arxiv.org/abs/1201.0490. [39] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019. [40] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991–25004, 2022. [41] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. [42] James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science, 22(10):1087–1096, 1976. [43] Francesco Laio and Stefania Tamea. Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrology and Earth System Sciences, 11(4):1267–1277, 2007. Appendix A Datasets Our benchmark comprises 41 datasets from BeyondArena [30] . We exclude maps_router_eta_1m, delivery_eta_1m and cooking_time_1m from the evaluation due to their combination of high numerical dimensionality (992 for maps_router_eta_1m), high categorical cardinality (135131 for cooking_time_1m) and large sample size (1.25 million for all three of them). Under our chosen preprocessing and training pipeline, these characteristics make exhaustive hyperparameter tuning infeasible within the 24-hour computational budget imposed for each dataset. We use the lite subset of the benchmark (i.e., the first split of each dataset). Importantly, GAR[MSE], GAR[MAE] and Ordinal Entropy on MLP show abnormal behavior and fail to converge on airfoil_self_noise and on telemonitoring_parkinsons_biomedical_voice_measurements. To retain these datasets in the aggregate comparisons, we replace each failed result with the worst result among the methods that converged on the same dataset using the MLP backbone. Batch sizes. For non-RealMLP models we determine the batch size from the number of training examples ntrainn_train: B(ntrain)=32,ntrain≤2,800,64,2,800<ntrain≤4,500,128,4,500<ntrain≤6,400,256,6,400<ntrain≤32,000,512,32,000<ntrain≤108,000,1024,ntrain>108,000.B(n_train)= cases32,&n_train≤ 2,800,\\ 64,&2,800<n_train≤ 4,500,\\ 128,&4,500<n_train≤ 6,400,\\ 256,&6,400<n_train≤ 32,000,\\ 512,&32,000<n_train≤ 108,000,\\ 1024,&n_train>108,000. cases For RealMLP we use default batch sizes from PyTabkit - 256 for training and 1024 for prediction. Appendix B Implementation Details B.1 Experimental protocol Data preprocessing. For each dataset, the same preprocessing is used. First, we treat binary features as categorical. Both binary and categorical features are then transformed using ordinal encoding. By default, numerical features are processed via a slightly modified version of quantile normalization from Scikit-learn [38] (see the code). Finally, targets are standardized to zero mean and unit variance, though all final metrics are computed on the original scale. The only exception is the evaluation of decoders [4], where we follow the authors’ protocol by either mapping targets to [0,1][0,1] (for the normalized decoder) or applying no transformations (for the unnormalized version). Training Process. We use Muon [12] to optimize all layer weight matrices, while AdamW [32] is used for bias parameters and the output-layer weight matrix. We do not apply learning rate schedulers. We apply global gradient clipping to 1.0. For each dataset, we use the predefined batch size (see Appendix ) and continue training until there are patience consecutive epochs without improvements on the validation set; we set patience=16 patience=16. Hyperparameter tuning. For each experiment, we carefully tune optimizer, objective and backbone hyperparameters using Optuna’s TPE sampler [39], for 100 trials with 24-hour tune time limitation. We use RMSE computed on the validation split as the optimization objective. We provide full parameter spaces in Appendix . Evaluation. After the best hyperparameters are found, we run the model with those parameters on 15 different random seeds and report the performance on the test part of the dataset. For each dataset, we use mean and standard deviation of test metrics (RMSE) over these random seeds to compare methods as described in Appendix . B.2 Ranks and metrics As we focus exclusively on regression tasks, we report Root Mean Squared Error (RMSE) as a primary metric. As mentioned in Appendix , all models and objectives are evaluated on 15 random seeds. We compare methods using two aggregation approaches. Relative Improvement. For each dataset, we compute the mean RMSE over 15 seeds and calculate its relative improvement (in %) over the baseline (typically the standard MSE objective). These values are shown in the right panel of Figure . Meanwhile, in the “Improv. (%)” columns of tables, we further aggregate them and report their median over the benchmark datasets. Ranks. For each dataset, we assign ranks based on a conservative order relation. Method A (with mean RMSE AμA_μ and standard deviation AσA_σ) is considered significantly better than method B if Aμ+Aσ<BμA_μ+A_σ<B_μ (We use this approach for finding W/T/L with A being the reference method). Intuitively, this approach groups methods into “tiers,” ensuring that negligible gains are not counted as wins on individual datasets. Specifically, ranks are computed independently for each dataset using a top-down procedure. We first sort the methods in ascending order of their mean RMSE. The best model is assigned rank 1 and serves as the initial reference. We then iterate through the remaining models: all methods that are not significantly worse than the current reference (according to the aforementioned rule) also receive rank 1. The first model that fails this criterion is assigned rank 2 and becomes the new reference for subsequent comparisons. This process continues until all models are ranked. The left panel of Figure illustrates the mean and standard deviation of these ranks across the benchmark. B.3 Support construction for distributional losses For every binning-based distributional objective we construct the support [a,b][a,b] as follows: (i) we compute the range between the 0.005- and 0.995-quantiles of the training labels distribution (we employ quantiles rather than min-max values to mitigate the impact of outliers while capturing the effective support of the distribution); (i) we expand this range by adding the same interval on both the left and right sides; (i) we then uniformly partition it into bins. The uniform binning is the most common in distributional regression research [1, 2] and performs stably in our experiments. B.4 Advanced models from Section Here, we provide implementation details for the experiments in Section . We use TabM-mini with PLE-Q embeddings [40], following the configuration and implementation of Gorishniy et al. [31]. We use TabPack equipped with numerical embeddings from Gorishniy et al. [33]. For all objectives on TabPack we do not add any new sampling parameters: We fix num_quantiles and num_bins as 384 (the same as the width) for all probabilistic objectives. Additionally, for AdaQR we fix regulizer as 0.2. We use the RealMLP [34] implementation from PyTabkit and add HL-Gaussian, quantile regression, AdaQR to that implementation. We use 100 Optuna TPE sampler tuning iterations as for other models and stop tuning at the 24-hour time limit. We use the “tabarena-new” hyperparameter search-space. B.5 AdaQR implementation The implementation of AdaQR includes the following details: 1. Alongside the quantile predictions, the model outputs a scalar logit aθ(x)a_θ(x) for the smoothing scale: σl(x)=|aθ(x)+b0|+ε,ε=10−5. _l(x)=|a_θ(x)+b_0|+ , =10^-5. Before training, we compute a fixed offset b0b_0 so that aθ0(x)+b0a_ _0(x)+b_0 has mean 0.50.5 over the training set, separately for each ensemble member. All scales are expressed in standardized target units. 2. The parameter regulizer in our implementation corresponds to 1−λπ1-λ π, rather than λ itself in the AdaQR objective (Section ). Its tuning space is provided in Appendix . 3. We use equally spaced quantile levels from 0.0010.001 to 0.9990.999. At inference, we average the predicted quantiles, as in standard quantile regression; the smoothing-scale output is not used. B.6 Hyperparameter Tuning Spaces In this section, we provide tuning spaces for the evaluated objectives and methods. Tables – summarize the optimizer and architecture tuning spaces for different models; objective-specific hyperparameter tuning spaces are provided in Table . Parameter names correspond to their implementation in the source code. In the case of the RealMLP backbone we use the “tabarena-new” hyperparameter search-space for RealMLP-specific parameters. Table 7: The hyperparameter tuning space for MLP. Here, (A) is the default space and (B) is used when evaluating Decoder[U] and Decoder[N] (see Section ). Parameter Distribution # layers UniformInt[1, 6] Width (hidden size) UniformInt[64, 1024] Dropout rate 0.0, Uniform[0.0, 0.5] Learning rate (Muon) LogUniform[1e-4, 3e-2] Learning rate (AdamW) (A) LogUniform[3e-5, 1e-3], (B) LogUniform[3e-6, 5e-4] Weight decay (AdamW) LogUniform[5e-3, 5] Table 8: The hyperparameter tuning space for TabM. Parameter Distribution # layers UniformInt[1, 4] Width (hidden size) UniformInt[64, 1024] Dropout rate 0.0, Uniform[0.0, 0.5] Learning rate (Muon) LogUniform[1e-3, 1e-1] Learning rate (AdamW) LogUniform[5e-5, 3e-3] Weight decay (AdamW) LogUniform[1e-3, 1] d_embedding UniformInt[8, 32] n_bins UniformInt[2, 128] Table 9: The sampling space for TabPack Parameter Distribution # layers UniformInt[1, 3] Width (hidden size) 384 Dropout rate 0.0, Uniform[0.0, 0.5] Learning rate (Muon) LogUniform[1e-3, 1e-1] Learning rate (AdamW) LogUniform[1e-4, 5e-3] Weight decay (AdamW) LogUniform[1e-3, 1] d_embedding UniformInt[8, 32] init_scale LogUniform[1e-2, 1e2] Table 10: Hyperparameter tuning space for different regression objectives. The notation follows Section ; refer to the source code for any naming ambiguities. For TabPack, the numbers of bins and quantiles are fixed at 384, and AdaQR’s regulizer is fixed at 0.2 (Appendix ). Objective Parameter Distribution MSE — — MAE — — Huber delta LogUniform[1e-2, 100] NLL — — β-NLL beta Uniform[0, 1] MDN mixture_size UniformInt[2, 64] Hard Binning n_bins UniformInt[32, 1024] Soft Binning n_bins UniformInt[32, 1024] soft_sigma LogUniform[1e-2, 5] HL-Gaussian hl_num_bins UniformInt[32, 1024]; TabPack: 384 sigma LogUniform[1e-2, 5] Joint End-To-End alpha 0.0, LogUniform[1e-5, 0.5] lambda_auto LogUniform[1e-4, 1] lambda_kl LogUniform[1e-4, 1] GAR[MAE] alpha LogUniform[1e-2, 100] GAR[MSE] alpha LogUniform[1e-2, 100] Decoder[U] n_blocks UniformInt[1, 6] d_block UniformInt[64, 512] dropout 0.0, Uniform[0.0, 0.5] n_heads Categorical[1, 2, 4, 8] temperature LogUniform[0.01, 0.5] top_k UniformInt[1, 9] base 10 num_exponent_digits 1 num_mantissa_digits 5 Decoder[N] n_blocks UniformInt[1, 6] d_block UniformInt[64, 512] dropout 0.0, Uniform[0.0, 0.5] n_heads Categorical[1, 2, 4, 8] temperature LogUniform[0.01, 0.5] top_k UniformInt[1, 9] base Categorical[2, 4, 8, 10] length UniformInt[4, 16] Softmax-MSE min_value_relative Uniform[-0.1, 0.1] max_value_relative Uniform[0.9, 1.1] num_bins UniformInt[4, 256] Ordinal Entropy lambda_d LogUniform[1e-5, 0.1] ordinal_entropy_samples UniformInt[10, 1000] ROT-REG lambda_ot LogUniform[1e-3, 100] opt_transport_samples UniformInt[2, 200] gamma LogUniform[0.1, 100] y_dim UniformInt[2, 32] Quantile Loss num_quantiles UniformInt[32, 1024] AdaQR num_quantiles UniformInt[32, 1024]; TabPack: 384 regulizer Uniform[1e-3, 1.0]; TabPack: 0.2 AdaQR†AdaQR num_quantiles UniformInt[32, 1024] regulizer Uniform[1e-3, 1.0] scale_anchor_weight LogUniform[1e-3, 2] AdaQR w. global σl _l num_quantiles UniformInt[32, 1024] sigma LogUniform[1e-4, 8] Appendix C Detailed description of regression objectives Notation. We consider supervised regression tasks on tabular data. Let D=(xi,yi)i=1nD=\(x_i,y_i)\_i=1^n denote the dataset, where xi∈ℝdx_i ^d is the feature vector and yi∈ℝy_i is the label for the i-th datapoint. We denote the deep learning model by fθ:ℝd→ℝd′f_θ:R^d ^d , where d′d is the output dimension, which is not necessarily equal to 1. Let l(⋅,⋅)l(·,·) be the loss function. The loss on a given sample is then defined as l=l(fθ(x),y)l=l(f_θ(x),y). C.1 Losses C.1.1 Traditional losses The first group of objectives in our comparison includes canonical regression losses. Here, any values denoted by a hat ⋅ · correspond to predictions produced by our model. • MSE (Mean Squared Error, or l2l_2 loss) is the standard objective for most regression tasks: l=l2(y^,y)=(y^−y)2l=l_2( y,y)=( y-y)^2 • MAE (Mean Absolute Error, or l1l_1 loss) provides robustness against outliers by penalizing errors linearly: l=l1(y^,y)=|y^−y|l=l_1( y,y)=| y-y| • Huber loss [29] is a middle ground between MSE and MAE: l=lδ(y^,y)=12(y−y^)2,if |y^−y|≤δ⋅|y−y^|−12δ2,if |y^−y|>δl=l_δ( y,y)= cases 12(y- y)^2,&if | y-y|≤δ\\ δ·|y- y|- 12δ^2,&if | y-y|>δ cases C.1.2 Binning-based losses Among various regression objectives, the binning-based losses [41, 1, 2] stand out as a promising option for regression tasks and are reported to be beneficial across diverse domains [3]. The core idea is to discretize scalar targets using a grid of consecutive bins. Formally, consider a sample (x,y)∈D(x,y)∈ D. The regression label y is mapped to a probability vector =(p1,…,pd′)p=(p_1,…,p_d ) over the d′d bins on the support [a,b][a,b]. Here, the i-th bin is the interval [li,ri][l_i,r_i] with center cic_i. Given the model’s logits fθ(x)f_θ(x), let =softmax(fθ(x))q=softmax(f_θ(x)). The model is trained to approximate the target distribution p via the KL divergence: l=lKL(fθ(x),y)=DKL(∥).l=l_ KL(f_θ(x),y)=D_ KL(p\,\|\,q). At inference, model predictions are computed by taking the expectation over the bins: y^=∑i⋅ci. y= _iq_i· c_i. The particular form of the target distribution is typically the same for all datapoints and using different forms of the target distributions results in different regression objectives. Below, we describe several binning-based objectives that we include in the comparison. Hard Binning [41, 2] maps each y to a one-hot vector p by assigning it to the nearest bin center. Soft Binning [2] improves Hard Binning by performing soft mapping to the dense probability vector: =softmax(−|c1−y|22σ2,…,−|cd′−y|22σ2),p=softmax (- |c_1-y|^22σ^2,…,- |c_d -y|^22σ^2 ), where σ is a global (shared across the dataset) hyperparameter controlling the spread of probability mass over the bins. HL-Gaussian [1] constructs p by discretizing the truncated Gaussian distribution with mean y and fixed variance σ2σ^2 over [a,b][a,b]. Specifically, given the density ρ(t∣y)=1Z2πσ2e−(t−y)22σ2ρ(t y)= 1Z 2πσ^2e^- (t-y)^22σ^2 the bin probabilities are i=1Z(Φ(ri−yσ)−Φ(li−yσ))p_i= 1Z ( ( r_i-yσ )- ( l_i-yσ ) ) where Φ is the cdf-function for (0,1)N(0,1) and Z is the normalization constant. Similarly to Soft Binning, σ is a global hyperparameter that effectively determines the smoothness of p. Joint end-to-end encoding [2] is a binning-based objective where the target distribution p itself is learned. Instead of using a fixed rule (as in the methods above), the mapping y↦y is parameterized by an auxiliary autoencoder-style model and trained end-to-end along with the main model. This is achieved by adding a reconstruction term (recovering y from p) to the DKL(∥)D_ KL(p\,\|\,q) loss. C.1.3 Quantile loss Quantile loss [9] offers an alternative approach to modeling conditional distributions p(y∣x)p(y x) by directly predicting specific quantiles of the target variable. Specifically, for a given input x, the model predicts N quantiles y^τ1,y^τ2,…,y^τN y_ _1, y_ _2,…, y_ _N corresponding to quantile levels τ1,τ2,…,τN∈(0,1) _1, _2,…, _N∈(0,1). Each quantile prediction y^τi y_ _i is trained using the pinball loss function: ρτi(y,y^τi)=τi(y−y^τi),if y≥y^τi(1−τi)(y^τi−y),if y<y^τi _ _i(y, y_ _i)= cases _i(y- y_ _i),&if y≥ y_ _i\\ (1- _i)( y_ _i-y),&if y< y_ _i cases where τi∈(0,1) _i∈(0,1) represents the target quantile level. The overall loss function is then computed as the average pinball loss across all N quantiles: l=lQR(fθ(x),y)=1N∑i=1Nρτi(y,fθ,i(x))l=l_ QR(f_θ(x),y)= 1N _i=1^N _ _i(y,f_θ,i(x)) where fθ,i(x)f_θ,i(x) denotes the model’s prediction for the i-th quantile. During inference, the final prediction y y is typically obtained by taking the average of all predicted quantiles: y^=1N∑i=1Ny^τi. y= 1N _i=1^N y_ _i. This approach provides a more direct characterization of the conditional distribution compared to binning methods, while avoiding the discretization of the output space. C.1.4 Modeling full conditional distributions in the arbitrary form An alternative paradigm to solve a regression problem has been employed in several recent works [16, 4] that model the conditional distributions p(y|x)p(y|x) without any assumptions about their parametric form. In our experimental comparison, we include the most recent approach proposed by Song and Bahri [4] that relies on the autoregressive decoder. Specifically, the approach uses a small autoregressive decoding head on top of an MLP backbone. Both MLP and decoder are trained jointly for the next-token (where tokens represent digits) prediction with a cross-entropy loss. The decoder predicts the sequence of tokens t1,…,tkt_1,…,t_k that represent the target y in an autoregressive manner. Formally, let ϕ(x)φ(x) be the representation produced by the MLP encoder. The model estimates the conditional distribution p(y∣x)p(y x) via the following factorization: p(y∣x)=∏i=1kp(ti∣t1,…,ti−1,ϕ(x)).p(y x)= _i=1^kp(t_i t_1,…,t_i-1,φ(x)). In our study, we evaluate two versions of this decoder described by the method’s authors. In the normalized variant, targets are mapped to the [0,1][0,1] interval using min-max transformation before tokenization. In the unnormalized variant, targets are represented as floating-point numbers in a base-B system: y=±Be⋅my=± B^e· m, where the exponent e and mantissa m∈[1,B)m∈[1,B) are predicted. C.1.5 Other losses In this section we list several additional regression objectives that are included in our evaluation. • GAR (Gradient aligned regression) [5] combines point-wise and pair-wise terms to better match the gradient and magnitude of the target function. • Softmax-MSE [2] uses a softmax binning head followed by computing the expectation and MSE loss: l=(∑i⋅ci−y)2l= ( _iq_i· c_i-y )^2 where cic_i and q follow the binning losses notation from Section . • Ordinal Entropy loss [6] is an objective that enables NNs to learn better feature representations Z (specifically, to maximize mutual information ℐ(Z,Y)=ℋ(Z)−ℋ(Z∣Y)I(Z,Y)=H(Z)-H(Z Y)), by adding two additional terms to the initial regression loss. • ROT-REG [8] is an optimal transport based regularizer that similarly allows the model to enhance learned representations by tightening them. • NLL assumes p(y|x)=(μ(x),σ2(x))p(y|x)=N(μ(x),σ^2(x)) and minimizes the negative log-likelihood: l=12logσ^2+(μ^−y)22σ^2.l= 12 σ^2+ ( μ-y)^22 σ^2. • β-NLL [22] extends NLL by reweighting each sample by ⌊σ^2β⌋ σ^2β to improve optimization stability, where ⌊⋅⌋ · denotes the stop-gradient operation. • Mixture Density Networks [15] model the conditional distribution p(y|x)p(y|x) as a Gaussian mixture. Appendix D Detailed description of the dataset used in Section To evaluate the robustness of our method under strictly controlled noise conditions, we construct both homoskedastic and heteroskedastic synthetic datasets. Below are the details regarding the process of generation. Dataset Size and Splits: The total dataset size is N=50,000N=50,000. We split this into 20,000 training examples, 15,000 validation examples, and 15,000 test examples. Importantly, while the training targets contain injected noise, the validation and test targets are completely noise-free (ycleany_clean). This allows us to accurately evaluate how well the models uncover the true underlying conditional expectation. Base Network Architecture: We use a standardized Multi-Layer Perceptron (MLP) generator to create non-linear mappings. To ensure stable scales across different initializations, the outputs of this generator are explicitly standardized to have zero mean and unit variance using statistics calculated over an initial batch of 10,00010,000 Gaussian noise samples. All MLPs use hidden dimensions of 256 and ReLU activations. Feature Generation (X): 1. We sample a 40-dimensional latent from the standard multivariate Gaussian distribution: z∼(0,I40)z (0,I_40). 2. We map z through a 3-layer MLP generator to produce a 100-dimensional informative feature vector xreal∈ℝ100x_real ^100. We use this step to imitate the complex relationships between features that are often found in tabular data. 3. To introduce irrelevant features, we sample 16-dimensional standard Gaussian noise xnoisy∼(0,I16)x_noisy (0,I_16). 4. The final feature vector is the concatenation of the informative and noisy features: x=[xreal,xnoisy]∈ℝ116x=[x_real,x_noisy] ^116. Target Generation (Y): The clean target values, f(x)f(x), are generated by passing the informative features xrealx_real through a 2-layer MLP fθf_θ. To create the noisy training targets ynoisyy_noisy, we inject Gaussian noise ϵ∼(0,1)ε (0,1) according to one of two settings: • Homoskedastic Dataset: The variance is constant across the entire input space. ynoisy=yclean+σϵ,where σ=4.0y_noisy=y_clean+σε, σ=4.0 • Heteroskedastic Dataset: The standard deviation σ(x)σ(x) depends non-linearly on the informative features. We pass xrealx_real through another 2-layer MLP gϕg_φ to obtain a scalar. This scalar is pushed through a Standard Normal Cumulative Distribution Function Φ to bound it in (0,1)(0,1). We then scale, shift, and exponentiate it to create a smooth, variable standard deviation profile: σ(x)=exp(Φ(gϕ(xreal))×4.5−2.0)σ(x)= ( (g_φ(x_real))× 4.5-2.0\ ) ynoisy=yclean+σ(x)ϵy_noisy=y_clean+σ(x)ε To ensure exact reproducibility, the process uses manual random seeds before any generation. Appendix E Quantile loss and Energy Distance There exists a well-known relation between the quantile loss and the Continuous Ranked Probability Score (CRPS) [42, 43]: CRPS(F,y) (F,y) =∫−∞+∞(F(z)−z≥y)2dz = _-∞^+∞ (F(z)- 1\z≥ y\ )^2\,dz =2∫01ρτ(y,QF(τ))dτ≈2⋅lQR(QF(⋅),y), =2 _0^1 _τ(y,Q_F(τ))\,dτ≈ 2· l_QR(Q_F(·),y), where F is a CDF, QFQ_F is its quantile function, and ρτ _τ is the pinball loss (Section ). The approximation replaces the integral over quantile levels with an average over the finite uniform grid used by our models. Thus, up to this discretization and a constant factor, minimizing the quantile loss is equivalent to minimizing CRPS. For real-valued random variables with finite first moments, the squared Energy Distance is [37] 2(F,G)=2|X−Y|−|X−X′|−|Y−Y′|,D^2(F,G)=2E|X-Y|-E|X-X |-E|Y-Y |, where X,X′∼FX,X F and Y,Y′∼GY,Y G are mutually independent. When G=δyG= _y, the Dirac distribution concentrated at y, this gives CRPS(F,y)=|X−y|−12|X−X′|=122(F,δy).CRPS(F,y)=E|X-y|- 12E|X-X |= 12D^2(F, _y). This identity provides the Energy Distance interpretation of quantile loss used in Section : the predicted distribution is compared with a point mass at the observed target. AdaQR replaces this point mass with an input-dependent Gaussian smoothing distribution. Appendix F AdaQR: Full Objective and Proof of Proposition F.1 Full objective in terms of quantiles Denoting the predicted quantiles as qiq_i at levels 0≤τi≤10≤ _i≤ 1, i=1,…,Ni=1,…,N, the AdaQR loss expands as follows: 122(F,(y,σl2))+λσl 12D^2 (F,\,N(y, _l^2) )+λ _l = =\, X,X′∼F[Y,Y′∼(y,σl2)[‖X−Y‖−12‖X−X′‖−12‖Y−Y′‖]]+λσl E _X,X F [ E _Y,Y (y, _l^2) [\|X-Y\|- 12\|X-X \|- 12\|Y-Y \| ] ]+λ _l ≈ ≈\, Y,Y′∼(y,σl2)[1N∑i=1N‖qi−Y‖−2N∑i=1Nτiqi+1N∑i=1Nqi−12‖Y−Y′‖]+λσl E _Y,Y (y, _l^2) [ 1NΣ _i=1^N\|q_i-Y\|- 2NΣ _i=1^N _iq_i+ 1NΣ _i=1^Nq_i- 12E\|Y-Y \| ]+λ _l = =\, 1N∑i=1N((qi−y)(2Φ(qi−yσl)−1)+2σl2πe−(qi−y)22σl2)−2N∑i=1Nτiqi+1N∑i=1Nqi−σlπ+λσl, 1N _i=1^N ((q_i-y) (2 \! ( q_i-y _l )-1 )+ 2 _l 2πe^- (q_i-y)^22 _l^2 )- 2N _i=1^N _iq_i+ 1N _i=1^Nq_i- _l π+λ _l, (2) where Φ(⋅) (·) is the standard normal cdf. F.2 Proof of Proposition Proposition 1 (AdaQR learns proportional uncertainty). Under the following assumptions: (A1) the number of quantiles is large enough to ignore discretization effects; (A2) both the predicted distribution F and the target distribution are Gaussian; (A3) the optimizer converges to Bayes-optimal predictions; (A4) the label noise is Gaussian: y|ytrue∼(ytrue,σt2)y y_true (y_true,\, _t^2), (A5) 0<λ<1/π0<λ<1/ π and σt>0 _t>0 the AdaQR objective () has a unique optimum in σl _l, and the optimal smoothing parameter satisfies: σl2=(1−πλ)21−(1−πλ)2σt2. _l^2= (1- πλ)^21-(1- πλ)^2\, _t^2. (3) In particular, σl2∝σt2 _l^2 _t^2, so AdaQR correctly captures the relative heteroskedasticity of the data. Proof. Fix an input x and define the expected objective ℒ¯(σq,σl)=y|x[ℒ(F,y,σl)]. L( _q, _l)=E_y x\! [L(F,y, _l) ]. By (A2) and (A3), we model F=(ytrue,σq2)F=N(y_true, _q^2), and by (A4), for each observed label y the target distribution is Gy=(y,σl2)G_y=N(y, _l^2) with y∼(ytrue,σt2)y (y_true, _t^2). The three terms of the Energy Distance simplify via the Half-Normal expectation formula |Z|=2/πσE|Z|= 2/π\,σ for Z∼(0,σ2)Z (0,σ^2). Note that the relevant quantities are the centered differences X−ytrueX-y_true, Y−yY-y, and y−ytruey-y_true, which are independent and zero-mean (by (A2) and (A4)): • X−Y=(X−ytrue)−(y−ytrue)−(Y−y)X-Y=(X-y_true)-(y-y_true)-(Y-y) is (0,σq2+σt2+σl2)N(0,\, _q^2+ _t^2+ _l^2); • X−X′=(X−ytrue)−(X′−ytrue)X-X =(X-y_true)-(X -y_true) is (0, 2σq2)N(0,\,2 _q^2), since X,X′X,X are i.i.d. samples from F; • conditionally on a shared label y, the samples Y,Y′|yY,Y y are i.i.d. from (y,σl2)N(y, _l^2), so Y−Y′|yY-Y y is (0, 2σl2)N(0,\,2 _l^2); crucially, this conditional law does not depend on y, hence it also holds marginally after averaging over y|xy x. Combining these and taking the expectation over y|xy x, the first term of the expected objective equals: y|x[122(F,Gy)]=1π(2(σq2+σt2+σl2)−σq−σl),E_y x\! [ 12D^2(F,G_y) ]= 1 π ( 2( _q^2+ _t^2+ _l^2)- _q- _l ), so, under (A1)–(A5), the expected objective is: ℒ¯(σq,σl)=1π(2(σq2+σt2+σl2)−σq−σl)+λσl. L( _q, _l)= 1 π ( 2( _q^2+ _t^2+ _l^2)- _q- _l )+λ _l. Setting partial derivatives to zero: ∂ℒ¯∂σq=1π(2σq2(σq2+σt2+σl2)−1)=0,∂ℒ¯∂σl=1π(2σl2(σq2+σt2+σl2)−1)+λ=0. cases ∂ L∂ _q= 1 π\! ( 2 _q 2( _q^2+ _t^2+ _l^2)-1 )=0,\\[8.0pt] ∂ L∂ _l= 1 π\! ( 2 _l 2( _q^2+ _t^2+ _l^2)-1 )+λ=0. cases The first equation gives σq2=σl2+σt2 _q^2= _l^2+ _t^2. Substituting into the second: 4πσl2=(1π−λ)2⋅4(σl2+σt2)⟹σl2=(1−πλ)2(σl2+σt2), 4π _l^2= ( 1 π-λ )^2· 4( _l^2+ _t^2) _l^2=(1- πλ)^2( _l^2+ _t^2), and solving for σl2 _l^2 yields (). It remains to verify that this stationary point is the unique minimum. For σt>0 _t>0, the function 2(σq2+σl2+σt2) 2( _q^2+ _l^2+ _t^2) is strictly convex in (σq,σl)( _q, _l) because its Hessian is positive definite everywhere. The remaining terms, −σq/π+(−1/π+λ)σl- _q/ π+(-1/ π+λ) _l, are linear. Hence ℒ¯ L is strictly convex, and under (A5) the stationary point found above — which is positive — is its unique global minimum. ∎ Appendix G Scale regularization for RealMLP For RealMLP, we augment AdaQR with an auxiliary scale regularizer, denoting the resulting objective by AdaQR†. Below, we describe its motivation, formulation, and effect on uncertainty estimation. G.1 Uncertainty estimation in RealMLP Proposition characterizes the population optimum under Gaussian assumptions. In practice, however, there is no guarantee that training reaches this optimum. In particular, a model can correctly rank inputs by uncertainty while producing inaccurate absolute scales. We compare TabM and RealMLP on our heteroskedastic synthetic dataset, where the true data uncertainty is known, converting smoothing scales into noise-scale estimates via (). TabM already recovers the underlying data uncertainty reasonably well and consistently across ensemble members (Figure (a)). RealMLP shows larger scale deviations and greater variation across members (Figure (b)), suggesting that these difficulties limit the gains of vanilla AdaQR on this backbone. This motivates an auxiliary regularizer for the scales inferred from the quantile spread and the smoothing output. G.2 Auxiliary scale regularization We augment AdaQR with a regularizer that provides a direct training signal for the predicted scales. Following the notation of the proof in Appendix , let r=1−πλr=1- πλ, with 0<r<10<r<1. Under the assumptions of Proposition , the optimal scales satisfy σt=1−r2σq=1−r2rσl. _t= 1-r^2\, _q= 1-r^2r\, _l. This motivates two estimates of the noise standard deviation: sq=1−r2σ^q,sσ=1−r2rσl,s_q= 1-r^2\, σ_q, s_σ= 1-r^2r\, _l, where, for N predicted quantiles qjq_j at levels τj _j, we estimate the quantile-distribution scale as σ^q2=1N∑j=1N(qj−q¯)21N∑j=1N[Φ−1(τj)]2,q¯=1N∑j=1Nqj. σ_q^2= 1N _j=1^N(q_j- q)^2 1N _j=1^N[ ^-1( _j)]^2, q= 1N _j=1^Nq_j. Here, Φ is the standard normal CDF. The denominator accounts for the finite, symmetric quantile grid, recovering σq2 _q^2 exactly for Gaussian quantiles. We supervise both estimates using CRPS evaluated at Gaussian CDFs Fμ,s(z)=Φ(z−μs),μ=stopgrad(q¯),F_μ,s(z)= \! ( z-μs ), μ=stopgrad( q), where stopgradstopgrad preserves its argument’s value but blocks gradients through it. The Gaussian CRPS has the closed form CRPS(Fμ,s,y)=e(2Φ(e/s)−1)+2sϕ(e/s)−sπ,e=y−μ,CRPS(F_μ,s,y)=e (2 (e/s)-1 )+2sφ(e/s)- s π, e=y-μ, where ϕφ is the standard normal density. The resulting objective, denoted by AdaQR†, is ℒAdaQR†=ℒAdaQR+γ2[CRPS(Fμ,sq,y)+CRPS(Fμ,sσ,y)],L_AdaQR =L_AdaQR+ γ2 [CRPS(F_μ,s_q,y)+CRPS(F_μ,s_σ,y) ], averaged over the training batch, with γ≥0γ≥ 0 controlling the regularization strength. Detaching q¯ q blocks the auxiliary gradient through the predicted mean, while centering the quantiles makes the spread estimate invariant to a common shift. The regularizer thus directly trains the quantile spread and the smoothing scale, although it can still affect point predictions through the shared backbone. Under the assumptions of Proposition , both inferred scales equal σt _t at the population optimum. With the conditional mean correctly specified, s=σts= _t also minimizes the expected Gaussian CRPS. The auxiliary terms therefore share the same idealized scale optimum as AdaQR. The regularizer requires no architectural changes: we retain AdaQR’s scalar smoothing-scale output. At inference time, the point prediction remains the average of the predicted quantiles. G.3 Effect on RealMLP As shown in Figure (b) and (c), AdaQR† produces noise-scale estimates that more closely match the true data uncertainty and are more consistent across ensemble members. The improvement primarily concerns the absolute scales, as vanilla AdaQR already ranks inputs by uncertainty reasonably well. These results suggest that the auxiliary regularizer helps RealMLP recover the scale relationship predicted by our theoretical analysis. Figure 4: Data uncertainty estimated by AdaQR versus the true data uncertainty on a heteroskedastic synthetic dataset, shown on logarithmic axes. Both uncertainties are expressed as variances. AdaQR estimates are obtained using (), following the theoretical results in Section . Panels compare (a) TabM, (b) RealMLP, and (c) RealMLP with scale-regularized AdaQR†. Gray points show individual test inputs for one ensemble member; the orange line connects this member’s median estimates in equal-count bins of true uncertainty. Each blue line shows the corresponding bin medians for another ensemble member. The shaded band spans the minimum and maximum of these medians, rather than a confidence interval. Dashed diagonals indicate perfect agreement; downward triangles mark estimates below the displayed range. All three models capture relative variation in data uncertainty. Scale regularization reduces RealMLP’s deviations at low uncertainty and improves consistency across ensemble members. Appendix H Uncertainty Estimation Estimating data uncertainty. Following Kartashev et al. [11], we use a CatBoost model with dual outputs (μ(x)μ(x) and σ(x)σ(x)) trained via Gaussian negative log-likelihood on the real datasets, utilizing the predicted σ(x)σ(x) as our uncertainty estimate. To prevent data leakage, this estimator is fit exclusively on the training split and applied to validation and test points during inference. For the synthetic data, we use the ground-truth σ(x)σ(x). Figure extends the comparison with CatBoost to nine additional datasets. We use a single TabM [31] checkpoint with 32 branches. For each test input, we convert each branch’s learned scale to a data variance using () and average the resulting variances. Both methods’ estimates are expressed in standardized-target variance units. The positive correlations provide further evidence that AdaQR learns meaningful input-dependent uncertainty. Figure 5: AdaQR versus CatBoost data uncertainty on nine additional real-world datasets. Points represent test inputs; AdaQR estimates are averaged across TabM branches after conversion to variances. Both axes are logarithmic, and dashed lines indicate equal estimates (y=xy=x). ρS _S denotes Spearman’s rank correlation, computed over all test inputs; at most 8,000 points are displayed per dataset. Appendix I Extended Results In this section we present results from Section with Tiny subset from BeyondArena [30] included (Table ), full results from cohort based early stopping section (Table ) and all per-dataset results used in this paper (Table ). Table 11: Improv. (%) denotes the median relative improvement in RMSE compared to the standard MSE objective. W/T/L summarizes comparisons with MSE using the rule in Appendix . Model Objective Improv. (%) W / T / L MLP Quantile Loss +0.55 21 / 15 / 5 AdaQR +0.74 19 / 17 / 5 NLL +0.25 10 / 23 / 8 beta-NLL +0.32 12 / 21 / 8 TabPack Quantile Loss +0.46 18 / 21 / 2 AdaQR +0.45 19 / 21 / 1 RealMLP Quantile Loss +0.40 20 / 16 / 5 AdaQR† +0.90 25 / 9 / 7 TabM Quantile Loss +0.58 20 / 16 / 5 AdaQR +0.93 20 / 17 / 4 Table 12: Cohort-based vs. global checkpoint selection for an MLP trained with quantile loss or AdaQR on the seven largest BeyondArena datasets within our 41-dataset evaluation subset. Values are mean ± standard deviation of test RMSE over 15 random seeds; lower is better. Global denotes a single checkpoint shared by all test points; K=5K=5 and K=10K=10 denote per-cohort selection with five and ten cohorts. All checkpoints are selected on validation data. Objective Cohorts Global K=5K=5 K=10K=10 Electric Motor Temperature Quantile Uncertainty 2.64±0.132.64± 0.13 2.65±0.132.65± 0.13 2.70±0.222.70± 0.22 Quantile Random 2.64±0.132.64± 0.13 2.64±0.142.64± 0.14 2.64±0.132.64± 0.13 AdaQR Uncertainty 2.74±0.162.74± 0.16 2.72±0.152.72± 0.15 2.73±0.142.73± 0.14 Mercari Quantile Uncertainty 0.47298±0.002510.47298± 0.00251 0.47282±0.001810.47282± 0.00181 0.47296±0.001880.47296± 0.00188 Quantile Random 0.47298±0.002510.47298± 0.00251 0.47289±0.001780.47289± 0.00178 0.47297±0.001820.47297± 0.00182 AdaQR Uncertainty 0.47230±0.000710.47230± 0.00071 0.47228±0.000660.47228± 0.00066 0.47234±0.000740.47234± 0.00074 Climate Quantile Uncertainty 1.4541±0.00981.4541± 0.0098 1.4515±0.01041.4515± 0.0104 1.4524±0.01001.4524± 0.0100 Quantile Random 1.4541±0.00981.4541± 0.0098 1.4530±0.01031.4530± 0.0103 1.4539±0.01001.4539± 0.0100 AdaQR Uncertainty 1.4552±0.00981.4552± 0.0098 1.4539±0.00871.4539± 0.0087 1.4540±0.00861.4540± 0.0086 Rossmann Quantile Uncertainty 1354±541354± 54 1287±441287± 44 1280±431280± 43 Quantile Random 1354±541354± 54 1354±551354± 55 1355±551355± 55 AdaQR Uncertainty 1457±1251457± 125 1405±1121405± 112 1390±1081390± 108 Allstate Quantile Uncertainty 0.53917±0.000240.53917± 0.00024 0.53883±0.000200.53883± 0.00020 0.53882±0.000210.53882± 0.00021 Quantile Random 0.53917±0.000240.53917± 0.00024 0.53930±0.000220.53930± 0.00022 0.53936±0.000190.53936± 0.00019 AdaQR Uncertainty 0.53923±0.000130.53923± 0.00013 0.53888±0.000200.53888± 0.00020 0.53882±0.000170.53882± 0.00017 5G Energy Quantile Uncertainty 4.0562±0.02664.0562± 0.0266 4.0473±0.01994.0473± 0.0199 4.0496±0.00944.0496± 0.0094 Quantile Random 4.0562±0.02664.0562± 0.0266 4.0599±0.02104.0599± 0.0210 4.0669±0.01844.0669± 0.0184 AdaQR Uncertainty 4.0368±0.01204.0368± 0.0120 4.0358±0.01834.0358± 0.0183 4.0449±0.01474.0449± 0.0147 SF Permit Time Quantile Uncertainty 1.4600±0.00771.4600± 0.0077 1.4245±0.00531.4245± 0.0053 1.4178±0.00361.4178± 0.0036 Quantile Random 1.4600±0.00771.4600± 0.0077 1.4606±0.00331.4606± 0.0033 1.4594±0.00361.4594± 0.0036 AdaQR Uncertainty 1.4454±0.01621.4454± 0.0162 1.4389±0.01121.4389± 0.0112 1.4261±0.01021.4261± 0.0102 Table 13: Dataset-wise MLP results with AdamW on 41 Beyond datasets (split 0). Test RMSE is mean ± sample standard deviation (ddof=1ddof=1). Single uses 15 seeds; E5 uses three disjoint ensembles of five seeds. 5g_energy_consumption Objective Single E5 MSE 4.10±0.024.10± 0.02 4.054±0.0084.054± 0.008 HL-Gaussian 4.07±0.044.07± 0.04 4.04±0.014.04± 0.01 Quantile Loss 4.10±0.034.10± 0.03 4.06±0.014.06± 0.01 airfoil_self_noise Objective Single E5 MSE 1.85±0.061.85± 0.06 1.78±0.031.78± 0.03 HL-Gaussian 1.88±0.041.88± 0.04 1.84±0.021.84± 0.02 Quantile Loss 1.91±0.031.91± 0.03 1.87±0.021.87± 0.02 allstate_claims_severity Objective Single E5 MSE 0.5408±0.00020.5408± 0.0002 0.5401±0.00010.5401± 0.0001 HL-Gaussian 0.5398±0.00010.5398± 0.0001 0.5392±0.00000.5392± 0.0000 Quantile Loss 0.5401±0.00020.5401± 0.0002 0.5396±0.00010.5396± 0.0001 blood_tests_drink_prediction Objective Single E5 MSE 0.68±0.010.68± 0.01 0.680±0.0100.680± 0.010 HL-Gaussian 0.71±0.030.71± 0.03 0.70±0.020.70± 0.02 Quantile Loss 0.674±0.0070.674± 0.007 0.673±0.0040.673± 0.004 body_density_prediction Objective Single E5 MSE 0.0127±0.00050.0127± 0.0005 0.0125±0.00030.0125± 0.0003 HL-Gaussian 0.0130±0.00030.0130± 0.0003 0.0129±0.00010.0129± 0.0001 Quantile Loss 0.0128±0.00030.0128± 0.0003 0.0127±0.00000.0127± 0.0000 california_house_prices_2020 Objective Single E5 MSE 0.280±0.0060.280± 0.006 0.271±0.0010.271± 0.001 HL-Gaussian 0.271±0.0030.271± 0.003 0.2643±0.00100.2643± 0.0010 Quantile Loss 0.259±0.0050.259± 0.005 0.254±0.0010.254± 0.001 cirrhosis_patient_survival_prediction Objective Single E5 MSE 0.71±0.020.71± 0.02 0.697±0.0090.697± 0.009 HL-Gaussian 0.68±0.010.68± 0.01 0.677±0.0060.677± 0.006 Quantile Loss 0.72±0.020.72± 0.02 0.707±0.0050.707± 0.005 climate_model_weather_forecasting_1m Objective Single E5 MSE 1.49±0.011.49± 0.01 1.460±0.0041.460± 0.004 HL-Gaussian 1.48±0.021.48± 0.02 1.470±0.0081.470± 0.008 Quantile Loss 1.47±0.011.47± 0.01 1.453±0.0081.453± 0.008 coffee_rating_prediction Objective Single E5 MSE 1.52±0.051.52± 0.05 1.50±0.031.50± 0.03 HL-Gaussian 1.54±0.051.54± 0.05 1.522±0.0071.522± 0.007 Quantile Loss 1.50±0.051.50± 0.05 1.48±0.021.48± 0.02 concrete_compressive_strength Objective Single E5 MSE 4.9±0.14.9± 0.1 4.75±0.044.75± 0.04 HL-Gaussian 4.9±0.24.9± 0.2 4.78±0.034.78± 0.03 Quantile Loss 4.9±0.24.9± 0.2 4.67±0.094.67± 0.09 diamonds Objective Single E5 MSE 0.093±0.0020.093± 0.002 0.0894±0.00050.0894± 0.0005 HL-Gaussian 0.0877±0.00050.0877± 0.0005 0.0861±0.00010.0861± 0.0001 Quantile Loss 0.093±0.0020.093± 0.002 0.089±0.0010.089± 0.001 early_learning_predictors Objective Single E5 MSE 13.44±0.0413.44± 0.04 13.375±0.00813.375± 0.008 HL-Gaussian 13.36±0.0313.36± 0.03 13.28±0.0113.28± 0.01 Quantile Loss 13.40±0.0713.40± 0.07 13.313±0.00913.313± 0.009 electric_motor_temperature_prediction Objective Single E5 MSE 3.0±0.13.0± 0.1 2.88±0.072.88± 0.07 HL-Gaussian 2.78±0.072.78± 0.07 2.70±0.012.70± 0.01 Quantile Loss 3.1±0.23.1± 0.2 2.86±0.092.86± 0.09 fiat_500 Objective Single E5 MSE 891±18891± 18 885±8885± 8 HL-Gaussian 822±8822± 8 816.3±0.8816.3± 0.8 Quantile Loss 878±16878± 16 871±3871± 3 food_delivery_time Objective Single E5 MSE 8.07±0.018.07± 0.01 8.036±0.0058.036± 0.005 HL-Gaussian 8.12±0.018.12± 0.01 8.100±0.0058.100± 0.005 Quantile Loss 8.10±0.018.10± 0.01 8.079±0.0058.079± 0.005 forest_fires Objective Single E5 MSE 1.37±0.011.37± 0.01 1.361±0.0021.361± 0.002 HL-Gaussian 1.356±0.0091.356± 0.009 1.3546±0.00011.3546± 0.0001 Quantile Loss 1.351±0.0031.351± 0.003 1.3496±0.00061.3496± 0.0006 garments_worker_productivity Objective Single E5 MSE 0.101±0.0090.101± 0.009 0.096±0.0020.096± 0.002 HL-Gaussian 0.095±0.0030.095± 0.003 0.093±0.0010.093± 0.001 Quantile Loss 0.096±0.0070.096± 0.007 0.090±0.0010.090± 0.001 healthcare_insurance_expenses Objective Single E5 MSE 0.401±0.0020.401± 0.002 0.399±0.0010.399± 0.001 HL-Gaussian 0.398±0.0040.398± 0.004 0.395±0.0020.395± 0.002 Quantile Loss 0.389±0.0020.389± 0.002 0.3863±0.00090.3863± 0.0009 houses Objective Single E5 MSE 0.229±0.0020.229± 0.002 0.2252±0.00070.2252± 0.0007 HL-Gaussian 0.227±0.0020.227± 0.002 0.224±0.0010.224± 0.001 Quantile Loss 0.228±0.0020.228± 0.002 0.2245±0.00070.2245± 0.0007 immoscout_german_house_prices Objective Single E5 MSE 0.406±0.0030.406± 0.003 0.3977±0.00080.3977± 0.0008 HL-Gaussian 0.407±0.0020.407± 0.002 0.401±0.0010.401± 0.001 Quantile Loss 0.406±0.0030.406± 0.003 0.397±0.0010.397± 0.001 mercari_price_suggestion_1m Objective Single E5 MSE 0.4756±0.00050.4756± 0.0005 0.4635±0.00080.4635± 0.0008 HL-Gaussian 0.4731±0.00050.4731± 0.0005 0.4619±0.00000.4619± 0.0000 Quantile Loss 0.4752±0.00070.4752± 0.0007 0.4625±0.00030.4625± 0.0003 Table 14: Dataset-wise MLP results with AdamW (continued). Test RMSE: mean ± sample standard deviation. Single: 15 seeds; E5: three ensembles of five. mercedes_benz_greener_manufacturing Objective Single E5 MSE 8.45±0.078.45± 0.07 8.42±0.028.42± 0.02 HL-Gaussian 8.39±0.078.39± 0.07 8.34±0.038.34± 0.03 Quantile Loss 8.34±0.038.34± 0.03 8.325±0.0098.325± 0.009 miami_housing Objective Single E5 MSE 0.1498±0.00090.1498± 0.0009 0.1482±0.00040.1482± 0.0004 HL-Gaussian 0.149±0.0010.149± 0.001 0.1465±0.00080.1465± 0.0008 Quantile Loss 0.1488±0.00080.1488± 0.0008 0.1472±0.00030.1472± 0.0003 mutual_funds_india Objective Single E5 MSE 6.0±0.16.0± 0.1 5.83±0.085.83± 0.08 HL-Gaussian 5.9±0.15.9± 0.1 5.72±0.035.72± 0.03 Quantile Loss 5.9±0.15.9± 0.1 5.78±0.035.78± 0.03 obesity_estimation Objective Single E5 MSE 4.94±0.024.94± 0.02 4.919±0.0074.919± 0.007 HL-Gaussian 4.97±0.044.97± 0.04 4.94±0.024.94± 0.02 Quantile Loss 4.90±0.024.90± 0.02 4.899±0.0064.899± 0.006 physiochemical_protein Objective Single E5 MSE 3.52±0.013.52± 0.01 3.431±0.0063.431± 0.006 HL-Gaussian 3.43±0.023.43± 0.02 3.35±0.023.35± 0.02 Quantile Loss 3.52±0.033.52± 0.03 3.43±0.023.43± 0.02 qsar_aquatic_toxicity Objective Single E5 MSE 1.16±0.021.16± 0.02 1.149±0.0091.149± 0.009 HL-Gaussian 1.18±0.021.18± 0.02 1.17±0.011.17± 0.01 Quantile Loss 1.18±0.021.18± 0.02 1.173±0.0031.173± 0.003 qsar_fish_toxicity Objective Single E5 MSE 0.95±0.020.95± 0.02 0.938±0.0070.938± 0.007 HL-Gaussian 0.93±0.020.93± 0.02 0.905±0.0030.905± 0.003 Quantile Loss 0.973±0.0100.973± 0.010 0.968±0.0030.968± 0.003 qsar_tid_11 Objective Single E5 MSE 0.813±0.0080.813± 0.008 0.797±0.0040.797± 0.004 HL-Gaussian 0.804±0.0050.804± 0.005 0.792±0.0030.792± 0.003 Quantile Loss 0.811±0.0070.811± 0.007 0.795±0.0020.795± 0.002 rossmann_store_sales Objective Single E5 MSE 1474±1261474± 126 1449±841449± 84 HL-Gaussian 1420±371420± 37 1414±111414± 11 Quantile Loss 1187±531187± 53 1134±221134± 22 santander_transaction_value Objective Single E5 MSE 1.43±0.021.43± 0.02 1.399±0.0041.399± 0.004 HL-Gaussian 1.407±0.0031.407± 0.003 1.406±0.0011.406± 0.001 Quantile Loss 1.410±0.0041.410± 0.004 1.406±0.0021.406± 0.002 sat11_hand_algo_runtime Objective Single E5 MSE 2.1±0.22.1± 0.2 1.99±0.031.99± 0.03 HL-Gaussian 2.05±0.052.05± 0.05 1.96±0.011.96± 0.01 Quantile Loss 2.2±0.12.2± 0.1 2.07±0.072.07± 0.07 sberbank_housing_market_forecasting Objective Single E5 MSE 0.26±0.010.26± 0.01 0.253±0.0020.253± 0.002 HL-Gaussian 0.253±0.0030.253± 0.003 0.250±0.0020.250± 0.002 Quantile Loss 0.256±0.0090.256± 0.009 0.246±0.0020.246± 0.002 sf_permit_time Objective Single E5 MSE 1.45±0.021.45± 0.02 1.45±0.011.45± 0.01 HL-Gaussian 1.46±0.011.46± 0.01 1.460±0.0041.460± 0.004 Quantile Loss 1.44±0.011.44± 0.01 1.440±0.0031.440± 0.003 student_portuguese_performance Objective Single E5 MSE 3.04±0.053.04± 0.05 2.98±0.042.98± 0.04 HL-Gaussian 2.93±0.042.93± 0.04 2.893±0.0062.893± 0.006 Quantile Loss 2.98±0.042.98± 0.04 2.929±0.0092.929± 0.009 superconductivity Objective Single E5 MSE 10.2±0.110.2± 0.1 9.92±0.039.92± 0.03 HL-Gaussian 10.2±0.110.2± 0.1 9.85±0.039.85± 0.03 Quantile Loss 10.0±0.110.0± 0.1 9.81±0.069.81± 0.06 telemonitoring_parkinsons_biomedical_voice_measurements Objective Single E5 MSE 11±211± 2 11.0±0.711.0± 0.7 HL-Gaussian 10.9±0.710.9± 0.7 10.4±0.410.4± 0.4 Quantile Loss 10.6±0.610.6± 0.6 10.2±0.310.2± 0.3 video_game_fps_prediction Objective Single E5 MSE 0.0145±0.00090.0145± 0.0009 0.0136±0.00040.0136± 0.0004 HL-Gaussian 0.0154±0.00100.0154± 0.0010 0.0151±0.00040.0151± 0.0004 Quantile Loss 0.012±0.0010.012± 0.001 0.0101±0.00040.0101± 0.0004 video_transcoding_time_prediction Objective Single E5 MSE 0.273±0.0070.273± 0.007 0.263±0.0050.263± 0.005 HL-Gaussian 0.264±0.0070.264± 0.007 0.258±0.0020.258± 0.002 Quantile Loss 0.264±0.0050.264± 0.005 0.259±0.0020.259± 0.002 wine_quality Objective Single E5 MSE 0.677±0.0020.677± 0.002 0.672±0.0010.672± 0.001 HL-Gaussian 0.670±0.0030.670± 0.003 0.665±0.0020.665± 0.002 Quantile Loss 0.674±0.0030.674± 0.003 0.6655±0.00030.6655± 0.0003 wine_world_cost Objective Single E5 MSE 0.402±0.0080.402± 0.008 0.395±0.0040.395± 0.004 HL-Gaussian 0.409±0.0050.409± 0.005 0.401±0.0010.401± 0.001 Quantile Loss 0.407±0.0070.407± 0.007 0.399±0.0020.399± 0.002 Table 15: Extended Beyond results with the Muon optimizer. Single denotes one model; E5 denotes an ensemble of five. 5g_energy_consumption Model Objective Single E5 MLP MSE 4.07±0.034.07± 0.03 4.02±0.014.02± 0.01 MAE 4.17±0.034.17± 0.03 4.13±0.014.13± 0.01 Huber Loss 4.15±0.054.15± 0.05 4.09±0.024.09± 0.02 Soft Binning 4.13±0.034.13± 0.03 4.09±0.024.09± 0.02 Hard Binning 4.06±0.034.06± 0.03 4.054±0.0074.054± 0.007 Joint End-To-End 4.10±0.044.10± 0.04 4.06±0.024.06± 0.02 Decoder[N] 4.48±0.064.48± 0.06 4.22±0.054.22± 0.05 HL-Gaussian 4.13±0.044.13± 0.04 4.080±0.0044.080± 0.004 Decoder[U] 2753±106472753± 10647 2752±47592752± 4759 Softmax MSE 4.14±0.034.14± 0.03 4.08±0.024.08± 0.02 Ordinal Entropy 4.11±0.054.11± 0.05 4.056±0.0084.056± 0.008 ROT-REG 4.06±0.044.06± 0.04 4.00±0.034.00± 0.03 GAR [MAE] 4.05±0.044.05± 0.04 4.01±0.014.01± 0.01 GAR [MSE] 4.08±0.054.08± 0.05 4.020±0.0074.020± 0.007 MDN 4.04±0.034.04± 0.03 4.008±0.0104.008± 0.010 NLL 4.09±0.044.09± 0.04 4.03±0.014.03± 0.01 beta-NLL 4.08±0.034.08± 0.03 4.039±0.0054.039± 0.005 AdaQR 4.04±0.014.04± 0.01 4.017±0.0024.017± 0.002 Quantile Loss 4.06±0.034.06± 0.03 4.026±0.0064.026± 0.006 TabPack MSE 4.03±0.024.03± 0.02 – HL-Gaussian 4.06±0.034.06± 0.03 – Quantile Loss 4.01±0.024.01± 0.02 – AdaQR 4.01±0.024.01± 0.02 – RealMLP MSE 4.04±0.024.04± 0.02 4.027±0.0084.027± 0.008 HL-Gaussian 4.12±0.034.12± 0.03 4.11±0.024.11± 0.02 Quantile Loss 4.06±0.024.06± 0.02 4.04±0.014.04± 0.01 AdaQR†AdaQR 4.03±0.034.03± 0.03 4.01±0.024.01± 0.02 AdaQR w. global σ 4.03±0.024.03± 0.02 4.01±0.024.01± 0.02 TabM MSE 4.01±0.034.01± 0.03 3.99±0.013.99± 0.01 HL-Gaussian 3.99±0.023.99± 0.02 3.98±0.013.98± 0.01 Quantile Loss 4.01±0.044.01± 0.04 3.98±0.023.98± 0.02 AdaQR 4.00±0.024.00± 0.02 3.986±0.0063.986± 0.006 airfoil_self_noise Model Objective Single E5 MLP MSE 1.85±0.051.85± 0.05 1.79±0.021.79± 0.02 MAE 1.89±0.071.89± 0.07 1.82±0.021.82± 0.02 Huber Loss 1.91±0.051.91± 0.05 1.85±0.011.85± 0.01 Soft Binning 1.89±0.061.89± 0.06 1.804±0.0081.804± 0.008 Hard Binning 2.19±0.022.19± 0.02 2.172±0.0082.172± 0.008 Joint End-To-End 1.93±0.041.93± 0.04 1.86±0.011.86± 0.01 Decoder[N] 2.11±0.082.11± 0.08 1.944±0.0051.944± 0.005 HL-Gaussian 1.90±0.051.90± 0.05 1.84±0.031.84± 0.03 Decoder[U] 4±24± 2 3±13± 1 Softmax MSE 1.86±0.061.86± 0.06 1.81±0.041.81± 0.04 Ordinal Entropy – – ROT-REG 1.82±0.051.82± 0.05 1.77±0.021.77± 0.02 GAR [MAE] – – GAR [MSE] – – MDN 1.89±0.071.89± 0.07 1.825±0.0071.825± 0.007 NLL 1.91±0.061.91± 0.06 1.81±0.011.81± 0.01 beta-NLL 1.88±0.061.88± 0.06 1.84±0.031.84± 0.03 AdaQR 1.86±0.051.86± 0.05 1.83±0.031.83± 0.03 Quantile Loss 1.86±0.051.86± 0.05 1.815±0.0071.815± 0.007 TabPack MSE 1.70±0.031.70± 0.03 – HL-Gaussian 1.77±0.021.77± 0.02 – Quantile Loss 1.71±0.031.71± 0.03 – AdaQR 1.69±0.021.69± 0.02 – RealMLP MSE 1.55±0.021.55± 0.02 1.53±0.011.53± 0.01 HL-Gaussian 1.50±0.011.50± 0.01 1.491±0.0071.491± 0.007 Quantile Loss 1.43±0.021.43± 0.02 1.43±0.011.43± 0.01 AdaQR†AdaQR 1.47±0.021.47± 0.02 1.460±0.0071.460± 0.007 AdaQR w. global σ 1.53±0.011.53± 0.01 1.529±0.0091.529± 0.009 TabM MSE 1.64±0.031.64± 0.03 1.61±0.011.61± 0.01 HL-Gaussian 1.63±0.021.63± 0.02 1.60±0.011.60± 0.01 Quantile Loss 1.68±0.031.68± 0.03 1.64±0.011.64± 0.01 AdaQR 1.62±0.021.62± 0.02 1.60±0.011.60± 0.01 Table 16: Extended Beyond results with the Muon optimizer (continued). allstate_claims_severity Model Objective Single E5 MLP MSE 0.5400±0.00020.5400± 0.0002 0.5388±0.00010.5388± 0.0001 MAE 0.5408±0.00010.5408± 0.0001 0.5402±0.00000.5402± 0.0000 Huber Loss 0.5393±0.00020.5393± 0.0002 0.5388±0.00000.5388± 0.0000 Soft Binning 0.5378±0.00010.5378± 0.0001 0.5373±0.00000.5373± 0.0000 Hard Binning 0.5393±0.00010.5393± 0.0001 0.5389±0.00000.5389± 0.0000 Joint End-To-End 0.5394±0.00020.5394± 0.0002 0.5387±0.00000.5387± 0.0000 Decoder[N] 0.550±0.0020.550± 0.002 0.5426±0.00100.5426± 0.0010 HL-Gaussian 0.5386±0.00010.5386± 0.0001 0.5379±0.00010.5379± 0.0001 Decoder[U] 0.549±0.0010.549± 0.001 0.5428±0.00070.5428± 0.0007 Softmax MSE 0.5400±0.00010.5400± 0.0001 0.5392±0.00010.5392± 0.0001 Ordinal Entropy 0.5389±0.00010.5389± 0.0001 0.5384±0.00010.5384± 0.0001 ROT-REG 0.5392±0.00010.5392± 0.0001 0.5386±0.00000.5386± 0.0000 GAR [MAE] 0.5395±0.00010.5395± 0.0001 0.5389±0.00000.5389± 0.0000 GAR [MSE] 0.5397±0.00010.5397± 0.0001 0.5391±0.00000.5391± 0.0000 MDN 0.5389±0.00020.5389± 0.0002 0.5381±0.00010.5381± 0.0001 NLL 0.5400±0.00020.5400± 0.0002 0.5391±0.00000.5391± 0.0000 beta-NLL 0.5395±0.00020.5395± 0.0002 0.5383±0.00010.5383± 0.0001 AdaQR 0.5392±0.00010.5392± 0.0001 0.5386±0.00010.5386± 0.0001 Quantile Loss 0.5392±0.00020.5392± 0.0002 0.5384±0.00000.5384± 0.0000 TabPack MSE 0.5364±0.00020.5364± 0.0002 – HL-Gaussian 0.5357±0.00020.5357± 0.0002 – Quantile Loss 0.5357±0.00020.5357± 0.0002 – AdaQR 0.5358±0.00020.5358± 0.0002 – RealMLP MSE 0.5343±0.00030.5343± 0.0003 0.5333±0.00000.5333± 0.0000 HL-Gaussian 0.5329±0.00020.5329± 0.0002 0.5320±0.00010.5320± 0.0001 Quantile Loss 0.5331±0.00020.5331± 0.0002 0.5318±0.00010.5318± 0.0001 AdaQR†AdaQR 0.5335±0.00010.5335± 0.0001 0.5327±0.00010.5327± 0.0001 AdaQR w. global σ 0.5339±0.00010.5339± 0.0001 0.5332±0.00000.5332± 0.0000 TabM MSE 0.5344±0.00020.5344± 0.0002 0.5335±0.00010.5335± 0.0001 HL-Gaussian 0.5339±0.00020.5339± 0.0002 0.5330±0.00010.5330± 0.0001 Quantile Loss 0.5346±0.00030.5346± 0.0003 0.5336±0.00000.5336± 0.0000 AdaQR 0.5341±0.00030.5341± 0.0003 0.5325±0.00000.5325± 0.0000 blood_tests_drink_prediction Model Objective Single E5 MLP MSE 0.667±0.0090.667± 0.009 0.666±0.0060.666± 0.006 MAE 0.68±0.010.68± 0.01 0.682±0.0040.682± 0.004 Huber Loss 0.68±0.020.68± 0.02 0.677±0.0050.677± 0.005 Soft Binning 0.667±0.0050.667± 0.005 0.666±0.0030.666± 0.003 Hard Binning 0.70±0.020.70± 0.02 0.693±0.0030.693± 0.003 Joint End-To-End 0.671±0.0060.671± 0.006 0.670±0.0020.670± 0.002 Decoder[N] 0.86±0.040.86± 0.04 0.76±0.010.76± 0.01 HL-Gaussian 0.652±0.0080.652± 0.008 0.6506±0.00080.6506± 0.0008 Decoder[U] 0.74±0.030.74± 0.03 0.699±0.0090.699± 0.009 Softmax MSE 0.669±0.0060.669± 0.006 0.668±0.0040.668± 0.004 Ordinal Entropy 0.69±0.020.69± 0.02 0.678±0.0030.678± 0.003 ROT-REG 0.662±0.0100.662± 0.010 0.660±0.0050.660± 0.005 GAR [MAE] 0.68±0.020.68± 0.02 0.674±0.0050.674± 0.005 GAR [MSE] 0.674±0.0080.674± 0.008 0.672±0.0050.672± 0.005 MDN 0.66±0.010.66± 0.01 0.655±0.0070.655± 0.007 NLL 0.674±0.0080.674± 0.008 0.671±0.0050.671± 0.005 beta-NLL 0.674±0.0080.674± 0.008 0.668±0.0080.668± 0.008 AdaQR 0.687±0.0080.687± 0.008 0.685±0.0020.685± 0.002 Quantile Loss 0.678±0.0060.678± 0.006 0.678±0.0040.678± 0.004 TabPack MSE 0.66±0.010.66± 0.01 – HL-Gaussian 0.668±0.0090.668± 0.009 – Quantile Loss 0.654±0.0050.654± 0.005 – AdaQR 0.66±0.010.66± 0.01 – RealMLP MSE 0.74±0.030.74± 0.03 0.693±0.0050.693± 0.005 HL-Gaussian 0.666±0.0050.666± 0.005 0.665±0.0020.665± 0.002 Quantile Loss 0.670±0.0080.670± 0.008 0.668±0.0020.668± 0.002 AdaQR†AdaQR 0.68±0.010.68± 0.01 0.668±0.0060.668± 0.006 AdaQR w. global σ 0.666±0.0040.666± 0.004 0.666±0.0020.666± 0.002 TabM MSE 0.671±0.0080.671± 0.008 0.670±0.0010.670± 0.001 HL-Gaussian 0.67±0.010.67± 0.01 0.666±0.0040.666± 0.004 Quantile Loss 0.675±0.0070.675± 0.007 0.672±0.0030.672± 0.003 AdaQR 0.67±0.010.67± 0.01 0.672±0.0060.672± 0.006 Table 17: Extended Beyond results with the Muon optimizer (continued). body_density_prediction Model Objective Single E5 MLP MSE 0.013±0.0020.013± 0.002 0.0127±0.00020.0127± 0.0002 MAE 0.0121±0.00020.0121± 0.0002 0.0119±0.00010.0119± 0.0001 Huber Loss 0.0125±0.00040.0125± 0.0004 0.0123±0.00000.0123± 0.0000 Soft Binning 0.0137±0.00020.0137± 0.0002 0.0137±0.00010.0137± 0.0001 Hard Binning 0.0153±0.00080.0153± 0.0008 0.0143±0.00040.0143± 0.0004 Joint End-To-End 0.0136±0.00020.0136± 0.0002 0.0133±0.00010.0133± 0.0001 Decoder[N] 0.014±0.0010.014± 0.001 0.0127±0.00040.0127± 0.0004 HL-Gaussian 0.0127±0.00020.0127± 0.0002 0.0125±0.00020.0125± 0.0002 Decoder[U] 0.016±0.0020.016± 0.002 0.0137±0.00040.0137± 0.0004 Softmax MSE 0.0136±0.00040.0136± 0.0004 0.0131±0.00020.0131± 0.0002 Ordinal Entropy 0.0130±0.00060.0130± 0.0006 0.0123±0.00020.0123± 0.0002 ROT-REG 0.0131±0.00070.0131± 0.0007 0.0127±0.00050.0127± 0.0005 GAR [MAE] 0.0127±0.00050.0127± 0.0005 0.0121±0.00020.0121± 0.0002 GAR [MSE] 0.0127±0.00050.0127± 0.0005 0.0124±0.00010.0124± 0.0001 MDN 0.0127±0.00030.0127± 0.0003 0.0126±0.00010.0126± 0.0001 NLL 0.0130±0.00060.0130± 0.0006 0.0126±0.00030.0126± 0.0003 beta-NLL 0.0131±0.00040.0131± 0.0004 0.0125±0.00010.0125± 0.0001 AdaQR 0.0129±0.00040.0129± 0.0004 0.0126±0.00010.0126± 0.0001 Quantile Loss 0.0125±0.00020.0125± 0.0002 0.0124±0.00020.0124± 0.0002 TabPack MSE 0.0125±0.00050.0125± 0.0005 – HL-Gaussian 0.0129±0.00050.0129± 0.0005 – Quantile Loss 0.0116±0.00030.0116± 0.0003 – AdaQR 0.0123±0.00030.0123± 0.0003 – RealMLP MSE 0.0113±0.00010.0113± 0.0001 0.0113±0.00010.0113± 0.0001 HL-Gaussian 0.0112±0.00010.0112± 0.0001 0.0112±0.00000.0112± 0.0000 Quantile Loss 0.0114±0.00020.0114± 0.0002 0.0114±0.00010.0114± 0.0001 AdaQR†AdaQR 0.0115±0.00010.0115± 0.0001 0.0114±0.00010.0114± 0.0001 AdaQR w. global σ 0.0115±0.00010.0115± 0.0001 0.0115±0.00010.0115± 0.0001 TabM MSE 0.0126±0.00020.0126± 0.0002 0.0126±0.00010.0126± 0.0001 HL-Gaussian 0.0129±0.00010.0129± 0.0001 0.0128±0.00010.0128± 0.0001 Quantile Loss 0.0120±0.00040.0120± 0.0004 0.0119±0.00010.0119± 0.0001 AdaQR 0.0124±0.00020.0124± 0.0002 0.0123±0.00000.0123± 0.0000 california_house_prices_2020 Model Objective Single E5 MLP MSE 0.274±0.0020.274± 0.002 0.2710±0.00080.2710± 0.0008 MAE 0.260±0.0020.260± 0.002 0.2573±0.00080.2573± 0.0008 Huber Loss 0.267±0.0020.267± 0.002 0.258±0.0010.258± 0.001 Soft Binning 0.257±0.0020.257± 0.002 0.2531±0.00060.2531± 0.0006 Hard Binning 0.306±0.0030.306± 0.003 0.294±0.0020.294± 0.002 Joint End-To-End 0.275±0.0020.275± 0.002 0.2718±0.00090.2718± 0.0009 Decoder[N] 0.312±0.0040.312± 0.004 0.280±0.0020.280± 0.002 HL-Gaussian 0.2622±0.00090.2622± 0.0009 0.2597±0.00020.2597± 0.0002 Decoder[U] 0.306±0.0050.306± 0.005 0.2895±0.00090.2895± 0.0009 Softmax MSE 0.274±0.0040.274± 0.004 0.268±0.0020.268± 0.002 Ordinal Entropy 0.273±0.0050.273± 0.005 0.266±0.0020.266± 0.002 ROT-REG 0.255±0.0030.255± 0.003 0.253±0.0020.253± 0.002 GAR [MAE] 0.261±0.0030.261± 0.003 0.254±0.0020.254± 0.002 GAR [MSE] 0.275±0.0020.275± 0.002 0.2658±0.00090.2658± 0.0009 MDN 0.263±0.0040.263± 0.004 0.2573±0.00070.2573± 0.0007 NLL 0.262±0.0030.262± 0.003 0.260±0.0010.260± 0.001 beta-NLL 0.268±0.0030.268± 0.003 0.265±0.0020.265± 0.002 AdaQR 0.245±0.0020.245± 0.002 0.2406±0.00090.2406± 0.0009 Quantile Loss 0.268±0.0040.268± 0.004 0.262±0.0010.262± 0.001 TabPack MSE 0.2009±0.00060.2009± 0.0006 – HL-Gaussian 0.202±0.0010.202± 0.001 – Quantile Loss 0.202±0.0020.202± 0.002 – AdaQR 0.2002±0.00060.2002± 0.0006 – RealMLP MSE 0.2010±0.00040.2010± 0.0004 0.2001±0.00030.2001± 0.0003 HL-Gaussian 0.2022±0.00090.2022± 0.0009 0.2003±0.00070.2003± 0.0007 Quantile Loss 0.2004±0.00040.2004± 0.0004 0.1989±0.00040.1989± 0.0004 AdaQR†AdaQR 0.2007±0.00080.2007± 0.0008 0.1993±0.00030.1993± 0.0003 AdaQR w. global σ 0.201±0.0010.201± 0.001 0.1996±0.00050.1996± 0.0005 TabM MSE 0.212±0.0020.212± 0.002 0.2078±0.00040.2078± 0.0004 HL-Gaussian 0.206±0.0020.206± 0.002 0.2040±0.00060.2040± 0.0006 Quantile Loss 0.202±0.0010.202± 0.001 0.1997±0.00070.1997± 0.0007 AdaQR 0.206±0.0020.206± 0.002 0.2027±0.00030.2027± 0.0003 Table 18: Extended Beyond results with the Muon optimizer (continued). cirrhosis_patient_survival_prediction Model Objective Single E5 MLP MSE 0.72±0.020.72± 0.02 0.705±0.0050.705± 0.005 MAE 0.66±0.020.66± 0.02 0.655±0.0080.655± 0.008 Huber Loss 0.67±0.010.67± 0.01 0.668±0.0050.668± 0.005 Soft Binning 0.69±0.010.69± 0.01 0.693±0.0060.693± 0.006 Hard Binning 0.93±0.070.93± 0.07 0.83±0.020.83± 0.02 Joint End-To-End 0.74±0.010.74± 0.01 0.734±0.0020.734± 0.002 Decoder[N] 1.0±0.21.0± 0.2 0.80±0.080.80± 0.08 HL-Gaussian 0.78±0.030.78± 0.03 0.779±0.0080.779± 0.008 Decoder[U] 0.83±0.060.83± 0.06 0.76±0.020.76± 0.02 Softmax MSE 0.73±0.010.73± 0.01 0.725±0.0020.725± 0.002 Ordinal Entropy 0.72±0.020.72± 0.02 0.712±0.0050.712± 0.005 ROT-REG 0.76±0.040.76± 0.04 0.73±0.010.73± 0.01 GAR [MAE] 0.68±0.010.68± 0.01 0.678±0.0030.678± 0.003 GAR [MSE] 0.67±0.020.67± 0.02 0.671±0.0070.671± 0.007 MDN 0.68±0.020.68± 0.02 0.680±0.0060.680± 0.006 NLL 0.70±0.020.70± 0.02 0.69±0.010.69± 0.01 beta-NLL 0.64±0.010.64± 0.01 0.636±0.0020.636± 0.002 AdaQR 0.680±0.0100.680± 0.010 0.679±0.0020.679± 0.002 Quantile Loss 0.68±0.010.68± 0.01 0.681±0.0050.681± 0.005 TabPack MSE 0.77±0.030.77± 0.03 – HL-Gaussian 0.91±0.080.91± 0.08 – Quantile Loss 0.75±0.020.75± 0.02 – AdaQR 0.72±0.030.72± 0.03 – RealMLP MSE 0.70±0.060.70± 0.06 0.68±0.040.68± 0.04 HL-Gaussian 0.78±0.050.78± 0.05 0.74±0.030.74± 0.03 Quantile Loss 0.658±0.0090.658± 0.009 0.656±0.0010.656± 0.001 AdaQR†AdaQR 0.70±0.050.70± 0.05 0.69±0.010.69± 0.01 AdaQR w. global σ 0.634±0.0090.634± 0.009 0.633±0.0030.633± 0.003 TabM MSE 0.649±0.0070.649± 0.007 0.646±0.0010.646± 0.001 HL-Gaussian 0.67±0.010.67± 0.01 0.671±0.0060.671± 0.006 Quantile Loss 0.68±0.010.68± 0.01 0.676±0.0050.676± 0.005 AdaQR 0.68±0.010.68± 0.01 0.681±0.0010.681± 0.001 climate_model_weather_forecasting_1m Model Objective Single E5 MLP MSE 1.474±0.0091.474± 0.009 1.468±0.0061.468± 0.006 MAE 1.454±0.0081.454± 0.008 1.440±0.0061.440± 0.006 Huber Loss 1.47±0.011.47± 0.01 1.458±0.0041.458± 0.004 Soft Binning 1.470±0.0081.470± 0.008 1.460±0.0011.460± 0.001 Hard Binning 1.448±0.0071.448± 0.007 1.434±0.0031.434± 0.003 Joint End-To-End 1.46±0.011.46± 0.01 1.443±0.0071.443± 0.007 Decoder[N] 1.51±0.011.51± 0.01 1.455±0.0081.455± 0.008 HL-Gaussian 1.467±0.0081.467± 0.008 1.455±0.0041.455± 0.004 Decoder[U] 1.49±0.011.49± 0.01 1.444±0.0051.444± 0.005 Softmax MSE 1.49±0.011.49± 0.01 1.473±0.0051.473± 0.005 Ordinal Entropy 1.50±0.011.50± 0.01 1.481±0.0021.481± 0.002 ROT-REG 1.470±0.0081.470± 0.008 1.463±0.0051.463± 0.005 GAR [MAE] 1.451±0.0081.451± 0.008 1.442±0.0031.442± 0.003 GAR [MSE] 1.476±0.0091.476± 0.009 1.468±0.0021.468± 0.002 MDN 1.45±0.011.45± 0.01 1.437±0.0041.437± 0.004 NLL 1.45±0.011.45± 0.01 1.436±0.0051.436± 0.005 beta-NLL 1.436±0.0081.436± 0.008 1.4259±0.00081.4259± 0.0008 AdaQR 1.46±0.011.46± 0.01 1.444±0.0081.444± 0.008 Quantile Loss 1.45±0.011.45± 0.01 1.444±0.0051.444± 0.005 TabPack MSE 1.376±0.0041.376± 0.004 – HL-Gaussian 1.373±0.0041.373± 0.004 – Quantile Loss 1.365±0.0031.365± 0.003 – AdaQR 1.368±0.0031.368± 0.003 – RealMLP MSE 1.402±0.0051.402± 0.005 1.389±0.0031.389± 0.003 HL-Gaussian 1.372±0.0031.372± 0.003 1.365±0.0021.365± 0.002 Quantile Loss 1.379±0.0051.379± 0.005 1.371±0.0021.371± 0.002 AdaQR†AdaQR 1.380±0.0061.380± 0.006 1.372±0.0021.372± 0.002 AdaQR w. global σ 1.370±0.0031.370± 0.003 1.364±0.0011.364± 0.001 TabM MSE 1.394±0.0091.394± 0.009 1.388±0.0021.388± 0.002 HL-Gaussian 1.382±0.0051.382± 0.005 1.375±0.0051.375± 0.005 Quantile Loss 1.368±0.0091.368± 0.009 1.356±0.0031.356± 0.003 AdaQR 1.359±0.0041.359± 0.004 1.354±0.0031.354± 0.003 Table 19: Extended Beyond results with the Muon optimizer (continued). coffee_rating_prediction Model Objective Single E5 MLP MSE 1.48±0.021.48± 0.02 1.4771±0.00031.4771± 0.0003 MAE 1.48±0.051.48± 0.05 1.46±0.021.46± 0.02 Huber Loss 1.54±0.071.54± 0.07 1.51±0.021.51± 0.02 Soft Binning 1.43±0.041.43± 0.04 1.406±0.0081.406± 0.008 Hard Binning 1.53±0.041.53± 0.04 1.53±0.021.53± 0.02 Joint End-To-End 1.51±0.021.51± 0.02 1.503±0.0081.503± 0.008 Decoder[N] 1.41±0.051.41± 0.05 1.36±0.021.36± 0.02 HL-Gaussian 1.40±0.021.40± 0.02 1.390±0.0051.390± 0.005 Decoder[U] 1.5±0.11.5± 0.1 1.44±0.031.44± 0.03 Softmax MSE 1.44±0.021.44± 0.02 1.424±0.0051.424± 0.005 Ordinal Entropy 1.50±0.031.50± 0.03 1.495±0.0071.495± 0.007 ROT-REG 1.48±0.021.48± 0.02 1.465±0.0061.465± 0.006 GAR [MAE] 1.54±0.061.54± 0.06 1.52±0.021.52± 0.02 GAR [MSE] 1.53±0.051.53± 0.05 1.50±0.031.50± 0.03 MDN 1.56±0.021.56± 0.02 1.553±0.0031.553± 0.003 NLL 1.49±0.051.49± 0.05 1.47±0.021.47± 0.02 beta-NLL 1.54±0.031.54± 0.03 1.5312±0.00101.5312± 0.0010 AdaQR 1.50±0.071.50± 0.07 1.48±0.031.48± 0.03 Quantile Loss 1.57±0.051.57± 0.05 1.56±0.011.56± 0.01 TabPack MSE 1.34±0.021.34± 0.02 – HL-Gaussian 1.36±0.021.36± 0.02 – Quantile Loss 1.33±0.021.33± 0.02 – AdaQR 1.34±0.021.34± 0.02 – RealMLP MSE 1.31±0.041.31± 0.04 1.291±0.0101.291± 0.010 HL-Gaussian 1.28±0.021.28± 0.02 1.276±0.0091.276± 0.009 Quantile Loss 1.27±0.011.27± 0.01 1.269±0.0071.269± 0.007 AdaQR†AdaQR 1.259±0.0071.259± 0.007 1.252±0.0031.252± 0.003 AdaQR w. global σ 1.32±0.021.32± 0.02 1.316±0.0051.316± 0.005 TabM MSE 1.33±0.021.33± 0.02 1.311±0.0061.311± 0.006 HL-Gaussian 1.34±0.031.34± 0.03 1.33±0.011.33± 0.01 Quantile Loss 1.29±0.041.29± 0.04 1.28±0.021.28± 0.02 AdaQR 1.30±0.021.30± 0.02 1.289±0.0091.289± 0.009 concrete_compressive_strength Model Objective Single E5 MLP MSE 4.8±0.14.8± 0.1 4.59±0.034.59± 0.03 MAE 4.7±0.14.7± 0.1 4.52±0.064.52± 0.06 Huber Loss 4.59±0.084.59± 0.08 4.54±0.024.54± 0.02 Soft Binning 4.65±0.084.65± 0.08 4.58±0.024.58± 0.02 Hard Binning 5.64±0.065.64± 0.06 5.61±0.035.61± 0.03 Joint End-To-End 4.63±0.044.63± 0.04 4.59±0.014.59± 0.01 Decoder[N] 5.1±0.25.1± 0.2 4.73±0.084.73± 0.08 HL-Gaussian 4.88±0.094.88± 0.09 4.791±0.0054.791± 0.005 Decoder[U] 6.2±0.36.2± 0.3 5.21±0.085.21± 0.08 Softmax MSE 4.67±0.034.67± 0.03 4.63±0.014.63± 0.01 Ordinal Entropy 4.7±0.14.7± 0.1 4.62±0.094.62± 0.09 ROT-REG 4.73±0.064.73± 0.06 4.62±0.014.62± 0.01 GAR [MAE] 4.69±0.064.69± 0.06 4.63±0.044.63± 0.04 GAR [MSE] 4.58±0.044.58± 0.04 4.544±0.0054.544± 0.005 MDN 4.7±0.14.7± 0.1 4.66±0.024.66± 0.02 NLL 4.8±0.14.8± 0.1 4.57±0.054.57± 0.05 beta-NLL 4.63±0.074.63± 0.07 4.57±0.024.57± 0.02 AdaQR 4.9±0.14.9± 0.1 4.72±0.014.72± 0.01 Quantile Loss 4.57±0.044.57± 0.04 4.54±0.024.54± 0.02 TabPack MSE 4.14±0.094.14± 0.09 – HL-Gaussian 4.12±0.104.12± 0.10 – Quantile Loss 3.99±0.083.99± 0.08 – AdaQR 3.95±0.093.95± 0.09 – RealMLP MSE 4.00±0.064.00± 0.06 3.94±0.023.94± 0.02 HL-Gaussian 4.06±0.054.06± 0.05 4.020±0.0094.020± 0.009 Quantile Loss 3.92±0.063.92± 0.06 3.858±0.0083.858± 0.008 AdaQR†AdaQR 4.16±0.054.16± 0.05 4.111±0.0064.111± 0.006 AdaQR w. global σ 4.08±0.074.08± 0.07 4.019±0.0044.019± 0.004 TabM MSE 4.08±0.074.08± 0.07 3.964±0.0093.964± 0.009 HL-Gaussian 3.80±0.053.80± 0.05 3.76±0.013.76± 0.01 Quantile Loss 3.84±0.053.84± 0.05 3.80±0.033.80± 0.03 AdaQR 3.94±0.083.94± 0.08 3.86±0.023.86± 0.02 Table 20: Extended Beyond results with the Muon optimizer (continued). diamonds Model Objective Single E5 MLP MSE 0.0892±0.00080.0892± 0.0008 0.0856±0.00010.0856± 0.0001 MAE 0.0869±0.00080.0869± 0.0008 0.0857±0.00050.0857± 0.0005 Huber Loss 0.0879±0.00050.0879± 0.0005 0.0852±0.00010.0852± 0.0001 Soft Binning 0.0879±0.00060.0879± 0.0006 0.0876±0.00020.0876± 0.0002 Hard Binning 0.0879±0.00060.0879± 0.0006 0.0873±0.00040.0873± 0.0004 Joint End-To-End 0.0875±0.00040.0875± 0.0004 0.0870±0.00020.0870± 0.0002 Decoder[N] 0.094±0.0020.094± 0.002 0.0868±0.00010.0868± 0.0001 HL-Gaussian 0.0873±0.00040.0873± 0.0004 0.0867±0.00020.0867± 0.0002 Decoder[U] 0.097±0.0020.097± 0.002 0.0908±0.00010.0908± 0.0001 Softmax MSE 0.0890±0.00050.0890± 0.0005 0.0852±0.00020.0852± 0.0002 Ordinal Entropy 0.0867±0.00020.0867± 0.0002 0.0863±0.00010.0863± 0.0001 ROT-REG 0.0870±0.00030.0870± 0.0003 0.0862±0.00010.0862± 0.0001 GAR [MAE] 0.0883±0.00040.0883± 0.0004 0.0856±0.00030.0856± 0.0003 GAR [MSE] 0.0888±0.00050.0888± 0.0005 0.0869±0.00020.0869± 0.0002 MDN 0.0877±0.00080.0877± 0.0008 0.0844±0.00010.0844± 0.0001 NLL 0.0875±0.00040.0875± 0.0004 0.0858±0.00000.0858± 0.0000 beta-NLL 0.0879±0.00060.0879± 0.0006 0.0859±0.00030.0859± 0.0003 AdaQR 0.0878±0.00060.0878± 0.0006 0.0855±0.00030.0855± 0.0003 Quantile Loss 0.0877±0.00060.0877± 0.0006 0.0855±0.00010.0855± 0.0001 TabPack MSE 0.0826±0.00020.0826± 0.0002 – HL-Gaussian 0.0823±0.00010.0823± 0.0001 – Quantile Loss 0.0820±0.00010.0820± 0.0001 – AdaQR 0.0819±0.00010.0819± 0.0001 – RealMLP MSE 0.0817±0.00020.0817± 0.0002 0.0808±0.00000.0808± 0.0000 HL-Gaussian 0.0820±0.00020.0820± 0.0002 0.0812±0.00000.0812± 0.0000 Quantile Loss 0.0818±0.00030.0818± 0.0003 0.0813±0.00000.0813± 0.0000 AdaQR†AdaQR 0.0817±0.00030.0817± 0.0003 0.0811±0.00010.0811± 0.0001 AdaQR w. global σ 0.0811±0.00020.0811± 0.0002 0.0804±0.00010.0804± 0.0001 TabM MSE 0.0830±0.00030.0830± 0.0003 0.0823±0.00010.0823± 0.0001 HL-Gaussian 0.0829±0.00040.0829± 0.0004 0.0820±0.00010.0820± 0.0001 Quantile Loss 0.0826±0.00050.0826± 0.0005 0.0817±0.00030.0817± 0.0003 AdaQR 0.0821±0.00030.0821± 0.0003 0.0813±0.00010.0813± 0.0001 early_learning_predictors Model Objective Single E5 MLP MSE 13.38±0.0413.38± 0.04 13.32±0.0213.32± 0.02 MAE 13.23±0.0213.23± 0.02 13.212±0.00613.212± 0.006 Huber Loss 13.23±0.0113.23± 0.01 13.207±0.00313.207± 0.003 Soft Binning 13.25±0.0213.25± 0.02 13.21±0.0113.21± 0.01 Hard Binning 13.26±0.0213.26± 0.02 13.237±0.00413.237± 0.004 Joint End-To-End 13.33±0.0513.33± 0.05 13.25±0.0413.25± 0.04 Decoder[N] 15±115± 1 13.7±0.413.7± 0.4 HL-Gaussian 13.192±0.00913.192± 0.009 13.182±0.00513.182± 0.005 Decoder[U] 15±115± 1 13.9±0.513.9± 0.5 Softmax MSE 13.24±0.0213.24± 0.02 13.21±0.0113.21± 0.01 Ordinal Entropy 13.27±0.0313.27± 0.03 13.227±0.00213.227± 0.002 ROT-REG 13.24±0.0113.24± 0.01 13.225±0.00813.225± 0.008 GAR [MAE] 13.23±0.0213.23± 0.02 13.220±0.00813.220± 0.008 GAR [MSE] 13.22±0.0113.22± 0.01 13.202±0.00513.202± 0.005 MDN 13.26±0.0413.26± 0.04 13.21±0.0113.21± 0.01 NLL 13.37±0.0313.37± 0.03 13.31±0.0113.31± 0.01 beta-NLL 13.24±0.0213.24± 0.02 13.215±0.00613.215± 0.006 AdaQR 13.31±0.0313.31± 0.03 13.26±0.0213.26± 0.02 Quantile Loss 13.29±0.0413.29± 0.04 13.257±0.00913.257± 0.009 TabPack MSE 13.05±0.0213.05± 0.02 – HL-Gaussian 13.04±0.0213.04± 0.02 – Quantile Loss 13.04±0.0213.04± 0.02 – AdaQR 13.05±0.0313.05± 0.03 – RealMLP MSE 13.02±0.0313.02± 0.03 12.951±0.00912.951± 0.009 HL-Gaussian 13.11±0.0313.11± 0.03 12.993±0.00412.993± 0.004 Quantile Loss 13.01±0.0213.01± 0.02 12.953±0.00712.953± 0.007 AdaQR†AdaQR 13.09±0.0413.09± 0.04 13.025±0.00913.025± 0.009 AdaQR w. global σ 13.05±0.0213.05± 0.02 13.002±0.00813.002± 0.008 TabM MSE 13.06±0.0213.06± 0.02 13.033±0.00813.033± 0.008 HL-Gaussian 13.12±0.0313.12± 0.03 13.08±0.0313.08± 0.03 Quantile Loss 13.09±0.0313.09± 0.03 13.052±0.00913.052± 0.009 AdaQR 13.07±0.0313.07± 0.03 13.046±0.00413.046± 0.004 Table 21: Extended Beyond results with the Muon optimizer (continued). electric_motor_temperature_prediction Model Objective Single E5 MLP MSE 2.62±0.032.62± 0.03 2.61±0.022.61± 0.02 MAE 2.96±0.082.96± 0.08 2.92±0.032.92± 0.03 Huber Loss 2.65±0.062.65± 0.06 2.63±0.042.63± 0.04 Soft Binning 2.66±0.052.66± 0.05 2.64±0.022.64± 0.02 Hard Binning 3.7±0.23.7± 0.2 3.53±0.093.53± 0.09 Joint End-To-End 2.6±0.12.6± 0.1 2.55±0.052.55± 0.05 Decoder[N] 3.7±0.23.7± 0.2 3.31±0.093.31± 0.09 HL-Gaussian 2.66±0.032.66± 0.03 2.652±0.0072.652± 0.007 Decoder[U] 270±1005270± 1005 266±445266± 445 Softmax MSE 2.7±0.22.7± 0.2 2.5±0.12.5± 0.1 Ordinal Entropy 2.68±0.052.68± 0.05 2.65±0.022.65± 0.02 ROT-REG 2.9±0.22.9± 0.2 2.81±0.092.81± 0.09 GAR [MAE] 2.61±0.072.61± 0.07 2.56±0.022.56± 0.02 GAR [MSE] 2.6±0.12.6± 0.1 2.49±0.052.49± 0.05 MDN 2.88±0.102.88± 0.10 2.83±0.092.83± 0.09 NLL 2.62±0.032.62± 0.03 2.60±0.022.60± 0.02 beta-NLL 2.64±0.052.64± 0.05 2.618±0.0052.618± 0.005 AdaQR 2.7±0.22.7± 0.2 2.60±0.032.60± 0.03 Quantile Loss 2.6±0.12.6± 0.1 2.49±0.062.49± 0.06 TabPack MSE 2.5±0.12.5± 0.1 – HL-Gaussian 2.40±0.062.40± 0.06 – Quantile Loss 2.5±0.12.5± 0.1 – AdaQR 2.37±0.102.37± 0.10 – RealMLP MSE 2.36±0.042.36± 0.04 2.35±0.022.35± 0.02 HL-Gaussian 2.42±0.022.42± 0.02 2.41±0.022.41± 0.02 Quantile Loss 2.23±0.062.23± 0.06 2.20±0.022.20± 0.02 AdaQR†AdaQR 2.29±0.042.29± 0.04 2.258±0.0072.258± 0.007 AdaQR w. global σ 2.48±0.072.48± 0.07 2.45±0.032.45± 0.03 TabM MSE 2.4±0.12.4± 0.1 2.33±0.022.33± 0.02 HL-Gaussian 2.28±0.052.28± 0.05 2.27±0.022.27± 0.02 Quantile Loss 2.50±0.052.50± 0.05 2.49±0.012.49± 0.01 AdaQR 2.27±0.072.27± 0.07 2.24±0.032.24± 0.03 fiat_500 Model Objective Single E5 MLP MSE 861±9861± 9 859±2859± 2 MAE 854±7854± 7 852±2852± 2 Huber Loss 830±5830± 5 828.0±0.4828.0± 0.4 Soft Binning 807±5807± 5 803.8±0.9803.8± 0.9 Hard Binning 806±3806± 3 805±1805± 1 Joint End-To-End 828±4828± 4 827±3827± 3 Decoder[N] 1237±6321237± 632 1019±1781019± 178 HL-Gaussian 811±1811± 1 811.0±0.9811.0± 0.9 Decoder[U] 890±18890± 18 822±6822± 6 Softmax MSE 839±7839± 7 836±3836± 3 Ordinal Entropy 838±9838± 9 835±2835± 2 ROT-REG 834±8834± 8 831.7±0.7831.7± 0.7 GAR [MAE] 1056±4771056± 477 992±292992± 292 GAR [MSE] 829±7829± 7 826±5826± 5 MDN 828±5828± 5 823±2823± 2 NLL 844±13844± 13 841±4841± 4 beta-NLL 832±7832± 7 829±6829± 6 AdaQR 831±5831± 5 830±2830± 2 Quantile Loss 826±7826± 7 821±3821± 3 TabPack MSE 794±10794± 10 – HL-Gaussian 776±5776± 5 – Quantile Loss 807±19807± 19 – AdaQR 801±10801± 10 – RealMLP MSE 789±9789± 9 782±2782± 2 HL-Gaussian 791±6791± 6 783±2783± 2 Quantile Loss 787±5787± 5 783±2783± 2 AdaQR†AdaQR 781±6781± 6 774±2774± 2 AdaQR w. global σ 778±6778± 6 770±2770± 2 TabM MSE 809±7809± 7 800±1800± 1 HL-Gaussian 805±4805± 4 803±2803± 2 Quantile Loss 813±12813± 12 809±4809± 4 AdaQR 819±8819± 8 810±3810± 3 Table 22: Extended Beyond results with the Muon optimizer (continued). food_delivery_time Model Objective Single E5 MLP MSE 7.73±0.017.73± 0.01 7.715±0.0087.715± 0.008 MAE 7.81±0.037.81± 0.03 7.750±0.0087.750± 0.008 Huber Loss 7.75±0.017.75± 0.01 7.714±0.0077.714± 0.007 Soft Binning 7.748±0.0087.748± 0.008 7.742±0.0027.742± 0.002 Hard Binning 7.65±0.017.65± 0.01 7.636±0.0027.636± 0.002 Joint End-To-End 7.73±0.017.73± 0.01 7.725±0.0057.725± 0.005 Decoder[N] 8.3±0.28.3± 0.2 7.93±0.047.93± 0.04 HL-Gaussian 7.69±0.017.69± 0.01 7.680±0.0047.680± 0.004 Decoder[U] 8.24±0.098.24± 0.09 8.03±0.038.03± 0.03 Softmax MSE 7.78±0.037.78± 0.03 7.738±0.0057.738± 0.005 Ordinal Entropy 7.72±0.027.72± 0.02 7.692±0.0087.692± 0.008 ROT-REG 7.76±0.027.76± 0.02 7.74±0.017.74± 0.01 GAR [MAE] 7.82±0.027.82± 0.02 7.791±0.0087.791± 0.008 GAR [MSE] 7.75±0.027.75± 0.02 7.739±0.0047.739± 0.004 MDN 7.71±0.017.71± 0.01 7.703±0.0027.703± 0.002 NLL 7.77±0.017.77± 0.01 7.744±0.0097.744± 0.009 beta-NLL 7.75±0.017.75± 0.01 7.734±0.0047.734± 0.004 AdaQR 7.74±0.027.74± 0.02 7.707±0.0087.707± 0.008 Quantile Loss 7.70±0.017.70± 0.01 7.671±0.0037.671± 0.003 TabPack MSE 7.514±0.0057.514± 0.005 – HL-Gaussian 7.476±0.0067.476± 0.006 – Quantile Loss 7.497±0.0077.497± 0.007 – AdaQR 7.497±0.0077.497± 0.007 – RealMLP MSE 7.416±0.0067.416± 0.006 7.409±0.0027.409± 0.002 HL-Gaussian 7.395±0.0067.395± 0.006 7.385±0.0037.385± 0.003 Quantile Loss 7.394±0.0047.394± 0.004 7.381±0.0027.381± 0.002 AdaQR†AdaQR 7.399±0.0067.399± 0.006 7.381±0.0037.381± 0.003 AdaQR w. global σ 7.404±0.0027.404± 0.002 7.3971±0.00087.3971± 0.0008 TabM MSE 7.56±0.017.56± 0.01 7.540±0.0067.540± 0.006 HL-Gaussian 7.463±0.0097.463± 0.009 7.444±0.0057.444± 0.005 Quantile Loss 7.512±0.0077.512± 0.007 7.501±0.0027.501± 0.002 AdaQR 7.44±0.017.44± 0.01 7.428±0.0037.428± 0.003 forest_fires Model Objective Single E5 MLP MSE 1.37±0.011.37± 0.01 1.362±0.0021.362± 0.002 MAE 1.356±0.0091.356± 0.009 1.344±0.0081.344± 0.008 Huber Loss 1.349±0.0011.349± 0.001 1.3488±0.00071.3488± 0.0007 Soft Binning 1.362±0.0021.362± 0.002 1.3614±0.00021.3614± 0.0002 Hard Binning 1.355±0.0021.355± 0.002 1.3549±0.00021.3549± 0.0002 Joint End-To-End 1.351±0.0021.351± 0.002 1.3505±0.00041.3505± 0.0004 Decoder[N] 1.7±0.11.7± 0.1 1.50±0.021.50± 0.02 HL-Gaussian 1.353±0.0021.353± 0.002 1.3531±0.00041.3531± 0.0004 Decoder[U] 1.57±0.061.57± 0.06 1.45±0.071.45± 0.07 Softmax MSE 1.3519±0.00081.3519± 0.0008 1.3516±0.00001.3516± 0.0000 Ordinal Entropy 1.37±0.021.37± 0.02 1.359±0.0041.359± 0.004 ROT-REG 1.367±0.0091.367± 0.009 1.364±0.0031.364± 0.003 GAR [MAE] 1.352±0.0041.352± 0.004 1.350±0.0011.350± 0.001 GAR [MSE] 1.378±0.0091.378± 0.009 1.3736±0.00021.3736± 0.0002 MDN 1.349±0.0041.349± 0.004 1.348±0.0011.348± 0.001 NLL 1.37±0.011.37± 0.01 1.358±0.0031.358± 0.003 beta-NLL 1.376±0.0081.376± 0.008 1.374±0.0031.374± 0.003 AdaQR 1.3563±0.00061.3563± 0.0006 1.3563±0.00021.3563± 0.0002 Quantile Loss 1.3586±0.00081.3586± 0.0008 1.3585±0.00031.3585± 0.0003 TabPack MSE 1.362±0.0091.362± 0.009 – HL-Gaussian 1.339±0.0091.339± 0.009 – Quantile Loss 1.355±0.0011.355± 0.001 – AdaQR 1.360±0.0031.360± 0.003 – RealMLP MSE 1.36±0.011.36± 0.01 1.348±0.0021.348± 0.002 HL-Gaussian 1.359±0.0051.359± 0.005 1.357±0.0011.357± 0.001 Quantile Loss 1.34±0.011.34± 0.01 1.34±0.011.34± 0.01 AdaQR†AdaQR 1.338±0.0061.338± 0.006 1.337±0.0051.337± 0.005 AdaQR w. global σ 1.366±0.0081.366± 0.008 1.360±0.0031.360± 0.003 TabM MSE 1.374±0.0091.374± 0.009 1.373±0.0041.373± 0.004 HL-Gaussian 1.362±0.0051.362± 0.005 1.3601±0.00081.3601± 0.0008 Quantile Loss 1.355±0.0031.355± 0.003 1.3545±0.00091.3545± 0.0009 AdaQR 1.354±0.0011.354± 0.001 1.3537±0.00041.3537± 0.0004 Table 23: Extended Beyond results with the Muon optimizer (continued). garments_worker_productivity Model Objective Single E5 MLP MSE 0.099±0.0040.099± 0.004 0.097±0.0010.097± 0.001 MAE 0.101±0.0040.101± 0.004 0.100±0.0020.100± 0.002 Huber Loss 0.099±0.0040.099± 0.004 0.098±0.0020.098± 0.002 Soft Binning 0.102±0.0060.102± 0.006 0.099±0.0010.099± 0.001 Hard Binning 0.105±0.0050.105± 0.005 0.104±0.0040.104± 0.004 Joint End-To-End 0.099±0.0070.099± 0.007 0.096±0.0020.096± 0.002 Decoder[N] 0.098±0.0100.098± 0.010 0.092±0.0050.092± 0.005 HL-Gaussian 0.092±0.0020.092± 0.002 0.0910±0.00050.0910± 0.0005 Decoder[U] 0.11±0.010.11± 0.01 0.096±0.0050.096± 0.005 Softmax MSE 0.102±0.0070.102± 0.007 0.101±0.0040.101± 0.004 Ordinal Entropy 0.096±0.0030.096± 0.003 0.094±0.0020.094± 0.002 ROT-REG 0.101±0.0080.101± 0.008 0.0976±0.00100.0976± 0.0010 GAR [MAE] 0.096±0.0060.096± 0.006 0.094±0.0030.094± 0.003 GAR [MSE] 0.103±0.0070.103± 0.007 0.101±0.0030.101± 0.003 MDN 0.097±0.0050.097± 0.005 0.095±0.0010.095± 0.001 NLL 0.102±0.0060.102± 0.006 0.096±0.0050.096± 0.005 beta-NLL 0.098±0.0030.098± 0.003 0.097±0.0020.097± 0.002 AdaQR 0.100±0.0050.100± 0.005 0.0985±0.00050.0985± 0.0005 Quantile Loss 0.100±0.0050.100± 0.005 0.098±0.0020.098± 0.002 TabPack MSE 0.099±0.0030.099± 0.003 – HL-Gaussian 0.095±0.0030.095± 0.003 – Quantile Loss 0.093±0.0030.093± 0.003 – AdaQR 0.094±0.0030.094± 0.003 – RealMLP MSE 0.095±0.0040.095± 0.004 0.094±0.0030.094± 0.003 HL-Gaussian 0.092±0.0090.092± 0.009 0.089±0.0060.089± 0.006 Quantile Loss 0.087±0.0030.087± 0.003 0.0856±0.00090.0856± 0.0009 AdaQR†AdaQR 0.089±0.0020.089± 0.002 0.0884±0.00070.0884± 0.0007 AdaQR w. global σ 0.092±0.0030.092± 0.003 0.092±0.0010.092± 0.001 TabM MSE 0.099±0.0030.099± 0.003 0.098±0.0010.098± 0.001 HL-Gaussian 0.088±0.0030.088± 0.003 0.088±0.0020.088± 0.002 Quantile Loss 0.097±0.0030.097± 0.003 0.096±0.0020.096± 0.002 AdaQR 0.098±0.0030.098± 0.003 0.097±0.0020.097± 0.002 healthcare_insurance_expenses Model Objective Single E5 MLP MSE 0.401±0.0020.401± 0.002 0.400±0.0010.400± 0.001 MAE 0.398±0.0040.398± 0.004 0.394±0.0010.394± 0.001 Huber Loss 0.396±0.0050.396± 0.005 0.393±0.0020.393± 0.002 Soft Binning 0.404±0.0020.404± 0.002 0.4037±0.00050.4037± 0.0005 Hard Binning 0.400±0.0050.400± 0.005 0.3985±0.00090.3985± 0.0009 Joint End-To-End 0.399±0.0020.399± 0.002 0.398±0.0010.398± 0.001 Decoder[N] 0.400±0.0070.400± 0.007 0.394±0.0020.394± 0.002 HL-Gaussian 0.400±0.0010.400± 0.001 0.3995±0.00010.3995± 0.0001 Decoder[U] 0.43±0.020.43± 0.02 0.415±0.0030.415± 0.003 Softmax MSE 0.396±0.0020.396± 0.002 0.3955±0.00070.3955± 0.0007 Ordinal Entropy 0.403±0.0040.403± 0.004 0.4003±0.00080.4003± 0.0008 ROT-REG 0.407±0.0060.407± 0.006 0.404±0.0040.404± 0.004 GAR [MAE] 0.404±0.0060.404± 0.006 0.4001±0.00090.4001± 0.0009 GAR [MSE] 0.397±0.0020.397± 0.002 0.3955±0.00070.3955± 0.0007 MDN 0.391±0.0040.391± 0.004 0.390±0.0010.390± 0.001 NLL 0.397±0.0030.397± 0.003 0.395±0.0010.395± 0.001 beta-NLL 0.400±0.0030.400± 0.003 0.395±0.0010.395± 0.001 AdaQR 0.391±0.0020.391± 0.002 0.389±0.0010.389± 0.001 Quantile Loss 0.3912±0.00100.3912± 0.0010 0.3902±0.00010.3902± 0.0001 TabPack MSE 0.395±0.0030.395± 0.003 – HL-Gaussian 0.393±0.0050.393± 0.005 – Quantile Loss 0.387±0.0040.387± 0.004 – AdaQR 0.384±0.0030.384± 0.003 – RealMLP MSE 0.404±0.0050.404± 0.005 0.402±0.0020.402± 0.002 HL-Gaussian 0.391±0.0020.391± 0.002 0.3898±0.00060.3898± 0.0006 Quantile Loss 0.391±0.0060.391± 0.006 0.389±0.0040.389± 0.004 AdaQR†AdaQR 0.392±0.0070.392± 0.007 0.389±0.0030.389± 0.003 AdaQR w. global σ 0.397±0.0070.397± 0.007 0.395±0.0020.395± 0.002 TabM MSE 0.397±0.0050.397± 0.005 0.394±0.0010.394± 0.001 HL-Gaussian 0.389±0.0040.389± 0.004 0.388±0.0030.388± 0.003 Quantile Loss 0.379±0.0020.379± 0.002 0.3787±0.00060.3787± 0.0006 AdaQR 0.378±0.0010.378± 0.001 0.3777±0.00030.3777± 0.0003 Table 24: Extended Beyond results with the Muon optimizer (continued). houses Model Objective Single E5 MLP MSE 0.2273±0.00090.2273± 0.0009 0.2226±0.00010.2226± 0.0001 MAE 0.228±0.0010.228± 0.001 0.2217±0.00030.2217± 0.0003 Huber Loss 0.2242±0.00070.2242± 0.0007 0.2225±0.00030.2225± 0.0003 Soft Binning 0.226±0.0010.226± 0.001 0.2233±0.00040.2233± 0.0004 Hard Binning 0.225±0.0010.225± 0.001 0.2234±0.00050.2234± 0.0005 Joint End-To-End 0.2250±0.00100.2250± 0.0010 0.2243±0.00050.2243± 0.0005 Decoder[N] 0.234±0.0020.234± 0.002 0.224±0.0010.224± 0.001 HL-Gaussian 0.2238±0.00080.2238± 0.0008 0.2226±0.00030.2226± 0.0003 Decoder[U] 0.251±0.0040.251± 0.004 0.2315±0.00060.2315± 0.0006 Softmax MSE 0.230±0.0010.230± 0.001 0.2228±0.00050.2228± 0.0005 Ordinal Entropy 0.2266±0.00080.2266± 0.0008 0.2238±0.00040.2238± 0.0004 ROT-REG 0.2250±0.00080.2250± 0.0008 0.2237±0.00010.2237± 0.0001 GAR [MAE] 0.2241±0.00090.2241± 0.0009 0.2233±0.00020.2233± 0.0002 GAR [MSE] 0.2242±0.00050.2242± 0.0005 0.2225±0.00010.2225± 0.0001 MDN 0.2228±0.00080.2228± 0.0008 0.2215±0.00030.2215± 0.0003 NLL 0.2257±0.00100.2257± 0.0010 0.2220±0.00040.2220± 0.0004 beta-NLL 0.223±0.0010.223± 0.001 0.2214±0.00020.2214± 0.0002 AdaQR 0.2231±0.00090.2231± 0.0009 0.2213±0.00030.2213± 0.0003 Quantile Loss 0.2237±0.00090.2237± 0.0009 0.2223±0.00040.2223± 0.0004 TabPack MSE 0.2030±0.00090.2030± 0.0009 – HL-Gaussian 0.204±0.0020.204± 0.002 – Quantile Loss 0.2015±0.00100.2015± 0.0010 – AdaQR 0.2015±0.00090.2015± 0.0009 – RealMLP MSE 0.1949±0.00050.1949± 0.0005 0.1926±0.00030.1926± 0.0003 HL-Gaussian 0.195±0.0010.195± 0.001 0.1929±0.00020.1929± 0.0002 Quantile Loss 0.1917±0.00050.1917± 0.0005 0.1902±0.00030.1902± 0.0003 AdaQR†AdaQR 0.1958±0.00090.1958± 0.0009 0.1934±0.00040.1934± 0.0004 AdaQR w. global σ 0.196±0.0010.196± 0.001 0.1931±0.00080.1931± 0.0008 TabM MSE 0.202±0.0010.202± 0.001 0.1995±0.00070.1995± 0.0007 HL-Gaussian 0.200±0.0020.200± 0.002 0.197±0.0010.197± 0.001 Quantile Loss 0.200±0.0010.200± 0.001 0.198±0.0010.198± 0.001 AdaQR 0.1994±0.00080.1994± 0.0008 0.1986±0.00040.1986± 0.0004 immoscout_german_house_prices Model Objective Single E5 MLP MSE 0.407±0.0020.407± 0.002 0.4018±0.00100.4018± 0.0010 MAE 0.404±0.0020.404± 0.002 0.399±0.0010.399± 0.001 Huber Loss 0.404±0.0030.404± 0.003 0.398±0.0020.398± 0.002 Soft Binning 0.406±0.0020.406± 0.002 0.4013±0.00040.4013± 0.0004 Hard Binning 0.440±0.0030.440± 0.003 0.428±0.0020.428± 0.002 Joint End-To-End 0.407±0.0020.407± 0.002 0.403±0.0010.403± 0.001 Decoder[N] 0.423±0.0080.423± 0.008 0.409±0.0010.409± 0.001 HL-Gaussian 0.409±0.0020.409± 0.002 0.4041±0.00020.4041± 0.0002 Decoder[U] 0.429±0.0050.429± 0.005 0.409±0.0010.409± 0.001 Softmax MSE 0.409±0.0030.409± 0.003 0.402±0.0010.402± 0.001 Ordinal Entropy 0.410±0.0030.410± 0.003 0.4035±0.00100.4035± 0.0010 ROT-REG 0.409±0.0030.409± 0.003 0.402±0.0010.402± 0.001 GAR [MAE] 0.404±0.0020.404± 0.002 0.400±0.0010.400± 0.001 GAR [MSE] 0.406±0.0030.406± 0.003 0.4003±0.00100.4003± 0.0010 MDN 0.401±0.0020.401± 0.002 0.3976±0.00070.3976± 0.0007 NLL 0.405±0.0030.405± 0.003 0.4001±0.00010.4001± 0.0001 beta-NLL 0.404±0.0020.404± 0.002 0.3977±0.00030.3977± 0.0003 AdaQR 0.405±0.0020.405± 0.002 0.3987±0.00030.3987± 0.0003 Quantile Loss 0.404±0.0020.404± 0.002 0.3995±0.00070.3995± 0.0007 TabPack MSE 0.369±0.0010.369± 0.001 – HL-Gaussian 0.371±0.0010.371± 0.001 – Quantile Loss 0.366±0.0010.366± 0.001 – AdaQR 0.3656±0.00090.3656± 0.0009 – RealMLP MSE 0.368±0.0020.368± 0.002 0.3650±0.00060.3650± 0.0006 HL-Gaussian 0.368±0.0010.368± 0.001 0.3629±0.00070.3629± 0.0007 Quantile Loss 0.367±0.0010.367± 0.001 0.3641±0.00030.3641± 0.0003 AdaQR†AdaQR 0.365±0.0010.365± 0.001 0.3620±0.00010.3620± 0.0001 AdaQR w. global σ 0.368±0.0020.368± 0.002 0.365±0.0010.365± 0.001 TabM MSE 0.3726±0.00070.3726± 0.0007 0.3709±0.00030.3709± 0.0003 HL-Gaussian 0.369±0.0020.369± 0.002 0.3669±0.00050.3669± 0.0005 Quantile Loss 0.371±0.0010.371± 0.001 0.3694±0.00060.3694± 0.0006 AdaQR 0.369±0.0010.369± 0.001 0.3684±0.00040.3684± 0.0004 Table 25: Extended Beyond results with the Muon optimizer (continued). mercari_price_suggestion_1m Model Objective Single E5 MLP MSE 0.4746±0.00060.4746± 0.0006 0.4664±0.00080.4664± 0.0008 MAE 0.479±0.0020.479± 0.002 0.472±0.0020.472± 0.002 Huber Loss 0.4741±0.00030.4741± 0.0003 0.4666±0.00030.4666± 0.0003 Soft Binning 0.472±0.0020.472± 0.002 0.463±0.0010.463± 0.001 Hard Binning 0.4742±0.00080.4742± 0.0008 0.4649±0.00090.4649± 0.0009 Joint End-To-End 0.4770±0.00050.4770± 0.0005 0.4694±0.00040.4694± 0.0004 Decoder[N] 0.507±0.0050.507± 0.005 0.486±0.0010.486± 0.001 HL-Gaussian 0.4730±0.00020.4730± 0.0002 0.4625±0.00050.4625± 0.0005 Decoder[U] 0.513±0.0050.513± 0.005 0.477±0.0030.477± 0.003 Softmax MSE 0.4764±0.00070.4764± 0.0007 0.4696±0.00020.4696± 0.0002 Ordinal Entropy 0.474±0.0010.474± 0.001 0.4675±0.00070.4675± 0.0007 ROT-REG 0.4782±0.00020.4782± 0.0002 0.4705±0.00010.4705± 0.0001 GAR [MAE] 0.476±0.0020.476± 0.002 0.4668±0.00060.4668± 0.0006 GAR [MSE] 0.4748±0.00030.4748± 0.0003 0.4687±0.00010.4687± 0.0001 MDN 0.474±0.0020.474± 0.002 0.4661±0.00100.4661± 0.0010 NLL 0.4767±0.00050.4767± 0.0005 0.4679±0.00040.4679± 0.0004 beta-NLL 0.476±0.0010.476± 0.001 0.4685±0.00050.4685± 0.0005 AdaQR 0.4723±0.00070.4723± 0.0007 0.464±0.0010.464± 0.001 Quantile Loss 0.473±0.0030.473± 0.003 0.464±0.0010.464± 0.001 TabPack MSE 0.4630±0.00020.4630± 0.0002 – HL-Gaussian 0.4652±0.00010.4652± 0.0001 – Quantile Loss 0.4621±0.00010.4621± 0.0001 – AdaQR 0.4621±0.00020.4621± 0.0002 – RealMLP MSE 0.4633±0.00020.4633± 0.0002 0.4577±0.00020.4577± 0.0002 HL-Gaussian 0.4572±0.00030.4572± 0.0003 0.4512±0.00040.4512± 0.0004 Quantile Loss 0.4584±0.00030.4584± 0.0003 0.4534±0.00020.4534± 0.0002 AdaQR†AdaQR 0.4617±0.00020.4617± 0.0002 0.4560±0.00020.4560± 0.0002 AdaQR w. global σ 0.4567±0.00020.4567± 0.0002 0.4511±0.00020.4511± 0.0002 TabM MSE 0.4682±0.00040.4682± 0.0004 0.4654±0.00020.4654± 0.0002 HL-Gaussian 0.4647±0.00050.4647± 0.0005 0.4616±0.00020.4616± 0.0002 Quantile Loss 0.4676±0.00030.4676± 0.0003 0.4649±0.00020.4649± 0.0002 AdaQR 0.4599±0.00030.4599± 0.0003 0.4572±0.00020.4572± 0.0002 mercedes_benz_greener_manufacturing Model Objective Single E5 MLP MSE 8.6±0.18.6± 0.1 8.570±0.0078.570± 0.007 MAE 8.38±0.048.38± 0.04 8.34±0.028.34± 0.02 Huber Loss 8.40±0.058.40± 0.05 8.37±0.018.37± 0.01 Soft Binning 9.4±0.59.4± 0.5 9.0±0.19.0± 0.1 Hard Binning 8.51±0.088.51± 0.08 8.46±0.028.46± 0.02 Joint End-To-End 8.45±0.068.45± 0.06 8.365±0.0028.365± 0.002 Decoder[N] 8.7±0.28.7± 0.2 8.47±0.038.47± 0.03 HL-Gaussian 8.68±0.108.68± 0.10 8.61±0.048.61± 0.04 Decoder[U] 8.8±0.28.8± 0.2 8.54±0.108.54± 0.10 Softmax MSE 8.62±0.108.62± 0.10 8.50±0.038.50± 0.03 Ordinal Entropy 8.57±0.098.57± 0.09 8.54±0.058.54± 0.05 ROT-REG 8.5±0.18.5± 0.1 8.43±0.028.43± 0.02 GAR [MAE] 8.34±0.048.34± 0.04 8.32±0.028.32± 0.02 GAR [MSE] 8.55±0.098.55± 0.09 8.50±0.098.50± 0.09 MDN 8.5±0.18.5± 0.1 8.41±0.088.41± 0.08 NLL 8.6±0.18.6± 0.1 8.51±0.088.51± 0.08 beta-NLL 8.8±0.28.8± 0.2 8.67±0.038.67± 0.03 AdaQR 8.36±0.048.36± 0.04 8.35±0.038.35± 0.03 Quantile Loss 8.41±0.058.41± 0.05 8.395±0.0068.395± 0.006 TabPack MSE 8.48±0.068.48± 0.06 – HL-Gaussian 8.48±0.098.48± 0.09 – Quantile Loss 8.46±0.078.46± 0.07 – AdaQR 8.46±0.078.46± 0.07 – RealMLP MSE 8.31±0.028.31± 0.02 8.303±0.0078.303± 0.007 HL-Gaussian 8.35±0.048.35± 0.04 8.339±0.0088.339± 0.008 Quantile Loss 8.22±0.018.22± 0.01 8.217±0.0048.217± 0.004 AdaQR†AdaQR 8.25±0.038.25± 0.03 8.24±0.018.24± 0.01 AdaQR w. global σ 8.29±0.048.29± 0.04 8.28±0.038.28± 0.03 TabM MSE 8.34±0.058.34± 0.05 8.32±0.038.32± 0.03 HL-Gaussian 8.33±0.048.33± 0.04 8.32±0.018.32± 0.01 Quantile Loss 8.24±0.028.24± 0.02 8.23±0.028.23± 0.02 AdaQR 8.28±0.038.28± 0.03 8.27±0.018.27± 0.01 Table 26: Extended Beyond results with the Muon optimizer (continued). miami_housing Model Objective Single E5 MLP MSE 0.1446±0.00080.1446± 0.0008 0.1427±0.00020.1427± 0.0002 MAE 0.148±0.0010.148± 0.001 0.1454±0.00040.1454± 0.0004 Huber Loss 0.1450±0.00050.1450± 0.0005 0.1438±0.00020.1438± 0.0002 Soft Binning 0.150±0.0020.150± 0.002 0.1465±0.00040.1465± 0.0004 Hard Binning 0.1486±0.00050.1486± 0.0005 0.1480±0.00020.1480± 0.0002 Joint End-To-End 0.147±0.0010.147± 0.001 0.1441±0.00060.1441± 0.0006 Decoder[N] 0.155±0.0020.155± 0.002 0.149±0.0010.149± 0.001 HL-Gaussian 0.144±0.0010.144± 0.001 0.1423±0.00040.1423± 0.0004 Decoder[U] 0.155±0.0020.155± 0.002 0.1488±0.00040.1488± 0.0004 Softmax MSE 0.1434±0.00070.1434± 0.0007 0.1424±0.00000.1424± 0.0000 Ordinal Entropy 0.1428±0.00070.1428± 0.0007 0.1420±0.00020.1420± 0.0002 ROT-REG 0.1432±0.00070.1432± 0.0007 0.1425±0.00040.1425± 0.0004 GAR [MAE] 0.1441±0.00070.1441± 0.0007 0.1430±0.00050.1430± 0.0005 GAR [MSE] 0.1435±0.00080.1435± 0.0008 0.1427±0.00060.1427± 0.0006 MDN 0.1458±0.00100.1458± 0.0010 0.1450±0.00030.1450± 0.0003 NLL 0.1466±0.00080.1466± 0.0008 0.1442±0.00030.1442± 0.0003 beta-NLL 0.1441±0.00070.1441± 0.0007 0.1428±0.00000.1428± 0.0000 AdaQR 0.1430±0.00050.1430± 0.0005 0.1422±0.00040.1422± 0.0004 Quantile Loss 0.144±0.0010.144± 0.001 0.1431±0.00050.1431± 0.0005 TabPack MSE 0.1323±0.00040.1323± 0.0004 – HL-Gaussian 0.1323±0.00030.1323± 0.0003 – Quantile Loss 0.1320±0.00040.1320± 0.0004 – AdaQR 0.1321±0.00040.1321± 0.0004 – RealMLP MSE 0.1348±0.00040.1348± 0.0004 0.1336±0.00010.1336± 0.0001 HL-Gaussian 0.1331±0.00070.1331± 0.0007 0.1315±0.00040.1315± 0.0004 Quantile Loss 0.1343±0.00070.1343± 0.0007 0.1325±0.00010.1325± 0.0001 AdaQR†AdaQR 0.1322±0.00050.1322± 0.0005 0.1316±0.00020.1316± 0.0002 AdaQR w. global σ 0.1330±0.00030.1330± 0.0003 0.1319±0.00010.1319± 0.0001 TabM MSE 0.1335±0.00060.1335± 0.0006 0.1329±0.00010.1329± 0.0001 HL-Gaussian 0.1341±0.00060.1341± 0.0006 0.1336±0.00010.1336± 0.0001 Quantile Loss 0.133±0.0010.133± 0.001 0.1321±0.00030.1321± 0.0003 AdaQR 0.1325±0.00060.1325± 0.0006 0.1319±0.00030.1319± 0.0003 mutual_funds_india Model Objective Single E5 MLP MSE 5.9±0.25.9± 0.2 5.66±0.035.66± 0.03 MAE 5.9±0.15.9± 0.1 5.65±0.045.65± 0.04 Huber Loss 5.9±0.15.9± 0.1 5.77±0.045.77± 0.04 Soft Binning 5.64±0.105.64± 0.10 5.59±0.055.59± 0.05 Hard Binning 6.2±0.16.2± 0.1 6.09±0.046.09± 0.04 Joint End-To-End 5.6±0.15.6± 0.1 5.56±0.055.56± 0.05 Decoder[N] 6.5±0.26.5± 0.2 6.26±0.076.26± 0.07 HL-Gaussian 5.56±0.085.56± 0.08 5.49±0.055.49± 0.05 Decoder[U] 7.3±0.47.3± 0.4 6.39±0.066.39± 0.06 Softmax MSE 6.0±0.16.0± 0.1 5.88±0.045.88± 0.04 Ordinal Entropy 6.0±0.26.0± 0.2 5.72±0.025.72± 0.02 ROT-REG 5.9±0.15.9± 0.1 5.71±0.065.71± 0.06 GAR [MAE] 5.79±0.095.79± 0.09 5.68±0.065.68± 0.06 GAR [MSE] 5.9±0.15.9± 0.1 5.79±0.055.79± 0.05 MDN 5.8±0.15.8± 0.1 5.73±0.045.73± 0.04 NLL 5.89±0.095.89± 0.09 5.776±0.0085.776± 0.008 beta-NLL 5.9±0.15.9± 0.1 5.76±0.095.76± 0.09 AdaQR 5.84±0.075.84± 0.07 5.719±0.0085.719± 0.008 Quantile Loss 5.90±0.095.90± 0.09 5.76±0.035.76± 0.03 TabPack MSE 5.07±0.085.07± 0.08 – HL-Gaussian 5.24±0.085.24± 0.08 – Quantile Loss 4.97±0.074.97± 0.07 – AdaQR 5.08±0.085.08± 0.08 – RealMLP MSE 6±26± 2 5.3±0.25.3± 0.2 HL-Gaussian 5.17±0.065.17± 0.06 5.14±0.045.14± 0.04 Quantile Loss 5.11±0.055.11± 0.05 5.09±0.025.09± 0.02 AdaQR†AdaQR 5.16±0.075.16± 0.07 5.100±0.0085.100± 0.008 AdaQR w. global σ 5.05±0.075.05± 0.07 5.007±0.0085.007± 0.008 TabM MSE 5.16±0.075.16± 0.07 5.12±0.025.12± 0.02 HL-Gaussian 5.2±0.15.2± 0.1 5.19±0.065.19± 0.06 Quantile Loss 5.0±0.15.0± 0.1 4.94±0.064.94± 0.06 AdaQR 4.95±0.064.95± 0.06 4.90±0.054.90± 0.05 Table 27: Extended Beyond results with the Muon optimizer (continued). obesity_estimation Model Objective Single E5 MLP MSE 4.91±0.034.91± 0.03 4.904±0.0024.904± 0.002 MAE 4.85±0.024.85± 0.02 4.84±0.014.84± 0.01 Huber Loss 4.94±0.054.94± 0.05 4.92±0.034.92± 0.03 Soft Binning 4.91±0.044.91± 0.04 4.90±0.024.90± 0.02 Hard Binning 4.96±0.054.96± 0.05 4.944±0.0064.944± 0.006 Joint End-To-End 4.96±0.024.96± 0.02 4.94±0.024.94± 0.02 Decoder[N] 5.1±0.25.1± 0.2 4.85±0.034.85± 0.03 HL-Gaussian 4.90±0.034.90± 0.03 4.895±0.0064.895± 0.006 Decoder[U] 5.3±0.15.3± 0.1 5.17±0.085.17± 0.08 Softmax MSE 4.95±0.044.95± 0.04 4.93±0.024.93± 0.02 Ordinal Entropy 4.89±0.034.89± 0.03 4.88±0.024.88± 0.02 ROT-REG 4.94±0.014.94± 0.01 4.926±0.0054.926± 0.005 GAR [MAE] 4.86±0.024.86± 0.02 4.845±0.0104.845± 0.010 GAR [MSE] 4.92±0.034.92± 0.03 4.912±0.0104.912± 0.010 MDN 4.89±0.024.89± 0.02 4.877±0.0054.877± 0.005 NLL 4.88±0.024.88± 0.02 4.870±0.0104.870± 0.010 beta-NLL 4.91±0.024.91± 0.02 4.904±0.0054.904± 0.005 AdaQR 4.90±0.034.90± 0.03 4.89±0.014.89± 0.01 Quantile Loss 4.86±0.024.86± 0.02 4.854±0.0054.854± 0.005 TabPack MSE 4.95±0.054.95± 0.05 – HL-Gaussian 4.93±0.054.93± 0.05 – Quantile Loss 4.88±0.034.88± 0.03 – AdaQR 4.89±0.034.89± 0.03 – RealMLP MSE 4.91±0.064.91± 0.06 4.84±0.024.84± 0.02 HL-Gaussian 5.0±0.15.0± 0.1 4.95±0.044.95± 0.04 Quantile Loss 4.89±0.044.89± 0.04 4.87±0.024.87± 0.02 AdaQR†AdaQR 4.825±0.0074.825± 0.007 4.823±0.0024.823± 0.002 AdaQR w. global σ 4.86±0.024.86± 0.02 4.851±0.0084.851± 0.008 TabM MSE 4.90±0.034.90± 0.03 4.88±0.024.88± 0.02 HL-Gaussian 4.88±0.024.88± 0.02 4.869±0.0064.869± 0.006 Quantile Loss 4.87±0.024.87± 0.02 4.87±0.014.87± 0.01 AdaQR 4.89±0.044.89± 0.04 4.872±0.0074.872± 0.007 physiochemical_protein Model Objective Single E5 MLP MSE 3.38±0.013.38± 0.01 3.342±0.0033.342± 0.003 MAE 3.43±0.023.43± 0.02 3.36±0.023.36± 0.02 Huber Loss 3.38±0.013.38± 0.01 3.359±0.0073.359± 0.007 Soft Binning 3.33±0.023.33± 0.02 3.297±0.0063.297± 0.006 Hard Binning 3.36±0.013.36± 0.01 3.313±0.0053.313± 0.005 Joint End-To-End 3.37±0.023.37± 0.02 3.317±0.0083.317± 0.008 Decoder[N] 3.9±0.13.9± 0.1 3.50±0.023.50± 0.02 HL-Gaussian 3.35±0.013.35± 0.01 3.31±0.013.31± 0.01 Decoder[U] 4.0±0.34.0± 0.3 3.50±0.053.50± 0.05 Softmax MSE 3.37±0.013.37± 0.01 3.343±0.0043.343± 0.004 Ordinal Entropy 3.36±0.013.36± 0.01 3.334±0.0063.334± 0.006 ROT-REG 3.36±0.013.36± 0.01 3.342±0.0093.342± 0.009 GAR [MAE] 3.6±0.73.6± 0.7 3.34±0.073.34± 0.07 GAR [MSE] 3.39±0.023.39± 0.02 3.338±0.0033.338± 0.003 MDN 3.366±0.0093.366± 0.009 3.322±0.0043.322± 0.004 NLL 3.47±0.023.47± 0.02 3.35±0.013.35± 0.01 beta-NLL 3.39±0.023.39± 0.02 3.337±0.0093.337± 0.009 AdaQR 3.37±0.023.37± 0.02 3.340±0.0053.340± 0.005 Quantile Loss 3.39±0.013.39± 0.01 3.369±0.0023.369± 0.002 TabPack MSE 3.24±0.013.24± 0.01 – HL-Gaussian 3.278±0.0103.278± 0.010 – Quantile Loss 3.241±0.0083.241± 0.008 – AdaQR 3.242±0.0103.242± 0.010 – RealMLP MSE 3.168±0.0093.168± 0.009 3.124±0.0033.124± 0.003 HL-Gaussian 3.17±0.013.17± 0.01 3.107±0.0013.107± 0.001 Quantile Loss 3.16±0.013.16± 0.01 3.126±0.0033.126± 0.003 AdaQR†AdaQR 3.165±0.0053.165± 0.005 3.1342±0.00073.1342± 0.0007 AdaQR w. global σ 3.258±0.0063.258± 0.006 3.231±0.0013.231± 0.001 TabM MSE 3.338±0.0103.338± 0.010 3.312±0.0043.312± 0.004 HL-Gaussian 3.30±0.043.30± 0.04 – Quantile Loss 3.39±0.063.39± 0.06 3.33±0.013.33± 0.01 AdaQR 3.35±0.053.35± 0.05 3.27±0.013.27± 0.01 Table 28: Extended Beyond results with the Muon optimizer (continued). qsar_aquatic_toxicity Model Objective Single E5 MLP MSE 1.17±0.021.17± 0.02 1.16±0.011.16± 0.01 MAE 1.28±0.031.28± 0.03 1.256±0.0081.256± 0.008 Huber Loss 1.16±0.021.16± 0.02 1.156±0.0041.156± 0.004 Soft Binning 1.14±0.021.14± 0.02 1.140±0.0071.140± 0.007 Hard Binning 1.22±0.041.22± 0.04 1.211±0.0091.211± 0.009 Joint End-To-End 1.20±0.031.20± 0.03 1.179±0.0061.179± 0.006 Decoder[N] 1.41±0.061.41± 0.06 1.33±0.051.33± 0.05 HL-Gaussian 1.17±0.011.17± 0.01 1.171±0.0051.171± 0.005 Decoder[U] 1.28±0.061.28± 0.06 1.15±0.061.15± 0.06 Softmax MSE 1.19±0.031.19± 0.03 1.177±0.0091.177± 0.009 Ordinal Entropy 1.17±0.021.17± 0.02 1.16±0.011.16± 0.01 ROT-REG 1.17±0.021.17± 0.02 1.16±0.011.16± 0.01 GAR [MAE] 1.19±0.041.19± 0.04 1.18±0.021.18± 0.02 GAR [MSE] 1.19±0.011.19± 0.01 1.182±0.0041.182± 0.004 MDN 1.23±0.021.23± 0.02 1.223±0.0041.223± 0.004 NLL 1.16±0.021.16± 0.02 1.144±0.0041.144± 0.004 beta-NLL 1.18±0.011.18± 0.01 1.179±0.0041.179± 0.004 AdaQR 1.18±0.021.18± 0.02 1.17±0.011.17± 0.01 Quantile Loss 1.19±0.011.19± 0.01 1.19±0.011.19± 0.01 TabPack MSE 1.18±0.021.18± 0.02 – HL-Gaussian 1.19±0.031.19± 0.03 – Quantile Loss 1.14±0.021.14± 0.02 – AdaQR 1.15±0.031.15± 0.03 – RealMLP MSE 1.15±0.021.15± 0.02 1.13±0.011.13± 0.01 HL-Gaussian 1.10±0.011.10± 0.01 1.099±0.0011.099± 0.001 Quantile Loss 1.18±0.021.18± 0.02 1.178±0.0091.178± 0.009 AdaQR†AdaQR 1.23±0.031.23± 0.03 1.206±0.0051.206± 0.005 AdaQR w. global σ 1.18±0.031.18± 0.03 1.17±0.021.17± 0.02 TabM MSE 1.16±0.051.16± 0.05 1.12±0.031.12± 0.03 HL-Gaussian 1.13±0.031.13± 0.03 1.13±0.021.13± 0.02 Quantile Loss 1.16±0.031.16± 0.03 1.148±0.0071.148± 0.007 AdaQR 1.19±0.031.19± 0.03 1.172±0.0041.172± 0.004 qsar_fish_toxicity Model Objective Single E5 MLP MSE 0.93±0.010.93± 0.01 0.918±0.0070.918± 0.007 MAE 0.950±0.0060.950± 0.006 0.949±0.0020.949± 0.002 Huber Loss 0.951±0.0080.951± 0.008 0.950±0.0040.950± 0.004 Soft Binning 0.870±0.0050.870± 0.005 0.864±0.0030.864± 0.003 Hard Binning 1.01±0.041.01± 0.04 0.92±0.020.92± 0.02 Joint End-To-End 0.924±0.0070.924± 0.007 0.921±0.0050.921± 0.005 Decoder[N] 0.97±0.010.97± 0.01 0.942±0.0020.942± 0.002 HL-Gaussian 0.889±0.0040.889± 0.004 0.888±0.0020.888± 0.002 Decoder[U] 0.96±0.020.96± 0.02 0.905±0.0090.905± 0.009 Softmax MSE 0.91±0.010.91± 0.01 0.901±0.0060.901± 0.006 Ordinal Entropy 0.93±0.010.93± 0.01 0.924±0.0070.924± 0.007 ROT-REG 0.94±0.020.94± 0.02 0.934±0.0070.934± 0.007 GAR [MAE] 0.939±0.0090.939± 0.009 0.937±0.0060.937± 0.006 GAR [MSE] 0.95±0.020.95± 0.02 0.941±0.0060.941± 0.006 MDN 0.99±0.050.99± 0.05 0.96±0.010.96± 0.01 NLL 0.98±0.030.98± 0.03 0.95±0.020.95± 0.02 beta-NLL 0.943±0.0070.943± 0.007 0.941±0.0010.941± 0.001 AdaQR 0.97±0.010.97± 0.01 0.962±0.0050.962± 0.005 Quantile Loss 0.98±0.010.98± 0.01 0.965±0.0080.965± 0.008 TabPack MSE 0.93±0.020.93± 0.02 – HL-Gaussian 0.88±0.020.88± 0.02 – Quantile Loss 0.92±0.010.92± 0.01 – AdaQR 0.93±0.020.93± 0.02 – RealMLP MSE 0.95±0.010.95± 0.01 0.94±0.010.94± 0.01 HL-Gaussian 0.91±0.020.91± 0.02 0.895±0.0070.895± 0.007 Quantile Loss 0.95±0.020.95± 0.02 0.930±0.0080.930± 0.008 AdaQR†AdaQR 0.908±0.0090.908± 0.009 0.897±0.0060.897± 0.006 AdaQR w. global σ 0.94±0.010.94± 0.01 0.929±0.0040.929± 0.004 TabM MSE 0.95±0.010.95± 0.01 0.937±0.0070.937± 0.007 HL-Gaussian 0.94±0.010.94± 0.01 0.938±0.0030.938± 0.003 Quantile Loss 0.889±0.0080.889± 0.008 0.886±0.0060.886± 0.006 AdaQR 0.95±0.020.95± 0.02 0.933±0.0030.933± 0.003 Table 29: Extended Beyond results with the Muon optimizer (continued). qsar_tid_11 Model Objective Single E5 MLP MSE 0.784±0.0050.784± 0.005 0.772±0.0010.772± 0.001 MAE 0.783±0.0050.783± 0.005 0.7720±0.00090.7720± 0.0009 Huber Loss 0.786±0.0060.786± 0.006 0.775±0.0030.775± 0.003 Soft Binning 0.787±0.0040.787± 0.004 0.782±0.0020.782± 0.002 Hard Binning 0.84±0.010.84± 0.01 0.819±0.0080.819± 0.008 Joint End-To-End 0.786±0.0030.786± 0.003 0.7814±0.00050.7814± 0.0005 Decoder[N] 0.83±0.010.83± 0.01 0.8022±0.00080.8022± 0.0008 HL-Gaussian 0.784±0.0040.784± 0.004 0.779±0.0030.779± 0.003 Decoder[U] 0.87±0.010.87± 0.01 0.810±0.0040.810± 0.004 Softmax MSE 0.796±0.0060.796± 0.006 0.784±0.0030.784± 0.003 Ordinal Entropy 0.791±0.0080.791± 0.008 0.772±0.0020.772± 0.002 ROT-REG 0.783±0.0060.783± 0.006 0.772±0.0020.772± 0.002 GAR [MAE] 0.781±0.0050.781± 0.005 0.770±0.0020.770± 0.002 GAR [MSE] 0.792±0.0040.792± 0.004 0.7810±0.00090.7810± 0.0009 MDN 0.783±0.0050.783± 0.005 0.771±0.0020.771± 0.002 NLL 0.784±0.0050.784± 0.005 0.775±0.0010.775± 0.001 beta-NLL 0.790±0.0060.790± 0.006 0.7786±0.00070.7786± 0.0007 AdaQR 0.790±0.0060.790± 0.006 0.775±0.0040.775± 0.004 Quantile Loss 0.781±0.0050.781± 0.005 0.771±0.0010.771± 0.001 TabPack MSE 0.749±0.0040.749± 0.004 – HL-Gaussian 0.747±0.0050.747± 0.005 – Quantile Loss 0.741±0.0030.741± 0.003 – AdaQR 0.739±0.0030.739± 0.003 – RealMLP MSE 0.757±0.0020.757± 0.002 0.751±0.0010.751± 0.001 HL-Gaussian 0.752±0.0030.752± 0.003 0.7480±0.00090.7480± 0.0009 Quantile Loss 0.759±0.0040.759± 0.004 0.7520±0.00070.7520± 0.0007 AdaQR†AdaQR 0.750±0.0020.750± 0.002 0.7468±0.00050.7468± 0.0005 AdaQR w. global σ 0.746±0.0040.746± 0.004 0.7388±0.00070.7388± 0.0007 TabM MSE 0.772±0.0040.772± 0.004 0.7645±0.00040.7645± 0.0004 HL-Gaussian 0.782±0.0060.782± 0.006 0.768±0.0030.768± 0.003 Quantile Loss 0.769±0.0060.769± 0.006 0.759±0.0010.759± 0.001 AdaQR 0.775±0.0060.775± 0.006 0.7604±0.00080.7604± 0.0008 rossmann_store_sales Model Objective Single E5 MLP MSE 1340±1251340± 125 1296±551296± 55 MAE 1362±851362± 85 1333±421333± 42 Huber Loss 1315±671315± 67 1292±411292± 41 Soft Binning 1172±951172± 95 1115±191115± 19 Hard Binning 1195±651195± 65 1175±451175± 45 Joint End-To-End 1256±311256± 31 1218±251218± 25 Decoder[N] 1289±831289± 83 1164±81164± 8 HL-Gaussian 1218±1331218± 133 1142±301142± 30 Decoder[U] 1252±841252± 84 1144±411144± 41 Softmax MSE 1270±821270± 82 1245±431245± 43 Ordinal Entropy 1319±671319± 67 1301±341301± 34 ROT-REG 1291±681291± 68 1274±631274± 63 GAR [MAE] 1283±441283± 44 1265±161265± 16 GAR [MSE] 1293±521293± 52 1284±271284± 27 MDN 1586±2241586± 224 1496±1151496± 115 NLL 1360±871360± 87 1327±401327± 40 beta-NLL 1248±191248± 19 1234±41234± 4 AdaQR 1457±1301457± 130 1228±321228± 32 Quantile Loss 1354±561354± 56 1331±411331± 41 TabPack MSE 994±23994± 23 – HL-Gaussian 1011±191011± 19 – Quantile Loss 997±39997± 39 – AdaQR 1009±301009± 30 – RealMLP MSE 924±13924± 13 913±4913± 4 HL-Gaussian 1040±81040± 8 1023±41023± 4 Quantile Loss 961±20961± 20 947±2947± 2 AdaQR†AdaQR 984±19984± 19 964±9964± 9 AdaQR w. global σ 1012±181012± 18 993±5993± 5 TabM MSE 1149±1641149± 164 1045±341045± 34 HL-Gaussian 1204±521204± 52 1180±321180± 32 Quantile Loss 1084±591084± 59 1009±231009± 23 AdaQR 1054±441054± 44 1006±151006± 15 Table 30: Extended Beyond results with the Muon optimizer (continued). santander_transaction_value Model Objective Single E5 MLP MSE 1.409±0.0061.409± 0.006 1.403±0.0021.403± 0.002 MAE 1.408±0.0071.408± 0.007 1.396±0.0031.396± 0.003 Huber Loss 1.403±0.0091.403± 0.009 1.394±0.0051.394± 0.005 Soft Binning 1.399±0.0091.399± 0.009 1.389±0.0031.389± 0.003 Hard Binning 1.415±0.0061.415± 0.006 1.409±0.0031.409± 0.003 Joint End-To-End 1.412±0.0101.412± 0.010 1.397±0.0091.397± 0.009 Decoder[N] 1.47±0.011.47± 0.01 1.415±0.0081.415± 0.008 HL-Gaussian 1.406±0.0071.406± 0.007 1.397±0.0051.397± 0.005 Decoder[U] 1.49±0.031.49± 0.03 1.412±0.0061.412± 0.006 Softmax MSE 1.415±0.0061.415± 0.006 1.407±0.0021.407± 0.002 Ordinal Entropy 1.410±0.0061.410± 0.006 1.404±0.0041.404± 0.004 ROT-REG 1.407±0.0061.407± 0.006 1.401±0.0031.401± 0.003 GAR [MAE] 1.393±0.0031.393± 0.003 1.3912±0.00091.3912± 0.0009 GAR [MSE] 1.413±0.0071.413± 0.007 1.404±0.0041.404± 0.004 MDN 1.408±0.0091.408± 0.009 1.395±0.0061.395± 0.006 NLL 1.390±0.0061.390± 0.006 1.3881±0.00031.3881± 0.0003 beta-NLL 1.400±0.0071.400± 0.007 1.392±0.0021.392± 0.002 AdaQR 1.395±0.0041.395± 0.004 1.391±0.0031.391± 0.003 Quantile Loss 1.393±0.0051.393± 0.005 1.388±0.0011.388± 0.001 TabPack MSE 1.361±0.0041.361± 0.004 – HL-Gaussian 1.359±0.0071.359± 0.007 – Quantile Loss 1.359±0.0031.359± 0.003 – AdaQR 1.361±0.0051.361± 0.005 – RealMLP MSE 1.377±0.0061.377± 0.006 1.369±0.0031.369± 0.003 HL-Gaussian 1.367±0.0041.367± 0.004 1.362±0.0011.362± 0.001 Quantile Loss 1.364±0.0031.364± 0.003 1.3628±0.00081.3628± 0.0008 AdaQR†AdaQR 1.369±0.0051.369± 0.005 1.367±0.0011.367± 0.001 AdaQR w. global σ 1.377±0.0041.377± 0.004 1.365±0.0041.365± 0.004 TabM MSE 1.379±0.0071.379± 0.007 1.3743±0.00071.3743± 0.0007 HL-Gaussian 1.380±0.0071.380± 0.007 1.373±0.0061.373± 0.006 Quantile Loss 1.38±0.011.38± 0.01 1.370±0.0021.370± 0.002 AdaQR 1.38±0.011.38± 0.01 1.373±0.0071.373± 0.007 sat11_hand_algo_runtime Model Objective Single E5 MLP MSE 2.02±0.072.02± 0.07 1.991±0.0071.991± 0.007 MAE 2.01±0.042.01± 0.04 1.98±0.021.98± 0.02 Huber Loss 2.04±0.042.04± 0.04 2.009±0.0072.009± 0.007 Soft Binning 1.96±0.031.96± 0.03 1.946±0.0061.946± 0.006 Hard Binning 2.02±0.072.02± 0.07 1.91±0.021.91± 0.02 Joint End-To-End 2.01±0.032.01± 0.03 1.977±0.0041.977± 0.004 Decoder[N] 2.2±0.12.2± 0.1 1.98±0.051.98± 0.05 HL-Gaussian 1.93±0.041.93± 0.04 1.88±0.011.88± 0.01 Decoder[U] 2.2±0.22.2± 0.2 1.97±0.101.97± 0.10 Softmax MSE 2.10±0.062.10± 0.06 2.02±0.032.02± 0.03 Ordinal Entropy 2.00±0.062.00± 0.06 1.96±0.011.96± 0.01 ROT-REG 1.96±0.031.96± 0.03 1.94±0.011.94± 0.01 GAR [MAE] 1.98±0.031.98± 0.03 1.95±0.011.95± 0.01 GAR [MSE] 2.01±0.042.01± 0.04 1.982±0.0051.982± 0.005 MDN 2.01±0.062.01± 0.06 1.943±0.0081.943± 0.008 NLL 1.93±0.031.93± 0.03 1.909±0.0071.909± 0.007 beta-NLL 1.99±0.051.99± 0.05 1.971±0.0091.971± 0.009 AdaQR 1.96±0.041.96± 0.04 1.945±0.0051.945± 0.005 Quantile Loss 1.94±0.031.94± 0.03 1.92±0.011.92± 0.01 TabPack MSE 1.83±0.041.83± 0.04 – HL-Gaussian 1.89±0.041.89± 0.04 – Quantile Loss 1.85±0.041.85± 0.04 – AdaQR 1.84±0.021.84± 0.02 – RealMLP MSE 1.96±0.041.96± 0.04 1.95±0.021.95± 0.02 HL-Gaussian 1.97±0.031.97± 0.03 1.93±0.021.93± 0.02 Quantile Loss 1.83±0.021.83± 0.02 1.820±0.0101.820± 0.010 AdaQR†AdaQR 1.91±0.021.91± 0.02 1.898±0.0061.898± 0.006 AdaQR w. global σ 1.90±0.021.90± 0.02 1.889±0.0071.889± 0.007 TabM MSE 1.90±0.021.90± 0.02 1.887±0.0051.887± 0.005 HL-Gaussian 1.87±0.031.87± 0.03 1.858±0.0061.858± 0.006 Quantile Loss 1.93±0.021.93± 0.02 1.922±0.0071.922± 0.007 AdaQR 1.92±0.031.92± 0.03 1.89±0.031.89± 0.03 Table 31: Extended Beyond results with the Muon optimizer (continued). sberbank_housing_market_forecasting Model Objective Single E5 MLP MSE 0.27±0.020.27± 0.02 0.26±0.020.26± 0.02 MAE 0.263±0.0100.263± 0.010 0.256±0.0040.256± 0.004 Huber Loss 0.26±0.010.26± 0.01 0.254±0.0060.254± 0.006 Soft Binning 0.263±0.0090.263± 0.009 0.2521±0.00100.2521± 0.0010 Hard Binning 0.254±0.0030.254± 0.003 0.2498±0.00030.2498± 0.0003 Joint End-To-End 0.255±0.0050.255± 0.005 0.248±0.0020.248± 0.002 Decoder[N] 0.264±0.0050.264± 0.005 0.256±0.0030.256± 0.003 HL-Gaussian 0.27±0.030.27± 0.03 0.253±0.0020.253± 0.002 Decoder[U] 0.28±0.020.28± 0.02 0.252±0.0030.252± 0.003 Softmax MSE 0.261±0.0060.261± 0.006 0.257±0.0050.257± 0.005 Ordinal Entropy 0.261±0.0050.261± 0.005 0.2555±0.00020.2555± 0.0002 ROT-REG 0.260±0.0070.260± 0.007 0.255±0.0020.255± 0.002 GAR [MAE] 0.263±0.0090.263± 0.009 0.257±0.0020.257± 0.002 GAR [MSE] 0.26±0.010.26± 0.01 0.257±0.0020.257± 0.002 MDN 0.29±0.020.29± 0.02 0.2617±0.00090.2617± 0.0009 NLL 0.257±0.0040.257± 0.004 0.2552±0.00060.2552± 0.0006 beta-NLL 0.256±0.0030.256± 0.003 0.251±0.0010.251± 0.001 AdaQR 0.254±0.0030.254± 0.003 0.2489±0.00040.2489± 0.0004 Quantile Loss 0.28±0.030.28± 0.03 0.27±0.020.27± 0.02 TabPack MSE 0.238±0.0020.238± 0.002 – HL-Gaussian 0.237±0.0010.237± 0.001 – Quantile Loss 0.2341±0.00090.2341± 0.0009 – AdaQR 0.234±0.0010.234± 0.001 – RealMLP MSE 0.2390±0.00090.2390± 0.0009 0.2380±0.00050.2380± 0.0005 HL-Gaussian 0.240±0.0020.240± 0.002 0.2382±0.00030.2382± 0.0003 Quantile Loss 0.2324±0.00080.2324± 0.0008 0.2318±0.00060.2318± 0.0006 AdaQR†AdaQR 0.236±0.0020.236± 0.002 0.235±0.0010.235± 0.001 AdaQR w. global σ 0.239±0.0020.239± 0.002 0.2378±0.00070.2378± 0.0007 TabM MSE 0.260±0.0030.260± 0.003 0.259±0.0020.259± 0.002 HL-Gaussian 0.241±0.0020.241± 0.002 0.2393±0.00030.2393± 0.0003 Quantile Loss 0.236±0.0010.236± 0.001 0.2350±0.00050.2350± 0.0005 AdaQR 0.240±0.0040.240± 0.004 0.238±0.0010.238± 0.001 sf_permit_time Model Objective Single E5 MLP MSE 1.46±0.011.46± 0.01 1.455±0.0031.455± 0.003 MAE 1.47±0.011.47± 0.01 1.468±0.0101.468± 0.010 Huber Loss 1.51±0.021.51± 0.02 1.492±0.0031.492± 0.003 Soft Binning 1.469±0.0071.469± 0.007 1.467±0.0051.467± 0.005 Hard Binning 1.493±0.0101.493± 0.010 1.489±0.0021.489± 0.002 Joint End-To-End 1.48±0.011.48± 0.01 1.475±0.0051.475± 0.005 Decoder[N] 1.57±0.051.57± 0.05 1.54±0.051.54± 0.05 HL-Gaussian 1.46±0.011.46± 0.01 1.455±0.0031.455± 0.003 Decoder[U] 1.53±0.051.53± 0.05 1.50±0.021.50± 0.02 Softmax MSE 1.49±0.011.49± 0.01 1.486±0.0081.486± 0.008 Ordinal Entropy 1.48±0.021.48± 0.02 1.465±0.0041.465± 0.004 ROT-REG 1.44±0.011.44± 0.01 1.434±0.0051.434± 0.005 GAR [MAE] 1.46±0.011.46± 0.01 1.461±0.0031.461± 0.003 GAR [MSE] 1.494±0.0091.494± 0.009 1.492±0.0021.492± 0.002 MDN 1.46±0.021.46± 0.02 1.450±0.0071.450± 0.007 NLL 1.465±0.0051.465± 0.005 1.464±0.0031.464± 0.003 beta-NLL 1.47±0.011.47± 0.01 1.471±0.0021.471± 0.002 AdaQR 1.45±0.021.45± 0.02 1.441±0.0061.441± 0.006 Quantile Loss 1.460±0.0081.460± 0.008 1.459±0.0021.459± 0.002 TabPack MSE 1.448±0.0061.448± 0.006 – HL-Gaussian 1.448±0.0041.448± 0.004 – Quantile Loss 1.447±0.0051.447± 0.005 – AdaQR 1.449±0.0091.449± 0.009 – RealMLP MSE 1.47±0.011.47± 0.01 1.465±0.0061.465± 0.006 HL-Gaussian 1.47±0.011.47± 0.01 1.472±0.0041.472± 0.004 Quantile Loss 1.457±0.0101.457± 0.010 1.454±0.0051.454± 0.005 AdaQR†AdaQR 1.46±0.011.46± 0.01 1.454±0.0061.454± 0.006 AdaQR w. global σ 1.47±0.011.47± 0.01 1.463±0.0021.463± 0.002 TabM MSE 1.47±0.021.47± 0.02 1.46±0.011.46± 0.01 HL-Gaussian 1.48±0.011.48± 0.01 1.480±0.0051.480± 0.005 Quantile Loss 1.47±0.031.47± 0.03 1.46±0.021.46± 0.02 AdaQR 1.48±0.021.48± 0.02 1.474±0.0071.474± 0.007 Table 32: Extended Beyond results with the Muon optimizer (continued). student_portuguese_performance Model Objective Single E5 MLP MSE 3.09±0.093.09± 0.09 3.00±0.033.00± 0.03 MAE 3.03±0.083.03± 0.08 2.96±0.032.96± 0.03 Huber Loss 3.04±0.083.04± 0.08 2.99±0.032.99± 0.03 Soft Binning 2.99±0.092.99± 0.09 2.92±0.032.92± 0.03 Hard Binning 3.00±0.023.00± 0.02 2.979±0.0022.979± 0.002 Joint End-To-End 2.97±0.042.97± 0.04 2.94±0.022.94± 0.02 Decoder[N] 3.17±0.093.17± 0.09 2.997±0.0082.997± 0.008 HL-Gaussian 2.99±0.042.99± 0.04 2.96±0.012.96± 0.01 Decoder[U] 3.2±0.23.2± 0.2 3.2±0.33.2± 0.3 Softmax MSE 2.96±0.042.96± 0.04 2.92±0.022.92± 0.02 Ordinal Entropy 3.11±0.063.11± 0.06 3.06±0.013.06± 0.01 ROT-REG 3.08±0.083.08± 0.08 2.99±0.022.99± 0.02 GAR [MAE] 3.03±0.073.03± 0.07 2.94±0.022.94± 0.02 GAR [MSE] 3.01±0.063.01± 0.06 2.92±0.032.92± 0.03 MDN 2.99±0.072.99± 0.07 2.90±0.022.90± 0.02 NLL 3.03±0.083.03± 0.08 2.96±0.062.96± 0.06 beta-NLL 3.03±0.063.03± 0.06 2.98±0.032.98± 0.03 AdaQR 3.02±0.083.02± 0.08 2.96±0.032.96± 0.03 Quantile Loss 2.94±0.042.94± 0.04 2.90±0.012.90± 0.01 TabPack MSE 2.82±0.032.82± 0.03 – HL-Gaussian 2.81±0.052.81± 0.05 – Quantile Loss 2.84±0.032.84± 0.03 – AdaQR 2.81±0.042.81± 0.04 – RealMLP MSE 2.92±0.072.92± 0.07 2.821±0.0082.821± 0.008 HL-Gaussian 2.87±0.042.87± 0.04 2.849±0.0052.849± 0.005 Quantile Loss 2.84±0.022.84± 0.02 2.83±0.012.83± 0.01 AdaQR†AdaQR 2.79±0.032.79± 0.03 2.77±0.012.77± 0.01 AdaQR w. global σ 2.85±0.022.85± 0.02 2.83±0.012.83± 0.01 TabM MSE 2.92±0.032.92± 0.03 2.91±0.012.91± 0.01 HL-Gaussian 2.93±0.022.93± 0.02 2.92±0.012.92± 0.01 Quantile Loss 2.91±0.012.91± 0.01 2.889±0.0022.889± 0.002 AdaQR 2.92±0.022.92± 0.02 2.90±0.012.90± 0.01 superconductivity Model Objective Single E5 MLP MSE 9.76±0.049.76± 0.04 9.643±0.0069.643± 0.006 MAE 9.88±0.089.88± 0.08 9.74±0.059.74± 0.05 Huber Loss 9.88±0.069.88± 0.06 9.73±0.029.73± 0.02 Soft Binning 9.77±0.049.77± 0.04 9.67±0.019.67± 0.01 Hard Binning 10.06±0.0910.06± 0.09 9.86±0.019.86± 0.01 Joint End-To-End 9.73±0.039.73± 0.03 9.64±0.019.64± 0.01 Decoder[N] 10.8±0.310.8± 0.3 10.2±0.110.2± 0.1 HL-Gaussian 9.79±0.059.79± 0.05 9.705±0.0089.705± 0.008 Decoder[U] 12±112± 1 10.3±0.110.3± 0.1 Softmax MSE 9.77±0.039.77± 0.03 9.70±0.029.70± 0.02 Ordinal Entropy 9.73±0.049.73± 0.04 9.63±0.029.63± 0.02 ROT-REG 9.79±0.049.79± 0.04 9.67±0.029.67± 0.02 GAR [MAE] 9.92±0.079.92± 0.07 9.77±0.049.77± 0.04 GAR [MSE] 9.78±0.039.78± 0.03 9.66±0.019.66± 0.01 MDN 9.89±0.089.89± 0.08 9.64±0.019.64± 0.01 NLL 9.9±0.19.9± 0.1 9.67±0.049.67± 0.04 beta-NLL 9.75±0.049.75± 0.04 9.69±0.039.69± 0.03 AdaQR 9.75±0.059.75± 0.05 9.67±0.029.67± 0.02 Quantile Loss 9.74±0.049.74± 0.04 9.594±0.0059.594± 0.005 TabPack MSE 9.47±0.029.47± 0.02 – HL-Gaussian 9.50±0.039.50± 0.03 – Quantile Loss 9.44±0.039.44± 0.03 – AdaQR 9.42±0.049.42± 0.04 – RealMLP MSE 9.64±0.029.64± 0.02 9.607±0.0069.607± 0.006 HL-Gaussian 9.54±0.029.54± 0.02 9.477±0.0039.477± 0.003 Quantile Loss 9.62±0.039.62± 0.03 9.59±0.029.59± 0.02 AdaQR†AdaQR 9.51±0.039.51± 0.03 9.462±0.0109.462± 0.010 AdaQR w. global σ 9.55±0.029.55± 0.02 9.507±0.0109.507± 0.010 TabM MSE 9.71±0.079.71± 0.07 9.642±0.0069.642± 0.006 HL-Gaussian 9.65±0.039.65± 0.03 9.588±0.0039.588± 0.003 Quantile Loss 9.62±0.049.62± 0.04 9.53±0.019.53± 0.01 AdaQR 9.62±0.049.62± 0.04 9.58±0.029.58± 0.02 Table 33: Extended Beyond results with the Muon optimizer (continued). telemonitoring_parkinsons_biomedical_voice_measurements Model Objective Single E5 MLP MSE 12±212± 2 11.6±0.311.6± 0.3 MAE 11.3±0.511.3± 0.5 11.0±0.311.0± 0.3 Huber Loss 12±112± 1 11.40±0.0811.40± 0.08 Soft Binning 10.9±0.310.9± 0.3 10.6±0.210.6± 0.2 Hard Binning 11.2±0.711.2± 0.7 10.27±0.0510.27± 0.05 Joint End-To-End 10.2±0.210.2± 0.2 10.1±0.210.1± 0.2 Decoder[N] 13±113± 1 11.4±0.611.4± 0.6 HL-Gaussian 10.7±0.310.7± 0.3 10.5±0.110.5± 0.1 Decoder[U] 11.3±1.011.3± 1.0 10.2±0.210.2± 0.2 Softmax MSE 11.7±0.511.7± 0.5 11.0±0.111.0± 0.1 Ordinal Entropy – – ROT-REG 11.7±0.711.7± 0.7 11.2±0.311.2± 0.3 GAR [MAE] – – GAR [MSE] – – MDN 12±212± 2 10.7±0.610.7± 0.6 NLL 11.1±0.711.1± 0.7 10.6±0.310.6± 0.3 beta-NLL 11.4±0.611.4± 0.6 10.9±0.310.9± 0.3 AdaQR 11.1±0.611.1± 0.6 10.8±0.210.8± 0.2 Quantile Loss 12±112± 1 11.2±0.511.2± 0.5 TabPack MSE 11.3±0.511.3± 0.5 – HL-Gaussian 10.5±0.810.5± 0.8 – Quantile Loss 10.9±0.710.9± 0.7 – AdaQR 11.2±0.611.2± 0.6 – RealMLP MSE 10.6±0.410.6± 0.4 10.4±0.110.4± 0.1 HL-Gaussian 10.9±0.410.9± 0.4 10.7±0.310.7± 0.3 Quantile Loss 11.0±0.311.0± 0.3 10.88±0.0810.88± 0.08 AdaQR†AdaQR 10.8±0.410.8± 0.4 10.6±0.310.6± 0.3 AdaQR w. global σ 10.8±0.210.8± 0.2 10.8±0.110.8± 0.1 TabM MSE 11.1±0.211.1± 0.2 11.09±0.0511.09± 0.05 HL-Gaussian 11.1±0.411.1± 0.4 10.9±0.210.9± 0.2 Quantile Loss 10.9±0.110.9± 0.1 10.83±0.0310.83± 0.03 AdaQR 10.8±0.210.8± 0.2 10.7±0.110.7± 0.1 video_game_fps_prediction Model Objective Single E5 MLP MSE 0.008±0.0020.008± 0.002 0.0052±0.00030.0052± 0.0003 MAE 0.008±0.0020.008± 0.002 0.0064±0.00050.0064± 0.0005 Huber Loss 0.010±0.0020.010± 0.002 0.0077±0.00020.0077± 0.0002 Soft Binning 0.0040±0.00060.0040± 0.0006 0.0034±0.00020.0034± 0.0002 Hard Binning 0.046±0.0040.046± 0.004 0.0444±0.00050.0444± 0.0005 Joint End-To-End 0.0100±0.00090.0100± 0.0009 0.0084±0.00050.0084± 0.0005 Decoder[N] 0.026±0.0030.026± 0.003 0.022±0.0020.022± 0.002 HL-Gaussian 0.006±0.0010.006± 0.001 0.0044±0.00070.0044± 0.0007 Decoder[U] 0.039±0.0040.039± 0.004 0.034±0.0040.034± 0.004 Softmax MSE 0.010±0.0020.010± 0.002 0.0081±0.00010.0081± 0.0001 Ordinal Entropy 0.007±0.0010.007± 0.001 0.0063±0.00020.0063± 0.0002 ROT-REG 0.008±0.0020.008± 0.002 0.0074±0.00100.0074± 0.0010 GAR [MAE] 0.0053±0.00090.0053± 0.0009 0.0041±0.00050.0041± 0.0005 GAR [MSE] 0.0074±0.00080.0074± 0.0008 0.0067±0.00020.0067± 0.0002 MDN 0.014±0.0040.014± 0.004 0.010±0.0010.010± 0.001 NLL 0.010±0.0020.010± 0.002 0.008±0.0010.008± 0.001 beta-NLL 0.009±0.0010.009± 0.001 0.0065±0.00020.0065± 0.0002 AdaQR 0.0025±0.00080.0025± 0.0008 0.0019±0.00010.0019± 0.0001 Quantile Loss 0.0032±0.00080.0032± 0.0008 0.0025±0.00050.0025± 0.0005 TabPack MSE 0.0041±0.00050.0041± 0.0005 – HL-Gaussian 0.0037±0.00050.0037± 0.0005 – Quantile Loss 0.0035±0.00030.0035± 0.0003 – AdaQR 0.0032±0.00040.0032± 0.0004 – RealMLP MSE 0.0011±0.00000.0011± 0.0000 0.0010±0.00000.0010± 0.0000 HL-Gaussian 0.0011±0.00000.0011± 0.0000 0.0011±0.00000.0011± 0.0000 Quantile Loss 0.0012±0.00000.0012± 0.0000 0.0011±0.00000.0011± 0.0000 AdaQR†AdaQR 0.0010±0.00000.0010± 0.0000 0.0010±0.00000.0010± 0.0000 AdaQR w. global σ 0.0011±0.00000.0011± 0.0000 0.0011±0.00000.0011± 0.0000 TabM MSE 0.0029±0.00030.0029± 0.0003 0.0027±0.00000.0027± 0.0000 HL-Gaussian 0.0023±0.00020.0023± 0.0002 0.0022±0.00000.0022± 0.0000 Quantile Loss 0.0019±0.00040.0019± 0.0004 0.0017±0.00010.0017± 0.0001 AdaQR 0.0015±0.00020.0015± 0.0002 0.0014±0.00000.0014± 0.0000 Table 34: Extended Beyond results with the Muon optimizer (continued). video_transcoding_time_prediction Model Objective Single E5 MLP MSE 0.269±0.0070.269± 0.007 0.2594±0.00100.2594± 0.0010 MAE 0.261±0.0080.261± 0.008 0.255±0.0040.255± 0.004 Huber Loss 0.259±0.0070.259± 0.007 0.255±0.0050.255± 0.005 Soft Binning 0.252±0.0030.252± 0.003 0.250±0.0010.250± 0.001 Hard Binning 0.264±0.0030.264± 0.003 0.2628±0.00080.2628± 0.0008 Joint End-To-End 0.244±0.0050.244± 0.005 0.2402±0.00060.2402± 0.0006 Decoder[N] 0.249±0.0040.249± 0.004 0.244±0.0020.244± 0.002 HL-Gaussian 0.264±0.0090.264± 0.009 0.2585±0.00080.2585± 0.0008 Decoder[U] 0.26±0.020.26± 0.02 0.245±0.0030.245± 0.003 Softmax MSE 0.254±0.0070.254± 0.007 0.247±0.0020.247± 0.002 Ordinal Entropy 0.253±0.0050.253± 0.005 0.251±0.0010.251± 0.001 ROT-REG 0.265±0.0060.265± 0.006 0.259±0.0010.259± 0.001 GAR [MAE] 0.26±0.010.26± 0.01 0.251±0.0030.251± 0.003 GAR [MSE] 0.257±0.0060.257± 0.006 0.253±0.0020.253± 0.002 MDN 0.255±0.0040.255± 0.004 0.252±0.0020.252± 0.002 NLL 0.255±0.0070.255± 0.007 0.251±0.0030.251± 0.003 beta-NLL 0.253±0.0060.253± 0.006 0.250±0.0040.250± 0.004 AdaQR 0.250±0.0040.250± 0.004 0.247±0.0020.247± 0.002 Quantile Loss 0.257±0.0040.257± 0.004 0.254±0.0020.254± 0.002 TabPack MSE 0.241±0.0080.241± 0.008 – HL-Gaussian 0.259±0.0050.259± 0.005 – Quantile Loss 0.240±0.0070.240± 0.007 – AdaQR 0.240±0.0070.240± 0.007 – RealMLP MSE 0.270±0.0060.270± 0.006 0.269±0.0040.269± 0.004 HL-Gaussian 0.270±0.0050.270± 0.005 0.269±0.0030.269± 0.003 Quantile Loss 0.251±0.0040.251± 0.004 0.249±0.0010.249± 0.001 AdaQR†AdaQR 0.246±0.0040.246± 0.004 0.245±0.0020.245± 0.002 AdaQR w. global σ 0.251±0.0030.251± 0.003 0.2505±0.00040.2505± 0.0004 TabM MSE 0.243±0.0070.243± 0.007 0.238±0.0020.238± 0.002 HL-Gaussian 0.250±0.0040.250± 0.004 0.249±0.0020.249± 0.002 Quantile Loss 0.238±0.0050.238± 0.005 0.236±0.0030.236± 0.003 AdaQR 0.250±0.0060.250± 0.006 0.246±0.0010.246± 0.001 wine_quality Model Objective Single E5 MLP MSE 0.650±0.0030.650± 0.003 0.647±0.0020.647± 0.002 MAE 0.653±0.0030.653± 0.003 0.650±0.0020.650± 0.002 Huber Loss 0.648±0.0030.648± 0.003 0.6451±0.00040.6451± 0.0004 Soft Binning 0.640±0.0010.640± 0.001 0.6387±0.00040.6387± 0.0004 Hard Binning 0.646±0.0020.646± 0.002 0.6431±0.00030.6431± 0.0003 Joint End-To-End 0.650±0.0030.650± 0.003 0.6466±0.00080.6466± 0.0008 Decoder[N] 0.723±0.0060.723± 0.006 0.69±0.020.69± 0.02 HL-Gaussian 0.645±0.0020.645± 0.002 0.6435±0.00020.6435± 0.0002 Decoder[U] 0.77±0.060.77± 0.06 0.718±0.0100.718± 0.010 Softmax MSE 0.6457±0.00080.6457± 0.0008 0.6444±0.00010.6444± 0.0001 Ordinal Entropy 0.649±0.0020.649± 0.002 0.6477±0.00050.6477± 0.0005 ROT-REG 0.649±0.0030.649± 0.003 0.6452±0.00060.6452± 0.0006 GAR [MAE] 0.650±0.0030.650± 0.003 0.649±0.0010.649± 0.001 GAR [MSE] 0.646±0.0020.646± 0.002 0.6436±0.00040.6436± 0.0004 MDN 0.662±0.0040.662± 0.004 0.650±0.0020.650± 0.002 NLL 0.647±0.0010.647± 0.001 0.6448±0.00020.6448± 0.0002 beta-NLL 0.650±0.0020.650± 0.002 0.6470±0.00090.6470± 0.0009 AdaQR 0.646±0.0020.646± 0.002 0.6453±0.00090.6453± 0.0009 Quantile Loss 0.646±0.0020.646± 0.002 0.6435±0.00040.6435± 0.0004 TabPack MSE 0.623±0.0040.623± 0.004 – HL-Gaussian 0.621±0.0030.621± 0.003 – Quantile Loss 0.624±0.0040.624± 0.004 – AdaQR 0.622±0.0030.622± 0.003 – RealMLP MSE 0.625±0.0030.625± 0.003 0.618±0.0010.618± 0.001 HL-Gaussian 0.627±0.0040.627± 0.004 0.6188±0.00020.6188± 0.0002 Quantile Loss 0.625±0.0030.625± 0.003 0.620±0.0010.620± 0.001 AdaQR†AdaQR 0.622±0.0020.622± 0.002 0.618±0.0020.618± 0.002 AdaQR w. global σ 0.629±0.0020.629± 0.002 0.625±0.0020.625± 0.002 TabM MSE 0.631±0.0070.631± 0.007 0.625±0.0010.625± 0.001 HL-Gaussian 0.625±0.0030.625± 0.003 0.6191±0.00060.6191± 0.0006 Quantile Loss 0.631±0.0050.631± 0.005 0.625±0.0030.625± 0.003 AdaQR 0.630±0.0030.630± 0.003 0.6256±0.00050.6256± 0.0005 Table 35: Extended Beyond results with the Muon optimizer (continued). wine_world_cost Model Objective Single E5 MLP MSE 0.43±0.010.43± 0.01 0.4116±0.00080.4116± 0.0008 MAE 0.415±0.0050.415± 0.005 0.4119±0.00090.4119± 0.0009 Huber Loss 0.431±0.0100.431± 0.010 0.414±0.0070.414± 0.007 Soft Binning 0.440±0.0070.440± 0.007 0.421±0.0020.421± 0.002 Hard Binning 0.450±0.0060.450± 0.006 0.445±0.0020.445± 0.002 Joint End-To-End 0.43±0.010.43± 0.01 0.419±0.0060.419± 0.006 Decoder[N] 0.476±0.0090.476± 0.009 0.447±0.0060.447± 0.006 HL-Gaussian 0.426±0.0080.426± 0.008 0.414±0.0050.414± 0.005 Decoder[U] 0.49±0.010.49± 0.01 0.45±0.010.45± 0.01 Softmax MSE 0.427±0.0070.427± 0.007 0.416±0.0020.416± 0.002 Ordinal Entropy 0.422±0.0080.422± 0.008 0.411±0.0050.411± 0.005 ROT-REG 0.431±0.0090.431± 0.009 0.419±0.0030.419± 0.003 GAR [MAE] 0.415±0.0080.415± 0.008 0.4084±0.00090.4084± 0.0009 GAR [MSE] 0.417±0.0040.417± 0.004 0.4093±0.00030.4093± 0.0003 MDN 0.44±0.010.44± 0.01 0.420±0.0080.420± 0.008 NLL 0.417±0.0040.417± 0.004 0.4127±0.00030.4127± 0.0003 beta-NLL 0.419±0.0070.419± 0.007 0.414±0.0030.414± 0.003 AdaQR 0.423±0.0060.423± 0.006 0.4163±0.00090.4163± 0.0009 Quantile Loss 0.425±0.0070.425± 0.007 0.417±0.0030.417± 0.003 TabPack MSE 0.386±0.0030.386± 0.003 – HL-Gaussian 0.40±0.010.40± 0.01 – Quantile Loss 0.387±0.0030.387± 0.003 – AdaQR 0.392±0.0030.392± 0.003 – RealMLP MSE 0.383±0.0030.383± 0.003 0.381±0.0020.381± 0.002 HL-Gaussian 0.389±0.0040.389± 0.004 0.387±0.0030.387± 0.003 Quantile Loss 0.394±0.0030.394± 0.003 0.393±0.0010.393± 0.001 AdaQR†AdaQR 0.390±0.0090.390± 0.009 0.386±0.0040.386± 0.004 AdaQR w. global σ 0.389±0.0050.389± 0.005 0.386±0.0010.386± 0.001 TabM MSE 0.402±0.0030.402± 0.003 0.400±0.0020.400± 0.002 HL-Gaussian 0.395±0.0050.395± 0.005 0.392±0.0010.392± 0.001 Quantile Loss 0.398±0.0060.398± 0.006 0.395±0.0030.395± 0.003 AdaQR 0.404±0.0020.404± 0.002 0.4029±0.00060.4029± 0.0006