Paper deep dive
Loss Choice or Model Choice? The Role of Forecast Level in Cryptocurrency Volatility Forecasting
Andrzej Tokajuk, Jarosław A. Chudziak
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/24/2026, 4:34:11 AM
Summary
This study investigates the relative impact of training loss functions versus forecasting models on cryptocurrency volatility forecasts. Using five models (HAR, HAR-X, NeuralHAR, DLinear, LightGBM) and seven loss functions across five major cryptocurrencies, the authors find that raw comparisons show loss choice as the dominant source of variation. However, after aligning forecast levels using validation data, model choice becomes the primary driver of variation, and differences in Value-at-Risk (VaR) breach rates narrow substantially. The results indicate that much of the apparent superiority of certain loss functions in raw metrics is due to forecast level shifts rather than improved temporal dynamics.
Entities (21)
Relation Signals (9)
Loss Choice → dominates → Model Choice
confidence 95% · Before alignment, marginal score variation is greater across losses... The median is 2.91... making cross-loss differences about 2.9 times larger.
Model Choice → dominates → Loss Choice
confidence 95% · After alignment, model choice becomes the larger source of variation in the full five-model comparison... The loss-to-model ratio decreases from 2.91 to 0.67.
Forecast Level → explains → Raw Loss Variation
confidence 95% · Level share is the fraction of squared log-forecast differences explained by a constant offset... Alignment reduces cross-loss variation by 59%-88%.
Forecast Level → propagatesto → Value-at-Risk
confidence 95% · Most raw risk differences thus arise from persistent forecast-level shifts rather than daily signals.
QLIKE → targets → Mean
confidence 95% · QLIKE targets the mean.
MSE-log → targets → Geometric Mean
confidence 95% · MSE-log targets the conditional geometric mean.
Pinball → targets → Median
confidence 95% · Pinball targets the median.
HAR → hashigh → Level Share
confidence 90% · Level share is 89% for HAR but 56% for LightGBM.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Volatility forecasts play a central role in financial risk management because their overall level and day-to-day movements affect downstream decisions. Most studies compare forecasting models while keeping the training loss fixed. Yet losses emphasise different errors and can target different properties of future volatility, so raw comparisons may combine persistent forecast-level differences with differences in daily forecast movements. This leaves unresolved whether the importance of loss choice comes mainly from the forecast level it targets or from differences that remain after level adjustment. We address this gap through a comparison of seven losses and five models across major cryptocurrencies. Validation-based alignment adjusts the forecast level before the raw and aligned forecasts are evaluated using statistical scores and one-day Value-at-Risk. Before alignment, marginal score variation is greater across losses. After alignment, model choice becomes the larger source of variation in the full five-model comparison, while cross-loss differences in VaR breach rates narrow substantially. Our contribution is a comprehensive evaluation of loss and model choice that shows why losses can appear so influential in raw comparisons and how this interpretation changes when forecast level and downstream risk are considered explicitly.
Tags
Links
- Source: https://arxiv.org/abs/2609.27024v1
- Canonical: https://arxiv.org/abs/2609.27024v1
Trouble viewing inline? Open PDF directly →
Full Text
21,189 characters extracted from source content.
Expand or collapse full text
Loss Choice or Model Choice? The Role of Forecast Level in Cryptocurrency Volatility Forecasting Andrzej Tokajuk Jarosław A. Chudziak Affiliation: Warsaw University of Technology, Warsaw, Poland E-mail andrzej.tokajuk.stud,jaroslaw.chudziak@pw.edu.pl Abstract Volatility forecasts play a central role in financial risk management because their overall level and day-to-day movements affect downstream decisions. Most studies compare forecasting models while keeping the training loss fixed. Yet losses emphasize different errors and can target different properties of future volatility, so raw comparisons may combine persistent forecast-level differences with differences in daily forecast movements. This leaves unresolved whether the importance of loss choice comes mainly from the forecast level it targets or from differences that remain after level adjustment. We address this gap through a comparison of seven losses and five models across major cryptocurrencies. Validation-based alignment adjusts the forecast level before the raw and aligned forecasts are evaluated using statistical scores and one-day Value-at-Risk. Before alignment, marginal score differences are larger across training losses. After alignment, they are larger across model configurations in the full comparison, while cross-loss differences in VaR breach rates narrow substantially. Our contribution is a comprehensive evaluation of loss and model choice that shows why losses can appear so influential in raw comparisons and how this interpretation changes when forecast level and downstream risk are considered explicitly. Keywords: Loss functions Volatility forecasting Cryptocurrency Value-at-Risk 1 Introduction Volatility forecasts translate market movements into estimates used for position sizing, capital allocation and Value-at-Risk (VaR). Their usefulness depends on their overall level and movements from one day to the next. Forecasts can follow similar daily paths while implying substantially different levels of financial risk. Volatility forecasting research has mainly focused on model design, with the training loss typically held fixed when competing models are evaluated. This choice is not neutral: different losses can favour different quantities of the conditional distribution [6], leading the same model trained on the same data to produce different forecasts. Raw score differences may thus combine persistent forecast-level shifts with differences in daily forecast movements. Because forecast level feeds directly into risk measures, this ambiguity affects both statistical evaluation and practical use. These considerations lead to three research questions: RQ1 How large is the effect of the loss function relative to the effect of the forecasting model? RQ2 Does the loss mainly change the forecast level, or does it also change its day-to-day dynamics? RQ3 How do these differences affect one-day VaR before and after forecast-level alignment? We investigate these questions through a walk-forward comparison of seven losses and five models across five major cryptocurrencies, holding non-loss settings fixed. We compare marginal score variation across the two factors, align forecast levels using validation data, and evaluate raw and aligned forecasts using statistical scores and one-day VaR. The analysis shows that forecast level is central to interpreting differences attributed to training loss, while the differences remaining after alignment depend on the model. The VaR evaluation further shows that this distinction carries into risk estimates. Our contribution is a comprehensive evaluation of loss and model choice that clarifies what raw comparisons capture, what remains after forecast-level adjustment, and why this distinction matters for financial risk. 2 Related Work Volatility forecasting research has primarily focused on model design, with HAR remaining a strong baseline across horizons [3]. Evidence is mixed: machine-learning gains in U.S. equities are strongest at longer horizons and with informative predictors [2]. Broader comparisons find no general advantage of nonlinear models over linear alternatives [1], while cryptocurrency results depend on the asset, evaluation metric and forecast horizon [4]. In contrast, the role of the training loss has received much less attention in volatility forecasting, although the broader forecasting literature shows that it is not merely an optimisation choice. A loss determines which property of the target distribution a model is trained to estimate, so forecasts obtained with different losses need not represent the same conditional quantity [6]. This distinction is particularly important for volatility, which is not observed directly and must be approximated using a proxy. Forecast rankings may therefore depend on both the evaluation score and the volatility proxy [7, 13]. In related financial research, Transformer architectures have been compared for stock selection [9], while a separate study examines how training losses affect portfolio outcomes within a fixed Transformer [10]. For volatility forecasting, QLIKE training has improved finite-sample HAR forecasts [14]. These studies leave open how loss choice compares with model choice and how much of the difference reflects forecast level. We examine both questions and their implications for VaR. 3 Methodology In this section, we define forecasts targeted by each loss, compare loss and model effects, and use alignment to distinguish forecast level from day-to-day dynamics. 3.1 Loss Functions The training loss determines which errors a model avoids and hence which feature of future variance it estimates. Let h>0h>0 denote the observed variance proxy and f>0f>0 its one-day-ahead forecast. We use the log error u=logh−logfu= h- f and the variance ratio r=h/fr=h/f. An exact forecast gives u=0u=0 and r=1r=1, while u>0u>0 or r>1r>1 indicates underprediction. Each loss can favour a different forecast of ht+1h_t+1 given ℱtF_t - we call this its forecast target. All means and medians in Table 1 are conditional on ℱtF_t. Huber-log uses ρδ(u)=u2/2 _δ(u)=u^2/2 for |u|≤δ|u|≤δ and δ(|u|−δ/2)δ(|u|-δ/2) otherwise, with δ=1.0δ=1.0. Here mδm_δ is the log-variance value that minimizes the conditional expected Huber loss, and exp(mδ) (m_δ) transforms it back to variance units. Table 1: Per-observation losses and the forecasts they target. Group Loss L(h,f)L(h,f) Forecast target Log error MSE-logMSE-log u2u^2 geometric mean Huber-log ρδ(u) _δ(u) exp(mδ) (m_δ) Pinball |u|/2|u|/2 median Mean target QLIKEQLIKE r−logr−1r- r-1 mean Patton-b f−h+hlogr,b=−1f-h+h r,\ b=-1 mean Relative error HMAE |r−1||r-1| h-weighted median HMSE (r−1)2(r-1)^2 [h2∣ℱt]/[h∣ℱt]E[h^2 _t]/E[h _t] MSE-log targets the conditional geometric mean, whereas Huber-log gives less weight to large log errors. HMAE instead weights observations by h. During variance spikes, these losses can therefore favour different forecast levels. 3.2 Comparison and Forecast-Level Alignment We compare all five models under all seven losses with fixed settings. The primary comparison retains native predictors, so model choice includes differences in inputs and history length; shared inputs provide a robustness check. Unequal numbers of losses and models make their ranges incomparable. We instead average absolute differences over all 21 loss pairs and 10 model pairs. In block b, let qom(b)q_om(b) be log mean test QLIKE for loss o and model m. Let q¯o(b) q_o(b) average this score across models and q¯m(b) q_m(b) across losses. We define ΔL(b)=121∑o<o′|q¯o(b)−q¯o′(b)|,ΔM(b)=110∑m<m′|q¯m(b)−q¯m′(b)|. _L(b)= 121 _o<o | q_o(b)- q_o (b) |, _M(b)= 110 _m<m | q_m(b)- q_m (b) |. The median of ΔL(b)/ΔM(b) _L(b)/ _M(b) over 25 blocks summarizes marginal score differences; values above one indicate greater differences across losses. This descriptive comparison does not separate loss-model interactions. To isolate forecast level, we estimate one validation constant c per loss-model fit and apply it to test forecasts: c=1|V|∑t∈Vhtft,ftaligned=cft,c= 1|V| _t∈ V h_tf_t, f^aligned_t=cf_t, (1) where V contains the validation dates. This c minimizes validation QLIKE over constant rescalings; using it on every date changes forecast level while preserving relative movements. For two losses within the same model and block, let dt=logfo,t−logfo′,td_t= f_o,t- f_o ,t with test average d¯ d. Subtracting d¯ d removes a constant multiplicative offset and leaves time-varying differences. We define LevelShare=1−∑t(dt−d¯)2∑tdt2.LevelShare=1- _t(d_t- d)^2 _td_t^2. Values near one indicate mainly level differences; values near zero indicate time-varying differences. LevelShare uses test forecasts only and does not determine the validation-based alignment constants. 4 Experimental Setup This section presents the data, models, training and evaluation. Chronological data and fixed non-loss settings help attribute differences to the training loss rather than tuning. Data and target. We use daily Binance USDT spot OHLCV data for BTC, ETH, BNB, XRP and ADA through 31 May 2026. BTC and ETH begin on 5 October 2017, and the others at their listing dates. Volatility proxies are computed before lagged predictors, so every input for day t+1t+1 is available by day t. The target is next-day unannualised Garman-Klass variance [5]; Parkinson [12] and Rogers-Satchell [15] test measurement robustness. Forecasting models. We compare HAR, HAR-X, NeuralHAR, DLinear and LightGBM. HAR [3] uses daily, weekly and monthly log variance; HAR-X adds returns, leverage, alternative proxies, volume and lagged BTC variance for altcoins. NeuralHAR and LightGBM [8] share these predictors, whereas DLinear [17] uses 48-day target history. All models predict log variance z z, transformed to f=exp(z^)f= ( z). Training and walk-forward design. HAR and HAR-X are estimated deterministically; the other models use three seeds and validation-based early stopping. Five expanding folds contain 1,013-2,529 training, 253 validation and 379 test days, with disjoint test windows. Scalers use training data, validation controls early stopping and alignment, and all 25 asset-fold blocks receive equal weight. Evaluation and uncertainty. We use QLIKE from Table 1 as the primary score; MSE-log, HMAE and alternative proxies test robustness. Scores are averaged over test dates and seeds within each asset-fold block. Confidence intervals use 1000 stationary bootstrap samples with expected block length 10 and identical resampled dates across forecasts. We use one-day 5% Gaussian VaR, VaRt=−q0.05ftVaR_t=-q_0.05 f_t, as a common diagnostic of forecast-level effects, where q0.05q_0.05 is the standard normal quantile. We compare raw and aligned magnitudes and breach rates; the Kupiec test [11] assesses asset-level coverage. 5 Results The results follow the three research questions. We first compare score differences across losses and models, then determine how much of the loss effect comes from forecast level rather than day-to-day dynamics, and finally examine how these differences affect VaR. 5.1 RQ1: Loss Choice Is a Major Source of Raw Variation To answer RQ1, the ratio from Section 3.2 equals one for equal marginal loss and model variation. Its median is 2.91 (95% CI [2.27, 3.25]), making cross-loss differences about 2.9 times larger. It remains above one with shared predictors (2.73, 95% CI [2.01, 3.44]) and across assets (1.44-4.55). Table 2: Mean test QLIKEQLIKE across 25 blocks; lower is better and bold marks column minima. Stochastic models average three seeds. Loss HAR HAR-X DLinear N-HAR LGBM MSE-logMSE-log 0.772 0.734 0.751 0.718 0.684 Huber-log 0.812 0.771 0.786 0.746 0.703 Pinball 0.836 0.785 0.808 0.763 0.713 QLIKEQLIKE 0.592 0.578 0.594 0.576 0.637 Patton-b 0.597 0.591 0.677 0.586 0.625 HMAE 0.728 0.704 0.646 0.709 0.649 HMSE 1.243 1.178 0.873 1.051 0.659 Table 2 reports every combination of loss and model. QLIKE and Patton generally lead under QLIKE evaluation, whereas HMSE performs poorly for HAR and HAR-X but remains competitive for LightGBM. Rankings are not universal because a score favours its own target. The loss-to-model ratio remains above one under MSE-log (3.71) and HMAE (3.98). Relative to Garman-Klass, loss rankings have Spearman correlations of 1.000 with Rogers-Satchell and 0.964 with Parkinson. Thus, the raw loss effect is robust to both the evaluation score and the volatility proxy. 5.2 RQ2: Forecast Level Explains Most Raw Loss Variation The ordering in Table 2 follows the targets in Table 1: log-error losses produce the lowest levels, QLIKE and Patton target conditional mean variance, and relative-error losses produce higher levels. RQ2 tests whether these level differences drive the raw loss effect. Table 3: Level share and cross-loss log-QLIKE variation by model; alignment uses validation only. Model Level share Raw variation Aligned Reduction HAR 89% 0.322 0.040 88% HAR-X 85% 0.307 0.049 84% NeuralHAR 80% 0.246 0.058 76% DLinear 71% 0.186 0.073 61% LightGBM 56% 0.121 0.050 59% Level share is the fraction of squared log-forecast differences explained by a constant offset. It is 89% for HAR but 56% for LightGBM, which retains more loss-specific daily variation. Alignment reduces cross-loss variation by 59%-88% across models and 77% overall (95% CI [72%, 79%]). The loss-to-model ratio decreases from 2.91 to 0.67 (95% CI [0.56, 0.99]), placing the aligned interval below one. Ratios remain below one under MSE-log, HMAE and for every asset. With shared predictors, the estimate is 0.81 (CI [0.66, 1.22]), so the reversal is statistically clear only in the full grid. This extends recent HAR evidence: QLIKE-training gains can partly reflect movement toward the level favoured by the evaluation loss, not better temporal learning alone [14]. 5.3 RQ3: Forecast-Level Differences Propagate to Value-at-Risk RQ3 moves from forecast comparisons to measured risk. Loss rankings under QLIKE and MSE-log have Spearman correlation −0.200-0.200, so the preferred loss depends on how quality is defined. Table 4 tests whether the level differences from RQ2 also change VaR magnitude and breach rates. Table 4: One-day 5% Gaussian VaR by loss. Raw magnitude is relative to QLIKE; bold marks breach rates closest to 5%. Loss Raw magnitude Raw breach rate Aligned breach rate Pinball -21.3% 6.31% 3.60% Huber-log -20.6% 6.23% 3.61% MSE-logMSE-log -19.1% 5.92% 3.57% QLIKEQLIKE +0.0% 3.68% 3.59% Patton-b +0.8% 3.68% 3.57% HMAE +24.7% 2.19% 3.48% HMSE +68.7% 1.28% 3.55% A calibrated 5% VaR is breached on about 5% of test days; lower rates are more conservative. Before alignment, log-error losses produce smaller VaR and more breaches, while HMSE is most conservative and 68.7% larger than QLIKE. Rates span 1.3%-6.3%. Alignment narrows this range to 3.5%-3.6% and removes 97% of its cross-loss variation. Most raw risk differences thus arise from persistent forecast-level shifts rather than daily signals, consistent with evidence that statistical and financial criteria need not rank methods alike [16]. The Kupiec test rejects correct coverage in 128 of 175 asset-loss-model series before alignment and 109 after it. Aggregate aligned breach rates remain below 5%, so convergence across losses does not imply correct VaR calibration. Alignment optimizes QLIKE rather than tail coverage; remaining discrepancies may reflect the Gaussian assumption, the volatility proxy or forecast error. 6 Discussion This study investigated whether cross-loss differences reflect volatility dynamics or mainly forecast levels. A key finding is that the balance depends on the model: forecast level explains most of the HAR difference, whereas LightGBM shows more loss-specific daily variation. Raw results capture the complete loss effect; aligned results show what remains after validation-based level adjustment, including possible residual test-period level differences. The evidence covers five cryptocurrencies, one-day forecasts, five models and seven losses, so the balance may differ across markets or horizons. Fixed settings exclude loss-specific tuning, while constant alignment cannot correct time-varying bias. VaR results also depend on OHLC-based proxies and a Gaussian return model. Raw and aligned results therefore answer complementary questions. Reporting both indicates whether differences in QLIKE or VaR are primarily associated with forecast level or day-to-day forecast differences. 7 Conclusion and Future Work In this paper, we examined how the training loss shapes cryptocurrency volatility forecasts relative to model choice. We found that much of the large raw loss effect comes from forecast level rather than distinct dynamics, although its importance depends on the model. After validation-based alignment, marginal score differences are larger across model configurations in the full grid, while the shared-input comparison remains inconclusive. We also observed that alignment substantially narrows cross-loss differences in one-day VaR breach rates, showing how the forecast level induced during training carries into a downstream measure of financial risk. The contribution of this work is a joint analysis of loss choice, model choice and forecast level. By reporting raw and aligned results, the study explains why a loss can appear superior under a statistical score without necessarily learning better temporal dynamics. These findings offer a more transparent way to interpret volatility benchmarks and their consequences for risk estimates. Future work can extend the analysis to other markets and horizons, loss-specific tuning and non-Gaussian risk models. References [1] Branco, R.R., Rubesam, A., Zevallos, M.: Forecasting realized volatility: does anything beat linear models? Journal of Empirical Finance 78, 101524 (2024) [2] Christensen, K., Siggaard, M., Veliyev, B.: A machine learning approach to volatility forecasting. Journal of Financial Econometrics 21(5), 1680-1727 (2023) [3] Corsi, F.: A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7(2), 174-196 (2009) [4] Dudek, G., Fiszeder, P., Kobus, P., Orzeszko, W.: Forecasting cryptocurrencies volatility using statistical and machine learning methods: a comparative study. Applied Soft Computing 151, 111132 (2024) [5] Garman, M.B., Klass, M.J.: On the estimation of security price volatilities from historical data. The Journal of Business 53(1), 67-78 (1980) [6] Gneiting, T.: Making and evaluating point forecasts. Journal of the American Statistical Association 106(494), 746-762 (2011) [7] Hansen, P.R., Lunde, A.: Consistent ranking of volatility models. Journal of Econometrics 131(1-2), 97-121 (2006) [8] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.-Y.: LightGBM: a highly efficient gradient boosting decision tree. In: Advances in Neural Information Processing Systems, vol. 30, p. 3146-3154 (2017) [9] Kwiatkowski, J., Chudziak, J.A.: Comparing transformer models for stock selection in quantitative trading. In: Modeling Decisions for Artificial Intelligence (MDAI 2025). LNCS, vol. 15957, p. 239–250. Springer, Cham (2026). [10] Kwiatkowski, J., Chudziak, J.A.: On evaluating loss functions for stock ranking: An empirical analysis with transformer model. In: Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 4910-4914 (2025) [11] Kupiec, P.H.: Techniques for verifying the accuracy of risk measurement models. The Journal of Derivatives 3(2), 73-84 (1995) [12] Parkinson, M.: The extreme value method for estimating the variance of the rate of return. The Journal of Business 53(1), 61-65 (1980) [13] Patton, A.J.: Volatility forecast comparison using imperfect volatility proxies. Journal of Econometrics 160(1), 246-256 (2011) [14] Puke, M., Schweikert, K.: Coherent forecasting of realized volatility. Journal of Forecasting 45(4), 1714-1729 (2026) [15] Rogers, L.C.G., Satchell, S.E.: Estimating variance from high, low and closing prices. The Annals of Applied Probability 1(4), 504-512 (1991) [16] Tokajuk, A., Chudziak, J.A.: Partial multivariate transformer as a tool for cryptocurrencies time series prediction. In: 2025 IEEE 37th International Conference on Tools with Artificial Intelligence (ICTAI), p. 1007-1011 (2025) [17] Zeng, A., Chen, M., Zhang, L., Xu, Q.: Are transformers effective for time series forecasting? In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, p. 11121-11128 (2023)