Paper deep dive

Pseudo-Label-Triggered Retraining from Forecast Errors for Online Time Series Forecasting

Yeryeong Kwak, Yoo-Min Jung, Jonghun Park

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 65

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Real-world time series forecasting systems operate under non-stationary data streams, where forecasting performance may degrade over time. Although retraining can recover the performance, it incurs non-trivial computational and operational costs. Under limited deployment resources, the key challenge is therefore not only how to retrain but also when to retrain. While existing retraining policies often rely on indirect indicators such as drift alarms or model staleness, we instead use realized forecast errors as direct deployment feedback. In this paper, we propose PILOT (Pseudo-label-Informed Learned Online Trigger), an online retraining framework that learns when to retrain from forecast-error dynamics. Since ground-truth retraining labels are unavailable, PILOT constructs a pseudo-label from future increases in forecast error and trains a lightweight scorer to predict it from observed error states. At deployment, PILOT uses only completed forecast errors and serves as a plug-in module for arbitrary forecasting backbones without architectural modification. We evaluate PILOT under standard multivariate forecasting settings across eight benchmarks with three representative backbones---DLinear, iTransformer, and TimesNet. Across all three backbones, PILOT achieves state-of-the-art average-rank performance among retraining policies while maintaining a favorable performance--efficiency trade-off.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

61,757 characters extracted from source content.

Expand or collapse full text

Pseudo-Label-Triggered Retraining from Forecast Errors for Online Time Series Forecasting Conference: preprint; ; preprintCCS: Computing methodologies Online learning settingsCCS: Applied computing ForecastingCCS: Mathematics of computing Time series analysis Yeryeong Kwak Note: Equal contribution Note: Corresponding author email: yeroong9698@snu.ac.kr Affiliation: Seoul National University, Seoul, Republic of Korea , Yoo-Min Jung email: pamela7384@gmail.com Affiliation: Seoul National University, Seoul, Republic of Korea and Jonghun Park email: jonghun@snu.ac.kr Affiliation: Seoul National University, Seoul, Republic of Korea 2026 Abstract. Real-world time series forecasting systems operate under non-stationary data streams, where forecasting performance may degrade over time. Although retraining can recover the performance, it incurs non-trivial computational and operational costs. Under limited deployment resources, the key challenge is therefore not only how to retrain but also when to retrain. While existing retraining policies often rely on indirect indicators such as drift alarms or model staleness, we instead use realized forecast errors as direct deployment feedback. In this paper, we propose PILOT (Pseudo-label-Informed Learned Online Trigger), an online retraining framework that learns when to retrain from forecast-error dynamics. Since ground-truth retraining labels are unavailable, PILOT constructs a pseudo-label from future increases in forecast error and trains a lightweight scorer to predict it from observed error states. At deployment, PILOT uses only completed forecast errors and serves as a plug-in module for arbitrary forecasting backbones without architectural modification. We evaluate PILOT under standard multivariate forecasting settings across eight benchmarks with three representative backbones—DLinear, iTransformer, and TimesNet. Across all three backbones, PILOT achieves state-of-the-art average-rank performance among retraining policies while maintaining a favorable performance–efficiency trade-off. Keywords: pseudo-labeling, model retraining, time series forecasting, online learning, concept drift †c-license: by 1. Introduction Real-world time series forecasting models operate in online settings (Hoi et al., 2021) and face distribution shifts (Liu et al., 2023; Lu et al., 2019) that can progressively degrade forecasting performance (Zhang et al., 2024). Retraining on recent observations can restore performance (Liu et al., 2023; Mahadevan and Mathioudakis, 2024; Zanotti, 2025). However, retraining incurs non-trivial computational overhead (Mahadevan and Mathioudakis, 2024; Zanotti, 2025; Hoffman et al., 2024), may disrupt deployment pipelines (Derakhshan et al., 2019), and does not necessarily improve forecasting performance (Regol et al., 2025; Zanotti, 2025). Under resource constraints, retraining should therefore be invoked selectively rather than routinely (Zliobaite et al., 2015; Mahadevan and Mathioudakis, 2024; Regol et al., 2025; Hoffman et al., 2024). The key question is therefore not only how to retrain, but also when to retrain. Existing policies trigger updates based on predicted future performance (Regol et al., 2025), model staleness or expected return (Mahadevan and Mathioudakis, 2024; Zliobaite et al., 2015), or statistical changes in the data stream (Bifet and Gavaldà, 2007; Raab et al., 2020). However, these criteria do not directly use realized forecasting performance as deployment feedback. In contrast, forecasting systems naturally produce direct deployment feedback once target observations become available: forecast errors. Prior work has primarily used them for monitoring (Trigg, 1964; Grundy et al., 2026). We instead use recent error dynamics to anticipate performance degradation and selectively allocate retraining computation. In this paper, we propose PILOT (Pseudo-label-Informed Learned Online Trigger), a lightweight framework that learns retraining decisions from forecast-error dynamics. During offline training, PILOT constructs a pseudo-label from future forecast-error increase relative to a recent baseline and trains a compact scorer to predict this signal from recent error states. This supervision requires no counterfactual retraining simulation. At deployment, the scorer triggers retraining using only realized forecast errors. Since it relies only on forecast errors, PILOT can be attached to different forecasting backbones without architectural modification. Our contributions are as follows: • We formulate retraining timing in online forecasting as a decision problem driven by realized forecast-error dynamics. • To the best of our knowledge, PILOT is the first approach in online time series forecasting to use pseudo-labeling for retraining decisions, enabling the decision module to be trained without ground-truth retraining labels. • Unlike representative retraining studies using synthetic or custom-designed streams (Mahadevan and Mathioudakis, 2024; Regol et al., 2025), we evaluate PILOT on eight widely adopted multivariate forecasting benchmarks. • Across three representative backbones spanning linear, Transformer, and convolutional architectures (Zeng et al., 2023; Liu et al., 2024; Wu et al., 2023), PILOT achieves state-of-the-art performance among retraining policies with a favorable performance–efficiency trade-off. (a) Training Phase (b) Inference Phase Figure 1. Overall framework of PILOT (Pseudo-label-Informed Learned Online Trigger). Gray snowflake modules are frozen while light-yellow modules are updatable in the corresponding phase. A two-panel schematic of the proposed retraining method. In both panels, modules shown in gray with a snowflake icon are frozen, while modules shown in light yellow have their parameters updated in the corresponding phase: the scorer is trained in panel (a), and the backbone is retrained in panel (b). The top panel illustrates the offline training phase: a training input window spanning from time $t-L$ to $t+H$ is fed into the frozen forecasting backbone $ _φ$, which produces a forecast $ _t$. The residual $ _t = _t - _t$ between the ground-truth future segment $ _t$ and the forecast is summarized into a five-dimensional forecast-error state vector $ _t$ consisting of the average signed residual, the mean absolute error, the mean squared error, the rolling mean absolute error, and the rolling root mean squared error. The mean squared error component of $ _t$ is passed through a pseudo-label construction branch that produces a pseudo-label $g_t$. The standardized error state sequence $ _t$ over the recent $N$ steps is passed to the trainable retraining scorer, which aggregates the sequence via average pooling and max pooling and concatenates the result with the current-step vector before feeding it into a multi-layer perceptron (MLP) that outputs a scalar retraining score $s_t$, supervised by $g_t$ via the Huber loss. The bottom panel illustrates the online inference phase: the forecasting backbone $ _φ$ processes a test input window and produces a forecast whose residual $ _t$ forms the current error-state $ _t$, which is passed to the frozen retraining scorer to yield a scalar score $s_t$. The score is then online-calibrated and compared against a threshold $θ$ to produce a binary retraining action $a_t = [ s_t > θ]$. A backbone update equation at the top indicates that when $a_t = 1$, the backbone parameters are updated on a recent training buffer $B_t$ via the retraining operator $U$. A timeline at the bottom illustrates the triggering schedule: an initial warm-up period of length $N_wu$, during which triggering is disabled, is followed by an active triggering region, then a cooldown period of length $N_cd$ after the most recent retraining time $t_last$, during which triggering is again disabled, and finally another active triggering region. 2. Related Work Selective retraining policies. Despite its practical importance, relatively few studies have directly addressed when models should be retrained. Zliobaite et al. (Zliobaite et al., 2015) formulated adaptation as a trade-off between update cost and expected performance gain. CARA (Mahadevan and Mathioudakis, 2024) triggers retraining when model staleness cost exceeds retraining cost, whereas UPF (Regol et al., 2025) predicts future model performance and retrains when predicted quality deteriorates. These methods infer retraining need from cost, staleness, or predicted performance; PILOT instead learns from realized forecast-error dynamics. Drift and forecast-error monitoring. Drift detectors can also serve as update triggers: ADWIN (Bifet and Gavaldà, 2007) monitors changes with adaptive windows, while KSWIN (Raab et al., 2020) applies a Kolmogorov–Smirnov test to recent stream windows. However, detected distribution change need not imply that retraining is beneficial (Regol et al., 2025). Separately, forecast errors have long served as monitoring signals (Trigg, 1964), with recent work detecting model inadequacy directly from forecast-error changes (Grundy et al., 2026) and studying large forecast errors under structural change (Castle et al., 2026). PILOT connects these lines by using future performance degradation as pseudo-label supervision to learn retraining decisions from recent error dynamics. 3. Proposed Method Figure 1 summarizes the offline training and online inference phases of PILOT. During offline scorer training, the pretrained forecasting backbone is frozen, and its completed forecast errors are used to construct error states and a pseudo-label for training a lightweight scorer. At deployment, the scorer is frozen and uses only completed forecast errors to decide when the backbone should be retrained. 3.1. Problem Formulation We study online multivariate time series forecasting under non-stationary data streams. Let ℱϕtF_ _t denote the forecasting backbone at decision step t, with parameters ϕt _t. Given a lookback window t∈ℝL×D X_t∈R^L× D, it produces an H-step forecast (1) ^t=ℱϕt(t)∈ℝH×D, Y_t=F_ _t( X_t)∈R^H× D, where L, H, and D denote the lookback length, prediction horizon, and number of variates, respectively, and t∈ℝH×D Y_t∈R^H× D denotes the corresponding future segment. Once t Y_t is fully observed, the system chooses a binary retraining action at∈0,1a_t∈\0,1\ using only forecast errors whose full horizons are available. Let ℬtB_t denote a buffer of recent observations and U the retraining operator. The backbone parameters evolve as (2) ϕt+1=at(ϕt,ℬt)+(1−at)ϕt, _t+1=a_t\,U( _t,B_t)+(1-a_t) _t, where at=1a_t=1 triggers retraining and at=0a_t=0 keeps the current backbone unchanged. Let ℰtE_t denote the history of completed forecast errors observed up to time t. We base the retraining decision on ℰtE_t and seek to learn a decision function π that maps this error history to a retraining action, (3) at=π⁡(ℰt).a_t=π(E_t). PILOT realizes π by encoding ℰtE_t into a compact forecast-error state representation and learning its relation to future degradation. 3.2. Forecast-Error State Representation For each completed forecast, we compute the residual matrix (4) t=t−^t∈ℝH×D, R_t= Y_t- Y_t∈R^H× D, and summarize it over the prediction horizon and variates as (5) rt=Avg⁡(t),etMAE=Avg⁡(|t|),etMSE=Avg⁡(t2),r_t=Avg( R_t), e_t^MAE=Avg(| R_t|), e_t^MSE=Avg( R_t^2), where Avg⁡(⋅)Avg(·) denotes the mean over all entries in ℝH×DR^H× D. Here, rtr_t captures the average signed residual, while etMAEe_t^MAE and etMSEe_t^MSE measure the mean absolute error (MAE) and mean squared error (MSE), respectively. To provide a recent baseline, we additionally compute (6) htMAE=1K∑i=t−K+1teiMAE,htRMSE=1K∑i=t−K+1teiMSE,h_t^MAE= 1K _i=t-K+1^te_i^MAE, h_t^RMSE= 1K _i=t-K+1^te_i^MSE, and form the per-step state (7) t=[rt,etMAE,etMSE,htMAE,htRMSE]⊤∈ℝ5. c_t= [r_t,e_t^MAE,e_t^MSE,h_t^MAE,h_t^RMSE ] ∈R^5. The historical terms htMAEh_t^MAE and htRMSEh_t^RMSE let the scorer interpret current errors relative to a recent baseline rather than in isolation. Since these feature channels can have different scales across datasets and forecasting backbones, we standardize each channel using statistics computed from the scorer-training split. Let ∈ℝ5 μ_ c∈R^5 and ∈ℝ5 σ_ c∈R^5 denote the feature-wise mean and standard deviation, respectively. The standardized per-step feature vector is defined as (8) ~t=t−+ϵ, c_t= c_t- μ_ c σ_ c+ε, where ϵε is a small constant for numerical stability. Finally, we stack the most recent N standardized states: (9) t=[~t−N+1;~t−N+2;…;~t]∈ℝN×5. C_t= [ c_t-N+1;\, c_t-N+2;\,…\,;\, c_t ]∈R^N× 5. This tensor serves as the input to the retraining scorer described in Section 3.4. By incorporating the most recent N states, it provides richer temporal context than a single-step snapshot. 3.3. Pseudo-Label Construction Since ground-truth labels indicating retraining necessity are unavailable in practice, we construct an offline pseudo-label gtg_t on the scorer-training split scorerD_scorer. We use the change in future mean MSE relative to a recent baseline as the supervision signal, which can be constructed directly from the frozen backbone’s error trajectory without counterfactual retraining simulation. Specifically, mean degradation measures the change in future average MSE relative to a recent baseline, with positive values corresponding to degradation. This perspective is related to model staleness, which measures the performance cost of keeping an outdated model (Mahadevan and Mathioudakis, 2024). Let e¯[a,b)MSE e^MSE_[a,b) denote the average MSE over the time interval [a,b)[a,b): (10) e¯[a,b)MSE=1b−a∑i=ab−1eiMSE. e^MSE_[a,b)= 1b-a _i=a^b-1e_i^MSE. For a current window of length WcW_c and a future window of length WfW_f, we define (11) gt=e¯MSE[t,t+Wf)−e¯MSE[t−Wc,t).g_t= e^MSE_[t,t+W_f)- e^MSE_[t-W_c,t). Thus, gt>0g_t>0 indicates that the frozen backbone’s future average error exceeds its recent baseline, providing supervision for learning impending degradation. Following the principle of winsorization (Hastings Jr et al., 1947), we clip gtg_t to [−0.5,2.0][-0.5,2.0] before scorer training, affecting fewer than 1%1\% of targets. We further examine the extensibility of PILOT to alternative pseudo-labels in Section 4.5 and Appendix B. 3.4. Lightweight Retraining Scorer Given t C_t and gtg_t, we train a compact scorer that outputs a retraining score st∈ℝs_t∈R at each time step. We first aggregate the recent error history by feature-wise average pooling, max pooling, and the latest state: (12) t=[AvgPool⁡(t),MaxPool⁡(t),~t]∈ℝ15. p_t= [AvgPool( C_t),\,MaxPool( C_t),\, c_t ]∈R^15. Each component captures a different aspect of the recent error history: average pooling captures the typical recent behavior, max pooling the peak severity, and ~t c_t the current state. We then map t p_t to the scalar score sts_t using a single-hidden-layer multilayer perceptron (MLP) with ReLU activation and dropout: (13) st=MLP⁡(t).s_t=MLP( p_t). We train the scorer with the Huber loss (Huber, 1964), (14) ℒHuber=1|scorer|∑t∈scorerℓδHuber(st,gt),L_Huber= 1|D_scorer| _t _scorer _δ^Huber(s_t,g_t), which is robust to occasional large pseudo-label deviations. To mitigate the underrepresentation of positive pseudo-labels, we oversample scorer-training samples with gt>0g_t>0 (He and Garcia, 2009). The trained scorer therefore predicts future degradation from error states available at decision time. 3.5. Online Triggering At test time, the frozen scorer outputs a raw score sts_t from each state tensor t C_t. We maintain its running mean μs,t _s,t and standard deviation σs,t _s,t using Welford’s online update rule (Welford, 1962). Triggering is disabled until NwuN_wu score observations have been accumulated; afterward, we calibrate (15) s~t=st−μs,tσs,t+ϵ. s_t= s_t- _s,t _s,t+ε. This standardization follows common practice in online monitoring of data streams (Gama et al., 2014). We trigger retraining when the calibrated score exceeds θ, subject to a cooldown of NcdN_cd steps after the previous retraining time tlastt_last. The final retraining action is therefore defined as (16) at=[nt≥Nwu]⋅[s~t>θ]⋅[t−tlast≥Ncd],a_t=[n_t≥ N_wu]·[ s_t>θ]·[t-t_last≥ N_cd], where [⋅][·] denotes the indicator function and ntn_t is the number of accumulated scores. When at=1a_t=1, the backbone is warm-start retrained on ℬtB_t according to Section 3.1. Running score statistics continue to update across retraining events; pseudo-labels are never used at deployment. 4. Experiments 4.1. Experimental Setup Datasets. In contrast to prior retraining studies that evaluated synthetic or custom-designed streams (Mahadevan and Mathioudakis, 2024; Regol et al., 2025), we used eight widely used multivariate forecasting benchmarks, providing a realistic testbed for retraining decisions on naturally evolving real-world time series: ETTh1 and ETTh2 (hourly, 7 variates), ETTm1 and ETTm2 (15-minute, 7 variates), Electricity (hourly, 321 variates), Exchange (daily, 8 variates), Traffic (hourly, 862 variates), and Weather (10-minute, 21 variates) (Trindade, 2015; Lai et al., 2018; Zhou et al., 2021; Wu et al., 2021). Following the standard split convention adopted in prior long-term forecasting work (Zhou et al., 2021; Wu et al., 2021; Wu et al., 2023; Liu et al., 2024), the ETT datasets were split chronologically into training, validation, and test sets at a 60:20:20 ratio, while the other datasets were split at a 70:10:20 ratio. Since our method requires an additional split for training the retraining scorer, we further partitioned the standard validation set into two equal halves: one was used as the scorer-training split for constructing error states and training the retraining scorer, and the other was retained as a validation split. This yielded four chronologically ordered splits—backbone training, scorer training, validation, and test—with ratios of 60:10:10:20 for ETT datasets and 70:5:5:20 for the others. The backbone was pretrained only on the first split and shared throughout all methods, so differences in performance were attributable solely to each method’s retraining policy. Backbones. We evaluated PILOT with three forecasting backbones that span different architectural families: DLinear (Zeng et al., 2023) from the linear family; iTransformer (Liu et al., 2024) from the Transformer family; and TimesNet (Wu et al., 2023) from the convolutional family. Baselines. We compared against six baselines spanning no retraining, scheduled retraining, drift-based triggers, and cost- or prediction-based policies. Method-specific hyperparameters were selected on the standard validation split from the candidate sets described below, and the selected configuration was evaluated on the test split. • No Retrain never updates the backbone after initial training. • Periodic triggers retraining at a fixed interval, with the interval tuned from 50,100,200\50,100,200\ steps. • ADWIN (Bifet and Gavaldà, 2007) is an adaptive windowing drift detector that automatically adjusts its window size based on observed changes, applied to the per-step MSE summary etMSEe^MSE_t defined in Section 3.2. We tuned the significance level δ∈0.002,0.01,0.05δ∈\0.002,0.01,0.05\. • KSWIN (Raab et al., 2020) performs a Kolmogorov–Smirnov test over a sliding window of recent etMSEe^MSE_t values to detect distributional change. We tuned the significance level α∈0.001,0.01,0.05α∈\0.001,0.01,0.05\. • CARA (Mahadevan and Mathioudakis, 2024) is a cost-aware retraining policy that triggers an update when estimated model staleness exceeds the retraining cost. We tuned both staleness variants (TT and CC), with the threshold fixed at τ=0.05τ=0.05. • UPF (Regol et al., 2025) formulates retraining as a future-performance prediction problem using an ElasticNet-based forecaster. We adapted UPF to forecasting by replacing the original predicted quality signal with future MSE under the Gaussian-likelihood setting described in the original paper’s Appendix. We tuned the retraining cost penalty α∈0.05,0.5,1.0α∈\0.05,0.5,1.0\. Evaluation metrics. We report test MSE, the number of retraining events (#RT), average rank, and PILOT’s wins/losses against each baseline. Lower MSE and rank are better, while lower #RT indicates fewer model updates; MSE ties in win/loss counts are broken by fewer retraining events. For each backbone, we also report one-sided Wilcoxon signed-rank tests (Wilcoxon, 1945) over the eight datasets and three shared random seeds to assess whether PILOT yields lower MSE than each baseline. Implementation details. Across all methods, we used a lookback length of L=96L=96 and a prediction horizon of H=96H=96. The cooldown period was set to Ncd=100N_cd=100 hours and the retraining buffer length was 1000 steps; the same warm-start retraining protocol was used across all methods. Hour-based hyperparameters were converted to dataset-specific step counts using each sampling interval, preserving a consistent temporal scope across datasets. Unless otherwise specified, all results were aggregated over three random seeds. For PILOT, all hyperparameters except the trigger threshold θ were fixed across datasets and backbones. Although θ remained available for validation, it was fixed to 1.01.0 for all results reported in this paper to avoid dataset- or backbone-specific tuning. The error-state vector used a rolling history window of K=20K=20 steps, and the state tensor stacked the most recent N=24N=24 standardized vectors. The degradation labels used current and future windows of Wc=Wf=48W_c=W_f=48 hours. Triggering was disabled during an initial warm-up period of Nwu=50N_wu=50 steps, during which online score statistics were accumulated. The scorer was a single-hidden-layer MLP with a hidden dimension of 6464 and a dropout rate of 0.10.1. The source code is available at https://anonymous.4open.science/r/PILOT-D44F. 4.2. Forecasting Performance Table 1. Forecasting performance across eight datasets and three forecasting backbones. Each cell reports MSE with its standard deviation, followed by the average number of retraining events (#RT) in parentheses; No Retrain reports MSE only since its #RT is always zero. Bold: best, underline: second-best. Backbone Dataset No Retrain Periodic ADWIN ((2007)) KSWIN ((2020)) CARA ((2024)) UPF ((2025)) PILOT (ours) DLinear ETTh1 0.4572± 0.0005 0.4514± 0.0003 (16.0) 0.4533± 0.0002 (10.7) 0.4506± 0.0001 (32.0) 0.4562± 0.0013 (10.3) 0.4536± 0.0007 (15.3) 0.4501± 0.0003 (17.3) ETTh2 0.2343± 0.0085 0.2184± 0.0007 (16.0) 0.2185± 0.0013 (6.7) 0.2188± 0.0009 (31.0) 0.2343± 0.0085 (0.0) 0.2193± 0.0008 (24.7) 0.2185± 0.0009 (14.0) ETTm1 0.3651± 0.0027 0.3606± 0.0002 (35.0) 0.3586± 0.0003 (23.3) 0.3618± 0.0009 (34.0) 0.3594± 0.0006 (23.7) 0.3624± 0.0003 (24.0) 0.3606± 0.0007 (20.7) ETTm2 0.1756± 0.0106 0.1603± 0.0011 (35.0) 0.1625± 0.0006 (17.3) 0.1607± 0.0009 (34.0) 0.1629± 0.0011 (15.3) 0.1610± 0.0039 (13.3) 0.1598± 0.0031 (22.3) ECL 0.1963± 0.0000 0.1957± 0.0000 (25.0) 0.1963± 0.0000 (6.0) 0.1963± 0.0000 (50.0) 0.1962± 0.0001 (9.0) 0.1959± 0.0000 (3.0) 0.1962± 0.0002 (31.7) Exchange 0.0888± 0.0000 0.0785± 0.0003 (7.0) 0.0844± 0.0000 (1.0) 0.0785± 0.0004 (13.0) 0.0782± 0.0001 (4.0) 0.0888± 0.0000 (0.0) 0.0776± 0.0006 (23.0) Weather 0.1911± 0.0000 0.1827± 0.0001 (18.0) 0.1849± 0.0001 (12.0) 0.1828± 0.0002 (17.7) 0.2012± 0.0009 (2.0) 0.1831± 0.0001 (17.0) 0.1819± 0.0005 (14.0) Traffic 0.6546± 0.0009 0.6416± 0.0002 (17.0) 0.6424± 0.0009 (7.7) 0.6420± 0.0002 (32.0) 0.6425± 0.0002 (20.7) 0.6413± 0.0003 (19.0) 0.6410± 0.0005 (15.7) avg. rank 6.33 2.73 4.13 3.81 4.85 4.15 2.00 # of wins/losses 8/0 6/2 6/2 8/0 6/2 7/1 – Wilcoxon test (p) <0.001<0.001 0.008 0.002 <0.001<0.001 <0.001<0.001 <0.001<0.001 – iTransformer ETTh1 0.4672± 0.0000 0.4616± 0.0121 (16.0) 0.4565± 0.0058 (9.3) 0.4803± 0.0222 (30.3) 0.4733± 0.0159 (12.0) 0.4656± 0.0091 (15.3) 0.4459± 0.0008 (19.7) ETTh2 0.2468± 0.0014 0.2449± 0.0015 (16.0) 0.2441± 0.0026 (8.7) 0.2511± 0.0010 (30.7) 0.2468± 0.0014 (0.0) 0.2484± 0.0015 (26.0) 0.2415± 0.0017 (11.3) ETTm1 0.4852± 0.0260 0.4007± 0.0057 (35.0) 0.4023± 0.0029 (25.7) 0.4086± 0.0089 (33.3) 0.4033± 0.0024 (27.0) 0.4086± 0.0036 (25.0) 0.3786± 0.0034 (17.0) ETTm2 0.1750± 0.0027 0.1693± 0.0022 (35.0) 0.1715± 0.0015 (18.0) 0.1729± 0.0035 (33.3) 0.1823± 0.0062 (5.0) 0.1757± 0.0038 (16.0) 0.1680± 0.0064 (15.3) ECL 0.1661± 0.0027 0.1646± 0.0003 (25.0) 0.1669± 0.0012 (6.3) 0.1656± 0.0003 (49.7) 0.1664± 0.0008 (3.7) 0.1658± 0.0004 (5.3) 0.1630± 0.0006 (32.3) Exchange 0.0939± 0.0000 0.1000± 0.0011 (7.0) 0.0923± 0.0005 (2.0) 0.1085± 0.0017 (13.0) 0.1039± 0.0014 (8.3) 0.0939± 0.0000 (1.0) 0.1125± 0.0123 (20.7) Weather 0.1811± 0.0000 0.1803± 0.0001 (18.0) 0.1800± 0.0006 (14.0) 0.1816± 0.0006 (17.3) 0.1770± 0.0004 (2.0) 0.1792± 0.0005 (17.0) 0.1788± 0.0012 (13.3) Traffic 0.4754± 0.0000 0.4766± 0.0002 (17.0) 0.4762± 0.0001 (9.0) 0.4753± 0.0004 (32.0) 0.4773± 0.0001 (12.0) 0.4816± 0.0003 (7.0) 0.4783± 0.0036 (13.7) avg. rank 4.35 3.29 3.21 5.13 4.73 4.92 2.38 # of wins/losses 6/2 6/2 6/2 6/2 5/3 7/1 – Wilcoxon test (p) 0.008 0.017 0.047 0.001 0.003 0.004 – TimesNet ETTh1 0.6762± 0.0041 0.6287± 0.0098 (16.0) 0.6269± 0.0035 (12.3) 0.6619± 0.0164 (32.0) 0.6396± 0.0046 (10.7) 0.6194± 0.0043 (15.0) 0.6406± 0.0091 (18.0) ETTh2 0.3128± 0.0135 0.3327± 0.0093 (16.0) 0.3170± 0.0204 (8.7) 0.3452± 0.0082 (32.0) 0.3128± 0.0135 (0.0) 0.3432± 0.0104 (30.3) 0.3270± 0.0174 (15.7) ETTm1 0.5381± 0.0000 0.3980± 0.0004 (35.0) 0.4010± 0.0046 (25.7) 0.3991± 0.0010 (34.0) 0.4194± 0.0058 (25.3) 0.3977± 0.0013 (30.0) 0.3945± 0.0023 (20.0) ETTm2 0.1977± 0.0118 0.1808± 0.0025 (35.0) 0.1840± 0.0042 (20.0) 0.1810± 0.0038 (34.0) 0.1927± 0.0074 (5.0) 0.1850± 0.0013 (16.7) 0.1817± 0.0015 (17.7) ECL 0.1731± 0.0000 0.1619± 0.0002 (25.0) 0.1642± 0.0009 (5.3) 0.1626± 0.0004 (49.7) 0.1688± 0.0001 (4.0) 0.1652± 0.0010 (4.7) 0.1618± 0.0003 (24.0) Exchange 0.1828± 0.0001 0.1680± 0.0002 (7.0) 0.1702± 0.0023 (4.3) 0.1656± 0.0002 (13.0) 0.1659± 0.0001 (5.0) 0.1828± 0.0001 (0.0) 0.1659± 0.0043 (9.0) Weather 0.2011± 0.0057 0.1851± 0.0010 (18.0) 0.1831± 0.0013 (12.0) 0.1856± 0.0030 (17.3) 0.2009± 0.0043 (2.0) 0.1839± 0.0003 (17.0) 0.1827± 0.0051 (13.7) Traffic 0.5996± 0.0009 0.4630± 0.0008 (17.0) 0.4808± 0.0054 (11.3) 0.4577± 0.0013 (32.7) 0.4894± 0.0061 (8.0) 0.4607± 0.0010 (12.7) 0.4714± 0.0126 (17.0) avg. rank 6.17 3.00 3.71 3.58 4.90 3.73 2.92 # of wins/losses 7/1 5/3 6/2 5/3 5/3 6/2 – Wilcoxon test (p) <0.001<0.001 0.366 0.345 0.060 0.006 0.094 – Table 1 reports MSE, #RT, and average rank across eight datasets and three backbones. PILOT achieved the lowest average rank on all three backbones, with more wins than losses against every baseline. The Wilcoxon tests further supported this pattern: PILOT was significantly better (p<0.05p<0.05) in 14 of 18 backbone–baseline comparisons, including every baseline on DLinear and iTransformer, while the remaining four still favored PILOT in win/loss counts. At the individual-dataset level, PILOT obtained the best MSE on five of eight datasets with both DLinear and iTransformer, and on three with TimesNet. The #RT values further showed that these gains did not arise simply from more frequent retraining, as update frequency varied across datasets and baselines. While Periodic and ADWIN remained competitive on certain backbones, neither ranked best across all three. The more specialized CARA and UPF did not consistently surpass these simpler policies, while KSWIN even ranked below No Retrain on iTransformer. Overall, these results indicate that PILOT’s forecast-error-based supervision provides a reliable basis for retraining decisions across diverse forecasting architectures. Three scatter plots compare retraining methods on ETTm2 using DLinear, iTransformer, and TimesNet. The x-axis shows total retraining runtime, the y-axis shows relative MSE improvement over No Retrain, and marker size is proportional to the number of retraining events. Methods toward the upper-left achieve greater forecasting improvement with lower retraining runtime. Figure 2. Retraining performance–efficiency trade-off on ETTm2 across the three backbones. The x-axis shows total retraining runtime including decision overhead, the y-axis shows relative MSE improvement over No Retrain (%), and marker size is proportional to #RT. Methods closer to the upper-left corner achieve greater forecasting improvement at lower retraining cost. Three scatter plots compare retraining methods on ETTm2 using DLinear, iTransformer, and TimesNet. The x-axis shows total retraining runtime, the y-axis shows relative MSE improvement over No Retrain, and marker size is proportional to the number of retraining events. Methods toward the upper-left achieve greater forecasting improvement with lower retraining runtime. 4.3. Performance–Efficiency Trade-off The performance gains of PILOT in Table 1 were not achieved through indiscriminate retraining. Figure 2 further examines the performance–efficiency trade-off on ETTm2, one of the longest online test streams among the eight benchmarks and a setting with diverse retraining frequencies across policies. The figure compares relative improvement over No Retrain against total retraining runtime, with marker size proportional to #RT; detailed measurements are provided in Appendix A. PILOT lies on the performance–retraining-time Pareto frontier across all three backbones. Moreover, total retraining time closely tracked #RT, while the time per retraining event was broadly similar across methods (Appendix A). Thus, differences in cumulative retraining cost were driven primarily by how often each policy retrained. By combining competitive forecasting improvement with moderate retraining cost, PILOT consequently achieved a favorable performance–compute trade-off. The baselines display two recurring trade-off patterns. Periodic and KSWIN sit toward the right of each plot, paying high retraining cost for their performance gains. In contrast, CARA clusters in the lower-left region, keeping retraining cost low but yielding limited improvement; on iTransformer, CARA even underperformed No Retrain. The remaining policies occupy intermediate positions but do not reach the upper-left region across all three backbones. Figure 2 therefore confirms PILOT’s favorable operational efficiency. 4.4. Retraining Timing and Robustness Beyond aggregate performance and cost, we next isolate the role of retraining timing, examine scorer stability under successive updates, and analyze the resulting trigger behavior. Table 2. Effect of retraining timing on DLinear. Uniform performs the same number of retraining events as PILOT on each dataset, spacing them uniformly over the test stream. Bold: best. Method ETTh1 ETTh2 ETTm1 ETTm2 ECL Exch. Weat. Traff. No RT 0.4572 0.2343 0.3651 0.1756 0.1963 0.0888 0.1911 0.6546 Uniform 0.5619 0.2395 0.3576 0.1782 0.2041 0.0879 0.1853 0.6416 PILOT 0.4501 0.2185 0.3606 0.1598 0.1962 0.0776 0.1819 0.6410 Matched-budget timing analysis Sections 4.2 and 4.3 show that forecasting accuracy and retraining frequency alone do not reveal whether gains arise from well-timed updates. To isolate retraining timing from frequency, we compared PILOT with Uniform, which performs the same number of updates as PILOT but distributes them uniformly over the test stream. We conducted this analysis with DLinear, where Periodic achieved its best average rank among the three backbones in Table 1. As shown in Table 2, PILOT outperformed Uniform on seven of the eight datasets, except on ETTm1. Uniform also underperformed No Retrain on four datasets, showing that using the same update budget does not guarantee improvement. Together, these results provide empirical support for the retraining timing learned from mean-degradation supervision. Figure 3. Pearson correlation between sts_t and future degradation gtonlineg_t^online across the test stream. Lines show means across datasets and seeds; shading denotes the interquartile range. Closed-loop robustness The scorer is trained offline with a frozen backbone, whereas online retraining progressively changes the deployed backbone and its error dynamics. To examine whether the learned signal remains informative under this feedback, we divided each test stream into ten chronological segments and, within each segment, measured the Pearson correlation (Pearson, 1895) between sts_t and future degradation gtonlineg_t^online, defined as gtg_t computed with the active checkpoint ϕt _t held fixed over the future window. Figure 3 shows that the correlation remained consistently positive across all three backbones without systematic decline over the test stream. This indicates stable scorer–degradation alignment despite successive backbone updates. Table 3. Trigger quality averaged across the eight datasets and three backbones. Bold: best, underline: second-best. Method Trigger Precision (↑ ) Trigger Need (↑ ) Episode Hit Rate (↑ ) Detection Lag Rank (↓ ) Periodic 0.482 0.091 0.417 3.53 ADWIN 0.518 0.113 0.243 3.73 KSWIN 0.514 0.088 0.574 2.66 CARA 0.588 0.255 0.233 4.61 UPF 0.437 0.052 0.353 4.04 PILOT 0.707 0.356 0.468 2.43 Trigger quality analysis The matched-budget results showed that update frequency alone cannot explain forecasting performance. We therefore evaluated whether each policy’s triggers aligned with realized forecast degradation. Using the No Retrain error stream as a common reference, we defined degradation as an increase in future average error relative to the recent window. The four metrics capture two complementary perspectives: trigger selectivity and degradation-episode coverage. Trigger precision measures the fraction of triggers occurring during degradation, while trigger need measures the average normalized degradation magnitude at triggered steps. Episode hit rate measures the fraction of contiguous degradation episodes containing at least one trigger, whereas detection lag rank measures how quickly a method triggers after the start of a hit episode. Detection lags are converted to per-dataset ranks before aggregation to account for different sampling rates.11 1 Trigger-conditioned metrics omit settings with no triggers; missed episodes count as zero hits, and methods with no hits receive the worst detection lag rank. Table 3 reveals a clear trade-off among the baselines. CARA is highly selective but misses many degradation episodes and reacts late, whereas KSWIN provides broader and faster coverage with lower selectivity. UPF shows the weakest selectivity and only moderate episode coverage, while Periodic and ADWIN occupy intermediate positions. This balanced behavior of Periodic and ADWIN is consistent with their relatively strong forecasting performance in Table 1. PILOT is the only method to rank within the top two on all four metrics: it achieves the best trigger precision, trigger need, and detection lag rank, together with the second-best episode hit rate. Thus, PILOT combines selective triggering with broad and timely coverage of degradation episodes. (a) DLinear on ETTh1 (b) iTransformer on ETTh2 Figure 4. Trigger visualization of each retraining method for (4(a)) DLinear on ETTh1 and (4(b)) iTransformer on ETTh2.Two three-panel figures illustrate retraining behavior over time. The first set shows ETTh1 with DLinear: the rolling MSE fluctuates with several distinct surges between steps 500 and 2300; the error trend panel shows positive values during these surges, with shaded regions marking intervals of positive forecast degradation; PILOT concentrates triggers within shaded regions, and it achieves the lowest MSE in this case. The second set shows ETTh2 with iTransformer: the rolling MSE starts high (~1.4) and gradually decreases, with persistent surges around steps 1400 to 1800; the error trend shows alternating positive and negative regions; PILOT achieves the lowest MSE on this seed while triggering substantially fewer times than others around steps 1600 to 2500. Trigger timing analysis. Figure 4 complements Table 3 with temporal examples for DLinear–ETTh1 and iTransformer–ETTh2. Within each panel, the top row shows the rolling MSE of the No Retrain baseline; the middle row shows the corresponding degradation trend; and the bottom row shows each method’s trigger events for one random seed, with the corresponding test MSE in parentheses. Shaded regions in the middle and bottom rows mark intervals with positive forecast degradation. These intervals provide a reference for assessing how each method’s triggers align with forecast degradation. As shown in Figure 4(a), the ETTh1 rolling MSE exhibits several distinct surges, including repeated bursts around steps 1300–1800 and a sharp spike near 2300. Periodic and KSWIN trigger across both degradation and non-degradation intervals, whereas CARA triggers comparatively sparsely. PILOT concentrates its updates around major degradation intervals and achieves the lowest MSE on this seed. In Figure 4(b), the rolling MSE peaks early (around 1.4 within the first 200 steps) and gradually decreases, with persistent surges around steps 1400–1800. KSWIN and UPF trigger densely over the entire test stream, while CARA never triggers. PILOT remains comparatively sparse after the main degradation interval and again achieves the lowest MSE. Together, these cases visually reinforce the selective yet responsive behavior summarized in Table 3. 4.5. Ablation and Alternative Pseudo-Labels Table 4. Component ablation for PILOT on TimesNet. Parentheses report relative MSE change from PILOT. Dataset PILOT w/o Scorer† w/o History w/o Stacking ETTh1 0.6406 0.6893 (+7.61%+7.61\%) 0.6928 (+8.14%+8.14\%) 0.7004 (+9.34%+9.34\%) ETTh2 0.3270 0.3294 (+0.74%+0.74\%) 0.3320 (+1.54%+1.54\%) 0.3337 (+2.04%+2.04\%) ETTm1 0.3945 0.3973 (+0.71%+0.71\%) 0.4034 (+2.26%+2.26\%) 0.4024 (+2.00%+2.00\%) ETTm2 0.1817 0.1891 (+4.06%+4.06\%) 0.1810 (−0.38%-0.38\%) 0.1859 (+2.29%+2.29\%) ECL 0.1618 0.1708 (+5.55%+5.55\%) 0.1691 (+4.53%+4.53\%) 0.1697 (+4.90%+4.90\%) Exchange 0.1659 0.1624 (−2.14%-2.14\%) 0.1678 (+1.13%+1.13\%) 0.1729 (+4.21%+4.21\%) Weather 0.1827 0.1866 (+2.13%+2.13\%) 0.1861 (+1.86%+1.86\%) 0.1858 (+1.70%+1.70\%) Traffic 0.4714 0.4796 (+1.73%+1.73\%) 0.4800 (+1.83%+1.83\%) 0.4748 (+0.71%+0.71\%) avg. Δ% \% — +2.55%+2.55\% +2.61%+2.61\% +3.40%+3.40\% # of wins — 7 7 8 †This future-informed comparison directly thresholds the future-derived retraining label, which is unavailable at deployment and introduces information leakage. Component ablation. To identify which components are responsible for the gains of PILOT, we conducted an ablation study using the mean-degradation formulation with TimesNet across all eight datasets. Table 4 reports MSE and the relative MSE increase from PILOT. We compared PILOT against three ablated variants: w/o Scorer, which removes the learned scorer and directly thresholds the pseudo-label; w/o History, which removes the rolling historical summaries (htMAEh_t^MAE, htRMSEh_t^RMSE) from the state; and w/o Stacking, which uses only the current-step state without temporal stacking. PILOT outperformed its ablated counterparts in 22 of the 24 dataset–variant comparisons. Removing the learned scorer increased MSE by 2.55%2.55\% on average, with the largest increase on ETTh1 (+7.61%+7.61\%). Notably, w/o Scorer is a future-informed method that directly thresholds a pseudo-label unavailable at deployment; PILOT nonetheless outperformed it on seven of eight datasets. This result is consistent with a denoising or regularization effect of scorer learning and suggests that the learned scorer can yield more reliable retraining decisions than direct thresholding. Removing historical features increased MSE by 2.61%2.61\% on average and degraded seven of eight datasets, showing the value of interpreting current errors against a recent baseline. Removing temporal stacking caused the largest average increase (+3.40%+3.40\%) and degraded all eight datasets, confirming that recent error dynamics are more informative than a single-step state. Therefore, the three mechanisms are complementary: the scorer converts noisy pseudo-labels into reliable decisions, historical features provide a local reference for error severity, and stacking captures recent error dynamics. Table 5. Descriptive average MSE of alternative pseudo-label designs. Formatting indicates their average-rank positions relative to the baselines. Bold: outperforms the best baseline; underline: outperforms the second-best. Variant DLinear iTransformer TimesNet Counterfactual RT† Mean (main) 0.2857 0.2708 0.3157 No Cumulative 0.2856 0.2730 0.3164 No Relative 0.2857 0.2714 0.3189 No Binary util. 0.2872 0.2733 0.3150 Yes Ordinal util. 0.2878 0.2755 0.3137 Yes Voting 0.2873 0.2932 0.3191 Yes †Additional offline retraining simulations are required only to construct utility-based labels; they are not used during online inference. Alternative pseudo-labels. We additionally evaluated cumulative and relative degradation, binary and ordinal retraining utility, and a voting ensemble of the five scorers. Table 5 summarizes their average MSE and computational requirements; their definitions and detailed analysis are provided in Appendix B. The extension study shows that PILOT is not tied to a single supervision signal. The degradation-based variants yield similar average MSE on DLinear and iTransformer, while Ordinal Utility performs best on TimesNet. Overall, these results show that the framework accommodates alternative notions of retraining necessity, while Mean Degradation provides a simple and consistently effective default without counterfactual retraining simulation. 5. Conclusion We proposed PILOT, a pseudo-label-based online retraining method that constructs supervision from realized forecast errors. PILOT uses future mean forecast-error degradation as a pseudo-label and trains a lightweight scorer on recent error states to produce retraining decisions. It remains backbone-agnostic, requiring no modification to the forecasting model. Across the eight benchmarks and three backbones, PILOT achieved the lowest average rank among retraining policies while maintaining a favorable performance–retraining-time trade-off. Matched-budget and trigger analyses further showed that its gains arise from selective and timely retraining rather than merely from more frequent updates. Closed-loop analysis showed that scorer–degradation alignment remained stable despite successive backbone updates, while an ablation study confirmed that scorer learning, historical context, and temporal stacking each play distinct roles in producing selective retraining decisions. More broadly, these results suggest that forecast errors can serve not only as monitoring signals but also as effective sources of supervision for learning retraining decisions. Experiments with alternative pseudo-labels further showed that PILOT is not tied to a single supervision design, while mean degradation provides a simple and consistently effective default without requiring counterfactual retraining simulation. While PILOT showed promising results, several directions remain open. Its pseudo-label relies on fixed-length current and future windows, which may limit responsiveness to abrupt performance degradation. PILOT also supports only binary decisions, whereas richer actions such as recalibration, partial fine-tuning, or cost-aware update selection may further improve adaptation under evolving data streams. Appendix A Retraining Efficiency Table 6 reports #RT, total retraining time including decision overhead, time per RT, and maximum GPU memory averaged over three seeds, supporting the ETTm2 performance–efficiency comparison in Figure 2. Within each backbone, peak GPU memory and time per retraining event are broadly comparable across methods. Accordingly, total retraining cost is driven primarily by how often retraining is invoked, reinforcing the importance of deciding when to retrain—the central focus of this work. Table 6. Detailed retraining efficiency on ETTm2. Backbone Method #RT Total RT Time (s) Time / #RT (s) GPU Mem. (MB) DLinear Periodic 35.0 6.644 0.190 233.4 ADWIN 17.3 2.944 0.170 233.5 KSWIN 34.0 6.418 0.189 233.7 CARA 15.3 2.716 0.177 233.0 UPF 13.3 2.857 0.214 233.8 PILOT 22.3 4.349 0.195 232.6 iTransformer Periodic 35.0 10.810 0.309 1336.0 ADWIN 18.0 5.595 0.311 1342.8 KSWIN 33.3 10.085 0.303 1334.1 CARA 5.0 1.326 0.265 1336.1 UPF 16.0 5.243 0.328 1330.6 PILOT 15.3 5.195 0.339 1330.6 TimesNet Periodic 35.0 102.016 2.915 39.6 ADWIN 20.0 59.941 2.997 39.2 KSWIN 34.0 99.807 2.935 39.9 CARA 5.0 11.272 2.254 39.7 UPF 16.7 48.280 2.897 39.2 PILOT 17.7 49.115 2.780 39.2 Appendix B Alternative Pseudo-Labels PILOT constructs retraining supervision from future mean degradation, as described in Section 3.3. To examine its extensibility, Section 4.5 evaluates the same scorer with alternative pseudo-labels designed with degradation- and utility-based signals; their definitions and detailed results are provided below. B.1. Alternative Pseudo-Label Definitions The alternative single-label variants capture two notions of retraining necessity: (i) future degradation under the unchanged backbone (Mahadevan and Mathioudakis, 2024; Page, 1954), and (i) utility from immediate retraining (Regol et al., 2025; Zliobaite et al., 2015). B.1.1. Cumulative degradation Cumulative degradation focuses on the accumulated future deterioration by summing future MSE deviations from the recent baseline: (17) gtcum=∑i=t+Wf−1(eiMSE−e¯[t−Wc,t)MSE).g_t^cum= _i=t^t+W_f-1 (e_i^MSE- e^MSE_[t-W_c,\,t) ). This aligns with cumulative-sum monitoring statistics (Page, 1954). B.1.2. Relative degradation Relative degradation addresses scale differences by normalizing mean degradation by the recent error baseline: (18) gtrel=e¯MSE[t,t+Wf)−e¯MSE[t−Wc,t)e¯MSE[t−Wc,t)+ϵ.g_t^rel= e^MSE_[t,\,t+W_f)- e^MSE_[t-W_c,\,t) e^MSE_[t-W_c,\,t)+ε. This normalized formulation is closely related to the relative staleness cost used in cost-aware retraining (Mahadevan and Mathioudakis, 2024). These continuous degradation-based extensions use the same clipping rule as the mean-degradation formulation. B.1.3. Utility-based labels Utility-based labels instead compare future losses under stale and retrained backbones to assess the benefit of immediate retraining, following cost-sensitive retraining formulations (Zliobaite et al., 2015; Regol et al., 2025). At each candidate time step t, we compare keeping the current backbone unchanged with retraining it on the recent buffer. We then define retraining utility as (19) ut=e¯stale,tMSE−e¯retrained,tMSE,u_t= e^MSE_stale,\,t- e^MSE_retrained,\,t, where e¯stale,tMSE e^MSE_stale,\,t and e¯retrained,tMSE e^MSE_retrained,\,t are the average future errors resulting from the stale case and the retraining case, respectively. A positive value indicates that immediate retraining improves future forecasting performance. These labels require counterfactual retraining only offline and do not affect online inference. B.1.4. Binary retraining utility The binary retraining utility label indicates whether immediate retraining yields a positive utility: (20) gtbin=[ut>0].g_t^bin=[u_t>0]. We use a zero-margin rule and treat any positive utility as indicating a beneficial retraining decision. B.1.5. Ordinal retraining utility An ordinal version separates the retraining benefit into low-, medium-, and high-benefit cases. We discretize the scalar utility into three ordinal levels: (21) gtord=0,ut≤Q50,1,Q50<ut≤Q80,2,ut>Q80,g_t^ord= cases0,&u_t≤ Q_50,\\ 1,&Q_50<u_t≤ Q_80,\\ 2,&u_t>Q_80, cases where Q50Q_50 and Q80Q_80 are empirical utility quantiles estimated from the scorer-training split, following a simple quantile-based discretization strategy (Dougherty et al., 1995). Unlike continuous degradation labels, these discrete utility labels are used as-is without clipping. B.1.6. Voting ensemble We additionally combine the five pseudo-label scorers by majority voting on a shared backbone trajectory, following hard-voting strategies in ensemble learning (Dietterich, 2000; Kuncheva, 2004). At each step t, the five frozen scorers run in parallel on the same state tensor t C_t. Each scorer m is independently calibrated as in Equation 15 and thresholded to produce a binary vote at(m)a_t^(m). Retraining is triggered when at least three of the five scorers agree: (22) at=[∑m=15at(m)≥3],a_t=\! [ _m=1^5a_t^(m)≥ 3 ], subject to the same cooldown constraint as in Equation 16. Table 7. Dataset-level MSE of the alternative pseudo-label variants, averaged over three seeds. Backbone Dataset Cum. deg. Rel. deg. Binary util. Ordinal util. Voting DLinear ETTh1 0.4503 0.4503 0.4532 0.4537 0.4510 ETTh2 0.2170 0.2182 0.2189 0.2240 0.2178 ETTm1 0.3574 0.3574 0.3601 0.3578 0.3601 ETTm2 0.1605 0.1605 0.1629 0.1620 0.1618 ECL 0.1962 0.1962 0.1956 0.1958 0.1960 Exchange 0.0771 0.0777 0.0790 0.0805 0.0853 Weather 0.1850 0.1836 0.1862 0.1860 0.1855 Traffic 0.6412 0.6414 0.6419 0.6427 0.6412 iTransformer ETTh1 0.4475 0.4464 0.4513 0.4508 0.4860 ETTh2 0.2450 0.2445 0.2465 0.2459 0.2659 ETTm1 0.3881 0.3791 0.3827 0.4009 0.5007 ETTm2 0.1662 0.1656 0.1657 0.1658 0.1724 ECL 0.1635 0.1634 0.1635 0.1636 0.1701 Exchange 0.1193 0.1193 0.1193 0.1205 0.0914 Weather 0.1793 0.1768 0.1801 0.1785 0.1816 Traffic 0.4753 0.4763 0.4774 0.4778 0.4772 TimesNet ETTh1 0.6349 0.6366 0.5977 0.6221 0.6311 ETTh2 0.3299 0.3309 0.3245 0.3139 0.3354 ETTm1 0.4051 0.3987 0.4020 0.4121 0.4043 ETTm2 0.1807 0.1815 0.1819 0.1797 0.1829 ECL 0.1613 0.1619 0.1651 0.1631 0.1612 Exchange 0.1661 0.1682 0.1721 0.1627 0.1776 Weather 0.1839 0.1834 0.1848 0.1829 0.1863 Traffic 0.4687 0.4899 0.4917 0.4733 0.4740 B.2. Alternative Pseudo-Label Performance The full comparison evaluates Mean Degradation together with four single-label extensions and the voting ensemble. Table 7 reports the dataset-level MSE of the five extensions, complementing the across-dataset averages in Table 5. Table 7 shows substantial dataset-level variation across the extensions. Cumulative and Relative Degradation attain the lowest MSE on most DLinear datasets, while Relative Degradation is lowest on six of eight iTransformer datasets. TimesNet exhibits a more heterogeneous pattern: Ordinal Utility is lowest on four datasets, with the remaining datasets favoring different variants. Voting can be particularly effective in specific settings. For example, on iTransformer–Exchange, it is the only PILOT variant to achieve a lower MSE than No Retrain (0.0914 vs. 0.0939). Ethical Considerations All experiments use publicly available time series benchmarks containing no personally identifiable or sensitive individual-level information, and involve no human or animal subjects, protected attributes, or personal data collection. PILOT is a general-purpose retraining framework not designed for sensitive or safety-critical applications, and we identify no foreseeable risks involving privacy, unfair treatment, surveillance, or malicious use. We provide source code and document the experimental settings to support transparency and reproducibility. GenAI Usage Disclosure The authors used Anthropic’s Claude and OpenAI’s ChatGPT in preparing this manuscript. Claude was used throughout the implementation process, including source code writing, debugging, and figure/table generation, as well as for refining portions of the manuscript at the paragraph and wording level. ChatGPT was used to support manuscript writing, primarily for English language refinement. The authors determined the research direction, methodological choices, selection of forecasting backbones and baseline methods, and experimental procedures, and independently interpreted all experimental results. All scientific claims and the final manuscript content were reviewed and verified by the authors, who take full responsibility for the paper. References Bifet and Gavaldà (2007) A. Bifet and R. Gavaldà Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM International Conference on Data Mining, Philadelphia, PA, p. 443–448. External Links: Document Cited by: §1, §2, 3rd item, Table 1. Castle et al. (2026) J. L. Castle, J. A. Doornik, and D. F. Hendry A novel approach to forecasting after large forecast errors. Journal of Forecasting 45 (2), p. 837–849. External Links: Document Cited by: §2. Derakhshan et al. (2019) B. Derakhshan, A. Rezaei Mahdiraji, T. Rabl, and V. Markl Continuous deployment of machine learning pipelines. In Proceedings of the 22nd International Conference on Extending Database Technology, Konstanz, Germany, p. 397–408. External Links: Document Cited by: §1. Dietterich (2000) T. G. Dietterich Ensemble methods in machine learning. In Multiple Classifier Systems, Lecture Notes in Computer Science, Vol. 1857, p. 1–15. External Links: Document Cited by: §B.1.6. Dougherty et al. (1995) J. Dougherty, R. Kohavi, and M. Sahami Supervised and unsupervised discretization of continuous features. In Proceedings of the Twelfth International Conference on Machine Learning (ICML), p. 194–202. External Links: Document Cited by: §B.1.5. Gama et al. (2014) J. Gama, I. Zliobaite, A. Bifet, M. Pechenizkiy, and A. Bouchachia A survey on concept drift adaptation. ACM Computing Surveys 46 (4), p. 44:1–44:37. External Links: Document Cited by: §3.5. Grundy et al. (2026) T. Grundy, R. Killick, and I. Svetunkov Online detection of forecast model inadequacies using forecast errors. Journal of Time Series Analysis 47 (3), p. 715–726. External Links: Document Cited by: §1, §2. Hastings Jr et al. (1947) C. Hastings Jr, F. Mosteller, J. W. Tukey, and C. P. Winsor Low moments for small samples: a comparative study of order statistics. The Annals of Mathematical Statistics 18 (3), p. 413–426. Cited by: §3.3. He and Garcia (2009) H. He and E. A. Garcia Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering 21 (9), p. 1263–1284. Cited by: §3.4. Hoffman et al. (2024) K. Hoffman, S. Salerno, J. Leek, and T. McCormick Some models are useful, but for how long?: a decision theoretic approach to choosing when to refit large-scale prediction models. External Links: 2405.13926, Document Cited by: §1. Hoi et al. (2021) S. C. H. Hoi, D. Sahoo, J. Lu, and P. Zhao Online learning: a comprehensive survey. Neurocomputing 459, p. 249–289. External Links: Document Cited by: §1. Huber (1964) P. J. Huber Robust estimation of a location parameter. The Annals of Mathematical Statistics 35 (1), p. 73–101. External Links: Document Cited by: §3.4. Kuncheva (2004) L. I. Kuncheva Combining pattern classifiers: methods and algorithms. John Wiley & Sons. Cited by: §B.1.6. Lai et al. (2018) G. Lai, W. Chang, Y. Yang, and H. Liu Modeling long- and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, p. 95–104. External Links: Document Cited by: §4.1. Liu et al. (2024) Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long ITransformer: inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, Cited by: 4th item, §4.1, §4.1. Liu et al. (2023) Z. Liu, R. Godahewa, K. Bandara, and C. Bergmeir Handling concept drift in global time series forecasting. In Forecasting with Artificial Intelligence: Theory and Applications, p. 163–189. External Links: Document Cited by: §1. Lu et al. (2019) J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang Learning under concept drift: a review. IEEE Transactions on Knowledge and Data Engineering 31 (12), p. 2346–2363. External Links: Document Cited by: §1. Mahadevan and Mathioudakis (2024) A. Mahadevan and M. Mathioudakis Cost-aware retraining for machine learning. Knowledge-Based Systems 293, p. 111610. External Links: Document Cited by: §B.1.2, §B.1, 3rd item, §1, §1, §2, §3.3, 5th item, §4.1, Table 1. Page (1954) E. S. Page Continuous inspection schemes. Biometrika 41 (1–2), p. 100–115. External Links: Document Cited by: §B.1.1, §B.1. Pearson (1895) K. Pearson Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London 58, p. 240–242. Cited by: §4.4. Raab et al. (2020) C. Raab, M. Heusinger, and F. Schleif Reactive soft prototype computing for concept drift streams. Neurocomputing 416, p. 340–351. External Links: Document Cited by: §1, §2, 4th item, Table 1. Regol et al. (2025) F. Regol, L. Schwinn, K. Sprague, M. Coates, and T. Markovich When to retrain a machine learning model. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 51369–51404. Cited by: §B.1.3, §B.1, 3rd item, §1, §1, §2, §2, 6th item, §4.1, Table 1. Trigg (1964) D. W. Trigg Monitoring a forecasting system. Operational Research Quarterly 15 (3), p. 271–274. External Links: Document Cited by: §1, §2. Trindade (2015) A. Trindade ElectricityLoadDiagrams20112014. Note: UCI Machine Learning RepositoryDOI: https://doi.org/10.24432/C58C86 Cited by: §4.1. Welford (1962) B. P. Welford Note on a method for calculating corrected sums of squares and products. Technometrics 4 (3), p. 419–420. External Links: Document Cited by: §3.5. Wilcoxon (1945) F. Wilcoxon Individual comparisons by ranking methods. Biometrics bulletin 1 (6), p. 80–83. Cited by: §4.1. Wu et al. (2023) H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long TimesNet: temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, Cited by: 4th item, §4.1, §4.1. Wu et al. (2021) H. Wu, J. Xu, J. Wang, and M. Long Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, Vol. 34, Red Hook, NY, USA, p. 22419–22430. Cited by: §4.1, §4.1. Zanotti (2025) M. Zanotti On the retraining frequency of global models in retail demand forecasting. Machine Learning with Applications 22, p. 100769. External Links: Document Cited by: §1. Zeng et al. (2023) A. Zeng, M. Chen, L. Zhang, and Q. Xu Are transformers effective for time series forecasting?. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, Washington, DC, USA, p. 11121–11128. Cited by: 4th item, §4.1. Zhang et al. (2024) Y. Zhang, W. Chen, Z. Zhu, D. Qin, L. Sun, X. Wang, Q. Wen, Z. Zhang, L. Wang, and R. Jin Addressing concept shift in online time series forecasting: detect-then-adapt. External Links: 2403.14949, Document Cited by: §1. Zhou et al. (2021) H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang Informer: beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, Washington, DC, USA, p. 11106–11115. Cited by: §4.1, §4.1. Zliobaite et al. (2015) I. Zliobaite, M. Budka, and F. Stahl Towards cost-sensitive adaptation: when is it worth updating your predictive model?. Neurocomputing 150, p. 240–249. External Links: Document Cited by: §B.1.3, §B.1, §1, §1, §2.

Instant research discovery