Paper deep dive

LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

Haomin Wen, Ziyu Zhou, Qingxiang Liu, Siru Zhong, Yuxuan Liang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 139

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/19/2026, 4:43:21 AM

Summary

The paper introduces LiveHouse-TS, the first open-world living benchmark infrastructure for Time Series Foundation Models (TSFMs). It addresses the limitations of static benchmarks by implementing a prequential evaluation protocol that tests model robustness against distribution shifts and seasonal variations in real-time. The infrastructure includes a model entrance, live data streams from 17 datasets across 11 domains, and an evaluation house that updates a real-time leaderboard. Key metrics include RMSE, MAPE, CRPS, Temporal Stability, and Improvement.

Entities (14)

LiveHouse-TS · benchmark-infrastructure · 100%CRPS · metric · 95%Prequential Evaluation · evaluation-protocol · 95%RMSE · metric · 95%Time Series Foundation Models · model-class · 95%Gift-Eval · benchmark · 90%Improvement · metric · 90%LiveBench · benchmark · 90%Monash · benchmark · 90%static benchmarks · evaluation-method · 90%Temporal Stability · metric · 90%Chronos · model · 85%Moirai · model · 85%TimesFM · model · 85%

Relation Signals (10)

LiveHouse-TS → evaluates → Time Series Foundation Models

confidence 95% · LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs.

LiveHouse-TS → includesmetric → RMSE

confidence 95% · for point accuracy we use RMSE (↓)

LiveHouse-TS → includesmetric → CRPS

confidence 95% · For probabilistic forecasts we use CRPS (↓)

LiveHouse-TS → uses → Prequential Evaluation

confidence 95% · LiveHouse-TS enforces a strict prequential evaluation protocol

static benchmarks → contrastswith → LiveHouse-TS

confidence 90% · existing evaluation protocols predominantly rely on static benchmarks... To bridge this gap, we introduce LiveHouse-TS

LiveHouse-TS → includesmetric → Improvement

confidence 90% · We next introduce Improvement (↓) to test how reliably a model improves

LiveHouse-TS → includesmetric → Temporal Stability

confidence 90% · We define Temporal Stability (↓) as the standard deviation over evaluations

TimesFM → isexampleof → Time Series Foundation Models

confidence 85% · TimesFM [15] and Timer [38] follow a decoder-only design

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecasting. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real-world environments characterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluating models prequentially on real future data in open-world environments, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

136,151 characters extracted from source content.

Expand or collapse full text

LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models Haomin Wen 2,† , Ziyu Zhou 1,† , Qingxiang Liu 1,† , Siru Zhong 1,† , Yuxuan Liang 1,* 1 The Hong Kong University of Science and Technology (Guangzhou) 2 Shanghai Innovation Institute; † Equal contribution; * Corresponding author wenhaomin.whm,zziyuzhou,qingxiangliu737@gmail.com, yuxliang@outlook.com,szhong691@connect.hkust-gz.edu.cn Abstract Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecast- ing. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how mod- els behave in continuously evolving real-world environments char- acterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluat- ing models prequentially on real future data in open-world environ- ments, LiveHouse-TS shifts time series benchmarking from snap- shot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol. Code, dataset, and leaderboard are available at: https://huggingface.co/spaces/CityMindDev/LiveHouse-TS. 1 Introduction Time series forecasting is a foundational task across a wide spec- trum of industrial and scientific domains, ranging from energy management and financial planning to climate modeling. Time series foundation models (TSFMs) have shown to be highly promis- ing paradigm for zero-shot forecasting across domains, driven by large-scale pretraining and the ability to perform zero-shot infer- ence [3,15,20,24,56]. This paradigm shift has sparked massive research interest, yielding hundreds of papers in the last two years. Concurrently, the rapid evolution of these models drives the demand for reliable evaluation protocols. As shown in Figure 1, cur- rent practice relies almost entirely on static benchmarks (e.g., GIFT- eval [1], fev-bench [51], TSFM-Bench [30]), where public datasets are split into predetermined, frozen train and test windows. While static leaderboards offer a controlled environment for initial verifica- tion, they introduce a fundamental limitation: snapshot evaluation ignores the operational realities of real-world deployment, where non-stationarity, concept drift, and sudden exogenous shifts contin- uously alter the underlying data-generating processes. In a static benchmark, a model that captures the top spot remains there indefi- nitely because standard datasets yield a permanent, immutable rank once computed. In reality, two forecasting models might achieve Round 1 Round 3 (latest) HistoryTarget ✓scored ✓scored Target (predicted) Awaiting GT⏳ NOW Current time Future (unknown) eval Time line Inference& eval HistoryTarget History Round 2 History Target Time line Static Leaderboard LiveBench:Amodelisscoredmultipletimes Real-time Leaderboard FixedData Streaming Data Inference StaticBench:Amodelisscoredonce Figure 1: Comparison between proposed LiveHouse-TS and static benchmark. LiveHouse-TS introduce a streaming evalu- ation protocol to capture model robustness under real-world operational conditions. It eliminates data leakage by requir- ing models to forecast at the current world time before ground truth is observed, with evaluation and the leader- board updated continuously as new data arrives. the exact same performance metric (e.g., a Mean Absolute Error of 0.42) on a static split, rendering them indistinguishable offline. Yet, when subjected to a rolling, real-world timeline, one model might swiftly degrade under a seasonal shift while the other maintains consistent reliability. A model’s operational superiority is not a permanent attribute; its performance and ranking must be continuously tested and earned as the world changes. To address these limitations, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. We define our open-world setting as a temporally open system featur- ing continuous streaming observations and an extensible registry for dynamically expanding data sources. As illustrated in Figure 1, LiveHouse-TS enforces a strict prequential evaluation protocol: predictions must be made at the current world time before the cor- responding ground-truth values exist, with metrics updated contin- uously as new observations arrive. Ultimately, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. We list the detailed comparsion of LiveHouse-TS and current benchmarks in Table 1. Crucially, rather than serving as a one-off leaderboard, LiveHouse-TS is conceptualized as a living arXiv:2608.17299v1 [cs.AI] 18 Aug 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 1: Comparison between representative TSFM benchmarks and LiveHouse-TS. (Abbr; LR: Leakage-resistant by nature; RTL: Real-time Leaderboard; OND: Open-to-new-data; FT: Forecasting Task, Prob.: Probability Forecasting). LiveHouse-TS provides the first systematic solution for benchmarking the live performance of time series foundation models. BenchmarkYearDatasetEvaluationLeaderboard Live/Static #Domain #Data LR Test zero-shot?FTMultivariate RTL ONDMetrics Monash [22]2021Static720✘Point✘MASE, sMAPE, msMAPE, MAE, RMSE BasicTS [50]2023Static520✘point✔✘MAE, RMSE, MAPE, WAPE TFB [45]2024Static1041✘Point✔✘✔MAE, MSE, MASE, MSMAPE ProbTS [60]2024Static612✘✔Point/Prob.✔✘NMAE, CRPS CiK [55]2025Static79✘✔Prob.✘CRPS GIFT-Eval [1]2024Static723✘✔Point/Prob.✔✘MAPE, CRPS fev-bench [51]2025Static796✘✔Point/Prob.✔✘✔MASE, SQL BOOM [11]2025Static5-✘✔Point/Prob.✔✘MASE, CRPS TSFM-Bench [30]2025Static1021✘✔Point✔✘MAE, MSE Impermanent [21]2026Live11✔Point/Prob.✔✘MASE, CRPS TS-Arena [41]2026Live13✔Point✔MASE LiveHouse-TS2026Live1117✔Point/Prob.✔RMSE,MAPE, CRPS, Stability, Improvement time series infrastructure designed to spark and systematically an- swer new scientific questions vital to the community. For example, can model rankings be maintained long-term? Which models are genuinely robust under real-world deployment? In summary, our core contributions are: •New Paradigm: We identify a critical evaluation gap in the snapshot paradigm and introduce a streaming evaluation proto- col centered on continuous temporal validity to capture model robustness under real-world operational conditions. •New Evaluation Infrastructure: We propose LiveHouse-TS, a leakage-resistant open-world live benchmark infrastructure fea- turing a real-time leaderboard, an extensible registry for streams and models, and new metrics specifically tailored for temporal stability and monotone performance improvement. •New Insights: Current TSFMs generalize well for zero-shot fore- casting on real future data. However, the rankings on LiveHouse- TS differ from those on prior static benchmarks, suggesting strong performance on static benchmarks may not necessarily translate to practical deployment. 2 Related Work Time Series Foundation Models. TSFMs are pretrained on a large cross-domain time series corpus and then applied zero-shot or with light fine-tuning to unseen datasets [3,7,9,15,18,20,24,53,56]. They vary in tokenization, architecture, and pre-training objec- tives, including DeepAR, N-BEATS, N-HiTS, PatchTST, DLinear, and TimesNet [10,42,43,48,57,59]; later models include Informer, Autoformer, FEDformer, Pyraformer, Crossformer, SCINet, and TiDE [14,34,35,58,62–64]. We refer the reader to Liang et al. [32] for a broader survey. To name a few examples, TimesFM [15] and Timer [38] follow a decoder-only design that models time se- ries as patches, whereas Chronos [2,3] discretizes (via scaling and quantization) continuous values into a token vocabulary to reuse language-model backbones. In contrast, encoder-style masked pre- training is adopted by MOIRAI [56] (masked any-variate modeling) and MOMENT [24] (masked multi-task pretraining), while Lag- Llama [46] produces probabilistic forecasts from lag-based features. Beyond modeling choices, Toto [13] and TTM [16] emphasize ob- servability and lightweight deployment, and Time-MoE [52] and Moirai-MoE [36] scale up via sparse mixture-of-experts routing. In parallel, another line of work reprograms or fine-tunes frozen language models for forecasting [8, 29, 65]. Time Series Forecasting Benchmark. Early competitions and archives fixed the unit of comparison, as M4 and M5 [39, 40] stan- dardized point and probabilistic scoring over large series collections and the Monash archive [22] consolidated datasets into one format that later became a common pretraining source. A second wave then targeted fair and reproducible comparison, where TFB [45] and BasicTS [50] control preprocessing and dataset heterogene- ity while ProbTS [60] and CiK [55] broaden the evaluation axis to distributional and context-aware forecasting. Most recently, GIFT- Eval [1,31], TSFM-Bench [30], fev-bench [51], and BOOM [11] target foundation models directly to test their zero-shot ability. Across all three waves, a curated set of public series is frozen with predetermined train/test splits, and models are scored once over the held-out windows. Such a design introduces the potential data leakage and may not reflect model performance in real deployment (as discussed in Sec 1), which motivates our live and open-world benchmark. Overall, we refer to Table 1 for a detailed comparison with LiveHouse-TS and related ones. Live Benchmark. A growing number of works in language and code evaluation address data contamination by making the bench- marks themselves live, continuously refreshing test data or gating them by release date so that every scored example is released after the model training. LiveBench [54], LiveCodeBench [28], and the multimodal LiveXiv [49] follow this principle and are supported by literature that analyzes why static benchmarks fail once their data leaks into pretraining [23,47]. The methodological basis comes from the stream-learning literature, where prequential evaluation enforces that predictions are always made before observing the corresponding labels, thereby preventing look-ahead bias [19] and enabling adaptive performance tracking under concept drift [5]. This live paradigm is well-suited for benchmarking general time se- ries forecasting, since real-world series are generated continuously and provide a natural supply of strictly post-cutoff evaluation data. 3 Benchmark Details 3.1 Task Formulation We consider a time seriesy 푡 푡≥1 observed at regular time steps, where y 푡 ∈ R 퐷 collects퐷target variables. At time푡, the goal is LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Figure 2: Overall architecture of LiveHouse-TS, as an analogy in which models are performers and the benchmark is a live house. It contains three decoupled yet coordinated components:푖)a Model Entrance that “checks tickets” and standardizes heterogeneous forecasters before they take the stage;푖)Live Data that turns public streams into a rolling set of forecasting tasks according to the dataset description (e.g., frequency, context length, and forecast horizon); and푖)an Evaluation House (the live house) that enforces the future-only rule, and updates the leaderboard. to forecast the next퐻vectors given the most recent퐿observa- tions (with optional covariates/metadata). In this setting, a point forecasting model outputs ˆ y 푡+1:푡+퐻 = 푓 y 푡−퐿+1:푡 , x 푡−퐿+1:푡 , m ,(1) where x denotes covariates (e.g., calendar features) and m denotes metadata such as frequency or horizon. Probabilistic forecasting instead targets a full predictive distribution over future trajectories, 푃 y 푡+1:푡+퐻 | y 푡−퐿+1:푡 , x 푡−퐿+1:푡 , m .(2) Zero-shot time series forecasting refers to applying a pretrained model to a novel dataset or unseen series without fine-tuning, using only the provided context window at inference time. 3.2 Overall Architecture LiveHouse-TS is guided by the following three design principles: Remark: Design principles. (푖) Leakge-resistant: Tasks are constructed from continually arriving public streams, and designed to test model’s ability on the real future to prevent the potential data leakage. (푖) Fairness: Ensuring fair comparisons across methods over time, since methods that join at different times may be evaluated over different time spans. (푖) Easy-to-scale: It should be easy for researchers and practi- tioners to join the leaderboard or contribute a new data source. Since we hope LiveHouse-TS serves as an infrastructure to evalu- ate the model’s generalizability in the open-world environment. As in Figure 2, we realize these principles with three decoupled yet coordinated components—as an analogy in which models are performers and the benchmark is a live house:푖)a Model Entrance that “checks tickets” and standardizes heterogeneous forecasters before they take the stage;푖)Live Data that turns public streams into a rolling set of forecasting tasks according to a dataset de- scription (e.g., frequency, context length, and horizon); and푖)an Evaluation House (the live house) that enforces the future-only rule, and updates the leaderboard. Details are provided in Appx. B. Model Entrance, which exposes a unified forecasting interface for forecasters, including hosted TSFMs and lightweight statisti- cal baselines. Given a context window and dataset metadata (e.g., sampling frequency and horizon), each predictor is required to re- turn forecasts aligned with the requested prediction horizon. The entrance adapter then validates the output shape and converts het- erogeneous model outputs into a common scoring representation: a mean forecast for point-error metrics such as MSE/RMSE, a median forecast for quantile-based point metrics such as MAPE. And a fixed set of quantile forecasts at predefined levels for probabilistic met- rics such as CRPS when available. If a method only provides point forecasts, we treat the point prediction as a degenerate predictive distribution for the evaluator. This canonical representation en- sures that all methods, regardless of whether they are local TSFMs, user-submitted models, or statistical baselines, are scored by the same metric implementation under the same horizon and target alignment. Details in Appx. B.1. Live Data, where collectors periodically retrieve fresh observa- tions from multiple domains (see Sec 3.3 for more details). Each stream is cleaned and mapped into a shared schema before being Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. windowed into tasks. Rather than imposing a single global setting, task construction follows the per-dataset specification (context length, prediction horizon, and frequency), ensuring that all models evaluated on a given dataset receive identical inputs and targets while respecting the natural time scale of each stream. Evaluation House. For each newly created task, the evaluation engine retrieves the historical context available at issue time and packages it into a standardized forecasting instance. A future-only gate then compares the task timestamp with each model’s admission time, filtering out any tasks issued before the model entered the leaderboard. The remaining eligible tasks are dispatched through the unified forecasting interface, and their forecasts are evaluated once the corresponding future targets become observable. Details of the evaluation house are provided in Appx. B.3. 3.3 Streaming Data 3.3.1 Data Curation. LiveHouse-TS builds its evaluation stream from public, continuously updated time series rather than from a frozen archive. Figure 3 showcases representative examples from various domains. The current registry contains 17 benchmark datasets across 15 public sources, 11 domains, and 8 native frequencies, as detailed in Table 2. The registry contains both directly reported time series (e.g., sensor readings, market prices and macro indicators) and event-derived time series, where times- tamped events are aggregated into regular buckets, such as GDELT document volume and USGS earthquake counts. Each dataset is described by a single registry entry containing its source identifier, entity granularity, native data frequency, recommended evaluation frequency, history length, forecast horizon, target variable, and optional covariates. Fourteen of the seventeen datasets include co- variates for multivariate evaluation. History and forecast windows are measured in native-frequency steps. High-rate and daily opera- tional streams provide short live contexts from minutes to weeks; while monthly and annual series preserve the longer seasonal and structural context (Appx. C.5 Table 5). We treat the native data fre- quency as a property of the series. This design choice ensures that changing how often we fetch data does not alter the forecasting problem itself—only the native frequency does. 3.3.2 Data Characteristics. The curated datasets have two proper- ties: diversity (covering qualitatively different forecasting regimes) and liveness (streaming coming data). Diversity. To avoid a high score being driven by matching a single domain, sampling rate, or smoothness pattern, we curate datasets along four complementary axes: source/domain breadth, temporal-scale breadth, task-structure breadth, and dynamic richness (Figure 4). Concretely, the current registry spans 15 public sources, 11 domains, and 8 native frequencies from one second to one year. Figure 5 summarizes this coverage by domain, native sampling frequency, and their joint distribution across the 17 datasets. Hourly series are the most common (7 datasets), followed by 15-minute (3) and daily (2); the remaining datasets cover 1s, 6min, 10min, 1mo, and 1y regimes. This range allows LiveHouse-TS to evaluate short- horizon high-rate forecasting, ordinary sensor forecasting, event- volume forecasting, and slow low-frequency forecasting under one protocol. Moreover, LiveHouse-TS is paried with dynamic richness diverse temporal behaviors such as seasonality, bursts, and regime Diverse temporal patterns across LiveBench-TS datasetsnormalized x-axis and per-series y-scale BTC 1s15 min finance1s BTC hourly23 h finance1h BTC monthly5 y finance1mo China GDP63 y macro economy1y Air quality24 h air quality1h NWS temp24 h weather1h Tide level24 h ocean6min River flow23 h hydrology15min Buoy waves23 h ocean10min Pageviews5 d web attention1d News volume23 h news events15min Earthquakes23 h disaster events1h Figure 3: Dataset examples with representative temporal pat- terns. Each panel shows one selected dataset window at its native cadence; axes are normalized independently to high- light temporal shape rather than absolute magnitude. The upper-right label gives the displayed window span, and the lower-right label is the native sampling frequency. Figure 4: Four measurements for dataset diversity. LiveHouse- TS is diverse enough both in terms of domain breadth, tem- poral scale, statistical variability, and task structure. shifts, and task-structure breadth that spans distinct forecasting setups (e.g., horizons, targets, and available covariates). Liveness. Observations are collected from public sources as they are released, which enables evaluation on values that post- date a model’s participation in the leaderboard. The leaderboard therefore evolves as new collection rounds complete and previously issued forecasts become scoreable. Every canonical observation row links back to a raw response record and parser version, and the verification run parsed 2,672 observations across all 17 datasets. To keep the main paper focused, per-dataset registry fields and verification-slice statistics are reported in Appx. C.5. 3.3.3 Data pipeline. The crawled raw data are archived with re- quest metadata, and converted into the forecasting task used for model evaluation. A future-only filter drops any context row whose observation was not yet available at forecast issue time. Appx. C.1 details collection, canonical parsing, and task filtering. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 2: Overview of streaming data in LiveHouse-TS (푛=17 datasets, 15 public sources). Native freq. is the sampling rate; Eval freq. is the recommended evaluation frequency. Both exclude crawler polling frequency. Group# DomainsNative freq.Eval freq.SourcesTargets Environment5weather, air quality, weather-energy1h (4), 1d1h (4), 1dOpen-Meteo, NASA POWER, NWS, NOAA NCEItemperature, PM 2.5 Water3hydrology, ocean6min, 10min, 15min1hUSGS Water, NOAA CO-OPS, NOAA NDBCdischarge, water level, wave height Mobility1traffic15min15minGBFS Citi Bikeavailable bikes Finance4finance1s, 1h (2), 1mo1s, 1h (2), 1moBinance, CoinGeckoclose, market price Society/economy2web attention, macro-economy1d, 1y1d, 1yWikimedia, World Bankpageviews, GDP Events2news events, disaster events15min, 1h1dGDELT, USGS Earthquakeevent volume, count Sampling frequency 1s6min10min15min1h1d1mo1yDatasets Finance 1214 Weather 213 Ocean 112 Air Quality 11 Disaster Events 11 Hydrology 11 Macro Economy 11 News Events 11 Traffic 11 Weather Energy 11 Web Attention 11 11137211Total 12 datasets per (domain, frequency) Figure 5: Coverage by domain and native sampling frequency. LiveHouse-TS evaluation covers from short-horizon high- rate forecast to long-horizon low-rate forecast. 3.4 Evaluation Mechanism 3.4.1 Metrics. Like most static benchmark, for point accuracy we use RMSE (↓) computed on푧-normalized series to make magnitudes comparable across datasets, and we additionally report MAPE (↓) when targets are bounded away from zero. For probabilistic fore- casts we use CRPS (↓), estimated from the quantiles emitted by each model. To summarize performance across datasets, we report Aver- age Rank [61] (↓) across datasets, Win Rate [61] (↑) from pairwise wins, and an Elo rating [17] (↑) that weights wins over stronger opponents more heavily and yields a robust leaderboard score. We refer to Appx. B.4 for more details. Moreover, since LiveHouse-TS is a live benchmark, we addi- tionally introduce live-specific metrics that capture what static leaderboards cannot. Let푠 푡 denote a base metric computed within a time window (i.e., a day) indexed by푡 ∈ 1, . . .,푇on a forecasting task. We define Temporal Stability (↓) as the standard deviation over evaluations; it tests how stable a model’s performance is over time: Stability= v u t 1 푇 − 1 푇 ∑︁ 푡=1 (푠 푡 − ̄ 푠) 2 , ̄ 푠= 1 푇 푇 ∑︁ 푡=1 푠 푡 ,(3) We next introduce Improvement (↓) to test how reliably a model improves as new targets are revealed. Fix a target time푢(which will be realized later). As the benchmark evolves, the model may issue multiple forecasts for this same푢at different issue times 푡 1 < ·< 푡 푇 푢 ; let푠 푢,푘 be the resulting error (lower-is-better) once 푦 푢 is revealed. We quantify the monotone trend of푠 푢,푘 푇 푢 푘=1 with Kendall’s 휏 , 휏 푢 = 2 푇 푢 (푇 푢 − 1) ∑︁ 1≤푖<푗≤푇 푢 sign(푠 푢,푗 −푠 푢,푖 ), Improvement= 1 |U| ∑︁ 푢∈U 휏 푢 , (4) where more negative values indicate a consistently decreasing (i.e., improving) error sequence as the issue time approaches the target. We choose Kendall’s휏because it is non-linear and robust to spikes, capturing whether performance predominantly improves as new information arrives (details in Appx. B.4). Model A Model B Model C Model A entryModel B entryModel C entry Live Releases Past targes excluded for late entrants would cause leakage scored after entry not scored before entry Figure 6: Future-only evaluation under asynchronous model entry: models are scored only on targets released after they join the live leaderboard, to avoid leakage. 3.4.2 Fairness. Note that in the real case, the model can join the leaderboard at different times, which introduces a significant chal- lenge to fair comparison for the live benchmark: As shown in Figure 6, models that join later may have already observed the ground-truth outcomes from earlier forecasting rounds, so scoring them on earlier periods would give an information advantage and introduce potential data leakage. To address this challenge, we pro- pose evaluating each model only on targets released after it joins the leaderboard. This keeps comparisons fair across entry times. However, models can therefore have different evaluation hori- zons under the future-only evaluation paradigm; we propose two so- lutions(푖)fixed-horizon leaderboards (daily, weekly, and monthly) so that models are compared over the same evaluation window.(푖) pair-wise historical ranking ranks models using only the forecast- ing tasks on which they were evaluated together (see more details in Appx. E). For each eligible model pair, it computes a dataset- balanced win rate from per-release comparisons (using both RMSE and CRPS), then aggregates these pairwise win rates into a model- level score by macro-averaging over eligible opponents. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 3: Overall performance on LiveHouse-TS evaluated by RMSE, MAPE, and CRPS using Average Rank (↓), Win Rate (↑), and Elo Rating (↑). All baselines are ordered by their average rank across the three metrics. The best and second-best results in each row are highlighted in red bold andblue underline, respectively. TSFMs consistently outperform classical statistical methods. Moirai-2.0 dominates probabilistic forecasting, while TimesFM-2.5 and Chronos–2 excel primarily in point accuracy. MeasureMetric Foundation modelsClassical baselines Moi2TFMChr2TiRexTabPFNSundialChrBToto1ARIMAMovAvgETSSNaive Average Rank (↓) RMSE4.70 3.303.80 4.005.806.407.1010.707.457.758.009.00 MAPE 3.703.85 4.904.305.106.456.155.058.709.659.8010.35 CRPS 2.309.207.257.054.703.153.902.958.709.359.4510.00 Win Rate (↑) RMSE0.66 0.790.750.730.560.510.450.120.410.390.360.27 MAPE 0.760.740.650.700.630.510.530.630.300.210.200.15 CRPS 0.880.260.430.450.660.810.740.820.300.240.230.18 Elo Rating (↑) RMSE1218 12541225119411611070947637834867789804 MAPE 1253113911631251117799810641055846739650666 CRPS 1470780930100312431358 12651325726666623611 4 Experiments We conduct extensive experiments to answer the following four research questions: RQ1 (Zero-shot ability). Do TSFMs genuinely have strong zero-shot forecasting ability when evaluated on the real future? RQ2 (Static vs. live rankings). Do rankings of TSFMs on LiveHouse-TS change significantly compared to static ones? RQ3 (Drift robustness). How does TSFM performance evolve as the live distribution drifts? RQ4 (Ranking Stability.) Can model rankings be maintained long-term? Models. We evaluate six TSFMs in frozen zero-shot mode, namely TiRex [4], Chronos-2 [2], and TimesFM-2.5 [15]. Toto-1.0 [12], Moirai-2.0 [33] Chronos-Bolt [3], TabPFN-TS [25], Sundial [37]. We select these models as the union of the TSFMs compared in widely used benchmarks, including GIFT-Eval [1], fev-bench [51], and TIME [44]. Collectively, these models represent major paradigms of modern TSFMs, including tokenization-based forecasting (Chronos- 2), direct continuous-value prediction (Chronos-Bolt), decoder-only large-scale forecasting (TimesFM-2.5), retrieval-enhanced forecast- ing (TiRex), probabilistic modeling (Moirai-2.0), diffusion-based forecasting (Sundial), large-scale autoregressive pre-training (Toto- 1.0), and table foundation model adaptation for time series forecast- ing (TabPFN-TS). We also include four classical baselines, namely Seasonal Naive [26], Moving Average [6], ARIMA [6], and ETS [27], which provide the evaluation baselines. The foundation models are run with frozen weights and a fixed context window. To keep the result tables compact we abbreviate the models as Chr2 for Chronos-2, TFM for TimesFM-2.5, Toto1 for Toto-1.0, Moi2 for Moirai-2.0, MovAvg for Moving-Average, TabPFN for TabPFN-TS, Chr2 for Chronos-2, and SNaive for Seasonal-Naive, while TiRex, ARIMA, and ETS keep their origianl names. Settings. All models are scored with a rolling mode, so that a forecast at time푡uses only observations up to푡and is graded once the ground truth arrives. The detailed forecasting settings of each dataset are presented in Appx. C.4 Table 5. Performance is assessed with the point metrics RMSE, and MAPE and the probabilistic metric CRPS, together with the aggregate measures Average Rank, Win Rate, and Elo detailed in Appx. B.4. Following GIFT-Eval [1], each Rank reported in the tables assigns every model a per-dataset rank by the metric (best=1) and averages these ranks within the reported group. The evaluation horizon reported in the results is from 01/06/2026 to 01/07/2026. Among the 17 datasets, 10 are consistently available; the remaining 7 occasionally have missing observations during data collection. We therefore report results on the 10 stable datasets and reserve the other 7 for future benchmarks. 4.1 RQ1. Zero-shot ability We evaluate whether TSFMs deliver strong zero-shot forecasts by comparing their predictions across the 10 eligible datasets in the live snapshot. We summarize the overall results in Table 3 and report per-dataset scores aggregated by domain, sampling frequency, and forecasting horizon in Appx. D Table 14, Table 15, and Table 16. Overall. Table 3 presents the overall performance of all baselines across three ranking metrics computed from MAE, MAPE, and CRPS, with detailed per-dataset results reported in Appx. D Table 13. Overall, TSFMs consistently outperform statistical baselines. Specif- ically, Moirai-2.0 and TimesFM-2.5 achieve the best Average Rank, whereas Toto-1.0 performs consistently worse. The Elo and Win Rate results further reveal substantial differences among TSFMs. Moirai-2.0 dominates probabilistic forecasting, while TimesFM-2.5 excels primarily in point accuracy (also see Appx. E Figure 17 and Figure 18 for more results). By different domains. Figure 7 summarizes CRPS-based ranks across the domains; detailed results for other metrics, sampling frequencies, and forecasting horizons are reported in Appx. D Ta- ble 14, Table 15, and Table 16. Moirai-2.0 stays in the Top-2 CRPS rank on most domains, and different TSFMs lead under different do- mains, frequencies, and forecast horizons. Notably, the top-ranked Moirai-2.0 performs poorly on weather-related domains (Weather, Air Quality, and Ocean), which highlights a clear opportunity to improve performance on such domains. Findings. TSFMs consistently outperform classical statistical base- lines in the zero-shot setting, yet no single model dominates both point forecasting and probabilistic forecasting. Among tested TSFMs, Moirai-2.0 achives the best in probabilistic forecasting while TimesFM- 2.5 and TiRex are good at point forecasting. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Moi2 Sundial Toto1 ChrB TabPFN Chr2 TiRex ARIMA ETS MovAvg TFM SNaive Air Quality Finance Hazards Hydrology Ocean Weather Web 321457681291011 153339111068127 1423106758.58.51112 214.54.5310.510.5869127 3.533.55.5655910.59.55.512 2.73.81.73.23.7879.710109.39 127436591110812 2 4 6 8 10 12 CRPS-based Average Rank Figure 7: CRPS-based average rank across seven domains. The results show that different TSFMs excel in distinct domains. Notably, the top-ranked Moirai-2.0 performs poorly on weather-related domains (Weather, Air Quality, and Ocean). 4.2 RQ2. Static versus live rankings To avoid biases in a single leaderboard, we compare LiveHouse- TS with three popular static benchmarks, i.e., GIFT-Eval [1], fev- bench [51], and TIME [44]. The comparison is restricted to the eight shared TSFMs with identical model versions (TabPFN-TS is unavailable in TIME). All benchmarks are ranked by CRPS. Figure 8 presents the ranking comparison. Findings. The three static benchmarks exhibit remarkable agree- ment. Chronos-2 consistently ranks first, followed by TiRex and TimesFM-2.5, while Chronos-Bolt and Sundial remain near the bottom. In contrast, LiveHouse-TS produces a quite different rank- ing. Moirai-2.0 and Toto-1.0 rise to the top, while Chronos-2 and TimesFM-2.5 fall to the bottom. Together with the RQ1 observation that Chronos-2 and TimesFM-2.5 achieve strong point accuracy but poor probabilistic calibration, these results suggest that static benchmarks fail to capture aspects of robustness that become ap- parent only under continuous live evaluation. GIFT-Evalfev-benchTIMELiveHouse-TS 1 2 3 4 5 6 7 8 CRPS Rank Chronos-2 TiRex TimesFM-2.5 Moirai-2.0 Toto-1.0 TabPFN-TS Sundial Chronos-Bolt Figure 8: CRPS ranking of the shared TSFMs across three static benchmarks and the live benchmark LiveHouse-TS. The dotted segment for TabPFN-TS indicates that it is not evaluated in TIME. The three static benchmarks reach a similar consensus, whereas the live benchmark produces a pronounced ranking inversion, showing that offline (static) rankings can diverge from online performance and that a live benchmark is necessary to assess models under real de- ployment conditions. 4.3 RQ3. Drift robustness Unlike static benchmarks, which evaluate a fixed test set, the live benchmark continuously assesses baselines on newly arriving ob- servations and therefore reveals their robustness to temporal dis- tribution shift. Figure 9 reports the Average Rank of the proposed Temporal Stability (↓) and Improvement (↓) metrics. The detailed numderical results are provided in Appx. D Table 11 and Table 12. Moirai-2.0 achieves the best performance on both metrics, consis- tent with its top ranking in the live benchmark. In contrast, Chronos- 2 and TimesFM-2.5, which consistently lead the static benchmarks, rank near the bottom among TSFMs, indicating substantial degrada- tion under temporal drift. Toto-1.0 further illustrates the difference between average accuracy and robustness. Despite ranking sec- ond in live CRPS, its Stability rank is only 10.50, revealing large performance fluctuations over time. Findings. Drift robustness largely explains the ranking inversion observed in RQ2. Models that remain stable under evolving data also achieve stronger live benchmarking performance, while strong static accuracy alone does not guarantee robust deployment. These complementary metrics provide aspects of forecasting quality that static evaluations cannot capture. Moi2 TabPFN Sundial ChrB TFM TiRex Chr2 Toto1 ARIMA MovAvg SNaive ETS 0 2 4 6 8 10 12 Average Rank StabilityImprovement Figure 9: Average Rank of Temporal Stability (blue) and Im- provement (yellow). Moirai-2.0 ranks highest on both drift- robustness metrics, while several top static performers ex- hibit substantially lower temporal stability. 4.4 RQ4. Ranking Stability Can model rankings be maintained long-term? In this subsection, we investigate whether model rankings remain stable across consec- utive weekly snapshots of the live benchmark. Unlike RQ2, which compares static leaderboards with LiveHouse-TS, this analysis fo- cuses on temporal ranking dynamics under the same evaluation protocol. Figure 11 shows CRPS rankings from W27 to W30 for all twelve baselines. Finding. Rankings continue to evolve even over consecutive weekly snapshots. Rather than converging to a fixed ordering, the leading position alternates among Chronos-2, TiRex, TimesFM-2.5, and TabPFN-TS, while several mid-ranked models exchange positions across weeks. These observations suggest that no single model can dominated the live benchmark all the way, even though it is the best method in the static benchmark. This further highlights Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. 05.2806.0406.1106.1806.2506.2807.0107.03 Wiki (1D pageviews, Timestamp: 2026-06-27) 200 300 400 07.04 09:5013:0016:1019:2022:3007.05 01:0003:2005:4008:0010:20 Wave (10min buoy wave height, Timestamp: 2026-07-04 22:40) 4 2 0 2 ContextTruthMoirai-2.0Toto-1.0Chronos-2TimesFM-2.5ARIMA (a) Wiki and Wave. 07.15 19:0007.16 18:0007.17 17:0007.18 16:0007.19 18:0007.20 18:00 PM2.5 (1h PM2.5, Timestamp: 2026-07-19 18:00) 25 50 75 07.20 14:5517:4020:2507.21 01:5505:1007:10 Potomac (15 min Potomac river flow, Timestamp: 2026-07-21 01:55) 2600 2800 3000 ContextTruthTiRexChronos-BoltTabPFN-TSSundialARIMA (b) PM2.5 and Potomac. Figure 10: Forecasting visualization on Wiki, Wave, PM 2.5, and Potomac datasets. The left of the dashed line denotes the historical context and the right the forecasting horizon. Per-model visualizations are provided in Figure 13 and Figure 14. Although different TSFMs perform well on different datasets, they share common failure modes under evolving data distribu- tions, including oversmoothing, delayed adaptation, and underestimated distribution shifts. the necessity of the proposed LiveHouse-TS and its supporting in- frastructure for establishing a realistic testbed to constantly assess model performance under real-world deployment. TFM Chr2 TiRex TabPFN Moi2 Sundial ChrB Toto1 ARIMA MovAvg ETS SNaive 1 2 3 4 5 6 7 8 9 10 11 12 CRPS Rank W27W28W29W30 Figure 11: CRPS rankings over four consecutive weekly snap- shots. The dynamic rankings demonstrate that model perfor- mance evolves with the incoming data stream, highlighting the need for continuous live evaluation rather than a single leaderboard snapshot. 4.5 Case study We visualize the predictions on four representative datasets from different domains and with diverse temporal characteristics: Wiki (daily Wikimedia page views), PM2.5 (hourly air quality), Wave (10-minute buoy wave height), and Potomac river flow (15-minute hydrology). Figure 10 shows the visualization comparison of these TSFMs together with ARIMA, and we provide the per-model show- cases in Appx. D Figure 13 and 14. As shown in Figure 10(a), on Wiki dataset, Moirai-2.0 and Chronos- 2 can yield similar predictions with the ground truth throughout the forecasting horizon, whereas Toto-1.0 exhibits unstable oscillations with wider prediction intervals. Wave further reveals the complex- ity of long-term temporal dynamics. Chronos-2, TimesFM-2.5, and TiRex initially produce less fluctuating predictions, but their pre- dictions gradually weaken and tend towards constant trajectories rather than maintaining the underlying periodicity. As shown in Figure 10(b), although TiRex, TabPFN-TS, Chronos-Bolt and Sun- dial capture the overall upeard trend of PM2.5 in later stages, they consistently underestimate the sharp increase in the earlier stages, while ARIMA remaines anchored near historical levels and failed to predict the shift. On the Potomac River dataset, the river flow drops rapidly after the predicted boundary, but all the highlighted methods react too slowly, consistently exceeding the true trajectory despite varying rates of decline. Findings. Across these datasets, TSFM failures exhibit consistent patterns rather than isolated errors. Models smooth out abrupt PM 2.5 spikes, bias toward historical levels in Potomac, or collapse long-term forecasts into smoothed trajectories in Wave when pe- riodic dynamics are under-observed. Crucially, these errors often occur simultaneously, indicating that prediction consistency does not imply reliability. Static benchmarks struggle to identify this behavior, merely averaging results over a fixed horizon. In con- trast, live benchmarks iteratively evaluate TSFMs across evolving sources, revealing real-world performance degradation. 5 Conclusion We presented a benchmark and live leaderboard for evaluating time series foundation models across diverse datasets and forecast- ing horizons. By standardizing data processing, evaluation proto- cols, this work aims to make comparisons more transparent and reproducible for the community. Our results highlight both the strengths of modern TSFMs and the remaining gaps in robustness and generalization when conditions shift across domains. Future work will expand the dataset coverage and tasks, and incorporate richer modalities to better reflect real-world deployment needs. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Limitations and Ethical Considerations. LiveHouse-TS evalu- ates forecasting models using publicly accessible time series streams and is not designed to collect private or personally identifiable in- formation; therefore, individual consent is generally not applicable to the current datasets. Data sources are reviewed for accessibility, licensing, and provenance, and future contributors are expected to exclude sensitive personal data. Nevertheless, geographic, domain, availability, and measurement biases in the selected streams may affect model scores and rankings, which should not be interpreted as evidence of universal superiority or downstream fairness. Finally, the reported forecasts and rankings are research artifacts rather than operational advice and should not be used directly for high- stakes financial, environmental, or public-safety decisions without domain-specific validation and human oversight. References [1] Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. 2024. GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation. arXiv preprint arXiv:2410.10393 (2024). NeurIPS 2024 Workshop on Time Series in the Age of Large Models (TSALM). [2]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Yuyang Wang, and Michael Bohlke-Schneider. 2025. Chronos-2: From Univariate to Universal Forecasting. arXiv preprint arXiv:2510.15821 (2025). [3] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mer- cado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. 2024. Chronos: Learning the Language of Time Series. Transactions on Machine Learning Research (TMLR) (2024). arXiv:2403.07815. [4] Andreas Auer, Patrick Podest, Daniel Klotz, Sebastian Böck, Günter Klambauer, and Sepp Hochreiter. 2025. TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning. arXiv:2505.23719 [cs.LG] https://arxiv.org/abs/2505.23719 [5]Albert Bifet and Ricard Gavaldà. 2007. Learning from Time-Changing Data with Adaptive Windowing. In Proceedings of the 2007 SIAM International Conference on Data Mining (SDM). 443–448. doi:10.1137/1.9781611972771.42 [6] George E. P. Box, Gwilym M. Jenkins, Gregory C. Reinsel, and Greta M. Ljung. 2015. Time Series Analysis: Forecasting and Control (5 ed.). John Wiley & Sons. [7]Defu Cao, Michael Gee, Jinbo Liu, Hengxuan Wang, Wei Yang, Rui Wang, and Yan Liu. 2025. Conversational Time Series Foundation Models: Towards Explainable and Effective Forecasting. arXiv:2512.16022 [8] Defu Cao, Zijie Lei, Muyan Weng, Jiao Sun, and Yan Liu. 2026. Speaking Num- bers to LLMs: Multi-Wavelet Number Embeddings for Time Series Forecasting. arXiv:2606.26487 [9] Defu Cao, Wen Ye, Yizhou Zhang, and Yan Liu. 2025.TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model. arXiv:2409.02322 [cs.LG] https://arxiv.org/abs/2409.02322 [10]Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler-Canseco, and Artur Dubrawski. 2022. N-HiTS: Neural Hierar- chical Interpolation for Time Series Forecasting. arXiv preprint arXiv:2201.12886 (2022). https://arxiv.org/abs/2201.12886 [11]Ben Cohen, Emaad Khwaja, Youssef Doubli, Salahidine Lemaachi, Chris Lettieri, Charles Masson, Hugo Miccinilli, Elise Ramé, Qiqi Ren, Afshin Rostamizadeh, Jean Ogier du Terrail, Anna-Monica Toon, Kan Wang, Stephan Xie, Zongzhe Xu, Viktoriya Zhukova, David Asker, Ameet Talwalkar, and Othmane Abou- Amal. 2025. This Time is Different: An Observability Perspective on Time Series Foundation Models. arXiv preprint arXiv:2505.14766 (2025). [12]Ben Cohen, Emaad Khwaja, Youssef Doubli, Salahidine Lemaachi, Chris Lettieri, Charles Masson, Hugo Miccinilli, Elise Ramé, Qiqi Ren, Afshin Rostamizadeh, Jean Ogier du Terrail, Anna-Monica Toon, Kan Wang, Stephan Xie, Zongzhe Xu, Viktoriya Zhukova, David Asker, Ameet Talwalkar, and Othmane Abou- Amal. 2026. This Time is Different: An Observability Perspective on Time Series Foundation Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=1jDAYXfcS2 [13]Ben Cohen, Emaad Khwaja, Kan Wang, Charles Masson, Elise Ramé, Youssef Doubli, and Othmane Abou-Amal. 2024. Toto: Time Series Optimized Transformer for Observability. arXiv preprint arXiv:2407.07874 (2024). [14]Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. 2023. Long-term Forecasting with TiDE: Time-series Dense Encoder. arXiv preprint arXiv:2304.08424 (2023). https://arxiv.org/abs/2304.08424 [15]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2024. A Decoder- Only Foundation Model for Time-Series Forecasting. In Proceedings of the 41st International Conference on Machine Learning (ICML). arXiv:2310.10688. [16]Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam H. Nguyen, Wesley M. Gifford, Chandra Reddy, and Jayant Kalagnanam. 2024. Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Fore- casting of Multivariate Time Series. In Advances in Neural Information Processing Systems (NeurIPS). arXiv:2401.03955. [17]Arpad E. Elo. 1978. The Rating of Chessplayers, Past and Present. Arco Publishing, New York. [18]Yuchen Fang, Hao Miao, Yuxuan Liang, Liwei Deng, Yue Cui, Ximu Zeng, Yuyang Xia, Yan Zhao, Torben Bach Pedersen, Christian S. Jensen, Xiaofang Zhou, and Kai Zheng. 2026. Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review. IEEE Transactions on Knowledge and Data Engineering 38, 3 (2026), 2040–2063. [19]João Gama, Raquel Sebastião, and Pedro Pereira Rodrigues. 2013. On Evaluating Stream Learning Algorithms. Machine Learning 90, 3 (2013), 317–346. doi:10. 1007/s10994-012-5320-9 [20]Azul Garza, Cristian Challu, and Max Mergenthaler-Canseco. 2023. TimeGPT-1. arXiv preprint arXiv:2310.03589 (2023). [21] Azul Garza, Renée Rosillo, Rodrigo Mendoza-Smith, David Salinas, An- drew Robert Williams, Arjun Ashok, Mononito Goswami, and José Martín Juárez. 2026. Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting. arXiv preprint arXiv:2603.08707 (2026). [22] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. 2021. Monash Time Series Forecasting Archive. In Proceedings of the NeurIPS Track on Datasets and Benchmarks. arXiv:2105.06643. [23]Shahriar Golchin and Mihai Surdeanu. 2024. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. In The Twelfth International Conference on Learning Representations (ICLR). arXiv:2308.08493. [24] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. MOMENT: A Family of Open Time-Series Foundation Models. In Proceedings of the 41st International Conference on Machine Learning (ICML). arXiv:2402.03885. [25]Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. 2026. From Tables to Time: Extending TabPFN-v2 to Time Series Forecasting. arXiv:2501.02945 [cs.LG] https://arxiv.org/abs/2501.02945 [26] Rob J. Hyndman and George Athanasopoulos. 2018. Forecasting: Principles and Practice (2 ed.). OTexts, Melbourne, Australia. https://otexts.com/fpp2/ [27] Rob J. Hyndman, Anne B. Koehler, J. Keith Ord, and Ralph D. Snyder. 2002. A State Space Framework for Automatic Forecasting Using Exponential Smoothing Methods. International Journal of Forecasting 18, 3 (2002), 439–454. [28] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Repre- sentations (ICLR). arXiv:2403.07974. [29] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. 2024. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. In The Twelfth International Conference on Learning Representations (ICLR). arXiv:2310.01728. [30] Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Christian S Jensen, et al.2025. Tsfm-bench: A comprehensive and unified benchmark of foundation models for time series forecasting. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 5595–5606. [31]Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, and Bin Yang. 2024. FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting. arXiv preprint arXiv:2410.11802 (2024). [32]Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. 2024. Foundation Models for Time Series Analysis: A Tutorial and Survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). 6555–6565. doi:10.1145/3637528. 3671451 arXiv:2403.14735. [33]Chenghao Liu, Taha Aksu, Juncheng Liu, Xu Liu, Hanshu Yan, Quang Pham, Silvio Savarese, Doyen Sahoo, Caiming Xiong, and Junnan Li. 2026. Moirai 2.0: When Less Is More for Time Series Forecasting. arXiv:2511.11698 [cs.LG] https://arxiv.org/abs/2511.11698 [34]Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. 2022. SCINet: Time Series Modeling and Forecasting with Sample Con- volution and Interaction. In Advances in Neural Information Processing Systems. https://arxiv.org/abs/2106.09305 [35]Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. 2022. Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. In International Conference Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. on Learning Representations. https://iclr.c/virtual/2022/poster/6827 [36]Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmer- mann, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. 2024. Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts. arXiv preprint arXiv:2410.10469 (2024). [37]Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2025. Sundial: A Family of Highly Capa- ble Time Series Foundation Models. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=LO7ciRpjI5 [38]Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2024. Timer: Generative Pre-trained Transformers Are Large Time Series Models. In Proceedings of the 41st International Conference on Machine Learning (ICML). arXiv:2402.02368. [39]Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2020. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36, 1 (2020), 54–74. [40]Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2022. The M5 competition: Background, organization, and implementation. International Journal of Forecasting 38, 4 (2022), 1325–1336. [41]Marcel Meyer, Sascha Kaltenpoth, Henrik Albers, Kevin Zalipski, and Oliver Müller. 2026. TS-Arena – A Live Forecast Pre-Registration Platform. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. ACM, 9558–9568. doi:10.1145/3770855.3817515 [42]Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In International Conference on Learning Representations. https://arxiv.org/abs/2211. 14730 [43] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2020. N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Fore- casting. In International Conference on Learning Representations. https://arxiv. org/abs/1905.10437 [44] Zhongzheng Qiao, Sheng Pan, Anni Wang, Viktoriya Zhukova, Yong Liu, Xudong Jiang, Qingsong Wen, Mingsheng Long, Ming Jin, and Chenghao Liu. 2026. It’s TIME: Towards the Next Generation of Time Series Forecasting Benchmarks. arXiv preprint arXiv:2602.12147 (2026). [45]Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. 2024. TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. Proceedings of the VLDB Endowment (PVLDB) 17, 9 (2024), 2363–2377. arXiv:2403.20150. [46] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Hena Ghonia, Rishika Bhag- watkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopou- los, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zantedeschi, Yuriy Nevmyvaka, and Irina Rish. 2023. Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting. arXiv preprint arXiv:2310.08278 (2023). [47]Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP Evaluation in Trouble: On the Need to Mea- sure LLM Data Contamination for each Benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023. 10776–10787. arXiv:2310.18018. [48] David Salinas, Valentin Flunkert, and Jan Gasthaus. 2017. DeepAR: Proba- bilistic Forecasting with Autoregressive Recurrent Networks. arXiv preprint arXiv:1704.04110 (2017). https://arxiv.org/abs/1704.04110 [49] Nimrod Shabtay, Eli Schwartz, Assaf Arbelle, Peter Staar, Sivan Doveh, Kate Saenko, Leonid Karlinsky, and Raja Giryes. 2025. LiveXiv – A Multi-Modal Live Benchmark Based on Arxiv Papers Content. In The Thirteenth International Conference on Learning Representations (ICLR). arXiv:2410.10783. [50]Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, Zhao Zhang, Di Yao, Tao Sun, Guangyin Jin, Xin Cao, Gao Cong, Christian S. Jensen, and Xueqi Cheng. 2023. Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis. arXiv preprint arXiv:2310.06119 (2023). [51]Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, and Yuyang Wang. 2025. fev-bench: A Realistic Benchmark for Time Series Forecasting. arXiv preprint arXiv:2509.26468 (2025). [52]Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. 2025. Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. In The Thirteenth International Conference on Learning Repre- sentations (ICLR). arXiv:2409.16040. [53]Shiyu Wang, Juntong Ni, Ziyi Zhang, Baichuan Mo, Xinyue Zhong, Chengxin Wang, Yuchen Fang, Zhou Ye, and Yang Xiang. 2026. ConFlux: Multivariate Time Series in Flux, One Unified Forecast in Confluence. In International Conference on Machine Learning (ICML). [54]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. In The Thirteenth International Conference on Learning Representations (ICLR). Spotlight; arXiv:2406.19314. [55]Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, and Alexandre Drouin. 2025. Context is Key: A Benchmark for Forecasting with Essential Textual Information. In Proceedings of the 42nd International Conference on Machine Learning (ICML). arXiv:2410.18959. [56]Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Unified Training of Universal Time Series Forecasting Transformers. In Proceedings of the 41st International Conference on Machine Learning (ICML). arXiv:2402.02592. [57]Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In International Conference on Learning Representations. https://arxiv. org/abs/2210.02186 [58]Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Fore- casting. In Advances in Neural Information Processing Systems. https://arxiv.org/ abs/2106.13008 [59]Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting?. In Proceedings of the AAAI Conference on Artificial Intelligence. https://arxiv.org/abs/2205.13504 [60]Jiawen Zhang, Xumeng Wen, Zhenwei Zhang, Shun Zheng, Jia Li, and Jiang Bian. 2024. ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. arXiv:2310.07446. [61]Xiyuan Zhang, Danielle Maddix Robinson, Junming Yin, Nick Erickson, Ab- dul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu, Christos Faloutsos, Michael Mahoney, et al.2026. Mitra: Mixed synthetic priors for enhancing tabular foundation models. Advances in neural information processing systems 38 (2026), 15795–15840. [62] Yunhao Zhang and Junchi Yan. 2023. Crossformer: Transformer Utilizing Cross- Dimension Dependency for Multivariate Time Series Forecasting. In Interna- tional Conference on Learning Representations. https://openreview.net/forum?id= vSVLM2j9eie [63]Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence. https://arxiv.org/abs/2012.07436 [64]Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency Enhanced Decomposed Transformer for Long- term Series Forecasting. In International Conference on Machine Learning. https: //arxiv.org/abs/2201.12740 [65] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. 2023. One Fits All: Power General Time Series Analysis by Pretrained LM. In Advances in Neural Information Processing Systems (NeurIPS). arXiv:2302.11939. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY A Getting Started We introduce how external participants can connect their TSFMs and contribute datasets to the live leaderboard. Because the bench- mark updates in real time, each model must support sustainable repeated inference. Therefore, the leaderboard does not download model weights or execute user code. Instead, participants host a forecasting endpoint—such as a Hugging Face Space or Inference Endpoint—which the leaderboard calls via a standardized API. Par- ticipants manage inference resources, while the leaderboard handles data collection, evaluation, aggregation, and display. A.1 How to join the leaderboard To join the leaderboard, participants provide a public Hugging Face model repository, a public URL for the endpoint implementa- tion, and a stable HTTPS forecasting endpoint. Inference runs on participant-controlled infrastructure: the model owner supplies the inference compute, while the leaderboard handles live task gener- ation, evaluation, aggregation, and display. A paid Hugging Face Space, a participant-owned domain, and a public server IP are not required. The recommended workflow is: (1)Initialize the portable endpoint template, replace itsforecast_one function with the model inference logic, and start the service on the participant’s inference server. (2)Validate the local endpoint using a complete forecasting request. (3)Run the publishing helper. By default, it exposes the local service through a persistent Tailscale Funnel with managed HTTPS and validates the resulting public route. A stable institutional HTTPS endpoint may be supplied instead. (4)Submit the generated metadata and validation receipt after both the local and public endpoints pass validation. The default deployment requires Python 3, Docker Engine, and Docker Compose v2. When the Tailscale route is used, Tailscale 1.52 or later must also be installed on the inference server. The complete setup is: git clone https :// github.com/zhouziyu02/TS -Live.git cd TS -Live python3 -m venv .venv source .venv/bin/activate python -m pip install -r requirements.txt export MODEL_ID="your -hf -username/your -model" export DISPLAY_NAME="YourModelName" export CODE_URL="https :// github.com/your -org/your - endpoint" # Verify the container runtime and Compose installation. docker version docker compose version # Generate the portable forecasting service. python scripts/community_model_wizard.py init \ --output -dir forecast -service # Replace forecast_one and add model -specific dependencies # to forecast -service/requirements.txt. Publish this source # directory at CODE_URL before submitting the model. docker compose -f forecast -service/compose.yaml \ up -d --build # Check the container and wait for the local service. docker compose -f forecast -service/compose.yaml ps curl --fail --show -error \ --retry 30 --retry -delay 2 --retry -connrefused \ http ://127.0.0.1:7860/ health # Send a complete local forecasting request. python scripts/validate_external_model_endpoint.py \ --endpoint -url http ://127.0.0.1:7860/ forecast \ --model -id "$MODEL_ID" \ --allow -http # One -time Linux setup for the default Tailscale route. tailscale version sudo tailscale up sudo tailscale set --operator="$USER" tailscale status # Validate locally , create the persistent HTTPS Funnel , # validate the public route , and generate both artifacts. python scripts/community_model_wizard.py publish \ --model -id "$MODEL_ID" \ --display -name "$DISPLAY_NAME" \ --code -url "$CODE_URL" \ --output -dir community -submission The first Funnel activation may require the participant or a tailnet administrator to approve Funnel and HTTPS access in a browser. If the publishing helper displays an approval URL, the participant completes this one-time authorization and reruns the samepublishcommand. The helper allows up to ten minutes for the public DNS record and HTTPS route to become available. Participants who already operate a stable institutional HTTPS endpoint, or a named Cloudflare Tunnel with a stable hostname, may skip the Tailscale setup and run: python scripts/community_model_wizard.py publish \ --model -id "$MODEL_ID" \ --display -name "$DISPLAY_NAME" \ --code -url "$CODE_URL" \ --endpoint -url \ https :// forecast.your -domain.example/forecast \ --output -dir community -submission For each evaluation request, the endpoint receives only the causal target history, an opaque series identifier, the forecast horizon, fre- quency, and requested quantile levels. It does not receive future observations, ground truth, raw dataset names, private metric val- ues, or predictions from other models. The validator checksGET /healthand sends a complete request toPOST /forecast. It veri- fies the response length, numerical finiteness, requested quantiles, response-size limit, and HTTPS requirement before producing a successful receipt. The default Funnel is initiated from the inference server and therefore requires no inbound firewall rule, participant-owned do- main, or manually managed TLS certificate. Its*.ts.netaddress remains stable while the Tailscale device identity and MagicDNS name are retained. Tailscale Funnel is a low-friction option but remains a beta service with provider-defined bandwidth limits. Par- ticipants requiring a custom domain or stronger ingress guarantees should use an institutional HTTPS service or a named Cloudflare Tunnel. Temporary tunnel URLs are not accepted. After successful validation, the participant pastes the contents of community-submission/community_model.yamlandcommunity-submission/validator_receipt.json into the community model request form at https://github.com/ Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. zhouziyu02/TS-Live/issues/new?template=community-model.yml. The submitted endpoint is reviewed before being enabled. An accepted model is admitted only to future live evaluation rounds and is never backfilled on releases preceding its admission time. Its results appear after the next successful evaluation cycle. De- tailed release snapshots may be retained privately for auditing and metric recomputation, while only metric summaries and aggregate leaderboard tables are published. A.2 How to contribute new dataset As an open online benchmark, LiveHouse-TS accepts new public time series streams from the community. A contribution plugs into the same live loop as the built-in sources in Sec. 3.3: register the dataset semantics, ingest fresh observations on a recurring schedule, and let the evaluation house form and score forecasting tasks auto- matically. Contributors provide (i) a dataset-level specification that defines the forecasting problem and (i) an ingestion adapter that fetches the public source and emits normalized observations. The full attribute contract and preprocessing requirements are given in Appx. B.2; accepted sources are reviewed for licensing, stability, and schema compliance before entering the public registry. B Implementation Details of LiveHouse-TS B.1 Model Entrance. The model entrance is the component through which all forecasting methods are registered, adapted, and admitted into LiveHouse-TS. Its primary role is to decouple model-specific inference logic from the live data and evaluation pipeline. A model may be a hosted TSFM, an external service, a local predictor, or a statistical baseline; after passing through the entrance, however, all methods expose a uniform forecasting contract to the evaluation house. Model registry. Each model is specified by a registry entry con- taining a stable model identifier, a display name, the model type, organization metadata, links to the model and replication code, and the information needed to instantiate its adapter. For hosted TSFMs, this includes the remote model identifier and API backend. For community submissions, the entry may instead specify a vali- dated HTTPS forecast endpoint. For statistical baselines, the entry records the algorithm family and its fixed hyperparameters, such as the ARIMA order, moving-average window, or season length. The registry also records whether a model is enabled for public evalua- tion and the time from which its future-only evaluation window begins. This start time is important: LiveHouse-TS never backfills a newly added model on targets that were already observable before the model entered the benchmark. Unified forecasting contract. For every admitted evaluation task, the entrance presents the model with a causal context window, the sampling frequency, the prediction horizon, and minimal task metadata. Formally, the model receives x 1:푇 , a frequency descriptor 푓, and a horizon퐻, and is asked to return forecasts for x 푇+1:푇+퐻 . The standardized output contains a point forecast and, when available, predictive quantiles: ˆ 흁 1:퐻 , ˆ q 휏,1:퐻 휏∈Q , whereQdenotes the requested quantile levels. This contract is im- plemented through a common predictor abstraction, so the down- stream evaluator does not need method-specific logic for TSFM APIs, local predictors, endpoints, or baselines. Forecast arrays are checked for shape, horizon length, and finite numeric values before they are converted into the common evaluation format. Adapters for heterogeneous model outputs. Different TSFMs expose various native output formats. Some return means and quantile forecasts; others return samples, prediction intervals, or point forecasts. The model entrance normalizes these outputs into a uniform mean-plus-quantiles representation. If a model provides samples, empirical quantiles are computed from the sample paths. If a model exposes only point forecasts, the adapter constructs an approximate predictive distribution via repeated stochastic calls or residual-bootstrap perturbations. This normalization allows point and distributional metrics, such as the quantile-loss approximation to CRPS, to be computed by a single metric engine. Hosted and external models. Built-in hosted TSFMs are queried through authenticated forecast APIs with fixed request fields: his- torical target values, frequency, prediction length, and requested quantile levels. Community models can be integrated through a self-hosted HTTPS endpoint. In this mode, LiveHouse-TS treats the submitted model as a black box: it does not import user code, download user weights, or execute third-party dependencies inside the leaderboard process. By default, the endpoint receives only the causal context, an opaque series identifier, the frequency, the horizon, and the requested quantiles. Future targets, metric values, authentication tokens, raw leaderboard internals, and other models’ predictions are never sent to the endpoint. The adapter enforces request timeouts, retry limits, maximum context lengths, maximum response sizes, and forecast-validity checks. Statistical baselines. The same entrance also hosts non-pretrained reference methods, including seasonal naive, moving average, ARIMA, and exponential smoothing. These baselines are fit or instantiated only from the context window of the current task and therefore obey the same causal restriction as TSFMs. Since several classical methods naturally produce point forecasts, the entrance derives quantile forecasts using residual-bootstrap samples when proba- bilistic outputs are required. This makes the baselines compatible with both point-accuracy and distributional metrics without giving them any special treatment in the evaluator. Failure handling and auditability. The entrance isolates model failures from the rest of the benchmark. Transient API failures can be retried, while invalid responses, non-finite forecasts, short hori- zons, malformed quantiles, or endpoint errors are rejected before scoring. For each successful run, LiveHouse-TS stores model meta- data, task metadata, metric rows, and evaluation timestamps as persistent artifacts. These records make it possible to audit which adapter, model identifier, context window, horizon, and evaluation time produced each leaderboard entry. Thus, the model entrance provides both a practical integration layer for heterogeneous fore- casting systems and a reproducibility boundary that keeps the live evaluation protocol model agnostic. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY B.2 Live Data. As a public online benchmark, LiveHouse-TS is designed to grow beyond its initial registry. Researchers and practitioners can con- tribute additional public streams—for example, a weather station feed, an open mobility API, or a government statistics portal—as long as the source is openly accessible and can be mapped into the shared contract below. Each contribution reuses the same live loop as the built-in sources: raw responses are archived for provenance, observations are normalized into a common schema, forecast tasks are generated from per-dataset window settings, and the evaluation house scores only targets that become observable after a model joins the leaderboard. Contribution contract. A new source requires two complemen- tary pieces. First, a dataset specification that semantically defines the forecasting problem (domain, entity type, native frequency, evalua- tion frequency, context and horizon lengths, targets, and covariates). Second, an ingestion adapter that periodically retrieves the public endpoint, preserves immutable raw evidence, and emits normalized observation records. The dataset specification is the single source of semantic truth; polling cadence is an operational scheduling choice and is kept separate from the forecasting definition itself. Required dataset attributes. Each contributed dataset must de- clare the attributes below. •Identity: dataset identifier, source identifier, human-readable name, domain, entity granularity, and short English descrip- tion/background for downstream users. •Forecasting setup: nativedata_frequency, recommendedeval_frequency, history_length_stepsandforecast_horizon_steps(both measured in native-frequency steps),target_variables, and optional covariate_variables. •Behavior flags: whether the series is ready for direct zero-shot leaderboard scoring, and whether raw events must be aggregated to a regular grid before forecasting. History and horizon windows are always expressed in the native series frequency, so a high-rate stream and a monthly macro series can coexist without forcing a single global window length. Storage model. The data layer separates three concerns rather than relying on one monolithic training file. • Raw archive: immutable copies of public responses, together with request metadata, fetch time, content hash, and parser ver- sion, so every normalized value remains auditable. •Canonical store: relational long-format tables for datasets, enti- ties, variables, observations, forecast tasks, and quality/run logs. This layer is the source of truth. •Task export: ephemeral model-facing bundles that pair a con- text window with dataset metadata; future targets are exported separately for the evaluator only. Preprocessing requirements. After each fetch, an ingestion adapter must satisfy the following requirements. (1)Preserve provenance. Store the untouched response before parsing; never overwrite prior raw evidence. (2)Normalize entities and variables. Map each forecastable object (station, market, fleet, grid point, country, etc.) to a stable entity identifier, and map measurable quantities to variable identifiers with units and frequency hints. (3) Emit long-format observations. Represent each value as one record keyed by entity, timestamp, and variable. Every record must carry: • timestamp: when the measurement refers to; • available_time: when the value became knowable to a fore- caster; • ingest_time: when LiveHouse-TS retrieved it; •numericvalue,frequency,unit, and a link back to the raw evidence. Event streams (e.g., news documents or earthquake catalogs) must be bucketed to the declared native frequency when aggre- gation is required. (4) Generate forecast tasks. Instantiate tasks from the dataset specification: forecast issue time, context window, horizon win- dow, covariate set, and frequency must be identical for all mod- els evaluated on the same dataset. (5) Report data health. Record missingness, duplicates, freshness delay, outliers, and ingestion failures so unstable or stale sources do not silently enter the leaderboard. Leakage rule. Model inputs must never contain information that was unavailable at prediction time. For every context observation, available_time≤ forecast_issue_time. A measurement timestamp alone is therefore insufficient: the value must also have been publicly knowable before the forecast was issued. Future targets are kept evaluator-only and are never exposed as model inputs. This rule is the same leakage-resistant contract used by the built-in streams described in Appx. C.1. Once accepted, a contributed source inherits the same task generation, future-only gating, and leaderboard update loop as the core registry entries. B.3 Evaluation House. The evaluation house is the component in LiveHouse-TS that turns standardized model forecasts into auditable leaderboard records. It receives forecasting tasks from the live data stage and model predictions from the model entrance, then applies a fixed evaluation protocol to all admitted model–task pairs. Its design goal is to ensure that every reported score is computed from the same target window, the same metric implementation, and the same aggregation rule, regardless of the model’s inference backend or output format. Task admission and future-only scoring. For each refreshed data stream, the live data stage materializes a forecasting task con- sisting of a historical context window x 1:푇 , a prediction horizon퐻, dataset metadata, and the future target x 푇+1:푇+퐻 . The evaluation house first checks whether the task is eligible for a given model. Let 푎 푚 denote the accepted entrance time of model푚. A task is scored for푚only if its target window is generated from observations that become available after푎 푚 . This future-only gate prevents newly submitted models from being evaluated retrospectively on data that could have been inspected during model development or endpoint debugging. The same gate is applied to hosted TSFMs, external community endpoints, local wrappers, and statistical baselines. Forecast execution. After admission, the evaluation house dis- patches the task through the unified predictor interface. The model receives only the causal context, frequency, and horizon; the target values are held out until scoring. The returned forecast is converted Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. into a common representation containing a mean or median forecast and, when available, a set of predictive quantiles. The evaluator val- idates that all forecast arrays have the required horizon length and contain finite numeric values. Invalid, malformed, or incomplete outputs are rejected before metrics are computed, so downstream ranking is never based on partially parsed forecasts. Metric computation. The metric engine evaluates both point ac- curacy and probabilistic quality. For point forecasts, LiveHouse-TS reports metrics such as MSE, RMSE, MAE, MASE, MAPE, sMAPE, NRMSE, and ND when they are well defined. For probabilistic fore- casts, the evaluator computes interval and quantile-based scores, including MSIS and the mean weighted sum quantile loss, which is used as the CRPS-style distributional metric in the leaderboard. All metrics are computed on the same held-out target window for all models admitted to that task. Metrics that are undefined for a particular target, such as percentage errors near zero, are marked as unavailable rather than silently imputed. Reference baselines and relative gain. The evaluation house also maintains matched baseline scores for lightweight reference methods. In particular, Seasonal-Naive is used as a causal reference because it requires no pretraining and can be instantiated from the same context window as every other method. For live aggregate reporting, LiveHouse-TS computes a relative real-time gain (RTG) from matched MSE values: RTG(푚)= 100· MSE SNaive − MSE 푚 MSE SNaive + MSE 푚 . This bounded form gives positive values to models that improve over the seasonal-naive reference and negative values to models that underperform it, while avoiding instability when absolute errors are small. Aggregation and ranking. The evaluation house produces fine-grained and aggregate views. At the lowest level, it stores a metric row for each model–dataset–release combination. Dataset- level scores are obtained by averaging over releases from the same stream. Overall live scores assign equal weight to datasets, prevent- ing frequently refreshed streams from dominating. For grouped analysis, LiveHouse-TS reports GIFT-Eval-style aggregates by do- main, frequency, and prediction length. These grouped tables nor- malize MSE and CRPS against matched Seasonal-Naive scores, then aggregate across configurations. Ranks are computed from matched comparisons rather than from incomparable partial records. For compact overall presenta- tion, LiveHouse-TS reports average rank, pairwise win rate, and an Elo-style score based on shared releases. The rank computation uses the lower-is-better ordering of MSE and CRPS and only com- pares models on tasks where both models have valid scores. This preserves a fair comparison when some models join later or when an admitted forecast fails validation for a particular release. Persistence and reproducibility. Every successful evaluation produces persistent artifacts: the model name and identifier, dataset configuration, release timestamp, prediction horizon, metric values, and aggregation metadata. These artifacts are stored separately from the model execution code and are sufficient to reconstruct the public leaderboard tables. The system also records metadata for aggregate tables, including generation time, the normalization baseline, the aggregation rule, and the number of contributing models and configurations. This separation allows LiveHouse-TS to audit individual leaderboard entries, regenerate aggregate tables, and distinguish changes caused by new data from changes caused by model or code updates. Robustness of the online loop. Because LiveHouse-TS operates continuously, the evaluation house is designed to handle partial failures without interrupting the full benchmark. A failed model call, invalid endpoint response, missing quantile, or undefined met- ric affects only the corresponding model–task pair. Other models evaluated on the same task remain valid, and later releases can still contribute new evidence for the failed model. As new observations arrive, the same admission, inference, scoring, persistence, and ag- gregation steps are repeated. The evaluation house therefore serves as the reproducible boundary between live forecasting execution and public leaderboard publication. B.4 Evaluation Metrics We evaluate each models along two complementary perspectives. Basic metrics quantify the absolute quality and cost of a forecast on each dataset, while performance-rank metrics aggregate these per-dataset scores into a single comparable measure of relative standing across the whole benchmark. Throughout we let y 푡 ∈ R 퐶 denote the ground-truth vector at forecast step푡and ˆ y 푡 ∈ R 퐶 the corresponding prediction over a horizon of length퐻, where퐶is the forecast dimension and∥·∥ denotes a vector norm. MSE and RMSE. The mean squared error (MSE) and root mean squared error (RMSE) are lower-is-better point-accuracy metrics. MSE penalizes large deviations quadratically and RMSE reports the same quantity in the original data scale, MSE= 1 퐶퐻 퐻 ∑︁ 푡=1 y 푡 − ˆ y 푡 2 2 ,RMSE= √ MSE.(5) We compute both on푧-normalized series so that datasets with different magnitudes contribute comparably. MAPE. The mean absolute percentage error (MAPE) is a lower-is- better metric that expresses error relative to the magnitude of each target, which makes it scale-free and comparable across datasets, MAPE= 1 퐶퐻 퐻 ∑︁ 푡=1 (y 푡 − ˆ y 푡 )/y 푡 1 ,(6) Because MAPE is undefined when a target is zero and inflates near small targets, we report it on series whose values stay bounded away from zero and rely on MSE and RMSE elsewhere. CRPS. The continuous ranked probability score (CRPS) is a lower-is-better metric that assesses probabilistic forecasts. For a single coordinate with predicted cumulative distribution퐹and realized value푦 it is CRPS(퐹,푦)= ∫ ∞ −∞ 퐹(푧)− 1푧 ≥ 푦 2 d푧,(7) and we average it over the퐶coordinates of ˆ y 푡 and over the퐻 forecast steps. CRPS is a strictly proper scoring rule that jointly rewards calibration and sharpness, and it reduces to absolute error when the forecast is a point mass. We estimate it from the predicted quantiles emitted by each model. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Average Rank. This lower-is-better metric ranks models on each dataset by a basic metric and averages these ranks across datasets, providing a simple scale-free indicator of consistent standing. Win Rate. This higher-is-better metric is the fraction of pairwise comparisons a model wins. For each dataset and each opposing model it scores a win when its metric is better, and we report the proportion of wins over all such comparisons. Elo. Elo rating [17] is a higher-is-better metric that treats per- dataset head-to-head outcomes as matches and fits a rating푅 푚 to each model, where expected score of model푚 against model 푛 is E[푆 푚,푛 ]= 1 1+ 10 (푅 푛 −푅 푚 )/400 .(8) Ratings are updated from observed wins and losses, so Elo rewards beating strong competitors more than weak ones and yields a single rating robust to the inclusion or removal of individual models. C Dataset details The main paper summarizes benchmark coverage and task config- uration. This appendix provides additional background on series origins, physical or social quantities represented, and public acquisi- tion methods. All sources are openly accessible without proprietary API keys. Table 4 documents each dataset’s monitoring entity, fore- casting target, public endpoint, and background notes from our diversity verification samples. C.1 Data pipeline Collection. At each collection round, the pipeline loads the reg- istry, queries the corresponding public endpoints, and stores the response body without modification under a source-specific raw directory. Collection runs on a source-dependent schedule, from frequent polls for high-rate markets to daily polls for environmental and macroeconomic feeds. Each round appends new raw response records rather than overwriting prior responses; when sufficient ob- servations arrive, the pipeline exports fresh forecasting tasks whose future windows can be scored against newly observed ground truth. The raw layer keeps JSON, TXT, or ZIP responses together with request URL, request parameters, HTTP status, fetch time, content hash, parser version, and error messages when applicable. This raw response archive makes every benchmark observation traceable to the exact public response from which it was parsed. Canonical parsing. Successful raw responses are parsed into a small relational schema rather than a single monolithic CSV. The metadata tables store dataset definitions, entities, variables, raw response records, quality reports, and run logs. The central table is a long-format observation table: dataset_id, source_id, entity_id, variable_id, timestamp, available_time, ingest_time, value, frequency, unit, raw_id This design preserves provenance and supports heterogeneous sources with different entities, units, and update mechanisms. For event sources, such as GDELT and USGS Earthquake, raw events are converted into regular aggregate time series, e.g., document volume or earthquake counts per time bucket. Task export. Forecasting tasks are generated from the canonical observation table using the history and horizon lengths specified in the registry. Each task records a target entity, target variable, forecast issue time, context window, horizon window, covariate list, and frequency. The model-facing export containstask.json andcontext.csv; labels are isolated infuture_target.csvfor the evaluator. To prevent look-ahead leakage, the exporter enforces available_time≤ forecast_issue_time for every context row. Thus, a timestamp is insufficient for inclusion: values must also be available prior to prediction issuance. C.2 Per-dataset background and access C.3 Event-derived series Two datasets arrive as events rather than natively regular mea- surements. GDELT returns a timeline of relative news-document volume for the query “climate change” at 15-minute resolution. USGS earthquakes publishes a rolling one-week GeoJSON feed of global events. For both sources, we aggregate timestamped events into regular time buckets before forming forecasting tasks, so that event-derived streams follow the same leaderboard protocol as directly reported time series. C.4 Diversity verification samples Before live deployment we collected short public samples from every provider to confirm that each source can be downloaded and converted into a regular numeric series. The verification covered 17 datasets, 11 domains, and 2,672 representative observations in total; every source in Table 4 formed a usable time series in this check. Table 5 reports the resulting per-dataset registry windows and descriptive statistics; those numbers are verification slices, not fixed train/test sizes for the live leaderboard. These statistics are generated from the canonical observation rows produced by the data pipeline, not from manual measurements. For each dataset, the diversity-report script selects the configured target variable (or a documented display variable for visualization), chooses one representative entity or aggregate, deduplicates ob- servations by dataset, entity, variable, and timestamp, and then computes푁, mean, range, and standard deviation on the selected target values. The underlying verification slices come from source- dependent sample collection runs rather than from one identical calendar period imposed on every domain-frequency pair: high-rate feeds use short recent windows, backfillable daily or hourly sources use multi-day verification windows, and monthly or annual sources use their available historical records. This design lets the verifi- cation check whether each source is parseable and behaviorally distinct, while leaving the live benchmark free to keep collecting future observations. C.5 Dataset inventory and verification statistics Table 6 summarizes the behavioral patterns seen in the verification charts. They illustrate why the registry mixes smooth environmen- tal signals, volatile financial series, sparse attention counts, and event-driven streams under one benchmark. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 4: Background and public access information for each benchmark dataset. Endpoints are the base URLs used for collection; query parameters depend on entity, time window, and variables. Covariates may be omitted in univariate runs. We use the following abbreviations in subsequent tables and figures: BTC for Binance BTCUSDT, PM2.5 for Open-Meteo air quality, Quake for USGS earthquake aggregates, Potomac for USGS Potomac discharge, Water for NOAA CO-OPS water level, Wave for NOAA NDBC buoy observations, T2M for NASA POWER meteorology, KSFO for NWS KSFO observations, Temp2m for Open-Meteo weather, and Wiki for Wikimedia pageviews. DatasetDomainEntity & regionPrimary targetPublic endpointBackground notes Open-Meteo Shanghai weatherweatherShanghai grid (31.23 ◦ N, 121.47 ◦ E) 2 m temperaturehttps://api.open- meteo.com/v1/ forecast Smooth diurnal weather; hu- midity, wind, precipitation co- variates Open-Meteo Shanghai air qualityair qualityShanghai gridPM 2.5 https://air- quality- api.open- meteo.com/v1/air- quality Pollutant spikes; PM 10 , NO 2 , CO covariates NASA POWER Shanghai meteorologyweather-energy Shanghai pointAir temperature (T2M)https://power.larc.nasa.gov/api/ temporal/hourly/point Energy-orientedmeteorol- ogy; evaluated daily despite hourly native series USGS Potomac dischargehydrologyUSGS site 01646500, Po- tomac River River dischargehttps://waterservices.usgs.gov/nwis/iv/High-frequency hydrology; gauge height covariate NOAA CO-OPS San Francisco water level oceanTide station 9414290, San Francisco Water level https://api.tidesandcurrents.noaa.gov/ api/prod/datagetter Six-minute coastal measure- ments with tide-like oscilla- tion NOAA NDBC buoy 46013oceanNDBC buoy 46013 Significantwave height https://w.ndbc.noaa.gov/data/ realtime2/46013.txt Marine buoy observations; wind, pressure, temperature covariates GBFS Citi Bike station statustrafficNYC bike-share stationsAvailable bikes per sta- tion https://gbfs.citibikenyc.com/gbfs/en/ station_status.json Shared-mobility supply; ver- ification uses a system mean over sampled stations Binance BTCUSDT (hourly)financeBTCUSDT spot marketClose pricehttps://api.binance.com/api/v3/klinesHourly OHLCV with regime shifts Binance BTCUSDT (one-second)financeBTCUSDT spot marketClose pricehttps://api.binance.com/api/v3/klinesSecond-levelmarketmi- crostructure stress test Binance BTCUSDT (monthly)financeBTCUSDT spot marketClose pricehttps://api.binance.com/api/v3/klines Low-frequency crypto his- tory on a shared asset CoinGecko Bitcoin market chartfinanceBitcoin (market level)USD pricehttps://api.coingecko.com/api/v3/coins/ bitcoin/market_chart Market-cap and volume co- variates Wikimedia Time Series article viewsweb attentionenwiki article Time seriesDaily pageviewshttps://wikimedia.org/api/rest_v1/ metrics/pageviews/per- article/ Sparse daily attention signal NWS KSFO observationsweatherNWS station KSFO, San Francisco Temperature https://api.weather.gov/stations/KSFO/ observations Official station observations; wind, humidity, pressure co- variates NOAA NCEI NYC daily summariesweatherNCEIstation USW00014732,New York City Daily mean tempera- ture https://w.ncei.noaa.gov/access/ services/data/v1 Long-running official daily climate summaries World Bank China macro indicatorsmacro-economy China (country level)GDP (current US$)https://api.worldbank.org/v2/country/ CHN/indicator/ Annual macro series since 1961; population and inflation covariates GDELT climate-change timelinenews events Global query: “climate change” Documentvolume (15 min) https://api.gdeltproject.org/api/v2/doc/ doc Event-attentionstream; scored at daily frequency USGS global earthquake aggregatesdisaster eventsGlobal earthquake cata- log Hourlyearthquake count https://earthquake.usgs.gov/ earthquakes/feed/v1.0/summary/ all_week.geojson Sparse events bucketed by UTC hour Table 6: Representative dynamics observed in diversity veri- fication samples (June 2026). Domain groupObserved pattern in verification samples Weather & air qualityDiurnal structure (Open-Meteo, NWS) and pollutant spikes (PM 2.5 ) Weather-energySmooth exogenous meteorology suitable as covariate- rich context Hydrology & ocean Slower physical dynamics; sub-hourly coastal tides and buoy waves Urban mobilitySystem-level bike availability rises/falls with commut- ing demand Finance Heavy-tailed movement from 1 s to monthly BTC series; CoinGecko adds market-level context Web & macroSparse daily pageviews and slow annual GDP growth with long history News & disaster events Bursty document volume and hourly earthquake counts from sparse events C.6 Per-dataset verification charts Figure 12 shows the verification-slice target trajectories used to summarize the dynamic patterns in Table 6. Each panel corresponds to one registry dataset; together they illustrate the diversity of ob- served temporal behavior. These are not fixed test windows but short public samples used to check that each source can be parsed into a regular numeric series. For the Citi Bike verification chart, each point averages available bikes across 50 stations before con- necting the sequence; live evaluation can instead track individual stations or GBFS systems using the same public feed. C.7 Data attribution and usage We gratefully acknowledge Open-Meteo, NASA POWER, USGS, NOAA, GBFS/Citi Bike, Binance, CoinGecko, Wikimedia Foun- dation, World Bank Open Data, and GDELT as the public data LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 5: Per-dataset registry and verification statistics. Data freq. and Eval freq. are native series frequency and recommended scoring frequency. Hist. and Hor. are history and forecast horizon in native-frequency steps. Type distinguishes directly reported time series from event-derived time series.푁isrow_countin the diversity verification slice (not a fixed train/test size). Mean, Std, and Range summarize the selected target variable in that slice. DatasetDomainData freq. Eval freq. TargetHist. Hor. Type푁 MeanStdRange Open-Meteo weatherweather1h1htemperature_2m33624 direct3226.972.74624.0–33.7 Open-Meteo air qualityair_quality1h1hpm2_533624 direct3250.6822.3827.1–99.4 NASA POWER meteorologyweather_energy 1h1dT2M33624 direct11223.623.67917.4–30.9 USGS Potomac waterhydrology15min1husgs_000609624 direct2759,824652.98,770–11,100 NOAA CO-OPS water levelocean6min1hwater_level24060 direct2401.0120.558 −0.042–1.84 NOAA NDBC buoyocean10min1hwave_height14472 direct762.0340.1871.8–2.4 GBFS Citi Bike statustraffic15min15minnum_bikes_available964 direct69.4670.4629.26–10.5 Binance BTCUSDT hourlyfinance1h1hclose9624 direct2465,3301,45862,178–67,295 Binance BTCUSDT one-second finance1s1sclose90060 direct99961,66750.0961,550–61,772 Binance BTCUSDT monthlyfinance1mo1moclose6012 direct9740,95731,9813,434–115,764 CoinGecko Bitcoin marketfinance1h1hprice_usd9624 direct28965,3351,39761,557–67,327 Wikimedia pageviewsweb_attention1d1dpageviews307 direct6277.258.13187–339 NWS KSFO observationsweather1h1htemperature9624 direct31315.672.99912.0–21.1 NOAA NCEI daily summariesweather1d1ddaily_avg_temperature307 direct424.201.60221.7–25.6 World Bank China macromacro_economy 1y1ygdp_current_usd405 direct64 3.65e12 5.63e12 4.73e10–1.87e13 GDELT climate timelinenews_events15min1ddocument_volume67296 event-derived791.2490.3480.701–2.347 USGS earthquake aggregatesdisaster_events1h1dearthquake_count16824 event-derived2410.714.0674–23 Table 7: Per-dataset verification variability used to support the dynamic-richness criterion in Figure 4. CV is computed on the selected target variable in the verification slice asstd/|mean|. These short slices illustrate temporal behavior but are not fixed train/test windows. DatasetDomain푁CV Observed dynamic pattern Open-Meteo weatherweather320.102 Smooth hourly diurnal temperature cycle Open-Meteo air qualityair_quality320.442 Pollutant variation with short spikes NASA POWER meteorologyweather_energy 1120.156 Smooth exogenous meteorology over several days USGS Potomac waterhydrology2750.066 High-frequency river discharge with slow physical variation NOAA CO-OPS water levelocean2400.551 Sub-hourly tidal oscillation with values around zero NOAA NDBC buoyocean760.092 Marine wave-height fluctuations with meteorological covariates GBFS Citi Bike statustraffic60.049 Short mobility-supply sequence averaged over sampled stations Binance BTCUSDT hourlyfinance240.022 Intraday financial movement Binance BTCUSDT one-second finance999 8.1×10 −4 Second-level market microstructure with small relative variation over the sampled window Binance BTCUSDT monthlyfinance970.781 Long-horizon crypto regime shifts CoinGecko Bitcoin marketfinance2890.021 Market-level hourly Bitcoin price movement Wikimedia pageviewsweb_attention60.210 Sparse daily public-attention counts NWS KSFO observationsweather3130.191 Station weather with diurnal structure and irregular updates NOAA NCEI daily summariesweather40.066 Daily official climate summaries in a short verification slice World Bank China macromacro_economy641.545 Long-term annual macroeconomic trend GDELT climate timelinenews_events790.279 Bursty event-attention volume aggregated from documents USGS earthquake aggregatesdisaster_events240.380 Sparse hourly disaster-event counts providers summarized in Table 4. Users operating a live deployment should respect each provider’s terms of use, attribution require- ments, and request-rate limits. D Detailed Forecasting results For completeness, we report per-dataset performance in Tables 8, 9, and 10, which present the RMSE, MAPE, and CRPS of every model in the frozen online benchmark snapshot on each dataset. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. TS-Bench diversity charts17 datasets open meteo shanghai hourly - temperature 2mn=32 20 25 30 35 00:0015:3007:00 mean 26.97 range 24 to 33.72026-06-03 to 2026-06-04 open meteo shanghai air quality hourly - pm2 5n=32 20 40 60 80 100 00:0015:3007:00 mean 50.68 range 27.1 to 99.42026-06-03 to 2026-06-04 nasa power shanghai hourly - T2Mn=112 15 20 25 30 35 05-2805-3006-01 mean 23.62 range 17.42 to 30.942026-05-28 to 2026-06-01 usgs potomac iv - usgs 00060n=275 8k 9k 10.0k 11.0k 12.0k 08:0019:2506:50 mean 9,823.71 range 8,770 to 11,1002026-06-03 to 2026-06-04 noaa coops sf water level - water leveln=240 -0.5 0 0.5 1 1.5 2 08:0019:5707:54 mean 1.012 range -0.042 to 1.842026-06-03 to 2026-06-04 noaa ndbc 46013 realtime - wave heightn=76 1.8 2 2.2 2.4 08:1019:3006:50 mean 2.034 range 1.8 to 2.42026-06-03 to 2026-06-04 gbfs citibike current - mean num bikes availablen=6 9 9.5 10 10.5 08:3608:5609:15 mean 9.467 range 9.26 to 10.52026-06-04 to 2026-06-04 binance btcusdt 1h - closen=24 62.0k 64.0k 66.0k 68.0k 08:0019:3007:00 mean 65,329.8 range 62,178.1 to 67,294.522026-06-03 to 2026-06-04 binance btcusdt 1s - closen=999 61.5k 61.6k 61.7k 61.8k 07:5208:0108:09 mean 61,667.04 range 61,550 to 61,772.492026-06-10 to 2026-06-10 binance btcusdt 1mo - closen=97 0 50.0k 100k 150k 201820222026 mean 40,957.29 range 3,434.1 to 115,764.082018-06-01 to 2026-06-01 coingecko bitcoin 7d - price usdn=289 60.0k 62.0k 64.0k 66.0k 68.0k 07:5919:5907:58 mean 65,335.14 range 61,556.89 to 67,326.912026-06-03 to 2026-06-04 wikimedia time series hourly - pageviewsn=6 150 200 250 300 350 05-2905-3106-03 mean 277.17 range 187 to 3392026-05-29 to 2026-06-03 nws ksfo observations - temperaturen=313 10 15 20 25 08:0019:5007:40 mean 15.67 range 12 to 21.12026-06-03 to 2026-06-04 noaa ncei nyc daily summaries - daily max tem...n=4 21 22 23 24 25 26 05-2805-2905-31 mean 24.2 range 21.7 to 25.62026-05-28 to 2026-05-31 world bank china annual macro - gdp current usdn=64 0 5e+03B 1e+04B 1.5e+04B 2e+04B 196119922024 mean 3,645,292,602,302.34 range 47,310,737,754.23 to 18,743,803,170,827.201961-01-01 to 2024-01-01 gdelt climate timeline - document volumen=79 0.5 1 1.5 2 2.5 09:3021:0008:30 mean 1.249 range 0.701 to 2.3472026-06-03 to 2026-06-04 usgs earthquake all week hourly - earthquake...n=24 0 5 10 15 20 25 08:0019:3007:00 mean 10.71 range 4 to 232026-06-03 to 2026-06-04 Figure 12: Verification-slice target trajectories for all 17 registry datasets. Each panel plots the locally parsed public sample for one dataset and target variable; panel titles give the dataset identifier and selected target. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 8: Per-dataset RMSE (↓) on the online benchmark. In each column the best available model is in red bold and the second best is in blue underline. ModelBTCPM2.5QuakePotomacWaterWaveT2MKSFOTemp2mWiki Chronos-2483.61411.4202.83019.6750.3180.8921.2931.0781.074 8.420 TiRex497.749 10.8432.882 16.9580.3390.8991.3291.0031.025 9.889 TimesFM-2.5485.40711.1562.81217.5020.2340.8441.3321.0600.9888.711 Toto-1.01.162e+0340.6935.31458.6000.5391.0931.5781.2081.83596.250 Moirai-2.0569.01619.2263.61924.6630.3610.308 0.9090.5681.51113.782 Chronos-Bolt802.83220.3593.91728.4950.634 0.2990.943 0.5781.63033.431 TabPFN-TS716.18319.5863.93427.6560.5180.3071.141 0.5151.53820.472 Sundial1.344e+0319.1844.29824.2350.2940.4101.3130.8901.53920.726 Moving-Average774.68416.5763.26324.9450.7310.8082.1191.2851.70226.906 ETS543.17218.5203.17623.9100.8650.9795.0530.9647.67728.427 ARIMA665.23315.2833.27126.3750.7310.8073.0960.9302.61629.245 Seasonal-Naive847.25018.3343.98328.2090.7550.8731.5001.3291.37353.966 Table 9: Per-dataset MAPE (↓) on the online benchmark. In each column the best available model is in red bold and the second best is in blue underline. ModelBTCPM2.5QuakePotomacWaterWaveT2MKSFOTemp2mWiki Chronos-20.0100.3290.5650.0090.4690.1580.0430.0430.0430.061 TiRex0.0100.3150.5780.0070.5910.1640.0430.0380.0420.057 TimesFM-2.50.0100.3140.4930.0080.283 0.1450.0460.0400.0490.060 Toto-1.00.0110.3330.5090.0100.4260.1710.0340.0310.0400.210 Moirai-2.00.0080.341 0.4830.0090.4370.1780.0310.0310.045 0.046 Chronos-Bolt0.0110.3640.5460.0100.7220.176 0.0300.0310.0450.134 TabPFN-TS0.0100.3480.5560.0090.5470.1790.035 0.0260.0440.066 Sundial0.020 0.2820.5340.0080.6000.2040.0400.0460.0450.073 Moving-Average0.0140.4770.6280.0111.8580.1580.0750.0670.0680.200 ETS0.0100.7130.5850.0101.3400.2130.1760.0440.3210.203 ARIMA0.0140.4390.5800.0121.3780.1550.1000.0380.1150.191 Seasonal-Naive0.0150.5180.6410.0111.8630.1660.0520.0650.0520.361 Table 10: Per-dataset CRPS (↓) on the online benchmark. In each column the best available model is in red bold and the second best is in blue underline. ModelBTCPM2.5QuakePotomacWaterWaveT2MKSFOTemp2mWiki Chronos-20.1120.4450.3450.1390.1660.4450.0590.4380.0920.122 TiRex0.1180.4360.3470.1390.1740.4280.0560.4080.1000.118 TimesFM-2.50.1360.5180.4810.172 0.1470.5050.0710.4990.1110.161 Toto-1.00.010 0.2450.2730.0100.1860.1280.0270.0270.0360.132 Moirai-2.00.0080.260 0.2620.008 0.207 0.124 0.0270.0280.040 0.050 Chronos-Bolt0.0100.2720.2780.0100.3650.1290.0270.0280.0410.111 TabPFN-TS0.0100.3340.3710.0090.3390.1710.036 0.0260.0450.063 Sundial0.0180.2570.2960.0070.1510.1640.0330.0390.0400.054 Moving-Average0.1070.4900.3480.1320.5970.4510.1060.4350.1780.321 ETS0.0990.8290.3480.1110.5750.5160.2790.3010.9030.323 ARIMA0.1170.4570.3380.1170.5140.4580.1740.3450.3560.284 Seasonal-Naive0.1000.5830.5560.1140.6210.5690.0730.4160.1360.731 Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 11: Per-dataset Temporal Stability (↓) on the online benchmark, computed from release-level MSE histories. Cells with fewer than two releases are shown as –. In each column the best available model is in red bold and the second best is in blue underline. ModelBTCPM2.5QuakePotomacWaterWaveT2MKSFOTemp2mWiki Chronos-27.329e+05494.9848.9082.860e+030.2713.7561.75627.0491.953129.300 TiRex6.805e+05463.4528.5752.414e+03 0.3103.4791.65327.4491.788167.781 TimesFM-2.56.407e+05499.8148.5742.685e+030.1373.3451.96427.1911.673133.699 Toto-1.03.767e+068.411e+0326.7279.212e+030.6537.8897.3763.9336.6198.228e+03 Moirai-2.04.540e+05 185.7135.2502.734e+030.1920.088 0.3750.7371.53864.521 Chronos-Bolt6.818e+05 169.7015.2823.085e+030.358 0.0521.2350.9812.238238.712 TabPFN-TS4.069e+05307.8975.9722.749e+030.2880.081 0.928 0.7011.27368.714 Sundial1.307e+06645.449 5.049 2.020e+03 0.1170.1020.6662.2531.50926.124 Moving-Average9.664e+05571.0859.0403.870e+031.3523.5555.08026.0202.8551.196e+03 ETS9.012e+057.217e+038.0412.763e+031.4314.35329.08226.072209.6321.770e+03 ARIMA7.312e+05943.3938.2443.046e+031.0273.51011.48825.72810.2611.427e+03 Seasonal-Naive1.002e+06644.21612.6574.223e+031.3893.7502.02125.6721.8294.084e+03 Table 12: Per-dataset Improvement (↓; more negative is better) on the online benchmark, computed as Kendall휏on release-level MSE histories. Cells with fewer than two releases are shown as –. In each column the best available model is in red bold and the second best is in blue underline. ModelBTCPM2.5QuakePotomacWaterWaveT2MKSFOTemp2mWiki Chronos-20.5600.5880.3980.224-0.033-0.500-0.138-0.0250.5270.704 TiRex0.5650.5450.4290.242-0.010-0.5040.144-0.0160.5470.873 TimesFM-2.50.5310.5480.4310.247-0.051-0.428-0.002-0.0340.5850.479 Toto-1.0-0.3090.059-0.2730.0720.022-0.0850.3440.1230.0221.000 Moirai-2.0-0.355-0.013-0.3510.132-0.028 -0.639-0.048 -0.195-0.385-1.000 Chronos-Bolt0.0900.307-0.3710.1510.014-0.098-0.048-0.143-0.4291.000 TabPFN-TS-0.5050.200 -0.4150.1210.000-0.2370.023-0.086-0.1140.333 Sundial0.5030.066-0.2710.084-0.080-0.221 -0.505-0.134-0.0960.000 Moving-Average0.4580.5200.3510.176-0.321-0.395-0.391-0.0810.5250.423 ETS0.3450.5330.3740.221-0.267-0.476-0.3790.0540.2960.366 ARIMA0.3930.5550.4140.191-0.313-0.464-0.456-0.0790.3640.592 Seasonal-Naive0.4840.4590.3810.134 -0.329-0.419-0.036-0.0740.4910.477 Table 13: Overall zero-shot standing on the online benchmark aggregated over all datasets. For each metric we report Average Rank (↓), Win Rate (↑), and Elo (↑) computed from the per-dataset values. In each row the best available model is in red bold and the second best is in blue underline. MetricChr2TiRexTFMToto1Moi2ChrBTabPFNSundialARIMAETSMovAvgSNaive Rank (RMSE)↓3.804.00 3.3010.704.707.105.806.407.458.007.759.00 Win Rate (RMSE)↑0.745 0.727 0.7910.1180.6640.4450.5640.5090.4140.3640.3860.273 Elo (RMSE)↑12251194 1254637121894711611070834789867804 Rank (MAPE)↓4.904.303.855.05 3.706.155.106.458.709.809.6510.35 Win Rate (MAPE)↑0.6450.7000.7410.632 0.7550.5320.6270.5050.3000.2000.2140.150 Elo (MAPE)↑1163125111391055 125310641177998846650739666 Rank (CRPS)↓7.257.059.202.952.303.904.703.158.709.459.3510.00 Win Rate (CRPS)↑0.4320.4500.2550.8230.8820.7360.6640.8050.3000.2320.2410.182 Elo (CRPS)↑93010037801325 1470126512431358726623666611 Rank (Stability)↓6.905.805.3010.50 2.404.802.803.608.409.608.809.10 Win Rate (Stability)↑0.4640.5640.6090.136 0.8730.6550.836 0.7640.3270.2180.2910.264 Elo (Stability)↑9899921059648 1423107314101407764625814797 Rank (Improvement)↓8.509.609.207.45 2.956.004.804.406.606.205.906.40 Win Rate (Improvement)↑0.3180.2180.2550.414 0.8230.5450.6550.6910.4910.5270.5550.509 Elo (Improvement)↑849753838776 13389901161124298110591036977 LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 14: Zero-shot rankings on the online benchmark aggregated by domain. For each metric we report Average Rank (↓), Win Rate (↑), and Elo (↑) computed from the per-dataset values within each domain. In each row the best available model is in red bold and the second best is in blue underline. GroupMetricChr2TiRexTFMToto1Moi2ChrBTabPFNSundialARIMAETSMovAvgSNaive Air Quality Rank (RMSE)↓3.001.002.0012.009.0011.0010.008.004.007.005.006.00 Win Rate (RMSE)↑0.8181.0000.9090.0000.2730.0910.1820.3640.7270.4550.6360.545 Elo (RMSE)↑145117861601218684397548807132693111911059 Rank (MAPE)↓4.003.002.005.006.008.007.001.009.0012.0010.0011.00 Win Rate (MAPE)↑0.7270.8180.909 0.6360.5450.3640.4551.0000.2730.0000.1820.091 Elo (MAPE)↑131214381588119210688179501783695206553397 Rank (CRPS)↓7.006.0010.001.003.004.005.002.008.0012.009.0011.00 Win Rate (CRPS)↑0.4550.5450.182 1.0000.8180.7270.6360.9090.3640.0000.2730.091 Elo (CRPS)↑941106256017831444131211781606823208683399 Finance Rank (RMSE)↓1.003.002.0011.005.009.007.0012.006.004.008.0010.00 Win Rate (RMSE)↑1.0000.8180.9090.0910.6360.2730.4550.0000.5450.7270.3640.182 Elo (RMSE)↑177714411603400119068893821610661321819540 Rank (MAPE)↓4.004.004.007.501.007.504.0012.009.504.009.5011.00 Win Rate (MAPE)↑0.727 0.7270.7270.409 1.0000.4090.7270.0000.2270.7270.2270.091 Elo (MAPE)↑130013011301914172890913002286511303652414 Rank (CRPS)↓9.0011.0012.003.001.003.003.005.0010.006.008.007.00 Win Rate (CRPS)↑0.2730.0910.0000.818 1.0000.8180.8180.6360.1820.5450.3640.455 Elo (CRPS)↑680410218145017661445144711995501077814945 Hazards Rank (RMSE)↓2.003.001.0012.007.008.009.0011.006.004.005.0010.00 Win Rate (RMSE)↑0.909 0.818 1.0000.0000.4550.3640.2730.0910.5450.7270.6360.182 Elo (RMSE)↑159514491782217938823694392105613191190544 Rank (MAPE)↓7.008.002.00 3.001.005.006.004.009.0010.0011.0012.00 Win Rate (MAPE)↑0.4550.3640.9090.818 1.0000.6360.5450.7270.2730.1820.0910.000 Elo (MAPE)↑9358281603 14531777118210551317696548399207 Rank (CRPS)↓6.007.0011.002.001.003.0010.004.005.008.508.5012.00 Win Rate (CRPS)↑0.5450.4550.0910.909 1.0000.8180.1820.7270.6360.3180.3180.000 Elo (CRPS)↑106394241016011778145255313121175750752212 Hydrology Rank (RMSE)↓3.001.002.0012.006.0011.009.005.008.004.007.0010.00 Win Rate (RMSE)↑0.8181.0000.909 0.0000.5450.0910.2730.6360.3640.7270.4550.182 Elo (RMSE)↑145617841599216106439169011878121313944544 Rank (MAPE)↓5.001.002.50 8.005.008.005.002.5012.008.0010.5010.50 Win Rate (MAPE)↑0.6361.0000.8640.3640.6360.3640.6360.8640.0000.3640.1360.136 Elo (MAPE)↑117617631500831117582611791509 223825496497 Rank (CRPS)↓10.5010.5012.004.502.004.503.001.008.006.009.007.00 Win Rate (CRPS)↑0.1360.1360.0000.6820.9090.6820.8181.0000.3640.5450.2730.455 Elo (CRPS)↑48348422312531592 1250144417828041069676941 Ocean Rank (RMSE)↓6.007.004.00 9.504.004.504.003.007.2511.507.759.50 Win Rate (RMSE)↑0.5450.4550.727 0.2270.7270.6820.7270.8180.4320.0450.3860.227 Elo (RMSE)↑100292411657051251125212681304990439941759 Rank (MAPE)↓3.75 5.501.004.506.008.007.509.006.0010.507.259.00 Win Rate (MAPE)↑0.7500.591 1.0000.6820.5450.3640.4090.2730.5450.1360.4320.273 Elo (MAPE)↑12121061176210809588598587541066611967812 Rank (CRPS)↓5.005.005.503.50 3.505.506.003.009.0010.509.5012.00 Win Rate (CRPS)↑0.6360.6360.5910.7730.7730.5910.5450.8180.2730.1360.2270.000 Elo (CRPS)↑1158118010721363 1383120111241361775522725136 Weather Rank (RMSE)↓5.335.005.339.672.674.333.335.339.0010.0010.008.00 Win Rate (RMSE)↑0.6060.6360.6060.212 0.8480.6970.788 0.6060.2730.1820.1820.364 Elo (RMSE)↑1125117211687201306111112451075776668706928 Rank (MAPE)↓5.834.677.672.333.673.333.007.009.1711.0010.679.67 Win Rate (MAPE)↑0.5610.6670.394 0.8790.7580.7880.8180.4550.2580.0910.1210.212 Elo (MAPE)↑114312658841518128113091411951681426490641 Rank (CRPS)↓8.007.009.331.672.673.173.673.839.6710.0010.009.00 Win Rate (CRPS)↑0.3640.4550.242 0.9390.8480.8030.7580.7420.2120.1820.1820.273 Elo (CRPS)↑74185059617601575148214181420536494504623 Web Rank (RMSE)↓1.003.002.0012.004.0010.005.006.009.008.007.0011.00 Win Rate (RMSE)↑1.0000.8180.9090.0000.7270.1820.6360.5450.2730.3640.4550.091 Elo (RMSE)↑177214411602215131655011951071683821947387 Rank (MAPE)↓4.002.003.0011.001.007.005.006.008.0010.009.0012.00 Win Rate (MAPE)↑0.7270.9090.8180.091 1.0000.4550.6360.5450.3640.1820.2730.000 Elo (MAPE)↑131616001447407177493511931069822546684208 Rank (CRPS)↓6.005.008.007.001.004.003.002.009.0011.0010.0012.00 Win Rate (CRPS)↑0.5450.6360.3640.455 1.0000.7270.8180.9090.2730.0910.1820.000 Elo (CRPS)↑106311838189491787130214401596698402552209 Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 15: Zero-shot rankings on the online benchmark aggregated by sampling frequency. For each metric we report Average Rank (↓), Win Rate (↑), and Elo (↑) computed from the per-dataset values within each frequency group. In each row the best available model is in red bold and the second best is in blue underline. GroupMetricChr2TiRexTFMToto1Moi2ChrBTabPFNSundialARIMAETSMovAvgSNaive 6min Rank (RMSE)↓3.004.00 1.007.005.008.006.002.009.5012.009.5011.00 Win Rate (RMSE)↑0.8180.727 1.0000.4550.6360.3640.5450.9090.2270.0000.2270.091 Elo (RMSE)↑14361310 1780944118682010621602 626208625400 Rank (MAPE)↓4.006.00 1.002.003.008.005.007.0010.009.0011.0012.00 Win Rate (MAPE)↑0.7270.545 1.0000.909 0.8180.3640.6360.4550.1820.2730.0910.000 Elo (MAPE)↑13101068 1783159914468171182947549692399206 Rank (CRPS)↓3.004.00 1.005.006.008.007.002.00 9.0010.0011.0012.00 Win Rate (CRPS)↑0.8180.727 1.0000.6360.5450.3640.4550.9090.2730.1820.0910.000 Elo (CRPS)↑14361314 1779119010678189481602 693547399206 10min Rank (RMSE)↓9.0010.007.0012.003.00 1.002.004.005.0011.006.008.00 Win Rate (RMSE)↑0.2730.1820.4550.0000.818 1.0000.909 0.7270.6360.0910.5450.364 Elo (RMSE)↑6855599412141451 17841601131711843991054811 Rank (MAPE)↓3.505.00 1.007.009.008.0010.0011.002.0012.003.506.00 Win Rate (MAPE)↑0.7730.636 1.0000.4550.2730.3640.1820.0910.9090.0000.7730.545 Elo (MAPE)↑13791189 1778936692816560405159521113751064 Rank (CRPS)↓7.006.0010.002.001.003.005.004.009.0011.008.0012.00 Win Rate (CRPS)↑0.4550.5450.1820.909 1.0000.8180.6360.7270.2730.0910.3640.000 Elo (CRPS)↑943106255716021781145411781306689401818208 15min Rank (RMSE)↓3.00 1.002.0012.006.0011.009.005.008.004.007.0010.00 Win Rate (RMSE)↑0.818 1.0000.9090.0000.5450.0910.2730.6360.3640.7270.4550.182 Elo (RMSE)↑1456 17841599216106439169011878121313944544 Rank (MAPE)↓5.00 1.002.508.005.008.005.002.5012.008.0010.5010.50 Win Rate (MAPE)↑0.636 1.0000.8640.3640.6360.3640.6360.8640.0000.3640.1360.136 Elo (MAPE)↑1176 17631500831117582611791509223825496497 Rank (CRPS)↓10.5010.5012.004.502.00 4.503.001.008.006.009.007.00 Win Rate (CRPS)↑0.1360.1360.0000.6820.9090.6820.818 1.0000.3640.5450.2730.455 Elo (CRPS)↑483484223125315921250144417828041069676941 1h Rank (RMSE)↓3.673.673.5010.674.836.836.007.837.177.508.008.33 Win Rate (RMSE)↑0.7580.7580.7730.1210.6520.4700.5450.3790.4390.4090.3640.333 Elo (RMSE)↑116512001211686117710251117989870809829923 Rank (MAPE)↓5.424.835.173.753.175.084.336.339.179.8310.4210.50 Win Rate (MAPE)↑0.5980.6520.6210.7500.8030.6290.6970.5150.2580.1970.1440.136 Elo (MAPE)↑112311921027 13501254118112611013757603597642 Rank (CRPS)↓7.677.5010.17 1.832.173.254.833.758.679.429.259.50 Win Rate (CRPS)↑0.3940.4090.167 0.9240.8940.7950.6520.7500.3030.2350.2500.227 Elo (CRPS)↑810860601 16851574 141312001356670588608637 1D Rank (RMSE)↓1.003.002.00 12.004.0010.005.006.009.008.007.0011.00 Win Rate (RMSE)↑ 1.0000.8180.9090.0000.7270.1820.6360.5450.2730.3640.4550.091 Elo (RMSE)↑177214411602215131655011951071683821947387 Rank (MAPE)↓4.002.00 3.0011.00 1.007.005.006.008.0010.009.0012.00 Win Rate (MAPE)↑0.7270.9090.8180.091 1.0000.4550.6360.5450.3640.1820.2730.000 Elo (MAPE)↑131616001447407 177493511931069822546684208 Rank (CRPS)↓6.005.008.007.00 1.004.003.002.009.0011.0010.0012.00 Win Rate (CRPS)↑0.5450.6360.3640.455 1.0000.7270.8180.9090.2730.0910.1820.000 Elo (CRPS)↑10631183818949 1787130214401596698402552209 LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 16: Zero-shot rankings on the online benchmark aggregated by forecast horizon. For each prediction length and metric, models are first ranked within each dataset and then ranked by their average rank across datasets sharing that prediction length (best= 1). In each row the best available model is in red bold and the second best is in blue underline. Pred. lengthMetricChr2TiRexTFMToto1MoiBoltTabPFNSundialARIMAETSMovAvgSNaive 7 Rank (RMSE)132124756109811 Rank (MAPE)325817469111012 Rank (CRPS)65872 4319111012 24 Rank (RMSE)231124659781011 Rank (MAPE)546218371091112 Rank (CRPS)7692 14538101211 60 Rank (RMSE)341758629121011 Rank (MAPE)471238561091112 Rank (CRPS)35146872 9101112 72 Rank (RMSE)810712132451169 Rank (MAPE)710145961121238 Rank (CRPS)761021345911812 Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 Chr2 Wiki (1D pageviews, H = 30, L = 7) ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.00 1.25 1.50 1.75 Wave (10 min buoy wave height, H = 144, L = 72) ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 TiRex ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.0 1.5 2.0 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 TFM ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 Toto1 ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 4 2 0 2 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 Moi2 ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 ChrB ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.25 1.50 1.75 2.00 2.25 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 TabPFN-TS ContextActualPred. Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred. May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 Sundial ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 SNaive ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 MovAvg ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 ARIMA ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 May 28Jun 07Jun 17Jun 28Jul 03 200 300 400 ETS ContextActualPred.p10p90 Jul 04 09:5014:0018:1022:20Jul 05 01:4004:4007:40 1.2 1.4 1.6 1.8 ContextActualPred.p10p90 Figure 13: Forecasting visualizations on Wiki (left column) and Wave (right column) for all baselines. Each row corresponds to one baseline. The left side of the dash line shows the context windows, and the right side shows the forecasting horizons. LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 Chr2 PM2.5 (1h PM2.5, L = 96, H = 24) ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 Potomac (15 min river flow, L = 45, H = 24) ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 TiRex ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 TFM ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 Toto1 ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 Moi2 ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 ChrB ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 TabPFN-TS ContextActualPred. 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred. 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 Sundial ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 SNaive ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 MovAvg ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 ARIMA ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 07.15 19:0007.17 02:0007.18 09:0007.19 18:0007.20 13:00 20 40 60 80 100 ETS ContextActualPred.p10-p90 07.20 14:5518:2521:5507.21 01:5506:40 2600 2800 3000 ContextActualPred.p10-p90 Figure 14: Forecasting visualizations on PM 2.5 (left column) and Potomac (right column) for all baselines. Each row corresponds to one baseline. The left side of the dash line shows the context windows, and the right side shows the forecasting horizons. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Figure 15: Pairwise historical ranking under asynchronous model admission. The top panel shows representative resolved releases and model admission times. Models A–D have sufficient shared evidence, whereas Model E joined later and remains below the eligibility thresholds. The lower-left panel shows dataset-balanced pairwise win rates푊 푖푗 ; unavailable comparisons involving Model E are omitted rather than treated as ties. The lower-right panel aggregates eligible pairwise comparisons into model-level scores and official ranks. Numerical values are illustrative. E Pairwise Historical Ranking Models may enter a live leaderboard at different times. Directly comparing their average errors since admission can therefore be misleading: an early model may have experienced both volatile and calm periods, whereas a recently admitted model may have been evaluated only under the latest conditions. We address this cohort mismatch by comparing each pair of models only on releases that they completed in common. A release is one resolved causal forecasting task for a particu- lar dataset and future target window. At issue time, all admitted models receive the same historical context, sampling frequency, and forecast horizon. Their forecasts are frozen before the future target becomes available. Once the complete target window has been observed, the forecasts are scored and the task becomes a resolved release. Models evaluated on the same release therefore share the same context, target values, timestamps, normalization statistics, and metric implementation. Figure 15 connects the three stages of the ranking procedure. In the top panel, the horizontal axis represents historical time up to cutoff푇. Each colored dot denotes a representative resolved dataset– target-window release, with colors indicating different datasets. The dots are illustrative; the actual ranking uses every resolved release. The blue triangle on each model row marks its admission time. A model is evaluated only on releases issued after admission. Models A–D entered sufficiently early to accumulate overlapping evaluations from at least 30 releases, 5 datasets, and 7 days. Model E entered later and has only 12 shared releases from 3 datasets over 2 days. Its overlap is therefore insufficient for an official comparison. The lower-left panel records the pairwise win rate푊 푖푗 of row model 푖against column model푗. For example,푊 퐴퐵 =0.62 means that Model A obtains a dataset-balanced win rate of 62% against Model B. The reverse comparison is푊 퐵퐴 =0.38. Green cells indicate values above 0.5, red cells indicate values below 0.5, and gray cells are unavailable. In particular, missing comparisons involving Model E are omitted; they are not assigned a neutral value of 0.5. The lower-right panel averages each model’s eligible pairwise win rates with equal weight per opponent. This produces the model-level score푊 푖 used for ranking. Models A–D receive official ranks, while Model E remains Provisional until it accumulates sufficient shared evidence. E.1 Shared releases and pair eligibility Let푡 푟 denote the end time of the target window for release푟. At historical cutoff푇 , define 푅 푖 (푇)= 푟 : 푡 푟 ≤ 푇, MSE 푖푟 and CRPS 푖푟 are valid (9) as the valid release history of model푖. The shared release history of models 푖 and 푗 is 푅 푖푗 (푇)= 푅 푖 (푇)∩ 푅 푗 (푇).(10) LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY Thus, a release completed by only one member of the pair does not enter their comparison. Releases issued before the later model’s admission are automatically excluded. Let푑 푟 be the dataset associ- ated with release 푟 . The shared dataset set is 퐷 푖푗 (푇)= 푑 푟 : 푟 ∈ 푅 푖푗 (푇) ,(11) and the temporal coverage of the pair is span 푅 푖푗 (푇) = max 푟∈푅 푖푗 (푇) 푡 푟 − min 푟∈푅 푖푗 (푇) 푡 푟 .(12) A pair is eligible only if |푅 푖푗 (푇)| ≥ 30, |퐷 푖푗 (푇)| ≥ 5,span 푅 푖푗 (푇) ≥ 7 days. (13) These requirements prevent a model from receiving an official comparison based on a short period, a small number of releases, or a narrow selection of datasets. A pair that fails any requirement is treated as unavailable rather than as a tie. E.2Comparing point and probabilistic forecasts The leaderboard includes both probabilistic models and models that produce only point forecasts. Both model types are evaluated using MSE and CRPS under the same context-only normalization. Let푧 푟ℎ be the normalized target at horizon stepℎ. A probabilistic model provides a predictive mean 휇 푖푟ℎ and quantiles 푄 푖푟ℎ (휏) for Q=0.1, 0.2, . . ., 0.9.(14) Its MSE is computed from the predictive mean, while its CRPS is approximated using the quantile forecasts: CRPS 푖푟 = 2 퐻 푟 |Q| 퐻 푟 ∑︁ ℎ=1 ∑︁ 휏∈Q 휌 휏 ( 푧 푟ℎ −푄 푖푟ℎ (휏) ) ,(15) where 휌 휏 (푢)=푢 ( 휏 − I푢< 0 ) (16) is the pinball loss. A point-only model provides one forecast ˆ 푧 푖푟ℎ per horizon step. We represent it as a degenerate predictive distribution by assigning the point forecast to every required quantile: 푄 point 푖푟ℎ (휏)= ˆ 푧 푖푟ℎ , ∀휏 ∈ Q.(17) This conversion does not add artificial uncertainty: the model assigns all predictive mass to its point forecast. Because the quantile grid is symmetric around 0.5, substituting Eq.(17)into Eq.(15)gives CRPS point 푖푟 = 1 퐻 푟 퐻 푟 ∑︁ ℎ=1 | 푧 푟ℎ − ˆ 푧 푖푟ℎ | .(18) Hence, the CRPS of a point-only model reduces to its normalized MAE. Probabilistic models are evaluated on both the location and dispersion of their predictive distributions, while point models are evaluated as zero-uncertainty distributions. Since CRPS is defined for both ordinary and degenerate predictive distributions, their scores remain directly comparable. Model output type and any benchmark-side procedure used to construct quantiles are fixed be- fore the target is observed. Native quantiles are used when available; a declared point-only output is converted using Eq. (17). E.3 Release-level pairwise score Both MSE and CRPS are lower-is-better. For two error values푎and 푏, let 푐(푎,푏)=         1, 푎< 푏, 0.5, 푎=푏, 0, 푎> 푏. (19) On shared release 푟 , the score of model 푖 against model 푗 is 푠 푖푗푟 = 1 2 푐 MSE 푖푟 , MSE 푗푟 +푐 CRPS 푖푟 , CRPS 푗푟 .(20) The two metrics receive equal weight. A model receives푠 푖푗푟 =1 if it wins on both metrics and푠 푖푗푟 =0 if it loses on both. A split decision gives 0.5; a win and a tie give 0.75; and a loss and a tie give 0.25. Therefore, 푠 푖푗푟 ∈ 0, 0.25, 0.5, 0.75, 1, 푠 푗푖푟 = 1−푠 푖푗푟 .(21) The same rule applies to point–point, probabilistic–probabilistic, and point–probabilistic comparisons. The MSE component com- pares central forecast accuracy, while the CRPS component com- pares the corresponding predictive distributions. E.4 Dataset-balanced pairwise win rate Datasets resolve releases at different rates. Pooling all releases di- rectly would allow high-frequency streams to dominate the ranking. We therefore average scores within each shared dataset before av- eraging across datasets. Let 푅 푖푗,푑 (푇)= 푟 ∈ 푅 푖푗 (푇) : 푑 푟 =푑 (22) be the releases shared by models푖and푗for dataset푑. Their dataset-specific win rate is 푊 푖푗,푑 (푇)= 1 |푅 푖푗,푑 (푇)| ∑︁ 푟∈푅 푖푗,푑 (푇) 푠 푖푗푟 .(23) The dataset-balanced pairwise win rate is 푊 푖푗 (푇)= 1 |퐷 푖푗 (푇)| ∑︁ 푑∈퐷 푖푗 (푇) 푊 푖푗,푑 (푇).(24) Every shared dataset therefore receives equal weight, regardless of how many releases it produces. Complementarity of the release- level score implies 푊 푗푖 (푇)= 1−푊 푖푗 (푇).(25) This is visible in the lower-left panel of Figure 15: for example, 푊 퐴퐵 =0.62 and푊 퐵퐴 =0.38. A value above 0.5 means that model푖 wins more often than model푗after dataset balancing; it is a ranking score, not a statistical significance test. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Chronos-2 TiRex TimesFM-2.5 Moirai-2.0 TabPFN-TS Chronos-Bolt Sundial Seasonal-Naive Moving-Average Toto-1.0 ARIMA ETS 0.671 0.668 0.609 0.580 0.560 0.524 0.508 0.385 0.378 0.375 0.375 0.368 Figure 16: Overall model ranking in the latest leaderboard snapshot. Bars show the dataset-balanced pairwise win rate aggregated over eligible metric-specific comparisons, historical releases, and opponents. Chronos-2 achieves the highest overall score, followed closely by TiRex. Higher values are better. TimesFM-2.5 TiRex Chronos-2 TabPFN-TS Moirai-2.0 Chronos-Bolt Sundial Seasonal-Naive Moving-Average ETS ARIMA Toto-1.0 0.680 0.673 0.668 0.618 0.576 0.530 0.494 0.392 0.381 0.371 0.370 0.249 Figure 17: MSE-based ranking within the point-forecast track. Bars report dataset-balanced pairwise win rates over eligible historical releases and opponents. TimesFM-2.5 achieves the highest score. Higher values are better. Chronos-2 TiRex Moirai-2.0 TimesFM-2.5 Chronos-Bolt Toto-1.0 Sundial TabPFN-TS 0.634 0.615 0.546 0.456 0.448 0.445 0.430 0.426 Figure 18: CRPS-based ranking restricted to the eight TSFMs whose model families natively support probabilistic forecasting through quantiles or forecast samples. Chronos-2 achieves the highest score. Higher values are better. E.5 Model-level ranking and status Let푁 푖 (푇)be the set of opponents for which model푖satisfies Eq.(13). The model-level score is the macro-average 푊 푖 (푇)= 1 |푁 푖 (푇)| ∑︁ 푗∈푁 푖 (푇) 푊 푖푗 (푇).(26) Each eligible opponent receives equal weight. Thus, an opponent with a longer shared history does not dominate the final score merely because the pair has more releases. In Figure 15, Model A has 푊 퐴 = 푊 퐴퐵 +푊 퐴퐶 +푊 퐴퐷 3 = 0.62+ 0.55+ 0.58 3 = 0.583,(27) LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation ModelsConference acronym ’X, June 03–05, 2018, Woodstock, NY which gives it rank 1. Models are ordered by decreasing푊 푖 (푇). We construct an undirected comparison graph whose vertices are models and whose edges are eligible model pairs. A model receives an official rank only if (1) it has at least three eligible opponents; and (2) it belongs to the main connected component of the graph. Models that fail either condition remain Provisional. Missing pairs are omitted from Eq.(26); they are never imputed as 0.5. This is why Model E in the figure has neither a model-level score nor a rank, despite having completed some releases. Alongside the rank, the leaderboard reports the number of eligi- ble opponents, shared releases, shared datasets, and covered time span. These fields expose the amount of evidence supporting each result and distinguish an established ranking from a provisional one. F Overall Model Ranking Figures 16–18 provide three complementary views of the latest leaderboard snapshot up to cutoff time푇. Each score is a dataset- balanced pairwise win rate, averaged with equal weight over eligible opponents and historical releases; higher values indicate better relative performance. Figure 16 shows the overall eligibility-aware aggregation, where Chronos-2 ranks first (0.671), narrowly ahead of TiRex (0.668), followed by TimesFM-2.5 (0.609). To separate point accuracy from distributional forecast quality, Figure 17 ranks models using MSE within the point-forecast track. TimesFM-2.5 achieves the highest MSE-based win rate (0.680), fol- lowed by TiRex (0.673) and Chronos-2 (0.668). Figure 18 restricts the comparison to the eight TSFMs whose model families natively support probabilistic forecasts through quantiles or samples. Under CRPS, Chronos-2 ranks first (0.634), followed by TiRex (0.615) and Moirai-2.0 (0.546). Together, the three views show that point-forecast accuracy and probabilistic forecast quality need not produce the same ordering: TimesFM-2.5 leads the MSE track, whereas Chronos-2 leads the CRPS track and the overall aggregation. TiRex remains consistently competitive across all three views. Because the MSE and CRPS rankings use different eligible model sets and opponent groups, their absolute scores should be interpreted within each figure rather than compared directly across tracks. These rankings summarize relative historical performance and do not by themselves establish statistical significance or long-term rank stability. ,

Instant research discovery