Paper deep dive

TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments

Qingren Yao, Yaxuan Kong, Yuqi Nie, Yichen Li, Stefan Zohren, Anna Vettoruzzo, Qingsong Wen, Ming Jin, Joaquin Vanschoren

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 97

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%

Last extracted: 8/17/2026, 5:24:49 AM

Summary

The paper introduces TimeSage-EV, a live benchmark for evaluating agentic time series analysis in evolving environments. It addresses the limitations of existing static benchmarks by tracking 60 real institutional scenarios across 6 domains (agriculture, climate, energy, finance, healthcare, transportation) with 1,485 scenario-period QA pairs from Feb 2023 to May 2026. The benchmark evaluates Large Language Model (LLM) agents on their ability to maintain temporal validity, use cutoff-aware evidence, and adapt to recurring data releases. It introduces TimeSage-1.0, a self-evolving agent, and reveals significant performance gaps in temporal reasoning and adaptation among frontier models.

Entities (10)

TimeSage-EV · benchmark · 100%TimeSage-1.0 · modelagent · 95%Eindhoven University of Technology · organization · 90%Griffith University · organization · 90%LLM · technology · 90%University of Oxford · organization · 90%ENSO · domain-concept · 85%ChatTime-TSQA · benchmark · 80%LiveBench · benchmark · 80%TimeSeriesGym · benchmark · 80%

Relation Signals (9)

TimeSage-EV → evaluates → LLM

confidence 95% · TimeSage-EV evaluates state identification, data summarization, and outlook reasoning... Experiments with frontier LLM agents...

TimeSage-1.0 → isa → Self-evolving agent

confidence 95% · TimeSage-1.0, a novel self-evolving agent with a reusable analytical skill library

TimeSage-EV → createdby → University of Oxford

confidence 90% · Yaxuan Kong 2,3* ... 2 University of Oxford

TimeSage-EV → createdby → Griffith University

confidence 90% · Ming Jin 5† ... 5 Griffith University

TimeSage-EV → createdby → Eindhoven University of Technology

confidence 90% · Qingren Yao 1* ... 1 Eindhoven University of Technology

TimeSage-EV → coversdomain → ENSO

confidence 85% · Scenario:ENSO Update: Monthly... TimeSage-EV tracks 60 real institutional scenarios across 6 domains... climate

TimeSage-EV → comparesagainst → ChatTime-TSQA

confidence 80% · Existing time series QA benchmarks, such as ChatTime-TSQA... Existing benchmarks only partially cover this setting

TimeSage-EV → comparesagainst → TimeSeriesGym

confidence 80% · TimeSeriesGym (Cai et al., 2025) evaluates LLMs... These settings remain snapshot-based... TimeSage-EV addresses this gap

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Time series analysis in high-stakes domains relies on recurring data releases, where new observations can alter the evidence base and the validity of later conclusions. Existing time series QA benchmarks mostly rely on fixed snapshots, leaving temporal validity and cutoff-aware evidence use unevaluated. We introduce TimeSage-EV, a live benchmark for agentic time series analysis in evolving environments. It tracks 60 real institutional scenarios across 6 domains, comprising 1,485 scenario-period QA pairs from Feb 2023 to May 2026 and spanning monthly, weekly, daily, and irregular release cadences. At each period, large language model (LLM) agents receive time series data and source reports, while the withheld target release provides ground truth. TimeSage-EV evaluates state identification, data summarization, and outlook reasoning. Experiments with frontier LLM agents and TimeSage-1.0, a novel self-evolving agent with a reusable analytical skill library, reveal significant performance gaps across model tiers and recurring failures in temporal validity, exogenous context use, and adaptation. We release TimeSage-EV as a research resource with monthly updates, code, a leaderboard, and failure-mode analyses.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

93,803 characters extracted from source content.

Expand or collapse full text

TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments Qingren Yao 1* , Yaxuan Kong 2,3* , Yuqi Nie, Yichen Li 3 , Stefan Zohren 2 , Anna Vettoruzzo 1 , Qingsong Wen 4 , Ming Jin 5† , Joaquin Vanschoren 1† 1 Eindhoven University of Technology, 2 University of Oxford, 3 VulpiVox Intelligence, 4 Squirrel Ai Learning, 5 Griffith University q.yao@tue.nl; yaxuan.kong@eng.ox.ac.uk; mingjinedu@gmail.com; j.vanschoren@tue.nl DatasetCode Abstract Time series analysis in high-stakes domains relies on recurring data releases, where new observations can alter the evidence base and the validity of later conclusions. Existing time series QA benchmarks mostly rely on fixed snapshots, leaving temporal validity and cutoff- aware evidence use unevaluated. We introduce TimeSage-EV, a live benchmark for agentic time series analysis in evolving environments. It tracks 60 real institutional scenarios across 6 domains, comprising 1,485 scenario-period QA pairs from Feb 2023 to May 2026 and spanning monthly, weekly, daily, and irregular release cadences. At each period, large lan- guage model (LLM) agents receive time series data and source reports, while the withheld tar- get release provides ground truth. TimeSage- EV evaluates state identification, data summa- rization, and outlook reasoning. Experiments with frontier LLM agents and TimeSage-1.0, a novel self-evolving agent with a reusable an- alytical skill library, reveal significant perfor- mance gaps across model tiers and recurring failures in temporal validity, exogenous context use, and adaptation. We release TimeSage-EV as a research resource with monthly updates, code, a leaderboard, and failure-mode analyses. 1 Introduction Across high-stakes domains, from finance and pub- lic health to transportation, time series analysis is not a static prediction task but a recurring work- flow. Analysts must continuously ingest new data releases, update historical context, and synthesize numeric trends with institutional reports. Crucially, each new release shifts the available evidence dis- tribution, requiring strict adherence to timestamp- aware information cutoffs. We formalize this re- curring workflow as agentic time series analysis in evolving environments, requiring Large Language * Equal contribution. † Corresponding author. Models (LLMs) to maintain temporal validity while reasoning over dynamic multimodal evidence. Existing benchmarks only partially cover this setting, with two key limitations. (1) Time se- ries question answering (QA) benchmarks, such as ChatTime-TSQA (Wang et al., 2025), Time- MQA (Kong et al., 2025), and TSRBench (Yu et al., 2026), test temporal reasoning over time series, but usually construct fixed QA pairs over prede- fined time series tasks rather than evolving work- flows. (2) Agent-centric benchmarks evaluate cod- ing (Jimenez et al., 2024), web navigation (Zhou et al., 2024; Mialon et al., 2024), multi-hop re- trieval (Wei et al., 2025), and tool use (Yao et al., 2025), but are not designed around recurring time series analysis. They rarely require agents to track time-indexed evidence, compare new observations with historical context, and synthesize time series with institutional reports. Thus, it remains unclear whether LLM agents can perform such analytical work as the evidence changes over time. Constructing a benchmark for this dynamic set- ting presents three challenges.(1) Each scenario must be drawn from a stable institutional source with a recurring release schedule, ensuring the eval- uation reflects real-time monitoring rather than syn- thetic snapshots. (2) Each target period must im- pose a strict information cutoff. Agents can only access cumulative data and source documents pub- lished prior to the target release, while the target release itself is withheld for ground truth construc- tion and evidence verification. (3) Evaluation can- not be restricted to closed-form choices. To align with practical applications, it must assess both key analytical tasks and human-readable reports. These requirements motivate a benchmark design with timestamp-aware ground truth, traceable evidence, cutoff-valid inputs, and metrics that explicitly sep- arate factual correctness from report faithfulness. To address these challenges, we introduce TimeSage-EV, a live benchmark for agentic time arXiv:2608.14270v1 [cs.AI] 14 Aug 2026 Agent–Environment Interaction Within ttimepoint PublicInstitution Releases Benchmark Construction Evaluation Protocol Recurringpublicsources withTS&documents Cutoff-awarescenario &instancegeneration Agentsanalyzeevolving evidencewitheachperiod Structuredanswers& reportsareauto-scored 500initialsources 6domains 46finalpublishers TS+releaseddocs Agriculture Climate Energy Finance Healthcare Transportation 6domains 60accepted scenarios Scenariospecification Definequestions,entities, context&tags... Periodgeneration Createperiod-tinstances withreleaseschedule Temporalisolation Withholdtarget-period releaseforground-truth& evidenceverification LiveBenchmark 1,485scenario-periodQA 4releasecadences(daily/ weekly/monthly/irregular) 3difficultytiers(easy/medium/ hardtasks) KeypointAccuracy Correctnessofkeyfindings KeypointCoverage Completenessofrequired analyticalkeypoints ReportFaithfulness Faithfulnesstoevidence& documentsources ReportQuality Clarity,coherence...? Rule-basedmethod+LLMJudges FailureModes Reasoning failure Temporal misuses Incomplete reporting Environment Update UpdatePeriodically Observation LLMagent Report (Answer+ Summary) t (TS, docs) t &feedback t-1 Withcode&memory Live t à t+1 Figure 1: Overview of TimeSage-EV, supporting agentic time series tasks and a live leaderboard. TS: time series. series analysis in evolving environments. As shown in Figure 1, each scenario is a recurring task anchored to an institutional source, defined by a fixed domain, release cadence, difficulty tier and set of questions. TimeSage-EV tracks 60 sce- narios across six domains from February 2023 to May 2026, covering agriculture, climate, energy, finance, healthcare, and transportation. The scenar- ios are organized into three difficulty tiers: easy tasks focus on state identification; medium tasks re- quire change detection, composition analysis, and event detection; and hard tasks require attribution and forecast. Each scenario unfolds over monthly, weekly, daily, or irregular release periods. For each target period, the benchmark instanti- ates specific questions and structured answer fields, comprising 1,485 scenario-period QA pairs. The agent receives pre-cutoff time series data and docu- ments, while the target-period release is withheld for verification. The question types are annotated with a 27-tag taxonomy grouped into six fami- lies (Appendix A.3.1 Table 6): state identification, change detection, composition analysis, event de- tection, attribution, and outlook. The agent answers the questions and compiles them into a report. Crit- ically, as new observations arrive in periods, the agent must update its judgments and revise its re- port adhering to the rolling evidence cutoff. Our evaluation protocol scores the output along four axes: keypoint accuracy, keypoint coverage, report faithfulness, and report quality. This distin- guishes agents that compute correct answers from those that generate fluent but weakly grounded reports. We then evaluate frontier LLM agents and TimeSage-1.0, a self-evolving agent with a reusable skill library, on TimeSage-EV. The re- sults reveal large performance gaps across difficulty tiers and failures in memory management, tempo- ral misuse, and report synthesis. These findings show that current agents struggle to maintain cutoff- correct evidence and to adapt to new releases, of- fering diagnostic guidance for reliable agents. Our contributions are summarized as follows. • We formalize a novel task for agentic time se- ries analysis in evolving environments, requir- ing temporal isolation, cutoff-valid evidence use, and recurring analytical updates. •We construct TimeSage-EV, a live benchmark 1 comprising 60 scenarios across 6 domains, 3 tiers, and 1,485 scenario-period questions with ground truth and traceable evidence. •We design a rigorous multi-axis evaluation pro- tocol measuring keypoint accuracy, keypoint coverage, report faithfulness, and report quality across structured answers and analytical reports. •We benchmark frontier LLM agents and the diag- nostic TimeSage-1.0, uncovering failure modes in memory management, temporal misuse, and report synthesis to guide future research. 1 The benchmark is kept live through a maintenance script that runs automatic updates monthly. See Appendix A.2. 2 Related Work Prior work spans language-grounded time series analysis and long-horizon live agent benchmarks. However, live benchmarks for agentic time series analysis in evolving environments remain underex- plored, particularly in scenarios where agents must repeatedly analyze recurring data releases while adhering to cutoff-valid numerical and documen- tary evidence. Table 1 summarizes representative benchmarks across these dimensions. Table 1: Comparison of representative benchmarks.✔ denotes support,△ partial support, and✘ not covered. BenchmarkTSTextAgentLiveEvol.Cutoff Time-LLM (Jin et al., 2024)✔✘ ChatTime (Wang et al., 2025)✔✘ Time-MQA (Kong et al., 2025)✔✘ TimeSeriesExam (Cai et al., 2024)✔✘ WebArena (Zhou et al., 2024)✘✔✘ LiveBench (White et al., 2025)✘ △ △✔✘ ForecastBench (Karger et al., 2025)✘ △✔✘ TimeSeriesGym (Cai et al., 2025)✔✘ TimeSage-MT (Kong et al., 2026)✔✘ TimeSage-EV✔ TS: numerical time series evidence; Text: textual or documen- tary evidence; Evol.: the same scenario is revisited across release periods; Cutoff : period-specific evidence isolation. Language-grounded time series analysis. Time series analysis is increasingly framed as a language- facing task that connects numerical patterns with in- structions and domain context. LLMTime (Gruver et al., 2023) and Time-LLM (Jin et al., 2024) recast time series for language-model use, while Chat- Time (Wang et al., 2025), ChatTS (Xie et al., 2025), Time-MQA (Kong et al., 2025), TSRBench (Yu et al., 2026), TimeSeriesExam (Cai et al., 2024), TimeSage-MT (Kong et al., 2026), and Tempo- ralBench (Weng et al., 2026) evaluate temporal reasoning through natural-language questions. Be- yond question answering, TimeSeriesGym (Cai et al., 2025) evaluates LLMs on end-to-end time se- ries machine learning tasks. These settings remain snapshot-based, with fixed questions over static evidence, and do not evaluate how agents update analyses across recurring releases. Long-horizon live agent benchmarks. LLM agents are evaluated through tool use, retrieval, code execution, and memory, as in ReAct (Yao et al., 2023), Toolformer (Schick et al., 2023), and AutoGen (Wu et al., 2024). Benchmark tasks span web navigation, dialogue, and software engi- neering, including WebArena (Zhou et al., 2024), GAIA (Mialon et al., 2024), Tau-Bench (Yao et al., 2025), and SWE-bench (Jimenez et al., 2024). Live benchmarks such as LiveBench (White et al., 2025) and LiveCodeBench (Jain et al., 2024) reduce con- tamination through refreshed task pools, while ForecastBench (Karger et al., 2025) evaluates pre- dictions after outcomes resolve. Yet, these settings remain refreshed snapshots, rather than evolving scenarios with recurring releases and cumulative numerical-documentary evidence. Positioning TimeSage-EV. Existing work leaves a critical gap at the intersection of these two lines: live, continuous agent evaluation for recurring time series analysis. TimeSage-EV addresses this gap by evaluating agentic time series analysis in evolving environments, where agents revisit in- stitutional releases, combine evolving numerical and textual evidence, respect time cutoffs, and pro- duce both structured answers and grounded reports. TimeSage-EV moves beyond static QA task to eval- uate evolving time series analysis as a dynamic, long-horizon agentic workflow (Appendix A.1). 3 Methodology 3.1 Task Formulation We formulate agentic time series analysis in evolving environments as a recurring task over sequential release periods. In agentic time se- ries analysis, an agent must utilize tools to inspect time series, consult cutoff-valid documents, answer structured questions, and generate an evidence- based report. An evolving environment presents this task under changing information states, as new evidence arrives across subsequent periods (Fig- ure 2). We formalize this setting below and provide expanded definitions in Appendix A.2. Scenario Task. An environment for scenarios defines a recurring analytical task over time: E s = (D s , Q s , A s , e (s) t k t k ∈T s ), whereD s denotes the domain background,Q s is the recurring question template,A s is the struc- tured answer schema, ande (s) t k is the sequence of evidence states across release periodsT s = t 1 ,...,t T . The task specification(D s ,Q s ,A s ) remains fixed within a scenario, whilee (s) t k changes as new releases become available. Period Instance. For each target periodt k , the period instance is defined by the environment state e (s) t k = q t k , X ≤t k , R <t k , W (s) ≤t k , whereq t k is the natural-language question,X ≤t k is the cumulative time series table up through the Report EININO LANINA 2022-062023-062024-062025-062026-03 Nino 3.4 anomaly ( °C ) 0 -1 -2 -3 1 2 3 2024-06 2024-12 2025-06 2026-03 ReportReport Report Role: Operational Climate AnalystScenario:ENSO Update: Monthly Background: ENSO is the dominant driver of seasonal climate variability. ENSO alerts give downstream sectors lead time to prepare, but signal is uncertain and shifts monthly, requiring continual re-assessment. Input: timeseries.csv,cumulative Oceanic Niño Index time series since 1950; <Y-M>.shtml,CPC ENSO diagnostic reportsupdatedper month. Periodic Task:It‘s now<time> ①What’sthis month‘s ENSO alert status? ②What ocean and atmosphere conditions best support the assessment? Scenario Card Evolving Environment Agent Output Agent ①Final EI NINO ①LaNinaWatch ①Not Active ②[TS] Nino 3.4 drifted to −0.42°C; 6monthlydeclines, above threshold; [Doc] Walkercircu- lationstrengthened. ②[TS] Nino 3.4ho- vered near −0.04°C; DJF dip never cross- ed −0.5°C. [Doc] SmallLa Niña (forecastbyhuman). ①Final La Nina ②[TS] Nino 3.4at +0.11°C; 4-mo sub- threshold spellgone. [Doc] Subsurface index (180°–100°W) rose for fifth month. ②[TS] Niño 3.4 at +0.22°C; shed 1.8°C since 2023 peak; [Doc] Belowavg SSTs expanded in eastern Pacific. Figure 2: Evolving environment in TimeSage-EV with ENSO scenario. TS: time series, Doc: source document. target period, andR <t k contains source documents released strictly before the target period. The target period releaseR t k is withheld from the agent and used only for ground truth construction and evi- dence verification. Thus, the time series is visible, but the document is not. Finally,W (s) ≤t k denotes external web evidence available prior to the cutoff. By default,W (s) ≤t k = ∅, and it is enabled only for scenarios with cutoff-valid retrieval. Agent State and Output. Within this environment, the LLM agent maintains a stateS t k , which may encompass memory, reasoning traces, or reusable analysis artifacts. Given a period instance, the agent returns ˆy t k , ˆr t k , S t k+1 = Agent s e (s) t k , S t k , whereˆy t k is a schema-conformant structured an- swer andˆr t k is a cutoff-grounded report. Stateless agents useS t k =∅. Figure 2 illustrates this evolv- ing process. As the benchmark progresses from t k tot k+1 , the task specification remains fixed, but the evidence state expands: X ≤t k ⊆ X ≤t k+1 , R <t k ∪R t k ⊆ R <t k+1 . Thus, each step introduces a new evidence state in which the time series is extended, the previous target release becomes prior evidence, and the next target release is withheld for evaluation. The agent must continuously revise its judgments without vi- olating the time cutoff. As future releases are pub- lished, the same process creates new temporally isolated evaluation periods, making TimeSage-EV a continuously updated live benchmark. 3.2 Benchmark Construction TimeSage-EV is built from public institutional re- leases that recur on a fixed cadence, and every evaluation period of every scenario is reproducible from those releases. The pipeline proceeds in three stages: source-side data collection, scenario-side generation of period-level instances, and a qual- ity control pass that enforces temporal isolation. Figure 3 reports corpus statistics, while Figure 4 sketches the pipeline. Full prompts, scenario check- lists, and audit attestations are provided in Ap- pendix A.3. (a)QuestionType(%) State Identification Change Detection Outlook Event Detection Attribution Composition Analysis 26.6% 23.6% 14.3% 13.5% 13.1% 8.9% (c)DifficultyTier 20 20 20 Easy Medium Hard (d)ReleaseCadence Monthly Weekly Daily Irregular 5 5 11 39 0200102025500 (b)CorpusSummary 60 Scenarios 6 Domains 288 Keypoints 1,485 QAPairs 46 Publishers 3 Tiers 27 Tags Y23-Y26 Span 4 ScoreAxes Figure 3: TimeSage-EV corpus statistics. 3.2.1 Data Collection Step I: Domain scanning and source registry. We built a registry of 500 public institutional sources across six domains: agriculture, climate, energy, finance, healthcare, and transportation. Candidate sources include APIs, periodic briefs, and statistical reports from government agencies and international organizations. For each source, we record metadata, including publisher, source type, release cadence, historical span, and data format. Step I: Attribute-based filtering. We retain sources that are public, recurring, historically deep enough for repeated instantiation, grounded in timestamped time series, and tied to answers trace- able to artifacts. We exclude sources requiring private access and lacking period alignment. We balance domain coverage, source type, and release cadence so that the corpus spans monthly, weekly, daily, and irregular schedules. Step I: Source validation.Each surviving source is validated for all target periods before promotion. We verify that source artifacts can be recovered, time series remain stable and ordered, cumulative histories can be assembled for each tar- Source Registry Collect candidate data sources Attribute Filtering Pick sources meeting benchmark criteria Source Validation Verify sources through actual downloads Evidence Extraction Extract time series analysis evidence from reports Question Generation Create evaluable questions based on evidence GT Extraction Derive deterministic answers from data (GT: ground truth) Grounding Audit Check consistency across questions, ground truth & evidence Human Check Review and address violations Stage 1: Data Collection Stage 2: Scenario Generation Stage 3: Quality Control TimeSage-EV (Live) Figure 4: Overview of benchmark generation pipeline. Benchmark data is updated monthly via automated scripts. get period, and ground-truth fields are traceable to time series observations or quoted document spans. 3.2.2 Scenario Generation Step IV: Evidence collection. For each accepted sources, we construct period-level evidence pack- ages. Each package represents the agent-visible state at a given period, containing the time series and documents accumulated up to that point. We extract background information from the source description and time series-relevant analytical text from the documents. These materials provide the basis for subsequent question generation. Step V: Question generation. Using the evidence in Step IV, we generate questions and export a scenario-level contract for each source. To do this, we adapt question, answer-schema, and scoring templates from a 27-tag taxonomy, spanning six question categories (Appendix A.3.1 Table 6), into a scenario-specific contract. LLMs are used for this adaptation, while the predefined question–answer– scoring structure is preserved. The contract can be instantiated in every period. The question distribu- tion varies by difficulty: easy scenarios focus on state identification; medium scenarios require tem- poral pattern identification; and hard scenarios in- volve analytical reasoning that combines numerical changes with document-based explanations. This difficulty cliff is shown in Appendix A.7 Table 12: keypoint answer accuracy declines monotonically from easy to hard across all evaluated LLMs. Step VI: Ground-truth extraction. For each in- stance, we derive reproducible ground truth accord- ing to the scenario contract. For each period, we follow the answer schema and scoring method spec- ified in the scenario contract to construct ground truth, either through computation or LLM-assisted evidence finding, depending on the field type. For fields defined over the time series, we apply de- terministic rules to the time series table, including threshold-based classification, multi-variable com- parison, and cross-period comparison. For fields requiring textual evidence, we use the withheld target-period source during answer construction and verification, transcribing the statements or ex- planations from the document. 3.2.3 Quality Control Every scenario undergoes an automated ground- ing audit and human review before release. The grounding audit checks schema validity, tempo- ral consistency, variable and answer-space valid- ity, deterministic answer derivability, evidence sup- port, leakage prevention, and instance-level com- pleteness. Three human reviewers then verify the ground truth, the clarity of the questions, and the grounding of the evidence. A scenario is released only after passing both stages. The final corpus contains 60 scenarios from 6 domains and 1,485 period-level instances spanning February 2023 to May 2026, with a balanced 20/20/20 difficulty-tier split, as summarized in Figure 3. 3.3 Evaluation Protocol Each agent run produces two outputs at each pe- riod: a structured answer and a free-form report. We score them on four axes: (1) keypoint ac- curacy, which checks whether the structured an- swer matches the ground truth; (2) keypoint cover- age, which checks whether the report mentions the keypoints; (3) report faithfulness, which checks whether the report’s claims are supported by cutoff- valid evidence rather than future or unsupported information; and (4) report quality, which checks whether the report is clear, organized, and appro- priate for the scenario. The protocol combines rule-based scoring for fields with fixed answer for- mats, such as labels, yes/no values, and numbers, with LLM-based judging for the report’s coverage, faithfulness, and writing quality. Each axis is re- ported on a 0–100 scale, and the run-level score is the average of the scored axes. We report both inde- pendent evaluation, where the agent starts fresh for each period, and sequential evaluation, where the agent can carry state within a scenario but receives only prior-period evidence. Figure 5 shows a sce- nario example with the evaluation protocol. More details on the judge are shown in Appendix A.4. Keypoint Accuracy Keypoint Coverage Report Faithfulness Report Quality compare structured answer to Ground Truth keypoints Alert status Outlook Key drivers Uncertainties Support evidence Event token 100 100 89 57 100 100 mean score of 6 = 91 recall in report 1/6 ×100 = 16.67 100 cover uncover uncover uncover uncover uncover fact-level verdict all criteria met: 1 of 6 key points covered: 8 of 10 facts evidence backed: 8/10 ×100 = 80 1. time series supported: 7 2. report supported: 1 3. fact supported: 0 4. unbacked: 2 rubric-based check 1. CPC issue date stated 2. Alert label stated 3. Outlook 4. ≥ 1 driver & uncertainty with evidence Period Task (Nov 2024） TS up to Nov 2024 Prior CPC ENSO discussions 7 analytical questions Structured Answer Free-form Report The ONI history now extends to OND 2024, where Niño-3.4 is -0.30°C, down from - 0.21°C in SON 2024. In the prior 10 October 2024 CPC discussion, the alert remained La Niña Watch ... For November 2024, the 3-month outlook is La Niña, 57%, October-December 2024. Risks remain material because ... (91+ 16.67+80 + 100) / 4 = 71.92Aggregated Score: Figure 5: Evaluation for the El Niño–Southern Oscilla- tion (ENSO) monitoring scenario. TS: time series. 4 Experiments 4.1 Environment Setting Each evolving scenario is instantiated as a sandbox that periodically releases time series analysis tasks and coordinates information exposure, evaluation, and feedback. In each period, the environment publishes the task and exposes only causally admis- sible information prior to that time point. The agent completes the task within this temporally isolated sandbox and submits its output. Once the scenario source releases the corresponding report, the envi- ronment collects the ground-truth outcome, evalu- ates the output, returns feedback, and advances to the next period, repeating the cycle. With a collec- tion cutoff in May 2026, we snapshot the latest 24 periods per scenario, while an automated pipeline continuously pulls from the scenario sources so each environment updates at its own release ca- dence, and the benchmark remains live over time. All experiments in this paper use these 24 periods. 4.2 Experimental Setting Baseline Models. We usesmolagent(Roucher et al., 2025) as the baseline, equipped with mem- ory, search, and code execution.A memory- compression tool can be invoked by the agent or triggered automatically when the context ap- proaches the token limit, as detailed in Ap- pendix A.6. We evaluate six LLMs under this har- ness: Sonnet-4.6 (Anthropic, 2026), GPT-5.4 (Ope- nAI, 2026), Kimi-K2.6 (Kimi Team, 2025), Qwen- 3.5-397B-A17B-NVFP4-TBC (Qwen Team, 2026), Devstral-2-123B-Instruct-2512 (Mistral AI, 2025), and Gemma-4-31B-IT-NVFP4 (Google DeepMind, 2026). 2 All models use temperature=0 with a 32,768-token output budget; all other hyperparame- ters remain at provider defaults. Each rollout is limited to 20 steps or 2,700 seconds; runs that exhaust either budget without producing an an- swer are counted as failures. For the LLM judge axes (coverage, faithfulness, and quality), we use DeepSeek-V4-Flash (DeepSeek-AI, 2026), with the prompts shown in Appendix A.5.2 and verifica- tion in Appendix A.7.2. TimeSage-1.0. To examine whether agents can self-improve from feedback over long-horizon tasks, we propose TimeSage-1.0, asmolagent- based harness augmented with a self-evolving skill- library management system (see Appendix A.6 for more details). The system supports skill induc- tion, quality checking, and skill retrieval, allowing agents to convert reasoning traces into reusable workflows and deploy them in subsequent periods. 4.3 Main Results Overall performance. Across the full benchmark, GPT-5.4 achieves the strongest score (86.0), fol- lowed by Qwen-3.5-397B (84.2) and Sonnet-4.6 (82.4), as summarized in Figure 6. Performance is near-saturated on easy tasks, where all models score above 92, but drops as the task requires richer analytical synthesis. On hard tasks, the best model reaches only 72.7, while Kimi-K2.6 and Gemma- 4-31B fall to 59.3 and 53.1, respectively. This tier-wise gap indicates that agents can handle basic time series analysis tasks, including state and tem- poral pattern identification. However, large-scale evidence extraction and external-information rea- soning remain major bottlenecks in hard scenarios. Token cost. As shown on the right side of Figure 6, total token cost varies by more than 4×across mod- els, from 84M for GPT-5.4 to 364M for Sonnet- 4.6. GPT-5.4 achieves the highest overall score while using the fewest tokens, showing a clear effi- ciency advantage. In contrast, Sonnet-4.6 and other 2 Sonnet-4.6 and GPT-5.4 are accessed through the Open- Router API (https://openrouter.ai), while the remaining LLMs are served via local vLLM deployments. Figure 6: Left: Outcome scores across difficulty tiers and overall; error bars are per-difficulty-level percentile bootstrap 95% confidence intervals. Right: Total token cost. Table 2: Per-question-type scores, reported as mean with bootstrapped 95% CI in the formatx +high −low . Cell shading uses a shared scale (lowhigh); the best result in each column is shown in bold. ModelState IDChange Det.Comp. IDEvent Det.AttributionForecast Sonnet-4.690.9 +1.2 −1.2 92.9 +1.5 −1.7 85.6 +2.7 −2.7 67.9 +2.2 −2.1 42.6 +3.0 −3.2 41.2 +4.1 −4.0 GPT-5.493.8 +0.8 −0.9 96.0 +1.2 −1.2 86.9 +2.4 −2.7 75.4 +1.9 −1.9 46.2 +3.2 −3.4 42.7 +4.1 −3.9 Kimi-K2.684.2 +1.8 −1.7 84.9 +2.4 −2.4 80.8 +3.2 −3.1 61.0 +2.6 −2.5 39.3 +3.3 −3.4 33.4 +3.9 −3.7 Qwen-3.5-397B92.6 +0.9 −0.9 97.0 +0.9 −1.0 86.8 +2.4 −2.5 74.1 +1.9 −1.9 46.1 +3.3 −3.2 43.0 +4.0 −3.8 Devstral-2-123B83.4 +1.5 −1.6 76.5 +2.8 −2.6 79.7 +3.1 −3.2 64.7 +2.4 −2.4 46.7 +3.2 −3.0 42.8 +4.1 −4.3 Gemma-4-31B82.7 +1.7 −1.8 80.2 +2.4 −2.7 77.3 +3.4 −3.3 55.7 +2.7 −2.6 37.3 +3.4 −3.2 34.8 +4.0 −3.8 high-cost models fail to lead in accuracy. These results suggest that reasoning efficiency drives per- formance in long-horizon time series analysis. Live leaderboard. Figure 7 traces how overall scores evolve under continuous testing, where each LLM’s in-context history accumulates from period t− 23to the cutoff time. After an initial three- period ramp-up, all models reach a high plateau, indicating that early context is useful. The trajec- tories then diverge: GPT-5.4 maintains the highest and most stable score with no degradation, showing strong long-horizon memory robustness, whereas Gemma-4-31B and Kimi-K2.6 decline steadily, the former falling from 80 to nearly 63. This late-stage drop suggests that long-horizon history retrieval be- comes a bottleneck for weaker LLMs in time series analysis. This is a key advantage of TimeSage-EV over static benchmarks: rather than evaluating iso- lated snapshots, it exposes whether LLMs remain reliable under long-term deployment. Per-question-type analysis. The per-question- type results in Table 2 reveal where the difficulty arises across the six question families. State identi- fication, change detection, and composition ques- tions are comparatively tractable, with the strongest models reaching above 85–95. In contrast, attribu- tion and forecast questions remain difficult for all Overall Score 60 70 80 90 100 Period cutoff-time t-23t-16t-8 FutureRelease Sonnet-4.6 Qwen-3.5-397B Devstral-2-123B GPT-5.4Gemma-4-31B Kimi-K2.6 Figure 7: Live overall-score leaderboard across periods. Scores are tracked as information accumulates fromt− 23 to cutoff time. Shaded region marks future updates. models. The best attribution score is 46.7 and the best forecast score is 43.0. These results reveal a capability gap: agents can compute over time series, but struggle with temporal grounding for cautious forward-looking analysis. 4.4 Failure Mode (FM) Analysis FM1: Reasoning failure. Figure 8 shows that reasoning failures are a major failure mode across models, involving reasoning overrun and instruc- tion drift. Reasoning overrun occurs when a model follows a long reasoning path and times out, reflect- ing difficulty reaching the correct answer efficiently. Instruction drift occurs when long contexts cause the model to forget rules, such as code-execution protocols or bash-command constraints. These fail- ures highlight the need for efficient reasoning and memory management in evolving environments. FM2: Temporal misuse. Temporal errors, includ- ing temporal misreading, temporal miscalculation, and inference errors stems from a weak alignment between task semantics, time series data, and ev- idence. Agents may read incorrect rows or times- tamps, compute incorrect variables or time win- dows, or rely on mismatched evidence for forecast- ing and attribution. These errors show that tem- poral grounding, rather than data access, remains a core challenge in evolving time series analysis environments. FM3: Incomplete reporting. Incomplete report- ing is a failure mode, where agents recover correct structured fields but omit required key points or fail to ground them in the report. This occurs in part be- cause overly long reasoning chains make it difficult for agents to summarize all required points. Figure 8: Failure-mode distribution per model; right numbers indicate total score impact. 4.5 The Roles of Memory and Skill Induction Does Memory Help in Evolving Environments? Figure 9 compares the sequential and indepen- dent evaluation of Qwen-3.5-397B and Gemma- 4-31B. For Qwen-3.5-397B, memory provides a benefit: sequential evaluation outperforms indepen- dent evaluation over periods. This suggests that a persistent state helps agents reuse analysis patterns instead of solving each period from scratch. How- ever, for Gemma-4-31B, memory is less beneficial: sequential evaluation improves early performance, but its advantage narrows and reverses in later pe- riods. Memory is useful only when models can manage it: history can support future analysis but may become a burden for in-context management. Can Agents Self-Evolve Reusable Skills? To test whether agents can convert repeated feed- back into reusable analytical skills, we evalu- ate TimeSage-1.0, oursmolagent-based harness Figure 9: Sequential vs. independent evaluation for Qwen-3.5-397B (Top) and Gemma-4-31B (Bottom). equipped with a lightweight self-evolving skill- library manager. As shown in Figure 10 a, reusable skills improve overall performance, with the clear- est gains in medium- and hard-difficulty scenarios. To understand this benefit, we further examine ef- ficiency and skill usage: enabling reusable skills reduces token cost to0.82×(Figure 10 d), while the skill-invocation frequency peaks in hard scenar- ios (Figure 10 b), suggesting that reusable skills are most valuable in evolving time series analysis when reasoning is difficult. However, skill induction and reuse are concentrated in the early periods and do not persist across the entire horizon (Figure 10 c), revealing that maintaining an effective skill library remains challenging in evolving time series tasks. Overall Score 0 100 60 40 20 80 Skill Frequency (%) overalleasymediumhard SmolagentTimeSage-1.0 ∆=+5.83 ∆=-1.63 ∆=+6.80 ∆=+12.20 0 5 10 4.4 3.6 10.6 easymediumhard Token/period (K) 0 50 100 150 x1.09 ↑ x1.02 ↑ x0.82 ↓ easymediumhard Smolagent TimeSage-1.0 Percent 0 100 80 60 40 20 88% 20.8% Skill induction rate Skill reusing rate t-23t-17t-11t-5 (a)OverallScorebyDifficulty (b)Skill-Usage (c)SkillInduction&Reuse (d)TokenCostbyDifficulty Figure 10: Analysis of self-evolving skill mechanism on Gemma-4-31B under evolving evaluation environments. 5 Conclusion We introduced TimeSage-EV, a live benchmark for agentic time series analysis in evolving en- vironments. TimeSage-EV formalizes recurring, cutoff-valid analytical updates and evaluates agents across 60 institutional scenarios, 1,485 scenario- period QA pairs, and 4 complementary scoring axes. Experiments with frontier LLM agents and TimeSage-1.0 reveal persistent gaps in memory management, long-horizon reasoning, and tempo- ral validity. These findings establish TimeSage-EV as a diagnostic benchmark for developing robust an- alytical agents that can update time series analyses reliably as evidence continually evolves. Limitations TimeSage-EV is built from public institutional sources with recurring releases, and therefore does not cover enterprise data, dashboards, or settings where historical releases are unavailable or poorly archived. Although the benchmark spans 6 do- mains and multiple release cadences, it cannot exhaust the diversity of document formats, in- stitutional conventions, or analytical workflows. Furthermore, the dataset relies predominantly on English-language reports from established global institutions, which may introduce geographic and linguistic biases, and it focuses strictly on numer- ical time series and textual documents, excluding the visual interpretation of charts and graphs that frequently accompany real-world releases. Regarding our evaluation protocol, we combine rule-based scoring for structured fields with LLM- based judging for report coverage, faithfulness, and quality. Despite structured rubrics and evi- dence grounding, some residual judge noise and bias may remain. Our experimental baseline also relies on thesmolagentframework, meaning some observed failure modes, such as instruction drift or reasoning overrun, may be partially entangled with this specific harness rather than solely reflecting the underlying capabilities of the LLMs. The experiments in this paper use May 2026 snapshot. We will release a frozen, versioned snapshot of the benchmark containing all 1,485 scenario-period instances evaluated in this paper (capped at May 2026). Future researchers can eval- uate against this static version for direct compa- rability with our baselines, while simultaneously testing on the live branch to measure true out-of- distribution generalization on newly published in- stitutional releases. Intended Use TimeSage-EV is intended as an academic research benchmark for diagnosing LLM agents on recur- ring time series analysis, and agent output should be treated as decision-support signals rather than substitutes for professional judgment in high-stakes domains. Scores in TimeSage-EV should be inter- preted as diagnostic indicators of analytical behav- ior in covered scenarios rather than as guaranties of agent reliability in unseen sources, domains, or release schedules. Agents may also produce fluent but unsupported claims or violate temporal cutoffs in ways that appear authoritative, so outputs require human verification before any downstream use. Ethical Considerations TimeSage-EV strictly adheres to ethical research standards. It utilizes publicly available institutional releases in compliance with each source’s usage terms. The benchmark contains only aggregate time series and reports, with no personally identifi- able information (PII). Human reviewers verifying ground-truth correctness worked only with pub- lic, non-sensitive content. We emphasize that our agents’ analyses are designed as a decision-support tool for human experts, not a replacement for pro- fessional judgment in high-stakes domains, and users should be mindful of potential model hallu- cinations. Our approach does not involve PII or sensitive data, and we follow the Code of Ethics. Acknowledgments This work was financially supported by the Eu- ropean Union’s Horizon Europe research and innovation programme under grant agreement No. 101214398 (ELLIOT). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. We also acknowledge the Supercomputing Center of the Eindhoven Uni- versity of Technology (Home - TU/e Supercomput- ing Center) for providing access to and assistance with the various computing resources available. References Anthropic. 2026.Introducing claude sonnet 4.6.https://w.anthropic.com/news/ claude-sonnet-4-6. Yifu Cai, Arjun Choudhry, Mononito Goswami, and Artur Dubrawski. 2024.Timeseriesexam:A time series understanding exam.arXiv preprint arXiv:2410.14752. Yifu Cai, Xinyu Li, Mononito Goswami, Michał Wil- i ́ nski, Gus Welter, and Artur Dubrawski. 2025. Time- seriesgym: A scalable benchmark for (time series) machine learning engineering agents. arXiv preprint arXiv:2505.13291. DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence. Google DeepMind. 2026.Gemma 4 model card.https://ai.google.dev/gemma/docs/ core/model_card_4. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gor- don Wilson. 2023. Large language models are zero- shot time series forecasters. In Advances in Neural Information Processing Systems, volume 36. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- CodeBench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07974. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can language mod- els resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representa- tions (ICLR). Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. 2024. Time-LLM: Time series forecasting by reprogramming large language models. In Interna- tional Conference on Learning Representations. Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tet- lock. 2025. ForecastBench: A dynamic benchmark of AI forecasting capabilities. In The Thirteenth In- ternational Conference on Learning Representations (ICLR). Kimi Team. 2025. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. 2025. Time-MQA: Time series multi- task question answering with context enhancement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29736–29753, Vienna, Austria. Association for Computational Linguistics. Yaxuan Kong, Qingren Yao, Yuqi Nie, Yichen Li, Yilei Shao, Stefan Zohren, Anna Vettoruzzo, Joaquin Vanschoren, Ming Jin, and Qingsong Wen. 2026. Timesage-mt: A multi-turn benchmark for evaluat- ing agentic time series reasoning. arXiv preprint arXiv:2606.01498. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations (ICLR). Mistral AI. 2025.Introducing Devstral 2 and Mistral Vibe CLI.https://mistral.ai/news/ devstral-2-vibe-cli. OpenAI. 2026. GPT-5.4 thinking system card. Techni- cal report, OpenAI. Qwen Team. 2026. Qwen3.5-397B-A17B.https: //huggingface.co/Qwen/Qwen3.5-397B-A17B. Model card. Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 2025.‘smolagents‘: a smol library to build great agentic systems.https://github.com/ huggingface/smolagents. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Pro- cessing Systems, volume 36. Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. 2025. ChatTime: A unified multimodal time series foundation model bridging numerical and tex- tual data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12694– 12702. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McK- inney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. BrowseComp: A simple yet challeng- ing benchmark for browsing agents. arXiv preprint arXiv:2504.12516. Muyan Weng, Defu Cao, Wei Yang, Yashaswi Sharma, and Yan Liu. 2026. TemporalBench: A benchmark for evaluating LLM-based agents on contextual and event-informed time series tasks. arXiv preprint arXiv:2602.13272. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, and 1 oth- ers. 2025. LiveBench: A challenging, contamination- limited LLM benchmark. In The Thirteenth Inter- national Conference on Learning Representations (ICLR). Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Awadal- lah, Ryen W. White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling next-gen LLM applica- tions via multi-agent conversation. In Conference on Language Modeling. Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. 2025. ChatTS: Aligning time series with LLMs via synthetic data for enhanced understanding and rea- soning. Proceedings of the VLDB Endowment, 18(8). Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R. Narasimhan. 2025.τ-bench: A bench- mark for tool-agent-user interaction in real-world do- mains. In The Thirteenth International Conference on Learning Representations (ICLR). Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations (ICLR). Fangxu Yu, Xingang Guo, Lingzhi Yuan, Haoqiang Kang, Hongyu Zhao, Lianhui Qin, Furong Huang, Bin Hu, and Tianyi Zhou. 2026. TSRBench: A com- prehensive multi-task multi-modal time series reason- ing benchmark for generalist models. arXiv preprint arXiv:2601.18744. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2024. WebArena: A realistic web environment for build- ing autonomous agents. In The Twelfth International Conference on Learning Representations (ICLR). A Appendix A.1 Task Rationale Why recurring time series analysis is different. TimeSage-EV targets a setting that is not captured by standard time series forecasting, static time se- ries QA, or generic agent benchmarks. In recur- ring institutional analysis, the task specification remains fixed, but the evidence available to the an- alyst changes after each release. The agent must interpret the latest observation, compare it with his- torical context, use only cutoff-valid documents, and revise its report as the environment evolves. Why temporal isolation matters.A central mo- tivation of TimeSage-EV is to evaluate whether agents can reason within the information limit that would have been applied at a target period. With- out this constraint, an agent may answer using fu- ture reports, post-hoc explanations, or documents published after the target release. TimeSage-EV prevents this by exposing cumulative time series data and prior-period source documents while with- holding the target-period release for ground truth construction and evidence verification. Why agentic evaluation matters. Real analyt- ical workflows require more than reading a fixed table. Agents must inspect data files, use tools, consult documents, preserve state across periods, and produce both structured answers and natural- language reports. TimeSage-EV evaluates agentic time series analysis as an integrated workflow, ex- posing failures such as unsupported causal claims, poor use of evidence, and incomplete reporting. A.2 Core Concepts Live benchmark. We call TimeSage-EV a live benchmark because its evaluation set continuously expands as institutional releases are published. For each scenarios, new releases extendT s over time. Each extension updates the cumulative time series and the available prior-period evidence, withholds the target-period release, and keeps the task con- tract fixed. This expansion is automated rather than manual: each scenario carries a self-contained, re- runnable pipeline that re-pulls the source, rebuilds the cumulative inputs, withholds the newest release, and re-derives ground truth and evidence under the scenario’s frozen rules. A monthly maintenance script executes this pipeline across all scenarios, keeping TimeSage-EV live without re-authoring any questions and ground-truths. Evolving environment. An evolving environ- ment is a sequence of period-specific evidence for the recurring analytical task. The domain back- ground, question template, and answer schema re- main fixed within a scenario, while the time series extends, and the document set changes as the re- leases arrive. The agent must update its judgments for periods while respecting the cutoff point. Agentic time series analysis. Agentic time se- ries analysis refers to workflows in which an agent uses tools to inspect time series data, consult ex- ternal evidence, answer analytical questions, and produce task-specific output. TimeSage-EV stud- ies a distinctive setting in which the analytical task recurs across an evolving environment: the time series extends, the document set changes, and each period imposes a cutoff on evidence. The agent can maintain evidence, reasoning traces, or reusable analysis artifacts over periods. We therefore evalu- ate not only the structured answer, but also a free- form report that exposes the agent’s evidence use, temporal grounding, and reasoning. This mirrors real-world time series analysis, where conclusions are typically delivered as periodic analytical reports rather than standalone structured predictions. A.3 Benchmark Construction Section 3.2 and Figure 4 summarize the benchmark generation pipeline. This appendix expands that overview into a detailed construction protocol. Sec- tions A.3.1 through A.3.4 walk through the pipeline sequentially, from data collection and scenario gen- eration to quality control and human annotation. Section A.3.5 documents the data sources and re- ports volatility statistics. Finally, Section A.3.6 presents three illustrative scenario examples. A.3.1 Stage 1: Data Collection Step 1: Source registry. The pipeline is a domain-by-domain pass loop. Each domain pass begins from a target domain selected to close a cur- rent portfolio gap, such as an under-represented tier or release cadence. For that domain, the builder constructs a registry of candidate public sources, searching for recurring institutional data products that could fill the gap. We prioritize gov- ernment agencies, central banks, international or- ganizations and stable institutional publishers, in- cluding sources such as EIA, NOAA, CDC, FRED, BLS, the World Bank and IEA. Each candidate is recorded in eight standardized metadata fields: endpoint, authentication requirement, update fre- quency, historical span, data format, license, re- port links, and free-form notes. The step yields a per-domain candidate registry and exits when the domain has accumulated at least ten plausible can- didates or is exhausted, with every metadata field either filled in or explicitly marked unknown. Step 2: Attribute filtering.Taking the candidate registry as input, we first reject sources that cannot support a recurring benchmark task before writing any concrete question, and then assign a difficulty tier to the passed sources. •Metadata filtering: apply Table 3 to remove sources that are inaccessible, weakly time- indexed, insufficiently variable, or unsuitable for evaluation of evolving-agents. •Tier criteria: assign each surviving source to pre- liminary tiers based on Table 4: Easy for categor- ical state identification, Medium for temporal- pattern summarization, and Hard for report- grounded reasoning about causes or outlooks. Table 3: Metadata-level validity checks in Step 2. No.Validity check 1Public access. Source is publicly accessible without credentials, payment, or approval. 2Time series grounding. Task uses an observable times- tamped series with values for the queried period. 3 Query persistence. The same template applies to both historical and future periods. 4Temporal anchoring. The source provides a date, pe- riod, timestamp, or update window. 5 GT freshness. Ground truth updates at least monthly, except report-level hard tasks. 6 Temporal variability. Outputs vary over time and avoid near-constant majority labels. 7Evolving-agent fit. Task requires time-dependent re- trieval, update, comparison, or synthesis. Step 3: Execution validation. The third step takes each retained source–tier pair and verifies by execution so that it can produce valid period- level benchmark instances. The builder retrieves the primary time series, releases archive, normal- izes it into a period-indexed table, and retrieves the corresponding source releases, such as dashboards, briefs, bulletins, or analytical reports. As summa- rized in Table 5, each candidate must provide at least24periods aligned instances with traceable, time-isolated, and rule-evaluable evidence. The difficulty tier assigned in Step 2 is confirmed during execution: Easy scenarios must yield cate- Table 4: Difficulty-tier criteria across source, task, an- swer format, and evaluation in Step 2. TierCriteria EasySource: a data dashboard reporting current op- erational state as categorical indicators. Task: identify the current state (reasoning LOW, ca- pability dependence LOW). Format: a single categorical label. Eval: exact match against the correct option. Medium Source: a data brief with time series and descrip- tive commentary, but no causal analysis. Task: summarize recent changes across time points and sub-components (reasoning MEDIUM, de- pendence MEDIUM). Format: a structured schema (period value, direction, leading sub- component, short summary). Eval: LLM judge or keyword matching for per-field coverage and factual consistency. HardSource: an analytical report with trend judge- ments, drivers, risks, and outlooks. Task: rea- son about causes of change and what comes next (reasoning HIGH, dependence HIGH). Format: a structured schema (conclusion, key drivers, risks, uncertainties, evidence). Eval: LLM judge or keyword matching, with each claim traceable to a section, table, or figure. gorical state-identification instances, Medium sce- narios must yield summaries of temporal patterns, and Hard scenarios must yield report-grounded reasoning about causes or outlooks. Sources that pass the metadata checks in Step 2 but fail any execution-level check are rejected and recorded in the rejection log with the failure reason. A.3.2 Stage 2: Scenario Generation Step 4: Evidence collection. Taking each in- stance skeleton validated in Step 3 as input, the fourth step assembles the evidence state that an agent will see at evaluation time, which also serves as the traceable substrate for deriving ground truth. We preserve the raw source downloads and extract an evidence state comprising two parts: the cumula- tive time series analysis reports and the cumulative time series table. For periodt k , the report set con- tains all reports from the earliest available period throught k−1 , while the time series table contains all rows from the earliest available period through t k . The output is therefore a time-isolated evalua- tion state: the agent receives the cumulative table and the prior-period source documents, while the target-period report is withheld for ground-truth construction. Each scenario yields at least 24 such per-period states, one per instance. The evidence is accepted only if it passes two Table 5: Execution-level validity checks in Step 3. No.Validity check 1Period alignment. The time series value, source re- lease, and question instance refer to the same times- tamp, date, period, or update window. 2Minimum history. The source supports at least24 periods for backtesting and future live evaluation. 3Reference traceability. Time series analyses in re- leased reports are traceable to concrete public evidence, such as a field, label, value, table, figure, or report state- ment. 4Information isolation. The evidence package for pe- riodtuses only information available at or before the query time, excluding future releases, later revisions, and retrospective explanations. 5Objective evaluability. Time series analyses in re- leased reports can be derived by predefined rules and evaluated without relying on subjective preference. checks: (1) Completeness, ensuring sufficient cov- erage, temporal continuity, and plausible numeric scale; and (2) Column hygiene, keeping only gen- uine raw measurements while removing duplicate, constant, textual, rescaled, or already-derived fields unless explicitly allowed. Step 5: Question generation. Using the evi- dence from Step 4 and the assigned difficulty tier, Step 5 defines a scenario-level evaluation contract: a fixed question template, answer schema, and scor- ing rule shared across all periods. Concretely, we use an LLM (Claude Opus 4.6) to match each ques- tion intent in Table 6 against the time series analy- sis reports and extract the corresponding analysis paragraphs, which the next step transforms into the answer schema and scoring rule listed in Table 7. In the meantime, the question coverage in dif- ficulty tier is enforced: Easy uses at least one A- group tag; Medium uses at least three tags across A/B/C/D plus a universal summary; and Hard uses at least three tags across E/F/D plus a summary and traceable evidence. Before admission, each contract must pass three checks: (1) recurrence across sampled and full periods, (2) synchroniza- tion among tags, schema fields, and scoring rules, and (3) artifact grounding against the cumulative ta- ble, instance folder, and answer schema. Contracts that fail any check are rejected. Step 6: Ground-truth extraction. Step 6 pro- duces a reproducible, leak-free canonical answer. For tasks whose answers can be derived directly from the time series, such as state identification or change detection, we use an LLM (Claude Opus Table 6: Question taxonomy in step 5: 27 fine-grained question types in six main categories (A–F). IDQuestion intent A — State identification A1The current categorical state of the target. A2The latest reported value. A3The direction of the latest reading (up / down / flat). B — Change detection B1The change from the previous period. B2The direction of the period-over-period change. B3The magnitude of the recent multi-period trend. B4How many consecutive periods the trend has lasted. B5The deviation from a declared baseline (e.g., five-year average). B6Whether this period is a record extremum within a time window. B7The ordinal rank of this period within its history. C — Composition identification C1The set of leading contributors (top-N ). C2The single dominant contributor. C3The breakdown of the whole into components with their values or shares. C4The assignment of entities into named buckets. D — Event detection D1The anomalies or unusual values flagged this period. D2The classification or enumeration of discrete events. E — Attribution E1The single cause attributed to the change. E2The primary driver among several. E3The secondary drivers. E4The set of drivers, each paired with traceable evidence. E5The stakeholders affected by the change. E6The issuer’s forward guidance or committed response. E7A comparison with a named earlier period or season. F — Outlook F1The forward risks, paired with traceable evidence. F2The direction, regime, or tone of the outlook. F3The outlook with its confidence and horizon. F4A conditional outlook (“if X, then Y”). Table 7: Scoring-rule assignment by question type in step 5. Rule details are defined in Table 11. Scoring ruleQuestion types Atomic rules Label matchA1, A3, B2, B6, C2, D2, E1, E2, E5, F2 Numeric checkA2, B1, B3, B4, B5, B7 Set overlapC1, C4, D1, D2, E3, E5 LLM judgmentE1, period summary Composition rules Joint scoringE6, E7, F3, F4 Item-wise scoringC3, E4, F1 4.6) to generate executable code (e.g., a rule-based state identification script or a change-rate calcula- tion) that yields verifiable answers. For tasks whose answers must be inferred jointly from textual con- text and the time series, we also use the LLM to transform the evidence extracted in Step 5 into the structured schema. This procedure constructs the ground truth from the given questions under the fixed contract, yielding a verifiable per-period task. The target-period release is used only for ground- truth construction and verification, never as agent- visible evidence. Thus, each field is either rule- reproducible from the cumulative table or explic- itly grounded in the withheld source. Finally, the prompt is rendered from the same scenario contract, specifying the task, inputs, rules, requested fields, and JSON schema. A.3.3 Stage 3: Quality Control Step 7: Grounding audit.Step 7 verifies that the question, ground-truth, evidence, data, and source documents are mutually consistent and solvable un- der the intended information boundary. The audit applies eight gates covering schema, rules, data, evidence, and cross-instance consistency (Table 8); a scenario passes only when all gates have zero- failed outcomes. Table 8: The eight grounding-audit gates in Step 7. GateGrounding-audit check 1Schema conformance. Ground truth validates against the answer schema; its top-level keys equal the required set, and enum values stay within the declared options. 2 File completeness. Each instance carries its question, ground truth, evidence, time series, and schema files. 3Question–artifact consistency. Columns, files, and period labels named in the question exist and match the data; output-example keys and enum options equal the schema. 4Rule reproducibility. The deterministic derivation rule recomputes the stored ground truth (report-grounded fields exempt). 5Time series health. The cumulative table is strictly ascending in time. 6Evidence–data consistency. Numeric and key–value references in the evidence resolve to table cells, and ground-truth enum strings appear in the evidence. 7Quote grounding. Every verbatim quote in the support- ing evidence is a substring of the cited source document. 8Cross-instance health. Instance count matches the scenario, ground-truth fields vary across periods, and each table’s last row matches its folder period. Step 8: Human check. The final step adjudi- cates audit warnings and catches design viola- tions that are difficult to formalize. Human re- viewers decide whether each warning is by de- sign, re-check artifact quality against the full gate of Table 9, and remove leakage references. By- design warnings may be released, such as hard-tier evidence warnings, whitelisted constants, or be- nign period-formatting differences, whereas sce- narios are blocked or quarantined if they con- tain target-period leakage, revision-framing leak- age, column-hygiene defects, or instance-count de- fects. After the human check, accepted scenarios are promoted into the benchmark, portfolio statis- tics are recomputed, and the termination condition total ≥ 60∧ domains ≥ 5∧ min(tier) ≥ 10 is re-checked; if it is not met, the loop returns to Step 1 for the next domain pass. Table 9: The nine-section artifact quality gate in Step 8. No.Artifact quality check 1Time series completeness. Core columns are popu- lated, the date axis is continuous, and there are at least 24 rows (cf. Step 4). 2Column hygiene. No duplicate date parts, rescaled duplicates, constant columns, embedded narrative, or agent-derivable columns. 3GT leakage. No column’s latest value, value set, or sign restates the ground-truth answer (Easy categorical state exempted). 4 Difficulty–task alignment. Easy reads a latest-row state; Medium summarises across time from a brief; Hard reasons causally from a report, with no analytical conclusions baked into the table. 5 Question non-hallucination. Referenced fields and windows exist; the question recurs across at least three periods and instantiates over all periods; difficulty matches the tier. 6Schema–template consistency. Tier and domain match the directory, Easy options equal the schema enum, and declared columns equal the table header. 7Directory compliance. Required scenario, environ- ment, script, source, and per-instance files are present, with at least 24 instances. 8 GT traceability. The latest period’s evidence cites concrete source locations, and no field is inferable from the table alone (cf. Step 6). 9Information isolation. No table rows or evidence cita- tions postdate the instance period (V7). A.3.4 Human Annotations We recruit three human reviewers to evaluate the quality of TimeSage-EV. All reviewers are PhD students in Computer Science with at least full pro- fessional proficiency in English, and all consent to the use of their annotations for benchmark construc- tion and analysis. The reviewers examine scenario specifications, period-specific questions, structured ground truth, and supporting evidence. The re- view focuses on whether each question is clear and answerable under the cutoff, whether the ground- truth fields are schema-conformant and correct, and whether each answer is grounded in the cumulative time series or cutoff-valid source documents. Dis- agreements are resolved through discussion, and a scenario is released only after both automatic checks and human review pass. The following are the detailed instructions to guide the reviewers: Thank you for helping verify the quality of TimeSage-EV, a benchmark for recurring time series analysis under a temporal cutoff. Please follow the instructions below. For each scenario you will receive: a scenario specification, period-specific questions, an answer schema, the ground truth (ground_truth.json) with its evidence (gt_evidence.md), the cumulative timeseries.csv, and any prior-period source documents. Instructions 1. Review without help from other people, answer-generation tools, or search engines beyond the provided documents. There is no time limit. 2. For each scenario, check: (a) Clarity: Is the question clear, recurring, and answerable for the target period? (b) Cutoff validity: Can the answer be obtained without the target-period source or any future evidence? (c) Schema conformance: Do all ground-truth fields match the schema and allowed values? (d) Correctness: Are the structured answers correct for the target period? (e) Grounding: Is every field supported by the time series or cutoff-valid documents? (f) Variation: Does the scenario vary across periods enough for recurring evaluation? 3. If a scenario is unclear, incorrect, or ambiguous, flag it with the affected period, field, and a short explanation. 4. Return one of: PASS, PASS_WITH_MINOR_FIXES, or FAIL. Thank you for your contributions! A.3.5 Released Sources After the construction loop, Table 10 lists the sources that pass metadata filtering, execution val- idation, ground-truth extraction, automatic audits, and human review. These sources form the released TimeSage-EV corpus, which contains60scenarios across six domains, including agriculture, climate, energy, finance, healthcare, and transportation, and three tiers, with 20 scenarios per tier. Table 10: Public data sources for all 60 TimeSage sce- narios, grouped by domain. Tier: E = easy, M = middle, H = hard. Scenario names are shortened to the shortest unique prefix. T ScenarioCad. Source Agriculture E usda_cattlemon. esmis.nal.usda.gov E usda_chickenmon. esmis.nal.usda.gov E usda_cold_storage_ butter mon. esmis.nal.usda.gov E usda_cold_storage_ pork mon. esmis.nal.usda.gov M usda_cropwkly. esmis.nal.usda.gov M usda_droughtwkly. droughtmonitor.unl.edu M usda_wasdemon. usda.gov H fao_foodmon. fao.org H usda_fasmon. apps.fas.usda.gov H usda_usmon. ams.usda.gov; fred. stlouisfed.org Climate E noaa_cpc_aomon. cpc.ncep.noaa.gov E noaa_cpc_mjodaily psl.noaa.gov E noaa_cpc_onimon. cpc.ncep.noaa.gov E noaa_cpc_pnamon. cpc.ncep.noaa.gov M c3s_copernicusmon. climate.copernicus.eu M noaa_ncei_globalmon. ncei.noaa.gov M noaa_ncei_usmon. ncei.noaa.gov H iri_ensomon. iri.columbia.edu H noaa_cpc_ensomon. cpc.ncep.noaa.gov H noaa_cpc_monthlymon. cpc.ncep.noaa.gov Energy E eia_refinerywkly. eia.gov E eia_us_commercialwkly. eia.gov E eia_wtidaily fred.stlouisfed.org M eia_natural_gas_ monthly mon. eia.gov M eia_us_petroleummon. eia.gov M eia_wpsrwkly. eia.gov H eia_monthlymon. eia.gov H eia_natural_gas_ weekly wkly. eia.gov H iea_oilmon. iea.org H ief_comparativemon. ief.org Finance E cboe_vixdaily fred.stlouisfed.org E fred_sahmmon. fred.stlouisfed.org E fred_usdaily fred.stlouisfed.org M dallas_fedmon. dallasfed.org M fred_consumermon. fred.stlouisfed.org M philly_fedmon. philadelphiafed.org H bea_pcemon. fred.stlouisfed.org H boe_monetary8/yr bankofengland.co.uk H ecb_monetary8/yr ecb.europa.eu H frb_usmon. fred.stlouisfed.org Healthcare E cdc_nwsswkly. data.cdc.gov E cdc_uswkly. data.cdc.gov E hhs_uswkly. data.cdc.gov M bls_joltsmon. fred.stlouisfed.org M bls_usmon. fred.stlouisfed.org M ecdc_cdtrwkly. ecdc.europa.eu M paho_measlesbiwk. paho.org H bls_medicalmon. fred.stlouisfed.org H cdc_respwkly. data.cdc.gov H who_choleramon. who.int Transportation E bls_dieselwkly. fred.stlouisfed.org E bts_airlinemon. fred.stlouisfed.org E tsa_checkpointdaily tsa.gov M ata_truckmon. fred.stlouisfed.org M bls_ppimon. fred.stlouisfed.org M eurocontrol_monthlymon. eurocontrol.int M portla_monthlymon. portoflosangeles.org H cass_freightmon. cassinfo.com H freightos_weeklywkly. freightos.com H iata_airmon. iata.org Figure 11: Per-Scenario Answer Volatility Across Do- mains. We quantify how much each environment evolves over time using the per-period answer tran- sition rate (Figure 11), which measures how often a scenario’s structured answer fields change between consecutive periods. Across all 60 scenarios the rate spans 0.13 to 0.91, confirming that temporal volatility is not uniform but a scenario-level prop- erty the benchmark deliberately varies. A.3.6 Scenario Examples To illustrate the range of tasks covered by our benchmark, Figures 12–14 present three represen- tative scenarios spanning the Easy, Medium, and Hard difficulty tiers. Each scenario casts the model as a domain analyst who must read a cumulative time-series snapshot and analysis reports and an- swer a structured set of questions about the latest reporting period. The Easy scenario (Weekly US Diesel Price- Band Classification) in Figure 12 asks the model to read the most recent row of an EIA/FRED diesel price series and assign it to one of four fixed price bands, followed by a brief summary. This is a task that relies on data lookup and threshold logic. The Medium scenario (Monthly Healthcare La- bor Market Analysis) in Figure 13 draws on BLS JOLTS data for the Health Care and Social Assis- tance sector and additionally requires the model to compute month-over-month and year-over-year changes, infer their direction, and synthesize a short digest, thereby introducing multi-step quanti- tative reasoning over the time series. The Hard scenario (Monthly ENSO Climate Analysis), shown in Figure 14, is the most open- ended. Working from NOAA CPC’s monthly Figure 12: Easy scenario: Weekly US Diesel Price- Band Classification. Figure 13: Medium scenario: Monthly Healthcare La- bor Market Analysis. Figure 14: Hard scenario: Monthly ENSO Climate Analysis. ENSO Diagnostic Discussion Reports alongside the Niño 3.4 ONI time series history, the model must report six things: the current alert-system status, a three-month outlook, the dominant ocean–atmosphere drivers, any flagged uncertain- ties, event tokens drawn from a closed vocabulary, and a grounded summary with supporting evidence. A.4 Evaluation Methodology We evaluate each agent run along four independent axes to disentangle answer correctness from re- port quality: structured-answer accuracy on the one hand, and report coverage, faithfulness, and writ- ing quality on the other. This section first defines the evaluation unit in Section A.4.1, then detail the evaluation method for each axis in Section A.4.2. A.4.1 Evaluation Unit The atomic unit of evaluation is a (scenario, period) instance. For each instance, the agent produces two artifacts: a structured answer that fills the scenario answer schema and a free-form report that summa- rizes the answer. The answer schema decomposes each period time series analysis into a set of key- points corresponding to the elementary facts that a competent analysis must establish. The report serves as an analytical summary, integrating these keypoints and their evidence into human-readable text. We evaluate the two artifacts along four axes, each scored on a0–100scale: keypoint accuracy (ACC), keypoint coverage (COV), report faithful- ness (FAITH), and report quality (QUAL). Averag- ing them gives the overall instance score: AGG = mean ACC, COV, FAITH, QUAL . A.4.2 The Four Evaluation Axes Keypoint accuracy. This axis measures how well the structured answer matches the ground truth (GT). Each keypoint is scored by comparing the agent-provided value with the GT value accord- ing to the answer-type-specific rules in Table 11. The match score reflects the degree of agreement between the two values, and the final score is the average across keypoints, rescaled to 0–100. Keypoint coverage. This axis measures the re- port’s completeness: whether it communicates the content of each required keypoint. For each key- point, the judge assigns one of three labels: YES, PARTIAL or NO, which are scored as1.0,0.5and 0.0, respectively. The axis score is obtained by Table 11: Primitive match rules. Each rule compares the agent outputawith the ground-truth valueb, or output set A with ground-truth set B. RuleMatch score Label match 1 if a = b, else 0. Boolean match 1 if bool(a) = bool(b), else 0. Text match1 if stripped strings are equal, else 0. Numeric check 1 if|a−b|≤ τ abs , or|a−b|/|b|≤ τ rel when relative tolerance is set andb ̸= 0; else0. With no tolerance, exact equality is required. Set overlapDice/F1 overlap2|A∩B|/(|A| +|B|); two empty sets score1, and a single empty set scores 0. averaging these keypoint-level scores within the instance and rescaling the result to the range0–100. During the evaluation, the easy and medium tiers contain no reasoning questions. Therefore, a key- point anchored to a ground-truth scalar is marked covered without an LLM call. In contrast, the hard- tier uses the LLM-based evaluation path, since a verbatim scalar match alone does not guarantee that the underlying concept has been conveyed. The coverage judge prompt is given in Appendix A.5.2. Report faithfulness. This axis measures factual groundedness: the fraction of the report’s atomic claims supported by the sources. The judge first decomposes the report into a small set of atomic single-fact claims and then verifies each against a layered reference: the scenario keypoints (level 1), excerpts of scenario context (level 2), and general world knowledge (level 3). A claim grounded at any level counts as supported; claims that contra- dict a keypoint or are mere speculation count as unsupported. The axis is the fraction of supported claims, on a scale of0–100. The judge prompts are presented in the Appendix A.5.2. Report quality.This axis measures writing qual- ity alone, namely genre conformance, structure, and readability. The judge scores the report against the scenario tier’s rubric, which lists the genre, the signals a good report should contain, those it should avoid, and an exemplar; it marks each required sig- nal as fully hit, partially hit, or missed, and flags any forbidden signal. The axis rewards hits, half- credits partial hits, deducts violations, and normal- izes by the number of required signals, on a scale0– -100. Factual errors are not penalized here since the other axes handle them. The quality judge prompt is given in the Appendix A.5.2. A.4.3 Evaluation Examples Figure 15 shows a successful run on the BTS Air- line Load Factor scenario. The agent correctly fills the structured answer (is_year _high _load _factor= false), matching the ground truth and producing a perfect keypoint precision. Its report communicates every required keypoint—the 80.5% load factor for 2024-11 and the negative high-load flag—so coverage is also full. In faithfulness, each atomic claim is grounded either in the keypoints (TS) or the scenario context (DOC), with no unsup- ported speculation. The report additionally hits all required quality signals while avoiding forbidden ones, such as causal attribution or fabricated press- release citations. All four axes score100, giving an aggregate of AGG = 100. Figure 16 shows a failure on the WASDE Crop Balance scenario, where the agent over-commits to a single commodity. It answerscornfor the dom- inant U.S. commodity (GT:mixed) and reports a concrete ending stock level where the ground truth isN / A, so several precision keypoints score0; onlykp _us _mom _pctmatches. Coverage is cor- respondingly partial: the report conveys the level and month-over-month keypoints, but omits the event/anomaly callouts and the correct dominant- commodity framing. Faithfulness remains high, as the stated claims are individually grounded in the keypoints or context. Quality is satisfied but only half-credited on the signal requiring both the lat- est level and the signed monthly change. The axis scores (20,60,100,87.5) average toAGG = 66.88, illustrating how the four axes jointly separate an- swer correctness from report quality. A.5 Prompt Templates This appendix reports the prompt templates used by TimeSage-EV. Scenario-specific prompts are generated from these templates and the scenario specification. The fully rendered period prompts for all 1,485 period-level instances are released with the benchmark files. A.5.1 Agent Prompt System and sandbox instruction. At each pe- riod, the agent receives a task prompt together with the following sandbox instruction. scenario spe- cific fields such as required answer keys, tier, and optional web-search availability are filled out from the scenario specification. Figure 15: Successful case of agent output. Figure 16: Failed case of agent output. You are a markets/data analyst. All files you need are in the CURRENT WORKING DIRECTORY. The cwd contains: * timeseries.csv -- the historical data series * answer_schema.json -- required JSON shape for your answer * gt_source/<prior>.* -- prior-period reports (PDF/HTML/JSON) Available tools: * Python code execution * final_answer(answer=<dict>, report=<str>) -- submit your final output. You must call this tool exactly once to finish. Do not write the answer to a file. Do not emit a final textual reply instead of calling the tool. * web_search(query: str, n: int=5) -- audited, time-isolated, enabled only for scenarios with audited cutoff-valid retrieval. Sandbox rules: Authorized imports: pandas, numpy, json, csv, re, math, statistics, datetime, pathlib, io, collections, itertools, functools, pypdf. Forbidden builtins: open, eval, exec, compile, __import__, globals, locals. Forbidden modules include os, sys, subprocess, socket, urllib, requests. File / IO replacements: * Read text: Path('x.txt').read_text() * Read JSON: json.loads(Path('x.json').read_text()) * Read CSV: pd.read_csv('x.csv') * Read PDF: PdfReader('x.pdf').pages[0].extract_text() * List dir: [p.name for p in Path.cwd().iterdir()] Scenario tier: tier Required answer keys: required_keys Task rules: 1. Read the CSV and prior-period gt_source files if needed. Do not hallucinate values. 2. Never cite or read the current period's gt_source. Only prior-period source documents are allowed. 3. Your final answer dict must contain exactly the required keys. Your report must cite specific CSV cells or prior-period files. 4. End your run by calling final_answer(answer=..., report=...). Memory instruction. For sequential evaluation, the following memory instruction is appended. Memory management is optional in multi-period episodes. If your conversation memory becomes bloated with failed code attempts, long observations, or patterns already learned, you may call request_memory_compaction(reason=...). The framework will summarize what you have learned across past periods after the current period finishes, then reset memory to that summary before the next period. Do not call this for a single-period task. Skill prompt.When the skill library is enabled, a skill-index block is prepended to each period’s task prompt. The block has two forms, depending on whether the library is empty or populated. An agent may save a reusable function withinduct_skill and inspect the full body of a saved skill with retrieve_skill; saved skills become callable as live tools starting the next period. When the library is empty, the following seeding nudge is shown. No skills have been saved yet for this sweep. You are the first agent to run. If you write a small, reusable idiom this period (a schema-specific CSV loader, a date-to-period converter, a band classifier, a row lookup), save it with induct_skill so later periods can call it as a tool. Good candidates: small single-purpose functions returning a clean Python value. Bad candidates: scenario-specific one-off math, or the literal answer dict. Induct the skill BEFORE calling final_answer, while the code is fresh. When there is at least one skill, the reuse- enforcement form is shown instead. === Available skills === The following skills are already registered as tools for this period. INVOKE THESE SKILLS -- do NOT re-implement a listed skill inline. Reuse is faster, less error-prone, and consistent across periods. Example call: result = first_skill_name(...) Skills: - skill_name -- skill_description ... Use retrieve_skill(name=...) to read a skill's notes and full code before invoking if its description is not clear enough. If you find a new reusable idiom this period, save it with induct_skill. Period question prompt. Each period-level question is rendered from the scenario’s question template. It is now period_label. I need an evidence-based analysis for this period's release_name. Use /root/timeseries.csv and any prior-period files in /root/gt_source/. The current-period source document is not available. Please answer the following questions: 1. field_question_1 2. field_question_2 ... N. field_question_N Return a JSON object conforming to /root/answer_schema.json and a concise natural-language report grounded in cutoff-valid evidence. A.5.2 Evaluation Prompts Keypoint coverage judge. For report coverage, deterministic literal matching is used when possi- ble. Otherwise, the LLM judge receives the follow- ing prompt for each keypoint. You are checking whether a report communicates a predefined content unit (ACU). Reply with exactly one of: YES, PARTIAL, NO. Content unit: description: keypoint_description verification question: verification_question Report: agent_report Reply with the single word YES, PARTIAL, or NO and nothing else. Report faithfulness decomposition. For report faithfulness, the report is first decomposed into atomic claims. Decompose this report into AT MOST 10 atomic, verifiable claims. Each claim must be ONE scalar fact: one number, one named entity, one date, one direction, one classification -- never a compound sentence with multiple facts. Skip markdown headers, bullet labels, rhetorical sentences, hedges, and definitional preambles. Each claim should be standalone and grounded enough to verify against a data source. Return a JSON list of strings, no prose around it. Report: agent_report Report faithfulness verification. Each atomic claim is then verified against cutoff-valid reference sources. Verify a single atomic factual claim against reference sources. Claim: atomic_claim Reference sources: reference_sources Pick exactly one verdict from this set: tier1_supported (keypoint says so), tier2_supported (context says so), tier3_general_knowledge (only if the claim is encyclopedic and not contradicting any reference), contradicts_keypoint (clearly conflicts with a keypoint), speculation (unsupported by any reference). Return JSON: "verdict": "<one>" Report quality judge. Report quality is scored separately from factual correctness. You are scoring a report's WRITING quality only -- its genre conformance, structure, and readability. DO NOT score for factual correctness; that is evaluated separately by a different scorer. Tier genre: tier_genre Required signals: - required_signal_1 - required_signal_2 ... Forbidden signals: - forbidden_signal_1 - forbidden_signal_2 ... Exemplar of a good report: quality_exemplar Report to score: agent_report Return JSON: "required_hits": ["<signal>", ...], "required_partial": ["<signal>", ...], "required_misses": ["<signal>", ...], "forbidden_violations": ["<signal>", ...], "rationale": "<one sentence>" A.6 Agent Harnesses A.6.1 Baseline Harness: smolagent Agent framework. All agents are built on smolagents, a lightweight framework in which an LLM drives a multi-step loop of tool calls, each step appended to a running memory of prior actions and observations. We use its code-agent variant, where the model acts by writing executable Python rather than emitting structured tool-call arguments, which suits the data-manipulation nature of time series analysis and exposes a single uniform action channel across all models. Action surface. The baseline agent is a smolagentCode Agent in which Python is the action language: the agent emits code, the sandbox executes it, and the resulting observation becomes the next turn. Imports are restricted to a whitelist of analytical libraries, file access is mediated through path-based readers rather than raw file handles, and all networking modules are excluded by design, so the only sanctioned external access is the audited web-search tool. No default tools are added beyond those the protocol requires: the agent can submit its output, request memory compaction, and issue web searches. Memory management. Memory management uses a hybrid policy: a compaction is scheduled at the next period boundary whenever the agent re- quests that one or the most recent step’s input-token count exceed a fixed threshold. When triggered, the harness issues a single call to the agent’s own model to summarize the CSV layout, working code idioms, schema pitfalls, feedback patterns, and a one-line recap per past period, then re-seeds the conversation with that summary. Internet access. Our harness blocks all network libraries (e.g.,urllib,requests,socket). To preserve a broad action space, we expose a web- search tool as the sole network channel, which returns only historical archived snapshots through a three-stage pipeline: (1) Candidate discovery: SearXNG 3 resolves the query into a set of candi- date URLs; (2) Archive retrieval: each URL is queried against Common Crawl 4 with server-side temporal filtering, so that only captures predating the cutoff are returned; (3) Fallback: if no qualify- ing capture exists, the Wayback Machine 5 supplies the snapshot closest to the cutoff, which is dis- carded if dated after it. These stages guarantee that every retrieved page was archived strictly before the cutoff date. A.6.2 Self-Evolving Harness: TimeSage-1.0 WhySelf-Evolve. TimeSage-EVevaluates agents on a recurring analytical task whose evidence changes over time, raising a question that single-period evaluation cannot answer: can an agent improve future analyses by converting feedback and repeated reasoning patterns from earlier periods into reusable procedures?A standardsmolagentharness with tool use and memory carries previous context, but mixes reusable routines with transient observations, failed code attempts, and period-specific facts, so memory alone cannot distinguish a generally useful workflow from merely accumulated history. A diagnostic harness. TimeSage-1.0 addresses this question by keeping the agent interface, model configuration, task exposure, time budget, and eval- uation protocol aligned with the baseline, while adding only a lightweight self-evolving skill-library manager. This isolates the effect of reusable skills from that of changing the base model or supplying additional evidence. It is therefore a diagnostic con- dition rather than an optimized production agent, measuring whether agents can induce, validate, re- trieve, and reuse compact analysis routines across the repeated release periods of a scenario. Skill-library tools. TimeSage-1.0 augments the baselinesmolagentharness with a skill-library layer, leaving the sandbox, inputs, and required arti- facts unchanged. It adds two tools:induct_skill saves a small reusable Python function with a name, description, and notes when the agent identifies a recurring analysis idiom, andretrieve_skillre- turns the full body of a saved skill when its short description is insufficient for reliable use. 3 https://searxng.org 4 https://commoncrawl.org 5 https://web.archive.org Chronological registration.Accepted skills be- come callable tools from the next period, so a skill induced in the periodt k cannot alter the current answer but can affect behavior int k+1 and beyond. This preserves the chronological structure of the evolving environment. Prompting and reuse. The library is exposed through a period-level skill-index block. When empty, it seeds the agent toward compact reusable idioms, such as schema-specific loaders or row lookups, and away from one-off arithmetic and literal answer dictionaries. Once non-empty, the block lists the available skills, registers them as tools, and instructs the agent to invoke an existing skill rather than re-implement it inline. The prompt templates are given in the Appendix A.5. Lightweight quality control. Before a skill is exposed in later periods, a check enforces that it is a callable, self-contained routine with a usable interface and a faithful description, filtering out candidates that are overly period-specific, dupli- cate an existing skill, or encode a literal answer. The check is deliberately minimal, keeping the li- brary usable during evaluation without turning skill construction into a separate supervised process. A.7 Additional Results A.7.1 Breakdown of Main Results Table 12 provides a detailed breakdown of the performance of the model by difficulty tier, com- plementing the aggregate results reported in the main text. For each model, we report scores along four evaluation dimensions (Accuracy, Coverage, Faithfulness, and Quality) as well as the Overall score, separately for the Easy, Medium, and Hard tiers. Cell shading is computed within each tier- and-metric group, so colors are comparable across models within a column but not across tiers; the best entry in each group is shown in bold. Two patterns are worth highlighting. First, per- formance degrades consistently from Easy to Hard across all models and dimensions: while most mod- els exceed 90 Overall on the Easy tier, scores drop substantially on the Hard tier (from 72.70 for GPT- 5.4 down to 53.02 for Gemma-4-31B), indicating that the harder questions remain a discriminating factor. Second, the dimension-level breakdown re- veals that this degradation is driven primarily by Accuracy and Coverage, whereas Faithfulness and Quality remain comparatively stable. This suggests Table 12: Per-difficulty-tier scores across evaluation dimensions. Cell shading uses a shared scale within each tier-and-metric group (lowhigh); the best result in each group is shown in bold. ModelTierAccuracyCoverageFaithfulnessQualityOverall GPT-5.4 Overall 84.8977.3091.2090.5885.99 Easy 98.5498.6796.1293.0496.59 Medium93.4476.7996.6487.2888.54 Hard 62.0956.5880.4491.6972.70 Qwen-3.5-397B Overall 84.7275.5788.6687.9984.24 Easy 98.6199.8692.5097.6197.14 Medium92.4878.5291.8682.7986.41 Hard 62.5348.2181.3884.0169.03 Sonnet-4.6 Overall 82.7674.3284.4687.9782.38 Easy99.4797.9993.9597.1997.15 Medium89.0375.9187.2182.4383.65 Hard 59.3649.0372.0484.7566.29 Kimi-K2.6 Overall77.8870.0781.7481.9077.90 Easy98.7598.6594.5997.9797.49 Medium 82.8969.8881.2274.4077.10 Hard51.6841.8269.5274.0059.25 Devstral-2-123B Overall74.1469.0884.4381.3777.26 Easy 89.7092.2893.8293.3092.27 Medium73.1166.8179.3467.0371.57 Hard59.7648.4480.5684.9668.43 Gemma-4-31B Overall75.0569.0880.8776.2575.31 Easy 95.0399.4293.8291.3794.91 Medium82.2370.7683.7974.4877.81 Hard47.4337.0764.8263.0953.10 that on difficult inputs models tend to produce well- formed and faithful outputs that nonetheless miss or misidentify the correct content. GPT-5.4 and Qwen-3.5-397B are the strongest overall, while the gap between models widens on the Hard tier. A.7.2 LLM Judge Verification To validate the reliability of our LLM judge, we conducted a human–judge agreement study on 300 instances (5 sampled periods per scenario) of GPT- 5.4’s agent outputs scored by DeepSeek-V4-Flash. Each instance was independently labeled by a hu- man annotator—given the question, reference, and prediction—using the same discrete label space as the LLM judge: coverage (YES/PARTIAL/NO per keypoint), faithfulness (five fixed verdicts per atomic claim), and quality (hit/partial/miss/viola- tion per signal). As shown in Table 13, coverage Table 13: Human–LLM judge agreement across the three evaluation axes. Evaluation Axis Agreement Rate Cohen’s κ Coverage94.5%0.882 Faithfulness98.4%0.917 Quality76.0%0.665 and faithfulness exhibit strong alignment with hu- man judgments, while quality agreement is lower yet still substantial. It is likely because quality as- sessment is inherently more subjective, whereas the other two axes follow clearer, more objective rubrics. These results indicate that the LLM-judge setup can serve as a reliable proxy for human eval- uation under our rubric. A.8 AI Usage We used AI assistance during the writing and edit- ing process to improve the clarity, organization, and wording of the manuscript. The authors controlled scientific content, benchmark design, experimental analysis, and final writing decisions. AI tools were not used to generate benchmark ground truth with- out automatic and human verification. All reported results, figures, and tables are based on the authors’ benchmark artifacts and evaluation pipeline.

Instant research discovery