Paper deep dive
MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs
Zhi Lei, Chenxi Liu, Hao Miao, Wanghui Qiu, Bin Yang, Chenjuan Guo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:15:15 AM
Summary
MM-ISTS is a novel multimodal framework for Irregularly Sampled Time Series (ISTS) forecasting that integrates vision-text Large Language Models (MLLMs) with a specialized ISTS encoding module. It addresses the modality gap and irregular sampling challenges by converting ISTS into irregularity-aware images and statistical text prompts, utilizing an adaptive query-based feature extractor to compress MLLM knowledge, and employing a modality-aware gating mechanism for robust fusion.
Entities (6)
Relation Signals (4)
MM-ISTS → forecasts → ISTS
confidence 100% · MM-ISTS, a multimodal framework... facilitating ISTS forecasting
MM-ISTS → utilizes → MLLMs
confidence 98% · MM-ISTS, a multimodal framework augmented by vision-text large language models
Adaptive Query-Based Feature Extractor → compresses → MLLMs
confidence 95% · Adaptive Query-Based Feature Extractor that compresses high-dimensional MLLM tokens
Cross-Modal Vision-Text Encoding → transforms → ISTS
confidence 95% · Cross-Modal Vision-Text Encoding module that transforms ISTS into irregularity-aware visual and textual representations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across variables. Existing ISTS forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we achieve MM-ISTS, a multimodal framework augmented by vision-text large language models, that bridges temporal, visual, and textual modalities, facilitating ISTS forecasting. MM-ISTS encompasses a novel two-stage encoding mechanism. In particular, a cross-modal vision-text encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a temporal-variable encoder. Further, we propose an adaptive query-based feature extractor to compress the learned tokens of MLLMs, filtering out small-scale useful knowledge, which in turn reduces computational costs. In addition, a multimodal alignment module with modality-aware gating is designed to alleviate the modality gap across ISTS, images, and text. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions.
Tags
Links
- Source: https://arxiv.org/abs/2603.05997v1
- Canonical: https://arxiv.org/abs/2603.05997v1
Trouble viewing inline? Open PDF directly →
Full Text
59,349 characters extracted from source content.
Expand or collapse full text
M-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs Zhi Lei zhilei@stu.ecnu.edu.cn East China Normal UniversityShanghaiChina , Chenxi Liu chenxi.liu@cair-cas.org.hk Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of SciencesHong KongChina , Hao Miao haom@cs.aau.dk Department of Computing, The Hong Kong Polytechnic UniversityHong KongChina , Wanghui Qiu onehui@stu.ecnu.edu.cn East China Normal UniversityShanghaiChina , Bin Yang byang@dase.ecnu.edu.cn East China Normal UniversityShanghaiChina and Chenjuan Guo cjguo@dase.ecnu.edu.cn East China Normal UniversityShanghaiChina (2018) Abstract. Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across variables. Existing ISTS forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we achieve M-ISTS, a multimodal framework augmented by vision-text large language models, that bridges temporal, visual, and textual modalities, facilitating ISTS forecasting. M-ISTS encompasses a novel two-stage encoding mechanism. In particular, a cross-modal vision-text encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a temporal-variable encoder. Further, we propose an adaptive query-based feature extractor to compress the learned tokens of MLLMs, filtering out small-scale useful knowledge, which in turn reduces computational costs. In addition, a multimodal alignment module with modality-aware gating is designed to alleviate the modality gap across ISTS, images, and text. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions. ISTS Forecasting, Multimodal LLMs, Cross-Modal Alignment †copyright: acmlicensed†journalyear: 2018†doi: X.X†conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY†isbn: 978-1-4503-X-X/2018/06 1. Introduction The expanding instrumentation of processes throughout society with sensors yields a proliferation of time series data in various domains such as healthcare (Goldberger et al., 2000; Reyna et al., 2019), transportation (Fan, 2022; Tang et al., 2020), climate science (Rubanova et al., 2019; Brouwer et al., 2019), and astronomy (Vio et al., 2013; Scargle, 1982). Existing time series forecasting methods (Liu et al., 2025a; Nie et al., 2023; Wu et al., 2023) mainly focus on fully observed data and cannot adapt to irregularly sampled time series (ISTS) (Zhang et al., 2025a, 2024; Mercatali et al., 2024; Luo et al., 2025), which are more common in real-world scenarios due to sensor malfunctions, network failure, and varying sampling sources. ISTS exhibits asynchronous observations on non-uniform time intervals across variables (Schirmer et al., 2022), making it difficult to achieve accurate forecasting for potential informed decision-making (Li et al., 2025). Existing ISTS forecasting methods can be generally categorized into three paradigms based on how to model temporal irregularities. The first category focuses on continuous-time modeling (Zhang et al., 2025b; Oh et al., 2024), which often utilizes differential equations or state-space models to naturally handle uneven time gaps. The second category methods employ geometric deep learning (Yalavarthi et al., 2024; Li et al., 2025) or patch-based modeling (Zhang et al., 2024; Luo et al., 2025) to capture dependencies among asynchronous observations via graph connectivity or segmented temporal tokens, which often involve aggregation or pooling operations that may obscure fine-grained temporal patterns. Recently, another line of methods has emerged that applies pre-trained language models (PLMs) (Zhang et al., 2025a) for ISTS forecasting due to the generalization capabilities of PLMs. Despite these advancements, most existing methods remain confined to a single modality and rely solely on historical observations. These methods often overlook the rich semantic information and fine-grained temporal patterns. Recent studies demonstrate that additional modalities, such as text and images, are capable of providing complementary information to facilitate time series modeling (Zhong et al., 2025; Jin et al., 2024). Specifically, the textual data often contains contextual descriptions and dataset statistics, which can enhance the understanding of time series patterns. Thus, prompt-based methods (Liu et al., 2025a; Jin et al., 2024) emerge by mapping time series into prompts to help the LLMs understand the time series in depth. These methods often focus on addressing the modality gap between continuous time series and discrete text, aiming to alleviate information misalignment (Liu et al., 2025b). However, these methods struggle to capture fine-grained temporal patterns, i.e., the ability to learn subtle dynamics, which are particularly important for ISTS forecasting to alleviate the influence of irregularity. More recent studies address this problem by converting time series into their visual versions, such as line graphs or gray-scale images, enabling spatial pattern capturing, which is embedded in time series (Chen et al., ). Nonetheless, these vision-based methods fall short in learning contextual semantics, since they fail to incorporate domain-specific knowledge. As a result, we need a new kind of method that can bridge the temporal observations, textual data, and images. However, it is non-trivial to develop this kind of model, due to the following challenges. Although multimodal LLMs (MLLMs) offer a promising means to bridge this gap, leveraging their general reasoning capabilities, it is challenging to utilize MLLMs to ISTS. First, a significant representational discrepancy exists between sparse ISTS and the dense inputs required by MLLMs. Naive conversion methods, such as converting time series into standard images or plain text, may not capture the critical uneven time intervals or learn temporal patterns with missing observations. For instance, standard image resizing may distort temporal scales, while linear text serialization may lose the structural correlation across variables. Second, it is challenging to alleviate the modality gap, aligning the numerical temporal observations, text, and images. Irregular numerical observations often require high precision, whereas MLLMs operate on a coarse-grained semantic level (Zhong et al., 2025). It is urgent to invent an effective mechanism to align such threefold representations. To address these problems, we propose M-ISTS, which utilizes multimodal vision-text LLMs for ISTS forecasting. M-ISTS consists of four major components: the Cross-Modal Vision-Text Encoding module, the ISTS Encoding module, the Adaptive Query-Based Feature Extractor, and the Multimodal Alignment. Specifically, M-ISTS employs a Cross-Modal Vision-Text Encoding module to automatically transform ISTS to vision and text modality, which can preserve irregularity to the utmost. To better understand temporal correlations, we convert the original ISTS into 3-channel images, with channels for raw observational values, missingness masks, and time intervals, enabling the MLLMs to distinguish missing data. Further, we generate descriptive textual prompts with statistics (e.g., missing rates, variable ranges) and domain knowledge to enhance the reasoning capabilities of MLLMs. Moreover, to effectively capture temporal dynamics, we propose a customized ISTS Encoding module in parallel with the Vision-Text Encoding branch, which uses a multi-view embedding mechanism to project asynchronous observations, timestamps, and variable indices into a unified latent space. This is realized by a two-stage Transformer encoder that sequentially captures intra-series temporal dependencies and inter-series variable correlations, resulting in robust numerical representations. To efficiently align the high-dimensional semantic space of MLLMs with temporal features, we propose an Adaptive Query-Based Feature Extractor, which employs a set of learnable queries (Li et al., 2023) to interact with MLLMs with cross-modal alignment. This mechanism acts as an information bottleneck, compressing the vast visual-textual tokens into fixed-length representations aligned with temporal features, effectively filtering out redundant noise while reducing computational overhead during the fusion stage. Finally, a Multimodal Alignment Module is designed to fuse these three-fold multimodal representations. It includes a Modality-Aware Gating network that handles irregular data statistics (such as local sparsity levels and variance) to dynamically assign importance weights across the ISTS encoding module and cross-modal vision-text encoding module. This allows the model to leverage general knowledge from MLLMs when numerical data is missing or sparse, enabling accurate predictions with low-quality data. Our main contributions are summarized as follows: • To the best of our knowledge, we propose M-ISTS, the first multimodal ISTS forecasting framework augmented by vision-text LLMs. • We design a novel cross-modal vision-text encoding module that automatically converts ISTS into irregularity-aware images and prompt-enhanced text, with ISTS encoding extracting enriched temporal features. • An innovative adaptive query-based feature extractor is proposed to compress MLLM knowledge, including a modality-aware gating mechanism, which aligns heterogeneous multimodal features, mitigating the modality gap. • We report on experiments using real data, which offer evidence of the effectiveness of the proposals, showing that M-ISTS outperforms state-of-the-art baselines. 2. Related Work 2.1. Multi-modal Time Series Forecasting Recent studies have incorporated textual information (Liu et al., 2025c) for regular time series forecasting to enhance predictive performance beyond numerical signals. For example, Time-LLM (Jin et al., 2024) proposes a reprogramming framework that adapts large language models for general time series forecasting. It aligns time series and textual modalities by converting time series into text prototypes and introducing a Prompt-as-Prefix mechanism to guide the LLM in transforming time series patches for prediction. TimeCMA (Liu et al., 2025b) proposes a cross-modality alignment framework that aligns disentangled time-series embeddings with robust prompt-based embeddings from large language models. TimeKD (Liu et al., 2025a) introduces an cross-modal LLM-based framework for multivariate time series forecasting via privileged knowledge distillation. It uses a calibrated LLM teacher with privileged information to distill the learned knowledgeable representations into a lightweight student model. Beyond text and time series modalities, Time-VLM (Zhong et al., 2025) integrates extra vision modality by encoding time series as images and fusing the images, text, and time series with a vision-language model to enhance time series forecasting. Despite recent progress, multimodal large-model approaches are still primarily designed for regularly sampled time series (RSTS), while effective multimodal modeling for ISTS forecasting remains to be explored. 2.2. ISTS Forecasting Existing ISTS forecasting methods (Tashiro et al., 2021; Schirmer et al., 2022; Zhang et al., 2024; Mercatali et al., 2024; Luo et al., 2025) can be categorized into three categories. The first category focuses on modeling continuous temporal dynamics. Latent ODE (Rubanova et al., 2019) introduces continuous-time hidden state evolution via neural ordinary differential equations, while CRU (Schirmer et al., 2022) adopts a state-space formulation with Kalman filtering. The second category represents ISTS using relational or patch-based structures. GraFITi (Yalavarthi et al., 2024) formulates ISTS as a bipartite graph between variables and timestamps, T-PatchGNN (Zhang et al., 2024) segments ISTS into temporal patches and models dependencies via Transformers and GNNs, and HyperIMTS (Li et al., 2025) models ISTS with hypergraph structures. More recently, pre-trained large language models (LLMs) have been explored for ISTS forecasting. ISTS-PLM (Zhang et al., 2025a) investigates how representation schemes affect LLMs’ ability to model ISTS, and Time-IMM (Chang et al., 2025) introduces a benchmark focusing on realistic irregular sampling. Despite these advances, most existing approaches either concentrate on unimodal ISTS or lack global contextual information to adequately model complex and dynamic data patterns. 3. Preliminary 3.1. ISTS Definition We consider an irregularly-sampled time series (ISTS) consisting of N variables. For the n-th variable, its historical observations are denoted as on=(tin,xin)i=1Lno_n=\(t^n_i,x^n_i)\_i=1^L_n, where tin∈ℝt^n_i denotes the timestamp of the i-th observation and xin∈ℝx^n_i denotes the corresponding recorded value. The number of observations LnL_n may vary across variables, and the complete ISTS is denoted by =onn=1NO=\o_n\_n=1^N. 3.2. Canonical Representation In practice, a commonly adopted preprocessing strategy is the canonical pre-alignment representation (Che et al., 2018; Shukla and Marlin, 2021; Zhang et al., 2022, 2023; Rubanova et al., 2019; Brouwer et al., 2019; Bilos et al., 2021; Schirmer et al., 2022), which transforms the ISTS data O into a triplet (,,ℳ)(T,X,M). Here, =[tl]l=1L∈ℝLT=[t_l]_l=1^L ^L denotes the ordered set of unique timestamps obtained by merging all observation times across variables, i.e., =⋃n=1Ntini=1LnT= _n=1^N\t^n_i\_i=1^L_n. The value matrix =[[xln]n=1N]l=1L∈ℝL×NX=[[x^n_l]_n=1^N]_l=1^L ^L× N aligns multivariate observations on the unified timeline, where xlnx^n_l records the observed value of the n-th variable at timestamp tlt_l if available, and is filled with zero otherwise. To explicitly distinguish real observations from filled values, a binary mask ℳ=[[mln]n=1N]l=1L∈0,1L×NM=[[m^n_l]_n=1^N]_l=1^L∈\0,1\^L× N is introduced, where mln=1m^n_l=1 indicates that variable n is observed at time tlt_l, and mln=0m^n_l=0 otherwise. Together, the triplet (,,ℳ)(T,X,M) provides a fixed-shape representation on a shared temporal grid while preserving the original irregular sampling pattern through the observation mask. To better preserve the native irregular sampling characteristics of each variable, we refer to the representation strategy in ISTS-PLM (Zhang et al., 2025a) and adopt a variable-independent sequence representation for ISTS. Specifically, we do not merge timestamps across variables; instead, each variable maintains its own ordered timestamps, denoted as n=[tin]i=1Ln∈ℝLnT_n=[t^n_i]_i=1^L_n ^L_n for the n-th variable, where LnL_n is the number of observations of the n-th variable and tint^n_i represents the i-th observation timestamp of the n-th variable. 3.3. Irregularly-Sampled Time Series Forecasting Given an ISTS O, we define forecasting queries to specify which future values are to be predicted. For the n-th variable, a forecasting query is defined by a future timestamp qjnq^n_j satisfying qjn>maxitinq^n_j> _it^n_i, where j=1,…,Qnj=1,…,Q_n denotes the query timestamps for variable n. The set of forecasting queries is denoted as =qjnj=1Qnn=1NQ=\\q^n_j\_j=1^Q_n\_n=1^N. The goal of ISTS forecasting is to learn a model FθF_θ, parameterized by θ, that maps historical observations and forecasting queries to future value predictions. (1) Fθ(,)⟶^=x^jnj=1Qnn=1N,F_θ(O,Q)\; \; X= \\ x^n_j\_j=1^Q_n \_n=1^N, where x^jn x^n_j denotes the predicted value of the n-th variable at the j-th query timestamp qjnq^n_j. 4. Methodology We present M-ISTS, a multimodal framework designed to tackle the challenges of ISTS forecasting. The core idea of M-ISTS is to combine the precise numerical patterns learned from historical data with the general knowledge provided by pre-trained MLLMs. As illustrated in Figure 1, the framework comprises four components: (1) Cross-Modal Vision-Text Encoding module that transforms ISTS into irregularity-aware visual and textual representations; (2) Dual-stage ISTS Encoding branch that sequentially models intra-series temporal dynamics and inter-series variable correlations; (3) Adaptive Query-Based Feature Extractor that compresses high-dimensional MLLM tokens into compact, variable-aligned queries; and (4) a Multimodal Alignment module equipped with a Modality-Aware Gating mechanism for adaptive fusion. Figure 1. Overview of the proposed M-ISTS framework. 4.1. Cross-Modal Vision-Text Encoding To leverage the powerful reasoning capabilities of frozen MLLMs, we propose a cross-modal transformation mechanism that maps the sparse ISTS into the dense contextual spaces of vision and language, while carefully preserving critical irregularity patterns that are essential for forecasting. 4.1.1. Irregularity-Aware Image Construction Standard time series visualization methods, such as line plots or heatmaps, often discard fine-grained temporal dynamics and introduce noise. To address this issue, we introduce a tensor-based image construction scheme that encodes observation values, missingness patterns, and temporal intervals into separate image channels. Given an ISTS sample with N variables and maximum sequence length L, we construct a three-channel image ℐ∈ℝ3×N×LI ^3× N× L, where the height H=NH=N corresponds to variables and the width W=LW=L corresponds to the temporal axis. Since different variables in ISTS may be observed at different time points, we represent the timestamps, values, and masks independently for each variable. Each channel captures a distinct aspect: Observed Data channel records values: 0∈ℝN×LC_0 ^N× L, where each row corresponds to a variable and each column corresponds to a timestep. The pixel intensity reflects the time series value. Missingness Mask channel encodes the sparsity pattern: 1∈ℝN×LC_1 ^N× L. A pixel value of 1 indicates that the corresponding variable was observed at that time step, while 0 indicates a missing entry. Temporal Interval channel encodes the irregularity of sampling intervals. Since different variables may be observed at different time points, we compute the time gap independently for each variable. Specifically, for variable n at time step l, we compute δln=tln−tl−1nδ^n_l=t^n_l-t^n_l-1, with δ1n=0δ^n_1=0 for the first observation. We construct 2∈ℝN×LC_2 ^N× L where (2)n,l=δln(C_2)_n,l=δ^n_l. The final irregularity-aware image ℐI is obtained by stacking these three channels: (2) ℐ=(0,1,2)∈ℝ3×N×L.I=Stack(C_0,C_1,C_2) ^3× N× L. Before feeding into the MLLM, the image is resized to match the expected input resolution and normalized to the appropriate pixel value range. 4.1.2. Statistical-domin Text Prompting While the constructed image provides a visual representation of the ISTS, MLLMs also benefit from the textual context that activates their domain knowledge. We therefore construct a structured text prompt P that integrates statistical summaries of the data with explicit domain context and task descriptions. We first compute key summary statistics for each variable n∈1,…,Nn∈\1,…,N\. Specifically, for each variable, we calculate the mean value over observed entries as μn=∑l=1Lmlnxln∑l=1Lmln _n= _l=1^Lm^n_lx^n_l _l=1^Lm^n_l, the value range [xnmin,xnmax][x_n ,x_n ] where xnmin=minl:mln=1xlnx_n = _l:m^n_l=1x^n_l and xnmax=maxl:mln=1xlnx_n = _l:m^n_l=1x^n_l, and the missing rate ρn=1−1L∑l=1Lmln _n=1- 1L _l=1^Lm^n_l representing the proportion of missing observations. To mitigate noise from extremely sparse variables where statistics are unreliable, we apply a filtering mechanism: we format the statistics into a natural language description n=(μn,xnmin,xnmax)S_n=Format( _n,x_n ,x_n ) only if ρn≤τ _n≤τ, where τ is a sparsity threshold (e.g., τ=0.9τ=0.9); otherwise, n=∅S_n= . The final text prompt P is assembled by concatenating four types of instruction components: (3) =[,,,nn:ρn≤τ],P=[P_img,P_data,P_task,\S_n\_n: _n≤τ], where P_img provides instructions for interpreting the three-channel image, P_data supplies domain-specific context about the dataset, and P_task specifies the forecasting objective. 4.1.3. MLLM Feature Extraction The frozen MLLM encoder ℰE_MLLM jointly processes the irregularity-aware image ℐI and the structured text prompt P to produce a sequence of high-dimensional hidden states. The image is first processed by a vision encoder to obtain visual tokens, which are then combined with text tokens and fed through the language model backbone. We extract the hidden states from the deep layers of the MLLM as our multimodal representation, as these layers capture informative, rich features after cross-modal fusion: (4) =ℰ(ℐ,)∈ℝS×dm,E_MLLM=E_MLLM(I,P) ^S× d_m, where S is the number of tokens output by MLLM and dmd_m is the hidden dimension of the MLLM. The MLLM parameters remain frozen during training, which preserves the pre-trained knowledge. Following TimeCMA (Liu et al., 2025b), we pre-compute and store the MLLM token embeddings before training to improve computational efficiency. 4.2. ISTS Encoding While MLLMs excel at high-level contextual reasoning, they may not capture fine-grained numerical patterns as accurately as dedicated time series models. Our ISTS encoder addresses this limitation by operating on carefully designed embeddings followed by a Transformer encoder that models temporal dynamics within each variable and correlations across variables. 4.2.1. Multi-View Embedding Fusion In order to better capture the relationships within ISTS using Transformer architectures, we model them from different perspectives. Temporal Embedding. Unlike discrete positional encodings used in standard Transformers, ISTS requires embeddings that can handle continuous and irregularly spaced timestamps. We employ a learnable sinusoidal mapping ϕ:ℝ→ℝDφ:R ^D that captures both periodic patterns and linear temporal trends: (5) ϕ(t)d=ω0t+β0,d=0,sin(ωdt+βd),d>0,φ(t)_d= cases _0t+ _0,&d=0,\\ ( _dt+ _d),&d>0, cases where ωdd=0D−1\ _d\_d=0^D-1 and βdd=0D−1\ _d\_d=0^D-1 are learnable parameters. The first dimension captures linear time progression, while the remaining dimensions encode periodic patterns at different frequencies. Variable Embedding. To distinguish between different variables and enable the model to learn variable-specific patterns, we introduce a learnable embedding matrix ∈ℝN×DE_var ^N× D that assigns a unique representation n∈ℝDe_n^var ^D to each variable index n∈1,…,Nn∈\1,…,N\. These embeddings are learned during training and capture the intrinsic characteristics of each variable type. Value Embedding. For each observation, we need to encode both the numerical value and whether it was actually observed. We concatenate the observed value xlnx^n_l with its corresponding mask indicator mlnm^n_l and apply a linear projection: l,n=[xln,mln]+e_l,n^val=[x^n_l,m^n_l]W_val+b_val, where ∈ℝ2×DW_val ^2× D and ∈ℝDb_val ^D are learnable parameters. Embedding Fusion. The fused embedding for the l-th time step of variable n combines temporal and value information through a mask-gated mechanism: l,n=mln⋅ϕ(tln)+l,nz_l,n=m^n_l·φ(t^n_l)+e_l,n^val. The mask-gated design ensures that temporal information is weighted by observation presence. To enable the model to aggregate information at the variable level, we prepend the variable embedding ne_n^var as a learnable prompt token to each variable’s sequence, forming n=[n,1,n,…,L,n]∈ℝ(L+1)×DZ_n=[e_n^var,z_1,n,…,z_L,n] ^(L+1)× D. 4.2.2. Temporal-Variable Encoder Multivariate ISTS exhibit two types of dependencies: temporal dependencies within each variable and cross-variable dependencies. To disentangle and effectively model these two types of relationships, we employ a Temporal Encoder and a Variable Encoder using stacked Transformer layers. Temporal Encoder. We apply a multi-layer Transformer encoder ℱF_temp independently to each variable’s sequence to capture temporal patterns. The encoder consists of LtL_t stacked layers, where each layer applies multi-head self-attention followed by a feed-forward network. The attention mechanism allows each position to attend to all other positions within the same variable’s sequence: (6) (,,)=(⊤dk),Attention(Q,K,V)=Softmax ( QK d_k )V, where ,,∈ℝ(L+1)×dkQ,K,V ^(L+1)× d_k are the query, key, and value matrices obtained by linear projections, and dkd_k is the dimension per attention head. The multi-head attention extends this by computing h parallel attention heads and concatenating their outputs: (7) ()=(ℎ1,…,ℎh)O,MultiHead(Z)=Concat(head_1,…,head_h)W^O, where ℎi=(iQ,iK,iV)head_i=Attention(ZW^Q_i,ZW^K_i,ZW^V_i) and OW^O is the output projection matrix. The Temporal Encoder processes each variable’s sequence independently, producing n=ℱ(n)∈ℝ(L+1)×DH_n^temp=F_temp(Z_n) ^(L+1)× D. Since different variables may have different numbers of observed time points, we perform mask-aware aggregation to obtain a fixed-length representation n∈ℝDh_n ^D for each variable: (8) n=∑l=0Lm~ln⋅n,l∑l=0Lm~ln,h_n= _l=0^L m^n_l·H_n,l^temp _l=0^L m^n_l, where m~0n=1 m^n_0=1 for variable tokens and m~ln=mln m^n_l=m^n_l for l≥1l≥ 1. Variable Encoder. We then model the dependencies across different variables. The aggregated variable representations are stacked to form a matrix =[1,…,N]⊤∈ℝN×DH_agg=[h_1,…,h_N] ^N× D. We enhance these representations by adding the variable embeddings and apply a multi-layer Transformer encoder ℱF_var consisting of LvL_v layers to model cross-variable correlations. The Variable Encoder allows each variable’s representation to attend to and incorporate information from all other variables. The final output is =ℱ(+)∈ℝN×DH_ISTS=F_var(H_agg+E_var) ^N× D. 4.3. Adaptive Query-Based Feature Extractor While the MLLM output ∈ℝS×dmE_MLLM ^S× d_m encodes rich contextual information from both visual and textual modalities, it cannot be directly fused with the ISTS Encoding branch output for two key challenges: (1) the sequence length S of MLLM output varies depending on the input image and prompt, making it difficult to align with the fixed N variables; (2) the hidden dimension dmd_m typically differs from the ISTS encoder dimension D. To bridge this representational gap, we propose an Adaptive Query-Based Feature Extractor inspired by the Q-Former architecture (Li et al., 2023), which acts as a learnable information bottleneck that compresses and aligns the high-dimensional MLLM output into variable-aligned representations. We introduce a set of N learnable query tokens ∈ℝN×dmQ ^N× d_m, where each query is initialized to encode a latent variable-specific representation for one of the N variables. The queries interact with the MLLM output E_MLLM through K stacked layers, each consisting of self-attention among queries, cross-attention from queries to MLLM features, and a feed-forward network. For each layer, the computation proceeds as follows. Given input queries Q, we first apply layer normalization and self-attention to enable information exchange among the N variable queries: (9) ~=+(()), Q=Q+MultiHead_self(LN(Q)), where MultiHead_self denotes multi-head self-attention with queries, keys, and values all derived from the input. We then apply layer normalization to obtain ′=(~)Q =LN( Q), which serves as the query input to cross-attention. The cross-attention mechanism allows each variable query to selectively attend to and extract relevant information from the MLLM’s visual and textual representations. Specifically, we compute projected queries p=′QQ_p=Q W^Q, keys p=KK_p=E_MLLMW^K, and values p=VV_p=E_MLLMW^V, where Q,K,VW^Q,W^K,W^V are learnable projection matrices. The cross-attention is then computed as: (10) (′,,)=(pp⊤dk)p.CrossAttn(Q ,E_MLLM,E_MLLM)=Softmax ( Q_pK_p d_k )V_p. The cross-attended features are added back via a residual connection: (11) ^=~+(′,,). Q= Q+MultiHead_cross(Q ,E_MLLM,E_MLLM). Finally, a feed-forward network with layer normalization and residual connection produces ′=^+((^))Q = Q+FFN(LN( Q)). The output ′Q then serves as the input Q for further alignment. Through this iterative K-layer process, the query tokens progressively extract and compress relevant contextual information from the high-dimensional visual-textual context. After K layers of refinement, the final query states constitute our multimodal representation =′∈ℝN×dmH_M=Q ^N× d_m, which is explicitly aligned with the N variables in terms of the first dimension. 4.4. Multimodal Alignment To effectively integrate precise numerical patterns with contextual knowledge, we propose an alignment module that adaptively fuses ∈ℝN×DH_ISTS ^N× D and ∈ℝN×dmH_M ^N× d_m based on variable-specific data quality. Cross-Attention Fusion. Direct concatenation or addition of the two representations would not account for the contextual relationships between numerical patterns and MLLM-derived features. Instead, we use cross-attention to enable the numerical features to selectively query and incorporate relevant contextual information: (12) =(,,)∈ℝN×D,H_fused=CrossAttn(H_ISTS,H_M,H_M) ^N× D, where H_ISTS serves as query and H_M servers as keys and values. Modality-Aware Gating. Different variables in an ISTS may have different observation densities. For a densely observed variable, the numerical features from the ISTS encoder are reliable and should be paid more attention. Conversely, for a sparsely observed variable with high missing rates, the contextual information from MLLMs may provide more valuable information. To address such variable-specific differences in observation quality, we introduce a Modality-Aware Gating mechanism that adaptively balances the contributions of the two modalities for each variable. For each variable n, we compute a statistics vector n∈ℝ4s_n ^4 that summarizes its data characteristics: (13) n=[μn,σn,ρn,cn],s_n=[ _n, _n, _n,c_n], where μn _n is the mean of observed values, σn _n is the standard deviation, ρn _n is the missing rate, and cn=∑l=1Lmln/Lc_n= _l=1^Lm^n_l/L is the normalized observation count. A gating network G, implemented as a two-layer MLP with ReLU activation, maps this statistics vector to fusion weights: (14) [αn,αn]=((n))∈ℝ2,[ _n^num, _n^m]=Softmax(G(s_n)) ^2, where αn+αn=1 _n^num+ _n^m=1. The final fused representation for each variable is computed as a statistics-conditioned weighted combination: (15) [n]=αn⋅[n]+αn⋅[n].H_final[n]= _n^num·H_ISTS[n]+ _n^m·H_fused[n]. 4.5. ISTS Predictor Given the fused representation H_final and forecasting queries =qjnj=1Qnn=1NQ=\\q^n_j\_j=1^Q_n\_n=1^N, we generate predictions by conditioning variable features on target timestamps. For query qjnq^n_j, the prediction is generated via an MLP: (16) x^jn=([[n]∥qjn]). x^n_j=MLP([H_final[n]\,\|\,q^n_j]). The model is optimized end-to-end via MSE loss over all valid queries, with the MLLM backbone frozen: (17) ℒ=1∑n=1NQn∑n=1N∑j=1Qn(x^jn−xjn)2.L= 1 _n=1^NQ_n _n=1^N _j=1^Q_n( x^n_j-x^n_j)^2. Table 1. Overall performance comparison on four datasets. The best results are highlighted in bold, and the second-best are underlined. Dataset PhysioNet MIMIC Human Activity USHCN Metric MSE×10−3× 10^-3 MAE×10−2× 10^-2 MSE×10−2× 10^-2 MAE×10−2× 10^-2 MSE×10−3× 10^-3 MAE×10−2× 10^-2 MSE×10−1× 10^-1 MAE×10−1× 10^-1 DLinear 41.86 ± 0.05 15.52 ± 0.03 4.90 ± 0.00 16.29 ± 0.05 4.03 ± 0.01 4.21 ± 0.01 6.21 ± 0.00 3.88 ± 0.02 TimesNet 16.48 ± 0.11 6.14 ± 0.03 5.88 ± 0.08 13.62 ± 0.07 3.12 ± 0.01 3.56 ± 0.02 5.58 ± 0.05 3.60 ± 0.04 PatchTST 12.00 ± 0.23 6.02 ± 0.14 3.78 ± 0.03 12.43 ± 0.10 4.29 ± 0.14 4.80 ± 0.09 5.75 ± 0.01 3.57 ± 0.02 Crossformer 6.66 ± 0.11 4.81 ± 0.11 2.65 ± 0.10 9.56 ± 0.29 4.29 ± 0.20 4.89 ± 0.17 5.25 ± 0.04 3.27 ± 0.09 Graph Wavenet 6.04 ± 0.28 4.41 ± 0.11 2.93 ± 0.09 10.50 ± 0.15 2.89 ± 0.03 3.40 ± 0.05 5.29 ± 0.04 3.16 ± 0.09 MTGNN 6.26 ± 0.18 4.46 ± 0.07 2.71 ± 0.23 9.55 ± 0.65 3.03 ± 0.03 3.53 ± 0.03 5.39 ± 0.05 3.34 ± 0.02 StemGNN 6.86 ± 0.28 4.76 ± 0.19 1.73 ± 0.02 7.71 ± 0.11 8.81 ± 0.37 6.90 ± 0.02 5.75 ± 0.09 3.40 ± 0.09 CrossGNN 7.22 ± 0.36 4.96 ± 0.12 2.95 ± 0.16 10.82 ± 0.21 3.03 ± 0.10 3.48 ± 0.08 5.66 ± 0.04 3.53 ± 0.05 FourierGNN 6.84 ± 0.35 4.65 ± 0.12 2.55 ± 0.03 10.22 ± 0.08 2.99 ± 0.02 3.42 ± 0.02 5.82 ± 0.06 3.62 ± 0.07 GRU-D 5.59 ± 0.09 4.08 ± 0.05 1.76 ± 0.03 7.53 ± 0.09 2.94 ± 0.05 3.53 ± 0.06 5.54 ± 0.38 3.40 ± 0.28 SeFT 9.22 ± 0.18 5.40 ± 0.08 1.87 ± 0.01 7.84 ± 0.08 12.20 ± 0.17 8.43 ± 0.07 5.80 ± 0.19 3.70 ± 0.11 RainDrop 9.82 ± 0.08 5.57 ± 0.06 1.99 ± 0.03 8.27 ± 0.07 14.92 ± 0.14 9.45 ± 0.05 5.78 ± 0.22 3.67 ± 0.17 Warpformer 5.94 ± 0.35 4.21 ± 0.12 1.73 ± 0.04 7.58 ± 0.13 2.79 ± 0.04 3.39 ± 0.03 5.25 ± 0.05 3.23 ± 0.05 mTAND 6.23 ± 0.24 4.51 ± 0.17 1.85 ± 0.06 7.73 ± 0.13 3.22 ± 0.07 3.81 ± 0.07 5.33 ± 0.05 3.26 ± 0.10 Latent-ODE 6.05 ± 0.57 4.23 ± 0.26 1.89 ± 0.19 8.11 ± 0.52 3.34 ± 0.11 3.94 ± 0.12 5.62 ± 0.03 3.60 ± 0.12 CRU 8.56 ± 0.26 5.16 ± 0.09 1.97 ± 0.02 7.93 ± 0.19 6.97 ± 0.78 6.30 ± 0.47 6.09 ± 0.17 3.54 ± 0.18 Neural Flow 7.20 ± 0.07 4.67 ± 0.04 1.87 ± 0.05 8.03 ± 0.19 4.05 ± 0.13 4.46 ± 0.09 5.35 ± 0.05 3.25 ± 0.05 T-PatchGNN 5.11 ± 0.11 3.73 ± 0.11 1.66 ± 0.02 7.21 ± 0.14 2.79 ± 0.11 3.24 ± 0.07 5.03 ± 0.04 3.14 ± 0.09 KAFNet 5.47 ± 0.07 3.82 ± 0.12 1.70 ± 0.02 7.23 ± 0.04 2.70 ± 0.05 3.16 ± 0.03 5.19 ± 0.14 3.17 ± 0.10 ISTS-PLM 5.17 ± 0.13 3.67 ± 0.06 1.72 ± 0.05 7.18 ± 0.34 2.61 ± 0.07 3.19 ± 0.07 5.28 ± 0.05 3.03 ± 0.06 M-ISTS (Ours) 4.98 ± 0.11 3.54 ± 0.05 1.63 ± 0.02 6.85 ± 0.07 2.47 ± 0.01 3.06 ± 0.02 5.10 ± 0.03 2.94 ± 0.05 (a) Efficiency analysis. (b) Attention map. (c) Gating weights vs. sparsity. Figure 2. (a) Efficiency comparison, (b) cross-attention patterns, and (c) adaptive gating behavior. 5. Experiments 5.1. Experimental Setup 5.1.1. Datasets To comprehensively evaluate the effectiveness of the proposed method on ISTS forecasting, we conduct experiments on four widely used benchmark datasets: PhysioNet (Silva et al., 2012), MIMIC (Johnson et al., 2016), Human Activity (Vidulin et al., 2010), and USHCN (Menne et al., 2015). These datasets span diverse application domains, including healthcare, biomechanics, and climate science. We randomly split each dataset into training, validation, and test sets with a ratio of 6:2:26:2:2. 5.1.2. Baselines To comprehensively evaluate the effectiveness of the proposed method, we compare it against a diverse set of representative baseline models for the time series forecasting task. These baselines can be broadly categorized into three groups: (1) Regular Time Series Forecasting Models: DLinear (Zeng et al., 2023), TimesNet (Wu et al., 2023), PatchTST (Nie et al., 2023), Crossformer (Zhang and Yan, 2023), Graph WaveNet (Wu et al., 2019), MTGNN (Wu et al., 2020), StemGNN (Cao et al., 2020), CrossGNN (Huang et al., 2023), and FourierGNN (Yi et al., 2023). (2) ISTS Classification and Imputation Models: GRU-D (Che et al., 2018), SeFT (Horn et al., 2020), RainDrop (Zhang et al., 2022), Warpformer (Zhang et al., 2023), and mTAND (Shukla and Marlin, 2021). (3) ISTS Forecasting Models: Latent ODEs (Rubanova et al., 2019), CRU (Schirmer et al., 2022), Neural Flows (Bilos et al., 2021), T-PatchGNN (Zhang et al., 2024), KAFNet (Zhou et al., 2026), and ISTS-PLM (Zhang et al., 2025a). 5.2. Experimental Results 5.2.1. Main Results Table 1 presents the forecasting performance of M-ISTS and all baseline methods across four ISTS datasets. Key findings are summarized as follows. Different types of baselines show distinct performance gaps. Methods designed for regular time series perform the worst on most datasets, as they cannot naturally model irregular sampling patterns. ISTS classification and imputation models perform better by directly handling irregular data, but their lack of optimization for forecasting tasks leads to inferior results compared to specialized ISTS forecasting methods. Among the specialized ISTS forecasting models, T-PatchGNN, KAFNet, and ISTS-PLM represent the strongest competitors, for they have components specifically designed for irregularities. Our proposed M-ISTS achieves the best performance on most metrics and ranks second on the remaining ones, demonstrating consistent superiority across all datasets. On average, it outperforms all ISTS forecasting baselines by 14.3% in MSE and 15.1% in MAE. This strong and stable performance can be largely attributed to its effective method design for ISTS: a temporal-variable encoder captures temporal and cross-variable dependencies from numerical observations, while pre-trained MLLMs provide complementary multimodal information, and the multimodal alignment module effectively combines the strengths of both information sources. Compared with the LLM-based baseline ISTS-PLM, M-ISTS yields notable improvements. Specifically, it achieves 5.2% lower MSE and 4.6% lower MAE on the MIMIC dataset, and 5.4% lower MSE with 4.1% lower MAE on the Human Activity dataset, which verifies the advantages of MLLMs over single-modal LLM-based approaches in ISTS forecasting. 5.2.2. Ablation Study To thoroughly investigate the contribution of major components in M-ISTS, we conduct ablation studies on PhysioNet and Human Activity datasets. We compare the full model with four variants: w/o Text, which removes the statistical-semantic textual modality; w/o Image, which excludes the irregularity-aware image representation; w/o QBE, which replaces the proposed Query-Based Extractor (QBE) with simple average pooling over MLLM features; and w/o Align, which substitutes the multimodal alignment module with a naive element-wise addition. As illustrated in Figure 3, removing any component consistently leads to noticeable performance degradation on both datasets. Among all variants, w/o QBE suffers the most significant performance drop, by explicitly querying MLLM representations with variable-specific embeddings, QBE effectively compresses high-dimensional multimodal features while preserving variable-level correlations. w/o Text also shows a clear degradation, especially on the PhysioNet dataset. This suggests that the statistical-semantic textual descriptions provide valuable domain-level priors that help the MLLM better capture temporal dynamics. Similarly, the performance drop observed in w/o Image demonstrates the importance of the proposed irregularity-aware image construction. Finally, w/o Align consistently underperforms the full model. The proposed multimodal alignment enables the model to adaptively balance numerical and multimodal features. In contrast, simple element-wise addition fails to model cross-modal interactions, leading to poor performance. (a) PhysioNet (b) Human Activity Figure 3. Ablation study. (a) Learning rate (b) Batch size (c) QBE layers (d) TE/VE layers Figure 4. Hyperparameter sensitivity analysis on PhysioNet. 5.2.3. Parameter Sensitivity We investigate the sensitivity of M-ISTS to key hyperparameters on the PhysioNet dataset in Figure 4. We evaluate learning rates from 10−5,5×10−5,10−4,5×10−4\10^-5,5× 10^-5,10^-4,5× 10^-4\. As shown in Figure 4 (a), a learning rate of 10−510^-5 yields the best performance, while larger learning rates lead to performance degradation due to unstable optimization. We test batch sizes from 8,16,24,32\8,16,24,32\. Figure 4 (b) shows that smaller batch sizes achieve better performance, with batch size 8 being optimal. This benefit likely stems from the implicit regularization effect in stochastic gradient descent. We vary the number of Query-Based Extractor layers in 2,3,4\2,3,4\. As shown in Figure 4 (c), 3 layers achieve the best results, indicating that moderate depth provides sufficient capacity for feature extraction without overfitting. We also evaluate the depth of Temporal Encoder and Variable Encoder in 2,3,4\2,3,4\. Figure 4 (d) demonstrates that 3 layers yield optimal performance, balancing model expressiveness and generalization. 5.2.4. Efficiency Analysis In Figure 2 (a), we compared the computational efficiency of M-ISTS with two small models, such as KAFNet and T-PatchGNN, and an LLM-based model, ISTS-PLM. Since both M-ISTS and ISTS-PLM rely on large language models, we focused on comparing their efficiencies and MSE. ISTS-PLM fine-tunes the parameters of the language model during training, while M-ISTS fully freezes the backbone structure of the large language model and only trains the lightweight downstream modules. Therefore, the time required by M-ISTS per cycle is approximately half of that of ISTS-PLM, and it has significantly fewer trainable parameters. Although traditional methods like KAFNet and T-PatchGNN are faster due to their smaller model size, M-ISTS achieves significantly better prediction performance by leveraging the rich semantic knowledge in the frozen large language model, demonstrating a good balance between efficiency and accuracy. 5.2.5. Case Study Figure 2 (b) visualizes the attention map in the Multimodal Alignment module, where ISTS features selectively attend to informative multimodal tokens. The sparse attention pattern indicates that the model filters noise and focuses on the most relevant information from the MLLM output. Figure 2 (c) displays the relationship between modality-aware gating weights and variable sparsity. We observe a clear trend: variables with high missing rates are assigned higher weights for the multimodal branch, while densely observed variables rely more on the numerical branch. This empirically validates our design motivation that the outputs of MLLM serve as effective complementary information when numerical data is scarce. 5.2.6. Hidden layer of MLLM Table 2. Effect of Different Qwen2-VL Hidden Layers. Layer Position MSE×10−3× 10^-3 MAE×10−2× 10^-2 Last 2.49±0.032.49± 0.03 3.08±0.033.08± 0.03 3rd-to-last 2.47±0.012.47 0.01 3.06±0.023.06 0.02 5th-to-last 2.48±0.022.48± 0.02 3.06±0.023.06± 0.02 7th-to-last 2.49±0.022.49± 0.02 3.07±0.033.07± 0.03 We investigate the impact of extracting hidden states from different layers of the Qwen2-VL-2B-Instruct (Wang et al., 2024). As shown in Table 2, we evaluate four layer positions counting backward from the final layer on Human Activity (Vidulin et al., 2010). The results reveal that the 3rd-to-last layer achieves the best performance with the lowest MSE and MAE, as well as the smallest variance. Interestingly, extracting features from the last layer does not yield optimal results. This phenomenon can be attributed to the fact that the last layer of MLLMs is typically optimized for next-token prediction in language generation. In contrast, the intermediate layers retain richer and more generalizable multimodal semantic representations. Furthermore, we observe that extracting from earlier layers (e.g., 7th-to-last) leads to performance degradation, likely because cross-modal fusion in transformer-based MLLMs becomes more complete in deeper layers. Notably, despite these variations, all layer configurations yield comparable results with only minor fluctuations, indicating that M-ISTS is robust to the choice of hidden layer and that the MLLM representations are also beneficial across different depths. 6. Conclusion We present M-ISTS, a multimodal forecasting framework for ISTS. To overcome the representational and modality gaps inherent in ISTS, we introduce a cross-modal encoding strategy that transforms sparse ISTS into irregularity-aware visual and textual representations. By combining ISTS representations with a temporal–variable encoder and pretrained MLLMs, M-ISTS effectively captures both numerical dynamics and complementary semantic information. Furthermore, an adaptive query-based feature extractor is adopted to efficiently compress high-dimensional MLLM tokens into compact, variable-aligned queries. Finally, Multimodal Alignment further aligns multimodal features with variable-level temporal representations with cross-attention and a modality-aware gating mechanism. Experiments on multiple real-world benchmarks demonstrate consistent performance improvements over state-of-the-art ISTS forecasting methods and recent LLM-based baselines, highlighting the potential of multimodal learning for ISTS forecasting. References M. Bilos, J. Sommer, S. S. Rangapuram, T. Januschowski, and S. Günnemann (2021) Neural flows: efficient alternative to neural odes. In NeurIPS, p. 21325–21337. Cited by: §3.2, §5.1.2. E. D. Brouwer, J. Simm, A. Arany, and Y. Moreau (2019) GRU-ODE-Bayes: continuous modeling of sporadically-observed time series. In NeurIPS, p. 7377–7388. Cited by: §1, §3.2. D. Cao, Y. Wang, J. Duan, C. Zhang, X. Zhu, C. Huang, Y. Tong, B. Xu, J. Bai, J. Tong, and Q. Zhang (2020) Spectral temporal graph neural network for multivariate time-series forecasting. In NeurIPS, Cited by: §5.1.2. C. Chang, J. Hwang, Y. Shi, H. Wang, W. Peng, T. Chen, and W. Wang (2025) Time-imm: A dataset and benchmark for irregular multimodal multivariate time series. In NeurIPS, Cited by: §2.2. Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu (2018) Recurrent neural networks for multivariate time series with missing values. Scientific reports 8 (1), p. 6085. Cited by: §3.2, §5.1.2. [6] M. Chen, L. Shen, Z. Li, X. J. Wang, J. Sun, and C. Liu VisionTS: visual masked autoencoders are free-lunch zero-shot time series forecasters. In ICML, Cited by: §1. J. Fan (2022) Dynamic nonlinear matrix completion for time-varying data imputation. In AAAI, p. 6587–6596. Cited by: §1. A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. Peng, and H. E. Stanley (2000) PhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation 101 (23), p. e215–e220. Cited by: §1. M. Horn, M. Moor, C. Bock, B. Rieck, and K. M. Borgwardt (2020) Set functions for time series. In ICML, p. 4353–4363. Cited by: §5.1.2. Q. Huang, L. Shen, R. Zhang, S. Ding, B. Wang, Z. Zhou, and Y. Wang (2023) CrossGNN: confronting noisy multivariate time series via cross interaction refinement. In NeurIPS, Cited by: §5.1.2. M. Jin, S. Wang, L. Ma, Z. Chu, J. Y. Zhang, X. Shi, P. Chen, Y. Liang, Y. Li, S. Pan, and Q. Wen (2024) Time-llm: time series forecasting by reprogramming large language models. In ICLR, Cited by: §1, §2.1. A. E. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark (2016) MIMIC-i, a freely accessible critical care database. Scientific data 3 (1), p. 1–9. Cited by: §5.1.1. B. Li, Y. Luo, Z. Liu, J. Zheng, J. Lv, and Q. Ma (2025) HyperIMTS: hypergraph neural network for irregular multivariate time series forecasting. In ICML, Cited by: §1, §1, §2.2. J. Li, D. Li, S. Savarese, and S. C. H. Hoi (2023) BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, Vol. 202, p. 19730–19742. Cited by: §1, §4.3. C. Liu, H. Miao, Q. Xu, S. Zhou, C. Long, Y. Zhao, Z. Li, and R. Zhao (2025a) Efficient multivariate time series forecasting via calibrated language models with privileged knowledge distillation. In ICDE, p. 3165–3178. Cited by: §1, §1, §2.1. C. Liu, Q. Xu, H. Miao, S. Yang, L. Zhang, C. Long, Z. Li, and R. Zhao (2025b) TimeCMA: towards llm-empowered multivariate time series forecasting via cross-modality alignment. In AAAI, p. 18780–18788. Cited by: §1, §2.1, §4.1.3. C. Liu, S. Zhou, Q. Xu, H. Miao, C. Long, Z. Li, and R. Zhao (2025c) Towards cross-modality modeling for time series analytics: A survey in the LLM era. In IJCAI, p. 10564–10572. Cited by: §2.1. Y. Luo, B. Zhang, Z. Liu, and Q. Ma (2025) Hi-patch: hierarchical patch GNN for irregular multivariate time series. In ICML, Cited by: §1, §1, §2.2. M. Menne, C. Williams Jr, R. Vose, and D. Files (2015) Long-term daily climate records from stations across the contiguous united states. Cited by: §5.1.1. G. Mercatali, A. Freitas, and J. Chen (2024) Graph neural flows for unveiling systemic interactions among irregularly sampled time series. In NeurIPS, p. 57183–57206. Cited by: §1, §2.2. Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam (2023) A time series is worth 64 words: long-term forecasting with transformers. In ICLR, Cited by: §1, §5.1.2. Y. Oh, D. Lim, and S. Kim (2024) Stable neural stochastic differential equations in analyzing irregular time series data. In ICLR, Cited by: §1. M. A. Reyna, C. Josef, S. Seyedi, R. Jeter, S. P. Shashikumar, M. B. Westover, A. Sharma, S. Nemati, and G. D. Clifford (2019) Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. In CinC, p. 1–4. Cited by: §1. Y. Rubanova, T. Q. Chen, and D. Duvenaud (2019) Latent ordinary differential equations for irregularly-sampled time series. In NeurIPS, p. 5321–5331. Cited by: §1, §2.2, §3.2, §5.1.2. J. D. Scargle (1982) Studies in astronomical time series analysis. i-statistical aspects of spectral analysis of unevenly spaced data. Astrophys. J. 263, p. 835–853. Cited by: §1. M. Schirmer, M. Eltayeb, S. Lessmann, and M. Rudolph (2022) Modeling irregular time series with continuous recurrent units. In ICML, p. 19388–19405. Cited by: §1, §2.2, §3.2, §5.1.2. S. N. Shukla and B. M. Marlin (2021) Multi-time attention networks for irregularly sampled time series. In ICLR, Cited by: §3.2, §5.1.2. I. Silva, G. Moody, D. J. Scott, L. A. Celi, and R. G. Mark (2012) Predicting in-hospital mortality of icu patients: the physionet/computing in cardiology challenge 2012. In Cinc, p. 245–248. Cited by: §5.1.1. X. Tang, H. Yao, Y. Sun, C. C. Aggarwal, P. Mitra, and S. Wang (2020) Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In AAAI, p. 5956–5963. Cited by: §1. Y. Tashiro, J. Song, Y. Song, and S. Ermon (2021) CSDI: conditional score-based diffusion models for probabilistic time series imputation. In NeurIPS, p. 24804–24816. Cited by: §2.2. V. Vidulin, M. Lustrek, B. Kaluza, R. Piltaver, and J. Krivec (2010) Localization data for person activity. UCI Machine Learning Repository 10, p. C57G8X. Cited by: §5.1.1, §5.2.6. R. Vio, M. Díaz-Trigo, and P. Andreani (2013) Irregular time series in astronomy and the use of the lomb-scargle periodogram. Astron. Comput. 1, p. 5–16. Cited by: §1. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. CoRR abs/2409.12191. Cited by: §5.2.6. H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long (2023) TimesNet: temporal 2d-variation modeling for general time series analysis. In ICLR, Cited by: §1, §5.1.2. Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang (2020) Connecting the dots: multivariate time series forecasting with graph neural networks. In SIGKDD, p. 753–763. Cited by: §5.1.2. Z. Wu, S. Pan, G. Long, J. Jiang, and C. Zhang (2019) Graph wavenet for deep spatial-temporal graph modeling. In IJCAI, p. 1907–1913. Cited by: §5.1.2. V. K. Yalavarthi, K. Madhusudhanan, R. Scholz, N. Ahmed, J. Burchert, S. Jawed, S. Born, and L. Schmidt-Thieme (2024) GraFITi: graphs for forecasting irregularly sampled time series. In AAAI, p. 16255–16263. Cited by: §1, §2.2. K. Yi, Q. Zhang, W. Fan, H. He, L. Hu, P. Wang, N. An, L. Cao, and Z. Niu (2023) FourierGNN: rethinking multivariate time series forecasting from a pure graph perspective. NeurIPS 36, p. 69638–69660. Cited by: §5.1.2. A. Zeng, M. Chen, L. Zhang, and Q. Xu (2023) Are transformers effective for time series forecasting?. In AAAI, p. 11121–11128. Cited by: §5.1.2. J. Zhang, S. Zheng, W. Cao, J. Bian, and J. Li (2023) Warpformer: A multi-scale modeling approach for irregular clinical time series. In SIGKDD, p. 3273–3285. Cited by: §3.2, §5.1.2. W. Zhang, C. Yin, H. Liu, and H. Xiong (2025a) Unleashing the power of pre-trained language models for irregularly sampled time series. In SIGKDD, p. 3831–3842. Cited by: §1, §1, §2.2, §3.2, §5.1.2. W. Zhang, C. Yin, H. Liu, X. Zhou, and H. Xiong (2024) Irregular multivariate time series forecasting: A transformable patching graph neural networks approach. In ICML, Cited by: §1, §1, §2.2, §5.1.2. X. Zhang, M. Zeman, T. Tsiligkaridis, and M. Zitnik (2022) Graph-guided network for irregularly sampled multivariate time series. In ICLR, Cited by: §3.2, §5.1.2. Y. Zhang, X. Wang, X. Yu, Z. Zhou, X. Xu, L. Bai, and Y. Wang (2025b) DIFFODE: neural ODE with differentiable hidden state for irregular time series analysis. In ICDE, p. 1–14. Cited by: §1. Y. Zhang and J. Yan (2023) Crossformer: transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR, Cited by: §5.1.2. S. Zhong, W. Ruan, M. Jin, H. Li, Q. Wen, and Y. Liang (2025) Time-vlm: exploring multimodal vision-language models for augmented time series forecasting. In ICML, Cited by: §1, §1, §2.1. Z. Zhou, Y. Huang, Y. Wang, Y. Wu, J. T. Kwok, and Y. Liang (2026) Revitalizing canonical pre-alignment for irregular multivariate time series forecasting. In AAAI, Cited by: §5.1.2.