Paper deep dive
Sensor-Language-Action Models
Yuekai Xu, Zitao Shuai, Yuzhe Yang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.
Tags
Links
- Source: https://arxiv.org/abs/2610.08244v1
- Canonical: https://arxiv.org/abs/2610.08244v1
Trouble viewing inline? Open PDF directly →
Full Text
115,155 characters extracted from source content.
Expand or collapse full text
2026-10-7 Sensor-Language-Action Models Yuekai Xu 1* , Zitao Shuai 1* , Yuzhe Yang 1† 1 University of California, Los Angeles Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor- Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we presentOpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art.OpenSLAalso demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts. Model: https://hf.co/yang-ai-lab/OpenSLA Code: https://github.com/yang-ai-lab/OpenSLA Website: https://yang-ai-lab.github.io/OpenSLA 1. Introduction Intelligent systems do more than perceive: they observe, understand, and act. The value of sensing, likewise, lies not only in describing what is happening, but in informing the actions that follow [39,13]. For example, clinicians in operation rooms continuously integrate physiological signals with individual context, history, and prior interventions to decide whether to administer a medication, escalation, or continue observation [10]. Yet, existing sensor models are centered on observing and understanding. Early methods map sensor signals to fixed labels or outcomes [24,31], while recent sensor-language models enable richer semantic descriptions and open-ended reasoning over sensor observations [43,15,40]. However, these advances largely stop before action: downstream decisions are still modeled as separate prediction problems, each with its own formulation and action space [17,35]. As a result, existing approaches provide no common mechanism for relating heterogeneous sensor evidence, contextual information, and the diverse actions they may support. Natural language, on the other hand, offers a natural bridge between sensing and action. Unlike fixed labels, language can jointly express task intent, individual context, physiological state, and action semantics, allowing heterogeneous sensor-action problems to be represented in a common form. This suggests a broader modeling paradigm in which sensor observations are grounded in language and connected directly to actions, rather than treating understanding and decision-making as separate stages. We therefore ask: Can sensor data, language, and action be modeled through a unified framework? * Equal contribution (co-first authors). † Correspondence to: yuzhey@ucla.edu. arXiv:2610.08244v1 [cs.AI] 6 Oct 2026 Sensor-Language-Action Models Rapid breathing provides context for the recorded nebulizer. Why this action? What action follows? Clinical Operating Room Daily Life Electrocardiogram Respirations Heart Rate Continuous Glucose Monitor . . . . Individual context 67-years old male, complaint: fever, prior history: ... Multimodal Sensor Signals Action space Medication Procedures Diagnostics AntibioticsInsulin bolus.. Invasive access .. Arterial blood gas test .. Clinical CGM OR Necessity Category Category Carb amount Necessity Category OpenSLA OpenTSLM SensorLM GPT 5.6 Luna Deepseek V3.2 Observational Relational Resp waveform: fast breathing; rate proxy 39.1/min, IQR 1.00 ... Breathing rateis improving with R above expected range ... Global & Contextual Context includes presentation mentions fever or infection ... Action Evidences Multi-faceted Captions Multi-level Action Necessity: Yes Category: nebulizer_treatment Label: NEBULIZER What is happening? ECG spikes at... A B C Patient’s notes Patient’s notes Evidences ETCO2 Temp ECG rate RESP rate Necessity Zero-shot Multi-level action pred. Sensor-Language-Action Model OpenSLA Figure 1|Sensor-Language-Action (SLA) Model. We establish SLA as a unified paradigm for sensor-action modeling through a language interface over sensor observations, patient context, and actions. We instantiate SLA with OpenSLA and evaluate it across more than 116,000 individuals, 79 sensor modalities, and 60 action groups. In addition to the superior performance over state-of-the-art methods,OpenSLAsupports (A) multi-level action predictions, (B) sensor-state understanding, and (C) action-evidence grounding. To fill the gap, we introduce Sensor-Language-Action (SLA) modeling, a unified framework that connects sensor observations, natural-language context, and actions through a common multimodal interface (see Fig. 1). We further instantiate SLA withOpenSLA, to our knowledge, the first general framework for jointly modeling sensor observations, language, and actions.OpenSLAextends sensor intelligence from understanding observations to reasoning about and explaining the actions they support, enabled by three key contributions:❶A hierarchical, multi-faceted physiological captioning pipeline that aligns local sensor dynamics, cross-signal relations, individual context, and action- relevant evidence as language supervision;❷The curation of a large-scale benchmark spanning more than 116,000 individuals, 79 sensor modalities, 3 healthcare domains, 5 tasks, and 60 action groups; and ❸ a unified SLA architecture for hierarchical action prediction, state understanding, and action explanation that supports robust and scalable sensor-action modeling via language grounding. To rigorously evaluateOpenSLA, we benchmark it against state-of-the-art methods across six real- world cohorts and five core tasks spanning clinical care, operating rooms, and daily-life metabolic health. Extensive experiments demonstrate consistently strong performance in hierarchical action prediction, state understanding, and action explanation. Further analyses reveal intriguing properties on action-evidence agreement, transfer to unseen action labels and external clinical cohorts, and future-state information in the learned representations. Our contributions are as follows: • We formulate Sensor-Language-Action (SLA) modeling, a new framework that unifies sensor-action problems through multimodal language interface, and construct a large-scale benchmark spanning more than 116,000 individuals, 79 sensor modalities, 5 tasks, and 60 action groups. •We introduce a multi-faceted sensor captioning pipeline that jointly captures local sensor dynamics, cross-signal relations, individual-level context, and action-relevant evidence, providing language supervision tailored to sensor-action modeling. • We developOpenSLA, a unified SLA model that jointly supports hierarchical action prediction, state understanding, and action explanation, informed by systematic studies of how sensor signals, language, and actions should be modeled together. 2 Sensor-Language-Action Models Table 1 | Comparisons with representative sensor-language studies. Study DomainData ScaleCaption Content Clinical OR Daily # Dataset # Modality # Individuals # Action Individual Context Action Evidence Khasentino et al. [13] ✗ ✓3204.2k–✓✗ Langer et al. [15]✓✗ ✓53N/R–✓✗ Zhang et al. [43]✗ ✓45103k–✗ Xu et al. [40]✓✗ ✗51210k–✗ OpenSLA (Ours)✓779116K60✓ •We conduct extensive evaluations across diverse real-world tasks and verify the superior performance ofOpenSLAagainst SOTA methods. Further analyses confirm its capabilities on language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts. 2. Related Work Sensor-Action Modeling. Predicting subsequent actions or decisions from observed sensor signals has been a long-standing problem [17,35]. Existing approaches typically formulate this as supervised prediction, either learning direct mappings from sensor observations to action labels or incorporating additional textual context as conditioning information [32,8,23]. However, these approaches are less effective and flexible in incorporating heterogeneous individual context and other decision-relevant information. In contrast, we address this limitation by introducing language as a flexible interface for sensor-action modeling. Sensor and Time-Series Foundation Models. Time-series foundation models [2,37,6,41] have shown strong performance across diverse sensor-centered tasks, including forecasting, classification, and representation learning. Beyond general-purpose time-series models, sensor-focused foundation models further specialize in physiological and wearable signals, covering ECG [18] and PPG [27], continuous glucose monitoring [22,20], motion and wearable sensing [39], and sleep [31]. Despite this progress, existing sensor foundation models largely focus on understanding states of input sensor signals, rather than translating sensor dynamics into action-relevant semantics. In contrast,OpenSLA addresses this gap by jointly modeling sensor observations, language context, and actions through a shared sensor-language-action interface. Sensor-Language Modeling. Language models have increasingly been applied to sensor analysis through direct prompting or dedicated sensor-language models [30,19,15,40]. SensorLM [43] and SleepLM [40] leverage paired sensor-text data and multi-level captioning to align sensor representa- tions with language, demonstrating strong transferability across downstream tasks. OpenTSLM [15], in contrast, combines time-series encoders with pretrained LLMs to support zero-shot language-based interaction with medical signals. Such models often rely on captioning pipelines for paired sensor-text supervision, but both their data and modeling pipelines are primarily designed for sensor-text align- ment rather than action-relevant semantics. In this project, we introduce a multi-faceted captioning pipeline to capture a broader spectrum of information, forming the basis of our SLA model. 3. Sensor-Language-Action Dataset Construction Data Regime. We construct our sensor-action testbed from six development datasets and one external dataset spanning three real-world settings: clinical ICU prediction, operating room (OR) decision making, and daily-life continuous glucose monitoring (CGM). The clinical setting includes MC-MED [12], MIMIC-I [11], and MIMIC-IV [10]; the OR setting includes MOVER [28] and VitalDB [16]; 3 Sensor-Language-Action Models Individual context 60 years old male Chief complaint: - ABDOMINAL PAIN - NAUSEA Prior history: - Interstitial pulmonary disease - Hypoxemia ... Observational Relational Action The lead I ECG showed an irregular-periodicity proxy, with an estimated rate of 100.2 beats/min. Respiratory rate was elevated, with a median of 26 breaths/min and a range of 23–30 across 31 measurements. Numerical respiratory-rate measurements and the respiration waveform both indicated rapid breathing. Global Overall findings suggested a pain and nausea symptom burden, supported by the presenting complaints, elevated heart rate, and a pain score of 5. Findings that bear on pain or nausea management are high pain reported nausea provides symptom evidence to the antiemetic. ECG-I Pleth Resp R HR Pain Perf Actions SpO2 Time (minutes) 153010 5 025 20 AzithromycinCefepimeTarget: Antiemetic Figure 2|Multi-faceted sensor-action captioning pipeline. Our captioning pipeline integrates four comple- mentary facets of sensor-action information: (1) Observational captions summarize signal-derived features; (2) Relational captions describe cross-channel evidence; (3) Global captions capture patient-specific context; and (4) Action captions identify evidence from the preceding facets that is relevant to the action label. and the CGM setting includes MetaboNet [36] and PEDAP [33]. We reserve MIMIC-IV as an external clinical cohort to evaluate cross-cohort generalization. For each of the six development datasets, we construct individual-disjoint training, validation, and test splits, and combine the training sets within each setting. The clinical and OR settings contain heterogeneous waveforms and numerical measurements, including ECG, plethysmography, respiration, blood pressure, SpO 2 , and heart rate. The CGM domain contains glucose time series together with basal insulin delivery. Unified Multi-level Action Space. We formulate sensor-action modeling around a common question: given sensor observations and contextual information, what action should be taken? In the clinical and OR settings, actions include medications, procedures, diagnostic evaluations, and treatment adjustments, whereas CGM actions correspond to insulin bolus delivery. We organize actions across datasets into a shared three-level hierarchy:❶action necessity, indicating whether to act;❷action category, specifying the type of action; and❸fine-grained action label, identifying the specific action within a category. For example, administration ofImipenemcan be represented as “Yes” at the necessity level, “antibiotics” at the category level, and “Imipenem” at the fine-grained level. We derive action supervision from the recorded clinical decisions and interventions in each source dataset. Detailed dataset statistics and action definitions are provided in Appendix A. Multi-faceted Captioning Pipeline. We aim to capture physiological evidence from sensor signals together with individual context relevant to decision making. Specifically, we structure each generated caption around four complementary facets: sensor observations, physiological relationships, individual- level context, and action-relevant evidence (Fig. 2). Implementation details are in Appendix B. Observational Captions. These captions describe channel-level physiological findings derived from established measurements and waveform characteristics (e.g., heart rate, respiratory rate, rhythm patterns). They translate directly observable sensor properties into explicit physiological descriptions. Relational Captions. We group physiologically related measurements into complementary feature sets and characterize their temporal and cross-channel relationships. For example, respiratory rate and breathing waveforms are jointly considered to characterize respiratory status, allowing multiple measurements to provide complementary evidence for the same underlying physiological process. Global Captions. We integrate sensor-derived findings with individual-specific context, including available background information and prior user history, to summarize the overall physiological state. This places local sensor evidence within the broader context in which decisions are made. 4 Sensor-Language-Action Models OpenSLA-H Pretrained LLM Action heads Caption decoder Captioning loss Action pred. loss Sensor encoderText encoder Pretrained LLM Caption decoder Captioning loss Action pred. loss OpenSLA-B Action heads Sensor encoder Text encoder Pretrained LLM Action heads Action pred. loss Sensor encoder Text encoder Text decoder Captioning loss VLA-style SensorLM Text encoder Sensor encoder Text encoder Action heads Action pred. loss Supervised Sensor encoder Hierarchical Memory Cond. Captioning loss Cond. Contrasitive loss Figure 3|TheOpenSLAarchitecture and comparison with existing sensor modeling paradigms.OpenSLA integrates pretrained language knowledge with caption supervision for action-relevant sensor understanding and action-conditioned caption decoding for improved alignment between sensor, language, and actions. Action Captions. We condition on recorded action categories to identify decision-relevant evidence from sensor measurements and individual context, and verbalize how this evidence supports the corresponding action. This formulation explicitly connects physiological observations with the evidence underlying downstream decisions. Taken together, we construct a large-scale SLA testbed spanning six development datasets and the external MIMIC-IV cohort, with 79 sensor modalities, 116K individuals, and 60 unique actions across three real-world settings. Our study substantially expands the scale and coverage of prior work (Table 1), while additionally introducing structured action supervision. We generate multiple caption combinations and evaluate their effectiveness in Sec. 5. Unless otherwise specified, we use all four facets in our main experiments. 4. Unifying Sensor-Action Tasks through a Language Interface 4.1. Sensor-Language-Action Modeling We begin with a fundamental question: How can the knowledge encoded in language models be effectively used for sensor-action modeling? To answer this, we study four language-model-centric paradigms:❶ zero-shot inference with multimodal LLMs;❷VLA-style action-only fine-tuning [14];❸SensorLM- style sensor-language alignment [43]; and❹joint fine-tuning with action and language supervision. These paradigms progressively couple language with sensor-action learning, from using pretrained multimodal knowledge without adaptation to directly aligning sensor observations, language, and actions during training (Fig. 3). We compare these paradigms on action prediction using VitalDB. Joint action-language fine-tuning consistently performs best, outperforming VLA-style action-only training (Fig. 8), while maintaining a substantial margin over SensorLM-style sensor-language alignment and zero-shot multimodal LLMs, including GPT-5.6-Luna and DeepSeek-V3.2 (Table 3). These results motivate the Sensor-Language-Action (SLA) paradigm, which jointly models sensor observations, natural-language instructions, and individual context, with both actions and language as prediction targets. We next investigate how sensor representations should be integrated into the language backbone for SLA modeling. We consider three representative adaptation strategies, ranging from lightweight exter- nal alignment to direct adaptation of the LLM:❶LLaVA-style adapter training [21];❷Flamingo-style cross-attention injection [1]; and❸direct Low-Rank Adaptation (LoRA) [9]. Fig. 4 confirms direct LoRA adaptation achieves the strongest performance on most action-prediction tasks, outperforming both adapter-based and cross-attention-based alternatives. We therefore instantiateOpenSLAusing direct LoRA-based adaptation of the language backbone. 5 Sensor-Language-Action Models Sensor encoder Text encoder Pretrained LLM Action heads No decoder Sensor encoder Text encoder Pretrained LLM Action heads VLA-style Caption Sensor encoder Text encoder Pretrained LLM Action heads No condition Caption decoder Sensor encoder Text encoder Pretrained LLM Action heads LLaVA-style Sensor encoder Text encoder Pretrained LLM Action heads Flamingo-style Caption decoder Caption decoder Projector Perceiver resampler Sensor encoder Text encoder Pretrained LLM Action heads OpenSLA Caption decoder Sensor encoder Text encoder Pretrained LLM Action heads OpenSLA Caption decoder Necessity Category AUROC LoRA Necessity Category AUROC Figure 4|Comparison of SLA alignment strategies.OpenSLAdirectly adapts the pretrained LLM with LoRA, outperforming Flamingo-style cross-attention and LLaVA-style projection. 4.2. OpenSLA: Learning Sensor Dynamics through a Language Interface OpenSLA comprises three main components:❶an LLM backbone with structured action-prediction heads;❷a hierarchical sensor encoder; and❸a multimodal fusion decoder (Fig. 3). The model is jointly optimized for action prediction and autoregressive language generation, allowing action decisions and their sensor-grounded descriptions to be learned within a shared representation. Structured action prediction heads. To support multi-level action prediction, we attach three prediction heads to the final-token representation of the LLM hidden states H, corresponding to action necessity, action category, and fine-grained action label. The necessity head uses a binary linear classifier to determine whether an action occurs, while the category head uses independent binary classifiers to predict applicable action categories. For fine-grained prediction, we use a similarity- based head that supports a flexible action space. Given an action category, textual descriptions of its candidate fine-grained actions are encoded into embeddings, and the action whose embedding is most similar to the LLM representation is selected. Hierarchical sensor encoder. Sensor recordings can span long temporal windows and contain heterogeneous channel compositions. To efficiently represent these inputs, we design a hierarchical sensor encoder consisting of channel-wise signal encoding followed by a Hierarchical Memory module. Each channel is first independently processed by a frozen signal encoder, producing sensor tokens Z (0) ∈ℝ 푁 0 ×퐷 , where푁 0 denotes the number of tokens and퐷their feature dimension. The Hierarchical Memory then compresses these tokens across multiple resolutions while preserving both fine-grained sensor patterns and sample-level information. The Hierarchical Memory contains two complementary pathways. The first recursively aggregates sensor tokens across multiple resolutions. Starting from Z (0) , we apply퐿query layers, where the 푙-th layer uses푁 푙 learned queries to cross-attend to Z (푙−1) , producing Z (푙) ∈ℝ 푁 푙 ×퐷 , with푁 푙 < 푁 푙−1 . This recursive compression aggregates information over increasingly long temporal contexts. In parallel, a separate set of global queries attends directly to the original token sequence Z (0) to capture sample-level semantics. The outputs of the two pathways are concatenated to form the sensor tokens provided to the language backbone. Multimodal fusion decoder. Although the hierarchical sensor encoder provides compact multi- resolution representations, the LLM hidden states may not retain the fine-grained sensor evidence required for grounded caption generation. We therefore introduce a gated residual caption decoder that explicitly conditions language generation on both predicted actions and sensor evidence. Specifically, two parallel branches cross-attend to the predicted action profile and retrieved sensor evidence, respectively. The action profile P consists of an action-necessity token and action-category tokens, 6 Sensor-Language-Action Models Table 2 | Multi-level action prediction results for ICU clinical prediction. Method MC-MEDMIMIC-I NecessityCategoryFine-labelNecessityCategoryFine-label AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ BAcc ↑ AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ BAcc ↑ Supervised: ConvTrans [5]65.2 ±1.8 57.5 ±1.4 59.8 ±1.8 53.8 ±1.2 54.7 ±2.1 61.4 ±2.3 59.7 ±1.8 54.8 ±1.7 51.7 ±0.8 57.1 ±1.8 MMIM [7]61.9 ±1.8 53.4 ±1.2 64.6 ±1.7 56.0 ±1.0 53.5 ±2.3 66.0 ±2.1 61.6 ±1.8 64.5 ±1.6 54.4 ±0.7 60.2 ±1.8 TS Foundation Models: Chronos-2 [2]62.5 ±1.8 55.4 ±1.4 56.9 ±1.9 52.7 ±0.9 51.8 ±2.6 60.9 ±2.4 56.6 ±1.8 58.8 ±1.7 52.5 ±0.8 53.2 ±2.0 Moirai [37]62.5 ±1.8 57.4 ±1.5 54.4 ±2.1 51.9 ±1.1 51.7 ±2.4 57.7 ±2.4 55.7 ±1.8 58.7 ±1.8 52.8 ±0.8 53.4 ±1.8 Multi-modal LLMs: GPT-5.6-Luna [25]–60.6 ±1.6–63.1 ±1.3 59.5 ±2.4–54.8 ±1.2–64.0 ±1.2 79.5 ±1.6 DeepSeek-V3.2 [3]–56.1 ±1.8–58.2 ±1.3 54.4 ±2.4–55.6 ±1.8–60.2 ±0.9 72.2 ±2.0 TS-LLM: OpenTSLM [15]–62.9 ±1.5–52.8 ±0.7 50.1 ±1.1–71.2 ±1.7–66.0 ±1.3 67.1 ±1.4 SensorLM [43]–64.6 ±1.6–53.5 ±0.8 50.4 ±1.2–62.9 ±1.8–55.3 ±0.7 51.7 ±0.9 SLA: OpenSLA-B77.6 ±1.572.5 ±1.583.7 ±1.468.6±1.361.2±2.184.0 ±1.675.4 ±1.488.3 ±1.573.1±1.673.7 ±1.5 OpenSLA-H77.0±1.572.2±1.483.7 ±1.368.7 ±1.365.2 ±2.581.9±1.673.3±1.588.1±1.674.5 ±1.776.5±1.2 with each token combining the corresponding action embedding and predicted score. Conditioned on the LLM hidden states H and action profile P, the sensor branch retrieves the most relevant tokens from the waveform and numerical sensor representations to form sensor evidence S. The caption representation is then refined as H ′ = H+ 푔 푝 푓 푝 H, P + 푔 푠 푓 푠 H, S ,(1) where푓 푝 and푓 푠 denote cross-attention modules over the action profile and sensor evidence, respectively, and푔 푝 and푔 푠 are learnable gates. The refined states H ′ are passed through the original language-model head for autoregressive caption generation. Training objective. We denote the action prediction loss asL act , which aggregates the losses for action necessity, category, and fine-grained action prediction, and optimize caption generation using autoregressive cross-entropyL text . The overall training objective isL=L act + 휆 text L text , where휆 text controls the contribution of language supervision. During training, we adapt the LLM backbone using LoRA and jointly optimize the hierarchical sensor encoder and multimodal fusion decoder, while keeping the remaining pretrained modules frozen. Implementation details are in Appendix C.1. 5. Experiment & Results Baselines. We compareOpenSLAwith four families of sensor modeling methods.❶Supervised models. We train ConvTrans [5] and MMIM [7,8] from scratch.❷Time-series foundation models. We linearly probe Chronos-2 [2], Moirai [37], Gluformer [22] for action prediction.❸Sensor- language models. We adapt OpenTSLM [15] and SensorLM [43] using our training corpus.❹ Multimodal LLMs. We prompt GPT-5.6-Luna [25] and DeepSeek-V3.2 [3] through CodeAct-style Python execution [34]. Additional details are provided in Appendix C.2. Implementation details. We denote the base and hierarchical variants of our model byOpenSLA-B andOpenSLA-H, respectively. AllOpenSLAvariants use a Qwen-3.5-2B backbone adapted with LoRA. For fair comparison, all methods trained on our cohorts use the same training data and compute budget, with signal encoders initialized from the same domain-specific DINO-style pretrained models 7 Sensor-Language-Action Models Table 3 | Multi-level action prediction results for operating room decision making. Method MOVERVitalDB NecessityCategoryFine-labelNecessityCategoryFine-label AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ BAcc ↑ AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ BAcc ↑ Supervised: ConvTrans [5]47.6 ±3.9 48.0 ±3.3 62.0 ±3.8 56.9 ±3.3 56.0 ±5.2 52.1 ±3.4 51.4 ±2.7 70.9 ±4.2 66.0 ±3.9 51.4 ±12.6 MMIM [7]53.5 ±3.6 51.8 ±3.2 63.2 ±3.6 59.0 ±3.0 55.9 ±4.0 51.7 ±3.5 51.1 ±2.5 63.9 ±4.6 56.8 ±4.1 57.2 ±9.7 TS Foundation Models: Chronos-2 [2]43.6 ±4.0 46.3 ±3.0 63.3 ±3.7 60.5 ±3.3 57.5 ±5.2 53.9 ±3.4 52.1 ±2.6 65.7 ±4.0 62.3 ±2.6 58.0 ±12.0 Moirai [37]46.3 ±3.7 45.7 ±2.9 63.7 ±3.7 57.0 ±3.1 51.9 ±4.2 49.8 ±3.7 49.7 ±1.8 60.9 ±5.8 57.5 ±4.0 54.8 ±16.2 Multi-modal LLMs: GPT-5.6-Luna [25]–45.0 ±3.3–52.4 ±1.3 69.2 ±4.6–50.1 ±2.3–55.7 ±2.3 78.6 ±1.6 DeepSeek-V3.2 [3]–48.1 ±3.2–54.3 ±2.4 55.9 ±4.0–50.8 ±1.9–53.8 ±1.8 66.2 ±3.4 TS-LLM: OpenTSLM [15]–71.5 ±3.0–55.2 ±1.9 52.3 ±1.9–48.9 ±2.2–49.9 ±1.0 51.2 ±2.5 SensorLM [43]–49.9 ±1.0–52.9 ±1.9 50.1 ±0.2–52.1 ±1.9–51.0 ±0.6 51.7 ±1.0 SLA: OpenSLA-B83.4±3.174.2±3.080.3 ±2.769.0±2.766.4 ±3.274.8±2.859.0±1.785.9±1.579.5±2.178.9 ±1.0 OpenSLA-H85.9 ±2.780.6 ±2.879.9±3.169.9 ±2.367.6±2.677.1 ±2.466.3 ±2.486.3 ±1.480.8 ±1.878.5 ±1.3 Table 4 | Multi-level action prediction results for CGM-based metabolic health prediction. Method MetaboNetPEDAP NecessityCategoryNecessityCategory AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ Supervised: ConvTrans [5]66.5 ±1.5 62.2 ±1.3 53.8 ±0.9 52.7 ±0.7 79.0 ±2.8 72.8 ±2.6 70.5 ±2.8 66.2 ±2.5 MMIM [7]66.3 ±1.9 60.2 ±1.6 62.7 ±2.1 58.7 ±1.6 64.7 ±3.5 61.7 ±3.6 58.5 ±3.6 56.3±3.0 TS Foundation Models: Chronos-2 [2]63.6 ±1.5 60.3 ±1.3 52.5 ±0.7 51.7 ±0.5 71.7 ±3.2 66.1 ±2.6 55.8 ±1.9 54.0 ±1.5 GluFormer [22]52.0 ±1.1 51.3 ±0.8 51.0 ±1.2 50.5 ±0.9 53.5 ±1.3 52.2 ±1.2 53.3 ±2.7 52.8 ±1.6 Multi-modal LLMs: GPT-5.6-Luna [25]–49.9 ±0.4–50.4 ±0.2–52.0 ±0.8–50.1 ±0.3 DeepSeek-V3.2 [3]–51.2 ±0.7–50.5 ±0.3–52.5 ±0.5–50.6 ±0.3 TS-LLM: OpenTSLM [15]–62.1 ±1.5–53.9 ±1.1–64.6 ±3.7–51.6 ±1.2 SensorLM [43]–62.9 ±1.5–56.8 ±1.4–59.4 ±3.0–51.8 ±0.5 SLA: OpenSLA-B73.3 ±1.565.0 ±1.366.5±2.159.2±2.281.2 ±3.375.5 ±2.965.2±5.556.3±1.9 OpenSLA-H71.9±1.564.3±1.367.2 ±2.259.6 ±2.180.9±3.375.1±2.859.7 ±2.453.9 ±0.8 [26,31], where applicable. Training is capped to 24 hours on H200 GPU nodes. Further details are provided in Appendix C. Evaluation. We evaluateOpenSLAagainst the baselines across the clinical, OR, and CGM domains on five tasks spanning three capabilities.❶Action Prediction. We report macro-AUROC and balanced accuracy for action necessity and category prediction, and balanced accuracy for fine-grained action prediction. Metrics are macro-averaged across labels at each action granularity. Fine-grained performance is evaluated within ground-truth-positive action categories.❷State Understanding. Following [40], we report the mean absolute error (MAE) between sensor measurements extracted from generated captions and their corresponding ground-truth values.❸Action Explanation. 8 Sensor-Language-Action Models Table 5|Caption-based sensor-state estimation results. Results are evaluated using MAE ↓ across sensor-state estimation tasks. Method ClinicalOperating RoomCGM Cardiac Index ECG-I Rate RESP Rate Resp. Rate Temperature ETCO 2 Basal Rate Carb Amount Multi-modal LLMs: GPT-5.6-Luna [25]–28.4–0.072.70 DeepSeek-V3.2 [3]–26.58.27–0.433.70 TS-LLM: OpenTSLM [15]1.1321.46.712.660.665.470.581.20 SensorLM [43]–20.55.872.560.669.760.5413.5 SLA: OpenSLA-B0.9320.11.592.480.645.130.070.73 OpenSLA-H1.0115.91.171.410.473.180.350.38 ECG-I Pleth Resp R 1min HRV 5min HRV HR Perf Time (minutes) 15 3010 5 025 20 Ground truth OpenSLA ChatGPT 5.6 Luna For heart-rate channel, the dominant finding is elevated heart-rate pattern; HR median 100, range 95-108, n=30. The cardiac waveform channel shows variable ECG waveform periodicity; rate proxy 95.2/min, IQR 0.40. Across the recent monitored interval, pulse rate shows elevated heart-rate pattern; HR median 100, range 97-104, n=30 ECG Lead I: irregular Lead I rhythm proxy; rate proxy 95.2/min, IQR 0.40. At the decision time, the observed vital signs show sinus-rate cardiac activity with HR approximately 90 bpm Figure 5|Sensor-grounded caption generation.OpenSLAcaptures both sensor states and fine-grained signal statistics, while the LLM baseline misses key physiological details. Following prior sensor-language evaluation [43,40], we measure reference-based agreement between generated explanations and action-evidence captions using BERTScore [42]. Scores are computed with DistilBERT [29] following the setup of [38]. OpenSLApredicts actions across domains and granularities. We evaluate action necessity, category, and fine-grained action prediction across the three domains, where applicable. As shown in Table 2, Table 3, and Table 4,OpenSLAconsistently achieves strong performance across domains and action granularities, with particularly large gains for action necessity and category prediction. The results also reveal distinct domain-specific patterns. Sensor-only action prediction is most challenging in the OR setting, whileOpenSLAachieves its largest gains in the clinical and OR domains, suggesting that individual-level textual context is especially informative for context-dependent clinical decisions. Across most settings, performance improves from necessity prediction to category and fine-grained prediction, as the candidate action space becomes increasingly constrained once an intervention and its category are known. CGM is a notable exception: action-category prediction remains challenging because insulin decisions can depend on future carbohydrate intake that is unavailable in the observed context. Language supervision enriches sensor-action representations beyond action prediction. We further assess whether the generated captions accurately recover physiological quantities from the observed sensor history, without training an additional regression head. As shown in Table 5,OpenSLA yields the most accurate physiological state estimates across the compared methods, indicating that its generated language retains detailed information about the underlying sensor state. The gains are 9 Sensor-Language-Action Models Table 6|Action explanation results. BERTScore ↑ [42] (F1) measures semantic agreement between generated text and reference action-evidence captions. Method ClinicalOperating RoomCGM MC-MED MIMIC-I MOVER VitalDB MetaboNet PEDAP Multi-modal LLMs: GPT-5.6-Luna [25]75.175.170.771.476.176.6 DeepSeek-V3.2 [3]73.873.269.971.274.575.5 TS-LLM: OpenTSLM [15]79.580.381.780.784.483.2 SensorLM [43]77.278.867.969.876.176.4 SLA: OpenSLA-B81.578.681.580.985.687.6 OpenSLA-H82.080.981.080.780.476.7 Similar Action Unseen Action BAcc Zero-Shot Imipenem Prediction Unseen Action in Representation Space Imipenem (unseen) Opioid analgesicsAntibiotics Meropenem Others AB Figure 6|Generalization to unseen actions. (A) Zero-shot prediction of the held-out actionImipenem. (B) Visualization of the learnedOpenSLAaction representation space, where the unseenImipenemrepresentation lies near semantically related antibiotics. most pronounced in the clinical setting, where effective prediction requires integrating complementary information across multiple sensor modalities. OpenSLAcaptures action-relevant evidence through language. Beyond structured action prediction, we assess whether generated descriptions recover the evidence associated with each recorded action. Table 6 confirms thatOpenSLAachieves the strongest overall agreement with reference action-evidence captions across domains. Overall,OpenSLAmore faithfully captures the sensor and contextual evidence associated with downstream actions. To examine this grounding directly, we qualitatively compare generated captions with their underlying sensor observations on a representative case. As shown in Fig. 5,OpenSLAproduces more precise and physiologically grounded descriptions than zero-shot multimodal LLMs, including GPT-5.6-Luna and DeepSeek-V3.2, with closer agreement between the generated explanation and the observed sensor state. 6. Generalization and Predictive Structure in SLA Representations Generalization to unseen actions. Beyond in-distribution action prediction, we ask whether sensor- language-action alignment induces a representation space that supports generalization to previously unseen actions. We first examine the geometry of representations learned on MIMIC-I by projecting backbone features into a PCA-reduced space. As shown in Fig. 6, semantically related actions occupy 10 Sensor-Language-Action Models MAE Emb. Interpolation Reveals Continuous Action Variation 0.05 U Insulin Action (U) 14.4 U Interpolated Virtual Emb. Emb. of Real Sample 1.2 U 7.7 U OpenSLAReads out Future Glucose Status 17:10 Current Signal19:10 Future Signal21:10 Time (H:M) CGM Basal 훼=0.4 훼=0.8 Emb. Mean. CGM Min CGM Max CGM Mean CGM MinCGM MaxCGM Prob 휶:Interpolate ratio A B OpenSLA Glucose Summary Figure 7|Continuous action structure and future-state information emerge in SLA representations. (A) Interpolating between learned representations yields smoothly varying insulin actions among the nearest real samples, revealing continuous action structure in the embedding space. (B) Representations derived from current CGM observations support accurate readout of future glucose statistics, indicating that SLA representations retain information predictive of subsequent sensor states. nearby regions. For example,antibiotic medicationsform a coherent cluster, and an unseen actionImipenem(excluded from training but belonging to the same semantic category), is positioned near other antibiotics. Motivated by this structure, we test whetherOpenSLAcan predict unseen actions directly from the learned representation space. We hold outImipenemduring training and perform zero-shot prediction by matching sample representations to textual action embeddings through similarity. OpenSLAachieves the highest balanced accuracy on this unseen-action task (Fig. 6), suggesting that SLA alignment supports transfer through the semantic structure shared across related actions. Continuous action structure emerges in the learned representations. We next ask whether this organization extends beyond discrete action categories to continuous action spaces. We interpolate between representations associated with two distinct insulin actions. At each interpolated point, we retrieve the nearest real-sample representation and examine its corresponding action value. Fig. 7 verifies the retrieved insulin actions vary monotonically along the interpolation trajectory, revealing a continuous action structure within the learned representation space. Future sensor states are recoverable from SLA representations. We further ask whether rep- resentations learned without explicit future-state supervision retain information about subsequent sensor dynamics. For each 2-hour CGM observation window, we define future targets using the mean, minimum, maximum, and slope of glucose over the subsequent 2 hours. With the SLA backbone frozen, we train ridge readouts on these targets using individual-grouped nested cross-validation. Fig. 7 confirms thatOpenSLArepresentations enable accurate prediction of future glucose states. These results suggest that SLA learning captures predictive physiological structure beyond the action targets used during backbone training. Transfer to unseen datasets. To examine gen- eralization beyond the training cohorts, we di- rectly applyOpenSLAto the external MIMIC-IV cohort without adaptation. Table 7 shows that OpenSLAoutperforms baselines on both action necessity and category prediction. These results demonstrate that the learned sensor-language- action associations transfer to an unseen clinical cohort under a shared action vocabulary. Table 7|Zero-shot transfer to the external MIMIC-IV cohort. Method NecessityCategory AUROC ↑ BAcc ↑ AUROC ↑ BAcc ↑ OpenTSLM [15]–69.30–54.33 SensorLM [43]–67.70–50.02 OpenSLA-B81.7574.4074.6456.35 OpenSLA-H80.5273.0074.9058.49 11 Sensor-Language-Action Models Sensor encoder Text encoder Pretrained LLM Action heads No decoder Sensor encoder Text encoder Pretrained LLM Action heads VLA-style Caption Sensor encoder Text encoder Pretrained LLM Action heads No condition Caption decoder Sensor encoder Text encoder Pretrained LLM Action heads LLaVA-style Sensor encoder Text encoder Pretrained LLM Action heads Flamingo-style Caption decoder Caption decoder Projector Perceiver resampler Sensor encoder Text encoder Pretrained LLM Action heads OpenSLA Caption decoder Sensor encoder Text encoder Pretrained LLM Action heads OpenSLA Caption decoder Necessity Category AUROC LoRA Necessity Category AUROC Figure 8|Ablation of language supervision and action-conditioned decoding. Removing caption supervision yields the largest performance drop, while removing action conditioning or the caption decoder further reduces action prediction performance. The fullOpenSLAmodel achieves the strongest results across both necessity and category prediction. Which model components support action prediction? We ablate the main components ofOpenSLAto identify their contributions to sensor-action modeling. As shown in Fig. 8, the full model achieves the strongest overall performance. The largest degradation occurs under VLA-style action-only training, highlighting the benefit of introducing language supervision into sensor-action learning. Removing action conditioning or the multimodal fusion decoder further reduces performance, indicating that explicitly connecting action predictions with sensor-grounded language provides additional gains. Table 8|Ablation of caption supervision facets. AUROC ↑ (%) is used as the metric. Caption VariantsMOVERVitalDB Obs. Rel. Global Action Necessity Category Necessity Category ✓ ✗ ✗81.576.264.181.1 ✗ ✗ ✗✓80.073.360.980.2 ✓ ✗ ✗✓81.076.165.282.7 ✓ ✓ ✗79.276.465.982.6 ✓ ✗ ✓ ✓81.677.366.883.9 ✓ ✓ ✗✓80.776.464.482.2 ✓ ✓ ✓✗81.677.564.382.3 ✓83.480.374.885.9 Which caption facets support action pre- diction? To quantify the contribution of different caption components, we ab- late over caption combinations in Table 8. Adding global captions improves necessity and category AUROC on MOVER, although the gains are less consistent on VitalDB. Incorporating action captions yields the strongest overall results, improving neces- sity and category AUROC on both datasets relative to supervision without the action facet. These results indicate that action- relevant language provides useful auxiliary supervision for learning sensor-action associations. Table 9|Ablation of caption generation and action condi- tioning. AUROC ↑ (%) is used as the metric. Method MOVERVitalDB Necessity Category Necessity Category Action only78.172.154.172.8 No cond.80.477.266.384.4 No decoder78.376.465.184.1 OpenSLA-B83.480.374.885.9 Is caption supervision alone sufficient? Finally, we separate the benefit of language supervision from the mechanism used to connect language with action prediction. The action-only variant removes caption generation and its associated loss; the no- action-conditioning variant retains caption generation but removes conditioning on predicted actions; and the no-decoder vari- ant retains language supervision but gen- erates captions directly from the LLM backbone. Table 9 verifies that the full model achieves the highest AUROC across all four endpoints. Both caption-generating variants outperform action-only training, confirming the benefit of language supervision, while explicit action conditioning and the multimodal fusion decoder provide additional improvements. 12 Sensor-Language-Action Models 7. Discussion Limitation.OpenSLAhas not yet undergone clinical validation and is not intended for direct clinical decision-making. Real-world deployment would require prospective validation, safety evaluation, and appropriate regulatory review. Our current implementation trains separate models for each domain, in part because their action spaces are defined independently. An important next step is to develop a generalist SLA model that operates across domains through shared representations of heterogeneous actions. In addition, our experiments assume a fixed set of sensor modalities and input channels, whereas real-world sensing systems vary across devices, signal availability, and acquisition settings. Extending SLA to robustly handle such sensor heterogeneity remains an important direction for future work. Conclusion. We present Sensor-Language-Action (SLA) modeling, extending sensor intelligence from understanding what is observed toward reasoning about the actions those observations support. SLA uses language as a semantic interface that connects sensor observations, language, and actions within a shared modeling framework. We instantiate this paradigm via a large-scale benchmark, multi-faceted language supervision, andOpenSLA, achieving strong performance across action prediction, state understanding, and action explanation. Further analyses show generalization to unseen actions and datasets, semantic and continuous structure in the learned action space, and predictive information about future sensor states. Together, SLA provides a foundation for general sensor models that jointly reason over sensing, language, and action. Acknowledgments We gratefully acknowledge the support of NIH R21-EB038753, the Amazon Research Awards, the NVIDIA Academic Grant, and the Biswas Foundation. Any opinions, findings, conclusions, or rec- ommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders. References [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. [2]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerron, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael Bohlke-Schneider. Chronos-2: From univariate to universal forecasting. arXiv preprint arXiv:2510.15821, 2025. URL https://arxiv.org/abs/2510.15821. [3]DeepSeek-AI et al. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. URL https://arxiv.org/abs/2512.02556. [4]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019. 13 Sensor-Language-Action Models [5]Navid Mohammadi Foumani, Chang Wei Tan, Geoffrey I Webb, and Mahsa Salehi. Improving position encoding of transformers for multivariate time series classification: Nm foumani et al. Data mining and knowledge discovery, 38(1):22–48, 2024. [6]Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MO- MENT: A family of open time-series foundation models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 16115–16152. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/goswami24a.html. [7]Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.723. URLhttps: //aclanthology.org/2021.emnlp-main.723/. [8]Shahad Hardan, Mai A Shaaban, Jehad Abdalla, and Mohammad Yaqub. Affordable and real- time antimicrobial resistance prediction from multimodal electronic health records. Scientific Reports, 14(1):16464, 2024. [9] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. [10] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV. PhysioNet, June 2022. doi: 10.13026/7vcr-e114. URLhttps://doi.org/10. 13026/7vcr-e114. Version 2.0. [11]Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-I, a freely accessible critical care database. Scientific Data, 3:160035, 2016. doi: 10.1038/sdata. 2016.35. [12]Aman Kansal, Emma Chen, Boyang Tom Jin, Pranav Rajpurkar, and David A. Kim. MC-MED, multimodal clinical monitoring in the emergency department. Scientific Data, 12(1):1094, 2025. doi: 10.1038/s41597-025-05419-5. [13] Justin Khasentino, Anastasiya Belyaeva, Xin Liu, Zhun Yang, Nicholas A Furlotte, Chace Lee, Erik Schenck, Yojan Patel, Jian Cui, Logan Douglas Schneider, et al. A personal health large language model for sleep and fitness coaching. Nature Medicine, 31(10):3394–3403, 2025. [14] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. [15]Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A. Xu, Winnie Chow, Martin Maritsch, Robert Jakob, Ning Wang, Juncheng Liu, Aradhana Verma, Brian Han, Daniel Seung Kim, Henry Chubb, Scott Ceresnak, Aydin Zahedivash, Alexander Tarlochan Singh Sandhu, Fatima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver Aalami, Filipe Barata, and Paul Schmiedmayer. OpenTSLM: Time-series language models for reasoning over multivariate medical text- and time-series data. arXiv preprint arXiv:2510.02410, 2026. URL https://arxiv.org/abs/2510.02410. [16]Hyung-Chul Lee, Yoonsang Park, Soo Bin Yoon, Seong Mi Yang, Dongnyeok Park, and Chul-Woo Jung. VitalDB, a high-fidelity multi-parameter vital signs database in surgical patients. Scientific Data, 9(1):279, 2022. doi: 10.1038/s41597-022-01411-5. 14 Sensor-Language-Action Models [17]Seung Mi Lee, Garam Lee, Tae Kyong Kim, Trang Le, Jie Hao, Young Mi Jung, Chan-Wook Park, Joong Shin Park, Jong Kwan Jun, Hyung-Chul Lee, et al. Development and validation of a prediction model for need for massive transfusion during surgery using intraoperative hemodynamic monitoring data. JAMA network open, 5(12):e2246637, 2022. [18] Jun Li, Aaron D Aguirre, Valdery Moura Junior, Jiarui Jin, Che Liu, Lanhai Zhong, Chenxi Sun, Gari Clifford, M Brandon Westover, and Shenda Hong. An electrocardiogram foundation model built on over 10 million recordings. Nejm ai, 2(7):AIoa2401033, 2025. [19]Sirui Li, Shuhan Xiao, Mihir Joshi, Ahmed Metwally, Daniel McDuff, Wei Wang, and Yuzhe Yang. Hearts: Benchmarking llm reasoning on health time series. arXiv preprint arXiv:2603.06638, 2026. [20]Zechen Li, Keerthana Natarajan, Weizhi Zhang, Menglian Zhou, Simon A Lee, Yuwei Zhang, Maxwell A Xu, Zeinab Esmaeilpour, Flora D Salim, Mark Malhotra, et al. Glucofm: A dual-stream foundation model for continuous glucose monitoring. arXiv preprint arXiv:2605.30865, 2026. [21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. [22]Guy Lutsker, Gal Sapir, Smadar Shilo, Jordi Merino, Anastasia Godneva, Jerry R Greenfield, Dorit Samocha-Bonet, Raja Dhir, Francisco Gude, Shie Mannor, et al. A foundation model for continuous glucose monitoring data. Nature, 650(8103):978–986, 2026. [23]Ahmed A Metwally, A Ali Heydari, Daniel McDuff, Alexandru Solot, Zeinab Esmaeilpour, An- thony Z Faranesh, Menglian Zhou, Girish Narayanswamy, Maxwell A Xu, Xin Liu, et al. Insulin resistance prediction from wearables and routine blood biomarkers. Nature, pages 1–11, 2026. [24]Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jake Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, and Daniel McDuff. Scaling wearable foundation models. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=yb4QE6b22f. [25]OpenAI. GPT-5.6 Luna. OpenAI API documentation, 2026. URLhttps://developers.openai. com/api/docs/models/gpt-5.6-luna. Accessed September 18, 2026. [26]Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. URL https://arxiv.org/abs/2304.07193. [27]Arvind Pillai, Dimitris Spathis, Fahim Kawsar, and Mohammad Malekzadeh. Papagei: Open foundation models for optical physiological signals. In International Conference on Learning Representations, volume 2025, pages 48230–48261, 2025. [28]Muntaha Samad, Mirana Angel, Joseph Rinehart, Yuzo Kanomata, Pierre Baldi, and Maxime Cannesson. Medical informatics operating room vitals and events repository (MOVER): a public- access operating room database. JAMIA Open, 6(4):ooad084, 2023. doi: 10.1093/jamiaopen/ ooad084. 15 Sensor-Language-Action Models [29]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [30]Zitao Shuai, Zongzhe Xu, Yuntian Wu, Sirui Li, Tianhong Li, and Yuzhe Yang. Signal or noise? understanding generative models for real-world sensor time series. arXiv preprint arXiv:2607.04245, 2026. [31]Zitao Shuai, Zongzhe Xu, David Yang, Wei Wang, and Yuzhe Yang. OSF: On pre-training and scaling of sleep foundation models. arXiv preprint arXiv:2603.00190, 2026. URLhttps: //arxiv.org/abs/2603.00190. [32]Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding with deep neural networks. In Machine learning for healthcare conference, pages 322–337. PMLR, 2017. [33]R. Paul Wadwa, Zachariah W. Reed, Bruce A. Buckingham, Mark D. DeBoer, Laya Ekhlaspour, Gregory P. Forlenza, Melissa Schoelwer, John Lum, Craig Kollman, Roy W. Beck, and Marc D. Breton. Trial of hybrid closed-loop control in young children with type 1 diabetes. New England Journal of Medicine, 388(11):991–1001, 2023. doi: 10.1056/NEJMoa2210834. [34]Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 50208–50232. PMLR, 2024. URL https://proceedings.mlr.press/v235/wang24h.html. [35] Aaron C Weidman, Salim Malakouti, David D Salcido, Chase Zikmund, Ravi Patel, Leonard S Weiss, Michael R Pinsky, Gilles Clermont, Jonathan Elmer, Ronald K Poropatich, et al. A machine learning trauma triage model for critical care transport. JAMA Network Open, 8(6):e259639, 2025. [36]Miriam K. Wolff, Peter Calhoun, Eleonora Maria Aiello, Yao Qin, and Sam F. Royston. MetaboNet: The largest publicly available consolidated dataset for type 1 diabetes management. arXiv preprint arXiv:2601.11505, 2026. URL https://arxiv.org/abs/2601.11505. [37]Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 53140–53164. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/ woo24a.html. [38]Justin Xu, Xi Zhang, Javid Abderezaei, Julie Bauml, Roger Boodoo, Fatemeh Haghighi, Ali Ganjizadeh, Eric Brattain, Dave Van Veen, Zaiqiao Meng, et al. Radeval: A framework for radiology text evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 546–557, 2025. [39] Zongzhe Xu, Aakarsh Anand, Sarah Jiang, Chuntung Zhuang, Zitao Shuai, Sriram Sankararaman, and Yuzhe Yang. Inertia-1: An open exploration of wearable motion foundation models. arXiv preprint arXiv:2607.06617, 2026. [40]Zongzhe Xu, Zitao Shuai, Eideen Mozaffari, Ravi S. Aysola, Rajesh Kumar, and Yuzhe Yang. SleepLM: Natural-language intelligence for human sleep. arXiv preprint arXiv:2602.23605, 2026. URL https://arxiv.org/abs/2602.23605. 16 Sensor-Language-Action Models [41] Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. In The Eleventh International Conference on Learning Representations, 2023. [42]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. [43]Yuwei Zhang, Kumar Ayush, Siyuan Qiao, A. Ali Heydari, Girish Narayanswamy, Maxwell A. Xu, Ahmed A. Metwally, Shawn Xu, Jake Garrison, Xuhai Xu, Tim Althoff, Yun Liu, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Cecilia Mascolo, Xin Liu, Daniel McDuff, and Yuzhe Yang. SensorLM: Learning the language of wearable sensors. arXiv preprint arXiv:2506.09108, 2025. URL https://arxiv.org/abs/2506.09108. 17 Sensor-Language-Action Models A. Details of Datasets A.1. Dataset Splits and Statistics Dataset usage. Our development datasets include MC-MED [12] and MIMIC-I [11] for clinical care, MOVER [28] and VitalDB [16] for operating room (OR) care, and MetaboNet [36] and PEDAP [33] for continuous glucose monitoring (CGM). All datasets above contribute to training and evaluation splits. We additionally use MIMIC-IV [10] dataset as an external transfer cohort, which is fully unseen during training. Table 11 summarizes the cohort sizes, with a combined size of 116K across the seven datasets. Table 10 reports the resulting training, validation, and test pools after quality filtering for action evaluation. Across the six development datasets, the training pool contains more than 2.3 million windows. For evaluation, test windows are balanced with respect to action necessity; For test evaluation, we balance-sample windows with and without a target action, and apply the same strategy to 1,000 MIMIC-IV windows for external evaluation. Table 10 | Dataset usage and split sizes. Split ClinicalOperating RoomCGMExternal MC-MED [12] MIMIC-I [11] MOVER [28] VitalDB [16] MetaboNet [36] PEDAP [33] MIMIC-IV [10] Train90,41216,184113,86548,8001,945,570127,437– Validation4031292381,00010,00010,000– Test2,7533,4027952,00010,00010,0001,000 Table 11 | Full cohort sizes across the seven datasets. Dataset ClinicalOperating RoomCGM Total MC-MED [12] MIMIC-I [11] MIMIC-IV [10] MOVER [28] VitalDB [16] MetaboNet [36] PEDAP [33] Cohort size49,19635,27612226,3293,6571,09565 115,740 Sensor Modalities and Coverage. We use waveform signals and numerical measurements as sensor inputs across domains. As summarized in Table 12, waveform modalities include ECG, plethysmogra- phy, respiration, invasive pressure, capnography, EEG, and CGM glucose traces, while numerical inputs include vital signs, hemodynamic and ventilator measurements, laboratory values, and historical insulin delivery. We further perform dataset-specific quality control and sensor harmonization to ensure reliable model inputs. In particular, we have reconciled cross-dataset modality aliases and acquisition variants and filtered low-quality waveform windows where applicable. The resulting sample-level coverage is reported in Table 12. Waveform and numerical inputs provide broad and complementary availability across datasets, with near-complete coverage for several cohorts and sensor types. A.2. Sensor-Action Sample Construction Input State. Each sample is anchored at a decision time푇 0 and combines a historical sensor window with textual individual context. The input windows span 30 minutes for clinical care, 5 minutes for operating room (OR) care, and 2 hours for CGM. The CGM model input contains 24 five-minute slots preceding the decision slot. Text context records available individual background, presentation, and prior actions, depending on the source dataset. The recorded target action is stored separately as supervision. Action Targets. The action hierarchy addresses whether to act (necessity), which type of action to select (category), and which particular action to take (fine-grained label), where applicable. We 18 Sensor-Language-Action Models Table 12 | Sensor modalities and training-set coverage for development datasets. Sensor Type ModalitiesTraining Coverage (%) ClinicalMC-MED [12] MIMIC-I [11] WaveformECG leads; plethysmography; respiration; arterial and other invasive-pressure waveforms. 70.878.8 Numerical Vital signs (HR, R, SpO 2 , blood pressure, temperature); hemodynamics; ventilator measurements; oxygen delivery; glucose; neurological and pain scores; urine output. 99.042.9 Operating Room (OR)MOVER [28]VitalDB [16] WaveformECG leads; plethysmography; arterial and central venous pressure; capnography; airway pressure; EEG. 25.1100.0 NumericalVital signs; invasive and non-invasive blood pressure; cardiac output and related hemodynamics; ventilation and respiratory gases; BIS and EEG-derived indices; laboratory measurements; urine output. 99.6100.0 Continuous Glucose Monitoring (CGM)MetaboNet [36]PEDAP [33] WaveformGlucose time series.100.0100.0 NumericalHistorical basal-insulin delivery.100.0100.0 derive supervision for these levels from recorded actions within the target window. Clinical and operating room (OR) samples can have multiple category and label targets. The CGM task predicts bolus occurrence and its dose bin, without a separate fine-grained label vocabulary. Each sample pairs a historical sensor window with action targets defined relative to the decision time푇 0 . Table 13 summarizes the positive and negative sample construction rules and the action types across domains. Table 13 | Action targets and negative-sample construction. DatasetTarget intervalPositive definitionNegative construction MC-MED, MIMIC-I [푇 0 , 푇 0 + 5 min]At least one mapped clinical action. No mapped action; retained and history-matched evaluation controls. MOVER(푇 0 , 푇 0 + 2 min]A mapped medication, procedure, device, or diagnostic event. No mapped event; same-grid, within-case sampling preferentially matched by procedure progress. VitalDB(푇 0 , 푇 0 + 2 min]An explicit command or control change. No mapped change; uncertain initial or post-gap observations are excluded. MetaboNet, PEDAP Decision slot 푇 0 A canonical insulin-bolus onset. No onset; within-individual sampling excludes recent boluses and overlapping histories. Action Category. Source-specific action names and codes are mapped to categories and fine-grained labels based on clinical practice. Clinical and operating room (OR) samples may contain multiple actions, with repeated occurrences of the same label counted once within the target window. For CGM, necessity denotes bolus occurrence, and positive boluses are assigned to four mutually exclusive dose 19 Sensor-Language-Action Models categories: (0, 1), [1, 2), [2, 4), and [4,∞) U. The full action space contain 39 Clinical categories, 17 operating room (OR) categories, and four CGM dose categories, totaling 60 domain-specific action categories. Table 14 summarizes the number of action targets and their positive test-window support across domains, reported as minimum, median, and maximum counts. Table 14|Number of action categories and positive-sample support. Support is reported as the minimum / median / maximum number of positive samples across targets. Dataset Action CategoryPositive Window Support Categories Fine-grained labelsCategoryFine-grained label MC-MED [12]295612 / 41 / 13210 / 18.5 / 132 MIMIC-I [11]286012 / 75.5 / 43710 / 43 / 263 MOVER [28]132210 / 21 / 4810 / 13 / 28 VitalDB [16]51423 / 219 / 83123 / 218.5 / 343 MetaboNet [36]4N/A1,166 / 1,274 / 1,382N/A PEDAP [33]4N/A273 / 886.5 / 3,180N/A B. Multi-Faceted Caption Construction We construct caption targets through structured fact extraction followed by rule-based selection and text rendering. For each decision time푇 0 , the pipeline extracts findings from the preceding sensor window and available individual context. Intermediate records retaining for each finding its measurement identity, numerical summary, temporal information, and source. These records are organized into four complementary caption facets. B.1. Observational Caption Construction Observational captions describe individual measurements and waveform channels. For numerical observations, we summarized statistics such as the median, range, and number of observations, and associated these summaries with measurement-specific state descriptors. Sparsely sampled measurements are reported by their observed values. For waveforms, channel-specific extractors produce rate, periodicity, amplitude, and pattern descriptors where supported. waveform-derived estimates are reported separately from charted measurements. The selected findings are rendered with their measurement names and numerical summaries. When signal quality is insufficient, the caption notes limited availability instead of reporting the correspond- ing descriptors. For CGM, the summary covers the glucose level, range, trend, and latest reading, with range-occupancy and variability descriptions added when applicable. B.2. Relational Caption Construction Relational captions describe temporal structure and relationships between physiological measurements. Temporal descriptions are derived from the extracted trajectories, distinguishing directional changes from persistent or relatively unchanged states and are produced only when the observations are sufficient to support a trend. Where historical event records are available, the captions also summarize their timing and duration, such as earlier carbohydrate entries, insulin delivery, or basal-rate changes in CGM. 20 Sensor-Language-Action Models Cross-channel descriptions use predefined, physiologically related channel pairs rather than unre- stricted pairwise correlations. For example numerical vital signs and their corresponding waveforms. A joint statement is generated only when the relevant observations are available and satisfy the corresponding compatibility and quality rules. B.3. Global Caption Construction Global captions combine available individual context with selected physiological findings. Context includes recorded background information, presentation, and relevant history. The summary is chosen from a predefined set of state descriptions according to the findings that support each candidate. For clinical and operating room (OR) data, candidate states are ranked by the support provided by available findings and are presented together with their main supporting evidence. CGM uses predefined combinations of glucose dynamics and historical insulin or carbohydrate information to identify state descriptions. When no state description is sufficiently supported, the caption explicitly reports this. B.4. Action Caption Construction Action captions connect sensor observations and individual context with action-relevant evidence. We use predefined matching rules to select relevant findings from the extracted facts and summarize how they support or qualify an action-related interpretation. Where applicable, recorded action categories guide this selection, while the underlying evidence is drawn from pre-decision observations. These descriptions are intended as evidence associations, not as causal explanations or treatment recommendations. Finally, the selected facts are verbalized using sentence templates and phrase banks that vary wording while retaining measurement identities, values, and evidence qualifications. Example captions are provided in Appendix F.1, and the corresponding template pools are listed in Appendix F.2. C. Method Implementation Details C.1. OpenSLA Implementation Sensor Encoding. We process waveform and numerical inputs with separate sensor encoders. For the Clinical and OR domains, waveform signals are encoded using a pretrained DINO encoder adapted from [31], which remains frozen during SLA training. For CGM, we don’t use a large sensor encoder because the signal length is smaller. Instead, we use a trainable convolutional patchifier to convert glucose traces into temporal tokens. Numerical measurements are encoded with a trainable Transformer, where each observation is represented by combining its original embedding with measurement-identity and temporal embeddings. Structured Action Heads. All three heads operate on the backbone hidden state at the final input po- sition, denoted as h∈ℝ 푑 , where푑is the language hidden dimension.❶Necessity heads. Independent binary linear heads predict the presence of each action-necessity type and are trained with binary cross-entropyL nec .❷Category heads. Independent binary linear heads predict the action categories. They are trained with positive-class-weighted binary cross-entropyL cat .❸Fine-grained label heads. Within each action category, we obtain text embeddings for the candidate fine-grained actions and project them together with h into a shared embedding space. Candidate actions are scored by cosine similarity and trained with cross-entropy within each category, denoted asL label . 21 Sensor-Language-Action Models Together, the training objectiveL act used in Section 4 can be written as: L act = 휆 nec L nec + 휆 cat L cat + 휆 label L label ,(2) where휆 nec weights the action-necessity loss,휆 cat is for category prediction, and휆 label is the weight of fine-grained label loss. Hierarchical Memory. The Hierarchical Memory pathway uses two query stages. Local queries aggregate encoder features within each channel-time group, while temporal queries aggregate the resulting local memories across time within the same channel. Within each branch, local query parameters are shared across channel-time groups, and temporal query parameters are shared across channels. Channel embeddings are added at both query stages, with time embeddings additionally added to the local queries. The local, temporal, and global memory tokens are concatenated as the final memory representation. Each query stage consists of multi-head cross-attention followed by a residual feed-forward block. Waveform and numerical inputs use separate parameters, feature widths, and query budgets. For numerical inputs, observations are grouped by measurement identity and relative timestamp, and the per-measurement summary tokens are fed into the query hierarchy and are appended directly to the numerical memory. Multi-modal Fusion Decoder. Action tokens P represent the predicted action profile, including an action-necessity token and action-category tokens. Each token combines the corresponding action embedding with a projected prediction score. In the hierarchical configuration, waveform and numerical selectors retrieve relevant sensor tokens using the output hidden state H from the LLM backbone, together with the pooled action profile P. Waveform candidates are obtained from patch- level sensor features, while numerical candidates are obtained from observation tokens. Selected candidates are projected into the language hidden space and weighted by their retrieval scores to form sensor evidence S. The action-profile branch푓 푝 and sensor-evidence branch푓 푠 each contain cross-attention, a feed-forward block, and an output projection. Both operate on the original caption states H, attending to the action profile P and sensor evidence S. The refined caption states H ′ are then passed through the backbone’s original language-model head. Training uses ground-truth caption prefixes and predicted actions with weighted autoregressive cross-entropy loss, where signal-related caption tokens have been assigned a higher loss weight. C.2. Baseline and Ablation Details Supervised Baselines. We compare ConvTrans [5] and MMIM [7] withOpenSLAin the main action tables. We adapt MMIM to combine sensor recordings with language context using a trainable CNN and frozen BERT encoder [4]. Their representations are fused through an MLP and passed to task-specific action classifiers. We retain the mutual-information and contrastive objectives from the original implementation, and jointly optimize them with supervised action prediction, and do not use a caption-generation objective. TS Foundation Models. We linearly probe Chronos-2 [2], Moirai [37], and GluFormer [22] for action prediction. Sensor sequences are encoded into pretrained representations and pooled before being passed to linear prediction heads, while the pretrained encoders remain frozen. These sensor-only baselines do not receive language context. Zero-shot multi-modal LLMs. We perform zero-shot inference with state-of-the-art multi-modal LLMs, including GPT-5.6-Luna [25] and DeepSeek-V3.2 [3], through a CodeAct-style interface [34]. Instead of serializing long sensor recordings into the prompt, the models execute Python code to inspect pre-decision measurements and compute relevant summaries. Language context and tool outputs are then used for action prediction and evidence-caption generation. Action prompts provide 22 Sensor-Language-Action Models Table 15 | Training configurations for OpenSLA and adapted sensor-language baselines. Setting OpenSLAOpenTSLM [15]SensorLM [43] Optimization: OptimizerAdamWAdamWAdamW Peak learning rate5× 10 −5 5× 10 −5 5× 10 −5 Weight decay0.010.010.01 Warmup fraction0.100.100.10 Learning-rate decayLinearLinearLinear Gradient clipping norm0.50.50.5 LoRA rank / 훼 / dropout16 / 32 / 0.0516 / 32 / 0.05– Effective Batch Size: Clinical161664 Operating room (OR)161664 CGM1616256 the available action categories and the candidate fine-grained labels within each category. We directly evaluate the returned discrete predictions without model adaptation or probability-based scoring. TS Language Models.❶OpenTSLM. To accommodate the full sensor histories used byOpenSLA, we replace OpenTSLM’s native sensor encoder with our domain-specific sensor encoders. We retain the original SP and Flamingo fusion module and initialize their remaining components from their official LLama-based checkpoints [15]. Specifically, we use the model variant pretrained on ECG signals for clinical and operating room (OR) tasks, and use the one pretrained on general time-series for the CGM domain.❷SensorLM. We adapt SensorLM [43] to our datasets with minimal changes to its original formulation. It uses the same sensor encoders asOpenSLA, together with a unimodal text encoder and a multimodal decoder that cross-attends to sensor features. Ablation Variants. We construct ablations based on the configuration ofOpenSLA-B.❶Caption supervision and conditioning. The action-only variant removes caption generation and its training loss while retaining the action predictors. The no-decoder variant retains caption supervision but generates text directly from the language backbone without the residual fusion decoder. The no- action-conditioning variant retains the decoder but removes its predicted-action inputs.❷Language- backbone adaptation. We compare two alternatives to LoRA-based adaptation. The LLaVA-style variant [21] freezes the language backbone and maps sensor features into input tokens through trainable projectors, while the Flamingo-style variant [1] injects sensor features through gated cross- attention modules in the frozen backbone. Both retain the same sensor encoders, action heads, and caption supervision, with all task-specific modules trainable except the pretrained DINO encoder. C.3. Training Details Table 15 summarizes optimization settings and effective batch sizes including gradient accumulation. 23 Sensor-Language-Action Models D. Task Setup D.1. Action Prediction We evaluate action prediction at three levels corresponding to whether to act, which action category to select, and which fine-grained action to take. Predictions are compared with held-out action annotations derived from the source records. Clinical and operating room (OR) tasks support all three levels. CGM necessity indicates a bolus intervention, and its categories correspond to dose bins; there is no separate fine-grained label task. Text-generating baselines are evaluated by mapping their generated actions to the same target vocabulary. We report AUROC from continuous prediction scores and balanced accuracy (BAcc), defined as the mean of sensitivity and specificity. For the main action tables, thresholds maximize validation BAcc separately for each binary target, retaining the calibration scope of each evaluated model. Fine- grained labels are evaluated within ground-truth-positive category cohorts, averaging eligible labels within each category and then across categories. The±values summarize bootstrap uncertainty as half the width of 95% percentile intervals from 2,000 individual-level cluster resamples (surgical cases for VitalDB), displayed symmetrically around the point estimates. Checkpoints and evaluation thresholds remain fixed; these values reflect evaluation- cohort sampling uncertainty rather than variation across training runs. D.2. Caption-Based Evaluation State Understanding. We extract physiological quantities from generated captions and compare them with reference statistics computed from the observed history. Table 16 specifies the quantity, statistic, and readout for each task. We compute mean absolute error (MAE) on reference-eligible samples with a valid extracted prediction, using the reference unit. Action Explanation. We compare generated text with the action-evidence portion of the reference caption using BERTScore F1. We usedistilbert-base-uncased. The reported result is the mean per-sample F1, multiplied by 100. D.3. Unseen Action Classification To examine generalization to unseen fine-grained actions within a known category, we construct a balanced binary classification set from MIMIC-I. The target is Imipenem/Cilastatin, a single antibiotic label absent from the supervised action training targets, while the antibiotic category is seen during training. Zero-Shot Action Scoring.OpenSLArepresents candidate actions through textual descriptors rather than label-specific classifier weights. At inference, we encode the unseen descriptor,antibiotic: Imipenem/Cilastatin , by averaging its language-backbone token embeddings and applying the trained descriptor projection. Its cosine similarity with the projected multi-modal decision representa- tion provides the fine-grained action score. All model parameters remain fixed; no additional training on the unseen label is required. D.4. External-Cohort Evaluation We evaluate the Clinical models on waveform-linked MIMIC-IV samples without further adaptation to this cohort. Each sample provides pre-decision sensor history and language context, with actions mapped to the Clinical target vocabulary. We report necessity and category AUROC and BAcc; discrete 24 Sensor-Language-Action Models Table 16|Caption evaluation tasks and scoring rules. Definitions for the sensor-state and action- evidence tasks reported in the main results. Quantity / TaskReference StatisticPrediction ReadoutMetric Clinical: 30-minute sensor history Cardiac indexWindow median (L/min/m 2 ) Extract an explicit cardiac-index median. MAE↓ ECG-I rateECG-derived rate proxy (beats/min) Extract the rate attributed to ECG lead I. MAE↓ RESP rate Respiration-waveform rate proxy (breaths/min) Extract a rate attributed to the RESP waveform. MAE↓ Operating room: 5-minute sensor history; Respiratory rateWindow median (breaths/min) Extract an explicit median of measured R. MAE↓ TemperatureWindow median ( ◦ C) Extract an explicit temperature median.MAE↓ End-tidal CO 2 Window median (mmHg) Extract an explicit ETCO 2 median.MAE↓ CGM: 2-hour pre-decision history Basal rateMean recorded basal-insulin rate (U/h) Extract a mean/average basal rate.MAE↓ Carbohydrate amount Amount of the latest recorded carbohydrate event (g) Extract an amount paired with a pre-decision recency in the preceding 120 minutes, and choose the latest unambiguous event. MAE↓ Action-evidence text: all six development datasets Action-evidence agreement Ground-truth action-evidence text Compare generated captions, against the same reference evidence. BERTScore F1↑ [42] action predictions from text-generating baselines contribute BAcc but not AUROC. This experiment measures transfer to another MIMIC release and cohort. The evaluation cohort contains 1,000 waveform-qualified windows from 87 individuals, with 500 action-positive and 500 action-negative samples. Both classes use a common five-minute candidate grid and a five-minute target interval. Sampling approximately matches action prevalence to the MIMIC-I reference cohort while balancing pre-decision availability characteristics. E. Additional Results We provide additional results and analysis including sensor-language adaptation, caption supervision, generalization, and the information retained in learned representations. 25 Sensor-Language-Action Models E.1. Sensor–Language Adaptation We compare direct LoRA adaptation with two alternatives: a LLaVA-style variant that freezes the language backbone and learns sensor projectors, and a Flamingo-style variant that injects trainable cross-attention into the frozen backbone. Both retain the same action heads and caption supervision. As shown in Table 17, direct LoRA achieves the highest necessity and category AUROC on both MOVER and VitalDB datasets. Table 17|Sensor–language adaptation. Necessity and category AUROC (%) on operating room datasets. Method MOVERVitalDB NecessityCategoryNecessityCategory AUROC ↑ AUROC ↑ AUROC ↑ AUROC ↑ LLaVA-style [21]51.566.454.460.0 Flamingo-style [1]82.976.168.982.1 OpenSLA83.480.374.885.9 E.2. Unseen Action Generalization We further evaluate zero-shot prediction of Imipenem in the clinical domain, a fine-grained antibiotic label excluded from supervised action training.OpenSLAscores this unseen label using its textual representation without additional training. As shown in Table 18,OpenSLA-Hachieves the highest F1 scores and balanced accuracies. Table 18 | Unseen-action prediction results. Method Imipenem F1 ↑ BAcc ↑ OpenTSLM [15]17.454.8 GPT-5.6-Luna [25]42.961.9 DeepSeek-V3.2 [3]34.554.8 OpenSLA70.071.4 E.3. Future-State Readout We test whether current-state representations retain information about subsequent glucose dynamics. On the MetaboNet dataset, we construct paired current and future two-hour signals and extract the glucose mean, minimum, maximum, and slope over the following two hours. We then fit ridge regression on the frozen representation from each model and evaluate it with five-fold cross-validation. It is fitted using nested cross-validation, with five outer folds and three inner folds for regularization selection. Chronos-2 is evaluated separately using its native zero-shot forecasts. Table 19 reports glucose-summary MAE in mg/dL and slope MAE in mg/dL/min. TheOpenSLA variants have lower error on mean, minimum, and maximum glucose than the listed baselines. Slope errors are close:OpenSLA, OpenTSLM-SP, and SensorLM all round to 0.39. These results concern a trained readout of frozen representations, not zero-shot generation of future glucose values. 26 Sensor-Language-Action Models Table 19|Future-state readout on MetaboNet. MAE for the next two hours: mean, minimum, maximum, and slope. Method MetaboNet Mean ↓ Min ↓ Max ↓ Slope ↓ OpenTSLM [15]26.524.230.60.39 SensorLM [43]26.924.630.50.39 Chronos-2 [2]29.729.332.80.48 GluFormer [22]33.230.038.70.41 OpenSLA-B24.022.328.00.39 OpenSLA-H24.022.527.70.39 E.4. Contribution of Sensor Inputs We compareOpenSLAwith a context-only variant that removes waveform and numerical inputs while retaining the same pre-decision text context, hierarchical action heads, supervision text captions, and action-conditioned fusion decoder. As shown in Table 20, including sensor inputs improves necessity and category AUROC on both datasets. Table 20|Contribution of sensor inputs. We report the results (AUROC (%)) of action necessity and category prediction. Method MOVERVitalDB NecessityCategoryNecessityCategory AUROC ↑ AUROC ↑ AUROC ↑ AUROC ↑ Context-only79.875.764.983.4 OpenSLA83.480.374.885.9 Generated Caption Counterfactual Generated Caption T = 0 zoomed in T = 30 min + Patient Profile Observed Signal Higher ABP Signal For the plethysmography waveform, the dominant waveform pattern is plethpulsatile signal present (rate proxy 76.1/min, IQR 1.00) The pre-decision state is most consistent with possible cardiac or rhythm concern The dominant waveform pattern is plethpulsatile signal present (rate proxy 82.1/min, IQR 1.00). Across directly observed context and signals, the dominant state is possible cardiacor rhythm concern, with ecghigh ratepattern waveform finding; Figure 9|Caption responses to sensor substitution. Patient context is held fixed while the supplied sensor recording changes. E.5. Controlled Sensor Substitution We hold the patient context fixed and replace the observed sensor recording with a higher-ABP example. As shown in Fig. 9, this sensor substitution changes the generated caption: the Pleth rate 27 Sensor-Language-Action Models proxy increases from 76.1 to 82.1/min, and the state summary additionally identifies a high-rate ECG pattern, while the overall cardiac/rhythm concern remains unchanged. This controlled comparison demonstrates that the generated caption responds to changes in the sensor input, while remaining the patient context part unchanged. E.6. Hierarchical Memory Efficiency Table 21 comparesOpenSLA-BandOpenSLA-Hon paired validation windows from each domain, with the values to the right of each arrow corresponding toOpenSLA-H. We use batch size 1 on a GH200 GPU and measure forward latency for the sensor encoders, backbone, and necessity/category heads on the action prediction task; compression and speedup are computed as ratios of domain-level means. Hierarchical Memory reduces both token counts and forward latency on average in the Clinical and OR domains. For CGM, sensor sequences are already short, so the additional memory-processing overhead largely offsets the savings from compression, resulting in little runtime benefit. Table 21 | Hierarchical Memory efficiency. DomainSensor Tokens ↓ Compression ↑ Speedup ↑ Clinical5801.0→ 400.414.49×3.4× OR3915.8→ 838.04.67×1.7× CGM33.0→ 21.01.57×1× F. Additional Examples F.1. Multi-faceted Caption Examples Examples 3 and 4 are re-rendered with consistent waveform-evidence filtering; the remaining exam- ples reproduce saved reference captions. Examples Captions Generated by Our Data Pipeline ======================================== Example 1 ======================================= OBSERVATIONAL: Numeric: 1-minute heart-rate variability was evidence of hrv abnormal (1min_HRV median 10, range 5-32, n=19) and persistently outside the expected range. The heart rate series indicates tachycardia and is persistently outside the expected range (HR median 110, range 107-120, n=19). Waveform: The Pleth trace did not meet the quality threshold for morphology or rate-proxy statements. Because Resp produced incompatible rhythm or periodicity proxies, it was excluded from the waveform conclusion. Lead I waveform evidence includes tachycardic ECG waveform proxy (rate proxy 108.8/min, IQR 0.20). RELATIONAL: Temporal: Temporal pattern: short-window HRV is abnormal with little recent change across the current decision window; evidence of hrv abnormal. Near the decision point, heart-rate channel shows elevated heart rate and is abnormal but not changing much. Cross-signal: elevated cardiac rate across numeric and ECG evidence is present across the corresponding signal channels. GLOBAL: Context: Before the decision point, the contextual record contains reported shortness of breath at presentation. State summary: Across directly observed context and signals, the dominant state is possible cardiac or rhythm concern finding, with hrv abnormal context; rapid cardiac rate; ecg high rate pattern waveform finding. 28 Sensor-Language-Action Models ACTION: elevated heart-rate pattern (HR median 110, range 107-120, n=19); trend: abnormal but not changing much; elevated cardiac rate across numeric and ECG evidence; cardiac waveform channel showed high-rate Lead I periodicity (rate proxy 108.8/min, IQR 0.20) is relevant to circulatory support, although the available evidence is indirect or incomplete. ======================================== Example 2 ======================================= OBSERVATIONAL: Numeric: Across the current observation interval, perfusion signal shows poor perfusion signal; Perf median 1, range 1-1, n=30; persistently outside the expected range. For respiratory rate, the dominant finding is elevated respiratory rate; R median 24, range 17-29, n=30. Waveform: The Pleth trace did not meet the quality threshold for morphology or rate-proxy statements. The rate or periodicity proxies from Resp were internally inconsistent, so that channel was not used for waveform interpretation. ECG channel summary: ecg rate proxy normal; rate proxy 81.2/min, IQR 0.10. RELATIONAL: Temporal: Temporal pattern: Perf is sustained without a clear directional shift across the half-hour signal window; poor perfusion signal. Over the monitored interval, R is abnormal but not changing much with elevated respiratory rate. GLOBAL: Context: Available contextual evidence includes reported neurologic at presentation. State summary: The pre-decision state is most consistent with possible hemodynamic instability signal; key evidence is reduced peripheral perfusion proxy; weak Pleth waveform. ACTION: A possible reason for the infectious diagnostic testing workup is the pre-T0 finding tachypnea (R median 24, range 17-29, n=30); trend: persistently outside the expected range; perfusion concern (Perf median 1, range 1-1, n=30); trend: persistently outside the expected range; peripheral pulse waveform showed weak pulse waveform amplitude (IQR 0.00); no direct note statement confirms it. ======================================== Example 3 ======================================= OBSERVATIONAL: Numeric: The heart rate series indicates tachycardia and is abnormal with little recent change (HR median 107, range 102-111, n=17). Waveform: Because Pleth was only partly usable in this window, no morphology or rate inference was drawn from it. The available Resp trace was present, but its proxy features disagreed, so no single morphology or rate interpretation was reported. ECG Lead I: nonuniform cardiac waveform proxy; rate proxy 99.9/min, IQR 0.35. RELATIONAL: Temporal: Over the monitored interval, heart-rate channel is persistently outside the expected range with elevated heart-rate pattern. GLOBAL: Context: Presentation context includes chief complaint of hypertension or hypotension. State summary: The pre-decision state is most consistent with possible cardiac or rhythm concern context; key evidence is elevated heart-rate pattern; irregular cardiac waveform pattern. ACTION: The available state shows elevated heart rate (HR median 107, range 102-111, n=17); trend: outside the expected range without a strong trend; because this recorded treatment can represent context-specific care, these facts are described as background for cardiorenal volume management rather than a definitive rationale. 29 Sensor-Language-Action Models ======================================== Example 4 ======================================= OBSERVATIONAL: Numeric: For heart-rate channel, the dominant finding is elevated heart-rate pattern; HR median 100, range 95-108, n=30. Waveform: Proxy interpretation was deferred for Pleth because the available waveform was quality-constrained. Waveform availability for Resp was adequate, yet the derived proxies were discordant and therefore not interpreted as a single finding. The cardiac waveform channel shows variable ECG waveform periodicity; rate proxy 95.2/min, IQR 0.40. RELATIONAL: Temporal: Temporal pattern: heart rate is abnormal but not changing much across the current observation interval; heart rate above expected range. GLOBAL: Context: Pre-decision context includes arrival complaint includes psych or substance; reported medication or refill at presentation. State summary: The pre-decision state is most consistent with possible cardiac or rhythm concern signal; key evidence is tachycardia; irregular Lead I structure. ACTION: Because no direct note reason was available, heart rate above expected range (HR median 100, range 95-108, n=30); trend: sustained without a clear directional shift is retained as a possible indication for evaluation of altered mental status or neurologic concern. ======================================== Example 5 ======================================= OBSERVATIONAL: Numeric: The heart rate has limited charting that supports elevated heart-rate pattern (HR median 101, range 101-101, n=1); this is based on one pre-decision HR reading, without a reliable trajectory. rapid breathing pattern appears in one pre-decision R reading (R median 23, range 23-23, n=1); no temporal trend is assigned. Waveform: ECG channel summary: ECG high-rate proxy; rate proxy 100.0/min, IQR 1.00. For Plethysmography waveform, the dominant waveform pattern is uneven Pleth pulsatility (rate proxy 91.3/min, IQR 1.00). GLOBAL: Context: Before the decision point, the contextual record contains presentation mentions bleeding or gi bleed. State summary: Across directly observed context and signals, the dominant state is possible infection or sepsis concern signal, with rapid cardiac rate; R above expected range. ACTION: A secondary signal for symptom-control intervention is elevated heart-rate pattern (HR median 101, range 101-101, n=1; one pre-decision HR reading); no stronger domain-specific fact was selected. ======================================== Example 6 ======================================= OBSERVATIONAL: Numeric: Basal delivery was zero throughout the prior two hours. Waveform: Glucose during the preceding two hours stayed within 71-100 mg/dL, median 88 mg/dL, inside the 70-180 mg/dL range. In the two-hour window, the fitted slope of sensor glucose was +6 mg/dL per hour; glucose was flat, within 10 mg/dL per hour. The last reading before the decision was 94 mg/dL, 5 minutes before it. RELATIONAL: Temporal: Over the two-hour window before the decision, no bolus, carbohydrate entry, small delivery run or basal change was logged. GLOBAL: Context: Patient context: 48 yr old, a male patient, 104 kg body weight, height of 170 cm, body mass index 36.0 kg/m2, duration of diabetes 38 years, automated insulin 30 Sensor-Language-Action Models delivery. State summary: Pre-decision state: readings in 70-180 mg/dL throughout, changing under 10 mg/dL per hour. ACTION: Findings on the side of no bolus: pump basal of zero entering the decision. ======================================== Example 7 ======================================= OBSERVATIONAL: Numeric: No basal delivery was recorded in the prior two hours. Waveform: Glucose over the two-hour observation window had a median of 97 mg/dL, in the target range of 70-180 mg/dL, and went from a low of 78 to a high of 113 mg/dL. Glucose over the two-hour window before the decision: decreasing, -18 mg/dL per hour. Glucose at the last reading, 5 minutes before the decision, was 78 mg/dL. RELATIONAL: Temporal: In the two-hour lookback: no bolus, carbohydrate entry, small delivery run or basal change. GLOBAL: Context: Patient profile: 28 yr old, a male patient, a weight of 58 kg, 175 cm tall, BMI equal to 18.8 kg/m2, 11 years since diagnosis, automated insulin delivery (AID). State summary: Summary of the state before the decision: none of the defined states applied; evidence was limited. ACTION: Against a bolus: no pump basal in the slot before the decision. ======================================== Example 8 ======================================= OBSERVATIONAL: Numeric: Basal insulin stayed at zero for all of the last two hours. Insulin on board, slot before the decision: 0.13 U. Waveform: Sensor glucose in the preceding two hours: median 114 mg/dL, in the 70-180 mg/dL target range, lowest 101 and highest 122 mg/dL. Glucose over the two hours before the decision: stable (under 10 mg/dL per hour either way), -8 mg/dL per hour. The last CGM value was 101 mg/dL, 5 minutes before the decision. RELATIONAL: Temporal: No timed events (bolus, carbohydrate, small delivery run, basal change) in the last two hours. GLOBAL: Context: Known patient attributes: aged 43, male, a closed-loop system. State summary: State before the decision: a 70-180 mg/dL trace with a slope below 10 mg/dL per hour. ACTION: Findings opposing an insulin bolus: zero basal on the pump just before the decision. ======================================== Example 9 ======================================= OBSERVATIONAL: Numeric: Basal insulin stayed at zero for all of the two-hour window. Waveform: CGM readings in the last two hours spanned 88-136 mg/dL, with a median of 117 mg/dL, neither below 70 nor above 180 mg/dL. Over the two-hour period before the decision, glucose was dropping, at -20 mg/dL per hour. The reading closest to the decision was 88 mg/dL, 5 minutes earlier. RELATIONAL: Temporal: The record lists no bolus, carbohydrate entry, small delivery run or basal change for the two-hour period before the decision. 31 Sensor-Language-Action Models GLOBAL: Context: Patient record: 35 years old, of female sex, body weight 54 kg, 157 cm tall, BMI at 21.9 kg/m2, living with diabetes 30 years, automated insulin delivery. State summary: Overall state before the decision: no selected glycaemic state; direct evidence was limited. ACTION: Pre-decision evidence against an insulin bolus: pump basal at zero just before the decision. ======================================= Example 10 ======================================= OBSERVATIONAL: Numeric: A constant basal rate of 0.90 U/h over the two hours up to the decision delivered 1.80 U. Insulin on board immediately before the decision was 0.01 U. Waveform: Median sensor glucose over the two-hour window was 150 mg/dL, in the target range of 70-180 mg/dL; readings 142-153 mg/dL. For the two-hour period before the decision, glucose had a fitted slope of -5 mg/dL per hour and was flat (less than 10 mg/dL per hour). Latest glucose: 142 mg/dL at 5 minutes before the decision. RELATIONAL: Temporal: No timed events (bolus, carbohydrate, small delivery run, basal change) in the two hours prior to the decision. GLOBAL: Context: Patient details on record: aged 66, recorded as male, weighing 108 kg, a height of 185 cm, computed BMI 31.4 kg/m2, 26 years of diabetes, an insulin pump with CGM. State summary: Overall glycaemic state: glucose kept inside 70-180 mg/dL, slope under 10 mg/dL per hour. ACTION: No finding for or against an insulin bolus was selected from the observed window. F.2. Template pool OBSERVATIONAL NUMERIC_CAPTION_TEMPLATES = [ "modality_phrase was numeric_finding (stat_phrase) and trend_phrase.", "Across window_phrase, modality_phrase shows numeric_finding; stat_phrase; trend_phrase.", "The modality_phrase series indicates numeric_finding and is trend_phrase (stat_phrase).", "Recorded modality_phrase values support numeric_finding; stat_phrase; trend_phrase.", "For modality_phrase, the dominant finding is numeric_finding; stat_phrase.", "numeric_finding is present in the modality_phrase series and is trend_phrase.", ] NUMERIC_OBSERVATION_ONLY_TEMPLATES = [ "Observed modality_phrase during window_phrase (stat_phrase).", "The current modality_phrase summary is stat_phrase.", "Recorded modality_phrase values in window_phrase are summarized as stat_phrase.", ] NUMERIC_SPARSE_TEMPLATES = [ "modality_phrase has limited charting that supports numeric_finding (stat_phrase); this is based on sparse_observation_phrase, without a reliable trajectory.", "numeric_finding appears in sparse_observation_phrase (stat_phrase); no temporal trend is assigned.", "Evidence from sparse_observation_phrase is consistent with numeric_finding (stat_phrase); the values are retained but temporal detail is withheld because observations are sparse.", "The available values for modality_phrase indicate numeric_finding (stat_phrase), although the evidence consists of only sparse_observation_phrase.", ] LAB_RESULT_CAPTION_TEMPLATES = [ "modality_phrase: numeric_finding; stat_phrase.", "numeric_finding appears in recent modality_phrase results (stat_phrase).", "Recent lab evidence from modality_phrase shows numeric_finding; stat_phrase.", 32 Sensor-Language-Action Models "The resulted modality_phrase panel contributes numeric_finding: stat_phrase.", "stat_phrase supports numeric_finding in the modality_phrase category.", "Lab context before the decision point includes numeric_finding from modality_phrase; stat_phrase.", "modality_phrase result summary: numeric_finding; stat_phrase.", "A recent modality_phrase finding is numeric_finding, recorded as stat_phrase.", ] WAVEFORM_GENERIC_TEMPLATES = [ "modality_phrase: pattern_phrase; stat_phrase.", "The modality_phrase signal shows pattern_phrase (stat_phrase).", "modality_phrase provides waveform evidence of pattern_phrase; stat_phrase.", "For modality_phrase, the dominant waveform pattern is pattern_phrase (stat_phrase).", ] WAVEFORM_ECG_I_TEMPLATES = [ "ECG Lead I: pattern_phrase; stat_phrase.", "Lead I waveform evidence includes pattern_phrase (stat_phrase).", "The cardiac waveform channel shows pattern_phrase; stat_phrase.", "ECG channel summary: pattern_phrase; stat_phrase.", ] WAVEFORM_PLETH_TEMPLATES = [ "Pleth waveform: pattern_phrase; stat_phrase.", "The pulse-ox waveform shows pattern_phrase (stat_phrase).", "Peripheral pulse waveform summary: pattern_phrase; stat_phrase.", "Pleth channel evidence is summarized as pattern_phrase, with stat_phrase.", ] WAVEFORM_RESP_TEMPLATES = [ "Resp waveform: pattern_phrase; stat_phrase.", "The respiration waveform shows pattern_phrase (stat_phrase).", "Breathing waveform evidence includes pattern_phrase; stat_phrase.", "Respiration waveform summary: pattern_phrase; stat_phrase.", ] CGM_WAVEFORM_CAPTION_TEMPLATES = "glucose_summary": [ "Median sensor glucose over period was median mg/dL, band_phrase; readings lo-hi mg/dL.", "Over period, median glucose was median mg/dL, band_phrase, with readings from lo to hi mg/dL.", ], "glucose_trend": [ "Glucose was direction over period, with a fitted slope of slope mg/dL per hour.", "The fitted glucose slope over period was slope mg/dL per hour; glucose was direction.", ], "glucose_last": [ "The last reading before the decision was last mg/dL, m minutes before it.", "The most recent glucose, m minutes before the decision, was last mg/dL.", ], "time_in_ranges": [ "Time in ranges over period: p_lt54% below 54, p_lt70% below 70, p_70_180% at 70-180, p_gt180% above 180 and p_gt250% above 250 mg/dL.", "Of valid readings in period, p_70_180% were at 70-180 mg/dL, p_lt70% below 70 (p_lt54% below 54) and p_gt180% above 180 (p_gt250% above 250).", ], "variability": [ "The p10-p90 glucose spread over period was spread mg/dL, at least 60 mg/dL.", "Glucose variability in period: p10-p90 spread spread mg/dL, 60 mg/dL or more.", ], "no_glucose": [ "No usable sensor glucose was available in period.", "Sensor glucose was unavailable for period.", ], CGM_NUMERIC_CAPTION_TEMPLATES = "basal_zero_throughout": [ "No basal insulin was delivered in period.", 33 Sensor-Language-Action Models "Basal delivery was zero throughout period.", ], "basal_steady": [ "Basal insulin ran at a constant rate U/h over period, total U in total.", "Over period, basal delivery held at rate U/h, totalling total U.", ], "basal_varying": [ "Basal insulin over period averaged mean U/h, ranging min-max U/h, total U in total.", "Over period, basal delivery ranged min-max U/h (mean mean U/h), totalling total U.", ], "iob": [ "Insulin on board before the decision was iob U.", "Just before the decision, insulin on board was iob U.", ], "cob": [ "Carbohydrate on board before the decision was cob g.", "Just before the decision, carbohydrate on board was cob g.", ], RELATIONAL TEMPORAL_CAPTION_TEMPLATES = [ "Temporal pattern: modality_phrase is trend_phrase across window_phrase; numeric_finding.", "Over the monitored interval, modality_phrase is trend_phrase with numeric_finding.", "The observed trajectory for modality_phrase is trend_phrase; stat_phrase.", "Near the decision point, modality_phrase shows numeric_finding and is trend_phrase.", ] CROSS_SIGNAL_CAPTION_TEMPLATES = [ "numeric_finding and waveform_finding are directionally consistent.", "numeric_finding is connector_evidence waveform_finding.", "waveform_finding provides waveform support for numeric_finding.", "numeric_finding is not clearly matched by waveform evidence because uncertainty_phrase.", "cross_signal_finding: numeric_finding connector_additive waveform_finding.", "Together, numeric_finding and waveform_finding support clinical_state.", "cross_signal_finding is present, based only on available numeric and waveform evidence.", "waveform_finding and numeric_finding point to clinical_state, with proxy-level certainty.", "numeric_finding contrasts with waveform_finding; signal interpretation is uncertainty_phrase.", "Cross-signal evidence: cross_signal_finding; supporting_findings.", ] CROSS_SIGNAL_EXPLICIT_FACT_TEMPLATES = [ "cross_signal_finding is supported by the paired numeric and waveform observations.", "Matched numeric and waveform channels show cross_signal_finding.", "cross_signal_finding is present across the corresponding signal channels.", ] CGM_TEMPORAL_CAPTION_TEMPLATES = "bolus_event": [ "A dose U bolus was recorded m minutes before the decision.", "Bolus: dose U, m minutes before the decision.", ], "carb_event": [ "grams g of carbohydrate was recorded m minutes before the decision.", "Carbohydrate: grams g, m minutes before the decision.", ], "small_deliveries_one": [ "One run of deliveries under 0.25 U per row (total U) was not counted as a bolus.", "A run of sub-0.25 U deliveries totalling total U was not counted as a bolus.", ], "small_deliveries_many": [ "count runs of deliveries under 0.25 U per row (total U) were not counted as boluses.", "count runs of sub-0.25 U deliveries totalling total U were not counted as boluses.", ], "basal_zero_ended": [ 34 Sensor-Language-Action Models "Basal delivery was zero for dur minutes, ending end_m minutes before the decision.", "A dur-minute zero-basal stretch ended end_m minutes before the decision.", ], "basal_zero_ongoing": [ "Basal delivery had been zero for dur minutes and still was in the slot before the decision.", "A dur-minute zero-basal stretch ran up to the slot before the decision.", ], "basal_changes_one": [ "The basal rate changed once, m minutes before the decision.", "One basal rate change, m minutes before the decision.", ], "basal_changes_many": [ "The basal rate changed n times, last m minutes before the decision.", "n basal rate changes, the latest m minutes before the decision.", ], "temporal_empty": [ "No bolus, carbohydrate entry, small delivery run or basal change was recorded in period.", "period_capitalized has no bolus, carbohydrate entry, small delivery run or basal change.", ], GLOBAL CONTEXT_CAPTION_TEMPLATES = [ "Presentation context includes supporting_findings.", "Available contextual evidence includes supporting_findings.", "The available clinical context mentions supporting_findings.", "Before the decision point, the contextual record contains supporting_findings.", "Contextual cues before the decision point include supporting_findings.", "Relevant presentation and workflow context includes supporting_findings.", "Earlier clinical context includes supporting_findings.", "Pre-decision context includes supporting_findings.", ] GLOBAL_CAPTION_TEMPLATES = [ "Overall observed state before the decision: clinical_state, supported by supporting_findings.", "The pre-decision state is most consistent with clinical_state; key evidence is supporting_findings.", "Across directly observed context and signals, the dominant state is clinical_state, with supporting_findings.", ] CGM_CONTEXT_CAPTION_TEMPLATES = "context": [ "Patient context: items.", "Patient: items.", ], "context_empty": [ "No patient context was recorded.", "Patient context is not recorded.", ], CGM_GLOBAL_CAPTION_TEMPLATES = "global_states": [ "Overall state: states.", "Overall pre-decision state: states.", ], "global_none": [ "Overall state: no selected glycaemic state; direct pre-decision evidence was limited.", "Overall state: no glycaemic state was selected; direct pre-decision evidence was limited.", ], 35 Sensor-Language-Action Models ACTION TREATMENT_ACTION_EVIDENCE_TEMPLATES = "supported": [ "Observed findings relevant to action_evidence_domain_phrase include action_evidence_clause.", "The available pre-decision evidence for action_evidence_domain_phrase is action_evidence_clause.", ], "secondary_only_evidence": [ "The observed window includes action_evidence_clause; this is nonspecific supporting evidence for action_evidence_domain_phrase rather than a complete indication.", "action_evidence_clause is relevant to action_evidence_domain_phrase, although the available evidence is indirect or incomplete.", ], "fallback_visible_evidence": [ "Visible pre-decision findings include action_evidence_clause; together they provide indirect or mixed context for action_evidence_domain_phrase, not a complete indication.", "The observed state contains action_evidence_clause; these facts are relevant background for action_evidence_domain_phrase, although none is a direct domain-specific match.", ], "label_review_context_only": [ "Visible pre-decision findings include action_evidence_clause; they are retained as clinical context, not as a direct indication for action_evidence_domain_phrase.", "The available state shows action_evidence_clause; because this recorded treatment can represent context-specific care, these facts are described as background for action_evidence_domain_phrase rather than a definitive rationale.", ], "label_review_no_visible_context": [ "The recorded treatment belongs to a context-dependent category; no reliable pre-decision fact is stated as a direct indication.", "This action category can reflect context-specific care, and the visible window does not support a label-specific rationale statement.", ], "limited_observable_evidence": [ "The available pre-decision record contains limited observable evidence specific to this action domain.", "No compatible state fact was selected for an action-specific evidence statement in this window.", ], "no_immediate_treatment_evidence": [ "The observed pre-decision state does not assert a specific immediate treatment-level action from the available evidence.", "No immediate treatment domain is selected from the visible pre-decision context and signals.", ], DIAGNOSTIC_ACTION_EVIDENCE_TEMPLATES = "direct_note_indication": [ "The pre-decision note documents this diagnostic indication: action_evidence_clause.", "A documented indication before T0 was: action_evidence_clause.", ], "inferred_indication": [ "The action_evidence_domain_phrase workup was likely prompted by action_evidence_clause; this is an inferred indication from pre-T0 evidence, not an explicitly documented reason.", "Pre-T0 findings suggest a possible indication for action_evidence_domain_phrase: action_evidence_clause; the note did not state this rationale directly.", ], "inferred_general_indication": [ "The action_evidence_domain_phrase workup was likely part of broader diagnostic assessment, but no specific pre-T0 indication was available.", "A diagnostic indication for action_evidence_domain_phrase is plausible from the clinical setting, although the available record does not identify a specific reason.", ], "supported": [ "Diagnostic-workup evidence for action_evidence_domain_phrase includes action_evidence_clause.", "The pre-decision record supports action_evidence_domain_phrase through action_evidence_clause.", ], 36 Sensor-Language-Action Models "secondary_only_evidence": [ "The observed window includes action_evidence_clause; this is indirect supporting context for action_evidence_domain_phrase.", "Available context such as action_evidence_clause is relevant to action_evidence_domain_phrase, but is not a complete indication.", ], "fallback_visible_evidence": [ "Visible pre-decision findings include action_evidence_clause; they provide broad context for action_evidence_domain_phrase, not a definitive indication.", "The available state shows action_evidence_clause; its relationship to action_evidence_domain_phrase remains indirect or mixed.", ], "label_review_context_only": [ "Visible pre-decision findings include action_evidence_clause; they are clinical context for action_evidence_domain_phrase, not a label-specific reason.", "The available state shows action_evidence_clause; the diagnostic action is retained without inferring an exact indication.", ], "limited_observable_evidence": [ "A diagnostic action was recorded, but no specific pre-decision indication was identifiable from the available context and signals.", "The visible window provides limited observable evidence for this diagnostic workup.", ], "no_immediate_treatment_evidence": [ "No immediate treatment action is asserted; the observed decision context is diagnostic or otherwise non-treatment focused.", "The visible state does not establish an immediate treatment response, while diagnostic context may still be present.", ], OR_ACTION_EVIDENCE_TEMPLATES = "supported": [ "Observed findings relevant to action_evidence_domain_phrase include action_evidence_clause.", "The available pre-decision evidence for action_evidence_domain_phrase is action_evidence_clause.", "For the current decision, visible evidence linked to action_evidence_domain_phrase includes action_evidence_clause.", "Evidence aligned with action_evidence_domain_phrase includes action_evidence_clause.", "The available state includes action_evidence_clause; these observations are relevant to action_evidence_domain_phrase.", "Findings that bear on action_evidence_domain_phrase are action_evidence_clause.", ], "no_immediate_treatment_evidence": [ "No immediate action domain is selected from the visible pre-decision context and signals.", "The caption records the current state without assigning an immediate action domain.", "Current findings are described without linking them to a recorded immediate action.", "No action-specific rationale is asserted from the observed pre-decision state.", "The available pre-T0 state does not select a specific immediate response.", ], CGM_ACTION_EVIDENCE_TEMPLATES = "evidence_for": [ "Findings for an insulin bolus: findings.", "Evidence for a bolus: findings.", ], "evidence_against": [ "Findings against an insulin bolus: findings.", "Evidence against a bolus: findings.", ], "evidence_none": [ "No finding for or against an insulin bolus was selected from the observed window.", "No finding for or against a bolus was selected.", ], 37