Paper deep dive
Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine
Chenran Weng, Joo Seung Lee, Malini Mahendra, Anil Aswani
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/17/2026, 5:14:30 AM
Summary
This paper addresses the challenge of temporal redundancy in longitudinal clinical notes for offline reinforcement learning (RL) in mechanical ventilation. The authors propose a redundancy-aware multimodal state representation framework that decomposes clinical notes into history and innovation components using two methods: embedding-space singular value decomposition (SVD) and sentence-level diff operations. Evaluated on MIMIC-III data, these methods significantly outperform structured-only and raw-note baselines across multiple off-policy evaluation metrics, demonstrating that isolating new clinical information improves RL policy performance.
Entities (10)
Relation Signals (7)
MIMIC-III → usedin → Evaluation
confidence 99% · Using real-world ICU data... MIMIC-III mechanical ventilation trajectories
Redundancy-Aware Framework → improves → RL Performance
confidence 96% · state representations constructed by stripping temporal note redundancy significantly outperform... baselines... directly improves RL performance
Temporal Redundancy → degrades → State Representation Quality
confidence 95% · temporal redundancy... dilutes time-local updates and degrades state representation quality.
Sentence-level Diff → usedfor → Sentence-Level Decomposition
confidence 94% · an interpretable sentence-level diff operation that filters out previously documented sentences
Singular Value Decomposition → usedfor → Embedding-Space Decomposition
confidence 94% · an embedding-space decomposition using singular value decomposition on local history subspaces
Conservative Q-Learning → usedfor → Offline Policy Learning
confidence 93% · Our main offline RL policy is trained using Conservative Q-Learning (CQL)
Clinical Notes → complements → Structured EHR Data
confidence 90% · Clinical notes provide a natural complementary source of such context
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient's condition evolves. While reinforcement learning (RL) offers a promising framework for optimizing these sequential decisions, standard approaches rely primarily on structured electronic health record (EHR) data, missing crucial clinical context recorded in free-text notes. Integrating longitudinal clinical notes into RL state spaces is challenging because notes are heavily inflated by temporal redundancy, such as copy-forward text, templating, and repetitive documentation, which dilutes time-local updates and degrades state representation quality. To address this, we propose a redundancy-aware multimodal state representation framework that explicitly removes duplicated note text over time before policy learning. We evaluate two computationally efficient temporal decomposition strategies for removing duplicated note text: (1) an embedding-space decomposition using singular value decomposition on local history subspaces, and (2) an interpretable sentence-level diff operation that filters out previously documented sentences before text encoding. Using real-world ICU data, we demonstrate that state representations constructed by stripping temporal note redundancy significantly outperform both structured-only and raw-note baselines across multiple off-policy evaluation methods (Model-Based Rollouts, Fitted Q-Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust Evaluation). Our findings show that explicitly isolating new clinical information from repeated note text yields higher-quality state representations and directly improves RL performance for clinical decision support.
Tags
Links
- Source: https://arxiv.org/abs/2608.14157v1
- Canonical: https://arxiv.org/abs/2608.14157v1
Trouble viewing inline? Open PDF directly →
Full Text
47,322 characters extracted from source content.
Expand or collapse full text
Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine Chenran Weng Joo Seung Lee Malini Mahendra Anil Aswani Abstract Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient’s condition evolves. While reinforcement learning (RL) offers a promising framework for optimizing these sequential decisions, standard approaches rely primarily on structured electronic health record (EHR) data, missing crucial clinical context recorded in free-text notes. Integrating longitudinal clinical notes into RL state spaces is challenging because notes are heavily inflated by temporal redundancy, such as copy-forward text, templating, and repetitive documentation, which dilutes time-local updates and degrades state representation quality. To address this, we propose a redundancy-aware multimodal state representation framework that explicitly removes duplicated note text over time before policy learning. We evaluate two computationally efficient temporal decomposition strategies for removing duplicated note text: (1) an embedding-space decomposition using singular value decomposition on local history subspaces, and (2) an interpretable sentence-level diff operation that filters out previously documented sentences before text encoding. Using real-world ICU data, we demonstrate that state representations constructed by stripping temporal note redundancy significantly outperform both structured-only and raw-note baselines across multiple off-policy evaluation methods (Model-Based Rollouts, Fitted Q-Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust Evaluation). Our findings show that explicitly isolating new clinical information from repeated note text yields higher-quality state representations and directly improves RL performance for clinical decision support. Introduction Mechanical ventilation is an intervention in intensive care for patients unable to maintain adequate respiratory function. By delivering controlled air and oxygen, it assists or replaces spontaneous breathing. Effective management requires clinicians to continuously adjust ventilator settings as patient conditions evolve, while inappropriate settings can cause complications like ventilator-induced lung injury, diaphragm dysfunction, pneumonia, and oxygen toxicity (20). These risks make mechanical ventilation an important sequential decision-making problem. Offline reinforcement learning provides a natural framework for learning time-varying treatment strategies from retrospective ICU trajectories. Unlike supervised learning, which primarily imitates clinician actions, RL aims to optimize long-term outcomes from sequential states, actions, and rewards. Prior studies have applied RL to mechanical ventilation using ICU data (19; 12; 15; 7). However, these approaches mainly define patient states using structured EHR variables, which may not fully capture the clinical context considered during ventilator decisions. Clinical notes provide a natural complementary source of such context because they contain clinician assessments, treatment plans, and descriptions of patient status. However, using longitudinal notes directly is challenging because clinical documentation is often highly redundant over time due to templating, repeated documentation, copy-paste, and copy-forward practices. In a study of approximately 2.3 million inpatient notes, Vawdrey et al. found that 42% of inpatient notes used copy-paste and 19% of inpatient note content was copied (27). For offline RL, this creates a state representation problem. A raw note embedding may mix persistent historical content, repeated text, and newly introduced clinical updates, making decision-relevant changes harder to identify. Although modern language representation methods like large language models can provide a potential approach for summarizing or removing redundant information from clinical notes, applying them repeatedly to longitudinal note histories can be computationally and financially expensive due to the large number of patients and decision times in ICU trajectories. This motivates the need for a more efficient approach that can identify temporal redundancy while preserving clinically meaningful information for sequential decision-making. To address this challenge, we make the following contributions in this paper: 1. We introduce two computationally efficient note decomposition methods, one in embedding space and one at the sentence level, to separate history-related content from newly introduced information without relying on expensive large language model processing. These are our proposed methods for removing temporal note redundancy in a computationally efficient way. 2. We formulate a redundancy-aware multimodal state construction framework for offline RL in mechanical ventilation. 3. We evaluate the resulting state representations on MIMIC-I mechanical ventilation trajectories and show that they improve estimated policy value over structured-only and raw-note multimodal baselines across multiple off-policy evaluation methods. This supports the accuracy and efficacy of our approach. Related Work Mechanical Ventilation Optimization via RL RL has been increasingly studied for mechanical ventilation optimization, with prior work addressing dynamic ventilator control, model-based policy learning, interpretability, safety, and weaning. Peine et al. introduced VentAI, a tabular Q-learning framework that uses structured patient trajectories to recommend ventilation regimes and evaluate their association with mortality (19). Beyond this data-driven control formulation, Chen et al. explored a model-based hybrid soft actor-critic approach for learning ventilator settings (4), while Lee et al. emphasized interpretability and transparent off-policy evaluation for making learned ventilation policies more clinically meaningful (15). Another line of work focuses on safety and reward design: DeepVent applies conservative offline RL with clinically relevant intermediate rewards (12), Zhang et al. balance therapeutic effect and safety in ventilator recommendation (33), Eghbali et al. introduce uncertainty-aware recommendations through conformal deep Q-learning (8), and den Hengst et al. incorporate clinical guidelines into the RL learning signal and safety constraints (7). Related studies also consider mechanical ventilation weaning and extubation readiness using off-policy RL and inverse RL (21; 32). Multimodal EHR Representation Learning Multimodal EHR learning has shown that clinical notes can complement structured variables for medical prediction. Prior work fuses time-invariant features, time-series measurements, and clinical notes (31), and further shows that the gain from notes can depend strongly on the clinical information they contain rather than only on model architecture (10). Recent studies extend structured-text fusion to ICU outcome prediction (18; 23), and temporal modeling, including early sepsis prediction with physiological time series and ClinicalBERT note features (30), and cross-modal temporal pattern discovery in EHR trajectories (28). Broader multimodal EHR pretraining methods also aim to learn general representations across tasks (29). Closest to our setting, MORE-CLEAR incorporates clinical notes into offline RL state representations for sepsis treatment (16). However, these studies generally treat note representations as additional context, rather than asking how longitudinal note redundancy should be decomposed when notes become part of an RL state. Redundancy and Copy-Forward in Clinical Notes Clinical notes often contain repeated information because of templates, copy-paste, and copy-forward practices. Prior work has measured redundancy in EHR corpora and studied how it affects text mining performance (6). Other studies have described where duplicate information in electronic medical records comes from (25) and proposed practical tools for monitoring copy-paste use in clinical notes (27). This repeated content can have real effects: limiting copy-paste has been linked to better inpatient care quality (5), note bloat can affect deep learning models for clinical prediction (17), and recent work removes repeated chart text to make LLM-based clinical systems more efficient while keeping useful clinical information (3). These studies motivate redundancy reduction, but they do not address how repeated and newly introduced note content should be separated for sequential decision-making states; our work studies that gap in offline RL for mechanical ventilation. Method Mechanical Ventilation MDP We formulate mechanical ventilation management as a finite-horizon offline RL problem. Each patient trajectory is represented as a sequence of transitions (st,at,rt,st+1)t=1T,(s_t,a_t,r_t,s_t+1)_t=1^T, where sts_t denotes the patient state at time t, ata_t denotes the ventilator action, and rtr_t is the clinical reward. Following prior work on RL for mechanical ventilation, we use 4-hour decision intervals and define the structured MDP using the same observable clinical variables, action discretization, and reward function as in (15). This design isolate the effect of the proposed state representations while keeping the clinical decision problem fixed. The structured component of the state includes the following variables: • Respiratory: Respiratory rate, Spontaneous tidal volume, PaO2FiO2 Ratio, Mean airway pressure • Hemodynamic: Heart rate, Systolic BP, Diastolic BP • Blood Gas: SpO2, PaCO2, PaO2 • Miscellaneous: Sepsis, Weight, Age, GCS, Cumulative Fluid Balance Following the original MDP formulation, these observable variables are also augmented with a time-invariant propensity score estimated from additional patient-type features using logistic regression. In our experiments, this structured state serves as the baseline representation, to which different clinical-note representations are added. Action space. At each decision time, the action corresponds to a discretized ventilator setting. Following the same action definition as the prior work, the action is a tuple at=(atVt,atPEEP,atFiO2),a_t=(a_t^Vt,a_t^PEEP,a_t^FiO_2), where atVta_t^Vt denotes ideal-body-weight-adjusted tidal volume, atPEEPa_t^PEEP denotes positive end-expiratory pressure, and atFiO2a_t^FiO_2 denotes fraction of inspired oxygen. Although these ventilator settings are continuous in clinical practice, they are discretized into clinically meaningful bins through clustering analysis. This yields a finite discrete action space suitable for offline discrete RL. Reward function. We use the same clinically motivated reward function as prior work. The reward encourages improvement in blood oxygen saturation while penalizing aggressive ventilator settings. Specifically, the oxygenation component rewards increases in SpO2 only when both the current and next SpO2 values are below 95%, avoiding additional reward for increasing oxygen saturation beyond a clinically safe range: rSpO2(st,st+1)=st+1(SpO2)−st(SpO2)if st+1(SpO2)<95∧st(SpO2) <950otherwiser_SpO_2(s_t,s_t+1)= casess_t+1(SpO_2)-s_t(SpO_2)\\ $s_t+1$(SpO$_2$)$<95 s_t$(SpO$_2$) $<95$\\ 0 cases (1) The action penalty discourages high tidal volume and high inspired oxygen: ra(at)=−αatVt≥10−βatFiO2≥0.6r_a(a_t)=- 1\a_t^Vt≥ 10\- 1\a_t^FiO_2≥ 0.6\. The total reward is r(st,at,st+1)=rSpO2(st,st+1)+ra(at)r(s_t,a_t,s_t+1)=r_SpO_2(s_t,s_t+1)+r_a(a_t). We use this reward specification to maintain the same clinical objective across all state representations. Therefore, differences in policy performance can be attributed primarily to the proposed state representation rather than to changes in the action space or reward design. Problem Setup We study offline RL from ICU trajectories. At each time step t, the observed patient state contains structured clinical variables ut∈ℝdsu_t ^d_s and clinical note text DtD_t, yielding transitions =(ut,Dt,at,rt,ut+1,Dt+1)D=\(u_t,D_t,a_t,r_t,u_t+1,D_t+1)\ with action at∈a_t , reward rtr_t, and next state st+1s_t+1. To avoid temporal leakage, DtD_t contains only notes available up to the state-construction cutoff for that 4-hour interval; later notes are not used. A central challenge is that clinical notes contain substantial redundancy due to copy-forward, templating, and repeated historical content. As a result, a raw note embedding may mix together persistent background information and newly introduced clinical information, which can obscure the time-local patient state relevant for control. Our goal is therefore to construct a redundancy-aware note representation and fuse it with the structured variables for downstream offline RL. Our approach is based on the same principle across all variants: a note should be decomposed into a history component and an innovation component before it is used as part of the RL state. We consider two instantiations of this idea. First, we introduce an embedding-space temporal decomposition, where the current note embedding is projected onto a patient-local subspace spanned by previous notes, and the residual is treated as the innovation component. Second, we introduce a sentence-level diff decomposition, where we first compare the current note against previous notes at the sentence level, explicitly split the note text into historical and newly added sentences, and then embed these two text segments separately. The final RL state is formed by concatenating the structured variables with the compressed note-derived history and new-information features. Figure 1 summarizes the proposed pipeline. Starting from the MIMIC-I mechanical ventilation cohort, we align structured variables and clinical notes to the same 4-hour decision intervals. We then construct three state representations for comparison: a structured-only state, a raw-note multimodal baseline, and the proposed redundancy-aware multimodal states. The redundancy-aware representations decompose clinical notes into history-related and update-related components before combining them with structured clinical variables. The resulting states are used with the same offline policy-learning and off-policy evaluation pipeline, allowing us to isolate the effect of state representation. Figure 1: Overview of the redundancy-aware multimodal state construction and evaluation pipeline. Note Encoding For each note DtD_t, we compute xt=f(Dt)∈ℝdnx_t=f(D_t) ^d_n using ClinicalBERT (1). Intervals with no note text use a zero note vector. The same pretrained encoder is used for the raw-note baseline, the embedding-space decomposition, and the sentence-level diff decomposition, keeping the comparison focused on the redundancy-handling strategy rather than on the text encoder. Embedding-Space Temporal Decomposition The first representation variant performs deduplication after note embedding. The main idea is to characterize the subspace of note content already explained by recent note history and then extract the residual as the current innovation. Fix a time t, and consider the previous k notes from the same ICU stay: xt−k,xt−k+1,…,xt−1.x_t-k,x_t-k+1,…,x_t-1. For early time steps t<kt<k, we use the available history only; if the history is empty, the history component is set to zero and xtx_t is treated as new information. Define the local history mean μthist=1k∑i=t−kt−1xi _t^hist= 1k _i=t-k^t-1x_i. We then construct the centered history matrix Ht=[(xt−k−μthist)⊤(xt−k+1−μthist)⊤(xt−1−μthist)⊤]∈ℝk×dn.H_t= bmatrix(x_t-k- _t^hist) \\ (x_t-k+1- _t^hist) \\ \\ (x_t-1- _t^hist) bmatrix ^k× d_n. We compute Ht=UtΣtVt⊤H_t=U_t _tV_t , and let Vt,ℓV_t, contain the top ℓ right singular vectors, which span the recent-history subspace. With yt=xt−μthisty_t=x_t- _t^hist, the projection and residual are xtsvd,hist=μthist+Vt,ℓVt,ℓ⊤ytx_t^svd,hist= _t^hist+V_t, V_t, y_t and xtsvd,new=(I−Vt,ℓVt,ℓ⊤)ytx_t^svd,new=(I-V_t, V_t, )y_t. The original note embedding then admits the additive decomposition xt=xtsvd,hist+xtsvd,newx_t=x_t^svd,hist+x_t^svd,new. Here xtsvd,histx_t^svd,hist represents the part of the current note embedding explained by recent note history, while xtsvd,newx_t^svd,new captures the information not explained by the local history subspace. Sentence-Level Diff Decomposition The second representation variant performs deduplication before note embedding. The main idea is to explicitly separate the current note text into sentences that already appeared in the recent note history and sentences that are newly introduced at the current time step. Let t=(ct,1,ct,2,…,ct,mt)S_t=(c_t,1,c_t,2,…,c_t,m_t) denote the ordered sentence sequence of note DtD_t, where ct,jc_t,j is the j-th sentence in the note. Let the sentence history over the previous k notes be ℛt=t−k⊕t−k+1⊕⋯⊕t−1R_t=S_t-k _t-k+1 ·s _t-1, where ⊕ denotes temporal concatenation of sentence sequences. We apply a sentence-level diff operator based on the classical longest-common-subsequence formulation of differential file comparison (9): (thist,tnew)=Diff(t,ℛt)(S_t^hist,S_t^new)=Diff(S_t,R_t), where thistS_t^hist contains the current-note sentences aligned to the history sequence and tnewS_t^new contains the current-note sentences not aligned to the history sequence. We reconstruct two note texts: Dthist=concat(thist)D_t^hist=concat(S_t^hist) and Dtnew=concat(tnew)D_t^new=concat(S_t^new), where concat(⋅)concat(·) concatenates a sentence sequence into text. We then embed the two views separately: xtdiff,hist=f(Dthist)x_t^diff,hist=f(D_t^hist) and xtdiff,new=f(Dtnew)x_t^diff,new=f(D_t^new). If either sentence set is empty, we use the zero vector for the corresponding embedding. Compared with the embedding-space decomposition, the sentence-level diff decomposition removes redundancy earlier in the pipeline and does so in a more explicit and interpretable way. The “history” channel contains only text that can be traced to previous notes, while the “new” channel contains only text newly introduced at time t. This representation is particularly natural when note duplication occurs via sentence-level copy-forward. Compression of Note Components All note components remain high-dimensional and are therefore compressed before being used in downstream RL. We apply PCA fitted on the training set to each component separately, producing low-dimensional history and new-information features for both the embedding-space and sentence-level decompositions. This step controls state dimension and is not essential to the core redundancy decomposition. For the embedding-space decomposition, we obtain low-dimensional features ztsvd,hist∈ℝqsvd,histz_t^svd,hist ^q_svd,hist and ztsvd,new∈ℝqsvd,newz_t^svd,new ^q_svd,new from xtsvd,histx_t^svd,hist and xtsvd,newx_t^svd,new, respectively. For the sentence-level diff decomposition, we similarly obtain ztdiff,hist∈ℝqdiff,histz_t^diff,hist ^q_diff,hist and ztdiff,new∈ℝqdiff,newz_t^diff,new ^q_diff,new from xtdiff,histx_t^diff,hist and xtdiff,newx_t^diff,new. We summarize the different state constructions in Table 1. Both variants provide the RL model with two note-derived channels: a history-like channel and a newly introduced information channel. They differ in the stage at which redundancy is addressed. The embedding-space method performs decomposition after embedding, modeling redundancy as low-rank structure in the continuous representation space, whereas the sentence-level diff method removes redundancy before embedding by identifying explicit textual reuse. These approaches are complementary: the former is naturally aligned with continuous representation learning, while the latter provides an interpretable text-level decomposition for sentence-level copy-forward. In both cases, the goal is to construct note-based state representations that are less dominated by redundant content and more responsive to changes in the patient’s clinical condition. Method State Representation Structured-only ststruct=uts_t^struct=u_t Structured + raw note straw=[ut,ztraw]s_t^raw=[u_t,z_t^raw] Embedding-space SVD stsvd=[ut,ztsvd,hist,ztsvd,new]s_t^svd=[u_t,z_t^svd,hist,z_t^svd,new] Sentence-level diff stdiff=[ut,ztdiff,hist,ztdiff,new]s_t^diff=[u_t,z_t^diff,hist,z_t^diff,new] Table 1: Summary of state representations. Offline Policy Learning For each state representation in Table 1, we construct transitions (st,at,rt,st+1)(s_t,a_t,r_t,s_t+1) and train policies on the same training trajectories. This ensures that all downstream comparisons use the same patient population, action space, reward function, and policy-learning algorithm. Behavior cloning. We train a behavior cloning (BC) policy (2) to imitate historical clinician actions by solving the supervised classification problem. BC serves as a clinician-imitation baseline and is used to estimate the behavior policy for importance-ratio-based off-policy evaluation. Conservative Q-learning. Our main offline RL policy is trained using Conservative Q-Learning (CQL) (13), which mitigates overestimation and out-of-distribution action selection by learning a conservative Q-function. For each state representation, we train a separate CQL policy π^CQL(s)=argmaxa∈Qθ(s,a). π_CQL(s)= _a Q_θ(s,a). All policies use the same CQL algorithm and training procedure, with the only difference being the underlying state representation. Experiments Dataset This study uses data from MIMIC-I, a large-scale and openly accessible critical care database (11). We focus on ICU stays involving mechanical ventilation and follow the cohort construction and structured-data preprocessing pipeline of (15). In brief, patient trajectories are discretized into 4-hour intervals, and each episode contains demographic variables, laboratory measurements, vital signs, and ventilator-related variables. We include adult patients with documented 90-day mortality outcomes and recorded tidal volume measurements. In pre-processing procedure, continuous structured variables are winsorized to reduce the effect of extreme outliers, missing values are filled using a combination of forward/backward filling and KNN imputation, and trajectories are truncated to the first 72 hours of ventilation. This gives a maximum of 18 decision points per episode, with one decision point every 4 hours. In addition to the structured variables, we extract clinical notes associated with each ICU stay. Notes are aligned to the same 4-hour decision intervals as the structured observations. For each interval, we concatenate all notes whose timestamps fall within the interval, preserving the de-identified note time and note category as headings. The final note dataset contains 157,924 notes from 10,125 ICU stays, aligned to 147,997 decision intervals. Each ICU stay contains 15.60 notes on average. Off-Policy Evaluation We evaluate each learned target policy πe _e using four complementary off-policy estimators: (i) model-based rollouts (MB), (i) Fitted Q Evaluation (FQE), (i) Weighted Importance Sampling (WIS), and (iv) Weighted Doubly Robust evaluation (WDR). Each estimator carries different bias–variance tradeoffs, and we report all four to triangulate policy value. Throughout, γ denotes the discount factor and a trajectory τ=(s0,a0,r0,s1,a1,r1,…,sT)τ=(s_0,a_0,r_0,s_1,a_1,r_1,…,s_T) has length T. Model-Based Evaluation We learn a probabilistic transition model P^(s′∣s,a) P(s s,a) from the offline dataset, and evaluate πe _e by Monte Carlo rollouts in the learned model. Starting from each test-set initial state s0s_0, we sample a trajectory τ^=(s0,a^0,s^1,a^1,…) τ=(s_0, a_0, s_1, a_1,…) where a^t∼πe(⋅∣s^t) a_t _e(· s_t) and s^t+1∼P^(⋅∣s^t,a^t) s_t+1 P(· s_t, a_t), and accumulate the discounted reward V^MB(πe)=s0∼0[∑t=0T−1γtr^(s^t,a^t)], V^MB( _e)\;=\;E_s_0 _0 [ _t=0^T-1γ^t\, r( s_t, a_t) ], (2) where r^(⋅,⋅) r(·,·) is the same reward function used during policy training and 0D_0 is the empirical distribution of test-set initial states. To reduce variance from stochastic rollouts, we average over 5 independent trajectories per initial state. MB is unbiased only if the learned P P matches the true transition kernel; in practice it can be optimistic when πe _e visits state–action regions with sparse training support. Fitted Q Evaluation Fitted Q Evaluation (FQE) (14) estimates policy value by iteratively fitting a Q-function under the evaluation policy. The final value is computed from the learned Q-function: V^FQE(πe)=s0∼0[a0∼πe(⋅|s0)Qπe(s0,a0)]. V_FQE( _e)=E_s_0 _0 [E_a_0 _e(·|s_0)Q _e(s_0,a_0) ]. (3) Weighted Importance Sampling Weighted Importance Sampling (WIS) (22) estimates policy value by reweighting observed trajectories according to the ratio between the evaluation policy and the estimated behavior policy. We estimate the behavior policy using a behavior-cloning model with a small uniform mixture (ϵ=0.05ε=0.05). Weighted Doubly Robust Evaluation Weighted Doubly Robust evaluation (WDR) (26) combines importance sampling with the FQE value estimates to reduce variance while retaining correction from observed trajectories. Implementation Details We split the mechanical ventilation episodes into training and testing subsets in an 80:20 ratio. Within the training set, 5-fold cross-validation was used to select shared reward and policy-learning hyperparameters. Because this step tunes parameters that are shared across all state representations, cross-validation was carried out using the structured-only state representation, which serves as our baseline. We tuned the Conservative Q-Learning coefficient αCQL _CQL together with the reward-penalty coefficients α and β, which balance improving oxygenation (measured by per-step changes in SpO2) against discouraging aggressive ventilator settings. Aggressive actions were defined as Vtset index ≥6≥ 6 or FiO2 index ≥3≥ 3, corresponding to the higher action indices in our discretized action space. The binning intervals used to construct the action space are reported in Table 2. Bin Vtset PEEP FiO2 (%) 1 [0.00,3.94)[0.00,3.94) [0.00,7.04)[0.00,7.04) [0,38.13)[0,38.13) 2 [3.94,5.36)[3.94,5.36) [7.04,11.28)[7.04,11.28) [38.13,51.73)[38.13,51.73) 3 [5.36,6.54)[5.36,6.54) [11.28,16.47)[11.28,16.47) [51.73,62.12)[51.73,62.12) 4 [6.54,7.74)[6.54,7.74) [16.47,∞)[16.47,∞) [62.12,68.12)[62.12,68.12) 5 [7.74,9.12)[7.74,9.12) – [68.12,76.82)[68.12,76.82) 6 [9.12,11.11)[9.12,11.11) – [76.82,90.94)[76.82,90.94) 7 [11.11,∞)[11.11,∞) – [90.94,100.00][90.94,100.00] Table 2: Action space binning intervals. For each candidate setting, we trained a CQL policy on the four training folds and evaluated it on the validation fold using model-based rollouts. Each rollout was initialized from the observed initial state of an ICU stay, and at each subsequent step the learned CQL policy selected an action while the learned transition model predicted the next state. The SpO2 improvement for ICU stay i was the mean per-step change in inverse-transformed SpO2 along the simulated trajectory, ΔiSpO2=1Ti−1∑t=0Ti−2(SpO^2,i,t+1−SpO^2,i,t), _i^SpO_2= 1T_i-1 _t=0^T_i-2 ( SpO_2,i,t+1- SpO_2,i,t ), averaged across stays in the validation fold. To characterize the accuracy of the transition model used inside model-based evaluation, we evaluated it by one-step next-state prediction on the held-out test set. For next-step SpO2 prediction in physiological units, the model achieved a mean absolute error of 1.0801.080 and an ROC-AUC of 0.9030.903 for classifying whether SpO2 exceeded 94%94\%. The selected configuration maximized validation SpO2 improvement subject to a safety constraint on the rate of aggressive actions, yielding α∗=0.375α^*=0.375, β∗=0.75β^*=0.75, and αCQL=0.25 _CQL=0.25. For all note-based state representations, we fixed representation hyperparameters before policy learning. The history window was set to k=18k=18, matching the maximum number of 4-hour decision points in the 72-hour horizon; the embedding-space method used ℓ=2 =2 singular vectors, and each note-derived component was compressed to q=64q=64 dimensions using training-set PCA. These values were shared across representations to keep state dimensions comparable. All final models were trained with fixed hyperparameter settings after cross-validation. The CQL policies used a discount factor γ=0.99γ=0.99 and were trained with d3rlpy library (24), which uses a two-layer MLP encoder with 256256 hidden units per layer, the Adam optimizer with learning rate 6.25×10−56.25× 10^-5. The behavior cloning models used a two-layer fully connected encoder with [128,128][128,128] hidden units, learning rate 10−410^-4, batch size 512512, weight decay 10−410^-4, and the Adam optimizer. The transition model was a two-layer MLP with hidden dimension 256256, dropout probability 0.10.1, learning rate 10−310^-3, batch size 256256, weight decay 10−410^-4, the Adam optimizer, and early stopping on held-out next-state MSE. These settings were held fixed across state representations. Numerical Results Figure 2: Off-policy evaluation of state representations. Paired-bootstrap mean differences in estimated policy value (left row label minus right) with 95%95\% CIs. Filled diamonds: CI excludes zero. Bottom row (below dashed line): raw-note check. Aggregate OPE Comparisons Across State Representations Figure 2 reports paired bootstrap comparisons of estimated policy value across state representations. Positive values indicate higher estimated value for the representation on the left than for the baseline on the right. Filled diamonds denote comparisons whose 95%95\% bootstrap confidence intervals exclude zero, and hollow circles denote comparisons whose intervals include zero. We evaluate each comparison using four OPE estimators: FQE, WIS, WDR, and MB. Both redundancy-aware representations consistently improve over the structured-only baseline. The embedding-space SVD representation achieves estimated gains of +0.12+0.12 (FQE), +0.40+0.40 (WIS), +0.22+0.22 (WDR), and +0.73+0.73 (MB). The sentence-level diff representation achieves larger gains of +0.27+0.27, +0.64+0.64, +0.41+0.41, and +0.75+0.75 under the same estimators. All of these comparisons have confidence intervals that exclude zero. The redundancy-aware representations also improve over the raw-note baseline. The SVD representation improves over raw notes by +0.14+0.14 to +0.46+0.46, and the diff representation improves over raw notes by +0.28+0.28 to +0.69+0.69, depending on the estimator. In contrast, the raw-note representation does not show a consistent improvement over the structured-only state: its confidence intervals include zero under FQE, WIS, and WDR, and only MB gives a positive point estimate. This pattern suggests that the observed gains are not simply due to adding note embeddings, but to representing notes in a way that separates persistent clinical history from newly introduced information. FQE produces narrower bootstrap intervals than WIS and WDR, reflecting the difference between value function based estimation and trajectory level importance weighting. Since each estimator has different bias–variance tradeoffs, we base our conclusions on the consistent improvements observed across all four OPE methods. Case Study: Separating Stable Ventilation History To better understand how the redundancy-aware note representations affect policy recommendations, we examine a representative held-out trajectory from ICU stay 246432. At decision time t=8t=8, the clinician selected action 412, and all learned policies agreed with this choice. At the following decision time t=9t=9, the clinician again selected action 412. The policies learned from the structured-only, embedding-space SVD, and sentence-level diff representations also selected action 412, while the policy learned from the raw-note representation selected action 512. The notes around this decision are summarized in Table 3. At t=8t=8, the respiratory care note describes a stable ventilation status, with stable arterial blood gases and no ventilator changes required. At t=9t=9, the note contains substantial new clinical information, including planned surgery, possible transfusion, coagulation abnormalities, potassium replacement, and continued sedation/paralysis. However, the ventilation-related information remains unchanged: the patient continues to have good arterial blood gas results on 40% FiO2 and 5 PEEP, with clear bilateral breath sounds. This example illustrates the challenge of using raw note representations for ventilator control. The raw note embedding may capture newly introduced information that reflects broader clinical events but is not directly relevant to ventilator adjustment, causing the policy to change its recommendation. In contrast, the redundancy-aware representations preserve the stable ventilation context from previous notes while separating newly introduced information, allowing the learned policies to remain consistent with the clinician’s action. This suggests that redundancy reduction is not only about removing repeated text, but also about preserving clinically relevant history for sequential decision-making. Time Note type Excerpt and interpretation t=8t=8 Respiratory care note The patient remained intubated, sedated, and mechanically ventilated. The note reports stable ABGs, clear and equal bilateral lung sounds, and no ventilator changes required overnight. t=9t=9 Nursing note The note adds new information about planned return to the operating room, possible transfusion, elevated coagulation measures, potassium replacement, and continued sedation/paralysis. However, the ventilation-specific content is stable: good ABG on 40% FiO2 and 5 PEEP, with clear bilateral breath sounds. Table 3: Clinical notes around decision time t=9t=9 for ICU stay 246432. Conclusion In conclusion, we presented a redundancy-aware multimodal state representation framework for offline RL in mechanical ventilation. Our approach addresses the challenge that longitudinal clinical notes contain substantial temporal redundancy, which can obscure decision-relevant information when raw note embeddings are directly incorporated into RL states. Rather than modifying the RL algorithm, we focus on improving state construction by decomposing clinical notes into history-related and new-information components before policy learning. We introduce two computationally efficient implementations of this idea: an embedding-space temporal decomposition based on recent note history and a sentence-level diff decomposition that explicitly separates repeated and newly introduced text, which are able to successfully deal with temporally redundant information in clinical notes. Using MIMIC-I mechanical ventilation trajectories, we show that both redundancy-aware representations improve estimated policy value over structured-only and raw-note multimodal baselines across multiple off-policy evaluation methods. These results demonstrate that clinical notes can provide useful additional information for offline RL when their temporal redundancy is explicitly modeled. Separating persistent clinical context from newly introduced information helps construct more effective multimodal state representations for sequential decision-making. However, these findings should be interpreted within the limitations of retrospective offline RL. Off-policy evaluation estimates policy performance from observational data rather than providing direct evidence of clinical benefit, and learned policies may still be affected by dataset bias, distribution shift, and limited coverage of state-action pairs. In addition, our decomposition methods identify textual or representational changes, which may not always correspond to clinically meaningful updates for ventilator control. Finally, this study is based on a single ICU dataset, MIMIC-I, and further validation on more diverse clinical settings is needed before deployment. Future work will investigate combining redundancy-aware representations with uncertainty estimation, clinical interpretability methods, and prospective evaluation to develop more reliable multimodal RL systems for clinical decision support. References Alsentzer et al. (2019) E. Alsentzer, J. Murphy, W. Boag, W. Weng, D. Jindi, T. Naumann, and M. McDermott Publicly available clinical bert embeddings. In Proceedings of the 2nd clinical natural language processing workshop, p. 72–78. Cited by: Note Encoding. Bain and Sammut (1995) M. Bain and C. Sammut A framework for behavioural cloning. In Machine Intelligence 15, p. 103–129. Cited by: Behavior cloning.. Cahoon et al. (2026) J. L. Cahoon, C. Stanwyck, A. Aali, R. Madding, E. Sun, Y. Jiang, R. Dhanasekaran, and E. Alsentzer Clinical note bloat reduction for efficient llm use. External Links: 2604.16364 Cited by: Redundancy and Copy-Forward in Clinical Notes. Chen et al. (2022) S. Chen, X. Qiu, X. Tan, Z. Fang, and Y. Jin A model-based hybrid soft actor-critic deep reinforcement learning algorithm for optimal ventilator settings. Information sciences 611, p. 47–64. Cited by: Mechanical Ventilation Optimization via RL. Cheng et al. (2022) C. Cheng, D. Wu, J. Lu, C. Yu, H. Lin, M. Wang, and C. Cheng Restricted use of copy and paste in electronic health records potentially improves healthcare quality. Medicine 101 (4), p. e28644. Cited by: Redundancy and Copy-Forward in Clinical Notes. Cohen et al. (2013) R. Cohen, M. Elhadad, and N. Elhadad Redundancy in electronic health record corpora: analysis, impact on text mining performance and mitigation strategies. BMC bioinformatics 14 (1), p. 10. Cited by: Redundancy and Copy-Forward in Clinical Notes. den Hengst et al. (2024) F. den Hengst, M. Otten, P. Elbers, F. van Harmelen, V. François-Lavet, and M. Hoogendoorn Guideline-informed reinforcement learning for mechanical ventilation in critical care. Artificial intelligence in medicine 147, p. 102742. Cited by: Introduction, Mechanical Ventilation Optimization via RL. Eghbali et al. (2025) N. Eghbali, T. Alhanai, and M. M. Ghassemi Distribution-free uncertainty quantification in mechanical ventilation treatment: a conformal deep q-learning framework. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 27960–27968. Cited by: Mechanical Ventilation Optimization via RL. Hunt and McIlroy (1976) J. W. Hunt and M. D. McIlroy An algorithm for differential file comparison. Technical report Technical Report Computing Science Technical Report 41, Bell Laboratories. Cited by: Sentence-Level Diff Decomposition. Husmann et al. (2022) S. Husmann, H. Yèche, G. Rätsch, and R. Kuznetsova On the importance of clinical notes in multi-modal learning for ehr data. External Links: 2212.03044 Cited by: Multimodal EHR Representation Learning. Johnson et al. (2016) A. E. W. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark MIMIC-i, a freely accessible critical care database. Scientific Data 3, p. 160035. External Links: Document Cited by: Dataset. Kondrup et al. (2023) F. Kondrup, T. Jiralerspong, E. Lau, N. de Lara, J. Shkrob, M. D. Tran, D. Precup, and S. Basu Towards safe mechanical ventilation treatment using deep offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 15696–15702. Cited by: Introduction, Mechanical Ventilation Optimization via RL. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems 33, p. 1179–1191. Cited by: Conservative Q-learning.. Le et al. (2019) H. Le, C. Voloshin, and Y. Yue Batch policy learning under constraints. In International Conference on Machine Learning, p. 3703–3712. Cited by: Fitted Q Evaluation. Lee et al. (2025) J. S. Lee, M. Mahendra, and A. Aswani Matching-based off-policy evaluation for reinforcement learning applied to mechanical ventilation. In Proceedings of the ACM/IEEE International Conference on Connected Health: Applications, Systems and Engineering Technologies, p. 211–221. Cited by: Introduction, Mechanical Ventilation Optimization via RL, Mechanical Ventilation MDP, Dataset. Lim et al. (2025) Y. Lim, B. Jeon, S. Park, J. Lee, S. W. Choi, C. W. Jeong, H. Ryu, H. Lee, and H. Yang MORE-clear: multimodal offline reinforcement learning for clinical notes leveraged enhanced state representation. External Links: 2508.07681 Cited by: Multimodal EHR Representation Learning. Liu et al. (2022) J. Liu, D. Capurro, A. Nguyen, and K. Verspoor “Note bloat” impacts deep learning-based nlp models for clinical prediction tasks. Journal of biomedical informatics 133, p. 104149. Cited by: Redundancy and Copy-Forward in Clinical Notes. Lyu et al. (2023) W. Lyu, X. Dong, R. Wong, S. Zheng, K. Abell-Hart, F. Wang, and C. Chen A multimodal transformer: fusing clinical notes with structured ehr data for interpretable in-hospital mortality prediction. In AMIA Annual Symposium Proceedings, Vol. 2022, p. 719. Cited by: Multimodal EHR Representation Learning. Peine et al. (2021) A. Peine, A. Hallawa, J. Bickenbach, G. Dartmann, L. B. Fazlic, A. Schmeink, G. Ascheid, C. Thiemermann, A. Schuppert, R. Kindle, et al. Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care. NPJ digital medicine 4 (1), p. 32. Cited by: Introduction, Mechanical Ventilation Optimization via RL. Pham et al. (2017) T. Pham, L. J. Brochard, and A. S. Slutsky Mechanical ventilation: state of the art. In Mayo Clinic Proceedings, Vol. 92, p. 1382–1400. Cited by: Introduction. Prasad et al. (2017) N. Prasad, L. Cheng, C. Chivers, M. Draugelis, and B. E. Engelhardt A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. External Links: 1704.06300 Cited by: Mechanical Ventilation Optimization via RL. Precup et al. (2000) D. Precup, R. S. Sutton, and S. P. Singh Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, p. 759–766. Cited by: Weighted Importance Sampling. Ruan et al. (2025) Y. Ruan, D. J. Tan, L. Huang, and M. Feng Evidence-based multimodal fusion on structured ehrs and free-text notes for icu outcome prediction. External Links: 2501.04389 Cited by: Multimodal EHR Representation Learning. Seno and Imai (2022) T. Seno and M. Imai D3rlpy: an offline deep reinforcement learning library. Journal of Machine Learning Research 23 (315), p. 1–20. Cited by: Implementation Details. Steinkamp et al. (2022) J. Steinkamp, J. J. Kantrowitz, and S. Airan-Javia Prevalence and sources of duplicate information in the electronic medical record. JAMA network open 5 (9), p. e2233348. Cited by: Redundancy and Copy-Forward in Clinical Notes. Thomas and Brunskill (2016) P. Thomas and E. Brunskill Data-efficient off-policy policy evaluation for reinforcement learning. In International conference on machine learning, p. 2139–2148. Cited by: Weighted Doubly Robust Evaluation. Vawdrey et al. (2022) D. K. Vawdrey, C. Cauthorn, D. Francis, K. Hackenberg, G. Maloney, and B. A. Hohmuth A practical approach for monitoring the use of copy-paste in clinical notes. In AMIA Annual Symposium Proceedings, Vol. 2021, p. 1178. Cited by: Introduction, Redundancy and Copy-Forward in Clinical Notes. Wang et al. (2025) F. Wang, F. Wu, Y. Tang, and L. Yu Ctpd: cross-modal temporal pattern discovery for enhanced multimodal electronic health records analysis. In Findings of the Association for Computational Linguistics: ACL 2025, p. 6783–6799. Cited by: Multimodal EHR Representation Learning. Wang et al. (2023) X. Wang, J. Luo, J. Wang, Z. Yin, S. Cui, Y. Zhong, Y. Wang, and F. Ma Hierarchical pretraining on multimodal electronic health records. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 2839–2852. Cited by: Multimodal EHR Representation Learning. Wang et al. (2022) Y. Wang, Y. Zhao, R. Callcut, and L. Petzold Integrating physiological time series and clinical notes with transformer for early prediction of sepsis. External Links: 2203.14469 Cited by: Multimodal EHR Representation Learning. Yang and Wu (2021) B. Yang and L. Wu How to leverage the multimodal ehr data for better medical prediction?. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 4029–4038. Cited by: Multimodal EHR Representation Learning. Yu et al. (2019) C. Yu, J. Liu, and H. Zhao Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC medical informatics and decision making 19 (Suppl 2), p. 57. Cited by: Mechanical Ventilation Optimization via RL. Zhang et al. (2024) B. Zhang, X. Qiu, and X. Tan Balancing therapeutic effect and safety in ventilator parameter recommendation: an offline reinforcement learning approach. Engineering Applications of Artificial Intelligence 131, p. 107784. Cited by: Mechanical Ventilation Optimization via RL.