Paper deep dive
Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane, Sudip Mittal, Shahram Rahimi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/2/2026, 9:15:15 AM
Summary
This study empirically evaluates five context strategies (Full, Recent, Episodic, Semantic, Hybrid) for longitudinal clinical reasoning using the MedLoCoMo benchmark across four open-weight LLMs. The research investigates how context selection affects answer correctness, robustness to evidence distance, and model abstention on unsupported premises. Results indicate that retrieval-based strategies, particularly Episodic and Hybrid, achieve superior accuracy compared to Full or Recent context, especially as the distance between the query and supporting evidence increases. The findings suggest that reliable long-horizon reasoning depends critically on how relevant evidence is selected and presented, rather than merely the volume of history accessible.
Entities (11)
Relation Signals (8)
Hybrid → combines → Episodic
confidence 95% · Hybrid: Episodic chunks and semantic facts are maintained in separate memory stores and retrieved independently.
Hybrid → combines → Semantic
confidence 95% · Hybrid: Episodic chunks and semantic facts are maintained in separate memory stores and retrieved independently.
MedLoCoMo → evaluates → Longitudinal clinical reasoning
confidence 95% · MedLoCoMo serves as the primary benchmark for longitudinal clinical reasoning.
Episodic → outperforms → Recent Context
confidence 95% · Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant
Gemma4-26B → usedas → Correctness Judge
confidence 95% · We use Gemma4-26B [16] as the primary correctness judge
Semantic → uses → Patient-specific facts
confidence 95% · Semantic: Each session is transformed... into self-contained patient-specific facts
Episodic → uses → Encounter-level text chunks
confidence 95% · Episodic: The original history is divided into overlapping encounter-level text chunks.
Hybrid → outperforms → Full Context
confidence 90% · retrieval-based strategies, particularly Episodic and Hybrid, generally provide stronger performance despite operating over substantially less context.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence distance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2610.00562v1
- Canonical: https://arxiv.org/abs/2610.00562v1
Trouble viewing inline? Open PDF directly →
Full Text
46,806 characters extracted from source content.
Expand or collapse full text
Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning Taye Akinrele ∗ , Noorbakhsh Amiri Golilarz † , Subash Neupane ‡ , Sudip Mittal § , Shahram Rahimi ¶ Department of Computer Science, The University of Alabama toakinrele ∗ , namirigolilarz † , sneupane4 ‡ , smittal1 § , srahimi1 ¶ @ua.edu Abstract—Longitudinal clinical reasoning requires large lan- guage models (LLMs) to identify and integrate relevant evi- dence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence dis- tance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accu- racy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for reasoning. Index Terms—Long-horizon reasoning, longitudinal clinical reasoning, large language models, context strategies, memory- augmented LLMs, retrieval-augmented generation, clinical AI I. INTRODUCTION Clinical decisions often depend on evidence accumulated across a patient’s history rather than information contained within a single encounter [1]. Diagnostic findings, treatments, laboratory results, medication changes, and clinical decisions may be distributed across multiple hospitalizations and out- patient visits spanning months or years [2]. Longitudinal clinical reasoning therefore requires systems to identify and connect relevant evidence across time to understand disease progression, recurring symptoms, treatment changes, and prior care decisions. With the emergence of large language models (LLMs) with extended context windows, a straightforward approach to longitudinal clinical reasoning is to provide the model with a patient’s full available history. In principle, this keeps relevant evidence from prior encounters accessible during reasoning. However, the ability to accommodate long contexts does not guarantee that all information within them will be used effectively. MedOdyssey demonstrates that long-context LLMs remain sensitive to the position of relevant medical informa- tion, exhibiting a lost-in-the-middle effect as context grows [3]. LongMemEval similarly reports substantial degradation Longitudinal History Query Context Strategy Answer Answer Model Fig. 1: Overview of the experimental framework. Given a longitudinal patient history and query, a context strategy determines the information provided to the answer model. We compare Full Context, Recent Context, Episodic, Semantic, and Hybrid strategies under the same underlying histories and queries. over long multi-session histories, particularly for multi-session reasoning, temporal reasoning, and knowledge updating [4]. In longitudinal clinical settings, cross-admission reasoning remains particularly challenging [1]. These findings suggest that increasing context capacity alone does not solve the problem of identifying and reasoning over relevant evidence distributed across long histories. An alternative is to selectively construct the context pre- sented to the model. Memory-augmented systems can retrieve information at different levels of abstraction, such as indi- vidual experiences or consolidated facts [5]. However, exist- ing memory systems often combine representation, retrieval, consolidation, updating, and context construction within a single architecture, making it difficult to isolate how the context provided to the model affects reasoning performance. This motivates a more fundamental question: how should longitudinal clinical information be selected and organized for effective long-horizon reasoning? This study investigates this question through a controlled empirical evaluation on MedLoCoMo [1]. We compare five context strategies: Full Context, Recent Context, Episodic, Semantic, and Hybrid. These strategies range from direct access to longitudinal history to relevance-based retrieval of encounter-level and fact-level information. We examine how context strategy affects reasoning across clinical tasks, ro- bustness to increasing query-evidence distance, and abstention when the available evidence does not support the requested conclusion (illustrated in Fig. 1). We address three research arXiv:2610.00562v1 [cs.CL] 30 Sep 2026 questions: • RQ1: How do different context strategies affect long- horizon reasoning performance, and how does their ef- fectiveness vary across reasoning tasks? • RQ2: How does long-horizon reasoning performance change as the distance between relevant evidence and the query increases, and how does this degradation differ across context strategies? • RQ3: How does context strategy affect abstention when evidence is insufficient? Contributions • We provide a controlled evaluation of five context strategies for longitudinal clinical reasoning, enabling direct comparison of long-context, recency-based, and retrieval-based approaches. • We show that access to more longitudinal history does not necessarily improve reasoning performance: retrieval-based strategies, particularly Episodic and Hybrid, generally provide stronger performance de- spite operating over substantially less context. • We demonstrate that context strategy affects multiple dimensions of long-horizon reasoning, including an- swer correctness, performance on distant evidence, and abstention when the available history does not support the requested conclusion. The remainder of this paper is organized as follows. Sec- tion I reviews related work, followed by the problem state- ment in Section I. Section IV describes the methodology and experimental setup. Section V presents and discusses the results, and Section VI concludes the paper. I. RELATED WORK Longitudinal clinical reasoning requires models to integrate information distributed across repeated patient encounters, including changes in diagnoses, medications, laboratory re- sults, and treatments. Recent medical benchmarks show that this remains difficult for LLMs. MedLoCoMo [1] evaluates single- and cross-admission reasoning over longitudinal patient histories grounded in MIMIC-IV, while LongMedBench [6] evaluates factual retrieval, temporal reasoning, and clinical decision-making over longitudinal EHRs. Related work such as MediLongChat [2] similarly evaluates reasoning across extended medical dialogues. These studies demonstrate that access to a patient’s history alone does not guarantee effective longitudinal reasoning, particularly when relevant evidence is distributed across encounters. This challenge is closely related to research on long-term LLM memory and long-context reasoning. LongMemEval demonstrates substantial performance degradation in informa- tion extraction, multi-session reasoning, temporal reasoning, knowledge updating, and abstention over extended histories [4]. Similarly, the “lost-in-the-middle” phenomenon shows that information can become difficult for LLMs to use when it occurs between the beginning and end of a long context, even when the information remains within the model’s context win- dow [7]. These findings motivate approaches that selectively construct the context provided to an LLM rather than relying exclusively on full-context prompting. Memory-augmented approaches provide one mechanism for constructing such contexts. CoALA distinguishes episodic memory for past experiences, semantic memory for consol- idated knowledge, procedural memory for learned procedures, and working memory for information actively used during rea- soning [5]. Systems such as MemoryBank [8], MemGPT [9], and AdMem [10] illustrate different approaches to maintaining and retrieving information beyond an LLM’s immediate con- text. Generative agents, for example, retrieve past experiences according to relevance, recency, and importance and construct higher-level reflections [11]. More recent work compares se- mantic, episodic, and procedural memory strategies for long- context question answering, providing evidence that memory design can affect reasoning performance [12]. However, the evaluated memory functions are not independently controlled: episodic and procedural components are layered over semantic retrieval, while working memory remains fixed. MemTrace [13] further emphasizes the importance of distinguishing failures in accessing relevant information from failures in reasoning after that information has been retrieved. Taken together, prior work suggests that long-horizon per- formance depends not only on the amount of history avail- able to an LLM, but also on how that history is selected, organized, and provided during reasoning. We refer to these alternatives as context strategies. Existing clinical studies largely evaluate long-context prompting, retrieval methods, or complete memory architectures, making it difficult to compare direct-context and memory-based approaches under a common experimental setting. In particular, controlled evidence remains limited on whether different context strategies are better suited to different forms of longitudinal clinical reasoning, how their effectiveness changes as supporting evidence becomes more distant, and how they affect model behavior when the available history does not support the requested conclusion. I. PROBLEM STATEMENT Long-horizon reasoning requires a system to answer ques- tions using information accumulated over an extended history. Let H = S 1 ,S 2 ,...,S T denote a longitudinal history, where each S t represents information observed at a particular point in time, and let q denote a query about that history. The goal is to produce an answer ˆy supported by information contained in H. As the history grows, relevant evidence may constitute only a small portion of the available information and may be separated from the query by many intervening observations. We therefore define a context strategy C as a function that determines how information from the longitudinal history is selected, organized, and provided to the answer model. Given a history H and query q, E q = C(q,H),ˆy = g(q,E q ),(1) where E q is the context provided to the answer model g. Different context strategies may preserve, prioritize, or retrieve different parts of the longitudinal history. Conse- quently, providing more history does not necessarily provide more useful evidence for reasoning. This work investigates how context strategy affects answer correctness, robustness to evidence distance, and abstention under insufficient evidence. IV. METHODOLOGY This study evaluates how different context strategies affect an LLM’s ability to answer questions requiring information distributed across long, multi-session histories. We compare five strategies: full context, recentcontext, episodic, semantic, and hybrid (episodic + semantic). MedLoCoMo [1] serves as the primary benchmark for longitudinal clinical reasoning. To assess whether the observed effects generalize beyond the clin- ical domain, we additionally evaluate the same strategies on LongMemEval [4], a general-domain benchmark for long-term conversational memory (results are reported in Section V-E). A. Datasets MedLoCoMo: We evaluate on MedLoCoMo, a longitu- dinal clinical question-answering benchmark comprising 100 patient timelines organized as chronologically ordered hos- pital admissions. Each patient record contains admission- level conversations and evidence-linked question-answer pairs spanning single-admission and cross-admission reasoning. We converted the native per-patient representation into a com- mon flat schema containing the question, answer, longitudinal history, temporal metadata, and identifiers of the supporting admissions, allowing all context strategies to operate over the same underlying patient histories. For the experiments, we sampled 1,000 questions and used the same question set across all context strategies and answer models. The sample was approximately balanced across the seven combinations of MedLoCoMo’s scope and question_type fields, as shown in Table I. The five an- swerable categories comprise medical reasoning, care-plan ra- tionale, longitudinal progression, cross-admission comparison, and frequency pattern, while the two adversarial categories contain questions with unsupported premises. Together, the adversarial categories account for 286 of the 1,000 evaluation questions. Foranswerablequestions, answer_session_ids records the admissions containing the supporting evidence. For adversarial questions, we set answer_session_ids to the empty set because the evidence field supplied by MedLoCoMo identifies the near-miss admissions used to construct the unsupported question rather than genuine supporting evidence. The patient histories are substantially longer than the evi- dence budget available to the retrieval-based context strategies. As summarized in Table I, a patient timeline contains an average of 29.5 admission sessions and 1,662.3 dialogue TABLE I: Distribution of question types in the 1,000-question evaluation subset. Question TypeScopeCount% Medical reasoningSingle14214.2 Care-plan rationaleSingle14314.3 AdversarialSingle14314.3 Longitudinal progressionCross14314.3 Cross-admission comparisonCross14314.3 Frequency patternCross14314.3 AdversarialCross14314.3 Total1,000100.0 TABLE I: Longitudinal context statistics for the patient timelines. Token counts are estimated using the same approximation employed for experimental evidence budgeting. StatisticSessionsTurnsEst. Tokens Mean29.51,662.335,072.6 Median28.01,623.034,099.0 Min1791320,451 Max643,52274,255 turns, corresponding to approximately 35,073 tokens under the token-length estimator used in our experimental pipeline. The longest timeline contains 64 sessions and 3,522 turns, with an estimated length of 74,255 tokens. These statistics are computed over the 100 patient timelines, with each patient counted once rather than once per question. B. Context Strategies We evaluate five context strategies (Fig. 2): Full, Recent, Episodic, Semantic, and Hybrid. Full uses the available lon- gitudinal history up to the model’s context limit, while the remaining strategies operate under a common evidence budget. • Full context: The longitudinal history is provided ver- batim without relevance-based retrieval or compression. All sessions are included when they fit within the model’s context window; otherwise, the history is truncated to the available context limit. • Recent context: The most recent raw sessions are retained until the evidence budget is exhausted, without relevance- based retrieval or compression. This provides a budget- matched recency baseline. • Episodic: The original history is divided into overlapping encounter-level text chunks. Each chunk is embedded independently and ranked by cosine similarity to the query, with the highest-ranked chunks selected within the evidence budget. • Semantic: Each session is transformed through an LLM- based extraction step into self-contained patient-specific facts with source-session and temporal metadata. Facts are embedded independently, ranked by cosine similarity to the query, and selected within the evidence budget. • Hybrid: Episodic chunks and semantic facts are main- tained in separate memory stores and retrieved indepen- dently. Similarity-ranked candidates from both stores are Raw sessions Longitudinal conversations Chunking Raw conversation chunks EMBEDDING_MODEL Vector Index (episodic chunks embeddings) QueryEMBEDDING_MODEL Cosine similarity (query vs chunk vectors) Rank & select candidates highest similarity first Fit token budget Retrieved episodic evidence ANSWER_MODELGenerated output Raw sessions Longitudinal conversations Atomic facts + metadata (source date, session ID) EMBEDDING_MODEL Vector Index (semantic fact embeddings) EXTRACTION_MODEL QueryEMBEDDING_MODEL Cosine similarity (query vs fact vectors) Rank & select candidates highest similarity first Fit token budget Retrieved semantic evidence ANSWER_MODELGenerated output Raw sessions Chunking Raw conversation chunks EMBEDDING_MODEL Vector Index (episodic chunks embeddings) Raw sessions Atomic facts + metadata (source date, session ID) EMBEDDING_MODEL Vector Index (semantic fact embeddings) EXTRACTION_MODEL QueryEMBEDDING_MODEL Independent Retrieval (rank episodic chunks + semantic facts separately) Interleave (S1, E1, S2, E2,..) Shared token budget ANSWER_MODELGenerated output (A) FULL CONTEXT (C) EPISODIC (D) SEMANTIC (E) HYBRID (EPISODIC + SEMANTIC) (B) RECENT CONTEXT Full patient history All conversations ANSWER_MODELGenerated output Query Full patient history All conversations ANSWER_MODELGenerated outputQuery Recent-window selection Fig. 2: Context strategies evaluated for longitudinal clinical reasoning. Full and Recent provide direct-context baselines, while Episodic, Semantic, and Hybrid use relevance-based retrieval over encounter-level evidence, patient-specific facts, or their combination, respectively. interleaved by rank and selected under a shared evidence budget. C. Fact Extraction and Caching Semantic memory was constructed by extracting patient- specific factual statements from each session independently using Qwen3-4B [14]. We used LLM-based extraction because clinical conversations often express temporally and causally related facts that entity-level extraction does not capture. Each session was processed in a single structured-output call that returned at most ten self-contained facts, preserving medica- tions, diagnoses, procedures, findings, temporal information, negation, and uncertainty. Extracted facts were cached by session and extraction model, so all questions and conditions referencing the same session used an identical semantic rep- resentation. D. Answer Generation For each strategy, the retrieved context and query were provided to the answer model in a single structured-output call, as illustrated in Fig. 3. The model returned a JSON object containing the generated answer and the identifiers of the supporting excerpts. This format was adopted after preliminary testing showed that separately generating and parsing cita- tions was less reliable. Before evaluation, cited identifiers not present in the context provided to the model were discarded, preventing unsupported or hallucinated citations from being credited. Experiments were repeated across multiple answer Question: During the hospitalization from 2155-12-13 to 2155-12-16, why was Coumadin held during the initial post-procedure period? Gold: Risk of bleeding from the tube site. Full Context Raw longitudinal history 2155-12-13:“We’reholdingyour Coumadin for now. The tube site could bleed...” [Earlier and later history] Output: Coumadin was held to prevent bleeding at the nephrostomy tube site. Recent Context Most recent raw history 2155-12-19:“Coumadinwas restarted...” 2155-12-13: “The tube site could bleed...” Output: Coumadin was held to prevent bleeding at the nephrostomy tube site. Episodic Retrieved encounter [e6] 2155-12-13 Patient: “I’m worried about being on Coumadin.” Doctor: “The tube site could bleed.” Output: Coumadin was held to prevent bleeding at the nephrostomy tube site. Semantic Retrieved facts [s111] Coumadin is temporarily held because of bleeding risk from the tube. Source: 2155-12-13 Output: Coumadin was held because of bleeding risk from the tube. Hybrid (Episodic + Semantic) Retrieved fact [s111] Coumadin is being held because of bleeding risk from the tube. Retrieved encounter [e6]Doctor:“We’reholdingyour Coumadin for now. The tube site could bleed.” Output: Coumadin was held due to the risk of bleeding from the nephrostomy tube. Fig. 3: Illustrative example of the five context strategies. models while keeping the memory construction, retrieval pro- cedure, prompts, and evaluation protocol fixed. E. Evaluation Evaluation is organized around the three research questions. All conditions are evaluated on the same sampled questions to enable paired comparison. a) RQ1 - Answer Correctness: The primary outcome is end-to-end answer accuracy. A judge LLM from a different model family than the answer model evaluates each generated response against the gold answer as correct or incorrect based on semantic equivalence rather than exact string match, following the LLM-as-judge protocol of Zheng et al. [15]. For context strategy m, accuracy is defined as Accuracy m = 1 N m N m X i=1 y im ,(2) where N m is the total number of benchmark questions evaluated under context strategy m, and y im = ( 1, if the generated answer is judged correct, 0, if the answer is judged incorrect or generation fails. (3) Accuracy is reported both overall and by question type to examine how the effect of context strategy varies across reasoning tasks. Generation failures therefore remain in the denominator and are counted as incorrect, making this an end- to-end measure of system performance. We use Gemma4-26B [16] as the primary correctness judge and MedGemma-27B [17] as a domain-specific secondary judge; neither belongs to the model family of any answer model. Both judges receive the question, gold answer, and generated answer and output a binary CORRECT/INCORRECT label. We report agreement between the two judges using raw agreement and Cohen’s κ. b) RQ2 - Robustness to Evidence Distance: We measure evidence distance by the location of the gold supporting evidence relative to the end of the longitudinal history as- sociated with each question. Specifically, the session gap d i captures how far back in the history the supporting evidence occurs. For descriptive analysis, questions are grouped into five predefined distance bins (0–2, 3–5, 6–10, 11–20, and 21+ sessions), and end-to-end answer accuracy is reported for each context strategy within each bin. To test whether the relationship between evidence distance and answer correctness differs across context strategies, we fit the logistic regression model logit[P (Y im = 1)] = β 0 + M X m=2 β m C im +β d z i + M X m=2 γ m (C im z i ), (4) where Y im = 1 indicates a correct answer for question i under context strategy m, C im denotes the context-strategy indicators, and z i is the standardized continuous session gap, z i = d i − ̄ d s d ,(5) where ̄ d and s d denote the mean and standard deviation of the raw session gaps, respectively. The interaction coefficients γ m test whether the effect of evidence distance on answer correctness differs between each strategy and the reference strategy. Because each benchmark question is evaluated under multiple context strategies, we compute cluster-robust standard errors at the question level. We use distance bins only for descriptive analysis and visualization; the regression retains session gap as a continuous variable. As a sensitivity analysis, we additionally report distance-stratified accuracy restricted to successful generations. c) RQ3 - Abstention: For adversarial questions, which are constructed such that the requested conclusion is unsup- ported by the patient history, we evaluate whether the gen- erated response appropriately rejects the unsupported premise or otherwise indicates that the question cannot be answered from the available evidence. Responses are evaluated against the benchmark’s unanswerable gold response using the same semantic correctness judge as RQ1. d) Supplementary Metrics: We compute BERTScore F1 [18] between generated and gold answers using roberta-large without baseline rescaling, excluding failed or empty generations; coverage is the percentage of questions with a scorable response. For evidence grounding, each cited excerpt or fact is mapped to its source session, and the cited sessions are compared with the gold sessions on answerable questions. Citation coverage is the percentage of questions with at least one valid citation. Precision, recall, and F1 are computed per question (0 when nothing is cited) and then averaged. F. Implementation All experiments were conducted locally on a workstation with an NVIDIA GeForce RTX 5080 GPU, 64 GB RAM, and a 24-core Intel Core Ultra 9 285K processor running Ubuntu 24.04.4 LTS. Models were served using Ollama 1 . We evaluate qwen3-8b [14], mistral-7b [19], llama3.2-3b [20], and glm4-9b [21] as answer models. Full Context uses each model’s available context window, whereas Recent Context and the retrieval-based conditions are constrained to a common 7,000-token evidence budget. Semantic memory is constructed using qwen3-4b and both episodic and semantic memories are embedded using all-MiniLM-L6-v2 [22], [23] and ranked using cosine similarity. Answer-generation prompts and decoding settings are held fixed across context strategies for each answer model, while retrieval and context construction vary according to the strategy being evaluated. V. RESULTS & DISCUSSION We report results across four answer models with different context capacities. Qwen and Mistral use 32k-token context windows, while LLaMA and GLM-4 use 128k-token windows. This difference primarily affects Full Context, for which longer histories may be truncated to the model’s context limit. Recent, Episodic, Semantic, and Hybrid instead operate under the same evidence budget across models. Full Context results should therefore be interpreted with these differences in context capacity in mind. A. Overall Effect of Context Strategy We first examine how context strategy affects overall answer accuracy on MedLoCoMo. Using Gemma4 as the primary correctness judge, Hybrid achieves the highest accuracy for Qwen (52.6%), LLaMA (46.0%), and GLM-4 (55.5%), while Episodic performs best for Mistral (62.0%), closely followed by Hybrid (61.4%) (Fig. 4a). Overall, Hybrid and Episodic provide the strongest aggregate performance, although their relative advantage varies across answer models. We repeat the evaluation using MedGemma as a domain-specific judge (Fig. 4b). The two judges agree on 78.7% of evaluations, with a Cohen’s κ of 0.575. MedGemma assigns correct judgments more frequently in disagreements (13.2%) than Gemma4 (8.1%), resulting in generally higher absolute accu- racy (Fig. 4c). Nevertheless, the main strategy-level patterns are preserved, with Hybrid and Episodic remaining among the strongest conditions across models. Mistral Full Context produced a valid answer for only 179 of 1,000 questions due to generation failures; on successful generations its accuracy was 55.3%, comparable to other strategies. As a complementary measure of answer similarity, BERTScore F1 varies only modestly across strategies, ranging from 0.83 to 0.87 (Fig. 4d). Episodic and Hybrid achieve the highest or near-highest scores across answer models, but the differences are substantially smaller than those observed in 1 https://ollama.com TABLE I: Session-level citation grounding across context strategies. Cov- erage denotes the percentage of evaluated responses containing at least one citation that could be mapped to a source session. StrategyCoverage (%)Precision (%)Recall (%)F1 (%) Full Context44.922.027.621.2 Recent Context63.213.511.111.2 Episodic87.548.750.046.1 Semantic79.035.740.534.8 Hybrid87.941.347.940.4 judged accuracy. For example, Qwen achieves BERTScore F1 values of 0.86 with Recent Context and 0.87 with Hybrid, despite judged accuracies of 31.6% and 52.6%, respectively. This indicates that semantic similarity alone does not capture the context-dependent differences in answer correctness iden- tified by the correctness judge. BERTScore coverage is high for most conditions, with Mistral Full Context as a notable exception (17.9%). To examine evidence grounding, we compare the evidence explicitly cited by the model with the annotated supporting sessions. Each cited excerpt or semantic fact is mapped to its source session and compared with the gold evidence-session identifiers. Table I reports citation coverage, precision, recall, and F1 for questions with annotated supporting evidence. Episodic achieves the highest citation precision (48.7%), recall (50.0%), and F1 (46.1%), while Hybrid achieves the highest ci- tation coverage (87.9%) and the second-highest recall (47.9%). Full and Recent Context show substantially lower citation precision and recall, indicating weaker alignment between cited and annotated evidence sessions. B. Performance Across Reasoning Tasks The effect of context strategy varies across reasoning tasks (Table IV). No single strategy achieves the highest accuracy across all question types and answer models. For care-plan rationale, Hybrid achieves the highest accuracy for Mistral (90.91%), GLM-4 (88.11%), and LLaMA (71.33%), while Episodic achieves the highest accuracy for Qwen (88.11%). The highest-performing strategy for cross-admission compari- son, frequency-pattern, and longitudinal-progression questions also varies across answer models. Performance also varies substantially on the two adversarial question types. We ex- amine this behavior separately in Section V-D as part of RQ3. Overall, the task-level results demonstrate that the relative effectiveness of context strategies varies across both reasoning tasks and answer models. C. Effect of Query-Evidence Distance To examine the effect of evidence distance, we evaluate accuracy across five session-gap bins: 0–2, 3–5, 6–10, 11– 20, and 21+ sessions. Figure 5 shows a pronounced decline for Recent Context from the 0–2 to the 21+ bin for Qwen (83.8% to 36.1%), LLaMA (70.3% to 31.7%), GLM-4 (83.8% to 39.8%), and Mistral (91.9% to 53.8%). At 21+ sessions, Hybrid achieves 61.1%, 50.3%, and 59.6% for Qwen, LLaMA, and GLM-4, respectively, while Episodic achieves 62.8% for Mistral. Hybrid (Episodic + Semantic) EpisodicSemanticRecent ContextFull Context 0.0 0.2 0.4 0.6 0.8 1.0 End-to-end judged accuracy 52.6% 46.0% 55.5% 61.4% 50.4% 44.6% 52.5% 62.0% 43.7% 37.4% 41.6% 58.9% 31.6% 31.2% 41.4% 57.5% 45.4% 40.9% 48.7% 9.9% Overall accuracy (Gemma4) Answer model Qwen LLaMA GLM-4 Mistral (a) Hybrid (Episodic + Semantic) EpisodicSemanticRecent ContextFull Context 0.0 0.2 0.4 0.6 0.8 1.0 End-to-end judged accuracy 60.9% 52.4% 60.1% 64.2% 59.3% 51.6% 59.5% 63.6% 50.5% 46.5% 50.1% 58.7% 39.0% 35.5% 45.3% 51.1% 51.2% 48.2% 55.9% 10.7% Overall accuracy (MedGemma) Answer model Qwen LLaMA GLM-4 Mistral (b) Overall agreement Cohen's MedGemma only Gemma4 only 20 40 60 80 100 Answer model Qwen LLaMA GLM-4 Mistral (c) Full Context Recent Context EpisodicSemanticHybrid Qwen Mistral LLaMA GLM-4 Answer model 0.86 87.8% cov. 0.86 100.0% cov. 0.87 100.0% cov. 0.86 90.3% cov. 0.87 100.0% cov. 0.83 17.9% cov. 0.84 100.0% cov. 0.85 100.0% cov. 0.84 97.1% cov. 0.85 99.9% cov. 0.85 99.4% cov. 0.85 97.7% cov. 0.86 98.9% cov. 0.86 95.6% cov. 0.86 97.7% cov. 0.86 100.0% cov. 0.85 100.0% cov. 0.86 99.9% cov. 0.85 91.2% cov. 0.86 100.0% cov. BERTScore F1 by Context Strategy (d) Fig. 4: Overall accuracy and judge sensitivity on MedLoCoMo. (a–b) End-to-end accuracy across context strategies and answer models using Gemma4 and MedGemma as correctness judges, respectively; error bars indicate 95% Wilson confidence intervals. (c) Agreement between the two correctness judges, including overall agreement, Cohen’s κ, and directional disagreements. (d) BERTScore F1 between generated and reference answers for the primary Gemma4- evaluated runs; percentages within cells indicate the proportion of generations evaluable by BERTScore. We further fit a logistic regression of answer correct- ness on standardized session gap, context strategy, and their interaction, with cluster-robust standard errors by question. Under Episodic (the reference strategy), increasing distance is associated with lower correctness for Qwen (β = −0.168, p = .030), Mistral (β = −0.199, p = .013), and GLM-4 (β = −0.200, p = .012), but not LLaMA (β = −0.084, p = .261). Most strategy-by-distance interactions are not significant; the exception is Recent Context for LLaMA, which declines more steeply than Episodic (β =−0.216, p = .028). As a sensitivity analysis, we repeated the distance-stratified evaluation using only successful generations. The decline in Recent Context persisted across all four models, and Episodic and Hybrid remained above Recent Context in the 21+ session bin, indicating that generation failures alone do not explain the observed distance effect. D. Abstention on Unanswerable Questions Context strategy also affects how models respond when the available patient history does not support the requested conclusion. As shown in Table V, averaged across the four evaluated models, accuracy on unanswerable questions is low- est for Full Context (14.8%) and highest for Semantic retrieval (33.5%). Hybrid achieves the highest accuracy on answerable questions (63.1%) while maintaining relatively high accuracy on unanswerable questions (30.9%). Episodic shows a similar pattern, with 61.9% and 28.7% accuracy on answerable and unanswerable questions, respectively. Performance on unanswerable questions varies across mod- els and context strategies. Semantic achieves the highest average unanswerable-question accuracy, but its answerable- question accuracy (50.2%) is lower than that of Hybrid and Episodic. Moreover, the strategy with the highest unanswerable-question accuracy differs across models. Over- TABLE IV: Accuracy (%) by context strategy and question type on the MedLoCoMo dataset across four answer models. Each model is evaluated on 1,000 questions: Cross-Admission Adversarial (CA-Adv;n = 143), Cross-Admission Comparison (CA-Comp;n = 143), Cross-Admission Frequency Pattern (CA-Freq;n = 143), Cross-Admission Longitudinal Progression (CA-Long;n = 143), Single-Admission Adversarial (SA-Adv;n = 143), Single-Admission Care-Plan Rationale (SA-Care;n = 143), and Single-Admission Medical Reasoning (SA-Med;n = 142). Best performance within each answer model and question type is shown in bold. ModelStrategyCA-AdvCA-CompCA-FreqCA-LongSA-AdvSA-CareSA-Med Qwen-3-8B Full Context6.9951.0541.9653.8526.5766.4371.13 Recent Context0.7045.4538.4631.4716.7846.1542.25 Episodic1.4055.2449.6551.7530.0788.1176.76 Semantic0.0048.9534.2751.0537.0672.7361.97 Hybrid (E+S)2.8056.6447.5560.1436.3686.7178.17 Mistral-7B Full Context32.1711.191.401.402.1016.084.93 Recent Context59.4453.1553.1557.3458.7467.1353.52 Episodic46.8562.2455.2453.1550.3583.9282.39 Semantic52.4559.4452.4548.9572.0367.1359.86 Hybrid (E+S)44.0661.5449.6555.9451.0590.9176.76 GLM-4-9B Full Context6.2958.7447.5555.9418.1881.8272.54 Recent Context42.6646.1534.2734.9724.4864.3442.96 Episodic27.9760.1444.7654.5518.1884.6277.46 Semantic25.1740.5633.5739.8627.2767.1357.75 Hybrid (E+S)30.7760.8448.9551.0530.7788.1178.17 LLaMA-3.2-3B Full Context7.6953.8540.5641.2618.1867.8357.04 Recent Context14.6931.4732.8732.8727.9744.0634.51 Episodic30.7748.9538.4638.4623.7866.4365.49 Semantic27.2737.7636.3638.4626.5747.5547.89 Hybrid (E+S)30.0746.8545.4541.9621.6871.3364.79 all, these results show that context strategy influences both reasoning when supporting evidence is available and absten- tion when the requested conclusion is unsupported. E. Cross-Benchmark Evaluation on LongMemEval To assess whether the effects of context strategy extend beyond longitudinal clinical histories, we additionally evaluate the five strategies on LongMemEval [4]. LongMemEval is a benchmark for evaluating long-term conversational memory in LLMs, requiring models to retrieve and reason over infor- mation distributed across extended, multi-session interaction histories. This provides a complementary general-domain set- ting to MedLoCoMo while preserving the core challenge of identifying relevant evidence from long histories. Figure 6 shows that context strategy continues to have a substantial effect on end-to-end accuracy, although the mag- nitude varies across models. Hybrid achieves 45.4%, 41.6%, 43.8%, and 57.6% for Qwen, LLaMA, GLM-4, and Mistral, respectively, while Episodic achieves 41.2%, 36.8%, 44.2%, and 61.8%. Semantic retrieval is particularly effective for Mistral (57.4%) but lower for the other models (27.4-31.0%). Recent Context performs poorly for Qwen (9.0%), LLaMA (7.6%), and GLM-4 (16.0%), whereas Mistral reaches 66.4%, illustrating substantial model-dependent variation. Full Con- text similarly varies across models, ranging from 19.2% for Mistral to 38.0% for GLM-4. Overall, the cross-benchmark results support the broader finding that context selection materially affects long-horizon reasoning, while also showing that no single strategy behaves uniformly across models and domains. F. Discussion Our results show that providing more longitudinal history does not necessarily improve clinical reasoning. Episodic and Hybrid generally outperform Recent and Full Context, despite Full Context providing access to more of the patient history. This suggests that selectively retrieving relevant evidence can be more effective than simply increasing the amount of context available to the answer model. This distinction becomes more apparent as evidence be- comes more distant. Recent Context performs better when relevant evidence is nearby but degrades as the session gap increases, whereas Episodic and Hybrid retain higher accuracy at long distances, although their distance slopes do not dif- fer significantly from other strategies. However, performance varies across reasoning tasks and answer models, indicating that the effectiveness of a context strategy depends on both the reasoning requirement and the underlying model. Context strategy also influences how models handle ques- tions with unsupported premises. Semantic achieves the high- est average accuracy on adversarial questions, while Hybrid combines the highest average accuracy on answerable ques- tions with relatively strong adversarial-question performance. The strategy with the highest adversarial-question accuracy nevertheless differs across models. Together, these findings 0-23-56-1011-2021+ 0.0 0.2 0.4 0.6 0.8 1.0 (a) Qwen 0-23-56-1011-2021+ (b) LLaMA 0-23-56-1011-2021+ (c) GLM-4 0-23-56-1011-2021+ (d) Mistral End-to-end accuracy Evidence distance (session-gap bin) Hybrid (Episodic + Semantic)EpisodicSemanticRecent ContextFull Context Fig. 5: Accuracy across evidence-distance bins for each context strategy and answer model. Evidence distance is measured as the session gap between the query and its supporting evidence. Points indicate accuracy within each distance bin, with error bars showing 95% Wilson confidence intervals. TABLE V: Accuracy (%) on answerable and adversarial questions. Adver- sarial questions contain unsupported premises or request conclusions that cannot be established from the available patient history. Mean denotes the average across the four answer models. Bold indicates the highest model- specific accuracy within each context strategy and question category. StrategyModelAnswerableAdversarial Full Qwen56.916.8 Mistral7.017.1 LLaMA52.112.9 GLM-463.312.2 Mean44.814.8 Recent Qwen40.88.7 Mistral56.959.1 LLaMA35.221.3 GLM-444.533.6 Mean44.330.7 Episodic Qwen64.315.7 Mistral67.448.6 LLaMA51.527.3 GLM-464.323.1 Mean61.928.7 Semantic Qwen53.818.5 Mistral57.662.2 LLaMA41.626.9 GLM-447.826.2 Mean50.233.5 Hybrid Qwen65.819.6 Mistral66.947.6 LLaMA54.125.9 GLM-465.430.8 Mean63.130.9 show that context strategy affects multiple dimensions of long- horizon reasoning: answer correctness, performance on distant evidence, and abstention when the available history does not support the requested conclusion. h Hybrid (Episodic + Semantic) EpisodicSemanticRecent ContextFull Context 0.0 0.2 0.4 0.6 0.8 1.0 End-to-end judged accuracy 45.4% 41.6% 43.8% 57.6% 41.2% 36.8% 44.2% 61.8% 27.4% 29.4% 31.0% 57.4% 9.0% 7.6% 16.0% 66.4% 21.6% 25.4% 38.0% 19.2% Answer model Qwen LLaMA GLM-4 Mistral Fig. 6: End-to-end judged accuracy on LongMemEval across context strategies and answer models. Error bars indicate 95% Wilson confidence intervals. VI. CONCLUSION In this study, we investigated how context strategy af- fects long-horizon clinical reasoning by comparing Full, Re- cent, Episodic, Semantic, and Hybrid context across multiple open-weight language models. Episodic and Hybrid gener- ally achieve stronger answer accuracy, including on distant evidence, while performance on adversarial questions reveals that successful reasoning when evidence is available does not necessarily translate to successful abstention when it is not. Overall, the results demonstrate that reliable longitudinal reasoning depends not only on how much historical infor- mation is available, but also on how relevant evidence is selected and presented to the model. This study has several limitations. We evaluate a fixed set of context strategies and retrieval settings (a single embedding model and a 7,000-token evidence budget) with four small open-weight models (3-9B parameters) served locally, so results may differ for larger models or other configurations. Semantic facts were extracted with Qwen3-4B, which shares a model family with one answer model. MedLoCoMo histories are synthetic dialogues derived from MIMIC-IV records and may not reflect the noise and heterogeneity of real clinical documentation. Finally, correct- ness was assessed by LLM judges whose agreement was moderate and was not validated against clinician annotations. Future work will investigate adaptive approaches that select or combine context strategies based on the query and reasoning requirements, and evaluate their reliability in more realistic clinical settings. ACKNOWLEDGMENT The research reported in this publication was supported by the Predictive Analytics and Technology Integration (PATENT) Laboratory in the Department of Computer Science at The University of Alabama. REFERENCES [1] Z. Zhang et al., “MedLoCoMo: A long-context multi-session medi- cal dialogue benchmark for large language models,” arXiv preprint arXiv:2607.22566, 2026. [2] H. Hu et al., “Synthesis and evaluation of long-term history-aware medical dialogue,” in Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2026. [3] Y. Fan, H. Sun, K. Xue, X. Zhang, S. Zhang, and T. Ruan, “MedOdyssey: A medical domain benchmark for long context evaluation up to 200k tokens,” in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, p. 32–56. [4] D. Wu et al., “LongMemEval: Benchmarking long-context LLMs and memory systems over multi-session conversations,” in International Conference on Learning Representations (ICLR), 2025. [5] T. Sumers, S. Yao, K. Narasimhan, and T. Griffiths, “Cognitive architec- tures for language agents,” Transactions on Machine Learning Research, 2024. [6] Y. Chen, Z. Xu, X. Zhang, Z. Fan, W. Zhai, H. Xu, and Z. Liu, “LongMedBench: Benchmarking medical agents for long-horizon clin- ical decision-making,” arXiv preprint arXiv:2607.09322, 2026. [7] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, p. 157–173, 2024. [8] W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang, “MemoryBank: En- hancing large language models with long-term memory,” in Proceedings of the AAAI conference on artificial intelligence, vol. 38, no. 17, 2024, p. 19 724–19 731. [9] C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez, “MemGPT: Towards LLMs as operating systems,” arXiv preprint arXiv:2310.08560, 2023. [10] R. Wang, H. Lu, S. Liu, L. Dong, and J. Zhu, “AdMem: Advanced memory for task-solving agents,” arXiv preprint arXiv:2606.06787, 2026. [11] J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023. [12] A. Terranova, B. Ross, and A. Birch, “Evaluating long-term memory for long-context question answering,” arXiv preprint arXiv:2510.23730, 2025. [13] X. Deng, R. Zhong, H. Peng, X. Lu, Y. Wu, G. Li, B. Xu, Y. Yao, J. Fang, H. Cao et al., “MemTrace: Tracing and attributing errors in large language model memory systems,” arXiv preprint arXiv:2605.28732, 2026. [14] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, D. Li, D. Liu, F. Huang, H. Wei et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [15] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, p. 46 595–46 623. [16] GemmaTeam,“Gemma4technicalreport,”arXivpreprint arXiv:2607.02770, 2026. [17] A. Sellergren et al., “MedGemma technical report,” arXiv preprint arXiv:2507.05201, 2025. [18] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating text generation with BERT,” in International Conference on Learning Representations (ICLR), 2020. [Online]. Available: https://openreview.net/forum?id=S1x6ueSKPr [19] A. Q. Jiang, A. Sablayrolles, A. Mensch et al., “Mistral 7B,” arXiv preprint arXiv:2310.06825, 2023. [20] A. Grattafiori, A. Dubey, A. Jauhri et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [21] Team GLM, “ChatGLM: A family of large language models from GLM- 130B to GLM-4 all tools,” arXiv preprint arXiv:2406.12793, 2024. [22] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “MiniLM: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, p. 5776–5788. [23] N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019, p. 3982–3992.