Paper deep dive
CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives
Aiwei Ivy Zhang, Nimra Ishfaq, Mohit Chandra, Santiago Alvarez Lesmes, Adam Coscia, Khatiya Chelidze Moon, Xiaohan Ding, Munmun De Choudhury
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/18/2026, 4:25:24 AM
Summary
The paper introduces CliniCIRCA, a modular three-stage Large Language Model (LLM) framework designed to reconstruct longitudinal mental health patient journeys from unstructured Electronic Health Record (EHR) discharge summaries. The framework performs event extraction, temporal anchoring without explicit timestamps, and summarization. Using the MIMIC-III dataset, the authors demonstrate that CliniCIRCA can generate verified gold-standard timelines and silver-standard training data, which improve the performance of open-weight models through instruction tuning.
Entities (6)
Relation Signals (5)
CliniCIRCA â processes â Discharge Summaries
confidence 95% · We present CliniCIRCA... for... Reconstruction of Clinical Annals... from Raw EHR Narratives
CliniCIRCA â uses â MIMIC-III
confidence 95% · From 14,882 MIMIC-III mental health admissions, we first construct a benchmark...
Gemini 2.5 Pro â selectedfor â CliniCIRCA
confidence 90% · We select based on Stage 2 alone and then freeze Gemini 2.5 Pro as the only agent advancing to Stage 3.
CliniCIRCA â targets â Mental Health
confidence 90% · We specifically focus on bridging the gap in mental health care...
Instruction Tuning â improves â Open-weight Models
confidence 85% · Compared with zero- and few-shot prompting, instruction tuning generally improves five open-weight models...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In mental health care, reasoning over patient journeys is a key task for clinicians. Yet these journeys, encompassing a longitudinal progression of biological, psychological, and social events, are often spread across disparate unstructured text narratives, making temporal recovery challenging. We present CliniCIRCA, a multi-stage LLM framework for Calendar-anchored, Imprecision-aware Reconstruction of Clinical Annals. To our knowledge, CliniCIRCA is the first to temporally classify clinical events across unstructured discharge summaries without event-level timestamps. From 14,882 MIMIC-III mental health admissions, we first construct a benchmark of 52 discharge summaries on which CliniCIRCA produces 15,891 temporally tagged events. After correcting 629 errors based on a clinician-in-the-loop evaluation, we produce verified gold-standard labels. Finally, the corrected timelines drive a temporally grounded summarization stage that compresses each source 1.52 times into a date-grouped chronological record. We then scale the framework to generate 1,000 silver-standard timelines and evaluate them as training data. Compared with zero- and few-shot prompting, instruction tuning generally improves five open-weight models on event extraction, temporal tagging, and summarization across silver and clinician-verified evaluations.
Tags
Links
- Source: https://arxiv.org/abs/2609.19585v1
- Canonical: https://arxiv.org/abs/2609.19585v1
Trouble viewing inline? Open PDF directly â
Full Text
169,933 characters extracted from source content.
Expand or collapse full text
CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives Aiwei Ivy Zhang Nimra Ishfaq Mohit Chandra Santiago Alvarez Lesmes Affiliation: Georgia Institute of Technology University of Texas at Austin Northwell Health Adam Coscia Khatiya Chelidze Moon Affiliation: Georgia Institute of Technology University of Texas at Austin Northwell Health Xiaohan Ding Munmun De Choudhury Abstract In mental health care, reasoning over patient journeys is a key task for clinicians. Yet these journeys, encompassing a longitudinal progression of biological, psychological, and social events, are often spread across disparate unstructured text narratives, making temporal recovery challenging. We present CliniCIRCA, a multi-stage LLM framework for Calendar-anchored, Imprecision-aware Reconstruction of Clinical Annals. To our knowledge, CliniCIRCA is the first to temporally classify clinical events across unstructured discharge summaries without event-level timestamps. From 14,88214,882 MIMIC-I mental health admissions, we first construct a benchmark of 5252 discharge summaries on which CliniCIRCA produces 15,89115,891 temporally tagged events. After correcting 629629 errors based on a clinician-in-the-loop evaluation, we produce verified gold-standard labels. Finally, the corrected timelines drive a temporally grounded summarization stage that compresses each source 1.521.52 times into a date-grouped chronological record. We then scale the framework to generate 1,0001,000 silver-standard timelines and evaluate them as training data. Compared with zero- and few-shot prompting, instruction tuning generally improves five open-weight models on event extraction, temporal tagging, and summarization across silver and clinician-verified evaluations. 1 Introduction Every patient follows a clinical journey: a longitudinal progression of symptoms, diagnoses, treatments, life events, and responses to care that demonstrate how their health has evolved over time Kraljevic et al. (2024); Ruan et al. (2019). Clinicians draw on this progression to interpret a patientâs current presentation and guide treatment. Unfortunately, these journeys are often not documented as explicit timelines Loftus et al. (2024); Linhares et al. (2023); Olex and McInnes (2021). For example, discharge summaries are a rich source of longitudinal patient information. They synthesize prior history, hospital course, and clinical outcomes in free-form text, where events may be repeated, narrated nonsequentially, or linked only through relative temporal expressions and surrounding context K et al. (2025); Seinen et al. (2025); Lee et al. (2024); Kim et al. (2024). Recovering temporal structure is thus a central challenge in clinical NLP for modeling patient journeys from clinical text Amirahmadi et al. (2025); Makarov et al. (2025). Patient journeys are often spread across disparate text sources, limiting the use of these narratives by computational systems that require structured longitudinal representations Chaturvedi (2024); Moharasan and Ho (2019). As a result, information fragmentation shifts the burden of reconstruction to clinicians and computational systems. Clinicians must piece together the patientâs course under time and information constraints, while models must infer temporal relations that remain implicit Linhares et al. (2023); Gao et al. (2022); Olex and McInnes (2021). Figure 1: Methodology Overview Prior work treated event reconstruction as a set of separable subtasks, which makes temporal alignment challenging. For example, clinical summarization compresses lengthy records into coherent accounts, but often under-preserves temporality and rarely represents complex or uncertain event chronology explicitly Kruse et al. (2025); Cui et al. (2025); Croxford et al. (2025). Temporal information extraction preserves finer structure by extracting time expressions, events, and temporal relations, but most systems are built around predefined annotation schemas and pairwise relation classification rather than recovery of an open-ended patient trajectory Chaturvedi et al. (2025); Gumiel et al. (2022); Alfattni et al. (2020). Recent LLM timeline generation comes closer to recovering full patient trajectories Kumar et al. (2026); Noroozizadeh et al. (2025); Wang and Weiss (2025); Wang et al. (2024), but often relies on structured EHR timestamps, preselected note fragments, or curated case reports, and commonly represents time using a single relative offset. What remains missing is a unified, text-only reconstruction formulation that preserves event breadth, calendar timing, temporal uncertainty, and patient-level coherence. We specifically focus on bridging the gap in mental health care, where a patientâs course cannot be understood only from diagnoses and procedures alone. Engelâs framework emphasizes that illness emerges through the interaction of biological, psychological, and social factors Engel (1977); in mental health, these factors are often expressed through medication use, trauma, social history, changes in daily functioning, etc Saha et al. (2026). Such information is part of the trajectory rather than peripheral context, yet the combination of events differs across patients. Mental health records therefore expose the limitations of fixed schemas: a narrow schema would omit events that may define an individual cause, while an overly exhaustive one quickly becomes impractical. Hence, we formulate patient-journey recovery as open-vocabulary, calendar-anchored, and uncertainty-aware temporal reconstruction from raw clinical narratives. Specifically, journeys are reconstructed within a single discharge summary corresponding to one hospital admission, as a step toward multi-encounter longitudinal modeling. We introduce CliniCIRCA (Figure 1), a three-stage LLM framework to convert discharge summaries to coherent temporal clinical trajectories. CliniCIRCA produces two linked representations: Output A, an inspectable event-level timeline for computational use and auditing, and Output B, a clinician-readable account derived from the same temporal substrate. Our experiments demonstrate that patient journeys produced by a high-capacity model (Section 6) can provide effective supervision for substantially smaller models, particularly for atomic clinical event extraction, while temporal reconstruction remains reasoning-intensive. By transforming fragmented narratives into linked structured and readable representations, CliniCIRCA could support longitudinal chart review, clinical handoffs, and follow-up care while preserving an auditable connection to the underlying events. We have provided code in a repository here11 1 https://anonymous.4open.science/r/CliniCIRCA-E5F/. 2 Dataset We use MIMIC-I v1.4 Johnson et al. (2016); Johnson et al. (2015), a publicly available, de-identified critical-care database from the Beth Israel Deaconess Medical Center (2001â2012) in English. CliniCIRCA receives unprocessed discharge summaries without chunking, section parsing, or normalization. Structured EHR fields are used to select the cohort and provide three temporal anchors: (1) date of birth, (2) admission date, and (3) discharge date. We define the cohort at the admission level: an admission is retained if it contains at least one psychiatric diagnosis from the ICD-9 Mental Disorders chapter (codes 290â319) and can be paired with a discharge summary. This produces 14,88214,882 summaries from 12,27312,273 patients, each summary treated as a separate reconstruction instance. Because admissions are flagged by the presence of any Mental Disorders diagnosis code rather than by principal diagnosis, the cohort includes both primary psychiatric admissions and medical/surgical admissions with psychiatric comorbidity. More details can be found in Appendix A. 3 The CliniCIRCA Framework CliniCIRCA is a sequential framework comprising three LLM stages (Figure 1), each receiving the output of the previous stage and having a single main task. Stage 1 reads the raw discharge summary (obtained from the MIMIC-I Dataset) without preprocessing and extracts a high-recall list of atomic clinical events. Stage 2 resolves the timing of each event, tagging it with an ISO date and a certainty label anchored against temporal references; its output is Output A, a timeline of (ISO date and certainty tag, atomic event) pairs. Stage 3 aggregates these tagged events by date into Output B, a date-grouped chronological summary. Decomposing the task in this way allows each stage to be prompted, evaluated, and improved independently, and confines each temporal decision to the stage best positioned to make it. 4 Model and Agent Evaluation Setup 4.1 Candidate Pool For CliniCIRCA, we select 1111 LLM candidates based on reported performance across general-purpose and clinical benchmarks Yu et al. (2025); Sarvari and Al-fagih (2025); Shi et al. (2024). For proprietary models, we evaluate Gemini 2.5 Pro and 2.5 Flash Comanici et al. (2025); Claude Sonnet 4.6 Anthropic (2026) and Haiku 4.5 Anthropic (2025). For open-weight models, we evaluate Llama 3.3 70B Meta AI (2024), Qwen 3.6 35B-A3B Qwen Team (2026), Gemma 4 26B-A4B Gemma Team (2026), Mistral Small 3.2 24B-A4B Mistral AI (2025), Phi-4 Mini 3.8B Abouelenin and others (2025), and medical-domain LLMs - MedGemma 27B Sellergren et al. (2026) and Meditron 3 Qwen 2.5 7B OpenMeditron (2025). Model specifications, generation settings, and computational details are provided in Appendix B. 4.2 Implementation Setup We construct a benchmark set of 5252 MIMIC-I discharge summaries, averaging 1,7741,774 words each (range: 437437â3,5833,583; Appendix A.2). Of the 52 cases, 29 had a principal psychiatric diagnosis; the remaining 23 were medical or surgical admissions with psychiatric comorbidity. For the benchmark, every candidate is run through Stages 1-2, with no mixing of models across stages. The size is determined by a power analysis (Appendix C) for a pairwise Ï2Ï^2 comparison (large effect, α = 0.050.05, 9595% power), yielding N = 5252 needed to reach statistical significance Ki et al. (2025); Card et al. (2020); Dror et al. (2018). Each candidate is run through Stages 1-2 as a fixed pipeline configuration, producing 572572 modelâdocument outputs evaluated using the clinician-authored rubric described in Section 5.1. We retain the selected model for Stage 3 rather than optimizing a separate model for each stage. This avoids a combinatorial search over model combinations, reveals how one modelâs capabilities propagate through the CliniCIRCA framework, and simplifies reproducibility and deployment governance Umeton et al. (2024); de Hond et al. (2024); MeskĂł and Topol (2023). We select based on Stage 2 because the summary stage depends on the accuracy of its event-level temporal input. Following prior health-domain work that validates LLM judges against expert-vetted human labels Mittal et al. (2026), two human annotators and two clinicians, all with relevant annotation experience and high proficiency in English, form a feedback loop to evaluate the results. We establish human-evaluator agreement on both Stage-2 and 3 outputs. 5 Constructing Patient Timelines from Discharge Summaries 5.1 Stage 1 & 2: From Raw Discharge Summary to Tagged Timeline Faithful clinical timeline recovery requires both comprehensive event extraction and accurate temporal anchoring, yet extraction errors can propagate through temporal reasoning and timeline construction Gumiel et al. (2022); Alfattni et al. (2020). We therefore decompose the task into Stage 1 for event extraction and Stage 2 for temporal anchoring, allowing each component to be independently specified and evaluated. Full generation prompts can be found in Appendix D Tables A5 - A6. Model Dimension A Dimension B Dimension C Avg. ABC MED-P MED-R MED-F1 Gemini 2.5 Pro 5.00 5.00 4.83 4.94 0.6370.637 0.812 0.710 Claude Haiku 4.5 5.00 4.98ÂŻ 4.98 4.81ÂŻ 4.81 4.93ÂŻ 4.93 0.6100.610 0.7530.753 0.6720.672 Claude Sonnet 4.6 4.98ÂŻ 4.98 5.00 4.774.77 4.924.92 0.6140.614 0.7350.735 0.6660.666 Gemini 2.5 Flash 4.98ÂŻ 4.98 5.00 4.504.50 4.834.83 0.6170.617 0.762ÂŻ 0.762 0.6790.679 Qwen 3.6 35B-A3B 5.00 5.00 4.234.23 4.744.74 0.6300.630 0.7580.758 0.685ÂŻ 0.685 Gemma 4 26B-A4B 4.884.88 4.964.96 4.274.27 4.714.71 0.6360.636 0.7130.713 0.6690.669 Mistral Small 3.2 24B-A4B 4.964.96 4.98ÂŻ 4.98 3.883.88 4.614.61 0.6360.636 0.7400.740 0.6810.681 Llama 3.3 70B 4.774.77 4.904.90 3.443.44 4.374.37 0.6260.626 0.6730.673 0.6440.644 MedGemma 27B 4.98ÂŻ 4.98 4.98ÂŻ 4.98 2.672.67 4.214.21 0.6380.638 0.7290.729 0.6780.678 Meditron 3 (Qwen 2.5 7B) 3.713.71 4.734.73 2.002.00 3.483.48 0.680ÂŻ 0.680 0.2850.285 0.3610.361 Phi-4 Mini 3.8B 3.903.90 4.174.17 2.192.19 3.423.42 0.685 0.4610.461 0.5250.525 Table 1: Stage-2 model-selection results on the 52-sample benchmark. The highlighted row indicates the model selected for the final pipeline. Results are averaged over 52 samples. Dimension A: Completeness; Dimension B: Semantic Faithfulness; and Dimension C: Temporal Tag Accuracy. Dimensions A to C are Likert-scale averages reported to two decimal places. The best-performing value for each metric is shown in bold, and the second-best value is underlined. Tied values receive the same formatting. Inclusive extraction and anchor-based tagging. Stage 1 reads the raw discharge summary, applying no chunking, section parsing, or normalization, and extracts events inclusively and faithfully. Accordingly, Stage 2 then resolves each event against the admission date, discharge date, and date of birth, assigning exactly one of four temporal categories: 1)[DATE/RANGE] [EXACT]: a date written in the event text itself. 2) [DATE/RANGE] [APPROX]: no written date, but the event resolves to an anchor or a relative offset from one. 3) [PRE ADM]: framed as prior to or continuous through this admission (history, âstatus post,â demographics, undated chronic states), and 4) [INDETERMINATE]: a guardrail used only when none of the above applies and the timing genuinely cannot be determined. As specified above, to maximize precision, only explicitly written dates receive [EXACT]; any inferred or relative date resolves to [APPROX] at the coarsest defensible granularity (e.g., year-only when only a year is recoverable). Stage 2âs output is a list of temporally classified clinical events. Operationalizing the notion of âfaithfulnessâ. Because no reference timeline exists against which these tuples could be scored, a âgoodâ timeline must itself be operationalized. We define three clinician-informed criteria for evaluating the Stage-2 output (full definitions in Appendix E). A) Completeness: whether all clinically relevant events in the note are captured. B) Semantic faithfulness: whether each extracted event preserves the meaning of the source text. C) Temporal classification accuracy: whether the assigned date and tag are correct. Calibrating the automated judge. Following established practice in LLM evaluation to use a strong later-generation model Zheng et al. (2023), we select Gemini 3.1 Pro as the LLM evaluator, chosen to be at least as capable as any candidate it scores. Prior to selection, the evaluator is calibrated against expert human scoring. Only the calibrated LLM evaluator is used in the selection and scaling. Selecting the agent model. Following the agent selection mechanism as described in Section 4.2, Gemini 2.5 Pro is selected as the strongest configuration, ranking among the top candidates on every dimension and achieving 5.005.00 on Completeness, 5.005.00 on Semantic Faithfulness, and 4.834.83 on Temporal Tag Accuracy in Table 1. As a judge-independent check, we also report MEDCON Yim et al. (2023), the F1 overlap of UMLS medical concepts between each modelâs extracted events and the source note; once again, Gemini 2.5 Pro attains the highest MEDCON F1 (Table 1), corroborating its judge-scored ranking. We select based on Stage 2 alone and then freeze Gemini 2.5 Pro as the only agent advancing to Stage 3. 5.2 Validating the Stage 1-2 Timeline We validate the output of Stage 1-2 through a human review with two psychiatric clinicians during 6 iterative annotation sessions. The human review process was used both to evaluate the automated judge and to correct all confirmed errors, providing a clinician-verified gold reference dataset (N=52N=52). Evaluation was conducted on an internally hosted, custom web-based secure platform, demonstrated in Appendix F. We compare the calibrated Gemini 3.1 Pro evaluator with the clinician-verified gold annotation (see Appendix E for complete rubrics). Overall agreement is high, with Cohenâs Îș=0.655Îș=0.655. However, because only 3.96%3.96\% of events were labeled as errors, class imbalance likely depresses Îș Gwet (2008). This is reflected in a Gwetâs AC1 of 0.9720.972 and negative agreement of 0.9860.986, indicating strong agreement on events judged correct. Agreement is lower for the rare error class, with positive agreement of 0.6690.669. 5.3 Stage 3: From Timeline to Temporally-Guided Summary Next, Stage 3 tests if the verified event timeline can be rendered as a faithful, temporally grounded summary without re-reading the discharge note or reconstructing its chronology. Two clinicians defined a specification for the mental-health cohort that retains demographics and social history, reason for admission, key in-stay events, medication changes, discharge, and follow-up. Each event is grouped under its Stage 2 temporal label. Stage 3 is evaluated for faithfulness to the verified timeline and temporal accuracy using a separately calibrated Gemini 3.1 Pro evaluator validated against clinician judgments. Generation prompts and rubrics appear in Appendices H - I. Across the benchmark, Stage 3 reduces reading burden by roughly a third (31.5% mean and 33.8% median per-note; 34.0% corpus-wide), a stable 1.52 times compression that removes ⌠600 words per admission (Table 2). Metric Value Mean reduction (avg. per-note %) 31.53% Median per-note reduction 33.76% Corpus-level reduction (total words) 34.01% Compression factor 1.52Ă Mean words reduced per admission 603 Median words reduced 422 Table 2: Reading-burden reduction from Stage 3 summarization over the 52-note benchmark. Per-note percentages are averaged across notes; corpus-level reduction pools word counts across all notes. 5.4 Validating the Stage 3 Summary We compare the judgeâs error counts against human adjudication across 5252 summaries (Table A21). The judge flags 262262 errors, of which human annotators uphold 3939: a 6.76.7-fold inflation, averaging 4.34.3 spurious errors per summary. Agreement is substantial on whether a summary contains any error (Îș=0.77Îș=0.77, positive agreement = 0.84) and on the relative ordering of summaries by error count (Spearman Ï = 0.770.77). With clinician guidance, we develop a two-dimensional rubric (Tables A16-A17) to validate if generated summaries: (A) faithfully retained relevant mental-health and social events without distortion and (B) placed them accurately in time. See Appendix K for full metrics and results, and Appendix L for the internally hosted, custom web-based evaluation platform. 5.5 Qualitative Analysis We qualitatively investigate what this reconstructed timeline helps a reader (e.g., a clinician) understand. To examine the clinical content captured by the timelines, we analyze three clinician-verified gold cases using two World Health Organization frameworks: eight categories of mental disorders and associated symptoms World Health Organization (2025a), and three levels of risk factors: Individual, Family & Community, and Structural World Health Organization (2025b). We label a risk factor as co-documented rather than causal unless the record explicitly links it to the disorder. Across three exemplar cases, temporal reconstruction distinguishes long-standing risk factors from acute in-stay changes and diagnostic reassessment, while preserving whether their relationships are causal, co-documented, or unresolved rather than collapsing them into a single causal narrative. Full case analyses are provided in Appendix M. 6 Evaluation with Open-Weight Models Silverâ Silverâ Gold Model Zero Few IT Zero Few IT Zero Few IT Gemma4-31B .640.011.640_.011 .665.016.665_.016 .840.008.840_.008 .639.012.639_.012 .659.017.659_.017 .827.014.827_.014 â .727.013.727_.013 .628.016.628_.016 Qwen3.6-35B-A3B .536.014.536_.014 .642.013.642_.013 .839.009.839_.009 .535.021.535_.021 .647.017.647_.017 .834.012.834_.012 â .732.012.732_.012 .550.026.550_.026 Qwen3-30B-A3B .420.027.420_.027 .547.025.547_.025 .745.028.745_.028 .454.028.454_.028 .561.020.561_.020 .795.013.795_.013 â .279.045.279_.045 .606.033.606_.033 Mistral-24B .309.021.309_.021 .565.016.565_.016 .848.008.848_.008 .284.021.284_.021 .545.032.545_.032 .847.009.847_.009 â .676.015.676_.015 .715.028.715_.028 Meditron-7B .040.006.040_.006 .131.013.131_.013 .744.021.744_.021 .032.007.032_.007 .120.015.120_.015 .752.021.752_.021 â .034.004.034_.004 .331.024.331_.024 Table 3: Task A measured by embedding-based RM-F1@0.85. Silverâ trains and evaluates on the silver split; Silverâ trains on silver and evaluates on all 5252 clinician-verified examples; Gold uses four-fold cross-validation on the gold set alone. Results are point estimates ± bootstrap SD. Silverâ Silverâ Gold Model Zero Few IT Zero Few IT Zero Few IT Gemma4-31B .738.019.738_.019 .731.019.731_.019 .767.024.767_.024 .799.036.799_.036 .804.043.804_.043 .749.029.749_.029 â .668.029.668_.029 .721.035.721_.035 Qwen3.6-35B-A3B .635.021.635_.021 .687.020.687_.020 .742.019.742_.019 .567.020.567_.020 .715.025.715_.025 .731.022.731_.022 â .602.023.602_.023 .633.019.633_.019 Qwen3-30B-A3B .426.017.426_.017 .609.021.609_.021 .695.016.695_.016 .452.019.452_.019 .628.031.628_.031 .658.011.658_.011 â .231.038.231_.038 .425.020.425_.020 Mistral-24B .573.013.573_.013 .638.024.638_.024 .733.025.733_.025 .567.040.567_.040 .615.023.615_.023 .734.018.734_.018 â .620.012.620_.012 .680.007.680_.007 Meditron-7B .266.016.266_.016 .004.003.004_.003 .401.035.401_.035 .256.018.256_.018 .004.004.004_.004 .666.048.666_.048 â .005.004.005_.004 .359.022.359_.022 Table 4: Task B temporal tagging, scored by macro-averaged Fâ1F1 over the four temporal tags with the atomic events supplied as oracle input. Results are point estimates ± bootstrap SD. Silverâ Silverâ Gold Model Zero Few IT Zero Few IT Zero Few IT Gemma4-31B .671.011.671_.011 .647.011.647_.011 .792.010.792_.010 .636.015.636_.015 .616.019.616_.019 .749.017.749_.017 â .570.026.570_.026 .609.028.609_.028 Qwen3.6-35B-A3B .538.011.538_.011 .584.011.584_.011 .779.010.779_.010 .490.016.490_.016 .540.017.540_.017 .754.015.754_.015 â .497.029.497_.029 .516.032.516_.032 Qwen3-30B-A3B .504.012.504_.012 .532.012.532_.012 .705.014.705_.014 .474.015.474_.015 .522.015.522_.015 .674.021.674_.021 â .108.032.108_.032 .214.030.214_.030 Mistral-24B .513.012.513_.012 .538.012.538_.012 .770.011.770_.011 .450.019.450_.019 .486.017.486_.017 .738.017.738_.017 â .484.019.484_.019 .616.025.616_.025 Meditron-7B .361.016.361_.016 .328.023.328_.023 .690.018.690_.018 .293.021.293_.021 .244.031.244_.031 .663.021.663_.021 â .101.018.101_.018 .159.020.159_.020 Table 5: Task C timeline summarization measured by ROUGE-L. Models receive temporally tagged events and the source discharge summary as oracle input. Results are point estimates ± bootstrap SD. We evaluate whether CliniCIRCAâs intermediate outputs can serve as reusable supervision under three training and evaluation settings. Silverâ measures how well a model fits pipeline-generated supervision. We sample 1,0001,000 admissions disjoint from the 5252 gold examples, label them with the frozen Stage 1â2 pipeline, and split them into 750750 training, 100100 validation, and 150150 held-out test documents. This Stage 1 to 3 generation processed 61.8461.84M input and output tokens in total, averaging 61,83661,836 tokens per admission across the complete framework (Appendix A.3). Silverâ measures whether that supervision transfers to human-verified annotations: models trained on the same 750750 silver documents are evaluated on all 5252 gold examples. Gold measures what a small clinician-verified set alone can support. Because 5252 examples cannot sustain a stable held-out split, we run four-fold cross-validation and pool the out-of-fold predictions before computing metrics and bootstrap intervals. Throughout, the gold examples are excluded from silver labeling, silver-trained adapters, and few-shot exemplar selection to prevent unintentional leakage. Under the Gold setting, each document is evaluated only by the fold-specific model for which it was held out. 6.1 Experimental Setup Models. We evaluate five locally deployable open-weight models spanning dense and mixture-of-experts architectures, 77Bâ3535B parameter scales, and general-purpose versus clinical specialization: Gemma4-31B Gemma Team (2026), Mistral-24B Mistral AI (2025), Qwen3.6-35B-A3B Qwen Team (2026), Qwen3-30B-A3B Qwen Team (2025), and Meditron-7B OpenMeditron (2025). Qwen3-30B-A3B and Gemma4-31B were not evaluated during Stage 2 model selection, allowing us to test whether observed performance extends beyond the models previously selected. Prompting and tuning. For each task, we compare three settings under identical task prompts: zero-shot prompting, few-shot prompting with three training exemplars, and LoRA-based instruction tuning with one adapter per model and task. Instruction tuning is intended to align models to the expected output format and decision rules rather than to inject new clinical knowledge. Metrics. We evaluate Task A using relaxed-match F1 with a cosine similarity threshold of 0.850.85 (RM-F1@0.85) based on prior research Sharif et al. (2025); Zhang et al. (2019). Task B uses macro-F1 over four temporal categories, with a prediction considered correct when both the temporal label and resolved ISO date or range match the reference. Task C is evaluated using ROUGE-L; ROUGE-1 and ROUGE-2 are reported in Appendix N. Statistical comparison. We report point estimates on the full evaluation set with standard deviations from 2,0002,000 document-level bootstrap resamples. For comparisons on the same evaluation set, resamples are paired across systems, and differences are assessed using 95%95\% confidence intervals for the paired score difference. We describe a difference as reliable when its confidence interval excludes zero; otherwise, we make no claim that one system outperforms the other. 6.2 Results We examine if silver supervision improves open-weight models, transfers to clinician-verified annotations, and provides greater utility than the gold set alone. Note that zero-shot prompting requires no training fold, so its GOLD-condition performance is identical to the zero-shot SILVERâ condition; we therefore omit the redundant GOLD zero-shot column in Tables 3-5. Unless otherwise noted, comparisons below describe point-estimate rankings; we flag a difference as statistically reliable only where the paired bootstrap confidence interval excludes zero, and treat other orderings as suggestive rather than established. Task A: Atomic event extraction. Under Silverâ , instruction tuning yields higher RM-F1@0.85 scores than zero-shot and few-shot prompting for all five models (Table 3). Mistral-24B achieves the highest point estimate at 0.8470.847, compared with 0.2840.284 zero-shot and 0.5450.545 few-shot. Silver-trained models also retain similar point estimates when evaluation shifts from the silver test set to clinician-verified annotations. For example, Mistral-24B scores 0.8480.848 under Silverâ and 0.8470.847 under Silverâ . These results show that benefits of silver supervision extend beyond evaluation against the generated labels. Task B: Temporal tagging. Temporal tagging shows a less uniform effect of instruction tuning (Table 4). Under Silverâ , instruction tuning yields higher scores than both prompting baselines for four of the five models. Mistral-24B and Qwen3.6-35B-A3B reach 0.7340.734 and 0.7310.731, respectively. Gemma4-31B is the exception, achieving its highest score with few-shot prompting (0.8040.804), compared with 0.7990.799 zero-shot and 0.7490.749 after instruction tuning. Task C: Timeline summarization. Under Silverâ , instruction tuning yields higher ROUGE-L scores than both prompting baselines for all five models (Table 5). Qwen3.6-35B-A3B achieves the highest point estimate at 0.7540.754, followed by Gemma4-31B at 0.7490.749 and Mistral-24B at 0.7380.738. Meditron-7B shows the largest increase, rising from 0.2930.293 zero-shot and 0.2440.244 few-shot to 0.6630.663 after instruction tuning. Silver supervision is more effective than gold-only tuning at the current scale. Across all three tasks and all five models, instruction tuning on the silver dataset yields higher gold-set point estimates than training on the gold folds alone. For Task A, Mistral-24B achieves 0.8470.847 under Silverâ , compared with 0.7150.715 under Gold; Qwen3-30B-A3B scores 0.7950.795 and 0.6060.606, respectively. The same pattern appears in Task B. The gold-only results should not be interpreted as evidence that the silver annotations are of higher quality. Rather, they show that at the evaluated scale, broader coverage and larger volume of the silver dataset provide sufficient supervision for downstream models to learn the target tasks. Gold annotations remain essential as an independent benchmark for assessing whether models trained on silver supervision generalize to human-verified outputs. 7 Related Work 7.1 Temporal Annotation in Clinical NLP Clinical temporal NLP has traditionally focused on extracting typed events, time expressions, and pairwise temporal relations in benchmarks such as i2b2, THYME, and n2c2 Styler et al. (2014); Sun et al. (2013); Uzuner et al. (2011). Later work extended this paradigm toward absolute event timing and temporal intervals Leeuwenberg and Moens (2020); Frattallone-Llado et al. (2024). These methods established the foundations of clinical temporal reasoning, but generally assume predefined event types or annotated mentions Wang et al. (2024). CliniCIRCA instead reconstructs an open-vocabulary, patient-level timeline directly from narrative text, grounding events to calendar dates or ranges while preserving temporal uncertainty. 7.2 Constructing Event Timelines & Summaries from Clinical Narratives Recent work has used LLMs to generate event time sequences from clinical narratives through either single-pass extraction or multi-stage reconstruction Kumar et al. (2026); Noroozizadeh et al. (2025); Wang and Weiss (2025). Existing approaches make this task tractable by imposing different forms of structure. MedTimeline focuses on a predefined schema of chemotherapy events and does not model temporally vague events Wang et al. (2024). Frattallone-Llado et al. (2024) and Kumar et al. (2026) rely on structured EHR data to improve or calibrate event timing. MIMIC-IV-Ext-22MCTS applies chunking and retrieval to discharge summaries before generating events with relative timestamps Wang et al. (2025a). In contrast, Noroozizadeh et al. (2025) construct timelines from published case reports, which are typically more organized and curated than routine clinical narratives. Together, these studies demonstrate the feasibility and downstream value of narrative timeline generation, but they focus on settings that are narrower, more structured, or more curated than ours. Clinical summarization addresses a complementary goal: condensing lengthy records into coherent accounts for review and decision-making Luo et al. (2023); Lee et al. (2024); Croxford et al. (2025). Most summarization systems generate directly from the source note, combining event selection, temporal reasoning, and narrative generation in a single step that makes the underlying chronology difficult to inspect. CliniCIRCA instead decouples these decisions: it first constructs an event-level, uncertainty-aware timeline and then summarizes from that representation. The summary is therefore not an independent compression of the original note, but a readable realization of an auditable temporal substrate. 8 Broader Implications Beyond demonstrating the feasibility of patient-journey construction from discharge summaries, our findings reveal practical implications for clinical NLP and usage in real-world contexts. Reusable supervision. Clinical institutions often have abundant unstructured records but limited resources for expert manual annotation, secure computing resources, and access to large proprietary models Balasubramanian et al. (2025); Ntinopoulos et al. (2025); Grothey et al. (2025). CliniCIRCA offers a practical route from raw text to task-specific supervision: instruction tuning on only 1,0001,000 silver-labeled examples improved every model on extraction and summarization, and four of five on temporal tagging. These results suggest that modest pipeline-generated datasets can support smaller, locally deployable models without requiring expert labeling of an entire clinical corpus or complex schema development, as has been the case in prior research (Section 7.2). The necessity of human-in-the-loop oversight. Reliable evaluation is particularly important in clinical NLP because infrequent errors can remain consequential even when aggregate performance appears high Jiang et al. (2026); Hager et al. (2024). Prior work in psychiatric NLP similarly finds that LLM outputs may resemble expert responses on surface-level properties such as tone while diverging on clinically consequential dimensions Chandra et al. (2025); Wang et al. (2025b). Our automated judge agrees strongly with mental health clinicians on the large majority of correct events, but is less reliable in identifying the nature of rare errors, particularly those involving pre-admission medications and date granularity. Producing a trustworthy reference set therefore requires exhaustive clinician-guided adjudication, since expert review remains necessary to resolve ambiguous temporal decisions and determine which errors are clinically meaningful Bavaresco et al. (2025); Diekmann et al. (2025); Chen et al. (2024). Supporting longitudinal chart review. Reviewing longitudinal records is time- and effort-intensive, especially for multimorbid patients whose histories could be distributed across multiple clinical encounters and narrative documents Van Veen et al. (2024). Our proposed Stage 3 renders the event-level timeline as a readable patient journey while retaining its calendar anchors and temporal uncertainty. Such summaries could provide clinicians with a traceable overview before time-constrained follow-up visits, especially in contexts in which manually reconstructing a complex history from several discharge summaries may be impractical. That said, we underscore that they should complement rather than replace the source record, with individual claims remaining connected to inspectable event-level evidence. 9 Conclusion and Future Work CliniCIRCA shows that recovering a patient journey from clinical narrative is best treated not as a single generation problem, but as separable representational decisions: what happened, when it happened, and how it should be rendered for review. This decomposition makes the resulting chronology inspectable, exposes where temporal ambiguity remains unresolved, and enables each stage to be evaluated independently. Our evaluation further shows that automated judges are useful for locating problematic outputs, but not for replacing expert adjudication of rare temporal errors. At the same time, the reconstructed journeys provide effective supervision for smaller open-weight models, suggesting that structured intermediate representations can serve not only as outputs, but also as reusable training resources. Future work will extend CliniCIRCA to multi-document patient records, richer temporal representations, and additional note types, to support traceable longitudinal chart review and memory-aware clinical systems. Limitations Although we present a framework with clinician-designed and verified definitions and steps, we recognize several areas for improvement. Representational scope. Our date-centered schema supports chronological ordering but may impose artificial timing on atemporal facts or persistent states, such as sex, family history, and long-term substance use. Future work should compare alternatives, including atemporal labels, validity intervals, and separate state/event categories, and evaluate their effects on timeline fidelity, summarization, and borderline cases. Summarization salience. Stage 3 follows a clinician-authored specification for retaining, combining, and omitting events in the narrative summary. Because the same two clinicians developed this specification and conducted the faithfulness and temporal-accuracy evaluation described in Section 5.4, the results may understate disagreement in salience or interpretation. Independent evaluation would strengthen this validation. The specification also reflects one view of clinical relevance, since priorities may differ across clinicians and care settings. The resulting summaries should therefore not be treated as a single canonical account of a patientâs journey. Future work should include more diverse evaluators and clinical settings, salience policies, and multiple reference summaries. Psychiatric symptom representation in Stage 3. Our Stage 3 specification (Appendix H) excludes affect descriptions with vital signs and routine examination findings to reduce boilerplate. As a result, mood and affect trajectories â a core component of psychiatric symptom tracking, as illustrated by the depression case in Section 5.5 â are preserved in the auditable Output A timeline (e.g., via diagnosis and treatment events) but not always carried into the reader-facing Output B narrative in their own right. Revisiting this exclusion for the mental-health setting specifically, for example by retaining clinically significant affect changes as a distinct category rather than folding them under general exam findings, is a natural next step for future research. Single-document scope. CliniCIRCA reconstructs the course described within one discharge summary per admission rather than multiple notes from the full longitudinal record. This bounded setting isolates narrative reconstruction, but cannot reconcile duplicated, conflicting, or evolving information across admissions. Prior patient-journey methods are not directly comparable because they use different inputs, prompts, and context assumptions. We therefore compare 11 candidate models under the same framework (Section 4), but do not include a single-pass end-to-end baseline. Future work should ablate the modular design using the same LLM and extend it to multi-document records through cross-note entity resolution, temporal alignment, and representation of revisions in clinical understanding. Despite these limitations, our proposed framework CliniCIRCA provides a clinician-validated representation for open-vocabulary temporal reconstruction from raw clinical narratives. Ethical Considerations We use the MIMIC-I Clinical Database (v1.4), accessed through PhysioNet under its Credentialed Health Data Use Agreement (v1.5.0) with the required CITI training completed for all personnel with direct access or analysis of the data. LLMs were used in compliance with PhysioNetâs policy on responsible use of MIMIC data with LLMs and online services. We accessed Claude models through Amazon Bedrock and Gemini models through Vertex AI because the data were neither externally retained nor used for model training. All human evaluation was performed exclusively by the authors, without the recruitment of external human subjects. As a secondary analysis of de-identified records, this work is not human-subjects research and required no additional IRB review. Because source notes may unevenly document social history, trauma, or substance use across patients â for instance by diagnosis, demographic group, or documentation era â CliniCIRCAâs extraction and summarization stages could reproduce such documentation asymmetries rather than correct them; auditing this is an important direction for future work. We also note that pipeline errors are not equally consequential: a mis-tagged or fabricated event involving suicidality, self-harm, or medication timing carries substantially more clinical risk than a mis-tagged laboratory value. Given this, we emphasize that Stage 3 outputs (Output B) are intended to complement, not replace, the source record, and should remain linked to their inspectable Stage 1-2 source events (Output A) in any deployment setting within the healthcare system, particularly for safety-relevant content. Acknowledgments Zhang, Ding, and De Choudhury were partly supported through funds from The Childrenâs Healthcare of Atlanta Pediatric Technology Center at Georgia Tech. We thank Jiawei Zhou, Rijul Magu, Shravika Mittal, Viet Cuong Nguyen, Pinxian Lu, Owen Xingjian Zhang, Zeyu Hua, and Zikang Leng for valuable technical ideation and suggestions throughout this work. We are grateful to Franklin Ye Ruan for discussions of clinical and medical knowledge that informed the design of this study. We thank Sonakshi Sisodiya for early-stage annotation, and Andrew Zhao for technical and server support. We also thank Kaike Ping for statistical guidance and support. References Abouelenin et al. (2025) A. Abouelenin et al. Phi-4-Mini Technical Report: compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743. Cited by: Table A4, §4.1. Alfattni et al. (2020) G. Alfattni, N. Peek, and G. Nenadic Extraction of temporal relations from clinical free text: A systematic review of current approaches. Journal of Biomedical Informatics 108, p. 103488 (en). External Links: ISSN 15320464, Link, Document Cited by: §1, §5.1. Amirahmadi et al. (2025) A. Amirahmadi, F. Etminani, J. Björk, O. Melander, and M. Ohlsson Trajectory-Ordered Objectives for Self-Supervised Representation Learning of Temporal Healthcare Data Using Transformers: Model Development and Evaluation Study. JMIR Medical Informatics 13, p. e68138 (en). External Links: ISSN 2291-9694, Link, Document Cited by: §1. Anthropic (2025) Anthropic Claude Haiku 4.5 System Card. Note: System card External Links: Link Cited by: Table A4, §4.1. Anthropic (2026) Anthropic Claude Sonnet 4.6 System Card. Note: System card External Links: Link Cited by: Table A4, §4.1. Balasubramanian et al. (2025) J. B. Balasubramanian, D. Adams, I. Roxanis, A. B. De Gonzalez, P. Coulson, J. S. Almeida, and M. GarcĂa-Closas Leveraging large language models for structured information extraction from pathology reports. Journal of Pathology Informatics 19, p. 100521 (en). External Links: ISSN 21533539, Link, Document Cited by: §8. Bavaresco et al. (2025) A. Bavaresco, R. Bernardi, L. Bertolazzi, D. Elliott, R. FernĂĄndez, A. Gatt, E. Ghaleb, M. Giulianelli, M. Hanna, A. Koller, A. Martins, P. Mondorf, V. Neplenbroek, S. Pezzelle, B. Plank, D. Schlangen, A. Suglia, A. K. Surikuchi, E. Takmaz, and A. Testoni LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 238â255. External Links: Link, Document, ISBN 979-8-89176-252-7 Cited by: Appendix E, §8. Card et al. (2020) D. Card, P. Henderson, U. Khandelwal, R. Jia, K. Mahowald, and D. Jurafsky With Little Power Comes Great Responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, p. 9263â9274 (en). External Links: Link, Document Cited by: §4.2. Centers for Medicare & Medicaid Services and National Center for Health Statistics (2011) Centers for Medicare & Medicaid Services and National Center for Health Statistics ICD-9-CM Official Guidelines for Coding and Reporting. Note: Effective October 1, 2011U.S. Department of Health and Human Services External Links: Link Cited by: §A.1, Table A1. Champely (2020) S. Champely Pwr: basic functions for power analysis. Note: R package version 1.3-0 External Links: Link Cited by: Appendix C. Chandra et al. (2025) M. Chandra, S. Sriraman, G. Verma, H. S. Khanuja, J. S. Campayo, Z. Li, M. L. Birnbaum, and M. De Choudhury Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Use. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Albuquerque, New Mexico, p. 11083â11113 (en). External Links: Link, Document Cited by: §8. Chaturvedi et al. (2025) R. Chaturvedi, P. Baghershahi, S. Medya, and B. Di Eugenio Temporal relation extraction in clinical texts: a span-based graph transformer approach. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 25765â25788. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. Chaturvedi (2024) R. Chaturvedi Temporal Knowledge Graph Extraction and Modeling across Multiple Documents for Health Risk Prediction. In Companion Proceedings of the ACM Web Conference 2024, Singapore Singapore, p. 1182â1185 (en). External Links: ISBN 979-8-4007-0172-6, Link, Document Cited by: §1. Chen et al. (2024) G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang Humans or LLMs as the Judge? A Study on Judgement Bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 8301â8327 (en). External Links: Link, Document Cited by: §8. Cohen (1988) J. Cohen Statistical power analysis for behavioral sciences. 2nd edition, Lawrence Erlbaum Associates, Hillsdale, NY. Cited by: Appendix C. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, L. Marris, S. Petulla, C. Gaffney, A. Aharoni, N. Lintz, T. C. Pais, H. Jacobsson, I. Szpektor, N. Jiang, K. Haridasan, A. Omran, N. Saunshi, D. Bahri, G. Mishra, E. Chu, T. Boyd, B. Hekman, A. Parisi, C. Zhang, K. Kawintiranon, T. Bedrax-Weiss, O. Wang, Y. Xu, O. Purkiss, U. Mendlovic, I. Deutel, N. Nguyen, A. Langley, F. Korn, L. Rossazza, A. RamĂ©, S. Waghmare, H. Miller, N. Byrd, A. Sheshan, R. Hadsell, S. Bhardwaj, P. Janus, T. Rissa, D. Horgan, A. Abdagic, L. Belenki, J. Allingham, A. Singh, T. Guidroz, S. Srinivasan, H. Schmit, K. Chiafullo, A. Elisseeff, N. Jha, P. Kolhar, L. Berrada, F. Ding, X. Si, S. B. Mallick, F. Och, S. Erell, E. Ni, T. Latkar, S. Yang, P. Sirkovic, Z. Feng, R. Leland, R. Hornung, G. Wu, C. Blundell, H. Alvari, P. Huang, C. Yip, S. Deur, L. Liu, G. Surita, P. Duque, D. Damen, J. Jia, A. Guez, M. Mircea, A. Sinha, A. Magni, P. Stradomski, T. Marian, V. GaliÄ, W. Chen, H. Husain, A. Singhal, D. Grewe, F. Aubet, S. Song, L. Blanco, L. Rechis, L. Ho, R. Munoz, K. Zheng, J. Hamrick, K. Mather, H. Taitelbaum, E. Rutherford, Y. Lei, K. Chen, A. Shukla, E. Moreira, E. Doi, B. Isik, N. Shabat, D. RogoziĆska, K. Kolipaka, J. Chang, E. VuĆĄak, S. Venkatachary, S. Noghabi, T. Bharti, Y. Jun, A. Zaks, S. Green, J. Challagundla, W. Wong, M. Mohammad, D. Hirsch, Y. Cheng, I. Naim, L. Proleev, D. Vincent, A. Singh, M. Krikun, D. Krishnan, Z. Ghahramani, A. Atias, R. Aggarwal, C. Kirov, D. Vytiniotis, C. Koh, A. Chronopoulou, P. Dogra, V. Ion, G. Tyen, J. Lee, F. Weissenberger, T. Strohman, A. Balakrishna, J. Rae, M. Velic, R. de Liedekerke, O. Elyada, W. Yuan, C. Liu, L. Shani, S. Kishchenko, B. Alessio, Y. Li, R. Song, S. Kwei, O. Jankowski, A. Pappu, Y. Namiki, Y. Ma, N. Tripuraneni, C. Cherry, M. Ikonomidis, Y. Ling, C. Ji, B. Westberg, A. Wright, D. Yu, D. Parkinson, S. Ramaswamy, J. Connor, S. H. Yeganeh, S. Grover, G. Kenwright, L. Litchev, C. Apps, A. Tomala, F. Halim, A. Castro-Ros, Z. Li, A. Boral, P. Sho, M. Yarom, E. Malmi, D. Klinghoffer, R. Lin, A. Ansell, P. K. S, S. Zhao, S. Zuo, A. Santoro, H. Cheng, S. Demmessie, Y. Liu, N. Brichtova, A. Culp, N. Braun, D. Graur, W. Ng, N. Mehta, A. Phillips, P. Sundberg, V. Godbole, F. Liu, Y. Katariya, D. Rim, M. Seyedhosseini, S. Ammirati, J. Valfridsson, M. Malihi, T. Knight, A. Toor, T. Lampe, A. Ittycheriah, L. Chiang, C. Yeung, A. FrĂ©chette, J. Rao, H. Wang, H. Srivastava, R. Zhang, R. Rhodes, A. Brand, D. Weesner, I. Figotin, F. Gimeno, R. Fellinger, P. Marcenac, J. Leal, E. Marcus, V. Cotruta, R. Cabrera, S. Luo, D. Garrette, V. Axelrod, S. Baltateanu, D. Barker, D. Chen, H. Toma, B. Ingram, J. Riesa, C. Kulkarni, Y. Zhang, H. Liu, C. Wang, M. Polacek, W. Wu, K. Hui, A. N. Reyes, Y. Su, M. Barnes, I. Malhi, A. Siddiqui, Q. Feng, M. Damaschin, D. Pighin, A. Steiner, S. Yang, R. S. Boppana, S. Ivanov, A. Kandoor, A. Shah, A. Mujika, D. Huang, C. A. Choquette-Choo, M. Patel, T. Yu, T. Creswell, Jerry, Liu, C. Barros, Y. Razeghi, A. Roy, P. Culliton, B. Xiong, J. Pan, T. Strohmann, T. Powell, B. Seal, D. DeCarlo, P. Shyam, K. Katircioglu, X. Wang, C. Hardin, I. Odisho, J. Broder, O. Chang, A. Nair, A. Shtefan, M. OâBrien, M. Agarwal, S. Potluri, S. Goyal, A. Jhindal, S. Thakur, Y. Stuken, J. Lyon, K. Toutanova, F. Feng, A. Wu, B. Horn, A. Wang, A. Cullum, G. Taubman, D. Shrivastava, C. Shi, H. Tomlinson, R. Patel, T. Tu, A. M. Oflazer, F. Pongetti, M. Yang, A. A. TaĂŻga, V. Perot, N. W. Pierse, F. Han, Y. Drori, I. Iturrate, A. Chakrabarti, L. Yeung, D. Dopson, Y. Chen, A. Kulshreshtha, T. Guo, P. Pham, T. Schuster, J. Chen, A. Polozov, J. Xing, H. Zhou, P. Kacham, D. Kukliansky, A. Miech, S. Yaroshenko, E. Chi, S. Douglas, H. Fei, M. Blondel, P. Myla, L. Madmoni, X. Wu, D. Keysers, K. Kjems, I. Albuquerque, L. Yu, J. Dâsa, M. Plantan, V. Ionescu, J. S. Elias, A. Gupta, M. R. Vuyyuru, F. Alcober, T. Zhou, K. Ji, F. Hartmann, S. Puttagunta, H. Song, E. Amid, A. Stefanoiu, A. Lee, P. Pucciarelli, E. Wang, A. Raul, S. Petrov, I. Tian, V. Anklin, N. Nti, V. Gomes, M. Schumacher, G. Vesom, A. Panagopoulos, K. Bousmalis, D. Andor, J. Jacob, Y. Zhang, B. Rosgen, M. Kecman, M. Tung, A. Belias, N. Goodman, P. Covington, B. Wieder, N. Saxena, E. Davoodi, M. Huang, S. Maddineni, V. Roulet, F. Campbell-Ajala, P. G. Sessa, Xintian, Wu, G. Lai, P. Collins, A. Haig, V. Sakenas, X. Xu, M. Giustina, L. E. Shafey, P. Charoenpanit, S. Garg, J. Ainslie, B. Severson, M. G. Arenas, S. Pathak, S. Rajayogam, J. Feng, M. Bakker, S. Li, N. Wichers, J. Rogers, X. Geng, Y. Li, R. Jagerman, C. Jia, N. Olmert, D. Sharon, M. Mauger, S. Mariserla, H. Ma, M. Mohabey, K. Kim, A. Andreev, S. Pollom, J. Love, V. Jain, P. Agrawal, Y. Schroecker, A. Fortin, M. Warmuth, J. Liu, A. Leach, I. Blok, G. P. Girirajan, R. Aharoni, B. Uria, A. Sozanschi, D. Goldberg, L. Ionita, M. T. Ribeiro, M. Zlocha, V. Birodkar, S. Lachgar, L. Yuan, H. Choudhury, M. Ginsberg, F. Zheng, G. Dibb, E. Graves, S. Lokhande, G. Rasskin, G. Muraru, C. Quick, S. Tata, P. Sermanet, A. Chawla, I. Karo, Y. Wang, S. Zhang, O. Keller, A. Dragan, G. Su, I. Chou, X. Liu, Y. Tao, S. Prabhakara, M. Wilson, R. Liu, S. Wang, G. Evans, D. Du, A. Castaño, G. Prasad, M. E. Mahdy, S. Gerlach, M. Reid, J. Kahn, A. Zait, T. S. Pillai, T. Ulrich, G. Wang, J. Wassenberg, E. Farkash, K. Yalasangi, C. Wang, M. Bauza, S. Bucher, T. Liu, J. Yan, G. Leung, V. Sindhwani, P. Barnes, A. Singh, I. Jurin, J. Chang, N. K. Bhumihar, S. Eiger, G. Citovsky, B. Withbroe, Z. Li, S. Xue, N. D. Santo, G. Stoyanov, Y. Raimond, S. Zheng, Y. Gao, V. ListĂk, S. Kwasiborski, R. Saputro, A. Ozturel, G. Mallya, K. Majmundar, R. West, P. Caron, J. Wei, L. Castrejon, S. Vikram, D. Ramachandran, N. Dhawan, J. Park, S. Smoot, G. van den Driessche, Y. Blau, C. Malik, W. Liang, R. Hirsch, C. N. dos Santos, E. Weinstein, A. van den Oord, S. Lall, N. FitzGerald, Z. Jiang, X. Yang, D. Webster, A. Elqursh, A. Pope, G. Rotival, D. Raposo, W. Zhu, J. Dean, S. Alabed, D. Tran, A. Gupta, Z. Gleicher, J. Austin, E. Rosseel, M. Umekar, D. Das, Y. Sun, K. Chen, K. Misiunas, X. Zhou, Y. Di, A. Loo, J. Newlan, B. Li, V. Ramasesh, Y. Xu, A. Chen, S. Gandhe, R. Soricut, N. Gupta, S. Hu, S. El-Sayed, X. Garcia, I. Brusilovsky, P. Chen, A. Bolt, L. Huang, A. Gurney, Z. Zhang, A. Pritzel, J. Wilkiewicz, B. Seybold, B. K. Shamanna, F. Fischer, J. Dean, K. Gill, R. Mcilroy, A. Bhowmick, J. Selier, A. Yang, D. Cheng, V. Magay, J. Tan, D. Varma, C. Walder, T. Kocisky, R. Nakashima, P. Natsev, M. Kwong, I. Gog, C. Zhang, S. Dieleman, T. Jimma, A. Ryabtsev, S. Brahma, D. Steiner, D. Du, A. ĆœuĆŸul, M. ĆœaniÄ, M. Raghavachari, W. Gierke, Z. Zheng, D. Petrova, Y. Dauphin, Y. Liu, I. Kessler, S. Hand, C. Duvarney, S. Kim, H. Lee, L. Hussenot, J. Hui, J. Smith, D. Jain, J. Xia, G. S. Tomar, K. Amiri, D. Phan, F. Fuchs, T. Weyand, N. Tomasev, A. Cordell, X. Liu, J. Mallinson, P. Joshi, A. Crawford, A. Suggala, S. Chien, N. Fernando, M. Sanchez-Vargas, D. Williams, P. Crone, X. Luo, I. Karpov, J. Shan, T. Thurk, R. Strudel, P. Voigtlaender, P. Patil, T. Dozat, A. Khodaei, S. Singla, P. Ambroszczyk, Q. Wu, Y. Chang, B. Roark, C. Hegde, T. Ding, A. Filos, Z. Wu, A. S. Pinto, S. Liu, S. Khanna, A. Pandey, S. Mcloughlin, Q. Li, S. Haves, A. Zhou, E. Buchatskaya, I. Leal, P. de Boursac, N. Akazawa, N. Anderson, T. Chen, K. Somandepalli, C. Liang, S. Goenka, S. Winkler, A. Grushetsky, Y. Ding, J. Smith, F. Ye, J. Pont-Tuset, E. Li, R. Li, T. Golany, D. Wegner, T. Jiang, O. Barak, Y. Shangguan, E. VĂ©rtes, R. Wong, J. Bornschein, A. Tudor, M. Bevilacqua, T. Schaul, A. S. Rawat, Y. Zhao, K. Axiotis, L. Meng, C. McLean, J. Lai, J. Beattie, N. Kushman, Y. Liu, B. Kutzman, F. Lang, J. Ye, P. Netrapalli, P. Mishra, M. Khan, M. Goel, R. Willoughby, D. Tian, H. Zhuang, J. Chen, Z. Tsai, T. Kementsietsidis, A. Khare, J. Keeling, K. Xu, N. Waters, F. AltchĂ©, A. Popat, B. Mittal, D. Saxton, D. E. Badawy, M. Mathieu, Z. Zheng, H. Zhou, N. Ranka, R. Shin, Q. Duan, T. Salimans, I. Mihailescu, U. Shaham, M. Chang, Y. Assael, N. Dikkala, M. Izzard, V. Cohen-Addad, C. Graves, V. Feinberg, G. Chung, D. Strouse, D. Karmon, S. Sharifzadeh, Z. Ashwood, K. Pham, J. Blanton, A. Vasiloff, J. Barber, M. Geller, A. Zhou, F. Zubach, T. Huang, L. Zhang, H. Gupta, M. Young, J. Proskurnia, R. Votel, V. Gabeur, G. Barcik, A. Tripathi, H. Yu, G. Yan, B. Changpinyo, F. PavetiÄ, A. Coyle, Y. Fujii, J. G. Mendez, T. Zhou, H. Rajamani, B. Hechtman, E. Cao, D. Juan, Y. Tan, V. Dalibard, Y. Du, N. Clay, K. Yao, W. Jia, D. Vijaykumar, Y. Zhou, X. Bai, W. Hung, S. Pecht, G. Todorov, N. Khadke, P. Gupta, P. Lahoti, A. Autef, K. Duddu, J. Lee-Thorp, A. Bykovsky, T. Misiunas, S. Flennerhag, S. Thangaraj, J. McGiffin, Z. Nado, M. Kunesch, A. Noever, A. Hertz, M. Liang, V. Stone, E. Palmer, S. Daruki, A. Pramanik, S. PĂ”der, A. Kyker, M. Khan, E. Sluzhaev, M. Ritter, A. Ruderman, W. Zhou, C. Nagpal, K. Vodrahalli, G. Necula, P. Barham, E. Pavlick, J. Hartford, I. Shafran, L. Zhao, M. MikuĆa, T. Eccles, H. Shimokawa, K. Garg, L. Vilnis, H. Chen, I. Shumailov, K. Lee, A. Abdelhamed, M. Xie, V. Cohen, E. Hlavnova, D. Malkin, C. Sitawarin, J. Lottes, P. Coquinot, T. Yu, S. Kumar, J. Zhang, A. Mahendru, Z. Ahmed, J. Martens, T. Chen, A. Boag, D. Peng, C. Devin, A. Klimovskiy, M. Phuong, D. Vainstein, J. Xie, B. Ramabhadran, N. Howard, X. Yu, G. Goswami, J. Cui, S. Shleifer, M. Pinto, C. Yeh, M. Yang, S. Javanmardi, D. Ethier, C. Lee, J. Orbay, S. Kotecha, C. Bromberg, P. Shaw, J. Thornton, A. G. Rosenthal, S. Gu, M. Thomas, I. Gemp, A. Ayyar, A. Ushio, A. Selvan, J. Wee, C. Liu, M. Majzoubi, W. Yu, J. Abernethy, T. Liechty, R. Pan, H. Nguyen, Qiong, Hu, S. Perrin, A. Arora, E. Pitler, W. Wang, K. Shivakumar, F. Prost, B. Limonchik, J. Wang, Y. Gao, T. Cour, S. Buch, H. Gui, M. Ivanova, P. Neubeck, K. Chan, L. Kim, H. Chen, N. Goyal, D. Chung, L. Liu, Y. Su, A. Petrushkina, J. Shen, A. Joulin, Y. Xu, S. X. Lin, Y. Kulizhskaya, C. Chelba, S. Vasudevan, E. Collins, V. Bashlovkina, T. Lu, D. Fritz, J. Park, Y. Zhou, C. Su, R. Tanburn, M. Sushkov, M. Rasquinha, J. Li, J. Prendki, Y. Li, P. LV, S. Sharma, H. Fitoussi, H. Huang, A. Dai, P. Dao, M. Burrows, H. Prior, D. Qin, G. Pundak, L. L. Sjoesund, A. Khurshudov, Z. Zhu, A. Webson, E. Kemp, T. Tan, S. Agrawal, S. Sargsyan, L. Cheng, J. Stephan, T. Kwiatkowski, D. Reid, A. Byravan, A. H. Michaely, N. Heess, L. Zhou, S. Goenka, V. Carpenter, A. Levskaya, B. Wang, R. Roberts, R. Leblond, S. Chikkerur, S. Ginzburg, M. Chang, R. Riachi, Chuqiao, Xu, Z. Borsos, M. Pliskin, J. Pawar, M. Lustman, H. Kirkwood, A. Anand, A. Chaudhary, N. Kalb, K. Milan, S. Augenstein, A. Goldie, L. Prince, K. Raman, Y. Sun, V. Xia, A. Cohen, Z. Huo, J. Camp, S. Ellis, L. Zilka, D. V. Torres, L. Patel, S. Arora, B. Chan, J. Adler, K. Ayoub, J. Liang, F. Jamil, J. Jiang, S. Baumgartner, H. Sun, Y. Karov, Y. Akulov, H. Zheng, I. Cai, C. Fantacci, J. Rubin, A. R. Acha, M. Wang, N. DâSouza, R. Sathyanarayana, S. Dai, S. Rowe, A. Simanovsky, O. Goldman, Y. Kuang, X. Pan, A. Rosenberg, T. Rojas-Esponda, P. Dutta, A. Zeng, I. Jurenka, G. Farquhar, Y. Bansal, S. Iqbal, B. Roelofs, G. Joung, P. Beak, C. Ryu, R. Poplin, Y. Wu, J. Alayrac, S. Buthpitiya, O. Ronneberger, C. Habtegebriel, W. Li, P. Cavallaro, A. Wei, G. Bensky, T. Denk, H. Ganapathy, J. Stanway, P. Joshi, F. Bertolini, J. Lo, O. Ma, Z. Charles, G. Sampemane, H. Sahni, X. Chen, H. Askham, D. Gaddy, P. Young, J. Tan, M. Eyal, A. BraĆŸinskas, L. Zhong, Z. Wu, M. Epstein, K. Bailey, A. Hard, K. Lee, S. Goldshtein, A. Ruiz, M. Badawi, M. Lochbrunner, J. Kearns, A. Brown, F. Pardo, T. Weber, H. Yang, P. Jiang, B. Akin, Z. Fu, M. Wainwright, C. Zou, M. Gaba, P. Manzagol, W. Kan, Y. Song, K. Zainullina, R. Lin, J. Ko, S. Deshmukh, A. Jindal, J. Svensson, D. Tyam, H. Zhao, C. Kaeser-Chen, S. Baird, P. Moradi, J. Hall, Q. Guo, V. Tsang, B. Liang, F. Pereira, S. Ganesh, I. Korotkov, J. Adamek, S. Thiagarajan, V. Tran, C. Chen, C. Tar, S. Jain, I. Dasgupta, T. Bilal, D. Reitter, K. Zhao, G. Vezzani, Y. Gehman, P. Mehta, L. Beltrone, X. Dotiwalla, S. Guadarrama, Z. Abbas, S. Karp, P. Georgiev, C. Ferng, M. Brockschmidt, L. Peng, C. Hirnschall, V. Verma, Y. Bi, Y. Xiao, A. Dabush, K. Xu, P. Wallis, R. Parker, Q. Wang, Y. Xu, I. Safarli, D. Tewari, Y. Zhang, S. Kim, A. Gesmundo, M. Thomas, S. Levi, A. Chowdhury, K. Rao, P. Garst, S. Conway-Rahman, H. Ran, K. McKinney, Z. Xiao, W. Yu, R. Agrawal, A. Stjerngren, C. Ionescu, J. Chen, V. Sharma, J. Chiu, F. Liu, K. Franko, C. Sanford, X. Cai, P. Michel, S. Ganapathy, J. Labanowski, Z. Garrett, B. Vargas, S. Sun, B. Gale, T. Buschmann, G. Desjardins, N. Ghelani, P. Jain, M. Verma, C. Asawaroengchai, J. Eisenschlos, J. Harlalka, H. Kazawa, D. Metzler, J. Howland, Y. Jian, J. Ades, V. Shah, T. Gangwani, S. Lee, R. Ring, S. M. Hernandez, D. Reich, A. Sinha, A. Sathe, J. Kovac, A. Gill, A. Kannan, A. Dâolimpio, M. Sevenich, J. Whang, B. Kim, K. C. Sim, J. Chen, J. Zhang, S. Lall, Y. Matias, B. Jia, A. Friesen, S. Nasso, A. Thapliyal, B. Perozzi, T. Yu, A. Shekhawat, S. Huda, P. Grabowski, E. Wang, A. Sreevatsa, H. Dib, M. Hassen, P. Schuh, V. Milutinovic, C. Welty, M. Quinn, A. Shah, B. Wang, G. Barth-Maron, J. Frye, N. Axelsson, T. Zhu, Y. Ma, I. Giannoumis, H. Sedghi, C. Ye, Y. Luan, K. Aydin, B. Chandra, V. Sampathkumar, R. Huang, V. Lavrenko, A. Eleryan, Z. Hong, S. Hansen, S. M. Carthy, B. Samanta, D. Äevid, X. Wang, F. Li, M. Voznesensky, M. Hoffman, A. Terzis, V. Sehwag, G. Fidel, L. He, M. Cai, Y. He, A. Feng, M. Nikoltchev, S. Phatale, J. Chase, R. Lawton, M. Zhang, T. Ouyang, M. Tragut, M. H. Manshadi, A. Narayanan, J. Shen, X. Gao, T. Bolukbasi, N. Roy, X. Li, D. Golovin, L. Panait, Z. Qin, G. Han, T. Anthony, S. Kudugunta, V. Patraucean, A. Ray, X. Chen, X. Yang, T. Bhatia, P. Talluri, A. Morris, A. RaĆŸnatoviÄ, B. Brownfield, J. An, S. Peng, P. Kane, C. Zheng, N. Duduta, J. Kessinger, J. Noraky, S. Liu, K. Rong, P. VeliÄkoviÄ, K. Rush, A. Goldin, F. Wei, S. M. R. Garlapati, C. Pantofaru, O. Kwon, J. Ni, E. Noland, J. D. Trapani, F. Beaufays, A. G. Roy, Y. Chow, A. Turker, G. Cideron, L. Mei, J. Clark, Q. Dou, M. BoĆĄnjak, R. Leith, Y. Du, A. Yazdanbakhsh, M. Nasr, C. Kwak, S. S. Sheth, A. Kaskasoli, A. Anand, B. Lakshminarayanan, S. Jerome, D. Bieber, C. Chu, A. Senges, T. Shen, M. Sridhar, N. Ndebele, B. Beyret, S. Mohamed, M. Chen, M. Freitag, J. Guo, L. Liu, P. Roit, H. Chen, S. Yan, T. Stone, J. Co-Reyes, J. Cole, S. Scellato, S. Azizi, H. Hashemi, A. Jin, A. Iyer, M. Valentine, A. György, A. Ahuja, D. H. Diaz, C. Lee, N. Clement, W. Kong, D. Garmon, I. Watts, K. Bhatia, K. Gupta, M. Miecnikowski, H. Vallet, A. Taly, E. Loper, S. Joshi, J. Atwood, J. Chick, M. Collier, F. Iliopoulos, R. Trostle, B. Gunel, R. Leal-Cavazos, A. M. Hrafnkelsson, M. Guzman, X. Ju, A. Forbes, J. Emond, K. Chauhan, B. Caine, L. Xiao, W. Zeng, A. Moufarek, D. Murphy, M. Meng, N. Gupta, F. Riedel, A. Das, E. Lawal, S. Narayan, T. Sosea, J. Swirhun, L. Friso, B. Neyshabur, J. Lu, S. Girgin, M. Wunder, E. Yvinec, A. Pyne, V. Carbune, S. Rijhwani, Y. Guo, T. Doshi, A. Briukhov, M. Bain, A. Hitron, X. Wang, A. Gupta, K. Chen, C. Du, W. Zhang, D. Shah, A. Akula, M. Dylla, A. Kachra, W. Kuo, T. Zou, L. Wang, L. Xu, J. Zhu, J. Snyder, S. Menon, O. Firat, I. Mordatch, Y. Yuan, N. Ponomareva, R. Blevins, L. Moore, W. Wang, P. Chen, M. Scholz, A. Dwornik, J. Lin, S. Li, D. Antognini, T. I, X. Song, M. Miller, U. Kalra, A. Raveret, O. Akerlund, F. Wu, A. Nystrom, N. Godbole, T. Liu, H. DeBalsi, J. Zhao, B. Liu, A. Caciularu, L. Lax, U. Khandelwal, V. Langston, E. Bailey, S. Lattanzi, Y. Wang, N. Kovelamudi, S. Mondal, G. Guruganesh, N. Hua, O. Roval, P. WesoĆowski, R. Ingale, J. Halcrow, T. Sohn, C. Angermueller, B. Raad, E. Stickgold, E. Lu, A. Kosik, J. Xie, T. Lillicrap, A. Huang, L. L. Zhang, D. Paulus, C. Farabet, A. Wertheim, B. Wang, R. Joshi, C. Ko, Y. Wu, S. Agrawal, L. Lin, X. Sheng, P. Sung, T. Breland-King, C. Butterfield, S. Gawde, S. Singh, Q. Zhang, R. Apte, S. Shetty, A. Hutter, T. Li, E. Salesky, F. Lebron, J. Kanerva, M. Paganini, A. Nguyen, R. Vallu, J. Peter, S. Velury, D. Kao, J. Hoover, A. Bortsova, C. Bishop, S. Jakobovits, A. Agostini, A. Agarwal, C. Liu, C. Kwong, S. Tavakkol, I. Bica, A. Greve, A. GP, J. Marcus, L. Hou, T. Duerig, R. Moroshko, D. Lacey, A. Davis, J. Amelot, G. Wang, F. Kim, T. Strinopoulos, H. Wan, C. L. Lan, S. Krishnan, H. Tang, P. Humphreys, J. Bai, I. H. Shtacher, D. Machado, C. Pang, K. Burke, D. Liu, R. Aravamudhan, Y. Song, E. Hirst, A. Singh, B. Jou, L. Bai, F. Piccinno, C. K. Fu, R. Alazard, B. Meiri, D. Winter, C. Chen, M. Zhang, J. Heitkaemper, J. Lambert, J. Lee, A. Frömmgen, S. Rogulenko, P. Nair, P. Niemczyk, A. Bulyenov, B. Xu, H. Shemtov, M. Zadimoghaddam, S. Toropov, M. Wirth, H. Dai, S. Gollapudi, D. Zheng, A. Kurakin, C. Lee, K. Bullard, N. Serrano, I. Balazevic, Y. Li, J. Schalkwyk, M. Murphy, M. Zhang, K. Sequeira, R. Datta, N. Agrawal, C. Sutton, N. Attaluri, M. Chiang, W. Farhan, G. Thornton, K. Lin, T. Choma, H. Nguyen, K. Dasgupta, D. Robinson, I. ComĆa, M. Riley, A. Pillai, B. Mustafa, B. Golan, A. Zandieh, J. Lespiau, B. Porter, D. Ross, S. Rajayogam, M. Agarwal, S. Venugopalan, B. Shahriari, Q. Yan, H. Xu, T. Tobin, P. Dubov, H. Shi, A. Recasens, A. Kovsharov, S. Borgeaud, L. Dery, S. Vasanth, E. Gribovskaya, L. Qiu, M. Mahdieh, W. Skut, E. Nielsen, C. Zheng, A. Yu, C. G. Bostock, S. Gupta, A. Archer, C. Rawles, E. Davies, A. Svyatkovskiy, T. Tsai, Y. Halpern, C. Reisswig, B. Wydrowski, B. Chang, J. Puigcerver, M. H. Taege, J. Li, E. Schnider, X. Li, D. Dena, Y. Xu, U. Telang, T. Shi, H. Zen, K. Kastner, Y. Ko, N. Subramaniam, A. Kumar, P. Blois, Z. Dai, J. Wieting, Y. Lu, Y. Zeldes, T. Xie, A. Hauth, A. Ćąifrea, Y. Li, S. El-Husseini, D. Abolafia, H. Zhou, W. Ding, S. Ghalebikesabi, C. GuĂa, A. Maksai, Ă. Weisz, S. Arik, N. Sukhanov, A. Ćwietlik, X. Jia, L. Yu, W. Wang, M. Brand, D. Bloxwich, S. Kirmani, Z. Chen, A. Go, P. Sprechmann, N. Kannen, A. Carin, P. Sandhu, I. Edkins, L. Nooteboom, J. Gupta, L. Maggiore, J. Azizi, Y. Pritch, P. Yin, M. Gupta, D. Tarlow, D. Smith, D. Ivanov, M. Babaeizadeh, A. Goel, S. Kambala, G. Chu, M. Kastelic, M. Liu, H. Soltau, A. Stone, S. Agrawal, M. Kim, K. Soparkar, S. Tadepalli, O. Bunyan, R. Soh, A. Kannan, D. Kim, B. J. Chen, A. Halumi, S. Roy, Y. Wang, O. Sercinoglu, G. Gibson, S. Bhatnagar, M. Sano, D. von Dincklage, Q. Ren, B. Mitrevski, M. OlĆĄĂĄk, J. She, C. Doersch, Jilei, Wang, B. Liu, Q. Tan, T. Yakar, T. Warkentin, A. Ramirez, C. Lebsack, J. Dillon, R. Mathews, T. Cobley, Z. Wu, Z. Chen, J. Simon, S. Nath, T. Sainath, A. Bendebury, R. Julian, B. Mankalale, D. Äurko, P. Zacchello, A. R. Brown, K. Sodhia, H. Howard, S. Caelles, A. Gupta, G. Evans, A. Bulanova, L. Katzen, R. Goldenberg, A. Tsitsulin, J. Stanton, B. Schillings, V. Kovalev, C. Fry, R. Shah, K. Lin, S. Upadhyay, C. Li, S. Radpour, M. Maggioni, J. Xiong, L. Haas, J. Brennan, A. Kamath, N. Savinov, A. Nagrani, T. Yacovone, R. Kappedal, K. Andriopoulos, L. Lao, Y. Li, G. Rozhdestvenskiy, K. Hashimoto, A. Audibert, S. Austin, D. Rodriguez, A. Ruoss, G. Honke, D. Karkhanis, X. Xiong, Q. Wei, J. Huang, Z. Leng, V. Premachandran, S. Bileschi, G. Evangelopoulos, T. Mensink, J. Pavagadhi, D. Teplyashin, P. Chang, L. Xue, G. Tanzer, S. Goldman, K. Patel, S. Li, J. Wiesner, I. Zheng, I. Stewart-Binks, J. Han, Z. Li, L. Luo, K. Lenc, M. LuÄiÄ, F. Xue, R. Mullins, A. Guseynov, C. Chang, I. Galatzer-Levy, A. Zhang, G. Bingham, G. Hu, A. Hartman, Y. Ma, J. Griffith, A. Irpan, C. Radebaugh, S. Yue, L. Fan, V. Ungureanu, C. Sorokin, H. Teufel, P. Li, R. Anil, D. Paparas, T. Wang, C. Lin, H. Peng, M. Shum, G. Petrovic, D. Brady, R. Nguyen, K. Macherey, Z. Li, H. Singh, M. Yenugula, M. Iinuma, X. Chen, K. Kopparapu, A. Stern, S. Dave, C. Thekkath, F. Perot, A. Kumar, F. Li, Y. Xiao, M. Bilotti, M. H. Bateni, I. Noble, L. Lee, A. VĂĄzquez-Reina, J. Salazar, X. Yang, B. Wang, E. Gruzewska, A. Rao, S. Raghuram, Z. Xu, E. Ben-David, J. Mei, S. Dalmia, Z. Zhang, Y. Liu, G. Bansal, H. Pankov, S. Schwarcz, A. Burns, C. Chan, S. Sanghai, R. Liang, E. Liang, A. He, A. Stuart, A. Narayanan, Y. Zhu, C. Frank, B. Fatemi, A. Sabne, O. Lang, I. Bhattacharya, S. Settle, M. Wang, B. McMahan, A. Tacchetti, L. B. Soares, M. Hadian, S. Cabi, T. Chung, N. Putikhin, G. Li, J. Chen, A. Tarango, H. Michalewski, M. Kazemi, H. Masoom, H. Sheftel, R. Shivanna, A. Vadali, R. Comanescu, D. Reid, J. Moore, A. Neelakantan, M. Sander, J. Herzig, A. Rosenberg, M. Dehghani, J. Choi, M. Fink, R. Hayes, E. Ge, S. Weng, C. Ho, J. Karro, K. Krishna, L. N. Thiet, A. Skerry-Ryan, D. Eppens, M. Andreetto, N. Sarma, S. Bonacina, B. K. Ayan, M. Nawhal, Z. Shan, M. Dusenberry, S. Thakoor, S. Gubbi, D. D. Nguyen, R. Tsarfaty, S. Albanie, J. MitroviÄ, M. Gandhi, B. Chen, A. Epasto, G. Stephanov, Y. Jin, S. Gehman, A. Amini, J. Weber, F. Behbahani, S. Xu, M. Allamanis, X. Chen, M. Ott, C. Sha, M. Jastrzebski, H. Qi, D. Greene, X. Wu, A. Toki, D. Vlasic, J. Shapiro, R. Kotikalapudi, Z. Shen, T. Saeki, S. Xie, A. Cassirer, S. Bharadwaj, T. Kiyono, S. Bhojanapalli, E. Rosenfeld, S. Ritter, J. Mao, J. G. Oliveira, Z. Egyed, B. Bandemer, E. Parisotto, K. Kinoshita, J. Pluto, P. Maniatis, S. Li, Y. Guo, G. Ghiasi, J. Tarbouriech, S. Chatterjee, J. Jin, Katrina, Xu, J. Palomaki, S. Arnold, M. Sewak, F. Piccinini, M. Sharma, B. Albrecht, S. Purser-haskell, A. Vaswani, C. Chen, M. Wisniewski, Q. Cao, J. Aslanides, N. M. Phu, M. Sieb, L. Agubuzu, A. Zheng, D. Sohn, M. Selvi, A. Andreassen, K. Subudhi, P. Eruvbetine, O. Woodman, T. Mery, S. Krause, X. Ren, X. Ma, J. Luo, D. Chen, W. Fan, H. Griffiths, C. Schuler, A. Li, S. Zhang, J. Sarr, S. Luo, R. Patana, M. Watson, D. Naboulsi, M. Collins, S. Sidhwani, E. Hoogeboom, S. Silver, E. Caveness, X. Zhao, M. Rodriguez, M. Deines, L. Bai, P. Griffin, M. Tagliasacchi, E. Xue, S. R. Babbula, B. Pang, N. Ding, G. Shen, E. Peake, R. Crocker, S. S. Raghvendra, D. Swisher, W. Han, R. Singh, L. Wu, V. Pchelin, T. Munkhdalai, D. Alon, G. Bacon, E. Robles, J. Bulian, M. Johnson, G. Powell, F. T. Ferreira, Y. Li, F. Benzing, M. VelimiroviÄ, H. Soyer, W. Kong, Tony, NguyĂȘn, Z. Yang, J. Liu, J. van Amersfoort, D. Gillick, B. Sun, N. Rauschmayr, K. Zhang, S. Zhan, T. Zhou, A. Frolov, C. Yang, D. Vnukov, L. Rouillard, H. Li, A. Mandhane, N. Fallen, R. Venkataraman, C. H. Hu, J. Brennan, J. Lee, J. Chang, M. Sundermeyer, Z. Pan, R. Ke, S. Tong, A. Fabrikant, W. Bono, J. Gu, R. Foley, Y. Mao, M. Delakis, D. Bhaswar, R. Frostig, N. Li, A. Zipori, C. Hope, O. Kozlova, S. Mishra, J. Djolonga, C. Schiff, M. A. Merey, E. Briakou, P. Morgan, A. Wan, A. Hassidim, R. Skerry-Ryan, K. Sengupta, M. Jasarevic, P. Kallakuri, P. Kunkle, H. Brennan, T. Lieber, H. Mansoor, J. Walker, B. Zhang, A. Xie, G. ĆœuĆŸiÄ, A. Chukwuka, A. Druinsky, D. Cho, R. Yao, F. Naeem, S. Butt, E. Kim, Z. Jia, M. Jordan, A. Lelkes, M. Kurzeja, S. Wang, J. Zhao, A. Over, A. Chakladar, M. Prasetya, N. Jha, S. Ganapathy, Y. Cong, P. Shroff, C. Saroufim, S. Miryoosefi, M. Hammad, T. Nasir, W. Xi, Y. Gao, Y. Maeng, B. Hora, C. Cheng, P. Haghani, Y. Lewenberg, C. Lu, M. Matysiak, N. Raisinghani, H. Wang, L. Baugher, R. Sukthankar, M. Giang, J. Schultz, N. Fiedel, M. Chen, C. Lee, T. Dey, H. Zheng, S. Paul, C. Smith, A. Ly, Y. Wang, R. Bansal, B. Perz, S. Ricco, S. Blank, V. Keshava, D. Sharma, M. Chow, K. Lad, K. Jalan, S. Osindero, C. Swanson, J. Scott, A. IliÄ, X. Li, S. R. Jonnalagadda, A. S. Soudagar, Y. Xiong, B. Batsaikhan, D. Jarrett, N. Kumar, M. Shah, M. Lawlor, A. Waters, M. Graham, R. May, S. Ramos, S. Lefdal, Z. Cankara, N. Cano, B. OâDonoghue, J. Borovik, F. Liu, J. Grimstad, M. Alnahlawi, K. Tsihlas, T. Hudson, N. Grigorev, Y. Jia, T. Huang, T. P. Igwe, S. Lebedev, X. Tang, I. Krivokon, F. Garcia, M. Tan, E. Jia, P. Stys, S. Vashishth, Y. Liang, B. Venkatraman, C. Gu, A. Kementsietsidis, C. Zhu, J. Jung, Y. Bai, M. J. Hosseini, F. Ahmed, A. Gupta, X. Yuan, S. Ashraf, S. Nigam, G. Vasudevan, P. Awasthi, A. M. Gilady, Z. Mariet, R. Eskander, H. Li, H. Hu, G. Garrido, P. Schlattner, G. Zhang, R. Saxena, P. DeviÄ, K. Muralidharan, A. Murthy, Y. Zhou, M. Choi, A. Wongpanich, Z. Wang, P. Shah, Y. Xu, Y. Huang, S. Spencer, A. Chen, J. Cohan, J. Wang, J. Tompson, J. Wu, R. Haroun, H. Li, B. Huergo, F. Yang, T. Yin, J. Wendt, M. Bendersky, R. Chaabouni, J. Snaider, J. Ferret, A. Jindal, T. Thompson, A. Xue, W. Bishop, S. M. Phal, A. Sharma, Y. Sung, P. Radhakrishnan, M. Shomrat, R. Ingle, R. Vij, J. Gilmer, M. D. Istin, S. Sobell, Y. Lu, E. Nottage, D. Sadigh, J. Willcock, T. Zhang, S. Xu, S. Brown, K. Lee, G. Wang, Y. Zhu, Y. Tay, C. Kim, A. Gutierrez, A. Sharma, Y. Xian, S. Seo, C. Cui, E. Pochernina, C. Baetu, K. JastrzÄbski, M. Ly, M. Elhawaty, D. Suh, E. Sezener, P. Wang, N. Yuen, G. Tucker, J. Cai, Z. Yang, C. Wang, A. Muzio, H. Qian, J. Yoo, D. Lockhart, K. R. McKee, M. Guo, M. Mehrotra, A. Mendonça, S. V. Mehta, S. Ben, C. Tekur, J. Mu, M. Zhu, V. Krakovna, H. Lee, A. Maschinot, S. Cevey, H. Choe, A. Bai, H. Srinivasan, D. Gasaway, N. Young, P. Siegler, D. Holtmann-Rice, V. Piratla, K. Baumli, R. Yogev, A. Hofer, H. van Hasselt, S. Grant, Y. Chervonyi, D. Silver, A. Hogue, A. Agarwal, K. Wang, P. Singh, F. Flynn, J. Lipschultz, R. David, L. Bellot, Y. Yang, L. Le, F. Graziano, K. Olszewska, K. Hui, A. Maurya, N. Parotsidis, W. Chen, T. Oguntebi, J. Kelley, A. Baddepudi, J. Mauerer, G. Shaw, A. Siegman, L. Yang, S. Shetty, S. Roy, Y. Song, W. Stokowiec, R. Burnell, O. Savant, R. Busa-Fekete, J. Miao, S. Ghosh, L. MacDermed, P. Lippe, M. Dektiarev, Z. Behrman, F. Mentzer, K. Nguyen, M. Wei, S. Verma, C. Knutsen, S. Dasari, Z. Yan, P. Mitrichev, X. Wang, V. Shejwalkar, J. Austin, S. Sunkara, N. Potti, Y. Virin, C. Wright, G. Liu, O. Riva, E. Pot, G. Kochanski, Q. Le, G. Balasubramaniam, A. Dhar, Y. Liao, A. Bloniarz, D. Shukla, E. Cole, J. Lee, S. Zhang, S. Kafle, S. Vashishtha, P. Mahmoudieh, G. Chen, R. Hoffmann, P. Srinivasan, A. D. Lago, Y. B. Shalom, Z. Wang, M. Elabd, A. Sharma, J. Oh, S. Kothawade, M. Le, M. Monteiro, S. Yang, K. Alarakyia, R. Geirhos, D. Mincu, H. Garnes, H. Kobayashi, S. Mariooryad, K. Krasowiak, Zhixin, Lai, S. Mourad, M. Wang, F. Bu, O. Aharoni, G. Chen, A. Goyal, V. Zubov, A. Bapna, E. Dabir, N. Kothari, K. Lamerigts, N. D. Cao, J. Shar, C. Yew, N. Kulkarni, D. Mahaarachchi, M. Joshi, Z. Zhu, J. Lichtarge, Y. Zhou, H. Muckenhirn, V. Selo, O. Vinyals, P. Chen, A. Brohan, V. Mehta, S. Cogan, R. Wang, T. Geri, W. Ko, W. Chen, F. Viola, K. Shivam, L. Wang, M. C. Elish, R. A. Popa, S. Pereira, J. Liu, R. Koster, D. Kim, G. Zhang, S. Ebrahimi, P. Talukdar, Y. Zheng, P. Poklukar, A. Mikhalap, D. Johnson, A. Vijayakumar, M. Omernick, M. Dibb, A. Dubey, Q. Hu, A. Suman, V. Aggarwal, I. Kornakov, F. Xia, W. Lowe, A. Kolganov, T. Xiao, V. Nikolaev, S. Hemingray, B. Li, J. Iljazi, M. RybiĆski, B. Sandhu, P. Lu, T. Luong, R. Jenatton, V. Govindaraj, Hui, Li, G. Dulac-Arnold, W. Park, H. Wang, A. Modi, J. Pouget-Abadie, K. Greller, R. Gupta, R. Berry, P. Ramachandran, J. Xie, L. McCafferty, J. Wang, K. Gupta, H. Lim, B. BrataniÄ, A. Brock, I. Akolzin, J. Sproch, D. Karliner, D. Kim, A. Goedeckemeyer, N. Shazeer, C. Schmid, D. Calandriello, P. Bhatia, K. Choromanski, C. Montgomery, D. Dua, A. Ramalho, H. King, Y. Gao, L. Nguyen, D. Lindner, D. Pitta, O. Johnson, K. Salama, D. Ardila, M. Han, E. Farnese, S. Odoom, Z. Wang, X. Ding, N. Rink, R. Smith, H. T. Lehri, E. Cohen, N. Vats, T. He, P. Gopavarapu, A. Paszke, M. Patel, W. V. Gansbeke, L. Loher, L. Castro, M. Voitovich, T. von Glehn, N. George, S. Niklaus, Z. Eaton-Rosen, N. RakiÄeviÄ, E. Jue, S. Perel, C. Zhang, Y. Bahat, A. Pouget, Z. Xing, F. Huot, A. Shenoy, T. Bos, V. Coriou, B. Richter, N. Noy, Y. Wang, S. Ontanon, S. Qin, G. Makarchuk, D. Hassabis, Z. Li, M. Sharma, K. Venkatesan, I. Kemaev, R. Daniel, S. Huang, S. Shah, O. Ponce, Warren, Chen, M. Faruqui, J. Wu, S. AndaÄiÄ, S. Payrits, D. McDuff, T. Hume, Y. Cao, M. Tessler, Q. Wang, Y. Wang, I. Rendulic, E. Agustsson, M. Johnson, T. Lando, A. Howard, S. G. S. Padmanabhan, M. Daswani, A. Banino, M. Kilgore, J. Heek, Z. Ji, A. Caceres, C. Li, N. Kassner, A. Vlaskin, Z. Liu, A. Grills, Y. Hou, R. Sukkerd, G. Cheon, N. Shetty, L. Markeeva, P. Stanczyk, T. Iyer, Y. Gong, S. Gao, K. Gopalakrishnan, T. Blyth, M. Reynolds, A. Bhoopchand, M. Bilenko, D. Gharibian, V. Zayats, A. Faust, A. Singh, M. Ma, H. Jiao, S. Vijayanarasimhan, L. Aroyo, V. Yadav, S. Chakera, A. Kakarla, V. Meshram, K. Gregor, G. Botea, E. Senter, D. Jia, G. Kovacs, N. Sharma, S. Baur, K. Kang, Y. He, L. Zhuo, M. Kostelac, I. Laish, S. Peng, L. OâBryan, D. Kasenberg, G. R. Rao, E. Leurent, B. Zhang, S. Stevens, A. Salazar, Y. Zhang, I. Lobov, J. Walker, A. Porter, M. Redshaw, H. Ke, A. Rao, A. Lee, H. Lam, M. Moffitt, J. Kim, S. Qiao, T. Koo, R. Dadashi, X. Song, M. Sundararajan, P. Xu, C. Kawamoto, Y. Zhong, C. Barbu, A. Reddy, M. Verzetti, L. Li, G. Papamakarios, H. Klimczak-PluciĆska, M. Cassin, K. Kavukcuoglu, R. Swavely, A. Vaucher, J. Zhao, R. Hemsley, M. Tschannen, H. Ge, G. Menghani, Y. Yu, N. Ha, W. He, X. Wu, M. Song, R. Sterneck, S. Zinke, D. A. Calian, A. Marsden, A. C. Ruiz, M. Hessel, A. Gueta, B. Lee, B. Farris, M. Gupta, Y. Li, M. Saleh, V. Misra, K. Xiao, P. Mendolicchio, G. Buttimore, V. Krayvanova, N. Nayakanti, M. Wiethoff, Y. Pande, A. Mirhoseini, N. Lao, J. Liu, Y. Hua, A. Chen, Y. Malkov, D. Kalashnikov, S. Gupta, K. Audhkhasi, Y. Zhai, S. Kopalle, P. Jain, E. Ofek, C. Meyer, K. Baatarsukh, H. StrejÄek, J. Qian, J. Freedman, R. Figueira, M. Sokolik, O. Bachem, R. Lin, D. Kharrat, C. Hidey, P. Xu, D. Duan, Y. Li, M. Ersoy, R. Everett, K. Cen, R. Santamaria-Fernandez, A. Taubenfeld, I. Mackinnon, L. Deng, P. Zablotskaia, S. Viswanadha, S. Goel, D. Yates, Y. Deng, P. Choy, M. Chen, A. Sinha, A. Mossin, Y. Wang, A. Szlam, S. Hao, P. K. Rubenstein, M. Toksoz-Exley, M. Aperghis, Y. Zhong, J. Ahn, M. Isard, O. Lacombe, F. Luisier, C. Anastasiou, Y. Kalley, U. Prabhu, E. Dunleavy, S. Bijwadia, J. Mao-Jones, K. Chen, R. Pasumarthi, E. Wood, A. Dostmohamed, N. Hurley, J. Simsa, A. Parrish, M. Pajarskas, M. Harvey, O. Skopek, Y. Kochinski, J. Rey, V. Rieser, D. Zhou, S. J. Lee, T. Acharya, G. Li, J. Jiang, X. Zhang, B. Gipson, E. Mahintorabi, M. Gelmi, N. Khajehnouri, A. Yeh, K. Lee, L. Matthey, L. Baker, T. Pham, H. Fu, A. Pak, P. Gupta, C. Vasconcelos, A. Sadovsky, B. Walker, S. Hsiao, P. Zochbauer, A. Marzoca, N. Velan, J. Zeng, G. Baechler, D. Driess, D. Jain, Y. Huang, L. Tao, J. Maggs, N. Levine, J. Schneider, E. Gemzer, S. Petit, S. Han, Z. Fisher, D. Zelle, C. Biles, E. Ie, A. Fadeeva, C. Liu, J. V. Franco, A. Collister, H. Zhang, R. Wang, R. Zhao, L. Kieliger, K. Shuster, R. Zhu, B. Gong, L. Chan, R. Sun, S. Basu, R. Zimmermann, J. Hayes, A. Bapna, J. Snoek, W. Yang, P. Datta, J. A. Abdallah, K. Kilgour, L. Li, S. Mah, Y. Jun, M. RiviĂšre, A. Karmarkar, T. Spalink, T. Huang, L. Gonzalez, D. Tran, A. Nowak, J. Palowitch, M. Chadwick, E. Talius, H. Mehta, T. Sellam, P. FrĂ€nken, M. Nicosia, K. He, A. Kini, D. Amos, S. Basu, H. Jobe, E. Shaw, Q. Xu, C. Evans, D. Ikeda, C. Yan, L. Jin, L. Wang, S. Yadav, I. Labzovsky, R. Sampath, A. Ma, C. Schumann, A. Siddhant, R. Shah, J. Youssef, R. Agarwal, N. Dabney, A. Tonioni, M. Ambar, J. Li, I. Guyon, B. Li, D. Soergel, B. Fang, G. Karadzhov, C. Udrescu, T. Trinh, V. Raunak, S. Noury, D. Guo, S. Gupta, M. Finkelstein, D. Petek, L. Liang, G. Billock, P. Sun, D. Wood, Y. Song, X. Yu, T. Matejovicova, R. Cohen, K. Andra, D. DâAmbrosio, Z. Deng, V. Nallatamby, E. Songhori, R. Dangovski, A. Lampinen, P. Botadra, A. Hillier, J. Cao, N. Baddi, A. Kuncoro, T. Yoshino, A. Bhagatwala, M. Ranzato, R. Schaeffer, T. Liu, S. Ye, O. Sarvana, J. Nham, C. Kuang, I. Gao, J. Baek, S. Mittal, A. Wahid, A. Gergely, B. Ni, J. Feldman, C. Muir, P. Lamblin, W. Macherey, E. Dyer, L. Kilpatrick, V. Campos, M. Bhutani, S. Fort, Y. Ahmad, A. Severyn, K. Chatziprimou, O. Ferludin, M. Dimarco, A. Kusupati, J. Heyward, D. Bahir, K. Villela, K. Millican, D. Marcus, S. Bahargam, C. Unlu, N. Roth, Z. Wei, S. Gopal, D. Ghoshal, E. Lee, S. Lin, J. Lees, D. Lee, A. Hosseini, C. Fan, S. Neel, M. Wu, Y. Altun, H. Cai, E. Piqueras, J. Woodward, A. Bissacco, S. Haykal, M. Bordbar, P. Sundaram, S. Hodkinson, D. Toyama, G. Polovets, A. Myers, A. Sinha, T. Levinboim, K. Krishnakumar, R. Chhaparia, T. Sholokhova, N. B. Gundavarapu, G. Jawahar, H. Qureshi, J. Hu, N. Momchev, M. Rahtz, R. Wu, A. P. S, K. Dhamdhere, M. Guo, U. Gupta, A. Eslami, M. Schain, M. Blokzijl, D. Welling, D. Orr, L. Bolelli, N. Perez-Nieves, M. Sirotenko, A. Prasad, A. Kar, B. D. B. Pigem, T. Terzi, G. Weisz, D. Ghosh, A. Mavalankar, D. Madeka, K. Daugaard, H. Adam, V. Shah, D. Berman, M. Tran, S. Baker, E. Andrejczuk, G. Chole, G. Raboshchuk, M. Mirzazadeh, T. Kagohara, S. Wu, C. Schallhart, B. Orlando, C. Wang, A. Rrustemi, H. Xiong, H. Liu, A. Vezer, N. Ramsden, S. Chang, S. Mudgal, Y. Li, N. Vieillard, Y. Hoshen, F. Ahmad, A. Slone, A. Hua, N. Potikha, M. Rossini, J. Stritar, S. Prakash, Z. Wang, X. Dong, A. Nazari, E. Nehoran, K. Tekelioglu, Y. Li, K. Badola, T. Funkhouser, Y. Li, V. Yerram, R. Ganeshan, D. Formoso, K. Langner, T. Shi, H. Li, Y. Yamamori, A. Panda, A. Saade, A. S. Scarpati, C. Breaux, C. Carey, Z. Zhou, C. Hsieh, S. Bridgers, A. Butryna, N. Gupta, V. Tulsyan, S. Woo, E. Eltyshev, W. Grathwohl, C. Parks, S. Benjamin, R. Panigrahy, S. Dodhia, D. D. Freitas, C. Sauer, W. Song, F. Alet, J. Tolins, C. Paduraru, X. Zhou, B. Albert, Z. Zhang, L. Shu, M. Bansal, S. Nguyen, A. Globerson, O. Xiao, J. Manyika, T. Hennigan, R. Rong, J. Matak, A. Bakalov, A. Sharma, D. Sinopalnikov, A. Pierson, S. Roller, G. Brown, M. Gao, T. Fukuzawa, A. Ghafouri, K. Vassigh, I. Barr, Z. Wang, A. Korsun, R. Jayaram, L. Ren, T. Zaman, S. Khan, Y. Lunts, D. Deutsch, D. Uthus, N. Katz, M. Samsikova, A. Khalifa, N. Sethi, J. Sun, L. Tang, U. Alon, X. Luo, D. Yu, A. Nayyar, B. Petrini, W. Truong, V. Hellendoorn, N. Chinaev, C. Alberti, W. Wang, J. Hu, V. Mirrokni, A. Balashankar, A. Aharon, A. Mehta, A. Iscen, J. Kready, L. Manning, A. Mohananey, Y. Chen, A. Tripathi, A. Wu, I. Petrovski, D. Hwang, M. Baeuml, S. Chandrakaladharan, Y. Liu, R. Coaguila, M. Chen, S. Ma, P. Tafti, S. Tatineni, T. Spitz, J. Ye, P. Vicol, M. Rosca, A. PuigdomĂšnech, Z. Yahav, S. Ghemawat, H. Lin, P. Kirk, Z. Nabulsi, S. Brin, B. Bohnet, K. Caluwaerts, A. S. Veerubhotla, D. Zheng, Z. Dai, P. Petrov, Y. Xu, R. Mehran, Z. Xu, L. Zintgraf, J. Choi, S. A. Hombaiah, R. Thoppilan, S. Reddi, L. Lew, L. Li, K. Webster, K. Sawhney, L. Lamprou, S. Shakeri, M. Lunayach, J. Chen, S. Bagri, A. Salcianu, Y. Chen, Y. Donchev, C. Magister, S. NĂžrly, V. Rodrigues, T. Izo, H. Noga, J. Zou, T. Köppe, W. Zhou, K. Lee, X. Long, D. Eisenbud, A. Chen, C. Schenck, C. M. To, P. Zhong, E. Taropa, M. Truong, O. Levy, D. Martins, Z. Zhang, C. Semturs, K. Zhang, A. Yakubovich, P. Moreno, L. McConnaughey, D. Lu, S. Redmond, L. Weerts, Y. Bitton, T. Refice, N. Lacasse, A. Conmy, C. Tallec, J. Odell, H. Forbes-Pollard, A. Socala, J. Hoech, P. Kohli, A. Walton, R. Wang, M. Sazanovich, K. Zhu, A. Kapishnikov, R. Galt, M. Denton, B. Murdoch, C. Sikora, K. Mohamed, W. Wei, U. First, T. McConnell, L. C. Cobo, J. Qin, T. Avrahami, D. Balle, Y. Watanabe, A. Louis, A. Kraft, S. Ariafar, Y. Gu, E. Rives, C. Yoon, A. Rusu, J. Cobon-Kerr, C. Hahn, J. Luo, Yuvein, Zhu, N. Ahuja, R. Benenson, R. L. Kaufman, H. Yu, L. Hightower, J. Zhang, D. Ni, L. A. Hendricks, G. Wang, G. Yona, L. Jain, P. Barrio, S. Bhupatiraju, S. Velusamy, A. Dafoe, S. Riedel, T. Thomas, Z. Yuan, M. Bellaiche, S. Panthaplackel, K. Kloboves, S. Jauhari, C. Akbulut, T. Davchev, E. Gladchenko, D. Madras, A. Chuklin, T. Hill, Q. Yuan, M. Madhavan, L. Leonhard, D. Scandinaro, Q. Chen, N. Niu, A. Douillard, B. Damoc, Y. Onoe, F. Pedregosa, F. Bertsch, C. Leichner, J. Pagadora, J. Malmaud, S. Ponda, A. Twigg, O. Duzhyi, J. Shen, M. Wang, R. Garg, J. Chen, U. Evci, J. Lee, L. Liu, K. Kojima, M. Yamaguchi, A. Rajendran, A. Piergiovanni, V. K. Rajendran, M. Fornoni, G. Ibagon, H. Ragan, S. M. Khan, J. Blitzer, A. Bunner, G. Sun, T. Kosakai, S. Lundberg, N. Elue, K. Guu, S. Park, J. Park, A. Narayanaswamy, C. Wu, J. Mudigonda, T. Cohn, H. Mu, R. Kumar, L. Graesser, Y. Zhang, R. Killam, V. Zhuang, M. GimĂ©nez, W. A. Jishi, R. Ley-Wild, A. Zhai, K. Osawa, D. Cedillo, J. Liu, M. Upadhyay, M. Sieniek, R. Sharma, T. Paine, A. Angelova, S. Addepalli, C. Parada, K. Majumder, A. Lamp, S. Kumar, X. Deng, A. Myaskovsky, T. SaboliÄ, J. Dudek, S. York, F. de Chaumont Quitry, J. Nie, D. Cattle, A. Gunjan, B. Piot, W. Khawaja, S. Bang, S. Wang, S. Khodadadeh, R. R, P. Rawlani, R. Powell, K. Lee, J. Griesser, G. Oh, C. Magalhaes, Y. Li, S. Tokumine, H. N. Vogel, D. Hsu, A. BC, D. Jindal, M. Cohen, Z. Yang, J. Yuan, D. de Cesare, T. Bruguier, J. Xu, M. Roy, A. Jacovi, D. Belov, R. Arya, P. Meadowlark, S. Cohen-Ganor, W. Ye, P. Morris-Suzuki, P. Banzal, G. Song, P. Ponnuramu, F. Zhang, G. Scrivener, S. Zaiem, A. R. Rochman, K. Han, B. Ghazi, K. Lee, S. Drath, D. Suo, A. Girgis, P. Shenoy, D. Nguyen, D. Eck, S. Gupta, L. Yan, J. Carreira, A. Gulati, R. Sang, D. Mirylenka, E. Cooney, E. Chou, M. Ling, C. Fan, B. Coleman, G. Tubone, R. Kumar, J. Baldridge, F. Hernandez-Campos, A. Lazaridou, J. Besley, I. Yona, N. Bulut, Q. Wellens, A. Pierigiovanni, J. George, R. Green, P. Han, C. Tao, G. Clark, C. You, A. Abdolmaleki, J. Fu, T. Chen, A. Chaugule, A. Chandorkar, A. Rahman, W. Thompson, P. Koanantakool, M. Bernico, J. Ren, A. Vlasov, S. Vassilvitskii, M. Kula, Y. Liang, D. Kim, Y. Huang, C. Ye, D. Lepikhin, and W. Helmholz Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. External Links: 2507.06261, Link Cited by: Table A4, §4.1. Croxford et al. (2025) E. Croxford, Y. Gao, N. Pellegrino, K. Wong, G. Wills, E. First, M. Schnier, K. Burton, C. Ebby, J. Gorski, M. Kalscheur, S. Khalil, M. Pisani, T. Rubeor, P. Stetson, F. Liao, C. Goswami, B. Patterson, and M. Afshar Development and validation of the provider documentation summarization quality instrument for large language models. Journal of the American Medical Informatics Association 32 (6), p. 1050â1060 (en). External Links: ISSN 1067-5027, 1527-974X, Link, Document Cited by: §1, §7.2. Cui et al. (2025) H. Cui, A. Unell, B. Chen, J. A. Fries, E. Alsentzer, S. Koyejo, and N. H. Shah TIMER: temporal instruction modeling and evaluation for longitudinal clinical records. npj Digital Medicine 8 (1), p. 577 (en). External Links: ISSN 2398-6352, Link, Document Cited by: §1. de Hond et al. (2024) A. de Hond, T. Leeuwenberg, R. Bartels, M. van Buchem, I. Kant, K. G. Moons, and M. van Smeden From text to treatment: the crucial role of validation for generative large language models in health care. The Lancet Digital Health 6 (7), p. e441âe443. Note: doi: 10.1016/S2589-7500(24)00111-0 External Links: ISSN 2589-7500, Link, Document Cited by: §4.2. Diekmann et al. (2025) Y. Diekmann, C. Fensore, R. Carrillo-Larco, E. Castejon Rosales, S. Shiromani, R. Pai, M. Shah, and J. Ho LLMs as Medical Safety Judges: Evaluating Alignment with Human Annotation in Patient-Facing QA. In Proceedings of the 24th Workshop on Biomedical Language Processing, Viena, Austria, p. 217â224 (en). External Links: Link, Document Cited by: §8. Dror et al. (2018) R. Dror, G. Baumer, S. Shlomov, and R. Reichart The Hitchhikerâs Guide to Testing Statistical Significance in Natural Language Processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, p. 1383â1392 (en). External Links: Link, Document Cited by: §4.2. Engel (1977) G. L. Engel The Need for a New Medical Model: A Challenge for Biomedicine. Science 196 (4286), p. 129â136 (en). External Links: ISSN 0036-8075, 1095-9203, Link, Document Cited by: §1. Frattallone-Llado et al. (2024) G. Frattallone-Llado, J. Kim, C. Cheng, D. Salazar, S. Edakalavan, and J. C. Weiss Using Multimodal Data to Improve Precision of Inpatient Event Timelines. In Advances in Knowledge Discovery and Data Mining, D. Yang, X. Xie, V. S. Tseng, J. Pei, J. Huang, and J. C. Lin (Eds.), Vol. 14648, p. 322â334 (en). Note: Series Title: Lecture Notes in Computer Science External Links: ISBN 978-981-97-2240-2 978-981-97-2238-9, Link, Document Cited by: §7.1, §7.2. Gao et al. (2022) Y. Gao, D. Dligach, T. Miller, D. Xu, M. M. M. Churpek, and M. Afshar Summarizing patientsâ problems from hospital progress notes using pre-trained sequence-to-sequence models. In Proceedings of the 29th International Conference on Computational Linguistics, N. Calzolari, C. Huang, H. Kim, J. Pustejovsky, L. Wanner, K. Choi, P. Ryu, H. Chen, L. Donatelli, H. Ji, S. Kurohashi, P. Paggio, N. Xue, S. Kim, Y. Hahm, Z. He, T. K. Lee, E. Santus, F. Bond, and S. Na (Eds.), Gyeongju, Republic of Korea, p. 2979â2991. External Links: Link Cited by: §1. Gemma Team (2026) Gemma Team Gemma 4 Technical Report. External Links: 2607.02770, Link Cited by: Table A4, §4.1, §6.1. Grothey et al. (2025) B. Grothey, J. Odenkirchen, A. Brkic, B. Schömig-Markiefka, A. Quaas, R. BĂŒttner, and Y. Tolkach Comprehensive testing of large language models for extraction of structured data in pathology. Communications Medicine 5 (1), p. 96 (en). External Links: ISSN 2730-664X, Link, Document Cited by: §8. Gumiel et al. (2022) Y. B. Gumiel, L. E. Silva E Oliveira, V. Claveau, N. Grabar, E. C. Paraiso, C. Moro, and D. R. Carvalho Temporal Relation Extraction in Clinical Texts: A Systematic Review. ACM Computing Surveys 54 (7), p. 1â36 (en). External Links: ISSN 0360-0300, 1557-7341, Link, Document Cited by: §1, §5.1. Gwet (2008) K. L. Gwet Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61 (1), p. 29â48. External Links: Document Cited by: §5.2. Hager et al. (2024) P. Hager, F. Jungmann, R. Holland, K. Bhagat, I. Hubrecht, M. Knauer, J. Vielhauer, M. Makowski, R. Braren, G. Kaissis, and D. Rueckert Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine 30 (9), p. 2613â2622 (en). External Links: ISSN 1078-8956, 1546-170X, Link, Document Cited by: §8. Jiang et al. (2026) Z. Jiang, H. Chen, Y. Wu, Y. Qin, C. Pei, D. Zeng, B. Sheng, and T. Y. Wong Beyond multiple-choice questions: Rethinking evaluation frameworks for large language models for clinical medicine. Intelligent Medicine 6 (2), p. 109â115 (en). External Links: ISSN 26671026, Link, Document Cited by: §8. Jin et al. (2024) Y. Jin, M. Chandra, G. Verma, Y. Hu, M. De Choudhury, and S. Kumar Better to ask in english: cross-lingual evaluation of large language models for healthcare queries. In Proceedings of the ACM Web Conference 2024, W â24, New York, NY, USA, p. 2627â2638. External Links: ISBN 9798400701719, Link, Document Cited by: Appendix B. Johnson et al. (2016) A. E.W. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark MIMIC-I, a freely accessible critical care database. Scientific Data 3 (1), p. 160035 (en). External Links: ISSN 2052-4463, Link, Document Cited by: §2. Johnson et al. (2015) A. Johnson, T. Pollard, and R. Mark MIMIC-I Clinical Database. PhysioNet. External Links: Link, Document Cited by: §2. K et al. (2025) K. K, R. Thirukovalluru, and D. Carlson ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, Mumbai, India, p. 2822â2836 (en). External Links: Link, Document Cited by: §1. Ki et al. (2025) D. Ki, M. Carpuat, P. McNamee, D. Khashabi, E. Yang, D. Lawrie, and K. Duh Linguistic nepotism: trading-off quality for language preference in multilingual rag. arXiv preprint arXiv:2509.13930. Cited by: §4.2. Kim et al. (2024) M. K. Kim, C. Rouphael, J. McMichael, N. Welch, and S. Dasarathy Challenges in and Opportunities for Electronic Health Record-Based Data Analysis and Interpretation. Gut and Liver 18 (2), p. 201â208 (en). External Links: ISSN 1976-2283, 2005-1212, Link, Document Cited by: §1. Kraljevic et al. (2024) Z. Kraljevic, D. Bean, A. Shek, R. Bendayan, H. Hemingway, J. A. Yeung, A. Deng, A. Balston, J. Ross, E. Idowu, J. T. Teo, and R. J. B. Dobson Foresightâa generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study. The Lancet Digital Health 6 (4), p. e281âe290 (en). External Links: ISSN 25897500, Link, Document Cited by: §1. Kruse et al. (2025) M. Kruse, S. Hu, N. Derby, Y. Wu, S. Stonbraker, B. Yao, D. Wang, E. M. Goldberg, and Y. Gao Large language models with temporal reasoning for longitudinal clinical summarization and prediction. Findings of ACL. EMNLP. Conference on Empirical Methods in Natural Language Processing 2025, p. 20715 â 20735. External Links: Document Cited by: §1. Kumar et al. (2026) S. Kumar, S. Noroozizadeh, J. Kim, and J. C. Weiss Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment. arXiv. Note: Version Number: 1Other Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim (authors contributed equally) External Links: Link, Document Cited by: §1, §7.2. Lee et al. (2024) C. Lee, K. A. Vogt, and S. Kumar Prospects for AI clinical summarization to reduce the burden of patient chart review. Frontiers in Digital Health 6, p. 1475092. External Links: ISSN 2673-253X, Link, Document Cited by: §1, §7.2. Leeuwenberg and Moens (2020) A. Leeuwenberg and M. Moens Towards Extracting Absolute Event Timelines From English Clinical Reports. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, p. 2710â2719. External Links: ISSN 2329-9290, 2329-9304, Link, Document Cited by: §7.1. Li et al. (2023) J. Li, X. Cheng, X. Zhao, J. Nie, and J. Wen HaluEval: a large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 6449â6464. External Links: Link, Document Cited by: Appendix E. Linhares et al. (2023) C. D. G. Linhares, D. M. Lima, J. R. Ponciano, M. M. Olivatto, M. A. Gutierrez, J. Poco, C. Traina, and A. J. M. Traina ClinicalPath: A Visualization Tool to Improve the Evaluation of Electronic Health Records in Clinical Decision-Making. IEEE Transactions on Visualization and Computer Graphics 29 (10), p. 4031â4046. External Links: ISSN 1077-2626, 1941-0506, 2160-9306, Link, Document Cited by: §1, §1. Loftus et al. (2024) T. J. Loftus, J. A. Balch, J. L. Marquard, J. M. Ray, B. S. Alper, N. Ojha, A. Bihorac, G. Melton-Meaux, G. Khanna, and C. J. Tignanelli Longitudinal clinical decision support for assessing decisions over time: State-of-the-art and future directions. DIGITAL HEALTH 10, p. 20552076241249925 (en). External Links: ISSN 2055-2076, 2055-2076, Link, Document Cited by: §1. Luo et al. (2023) Z. Luo, Y. Ji, A. Gupta, Z. Li, A. Frisch, and D. He Towards Accurate and Clinically Meaningful Summarization of Electronic Health Record Notes: A Guided Approach. In 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Pittsburgh, PA, USA, p. 1â5. External Links: ISBN 979-8-3503-1050-4, Link, Document Cited by: §7.2. Makarov et al. (2025) N. Makarov, M. Bordukova, P. Quengdaeng, D. Garger, R. Rodriguez-Esteban, F. Schmich, and M. P. Menden Large language models forecast patient health trajectories enabling digital twins. npj Digital Medicine 8 (1), p. 588 (en). External Links: ISSN 2398-6352, Link, Document Cited by: §1. MeskĂł and Topol (2023) B. MeskĂł and E. J. Topol The imperative for regulatory oversight of large language models (or generative AI) in healthcare. npj Digital Medicine 6 (1), p. 120 (en). External Links: ISSN 2398-6352, Link, Document Cited by: §4.2. Meta AI (2024) Meta AI Llama 3.3 70B Instruct model card. Note: Hugging Face model cardAccessed July 27, 2026 External Links: Link Cited by: Table A4, §4.1. Mistral AI (2025) Mistral AI Mistral-Small-3.2-24B-Instruct-2506 model card. Note: Hugging Face model cardAccessed July 27, 2026 External Links: Link Cited by: Table A4, §4.1, §6.1. Mittal et al. (2026) S. Mittal, E. Kasson, L. Paraboschi, E. Laufenberg, J. Zhou, P. A. Cavazos-Rehg, T. Mitra, and M. De Choudhury Follow the rules (or not): community norms and AI-generated support in online health communities. In Proceedings of the International AAAI Conference on Web and Social Media, Note: To appear External Links: Link Cited by: §4.2. Moharasan and Ho (2019) G. Moharasan and T. Ho Extraction of Temporal Information from Clinical Narratives. Journal of Healthcare Informatics Research 3 (2), p. 220â244 (en). External Links: ISSN 2509-4971, 2509-498X, Link, Document Cited by: §1. Noroozizadeh et al. (2025) S. Noroozizadeh, S. Kumar, G. H. Chen, and J. C. Weiss Temporally annotated textual time series from PubMed Open Access clinical case reports. Health Informatics (en). External Links: Link, Document Cited by: §1, §7.2. Ntinopoulos et al. (2025) V. Ntinopoulos, H. Rodriguez Cetina Biefer, I. Tudorache, N. Papadopoulos, D. Odavic, P. Risteski, A. Haeussler, and O. Dzemali Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation. BMJ Health & Care Informatics 32 (1), p. e101139 (en). External Links: ISSN 2632-1009, Link, Document Cited by: §8. Olex and McInnes (2021) A. L. Olex and B. T. McInnes Review of Temporal Reasoning in the Clinical Domain for Timeline Extraction: Where we are and where we need to be. Journal of Biomedical Informatics 118, p. 103784 (en). External Links: ISSN 15320464, Link, Document Cited by: §1, §1. OpenMeditron (2025) OpenMeditron Meditron3-Qwen2.5-7B model card. Note: Hugging Face model cardAccessed July 27, 2026 External Links: Link Cited by: Table A4, §4.1, §6.1. Qwen Team (2025) Qwen Team Qwen3 technical report. External Links: 2505.09388, Link Cited by: §6.1. Qwen Team (2026) Qwen Team Qwen3.6-35B-A3B: agentic coding power, now open to all. External Links: Link Cited by: Table A4, §4.1, §6.1. Ruan et al. (2019) T. Ruan, L. Lei, Y. Zhou, J. Zhai, L. Zhang, P. He, and J. Gao Representation learning for clinical time series prediction tasks in electronic health records. BMC Medical Informatics and Decision Making 19 (S8), p. 259 (en). External Links: ISSN 1472-6947, Link, Document Cited by: §1. Saha et al. (2026) K. Saha, D. W. Yoo, V. Das Swain, and M. De Choudhury Life events as predictors of wellbeing outcomes. npj Digital Public Health 1 (1), p. 5 (en). External Links: ISSN 3091-2784, Link, Document Cited by: §1. Sarvari and Al-fagih (2025) P. Sarvari and Z. Al-fagih Rapidly Benchmarking Large Language Models for Diagnosing Comorbid Patients: Comparative Study Leveraging the LLM-as-a-Judge Method. JMIRx Med 6, p. e67661âe67661 (en). External Links: ISSN 2563-6316, Link, Document Cited by: §4.1. Seinen et al. (2025) T. M. Seinen, J. A. Kors, E. M. Van Mulligen, and P. R. Rijnbeek Using Structured Codes and Free-Text Notes to Measure Information Complementarity in Electronic Health Records: Feasibility and Validation Study. Journal of Medical Internet Research 27, p. e66910 (en). External Links: ISSN 1438-8871, Link, Document Cited by: §1. Sellergren et al. (2026) A. Sellergren, S. Kazemzadeh, T. Jaroensri, A. Kiraly, M. Traverse, T. Kohlberger, S. Xu, F. Jamil, C. Hughes, C. Lau, J. Chen, F. Mahvar, L. Yatziv, T. Chen, B. Sterling, S. A. Baby, S. M. Baby, J. Lai, S. Schmidgall, L. Yang, K. Chen, P. Bjornsson, S. Reddy, R. Brush, K. Philbrick, M. Asiedu, I. Mezerreg, H. Hu, H. Yang, R. Tiwari, S. Jansen, P. Singh, Y. Liu, S. Azizi, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. RamĂ©, M. Riviere, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Buchatskaya, J. Alayrac, D. Lepikhin, V. Feinberg, S. Borgeaud, A. Andreev, C. Hardin, R. Dadashi, L. Hussenot, A. Joulin, O. Bachem, Y. Matias, K. Chou, A. Hassidim, K. Goel, C. Farabet, J. Barral, T. Warkentin, J. Shlens, D. Fleet, V. Cotruta, O. Sanseviero, G. Martins, P. Kirk, A. Rao, S. Shetty, D. F. Steiner, C. Kirmizibayrak, R. Pilgrim, D. Golden, and L. Yang MedGemma technical report. External Links: 2507.05201, Link Cited by: Table A4, §4.1. Sharif et al. (2025) O. Sharif, J. Gatto, M. Basak, and S. M. Preum REGen: a reliable evaluation framework for generative event argument extraction. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 12146â12168. Cited by: §6.1. Shi et al. (2024) W. Shi, R. Xu, Y. Zhuang, Y. Yu, J. Zhang, H. Wu, Y. Zhu, J. C. Ho, C. Yang, and M. D. Wang EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 22315â22339 (en). External Links: Link, Document Cited by: §4.1. Styler et al. (2014) W. F. Styler, S. Bethard, S. Finan, M. Palmer, S. Pradhan, P. C. De Groen, B. Erickson, T. Miller, C. Lin, G. Savova, and J. Pustejovsky Temporal Annotation in the Clinical Domain. Transactions of the Association for Computational Linguistics 2, p. 143â154 (en). External Links: ISSN 2307-387X, Link, Document Cited by: §7.1. Sun et al. (2013) W. Sun, A. Rumshisky, and O. Uzuner Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. Journal of the American Medical Informatics Association 20 (5), p. 806â813 (en). External Links: ISSN 1067-5027, 1527-974X, Link, Document Cited by: §7.1. Umeton et al. (2024) R. Umeton, A. Kwok, R. Maurya, D. Leco, N. Lenane, J. Willcox, G. A. Abel, M. Tolikas, and J. M. Johnson GPT-4 in a Cancer Center â Institute-Wide Deployment Challenges and Lessons Learned. NEJM AI 1 (4) (en). External Links: ISSN 2836-9386, Link, Document Cited by: §4.2. Uzuner et al. (2011) Ă. Uzuner, B. R. South, S. Shen, and S. L. DuVall 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association 18 (5), p. 552â556 (en). External Links: ISSN 1527-974X, 1067-5027, Link, Document Cited by: §7.1. Van Veen et al. (2024) D. Van Veen, C. Van Uden, L. Blankemeier, J. Delbrouck, A. Aali, C. Bluethgen, A. Pareek, M. Polacin, E. P. Reis, A. SeehofnerovĂĄ, N. Rohatgi, P. Hosamani, W. Collins, N. Ahuja, C. P. Langlotz, J. Hom, S. Gatidis, J. Pauly, and A. S. Chaudhari Adapted large language models can outperform medical experts in clinical text summarization. Nature Medicine 30 (4), p. 1134â1142 (en). External Links: ISSN 1078-8956, 1546-170X, Link, Document Cited by: §8. Virtanen et al. (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, Ä°. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, p. 261â272. External Links: Document Cited by: Appendix C. Wang et al. (2025a) J. Wang, X. Niu, J. Kim, J. Shen, T. Zhang, and J. Weiss MIMIC-IV-Ext-22MCTS: A 22 Millions-Event Temporal Clinical Time-Series Dataset with Relative Timestamp for Risk Prediction. In Review. External Links: Link, Document Cited by: §7.2. Wang and Weiss (2025) J. Wang and J. C. Weiss A Large-Language Model Framework for Relative Timeline Extraction from PubMed Case Reports. AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science 2025, p. 598â606 (eng). External Links: ISSN 2153-4063 Cited by: §1, §7.2. Wang et al. (2024) L. Wang, Q. Lu, R. Li, S. Fu, and H. Liu Wonder at Chemotimelines 2024: MedTimeline: An End-to-End NLP System for Timeline Extraction from Clinical Narratives. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, Mexico, p. 483â487 (en). External Links: Link, Document Cited by: §1, §7.1, §7.2. Wang et al. (2025b) X. Wang, J. Zhang, G. Zhang, and H. Guo Feel the difference? a comparative analysis of emotional arcs in real and LLM-generated CBT sessions. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 19999â20017. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §8. World Health Organization (2025a) World Health Organization Mental disorders. Note: https://w.who.int/news-room/fact-sheets/detail/mental-disordersAccessed: 2026-07-13 Cited by: §5.5. World Health Organization (2025b) World Health Organization Mental health. Note: https://w.who.int/news-room/fact-sheets/detail/mental-health-strengthening-our-responseAccessed: 2026-07-14 Cited by: §5.5. Yim et al. (2023) W. Yim, Y. Fu, A. Ben Abacha, N. Snider, T. Lin, and M. Yetisgen ACI-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Nature Scientific Data. External Links: Link Cited by: §5.1. Yu et al. (2025) H. Yu, J. Zhou, L. Li, S. Chen, J. Gallifant, A. Shi, J. Sun, X. Li, J. He, W. Hua, M. Jin, G. Chen, Y. Zhou, Z. Li, T. Gupte, M. Chen, Z. Azizi, Q. Dou, B. P. Yan, Y. Xing, Y. Zhang, T. L. Assimes, D. S. Bitterman, X. Ma, L. Lu, and L. Fan Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education. Communications Medicine 6 (1), p. 27 (en). External Links: ISSN 2730-664X, Link, Document Cited by: §4.1. Zhang et al. (2019) T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi Bertscore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675. Cited by: §6.1. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS â23, Red Hook, NY, USA. Cited by: §5.1. Appendix A Dataset Extraction and Profile A.1 Mental Health Admissions Subset We operationalize mental-health-coded admissions using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM): an admission qualifies if it contains at least one diagnosis code from Chapter 5, Mental Disorders (290â319) (Centers for Medicare & Medicaid Services and National Center for Health Statistics, 2011) Centers for Medicare & Medicaid Services and National Center for Health Statistics (2011). Table A1 breaks down the key categories. Our framework uses only free-text discharge summaries from the MIMIC-I NOTEEVENTS table, which describe each patientâs hospital stay. We link each summary to structured MIMIC-I fields only to define the study cohort and provide date anchors: diagnosis codes from DIAGNOSES_ICD, date of birth from PATIENTS, and admission and discharge dates from ADMISSIONS. These structured fields are not used as model inputs or training labels. The discharge summaries are provided to the framework in their original form, without chunking, section detection, or text normalization. Code range Diagnostic category 290â294 Organic psychotic conditions 295â299 Other psychoses 300â316 Neurotic disorders, personality disorders, and other nonpsychotic mental disorders 317â319 Intellectual disabilities Table A1: Subcategories within ICD-9-CM Chapter 5, Mental Disorders (codes 290â319) Centers for Medicare & Medicaid Services and National Center for Health Statistics (2011). An admission is included in the mental-health-coded cohort if it contains at least one ICD-9-CM diagnosis code within the Mental Disorders chapter (290â319) A.2 Benchmark Input Text Profile Because both model evaluation and human validation require comparison against the complete source discharge summary, input length determines the amount of narrative context that must be processed and reviewed. Table A2 summarizes the raw-text lengths of the 52-document benchmark. The summaries contain 1,7741,774 words on average (median: 1,6241,624; SD: 821821), ranging from 437437 to 3,5833,583 words and totaling 92,22292,222 words. Character counts show similar variation, with a mean of 11,87111,871 characters and a range of 2,9522,952 to 24,80824,808. These statistics reflect the original discharge summaries supplied to the framework without chunking, section parsing, or text normalization, and illustrate the long-context processing and manual-review burden of the benchmark. Metric Mean Median SD Min Max Total Characters 11,871 11,042 5,481 2,952 24,808 617,267 Words 1,774 1,624 821 437 3,583 92,222 Table A2: Raw-text profile of the 52 discharge summaries in the benchmark. A.3 Token Usage for Silver-Standard Generation To quantify the inference required to construct the silver-standard supervision used in Section 6, we recorded the API token-usage metadata returned by Gemini 2.5 Pro for the 1,000 admissions processed through the frozen three-stage framework. Across Stages 1 to 3, generation processed 45.1345.13M input tokens and produced 16.7016.70M output tokens, for a combined total of 61.8461.84M tokens, or 61,83661,836 tokens per admission on average. Stage 1 accounted for 25.6625.66M tokens (41.541.5% of the total), Stage 2 for 19.5219.52M (31.631.6%), and Stage 3 for 16.6616.66M (26.926.9%). Table A3 provides the complete stage-level breakdown. These counts quantify the inference used to generate the 1,0001,000-example silver corpus in Section 6 and illustrate the computational demands of processing complete discharge summaries and their intermediate representations through a multi-stage long-context pipeline. The resulting corpus, however, can be reused across three downstream tasks and multiple open-weight student models, amortizing the initial teacher-generation cost. Of the reported input tokens, 16,177,25416,177,254 were served from cached context; because cached tokens are a subset of input tokens, they are not added again to the total. Stage Input Output Total Stage 1: Atomic Clinical Event Extraction 20,464,371 5,198,803 25,663,174 Stage 2: Date & Certainty Tagging 9,794,222 9,723,468 19,517,690 Stage 3: Temporally Guided Summarization 14,874,581 1,780,523 16,655,104 Total 45,133,174 16,702,794 61,835,968 Table A3: Token usage for generating the 1,000-example silver-standard corpus with the frozen Gemini 2.5 Pro. Total tokens are the sum of input and output tokens. The grand total corresponds to an average of 61,836 tokens per admission across all three stages. Of the input tokens, 16,177,254 were served from cached context and are already included in the input counts. Appendix B Agent Candidate Selection Following the selection of agents in Section 4.2, a candidate is eligible if it supports a native context window of at least 3232K tokens which is sufficient to fit our P9999 input without truncation. All candidates are evaluated with identical prompts and greedy decoding (t=0t=0). An open-weight candidate additionally has to be (i) downloadable under a license permitting research and on-premise inference and (i) loadable on our deployment stack across four L40S GPUs at bfloat16 precision. A proprietary candidate additionally has to be (i) served through a managed endpoint compatible with our institutional data-use agreement (e.g., Amazon Bedrock or Google Vertex AI) and (iv) cost-feasible at corpus scale. A complete table of agent specifications can be found in Table A4. Prior work recommends low decoding temperatures for classification and labelling tasks to stabilize outputs Jin et al. (2024). Because our stages demand both high precision and high recall, including sharp inclusion/exclusion at extraction and unambiguous tagging decisions downstream, we set temperature t=0t=0 (greedy decoding) for all three agentic stages. All other generation settings were held constant across models, and for open-weight models this produces byte-identical outputs across runs. Agent Model Provider Params (total/active) Arch. Gemini 2.5 Pro Comanici et al. (2025) Google (via Vertex AI) Undisclosed n/r Gemini 2.5 Flash Google (via Vertex AI) Undisclosed n/r Claude Sonnet 4.6 Anthropic (2026) Anthropic (via Bedrock) Undisclosed n/r Claude Haiku 4.5 Anthropic (2025) Anthropic (via Bedrock) Undisclosed n/r Llama 3.3 70B Meta AI (2024) Meta 70B Dense Qwen 3.6 35B-A3B Qwen Team (2026) Alibaba 35B / 3B MoE Gemma 4 26B-A4B Gemma Team (2026) Google 26B / 4B MoE MedGemma 27B Sellergren et al. (2026) Google 27B Dense Mistral Small 3.2 24B Mistral AI (2025) Mistral AI 24B Dense Meditron 3 (Qwen 2.5 7B) OpenMeditron (2025) OpenMeditron/EPFL 7B Dense Phi-4 Mini 3.8B Abouelenin and others (2025)s Microsoft 3.B Dense Table A4: Model agents included in the candidate pool. Models vary in provider, domain specialization, serving environment, parameter count, and architecture. Appendix C Benchmark Size Selection Via Power Analysis We fix the number of manually evaluated documents using an a priori power analysis for a chi-square test Cohen (1988). The required sample size is N=λâw2,N= λ^*w^2, (1) where w is Cohenâs effect-size index and λâλ^* is the noncentrality parameter at which the noncentral chi-square distribution Ï2â(dâf,λ)Ï^2(df,λ) reaches the target power relative to the critical value Ï1âα2â(dâf)Ï^2_1-α(df). For a pairwise comparison with one degree of freedom, we targeted a large effect size (w=0.5w=0.5), a significance level of α=0.05α=0.05, and power of 1âÎČ=0.951-ÎČ=0.95. This gives λâ=(z1âα/2+z1âÎČ)2=12.99,λ^*= (z_1-α/2+z_1-ÎČ )^2=12.99, (2) and therefore N=12.990.52=12.990.25=51.98.N= 12.990.5^2= 12.990.25=51.98. (3) We round this value up to N=52N=52 documents. Applying a finite-population correction for the full corpus of 14,882 documents produced an adjusted estimate of Nadj=51.80N_adj=51.80, leaving the required sample size effectively unchanged. At N=52N=52, the achieved power is 0.9500.950. The analysis is conducted using the pwr package Champely (2020) and independently verified using the noncentral chi-square distribution implemented in SciPy Virtanen et al. (2020). The sampling unit is the discharge-summary document. Thus, the independence assumption applies across sampled documents, and correlations among events within a document do not artificially increase the effective sample size. Appendix D Stage 1 & Stage 2 Generation Prompts We present the concrete definitions for each of the sub-dimensions of timeline-robustness. We provide the complete prompts used for Stages 1 and 2 in Tables A5 and A6. Each stage uses a fixed system prompt and a document-specific user-prompt template. The system prompts were applied unchanged across all candidate models. At inference time, only the bracketed placeholders were replaced with the corresponding discharge summary, extracted events, and admission anchors. No model-specific prompt tuning or manual preprocessing was performed. D.1 Stage 1: Atomic Clinical Event Extraction The Stage 1 prompt instructs the model to extract all clinical information from the discharge summary as atomic, self-contained events. The prompt prioritizes coverage and source faithfulness: repeated mentions are retained, negation is preserved, and the model is instructed not to correct the note, infer unstated information, or add clinical interpretation. Events are returned in their original document order as an unnumbered list. The complete prompt is shown in Table A5. D.2 Stage 2: ISO Date and Certainty Tagging The Stage 2 prompt assigns one temporal label to each event extracted in Stage 1. The model receives the full discharge summary as global context, together with the admission date, discharge date, date of birth, patient age, and the ordered list of atomic events. It is instructed to preserve the event text and assign exactly one of four labels: [EXACT], [APPROX], [PRE ADM], or [INDETERMINATE]. The prompt follows a conservative decision rule: explicitly written dates receive [EXACT], dates resolved from anchors or relative expressions receive [APPROX], events framed as prior history receive [PRE ADM], and [INDETERMINATE] is used only when no defensible temporal resolution is available. The model is explicitly prohibited from guessing dates. The complete prompt is shown in Table A6. Type Prompt System Prompt You are an atomic clinical event extraction engine. Your task is to extract a list of atomic clinical events (Presented as ATOMIC_CLINICAL_EVENTS) from an unstructured discharge summary. CLINICAL EVENTS: A clinical event is one clinically meaningful, patient-related unit of information documented in the note, expressed as a self-contained statement describing an action, condition, observation, or change in the patientâs clinical course. An event may describe something that: 1. Happened 2. Was observed 3. Was reported 4. Was performed 5. Was denied 6. Was known about the patient This includes, but is not limited to: Diagnoses, symptoms and signs, Procedures, lab results, imaging findings, medications, social history, behavioral events, family history, clinical decisions and plans, negated findings (these must be preserved exactly). ATOMIC CLINICAL EVENTS: We define an atomic clinical event as the smallest unit of clinical information expressible as a single subject-verb-object clause that cannot be further decomposed without loss of clinical meaning. Instructions: 1. Expand all abbreviations but keep the original in parentheses (e.g., âshortness of breath (SOB)â, âheart rate (HR) of 96â). 2. Extract every mention of every fact, even if the same fact appears in multiple sections. Do not deduplicate. 3. Do not correct errors, infer unstated information, or add clinical interpretation. Extract only what is explicitly written. 4. Negated findings must be preserved exactly (e.g., âThe patient has no thyromegaly"). 5. Preserve the tense from the original text. For structured data without verbs (such as labs and vitals), use âwas.â 6. Output the atomic clinical events in document order as a list, with each item starting with â- ". Do not number the bullets. Do not add explanations. Do not add any extra text. User Prompt DISCHARGE_SUMMARY: <discharge_summary> Table A5: Prompt used for atomic clinical event extraction Type Prompt System Prompt You are a temporal tagger for atomic clinical events. Assign exactly one temporal tag to each event in the given ATOMIC_CLINICAL_EVENTS list, producing the TAGGED_ATOMIC_CLINICAL_EVENTS list. Accurate tags let us reconstruct a patientâs timeline from an unstructured note, so a wrong date is worse than a coarse one â NEVER invent or guess a date. You are given the original note (DISCHARGE_SUMMARY) as global context and ANCHOR_DATES (admission date, discharge date, and patient age if stated). DATE FORMATS: - When a date is assigned, use the most specific format supported by the source: [Y-M-D], [Y-M], or [Y]. - For events spanning a range, use [START to END], where START and END can each be in any of the above formats (e.g., [2000-01-01 to 2000-01-02], [1998-11 to 2001-08-02]). - Never add precision the source lacks: month-only resolves to [Y-M]; year-only resolves to [Y]. TAGS: assign the one option that applies. [EXACT] â a date (or range) is written in the event text itself and applies to it. Use it as written, normalized, regardless of where it falls: a dated pre-admission event (e.g. a past surgery) is EXACT, not [PRE ADM]; a future follow-up date is EXACT. Examples include: - â[2000-01-01] [EXACT] The patient underwent a surgical procedure on 2000-01-01â - â[2010-03-04] [EXACT] On 2010-03-04 the patient was brought to the emergency department (ED)â - â[1980-05-13] [EXACT] The patient is to follow up with his primary care provider on 1980-05-13â (a written date is EXACT even when it falls after discharge) [APPROX] â no explicit date is written in the event string, the event resolves to an anchor (admission, discharge, or DOB). Covers: the admission/presentation encounter; a relative offset from an anchor, at the offsetâs precision; an in-stay event with no specific date (use the [admission to discharge] range); a plan or instruction issued at discharge. Examples (admission 2010-01-06, discharge 2010-01-10): Examples include: - â[2010-01-01] [APPROX] 5 days before admission, the patient attempted suicideâ - â[2003] [APPROX] The patient quit smoking 7 years agoâ - â[2010-01-06 to 2010-01-10] [APPROX] A procedure was held during the hospitalizationâ - â[2010-01-06] [APPROX] In the emergency department (ED), the patient did wellâ (a prior related event established the ED visit was on 2010-01-06; events tied to that visit resolve to it) [PRE ADM] â the event is framed as prior to or continuous through this admission â including but not necessarily limited to past medical history, âhistory of âŠâ, âstatus postâ/âs/pâ, prior events, family or social history with no exact or relative time mentions, demographics, or an undated chronic state. Carries no [DATE] tag in front. Examples include: - â[PRE ADM] The patient has a history of depressionâ - â[PRE ADM] The patient is a 37 year old femaleâ - â[PRE ADM] The patient was involved in an accidentâ - â[PRE ADM] The patientâs mother died at age 40 of cancerâ. Strict Guideline Prefer the three tags above. Use the last option, [INDETERMINATE], only as a last resort. [INDETERMINATE] â use ONLY after [EXACT], [APPROX], and [PRE ADM] have each been ruled out: no written date (not EXACT), no resolution to any anchor (not APPROX), and no indication it occurred before admission (not PRE ADM). The timing genuinely cannot be determined. This is not a fallback for uncertainty â if any of the first three plausibly applies, use that one. Carries no date. An example includes: - â[INDETERMINATE] An outpatient ultrasound was recommended at some pointâ Output Instructions 1. Preserve the exact wording of each input event in the output. Do not reword, reorder, correct errors, infer unstated facts, or add interpretation. 2. One bullet per event, starting with â- â. Tag block is â[DATE] [TAG]â for EXACT and APPROX, and the tag alone (â[PRE ADM]â or â[INDETERMINATE]â) for the two that carry no date. 3. Do not number bullets. Do not add any text outside the bullets. User Prompt ANCHOR_DATES: - Date of Birth: date_of_birth - Admission Date: <admission_date> - Discharge Date: <discharge_date> DISCHARGE_SUMMARY: <discharge_summary> ATOMIC_CLINICAL_EVENTS: <atomic_clinical_events> Table A6: Prompt used for date & certainty tagging Appendix E Stage 1 & Stage 2 Evaluation Rubrics Stage 2 is evaluated using four complementary dimensions. Completeness measures whether the extracted timeline captures the source-grounded clinical events in the discharge summary. Semantic faithfulness assesses whether each event preserves the meaning of the source, including negation, uncertainty, quantities, and other qualifiers. Temporal-tag accuracy evaluates both the assigned certainty category and the resolved date or date range. Anti-hallucination identifies events that introduce claims unsupported by the source note. Itâs important to note that anti-hallucination was evaluated only by human annotators. Unlike the other dimensions, this criterion requires determining whether any claim in an extracted event lacks support anywhere in the source note. Prior work shows that LLM-as-judge reliability varies substantially across tasks and evaluation properties, and that even strong LLMs remain imperfect at recognizing hallucinated content Bavaresco et al. (2025); Li et al. (2023). This limitation is especially consequential in the clinical domain, where subtle, plausible unsupported claims may be difficult to distinguish from source-grounded information. Because a missed hallucination would directly contaminate the clinician-verified reference set, we reserved this criterion for exhaustive human review rather than treating an LLMâs factuality judgment as ground truth. All rubrics are included in Tables A7 to A11. Component Dimension A: Completeness Guiding Question Did the extraction capture the source-grounded events from the note? Definition A source-grounded clinical event is a single, self-contained assertion documented in the note and concerning the patientâs care. This includes, but is not limited to, conditions, findings, procedures, medications, laboratory and vital-sign values, plans and instructions, demographics, allergies, admission and discharge facts, and family or social history. An assertion is source-grounded when it is stated in the note; it does not need to be clinically salient. Clinical salience is not assessed under this rubric. Exclusions An assertion is excluded only when its entire content is administrative, such as a clinician or provider name alone, telephone or fax number, pager number, address, ward or room identifier, or document-formatting artifact. An administrative detail appearing within an otherwise clinical assertion does not exclude that assertion. Reporting Report every missing event, including its source span in the note and a one-line description. The expected-event set is the union of the source-grounded candidate events and the missing events reported by the evaluator. Coverage is calculated as the number of captured expected events divided by the size of this union. Score 5 The candidate list captures at least 95% of the expected source-grounded clinical events. Score 4 The candidate list captures at least 80% but less than 95% of the expected source-grounded clinical events. Score 3 The candidate list captures at least 60% but less than 80% of the expected source-grounded clinical events. Score 2 The candidate list captures at least 40% but less than 60% of the expected source-grounded clinical events. Score 1 The candidate list captures less than 40% of the expected source-grounded clinical events. Table A7: Stage-2 evaluation rubric for Dimension A: Completeness. Component Dimension B: Semantic Faithfulness Guiding Question Does each candidate event preserve the clinical meaning of the source note while allowing the permitted normalizations? Scope Judge only the event text. Ignore the bracketed date and temporal-tag block preceding it, because temporal tags are evaluated separately. Within this dimension, assess only whether temporal expressions written inside the event text match those written in the source note. Definition An event preserves the source meaning only when all of the following properties survive rewriting: polarity or negation; epistemic status, including hedging and expressions such as âlikelyâ or âconcern forâ; experiencer, such as the patient versus a family member; quantities, units, doses, and laterality; severity, chronicity, and frequency; temporal expressions appearing in the text; and conditionality or hypotheticality. Each event receives a binary semantic judgment. A wording change that preserves all meaning-invariant properties is not an error. Hallucinations Events with no identifiable source assertion are hallucinations and are evaluated under Dimension D. They are excluded from the denominator for Dimension B. No event may be counted as an error under both Dimensions B and D. Reporting Report every event that fails, including its event ID, corresponding source span, and one error code from the following list: âą negation_lost: a negated finding is asserted, or an asserted finding is negated. âą uncertainty_lost: a hedged claim is asserted as fact, or a factual claim is incorrectly hedged. âą experiencer_shifted: a statement concerning a family member or another person is attributed to the patient, or vice versa. âą value_or_unit_altered: a quantity, unit, dose, count, or laterality differs from the source. âą qualifier_altered: severity, chronicity, or frequency differs from the source. âą temporal_cue_altered: a temporal expression in the event text differs from or is omitted relative to the source. âą conditionality_lost: a conditional or hypothetical statement is presented as an actual occurrence. âą abbreviation_misexpanded: an abbreviation is expanded to a meaning unsupported by the source. âą other_distortion: the meaning differs for a reason not covered by the other error codes. Normalization Separately record events whose wording deviates from the permitted normalizations while preserving all meaning-invariant properties using the flag normalization_deviation. These cases are not semantic errors and do not affect the score. Score 5 At least 95% of candidate events preserve the source meaning. Score 4 At least 80% but less than 95% of candidate events preserve the source meaning. Score 3 At least 60% but less than 80% of candidate events preserve the source meaning. Score 2 At least 40% but less than 60% of candidate events preserve the source meaning. Score 1 Less than 40% of candidate events preserve the source meaning. Table A8: Stage-2 evaluation rubric for Dimension B: Semantic Faithfulness. Component Dimension C: Temporal-Tag Accuracy Guiding Question For each candidate event, is the assigned temporal tag correct, and is the assigned date correct? Tag Format Every event carries exactly one temporal tag. [EXACT] and [APPROX] also carry a date or date range. [PRE ADM] and [INDETERMINATE] carry no date. [EXACT] A date is written in the event text itself. A date whose month and day are written and whose year is resolved from the note is also [EXACT]. For example, âon 5-9â in a note with admission year 2135 becomes [2135-05-09] [EXACT]. An event whose text contains no date is never [EXACT]. [APPROX] No date is written in the event text, but the event resolves to the admission date, discharge date, date of birth, a relative offset from one of these anchors, or a date established elsewhere in the note. An in-stay event without a specific date receives the admission-to-discharge range. A plan issued at discharge receives the discharge date. [PRE ADM] The event is framed as occurring before or continuing through the current admission, with no written or resolvable date. This includes history, âstatus postâ statements, demographics, undated chronic states, and undated family or social history. [INDETERMINATE] None of the other categories applies and the eventâs timing cannot be determined. This is a last-resort label. Date Precision A date is correct only when it matches the date that should have been assigned at the same level of precision. Assigning [2135-05] when [2135-05-09] was recoverable is an error, as is assigning [2135-05-09] when only [2135-05] was supported. For a date range, both endpoints must match. Event Score 1.0 Assign 1.0 when the temporal tag is correct and, for [EXACT] and [APPROX], the date is also correct. [PRE ADM] and [INDETERMINATE] receive 1.0 when the tag is correct because they carry no date. Event Score 0.5 Assign 0.5 only when both the assigned and correct tags are [EXACT] or [APPROX], and exactly one of the tag or date is correct. This includes cases where the tag is correct but the date is wrong, or the date is correct but [EXACT] and [APPROX] are interchanged. Event Score 0.0 Assign 0.0 in every other case. This includes assigning a no-date tag when a date-bearing tag is correct; assigning a date-bearing tag when a no-date tag is correct; confusing [PRE ADM] with [INDETERMINATE]; assigning both the wrong tag and wrong date; or emitting [EXACT] or [APPROX] without a date. Table A9: Stage-2 evaluation rubric for Dimension C: Temporal-Tag Accuracy (Part 1 of 2). Component Dimension C: Temporal-Tag Accuracy Reporting Report every event receiving a score below 1.0. Include its event ID, the temporal tag and date believed to be correct, and one error code from the following list: âą tag_over_specific: [EXACT] or [APPROX] was assigned where [PRE ADM] or [INDETERMINATE] was correct, meaning that a date was invented. âą tag_under_specific: [PRE ADM] or [INDETERMINATE] was assigned where a date was recoverable. âą tag_exact_approx_confused: [EXACT] and [APPROX] were interchanged. âą tag_preadm_indet_confused: [PRE ADM] and [INDETERMINATE] were interchanged. âą date_wrong_value: the assigned date contains the wrong day, month, or year. âą date_too_coarse: the date is correct but less precise than the note supports. âą date_too_precise: the date is more precise than the note supports. âą date_range_wrong: one or both endpoints of a date range are incorrect. âą date_missing: [EXACT] or [APPROX] carries no date. âą date_wrongly_present: [PRE ADM] or [INDETERMINATE] incorrectly carries a date. âą other_temporal_inaccuracy: another temporal error, accompanied by an explanation. Aggregate Rate Temporal-tag accuracy is calculated as the sum of the per-event scores divided by the number of assessed events, expressed as a percentage. Score 5 At least 95% temporal-tag accuracy. Score 4 At least 80% but less than 95% temporal-tag accuracy. Score 3 At least 60% but less than 80% temporal-tag accuracy. Score 2 At least 40% but less than 60% temporal-tag accuracy. Score 1 Less than 40% temporal-tag accuracy. Rate Calculation For Dimensions AâC, calculate rates to one decimal place. Do not round the rate before assigning its score band. Report the counts used in the calculation. Table A10: Stage-2 evaluation rubric for Dimension C: Temporal-Tag Accuracy (Part 2 of 2). Component Dimension D: Anti-Hallucination Guiding Question Does every factual claim in the candidate events come from the source note? Definition This dimension detects information added by the extraction. It does not assess information that was changed from an existing source assertion, and it does not penalize permitted rewrites. Decision Rule For each candidate event, determine whether the source note contains an assertion of which the event is a version, even if the event distorts that assertion. If such an assertion exists, the event is not a hallucination and any meaning error belongs under Dimension B. If no corresponding source assertion exists, the event is a hallucination. Hallucination Types âą fabricated_event: an action, finding, procedure, or occurrence that the source note never states. âą fabricated_attribute: a value, unit, dose, count, side, or qualifier attached to a real event when the source gives no such attribute. When the source gives a different value for a known event, the error belongs under Dimension B. âą fabricated_entity: a medication, test, diagnosis, or person that the source note never names. âą fabricated_date: a date written within the event text that does not appear in the source note. Errors in the bracketed date or temporal-tag block belong under Dimension C. âą unwarranted_inference: a causal, diagnostic, or prognostic conclusion not explicitly stated in the source note. Clinical reasoning stated in the source may be extracted, but unsupported reasoning introduced by the model may not. Do Not Assess Do not assess temporal tags or bracketed dates, which belong under Dimension C; meaning changes to events that have an identifiable source, which belong under Dimension B; or any permitted rewrite. Reporting Report every hallucinated event, including its event ID and complete event text. Score 1 No candidate event contains a hallucination. Score 0 At least one candidate event contains a hallucination. Table A11: Stage-2 evaluation rubric for Dimension D: Anti-Hallucination. Appendix F Stage 1 & Stage 2 Evaluation Platform Human evaluation of Stage-2 outputs was conducted through a purpose-built local web application (Figure A1). The tool runs as a self-contained Python program with a static HTML interface, requires no external services or database, and is accessed through SSH port forwarding so protected health information remains on the host machine. Model outputs are loaded into an ordered annotation queue, and annotators resume from their first unfinished document. The interface displays the full discharge summary alongside the model-generated events in source order, including their temporal and certainty tags. For each event, annotators independently assess temporal accuracy, certainty-tag accuracy, semantic faithfulness, and hallucination, with criterion-specific comment fields for flagged errors. To capture omissions, the interface also records the number of missing events and the corresponding source passages. The platform includes progress tracking, bulk confirmation for all-correct events, checks that prevent advancing with incomplete ratings, keyboard navigation, and immediate crash-safe saving to per-annotator JSON files. Completed work can therefore be interrupted and resumed without loss. Figure A1: Web interface of the Stage-2 event-level human annotation platform. A single subject is shown: the source discharge summary (left) beside the modelâs Stage-2 tagged event list (right), with per-event Yes/No controls for the four rubric dimensions (time tag, type tag, semantic faithfulness, hallucination) and a footer for logging missed events. All protected health information and note/event text are redacted (gray bars) for this figure. Appendix G Stage 1 & Stage 2 Evaluation Agreement and Error Analysis Table A12 reports agreement between the Gemini 3.1 Pro evaluator and clinician-guided human annotations across 15,891 Stage-2 events from 52 discharge summaries. We report Cohenâs Îș, Gwetâs AC1, and positive and negative agreement. Because confirmed errors are rare relative to correct events, Cohenâs Îș is sensitive to class prevalence; Gwetâs AC1 and the class-specific agreement measures provide complementary views of agreement under this imbalance. Overall agreement is high on the dominant non-error class: negative agreement reaches 0.9860.986, while Gwetâs AC1 is 0.9720.972. Agreement is lower on the rare error class, with positive agreement of 0.6690.669. This indicates that the evaluator and human reviewers agree strongly on which events are correct, but differ more often when identifying or characterizing errors. The disagreements are concentrated in temporal accuracy rather than semantic faithfulness. Although human annotation and the automated judge report nearly identical total numbers of temporal errors (608608 versus 602602), an event-level breakdown reveals a strong directional asymmetry: human review identified 175 over-specified tags on pre-admission medications and demographics, 169 of which the judge failed to flag, whereas judge-only findings were concentrated in overlapping date-range and date-granularity codes (Appendix G.1). Temporal-accuracy agreement reaches Îș=0.617Îș=0.617, whereas semantic-faithfulness negative agreement is 1.0001.000. Manual review identified two recurring sources of temporal disagreement. First, some models anchor home medications to the admission date using [APPROX], while clinicians classify them as [PRE ADM]; the automated evaluator does not consistently penalize this distinction. Second, the evaluator sometimes accepts a coarse date range even when the event can be resolved to a more specific in-stay anchor. Thus, disagreement is concentrated in borderline cases of pre-admission status and date granularity rather than being distributed uniformly across the task. The integrity of the gold set does not depend on evaluator agreement. Human annotators exhaustively reviewed all 15,891 events under clinician guidance and corrected every confirmed error after escalating errors to discussion with clinicians, regardless of whether the automated evaluator detected it. This process produced 52 human-corrected pairs of discharge summaries and tagged timelines. G.1 Temporal Error-Type Analysis To characterize the lower positive agreement on temporal accuracy, we compared the finalized event-level error codes assigned through exhaustive human review with those assigned by the calibrated LLM judge across the 52-note benchmark. Figure A2: Stage-2 temporal-error counts from human annotation and the calibrated LLM judge. Despite similar totals (608 versus 602), human review surfaced 175 over-specified tags on pre-admission medications and demographics that the judge almost entirely missed, while judge-only errors concentrated in overlapping date-range and granularity categories. As shown in Figure A2, the aggregate counts are similar: human annotation identified 608 temporal errors, compared with 602 identified by the judge. Human review found at least one temporal error in 50 of the 52 notes, whereas the judge identified an error in 46. The clearest asymmetry concerns over-specified temporal tags. Human annotation identified 175 such errors involving pre-admission medications (151 events) and patient demographics (24 events), of which the judge failed to flag 169 (96.6%). These are not cases in which the human and judge applied different error codes to the same behavior: the judge returned no temporal error at all. The pattern therefore represents a systematic false-negative class in which the judge accepts admission-anchored [EXACT] or [APPROX] labels for standing or pre-admission facts that should receive [PRE ADM]. The reverse asymmetry is qualitatively different. Of the 200 judge-only errors, 80 were coded as date_range_wrong and 56 as date_too_coarse. These codes describe closely related aspects of temporal granularity and may place the boundary between an incorrect range and an insufficiently precise date differently. Thus, similar aggregate error counts do not indicate equivalent error detection: human review exposes a clinically coherent error class that the judge largely misses, whereas much of the judgeâs excess is concentrated around rubric boundaries for range and granularity. G.2 Other Error-Types and Analysis The 2121 non-temporal errors are minor and mechanical, encompassing just 0.13%0.13\% of all 15,89115,891 events. Unlike the temporal errors, almost none reflect clinical misunderstanding. The dominant failure mode is over-precision: the model quietly hardens hedged or masked source text into confident assertions. Roughly half of the semantic errors convert uncertainty into fact (e.g., â? <*clinical event*>" becomes âpossible <*clinical event*>"), or add unsupported specificity to medications (such as "twice a day", "twice daily" appended to a drug the note never dosed). The rest stem from misreading MIMICâs de-identification masks and OCR-garbled tokens. The 7 hallucinations are similar in spirit â mostly home/admission medications and lab values carrying detail not fully grounded in the source â and the single completeness miss (1 missing patient date of birth event across 52 notes) is negligible. In short, on everything except temporal tagging the model is essentially faithful; its only residual weakness is a tendency to resolve ambiguity in the source rather than preserve it. Dimension Cohenâs Îș AC1 Pos. Neg. Overall 0.655 0.972 0.669 0.986 Temporal accuracy 0.617 0.971 0.631 0.986 Semantic faithfulness 0.611 0.999 0.611 1.000 Table A12: Agreement between the Gemini 3.1 Pro and gold annotation, calculated over 52 discharge summaries producing 15,891 events. AC1 denotes Gwetâs AC1; Pos. and Neg. denote positive and negative agreement on the rare error and dominant non-error classes, respectively. Anti-hallucination is excluded because it was evaluated only by human annotators. Appendix H Stage 3 Generation Prompt The Stage 3 prompt converts the tagged event timeline into a chronological prose summary. It specifies how temporal tags should be rendered, which clinical information should be included or excluded, how duplicate or overlapping events should be combined, and how factual and temporal fidelity should be preserved. Because of its length, the complete prompt is presented across three consecutive sub-tables (Table A13). Type Prompt System Prompt You will convert a tagged clinical event list into a temporally structured prose summary. INPUTS You are given two inputs: 1. ORIGINAL DISCHARGE SUMMARY â the source note. Use it only to clarify wording, expand abbreviations, or resolve unclear phrasing. Do not take any fact from it that is not present in the tagged event list. 2. TAGGED EVENT LIST â the authoritative source. Every fact and every date in the summary must come from this list. TEMPORAL RENDERING RULES (mandatory, no exceptions) Each event carries a temporal tag. Render each tag exactly as specified: âą [Y-M-D] [EXACT]: âOn Y-M-D, âŠâ âą [Y-M-D] [APPROX]: âApproximately on Y-M-D, âŠâ âą [Y-M-D to Y-M-D] [APPROX]: âApproximately from Y-M-D to Y-M-D, âŠâ âą [Y] [EXACT]: âIn Y, âŠâ âą [PRE ADM]: âBefore admission, âŠâ Rules: âą Write all dates in ISO format (Y-M-D), zero-padded, exactly as given in the tag. âą The word âapproximatelyâ applies to the date, not to the event. The event is certain; only its timing is approximate. Never write âthe patient was likely admittedâ or âit is likely that X happened.â âą Reproduce each eventâs negations and hedges exactly as written; do not add, delete, or replace them. âą If the event text restates timing already supplied by the tag, such as âadmitted on 2122-5-19â or âon postoperative day two,â delete the repeated timing phrase from the sentence. âą You may adjust grammar and tense minimally so that the sentence reads correctly after the temporal prefix. Do not change the clinical content. DEDUPLICATION AND MERGING âą Exact duplicates: If two events have the same tag and the same content, output the content once. âą Subsumption: If one eventâs content is fully contained in anotherâs, such as âThe patient is femaleâ and âThe patient is a 19-year-old female,â output only the more specific event. âą Merging: Events with identical tags (the same date and the same type) must be combined under one temporal prefix and joined into one or a few sentences. Never repeat the prefix, and never merge events across different dates or types. This includes recombining events split from one clinical statement, such as âHer incision was clean,â âwas dry,â and âwas intact.â âą Merge date of birth, age, and sex into one opening demographic sentence. Table A13: Prompt used for temporal-guided clinical summarization (Part 1 of 3). Type Prompt System Prompt INCLUDE Include only the following categories. If an event does not fall into one of these categories, omit it. 1. Patient demographics: date of birth, age, and sex. 2. The reason for admission: the clinical problem or finding that led to the admission, and any diagnostic workup or counseling directly related to it. 3. In-stay events, limited to the following types: a. Procedures and surgeries performed. b. Intraoperative findings and repairs. c. Admissions to and transfers between services or units. d. Intubation, extubation, and other airway or ventilation events. e. Placement or removal of devices, lines, tubes, or catheters. f. Treatments or interventions started or stopped. g. Diagnoses made or revised during the stay. h. Consultations obtained. 4. The discharge: date of discharge, disposition, discharge condition, and follow-up instructions. 5. Social history relevant to the patientâs mental health, unless it is described as non-contributory or insignificant in the source material. 6. Medications on admission and medications at discharge must each be rendered as a line-separated list under their own temporal prefix, with one medication per line copied verbatim from the event text. These are the only lists permitted in the output; all other content must be continuous prose, as in the following example: Before admission, the patient was taking: âą Medicine A at X dosage âą Medicine B Approximately on [discharge date], the patient was discharged on: âą Medicine C for Y duration âą Medicine D via route Z EXCLUDE Exclude the following categories entirely: 1. Laboratory values and laboratory results. 2. Vital-sign values. 3. Physical-examination findings, including normal and benign findings and descriptions of body habitus or affect. 4. Statements that a parameter or status remained stable, unchanged, or within normal limits. 5. Past medical history, past surgical history, and prior events that are not the reason for the current admission, unless the condition actively complicated the hospital course, required a consultation, or significantly altered medical management during the stay. 6. Allergies. FIDELITY âą Use only facts present in the tagged event list. âą Do not infer, interpret, diagnose, or add clinical judgment. âą Do not alter any numeric value, date, medication name, dose, route, or frequency. âą Preserve negations exactly. âNo evidence of metastatic diseaseâ must never become âevidence of metastatic disease.â âą Retain de-identification placeholders, such as [Last Name (STitle) 9125], exactly as written. Do not expand, guess, or replace them. âą Do not add a heading, preamble, commentary, or closing statement. Table A14: Prompt used for temporal-guided clinical summarization (Part 2 of 3). Type Prompt System Prompt ORDER AND FORMAT âą Output continuous prose, except for the two medication lists specified above. Do not use other bullet points, headings, or tables. âą Open with the demographic sentence. âą Order all remaining content chronologically by the start date of its tag. For events sharing a start date, preserve the order in which they appear in the tagged event list. âą Place [PRE ADM] events, which carry no date, together as one âBefore admissionâ group immediately before the admission date. âą Start a new paragraph at each of the following points: before admission, the admission date, and the discharge date. WORKED CONVERSIONS These examples demonstrate temporal rendering only. Whether an event is retained is determined by the INCLUDE and EXCLUDE rules above. Input: - [2000-01-01] [EXACT] The patient was admitted on 2000-01-01. Output: On 2000-01-01, the patient was admitted. Input: - [2000-01-03] [APPROX] She was extubated on postoperative day three. Output: Approximately on 2000-01-03, she was extubated. Input: - [2000-05-10 to 2000-05-30] [APPROX] The patient was admitted to the ICU service. Output: Approximately from 2000-05-10 to 2000-05-30, the patient was admitted to the ICU service. Input: - [PRE ADM] The patient was found to be unconscious. Output: Before admission, the patient was found to be unconscious. Input: - [2011] [EXACT] The patient is status post knee operation in 2011. Output: In 2011, the patient underwent knee operation. Merging two events sharing one tag: Input: - [2013-12-31] [APPROX] The patientâs social service worker recommended therapy. - [2013-12-31] [APPROX] The patientâs social service worker discussed treatment plans with the patientâs PCP. Output: Approximately on 2013-12-31, the patientâs social service worker recommended therapy and discussed treatment plans with the patientâs PCP. Subsumption: Input: - [PRE ADM] The patient is female. - [PRE ADM] The patient is a 58-year-old female. Output: Keep only the second event and incorporate it into the opening demographic sentence. Now produce the summary. User Prompt ORIGINAL_DISCHARGE_SUMMARY: <discharge_summary> TAGGED_EVENT_LIST: <tagged_event_list> Table A15: Prompt used for temporal-guided clinical summarization (Part 3 of 3). Appendix I Stage 3 Evaluation Rubrics Stage 3 is evaluated by both human annotators and the calibrated LLM evaluator using two shared dimensions. Faithfulness to Source measures whether each clinical claim in the generated summary is supported by the human-verified event timeline and preserves its original clinical meaning. Temporal Accuracy measures whether included events are rendered under the correct date and certainty tag and whether only events with identical temporal tags are grouped together. The human-verified timeline serves as the reference of record; the original discharge summary is used only to clarify unclear wording. Because selective content inclusion is expected in summarization, omitted events are not penalized under either dimension. The complete rubrics are provided in Tables A16 and A17. Component Dimension A: Faithfulness to Source Guiding Question Is every clinical event and fact in the summary traceable to the source discharge summary or the human-verified event timeline, with no invented content, and is its clinical meaning preserved? Temporal placement is evaluated separately under Dimension B. Reference Evaluate the candidate summary primarily against the human-verified event timeline. The source discharge summary may be used only to clarify an eventâs wording when the timeline is unclear. Do not reward or penalize content that appears in the source note but is absent from the verified timeline. Faithful Claim A claim is faithful when it is traceable to a corresponding event and preserves the eventâs clinical meaning. Permitted rewriting, such as expanding an abbreviation while retaining the original or supplying a subject and verb for structured content, is not an error when the meaning remains unchanged. The summary must include all mental health-related and social history events from the original source. Untraceable Claims A claim is unfaithful when no supporting event exists in the human-verified timeline or source note. This includes an invented event, fact, medication, diagnosis, or clinical detail. Meaning Distortions A claim is also unfaithful when it corresponds to a real event but changes its clinical meaning. Examples include: âą adding, dropping, or reversing a negation; âą adding or removing a hedge or uncertainty marker; âą changing a numeric value, dose, medication name, route, frequency, or laterality; âą introducing an unsupported causal explanation, diagnosis, or clinical rationale; or âą merging events into a new claim that neither source event supports. Do Not Assess Do not penalize omitted events, because selective inclusion is expected in a summary. Do not assess wrong dates, temporal tags, or temporal grouping under this dimension; these are evaluated under Dimension B. Rate The faithfulness rate is the percentage of clinical claims in the summary that are both traceable to the reference and meaning-preserving. Score 5 At least 95% of the summaryâs clinical claims are faithful. Score 4 At least 80% but less than 95% of the summaryâs clinical claims are faithful. Score 3 At least 60% but less than 80% of the summaryâs clinical claims are faithful. Score 2 At least 40% but less than 60% of the summaryâs clinical claims are faithful. Score 1 Less than 40% of the summaryâs clinical claims are faithful. Table A16: Stage-3 evaluation rubric for Dimension A: Faithfulness to Source. Component Dimension B: Temporal Accuracy Guiding Question Is every clinical claim in the summary placed under the same temporal tag assigned in the human-verified event timeline, and are only events with identical tags grouped together? Scope Evaluate temporal placement and grouping only. Whether an event is supported by the reference and whether its clinical meaning is preserved are assessed under Dimension A. Temporal Tags Every event carries one of the following temporal tags: âą [DATE] [EXACT], where DATE is a single ISO date or ISO date range; âą [DATE] [APPROX], where DATE is a single ISO date or ISO date range; or âą [PRE ADM], for an event occurring before admission without a resolved date. Valid Grouping A grouping is valid only when all events within it share an identical temporal tag: the same date or date range and the same certainty marker. Events with the same date but different [EXACT] and [APPROX] markers must not be merged. Misplacement A claim is temporally misplaced when it appears under a different date, date range, or certainty tag from the one assigned in the verified timeline. Examples include placing a discharge-day event on the admission date, rendering an [EXACT] event as a date range, or assigning a date to a [PRE ADM] event. Invalid Grouping A temporal grouping is incorrect when: âą events with different dates or date ranges are merged; âą [EXACT] and [APPROX] events are merged despite sharing the same date; âą a [PRE ADM] event is merged with a dated event; or âą events sharing one identical tag are unnecessarily split across multiple temporal prefixes. Do Not Assess Do not assess invented or meaning-distorted events under this dimension; these are evaluated under Dimension A. Do not penalize omitted events. The order of correctly tagged groups is not an error, provided each group appears under its correct temporal prefix. Rate The temporal-accuracy rate is the percentage of included clinical events that are both correctly placed and correctly grouped. Rates are calculated to one decimal place, without rounding before assigning the score band. The counts used to calculate the rate must also be reported. Score 5 At least 95% of included events are correctly placed and grouped. Score 4 At least 80% but less than 95% of included events are correctly placed and grouped. Score 3 At least 60% but less than 80% of included events are correctly placed and grouped. Score 2 At least 40% but less than 60% of included events are correctly placed and grouped. Score 1 Less than 40% of included events are correctly placed and grouped. Table A17: Stage-3 evaluation rubric for Dimension B: Temporal Accuracy. Appendix J Stage 3 LLM and Auto-Evaluator Results Model Faithfulness Temporal Accuracy MEDCON (summaries) Band / % Band / % P R F1 Gemini 2.5 Pro 5.00 / 99.93 4.75 / 95.33 0.725 0.519 0.599 Table A18: Evaluation results for summaries generated by Gemini 2.5 Pro. Faithfulness and temporal accuracy are reported as mean rubric bands and percentage scores. MEDCON precision, recall, and F1 measure clinical-concept overlap. As shown in Table A18, faithfulness is saturated: all 52 summaries score band 5, with 5 distorted claims overall. Temporal accuracy shows the mean likert band at 4.75 / 5, macro-mean 95.33% of events correctly placed (micro 96.17%, 257 errors overall; median 100%). MEDCON precision (0.7250.725) exceeds recall (0.5190.519), F1 0.5990.599. Asserted content is largely grounded, consistent with the faithfulness band; roughly half of source concepts are omitted. The judge scores only claims that were made and is therefore blind to this coverage gap, while MEDCON is insensitive to temporal placement. The two measure disjoint failure axes and are reported together for that reason. Appendix K Stage 3 Evaluation Agreement K.1 Annotation Protocol and Unit Definitions Silverâ Silverâ Gold Metric Model Zero Few IT Zero Few IT Zero Few IT ROUGE-1 Gemma4-31B .826±.008 .823±.008 .886±.006 .805±.011 .794±.013 .865±.013 â .743±.026 .729±.030 Qwen3.6-35B-A3B .749±.010 .791±.009 .879±.007 .717±.014 .754±.016 .867±.011 â .670±.036 .656±.037 Qwen3-30B-A3B .737±.013 .750±.013 .843±.012 .713±.014 .765±.012 .811±.018 â .136±.040 .296±.041 Mistral-24B .736±.011 .761±.011 .878±.007 .679±.021 .724±.017 .864±.011 â .698±.022 .763±.027 Meditron-7B .555±.021 .595±.022 .804±.017 .471±.030 .531±.031 .802±.021 â .178±.043 .230±.031 ROUGE-2 Gemma4-31B .711±.010 .693±.011 .804±.009 .675±.013 .649±.017 .776±.015 â .607±.024 .629±.028 Qwen3.6-35B-A3B .621±.011 .671±.011 .797±.009 .575±.016 .622±.018 .782±.012 â .555±.032 .551±.034 Qwen3-30B-A3B .598±.012 .617±.013 .748±.013 .567±.015 .613±.015 .708±.019 â .112±.033 .229±.034 Mistral-24B .600±.012 .640±.013 .795±.009 .540±.020 .597±.019 .778±.013 â .564±.021 .655±.026 Meditron-7B .422±.019 .463±.020 .709±.018 .336±.026 .399±.027 .696±.022 â .142±.035 .141±.025 Table A19: Additional lexical evaluation for Task C timeline summarization. We report ROUGE-1 and ROUGE-2 F1 under Silverâ , Silverâ , and Gold. Results are mean ± bootstrap SD over 2,000 document-level resamples. Higher is better. Human annotators reviewed all 52 benchmark summaries alongside the judgeâs output. The review interface presents the judgeâs findings as a list and allows the annotator to mark any finding not a mistake. Metric Value 95% CI Observed agreement (PoP_o) 0.904 [0.808, 0.981] Cohenâs Îș 0.773 [0.567, 0.947] Gwetâs AC1 0.835 [0.674, 0.966] Positive agreement 0.839 [0.667, 0.960] Negative agreement 0.932 [0.862, 0.986] Table A20: Agreement between the automated evaluator and human annotations. PoP_o denotes observed agreement. Confidence intervals are reported at the 95% level. K.2 Summary-level Detection Agreement Table A20 treats each summary as one unit and asks whether the judge and the annotator agree that it contains at least one error. This framing avoids committing to a claim inventory as the denominator, and at a positive rate of 35% the prevalence index falls to 0.40, so Cohenâs Îș and Gwetâs AC1 no longer diverge and both are interpretable. The judge flagged 18 summaries; the annotatorâs adjudication left at least one surviving flag in 13 of them and eliminated all flags in 5. K.3 Error-count concordance Metric Value 95% CI Totals (n=52n=52 summaries) Errors flagged by judge 262 â Errors upheld by annotator 39 â Inflation ratio 6.7Ă6.7Ă â Concordance Linâs C 0.103 [0.031, 0.345] ICC(A,1) 0.105 [0.032, 0.349] Pearsonâs r 0.511 [0.119, 0.818] Spearmanâs Ï 0.768 [0.573, 0.934] BlandâAltman analysis Bias (judge â annotator) +4.29+4.29 SD =15.32=15.32 Limits of agreement â25.7-25.7 to +34.3+34.3 Table A21: Error-count concordance between the automated judge and human annotator. Per-summary agreement between the judgeâs error count and the number of errors upheld by the annotator. C denotes Linâs concordance correlation coefficient; ICC(A,1) denotes the absolute-agreement intraclass correlation coefficient. Table A21 compares the number of errors the judge reports in each summary against the number the annotator upheld. The two disagree sharply in magnitude and agree well in order. Linâs concordance correlation coefficient is 0.103 and ICC(A,1) is 0.105, both near the floor, because the judge over-reports by a mean of 4.3 errors per summary with 95% limits of agreement spanning â25.7-25.7 to +34.3+34.3 â an interval wider than the error count of any but the most heavily flagged summary. Spearmanâs Ï is 0.768, however: the judge ranks summaries by error burden much as the annotator does. The gap between Spearmanâs Ï (0.768) and Pearsonâs r (0.511) reflects a single influential summary, subject 7676, at 100 judge errors against 5 upheld; rank-based statistics absorb it and moment-based ones do not. Taken together these results characterize the judge as a triage and ranking instrument rather than a measurement one. Its absolute error counts should not be reported as validated error rates, and any downstream comparison between generation models should use the judgeâs ordering rather than its magnitudes. Appendix L Stage 3 Evaluation Platform For Stage 3, we built a self-contained, locally-run web application for human review and correction of Gemini 2.5 Pro-generated Stage 3 summaries. The review screen has three regions (Figure A3). The left panel shows the source material, toggling between the original discharge summary and the Stage 2 gold event list, and is read-only. The right panel presents the modelâs Stage 3 summary in an editable field: rather than only flagging problems, the reviewer corrects the summary in place to produce a gold summary. Edits autosave, and an "edited" indicator with a revert-to-original control distinguishes corrected from untouched summaries; the correction is stored as a separate stage3_gold_summary field, preserving the modelâs original output for comparison. The bottom panel surfaces an LLM-judge evaluation as a scaffold for verification. The summary is scored on two rubric criteria as listed in Appendix I. Figure A3: The human review and correction platform. For each subject, the reviewer compares the modelâs Stage 3 summary against the source and corrects it in place. Left: source panel, toggled between the discharge summary and Stage 2 gold events. Right: the modelâs Stage 3 summary in an editable field, saved as a separate field. Patient-text regions from the original MIMIC I source materials are redacted (grey). Appendix M Qualitative Analysis Cases In this section, we analyze three clinician-verified gold cases using WHO categories of mental disorders, symptoms, and risk factors as mentioned in Section 5.5, grounding each assignment in the gold timeline and treating unestablished links as co-documented rather than causal. M.1 Case A: Chronic Disability and an In-Stay Depression Case A comprises 306 gold events across a 3-week admission. The record describes roughly three decades of C5 quadriplegia, an Individual-level risk factor that Stage 2 resolves to a year-only [APPROX] date some thirty years before admission, and a marked deterioration in ventilatory status during the stay. The note itself states the sequence, attributing the patientâs depressed mood to that deterioration, so the link is explicit rather than inferred by us. Depression is documented as a diagnosis: pharmacotherapy for situational depression is started during the admission, and psychiatry is consulted on the 11th hospital day. The symptoms recorded are suicidal thoughts and refusal of food; at discharge the note records restored hope, concentration and sleep, and continuation of an SSRI. Several symptoms are therefore attested only through their documented reversal, an implicit form of expression that an extraction system keyed to symptom mentions alone would miss. What makes this case instructive is where the onset falls. The background risk factor begins decades before admission, whereas the depressive episode emerges during the hospital stay, making their temporal relationship difficult to recover from the narrative alone. Only a calendar-anchored representation places them on one ordered chain while preserving that separation. The distinction is also fragile in exactly the way our error analysis predicts: had the onset been tagged [PRE ADM] rather than resolved to the admission interval, the clinically salient fact that this depression arose during the stay would disappear from the timeline entirely. This is the same boundary that accounts for the largest class of temporal disagreements in Section G. M.2 Case B: Depression Across All Three WHO Risk Levels Case B contains 494 gold events spanning a 1.5-month admission after an intentional overdose. It is the only case in our sample with documented risk factors across all three WHO levels. At the individual level, the record describes long-standing alcohol misuse across the past medical history, social history, and discharge diagnoses. At the Family & Community level, it notes limited social support, poor self-care, and the recent loss of a living arrangement with a relative. At the Structural level, it identifies multiple social stressors and a lack of insurance coverage. These links are stated directly in the record, which cites them as reasons for involving psychiatry and social work. The associated symptoms and behaviors include the presenting overdose, two earlier suicide attempts by different means, a discharge diagnosis noting multiple attempts, and a documented reduction in suicidal intent during the admission. This case highlights two important features of the reconstruction. First, the structural risk factors are exactly the kind of information that a fixed clinical schema might omit, even though they are central to the explanation recorded by the care team. This illustrates the need for the open-vocabulary representation described in Section 1. Second, the record contains an unresolved contradiction: psychiatry documents that the patient was not currently depressed during the admission, while depression remains listed among the discharge diagnoses. M.3 Case C: A Risk Cascade Under an Unsettled Diagnosis Case C contains 227 gold events spanning a 12-day admission and includes four documented risk factors, only one of which the source explicitly links to the diagnosis. The record identifies more than two decades of alcohol misuse as clinically relevant to the patientâs diagnosis. It also documents childhood sexual abuse by a caregiver, job loss attributed to drinking, and living alone after the end of a relationship about a year before admission. Because the note does not connect these latter factors directly to the disorder, we treat them as co-documented rather than causal. Bipolar disorder appears as an established diagnosis, supported by mood-stabilizing treatment at admission, repeated overdose-related hospitalizations, a prior psychiatric admission after a suicide attempt, and multiple presentations following suicide threats. What distinguishes this case is that the record later revises its own clinical interpretation. During the admission, psychiatry questions whether the bipolar diagnosis is accurate and considers whether long-term alcohol use may have contributed to a misdiagnosis. Most background risk factors are undated and therefore tagged [PRE ADM], whereas this diagnostic reconsideration is placed within the admission interval. Temporal anchoring makes the progression visible: long-standing background factors, an established diagnosis, and then in-stay doubt about that diagnosis. The case also shows that uncertainty is not only temporal; it also concerns causality and diagnosis. Finally, it demonstrates the limits of surface matching. During screening, a statement about episodic drinking was incorrectly matched to an eating-disorder symptom term and was removed after clinician review, illustrating why lexical overlap alone cannot support reliable qualitative interpretation. Appendix N Additional Results for Evaluation We present additional evaluation results for the open-weight models (Table A19).