Paper deep dive
Hindsight Bias in Clinical Temporal Reasoning: How Future Data Exposure Affects Large Language Model Judgment
Misaki Matsuura, Sayantan Kumar, Ojas Kadam, Jeremy C. Weiss
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/20/2026, 3:08:27 AM
Summary
This paper introduces a paired benchmark to measure hindsight bias in clinical temporal reasoning using Large Language Models (LLMs). The benchmark utilizes 171 case reports from PubMed Central (40 sepsis, 131 GLP-1/diabetes) represented as textual narratives and Textual Time Series (TTS). By comparing model answers on truncated (masked) versus complete (full) timelines, the study demonstrates that exposure to future data significantly increases hindsight bias rates (HBR, AIR, ∆HTR) without necessarily reducing overall accuracy, highlighting a critical flaw in retrospective evaluation methods for clinical AI.
Entities (13)
Relation Signals (6)
PubMed Central Open Access Subset → contains → Sepsis Cohort
confidence 95% · It contains 171 case reports from the PubMed Central Open Access Subset---40 sepsis and 131 GLP-1/diabetes cases
PubMed Central Open Access Subset → contains → GLP-1/Diabetes Cohort
confidence 95% · It contains 171 case reports from the PubMed Central Open Access Subset---40 sepsis and 131 GLP-1/diabetes cases
Hindsight Bias Rate (HBR) → measures → Hindsight Bias
confidence 95% · Hindsight bias rate (HBR)... captures different signals of hindsight bias.
Temporal Masking → reduces → Hindsight Bias
confidence 95% · temporal masking reduces bias without lowering accuracy.
GPT 5.6 Sol → exhibits → Hindsight Bias
confidence 90% · Across GPT 5.6 Sol... full timeline exposure produces consistent hindsight-sensitive shifts
Textual Time Series (TTS) → usedin → Temporal Masking
confidence 90% · Models answer each question using either a TTS truncated at the cutoff or the complete timeline
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Clinical decisions are prospective, but clinical language models are often evaluated on retrospective records that reveal the final diagnosis, treatment response, and outcome. Such evaluations may reward the use of future information rather than reasoning under the uncertainty present at the decision point. We introduce a paired benchmark for measuring outcome-conditioned shifts consistent with hindsight bias in clinical temporal reasoning. It contains 171 case reports from the PubMed Central Open Access Subset---40 sepsis and 131 GLP-1/diabetes cases---represented as both textual narratives and human-annotated and LLM-generated textual time series (TTS). For each case, questions are tied to a clinically meaningful cutoff and paired with a prospective reference answer and an outcome-consistent \emph{hindsight trap}. Models answer each question using either a TTS truncated at the cutoff or the complete timeline; additional conditions vary the narrative source (original or synthetic) and TTS annotation source (human or LLM). We evaluate accuracy (Acc), hindsight trap rate (HTR), answer instability rate (AIR), and hindsight bias rate (HBR), each of which captures different signals of hindsight bias. Across GPT 5.6 Sol, Gemma 4, GLM 5.2, and Opus 5, full timeline exposure produces consistent hindsight-sensitive shifts, while temporal masking reduces bias without lowering accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2609.13454v1
- Canonical: https://arxiv.org/abs/2609.13454v1
Trouble viewing inline? Open PDF directly →
Full Text
75,999 characters extracted from source content.
Expand or collapse full text
Submitted to ML4H 2026: Proceedings TrackMachine Learning for Health (ML4H) 2026 Hindsight Bias in Clinical Temporal Reasoning: How Future Data Exposure Affects Large Language Model Judgment Misaki Matsuura 1 and Sayantan Kumar 2 and Ojas Kadam 3 and Jeremy C. Weiss 2 1 Case School of Engineering, Case Western Reserve University, USA 2 National Library of Medicine, National Institutes of Health, USA 3 Rice University University, USA Abstract Clinical decisions are prospective, but clinical language models are often evaluated on retro- spective records that reveal the final diagnosis, treatment response, and outcome. Such evalua- tions may reward the use of future information rather than reasoning under the uncertainty present at the decision point. We introduce a paired benchmark for measuring outcome- conditioned shifts consistent with hindsight bias in clinical temporal reasoning.It contains 171 case reports from the PubMed Central Open Access Subset—40 sepsis and 131 GLP- 1/diabetes cases—represented as both textual narratives and human-annotated and LLM- generated textual time series (TTS). For each case, questions are tied to a clinically meaning- ful cutoff and paired with a prospective refer- ence answer and an outcome-consistent hind- sight trap. Models answer each question using either a TTS truncated at the cutoff or the com- plete timeline; additional conditions vary the narrative source (original or synthetic) and TTS annotation source (human or LLM). We evalu- ate accuracy (Acc), hindsight trap rate (HTR), answer instability rate (AIR), and hindsight bias rate (HBR), each of which captures differ- ent signals of hindsight bias. Across GPT 5.6 Sol, Gemma 4, GLM 5.2, and Opus 5, full time- line exposure produces consistent hindsight- sensitive shifts, while temporal masking reduces bias without lowering accuracy. Keywords: hindsight bias; clinical temporal reasoning; large language models; clinical ques- tion answering, benchmark evaluation Data and Code Availability The source case reports are publicly available through the PubMed Central Open Access Subset. Code will be provided as supplementary material. Institutional Review Board (IRB) This study uses publicly available case reports and involves no participant recruitment, intervention, or access to private identifiable data. 1. Introduction Clinical decision-making is inherently prospective. A clinician deciding whether to admit, treat, escalate, or discharge a patient must act using only the ev- idence available at that moment, before the subse- quent trajectory and outcome are known. Clinical artificial intelligence systems inherit the same infor- mation constraint: a model intended to support a decision at time t should be evaluated using infor- mation available by time t, rather than information documented later in the episode. This evaluation setting creates the conditions for hindsight bias, in which knowledge of an outcome changes judgments about what was knowable or pre- dictable beforehand (Fischhoff, 1975). Hindsight bias has been observed in physicians estimating diagnostic probabilities (Arkes et al., 1981; Dawson et al., 1988) and evaluating earlier care. In a randomized vignette study, clinicians judged identical antecedent care to recovery or mortality outcomes more harshly after a death outcome in two of three cases, and greater se- niority did not eliminate the effect (Banham-Hall and Stevens, 2019). However, not every answer change after additional evidence is a bias. If the task asks for the final di- agnosis, later evidence should appropriately update the answer. The narrower failure mode studied here arises when later information changes a judgment whose intended target is what was supportable at an earlier reference point. Measuring this failure re- quires holding the clinical question and antecedent evidence fixed while varying access to post-reference © M.M. , S. Kumar, O. Kadam & J.C. Weiss. arXiv:2609.13454v1 [cs.CL] 11 Sep 2026 Hindsight Bias in Clinical Temporal Reasoning Figure 1: Benchmark pipeline for measuring hindsight-sensitive clinical reasoning. Models answer the same question using masked and complete case trajectories, and response changes are evaluated for correctness, instability, and movement toward an outcome-consistent hindsight trap. information. Prior paired clinical vignette studies have shown that general and reasoning-focused LLMs can exhibit hindsight bias when outcome cues are ma- nipulated, but these evaluations are based on indi- vidual vignettes rather than a corpus of longitudinal clinical cases (Wang and Redelmeier, 2024; Degany et al., 2025). ExAnte evaluates temporal leakage in general domain QA by imposing explicit temporal cutoffs (Liu et al., 2026), but its objective differs from ours. It evaluates whether answers are supported by information available before a specified time, whereas we ask whether exposure to a patient’s subsequent clinical trajectory changes a model’s judgment about what was supportable at an earlier decision point. This requires paired full–masked evaluation with a fixed prospective reference answer and an outcome- consistent hindsight trap, allowing us to measure not only correctness but the direction of within-question answer changes. To the best of our knowledge, no existing clinical benchmark applies this design across a corpus of real longitudinal case trajectories. Contributions. We introduce (1) a paired bench- mark for clinical hindsight bias that compares masked and complete trajectories using prospective reference answers and predefined hindsight traps; (2) comple- mentary metrics (∆HTR, AIR, and HBR) for mea- suring directional hindsight-sensitive changes; and (3) a cross-model evaluation across two clinical co- horts and multiple input representations showing that temporal masking reduces hindsight bias with- out reducing accuracy. 2. Related Work Here, we present an overview of related works. A detailed version can be found in Appendix A. Generaltemporalreasoning. TRAM, TimeBench, Test of Time, and TimE evaluate event ordering, temporal arithmetic, duration, and multistep reasoning in synthetic or general domain settings (Wang and Zhao, 2024; Chu et al., 2024; Fatemi et al., 2025; Wei et al., 2025). ExAnte is clos- est to our temporal setup. It imposes explicit cutoffs and measures leakage from post-cutoff knowledge, but its tasks are non-clinical (Liu et al., 2026). Our benchmark instead tests whether later events in a patient trajectory shift a judgment about an earlier clinical point toward the hindsight trap. Clinical question answering over longitudinal records. MedAlign, MIMIC-Instr, EHRNoteQA, and EHRSQL evaluate instruction following or ques- tion answering over longitudinal notes and struc- 2 Hindsight Bias in Clinical Temporal Reasoning tured EHR data (Fleming et al., 2024; Wu et al., 2024; Kweon et al., 2024; Lee et al., 2022). More recent benchmarks target temporal reasoning di- rectly.TIMER grounds instructions to patient- specific timestamps, ASCENT evaluates diagnosis as evidence accumulates, and RealICU evaluates win- dowed ICU decisions using hindsight-informed labels (Cui et al., 2025; Choi et al., 2026; Shen et al., 2026). These tasks measure temporal grounding or stepwise reasoning. We pair the same question and pre-cutoff evidence with complete and masked timelines, allow- ing measurement of whether the answer changes and whether it moves in an outcome-consistent direction. Reasoning from clinical case reports. Med- CaseReasoning evaluates diagnostic answers and rea- soning derived from open access case reports, while MedR-Bench covers examination, diagnosis, and treatment planning (Wu et al., 2025; Qiu et al., 2025). Prior work also reconstructs time-localized sepsis tra- jectories from PubMed Central case reports as tex- tual time series (Noroozizadeh and Weiss, 2026). We build on this representation but study a different question: not only whether a model reaches the final conclusion, but whether that conclusion was support- able before later events were known. 3. Methods The overall pipeline is illustrated in Figure 1, and all prompts used are listed in Appendix C. 3.1. Study design and data source Case reports were drawn from the PubMed Central Open Access (PMC OA) Subset provided by National Library of Medicine (2025). We used two cohorts comprising 171 case reports: 40 sepsis cases and 131 cases of type 2 diabetes patients taking Glucagon-like Peptide Receptor Agonists (GLP1-RA) medications. The sepsis and GLP1-RA cohorts build on previously published works (Noroozizadeh and Weiss, 2026; Ku- mar and Weiss, 2026), which provide independently constructed timelines from clinically trained annota- tors and LLM extractors. The two cohorts were se- lected to evaluate the benchmark across contrasting clinical trajectories: comparatively dense, acute care episodes in sepsis and more heterogeneous longitudi- nal trajectories in GLP-1/diabetes. 3.2. Clinical representations Each case i was represented by its original case re- port narrative N i and by two textual time series Case: 58-year-old male admitted with septic shock sec- ondary to pneumonia. [...] Cutoff time: 24 hours post-admission Visible to answering model: Question: What probability range best represents the like- lihood of requiring mechanical ventilation within 48 hours, given the patient’s current respiratory status and haemo- dynamic profile? Format: Multiple choice (ordinal) Options: [A.] <10%, [B.] 10–30%, [C.] 31–60%, [D.] >60% Hidden evaluation metadata: a ∗ (reference answer): A. <10% a trap (hindsight trap): D. >60% Trap explanation: The patient was intubated at 36 hours; a model with access to the full trajectory may inflate the prospective probability estimate in light of this outcome. Eval target: both Cutoff event: Arterial blood gas showing mild hypox- aemia; vasopressor requirement stable. Figure 2: Representative benchmark question. The visible component (above the divider) is shown to the answering model; the hidden metadata (below) is used only for evalua- tion. The reference answer a ∗ is what was defensible at the cutoff, while the hindsight trap answer a trap is what may seem more compelling once the eventual outcome is known. (TTS): a human-annotated TTS T H i and an indepen- dently generated LLM TTS T L i . A TTS is a sequence T s i = (e s ij ,t s ij ) n s i j=1 , s ∈ H, L, where e s ij is a clinical event and t s ij is its relative time in hours. This representation preserves the distinction between narrative order and clinical event order, which may diverge in retrospectively written case reports. 3.3. Question construction Questions were generated with GPT 5.6 Sol with the default medium reasoning effort from the origi- nal narrative N i and human TTS T H i . For each case, the generator returned ten questions as a schema- constrained JSON array; outputs that failed valida- tion received one repair attempt. Across 171 cases, this yielded 1,710 questions. Figure 2 shows a rep- resentative item, including the fields visible to the answering model and the hidden metadata used for evaluation. Questionswereassignedto accuracy, hindsight bias, or both. accuracy items targeted verifiable answers with low hindsight sensitivity; 3 Hindsight Bias in Clinical Temporal Reasoning hindsight bias items targeted subjective judgments such as likelihood and treatment appropriateness that could shift after later evidence; and both items required a prospectively supported answer and a plausible trap.Each case targeted a 2/3/5 split across these categories. The benchmark included free response, true/false, and four-option multiple choice questions, with mul- tiple choice items labeled categorical or ordinal. Ordinal options used explicit ranges, with a ∗ q and a trap q placed at maximally separated positions. The prompt limited each reasoning type to two questions per case to maximize diversity. It also omitted ex- plicit temporal references from visible question text so that prospective reasoning depended on the sup- plied context rather than an overt instruction. 3.4. Temporal masking and cutoff position For each question q, the generator supplied a numeric cutoff c q in the time coordinate of the human TTS. Given a TTS T s i , we retained events at or before the cutoff: T s iq,masked =(e s ij ,t s ij )∈ T s i : t s ij ≤ c q (1) while T s i,full = T s i retained the complete trajectory. The same numeric cutoff was applied to the human and LLM TTS. After sorting valid rows by numeric time and pre- serving source order within ties, let k q ∈ 1,...,n i denote the chronological index matching the cutoff. We define RelPos(q) = k q −1 n i −1 , where 0 and 1 denote the first and last recorded events, respectively. Constructing synthetic case reports. To test whether our results depended on the original case re- port prose or persisted under an independently re- constructed narrative representation, we generated synthetic narratives from the TTS. A separate nar- rative was generated for every question q, cutoff c iq , and annotation source s ∈ H, L: each was derived from the masked TTS T s,mask iq rather than the full TTS, so no post-cutoff information could leak into the narrative seen by the answering model. A single case report i with Q i questions and two annotation sources therefore yields up to 2Q i distinct synthetic narratives, guaranteeing strict temporal alignment between the narrative and the TTS in each paired condition. For each condition, narratives were gener- ated with the same model used for answering. Table 1: Input conditions.H/L-TTS denote human/LLM-generated textual time series; F/M denotes full/masked variants. Condition Model input 0Original narrative 1.1Original narrative + H-TTS (F/M) 1.2Original narrative + L-TTS (F/M) 2.1H-TTS only (F/M) 2.2L-TTS only (F/M) 3.1Synthetic narrative + H-TTS (F/M) 3.2Synthetic narrative + L-TTS (F/M) 3.5. Answer generation We evaluated GPT-5.6 Sol, Gemma-4-31B, GLM- 5.2-FP8, and Opus 5. Each model answered every question independently and received only the visible question fields—question, format, and options— together with the context specified by Table 1. The rest were withheld; see Figure 2. Responses contained an answer and brief supporting evidence. No model had access to external retrieval, tools, or the web. Hindsight warning sensitivity analysis. We re- peated the Sepsis experiments for GPT and GLM after appending explicit instructions to use only con- temporaneously available information and disregard the final outcome. All other procedures were un- changed. 3.6. Experimental conditions We evaluated seven input conditions (Table 1, Ap- pendix B) using the same question set throughout. Condition 0 provided the complete original narrative as an unpaired retrospective baseline. Conditions 1.x combined that narrative with full or masked human- or LLM-generated TTS; only the TTS differed within each pair, so the narrative could still reveal post- cutoff information. Conditions 2.x used TTS alone and therefore directly isolated the effect of tempo- ral masking. Conditions 3.x paired each TTS with a synthetic narrative generated from the corresponding masked or full information scope. Condition 0 is unpaired and therefore supports only Acc and HTR. Conditions 1.x, 2.x, and 3.x pro- vide matched masked and full responses and addi- tionally support ∆Acc, ∆HTR, AIR, and HBR. 3.7. Evaluation Let a q,z denote the model answer to question q under visibility condition z ∈ full, masked. We compare each answer with the prospective reference a ∗ q and 4 Hindsight Bias in Clinical Temporal Reasoning the hindsight trap a trap q . Metrics were computed per question and then averaged. Structured responses were scored by exact match. Free responses were scored by a GPT-5.6 Sol judge against the prospective reference and hindsight-trap answers using0, 0.5, 1 semantic-agreement scores. 3.7.1. Individual metrics Accuracy (Acc). Acc measures agreement with the prospective reference answer a ∗ q and is defined for accuracy and both questions, which comprise 70% of the dataset. For true/false and multiple choice questions, Acc q (a) = 1[a = a ∗ q ]∈0, 1, whereas free response questions receive Acc q (a) ∈ 0, 0.5, 1 from the judge, with 0.5 denoting partial correctness. Hindsight trap rate (HTR). HTR measures agreement with the outcome-consistent hindsight trap a trap q and is defined for hindsight bias and both questions, which comprise 80% of the dataset. For true/false and categorical multiple choice ques- tions, HTR q (a) = 1[a = a trap q ]∈0, 1,(2) whereas free response questions receive HTR q (a) ∈ 0, 0.5, 1 from the judge. For ordinal multiple choice questions, HTR is graded by movement from a ∗ q to- ward a trap q . In Figure 2, for example, answers B, C, and D receive 1 3 , 2 3 , and 1, respectively, yielding HTR q (a)∈0, 1 3 , 2 3 , 1. 3.7.2. Paired metrics Delta accuracy (∆Acc). ∆Acc q = Acc q (a q,masked )− Acc q (a q,full ).(3) A positive ∆Acc q indicates greater correctness under masking, meaning post-cutoff information reduces average accuracy. Delta hindsight trap rate (∆HTR). ∆HTR q = HTR q (a q,full )− HTR q (a q,masked ).(4) A positive ∆HTR q indicates greater trap attraction under full exposure, meaning post-cutoff information increases hindsight bias. Answer instability rate (AIR). AIR records any answer change, regardless of direction: AIR q = 1[a q,full ̸≡ a q,masked ]∈0, 1.(5) AIR upper-bounds any bias effect because a model answer must change to exhibit hindsight bias. Hindsight bias rate (HBR). HBR captures the joint event that a model is fully correct under mask- ing and moves toward the trap under full exposure. It is defined only for both questions, for which both a ∗ q and a trap q are available. Let I M q = 1[Acc q (a q,masked ) = 1]. Then HBR q = I M q · HTR q (a q,full ),(6) so HBR is nonzero only when the masked answer is correct. 3.7.3. Statistical analyses We estimated 95% confidence intervals using a case- clustered bootstrap with 2,000 replicates. Cases were sampled with replacement, and all questions and paired responses from each sampled case were re- tained. Sepsis and GLP-1/diabetes were analyzed separately. For metrics evaluated relative to zero, effects were considered statistically significant when the 95% bootstrap confidence interval excluded zero. 3.8. Human validation Four clinical annotators (one physician and three medical or physician assistant trainees) reviewed an error-enriched 10% sample of cases, split into two non-overlapping subsets with two annotators each. Annotators assessed whether each question was clini- cally reasonable and whether they agreed with the prospective reference and hindsight-trap answers. Sampling and validation details are provided in Ap- pendix G. 4. Results Figure 3 summarizes Accuracy and Hindsight Trap Rate (HTR) for GPT-5.6 Sol on the GLP and Sepsis datasets. Condition 0 serves as the full information baseline, while Conditions 1.1–3.2 allow direct com- parison between the full and masked settings. Results for Gemma 4, GLM 5.2, and Opus 5 are provided in Appendix E. Naive narrative answering already shows sub- stantial hindsight bias. Before considering the paired masking experiments, condition 0 establishes that hindsight bias is already a meaningful problem when the model answers questions directly from the clinical narrative. As shown in Figure 3C and Fig- ure 3D, in both datasets, the hindsight trap rate (HTR) is significantly greater than zero. 14–15% of responses are hindsight-biased when naively answer- ing questions based on narratives. Paired masking reveals robust hindsight bias effects. The paired conditions (1.x–3.x) allow hind- sight bias to be quantified directly using three com- plementary metrics: ∆HTR, AIR, and HBR. As seen 5 Hindsight Bias in Clinical Temporal Reasoning 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 Accuracy (A) GLP Full Masked 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 (B) Sepsis Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 Hindsight Trap Rate (HTR) (C) Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 (D) Full Masked Individual Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. Figure 3: Accuracy and Hindsight Trap Rate (HTR) across experimental conditions for GPT for the GLP and sepsis datasets. Condition 0 represents the full information setting. For Conditions 1.1–3.2, results are shown for both the full and masked settings. Points indicate observed values and error bars indicate 95% bootstrap confidence intervals. in Figure 4, across both datasets, AIR and HBR are significantly greater than zero in every paired con- dition, indicating that access to future information systematically changes model responses and, more stringently, can shift initially correct prospective an- swers toward predefined hindsight traps. This is par- ticularly notable for HBR, which is a strict subset of AIR and therefore distinguishes hindsight-biased shifts from general answer instability. The evidence is strongest under strict temporal masking (Condi- tions 2.x and 3.x), where all three metrics are signif- icantly positive across both datasets. For example, AIR reaches 16.0% in Sepsis Condition 2.2, while HBR reaches 5.5% in the same condition, demon- strating that a meaningful subset of responses not only changed after future information was revealed but shifted specifically from a prospectively correct answer toward the hindsight trap. Overall, the con- sistency of ∆HTR, AIR, and HBR under strict mask- ing shows that the observed effects are not merely general response instability: access to future clinical information systematically shifts model judgments toward hindsight-consistent answers. Condition 1 provides weaker temporal isola- tion. In conditions 1.1 and 1.2, ∆HTR remains small and non-significant despite significant AIR and HBR (Figure 4). Unlike conditions 2.x and 3.x, the masked variant in condition 1 retains the complete original narrative, which may contain post-cutoff or outcome-relevant information. Consequently, the full–masked comparison isolates only the effect of masking the TTS, not the effect of removing future information from the model’s entire input. The sig- nificant AIR and HBR nevertheless show that chang- ing TTS access can alter responses, including shifts from prospectively correct answers toward hindsight traps, even when the retrospective narrative remains available. Human versus LLM TTS. Across both datasets, the human-TTS variants generally achieve slightly higher absolute accuracy than the corresponding LLM-TTS variants. This comparison should be in- terpreted in light of the benchmark construction: the human TTS was used as the reference representa- tion during question generation. Discrepancies in 6 Hindsight Bias in Clinical Temporal Reasoning 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 0.25 Rate (A) GLP 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition (B) Sepsis Paired Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. ΔHTR AIRHBR Figure 4: Hindsight bias metrics across paired experimental conditions for GPT for the GLP and sepsis datasets. We report ∆HTR, AIR, and HBR for Conditions 1.1–3.2. Points indicate observed values and error bars indicate 95% bootstrap confidence intervals. the LLM-reconstructed TTS—including differences in event content or timing—may therefore omit in- formation needed to answer a question or place infor- mation on the wrong side of its masking cutoff. The synthetic narrative conditions support the masking interpretation. Conditions 3.x are especially informative because they retain a narra- tive style prompt while restoring temporal control through a synthetic narrative. Their results closely track the strict masking pattern seen in 2.x: all hind- sight bias metrics are significantly positive, and their magnitudes are much closer to 2.x than to the atten- uated 1.x conditions. This suggests that the weaker effects in 1.x are not due to narrative prompting per se, but to leakage from the original unmasked nar- rative. The synthetic narrative therefore appears to preserve the benefits of a narrative format without undermining the masking manipulation. Bias reduction does not come at the cost of accuracy. In contrast to the consistent hindsight bias signal, Figure 3A and Figure 3B highlight that ∆Acc is not significant in any paired condition in either dataset. Although some point estimates are positive and others negative, there is no reliable evi- dence that access to the full case improves accuracy relative to the masked version. This is a central result that masking reduces hindsight bias without imposing a detectable accuracy penalty. Put differ- ently, future information is systematically biasing the model’s answers, but not meaningfully helping it an- swer correctly. This strengthens the case for tempo- rally masked evaluation as a way to reduce hindsight bias while preserving task performance. Explicit hindsight warning attenuates but does not eliminate the effect. As a sensitiv- ity analysis, we repeated the Sepsis evaluation for GPT and GLM with explicit instructions to avoid hindsight. The warning generally reduced hindsight- related metrics, particularly under strict masking, while accuracy remained broadly similar. However, AIR and HBR remained positive across paired con- ditions, indicating that hindsight-sensitive response changes persisted despite the explicit instruction. Full results are reported in Appendix F. Human validation supports benchmark la- bel validity. Human review of the error-enriched validation sample found that 95.9% of questions were judged clinically reasonable, with agreement rates of 98.7% for the prospective reference answers and 97.1% for the hindsight-trap answers. Inter- annotator agreement was 91.8% for question rea- sonableness, 99.2% for the prospective answer, and 94.1% for the hindsight trap. Detailed results are reported in Appendix G. 5. Discussion Principal findings. Across both cohorts, exposure to the complete timeline shifted model responses to- 7 Hindsight Bias in Clinical Temporal Reasoning ward outcome-consistent alternatives.In the un- paired narrative only baseline, the hindsight trap rate was 14–15%; because this condition has no masked counterpart, it is best interpreted as a descriptive baseline rate rather than a causal estimate of hind- sight bias. The paired conditions provide stronger evidence. AIR and HBR were positive across all paired input settings, and strict temporal masking (conditions 2.x and 3.x) produced the largest changes in trap attraction, with ∆HTR ranging from 5.8% to 9.6% and HBR from 3.3% to 5.5%.By con- trast, ∆Acc was statistically insignificant.Thus, later information changed a subset of answers in the outcome-consistent direction without a correspond- ing improvement in overall accuracy. The condition comparisons clarify the role of in- formation control. In conditions 1.x, the original narrative remained complete in both variants, so fu- ture information was shared across the pair and the masked–full contrast was attenuated. Conditions 2.x removed the narrative, while conditions 3.x paired each TTS with a synthetic narrative generated from the corresponding information scope. The larger ef- fects in these strictly controlled settings suggest that the information boundary, rather than narrative for- mat alone, drives the measured difference. Human- and LLM-generated TTS produced the same quali- tative pattern, although effect sizes varied by cohort and condition. While our experiments do not iden- tify the source of the observed bias, its persistence after explicit hindsight warning prompting suggests that instructions alone may be insufficient to elimi- nate it. This result suggests that input level temporal control may be more reliable than instruction alone when evaluating judgments about an earlier clinical state. Validity of benchmark labels. Human review provides additional support for the validity of the GPT-generated questions, prospective reference an- swers, and hindsight traps (Section 3.8; Appendix G). Importantly, the validation set was intentionally en- riched for cases in which models made errors or ex- hibited hindsight-sensitive behavior—the same cases that contribute most directly to our evidence of hind- sight bias. Positive human validation on this chal- lenging subset therefore strengthens confidence that the benchmark’s central findings are not driven by invalid questions or labels. Furthermore, our various metrics provide robust- ness to annotation error by examining hindsight bias from different angles that rely on different annotation components: AIR requires only an answer change, Acc depends on the prospective reference answer, HTR depends on the hindsight-trap answer, and HBR combines both. Convergence across these com- plementary measures, particularly under strict mask- ing, makes it less likely that the observed hindsight- bias signal is an artifact of any single annotation label or metric definition. Limitations. Our studies have several limitations. First, the reference time was hidden and questions were intentionally unanchored, so some full–masked differences may reflect changes in task interpretation rather than hindsight bias. Explicitly specifying the reference time and allowable evidence could reduce this ambiguity, but such tightly structured questions may not reflect real clinical deployments, where mod- els may need to infer the relevant temporal context from available information. Second, free-response scoring relies on an LLM judge and may be sensitive to paraphrase or judge-specific variation. However, the judge only compares responses with predefined reference and hindsight-trap answers, and most ques- tions use structured formats, reducing reliance on LLM-based scoring. Third, published case reports overrepresent unusual and educational cases, limit- ing generalizability to routine clinical practice. We use these datasets because they are publicly avail- able and, importantly, provide manually annotated TTS, enabling controlled evaluation of temporal rea- soning with high-quality representations of the clin- ical trajectories. Fourth, temporal masking relies on recorded timestamps, which may be imprecise or misplace information relative to a cutoff. Moreover, observation time may not reflect the true onset or duration of a clinical state. Future directions. Evaluations of earlier clinical decisions should restrict model inputs to information available at the decision point and report directional measures of outcome-sensitive change alongside ac- curacy. Future work should extend the benchmark to routine longitudinal EHRs, larger expert-reviewed samples, and counterfactual outcome controls, while investigating training strategies and model archi- tectures that better enforce prospective information boundaries. 6. Conclusion We introduced a paired benchmark for testing whether later clinical information changes LLM judg- ments about an earlier decision point. Across two 8 Hindsight Bias in Clinical Temporal Reasoning clinical cohorts and four models, complete trajecto- ries increased movement toward outcome-consistent hindsight traps, whereas temporal masking reduced these shifts without a detectable loss in accuracy. These findings show that retrospective clinical eval- uation can overstate prospective reasoning perfor- mance and motivate evaluation protocols that restrict model inputs to information available at the target decision time. Acknowledgements This research was supported by the Intramural Re- search Program of the National Institutes of Health (NIH) and utilized the computational resources of the NIH HPC Biowulf cluster. The contributions of the NIH author(s) are considered Works of the United States Government. The findings and conclusions presented in this paper are those of the author(s) and do not necessarily reflect the views of the NIH or the U.S. Department of Health and Human Services. References H. R. Arkes, R. L. Wortmann, P. D. Saville, and A. R. Harkness.Hindsight bias among physi- cians weighing the likelihood of diagnoses. Journal of Applied Psychology, 66(2):252–254, 1981. doi: 10.1037/0021-9010.66.2.252. Edward Banham-Hall and Sian Stevens. Hindsight bias critically impacts on clinicians’ assessment of care quality in retrospective case note review. Clin- ical Medicine, 19(1):16–21, 2019. doi: 10.7861/clin medicine.19-1-16. Yera Choi, Yeong Hwa Kim, JaeDeok Lee, Taekang Kim, Sangdoo Yun, and Seong-Eun Moon. AS- CENT: A benchmark for evaluating and advanc- ing stepwise diagnostic reasoning in large language models on common clinical scenarios. In Proceed- ings of the 7th Conference on Health, Inference, and Learning, volume 333 of Proceedings of Ma- chine Learning Research, pages 404–426. PMLR, 2026. URL https://proceedings.mlr.press/v333/c hoi26a.html. Zheng Chu, Jingchang Chen, Qianglong Chen, Wei- jiang Yu, Haotian Wang, Ming Liu, and Bing Qin. TimeBench: A comprehensive evaluation of tempo- ral reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1204–1228, Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl- long.66. URL https://aclanthology.org/2024.acl-long.66/. Hejie Cui, Alyssa Unell, Bowen Chen, Jason Alan Fries, Emily Alsentzer, Sanmi Koyejo, and Nigam H. Shah. TIMER: Temporal instruction modeling and evaluation for longitudinal clinical records. npj Digital Medicine, 8:577, 2025. doi: 10.1038/s41746-025-01965-9. URL https://w. nature.com/articles/s41746-025-01965-9. Neal V. Dawson, Hal R. Arkes, Carl Siciliano, Richard Blinkhorn, Mark Lakshmanan, and Mary Petrelli. Hindsight bias: An impediment to ac- curate probability estimation in clinicopathologic conferences. Medical Decision Making, 8(4):259– 264, 1988. doi: 10.1177/0272989X8800800406. Or Degany, Sahar Laros, Daphna Idan, and Sharon Einav. Evaluating the o1 reasoning large language model for cognitive bias: a vignette study. Critical Care, 29(1):376, 2025. doi: 10.1186/s13054-025-0 5591-5. Bahare Fatemi, Seyed Mehran Kazemi, Anton Tsit- sulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. Test of time: A benchmark for evaluating LLMs on temporal reasoning. In Inter- national Conference on Learning Representations, 2025. URL https://proceedings.iclr.c/paper files /paper/2025/hash/eb7295a8bc613b375726659c2ec d6f14-Abstract-Conference.html. Baruch Fischhoff. Hindsight is not equal to foresight: The effect of outcome knowledge on judgment un- der uncertainty. Journal of Experimental Psychol- ogy: Human Perception and Performance, 1(3): 288–299, 1975. doi: 10.1037/0096-1523.1.3.288. Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott Adams, Oluseyi Fayanju, Shreya J. Shah, Thomas Savage, Ethan Goh, Ak- shay S. Chaudhari, Nima Aghaeepour, Christo- pher Sharp, Michael A. Pfeffer, Percy Liang, 9 Hindsight Bias in Clinical Temporal Reasoning Jonathan H. Chen, Keith E. Morse, Emma P. Brunskill, Jason A. Fries, and Nigam H. Shah. MedAlign: A clinician-generated dataset for in- struction following with electronic medical records. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 38, pages 22021–22030, 2024. doi: 10.1609/aaai.v38i20.30205. URL https://o js.aaai.org/index.php/AAAI/article/ view/30205. Sayantan Kumar and Jeremy C. Weiss. Temporally phenotyping GLP-1RA case reports with large lan- guage models: A textual time series corpus and risk modeling. medRxiv, 2026. doi: 10.64898/2026.04.0 5.26350197. Preprint. Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwanghyun Kim, Jeewon Yang, Seunghyun Won, and Edward Choi. EHRNoteQA: An LLM benchmark for real-world clinical practice using discharge summaries. In Ad- vances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-3958. URL https://proceedings.neurips.c/paper f iles/paper /2024/hash/e15c4af f22f 12c4986c1f cb4e941e03 -Abstract-Datasets andBenchmarksTrack.html. Datasets and Benchmarks Track. Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong-Yeup Kim, and Edward Choi. EHRSQL: A practical text-to-SQL benchmark for electronic health records. In Advances in Neural Information Processing Systems, volume 35, 2022. doi: 10.522 02/068431-1134. URL https://proceedings.neurip s.c/paper f iles/paper/2022/hash/643e347250cf9 289e5a2a6c1ed5e42e-Abstract-DatasetsandBen chmarks.html. Datasets and Benchmarks Track. Yachuan Liu, Xiaochun Wei, Lin Shi, Xinnuo Li, Bo- han Zhang, Paramveer Dhillon, and Qiaozhu Mei. ExAnte: A benchmark for ex-ante inference in large language models. In Proceedings of the 19th Conference of the European Chapter of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 1551–1571, Rabat, Morocco, March 2026. Association for Computational Lin- guistics. doi: 10.18653/v1/2026.eacl-long.72. URL https://aclanthology.org/2026.eacl-long.72/. National Library of Medicine. Pubmed central open access subset (PMC OA).National Library of Medicine Open Data Portal, 2025. URL https://da tadiscovery.nlm.nih.gov/Literature/PubMed-Cen tral-Open-Access-Subset-PMC-OA-/qbit-hxvw. Last updated June 18, 2025; accessed August 8, 2026. Shahriar Noroozizadeh and Jeremy Weiss. Recon- structing sepsis trajectories from clinical case re- ports using LLMs: The textual time series corpus for sepsis. In Proceedings of the 7th Conference on Health, Inference, and Learning, volume 333 of Proceedings of Machine Learning Research, pages 10–51. PMLR, 2026. URL https://proceedings.ml r.press/v333/noroozizadeh26a.html. Pengcheng Qiu, Chaoyi Wu, Shuyu Liu, Yanjie Fan, Weike Zhao, Zhuoxia Chen, Hongfei Gu, Chuanjin Peng, Ya Zhang, Yanfeng Wang, and Weidi Xie. Quantifying the reasoning abilities of LLMs on clin- ical cases. Nature Communications, 16:9799, 2025. doi: 10.1038/s41467- 025- 64769-1. URL https: //w.nature.com/articles/s41467-025-64769-1. Chengzhi Shen, Weixiang Shen, Tobias Susetzky, Chen Chen, Jun Li, Yuyuan Liu, Xuepeng Zhang, Zhenyu Gong, Daniel Rueckert, and Jiazhen Pan. RealICU: Do LLM agents understand long-context ICU data?a benchmark beyond behavior im- itation.arXiv preprint arXiv:2605.13542, 2026. doi: 10.48550/arXiv.2605.13542. URL https: //arxiv.org/abs/2605.13542. Jonathan Wang and Donald A. Redelmeier. Cogni- tive biases and artificial intelligence. NEJM AI, 1 (12):AIcs2400639, 2024. doi: 10.1056/AIcs240063 9. Yuqing Wang and Yun Zhao. TRAM: Benchmarking temporal reasoning for large language models. In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 6389–6415, Bangkok, Thailand, August 2024. Association for Computa- tional Linguistics. doi: 10.18653/v1/2024.f inding s-acl.382. URL https://aclanthology.org/2024.fin dings-acl.382/. Shaohang Wei, Wei Li, Feifan Song, Wen Luo, Tianyi Zhuang, Haochen Tan, Zhijiang Guo, and Houfeng Wang. TimE: A multi-level benchmark for tem- poral reasoning of LLMs in real-world scenarios. In Advances in Neural Information Processing Sys- tems, volume 38, 2025. doi: 10.52202/085713-3082. URL https://papers.neurips.c/paper f iles/paper /2025/hash/84f 1e188c2be52f 89f6e206bc37d092 10 Hindsight Bias in Clinical Temporal Reasoning d-Abstract-Datasets andBenchmarksTrack.htm l. Datasets and Benchmarks Track. Kevin Wu, Eric Wu, Rahul Thapa, Kevin Wei, An- gela Zhang, Arvind Suresh, Jacqueline J. Tao, Min Woo Sun, Alejandro Lozano, and James Zou. MedCaseReasoning: Evaluating and learning diag- nostic reasoning from clinical case reports. arXiv preprint arXiv:2505.11733, 2025. doi: 10.48550/a rXiv.2505.11733. URL https://arxiv.org/abs/2505 .11733. Zhenbang Wu, Anant Dadu, Mike Nalls, Faraz Faghri, and Jimeng Sun. Instruction tuning large language models to understand electronic health records. In Advances in Neural Information Pro- cessing Systems, volume 37, 2024. doi: 10.52202 /079017-1737. URL https://proceedings.neurips. c/paper f iles/paper/2024/hash/62986e0a78780 fe5f 17b495aeded5bab-Abstract-DatasetsandBen chmarksTrack.html. Datasets and Benchmarks Track. 11 Hindsight Bias in Clinical Temporal Reasoning Supplementary Material This supplementary material provides detailed infor- mation organized as follows: • Appendix A: Detailed literature review. • Appendix B: Experimental conditions and in- put configurations. • Appendix C: Prompts used for question con- struction, answer generation, evaluation, and synthetic narrative generation. • Appendix D: Distribution of temporal masking cutoffs across the clinical trajectories. • Appendix E: Complete results across other models (Gemma, GLM, Opus). • Appendix F: Sensitivity analysis with explicit hindsight warning. • Appendix G: Human validation of benchmark questions, prospective answers, and hindsight traps. Appendix A. Detailed Literature Review General temporal reasoning Several bench- marks evaluate temporal reasoning in general do- main LLMs. TRAM covers event order, arithmetic, frequency, and duration (Wang and Zhao, 2024); TimeBench provides a hierarchical evaluation across a broad range of temporal phenomena (Chu et al., 2024); Test of Time uses controlled synthetic tasks to study temporal logic and sensitivity to problem structure (Fatemi et al., 2025); and TimE evaluates multi-level reasoning over Wikipedia, news, and di- alogue scenarios (Wei et al., 2025). Most closely re- lated, ExAnte evaluates inference under explicit tem- poral cutoffs and measures leakage from post-cutoff knowledge across several non-clinical tasks (Liu et al., 2026). These benchmarks establish that temporal constraints remain difficult for LLMs, but they do not study clinical judgments or quantify whether ex- posing the later course of the same patient trajectory moves an answer toward a clinically defined hindsight trap. Clinical question answering over longitu- dinal records. Clinical benchmarks increasingly test LLMs on realistic or longitudinal EHR tasks. MedAlign contains clinician-generated instructions grounded in longitudinal records (Fleming et al., 2024); MIMIC-Instr provides large-scale instruction- following examples derived from MIMIC-IV (Wu et al., 2024); and EHRNoteQA evaluates questions spanning one or more discharge summaries (Kweon et al., 2024). EHRSQL tests text-to-SQL reason- ing over structured EHR databases, including time- sensitive questions and temporal expressions (Lee et al., 2022). TIMER explicitly links instructions to timestamps and evaluates temporal boundary adher- ence, trend detection, and chronological precision in longitudinal EHRs (Cui et al., 2025). ASCENT eval- uates stepwise diagnostic reasoning as evidence accu- mulates under incomplete information (Choi et al., 2026). RealICU evaluates windowed ICU decision support using prefix-only observations and labels produced after physicians review the full trajectory (Shen et al., 2026); unlike our paired design, it uses hindsight to construct reference labels rather than measuring how a model’s own answer changes after outcome exposure. These works evaluate retrieval, temporal synthesis, instruction following, evolving diagnosis, or prospective performance against retro- spectively informed labels. We instead hold the ques- tion and prospective reference target fixed and com- pare masked with complete-record exposure to mea- sure the directional effect of later information on the model’s response. Reasoning from clinical case reports. Clinical case reports have also been used to construct chal- lenging reasoning benchmarks. MedCaseReasoning pairs diagnostic cases with detailed reasoning state- ments derived from open-access case reports and evaluates both final diagnosis and reasoning recall (Wu et al., 2025). MedR-Bench structures case re- ports into examination recommendation, diagnostic decision-making, and treatment-planning tasks (Qiu et al., 2025). Related work has reconstructed time- localized sepsis trajectories from PubMed Open Ac- cess case reports as textual time series (Noroozizadeh and Weiss, 2026). These resources improve the real- ism and granularity of clinical reasoning evaluation, but primarily assess whether a model reaches or ex- plains a final clinical conclusion. Our benchmark in- stead asks when a conclusion was supportable and 12 Hindsight Bias in Clinical Temporal Reasoning measures how the same answer changes when post- cutoff events become visible. Appendix B. Experimental conditions We evaluated seven input conditions (Table 1), each defining what information the answering model has access to when responding to a question. The ques- tion set itself was identical across all conditions: all questions were generated from the original narrative N i paired with the full human TTS T H i , and the an- swering model never sees the generation inputs or the hidden evaluation metadata. Condition 0 Condition 0 serves as a naive baseline representing the setting most common in current clin- ical QA systems, where the model is given a complete retrospective case report and asked to answer ques- tions without any temporal masking. Because case reports are written retrospectively and may reference diagnoses, treatments, and outcomes in any order re- gardless of when they occurred, there is no reliable way to mask post-cutoff information from a narra- tive. Condition 0 therefore reflects the upper bound of potential hindsight exposure under narrative-only input. Condition 1.x Conditions 1.x address this limita- tion partially by pairing the original narrative with a temporally masked TTS. The narrative component is held fixed and complete across the full and masked variants; only the TTS is truncated at the cutoff. While masking the TTS removes explicit post-cutoff temporal data, the original narrative may still im- plicitly contain outcome information, meaning that conditions 1.x achieve only partial information con- trol. Condition 2.x Conditions 2.x remove the narra- tive entirely, providing the TTS alone. This yields the cleanest paired comparison between prefix-only and complete temporal information, as the TTS is the sole information source and can be precisely trun- cated at c q . Condition 3.x Conditions 3.x also achieve rigor- ous temporal control. Conditions 3.x are motivated by the complementary limitations of conditions 1.x and 2.x: while condition 1.x retains a narrative but cannot fully mask it, and condition 2.x achieves strict masking but strips away the clinical context that a narrative provides, conditions 3.x combine both desiderata by supplying a narrative that is simultane- ously present and strictly temporally controlled. The synthetic narrative presented to the answering model is derived from the same masked TTS used in that condition variant, as described in Section 3.2. Be- cause the synthetic narrative is generated from pre- cutoff input, it cannot contain implicit references to subsequent events. Paired with a masked TTS, con- ditions 3.x ensure that all information available to the answering model — both narrative and temporal — is strictly limited to what was observable before the cutoff, a guarantee that conditions 1.x cannot pro- vide. 13 Hindsight Bias in Clinical Temporal Reasoning Appendix C. Prompts Question generation prompt You are generating difficult clinical reasoning questions from case reports to build a benchmark evaluating accuracy and hindsight bias in language models. Questions should be genuinely hard—the answering model should make errors, especially by using outcome knowledge to revise past assessments it should not revise. Generate 10 questions per case report with a mix of formats (free response, multiple choice with 4 options, true/false) and the following distribution by evaluation target: • accuracy (20%): verifiable from case facts; answer does not change with more timeline information • hindsight bias (30%): answer is judgment-based (likelihood, severity, risk); no objective ground truth, but a model with outcome knowledge is strongly tempted to revise its answer • both (50%): defensible correct answer anchored to case facts or clinical guidelines, AND strongly tempting to revise with outcome knowledge Rules: • Questions must be clinically focused, not meta-questions about the data. Never phrase a question in terms of what the TTS “establishes,” “shows,” “records,” or “contains.” Ask about the clinical facts themselves (e.g., “Did angiography occur before embolization?” not “Does the TTS establish a strict chronological sequence between angiography and embolization?”). • Do not include any time reference in the question text. Questions must be atemporal—e.g., “What was the probability of a serious infectious complication?” not “At readmission, what was the probability...”. The answering model’s timeline is controlled externally through masking; an explicit time reference tips off prospective reasoning. • Anchor to early ambiguous presentations where findings were indistinguishable from benign alternatives, not to late confirmed events. • The hindsight trap must be the most attractive answer for a model that knows the outcome—the correct prospective answer should feel conservative or uncertain by comparison. • No more than 2 questions of the same reasoning type (likelihood, severity, causal attribution, treatment appropriateness, diagnostic classification, temporal ordering). • For ordinal multiple choice questions: use exact probability ranges instead of vague labels (e.g., “A. <10%”, “B. 10– 30%”, “C. 30–60%”, “D. >60%”). The correct answer and hindsight trap must be at opposite ends of the scale—if the correct answer is the lowest option, the hindsight trap must be the highest, and vice versa. You are also provided with the textual time series (TTS) for this case—a JSON array of "event": "...", "time": <hours> tuples ordered chronologically by time. Use the TTS to: • Identify events that the narrative describes out of chronological order. These are especially valuable for hindsight bias questions—the narrative’s non-chronological framing may lead a model reading only the narrative to assume a different temporal order than what actually occurred. Design questions that target these moments, where a model relying on narrative framing would answer differently than one reasoning from the true chronological sequence. • Identify tight event clusters (multiple events at similar timestamps) that represent moments of rapid clinical change— these are high-yield anchors for ambiguous early presentations. Return a valid JSON array only. Each element: "question": "...", "format": "free response | multiple choice | true/false", "options": ["A. ...", "B. ...", "C. ...", "D. ..."] or null, "options type": "ordinal | categorical | null", "evaltarget": "accuracy | hindsightbias | both", "referencetime": "clinical moment the question is implicitly about", "earlycutoff": "event": "specific event label from narrative just after reference time", "time": "time expression as written in the TTS" , "correctanswer": "...", "hindsighttrap": "the most tempting wrong answer for a model with full-case access; null if evaltarget is accuracy", "hindsight trapexplanation": "one sentence naming the post-cutoff fact that makes the trap so tempting; null if evaltarget is accuracy" 14 Hindsight Bias in Clinical Temporal Reasoning Answer generation prompt: Condition 0 (Narrative only) You answer clinical reasoning questions using only the provided case report. For each request, you will receive: 1. A case report. 2. One JSON question object containing only these fields: "question", "format", and "options". Answer only from the visible case report and the visible question object. Use general medical knowledge only to interpret clinical language, not to invent patient-specific facts. Rules: • If "format" is "true/false", answer exactly "true" or "false". • If "format" is "multiple choice", answer with exactly one option string from "options", including the option letter and text. • If "format" is "free response", answer concisely in one to three sentences. • Always include an "evidence" field explaining why you gave the answer. • The evidence should be a concise case-grounded rationale, ideally naming the relevant event, symptom, test, treatment, timing, or absence of needed information from the case report. • Do not include reasoning, citations, markdown, or extra text outside the JSON. Return only valid JSON in this exact shape: "answer": "...", "evidence": "..." Answer generation prompt: Conditions 1 and 3 (Narrative + TTS) You answer clinical reasoning questions using the provided case report and its textual time series (TTS). For each request, you will receive: 1. A case report. 2. A TTS—a JSON array of "event": "...", "time": <hours> tuples ordered chronologically by time relative to an arbitrary anchor event. 3. One JSON question object containing only these fields: "question", "format", and "options". Answer using both the case report and the TTS. When they conflict, trust the TTS for event ordering and timing, and the narrative for clinical detail. Use general medical knowledge only to interpret clinical language, not to invent patient-specific facts. Rules: • If "format" is "true/false", answer exactly "true" or "false". • If "format" is "multiple choice", answer with exactly one option string from "options", including the option letter and text. • If "format" is "free response", answer concisely in one to three sentences. • Always include an "evidence" field explaining why you gave the answer. • The evidence should be a concise rationale grounded in the case report or TTS, naming the relevant event, timestamp, symptom, test, treatment, or absence of needed information. • Do not include reasoning, citations, markdown, or extra text outside the JSON. Return only valid JSON in this exact shape: "answer": "...", "evidence": "..." 15 Hindsight Bias in Clinical Temporal Reasoning Answer generation prompt: Condition 2 (TTS only) You answer clinical reasoning questions using only the provided textual time series (TTS). For each request, you will receive: 1. A TTS—a JSON array of"event": "...", "time": <hours> tuples ordered chronologically by time relative to an arbitrary anchor event. 2. One JSON question object containing only these fields: "question", "format", and "options". Answer using only the TTS. Use general medical knowledge only to interpret clinical language, not to invent patient- specific facts. Rules: • If "format" is "true/false", answer exactly "true" or "false". • If "format" is "multiple choice", answer with exactly one option string from "options", including the option letter and text. • If "format" is "free response", answer concisely in one to three sentences. • Always include an "evidence" field explaining why you gave the answer. • The evidence should be a concise rationale grounded in the TTS, naming the relevant event, timestamp, or absence of needed information. • Do not include reasoning, citations, markdown, or extra text outside the JSON. Return only valid JSON in this exact shape: "answer": "...", "evidence": "..." Hindsight warning prompt For the sensitivity analysis experiments, the answer-generation prompts for Conditions 0–3 were identical to the corresponding prompts above, except that the following two sentences were appended to the beginning (for GPT) or end (for GLM) of each prompt: Use your medical expertise for this reasoning task; do not let hindsight cloud your judgment. Use only knowledge available at the time; do not let the final outcome of the case affect your answer. Synthetic case generation prompt: Condition 3 You are a medical expert tasked with reconstructing a synthetic clinical case report from a patient timeline presented in textual time-series (TTS) format. The input contains two columns: event and time in hours, where the timeline conventions are as follows: Time 0 represents the patient’s hospital admission; events occurring before admission have negative time values and events occurring after admission have positive time values. Guidelines: • Use clinical and natural language to convert fragmented timeline entries into a readable narrative. • Follow the order in which events appear in the TTS file as the narrative sequence. • Central events include: symptom onset or major clinical changes; hospital admission, transfer, discharge, or death; diagnostic tests and clinically significant results; diagnoses; procedures and interventions; medication initiation, discon- tinuation, or major adjustment; complications and treatment responses. • Express the timing of central events relative to nearby central events (e.g., “two weeks after that,” “one month after presentation,” “shortly after admission”) using any unit of time (hours, days, weeks, months, years) that is the most natural to read. • Use a standalone or absolute timepoint only when there is no clear anchoring event or when it is clearer than a relative phrase. • Include only information explicitly supported by the input timeline. • Do not infer, invent, or add diagnoses, symptoms, treatments, results, outcomes, demographic details, or causal relation- ships that are not documented. 16 Hindsight Bias in Clinical Temporal Reasoning • When the timeline is ambiguous or incomplete, report the information conservatively without any speculation. Output Requirements: Return only the synthetic clinical case report. Do not include headings about the task, explanations, notes, disclaimers, or commentary outside the report. Example input: 42-year-old | 0 female | 0 no pathological history | 0 consulted the rheumatology clinic | 0 initiated subcutaneous semaglutide for weight loss | -1008 starts complaining of myalgia | -672 edema in the four limbs | -672 review of symptoms was negative | -672 new contrast MRI took place | +730 notable decrease in inflammatory changes in the limb muscles | +730 Example output: A 42-year-old female patient with no pathological history consulted the rheumatology clinic. Six weeks before the consultation, she had initiated subcutaneous semaglutide for weight loss. Two weeks after that, she starts complaining of myalgia and edema in the four limbs. The review of symptoms by systems was negative. One month after presentation, a new contrast MRI showed a notable decrease in the extent of inflammatory changes in the limb muscles. GPT judge prompt: Individual metrics for free-response questions You are evaluating an answer to a clinical temporal-QA question. Return only JSON. Decide clinical meaning, not exact wording. • accuracy score must be 1 for fully correct, 0.5 for partially correct, and 0 for incorrect. • htrscore must be 1 if the answer fully matches the hindsighttrap, 0.5 if it partially reflects the hindsight trap, and 0 if it does not reflect the trap. • If hindsighttrap is null, htrscore must be 0. GPT judge prompt: Paired metrics for free-response questions You are evaluating paired answers to a clinical temporal-QA question. Return only JSON. Decide clinical meaning, not exact wording. • fullequalsmasked is true if the full-condition and masked-condition answers mean the same thing clinically. • For fullaccuracyscore and maskedaccuracyscore, use 1 for fully correct, 0.5 for partially correct, and 0 for incorrect. • For fullhtrscore and maskedhtrscore, use 1 if the answer fully matches the hindsighttrap, 0.5 if it partially reflects the hindsight trap, and 0 if it does not reflect the trap. • If hindsighttrap is null, both HTR scores and hbrscore must be 0. • For hbr score, give hindsight-bias credit only to the extent that the masked answer is correct and the full answer reflects the trap; use 1, 0.5, or 0. 17 Hindsight Bias in Clinical Temporal Reasoning Appendix D. Distribution of Masking Cutoffs To characterize where masking occurs within the clin- ical timelines, we define the relative masking cutoff position as the chronological index of the cutoff event within each TTS, normalized to the interval [0, 1], where 0 and 1 correspond to the beginning and end of the timeline, respectively. This measure reflects po- sition within the sequence of recorded events rather than elapsed clinical time. Table D.1 summarizes the distribution of relative cutoff positions, RelPos(q), across the two datasets. In both cases, the mean and median were near the midpoint of the timeline, and the standard deviations indicate substantial spread across temporal positions. The observed ranges were also broad, showing that questions span much of the clinical trajectory rather than clustering within a narrow temporal window. Table D.1: Distribution of relative cutoff posi- tion RelPos(q) across questions in each dataset. SepsisGLP Mean0.4990.565 Median0.5000.581 SD0.2060.206 Min0.0290.000 Max0.9451.000 Figure D.1 shows that masking points are dis- tributed broadly across the clinical timeline rather than being concentrated at a single position. Sep- sis cutoffs tend to occur somewhat earlier than GLP cutoffs, particularly in the middle of the timeline, al- though both datasets span nearly the full range of rel- ative positions. This distribution indicates that the prospective evaluation is conducted at diverse stages of the underlying clinical trajectories rather than at a fixed or narrowly defined point in each case. Appendix E. Results for Other Models Figures E.1–E.6 report the case-level 95% boot- strap confidence intervals for the individual and paired evaluation metrics for Gemma-4, GLM-5.2, and Opus-5. Comparison across models. Across GPT-5.6 Sol, Gemma-4, GLM-5.2, and Opus-5, accuracy re- 0.00.20.40.60.81.0 Relative masking cutoff position 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative incidence Cumulative Distribution of Masking Cutoff Positions GLP (n=1277) Sepsis (n=400) Figure D.1: Empirical cumulative distribution of rel- ative masking cutoff positions for GLP and Sepsis. Relative position is defined using the chronological index of the cut- off event within the TTS and normalized to [0, 1], with 0 representing the begin- ning and 1 the end of the recorded event sequence. mained generally high, with relatively modest full– masked differences for most paired conditions (Fig- ure E.7).Condition 0 is an unpaired original- narrative baseline and is therefore shown only un- der full information. In contrast to accuracy, HTR showed a clearer masking effect: full and masked HTR were similar in the 1.x conditions, whereas masked HTR was consistently lower across models in the strict 2.x and 3.x conditions (Figure E.8). The paired metrics reinforce this pattern (Fig- ure E.9). ∆HTR was near zero in the 1.x condi- tions but consistently positive under strict masking, while AIR also increased substantially in the 2.x and 3.x conditions. Positive HBR further indicates that some answer changes were directional, shifting from the prospective reference under masking toward the hindsight trap when post-cutoff information was re- stored. Although effect sizes varied across models and datasets, the same qualitative pattern appeared across all four model families. Cross-model differences should be interpreted cau- tiously. GPT was the only answerer from the same family as the question generator, and GPT was also used to judge free-response answers; however, the similar strict-masking pattern among Gemma, GLM, 18 Hindsight Bias in Clinical Temporal Reasoning 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 Accuracy (A) GLP Full Masked 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 (B) Sepsis Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 Hindsight Trap Rate (HTR) (C) Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 (D) Full Masked Individual Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. Figure E.1: Accuracy and HTR results for Gemma with case-level 95% bootstrap confidence intervals. 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 0.25 Rate (A) GLP 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition (B) Sepsis Paired Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. ΔHTR AIRHBR Figure E.2: Hindsight bias metrics for Gemma with case-level 95% bootstrap confidence intervals. 19 Hindsight Bias in Clinical Temporal Reasoning 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 Accuracy (A) GLP Full Masked 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 (B) Sepsis Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 Hindsight Trap Rate (HTR) (C) Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 (D) Full Masked Individual Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. Figure E.3: Accuracy and HTR results for GLM with case-level 95% bootstrap confidence intervals. 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 0.25 Rate (A) GLP 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition (B) Sepsis Paired Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. ΔHTR AIRHBR Figure E.4: Hindsight bias metrics for GLM with case-level 95% bootstrap confidence intervals. 20 Hindsight Bias in Clinical Temporal Reasoning 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 Accuracy (A) GLP Full Masked 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1.00 (B) Sepsis Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 Hindsight Trap Rate (HTR) (C) Full Masked 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 (D) Full Masked Individual Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. Figure E.5: Accuracy and HTR results for Opus-5 with case-level 95% bootstrap confidence intervals. 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 0.25 Rate (A) GLP 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition (B) Sepsis Paired Metrics by Condition * Masked condition is strictly protected from post-cutoff information leakage. ΔHTR AIRHBR Figure E.6: Hindsight bias metrics for Opus-5 with case-level 95% bootstrap confidence intervals. 21 Hindsight Bias in Clinical Temporal Reasoning and Opus suggests that the overall effect is not spe- cific to GPT. The plots also suggest that lower- performing model–condition combinations can ex- hibit greater degradation under masking, but this re- lationship is not monotonic across models or datasets. Thus, overall model quality may contribute to robust- ness, but accuracy and susceptibility to hindsight in- formation appear to capture distinct aspects of model behavior. 22 Hindsight Bias in Clinical Temporal Reasoning 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * 0.800 0.825 0.850 0.875 0.900 0.925 0.950 0.975 1.000 Accuracy GLP — Full 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Sepsis — Full 0 (N/A) 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.800 0.825 0.850 0.875 0.900 0.925 0.950 0.975 1.000 Accuracy GLP — Masked 0 (N/A) 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition Sepsis — Masked Model Comparison: Accuracy * Masked condition is strictly protected from post-cutoff information leakage. GPTGemmaGLMOpus Figure E.7: Accuracy across models and input conditions. Accuracy for GPT-5.6 Sol, Gemma-4, GLM-5.2, and Opus-5 on GLP and Sepsis under full (top) and masked (bottom) information. Asterisks denote strict masking conditions. 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * 0.00 0.05 0.10 0.15 0.20 0.25 Hindsight Trap Rate (HTR) GLP — Full 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Sepsis — Full 0 (N/A) 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 0.25 Hindsight Trap Rate (HTR) GLP — Masked 0 (N/A) 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition Sepsis — Masked Model Comparison: Hindsight Trap Rate * Masked condition is strictly protected from post-cutoff information leakage. GPTGemmaGLMOpus Figure E.8: Hindsight Trap Rate across models and input conditions. HTR for GPT-5.6 Sol, Gemma-4, GLM-5.2, and Opus-5 on GLP and Sepsis under full (top) and masked (bottom) information. Asterisks denote strict masking conditions. 23 Hindsight Bias in Clinical Temporal Reasoning −0.05 0.00 0.05 0.10 0.15 0.20 0.25 Δ HTR GLPSepsis −0.05 0.00 0.05 0.10 0.15 0.20 0.25 AIR 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 0.25 HBR 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition Model Comparison: Paired Hindsight-Bias Metrics * Masked condition is strictly protected from post-cutoff information leakage. GPTGemmaGLMOpus Figure E.9: Paired hindsight-bias metrics across models. ∆HTR, Answer Instability Rate (AIR), and Hindsight Bias Rate (HBR) for GPT, Gemma, GLM, and Opus on GLP and Sepsis. Asterisks denote strict masking conditions. 24 Hindsight Bias in Clinical Temporal Reasoning Appendix F. Sensitivity Analysis with Explicit Hindsight Warning Figures F.1–F.2 and F.3–F.4 report the sensitivity analysis using explicit hindsight warning for GPT and GLM, respectively. Error bars indicate case-level 95% bootstrap confidence intervals. The modification in prompts can be found in Appendix C. 0.850 0.875 0.900 0.925 0.950 0.975 1.000 Accuracy (A) GPT — Sepsis 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 Hindsight Trap Rate (HTR) (B) Original vs. Explicit Hindsight Warning Prompt * Masked condition is strictly protected from post-cutoff information leakage. Original promptHindsight warningFullMasked Figure F.1: Accuracy and hindsight trap rate (HTR) across conditions for GPT under both prompt versions. Error bars show case- level 95% bootstrap confidence intervals. Across both models, explicit hindsight warning at- tenuated hindsight-biased responding but did not eliminate it.The reduction was most apparent in Conditions 2.1–3.2, where full-condition HTR, ∆HTR, and HBR were generally lower under the modified prompt. AIR was also reduced, particularly for GPT, but remained positive across the paired con- ditions, as did HBR, indicating persistent hindsight- sensitive answer changes despite the explicit warning. Accuracy remained broadly similar between prompt versions for both models, suggesting that the reduc- 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 Rate GPT — Sepsis Effect of Explicit Hindsight Warning * Masked condition is strictly protected from post-cutoff information leakage. Original promptHindsight warning ΔHTR AIRHBR Figure F.2: Hindsight bias metrics across paired con- ditions for GPT under both prompt ver- sions.Metrics include the change in hindsight trap rate (∆HTR), AIR, and HBR. Error bars show case-level 95% bootstrap confidence intervals. tions in hindsight-related metrics were not driven by a general deterioration in task performance. The con- sistency of this pattern across GLM and GPT indi- cates that models can respond to explicit instructions against hindsight bias, but that simple prompt-level mitigation is insufficient to eliminate the effect. 25 Hindsight Bias in Clinical Temporal Reasoning 0.850 0.875 0.900 0.925 0.950 0.975 1.000 Accuracy (A) GLM — Sepsis 01.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition 0.00 0.05 0.10 0.15 0.20 Hindsight Trap Rate (HTR) (B) Original vs. Explicit Hindsight Warning Prompt * Masked condition is strictly protected from post-cutoff information leakage. Original promptHindsight warningFullMasked Figure F.3: Accuracy and hindsight trap rate (HTR) across conditions for GLM under both prompt versions. Error bars show case- level 95% bootstrap confidence intervals. 1.11.2 2.1 * 2.2 * 3.1 * 3.2 * Condition −0.05 0.00 0.05 0.10 0.15 0.20 Rate GLM — Sepsis Effect of Explicit Hindsight Warning * Masked condition is strictly protected from post-cutoff information leakage. Original promptHindsight warning ΔHTR AIRHBR Figure F.4: Hindsight bias metrics across paired con- ditions for GLM under both prompt ver- sions.Metrics include the change in hindsight trap rate (∆HTR), AIR, and HBR. Error bars show case-level 95% bootstrap confidence intervals. 26 Hindsight Bias in Clinical Temporal Reasoning Appendix G. Human Validation This appendix provides additional details on the hu- man validation described in Section 3.8, including the error-enriched sampling procedure and validation re- sults. Annotators and review procedure. Four clini- cal annotators participated: one physician and three medical or physician assistant trainees. We sampled 10% of cases for review and divided them into two non-overlapping 5% subsets, each independently eval- uated by two annotators. For each question, annota- tors assessed (1) whether the question was clinically reasonable, (2) whether they agreed with the prospec- tive reference answer, and (3) whether they agreed with the hindsight-trap answer. All judgments were binary (yes/no). Error-enriched sampling. Rather than sampling uniformly, we preferentially selected cases in which the evaluated model exhibited errors or hindsight- sensitive behavior. For each case, we computed an error-proneness score (EPS) from four model-derived quantities: 1−Accuracy and HTR under Condition 0, and AIR and HBR under Condition 2.1. Each quan- tity was aggregated to the case level and min–max normalized within cohort: EPS i = 0.4 (1− Accuracy) i + 0.2 ] HTR i + 0.2 g AIR i + 0.2 ] HBR i . (G.1) Cases were sampled without replacement with weights w i ∝ exp EPS i τ , τ = 0.3. Thus, a 0.3-unit increase in EPS corresponds to an approximately e≈ 2.72-fold increase in relative sam- pling weight. This design provides a targeted stress test of benchmark annotations in cases most relevant to the observed model errors and hindsight effects. Human validation results. The validation sam- ple comprised 170 questions, each reviewed by two annotators.Agreement with the benchmark an- notations was high across question reasonableness, prospective reference answers, and hindsight-trap an- swers, with sample agreement ranging from 95.9% to 98.7% (Table G.1). Inter-annotator agreement was similarly high (91.8–99.2%). Because sampling favored cases with higher EPS, we additionally report inverse-probability-weighted estimates based on the cohort-specific sampling de- sign. These closely matched the corresponding sam- ple estimates (Table G.1) and should be interpreted as design-weighted point estimates; uncertainty inter- vals would require case-level resampling. 27 Hindsight Bias in Clinical Temporal Reasoning Table G.1: Human validation results. Sample estimates are calculated from the error-enriched validation set; weighted estimates use inverse-probability weights from the cohort-specific sampling design. AssessmentSample agreementApplicable assessmentsWeighted agreement Agreement with benchmark annotations Question clinically reasonable95.9%326/34095.8% Prospective reference answer98.7%235/23899.0% Hindsight-trap answer97.1%264/27296.5% Inter-annotator agreement Question clinically reasonable91.8%156/17091.6% Prospective reference answer99.2%118/11999.5% Hindsight-trap answer94.1%128/13693.0% Note. Sample agreement is calculated from individual annotator assessments. Inter-annotator agreement is the propor- tion of applicable questions for which both annotators assigned to the subset gave the same judgment. Not-applicable answer assessments are excluded from the corresponding denominators. 28