Paper deep dive
MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical Interaction
Ruoyu Wu, Shenfu Xie, Yinqian Sun, Haibo Tong, Feifei Zhao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Interactive clinical agents must gather decisive evidence and convert it into grounded actions under partial observability. A correct final diagnosis alone does not show that an agent respected evidence and care-process constraints. We introduce MediSkill-Evo, a clinical agent that evolves governed process knowledge without backbone fine-tuning. It separates experience into four typed banks for clinical skills, process rules, symbolic schemas, and measurement procedures. Provenance, support, replay, and controller-defined safety checks govern publication to a frozen test-time snapshot. A Process-Constrained Preference Harness binds evidence to its source, rejects controller-invalid candidates, and ranks actions with a safety-prioritized Clinical Process Critic. We evaluate complete agent systems across two backbone endpoints and six controlled stress dimensions under the same Doctor-turn limit. On 300 held-out Qwen encounters, MediSkill-Evo improves diagnosis accuracy from 61.33 percent to 69.00 percent and treatment-intent coverage from 33.62 percent to 66.44 percent, while reducing automatically scored critical failures from 31.00 percent to 16.33 percent relative to AgentClinic. On 180 hard-isolation conditions derived from 30 cases, target recovery reaches 93.61 percent under patient-behavior pressure, 100.00 percent for temporal evidence, and 92.22 percent for triage red flags. An exploratory 100-case MedSAM comparison evaluates request-gated tool-interface feasibility. These results provide descriptive end-to-end evidence for the complete system on fixed evaluation suites, not causal evidence for an individual bank or clinical validation of the automatic judge.
Tags
Links
- Source: https://arxiv.org/abs/2608.23397v1
- Canonical: https://arxiv.org/abs/2608.23397v1
Trouble viewing inline? Open PDF directly →
Full Text
129,661 characters extracted from source content.
Expand or collapse full text
MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical Interaction Ruoyu Wu 1,2,3,* Shenfu Xie 1,2,3,* Yinqian Sun 1,2,3,4 Haibo Tong 1,2,3,5 Feifei Zhao 1,2,3,5,† 1 Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences; 2 Beijing Key Laboratory of Safe AI and Superalignment; 3 Beijing Institute of AI Safety and Governance; 4 School of Artificial Intelligence, University of Chinese Academy of Sciences; 5 Long-term AI Beijing, China Abstract Interactive clinical agents must acquire decisive evidence and convert it into grounded actions under partial observability; a correct final label alone cannot certify that this process respected the benchmark’s evidence and care-process contracts. Existing experience memories typically place reusable strategies, process rules, evidence semantics, and visual procedures behind one retrieval interface, although these knowledge types demand dif- ferent scopes and safeguards. We introduceMediSkill-Evo, a clinical agent that evolves governed process knowledge without backbone fine-tuning. Four typed banks separately maintain Clin- ical Skills, Process Rules, Symbolic Schemas, and Measurement procedures, while provenance, support, replay, and controller- defined safety checks govern publication into a frozen test-time snapshot. A Process-Constrained Preference Harness binds ev- idence to its source, rejects controller-invalid candidates, and ranks actions through a safety-prioritized Clinical Process Critic. Frozen-suite evaluations across two backbone endpoints and six controlled stress dimensions compare complete agent systems under the same Doctor-turn ceiling; internal calls, tokens, and semantic question load are not matched. On 300 held-out Qwen encounters, MediSkill-Evo raises diagnosis accuracy from 61.33% to 69.00% and reference treatment-intent coverage from 33.62% to 66.44%, while reducing automatically scored critical failures from 31.00% to 16.33% relative to AgentClinic. On 180 hard-isolation conditions derived from 30 cases, its target recovery is 93.61% under patient-behavior pressure, 100.00% for temporal evidence, and 92.22% for triage red flags; three dimension-specific auto- matic failure indicators are zero for all compared systems and therefore are not evidence of unique superiority. An exploratory 100-case MedSAM comparison is reported only as request-gated tool-interface feasibility. These results are descriptive end-to-end evidence for the complete system on the evaluated fixed suites, not causal evidence for an individual bank or clinical validation of the automatic judge. CCS Concepts •Computing methodologies→Multi-agent systems; •Infor- mation systems→Information retrieval. Keywords clinical agents, self-evolving memory, process-constrained reason- ing, multimodal tool interface 1 Introduction A clinically useful agent must do more than name a disease. Under partial observability, it must elicit decisive history, request and in- terpret examinations, update its assessment, and recommend treat- ment under safety and urgency constraints. An unavailable test is not a negative finding, and a correct label can still conceal an un- supported or unsafe trajectory. The central problem is therefore how an agent can improve from experience while preserving the evidence boundaries and care obligations that make each action trustworthy. Figure3shows this problem in text-only and multi- modal encounters. Medical agents increasingly support interactive clinical work. AgentClinic [14] models partially observable doctor–patient– measurement encounters; MDAgents [5] adapts collaboration to case complexity; EHRAgent [15] and MMedAgent [6] connect models to executable EHR code and multimodal tools; Reflec- Tool [7] verifies tool use from experience; and MEDDxAgent [12] coordinates specialized modules for interactive differential diagno- sis. AI Hospital measures symptom collection, examination choice, and diagnosis in multi-turn simulation [ 26], while 3MDBench stud- ies multimodal telemedical dialogue [27]. Together, they establish interaction, specialization, tool use, and workflow-level evaluation as an emerging baseline rather than a contribution unique to this paper. Our narrower question is how trajectory-derived knowl- edge with different epistemic roles can be published and exercised through type-dependent validation and decision authority. Self-evolving agents provide a parameter-efficient route to this goal. Reflexion [16] stores verbal feedback and ExPeL [24] con- solidates cross-trial insights; Voyager [ 19], ICAL [13], and Agent Workflow Memory [20] distill executable skills, multimodal ab- stractions, or workflows. MemP [3] builds updateable procedural instructions, SkillWeaver [25] discovers reusable skills through practice, and MemBench separates factual and reflective memory while evaluating effectiveness, efficiency, and capacity [28]. These methods show that completed trajectories can become reusable ex- ternal knowledge without backbone updates. MediSkill-Evo does not claim the first structured memory or workflow evaluation; it proposes a particular complete-system interface in which arti- fact type controls validation, retrieval scope, and benchmark-time authority. Clinical interaction motivates preventing inferred, miss- ing, or tool-derived information from silently becoming observed fact. * Ruoyu Wu and Shenfu Xie contributed equally to this work. † Feifei Zhao is the corresponding author. 1 arXiv:2608.23397v1 [cs.AI] 24 Aug 2026 WSDM ’27, February 15–19, 2027, Hong KongWu et al. Figure 1: Online inference and offline learning and governance in MediSkill-Evo. Left: the online pipeline retrieves four-bank context, generates and verifies candidate actions, executes the selected action, and incorporates environment feedback. Right: completed trajectories are reflected into typed memory operations, validated for label/evidence leakage, controller-defined safety, support, and replayability, and published as the next memory snapshot. Decision-critical Harness prompt (abridged) SYSTEM You are a strict Clinical Process Reward Model critic. Score candidate Doctor actions using only the visible dialogue state and general clinical safety. Diagnostic specificity and evidence alignment matter more than sounding complete. Do not use hidden gold diagnoses. Return strict JSON. RUNTIME INPUT state: <visible evidence, process ledger, requested tests, turn constraints> candidates: <structured next-action candidates> DECISION RULES 1. Prefer the action that resolves the most important remaining diagnostic or safety gap. 2. A test request contains one atomic item; penalize broad, packed, repeated, or low- value testing. 3. Never treat missing orRESULTS_UNAVAILABLEevidence as normal, negative, or reas- suring. 4. Strongly penalize missed dangerous diagnoses, delayed escalation, unsafe reassur- ance, and unmet treatment prerequisites. 5. On the final turn, onlyDIAGNOSIS_READYis valid; score diagnostic support, manage- ment, safety checks, and follow-up. OUTPUT ``scores′:[ ``candidate_index′:푖, ``process_score′:⋅, ``diagnostic_specificity_score′:⋅, ``evidence_alignment_score′:⋅, ``treatment_completeness_score′:⋅, ``safety_score′:⋅, ``triage_escalation_score′:⋅, ``test_efficiency_score′:⋅, ``constraint_violations′:[...], ``valid′:true|false ] Figure 2: The decision-critical Clinical Process Critic tem- plate. Runtime slots are populated at each turn; retry-only instructions are omitted. Agent harnesses determine whether typed knowledge actually changes behavior. ReAct [23] interleaves reasoning and envi- ronment actions; AgentBench [ 8] and AgentBoard [10] evaluate multi-step progress; AppWorld [18], T-Eval [1], ToolSandbox [9], and휏-bench [ 22] expose executable state transitions, tool policies, and interaction reliability. In medicine, the harness must addition- ally bind every result to a valid request, keep unavailable evidence unknown, preserve registered process obligations, and apply benchmark-defined safety checks without access to the hidden diagnosis. The proposed interface turns evolving knowledge into controller-valid, evidence-grounded action; it is not a claim of independent clinical legality or safety. We introduceMediSkill-Evo, which organizes evolving expe- rience around the clinical process. Completed trajectories propose updates to four typed banks: Clinical Skill stores case-level strate- gies, Process Rule stores workflow constraints, Symbolic Schema defines evidence provenance and controller-valid transitions, and Measurement stores visual procedures. Provenance, replay, sup- port, and safety checks govern publication into the next frozen snapshot. During an encounter, a Process-Constrained Preference Harness retrieves state-relevant objects, verifies alternative actions against symbolic and process constraints, and selects through a safety-prioritized Clinical Process Critic. The key insight is that memory becomes dependable when its type determines not only what is retrieved, but also how it is validated and how much au- thority it receives at decision time. We evaluate this complete design at four levels. On 300 held- out FullChain encounters with Qwen3.6-Flash, MediSkill-Evo differs from AgentClinic by+7.67diagnosis points,+32.82refer- ence treatment-intent coverage points, and−14.67automatically scored critical-failure points; DeepSeek-V4-Flash shows the same direction on evidence, treatment-intent, safety, and gated inter- action metrics. Across 180 controlled hard-isolation conditions, the primary dimension metrics show+17.22points over the strongest baseline in patient-behavior target recovery,+27.22in 2 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong temporal-evidence recovery, and+34.44in triage-red-flag recov- ery. Conventional diagnosis and treatment scores are retained as auxiliary outcomes rather than the endpoint of this stress evaluation. An exploratory MedSAM-enabled condition is 3.24 points higher on the multimodal core score. Component results are hypothesis-generating observations about internal tradeoffs rather than bank- or stage-specific causal estimates. Our contributions are threefold: •We propose MediSkill-Evo as a complete agent sys- tem combining four-bank self-evolution and a Process- Constrained Preference Harness for evidence-grounded, safety-constrained clinical interaction. •We introduce FullChain for history-to-prescription work- flows, a controlled hard-isolation stress benchmark for care-process failures, and an exploratory request-gated NEJM visual-tool interface suite. •Under the same Doctor-turn ceiling, complete-system com- parisons across two backbone endpoints and three frozen benchmarks report diagnosis, evidence-acquisition, process, and automatically scored safety outcomes; internal compute is not matched, and component results do not isolate indi- vidual mechanisms. 2 Method 2.1 Overall Architecture MediSkill-Evo supports partially observable, interactive clinical de- cision making while continually revising external knowledge from completed training trajectories rather than updating backbone pa- rameters. At turn푡, the visible state contains only the presented case information, current observation, dialogue history, and ex- aminations that have been requested and returned; reference diag- noses, evaluator labels, and unrevealed results remain inaccessible. Given the four-bank snapshotℬ 푒 published after evolution stage 푒, the system retrieves relevant, provenance-traceable knowledge. The Doctor uses the resulting augmented state푠 푡 to ask a history question, request an examination, or submit a clinical plan, and new evidence enters the next state: 푣 푡 = (푥 vis , 표 푡 , ℎ 푡 , 푧 푡 ),푢 푡 =ℛ(푣 푡 ;ℬ 푒 ),푠 푡 = (푣 푡 , 푢 푡 ).(1) This formulation separates case-visible evidence from external knowledge and prevents retrieval from introducing hidden labels for the current case. Figure1summarizes the online inference and offline gover- nance workflows of MediSkill-Evo. MediSkill-Evo consists of a four-bank self-evolution layer and aProcess-Constrained Preference Harness. After each training case, the former routes trajectory- derived experience to the Clinical Skill, Process Rule, Symbolic Schema, and Measurement banks, which respectively represent clinical strategies, cross-case workflow rules, evidence bound- aries, and visual measurement procedures; provenance, safety, and replay checks govern publication of the next frozen snap- shot. Within an encounter, the Harness retrieves state-relevant objects from these banks. It directly executes a rule-mandated process checkpoint when one is triggered; otherwise, it generates structured candidates and selects an executable action through symbolic verification and Clinical Process Critic comparison. Here, preferencedenotes test-time ranking of candidates generated for the same state, not parameter-level preference learning or rein- forcement learning. Encounter traces support only later evolution stages and never revise a case in progress. 2.2 Four-Bank Self-Evolution Four-bank self-evolution converts completed case trajectories into reusable, auditable external knowledge. For training case 푖,휏 푖 = (푣 푡 , 푢 푡 , 푎 푡 , 표 푡+1 푇 푡=0 , 푦 푖 )records visible states, retrieved knowledge, executed actions, environment responses, and the post-encounter evaluation, separating what the system observed, retrieved, and executed. Only after termination does the reflector generate update proposals, so evolution cannot alter the case in progress. Each bank manages typed artifacts comprising content, applicability scope, trajectory provenance, and lifecycle status. Active artifacts are not overwritten in place; add, merge, patch, deprecate, or discard operations are proposed for the next snap- shot. The banks share this merge, validation, and publication protocol but retain distinct knowledge boundaries and checks. This separation lets each bank evolve independently without mix- ing clinical strategies, workflow constraints, evidence semantics, and visual procedures. The four types separate content from decision authority. Clin- ical Skills encode scoped diagnostic and management strategies; Process Rules encode cross-disease required or prohibited bench- mark actions; Symbolic Schemas define controller-valid evidence sources and state transitions; and Measurement Skills encode image-specific observation procedures without returning a diag- nosis. In the reported offline system, “required” means enforced relative to the registered benchmark contract, not endorsed by an external guideline or clinician. Deterministic evidence semantics and registered safety prerequisites outrank trajectory-derived rules, which cannot create facts or override those constraints. For deployment, a learned regularity would remain advisory unless an identified guideline or expert policy supplied its authority and independent validation justified hard enforcement. The sup- plementary material specifies artifact fields, provenance, conflict resolution, and lifecycle operations. Let푏 ∈ 퐶, 푃, 푆, 푀index the four banks,Δ 푏 1∶푁 denote proposals from the ordered training cases, and푈 푏 and푉 푏 be the typed merge and validation operators. The next snapshot is ℬ 푏 푒+1 = 푉 푏 ( 푈 푏 (ℬ 푏 푒 , Δ 푏 1∶푁 ), 휏 푖 푁 푖=1 ) .(2) The merge operator organizes proposals by artifact identity, se- mantic overlap, and applicability scope, removing duplicates while preserving revision provenance. The validator checks type con- sistency, provenance, label/evidence leakage, controller-defined safety, and replayability on the proposal-generating trajectories; it does not estimate generalization to unseen cases or confer clin- ical authority. Valid artifacts enter the next immutable snapshot, whereas insufficient, conflicting, or out-of-bound artifacts are withheld, disabled, or rejected. This ties each inference trace to a determinate knowledge version. At test time, the final snapshot is frozen and reflection and knowledge writes are disabled. 3 WSDM ’27, February 15–19, 2027, Hong KongWu et al. 2.3 Process-Constrained Preference Harness At each turn, the Harness constructs푣 푡 from the presented task and acquired evidence and registers Patient responses, Doctor actions, and Measurement outputs as provenance-bearing facts. Every ex- amination result is bound to the normalized request that elicited it: an available result is returned only after the request, whereas an absent result is markedRESULTS_UNAVAILABLEand remains un- known. Clinical Skills then pass retrieval and semantic gating, Pro- cess Rules form the dynamic ledger, Symbolic Schemas expose evi- dence boundaries, and Measurement Skills guide visual inspection after an image request. Their outputs constitute푢 푡 , with external knowledge and case evidence recorded separately. Before generating candidates, the Harness checks whether an active Process Rule mandates a deterministic process action. A trig- gered action is executed with its rule and evidence recorded; other- wise, the Doctor generates structured candidates containing an ac- tion, target, rationale, expected information value, supporting ev- idence, and safety risks. In non-final turns, each examination can- didate requests one atomic item to permit direct comparison of in- formation value. Final-turn candidates instead provide all required diagnosis, evidence, management, safety, and follow-up fields. The Symbolic Verifier first removes candidates that use un- available or controller-invalid evidence, omit required final fields, or violate registered treatment-safety prerequisites. The Clini- cal Process Critic scores the remaining candidates for process quality, diagnostic specificity, evidence alignment, treatment com- pleteness, safety, triage, and examination efficiency. Step-level selection emphasizes process advancement and information effi- ciency, whereas final selection emphasizes diagnosis, evidence, and treatment completeness. With stage푟 ∈ step,final, hard- constraint indicator퐻, critic dimensions풟 푟 , and soft-constraint set풫 푟 , selection is unified as 푐 ∗ 푡 =arg max 푐 푖 푡 ∶ 퐻(푠 푡 ,푐 푖 푡 )=1 [ ∑ 푑∈풟 푟 휆 푟 푑 푝 푖 푑,푡 − ∑ 푘∈풫 푟 휆 푟 pen,푘 푞 푖 푡,푘 ] ,푎 ∗ 푡 = 푎(푐 ∗ 푡 ). (3) The first term is the stage-specific process score, and the second penalizes repeated examinations, inefficiency, and repairable struc- tural defects. Hard-invalid candidates cannot re-enter through fi- nite penalties. If none meets the safety threshold, bounded regener- ation proceeds without relaxing hard constraints; persistent failure yields safe termination and human escalation. Candidates, verifier outputs, scores, and selections remain in the audit trace. Figure 2reproduces the decision-critical portion of the Clini- cal Process Critic prompt. We expose this prompt because it de- fines the evidence boundary, the safety priorities, and the typed scoring interface that operationalize preference selection. Runtime state and candidates replace the bracketed slots; omitted instruc- tions concern only schema recovery and serialization. The supple- mentary material provides the learning, measurement, candidate- generation, preference, and final-safety templates needed to repro- duce the complete prompt-driven path. The selectedASK,REQUEST_TEST, orDIAGNOSIS_READYaction is sent to the Patient, Measurement, or final-response component, and the return updates the evidence state. A final response under- goes schema validation, diagnosis-blind safety review, and a sep- arate risk-auditor call that checks diagnostic support, dangerous alternatives, treatment contraindications, and management inten- sity. A Final Rewriter incorporates required corrections, and a Re- lease Certifier performs the final check. This bounded rewrite pro- cess withholds an uncertified plan at the retry limit and returns safe termination and human escalation instead. 3 Experiments 3.1 Implementation Details We evaluate MediSkill-Evo with two hosted backbone endpoints, Qwen3.6-Flash [21] and DeepSeek-V4-Flash [2], from the Qwen and DeepSeek model families. Within each comparison, the Doctor and moderator use the same backbone, while the Patient and Mea- surement environments, evaluation code, case order, and Doctor- turn ceiling are held fixed; each Doctor-side system retains its na- tive experience-reuse and control procedure. This ceiling does not match internal calls, tokens, words, atomic question load, latency, cost, or patient burden. Registered evaluation calls use temperature zero and one observed rollout for every case–configuration pair; provider-side nondeterminism is therefore outside the finite-suite estimand. The Process-Constrained Preference Harness generates three candidates at each non-deterministic decision point and uses the safety-priority gate. Text-only FullChain and stress evaluations permit at most six Doctor inferences per case. Their Clinical Skill, Process Rule, and Symbolic Schema snapshots are frozen through- out evaluation, while the Measurement Bank is inactive because no medical image is available. The exploratory multimodal evaluation uses Qwen3.6-Flash, three candidates, and at most eight Doctor inferences per case. We use 20 shards for both training and testing. The three Doctor banks are initialized from the same previously learned snapshot and remain frozen. The paired conditions share the backbone, cases, interaction environment, and Doctor-turn ceiling, but not internal compute; no token- or cost-normalized efficiency claim is made. Following Section 2.2, proposed procedures are checked against their completed training trajectories before publication, and the condition-specific Measurement Bank is frozen for the 100-case test. The MedSAM [11] condition uses the ViT-B check- point. The paired no-MedSAM and MedSAM runs share the model, datasets, split hashes, Doctor-bank hashes, candidate count, and inference budget; their manifest-verified tool intervention is whether MedSAM may be called by the Measurement Agent, while each condition learns its corresponding Measurement Bank. Because masks are not anatomically adjudicated and the banks differ, this is a request-gated interface case study, not evidence that segmentation improves clinical reasoning. The experiments characterize four aspects of the submitted complete system: end-to-end behavior at two backbone endpoints; constraint-following under stress; a modular visual-tool interface; and hypothesis-generating component tradeoffs. Metric defini- tions appear in Section 3.3, with complete evaluator prompts in the supplementary material. Manifests bind model aliases, registered configurations, dataset splits, bank and registry hashes, shard assignments, and code revisions; all main tables report case-level 4 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong (a) Text-only FullChain interaction(b) Multimodal NEJM interaction INITIAL 24-year-old womanwith sudden right-upper-quadrant pain after a fatty meal, nausea, and vomiting. DOCTOR When did the pain start, how severe is it, and does it radiate to your back or shoulder? PATIENT Constant for 10 hours, 8/10, radiating to the right shoulder blade; worse after eating. DOCTOR REQUEST TEST: Complete_Blood_Count MEASUREMENT WBC 11.7×10 3 /μL with neutrophil predominance. DOCTOR REQUEST TEST: Right_Upper_Quadrant_Ultrasound MEASUREMENT RESULTS_UNAVAILABLE; the result remains unknown, not negative. DOCTOR Is there a sharp catch in your breath with right-upper-quadrant pressure? PATIENT Yes—I flinch and cannot take a full breath. FINAL Acute cholecystitis. NPO, IV fluids, antibiotics after prerequisite checks, urgent surgical evaluation, and alternative imaging because ultra- sound is unavailable. INITIAL 95-year-old womanwith four weeks of dyspnea and dry cough; resting SpO 2 is 83%. DOCTOR How long have you taken nitrofurantoin? PATIENT Daily for six months. DOCTOR: REQUEST TEST: NEJM_Medical_Image MEASUREMENT AGENT: optional MedSAM Raw radiograph Raw chest CT X-ray overlay CT overlay MEASUREMENT Bilateral opacities on radiography; CT shows ground- glass opacity, septal thickening, crazy paving, and consolidation. MedSAM localizes these regions; the agent re-inspects the original pixels. DOCTOR REQUEST TEST: Complete_Blood_Count MEASUREMENT Neutrophilic leukocytosis; no eosinophilia. FINAL Nitrofurantoin-induced pulmonary toxicity. Stop nitrofurantoin, provide oxygen and monitored admission, exclude infection, and obtain pul- monary consultation. Figure 3: Two frozen-test interactions scored diagnosis-correct by the automatic evaluator. Left: targeted history, available lab- oratory evidence, an unavailable ultrasound, and a focused physical examination support acute cholecystitis without treating a missing result as negative. Right: medication history, original radiograph and CT evidence, optional MedSAM localization, and a follow-up blood count support nitrofurantoin-induced pulmonary toxicity; this label does not validate anatomy or man- agement. Table 1: Evaluation datasets. FullChain encounters derive from MIMIC-IV [4] and use the AgentClinic interaction pro- tocol [14]; multimodal cases derive from the NEJM image collection [17]. Stress counts denote controlled conditions; the 420/180 conditions are derived from 70/30 disjoint un- derlying clinical cases. DatasetTrain Test Modality MIMIC-IV FullChain700 300 Text Controlled clinical stress420 180 Text NEJM FullChain Interactive200 100 Text + image point estimates on the fixed suites. We do not claim population confidence intervals, hosted-service repeatability, computational efficiency, or single-component causal effects. 3.2 Datasets Table1summarizes the three evaluation settings and their fixed train–test splits. MIMIC-IV FullChain encounters.We construct interactive clin- ical encounters from MIMIC-IV-derived records [4]. Each record preserves its source identifiers and contributing structured ta- bles, including admissions, diagnoses, demographics, laboratory and microbiology events, prescriptions, pharmacy records, and procedures. These fields are transformed into an AgentClinic- compatible OSCE case [ 14] containing a Doctor objective, patient profile, physical findings, requestable examinations, a reference diagnosis, management targets when available, safety constraints, and evaluator-only targets. The resulting split contains 700 train- ing encounters and 300 held-out encounters; a direct identifier audit finds zero shared subjects and zero shared admissions across the split. During interaction, the Doctor sees only the initial objec- tive, accumulated dialogue, and results returned after its requests; reference diagnoses and scoring targets remain hidden. Controlled clinical stress benchmark.The controlled stress benchmark asks whether an agent preserves process correctness and safety when a specific care obligation becomes difficult, not merely whether it still predicts the reference diagnosis. We construct six source-grounded variants of each FullChain case: diagnosis difficulty, evidence completeness, patient behavior com- plexity, treatment and prescription safety, temporal dynamics, and triage safety. The training benchmark contains six variants for each of 70 cases (420 conditions). The held-out Stress V2 bench- mark applies all six dimensions to 30 disjoint source cases (180 conditions) and uses field-level hard isolation: a deterministic con- troller, rather than an LLM prompt, owns delayed or permanently unavailable values and releases an exact source value only after a permitted question, examination, or test request. The Doctor and its retrieval and decision modules never receive the dimen- sion, subtype, fact identifiers, trigger concepts, hidden values, or evaluator targets. Stress V2 changes only the visibility and timing of source- supported facts. It delays discriminating evidence, makes one fact permanently unavailable while preserving independent solv- ability, requires focused recovery of fragmented patient facts or treatment prerequisites, delays source-supported timeline facts, or withholds a real urgent red flag until an appropriate screen. It does not invent distractor symptoms, refusals, worsening events, contraindications, or red flags. The 180 test observations therefore represent six paired conditions on 30 underlying cases, not 180 5 WSDM ’27, February 15–19, 2027, Hong KongWu et al. Table 2: Held-out Stress V2 dimensions and their enforced process obligations. Hidden values remain outside every runtime LLM prompt and are released only by the determin- istic controller. DimensionPrimary capability under evaluation Diagnosis difficultyRecovering decisive source evidence through the correct action chan- nel, using released evidence, and avoiding premature closure. Evidence completeness Requesting the missing item, treating unavailability as unknown, us- ing alternative evidence, and avoiding a fabricated result. Patient behavior com- plexity Recovering patient facts through focused, respectful questions with- out changing the underlying disease facts. Treatment and prescrip- tion Verifying source-supported prerequisites and choosing conditional treatment, safe deferral, or an alternative rather than an unsafe ac- tion. Temporal dynamicsRecovering the original timeline and integrating it into reassessment, treatment, monitoring, or escalation. Triage safetyRecovering a real red flag, escalating appropriately, and avoiding un- safe reassurance. independent patients. Construction checks enforce disjoint under- lying cases, complete six-dimension coverage, unique condition identifiers, target reachability through the correct action channel, and absence of secret values from every runtime model prompt. Table2summarizes the exact held-out subtypes and the process obligation tested by each dimension. NEJM FullChain Interactive.We build the multimodal bench- mark from a frozen pool of 300 NEJM image cases [17]. First, we standardize every item as a diagnosis task: 231 source questions already ask for a diagnosis, while an LLM rewrites 69 questions about causes, findings, treatments, organisms, or measurements into source-grounded diagnosis questions. A second LLM-driven conversion then decomposes each case into patient-knowable history, bedside examination findings, canonical requestable tests, required history questions, and required tests. The converter is prohibited from inventing negative history, normal findings, vital signs, laboratory values, treatments, or outcomes, and it cannot expose the reference diagnosis in any Doctor-visible field. A verifier audits the decomposition for unsupported facts, answer leakage, misplaced evidence, and invalid test names before the final validation pass. The image is deterministically registered as the requestable NEJM_Medical_Imageexamination, while other laboratory, imag- ing, pathology, and physical evidence retain canonical test names. The visible image-request name specifies an interaction affor- dance, not its content, finding, or diagnosis. All 300 records follow the same request-gated interaction contract: only available exam- ination names and initial bedside findings are visible at the start, and an unobserved or unavailable result cannot be interpreted as negative. We use the fixed manifest to assign 200 cases to training and 100 to a frozen test split. Dataset and registry hashes bind the generated files to that split; all 300 cases pass final validation, contain a registered image request, and preserve hidden-label separation. 3.3 Evaluator and Metrics The offline evaluator scores clinical outcomes and observable pro- cess quality from the completed Doctor–Patient–Measurement trajectory and evaluator-only targets after inference. It receives no method identity, and the same evaluator path is applied to every comparator. The semantic judge uses the registered moderator alias—the comparison backbone within each FullChain block—at temperature zero. Registered test names, Stress V2 release events, and output-schema validity are checked deterministically, while the judge handles semantic equivalence and must cite support- ing trajectory turns; malformed or unsupported outputs receive no credit. In the 300-case FullChain test set, required-history, required-test, and management target lists are nonempty for every case. In the NEJM test set, all cases have an image-test target and seven cases have an empty required-history list; macro recall uses the same defined-target convention for both paired conditions. The stress evaluator deliberately separates process measure- ment from conventional task outcomes. Controller events deter- mine whether a target fact was recovered; a dimension-specific semantic check then asks only whether released evidence was used, unavailable evidence remained unknown, a treatment pre- requisite led to safe action or deferral, a timeline was integrated, or a red flag triggered escalation. These dimension metrics are the primary Stress V2 endpoints. Diagnosis, treatment, the regis- tered stress composite, general safety and critical-failure labels, and interaction economy are retained as auxiliary system out- comes so that process gains cannot conceal a collapse in ordinary clinical performance. Table 3summarizes these measures; all rates are macro-averaged percentages over their declared eligi- ble sets, and the supplementary material provides the complete prompts, denominators, and aggregation rules. Treatment-intent, safety, critical-failure, and semantic stress labels are operational outputs of this automatic evaluator, not independently clinician- adjudicated clinical outcomes; they support method-blind within- benchmark comparison but do not establish construct calibration, clinical certification, or prospective validity. 3.4 Cross-Backbone FullChain Results We compare AgentClinic [14], the original Doctor without an experience bank, with our structured Agent-KB implementation, ExPeL [ 24], MemP [3], Reflexion [16], SkillWeaver [25], and MediSkill-Evo on the same 300 held-out encounters. Table4 groups the two backbone-specific evaluations. Every method uses the same AgentClinic interaction environment, backbone, case order, Doctor-turn ceiling, and frozen test-time memory. This is an end-to-end system comparison: it preserves native memory/- control flow and neither isolates memory representation from its Harness nor normalizes internal calls, tokens, or monetary cost. With Qwen3.6-Flash, MediSkill-Evo has the strongest joint au- tomatic benchmark outcome: diagnosis accuracy reaches 69.00%, 7.67 points above AgentClinic, while reference treatment-intent coverage rises from 33.62% to 66.44%. The same run raises evidence recall by 36.12 points and required-history recall by 86.74 points, and reduces automatically scored critical failures from 31.00% to 16.33%. The supplementary material reports per-case better/tie/- worse transitions from these existing paired traces without resam- pling or re-judging. This alignment shows that the accepted final di- agnosis co-occurs with a more complete observable trajectory un- der the benchmark definitions rather than label prediction alone. 6 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong DeepSeek-V4-Flash shows the same process-metric pattern at the second tested endpoint. MediSkill-Evo leads seven of the eight non-diagnosis metrics: treatment-intent coverage reaches 47.30%, evidence recall 40.11%, and history recall 73.65%; automatically scored safety violations fall from 20.33% to 3.33%, critical failures from 52.00% to 33.00%, and unnecessary examinations from 45.04% to 18.33% relative to AgentClinic. Gated interaction efficiency con- sequently rises from 7.58% to 27.56%. These two endpoints support cross-backbone consistency within the evaluated families, not a claim of model-family-wide generalization. Table 3: Metrics used in our evaluation. The upper block cov- ers standard and multimodal cases; the lower block covers controlled clinical stress. Arrows indicate the preferred di- rection. GroupMetrics and meaning Standard clinical and multimodal evaluation OutcomeDx↑: accepted final diagnosis;Tx/Rx↑: reference treat- ment intents covered, with unsafe care penalized. EvidenceHist.↑: required history elicited;Tests↑: required ex- aminations requested; Evid. ↑ : joint history-and-test coverage. SafetySafety↓: observable safety violation;Critical↓: criti- cal diagnostic, treatment, or triage failure. EconomyUnnec.↓: unjustified examinations;Int.Eff.↑: gated interaction efficiency, not compute efficiency;Core↑: composite automatic score penalized for critical failure. MedSAMMasks: cases with at least one non-empty MedSAM mask. Controlled clinical stress evaluation DiagnosisDTR: delayed target recovered;DUR: recovered dis- criminator used;PCR: premature closure. EvidenceHqR: unavailable history queried;KTR: unavailable test requested;UHS: unavailability handled safely; HUR: unavailable result hallucinated. BehaviorBTR: target patient facts recovered;FQS: focused- question adequacy;BFR: target-recovery failure. TreatmentTSR: treatment-safety prerequisites recovered;SDF: safe deferral or alternative;UAR: unsafe action. TemporalTER: timeline evidence recovered;TIR: recovered timeline integrated;TFR: integration failure. TriageRFR: red-flag evidence recovered;EA: escalation ade- quacy;URR: unsafe reassurance. 3.5 Process Correctness and Safety under Controlled Stress Stress V2 asks a process question: after decisive evidence is de- layed, unavailable, fragmented, or safety-critical, does the agent recover what can be observed and respond without inventing ev- idence or taking an unsafe action? We therefore organize the ex- periment around the controller-grounded process and safety end- points in Fig. 4. Its left panel summarizes dimension-aware pro- cess completion, while its right panel exposes the concrete recov- ery, use, and failure submetrics behind each dimension. Table 5re- tains diagnosis, treatment, and other conventional system metrics as an auxiliary check, not as the organizing claim. AgentClinic [ 14], MemP [3], Reflexion [16], and MediSkill-Evo are evaluated on the same 180 conditions with the same controller, evaluator, case or- der, and six-action budget. All saved predictions are retained; no method is filtered after interaction. The stress profile is not uniformly favorable, but it identifies where the complete system changes the care process. In Fig.4(b), MediSkill-Evo recovers 93.61% of target patient facts, 17.22 points above the strongest baseline; it recovers all source-supported time- line targets, a 27.22-point margin; and it recovers 92.22% of delayed triage red flags, a 34.44-point margin. Every recovered timeline target is integrated, and all evaluated triage plans provide ade- quate escalation without unsafe reassurance. Under unavailable evidence, MediSkill-Evo queries every hidden-history subtype and never fabricates an unavailable result; however, it requests only 30.00% of unavailable-test subtypes, below all comparators. Diagnosis-pressure recovery is likewise 68.89%, below Reflexion’s 77.22%, even though recovered discriminators are always used and no method is scored for premature closure. Treatment-prerequisite recovery remains low for every system (33.33–36.67%); all four nevertheless use a safe alternative or explicit deferral and avoid a dimension-scored unsafe action. The left panel summarizes these outcomes without letting any single submetric stand in for a dimension; the detailed heatmap shows that ceiling-valued condi- tional behavior does not erase the fixed-denominator acquisition bottlenecks. The auxiliary results verify that this process emphasis does not come from discarding ordinary task performance. MediSkill-Evo has the highest registered stress-process score (83.79%), required- action recall (88.30%), treatment-intent coverage (82.40%), and core score (80.03%); its diagnosis accuracy is 93.89%, 2.78 points below AgentClinic. General safety violations and critical failures are 1.11% and 2.78%, respectively, compared with AgentClinic’s 0.00% and 1.11%. Thus, the experiment supports stronger process completion in the patient-behavior, temporal, and triage dimen- sions, not uniform dominance in either diagnosis or safety. The patterns are descriptive evidence for the complete system on 30 paired source cases and do not causally identify an individual bank or Harness stage. 3.6 Modular Visual Measurement on NEJM Cases We evaluate MemP, Reflexion, and two MediSkill-Evo tool condi- tions on the frozen 100-case NEJM [17] test split. The MediSkill- Evo conditions freeze the three Doctor banks and evolve a condition-specific Measurement Bank on the corresponding 200 training cases. One condition analyzes original images without segmentation; the other permits the Measurement Agent to invoke MedSAM [ 11] and use masks and overlays as localization aids. Table6separates unconditional case coverage from metrics that can be computed only after a diagnosis-ready output. 7 WSDM ’27, February 15–19, 2027, Hong KongWu et al. Table 6: Four-agent results on 100 multimodal NEJM image cases [17]. Ready and Masks are counts; all reported Dx and Core values use all 100 cases. The baseline dimension export does not report Hist., Tests, or Unnec.; dashes prevent mix- ing those runs with earlier conditional summaries. Other values are percentages. Agent/conditionReady Dx↑Hist.↑Tests↑Unnec.↓Core↑Masks MemP100/100 38.00– 35.95 0/100 Reflexion100/100 39.00– 40.29 0/100 MediSkill-Evo (raw)100/10037.0075.58100.005.0044.690/100 MediSkill-Evo (+MedSAM)100/10040.0079.50100.006.5847.9334/100 All four methods produce completed outputs for the 100-case split. In the updated dimension summary, MemP and Reflexion reach diagnosis accuracies of 38.00% and 39.00% and core scores of 35.95% and 40.29%, respectively. The export does not contain their history, test, or unnecessary-test aggregates, so Table6 leaves those cells unreported. Within the paired MediSkill-Evo comparison, the exploratory MedSAM-enabled condition is 3.00 points higher in diagnosis accuracy, 3.92 points higher in required- history recall, and 3.24 points higher in core-case score than the original-image condition. The Measurement Agent produces 54 non-empty masks in 34 cases and analyzes the remaining cases directly from original pixels. This selective use illustrates the intended interface: MedSAM can contribute localization when a valid region is produced, while the reporting contract preserves an original-pixel path for every case. Because the Measurement Banks are condition-specific and the three-case diagnosis dif- ference has no repeatability or localization validation, this is feasibility evidence for a tool-bearing system condition, not a MedSAM improvement or segmentation-only causal estimate. 3.7 Component Analysis We rerun the no-memory reference, three bank removals, and the full configuration on the same fixed 100-case subset of held-out FullChain encounters. Every row contains 100 diagnosis-ready outputs and zero error rows. Table7is a matched, hypothesis- generating comparison of observed system behaviors; one frozen run per profile does not establish that a bank is necessary or causally beneficial. Table 7: Automatic component diagnostics on the same 100 held-out FullChain cases. All profiles use the same frozen model endpoint, evaluator, case set, and inference budget; the switches remove the indicated Clinical, Process, or Sym- bolic bank. Values are percentages from one run per pro- file, so differences are descriptive rather than repeatability- adjusted causal estimates. VariantDx↑Tx/Rx↑Safe.↓Crit.↓Unnec.↓ No memory 67.00 36.50 1.00 30.0011.50 w/o Clinical 74.00 63.65 1.00 24.00 29.75 w/o Process 71.00 60.70 8.00 36.00 30.50 w/o Symbolic 70.00 68.900.0015.00 19.50 Full76.0069.400.0013.0019.17 No profile dominates every endpoint. The full system has the highest diagnosis accuracy (76.00%) and treatment-intent coverage (69.40%), the lowest critical-failure rate (13.00%), and a tied-lowest safety-violation rate (0.00%). Conversely, no memory has the low- est unnecessary-test rate (11.50%) but also the lowest treatment coverage (36.50%) and the highest critical-failure rate (30.00%). Among the bank removals, removing the Clinical bank yields the highest diagnosis accuracy (74.00%), whereas removing the Symbolic bank yields the highest treatment coverage (68.90%), the Table 4: Main automatic operational results (%) on the same 300 held-out FullChain encounters. Backbones represent the Qwen [21] and DeepSeek [2] families; baseline methods are cited in the accompanying text. Higher is better except for Safety, Critical, and Unnec. Bold marks the best result within each backbone; shaded rows denote MediSkill-Evo. BackboneMethod Benchmark outcome Evidence acquisitionRisk and economy Int.Eff.↑ Dx↑Tx/Rx↑Evid.↑Hist.↑Tests↑Safety↓Critical↓Unnec.↓ Qwen3.6-Flash AgentClinic61.3333.62 12.31 11.70 13.461.6731.0015.89 39.60 Agent-KB61.0038.43 14.52 11.26 17.450.6732.3313.8241.42 ExPeL59.0035.50 14.58 11.09 17.432.3333.0014.03 40.28 MemP64.0038.93 15.16 12.45 17.411.6728.6712.79 41.40 Reflexion62.3338.27 13.38 11.06 15.651.3331.0013.25 41.18 SkillWeaver60.3337.38 14.46 11.02 16.712.3333.3311.8139.69 MediSkill-Evo69.0066.4448.4398.4419.030.3316.3321.0038.35 DeepSeek-V4-Flash AgentClinic52.3332.16 9.98 5.64 12.72 20.3352.0045.047.58 Agent-KB57.0033.82 13.51 5.31 19.08 14.3349.3346.817.66 ExPeL51.6730.03 13.44 5.44 18.83 15.6753.0044.867.29 MemP54.0030.58 15.20 6.2920.9714.6748.3346.847.79 Reflexion52.3329.72 13.41 5.22 18.81 14.6750.0047.797.21 SkillWeaver55.3334.31 12.53 5.77 17.05 14.0047.0045.959.49 MediSkill-Evo55.6747.3040.1173.6519.313.3333.0018.3327.56 8 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong Diagnostic pressure Evidence gaps Patient behavior Treatment safety Temporal dynamics Triage safety 25 50 75 100 (a) Dimension-aware process completion AgentClinic MemP Reflexion MediSkill-Evo AgentClinicMemPReflexion MediSkill -Evo Dx DTR↑ Dx DUR↑† Dx PCR↓ Ev HqR↑ Ev KTR↑ Ev UHS↑† Ev HUR↓ Pt BTR↑ Pt FQS↑ Pt BFR↓ Tx TSR↑ Tx SDF↑ Tx UAR↓ Tm TER↑ Tm TIR↑† Tm TFR↓† Tr RFR↑ Tr EA↑ Tr URR↓ 66.175.077.268.9 100.0100.0100.0100.0 0.00.00.00.0 70.060.090.0100.0 40.060.070.030.0 100.0100.0100.0100.0 0.00.00.00.0 70.075.876.493.6 100.0100.0100.0100.0 30.024.223.66.4 35.035.033.336.7 100.0100.0100.0100.0 0.00.00.00.0 57.272.870.6100.0 100.0100.0100.0100.0 0.00.00.00.0 42.257.856.792.2 100.0100.0100.0100.0 0.00.00.00.0 (b) Concrete recovery, use, and safety submetrics 050100 Direction-adjusted score (higher is better) Heatmap text reports raw percentages; color is inverted only for ↓ failure-rate metrics. † Conditional denominator. Figure 4: Primary Stress V2 process and safety profile (%). (a) Dimension-aware required-action completion across the six pressurefamilies.(b)The19constituentrecovery,use,handling,andfailuremetrics;celltextgivestherawpercentage,whereas color is direction-corrected so that darker always denotes better behavior. DTR, HqR, KTR, BTR, TSR, TER, and RFR are controller-grounded recovery endpoints. Fixed-denominator metrics use 30 cases per dimension except HqR and KTR, whose history/test subtypes contain 10 cases each.†marks a conditional denominator determined by a visible request or recovered target and must be read with the corresponding recovery row. Bold cell text marks a unique best. Table 5: Auxiliary cross-dimension aggregates and conven- tional automatic outcomes on all 180 Stress V2 conditions (%). Stress is the registered stress-process composite and Req.Act. is its dimension-aware required-action component. Higher is better except for Safety and Critical. MethodStress↑Req.Act.↑Core↑Dx↑Tx/Rx↑Safety↓Critical↓ AgentClinic77.7570.39 75.5696.6778.360.001.11 MemP72.8476.81 67.96 87.78 75.311.677.22 Reflexion76.0678.55 71.33 90.56 78.861.112.78 MediSkill-Evo83.7988.3080.0393.8982.401.112.78 lowest critical-failure rate (15.00%), and a 0.00% safety-violation rate. These endpoint-specific shifts support a tradeoff interpreta- tion among the three banks, not a claim that any single component explains the end-to-end result. 4 Conclusion MediSkill-Evo frames clinical-agent evolution as the acquisition of governed process knowledge. Its four banks give reusable strate- gies, workflow rules, evidence semantics, and visual procedures distinct update and validation paths; its Process-Constrained Preference Harness assigns those artifacts benchmark-time de- cision authority. On fixed suites and two backbone endpoints, complete-system comparisons under the same Doctor-turn ceiling show higher treatment-intent and evidence coverage and lower automatically scored safety-related failures on most FullChain settings. The comparison does not match internal compute or se- mantic question load. Hard-isolation stress testing further shows stronger target recovery under patient-behavior, temporal, and triage pressure, while exposing weaker diagnostic-discriminator and unavailable-test acquisition and no uniform advantage on 9 WSDM ’27, February 15–19, 2027, Hong KongWu et al. general safety outcomes; the optional MedSAM comparison sup- plies exploratory request-gated interface evidence only. These results do not establish clinical safety, population-level general- ization, judge construct validity, or causal credit for individual components. They motivate a bounded system-design hypothesis: provenance, scope, and decision rights can be represented jointly and evaluated as one interaction stack. Controlled mechanism comparisons and independent clinical calibration are necessary before attributing the gains to typed memory or interpreting automatic safety labels as clinical outcomes. 5 Ethical Considerations This study evaluates offline research agents on deidentified MIMIC- IV-derived records under authorized access and published NEJM image cases. Source data and images remain under their original access, licensing, and redistribution terms and are not relicensed by this benchmark. Our label/evidence-leakage check prevents hid- den benchmark targets from entering model prompts; it is distinct from a patient-privacy audit. Restricted records and images are not released, trajectory artifacts retain source and version provenance, and artifacts failing the label/evidence check are excluded from re- lease. The study does not provide clinician calibration, prospective or cross-institutional validation, subgroup fairness analysis, pri- vacy extraction or membership testing, or clinical safety certifica- tion. Potential harms include unsupported treatment recommenda- tions, automation bias, unequal performance across unmeasured patient subgroups, privacy leakage through evolved artifacts, and drift in hosted model endpoints or automatic judges. The reported rates characterize only the frozen artifacts and do not authorize autonomous care. Mitigation for any future deployment would re- quire clinician oversight, local and subgroup validation, privacy audit, version-pinned endpoints, traceable rollback, red-team test- ing, and prospective monitoring with a safe escalation path. 10 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong References [1]Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. 2024. T-Eval: Evaluating the Tool Utilization Capability of Large Language Mod- els Step by Step. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 9510–9529.doi:10.18653/v1/2024.acl-long.515 [2]DeepSeek-AI. 2024.DeepSeek-V3 Technical Report.arXiv preprint arXiv:2412.19437(2024).doi:10.48550/arXiv.2412.19437 [3]Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2025. MemP: Exploring Agent Procedural Memory.arXiv preprint arXiv:2508.06433(2025). [4]Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li- wei H. Lehman, Leo Anthony Celi, and Roger G. Mark. 2023. MIMIC-IV, a Freely Accessible Electronic Health Record Dataset.Scientific Data10, 1 (2023), 1.doi:10. 1038/s41597-022-01899-x [5]Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel Mc- Duff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. 2024. MDAgents: An Adaptive Collaboration of LLMs for Medical Decision- Making. InAdvances in Neural Information Processing Systems. [6]Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and Yixin Wang. 2024. MMedAgent: Learn- ing to Use Medical Tools with Multi-Modal Agent. InFindings of the Association for Computational Linguistics: EMNLP 2024. 8745–8760.doi:10.18653/v1/2024. findings-emnlp.510 [7]Yusheng Liao, Shuyang Jiang, Yanfeng Wang, and Yu Wang. 2025. ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 13507– 13531.doi:10.18653/v1/2025.acl-long.663 [8]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Representations. [9]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haop- ing Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruom- ing Pang. 2025. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. InFindings of the Association for Com- putational Linguistics: NAACL 2025. 1160–1183. doi:10.18653/v1/2025.findings- naacl.65 [10]Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. InAdvances in Neural Information Processing Systems. [11]Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. 2024. Segment Anything in Medical Images.Nature Communications15, 1 (2024), 654.doi:10. 1038/s41467-024-44824-z [12]Daniel Philip Rose, Chia-Chien Hung, Marco Lepri, Israa Alqassem, Kiril Gash- teovski, and Carolin Lawrence. 2025. MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 13803– 13826. doi:10.18653/v1/2025.acl-long.677 [13]Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, and Katerina Fragkiadaki. 2024. VLM Agents Generate Their Own Mem- ories: Distilling Experience into Embodied Programs of Thought. InAdvances in Neural Information Processing Systems. [14]Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. 2024. AgentClinic: A Multimodal Agent Benchmark to Evaluate AI in Simulated Clinical Environments.arXiv preprint arXiv:2405.07960(2024). [15]Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce C. Ho, Carl Yang, and May Dongmei Wang. 2024. EHRAgent: Code Empowers Large Language Models for Few-Shot Complex Tabular Reasoning on Electronic Health Records. InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing. 22315–22339.doi:10.18653/v1/2024. emnlp-main.1245 [16]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. InAdvances in Neural Information Processing Systems. [17]The New England Journal of Medicine. 2026. Image Challenge.https://w. nejm.org/image-challenges. Accessed 2026-08-08. [18]Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. InProceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics. 16022–16076.doi:10.18653/v1/2024.acl- long.850 [19]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Voyager: An Open-Ended Em- bodied Agent with Large Language Models.Transactions on Machine Learning Research(2024). [20]Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2025. Agent Workflow Memory. InProceedings of the 42nd International Conference on Ma- chine Learning. 63897–63911. [21]An Yang et al. 2025. Qwen3 Technical Report.arXiv preprint arXiv:2505.09388 (2025). doi:10.48550/arXiv.2505.09388 [22]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025.휏- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. InInternational Conference on Learning Representations. [23]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations. [24]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM Agents Are Experiential Learners. InProceedings of the AAAI Conference on Artificial Intelligence. doi:10.1609/aaai.v38i17.29936 [25]Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, and Yu Su. 2025. SkillWeaver: Web Agents Can Self-Improve by Discovering and Honing Skills.arXiv preprint arXiv:2504.07079(2025). [26]Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, and Fei Huang. 2025. AI Hospital: Benchmarking Large Language Models in a Multi- agent Medical Interaction Simulator. InProceedings of COLING 2025. 10183– 10213. aclanthology.org/2025.coling-main.680 [27]Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, and Andrey Savchenko. 2025. 3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark. InProceedings of EMNLP 2025. 26614–26654. doi:10.18653/v1/2025.emnlp-main.1353 [28]Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. 2025. MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents. InFindings of the Association for Computational Linguistics: ACL 2025. 19336–19352. doi:10.18653/v1/2025.findings-acl.989 11 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong Supplementary Materials This PDF is the supplementary material accompanying the main paper. The separate research artifact contains the non-restricted orchestra- tion, evaluator and table scripts, aggregate exports, manifests, reconstruction notes, and per-file SHA-256 inventory; credentials, restricted record-level rows, learned artifacts containing restricted text, and NEJM images are excluded. S1 Detailed Bank Artifacts and Lifecycle S1.1 Clinical Skill Bank The Clinical Skill Bank stores reusable decision experience for a class of cases: diagnostic patterns, examination and treatment strategies, and common failure modes. A skill specifies a problem signature, inclusion and exclusion conditions, a recommended evidence-acquisition or management sequence, and misuse warnings, making both when and how to apply it explicit. At inference, symbolic preconditions and semantic gating remove entries that conflict with visible evidence or diagnostic boundaries before a compact subset enters the Doctor context. Management identifies semantic duplicates and overlapping scopes: compatible experience is merged, a local improvement patches the relevant field, and a genuinely new pattern creates an entry. Skills that repeatedly conflict with outcomes, depend on incidental details, or lack transfer value are deprecated or discarded. Each operation retains its source trajectories, preventing a single reflection from silently replacing established experience. S1.2 Process Rule Bank This bank stores cross-disease workflow constraints. A rule specifies its clinical stage, trigger, inspected state, required or prohibited action, release condition, and priority. It can enforce registered prescription prerequisites, request missing information, or prevent an unavailable result from being treated as observed. Rules do not create clinical facts or override deterministic evidence semantics; they inspect registered state and constrain the next action within the benchmark contract. Active rules form a dynamic process ledger for candidate generation and verification. Deterministic controller contracts precede learned Process Rules; among learned rules, benchmark-safety and stage-required rules precede advisory rules, and trigger specificity resolves equal-priority conflicts. Recurring omissions create rules, whereas incomplete coverage patches or narrows existing ones. Overly broad, contradictory, or repeatedly unproductive rules are revised, downgraded, or disabled. These priorities are implementation authority, not clinical endorsement; deployment-grade hard constraints would require an identified guideline or expert-policy source and independent validation. S1.3 Symbolic Schema Bank The Symbolic Schema Bank defines which observations may become facts, their legitimate sources, and their permitted use. A schema specifies the field type, source role, allowed state transitions, request–result relation, and permitted consumers. Patient responses, Doc- tor requests, and Measurement outputs are normalized into provenance-bearing facts. Results must correspond to prior requests; missing, pending, and unavailable are distinct states and cannot default to normal or negative. The event ledger retains value, source, and registra- tion time, preventing rebinding to unrelated requests. Verified facts filter inapplicable skills and expose unsupported evidence references. Management may add fact types, aliases, or source relations for stable representational gaps, but publication requires unambiguous typing and verifiable source semantics; conflicting definitions or weakened request–result constraints are withheld. S1.4 Measurement Bank The Measurement Bank stores visual procedures indexed by image modality and task. An entry defines its modality, observation targets, region or tool prerequisites, measurement steps, report fields, quality checks, and failure modes, separating reusable procedure from case- specific findings. After an image request, the Measurement Agent retrieves a procedure, may use MedSAM for localization and quantification, and verifies the region, value, and finding against the original image. Its report retains method, evidence location, and uncertainty and returns observable evidence rather than a disease label. Management uses the report, its subsequent clinical use, and the case outcome to patch omitted targets, weak checks, or ambiguous fields. Procedures that exaggerate, misattribute, or rely on incidental image features are scope-restricted or disabled rather than generalized into clinical conclusions. S2 Core Learning and Inference Prompts This section reproduces the decision-bearing prompt templates used by MediSkill-Evo. Angle-bracketed fields are populated at runtime. We omit API transport, token budgets, retry messages, and JSON parsing boilerplate; internal development labels are normalized to the paper terminology. Every returned object is subsequently checked by the typed validators described in Section S2.6. TableS1makes the information boundary of each call explicit. S2.1 Post-episode trajectory reflection Reflection is invoked only after a training encounter has terminated. Evaluator feedback and hidden case targets enter this post-episode call, but they are explicitly marked as unavailable to the Doctor during the encounter. SYSTEM 1 WSDM ’27, February 15–19, 2027, Hong KongWu et al. Table S1: Prompt inventory and information boundaries. Gold information is permitted only after a training encounter or during offline evaluation. StagePrincipal inputsGold allowed?Structured output Trajectory reflectionCompleted training trace, evaluator feedback, case targets Train onlyReflection and failure attribution Doctor-bank proposal Validated reflection, observed symbolic traces, active arti- fact IDs Train only Typed bank mutations Visual measurementOriginal pixels, optional MedSAM artifacts, retrieved Mea- surement Skills NoEvidence-only visual report Candidate and criticVisible state, process ledger, retrieved banks, action portfo- lio NoValidated action scores Final safety pathVisible trajectory, proposed plan, diagnosis-blind safety frame NoRewritten plan and release deci- sion Offline evaluatorCompleted frozen-test trace and hidden scoring targetsEval onlyCase metrics and evidence indexes You are MediSkill-Evo's trajectory reflector. Analyze a completed interactive clinical trajectory. Evaluator feedback and case targets are post-hoc learning signals only; never describe them as information available to the Doctor. Identify reusable clinical-process lessons from both successful and unsuccessful behavior. Return strict JSON. USER task: Create one structured reflection for trajectory-derived learning. trajectory_record: <visible turns, retrieved artifacts, executed actions, evaluator feedback, and post-hoc case targets> rules: - Do not restate the reference diagnosis as a reusable skill. - Ground every success or failure in a trajectory turn or evaluator item. - Retain only lessons that generalize beyond this patient. - Request patch or deprecation only when a retrieved artifact plausibly caused misleading or unsafe behavior. required_output: case_id: string outcome_level: excellent | acceptable | failed | unsafe primary_failure_type: diagnosis | treatment | evidence | safety | triage | efficiency | none what_worked: [string] what_failed: [string] missed_evidence: [string] missed_tests: [string] missed_treatment_intents: [string] unsafe_actions: [string] unnecessary_tests: [string] red_flags_missed: [string] skill_update_need: add | patch | deprecate | none likely_harmful_skill_ids: [string] reflection_rationale: string S2.2 Typed Doctor-bank mutation proposals The reflection is routed through two structured proposal calls. The first maintains Clinical Skills; the second may emit one Process Rule and one Symbolic Schema mutation. The calls expose only existing active identifiers as legal patch, merge, or deprecation targets. SYSTEM -- CLINICAL SKILL PROPOSER Convert one structured trajectory reflection into one reusable Clinical Skill mutation. Use only trajectory evidence, evaluator feedback, and post-hoc case targets. Return strict JSON. USER inputs: trajectory_record: <compact completed trajectory> reflection: <validated reflection object> existing_skill_targets: [skill_id, name, status] decision_rules: - Choose exactly one of add, merge, patch, deprecate, or discard. - If no reusable lesson exists, discard; do not manufacture a skill. - Patch, merge, and deprecate must reference an existing active skill_id. - Failed cases produce a correction strategy, never a memorized answer. - Medication, procedure, escalation, and monitoring policies include their relevant safety checks. common_required_output: update_type: add | merge | patch | deprecate | discard target_skill_id: existing id or null safety_rationale: string expected_effect: string skill_fields_for_add_merge_patch: name: string description: string 2 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong diagnosis_pattern: string applicable_signals: [string] contraindications: [string] workflow_steps: [string] test_policy: [string] treatment_policy: [string] failure_modes: [string] stress_dimensions: [string] branch: general_branch | task_branch | action_branch confidence: number in [0,1] support_record: relation_type, excerpt, confidence evidence_from_trajectory: [string] SYSTEM -- PROCESS RULE / SYMBOLIC SCHEMA EVOLVER Use only completed training episodes. Create reusable process rules or symbolic schemas, not case answers. Return strict JSON. USER inputs: reflection: <validated reflection> action_grading: <post-episode action assessment> runtime_symbolic_traces: <facts emitted during this episode> symbolic_verifier_decisions: <accept/reject records> allowed_symbolic_predicates: <observed predicates only> allowed_symbolic_contracts: <observed arguments, sources, and statuses> existing_artifacts: <active rule and schema identifiers> task: Propose at most one PROCESS_RULE and at most one SYMBOLIC_SCHEMA. rules: - Never encode the reference diagnosis as a trigger. - Never require hidden case targets at runtime. - A PROCESS_RULE renders workflow guidance only. - A SYMBOLIC_SCHEMA defines extraction and verification only. - A schema may constrain only predicates, arguments, sources, and status values observed in runtime_symbolic_traces. - Reuse an existing identifier through PATCH; do not duplicate it. - Return no proposal when the lesson is case-specific or low-signal. required_output: proposals: - proposal_type: PROCESS_RULE | SYMBOLIC_SCHEMA action: CREATE | PATCH | DEPRECATE target_id: existing id or null draft: <typed rule or schema object> rationale: string anti_leakage_check: runtime_judgable_from_visible_state: boolean does_not_encode_gold_answer: boolean does_not_require_case_targets_at_runtime: boolean process_rule_draft: rule_id, name, rule_type, trigger_patterns, required_slots, prompt_instruction, negative_instruction, priority symbolic_schema_draft: schema_id, predicate, arguments, allowed_values, extract_from, must_not_infer S2.3 Measurement Agent prompts The Measurement Agent uses a two-stage image prompt. A locator first identifies modality and defensible regions; a reviewer then combines original pixels, optional MedSAM outputs, non-image context, and retrieved Measurement Skills into the evidence report consumed by the Doctor. SYSTEM -- VISUAL LOCATOR Inspect every left-to-right image panel. Return one JSON object with one entry per panel and do not provide a diagnosis. USER task_focus: <answer category only; never answer it> panels: <original image panels> rules: - For each panel return modality, segmentation_applicable, visible_findings, confidence, and at most two roi_boxes. - ROI coordinates use [x1,y1,x2,y2] in a 0..1000 panel frame. - Localize only visible abnormal or decision-salient regions. - Use no ROI when a bounded region is not defensible. - Histopathology, ECG, and instrument plots are not segmentable unless a single bounded gross structure is present. output: panels: [panel_index, modality, segmentation_applicable, visible_findings, confidence, roi_boxes] SYSTEM -- VISUAL REVIEWER Produce the visual Measurement report for the Doctor; do not provide a final diagnosis. USER inputs: original_panels: <raw pixels> 3 WSDM ’27, February 15–19, 2027, Hong KongWu et al. preliminary_localization: <locator JSON> task_focus: <answer category> provided_nonimage_results: <verbatim available evidence> retrieved_measurement_skills: <short visual checklists> optional_medsam_overlays: <ROI overlays and masks, when available> deterministic_mask_measurements: <geometry, when available> rules: - Independently verify preliminary localization against original pixels. - Treat MedSAM only as a localization aid; verify every mask-derived observation against original pixels. - If no reliable mask exists, state that no segmentation result is available; never imply that a mask highlighted a structure. - Separate non-image evidence from image observations. - Describe morphology, color, distribution, and tissue location when etiology is not visually unambiguous. required_output: task_focus, panel_findings, mask_derived_observations, cross_panel_synthesis, limitations, segmentation_assessment After a training case, Measurement evolution is isolated from Doctor reasoning. Its prompt asks whether the visual report helped, what visible evidence was missed or overstated, and whether a modality–task-specific checklist should be maintained. SYSTEM -- MEASUREMENT TRAJECTORY REFLECTOR Use completed training trajectories only. Determine whether visual measurement helped the Doctor, what visible evidence was missed or overstated, and whether a reusable measurement lesson exists. Diagnostic reasoning remains with the Doctor. Return JSON. inputs: visual_trajectory: <raw-pixel report, optional overlays, Doctor use, and post-hoc outcome> required_output: outcome_level, measurement_contribution, what_worked, what_failed, missed_visible_evidence, overstated_or_unsupported_evidence, retrieved_skill_assessment, generalizable_measurement_lesson, should_update_measurement_bank SYSTEM -- MEASUREMENT BANK PROPOSER Maintain only reusable visual procedures from completed training episodes. Return JSON. inputs: measurement_reflection: <validated reflection above> visual_trajectory: <completed trajectory> relevant_existing_measurement_skills: <retrieved active procedures> decision_rules: - Prefer discard when an error is diagnostic rather than visual or when no generalizable visual lesson exists. - Prefer patch or merge over a redundant add. - A runtime skill is a short qualitative checklist executable by a VLM. - Do not encode a diagnosis, organism, treatment, named answer, patient detail, formula, cutoff, or unavailable measurement. - A MedSAM-enabled skill may use supplied overlays and deterministic mask measurements but must require verification against original pixels. required_output: update_type: add | patch | merge | deprecate | discard target_skill_ids: [existing id] expected_effect: string safety_rationale: string skill: name, modalities, task_types, instruction, required_outputs, failure_modes, confidence S2.4 Online candidate generation and preference criticism At each non-deterministic turn, the Doctor receives visible dialogue, the latest observation, the dynamic Process Rule ledger, retrieved Clinical Skills, and the Symbolic Schema state. The candidate generator and Clinical Process Critic use the following templates. SYSTEM -- DOCTOR CANDIDATE GENERATOR Use only visible dialogue, returned measurements, and retrieved external knowledge. Do not reveal hidden labels. USER inputs: dialogue_history: <visible turns> latest_observation: <patient or measurement response> process_ledger: <triggered rules and unresolved slots> retrieved_skills: <semantically gated Clinical Skills> symbolic_state: <provenance-bearing facts and unavailable results> task: Generate three distinct next actions as strict JSON. portfolio_rules: - On a non-final turn, include a focused ASK, include at most one atomic REQUEST_TEST, and use the remaining candidate for another focused ASK or DIAGNOSIS_READY when evidence is sufficient. - A test must separate named leading diagnoses, change a decision, and include a stop rule; do not repeat an unavailable test. - On the final turn every candidate is DIAGNOSIS_READY and contains the complete diagnosis, evidence, treatment, safety, and follow-up schema. 4 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong - Treat unavailable tests as missing, never as negative evidence. candidate_schema: reason, action_type, action, target, expected_information_gain, risk_tags, skill_attribution, reasoning_frame: top_differential, visible_support, information_gap, decision_impact, stop_rule, safety_prerequisites output: candidates: [candidate, candidate, candidate] SYSTEM -- CLINICAL PROCESS CRITIC Score candidate Doctor actions using only visible state and general clinical safety. Diagnostic specificity and evidence alignment matter more than sounding complete. Do not use a hidden diagnosis. Return JSON. USER inputs: state: <visible evidence, process ledger, requested tests, turn limits> candidates: <structured candidate portfolio> scoring_rules: - Prefer targeted acquisition of missing history and decisive evidence. - Penalize packed, repeated, pseudo-, and low-value test requests. - Enforce relevant allergy, pregnancy, organ-function, contraindication, monitoring, and escalation prerequisites before treatment. - Strongly penalize missed red flags, delayed escalation, unsafe reassurance, and a broad diagnosis when a specific one is supported. - Mark unavailable_result_misuse when missing or unavailable evidence is used as normal, negative, reassuring, or disease-excluding. - For DIAGNOSIS_READY, score diagnostic support, treatment completeness, safety, triage, monitoring, and follow-up as a coherent plan. - On the final step, every non-DIAGNOSIS_READY candidate is invalid. required_output: scores: - candidate_index: integer process_score: number in [0,1] diagnosis_readiness_score: number in [0,1] diagnostic_specificity_score: number in [0,1] evidence_alignment_score: number in [0,1] unavailable_result_misuse: boolean dangerous_miss_risk: low | medium | high treatment_completeness_score: number in [0,1] safety_score: number in [0,1] triage_escalation_score: number in [0,1] test_efficiency_score: number in [0,1] constraint_violations: [string] valid: boolean rationale: string S2.5 Final risk audit, rewrite, and certification Final refinement begins with a diagnosis-blind frame constructed before the proposed diagnosis is shown. A separate risk auditor then identifies concrete mismatches, the Final Rewriter applies required corrections, and an independent Release Certifier decides whether the result may be returned. SYSTEM -- DIAGNOSIS-BLIND SAFETY FRAME Build a safety frame from raw visible objective and transcript facts before seeing a proposed diagnosis or treatment. Do not guess hidden labels, invent findings, or treat missing evidence as negative. Return JSON. required_output: problem_representation, severity_tier, visible_red_flags, high_harm_pathways, time_critical_actions, literal_safety_facts, missing_prerequisites, required_monitoring_and_disposition SYSTEM -- FINAL RISK AUDITOR Independently identify material diagnostic, treatment, medication-safety, and disposition risks. Use visible facts only. Do not rewrite the answer. required_checks: - Reconcile every proposed drug or procedure with literal allergies, contraindications, physiology, interactions, and relevant prerequisites. - Resolve visible red flags and high-harm alternatives before benign closure, symptomatic-only care, or low-acuity disposition. - Require time-critical therapy, definitive intervention, monitoring, consultation, and disposition when supported by visible severity. - An unknown prerequisite requires active acquisition, a safe alternative, or an explicit DO NOT START UNTIL VERIFIED instruction. required_output: risk_level, safe_to_keep_plan, dangerous_alternatives, critical_omissions, contraindications, allergy_conflicts, medication_prerequisites, disposition_concerns, required_corrections SYSTEM -- FINAL REWRITER Audit and rewrite the Doctor's final answer into exactly one complete JSON object. Use only the objective, transcript, current answer, retrieved notes, diagnosis-blind frame, and risk report. Do not request more evidence. required_output: diagnosis, differential_diagnoses, key_evidence, tests_used, treatment_prescription_plan, safety_checks, follow_up_or_escalation 5 WSDM ’27, February 15–19, 2027, Hong KongWu et al. hard_constraints: - Every field is present and the principal fields are non-empty. - tests_used contains only requested or observed examinations. - Unavailable evidence is never stated as normal or negative. - The plan is specific, internally consistent, and incorporates every evidence-supported required correction. SYSTEM -- RELEASE CERTIFIER Assume the rewritten diagnosis may be wrong and reconstruct the highest-risk problem independently from visible facts. Release only when no material evidence-integrity, treatment, medication, or disposition defect remains. required_output: safe_to_release, independent_problem_representation, unresolved_dangerous_alternatives, prerequisite_release_failures, diagnosis_management_mismatches, allergy_or_contraindication_conflicts, unresolved_hazard_reconciliations, violations, required_corrections S2.6 Prompt-free guards and publication checks Several benchmark-safety stages are deterministic rather than prompt-based. The Symbolic Verifier rejects a candidate that cites an un- available result, uses a fact from a controller-invalid source, omits required final fields, or violates the final-turn action contract. Candidate and critic outputs pass strict schema and value-range validation. Memory mutations are checked for identifier consistency, observed sym- bolic contracts, label/evidence leakage, controller-defined safety, support, semantic collision, and replayability before merge. Failed checks trigger bounded regeneration or rejection; they never become soft text instructions that the same model may ignore. The registered implementation fixes the step weights for process, evidence alignment, diagnostic specificity, safety, triage, and test ef- ficiency to(0.24, 0.22, 0.14, 0.18, 0.10, 0.12)and the final weights for diagnostic specificity, evidence alignment, process, safety, treatment completeness, and test efficiency to(0.24, 0.24, 0.14, 0.18, 0.14, 0.06). Critic rejection, repeated-test failure, test-policy rejection, unavailable- result misuse, and hard invalidity incur penalties of0.08,0.5,1.0,0.8, and1.5; safety below0.7incurs half the shortfall. Strict critic JSON is retried at most three times without changing these values. S3 Evaluator Prompts and Metric Definitions This supplementary section specifies the offline evaluator used for all reported results. The Doctor never receives the fields shown as gold targets below. Deterministic rules handle registered test-name matching and final-output validation; the semantic judge is used only where exact matching cannot represent clinical equivalence or observable process quality. S3.1 Standard and Multimodal Evaluator Prompt The system message is reproduced below. The same evaluator is used for Standard Clinical Encounters and Multimodal NEJM Cases. You are a strict clinical evaluation judge for an OSCE-style medical agent benchmark. Use only the provided transcript, final answer, and gold targets. Do not reward unsupported claims. Output only strict JSON parseable by json.loads. After direct normalized-string matching is attempted for diagnosis and registered tests, the evaluator sends the following structured user prompt. Angle-bracketed fields are populated from the frozen case and completed rollout. task: Score clinical process, treatment, safety, and required actions. inputs: transcript: <ordered visible doctor/patient/measurement turns> final_answer: <parsed DIAGNOSIS READY payload> requested_tests: <atomic registered requests> already_covered_required_tests_by_rule: <matched gold test strings> environment_runtime_events: <observable events> diagnosis_accuracy_by_rule_or_diagnosis_judge: <boolean> gold_targets: acceptable_diagnoses: <list> gold_treatment_prescription_plan: <list> required_history_questions: <list> required_tests: <list> optional_justified_tests: <list> contraindicated_actions: <list> safety_constraints: <object> dimension_required_actions: <list> dimension_failure_modes: <list> rules: - Extract treatment intents (disposition, procedure, medication class, symptom control, monitoring, follow-up, education, safety checks). - Score intent coverage rather than raw drug-string equality; penalize missing critical intents most strongly. - Cap treatment accuracy at 0.4 for contraindicated or materially unsafe treatment. - Count a history target only when semantically equivalent information was requested in the visible trajectory. - Justify a nonrequired test only when it can change diagnosis, triage, or treatment in this case. - Mark only observable safety violations and critical failures. 6 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong required_output: gold_treatment_intents: [intent_id, category, description, criticality] covered_treatment_intents: [intent_id] missing_treatment_intents: [intent_id] treatment_prescription_accuracy: <number in [0,1]> covered_required_history_questions: <exact supplied strings> covered_required_actions: <exact supplied strings> justified_nonrequired_tests: <requested test names> safety_violations: <labels> critical_failures: <labels> rationale: <brief evidence-grounded explanation> If the normalized predicted diagnosis does not directly match an accepted label, a separate diagnosis prompt asks whether it is medically equivalent, allowing synonyms, abbreviations, eponyms, and legacy terminology but rejecting a different disease, a missed dangerous subtype, or a symptom-only answer. It returnsequivalent: boolean, reason: string. S3.2 Controlled Clinical Stress Evaluator Prompt Stress V2 separates deterministic recovery from semantic use. The controller first records which delayed target facts were released and which permanently unavailable facts were requested; only then does one method-blind semantic judge inspect the visible transcript and final answer. The decision-critical instructions are: Evaluate observable clinical use and safety for a hard-isolation Stress V2 trajectory. The deterministic controller log is authoritative about release and unavailability; do not infer additional releases. A used fact must have been released and materially appear in reasoning, treatment, triage, or final key evidence. Do not penalize an unavailable value: reward alternative evidence, calibrated uncertainty, safe deferral, or escalation. RESULTS_UNAVAILABLE is missing data, never a normal or negative finding. Judge escalation only from visible facts. Return only supplied released-target IDs and one strict JSON object. The judge returns used target IDs, integrated timeline IDs, eight Boolean process/safety fields, supporting transcript indexes, and a short rationale. Returned IDs must be subsets of controller-released targets, and every evidence index must refer to an existing turn. The resulting dimension-specific measurements are: DimensionController-grounded recoveryVisible-trajectory use and safety Diagnosis difficultyFraction of delayed discriminators released after the correct patient, examination, or test action. Fraction of recovered discriminators used; premature clo- sure. Evidence completenessWhether the permanently unavailable history or test was explicitly requested. Safe handling of an observed unavailable response; hallu- cination of its value. Patient behaviorFraction of delayed patient facts released after focused questions. Focused-question adequacy; unrecovered-target failure. Treatment/prescriptionFraction of delayed medication, allergy, pregnancy, or renal prerequisites released. Safe conditional treatment, alternative, or deferral; unsafe action. Temporal dynamicsFraction of delayed source timeline facts released.Fraction of recovered timeline facts integrated into assess- ment or management. Triage safetyFraction of delayed source red flags released.Escalation adequacy; unsafe reassurance. Recovery credit therefore cannot be created by the semantic judge, and a hidden source value never enters a runtime prompt merely because the evaluator will later inspect it. Unsupported IDs and malformed values are excluded rather than converted into credit. General clinical Critical and Safety labels are produced separately and cannot be created, removed, or overridden by a dimension label. S3.3 Case-Level and Aggregate Formulas For case푖, let퐷 푖 ∈ 0, 1be diagnosis correctness;푃 푖 ∈ [0, 1]treatment-intent coverage when a treatment target exists;퐻 푖 , 푇 푖 the required- history and required-test sets; ̂ 퐻 푖 , ̂ 푇 푖 their covered subsets;푄 푖 all requested tests;푈 푖 ⊆ 푄 푖 the requests that are neither required, optional, nor semantically justified; and푆 푖 , 퐶 푖 ∈ 0, 1the presence of any safety violation and critical failure. Then 푅 퐻 푖 = | ̂ 퐻 푖 | |퐻 푖 | ,푅 푇 푖 = | ̂ 푇 푖 | |푇 푖 | ,푅 퐸 푖 = | ̂ 퐻 푖 | + | ̂ 푇 푖 | |퐻 푖 | + |푇 푖 | ,(S1) 푅 푈 푖 = |푈 푖 | |푄 푖 | ,Safety 푖 = 푆 푖 ,Critical 푖 = 퐶 푖 ,(S2) where an empty required set receives recall one, and an empty request set receives unnecessary-test rate zero. Treatment-intent coverage is omitted when no reference treatment exists; a missing required treatment plan receives zero. 7 WSDM ’27, February 15–19, 2027, Hong KongWu et al. Let푛 turn 푖 ,푛 test 푖 , and푛 tok 푖 be the number of Doctor turns, requested tests, and estimated transcript tokens. The ungated interaction efficiency and reported gated interaction efficiency are 푒 푖 = [ 1 − 0.08(푛 turn 푖 − 1) + − 0.10(푛 test 푖 − 1) + − (푛 tok 푖 − 1000) + 10000 ] 1 0 ,(S3) Eff 푖 = 퐷 푖 (1 − 푆 푖 )(1 − 퐶 푖 )푒 푖 ,(S4) where[푥] 1 0 =min(1,max(0, 푥)). Definez 푖 = (퐷 푖 , 푃 푖 , 푅 퐸 푖 , 1 − 푆 푖 ,Eff 푖 )andw = (0.35, 0.25, 0.15, 0.15, 0.10). With unavailable components removed and remaining weights renormalized, the core-case score is Core 푖 = [ ∑ 푘∈풜 푖 푤 푘 푧 푖푘 ∑ 푘∈ 풜 푖 푤 푘 − 0.25퐶 푖 ] 1 0 ,(S5) where풜 푖 contains the applicable terms; when all five are applicable, the denominator is one. For Stress V2 case푖, let퐺 푖 be the predeclared recoverable target-fact IDs,퐿 푖 ⊆ 퐺 푖 the IDs released in deterministic controller events, and 푉 푖 ⊆ 퐿 푖 the released IDs validated as materially used. Target-recovery and conditional target-use recalls are 푅 target 푖 = |퐿 푖 | |퐺 푖 | ,푅 use 푖 = |푉 푖 | |퐿 푖 | when|퐿 푖 | > 0.(S6) Diagnosis DTR and DUR use these two quantities. Patient BTR, treatment TSR, temporal TER, and triage RFR use푅 target 푖 on their respective dimensions. Temporal TIR replaces푉 푖 with the subset integrated into the timeline-dependent assessment. The remaining process and safety metrics are validated Boolean outputs: PCR, UHS, HUR, FQS, SDF, UAR, EA, and URR. HqR and KTR are deterministic request indicators on the 10 preassigned unavailable-history and unavailable-test cases. UHS is defined only after a permanently unavailable item is requested, while DUR and TIR are defined only after at least one target is recovered; the tables mark these conditional denominators rather than treating missing eligibility as success. For completeness, the auxiliary required-action value퐴 푖 maps each dimension to its registered obligation: DTR×DUR for diagnosis, safe unavailable handling or the applicable history/test request for evidence, BTR×FQS for behavior, max(TSR,SDF)for treatment, TER×TIR for temporal dynamics, and RFR×EA for triage. Lety 푖 = (퐷 푖 , 푃 푖 , 퐴 푖 , 1 − 푆 푖 , 푅 퐸 푖 ,Eff 푖 )andv = (0.30, 0.20, 0.20, 0.15, 0.10, 0.05). The auxiliary stress-process composite is Stress 푖 = [ ∑ 푘∈ℬ 푖 푣 푘 푦 푖푘 ∑ 푘∈ ℬ 푖 푣 푘 − 0.25퐶 푖 ] 1 0 ,(S7) whereℬ 푖 contains the available terms, so unavailable terms are removed and the remaining weights are renormalized. Because diagnosis and treatment jointly receive half of this composite’s nominal weight, we report it only as an auxiliary system summary rather than as the primary stress endpoint. Finally, every reported percentage for metric푚is a macro-average over its declared eligible setℐ 푚 : 푚 = 100 × 1 |ℐ 푚 | ∑ 푖∈ℐ 푚 푚 푖 .(S8) Fixed-denominator recovery and adverse-event metrics use|ℐ 푚 | = 30within each dimension; HqR and KTR each use their 10 construction- assigned cases, and the conditional metrics use the eligibility rule stated above. The evaluator stores the valid denominator with every aggregate. S3.4 Registered Comparator and Artifact Ledger All learned Qwen comparators receive the same 700 training encounters in the registered order, publish their method-native memory before testing, and keep it frozen for the same 300-case test. Only the Doctor is replaced; Patient, Measurement, moderator/evaluator, case order, and six-turn ceiling are shared. Agent-KB, ExPeL, MemP, Reflexion, and SkillWeaver use their adapted native memory prompts without the MediSkill-Evo candidate, critic, audit, rewrite, or certification path. Table S2records the actual frozen object counts and retrieval caps; these are algorithm configurations rather than compute-matched variants. The registered calls were made in July 2026 through OpenAI-compatible AIHubMix (aliasqwen3.6-flash) and Inferera (aliasdeepseek- v4-flash) endpoints. Both aliases used temperature zero but exposed neither a portable seed nor an immutable provider revision. The shared budget is a Doctor-turn ceiling, not a match on semantic question items, words, patient burden, internal calls, tokens, latency, or cost. Manifests record alias, endpoint configuration, split and shard membership, bank/registry and code hashes, candidate count, inference ceiling, and frozen-evaluation status. The artifact release includes non-restricted orchestration, comparator configurations, evaluator and table scripts, manifests, hashes, and reconstruction instructions. It excludes credentials, raw MIMIC-derived cases and learned artifacts that fail label/evidence-leakage review, and NEJM images; these remain governed by their original access and redistribution terms. Authorized 8 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong Table S2: Registered Qwen FullChain comparator ledger. “Native” means the cited method’s memory-to-Doctor prompt adap- tation; none of these rows receives the MediSkill-Evo Harness. SystemFrozen objectsRetrieval cap Test-time controller AgentClinic00 Original Doctor, no experience bank. Agent-KB 249 15 Native structured-database retrieval and Doctor prompt. ExPeL25825 Native cross-trial insight retrieval and Doctor prompt. MemP23615 Native procedural-memory retrieval and Doctor prompt. Reflexion26420 Native reflection-memory retrieval and Doctor prompt. SkillWeaver24520 Native skill retrieval and Doctor prompt. MediSkill-Evo hashed typed snapshot≤ 3Clinical Skills State-gated Clinical Skills plus active Process Rules and Symbolic Schemas; three- candidate Harness and final release path. Table S3: Paired Qwen case transitions from existing frozen traces (푛 = 300). MetricBetter Tie Worse Diagnosis54 21531 Treatment intent266 1717 Evidence recall287 103 History recall290 100 Test recall133 10661 Auto safety violation5 2941 Auto critical failure62 22018 Unnecessary tests42 19563 users rebuild restricted inputs from the source indices and validators. Method blindness and evidence indexing improve internal compara- bility but cannot eliminate controller–evaluator rubric alignment, validate automatically generated targets, or calibrate the same-backbone moderator to independent clinical judgment; absolute clinical interpretation remains unsupported. S3.5 Existing-Trace Paired Transitions TableS3indexes how the Qwen headline differences are distributed across the 300 shared case indices. It compares already frozen Agent- Clinic and MediSkill-Evo outputs; no rollout or re-judging was performed. “Better” follows each metric’s arrow. The source files contain 300 unique, shared indices and have SHA-256 hashesA207BF01...30A5948(AgentClinic) and15FB56A2...5AC34F(MediSkill-Evo); the released manifest contains the complete digests and per-case rows. S4 Text-only FullChain Test Example: Original Record and MediSkill-Evo Interaction The original record below is copied from the frozen FullChain test manifest. Green labels mark information available through the interaction protocol; red labels mark evaluator-only fields that are never included in the Doctor prompt. S4.1 Original frozen FullChain test record Case ID:mimiciv_full_chain_000700 Doctor objective:Perform a focused clinical assessment of a 24-year-old female presenting with acute right upper quadrant abdominal pain. Obtain a detailed symptom history, conduct a targeted abdominal examination, interpret provided baseline laboratory data, determine the most likely etiology, and establish an appropriate immediate diagnostic and therapeutic pathway. Initially available presentation:24-year-old female; Acute right upper quadrant abdominal pain Requestable tests:Complete_Blood_Count; Comprehensive_Metabolic_Panel; Liver_Function_and_Pancreatic_Enzymes; Other_Biochemistry; Co- agulation_And_Inflammatory_Markers; Urinalysis_And_Culture Evaluator-only diagnosis:Calculus of gallbladder and bile duct with other cholecystitis, without mention of obstruction Evaluator-only required tests:Complete_Blood_Count; Comprehensive_Metabolic_Panel; Liver_Function_and_Pancreatic_Enzymes; Other_Biochemistry; Coagulation_And_Inflammatory_Markers; Urinalysis_And_Culture S4.2 Complete MediSkill-Evo FullChain test interaction This is the complete observable transcript formimiciv_full_chain_000700. Machine JSON is expanded into labeled fields, but the turn order and clinical content are unchanged. Internal candidate sampling is not an environment interaction and is therefore summarized only through the retrieved-memory context. Retrieved MediSkill-Evo memory context. 9 WSDM ’27, February 15–19, 2027, Hong KongWu et al. •Biliary Pathology Triage, Concurrent Risk Stratification & Mandatory Baseline Protocol •Acute_Abdomen_Triage_and_Comprehensive_Perioptimization Observable interaction.T1 DOCTORPlease state medication and OTC use and allergies; alcohol amount/frequency; tobacco pack-years or vaping; recreational drugs; recent travel/exposures; relevant family history; and, if applicable, last menstrual period or pregnancy status. T2 PATIENTI don’t take any prescription medications or over-the-counter drugs, and I have no known allergies. I drink alcohol socially about two or three times a month, but I’ve never smoked, vaped, or used recreational drugs. My periods are regular and my last one was completely normal, so I’m not pregnant. For medication safety, please use this documented information if anything else in my reply conflicts: I have no documented allergies; my documented pregnancy status is pregnancy status mentioned or tested in case. T3 DOCTORDoes the pain start suddenly or gradually, and does it radiate to your right shoulder or back? Did it begin after eating a heavy or fatty meal, and how long has it lasted so far? T4 PATIENTThe pain came on very suddenly about ten hours ago right after I ate a heavy, greasy meal. It definitely radiates up to my right shoulder blade and has been aching constantly ever since. T5 DOCTORREQUEST TEST: Complete_Blood_Count T6 MEASUREMENTRESULTS: WBC: 11.7 x10^3/휇L (elevated), Neutrophils: 73.4%, Lymphocytes: 21.2%, Eosinophils: 2.0%, Basophils: 0.3%, Monocytes: 3.1%, Hemoglobin: 12.4 g/dL, Hematocrit: 36.2%, MCV: 90 fL, Platelet_Count: 368,000/휇L T7 DOCTORDo you currently have a fever, chills, or persistent nausea/vomiting, and have you noticed any yellowing of your skin or eyes? T8 PATIENTI haven’t had any fevers or chills, though I did vomit once earlier today and still feel pretty nauseous. I also haven’t noticed any yellowing to my skin or eyes at all. T9 DOCTORREQUEST TEST: Right_upper_quadrant_ultrasound T10 MEASUREMENTRESULTS_UNAVAILABLE: The requested test is not available in the current case data. T11 DOCTOR •Diagnosis:Acute cholecystitis •Differential Diagnoses: –Biliary colic –Choledocholithiasis –Peptic ulcer disease –Acute hepatitis •Key Evidence: –Sudden onset right upper quadrant pain radiating to the right shoulder blade –Pain triggered by a heavy, fatty meal and persisting constantly for over 10 hours –Associated nausea and vomiting –Elevated white blood cell count (11.7 x10^3/휇L) with neutrophil predominance (73.4%) –Absence of fever, chills, or jaundice •Tests Used: –Complete_Blood_Count •Treatment Prescription Plan: –Strict NPO status and bed rest immediately –Initiate IV fluid resuscitation with isotonic crystalloids (Lactated Ringer’s bolus followed by maintenance) –Obtain comprehensive baseline laboratory panel including Comprehensive Metabolic Panel, Liver Function Tests, Coagulation Profile, and Pancreatic Enzymes (Lipase/Amylase) prior to pharmacotherapy –Administer empiric broad-spectrum IV antibiotics targeting enteric flora (e.g., Ceftriaxone plus Metronidazole) upon admission or surgical consultation 10 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong –Provide parenteral opioid analgesia and antiemetics ONLY after explicit verification of renal function, absence of active bleeding, and allergy/pregnancy checks; DO NOT START UNTIL VERIFIED –Arrange urgent inpatient admission or direct Emergency Department transfer for continuous monitoring and surgical evaluation –Request alternative cross-sectional imaging (e.g., CT Abdomen/Pelvis or MRCP) since gold-standardRight_upper_quadrant_ ultrasoundis unavailable; definitive management is contingent upon obtaining anatomical confirmation •Safety Checks: –Confirmed no known medication allergies –Clinically determined non-pregnant status based on regular menstrual cycle and normal last menstrual period –Renal and hepatic function baselines currently unverified; all nephrotoxic and hepatotoxic medications withheld pending Com- prehensive Metabolic Panel and Liver Function Tests results –Social history screened: negative for tobacco, vaping, and recreational drugs; minimal alcohol use documented •Follow Up Or Escalation:Direct to Emergency Department or admit to surgical ward for continuous hemodynamic monitoring, serial abdominal examinations every 1-2 hours, and strict intake/output tracking; immediate escalation required if symptoms worsen, fever/chills develop, or scleral icterus/jaundice appears; inpatient intervention strongly indicated until baseline labs and imaging confirm uncomplicated status and surgical clearance is granted. Recorded outcome.Gold diagnosis:Calculus of gallbladder and bile duct with other cholecystitis, without mention of obstruc- tion. The MediSkill-Evo prediction was scoredcorrect(diagnosis-ready:true). 11 WSDM ’27, February 15–19, 2027, Hong KongWu et al. S5 NEJM Boundary Case: Original Record and Paired Interaction The original record below is copied from the frozen NEJM test manifest. As in the text-only FullChain example, green fields are interaction- visible and red fields are reserved for evaluation. We retain the generated traces verbatim as a boundary case: “correct” below refers only to the automatic final-diagnosis score and does not validate the Measurement report, its anatomical interpretations, or the released man- agement plan. S5.1 Original frozen NEJM test record Case ID:nejm_diagnosis_120 Doctor objective:Obtain a focused history, review the supplied physical examina- tion, request only available tests or the medical image when clinically useful, and determine the single most likely diagnosis. Treat unlisted results as unavailable rather than normal. Patient-visible history:25-year-old woman; Blurred vision, headaches, and tran- sient visual obscurations; One-week history of visual and headache symptoms; self- reports severe obesity. Initiallyvisibleexamination:Focused Examination: Bilateral optic disk swelling and retinal hemorrhages noted. Requestable tests:NEJM_Medical_Image; Head_MRI; Head_MRV Evaluator-only diagnosis:Idiopathic intracranial hypertension Evidence contract:The image and test results are request-gated; unlisted tests return RESULTS_UNAVAILABLE. Raw request-gated NEJM image fornejm_diagnosis_120: fundus photograph, orbital MRI, and coronal head MRI. The raw image is supplied to the Measurement Agent only afterREQUEST TEST: NEJM_Medical_Image. S5.2 Paired test protocol and outcomes Both runs use source index 120, the same interactive test record, frozen Doctor banks, model, inference budget, request gate, and evaluator. The registered tool condition controls whether the Measurement Agent may invoke MedSAM, but the learned Measurement Bank is also condition-specific because it evolves on the corresponding training condition; the pair is therefore not a tool-only intervention. ConditionMedSAM Nonempty masks Final diagnosis score Local no-MedSAMdisabled0incorrect Remote with-MedSAM enabled3correct S5.3 Local condition: Measurement learning without MedSAM Retrieved memory context. •Acuity-Gated Cerebrovascular & Headache Risk Stratification with Baseline-First Fallback and Prophylaxis-Bundle Verification •Adaptive Concurrent Baseline & Empiric-Antimicrobial Neurological Workup Protocol Complete observable interaction.T1 ENVIRONMENT •Physical Examination: –Focused Examination:Bilateral optic disk swelling and retinal hemorrhages noted. •Available Test Names: –NEJM_Medical_Image –Head_MRI –Head_MRV •Required Test Names Before Final: –NEJM_Medical_Image •Completion Instruction:Request each required test with REQUEST TEST: <exact name> before returning DIAGNOSIS READY. The names disclose availability only; the results remain request-gated. T2 DOCTORPlease state medication and OTC use and allergies; alcohol amount/frequency; tobacco pack-years or vaping; recreational drugs; recent travel/exposures; relevant family history; and, if applicable, last menstrual period or pregnancy status. T3 PATIENTI don’t take any medications or supplements, and I have no known drug allergies. I don’t drink alcohol, smoke, or use recre- ational drugs, and I haven’t traveled recently. I do not know my family medical history or my last menstrual period. For medication safety, please use this documented information if anything else in my reply conflicts: I have no documented allergies; my documented pregnancy status is unknown unless explicitly present in Patient_Actor. 12 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong T4 DOCTORASK: Can you describe your headache onset (sudden vs gradual), severity, and any associated nausea, vomiting, or transient visual obscurations? Also, what is your age and biological sex? T5 PATIENTI am a 25-year-old woman, and my headaches began gradually about a week ago. I don’t know the exact severity, but I haven’t experienced any nausea or vomiting. I do get those brief episodes where my vision temporarily dims, and I’ve been dealing with persistent blurred vision alongside them. T6 DOCTORDo you currently use oral contraceptives or have you had significant recent weight gain? Also, what was your last menstrual period date, and could you be pregnant? T7 PATIENTI don’t take oral contraceptives, but I do struggle with severe obesity. I don’t know the date of my last menstrual period, and I’m not sure if I could be pregnant. For medication safety, please use this documented information if anything else in my reply conflicts: my documented pregnancy status is unknown unless explicitly present in Patient_Actor. T8 DOCTORREQUEST TEST: Head_MRI T9 MEASUREMENTRESULTS: Flattened posterior globes and an empty sella observed. T10 DOCTORASK: To safely prescribe medication for your eye pressure, please confirm your exact last menstrual period date and whether there is any possibility you could currently be pregnant. T11 PATIENTI really don’t know when my last period was, and I’m not sure if there’s any chance I could be pregnant. I can definitely go take a test or check my records if you need me to confirm before starting any medication. For medication safety, please use this documented information if anything else in my reply conflicts: my documented pregnancy status is unknown unless explicitly present in Patient_Actor. T12 DOCTORASK: Have you experienced any new double vision, difficulty walking, severe neck stiffness, or worsening confusion over the past 24 hours? T13 PATIENTNo, I haven’t experienced any double vision, trouble walking, a stiff neck, or confusion recently. My symptoms have just been the headaches and those brief episodes where my vision temporarily dims or gets blurry. T14 DOCTORREQUEST TEST: NEJM_Medical_Image T15 MEASUREMENT Image input to the Measurement Agent (no MedSAM) Panel 0: fundus photograph Panel 1: orbital MRI Panel 2: coronal head MRI No segmentation mask or overlay is produced in this condition. •Report: 13 WSDM ’27, February 15–19, 2027, Hong KongWu et al. –Task Focus:diagnosis –Panel Findings: ∗panel index: 0; modality: Fundus photography; visible findings: Color fundus image of the retina showing the optic disc on the left with distinct margins. The retinal vasculature is visible radiating from the disc. A bright, linear reflection artifact is present centrally over the macula. There are no obvious large hemorrhages or exudates in the visible field.; confidence: high ∗panel index: 1; modality: MRI (Orbital); visible findings: Axial MRI slice of the orbits. Both globes are visualized. White arrows point to focal, hyperintense lesions located at the posterior pole of both eyes, within the vitreous cavity adjacent to the retina. These lesions appear as small, rounded nodules.; confidence: high ∗panel index: 2; modality: MRI (Brain); visible findings: Coronal T2-weighted MRI of the brain. The lateral ventricles are visible. An asterisk marks a region in the midline posteriorly, likely involving the pineal gland or quadrigeminal plate cistern area. The surrounding brain parenchyma shows normal sulcal and gyral patterns for the age.; confidence: high –Mask Derived Observations:None recorded. –Cross Panel Synthesis:The multimodal imaging reveals bilateral ocular findings characterized by focal, hyperintense nodules at the posterior poles of the globes on axial orbital MRI. Concurrently, the coronal brain MRI identifies a discrete lesion in the midline posterior region near the pineal gland. While the fundus photograph provides a view of the retinal surface, it does not clearly delineate the specific posterior wall nodules seen on the MRI sequences. –Limitations:This report is based on a raw-image control condition; no MedSAM mask was used, and segmentation was not performed. Visual assessment is limited to qualitative description of morphology and spatial localization without automated quan- titative metrics. –Segmentation Assessment:No segmentation was performed. •Medsam Enabled:False •Task Focus: –Answer Type:diagnosis –Measurement Guidance:The requested answer type is a diagnosis; provide discriminative visible morphology but leave disease naming to the Doctor. •Provided Exam Context: –Focused Examination:Bilateral optic disk swelling and retinal hemorrhages noted. •Measurement Skill Ids: –vm_evolved_g379_09bc12e9013 –vm_evolved_g341_6a22b2bd3831 –vm_evolved_g96_58af12a19760 •Segmentation Results:None recorded. •Limitations:No MedSAM segmentation was used in this control condition. T16 DOCTOR •Diagnosis:Secondary Intracranial Hypertension (suspected pineal region mass) •Differential Diagnoses: –Cerebral Venous Sinus Thrombosis –Idiopathic Intracranial Hypertension –Meningitis/Encephalitis •Key Evidence: –25-year-old obese female –Gradual onset headache x 1 week with transient visual obscurations and persistent blurred vision –Bilateral optic disk swelling and retinal hemorrhages on focused examination –Head MRI showing flattened posterior globes and empty sella sign –NEJM Medical Image revealing discrete midline posterior lesion near pineal gland and bilateral posterior pole ocular nodules •Tests Used: –Head_MRI –NEJM_Medical_Image •Treatment Prescription Plan: –Disposition: Admit to hospital or place in ED observation unit for continuous neurological monitoring and urgent specialist eval- uation. 14 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong –Consultations: Immediate Neurosurgery and Ophthalmology consultation for management of suspected elevated ICP and posterior midline lesion. –Procedures: Defer lumbar puncture until contrast MRI definitively excludes mass effect or hydrocephalus and neurosurgery clears the procedure. –Medications: Acetazolamide 500 mg PO twice daily - DO NOT START UNTIL VERIFIED (requires confirmed negative urine beta- hCG and normal Basic Metabolic Panel). –Medications: Topiramate 25 mg PO nightly - DO NOT START UNTIL VERIFIED (requires confirmed negative urine beta-hCG and renal function assessment). –Non-pharmacologic management: Strict bed rest with head elevation at 30 degrees, dietary sodium restriction (<2g/day), and avoidance of NSAIDs or empiric anticoagulants pending hemorrhage/CVST exclusion. –Monitoring: Serial automated visual field testing and neurological checks every 4 hours; monitor serum potassium, bicarbonate, and creatinine monthly if acetazolamide is eventually initiated. –Obtain immediate blood pressure measurement to definitively exclude hypertensive emergency given papilledema and headache. •Safety Checks: –Pregnancy status explicitly unknown: Obtain urine beta-hCG immediately prior to any pharmacotherapy, especially teratogenic agents like topiramate or acetazolamide. –Allergies: None documented. –Renal/Hepatic: Verify Basic Metabolic Panel before initiating acetazolamide; avoid in severe renal impairment. –Contraindications: Avoid tetracycline antibiotics and high-dose vitamin A supplements which can exacerbate intracranial hyper- tension. –Mass effect risk: Hold lumbar puncture until neuroimaging confirms absence of obstructive hydrocephalus or space-occupying lesion to prevent herniation. –Hemodynamic: Obtain immediate blood pressure measurement to rule out hypertensive emergency. •Follow Up Or Escalation:Return to ED immediately for sudden vision loss, severe refractory headache, projectile vomiting, or focal neurological deficits. Maintain inpatient admission or ED observation until Neurosurgery and Ophthalmology complete urgent evaluation, baseline labs (including hCG and BMP) are reviewed, and definitive imaging (Head_MRV) is obtained to rule out cerebral venous sinus thrombosis. Recorded outcome.Gold diagnosis:Idiopathic intracranial hypertension. The no-MedSAM prediction was scoredincorrect. S5.4 Remote condition: Measurement learning with MedSAM Retrieved memory context. •Acuity-Gated Cerebrovascular & Headache Risk Stratification with Baseline-First Fallback and Prophylaxis-Bundle Verification •Adaptive Concurrent Baseline & Empiric-Antimicrobial Neurological Workup Protocol Complete observable interaction.T1 ENVIRONMENT •Physical Examination: –Focused Examination:Bilateral optic disk swelling and retinal hemorrhages noted. •Available Test Names: –NEJM_Medical_Image –Head_MRI –Head_MRV •Required Test Names Before Final: –NEJM_Medical_Image •Completion Instruction:Request each required test with REQUEST TEST: <exact name> before returning DIAGNOSIS READY. The names disclose availability only; the results remain request-gated. T2 DOCTORPlease state medication and OTC use and allergies; alcohol amount/frequency; tobacco pack-years or vaping; recreational drugs; recent travel/exposures; relevant family history; and, if applicable, last menstrual period or pregnancy status. T3 PATIENTI don’t take any medications or have any allergies. I don’t know about my family history, recent travel, or my last menstrual period and pregnancy status. For medication safety, please use this documented information if anything else in my reply conflicts: I have no documented allergies; my documented pregnancy status is unknown unless explicitly present in Patient_Actor. 15 WSDM ’27, February 15–19, 2027, Hong KongWu et al. T4 DOCTORCould you describe any headaches you are experiencing, particularly their onset, severity, and whether they worsen when lying down or with coughing? Have you noticed any sudden vision changes, double vision, or nausea? T5 PATIENTMy headaches have been going on for about a week and feel quite severe, particularly when I lie down or bend over. I’ve also experienced brief moments where my vision suddenly grays out or blacks out, alongside persistent blurriness. I haven’t noticed any double vision or nausea. T6 DOCTORREQUEST TEST: Head_MRI T7 MEASUREMENTRESULTS: Flattened posterior globes and an empty sella observed. T8 DOCTORREQUEST TEST: Head_MRV T9 MEASUREMENTRESULTS: Transverse sinus stenoses present without evidence of obstruction or thrombosis. T10 DOCTORCould you please specify your alcohol consumption frequency and amount, any tobacco use or vaping history, and whether you take any recreational substances? Additionally, what is your current height and weight? T11 PATIENTI don’t drink alcohol, use tobacco or vaping products, or take any recreational substances. I don’t have my exact height and weight recorded, but I do know that I have severe obesity. T12 DOCTORCould you please confirm your current pregnancy status and provide your exact height and weight to calculate your BMI? T13 PATIENTI don’t know my current pregnancy status, and I don’t have my exact height and weight recorded. I can only tell you that I have severe obesity. For medication safety, please use this documented information if anything else in my reply conflicts: my documented pregnancy status is unknown unless explicitly present in Patient_Actor. T14 DOCTORREQUEST TEST: NEJM_Medical_Image 16 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong T15 MEASUREMENT Image input to the Measurement Agent Panel 0: fundus photograph Panel 1: orbital MRI Panel 2: coronal head MRI MedSAM segmentation overlays Panel 1, ROI 0Panel 1, ROI 1 Panel 2, ROI 0 •Report: –Task Focus:diagnosis –Panel Findings: ∗panel index: 0; modality: Fundus Photography; visible findings: Color fundus photograph of the retina. The optic disc is visible on the left with distinct margins. Retinal vessels radiate from the disc. A bright, linear reflection artifact is present centrally over the macula. There are no obvious large hemorrhages or exudates in this view.; confidence: 0.95; roi boxes: ∗panel index: 1; modality: MRI (Orbits); visible findings: Axial MRI scan of the orbits showing both globes. White arrows point to hyperintense foci located at the posterior aspect of the globe, adjacent to the optic nerve head region bilaterally. These appear as small, rounded areas of high signal intensity within the vitreous cavity near the retinal surface.; confidence: 0.98; roi boxes: 420; 380; 460; 420; 720; 380; 760; 420 ∗panel index: 2; modality: MRI (Brain); visible findings: Coronal T2-weighted MRI of the brain. An asterisk marks a focal area of abnormality at the inferior aspect of the midline, likely involving the cerebellar vermis or fourth ventricle region. The lesion appears hypointense relative to the surrounding CSF and brain parenchyma.; confidence: 0.95; roi boxes: 480; 850; 520; 890 –Mask Derived Observations: Implementation note:solidity divides integer selected-pixel area by OpenCV’s continuous contour-hull area. For very small masks these discrete/continuous conventions can produce values slightly above one; solidity is an auxiliary report field and is not used for case selection, diagnosis, or scoring. ∗panel index: 1; modality: MRI (Orbits); box xyxy: 277; 157; 304; 173; quantitative features: valid: True; area ratio: 0.000402; prompt box coverage: 0.25463; component count: 1; largest component fraction: 1.0; centroid normalized xy: 0.4285; 0.4038; bbox normalized xyxy: 0.4221; 0.3889; 0.4387; 0.4203; bbox extent: 0.769231; circularity: 0.921625; solidity: 1.128205; inside gray mean: 17 WSDM ’27, February 15–19, 2027, Hong KongWu et al. 80.464; surrounding ring gray mean: 75.806; inside minus ring gray: 4.658; edge fraction inside: 0.0; touches image boundary: False; touches prompt boundary: True ∗panel index: 1; modality: MRI (Orbits); box xyxy: 475; 157; 502; 173; quantitative features: valid: True; area ratio: 0.00027; prompt box coverage: 0.171296; component count: 1; largest component fraction: 1.0; centroid normalized xy: 0.746; 0.403; bbox normal- ized xyxy: 0.7352; 0.3961; 0.7564; 0.4155; bbox extent: 0.660714; circularity: 0.570172; solidity: 1.049645; inside gray mean: 82.041; surrounding ring gray mean: 83.578; inside minus ring gray: -1.537; edge fraction inside: 0.0; touches image boundary: False; touches prompt boundary: True ∗panel index: 2; modality: MRI (Brain); box xyxy: 201; 351; 218; 368; quantitative features: valid: True; area ratio: 0.000656; prompt box coverage: 0.394464; component count: 1; largest component fraction: 1.0; centroid normalized xy: 0.4967; 0.8671; bbox nor- malized xyxy: 0.4833; 0.8551; 0.5119; 0.8841; bbox extent: 0.791667; circularity: 0.984702; solidity: 1.123153; inside gray mean: 183.833; surrounding ring gray mean: 181.55; inside minus ring gray: 2.283; edge fraction inside: 0.192982; touches image bound- ary: False; touches prompt boundary: True –Cross Panel Synthesis:The fundus photography shows a relatively normal-appearing retina with a central reflection artifact, lacking obvious large hemorrhages or exudates despite the clinical note of swelling. The orbital MRI reveals bilateral, symmetric, hyperintense foci at the posterior pole of the globes, corresponding anatomically to the optic nerve head region. The brain MRI demonstrates a focal hypointense lesion in the midline inferiorly, potentially involving the cerebellar vermis or fourth ventricle. The combination of bilateral posterior eye findings and a midline brain lesion suggests a systemic or syndromic process affecting these specific anatomical regions. –Limitations:The segmentation masks for the orbital MRI lesions show high circularity but low prompt box coverage, indicating they capture the core of the hyperintense foci but may not fully delineate their irregular boundaries against the vitreous. The brain lesion mask has a higher coverage ratio but its exact anatomical relationship to the fourth ventricle vs. vermis is difficult to confirm without sagittal/coronal multi-planar reconstruction. The fundus image lacks color contrast resolution to definitively rule out subtle micro-hemorrhages or exudates mentioned in the clinical history. –Segmentation Assessment:Masks serve only as approximate anatomical localization aids, not precise pathological quantifiers. Cross-verification against raw pixels confirms the orbital masks align with the hyperintense foci pointed out by arrows. The brain mask aligns with the asterisk-marked region. However, automated masks cannot inherently differentiate subtle pathological textures from normal background tissue; final morphological assessments rely on direct visual inspection of raw images. •Medsam Enabled:True •Task Focus: –Answer Type:diagnosis –Measurement Guidance:The requested answer type is a diagnosis; provide discriminative visible morphology but leave disease naming to the Doctor. •Provided Exam Context: –Focused Examination:Bilateral optic disk swelling and retinal hemorrhages noted. •Measurement Skill Ids: –vm_evolved_g38_48bfeb9bde45 –vm_evolved_g22_71611f3e24ce –vm_evolved_g225_2b3401c40f64 •Segmentation Results: –∗Panel Index:1 ∗Modality:MRI (Orbits) ∗Box Xyxy:277; 157; 304; 173 ∗Mask Area Ratio:0.000402 ∗Mask Nonempty:True ∗Quantitative Features:valid: True; area ratio: 0.000402; prompt box coverage: 0.25463; component count: 1; largest component fraction: 1.0; centroid normalized xy: 0.4285; 0.4038; bbox normalized xyxy: 0.4221; 0.3889; 0.4387; 0.4203; bbox extent: 0.769231; circularity: 0.921625; solidity: 1.128205; inside gray mean: 80.464; surrounding ring gray mean: 75.806; inside minus ring gray: 4.658; edge fraction inside: 0.0; touches image boundary: False; touches prompt boundary: True –∗Panel Index: 1 ∗Modality:MRI (Orbits) ∗Box Xyxy:475; 157; 502; 173 ∗Mask Area Ratio:0.00027 ∗Mask Nonempty:True 18 MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical InteractionWSDM ’27, February 15–19, 2027, Hong Kong ∗Quantitative Features:valid: True; area ratio: 0.00027; prompt box coverage: 0.171296; component count: 1; largest component fraction: 1.0; centroid normalized xy: 0.746; 0.403; bbox normalized xyxy: 0.7352; 0.3961; 0.7564; 0.4155; bbox extent: 0.660714; circularity: 0.570172; solidity: 1.049645; inside gray mean: 82.041; surrounding ring gray mean: 83.578; inside minus ring gray: -1.537; edge fraction inside: 0.0; touches image boundary: False; touches prompt boundary: True –∗Panel Index:2 ∗Modality:MRI (Brain) ∗Box Xyxy:201; 351; 218; 368 ∗Mask Area Ratio:0.000656 ∗Mask Nonempty:True ∗QuantitativeFeatures:valid: True; area ratio: 0.000656; prompt box coverage: 0.394464; component count: 1; largest component fraction: 1.0; centroid normalized xy: 0.4967; 0.8671; bbox normalized xyxy: 0.4833; 0.8551; 0.5119; 0.8841; bbox extent: 0.791667; circularity: 0.984702; solidity: 1.123153; inside gray mean: 183.833; surrounding ring gray mean: 181.55; inside minus ring gray: 2.283; edge fraction inside: 0.192982; touches image boundary: False; touches prompt boundary: True •Limitations:MedSAM masks are prompt-conditioned localization aids; preserve the raw-image findings. T16 DOCTOR •Diagnosis:Idiopathic Intracranial Hypertension (IIH) •Differential Diagnoses: –Posterior Fossa Mass Lesion with Obstructive Hydrocephalus –Cerebral Venous Sinus Thrombosis –Meningitis/Infectious Etiology –Malignant Hypertension •Key Evidence: –Severe obesity –Headaches worsening with recumbency and bending over (positional/Valsalva exacerbation) –Transient visual obscurations and persistent blurriness –Initial focused examination showing bilateral optic disk swelling and retinal hemorrhages –MRI demonstrating flattened posterior globes, empty sella, and transverse sinus stenoses without thrombosis –MRI revealing a focal hypointense midline inferior lesion near the cerebellar vermis/fourth ventricle •Tests Used: –Head_MRI –Head_MRV –NEJM_Medical_Image •Treatment Prescription Plan: –Urgent Neurosurgery consultation and recommendation for hospital admission for continuous neurologic monitoring and evalu- ation of the posterior fossa lesion and elevated ICP. –Ophthalmology referral for formal visual field testing and standardized fundus documentation to reconcile initial exam findings with imaging. –DO NOT START UNTIL VERIFIED: Acetazolamide 500 mg orally twice daily (titrating to 1000 mg twice daily) to reduce CSF production; withhold pending confirmed non-pregnancy status and baseline BMP/LFTs. –Non-drug management: Head elevation to 30 degrees, structured weight reduction program targeting 5-10% body weight loss, dietary sodium restriction (<2g/day), strict avoidance of Vitamin A supplements and tetracycline-class antibiotics. –Obtain mandatory baseline laboratories immediately: CBC, comprehensive metabolic panel (BMP), coagulation profile, liver func- tion tests, and quantitative serum/urine pregnancy test prior to any pharmacotherapy. –Strict avoidance of NSAIDs, anticoagulants, and antiplatelet agents until hemorrhage risk and mass effect are fully characterized. •Safety Checks: –Pregnancy status is currently unknown; acetazolamide carries teratogenic risk and must be withheld until a quantitative serum/urine pregnancy test confirms non-pregnancy. –Renal and hepatic baselines are unverified; acetazolamide requires renal clearance and may precipitate metabolic acidosis or elec- trolyte depletion. Baseline BMP/LFTs are mandatory before first dose. –No known drug allergies documented; explicit verification required prior to dispensing. –Posterior fossa lesion and signs of elevated ICP require urgent neurosurgical evaluation to rule out obstructive hydrocephalus or herniation risk. 19 WSDM ’27, February 15–19, 2027, Hong KongWu et al. •Follow Up Or Escalation:Admit to a monitored neurological unit for frequent neuro checks and urgent specialist consultations; return to Emergency Department immediately for sudden permanent vision loss, thunderclap headache, projectile vomiting, or altered mental status. Recorded outcome.Gold diagnosis:Idiopathic intracranial hypertension. The MedSAM prediction was scoredcorrectby the auto- matic diagnosis metric; this label does not endorse the visual evidence or management plan. Interpretation.This pair is a qualitative failure-boundary illustration rather than positive clinical evidence or a causal estimate. In both paths, source annotations are overinterpreted as pathology; in the MedSAM path, an unverified posterior-fossa finding propagates into the differential and escalation plan despite the correct final label. The diagnosis-blind release checks enforce provenance, prerequisites, and conservative disposition but cannot determine whether an image interpretation is clinically true. This trace therefore exposes a limitation of the current governance stack and motivates independent image adjudication; it must not be read as evidence that MedSAM improved this case. 20