Paper deep dive
PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs
Jinyue Li, Yuci Liang, Qiankun Li, Xinheng Lyu, Jiayu Qian, Huabao Chen, Kun Wang, Zhigang Zeng, Anil Anthony Bharath, Yang Liu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 1:06:44 AM
Summary
PathMem is a memory-centric multimodal framework for pathology MLLMs that integrates structured domain knowledge via a long-term memory (LTM) knowledge graph and a Memory Transformer. It dynamically transforms LTM into working memory (WM) to improve diagnostic reasoning, achieving state-of-the-art performance on WSI-Bench report generation and open-ended diagnosis.
Entities (5)
Relation Signals (3)
PathMem → evaluatedon → WSI-Bench
confidence 100% · PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation
PathMem → utilizes → Memory Transformer
confidence 98% · PathMem... introduces a Memory Transformer that models the dynamic transition from LTM to working memory
PathMem → constructsltmfrom → PubMed
confidence 95% · We first construct a high-quality pathology knowledge graph as LTM via deep semantic search over PubMed.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multimodal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation (12.8% WSI-Precision, 10.1% WSI-Relevance) and open-ended diagnosis by 9.7% and 8.9% over prior WSI-based models.
Tags
Links
- Source: https://arxiv.org/abs/2603.09943v1
- Canonical: https://arxiv.org/abs/2603.09943v1
Trouble viewing inline? Open PDF directly →
Full Text
57,477 characters extracted from source content.
Expand or collapse full text
PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs Jinyue Li 1,∗ , Yuci Liang 2,∗ , Qiankun Li 3† , Xinheng Lyu 2 , Jiayu Qian 3 , Huabao Chen 1 , Kun Wang 3 , Zhigang Zeng 4 , Anil Anthony Bharath 5 , and Yang Liu 3 1 University of Science and Technology of China 2 Shenzhen University 3 Nanyang Technological University 4 Huazhong University of Science and Technology 5 Imperial College London, Imperial Global Singapore Abstract. Computational pathology demands both visual pattern recog- nition and dynamic integration of structured domain knowledge, includ- ing taxonomy, grading criteria, and clinical evidence. In practice, di- agnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision–language reasoning capa- bilities, they lack explicit mechanisms for structured knowledge inte- gration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic stan- dards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multi- modal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory re- finement for downstream reasoning. PathMem achieves SOTA per- formance across benchmarks, improving WSI-Bench report generation (+12.8% WSI-Precision, +10.1% WSI-Relevance) and open-ended di- agnosis by +9.7% and +8.9% over prior WSI-based models. Keywords: Meta-Cognition Memory· Computational Pathology· MLLM 1 Introduction Computational pathology is a knowledge-intensive and cognitively complex dis- cipline [1, 12, 30, 34, 36, 42]. Accurate diagnosis requires not only visual pattern recognition from histopathological images but also the dynamic integration of long-term expert knowledge [45], including disease taxonomy [38], grading cri- teria [27], and evolving clinical evidence [15]. In practice, diagnostic reasoning requires linking morphological observations with formal diagnostic and grad- ing standards. Although recent multimodal large language models (MLLMs) [5, arXiv:2603.09943v1 [cs.AI] 10 Mar 2026 2Li et al. Green: matched ground-truth content; Red: deviations; Blue: Correct information from long-term memory; Underlined: template language. Given the Nottingham grading system mentioned, what is the grading of this carcinoma? BRCA/TCGA-C8-A12Y PathMem(Ours): The carcinoma is classified as Nottingham Grade I, with a total score of 6 out of 9. This grading reflects a tubule formation score of less than 10%, moderate nuclear pleomorphism with a score of approximately 2, and a mitotic count score of about 2. This intermediate grade indicates a moderately differentiated tumor, suggesting a moderate level of malignancy and potential for aggressiveness. GPT-4o: Grade I Quilt-LLaVA: Grade I WSI-LLaVA: This carcinoma is graded as Nottingham Histologic Grade 2, with a total score of 6 out of 9. This intermediate rating suggests that it achieves a balance between invasive and non-invasive characteristics. Long-term memory is activated as working memory Got it ! Text Visual retrieve Working memory -embeddings activation Working memory Awaken my memory ！ Fig. 1: Overview of PathMem. WM activates relevant LTM and transforms them into an updated WM for interpretable reasoning. 31,47] have demonstrated promising vision-language reasoning capabilities, they largely operate as parametric black boxes and lack explicit mechanisms for struc- tured knowledge integration and interpretable memory control [32]. Therefore, incorporating structured, controllable, and interpretable knowledge integration mechanisms on top of multimodal reasoning capabilities has become a critical step toward achieving clinically reliable and professionally robust intelligent sys- tems in computational pathology. In clinical practice, pathologists rely on a hierarchical memory process [6,19, 21,33,43]: accumulated domain knowledge forms long-term memory (LTM) [4], while case-specific evidence selectively activates relevant knowledge into working memory (WM) for reasoning [3], and iteratively refine their hypotheses. Existing retrieval-augmented generation (RAG) methods attempt to introduce external knowledge [17,25,36,49], but their retrieval pipelines are typically static and fail to model the dynamic and adaptive nature of human memory transformation. This cognitive process, characterized by memory retrieval, selective activation, and meta-cognitive regulation, is largely absent in current MLLMs [2,14,20,22, 26,48]. In this paper, we propose a memory-centric multimodal framework for pathol- ogy MLLMs (Fig. 1). We first construct a high-quality pathology knowledge graph as LTM via deep semantic search over PubMed. The knowledge graph is encoded into semantic embeddings aligned with the multimodal backbone. We then design a Memory Transformer that explicitly models the transition from LTM to WM. Given visual and textual embeddings, we perform: Static activa- tion, which ranks knowledge entries via cosine similarity; Dynamic activation, which jointly projects multimodal and knowledge embeddings to compute global relevance. An adaptive selection strategy determines the boundary of activated knowledge, transferring only highly relevant entries into WM for final reason- PathMem3 ing. This mechanism enables controlled and interpretable memory enhancement during inference. The contributions of this paper are summarized as follows. ❶ High-Quality LTM Construction: We build a structured pathology knowl- edge graph via deep search over PubMed, providing a scalable and updatable repository that simulates accumulated expert-level domain knowledge. ❷ Memory-Driven Pathology MLLM Architecture: We introduce an ex- plicit long-term/working memory paradigm into multimodal pathology mod- eling, enabling cognitively grounded and knowledge-aware reasoning beyond purely parametric inference. ❸ Dynamic–Static Memory Controller with Adaptive Activation: We propose a dual-mode memory activation mechanism with self-adaptive selec- tion, explicitly modeling the transformation from LTM to WM for context- aware, interpretable, and enhanced inference. ❹ PathMem achieves the SOTA performance across multiple Bench- marks: In report generation, it improves WSI-Precision by 12.8% and WSI- Relevance by 10.1%. In open-ended diagnosis, it further yields gains of 9.5% and 8.9%, respectively. By replacing static retrieval with explicit memory transforma- tion, our framework enables interpretable, memory-aware reasoning aligned with human cognition in pathology MLLMs. 2 Related Work 2.1 Pathology MLLM With the rapid development of computational pathology, the availability of large- scale whole slide images (WSIs) paired with diagnostic reports has catalyzed the rise of pathology-oriented MLLMs [1, 13, 46]. By pretraining on hundreds of thousands of WSIs and billions of tiles, recent foundation models encode rich slide-level morphological and clinical semantics, enabling transfer to diverse downstream tasks such as cancer subtyping, prognosis prediction, cross-modal re- trieval, and report generation. For instance, Prov-GigaPath [46] and TITAN [13] emphasize large-scale whole-slide modeling and vision–language alignment, while PRISM [39] and PathAlign [1] demonstrate the effectiveness of language-aligned slide embeddings for zero-shot and generative applications. Building upon these slide-level foundations, recent pathology MLLMs further integrate instruction tuning, task unification, and computational efficiency into a cohesive framework. PathAsst [44] and CPath-Omni [42] construct unified multimodal assistants that consolidate patch- and slide-level tasks under large-scale image–text supervision. SlideChat [10] and WSI-LLaVA [30] enhance morphology-aware reasoning and conversational capabilities through dedicated WSI-benchmarks and instruction 4Li et al. data. To address the prohibitive cost of gigapixel modeling, LoC-Path [21] in- troduces redundancy-aware compression for efficient slide representation, while ChatEXAONEPath [29] strengthens expert-level clinical reasoning via retrieval- based data construction. In summary, while prior pathology MLLMs achieve strong slide-level un- derstanding through large-scale pretraining and alignment, they lack explicit mechanisms to balance long-term knowledge and short-term contextual mem- ory, limiting stable and consistent clinical reasoning. 2.2 Knowledge in pathology Pathology requires integrating morphological patterns, molecular alterations, and clinical context, WSIs introduce substantial scale and heterogeneity. Conse- quently, explicit domain knowledge is essential for robust computational pathol- ogy [23,28,35,41,50]. Recent pathology foundation models aim to learn transfer- able visual representations, yet purely data-driven approaches often suffer from limited task generalization and domain robustness. To mitigate this, knowledge has been incorporated as an inductive bias during pretraining and adaptation. Unified knowledge distillation aggregates expert and self-knowledge to enhance cross-task generalization [35], while knowledge-guided adaptation leverages vi- sual and textual priors to improve cross-domain robustness and demographic fairness [23]. These works highlight the role of knowledge in stabilizing and gen- eralizing foundation models. Beyond improving robustness, another line of work aligns WSIs with explicit expert concepts. Knowledge concept-based MIL frame- works connect slide representations with disease-specific linguistic concepts de- rived from medical literature [50], enabling semantically grounded modeling. In multimodal prognosis prediction, prior and textual knowledge guide phenotype extraction and genotype interaction [18,40], reducing modality gaps and enhanc- ing clinically meaningful integration. Recently, knowledge has been injected into pathology vision-language models to promote structured reasoning. By align- ing WSIs with medical knowledge graphs and optimizing reasoning trajectories, knowledge-guided policy learning enables evidence-linked, chain-of-thought di- agnosis [28], improving interpretability beyond outcome-level prediction. Despite progress, most knowledge-enhanced methods rely on implicit supervi- sion or proprietary RAG pipelines, limiting reproducibility and long-term knowl- edge utilization. ♣ Different from prior knowledge-injection strategies that treat knowledge as auxiliary supervision or external retrieval, we approach en- hancement from a memory-aware perspective by constructing a high-quality pathology knowledge base and integrating it as structured LTM within pathology MLLMs, enabling persistent, explicit, and interpretable reasoning. 3 Method 3.1 Unified Model Interface Problem Formulation. We formulate LTM construction as a literature-grounded knowledge aggregation problem. Given a domain-specific query q 0 , we construct PathMem5 Knowledge AcquisitionDeepSearch-based KG Construction GPT-4o 2、Information Extraction 3、Filtering low confidence Long-term Memory (KG) confidence>0.8 ? pubmed_search(PubMed) LLM-driven KG construction molecular expression molecular expression Morphological features Primary-metastatic relationship Long-term memory provides valuable structured expertise, we use KG to represent. example1: Molecules → Progression → Transfer lung cancer → HAS_EXPRESSION → TRPA1 mRNA upregulated → HAS_DIAGNOSTIC_CLUE → Tumor stage ↑ & metastasis risk ↑ → METASTASIS_TO → brain example2: IHC marker → Prognosis → Treatment response lung squamous cell carcinoma → HAS_IHC_MARKER → mPRα aberrantly overexpressed → HAS_DIAGNOSTIC_CLUE → Poor overall survival → HAS_DIAGNOSTIC_CLUE → Increased therapeutic resistance 1、Scraping Summary Literature retrieved from PubMed Fig. 2: LTM construction pipeline for pathology knowledge graphs via iterative liter- ature retrieval and LLM-based information extraction. a weighted directed multigraph G encoding disease–feature–evidence relations extracted from biomedical abstracts. Formally, the resulting LTM is defined as G = (V,R,E,W,Φ,Ψ ),(1) whereV denotes the global entity set,R denotes the predefined relation schema, E ⊆ V ×R×V denotes the set of directed relational triples, W : E → [0, 1] assigns calibrated confidence weights to edges, Φ denotes the entity normaliza- tion mapping, and Ψ denotes the feature-based inverted index for retrieval. The objective is to maximize structured knowledge coverage while controlling redun- dancy and uncertainty propagation. 3.2 LTM Construction As illustrated in Fig. 2, we construct the LTM through an iterative pipeline that integrates literature retrieval with LLM-based information extraction to progressively build and refine the pathology knowledge graph. ♣ Hash-Based Deduplication and Memory Monotonicity. To guarantee monotonic growth of memory without redundancy, we define a deterministic hashing operatorH(·) and compute h i = H (Normalize(a i )), where Normalize(·) removes formatting noise, punctuation variation, and case differences before hashing. Let M (t) h de- note the set of previously observed hash values at iteration t. The deduplicated abstract set is then defined as A ∗ = n a i h i /∈M (t) h o , M (t+1) h =M (t) h ∪h i , (2) here h i is the hash signature of abstract a i , M (t) h is the memory of previously seen abstracts, andA ∗ is the effective corpus after deduplication. This guaran- tees that identical textual evidence does not repeatedly influence the long-term 6Li et al. graph. ❥ LLM-Based Structured Knowledge Extraction. Each dedupli- cated abstract a i ∈ A ∗ is processed by a large language model extractor f θ parameterized by θ: T i = f θ (a i ) = (s ij ,r ij ,o ij ,c ij ,z ij ) K i j=1 , here K i is the number of candidate triples extracted from abstract a i , s ij ∈ V is the subject entity, r ij ∈R is the relation type, o ij ∈V is the object entity, c ij ∈ [0, 1] is the model-estimated confidence score, and z ij ∈R d is the latent semantic embed- ding of the triple, capturing contextual representation for downstream similarity consolidation. ✤ Confidence-Constrained Triple Filtering. To ensure high precision in LTM, we introduce a minimum confidence threshold τ. The retained triple set from abstract a i is defined as T τ i =(s ij ,r ij ,o ij ,c ij ,z ij )∈T i | c ij ≥ τ, (3) here τ ∈ (0, 1) controls the trade-off between recall and precision. Only triples whose estimated confidence exceeds τ are injected into the knowledge graph. ♠ Probabilistic Multi-Evidence Fusion. Since identical semantic triples may appear across multiple abstracts, we perform probabilistic evidence aggre- gation. For a canonical triple (s,r,o) appearing in m independent sources, with confidence scores c (1) ,c (2) ,...,c (m) , the aggregated edge weight is defined as W (s,r,o) = 1− m Y k=1 1− α· c (k) · exp −F z (k) − ̄ z 2 2 , (4) where α ∈ (0, 1] is a global scaling coefficient, c (k) is the confidence score from the k-th abstract, z (k) is the embedding representation of the k-th evidence instance, ̄ z is the centroid embedding of all supporting instances, ∥ · ∥ 2 de- notes Euclidean norm, andF > 0 controls embedding-consistency penalization. This formulation follows a noisy-or probabilistic fusion mechanism, where multi- source corroboration increases edge certainty, while embedding inconsistency reduces effective contribution. ♦ Feature-Oriented Memory Indexing. To support efficient retrieval during reasoning, we construct a feature-inverted index Ψ (f ) = (s,r,o)∈E | f = s ∨ f = o, where f ∈V denotes a histopathologi- cal feature entity, and Ψ (f ) returns all triples involving that feature. Addition- ally, disease nodes are defined asV disease =v ∈V | Type(v) = Disease. These memory indices enable subgraph extraction from microscopic features. 3.3 Memory Transformer: Knowledge-Aware LTM→WM Transition As illustrated in Fig. 3, the Memory Transformer serves as the core mod- ule that bridges the LTM and WM through structured embedding interactions. ♣ LTM Embedding Space. Let the constructed knowledge graph define a structured LTM space represented by an embedding bank Q = m i ∈R d i = 1,...,N , Q∈R N×d ,(5) here N denotes the total number of knowledge entities stored in LTM, d is the shared embedding dimensionality with the multimodal backbone, and each PathMem7 Human-like working memory framework Provide a concise report on the histopathological features and the diagnosis of the whole slide image？ Visual - Encoder ( Prov - GigaPath ) Visual - Encoder ( Prov - GigaPath )Text-EncoderText-Encoder Long-Term Memory Text Encoder Text Decoder Top - K tokens Static activation Δ 1 Δ 2 Δ 3 Δ 4 Δ 5 neighborhood differences ActivateInactive Dynamic activation Cosine similarity MLP Memory Transformer Current Working Memory Memory Transformer Long-Term Memory Provide a concise report on the histopathological features and the diagnosis of the whole slide image. ID 527：Pulmonary basaloid squamous cell carcinoma： → HAS_ARCHITECTURE → solid basaloid nests → Morphologic inference → Predominantly solid architecture observed → Diagnostic implication → Poor differentiation pattern supported ID 526：Pulmonary basaloid squamous cell carcinoma → HAS_GRADE → high-grade → Semantic equivalence → High-grade ≈ poorly differentiated → Diagnostic inference → Poorly differentiated carcinoma ... ... ... ... Microscopic observation of the pathology slide reveals a poorly differentiated adenocarcinoma with a predominant solid growth pattern (ID: 638,582), comprising 90% of the tumor, and a small acinar component making up the remaining 10%. The tumor cells exhibit signet ring cell differentiation and are characterized by the presence of mucin-filled vacuoles displacing the nucleus to the periphery. There is no evidence of angiolymphatic invasion, and the tumoral stroma is notably fibrous and hyalinized. The surrounding lung tissue shows emphysematous changes and the presence, distal to the tumorous area, of a carcinoid tumorlet. Based on these observations, the diagnosis is poorly-differentiated adeno-carcinoma (ID: 526,527). Update Working Memory ... ... Cut off 2 5 3 1 4 , , , ...... Token selection – Top-K Update Working Memory Activation path Scores Ranking ... ......... Fig. 3: Framework of PathMem. A memory-augmented MLLMs for computational pathology that aligns visual, textual, and knowledge graph representations, and adap- tively activates LTM for knowledge-grounded reasoning about pathology. m i encodes a graph-aware knowledge representation learned from literature- grounded triples. The memory space Q is fixed during inference and serves as an external, persistent knowledge reservoir. Given an input multimodal sequence X∈R T×d , we compute a normalized query representation q = ̄ x t (∥ ̄ x t ∥ 2 + ε) −1 , here x t denotes the t-th token embedding,∥·∥ 2 is the Euclidean norm, and ε is a small constant for numerical stability. The vector q∈R d summarizes the global semantic context of the current input. ❥ Structured Memory Selection. We define a relevance operator over the LTM space: J = Softmax Qq √ d +N , J ∈R N .(6) In this formulation, Qq denotes the similarity between all memory tokens and the query, 1 √ d is the scaled dot-product normalization factor, andN ∈R N is a structural masking vector that suppresses previously selected indices or enforces diversity constraints. The resulting vectorJ represents a probability distribution over the entire memory bank, encoding query-conditioned relevance scores. To obtain a sparse WM, we apply top-k selection: I = TopK (J,k), Q work =J I ⊙Q I ,(7) 8Li et al. here k ≪ N ensures sparsity, I denotes the selected indices, Q I ∈R k×d is the retrieved subset, and ⊙ denotes row-wise scaling by relevance weights. This operation converts dense LTM into a compact WM block. ♠ Memory- Augmented Transformer Input. The WM is then prepended to the original sequence: X ⋆ = Concat (Q work ,X)∈R (k+T)×d . The augmented sequence X ⋆ is fed into the Transformer encoder, allowing self-attention to jointly model long- term knowledge tokens and input-specific representations. Through this mech- anism, external structured memory is dynamically projected into the model’s space, enabling evidence-aware reasoning without expanding parameters. 4 Experiments 4.1 Experimental Setup Datasets Our model is trained and evaluated on WSI-Bench, and further validated on three external datasets in a zero-shot manner. Internal Dataset. We use WSI-Bench, a large-scale morphology-aware benchmark constructed from The Cancer Genome Atlas (TCGA). WSI-Bench contains 9,850 WSIs span- ning 30 cancer types and comprises 179,569 VQA pairs (approximately 180k), covering four pathological capabilities: morphological analysis, diagnosis, treat- ment planning, and report generation. The dataset includes both open-ended and close-ended (multiple-choice) questions. Following the original split, 9,642 WSIs (175,450 VQA pairs) are used for training, and 208 WSIs (4,119 VQA pairs) are reserved for testing. All evaluation metrics, including WSI-Precision, WSI- Relevance, accuracy, and conventional NLU metrics (e.g., BLEU and ROUGE), are kept identical to ensure direct comparability with the baseline version. Ex- ternal Datasets. To evaluate generalization ability, we conduct zero-shot vali- dation on three publicly available WSI-level benchmarks: (1) WSI-VQA [8], (2) SlideBench-VQA (BCNB) [10], and (3) CPTAC-NSCLC. Following the prepro- cessing protocol in the original study, we exclude clinically unreasonable or non- WSI-observable questions (e.g., survival time prediction, immunohistochemical markers when unavailable, and 3D tumor size estimation). No additional fine- tuning is performed on these external datasets. Implementation Details. WSI Encoder. We adopt a dual-branch visual encoder to capture both local and global contextual information of gigapixel WSIs. For patch-level representation, 256×256 image tiles are extracted at a fixed magnification and encoded using a DINOv2-pretrained tile-level encoder. For slide-level modeling, aggregated patch embeddings are processed by a LongNet- based transformer to capture long-range dependencies across the entire slide. The resulting visual features are projected into the language embedding space via a learnable linear projection layer. Language Model. We employ a pretrained large language model (LLM) as the text backbone. The visual features are aligned with textual embeddings and injected into the LLM through a projection module. During training, different PathMem9 components are selectively frozen depending on the training stage, following a staged optimization strategy. Three-Stage Training Strategy. Consistent with the original framework, training is conducted in three stages: 1. WSI-Text Alignment: Contrastive learning is performed between slide- level visual embeddings and report-level textual embeddings to bridge the cross-modal gap. 2. Feature Space Alignment: The pretrained WSI encoder is connected to the LLM via a projection layer. The visual encoder and LLM are frozen, and only the projection layer is optimized. 3. Task-Specific Instruction Tuning: The projection layer and LLM are jointly optimized on WSI-Bench tasks, while keeping the WSI encoder frozen. Evaluation Metrics. We evaluate the model using both general language generation metrics and pathology-specific metrics. For open-ended report gen- eration tasks, we adopt BLEU, ROUGE-L, and METEOR to measure lexical overlap and semantic similarity between generated texts and ground-truth re- ports. To assess pathology-aware correctness and relevance, we further employ WSI-Precision and WSI-Relevance following WSI-LLaVA [30], which evaluate the accuracy of predicted pathological findings and their alignment with slide- level visual evidence, respectively. Training Settings. We train the model with DeepSpeed on 4 NVIDIA A100 GPUs using the AdamW optimizer. The initial learning rate is 2× 10 −4 with a cosine scheduler and a 0.03 warmup ratio, and the model is trained for 3 epochs. The per-device batch size is 32. For parameter- efficient fine-tuning, we use LoRA with r = 128 and α = 256. The multimodal projector uses a learning rate of 2× 10 −5 . 4.2 Main Result Quantitative analysis Obs. ❶: Consistent Performance Gains of PathMem Across Patho- logical Tasks. Tab. 1 presents the comparison on WSI-Bench. PathMem achieves the best overall average of 0.768, surpassing WSI-LLaVA at 0.754, Quilt LLaVA at 0.721, WSI-VQA at 0.590, and GPT-4o at 0.507, demonstrating su- perior comprehensive capability. In Morphological Analysis, PathMem attains the highest open ended precision and close ended accuracy at 0.535 and 0.972, with competitive relevance at 0.542. For Diagnosis, it leads in precision, rele- vance, and accuracy with 0.707, 0.701, and 0.868, outperforming all baselines. In Treatment Planning, PathMem achieves 0.796 precision, 0.792 relevance, and 1.000 accuracy, maintaining the best balance across metrics. Notably, thumbnail based models show clear degradation. GPT-4o yields only 0.220 precision and 0.471 accuracy in Morphological Analysis, highlighting the necessity of WSI level modeling. Overall, PathMem consistently improves open ended reasoning and achieves the strongest average performance across all pathological tasks. 10Li et al. Table 1: Quantitative evaluation of WSI and general-purpose MLLMs on our WSI- Bench across three pathological capabilities. WSI-P: WSI-Precision, WSI-R: WSI- Relevance, Acc: accuracy, open: open-ended question, and close: close-ended question. Morphological AnalysisDiagnosisTreatment Planning Average ModelInputOpenCloseOpenCloseOpenClose WSI-P WSI-RAccWSI-P WSI-R Acc WSI-P WSI-R Acc Quilt-LLaVA [37] Thumbnail 0.448 0.4470.9470.586 0.604 0.849 0.788 0.816 1.000 0.721 GPT-4o [24]Thumbnail 0.220 0.2040.4710.472 0.457 0.530 0.496 0.841 0.875 0.507 WSI-VQA [9]WSI0.395 0.4620.7580.436 0.525 0.577 0.791 0.595 0.771 0.590 WSI-LLaVA [30]WSI0.488 0.610 0.9510.610 0.612 0.863 0.810 0.845 1.000 0.754 PathMem (Ours)WSI0.535 0.5420.9720.707 0.701 0.868 0.796 0.792 1.000 0.768 Obs. ❷: Superior Report Generation Performance of PathMem. Tab. 2 reports the results for report generation. PathMem achieves the best perfor- mance across all metrics, with BLEU-1 to BLEU-4 scores of 0.548, 0.420, 0.347, and 0.302, consistently outperforming WSI-LLaVA (0.480, 0.358, 0.287, 0.240) and other baselines. In semantic evaluation, PathMem attains 0.536 ROUGE-L and 0.531 METEOR, surpassing WSI-LLaVA (0.490 / 0.465) and significantly exceeding GPT-4o (0.132 / 0.167), indicating stronger content coverage and alignment with reference reports. It also achieves the highest WSI-Precision and WSI-Relevance (0.508 / 0.530), compared with 0.380 / 0.429 for WSI-LLaVA, re- flecting improved slide-level grounding and clinical relevance. Overall, consistent gains across lexical, semantic, and WSI-specific metrics demonstrate the superi- ority of PathMem in generating accurate and pathology-coherent reports. Table 2: Quantitative evaluation of WSI and general-purpose MLLMs in report gen- eration task on WSI-Bench. ModelsBLEU-1BLEU-2BLEU-3BLEU-4ROUGE-LMETEOR WSI-Precision WSI-Relevance GPT-4o [24]0.2020.0690.0300.0160.1320.1670.0670.138 Quilt-LLaVA [37]0.4740.3510.2820.2360.4750.4600.3240.333 MI-Gen [7]0.4030.3060.2480.2090.4460.4070.3100.377 Hist-Gen [16]0.4060.3070.2480.2080.4480.4160.3000.397 WSI-LLaVA [30]0.4800.3580.2870.2400.4900.4650.3800.429 PathMem (Ours) 0.548± 0.151 0.420± 0.173 0.347± 0.188 0.302± 0.196 0.536± 0.155 0.531± 0.158 0.508± 0.147 0.530± 0.144 Obs. ❸: Strong Zero-Shot Generalization of PathMem Across Exter- nal Benchmarks. Tab. 3 presents the zero-shot external validation results. PathMem achieves the best performance on all benchmarks, demonstrating strong generalization across datasets and tasks. On WSI-VQA, PathMem at- tains 0.572, outperforming WSI-LLaVA at 0.546 and substantially exceeding Quilt-LLaVA at 0.132. On SlideBench VQA BCNB, PathMem achieves the highest scores in all categories, including Tumor at 0.918, Grade at 0.482, and Subtype at 0.313, yielding the best average of 0.571. In comparison, WSI-LLaVA obtains an average of 0.553, while WSI-VQA shows clear instability with an av- erage of 0.113 and zero performance on Subtype. On CPTAC NSCLC, Path- Mem reaches 0.754, surpassing WSI-LLaVA at 0.721 and markedly improving over Quilt-LLaVA at 0.603. Overall, consistent gains across external datasets confirm the robustness and cross domain generalization ability of PathMem. PathMem11 Table 3: Zero-shot external validation on WSI-VQA, SlideBench-VQA (BCNB), and CPTAC-NSCLC. ModelInput WSI-VQA SlideBench-VQA (BCNB) CPTAC Tumor Grade Subtype Avg Quilt-LLaVA [37] Thumb0.1320.113 0.194 0.181 0.163 0.603 WSI-VQA [9]WSI0.4690.039 0.300 0.000 0.113 0.000 SlideChat [11]WSI-0.902 0.231 0.175 0.436- WSI-LLaVA [30]WSI0.5460.905 0.463 0.292 0.553 0.721 PathMem (Ours) WSI0.5720.918 0.482 0.313 0.571 0.754 T-answer extract claims: Scoring results: claim 1: "Microscopic observation reveals sheets and nests of atypical squamous cells with abundant eosinophilic cytoplasm and significant nuclear atypia.", claim 2: "Nuclear atypia is characterized by increased nuclear size, irregular nuclear contours, and hyperchromasia.", claim 3: "The architecture predominantly lacks keratinization, aligning with a non-keratinizing histological variant.", claim 4: "Morphological assessment indicates poorly differentiated squamous cells with grade I differentiation.", claim 5: "Grade I differentiation is characterized by highly disorganized cell arrangements and pronounced pleomorphism.", claim 6: "There is moderate tumor necrosis indicating areas of cell death.", claim 7: "There is no evidence of lymphatic, venous, or arterial invasion by the tumor.", claim 8: "Non-neoplastic lung tissue exhibits signs of emphysema, pointing to alveolar damage and airspace enlargement.", claim 9: "The diagnosis is squamous cell carcinoma, non-keratinizing type, poorly differentiated (grade I)." PathMem (Ours): Microscopic observation of the pathology slide reveals a poorly differentiated squamous cell carcinoma characterized by a lack of glandular formation, indicating a significant deviation from normal tissue architecture. The tumor cells exhibit marked nuclear pleomorphism, with notable variability in nuclear size and shape. There is a high mitotic activity observed, indicative of a rapid rate of cell division. Vascular invasion is not identified, and there is no evidence of necrosis or calcification within the examined tissue. Based on these observations, the diagnosis is poorly-differentiated squame cell carcenoma. LUSC/TCGA-21-1083 GPT-4o: Microscopic observation of the pathology slide reveals a poorly differentiated adenocarcinoma with a predominant bronchioloalveolar pattern. The tumor cells exhibit significant nuclear pleomorphism, characterized by marked variability in nuclear size and shape, and a high mitotic rate, indicating a rapid rate of cell division. There is evidence of vascular invasion, with tumor emboli present within vascal spaces. The surrounding lung parenchyma shows signs of emphysema and chronic obstructive pulmonary disease (COPD), with mild interstitial fibrosis and mild chronic inflammation. Based on these observations, the diagnosis is poorly-differentiated adenoacanthoma. Quilt-LLaVA: Microscopic observation of the pathology slide reveals a moderately differentiated adenocarcinoma characterized by irregular clusters of epithelial cells infiltrating the stroma. The tumor cells form glandular structures that moderately deviate from normal tissue architecture. The nuclei exhibit moderate variability in size and shape, indicative of nuclear pleomorphism. There is a moderate rate of cell division observed. Vascular invasion is not present. Areas of necrosis are not noted. Based on these observations, the diagnosis is moderately-differentiated adeno-glandular carcinoma. WSI-LLaVA: Microscopic observation of the pathology slide reveals a poorly differentiated squamous cell carcinoma with a lack of glandular differentiation. The tumor cells exhibit a high degree of nuclear pleomorphism, with significant variability in nuclear size and shape. There is a high mitotic rate, indicating a rapid rate of cell division. The carcinomatous cells are seen invading the surrounding stroma and are present within small blood vessels, indicating vascular invasion. Additionally, there is no evidence of necrosis or calcification within the tumor. The surrounding non-neoplastic lung tissue shows emphysematous changes. Based on these observations, the diagnosis is poorly- differentiated squarous cell adenocarcinoma. T-answer: Microscopic observation of the pathology slide reveals sheets and nests of atypical squamous cells with abundant eosinophilic cytoplasm and significant nuclear atypia, characterized by increased nuclear size, irregular nuclear contours, and hyperchromasia. The architecture predominantly lacks keratinization, aligning with a non-keratinizing histological variant. Morphological assessment indicates poorly differentiated squamous cells with grade I differentiation, characterized by highly disorganized cell arrangements and pronounced pleomorphism. There is moderate tumor necrosis indicating areas of cell death, but no evidence of lymphatic, venous, or arterial invasion by the tumor. Additionally, non-neoplastic lung tissue exhibits signs of emphysema, pointing to alveolar damage and airspace enlargement. Based on these observations, the diagnosis is squamous cell carcinoma, non-keratinizing type, poorly differentiated (grade I). 0.3 0.7 0.3 1.0 1.0 0 1.0 0 0.7 0 0.7 0 0 0.7 0 0 1.0 0 0 0.3 0 0 0.3 0 1.0 0 0 0.3 0.7 0 0.3 0.7 0 0 1.0 0 Q：What are the main observations in the whole slide histopathology image, and what is the likely diagnosis? Fig. 4: Qualitative comparison of generated reports by our method and three baseline approaches on the report generation task. (Red highlights denote incorrect content, while green highlights denote correct content, and orange highlights denote missing content from T-answer.) Qualitative analysis Obs. ❶: The Superior Reasoning Capability of PathMem. As shown in Fig. 4, we present a qualitative comparison for case LUSC/TCGA-21- 1083. The ground truth is poorly differentiated (Grade I), non-keratinizing squamous cell carcinoma. Overall Diagnosis. PathMem correctly predicts poorly differentiated squamous cell carcinoma, fully consistent with the refer- ence. WSI-LLaVA captures squamous features but outputs an inconsistent hy- brid diagnosis. GPT-4o and Quilt-LLaVA misclassify the case as adenocarci- noma variants, reflecting confusion between squamous and glandular differen- tiation. Fine-Grained Morphology. For squamous differentiation (Claims 1, 3–5), PathMem best captures poor differentiation and marked pleomorphism, with partial recognition of non-keratinizing features. Other models show weaker 12Li et al. alignment and often hallucinate glandular structures. For nuclear atypia (Claim 2), all models detect pleomorphism, but PathMem demonstrates greater consis- tency. For necrosis and invasion (Claims 6–7), performance is generally limited; however, PathMem correctly identifies the absence of vascular invasion. For background lung changes (Claim 8), WSI-LLaVA and Quilt-LLaVA note em- physematous features, whereas PathMem focuses on tumor characteristics. Obs. ❷: Metric Evidence Supporting PathMem’s Reasoning Strength The quantitative results closely mirror the color-coded qualitative comparison (Fig. 5). Hist-Gen exhibits numerous red deviations and orange omissions, leading to low semantic overlap (BLEU-4: 0.025; ROUGE-L: 0.182) and limited pathological relevance (WSI-P: 0.257; WSI-R: 0.230). WSI-LLaVA shows mod- erate improvement, with fewer deviations but continued omission of key invasive features. This corresponds to slightly higher BLEU-4 (0.043), ROUGE-L (0.250), and WSI-based metrics (WSI-P: 0.311; WSI-R: 0.294). In contrast, PathMem demonstrates predominantly green matches with minimal deviations, alongside multiple blue knowledge graph–grounded concepts. This results in substantially improved lexical and clinical alignment (BLEU-4: 0.361; ROUGE-L: 0.583; ME- TEOR: 0.678) and markedly higher WSI-Precision and recall (0.596 / 0.592). Overall, reductions in deviations and missing findings correspond directly to gains across both NLU and WSI-based metrics, while knowledge graph ground- ing further enhances diagnostic consistency. ♣ Answer-Aligned Memory Path. Visual Cues (solid growth, poor differen- tiation, papillary areas, vascular invasion, fibrotic lung background) → Static LTM Retrieval (lung primary site, NSCLC spectrum, solid/high-grade archi- tecture) → Dynamic Multimodal Reweighting (↑ glandular signals + ↑ squamous patterns; ↓ irrelevant entities) → Adaptive LTM→WM Trans- fer (lung context + dual-lineage semantics + invasion knowledge) → WM- Integrated Diagnosis: Poorly Differentiated Adenosquamous Carci- noma (+ angiolymphatic invasion [LTM ] + focal visceral pleural in- volvement [LTM ] + background emphysema [LTM ]). The highlighted di- agnostic elements emerge through structured LTM activation and controlled LTM→WM transition, contributing to the final textual output beyond purely parametric generation. Table 4: Ablation study on static and dynamic KG retrieval. The full model with both static and dynamic LTM achieves the best performance across all metrics. D denotes Dynamic-activation, and S denotes Static-activation LTM. Version BaselineLTMPathMem -DSBLEU-1BLEU-2BLEU-3BLEU-4ROUGE-LMETEORWSI-PWSI-R a✓0.481± 0.1810.357± 0.1960.287± 0.2050.241± 0.2120.477± 0.1770.464± 0.1780.404± 0.1620.445± 0.163 b✓0.528± 0.1530.405± 0.1760.332± 0.1950.284± 0.1980.518± 0.1680.508± 0.1640.492± 0.1550.516± 0.147 c✓0.521± 0.1610.396± 0.1830.324± 0.2050.278± 0.2050.510± 0.1700.511± 0.1630.489± 0.1580.510± 0.148 d✓0.548± 0.1510.420± 0.1730.347± 0.1880.302± 0.1960.536± 0.1550.531± 0.1580.508± 0.1470.530± 0.144 PathMem13 ModelModel Responses WSI Metrics WSI-PWSI-R NLU Metrics Ground Truth Hist- Gen WSI- LLaVA PathMem N/AN/A BLEU-1: 0.274 BLEU-2: 0.100 BLEU-3: 0.045 BLEU-4: 0.025 ROUGE-L: 0.182 METEOR: 0.204 BLEU-1: 0.319 BLEU-2: 0.141 BLEU-3: 0.071 BLEU-4: 0.043 ROUGE-L: 0.250 METEOR: 0.258 BLEU-1: 0.636 BLEU-2: 0.519 BLEU-3: 0.426 BLEU-4: 0.361 ROUGE-L: 0.583 METEOR: 0.678 Microscopic observation of the pathology slide reveals poorly differentiated clusters of adenocarcinoma cells exhibiting significant architectural disorganization and extensive necrosis. There is an evident absence of glandular structure formation, indicative of poor differentiation. The slide also shows notable angiolymphatic invasion and evidence of visceral pleural invasion, as the tumor has penetrated the pleural surface. Additionally, the presence of multiple satellite nodules further exemplifies the invasive nature of this carcinoma. Background lung tissue displays changes consistent with emphysema. Based on these observations, the diagnosis is invasive poorly differentiated adenocarcinoma. Histological evaluation demonstrates an aggressive epithelial malignancy of gland-forming origin with limited differentiation. The neoplasm is largely composed of sheets of tumor cells arranged in a compact pattern, while a minor proportion displays focal glandular architecture. Many cells contain abundant intracellular mucin that compresses the nuclei toward the cell membrane, producing a signet-ring–like appearance. The supporting stroma is dense with areas of fibrosis and hyaline change. No vascular or lymphatic permeation is identified. Adjacent pulmonary parenchyma exhibits features of chronic airspace enlargement, and a small neuroendocrine tumorlet is noted away from the primary lesion. Overall findings are consistent with a high-grade pulmonary adenocarcinoma. The examined lung specimen reveals a malignant glandular tumor with marked loss of structural organization. Most of the lesion consists of densely packed tumor cells, whereas only a small fraction forms rudimentary acinar structures. Numerous cells demonstrate intracellular mucous accumulation with peripheral displacement of nuclei. The tumor microenvironment shows prominent stromal sclerosis. Invasion into vascular or lymphatic channels is not observed. Background lung tissue is notable for airspace destruction compatible with emphysema, and a minute carcinoid proliferation is present at a distance from the main mass. These morphological features support the interpretation of a poorly differentiated carcinoma of adenocarcinomatous type arising in the lung. Microscopic examination of the pathology slide reveals poorly differentiated clusters of adenocarcinoma cells with a predominant solid growth pattern and marked architectural disorganization. There is minimal glandular structure formation, consistent with poor differentiation, and focal areas show necrotic change. The tumor demonstrates evidence of angiolymphatic invasion, and focal visceral pleural involvement is identified, indicating penetration toward the pleural surface. Scattered small nodular extensions are present within the surrounding parenchyma, supporting the invasive behavior of the carcinoma. Background lung tissue shows changes consistent with emphysema. Based on these findings, the diagnosis is invasive poorly differentiated adenocarcinoma. 0.2570.230 0.3110.294 0.5960.592 BLEU-1: — BLEU-2: — BLEU-3: — BLEU-4: — ROUGE-L: — METEOR: — Fig. 5: Comparison of WSI-based and NLU-based evaluations. Green indicates agree- ment with the reference, red denotes deviations, orange marks missing ground-truth content, underlined text reflects template-style language, and blue highlights knowl- edge graph–grounded concepts. 4.3 Analysis and Discussion Ablation Study on Static and Dynamic KG Retrieval. Tab. 4 presents the ablation results on static and dynamic KG retrieval. Both mechanisms con- sistently improve performance over the baseline (Version a). Adding dynamic LTM (Version b) increases BLEU-4 from 0.241 to 0.284 and WSI-P from 0.404 to 0.492. Static LTM (Version c) also yields clear gains, with BLEU-4 reaching 0.278 and WSI-R 0.510. Dynamic LTM achieves slightly higher BLEU scores than static LTM, for example 0.284 versus 0.278 in BLEU-4, while static LTM attains a marginally higher METEOR score, 0.511 versus 0.508. The full model 14Li et al. (Version d) achieves the best results on all metrics, including 0.302 BLEU-4, 0.536 ROUGE-L, 0.531 METEOR, 0.508 WSI-P, and 0.530 WSI-R. Compared with the baseline, BLEU-4 improves by 0.061 and WSI-P by 0.104. These results indicate that static and dynamic retrieval provide complementary benefits, and their combination is essential for optimal performance. Table 5: Sensitivity analysis on the maximum number of activated KG tokens (Top-K). Performance comparison under different limits for dynamic and static LTM retrieval. D denotes Dynamic-activation, and S denotes Static-activation LTM. Version Token CapPathMem DSBLEU-1BLEU-2BLEU-3BLEU-4ROUGE-LMETEORWSI-PWSI-R a 110.517± 0.1610.393± 0.1840.320± 0.1980.274± 0.2100.508± 0.1720.500± 0.1690.477± 0.1630.496± 0.151 b 220.525± 0.1600.402± 0.1810.330± 0.2040.284± 0.2050.517± 0.1680.509± 0.1640.484± 0.1590.497± 0.148 c 330.531± 0.1530.410± 0.1750.339± 0.1930.293± 0.1960.525± 0.1620.514± 0.1610.493± 0.1550.513± 0.147 d 440.537± 0.1540.414± 0.1730.340± 0.1940.292± 0.1970.528± 0.1580.517± 0.1610.502± 0.1490.526± 0.146 g 550.542± 0.1510.418± 0.1740.346± 0.1880.298± 0.1970.532± 0.1560.523± 0.1580.506± 0.1470.528± 0.143 Sensitivity Analysis of Top-K KG Tokens. Tab. 5 presents the sensitivity analysis on the maximum number of activated KG tokens for dynamic and static LTM retrieval. Performance consistently improves as Top-K increases from 1 to 5. With Top-K equals 1, BLEU-4 is 0.274 and WSI-P is 0.477. Increasing Top-K to 3 raises BLEU-4 to 0.293 and WSI-R to 0.513. The best results are achieved at Top-K equals 5, with 0.298 BLEU-4, 0.532 ROUGE-L, 0.523 METEOR, 0.506 WSI-P, and 0.528 WSI-R. Compared with Top-K equals 1, BLEU-4 improves by 0.024 and WSI-P by 0.029. The gains diminish beyond Top-K equals 3, for example BLEU-4 increases only from 0.293 to 0.298 when moving from 3 to 5. This indicates that a moderate token budget captures most useful knowledge, while larger limits provide limited but consistent improvements. 5 Conclusion We propose PathMem, a memory-augmented MLLM for computational pathol- ogy that explicitly models long- and working-memory through a PubMed-derived pathology knowledge graph and a dual-stage Memory Transformer. Experiments on WSI-Bench and three zero-shot benchmarks show consistent improvements over SOTA models across core pathology tasks. Ablation and sensitivity stud- ies confirm the effectiveness of dual-mode KG retrieval and moderate Top-K token settings. PathMem introduces an interpretable, reproducible framework for knowledge-enhanced multimodal reasoning in digital pathology, with future work focusing on expanding KG coverage, improving WSI processing efficiency, and conducting clinical validation. References 1. Ahmed, F., Sellergren, A., Yang, L., Xu, S., Babenko, B., Ward, A., Olson, N., Mohtashamian, A., Matias, Y., Corrado, G.S., et al.: Pathalign: A vision-language PathMem15 model for whole slide images in histopathology. arXiv preprint arXiv:2406.19578 (2024) 1, 3 2. An, T.: Cognitive workspace: Active memory management for llms–an empirical study of functional infinite context. arXiv preprint arXiv:2508.13171 (2025) 2 3. Baddeley, A.: Working memory. In: Memory, p. 101–152. Routledge (2025) 2 4. Bei, Y., Wei, T., Ning, X., Zhao, Y., Liu, Z., Lin, X., Zhu, Y., Hamann, H., He, J., Tong, H.: Mem-gallery: Benchmarking multimodal long-term conversational memory for mllm agents. arXiv preprint arXiv:2601.03515 (2026) 2 5. Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Cornia, M., Cucchiara, R.: The revolution of multimodal large language models: A survey. Findings of the association for computational linguistics: ACL 2024 p. 13590– 13618 (2024) 1 6. Chen, J., Cai, L., Wang, Z., Huang, Y., Jiang, S., Huang, S., Wang, H., Zhang, Y.: Pathagent: Toward interpretable analysis of whole-slide pathology images via large language model-based agentic reasoning. arXiv preprint arXiv:2511.17052 (2025) 2 7. Chen, P., Li, H., Zhu, C., Zheng, S., Shui, Z., Yang, L.: Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images. In: In- ternational Conference on Medical Image Computing and Computer-Assisted In- tervention. p. 546–556. Springer (2024) 10 8. Chen, P., Zhu, C., Zheng, S., Li, H., Yang, L.: Wsi-vqa: Interpreting whole slide images by generative visual question answering. In: European Conference on Com- puter Vision. p. 401–417. Springer (2024) 8 9. Chen, P., Zhu, C., Zheng, S., Li, H., Yang, L.: Wsi-vqa: Interpreting whole slide images by generative visual question answering. In: European Conference on Com- puter Vision. p. 401–417. Springer (2025) 10, 11 10. Chen, Y., Wang, G., Ji, Y., Li, Y., Ye, J., Li, T., Hu, M., Yu, R., Qiao, Y., He, J.: Slidechat: A large vision-language assistant for whole-slide pathology image understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 5134–5143 (2025) 3, 8 11. Chen, Y., Wang, G., Ji, Y., Li, Y., Ye, J., Li, T., Zhang, B., Pei, N., Yu, R., Qiao, Y., et al.: Slidechat: A large vision-language assistant for whole-slide pathology image understanding. arXiv preprint arXiv:2410.11761 (2024) 11 12. Cifci, D., Veldhuizen, G.P., Foersch, S., Kather, J.N.: Ai in computational pathol- ogy of cancer: improving diagnostic workflows and clinical outcomes? Annual Re- view of Cancer Biology 7(1), 57–71 (2023) 1 13. Ding, T., Wagner, S.J., Song, A.H., Chen, R.J., Lu, M.Y., Zhang, A., Vaidya, A.J., Jaume, G., Shaban, M., Kim, A., et al.: A multimodal whole-slide foundation model for pathology. Nature medicine p. 1–13 (2025) 3 14. Dong, C.V., Lu, Q., Norman, K.A., Michelmann, S.: Towards large language models with human-like episodic memory. Trends in Cognitive Sciences (2025) 2 15. Fountzilas, E., Pearce, T., Baysal, M.A., Chakraborty, A., Tsimberidou, A.M.: Convergence of evolving artificial intelligence and machine learning techniques in precision oncology. NPJ Digital Medicine 8(1), 75 (2025) 1 16. Guo, Z., Ma, J., Xu, Y., Wang, Y., Wang, L., Chen, H.: Histgen: Histopathology report generation via local-global feature encoding and cross-modal context inter- action. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. p. 189–199. Springer (2024) 10 17. Hamza, A., Ahn, Y.H., Lee, S., Kim, S.T., et al.: Llava needs more knowledge: Retrieval augmented natural language generation with knowledge graph for ex- 16Li et al. plaining thoracic pathologies. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, p. 3311–3319 (2025) 2 18. He, Q., Tan, H., Xiao, B., Tan, Y., Peng, X., Peng, C., Yue, X., Jiang, L., Cao, Y., Lv, F.J., et al.: Prior knowledge-guided multimodal deep learning system for biomarker exploration and prognosis prediction of urothelial carcinoma. npj Digital Medicine (2025) 4 19. Hou, W., Cheng, Y., Xu, K., Hu, Y., Li, W., Liu, J.: Memory-augmented multimodal llms for surgical vqa via self-contained inquiry. arXiv preprint arXiv:2411.10937 (2024) 2 20. Hou, Y., Tamoto, H., Zhao, Q., Miyashita, H.: Synapticrag: Enhancing temporal memory retrieval in large language models through synaptic mechanisms. In: Find- ings of the Association for Computational Linguistics: ACL 2025. p. 20422–20436 (2025) 2 21. Hu, Q., Lyu, W., Xu, M., Qi, K., Hu, X., Gupta, S., Zhou, J., Chen, C.: Loc- path: Learning to compress for pathology multimodal large language models. arXiv preprint arXiv:2512.05391 (2025) 2, 4 22. Huang, W., Wang, Z., Lin, H., Wang, S., Xu, B., Li, Q., Zhu, B., Yang, L., Qin, C.: Ama: Adaptive memory via multi-agent collaboration. arXiv preprint arXiv:2601.20352 (2026) 2 23. Huang, Y., Zhao, W., Zhang, Z., Chen, Y., Fu, Y., Wu, F., Jiang, Y., Liang, L., Wang, S., Yu, L.: Knowledge-guided adaptation of pathology foundation models effectively improves cross-domain generalization and demographic fairness. Nature Communications (2025) 4 24. Islam, R., Moushi, O.M.: Gpt-4o: The cutting-edge advancement in multimodal llm. Authorea Preprints (2024) 10 25. Jabal, M.S., Warman, P., Zhang, J., Gupta, K., Jain, A., Mazurowski, M., Wig- gins, W., Magudia, K., Calabrese, E.: Open-weight language models and retrieval- augmented generation for automated structured data extraction from diagnostic reports: assessment of approaches and parameters. Radiology: Artificial Intelligence 7(3), e240551 (2025) 2 26. Jiang, H., Chen, J., Pan, Y., Chen, L., You, W., Zhou, Y., Zhang, R., Sikora, A., Zhao, L., Abate, Y., et al.: Synapse: Empowering llm agents with episodic-semantic memory via spreading activation. arXiv preprint arXiv:2601.02744 (2026) 2 27. Jiang, R., Yin, X., Yang, P., Cheng, L., Hu, J., Yang, J., Wang, Y., Fu, X., Shang, L., Li, L., et al.: A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular marker discovery of gliomas. Nature Machine Intelligence 6(8), 876–891 (2024) 1 28. Jiang, S., Liu, F., Wang, Z., Cai, L., Zhang, Y.: Pathreasoner-r1: Instilling struc- tured reasoning into pathology vision-language model via knowledge-guided policy optimization. arXiv preprint arXiv:2601.21617 (2026) 4 29. Kim, S., Lee, S., Jang, J.: Chatexaonepath: An expert-level multimodal large language model for histopathology using whole slide images. arXiv preprint arXiv:2504.13023 (2025) 4 30. Liang, Y., Lyu, X., Chen, W., Ding, M., Zhang, J., He, X., Wu, S., Xing, X., Yang, S., Wang, X., et al.: Wsi-llava: A multimodal large language model for whole slide image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 22718–22727 (2025) 1, 3, 9, 10, 11 31. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 1 PathMem17 32. Liu, Y., Liang, Z., Wang, Y., Wu, X., Tang, F., He, M., Li, J., Liu, Z., Yang, H., Lim, S., et al.: Unveiling the ignorance of mllms: Seeing clearly, answering incorrectly. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 9087–9097 (2025) 2 33. Liu, Y., Lin, X., Shang, Y., Li, Y., Wang, S., Cao, Y.: Pathmind: A retrieve- prioritize-reason framework for knowledge graph reasoning with large language models. arXiv preprint arXiv:2511.14256 (2025) 2 34. Liu, Y., Carrero, Z.I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., Kather, J.N.: Benchmarking large language model-based agent systems for clinical decision tasks. npj Digital Medicine (2026) 1 35. Ma, J., Guo, Z., Zhou, F., Wang, Y., Xu, Y., Li, J., Yan, F., Cai, Y., Zhu, Z., Jin, C., et al.: A generalizable pathology foundation model using a unified knowledge distillation pretraining framework. Nature Biomedical Engineering p. 1–20 (2025) 4 36. Omar, M., Ullanat, V., Loda, M., Marchionni, L., Umeton, R.: Chatgpt for digital pathology research. The Lancet Digital Health 6(8), e595–e600 (2024) 1, 2 37. Seyfioglu, M.S., Ikezogwo, W.O., Ghezloo, F., Krishna, R., Shapiro, L.: Quilt- llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13183–13192 (2024) 10, 11 38. Shafique, A., Gonzalez, R., Pantanowitz, L., Tan, P.H., Machado, A., Cree, I.A., Tizhoosh, H.R.: A preliminary investigation into search and matching for tumor discrimination in world health organization breast taxonomy using deep networks. Modern Pathology 37(2), 100381 (2024) 1 39. Shaikovski, G., Casson, A., Severson, K., Zimmermann, E., Wang, Y.K., Kunz, J.D., Retamero, J.A., Oakley, G., Klimstra, D., Kanan, C., et al.: Prism: A multi- modal generative foundation model for slide-level histopathology. arXiv preprint arXiv:2405.10254 (2024) 3 40. Shi, J., Li, C., Zheng, Y., Fu, H.: Kpvg: Knowledge-prompted vision-genomics model for cancer survival prediction in whole slide images. Information Fusion p. 103660 (2025) 4 41. Smart, J., Roux, M.: A model for medical knowledge representation application to the analysis of descriptive pathology reports. Methods of information in medicine 34(04), 352–360 (1995) 4 42. Sun, Y., Si, Y., Zhu, C., Gong, X., Zhang, K., Chen, P., Zhang, Y., Shui, Z., Lin, T., Yang, L.: Cpath-omni: A unified multimodal foundation model for patch and whole slide image analysis in computational pathology. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 10360–10371 (2025) 1, 3 43. Sun, Y., Wu, H., Zhu, C., Zheng, S., Chen, Q., Zhang, K., Zhang, Y., Wan, D., Lan, X., Zheng, M., et al.: Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. In: European Conference on Computer Vision. p. 56–73. Springer (2024) 2 44. Sun, Y., Zhu, C., Zheng, S., Zhang, K., Sun, L., Shui, Z., Zhang, Y., Li, H., Yang, L.: Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, p. 5034–5042 (2024) 3 45. Verghese, G., Lennerz, J.K., Ruta, D., Ng, W., Thavaraj, S., Siziopikou, K.P., Naidoo, T., Rane, S., Salgado, R., Pinder, S.E., et al.: Computational pathology in cancer diagnosis, prognosis, and prediction–present day and prospects. The Journal of pathology 260(5), 551–563 (2023) 1 18Li et al. 46. Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al.: A whole-slide foundation model for digital pathology from real-world data. Nature p. 1–8 (2024) 3 47. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. National Science Review 11(12), nwae403 (2024) 1 48. Yoshida, R., Isono, S., Kajikawa, K., Someya, T., Sugimoto, Y., Oseki, Y.: If atten- tion serves as a cognitive model of human memory retrieval, what is the plausible memory representation? In: Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). p. 9795–9812 (2025) 2 49. Zhang, W., Guo, J., Zhang, H., Zhang, P., Chen, J., Zhang, S., Zhang, Z., Yi, Y., Bu, H.: Patho-agenticrag: towards multimodal agentic retrieval- augmented generation for pathology vlms via reinforcement learning. arXiv preprint arXiv:2508.02258 (2025) 2 50. Zhao, W., Guo, Z., Fan, Y., Jiang, Y., Yeung, M.C., Yu, L.: Aligning knowledge concepts to whole slide images for precise histopathology image analysis. npj Dig- ital Medicine 7(1), 383 (2024) 4