Paper deep dive

SAGE: Semantic Explainability of Attention-Based Survival Models in Computational Pathology

Abdallah Lamane, Abdul Rahman Diab, Ren-Chin Wu, William Lotter

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 29

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/5/2026, 4:30:50 AM

Summary

The paper introduces SAGE (Semantic Attention Global Explanations), a post-hoc framework for extracting global, language-grounded explanations from frozen Attention-Based Multiple Instance Learning (ABMIL) models used in computational pathology. SAGE utilizes a pathology vision-language model (VLM) to score image patches against a dictionary of 25 histological concepts, aggregating these scores using the ABMIL model's attention weights to quantify the relationship between specific histological features and patient survival risk across cohorts. Applied to seven TCGA cancer cohorts, SAGE successfully recovered known prognostic features (e.g., necrosis as adverse, epithelial cells as favorable) and identified cancer-specific biology (e.g., angiogenic signatures in renal cell carcinoma). The study demonstrates that SAGE provides scalable, model-agnostic interpretability, validating that the concept dictionary captures significant prognostic information encoded by foundation models.

Entities (10)

SAGE · framework · 98%ABMIL · model-architecture · 95%Necrosis · histological-concept · 95%Renal Cell Carcinoma (KIRC) · disease-cohort · 95%TCGA · dataset · 95%CONCH · foundation-model · 92%Angiogenesis · histological-concept · 90%Epithelial Cells · histological-concept · 90%MUSK · foundation-model · 90%UNI2 · foundation-model · 90%

Relation Signals (7)

SAGE → analyzes → TCGA

confidence 95% · Applied to survival prediction using seven TCGA cancer cohorts

SAGE → appliesto → Renal Cell Carcinoma (KIRC)

confidence 95% · revealing cancer-specific biology, including a favorable angiogenic signature in renal cell carcinoma

SAGE → identifies → Necrosis

confidence 95% · SAGE recovered established prognostic features, such as the adverse association of necrosis

SAGE → uses → ABMIL

confidence 95% · SAGE extracts global explanations from a frozen ABMIL model.

SAGE → identifies → Angiogenesis

confidence 90% · revealing cancer-specific biology, including a favorable angiogenic signature in renal cell carcinoma

SAGE → identifies → Epithelial Cells

confidence 90% · Averaged across cohorts, the concept most associated with favorable survival predictions is epithelial cells

SAGE → uses → CONCH

confidence 90% · Our core experiments use CONCH as both the vision encoder for ABMIL fitting and explainability through SAGE.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Attention-based multiple instance learning (ABMIL) is the predominant approach for slide-level prediction in computational pathology, yet its attention maps provide only local explanations: they indicate where a model focuses but not which histological features drive its predictions or how the model behaves across a patient cohort. We present Semantic Attention Global Explanations (SAGE), a post-hoc framework that extracts global, language-grounded explanations from a frozen ABMIL model. Using a pathology vision-language model, SAGE scores image patches against a dictionary of 25 histological concepts, aggregates these scores according to the model's learned attention, and quantifies how each concept relates to prediction risk across a cohort. Applied to survival prediction using seven TCGA cancer cohorts and three foundation models, SAGE recovered established prognostic features, such as the adverse association of necrosis, while revealing cancer-specific biology, including a favorable angiogenic signature in renal cell carcinoma consistent with known molecular subtypes. Ablation studies demonstrated that these associations depend on the model's learned attention rather than concept prevalence alone, and that the concept dictionary captures much of the prognostic information encoded by the foundation model features. Through semantically-grounded explanations, SAGE provides a scalable, model-agnostic framework for understanding what ABMIL survival models learn, enabling pathologists to interpret model behavior at the cohort level and offering the potential for biomarker identification.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

27,714 characters extracted from source content.

Expand or collapse full text

SAGE: Semantic Explainability of Attention-Based Survival Models in Computational Pathology Abdallah Lamane 1,2 , Abdul Rahman Diab 2 , Ren-Chin Wu 3 , and William Lotter 2,4,5 1 Massachusetts Institute of Technology, Cambridge, MA, USA 2 Department of Data Science, Dana-Farber Cancer Institute, Boston, MA 3 Department of Pathology, Dana-Farber Cancer Institute, Boston, MA 4 Department of Pathology, Brigham and Women’s Hospital, Boston, MA 5 Harvard Medical School, Boston, MA lotterb@ds.dfci.harvard.edu Abstract. Attention-based multiple instance learning (ABMIL) is the predominant approach for slide-level prediction in computational pathol- ogy, yet its attention maps provide only local explanations: they indicate where a model focuses but not which histological features drive its pre- dictions or how the model behaves across a patient cohort. We present Semantic Attention Global Explanations (SAGE), a post-hoc framework that extracts global, language-grounded explanations from a frozen AB- MIL model. Using a pathology vision-language model, SAGE scores im- age patches against a dictionary of 25 histological concepts, aggregates these scores according to the model’s learned attention, and quantifies how each concept relates to prediction risk across a cohort. Applied to survival prediction using seven TCGA cancer cohorts and three founda- tion models, SAGE recovered established prognostic features, such as the adverse association of necrosis, while revealing cancer-specific biology, including a favorable angiogenic signature in renal cell carcinoma con- sistent with known molecular subtypes. Ablation studies demonstrated that these associations depend on the model’s learned attention rather than concept prevalence alone, and that the concept dictionary captures much of the prognostic information encoded by the foundation model features. Through semantically-grounded explanations, SAGE provides a scalable, model-agnostic framework for understanding what ABMIL survival models learn, enabling pathologists to interpret model behavior at the cohort level and offering the potential for biomarker identification. Keywords: Concept-based explanations· Global explanations· Foun- dation models· Computational pathology· Survival analysis. 1 Introduction Attention-based multiple instance learning (ABMIL) [4] is a standard approach for weakly supervised tasks in computational pathology, including emerging ap- plications in survival prediction directly from H&E whole-slide images (WSIs). arXiv:2608.02803v1 [cs.CV] 3 Aug 2026 2A. Lamane et al. As these models move toward clinical decision support, explainability is essen- tial for adoption. ABMIL offers local, spatial explainability through attention heatmaps for individual slides, but these maps highlight only regions of interest and leave the underlying biological semantics, namely the histological concepts the model associates with favorable or adverse outcomes, opaque. In addition to semantic explainability, there is a need for global explanations that go beyond individual slides to characterize what a model has learned overall. Such global views can help check biological grounding, facilitate trust, and sup- port knowledge discovery. Concept-bottleneck and concept-guided approaches [7, 14, 9] address this by training models to predict through predefined human- interpretable concepts, but they typically require extensive concept annotations and have limited flexibility once trained. We present a global semantic explainability framework for ABMIL survival models that does not require retraining or additional annotations. Our approach leverages pathology-specific vision-language models (VLMs) to map ABMIL at- tention weights onto a dictionary of pathology concepts, and then aggregates concept scores across slides into a cohort-level explainability profile. We apply our framework to seven TCGA cohorts and three pathology foundation models, where we find that the recovered semantic profiles capture aspects of known biology, with both pan-cancer and cancer-specific prognostic trends. 2 Related Work Pathology foundation models. CONCH [10] and MUSK [15] are pathology VLMs trained to align histology images with text captions in a shared embedding space. Vision-only foundation models, such as UNI2 [1], are commonly trained using self-distillation and contrastive self-supervised learning objectives. Concept- and language-grounded explainability in MIL. A growing line of work incorporates textual concepts into the WSI classifier and trains it end-to-end. ConceptMIL [14] is a label-free concept bottleneck model whose prediction is a linear combination of VLM-predicted concepts on the top attended patches; Con- cepPath [18] extracts expert concepts with GPT-4 and learns a concept-guided hierarchical attention; VLEER [12] concatenates retrieved text embeddings to the visual features before training an MIL aggregator. All three focus on clas- sification tasks rather than survival, and their concept vocabularies are directly incorporated into the predictive representation or model training pipeline. Vision-language survival analysis. VLSA [9] performs survival prediction with language: it encodes ordinal prognostic text priors and uses them as auxiliary signals that guide instance aggregation during training. Conversely, our frame- work is designed to explain existing, frozen survival models, making it applicable to vision-only attention-MIL predictors not designed with text in mind. More- over, rather than imposing language priors and subsequently determining their attribution scores, SAGE reads out concept associations from a model’s atten- tion, allowing different vocabularies to be queried retrospectively using the same model. Semantic Explainability of Survival Models3 Fig. 1. SAGE: A framework for global semantic explanations of trained ABMIL models. 3 Semantic Attention Global Explanations (SAGE) Setup and notation. A WSI is represented as a bag of N patches X = x 1 ,...,x N , which are processed by a frozen vision encoder (i.e. foundation model) h v to produce embeddings h v (x i ). An ABMIL network assigns each patch an attention weight a i and aggregates patch-level representations into a slide- level representation z: a i = exp w ⊤ tanh(V h v (x i )) P N j=1 exp w ⊤ tanh(V h v (x j )) , z = N X i=1 a i h v (x i ),(1) with learnable parameters w,V . A downstream head then maps z to a slide-level prediction (e.g. a survival score). In SAGE we treat the entire ABMIL predictor as frozen and use only its attention weights and output scores. Language-grounded semantic projection. We define a dictionary of T = 25 pathology concepts that encompass common cell types (e.g. lymphocytes), cell descriptors (e.g. mitotic figures), and microenvironmental features (e.g. necrosis) often used in interpreting histology slides; the full list (Table 1) was curated under pathologist guidance. The framework is flexible to the terms included, and users can add or remove terms as needed. For each concept y j ,j ∈ [1,T], the text encoder of a pathology VLM g t generates an embedding g t (y j ). For robustness, we use a prompt-ensembling 4A. Lamane et al. Table 1. The 25-term pathology concept dictionary, grouped by category. CategoryTerms Cell typeslymphocytes, plasma cells, macrophages, neutrophils, eosinophils, fibroblasts, endothelial cells, epithelial cells, adipocytes, cancer cells Cell / nuclear descriptors mitotic figures, apoptosis, pleomorphism, hyperchromasia, nucleolar enlargement, dysplasia, poor differentiation Microenvironment & architecture angiogenesis, desmoplasia, scarring, necrosis, lymphoid aggregates, hypercellularity, lymphovascular invasion, perineural invasion strategy, averaging over 22 natural-language templates provided by the CONCH repository [10]. For each patch x i , we use the VLM’s vision encoder g v to compute the cosine similarity s i,j = ⟨g v (x i ),g t (y j )⟩/(∥g v (x i )∥g t (y j )∥) and aggregate it into an attention-weighted, patient-level score S j = P N i=1 a i s i,j using the attention weights of the ABMIL model. Intuitively, S j quantifies how strongly the ABMIL model attends to morphology semantically related to concept j on a slide, using a VLM to quantify patch-level semantic similarity. We note that the framing of our method allows SAGE to leverage different vision encoders for the ABMIL predictor and the vision-language cosine similarity computation. Our goal is to explain the ABMIL predictor itself, with SAGE transforming the localized attention weights into semantically meaningful scores. From concept scores to global explanations. Given a trained ABMIL model and a concept dictionary, SAGE produces a vector (S j ) T j=1 of attention- weighted concept scores for each patient. To obtain a global explanation of what the model has learned, we compare these scores to the model’s own predictions across patients. In our experiments, the ABMIL output is a continuous survival score, and we summarize the association between each concept and model pre- dictions using a rank correlation coefficient over the cohort, as described below. Explaining prognostic models with SAGE. For survival prediction, the slide representation z is mapped to discrete-time hazard logits l k over predefined time bins k: h k = σ(l k ), P k = Y j≤k (1− h j ),(2) where h k is the hazard and P k the discrete survival probability at bin k. As is common in prognostication studies, we define a cumulative prognostic score ˆτ = P k P k , which increases with predicted survival time. The ABMIL parameters are trained with a discrete-time negative log-likelihood survival loss [16], and then frozen for the SAGE analysis. Statistical association analysis. For each concept y j , we compute the Spear- man rank correlation r between S j and the ABMIL-predicted prognostic score Semantic Explainability of Survival Models5 ˆτ on test-set patients. We perform five-fold cross validation for all experiments, reporting the cross-fold mean correlation with confidence intervals computed via bootstrapping and p-values computed via permutation (5000 resamples and permutations respectively, both performed within folds). Since ˆτ increases with predicted survival time, a positive r indicates that higher values of concept y j co- occur with patients the model predicts to survive longer (favorable association), and a negative r indicates the opposite (adverse association). 4 Experiments and Results 4.1 Data and Setup We use seven TCGA cohorts (BRCA, BLCA, CESC, COAD, KIRC, LGG, LUAD), with disease-specific survival (DSS) as the endpoint. We sample one slide per patient, resulting in cohort sizes of 1014, 370, 246, 398, 479, 478 and 426 respectively. Our core experiments use CONCH as both the vision encoder for ABMIL fitting and explainability through SAGE. We additionally perform sensitivity analyses using MUSK and UNI2. WSIs are processed and tiled using PathFMTools [2] with foreground tissue segmented via the HEST pipeline [5]. ABMIL training. The discrete-time survival head uses four time bins set to the training-fold event-time quartiles. Training is performed for up to 40 epochs using the Adam optimizer [6] and early stopping (patience 8). Five-fold cross-validation is used, stratified by event-time quartile, where for each fold, three splits are used for training, one for validation, and one for testing. For each (cohort, model) pair, we sweep the same small grid: learning rate ∈ 5×10 −5 , 10 −4 , 2×10 −4 and hidden dimension ∈ 256, 384, 512. We then select the values leading to the highest mean validation C-index across folds before proceeding to testing. 4.2 Prognostic Performance An ABMIL predictor using the CONCH vision embeddings attains a mean test C-index of 0.625 across the seven cohorts, ranging from 0.77± 0.03 on KIRC to 0.53 ± 0.09 on BLCA. These concordance values are in line with existing work evaluating foundation model-based ABMIL survival models on TCGA [8], providing a basis for our explainability analysis. 4.3 Semantic Explainability The SAGE profile of the CONCH ABMIL predictor for each cohort is summa- rized in Fig. 2, which shows the Spearman correlation between each concept and the prognosis predictions. Averaged across cohorts, the concept most asso- ciated with favorable survival predictions is epithelial cells, with necrosis most associated with adverse predictions. These findings are biologically intuitive as epithelial cells represent normal tissue, and necrosis indicates cell death, which has been shown to be negatively prognostic in several cancers [17, 13]. 6A. Lamane et al. epithelial cells hypercellularity endothelial cells dysplasia lymphocytes lymphoid aggregates hyperchromasia nucleolar enlargement plasma cells fibroblasts scarring adipocytes poor differentiation angiogenesis macrophages eosinophils apoptosis lymphovascular invasion mitotic figures desmoplasia neutrophils perineural invasion pleomorphism cancer cells necrosis BRCA BLCA CESC COAD KIRC LGG LUAD 0.29*0.49*0.60*0.050.39*0.44*0.19*-0.06*0.25*0.42*0.38*0.56*-0.17*0.21*0.36*0.13*0.03-0.27*-0.33*-0.30*0.12*-0.44*-0.35*-0.15*-0.18* 0.43*0.25*0.47*0.16*0.27*0.23*0.13*0.32*0.12*0.25*0.050.30*0.100.14*0.30*0.20*-0.03-0.20*-0.14*-0.25*-0.04-0.23*-0.10-0.26*-0.13* 0.050.16*-0.04-0.030.28*0.03-0.060.100.57*-0.04-0.17*-0.28*0.26*-0.13*-0.41*-0.25*-0.37*0.45*0.60*0.27*-0.44*0.44*0.21*-0.20*-0.34* 0.02-0.00-0.15*0.34*-0.17*-0.10*0.42*0.38*-0.22*-0.02-0.08-0.060.09-0.13*-0.23*0.010.040.29*0.37*0.19*-0.19*0.15*0.25*0.08-0.21* 0.76*0.60*0.83*0.52*0.52*0.69*0.12*0.20*0.55*0.17*0.37*0.49*-0.040.47*0.40*0.080.30*-0.33*-0.45*-0.56*0.12*-0.45*-0.61*-0.45*-0.11* 0.26*0.51*0.050.37*-0.12*-0.12*-0.030.46*-0.21*-0.27*-0.11*-0.13*0.42*-0.40*-0.28*-0.34*-0.24*-0.28*-0.19*-0.13*-0.22*-0.24*-0.37*-0.23*0.06 0.39*0.10*0.24*0.32*0.17*0.12*0.45*-0.23*-0.020.35*0.35*-0.14*-0.070.03-0.020.17*-0.07-0.09-0.28*0.10*-0.09-0.02-0.090.09-0.41* 0.75 0.50 0.25 0.00 0.25 0.50 0.75 Spearman r Fig. 2. SAGE profiles for prognosis prediction across seven TCGA cohorts using CONCH. Spearman r between concept scores and predicted prognosis is shown per concept and cohort, with concepts sorted by mean r; positive values indicate corre- lation with longer predicted survival, negative values with shorter survival. Asterisks mark p < 0.05 via permutation test. Across cancer types, the most pronounced SAGE profile is observed in re- nal cell carcinoma (KIRC), which also corresponds to the highest prognostic performance. Concept scores for endothelial cells (cells that line blood vessels; r = 0.83± 0.07), plasma cells (blood cells; r = 0.55± 0.19), and angiogenesis (blood vessel growth; r = 0.47±0.11) all correlate with longer predicted survival. This aligns with findings that angiogenic molecular subtypes of KIRC are associ- ated with favorable prognosis [11]. Another notable pattern is the strong negative association between neutrophils and predicted prognosis in CESC (cervical squa- mous cell carcinoma and endocervical adenocarcinoma), also aligning with prior biomarker studies [3]. Other associations are less clear-cut, such as that between hypercellularity and longer predicted survival (second highest correlation across concepts). Hypercellularity can indicate high proliferation of cancer cells but may also reflect dense immune infiltration; inspection of example patches with high hypercellularity scores and high ABMIL attention revealed dense immune infiltrates (Fig. 3), potentially explaining the favorable association. 4.4 Dictionary Sufficiency: A Text-Only Survival Model A prerequisite for interpreting the concept scores as meaningful is that the 25- term dictionary itself captures prognostic signal. We tested this directly by train- ing an ABMIL survival model whose per-patch features are replaced by the patch-concept similarities to the dictionary (one channel per prompt template of each term, in CONCH’s similarity space). Despite this compression to an interpretable, concept-only representation, the text-only model is competitive across all seven cohorts, achieving an average C-index that matches the vision- based model (0.625; Table 2). This performance thus validates the dictionary as a meaningful prognostic basis for explaining vision-based ABMIL models. The comparable prognostic accuracy further suggests that the text-based model could serve as a standalone alternative approach more broadly. Semantic Explainability of Survival Models7 epithelial cells BRCABLCACESCCOADKIRCLGGLUAD hypercellularity endothelial cells dysplasia lymphocytes lymphoid aggregates hyperchromasia nucleolar enlargement plasma cells fibroblasts scarring adipocytes poor differentiation angiogenesis BRCABLCACESCCOADKIRCLGGLUAD macrophages eosinophils apoptosis lymphovascular invasion mitotic figures desmoplasia neutrophils perineural invasion pleomorphism cancer cells necrosis Fig. 3. Example patches for each concept. For each concept and cohort, a representative patch is sampled from slides with high concept scores and high ABMIL attention (top 10% for each; based on CONCH). 4.5 Attention vs. Zero-Shot Aggregation: An Ablation Our semantic scores weight patch-concept similarities by the survival model’s attention, so a natural ablation asks whether that weighting adds anything over uniform (equal-weight) mean similarity. We recomputed slide-level con- cept scores using uniform patch weights, and compared the resulting concept- predicted prognosis correlations. We find that the correlative trends exist with uniform weighting, and become sharpened with attention weighting, as illus- trated in Fig. 4a for KIRC and BRCA. Across all 175 concept-cohort pairs attention-weighting increases |r| for 75% of them (mean ∆|r| = +0.07; one- sided paired Wilcoxon p = 4× 10 −13 ). These results suggest that the attention- weighting better explains the model’s predictions, as expected given ABMIL’s formulation, and that the model may indeed be learning to focus on underlying discriminative features in the data. 4.6 Foundation Model Ablation: Does the Choice of VLM Matter? We repeat the analysis with two further encoders, UNI2 [1] and MUSK [15], on two cohorts (BRCA, the largest, and KIRC, the most prognostic). As UNI2 is a vision-only model, we use its features to train the ABMIL model, but score the attended patches against concepts in both CONCH’s and MUSK’s text-image spaces. The vision-based prognostic performance is similar between the three models (Table 3). The text-only representation, however, is competitive only for 8A. Lamane et al. Table 2. Prognostic performance on TCGA cohorts (test C-index, mean± std over 5 folds). The baseline ABMIL predictor based on CONCH vision embeddings aligns with performance reported in literature. Using the text-only similarity embeddings achieves similar performance. Bold marks the better of the two per cohort. Cohortn CONCH (vision)text-only BRCA 1014 0.612± 0.060.582± 0.07 BLCA3700.531± 0.090.584± 0.08 CESC2460.588± 0.090.592± 0.06 COAD 3980.626± 0.110.627± 0.08 KIRC4790.774± 0.030.760± 0.03 LGG4780.669± 0.110.633± 0.06 LUAD4260.577± 0.060.597± 0.05 Average0.6250.625 Table 3. Foundation model ablation. Test C-index (mean± std over 5 folds) is dis- played for BRCA and KIRC when using each model’s vision encoder, and when using the text-only embedding variant for CONCH and MUSK. Bold = best per column. BRCAKIRC Encodervisiontext-onlyvisiontext-only CONCH 0.612± 0.06 0.582± 0.07 0.774± 0.03 0.760± 0.03 UNI20.659± 0.07—0.751± 0.05— MUSK0.605± 0.11 0.527± 0.09 0.740± 0.03 0.595± 0.15 CONCH: collapsing each patch to MUSK’s 25-concept similarities drops con- cordance to 0.53/0.59 on BRCA/KIRC. This result suggests that the predictive value of the interpretable space hinges on the quality of the VLM’s text-image alignment, and supports the use of CONCH in our pipeline. Across all concepts, UNI2 tracks CONCH almost perfectly on KIRC (r = 0.98 in rank order of concepts) and moderately on BRCA (0.31), suggesting that different ABMIL predictors may rely on both shared and distinct features. MUSK, with its own vision-language space, is more distinct from CONCH but still exhibits positive rank correlation (r = 0.48/0.19 for KIRC/BRCA; see also Fig. 4b). 5 Discussion Limitations and future work. Concept associations are measured against the ABMIL model’s predictions, so they should be interpreted as hypothesis- generating rather than causal. While the use of a VLM provides a scalable means to assess concept similarity, the model may make errors and have non-uniform accuracy across concepts. The default list of 25 concepts may miss cancer-specific or subtle morphological features, so users are encouraged to customize this list for specific applications. We tested seven TCGA cohorts, a common prognostication benchmark, but future validation in other cohorts is important. Semantic Explainability of Survival Models9 0.50.00.5 pleomorphism desmoplasia mitotic figures perineural invasion cancer cells lymphovascular invasion necrosis poor differentiation eosinophils hyperchromasia neutrophils fibroblasts nucleolar enlargement apoptosis scarring macrophages angiogenesis adipocytes dysplasia lymphocytes plasma cells hypercellularity lymphoid aggregates epithelial cells endothelial cells KIRC 0.50.00.5 perineural invasion pleomorphism mitotic figures desmoplasia lymphovascular invasion necrosis poor differentiation cancer cells nucleolar enlargement apoptosis dysplasia neutrophils eosinophils hyperchromasia angiogenesis plasma cells epithelial cells macrophages scarring lymphocytes fibroblasts lymphoid aggregates hypercellularity adipocytes endothelial cells BRCA Correlation with prognosis prediction uniformattention-weighted favorable (r > 0)adverse (r < 0) 0.00.20.40.60.8 Spearman r nucleolar enlargement apoptosis cancer cells dysplasia epithelial cells poor differentiation hypercellularity endothelial cells necrosis lymphoid aggregates hyperchromasia macrophages pleomorphism lymphocytes mitotic figures plasma cells neutrophils perineural invasion eosinophils angiogenesis lymphovascular invasion fibroblasts scarring desmoplasia adipocytes KIRC BRCA (a)(b) Fig. 4. (a) Attention ablation. Spearman r between concept scores and prognosis pre- diction under uniform (open) vs. attention-weighted (filled) aggregation; bars = 95% bootstrap CI. Attention sharpens the associations present in the uniform baseline. (b) Similarity in patch-level concept scores between CONCH and MUSK. For each con- cept, the mean per-slide Spearman r between CONCH and MUSK patch-level scores is displayed, separately for BRCA and KIRC. Architectural/stromal concepts show higher concordance (top) than fine cytological ones (bottom). Conclusion. SAGE is a lightweight, plug-and-play framework that translates ABMIL attention into global concept associations. Applied across seven TCGA cohorts and three foundation models, the recovered profiles often align with biological knowledge and are strongest where the model is most accurate (notably KIRC). Through semantic explainability, SAGE may help foster physician trust in emerging prognostic applications while also advancing our understanding of the biological relationships between histological features and patient outcomes. Code Availability. Code is available at https://github.com/lotterlab/sage. Acknowledgments. W.L. acknowledges funding support from the Ellison Founda- tion, the Wong Family Award, the Louis B. Mayer Foundation, the National Institute of Biomedical Imaging and Bioengineering award R21EB035247, and the National Li- brary of Medicine award R01LM014775. Disclosure of Interests. The authors have no competing interests to declare. References 1. Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F.K., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L.L., Wang, J.J., Vaidya, A., Le, L.P., Gerber, G., Sahai, S., Williams, W., Mahmood, F.: Towards a general-purpose foundation model for computational pathology. Nat. Med. 30(3), 850–862 (Mar 2024) 10A. Lamane et al. 2. Diab, A.R., Karn, E.E., Wu, R., Ruiz, E.S., Lotter, W.: Leveraging founda- tion models for histological grading in cutaneous squamous cell carcinoma using PathFMTools. In: Machine Learning for Health Symposium. PMLR (May 2026) 3. He, M., Wang, Y., Zhang, G., Cao, K., Yang, M., Liu, H.: The prognostic sig- nificance of tumor-infiltrating lymphocytes in cervical cancer. J. Gynecol. Oncol. 32(3), e32 (May 2021) 4. Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning. PMLR (Jul 2018) 5. Jaume, G., Doucet, P., Song, A.H., Lu, M.Y., Almagro-Pérez, C., Wagner, S.J., Vaidya, A.J., Chen, R.J., Williamson, D.F.K., Kim, A., Mahmood, F.: HEST-1k: A dataset for spatial transcriptomics and histology image analysis. arXiv (2024) 6. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv (2014) 7. Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., Liang, P.: Concept bottleneck models. ICML (2020) 8. Lei, W., Tan, Y., Li, A., Chen, H., Tian, H., Li, R., Jiang, Z., Yan, F., Zhang, X., Zhang, S.: Unifying multiple foundation models for advanced computational pathology. arXiv (Mar 2025) 9. Liu, P., Ji, L., Gou, J., Fu, B., Ye, M.: Interpretable vision-language survival anal- ysis with ordinal inductive bias for computational pathology. International Con- ference on Learning Representations 2025, 99062–99090 (May 2025) 10. Lu, M.Y., Chen, B., Williamson, D.F.K., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G., Parwani, A.V., Zhang, A., Mahmood, F.: A visual-language foundation model for computational pathology. Nat. Med. (2024) 11. Motzer, R.J., Banchereau, R., Hamidi, H., Powles, T., McDermott, D., Atkins, M.B., Escudier, B., Liu, L.F., Leng, N., Abbas, A.R., Fan, J., Koeppen, H., Lin, J., Carroll, S., Hashimoto, K., Mariathasan, S., Green, M., Tayama, D., Hegde, P.S., Schiff, C., Huseni, M.A., Rini, B.: Molecular subsets in renal cancer determine outcome to checkpoint and angiogenesis blockade. Cancer Cell (Dec 2020) 12. Nguyen, A.T., Byeon, K., Kim, K., Kwak, J.T.: VLEER: Vision and language embeddings for explainable whole slide image representation. arXiv (Feb 2025) 13. Pollheimer, M.J., Kornprat, P., Lindtner, R.A., Harbaum, L., Schlemmer, A., Re- hak, P., Langner, C.: Tumor necrosis is a new promising prognostic factor in col- orectal cancer. Hum. Pathol. 41(12), 1749–1757 (Dec 2010) 14. Sun, S., Tessier, L., Meeuwsen, F., Grisi, C., van Midden, D., Litjens, G., Baum- gartner, C.F.: Label-free concept based multiple instance learning for gigapixel histopathology. arXiv [cs.CV] (Jan 2025) 15. Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., Li, Y., Bergstrom, C., Gopaulchan, M., Kim, T., Yu, K.H., Willens, S., Olguin, F.M., Nirschl, J.J., Neal, J., Diehn, M., Yang, S., Li, R.: A vision-language foundation model for precision oncology. Nature 638(8051), 769–778 (Feb 2025) 16. Zadeh, S.G., Schmid, M.: Bias in cross-entropy-based training of deep survival networks. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 3126–3137 (Sep 2021) 17. Zhang, L., Zha, Z., Qu, W., Zhao, H., Yuan, J., Feng, Y., Wu, B.: Tumor necrosis as a prognostic variable for the clinical outcome in patients with renal cell carcinoma: a systematic review and meta-analysis. BMC Cancer 18(1), 870 (Sep 2018) 18. Zhao, W., Guo, Z., Fan, Y., Jiang, Y., Yeung, M.C.F., Yu, L.: Aligning knowledge concepts to whole slide images for precise histopathology image analysis. NPJ Digit. Med. 7(1), 383 (Dec 2024)

Instant research discovery