Paper deep dive
CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs
Naren Akash, Arihanth Tadanki, Jayanthi Sivaswamy
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/31/2026, 4:13:09 AM
Summary
The paper introduces CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. It utilizes graph transformers to extract features from specific anatomical regions and models inter-region relationships using learnable edge embeddings. CheXtriev outperforms state-of-the-art global and local methods in retrieval accuracy and ranking quality, particularly for less prevalent findings, by leveraging evidence-based anatomical context.
Entities (21)
Relation Signals (19)
Arihanth Tadanki β affiliatedwith β IIIT-Hyderabad
confidence 95% Β· Affiliation: International Institute of Information Technology Hyderabad, India
Naren Akash β affiliatedwith β IIIT-Hyderabad
confidence 95% Β· Affiliation: International Institute of Information Technology Hyderabad, India
Jayanthi Sivaswamy β affiliatedwith β IIIT-Hyderabad
confidence 95% Β· Affiliation: International Institute of Information Technology Hyderabad, India
CheXtriev β targetsfinding β Lung Opacity
confidence 95% Β· we focus on nine specific findings: lung opacity (LO)
CheXtriev β targetsfinding β Consolidation
confidence 95% Β· consolidation (CONS)
CheXtriev β targetsfinding β Pneumothorax
confidence 95% Β· pneumothorax (PTX)
CheXtriev β targetsfinding β Pleural Effusion
confidence 95% Β· pleural effusion (PE)
CheXtriev β targetsfinding β Atelectasis
confidence 95% Β· atelectasis (AT)
CheXtriev β targetsfinding β Enlarged Cardiac Silhouette
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18% to 26% in retrieval accuracy and 11% to 23% in ranking quality. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.28137v1
- Canonical: https://arxiv.org/abs/2608.28137v1
Trouble viewing inline? Open PDF directly β
Full Text
32,754 characters extracted from source content.
Expand or collapse full text
00footnotetext: N. Akash and A. Tadanki contributed equally. CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs Supplementary Material Naren Akash R J Arihanth Tadanki Jayanthi Sivaswamy E-mail https://cvit.iiit.ac.in/mip/projects/chextriev Affiliation: Center for Visual Information Technology, International Institute of Information Technology Hyderabad, India E-mail naren.akash@research.iiit.ac.in Affiliation: International Institute of Information Technology Hyderabad, India Abstract We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18%18\% to 26%26\% in retrieval accuracy and 11%11\% to 23%23\% in ranking quality. The code is available at https://github.com/cvit-mip/chextriev. Keywords: Case-based Retrieval Graph Transformer Radiographs 1 Introduction In clinical practice, expertise accumulates with experiences [1]. Clinicians often use case-based reasoning [2] to understand and diagnose patients by drawing parallels between past cases and current presentations. This analogy-driven approach hinges on effectively retrieving relevant past cases, making medical image retrieval (MIR) a cornerstone of evidence-based clinical decision-making. MIR facilitates the identification of similar past cases, allowing clinicians to: (i) refine diagnosis by suggesting investigative features or proposing alternative diagnoses, (i) optimize treatment planning based on past outcomes, and (i) enhance explanation and failure recovery by identifying clinically relevant past errors and their corrective actions [3]. As such, robust MIR systems hold significant potential to improve patient outcomes and advance medical care. Despite its promise, MIR presents distinct challenges [4]. Unlike general-purpose image retrieval, which focuses on global image regions, MIR has to contend with images that exhibit similar global features across patients, with subtle, fine-grained markers serving as critical disease indicators. This is particularly evident in chest radiographs, the most common yet challenging imaging modality. Its interpretation is prone to errors due to the two-dimensional projection of three-dimensional structures, leading to the superimposition of anatomies, low-contrast features, and multiple subtle, non-specific abnormalities. Studies suggest error rates as high as 30% in patients with abnormal findings and with up to 56% disagreement among radiologists [5]. MIR, especially for cases with multiple findings, remains understudied despite major strides in computer vision for multi-label settings [6]. Automated case-based retrieval of chest radiographs necessitates learning discriminative yet informative representations that can help bridge the gap between low-level visual features and high-level clinical interpretations. Hashing, widely used for computational and storage efficiency, maps high-dimensional features to compact hash codes. Advances in deep learning offer end-to-end training of deep hashing models such as DRH [7], OSDH [8] and SH-EBM [9] for multi-label chest radiographs, automatically learning both features and hash codes. However, they often lose crucial information related to disease classification and the regions of interest. This has been addressed recently with attention-based triplet hashing (ATH) [1] to implicitly focus on specific regions. ATH still relies on global image representation, potentially limiting its ability to capture fine-grained clinical information. Other emerging cross-modal approaches, such as MMDL [11] and X-TRA [12], incorporate text reports. Existing MIR approaches use image representations which do not fully leverage radiologistsβ systematic approach [13] to read chest radiographs, where the anatomical location of observed findings is often used for differential diagnosis. While focusing on anatomy has shown promise in classification (AnaXNet [2]), change detection (CheXRelNet [15]), and report generation (RGRG [16]), to our best knowledge, no work has attempted to incorporate anatomical reasoning into the retrieval process itself. In this paper, we focus on MIR for chest radiographs aiming to find relevant cases solely from visual content. Our approach addresses the limitations of current multi-label retrieval methods by leveraging anatomy-aware representation learning. Unlike existing methods that rely on global features, we explicitly target the subtle, yet critical informative details from specific anatomical regions and fuse them to learn a richer representation that incorporates the interplay between anatomical location and radiological findings. This contextualization, grounded in evidence-based anatomy[17], leads to more accurate, effective and efficient retrieval compared to methods using solely global features. Furthermore, we incorporate anatomy-aware saliency to highlight regions contributing to the retrieval decision to build model understanding and transparency. 2 Method Figure 1: A schematic illustration of CheXtriev. Case-based chest radiograph retrieval systems identify images from a collection which are similar to a given query image. This is achieved by analyzing the visual content of the radiographs as well as focusing on the presence and location of anatomical abnormalities. In this work, we propose a system called CheXtriev, which first extracts informative features from anatomically defined regions and constructs a graph where nodes represent regions and edges capture spatial relationships and potential co-occurrences between findings. The graph leverages learnable location and edge embeddings to capture spatial context and relationships between regions, allowing a graph transformer architecture with global edge-aware attention and gated residuals to learn robust global-local context representations for accurate, effective and efficient retrieval. An overview of our proposed pipeline is shown in Figure 1. 2.1 Thoracic Radioanatomic Region Representation Extraction Radiologists systematically analyse chest radiographs by sequentially assessing anatomical structures and interfaces to precisely detect abnormalities. Inspired by this, we identify eighteen essential anatomical regions in frontal radiographs. These regions include the right and left lungs, encompassing their apical, upper, mid, and lower zones, as well as the right and left hilar structures, costophrenic angles, mediastinum (and its upper portion), cardiac silhouette, and trachea. We obtain the anatomical region representations R=rii=1NββNΓDRR=\r_i\_i=1^N ^NΓ D_R, where N is the number of regions and each rir_i of dimension DRD_R is extracted by a fixed, pretrained ResNet50 feature extractor after the last global average pooling layer. 2.2 Modeling Inter-Region Relationships with Graph Transformer The complex diagnostic reasoning process of a radiologist relies on anatomical interdependence, spatial pattern recognition, and disease co-occurrence. We design a novel graph transformer [18] framework, to capture both global and local contexts and learn latent relationships between different regions. Feature projection and location embeddings. We project each of the regionβs features riββDRr_i ^D_R with a two-layer perceptron (MLP) with one ReLU non-linearity. We learn N location embedding vectors ELββDRΓNE^L ^D_RΓ N to distinguish between one anatomical region from the other. Combining the features with corresponding embedding vectors provides a rich representation r~i=MLPβ(ri)+EiL r_i=MLP(r_i)+E^L_i. Edge-aware graph attention. To model anatomical relationships, we construct a graph with learnable continuous edges to capture inter-region dependencies. Each edge in the graph connects two regions, denoted as j and i (neighbour and reference region, respectively). We calculate C-way multi-head attention that incorporates both region features and learnable edge information as follows. Reference region features hih_i and neighbouring region features hjh_j are transformed into query qc,i(l)ββDTq_c,i^(l) ^D_T and key kc,j(l)ββDTk_c,j^(l) ^D_T vectors using trainable weights Wc,q(l)W_c,q^(l), Wc,k(l)W_c,k^(l) and biases bc,q(l)b_c,q^(l) and bc,k(l)b_c,k^(l). qc,i(l)=Wc,q(l)βhi(l)+bc,q(l)q_c,i^(l)=W_c,q^(l)h_i^(l)+b_c,q^(l); kc,j(l)=Wc,k(l)βhj(l)+bc,k(l)k_c,j^(l)=W_c,k^(l)h_j^(l)+b_c,k^(l). Here, l corresponds to ltβhl^th layer. Edge features eiβje_ij are learnt and encoded and added into the key vector as additional information for each layer, allowing the model to capture context-specific latent relationships beyond inherent node features: ec,iβj=Wc,eβeiβj+bc,e_c,ij=W_c,ee_ij+b_c,e and Ξ±c,iβj(l)=β¨qc,i(l),kc,j(l)+ec,iβjβ©βuββ‘(i)β¨qc,i(l),kc,u(l)+ec,iβuβ© _c,ij^(l)= q_c,i^(l),k_c,j^(l)+e_c,ij Ξ£_u (i) q_c,i^(l),k_c,u^(l)+e_c,iu , where β¨q,kβ©=expβ‘(qTβkDT) q,k = ( q^Tk D_T ) is exponential scale cosine similarity function and DTD_T is the hidden size of each head. We calculate a weighted sum of neighbouring region features, incorporating both node and edge information, and make a message aggregation to the reference region as: vc,j(l)=Wc,v(l)hj(l)+bc,v(l);h^i(l+1)=β₯c=1Cβjββ‘(i)Ξ±c,iβj(l)(vc,j(l)+ec,iβj)v_c,j^(l)=W_c,v^(l)h_j^(l)+b_c,v^(l);\ \ h_i^(l+1)=\|_c=1^C _j (i) _c,ij^(l)(v_c,j^(l)+e_c,ij). Multi-level feature aggregation. We introduce shared-source gated residual connections to selectively aggregate global-local context-aware features as: gi(l)=Wr(l)βhi(l)+br(l)g_i^(l)=W_r^(l)h_i^(l)+b_r^(l); Ξ²i(l)=sigmoidβ(WgOPEN(l))β[h^i(l);ri(l);h^i(l+1)βri(l)]) _i^(l)=sigmoid(W_g^(l))[ h_i^(l);r_i^(l); h_i^(l+1)-r_i^(l)]); hi(l+1)=ReLUβ(LayerNormβ((1βΞ²i(l))βh^i(l+1)+Ξ²i(l+1)βhi(0))).h_i^(l+1)=ReLU(LayerNorm((1- _i^(l)) h_i^(l+1)+ _i^(l+1)h_i^(0))). The aggregated, hierarchical representation incorporates more comprehensive, fine-grained information at different levels of granularity to better detect abnormalities among different regions. Classifier. We average the multi-head output on the last layer L of graph transformer and remove non-linearities: h^i(L+1)=1Cββc=1Cβjββ‘(i)Ξ±c,iβj(L)β(vc,j(L)+ec,iβj) h_i^(L+1)= 1C _c=1^C _j (i) _c,ij^(L)(v_c,j^(L)+e_c,ij); hi(L+1)=(1βΞ²i(L))βh^i(L+1)+Ξ²i(L)βgi(L)h_i^(L+1)=(1- _i^(L)) h_i^(L+1)+ _i^(L)g_i^(L). We feed the mean-pooled features into shared linear layers for multi-label classification using class-wise binary cross-entropy loss. We perform normalization on the final layer features to obtain dense vectors that capture anatomy-aware representations for similarity search. 2.3 Efficient Retrieval and Visual Interpretability for Similarity To efficiently retrieve relevant chest radiographs, we adopt a fast and exact k-nearest neighbour retrieval model using FAISS [19]. It builds a flat index for our encoded, dense data embeddings, enabling efficient inner product similarity search with the query vector. This non-parametric approach effectively identifies radiographs with the highest cosine similarity from the pre-built vector database. In order to visually interpret the results, we adopt an occlusion-based method [20] and derive similarity-based saliency maps. Instead of using small, sliding occlusions, we directly occlude pre-defined anatomical regions within the retrieved image. We then calculate the cosine similarity between the query image and each occluded version. Regions causing significant drops in cosine similarity are deemed critical for overall similarity, while those with minimal change are considered less important. This analysis is visualized as a saliency map, highlighting the most influential regions for similarity between the query and retrieved image. 3 Experiments 3.1 Data We use the MIMIC-CXR-JPG v2.0.0 [5] database for all our experiments and follow the official MIMIC-CXR [4] data splits. We select the frontal chest radiograph images (PA or AP view) with valid bounding box coordinates for all eighteen anatomical regions. These annotations are provided by the Chest ImaGenome v1.0.0 [3] dataset, which is constructed from MIMIC-CXR. This resulted in 226,473 training, 1,863 validation, and 3,191 testing images. Similar to AnaXNet [2], we focus on nine specific findings: lung opacity (LO), pleural effusion (PE), atelectasis (AT), enlarged cardiac silhouette (ECS), pulmonary edema/hazy opacity (PE/HO), pneumothorax (PTX), consolidation (CONS), fluid overload/heart failure (FO/HF) and pneumonia (PN). We consider a radiograph positive for a finding if any region is labelled positive for that finding. We evaluate the retrieval performance by querying each test radiograph with findings against the remaining test set as the database. 3.2 Experimental Settings In our experiments, we resize all chest radiographs to 224Γ224224Γ 224. The graph transformer consists of 2 layers with 8 multi-head attention heads and the model dimension is set to DT=64D_T=64. We utilize the Adam optimizer with a learning rate of 1βeβ41e-4 in PyTorch Lightning. We incorporate early stopping based on validation loss with patience of 44 evaluations. Training is limited to a maximum of 3030 epochs with a batch size of 1616 and gradient accumulation occurs every 88 epochs. Further, we use a learning rate schedule with a reduction factor of 0.10.1 and set gradient clipping to 0.50.5. The training process was distributed across four NVIDIA GeForce RTX-20802080 Ti GPUs. Metrics. We use three metrics [1] to assess the precision and quality of retrieval: (i) Average Precision (AP) measures the average position of relevant cases within the retrieved list, thus evaluating the ranking quality, (i) Hit Ratio (HR) indicates the proportion of retrieved cases that are relevant, thus reflecting the retrieval effectiveness, (i) Reciprocal Rank (R) is the reciprocal of the rank of the first relevant case in the returned list, assessing the retrieval efficiency. 4 Results and Discussion Various variants of the proposed solution, namely, CheXtriev, were constructed to assess the contributions of various components. First, we present the evaluation results of these variants. Then, we highlight the strengths of CheXtriev, such as learning from anatomy-aware features via a local approach using the results of benchmarking experiments. Table 1: Impact of design choices in CheXtriev on retrieval and ranking performance. Here, IRM and MLF denote inter-anatomic region modelling using graph transformers and multi-level features with gated residuals, respectively. Variants IRM Edge Connectivity Location MLF mAP mHR mRR V0 β β β β 51.8 39.7 54.0 V1 GT Shared Binary β Global 52.9 39.9 55.2 V2 GT Shared Uniform β Global 53.9 40.8 56.3 V3 GT Shared Uniform Learnable Global 54.0 40.9 56.4 V4 GT Unique RiβRjR_i-R_j Learnable Local 51.8 39.8 53.7 V5 GT Unique RiβRjR_i-R_j β Global 54.6 40.9 57.1 V6 GT Unique RiβRjR_i-R_j Learnable Global 55.1 40.7 57.4 CheXtriev variants. The performance results of the variants are presented in Table 1. The baseline variant (V0) solely relies on the mean pooling of extracted region features and achieves 51.8%51.8\% mAP, 39.7%39.7\% mHR, and 54.0%54.0\% mRR. The consistent performance boost observed from V1 to V6 (in global MLF), ranges from +2.1%+2.1\% (in mAP for V1) to +6.4%+6.4\% (for V6); this underscores the advantage of graph transformers with fully connected unique learnable edges over uniform edge-sharing schemes and naive handcrafted adjacency. This result also suggests that the retrieval task benefits from considering all region pairs, potentially capturing intricate latent relationships between regions. It can be seen that local gated residual connections (V4) lead to a significant drop in performance (6.38%6.38\% lower mAP and 6.89%6.89\% lower mRR) relative to a global one (V6). This emphasizes the value of global gated residual connections with selective refinement for learning multi-level features. The fact that V6 and V3 outperform V5 and V2, respectively, suggests that the learnable location embedding improves model performance by capturing crucial spatial context for accurate ranking. Assessment of anatomy-aware feature extraction. A trend that can be observed from Table 1 is that relative to the baseline V0 which lacks inter-region relationship modelling, all graph transformer (GT) variants have a modest but consistent improvement in identifying relevant cases (similar mHR). This suggests that, in general, anatomy-aware feature extraction is more effective (than a strategy that does not consider anatomical relationships) at retrieving relevant cases, and the design of CheXtriev (V6) gives it an additional edge to precisely rank and retrieve these cases (higher mAP and mRR). Table 2: Comparison of top-5 retrieval performance on MIMIC-CXR dataset against global baselines (CNN, ATH) and a local variant (AnaXNet). (.)β(.) indicates p<0.05p<0.05. Global CNN ATH [1] AnaXNet [2] CheXtriev Findings AP HR R AP HR R AP HR R AP HR R LO 90.4β90.4 85.9β85.9 92.2 88.2β88.2 82.2β82.2 90.9 88.8β88.8 82.9β82.9 91.2 91.7 87.6 93.3 PE 68.6β68.6 54.7β54.7 72.6 62.1β62.1 46.3β46.3 66.1 63.7β63.7 46.2β46.2 67.2 71.4 59.4 74.7 AT 60.3β60.3 44.1β44.1 64.2 54.3β54.3 36.8β36.8 57.8 59.1β59.1 42.3β42.3 62.8 64.4 49.1 67.6 ECS 64.7β64.7 48.3β48.3 67.9 61.3β61.3 45.4β45.4 64.9 64.7β64.7 48.5β48.5 68.4 69.4 56.0 73.7 PE/HO 56.5β56.5 40.2β40.2 59.2 51.8β51.8 34.7β34.7 54.7 53.5β53.5 34.9β34.9 57.4 62.5 47.8 65.8 PTX 22.4β22.4 8.5β8.5 22.7 8.9β8.9 4.0β4.0 9.1 31.1 12.7 31.8 31.5 12.5 32.4 CONS 24.2β24.2 13.2 25.2 23.3β23.3 11.8β11.8 23.8 27.6 12.5 28.3 31.6 14.6 32.7 FO/HF 14.8β14.8 7.3β7.3 14.8 15.8β15.8 7.0β7.0 16.2 17.8β17.8 7.2β7.2 17.9 28.8 11.8 29.3 PN 39.0β39.0 23.0β23.0 41.3 37.1β37.1 21.9β21.9 38.8 41.4β41.4 23.6β23.6 43.4 44.7 27.1 47.2 Mean 49.0 36.1 51.1 44.8 32.2 46.9 49.7 34.5 52.0 55.1 40.7 57.4 wMean 67.0 55.2 69.6 63.1 50.5 66.1 65.7 52.2 68.6 70.8 59.5 73.4 Figure 2: Retrieval performance and saliency map analysis for a sample query image with enlarged cardiac silhouette (ECS). Each row displays the top 4 retrieved images and their corresponding occlusion-based saliency maps generated by CheXtriev, AnaXNet, ATH and Global CNN (top to bottom). A retrieved image is considered correct if it matches the specific finding of interest, in this case ECS, as the query image. Assessment of global vs. local approaches. We compare the performance results of CheXtriev, against both global and local methods. Table 3 presents these results for top five retrieved images. Two global baselines are considered, one based on ResNet-50 extracted dense features (column 1) and ATH [1], a SOTA retrieval approach (column 2). A studentβs t-test was done to establish the statistical significance of the difference in results between CheXtriev and the baselines, and the significant values (p<0.05)(p<0.05) are marked with an asterisk. Relative to both the global baselines, CheXtriev has higher AP values across all nine investigated findings, with these ranging from 91.7% (LO) to 28.8% (FO/HF). This trend also holds for HR and R metrics. Both macro-mean and weighted mean metrics are reported in the last two rows of Table 3 to draw insights into the overall retrieval efficacy from an unbalanced database. Macro-mean assigns equal weightage to all findings, which may introduce a bias to frequently occurring classes. Hence, the weighted mean assigns weights to classes inversely proportional to their frequency. It can be observed that the weighted mean is higher than the macro mean for all metrics. The mAP improvement over the baselines ranges from at least 12% (first column) to a notable 23% (second column). The noteworthy point is that the improvement is quite significant for classes with lower prevalence, such as FO/HF (+82.3% to +94.6%), PTX (+40.6% to +253.9%), CONS (+30.6% to +35.6%), PN (+14.6% to +20.5%), demonstrating CheXtrievβs ability to learn more discriminative and powerful visual representations compared to global methods. These results point to CheXtrievβs ability to accurately rank and efficiently retrieve relevant cases, even for less frequent classes. We also compare CheXtriev against AnaXNet, a SOTA local approach designed for classification tasks. CheXtriev exhibited significant improvements in mean AP, particularly for findings associated with known blind spots [24, 25] in chest radiographs, such as lung apices, hilar structures and inferior lung bases (for example, FO/HF +61.80%, PE/HO +16.82%, CONS +14.49% higher). This can be attributed to CheXtrievβs global edge-aware attention mechanism and fully connected learnable continuous edges, which enable it to capture anatomical relationships and latent finding co-occurrences more effectively than AnaXNetβs naive handcrafted binary edges and limited expressiveness from neighborhood aggregation. Additionally, CheXtrievβs location embeddings and gated residual connections have aided in learning superior visual representations, building upon the strengths of local approaches to capture well the subtle abnormalities in areas prone to human error during interpretation. The results also indicate classification-optimized features may not be the most effective for retrieval. Interpretability analysis. We analyze saliency maps [20] to assess which anatomical regions influenced retrieval for each query image (see Figure 2). The first three retrieved images by CheXtriev all exhibit ECS, with the saliency maps highlighting a focus on the cardiac silhouette region. However, in the saliency map for the fourth retrieved image, the modelβs attention diffused throughout the lung region, indicating an incorrect retrieval. Some images retrieved by AnaXNet lack any findings for ECS. Even though ATH and CNN manage to retrieve correct cases, the corresponding saliency maps highlight irrelevant regions. This suggests other models might be overly sensitive to visual changes in irrelevant parts of the image, potentially due to weaknesses in encoding spatial information and context within the image. We also note that these saliency maps may not be meaningful for methods that do not use anatomy-aware features, underscoring the additional benefits of our approach. 5 Conclusion In this work, we propose CheXtriev, a graph-based radiograph retrieval model inspired by the systematic approach radiologists use to interpret radiographs and grounded in evidence-based anatomy. Key novel features of our approach were explicitly targeting informative details from specific anatomical regions, modelling the interplay between anatomical location and findings and fusing them into a richer anatomy-aware representation. Superior results over existing methods underscore the benefit of this contextualization in achieving more accurate, effective, and efficient case retrieval, particularly for the less prevalent findings. A preliminary analysis with anatomy-aware saliency maps indicates that it may be possible to use them, albeit with some further extension to cover multiple findings, for interpretability of the retrieved results. Overall, CheXtriev offers a promising approach for medical image retrieval tasks, particularly in chest radiography, where subtle anatomical variations hold significant diagnostic value. Acknowledgements We thank Dr. L.T. Kishore for his insights on how radiologists analyze and interpret chest radiographs in clinical practice. Disclosure of Interests. There are no conflicts of interest to declare. References [1] Kolodner, Janet L.: The Role of Experience in Development of Expertise. In: Annual AAAI Conference on Artificial Intelligence (1982). [2] Slade, Stephen: Case-based Reasoning: A Research Paradigm. In: AI Magazine, 12(1), 42β42 (1991). [3] Kolodner, Janet L., et al.: Using Experience in Clinical Problem Solving: Introduction and Framework. IEEE Transactions on Systems, Man, and Cybernetics, 17(3), 420β431 (1987). [4] Li, Zhongyu, et al.: Large-scale Retrieval for Medical Image Analytics: A Comprehensive Review. Medical image analysis 43, 66β84, (2018). [5] Gefter, Warren B., et al.: Commonly Missed Findings on Chest Radiographs: Causes and Consequences. Chest, 163(3), 650β661 (2023). [6] Rodrigues, Josiane, et al.: Deep Hashing for Multi-label Image Retrieval: A Survey. Artificial Intelligence Review, 53, 5261β5307 (2020). [7] Conjeti, Sailesh, et al.: Hashing with Residual Networks for Image Retrieval. In: Medical Image Computing and Computer Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2017). [8] Chen, Zhixiang, et al.: Order-Sensitive Deep Hashing for Multimorbidity Medical Image Retrieval. In: Medical Image Computing and Computer Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2018). [9] Huang, Peng, et al.: Energy-Based Supervised Hashing for Multimorbidity Image Retrieval. In: Medical Image Computing and Computer Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2021). [10] Fang, Jiansheng, et al.: Deep Triplet Hashing Network for Case-based Medical Image Retrieval. Medical Image Analysis 69, 101981 (2021). [11] Yu, Yang, et al.: Multimodal Multitask Deep Learning for X-Ray Image Retrieval. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2021). [12] van Sonsbeek, Tom, et al.: X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation. In: International Conference on Information Processing in Medical Imaging - IPMI, LNCS. Springer, Cham. (2023). [13] Raoof, Suhail, et al.: Interpretation of Plain Chest Roentgenogram. Chest, 141(2), 545β558 (2012). [14] Agu, Nkechinyere N., et al.: AnaXNet: Anatomy Aware multi-label Finding Classification in Chest X-ray. In: Medical Image Computing and Computer Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2021). [15] Karwande, Gaurang, et al.: CheXRelNet: An Anatomy-Aware Model for Tracking Longitudinal Relationships Between Chest X-rays. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2022). [16] Tanida, Tim, et al. Interactive and Explainable Region-guided Radiology Report Generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2023). [17] Yammine, Kaissar: Evidence-based Anatomy. Clinical Anatomy, 27(6), 847β852, (2014). [18] Dwivedi, Vijay Prakash, et al.: A Generalization of Transformer Networks to Graphs. In: AAAI 2021 Workshop on Deep Learning on Graphs: Methods and Applications (2020). [19] Douze, Matthijs, et al.: The Faiss Library. arXiv preprint arXiv:2401.08281 (2024). [20] Hu, Brian, et al.: X-MIR: Explainable Medical Image Retrieval. In IEEE/CVF Winter Conference on Applications of Computer Vision (2022). [21] Johnson, Alistair, et al.: MIMIC-CXR-JPG, A Large Publicly Available Database of Labeled Chest Radiographs. arXiv preprint arXiv:1901.07042 (2019). [22] Johnson, Alistair, et al.: MIMIC-CXR, a De-identified Publicly Available Database of Chest Radiographs with Free-text Reports. Scientific Data, 6(1), 317 (2019). [23] Wu, Joy T, et al.: Chest Imagenome Dataset for Clinical Reasoning. In: Advances in Neural Information Processing Systems (2021). [24] Ropp, Alan, et al. Did I Miss That: Subtle and Commonly Missed Findings on Chest Radiographs. Current Problems in Diagnostic Radiology, 44(3), 277β289 (2015). [25] de Groot, Patricia M., et al. Pitfalls in Chest Radiographic Interpretation: Blind Spots. Seminars in Roentgenology, 50(3), 197β209, (2015). CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs 00footnotetext: N. Akash and A. Tadanki contributed equally. Naren Akash R J Arihanth Tadanki Jayanthi Sivaswamy Table 3: Comparison of top-3 retrieval performance on MIMIC-CXR dataset against global baselines (CNN, ATH) and a local variant (AnaXNet). (.)β(.) [rgb]0,0,1 indicates p<0.05p<0.05. Global CNN ATH [1] AnaXNet [2] CheXtriev Findings AP HR R AP HR R AP HR R AP HR R LO 91.4β 85.8β 91.9β 89.7β 82.5β 90.6β 90.7β 84.0β 91.7β 92.5 88.2 93.0 PE 70.0β 56.2β 71.2 62.6β 47.6β 63.9β 64.8β 48.6β 65.7β 72.1 59.4 73.0 AT 60.8β 45.2β 61.9β 53.9β 36.8β 54.5β 59.7β 42.5β 61.1β 64.9 48.5 65.8 ECS 64.5β 49.3β 65.6β 61.6β 45.9β 62.5β 64.5β 48.3β 65.2β 70.5 55.4 71.7 PE/HO 55.3β 40.7β 56.1β 51.0β 36.0β 51.7β 53.7β 36.3β 54.5β 64.0 49.1 65.2 PTX 20.7β 11.5β 21.0β 7.6β 4.2β 7.8β 32.0 16.3 32.1 29.7 14.9 29.7 CONS 21.6β 12.9 21.8β 20.9β 13.0β 21.1β 25.6 15.0 25.8 27.2 15.9 27.7 FO/HF 11.2β 6.5β 11.3β 13.6β 7.3β 13.7β 15.0β 7.7β 15.1β 26.3 13.6 26.4 PN 37.1β 22.8β 37.5β 34.3β 21.4β 34.6β 39.4β 26.2β 39.7β 44.8 28.2 45.4 Mean 48.1 36.8 48.7 43.9 32.7 44.5 49.5 36.1 50.1 54.7 41.5 55.3 wMean 67.1 55.7 67.9 63.2 51.0 64.0 66.3 53.4 67.1 71.3 59.8 72.0 Table 4: Comparison of top-10 retrieval performance on MIMIC-CXR dataset against global baselines (CNN, ATH) and a local variant (AnaXNet). (.)β(.) [rgb]0,0,1 indicates p<0.05p<0.05. Global CNN ATH [1] AnaXNet [2] CheXtriev Findings AP HR R AP HR R AP HR R AP HR R LO 89.0β 86.1β 92.2β 86.2β 82.0β 90.9β 87.3β 83.5β 91.9β 90.5 87.7 93.2 PE 64.4β 54.1β 73.2 58.2β 45.2β 67.2β 59.3β 46.4β 68.9β 67.6 57.5 75.1 AT 55.7β 43.5β 65.0β 50.5β 36.1β 59.2β 55.1β 41.4β 65.0β 59.7 47.6 68.6 ECS 59.8β 48.5β 68.5β 57.3β 43.8β 65.7β 59.5β 46.1β 68.3β 64.6 53.1 74.1 PE/HO 52.3β 39.1β 60.2β 48.4β 32.8β 56.1β 50.2β 34.0β 59.1β 59.2 46.3 68.0 PTX 23.6 6.8β 24.6β 10.3β 4.3β 11.3β 32.1 9.5β 35.3 31.4 11.5 35.4 CONS 26.3β 13.2 28.9 24.8β 10.8β 26.2β 27.2 11.3 30.7 30.7 14.1 33.8 FO/HF 16.5β 6.7β 16.9β 16.6β 6.7β 18.3β 18.9β 6.3β 20.3β 29.8 10.7 32.1 PN 38.1β 22.3β 44.2β 36.2β 22.2β 41.6β 39.5β 22.4β 45.5β 42.8 22.5 50.7 Mean 47.3 35.6 52.6 43.2 31.5 48.5 47.7 33.4 53.9 52.9 39.3 59.0 wMean 64.2 54.9 70.5 60.5 49.7 67.1 62.8 51.7 69.9 67.9 58.3 74.3 Table 5: Statistics of the Chest ImageGenome [3] test dataset, which includes frontal chest radiographs (PA or AP view) with valid bounding box annotations for 18 anatomical regions. The data split is based on the official MIMIC-CXR [4, 5] data splits. Radiological Findings #Radiographs Lung Opacity 2679 Pleural Effusion 1340 Atelectasis 1241 Enlarged Cardiac Silhouette 1212 Pulmonary Edema / Hazy Opacity 819 Pneumonia 643 Consolidation 334 Fluid Overload / Heart Failure 169 Pneumothorax 96 References [1] Fang, Jiansheng, et al.: Deep Triplet Hashing Network for Case-based Medical Image Retrieval. Medical Image Analysis 69, 101981 (2021). [2] Agu, Nkechinyere N., et al.: AnaXNet: Anatomy Aware multi-label Finding Classification in Chest X-ray. In: Medical Image Computing and Computer Assisted Intervention - MICCAI, LNCS. Springer, Cham. (2021). [3] Wu, Joy T, et al.: Chest Imagenome Dataset for Clinical Reasoning. In: Advances in Neural Information Processing Systems (2021). [4] Johnson, Alistair, et al.: MIMIC-CXR, a De-identified Publicly Available Database of Chest Radiographs with Free-text Reports. Scientific Data, 6(1), 317 (2019). [5] Johnson, Alistair, et al.: MIMIC-CXR-JPG, A Large Publicly Available Database of Labeled Chest Radiographs. arXiv preprint arXiv:1901.07042 (2019).