Paper deep dive
From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
Haowen Gu, Gensheng Pei, Junzhu Mao, Qiong Wang, Mingwu Ren, Yazhou Yao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/28/2026, 4:18:17 AM
Summary
The paper introduces MedREAL, a unified framework for Medical Visual Question Answering (Med-VQA) and segmentation that bridges the gap between high-level clinical reasoning and pixel-level spatial grounding. It utilizes a Seg Anchored Reasoning Pooling (SARP) module to distill semantic evidence from MLLM hidden states using a [SEG] token, and a Reasoning-to-Visual (R2V) fusion mechanism to inject these features into a segmentation pipeline. The authors also present MedRAVS-13K, a new dataset of 13,824 samples across four imaging modalities, and demonstrate that MedREAL achieves state-of-the-art performance with 68.49% gIoU and 70.47% cIoU.
Entities (12)
Relation Signals (10)
MedREAL → usesdataset → MedRAVS-13K
confidence 97% · To facilitate this paradigm, we construct MedRAVS-13K... Extensive experiments demonstrate that MedREAL significantly outperforms state-of-the-arts...
MedREAL → achievesmetric → cIoU 70.47%
confidence 95% · MedREAL significantly outperforms state-of-the-arts, achieving ... 70.47% cIoU...
MedREAL → achievesmetric → gIoU 68.49%
confidence 95% · MedREAL significantly outperforms state-of-the-arts, achieving 68.49% gIoU...
MedREAL → containsmodule → SARP
confidence 95% · MedREAL introduces Seg Anchored Reasoning Pooling (SARP) to distill task-relevant semantic evidence...
MedREAL → containsmodule → R2V
confidence 95% · a Reasoning-to-Visual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features...
MedRAVS-13K → generatedby → Qwen3-VL-235B-A22B-Thinking
confidence 92% · We leverage the advanced reasoning capabilities of Qwen3-VL-235B-A22B-Thinking to generate clinically accurate question-answering (QA) pairs.
MedRAVS-13K → includessource → ISIC 2018
confidence 90% · We augment four heterogeneous medical imaging datasets (i.e., ... ISIC-2018 [12]...)
MedRAVS-13K → includessource →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question Answering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose \textsc{\textsc{MedREAL}} (\textbf{Med}ical \textbf{RE}asoning-driven \textbf{A}nswering and \textbf{L}ocalization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, \textsc{MedREAL} introduces \textbf{S}eg \textbf{A}nchored \textbf{R}easoning \textbf{P}ooling (SARP) to distill task-relevant semantic evidence directly from \texttt{[SEG]} tokens within the MLLM's hidden states. Furthermore, a \textbf{R}easoning-to-\textbf{V}isual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset comprising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that \textsc{MedREAL} significantly outperforms state-of-the-arts, achieving 68.49\% gIoU and 70.47\% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, \textsc{MedREAL} provides a robust, interpretable framework for reasoning-driven medical image analysis.
Tags
Links
- Source: https://arxiv.org/abs/2608.26856v1
- Canonical: https://arxiv.org/abs/2608.26856v1
Trouble viewing inline? Open PDF directly →
Full Text
52,691 characters extracted from source content.
Expand or collapse full text
From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation Haowen Gu 1,2 , Gensheng Pei 3 , Junzhu Mao 1,2 , Qiong Wang 1,2 , Mingwu Ren 1,2 (B) , and Yazhou Yao 1,2 (B) 1 Nanjing University of Science and Technology, Nanjing, China 2 State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China 3 Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea yazhou.yao@njust.edu.cn, renmingwu@mail.njust.edu.cn https://github.com/NUST-Machine-Intelligence-Laboratory/MedREAL Abstract. Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question An- swering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose MedREAL (Medical REasoning-driven Answering and Localization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, MedREAL in- troduces Seg Anchored Reasoning Pooling (SARP) to distill task-relevant semantic evidence directly from [SEG] tokens within the MLLM’s hid- den states. Furthermore, a Reasoning-to-Visual (R2V) fusion mecha- nism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset com- prising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that MedREAL signifi- cantly outperforms state-of-the-arts, achieving 68.49% gIoU and 70.47% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, MedREAL provides a robust, interpretable framework for reasoning-driven medical image analysis. Keywords: Medical Visual Question Answering· Multimodal Large Language Models· Visual Grounding· Reasoning-driven Segmentation 1 Introduction Medical visual question answering (Med-VQA) aims to answer clinically rele- vant questions about medical images, requiring models to jointly understand visual content and perform domain-specific reasoning. Beyond answer predic- tion, clinical practice often demands explicit localization of the visual evidence arXiv:2608.26856v1 [cs.CV] 27 Aug 2026 2H. Gu et al. Question: Can you tell me what is this on my skin? This is a dermatoscopicview of a pigmented lesion TextSegment bbox promptpoint Segmentor Text Segment Question: What surrounding tissue changes are visible adjacent to the focal breast lesion in the ultrasound image? VLM Vision Encoder Mask Decoder Answer: There is evidence of architectural distortion ... malignant lesions. Here is the [seg]. SARP R2V TextSegment Medical VQAPromptableSegmentation MedREAL(ours) No Localization Only provides text answer without visual grounding. Prompt-dependent Required manual prompt(bbox/point/text) and lacks semantic understanding. Answer-consistent Localization Predicts segmentation aligned with the question semantics without manual prompts Explainable Provides reasoning-based evidence supporting the answer. Fig. 1: Comparison between Medical VQA, promptable segmentation, and the pro- posed MedREAL framework. MedREAL introduces an explicit evidence token to extract reasoning-aligned features, which are fused with visual context to produce answer-consistent evidence localization. that supports a decision, such as identifying a lesion, anatomical structure, or pathological region. This requirement highlights a critical limitation of existing Med-VQA systems: while recent multimodal large language models (MLLMs) [1,3,13,34,35,51] demonstrate strong reasoning capabilities, they typically pro- vide answers without grounding them in pixel-level evidence, thereby reducing interpretability and limiting clinical trust. Most medical image segmentation methods [6,14,38,39,52,61] demonstrate strong capability in spatial localization, yet remain limited in high-level semantic understanding. Although recent promptable foundation models such as SAM3 [10] and its medical variant MedSAM3 [32] enable concept segmentation from text-conditioned inputs, their semantic modeling remains shallow and largely relies on lexical alignment rather than complex clinical reasoning. As a result, these approaches struggle to capture the multi-step diagnostic logic required in real-world scenarios, where segmentation should be guided by question-driven semantic evidence rather than generic visual prompts. This creates a semantic disconnect between high-level diagnostic logic and pixel-level spatial grounding. Overcoming this challenge is essential for building integrated systems capable of joint medical answering and localization, ensuring that every diagnostic response is backed by spatially consistent visual justification. Simply concatenating MLLMs with segmentation models, however, is subop- timal. Although MLLM hidden states contain rich multimodal information, they are often dominated by global context and broad linguistic patterns, making it difficult to isolate the specific visual features that serve as reasoning evidence. Furthermore, directly feeding text embeddings into a segmentation model leads to poor spatial alignment, as the connection between token-level reasoning and pixel-level localization is not explicitly modeled. As illustrated in Figure 1, an effective solution must bridge this gap by explicitly identifying reasoning-related MedREAL: From Reasoning to Pixels3 evidence within MLLM representations and transforming it into spatially mean- ingful signals to guide the segmentation process. To address these challenges, we propose MedREAL (Medical REasoning- driven Answering and Localization), a unified framework that aligns linguistic reasoning with pixel-level localization within a reasoning-aware latent space. Specifically, we embed an explicit evidence token, [SEG], into the Med-VQA generation sequence, utilizing its hidden representation as a reasoning anchor to distill the critical semantic evidence that justifies the predicted answer. Building upon this design, we introduce the Seg-Anchored Reasoning Pooling (SARP) module. SARP leverages the [SEG] representation as a guiding prior to distill reasoning-relevant features from the MLLM’s terminal layers. By effectively sup- pressing irrelevant global semantics, SARP yields a compact, evidence-aligned representation tailored to the target region. Furthermore, we propose a Reasoning- to-Visual (R2V) Fusion mechanism, which integrates these distilled reasoning features with the global visual context extracted from the segmentor’s image encoder. This operation produces a semantically informed conditioning vector that robustly guides the SAM-based mask decoding stage. With this unified ar- chitecture, segmentation is driven not only by visual appearance but also by the model’s internal diagnostic reasoning, allowing MedREAL to achieve spatially precise and answer-consistent evidence localization. Another fundamental challenge in this domain is the scarcity of benchmarks that simultaneously evaluate clinical reasoning and spatial grounding. Current Med-VQA datasets [18, 28, 33] predominantly prioritize textual accuracy with- out pixel-level supervision; conversely, existing medical segmentation datasets [2,5,12,22,44,48] typically lack reasoning-oriented queries. To enable the training and evaluation of reasoning-driven segmentation, we construct MedRAVS-13K (Medical Reasoning, Answering, and Visual Segmentation), a reasoning-aware benchmark comprising 13,824 expertly curated samples. We augment four het- erogeneous medical imaging datasets (i.e., BUSI [2], COVID-QU-Ex [48], ISIC- 2018 [12], and Kvasir-SEG [22]), spanning ultrasound, X-ray, dermoscopy, and endoscopy modalities. We enrich these datasets with clinically oriented question- answer pairs that explicitly refer to specific regions of interest. Each QA pair is matched with a verified pixel-level segmentation mask, forming perfectly aligned image-text-mask triplets. This provides the structured supervision necessary to link high-level diagnostic reasoning with precise spatial localization. Extensive experiments on the MedRAVS-13K dataset demonstrate that our MedREAL significantly enhances segmentation quality. Compared to existing representative methods, MedREAL achieves state-of-the-art performance with an overall gIoU of 68.49% and cIoU of 70.47%. Crucially, by aligning linguistic reasoning with pixel-level grounding, our method ensures that the generated evidence masks are semantically consistent with the predicted clinical answers. This synergy effectively mitigates the “black-box” nature of traditional MLLMs, offering substantially improved interpretability and reliability for clinical decision support systems. In summary, our primary contributions are fourfold: 4H. Gu et al. – We propose MedREAL, a unified framework that couples Med-VQA rea- soning with pixel-level grounding by embedding explicit [SEG] tokens as reasoning anchors to distill semantic evidence from MLLM hidden states. – We introduce the SARP module and R2V fusion mechanism to effectively extract reasoning-relevant features and systematically inject them into a SAM-based pipeline for precise mask decoding. – We construct MedRAVS-13K, a comprehensive benchmark dataset of 13,824 samples across four diverse imaging modalities, providing a robust founda- tion of clinical QA pairs and verified pixel-level masks. – Extensive evaluations show that MedREAL significantly outperforms ex- isting methods, achieving 68.49% gIoU and 70.47% cIoU, while ensuring high semantic consistency between linguistic reasoning and spatial evidence. 2 Related Work Medical Visual Question Answering. Medical visual question answering aims to empower models to resolve clinically relevant queries by synthesizing vi- sual content with domain-specific medical knowledge. While early efforts primar- ily focused on fusion strategies between static image features and linguistic en- codings [16,17,54,62], the field has recently been transformed by MLLMs. These advanced architectures [11,27,29,47,49] leverage massive pretraining to demon- strate sophisticated cross-modal reasoning, enabling them to interpret complex clinical descriptions and generate nuanced natural language responses. However, a critical gap remains as these models predominantly operate on global seman- tic representations. Despite their linguistic fluency, they typically fail to provide explicit pixel-level grounding for their predictions. This absence of spatial evi- dence makes it difficult to verify the decision rationale against specific pathologi- cal regions, thereby limiting their interpretability and clinical trustworthiness in high-stakes diagnostic environments. To overcome this, MedREAL embeds an [SEG] token as a reasoning anchor to distill local semantic evidence, successfully bridging the gap between linguistic answers and precise spatial grounding. Medical Image Segmentation. Medical image segmentation is a fundamental task in medical image analysis, aiming to accurately delineate lesions or anatom- ical structures to support diagnosis and treatment planning. Traditional meth- ods [9,20,41,46,53,59] rely on U-Net [45] and its variants, which perform pixel- wise classification through fully convolutional networks. More recently, prompt- able segmentation approaches, such as SAM [39], have demonstrated strong gen- eralization across multiple modalities and large-scale datasets, enabling segmen- tation guided by points, bounding boxes, or textual prompts. However, these methods [23, 25, 52, 60, 63] typically rely solely on visual information and lack language-based reasoning constraints, making it challenging to perform question- driven segmentation. In clinical diagnostic workflows, it is essential for models to provide localized evidence that directly corresponds to a specific query; there- fore, the effective integration of linguistic reasoning with spatial segmentation remains an open and critical challenge. Unlike conventional visual-centric mod- els, our MedREAL framework explicitly injects MLLM-derived clinical logic MedREAL: From Reasoning to Pixels5 Question: What surrounding tissue changes are visible adjacent to the focal breast lesion in the ultrasound image? er Answer: There is evidence of architectural distortion ....... malignant lesions. Here is the [seg]. Vision Encoder Mask Decoder bbox VLM LoRA [seg] reasoning tokens 풉 reason SARP 풉 seg Concat R2V 푭 풗 img 풄 Fig. 2: Overview of MedREAL. A medical image and question are processed by a VLM producing answer tokens including [SEG]. SARP distills reasoning-relevant fea- tures from [SEG], which are fused with global visual features via R2V Fusion to con- dition the mask decoder, producing an evidence-aligned segmentation mask guided by both reasoning and visual context. into the segmentation pipeline via the R2V mechanism, ensuring masks are di- rectly driven by diagnostic reasoning. Reasoning Guided Segmentation. Reasoning-guided segmentation aims to incorporate high-level semantic reasoning into pixel-level predictions. Existing studies [7, 8, 15, 19, 26, 36, 43, 50, 55–58] have largely focused on natural scenes, predominantly relying on a single [seg] token to represent the entire target se- mantic mask. While this design enables coarse alignment between language and the predicted mask, it represents the target as a holistic concept and overlooks fine-grained evidence that is critical for precise localization. In medical imaging, this line of research is still in its early stages. Some approaches [21,32] attempt to fuse attention maps or intermediate features from question-answering models with segmentation networks to enhance semantic consistency, but they typically rely on global language representations or simple feature concatenation, lack- ing explicit anchoring mechanisms to localize regions relevant to the question. Moreover, the scarcity of high-quality reasoning-annotated datasets for medi- cal tasks makes joint training and evaluation challenging. MedREAL uniquely addresses these bottlenecks by utilizing the SARP module to extract localized, fine-grained semantic evidence rather than holistic representations, supported by our comprehensive MedRAVS-13K benchmark for rigorous evaluation. 3 Method 3.1 Overall Pipeline We propose MedREAL, a unified framework that couples medical visual ques- tion answering with pixel-level segmentation by explicitly injecting multimodal reasoning semantics into the localization process. The overall architecture is il- lustrated in Figure 2. 6H. Gu et al. Unlike standard pipeline models that treat text generation and segmentation as isolated steps, MedREAL unifies them within a reasoning-aware latent space. Given a medical image I and a clinical text query Q, the multimodal large lan- guage model (MLLM) autoregressively generates a textual response A alongside a sequence of latent hidden states H. To explicitly bridge language and vision, we prompt the MLLM to output a dedicated [SEG] token when localization is required. The language generation process can be formulated as: A,H =F MLLM (I,Q)(1) where H ∈R B×L×D encapsulates the dense semantic representations of the generated reasoning process. Rather than relying solely on generic visual prompts, MedREAL leverages this textual reasoning to guide the spatial decoding. We introduce the Seg- Anchored Reasoning Pooling (SARP) module to extract a compact reasoning representation h reason from H. Subsequently, the Reasoning-to-Visual (R2V) Fusion module dynamically integrates h reason with the visual feature maps F extracted by the segmentor’s image encoder. This yields a semantically informed prompt c, which is fed into the mask decoderD mask to predict the final evidence mask M: M =D mask (F,c).(2) By formulating the pipeline in this cohesive manner, the segmentation mask is strictly conditioned on the diagnostic logic formulated by the MLLM, ensuring semantic alignment between the textual answer and the localized visual evidence. 3.2 Seg-Anchored Reasoning Pooling While the [SEG] token acts as a trigger for the segmentation task, extracting features exclusively from its corresponding hidden state is suboptimal. Clinical reasoning is inherently distributed across the preceding linguistic sequence; a single token lacks the capacity to encapsulate complex diagnostic multi-step logic. To address this, we propose the SARP module, which utilizes the [SEG] token as an attention anchor to aggregate reasoning-relevant semantics from the broader context. Let H ∈R B×L×D denote the terminal-layer hidden states of the MLLM. We first isolate the representation of the [SEG] token using a binary positional mask m s ∈ 0, 1 L . To ensure robustness against potential sequence shifts, the segmentation anchor h seg ∈R D is computed via masked average pooling: h seg = P L i=1 m s i H i P L i=1 m s i + ε ,(3) where ε is a small constant to prevent zero division. This anchor explicitly captures the model’s semantic state at the exact moment spatial grounding is invoked. To adaptively filter out irrelevant linguistic noise (e.g., grammatical structures or generic clinical boilerplate) and concentrate on specific diagnostic MedREAL: From Reasoning to Pixels7 evidence, we project h seg into a query vector q. Concurrently, the preceding rea- soning sequence within H is linearly mapped into key K and value V matrices. An attention-based pooling mechanism is then applied to measure the relevance of each reasoning token to the segmentation trigger: α = Softmax qK ⊤ √ d k ,(4) where d k is the scaling factor based on the hidden dimension. The final reasoning representation h reason is derived through a weighted summation: h reason = αV .(5) This operation distills a compact, high-density semantic vector that explicitly encodes the clinical rationale necessitating the segmentation, providing a highly informative prior for the subsequent localization phase. 3.3 Reasoning-to-Visual Fusion Although h reason is semantically rich, it exists entirely within the linguistic latent space and lacks the spatial inductive biases required for precise pixel-level mask generation. To translate abstract diagnostic logic into an explicit spatial guid- ance signal, we introduce the R2V Fusion module, which bridges the extracted reasoning with global anatomical context. Given the multi-scale visual feature map F ∈R B×C×H ′ ×W ′ from the seg- mentor’s image encoder, we first apply global average pooling to collapse the spatial dimensions. A linear projection followed by a non-linear activation σ maps this representation into a global visual context vector v img ∈R B×D c : v img = σ Linear AvgPool(F ) . (6) This vector provides a structural anatomical prior. To enable deep multi- modal interaction, we concatenate v img with the reasoning representation h reason along the channel dimension. The fused tensor is then projected through a two- layer multi-layer perceptron (MLP) parameterized by φ: c = φ [v img ,h reason ] ,(7) where [·,·] denotes the concatenation operation. The resulting conditional vector c fundamentally alters the segmentation paradigm: rather than relying on geo- metric prompts (e.g., points or boxes), c acts as a semantically-informed pseudo- prompt for the SAM-based mask decoder. This ensures the decoded mask strictly adheres to the diagnostic evidence articulated by the MLLM. 3.4 Optimization Objective The entire MedREAL framework is trained end-to-end, jointly optimizing both the text generation quality and the spatial segmentation accuracy. The total loss 8H. Gu et al. L is formulated as a weighted combination: L = λ txt L CE ( ˆ Y txt ,Y txt )+λ mask h λ bce L BCE ( ˆ M,M)+λ dice L DICE ( ˆ M,M) i .(8) Here, L CE represents the standard auto-regressive cross-entropy loss applied to the textual response. For the spatial localization, the mask loss is computed as a weighted sum of the per-pixel Binary Cross-Entropy (L BCE ) and the Dice loss (L DICE ), which handles class imbalance in medical lesions. This composite objective effectively bridges the MLLM and the visual segmentor, forcing the shared latent space to align linguistic logic with pixel-level boundaries. 4 Dataset 4.1 Data Source MedRAVS-13K Fig. 3: Overview of MedRAVS-13K. As summarized in Figure 3, we con- struct MedRAVS-13K (Medical Reasoning, Answering, and Visual Segmentation) to rigorously evalu- ate the MedREAL framework. We integrate four diverse publicly avail- able sources: BUSI [2], COVID- QU-Ex [48], ISIC-2018 [12], and Kvasir-SEG [22]. Unlike conven- tional benchmarks that treat text and masks in isolation, MedRAVS- 13K explicitly aligns fine-grained clinical question-answering with pre- cise visual segmentation masks. This design enables a dual assessment of sophisticated diagnostic reasoning and localized evidence generation. The dataset spans multiple modal- ities, e.g., ultrasound, X-ray, der- moscopy, and endoscopy, covering critical organs such as the breast, lung, skin, and colon. These sources exhibit substantial heterogeneity in imaging physics and lesion morphology, ranging from low-contrast sonographic bound- aries to intricate endoscopic mucosal textures. Such multi-modality composition provides a robust and challenging testbed for evaluating reasoning-driven seg- mentation across clinically distinct scenarios. 4.2 Data Generation and Curation Figure 4 details the multi-stage generation and curation pipeline designed to align clinical reasoning with pixel-level annotations. For each source dataset, we MedREAL: From Reasoning to Pixels9 Source data: Diverse medical modalities and ground truth masks BUSI ISIC-2018 Kvasir-SEG Multi-source Medical Data COVID-QU-Ex Qwen3-VL-235B-A22B- Thinking VQA Generation Prompt Template [seg] TEXT Prompt Engineering 10% LLM-based medical semantic consistency Human expert review Automatic Filter & Human Verification Q: Can you tell me what is this on my skin? A: This is a pigmented lesion, Here is the [seg]. Rule-basedFinalCheck Verify Q&A is not None, [seg] token is included. Data storage Stored in JSON format Fig. 4: Dataset Generation Pipeline of MedRAVS-13K. The process illustrates the integration of raw images and masks to generate reasoning-aware text, followed by rigorous quality assurance. develop modality- and organ-specific prompts that incorporate the raw medical images alongside their ground-truth masks. We leverage the advanced reasoning capabilities of Qwen3-VL-235B-A22B-Thinking to generate clinically accurate question-answering (QA) pairs. Crucially, a [SEG] token is explicitly embedded within each generated sequence to denote the region of interest. This mechanism ensures that the textual rationale is strictly grounded in the annotated anatomy or lesion, compelling the model to bridge abstract diagnostic logic with concrete visual evidence. To guarantee data fidelity, we implement a two-tier quality assurance proto- col. First, an automated filtering stage employs a powerful large language model to evaluate the medical plausibility, semantic consistency, and visual relevance of each generated pair, discarding low-quality outputs. Second, we perform meticu- lous manual verification on a 10% subset of the data to confirm clinical accuracy and verify the spatial alignment between the textual queries and the segmenta- tion masks. To mitigate potential biases arising from class and modality imbal- ances, we apply a controlled sampling strategy across the pathological categories during the final split construction, ensuring a representative benchmark. Ulti- mately, MedRAVS-13K comprises over 13,000 expertly curated samples. Each instance seamlessly integrates a medical image, a [SEG]-augmented QA pair, and a corresponding pixel-level mask. These queries encompass diverse diagnostic in- tents, including polar (yes/no), descriptive, and localization-focused questions. Because the masks exhibit substantial variation in scale and morphology across the four modalities, the dataset presents highly challenging visual grounding scenarios that demand the joint optimization of reasoning, segmentation, and answer prediction. 10H. Gu et al. Table 1: Referring phrases used for different datasets and semantic classes. These ground-truth derived templates serve as explicit prompts for non-reasoning baselines. DatasetClassTrain Samples Val Samples Referring Phrase BUSI [2] Benign915396benign breast mass Malignant441189malignant breast tumor Normal276123normal breast tissue COVID-QU-Ex [48] COVID-192330582COVID-19 lung infection Non-COVID1162291non-COVID lung infection Normal1164291normal lung region ISIC-2018 [12]Skin Lesion2593100skin lesion region Kvasir-SEG [22] Polyp2615356polyp region 5 Experiments 5.1 Experiment Setup Datasets. To evaluate the performance of MedREAL, all experiments are con- ducted on the MedRAVS-13K dataset as described in Sec. 4. For comparative analysis involving baseline models that lack inherent reasoning capabilities, we utilize standard category labels to implement the segmentation task. Detailed specifications regarding the prompting mechanisms for various architectures are provided in the subsequent sections. Evaluation Metrics. Following established protocols in reasoning segmenta- tion, we adopt gIoU and cIoU as primary metrics. gIoU is defined as the arith- metic mean of per image IoU scores, while cIoU represents the ratio of total intersection to total union across the dataset. Since cIoU is heavily biased to- ward large area objects and prone to instability, gIoU is preferred in medical contexts where lesion scales vary significantly, as it provides a more balanced measure of segmentation quality. Implementation Details. Our model is trained on two NVIDIA 48G A6000 GPUs using the DeepSpeed [42] engine to enable memory-efficient optimization and stable large scale multimodal training. We adopt the AdamW [37] optimizer with an initial learning rate of 2×10 −4 and apply weight decay for regularization. Unless otherwise specified, we adopt Qwen3-VL-2B-Instruct [4] as the MLLM and SAM [24] as the segmentor for all experiments. Following the optimization strategy in LISA [26], we set the loss weights λ txt and λ mask to 1.0. For the mask supervision components, λ bce and λ dice are assigned values of 2.0 and 0.5. 5.2 Comparison with State-of-the-Art Methods To evaluate MedREAL, we benchmark it against three distinct paradigms: referring-based segmentation (i.e., OVSeg [30], SAM3 [10], MedSAM3 [32]), agent-based iterative systems (i.e., SAM3-Agent [10], MedSAM3-Agent [32]), and end-to-end reasoning segmentation (i.e., LISA [26]). MedREAL: From Reasoning to Pixels11 Table 2: Comparison of gIoU and cIoU across different datasets. Best results are highlighted in bold, and second-best are underlined . Method BUSICOVID-QU-Ex ISIC-2018 Kvasir-SEG Overall gIoU cIoU gIoU cIoU gIoU cIoU gIoU cIoU gIoU cIoU Refering-based Segmentation OVSeg [30]4.72 4.79 17.59 16.75 26.19 24.56 13.88 13.58 13.48 18.75 SAM3 [10]17.69 0.68 0.00 0.00 1.04 1.78 0.00 0.00 5.43 1.39 MedSAM3 [32]62.93 58.9771.56 75.03 83.03 80.4381.82 82.02 70.99 78.28 Agent-based Segmentation SAM3-Agent [10]41.61 19.67 5.24 8.12 7.17 0.92 4.08 3.76 16.21 4.20 MedSAM3-Agent [32] 58.29 55.10 46.88 46.23 82.6580.88 71.89 62.84 55.71 74.33 Reasoning-based Segmentation LISA [26]41.76 48.84 62.30 63.88 71.15 56.06 57.50 56.30 55.70 56.05 MedREAL (Ours) 69.15 62.52 64.51 66.9282.21 71.74 76.3569.7368.4970.47 Table 3: Comparison of text generation performance between LISA and our method. BLEUROUGE MethodBLEU BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L LISA [26]24.04 56.94 29.69 18.88 13.1559.8133.4148.11 MedREAL (Ours) 89.12 98.75 97.74 97.00 96.26 96.6995.8496.69 Because referring-based purely structural models lack intrinsic reasoning ca- pabilities, they cannot process the complex VQA queries in MedRAVS-13K di- rectly. To accommodate them and establish an upper-bound for structural de- coding, we provide explicit, oracle-like text prompts derived from the ground- truth category of each query, with detailed templates summarized in Table 1. For the agent-based approaches, we deploy Qwen3-VL-8B-Thinking as the rea- soning engine to autonomously generate prompts for the segmentors (capped at 20 rounds). Finally, LISA is evaluated as the primary reasoning-segmentation baseline, utilizing LLaVA-7B [29] as its multimodal backbone. Quantitative Segmentation Results. Table 2 presents the quantitative lo- calization performance. When compared to the direct end-to-end reasoning base- line (LISA), MedREAL demonstrates overwhelming superiority, improving the overall gIoU from 55.70% to 68.49%. This substantial margin validates that our SARP and R2V modules extract far more precise spatial priors than LISA’s holistic [SEG] embedding approach. This advantage is particularly evident in high-noise environments like the BUSI ultrasound dataset, where MedREAL achieves 69.15% gIoU compared to LISA’s 41.76%, proving our method’s ro- bustness against low-contrast boundaries and speckle noise. It is crucial to correctly contextualize the performance of MedSAM3. While it reaches high metrics (e.g., 70.99% overall gIoU), this purely structural model benefits heavily from explicit ground-truth category prompts (as defined in Ta- ble 1) and extensive pretraining on the original source datasets. However, this 12H. Gu et al. label-driven paradigm fails entirely in real-world diagnostic workflows that re- quire multi-step logic rather than explicit naming. This vulnerability is starkly exposed when evaluating MedSAM3-Agent: once forced to autonomously deduce the prompt from complex VQA queries rather than relying on oracle labels, its overall gIoU drastically collapses to 55.71%. The performance degradation is exceptionally severe on the COVID-QU-Ex dataset (dropping from 71.56% to 46.88%), as the decoupled reasoning engine struggles to comprehend complex pathological descriptions, generating inaccurate textual prompts that mislead the segmentation backbone. In contrast, MedREAL operates entirely on nat- ural language questions without predefined category hints or oracle labels, yet achieves highly competitive performance (68.49% overall gIoU). By seamlessly bridging the gap between abstract diagnostic reasoning and spatial grounding within a unified latent space, MedREAL establishes a new state-of-the-art for reasoning-driven medical architectures. Text Generation Quality. Beyond spatial localization, clinically integrated systems must maintain high diagnostic articulation. Table 3 compares the textual VQA generation quality of MedREAL against LISA. Our framework achieves near-perfect BLEU [40] and ROUGE [31] scores, outperforming the baseline by massive margins. These results convincingly demonstrate that extracting rea- soning semantics for visual guidance does not compromise the MLLM’s inherent linguistic capabilities. Instead, our unified architecture successfully harmonizes high-fidelity text generation with precise pixel-level evidence grounding, offering a complete and trustworthy diagnostic response. 5.3 Ablation Studies To rigorously investigate the individual contributions of Box Guidance, the Seg-Anchored Reasoning Pooling (SARP) module, and the Reasoning-to-Visual (R2V) fusion mechanism, we conduct comprehensive ablation experiments. To ensure fair comparison and preserve parameter parity, ablated modules are sub- stituted with equivalent linear projections rather than simply being bypassed. The quantitative impacts across gIoU and cIoU metrics are detailed in Table 4. Table 4: Ablation study isolating the contributions of bounding-box guidance, SARP, and R2V modules. Box Guide SARP R2V gIoU (%) cIoU (%) ✗11.3519.69 ✓✗45.0252.84 ✗✓36.2437.74 ✓✗✓23.2247.12 ✓✗45.5046.96 ✓68.4970.47 The Necessity of Spa- tial Priors. Removing all three key components de- grades performance to a baseline of 11.35% gIoU, re- flecting the inherent diffi- culty of zero-shot spatial grounding without explicit localization priors. Reintro- ducing only Box Guidance provides a sharp increase to 45.02% gIoU. While coarse spatial anchors offer strong MedREAL: From Reasoning to Pixels13 regional cues, this configuration still lacks the fine-grained semantic alignment required to delineate precise lesion boundaries. The Interdependence of SARP and R2V. The interplay between reasoning extraction and visual fusion proves critical. When SARP is active but R2V is replaced by a linear mapping, the model achieves 45.50% gIoU, comparable to the Box Guidance baseline, yet struggles with cIoU. This reveals that while SARP successfully distills concentrated evidence from the reasoning sequence, these semantic features remain bottlenecked without a dedicated fusion mechanism to project them into the visual space. Conversely, enabling R2V without SARP yields a detrimental 23.22% gIoU; forcefully injecting unrefined, global linguistic representations directly into the visual pipeline introduces severe semantic noise, actively disrupting spatial localization. Synergistic Integration. As shown in Table 4, enabling both SARP and R2V without Box Guidance yields 36.24% gIoU, demonstrating that reasoning-driven conditioning can partially compensate for missing spatial priors, but remains insufficient on its own. The full MedREAL architecture seamlessly integrates all three components to establish the upper bound (68.49% gIoU / 70.47% cIoU). This confirms a highly complementary relationship: Box Guidance anchors the general region, SARP distills the exact diagnostic rationale, and R2V effectively translates this rationale into actionable spatial conditioning. Table 5: Effect of MLLM scaling on reasoning- driven segmentation performance. Upgrading the backbone directly translates to superior localization. Backbone MLLMgIoU (%) cIoU (%) Qwen3-VL-2B-Instruct68.4970.47 Qwen3-VL-4B-Instruct 75.96 70.26 Scaling Behavior. Finally, we explore the scaling behav- ior of the underlying multi- modal foundation model. As reported in Table 5, up- grading the backbone from Qwen3-VL-2B-Instruct to the 4B-Instruct variant, while freez- ing the rest of the MedREAL architecture, triggers a sub- stantial gIoU improvement from 68.49% to 75.96%. This empirically validates that our framework efficiently lever- ages the enhanced reasoning capabilities of larger MLLMs, dynamically trans- lating superior linguistic logic into higher-quality evidence masks. 5.4 Qualitative Analysis Robustness across Modalities. Figure 5 evaluates segmentation across four challenging clinical scenarios. General-domain models (e.g., OVSeg [30], SAM3 [10]) fail on low-contrast ultrasound (Row 1) and diffuse X-ray infections (Row 2) due to limited domain knowledge. While MedSAM3 [32] improves boundary pre- diction through domain pretraining, its decoupled reasoning makes it sensitive to visual noise, such as endoscopic blood artifacts (Row 4). LISA [26] narrows the semantic gap but relies on a single, global [seg] token. This design limits 14H. Gu et al. ImageOVSegSAM3MedSAM3SAM3 AgentMedSAM3 AgentMedREAL(Ours)LISAGT Fig. 5: Qualitative comparison with competing methods across four diverse medical imaging modalities (Top to Bottom: BUSI [2], COVID-QU-Ex [48], ISIC-2018 [12], and Kvasir-SEG [22]). MedREAL consistently produces precise evidence masks that are robust to structural artifacts and highly aligned with the underlying clinical reasoning. What is the anatomical location and key morphological characteristics of the mucosal lesion observed in this endoscopic image? Question The lesion is situated within the colonic lumen, .... Its surface demonstrates a granular texture with subtle vascular patterns and a slightly erythematous appearance. The [seg] region shows .... GT The lesion is situated within the colonic lumen... Its surface demonstrates a granular texture with subtle vascular patternsand a slightly erythematous appearance. The [SEG] shows ... MedREAL The lesion is located in the cecum... It exhibits a uniform erythematous appearance without ulceration or nodularity. The [SEG] demonstrates ... LISA (a) What radiological features are present in the right lower lobe that suggest a viral pneumonia in this COVID-19 case? Question The right lower lobe demonstrates bilateral ground-glass opacities with ....... [seg] GT The right lower lobe demonstrates bilateral ground-glass opacities .... typical of viral pneumonitis. Here is the [SEG]. MedREAL The right lower lobe demonstrates...... showing diffuse involvement of the lung parenchyma. [SEG] LISA (b) Fig. 6: Qualitative comparison between MedREAL and LISA. MedREAL demon- strates superior semantic consistency between the generated VQA text and the spatial segmentation mask. precise spatial localization, causing LISA to over-segment irregular skin lesion boundaries with small color changes (Row 3). In contrast, MedREAL directly feeds diagnostic reasoning into the visual decoder via SARP and R2V. This rich semantic information ensures robust artifact removal and accurate bound- ary segmentation across all modalities. Unlike baseline methods that segment MedREAL: From Reasoning to Pixels15 the most visually dominant structures, our approach localizes targets guided by underlying clinical logic. Consequently, MedREAL reliably distinguishes true pathological regions from structurally similar healthy tissues. Reasoning-to-Mask Consistency. Figure 6 analyzes the internal consistency between generated VQA responses and spatial masks. LISA [26] frequently ex- hibits a serious mismatch: in the endoscopic case (Figure 6a), its incorrect text prediction of the polyp leads to a false-positive mask, while in the X-ray case (Figure 6b), it correctly localizes the infection in text but completely fails to lo- cate it spatially. Such unpredictable misalignment greatly reduces user trust in critical medical scenarios, as the visual evidence fails to support the diagnostic text. MedREAL resolves this mismatch. By conditioning the spatial decoding on extracted reasoning features rather than general prompts, our predicted mask serves as an accurate, pixel-level representation of the correct diagnostic text. This reliable alignment turns the MLLM from a black-box predictor into an interpretable and verifiable clinical tool. 6 Conclusion In this paper, we present MedREAL, a unified framework that bridges the semantic gap between high-level diagnostic reasoning and pixel-level spatial grounding in medical multimodal large language models. By introducing the Seg- Anchored Reasoning Pooling (SARP) and Reasoning-to-Visual (R2V) Fusion mechanisms, our approach effectively distills localized semantic evidence from the model’s internal reasoning sequence to guide precise mask decoding. Sup- ported by our newly curated MedRAVS-13K benchmark, extensive evaluations across four diverse imaging modalities confirm that MedREAL achieves superior spatial alignment (68.49% gIoU) compared to existing reasoning-segmentation architectures. Crucially, by strictly coupling linguistic outputs with visual evi- dence, our method significantly enhances the interpretability and trustworthiness of AI-assisted clinical decision support. Limitations and Future Work. Currently, MedREAL operates exclusively on 2D imaging modalities and incurs computational overhead due to the autore- gressive nature of the MLLM backbone. To address these constraints, our future work focuses on integrating 3D visual encoders into the R2V module to sup- port volumetric data (e.g., CT and MRI) and exploring knowledge distillation to accelerate inference for real-time clinical deployment. 7 Acknowledgement This work was supported by the National Defense Science and Technology In- dustry Bureau Technology Infrastructure Project (JSZL2024606C001). 16H. Gu et al. References 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 2. Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A.: Dataset of breast ultrasound images. Data in brief 28, 104863 (2020) 3. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023) 4. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., , et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 5. Bilic, P., Christ, P., Li, H.B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G.E.H., Chartrand, G., et al.: The liver tumor segmentation benchmark (lits). Medical image analysis 84, 102680 (2023) 6. Butoi, V.I., Ortiz, J.J.G., Ma, T., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Uni- verseg: Universal medical image segmentation. In: CVPR. p. 21438–21451 (2023) 7. Cai, X., Li, L., Pei, G., Chen, T., Pan, J., Yao, Y., Wang, W.: Unbiased object detection beyond frequency with visually prompted image synthesis. In: The Four- teenth International Conference on Learning Representations (2026) 8. Cai, X., Pei, G., Sun, Z., Yao, Y., Shen, F., Wang, W.: Iris: Bringing real-world priors into diffusion model for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 26909– 26919 (2026) 9. Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin- unet: Unet-like pure transformer for medical image segmentation. In: ECCV. p. 205–218. Springer (2022) 10. Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., et al.: Sam 3: Segment anything with concepts. In: ICLR (2026) 11. Chen, J., Gui, C., Ouyang, R., Gao, A., Chen, S., Chen, G.H., Wang, X., Cai, Z., Ji, K., Wan, X., et al.: Towards injecting medical visual knowledge into multimodal llms at scale. In: EMNLP. p. 7346–7370 (2024) 12. Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al.: Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 (2019) 13. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025) 14. Fan, W., Fang, H., Li, R., Lin, Y., An, C., Luo, X.: Anatomy-Aware Frequency- Attention Transformer Networks for Liver Couinaud CT/MR Segmentation . In: MICCAI. vol. LNCS 15960, p. 55 – 65. Springer Nature Switzerland (October 2025) 15. Ghezloo, F., Seyfioglu, M.S., Soraki, R., Ikezogwo, W.O., Li, B., Vivekanandan, T., Elmore, J.G., Krishna, R., Shapiro, L.: Pathfinder: A multi-modal multi-agent sys- tem for medical diagnostic decision-making applied to histopathology. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision. p. 23431– 23441 (2025) MedREAL: From Reasoning to Pixels17 16. Gong, H., Chen, G., Liu, S., Yu, Y., Li, G.: Cross-modal self-attention with multi- task pre-training for medical visual question answering. In: ICMR. p. 456–460 (2021) 17. Gu, H., Pei, G., Sun, Z., Ren, M., Shu, X., Yao, Y., Shen, F.: Medfg-vqa: Low- frequency memory and graph attention for lightweight medical vqa. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 42755–42764 (2026) 18. He, X., Zhang, Y., Mou, L., Xing, E., Xie, P.: Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 (2020) 19. Howlader, P., Nguyen-Canh, H., Das, S., Xu, J., Le, H., Samaras, D.: Cora: Consistency-guided semi-supervised framework for reasoning segmentation. In: WACV. p. 5934–5944 (2026) 20. Huang, H., Lin, L., Tong, R., Hu, H., Zhang, Q., Iwamoto, Y., Han, X., Chen, Y.W., Wu, J.: Unet 3+: A full-scale connected unet for medical image segmentation. In: ICASSP. p. 1055–1059. Ieee (2020) 21. Huang, S., Liang, H., Wang, Q., Zhong, C., Zhou, Z., Shi, M.: Seg-sam: Semantic-guided sam for unified medical image segmentation. arXiv preprint arXiv:2412.12660 (2024) 22. Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., De Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: M. p. 451–462 (2019) 23. Jiang, C., Ding, T., Song, C., Tu, J., Yan, Z., Shao, Y., Wang, Z., Shang, Y., Han, T., Tian, Y.: Medical sam3: A foundation model for universal prompt-driven medical image segmentation. arXiv preprint arXiv:2601.10880 (2026) 24. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything (2023), https://arxiv.org/abs/2304.02643 25. Konwer, A., Yang, Z., Bas, E., Xiao, C., Prasanna, P., Bhatia, P., Kass-Hout, T.: Enhancing sam with efficient prompting and preference optimization for semi- supervised medical image segmentation. In: CVPR. p. 20990–21000 (2025) 26. Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: CVPR. p. 9579–9589 (2024) 27. Lai, Y., Zhong, J., Li, M., Zhao, S., Li, Y., Psounis, K., Yang, X.: Med-r1: Rein- forcement learning for generalizable medical reasoning in vision-language models. TMM (2026) 28. Lau, J.J., Gayen, S., Ben Abacha, A., Demner-Fushman, D.: A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5(1), 180251 (2018) 29. Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. NeurIPS 36, 28541–28564 (2023) 30. Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Marculescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: CVPR. p. 7061–7070 (2023) 31. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text sum- marization branches out. p. 74–81 (2004) 32. Liu, A., Xue, R., Cao, X.R., Shen, Y., Lu, Y., Li, X., Chen, Q., Chen, J.: Medsam3: Delving into segment anything with medical concepts. arXiv preprint arXiv:2511.19046 (2025) 18H. Gu et al. 33. Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., Wu, X.M.: Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In: ISBI. p. 1650–1654. IEEE (2021) 34. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llava-next: Improved reasoning, ocr, and world knowledge (January 2024), https://llava-vl.github. io/blog/2024-01-30-llava-next/ 35. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023) 36. Liu, Y., Peng, B., Zhong, Z., Yue, Z., Lu, F., Yu, B., Jia, J.: Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520 (2025) 37. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2019), https: //arxiv.org/abs/1711.05101 38. Lv, X., Dong, X., Wang, L., Yang, J., Zhao, L., Pu, B., Jin, Z., Li, X.: Test-time domain generalization via universe learning: A multi-graph matching approach for medical image segmentation. In: CVPR. p. 15621–15631 (2025) 39. Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature communications 15(1), 654 (2024) 40. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL. p. 311–318 (2002) 41. Pei, G., Chen, T., Wang, Y., Cai, X., Shu, X., Zhou, T., Yao, Y.: Seeing what matters: Empowering clip with patch generation-to-selection. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 24862–24872 (2025) 42. Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: ACM SIGKDD. p. 3505–3506 (2020) 43. Ren, Z., Huang, Z., Wei, Y., Zhao, Y., Fu, D., Feng, J., Jin, X.: Pixellm: Pixel reasoning with large multimodal model. In: CVPR. p. 26374–26383 (2024) 44. Riedel, E.O., de la Rosa, E., Baran, T.A., Petzsche, M.H., Baazaoui, H., Yang, K., Musio, F.A., Huang, H., Robben, D., Seia, J.O., et al.: Isles’24–a real-world longitudinal multimodal stroke dataset. arXiv preprint arXiv:2408.11142 (2024) 45. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. p. 234–241. Springer (2015) 46. Sun, S., Gu, H., Xie, C., Ren, Y., Ren, M., Zhang, H.: Bridging granularity gaps: Hierarchical semantic learning for cross-domain few-shot segmentation. In: Pro- ceedings of the AAAI Conference on Artificial Intelligence. vol. 40, p. 9215–9223 (2026) 47. Sun, Z., Yao, Y., Liu, T., Li, Z., Shen, F., Tang, J.: Jo-snc: Combating noisy la- bels through fostering self-and neighbor-consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025) 48. Tahir, A.M., Chowdhury, M.E., Khandakar, A., Rahman, T., Qiblawey, Y., Khur- shid, U., Kiranyaz, S., Ibtehaz, N., Rahman, M.S., Al-Maadeed, S., et al.: Covid-19 infection localization and severity grading from chest x-ray images. Computers in biology and medicine 139, 105002 (2021) 49. Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al.: Towards generalist biomedical ai. Nejm Ai 1(3), AIoa2300138 (2024) 50. Wang, J., Ke, L.: Llm-seg: Bridging image segmentation and large language model reasoning. In: CVPR. p. 1765–1774 (2024) 51. Wei, H., Sun, Y., Li, Y.: Deepseek-ocr: Contexts optical compression. arXiv preprint arXiv:2510.18234 (2025) MedREAL: From Reasoning to Pixels19 52. Wei, X., Cao, J., Jin, Y., Lu, M., Wang, G., Zhang, S.: I-medsam: Implicit medical image segmentation with segment anything. In: ECCV. p. 90–107. Springer (2024) 53. Wu, J., Zhang, Y., Tang, X.: Simultaneous tissue classification and lateral ventricle segmentation via a 2d u-net driven by a 3d fully convolutional neural network. In: EMBC. p. 5928–5931. IEEE (2019) 54. Xu, J., Pei, G., Liu, H., Yao, Y.: Gsv2x: Geometry-aware uncertainty modeling and orthogonal fusion for robust roadside perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 21409–21419 (2026) 55. Yan, C., Wang, H., Yan, S., Jiang, X., Hu, Y., Kang, G., Xie, W., Gavves, E.: Visa: Reasoning video object segmentation via large language models. In: ECCV. p. 98–115. Springer (2024) 56. Yang, Z., Pei, G., Chen, T., Yuan, X., Zhang, H., Shu, X., Yao, Y.: Beyond quadratic: Linear-time change detection with rwkv. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, p. 11811–11819 (2026) 57. Yang, Z., Pei, G., Chen, T., Zhou, Y., Zhou, T., Yao, Y., Shen, F.: Efficiency follows global-local decoupling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 25524–25535 (2026) 58. Yang, Z., Pei, G., Yao, Y., Zhou, T., Ding, L., Shen, F.: ChangeTitans: Toward remote sensing change detection with neural memory. IEEE Transactions on Geo- science and Remote Sensing 63, 1–14 (2025) 59. Yin, J., Chen, T., Chen, Y., Pei, G., Shu, X., Yao, Y., Shen, F.: Pca-seg: Revisiting cost aggregation for open-vocabulary semantic and part segmentation. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 27633–27643 (June 2026) 60. Zhang, W., Wu, H., Qin, J.: Domesticating sam for breast ultrasound image seg- mentation via spatial-frequency fusion and uncertainty correction. In: ECCV. p. 20–37. Springer (2024) 61. Zhang, Z., Yin, G., Zhang, B., Liu, W., Zhou, X., Wang, W.: A semantic knowledge complementarity based decoupling framework for semi-supervised class-imbalanced medical image segmentation. In: CVPR. p. 25940–25949 (2025) 62. Zhou, Y., Kang, X., Ren, F.: Employing inception-resnet-v2 and bi-lstm for medical domain visual question answering. In: CLEF. p. 1–11 (2018) 63. Zhu, J., Hamdi, A., Qi, Y., Jin, Y., Wu, J.: Medical sam 2: Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874 (2024)