Paper deep dive
MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models
Yuan Wang, Hualiang Wang, Yixin Chen, Songtao Jiang, Shujian Gao, Jiaming Lin, Siming Fu, Jian Wu, Zuozhu Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/16/2026, 2:58:42 AM
Summary
The paper introduces MedUP, a Medical Vision-Language Model (Med-VLM) that unifies visual perception (segmentation/grounding) and language understanding within a shared token space. It utilizes UniMedTok, a region tokenizer that encodes masks as discrete tokens, allowing seamless interleaving with text. The model is trained on UniMed-Train, a 1.84M-instance corpus, and evaluated on UniMed-Bench, demonstrating superior performance over native, agentic, and dual-decoder Med-VLMs.
Entities (10)
Relation Signals (7)
MedUP → uses → UniMedTok
confidence 98% · At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary
MedUP → evaluatedon → UniMed-Bench
confidence 97% · introduce UniMed-Bench for unified evaluation... MedUP achieves strong and consistent performance across all three tasks in UniMed-Bench
MedUP → trainedon → UniMed-Train
confidence 97% · We curate UniMed-Train... Extensive experiments show that MedUP outperforms...
MedUP-H → basedon → HuluMed-4B
confidence 95% · Our main models are MedUP-H, built on HuluMed-4B
MedUP-Q → basedon → Qwen3-VL-4B
confidence 95% · and MedUP-Q, built on Qwen3-VL-4B
UniMed-Train → containstask → Seg-CoT
confidence 90% · UniMed-Train... comprising... 4,000 CoT-based text-to-mask reasoning samples
MedUP → outperforms → MedSAM
confidence 85% · remaining competitive with specialist medical segmentors (e.g., MedSAM1–3) on text-guided segmentation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.
Tags
Links
- Source: https://arxiv.org/abs/2608.10635v1
- Canonical: https://arxiv.org/abs/2608.10635v1
Trouble viewing inline? Open PDF directly →
Full Text
71,022 characters extracted from source content.
Expand or collapse full text
MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models Yuan Wang1 Hualiang Wang1 Yixin Chen1 Songtao Jiang1 Shujian Gao2 Jiaming Lin1 Siming Fu1 Jian Wu1 Zuozhu Liu*, 1 1 Zhejiang University 2 Fudan University yuan2.24, zuozhuliu@intl.zju.edu.cn Abstract Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling. MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models Yuan Wang1 Hualiang Wang1 Yixin Chen1 Songtao Jiang1 Shujian Gao2 Jiaming Lin1 Siming Fu1 Jian Wu1 Zuozhu Liu*, 1 1 Zhejiang University 2 Fudan University yuan2.24, zuozhuliu@intl.zju.edu.cn 1 Introduction Fueled by massive medical vision-language corpora, Medical Vision-Language Models (Med-VLMs) have risen to prominence through a unified paradigm: verbalizing anything they see. This formulation endows them with powerful visual understanding and versatile linguistic generation capabilities, substantially advancing tasks ranging from visual question answering and report generation to medical reasoning (Li et al., 2023; Moor et al., 2023; Wu et al., 2025a; Zhang et al., 2024; Chen et al., 2024; Wang et al., 2026, 2025c; Liu et al., 2024). Among these tasks, precise visual perception, such as grounding and segmentation, serves as an indispensable prerequisite for trustworthy medical image understanding, providing explicit localizations of pathological regions and key visual cues before decision-making (Kirillov et al., 2023; Ma et al., 2024; Luo et al., 2025). Figure 1: Prior medical VLMs either lack native text-guided segmentation, rely on external tools, or suffer from the gap between text and images. MedUP introduces UniMedTok, a native mask-token interface that unifies text-guided segmentation, region-grounded understanding, and medical VQA within one VLM. However, the paradigm of verbalizing anything constrains how Med-VLMs achieve perception natively: existing models resort to a text-centric strategy, wherein spatial references are verbalized as discrete numerical strings, such as bounding-box coordinates and segmentation keypoints (Chen et al., 2023), as shown in Figure 1 (A). This formulation suffers from inherent limitations: Med-VLMs lack spatial sensitivity to such coordinate strings, fundamentally disconnecting the localization semantics they encode from the visual feature space. Alternatively, a line of work pursues an orthogonal solution: equipping Med-VLMs with external segmentation modules. Representative approaches include tool-using agent models, as shown in Figure 1 (B), which orchestrate off-the-shelf models (e.g., SAM) as callable tools during inference (Li et al., 2024; Jiang et al., 2026), and dual-decoder architectures, as shown in Figure 1 (C), which completely decouple the output space into an LLM branch for linguistic content understanding and a dedicated visual decoder for segmentation mask prediction (Lai et al., 2024; Huang et al., 2025a, b). Both paradigms, however, introduce notable drawbacks: they incur substantial additional parameters and architectural complexity. More fundamentally, whether by delegating perception to external tools or routing it through a separate decoder, the decoupling of understanding and perception creates a significant representation gap between the two capabilities. As a result, this decoupled design often leads to ineffective region-language alignment, as we empirically verify in Table 4, where decoupled Med-VLMs deliver underwhelming performance on visually-grounded tasks. These observations motivate a fundamental question: Can we natively unify understanding and perception of Med-VLMs within a shared representation space? We argue that the key insight is to tokenize regions into discrete mask tokens within the shared token space of language, i.e., region as language. Guided by this principle, we present MedUP, a medical VLM for unified region-language modeling. At its core lies UniMedTok, a native region tokenizer that encodes medical regions as discrete mask tokens and then aligns them with their language counterparts (Wang et al., 2025b; Zhou et al., 2026; Lai et al., 2024). Consequently, MedUP can seamlessly interleave mask tokens with text in a single sequence, grounding pathological findings to precise regions, and describing arbitrary regions in natural language, thereby achieving unified perception and understanding. We train MedUP in two stages. In Stage 1, UniMedTok is pretrained via masked region reconstruction, where it learns to encode masks into discrete tokens and decode them back into masks. In Stage 2, we align UniMedTok with the VLMs, teaching it to natively “speak” mask tokens within text sequences. To this end, we curate a large-scale training corpus, UniMed-Train, comprising 902,648 text-guided segmentation samples, 902,648 region-grounded understanding samples, 4,000 CoT-based text-to-mask reasoning samples, and 27,738 standard medical image understanding samples, 1,837,034 training instances in total. To evaluate MedUP systematically, we further build UniMed-Bench, a unified medical region-language benchmark with three tasks: Medical VQA, Text-Guided Segmentation, and Region-Grounded Understanding. This benchmark is designed to test not only whether a model can answer questions correctly, but also whether it can associate answers with the right medical regions and operate bidirectionally between language and masks. Under this protocol, we compare general-domain VLMs, medical VLMs, adapted mask-token baselines, and MedUP. We further introduce Seg-CoT task, a segmentation-oriented chain-of-thought paradigm for text-to-mask prediction (Wei et al., 2022; Lai et al., 2024). Instead of treating mask generation as a direct decoding problem alone, Seg-CoT encourages the model to produce segmentation through intermediate reasoning about anatomy, abnormality attributes, and localization cues. This improves semantic grounding and makes mask prediction more compatible with the reasoning behavior already exhibited by large vision-language models. Extensive experiments demonstrate that MedUP achieves strong and consistent performance across all three tasks in UniMed-Bench, outperforming native Med-VLMs, agentic Med-VLMs, and dual-decoder models, while remaining competitive with specialist medical segmentors (e.g., MedSAM1–3) on text-guided segmentation. Our contributions are three-fold: Architectural Contribution. We propose MedUP, a Med-VLM equipped with UniMedTok, a native region tokenizer that encodes masks as discrete tokens within the LLM vocabulary, unifying perception and understanding. Data Contribution. We curate UniMed-Train (1.84M instances) and UniMed-Bench, providing the large-scale region-language corpus and benchmark for bidirectional medical region-language evaluation, including Seg-CoT, a new reasoning-guided segmentation paradigm. Empirical Contribution. MedUP consistently outperforms native, agentic, and dual-decoder Med-VLMs across all tasks, while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding-perception modeling. Figure 2: Overview of MedUP. MedUP is built on UniMedTok, a native mask-token interface for grounded medical vision-language modeling. Stage 1 learns a medical mask tokenizer that converts region masks into compact discrete tokens, and Stage 2 trains the VLM on four supervision streams from UniMed-Train: Medical VQA, Text-Guided Segmentation, Region-Grounded Understanding, and Seg-CoT. The same interface supports both mask-as-output and mask-as-input inference and is evaluated on UniMed-Bench. 2 Methods 2.1 Problem Formulation We study grounded medical vision-language modeling under a unified region-language setting. Given a medical image I, a text instruction or question X, and an optional region mask M, the model is required to support three downstream task types: (1) Medical VQA, where the output is a free-form textual answer; (2) Text-Guided Segmentation, where the output is a segmentation mask corresponding to a language description; and (3) Region-Grounded Understanding, where the model receives a target region and generates a clinically meaningful description, label, or answer conditioned on that region. The goal is to model these tasks in one autoregressive framework rather than by coupling separate segmentation and language systems. Formally, we introduce a mask serialization operator S(⋅)S(·) that maps a dense region mask into a short token span, and denote the output sequence by Y. MedUP models all tasks with a single conditional autoregressive distribution pθ(Y∣I,X)=∏t=1|Y|pθ(yt∣I,X,y<t).p_θ(Y I,X)= _t=1^|Y|p_θ(y_t I,X,y_<t). (1) The three tasks differ only in how the input-output pair is instantiated. For Medical VQA, the target sequence is a textual answer A. For Text-Guided Segmentation, the target sequence is the serialized mask span S(M)S(M). For Region-Grounded Understanding, the serialized region span S(M)S(M) is appended to the instruction as part of the conditioning context, and the model predicts the textual answer A. This formulation reduces region understanding and region generation to next-token prediction in a shared text-mask space. 2.2 Overview MedUP is built around UniMedTok, a native mask-token interface that places medical regions in the same autoregressive space as text. The system has two stages. In Stage 1, we train a medical mask tokenizer to convert a region mask into compact discrete codes and reconstruct the mask from them. In Stage 2, we expand the VLM vocabulary with mask tokens, convert all region-related supervision into text-mask sequences, and jointly train on the four streams of UniMed-Train: Medical VQA, Text-Guided Segmentation, Region-Grounded Understanding, and Seg-CoT. Figure 2 summarizes the two-stage design of MedUP. This design separates mask representation learning from region-language modeling. The tokenizer is responsible for faithful bidirectional conversion between dense masks and discrete codes, while the VLM only needs to learn how to read and generate these codes in context. As a result, UniMedTok avoids adding a trainable segmentation head inside the language model. Full tokenizer architecture and implementation details are deferred to Appendix C. 2.3 Stage 1: Medical Mask Tokenizer We implement UniMedTok as an image-conditioned vector-quantized mask autoencoder. Given an image I and region mask M, the tokenizer encodes the mask into a continuous representation and then compresses it with residual vector quantization into an ordered two-code representation q=Q(Etok(I,M))=[c1,c2],c1,c2∈0,…,255,q=Q(E_tok(I,M))=[c_1,c_2], c_1,c_2∈\0,…,255\, (2) yielding an MT256×2 tokenization scheme. The discrete code pair is then decoded, conditioned on the same image, back into a dense mask M^=Dtok(I,q) M=D_tok(I,q). Because decoding remains image-conditioned, the codes act as compact region prompts rather than standalone pixel descriptions. Stage 1 is trained as a mask reconstruction objective with quantization regularization, and the tokenizer is frozen after convergence for all downstream Stage-2 data construction and inference. In practice, we use a medical SAM2-style backbone, non-shared codebooks, and image-plus-box conditioning for localization stability. The full architecture, residual quantization procedure, and losses are provided in Appendix C. 2.4 Stage 2: Mask Tokens as Language Vocabulary expansion. After training the tokenizer, we convert each discrete code into a textual special token. Specifically, we add a start token <|mt_start|>, an end token <|mt_end|>, and 512 mask code tokens <|mt_0000|> to <|mt_0511|> to the VLM vocabulary. Since each mask uses two codebook levels of size 256, the first token corresponds to the first codebook and the second token corresponds to the second codebook with an offset of 256. A mask is therefore represented as S(M)=[ts,tc1,t256+c2,te]S(M)=[t_s,\,t_c_1,\,t_256+c_2,\,t_e] (3) This textualization turns each mask into a short, language-compatible span that can be inserted into prompts or generated as output. Mask-as-input. For mask-grounded understanding, we encode the target region with the frozen tokenizer and insert the resulting mask-token span into the user prompt: Xreg=[X;S(M)].X_reg=[X;S(M)]. (4) The model then answers questions conditioned on both the image and the explicit region reference. This formulation allows region-level reasoning without modifying the base VLM architecture. In contrast to crop-based or overlay-based prompting, the mask is represented in a symbolic form that can be composed with arbitrary text instructions. Mask-as-output. For text-guided segmentation, the VLM autoregressively generates a mask-token span in response to a referring instruction, q^=[c^1,c^2]∼pθ(⋅∣I,X),M^=Dtok(I,q^), q=[ c_1, c_2] p_θ(· I,X), M=D_tok(I, q), (5) where the generated tokens are parsed into a code pair before decoding. Thus, the VLM itself only generates short discrete codes, while pixel-level reconstruction is handled by the frozen tokenizer learned in Stage 1. Figure 3: Overview of the unified grounded medical framework and the UniMed corpus. The framework unifies segmentation, region understanding, medical VQA, and Seg-CoT reasoning through shared mask tokens across seven imaging modalities. 2.5 Unified Multi-Task Training We train the Stage-2 VLM with mixed supervision from the four streams of UniMed-Train. All samples are converted into standard conversational sequences, so optimization remains the usual autoregressive next-token loss: ℒstage2=−∑(I,X,Y)∈∑t=1|Y|logpθ(yt∣I,X,y<t),L_stage2=- _(I,X,Y) _t=1^|Y| p_θ(y_t I,X,y_<t), (6) where =∪k=14kD= _k=1^4D_k merges Medical VQA, Text-Guided Segmentation, Region-Grounded Understanding, and Seg-CoT. This unified objective lets the model answer image-level medical questions, generate mask tokens from text, interpret mask tokens as symbolic region references, and perform reasoning-augmented text-to-mask prediction within a single training pipeline. In Stage 2, the tokenizer is frozen and only the VLM is optimized. No segmentation-specific reconstruction loss is used in this stage; cross-task transfer is induced entirely by next-token prediction over mixed text-mask sequences. For Seg-CoT specifically, the target is written as a concatenated reasoning-and-mask sequence Y=[R;S(M)]Y=[R;S(M)], where R denotes the intermediate textual rationale. Detailed optimization settings and backbone-specific implementation choices are provided in Appendix C. 3 UniMed-Train and UniMed-Bench 3.1 UniMed-Train: Stage-2 Training Corpus As illustrated in Figure 3, Stage 2 is trained on UniMed-Train, a four-stream medical instruction corpus covering Text-Guided Segmentation, Region-Grounded Understanding, Medical VQA, and Seg-CoT. The current release contains 902,648 text-guided segmentation samples, 902,648 region-grounded understanding samples, 27,738 Medical VQA samples, and 4,000 reasoning-augmented Seg-CoT samples, for a total of 1,837,034 instances. The two mask-centric streams are constructed from 80+180+1 medical segmentation datasets using the frozen Stage-1 tokenizer: one stream trains mask-as-output generation from referring text, while the other trains mask-as-input understanding by inserting the serialized region into the question context. Medical VQA preserves image-level clinical reasoning, and Seg-CoT adds intermediate anatomical, attribute, and localization reasoning before the final mask-token span. Because not all datasets are equally compatible with a compact two-token mask representation, we apply round-trip filtering before Stage 2: each ground-truth mask is encoded and decoded by the tokenizer, and low-fidelity datasets are downsampled according to reconstruction quality. This filtering is applied consistently to both mask-centric streams and improves training stability. Detailed construction templates, prompt formats, and filtering procedures are deferred to Appendix C.4, Appendix C.2, and Figure 6. 3.2 UniMed-Bench: Unified Evaluation Benchmark We build UniMed-Bench as a held-out benchmark for unified grounded medical vision-language evaluation. It covers three tasks: Medical VQA, Text-Guided Segmentation, and Region-Grounded Understanding. Medical VQA includes 8,273 test questions. Text-Guided Segmentation contains 219,636 test samples from an 80-dataset benchmark family. Region-Grounded Understanding is built from the same 80 datasets, with 218,244 v2_tokens samples and 219,257 v1_masks samples. In the current benchmark instantiation, most region-understanding questions are concise category- or label-oriented prompts. In the main paper, we report answer accuracy for Medical VQA, Dice/IoU for Text-Guided Segmentation, and exact match for Region-Grounded Understanding, with weighted token recall used as a complementary detailed metric. Full evaluation details are provided in Appendix C.2. Figure 4: Effect of training data scale on Text-Guided Segmentation and Region-Grounded Understanding. (A) Increasing the training data scale from 10% to 100% consistently improves weighted Dice and weighted IoU in text-guided segmentation, with a 0.099 gain in weighted Dice. (B) Region-grounded understanding also improves steadily with scale, and v2_tokens consistently outperforms v1_masks. 4 Experiments Table 1: Main results on UniMed-Bench. We report sample-weighted overall accuracy for Medical VQA, macro Dice for Text-Guided Segmentation, and exact match for Region-Grounded Understanding. For baselines without native mask tokens, Region-Grounded Understanding is evaluated under v1_masks. Consistent with the Introduction, the comparison set includes native medical VLM baselines, agentic grounded models, and dual-decoder or externally grounded baselines. For readability, the table regroups these methods by their concrete region interface or grounding mechanism. Methods without complete three-task coverage are reported in Tables 4, 3 and 5. Method Region Interface -Level Reasoning -Grounded Capabilities Medical VQA Acc. ↑ Text-Guided Seg. mDice ↑ Region-Grounded Und. EM ↑ Medical VLMs without native mask-token interfaces HealthGPT-M3 image-level only 45.3 – 4.4 UniBiomed image-level only 7.7 37.9 0.0 Dual-decoder or externally grounded baselines LISA++ external mask decoder 30.2 19.2 0.0 SAM4MLLM SAM-assisted grounding 21.5 14.8 3.0 Agentic grounded medical models MMedAgent agent + external tools 4.5 27.9 0.0 MedUP-Q native mask tokens 63.5 64.4 78.5 MedUP-H native mask tokens 66.5 67.9 81.8 Table 2: Medical VQA results on UniMed-Bench. We report closed-question accuracy. Model SLAKE PathVQA VQA-RAD Baselines LLaVA-Med 0.8534 0.9121 0.8419 MedGemma 0.8269 0.6299 0.8207 UniBiomed 0.1490 0.1607 0.1213 LISA++ 0.5769 0.5851 0.5478 SAM4MLLM 0.5192 0.3586 0.4081 MMedAgent 0.0673 0.0717 0.3603 MedUP-Q 0.9063 0.9219 0.8606 MedUP-H 0.9135 0.9354 0.8493 Table 3: Per-modality micro Dice on text-guided medical image segmentation. Specialist segmentors receive oracle visual prompts, whereas MedUP receives only text instructions. Micro Dice is computed from globally accumulated intersections and mask areas within each modality group. Modality Segmentors / VLM Baselines MedSAM1 MedSAM2 MedSAM3 BiomedParse v2 UniBiomed LISA++ SAM4MLLM MMedAgent MedUP-H MedUP-Q CT 0.8431 0.7006 0.5007 0.3646 0.4782 0.1353 0.0685 0.3980 0.9206 0.9137 MR 0.7437 0.6709 0.3873 0.3173 0.4923 0.1059 0.0519 0.2774 0.7414 0.6891 Ultrasound 0.8849 0.7754 0.5667 0.4033 0.3718 0.2399 0.1411 0.2509 0.8428 0.8265 Endoscopy 0.9257 0.9407 0.9017 0.0030 0.7472 0.5863 0.3601 0.4708 0.8760 0.8361 Fundus 0.9257 0.9353 0.6958 0.0000 0.4710 0.0712 0.0925 0.2090 0.8566 0.8607 Dermoscopy 0.9485 0.9424 0.9095 0.0004 0.5567 0.5398 0.4172 0.4854 0.8473 0.8616 X-ray 0.9614 0.9501 0.9601 0.0137 0.9344 0.7615 0.4358 0.6572 0.9523 0.9603 Overall 0.8858 0.8104 0.6434 0.2204 0.5589 0.1955 0.1072 0.4123 0.8906 0.8885 Table 4: Protocol comparison for Region-Grounded Understanding on UniMed-Bench across the Qwen-based and Hulu-based backbones. v1_masks exposes the target region visually, whereas v2_tokens uses discrete mask tokens as the region reference. Each cell reports EM and Token Recall. Backbone Protocol EM (%) Recall (%) Qwen-based v1_masks 49.8 49.8 2_tokens 78.5 78.5 Hulu-based v1_masks 49.8 49.8 2_tokens 81.8 81.8 Table 5: Per-modality weighted token recall on Region-Grounded Understanding. Baselines use v1_masks, while MedUP-H/Q use v2_tokens. Modality / VLM Baselines VLM Baselines UniBiomed LISA++ SAM4MLLM MMedAgent LLaVA-Med MedGemma MedUP-H MedUP-Q CT 0.0499 0.0005 0.0495 0.0000 0.0382 0.3123 0.9347 0.9242 MR 0.0137 0.0000 0.0011 0.0000 0.0119 0.3038 0.6271 0.5568 Ultrasound 0.0006 0.0000 0.0000 0.0000 0.0936 0.9857 0.8762 0.8206 Endoscopy 0.3529 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Fundus 0.0000 0.0000 0.0000 0.0000 0.0535 0.6310 0.9450 0.9609 Dermoscopy 0.0000 0.0000 0.0000 0.0000 0.2237 0.9956 0.9967 1.0000 X-ray 0.1404 0.1754 0.9561 0.0000 0.4386 1.0000 1.0000 1.0000 Overall 0.0359 0.0004 0.0302 0.0000 0.0297 0.3277 0.8183 0.7845 4.1 Experimental Setup We train MedUP on UniMed-Train and evaluate it on UniMed-Bench. Our main models are MedUP-H, built on HuluMed-4B Jiang et al. (2025b), and MedUP-Q, built on Qwen3-VL-4B (Bai et al., 2025). Both variants share the same Stage-1 MT256x2 tokenizer, round-trip-filtered training data, and four-stream Stage-2 objective. Medical VQA is evaluated on SLAKE, PathVQA, and VQA-RAD with overall accuracy; Text-Guided Segmentation is evaluated on 80 datasets with mean Dice; and Region-Grounded Understanding is evaluated on the same dataset family under both v2_tokens and v1_masks with exact match. Consistent with the framing in the Introduction, we compare MedUP against three main baseline families: native medical VLMs without native mask-token interfaces, represented by HealthGPT-M3 Lin et al. (2025) and UniBiomed Wu et al. (2025b); dual-decoder or externally grounded baselines, represented by LISA++ (Lai et al., 2024) and SAM4MLLM; and agentic grounded medical models, represented by MMedAgent Li et al. (2024). Because these methods expose regions through different concrete mechanisms, the main table further regroups them by region interface or grounding mechanism for presentation clarity. Unless otherwise stated, all methods are evaluated under the same data splits, task formats, and decoding settings. Detailed optimization settings are provided in Appendix C. 4.2 Main Results on the Unified Benchmark Table 4 reports the unified comparison across Medical VQA, Text-Guided Segmentation, and Region-Grounded Understanding. Across both backbones, MedUP achieves the strongest overall results among the methods included in this three-task setting, outperforming native medical VLM baselines, agentic grounded models, and dual-decoder or externally grounded baselines on the grounded tasks while remaining strong on Medical VQA. MedUP-H reaches 66.566.5 accuracy, 67.967.9 mDice, and 81.881.8 exact match, while MedUP-Q reaches 63.563.5, 64.464.4, and 78.578.5, showing that the proposed interface transfers across both medical-domain and more general multimodal foundations. 4.3 Comparison with Specialist Medical Segmentors Table 3 compares MedUP with specialist medical segmentors on text-guided segmentation. Although specialist models receive stronger oracle visual prompts, MedUP remains competitive across modalities and substantially outperforms the grounded / VLM baselines under micro-Dice. This highlights the practical value of native text-driven segmentation: MedUP trades some oracle-prompt advantage for a much more flexible language interface while retaining strong dense localization performance. 4.4 Protocol Study for Mask-Grounded Understanding To understand whether discrete mask tokens are an effective interface for region-grounded understanding, we compare MedUP with alternative region-presentation protocols. The v1_masks setting exposes the target region visually, whereas v2_tokens represents the region in a language-compatible token space. This experiment isolates the benefit of the interface itself. Table 4 shows a large protocol gap across both the Qwen-based and Hulu-based backbones, while Table 5 reports the per-modality weighted token recall comparison. Across all settings, v2_tokens consistently outperforms v1_masks, indicating that discrete mask tokens provide a more effective interface for region-grounded understanding. The results further suggest that the region interface is a core factor in connecting localized visual evidence with language reasoning. Under v1_masks, the model must additionally map visual overlays to textual semantics, which becomes fragile for small or anatomically ambiguous regions. In contrast, v2_tokens places both region references and linguistic context within a shared token space, reducing the representation gap between visual grounding and language reasoning and leading to substantially stronger performance. 4.5 Effect of Training Data Scale Figure 4 shows that both grounded tasks improve as the amount of Stage-2 training data increases. For Text-Guided Segmentation, weighted Dice and weighted IoU rise consistently from 10% to 100% data scale, with a total Dice gain of 0.099. Region-Grounded Understanding follows the same trend, and v2_tokens remains stronger than v1_masks at every scale. These results indicate that the proposed interface continues to benefit from additional supervision and remains favorable throughout the tested data regime. 4.6 Per-Task and Protocol Analysis Beyond the unified main table, Tables 4, 3, 4, and 5 provide task-specific views of the benchmark. Table 4 reports the closed-question Medical VQA breakdown on SLAKE, PathVQA, and VQA-RAD. Table 3 compares text-guided segmentation against specialist medical segmentors and grounded VLM baselines under per-modality micro Dice. Tables 4 and 5 analyze region-grounded understanding through protocol comparison and modality-level token recall, respectively. Figure 4 complements these tables with a training-scale analysis for the two grounded tasks. We leave additional ablations such as round-trip filtering, token budget, and Seg-CoT training effects for future versions once the corresponding experimental evidence is included. 4.7 Effect of Round-trip Filtering Figure 5: Effect of round-trip filtering on text-guided segmentation. We compare training with and without round-trip filtering for both MedUP-Q and MedUP-H. Filtering low-fidelity mask-token supervision improves macro Dice consistently across backbones, yielding gains of +7.1 for MedUP-Q and +9.3 for MedUP-H. Figure 5 isolates the contribution of our round-trip filtering strategy. Without filtering, tokenizer reconstruction errors introduce noisy supervision into Stage-2 mask generation, especially on datasets with small, irregular, or semantically ambiguous regions. After filtering, both backbones improve substantially: MedUP-Q increases from 57.3 to 64.4 mDice, while MedUP-H rises from 58.6 to 67.9. The larger gain on MedUP-H suggests that stronger medical backbones can better exploit cleaner tokenized mask supervision once low-fidelity training cases are removed. 4.8 Qualitative Analysis We provide qualitative examples for text-guided segmentation, region-grounded understanding, and failure cases. In particular, we visualize how Seg-CoT changes generation behavior, whether the predicted masks are semantically aligned with the reasoning trace, and how the same region is handled under crop, overlay, and mask-token protocols. Details are shown in Appendix D. 5 Conclusion We introduced MedUP, a family of grounded medical vision-language models built on UniMedTok, a unified mask-token interface for region-language modeling. By treating masks as discrete language-compatible tokens, MedUP unifies Medical VQA, Text-Guided Segmentation, and Region-Grounded Understanding within a single autoregressive framework. We further organized training as the four-stream UniMed-Train corpus and evaluation as the three-task UniMed-Bench, with Seg-CoT improving text-to-mask generation through intermediate anatomical and localization reasoning. These results suggest that native region-language interfaces are a promising direction for building grounded medical VLMs. Limitations While MedUP shows that a native mask-token interface is effective for unified medical understanding and perception, several aspects remain open for further study. Our current region representation is intentionally compact, and future work may explore whether richer tokenizations are helpful for some very small, irregular, or visually subtle structures. Our evaluation also focuses primarily on offline benchmark settings, and it would be valuable to further study behavior in more deployment-oriented scenarios such as interactive refinement, longitudinal workflows, or distribution shift. In addition, although we study two backbones under a shared interface, broader validation across model scales and training regimes would help better characterize the generality of the proposed design. We view these as natural next steps for extending native region-language modeling toward more realistic medical applications. References Aho and Ullman (1972) Alfred V. Aho and Jeffrey D. Ullman. 1972. The Theory of Parsing, Translation and Compiling, volume 1. Prentice-Hall, Englewood Cliffs, NJ. American Psychological Association (1983) American Psychological Association. 1983. Publications Manual. American Psychological Association, Washington, DC. Ando and Zhang (2005) Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. Andrew and Gao (2007) Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40. Bai et al. (2025) Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cao et al. (2023) Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. 2023. Swin-unet: Unet-like pure transformer for medical image segmentation. In Computer Vision – ECCV 2022 Workshops, pages 205–218. Springer. Chandra et al. (1981) Ashok K. Chandra, Dexter C. Kozen, and Larry J. Stockmeyer. 1981. Alternation. Journal of the Association for Computing Machinery, 28(1):114–133. Chen et al. (2024) Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. Preprint, arXiv:2406.19280. Chen et al. (2023) Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm’s referential dialogue magic. Preprint, arXiv:2306.15195. Gusfield (1997) Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK. Hatamizadeh et al. (2022) Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R. Roth, and Daguang Xu. 2022. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 574–584. Huang et al. (2025a) Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. 2025a. Towards a multimodal large language model with pixel-level insight for biomedicine. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4):3779–3787. Huang et al. (2025b) Yu Huang, Zelin Peng, Yichen Zhao, Piao Yang, Xiaokang Yang, and Wei Shen. 2025b. Medseg-r: Reasoning segmentation in medical images with multimodal large language models. Preprint, arXiv:2506.10465. Isensee et al. (2021) Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. 2021. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211. Jiang et al. (2025a) Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, Jin Hao, Zijian Chen, Ruijia Wu, Tao Tang, Junhui Lv, Hongxia Xu, Hongwei Wang, Jun Xiao, Bin Feng, and 6 others. 2025a. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. Preprint, arXiv:2510.08668. Jiang et al. (2025b) Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, and 1 others. 2025b. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668. Jiang et al. (2026) Yankai Jiang, Qiaoru Li, Binlu Xu, Haoran Sun, Chao Ding, Junting Dong, Yuxiang Cai, Xuhong Zhang, and Jianwei Yin. 2026. Ibisagent: Reinforcing pixel-level visual reasoning in mllms for universal biomedical object referring and segmentation. Preprint, arXiv:2601.03054. Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026. Lai et al. (2024) Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589. Li et al. (2024) Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and 1 others. 2024. Mmedagent: Learning to use medical tools with multi-modal agent. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8745–8760. Li et al. (2023) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Advances in Neural Information Processing Systems, volume 36. Datasets and Benchmarks Track. Lin et al. (2025) Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, and 1 others. 2025. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. arXiv preprint arXiv:2502.09838. Liu et al. (2024) Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Tianyi Zhou, and Zuozhu Liu. 2024. Medcot: Medical chain of thought via hierarchical expert. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17371–17389. Luo et al. (2025) Lingxiao Luo, Bingda Tang, Xuanzhong Chen, Rong Han, and Ting Chen. 2025. Vividmed: Vision language model with versatile visual grounding for medicine. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1687–1704. Ma et al. (2024) Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. 2024. Segment anything in medical images. Nature Communications, 15(1):654. Moor et al. (2023) Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Proceedings of the 3rd Machine Learning for Health Symposium, volume 225 of Proceedings of Machine Learning Research, pages 353–367. PMLR. Peng et al. (2024) Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Kosmos-2: Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations. Rasooli and Tetreault (2015) Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv:1503.06733. Version 2. Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer. Sellergren et al. (2026) Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cian Hughes, Charles Lau, and 1 others. 2026. Medgemma technical report. Preprint, arXiv:2507.05201. Tong et al. (2025) Qinyue Tong, Ziqian Lu, Jun Liu, Yangming Zheng, and Zheming Lu. 2025. Medisee: Reasoning-based pixel-level perception in medical images. Preprint, arXiv:2504.11008. Trinh et al. (2026) Quoc-Huy Trinh, Minh-Van Nguyen, Jun Zeng, Debesh Jha, and Ulas Bagci. 2026. Prs-med: Position reasoning segmentation in medical imaging. Preprint, arXiv:2505.11872. Wang et al. (2025a) Guoxin Wang, Jun Zhao, Xinyi Liu, Yanbo Liu, Xuyang Cao, Chao Li, Zhuoyun Liu, Qintian Sun, Fangru Zhou, Haoqiang Xing, and Zhenhong Yang. 2025a. Citrus-v: Advancing medical foundation models with unified medical image grounding for clinical reasoning. Preprint, arXiv:2509.19090. Wang et al. (2025b) Tao Wang, Changxu Cheng, Lingfeng Wang, Senda Chen, and Wuyue Zhao. 2025b. Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23267–23278. Wang et al. (2026) Yuan Wang, Shujian Gao, Jiaxiang Liu, Songtao Jiang, Xia Haoxiang, Xiaotian Zhang, Zhaolu Kang, Yemin Wang, and Zuozhu Liu. 2026. Beyond n-grams: A hierarchical reward learning framework for clinically-aware medical report generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33719–33727. Wang et al. (2025c) Yuan Wang, Jiaxiang Liu, Shujian Gao, Bin Feng, Zhihang Tang, Xiaotang Gai, Jian Wu, and Zuozhu Liu. 2025c. V2t-cot: From vision to text chain-of-thought for medical reasoning and diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 658–668. Springer. Wang et al. (2025d) Ziyue Wang, Junde Wu, Linghan Cai, Chang Han Low, Xihong Yang, Qiaxuan Li, and Yueming Jin. 2025d. Medagent-pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow. arXiv preprint arXiv:2503.18968. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Wu et al. (2025a) Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. 2025a. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communications, 16:7866. Wu et al. (2025b) Linshan Wu, Yuxiang Nie, Sunan He, Jiaxin Zhuang, Luyang Luo, Tao Li, Zhuoyao Xie, Dexuan Chen, Yinghua Zhao, Neeraj Mahboobani, Varut Vardhanabhuti, Ronald Cheong Kin Chan, Yifan Peng, Pranav Rajpurkar, and Hao Chen. 2025b. Unibiomed: A universal foundation model for grounded biomedical image interpretation. Preprint, arXiv:2504.21336. Zhang et al. (2024) Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D. Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, and 4 others. 2024. A generalist vision-language foundation model for diverse biomedical tasks. Nature Medicine, 30(11):3129–3141. Zhou et al. (2026) Yikang Zhou, Tao Zhang, Dengxian Gong, Yuanzheng Wu, Ye Tian, Haochen Wang, Haobo Yuan, Jiacong Wang, Lu Qi, Hao Fei, Anran Wang, Zhuochen Wang, Yujing Wang, Cheng Chen, Shunping Ji, and Xiangtai Li. 2026. Samtok: Representing any mask with two words. Preprint, arXiv:2601.16093. Appendix In this appendix, we provide additional related-work discussion, dataset statistics, implementation details, qualitative case studies, and detailed benchmark analysis. The content structure is outlined as follows: • Section A - Related Work • Section B - UniMed Dataset Overview • Section C - Implementation Details – Section C.1 - Overall Pipeline – Section C.2 - Evaluation Details – Section C.3 - Hyperparameters – Section C.4 - Prompt and Data Format • Section D - Case Study of Seg-CoT • Section E - UniMed-Bench Detailed Analysis Appendix A Related Work A.1 Medical Image Segmentation. Medical image segmentation has long served as the foundation of pixel-level medical perception. Classical encoder–decoder architectures such as U-Net established dense prediction as a standard formulation for biomedical image analysis, while nnU-Net further showed the importance of self-configuring pipelines and task-adaptive training protocols (Ronneberger et al., 2015; Isensee et al., 2021). Transformer-based models such as Swin-Unet and UNETR extend this paradigm by incorporating long-range spatial modeling for 2D and 3D medical images (Cao et al., 2023; Hatamizadeh et al., 2022). More recently, promptable segmentation models such as SAM and MedSAM have shifted segmentation toward interactive region localization, enabling strong generalization across anatomical structures, lesions, and imaging modalities (Kirillov et al., 2023; Ma et al., 2024). Despite their strong localization capability, these methods are primarily designed for mask prediction or region delineation. They do not naturally support open-ended medical language interaction, diagnosis-oriented reasoning, or bidirectional mask-language understanding. Thus, traditional and promptable medical segmentors can localize regions but remain largely disconnected from grounded medical vision-language modeling. A.2 Pixel-Level Understanding in Medical MLLMs. Recent medical vision-language models have demonstrated strong capabilities in medical visual question answering, report understanding, clinical dialogue, and diagnosis-oriented reasoning (Li et al., 2023; Moor et al., 2023; Zhang et al., 2024; Chen et al., 2024; Jiang et al., 2025a; Sellergren et al., 2026). However, most of these models still represent visual evidence at the image level, making it difficult to associate generated language with precise anatomical structures or abnormal regions. To bridge language reasoning with fine-grained spatial grounding, recent pixel-level MLLMs introduce segmentation into multimodal reasoning. A representative direction is LISA-style reasoning segmentation, where MLLMs generate implicit segmentation representations that are decoded into masks through external segmentation modules (Lai et al., 2024). Subsequent medical adaptations further extend this paradigm to biomedical grounding, clinical reasoning, and reasoning-guided segmentation (Huang et al., 2025a; Wang et al., 2025a; Wu et al., 2025b; Tong et al., 2025; Trinh et al., 2026; Huang et al., 2025b), marking an important transition from image-level medical understanding toward grounded pixel-level perception. Despite this progress, many existing grounded medical MLLMs remain decoder-centric or tool-centric. LISA-style methods typically rely on implicit segmentation embeddings and external decoders, while recent agentic approaches such as IBISAgent reformulate segmentation as iterative reasoning and interaction with external segmentation tools (Jiang et al., 2026; Wang et al., 2025d). Although these methods improve grounding and refinement ability, masks are still treated as outputs of decoders or tools rather than native representations within the autoregressive language space. Another emerging direction explores mask-as-language modeling, where segmentation masks are represented as discrete language-compatible tokens for autoregressive prediction. Recent methods such as SAMTok and HiMTok demonstrate the feasibility of unified text-mask interaction within a shared token space (Zhou et al., 2026; Wang et al., 2025b). However, existing mask-token approaches mainly focus on general-domain visual grounding, whereas medical grounded understanding introduces additional challenges including anatomical ambiguity, abnormality semantics, clinically meaningful localization, and segmentation-oriented reasoning. MedUP addresses this medical setting through a unified region-language interface, with UniMedTok integrating Medical VQA, Text-Guided Segmentation, Region-Grounded Understanding, and segmentation-oriented reasoning within a single grounded medical VLM. Appendix B UniMed Dataset Overview Figure 6: Overview of UniMed-Train and UniMed-Bench. UniMed-Train combines four supervision streams for Stage-2 training, while UniMed-Bench provides held-out evaluation across Medical VQA, Text-Guided Segmentation, and Region-Grounded Understanding. Figure 6 summarizes both the training composition of UniMed-Train and the evaluation coverage of UniMed-Bench. As shown in the left donut chart, UniMed-Train is dominated by the two mask-centric supervision streams, with 902,648 Text-Guided Segmentation samples and 902,648 Region-Grounded Understanding samples, together accounting for roughly 98% of the full corpus. The remaining supervision comes from 27,738 Medical VQA samples and 4,000 Seg-CoT samples. This mixture reflects the core design of MedUP: image-level reasoning is preserved, but the majority of Stage-2 learning signal is devoted to teaching the model how to read, generate, and reason over localized medical regions through the shared mask-token interface. The right panel highlights the modality and dataset diversity of UniMed-Bench. The benchmark spans 80 held-out datasets across seven imaging modalities, with MR and CT forming the largest portions (46.2% and 30.0%, respectively), followed by Endoscopy and Fundus (7.5% each), Ultrasound and Dermoscopy (3.8% each), and a smaller X-ray portion (1.2%). The nested benchmark chart further shows that this coverage is not concentrated in only a few datasets: the inner ring captures modality-level balance, while the outer ring exposes substantial dataset-level variation in scale. Together, these statistics illustrate that UniMed-Bench evaluates unified region-language modeling not only on dominant cross-sectional modalities such as CT and MR, but also on long-tail clinical settings where visual appearance, anatomy, and grounding difficulty differ substantially. Appendix C Implementation Details We summarize the concrete implementation of MedUP, including the two-stage training pipeline, evaluation protocol, optimization settings, and the prompt/data format used to unify the three tasks in UniMed-Bench. C.1 Overall Pipeline Our implementation follows a two-stage design. In Stage 1, we train a VQ-based mask tokenizer on medical segmentation data using a SAM2-style image encoder and a residual vector quantization bottleneck. The tokenizer compresses each binary region mask into a compact MT256x2 representation with codebook size 256, codebook depth 2, non-shared codebooks, and latent dimension 256. All images are resized to 1024×1024 before tokenizer encoding and decoding. After Stage 1 converges, we freeze the tokenizer and export its weights for downstream use. We then expand the Stage-2 backbone vocabulary with 514 mask-related special tokens: <|mt_start|>, <|mt_end|>, and 512 code tokens from <|mt_0000|> to <|mt_0511|>. These tokens form the native region-language interface used by MedUP. In Stage 2, all training streams are converted into a unified conversational format and optimized with standard autoregressive next-token prediction. For text-guided segmentation, the model predicts a short mask-token span; for region-grounded understanding, the mask-token span is inserted into the user prompt as a symbolic region reference; for Medical VQA, the model answers directly from the image-question pair. Our released medical training pipeline supports both Qwen3-VL-4B and HuluMed-4B backbones under the same mask-token interface. Round-trip filtering. Before Stage 2, we apply dataset-level round-trip filtering to reduce noisy mask supervision. Specifically, each ground-truth mask is first encoded by the Stage-1 tokenizer and then decoded back into a dense mask. We compute reconstruction quality with Dice and IoU and derive a dataset-level score from their mean. In our filtering script, low-quality datasets are downsampled according to score-based keep ratios, while high-quality datasets are kept in full. This filtered data is used consistently for both text-guided segmentation and region-grounded understanding. C.2 Evaluation Details Medical VQA. We evaluate image-level reasoning on SLAKE, PathVQA, and VQA-RAD. Following the evaluation scripts used in our codebase, each closed question is rewritten with a short answer instruction (“Answer the question using a single word or phrase”), while open questions request concise responses. For the main paper, we report closed-question accuracy. Our evaluation scripts additionally support exact match, token recall, and optional external LLM judging for open-ended answers. Text-Guided Segmentation. We evaluate text-guided segmentation on the 80-dataset medical benchmark family. At inference time, the model generates textual outputs containing mask tokens. These tokens are parsed and decoded back into dense binary masks with the frozen Stage-1 tokenizer and the original medical image. We then compute Dice and IoU. In the main table, we report macro Dice; in the appendix and internal analysis, we additionally report weighted and micro aggregation. Region-Grounded Understanding. We evaluate region-grounded understanding on the same 80-dataset benchmark family under two protocols. In v1_masks, the target region is shown visually through an overlay mask; in v2_tokens, the same region is represented by a discrete mask-token span and inserted into the question template. For the main paper, we use exact match as the primary metric and additionally report token recall in the detailed appendix analysis. Sharded inference. For the large 80-dataset evaluations, our codebase performs sharded inference across multiple GPUs and merges shard-level predictions into a final summary file. This is the default setting used by our task-2 and task-3 evaluation scripts. C.3 Hyperparameters Table 6 summarizes the key hyperparameters used in the current implementation of MedUP. We implement MedUP on top of the Qwen3-VL-4B and HuluMed-4B backbones, referred to as MedUP-Q and MedUP-H, respectively. All Stage-2 fine-tuning experiments are conducted using 8 NVIDIA H20 GPUs. Component Setting Stage-1 tokenizer VQ-SAM2 with codebook size 256256, depth 22, non-shared codebooks Stage-1 input size 1024×10241024× 1024 Stage-1 optimizer AdamW, learning rate 4×10−54× 10^-5, weight decay 0.050.05 Stage-1 training batch size 88/GPU, 1 epoch, warmup ratio 0.050.05 Stage-1 checkpoint interval every 50005000 iterations Stage-2 backbones MedUP-Q (Qwen-based), MedUP-H (Hulu-based) Stage-2 optimizer AdamW, learning rate 2×10−52× 10^-5, weight decay 0.050.05 Stage-2 training batch size 11/GPU, gradient accumulation 88, 1 epoch Stage-2 schedule linear warmup (0.050.05) + cosine decay Stage-2 precision BF16 mixed precision Stage-2 adaptation LoRA with rank 128128, alpha 256256, dropout 0.050.05 Stage-2 vision encoder frozen Stage-2 max length 81928192 (Qwen3-VL-4B) / 1638416384 (HuluMed-4B) Stage-2 checkpoint interval every 10001000 iterations Table 6: Key implementation hyperparameters used by MedUP. The table reflects the training configurations used in our local codebase for the reported Qwen-based and Hulu-based models. C.4 Prompt and Data Format All tasks are converted into a standard conversational format so that training remains pure autoregressive next-token prediction. For example, a text-guided segmentation sample is represented as an image-conditioned instruction followed by a short JSON-like answer containing the mask-token span and its label. A typical target format is: ["mask_2d": "<|mt_start|><|mt_0230|><|mt_0345|><|mt_end|>", "label": "liver"] For region-grounded understanding, the mask-token span is inserted directly into the question, e.g., “Which category does region <|mt_start|>...<|mt_end|> belong to in this medical image?” In our current benchmark and training templates, these prompts are typically instantiated as short category- or label-oriented region understanding questions. For Medical VQA, we use ordinary image-question-answer conversations without mask tokens. This unified textual interface is important for our implementation: the language model only needs to predict short symbolic spans, while dense pixel reconstruction is delegated to the frozen Stage-1 tokenizer decoder. As a result, MedUP can support segmentation generation and region-grounded understanding without introducing an additional trainable dense decoder in Stage 2. Appendix D Case Study of Seg-CoT We provide qualitative examples from the Seg-CoT portion of UniMed-Train to illustrate how reasoning and mask prediction are interleaved under the UniMedTok interface. Each card shows the question, the original image, the pseudo segmentation overlay, and the corresponding reasoning trace with embedded mask-token outputs. These examples fall into two patterns. The first pattern is single-mask Seg-CoT, where the model reasons about the target anatomy or semantic category and emits one final mask-token span for the identified region. The second pattern is multi-mask Seg-CoT, where the reasoning trajectory is decomposed into multiple localization steps, and each step produces its own mask-token span. This latter form is especially useful for bilateral structures, hierarchical anatomy, and context-to-target localization, where intermediate grounded regions help the model progressively refine its interpretation. CT Prompt: What is the anatomical location of the area shown in the image? A. Shoulder and upper back region B. Abdominal region C. Chest region. D. Hip region Original Image Segmentation Overlay Chain-of-Thought Reasoning The image shows a cross-sectional view of the thoracic cavity, with visible lung parenchyma and mediastinal structures. The highlighted region corresponds to the chest area, ruling out the shoulder, abdominal, and hip regions based on anatomical landmarks and tissue types. ["mask_2d": "<|mt_start|> <|mt_0180|><|mt_0087|><|mt_end|>", "label": "Chest region."] Final answer: (C) X-Ray Prompt: What content appears in this image? A. Kidney B. Brain C. Lung D. Bone Original Image Segmentation Overlay Chain-of-Thought Reasoning The image shows a large, dark area with branching structures that resemble airways and blood vessels. ["mask_2d": "<|mt_start|> <|mt_0073|><|mt_0089|><|mt_end|>", "label": "Lung"] The highlighted region corresponds to the lung tissue, ruling out the kidney, brain, and bone based on their distinct anatomical features and density. Final answer: (C) Ultrasound Prompt: What structure is depicted in this ultrasound image? A. Femoral Vein B. Adrenal Gland C. portal vein D. Liver parenchyma Original Image Segmentation Overlay Chain-of-Thought Reasoning The image shows a tubular structure with an anechoic center and echogenic walls, which is characteristic of a blood vessel. ["mask_2d": "<|mt_start|> <|mt_0101|><|mt_0168|><|mt_end|>", "label": "portal vein"] The location and appearance are consistent with the portal vein, ruling out the femoral vein, adrenal gland, and liver parenchyma based on size, shape, and acoustic profile. Final answer: (C) Dermoscopy Prompt: What anatomical part is illustrated in this picture? A. Bones B. Skin C. Reproductive system D. Respiratory system Original Image Segmentation Overlay Chain-of-Thought Reasoning The image shows a close-up view of a textured surface with visible pores and hair follicles, which are characteristic features of the skin. ["mask_2d": "<|mt_start|> <|mt_0003|><|mt_0196|><|mt_end|>", "label": "Skin"] The highlighted region corresponds to the skin, ruling out bones, reproductive system, and respiratory system based on the absence of bone structure, internal organs, or airways. Final answer: (B) CT Prompt: Step through the two lung fields visible in this thoracic CT slice and emit one segmentation mask per side. Original Image Segmentation Overlay [Multi-mask Overlay] Chain-of-Thought Reasoning The image is an axial CT slice through the lower thorax, with bilateral pulmonary fields flanking the central mediastinum. Step one: I segment the right lung field, identified by its low-density air-filled parenchyma lateral to the heart. ["mask_2d": "<|mt_start|> <|mt_0173|><|mt_0034|><|mt_end|>", "label": "right lung"] Step two: I then segment the contralateral left lung, which exhibits a similar reticular pulmonary texture and rib-cage boundary. ["mask_2d": "<|mt_start|> <|mt_0428|><|mt_0091|><|mt_end|>", "label": "left lung"] The two masks together delineate the complete bilateral pulmonary anatomy at this slice level. X-Ray Prompt: Identify each lung field separately on this chest radiograph and return one segmentation mask per side. Original Image Segmentation Overlay [Multi-mask Overlay] Chain-of-Thought Reasoning The frontal chest X-ray reveals bilateral lung fields with the central mediastinal silhouette between them. Step one: I begin with the right lung, segmenting its radiolucent area within the rib cage. ["mask_2d": "<|mt_start|> <|mt_0265|><|mt_0142|><|mt_end|>", "label": "right lung"] Step two: I then segment the contralateral left lung, bounded laterally by the chest wall and medially by the cardiac silhouette. ["mask_2d": "<|mt_start|> <|mt_0073|><|mt_0089|><|mt_end|>", "label": "left lung"] Both pulmonary fields are now isolated as two distinct masks suitable for downstream side-specific assessment. Ultrasound Prompt: Provide two masks for this ultrasound image: the surrounding hepatic parenchyma and the portal vein within it. Original Image Segmentation Overlay [Multi-mask Overlay] Chain-of-Thought Reasoning The ultrasound image displays the right upper quadrant centered on hepatic anatomy. Step one: I first segment the surrounding hepatic parenchyma, characterized by its homogeneous mid-level echotexture. ["mask_2d": "<|mt_start|> <|mt_0314|><|mt_0207|><|mt_end|>", "label": "liver parenchyma"] Step two: within this organ context, I refine the segmentation to the portal vein, recognized by its anechoic lumen and echogenic walls. ["mask_2d": "<|mt_start|> <|mt_0101|><|mt_0168|><|mt_end|>", "label": "portal vein"] The hierarchical masks separate the organ background from the target vessel, enabling vessel-in-organ reasoning downstream. Dermoscopy Prompt: Return two masks for this dermoscopy image: the surrounding skin region and the central pigmented lesion. Original Image Segmentation Overlay [Multi-mask Overlay] Chain-of-Thought Reasoning The dermoscopy image captures a pigmented region on the skin surface. Step one: I segment the surrounding healthy skin area as anatomical context. ["mask_2d": "<|mt_start|> <|mt_0003|><|mt_0196|><|mt_end|>", "label": "skin"] Step two: I then refine the segmentation to isolate the pigmented lesion located at the center, distinguished by its darker tone and irregular border. ["mask_2d": "<|mt_start|> <|mt_0152|><|mt_0086|><|mt_end|>", "label": "skin lesion"] The two masks separate the anatomical context from the lesion target, supporting downstream pathology-versus-context reasoning. Appendix E UniMed-Bench Detailed Analysis This appendix presents benchmark subset examples for four dataset-level metrics. The subset is constrained so that each listed dataset satisfies a strict criterion: in that row, either MedUP-H or MedUP-Q achieves the best score (including ties). Within each row, the best and second-best method scores are marked by bold and underline. For readability, we show up to 20 datasets per metric, prioritized by the higher value between MedUP-H and MedUP-Q. The resulting subsets emphasize datasets where our unified token-based models are competitive at the top level, while still exposing cross-method differences on the same benchmark slices. In text-guided segmentation, this highlights both near-saturated datasets and structurally harder datasets where method gaps remain visible. In region-grounded understanding, it also shows where multiple methods hit ceiling-level scores versus where token-level naming and exact matching are still challenging. Table 7: Benchmark subset results for mean Dice on text-guided segmentation. Modality Dataset Samples Segmentors / VLM Baselines MedSAM1 MedSAM2 MedSAM3 BiomedParse UniBiomed LISA++ SAM4MLLM MMedAgent MedUP-H MedUP-Q CT finding-lungs-in-ct-data_2d 54 0.7481 0.0951 0.7205 0.2666 0.2839 0.6701 0.4281 0.0519 0.9576 0.9582 CT VESSEL2012 2,082 0.8034 0.1436 0.0031 0.0039 0.0176 0.4534 0.4428 0.1755 0.9273 0.9396 CT MSD_Spleen 146 0.9249 0.7612 0.8838 0.3994 0.7433 0.0675 0.0588 0.0809 0.9374 0.9389 CT PleThora 3,313 0.8004 0.7373 0.1055 0.0098 0.0025 0.3188 0.3486 0.0306 0.9179 0.9113 CT Continuous_Registration_task1 139 0.6704 0.7834 0.8009 0.2900 0.0286 0.2508 0.2507 0.2873 0.9013 0.9042 MR CMRxMotions 313 0.8597 0.7876 0.5084 0.2492 0.3001 0.0909 0.0452 0.1494 0.8623 0.8493 CT AbdomenCT1K 37,571 0.8325 0.6491 0.6379 0.2202 0.4402 0.1230 0.0890 0.3976 0.8561 0.8394 CT FLARE21 8,211 0.8054 0.6077 0.6191 0.3602 0.5453 0.1232 0.1050 0.3769 0.8450 0.8386 CT KiTS 3,313 0.7361 0.5738 0.6170 0.0831 0.3284 0.0844 0.0599 0.3690 0.8414 0.8164 MR mnms2 1,919 0.7908 0.7701 0.5202 0.3488 0.3094 0.0985 0.0467 0.1940 0.8407 0.8151 CT Colorectal_Liver_Metastases 2,060 0.7650 0.4567 0.5880 0.1410 0.6192 0.1835 0.1399 0.6354 0.7929 0.7682 CT KiTS2021 5,622 0.7048 0.5266 0.6219 0.0800 0.3275 0.0693 0.0442 0.3762 0.7863 0.7661 CT KiTS2023 5,816 0.6845 0.5044 0.5896 0.0826 0.3307 0.0784 0.0542 0.4156 0.7747 0.7434 CT SegRap2023_cta 6,597 0.6261 0.4815 0.2165 0.0756 0.0678 0.0763 0.0688 0.0558 0.7180 0.7020 CT SegRap2023_ct 6,597 0.6261 0.4815 0.2165 0.0747 0.0678 0.0763 0.0688 0.0558 0.7177 0.7037 MR BraTS2021 15,929 0.5836 0.5463 0.4015 0.0680 0.3848 0.0964 0.0779 0.2709 0.5881 0.5294 X-ray sz_cxr 114 0.9602 0.9477 0.9586 0.0122 0.9331 0.7557 0.4345 0.6581 0.9485 0.9436 Fundus drishti_gs_od 51 0.9613 0.9710 0.9591 0.0000 0.6814 0.0641 0.4387 0.8767 0.9315 0.9351 Endoscopy kvasircapsule_seg 11 0.9533 0.9527 0.9309 0.0000 0.5627 0.7868 0.7506 0.6338 0.9154 0.9204 Fundus ichallenge_adam_task2 61 0.9559 0.9645 0.9367 0.0000 0.8011 0.0285 0.0601 0.1823 0.9190 0.9203 Table 8: Benchmark subset results for mean IoU on text-guided segmentation. Modality Dataset Samples Segmentors / VLM Baselines MedSAM1 MedSAM2 MedSAM3 BiomedParse UniBiomed LISA++ SAM4MLLM MMedAgent MedUP-H MedUP-Q CT finding-lungs-in-ct-data_2d 54 0.6023 0.0580 0.6328 0.1663 0.1819 0.5136 0.2831 0.0362 0.9293 0.9301 CT VESSEL2012 2,082 0.7223 0.1326 0.0017 0.0022 0.0105 0.3845 0.3205 0.1387 0.8945 0.9101 CT MSD_Spleen 146 0.8635 0.7021 0.8198 0.3125 0.6694 0.0351 0.0305 0.0746 0.8919 0.8933 CT PleThora 3,313 0.7358 0.6797 0.0822 0.0070 0.0013 0.2041 0.2286 0.0174 0.8665 0.8608 CT Continuous_Registration_task1 139 0.5988 0.7124 0.6882 0.2030 0.0162 0.1503 0.1502 0.2208 0.8396 0.8419 CT AbdomenCT1K 37,571 0.7445 0.5830 0.5410 0.1817 0.3742 0.0725 0.0497 0.3456 0.7983 0.7816 CT KiTS 3,313 0.6205 0.4952 0.5224 0.0611 0.2530 0.0462 0.0317 0.2987 0.7943 0.7683 CT FLARE21 8,211 0.7110 0.5424 0.5254 0.2979 0.4621 0.0716 0.0598 0.3240 0.7923 0.7842 MR CMRxMotions 313 0.7720 0.6947 0.3766 0.2002 0.2378 0.0513 0.0236 0.1003 0.7846 0.7714 MR mnms2 1,919 0.6980 0.6836 0.3804 0.2681 0.2267 0.0573 0.0242 0.1421 0.7593 0.7309 CT Colorectal_Liver_Metastases 2,060 0.6626 0.4034 0.4675 0.1142 0.5365 0.1083 0.0811 0.5711 0.7410 0.7169 CT KiTS2021 5,622 0.5966 0.4548 0.5343 0.0589 0.2583 0.0377 0.0230 0.3041 0.7375 0.7177 CT KiTS2023 5,816 0.5776 0.4348 0.4985 0.0606 0.2627 0.0433 0.0285 0.3368 0.7232 0.6946 CT MMWHS_CT 1,591 0.6565 0.5065 0.2007 0.0051 0.1731 0.0246 0.0388 0.2069 0.6573 0.6342 CT SegRap2023_cta 6,597 0.5127 0.4026 0.1760 0.0547 0.0472 0.0487 0.0451 0.0426 0.6219 0.6053 CT SegRap2023_ct 6,597 0.5127 0.4026 0.1760 0.0541 0.0472 0.0487 0.0451 0.0426 0.6216 0.6068 CT WORD 6,854 0.5690 0.4856 0.2812 0.1577 0.3731 0.0471 0.0490 0.2820 0.5884 0.5600 MR BraTS2021 15,929 0.4614 0.4366 0.3192 0.0447 0.2951 0.0553 0.0442 0.2010 0.4864 0.4261 MR BraTS2023_GLI 16,388 0.4588 0.4225 0.3981 0.1512 0.3414 0.0540 0.0451 0.1620 0.4674 0.4086 MR BraTS2019 5,035 0.4351 0.3689 0.2655 0.0483 0.2509 0.0609 0.0446 0.1541 0.4396 0.3688 Table 9: Benchmark subset results for weighted EM on region-grounded understanding. Modality Dataset Samples / VLM Baselines VLM Baselines UniBiomed LISA++ SAM4MLLM MMedAgent LLaVA-Med MedGemma MedUP-H MedUP-Q Ultrasound CETUS2014 3,262 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 CT Continuous_Registration_task1 139 0.0000 0.0072 0.0647 0.0000 0.0144 0.7842 1.0000 1.0000 MR Heart_Seg_MRI 51 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 MR braimMRI 258 0.0000 0.0000 0.0000 0.0000 0.0000 0.9806 1.0000 1.0000 Endoscopy cvc_clinicdb 123 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Fundus drishti_gs_od 51 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy endovis15 123 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy hyper-kvasir-segmented-images 200 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Fundus ichallenge_adam_task2 61 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Dermoscopy isic2018_task1 90 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy kvasir_seg 40 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy kvasir_seg_aliyun 200 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy kvasircapsule_seg 11 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 X-ray sz_cxr 114 0.0000 0.1754 0.9561 0.0000 0.3158 1.0000 1.0000 1.0000 Ultrasound TDSC-ABUS2023 254 0.0000 0.0000 0.0000 0.0000 0.0000 0.9252 1.0000 0.9961 MR Prostate_MRI_Segmentation_Dataset 175 0.0000 0.0000 0.1371 0.0000 0.0743 0.9943 1.0000 0.9943 MR SPPIN2023 923 0.0000 0.0000 0.0000 0.0000 0.0000 0.4670 1.0000 0.9935 CT MSD_Spleen 146 0.0000 0.0000 0.0000 0.0000 0.0000 0.8973 1.0000 0.9932 MR MSD_Heart 133 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 0.9850 CT finding-lungs-in-ct-data_2d 54 0.0000 0.1667 0.8148 0.0000 0.0556 0.9815 1.0000 0.9815 Table 10: Benchmark subset results for weighted token recall on region-grounded understanding. Modality Dataset Samples / VLM Baselines VLM Baselines UniBiomed LISA++ SAM4MLLM MMedAgent LLaVA-Med MedGemma MedUP-H MedUP-Q Ultrasound CETUS2014 3,262 0.0000 0.0000 0.0000 0.0000 0.1257 1.0000 1.0000 1.0000 CT Continuous_Registration_task1 139 0.0432 0.0072 0.0647 0.0000 0.0144 0.7842 1.0000 1.0000 MR Heart_Seg_MRI 51 0.0000 0.0000 0.0000 0.0000 0.1961 1.0000 1.0000 1.0000 MR braimMRI 258 0.0000 0.0000 0.0000 0.0000 0.0000 0.9806 1.0000 1.0000 Endoscopy cvc_clinicdb 123 0.2602 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Fundus drishti_gs_od 51 0.0000 0.0000 0.0000 0.0000 0.0784 1.0000 1.0000 1.0000 Endoscopy endovis15 123 0.2276 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy hyper-kvasir-segmented-images 200 0.4450 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Fundus ichallenge_adam_task2 61 0.0000 0.0000 0.0000 0.0000 0.1148 1.0000 1.0000 1.0000 Dermoscopy isic2018_task1 90 0.0000 0.0000 0.0000 0.0000 0.2111 1.0000 1.0000 1.0000 Endoscopy kvasir_seg 40 0.5250 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy kvasir_seg_aliyun 200 0.3800 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 Endoscopy kvasircapsule_seg 11 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 X-ray sz_cxr 114 0.1404 0.1754 0.9561 0.0000 0.4386 1.0000 1.0000 1.0000 Ultrasound TDSC-ABUS2023 254 0.0000 0.0000 0.0000 0.0000 0.1063 0.9252 1.0000 0.9961 MR Prostate_MRI_Segmentation_Dataset 175 0.0000 0.0000 0.1600 0.0000 0.0971 0.9943 1.0000 0.9943 MR SPPIN2023 923 0.0000 0.0000 0.0000 0.0000 0.0000 0.4670 1.0000 0.9935 CT MSD_Spleen 146 0.0205 0.0000 0.0000 0.0000 0.0000 0.8973 1.0000 0.9932 MR MSD_Heart 133 0.0000 0.0000 0.0000 0.0000 0.0376 1.0000 1.0000 0.9850 CT finding-lungs-in-ct-data_2d 54 0.0741 0.1667 0.8148 0.0000 0.2037 0.9815 1.0000 0.9815