Paper deep dive

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Quanhao Zhu, Bo Xu, Rui Lin, Chenyuan Wang, Yu Shao, Boling Zhu, Jiuyan Sun, Liang Zhao, Hongfei Lin, Feng Xia

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 111

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%

Last extracted: 9/28/2026, 3:53:31 AM

Summary

The paper introduces UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound images using Large Vision-Language Models (VLMs). It comprises 40 public datasets across 13 anatomical categories, featuring three tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation. The authors also propose UltraG-Agent, a framework combining an open-source MLLM with the ultrasound-specific segmentation model UltraSAM3 to improve both semantic prediction and pixel-level visual grounding.

Entities (10)

UltraG-Agent · modelframework · 99%UltraG-Bench · benchmark · 99%Ultrasound · medical-modality · 99%Evidence-grounded Report Generation · task · 98%Evidence-grounded VQA · task · 98%Instruction-guided Segmentation · task · 98%UltraSAM3 · model · 97%13 Anatomical Categories · concept · 95%SAM3 · model · 95%GPT-5.5 · model · 90%

Relation Signals (8)

UltraG-Bench → containstask → Evidence-grounded VQA

confidence 99% · UltraG-Bench ... comprises three progressive tasks: ... evidence-grounded VQA

UltraG-Bench → containstask → Evidence-grounded Report Generation

confidence 99% · UltraG-Bench ... comprises three progressive tasks: ... evidence-grounded report generation

UltraG-Bench → containstask → Instruction-guided Segmentation

confidence 99% · UltraG-Bench ... comprises three progressive tasks: instruction-guided segmentation

UltraG-Bench → coversmodality → Ultrasound

confidence 99% · UltraG-Bench ... for evaluating pixel-level evidence grounding in ultrasound.

UltraG-Agent → usesmodel → UltraSAM3

confidence 98% · UltraG-Agent ... combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3.

UltraG-Bench → coversanatomy → 13 Anatomical Categories

confidence 95% · UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories

UltraG-Agent → usesmodel → MLLM

confidence 95% · UltraG-Agent ... uses an open-source MLLM for instruction understanding

UltraSAM3 → isbasedon → SAM3

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predictions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding. Our dataset and code are available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

107,978 characters extracted from source content.

Expand or collapse full text

ULTRAG-BENCH: A MULTI-TASK BENCHMARK FOR ASSESSING LARGE VISION-LANGUAGE MODELS ON PIXEL-LEVEL EVIDENCE GROUNDING IN ULTRASOUND Quanhao Zhu, Bo Xu, Rui Lin, Chenyuan Wang, Yu Shao, Boling Zhu, Jiuyan Sun, Liang Zhao, Hongfei Lin Dalian University of Technology zhuqh19@gmail.com boxu@dlut.edu.cn lr983088162@mail.dlut.edu.cn chenyuanwang@mail.dlut.edu.cn python@mail.dlut.edu.cn SinsapZ@outlook.com 2093145195@mail.dlut.edu.cn liangzhao@dlut.edu.cn hflin@dlut.edu.cn Feng Xia RMIT University f.xia@ieee.org ABSTRACT Ultrasound is one of the most widely used medical imaging modalities, and recent large vision- language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predic- tions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding. Our dataset and code are available at https://github.com/zhuqh19/UltraG-Bench. 1 Introduction Ultrasound has become one of the most widely used medical imaging modalities in clinical practice owing to its lack of ionizing radiation, real-time imaging capability, and relatively low cost. Compared with CT and MRI, ultrasound devices are more portable and less expensive, making them highly accessible across diverse clinical settings [1]. Accurate recognition, localization, and interpretation of anatomical categories and lesions in ultrasound images are fundamental to clinical diagnosis and decision-making [2]. Nowadays, supervised segmentation methods and general foundation models have continuously improved pixel-level localization of organs and lesions [3,4]. On the other hand, vision-language models (VLMs) and multimodal large language models (MLLMs) are increasingly being applied to ultrasound image understanding, including tasks such as arXiv:2609.30928v1 [cs.CV] 25 Sep 2026 Running Title for Header visual question answering (VQA) and report generation [2]. However, a substantial gap remains between pixel-level visual localization and high-level semantic understanding in current models. Conventional segmentation models can provide precise spatial boundaries but generally lack open-ended language understanding and semantic generation capabilities, whereas VLMs and MLLMs can answer questions or generate image descriptions but often fail to provide pixel-level visual evidence that is aligned with their semantic predictions [5,6,7,8]. This gap is particularly critical in medical applications, where clinical reasoning depends not only on “what the model predicts”, but also on “which regions in the image support that prediction.” However, this gap has not yet been systematically evaluated by existing benchmarks, leaving the extent of the disconnect between semantic understanding and pixel-level visual grounding in current models poorly understood. Thyroid Carotid artery Breast Liver Cardiac Nerve Fetal Prostate Ovarian Kidney Muscle Abdomen Lung Organ: Abdomen Instruction: Segment the kidney in this image. Answer: This is the [seg]. Instruction-guided Segmentation Organ: Breast Question: Is there a visible breast lesion? Answer: Yes, this is the breast tumor. Evidence-grounded VQA Organ: Kidney Instruction: Generate an ultrasound report. Answer: The kidney is located at the left of ...... Evidence-grounded Report Generation 13 Anatomical Categories 138832 Ultrasound images 181952 Segmentation masks 331125 Seg Instructions 666779 Grounded VQA pairsGrounded Reports 138832 Evidence-grounded VQA Instruction-guided Segmentation Figure 1: Overview of UltraG-Bench, including 13 anatomical categories, benchmark statistics, and representative examples of the three tasks: Instruction-guided Segmentation, Evidence-grounded VQA, and Evidence-grounded Report Generation. Existing benchmarks often evaluate semantic understanding and pixel-level visual localization in isolation [9,10,11, 12,13]: segmentation benchmarks primarily measure the overlap between predicted and ground-truth masks using metrics such as IoU and Dice, whereas VQA and report-generation benchmarks mainly assess the correctness of the generated answers or text. As a result, a model may produce the correct answer while relying on the wrong visual region, or generate fluent and semantically plausible descriptions without providing consistent pixel-level evidence for its claims. Recently, visual evidence-grounded datasets such as GEMeX [14] and MIMIC-ILS [15] have emerged, but such evaluation remains lacking in ultrasound, especially at the pixel level. To address these challenges, we introduce UltraG-Bench as shown in Figure 1, a large-scale multi-task benchmark for Pixel-level Evidence Grounding in ultrasound. UltraG-Bench annotates 40 public ultrasound segmentation datasets spanning 13 anatomical categories, including the abdomen, breast, heart, carotid artery, fetus, kidney, liver, lung, muscle, nerve, ovary, prostate, and thyroid, comprising 138,832 ultrasound images and 181,952 segmentation masks. Based on these pixel-level annotations, we construct three tasks from visual perception to semantic understanding: instruction-guided segmentation, which evaluates whether a model can localize the correct target from a natural-language instruction; evidence-grounded VQA, which assesses answer correctness and the accuracy of the corresponding visual evidence; evidence-grounded report generation, which examines whether a model can consistently associate different semantic statements with the appropriate pixel-level regions during report generation. During dataset construction, we explicitly preserve the correspondence between textual statements and the original segmentation masks, while restricting diagnostic, pathological, and treatment-related claims that are unsupported by the available pixel-level annotations, 2 Running Title for Header thereby ensuring that each grounded statement is backed by traceable visual evidence. Based on this benchmark, we systematically evaluate 15 state-of-the-art models, including closed-source general-purpose MLLMs, open-source general-purpose MLLMs, medical MLLMs, and pixel-level grounded medical MLLMs. Furthermore, we observe a clear complementarity between high-level semantic understanding and fine-grained pixel localization in existing models: MLLMs excel at interpreting complex natural-language instructions and generating semantic responses, whereas specialized ultrasound segmentation models provide stronger pixel-level structural localization. This motivates a further question: Can more reliable Pixel-level Evidence Grounding be achieved through collaboration among pretrained models? To explore this, we construct UltraG-Agent, an ultrasound grounding agent that uses an open-source MLLM for instruction understanding, task planning, and response generation, while employing UltraSAM3 [16], an ultrasound concept-driven segmentation model, as the visual module for target segmentation. Experiments show that UltraG-Agent achieves state-of-the-art performance on UltraG-Bench. The main contributions are as follows: •UltraG-Bench: We annotate 40 public ultrasound segmentation datasets spanning 13 anatomical categories, comprising 138,832 ultrasound images and 181,952 segmentation masks. We construct three tasks, including instruction-guided segmentation, mask-grounded VQA, and mask-grounded report generation, to evaluate capabilities of existing models. •Comprehensive evaluation: Under a unified evaluation framework, we benchmark 16 state-of-the-art models, including closed-source and open-source general MLLMs, medical MLLMs, pixel-level grounded medical MLLMs, and prompt-driven segmentation models. •UltraG-Agent: We propose an ultrasound agent framework that combines the instruction understanding and task-planning capabilities of an open-source MLLM with the specialized ultrasound segmentation capability of UltraSAM3. UltraG-Agent achieves state-of-the-art performance among all evaluated models. 2 Related Work Medical Evidence-Grounded Benchmarks.Early medical VQA datasets, such as VQA-RAD, SLAKE, PathVQA, and OmniMedVQA, pair medical images with questions and answers [9,10,11,12]. They provide important bench- marks for medical visual understanding and question answering. As interpretability and visual evidence become more important, later datasets begin to link text with local image regions. MS-CXR, Chest ImaGenome, and PadChest-GR provide annotations that connect radiology findings, anatomical structures, and imaging attributes with corresponding bounding boxes in chest X-rays [17,18,19]. These annotations enable image–text alignment to be evaluated at the region level. For VQA, GEMeX [14] introduces textual explanations and visual regions into Med-VQA. GEMeX- RMCoT [20] further links intermediate reasoning steps to relevant anatomical regions. MIMIC-ILS [15] extends referring image segmentation to chest X-rays, providing finer pixel-level evidence. However, existing benchmarks focus on radiological modalities, while ultrasound remains lacking. Moreover, most datasets focus on a single task. In contrast, UltraG-Bench builds a multi-task Pixel-level Evidence Grounding benchmark from 40 public ultrasound segmentation datasets. We have included a comparison with existing medical evidence-grounded benchmarks in Table 1. Table 1: Comparison of UltraG-Bench with representative medical evidence-grounded datasets. BenchmarkModalityScaleSpatial EvidenceRegion GroundingInstr.-Seg.Grounded VQAGrounded ReportCoverage MS-CXRCXR1,162 pairsBounding Box✓✗Chest Chest ImaGenomeCXR242,072 imagesBounding Box✓✗Chest PadChest-GRCXR4,555 studiesBounding Box✓✗✓Chest GEMeXCXR1.61M VQA pairsBounding Box✓✗✓✗Chest GEMeX-RMCoTCXR202K instancesBounding Box✓✗✓✗Chest MIMIC-ILSCXR1.1M pairsPixel Mask✓✗Chest UltraG-BenchUltrasound1.14M annotationsPixel Mask✓13 anatomies Large Vision-Language Models for Medical Evidence Grounding. Recently, both general-purpose and medical multimodal large language models have made rapid progress in medical image understanding. General MLLMs such as GPT [21], Claude [22], Qwen-VL [23], InternVL [24], and MiniCPM-V [25] show strong visual understanding, natural-language interaction, and open-ended reasoning. Medical MLLMs such as LLaVA-Med [26], MedGemma [27], and Lingshu [28] further improve medical image and clinical understanding through medical pretraining or instruction tuning. However, these models are often unclear whether their predictions are supported by the correct visual evidence. To provide fine-grained spatial outputs, another line of work endows MLLMs with segmentation capability. LISA [29] connects special tokens such as ‘[SEG]‘ to a segmentation decoder and enables reasoning segmentation from complex 3 Running Title for Header Data GatheringTask Definition Annotation Pipeline Public Data Gathering from the Internet License Check and Approval Clinical Experts Consultations Progressive Tasks Instruction-guided Segmentation 1 2 3 Evidence-grounded VQA Evidence-grounded Report Generation Segmentation Instruction --> Target Mask Question Answering with Supporting Evidence Mask Structured Report with Evidence Mask 1 2 3 4 Format Standardization Unified Image Size, Labels and Data Schema Evidence-Constrained Annotation Generate Annotation with Associated Evidence Mask Automated Validation lSchema Consistency lEvidence-Description Alignment lClinical Plausibility Checking Manual Review Manual Verfication and Quality Assurance UltraG-Bench Figure 2: Construction pipeline of the UltraG-Bench, including public data collection, progressive task design, evidence- constrained annotation, automated validation, and expert review. language instructions. Later models such as UniBiomed [30] extend this idea to medical images and produce pixel-level masks from medical text instructions. However, their true capability for ultrasound Pixel-level Evidence Grounding remains underexplored. More recently, prompt-driven segmentation models such as SAM3 [31] have emerged. Inspired by this paradigm, we develop UltraG-Agent to evaluate agent-based grounding on our benchmark. 3 UltraG-Bench Overview.We construct UltraG-Bench, a multi-task benchmark for evaluating Pixel-level Evidence Grounding in ultrasound images. UltraG-Bench is built upon 40 publicly available ultrasound image segmentation datasets, covering 13 anatomical categories, including the abdomen, breast, heart, carotid artery, fetus, kidney, liver, lung, muscle, nerve, ovary, prostate, and thyroid. Based on the pixel-level segmentation masks provided by the original datasets, we further construct three progressively challenging tasks that bridge visual perception and semantic understanding. The overall construction pipeline is illustrated in Figure 2. 3.1 Task Definition Instruction-guided Segmentation. Instruction-guided Segmentation evaluates whether a model can recognize and segment a target structure according to a natural-language instruction. Given an ultrasound image and a textual instruction describing the target object, such as “Segment the kidney in this image.”, the model is required to understand the target concept specified by the instruction and output the corresponding pixel-level segmentation region. Each instruction is explicitly associated with one or more target masks from the original dataset. Therefore, this task jointly evaluates natural-language instruction understanding and fine-grained spatial localization. Evidence-grounded VQA. Evidence-grounded VQA extends conventional medical visual question answering by additionally requiring pixel-level visual evidence that supports the generated answer. Given an ultrasound image and a related question, the model is required to generate a textual answer while simultaneously segmenting the region that supports the answer. Unlike conventional VQA benchmarks that evaluate only answer correctness, UltraG-Bench 4 Running Title for Header explicitly preserves the correspondence among the question, answer, and original segmentation mask. Consequently, even if a model produces a semantically correct answer, it is not considered to achieve reliable evidence grounding if the predicted visual evidence does not align with the ground-truth target region. This task therefore jointly evaluates semantic understanding and answer-evidence consistency. Evidence-grounded Report Generation. Evidence-grounded Report Generation further extends pixel-level visual grounding to long-form text generation. Given an ultrasound image and a report-generation instruction, the model is required to generate a structured ultrasound description and associate image-supported semantic statements with their corresponding pixel-level evidence regions. Specifically, statements in the generated report are linked to target masks, establishing fine-grained statement–evidence correspondence. This task evaluates not only the semantic quality of the generated text, but also whether the model can maintain consistency between textual descriptions and pixel-level visual evidence throughout a longer generation process. Diagnostic, pathological, treatment-related, or other clinical inferences that cannot be supported by the available pixel-level annotations are excluded from grounded statements, thereby avoiding supervision based on untraceable visual evidence. 3.2 Dataset Construction 3.2.1 Dataset Collection and Preprocessing We collect publicly available ultrasound image segmentation datasets. Before incorporating them into UltraG-Bench, we verify their public-use licenses and organize their images, segmentation annotations, and target-category information. The resulting collection covers 13 anatomical categories. Detailed descriptions of our benchmark, including data sources, target categories, and statistics, are provided in the Appendix A.1. Since the source datasets differ substantially in image organization, mask representation, category naming, and annotation format, we first perform unified preprocessing. Specifically, segmentation annotations from different sources are converted into a COCO-style annotation format. Each image is assigned a unique identifier, and every segmentation mask is explicitly associated with its corresponding target-category name. The standardized representation contains unified image information, target categories, segmentation regions, and annotation identifiers, providing a consistent pixel-level evidence representation for the three downstream tasks. After standardization, UltraG-Bench contains 138,832 ultrasound images and 181,952 pixel-level segmentation masks. These standardized target-mask correspondences serve as the foundation for subsequent annotation. 3.2.2 Evidence-constrained Annotation Based on the standardized segmentation data, we further construct annotations for the three tasks. Instead of allowing an MLLM to freely generate questions or reports, we adopt an Evidence-constrained Annotation strategy. For each image, the existing target-category names and their corresponding segmentation masks are first extracted and used as the factual basis available during annotation generation. The detailed annotation prompts are provided in the Appendix A.3. We use GPT-5.5 [21] to generate annotations. For each sample, the model is only allowed to construct segmentation instructions, VQA pairs, and grounded reports based on the provided target names and pixel-level evidence. For Instruction-guided Segmentation, we generate target segmentation instructions with diverse expressions while preserving their correspondence with the target masks. For example, “Segment the liver.” is directly grounded by the liver mask. For Evidence-grounded VQA, we construct questions and answers related to visible targets, spatial locations, object counts, and other information supported by the available annotations. For example, given the question “Which abdominal structure occupies the upper-central region of the image?”, the answer is “Liver.”, and the corresponding liver mask is used as supporting visual evidence. For Evidence-grounded Report Generation, we generate structured ultrasound descriptions based on the target categories and their pixel-level masks, focusing primarily on four types of information: presence, category, location, and size. For example, the model may generate “A visible kidney target is present in the middle-right region, with an approximate size of 77×45 pixels.” The key information contained in the report is therefore directly supported by the ground-truth masks. The entire annotation process follows a strict evidence constraint. The annotation model is not allowed to introduce diagnoses, pathological attributes, treatment recommendations, or other medical conclusions that cannot be supported by the available pixel-level annotations. In other words, segmentation masks are primarily used to support visual facts such as target presence, spatial location, and extent, rather than serving as evidence for additional clinical attributes. In this way, the annotations are not generated independently of the segmentation labels, but remain traceably connected to 5 Running Title for Header UltraG-Agent Task Selection Input Ultrasound Image User Query Determine the task MLLM Instruction-guided Segmentation Evidence-grounded VQA Evidence-grounded Report Generation Target Selection Select the appropriate segmentation target Segmentatin Target lThyroid nodule lBreast tumor lLiver tumor lLeft Ventricle lProstate ...... Prompt-driven Seg. Segmentation Targe Prompt: Kidney Prompt-driven Segmentation model (SAM3-based) Generated Target Mask Output Visualized Evidence Textual Answer OR Structured Report MLLM Figure 3: Overview of UltraG-Agent. the original pixel-level annotations, ensuring that every grounded prediction in UltraG-Bench can be traced back to explicit visual evidence. 3.2.3 Dataset Validation To ensure annotation reliability, we adopt a two-stage validation pipeline consisting of automated validation and expert review. Automated checks assess data completeness, formatting consistency, image–annotation correspondence, and task-specific alignment between textual annotations and pixel-level evidence. Samples that fail these checks are removed or regenerated. For manual review, two reviewers with training in ultrasound imaging independently assess 10% samples from each anatomical structure, with representative coverage across categories and annotation types. They verify the consistency between annotations and their corresponding pixel-level evidence across all three tasks. Disagreements are jointly reviewed to reach consensus. 3.3 Statistics UltraG-Bench contains 331,125 instruction-guided segmentation instructions, 666,779 evidence-grounded VQA pairs, and 138,832 evidence-grounded reports, forming a multi-level evaluation framework that ranges from target segmentation and visual question answering to report generation. Detailed statistics are provided in Appendix A.1. 4 UltraG-Agent SAM3 [31] is a prompt-driven segmentation model that supports open-vocabulary text prompts. It can perform segmentation according to text prompts, and further employs an MLLM-based agent to translate complex user instructions into executable segmentation targets. Inspired by this framework, we further propose UltraG-Agent, which combines the semantic understanding capability of a multimodal large language model with the fine-grained pixel-level localization capability of a prompt-driven ultrasound segmentation model to achieve more reliable Pixel-level Evidence Grounding. UltraG-Agent does not require additional end-to-end training. Instead, it consists of a pretrained MLLM without task-specific fine-tuning and UltraSAM3 [16], a prompt-driven segmentation model adapted to the ultrasound domain. The overall framework is illustrated in Figure 3. Given an ultrasound image and a user instruction, the MLLM first performs task selection to determine whether the current request corresponds to Instruction-guided Segmentation, Evidence-grounded VQA, or Evidence-grounded Report Generation. It then selects the segmentation target required for the task based on the image, the user instruction, and a predefined set of ultrasound target categories, and converts the selected target into a concise text prompt for segmentation. Depending on the task, the MLLM can select one or multiple targets and generate a corresponding UltraSAM3 prompt for each target. 6 Running Title for Header After target selection, UltraSAM3 performs prompt-driven segmentation using the generated text prompts and produces pixel-level masks for the selected targets. For segmentation tasks, the predicted masks are directly returned as the output. For VQA and report generation, the system further extracts visual information from the predicted masks, including target presence, category, spatial location, and extent, and uses these mask-derived facts as reliable visual evidence to constrain the final text generation. UltraG-Agent ultimately outputs the visualized pixel-level evidence together with the corresponding textual answer or structured report, thereby combining the high-level semantic understanding capability of the MLLM with the fine-grained spatial localization capability of UltraSAM3. Detailed formulations and prompting strategies are provided in the Appendix A.4. 5 Experiment 5.1 Experimental Setting We evaluate UltraG-Bench on 14 representative MLLMs, including closed-source models, open-source general-purpose models, medical MLLMs, and pixel-level grounded medical MLLMs. To examine whether UltraG-Agent can extend prompt-driven segmentation models with VQA and report generation capabilities while preserving or improving their segmentation performance, we additionally evaluate BiomedParse [32] and UltraSAM3 [16] as prompt-driven segmentation baselines. We further evaluate four UltraG-Agent variants with different MLLMs and prompt-driven segmentation models. All models are evaluated under a unified protocol across the three tasks of UltraG-Bench. Detailed baseline introduction and the prompts used for evaluation are given in Appendix A.2 and Appendix A.5, respectively. 5.2 Evaluation Metric Instruction-guided Segmentation. We use Intersection over Union (IoU) and Dice coefficient (Dice) to evaluate the overlap between the predicted masks and the ground-truth masks. Evidence-grounded VQA.We report Answer Accuracy (Acc.) to evaluate the semantic correctness of the generated answers. Since a correct answer does not necessarily imply that the model relies on the correct visual evidence, we further introduce Grounded Accuracy (G-Acc.). A prediction is counted as correct by G-Acc. only when both the textual answer is correct and the predicted evidence region is sufficiently consistent with the corresponding ground-truth mask. Specifically, G-Acc. = 1 N N X i=1 I a i = ˆa i ∧ IoU( ˆ M i ,M i )≥ τ ,(1) where τ is the predefined grounding threshold 0.5. Evidence-grounded Report Generation.We use ROUGE-L to measure the lexical similarity between the generated and reference reports. To further evaluate whether the generated report correctly captures the key visual evidence, we introduce Semantic Accuracy (SemAcc). Specifically, we assess four semantic attributes: presence, which indicates whether the target is correctly identified as present or absent; category, which evaluates whether the target structure is correctly recognized; location, which measures whether the spatial position of the target is correctly described; and size, which evaluates whether the reported target extent is consistent with the ground-truth mask. These attributes correspond to binary indicatorspresence_ok,category_ok,location_ok, andsize_ok. The report-level semantic accuracy is defined as SemAcc = 1 4 (presence_ok + category_ok + location_ok + size_ok),(2) and the final score is averaged over all evaluated samples. Detailed definitions and evaluation of the four attributes are provided in the Appendix A.6 and Appendix A.7, respectively. 5.3 Evaluation Results 5.3.1 Main Results Table 2 summarizes the overall performance of different models on the three tasks of UltraG-Bench. General-purpose MLLMs show relatively strong semantic understanding but limited pixel-level grounding. For example, InternVL3.5-8B and GPT-5.4 achieve VQA accuracies of 0.3678 and 0.3631, while their G-Acc. scores remain only 0.0081 and 0.0609, respectively. Medical MLLMs exhibit a similar pattern: models such as MedGemma-27B achieve competitive VQA and report-generation performance, but all obtain G-Acc. below 0.006 and relatively low segmentation IoU. In contrast, 7 Running Title for Header Table 2: Experimental Results on UltraG-Bench. The best, second-best, and third-best results among all models are highlighted ingreen,blue, andyellow, respectively. For UltraG-Agent,I,Q, andLrefer to InternVL3.5-8B, Qwen3-VL-8B, and Lingshu-7B, while US3 and S3 refer to UltraSAM3 and SAM3, respectively. Model Instruction-guided Seg.Evidence-grounded VQAEvidence-grounded Report Gen. IoU↑Dice↑Acc.↑G-Acc.↑ROUGE-L↑SemAcc↑ Closed-source MLLMs Claude-4.50.15440.21800.33270.02140.13340.3281 GPT-5.40.26530.37180.36310.06090.22640.3577 Open-source General MLLMs Qwen3-VL-4B0.10690.15370.21970.01370.07930.1899 MiniCPM-V 2.60.06420.08540.02540.00000.15540.2592 Qwen3-VL-8B0.10610.16140.19380.00710.12740.2930 Qwen3-VL-32B0.11360.15950.30950.01130.22110.3025 InternVL3.5-8B0.09740.14620.36780.00810.19030.3113 InternVL3.5-14B0.11620.17550.35280.01760.24080.3426 Open-source Medical MLLMs LLaVA-Med0.13690.20060.22900.00000.16900.2822 MedGemma-4B0.10810.17020.32070.00250.13550.2938 Lingshu-7B0.07150.11140.32920.00000.16660.2975 Lingshu-32B0.09200.14130.29490.00460.19400.3000 MedGemma-27B0.16160.24240.36680.00560.21580.3436 Pixel-level Grounded Medical MLLMs UniBiomed0.15500.20680.02670.00020.12170.0567 Prompt-driven Segmentation Models BiomedParse0.26450.3303- -- -- -- - UltraSAM30.54780.6190- -- -- -- - Ours UltraG-Agent I+S3 0.04320.04830.08510.00000.17510.1568 UltraG-Agent L+US3 0.52920.6114 0.54180.33510.38050.6225 UltraG-Agent Q+US3 0.54350.62280.56840.42220.33000.5766 UltraG-Agent I+US3 0.55640.63840.59700.45310.39710.6687 the pixel-level grounded medical model UniBiomed shows stronger segmentation capability but substantially weaker semantic performance. Prompt-driven segmentation models exhibit much stronger localization, with BiomedParse and UltraSAM3 achieving IoU/Dice scores of 0.2645/0.3303 and 0.5478/0.6190, respectively, but they cannot perform VQA or report generation. In contrast, UltraG-Agent achieves substantial improvements across all three tasks, demonstrating the effectiveness of combining MLLM-based semantic understanding with ultrasound-specific pixel-level localization. Among all variants, UltraG-Agent I+US3 achieves the best performance on all six metrics, outperforming the best baseline by 0.2911, 0.2666, 0.2292, 0.3922, 0.1563, and 0.3110 in IoU, Dice, Acc., G-Acc., ROUGE-L, and SemAcc, respectively. Notably, compared with UltraSAM3 alone, UltraG-Agent I+US3 further improves IoU/Dice from 0.5478/0.6190 to 0.5564/0.6384, while additionally enabling VQA and report generation. Further comparison shows that UltraSAM3 is a key factor behind these gains: replacing it with the original SAM3 leads to substantial performance drops across 8 Running Title for Header 10 20 30 40 50 60 70 80 10 20 30 40 50 60 70 80 90 10 20 30 40 50 60 70 80 90 100 Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 10 20 30 40 50 60 70 80 90 10 20 30 40 50 60 70 80 90 100 Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 10 20 30 40 50 60 70 80 90 10 20 30 40 50 60 70 80 90 100 Evidence-grounded Report Gen. (SemAcc) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 GPT-5.4-mini Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 Claude Haiku 4.5 Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 InternVL3.5-14B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 InternVL3.5-8B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 Qwen3-VL-32B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 Qwen3-VL-8B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 MiniCPM-V 2.6 Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 Qwen3-VL-4B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 MedGemma-27B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 Lingshu-32B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 Lingshu-7B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 MedGemma-4B Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 LLaVA-Med Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 UniBiomed Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 UltraG-Agent I+SAM3 Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 UltraG-Agent Q+US3 Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 UltraG-Agent L+US3 Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 (a) Instruction-guided Seg. (IoU) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 (b) Evidence-grounded VQA (Acc.) Abdomen Breast Cardiac Carotid Kidney Lung NerveOvarian Prostate Thyroid Fetal Liver Muscle 10 20 30 40 50 60 70 80 90 100 (c) Evidence-grounded Report Gen. (SemAcc) GPT-5.4-mini Claude Haiku 4.5 InternVL3.5-14B InternVL3.5-8B Qwen3-VL-32B Qwen3-VL-8B MiniCPM-V 2.6 Qwen3-VL-4B MedGemma-27B Lingshu-32B Lingshu-7B MedGemma-4B LLaVA-Med UniBiomed UltraG-Agent I + SAM3 UltraG-Agent Q + US3 UltraG-Agent L + US3 UltraG-Agent I + US3 UltraG-Agent I+US3 Figure 4: Fine-grained organ-level comparison across 13 anatomical categories for the three tasks in UltraG-Bench. UltraG-Agent variants with UltraSAM3 show consistently strong performance across diverse organs. segmentation, VQA, and report generation. Meanwhile, different MLLM backbones yield only modest differences and all maintain strong performance. Overall, these results indicate that reliable Pixel-level Evidence Grounding depends on both high-level semantic reasoning and specialized ultrasound pixel-level localization, with UltraSAM3 serving as a critical foundation. 5.3.2 Fine-grained Organ-level Analysis Figure 4 presents the fine-grained performance of different models across 13 anatomical categories. Clear performance variations can be observed across organs, indicating that the difficulty of Pixel-level Evidence Grounding is jointly affected by anatomical structure and task type. In addition, both general-purpose and medical MLLMs exhibit substantial performance fluctuations across organs, particularly on Instruction-guided Segmentation, where their pixel- level localization ability is generally limited. Detailed case study and error analysis are provided in Appendix A.8 and A.9. In contrast, UltraG-Agent variants with UltraSAM3 achieve broader radar coverage across most anatomical categories and show relatively consistent trends across different MLLM backbones. This indicates that the advantage of UltraG- Agent is not driven by only a few specific organs, but generalizes more consistently across diverse ultrasound anatomical structures. 6 Conclusion In this work, we construct UltraG-Bench, a multi-task benchmark for evaluating Pixel-level Evidence Grounding in ultrasound. By annotating 40 public ultrasound segmentation datasets, UltraG-Bench provides unified evaluation across Instruction-guided Segmentation, Evidence-grounded VQA, and Evidence-grounded Report Generation. Our experiments reveal a clear gap between semantic understanding and pixel-level localization in existing MLLMs. To bridge this gap, we further propose UltraG-Agent. Extensive experiments show that UltraG-Agent consistently improves both semantic prediction and visual grounding performance. Reproducibility statement To support reproducibility, detailed dataset information and baseline descriptions are provided in Appendix A.1 and A.2. Benchmark construction details, including annotation prompts and quality validation, are described in Appendix A.3. The implementation details of UltraG-Agent and the complete evaluation protocols are provided in Appendix A.4 and A.5, respectively. Detailed definitions and attribute-wise analyses of Semantic Accuracy are given in Appendix A.6 and A.7. Additional case studies and error analyses are provided in Appendix A.8 and A.9. 9 Running Title for Header Our benchmark and associated code have been released anonymously athttps://anonymous.4open.science/r/ UltraG-Bench-F908. The Use of Large Language Models We used ChatGPT only for language refinement and writing assistance. All generated contents were carefully reviewed and revised. References [1]Kenneth A Fleming, Susan Horton, Michael L Wilson, Rifat Atun, Kristen DeStigter, John Flanigan, Shahin Sayed, Pierrick Adam, Bertha Aguilar, Savvas Andronikou, et al. The lancet commission on diagnostics: transforming access to diagnostics. The Lancet, 398(10315):1997–2050, 2021. [2]Xiaoqing Guo, Mohammad Alsharid, He Zhao, Yipei Wang, Jayne Lander, Aris T Papageorghiou, and J Alison Noble. A visually grounded language model for fetal ultrasound understanding. Nature Biomedical Engineering, pages 1–17, 2026. [3]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015. [4]Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature communications, 15(1):654, 2024. [5]Peder L Myhre, Bjørnar Grenne, Federico M Asch, Victoria Delgado, Rohan Khera, Stéphane Lafitte, Roberto M Lang, Patricia A Pellikka, Partho P Sengupta, Sreekanth Vemulapalli, et al. Artificial intelligence-enhanced echocardiography in cardiovascular disease management. Nature Reviews Cardiology, 23(3):164–182, 2026. [6] Kai Jin, Qixuan Sun, Daohuan Kang, Ziyao Luo, Tao Yu, Wenzheng Han, Yi Zhang, Meng Wang, Danli Shi, and Andrzej Grzybowski. Grounded report generation for enhancing ophthalmic ultrasound interpretation using vision-language segmentation models. npj Digital Medicine, 9(1):99, 2026. [7]Bo Xu, Quanhao Zhu, Qingchen Zhang, Mengmeng Wang, Liang Zhao, Hongfei Lin, Jing Ren, and Feng Xia. Echogpt: An interactive cardiac function assessment model for echocardiogram videos. In IJCAI, pages 7858–7866, 2025. [8]Bo Xu, Quanhao Zhu, Boling Zhu, Chenyuan Wang, Liang Zhao, Hongfei Lin, and Feng Xia. Medvcot: Bridging the modality gap in medical vqa through latent visual reasoning. [9]Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):180251, 2018. [10]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021. [11] Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020. [12] Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22170–22183. IEEE, 2024. [13] Yanlin Li, Minghui Guo, Kaiwen Zhang, Shize Zhang, Yiran Zhao, Haodong Li, Congyue Zhou, Weijie Zheng, Yushen Yan, Shengqiong Wu, Wei Ji, Lei Cui, Furu Wei, Hao Fei, Mong-Li Lee, and Wynne Hsu. Unim: A unified any-to-any interleaved multimodal benchmark. In Proceedings of the CVPR, 2026. [14]Bo Liu, Ke Zou, Li-Ming Zhan, Zexin Lu, Xiaoyu Dong, Yidi Chen, Chengqiang Xie, Jiannong Cao, Xiao-Ming Wu, and Huazhu Fu. Gemex: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 21310–21320. IEEE, 2025. [15]Geon Choi, Hangyul Yoon, Hyunju Shin, Hyunki Park, Sang Hoon Seo, Eunho Yang, and Edward Choi. Instruction- guided lesion segmentation for chest x-rays with automatically generated large-scale dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1482–1492, 2026. 10 Running Title for Header [16]Bo Xu, Quanhao Zhu, Rui Lin, Boling Zhu, Chenyuan Wang, Hongfei Lin, Feng Xia, and Chenhua Ji. Ultrasam3: A concept-driven foundation model for universal ultrasound image segmentation. arXiv preprint arXiv:2607.29200, 2026. [17] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. In European conference on computer vision, pages 1–21. Springer, 2022. [18]Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper S Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. Chest imagenome dataset for clinical reasoning. arXiv preprint arXiv:2108.00316, 2021. [19] Daniel Coelho De Castro, Aurelia Bustos, Shruthi Bannur, Stephanie L Hyland, Kenza Bouzid, Maria Teodora Wetscherek, Maria Dolores Sánchez-Valverde, Lara Jaques-Pérez, Lourdes Pérez-Rodríguez, Kenji Takeda, et al. Padchest-gr: A bilingual chest x-ray dataset for grounded radiology report generation. NEJM AI, 2(7):AIdbp2401120, 2025. [20]Bo Liu, Xiangyu Zhao, Along He, Yidi Chen, Huazhu Fu, and Xiao-Ming Wu. Gemex-rmcot: An enhanced med-vqa dataset for region-aware multimodal chain-of-thought reasoning. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 13213–13220, 2025. [21] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [22] AI Anthropic. System card: Claude opus 4 & claude sonnet 4. Claude-4 Model Card, 2025. [23] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [24]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. [25]Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. [26]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in neural information processing systems, 36:28541–28564, 2023. [27]Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. [28] Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: Generalist foundation model for unified multimodal medical understanding and reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. [29] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9579–9589. IEEE, 2024. [30] Linshan Wu, Yuxiang Nie, Sunan He, Jiaxin Zhuang, Luyang Luo, Tao Li, Zhuoyao Xie, Dexuan Chen, Yinghua Zhao, Neeraj Mahboobani, et al. A universal foundation model for grounded biomedical image interpretation. Nature Communications, 2026. [31]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. In International conference on learning representations, volume 2026, pages 138846–138923, 2026. [32]Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lee, Sid Kiblawi, Tristan Naumann, Jianfeng Gao, Angela Crabtree, Jacob Abel, et al. A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods, 22(1):166–176, 2025. [33]José Ignacio Orlando. AbdomenUS: US Simulation and Segmentation.https://w.kaggle.com/datasets/ ignaciorlando/ussimandsegm, 2018. [34]Anna Pawłowska, Anna ́ Cwierz-Pie ́ nkowska, Agnieszka Domalik, Dominika Jagu ́ s, Piotr Kasprzak, Rafał Matkowski, Łukasz Fura, Andrzej Nowicki, and Norbert ̇ Zołek. Curated benchmark dataset for ultrasound based breast lesion analysis. Scientific Data, 11(1):148, 2024. 11 Running Title for Header [35]Ali Abbasian Ardakani, Afshin Mohammadi, Mohammad Mirza-Aghazadeh-Attari, and U Rajendra Acharya. An open-access breast lesion ultrasound image database: Applicable in artificial intelligence studies. Computers in Biology and Medicine, 152:106438, 2023. [36]Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020. [37]Wilfrido Gómez-Flores, Maria Julia Gregorio-Calas, and Wagner Coelho de Albuquerque Pereira. Bus-bra: A breast ultrasound dataset for assessing computer-aided diagnosis systems. Medical physics, 51(4):3110–3123, 2024. [38]Moi Hoon Yap, Gerard Pons, Joan Marti, Sergi Ganau, Melcior Sentis, Reyer Zwiggelaar, Adrian K Davison, and Robert Marti. Automated breast ultrasound lesions detection using convolutional neural networks. IEEE journal of biomedical and health informatics, 22(4):1218–1226, 2017. [39] Ahmed Iqbal and Muhammad Sharif. Memory-efficient transformer network with feature fusion for breast tumor segmentation and classification task. Engineering Applications of Artificial Intelligence, 127:107292, 2024. [40]Noelia Vallez, Gloria Bueno, Oscar Deniz, Miguel Angel Rienda, and Carlos Pastor. Bus-uclm: Breast ultrasound lesion segmentation dataset. Scientific Data, 12(1):242, 2025. [41]Yanjun Guo, Xingguang Duan, Chengyi Wang, and Huiqin Guo. Segmentation and recognition of breast ultrasound images based on an expanded u-net. Plos one, 16(6):e0253202, 2021. [42] Sarah Leclerc, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, et al. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE transactions on medical imaging, 38(9):2198– 2210, 2019. [43]Jiewen Yang, Xinpeng Ding, Ziyang Zheng, Xiaowei Xu, and Xiaomeng Li. Graphecho: Graph-driven unsuper- vised domain adaptation for echocardiogram video segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11878–11887, 2023. [44]Tianchen Wang, Zhihe Li, Meiping Huang, Jian Zhuang, Shanshan Bi, Jiawei Zhang, Yiyu Shi, Hongwen Fei, and Xiaowei Xu. Echocp: An echocardiography dataset in contrast transthoracic echocardiography for patent foramen ovale diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 506–515. Springer, 2021. [45] David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):252–256, 2020. [46] Charitha D Reddy, Leo Lopez, David Ouyang, James Y Zou, and Bryan He. Video-based deep learning for automated assessment of left ventricular ejection fraction in pediatric patients. Journal of the American Society of Echocardiography, 36(5):482–489, 2023. [47]Unity Imaging Collaborative. Unity Imaging Echocardiography Datasets.https://data.unityimaging.net/, 2024. [48] Yuan Bi, Zhongliang Jiang, Ricarda Clarenbach, Reza Ghotbi, Angelos Karlas, and Nassir Navab. Mi-segnet: Mutual information-based us segmentation for unseen domain generalization. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 130–140. Springer, 2023. [49]M Sofia Sappia, Chris L de Korte, Bram van Ginneken, Dean Ninalga, Satoshi Kondo, Satoshi Kasai, Kousuke Hirasawa, Tanya Akumu, Carlos Martín-Isla, Karim Lekadir, et al. Acouslic-ai challenge report: Fetal abdominal circumference measurement on blind-sweep ultrasound data from low-income countries. Medical image analysis, 105:103640, 2025. [50]Karine Souza Da Correggio, R Noya Galluzzo, Luıs Otávio Santos, F Soares Muylaert Barroso, T Zimmer- mann Loureiro Chaves, A Sherlley Casimiro Onofre, and Aldo von Wangenheim. Fetal abdominal structures segmentation dataset using ultrasonic images. Mendeley Data, 1:1, 2023. [51]Vahid Ashkani Chenarlogh, Mostafa Ghelich Oghli, Ali Shabanzadeh, Nasim Sirjani, Ardavan Akhavan, Isaac Shiri, Hossein Arabi, Morteza Sanei Taheri, and Mohammad Kazem Tarzamni. Fast and accurate u-net model for fetal ultrasound image segmentation. Ultrasonic imaging, 44(1):25–38, 2022. [52] Thomas LA van den Heuvel, Dagmar de Bruijn, Chris L de Korte, and Bram van Ginneken. Automated measurement of fetal head circumference using 2d ultrasound images. PloS one, 13(8):e0200412, 2018. [53] Bai Jieyun and Ou ZhanHong. Pubic symphysis-fetal head segmentation and angle of progression, March 2024. 12 Running Title for Header [54]Wu Songxiong, Zhang Hongyuan, Ye Tingting, X Haoyu, Z Ping, S Qingjun, W Panying, H Bingsheng, D Lei, and W Guangyao. Focus: four-chamber ultrasound image dataset for fetal cardiac biometric measurement. Zenodo10, 5281, 2025. [55]Rohit Singla, Cailin Ringstrom, Grace Hu, Victoria Lessoway, John Reid, Christopher Nguan, and Robert Rohling. The open kidney ultrasound data set. In International Workshop on Advances in Simplifying Medical Ultrasound, pages 155–164. Springer Nature Switzerland, 2023. [56] JeevaWS.Ultrasound Normal Kidney Image Dataset.https://universe.roboflow.com/jeevaws/ ultrasound-normal-kidney-image, 2025. [57] Xu Yiming, Zheng Bowen, Liu Xiaohong, Wu Tao, Ju Jinxiu, Wang Shijie, Lian Yufan, Zhang Hongjun, Liang Tong, Sang Ye, Jiang Rui, Wang Guangyu, Ren Jie, and Chen Ting. Annotated ultrasound liver images, November 2022. [58]James R McLaughlan, Lewis Howell, and Nicola Ingram. Lung ultrasound covid phantom dataset used for training machine learning model. 2024. [59]Rebecca Cunningham and Neil Cronin. FALLMUD: FAscicle Lower Leg Muscle Ultrasound Dataset.https: //kalisteo.cea.fr/index.php/fallmud/, 2020. [60]Clyde J Belasso, Bahareh Behboodi, Habib Benali, Mathieu Boily, Hassan Rivaz, and Maryse Fortin. Luminous database: lumbar multifidus muscle segmentation from ultrasound images. BMC Musculoskeletal Disorders, 21(1):703, 2020. [61]Francesco Marzola, Nens Van Alfen, Jonne Doorduin, and Kristen M Meiburger. Deep learning segmentation of transverse musculoskeletal ultrasound images for neuromuscular disease assessment. Computers in biology and medicine, 135:104623, 2021. [62]Yi Ding, IEEE Member, Qiqi Yang, Yiqian Wang, Dajiang Chen, Zhiguang Qin, and Jian Zhang. Mallesnet: A multi-object assistance based network for brachial plexus segmentation in ultrasound images. Medical Image Analysis, 80:102511, 2022. [63]Qi Zhao, Shuchang Lyu, Wenpei Bai, Linghan Cai, Binghao Liu, Guangliang Cheng, Meijing Wu, Xiubo Sang, Min Yang, and Lijiang Chen. Mmotu: A multi-modality ovarian tumor ultrasound image dataset for unsupervised cross-domain semantic segmentation. arXiv preprint arXiv:2207.06799, 2022. [64]W Shao and W Brisbane. Micro-ultrasound prostate segmentation dataset. URL: https://doi. org/10.5281/zenodo, 10475293, 2024. [65]Zachary Baum, Shaheer Saeed, Zhe Min, Yipeng Hu, and Dean Barratt. Mr to ultrasound registration for prostate challenge-dataset. In Medical Image Computing and Computer Assisted Intervention–MICCAI, volume 2023, 2023. [66]Lina Pedraza, Carlos Vargas, Fabián Narváez, Oscar Durán, Emma Muñoz, and Eduardo Romero. An open access thyroid ultrasound image database. In 10th International symposium on medical information processing and analysis, volume 9287, pages 188–193. SPIE, 2015. [67]Yuchen Wang, Zhongyu Li, Xiangxiang Cui, Liangliang Zhang, Xiang Luo, Meng Yang, and Shi Chang. Key- frame guided network for thyroid nodule recognition using ultrasound videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 238–247. Springer, 2022. [68]Markus Krönke, Christine Eilers, Desislava Dimova, Melanie Köhler, Gabriel Buschner, Lilit Schweiger, Lemonia Konstantinidou, Marcus Makowski, James Nagarajah, Nassir Navab, et al. Tracked 3d ultrasound and deep neural network-based thyroid segmentation reduce interobserver variability in thyroid volumetry. Plos one, 17(7):e0268550, 2022. [69] Haifan Gong, Jiaxin Chen, Guanqi Chen, Haofeng Li, Guanbin Li, and Fei Chen. Thyroid region prior guided attention for ultrasound segmentation of thyroid nodules. Computers in biology and medicine, 155:106389, 2023. [70]Haifan Gong, Guanqi Chen, Ranran Wang, Xiang Xie, Mingzhi Mao, Yizhou Yu, Fei Chen, and Guanbin Li. Multi-task learning for thyroid nodule segmentation with thyroid region prior. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 257–261. IEEE, 2021. [71] Stanford AIMI. Thyroid ultrasound cine-clip, April 2026. 13 Running Title for Header A Appendix A.1 Datasets Details Source Dataset Statistics.Table 3 summarizes the 40 ultrasound segmentation datasets spanning 13 organs that serve as sources for UltraG-Bench. The selected source data comprises 138,832 images and 181,952 segmentation masks across the training and test splits. Task-wise Anatomical Distribution. Figure 5 shows the anatomical-category distributions of the three tasks in UltraG-Bench. Overall, the three tasks exhibit similar distributions across the 13 anatomical categories. Cardiac and thyroid samples constitute the largest proportions, accounting for 38%–41% and 26%–27% of the annotations, respectively, followed by fetal, muscle, and prostate categories. The remaining anatomical categories occupy smaller proportions. Despite differences in the total number of annotations across tasks, their anatomical compositions remain broadly consistent, providing comparable category coverage for evaluating pixel-level evidence grounding from segmentation to VQA and report generation. 38% 6% 26% 9% 8% Instruction-guided Segmentation n = 331,125 40% 5% 27% 11% 7% Evidence-grounded VQA n = 666,779 41% 5% 27% 11% 7% Evidence-grounded Report Generation n = 138,832 Abdomen Ovarian Breast Prostate Cardiac Thyroid Carotid artery Fetal Kidney Liver Lung Muscle Nerve Figure 5: Anatomical-category distributions of the three tasks in UltraG-Bench across 13 anatomical categories. A.2 Baseline Introduction We evaluate a diverse set of baseline models covering closed-source general-purpose MLLMs, open-source general- purpose MLLMs, medical MLLMs, pixel-level grounded medical MLLMs, and prompt-driven segmentation models. The evaluated baselines are briefly introduced below. Claude-4.5.Claude-4.5 [22] is a proprietary multimodal large language model developed by Anthropic, with strong general-purpose capabilities in visual understanding and multimodal reasoning. We include it as a representative closed-source MLLM for evaluating semantic understanding and visual grounding in ultrasound images. GPT-5.4.GPT-5.4 [21] is a proprietary frontier model developed by OpenAI with advanced multimodal understanding and reasoning capabilities. We evaluate GPT-5.4 as a strong closed-source general-purpose baseline across all three tasks of UltraG-Bench. Qwen3-VL Series.Qwen3-VL [23] is an open-source vision-language model family designed for general multimodal perception and reasoning, supporting diverse visual inputs and fine-grained visual understanding. We evaluate the 4B, 8B, and 32B variants to investigate the effect of model scale on ultrasound Pixel-level Evidence Grounding. MiniCPM-V 2.6.MiniCPM-V 2.6 [25] is a compact 8B multimodal model designed for efficient image, multi-image, and video understanding. Despite its relatively small model size, it provides strong general-purpose visual-language capabilities and serves as an efficient open-source MLLM baseline. 14 Running Title for Header Table 3: Statistics of the 40 ultrasound datasets retained in the Grounded-US annotations. OrganDatasetTrain ImagesTrain MasksTest ImagesTest Masks AbdomenAbdomenUS [33]7843,079198781 BreastBrEast [34]20440252102 BreastBUID [35]1853704794 BreastBUSI [36]6241,056156274 BreastBUS_BRA [37]1,5003,000375750 BreastBUS_DatasetB [38]1302603366 BreastBUS_UC [39]6481,294163326 BreastBUS_UCLM [40]546408137120 BreastS1 [41]192388918 CardiacCAMUS [42]1,6004,8004001,200 CardiacCardiacUDC [43]1,7175,8925331,754 CardiacEchoCP [44]4341,696132521 CardiacEchoNet_Dynamic [45]17,47617,4762,5502,550 CardiacEchoNet_Pediatric [46]12,07112,0713,3003,300 CardiacUnity [47]13,1365,6753,2831,367 Carotid artery CCA [48]2,3072,195540540 FetalACOUSLIC [49]5,3275,3271,2931,293 FetalFASS [50]1,27010,0123182,506 FetalFast_UNet [51]1,1281,128283283 FetalHC [52]799799200200 Fetalfh_ps [53]3,2006,4008001,600 Fetalfocus [54]25050050100 KidneyKidneyUS [55]3883889898 KidneyUltrasound_Normal_Kidney [56]8641,728216432 LiverAnnotated_Ultrasound_Liver [57]588510147125 LungLUSS [58]4512,977113775 MuscleFALLMUD [59]650655163326 MuscleLUMINOUS [60]2693026878 MuscleSTMUS_NDA [61]6,5296,5301,6321,632 NerveUPBD [62]7643,441191935 OvarianOTU_2d [63]1,1751,191294298 OvarianOTU_3d [63]1361463440 ProstateMicroSeg [64]2,0782,085542542 ProstateRegPro [65]3,5163,786367396 ThyroidDDTI [66]509510128128 ThyroidKFGNet [67]1311613337 ThyroidSegthy [68]9,93819,1871,9523,482 ThyroidTG3K [69]2,8683,056717772 ThyroidTN3K [70]2,7943,063699757 ThyroidThyroid_US_Cineclip [71]13,56513,5653,8453,845 Total–112,741147,50926,09134,443 InternVL3.5 Series. InternVL3.5[24] is an open-source multimodal model family that improves visual-language understanding, reasoning, and inference efficiency through enhanced multimodal training and reinforcement learning. We evaluate its 8B and 14B variants as representative general-purpose MLLMs with strong multimodal reasoning capabilities. 15 Running Title for Header LLaVA-Med. LLaVA-Med [26] adapts a general-purpose vision-language assistant to the biomedical domain us- ing large-scale biomedical image-text data and instruction tuning. It is designed for open-ended conversational understanding of biomedical images and serves as a representative early medical MLLM. MedGemma Series.MedGemma [27] is a family of medical vision-language foundation models built upon Gemma and specifically adapted for medical image and text understanding. We evaluate the 4B and 27B variants to assess whether medical-domain adaptation and increased model capacity improve ultrasound evidence grounding. Lingshu Series.Lingshu [28] is a generalist medical MLLM designed for unified multimodal medical understanding and reasoning through large-scale medical data curation and multi-stage training. We evaluate the 7B and 32B variants, which support a broad range of medical imaging modalities and tasks including medical VQA and report generation. UniBiomed. UniBiomed [30] is a grounded biomedical vision-language foundation model that jointly performs semantic interpretation and pixel-level segmentation of biomedical targets. Unlike conventional medical MLLMs that primarily generate textual responses, UniBiomed explicitly connects generated medical findings with localized visual regions, making it particularly relevant to Pixel-level Evidence Grounding. BiomedParse.BiomedParse [32] is a biomedical image parsing foundation model that unifies segmentation, detection, and recognition across diverse biomedical imaging modalities. It supports text-prompted segmentation of biomedical objects and is therefore included as a representative prompt-driven biomedical segmentation model. UltraSAM3. UltraSAM3 [16] is a concept-driven foundation model specifically developed for universal ultrasound image segmentation. By adapting SAM3 to ultrasound-specific image–mask–concept triplets, it enables text-based target specification and provides strong prompt-driven pixel-level localization across diverse ultrasound anatomical structures. A.3 Benchmark Construction Details Annotation Details. For each ultrasound image, we first extract evidence facts from the original COCO-format segmentation annotations, including target presence, category, mask identity, coarse image location, pixel extent, mask area ratio, boundary information, and the number of distinct targets. These mask-derived facts serve as the only visual evidence available during annotation. Based on them, we construct three types of annotations: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation. The annotation process follows strict evidence constraints: generated text is restricted to information directly supported by the supplied labels and mask geometry, while unsupported diagnostic, pathological, treatment-related, or metadata-related statements are prohibited. After generation, all annotations are automatically normalized and validated to ensure valid evidence references and task-format consistency. Segmentation instructions and grounded reports are deterministically constructed from mask- derived facts, while VQA pairs are further filtered according to clinical relevance and evidence validity; invalid outputs are replaced by rule-based fallback annotations. The same annotation pipeline is applied to all 13 anatomical categories by replacing the organ-specific target with organ. Prompt Create a grounded multi-task ultrasound sample for organ. Return valid JSON only. Use only the supplied facts. Theorganlabel identifies the segmentation target. Do not introduce any clinical attribute that is not explicitly supported by the supplied labels or mask geometry. Do not invent diagnoses, pathological findings, prognosis, treatment, recommendations, or other unsupported clinical conclu- sions. Never ask about or mention file names, case numbers, slice numbers, image IDs, dataset/source names, split names, paths, dates, annotations, mask IDs, or metadata in user-facing text. Every VQA question must be directly supported by the available organ evidence, such as: • target presence or visibility; • image-region location; • pixel extent or size; • area ratio; • boundary availability; • number of visible targets. Use [SEG] only when target_mask_ids is non-empty. Mask IDs may appear only in ID fields. 16 Running Title for Header The supplied masks support target presence, approximate image location, pixel-level size and extent, target count, and boundary information. For questions about a visible organ target, set requires_grounding=true and evidence_source=mask_geometry. If no target mask is available, generate only absence or not-applicable questions with requires_grounding=false. Return exactly the following top-level keys: segmentation_tasks: 2–4 objects with task_id, instruction, answer, target_mask_ids, and answerability. grounded_vqa : 4–6 objects withqa_id,question,answer,answer_type,requires_grounding,evidence_mask_ids, reasoning_type, and evidence_source. grounded_report : an object with a report prompt and 5–7 sentence objects containingsection,text,evidence_mask_ids, and finding_type. Facts: mask-derived evidence facts Automated Quality Validation. We perform rule-based automatic quality validation for every generated sample before manual review. The validator checks five aspects. First, masks must have unique identifiers, valid anatomical categories, positive geometry and area, valid area ratios, coordinates within image boundaries, and no duplicated geometry. Second, each sample must contain 2–4 instruction-guided segmentation annotations, where every referenced target mask exists and the[SEG]token is used if and only if a valid mask is referenced. Third, each sample must contain 4–6 evidence-grounded VQA pairs with non-empty question–answer content, valid evidence masks, and consistentrequires_groundingflags. Fourth, grounded reports must contain 5–7 sentences with at least Findings and Impression, and all statements about target presence, category, location, extent, and count must agree with the referenced masks. Finally, we reject unsupported content beyond the available pixel-level evidence, including unannotated anatomical attributes, unsupported laterality or physical measurements, diagnostic or pathological claims, treatment recommendations, and metadata-related statements. Organ-specific constraints are further defined for each organ. Issues are categorized as critical, major, minor, or informational, with penalties of 25, 10, 3, and 0 points from an initial score of 100; samples containing any critical or major issue are flagged as failing automatic validation. A.4 UltraG-Agent Implementation Details UltraG-Agent follows a modular inference pipeline that separates high-level instruction understanding from pixel-level visual grounding. Given an ultrasound imageI, a user instructionq, and a predefined target-category setC, an MLLM first acts as a task planner to identify the task type and the anatomical targets required to answer the instruction: P =M plan (I,q,C) = τ,(c k ,p k ) K k=1 ,(3) whereτ ∈Segmentation, VQA, Reportdenotes the inferred task,c k ∈Cis the selected target category, andp k is a concise text prompt generated for UltraSAM3. Multiple targets are selected only when required by the instruction. For each planned target, UltraSAM3 independently performs prompt-driven segmentation: (m k,j ,s k,j ) N k j=1 =S US3 (I,p k ),(4) wherem k,j ands k,j denote thej-th candidate mask and its confidence score, respectively. After filtering invalid candidates, the highest-scoring mask is selected as ˆm k = arg max m k,j s k,j .(5) We then extract mask-derived evidence g k =G( ˆm k ) =b k , l k , a k , r k , d long k , d short k ,(6) including the bounding boxb k , image-region locationl k , pixel areaa k , area ratior k , and long- and short-axis pixel extents. These quantities form the authoritative visual evidence used by the subsequent response generation stage. The final response is conditioned on the original instruction, the task plan, and the mask-derived evidence: y =M resp I,q,P,(c k ,g k ) K k=1 .(7) For instruction-guided segmentation, the output consists of the textual response and the corresponding predicted mask. For evidence-grounded VQA, the answer is paired with the union of the masks referenced by the response: M VQA = [ k∈T a ˆm k ,(8) 17 Running Title for Header whereT a denotes the targets supporting the answer. For evidence-grounded report generation, each report statementy i is associated with its own evidence mask, M i = [ k∈T i ˆm k ,(9) thereby preserving fine-grained statement–evidence correspondence throughout the generated report. Mask-derived facts are treated as authoritative during response generation, preventing the MLLM from modifying target presence, location, pixel extent, or area measurements inferred from UltraSAM3. A.5 Evaluation Details Prompt-driven Segmentation Models Evaluation Details.For prompt-driven segmentation models, the ultrasound image and its segmentation instruction are directly provided as input. The predicted mask is evaluated against the ground-truth segmentation using IoU and Dice, following the same protocol as Instruction-guided Segmentation. UltraG-Agent Evaluation Details. The same evaluation samples, reference annotations, mask decoding procedure, and metric implementation are used for UltraG-Agent to ensure direct comparability with the MLLM baselines. MLLM Evaluation Details. We evaluate all MLLMs under a unified zero-shot protocol across the three tasks of UltraG-Bench. For each sample, the model receives the original ultrasound image together with a task-specific textual prompt and is required to return a structured JSON response. For spatial predictions, models may output either a polygon or a bounding box in the original-image pixel coordinate system; these predictions are rasterized into binary masks and compared with the corresponding ground-truth segmentation masks. All API-based models are evaluated with temperature0to reduce sampling variance, and the same task prompts and output constraints are used across different models. No ground-truth mask coordinates or reference answers are provided to the model during inference. For report generation, only the required finding-type structure is specified to standardize the output format, while the reference report content and evidence masks remain hidden. For Instruction-guided Segmentation, we report IoU and Dice between the predicted and ground-truth masks. For Evidence-grounded VQA, Answer Accuracy measures semantic correctness, while Grounded Accuracy further requires the answer to be correct and the predicted evidence region to achieve an IoU above0.5with the ground-truth evidence mask. For Evidence-grounded Report Generation, ROUGE-L evaluates textual similarity, while Semantic Accuracy measures the correctness of four evidence-supported attributes: target presence, category, location, and size. Location is evaluated using the predefined3× 3image-region grid, and predicted pixel extent is considered correct when both dimensions are within a 20% tolerance of the reference values. Evaluation Prompts. The following prompts are used consistently for all evaluated MLLMs. Here,organ,W, and H denote the anatomical category and the original image width and height, respectively. Instruction-guided Segmentation Prompt You are given a clinical ultrasound image and a segmentation instruction. Return valid JSON only with keys: answer, predicted_mask. predicted_maskmust be"type":"polygon","points":[[x,y],...],"type":"bbox","bbox_xywh":[x,y,w,h], or null. Use original-image pixel coordinates. Include [SEG] in answer if a target is present. Do not use annotation, annotated, dataset, mask id, or metadata. Input: "image_width": W, "image_height": H, "instruction": "instruction" Evidence-grounded VQA Prompt You are given a clinical ultrasound image and a question. Return valid JSON only with keys: answer, predicted_mask. If the answer depends on a visible target, organ, or structure, providepredicted_maskas a polygon or bbox in original-image pixel coordinates; otherwise use null. Do not use annotation, annotated, dataset, mask id, or metadata. 18 Running Title for Header Input: "organ": "organ", "image_width": W, "image_height": H, "question": "question" Evidence-grounded Report Generation Prompt You are given a clinical ultrasound image. Return valid JSON only with key report. report must be a list of sentence objects with section, text, finding_type, and predicted_mask. For spatialfinding_typevalues,predicted_maskmust localize the supporting target, organ, or structure as a polygon or bbox in original-image pixel coordinates. Use null predicted mask for non-spatial statements. Use the provided finding-type structure when applicable. Use concise benchmark-style wording. Do not invent ultrasound descriptors if uncertain. Do not invent diagnostic grades, pathology classes, or treatment recommendations, and do not use cm unless a scale marker is visible. If estimating size, use pixel coordinates or pixel extent only. Do not use annotation, annotated, dataset, mask id, or metadata. Input: "organ": "organ", "image_width": W, "image_height": H, "prompt": "report_instruction" A.6 Detailed Definition of Semantic Accuracy To complement lexical similarity metrics, we evaluate whether a generated report preserves the key visual evidence represented by the reference annotation. Specifically, Semantic Accuracy (SemAcc) consists of four binary attributes: target presence, anatomical category, spatial location, and target size. LetRand ˆ Rdenote the reference and generated reports, respectively. Each attribute produces a binary indicator in0, 1, as defined below. Target Presence. The presence attribute evaluates whether the generated report correctly describes the target as present or absent. We extract a binary target-presence state from both the reference and generated reports: p(R), p( ˆ R)∈0, 1,(10) where 1 denotes that the target is present and 0 denotes that the target is absent. Presence correctness is defined as presence_ok = I h p( ˆ R) = p(R) i ,(11) whereI[·]is the indicator function. In practice, explicit absence expressions (e.g., “no target”, “target is absent”, or “target is not visible”) are interpreted as negative evidence, whereas explicit mentions of the target structure indicate target presence. Anatomical Category.The category attribute measures whether the generated report correctly identifies the anatomi- cal target specified by the reference annotation. LetC(R)andC( ˆ R)denote the sets of target categories mentioned in the reference and generated reports, respectively. We define category_ok = I h C(R)⊆C( ˆ R) i .(12) Thus, all target categories required by the reference report must be correctly identified in the generated report. Spatial Location.We represent target location using a3× 3spatial grid over the original image. Given an image of width W and height H , and the center of the reference target mask (x c ,y c ), its horizontal position is defined as h(x c ) =    left, x c < W/3, central, W/3≤ x c < 2W/3, right, x c ≥ 2W/3, (13) 19 Running Title for Header and its vertical position is v(y c ) =    upper, y c < H/3, middle, H/3≤ y c < 2H/3, lower, y c ≥ 2H/3. (14) The resulting location label is ℓ = (v(y c ),h(x c )),(15) corresponding to one of nine regions such as upper-left, middle-central, or lower-right. LetL(R)andL( ˆ R)denote the spatial-location labels extracted from the reference and generated reports. Location correctness is defined as location_ok = I h L( ˆ R) =L(R) i .(16) Target Size. The size attribute evaluates whether the pixel-level extent reported by the model is consistent with that derived from the ground-truth mask. Let(d 1 ,d 2 )denote the two reference pixel extents and( ˆ d 1 , ˆ d 2 )denote the corresponding values extracted from the generated report. Since the ordering of the long and short axes may vary in natural-language generation, we consider both direct and reversed matching. For each reference dimensiond, the allowable tolerance is δ(d) = max(1, 0.2d),(17) i.e., either one pixel or 20% of the reference dimension, whichever is larger. We define Match ( ˆ d 1 , ˆ d 2 ), (d 1 ,d 2 ) = 2 i=1 | ˆ d i − d i |≤ δ(d i ) .(18) The final size indicator is size_ok = I h Match ( ˆ d 1 , ˆ d 2 ), (d 1 ,d 2 ) ∨ Match ( ˆ d 1 , ˆ d 2 ), (d 2 ,d 1 ) i . (19) Semantic Accuracy. The four indicators are equally weighted to obtain the report-level semantic accuracy: SemAcc = 1 4 (presence_ok + category_ok + location_ok + size_ok).(20) The final SemAcc reported for a model is the average over all evaluated report samples: SemAcc final = 1 N N X n=1 SemAcc (n) ,(21) where N denotes the number of evaluated reports. A.7 Attribute-wise Analysis of Semantic Accuracy Table 4 provides a detailed breakdown of Semantic Accuracy into its four constituent attributes: presence, category, location, and size. Existing MLLMs generally perform well on target presence, indicating that they can often recognize whether the relevant anatomical structure is visible in an ultrasound image. However, their performance decreases substantially for more fine-grained attributes. In particular, most general-purpose and medical MLLMs achieve very low accuracy on location and size, suggesting that semantic recognition does not necessarily translate into precise spatial understanding. Category recognition is comparatively stronger, but still remains substantially below the best UltraG-Agent variants. UltraG-Agent shows clear improvements in the attributes that require pixel-level visual evidence. UltraG-Agent L+US3 achieves the highest presence accuracy of 0.9614, while UltraG-Agent I+US3 obtains the best results for category, location, and size, reaching 0.7254, 0.6462, and 0.4606, respectively. The gains are particularly pronounced for location and size, where conventional MLLMs achieve only limited performance. Moreover, replacing UltraSAM3 with the original SAM3 leads to substantial degradation across these attributes, further demonstrating the importance of ultrasound-specific pixel-level localization. Overall, the attribute-level results show that the improvement in SemAcc primarily comes from stronger alignment between semantic predictions and fine-grained spatial evidence, rather than from target recognition alone. 20 Running Title for Header Table 4: Detailed comparison of the four semantic attributes used in Semantic Accuracy on UltraG-Bench. The best, second-best, and third-best results among all models are highlighted ingreen,blue, andyellow, respectively. For UltraG-Agent,I,Q, andLrefer to InternVL3.5-8B, Qwen3-VL-8B, and Lingshu-7B, whileUS3andS3refer to UltraSAM3 and SAM3, respectively. ModelPresence↑Category↑Location↑Size↑ Closed-source MLLMs Claude-4.50.87890.38710.00670.0001 GPT-5.40.81390.48240.03050.0418 Open-source General-purpose MLLMs Qwen3-VL-4B0.52310.23280.00010.0065 MiniCPM-V 2.60.70700.29780.00020.0020 Qwen3-VL-8B0.81730.29870.00050.0000 Qwen3-VL-32B0.82750.32230.01810.0007 InternVL3.5-8B0.85080.36780.00420.0025 InternVL3.5-14B0.93150.40920.00900.0113 Medical MLLMs LLaVA-Med0.85900.24160.00000.0002 MedGemma-4B0.89320.34840.00160.0003 Lingshu-7B0.74700.41210.00000.0000 Lingshu-32B0.67890.48360.00000.0054 MedGemma-27B0.86470.44180.00070.0037 Pixel-level Grounded Medical MLLMs UniBiomed0.24720.06460.00000.0003 UltraG-Agent (Ours) UltraG-Agent I+S3 0.14180.40680.03820.0000 UltraG-Agent L+US3 0.96140.58530.58020.4335 UltraG-Agent Q+US3 0.95000.50240.59920.3997 UltraG-Agent I+US3 0.95020.72540.64620.4606 A.8 Case Study Instruction-guided Segmentation Case Study.Figure 6 presents a representative breast lesion segmentation example. UltraG-Agent produces pixel-level masks that closely follow the ground-truth lesion boundary, with UltraG-Agent I+US3 showing particularly accurate spatial alignment and boundary delineation. UltraSAM3 and BiomedParse also achieve reasonable localization, reflecting the advantage of dedicated prompt-driven segmentation models for fine-grained spatial prediction. In contrast, most standalone MLLMs struggle to produce precise pixel-level outputs, often returning coarse bounding boxes, incomplete contours, or regions that only partially overlap with the target. This case illustrates the substantial gap between recognizing the requested anatomical target and accurately delineating its spatial extent. Figure 7 provides another qualitative comparison under the same instruction-guided segmentation setting. A similar trend can be observed: UltraG-Agent generates a more complete and spatially consistent target mask, while many standalone MLLMs provide only approximate localization or fail to capture the target boundary accurately. Although prompt-driven segmentation models generally exhibit stronger localization ability than conventional MLLMs, UltraG- 21 Running Title for Header C: _studies_max_coverage_v3 _model_coverage\ vqa\02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Q:Segment the breast lesion UniBiomed Breast Instruction-guided Segmentation Case Study UltraG-Agent I+US3 UltraG-Agent L+US3 Claude 4.5 GPT 5.4 InternVL3.5-8B InternVL3.5-14B Lingshu-7BLingshu-32B LLava-Med MedGemma-4BMedGemma-27BMiniCPM-V-2.6Qwen3VL-32B Qwen3VL-8B Qwen3VL-4B BiomedParse UltraSAM3 Figure 6: Breast Instruction-guided Segmentation Case Study. Agent retains this fine-grained segmentation capability while additionally supporting the semantic understanding required by the other tasks in UltraG-Bench. Together, the two cases demonstrate the complementary benefit of combining MLLM-based instruction understanding with ultrasound-specific pixel-level segmentation. Evidence-grounded VQA Case Study. Figure 8 presents a representative thyroid VQA example. UltraG-Agent correctly identifies that a single thyroid nodule is present and provides a well-aligned segmentation mask as pixel-level evidence. In contrast, several compared MLLMs either return an answer without valid grounding or generate incorrect and unsupported descriptions. This case demonstrates that a semantically plausible answer alone is insufficient without spatial evidence aligned with the target. Figure 9 shows a representative ovarian VQA example. UltraG-Agent correctly localizes the ovarian tumor target to the middle-central image region and grounds the answer with the corresponding segmentation mask. By comparison, the other models produce inaccurate spatial descriptions, unsupported anatomical references, or responses without valid pixel-level grounding. This case highlights the importance of explicitly aligning semantic answers with fine-grained visual evidence. Evidence-grounded Report Generation Case Study.Figure 10 presents a representative prostate ultrasound report generation example. UltraG-Agent produces a structured report that closely follows the reference in target presence, 22 Running Title for Header C: _studies_max_coverage_v3 _model_coverage\ vqa\02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Q:Segment the visible fetal head UniBiomed Fetal Instruction-guided Segmentation Case Study UltraG-Agent I+US3 UltraG-Agent L+US3 Claude 4.5 GPT 5.4 InternVL3.5-8B InternVL3.5-14B Lingshu-7BLingshu-32B LLava-Med MedGemma-4BMedGemma-27BMiniCPM-V-2.6Qwen3VL-32B Qwen3VL-8BQwen3VL-4B BiomedParse UltraSAM3 Figure 7: Fetal Instruction-guided Segmentation Case Study. middle-central location, pixel extent, and visible-area proportion, while providing a corresponding segmentation mask. In contrast, UniBiomed generates largely nonspecific statements despite producing a plausible mask, Qwen3-VL-32B introduces substantial errors in location and size estimation, and Lingshu-32B further hallucinates unsupported clinical attributes and physical measurements. This example shows that accurate report generation requires not only recognizing the target, but also grounding each spatial description in reliable pixel-level evidence. Figure 11 shows a liver ultrasound report generation example. UltraG-Agent correctly identifies the liver lesion, localizes it to the middle-central image region, and reports its pixel-level extent and image-area proportion with a mask that closely matches the reference. By comparison, UniBiomed produces semantically incorrect content referring to lung findings, while Claude-4.5 and GPT-5.4 fail to identify the focal liver lesion and instead generate unsupported or contradictory descriptions. The case highlights the benefit of evidence-constrained generation, where report content is explicitly tied to the localized ultrasound target rather than generated from semantic cues alone. A.9 Error Analysis Instruction-guided Segmentation Error Analysis. Figure 12 illustrates a challenging abdominal ultrasound case in which most models fail to accurately delineate the visible liver region. UltraG-Agent and UltraSAM3 localize only a small region near the upper-central part of the image, resulting in a substantial mismatch with the ground-truth 23 Running Title for Header C: _studies_max_coverage_v3 \ max_model_coverage \02_DDTI_coco_DDTI_coco_test_100_vqa _2 .png Ground Truth Reference Answer: 1 Q:How many thyroid nodules are present in the image? UltraG-Agent I+US3 Answer: One target is present.[SEG] UniBiomed Answer: A parenchymal nodule with a diameter of 1 cm was observed in the superior segment of the right lung lower lobe. It was evaluated in favor of chronic parenchymal disease. the right lung lower lobe is not ventilated.[SEG] GPT 5.4 Answer: 1.[NO_SEG] MedGemma-27B Answer: There are 2 thyroid nodules present in the image.[NO_SEG] MedGemma-4B Answer: 2.[NO_SEG] Thyroid Evidence-grounded VQA Case Study Figure 8: Thyroid Evidence-grounded VQA Case Study. mask, while BiomedParse and UniBiomed produce incomplete or fragmented predictions. Standalone MLLMs exhibit more severe failure modes, including coarse bounding boxes, substantial over-segmentation, spatially shifted regions, and empty predictions. This case highlights the difficulty of segmenting targets with weakly defined boundaries and irregular visible extent in ultrasound. It also reveals a limitation of UltraG-Agent: when the underlying prompt-driven segmentation model misidentifies the target region, the localization error is propagated to the final grounded output, indicating that further improvements in ultrasound-specific segmentation remain important. Evidence-grounded VQA Error Analysis.Figure 13 presents a challenging lung ultrasound VQA case that exposes different failure modes in semantic prediction and visual grounding. UltraG-Agent Q+US3 correctly identifies the A-line location as upper-central and provides a corresponding grounded region, while UltraG-Agent L+US3 produces an incorrect middle-central localization despite generating a plausible mask. InternVL3.5-14B predicts the correct textual location but does not provide valid pixel-level evidence, illustrating that semantic correctness alone is insufficient for grounded VQA. UniBiomed fails to identify the relevant target, whereas Qwen3VL-32B generates an over-detailed description with inaccurate localization and an overly broad evidence region. This example highlights that evidence- grounded VQA requires both correct semantic reasoning and spatially consistent localization, and errors in either component lead to failure under the joint evaluation criterion. Evidence-grounded Report Generation Error Analysis.Figure 14 presents a challenging cardiac ultrasound case involving multiple anatomical structures. The reference report requires grounded descriptions of the left ventricular endocardium, left ventricular epicardium, and left atrium, whereas UltraG-Agent identifies only a single left-ventricular target and further assigns an incorrect upper-central location. This illustrates a limitation of the current agent pipeline when an instruction requires simultaneous planning and grounding of multiple structures. UniBiomed fails to produce relevant cardiac findings, while MedGemma-27B recognizes several cardiac chambers but introduces unsupported structures and provides an imprecise evidence region. GPT-5.4 generates clinically plausible descriptions of cardiac 24 Running Title for Header C: _studies_max_coverage_v3 _model_c overage \02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Reference Answer: The target is located in the middle-central region. Q:Where is the ovarian tumor target located in the image? UltraG-Agent L+US3 Answer: The ovarian tumor target is located in the middle-central part of the image.[SEG] UniBiomed Answer: No findings. No space-occupying lesion was detected in the liver that entered the cross-sectional area.[SEG] GPT 5.4 Answer: The ovarian tumor target is located in the central to right-upper portion of the ovary, appearing as a hypoechoic (dark) cystic or solid lesion within the ovarian parenchyma.[NO_SEG] MedGemma-27B Answer: The ovarian tumor target is located in the lower part of the image, within the uterus, near the cervix.[NO_SEG] Lingshu-32B Answer: The ovarian tumor target is located in the right ovary.[NO_SEG] Ovarian Evidence-grounded VQA Case Study Figure 9: Ovarian Evidence-grounded VQA Case Study. anatomy, but adds unsupported attributes such as normal morphology and valve function without corresponding pixel- level evidence. Overall, this case shows that evidence-grounded report generation remains particularly challenging when multiple anatomical targets must be jointly identified, localized, and described, and that errors in target planning can propagate to the final grounded report. 25 Running Title for Header C: _studies_max_coverage_v3 _model_c overage \02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Reference Answer: [Findings] The prostate is visible in the middle-central image region. [Findings] The prostate center lies in the middle-central image region. [Findings] Its visible extent is approximately 695.0 x 486.0 pixels. [Findings] The visible prostate area occupies approximately 20.36% of the image. [Impression] Visible prostate gland target for segmentation. Q:Generate a structured prostate ultrasound description. Prostate Evidence-grounded Report Generation Case Study UltraG-Agent I+US3 Answer: [Findings] The prostate is visible in the middle-central location. [Findings] The center of the prostate lies in the middle-central location. [Findings] The prostate measures 625 x 475 pixels. [Findings] The prostate occupies 17.78% of the image area. [Impression] The prostate is identified as the visible segmentation target. UniBiomed Answer: [Findings] nonspecific. [Findings] the lesion area is nonspecific. [Findings] [SEG] [Findings] nonspecific. the lesion area is nonspecific. Qwen3VL-32B Answer: [grounding_scope] The image displays a grayscale ultrasound view of a pelvic structure consistent with the prostate. [organ_presence] Prostate is present in the image. [organ_extent] The prostate extends from the upper left to the lower right of the image, occupying the central region. [organ_location] The prostate is located centrally within the pelvic field, with the right lobe appearing more prominent. [organ_size] The prostate spans approximately 750 pixels in width and 1350 pixels in height. [segmentation_impression] The prostate exhibits heterogeneous echotexture with areas of increased and decreased echogenicity. Lingshu-32B Answer: [Findings] The prostate gland is visualized in the midline, posterior to the pubic symphysis. [Findings] The prostate appears enlarged, measuring approximately 4.5 x 3.5 x 3.0 cm in its greatest dimensions. [Findings] The echotexture of the prostate is heterogeneous, with areas of increased echogenicity suggestive of prostatic hyperplasia. [Findings] No focal lesions or masses are identified within the prostate gland. [Findings] The surrounding periprostatic tissue appears unremarkable. [Findings] The seminal vesicles are not clearly visualized in this image. [Findings] The bladder wall appears smooth and without thickening. [Findings] No evidence of hydronephrosis or hydroureter is noted in this view. [NO_SEG] Figure 10: Prostate Evidence-grounded Report Generation Case Study. 26 Running Title for Header C: _studies_max_coverage_v3 _model_c overage \02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Reference Answer: [Findings] 1 visible liver lesion target is centered in the middle-central region. [Findings] The supplied category labels this liver lesion as malignant. [Findings] The largest labeled target is the liver lesion with a pixel extent of approximately 500.0 x 353.0 pixels. [Assessment] The liver ultrasound targets are: liver lesion. [Impression] Visible liver ultrasound targets: liver lesion. Q:Generate a structured liver ultrasound description. UltraG-Agent I+US3 Answer: [Findings] The liver lesion is visible in the middle-central image region. [Findings] Its visible extent is approximately 485.0 x 351.0 pixels. [Findings] The visible liver lesion area occupies approximately 15.70%. [Assessment] The available evidence supports liver lesion localization. [Impression] Visible liver lesion target for segmentation. UniBiomed Answer: [Findings] ground glass density. [Findings] in the upper lobe of the right lung, there is an area of increase in density consistent with the change in ground glass density observed in the previous examination. [Findings] ground glass density. in the upper lobe of the right lung, there is an area of increase in density consistent with the change in ground glass density observed in the previous examination. Claude 4.5 Answer: [Organ] Liver visible in longitudinal view. [Parenchyma] Hepatic parenchyma demonstrates homogeneous echotexture without focal lesions. [Liver extent] Liver margins span approximately 480 pixels in vertical extent. [Lesions] No focal lesions detected. [Impression] Homogeneous liver without evidence of focal pathology. GPT 5.4 Answer: [anatomic_summary] Liver visualized in a single ultrasound view. [largest_target_extent] Hepatic parenchyma occupies the central and right- sided image field. [liver lesion_presence_location] No focal liver lesion is identified in this image. [segmentation_impression] Diffuse increased hepatic echogenicity is present. Liver Evidence-grounded Report Generation Case Study Figure 11: Liver Evidence-grounded Report Generation Case Study. 27 Running Title for Header C: _studies_max_coverage_v3 _model_coverage\ vqa\02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Q:Segment the visible liver UniBiomed Abdomen Instruction-guided Segmentation Error Analysis UltraG-Agent I+US3 UltraG-Agent L+US3 Claude 4.5 GPT 5.4 InternVL3.5-8BInternVL3.5-14B Lingshu-7BLingshu-32B LLava-Med MedGemma-4BMedGemma-27BMiniCPM-V-2.6Qwen3VL-32B Qwen3VL-8B Qwen3VL-4B BiomedParse UltraSAM3 Figure 12: Abdomen Instruction-guided Segmentation Error Analysis. 28 Running Title for Header C: _studies_max_coverage_v3 \ max_model_coverage \02_DDTI_coco_DDTI_coco_test_100_vqa _2 .png Ground Truth Reference Answer: Upper Central Q:What is the approximate image- region location of the A-line? UltraG-Agent Q+US3 Answer: upper-central.[SEG] UltraG-Agent L+US3 Answer: The A-line target is located in the middle-central region of the image.[SEG] InternVL3.5-14B Answer: The A-line target is located in the upper central region of the image.[NO_SEG] Qwen3VL-32B Answer: The A-lines are visible as horizontal, equidistant, hyperechoic lines extending from the pleural line, located in the upper portion of the image.[NO_SEG] Lung Evidence-grounded VQA Error Analysis UniBiomed Answer: no findings. no space-occupying lesion was detected in the liver that entered the cross-sectional area.[NO_SEG] Figure 13: Lung Evidence-grounded VQA Error Analysis. 29 Running Title for Header C: _studies_max_coverage_v3 _model_c overage \02_DDTI_coco_DDTI_coco_test_100_vqa_2 .png Ground Truth Reference Answer: [Findings] Supported boundaries are available for left ventricular endocardium, left ventricular epicardium, left atrium. [Findings] The left ventricular endocardium is centered in the middle-central image region. [Findings] The left ventricular epicardium is centered in the middle-central image region. [Findings] The left atrium is centered in the upper-central image region. Q:Generate a structured grounded echocardiography description. UltraG-Agent L+US3 Answer: [Findings] The left ventricle is visible in the upper-central region. [Findings] The long axis of the left ventricle measures 152 pixels, and the short axis measures 127 pixels. [Findings] The left ventricle occupies 4.75% of the image area. [Impression] The left ventricle is the visible segmentation target. UniBiomed Answer: [Findings] no findings. [Findings] no space-occupying lesion was detected in the liver that entered the cross-sectional area. [Findings] [SEG] [Findings] no findings. no space-occupying lesion was detected in the liver that entered the cross-sectional area. MedGemma-27B Answer: [Overall] Echocardiogram obtained. [Chamber] Left ventricle is visualized. [Chamber] Right ventricle is visualized. [Chamber] Left atrium is visualized. [Chamber] Right atrium is visualized. GPT 5.4 Answer: [Findings] The heart is visualized in the ultrasound image. [Findings] The left ventricle appears to be of normal size and shape. [Findings] The right ventricle appears to be of normal size and shape. [Findings] The interventricular septum appears to be intact and of normal thickness. [Findings] The mitral valve appears to be open and functioning normally. Cardiac Evidence-grounded Report Generation Error Analysis Figure 14: Cardiac Evidence-grounded Report Generation Error Analysis. 30

Instant research discovery