Paper deep dive
MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction
Yulin Fu, Junren Wang, Guangjing Yang, Zhangyuan Yu, Wanran Sun, Jiabao Zhou, Jin Yin, Qicheng Lao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 83%
Last extracted: 10/9/2026, 8:38:30 AM
Summary
The paper introduces MedBenchAgent, a multi-agent framework for the systematic automation of medical Vision-Language Model (VLM) benchmark construction. It formulates benchmark creation as 'constrained compilation,' utilizing a Benchmark Intermediate Representation (BIR) to separate specification planning from item instantiation. The framework achieves a Task-Space F1 of 90.9%, outperforming direct and prior-guided induction methods, and demonstrates portability across specialized medical domains.
Entities (22)
Relation Signals (16)
MedBenchAgent â uses â Benchmark Intermediate Representation
confidence 95% ¡ we introduce MedBenchAgent, a multi-agent framework with a Benchmark Intermediate Representation (BIR)
MedBenchAgent â implements â Constrained Compilation
confidence 92% ¡ We formulate benchmark construction as constrained compilation... Based on this formulation, we introduce MedBenchAgent
MedBenchAgent â achieves â Task-Space F1
confidence 90% ¡ MedBenchAgent achieves a Task-Space F1 of 90.9%
MedBenchAgent â consistsof â SpecificationAgent
confidence 88% ¡ Planning implements F plan via SpecificationAgent, TaskPlanningAgent, and MedicalAlignAgent
MedBenchAgent â consistsof â TaskPlanningAgent
confidence 88% ¡ Planning implements F plan via SpecificationAgent, TaskPlanningAgent, and MedicalAlignAgent
MedBenchAgent â consistsof â MedicalAlignAgent
confidence 88% ¡ Planning implements F plan via SpecificationAgent, TaskPlanningAgent, and MedicalAlignAgent
MedBenchAgent â consistsof â GenerationAgent
confidence 85% ¡ GenerationAgent operates at the caseâtask level to instantiate evaluation items
MedBenchAgent â consistsof â AuditAgent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation items within predefined benchmark specifications. We study the broader problem of automatically deriving the specification itself: what to evaluate, which annotations support each task, and how to translate this evidence into reliable evaluation items. We formulate benchmark construction as constrained compilation, in which the benchmark specification is progressively derived from evaluation requirements, heterogeneous annotations, and medical knowledge. Based on this formulation, we introduce MedBenchAgent, a multi-agent framework with a Benchmark Intermediate Representation (BIR) that encodes task definitions, evidence mappings, evaluation protocols, and item specifications across construction stages. MedBenchAgent separates planning, which derives and verifies the specification, from instantiation, which constructs and audits items under the locked specification. MedBenchAgent achieves a Task-Space F1 of 90.9%, outperforming direct task induction (79.2-80.0%) and prior-guided induction (85.1%); 994 of 1,000 sampled items from correctly identified tasks pass human audit. We further demonstrate portability to a specialized medical domain and evaluate twelve VLMs, revealing task- and setting-specific variation obscured by aggregate scores. These results establish constrained compilation as a scalable and auditable framework for medical VLM benchmark construction beyond question generation.
Tags
Links
- Source: https://arxiv.org/abs/2610.11312v1
- Canonical: https://arxiv.org/abs/2610.11312v1
Trouble viewing inline? Open PDF directly â
Full Text
104,832 characters extracted from source content.
Expand or collapse full text
MEDBENCHAGENT: TOWARDS SYSTEMATIC AUTOMA- TION OF MEDICAL VLM BENCHMARK CONSTRUCTION Yulin Fu 1,â ,Junren Wang 2,3,â ,Guangjing Yang 1 ,Zhangyuan Yu 1 ,Wanran Sun 1 , Jiabao Zhou 1 , Jin Yin 2,3 , Qicheng Lao 1 1 Beijing University of Posts and Telecommunications 2 West China Hospital 3 Sichuan Provincial Engineering Research Center of Intelligent Diagnosis and Treatment of Breast Diseases â Equal contribution. yulin.fu, qicheng.lao@bupt.edu.cn ABSTRACT Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation items within predefined benchmark specifications. We study the broader problem of automatically deriving the specification itself: what to evaluate, which annotations support each task, and how to translate this evidence into reliable evaluation items. We formulate benchmark construction as constrained compilation, in which the benchmark specification is progressively derived from evaluation requirements, heterogeneous annotations, and medical knowledge. Based on this formulation, we introduce MedBenchAgent, a multi-agent framework with a Benchmark Inter- mediate Representation (BIR) that encodes task definitions, evidence mappings, evaluation protocols, and item specifications across construction stages. MedBen- chAgent separates planning, which derives and verifies the specification, from instantiation, which constructs and audits items under the locked specification. MedBenchAgent achieves a Task-Space F1 of 90.9%, outperforming direct task induction (79.2â80.0%) and prior-guided induction (85.1%); 994 of 1,000 sampled items from correctly identified tasks pass human audit. We further demonstrate portability to a specialized medical domain and evaluate twelve VLMs, revealing task- and setting-specific variation obscured by aggregate scores. These results es- tablish constrained compilation as a scalable and auditable framework for medical VLM benchmark construction beyond question generation. 1INTRODUCTION Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible as richly annotated imaging datasets provide reusable supervision and large language models (LLMs) enable scalable generation of evaluation items (Zhang et al., 2023; Hu et al., 2024; Chen et al., 2024). Yet a benchmark is more than a collection of images and questions: it must specify what capabilities are evaluated, which source annotations validly support them, how annotation evidence is transformed into reference answers, how evaluation items are instantiated, and how model responses are elicited and assessed. Collectively, we refer to these decisions as the benchmark specification, which in turn determines the claims that the resulting evaluation can support. In medical imaging, establishing such a specification requires interpreting heterogeneous annotation schemas, units, terminology, and label semantics, making manual benchmark design time-consuming and expertise-intensive. Automating the specification itself, rather than only large-scale item generation, is therefore an important problem for scalable and reliable medical VLM evaluation. Existing work approaches automated benchmark construction from two directions. Medical systems construct benchmarks from domain data, but operate within predefined clinical scopes, reporting structures, or question schemas (Xie et al., 2026; Liu et al., 2026a). General-purpose systems offer greater flexibility by using declarative or agentic workflows to translate evaluation requirements into 1 arXiv:2610.11312v1 [cs.AI] 8 Oct 2026 benchmark construction (Li et al., 2025; Xiong et al., 2026; Butt et al., 2025). However, they do not explicitly constrain medical benchmark specification by annotation-supported task validity, reference provenance, and medical knowledge. The remaining challenge, then, is to combine flexible benchmark specification with the semantic and evidential constraints imposed by evaluation requirements and heterogeneous medical data. We address this challenge with MedBenchAgent, a multi-agent framework for systematic medical VLM benchmark construction. We cast benchmark construction as constrained compilation, in which the benchmark specification is progressively established from an intended evaluation scope, heterogeneous source annotations, and medical knowledge, with each confirmed decision narrowing the space of subsequent construction. To make these constraints explicit and preserve them across stages, we introduce a Benchmark Intermediate Representation (BIR) that records the evaluation scope, task hierarchy, annotation evidence mappings, evaluation protocols, sample allocations, and item-construction specifications. Together, constrained compilation and the BIR provide an explicit mechanism for carrying semantic and evidential constraints from benchmark specification into downstream construction. MedBenchAgent operationalizes this formulation through a staged multi-agent workflow for medical benchmark construction: 1) planning resolves and verifies the specification against dataset annotations and medical references before locking the BIR; and 2) instantiation constructs and audits evaluation items under this fixed specification. In this way, MedBenchAgent extends automation from predefined medical scopes or question structures to benchmark specification itself, while preserving the annotation and medical constraints required for reliable construction. This formulation enables empirical evaluation of both the benchmark-construction process and the benchmark produced by MedBenchAgent. In construction analysis, MedBenchAgent achieves a Task-Space F1 of 90.9%, compared with 79.2â80.0% for direct task induction and 85.1% when the same task prior is provided to the direct model. For correctly identified tasks, 994 of 1,000 sampled items pass human audit. Together, these results suggest that open-ended decisions about what should be evaluated are substantially more challenging than evidence mapping and item construction once benchmark semantics are constrained. We further instantiate the workflow in a specialized medical domain under a different annotation schema and evaluate twelve VLMs on the resulting benchmark. Model evaluation reveals substantial variation across tasks, imaging settings, and evaluation metrics that aggregate scores can obscure, reinforcing fine-grained benchmark specification as an important component of medical VLM evaluation. Overall, this work makes three contributions: (1) Conceptually, we model the construction of medical VLM benchmarks and formulate it as constrained compilation; (2) Methodologically, we introduce a multi-agent MedBenchAgent framework and its Benchmark Intermediate Representation (BIR); and (3) Empirically, we derive insights into both the benchmark-construction process and medical VLM evaluation, while providing reusable benchmark artifacts. 2RELATED WORK Medical VLM benchmarks. Medical VLM benchmarks have progressed from manually authored questionâanswer datasets such as VQA-RAD and SLAKE (Lau et al., 2018; Liu et al., 2021), to scal- able reuse of imageâcaption and classification supervision in PMC-VQA and OmniMedVQA (Zhang et al., 2023; Hu et al., 2024), and more recently to task-structured evaluation over heterogeneous datasets in GMAI-MMBench and MMBU (Chen et al., 2024; DâCunha et al., 2026). Automated benchmark construction. General-purpose benchmark automation has increasingly adopted declarative or agentic construction, whereas medical systems have favored domain-specific, more deterministic workflows. AutoBencher constructs benchmarks from specified evaluation desider- ata, BenchmarkAgent translates high-level evaluation requirements into construction procedures, and BenchAgents coordinates multiple agents across benchmark-creation stages (Li et al., 2025; Xiong et al., 2026; Butt et al., 2025). In the medical domain, EHRBench uses a task-specific template pipeline to instantiate clinical QA from EHR data (Xie et al., 2026), while Liu et al. (2026a) packages schema- and report-grounded oncology VQA generation as a reusable agent skill. 2 3WHAT MAKES A GOOD MEDICAL VLM BENCHMARK? 3.1BENCHMARK FORMULATION A medical VLM benchmark must specify both what is evaluated and how model performance is assessed. Building on medical benchmarking frameworks that distinguish the evaluation objective, challenge data, reference annotations, and assessment procedures (Karargyris et al., 2023; Maier-Hein et al., 2020), and following recent task-oriented medical VLM benchmarks, we additionally specify the task definitions and queries presented to the model for visionâlanguage evaluation. We represent a benchmark by its intended evaluation scopeU, task spaceT, evaluation-item setI, and evaluation protocolP . A medical VLM benchmark is therefore defined as B = (U,T ,I,P), I =Κ i = (x i ,t i ,q i ,r i ) N i=1 ,(1) wherex i is a medical imaging case,t i âTthe evaluated task,q i the model-facing query, andr i the benchmark reference;P specifies each taskâs response convention and scoring rule. Because eachr i is a benchmark-level reference, not a raw annotation copied directly from a source dataset, its provenance and semantics must be recorded. The CLAIM 1 guideline requires medical imaging studies to document the source, definition, and acquisition procedure of the reference standard (Tejani et al., 2024). We represent the source datasets as D =D m M m=1 , D m = (X m ,A m , ÎŁ m ),(2) whereX m is the set of imaging cases in datasetm,A m is its annotation collection, andÎŁ m specifies the annotation schema, annotation unit, granularity, label space, and field semantics. An annotation recordaâA m represents evidence encoded under the dataset-specific specificationÎŁ m ; it becomes a benchmark referencer i only after its semantics have been matched to a taskt i and mapped into the common benchmark representation. Benchmark construction must therefore determine not only whether the available annotations support a proposed task, but also how the relevant annotation evidence should be interpreted as its reference answer. We therefore abstract benchmark construction as F : (D,K,U )7âB,(3) whereKdenotes the medical knowledge required to interpret clinical concepts and evaluation semantics. Once constructed, the benchmark can be applied to a model setMto produce empirical evidenceE:(B,M) ââ E. This separates benchmark construction, which defines the evaluation object, from benchmark application, which produces evidence about model behavior under the specified scope and protocol. We use benchmark specification to denote the pre-instantiation decisions that determine the evaluation scope, dataset-specific task definitions and granularity, supporting evidence and reference construction, evaluation protocol, sample allocation, and item-construction rules. The concrete item setI is produced by instantiating this specification over the source cases. 3.2DESIDERATA FOR BENCHMARK CONSTRUCTION The formulation above motivates three desiderata for medical VLM benchmark construction. D1: Evidence Groundedness.Because benchmark construction maps source annotations to evaluation tasks and reference answers, annotation availability alone is insufficient to justify either output. Evidence groundedness requires that each task be supported by the schema and semantics of its source annotations and that each item-level referencer i be entailed by the annotation evidence used to construct it. SLAKE uses physician-provided annotations, while EHRXQA uses templates and executable queries (Liu et al., 2021; Bae et al., 2023). D2: Medical Alignment.Annotation support establishes that a task is constructible, but does not determine how it should be expressed under the intended medical evaluation scope. Medical alignment requires task definitions, response spaces, and queries presented to the model to be consistent with established medical concepts and terminology, and not merely to verbalize labels to one dataset. GMAI-MMBench reflects this desideratum by organizing evaluation around clinically defined tasks (Chen et al., 2024). 1 Checklist for Artificial Intelligence in Medical Imaging 3 ¡ Align Item Specifications with Medical Knowledge ¡ Instantiate & Review Items ¡ Audit & Finalize Benchmark Input Resources Five-Role Orchestration Planning Instantiation User Requirement ¡ Natural Language Datasets SpecificationAgent TaskPlanningAgent MedicalAlignAgent GenerationAgent AuditAgent Medical References Guideline Standard ... ... ¡ Specify Evaluation Scope ¡ Determine Supported Tasks A B C D A B C D filter support Supported Tasks metric & sample F1 80 ... ... ... ... ¡ question formulation ¡ response space ¡ reference normalization Medically Grounded Item Design mIoU 120 Extract Scope Requirements Derive case-task pairs Review Sample Repeat Ă Generate reference answer query ¡ medical domain ¡ evaluation objective ¡ question language ¡ benchmark size Evaluation Scope ... Retrieval & Rerank RAG item benchmark items Benchmark raw benchmark items Quality Audit Filtering & Rebalancing Tasks Lock BIR Dataset Annotations evaluation scope task space evidence mapping evaluation protocol sample allocation item-construction specification BIR ... ... Figure 1: Overview of MedBenchAgent. MedBenchAgent coordinates five agents through a shared Benchmark Intermediate Representation (BIR). Planning specifies tasks and their constraints within the BIR, whereas instantiation constructs evaluation items under the locked specifications. D3: Evaluative Utility.The application(B,M) â Efurther implies that evidence-grounded and medically aligned items do not necessarily yield informative evidence about model behavior. Evaluative utility requires the benchmark to support separate assessment of model capabilities and preserve interpretable differences across tasks and imaging settings. OmniMedVQA, MultiMedEval, and GMAI-MMBench exemplify this desideratum through evaluation across multiple datasets, tasks, modalities, and model families (Hu et al., 2024; Royer et al., 2024; Chen et al., 2024). 3.3BENCHMARK CONSTRUCTION AS CONSTRAINED COMPILATION We therefore formulate benchmark construction as constrained compilation: planning progressively establishes the benchmark specification, and each confirmed decision restricts the admissible choices of subsequent stages within the mappingF. The evaluation scope constrains the relevant task categories; annotation semantics determine which concrete prediction targets can be instantiated within them and which evidence can support their references; medical knowledge and evaluation requirements further constrain task expression and assessment. This process requires an explicit intermediate state that preserves confirmed decisions across stages. Section 4 instantiates this state as the BIR and locks it before item construction. 4MEDBENCHAGENT Figure 1 summarizes the five-agent workflow and the separation between planning and instantiation, while Algorithm 1 formalizes the corresponding BIR updates and stage-wise construction procedure. 4.1INPUTS AND BENCHMARK INTERMEDIATE REPRESENTATION MedBenchAgent takes three benchmark-specific resources as input: a natural-language user require- ment, medical imaging datasets, and medical reference materials. Each dataset-specific BIR contains a task-category priorT 0 , represented as an unexpanded task tree. The user requirement udefines the intended evaluation scopeUand specifies additional user constraints on benchmark construction. The source collectionD = D m M m=1 consists of medical imaging cases and their dataset-specific annotations. The medical knowledge resourceKconsists of clinical guidelines, reporting standards, and other domain references relevant to the target setting. The initial task definitions inT 0 are distilled from the task-design principles of existing medical VLM benchmarks and specify candidate tasks and their annotation-support conditions. 4 Section 3 formulates benchmark construction asF : (D,K,U )7âB. MedBenchAgent implements this transformation through two coupled stages: Z = F plan (D,K,U ;T 0 ), I = F inst (D,Z),(4) whereZ denotes the benchmark design produced during planning. For each source datasetD m , planning maintains a dataset-specific Benchmark Intermediate Rep- resentation (BIR). After the evaluation scope is confirmed, its task-space component is initialized as the unexpanded task treeT 0 . Candidate nodes are retained, removed, or expanded according to their support conditions, the evaluation scope, and available annotation evidence. Planning completes their evidence mappings, evaluation protocols, and sample allocations, while task design supplies the item-construction specifications. The resulting state is then confirmed and locked: BIR (0) m = (U,T 0 , 0, 0, 0, 0),BIR lock m = (U,T m , Î m ,P m ,n m , Ψ m ).(5) Here,T m is the dataset-specific task tree obtained by retaining supported nodes and expanding decomposable tasks into distinct prediction targets. Ordinary tasks remain leaf nodes, whereas decomposable tasks have question types as their children. Each child must remain within its parentâs task semantics and have annotation support for its own prediction target. The middle four components jointly describe this tree and its node specifications:Î m (t)records supporting annotation evidence, P m (t)specifies the evaluation protocol, andn m (t)denotes the sample allocation associated with task nodet. These specifications are completed for the retained atomic tasks during planning; the zero entries in the initial BIR denote components not yet populated for the dataset.Ψ m (t)contains the corresponding medically grounded item-construction specification. We writeLeaves(T m )for the retained atomic evaluation tasks, so a decomposable parent is not counted in addition to its children. The planning output is thereforeZ =BIR lock m M m=1 . EachBIR lock m encodes the confirmed dataset- level benchmark specification, while instantiation produces the corresponding audited item setI m . Dataset-specific leaf tasks, item sets, and protocols are then consolidated across datasets, with T = S M m=1 Leaves(T m ), to form the final benchmarkB = (U,T ,I,P). 4.2PLANNING: COMPILING AND LOCKING THE BIR Planning implementsF plan via SpecificationAgent, TaskPlanningAgent, and MedicalAlignAgent, which structure requirements, determine supported tasks, and design task items, respectively. The CKPT markers in Algorithm 1 denote critical verification checkpoints at consequential planning transitions, validating the proposed BIR state before commitment through deterministic checks, human review, model-based supervision, or hybrid mechanisms. Algorithm 1 Medical VLM benchmark construction Require: User requirement u, source datasetsD, medical referencesK Ensure: BenchmarkB = (U,T ,I,P) 1: U CKPT ââSPECIFY(u) ;BIR (0) m = (U,T 0 , 0, 0, 0, 0) M m=1 ; I,I raw,m ââ , âm 2: for allD m âD do 3:(T m , Î m ,P m ,n m ) CKPT ââ ASSESSTASKSUPPORT(D m , BIR (0) m ) 4:BIR m â (U,T m , Î m ,P m ,n m , 0) 5:Ψ m CKPT ââ DESIGNTASKITEMS(K,U,T m , Î m ,P m ) 6:BIR lock m â (U,T m , Î m ,P m ,n m , Ψ m ) 7:for all eligible caseâtask pairs (x i ,t) with tâ Leaves(T m ) do 8:e i â EXTRACTEVIDENCE(x i , Î m (t)) 9:r i â DERIVEREFERENCE(e i , Ψ m (t)) 10:q i â GENERATEQUERY(x i ,t,e i , Ψ m (t)) 11:Κ i â (x i ,t,q i ,r i ) 12:ifREVIEWITEM(Κ i ,e i , BIR lock m )thenI raw,m âI raw,m âŞÎš i 13:end for 14: I m â AUDIT(I raw,m , BIR lock m ,D m ) 15: I âIâŞI m 16: end for 17: T â M [ m=1 Leaves(T m ); P â MERGEPROTOCOLS(P m M m=1 ) 18: returnB = (U,T ,I,P) SpecificationAgent first converts the open-ended natural-language require- mentuinto a structured evaluation scopeU. The specification resolves ambiguous dimensions of the re- quested benchmark before constrain- ing subsequent planning. TaskPlanningAgent then extends the task-category prior into the dataset- specific task treeT m using the evalu- ation scope and annotation evidence inD m . A category is retained only when its support conditions are satis- fied, andÎ m records the eligible an- notation fields to construct references for its tasks. When annotations ex- pose multiple distinct prediction tar- gets within a supported task, the agent expands it into child tasks, each rep- resenting an annotation-supported an- swer axis within parent semantics. Semantic duplicates are removed across retained atomic tasks, 5 Tasks Datasets Annotations Finding Categories Lesion Characteristics Bounding Box Histology Pathology Status VinDr-Mammo (8,684) BUS-BRA (5,000) BrEaST (2,564) BI-RADS (4,006) Breast Composition (2,700) (2,455) (1,731) (3,708) (1,371) (1,506) A (2,700) FT (1,226) FA (1,731) AC (4,006) DX (1,371) PS (1,506) Det (2,479) VG (1,229) A â Anatomy Attribute Query: What is the breast tissue composition as seen in this ultrasound image? Answer: Homogeneous: fibroglandular Query: What is the breast tissue composition as seen in this ultrasound image? FT â Finding Type Query: What type of finding does this mammogram show? Answer: Mass FA â Finding Attribute Query: What is the shape of the mass observed on the ultrasound? Answer: Irregular AC â Assessment Category Query: Which BI-RADS category applies to this breast ultrasound finding? Answer: BI-RADS 3: Probably Benign DX â Diagnosis Query: What histologic diagnosis is most consistent with this ultrasound finding? Answer: Cyst PS â Pathology Status Query: Classify this breast lesion seen on ultrasound: Answer: Benign Det â Detection Query: Where is the suspicious region located in this mammogram? Please draw a bounding box around it. Reference: Bounding box VG â Visual Grounding Query: Can you draw a bounding box around the Mass in this mammogram? Reference: Bounding box Figure 2: Benchmark construction and representative items. Three source datasets are mapped through heterogeneous annotations to the retained evaluation tasks, with one representative item shown for each task. Numbers in parentheses denote item counts. establishing the task support required by Desideratum D1. TaskPlanningAgent selects a compatible primary evaluation metric for each retained atomic task. The resulting task plan passes through CKPT before commitment to the BIR, after which deterministic allocation assignsn m . Preserving prediction targets provides the basis for separate task-level analysis under Desideratum D3. MedicalAlignAgent subsequently converts each retained task into a medically meaningful and exe- cutable item-construction specificationΨ m . It adopts retrieval-augmented generation (RAG) (Lewis et al., 2020) over the relevant medical references inKto ground the design in medical knowledge. Grounded on these references and constrained by the retained task semantics and authorized anno- tation evidence, it specifies the taskâs question semantics, response space, and reusable question formulations, together with the normalization to benchmark-level reference answers. This step connects an evidence-supported task to its medical evaluation semantics, thereby supporting Desider- atum D2. After the complete set of task designs passes through CKPT, the resultingBIR lock m is fixed for downstream instantiation. Downstream stages may read the confirmed task definitions, supporting evidence, protocols, sample allocations, and construction specifications, but cannot revise them. 4.3INSTANTIATION: CONSTRUCTING ITEMS UNDER THE LOCKED BIR Instantiation realizesF inst in Eq. 4 by applyingBIR lock m to the source annotations without reopening the confirmed benchmark design. As summarized in Algorithm 1, GenerationAgent operates at the caseâtask level to instantiate evaluation items, whereas AuditAgent operates over the resulting item set to verify benchmark-level consistency. For each eligible caseâtask pair witht i â Leaves(T m ), GenerationAgent constructs an evaluation itemΚ i = (x i ,t i ,q i ,r i )under the corresponding task specificationΨ m (t i ). The model generates only the queryq i , varying its wording within the locked task semantics and authorized annotation evidence. Deterministic code derives the reference answerr i from the supporting evidence specified byÎ m (t i ) and, where applicable, organizes the response options according to the confirmed construction specification. A subsequent model review call screens the generated item for task drift, ambiguity, unsupported additions, and answer leakage without modifying the fixed reference. This separation allows linguistic variation in the query while preserving annotation-grounded reference answers and the item-level entailment requirement of Desideratum D1. AuditAgent subsequently checks the generated item set for consistency with the locked task-node specifications, source annotations, and benchmark-level requirements. Deterministic checks verify annotation support, response and reference consistency, and exact duplication, while accepted multiple-choice items are balanced with respect to option positions. AuditAgent additionally performs task-stratified consistency screening to identify issues such as task drift, answer leakage, ambiguity, and semantic duplication. The items passing the deterministic audit constitute the finalized evaluation- item setI and complete the resulting benchmarkB = (U,T ,I,P). 6 Table 1: Task-space construction and evidence-mapping results. Task-Space Precision (P), Recall (R), and F1 mea- sure agreement with researcher review; EMA measures agreement of supporting annotation fields (%). Task-Space ReviewEvidence Model TPFPFNPâRâF1âEMAâ DeepSeek-V4-Flash198270.4 90.5 79.2100.0 Qwen3.5-9B198270.4 90.5 79.2 100.0 MedGemma-27B209169.0 95.2 80.0100.0 DeepSeek-V4-Flash +T 0 206176.9 95.2 85.1 100.0 MedBenchAgent203187.0 95.2 90.9100.0 Table 2: Human consistency audit of finalized items. Pass rates are com- puted over 1,000 items sampled uni- formly from the finalized benchmark. CriterionPassFailPass Rate Reference groundedness1,0000100.0% Taskâquery alignment996499.6% Unambiguity1,0000100.0% Evaluation integrity994699.4% All criteria994699.4% 5EXPERIMENTS: BENCHMARK CONSTRUCTION AND EVALUATION 5.1EXPERIMENTAL SETUP Construction. Figure 2 summarizes the resulting benchmark construction. We instantiate Med- BenchAgent using three medical imaging datasets: VinDr-Mammo (Nguyen et al., 2023), BrEaST (PawĹowska et al., 2024), and BUS-BRA (G Ě omez-Flores et al., 2024). The medical knowledge resource is grounded in the relevant sections of the ACR BI-RADS v2025 Manual (Destounis et al., 2025; Leung et al., 2025). For the reported runs, we instantiate the planning checkpoints with researcher review. Evaluation. We evaluate twelve VLMs spanning general-purpose and medical-domain model families, multiple parameter scales, and different inference regimes. The general-purpose models include InternVL3-8B (Zhu et al., 2025), Qwen2.5-VL (7B, 32B) (Bai et al., 2025), and Qwen3-VL (8B, 32B), including an 8B thinking variant (Qwen Team, 2025). The medical models include MedGemma-4B (Google Research & Google DeepMind, 2025), Lingshu (7B, 32B) (LASA Team et al., 2025), Hulu-Med-7B (Jiang et al., 2025), QoQ-Med-7B (Dai et al., 2025), and BreastGPT- 8B (Liu et al., 2026b). Multiple-choice tasks are evaluated with Accuracy (Acc.) and Macro-F1 (F1), while spatial tasks use mean intersection over union (mIoU) andAcc@IoU 0.5 . When a task contains multiple fine-grained tasks, metrics are computed separately and macro-averaged with equal weight. Implementation. We host DeepSeek-V4-Flash (DeepSeek-AI, 2026) as the language-model back- end with vLLM (Kwon et al., 2023), and Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B (Zhang et al., 2025) for retrieval and reranking. Model evaluation uses deterministic decoding and FlashAttention-2 (Dao, 2024), and all experiments are run on NVIDIA B200 GPUs. 5.2BENCHMARK CONSTRUCTION ANALYSIS Task-space construction. MedBenchAgent uses BIR-mediated task planning to constrain task- space construction withUandÎŁ m , rather than inducing the task space in a single open-ended step. We evaluate this strategy against direct baselines using the same inputs in a single model call, and additionally provide DeepSeek-V4-Flash with the initial task definitionsT 0 . The direct baselines use MedGemma-27B (Google Research & Google DeepMind, 2025), Qwen3.5-9B (Qwen Team, 2026), and DeepSeek-V4-Flash. We evaluate against a reference task spaceT â , defined before MedBenchAgent and finalized prior to these experiments by dataset-familiar researchers with informal input from a clinical collaborator. Proposals are matched toT â by prediction target and annotation semantics rather than surface wording. We report Task-Space Precision, Recall, and F1 from TP, FP, and FN counts; for matched tasks, Evidence-Mapping Agreement (EMA) measures agreement with researcher-confirmed supporting fields. As shown in Table 1, BIR-mediated construction achieves 90.9% Task-Space F1, compared with 79.2â80.0% for direct induction. ProvidingT 0 improves direct DeepSeek-V4-Flash induction to 85.1% F1, while MedBenchAgent further increases precision from 76.9% to 87.0% at the same 95.2% recall, reducing false positives from six to three. All methods achieve 100% EMA on matched tasks. These results show that the BIR-mediated design primarily improves the selection of valid prediction targets, while supporting-field identification is reliable once the task is correctly identified. 7 Table 3: Evaluation performance on tasks with choices. Results are reported as Accuracy / Macro-F1 (%).â denotes thinking variants. Task abbreviations (A, FT, etc.) are defined in Figure 2. AnatomyFindingAssessment & Diagnosis Model AAFTFAACDXPS Overall General-purpose VLMs InternVL3-8B36.59 / 17.3019.09 / 8.6542.28 / 30.8616.12 / 12.8235.35 / 22.8648.95 / 40.2533.06 / 22.12 Qwen2.5-VL-7B48.86 / 23.0829.77 / 10.1741.45 / 29.2821.35 / 11.9333.76 / 22.6751.03 / 41.8637.70 / 23.16 Qwen3-VL-8B-I34.89 / 18.5616.31 / 9.0329.84 / 22.5739.11 / 19.7439.98 / 29.6560.55 / 47.8536.78 / 24.57 Qwen3-VL-8B-T â 25.52 / 14.1815.09 / 7.2956.93 / 35.1338.58 / 20.0335.13 / 24.1553.05 / 42.5237.38 / 23.88 Qwen2.5-VL-32B44.68 / 21.5954.00 / 12.6746.20 / 29.5526.83 / 16.7943.26 / 26.6952.92 / 42.4144.65 / 24.95 Qwen3-VL-32B37.31 / 20.4754.65 / 13.2537.08 / 28.4234.22 / 20.5939.37 / 29.8359.37 / 54.2843.67 / 27.81 Medical VLMs MedGemma-4B22.39 / 15.6923.16 / 10.8447.55 / 33.9417.98 / 15.3830.20 / 23.7555.78 / 44.3232.84 / 23.99 Lingshu-7B44.27 / 26.1332.22 / 12.2856.84 / 38.6625.28 / 20.8552.18 / 29.2971.21 / 52.5347.00 / 29.96 Hulu-Med-7B31.62 / 16.6317.05 / 9.1951.14 / 35.4922.75 / 18.8149.64 / 34.7064.14 / 51.5939.39 / 27.73 QoQ-Med-7B â 48.56 / 22.7253.83 / 12.0366.09 / 37.8520.43 / 10.2441.45 / 24.0742.14 / 30.4445.42 / 22.89 BreastGPT-8B36.03 / 26.4135.73 / 16.9675.22 / 52.5040.72 / 31.5465.09 / 47.0678.58 / 71.3055.23 / 40.96 Lingshu-32B50.89 / 24.7042.66 / 14.1757.54 / 43.0134.19 / 21.6549.04 / 24.5470.08 / 55.2550.73 / 30.55 Family-level Summary General-purpose Avg.37.98 / 19.1931.49 / 10.1842.30 / 29.3029.37 / 16.9837.81 / 25.9754.31 / 44.8638.87 / 24.41 Medical Avg.38.96 / 22.0434.11 / 12.5859.06 / 40.2426.89 / 19.7447.93 / 30.5763.65 / 50.9045.10 / 29.35 Human audit of finalized items. MedBenchAgent promotes consistent item construction by locking item-construction requirements in the confirmed specification before instantiation. To assess whether these constraints are preserved in the finalized benchmark, we sample 1,000 items without replacement and audit each for reference groundedness, taskâquery alignment, unambiguity, and evaluation integrity against its annotation evidence and specification. An item passes only when all four criteria are satisfied. Table 2 shows that 994 items pass, yielding a 99.4% consistency rate. Six failures include four VinDr-Mammo finding-type questions that ask for BI-RADS assessment categories and two BrEaST diagnosis questions with unsupported additions. These cases localize remaining errors to template semantics and evidence control during generation, distinguishing item-expression errors from the task-specification errors identified in the task-space analysis. The high consistency rate supports locked specifications for large-scale item construction while identifying targets for semantic verification. Detailed setups and additional analyses are provided in Appendix B. 5.3MODEL EVALUATION Table 4: Spatial localization performance. Re- sults are reported as mIoU / Acc@IoU 0.5 (%). VinDr-MammoBUS-BRA Model DetVGDet General-purpose VLMs InternVL3-8B3.30 / 0.243.15 / 0.3326.00 / 11.20 Qwen2.5-VL-7B7.60 / 3.918.77 / 6.5128.94 / 22.24 Qwen3-VL-8B-I10.72 / 7.0013.68 / 10.9851.11 / 56.40 Qwen3-VL-8B-T â 10.94 / 7.3210.42 / 7.0037.50 / 32.88 Qwen2.5-VL-32B9.70 / 6.1010.83 / 7.3240.63 / 37.52 Qwen3-VL-32B15.40 / 12.8618.01 / 16.8455.36 / 62.08 Medical VLMs MedGemma-4B1.54 / 0.001.57 / 0.0015.73 / 3.68 Lingshu-7B4.41 / 0.813.65 / 0.654.71 / 0.88 Hulu-Med-7B0.01 / 0.000.00 / 0.006.97 / 2.96 QoQ-Med-7B â 6.97 / 1.066.97 / 1.3045.63 / 43.52 BreastGPT-8B5.34 / 0.905.44 / 1.1472.62 / 82.80 Lingshu-32B4.70 / 1.465.62 / 1.9512.46 / 4.40 Task-oriented specification preserves non- redundant evaluation evidence. MedBenchA- gent retains annotation-supported prediction tar- gets as separate tasks and records their evalua- tion protocols in the BIR. We test whether these choices affect model assessment beyond increas- ing benchmark granularity. As shown in Ta- ble 3, medical VLMs outperform general-purpose VLMs by 6.23 points in Overall Acc. but trail by 2.48 points on AC, indicating that the aggregate advantage is not uniform across tasks. To quan- tify such changes, we compare all 66 model pairs and find that 12â24 pairs (18.2â36.4%) reverse or- dering between Overall Acc. and individual-task Acc.; within tasks, 4â15 pairs (6.1â22.7%) reverse between Acc. and F1. MedBenchAgent also preserves localization annotations as spatial prediction targets rather than converting them into choice-based questions. For example, Lingshu-7B has higher Overall Acc. than Qwen2.5-VL-32B (47.00% vs. 44.65%), while Qwen2.5-VL-32B performs better on all six spatial measures in Table 4. Across these measures, 28â34 of 66 model pairs (42.4â51.5%) reverse 8 Table 5: Multiple-choice performance under the blurred-image intervention. Results are reported as Acc. / F1 (%). â Overall denotes the original-to-blurred score drop (p) AnatomyFindingAssessment & DiagnosisOverallâ Overall Model AAFTFAACDXPSAcc/F1Acc/F1 Random Guess25.00 / 20.6412.50 / 8.0236.90 / 30.9519.76 / 16.8425.00 / 22.3941.67 / 37.8326.81 / 22.78â MedGemma-4B11.56 / 11.588.24 / 5.8839.06 / 27.991.83 / 3.3821.06 / 16.7340.44 / 36.5920.36 / 17.0212.48 / 6.97 Lingshu-7B26.78 / 19.6016.48 / 7.0355.03 / 27.9632.85 / 16.0835.98 / 18.7461.92 / 32.2838.17 / 20.288.83 / 9.68 Hulu-Med-7B19.06 / 16.625.71 / 4.6954.83 / 28.0731.07 / 15.8938.27 / 22.0152.23 / 41.1233.53 / 21.405.86 / 6.33 BreastGPT-8B36.66 / 18.0042.01 / 11.4858.54 / 28.8120.12 / 11.3730.70 / 18.4935.55 / 21.9737.26 / 18.3517.97 / 22.61 ordering relative to Overall Acc. Thus, rankings vary substantially across evaluation forms. These results support Desideratum D3: preserving annotation-supported tasks across capability dimensions and assigning task-appropriate metrics materially changes the evidence provided by the benchmark, motivating decomposition of aggregate results by prediction target to reveal capability differences obscured by Overall summaries. Additional results are provided in Appendix D. Annotation grounding provides a model-independent reference layer. MedBenchAgent treats the annotations associated with each source image as the reference standard. This design places trust in clinician annotations rather than external model judgments, avoiding an additional source of uncertainty. To test this strategy, we replace every evaluation image with a Gaussian-blurred image while keeping all non-image benchmark components unchanged. Across four medical VLMs, Acc. decreases by 5.86â17.97 percentage points (p) and F1 by 6.33â22.61 p (Table 5). The consistent degradation indicates that benchmark performance remains sensitive to visual evidence from the corresponding image, despite references being derived directly from source annotations rather than model-based visual judgments. 5.4CROSS-DOMAIN CASE STUDY To assess the cross-domain portability of MedBenchAgent, we apply the complete construction workflow under a specialized medical setting using ISIC 2018 (Codella et al., 2019; Tschandl et al., 2018), with a distinct dataset and annotation schema. Qwen2.5 7B Qwen3 32B MedGemma 4B Lingshu 32B 0 20 40 60 80 100 Score (%) (a) Finding Attribute Accuracy Macro-F1 Qwen2.5 7B Qwen3 32B Lingshu 32B MedGemma 4B (b) Diagnosis Accuracy Macro-F1 (c) Detection Acc@IoU 0.5 mIoU Qwen2.5 7B Qwen3 32B Lingshu 32B MedGemma 4B Figure 3: Cross-domain instantiation on dermoscopy. The resulting task space and represen- tative VLM results are summarized in Figure 3. Representative VLMs show different relative strengths across pre- diction targets rather than a uniform performance pattern. This indicates that the same workflow can instantiate a domain-appropriate task structure under a distinct annotation schema These results provide cross-domain evidence for the portability of MedBenchAgent across annotation semantics and task definitions. Additional details are provided in Appendix C. 6CONCLUSION We introduced MedBenchAgent, a multi-agent framework that treats medical VLM benchmark construction as constrained compilation and makes benchmark specification an explicit, auditable object of construction. Through the Benchmark Intermediate Representation (BIR), MedBenchA- gent separates specification planning from downstream item instantiation, allowing construction decisions to be evaluated stage by stage rather than only through the final benchmark. Our analy- ses identify open-ended task-space specification as the primary source of construction uncertainty, while annotation-evidence mapping and item instantiation become substantially more reliable once benchmark semantics are fixed. Evaluation of the resulting benchmark further reveals task-, metric-, and imaging-dependent variation across VLMs that aggregate scores can obscure. Together with the cross-domain instantiation under a specialized medical setting, these results show that con- 9 strained, multi-agent benchmark construction can support both scalable medical VLM evaluation and systematic study of the benchmark-construction process itself. Limitations and Future Directions. While MedBenchAgent moves medical benchmark construction towards greater automation, limited human involvement remains at planning checkpoints; we will further reduce this reliance by introducing supervisory agents to verify intermediate specifications. More broadly, MedBenchAgent provides a basis for comparing controlled benchmark compositions and for constructing larger collections of disease-, anatomy-, and modality-specific benchmarks. REFERENCES Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, and Edward Choi. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, p. 3867â3880. Curran Associates, Inc., 2023. doi: 10.52202/075280-0170. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502.13923, 2025. doi: 10.48550/ARXIV.2502.13923. Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, and Vidhisha Balachandran. Benchagents: Automated benchmark creation with agent interaction. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025. Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37: 94327â94427, 2024. Noel C. F. Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen W. Dusza, David A. Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael A. Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC). CoRR, abs/1902.03368, 2019. URL http://arxiv.org/abs/1902.03368. David Dai, Peilin Chen, Chanakya Ekbote, and Paul Liang. Qoq-med: Building multimodal clinical foundation models with domain-aware grpo training. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, p. 37406â37453. Curran Associates, Inc., 2025. doi: 10.52202/085 713-1256. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, p. 35549â35562, 2024. Ryan DâCunha, Alejandro Lozano, Xiaoxiao Sun, Daniel Vela Jarquin, Min Woo Sun, Josiah Aklilu, James Burgess, Yuhui Zhang, Ryan Nayebi, Paola Avila, Robayo, Jin Ye, Ming Hu, Zhongying Deng, Junjun He, Xin Chen, Yue Yao, Robert Tibshirani, Jeffrey J. Nirschl, and Serena Yeung-Levy. MMBU: A massive multi-modal biomedical understanding benchmark to probe the perception capabilities of vision-language models. CoRR, abs/2606.06696, 2026. doi: 10.48550/ARXIV.2606.06696. DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence. CoRR, abs/2606.19348, 2026. doi: 10.48550/ARXIV.2606.19348. S. V. Destounis, S. M. Friedewald, L. J. Grimm, S. P. Poplack, and J. S. Sung. Mammography. In ACR BI-RADS v2025 Manual. American College of Radiology, Reston, VA, 2025. 10 Wilfrido G Ě omez-Flores, Maria Julia Gregorio-Calas, and Wagner Coelho de Albuquerque Pereira. Bus-bra: a breast ultrasound dataset for assessing computer-aided diagnosis systems. Medical physics, 51(4):3110â3123, 2024. Google Research and Google DeepMind. Medgemma technical report. CoRR, abs/2507.05201, 2025. doi: 10.48550/ARXIV.2507.05201. Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22170â22183, 2024. Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, Jin Hao, Zijian Chen, Ruijia Wu, Tao Tang, Junhui Lv, Hongxia Xu, Hongwei Wang, Jun Xiao, Bin Feng, Fudong Zhu, Kenli Li, Weidi Xie, Jimeng Sun, Jian Wu, and Zuozhu Liu. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. CoRR, abs/2510.08668, 2025. doi: 10.48550/ARXIV.2510.08668. Alexandros Karargyris, Renato Umeton, Micah J Sheller, Alejandro Aristizabal, Johnu George, Anna Wuest, Sarthak Pati, Hasan Kassem, Maximilian Zenk, Ujjwal Baid, et al. Federated benchmarking of medical artificial intelligence with medperf. Nature machine intelligence, 5(7):799â810, 2023. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611â626, 2023. LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. CoRR, abs/2506.07044, 2025. doi: 10.48550/ARXIV.2506.07044. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):180251, 2018. J. W. T. Leung, J. A. Baker, R. Hooley, V. A. Loving, and J. A. Rapelyea. Ultrasound. In ACR BI-RADS v2025 Manual. American College of Radiology, Reston, VA, 2025. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K Ě uttler, Mike Lewis, Wen-tau Yih, Tim Rockt Ě aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, p. 9459â9474. Curran Associates, Inc., 2020. Xiang Li, Farzaan Kaiyom, Evan Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, p. 94357â94386, 2025. Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), p. 1650â1654. IEEE, 2021. Bo Liu, Hanxue Gu, Xiangru Li, Zheren Zhu, Jacob Ellison, Kang Wang, Janine M Lupo, Yang Yang, and Hui Lin. Automated report-derived oncology vqa benchmark for evaluating vision-language models on 3d medical imaging. arXiv preprint arXiv:2606.02809, 2026a. Yang Liu, Jiajin Zhang, Danyang Tu, Yaojun Hu, Jiao Qu, Jiuyu Zhang, Yu Shi, Wei Fang, Shi Gu, Ling Zhang, and Yingda Xia. Breastgpt: A multimodal large language model for the full spectrum of breast cancer clinical routine. CoRR, abs/2606.04911, 2026b. doi: 10.48550/ARXIV.2606.0491 1. 11 Lena Maier-Hein, Annika Reinke, Michal Kozubek, Anne L Martel, Tal Arbel, Matthias Eisenmann, Allan Hanbury, Pierre Jannin, Henning M Ě uller, Sinan Onogur, et al. Bias: Transparent reporting of biomedical image analysis challenges. Medical image analysis, 66:101796, 2020. Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu. Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data, 10(1):277, 2023. Anna PawĹowska, Anna Ě Cwierz-Pie Ě nkowska, Agnieszka Domalik, Dominika Jagu Ě s, Piotr Kasprzak, RafaĹ Matkowski,Ĺukasz Fura, Andrzej Nowicki, and Norbert Ě ZoĹek. Curated benchmark dataset for ultrasound based breast lesion analysis. Scientific Data, 11(1):148, 2024. Qwen Team. Qwen3-vl technical report. CoRR, abs/2511.21631, 2025. doi: 10.48550/ARXIV.2511. 21631. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen .ai/blog?id=qwen3.5. Corentin Royer, Bjoern Menze, and Anjany Sekuboyina. Multimedeval: A benchmark and a toolkit for evaluating medical vision-language models. In Ninon Burgos, Caroline Petitjean, Maria Vakalopoulou, Stergios Christodoulidis, Pierrick Coupe, Herv Ě e Delingette, Carole Lartizien, and Diana Mateus (eds.), Proceedings of The 7nd International Conference on Medical Imaging with Deep Learning, volume 250 of Proceedings of Machine Learning Research, p. 1310â1327. PMLR, 03â05 Jul 2024. URL https://proceedings.mlr.press/v250/royer24a.html. Ali S Tejani, Michail E Klontzas, Anthony A Gatti, John T Mongan, Linda Moy, Seong Ho Park, Charles E Kahn Jr, and CLAIM 2024 Update Panel. Checklist for artificial intelligence in medical imaging (claim): 2024 update. Radiology: Artificial Intelligence, 6(4):e240300, 2024. Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data, 5:180161, 2018. doi: 10.1038/sdata.2018.161. Yuzhang Xie, Keqi Han, Yunpeng Xiao, Hejie Cui, Guanchen Wu, Ziyang Zhang, Kai Shu, Jiaying Lu, Xiao Hu, and Carl Yang. Ehrbench: An automated and reliable ehr-based benchmark for clinical decision making with llms. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD â26, p. 10068â10079, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400722592. doi: 10.1145/3770855.3817 571. Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai, Bokang Yang, Wencheng Han, Xiao-Hui Li, and Xiangyu Yue. Benchmark everything everywhere all at once, 2026. URLhttps: //arxiv.org/abs/2606.06462. Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embed- ding: Advancing text embedding and reranking through foundation models, 2025.URL https://arxiv.org/abs/2506.05176. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. CoRR, abs/2504.10479, 2025. doi: 10.48550/ARXIV.2504.10479. 12 Table 6: TaskPlanningAgentâs initial retained task proposals and supporting fields before task-space review. DatasetRaw proposed taskRaw supporting field(s) BrEaST: 11 retained proposals BrEaST anatomy_attribute.tissue_com position study.annotation.tissue_composition BrEaST finding_attribute.shape image.annotation.shape BrEaST finding_attribute.margin image.annotation.margin BrEaST finding_attribute.echogenici ty image.annotation.echogenicity BrEaST finding_attribute.posterior_ features image.annotation.posterior_features BrEaST finding_attribute.calcificat ions image.annotation.calcifications BrEaST finding_attribute.halo image.annotation.halo BrEaST finding_attribute.skin_thick ening image.annotation.skin_thickening BrEaST assessment_category image.annotation.birads BrEaST diagnosis image.annotation.diagnosis BrEaST pathology_status image.annotation.classification BUS-BRA: 6 retained proposals BUS-BRA finding_type image.annotation.histology BUS-BRA assessment_category image.annotation.birads BUS-BRA diagnosis image.annotation.histology BUS-BRA pathology_status image.annotation.pathology BUS-BRA detection image.annotation.bbox[ * ] BUS-BRA visual_grounding image.annotation.bbox[ * ]; image.annotation.histology VinDr-Mammo: 6 retained proposals VinDr view_identification image.annotation.view_position VinDr anatomy_attribute.breast_den sity image.annotation.breast_density VinDr finding_type image.annotation.findings[ * ].finding_categories[ * ] VinDr assessment_category image.annotation.breast_birads VinDr detection image.annotation.findings[ * ].bbox[ * ] VinDr visual_grounding image.annotation.findings[ * ].bbox[ * ]; image.annotation.findings[ * ].finding_categories[ * ] ACONSTRUCTED BREAST-IMAGING BENCHMARK A.1RAW TASK PROPOSALS AND EVIDENCE MAPPINGS Table 6 transcribes every task that TaskPlanningAgent markedretainin the initial taskplans/v1.jsonoutput, before researcher confirmation. It shows the modelâs original task decomposition and its exact mapping from each proposed task to authorized annotation evidence. The count-affecting review decisions are reported separately in Appendix B.2. A.2AUDITED REPRESENTATIVE ITEMS The 20 entries in Figures 4 and 5 give one audited item for every atomic datasetâtask pair instantiated in the realized breast-imaging benchmark, spanning BrEaST, BUS-BRA, and VinDr-Mammo. For each entry, we show the source dataset, task definition, representative image, generated query, and corresponding benchmark reference answer. BCONSTRUCTION ANALYSIS AND BENCHMARK VALIDATION B.1REFERENCE TASK SPACE AND TASK-SPACE EVALUATION PROTOCOL Baselines. We compare MedBenchAgent with three direct task-planning baselines to evaluate BIR-mediated task-space compilation against direct task induction. All methods receive the same confirmed evaluation scopeUand the same dataset-specific profile derived from the normalized annotation schemaÎŁ m . MedBenchAgent compiles the initial task treeT 0 within the BIR against the available annotation evidence through task retention and expansion. The direct baselines instead construct a complete set of prediction targets directly fromUandÎŁ m in a single model call per dataset. This comparison therefore evaluates catalog-initialized BIR compilation against direct task induction 13 BrEaST finding_attribute.shape Task description Shape of the target finding or abnormal region Annotation shape: oval Question How would you describe the shape of the lesion in this ultrasound image? Reference option: C Options A.roundB.irregular C.oval BrEaST finding_attribute.margin Task description Margin characteristic of the target finding or abnormal region Annotation margin: not circumscribed - indistinct Question Examine the ultrasound image and determine whether the lesion margin is circumscribed or not circumscribed. Reference option: B Options A.Circumscribed B.Not circumscribed BrEaST finding_attribute.echogenicity Task description Echogenicity of the target finding or abnormal region Annotation echogenicity: anechoic Question What is the echogenicity pattern of the lesion observed on this ultrasound image? Reference option: B Options A.hypoechoic B.anechoic C.heterogeneous D.isoechoic BrEaST finding_attribute. posterior_features Task description Posterior acoustic features of the target finding or abnormal region Annotation posterior_features: shadowing Question Given the ultrasound image, what is the correct description of the posterior acoustic features observed? Reference option: B Options A.Combined pattern B.Shadowing C. Enhancement D. No posterior features BrEaST finding_attribute.halo Task description Presence of a halo sign around the target finding or abnormal region Annotation halo: no Question In this breast ultrasound, does the abnormal region demonstrate a halo sign? Reference option: A Options A.NoB.Yes BrEaST finding_attribute. skin_thickening Task description Presence of skin thickening associated with the target finding or abnormal region Annotation skin_thickening: no Question Does the imaged breast lesion demonstrate skin thickening? Reference option: A Options A.AbsentB.Present BrEaST pathology_status Task description Determine the benign, malignant, or other pathological status associated with the imaging finding or case. Annotation classification: benign Question What is the pathological classification of the lesion seen in this breast ultrasound? Reference option: B Options A.MalignantB.Benign C.Normal BUS-BRA pathology_status Task description Determine the benign, malignant, or other pathological status associated with the imaging finding or case. Annotation pathology: benign Question Which of the following best describes the pathology of the finding seen in this ultrasound image: benign or malignant? Reference option: B Options A.MalignantB.Benign VinDr-Mammo anatomy_attribute. breast_density Task description Breast density category (A, B, C, D) indicating tissue composition Annotation breast_density: DENSITY C Question According to the ACR BI-RADS classification, which breast density category is depicted in this mammogram? Reference option: A Options A.DENSITY C B.DENSITY B C.DENSITY D D.DENSITY A BUS-BRA detection Task description Locate a lesion or abnormal region in the image without identifying the target by a specific finding name or description. Annotation bbox: [47, 92, 161, 158] Question Can you provide the bounding box that localizes the area of concern in this ultrasound image? Response: bounding box Response bbox pixels: [47, 92, 161, 158] VinDr-Mammo detection Task description Locate a lesion or abnormal region in the image without identifying the target by a specific finding name or description. Annotation bbox: [779.625, 612.034, 1894.66, 2014.29] Question Please draw a bounding box around the area of concern that should be examined further. Response: bounding box Response bbox pixels: [779.625, 612.034, 1894.66, 2014.29] VinDr-Mammo visual_grounding Task description Locate the single image region corresponding to one clearly specified, localized, image-observable finding or imaging sign. Annotation bbox: [9.002, 1212.4, 713.925, 2705.1799] Question Where is the Global Asymmetry located in this mammogram? Response: bounding box Response bbox pixels: [9.002, 1212.4, 713.925, 2705.1799] BrEaST anatomy_attribute. tissue_composition Task description Determine the global tissue composition of the breast anatomy Annotation tissue_composition: heterogeneous: predominantly fat Question What is the appropriate classification for the breast tissue composition observed in this ultrasound image? Reference option: C Options A.Heterogeneous: predominantly fibroglandular B.Homogeneous: fibroglandular C.Heterogeneous: predominantly fat D.Homogeneous: fat BrEaST finding_attribute. calcifications Task description Presence and type of calcifications in the target finding or abnormal region Annotation calcifications: no Question What type of calcifications is present in the abnormal area seen on this breast ultrasound? Reference option: A Options A.No calcifications B.Intraductal calcifications C.Indefinable calcifications D.Calcifications in a mass BrEaST assessment_category Task description Determine the standardized reporting category assigned to an imaging finding or case. Annotation birads: 5 Question Using the BI-RADS lexicon, how should this breast ultrasound finding be categorized? Reference option: A Options A.BI-RADS 5: Highly Suggestive of Malignancy B.BI-RADS 2: Benign C.BI-RADS 4c: High Suspicion D.BI-RADS 1: Negative E.BI-RADS 4a: Low Suspicion F.BI-RADS 4b: Moderate Suspicion G.BI-RADS 3: Probably Benign BrEaST diagnosis Task description Determine the most likely diagnosis for the imaged finding or case. Annotation diagnosis: Invasive carcinoma of no special type (NST) Question What diagnosis is most consistent with the breast lesion seen on this ultrasound? Reference option: A Options A.Invasive carcinoma of no special type (NST) B.Fibroadenoma C.Benign mammary dysplasia D.Invasive lobular carcinoma BUS-BRA assessment_category Task description Determine the standardized reporting category assigned to an imaging finding or case. Annotation birads: 3 Question According to the standardized breast imaging reporting system, which category best describes this ultrasound finding? Reference option: B Options A.BI-RADS 5: Highly Suggestive of Malignancy B.BI-RADS 3: Probably Benign C.BI-RADS 4: Suspicious D.BI-RADS 2: Benign BUS-BRA diagnosis Task description Determine the most likely diagnosis for the imaged finding or case. Annotation histology: invasive ductal carcinoma Question What histologic diagnosis is most consistent with the breast lesion seen on ultrasound? Reference option: B Options A.Fibrocystic changes B.Invasive ductal carcinoma C.Cyst D.Fibroadenoma VinDr-Mammo finding_type Task description Identify the explicitly annotated visual category of the primary imaging finding. Annotation finding_categories: Suspicious Calcification Question What mammographic finding is demonstrated in this image? Reference option: G Options A.Focal Asymmetry B.Nipple Retraction C.Skin Thickening D.Suspicious Lymph Node E.Mass F.Architectural Distortion G.Suspicious Calcification H.Asymmetry VinDr-Mammo assessment_category Task description Determine the standardized reporting category assigned to an imaging finding or case. Annotation breast_birads: BI-RADS 2 Question For this mammography study, which BI-RADS assessment category is most appropriate? Reference option: C Options A.BI-RADS 5: Highly Suggestive of Malignancy B.BI-RADS 3: Probably Benign C.BI-RADS 2: Benign D.BI-RADS 1: Negative E.BI-RADS 4: Suspicious Figure 4: Representative audited items for 12 datasetâtask mappings in the breast-imaging bench- mark. Each card shows the source dataset and task, source annotation, representative image, generated query, and benchmark reference or spatial response. under the same evaluation scope and dataset profile. Direct+T 0 augments the DeepSeek-V4-Flash direct baseline with the initial BIR task definitions as prompt reference. 14 BrEaST finding_attribute.shape Task description Shape of the target finding or abnormal region Annotation shape: oval Question How would you describe the contour of the lesion in this ultrasound image? Reference option: C Options A.roundB.irregular C.oval BrEaST finding_attribute.margin Task description Margin characteristic of the target finding or abnormal region Annotation margin: not circumscribed - indistinct Question Examine the ultrasound image and determine whether the lesion margin is circumscribed or not circumscribed. Reference option: B Options A.Circumscribed B.Not circumscribed BrEaST finding_attribute.echogenicity Task description Echogenicity of the target finding or abnormal region Annotation echogenicity: anechoic Question What is the echogenicity pattern of the lesion observed on this ultrasound image? Reference option: B Options A.hypoechoic B.anechoic C.heterogeneous D.isoechoic BrEaST finding_attribute. posterior_features Task description Posterior acoustic features of the target finding or abnormal region Annotation posterior_features: shadowing Question Given the ultrasound image, what is the correct description of the posterior acoustic features observed? Reference option: B Options A.Combined pattern B.Shadowing C. Enhancement D. No posterior features BrEaST finding_attribute.halo Task description Presence of a halo sign around the target finding or abnormal region Annotation halo: no Question In this breast ultrasound, does the abnormal region demonstrate a halo sign? Reference option: A Options A.NoB.Yes BrEaST finding_attribute. skin_thickening Task description Presence of skin thickening associated with the target finding or abnormal region Annotation skin_thickening: no Question Does the imaged breast lesion demonstrate skin thickening? Reference option: A Options A.AbsentB.Present BrEaST pathology_status Task description Determine the benign, malignant, or other pathological status associated with the imaging finding or case. Annotation classification: benign Question What is the pathological classification of the lesion seen in this breast ultrasound? Reference option: B Options A.MalignantB.Benign C.Normal BUS-BRA pathology_status Task description Determine the benign, malignant, or other pathological status associated with the imaging finding or case. Annotation pathology: benign Question Which of the following best describes the pathology of the finding seen in this ultrasound image: benign or malignant? Reference option: B Options A.MalignantB.Benign VinDr-Mammo anatomy_attribute. breast_density Task description Breast density category (A, B, C, D) indicating tissue composition Annotation breast_density: DENSITY C Question According to the ACR BI-RADS classification, which breast density category is depicted in this mammogram? Reference option: A Options A.DENSITY C B.DENSITY B C.DENSITY D D.DENSITY A BUS-BRA detection Task description Locate a lesion or abnormal region in the image without identifying the target by a specific finding name or description. Annotation bbox: [47, 92, 161, 158] Question Can you provide the bounding box that localizes the area of concern in this ultrasound image? Response: bounding box Response bbox pixels: [47, 92, 161, 158] VinDr-Mammo detection Task description Locate a lesion or abnormal region in the image without identifying the target by a specific finding name or description. Annotation bbox: [779.625, 612.034, 1894.66, 2014.29] Question Please draw a bounding box around the area of concern that should be examined further. Response: bounding box Response bbox pixels: [779.625, 612.034, 1894.66, 2014.29] VinDr-Mammo visual_grounding Task description Locate the single image region corresponding to one clearly specified, localized, image-observable finding or imaging sign. Annotation bbox: [9.002, 1212.4, 713.925, 2705.1799] Question Where is the Global Asymmetry located in this mammogram? Response: bounding box Response bbox pixels: [9.002, 1212.4, 713.925, 2705.1799] BrEaST anatomy_attribute. tissue_composition Task description Determine the global tissue composition of the breast anatomy Annotation tissue_composition: heterogeneous: predominantly fat Question What is the appropriate classification for the breast tissue in this ultrasound image? Reference option: C Options A.Heterogeneous: predominantly fibroglandular B.Homogeneous: fibroglandular composition observed D.Homogeneous: fat BrEaST finding_attribute. calcifications Task description Presence and type of calcifications in the target finding or abnormal region Annotation calcifications: no Question What type of calcifications is present in the this breast ultrasound? Reference option: A Options A.No calcifications B.Intraductal calcifications abnormal area seen on calcifications D.Calcifications in a mass BrEaST assessment_category Task description Determine the standardized reporting category assigned to an imaging finding or case. Annotation birads: 5 Question Using the BI-RADS lexicon, how should this breast categorized? Reference option: A Options A.BI-RADS 5: Highly Suggestive of Malignancy C. ultrasound finding be B.BI-RADS 2: Benign BI-RADS 4c: High Suspicion D.BI-RADS 1: Negative E.BI-RADS 4a: Low Suspicion F.BI-RADS 4b: Moderate Suspicion G.BI-RADS 3: Probably Benign BrEaST diagnosis Task description Determine the most likely diagnosis for the imaged finding or case. Annotation diagnosis: Invasive carcinoma of no special type (NST) Question What diagnosis is most consistent with the breast lesion seen on this ultrasound? Reference option: A Options A.Invasive carcinoma of no special type (NST) B.Fibroadenoma C.Benign mammary dysplasia D.Invasive lobular carcinoma BUS-BRA assessment_category Task description Determine the standardized reporting category assigned to an imaging finding or case. Annotation birads: 3 Question According to the standardized breast imaging reporting system, which category best describes this ultrasound finding? Reference option: B Options A.BI-RADS 5: Highly Suggestive of Malignancy B.BI-RADS 3: Probably Benign C.BI-RADS 4: Suspicious D.BI-RADS 2: Benign BUS-BRA diagnosis Task description Determine the most likely diagnosis for the imaged finding or case. Annotation histology: invasive ductal carcinoma Question What histologic diagnosis is most consistent with the breast lesion seen on ultrasound? Reference option: B Options A.Fibrocystic changes B.Invasive ductal carcinoma C.Cyst D.Fibroadenoma VinDr-Mammo finding_type Task description Identify the explicitly annotated visual category of the primary imaging finding. Annotation finding_categories: Suspicious Calcification Question What mammographic finding is demonstrated in this image? Reference option: G Options A.Focal Asymmetry B.Nipple Retraction C.Skin Thickening D.Suspicious Lymph Node E.Mass F.Architectural Distortion G.Suspicious Calcification H.Asymmetry VinDr-Mammo assessment_category Task description Determine the standardized reporting category assigned to an imaging finding or case. Annotation breast_birads: BI-RADS 2 Question For this mammography assessment category Reference option: C Options A.BI-RADS 5: Highly Suggestive of Malignancy BI-RADS 3: Probably Benign BI-RADS 2: Benign D.BI-RADS 1: Negative E. study, which BI-RADS B. is most appropriate? C. BI-RADS 4: Suspicious C.H pr e e t d e o ro m g i e n n a e n o tl u y s: fat C.Indefinable Figure 5: Continued from Figure 4. Representative samples from different datasetâtask mappings, completing all 20 pairs. For this comparison, both MedBenchAgent and the direct baselines are evaluated before human task-space confirmation. The MedBenchAgent result is the initial task-space proposal produced by TaskPlanningAgent, while each direct baseline is evaluated from its raw model output. All LLM baselines return a single JSON object through an OpenAI-compatible endpoint and are hosted locally on 1â2 NVIDIA B200 GPUs. Reference task-space design. We designed the breast-imaging reference task spaceT â from the perspective of medical VLM evaluation, with the goal of covering the major stages of imaging diagnosis across the breast cancer diagnostic workflow. Informed by discussions within the research team and with a clinical collaborator, we abstracted this workflow into eight evaluation phases: image recognition and validity, global tissue characterization, lesion detection and localization, finding characterization, diagnostic classification, standardized assessment, management and integrated reasoning, and report generation. These phases provide a clinically structured progression from basic 15 image interpretation to increasingly integrated diagnostic decisions, while remaining expressed as model-evaluable visual prediction capabilities. We then applied this staged framework using the task definitions and reference annotations available in the public breast-imaging datasets considered in this study. For each phase, we defined a predic- tion target only when the corresponding annotations provided a clear reference standard, and we represented distinct prediction targets as separate atomic tasks. The framework defines the intended clinical coverage, while each dataset determines which stages have annotation-supported targets. Stages without an eligible target remain in the framework, showing both the covered capabilities and the parts of the broader imaging workflow not supported by the available public annotations. Figure 6 reports the resulting dataset-conditioned reference task space. Under the atomic matching protocol below, the table contains 21 reference prediction targets in total. A phase may contain multiple atomic targets when the available annotations support distinct prediction targets, whereas a categorical prediction target such as VinDr finding-type classification remains one task even when its answer space contains multiple finding categories. Independence from automated task-space construction. The staged framework and the resulting T â were developed before the automated task-space experiments and were fixed prior to evaluating MedBenchAgent or any direct-construction baseline.T â was not provided to any evaluated con- struction method, and the reference task space was not added to, removed from, split, or merged in response to method outputs. It therefore serves only as the external reference for measuring agree- ment between automatically constructed task spaces and the independently designed breast-imaging evaluation space. Human involvement in task-space evaluation. Development of the reference framework and adjudication of method outputs were separated from the automated construction procedures. The reference-space design involved benchmark-design discussions among dataset-familiar researchers with clinical input. For the Task-Space F1 analysis reported in this work, matching and adjudication between method proposals and the fixed reference task space were performed by a researcher using the predefined rules below. Prediction-target matching. The unit of evaluation was a dataset-conditioned prediction target. An ordinary task counted once; a decomposable parent category did not count separately, whereas each distinct prediction target under that parent was evaluated as one task. Matching was based on whether the proposal and reference represented the same prediction target, regardless of task identifier, surface wording, evaluation metric, response format, or JSON nesting. A raw proposal was counted as a true positive (TP) when it matched one reference prediction target inT â . A proposal without a corresponding reference task was counted as a false positive (FP), including unsupported, out-of-scope, shortcut-prone, or semantically redundant proposals. A reference task inT â without a semantically equivalent proposal was counted as a false negative (FN). Matching was one-to-one. If multiple proposals represented the same reference prediction target, at most one was matched as a TP and each additional duplicate was counted as an FP. If a single proposal conflated multiple distinct prediction targets, it could match at most one reference task, with each remaining unmatched target counted as an FN. Differences that did not change the prediction targetâincluding the evaluation metric, response format, bounding-box coordinate convention, surface wording, or exact annotation-field spellingâdid not affect task-level matching. Evidence-Mapping Agreement. For task proposals matched toT â , we additionally evaluated whether the method identified the annotation fields required to support the corresponding reference construction. Evidence-Mapping Agreement (EMA) was computed only over matched TPs. For each matched task, agreement was assigned a value of one when the method-proposed supporting-field set was identical to the supporting-field set confirmed during review against the dataset annotation schema ÎŁ m , and zero otherwise. FPs and FNs were excluded from the EMA denominator. Metrics and aggregation. For each methodâdataset pair, we computed P = TP TP + FP , R = TP TP + FN , F 1 = 2TP 2TP + FP + FN . We additionally report the number of task-space discrepancies between a raw proposal andT â , defined asFP + FN. Overall results micro-aggregate TP, FP, and FN across the three datasets 16 Diagnostic breast imaging Patient presentation screen-detected abnormality or symptoms / signs Global tissue characterization Abnormality detection and localization Imaging finding characterization BI-RADS assessment Breast density VinDr Tissue composition BrEaST Lesion detection VinDr / BUS-BRA Visual grounding VinDr Finding type VinDr Finding attributes shape, margin, echogenicity, posterior features, calcifications, halo, skin thickening BrEaST Assessment category VinDr / BrEaST / BUS-BRA BI-RADS 0 Incomplete BI-RADS 1/2 Negative / benign BI-RADS 3 Probably benign BI-RADS 4/5 Suspicious Additional diagnostic imaging Routine screening / clinical follow-up Short-interval follow-up Biopsy / tissue sampling Pathology result when tissue sampling is performed Pathology status BrEaST / BUS-BRA Reference-confirmed diagnosis BrEaST / BUS-BRA Figure 6: Clinical breast-imaging workflow and its annotation-supported reference task space. Tasks are attached to the corresponding stages of the imaging workflow. before applying these equations. Thus, datasets contribute according to their numbers of reference and proposed prediction targets rather than receiving equal dataset weight. Direct task-induction baseline. The direct baselines receive the same confirmed benchmark requirement and dataset profile as MedBenchAgent, but do not receive the predefined task catalogue T 0 . They induce the complete task space in a single model call under the same output schema and evidence-admissibility rules, including annotation-supported prediction targets, atomic answer axes, semantic-duplicate removal, and explicit spatial-evidence requirements. The complete baseline prompts and serialization instructions are provided with the released code. B.2TASK-SPACE RESULTS AND DISCREPANCY ANALYSIS Tables 7 and 8 detail task-space discrepancies for MedBenchAgent and the three direct baselines. Their matched tasks have supporting-field mappings consistent with the reference, as reflected by the EMA results in Table 1. The discrepancy patterns also differ qualitatively across construction strategies. MedBenchAgentâs residual errors are concentrated in semantic boundary cases, including a redundant BUS-BRA finding- type proposal, a visual-grounding proposal that did not define a capability distinct from generic detection, a shortcut-prone VinDr view-identification task, and the omitted finding-level assessment target. In contrast, the direct baselines more frequently introduce nonvisual metadata, acquisition or marker shortcuts, invalid spatial evidence, or omit otherwise supported visual prediction targets. This distinction helps explain why Evidence-Mapping Agreement is perfect for matched tasks even when task-space precision differs: once the intended prediction target is identified, annotation-field selection is comparatively stable, whereas deciding which targets should constitute the benchmark remains more difficult. B.3HUMAN ITEM AUDIT AND RESIDUAL ERRORS Sampling and review setting. The audit population comprised all 16,248 finalized items accepted by the automatic screening and audit procedures in the completed BrEaST, BUS-BRA, and VinDr- 17 Table 7: Task-space discrepancies affecting TP, FP, or FN counts for MedBenchAgent and Direct DeepSeek. Retained true positives are omitted. MethodDatasetProposal or confirmed taskLabelReview rationale MedBenchAgentBUS-BRAfinding typeFPDuplicated diagnosis on the histology answer axis. MedBenchAgentBUS-BRAvisual groundingFPDid not define a capability distinct from generic lesion detec- tion. MedBenchAgentVinDrview identificationFPBurned-in view markers exposed the target as a shortcut. MedBenchAgentVinDrfinding-level assessmentFNFinding-level BI-RADS supported a distinct, initially omitted answer axis. Direct DeepSeekBrEaSTclinical signsFPNonvisual clinical metadata. Direct DeepSeekBrEaSTpatient symptomsFPNonvisual patient information. Direct DeepSeekBrEaSTverification methodFPNonvisual examination/pathology metadata. Direct DeepSeekBrEaSTtumor localizationFPTreated a mask as bounding-box evidence. Direct DeepSeekBrEaSTdiagnosisFN The explicit diagnosis field was not proposed as an indepen- dent task. Direct DeepSeekBUS-BRAside identificationFPLaterality metadata was outside the intended visual scope. Direct DeepSeekBUS-BRAdevice identificationFPAcquisition-device metadata was outside the clinical visual scope. Direct DeepSeekVinDrlaterality classificationFPMetadata/marker shortcut rather than the intended image semantics. Direct DeepSeekVinDrview-position classificationFPBurned-in view marker leaked the target. Direct DeepSeekVinDrvisual groundingFNFinding category was not paired with its box. Table 8: Task-space discrepancies affecting TP, FP, or FN counts for Direct Qwen and Direct MedGemma. Retained true positives are omitted. MethodDatasetProposal or confirmed taskLabelReview rationale Direct QwenBrEaSTpalpability assessmentFPClinical palpation is not image-answerable. Direct QwenBrEaSTinterpretation classificationFPA source conclusion is not an independent visual capabil- ity. Direct QwenBrEaSTverification methodFPNonvisual confirmation metadata. Direct QwenBrEaSTpatient symptom statusFPNonvisual patient information. Direct QwenBUS-BRAlateralizationFPLaterality metadata was outside the intended scope. Direct QwenBUS-BRAdevice identificationFP Acquisition-device metadata was outside the intended scope. Direct QwenVinDrlaterality classificationFPMetadata/marker shortcut. Direct QwenVinDrview-position classificationFPBurned-in view marker leaked the target. Direct QwenVinDrbreast densityFN A varying breast-density field supported an omitted global answer axis. Direct QwenVinDrvisual groundingFNFinding category was not paired with its box. Direct MedGemmaBrEaSTanatomy classificationFPAnatomy was constant and could not form a valid classifi- cation task. Direct MedGemmaBrEaSTmodality classificationFPModality was constant. Direct MedGemmaBrEaSTinterpretation classificationFP Source interpretation was not an independent visual capa- bility. Direct MedGemmaBrEaSTage classificationFPPatient age was nonvisual metadata. Direct MedGemmaBrEaSTpatient symptomsFPPatient symptoms were nonvisual. Direct MedGemmaBrEaSTsigns classificationFP Clinical-sign metadata was outside the target visual scope. Direct MedGemmaBrEaSTverification classificationFPNonvisual confirmation metadata. Direct MedGemmaVinDrlaterality classificationFPMetadata/marker shortcut. Direct MedGemmaVinDrview-position classificationFPBurned-in view marker leaked the target. Direct MedGemmaVinDrvisual groundingFNFinding category was not paired with its box. Mammo runs. We globally shuffled this combined population and sampled 1,000 items uniformly without replacement. We reviewed each sampled item using the predefined criteria below. The review materials comprised the dataset and task identifiers, response format, question, options, benchmark reference, and complete normalized annotation. The audit evaluated construction consistency against the source annotations and locked task specification. Audit criteria. Four binary criteria were applied. Reference groundedness required the benchmark reference to follow directly from the authorized annotation evidence. Taskâquery alignment required the question to express the locked prediction target. Unambiguity required the question to admit one intended answer under the supplied evidence. Evaluation integrity required the absence of answer leakage, overlapping response options, questionâreference mismatch, response-format mismatch, and unsupported additions. An item was considered valid only when all four criteria passed: ValidItem i = 1G i ⧠A i ⧠U i ⧠I i , 18 Table 9: ISIC 2018 tasks, supporting annotation fields, and realized item counts. TaskSupporting fieldResp.MetricPlannedRealized Diagnosis image.annotation.diagnosisletterMacro-F11,6671,512 Detection image.annotation.findings[ * ] .bbox[ * ] bboxmIoU1,6661,000 Globules presence image.annotation.attribute_presence .globules letterAccuracy334334 Milia-like cyst presence image.annotation.attribute_presence .milia_like_cyst letterMacro-F1334334 Negative network presence image.annotation.attribute_presence .negative_network letterMacro-F1333333 Pigment network presence image.annotation.attribute_presence .pigment_network letterAccuracy333333 Streaks presence image.annotation.attribute_presence .streaks letterMacro-F1333333 Total5,0004,179 whereG i ,A i ,U i , andI i denote reference groundedness, taskâquery alignment, unambiguity, and evaluation integrity, respectively. Residual errors. Six of the 1,000 sampled items failed at least one audit criterion. Four VinDr- Mammo finding-type items exhibited the same template-level taskâquery mismatch: the generated question asked for a BI-RADS assessment category while the locked answer space contained finding categories. Two BrEaST diagnosis items introduced unsupported contextual details and therefore failed evaluation integrity. Importantly, the fixed annotation-derived reference answers remained supported by the authorized evidence in all six cases. Across the complete audit, there were four taskâquery-alignment failures and six evaluation-integrity failures, with no reference-groundedness or unambiguity failures. CCROSS-DOMAIN PORTABILITY STUDY: DERMOSCOPY C.1DATASET, TASK SPACE, AND REALIZED ALLOCATION The cross-domain run used the ISIC 2018 challenge data from its three official task sources: lesion segmentation (Task 1), dermoscopic-attribute detection (Task 2), and disease classification (Task 3). Tasks 1 and 2 share 1,000 held-out dermoscopic images, while Task 3 contributes 1,512 held-out diagnosis images. After deduplicating the shared Task 1/2 identifiers, the normalized corpus contains 2,512 unique images. Diagnosis labels are available for the Task 3 subset, whereas lesion masks and five dermoscopic-attribute masks are available for the Task 1/2 subset. Because these annotations are defined on different image subsets, no single task is eligible for all 2,512 images. We apply the workflow to the ISIC annotation schema and evaluate the resulting task-to-evidence mappings, item consistency, and model performance. Task space and realized allocation. The requested 5,000-item allocation yielded 4,179 realized items because of source-data eligibility limits. All eligible diagnosis (1,512) and lesion-localization (1,000) cases were retained, with the remaining 1,667 items allocated across the five dermoscopic- attribute tasks. We checked the initial task plan against the documented ISIC annotation schema and label semantics. All seven proposed tasks and their supporting-field mappings were retained without task-space revision. C.2ITEM CONSISTENCY AND REPRESENTATIVE EXAMPLES Item consistency. We additionally performed a consistency check on 200 finalized ISIC items using the same annotation- and contract-based criteria as the breast-imaging audit. All sampled items passed checks of referenceâevidence consistency, taskâquery consistency, and evaluation-format integrity against the normalized annotations and locked task contracts. Figure 7 provides one audited representative item for each of the seven dermoscopy tasks using the same presentation format as the breast-imaging examples. 19 ISIC 2018 finding_attribute. globules_presence Task description Presence of globules in the lesion Annotation attribute_presence.globules: true Question Does the dermoscopic image of the skin lesion show globules? Reference option: B Options A.AbsentB.Present ISIC 2018 finding_attribute. milia_like_cyst_presence Task description Presence of milia-like cysts in the lesion Annotation attribute_presence.milia_like_ cyst: true Question What is the status of milia-like cysts in this dermoscopic image? Reference option: A Options A.PresentB.Absent ISIC 2018 finding_attribute. negative_network_presence Task description Presence of negative network in the lesion Annotation attribute_presence.negative_ network: true Question Does this lesion exhibit a negative network? Reference option: A Options A.PresentB.Absent ISIC 2018 finding_attribute. pigment_network_presence Task description Presence of pigment network in the lesion Annotation attribute_presence.pigment_ network: true Question Does the dermoscopic image show a pigment network? Reference option: B Options A.AbsentB.Present ISIC 2018 finding_attribute. streaks_presence Task description Presence of streaks in the lesion Annotation attribute_presence.streaks: true Question Regarding streaks in this dermoscopic image, which finding is observed? Reference option: A Options A.PresentB.Absent ISIC 2018 detection Task description Locate a lesion or abnormal region in the image without identifying the target by a specific finding name or description. Annotation bbox: [984, 684, 2184, 1584] Question Please outline the skin lesion in the image by providing the bounding box coordinates. Response: bounding box Response bbox pixels: [984, 684, 2184, 1584] ISIC 2018 diagnosis Task description Determine the most likely diagnosis for the imaged finding or case. Annotation diagnosis: BCC Question Given the dermoscopic image, what is the most likely diagnosis? Reference option: D Options A.Melanoma B.Melanocytic nevus C.Dermatofibroma D.Basal cell carcinoma E.Vascular lesion F.Actinic keratosis / Bowen's disease G.Benign keratosis Figure 7: Representative audited ISIC 2018 items for all seven dermoscopy tasks. C.3MODEL EVALUATION Table 10 reports performance across diagnosis, dermoscopic attributes, and localization. Different checkpoints lead on different task metrics, providing complementary evaluation profiles. Together with the task-space review and item audit, these results demonstrate application of MedBenchAgent to the ISIC annotation schema. Table 10 shows that the cross-domain benchmark preserves substantial variation across task families and model checkpoints. No single checkpoint exhibits uniformly strong performance across diagnosis, dermoscopic attributes, and lesion localization, and the relative behavior of general-purpose and medical VLMs varies across these task types. In particular, categorical performance does not reliably predict spatial-localization performance. These results are consistent with the breast-imaging evaluation in showing that aggregate model quality cannot be inferred from one task family alone, while providing evidence that the same construction workflow can be instantiated under a different annotation schema and clinical vocabulary. 20 Table 10: Evaluation performance on ISIC 2018. Choice tasks report Accuracy / Macro-F1 (%), while detection reports mIoU / Acc@IoU0.5 (%). Best results for each metric are shown in bold. Finding Attribute ModelDiagnosis GlobulesMilia-likeNeg. net.Pigment net.Streaks Detection General-purpose VLMs InternVL3-8B20.7 / 15.331.4 / 31.170.7 / 56.645.3 / 35.072.4 / 63.439.3 / 35.750.5 / 52.2 Qwen2.5-VL-7B24.2 / 14.657.8 / 53.780.8 / 55.583.8 / 47.459.5 / 59.333.6 / 30.775.1 / 86.6 Qwen3-VL-8B-I53.2 / 22.828.4 / 27.580.2 / 61.880.2 / 47.359.5 / 58.458.0 / 44.780.3 / 91.4 Qwen3-VL-8B-T â 30.9 / 18.660.5 / 52.983.2 / 61.185.0 / 48.257.1 / 56.082.9 / 47.071.2 / 83.6 Qwen2.5-VL-32B29.8 / 20.825.1 / 23.646.7 / 40.228.5 / 24.372.4 / 64.515.9 / 15.976.8 / 87.6 Qwen3-VL-32B52.8 / 30.370.1 / 58.284.4 / 55.174.2 / 45.858.0 / 57.388.0 / 55.180.2 / 92.2 Medical VLMs MedGemma-4B47.8 / 31.525.7 / 24.066.8 / 52.545.3 / 33.579.0 / 71.759.5 / 47.731.8 / 29.1 Lingshu-7B10.6 / 7.068.3 / 57.282.9 / 53.946.8 / 36.276.9 / 71.446.5 / 38.411.7 / 6.8 Hulu-Med-7B56.3 / 34.475.4 / 47.483.8 / 59.095.2 / 48.840.8 / 38.676.6 / 48.90.7 / 0.2 QoQ-Med-7B â 49.5 / 21.469.2 / 48.284.7 / 55.690.7 / 47.641.1 / 39.275.7 / 46.825.8 / 11.3 Lingshu-32B24.6 / 16.871.9 / 60.375.7 / 53.544.1 / 33.972.4 / 69.257.4 / 44.844.5 / 53.3 DADDITIONAL MODEL-EVALUATION RESULTS D.1EVALUATION PROTOCOL For reporting, fine-grained question types in Table 3 are first scored separately and macro-averaged within their parent task where applicable, while dataset-specific results for the same task family are then macro-averaged with equal weight, and the reported Overall score macro-averages the resulting task-family scores. In this section, we report the complete per-dataset model-evaluation results and detailed evaluation protocol. Tables 11, 13, and 14 use the same two-level template to report dataset-level task results for BrEaST, BUS-BRA, and VinDr-Mammo. The abbreviations are A (anatomy attribute), FA (finding attribute), FT (finding type), AC (assessment category), DX (diagnosis), and PS (pathology status). Table 12 separately expands BrEaST FA into its seven question types. Table 15 reports mIoU and Acc@IoU 0.5 for each datasetâtask pair. Scores are computed over all evaluated items, assigning zero IoU when no valid bounding box is returned. Inputs and response parsing. Each model received the image, question, and lettered options when applicable, without the reference answer. Model-specific chat templates were applied with deterministic decoding. Multiple-choice predictions were parsed as a single valid option letter and evaluated using Accuracy and Macro-F1 over semantic answer classes. Invalid responses remained incorrect. Spatial predictions were parsed as a single four-number bounding box, accepting the supportedbboxorbbox2dfields, bare lists or tuples, and JSON-fenced outputs. Missing, malformed, nonfinite, degenerate, out-of-range, or multiple distinct boxes were invalid and received zero IoU. Coordinate conversion and scoring. We strictly follow the protocols of all models that claim to have spatial evaluation capabilities. Predictions were converted from the configured model- specific coordinate representation to absolute original-image coordinates before scoring. Qwen3-VL, InternVL3, and BreastGPT used a fixed[0, 1000]coordinate space; Qwen2.5-VL, Lingshu, and QoQ- Med used coordinates in the resized model input; and Hulu-Med used original-image coordinates. MedGemma used normalized[0, 1]coordinates in[y 1 ,x 1 ,y 2 ,x 2 ]order, which were reordered and rescaled. The effective coordinate size was recorded with each prediction. Spatial metrics were computed over all evaluated items, including invalid outputs. Accordingly, mIoU andAcc@IoU 0.5 measure end-to-end bounding-box prediction under the specified response protocol. D.2PAIRWISE MODEL-ORDERING ANALYSIS The summary statistics of model-ordering reversals are reported in Section 5.3. Here, we provide the detailed pairwise model-ordering analysis and describe how these statistics are computed. For two 21 Table 11: Multiple-choice evaluation on BrEaST by task type. Results are reported as Accuracy / Macro-F1 (%). A contains 200 tissue-composition items; FA equally averages its seven question types; AC and PS each contain 256 items; and DX contains 121 items. â denotes thinking variants. AnatomyFindingAssessment & Diagnosis Model AAFAACDXPS General-purpose VLMs InternVL3-8B27.5 / 10.842.3 / 30.918.8 / 10.128.9 / 16.350.8 / 33.4 Qwen2.5-VL-7B33.0 / 19.541.5 / 29.316.4 / 10.534.7 / 21.354.7 / 36.4 Qwen3-VL-8B-I27.5 / 10.929.8 / 22.620.7 / 16.738.8 / 26.960.9 / 40.3 Qwen3-VL-8B-T â 28.0 / 13.456.9 / 35.121.1 / 19.338.0 / 24.549.2 / 32.7 Qwen2.5-VL-32B25.0 / 15.246.2 / 29.619.1 / 13.248.8 / 27.347.3 / 31.6 Qwen3-VL-32B17.5 / 12.937.1 / 28.420.7 / 20.835.5 / 27.762.9 / 53.0 Medical VLMs MedGemma-4B26.5 / 16.047.5 / 33.919.5 / 13.444.6 / 32.451.6 / 36.9 Lingshu-7B33.5 / 24.056.8 / 38.724.6 / 25.046.3 / 18.869.1 / 40.4 Hulu-Med-7B27.0 / 11.351.1 / 35.520.7 / 16.044.6 / 24.461.7 / 41.3 QoQ-Med-7B â 30.0 / 20.766.1 / 37.820.7 / 13.743.0 / 23.548.8 / 31.5 BreastGPT-8B21.5 / 19.975.2 / 52.536.3 / 36.160.3 / 39.768.4 / 55.5 Lingshu-32B31.5 / 22.257.5 / 43.023.4 / 18.747.1 / 20.367.2 / 42.3 models i and j, an ordering reversal between scores s and s Ⲡoccurs when (s i â s j )(s Ⲡi â s Ⲡj ) < 0. With 12 evaluated VLMs, there are 12 2 = 66unordered model pairs. Ties are not counted as reversals. Overall versus task-level evaluation.We compare the ordering induced by Overall Acc. in Table 3 with the Acc. ordering for each task category. The numbers of reversed pairs are 19 (A), 22 (FT), 18 (FA), 24 (AC), 12 (DX), and 22 (PS), corresponding to 18.2â36.4% of all model pairs. Acc. versus F1. Within each task category, we compare model ordering under Acc. and F1. The numbers of reversed pairs are 13 (A), 11 (FT), 5 (FA), 15 (AC), 15 (DX), and 4 (PS), corresponding to 6.1â22.7%. Choice-based versus spatial evaluation. We compare Overall Acc. with each of the six spatial measures in Table 4. The numbers of reversed model pairs are 33 and 32 for VinDr-Mammo detection, 34 and 32 for VinDr-Mammo visual grounding, and 29 and 28 for BUS-BRA detection, for mIoU and Acc@IoU 0.5 , respectively. Thus, 28â34 of the 66 model pairs (42.4â51.5%) reverse their ordering across the spatial measures. D.3ADDITIONAL ANALYSIS OF MODEL BEHAVIOR The results allow us to examine how aggregate model performance is composed and where model comparisons change with the evaluation target, metric, or imaging setting. We first analyze variation within task families, then examine changes across model scales and thinking variants, and finally compare categorical and spatial performance across imaging settings. Throughout, comparisons refer to the evaluated checkpoints under the reported protocols. D.3.1TASK GRANULARITY AND METRIC-DEPENDENT RANKINGS Strong task-family performance can coexist with substantial variation among its constituent prediction targets. BreastGPT-8B achieves the highest aggregate finding-attribute scores, yet its BrEaST Macro- F1 ranges from 78.0 on margin and 63.7 on halo to 33.2 on posterior features and 35.0 on calcifications (Table 12). The aggregate FA score therefore combines markedly different attribute-level results. Retaining these prediction targets separately identifies which attributes contribute to a modelâs overall advantage and which remain comparatively weak under their respective evaluation settings. 22 Table 12: Multiple-choice evaluation on BrEaST: finding-attribute question types. Results are reported as Accuracy / Macro-F1 (%). Best results for each metric are shown in bold. Finding Attribute Model ShapeMarginEcho.PosteriorCalc.HaloSkin General-purpose VLMs InternVL3-8B54.5 / 30.953.2 / 53.046.3 / 33.121.8 / 15.645.2 / 23.026.0 / 24.448.8 / 35.9 Qwen2.5-VL-7B55.4 / 26.858.3 / 47.440.3 / 28.319.4 / 8.127.8 / 14.747.6 / 44.641.4 / 35.1 Qwen3-VL-8B-I34.3 / 28.647.2 / 42.225.5 / 16.237.3 / 24.850.4 / 24.011.4 / 16.72.7 / 5.6 Qwen3-VL-8B-T â 56.6 / 34.458.3 / 45.861.5 / 28.422.2 / 12.459.9 / 30.463.4 / 46.676.6 / 47.8 Qwen2.5-VL-32B57.4 / 32.355.2 / 45.361.0 / 35.122.2 / 12.471.0 / 28.647.2 / 43.89.4 / 9.4 Qwen3-VL-32B58.7 / 39.932.9 / 40.345.0 / 34.723.8 / 17.952.0 / 24.715.9 / 17.331.2 / 24.2 Medical VLMs MedGemma-4B52.9 / 40.854.8 / 52.839.8 / 23.352.8 / 24.629.0 / 14.242.3 / 39.661.3 / 42.3 Lingshu-7B58.3 / 45.663.9 / 60.263.6 / 32.613.9 / 9.444.0 / 18.865.4 / 48.588.7 / 55.5 Hulu-Med-7B58.3 / 34.159.5 / 50.058.9 / 38.625.8 / 19.117.9 / 13.364.6 / 45.073.0 / 48.4 QoQ-Med-7B â 52.1 / 30.665.1 / 62.251.9 / 30.557.9 / 25.875.4 / 27.272.0 / 41.888.3 / 46.9 BreastGPT-8B73.6 / 56.178.6 / 78.070.6 / 45.354.8 / 33.289.3 / 35.070.3 / 63.789.5 / 56.3 Lingshu-32B59.9 / 47.567.9 / 64.462.8 / 44.723.0 / 21.045.2 / 22.965.4 / 44.678.5 / 56.1 Table 13: Multiple-choice evaluation on BUS-BRA by task type. Results are reported as Accuracy / Macro-F1 (%). â denotes thinking variants. Best results for each metric are shown in bold. Assessment & Diagnosis Model ACDXPS General-purpose VLMs InternVL3-8B25.7 / 24.141.8 / 29.447.1 / 47.1 Qwen2.5-VL-7B37.0 / 15.632.8 / 24.147.4 / 47.3 Qwen3-VL-8B-I35.3 / 20.541.1 / 32.460.2 / 55.4 Qwen3-VL-8B-T â 31.2 / 20.932.2 / 23.856.9 / 52.3 Qwen2.5-VL-32B34.1 / 21.237.8 / 26.058.6 / 53.2 Qwen3-VL-32B36.2 / 20.843.2 / 31.955.8 / 55.6 Medical VLMs MedGemma-4B22.6 / 22.415.8 / 15.160.0 / 51.8 Lingshu-7B38.6 / 26.358.1 / 39.873.3 / 64.6 Hulu-Med-7B26.3 / 25.454.6 / 45.066.6 / 61.8 QoQ-Med-7B â 36.9 / 14.539.9 / 24.735.4 / 29.4 BreastGPT-8B45.0 / 36.569.8 / 54.488.8 / 87.1 Lingshu-32B33.3 / 28.051.0 / 28.773.0 / 68.1 Table 14: Multiple-choice evaluation on VinDr-Mammo by task type. Results are reported as Accuracy / Macro-F1 (%). Best results for each metric are shown in bold. AnatomyFindingAssessment & Diagnosis Model AAFTAC General-purpose VLMs InternVL3-8B45.7 / 23.819.1 / 8.73.9 / 4.2 Qwen2.5-VL-7B64.7 / 26.629.8 / 10.210.7 / 9.6 Qwen3-VL-8B-I42.3 / 26.216.3 / 9.061.4 / 22.0 Qwen3-VL-8B-T â 23.0 / 14.915.1 / 7.363.4 / 19.9 Qwen2.5-VL-32B64.4 / 28.054.0 / 12.727.3 / 15.9 Qwen3-VL-32B57.1 / 28.054.6 / 13.345.8 / 20.1 Medical VLMs MedGemma-4B18.3 / 15.423.2 / 10.811.8 / 10.3 Lingshu-7B55.0 / 28.332.2 / 12.312.7 / 11.3 Hulu-Med-7B36.2 / 22.017.0 / 9.221.2 / 15.1 QoQ-Med-7B â 67.1 / 24.753.8 / 12.03.7 / 2.5 BreastGPT-8B50.6 / 32.935.7 / 17.040.8 / 22.1 Lingshu-32B70.3 / 27.242.7 / 14.245.8 / 18.3 23 Table 15: Spatial evaluation on BUS-BRA and VinDr-Mammo. Results report mIoU and Acc@IoU 0.5 (%). Best results for each metric are shown in bold. BUS-BRA DetectionVinDr DetectionVinDr Visual Grounding Model mIoUAcc@.5mIoUAcc@.5mIoUAcc@.5 General-purpose VLMs InternVL3-8B26.011.23.30.23.20.3 Qwen2.5-VL-7B28.922.27.63.98.86.5 Qwen3-VL-8B-I51.156.410.77.013.711.0 Qwen3-VL-8B-T â 37.532.910.97.310.47.0 Qwen2.5-VL-32B40.637.59.76.110.87.3 Qwen3-VL-32B55.462.115.412.918.016.8 Medical VLMs MedGemma-4B15.73.71.50.01.60.0 Lingshu-7B4.70.94.40.83.60.7 Hulu-Med-7B7.03.00.00.00.00.0 QoQ-Med-7B â 45.643.57.01.17.01.3 BreastGPT-8B72.682.85.30.95.41.1 Lingshu-32B12.54.44.71.55.62.0 The choice of metric also changes comparisons within an individual attribute. On posterior features, QoQ-Med-7B achieves higher Accuracy than BreastGPT-8B (57.9 versus 54.8), whereas BreastGPT- 8B achieves higher Macro-F1 (33.2 versus 25.8). The same ordering reversal occurs on halo: QoQ-Med-7B leads in Accuracy (72.0 versus 70.3), but BreastGPT-8B leads in Macro-F1 (63.7 versus 41.8). These reversals extend the metric-dependent rankings observed in the main results to atomic prediction targets. Accuracy weights individual items equally, while Macro-F1 averages class-specific F1 scores; consequently, the two metrics capture different aspects of performance over the answer space. Reporting both preserves distinctions that either metric alone would omit. D.3.2CHANGES ACROSS MODEL SCALES AND THINKING VARIANTS Similar aggregate scores can conceal large changes in the underlying task profile. Qwen3-VL-8B-T exceeds Qwen3-VL-8B-I by only 0.60 Accuracy points overall, but the task-level changes are much larger (Table 3). FA improves by 27.09 points, while A, FT, AC, DX, and PS decrease by 9.37, 1.22, 0.53, 4.85, and 7.50 points, respectively. Under the reported equal weighting of task families, the FA improvement contributes approximately 4.52 points to Overall Accuracy, while the other five tasks jointly contribute approximatelyâ3.91points. The small net change thus results from opposing task-level changes rather than uniformly similar performance. The FA improvement is itself concentrated in particular attributes. On BrEaST, halo Accuracy increases from 11.4 to 63.4 and skin-thickening Accuracy from 2.7 to 76.6, whereas posterior-feature Accuracy decreases from 37.3 to 22.2 (Table 12). With seven equally weighted attributes, the halo and skin-thickening increases together contribute approximately 18.0 points to the roughly 27.1-point FA improvement, based on the rounded attribute scores. Decomposing the comparison at both levels therefore reveals a pattern that the Overall score obscures: a substantial gain within one task family is driven largely by a subset of its prediction targets and offset by declines elsewhere. Changes across model scales also vary by task. Within Lingshu, the 32B checkpoint improves Overall Accuracy from 47.00 to 50.73 and FA Macro-F1 from 38.66 to 43.01 relative to the 7B checkpoint, while DX Macro-F1 decreases from 29.29 to 24.54. In contrast, Qwen3-VL-32B improves over Qwen3-VL-8B-I on aggregate multiple-choice performance and all three reported breast-imaging spatial settings (Tables 3 and 4). These checkpoint comparisons show that aggregate improvements can have different task-level compositions. They support examining the resulting performance profile rather than using model scale or the thinking designation as a substitute for task-specific evaluation. D.3.3PERFORMANCE ACROSS IMAGING SETTINGS AND SPATIAL TASKS Relative performance on categorical tasks does not consistently carry over to bounding-box prediction. On ISIC, Hulu-Med-7B achieves the highest diagnosis Accuracy and Macro-F1 among the evaluated checkpoints (56.3 and 34.4), but obtains 0.7 mIoU and 0.2Acc@IoU 0.5 on detection. Qwen3-VL-8B- 24 I achieves lower diagnosis scores (53.2 and 22.8), yet substantially higher detection scores (80.3 and 91.4; Table 10). These results describe complementary task profiles within the same dataset family. Because diagnosis and detection use different ISIC annotation subsets, the comparison concerns dataset-level performance rather than paired success and failure on individual images. The spatial scores measure end-to-end box prediction, including response validity under the evaluation protocol. Spatial rankings also change across imaging settings. BreastGPT-8B leads Qwen3-VL-32B on BUS-BRA detection, withAcc@IoU 0.5 of 82.80 versus 62.08, whereas Qwen3-VL-32B leads on VinDr-Mammo detection, with 12.86 versus 0.90 (Table 4). This ranking reversal shows that a modelâs advantage within one localization setting need not persist in another. Reporting each datasetâtask pair separately preserves this distinction and prevents a pooled spatial score from obscuring where the observed advantage applies. 25