Paper deep dive

OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis

Tianwei Lin, Zhongwei Qiu, Wenqiao Zhang, Jiang Liu, Yihan Xie, Mingjian Gao, Zhenxuan Fan, Zhaocheng Li, Sijing Li, Zhongle Xie, Peng LU, Yueting Zhuang, Ling Zhang, Beng Chin Ooi, Yingda Xia

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 86

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 7/21/2026, 2:15:47 AM

Summary

The paper introduces OmniCT, a unified slice-volume Large Vision-Language Model (LVLM) for Computed Tomography (CT) analysis. It addresses the fragmentation between slice-driven and volume-driven models by proposing Spatial Consistency Enhancement (SCE) and Organ-level Semantic Enhancement (OSE). SCE uses Volumetric Slice Composition and Tri-axial Positional Embedding to unify 2D/3D representations, while OSE aligns anatomical regions for semantic enhancement. The authors also introduce MedEval-CT, a large-scale dataset and benchmark for evaluating CT understanding.

Entities (10)

OmniCT · model · 98%MedEval-CT · datasetbenchmark · 97%Computed Tomography · modality · 95%Organ-level Semantic Enhancement · method · 95%Spatial Consistency Enhancement · method · 95%Tri-axial Positional Embedding · technique · 92%Volumetric Slice Composition · technique · 92%Large Vision-Language Models · concept · 90%Mixture of Experts Hybrid Projection · technique · 90%TotalSegmentor · tool · 88%

Relation Signals (9)

OmniCT → evaluatedon → MedEval-CT

confidence 97% · MedEval-CT: the largest slice-volume CT dataset and hybrid benchmark... OmniCT consistently outperforms existing methods

MedEval-CT → contains → MedEval-CT-Bench

confidence 95% · Based on the MedEval-CT-Dataset, we further construct MedEval-CT-Bench

MedEval-CT → contains → MedEval-CT-Dataset

confidence 95% · MedEval-CT-Dataset. We introduce MedEval-CT-Dataset, the largest unified CT imaging resource to date

OmniCT → uses → Organ-level Semantic Enhancement

confidence 95% · (ii) Organ-level Semantic Enhancement (OSE): segmentation and ROI localization explicitly align anatomical regions

OmniCT → uses → Spatial Consistency Enhancement

confidence 95% · We present OmniCT... which makes three contributions: (i) Spatial Consistency Enhancement (SCE)

Spatial Consistency Enhancement → includes → Tri-axial Positional Embedding

confidence 92% · SCE leverages... Tri-Axial Positional Embedding (TPE)

Spatial Consistency Enhancement → includes → Volumetric Slice Composition

confidence 92% · SCE leverages Volumetric Slice Composition (VSC), Tri-Axial Positional Embedding (TPE), and MoE Hybrid Projection (MHP)

Spatial Consistency Enhancement → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Computed Tomography (CT) is one of the most widely used and diagnostically information-dense imaging modalities, covering critical organs such as the heart, lungs, liver, and colon. Clinical interpretation relies on both slice-driven local features (e.g., sub-centimeter nodules, lesion boundaries) and volume-driven spatial representations (e.g., tumor infiltration, inter-organ anatomical relations). However, existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding: slice-driven LVLMs show strong generalization but lack cross-slice spatial consistency, while volume-driven LVLMs explicitly capture volumetric semantics but suffer from coarse granularity and poor compatibility with slice inputs. The absence of a unified modeling paradigm constitutes a major bottleneck for the clinical translation of medical LVLMs. We present OmniCT, a powerful unified slice-volume LVLM for CT scenarios, which makes three contributions: (i) Spatial Consistency Enhancement (SCE): volumetric slice composition combined with tri-axial positional embedding that introduces volumetric consistency, and an MoE hybrid projection enables efficient slice-volume adaptation; (ii) Organ-level Semantic Enhancement (OSE): segmentation and ROI localization explicitly align anatomical regions, emphasizing lesion- and organ-level semantics; (iii) MedEval-CT: the largest slice-volume CT dataset and hybrid benchmark integrates comprehensive metrics for unified evaluation. OmniCT consistently outperforms existing methods with a substantial margin across diverse clinical tasks and satisfies both micro-level detail sensitivity and macro-level spatial reasoning. More importantly, it establishes a new paradigm for cross-modal medical imaging understanding. Our project is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,875 characters extracted from source content.

Expand or collapse full text

OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis Tianwei Lin ∗1,2 , Zhongwei Qiu ∗2,3,1 , Wenqiao Zhang †1 , Jiang Liu 1 , Yihan Xie 1 , Mingjian Gao 1 , Zhenxuan Fan 1 , Zhaocheng Li 1 , Sijing Li 1,2 , Zhongle Xie 1 , Peng Lu 1 , Yueting Zhuang 1 , Ling Zhang 2 , Beng Chin Ooi 1 , Yingda Xia 2 1 Zhejiang University 2 DAMO Academy, Alibaba Group 3 Hupan Lab ∗ Equal contribution, † Corresponding author Computed Tomography (CT) is one of the most widely used and diagnostically information-dense imaging modalities, covering critical organs such as the heart, lungs, liver, and colon. Clinical interpretation relies on both slice-driven local features (e.g., sub-centimeter nodules, lesion boundaries) and volume-driven spatial representations (e.g., tumor infiltration, inter-organ anatomical relations). However, existing Large Vision–Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding: slice-driven LVLMs show strong generalization but lack cross-slice spatial consistency, while volume-driven LVLMs explicitly capture volumetric semantics but suffer from coarse granularity and poor compatibility with slice inputs. The absence of a unified modeling paradigm constitutes a major bottleneck for the clinical translation of medical LVLMs. We present OmniCT, a powerful unified slice–volume LVLM for CT scenarios, which makes three contributions: (i) Spatial Consistency Enhancement (SCE): volumetric slice composition combined with tri-axial positional embedding that introduces volumetric consistency, and an MoE hybrid projection enables efficient slice–volume adaptation; (i) Organ-level Semantic Enhancement (OSE): segmentation and ROI localization explicitly align anatomical regions, emphasizing lesion- and organ-level semantics; (i) MedEval-CT: the largest slice–volume CT dataset and hybrid benchmark integrates comprehensive metrics for unified evaluation. OmniCT consistently outperforms existing methods with a substantial margin across diverse clinical tasks and satisfies both micro-level detail sensitivity and macro-level spatial reasoning. More importantly, it establishes a new paradigm for cross-modal medical imaging understanding. Our project is available at https://github.com/ZJU4HealthCare/OmniCT. Email: lintw, wenqiaozhang@zju.edu.cn, qiuzhongwei.qzw@alibaba-inc.com Date: March 3, 2026 1 Introduction Large Vision–Language Models (LVLMs) have become a cornerstone of multi-modal artificial intelligence, demonstrating strong cross-modal representation and reasoning capabilities in both image understanding (Qiu et al., 2023; Li et al., 2024a; Zhu et al., 2025; Bai et al., 2025b) and video perception (Lin et al., 2023a; Li et al., 2024a; Zhang et al., 2025a; Yuan et al., 2025). Benefiting from large-scale pre-training and modality alignment, LVLMs achieve remarkable performance in open-domain tasks (Qiu et al., 2022; Yue et al., 2024; Fu et al., 2025), excelling in both generation and reasoning. These advances establish LVLMs as a universal paradigm for unified vision–language modeling, where the joint modeling of 2D and 3D modalities has emerged as a key design principle. In recent years, the potential of LVLMs in medical imaging has received increasing attention, with exploration in radiological imaging being particularly notable (Wu et al., 2025; Xu et al., 2025). However, most existing methods are tailored to process either CT slices (Chen et al., 2024a; Lin et al., 2025) or volumetric data (Bai et al., 2024; Hamamci et al., 2024c), with limited focus on cooperative processing. Slice-driven models leverage large-scale 2D pre-training to achieve strong vision–language alignment and perform well in tasks such as lesion arXiv:2602.16110v2 [cs.CV] 1 Mar 2026 Figure 1 (a) is the statistics of the proposed MedEval-CT-Dataset. (b) describes the simplified architecture of proposed OmniCT. (c) shows that OmniCT consistently surpasses all baselines on both slice-driven and volume-driven CT benchmarks. detection and radiology report description, yet they fail to capture cross-slice spatial consistency. In contrast, volume-driven models explicitly model voxel-level spatial structures, offering advantages in holistic spatial representation and organ-level reasoning. Nevertheless, these models often lack sensitivity to fine-grained abnormalities and boundary morphology, and their architectures are difficult to adapt to slice-level tasks, thereby limiting their applicability across diverse medical scenarios. This persistent dichotomy between slice and volume modeling constitutes a major bottleneck in the development of medical LVLMs. Among various medical imaging modalities, CT is one of the most widely used and dense in information, with hundreds of million performed each year globally. CT can cover critical organs such as the heart, lungs, liver, and colon, and is widely applied in essential tasks including disease screening (Hu et al., 2025), lesion assessment (Li et al., 2025b; Shui et al., 2025), and tumor staging (Bassi et al., 2025; Qiu et al., 2025). Its diagnostic process relies on both slice-level local imaging cues, such as sub-centimeter pulmonary nodules or hepatic lesion boundaries, and volume-level spatial–topological representations, such as tumor infiltration ranges or inter-organ anatomical relationships. Modeling along a single dimension alone cannot meet these dual requirements. Therefore, integrating the complementary strengths of 2D and 3D modeling within a unified framework is not only a central scientific challenge for CT understanding but also an inevitable step toward the clinical translation of medical LVLMs. We propose OmniCT (see Fig. 1), a powerful unified slice–volume LVLM for CT-centric understanding, which preserves the cross-modal alignment and generalization strengths of 2D models while integrating the spatial structural awareness of 3D models. To bridge the modality gap between slice and volume representations, we introduce a Spatial Consistency Enhancement (SCE) strategy. Unlike generic LVLMs that rely on frame sampling or key-frame stacking strategies (Xu et al., 2024; Li et al., 2024b; Huang et al., 2024), SCE performs Volumetric Slice Composition (VSC) by structurally combining adjacent slices along the channel dimension into locally consistent volumetric units, thereby retaining contextual spatial transitions. It further incorporates a Tri-axial Positional Encoding (TPE), which injects 3D positional encodings into visual representations to enable volumetric awareness while maintaining compatibility with slice-based inputs. In addition, a MoE Hybrid Projection (MHP) dynamically aligns slice and volume features within a shared representation space, ensuring semantic unification with the Large Language Models (LLMs). Overall, SCE injects robust 2D/3D spatial priors while achieving a balance between efficiency and adaptability. In clinical diagnosis, image interpretation is performed at the organ level, where observations and lesion localization are conducted within this scope (Shui et al., 2025). Building on this clinical requirement, we propose Organ-level Semantic Enhancement (OSE). OSE performs task-guided anatomical region localization, explicitly projecting critical organ regions into the token representation space and fusing them with global Figure 2 The architecture of OmniCT, a unified slice–volume LVLM paradigm. visual context, thereby embedding organ-centric semantics into the representation. It then applies an adaptive aggregation to compress long-sequence representations: this mechanism preserves overall information coverage while adaptively magnifying smaller organ regions and compressing larger ones, thus highlighting the most diagnostically relevant structures. In this way, OSE explicitly incorporates region priors with high task-relevant semantic load while improving the relevance and interpretability of models in clinical tasks. Existing medical benchmarks (Hu et al., 2024; Yue et al., 2024; Yamagishi et al., 2025) often adopt multi- modality designs to evaluate the general capability of LVLMs, yet they fall short in task alignment and clinical representativeness for CT interpretation. To address this gap, we introduce MedEval-CT, the first holistic evaluation framework dedicated to CT images. At the data level, MedEval-CT-Dataset consolidates 1.7M slice-driven and volume-driven VQA samples across 7 clinical task types, establishing the largest CT resource to date (Fig. 1 (a)). At the benchmark level, MedEval-CT-Bench organizes hybrid evaluations along clinical problem types and organ distributions. At the toolkit level, MedEval-CT-Factory standardizes input handling, feature construction, and multi-dimensional metrics, supporting statistical, semantic, and LLM-based evaluations. Collectively, MedEval-CT institutionalizes fairness and comparability in medical LVLM evaluation, while providing a scalable foundation for larger and more complex clinical scenarios. Experimental results on multiple CT-centric benchmarks show that OmniCT achieves substantial improvements over existing methods, as illustrated by the radar chart in Fig. 1 (c), validating the effectiveness of proposed unified slice–volume modeling paradigm. Our main contributions are: • Unified LVLM Paradigm for CT Imaging: Bridges the gap between slice and volume representations, injecting 3D spatial priors while retaining the efficiency of 2D alignment. • Representation Enhancements: We design SCE and OSE to bridge slice–volume gaps and embed organ-centric semantics, yielding spatially coherent and clinically meaningful representations. • MedEval-CT: Establishes the first holistic evaluation suite for CT imaging, augmented with 1.7M multimodal VQA samples, enabling fair, comparable, and scalable assessment of medical LVLMs. •Substantial Performances and Strong Baseline: OmniCT outperforms all medical LVLMs and general LVLMs with a significant margin across multiple slice- and volume-driven CT benchmarks, establishing a strong baseline for future research towards clinical medical LVLMs. 2 Methodology We propose OmniCT, a unified slice–volume LVLM for CT-centric understanding (Fig. 2). Unlike prior medical LVLMs restricted to either 2D slices or 3D volumes, OmniCT incorporates SCE and OSE to enable comprehensive CT representation. 2.1 Spatial Consistency Enhancement To bridge the representational gap between slices and volumes, we propose Spatial Consistency Enhancement (SCE) module, which injects volumetric priors into LLM while remaining compatible with slice-driven approaches. SCE leverages Volumetric Slice Composition (VSC), Tri-Axial Positional Embedding (TPE), and MoE Hybrid Projection (MHP) to unify 2D slices and 3D volumes into the LLM space, enabling localized spatial perception, spatial position encoding, and seamless alignment of slice/volume representations within the LLM space, respectively. Volumetric Slice Composition. For a 3D CT volumeV ∈ R D×H×W , whereD,H,Wrepresent the dimensions along the z, y, and x directions, respectively, VSC structurally concatenates adjacent slices along the z axis to construct locally consistent volumetric units:ˆs i =Concat(V 3i−2 ,V 3i−1 ,V 3i )for i= 1,...,⌊D/3⌋, where ˆs i ∈ R 3×H×W represents a reassembled unit that preserves cross-slice spatial transitions. For independent 2D slice inputsS=s 1 ,...,s n ,s i ∈ R 1×H×W , we simply replicates i along the channel axis to constructˆs i . In this way, both 2D slices and 3D volumes are unified as a series of reassembled units ˆ S=ˆs i |i∈[1,n], andˆs i has a size of 3× H × W, where 3 is the channel number. Tri-Axial Positional Embedding. Through volumetric slice composition, 2D slices or 3D volume are transposed into unified units ˆ S of sizeN s ×3×H×W, which are processed by a vision encoderφ v (·| θ v ) with parameters θ v to obtain patch-level visual tokens F: F = φ v ( ˆ S | θ v ) =φ v (ˆs 1 | θ v ),...,φ v (ˆs N s | θ v )∈ R N s ×H ′ ×W ′ ×d v .(1) Here,H ′ = H K andW ′ = W K denote the spatial dimensions before flattening the patch features, and the patch size for tokenization is 3×K×K.N s represents the number of unified reassembled units and can be regarded as a new depth dimension of reassembled units. To summarize,N s reassembled units are as inputs to generate N s × 1× H ′ × W ′ tokens with a dimension of d v . To explicitly inject global volumetric awareness, we construct sinusoidal positional encodingsP=P N s ,P H ′ ,P W ′ along the depthN s , heightH ′ , and widthW ′ dimensions of the reassembled units. This yields tokensZ enriched with 3D positional priors: Z =F ⊕ P =F ⊕ P N s ⊕ P H ′ ⊕ P W ′ ,Z ∈ R N s ×H ′ ×W ′ ×(d v +d z +d y +d x ) ,(2) where ⊕ denotes concatenating tokens with positional encodings along the feature dimension. MoEHybrid Projection. To mitigate token explosion and reduce redundancy in the visual token representation of volumetric units for native volume input, we first perform a token-level unshuffle operation onZ. This operation clusters spatially adjacentm× mtokens into more representations while preserving spatial relationships, resulting in newly generated token representations ˆ Z: ˆ Z =U(Z), ˆ Z ∈ R N s ×(H ′ /m)×(W ′ /m)×[(d v +d z +d y +d x )×m 2 ] ,(3) whereUdenotes the token-level unshuffle operation, withm= 1 for slice inputs to preserve original resolution. Subsequently, we employ a slice–volume Mixture of Experts Hybrid Projection (MHP)ψ(· | θ p ) to align features with the LLM’s representation space, formally expressed as: ˆ F = ψ( ˆ Z | θ p =W s ,W v ,W share ) = W share σ(W s ˆ Z · 1 slice + W v ˆ Z · 1 volume ),(4) whereσ(·) denotes the GELU activation function, and 1 slice and 1 volume are binary indicator functions that represent routing conditions for the slice and volume features, respectively (1 if the condition is satisfied, and 0 otherwise). The final tokens ˆ Fhas a size ofL× d f , whereL=N s × H ′ m × W ′ m represents the total number of tokens, andd f denotes the output feature dimension of MHP, which takes an input feature dimension of (d v + d z + d y + d x )× m 2 . Overall, the above SCE process generates unified CT tokens that are compatible with both slices and volumes, while embedding spatial positional awareness. These unified tokens are subsequently projected into the LLM representation space via MHP, serving as the input tokens for the LLM. 2.2 Organ-level Semantic Enhancement CT images are typically represented as high-resolution 3D volumes, often consisting of more than 150 axial slices with an in-plane resolution of 512× 512, whereas lesions usually occupy only a small and localized region. To enable clinically practical LVLMs capable of identifying abnormal features within such high-dimensional data, we introduce an Organ-level Semantic Enhancement (OSE) module within our unified framework, which consists of three components: anatomical region localization, semantic feature aggregation, and context fusion. Anatomical region localization. Given the visual token representation ˆ F ∈ R L×d h produced by SCE, we perform region-wise selection based on spatial priors of the target organo. The organ mask is denoted as M o ∈ R D×H×W , including 117 anatomical structures, which is generated by TotalSegmentor (Wasserthal et al., 2023). This mask ofD× H × Wis mapped to the token size by leveraging the scaling relationship between pixels and vision tokens, resulting in the organ-specific subset: ˆ F o = ˆ F[ ˆ M o ],where [ ˆ M o ] denotes mask-based indexing for token selection. ˆ F o represents the selected tokens of sizeL o × d h for organoby the organ mask ˆ M o . Adaptive organ-level feature aggregation. Since different organs exhibit significant variation in scale and token length, directly concatenating them with text tokens can lead to severe length imbalance. To address this issue, we design a fixed-dimensional discriminative aggregation functionAgg(·), which compresses ˆ F o into a unified size: ˆ f o = Agg( ˆ F o ), ˆ f o ∈ R L c ×d h , ˆ F o ∈ R L o ×d h ,(5) whereL c denotes the fixed number of aggregated tokens compressed fromL o . This aggregation not only reduces token redundancy but also introduces a “magnification effect" for small organs, enhancing fine-grained lesion features. Simultaneously, it applies a “compression effect" to large organs or global regions, effectively minimizing redundancy and preserving essential information. Finally, the organ-level aggregated representation ˆ f o is concatenated with the global visual tokens ˆ F to generate global-local vision tokens ˆ F OSE : ˆ F OSE = [ ˆ F; ˆ f o ],and combined with text tokensEas input to the LLM backbone, forming a semantically enhanced multimodal representation. Overall, OSE enhances discriminative capability at the local (organ) level while maintaining contextual consistency at the global level, thus delivering more relevant and interpretable representations for downstream clinical reasoning tasks. 2.3 Training Strategy After applying Spatial Consistency Enhancement and Organ-level Semantic Enhancement, we obtain enhanced medical visual features ˆ F OSE ∈ R (L+L c )×d h . Meanwhile, the text queryQ=q 1 ,...,q m is embedded with text embedding matrix φ t (·|θ t ): E = φ t (Q|θ t )∈ R m×d h . The two modalities are concatenated into a unified inputT= [ ˆ F OSE ;E], which is fed into the LLM to model the conditional probability distribution. The overall optimization objective is formulated as minimizing the autoregressive cross-entropy loss: min θ E (T ,y)∼D   − N y X t=1 logP(y t | y <t ;T ;θ)   , θ = ( θ p ,Pretraining Stage, θ p ,θ llm , Instruction Tuning Stage. (6) 3 Dataset Current medical benchmarks predominantly emphasize broad multi-modal capabilities, yet fall short in capturing the domain-specific demands of CT-based clinical interpretation. To bridge this gap, we introduce MedEval-CT, the first holistic evaluation framework for CT understanding, structured along three comple- mentary dimensions: Datasets (MedEval-CT-Dataset), Benchmarks (MedEval-CT-Bench), and Tools (Data Orchestration Engine and MedEval-CT-Factory). Figure 3 (a) and (b) illustrate the data distribution of MedEval-CT-Bench at the slice and volume levels, respectively, encompassing both the clinical-based categorization (4 types: GIR, MAI, AII, and CRD) and the organ-level distribution (13 organs). (c) presents the data engineering pipeline. 3.1 MedEval-CT MedEval-CT-Dataset. We introduce MedEval-CT-Dataset, the largest unified CT imaging resource to date, comprising over 1.7 million VQA samples from 170,280 independent 3D volumes and 327,063 standalone 2D slices. The two subsets are collected from distinct, non-overlapping sources, ensuring that the 2D slices are neither extracted nor sampled from the 3D volumes. To fully leverage the rich spatial structure and clinical semantics of volumetric CT data, each 3D volume is annotated with multiple VQA instances covering diverse diagnostic perspectives, whereas the 2D subset mainly supports slice-level interpretation with typically one question per slice. Overall, the dataset enables comprehensive evaluation of both 2D clinical understanding and 3D volumetric perception, accounting for 16.3% and 83.7% of the data, respectively. As shown in Fig. 3, the dataset is systematically partitioned across three dimensions: task types, clinical categories, and organs, enabling multi-faceted evaluation of LVLMs. For task types, it spans seven medical VQA scenarios, from structured to open-ended tasks: (1) Free-form QA (38.0%) (2) Multiple Choice (18.0%) (3) Short Answers (13.7%) (4) Report Generation (11.5%) (5) Caption (7.2%) (6) Dialogue (6.2%) and (7) Judgment (5.5%). Clinical categories reflect increasing difficulty, progressing from basic anatomical recognition to expert-level reasoning: (1) General Imaging Recognition (GIR, 21.9%) (2) Medical Abnormality Identification (MAI, 23.6%) (3) Advanced Imaging Interpretation (AII, 34.5%) and (4) Clinical Reasoning and Decision (CRD, 19.9%). Organ-wise, it covers lungs (32.4%), vessels (8.2%), heart (6.6%), liver (4.9%), kidneys (4.3%), and additional regions like spine, trachea, and esophagus, ensuring robust anatomical diversity. Overall, MedEval-CT surpasses existing datasets in scale and granularity, providing high-resolution distributions across tasks, clinical expertise, and organs to advance the development of LVLMs for CT imaging. Data sources and other details are presented in Table 8 and Appendix E. MedEval-CT-Bench. Based on the MedEval-CT-Dataset, we further construct MedEval-CT-Bench, the first systematic hybrid benchmark tailored for slice-volume CT. Its design emphasizes task–organ dual balance: on the one hand, we perform stratified sampling across different clinical problem types (GIR, MAI, AII, and CRD), ensuring full task-spectrum coverage from low-level interpretation to high-level reasoning; on the other hand, we maintain balanced organ representation, strengthening core organs (heart, lungs, liver, kidneys, etc.) while retaining long-tail structures (spine, trachea, esophagus, etc.), thereby guaranteeing fairness and comparability in clinical. To further improve clinical semantic fidelity, we propose clinical-granularity rewriting, which refines test questions to a more fine-grained clinical level and adds more confounding answer options while maintaining their diagnostic intent, ensuring they better reflect the variations encountered in real-world diagnostic scenarios. In summary, MedEval-CT-Bench represents significant advancements in task hierarchy, organ-level balance, and clinical authenticity, offering a more rigorous and demanding benchmark for CT understanding evaluation. Data Orchestration Engine. We introduce a Data Orchestration Engine to support the construction of MedEval-CT. The engine comprises four complementary modules that collaborate across key stages, forming a self-consistent medical knowledge pipeline. It enables end-to-end capabilities for large-scale sampling, clinical consistency verification, structured task mapping, and semantic refinement: (i) Corpus Selector: Combines LVLM capabilities with rule-based constraints to filter CT samples from multi-source imaging datasets, ensuring representativeness across modality (2D slice/3D volume), anatomy (heart, lungs, liver, etc.), resolution, and image quality. (i) Integrity Verifier: Leverages multi-modal reasoning and rule-based checks, supplemented by a 10% manual audit, to guarantee alignment between images and texts in modality, organ semantics, and pairing consistency. (i) Task Mapper: Maps qualified samples to four clinical task categories and thirteen organ classes, ensuring balanced task complexity and anatomical coverage in MedEval-CT-Bench. (iv) Semantic Refiner: Rewrites test questions under clinical context, introducing synonymous phrasing, terminology variations, and subtle confounding options to generate semantically similar but more discriminative multiple-choice items, thereby enhancing the benchmark’s ability to evaluate clinical reasoning. Overall, the engine constructs a large-scale yet distribution-balanced MedEval-CT-Dataset while ensuring that MedEval-CT-Bench achieves reliability in terms of task hierarchy, organ balance, and clinical authenticity. Details are provided in the Appendix F. MedEval-CT-Factory. We introduce MedEval-CT-Factory, an institutionalized evaluation factory designed to address the heterogeneity of inputs, features, and outputs in medical LVLMs. At the input level, the Factory standardizes diverse CT data formats, including DICOM, NIfTI, arrays, and slice sequences, enabling seamless 2D/3D processing. At the feature level, it unifies model inputs (single images, multi-slice sequences, videos, or volumes) via frame sampling, resampling, and projection strategies. At the output level, it provides a multi-layer evaluation protocol, ranging from statistical metrics (BLEU (Papineni et al., 2002), ROUGE (Lin, 2004)), to semantic metrics (BERTScore (Zhang et al., 2019), embedding similarity (Zhang et al., 2025b)), and further to LLM-based evaluation simulating clinical reasoning. Overall, MedEval-CT-Factory streamlines complex engineering workflows into a standardized framework, ensuring comparability across models. Serving as the fourth pillar of the MedEval-CT paradigm alongside the Dataset, Bench, and Data Engine, it is designed as a toolbox to enhance both the efficiency and fairness of LVLM evaluation in the CT domain. (The framework and details of MedEval-CT-Factory are shown in Appendix G). 4 Experiments Baseline Comparisons. To comprehensively evaluate the performance of OmniCT against existing open-source medical LVLMs as well as strong general-purpose LVLMs, we select a diverse set of baseline models that systematically cover both the general-to-medical spectrum and the 2D-to-3D setting. For the 2D slice-based benchmarks, we include general-purpose LVLMs such as InternVL3 (Zhu et al., 2025), Qwen2.5-VL (Bai et al., 2025b), and GPT-5 (Wang et al., 2025), together with representative medical-domain models including HealthGPT (Lin et al., 2025), HuatuoGPT-Vision-Qwen2.5 (Chen et al., 2024b), MedGemma-4B-IT (Sellergren et al., 2025), MedVLM-R1-2B (Pan et al., 2025), Lingshu (Xu et al., 2025), and RadFM (Wu et al., 2025). For the 3D volume-based benchmarks, we compare OmniCT with strong general LVLMs (MiniCPM-V-4.5 (Yu et al., 2025), Qwen2.5-VL, and GPT-5) as well as specialized 3D medical LVLMs, including M3D-LaMed (Bai et al., 2024), CT-CHAT (Hamamci et al., 2024a), and RadFM. The Data Details, Model Details, and Implementation Details are shown in Appendix E. 4.1 Main Experiments Table 1 Ablation analysis of OmniCT. SCE OSE Public Bench.MedEval-CT-Bench 2D3DOrgan TaskAvg. --78.6862.1776.5178.4177.62 ✓-80.14 63.6876.79 78.69 78.06 -✓ 80.74 65.3777.02 79.42 78.62 ✓81.4566.1578.2480.2779.62 Slice-driven Understanding. We systematically evaluate OmniCT on four mainstream VQA benchmarks.As shown in the Table 2, medical LVLMs (e.g., HuatuoGPT-V-Qwen2.5, MedGemma) demonstrate relatively strong medical semantic understanding in certain tasks but remain limited in overall performance, often encountering bottlenecks on complex tasks. In contrast, general LVLMs achieve competitive or even superior results on some benchmarks, reflecting their strengths in language reasoning, but lack adaptation to CT images. For comparison, RadFM, although capable of handling both slice and volume inputs, achieves the weakest performance across all slice benchmarks, with an average score of only 32.12, failing to meet the demands of fine-grained CT tasks. Under the same evaluation protocol, OmniCT consistently surpasses existing models at both 3B and 7B scales: the 7B version achieves an average Table 2 The comparison of OmniCT with other LVLMs on 2D CT benchmarks. Model#Params SLAKEVQA-RADOmniMedVQARadFig-VQA Avg. Close OpenClose OpenTask1 Task2Easy Medium Hard Med-LVLM (Slice-centric) HealthGPT4B74.74 56.3371.88 33.4557.36 54.2070.81 71.22 72.9062.54 HuatuoGPT-V-Qwen2.57B72.68 44.1972.92 35.6871.07 83.0776.56 73.76 71.7466.85 MedGemma-4B-IT4B68.04 53.9556.25 33.5561.42 67.0064.59 65.40 64.2059.38 MedVLM-R1-2B2B----59.90 67.3755.50 54.51 55.56- Lingshu7B80.93 74.2375.00 34.6268.02 69.9777.51 78.48 75.2270.44 General-LVLM InternVL38B73.20 60.8869.79 34.7963.96 71.7868.42 70.46 68.5564.65 Qwen2.5-VL8B69.59 47.8669.59 35.5462.94 65.9265.07 69.45 67.1061.45 GPT-5-78.35 45.8670.83 41.0567.00 69.1080.86 78.90 81.7468.19 Med-LVLM (Multi-granularity) RadFM14B51.03 43.8853.12 20.2930.97 28.2923.92 19.75 17.8332.12 OmniCT (Ours)3B77.84 85.3270.83 30.0197.46 97.2579.43 82.03 79.1377.71 OmniCT (Ours)7B 85.05 87.2076.04 36.3297.9798.7082.30 85.82 83.6281.45 score of 81.45, exceeding the second-best model Lingshu by more than +11.01. These results demonstrate the robustness and comprehensiveness of OmniCT on slice-driven tasks. Volume-driven Understanding. As shown in Table 3, we further assess OmniCT on M3D, CT-RATE, and 3D- RAD to evaluate its volumetric perception capability for CT volume. Results show that existing volume-driven medical LVLMs (e.g., M3D-LaMed-7B/4B, CT-CHAT) achieve strong performance on specific subtasks—for example, CT-CHAT reaches 86.46 on CT-RATE multi-choice—but their overall averages remain below 36, highlighting limitations in coverage and stability. General LVLMs also exhibit strong cross-modal generalization in certain volume tasks, with GPT-5 leading multiple subtasks on 3D-RAD; however, their performance is highly inconsistent and lacks domain adaptation to CT volume. By contrast, OmniCT achieves clear advantages at both 3B and 7B scales: the 3B version reaches 87.38 on CT-RATE multi-choice with an average of 63.48, while the 7B version achieves 85.69 on the LTD task of 3D-RAD, pushing its overall average to 66.15—significantly outperforming all compared models. In addition, considering the significant importance of CT report generation, we performed 18-class abnormality label prediction for the report generation task on CT-RATE using fine-tuned RadBERT (Yan et al., 2022). The results show that OmniCT outperforms most volume-driven CT models and previous unified models (see Table 16), and performs similarly to models specifically designed for CT volume report generation (Hamamci et al., 2024b; Di Piazza et al., 2025). This validates the superiority of OmniCT in 3D spatial modeling and cross-task consistency. Across both slice-driven and volume-driven benchmarks, OmniCT demonstrates stable and comprehensive superiority at different scales, highlighting its holistic perception of spatial–semantic features in CT volume understanding tasks. 4.2 Ablation Analysis We conduct a systematic ablation study on the proposed SCE and OSE modules across multiple public 2D/3D CT benchmarks and our MedEval-CT-Bench. Notably, we keep the VSC and the MHP fixed throughout all experiments, since they serve as fundamental mechanisms for coupling 2D slice understanding with 3D volumetric perception. Under this prerequisite design, we further analyze the contribution of SCE and OSE. Results are summarized in Table 1. On the 2D slice benchmarks, the baseline achieves an average score of 79.38; introducing SCE alone improves performance to 80.14, while adding OSE alone yields 80.74. When both are combined, the performance further increases to 81.45, achieving the best results. On the 3D volume benchmarks, the baseline starts at 62.17; adding SCE improves it to 63.68, while adding OSE alone boosts it to 65.37. The complete model combining both modules reaches the highest score of 66.15. On the MedEval-CT-Bench, OmniCT consistently outperforms the baseline with the addition of the SCE and OSE. The improvements in both organ-level and clinical-level tasks further validate the effectiveness of these two modules. Overall, both SCE and OSE contribute significantly to performance gains, with even stronger effects observed on volume-driven tasks, demonstrating the effectiveness and complementarity of the proposed Table 3 The comparison of OmniCT with other LVLMs on 3D CT benchmarks. Model#Params M3DCT-RATE3D-RAD Avg. Cap Close Open Multi choice Clinical Entity Report I.O. A.D. E.D. STD. LTD. Med-LVLM (Volume-centric) M3D-LaMed-7B7B24.79 75.78 56.0947.44 18.1516.1816.85 16.71 18.00 25.47 24.1730.88 M3D-LaMed-4B4B46.30 75.08 53.8359.29 13.6613.4617.60 17.49 40.25 25.40 24.3135.15 CT-CHAT8B21.21 35.88 21.8186.46 49.9546.7631.56 29.98 45.33 12.95 13.6835.97 General-LVLM MiniCPM-V-4_59B18.44 43.20 26.8969.21 26.2123.2128.03 29.80 30.98 12.70 16.3229.54 Qwen2.5-VL8B22.62 48.64 28.9961.34 37.5126.8430.51 30.60 41.28 9.19 13.0531.87 GPT-5-21.66 50.36 33.6064.27 34.4424.8632.98 35.22 67.00 59.07 77.9745.59 Med-LVLM (Multi-granularity) RadFM14B22.62 30.39 19.8263.93 19.5217.9223.25 24.67 29.20 44.11 42.9930.77 OmniCT (Ours)3B27.75 81.24 62.1687.38 63.4351.6752.02 51.43 84.75 64.43 72.0563.48 OmniCT (Ours)7B26.61 83.84 63.8889.80 63.9952.4853.68 51.97 87.77 67.91 85.6966.15 enhancements. Moreover, MHP is a crucial component for synergizing 2D and 3D modalities. We provide further analysis of its beneficial role in cross-modal generalization in Appendix H.2 (i). 4.3 In-Depth Study (i) Performance Advantages of Mixed Data Training. As shown in Figure 4 (a), OmniCT consistently achieves the best performance across different proportions of mixed data, demonstrating its strong adaptability to cross-modal modeling. OmniCT exhibits strong performance even under single-modality training. We attribute this behavior to the combination of a unified single-tower semantic space and the MHP, which together enable projection patterns learned from slices to extend naturally to volumes, and symmetrically allow volume-trained representations to transfer back to slice. A more detailed analysis of this mechanism is provided in Appendix H.2 (vi). We conduct balanced sampling to verify that OmniCT can facilitate effective knowledge fusion between 2D and 3D modalities through mixed training. (i) 2D Encoders vs. 3D Encoders. Given the inherent differences in design objectives and input modes, directly applying 3D encoders to 2D inputs often requires artificial adaptations such as depth replication, which compromises the fairness of comparison. Therefore, we conduct evaluations in native 3D settings. As shown in Figure 4 (b), even though M3D-CLIP (Bai et al., 2024) is pretrained with contrastive learning on the M3D dataset, it does not exhibit a clear advantage over 2D encoders such as DINOv3 (245 tokens) (Siméoni et al., 2025) and SigLIP (405 tokens) (Zhai et al., 2023), despite using the largest number of visual tokens (512). These results indicate that, at this stage, 2D encoders not only provide a more natural compatibility with both 2D and 3D inputs but also demonstrate stronger generalization across tasks, organs, and modalities. To assess the generality of this finding, we additionally evaluate several recent native 3D encoders (Wan et al., 2025; Wang et al., 2023) under the same protocol; the results are provided in Table 4. It is worth noting that we are not claiming that 2D features can fully represent 3D volumes. Instead, we offer a more measured assessment: at the current stage, through structured reorganization and volume-level embedding, more generalizable 2D encoders can robustly carry 3D spatial information. This design does not collapse dimensionality; rather, it retains spatial structures and relationships that remain interpretable from a 3D perspective on top of a 2D semantic backbone. (i) Organ- and Task-level Performance Analysis. On the organ level (Figure 4 (c)), OmniCT consistently outperforms baselines across the chest, transition zone, and abdomen. The advantage is particularly striking for anatomically challenging small organs such as the pancreas and esophagus, where most existing LVLMs suffer severe performance degradation. This highlights OmniCT’s ability to capture fine-grained organ semantics and boundary cues, effectively filling a critical blind spot of prior models in handling complex anatomical structures. On the task level (Figure 4 (d)), performance shows a clear gradient with respect to clinical difficulty: while most models display a significant gap between low-level anatomical recognition and high-level reasoning, OmniCT maintains consistently strong results across all levels, substantially narrowing this gap. This stability demonstrates that OmniCT not only enhance local anatomical discriminability but Figure 4 (a) Comparison of OmniCT with 2D/3D LVLMs on 2D/3D benchmarks using 30%, 100% training data of 2D, 3D, and mixed 2D/3D. (b) The study of using a 3D vision encoder, 2D vision encoders by different pre-training ways. (c) Per-organ performance heatmap of 2D/3D models and OmniCT on 2D/3D MedEval-CT-Bench. (d) Performance heatmaps by clinical task category and bar charts comparing performance with clinical knowledge requirements across task categories. Table 4 2D vs. 3D Encoder Comparison. EncoderToken Budget RatioPlanePhaseOrganAbnormalityLocationAvg. M3D-CLIP1.26×99.084.877.178.263.180.4 VideoMAEv20.97×91.374.776.975.362.876.2 Wan2.1-VAE1.42×97.976.477.174.763.978.0 DINOv30.61×99.588.077.878.965.381.9 SigLip 1.00×99.590.278.479.267.482.9 also reinforce consistency in clinical reasoning. 5 Related Work Slice-driven Medical LVLMs. With the success of Large Language Models in general language (Liu et al., 2024; Yang et al., 2025) and vision (Qiu et al., 2020; Bai et al., 2025a), early explorations focused on adapting general LVLM paradigms to the medical domain, such as LLaVA-Med (Li et al., 2023) and Med-Flamingo (Moor et al., 2023), which leveraged medical image–text pairs and instruction data to enable initial medical capabilities. Subsequently, a series of more general-purpose medical LVLMs emerged, including RadFM (Wu et al., 2025), BiomedGPT (Luo et al., 2023), HuatuoGPT-Vision (Chen et al., 2024a), and Lingshu (Xu et al., 2025). These models advanced the field through large-scale data curation (Bansal et al., 2024), reasoning-enhanced training strategies (Pan et al., 2025; Xu et al., 2025), multi-task generalization (Jiang et al., 2024), and domain-specific knowledge integration (Sellergren et al., 2025). Recently, models such as CXR-LLaVA (Lee et al., 2025) and EyecareGPT (Li et al., 2025a) have demonstrated stronger adaptability and diagnostic value in modality-specific and specialty-oriented tasks (Xie et al., 2025; Hao et al., 2025). Nevertheless, despite substantial progress in data scale, architectural design, and task diversity, slice-driven medical LVLMs remain constrained by their reliance on planar inputs, limiting their ability to capture the spatial consistency and cross-slice dependencies essential for CT understanding. Volume-driven Medical LVLMs. To overcome the limitations of 2D modeling, research has increasingly turned to 3D volumetric imaging, employing dedicated datasets, 3D encoders, and cross-modal alignment modules to strengthen spatial modeling in clinical tasks (Wu et al., 2025). M3D-LaMed (Bai et al., 2024) established a comprehensive evaluation system across multiple volumetric medical tasks, while CT-CHAT (Hamamci et al., 2024c) introduced paired chest CT data and an architecture tailored for fine-grained analysis and dialog-based interaction. At the methodological level, Med-2E3 (Shi et al., 2024) combined 2D and 3D encoders and enhanced reasoning consistency through dynamic cross-slice scoring, whereas Med3DInsight (Chen et al., 2024c) aligned a 3D encoder with a 2D LVLM, achieving strong performance in both segmentation and classification. Nevertheless, the lack of a unified clinical evaluation framework and efficient slice–volume collaboration mechanisms continues to limit adaptability and scalability. 6 Conclusion We propose OmniCT, a unified slice-volume LVLM for CT analysis. Through the proposed SCE and OSE modules, OmniCT achieves spatially coherent and clinically grounded representations, leading OmniCT to realize new state-of-the-art performances on multiple benchmarks. We further propose MedEval-CT, a unified, fair, and comprehensive evaluation framework for 2D/3D CT analysis. Detailed evaluations reveal that existing general-purpose and medical LVLMs exhibit significant performance biases across clinical tasks for different organs. In contrast, OmniCT demonstrates exceptional capability with balanced performance across all organs, which will encourage LVLMs to focus on enhancing clinical capabilities for various organs in the CT domain. Acknowledgement This work has been supported in part by the NSFC (No. 62436007), the China Postdoctoral Science Foundation under Grant Number 2024M752794, the ZJNSF (No. LZ25F020004), the Key Research and Development Projects in Zhejiang Province (No. 2025C01128, 2025C01030, 2025C02156), Zhejiang University Education Foundation Qizhen Scholar Foundation. References Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Anthropic AI. The claude 3 model family: Opus, sonnet, haiku. 2024.https://api.semanticscholar.org/CorpusID: 268232499. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578, 2024. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b. Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. Hritik Bansal, Daniel Israel, Siyan Zhao, Shufan Li, Tung Nguyen, and Aditya Grover. Medmax: Mixed-modal instruction tuning for training biomedical assistants. arXiv preprint arXiv:2412.12661, 2024. Pedro RAS Bassi, Mehmet Can Yavuz, Kang Wang, Xiaoxi Chen, Wenxuan Li, Sergio Decherchi, Andrea Cavalli, Yang Yang, Alan Yuille, and Zongwei Zhou. Radgpt: Constructing 3d image-text tumor datasets. arXiv preprint arXiv:2501.04678, 2025. Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280, 2024a. Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024b. https://arxiv.org/abs/2406.19280. Qiuhui Chen, Huping Ye, and Yi Hong. Med3dinsight: Enhancing 3d medical image understanding with 2d multi-modal large language models. arXiv preprint arXiv:2403.05141, 2024c. Theo Di Piazza, Carole Lazarus, Olivier Nempont, and Loic Boussel. Ct-agrg: Automated abnormality-guided report generation from 3d chest ct volumes. In 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), pages 01–05. IEEE, 2025. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025. Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, and Zuozhu Liu. 3d-rad: A comprehensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks. arXiv preprint arXiv:2506.11147, 2025. Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, et al. A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero-shot detection of abnormalities. CoRR, 2024a. Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. Ct2rep: Automated radiology report generation for 3d medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 476–486. Springer, 2024b. Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Doga, Omer Faruk Durugol, Weicheng Dai, Murong Xu, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834, 2024c. Jing Hao, Yuxuan Fan, Yanpeng Sun, Kaixin Guo, Lizhuo Lin, Jinrong Yang, Qi Yong H Ai, Lun M Wong, Hao Tang, and Kuo Feng Hung. Towards better dental ai: A multimodal benchmark and instruction dataset for panoramic x-ray analysis. arXiv preprint arXiv:2509.09254, 2025. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pages arXiv–2507, 2025. Can Hu, Yingda Xia, Zhilin Zheng, Mengxuan Cao, Guoliang Zheng, Shangqi Chen, Jiancheng Sun, Wujie Chen, Qi Zheng, Siwei Pan, et al. Ai-based large-scale screening of gastric cancer from noncontrast ct imaging. Nature Medicine, pages 1–9, 2025. Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024. De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In European Conference on Computer Vision, pages 202–218. Springer, 2024. Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models. arXiv preprint arXiv:2404.10237, 2024. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. Seowoo Lee, Jiwon Youn, Hyungjin Kim, Mansu Kim, and Soon Ho Yoon. Cxr-llava: a multimodal large language model for interpreting chest x-ray images. European Radiology, pages 1–13, 2025. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023. Sijing Li, Tianwei Lin, Lingshuai Lin, Wenqiao Zhang, Jiang Liu, Xiaoda Yang, Juncheng Li, Yucheng He, Xiaohui Song, Jun Xiao, et al. Eyecaregpt: Boosting comprehensive ophthalmology understanding with tailored dataset, benchmark and model. arXiv preprint arXiv:2504.13650, 2025a. Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2024b. Yingtai Li, Haoran Lai, Xiaoqian Zhou, Shuai Ming, Wenxin Ma, Wei Wei, and S. Kevin Zhou. More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era . In proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2025, volume LNCS 15966. Springer Nature Switzerland, September 2025b. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023a. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, et al. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. arXiv preprint arXiv:2502.09838, 2025. Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. Springer, 2023b. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442, 2023. Vishal Monga and Brian L Evans. Perceptual image hashing via feature points: performance evaluation and tradeoffs. IEEE transactions on Image Processing, 15(11):3452–3465, 2006. Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023. OpenAI. Gpt-5 model documentation.https://openai.com/zh-Hans-CN/index/introducing-gpt-5/, 2025. Ac- cessed: 2025-08-07. Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 337–347. Springer, 2025. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. Zhongwei Qiu, Kai Qiu, Jianlong Fu, and Dongmei Fu. Dgcn: Dynamic graph convolutional network for efficient multi-person pose estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11924–11931, 2020. Zhongwei Qiu, Huan Yang, Jianlong Fu, and Dongmei Fu. Learning spatiotemporal frequency-transformer for compressed video super-resolution. In European conference on computer vision, pages 257–273. Springer, 2022. Zhongwei Qiu, Qiansheng Yang, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Chang Xu, Dongmei Fu, and Jingdong Wang. Psvt: End-to-end multi-person 3d pose and shape estimation with progressive video transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21254–21263, 2023. Zhongwei Qiu, Hanqing Chao, Tiancheng Lin, Wanxing Chang, Zijiang Yang, Wenpei Jiao, Yixuan Shen, Yunshuo Zhang, Yelin Yang, Wenbin Liu, et al. Bridging local inductive bias and long-range dependencies with pixel-mamba for end-to-end whole slide image analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22738–22747, 2025. Ronan L.M. Rocov2-radiology (revision 5d66908), 2024.https://huggingface.co/datasets/eltorio/ ROCOv2-radiology. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. Yiming Shi, Xun Zhu, Ying Hu, Chenyi Guo, Miao Li, and Ji Wu. Med-2e3: A 2d-enhanced 3d medical multimodal large language model. arXiv preprint arXiv:2411.12783, 2024. Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang, Xianghua Ye, Tingbo Liang, Qi Zhang, et al. Large-scale and fine-grained vision-language pre-training for enhanced ct image understanding. arXiv preprint arXiv:2501.14548, 2025. Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2, 2024. Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549–14560, June 2023. Shansong Wang, Mingzhe Hu, Qiang Li, Mojtaba Safari, and Xiaofeng Yang. Capabilities of gpt-5 on multimodal medical reasoning. arXiv preprint arXiv:2508.08224, 2025. Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalsegmentator: robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence, 5(5):e230024, 2023. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communications, 16(1):7866, 2025. Yihan Xie, Sijing Li, Tianwei Lin, Zhuonan Wang, Chenglin Yang, Yu Zhong, Wenqiao Zhang, Haoyuan Li, Hao Jiang, Fengda Zhang, et al. Heartcare suite: Multi-dimensional understanding of ecg with raw multi-lead signal modeling. arXiv preprint arXiv:2506.05831, 2025. Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044, 2025. Yosuke Yamagishi, Shouhei Hanaoka, Yuta Nakamura, Tomohiro Kikuchi, Akinobu Shimizu, Takeharu Yoshikawa, and Osamu Abe. Radfig-vqa: A multi-imaging-modality radiology benchmark for evaluating vision-language models in clinical practice. In MICCAI2025 Joint Springer proceedings: 1st Workshop on Multimodal Large Language Models (MLLMs) in Clinical Practice. Springer, 2025. in press. An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. Radbert: adapting transformer-based language models to radiology. Radiology: Artificial Intelligence, 4(4):e210258, 2022. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, Bokai Xu, Junbo Cui, Yingjing Xu, Liqing Ruan, Luoyuan Zhang, Hanyu Liu, Jingkun Tang, Hongyuan Liu, Qining Guo, Wenhao Hu, Bingxiang He, Jie Zhou, Jie Cai, Ji Qi, Zonghao Guo, Chi Chen, Guoyang Zeng, Yuxuan Li, Ganqu Cui, Ning Ding, Xu Han, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe, 2025.https://arxiv.org/abs/2509.18154. Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video llm. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18970–18980, 2025. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025a. Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025b. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. Appendix A Appendix Section B. LLM usage statement. Section C. Notation Table. Section D. Extended Related Work. Section E. Implementation Details. Section F. Data Orchestration Engine. Section G. Mechanism of MedEval-CT-Factory Section H. Supplementary experiments. B LLM usage statement In this work, we primarily employ LLMs in the following two aspects: (i) Data construction: We use LLMs for data annotation, cleaning, filtering, and rewriting, thereby building MedEval-CT-Dataset and MedEval- CT-Bench. (i) Manuscript polishing: We leverage LLMs to review the grammar and improve the clarity and accuracy of the manuscript, ensuring that our methodology is properly and comprehensively presented. (i) Model evaluation: We include several advanced LLMs as baseline methods for comparison. For open-weight models, we download the publicly released checkpoints and conduct inference locally. For closed-source models, we perform evaluation through their official APIs. All baseline results are obtained under a unified evaluation protocol to ensure fairness and comparability. C Notation Table To provide a comprehensive overview of the notations used throughout the paper, we present a summary of key symbols and their corresponding definitions in Table 5. This table serves as a convenient quick reference, covering the main variables and operators involved in our formulation. We hope that this notation list facilitates the understanding of our methodology and improves the readability of the paper by enabling readers to easily recall the meaning of each symbol. D Extended Related Work In recent times, the release of multiple Multi-modal Large Language Models (MLLMs) has driven innovations in vision-language fusion, long temporal sequence processing, and scenario adaptability (Hong et al., 2025; Team et al., 2025), laying a solid foundation for cross-domain applications. Prominent foundation models such as Qwen2.5-VL (Bai et al., 2025b), GPT-4o (Achiam et al., 2023), Claude 3.5 (AI, 2024), InternVL3 (Zhu et al., 2025), and the latest GPT-5 (Wang et al., 2025) have continuously advanced in multi-modal understanding, long-sequence processing, multi-task learning, and vertical domains like healthcare (Arora et al., 2025), demonstrating exceptional potential. These advancements are primarily driven by high-quality data curation and iterative algorithmic optimization. However, as foundation models, maintaining a balance between general-purpose capabilities and domain-specific expertise remains a significant challenge. E Implementation Details DataDetails. For 2D slice evaluation, we construct test sets based on SLAKE (Liu et al., 2021), VQA-RAD (Lau et al., 2018), OmniMedVQA (Hu et al., 2024), and RadFig (Yamagishi et al., 2025), where all samples are systematically filtered by the data engine (Section 3) to retain only high-quality CT VQA data. For 3D volume evaluation, we adopt existing benchmarks including M3D (Bai et al., 2024), CT-RATE (Hamamci et al., 2024a), and 3D-RAD (Gai et al., 2025) to cover the full spectrum of CT volumetric scenarios. Regarding evaluation metrics, Accuracy is used for closed-end and multiple-choice tasks, while open-ended QA is assessed by a weighted combination of BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), Token-F1 (Saab et al., 2024), and BERTScore (Zhang et al., 2019), balancing lexical matching with semantic alignment to achieve multi-level quality measurement. For data pre-processing, all CT volumes with preserved metadata are windowed to [−1000, 1000] and resampled to 32× 384× 384. Model Details and Implementation Details. We use siglip-so400m-patch14-384 (Zhai et al., 2023) as the vision encoder and Qwen2.5 (Team, 2024) as the backbone LLM, with AdamW (Loshchilov and Hutter, 2017) as the optimizer. During pretraining, only the MoE Hybrid Projection is updated to perform cross-modal alignment, with a learning rate of 2×10 −4 . In the vision instruction tuning stage, both the projection layer and LLM parameters are optimized, with the learning rate reduced to 5×10 −5 to ensure stable convergence. All experiments are trained with a global batch size of 256 and a warmup–cosine learning rate scheduler. Unless otherwise specified, experiments are conducted under the 7B parameter scale. The specific hyperparameter settings can be found in Table 6. F Data Orchestration Engine The construction of MedEval-CT is powered by the proposed Data Orchestration Engine, which integrates four complementary modules. Specifically, the Corpus Selector and Integrity Verifier are implemented with Qwen2.5-VL-72B (Bai et al., 2025b), while the Task Mapper and Semantic Refiner leverage Qwen3-237B- A3B (Yang et al., 2025), thereby exploiting the complementary strengths of different models in large-scale data filtering and semantic refinement. The specific prompt designs for each module can be found in Fig. 5. To ensure the reliability and independence of MedEval-CT, we proactively conduct a systematic audit of potential data overlap during its construction. For the slice datasets, we note that ROCOv2 (Ronan L.M., 2024), PubMedVision (Chen et al., 2024b), LLaVA-Med (Li et al., 2023), and RadFig-VQA (Yamagishi et al., 2025) originate from PMC-OA (Lin et al., 2023b). Although these four datasets follow their own automated curation pipelines, we further apply a two-stage deduplication strategy within them: perceptual image hashing (pHash) (Monga and Evans, 2006) is used to cluster visually similar images, followed by BiomedCLIP (Zhang et al., 2023) feature matching to remove image–text pairs with high semantic similarity. For the volume datasets, we strictly adhere to the official splits of M3D (Bai et al., 2024), CT-RATE (Hamamci et al., 2024a), and 3D-RAD (Gai et al., 2025); even in cases where datasets share underlying CT volumes, we avoid any cross-dataset training or evaluation. Throughout the pipeline, we retain only modality-consistent and high-quality CT scans, filtering out blurry, artifact-heavy, or low-resolution samples. These measures allow MedEval-CT to maintain strict separation in data sourcing, partitioning, and deduplication, effectively minimizing the risks of training–testing contamination and data leakage. To further address the concern that using Qwen-family models in both the data pipeline and the LLM base model might introduce family-specific bias or circularity, we additionally instantiate OmniCT with a different LLM backbone, Phi-4-mini (Abouelenin et al., 2025), while keeping the training data and optimization protocol unchanged. As shown in Table 7, OmniCT with Phi-4-mini achieves performance that is highly comparable to, and on several benchmarks slightly better than, the Qwen2.5-3B variant across both slice- driven (SLAKE, VQA-RAD, OmniMedVQA, RadFig-VQA) and volume-driven (M3D, CT-RATE, 3D-RAD, MedEval-CT-Bench) benchmarks. This consistency indicates that the observed gains mainly stem from the proposed unified framework itself rather than from any base model-specific preference or bias induced by the models used in the data construction pipeline. G Mechanism of MedEval-CT-Factory The logical structure of MedEval-CT-Factory is illustrated in Figure 6. This section explains its design motivations and module responsibilities from a framework-level perspective. Unified Processing of Heterogeneous Formats. MedEval-CT-Factory begins at the input level, where commonly Figure 5 Prompt template of data orchestration engine for generating MedEval-CT. used medical imaging formats are standardized. Medical data often come in diverse forms such as DICOM, NIfTI, NRRD, 3D arrays, RGB slices, and slice sequences, which differ significantly in metadata organiza- tion, spatial resolution, and storage layouts. The Factory maps these heterogeneous inputs into a unified representation through designated loading rules, enabling subsequent modules to perform slice-volume unified processing without relying on format-specific operations. Lightweight but Unified Feature Construction. Building on the standardized inputs, the Factory provides a lightweight yet flexible feature construction layer. Instead of enforcing any model-specific preprocessing pipeline, it offers general-purpose mechanisms such as frame sampling, slice aggregation, 2D–3D projection, and resampling, allowing various LVLMs to interface with the evaluation workflow in a consistent manner. Figure 6 MedEval-CT-Factory enables standardized prepro- cessing and fair, consistent evaluation of medical LVLMs across benchmarks. Multi-dimensional Evaluation Protocols. At the out- put level, MedEval-CT-Factory integrates multi- ple evaluation strategies to accommodate the di- versity of outputs produced by medical LVLMs. Rather than imposing a rigid scoring pipeline, it provides a composable and extensible evalu- ation space: (i) statistical metrics (BLEU (Pa- pineni et al., 2002), ROUGE (Lin, 2004), ME- TEOR (Banerjee and Lavie, 2005)) for measur- ing surface-level textual alignment; (i) semantic metrics (BERTScore (Zhang et al., 2019), embed- ding similarity (Zhang et al., 2025b)) for assessing semantic correspondence; and (i) LLM-based eval- uation for simulating clinical reasoning, offering more qualitative judgments aligned with medical scenarios. Users may flexibly select appropriate evaluation layers according to task requirements without being restricted to a single metric. Overall, the Factory provides a structured, extensi- ble, and model-agnostic framework for conducting consistent and reproducible CT LVLM evaluation. Although not all modules are used in every experiment, its modular design offers room for future extensions. H Supplementary experiments. H.1 MedEval-CT-Bench Across both MedEval-CT-Bench-2D (Table 9) and MedEval-CT-Bench-3D (Table 10), OmniCT consistently achieves the highest overall performance, with averages of 79.80 and 77.63, respectively, surpassing strong baselines such as GPT-5-mini (OpenAI, 2025), Lingshu, CT-CHAT, and M3D-LaMed. It demonstrates robust gains across diverse organs (e.g., liver, kidneys, heart, spine) and task levels (GIR, MAI, AII, CRD), excelling particularly in advanced interpretation and reasoning. These results highlight the effectiveness of our unified slice–volume paradigm in delivering stable, cross-task generalization and comprehensive CT understanding. H.2 Supplementary Ablation. (i) Analysis of Cross-Modal Generalization. To understand the source of OmniCT’s cross-modal generalization, we analyze the roles of (i) the unified single-tower semantic space and (i) the MoE Hybrid Projection (MHP). The single-tower backbone embeds 2D slices and 3D volumes into a shared semantic neighborhood, preventing the semantic drift commonly observed in dual-encoder designs. MHP further learns a modality-adaptive mapping from visual tokens to the LLM space, allowing the projection behavior learned from 2D slices to transfer effectively to 3D representations, and vice versa. To disentangle the contributions of the two components, we compared a dual-tower without MHP configuration against the single-tower with MHP under the same training setup, and observed a substantial degradation in cross-modal generalization. The results are reported in Table 11. Therefore, these two components form a coherent mechanism that supports cross-modal transfer: unified semantics provide a common representational anchor, and MHP supplies the flexibility needed to align slice- and volume-based tokens under a unified LVLM interface. (i) t-SNE Visualization of MoE Hybrid Projection. To further examine whether the two experts in the MHP module learn distinguishable token transformations, we project their output embeddings into a 2D space using t-SNE. As shown in Fig. 7, the features routed to the 2D expert and the 3D expert form two clearly separated clusters. This separation emerges without any explicit supervision enforcing modality-specific behavior; instead, it arises from the structural differences in the inputs (e.g., voxelized tokens with VSC/TPE for 3D vs. planar tokens for 2D) and their decoupled optimization paths before entering the shared semantic space. The visualization supports that the two experts encode modality-dependent transformations, serving the intended role of normalizing heterogeneous inputs before alignment with the LLM. (i) Analysis of Organ-level Semantic Enhancement. The OSE module leverages organ segmentation as a structural regional prior rather than a supervision target. The segmentation masks indicate organ regions with high semantic load for typical CT-based reasoning, from which OSE aggregates a compact set of discriminative tokens, while all global tokens are preserved in the feature stream. In this way, OSE explicitly strengthens organ-level semantics without sacrificing global context. Since the module relies on organ-level structural consistency instead of pixel-level boundary fitting, the high stability of TotalSegmentor in thoracoabdominal CT (average Dice 94.3% (Wasserthal et al., 2023)) is well suited for providing such regional cues. To evaluate the effectiveness of OSE and to rule out potential bias introduced by the segmentation model, we designed three alternative strategies: (i) removing ROI regions, (i) random ROI pooling, and (i) directly concatenating native ROI tokens. As shown in Table 12, removing ROIs yields the expected performance drop; random pooling brings limited gains mainly due to weak alignment effects arising from repeated visual tokens; and direct concatenation of native ROI tokens produces variable-length sequences that prevent stable semantic compression and offer no performance benefit. In contrast, OSE’s fixed-dimensional adaptive aggregation preserves global information coverage while emphasizing diagnostically critical regions, making it better suited to the structured requirements of medical image analysis. (iv) Ablation of Adaptive Feature Aggregation. We further examined the impact of different 2D/3D aggregation token settings (m 2D ,m 3D ) on model performance (Table 13). The results show that moderate aggregation (e.g.,m 2D = 81,m 3D = 90) consistently improves both 2D and 3D performance compared to the baseline without OSE. As the aggregation context continues to grow, the gains diminish and eventually decline, likely due to excessive semantic overlap with global features that disperses the model’s effective visual attention. Overall, these observations indicate that an appropriately sized set of aggregated tokens can effectively enhance organ-level semantics, increase the information density of visual tokens, and maintain a favorable balance between accuracy and computational cost. (v) Robustness of MedEval-CT-Bench to Answer Leakage To reduce the risk that models exploit language artifacts instead of visual evidence, MedEval-CT-Bench’s multiple-choice questions are constructed with a clinical-granularity refiner that rewrites prompts using synonymous expressions, refines clinical wording, and injects stronger distractor options. This preserves the underlying diagnostic intent while weakening template-like phrasing, simple co-occurrence patterns, and answer-position biases. We further conduct two stress tests on MedEval-CT-Bench: (i) an image–question mismatch setting, where questions are randomly paired with incorrect CT scans/volumes, and (i) a noise substitution setting, where images are replaced by noise. As shown in Table 14, both 2D (6-way choice, random≈16.7%) and 3D (4-way choice, random≈25%) accuracies drop sharply toward near-random levels under mismatch/noise, while remaining high with normal inputs. (vi) Unified Representation Gains. To further examine the feasibility and utility of using a 2D encoder as the semantic backbone for incorporating 3D spatial cues, we conduct a balanced subsampling study across slice-driven and volume-driven data. Specifically, we perform controlled ablations using 25%, 50%, and 100% of the available samples for each modality (results in Table 15). Across all settings, joint training consistently yields measurable performance gains. These results indicate that, under the current scale of available pretraining resources, 2D encoders exhibit more mature semantic generalization and thus serve as a reliable representational anchor for constructing 3D inputs. With structured spatial injection, the unified representation acquires effective volumetric awareness, enabling synergistic improvements across both slice- Figure 7 t-SNE plot showing distinct clusters of 2D and 3D expert features after MHP module. and volume-level tasks. H.3 Case Study In Figure 8, the Ground Truth highlights portal hypertension, cirrhotic changes, intrahepatic bile duct cystic dilatations with the central dot sign, and additional renal and gallstones. M3D-LaMed-7B, while mentioning hepatic and portal abnormalities, overemphasizes tumor invasion with incorrect findings, and GPT-5 completely deviates with irrelevant head CT descriptions. In contrast, OmniCT accurately captures the key clinical features—portal hypertension, splenomegaly, mild ascites, cirrhosis, Caroli disease signs, and urinary/gallstones—closely matching the Ground Truth with clinically coherent language, demonstrating its superior spatial–semantic consistency in chest–abdominal CT interpretation. Figure 8 A case study comparing CT findings generated by different medical LVLMs with the clinical ground truth. Table 5 Notations used throughout this paper. NotationDescription V ∈ R D×H×W 3D CT volume with depth D, height H, and width W V j ∈ R 1×H×W j-th 2D slice extracted from the 3D volume s i ∈ R 1×H×W Independent 2D slice input S =s 1 ,...,s n Collection of independent 2D slice inputs ˆs i = Concat(V 3i−2 ,V 3i−1 ,V 3i ) Reassembled volumetric unit ˆ S =ˆs i | i∈ [1,n]Set of all reassembled slice units N s Number of reassembled units (new depth dimension) φ v (·| θ v )Vision encoder with parameters θ v F ∈ R N s ×H ′ ×W ′ ×d v Patch-level visual tokens extracted from ˆ S H ′ = H K , W ′ = W K Spatial resolution of patch features after tokenization KPatch size (stride along spatial dimensions) d v Dimension of visual tokens P =P N s ,P H ′ ,P W ′ Sinusoidal positional encodings along depth, height, width ZTokens enriched with tri-axial positional priors d z ,d y ,d x Feature dimensions of depth/height/width positional encodings UToken-level unshuffle operation mWindow size for unshuffle (m=1 for slice input) ˆ ZToken representations after unshuffle ψ(·| θ p )Slice–volume MoE Hybrid Projection function θ p =W s ,W v ,W share Parameters of MoE Hybrid Projection W s ,W v ,W share Slice-specific, volume-specific, and shared projection matrices 1 slice ,1 volume Binary indicator functions for slice/volume routing σ(·)GELU activation function ˆ F ∈ R L×d f Projected tokens aligned with LLM space L = N s H ′ m W ′ m Total number of projected tokens d f Output feature dimension of MoE Hybrid Projection M o ∈ R D×H×W Organ mask for organ o (from TotalSegmentor, 117 structures) ˆ M o Organ mask mapped to token resolution ˆ F o ∈ R L o ×d h Subset of tokens selected by ˆ M o L o Number of tokens belonging to organ o Agg(·)Organ-level feature aggregation function ˆ f o ∈ R L c ×d h Aggregated tokens for organ o L c Fixed number of aggregated tokens after compression ˆ F OSE Global-local vision tokens after OSE fusion Q =q 1 ,...,q m Input text query sequence φ t (·|θ t )Text embedding function with parameters θ t E ∈ R m×d h Text token embeddings T = [ ˆ F OSE ;E]Unified multimodal input sequence y = (y 1 ,...,y m )Target output sequence P(y t | y <t ;T ;θ)Conditional probability distribution from LLM DTraining data distribution θ llm Parameters of the LLM backbone Table 6 Overview of the hyperparameter settings used for training OmniCT-3B and OmniCT-7B across two stages. OmniCT-3BOmniCT-7B Hyperparameter Stage-1Stage-2Stage-1Stage-2 OptimizerAdamWAdamWAdamWAdamW Learning Rate of Adapter2e-45e-52e-45e-5 Learning Rate of LLM-5e-5-5e-5 Global Batch Size256256256256 Weight Decay0000 LR Schedulerwarm up-cosinewarm up-cosinewarm up-cosinewarm up-cosine Warmup Ratio0.030.030.030.03 Epoch2121 Max Sequence Length2048204820482048 Table 7 Performance of OmniCT with different LLM backbones on 2D and 3D CT benchmarks. Base Model#ParamsSLAKEVQA-RAD OmniMed VQA RadFig -VQA M3DCT-RATE3D-RAD MedEval-CT -Bench Qwen2.5-3B3B81.650.497.480.257.167.564.975.9 Phi-4-mini3.8B81.155.396.984.4 56.467.466.976.2 Table 8 Summary of the datasets included in MedEval-CT, with their task types, training/test sizes, sources, and licenses. DatasetTask TypeTrain SizeTest SizeSourceLicense Volume-driven Datasets M3D-CAPReport116065100RadiopaediaApache-2.0 M3D-VQA Multiple Choice 2409295000 RadiopaediaApache-2.0 Short Answer2409295000 CT-RATEv2 Report938226076 Istanbul Medipol University, Mega Hospital C-BY-NC-SA-4.0 Free-form QA69376024149 3D-RAD Short Answer97094692 Istanbul Medipol University, Mega Hospital C-BY-NC-SA-4.0 Judgment10017023472 Multiple Choice263165746 Slice-driven Datasets ROCOv2Caption18663-PMC-OAApache-2.0 PubMedVision Caption113142- PMC-OAApache-2.0 Dialogue112649- RadFig-VQAMultiple Choice466962084PMC-OACC-BY-NC-SA-4.0 OmniMedVQAMultiple Choice142301579Part of 73 datasetsCC BY & Apache-2.0 LLaVA-MedDialogue10622-PMC-15MApache-2.0 MEDPIX-ClinQA Caption3895- MEDPIX 2.0Apache-2.0 Dialogue3895- VQA-RAD Short Answer104061 MEDPIXCC0 Judgment124896 SLAKE Short VQA2598234 Medical Segmentation Decathlon, ChestXray-NIHCC, CHAOS C-BY-4.0 Judgment2280194 Table 9 MedEval-CT-Bench-2D ModelQwen2.5-VLInternVL3-8BRadFMLingshuHealthGPTMedVLM-R1MedGemma-4BGPT-5-miniOurs lungs72.5066.2518.9970.0063.7570.0066.2571.2574.68 heart 69.7665.855.3977.0768.2961.9566.8376.4780.39 liver76.9570.1710.2086.4475.2562.7168.4775.5982.31 spleen57.3854.1010.0068.8559.0247.5460.6667.2168.33 kidneys72.4875.199.3480.6265.1259.3060.4778.2982.10 pancreas72.0066.008.0884.0071.0062.0066.0075.0078.79 stomach64.9168.427.1471.9373.6856.1464.9178.9575.00 bowel 67.8470.279.7669.4672.1663.5162.4371.0883.47 esophagus58.3358.338.5758.3369.4450.0052.7863.8980.00 trachea59.2651.857.6959.2666.6751.8559.2662.9673.08 vessels 65.8870.987.8776.4769.8054.5165.1075.2080.71 spine73.3381.676.7881.6776.6766.6766.6770.0083.05 others72.2578.759.2785.2576.0078.0076.5072.2586.47 GIR67.7572.1613.9575.4166.3664.9767.7572.5670.70 MAI 67.0372.538.2987.9176.9284.0777.4756.0488.95 AII75.6269.719.7376.7665.9062.2959.0569.9086.83 CRD69.1471.116.9578.3374.6760.2367.8279.1581.78 Average68.3868.439.2975.7570.0462.1065.2071.5279.80 Table 10 MedEval-CT-Bench-3D ModelQwen2.5-VLMiniCPM-V-4.5CT-CHATM3D-LaMed-Phi-3-4BM3D-LaMed-Llama-2-7BGPT-5-miniOurs lungs52.0747.6652.0754.2762.2653.7267.22 heart54.4559.1651.8358.6463.8754.4586.91 liver55.1455.3960.4075.9467.4262.9180.20 spleen51.8845.9464.3876.8874.0648.4471.25 kidneys47.8741.6055.8971.6880.2044.1178.20 pancreas 41.3840.5256.4776.7270.6943.5375.00 stomach36.9240.6562.6268.2271.5052.8064.02 bowel47.3240.2860.5674.9377.7547.6180.00 esophagus44.6846.8148.9440.4376.6029.7963.83 trachea61.4165.2784.5768.8182.9664.3189.39 vessels 50.8851.6354.1464.1671.6852.1384.46 spine45.9852.8777.0167.8272.4155.1783.91 others 51.6348.1261.1564.9168.4252.6381.45 GIR45.8339.3446.4575.9880.8847.9277.82 MAI 61.8964.3276.3466.7570.0851.1586.06 AII46.4045.2963.1067.2673.2851.0677.18 CRD 49.4748.0056.0163.9264.4558.1272.78 Average49.7248.9960.7066.9072.2751.1777.63 Table 11 Ablation of MoE Hybrid Projection. Training StrategyPerf. 2DPerf. 3DAvg. SigLip + M3D-CLIP (w/o MHP)34.5730.5832.58 SigLip + Siglip (w/ MHP)55.3048.6151.96 Table 12 Ablation analysis of adaptive organ-level feature aggregation. ROI StrategyPerf. 2DPerf. 3DAvg. No OSE78.6862.1770.43 OSE w/ native ROI78.3762.2470.31 OSE w/ random ROI 80.1364.2272.18 OSE w/ adaptive ROI81.4566.1573.80 Table 13 Ablation of the OSE aggregation ratios for 2D and 3D tokens.m 2D andm 3D denote the numbers of aggregated organ-level tokens for 2D slices and 3D volumes, respectively. m 2D m 3D Perf. 2DPerf. 3DAvg. 0078.6862.1770.43 364080.6663.8172.24 819081.4566.1573.80 144160 81.2366.0473.64 22525080.6465.4873.06 Table 14 Organ-level accuracy on MedEval-CT-Bench under normal, image–question mismatch, and noise settings Organ2D Normal2D Mismatch2D Noise3D Normal3D Mismatch3D Noise lungs74.723.119.567.229.427.3 heart80.424.014.986.931.820.1 liver82.319.920.980.230.525.8 spleen68.313.717.471.328.429.0 kidneys82.125.121.178.222.322.6 pancreas78.824.915.275.030.223.4 stomach 75.021.714.464.019.022.7 bowel 83.522.219.480.030.127.9 esophagus 80.020.318.063.828.629.2 trachea73.118.020.689.421.824.1 vessels80.723.919.884.530.128.6 spine 83.117.217.883.928.917.1 others86.522.914.681.529.727.0 Table 15 Ablation study of unified representation gains. Compared to 2D-only and 3D-only training, mixed training consistently improves performance across all data scales while preserving the same 2D/3D ratio. Training StrategyRatioSLAKEVQA-RADRadFig-VQAM3D-VQACT-RATE3D-RAD 2D-Only25%70.662.577.2--- 3D-Only25%---65.884.565.4 Mixed25%72.262.578.269.984.966.9 2D-Only50%75.866.779.4--- 3D-Only50%---72.786.267.2 Mixed50% 77.370.879.873.687.168.1 2D-Only100%81.072.378.2--- 3D-Only100%---74.486.668.6 Mixed100%81.271.881.974.786.669.3 Table 16 Performance of 18 types of anomaly label prediction. ModelPrecisionRecallF1 RadFM13.16.47.2 M3D-LaMed-7B8.12.53.5 M3D-LaMed-4B16.58.49.6 CT-CHAT 24.338.827.2 CT2Rep41.638.136.7 CT-AGRG37.855.442.1 OmniCT41.736.536.3

Instant research discovery