Paper deep dive
Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models
Maulik Chevli, Johannes Brandt, Rickmer Braren, Daniel Rueckert, Philip Müller
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/8/2026, 4:32:29 AM
Summary
This paper benchmarks ten frozen 3D CT foundation models across three thoracic CT cohorts to evaluate their ability to detect incidental findings without fine-tuning. The study finds no universal state-of-the-art model; performance varies by evaluation context (k-NN, zero-shot, linear probing). While models with fine-grained tokenization and vision-language alignment generally perform best, the primary determinant of detectability is physical: findings with high contrast and large spatial extent are reliably detected, whereas small, low-contrast focal lesions remain challenging for all encoders due to limitations in globally pooled embeddings.
Entities (12)
Relation Signals (8)
focal lesions → ischallengingfor → globally pooled embeddings
confidence 95% · small, low-contrast focal lesions remain a persistent challenge across all evaluated encoders... attributed to the inherent limitations of globally pooled embeddings
CT-SSG → issupervised → CT-SSG
confidence 95% · With the exception of the supervised CT-SSG
detectability → scaleswith → contrast
confidence 95% · a finding's detectability scales with its contrast against surrounding tissue
detectability → scaleswith → spatial extent
confidence 95% · a finding's detectability scales with its contrast against surrounding tissue and its spatial extent
FlexiCT → performsbestin → zero-shot prompting
confidence 90% · FlexiCT maximizes skill
COLIPRI → performsbestin → Linear Probing
confidence 90% · COLIPRI maximizes AUROC
CT-RATE → isusedby → FlexiCT
confidence 85% · Five models were exposed to the CT-RATE dataset during pretraining... FlexiCT
CT-RATE → isusedby → COLIPRI
confidence 85% · Five models were exposed to the CT-RATE dataset during pretraining... COLIPRI
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Routine CT interpretation is inherently comprehensive, capturing incidental findings across the entire scan volume. 3D CT foundation models could assist this process by providing generalizable representations of anatomy and pathology. To evaluate their diagnostic breadth, we benchmark ten frozen CT encoders across three cohorts of thoracic CT scans, including an unseen internal clinical dataset, using $k$-nearest neighbors, zero-shot prompting, and linear probing. We find no universal state-of-the-art, with rankings fluctuating significantly depending on the evaluation context. While models combining fine-grained image tokenization with vision-language alignment generally perform best, a lightweight supervised encoder remains highly competitive, demonstrating that explicit labels can effectively substitute for scale. Crucially, rather than model architecture, we observe that the primary determinant of performance is a physical bottleneck: a finding's detectability scales with its contrast against surrounding tissue and its spatial extent. Through controlled within-organ comparisons, we empirically demonstrate that widespread or high-contrast abnormalities, such as devices and effusions, are reliably recovered. Conversely, small, low-contrast focal lesions remain a persistent challenge across all evaluated encoders. We attribute this to the inherent limitations of globally pooled embeddings, suggesting that accurately representing small, low-contrast structures will require region- or lesion-level pretraining.
Tags
Links
- Source: https://arxiv.org/abs/2608.05960v1
- Canonical: https://arxiv.org/abs/2608.05960v1
Trouble viewing inline? Open PDF directly →
Full Text
33,604 characters extracted from source content.
Expand or collapse full text
Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models Maulik Chevli 1⋆ , Johannes Brandt 1⋆ , Rickmer Braren 1,4 , Daniel Rueckert 1,2,3 , and Philip Müller 1 1 Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany 2 Dept. of Computing, Imperial College London, UK 3 Munich Center for Machine Learning (MCML), Munich, Germany 4 Dept. of Diagnostic and Interventional Radiology, UKE Hamburg, Germany Abstract. Routine CT interpretation is inherently comprehensive, cap- turing incidental findings across the entire scan volume. 3D CT founda- tion models could assist this process by providing generalizable represen- tations of anatomy and pathology. To evaluate their diagnostic breadth, we benchmark ten frozen CT encoders across three cohorts of thoracic CT scans, including an unseen internal clinical dataset, usingk-nearest neighbors, zero-shot prompting, and linear probing. We find no universal state-of-the-art, with rankings fluctuating significantly depending on the evaluation context. While models combining fine-grained image tokeniza- tion with vision-language alignment generally perform best, a lightweight supervised encoder remains highly competitive, demonstrating that ex- plicit labels can effectively substitute for scale. Crucially, rather than model architecture, we observe that the primary determinant of per- formance is a physical bottleneck: a finding’s detectability scales with its contrast against surrounding tissue and its spatial extent. Through controlled within-organ comparisons, we empirically demonstrate that widespread or high-contrast abnormalities, such as devices and effusions, are reliably recovered. Conversely, small, low-contrast focal lesions re- main a persistent challenge across all evaluated encoders. We attribute this to the inherent limitations of globally pooled embeddings, suggesting that accurately representing small, low-contrast structures will require region- or lesion-level pretraining. Keywords: Foundation models· Chest CT· Incidental findings· Multi- abnormality classification· Frozen features· Transfer evaluation. 1 Introduction Routine CT interpretation is inherently comprehensive: Regardless of the pri- mary indication, radiologists inspect the entire volume [11,3] and report any anatomical deviations, including incidental findings [9]. 3D CT foundation mod- els (FMs) can assist with this comprehensive review because they encode broad, ⋆ These authors contributed equally. arXiv:2608.05960v1 [cs.CV] 6 Aug 2026 2M. Chevli, J. Brandt et al. Evaluation Datasets multi-center | multi label | 3D CT RAD-ChestCT CT-RATE In-house ... 3D CT Foundation Models Volumetric Encoders k-N | Zero-shot | Linear probing ... LATENT SPACE EVALUATE ENCODE 1,564 Scans 2,284 Scans 3,001 Scans Fig. 1: Evaluation Overview. Three 3D thoracic CT datasets are encoded us- ing frozen foundation models. The extracted representations are then evaluated across multiple findings classification tasks using k-N, zero-shot classification, and linear probing readouts. generalizable representations of anatomy and pathology. This generalizability is critical because medical pathologies follow a heavy-tailed distribution [20], making it infeasible to train dedicated classifiers for every possible abnormal- ity. Furthermore, while hospitals may possess the infrastructure to run inference locally, they rarely have the labeled datasets necessary to fine-tune task heads for hundreds of rare conditions [12]. While fine-tuning remains a practical route for targeted applications, a true foundation model should, to a meaningful de- gree, inherently capture the diverse abnormalities present in a CT within its embeddings. Assessing whether current models achieve this fundamental capa- bility requires isolating the latent space and benchmarking their frozen repre- sentations directly. We therefore ask: Are current CT foundation models capable of incidental-finding detection without any fine-tuning, does a consistently best- performing model emerge, and where do their representational limitations lie? To answer this, we evaluate ten 3D CT encoders in their frozen state on three thoracic CT datasets: CT-RATE [10], RAD-ChestCT [8], and an in-house hospital cohort guaranteed to be unseen by any model. To assess models’ utility under data and compute scarcity, we avoid fine-tuning entirely and evaluate them solely through zero-shot prompting, k-nearest-neighbour classification, and linear probing. While no single model wins across all readouts, architectures utilizing fine-grained image tokenizers combined with report-aligned pre-training consistently lead performance. Furthermore, we observe that diagnostic difficulty is governed not by the choice of encoder, but by characteristics of the findings like its spatial extent and contrast, leaving small focal lesions challenging for all encoders. Ultimately, this benchmark offers independent insights to inform the selection of baseline encoders and highlights critical representational bottlenecks for future model development. 2 Results We evaluate ten 3D CT FMs as shown in the Fig. 2a, out of these, three models (CT-FM [17], Curia-2 [18], VoxelFM [16]) use image-only pre-training, CT-SSG Frozen-Feature Benchmark of 3D CT Foundation Models3 Encoder Obj. TextPatchDimCT-RATE FlexiCT VLLLM 8 3 1728 ✓ COLIPRI VLsmall 8 3 768 ✓ Pillar-0 VLLLM 8 2 ⋅4 1152 – CT-SSG Sup none–512 ✓ SPECTRE VLsmall 16 2 ⋅8 1080 ✓ Curia-2 SSLnone 16 2 768 – Merlin VLsmall–2048 – VoxelFM SSLnone 14 3 864 – CT-CLIP VLsmall 20 2 ⋅10 512 ✓ CT-FM SSLnone–512 – Patch in voxels; – : CNN/graph; Curia-2: 2D axial slices 0.50.60.7 1,564 scans · 18 labels CT-RATE 0.550.600.65 2,284 scans · 92 labels RadChest 0.60.7 3,001 scans · 87 labels In-house 0.50.70.9 Bones Mediastinum Airways Lung Cardiac Pleura Devices RadChest 0.50.70.9 In-house 10 2 10 3 0.00 0.05 0.10 0.15 0.20 0.25 macro probe skill RadChest 10 2 10 3 In-house a Encoder configuration and macro AUROC per cohort macro AUROC b Per-organ capability linear-probe AUROC c Data efficiency labelled training scans cosine kNNzero-shot FlexiCT COLIPRI Pillar-0 all-10 range median FlexiCT COLIPRI VoxelFM CT-SSG Curia-2 SPECTRE Pillar-0 Merlin CT-FM CT-CLIP Fig. 2: Benchmarking out-of-the-box capability and data efficiency. (a) Encoder configurations and macro AUROC. Models combining vision– language alignment with fine-grained tokenizers (FlexiCT, COLIPRI) consis- tently lead kNN and zero-shot performance across both public and unseen ex- ternal cohorts. (b) Per-organ linear probing confirms this architectural advan- tage persists across anatomical regions, though absolute detectability remains organ-dependent. (c) Data efficiency curves demonstrate that these leading rep- resentations scale log-linearly with available data. Surprisingly, despite its strong overall performance, Pillar-0 rapidly hits a representational ceiling and saturates. Confidence intervals are omitted for readability as they are negligibly small. [7] is supervised trained, while the remainder (CT-CLIP [10], COLIPRI [19], SPECTRE [6], Merlin [4], Pillar-0 [2], FlexiCT [13]) use vision-language align- ment. Five of these models used the CT-RATE training set during (pre)training. We evaluate the models using AUROC and prevalence-normalized PR-AUC-, also know as skill: for a finding of prevalence π, skill = (AP−π)/(1−π), where AP is the average precision, skill values range from 0 to 1 being the perfect score. Our evaluation yields no single optimal model; rankings depend on how the embeddings are read (Fig. 2b, 2c). Under cosine kNN, the top three encoders on every cohort are FlexiCT, COLIPRI, and Pillar-0, which are the transformers with the finest image tokenizer (8-voxel patches; Fig. 2a), whereas the coarsest- tokenizer transformer (CT-CLIP, 20×20×10) ranks last. This patch-granularity advantage strongly co-occurs with report-aligned pretraining, demonstrating that 4M. Chevli, J. Brandt et al. device fluid calcif. texture skeletal focal COLIPRI FlexiCT VoxelFM Curia-2 SPECTRE CT-SSG Pillar-0 Merlin CT-FM CT-CLIP 0.820.810.740.760.690.68 0.840.790.720.730.660.65 0.790.750.690.700.700.61 0.800.780.690.670.660.60 0.750.750.700.710.660.61 0.790.750.710.690.620.60 0.810.710.660.680.600.62 0.770.740.670.650.610.57 0.680.660.630.590.590.54 0.560.570.580.550.550.52 Capability map: linear-probe AUROC by encoder and finding type device fluid calcif. texture skeletal focal 0.5 0.6 0.7 0.8 AUROC Readout decomposition: present (kNN) < extractable (probe); some extractable signal is not language-reachable chance kNN linear probe zero-shot 0.5 0.6 0.7 0.8 0.9 Fig. 3: Classification difficulty is dictated by finding type, not encoder. Left: Linear-probe AUROC by model and finding type. The easy-to-hard hi- erarchy (columns, left to right) is highly consistent across all encoders: de- vices and fluid are universally accessible, while focal lesions remain challenging. Right: Readout decomposition averaged across the six report-aligned encoders. Although linear probing extracts more latent signal than zero-shot prompting or k-N retrieval, the underlying difficulty hierarchy is preserved. Focal abnor- malities yield the lowest performance across all readouts. fine-grained image tokenization combined with vision–language alignment gen- erally yields the strongest out-of-the-box representations for retrieval. Crucially, Pillar-0 never saw CT-RATE yet remains in the top three on the two held-out cohorts, indicating that these shared architectural traits confer genuine gener- alization. However, data efficiency curves (Fig. 2c) reveal that Pillar-0 rapidly hits a representational ceiling and saturates as labeled data increases, eventually being overtaken by CT-SSG. As the only supervised encoder and the small- est embedding (512-d), CT-SSG stays highly competitive across both kNN and linear probing on all three cohorts. It matches or beats every self-supervised encoder and both convolutional models, suggesting that explicit labels can effec- tively substitute for scale. Under linear probing, COLIPRI maximizes AUROC and FlexiCT maximizes skill, at which point the strict patch-granularity order- ing dissolves and the self-supervised VoxelFM rises to the front. CT-CLIP and CT-FM consistently underperform. Performance differences across distinct findings are substantially larger than the differences between models on any given finding. We sort every label into six phenotypic classes defined by visual morphology: calcification, devices, fluid, texture, focal, and skeletal (see Methods). The easy-to-hard ordering across these types barely moves from one encoder to the next (Kendall W = 0.89, Fig. 3a): each encoder reads support devices off easily, handles diffuse fluid and texture changes moderately, and struggles most with focal lesions. This limitation is rep- resentational rather than an artefact of the readout. The linear probe extracts more signal than kNN or zero-shot, yet focal performance stays low even under the probe (Fig. 3b). Moreover, it is selective within the focal class: larger solid lesions stay retrievable (solid nodule skill value 0.26, mediastinal lymphadenopa- thy 0.29) while small subsolid ones sink toward chance (ground-glass nodule skill 0.05, part-solid nodule 0.02). Frozen-Feature Benchmark of 3D CT Foundation Models5 0.50.60.70.80.9 Pleura (In-house) Pleura (RAD-ChestCT) Pericardium (RAD-ChestCT) Lung nodule (In-house) thickeningeffusion thickeningeffusion thickeningeffusion subsolidsolid ↑ contrast (same organ) 0.50.60.70.80.9 Calcification (In-house) Ground-glass (In-house) Vessel Calcification (In-house) faint lung focibulk coronary focaldiffuse carotid (small)aortic (large) ↑ extent (same density) subtle member conspicuous member macro AUROC over 10 encoders Fig. 4: Contrast and spatial extent govern detectability. Controlled within-organ comparisons demonstrate that increasing either contrast (left) or spatial extent (right) improves detection. Conspicuous findings (blue) system- atically yield higher AUROCs than their subtle counterparts (orange) across all evaluated models. This indicates that a finding’s physical footprint, not model architecture, is the primary bottleneck for incidental finding detection. This difficulty ordering is driven by a straightforward empirical pattern: a finding’s detectability scales with its contrast against surrounding tissue and its spatial extent. To isolate these physical effects from prevalence and anatomy, we conducted controlled within-organ comparisons (Fig. 4). Holding extent fixed while increasing contrast (e.g., pleural and pericardial effusion vs. thickening; solid vs. subsolid nodule) improves macro AUROC by +0.18 on average. Con- versely, holding contrast fixed and increasing extent (e.g., bulk vs. faint calci- fication; diffuse vs. focal ground-glass) yields a +0.16 improvement. All seven within-organ comparisons are positive and bootstrap-significant, with the ten evaluated encoders demonstrating near-universal agreement on this direction- ality (68 of 70 comparison× encoder pair). This pattern replicates robustly across both the public and unseen internal cohorts. Consequently, we observe that the representational bottleneck is primarily physical: low-contrast, small- extent abnormalities, which are precisely the subsolid focal lesions highlighted above, remain fundamentally challenging for all globally pooled encoders. Finally, read-out performance varies across the finding prevalence range. The linear probe remains the strongest readout overall. For the rarest findings, zero-shot prompting yields a paired advantage of approximately 0.10 AUROC over kNN (Fig. 5). However, this advantage disappears when evaluated using prevalence-normalized skill. Under the skill metric, both zero-shot and kNN readouts score near zero on the rarest findings. 3 Discussion and Outlook We investigated whether the frozen embeddings of current 3D CT foundation models possess the representational breadth necessary to capture the wide ar- ray of incidental findings encountered in routine interpretation. Our results show that no single encoder consistently wins across all cohorts and readouts. Instead, the fundamental factor that transfers across cohorts is a physical bottleneck. Specifically, a finding’s detectability is heavily influenced by its contrast and 6M. Chevli, J. Brandt et al. spatial extent. This leaves low-contrast, small-extent findings, such as subsolid focal lesions, universally challenging for globally pooled embeddings, approach- ing chance performance even under linear probing (ground-glass nodule skill value 0.05, part-solid nodule 0.02). This physical limitation clarifies and extends recent evaluations of frozen foundation models. While prior work demonstrates that frozen features support in-distribution oncology diagnosis but falter on prognosis [1], our benchmark- ing across a wide spectrum of incidental findings elucidates why frozen embed- dings systematically fail to capture certain pathologies. A globally pooled em- bedding inherently attenuates the subtle spatial signals of small lesions. In con- trast, purpose-built systems that rely on explicit spatial localization [15,5] and per-structure radiomics achieve strong performance on incidental findings [14], underscoring the necessity of spatial grounding. We acknowledge several limitations: evaluating frozen embeddings bounds achievable peak performance, and the NLP-based extraction of labels across all three cohorts introduces noise that disproportionately affects the rarest find- ings. However, the difficulty ordering by finding type is highly consistent across all three cohorts (Kendall W=0.89), despite each using an independently con- structed label pipeline. This strong agreement makes label noise an unlikely cause of the focal-lesion failure, indicating the pattern is representational. These findings provide concrete directions for future model development. Be- cause diagnostic difficulty is governed by contrast and extent rather than model capacity, overcoming this will likely require region- or lesion-level pretraining ob- jectives that explicitly preserve small, low-contrast structures. Furthermore, the performance gap between linear probing and zero-shot retrieval suggests that rel- evant visual signals are present in the embeddings but remain inaccessible to the text encoder. Improving fine-grained vision–language alignment, perhaps cou- pled with lightweight, pattern-specific detection heads, offers a more promising route to robust clinical translation than relying on a single global vector. 4 Data and Method Datasets. We evaluate on three distinct chest CT cohorts. The CT-RATE [10] validation set initially contains 3,039 thoracic CT volumes from 1,304 patients. To prevent leakage from near-duplicate reconstructions, we filter this to one re- construction per study, yielding 1,564 scans. This split is strictly disjoint from the CT-RATE training data seen by five of the evaluated models. The RadCh- estCT [8] validation set comprises 2,284 single-scan patients. Our unseen internal hospital cohort includes 3,001 scans from 2,087 patients. Abnormality labels span the thoracic field of view and are derived from radiology reports: CT-RATE (18 labels) uses a fine-tuned text model, while the internal cohort (87 labels) utilizes the RATE framework [2]. RadChestCT uses a radiologist-validated rule-based extractor; we aggregate its fine-grained location annotations into 92 labels by pooling across anatomical regions, preserving regional splits only for inherently multi-organ findings (cancer, mass, lymphadenopathy, calcification). Because pa- Frozen-Feature Benchmark of 3D CT Foundation Models7 tients in CT-RATE and the internal cohort contribute multiple scans, we eval- uate both using strictly patient-grouped cross-validation. Models. We evaluate 10 models capable of multi-abnormality classification. With the exception of the supervised CT-SSG [7], all models utilize self-supervised learning: either image-only (CT-FM [17], Curia-2 [18], VoxelFM [16]) or vision– language alignment (CT-CLIP [10], COLIPRI [19], SPECTRE [6], Merlin [4], Pillar-0 [2], FlexiCT [13]). Pretraining data varies significantly across the lineup. Five models were exposed to the CT-RATE dataset during pretraining (indicated in Fig. 2). Furthermore, anatomical exposure ranges from strictly chest-specific (COLIPRI, CT-CLIP, CT-SSG, Pillar-0) and thoraco-abdominal (SPECTRE), to broad multi-region or whole-body collections (FlexiCT, VoxelFM, Curia-2, CT-FM), or exclusively abdomen–pelvis without chest exposure (Merlin). Conse- quently, the domain overlap with our chest-dominated evaluation cohorts differs markedly among encoders. We evaluate each model strictly as a frozen feature extractor using its author-prescribed preprocessing. Evaluation. We read the frozen embeddings using three distinct methods: cosine-weighted k-nearest neighbours (k = 5); zero-shot classification (for vision– language models); and linear probing. For zero-shot classification, we score each finding against a fixed prompt pair—“A chest CT scan showing finding.” and “A chest CT scan showing no finding.”—computing σ(cos(z,t + )− cos(z,t − )) over the ℓ 2 -normalized image and text embeddings. The scan’s own report is never used. The linear probe is a one-vs-rest ℓ 2 -regularized logistic regression (C = 1, balanced classes) fit on the raw embeddings. All three readouts are eval- uated using the same five-fold cross-validation, strictly grouped by patient to prevent leakage. We report 95% confidence intervals from 1,000 paired patient- level bootstrap resamples, computing macro averages over findings with at least 20 positives. Finding Types. To analyze representational capability by finding character- istics rather than individual label, we group all labels across the three co- horts into six broad radiological phenotypes: calcification (e.g., coronary de- posits), devices (e.g., pacemakers, stents), fluid collections and morphometric changes (e.g., effusions, cardiomegaly), diffuse texture patterns (e.g., atelec- tasis, ground-glass attenuation), focal space-occupying lesions (e.g., nodules, tumours), and skeletal findings (e.g., fractures, arthritis). The complete list is available on our github repository. This taxonomy is applied uniformly across all cohorts, serving as our primary unit of analysis for capability mapping (Fig. 3a). We measure the consistency of difficulty ordering across encoders using Kendall’s coefficient of concordance (W) on the resulting matrix. For the contrast× ex- tent analysis (Fig. 4), we isolate these physical variables from prevalence and anatomy by constructing controlled within-organ comparisons: pairs of findings in the same organ that differ along a single axis—either contrast (e.g. effusion vs. thickening) or extent (e.g. diffuse vs. focal ground-glass). For each pair, we test the difference in macro AUROC (averaged across all ten encoders) using a patient-grouped paired bootstrap, reporting how many encoders agree on the direction of the effect. 8M. Chevli, J. Brandt et al. FlexiCTCOLIPRISPECTREPillar-0MerlinCT-CLIPCuria-2CT-SSGVoxelFMCT-FMFlexiCTCOLIPRISPECTREPillar-0MerlinCT-CLIPCuria-2CT-SSGVoxelFMCT-FMFlexiCTCOLIPRISPECTREPillar-0MerlinCT-CLIP nodule lung 76% opacity 54% coronary artery disease 53% calcification cardiac 52% calcification other 35% cancer lung 33% groundglass 32% atelectasis 30% atherosclerosis 30% calcification vascular 29% lymphadenopathy axillary 2% aneurysm 2% secretion 2% bronchiolitis 2% lucency 2% bronchitis 2% bronchiolectasis 1% distention 1% heart failure 1% congestion 1.0% Lung nodule 45% Lung opacity 39% Arterial wall calcification 28% Pulmonary fibrotic sequela 27% Lymphadenopathy 25% Coronary artery wall calcification 25% Atelectasis 23% Emphysema 20% Consolidation 19% Hiatal hernia 14% Pleural effusion 12% Peribronchial thickening 11% Bronchiectasis 11% Cardiomegaly 10% Medical material 10% Mosaic attenuation pattern 8% Interlobular septal thickening 8% Pericardial effusion 7% coronary atherosclerosis 44% aortic atherosclerosis 40% consolidation 36% ground glass opacity 30% atelectasis 26% effusion 25% cvc 25% solid 18% lymphadenopathy 17% valve calcifications 13% mosaic attenuation 2% emphysema paraseptal 2% valve replacement 2% steatosis 2% chronic pancreatitis 1% mass 1% cabg 1% mass 1% implants 1.0% pneumomediastinum 0.7% kNNprobezero-shot 65676372596160636055 67676267565757635653 77806979716769816853 76786877696469786553 60585862565556605251 78756878636666706354 67685870545657615651 66686365566059665757 59615663595556625549 63646163625759645951 52545256524954524949 59565355565359545447 53534765615354555548 50505048495247515147 49484951495152515356 52555150486055495250 63686560595251626153 54544853505249514948 72755471636760756359 62665362645958615358 61646167555257605655 82828081585762786358 84878084777282738068 61655764605359565652 69676967626062626155 84878286766880708068 68756966585658635957 70726368625663676356 85888385625663806358 61596457616159565755 92919185807086847969 67656367645564676462 68676163575557695653 83857875737377747665 78816767675959676462 72706865626167736656 73757272625965726256 71746967635767626259 81817279766774816863 75776875726371756759 77777476655963726058 78777477655864766257 75746869696567726461 94948992918486947974 91916468675465865951 59645667555557545754 69656867645962676258 62665864585662605653 59626063585958585453 67605468494850645049 62765768565558565358 79675458555061485352 56505149504857505353 55505053474853515553 57505357535661565056 50515451534852535951 79645863535363616953 58585360534754636055 72766966666866697263 72746862606162686858 84857979787280867770 81837675767178847569 65686362586159625853 84807874696277777862 71757067585962726657 73736860646268707160 66716361656163686256 70756963676469736859 50555175575642574852 70705963636968625756 68686162596557645751 54565461535661575849 51634253404937424255 69736767575969626766 86918684777283799163 63646855636351506652 84887284808383808872 72736468667475667561 66716569575761636354 82868375666373847962 90928884867789868981 64746659636064636558 70747261666466667162 89928887847686868881 74807364646467707162 78787768706572807866 90918785736475858165 67676956676869686661 95969489898095909390 73787464716471777566 74777265616264707255 92928884868589889082 85907873776777797869 82858279747179838171 84868375747176838170 84847976727080807769 85868179807381878173 81847872777278817970 79827869686769777463 83838074706772827964 79807867737174777871 97979592958793959389 95948078846091927765 64736366596161606258 77787769706877767568 72776765686671706959 85858277787473828463 83857682616679807764 92937587817891818363 83798078806385556864 62706954646562636966 52635453455852544743 89868193877493868269 73666670686375678067 1008179946872771009384 91847884807176847662 707063505256 696348626359 698342767469 807440727367 656250585858 645961565853 697054625451 656058655360 696946656358 696747616662 675251585747 676947646669 506446616661 515755503946 656446555750 556155414949 806850667847 596660626756 728057747381 746150707572 546446485154 838269796258 808949827771 546556515354 726833715560 828954877671 697052605760 756954616360 918984846753 655644526667 969434887459 757344675561 727450585757 928838847475 537348756857 737326696966 817831797266 878035785856 858460807670 828261727670 818167776865 828348776563 767139725862 979635949283 769341566152 596651505350 777533736663 757343667065 757941737276 908659556464 878945727176 858151417351 577147686762 667346605864 868252677472 536359444756 777549444459 908566687767 RadChest high-prev RadChest low-prev CT-RATE high-prev CT-RATE low-prev In-house high-prev In-house low-prev 0.50.60.70.80.9 AUROC organ Lung Airways Pleura Mediastinum Cardiac Devices Bones Extrathoracic Breast Abdomen Neck Fig. 5: Comprehensive performance landscape across findings, cohorts, and readouts. Per-finding AUROC for high- and low-prevalence abnormalities across three datasets, evaluated via kNN, linear probing, and zero-shot prompt- ing. The prominent horizontal banding confirms that diagnostic difficulty is in- herently tied to the finding itself, rather than the encoder. While linear probing consistently maximizes extractable signal, rare (low-prevalence) and focal find- ings remain universally challenging across all models and readout strategies. Conversely, structural abnormalities like support devices and fluid collections are robustly detected by nearly every encoder. Furthermore, the vertical band- ing reveals that while fine-grained tokenizers drive peak performance among the leading models, coarser architectures like CT-CLIP systematically collapse to- ward chance. The persistence of these difficulty patterns in the bottom block, our unseen internal hospital cohort, demonstrates that these representational bottlenecks are fundamental to the models themselves, rather than artifacts of the public pretraining datasets. Frozen-Feature Benchmark of 3D CT Foundation Models9 Acknowledgments. We thank the members of the IDERHA consortium - https://w.iderha.org/. This project is supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under grant agreement No. 101112135. The JU receives support from the European Union’s Horizon Europe research and innovation programme and COCIR, EFPIA, Europa Bío, MedTech Europe, and Vaccines Europe. Funded by the European Union, the private members, and those contributing partners of the IHI JU. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the afore- mentioned parties; neither of the aforementioned parties can be held responsible for them. Compliance with Ethical Standards. All procedures complied with the Dec- laration of Helsinki and relevant institutional guidelines. The retrospective anal- ysis of the in-house dataset was approved by the Technical University of Munich Ethics Committee (87/18S), with a waiver of informed consent. Use of the ex- ternal CT-RATE and RAD-Chest CT datasets was governed by their respective institutional approvals: the Istanbul Medipol University Clinical Research Ethics Committee (E-10840098-772.02-6841) and the Duke University Health System IRB. All data across the three datasets were fully anonymized, collected retro- spectively, and processed under informed consent waivers and HIPAA compliance where applicable. Code availability. Code to reproduce the frozen-encoder readouts (AUROC and prevalence-normalized skill under k-N, zero-shot, and linear probing) on the two public cohorts (RAD-ChestCT, CT-RATE) is available at https:// github.com/maulikchevli/frozen-lexpert. Pretrained weights are obtained from each model’s public release; the restricted internal cohort is not included. References 1. Aerts, H.J.W.L., Pai, S., Hadzic, I., Fedorov, A., Mak, R.H.: Foundation model embeddings for quantitative tumor imaging biomarkers. Research Square (2025). https://doi.org/10.21203/rs.3.rs-6630446/v1, preprint rs.3.rs-6630446 2. Agrawal, K.K., Liu, L., Lian, L., Nercessian, M., Harguindeguy, N., et al.: Pillar-0: a new frontier for radiology foundation models. arXiv preprint arXiv:2511.17803 (2025) 3. Aydin, S., Ece, B., Cakmak, V., Kocak, B., Onur, M.R.: Emergency radiology: roadmap for radiology departments. Japanese Journal of Radiology 43(10), 1606– 1617 (2025) 4. Blankemeier, L., Cohen, J.P., Kumar, A., Van Veen, D., Gardezi, S.J.S., et al.: Merlin: a vision language foundation model for 3d computed tomography. arXiv preprint arXiv:2406.06512 (2024) 5. Brandt, J., Chevli, M., Braren, R., Kaissis, G., Müller, P., Rueckert, D.: Lungevaty: A scalable, open-source transformer-based deep learning model for lung cancer risk prediction in ldct screening. In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI). p. 1–5. IEEE (2026) 10M. Chevli, J. Brandt et al. 6. Claessens, C., Viviers, C., D’Amicantonio, G., Bondarev, E., van der Sommen, F.: Scaling self-supervised and cross-modal pretraining for volumetric ct transformers. arXiv preprint arXiv:2511.17209 (2025) 7. Di Piazza, T., Lazarus, C., Nempont, O., Boussel, L.: Structured spectral graph representation learning for multi-label abnormality analysis from 3d ct scans. arXiv preprint arXiv:2510.10779 (2025) 8. Draelos, R.L., Dov, D., Mazurowski, M.A., Lo, J.Y., Henao, R., Rubin, G.D., Carin, L.: Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical Image Analysis 67, 101857 (2021). https://doi.org/10.1016/j.media.2020.101857 9. Evans, C.S., Arthur, R., Kane, M., Omofoye, F., Chung, A.E., Moreton, E., Moore, C.: Incidental radiology findings on computed tomography studies in emergency department patients: a systematic review and meta-analysis. Annals of emergency medicine 80(3), 243–256 (2022) 10. Hamamci, I.E., Er, S., Almas, F., Simsek, A.G., Esirgun, S.N., et al.: Developing generalist foundation models from a multimodal dataset for 3d computed tomog- raphy. arXiv preprint arXiv:2403.17834 (2024) 11. Herring, W.: Learning radiology: recognizing the basics. Elsevier Health Sciences (2019) 12. Hoelzle, C.S., Brandt, J., Mueller, J.C., Klug, M., Westphal, J., Rueckert, D., Chevli, M., Fintelmann, F.J.: Longitudinal language-model reasoning enables au- tomated labeling of lung cancer recurrence from unstructured clinical records. Research Square (2026). https://doi.org/10.21203/rs.3.rs-9550278/v1, https://doi. org/10.21203/rs.3.rs-9550278/v1 13. Li, Y., Gao, Y., Dong, H., Lai, Y., Wang, S., Safari, M., Baciak, J.E., Yang, X.: Universal ct representations from anatomy to disease phenotype through agglom- erative pretraining. arXiv preprint arXiv:2605.21906 (2026) 14. Marcinkiewicz, A.M., Buchwald, M., Shanbhag, A., Bednarski, B.P., Killekar, A., Miller, R.J.H., Builoff, V., Lemley, M., Berman, D.S., Dey, D., Slomka, P.J.: AI for multistructure incidental findings and mortality prediction at chest CT in lung can- cer screening. Radiology 312(3), e240541 (2024). https://doi.org/10.1148/radiol. 240541 15. Mikhael, P.G., Wohlwend, J., Yala, A., Karstens, L., Xiang, J., Takigami, A.K., Bourgouin, P.P., Chan, P., Mrah, S., Amayri, W., Juan, Y.H., Yang, C.T., Wan, Y.L., Lin, G., Sequist, L.V., Fintelmann, F.J., Barzilay, R.: Sybil: A validated deep learning model to predict future lung cancer risk from a single low-dose chest computed tomography. Journal of Clinical Oncology 41(12), 2191–2200 (2023) 16. Moreno-Aguado, R., Magallón, A., Moreno, V., Fang, Y., Yang, G.: Learning ro- bust visual features in computed tomography enables efficient transfer learning for clinical tasks. arXiv preprint arXiv:2604.04133 (2026) 17. Pai, S., et al.: Vision foundation models for computed tomography. arXiv preprint arXiv:2501.09001 (2025) 18. Saporta, A., Callard, B., Dancette, C., Khlaut, J., Corbière, C., Butsanets, L., Prat, A., Manceron, P.: Curia-2: scaling self-supervised learning for radiology foundation models. arXiv preprint arXiv:2604.01987 (2026) 19. Wald, T., Hamamci, I.E., Gao, Y., Bond-Taylor, S., et al.: Comprehensive language–image pre-training for 3d medical image understanding. arXiv preprint arXiv:2510.15042 (2025) 20. Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: A sur- vey. IEEE transactions on pattern analysis and machine intelligence 45(9), 10795– 10816 (2023)