Paper deep dive
FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs
David Restrepo, Chenwei Wu, Luis Filipe Nakayama, Miguel L. Martins, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging analysis, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark for evaluating retinal fundus models that considers vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted through supervised fine-tuning with low-rank adaptation (LoRA). Results show that no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards
Tags
Links
- Source: https://arxiv.org/abs/2609.33158v1
- Canonical: https://arxiv.org/abs/2609.33158v1
Trouble viewing inline? Open PDF directly â
Full Text
67,487 characters extracted from source content.
Expand or collapse full text
FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs David Restrepo Chenwei Wu Luis Filipe Nakayama Miguel L. Martins Stergios Christodoulidis Maria Vakalopoulou Enzo Ferrante CentraleSupĂ©lec, UniversitĂ© Paris-Saclay, France Cancer Data Science Unit, IHU PRISM, UniversitĂ© Paris-Saclay, CentraleSupĂ©lec, Gustave Roussy, INSERM, France University of Michigan, USA Federal University of SĂŁo Paulo, Brazil University of Porto, Portugal Aalborg University, Denmark Universidad de Buenos Aires, Argentina Abstract Progress in AI-based retinal image analysis has accelerated with the adoption of foundation models, yet evaluating their real-world reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging analysis, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark for evaluating retinal fundus models that considers vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted through supervised fine-tuning with low-rank adaptation (LoRA). Results show that no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards. 1 Introduction Retinal fundus photography is one of the most widely used imaging modalities for large-scale screening and monitoring for vision-threatening conditions [35]. However, it presents a unique challenge for foundation-model evaluation and training due to the confluence of dataset shift and label noise [10]. Beyond technical variations in acquisition (e.g., field of view, illumination), annotated labels exhibit significant inter-grader variability and differing levels of clinical expertise [10]. Foundation models trained on large datasets often overfit to overrepresented populations or specific acquisition hardware, creating hidden biases that limit clinical utility and fairness when deployed in diverse, real-world healthcare systems [31, 19, 27]. The evaluation problem has become more urgent as AI-based retinal image analysis has moved beyond task-specific classifiers. Current systems include foundation vision-only encoder models (VMs) [42, 26], dual encoder-only vision-language models (VLMs)[27, 32, 5], and generative instruction-tuned multimodal large language models (MLLMs)[29, 30]. These systems expose different prediction interfaces: the VM and VLM encoders generate embeddings and multi-modal embeddings, and the MLLMs return token logits to be used in an auto-regressive way. Without a shared evaluation layer, apparent gains can reflect changes in prompt template, threshold, calibration convention, or label mapping rather than real clinical transfer. Existing resources cover important pieces of this problem, but not the full benchmark need. Single datasets and challenges such as IDRiD [24], RFMiD [23], BRSET [17], mBRSET [41], PAPILA [9], and G1020 [2] provide valuable disease labels and sometimes demographic or image-quality metadata, but they do not by themselves test cross-dataset transfer across model families (VM, VLM-encoders, MLLMs). Recent MLLM and LVLM benchmarks, such as FunBench [39] and LMOD [25], evaluate ophthalmic reasoning, but they are not designed as a harmonized evaluation suite for measuring disease-classification transfer across the full architectural spectrum covering VMs, VLMs, and MLLMs systems together. Table 1 summarizes this gap: prior resources usually cover either diseases, metadata, or generative models, while few combine evaluations on some of the Diabetic Retinopathy (DR), referable DR, glaucoma, out-of-domain evaluation, fairness, quality robustness, VLMs, MLLMs, and fine-tuning transfer in one framework. In this study, we present FOCUS, a benchmark for comparing foundation models from vision-only encoders to generative multimodal models. The benchmark asks a deliberately practical question: when foundation models are adapted to domain-specific fundus data, do they improve the target task only on the training dataset, or do they also preserve ranking performance, calibration, subgroup behavior, and image-quality robustness? By combining ten fundus datasets across multiple regions and acquisition conditions, FOCUS creates a broader test set for out-of-distribution tasks and demographic variation than any single dataset can provide. Our contributions are: (1) a dataset registry and task harmonization layer for binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy across ten fundus datasets; (2) a unified evaluation layer covering 532 modelâdataset-task-method configurations, including 190 for VMs, 228 for VLMs, and 114 for MLLMs; (3) an evaluation across four axes: performance, calibration, fairness, and image-quality robustness; and (4) a Supervised Fine Tuning (SFT) Low-rank adaptation (LoRA) analysis for MLLMs for the four evaluation axes, comparing each adapter with its matching base model on same-dataset test sets and same-task external datasets. We release the code and modular pipelines for benchmark evaluation and dataset downloading/cleaning through a repository, together with an interactive visualization dashboard. 2 Related Work VMs and VLMs for fundus imaging. Retinal AI has moved from task-specific "specialist" models toward foundation models that differ in domain pretraining and output interface. FOCUS therefore keeps VM, VLM, and MLLM interfaces separate while evaluating them through one harmonized retinal analysis layer. General VM/VLM backbones such as ViT [8], DINOv2 [20], DINOv3 [34], CLIP [28], and SigLIP2 [37] provide broad visual or image-text representations. Medical and ophthalmic variants adapt these ideas to medical images: RETFound [42] and VisionFM [26] specialize VM pretraining for ophthalmology, MedSigLIP [30] adapts dual-encoder image-text modeling to medical images, and FLAIR [33], RET-CLIP [5], and EyeCLIP [32] focus on retinal or ophthalmic VLM pretraining. MLLMs for fundus imaging. Generative MLLMs add a prompt-based interface with text outputs and token probabilities. General models such as Gemma-3 [7], Qwen3-VL [1], and LLaVA-NeXT [13] provide broad image-language reasoning, while medical or ophthalmic models such as LLaVA-Med [12], Med-Flamingo [14], MedGemma [30], MedGemma 1.5 [29], and OphGLM [4] adapt generative VLMs using biomedical captions, medical instruction tuning, or ophthalmology chat data. Benchmarking retinal and ophthalmic models. Public fundus datasets and challenges have been essential for diabetic retinopathy, referable diabetic retinopathy, glaucoma, and multi-disease recognition, including BRSET[16], mBRSET[41, 18], PAPILA[9], RFMiD[22], RFMiD 2.0[23], IDRiD[24], Messidor-2[11], G1020[2], REFUGE[21], and JSIEC1000 [3]. It is common for methodological studies to conduct their experiments on a limited set of datasets, limited or heterogeneous model interfaces and downstream performance metrics. Ophthalmological benchmarks have thus been proposed to evaluate models accross methodologies in a unified way. FunBench evaluates fundus-reading skills of MLLMs with vision-question answering (VQA) tasks [39]; LMOD focuses on measuring large-scale ophthalmological VLM vulnerabilities, such as hallucinations [25]; MultiEYE and MIRAGE emphasize multimodal or optimal coherence tomography (OCT)-centered retinal analysis [38, 15]. While these provide valuable insights into specific failure modes, they often focus on a single model paradigm (e.g., generative models) or lack a harmonized framework for cross-dataset disease transfer. Figure 1: Multimetric arena view. Spider plots summarize the top 6 VM, VLM-encoders, and MLLM configurations across AUROC, AUPRC, accuracy, calibration, fairness diagnostics, and image-quality robustness. Colors distinguish models; LP = Linear probing, ZS = Zero-shot, ZS Prompt = Zero-shot prompting. This view makes visible that strong ranking performance need not imply strong calibration or balanced subgroup/quality behavior. The proposed benchmark. FOCUS was created specifically for colored fundus photographs and it is designed to assess whether the foundation models generalize beyond their training or fine-tuning datasets. As a whole, the data in FOCUS is diverse in terms of acquisition conditions, disease tasks and demographic variables (e.g., age, gender, geography). Specifically, it provides a plug-and-play evaluation tool covering a wide range of population and acquisition shifts for same-task out-of-domain evaluation, while simultaneously harmonizing the available metadata to facilitate analysis of prediction calibration, age/sex subgroup gaps, and image-quality robustness. Table 1 summarizes this gap. Table 1: Coverage comparison with related fundus datasets, challenges, and ophthalmic benchmarks. Resource Type DR Ref. DR Glauc. multi- reg. Fair. Qual. VM VLM MLLM FT trans. IDRiD [24] DR challenge Y Y â â â â D â â â RFMiD / RFMiD 2.0 [22, 23] Multi-disease fundus datasets Y D D â â â D â â â BRSET / mBRSET [17, 16, 18, 41] Brazilian fundus datasets Y Y Y D D D D â â â PAPILA / G1020 / REFUGE [9, 2, 21] Glaucoma datasets/challenges â â Y D D â D â â â FunBench [39] Fundus MLLM VQA benchmark VQA VQA VQA Y â â â â Y â LMOD [25] Ophthalmology LVLM benchmark VQA VQA VQA Y D â â D Y â MultiEYE / MIRAGE [38, 15] Multimodal/OCT retinal benchmarks D â D Y â â Y D D â FOCUS Cross-dataset fundus benchmark Y Y Y Y Y Y Y Y Y Y Y: explicit benchmark evaluation; D: labels/metadata available but not a cross-model benchmark axis; VQA: broader visual-question-answering or MLLM evaluation; â: not central to the resource. 3 FOCUS Benchmark Design 3.1 Datasets and tasks FOCUS targets color fundus photography datasets with complementary metadata relating to the acquisition conditions, demographics, and a plurality of ophthalmological downstream tasks. The current dataset registry includes BRSET [17, 16], mBRSET [18, 41], PAPILA [9], RFMiD [22], RFMiD 2.0 [23], IDRiD [24], Messidor-2 labels [11], REFUGE [21], G1020 [2], and JSIEC1000 [3]. Not all datasets are uniformly annotated, thus task-specific analysis are conditioned only on the subset of data containing valid ground truth annotations. An overview of the datasets can be seen in Appendix B. Task harmonization. FOCUS harmonizes dataset-specific annotations into three binary clinical tasks: any diabetic retinopathy (DR), referable DR, and glaucomatous optic neuropathy (GON). Any DR captures early pathological signs using ICDR grade â„1â„ 1 [40], while referable DR reflects the screening decision to refer moderate non-proliferative DR or worse (ICDR grade â„2â„ 2) and/or suspicious macular edema. We use GON rather than definitive glaucoma because fundus photography captures structural risk markers, such as neuroretinal rim thinning or increased vertical cup-to-disc ratio; labels are harmonized using VCDR â„0.6â„ 0.6 or specialist annotation of glaucomatous damage [6]. Together, these tasks span localized vascular lesions, referral-level disease, and global optic-disc structure. 3.2 Model evaluation This benchmark separates two axes that are often conflated: domain specificity and output interface (see Table 2). VMs are evaluated through frozen-feature linear probes, including retinal foundation encoders such as RETFound [42]. VLMs are evaluated through zero-shot image-text embedding matching and frozen-image-feature linear probing. MLLMs are evaluated prompting the model to return binary outputs (âyesâ or ânoâ), and then using the first-token positive-class probabilities (see Appendix E.3 for details). Table 2: Model families represented in the current FOCUS result bundle. Family Domain Architectures Interface VM Generalist ViT-L/16, DINOv2-L, DINOv3-ViT-L Image embeddings VM Ophthalmic RETFound variants, VisionFM-Fundus Image embeddings VLM Generalist CLIP, SigLIP2, MedSigLIP Joint image-text embeddings VLM Ophthalmic EyeCLIP, RET-CLIP, FLAIR Joint image-text embeddings MLLM Generalist Qwen3-VL, Gemma-3, LLaVA-NeXT Text output and token logits MLLM Medical MedGemma variants Text output and token logits 3.3 Evaluation axes FOCUS is organized around four primary evaluation axes. Performance is assessed using threshold-free metrics (e.g. AUROC) that summarize how well model scores separate positive from negative cases across operating points, together with threshold-dependent metrics that reflect classification quality at a fixed decision threshold (e.g. Accuracy). Calibration measures whether predicted positive-class scores correspond to the observed frequency of positive classes for each harmonized task. Fairness summarizes differences in performance across age and sex subgroups (when the required metadata is available). Robustness evaluates whether performance and calibration remain stable across image-quality strata for datasets with quality annotations. Image quality is taken from the dataset-provided quality metadata when available, typically reflecting acquisition or grading quality labels assigned during dataset curation. Formal metric definitions and notation are provided in Appendix G. Fine-tuned MLLMs are evaluated separately from the base benchmark. We train SFT LoRA adapters only on BRSET and mBRSET, because these are the configured training datasets that support all three harmonized tasks and include the row-level metadata needed for performance, calibration, fairness, and image-quality robustness analyses. For each modelâtraining datasetâtask combination, we measure in-dataset performance on the held-out test split of the training dataset and same-task transfer on every other benchmark dataset that supports the task. Results are reported as metric deltas relative to the matching base MLLM evaluated on the same test dataset and task. Figure 2: Benchmark pipeline. FOCUS connects dataset/task harmonization, model-interface-specific inference, multidimensional evaluation, LoRA adaptation tests, and arena-style reporting. The intent is to measure whether fundus-specific adaptation generalizes rather than only improving the dataset used for training. 4 Empirical Results The base benchmark suite covers 532 modelâdatasetâtaskâmethod configurations: of those 190 correspond to VMs, 228 to VLMs, and 114 to MLLMs. The language-model fine-tuning suite covers 228 SFT adapter evaluations and their full details are in Appendices C and F. Tab.3 shows the method-average results which we will unpack in the rest of this section : across four axes of performance, calibration, fairness and robustness, no single model or model class dominates. Finding 1: Generalist pretrained VM encoder remains a surprisingly strong and robust retinal baseline. General pretrained VM encoders achieve the strongest mean AUROC in FOCUS, surpassing ophthalmic-domain VM encoders across most datasets. As shown in Tab. 4, the largest gap appears on mBRSET (0.162), the only mobile-phone fundus dataset in the benchmark. This suggests that current ophthalmic VM pretraining may be less robust to acquisition shifts outside conventional desktop fundus photography. Because mBRSET also differs in population and label composition, this gap should not be interpreted as isolating device shift alone; rather, it highlights a deployment-relevant setting where ophthalmic-domain pretraining underperforms generalist visual pretraining. Strong generalist visual representations remain difficult baselines to beat, especially under cross-dataset and device-shift evaluation. As for other model groups, VLM performance depends on the evaluation interface: Ophthalmic VLMs outperform general VLMs on mean AUROC in zero-shot settings but fall short in linear probing, showing lower performance ceiling. Lastly, medical MLLMs substantially outperform general MLLMs. Table 3: Method-aware mean metrics by model type and task. Higher is better except for ECE. Task Model type Method Accuracy AUROC AUPRC ECE Fairness Quality Shift Binary DR General VM Linear probing 0.813 0.867 0.782 0.188 0.932 0.913 0.739 Ophthalmic VM Linear probing 0.720 0.690 0.537 0.206 0.922 0.910 0.703 General VLM-encoders Linear probing 0.819 0.865 0.784 0.179 0.936 0.874 0.746 General VLM-encoders Zero-shot 0.477 0.547 0.346 0.311 0.948 0.897 0.840 Ophthalmic VLM-encoders Linear probing 0.780 0.814 0.702 0.151 0.916 0.911 0.801 Ophthalmic VLM-encoders Zero-shot 0.747 0.698 0.522 0.215 0.954 0.900 0.845 General MLLMs Zero-shot prompting 0.521 0.685 0.457 0.435 0.948 0.911 0.742 Medical MLLMs Zero-shot prompting 0.791 0.868 0.707 0.194 0.936 0.939 0.820 Referable DR General VM Linear probing 0.858 0.850 0.556 0.194 0.907 0.857 0.741 Ophthalmic VM Linear probing 0.765 0.697 0.363 0.233 0.913 0.916 0.799 General VLM-encoders Linear probing 0.845 0.853 0.544 0.189 0.908 0.839 0.860 General VLM-encoders Zero-shot 0.654 0.596 0.314 0.209 0.940 0.907 0.817 Ophthalmic VLM-encoders Linear probing 0.776 0.770 0.528 0.178 0.910 0.903 0.674 Ophthalmic VLM-encoders Zero-shot 0.440 0.672 0.360 0.322 0.952 0.930 0.739 General MLLMs Zero-shot prompting 0.517 0.651 0.393 0.433 0.932 0.940 0.446 Medical MLLMs Zero-shot prompting 0.775 0.844 0.700 0.208 0.916 0.959 0.389 Glaucoma General VM Linear probing 0.712 0.742 0.471 0.259 0.872 0.884 0.747 Ophthalmic VM Linear probing 0.707 0.591 0.312 0.263 0.915 0.872 0.903 General VLM-encoders Linear probing 0.717 0.745 0.474 0.248 0.924 0.875 0.771 General VLM-encoders Zero-shot 0.516 0.488 0.226 0.317 0.921 0.869 0.880 Ophthalmic VLM-encoders Linear probing 0.721 0.719 0.527 0.230 0.919 0.919 0.806 Ophthalmic VLM-encoders Zero-shot 0.757 0.541 0.266 0.162 0.940 0.924 0.878 General MLLMs Zero-shot prompting 0.730 0.630 0.418 0.259 0.964 1.000 0.489 Medical MLLMs Zero-shot prompting 0.816 0.656 0.470 0.176 0.957 1.000 0.482 Overall General VM Linear probing 0.803 0.828 0.617 0.209 0.899 0.885 0.654 Ophthalmic VM Linear probing 0.733 0.667 0.414 0.231 0.917 0.899 0.762 General VLM-encoders Linear probing 0.802 0.829 0.614 0.201 0.923 0.863 0.661 General VLM-encoders Zero-shot 0.552 0.549 0.303 0.275 0.934 0.891 0.800 Ophthalmic VLM-encoders Linear probing 0.763 0.773 0.592 0.182 0.915 0.911 0.732 Ophthalmic VLM-encoders Zero-shot 0.636 0.647 0.395 0.240 0.948 0.918 0.729 General MLLMs Zero-shot prompting 0.575 0.658 0.423 0.388 0.950 0.950 0.603 Medical MLLMs Zero-shot prompting 0.791 0.803 0.642 0.194 0.939 0.966 0.567 Additional task-level comparisons and leaderboard views can be found in the Appendix, in Figures 26 and 27. Table 4: Per-dataset mean AUROC for VM encoders. Î=AUROCgenâAUROCoph =AUROC_gen-AUROC_oph. Dataset Country Device Gen. Oph. Î mBRSET Brazil mobile 0.754 0.592 +0.162 IDRiD India desktop 0.825 0.692 +0.133 BRSET Brazil desktop 0.822 0.700 +0.122 JSIEC1000 China desktop 0.958 0.851 +0.106 Messidor-2 France desktop 0.836 0.759 +0.077 RFMiD India desktop 0.914 0.849 +0.066 PAPILA Spain desktop 0.766 0.710 +0.056 G1020 Germany desktop 0.631 0.614 +0.017 RFMiD-2 India desktop 0.846 0.918 -0.072 Figure 3: Performance vs. reliability and fairness, by model family. Each point is one (model, method) configuration; rows index axis (top: age equalised-odds gap; bottom: ECE), columns index family (CV / VLM / MLLM). Marker colour encodes model type, marker shape encodes adaptation method (circle = linear probe, triangle = zero-shot, square = prompted). Both axes are inverted so that âupâ is desirable on every panel. The dotted line in each panel is a least-squares fit on that panelâs points; red denotes a positive AUROCâgap slope (a rankingâaxis trade-off) and green a negative slope (aligned). Pearsonâs r is reported in each panel (*p<0.05p<0.05, **p<0.01p<0.01, ***p<0.001p<0.001). Finding 2: Choosing a strong AUROC model does not necessarily sacrifice calibration, but it can sacrifice demographic equity. As shown in Fig. 3, not all deployment axes relate to AUROC in the same way. Calibration tends to improve with ranking performance, whereas rate-based demographic fairness gaps tend to worsen. For calibration, ECE is negatively correlated with AUROC, indicating that better ranking models are also better calibrated on average. This pattern holds especially strong within VM and VLM configurations. Fairness behaves differently. At the family level, VM encoders achieve the highest mean AUROC. The reversal is clearest for age subgroup fairness: VM models have the largest age equalized-odds gap, while MLLMs have the smallest. Demographic parity, equal opportunity, and equalized odds gaps all show positive correlations with mean AUROC, with equal opportunity and equalized odds reaching statistical significance. Additionally, different adaptation methods can amplify the disparity. For the VLMs with both zero-shot and linear-probe results, linear probing improves AUROC for most models but increases the fairness gap for all six. Finding 3: Glaucoma is the task cliff where retinal AI progress stalls. Figure 4: Glaucoma is the clearest task-level failure in FOCUS. As shown in Fig.4, glaucoma is the obvious task domain where current retinal AI progress lags behind. Every adapted model family drops when moving from diabetic retinopathy to glaucoma. Generalist CV encoders fall from 0.872 AUROC on binary DR to 0.701 on glaucoma. Generalist VLMs with linear probing fall from 0.868 to 0.714. Ophthalmic CV and VLM encoders each lose at least 0.10 AUROC. The largest drop is for medical MLLMs, which match the strongest models on binary DR but fall from 0.868 to 0.642 on glaucoma. This pattern shows that glaucoma is not solved by domain pretraining, medical instruction tuning, model scale, or lightweight adaptation. The best glaucoma group is generalist VLMs with a linear probe, reaching only 0.714 mean AUROC. On G1020, the hardest external glaucoma dataset, the best single configuration reaches 0.746 AUROC. Zero-shot VLMs fail most severely. Generalist VLM zero-shot evaluation reaches only near chance mean AUROC on glaucoma. Because glaucoma labels and imaging protocols vary substantially across datasets, this gap may reflect both intrinsic visual difficulty and dataset-level heterogeneity. Nevertheless, the consistency of the drop across model families indicates that current foundation-model strategies do not remove this bottleneck. This is clinically important because glaucoma is a leading cause of irreversible blindness [36] and should not be overlooked in retinal foundation-model evaluation. Finding 4: Specialization beats scale for MLLMs. The MLLM comparison separates two factors that are often conflated: parameter count and medical specialization. The result is clear: medical tuning gives the main gain, while additional scale does not reliably improve retinal performance. At fixed 27B scale, MedGemma-27B strongly outperforms Gemma-3-27B, increasing mean AUROC from 0.648 to 0.797 and reducing ECE from 0.36 to 0.19. This shows that medical specialization improves both ranking and calibration. Scaling within the medical family tells a different story. Within the evaluated MedGemma models, scaling from 4B to 27B does not improve mean AUROC. The gain is more consistent with domain specialization than with parameter count alone. The reliability axes also show that scale is not a clean solution. Fairness and image-quality gaps vary across models rather than following parameter count. The main takeaway is that medical specialization improves retinal MLLMs, but scaling from 4B to 27B does not guarantee better AUROC or uniformly better deployment behavior. Figure 5: Medical specialization improves retinal MLLMs, but scaling from 4B to 27B does not guarantee better AUROC or uniformly better deployment behavior. Finding 5: LoRA SFT improves calibration and thresholded decisions, not ranking. As shown in Tab. 5 and Fig. 6 supervised fine-tuning helps MLLMs behave more consistently at the decision threshold, but it does not meaningfully improve average disease ranking. In-domain SFT improves accuracy by +0.033 and ECE by -0.040, both significant after correction. The same pattern mostly transfers OOD, with +0.012 accuracy and -0.018 ECE. In contrast, AUROC barely changes: +0.004 in-domain and +0.002 OOD, with non-significant q-values. This means SFT is not teaching the models a substantially better ordering of positive and negative retinal cases. Instead, it mainly changes how the model maps its existing signal into binary answers and confidence scores. Table 5: Language-model fine-tuning transfer summary. Deltas are SFT metric minus matching base MLLM metric; negative Î indicates improved calibration. Bootstrap q-values apply BenjaminiâHochberg correction within each metric family. Test domain AUROC Î q Î q Î q In-domain 0.776 0.004 0.477 0.033 0.004 -0.040 0.004 OOD 0.724 0.002 0.521 0.012 0.005 -0.018 0.004 Figure 6: Language-model fine-tuning deltas. Left: aggregate deltas versus base MLLMs on the same test dataset/task. Right: model-level deltas show that SFT effects are not uniform across base MLLMs or domains (negative Î indicates improved calibration). 5 Discussion FOCUS is designed to test whether foundation-model adaptation to fundus photography generalizes beyond the narrow training setting. The current base benchmark shows that model interface and domain specialization interact in nontrivial ways: general VM encoders are strong and stable, medical MLLMs are competitive but not uniformly dominant, and VLMs can be weak in zero-shot form while becoming strong under linear probing. The language-model fine-tuning analysis adds a second layer: fine-tuning can improve in-domain thresholded behavior, but the same adapter may produce smaller or less reliable gains on external datasets. These findings argue against a single retinal AI leaderboard. A clinically relevant benchmark should expose the trade-offs among ranking performance, calibration, subgroup behavior, image-quality robustness, and OOD transfer. FOCUS makes these axes explicit and keeps base and fine-tuned analyses separated so future work can measure whether adaptation helps, harms, or merely shifts failure modes. 6 Limitations and Intended Use FOCUS is not a clinical validation study and should not be used to claim deployment readiness. Dataset labels are inherited from heterogeneous public or credentialed sources and may reflect different grading protocols. Some datasets lack demographic or image-quality metadata, limiting fairness and robustness analyses. Subgroup gaps are descriptive diagnostics, not causal fairness claims. Generative-model results depend on prompt templates, answer parsing, and token-probability availability. Benchmark scores should therefore be interpreted as evidence about controlled retrospective evaluation, not as evidence of safety for diagnosis or screening. 7 Reproducibility Code and benchmark results are available through a repository and an interactive leaderboard dashboard. The repository documentation describes the modular dataset and model interfaces, which can be used for dataset downloading/cleaning and to support additional fundus datasets, label mappings, tasks, and model families. Acknowledgments and Disclosure of Funding Funding. This work was supported by the European Unionâs Horizon Europe research and innovation programme under the Marie SkĆodowska-Curie COFUND grant agreement No. 101127936 (DeMythif.AI), and by France 2030 funding managed by the French National Research Agency (ANR) under the IA Cluster programme (DATAIA Cluster; grant No. ANR-23-IACL-0003). Computational resources. This work was granted access to the HPC resources of the Jean Zay supercomputer operated by IDRIS (CNRS) and to the Ruche MĂ©socentre of UniversitĂ© Paris-Saclay. Competing interests. The authors declare no competing interests. References [1] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §2. [2] M. N. Bajwa, G. A. P. Singh, W. Neumeier, M. I. Malik, A. Dengel, and S. Ahmed (2020) G1020: a benchmark retinal fundus image dataset for computer-aided glaucoma detection. In 2020 International Joint Conference on Neural Networks (IJCNN), p. 1â7. Cited by: §1, §2, Table 1, §3.1. [3] L. Cen, J. Ji, J. Lin, S. Ju, H. Lin, T. Li, Y. Wang, J. Yang, Y. Liu, S. Tan, et al. (2021) Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nature communications 12 (1), p. 4828. Cited by: §2, §3.1. [4] Z. Deng, W. Gao, C. Chen, Z. Niu, Z. Gong, R. Zhang, Z. Cao, F. Li, Z. Ma, W. Wei, et al. (2024) OphGLM: an ophthalmology large language-and-vision assistant. Artificial Intelligence in Medicine 157, p. 103001. Cited by: §2. [5] J. Du, J. Guo, W. Zhang, S. Yang, H. Liu, H. Li, and N. Wang (2024) Ret-clip: a retinal image foundation model pre-trained with clinical diagnostic reports. In International conference on medical image computing and computer-assisted intervention, p. 709â719. Cited by: §1, §2. [6] P. J. Foster, R. Buhrmann, H. A. Quigley, and G. J. Johnson (2002) The definition and classification of glaucoma in prevalence surveys. British journal of ophthalmology 86 (2), p. 238â242. Cited by: §3.1. [7] G. T. A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramâe, M. RiviĂšre, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. I. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. Gyorgy, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Pluciânska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. M. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. StaĆczyk, P. D. Tafti, R. Shivanna, R. Wu, R. Pan, R. A. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. PĂ”der, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. S. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, D. Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot (2025) Gemma 3 technical report. ArXiv abs/2503.19786. External Links: Link Cited by: §2. [8] A. Kolesnikov, A. Dosovitskiy, D. Weissenborn, G. Heigold, J. Uszkoreit, L. Beyer, M. Minderer, M. Dehghani, N. Houlsby, S. Gelly, T. Unterthiner, and X. Zhai (2021) An image is worth 16x16 words: transformers for image recognition at scale. Cited by: §2. [9] O. Kovalyk, J. Morales-SĂĄnchez, R. VerdĂș-Monedero, I. SellĂ©s-Navarro, A. PalazĂłn-Cabanes, and J. Sancho-GĂłmez (2022) PAPILA: dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment. Scientific Data 9 (1), p. 291. Cited by: §1, §2, Table 1, §3.1. [10] J. Krause, V. Gulshan, E. Rahimy, P. Karth, K. Widner, G. S. Corrado, L. Peng, and D. R. Webster (2018) Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125 (8), p. 1264â1272. Cited by: §1. [11] J. Krause, V. Gulshan, E. Rahimy, P. Karth, K. Widner, G. S. Corrado, L. Peng, and D. R. Webster (2018) Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125 (8), p. 1264â1272. Cited by: §2, §3.1. [12] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao (2023) Llava-med: training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36, p. 28541â28564. Cited by: §2. [13] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024) Llavanext: improved reasoning, ocr, and world knowledge. Cited by: §2. [14] M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar (2023) Med-flamingo: a multimodal medical few-shot learner. In Machine learning for health (ML4H), p. 353â367. Cited by: §2. [15] J. Morano, B. Fazekas, E. SĂŒkei, R. Fecso, T. Emre, M. Gumpinger, G. Faustmann, M. Oghbaie, U. Schmidt-Erfurth, and H. BogunoviÄ (2025) Multimodal foundation model and benchmark for comprehensive retinal oct image analysis. NPJ Digital Medicine 8 (1), p. 576. Cited by: §2, Table 1. [16] L. F. Nakayama, M. Goncalves, L. Z. Ribeiro, H. Santos, D. Ferraz, F. Malerbi, L. A. Celi, and C. Regatieri (2023) A brazilian multilabel ophthalmological dataset (brset). PhysioNet 13026, p. 2. Cited by: §2, Table 1, §3.1. [17] L. F. Nakayama, D. Restrepo, J. Matos, L. Z. Ribeiro, F. K. Malerbi, L. A. Celi, and C. S. Regatieri (2024) BRSET: a brazilian multilabel ophthalmological dataset of retina fundus photos. PLOS Digital Health 3 (7), p. e0000454. Cited by: §1, Table 1, §3.1. [18] L. F. Nakayama, L. Zago Ribeiro, D. Restrepo, N. Santos Barboza, R. Dias Fiterman, M. Vieira Sousa, A. Pereira, C. Regatieri, F. Malerbi, and R. Andrade (2024) MBRSET, a mobile brazilian retinal dataset. PhysioNet https://doi. org/10.13026/QXPD-1Y65. Cited by: §2, Table 1, §3.1. [19] C. Nielsen, E. A. Stanley, M. Wilms, and N. D. Forkert (2025) Assessment of demographic bias in retinal age prediction machine learning models. Frontiers in Artificial Intelligence 8, p. 1653153. Cited by: §1. [20] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §2. [21] J. I. Orlando, H. Fu, J. B. Breda, K. Van Keer, D. R. Bathula, A. Diaz-Pinto, R. Fang, P. Heng, J. Kim, J. Lee, et al. (2020) Refuge challenge: a unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Medical image analysis 59, p. 101570. Cited by: §2, Table 1, §3.1. [22] S. Pachade, P. Porwal, D. Thulkar, M. Kokare, G. Deshmukh, V. Sahasrabuddhe, L. Giancardo, G. Quellec, and F. MĂ©riaudeau (2021) Retinal fundus multi-disease image dataset (rfmid): a dataset for multi-disease detection research. Data 6 (2), p. 14. Cited by: §2, Table 1, §3.1. [23] S. Panchal, A. Naik, M. Kokare, S. Pachade, R. Naigaonkar, P. Phadnis, and A. Bhange (2023) Retinal fundus multi-disease image dataset (rfmid) 2.0: a dataset of frequently and rarely identified diseases. Data 8 (2), p. 29. Cited by: §1, §2, Table 1, §3.1. [24] P. Porwal, S. Pachade, R. Kamble, M. Kokare, G. Deshmukh, V. Sahasrabuddhe, and F. Meriaudeau (2018) Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data 3 (3), p. 25. Cited by: §1, §2, Table 1, §3.1. [25] Z. Qin, Y. Yin, D. Campbell, X. Wu, K. Zou, N. Liu, Y. C. Tham, X. Zhang, and Q. Chen (2025) LMOD: a large multimodal ophthalmology dataset and benchmark for large vision-language models. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 2501â2522. External Links: Link, Document, ISBN 979-8-89176-195-7 Cited by: §1, §2, Table 1. [26] J. Qiu, J. Wu, H. Wei, P. Shi, M. Zhang, Y. Sun, L. Li, H. Liu, H. Liu, S. Hou, et al. (2023) Visionfm: a multi-modal multi-task vision foundation model for generalist ophthalmic artificial intelligence. arXiv preprint arXiv:2310.04992. Cited by: §1, §2. [27] D. Queiroz, A. Carlos, A. Anjos, and L. Berton (2026) Fair foundation models for medical image analysis: challenges and perspectives. ACM Transactions on Computing for Healthcare 7 (2), p. 1â24. Cited by: §1, §1. [28] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748â8763. Cited by: §2. [29] A. Sellergren, C. Gao, F. Mahvar, T. Kohlberger, F. Jamil, M. Traverse, A. Tono, B. Sadjad, L. Yang, C. Lau, et al. (2026) Medgemma 1.5 technical report. arXiv preprint arXiv:2604.05081. Cited by: §1, §2. [30] A. Sellergren, S. Kazemzadeh, T. Jaroensri, A. Kiraly, M. Traverse, T. Kohlberger, S. Xu, F. Jamil, C. Hughes, C. Lau, et al. (2025) Medgemma technical report. arXiv preprint arXiv:2507.05201. Cited by: §1, §2, §2. [31] L. Seyyed-Kalantari, H. Zhang, M. B. McDermott, I. Y. Chen, and M. Ghassemi (2021) Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine 27 (12), p. 2176â2182. Cited by: §1. [32] D. Shi, W. Zhang, J. Yang, S. Huang, X. Chen, P. Xu, K. Jin, S. Lin, J. Wei, M. Yusufu, et al. (2025) A multimodal visualâlanguage foundation model for computational ophthalmology. npj digital medicine 8 (1), p. 381. Cited by: §1, §2. [33] J. Silva-Rodriguez, H. Chakor, R. Kobbi, J. Dolz, and I. B. Ayed (2025) A foundation language-image model of the retina (flair): encoding expert knowledge in text supervision. Medical Image Analysis 99, p. 103357. Cited by: §2. [34] O. SimĂ©oni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. (2025) Dinov3. arXiv preprint arXiv:2508.10104. Cited by: §2. [35] M. Soleimani, F. Alipour, Y. Taghavi, M. Fatemipour, H. Hakimi, Z. Jamali, P. Khalili, F. Ayoobi, M. Sheikh, R. Tavakoli, et al. (2023) Single-field fundus photography for screening of diabetic retinopathy: the prevalence and associated factors in a population-based study. Diabetes Therapy 14 (1), p. 205â217. Cited by: §1. [36] Y. Tham, X. Li, T. Y. Wong, H. A. Quigley, T. Aung, and C. Cheng (2014) Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. Ophthalmology 121 (11), p. 2081â2090. Cited by: §4. [37] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. (2025) Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §2. [38] L. Wang, C. Qi, C. Ou, L. An, M. Jin, X. Kong, and X. Li (2024) MultiEYE: dataset and benchmark for oct-enhanced retinal disease recognition from fundus images. IEEE Transactions on Medical Imaging 44 (4), p. 1711â1722. Cited by: §2, Table 1. [39] Q. Wei, K. Qian, and X. Li (2025) FunBench: benchmarking fundus reading skills of mllms. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 278â288. Cited by: §1, §2, Table 1. [40] C. P. Wilkinson, F. L. Ferris I, R. E. Klein, P. P. Lee, C. D. Agardh, M. Davis, D. Dills, A. Kampik, R. Pararajasegaram, J. T. Verdaguer, et al. (2003) Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110 (9), p. 1677â1682. Cited by: §3.1. [41] C. Wu, D. Restrepo, L. F. Nakayama, L. Zago Ribeiro, Z. Shuai, N. S. Barboza, M. L. V. Sousa, R. D. Fitterman, A. D. A. Pereira, C. V. S. Regatieri, et al. (2025) A portable retina fundus photos dataset for clinical, demographic, and diabetic retinopathy prediction. Scientific Data 12 (1), p. 323. Cited by: §1, §2, Table 1, §3.1. [42] Y. Zhou, M. A. Chia, S. K. Wagner, M. S. Ayhan, D. J. Williamson, R. R. Struyven, T. Liu, M. Xu, M. G. Lozano, P. Woodward-Court, et al. (2023) A foundation model for generalizable disease detection from retinal images. Nature 622 (7981), p. 156â163. Cited by: §1, §2, §3.2. Appendix A Benchmark Card Name. FOCUS: Foundation Ophthalmic Cross-Dataset Understanding under Shift. Purpose. FOCUS is intended for retrospective evaluation of retinal fundus models under dataset shift. The core question is whether foundation-model adaptation to fundus photography improves generalization and robustness, rather than only improving the test split of the dataset used for fine-tuning. Intended use. Researchers may use FOCUS to compare model-family transfer, calibration, subgroup gaps, image-quality robustness, and supervised language-model fine-tuning behavior across harmonized retinal tasks. Non-intended use. FOCUS is not intended for clinical deployment validation, diagnosis, screening triage, regulatory approval, or patient-level decision making. Primary reporting axes. The benchmark reports AUROC, AUPRC, accuracy, F1, expected calibration error (ECE), fairness gaps for binary age/sex subgroups where metadata are available, and image-quality robustness gaps where quality strata are available. Metrics outside these implemented axes are omitted from the current report. Appendix B Dataset and Task Matrix Dataset Binary DR Referable DR Glaucoma BRSET Yes Yes Yes mBRSET Yes Yes Yes PAPILA No No Yes RFMiD Yes Yes No RFMiD 2.0 Yes Yes No IDRiD Yes Yes No Messidor-2 Yes Yes No REFUGE No No Yes G1020 No No Yes JSIEC1000 Yes Yes No Table 6: Configured dataset-task availability. A model is evaluated on a dataset-task pair only when the task label is available after harmonization. Table 7: Datasets configured in FOCUS. Tasks use the harmonized definitions in Sec. 3.1; Fairness and robustness analyses are computed only when metadata are available. Dataset Source Tasks Extra metadata BRSET Brazil Any DR, ref. DR, GON Age, sex, image quality mBRSET Brazil Any DR, ref. DR, GON Age, sex, image quality PAPILA Spain GON Age, sex RFMiD India Any DR, ref. DR â RFMiD 2.0 India Any DR, ref. DR â IDRiD India Any DR, ref. DR â Messidor-2 France Any DR, ref. DR â REFUGE China GON â G1020 Germany GON â JSIEC1000 China Any DR, ref. DR â Appendix C Experiment Design C.1 Base benchmark The base benchmark evaluates three model families: VM encoders, VLMs, and MLLMs. The current result bundle contains 532 evaluated configurations: 190 VM configurations, 228 VLM configurations, and 114 MLLM configurations. Each configuration corresponds to one model, dataset, task, and evaluation method. Data splits. For VM and VLM linear probing, the image encoder is frozen. The linear head is trained on the datasetâs training split and evaluated on its test split. For VLM zero-shot and MLLM prompting, no task-specific fitting is performed; the model is evaluated on the test split. Missing labels are removed before metric calculation. Task labels. All primary tasks are binary. Dataset-specific labels are harmonized into three target definitions: any diabetic retinopathy, referable diabetic retinopathy, and glaucoma. C.2 Fine-tuned MLLM transfer The fine-tuned analysis evaluates supervised LoRA adapters trained on BRSET or mBRSET for one task at a time and tests them on every configured dataset that supports the same task. The current SFT analysis contains 228 evaluations: 36 in-domain tests where the training and test datasets match, and 192 same-task out-of-domain (OOD) tests where they differ. Each adaptation result is indexed by the training model, training dataset, training task, test dataset, test task, and evaluation split. The analysis keeps only same-task transfer pairs, because the current metrics are binary task metrics rather than cross-task semantic transfer metrics. Appendix D Compute Resources All experiments were run on a shared high-performance computing environment using anonymized Slurm-style batch jobs. GPU jobs used NVIDIA H100-class accelerators with 80GB memory per GPU, CUDA 12.4, cuDNN 9.2, and NCCL 2.21. CPU-only jobs were used for dataset/model download orchestration and result aggregation. Table 8 reports the resources requested by the released job scripts; wall-clock values are conservative job time limits rather than exact runtimes. Experiment stage Worker type Memory Batch size Time limit Dataset downloads CPU, 4 cores system memory â 8h Model downloads CPU, 4 cores system memory â 2h VM base evaluation 1 H100 GPU, 24 CPU cores 80GB GPU 16 20h VLM base evaluation 1 H100 GPU, 24 CPU cores 80GB GPU 16 20h MLLM base prompting 1 H100 GPU, 24 CPU cores 80GB GPU 16 20h SFT LoRA training 4 H100 GPUs, 48 CPU cores 4Ă80GB GPU 1â4 18h SFT adapter evaluation 1 H100 GPU, 24 CPU cores 80GB GPU 16 20h Base result analysis CPU, 4 cores system memory â 2h FT result analysis/significance CPU, 4 cores system memory â 2h Table 8: Compute resources used by the benchmark jobs. The SFT LoRA batch size is 1 with gradient accumulation 8 for 27B-class models and 4 with gradient accumulation 4 for smaller models. VM/VLM/MLLM base evaluation jobs iterate over the configured modelâdatasetâtaskâmethod grid for each family. Appendix E Inference Setup by Model Type E.1 VM encoders For a frozen VM encoder fΞf_Ξ, each image xix_i is mapped to an embedding zi=fΞâ(xi).z_i=f_Ξ(x_i). The benchmark trains a balanced logistic-regression probe on the training split: âi=wâ€âzi+b,si=ÏâĄ(âi), _i=w z_i+b, s_i=Ï( _i), where sis_i is the positive-class score. The predicted label is the linear probeâs class prediction y^i y_i. E.2 VLMs VLMs are evaluated in two modes. In linear-probing mode, the image tower is frozen and the same logistic-regression formulation as above is applied to VLM image embeddings. In zero-shot mode, the task provides a negative and positive text prompt. Let viv_i be the image embedding and t0,t1t_0,t_1 the text embeddings. The implementation computes logits aiâc=100âviâ€âtc,câ0,1,a_ic=100\,v_i t_c, câ\0,1\, after the model-specific embedding normalization used by the VLM wrapper. The positive-class score is si=expâĄ(aiâ1)expâĄ(aiâ0)+expâĄ(aiâ1),s_i= (a_i1) (a_i0)+ (a_i1), and y^i=argâĄmaxcâĄaiâc y_i= _ca_ic. E.3 MLLMs MLLMs are evaluated with task-specific prompts that require a binary answer. For an image xix_i and prompt qiq_i, the model defines a next-token distribution over the vocabulary after the chat template has been applied: pΞâ(vâŁxi,qi)=softmaxâĄ(zi)v,p_Ξ(v x_i,q_i)=softmax(z_i)_v, where ziz_i are the logits for the first generated token. We extract the probabilities assigned to affirmative and negative answer tokens from this first-token distribution. To reduce tokenizer-format dependence, each answer word is evaluated using single-token variants with no prefix, a leading space, and a leading newline: =yes, Yes, ,Yes, yes, ,Y=\ yes, Yes, nYes, Yes, yes, nyes\, =no, No, ,No, no, .N=\ no, No, nNo, No, no, nno\. Only variants that tokenize to a single token are used. The affirmative and negative first-token probabilities are pi+=maxvââĄpΞâ(vâŁxi,qi),piâ=maxvââĄpΞâ(vâŁxi,qi).p_i^+= _v p_Ξ(v x_i,q_i), p_i^-= _v p_Ξ(v x_i,q_i). The positive-class score used for AUROC, AUPRC, and calibration is si=pi+.s_i=p_i^+. The hard predicted label is parsed from the generated text by taking the first standalone yes/no answer: y^i=1if the parsed answer is yes,0if the parsed answer is no. y_i= cases1&if the parsed answer is yes,\\ 0&if the parsed answer is no. cases Rows without a valid parsed yes/no label or without a valid first-token affirmative score are excluded from metric calculation. MLLM prompts. For binary diabetic retinopathy: Based on the fundus image, does this eye show diabetic retinopathy? Respond with yes if any level of diabetic retinopathy is present, or no otherwise. Respond only with âyesâ or ânoâ without additional commentary. For referable diabetic retinopathy: Based on the fundus image, does this eye show referable diabetic retinopathy? Respond with yes if the image should be referred for diabetic retinopathy evaluation or treatment, or no otherwise. Respond only with âyesâ or ânoâ without additional commentary. For glaucoma: Based on the fundus image, does this eye show glaucoma or glaucomatous optic disc changes? Respond with yes if glaucoma or suspicious glaucomatous changes are present, or no otherwise. Respond only with âyesâ or ânoâ without additional commentary. E.4 Fine-tuned MLLMs Fine-tuned MLLMs use the same inference pipeline as base MLLMs, with an SFT LoRA adapter loaded into the base model before generation. The same parser and scoring pipeline is used for base and fine-tuned models so that deltas reflect adaptation rather than evaluation changes. Appendix F Language Model Fine-tuning Details Training grid. The current FT grid trains the six configured MLLMs on BRSET and mBRSET for any diabetic retinopathy, referable diabetic retinopathy, and glaucoma. The evaluated MLLMs are Gemma-3-27B, MedGemma-4B, MedGemma-1.5-4B, LLaVA-NeXT-8B, Qwen3-VL-8B, and MedGemma-27B. Adapter configuration. SFT uses LoRA adapters targeting all linear modules with rank r=16r=16, no bias terms, and causal language modeling as the PEFT task type. The implementation uses LoRA dropout 0.05, bf16 training, gradient checkpointing, and the configured 16-bit quantization mode. Objective. Supervised fine-tuning optimizes the assistant answer tokens with cross-entropy: âSFT=ââjâilogpΞ(aiâjâŁxi,qi,ai,<j),L_SFT=- _j _i p_Ξ(a_ij x_i,q_i,a_i,<j), where xix_i is the fundus image, qiq_i is the task prompt, and iA_i indexes the assistant answer tokens. The assistant target is the binary answer text. The SFT configuration uses one epoch, learning rate 2Ă10â52Ă 10^-5, LoRA alpha 16, batch size 1 with gradient accumulation 8 for 27B-class models, and batch size 4 with gradient accumulation 4 for smaller models. Appendix G Metric Definitions Let D=(yi,y^i,si)i=1nD=\(y_i, y_i,s_i)\_i=1^n, where yiâ0,1y_iâ\0,1\ is the ground-truth label, y^iâ0,1 y_iâ\0,1\ is the predicted label, and siâ[0,1]s_iâ[0,1] is the positive-class score. Metrics are set to missing when the required labels, predictions, scores, or class diversity are unavailable. Accuracy. Accuracy=1nâi=1n[y^i=yi].Accuracy= 1n _i=1^n1[ y_i=y_i]. Precision, recall, and F1. With TâPTP, FâPFP, and FâNFN computed for the positive class, Precision=TâPTâP+FâP,Recall=TâPTâP+FâN,Precision= TPTP+FP, = TPTP+FN, F1=2âPrecisionâĄRecallPrecision+Recall.F1= 2\,Precision\,RecallPrecision+Recall. AUROC. Let P=i:yi=1P=\i:y_i=1\ and N=j:yj=0N=\j:y_j=0\. AUROC is the probability that a positive example receives a higher score than a negative example, with ties receiving half credit: AUROC=1|P|â|N|âiâPâjâN([si>sj]+12[si=sj]).AUROC= 1|P||N| _iâ P _jâ N (1[s_i>s_j]+ 121[s_i=s_j] ). AUPRC. The implementation uses scikit-learn average precision. Sorting examples by descending sis_i, let PkP_k and RkR_k be precision and recall after the k-th threshold. Then AUPRC=âk(RkâRkâ1)âPk.AUPRC= _k(R_k-R_k-1)P_k. Expected calibration error. The implemented ECE is positive-class probability calibration, not correctness-confidence calibration. Divide [0,1][0,1] into M=10M=10 equal-width bins BmB_m according to sis_i. For each non-empty bin, scoreâĄ(Bm)=1|Bm|ââiâBmsi,posâĄ(Bm)=1|Bm|ââiâBmyi.score(B_m)= 1|B_m| _iâ B_ms_i, (B_m)= 1|B_m| _iâ B_my_i. ECE is ECE=âm=1M|Bm|nâ|scoreâĄ(Bm)âposâĄ(Bm)|.ECE= _m=1^M |B_m|n |score(B_m)-pos(B_m) |. Lower ECE is better. Appendix H Fairness Metrics For a binary subgroup attribute aiâ0,1a_iâ\0,1\, let Gg=i:ai=gG_g=\i:a_i=g\. Gaps are reported as max-minus-min across the two groups, so lower is better. These are descriptive subgroup diagnostics and should not be interpreted as causal fairness estimates. Demographic parity gap. ÎDP=|PrâĄ(Y^=1âŁG1)âPrâĄ(Y^=1âŁG0)|. _DP= | ( Y=1 G_1)- ( Y=1 G_0) |. Accuracy gap. ÎAcc=|AccuracyâĄ(G1)âAccuracyâĄ(G0)|. _Acc= |Accuracy(G_1)-Accuracy(G_0) |. Equal opportunity gap. With TPRâĄ(G)=PrâĄ(Y^=1âŁY=1,G)TPR(G)= ( Y=1 Y=1,G), ÎTPR=|TPRâĄ(G1)âTPRâĄ(G0)|. _TPR= |TPR(G_1)-TPR(G_0) |. False-positive-rate gap. With FPRâĄ(G)=PrâĄ(Y^=1âŁY=0,G)FPR(G)= ( Y=1 Y=0,G), ÎFPR=|FPRâĄ(G1)âFPRâĄ(G0)|. _FPR= |FPR(G_1)-FPR(G_0) |. Equalized odds gap. The implemented equalized odds summary is the maximum of the true-positive-rate and false-positive-rate gaps: ÎEO=maxâĄ(ÎTPR,ÎFPR). _EO= ( _TPR, _FPR). It is not the average of TPR and FPR gaps. AUROC gap. ÎAUROC=|AUROCâĄ(G1)âAUROCâĄ(G0)|, _AUROC= |AUROC(G_1)-AUROC(G_0) |, when both subgroup AUROCs are defined. Appendix I Subgroup Construction Age. Age is binarized within the analyzed result rows using the median valid age as the threshold: ai=[agei>median(age)].a_i=1[age_i>median(age)]. This avoids selecting a very young cutoff that would produce tiny minority groups in older retinal cohorts. Sex. Sex is normalized to male/female using the dataset-specific metadata column when available. Rows with unrecognized or missing sex values are excluded from sex subgroup calculations. Image quality. Robustness uses binary image-quality strata. For BRSET, the provided image-quality label is binarized into adequate versus inadequate quality. For mBRSET, the provided final quality flag is binarized into accepted versus not accepted quality. Minimum group sizes. The default minimum subgroup size is 10. Image-quality robustness uses the configured minimum of 5. If fewer than two groups remain or the smaller group is below the threshold, the metric is skipped in the summary tables. Appendix J Robustness Metrics Robustness is computed overall for each model, dataset, and task, rather than inside every fairness subgroup. This avoids creating many small partitions. For a binary image-quality attribute Qâ0,1Qâ\0,1\ and a metric M, ÎQ,M=|MâĄ(Q=1)âMâĄ(Q=0)|. _Q,M= |M(Q=1)-M(Q=0) |. The implementation reports image-quality gaps for accuracy, AUROC, AUPRC, F1, and ECE when both quality groups are available. Appendix K Fine-tuning Delta Metrics For a fine-tuned adapter A, test dataset d, and task t, the FT analysis joins the adapter metric with the base MLLM metric for the same model, dataset, and task. For any metric M, ÎâMâ(A,d,t)=MFTâ(A,d,t)âMbaseâ(modelâĄ(A),d,t). M(A,d,t)=M_FT(A,d,t)-M_base(model(A),d,t). Positive deltas are better for AUROC, AUPRC, accuracy, and F1. Negative deltas are better for ECE. The analysis also records OOD(A,d)=[train(A)â d].OOD(A,d)=1[train(A)â d]. K.1 Uncertainty for fine-tuning deltas The FT significance analysis uses paired predictions rather than only aggregate summaries. For each adapterâtest comparison c, examples are aligned by image identifier between the fine-tuned adapter and the corresponding base MLLM: c=(yi,y^ibase,sibase,y^iFT,siFT)i=1nc,D_c=\(y_i, y^base_i,s^base_i, y^FT_i,s^FT_i)\_i=1^n_c, where sis_i is the positive-class score. The observed paired delta is ÎŽ^c,M=MâĄ(yi,y^iFT,siFTi=1nc)âMâĄ(yi,y^ibase,sibasei=1nc). ÎŽ_c,M=M(\y_i, y^FT_i,s^FT_i\_i=1^n_c)-M(\y_i, y^base_i,s^base_i\_i=1^n_c). For per-comparison uncertainty, bootstrap sample b draws ncn_c indices with replacement from 1,âŠ,nc\1,âŠ,n_c\ and recomputes ÎŽ^c,M(b) ÎŽ^(b)_c,M. The 95% interval is the empirical [2.5,97.5][2.5,97.5] percentile interval. The two-sided bootstrap p-value is pboot=2âminâĄ(1+âb[ÎŽ^(b)c,Mâ€0]B+1,1+âb[ÎŽ^(b)c,Mâ„0]B+1),p_boot=2 ( 1+ _b1[ ÎŽ^(b)_c,M†0]B+1, 1+ _b1[ ÎŽ^(b)_c,Mâ„ 0]B+1 ), clipped at 1. For the domain-level summaries in the main paper, the analysis bootstraps the mean of the paired deltas across SFT adapterâtest comparisons within each test domain and metric. Reported q-values apply BenjaminiâHochberg correction within each metric family. Appendix L Current Summary Tables L.1 AUROC by task and model type Model type Binary DR Glaucoma Referable DR General Vision Encoder Model (VM) 0.867 0.742 0.850 Ophthalmic Vision Encoder Model (VM) 0.690 0.591 0.697 General Multimodal LLM (MLLM) 0.685 0.630 0.651 Medical Multimodal LLM (MLLM) 0.868 0.656 0.844 General Dual Encoder Vision Language Model (VLM-encoders) 0.706 0.616 0.724 Ophthalmic Dual Encoder Vision Language Model (VLM-encoders) 0.756 0.630 0.721 Table 9: Mean AUROC by task and model type in the current benchmark result bundle. L.2 Language model fine-tuning summary Domain AUROC AUPRC Accuracy F1 ECE In-domain 0.776 0.619 0.826 0.511 0.164 OOD 0.724 0.513 0.675 0.402 0.290 Table 10: Mean SFT metrics by test domain. The corresponding delta estimates are summarized in the main paper. Domain Metric Mean delta 95% CI low 95% CI high q In-domain AUROC 0.004 -0.002 0.010 0.521 In-domain Accuracy 0.033 0.003 0.076 0.004 In-domain ECE -0.040 -0.085 -0.008 0.004 OOD AUROC 0.002 -0.002 0.006 0.521 OOD Accuracy 0.012 0.005 0.021 0.005 OOD ECE -0.018 -0.026 -0.009 0.004 Table 11: Bootstrap uncertainty for domain-level SFT deltas. Positive ECE deltas indicate worse calibration, while negative ECE deltas indicate improved calibration. Intervals are estimated by paired resampling over matched base and fine-tuned predictions. L.3 Dataset and model coverage Figure 7: Fine-grained model performance. Task-specific model rankings expose differences that are hidden by family averages. This larger version is kept in the supplement because it is useful for inspection but too dense for the main narrative. Figure 8: Task-specific leaderboards. Top AUROC configurations differ by task, reinforcing that a benchmark for fundus foundation models should expose disease-specific strengths rather than collapse all results into a single score. Figure 9: Model-by-dataset AUROC heatmap. Rows are model/method configurations and columns are datasets. This plot is intended to reveal dataset-specific failures that are hidden by averaging over all datasets. Figure 10: Dataset-level family metrics. AUROC, AUPRC, accuracy, and ECE are shown by dataset and model family. Datasets are categorical sources rather than a time series, so bars are used instead of line plots. L.4 Calibration diagnostics Figure 11: Calibration by model type. Mean ECE is summarized by dataset and task for each model type. Empty cells indicate unavailable dataset-task combinations or missing valid scores. Figure 12: VM calibration grid. Mean ECE for VM encoder configurations, organized by dataset and task. Figure 13: VLM calibration grid. Mean ECE for VLM configurations, including both zero-shot and linear-probing results where available. Figure 14: MLLM calibration grid. Mean ECE for prompted MLLM configurations by dataset and task. L.5 Calibration curves The following sheets restore the raw calibration-curve diagnostics. Each sheet fixes one model type and lays out dataset/task panels; lines show the top configurations within that model type. These figures are intentionally supplementary because they are large and are meant for error analysis rather than quick comparison. Figure 15: Calibration curves for general VM encoders. Each panel is a dataset/task pair; the diagonal indicates perfect calibration. Figure 16: Calibration curves for ophthalmic VM encoders. Each panel is a dataset/task pair; the diagonal indicates perfect calibration. Figure 17: Calibration curves for general VLMs. Each panel is a dataset/task pair; the diagonal indicates perfect calibration. Figure 18: Calibration curves for ophthalmic VLMs. Each panel is a dataset/task pair; the diagonal indicates perfect calibration. Figure 19: Calibration curves for general MLLMs. Each panel is a dataset/task pair; the diagonal indicates perfect calibration. Figure 20: Calibration curves for medical MLLMs. Each panel is a dataset/task pair; the diagonal indicates perfect calibration. L.6 Family-specific and adaptation diagnostics Figure 21: VM multimetric spider plot. Top VM configurations are compared across ranking, thresholded performance, calibration, and available robustness/fairness diagnostics. Figure 22: VLM multimetric spider plot. Top VLM configurations show how linear probing changes not only AUROC but also calibration and thresholded behavior. Figure 23: MLLM multimetric spider plot. Top MLLM configurations differ not only in AUROC but also in AUPRC, thresholded accuracy, calibration, and available robustness/fairness diagnostics. Figure 24: VLM zero-shot versus linear-probing comparison. Linear probing consistently changes the operating point relative to zero-shot image-text matching, motivating method-aware reporting. Figure 25: Language-model fine-tuning deltas by training task. SFT effects are not uniform across binary DR, referable DR, and glaucoma. Appendix M Task-Level Comparisons Figure 26: Task-level model-type comparison. Mean AUROC by disease task and model type. Splitting the comparison by task avoids hiding glaucoma-specific weaknesses inside a pooled average. Appendix N Leaderboards Figure 27: Model-type-stratified leaderboards. Each panel lists the leading model/method configurations within one model type, with separate bars for binary DR, referable DR, and glaucoma. This replaces a single pooled leaderboard so task and model-type effects remain visually distinct. Appendix O Interface and Adaptation Figure 28: Interface and adaptation summary. Left: mean AUROC by model type. Right: VLM linear probing and zero-shot performance summarized in one task-stratified view. Appendix P Reliability Matrix Figure 29: Reliability matrix by model type. Higher values are better. Fairness and quality robustness summarize available metadata-backed gaps, so these cells should be read as coverage-aware diagnostics rather than universal claims for every dataset. Appendix Q Code and Result Artifacts The released repository contains modular dataset loaders, task-label harmonization rules, model-interface wrappers, evaluation routines, and figure-generation utilities. The benchmark artifacts include machine-readable summaries for aggregate performance, calibration, subgroup fairness, image-quality robustness, coverage, and language-model fine-tuning analyses. These artifacts are designed to support both the manuscript figures and the public arena dashboard. The dataset and model modules are organized as registries rather than one-off analysis code. New datasets can be added by specifying metadata fields, label mappings, supported tasks, and split definitions; new models can be added by implementing the relevant VM, VLM, or MLLM interface. This design keeps the benchmark extensible as additional fundus datasets, acquisition variables, and model families become available. Appendix R Dashboard Organization The dashboard is organized around the same analysis dimensions used in the benchmark: arena, performance, calibration, fairness, robustness, model types, tasks, coverage, and leaderboard tables. Fine-tuned SFT adapter analyses are presented as a separate adaptation view so that base benchmark results are not mixed with fine-tuning experiments. Appendix S Ethics and Limitations FOCUS is retrospective and should not be interpreted as clinical validation. Labels may be noisy or non-equivalent across sources. Demographic and acquisition metadata are incomplete. Subgroup gaps are descriptive and may be unstable for small strata. Dataset licenses may restrict image redistribution. Public dashboards should therefore release metrics, metadata, manifests, and code only where permitted.