Paper deep dive
DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation
Jiaxuan Li, Qing Xu, Xiangjian He, Yue Li, Daokun Zhang, Fiseha B. Tesema, Rong Qu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/8/2026, 3:39:43 AM
Summary
The paper introduces DistMedVL, a probabilistic vision-language framework for medical image segmentation that addresses aleatoric and epistemic uncertainty. It utilizes a Probabilistic Cross-Modal Adapter (PCM-Adapter) containing a Mahalanobis Alignment Module (MAM) and a Distribution Flow Module (DFM) to perform variance-conditioned matching and vision-guided refinement of textual distributions, outperforming state-of-the-art methods on eight benchmarks with only 6.3M trainable parameters.
Entities (9)
Relation Signals (8)
DistMedVL → contains → PCM-Adapter
confidence 95% · DistMedVL ... introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter)
PCM-Adapter → contains → MAM
confidence 95% · The PCM-Adapter comprises two sequential modules ... Mahalanobis Alignment Module (MAM)
PCM-Adapter → contains → DFM
confidence 95% · The PCM-Adapter comprises two sequential modules ... Distribution Flow Module (DFM)
MAM → models → Gaussian distributions
confidence 95% · MAM that models textual tokens as Gaussian distributions
MAM → uses → Mahalanobis distance
confidence 95% · MAM ... computes patch-text compatibility via Mahalanobis distance
DistMedVL → addresses → Aleatoric uncertainty
confidence 90% · jointly mitigate aleatoric and epistemic uncertainty
DistMedVL → addresses → Epistemic Uncertainty
confidence 90% · jointly mitigate aleatoric and epistemic uncertainty
DistMedVL → buildson → CLIPSeg
confidence 90% · inputs are processed by the frozen vision and text encoders of CLIPSeg
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding, yet remains hindered by uncertainty in both modalities under real-world clinical conditions. Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries and epistemic uncertainty from limited training data, leading to fragile performance under domain shift. To address this issue, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty. Specifically, the PCM-Adapter comprises two sequential modules for progressive probabilistic alignment. We first devise a Mahalanobis Alignment Module (MAM) that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, yielding variance-conditioned matching that downweights unreliable feature dimensions. Moreover, we devise a Distribution Flow Module (DFM) that estimates modality-wise confidence parameters and performs vision-guided refinement of textual distributions, accommodating distributional variation across imaging modalities. Extensive experiments across eight medical segmentation benchmarks demonstrate that DistMedVL outperforms state-of-the-art methods with only 6.3M trainable parameters, exhibiting superior data efficiency, perturbation robustness and cross-dataset generalization.
Tags
Links
- Source: https://arxiv.org/abs/2608.05683v1
- Canonical: https://arxiv.org/abs/2608.05683v1
Trouble viewing inline? Open PDF directly →
Full Text
57,425 characters extracted from source content.
Expand or collapse full text
DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation Jiaxuan Li, Qing Xu, Xiangjian He, Yue Li, Daokun Zhang, Fiseha B. Tesema, Rong Qu, This work is partially supported by the Yongjiang Technology Innovation Project (2022A-097-G), Zhejiang Department of Transportation General Research and Development Project (2024039), and National Natural Science Foundation of China grant (UNNC: B0166). (Equal contribution: Jiaxuan Li and Qing Xu, Corresponding authors: Xiangjian He) J. Li and Q. Xu are with School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang, China, and with School of Computer Science, University of Nottingham, UK (e-mail: jiaxuan.li@nottingham.edu.cn). X. He, Y. Li, F. B. Tesem, D. Zhang are with School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang, China (e-mail: sean.he@nottingham.edu.cn). R. Qu is with School of Computer Science, University of Nottingham, UK (e-mail: rong.qu@nottingham.ac.uk). Abstract Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding, yet remains hindered by uncertainty in both modalities under real-world clinical conditions. Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries and epistemic uncertainty from limited training data, leading to fragile performance under domain shift. To address this issue, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty. Specifically, the PCM-Adapter comprises two sequential modules for progressive probabilistic alignment. We first devise a Mahalanobis Alignment Module (MAM) that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, yielding variance-conditioned matching that downweights unreliable feature dimensions. Moreover, we devise a Distribution Flow Module (DFM) that estimates modality-wise confidence parameters and performs vision-guided refinement of textual distributions, accommodating distributional variation across imaging modalities. Extensive experiments across eight medical segmentation benchmarks demonstrate that DistMedVL outperforms state-of-the-art methods with only 6.3M trainable parameters, exhibiting superior data efficiency, perturbation robustness and cross-dataset generalization. †publicationid: pubid: I Introduction Medical image segmentation is fundamental to clinical applications such as computer-aided diagnosis, treatment planning and surgical navigation [27, 2, 34]. In clinical practice, medical images are inherently accompanied by rich textual information, including radiology reports, diagnostic descriptions and clinical notes, which provide complementary semantic context about anatomical structures and pathological conditions. This naturally multimodal clinical workflow motivates cross-modal learning paradigms that jointly leverage visual and textual modalities to improve the accuracy and generalizability of medical image segmentation [40]. Such paradigms are particularly beneficial for medical segmentation, where textual descriptions of lesion characteristics, locations, and pathological findings can provide essential semantic guidance for delineating structures that are visually ambiguous across different imaging modalities. Early medical image segmentation methods primarily focused on unimodal visual learning. CNN-based architectures [33, 42] established foundational feature extraction capabilities through hierarchical local pattern learning, while Transformer-based approaches [6, 5] introduced global self-attention mechanisms for capturing long-range dependencies. Hybrid architectures [7, 16] further combined the strengths of both paradigms, achieving enhanced pixel-level prediction across various imaging modalities. Despite their success, these unimodal methods are inherently confined to visual feature representations, leaving the potential of language-based guidance for more flexible segmentation unexplored. To exploit the complementary textual information available in clinical settings, vision-language models (VLMs), pre-trained on large-scale image-text pairs to learn aligned cross-modal representations [31], have been increasingly adopted for medical image segmentation through strategies such as lightweight adapter-based transfer [11], text-guided feature fusion [26, 45], and cross-modal attention mechanisms [32]. Figure 1: (a) Overview of the DistMedVL architecture, where only the PCM adapter and decoder are trainable. (b) Detail of the PCM Adapter, which maps cross-modal features to probabilistic embeddings. While these vision-language approaches achieve promising performance, their deterministic cross-modal alignment paradigm remains inherently fragile under domain shift and data-scarce scenarios. In real-world clinical settings, image quality is frequently degraded by device heterogeneity and acquisition noise [41], while paired textual inputs often contain semantic ambiguity or erroneous annotations [17]. Furthermore, the scarcity of large-scale annotated medical datasets [37] amplifies model uncertainty in underrepresented regions and rare pathologies. Under such conditions, deterministic alignment treats visual and textual representations as fixed-point embeddings without assessing per-dimension feature reliability [44], overlooking both the aleatoric uncertainty inherent in noisy inputs and the epistemic uncertainty arising from limited training data. These limitations motivate the need for probabilistic modeling in cross-modal alignment. To address this issue, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen vision-language encoders to explicitly model representational uncertainty. The PCM-Adapter comprises two sequential modules for progressive cross-modal probabilistic alignment. The Mahalanobis Alignment Module (MAM) models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, where bilateral variances estimated from both modalities modulate the matching scores to naturally downweight unreliable feature dimensions, enabling anisotropic alignment that accounts for both aleatoric and epistemic uncertainty. The Distribution Flow Module (DFM) estimates modality-wise confidence parameters and performs vision-guided refinement of textual distributions through a reliability-gated mechanism, enabling text representations to adaptively adjust based on visual evidence rather than serving as a fixed semantic prior. Together, the two modules form a progressive alignment pipeline that jointly addresses the uncertainty of both modalities. We comprehensively evaluate DistMedVL from both epistemic and aleatoric uncertainty perspectives, including data efficiency under limited supervision, cross-domain generalization under distribution shift, and perturbation robustness under degraded input quality. Extensive experiments on eight medical segmentation benchmarks demonstrate that DistMedVL consistently outperforms state-of-the-art methods with superior robustness under varying uncertainty conditions. The main contributions of this work are summarized as follows. • We propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight PCM-Adapter into frozen vision-language encoders to jointly mitigate aleatoric and epistemic uncertainty in cross-modal medical image segmentation. • We devise a Mahalanobis Alignment Module (MAM) that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, replacing deterministic matching with variance-aware distribution-level alignment. • We devise a Distribution Flow Module (DFM) that estimates modality-wise confidence parameters and employs a reliability gate to suppress high-variance visual features, enabling adaptive vision-guided refinement of textual distributions. • Extensive experiments on eight medical segmentation benchmarks demonstrate that DistMedVL achieves state-of-the-art performance with only 6.3M trainable parameters under varying levels of uncertainty. I Related Work Figure 2: Overall framework of the proposed method. (a) Mahalanobis Distance Module (MDM), which maps image features to text features. (b) Distribution Flow Module (DFM), which maps text features to global image semantics. I-A Unimodal Medical Image Segmentation Medical image segmentation has advanced significantly through deep learning architectures [4]. CNN-based methods centered on the U-Net framework [33] established foundational encoder-decoder designs, with subsequent works such as FatNet [42] and DCSAU-Net [43] enhancing multi-scale feature extraction and target awareness. To address the limited receptive field of convolutions, Transformer-based approaches such as Swin-UNet [6] introduced self-attention mechanisms for global context modeling, while hybrid CNN-Transformer architectures including TransUNet [7], H2Former [16] and CFFormer [25] combined local and global feature extraction for enhanced pixel-level prediction. Despite their strong performance across diverse imaging modalities, these methods produce deterministic representations that lack the capacity to model uncertainty, tending to yield over-confident predictions in data-scarce scenarios and out-of-distribution regions. I-B Vision-Language Models for Medical Image Segmentation Vision-language models (VLMs) such as CLIP [31] learn aligned cross-modal representations from large-scale image-text pairs, demonstrating strong transferability across downstream tasks. This paradigm has been extended to the medical domain through models such as PubMedCLIP [12] and UniMedCLIP [22], which leverage domain-specific image-text pairs for medical representation learning. For medical image segmentation, a prevalent strategy introduces lightweight adapters upon frozen VLM encoders to achieve cross-modal segmentation at minimal computational cost. VLSM-Adapter [11] inserts adapter modules into both modality encoders but operates each modality independently without explicit cross-modal interaction. CausalCLIPSeg [8] employs causal intervention on visual features to suppress confounding factors, yet operates solely on visual representations and requires end-to-end fine-tuning with substantial computational overhead. MedCLIPSeg [23] introduces probabilistic cross-modal alignment, yet its alignment metric reduces uncertainty to a scalar correction term via a variance-penalized inner product, rather than modeling the full distributional structure between modalities. These methods uniformly rely on deterministic or shallow probabilistic alignment, leaving the heterogeneous distributional gap between visual and textual representations insufficiently addressed. I-C Uncertainty Estimation and Probabilistic Representations Uncertainty estimation in deep learning encompasses two complementary types: aleatoric uncertainty originating from inherent data noise that is irreducible regardless of training data volume, and epistemic uncertainty arising from limited model knowledge that can be progressively reduced with additional data [21]. Classical approaches such as MC Dropout [13] and Deep Ensemble [24] estimate predictive uncertainty through post-hoc model-level inference via multiple stochastic forward passes or multiple independently trained models, but lack explicit modeling of uncertainty at the feature representation level. Recent work proposes density-based methods to disentangle both uncertainty types within a single forward pass [30], offering a more principled approach to feature-level uncertainty estimation. In cross-modal settings, images and text inherently exhibit one-to-many correspondences that deterministic point-to-point alignment cannot capture [9]. PCME [9] addresses this by modeling features as Gaussian distributions for probabilistic cross-modal matching, while ProbVLM [39] estimates distributions over frozen VLM embeddings in a post-hoc manner. However, these probabilistic global matching methods remain insufficient for dense prediction tasks under distribution shift, and how to explicitly incorporate uncertainty into cross-modal feature alignment remains an open challenge. I Methodology I-A Overview of DistMedVL We present the DistMedVL framework in Fig. 1(a). Given a medical image and its paired textual description, both inputs are processed by the frozen vision and text encoders of CLIPSeg [29]. To enable cross-modal probabilistic alignment at multiple semantic levels, we insert lightweight PCM-Adapters at layers 2,4,6,8\2,4,6,8\ of both encoders. As illustrated in Fig. 1(b), each PCM-Adapter comprises a compression layer, two sequential modules that are the Mahalanobis Alignment Module (MAM) for text-to-vision probabilistic matching and the Distribution Flow Module (DFM) for vision-guided textual distribution refinement, and an expansion layer. The updated visual features are fed into a fine-tuned CLIPSeg decoder for segmentation prediction, while both modality features are additionally projected into a shared embedding space for contrastive alignment. I-B Probabilistic Cross-Modal Adapter Existing vision-language adapters [11] typically insert independent modules into each modality encoder without explicit cross-modal interaction, leaving the alignment to be implicitly learned from downstream supervision alone. To achieve explicit probabilistic alignment while preserving the pre-trained encoder representations, we devise the PCM-Adapter as a bottleneck-style module that operates in a compressed feature space. At each selected layer ℓ , the intermediate visual features v(ℓ)∈ℝN×CF_v^( ) ^N× C and textual features t(ℓ)∈ℝL×KF_t^( ) ^L× K are first compressed and then restored via linear projections: v(ℓ)↓ _v^( ) =v(ℓ)v↓,~v(ℓ)=^v(ℓ)v↑, =F_v^( )W_v , F_v^( )= F_v^( )W_v , (1) t(ℓ)↓ _t^( ) =t(ℓ)t↓,~t(ℓ)=^t(ℓ)t↑, =F_t^( )W_t , F_t^( )= F_t^( )W_t , where v↓∈ℝC×MW_v ^C× M, t↓∈ℝK×MW_t ^K× M are compression projections that map both modalities into a shared M-dimensional space, and v↑∈ℝM×CW_v ^M× C, t↑∈ℝM×KW_t ^M× K are expansion projections that restore the original dimensionality. This bottleneck design not only reduces computational overhead but also filters out redundant feature dimensions before probabilistic alignment. Within this compressed space, the MAM and DFM are applied sequentially: ^v(ℓ) F_v^( ) =MAM(ℓ)(v(ℓ)↓,t(ℓ)↓), =MAM^( )\! (F_v^( ) ,\;F_t^( ) ), (2) ^t(ℓ) F_t^( ) =DFM(ℓ)(^v(ℓ),t(ℓ)↓), =DFM^( )\! ( F_v^( ),\;F_t^( ) ), where the output of the MAM, ^v(ℓ) F_v^( ), is directly used as the visual input to the DFM at the same layer, as illustrated in Fig. 1(b). This sequential coupling is motivated by the observation that DFM benefits from receiving uncertainty-refined visual features rather than raw encoder outputs, enabling cascaded cross-modal refinement. The restored features ~v(ℓ) F_v^( ) and ~t(ℓ) F_t^( ) subsequently replace the original features as inputs to the next transformer layer. In this way, the PCM-Adapter realizes a bottleneck-style probabilistic alignment that enriches both modalities with cross-modal uncertainty-aware information while seamlessly preserving the frozen encoder representations. I-C Mahalanobis Alignment Module Deterministic cross-modal matching methods, such as cosine similarity, treat all feature dimensions equally and produce fixed-point alignment, making them susceptible to noisy or ambiguous dimensions that are prevalent in clinical imaging data. Unlike cosine similarity, which only measures angular proximity in the feature space, the Mahalanobis distance incorporates the covariance structure of the underlying distributions, allowing the matching criterion to adaptively scale each feature dimension according to its estimated reliability. To this end, we devise MAM that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, enabling variance-conditioned anisotropic alignment that inherently downweights unreliable feature dimensions, as illustrated in Fig. 2(a). Given the compressed features v(ℓ)↓F_v^( ) and t(ℓ)↓F_t^( ) , we first project both into a compact H-dimensional space via linear layers with GELU activation to obtain ∈ℝN×HV ^N× H and ∈ℝL×HT ^L× H. Within this space, we further apply separate semantic extractors followed by LayerNorm to obtain modality-specific representations v∈ℝN×DZ_v ^N× D and t∈ℝL×DZ_t ^L× D. The LayerNorm is applied along the feature dimension independently for each token, ensuring that subsequent variance estimation reflects genuine uncertainty rather than heterogeneous feature scales. The MAM then models the textual semantic representations as Gaussian distributions by estimating per-token means and variances: t μ_t =Wμt∈ℝL×D, =W_μZ_t ^L× D, (3) t2 σ_t^2 =softplus(Wσt)∈ℝL×D, =softplus(W_σZ_t) ^L× D, where Wμ,Wσ∈ℝD×DW_μ,W_σ ^D× D are learnable transformations applied to each token independently, and softplus(⋅)softplus(·) ensures strictly positive variances. Unlike textual tokens, which carry relatively sparse semantic cues, visual patch embeddings encode substantially denser spatial information. Directly estimating per-patch variance would yield noisy and unreliable estimates due to the redundancy and ambiguity inherent in dense visual features. We instead aggregate all patch embeddings into a global visual representation ¯v=1N∑n=1Nvn∈ℝD Z_v= 1N _n=1^NZ_v^n ^D and estimate an image-level variance v2=softplus(Wσv¯v)∈ℝD σ_v^2=softplus(W_ _v Z_v) ^D, where Wσv∈ℝD×DW_ _v ^D× D maps the global representation into a per-dimension visual uncertainty estimate. The bilateral variance is then obtained by fusing both sources: 2=t2+sigmoid(λ)⊙v2+ϵ, σ^2= σ_t^2+sigmoid( θ_λ) σ_v^2+ε, (4) where λ∈ℝD θ_λ ^D is a learnable parameter that controls how much image-level uncertainty is incorporated into each dimension, and sigmoid(⋅)sigmoid(·) constrains the visual variance contribution to (0,1)(0,1). Since v2∈ℝD σ_v^2 ^D is a single image-level estimate while t2∈ℝL×D σ_t^2 ^L× D is per-token, v2 σ_v^2 is broadcast across all L rows, allowing each token to share a common image-level uncertainty prior while retaining its own token-specific textual variance. ϵε is a small constant for numerical stability. Since the bilateral variance provides per-dimension estimates, the covariance of each textual Gaussian is naturally diagonal, eliminating the need for full matrix inversion. The squared Mahalanobis distance between the n-th visual patch and the l-th textual Gaussian thus reduces to: dM2(vn,tl)=1D∑d=1D(Zv,dn−μt,dl)2σd2,l,d_M^2(Z_v^n, μ_t^l)= 1 D _d=1^D (Z_v,d^n- _t,d^l)^2 _d^2,l, (5) where the 1D 1 D factor normalizes the distance to prevent magnitude scaling with dimensionality. Dimensions with larger variance contribute less to the overall distance, effectively suppressing unreliable feature dimensions in the alignment. By replacing point-wise matching with distribution-level alignment, the MAM inherently accounts for aleatoric uncertainty through per-dimension variance weighting and mitigates the adverse effect of epistemic uncertainty, since underrepresented features naturally exhibit higher variance and are thus automatically downweighted. Computing Eq. 5 for all patch-token pairs yields the distance matrix M∈ℝN×LD_M ^N× L, which is converted into a probabilistic compatibility matrix via temperature-scaled softmax: =softmax(−M/τ)∈ℝN×L,A=softmax (-D_M/τ ) ^N× L, (6) where τ is a learnable temperature parameter constrained to [τmin,τmax][ _ , _ ] for training stability, and the negation ensures that smaller distances yield higher compatibility scores. The resulting A serves as a soft assignment matrix that captures the probabilistic affinity between visual patches and textual tokens. Based on this assignment, a dense semantic prior =∈ℝN×HS=AT ^N× H is generated for each visual patch through weighted aggregation over textual features. The semantic prior is then adaptively integrated into the visual representation via a gating mechanism: ^v(ℓ)=Wo(+g()⊙), F_v^( )=W_o\! (V+g(V) ), (7) where g(⋅)g(·) is a two-layer MLP with GELU activation and sigmoid output that produces a gate g()∈ℝN×Hg(V) ^N× H, ⊙ is element-wise multiplication, and Wo∈ℝH×MW_o ^H× M projects the fused representation back to the compressed dimensionality. The residual connection ensures that the original visual representation V is preserved as the base, while the element-wise gating g()⊙g(V) allows each patch to selectively absorb relevant textual semantics. When the textual prior is unreliable or irrelevant, the gate outputs approach zero, effectively bypassing the semantic prior and preserving the original visual features. In this way, the MAM replaces deterministic point-to-point matching with variance-aware distribution-level alignment, where unreliable feature dimensions are naturally downweighted through the bilateral uncertainty encoded in the Mahalanobis distance. I-D Distribution Flow Module Rather than applying MAM in reverse for the vision-to-text direction, we design a separate Distribution Flow Module to account for the granularity asymmetry between the two modalities, as illustrated in Fig. 2(b). In the text-to-vision direction, visual patches can be sparsely matched to a small set of semantically relevant text tokens, making fine-grained probabilistic alignment tractable. However, reversing this formulation would require each text token to attend over a large number of spatially dense patches, resulting in diffuse attention maps that fail to capture discriminative visual cues. The DFM therefore adopts a global semantic aggregation strategy that distills both modalities into compact distributional summaries. However, aggregating dense visual patches into a single global representation raises a concern: the reliability of this summary depends on whether the patches reach semantic consensus. The DFM therefore incorporates a variance-driven reliability mechanism that assesses inter-patch semantic consistency and adaptively modulates the visual contribution accordingly. Given the vision-enhanced features ^v(ℓ) F_v^( ) from the MAM and the original textual features t(ℓ)↓F_t^( ) , we project both into a compact K-dimensional semantic space via linear projections Wv,Wt∈ℝM×KW_v,W_t ^M× K to obtain vD_v and tD_t, which encourages more compact semantic aggregation. We then aggregate over visual patches to obtain a global semantic mean v,g μ_v,g and an inter-patch variance v,g2 σ^2_v,g that quantifies the degree of semantic disagreement among patches along each feature dimension. A reliability gate transforms this variance into per-dimension reliability scores: v,g μ_v,g =1N∑i=1Nv(i),v,g2=1N∑i=1N(v(i)−v,g)2, = 1N _i=1^ND_v^(i), σ^2_v,g= 1N _i=1^N (D_v^(i)- μ_v,g )^2, (8) γ =sigmoid((v,g2))∈ℝK, =sigmoid\! (G\! ( σ^2_v,g ) ) ^K, where v(i)∈ℝKD_v^(i) ^K denotes the projected embedding of the i-th visual patch, and (⋅)G(·) is a two-layer MLP with GELU activation. The global mean v,g μ_v,g captures the overall semantic content of the image, while the global variance v,g2 σ^2_v,g measures the inter-patch semantic consistency along each dimension. The gate is designed by default to high reliability at initialization, and learns during training to assign lower reliability scores to dimensions with higher inter-patch variance, adaptively suppressing uncertain visual signals before they propagate into the textual refinement. The semantic strength parameters for both modalities are then estimated as: v β_v =[softplus(Wβvv,g)+ϵ]⊙, = [softplus\! (W_ _v μ_v,g )+ε ] γ, (9) t β_t =softplus(Wβtt,g)+ϵ, =softplus\! (W_ _t μ_t,g )+ε, where t,g=1L∑j=1Lt(j)∈ℝK μ_t,g= 1L _j=1^LD_t^(j) ^K denotes the global textual mean, Wβv,Wβt∈ℝK×KW_ _v,W_ _t ^K× K are learnable projections, and softplus(⋅)softplus(·) ensures strictly positive values. The element-wise multiplication with γ suppresses v β_v in dimensions where visual patches are semantically inconsistent, preventing unreliable visual signals from corrupting the textual refinement. The refined textual semantic strength is obtained via a residual distribution flow: t∗=t+MLPf(v∥t), β_t^*= β_t+MLP_f\! ( β_v\| β_t ), (10) where ∥\| denotes concatenation and MLPf(⋅)MLP_f(·) is a two-layer MLP with GELU activation that learns a residual correction in the concentration parameter space. The residual formulation ensures that the visual distribution exerts a controlled adjustment rather than replacing the textual distribution entirely. The refined parameter is then projected back and broadcast across all L text tokens to produce the final vision-guided textual feature: ^t(ℓ)=t(ℓ)↓+MLPr(t∗), F_t^( )=F_t^( ) +MLP_r\! ( β_t^* ), (11) where MLPr(⋅)MLP_r(·) projects t∗ β_t^* back into the original feature space and the offset is broadcast uniformly across all L text tokens. This uniform broadcast reflects the global nature of the vision-to-text refinement that, rather than producing token-specific corrections, DFM applies a shared distributional shift, which adjusts all text tokens based on the aggregated visual evidence. In this way, DFM enables adaptive vision-guided refinement of textual distributions, where only reliable visual evidence with high inter-patch semantic consensus contributes to the cross-modal alignment, ensuring the robust distributional correction under varying input uncertainty. I-E The DistMedVL Architecture Designs DistMedVL builds upon CLIPSeg [29], a vision-language architecture pre-trained on natural image-text pairs. The framework adopts the ViT-B/16 vision encoder ℰvE_v and the Transformer-based text encoder ℰtE_t, both consisting of 12 transformer layers, as frozen feature extractors. Given a medical image vX_v and its paired textual description tX_t, the encoders produce visual features v∈ℝN×CF_v ^N× C and textual features t∈ℝL×KF_t ^L× K, where N and L denote the number of visual patches and text tokens, and C and K the corresponding feature dimensionalities. The PCM-Adapters are inserted at layers 2,4,6,8\2,4,6,8\ to perform cross-modal probabilistic alignment at multiple semantic levels, while only the adapter parameters and the CLIPSeg decoder are updated during training. The final features vF_v and tF_t are fed into the CLIPSeg decoder D, which is fine-tuned during training to generate segmentation predictions. To maintain global cross-modal alignment throughout training, we additionally project vF_v and tF_t into a shared embedding space via global average pooling followed by learnable linear projections for contrastive learning. The training objective combines a segmentation loss and a contrastive loss: ℒseg=12(ℒBCE+ℒDice),ℒDistMedVL=λ1⋅ℒseg+λ2⋅ℒcon, splitL_seg&= 12\! (L_BCE+L_Dice ),\\ L_ DistMedVL&= _1·L_seg+ _2·L_con, split (12) where ℒconL_con follows the CLIP contrastive formulation [31], and λ1=0.5 _1=0.5, λ2=0.1 _2=0.1 are loss weighting coefficients following MedCLIPSeg [23]. The entire framework introduces only 6.3M trainable parameters from the PCM-Adapters and decoder, while the frozen encoders contribute the remaining parameters. To rigorously evaluate the generalization capability of the proposed framework, no data augmentation is employed during training beyond image resizing, ensuring that performance gains are attributable to the model architecture rather than augmentation strategies. IV Experiments IV-A Datasets We evaluate DistMedVL across 8 benchmark datasets spanning four imaging modalities. Five datasets, which are TN3K [14], BUSI [1], ISIC-2016 [15], Kvasir-SEG [20] and Covid19 [10],are used for training and evaluation. Three additional datasets, which are BUID [3], CVC-ClinicDB [36] and PKTN [35], serve exclusively as out-of-distribution targets for domain generalization. The dataset splits are summarized in Table I. For BUSI, ISIC-2016, Kvasir-SEG, Covid19, BUID and CVC-ClinicDB, we adopt the dataset splits and text prompt annotations from MedCLIPSeg [23]. For TN3K and PKTN, text descriptions are generated using Qwen2.5 [19], guided by dataset descriptions and ground-truth masks. TABLE I: Dataset splits for data efficiency and domain generalization experiments. The top five datasets are used for training and evaluation under varying data ratios. The bottom three datasets serve exclusively as out-of-distribution targets for domain generalization, where all available samples are used for testing. Dataset Train 10% Train 25% Train 50% Train 100% Validation Test TN3K [14] 259 647 1295 2591 288 614 BUSI [1] 62 156 312 624 78 78 ISIC-2016 [15] 80 202 405 810 90 379 Kvasir-SEG [20] 80 200 400 800 100 100 Covid19 [10] 571 1429 2858 5716 1429 2113 BUID [3] - - - - - 232 CVC-ClinicDB [36] - - - - - 612 PKTN [35] - - - - - 1005 TABLE I: Data efficiency and epistemic uncertainty analysis. Models are trained with 10%, 25%, 50%, and 100% of the training data to evaluate segmentation performance under varying levels of epistemic uncertainty induced by limited supervision. Results are averaged across 5 datasets. Model 10% 25% 50% 100% Dice NSD Dice NSD Dice NSD Dice NSD Unimodal Approaches UNet [33] 62.40 66.81 70.26 74.61 75.64 80.04 80.38 84.65 DCSAU-Net [43] 60.06 64.72 67.46 71.79 75.87 80.30 80.05 84.61 FAT-Net [42] 73.95 78.42 80.09 84.41 82.19 86.39 84.51 88.70 Swin-Unet [6] 75.39 80.70 80.09 84.67 82.70 87.07 84.25 88.63 Trans-Unet [7] 77.11 81.93 78.78 83.47 82.24 86.82 83.42 87.90 CFFormer [25] 76.10 81.12 79.35 83.89 82.40 86.65 84.51 88.67 Crossmodal Approaches TGANet [38] 76.68 81.24 81.75 86.19 84.36 88.60 85.93 90.39 LViT [26] 68.16 72.63 77.72 82.22 81.24 85.68 84.09 88.39 CLIPSeg [28] 77.05 82.49 80.66 85.74 83.29 88.46 84.76 89.80 LanGuideMedSeg [46] 77.31 81.79 79.28 83.47 84.77 89.44 86.45 91.17 VLSM-Adapter [11] 79.65 84.79 82.25 87.38 83.42 86.72 84.91 90.20 RecLMIS [18] 71.64 76.27 77.96 82.36 82.42 86.85 85.69 90.01 CausalCLIPSeg [8] 72.03 76.72 79.17 83.83 83.59 88.07 85.90 90.37 MedCLIPSeg [23] 78.27 83.38 83.06 87.99 85.01 89.78 86.44 91.07 DistMedVL (Ours) 81.50 86.54 84.63 89.44 86.22 90.91 87.63 92.36 IV-B Implementation Details All experiments are conducted on a single NVIDIA RTX A6000 GPU (48GB). We use a batch size of 24 and optimize the model using the AdamW optimizer with a learning rate of 3×10−43× 10^-4 and default betas (0.9,0.999)(0.9,0.999). The learning rate is scheduled by cosine annealing with a minimum of 1×10−41× 10^-4 over 100 epochs. Input images are resized to 224×224224× 224 and normalized using ImageNet statistics. No data augmentation is applied during training or testing. IV-C Evaluation Metrics We adopt two complementary metrics to comprehensively assess segmentation quality. The Dice Similarity Coefficient (DSC) measures the volumetric overlap between the predicted segmentation mask P and the ground truth G, and it is particularly sensitive to region-level accuracy. The Normalized Surface Distance (NSD) evaluates boundary quality by computing the fraction of predicted boundary points that lie within a specified tolerance distance τ of the ground-truth boundary, providing a complementary measure of contour precision that is clinically relevant for delineating anatomical structures with complex or ambiguous boundaries. We report both metrics as percentages, where higher values indicate better segmentation accuracy and boundary delineation, respectively. Figure 3: Qualitative comparison of vision-language models on medical image segmentation under the 100% training data setting. White regions indicate correctly segmented areas, green regions denote under-segmentation, and red regions denote over-segmentation. TABLE I: Domain generalization results. Models are trained on the source domain and evaluated on out-of-distribution target datasets. Three groups of experiments are conducted under varying levels of aleatoric uncertainty. Method Breast Ultrasound Colonoscopy Thyroid Nodule Source: BUSI Target: BUID Source: Kvasir-SEG Target: CVC-Clinic Source: TN3K Target: PKTN Dice NSD Dice NSD Dice NSD Dice NSD Dice NSD Dice NSD Unimodal Approaches UNet [33] 70.67 74.86 71.44 76.01 82.13 85.34 57.96 61.95 81.08 86.49 46.06 51.67 DCSAU-Net [43] 77.28 81.65 69.37 74.18 74.37 78.09 50.09 54.33 80.70 86.12 47.76 53.61 FAT-Net [42] 80.37 84.44 77.41 82.35 89.23 92.34 79.08 83.28 81.94 87.18 56.90 63.53 Swin-Unet [6] 78.64 83.07 73.11 77.44 89.55 92.61 79.04 83.76 81.97 87.57 60.28 68.34 TransUnet [7] 75.91 80.18 77.42 82.35 89.80 93.62 81.43 86.52 80.92 86.61 54.86 62.38 CFFormer [25] 79.46 83.61 75.19 79.65 89.43 92.44 79.70 84.04 81.47 86.63 60.32 67.86 Crossmodal Approaches TGANet [38] 82.26 86.37 79.77 84.50 90.00 93.57 80.15 84.31 82.47 88.43 61.65 69.27 LViT [26] 81.42 85.86 64.37 69.60 87.08 90.43 68.16 73.76 80.71 85.91 54.10 60.27 CLIPSeg [28] 81.88 86.82 55.81 61.23 86.92 90.46 81.09 86.67 80.56 86.72 63.02 72.84 LanGuideMedSeg [46] 84.65 88.72 81.77 86.76 90.19 93.76 83.60 87.94 82.53 88.64 65.64 73.28 VLSM-Adapter [11] 80.96 86.55 79.53 84.78 89.61 93.76 82.50 88.09 80.65 86.92 56.78 66.08 RecLMIS [18] 83.71 88.09 68.78 73.22 87.36 90.46 76.10 80.63 82.09 87.25 57.67 64.54 CausalCLIPSeg [8] 83.43 87.92 76.82 81.72 90.41 93.79 79.17 83.93 81.67 87.13 57.52 64.96 MedCLIPSeg [23] 84.32 88.89 82.66 87.27 90.05 93.72 82.35 87.63 82.37 88.34 63.49 72.15 DistMedVL (Ours) 86.01 90.65 83.43 88.20 90.98 94.78 84.64 89.67 84.39 90.26 68.02 77.48 Figure 4: Visualization of domain shift between source and target datasets via PCA projection of extracted features. The Maximum Mean Discrepancy (MMD) quantifies the distributional discrepancy between domains. Arrows indicate the direction of distribution shift from source to target centroids. IV-D Comparison with State-of-the-Art Methods We compare DistMedVL with both unimodal approaches (UNet [33], DCSAU-Net [43], FAT-Net [42], Swin-Unet [6], TransUNet [7], CFFormer [25]) and crossmodal approaches (TGANet [38], LViT [26], CLIPSeg [28], LanGuideMedSeg [46], VLSM-Adapter [11], RecLMIS [18], CausalCLIPSeg [8], MedCLIPSeg [23]). To simultaneously assess data efficiency and epistemic uncertainty, we train all models with 10%, 25%, 50% and 100% of the training data, since reducing the training set size naturally increases epistemic uncertainty while keeping aleatoric uncertainty constant within the same distribution [21]. Evaluations are averaged across the five benchmark datasets. As shown in Table I, DistMedVL consistently outperforms all competing methods across all data regimes. At full supervision (100% training data), our method achieves 87.63% Dice and 92.36% NSD, surpassing the best-competing crossmodal approach MedCLIPSeg [23] by 1.19% in Dice and 1.29% in NSD, and outperforming the strongest unimodal method by over 3% in both metrics. Under high epistemic uncertainty with only 10% training data, the performance gap widens further, with DistMedVL achieving at least 1.85% Dice and 1.75% NSD improvement over the best-competing approach. This widening advantage under data scarcity suggests that probabilistic alignment becomes increasingly beneficial as epistemic uncertainty grows, since the variance-conditioned matching inherently downweights unreliable features that arise from insufficient training. The consistent gains across all data ratios demonstrate that our approach provides robust benefits regardless of the epistemic uncertainty level. Fig. 3 presents qualitative segmentation results across the five benchmark datasets. When competing methods produce fragmented predictions or fail to accurately localize pathological regions, DistMedVL retains precise lesion capture and boundary delineation. This is attributed to the variance-aware alignment in the PCM-Adapter that suppresses unreliable feature dimensions during cross-modal matching, reducing the impact of both imaging noise and model uncertainty on the final prediction. IV-E Domain Generalization To further evaluate robustness against epistemic uncertainty arising from distribution shift, we conduct cross-domain experiments where models are trained on a source domain and directly evaluated on unseen target domains without fine-tuning. We select three source-target pairs representing progressively increasing domain shift: breast ultrasound (BUSI [1] → BUID [3]), colonoscopy (Kvasir-SEG [20] → CVC-ClinicDB [36]), and thyroid ultrasound (TN3K [14] → PKTN [35]). As illustrated in Fig. 4, PCA projections of CLIP-extracted features confirm the ordering that the breast ultrasound pair exhibits substantial distributional overlap, the colonoscopy pair shows moderate divergence, while the thyroid pair displays the most significant separation with the highest MMD value, indicating the most severe epistemic uncertainty. As shown in Table I, DistMedVL consistently outperforms all competing methods across all three domain shift settings. In the breast ultrasound transfer (BUSI → BUID), our model surpasses MedCLIPSeg [23] on the target domain by 0.77% in Dice and 0.93% in NSD. In the colonoscopy transfer (Kvasir-SEG → CVC-ClinicDB), DistMedVL achieves 84.64% Dice on the target domain, maintaining a 1.04% Dice lead over the best-competing method. In the most challenging thyroid transfer (TN3K → PKTN), where epistemic uncertainty is highest as evidenced by the largest MMD in Fig. 4, our model achieves a substantial 2.38% Dice and 4.20% NSD advantage over the state-of-the-art. Notably, the performance advantage of DistMedVL grows as the domain shift intensifies, demonstrating that probabilistic alignment becomes increasingly beneficial under higher epistemic uncertainty. This property is particularly relevant for real-world clinical deployment, where models inevitably encounter unseen imaging protocols and patient populations. Rather than degrading gracefully like deterministic baselines, DistMedVL appears to leverage the additional uncertainty signal to recalibrate its predictions, suggesting that explicit distributional modeling offers a more robust foundation for generalization than point-estimate representations alone. Figure 5: Qualitative comparison of uncertainty maps. Warmer colors indicate higher predictive uncertainty, while cooler colors denote lower uncertainty. IV-F Perturbation Robustness To evaluate robustness against aleatoric uncertainty, we introduce controlled perturbations to both input modalities: Gaussian blur (7×77× 7 kernel, σ=2σ=2) for images, and spatial orientation inversion for text prompts. We evaluate three scenarios: image-only, text-only, and joint perturbation. We compare against MedCLIPSeg [23], which also explicitly addresses uncertainty, on TN3K [14] (high intrinsic speckle noise) and Kvasir-SEG [20] (relatively clean data) to examine degradation across varying levels of intrinsic aleatoric uncertainty. As shown in Table IV, DistMedVL exhibits substantially smaller performance degradation under all perturbation scenarios. Under image perturbation on TN3K, our Dice decreases by only 1.08%, compared to 2.44% for MedCLIPSeg. On Kvasir-SEG, our degradation under image perturbation is also lower than that of MedCLIPSeg. The advantage is more pronounced under text perturbation, where our model benefits from the MAM’s distribution-level alignment that downweights high-variance dimensions regardless of the query state, unlike MedCLIPSeg’s query-dependent variance penalty that weakens when the query itself is corrupted. The DFM further reinforces robustness through its reliability gate, which detects perturbation-induced inter-patch inconsistency and suppresses unreliable visual signals. Under joint perturbation where aleatoric uncertainty is maximized, DistMedVL continues to maintain significantly lower degradation across both datasets, demonstrating the resilience of our dual uncertainty-aware alignment against aleatoric uncertainty across both modalities. TABLE IV: Perturbation experiment results. IP: Gaussian blur applied to input images. TP: spatial directional cues in text prompts inverted. ✓/✗ indicate presence/absence of perturbation. Model IP TP TN3K Kvasir-SEG Dice NSD Dice NSD MedCLIPSeg [23] ✗ ✗ 82.37 88.34 90.05 93.72 MedCLIPSeg [23] ✓ ✗ 79.93 85.63 89.54 93.23 MedCLIPSeg [23] ✗ ✓ 76.96 82.26 89.31 93.06 MedCLIPSeg [23] ✓ ✓ 74.82 80.10 89.15 92.91 DistMedVL (Ours) ✗ ✗ 84.39 90.26 90.98 94.78 DistMedVL (Ours) ✓ ✗ 83.31 88.97 90.58 94.45 DistMedVL (Ours) ✗ ✓ 80.83 86.42 90.74 94.53 DistMedVL (Ours) ✓ ✓ 79.00 84.49 90.23 94.03 IV-G Ablation Study Table V presents the ablation study of the PCM-Adapter components. The decoder-only baseline without adapter insertion achieves 81.12% Dice and 86.28% NSD, as the frozen encoder pretrained on natural images lacks domain-specific priors for medical image segmentation. Inserting the MAM alone improves Dice to 85.76%, while the DFM alone yields 85.02%, confirming the effectiveness of both modules in bridging the domain gap through intermediate feature adaptation. Combining both modules in a parallel configuration further improves performance on BUSI (86.96% Dice), demonstrating their complementary contributions. However, the sequential configuration, where the DFM receives uncertainty-refined features from the MAM, achieves the best overall performance across both datasets, particularly on Kvasir-SEG (90.98% Dice vs. 90.35% in parallel). This validates our design motivation: the MAM first produces uncertainty-refined visual features, providing a more reliable input for the subsequent DFM to perform global semantic alignment, whereas the parallel configuration operates on raw features independently and cannot benefit from this cascaded refinement. TABLE V: Ablation study on adapter configurations within the PCM Adapter. ✓/✗ denote the presence/absence of each module. Seq. indicates sequential arrangement. MDM DFM Sequential BUSI Kvasir-SEG Dice NSD Dice NSD ✗ ✗ ✗ 81.12 86.28 87.94 91.77 ✓ ✗ ✗ 85.76 90.52 90.21 93.98 ✗ ✓ ✗ 85.02 90.55 89.68 93.35 ✓ ✓ ✗ 86.96 90.67 90.35 94.11 ✓ ✓ ✓ 86.01 90.65 90.98 94.78 Fig. 5 presents predictive uncertainty maps estimated via Monte Carlo Dropout with 30 forward passes. Compared to VLSM-Adapter [11] and MedCLIPSeg [23], DistMedVL consistently concentrates high-entropy regions along segmentation boundaries while suppressing spurious uncertainty in both foreground and background. This indicates well-calibrated posterior estimates and demonstrates that the PCM-Adapter preserves fine-grained boundary delineation where competing methods produce spatially diffuse uncertainty. V Conclusion In this paper, we propose DistMedVL, a probabilistic vision‑language framework for uncertainty‑aware medical image segmentation. We introduce a lightweight PCM‑Adapter consisting of two sequential modules, Mahalanobis Alignment and Distribution Flow, which explicitly mitigate the impact of uncertainty on segmentation through variance‑conditioned cross‑modal alignment and reliability‑gated distributional refinement. To thoroughly evaluate its robustness against uncertainty, we conduct experiments on data efficiency, cross‑domain generalization, and perturbation robustness. The results demonstrate that DistMedVL effectively suppresses uncertainty across diverse scenarios. References [1] W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy (2020) Dataset of breast ultrasound images. Data Brief 28, p. 104863. Cited by: §IV-A, §IV-E, TABLE I. [2] S. M. Anwar, M. Majid, A. Qayyum, M. Awais, M. Alnowami, and M. K. Khan (2018) Medical image analysis using convolutional neural networks: a review. J. Med. Syst. 42 (11), p. 226. Cited by: §I. [3] A. A. Ardakani, A. Mohammadi, M. Mirza-Aghazadeh-Attari, and U. R. Acharya (2023) An open-access breast lesion ultrasound image database: applicable in artificial intelligence studies. Comput. Biol. Med. 152, p. 106438. Cited by: §IV-A, §IV-E, TABLE I. [4] R. Azad, E. K. Aghdam, A. Rauland, Y. Jia, A. H. Avval, A. Bozorgpour, S. Karimijafarbigloo, J. P. Cohen, E. Adeli, and D. Merhof (2024) Medical image segmentation review: the success of u-net. IEEE Trans. Pattern Anal. Mach. Intell. 46 (12), p. 10076–10095. Cited by: §I-A. [5] R. Azad, M. Heidari, M. Shariatnia, E. K. Aghdam, S. Karimijafarbigloo, E. Adeli, and D. Merhof (2022) Transdeeplab: convolution-free transformer-based deeplab v3+ for medical image segmentation. In Proc. PRIME Workshop, p. 91–102. Cited by: §I. [6] H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, and M. Wang (2022) Swin-unet: unet-like pure transformer for medical image segmentation. In ECCV, p. 205–218. Cited by: §I, §I-A, §IV-D, TABLE I, TABLE I. [7] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou (2021) Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306. Cited by: §I, §I-A, §IV-D, TABLE I, TABLE I. [8] Y. Chen, M. Wei, Z. Zheng, J. Hu, Y. Shi, S. Xiong, X. X. Zhu, and L. Mou (2024) Causalclipseg: unlocking clip’s potential in referring medical image segmentation with causal intervention. In MICCAI, p. 77–87. Cited by: §I-B, §IV-D, TABLE I, TABLE I. [9] S. Chun, S. J. Oh, R. S. De Rezende, Y. Kalantidis, and D. Larlus (2021) Probabilistic embeddings for cross-modal retrieval. In CVPR, p. 8415–8424. Cited by: §I-C. [10] A. Degerli, S. Kiranyaz, M. E. Chowdhury, and M. Gabbouj (2022) Osegnet: operational segmentation network for covid-19 detection using chest x-ray images. In ICIP, p. 2306–2310. Cited by: §IV-A, TABLE I. [11] M. Dhakal, R. Adhikari, S. Thapaliya, and B. Khanal (2024) Vlsm-adapter: finetuning vision-language segmentation efficiently with lightweight blocks. In MICCAI, p. 712–722. Cited by: §I, §I-B, §I-B, §IV-D, §IV-G, TABLE I, TABLE I. [12] S. Eslami, C. Meinel, and G. De Melo (2023) Pubmedclip: how much does clip benefit visual question answering in the medical domain?. In Findings Assoc. Comput. Linguist.: EACL, p. 1181–1193. Cited by: §I-B. [13] Y. Gal and Z. Ghahramani (2016) Dropout as a bayesian approximation: representing model uncertainty in deep learning. In ICML, p. 1050–1059. Cited by: §I-C. [14] H. Gong, J. Chen, G. Chen, H. Li, F. Chen, and G. Li (2022) Thyroid region prior guided attention for ultrasound segmentation of thyroid nodules. Comput. Biol. Med. 106389, p. 1–12. Cited by: §IV-A, §IV-E, §IV-F, TABLE I. [15] D. Gutman, N. C. Codella, E. Celebi, B. Helba, M. Marchetti, N. Mishra, and A. Halpern (2016) Skin lesion analysis toward melanoma detection: a challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1605.01397. Cited by: §IV-A, TABLE I. [16] A. He, K. Wang, T. Li, C. Du, S. Xia, and H. Fu (2023) H2former: an efficient hierarchical hybrid transformer for medical image segmentation. IEEE Trans. Med. Imaging 42 (9), p. 2763–2775. Cited by: §I, §I-A. [17] W. Huang, C. Li, H. Yang, J. Liu, Y. Liang, H. Zheng, and S. Wang (2024) Enhancing the vision–language foundation model with key semantic knowledge-emphasized report refinement. Med. Image Anal. 97, p. 103299. Cited by: §I. [18] X. Huang, H. Li, M. Cao, L. Chen, C. You, and D. An (2024) Cross-modal conditioned reconstruction for language-guided medical image segmentation. IEEE Trans. Med. Imaging 44 (4), p. 1821–1835. Cited by: §IV-D, TABLE I, TABLE I. [19] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. (2024) Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Cited by: §IV-A. [20] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. De Lange, D. Johansen, and H. D. Johansen (2019) Kvasir-seg: a segmented polyp dataset. In M, p. 451–462. Cited by: §IV-A, §IV-E, §IV-F, TABLE I. [21] A. Kendall and Y. Gal (2017) What uncertainties do we need in bayesian deep learning for computer vision?. Adv. Neural Inf. Process. Syst. 30. Cited by: §I-C, §IV-D. [22] M. U. Khattak, S. Kunhimon, M. Naseer, S. Khan, and F. S. Khan (2024) Unimed-clip: towards a unified image-text pretraining paradigm for diverse medical imaging modalities. arXiv preprint arXiv:2412.10372. Cited by: §I-B. [23] T. Koleilat, H. Asgariandehkordi, O. Nejatimanzari, B. Barile, Y. Xiao, and H. Rivaz (2026-06) MedCLIPSeg: probabilistic vision-language adaptation for data-efficient and generalizable medical image segmentation. In CVPR, p. 1406–1417. Cited by: §I-B, §I-E, §IV-A, §IV-D, §IV-D, §IV-E, §IV-F, §IV-G, TABLE I, TABLE I, TABLE IV, TABLE IV, TABLE IV, TABLE IV. [24] B. Lakshminarayanan, A. Pritzel, and C. Blundell (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inf. Process. Syst. 30. Cited by: §I-C. [25] J. Li, Q. Xu, X. He, Z. Liu, D. Zhang, R. Wang, R. Qu, and G. Qiu (2026) CFFormer: cross cnn-transformer channel attention and spatial feature fusion for improved segmentation of heterogeneous medical images. Expert Syst. Appl. 295, p. 128835. Cited by: §I-A, §IV-D, TABLE I, TABLE I. [26] Z. Li, Y. Li, Q. Li, P. Wang, D. Guo, L. Lu, D. Jin, Y. Zhang, and Q. Hong (2023) Lvit: language meets vision transformer in medical image segmentation. IEEE Trans. Med. Imaging 43 (1), p. 96–107. Cited by: §I, §IV-D, TABLE I, TABLE I. [27] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez (2017) A survey on deep learning in medical image analysis. Med. Image Anal. 42, p. 60–88. Cited by: §I. [28] T. Lüddecke and A. Ecker (2022) Image segmentation using text and image prompts. In CVPR, p. 7086–7096. Cited by: §IV-D, TABLE I, TABLE I. [29] T. Lüddecke and A. Ecker (2022-06) Image segmentation using text and image prompts. In CVPR, p. 7086–7096. Cited by: §I-A, §I-E. [30] J. Mukhoti, A. Kirsch, J. Van Amersfoort, P. H. Torr, and Y. Gal (2023) Deep deterministic uncertainty: a new simple baseline. In CVPR, p. 24384–24394. Cited by: §I-C. [31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In ICML, p. 8748–8763. Cited by: §I, §I-B, §I-E. [32] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu (2022) Denseclip: language-guided dense prediction with context-aware prompting. In CVPR, p. 18082–18091. Cited by: §I. [33] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In MICCAI, p. 234–241. Cited by: §I, §I-A, §IV-D, TABLE I, TABLE I. [34] S. Shurrab and R. Duwairi (2022) Self-supervised learning methods and applications in medical imaging analysis: a survey. PeerJ Comput. Sci. 8, p. e1045. Cited by: §I. [35] X. Sun, B. Wei, Y. Jiang, L. Mao, and Q. Zhao (2024) CLIP-tnseg: a multi-modal hybrid framework for thyroid nodule segmentation in ultrasound images. External Links: 2412.05530 Cited by: §IV-A, §IV-E, TABLE I. [36] N. Tajbakhsh, S. R. Gurudu, and J. Liang (2015) Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans. Med. Imaging 35 (2), p. 630–644. Cited by: §IV-A, §IV-E, TABLE I. [37] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. N. Chiang, Z. Wu, and X. Ding (2020) Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Med. Image Anal. 63, p. 101693. Cited by: §I. [38] N. K. Tomar, D. Jha, U. Bagci, and S. Ali (2022) TGANet: text-guided attention for improved polyp segmentation. In MICCAI, p. 151–160. Cited by: §IV-D, TABLE I, TABLE I. [39] U. Upadhyay, S. Karthik, M. Mancini, and Z. Akata (2023) Probvlm: probabilistic adapter for frozen vison-language models. In ICCV, p. 1899–1910. Cited by: §I-C. [40] Z. Wang, Z. Wu, D. Agarwal, and J. Sun (2022) Medclip: contrastive learning from unpaired medical images and text. In EMNLP, p. 3876–3887. Cited by: §I. [41] R. Wen, H. Yuan, D. Ni, W. Xiao, and Y. Wu (2024) From denoising training to test-time adaptation: enhancing domain generalization for medical image segmentation. In WACV, p. 464–474. Cited by: §I. [42] H. Wu, S. Chen, G. Chen, W. Wang, B. Lei, and Z. Wen (2022) FAT-net: feature adaptive transformers for automated skin lesion segmentation. Med. Image Anal. 76, p. 102327. Cited by: §I, §I-A, §IV-D, TABLE I, TABLE I. [43] Q. Xu, Z. Ma, W. Duan, et al. (2023) DCSAU-net: a deeper and more compact split-attention u-net for medical image segmentation. Comput. Biol. Med. 154, p. 106626. Cited by: §I-A, §IV-D, TABLE I, TABLE I. [44] Y. Xu and T. Zhang (2024) From deterministic to probabilistic: a novel perspective on domain generalization for medical image segmentation. arXiv preprint arXiv:2412.05572. Cited by: §I. [45] Z. Yang, J. Wang, Y. Tang, K. Chen, H. Zhao, and P. H. Torr (2022) Lavt: language-aware vision transformer for referring image segmentation. In CVPR, p. 18155–18165. Cited by: §I. [46] Y. Zhong, M. Xu, K. Liang, K. Chen, and M. Wu (2023) Ariadne’s thread: using text prompts to improve segmentation of infected areas from chest x-ray images. In MICCAI, p. 724–733. Cited by: §IV-D, TABLE I, TABLE I.