Paper deep dive
Topology-Informed Prompt-Conditioned Universal Segmentation of Uterine Structures from Ultrasound and MRI
Yongheng Sun, Yuexi Gu, Jingwen Sun, Maureen Kohi, Mingxia Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-structure segmentation of the uterus is important for computer-assisted screening, diagnosis, and treatment planning of uterine diseases, where ultrasound and MRI provide complementary clinical information. However, developing a unified model across these modalities is challenging due to their substantially different image appearances, anatomical contexts, spatial resolutions, and label spaces. Moreover, existing datasets often define different segmentation targets, making joint learning challenging and potentially leading to negative transfer across heterogeneous tasks. To this end, we propose a Topology-informed Prompt-conditioned Universal Segmentation (TPUS) framework for segmenting multiple uterine structures across ultrasound and MRI. TPUS introduces a graph-based multi-dataset backbone comprising modality-specific stems and a modality-shared graph-based encoder-decoder to support modality-sensitive input adaptation, structural feature reasoning, and joint representation learning across heterogeneous uterine segmentation tasks. In addition, TPUS uses task-aware class prompts to condition the segmentation process for different datasets and label spaces, a dynamic convolutional adaptation module to generate task-specific output responses, and a topology-informed loss to encourage anatomically consistent predictions. Experiments on a uterine ultrasound dataset and a T2-weighted uterine myoma MRI dataset demonstrate that TPUS achieves Dice scores of 0.898 and 0.693 on the two held-out test sets, respectively, outperforming several generic and universal segmentation baselines. Source code can be accessed at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.06494v1
- Canonical: https://arxiv.org/abs/2610.06494v1
Trouble viewing inline? Open PDF directly →
Full Text
23,665 characters extracted from source content.
Expand or collapse full text
Topology-Informed Prompt-Conditioned Universal Segmentation of Uterine Structures from Ultrasound and MRI Yongheng Sun Yuexi Gu Jingwen Sun Maureen Kohi Mingxia Liu Abstract Multi-structure segmentation of the uterus is important for computer-assisted screening, diagnosis, and treatment planning of uterine diseases, where ultrasound and MRI provide complementary clinical information. However, developing a unified model across these modalities is challenging due to their substantially different image appearances, anatomical contexts, spatial resolutions, and label spaces. Moreover, existing datasets often define different segmentation targets, making joint learning challenging and potentially leading to negative transfer across heterogeneous tasks. To this end, we propose a Topology-informed Prompt-conditioned Universal Segmentation (TPUS) framework for segmenting multiple uterine structures across ultrasound and MRI. TPUS introduces a graph-based multi-dataset backbone comprising modality-specific stems and a modality-shared graph-based encoder-decoder to support modality-sensitive input adaptation, structural feature reasoning, and joint representation learning across heterogeneous uterine segmentation tasks. In addition, TPUS uses task-aware class prompts to condition the segmentation process for different datasets and label spaces, a dynamic convolutional adaptation module to generate task-specific output responses, and a topology-informed loss to encourage anatomically consistent predictions. Experiments on a uterine ultrasound dataset and a T2-weighted uterine myoma MRI dataset demonstrate that TPUS achieves Dice scores of 0.898 and 0.693 on the two held-out test sets, respectively, outperforming several generic and universal segmentation baselines. Source code can be accessed online. Index Terms: Uterine Structure Segmentation, Ultrasound, Magnetic Resonance Imaging †address: 1Department of Radiology and BRIC, UNC at Chapel Hill, Chapel Hill, NC 27599, USA 1 INTRODUCTION Multi-structure segmentation of the uterus is an important task in medical image analysis, supporting computer-assisted screening, diagnosis, disease characterization, and treatment planning for a wide range of uterine diseases [1, 6]. Accurate delineation of the uterus and its internal structures, such as the uterine cavity and myometrium, as well as focal lesions including uterine fibroids and Nabothian cysts, can provide quantitative anatomical and pathological information for clinical assessment [6]. Ultrasound and MRI are two commonly used imaging modalities for evaluating uterine anatomy and pathology [1, 6], offering complementary information regarding accessibility, spatial detail, soft-tissue contrast, and tissue characterization. Developing robust methods to segment multiple uterine structures and lesions across ultrasound and MRI could facilitate comprehensive automated analysis of uterine diseases. However, achieving unified segmentation across ultrasound and MRI remains challenging due to substantial differences between the two modalities in image appearance, anatomical context, spatial resolution, and tissue contrast [1, 6]. Beyond these modality-specific differences, publicly available datasets are often designed for different clinical objectives and therefore provide heterogeneous segmentation targets and label definitions. For example, one dataset may focus on delineating the uterus and its internal structures, whereas another may primarily target uterine lesions such as fibroids. Consequently, directly combining such datasets for joint model training can introduce substantial task heterogeneity and lead to negative transfer, where knowledge beneficial to one segmentation task may interfere with another. Most existing methods are designed for a single modality, dataset, or fixed label space, and thus are not well suited for jointly learning heterogeneous uterine segmentation tasks across ultrasound and MRI. Figure 1: Overview of the proposed TPUS framework. (a) Modality-specific stems process input 3D US or MRI volumes. (b) The modality-shared graph-based encoder-decoder with topology blocks (TopoBlocks) extracts multi-scale features, with auxiliary heads at decoders D1D_1–D3D_3. (c) Task-aware class prompts from CLIP [8] and the E5E_5 bottleneck condition a dynamic head at D4D_4 for full-resolution prediction. To address these challenges, we propose a Topology-informed Prompt-conditioned Universal Segmentation (TPUS) framework for unified multi-structure segmentation across uterine ultrasound and MRI. As shown in Fig. 1, TPUS is designed to jointly learn representations across heterogeneous datasets while adapting the segmentation process to dataset-specific label spaces. Specifically, we introduce task-aware class prompts to explicitly condition the model on different segmentation targets, enabling flexible prediction across heterogeneous label definitions. A dynamic convolutional adaptation module further generates task-specific output responses, allowing the representation to be effectively adapted to different modalities and segmentation tasks. To preserve anatomical consistency, we incorporate a topology-informed loss that encourages structurally plausible predictions. In addition, we develop a graph-based multi-dataset backbone comprising modality-specific stems and a modality-shared graph-based encoder-decoder, enabling modality-sensitive input adaptation and joint representation learning across heterogeneous uterine segmentation tasks. These components enable TPUS to jointly learn from ultrasound and MRI while mitigating negative transfer and improving multi-structure uterine segmentation. Experiments on uterine ultrasound and T2-weighted MRI datasets demonstrate that the proposed TPUS consistently outperforms several generic and universal segmentation methods. 2 METHODOLOGY The proposed TPUS addresses cross-modality appearance variation and dataset-specific label spaces through modality-sensitive input adaptation, joint representation learning, prompt-conditioned prediction, and topology-informed deep supervision (Fig. 1). 2.1 Graph-Based Multi-Dataset Backbone The graph-based multi-dataset backbone comprises modality-specific stems and a modality-shared graph-based encoder-decoder. Let =(d)d=1KD=\D^(d)\_d=1^K denote K datasets, where dataset d provides an image (d)x^(d) and a label map (d)y^(d) defined over its own class set dC_d. TPUS follows the MultiTalent multi-dataset paradigm [10], while separating modality-sensitive input processing from shared anatomical representation learning. As shown in Fig. 1(a), each dataset has an independent stem dS_d comprising a ×33\!×\!3\!×\!3 convolution, instance normalization, and LeakyReLU that maps its one-channel volume to 32 channels: 0(d)=d((d)).z^(d)_0=S_d\! (x^(d) ). (1) At each iteration, one sampled volume is routed through its corresponding stem. With the reported batch size of one, the concatenation operator in Fig. 1(a) simply forwards the selected stem output rather than pairing US and MRI volumes. The output then enters the modality-shared six-level residual U-shaped encoder-decoder in Fig. 1(b), whose encoder widths are 32, 64, 128, 256, 320, and 320 channels and whose decoder widths are 320, 256, 128, 64, and 32 channels. Resolution-matched skip connections link the two paths; all parameters after the stems are shared. Within the modality-shared graph-based encoder-decoder, we adapt the Pool graph neural network (GNN) and Swin GNN [9] into several topology blocks (TopoBlocks) deployed at encoder levels E2E_2–E5E_5 and decoder levels D0D_0–D2D_2; high-resolution levels E0E_0–E1E_1 and D3D_3–D4D_4 retain convolutional residual blocks. The TopoBlock shown in the bottom right of Fig. 1 applies graph reasoning and feed-forward network (FFN) refinement residually to feature F: ^p F_p =p()+, =G_p(F)+F, p _p =FFN(^p)+^p, =FFN( F_p)+ F_p, ^w F_w =w(p)+p, =G_w(F_p)+F_p, ′ =FFN(^w)+^w, =FFN( F_w)+ F_w, (2) where pG_p denotes Pool GNN, which builds a dynamic k-nearest-neighbor graph after max pooling and restores resolution by max unpooling, and wG_w denotes Swin GNN, which builds dynamic graphs within shifted 3D windows. Both use max-relative graph convolution to complement the backbone’s local convolutional bias. 2.2 Prompt-Conditioned Dynamic Head Figure 1(c) shows the class-text, image-context, and prediction paths of the dynamic head. For each class c∈dc _d, including background, we form a task-aware prompt, such as “uterus in a gynecologic ultrasound image” or “uterine myoma in a sagittal T2 pelvic MRI image.” A frozen CLIP ViT-B/32 text encoder [8] yields an embedding d,c∈ℝ512e_d,c ^512. Then TPUS combines class semantics with image context to generate the classifier. The E5E_5 bottleneck feature B and class embedding are projected to 256 dimensions: =Conv(GAP(ReLU(GN()))), =Conv\! (GAP (ReLU(GN(B)) ) ), d,c _d,c =ReLU(td,c+t), =ReLU(W_te_d,c+b_t), d,c θ_d,c =ℋ([∥d,c]), =H\! ([i\,\|\,t_d,c] ), (3) where [⋅∥⋅][·\|·] denotes channel-wise concatenation at the circled C in Fig. 1(c), the shared ×11\!×\!1\!×\!1 controller ℋH maps the 512-D condition to d,c∈ℝ153 θ_d,c ^153, GAP denotes global average pooling, ReLU represents the rectified linear unit, and GN represents group normalization. In the prediction path, the final D4D_4 decoder feature 4∈ℝ32×D×H×WF_4 ^32× D× H× W is transformed into an eight-channel map: =Conv(ReLU(GN(4))).Q=Conv\! (ReLU(GN(F_4)) ). The dynamic convolution (DynConv) block in Fig. 1(c) comprises three class-wise dynamic convolutions parameterized by d,c θ_d,c: d,c1 ^1_d,c =σ(d,c1∗+d,c1), =σ(W^1_d,c*Q+b^1_d,c), (4) d,c2 ^2_d,c =σ(d,c2∗d,c1+d,c2), =σ(W^2_d,c*q^1_d,c+b^2_d,c), d,c _d,c =d,c3∗d,c2+d,c3, =W^3_d,c*q^2_d,c+b^3_d,c, where ∗* is a ×11\!×\!1\!×\!1 convolution, σ is ReLU, and the channel dimensions are →18\!→\!8\!→\!8\!→\!1. Each class therefore requires (82+8)+(82+8)+(8+1)=153(8^2+8)+(8^2+8)+(8+1)=153 generated parameters. The controller is shared across datasets and classes, but the resulting filters depend on both image and class. CLIP embeddings remain fixed, whereas their projection and the controller are trainable. This head replaces the fixed D4D_4 classifier and produces the native multiclass logits; softmax yields the corresponding prediction. Unlike previous CLIP-driven dynamic convolution [5], which predicts a fixed foreground vocabulary using independent sigmoid outputs, TPUS uses task-aware class prompts to generate background-inclusive softmax logits in each dataset’s native label space. 2.3 Topology-Informed Deep Supervision We apply deep supervision to outputs at decoder DsD_s, s∈1,2,3,4s∈\1,2,3,4\. Let so^s and sy^s denote the logits and corresponding target at DsD_s. We define pairwise exclusion relations among a dataset’s foreground labels. Unlike previous topology-informed losses [9], which require manually specified relations for a fixed label space, TPUS automatically derives pairwise exclusions from each dataset’s native foreground labels. Hard predictions sP^s are obtained by taking the class-wise argmax of the softmax probabilities. A ×33\!×\!3\!×\!3 kernel is then applied to sP^s to identify violations within each voxel’s 26-connected neighborhood, yielding a binary critical-voxel mask sV^s. For batch size N and voxel domain Ωs _s, the topological interaction (TI) term applies cross-entropy only at these voxels: ℒTs=1N∑n=1N∑v∈ΩsVn,vsCE(n,vs,n,vs).L^s_T= 1NΣ _n=1^NΣ _v∈ _sV^s_n,v\,CE(o^s_n,v,y^s_n,v). (5) For a single foreground class, the exclusion set is empty and ℒTs=0L_T^s=0; the four UMD foreground classes yield six exclusion pairs. The total loss is formulated as follows: ℒ=∑s=14wsℒs=∑s=14ws(ℒDices+ℒCEs+λTℒTs),L=Σ _s=1^4w_sL^s=Σ _s=1^4w_s (L^s_Dice+L^s_CE+ _TL^s_T ), (6) where λT _T is empirically set as 10−610^-6. We use normalized deep-supervision weights ws=2s−1/15w_s=2^s-1/15 for DsD_s (s∈1,2,3,4s∈\1,2,3,4\). Only the full-resolution dynamic output at D4D_4 is used at inference. Figure 2: Visual comparison of segmentation results on the UterUS (top) and UMD (bottom) datasets. Shown are the input, competing methods, the proposed TPUS, and ground truth. Different colors denote distinct tissues. 3 EXPERIMENTS 3.1 Data and Implementation The UterUS dataset [1] comprises 3D US volumes with binary uterine-region masks, excluding cases with fibroids, adenomyosis, and other acquired abnormalities. The Uterine Myoma MRI Dataset (UMD) [6] provides labels for the uterine wall, uterine cavity, myoma, and Nabothian cyst (when present) on T2-weighted MRI of patients with surgically and pathologically confirmed fibroids. Training/validation/held-out test sets contain 100/10/28 cases for UterUS and 191/21/54 for UMD. Following MultiTalent [10], we adopt its nnU-Net-based preprocessing [4] (1-m isotropic resampling and z-score normalization) and data augmentation. Training uses 1923192^3 patches, batch size 1, and 1,000 epochs of 250 iterations with Nesterov SGD (momentum 0.99, weight decay ×10−53\!×\!10^-5; polynomial decay from 10−310^-3). All methods share splits, preprocessing, and postprocessing. Dice and average symmetric surface distance (ASSD) are reported as mean± over cases per foreground class, then macro-averaged; ASSD is omitted for empty surfaces. Table 1: Segmentation results on UterUS and UMD. Method UterUS UMD Dice ↑ ASSD (m) ↓ Dice ↑ ASSD (m) ↓ UNETR [3] 0.792± 0.110 0.846∗± 0.420 0.330± 0.199 18.899∗± 21.245 SwinUNETR [2] 0.864± 0.071 0.593∗± 0.304 0.347± 0.200 15.065∗± 20.817 CLIP-UM [5] 0.861± 0.054 0.637∗± 0.251 0.144± 0.137 19.165∗± 11.165 UniSeg [11] 0.498± 0.160 1.715∗± 0.513 0.599± 0.242 2.732∗± 4.737 DoDNet [12] 0.891± 0.050 0.760∗± 1.169 0.433± 0.197 17.143∗± 19.048 PaNN [13] 0.780± 0.095 2.242∗± 2.099 0.273± 0.158 20.466∗± 9.162 MultiTalent [10] 0.895± 0.051 0.448∗± 0.162 0.680± 0.257 1.775∗± 3.880 TPUS (Ours) 0.898± ±0.050 0.432± ±0.156 0.693± ±0.254 1.684± ±4.680 3.2 Experimental Results We compare the proposed TPUS with task-specific UNETR [3] and SwinUNETR [2], and five joint-training methods: CLIP-UM with CLIP embeddings [5], UniSeg with task prompts [11], DoDNet with task-conditioned filters [12], PaNN with size priors [13], and MultiTalent with dataset-specific heads [10]. Table 1 reports results of the eight methods on UterUS and UMD. Table 1 shows that TPUS achieves the best Dice and ASSD on UterUS (0.898 and 0.432 m vs. 0.895 and 0.448 m for MultiTalent) and UMD (0.693 and 1.684 m vs. 0.680 and 1.775 m). Two-sided paired Wilcoxon signed-rank tests comparing TPUS with each competing method in terms of ASSD demonstrate that TPUS achieves statistically significantly better performance (p<<0.050.05, marked with ∗). Figure 2 qualitatively compares different methods on both datasets. On the challenging 3D US dataset (UterUS), competing methods often exhibit target fragmentation or under-segmentation due to acoustic shadowing and low soft-tissue contrast, whereas TPUS produces compact, anatomically contiguous, and boundary-complete uterine masks. On the multi-class T2-weighted MRI dataset (UMD), TPUS suppresses noisy and distant false positives while preserving the uterine wall and cavity and accurately delineating focal lesions, such as myomas and Nabothian cysts, with sharp boundaries. This highlights TPUS’s ability to maintain structural integrity and adapt to heterogeneous multi-structure label spaces. Table 2: Ablation results on all 54 test cases from UMD. Configuration Dice ↑ ASSD (m) ↓ Base 0.677± 0.258 1.834± 4.608 Base + Prompt 0.678± 0.253 1.735± 4.128 Base + Prompt + TopoBlock 0.680± 0.259 1.718± 3.383 Base + Prompt + DynConv 0.684± 0.255 1.606± 4.787 Base + TI 0.686± 0.255 1.536± ±3.237 TPUS (Ours) 0.693± ±0.254 1.684± 4.680 3.3 Ablation Study Table 2 evaluates each TPUS component under a matched protocol on UMD. “Base” denotes the MultiTalent model retrained under our protocol. Adding prompt conditioning (Base+Prompt) yields a slight Dice gain, while TopoBlock integration (Base+Prompt+TopoBlock) improves multi-scale long-range reasoning, and dynamic convolution (Base+Prompt+DynConv) enhances dataset-specific adaptation. Topology-informed deep supervision (Base+TI) reduces boundary errors. Combining all components in TPUS achieves the highest Dice, demonstrating their complementary contributions. 3.4 Influence of Prompt-Encoding Strategy We evaluate the impact of different prompt-encoding strategies within the TPUS framework, comparing our default CLIP text encoder against a prompt-free FiLM approach [7], the UniSeg-style task prompt [11], and a randomly initialized learnable embedding. As reported in Table 3, removing explicit prompt conditioning or using alternative schemes slightly degrades segmentation accuracy. The prompt-free FiLM and UniSeg-style prompt variants achieve Dice score of 0.893, while the random embedding strategy reaches 0.896. The proposed pretrained CLIP text encoder achieves the highest Dice and lowest ASSD scores (p<<0.050.05 via two-sided paired Wilcoxon signed-rank tests). These results suggest that semantically rich CLIP-derived prompts used in TPUS provide more effective guidance for segmentation. Table 3: TPUS with different prompt-encoding methods on UterUS. Method Dice ↑ ASSD (m) ↓ p-value Prompt-free FiLM [7] 0.893± 0.048 0.456± 0.154 ×10−81.72\!×\!10^-8 UniSeg-style Prompt [11] 0.893± 0.050 0.463± 0.149 ×10−41.53\!×\!10^-4 Random Embedding 0.896± 0.047 0.448± 0.148 ×10−31.17\!×\!10^-3 CLIP Text Encoder (Ours) 0.898± ±0.050 0.432± ±0.156 – 4 CONCLUSION This work presents TPUS, a unified framework for multi-structure uterine segmentation across ultrasound and MRI. TPUS integrates a graph-based multi-dataset backbone comprising modality-specific stems and a modality-shared graph-based encoder-decoder, prompt-conditioned dynamic prediction, and topology-informed deep supervision for heterogeneous uterine ultrasound and MRI datasets. It achieves the highest Dice and lowest ASSD among the compared methods on both held-out test sets. A current limitation is that evaluation is restricted to one ultrasound and one MRI dataset; future work will assess scalability and generalizability by integrating additional datasets with diverse acquisition protocols and label spaces. 5 COMPLIANCE WITH ETHICAL STANDARDS De-identified UterUS and UMD data were analyzed in this work. The UMD dataset had Beijing Shijitan Hospital ethics approval (sjtkyll-lx-2022(1)) and waived consent; this secondary analysis required no additional approval. 6 ACKNOWLEDGMENTS This work was supported in part by NIH grants AG073297, AG082938, EB035160, and NS134849. The authors declare no relevant financial or non-financial conflicts of interest. References [1] E. Bonevs, M. Gergolet, C. Bohak, vZ. Lesar, and M. Marolt (2024) Automatic segmentation and alignment of uterine shapes from 3D ultrasound data. Comput. Biol. Med. 178, p. 108794. Cited by: §1, §1, §3.1. [2] A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H. R. Roth, and D. Xu (2022) Swin UNETR: swin transformers for semantic segmentation of brain tumors in MRI images. In BrainLes, p. 272–284. Cited by: §3.2, Table 1. [3] A. Hatamizadeh, Y. Tang, V. Nath, et al. (2022) UNETR: transformers for 3D medical image segmentation. In WACV, p. 574–584. Cited by: §3.2, Table 1. [4] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein (2021) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18 (2), p. 203–211. Cited by: §3.1. [5] J. Liu, Y. Zhang, J.-N. Chen, et al. (2023) CLIP-driven universal model for organ segmentation and tumor detection. In ICCV, p. 21152–21164. Cited by: §2.2, §3.2, Table 1. [6] H. Pan, M. Chen, W. Bai, et al. (2024) Large-scale uterine myoma MRI dataset covering all FIGO types with pixel-level annotations. Sci. Data 11 (1), p. 410. Cited by: §1, §1, §3.1. [7] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. C. Courville (2018) FiLM: visual reasoning with a general conditioning layer. In AAAI, p. 3942–3951. Cited by: §3.4, Table 3. [8] A. Radford, J. W. Kim, C. Hallacy, et al. (2021) Learning transferable visual models from natural language supervision. In ICML, Vol. 139, p. 8748–8763. Cited by: Figure 1, §2.2. [9] P. Shi, X. Guo, Y. Yang, C. Ye, and T. Ma (2023) NexToU: efficient topology-aware U-Net for medical image segmentation. arXiv preprint arXiv:2305.15911. Cited by: §2.1, §2.3. [10] C. Ulrich, F. Isensee, T. Wald, M. Zenk, M. Baumgartner, and K. H. Maier-Hein (2023) MultiTalent: a multi-dataset approach to medical image segmentation. In MICCAI, p. 648–658. Cited by: §2.1, §3.1, §3.2, Table 1. [11] Y. Ye, Y. Xie, J. Zhang, Z. Chen, and Y. Xia (2023) UniSeg: a prompt-driven universal segmentation model as well as a strong representation learner. In MICCAI, p. 508–518. Cited by: §3.2, §3.4, Table 1, Table 3. [12] J. Zhang, Y. Xie, Y. Xia, and C. Shen (2021) DoDNet: learning to segment multi-organ and tumors from multiple partially labeled datasets. In CVPR, p. 1195–1204. Cited by: §3.2, Table 1. [13] Y. Zhou, Z. Li, S. Bai, et al. (2019) Prior-aware neural network for partially-supervised multi-organ segmentation. In ICCV, p. 10672–10681. Cited by: §3.2, Table 1.