Paper deep dive

Hybrid Diffusion Model for Breast Ultrasound Image Augmentation

Farhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura J. Brattain

Year: 2026Venue: arXiv preprintArea: eess.IVType: PreprintEmbeddings: 16

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 3/31/2026, 1:38:09 AM

Summary

The paper proposes a hybrid diffusion-based augmentation framework for breast ultrasound (BUS) images, combining Stable Diffusion v1.5 with Low-Rank Adaptation (LoRA) and Textual Inversion (TI) to improve image fidelity and address class imbalance. The method incorporates an image-to-image (img2img) refinement stage to preserve ultrasound-specific textures, resulting in a significant reduction in Frechet Inception Distance (FID) while maintaining classification performance on the BUSI dataset.

Entities (5)

BUSI · dataset · 100%LoRA · technique · 100%ResNet18 · model-architecture · 100%Stable Diffusion v1.5 · model-architecture · 100%Textual Inversion · technique · 100%

Relation Signals (3)

Hybrid Diffusion Framework → augments → BUSI

confidence 100% · We used the proposed framework to generate synthetic samples for underrepresented classes, addressing data imbalance in the open-source Kaggle breast ultrasound image (BUSI) dataset

ResNet18 → classifies → BUSI

confidence 100% · ResNet18 was used to classify the three classes.

Hybrid Diffusion Framework → uses → Stable Diffusion v1.5

confidence 100% · We implemented Stable Diffusion v1.5 (SD1.5) [8], fine-tuned with low-rank adaptation (LoRA)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We propose a hybrid diffusion-based augmentation framework to overcome the critical challenge of ultrasound data augmentation in breast ultrasound (BUS) datasets. Unlike conventional diffusion-based augmentations, our approach improves visual fidelity and preserves ultrasound texture by combining text-to-image generation with image-to-image (img2img) refinement, as well as fine-tuning with low-rank adaptation (LoRA) and textual inversion (TI). Our method generated realistic, class-consistent images on an open-source Kaggle breast ultrasound image dataset (BUSI). Compared to the Stable Diffusion v1.5 baseline, incorporating TI and img2img refinement reduced the Frechet Inception Distance (FID) from 45.97 to 33.29, demonstrating a substantial gain in fidelity while maintaining comparable downstream classification performance. Overall, the proposed framework effectively mitigates the low-fidelity limitations of synthetic ultrasound images and enhances the quality of augmentation for robust diagnostic modeling.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

16,098 characters extracted from source content.

Expand or collapse full text

HYBRID DIFFUSION MODEL FOR BREAST ULTRASOUND IMAGE AUGMENTATION Farhan Fuad Abir 1,2 Sanjeda Sara Jennifer 3 Niloofar Yousefi 4 Laura J. Brattain 2 1 Electrical and Computer Engineering, University of Central Florida 2 Department of Medicine, University of Central Florida College of Medicine 3 Department of Computer Science, University of Central Florida 4 Industrial Engineering and Management Systems, University of Central Florida ABSTRACT We propose a hybrid diffusion-based augmentation frame- work to overcome the critical challenge of ultrasound data augmentation in breast ultrasound (BUS) datasets. Unlike conventional diffusion-based augmentations, our approach improves visual fidelity and preserves ultrasound texture by combining text-to-image generation with image-to-image (img2img) refinement, as well as fine-tuning with low-rank adaptation (LoRA) and textual inversion (TI). Our method generated realistic, class-consistent images on an open-source Kaggle breast ultrasound image dataset (BUSI). Compared to the Stable Diffusion v1.5 baseline, incorporating TI and img2img refinement reduced the Fr ́ echet Inception Distance (FID) from 45.97 to 33.29, demonstrating a substantial gain in fidelity while maintaining comparable downstream clas- sification performance.Overall, the proposed framework effectively mitigates the low-fidelity limitations of synthetic ultrasound images and enhances the quality of augmentation for robust diagnostic modeling. Index Terms— Breast Ultrasound, Diffusion Models, Textual Inversion, LoRA, Data Augmentation 1. INTRODUCTION Ultrasound is a key medical imaging modality for breast can- cer diagnosis due to its affordability, non-invasive nature, and real-time imaging capabilities [1]. However, deep learn- ing–based ultrasound interpretation faces persistent chal- lenges due to limited and imbalanced datasets that constrain model generalizability across lesion types. Traditional aug- mentation techniques, such as flipping, rotation, and intensity variations, offer only limited diversity [2]. Early generative approaches using Generative Adversarial Networks (GANs) often produce artifacts and fail to replicate the speckle noise and subtle tissue textures critical for diagnosis [3]. Diffusion models progressively denoise latent represen- tations, yielding structural detail essential for ultrasound synthesis.These models have been widely used for de- noising, despeckling [4, 5] and text-conditioned genera- tion of echocardiograms or breast ultrasound (BUS) images [6, 7]. These methods showed promising results of using diffusion-based image generation for BUS. However, exist- ing diffusion-based BUS generation tends to produce overly smooth images that lack the characteristic speckle noise and structural complexity of real BUS images. To address the research gap, we propose a hybrid diffusion augmentation approach to improve the fidelity of ultrasound synthesis and the severe class imbalance in breast ultrasound datasets. Our framework integrates semantic conditioning with text-to-image (text2img) generation and structural re- finement with image-to-image (img2img) diffusion [8]. First, we implemented Stable Diffusion v1.5 (SD1.5) [8], fine-tuned with low-rank adaptation (LoRA) [9] and enhanced with tex- tual inversion (TI) [10] to embed domain-specific ultrasound concepts into the model’s latent space. Then, we use img2img for retaining fine-grained BUS structures. This hybrid strat- egy retains ultrasound texture, tissue heterogeneity, and le- sion boundaries more faithfully than other diffusion-based approaches. We used the proposed framework to generate synthetic samples for underrepresented classes, addressing data imbalance in the open-source Kaggle breast ultrasound image (BUSI) dataset [11]. By producing class-consistent and anatomically coherent ultrasound images, the method improves downstream classification performance. 2. METHODS 2.1. Dataset Description We used the Kaggle open-source Breast Ultrasound Images (BUSI) dataset consisting of 780 images categorized as be- nign (437), malignant (210), and normal (133) [11]. All im- ages were resized and normalized. The train-validation split (80-20) had 623 training images (349 benign, 168 malignant, 106 normal) and 157 validation images (88 benign, 42 malig- nant, 27 normal), with benign being the majority class. 2.2. Hybrid Diffusion Model Fig.1 illustrates the three-stage model workflow: prompt gen- eration, model fine-tuning, and synthetic image generation. arXiv:2603.26834v1 [eess.IV] 27 Mar 2026 Prompt Generator Image Label “ultrasound image of a malignant breast lesion.” Prompt malignant Prompt Generation Images Prompts Stable Diffusion v1.5 LoRA Finetuned SD1.5 Text2Img LoRA Finetuning Prompts Generated Images Finetuned SD1.5 Text2Img SD1.5 Img2Img w/ LoRA weights Final Synthetic Images Synthetic Image Generation Images Token <ultrasound> Stable Diffusion v1.5 Textual Inversion Token Embedding <ultrasound> Token Generation Token Embedding <ultrasound> (a)(b) (c) Fig. 1. Overview of the proposed hybrid diffusion-based image generation framework for breast ultrasound. The method consists of three main stages. (a) The preprocessing stage converts the labels into descriptive prompts. (b) LoRA finetuning and Token Generation adapts the Stable Diffusion v1.5 using LoRA for image–prompt alignment and Textual Inversion for learning domain-specific <ultrasound> token. (c) In the final workflow, the prompts and learned token <ultrasound> are passed through the finetuned text-to-image (text2img) model to generate synthetic images, which are further refined using an image-to-image (img2img) stage with LoRA weights, yielding the final synthetic ultrasound images. Prompt Generation. We derived class-specific prompts from BUSI labels (normal, benign, malignant) to guide the generation process.These prompts were mapped to radiology-style descriptions such as “ultrasound image of a benign (or malignant) breast lesion.” For the normal class, the prompt was: “ultrasound image of normal breast tissue.” Model Fine-Tuning. Firstly, we used low-rank adap- tation (LoRA) to fine-tune the Stable Diffusion model with the images and associated text prompts.This facilitated domain-specific feature learning while avoiding full-scale model pretraining.Then we introduced a custom token <ultrasound> using TI to enhance the model’s under- standing of ultrasound-specific prompts. TI learned a new embedding that captured characteristic texture and speckle patterns from representative BUSI images. This embedding was incorporated into all prompts to strengthen domain align- ment and ensure consistent synthesis across BUS classes. Synthetic Image Generation. At first, the TI embedding was imported alongside the LoRA adapters. Then we gener- ated the synthetic BUS images from the finetuned text2img SD1.5 model. Additionally we applied Stable Diffusion’s img2img model to enhance the fidelity of the generated out- puts. This refinement process was performed with an empir- ically chosen low denoising strength of 0.3. Afterward, we added synthetic images to each class of the training set, yield- ing 350 samples per class to match the majority class, while the validation set remained unchanged. 2.3. Evaluation Metrics We used accuracy, F1-score, Area Under the Receiver Oper- ating Characteristic Curve (AUC-ROC), positive predictive value (PPV), and recall to evaluate classification perfor- mance. Moreover, we used the Fr ́ echet Inception Distance (FID), which quantifies the similarity between real and syn- thetic image distributions. FID scores were computed using Inception v3 features on 780 real and synthetic images. 3. EXPERIMENTS AND RESULTS 3.1. Experimental Setup We conducted five experiments using original ultrasound im- ages as the baseline. ResNet18 was used to classify the three classes. (1) Baseline (Original): trained only on original ultrasound images; (2) Original + SD1.5: augmented with synthetic samples generated by SD1.5; (3) Original + SD1.5 + img2img: adding img2img translation for style adaptation; (4) Original + SD1.5 + TI: leveraging textual inversion for domain-specific embedding alignment; and (5) Original + SD1.5 + TI + img2img: combining all components. ResNet18 was trained with the Adam optimizer (learning rate = 0.0001, batch size = 16, 30 epochs) using cross-entropy loss. We fine-tuned SD1.5 using the standard denoising ob- jective whereas for TI we updated the learned token embed- ding. Standard augmentations (horizontal flip, normalization) were applied to improve generalization. All experiments ran Table 1. Impact of each component in the hybrid diffusion framework on classification and image quality metrics. ComponentsAccuracy↑ F1-Score↑ AUC-ROC↑ PPV↑ FID↓ Baseline (Original Images)0.9040.8870.9790.890- Original + SD1.50.9170.9050.9860.90145.97 Original + SD1.5 + img2img0.8980.8790.9780.87838.34 Original + SD1.5 + TI0.9240.9120.9800.90645.66 Original + SD1.5 + TI + img2img0.9050.8840.9750.89 37.18 Malignant Real SD1.5 SD1.5 + Img2Img Benign Malignant Normal SD1.5 + TI SD1.5 + TI + Img2Img Fig. 2. Comparison of real ultrasound images and synthetic variants generated by 4 different approaches based on SD1.5. Rows correspond to breast lesion categories: benign (top), malignant (middle), and normal (bottom). Columns show (from left to right): real images, baseline SD1.5 generations, SD1.5 with img2img refinement, SD1.5 with TI, and SD1.5 combined with TI and img2img. The img2img refinement increases the fidelity by improving ultrasound textures. on a workstation with a 12th Gen Intel Core i5-12500 CPU, 128 GB RAM, and an NVIDIA RTX A4000 (16 GB) GPU on Ubuntu Linux. Model training and diffusion-based syn- thesis were implemented with Python, PyTorch [12], and the Hugging Face diffusers library [13]. 3.2. Results Table 1 summarizes the impact of LoRA fine-tuning, TI, and img2img refinement. Using SD1.5 text2img augmenta- tion with real data improved the baseline to Acc 0.917, F1 0.905, and the highest AUC-ROC 0.986 (FID 45.97). In- corporating TI achieved the best overall classification (Acc 0.924, F1 0.912, PPV 0.906), demonstrating that the learned domain token enhances class-aware synthesis (FID 45.66). Img2img consistently lowered FID (to 38.34 without TI and 37.18 with TI) but slightly reduced accuracy and F1. AUC- ROC remained ≥ 0.975 across all settings, indicating stable discriminative performance. Overall, the classifier achieved comparable classification performance on the original imbal- anced data and the synthetic dataset. However, our method shows significant decrease in FID score which highlights the improved visual fidelity of the generated images. 3.3. Qualitative Analysis Fig.2 shows examples of synthetic BUS images of 3 differ- ent classes. The images demonstrate that finetuned SD1.5 captures ultrasound features, such as lesion boundaries, in- ternal textures, and background anatomy. Malignant lesions display irregular, heterogeneous texture, while benign ones exhibit smoother contours with homogeneous texture. Nor- mal cases show no lesions and variable breast tissues. In ad- dition, img2img refinement helps preserve fine-grained tissue patterns relative to SD1.5, while textual inversion enhances semantic consistency. Overall, the results suggest the syn- thetic images retain clinically meaningful traits. This visual validation complements the quantitative improvements, high- lighting the capability of the proposed method for generating high-fidelity BUS images. 4. CONCLUSION We presented a hybrid diffusion–based augmentation frame- work for breast ultrasound images that integrates prompt- driven text2img synthesis with LoRA finetuning and Tex- tual Inversion, with an img2img refinement stage. Applied to the Kaggle BUS dataset (BUSI), our method generated high-fidelity breast ultrasound images. The refinement stage significantly improved visual quality, retaining ultrasound characteristics relevant for diagnosis. Although the small dataset failed to capture the data imbalance issue, the pro- posed method achieved comparable classification perfor- mance and substantially improved the FID score. Future work will focus on closing this gap via task-aware generation, such as conditioning on lesion masks or structure- preserving priors, jointly optimizing with a diagnostic en- coder, and weighting synthetic samples by classifier confi- dence. We will also further refine our approach on larger datasets. Finally, we will assess generalizability to other med- ical imaging modalities, such as MRI, CT, and X-ray. 5. COMPLIANCE WITH ETHICAL STANDARDS This study uses the publicly available Breast Ultrasound Im- ages (BUSI) [11], which is fully anonymized and does not contain any personally identifiable information. Therefore, no additional ethical approval was required for this work. 6. ACKNOWLEDGEMENTS No funding was received for this study, and the authors have no relevant financial or non-financial interests to disclose. 7. REFERENCES [1] Adeel Khalid, Adnan Mehmood, Ali Alabrah, Bandar F. Alkhamees, Faisal Amin, Hani AlSalman, and Gyu- Sung Choi, “Breast cancer detection and prevention us- ing machine learning,” Diagnostics (Basel), vol. 13, no. 19, p. 3113, 10 2023. [2] E. Goceri, “Medical image data augmentation: tech- niques, comparisons and interpretations,” Artificial In- telligence Review, vol. 56, p. 12561–12605, 2023. [3] Yuliana Jim ́ enez-Gaona, Diana Carri ́ on-Figueroa, Va- sudevan Lakshminarayanan, and Mar ́ ıa Jos ́ e Rodr ́ ıguez- ́ Alvarez,“Gan-based data augmentation to improve breast ultrasound and mammography mass classifica- tion,” Biomedical Signal Processing and Control, vol. 94, p. 106255, 2024. [4] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, “Adding conditional control to text-to-image diffusion models,” arXiv preprint arXiv:2302.05543, 2023. [5] Hojat Asgariandehkordi, Sobhan Goudarzi, Mostafa Sharifzadeh, Adrian Basarab, and Hassan Rivaz, “De- noising plane wave ultrasound images using diffusion probabilistic models,” IEEE Transactions on Ultrason- ics, Ferroelectrics, and Frequency Control, 2024. [6] Benoit Freiche, Anthony El-Khoury, Ali Nasiri-Sarvi, Mahdi S Hosseini, Damien Garcia, Adrian Basarab, Mathieu Boily, and Hassan Rivaz,“Ultrasound im- age generation using latent diffusion models,” arXiv preprint arXiv:2502.08580, 2025. [7] Seok-Hwan Oh, Guil Jung, MyeongGee Kim, Young- Min Kim, Hyeon-Jik Lee, Sang-Yun Kim, Hyuk-Sool Kwon, and Hyeon-Min Bae, “Breast tumor image syn- thesis based on diffusion probabilistic model,” in 2024 IEEE Ultrasonics, Ferroelectrics, and Frequency Con- trol Joint Symposium (UFFC-JS). IEEE, 2024, p. 1–4. [8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ̈ orn Ommer, “High-resolution im- age synthesis with latent diffusion models,” in Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, p. 10684–10695. [9] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al., “Lora: Low-rank adaptation of large lan- guage models.,” ICLR, vol. 1, no. 2, p. 3, 2022. [10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or, “An image is worth one word: Personalizing text- to-image generation using textual inversion,”arXiv preprint arXiv:2208.01618, 2022. [11] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy, “Dataset of breast ultrasound images,” Data in brief, vol. 28, p. 104863, 2020. [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, p. 770–778. [13] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca,Nathan Lambert,Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf, “Diffusers:State-of-the-artdiffusionmodels,” https://github.com/huggingface/diffusers, 2022.

Instant research discovery