Paper deep dive
TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation
Hyeonseop Song, Seokhun Choi, Hoseok Do
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/10/2026, 4:39:59 AM
Summary
The paper introduces TMI, a hybrid data synthesis framework for long-tailed instance segmentation that combines text-to-image (T2I) generation for scene diversity with image-to-image (I2I) editing for realistic, accurate augmentation. It addresses limitations of existing methods by employing a teacher-student scheme for reliable pseudo-labeling and a novel "place-and-verify" I2I module called VRAIN to specifically target rare classes. Evaluated on the LVIS benchmark, TMI significantly improves overall and rare-class average precision, demonstrating effective scaling with backbone capacity.
Entities (8)
Relation Signals (6)
TMI โ evaluatedon โ LVIS
confidence 96% ยท On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points
TMI โ addresses โ Long-Tailed Instance Segmentation
confidence 95% ยท Large-vocabulary instance segmentation is constrained by long-tailed category distributions... To address these issues, we propose a hybrid pipeline...
TMI โ incorporates โ Text-to-Image
confidence 94% ยท hybrid pipeline coupling T2I generation with context-aware image-to-image (I2I) editing.
TMI โ incorporates โ Image-to-Image
confidence 94% ยท hybrid pipeline coupling T2I generation with context-aware image-to-image (I2I) editing.
VRAIN โ specializesin โ Rare Classes
confidence 92% ยท To strengthen supervision for rare classes, we introduce VRAIN, a novel I2I editor.
Teacher-Student Scheme โ ensures โ Label Reliability
confidence 90% ยท a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-vocabulary instance segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. While data synthesis offers a promising alternative, current paradigms have complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle on rare classes, whereas copy-paste methods compromise contextual realism. To address these issues, we propose a hybrid pipeline coupling T2I generation with context-aware image-to-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories. To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation. On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity. Our project page is available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2607.08201v1
- Canonical: https://arxiv.org/abs/2607.08201v1
Trouble viewing inline? Open PDF directly โ
Full Text
75,058 characters extracted from source content.
Expand or collapse full text
TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation Hyeonseop Song โ , Seokhun Choi โ , and Hoseok Do โ AI Lab, CTO Division, LG Electronics, Republic of Korea hyeonseop.song, seokhun.choi, hoseok.do@lge.com Project page: https://seokhunchoi.github.io/TMI Abstract. Large-vocabulary instance segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. While data synthesis offers a promising alternative, current paradigms have complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle on rare classes, whereas copy-paste methods compromise contextual realism. To address these issues, we pro- pose a hybrid pipeline coupling T2I generation with context-aware image- to-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories. To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation. On the LVIS benchmark, our method surpasses existing baselines, improv- ing overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity. Keywords: Generative Data Synthesis ยท Image Diffusion Models ยท Hy- brid Data Pipeline ยท Long-Tailed Instance Segmentation 1 Introduction Instance segmentation is a critical task, underpinning diverse applications from autonomous driving [4,5] and robotics [16,46,52] to visual understanding [8,26, 43] and image editing [21,25,40]. However, achieving robust performance relies on large, densely annotated datasets, which are costly to build. This bottleneck is particularly acute for large-vocabulary benchmarks such as LVIS [9] with their long-tailed category distributions. While methods like re-weighting [33, 34, 54], balanced sampling [3,11,54], and classifier calibration [29,37,38] help alleviate the imbalance, they cannot resolve the underlying data scarcity of rare categories [9]. * Equal contribution. โ Corresponding author. arXiv:2607.08201v1 [cs.CV] 9 Jul 2026 2H. Song & S. Choi et al. AP rare 24.0 24.4 25.6 29.6 33.9 MosaicFusion DiverGen X-Paste AP all Real-only Ours Data Synthesis Approach Label Accuracy Realism Scene Diversity T2I (a) Mosaic (b) Our T2I I2I (c) Copy-Paste (d) Our Edit T2I+I2I(e) Our Hybrid 38.1 34.1 35.1 36.7 34.5 I2I-EditingT2I-Generation Baseline Ours โAdd a red and yellow striped bait to the floor...โ Real (c)(a) (d)(b) Adaptive Refine Static Label Fig. 1: Mutually Complementary T2I-I2I Data Synthesis. (a) Existing T2I methods (e.g., MosaicFusion [47]) offer diverse scenes but lack label accuracy, while (c) Copy-Paste-based I2I methods (e.g., X-Paste [53] and DiverGen [6]) offer accurate labels but reduce realism. (e) Our hybrid framework combines two proposed modules: (b) a T2I branch for scene diversity with reliable labels via teacher-student adaptive pseudo-labeling, and (d) context-aware I2I edits for realistic, accurate instance-level supervision. This unified approach achieves state-of-the-art results on LVIS benchmark. This data scarcity has motivated recent work on synthetic data generation, leveraging advances in image generative modeling [12, 17, 32, 51] as a scalable alternative to manual annotation for improving long-tailed instance segmen- tation. Current generative data synthesis methods fall into three paradigms: text-to-image (T2I), label-to-image (L2I), and image-to-image (I2I). T2I meth- ods [27, 44, 47], which first generate images from textual prompts and then as- sign semantic labels via pseudo-labeling, can introduce label noise, particularly for rare or fine-grained categories. L2I approaches [19,49,50] synthesize images conditioned on segmentation masks, enabling precise layout control via mask- to-image modules such as ControlNet [51]. However, scaling this approach to large category sets remains challenging, often leading to label-image mismatches and inaccurate annotations. In contrast, I2I approaches [6,24,53] augment real datasets by adding new objects via copy-paste or inpainting, offering accurate annotations for added instances. However, existing paste-based methods often suffer from domain gaps (e.g., illumination mismatches), while inpainting strug- gles with plausible mask placement. Interestingly, when paste-based approaches explicitly target rare classes, performance on those classes often decreases rather than improves [6]. This counterintuitive result suggests that naive pasting often harms contextual realism, leading the segmentation model to overfit to synthetic artifacts rather than learning true representations. To overcome these limitations, we propose a hybrid data synthesis frame- work that strategically integrates T2I generation with context-aware I2I editing TMI3 as illustrated in Fig. 1. Our T2I branch generates diverse images of broad cat- egories and scenes, from which pseudo labels are obtained through an offline labeler. Since offline pseudo labels are prone to noise due to the domain gap between real and generated images, we additionally employ a teacher-student scheme [35] that adapts to the T2I domain and progressively refines pseudo- label quality. However, T2I generation alone is insufficient for rare classes, which remain difficult to label reliably. We address this limitation with our novel I2I branch, VRAIN (Verified Rare-class Augmentation via INstructed editing), to enhance rare-class representation. VRAIN operates as a two-stage pipeline: (i) it first performs context-aware rare-class placement via instruction-based edit- ing to ensure natural scene integration; (i) it then verifies the editโs semantic consistency and visual fidelity, generating a precise, high-fidelity annotation for the successfully integrated instance. This two-stage โplace-and-verifyโ process directly mitigates the contextual inconsistency of copy-paste methods and the mask-placement ambiguity inherent in inpainting-based approaches. By yielding semantically coherent and visually natural edits, this process alleviates domain gaps and compositional artifacts, while also enabling targeted augmentation for rare categories. Experimentally, our approach achieves state-of-the-art results on the LVIS benchmark, significantly outperforming existing paste-based augmentation and T2I baselines. Notably, it demonstrates remarkable effectiveness for both overall and underrepresented categories. Furthermore, our method scales effectively with larger backbones, demonstrating its practicality as a scalable solution for data generation in large-vocabulary instance segmentation. In summary, our key contributions are as follows: โ We propose a unified T2I-I2I data synthesis framework that leverages T2I for scene diversity and I2I editing for enhanced realism and rare-class repre- sentation. โ We design VRAIN, a novel โplace-and-verifyโ I2I pipeline, which leverages instruction-based editing for semantically coherent placement and introduces a verification stage to generate accurate annotations. โ We achieve state-of-the-art performance on the LVIS benchmark, especially on rare categories and demonstrate effective scaling with larger backbone models. 2 Related Work 2.1 Image Generative Models High-quality T2I generation is dominated by diffusion models, which have rapidly evolved from DDPM [12] and latent diffusion [32] to more efficient flow-matching formulations [20,22]. Modern architectures such as Flux [17] and Qwen-Image [42] further boost fidelity and scalability by adopting Diffusion Transformer (DiT) [30] backbones with flow-matching objectives. These T2I advances also empower instruction-based I2I editingโpioneered by InstructPix2Pix [2] and extended by 4H. Song & S. Choi et al. unified frameworks like Flux-Kontext [18] and Qwen-Image-Edit [42]โenabling semantically consistent, language-guided generation and editing. Our dataset synthesis framework is built on these generative and editing developments. 2.2 Data Generation for Segmentation Task Recent advances in generative modeling have led to various data generation strategies for segmentation, broadly categorized as T2I, L2I, and I2I. T2I meth- ods generate images from textual prompts, often using diffusion attention maps [27, 45,47] or fine-tuning perception decoders [44] for pseudo-labeling in few-shot set- tings. For example, MosaicFusion [47] divides the image canvas into mosaic [1]- like regions and simultaneously generates multiple objects. However, most T2I approaches are effective on small-scale data or for a limited number of cate- gories [27,45], struggling to scale to large-scale, in-the-wild scenarios. L2I methods generate images conditioned on segmentation masks, enabling precise control over object layout. Some works [19, 49] utilize mask-to-image modules like ControlNet [48, 51], while others [50] extend this framework by designing text-to-mask modules for mask diversity. However, extending L2I to large-vocabulary datasets such as LVIS [9] remains challenging. Mask-only con- ditioning often fails to faithfully render the semantics of numerous fine-grained categories, leading to label-image mismatches. I2I methods expand datasets by editing existing images. Copy-paste tech- niques, such as X-Paste [53] and DiverGen [6], leverage generative models to create diverse instance pools for composition onto real images. This success- fully covers large categories where L2I methods struggle; however, the pasted instances often appear visually distinct from their surroundings (e.g., illumi- nation or blending mismatches), limiting training gains. Inpainting-based ap- proaches [14,24] synthesize objects into masked regions. While effective in con- strained domains (e.g., road scenes [14,24]), they have not scaled to in-the-wild, large-category datasets like LVIS, due to the challenges of realistic mask place- ment and high semantic diversity of scenes. Our hybrid framework addresses these complementary failures. We harness T2I for diversity but mitigate its label noise by employing prompt-consistent filtering and teacher-student refinement. We then use our I2I branch to address both T2Iโs rare-class weakness and prior I2Iโs realism gap, providing high-fidelity, targeted augmentation that scales effectively on LVIS. 3 Methods We generate synthetic training data through two complementary paradigms: (i) T2I generation for diverse scene synthesis, and (i) I2I editing for context-aware augmentation of real images. The T2I branch (Sec. 3.1) provides broad seman- tic coverage across all categories in the long-tailed dataset. However, assigning accurate annotations remains a challenge; conventional offline models, trained exclusively on real images, produce noisy pseudo-labels due to the domain gap TMI5 Real Dataset ํ ํซํํํฅ ํผ real ,ํด real ํช ํ ~ํผ ํช ํํฅ (ํ) Real Data Distribution Rare Classes ํช ํํํ All Classes ํช ํํฅ Count Complementary Data Generation Gen Counter barge machine gun sparkler ํช ํ ~ํบ ํช ํํํ (ํธ) ํช ํ ํํ ํดํผํ T2I I2I Dataset ํ ํํํ ํผ edit ,ํด edit ํํํํ โ : Add a lit sparkler (ํ โ ) to the grass...โ T2I Dataset ํ ํํํ water gun sparkler flip-flop person smoothie ํถ ํ ํผ ํ ํด ํ EMA Aug. ํด ํญํํํํกํํซ ํด ํฌํญํฎํํํงํญ Model Training offline Aug. ํ ํซํํํฅ ํ ํํํ ํด ํ ํด ํ ํ แ ํด ํ teacher ํผ ํ ํ ํํํ Pseudo-Label Adaptation Filtering Memory ํญ ํญํํฑํญ (โ,ํช ํ ) Iteration = 0Iteration = 100K Pseudo-Label Adaptation Model Training I2I +1 Gen Counter Update Fig. 2: Overview of Hybrid Data Generation and Training Framework. We combine two complementary paradigms: (i) T2I generation for diverse scenes over all categories C all and (i) I2I editing (VRAIN) for rare categories C I2I with accurate instance-level supervision. (i) A student M student jointly learns from both D T2I and D I2I . For D T2I images I m , an EMA teacher M teacher generates refined pseudo-labels ฬ A teacher m , merged with offline labels A P m to form the final supervision A m . As training proceeds, the teacher adapts to the T2I domain, progressively improving initially noisy pseudo-labels (pseudo-label adaptation). For example, rare classes (e.g., sparkler and water gun)โmissed by the offline labeler and initially unlabeledโare gradually labeled correctly as the teacher adapts with the support from rare-class-targeted I2I data. between real and generated images. We therefore treat this T2I set as a source of weakly-labeled data, D T2I . In contrast, our I2I branch (VRAIN, Sec. 3.2) enhances rare-class representation. Because these edits are driven by specific in- structions, the identity and location of the newly inserted instance are explicitly known, yielding accurate, instance-level labeled data (D I2I ) that provide reliable supervision for rare categories where T2I struggles. Our final synthesized dataset is the union of these two complementary sets, D gen = D T2I ,D I2I . Finally, we introduce a teacher-student scheme (Sec. 3.3) to train the student model on this hybrid dataset. In this semi-supervised loop, an EMA-updated teacher gener- ates dynamic, online pseudo-labels for the D T2I . The student is jointly trained on these progressively refined T2I pseudo-labels to expand category and scene di- versity, while also incorporating the labeled I2I data for accurate, instance-level supervision. An overview of the entire process is depicted in Fig. 2. 3.1 Text-to-Image Generation Our T2I method consists of two stages: (i) generating a set of text prompts and corresponding images, and (i) offline pseudo-labeling with text-consistent filtering. First, to generate text prompt set T of size N text , we randomly sample a subset C m of size l from the total category set C all . We then create a descriptive prompt t m composed of these selected classes using GPT-4o [13]: T =t m | t m = GPT-4o(C m ).(1) 6H. Song & S. Choi et al. MethodAP box AP mask AP box r AP mask r (a) Real-only34.5 30.8 24.0 21.6 (b) High-threshold Filtering 35.8 32.0 31.2 28.0 (c) Text-consistent Filtering 36.7 33.0 32.1 28.9 Table 1: Comparison of T2I Pseudo-label Filtering Strategies. We compare models trained with T2I data, where the raw predictions ห A P m are filtered using either (b) a high-threshold strategy or (c) our text-consistent strategy. Our method (c) re- tains semantically valid, low-confidence instances that high-threshold filtering discards, resulting in improved overall and rare-class AP. Subsequently, we generate an image I m from the created text prompt using the image generation model ฮฆ T2I [17] and construct N T2I unlabeled T2I dataset as image-category pairs: D T2I =(I m ,C m )| I m = ฮฆ T2I (t m ) Second, we pre-compute a set of offline labels for this dataset. We process each image I m with a pre-trained public instance segmentation model [57] M P to obtain an initial set of raw predictions, ห A P m = M P (I m ). To leverage the prior knowledge that I m was generated from the prompt t m containing classes C m , we apply a text-consistent filtering process F text . For the k-th predicted instance annotation a P k , this filter retains only the instances whose predicted classes belong to C m : A P m = F text ( ห A P m ,C m ) = n a P k | a P k โ ห A P m , class(a P k )โ C m o . (2) The effectiveness of this text-consistent filtering is validated in Table 1. Our filter- ing method yields superior results compared to high-threshold instance mining. This demonstrates that our filtering strategy, which leverages prompt knowledge, is more effective than relying solely on high-confidence scores. It successfully re- tains valid instances, even those with low-confidence, that are consistent with the text prompt. These filtered offline labels, A P m , are then used as one of the supervision sources in our training pipeline, detailed in Sec. 3.3. 3.2 Image-to-Image Editing While our T2I branch provides broad diversity, it struggles to produce accurate instance-level annotations, especially for rare categories prone to label noise. A high-quality seed set of verified rare-class annotations is therefore essential, as it bootstraps the online pseudo-labeling methods (e.g., EMA teacher [35]). I2I ap- proaches, which can add new objects with accurate annotations, offer a promis- ing solution. However, prior I2I methods suffer from critical flaws: copy-paste techniques produce contextually inconsistent artifacts, while inpainting-based methods struggle to identify suitable mask regions in complex scenes. Motivated by these limitations, we propose VRAIN (Verified Rare-class Augmentation via INstructed editing), illustrated in Fig. 3. VRAIN is a principled two-stage pipeline designed to overcome these realism and placement challenges, perform- ing (1) context-aware rare-class placement via instruction-based editing, and (2) verification of the editโs fidelity to generate trustworthy annotation. TMI7 โAdd a small colorful rag doll(c โ )to the right side of the plate. The rag doll has a friendly face and is made of soft fabric.โ inst โ SSIM Diff. โ Rare-class Proposal Coarse Area Open-vocab Detection Object Mask โขVLM Verification โฃAnnotation Updater โกInstruction- based Editor (ํฝ ํํํ ) c โ ํจ ํํํญ SAM ํฐ ํซํํํฅ ํฐ ํํํขํญ PLACE: Proposal & Synthesis VERIFY: Verification & Annotation ํฐ ํํํขํญ ํํจํฑ ํจ ํํํขํญ ํจ ํซํํํฅ Fig. 3: VRAIN โPlace-and-Verifyโ Pipeline. Our two-stage framework ensures high-fidelity I2I editing. (i) Place: A VLM proposes a semantically coherent instruc- tion (inst โ ) for inserting a rare class (c โ ) that fits naturally within I real . An instruction- based editor ฮฆ I2I then synthesizes I edit . (i) Verify: I edit is then validated. The new instance is localized via SSIM difference and open-vocab detection, semantically con- firmed by a VLM filter, and masked using SAM. Finally, an annotation updater resolves occlusions with A real , integrating the new instance to produce the trustworthy final an- notation A edit . Rare-class Proposal and Synthesis. A central challenge in rare-class gener- ation is selecting a semantically fitting category and determining its placement, ensuring the inserted instances are contextually coherent with the scene. This process is key to reducing the unrealistic compositions of previous copy-paste methods. To this end, we leverage a Vision Language Model (VLM) [56] to ensure semantic consistency, by selecting the most suitable category from a can- didate set for a given image, and then generating its placement instruction. This candidate set (C q ) is formed by sampling Q categories from all target categories C I2I for a given real image I real from the training datasetD real =(I real ,A real ). The sampling is performed using a softmax-based sampler (C q โผ S C I2I (Q)) that is weighted by the generation count of each class (initialized to zero), penalizing frequently generated classes. The VLM is then prompted with I real and the can- didate set C q to recommend a suitable category c โ along with the corresponding textual instruction inst โ : (c โ , inst โ ) = VLM(I real , C q ),(3) where inst โ is the natural language instruction fed to the editor, while c โ is the explicit category label passed to the annotation stage (see the supplementary material for details). Finally, inst โ is used by an instruction-based editing model ฮฆ I2I [18] to produce the edited image: I edit = ฮฆ I2I (I real ,inst โ ). Verification and Annotation. Although the instruction-based editor ฮฆ I2I produces visually plausible results, these edits may suffer from contextual mis- matches or rendering artifacts. To ensure annotation quality, VRAIN performs a 8H. Song & S. Choi et al. verification and annotation stage that evaluates both semantic consistency and visual fidelity of each edited image. First, to locate the edited region, we com- pute a structural similarity (SSIM [41]) difference map between I real and I edit , thresholding at ฯ edit to identify modified pixels. Second, to detect the new ob- ject, we apply an open-vocabulary detector [7] within this coarse area, searching for the intended category c โ and producing a set of candidate detections A det . However, A det may contain false-positives or artifacts. To semantically verify these candidates, we introduce a VLM-based filter, F VLM (A det ,c โ ). For each detection a k โA det , we prepare an image I box edit (a k ) by overlaying its bounding box in red on the edited image. The VLM is then prompted with this visualization along with a template-based question t c โ (e.g., โIs the red bounding box in the image a c โ ?โ). The VLM outputs a binary โYesโ or โNoโ response. A detection a k is confirmed only if the VLM responds โYesโ: F VLM (A det ,c โ ) = n a k | a k โA det , VLM(I box edit ,t c โ ;a k )= Yes o . (4) For all verified objects in F VLM , we obtain instance masks utilizing Segment Anything Model (SAM) [31]. These new masks are then added to the original annotation A real . However, this integration can create occlusions, where new SAM masks overlap with an existing ground-truth mask from A real . Thus, we update in A real by subtracting the new SAM masks from any overlapping original masks to produce the final annotation A edit . If this process yields at least one verified instance mask, the generation count for category c โ is incremented by one. This entire procedure is repeated until the total count for C I2I reaches the target N I2I , resulting in the final dataset D I2I =(I edit ,A edit ) of size N I2I . 3.3 Model Training The straightforward way to train an instance segmentation model [55] using the two complementary synthesized datasets, D gen = D T2I ,D I2I , is to combine them with the real dataset D real for joint training. However, unlike D real or D I2I , all images in D T2I are fully generated, which can lead to domain gaps and unreliable pseudo-labels using a static labeler trained only on real data. To address this issue, we adopt an EMA-based teacher-student scheme [35], where the student model M student learns from supervision signals generated by the teacher model M teacher . Its parameters are updated as ฮ teacher โ ฮณฮ teacher + (1โฮณ)ฮ student , where ฮ teacher and ฮ student denote the parameters of the teacher and student models, respectively, and ฮณ is the EMA decay rate. The teacher gradually adapts to the T2I domain during training, thereby producing more accurate pseudo-labels for T2I data. Moreover, because the teacher is also trained on the high-quality, rare-class-focused D I2I , it becomes more adept at assigning more precise labels even for rare categories in D T2I . Specifically, at each training step, a batch is sampled fromD real ,D I2I ,D T2I . For images in D real and D I2I , the student is trained directly using their ground- truth annotations (A real and A edit ). On the other hand, for images in D T2I , the TMI9 student is trained on hybrid pseudo-labels created by processing and merging the offline (A P m ) and new online (A teacher m ) labels generated by the teacher. This training-time process for online labels is as follows. First, the online EMA-based teacher M teacher produces raw predictions ห A teacher m , which are then split into two groups according to their predicted class หc and the sampled cate- gory set C m . Prompt-consistent instances (หcโ C m ) are filtered using a class-wise adaptive threshold. This threshold is dynamic; we maintain a per-category run- ning memory of recent confidences, and the threshold for a given class is set to ฯ label times the mean confidence of that memory, allowing the filter to adapt as the teacher improves. Prompt-inconsistent instances (หc /โ C m ) are assigned an โunlabeledโ class if they are detected with high confidence above ฯ unlabel times the mean confidence. These โunlabeledโ instances are excluded only from the classification loss but are still included in box and mask losses [55], pro- viding implicit localization supervision from objects that were confidently de- tected but not explicitly specified in the prompt. These two sets of processed labels (the adaptively-thresholded and the unlabeled instances) together form the complete online pseudo-label set, ฬ A teacher m . Finally, we merge this dynamic online set ฬ A teacher m with the static offline pseudo-labels A P m via IoU-based match- ing to produce the studentโs final supervision, A m . This strategy combines their complementary strengths: the offline set A P m provides a stable, precision-oriented baseline of high-confidence instances with conservative coverage of common cat- egories, while the online set ฬ A teacher m introduces progressively refined, domain- adapted labels, especially for rare classes thanks to the auxiliary guidance from I2I data. The student M student thus learns from both a robust baseline and dy- namic, in-domain corrections. 4 Experiments 4.1 Experimental Settings Dataset. We conducted experiments on LVIS [9], a large-scale instance segmen- tation dataset featuring a long-tailed distribution of 1,203 categories. It contains 100K training and 20K validation images with 2M annotations. Based on the number of training images per category, categories are divided into three fre- quency groups: rare (1โ10 images; 337 categories), common (11โ100 images; 461 categories), and frequent (>100 images; 405 categories). Baselines. We compare our method against three baselines: MosaicFusion [47] (a T2I method), X-Paste [53], and DiverGen [6] (I2I copy-paste methods). We use CenterNet2 [55], a widely used model for large-vocabulary instance segmen- tation, for all our experiments, with two backbone configurations: ResNet-50 [10] (640 resolution, batch size 32) and Swin-L [23] (896 resolution, batch size 16). Following the convention of prior work [6,53], all models are trained for 180K it- erations, initializing from their respective โReal-onlyโ pretrained checkpoints. To ensure a fair comparison under identical experimental settings, we reproduced 10H. Song & S. Choi et al. Backbone MethodT2I Dataset I2I Dataset AP box AP mask AP box r AP mask r ResNet-50 Real-only [55] --34.5 30.8 24.0 21.6 ResNet-50 MosaicFusion [47] โ34.1 30.4 24.4 22.5 ResNet-50 DiverGen [6] โ35.1 31.2 25.6 23.8 ResNet-50 X-Paste [53] โ36.7 33.0 29.6 27.8 ResNet-50 Ours โ38.1 34.0 33.9 31.7 Swin-L Real-only [55] --47.5 42.3 41.4 36.8 Swin-L MosaicFusion [47] โ47.7 42.8 41.3 37.5 Swin-L DiverGen [6] โ49.6 44.2 44.5 39.8 Swin-L X-Paste [53] โ50.1 44.4 48.2 43.3 Swin-L Ours โ50.7 45.2 49.1 44.0 Table 2: Comparison with Baselines on LVIS validation Dataset. Our hybrid T2I-I2I approach consistently outperforms all baselines: the Real-only model, the T2I- only method (MosaicFusion), and copy-paste I2I methods (DiverGen, X-Paste). This superior performance holds across both ResNet-50 and Swin-L backbones, delivering particularly strong gains on rare-class metrics and demonstrating the effectiveness of combining both data synthesis paradigms. DiverGen using their official code, generating 1K synthetic instance pool per category (1.2M in total) with two generative models, and also reproduced Mo- saicFusion following its original implementation. For the evaluation metrics, we evaluate performance using LVIS box average precision AP box and mask average precision AP mask . We also report rare-category results AP box r and AP mask r . Implementation Details. We generated 200K T2I images (N T2I = 200K) using Flux.1-dev [17] from a set of 40K prompts (N text = 40K), where each prompt was created by randomly sampling l โ [5, 10] categories from the total LVIS category set. We also generated 80K I2I images (N I2I = 80K) from the LVIS training set using Flux.1 Kontext-dev [18], targeting rare LVIS categories with Q = 5 and an edit-region threshold ฯ edit = 5. We set the EMA decay rate to ฮณ = 0.999, the same as the baselines, but in our framework it is used to adaptively refine pseudo-labels rather than just for evaluation. We set ฯ label = 0.7 and ฯ unlabel = 1.2 for dynamic threshold in online pseudo-labeling with running memory size 100 per class. We used InternVL3-14B [56] as the VLM for VRAINโs recommendation and verification steps. Further details on VLM implementation including prompt creation are provided in the supplementary material. 4.2 Comparison We evaluated our approach against various data synthesis methods, as shown in Table 2. MosaicFusion, a rare-class-targeted T2I method, offers only marginal gains over the real-only model, indicating that despite its scene diversity, the limited label accuracy of T2I generation constrains performance on long-tailed datasets. Copy-paste I2I methods like DiverGen and X-Paste provide more ac- curate instance-level labels and achieve larger improvements, yet still lag behind our hybrid T2I-I2I framework. Our method uniquely combines the strengths of TMI11 training iteration ํผ ํ แ ํด ํ ํกํํํโํํ (a) "A toddler holding a teddy bear sits on a bathroom floor near slippers, a toilet in the background, and a quesadilla on a plate beside them, warm indoor lighting, candid photography style." แ ํด ํ ํกํํํโํํ (b) "A kitchen counter with a spice rack beside a rolling pin and strawberries, a screwdriver and telephoto lens near a beanie, traffic light visible through the window, warm natural lighting, professional photography." training iteration ํผ ํ แ ํด ํ ํกํํํโํํ แ ํด ํ ํกํํํโํํ Fig. 4: Examples of Pseudo-label Adaptation in T2I images. In each T2I prompt, boldfaced and underlined words indicate the target categories for generation. As the teacher model updates, it adapts to the T2I domain and enhances pseudo-label ( ฬ A teacher m ) qualityโ(a) improving the toddler mask and labeling previously missed ob- jects (e.g., quesadilla), and (b) refining masks (e.g., spice rack, rolling pin) while newly labeling a telephoto lens. MethodAP box AP mask AP box r AP mask r (a) Real-only47.5 42.3 41.4 36.8 (b) MosaicFusion [47]47.7 42.8 41.3 37.5 (c) DiverGen (rare-target) [6] 47.3 42.2 34.5 31.2 (d) D I2I only (VRAIN)48.1 43.3 42.9 39.5 Table 3: Comparison with Rare-class-targeted Data Synthesis Approaches. Among methods generating data for rare categories, (c) DiverGen shows decreased rare- class performance due to reduced realism, whereas (d) our I2I approach effectively and practically improves rare-class performance with high visual fidelity, accurate labels. both paradigms. It leverages T2I for broad diversity like MosaicFusion, but uses our teacher-student adaptive pseudo-labeling to ensure high quality label. By further incorporating VRAIN (I2I) to inject rare-class data with precise instance- level labels, it enhances performance on underrepresented categories and boot- straps online pseudo-labeling, boosting overall accuracy. Fig. 4 demonstrates the EMA teacherโs progressive improvement of pseudo-labels on T2I images during trainingโincluding the detection and mask refinement of previously missed rare- class objects such as quesadilla and telephoto lensโeffectively bootstrapping the online pseudo-labeling process. This synergy achieves the largest gains across all categories and especially for rare classes, improving AP box from 34.5โ 38.1 (ResNet-50) and 47.5โ 50.7 (Swin-L) overall, and AP box r from 24.0โ 33.9 (ResNet-50) and 41.4โ 49.1 (Swin-L) for rare categories. To further analyze rare-class-focused strategies, we compare approaches that synthesize data exclusively for rare categories, as shown in Table 3. Although designed for rare-class enhancement, MosaicFusion again provides a marginal improvement over the real-only model, indicating that the limited label reliabil- ity of T2I generation constrains its effectiveness. Notably, the rare-class-focused variant of DiverGen catastrophically underperforms the real-only baseline on rare categories (AP box r 41.4โ 34.5). This contradictory result highlights that naive copy-pasting harms performance due to its poor contextual coherence and realism, encouraging overfitting to pasted instances rather than learning true representations. In contrast, our rare-class-focused I2I approach (VRAIN) suc- cessfully boosts rare-class performance, benefiting from high visual fidelity and 12H. Song & S. Choi et al. ํผ ํ ํด ํ ํ (a) "A backyard scene with a garden hose coiled near a wok on a patio table, a dove perched nearby, potatoes in a basket, and money scattered on the ground, soft natural daylight photography." แ ํด ํ teacher โํด ํ ํ (b) "A coastal souvenir shop with a walrus statue near a signboard, pottery on shelves, a fire extinguisher on the wall, a puppet hanging from the cornice, warm afternoon light, casual photography." ํผ ํ ํด ํ ํ แ ํด ํ teacher โํด ํ ํ Fig. 5: Examples of Final Pseudo-labels for T2I images. In each T2I prompt, boldfaced and underlined words indicate the target categories for generation. Teacher- generated pseudo-labels (red) are merged (โ) with offline pseudo-labels (blue) to form the final supervision. As the teacher adapts to the T2I domain, it successfully labels rare classesโ(a) dove and (b) walrus. MethodAP box AP mask AP box r AP mask r (a) Real-only47.5 42.3 41.4 36.8 (b) M teacher only49.4 43.9 47.9 43.0 (c) M P only49.9 44.9 45.3 41.7 (d) M teacher and M P 50.3 45.1 47.5 43.8 Table 4: Ablation Study on T2I Pseudo-label Supervision. We evaluate the impact of different pseudo-label sources: offline pseudo labeler (M P ), EMA teacher (M teacher ), and their combination. The results show that M P provides stable super- vision for common categories, M teacher adaptively improves rare-class coverage, and combining both leverages complementary strengths. precise instance-level labeling. These results demonstrate the effectiveness and practical applicability of our method for improving underrepresented categories. 4.3 Ablation Study Effect of EMA-Teacher Labeling. To evaluate the effectiveness of our EMA- teacher framework, we conducted an ablation study on T2I data labeling, as shown in Table 4. Specifically, we trained separate instance segmentation mod- els using either the offline pseudo labels (A P m ) or the EMA teacher-generated labels ( ฬ A teacher m ) as supervision. The model trained with A P m achieves relatively higher AP on common categories, reflecting the stable supervision provided by the offline pseudo labels. In contrast, the model trained with ฬ A teacher m leverages pseudo labels that are progressively updated by the EMA teacher throughout training, adaptively correcting errors and improving AP on underrepresented rare classes. Combining both M P and M teacher leverages their complementary strengths, maintaining high accuracy for common categories while improving rare-class coverage, as visually illustrated in Fig. 5. Reliability of VRAINโs Verification and Annotation. To quantitatively assess the reliability of our VLM-based verification module in VRAIN, we con- struct a controlled perturbation benchmark from the LVIS validation set (244k instances). We generate negative samples by randomly perturbing instance class TMI13 "Addablackheadset (ํ โ )totheskier'shead. The headsethasasleekdesignand ispositionedovertheskier'sears." "Addablackheadset (ํ โ )totheheadof thedog. The headsethasearcupsand amicrophone." (a) Filtered-out Case(b) Filtered-in Case ํผ real ํผ edit ํผ real ํผ edit Fig. 6: Qualitative Examples of the VRAIN Verification Stage. (a) A โhelmetโ is added instead of the requested โheadsetโ (c โ ), creating a semantic mismatch, and is thus filtered out; (b) The target โheadsetโ is correctly added, passing the verification. MethodPrecision Recall FP Rate FN Rate (a) CLIP (Crop)0.886 0.623 0.1870.377 (b) VLM (Crop)0.975 0.521 0.0310.478 (c) VLM (Ours)0.984 0.846 0.033 0.154 Table 5: VLM Verification Reliability on LVIS Validation. Compared to prior CLIP-based verification, our spatially grounded VLM verifier achieves a superior pre- cisionโrecall trade-off with lower FP and FN rates. MethodAP box AP mask AP box r AP mask r (a) Real-only47.5 42.3 41.4 36.8 (b) D I2I only (w/o verification) 47.4 42.5 39.5 36.0 (c) D I2I only (VRAIN)48.1 43.3 42.9 39.5 Table 6: Ablation Study on VLM Verification in VRAIN. We compare I2I data with and without VLM-based filtering. Removing the verification step (b) reduces both overall and rare-class AP, indicating that the VLM filter is crucial for retaining semantically correct and relevant instances. labels with 30% probability while preserving the original images and bounding boxes. We measure false positive (FP) rate when the verifier incorrectly accepts perturbed labels and false negative (FN) rate when it rejects correct ones. We compare three verification strategies: (a) CLIP similarity on crops (โa photo of [class]โ), (b) crop-based VLM verification (โIs the image a [class]?โ), and (c) our spatially grounded red-box VLM (โIs the red bounding box a [class]?โ). As shown in Table 5, the VLM verifier significantly reduces the FP rate compared to CLIP (0.187 โ 0.033). Moreover, explicitly providing spatial grounding significantly reduces the FN rate (0.478 โ 0.154) compared to the crop-based VLM, achiev- ing a balanced trade-off between precision and recall. We further evaluate the impact of the verification step by generating I2I data without applying VLM filtering. As shown in Table 6-(b), generating I2I data without VLM verification degrades both overall and rare-class performance, highlighting the importance of semantic consistency filtering for reliable synthetic supervision. Qualitative examples of filtered and retained samples are shown in Fig. 6. Beyond semantic verification, we validate the mask annotation quality of VRAIN using the ORIDa benchmark [15], which contains 108k real before/after image pairs with physically placed objects and ground-truth annotations. By treating the before and after images as I real and I edit respectively, we apply 14H. Song & S. Choi et al. MethodAP box AP mask AP box r AP mask r (a) Real-only47.5 42.3 41.4 36.8 (b) w/o D I2I 50.3 45.1 47.5 43.8 (c) Ours50.7 45.2 49.1 44.0 Table 7: Ablation Study on I2I Dataset. Excluding D I2I lowers overall and rare- class performance, emphasizing the value of high-fidelity, rare-class-targeted I2I data that collaboratively complements T2I supervision. our annotation pipeline and compare the resulting masks against the ground- truth. On this benchmark, our method achieves 0.925 mAP on mask annotations, demonstrating high annotation fidelity comparable to human-level quality [9,36]. Contribution of the I2I Data. Finally, we assessed the overall contribution of the generated high-fidelity I2I data (D I2I ) to the hybrid training framework. As shown in Table 7, excluding D I2I leads to a noticeable performance drop, particularly for rare categories (AP box r 49.1โ 47.5, AP mask r 44.0โ 43.8). This confirms thatD I2I successfully provides targeted rare-class coverage that comple- ments the T2I data. By integrating these reliable, instance-level annotations, the model learns more robust representations for underrepresented classes, further boosting overall accuracy. 5 Conclusion We present a unified data synthesis framework that bridges T2I generation and context-aware I2I editing for large-vocabulary instance segmentation. Our ap- proach effectively couples the diversity and scalability of T2I synthesis with the realism and precise rare-class supervision of I2I editing. Through prompt- consistent filtering and teacher-student refinement, the T2I branch achieves re- liable label quality despite domain gaps, while the proposed VRAIN framework provides high-fidelity, contextually coherent rare-class augmentation that rein- forces the teacher-student learning process. Experiments on LVIS show that this hybrid strategy surpasses existing T2I and paste-based baselines, particularly on rare categories, and scales favorably with larger backbones. We believe this integration of complementary generative paradigms offers a promising approach to boosting model performance across diverse and underrepresented categories in long-tailed instance segmentation. Limitation. Our hybrid approach relies on off-the-shelf generative models, which do not perfectly adhere to input instructions. For example, the T2I branch uses prompt-consistent filtering to retain relevant instances, but it may still fail to realize certain categories specified in the prompt. Similarly, the I2I can ignore positional instructions and place objects incorrectly. Although these approaches help mitigate label noise and contextual inconsistency, achieving perfect fidelity and precise instance-level control across all synthetic data remains a persistent challenge. TMI15 Supplementary Material of TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation A Additional Experiments A.1 Ablation Study on Synthetic Data Scale and Composition We analyze the effect of the scale and composition of synthetic data through an ablation study (Table A). First, synthetic data consistently improves perfor- mance over the real-only baseline (a), confirming the benefit of augmenting real images with our synthetic samples (bโe). Second, increasing the scale of synthetic data enhances performance: using the full T2I and I2I sets (e) outperforms the 50% subset (d), demonstrating favorable scaling of the synthetic data. Finally, combining the two synthetic branches yields the best model. The I2I branch contributes rare-focused target selection, high-quality instance-level labels, and context-aware realism, while the T2I branch, annotated via our combined of- fline and online adaptive labeling pipeline, provides broad category coverage and high-quality pseudo labels. Together, this combination boosts both overall and rare-class performance over each individual branch (b) and (c) alone. Method# T2I # I2I AP box AP mask AP box r AP mask r (a) Real-only0%0% 47.5 42.3 41.4 36.8 (b) T2I-only100% 0% 50.3 45.1 47.5 43.8 (c) I2I-only0%100% 48.1 43.3 42.9 39.5 (d) T2I + I2I (50%)50% 50% 50.2 45.0 47.5 43.1 (e) T2I + I2I (100%; Ours) 100% 100% 50.7 45.2 49.1 44.0 Table A: Ablation Study on Synthetic Data Scale and Composition. Synthetic data improves performance over the real-only baseline (a), and larger amounts yield further gains (d โ e). Adding I2I to T2I-only (b โ e) boosts rare-class AP, while adding T2I to I2I-only (c โ e) benefits from our merged offline-online T2I supervision. 16H. Song & S. Choi et al. A.2 Analysis on Progressive Teacher Adaptation We introduce an EMA-based teacher-student framework combined with text- consistent filtering to progressively refine the pseudo labels on T2I data. To assess the effectiveness of our design, we focus on two key aspects: (1) progressive, adaptive pseudo labels generated by the EMA teacher, and (2) the use of prompt- inconsistent predictions as unlabeled instances. Does the teacher effectively adapt to the T2I domain, and is it impor- tant? In our training strategy for the T2I dataset, the teacher model M teacher is updated via EMA and gradually adapts to the T2I domain while being guided by rare-class-focused I2I supervision, producing online pseudo labels ฬ A teacher m . To verify the effectiveness of this progressive adaptation, we compare it against a control variant where the teacherโs pseudo labels ฬ A teacher m on T2I data are frozen at initialization. All other architecture and training hyperparameters are kept identical; the only difference is whether T2I pseudo labels are progressive or static. As shown in Table B, the fixed-label variant consistently underper- forms our progressive teacher across all AP metrics, with particularly pronounced degradation on rare categories. This demonstrates that progressively adapting the EMA teacher to the T2I domain and refining pseudo labels over training is crucial for improving performance, particularly on underrepresented categories. MethodAP box AP mask AP box r AP mask r (a) Real-only47.5 42.3 41.4 36.8 (b) ฬ A teacher m (frozen)48.8 43.8 45.5 41.2 (c) ฬ A teacher m (progressive)49.4 43.9 47.9 43.0 Table B: Experiments on the Effectiveness of Adaptive Pseudo Label in T2I Dataset. We compare static T2I pseudo labels frozen at initialization (fixed- label) versus refreshed pseudo labels from the EMA-updated teacher (progressive). Refreshing pseudo labels over training yields consistent AP gains, especially on rare categories. Is it beneficial to treat high-confident prompt-inconsistent predictions as unlabeled? We analyze the role of unlabeled instances in our T2I training setup. Recall that we sample a category set C m from the long-tailed taxonomy to construct the text prompt, but the synthesized T2I images often include additional instances that are not explicitly specified in the prompt. Our method applies text-consistent filtering by utilizing the set C m , keeping only prompt- consistent instances whose predicted class belongs to C m . While this improves the precision of T2I pseudo labels, it also removes many visually present objects outside C m , leading to a sparsely annotated object detection (SAOD) [28, 39] scenario where frequently occurring objects can remain unannotated and are treated as background during training. TMI17 To alleviate this issue, we reuse predictions that are high-confidence yet in- consistent with the prompt as unlabeled instances. These instances are excluded from the classification loss but included in the box and mask losses [55], pro- viding pure localization supervision that encourages the model to recognize that โthere is an object hereโ without enforcing a potentially incorrect class label. To verify the effectiveness of this approach, we compare two variants against our method on T2I data, as reported in Table C: (b) no-unlabeled variant that dis- cards all prompt-inconsistent predictions and relies only on prompt-consistent labels, (c) label-as-is variant that treats high-score but prompt-inconsistent pre- dictions as labeled instances with the teacherโs predicted class, and (d) our method, which instead treats high-score but prompt-inconsistent predictions as unlabeled instances. The no-unlabeled variant (b) suffers a noticeable drop in overall AP, with larger gaps on categories that frequently appear but are not prompted, highlighting that simply ignoring prompt-inconsistent predictions is ineffective under a sparsely annotated regime. The label-as-is variant, while pro- viding additional supervision, underperforms our method and even falls below the no-unlabeled variant, indicating that directly treating prompt-inconsistent predictions as labeled instances is sensitive to label noise from incorrect class as- signments, underscoring the necessity of prompt-consistent filtering. In contrast, our unlabeled instance strategy achieves the best performance among the three, demonstrating that combining the prompt-consistent filtering for precise T2I la- bels with the use of high-confidence prompt-inconsistent predictions solely for localization supervision is a robust and effective choice for handling the sparsely annotated nature of T2I data. MethodAP box AP mask AP box r AP mask r (a) Real-only47.5 42.3 41.4 36.8 (b) M teacher only (w/o unlabel)47.3 42.4 44.8 41.1 (c) M teacher only (unlabel โ label) 46.8 41.8 42.2 39.1 (d) M teacher only49.4 43.9 47.9 43.0 Table C: Experiments on the Effectiveness of Unlabeled Pseudo Labels. Treating high-confidence prompt-inconsistent predictions as unlabeled (d) improves both overall and rare-class AP. Discarding them (b) or treating them as labeled in- stances (c) leads to lower performance, highlighting the importance of unlabeled su- pervision in sparsely annotated T2I data. 18H. Song & S. Choi et al. B Implementation Details B.1 T2I Generation Textual Prompt Set Generation. To generate the textual prompt set T of size N text , we create each text prompt by randomly selecting l categories from the total category set C all , where l is chosen independently for each prompt in the range 5โ10 to ensure diverse compositions. These selected classes form the set C m for that prompt, and are provided to GPT-4o [13] using the following instruction template: Create a natural , concise image generation prompt using the selected object classes. The prompt should describe a realistic scene where these objects naturally coexist. Selected classes: selected_classes CRITICAL REQUIREMENTS: 1. MANDATORY: Include EVERY SINGLE selected object class in the prompt - NO EXCEPTIONS 2. Each object must be explicitly mentioned by name in the final prompt 3. Create a natural scene description (not just object listing) 4. Keep total prompt under 40 words 5. Don โt specify colors - let AI choose naturally 6. Focus on realistic spatial relationships and context 7. End with simple lighting/photography style VERIFICATION CHECKLIST - Before submitting , ensure: - Every object from the selected classes list appears in the prompt - No object is missing or omitted - All objects are mentioned by their exact names STRUCTURE: - Describe a realistic scene where these objects belong together - Mention spatial relationships (on , near , beside , etc.) - Brief setting and atmosphere - ENSURE ALL OBJECTS ARE INCLUDED Example formats: "A kitchen scene with ceramic bowl on wooden cutting board beside leather handbag and steel refrigerator , warm natural lighting , professional photography" "Living room interior featuring armchair near coffee table with lamp and books , cozy atmosphere , soft lighting" "Garden setting with watering can beside flower pot and gardening tools on wooden bench , natural daylight" IMPORTANT: If you cannot fit all objects naturally in 40 words , prioritize including ALL objects over perfect grammar. Please provide your response in the following JSON format: โjson "prompt ": "your scene description (max 40 words , MUST include ALL objects)" TMI19 As an illustrative example, the randomly selected classes for a prompt may be gorilla, lemon, beer bottle, pocketknife, carton, napkin/table napkin/servi- ette, spatula, lollipop, and swimsuit/swimwear/bathing suit/swimming costume/- bathing costume/swimming trunks/bathing trunks. Given thisselected_classes, GPT-4o generates a textual prompt such as โA picnic scene with a gorilla beside a carton holding lemons, a beer bottle, a lollipop, and a pocketknife on a napkin, with a spatula and swimwear nearby, under soft natural daylightโ, which is a natural scene description that includes all selected object classes. The prompt is then used by the T2I model ฮฆ T2I [17] to generate the corresponding image I m . In practice, we generate five images per textual prompt to increase visual diversity. Fig. A shows representative examples of our T2I images created from the textual prompt set. โA picnic scene with a gorilla beside a carton holding lemons, a beer bottle, a lollipop, and a pocketknife on a napkin, with a spatula and swimwear nearby, under soft natural daylight.โ "A cozy kitchen with a polar bear plush near an alarm clock and calendar on the counter, kiwi fruit and sausage on a plate, flower arrangement by the window, taxi visible through glass, soft daylight photography." โA backyard party with a dalmatian wearing a dog collar near a tote bag, cowbell hanging on a trunk, sparkler fireworks lit beside a beachball, cigar box on table, drill and projector nearby, evening photography.โ โA picnic scene with a wok on a blanket, thermos bottle nearby, penny coin and poker chip scattered beside, sparkler fireworks held in hand, sunset atmosphere, soft golden-hour lighting, realistic photography.โ โA beachside holiday home featuring a Christmas tree near an automatic washer, a surfboard leaning on the wall, a brass plaque above, a lanyard hanging nearby, and cooking utensils on the countertop, soft natural lighting.โ Fig. A: Examples of T2I Images. In each T2I prompt, boldfaced and underlined words indicate the target categories for generation. 20H. Song & S. Choi et al. B.2 I2I Editing Rare-class Proposal. In the rare-class proposal stage, we identify semantically suitable categories and determine their placement within a given real image. For each image I real , we sample a candidate set C q of Q classes from target rare categories C I2I using a softmax-based sampler weighted by previous generation counts. Specifically, the sampling weight w(c) for each category cโ C I2I is com- puted as w(c) = exp(โcount(c)), where count(c) is the total generation count for c. The probability of selecting c is P(c) = w(c)/ P c โฒ โC I2I w(c โฒ ), and we draw Q classes according to P(c) to form the candidate set C q . This procedure en- courages balanced augmentation across rare classes. Once C q is obtained, a Vision-Language Model (VLM) [56] is prompted using a natural language template to select the most semantically appropriate category c โ and generate a corresponding instruction inst โ . The prompt template is as follows: I want to inpaint a semantically natural object into the provided image. Please choose which of the following Q classes is most appropriate for pasting. It is important to exclude any objects already present in the image. Additionally , please provide an instruction in the format โAdd <attribute_phrase> <class_name> to <location>. <extra_sentences>โ to inpaint the selected class into the provided image. Candidates: class_name_str Answer using the following format without any explanation: Objects: List the names of the objects in the provided image. Appropriate class: <number>. <class_name> Instruction: an instruction in the format โAdd <attribute_phrase> <class_name> to <location>. <extra_ sentences>โ - <class_name>: the selected class. - <attribute_phrase>: an attribute phrase (color , shape , brand , etc.). - <location>: the location where the object should be added. - <extra_sentences>: 1--2 sentences describing additional characteristics of the object (e.g., color , material , distinguishing features). Examples: "Add a red chair to the left side of the room. The chair has wooden legs and a cushioned seat." "Add a blue backpack to the floor. The backpack has multiple compartments and padded straps ." "Add a yellow folded umbrella near the child on the right side of the image. The umbrella has a curved handle and a cartoon character print ." Here, we set Q = 5 candidate classes per image, and class_name_str enumerates these classes with numbered labels for clarity. For instance, for the real image I real in the first row of Fig. G, where a man is skiing downhill, sup- TMI21 pose the candidate set C q contains the following five classes: garbage, gondo- la/gondola boat, comic book, sparkler/sparkler fireworks, and spear/lance. Then, class_name_str would list them as: 1. garbage 2. gondola/gondola boat 3. comic book 4. sparkler/sparkler fireworks 5. spear/lance From this set, the VLM selects the most semantically appropriate class c โ and generates a corresponding natural language instruction inst โ , which is then used by the instruction-based editor to inpaint the object into the real image. For example, the VLM may choose comic book as c โ and produce the instruction โAdd a colorful comic book to the skierโs left hand. The comic book has vibrant illustrations and text.โ, which guides the editor to accurately place and render the object in the scene. Following this instruction, the instruction-based editor produces the final edited image I edit , showing the skier holding the inpainted comic book with coherent appearance and geometry. Verification. In the verification stage, we aim to ensure that each edited object is both semantically correct and visually accurate. Candidate detections from the edited image are first localized using a structural similarity (SSIM [41]) difference map or an open-vocabulary detector [7], producing a set of potential objects. To semantically verify these candidates, we employ a VLM [56] that is prompted with a visualization of the edited image where the candidate bounding box is overlaid in red (see Fig. 6 in the main text for visualization examples). The VLM is asked whether the object within the red box corresponds to the intended category using the following template: Is the red bounding box in the image a class_name? Answer in the following format without further explanation: "Answer ": Answer here with "Yes" if the red bounding box is a class_name, "No" otherwise. Here,class_name is c โ . Only candidates that receive a โYesโ from the VLM are retained, and instance masks are then obtained for these verified objects using SAM. This ensures that only semantically consistent objects are added to the final annotation. 22H. Song & S. Choi et al. C Computational Costs We analyze the computational cost of our data generation pipeline on a single NVIDIA H100 GPU. T2I generation takes 9.2s per sample, including 8s for image synthesis and 1.2s for annotation. I2I generation requires 20.6s per sample, consisting of rare-class proposal (1.96s), image editing (18s), and annotation with verification (0.64s). Notably, the verification overhead accounts for less than 3.5% of the total I2I time, indicating that the proposed โplace-and-verifyโ design does not introduce a significant computational bottleneck. Since this process is performed once offline for dataset construction, it remains substantially more scalable than manual annotation. D Failure Modes and Statistics in VRAIN We visualize representative failure modes and filtering statistics in Fig. B. The rejected samples are categorized into four distinct types. Structural anomalies are filtered by SSIM: (a) no visible change where the editor fails to alter the image, and (b) excessive edits that disrupt the original scene geometry. Object class errors are subsequently filtered by two-tage semantic verification process: (c) instances that fail to be localized by the open-vocabulary detector searching for the target class, and (d) fine-grained semantic mismatches further rejected by the VLM (e.g., synthesizing a phone instead of a phonebook). Fig. B-(b) illustrates an infeasible proposal where the VLM suggested โAdd a tambourine to the tableโ in a scene without a visible table. This led the editor to hallucinate a table, which was subsequently rejected by the SSIM-based structural consistency check. Although the overall acceptance rate is 52%, this strict filtering strategy prioritizes quality over quantity, ensuring high-fidelity and context-consistent synthetic instances. Such a quality-first design is critical for bootstrapping T2I training, enabling more reliable learning of rare categories. (a) No Change (3%)(b) Excessive (14%)(c) O.V. Filtered (8%) ํผ edit ํผ real (d) VLM Filtered (23%) ํผ edit ํผ real ํผ edit ํผ real ํผ edit ํผ real Fig. B: VRAIN Filtering Visualization and Statistics. Representative failure cases and the rejection rate of each verification filter stage. The instructions are: (a) โAdd a silver file (file tool) to the shelf below the television. The file has a metallic finish and is commonly used for smoothing surfaces.โ; (b) โAdd a wooden tambourine to the table in front of the man. The tambourine has a circular shape with jingle bells inside and a light brown color.โ (the table is a hallucinated target location); (c) โAdd a red unicycle to the left side of the ramp. The unicycle has a black seat and silver spokes.โ; (d) โAdd a white phonebook to the left side of the sink. The phonebook has a rectangular shape and is similar in color to the bathroom tiles.โ TMI23 E More Results E.1 T2I Dataset Analysis We provide more examples illustrating the evolution of teacher-generated pseudo labels on T2I images in Fig. C and D. The first column shows the T2I image, and the following three columns depict the teacherโs predictions at early, intermedi- ate, and late training stages, respectively, revealing the progressive refinement achieved via our EMA-based adaptation. We also present the constructed final supervision for T2I data in Fig. E and F. The 1st and 4th columns display the T2I images, the 2nd and 5th columns show their offline pseudo labels, and the 3rd and 6th columns visualize the final merged labels, which combine the offline labels with the teacherโs refined predictions. These examples highlight how both sources jointly contribute to producing reliable supervision for training on T2I images. E.2 I2I Dataset Examples We present additional examples of data generated by our I2I editing pipeline. Each row consists of four columns: the original image I real , the original an- notation A real , the edited image I edit (produced by the instruction-based editor ฮฆ I2I ), and the final edited annotation A edit (obtained after VLM verification and SAM mask integration). Examples across multiple scenes are shown in Fig. Gโ L, illustrating that VRAIN inserts rare-class objects naturally while preserving annotation quality. 24H. Song & S. Choi et al. training iteration ํผ ํ แ ํด ํ ํกํํํโํํ แ ํด ํ ํกํํํโํํ Fig. C: Examples of Pseudo-label Adaptation in T2I images. TMI25 training iteration ํผ ํ แ ํด ํ ํกํํํโํํ แ ํด ํ ํกํํํโํํ Fig. D: Examples of Pseudo-label Adaptation in T2I images. 26H. Song & S. Choi et al. ํผ ํ ํด ํ ํ แ ํด ํ teacher โํด ํ ํ ํผ ํ ํด ํ ํ แ ํด ํ teacher โํด ํ ํ Fig. E: Examples of Final Pseudo-labels for T2I images. Teacher-generated pseudo-labels (red) are merged (โ) with offline pseudo-labels (blue) to form the final supervision. TMI27 ํผ ํ ํด ํ ํ แ ํด ํ teacher โํด ํ ํ ํผ ํ ํด ํ ํ แ ํด ํ teacher โํด ํ ํ Fig. F: Examples of Final Pseudo-labels for T2I images. Teacher-generated pseudo-labels (red) are merged (โ) with offline pseudo-labels (blue) to form the final supervision. 28H. Song & S. Choi et al. ํด edit ํผ real ํด real ํผ edit Fig. G: Examples of I2I Dataset. TMI29 ํด edit ํผ real ํด real ํผ edit Fig. H: Examples of I2I Dataset. 30H. Song & S. Choi et al. ํด edit ํผ real ํด real ํผ edit Fig. I: Examples of I2I Dataset. TMI31 ํด edit ํผ real ํด real ํผ edit Fig. J: Examples of I2I Dataset. 32H. Song & S. Choi et al. ํด edit ํผ real ํด real ํผ edit Fig. K: Examples of I2I Dataset. TMI33 ํด edit ํผ real ํด real ํผ edit Fig. L: Examples of I2I Dataset. 34H. Song & S. Choi et al. References 1. Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020) 2. Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 18392โ18402 (2023) 3. Chang, N., Yu, Z., Wang, Y.X., Anandkumar, A., Fidler, S., Alvarez, J.M.: Image- level or object-level? a tale of two resampling strategies for long-tailed detection. In: International conference on machine learning. p. 1463โ1472. PMLR (2021) 4. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 5. De Brabandere, B., Neven, D., Van Gool, L.: Semantic instance segmentation for autonomous driving. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). p. 478โ480 (2017). https://doi.org/10. 1109/CVPRW.2017.66 6. Fan, C., Zhu, M., Chen, H., Liu, Y., Wu, W., Zhang, H., Shen, C.: Divergen: Improving instance segmentation by learning wider data distribution with more diverse generative data. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. p. 3986โ3995 (2024) 7. Fu, S., Yang, Q., Mo, Q., Yan, J., Wei, X., Meng, J., Xie, X., Zheng, W.S.: Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 14987โ14997 (2025) 8. Guo, H., Zhu, H., Peng, S., Wang, Y., Shen, Y., Hu, R., Zhou, X.: Sam-guided graph cut for 3d instance segmentation. In: Computer Vision โ ECCV 2024: 18th European Conference, Milan, Italy, September 29โOctober 4, 2024, Proceedings, Part XLVIII. p. 234โ251. Springer-Verlag, Berlin, Heidelberg (2024). https:// doi.org/10.1007/978-3-031-73195-2_14, https://doi.org/10.1007/978-3- 031-73195-2_14 9. Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 5356โ5364 (2019) 10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p. 770โ778 (2016) 11. He, Y.Y., Zhang, P., Wei, X.S., Zhang, X., Sun, J.: Relieving long-tailed instance segmentation via pairwise class balance. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. p. 7000โ7009 (2022) 12. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840โ6851 (2020) 13. Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024) 14. de Jorge, P., Volpi, R., Dokania, P.K., Torr, P.H., Rogez, G.: Placing objects in con- text via inpainting for out-of-distribution segmentation. In: European Conference on Computer Vision. p. 456โ473. Springer (2024) TMI35 15. Kim, J., Han, S., Jeong, J., Choi, J., Kim, D., Kim, S.J.: Orida: Object-centric real-world image composition dataset. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 3051โ3060 (2025) 16. Kimhi, M., Vainshtein, D., Baskin, C., Di Castro, D.: Robot instance segmentation with few annotations for grasping. In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). p. 7939โ7949. IEEE (2025) 17. Labs, B.F.: Flux. https://github.com/black-forest-labs/flux (2024) 18. Labs, B.F., Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dock- horn, T., English, J., English, Z., Esser, P., et al.: Flux. 1 kontext: Flow match- ing for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742 (2025) 19. Li, Y., Keuper, M., Zhang, D., Khoreva, A.: Adversarial supervision makes layout- to-image diffusion models thrive. In: ICLR (2024) 20. Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022) 21. Liu, C., Li, X., Ding, H.: Referring image editing: Object-level image editing via referring expressions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 13128โ13138 (June 2024) 22. Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022) 23. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 10012โ10022 (2021) 24. Loiseau, T., Vu, T.H., Chen, M., Pรฉrez, P., Cord, M.: Reliability in semantic seg- mentation: Can we use synthetic data? In: European Conference on Computer Vision. p. 442โ459. Springer (2024) 25. Luo, W., Yang, S., Zhang, X., Zhang, W.: Siedob: Semantic image editing by dis- entangling object and background. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1868โ1878 (June 2023) 26. Molina, J.M., Llerena, J.P., Usero, L., Patricio, M.A.: Advances in instance seg- mentation: Technologies, metrics and applications in computer vision. Neurocom- puting 625, 129584 (2025). https://doi.org/https://doi.org/10.1016/j. neucom.2025.129584, https://w.sciencedirect.com/science/article/pii/ S0925231225002565 27. Nguyen, Q., Vu, T., Tran, A., Nguyen, K.: Dataset diffusion: Diffusion-based syn- thetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems 36, 76872โ76892 (2023) 28. Niitani, Y., Akiba, T., Kerola, T., Ogawa, T., Sano, S., Suzuki, S.: Sampling tech- niques for large-scale object detection from sparsely annotated objects. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 6510โ6518 (2019) 29. Pan, T.Y., Zhang, C., Li, Y., Hu, H., Xuan, D., Changpinyo, S., Gong, B., Chao, W.L.: On model calibration for long-tailed object detection and instance segmen- tation. Advances in Neural Information Processing Systems 34, 2529โ2542 (2021) 30. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 4195โ4205 (2023) 31. Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rรคdle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024) 36H. Song & S. Choi et al. 32. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 10684โ10695 (2022) 33. Tan, J., Lu, X., Zhang, G., Yin, C., Li, Q.: Equalization loss v2: A new gradient bal- ance approach for long-tailed object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 1685โ1694 (2021) 34. Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., Yan, J.: Equalization loss for long-tailed object recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 11662โ11671 (2020) 35. Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems 30 (2017) 36. Tschirschwitz, D.E., Rodehorst, V.: Label convergence: Defining an upper perfor- mance bound in object recognition through contradictory annotations. In: Proceed- ings of the Winter Conference on Applications of Computer Vision. p. 6848โ6857 (2025) 37. Wang, T., Li, Y., Kang, B., Li, J., Liew, J.H., Tang, S., Hoi, S., Feng, J.: Classification calibration for long-tail instance segmentation. arXiv preprint arXiv:1910.13081 (2019) 38. Wang, T., Li, Y., Kang, B., Li, J., Liew, J., Tang, S., Hoi, S., Feng, J.: The devil is in classification: A simple framework for long-tail instance segmentation. arXiv preprint arXiv:2007.11978 (2020) 39. Wang, T., Yang, T., Cao, J., Zhang, X.: Co-mining: Self-supervised learning for sparsely annotated object detection. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, p. 2800โ2808 (2021) 40. Wang, X., Darrell, T., Rambhatla, S.S., Girdhar, R., Misra, I.: Instancediffusion: Instance-level control for image generation. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. p. 6232โ6242 (2024) 41. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600โ612 (2004). https://doi.org/10.1109/TIP.2003.819861 42. Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.m., Bai, S., Xu, X., Chen, Y., et al.: Qwen-image technical report. arXiv preprint arXiv:2508.02324 (2025) 43. Wu, S., Fei, H., Chua, T.s.: Universal scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 14158โ14168 (June 2025) 44. Wu, W., Zhao, Y., Chen, H., Gu, Y., Zhao, R., He, Y., Zhou, H., Shou, M.Z., Shen, C.: Datasetdm: Synthesizing data with perception annotations using diffu- sion models. Advances in Neural Information Processing Systems 36, 54683โ54695 (2023) 45. Wu, W., Zhao, Y., Shou, M.Z., Zhou, H., Shen, C.: Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 1206โ1217 (2023) 46. Xie, C., Mousavian, A., Xiang, Y., Fox, D.: Rice: Refining instance masks in clut- tered environments with graph neural networks. In: Conference on Robot Learning. p. 1655โ1665. PMLR (2022) 47. Xie, J., Li, W., Li, X., Liu, Z., Ong, Y.S., Loy, C.C.: Mosaicfusion: Diffusion mod- els as data augmenters for large vocabulary instance segmentation. International Journal of Computer Vision 133(4), 1456โ1475 (2025) TMI37 48. Xue, H., Huang, Z., Sun, Q., Song, L., Zhang, W.: Freestyle layout-to-image synthe- sis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 14256โ14266 (2023) 49. Yang, L., Xu, X., Kang, B., Shi, Y., Zhao, H.: Freemask: Synthetic images with dense annotations make stronger segmentation models. Advances in Neural Infor- mation Processing Systems 36, 18659โ18675 (2023) 50. Ye, H., Kuen, J., Liu, Q., Lin, Z., Price, B., Xu, D.: Seggen: Supercharging segmen- tation models with text2mask and mask2img synthesis. In: European Conference on Computer Vision. p. 352โ370. Springer (2024) 51. Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 3836โ3847 (2023) 52. Zhang, Y., Yin, M., Bi, W., Yan, H., Bian, S., Zhang, C.H., Hua, C.: Zisvfm: Zero-shot object instance segmentation in indoor robotic environments with vision foundation models. IEEE Transactions on Robotics (2025) 53. Zhao, H., Sheng, D., Bao, J., Chen, D., Chen, D., Wen, F., Yuan, L., Liu, C., Zhou, W., Chu, Q., et al.: X-paste: Revisiting scalable copy-paste for instance seg- mentation using clip and stablediffusion. In: International Conference on Machine Learning. p. 42098โ42109. PMLR (2023) 54. Zhao, Y., Chen, S., Chen, Q., Hu, Z.: Combining loss reweighting and sample resampling for long-tailed instance segmentation. In: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). p. 1โ5 (2023). https://doi.org/10.1109/ICASSP49357.2023.10094303 55. Zhou, X., Koltun, V., Krรคhenbรผhl, P.: Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461 (2021) 56. Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) 57. Zong, Z., Song, G., Liu, Y.: Detrs with collaborative hybrid assignments training. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 6748โ6758 (2023)