Paper deep dive
InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting
Duc Vu, Kien Nguyen, Trong-Tung Nguyen, Ngan Nguyen, Phong Nguyen, Khoi Nguyen, Cuong Pham, Anh Tran
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/26/2026, 1:48:01 AM
Summary
InverFill is a one-step inversion method designed for few-step diffusion-based image inpainting. It addresses the semantic misalignment and harmonization issues caused by random Gaussian noise initialization in few-step models by mapping masked images into semantically informed noise latents. The method introduces a Re-Blending operation to prevent information leakage and a Gaussian regularization loss to ensure the inverted latent aligns with the expected noise distribution, enabling high-fidelity inpainting without costly retraining.
Entities (5)
Relation Signals (3)
InverFill → uses → Re-Blending
confidence 98% · we introduce the Re-Blending operation to mitigate information leakage
InverFill → uses → Gaussian regularization loss
confidence 98% · we introduce a Gaussian regularization loss to align the inverted noise latent
InverFill → improves → few-step inpainting
confidence 95% · InverFill, an efficient one-step inversion network that significantly improves performance of few-step inpainting
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization.
Tags
Links
- Source: https://arxiv.org/abs/2603.23463v1
- Canonical: https://arxiv.org/abs/2603.23463v1
Trouble viewing inline? Open PDF directly →
Full Text
64,311 characters extracted from source content.
Expand or collapse full text
InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting Duc Vu1⋆ Kien Nguyen1⋆ Trong-Tung Nguyen1⋆ Ngan Nguyen1⋆ Phong Nguyen1 Khoi Nguyen1 Cuong Pham1,2 Anh Tran1 1 Qualcomm AI Research† 2 Posts & Telecommunications Inst. of Tech., Vietnam ducvu, kienn, tunnguy, ngannguy, phongnh, khoi, pcuong, anhtra@qti.qualcomm.com cuongpv@ptit.edu.vn Abstract Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization. 00footnotetext: ⋆ Equal Contribution † Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. 1 Introduction Recent generative models enable photorealistic and detail-rich visual synthesis across many tasks [39, 43, 3, 52, 34]. Among them, text-guided image inpainting has become a key direction, aiming to fill masked regions with content that is semantically aligned with the prompt and visually consistent with the background. Progress in this area is largely driven by large-scale text-to-image diffusion models [43, 20, 5]. To adapt pretrained models for inpainting, recent methods rely on blended sampling or fine-tuning with spatially aware architectures, exploiting strong pretrained priors for seamless results. Early approaches fine-tune the full diffusion U-Net with mask conditioning [43, 38, 56], while adapter-based methods like BrushNet [16] add lightweight trainable branches to frozen backbones. Training-free methods [28, 14, 22] instead use guided sampling or attention manipulation. Despite their effectiveness, most techniques require many sampling steps, resulting in high latency and limiting real-time deployment. This underscores the need for faster inpainting solutions. While many few-step text-to-image diffusion models exist [47, 46, 27, 4], adapting them for image inpainting is nontrivial. A natural solution is blended sampling [2], where predictions are iteratively merged with the unmasked regions. This strategy works well for multi-step diffusion models, where gradual denoising allows the synthesized content to blend smoothly with the preserved context. In the few-step regime, however, each denoising step induces much larger updates, leading to semantic misalignment between the initial noise and the masked content, and ultimately causing poor harmonization with the surrounding background. To the best of our knowledge, TurboFill [55] is the only successful few-step, text-guided specialized inpainting model. It reduces inference steps using a 3-step adversarial scheme that trains an inpainting adapter on top of a distilled few-step text-to-image model [59]. However, this design is complex, requires real-image supervision, and is computationally heavy. Moreover, prior inpainting methods follow the standard diffusion process [13, 49], which always starts from pure Gaussian noise. This gives the model no initial clue about the semantics or structure of the unmasked regions, often causing a semantic mismatch between the inpainted content and its surrounding context. Multi-step models can gradually correct this mismatch, but few-step or one-step models have no such allowance, leaving little room to recover from the initial randomness. As a result, inpainting under low NFEs tends to produce blurry, poorly integrated regions and degraded overall fidelity. To this end, we introduce InverFill, an efficient one-step inversion network that significantly improves performance of few-step inpainting with minimal overhead. As shown in LABEL:fig:teaser, InverFill maps the masked image into an inverted noise latent, replacing random Gaussian initialization with a semantically informed noise for few-step inpainting. Although diffusion inversion has been explored for editing and inpainting [30, 29, 6, 22], we are the first to design a one-step inversion customized for inpainting. While SwiftEdit [35] proposes a one-step inversion framework for image editing, a naive adaptation to inpainting fails for two reasons. First, training on masked images causes substantial leakage from the visible regions into the inverted noise latent. Second, its reconstruction objective does not constrain the inverted latent to follow the required Gaussian distribution. To overcome this, we introduce Re-Blending to prevent information leakage and a Gaussian regularization loss to ensure the inverted noise latent aligns with the expected noise distribution. Our training pipeline is image-free, requiring no curated image–mask–text triplets and no multi-stage procedures. With these designs, InverFill enhances few-step inpainting and enables few-step text-to-image models to perform on par with specialized inpainting systems, without any finetuning while introducing negligible latency. Our contributions are summarized as follows: • We propose InverFill, an efficient one-step inversion network for few-step image inpainting, which generates semantically informed initial noise to improve inpainting quality while introducing minimal overhead. • We introduce the Re-Blending operation to mitigate information leakage during training while preserving key semantics in the inverted noise latent for inpainting. • We introduce a Gaussian regularization loss to align the inverted noise latent with the expected Gaussian distribution, enhancing stability and quality. • Our method features a highly simplified, image-free training pipeline that eliminates the need for image-mask-text triplets and complex multi-stage training. • We demonstrate that InverFill significantly boosts the performance of existing few-step inpainting models and enables few-step text-to-image models to perform high-quality inpainting without any task-specific fine-tuning. 2 Related Works Figure 1: Inversion Network Training: We train an inversion network, θF_θ, to invert a masked image to a noise latent z^T z_T such that, after blending with random Gaussian noise to form z^Tblend z_T^blend, the latent enables high-fidelity, well-harmonized reconstruction of the original image. Figure 2: Inpainting Pipeline: The inversion network extracts the latent z^T z_T from a masked image, which is blended with random noise to form z^Tblend z_T^blend and then fed into a few-step inpainting pipeline to generate the final image. (Zoom in for details) 2.1 Fast Text-to-image Diffusion Models Traditional multi-step diffusion models [43, 39, 5, 13] are known for slow sampling, often requiring dozens to hundreds of neural function evaluations (NFEs) per image. Recent diffusion distillation methods [44, 50, 27, 33, 7, 36] significantly accelerate generation by aligning the student model’s prediction trajectory with that of a pre-trained multi-step teacher, enabling few-step (4-8 step) inference. Progressive Distillation [44] repeatedly distills from the teacher while halving the number of steps at each stage, preserving high sample quality while reducing from thousands of steps. Consistency Model [50, 27, 31] enforces self-consistency in predictions via either distillation-based or distillation-free objectives. ADD [47] and LADD [46] employs a combination of adversarial training and score distillation for turning pretrained multi-step diffusion models into few-step diffusion model. SANA-Sprint [4] accelerates sampling with a training-free transformation into TrigFlow [25], followed by few-step training with dense time embeddings, QK-normalization, and max-time weighting. 2.2 Image Inpainting Approaches Image inpainting fills missing regions so they blend naturally with the surrounding context. Early methods [26, 6, 40] use GANs or unconditional diffusion models trained on specific datasets [24, 18, 8]. For instance, RePaint [26] uses an unconditional DDPM [13] as a generative prior and blends available pixels into the sampling process. Text-to-image diffusion models provide strong image–text priors for text-guided inpainting, which demands both realistic content completion and semantic alignment with the prompt. BrushNet [16] fine-tunes both a pretrained text-to-image model and an additional conditional branch for inpainting, and then relies on multi-step sampling to produce coherent results. Meanwhile, Blended Latent Diffusion [2] guides the multi-step sampling process using a blending operation, gradually aligning the inpainting content with the surrounding background from the source image. Such methods require many NFEs to achieve high-quality results. As fast few-step generative models emerge [27, 4, 46], reducing the number of sampling steps becomes increasingly necessary, motivating the study of few-step inpainting. A straightforward idea is to apply similar blending strategies on few-step text-to-image models. However, extending inpainting to few-step diffusion models [27, 4, 46] remains challenging, as blended sampling alone is insufficient to produce coherent results, leading to poor visual quality, as shown in LABEL:fig:teaser. TurboFill [55] addresses this by training an inpainting adapter on a few-step text-to-image generation model with a complex 3-step adversarial training scheme, which requires extensive real-image supervision. Moreover, TurboFill exclusively explores on UNet–based architectures [43, 39], leaving its generalization to other models questionable. Hence, few-step inpainting remains under-explored. 2.3 Diffusion-based Inversion While diffusion models generate images by progressively denoising a noisy latent, diffusion inversion methods [49, 30, 17, 11, 45] perform the reverse: recovering an inverted latent that faithfully reconstructs the original image when re-denoised. Such inversion is essential for reconstruction, latent exploration, and downstream editing. DDIM Inversion [49] introduced a deterministic reverse process by linearizing noise prediction across adjacent steps, an approximation effective for models with many sampling iterations [9, 43, 20, 39]. This enables reversed sampling for faithful reconstruction and editing. Null-text Inversion [30] refines null-text embeddings via costly iterative optimization, whereas Direct Inversion [17] eliminates this optimization by decoupling reconstruction and editing pathways. However, the linear approximation used in prior inversion methods breaks down for few-step diffusion models [50, 27, 47], resulting in poor inversion quality. Recent works [11, 45] therefore develop inversion techniques tailored to few-step models [39, 20]. Renoise [11] refines noise latents using fixed-point iteration combined with step-wise averaging, while GNRI [45] formulates inversion as a scalar root-finding problem solved with 1-2 Newton–Raphson iterations per step. These methods significantly accelerate and stabilize inversion compared to multi-step approaches. Recently, SwiftEdit [35] pushes this further with a one-step inversion network trained for one-step diffusion models [23, 60, 59, 33, 7]. This network directly maps source images into its noise latent in a single forward pass, enabling fast image reconstruction and editing with minimal overhead. Inspired by this, we incorporate a similar inversion network into our inpainting framework, enhanced with refinements and dedicated training objectives to enable efficient, high-quality few-step inpainting. 3 Preliminaries 3.1 Text-to-Image Diffusion Models. Text-to-image diffusion models synthesize images by aligning textual inputs with corresponding visual features. State-of-the-art methods primarily use latent diffusion [43, 39], where a Variational Auto-Encoder (VAE) [19] encoder ℰE maps an image I to a latent z. The denoising process comprises a fixed forward noising step and a learned reverse step. In the forward process, a clean latent z0=ℰ(I)z_0=E(I) is gradually corrupted into Gaussian noise over T timesteps via a Markov chain q(zt|zt−1)q(z_t|z_t-1) with a variance schedule βt _t: q(zt|zt−1)=(zt;1−βtzt−1,βt).q(z_t|z_t-1)=N(z_t; 1- _tz_t-1, _tI). (1) zt=1−βtzt−1+βtϵ,where ϵ∼(0,).z_t= 1- _tz_t-1+ _tε, ε (0,I). (2) Given an input noise zTz_T sampled from Eq. 1 and a text prompt c, the training objective of the denoising network ϵθ _θ at timestep t is defined as: minθz0,c,ϵ∼(1,T),t∼(0,I)‖ϵ−ϵθ(zt,t,c)‖22 _θE_z_0,c,ε (1,T),t (0,I) \|ε- _θ(z_t,t,c) \|_2^2 (3) During inference, ϵθ _θ iteratively estimates and removes the noise from the noisy image across T timesteps. In practice, large T are required to gradually refine the image, ensuring high-quality generation. In contrast, few-step models apply large, discrete updates at each step, which limits the opportunity for smooth adjustments. Any intermediate modification, such as blending, can easily disrupt the denoising trajectory, leading to artifacts or failed reconstructions. 3.2 Image Inpainting Problem Definition. Given a masked image Im∈ℝH×W×CI_m ^H× W× C with missing content defined by a binary mask M∈0,1H×W×CM∈\0,1\^H× W× C, where 0 denotes unmasked regions and 1 denotes masked regions, image inpainting aims to generate IinpaintI_inpaint within the masked region to form a composited image I=Im⊙(1−M)+Iinpaint⊙MI=I_m (1-M)+I_inpaint M such that the inpainted regions are semantically aligned with a text prompt c and visually consistent with the unmasked context, accurately reflecting what and where to inpaint. Blended Sampling Strategy. This inpainting approach, exemplified by Blended Latent Diffusion (BLD) [2], gradually blends known information from unmasked regions with generated content in the masked areas. Given a masked image ImI_m and a corresponding binary mask M (resized to m in the latent space), the initial masked latent representation is computed as z0m=ℰ(Im)z_0^m=E(I_m). During the reverse diffusion process, at each timestep t, BLD adds noise to the known regions of the original latent z0mz_0^m following Eq. 1, yielding ztmz_t^m. As described in Eq. 4, BLD then blends ztmz_t^m with the predicted denoised latent z^t z_t using the mask m. The resulting blended latent serves as the input for the subsequent denoising step at t−1t-1, ensuring a seamless transition between the unmasked context and the newly generated content. ztblend=ztm⊙(1−m)+z^t⊙mz_t^blend=z^m_t (1-m)+ z_t m (4) Figure 3: Failure of BLD in few-step models (SDXL-Turbo, 4 steps) is illustrated in Column 3 and corrected by our method in Column 4. (Zoom in for details) 4 Method In Sec. 4.1, we analyze the failure of blended sampling in few-step models and outline the motivations behind InverFill. The key component of our system is a one-step inversion network tailored for inpainting. We will present an overview on this network (Sec. 4.2), followed by our proposed components in training pipeline (Secs. 4.3, 4.4, 4.5, 4.6 and 4.7). Finally, we present our inpainting pipeline in Sec. 4.8. Fig. 1 illustrates the training pipeline of our inversion network, while Fig. 2 shows the full inpainting inference pipeline. 4.1 Motivation Blended Sampling Strategy for Few-Step Model. While BLD is effective for multi-step diffusion models, applying it directly to few-step models significantly reduces inpainting quality. As shown in Fig. 3, blending under few-step inference introduces semantic and stylistic inconsistencies between the generated and unmasked regions, yielding visible artifacts. This limitation originates from the initialized random Gaussian noise in the reverse process: multi-step models progressively refine this noise and adapt to the context of the unmasked regions within ImI_m, whereas few-step models make large ODE updates and lack sufficient refinement steps. When initialized from semantically distant noise, few coarse updates cannot correct the mismatch. Thus, effective few-step blending requires initializing zTz_T semantically aligned with the unmasked regions of the image. Inversion for Image Inpainting. A promising direction for mitigating semantic misalignment is diffusion inversion, which maps the unmasked image into the final noise latent zTz_T. However, existing inversion methods are iterative and introduce considerable overhead, contradicting the efficiency requirements of few-step sampling. A one-step inversion is critical for fast inference, as demonstrated by SwiftEdit [35], which provides efficient and semantically coherent initialization. Nonetheless, directly applying SwiftEdit to inpainting is unsuitable for two reasons: (1) it is not explicitly designed for processing masked inputs, which causes information leakage during training, and (2) its training objectives do not enforce the inverted noise to follow the required Gaussian prior, resulting in distributional mismatch and degraded reconstructions. To overcome these limitations, we introduce InverFill, a one-step inversion network designed for inpainting. InverFill (1) operates directly on masked images to produce semantically aligned initial noise latents, and (2) explicitly regularizes the inverted noise to match the Gaussian prior. Addressing both issues enables InverFill to achieve high-fidelity, coherent inpainting within the few-step regime. 4.2 Masked Image Inversion Network Problem Definition. Given a pretrained one-step text-to-image generator G, we aim to develop a one-step inversion network θF_θ that is tailored for the inpainting purpose. Specifically, given a ground-truth image IgtI_gt and a masked image ImI_m produced from IgtI_gt using a binary mask M, i.e., Im=Igt⊙(1−M)I_m=I_gt (1-M), their image latents are z0=ℰ(Igt)z_0=E(I_gt) and z0m=ℰ(Im)z_0^m=E(I_m), where ℰE is the VAE encoder. We train θF_θ to map z0mz_0^m and text prompt c to an inverted noise latent. The network is optimized so that passing this latent through G produces a predicted image latent z^0 z_0 resembling the original latent z0z_0. The predicted noise latent should yield a reconstruction where (1) the background faithfully preserves the masked input, and (2) the generated region harmonizes with the background while remaining consistent with the text prompt and the unmasked content of ImI_m. Inversion Network Architecture. Following [35], θF_θ shares the architecture of the one-step generator G and inherits its pretrained weights during as initialization. Masked Image Training. To adapt our inversion network to masked image inputs, we leverage the one-step generator G to synthesize training image–mask–prompt triplets on the fly. Given a text prompt c and random Gaussian noise ϵ∼(0,I)ε (0,I), G produces a ground-truth image latent z0=(ϵ,c)z_0=G(ε,c) and its corresponding image Igt=(z0)I_gt=D(z_0), where D is the VAE decoder. To ensure robustness and prevent overfitting to specific masks, we randomly sample a mask M of diverse shapes and brush types and apply it to IgtI_gt to generate the masked image ImI_m. The subsequent masked image latent z0m=ℰ(Im)z_0^m=E(I_m) serves as input to our one-step inversion network θF_θ, which predicts the inverted noise latent z^T z_T. In the following sections, we introduce our objective functions and describe how we optimize and integrate z^T z_T to achieve a high-quality reconstruction of z0z_0. Figure 4: Effects of the proposed Re-Blending operation during training, without reg L_reg. (Zoom in for details) 4.3 Reconstruction Objectives Similar to SwiftEdit [35], we apply reconstruction losses in both the noise latent (ℒnoiseL_noise) and image latent (ℒimageL_image) spaces. Since our inversion network operates on a masked image ImI_m, applying ℒnoiseL_noise over the entire predicted latent z^T z_T is suboptimal: the regions of z0mz^m_0 corresponding to the masked areas of ImI_m contain no meaningful information, and penalizing these regions can hinder training. Therefore, we restrict ℒnoiseL_noise to the unmasked regions. Our reconstruction objectives are formulated as follows: ℒnoise=‖(1−m)⊙z^T−(1−m)⊙ϵ‖22,L_noise=\|(1-m) z_T-(1-m) ε\|_2^2, (5) ℒimage=‖z^0−z0‖22L_image=\| z_0-z_0\|_2^2 (6) ℒrecons=λnoise∗ℒnoise+λimage∗ℒimageL_recons= _noise*L_noise+ _image*L_image (7) 4.4 Re-Blending Operation Our inversion network maps the unmasked image content to a noise latent z^T z_T. In SwiftEdit [35], ℒnoiseL_noise ensures that the predicted noise latent z^T z_T preserves details of the complete input image I in the noise latent space. However, for inpainting tasks, our inversion network θF_θ only receives the incomplete masked image ImI_m to predict z^T z_T. Consequently, our masked loss ℒnoiseL_noise in Eq. 5 causes training bias towards the unmasked regions. This bias causes image-space structural patterns from ImI_m to leak into z^T z_T, while regions corresponding to the mask exhibit low variance and artifacts. As a result, z^T z_T deviates significantly from the Gaussian distribution expected by the diffusion model. During training, this distributional mismatch leads G to collapse when computing z^0=(z^T,c) z_0=G( z_T,c), producing the incoherent, artifact-filled outputs illustrated in Fig. 4. To address this, we introduce a Re-Blending operation. During training and inference, the masked regions of the predicted noise latent z^T z_T are replaced with random Gaussian noise ϵ′∼(0,I)ε (0,I), partially restoring the latent to the expected distribution and recovering key semantic features, as shown in Fig. 4. Following Eq. 8, generator G inputs the corrected latent z^Tblend z_T^blend to produce the final output z^0 z_0. z^Tblend=z^T⊙(1−m)+ϵ′⊙m,z^0=(z^Tblend,c) z_T^blend= z_T (1-m)+ε m, z_0=G( z_T^blend,c) (8) where m is latent-space mask downsampled from M. Figure 5: Ablation Study on ℒregL_reg. Without the regularization loss, the model fails to preserve the background during reconstruction and produces blurred, low-detail outputs. With the loss, the background is well preserved and fine image details are restored. 4.5 Gaussian Regularization As shown in Fig. 4, Re-Blending mitigates information leakage and partially restores the Gaussian structure of z^Tblend z_T^blend, but the predicted latent z^0 z_0 recovers only key semantics. The output fails to preserve background, and the generated content within masked regions remains blurry and low-detail, as shown in Fig. 5. Despite Re-Blending’s partial correction, z^Tblend z_T^blend still deviates from the standard Gaussian expected by the generator. This occurs because ℒnoiseL_noise in Eq. 5 focuses solely on the unmasked regions. While the image-space loss ℒimageL_image in Eq. 6 indirectly encourages the injected noise ϵ′ε to harmonize with z^T z_T to better reconstruct image latent, it cannot fully enforce Gaussian consistency due to the lack of direct supervision with ground-truth Gaussian noise within the masked regions. As a result, the added ϵ′ε remains visually distinct from the background inverted noise in z^T z_T (Column 2 in Fig. 5), indicating that z^Tblend z_T^blend is still far from the target Gaussian distribution. To address this issue, inspired by [15], we introduce an additional Gaussian regularization term. This term explicitly encourages Gaussian distribution on the blended latent z^Tblend z_T^blend by matching its statistical moments with the theoretical moments of a standard Gaussian. Let μn _n be the n-th theoretical moment of a standard Gaussian. The moment-matching loss for the n-th moment is defined as: ℒn=‖|1D∑k=1D(z^Tblend)n|1n−μn1n‖,L_n= \| | 1D _k=1^D ( z_T^blend )^n | 1n- _n 1n \|, (9) where D=c×h×wD=c× h× w is the total number of pixels of z^Tblend z_T^blend. Our final regularization loss, ℒregL_reg, is the sum of the losses for the first and second moments, corresponding to the mean and variance of Gaussian distribution: ℒreg=∑n∈1,2ℒnL_reg= _n∈\1,2\L_n (10) As shown in Fig. 5, Gaussian Regularization Loss during training helps z^Tblend z_T^blend better align with the Gaussian prior, enabling faithful reconstruction of the original image while preserving the background, as confirmed in Tab. 2. Table 1: Quantitative comparison of InverFill against few-step and multi-step diffusion inpainting baselines on BrushBench and MagicBrush. NFEs denotes the number of function evaluations. ↑ indicates that higher is better, ↓ indicates that lower is better Type Method NFEs BrushBench MagicBrush Runtime↓ IR↑×10_× 10 HPS↑×102_× 10^2 AS↑ CLIP↑ IR↑×10_× 10 HPS↑×102_× 10^2 AS↑ CLIP↑ (seconds) SANA-Sprint 0.6B 2 11.02 26.21 6.05 27.12 2.55 25.07 5.32 25.67 0.37 !15 SANA-Sprint 0.6B + InverFill 2 11.65 27.93 6.15 27.17 3.04 25.37 5.42 25.71 0.43 (+0.06) SANA-Sprint 0.6B 4 10.82 26.34 6.00 27.11 2.56 25.12 5.37 25.63 0.45 !15 SANA-Sprint 0.6B + InverFill 4 11.76 27.83 6.18 27.19 3.14 25.47 5.43 25.74 0.51 (+0.06) SDXL Turbo 4 11.42 28.20 6.06 27.26 3.51 25.76 5.46 25.79 0.66 !15 SDXL Turbo + InverFill 4 12.38 28.44 6.08 27.67 3.75 25.84 5.48 26.08 0.70 (+0.04) SDXL Turbo + BrushNet 4 12.56 28.26 6.00 27.51 4.20 24.92 5.20 25.62 0.70 !15 Few-step SDXL Turbo + BrushNet + InverFill 4 12.63 28.43 6.03 27.62 4.154.15 25.10 5.23 25.68 0.74 (+0.04) !20 SANA 0.6B 20 12.12 27.04 6.17 27.49 3.68 24.11 5.48 25.93 1.18 !20 HD-Painter 30 12.82 28.17 6.30 27.43 3.59 24.60 5.65 25.87 23.45 !20 SDXL-Inpainting 30 13.16 28.92 6.37 27.15 3.91 24.13 5.51 25.50 3.35 !20Multi-step SDXL + BrushNet 30 13.26 28.28 6.26 27.54 3.94 24.28 5.46 25.60 4.31 Figure 6: Our method achieves qualitative results comparable to multi-step SDXL-Inpainting and is on par with BrushNet (4 steps), as shown in Columns 7 and 8. Notably, this performance is obtained using only text prompts during training, whereas competing methods rely on full text–image–mask supervision. Moreover, integrating our approach with BrushNet further enhances semantic coherence. 4.6 Improving Quality with Adversarial Loss Previous works [60, 59, 4] show that adversarial losses during training improve visual quality. Following LADD [46], we use the frozen teacher model to define a latent feature space for adversarial supervision, with multiple discriminator heads on intermediate layers for stable, efficient distillation. In our training, we treat the original image latent z0z_0 as real and the predicted image latent z^0 z_0 as fake to train the inversion model and discriminator as follows: ℒadvG(θ)=−z^0,t[∑kDψ,k(Gpre(z^t,t,c))]L_adv^G(θ)=-E_ z_0,t [ _kD_ψ,k (G_pre ( z_t,t,c ) ) ] (11) ℒadvD _adv^D (ψ)=z0,t[∑kReLU(1−Dψ,k(Gpre(zt,t,c)))] (ψ)=E_z_0,t [ _kReLU (1-D_ψ,k (G_pre (z_t,t,c ) ) ) ] (12) +z^0,t[∑kReLU(1+Dψ,k(Gpre(z^t,t,c)))] +E_ z_0,t [ _kReLU (1+D_ψ,k (G_pre ( z_t,t,c ) ) ) ] where ztz_t, z^t z_t are noisy versions of original image latent z0z_0 and predicted image latent z^0 z_0 at timesteps t. GpreG_pre denotes a frozen multi-step teacher model. Dψ,kD_ψ,k denotes discriminator heads at the k-th intermediate layers of GpreG_pre. 4.7 Final Objectives Our final training objective for θF_θ is defined as follows: ℒfinal=λrecons∗ℒrecons+λreg∗ℒreg+λadv∗ℒadvL_final= _recons*L_recons+ _reg*L_reg+ _adv*L_adv (13) 4.8 Inpainting Pipeline Fig. 2 illustrates our inpainting pipeline, which closely follows the blended sampling strategy described in Sec. 3.2. However, instead of initializing with random Gaussian noise, we employ our trained inversion model θF_θ to predict the inverted noise latent z^T z_T and obtain the blended latent z^Tblend z_T^blend using Eq. 8. This blended latent z^Tblend z_T^blend serves as the Gaussian noise input to the inpainting process. 5 Experiments 5.1 Training Details We train InverFill on Sana-Sprint 0.6B [4] and SDXL-Turbo [47], which represent two common diffusion architectures: DiT and UNet. All training is performed on four NVIDIA A100 40GB GPUs for 8-10 hours. During training, we randomly sample text prompts from BrushData [16] and MSCOCO [21]. We use a total batch size of 32 and a learning rate of 1×10−51× 10^-5 with AdamW optimizer. 5.2 Evaluation Setup Dataset. We perform evaluation on inpainting BrushBench [16], with 600 images and annotated masks, and image editing MagicBrush [61] benchmark. For inpainting, we adapt MagicBrush’s 535-image test set using its captions and masks. Each image includes multiple segmentation and random masks, providing a diverse and challenging evaluation for inpainting performance. All experiments and evaluations were performed using 102421024^2 resolution. Evaluation Metrics. We evaluate our results from two criteria: image generation quality and text alignment. • Image Generation Quality. We use three human-aligned metrics: ImageReward (IR) [57], HPS v2 (HPS) [53], and Aesthetic Score (AS) [48]. IR and HPS are reward models trained on large-scale human preference data, while AS is a linear model trained to predict perceptual quality. • Text Alignment. We measure text–image alignment using CLIP Similarity (CLIP) [41], which quantifies how well the inpainted images match their prompts. Baselines. We evaluate InverFill on state-of-the-art few-step text-to-image diffusion models, SANA-Sprint 0.6B [4] and SDXL-Turbo [47], using the blended sampling strategy in Sec. 3.2 for inpainting. We report results using 2- and 4-step NFE settings for SANA-Sprint and 4-step for SDXL-Turbo. Following [55], we integrate SDXL-Turbo [39] with BrushNet [16] to evaluate InverFill using few-step specialized inpainting models that do not rely on the blended sampling strategy. For reference, we report results from multi-step models, including Sana 0.6B [54], HD-Painter [28], SDXL-Inpainting [39] and SDXL with BrushNet [16]. Table 2: Effects of reg L_reg on SANA-Sprint 0.6B [4] with 2 NFEs on BrushBench [16]. All models were evaluated at 5000 iterations. Method IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ w/o ℒregL_reg 11.11 26.69 6.08 27.13 w/ ℒregL_reg 11.40 27.22 6.12 27.15 5.3 Quantitative Results As shown in Tab. 1, InverFill consistently improves performance across few-step diffusion settings. When integrated with SANA-Sprint and SDXL-Turbo under blended sampling, InverFill boosts all metrics on BrushBench and MagicBrush. For example, SANA-Sprint (2 NFEs) + InverFill raises IR from 11.02 to 11.65 on BrushBench and 2.55 to 3.04 on MagicBrush. InverFill also strengthens specialized inpainting model. In BrushNet + InverFill (4 NFEs), IR improves from 12.56 to 12.63 and HPS from 28.26 to 28.43. Regarding text alignment, InverFill achieves substantial gains in CLIP scores. Notably, InverFill-equipped few-step models match or surpass multi-step methods while remaining efficient; SDXL-Turbo + InverFill (4 NFEs) outperforms HD-Painter (30 NFEs) on key metrics. Despite these gains, InverFill introduces extremely minimal overhead, only 0.06s on SANA-Sprint and 0.04s on SDXL. 5.4 Qualitative Results Fig. 6 shows that integrating InverFill improves coherence and background harmonization. Without using real images, InverFill achieves quality comparable to BrushNet (4-step SDXL-Turbo), which relies on an inpainting dataset of real images [16]. Moreover, combining InverFill with the BrushNet + SDXL-Turbo pipeline further boosts semantic quality, indicating that InverFill can also strengthen specialized few-step inpainting systems. 5.5 Enhanced Caption for BrushBench Motivation. A limitation of BrushBench [16] is its reliance on simple, short prompts, which limits evaluation of text understanding and compositional generation. Modern models, SDXL [39] with dual text encoders and SANA-Sprint [4] with Gemma-2 [42], are built for more context-heavy prompts. Therefore, we expand BrushBench captions using Qwen3 [58] with detailed foreground and background descriptions, enabling more comprehensive evaluation of text alignment and visual coherence in inpainting. Quantitative Results. Tab. 3 shows that InverFill remains effective under detailed, complex prompts, improving all baselines and demonstrating robustness in text-rich settings. For SANA-Sprint, CLIP gains exceed those with simple prompts in Tab. 1, indicating stronger visual–text alignment and better use of large encoders like Gemma-2. Table 3: Quantitative comparison of InverFill against few-step and multi-step diffusion inpainting baselines on BrushBench with enhanced prompts. ↑ indicates that higher is better. Method NFEs IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ SANA-Sprint 0.6B 2 7.53 25.81 6.09 28.10 !15 SANA-Sprint 0.6B + InverFill 2 8.55 26.49 6.19 28.26 SANA-Sprint 0.6B 4 7.32 25.81 6.11 28.15 !15 SANA-Sprint 0.6B + InverFill 4 8.54 26.50 6.21 28.29 SDXL Turbo 4 8.09 26.85 6.12 28.25 !15 SDXL Turbo + InverFill 4 9.00 27.11 6.12 28.70 SDXL Turbo + BrushNet 4 9.35 27.22 6.06 28.87 !15 Few-step SDXL Turbo + BrushNet + InverFill 4 9.53 27.37 6.10 28.91 !20 SANA 0.6B 20 9.34 26.77 6.24 28.63 !20 HD-Painter 30 9.60 27.83 6.35 28.63 !20 SDXL-Inpainting 30 9.90 27.43 6.38 28.38 !20 Multi-step SDXL + BrushNet 30 10.41 28.11 6.34 28.76 6 Conclusion In this work, we introduce InverFill, a lightning-fast one-step inversion network explicitly designed for image inpainting that enhances existing few-step inpainting methods. Extensive experiments show that InverFill produces high-quality inpainting results while adding as few as 0.06 seconds of overhead. References [1] E. Agustsson and R. Timofte (2017-07) NTIRE 2017 challenge on single image super-resolution: dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Cited by: §11. [2] O. Avrahami, O. Fried, and D. Lischinski (2023) Blended latent diffusion. ACM transactions on graphics (TOG) 42 (4), p. 1–11. Cited by: §1, §2.2, §3.2. [3] M. Brack, F. Friedrich, K. Kornmeier, L. Tsaban, P. Schramowski, K. Kersting, and A. Passos (2024) Ledits++: limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8861–8870. Cited by: §1. [4] J. Chen, S. Xue, Y. Zhao, J. Yu, S. Paul, J. Chen, H. Cai, E. Xie, and S. Han (2025) SANA-sprint: one-step diffusion with continuous-time consistency distillation. CoRR. Cited by: §1, §2.1, §2.2, §4.6, §5.1, §5.2, §5.5, Table 2, Table 2, §7. [5] J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Z. Wang, J. Kwok, P. Luo, H. Lu, and Z. Li (2024) PixArt-α: fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations (ICLR), Cited by: §1, §2.1. [6] C. Corneanu, R. Gadde, and A. M. Martinez (2024) Latentpaint: image inpainting in latent space with diffusion models. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 4334–4343. Cited by: §1, §2.2. [7] T. Dao, T. H. Nguyen, T. Le, D. Vu, K. Nguyen, C. Pham, and A. Tran (2024) Swiftbrush v2: make your one-step diffusion model better than its teacher. In European Conference on Computer Vision, p. 176–192. Cited by: §2.1, §2.3. [8] T. T. Dao, D. H. Vu, C. Pham, and A. Tran (2024) Efhq: multi-purpose extremepose-face-hq dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22605–22615. Cited by: §2.2. [9] P. Dhariwal and A. Q. Nichol (2021) Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: Link Cited by: §2.3. [10] R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau (2023) Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2426–2436. Cited by: §14. [11] D. Garibi, O. Patashnik, A. Voynov, H. Averbuch-Elor, and D. Cohen-Or (2024) Renoise: real image inversion through iterative noising. In European Conference on Computer Vision, p. 395–413. Cited by: §2.3, §2.3. [12] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: §11. [13] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.1, §2.2. [14] T. Hsiao, B. Ruan, S. Tsai, Y. Wu, and H. Shuai (2024) Freecond: free lunch in the input conditions of text-guided inpainting. arXiv preprint arXiv:2412.00427. Cited by: §1. [15] J. Hwang, J. Kim, and M. Sung (2025) Moment-and power-spectrum-based gaussianity regularization for text-to-image models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §4.5. [16] X. Ju, X. Liu, X. Wang, Y. Bian, Y. Shan, and Q. Xu (2024) Brushnet: a plug-and-play image inpainting model with decomposed dual-branch diffusion. In European Conference on Computer Vision, p. 150–168. Cited by: §1, §2.2, §5.1, §5.2, §5.2, §5.4, §5.5, Table 2, Table 2, InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting. [17] X. Ju, A. Zeng, Y. Bian, S. Liu, and Q. Xu (2024) PnP inversion: boosting diffusion-based editing with 3 lines of code. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.3. [18] T. Karras, S. Laine, and T. Aila (2019) A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4401–4410. Cited by: §11, §2.2. [19] D. P. Kingma and M. Welling (2014) Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), Cited by: §3.1. [20] B. F. Labs (2024) FLUX. Note: https://github.com/black-forest-labs/flux Cited by: §1, §2.3, §2.3. [21] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: §5.1. [22] H. Liu, Y. Wang, and M. Wang One stone with two birds: a null-text-null frequency-aware diffusion models for text-guided image inpainting. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1, §1. [23] X. Liu, X. Zhang, J. Ma, J. Peng, and Q. Liu (2024) Instaflow: one step is enough for high-quality diffusion-based text-to-image generation. In International Conference on Learning Representations, Cited by: §2.3. [24] Z. Liu, P. Luo, X. Wang, and X. Tang (2015-12) Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), Cited by: §2.2. [25] C. Lu and Y. Song (2025) Simplifying, stabilizing and scaling continuous-time consistency models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.1. [26] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool (2022) Repaint: inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11461–11471. Cited by: §2.2. [27] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao (2023) Latent consistency models: synthesizing high-resolution images with few-step inference. External Links: 2310.04378 Cited by: §1, §2.1, §2.2, §2.3. [28] H. Manukyan, A. Sargsyan, B. Atanyan, Z. Wang, S. Navasardyan, and H. Shi (2023) Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §5.2. [29] D. Miyake, A. Iohara, Y. Saito, and T. Tanaka (2025) Negative-prompt inversion: fast image inversion for editing with text-guided diffusion models. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 2063–2072. Cited by: §1. [30] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or (2023-06) NULL-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6038–6047. Cited by: §1, §2.3. [31] A. Nguyen, V. Van Nguyen, D. Vu, T. T. Dao, C. Tran, T. Tran, and A. T. Tran Improved training technique for shortcut models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2.1. [32] K. Nguyen, A. Tran, and C. Pham (2025) SuMa: a subspace mapping approach for robust and effective concept erasure in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 19587–19596. Cited by: §14. [33] T. H. Nguyen and A. Tran (2024) Swiftbrush: one-step text-to-image diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7807–7816. Cited by: §2.1, §2.3. [34] T. Nguyen, D. Nguyen, A. Tran, and C. Pham (2024) FlexEdit: flexible and controllable diffusion-based object-centric image editing. arXiv preprint arXiv:2403.18605. Cited by: §1. [35] T. Nguyen, Q. Nguyen, K. Nguyen, A. Tran, and C. Pham (2025-06) SwiftEdit: lightning fast text-guided image editing via one-step diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 21492–21501. Cited by: §1, §2.3, §4.1, §4.2, §4.3, §4.4, Figure 7, Figure 7, Table 5, §8, Table 7, Table 8. [36] V. Nguyen, A. Nguyen, T. Dao, K. Nguyen, C. Pham, T. Tran, and A. Tran (2025) Supercharged one-step text-to-image diffusion models with negative prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 18004–18013. Cited by: §2.1. [37] V. Nguyen and V. M. Patel (2025) CGCE: classifier-guided concept erasure in generative models. arXiv preprint arXiv:2511.05865. Cited by: §14. [38] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen (2021) Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. Cited by: §1. [39] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2024) SDXL: improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, Cited by: §1, §2.1, §2.2, §2.3, §2.3, §3.1, §5.2, §5.5. [40] K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn (2022) Diffusion autoencoders: toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10619–10629. Cited by: §2.2. [41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: 2nd item. [42] M. Rivière, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, et al. (2024) Gemma 2: improving open language models at a practical size. CoRR. Cited by: §5.5. [43] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §1, §2.1, §2.2, §2.3, §3.1. [44] T. Salimans and J. Ho (2022) Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. [45] D. Samuel, B. Meiri, H. Maron, Y. Tewel, N. Darshan, S. Avidan, G. Chechik, and R. Ben-Ari (2025) Lightning-fast image inversion and editing for text-to-image diffusion models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.3, §2.3. [46] A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, and R. Rombach (2024) Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, p. 1–11. Cited by: §1, §2.1, §2.2, §4.6. [47] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach (2024) Adversarial diffusion distillation. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXXVI, Berlin, Heidelberg, p. 87–103. External Links: ISBN 978-3-031-73015-3, Link, Document Cited by: §1, §2.1, §2.3, §5.1, §5.2. [48] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. (2022) Laion-5b: an open large-scale dataset for training next generation image-text models. Advances in neural information processing systems 35, p. 25278–25294. Cited by: 1st item. [49] J. Song, C. Meng, and S. Ermon (2021) Denoising diffusion implicit models. In International Conference on Learning Representations, Cited by: §1, §2.3. [50] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: §2.1, §2.3. [51] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky (2022) Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 2149–2159. Cited by: §11. [52] Y. Wang, W. Yang, X. Chen, Y. Wang, L. Guo, L. Chau, Z. Liu, Y. Qiao, A. C. Kot, and B. Wen (2024) Sinsr: diffusion-based image super-resolution in a single step. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 25796–25805. Cited by: §1. [53] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li (2023) Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. CoRR. Cited by: 1st item. [54] E. Xie, J. Chen, J. Chen, H. Cai, H. Tang, Y. Lin, Z. Zhang, M. Li, L. Zhu, Y. Lu, et al. (2024) Sana: efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629. Cited by: §5.2. [55] L. Xie, D. Pakhomov, Z. Wang, Z. Wu, Z. Chen, Y. Zhou, H. Zheng, Z. Zhang, Z. Lin, J. Zhou, et al. (2025) TurboFill: adapting few-step text-to-image model for fast image inpainting. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 7613–7622. Cited by: §1, §2.2, §5.2. [56] S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang (2023) Smartbrush: text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 22428–22437. Cited by: §1. [57] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong (2023) Imagereward: learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36, p. 15903–15935. Cited by: 1st item. [58] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §11, §5.5. [59] T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and B. Freeman (2024) Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems 37, p. 47455–47487. Cited by: §1, §2.3, §4.6. [60] T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park (2024) One-step diffusion with distribution matching distillation. In CVPR, Cited by: §2.3, §4.6. [61] K. Zhang, L. Mo, W. Chen, H. Sun, and Y. Su (2023) Magicbrush: a manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems 36, p. 31428–31449. Cited by: §5.2. Supplementary Material We first present ablations on the loss weights in Sec. 7. Sec. 8 compares our method with other regularization techniques, while Sec. 10 evaluates alternative inversion methods. Additional ablations for our proposed components are in Sec. 9. Sec. 15 includes qualitative comparisons. Note: All experiments in both the main paper and supplementary use 102421024^2 resolution. For all supplementary results, we use BrushBench [16] with its original captions. 7 Loss Weight Ablations We evaluate the impact of reconstruction weights λnoise _noise and λimage _image in ℒreconsL_recons (Sec. 4.3), along with the Gaussian regularization λreg _reg (Sec. 4.5) and adversarial weights λadv _adv (Sec. 4.6), using SANA-Sprint 0.6B [4]. All experiments use the Re-Blending operation (Sec. 4.4) during training and inference. For LADD adversarial loss, the discriminator learning rate is set to 1×10−61× 10^-6. Detailed results are provided in Tab. 4. Table 4: Ablation study on key hyperparameters for each component. The best setting from each block is propagated to the next. Method λnoise _noise λimage _image λreg _reg λadv _adv IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ Sec. 4.3 !15 2.0 !15 1.0 0 0 11.09 26.69 6.04 27.10 !15 1.0 !15 2.0 0 0 10.91 26.67 6.06 27.05 !15 1.0 !15 1.0 0 0 11.11 26.68 6.08 27.13 Sec. 4.5 1.0 1.0 !15 0.25 0 11.12 26.55 6.09 27.13 1.0 1.0 !15 0.5 0 11.40 27.22 6.12 27.15 1.0 1.0 !15 1.0 0 11.36 26.58 6.10 27.14 1.0 1.0 !15 2.0 0 11.03 26.56 6.09 27.17 Sec. 4.6 1.0 1.0 0.5 !15 0.25 11.57 27.36 6.14 27.16 1.0 1.0 0.5 !15 0.5 11.65 27.93 6.15 27.17 1.0 1.0 0.5 !15 1.0 11.60 27.61 6.15 27.17 Tab. 4 summarizes the ablation results on the loss-weight components. Based on this study, we use the final weights λnoise=1.0 _noise=1.0, λimage=1.0 _image=1.0, λreg=0.5 _reg=0.5, and λadv=0.5 _adv=0.5 for all experiments reported in Tabs. 1 and 3. 8 Comparison with Regularization Loss in SwiftEdit We perform an ablation to compare our regularization loss ℒregL_reg with the Score Distillation Sampling loss ℒSDSL_SDS used in SwiftEdit [35]. As shown in Tab. 5, applying ℒregL_reg consistently outperforms ℒSDSL_SDS across all metrics (IR, HPS, AS, and CLIP), demonstrating its effectiveness in preserving image fidelity. Fig. 7 illustrates the qualitative difference between the two losses. With the SDS-based loss, the reconstruction collapses, as the inverted noise is over-regularized and loses the semantic structure of the original image, producing blurry and unrecognizable results. In contrast, our Gaussian regularization loss ℒregL_reg preserves the semantic content and enables high-fidelity reconstruction from the inverted noise. Table 5: Quantitative comparison between the SDS loss ℒSDSL_SDS and our Gaussian regularization loss ℒregL_reg. For a fair evaluation, both methods are tested on SANA-Sprint 0.6B using 2 NFEs. Method IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ ℒSDSL_SDS [35] 11.18 26.50 6.10 27.12 ℒregL_reg (Ours) 11.40 27.22 6.12 27.15 Figure 7: Qualitative comparison between our proposed regularization loss (ℒregL_reg) and the Score Distillation Sampling (SDS) loss (ℒSDSL_SDS) from SwiftEdit [35]. This visualization shows that our ℒregL_reg is crucial for preserving the original image content, while using ℒSDSL_SDS leads to significant information loss and poor reconstruction. Method IR×10 ↑ HPS×102_× 10^2 AS ↑ CLIP↑ Runtime (seconds)↓ DDIMInv (w/o Blending) (50 steps) + SDXL-Turbo 4.10 22.95 5.30 26.40 4.18 DDIMInv (w/ Blending) (50 steps) + SDXL-Turbo 12.11 28.21 6.04 27.32 4.32 InverFill (Ours) + SDXL-Turbo 12.38 28.44 6.08 27.67 0.74 Table 6: Quantitative comparison of one-step InverFill versus the 50-step DDIM inversion baseline on BrushBench. For a fair comparison, we run SDXL-Turbo with 4 NFEs. 9 Ablation of Proposed Components To better understand the contribution of each part in our framework, we conducted an ablation study on both SANA-Sprint 0.6B (Tab. 7) and SDXL-Turbo (Tab. 8). We established a baseline for comparison by training a model with the reconstruction loss from Sec. 4.3, using the masked image as input. From this starting point, we then incrementally added our proposed components: Re-Blending (Sec. 4.4), Gaussian Regularization (Sec. 4.5), and the LADD adversarial loss (Sec. 4.6). Our results show that each component contributes incremental gains in performance. As shown in Tab. 7, introducing the Re-Blending operation increases the IR score from 7.93 to 11.11. The further addition of Gaussian Regularization expands this improvement, and incorporating the LADD adversarial loss leads to the highest scores, with an IR of 11.65 and an HPS of 27.93. A similar pattern of improvement is also noted in the experiments with SDXL-Turbo (Tab. 8). This evaluation suggests that all three components contribute effectively, collectively leading to the performance of the full InverFill model. Table 7: Ablation of components on SANA-Sprint 0.6B (2 NFEs). Method IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ !15 Baseline [35] 7.93 24.79 5.96 26.40 InverFill + Re-Blending (Sec. 4.4) 11.11 26.68 6.08 27.13 + Gaussian Reg. (Sec. 4.5) 11.40 27.22 6.12 27.15 + LADD (Sec. 4.6) 11.65 27.93 6.15 27.17 Table 8: Ablation of components on SDXL-Turbo (4 NFEs). Method IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ !15 Baseline [35] 10.64 26.46 6.03 26.56 InverFill + Re-Blending (Sec. 4.4) 11.33 27.18 6.03 27.16 + Gaussian Reg. (Sec. 4.5) 12.14 28.14 6.06 27.57 + LADD (Sec. 4.6) 12.38 28.44 6.08 27.67 10 Other Inversion Approaches We quantitatively compare InverFill with a 50-step DDIM inversion process, using SDXL for inversion and SDXL-Turbo blended sampling for inpainting. Based on Fig. 8, directly applying DDIM inversion to a masked image fails to encode the masked regions, producing smooth, gray, null-like structures in those areas. This loss of content significantly degrades performance, as reflected in the low scores reported in the first row of Tab. 6. Next, we apply our proposed Re-Blending operation (Sec. 4.4) to the DDIM-inverted noise. While this fills the previously null-like regions, the resulting model (Row 2 in Tab. 6) still struggles with scene harmonization. In contrast, our one-step InverFill method (Row 3) achieves higher scores across all metrics and is significantly more efficient, running in just 0.74 seconds, nearly six times faster than the 50-step DDIM process (4.32 seconds). Combined with the improved qualitative harmonization in Fig. 9, these results demonstrate that InverFill is both substantially more effective and practical. Figure 8: Visualization of DDIM Inversion results. The masked regions are not encoded, producing smooth, null-like areas in the inverted noise and causing loss of content. Figure 9: Qualitative comparison between InverFill and DDIM Inversion. InverFill achieves substantially better scene harmonization and semantic consistency. Table 9: Quantitative results on FFHQ, DIV2K, and BrushBench. Red, Blue, and Black denote scores on FFHQ, DIV2K, and BrushBench, respectively. Method NFEs FFHQ / DIV2K / BrushBench FID↓ IR×10↑ HPS×102_× 10^2↑ AS↑ CLIP↑ LPIPS↓ SSIM↑ SANA-Sprint 0.6B 2 27.12 2.81 / 5.22 22.42 / 26.78 5.12 / 5.77 23.55 / 28.26 0.184 / 0.193 / 0.144 0.704 / 0.572 / 0.769 !15SANA-Sprint 0.6B + InverFill 2 26.53 5.27 / 5.87 23.26 / 27.17 5.31 / 5.89 23.65 / 28.43 0.172 / 0.182 / 0.138 0.719 / 0.575 / 0.771 SANA-Sprint 0.6B 4 27.32 2.66 / 5.25 22.50 / 26.83 5.17 / 5.79 23.84 / 28.31 0.184 / 0.192 / 0.140 0.706 / 0.573 / 0.774 !15SANA-Sprint 0.6B + InverFill 4 26.42 5.27 / 5.83 23.32 / 27.15 5.37 / 5.92 23.88 / 28.38 0.169 / 0.181 / 0.134 0.708 / 0.574 / 0.774 SDXL Turbo 4 26.32 7.37 / 4.71 25.73 / 26.81 5.67 / 5.92 25.24 / 28.21 0.269 / 0.292 / 0.139 0.626 / 0.454 / 0.813 !15SDXL Turbo + InverFill 4 25.90 8.35 / 5.27 26.14 / 27.03 5.76 / 5.95 25.29 / 28.25 0.262 / 0.287 / 0.133 0.655 / 0.455 / 0.815 SDXL Turbo + BrushNet 4 25.55 7.86 / 5.11 25.05 / 26.05 5.53 / 5.76 24.72 / 28.41 0.204 / 0.469 / 0.185 0.728 / 0.292 / 0.755 !15SDXL Turbo + BrushNet + InverFill 4 25.49 7.91 / 5.17 25.17 / 26.18 5.55 / 5.79 24.85 / 28.39 0.206 / 0.469 / 0.178 0.727 / 0.293 / 0.757 11 Additional Experiments We evaluate InverFill on FFHQ [18] and DIV2K [1] to assess robustness across diverse mask configurations and standard benchmarks, with results reported in Tab. 9. For all evaluations, we use the same checkpoints as in Sec. 5 without any modification or fine-tuning. Datasets. For FFHQ, we sample 10K images. For DIV2K, we use 900 images from the training and validation sets. Following the same settings in Sec. 5.5, prompts are generated using Qwen-3 [58]. Mask Settings. For both FFHQ and DIV2K, we adopt LaMa’s [51] strategy with polygonal thick- and thin-stroke masks, and additionally include rectangular masks covering half of the image. Masks are randomly sampled from these configurations to ensure a diverse evaluation. Additional Metrics. In addition to perceptual quality metrics, we report LPIPS and SSIM to assess consistency, including results from the BrushBench evaluation. For FFHQ, we additionally report FID [12]. 12 Analysis of the Inversion Effect We analyze the effect of the inversion network to explain why initializing from well-aligned noise yields more coherent and consistent outputs. Our intuition is that such noise encodes the blending trajectory and preserves background information, thereby enabling smoother blending during the denoising process. To further validate this observation, we compute LPIPS between the predicted x0x_0 in background regions at intermediate timesteps and the input image, and report the results in Fig. 10. We observe that initialization with well-aligned noise consistently yields significantly lower LPIPS than random initialization, supporting our hypothesis. Moreover, Fig. 10 provides insight into the effectiveness of the Gaussian regularization loss: the Jensen–Shannon divergence (JSD) with respect to the Gaussian distribution is substantially reduced when this regularization is applied, leading to better-aligned latent noise while also satisfying the required Gaussian distribution for diffusion models, and consequently yielding stable and coherent reconstructions. Figure 10: Quantitative analysis of inversion effects. We report LPIPS in background regions at intermediate timesteps and JSD with respect to the Gaussian prior. Lower values indicate better alignment. Red bars denote results without InverFill, while Green bars denote results with InverFill. 13 Failure Cases We report representative failure cases in Fig. 11. Overall, the main limitation of our method stems from color inconsistencies between the inpainted region and the background. Figure 11: Representative failure cases of our method. While InverFill improves overall coherence, it may produce color inconsistencies between the inpainted regions and the background. 14 Societal Impacts Our work aims to provide a practical tool for creative professionals, facilitating tasks such as photo restoration and object removal. We acknowledge that realistic image manipulation technologies can be misused to generate deceptive content. To mitigate such risks, we advocate for the parallel development of detection methods [32, 10, 37] for AI-manipulated media and encourage the responsible use of these technologies. 15 More Qualitative Results To provide a comprehensive visual comparison of InverFill, Figs. 12 and 13 present an expanded set of qualitative results, further illustrating the improvements in coherence and background harmonization highlighted in our work. Figure 12: More qualitative comparison on BrushBench (Zoom in for best view) Figure 13: More qualitative comparison on BrushBench (Zoom in for best view)