Paper deep dive

GuardPaint:SpeculativeSafetyDecodingforText-to-ImageGeneration

Shreyash Dhoot, Paras Dhiman, Arsh Abbas Naqvi, Aranbi Dutta, Aman Chadha, Vinija Jain, Amitava Das

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 111

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/25/2026, 7:35:26 AM

Summary

GuardPaint is a speculative decoding framework designed to enhance safety in Text-to-Image (T2I) diffusion models by intervening during the denoising trajectory. It utilizes a lightweight auditor to detect and localize unsafe regions in intermediate images, a policy-aligned inpainter to generate safe repair candidates, and a guarded tournament to select edits that improve policy compliance while preserving prompt fidelity and perceptual quality. The method is plug-and-play, requiring no modification to the base model weights, and is evaluated across UNet-based (SD 1.5, SDXL, SD 3.5) and flow-matching (FLUX.1-dev) architectures against various jailbreak attacks.

Entities (14)

GuardPaint · method · 100%Auditor-Scorer · component · 95%FLUX.1 [Dev] · model · 95%Guarded Tournament · mechanism · 95%Policy-Aligned Inpainter · component · 95%SDXL · model · 95%Stable Diffusion 1.5 · model · 95%Stable Diffusion 3.5 · model · 95%DACA · attack · 90%JailBreakDiffBench · benchmark · 90%MMA · attack · 90%PGJ · attack · 90%RABell · attack · 90%SneakPrompt · attack · 90%

Relation Signals (13)

GuardPaint → testedonmodel → Stable Diffusion 1.5

confidence 95% · UNet/flow-matching models including SD~1.5... GuardPaint reduces attack success

GuardPaint → testedonmodel → SDXL

confidence 95% · UNet/flow-matching models including... SDXL... GuardPaint reduces attack success

GuardPaint → testedonmodel → Stable Diffusion 3.5

confidence 95% · UNet/flow-matching models including... SD~3.5... GuardPaint reduces attack success

GuardPaint → testedonmodel → FLUX.1 [Dev]

confidence 95% · UNet/flow-matching models including... FLUX.1-dev... GuardPaint reduces attack success

GuardPaint → uses → Auditor-Scorer

confidence 95% · A lightweight auditor monitors intermediate images, localizes unsafe regions, and triggers surgical inpainting repair only where needed.

GuardPaint → uses → Policy-Aligned Inpainter

confidence 95% · Candidate repairs are generated by a policy-aligned inpainter

GuardPaint → uses → Guarded Tournament

confidence 95% · selected through a guarded tournament that accepts edits only when they improve policy compliance

GuardPaint → evaluatedon → JailBreakDiffBench

confidence 90% · We evaluate under the JailBreakDiffBench protocol

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Text-to-image (T2I) diffusion models offer powerful visual generation, but their controllability creates a critical safety challenge: adversarial prompts can steer the denoising trajectory toward policy-violating content such as explicit nudity or graphic violence. Existing safeguards mostly act before generation through prompt filtering or after generation through image classification, leaving the diffusion process itself unguarded and often yielding only refusal rather than safe visual repair. We introduce GuardPaint, a speculative decoding framework for safe T2I generation that intervenes inside the diffusion trajectory without modifying the base model. A lightweight auditor monitors intermediate images, localizes unsafe regions, and triggers surgical inpainting repair only where needed. Candidate repairs are generated by a policy-aligned inpainter and selected through a guarded tournament that accepts edits only when they improve policy compliance while preserving prompt fidelity and perceptual quality. Across five jailbreak families SneakPrompt, MMA, PGJ, DACA, and RABell and UNet/flow-matching models including SD~1.5, SDXL, SD~3.5, and FLUX.1-dev. GuardPaint reduces attack success and harmful generations with minimal degradation to image quality, prompt fidelity, and benign behavior. Content warning: This paper contains examples involving nudity and violence that some readers may find disturbing, distressing, or offensive.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

107,602 characters extracted from source content.

Expand or collapse full text

GuardPaint: Speculative Safety Decoding for Text-to-Image Generation * Shreyash Dhoot 1 , Paras Dhiman 1 , Arsh Abbas Naqvi 1 , Arnabi Dutta 1 Aman Chadha 2,† , Vinija Jain 3,† , Amitava Das 1 2 Apple, USA 3 Meta, USA 1 Pragya Lab, BITS Pilani Goa, India Abstract Text-to-image (T2I) diffusion models offer powerful visual generation, but their controlla- bility creates a critical safety challenge: adver- sarial prompts can steer the denoising trajectory toward policy-violating content such as explicit nudity or graphic violence. Existing safeguards mostly act before generation through prompt fil- tering or after generation through image classi- fication, leaving the diffusion process itself un- guarded and often yielding only refusal rather than safe visual repair. We introduce GuardPaint, a speculative decod- ing framework for safe T2I generation that in- tervenes inside the diffusion trajectory without modifying the base model. A lightweight au- ditor monitors intermediate images, localizes unsafe regions, and triggers surgical inpainting repair only where needed. Candidate repairs are generated by a policy-aligned inpainter and selected through a guarded tournament that ac- cepts edits only when they improve policy com- pliance while preserving prompt fidelity and perceptual quality. Across five jailbreak families—SneakPrompt, MMA, PGJ, DACA, and RABell—and UNet/flow-matching models including SD 1.5, SDXL, SD 3.5, and FLUX.1-dev. GuardPaint reduces attack success and harmful generations with minimal degradation to image quality, prompt fidelity, and benign behavior. Content warning: This paper contains examples involving nudity and violence that some readers may find disturbing, distressing, or offensive. 1 The Case for Plug-and-Play Safety Alignment in Text-to-Image Generation Text-to-image (T2I) diffusion models now produce high-fidelity images from open-ended prompts, but * To further research in the field we release our code here. † This work was conducted outside the authors’ primary professional roles. the same controllability creates a sharp safety fail- ure mode: adversarial prompts can steer the de- noising trajectory toward policy-violating content, including explicit nudity and graphic violence. Re- cent attacks exploit prompt obfuscation, token sub- stitution, multilingual or Unicode perturbations, and semantic rewriting to bypass surface-level safe- guards (Yang et al., 2024b,a; Tsai et al., 2024; Deng and Chen, 2024; Ma et al., 2025; Struppek et al., 2023; Ba et al., 2024). As T2I systems move into creative and public-facing deployment, safety mechanisms must block harmful content without unnecessarily degrading benign generations. Existing defenses mostly protect the boundaries of generation. Prompt-level safeguards detect, rewrite, or normalize unsafe inputs before sam- pling (Wu et al., 2024; Liu et al., 2024b), while post-hoc classifiers intervene only after an image has already been produced. These methods can refuse or flag content, but they do not guard the diffusion trajectory and rarely provide a compli- ant visual alternative. Other methods modify the generator itself through concept erasure, model editing, safe denoising, inference-time steering, or preference alignment (Schramowski et al., 2023; Gandikota et al., 2023; Meng et al., 2026; Kim et al., 2025; Wallace et al., 2024; Li et al., 2024a; Borso et al., 2025; Hong et al., 2025; Park et al., 2024; Li et al., 2024b; Zhang et al., 2025; Liu et al., 2024a). While effective in specific settings, they of- ten require model-specific retraining, global distri- butional changes, or do not explicitly repair unsafe spatial regions during decoding. This motivates a central systems question: Can safety alignment be introduced as a modular decoding-time layer without modifying the underlying generator? We argue that such a layer requires moving from boundary-level filtering to trajectory-level repair. Rather than rejecting prompts before sampling or arXiv:2608.21869v1 [cs.CV] 22 Aug 2026 Table 1: Positioning of GUARDPAINT. Prior T2I safety methods filter prompts, classify outputs, edit weights, steer generation, or align model distributions. GUARDPAINT combines trajectory-level intervention, localized repair, safe visual alternatives, plug-and-play deployment, and tournament-based repair selection. Method family Trajectory level Localized repair Safe visual alternative No base retraining Tournament selection Prompt filtering / rewriting (Wu et al., 2024; Liu et al., 2024b) ✗✓✗ Post-hoc safety classifiers✗✓✗ Concept erasure / model editing (Schramowski et al., 2023; Gandikota et al., 2023) ✗ Inference-time safety steering (Meng et al., 2026; Kim et al., 2025) ✓✗✓✗ Preference-aligned diffusion models (Wallace et al., 2024; Li et al., 2024a; Borso et al., 2025; Hong et al., 2025; Park et al., 2024; Li et al., 2024b; Zhang et al., 2025) ✗✓✗ Preference-tuned inpainting (Liu et al., 2024a)✗✓✗ GUARDPAINT✓ classifying images after generation, a safety layer should monitor the denoising process itself, detect when unsafe structure emerges, and intervene lo- cally. The desired intervention is plug-and-play across evolving T2I backbones, localized to unsafe regions, and selective enough to reject edits that harm prompt fidelity or perceptual quality. GUARDPAINT follows this premise.In- spired by speculative decoding in language mod- els (Leviathan et al., 2023; Chen et al., 2023), the frozen diffusion generator acts as a draft model and a lightweight auditor acts as a verifier. At se- lected denoising steps, the auditor localizes unsafe regions; a policy-aligned inpainter proposes candi- date repairs; and a guarded tournament ranks these candidates, accepting an edit only when it improves auditor-defined policy compliance while preserv- ing fidelity and perceptual quality. The result is a plug-and-play safety alignment layer that converts unsafe generations into safe, semantically coherent visual alternatives rather than blank refusals. Contributions. • We introduce plug-and-play safety alignment for T2I diffusion models via decoding-time tra- jectory intervention instead of generator retrain- ing. •We propose GUARDPAINT, a speculative safety decoding framework combining adversarial au- diting, localized inpainting repair, and guarded candidate ranking. • We introduce a guarded tournament align- ment mechanism that accepts repairs only when they improve auditor-defined policy compliance while preserving fidelity and perceptual quality. •We demonstrate deployment across UNet-based and flow-matching T2I architectures without modifying base-model weights. 2 GuardPaint: Speculative Safety Decoding Overview. Letpbe the prompt and letz t 0 t=T denote the denoising trajectory of a frozen gen- eratorΦ base .GUARDPAINT treats this trajec- tory as a sequence of auditable intermediate states. At selected timestepst ∈ T audit , the la- tent is decoded into an audit viewI t = Dec(z t ). A lightweight Auditor-ScorerA s estimates three quantities: whether intervention is needed, where unsafe content is localized, and whether a proposed repair preserves safety, faithfulness, and percep- tual quality. If no violation is detected, denoising continues unchanged. If a violation is detected, a policy-aligned inpainterA g proposes local re- pairs, and a guarded tournament accepts an edit only when it improves the calibrated safety objec- tive while passing fidelity and quality gates. This yields a draft–audit–repair–verify loop. The frozen diffusion model is the draft; the audi- tor is the verifier; the inpainter proposes localized repairs; and the tournament performs selective ac- ceptance. The key invariant is an auditor-defined non-regression guard: when no candidate passes the guard, the base trajectory is preserved. Figure 1: GUARDPAINT inference pipeline. At audited denoising steps, the current latent is decoded into an intermediate image and passed to the Auditor-Scorer. If no unsafe region is detected, the base trajectory continues unchanged. If a suspect region is detected, the tournament policy proposesNinpainting configurations, the policy-aligned inpainter generatesNlocal repair candidates, and the guarded tournament selects a winner for latent reinsertion. 2.1 Auditor-Scorer: Detecting When and Where to Intervene The Auditor-ScorerA s is a multi-task model with three roles: trigger repair, localize unsafe regions, and score candidate repairs. It uses a ResNet-101 backbone pretrained on ImageNet (He et al., 2016). Promptandtimestepconditioning. A lightweight BiLSTM text encoder with word embeddings (d text = 512) encodes the prompt. Spatial image features attend to prompt-token features through 8-head cross-attention with pre-LayerNorm, yielding a prompt-conditioned visual representation for the faithfulness branch. A 3-layer timestep MLP,128→ 256→ 512with SiLU activations, embedst/T ∈ [0, 1]. Two FiLM projections (Perez et al., 2018) produce(γ,β) modulation parameters, allowing the auditor to calibrate risk differently for early noisy views and late semantically formed images. Multi-task outputs. The auditor produces six outputs, organized by function: 1.Triggering:binary adversarial probabil- ityˆy adv ∈ [0, 1]and class logits over safe, nudity, violence. 2. Localization:per-class risk mapsr c ∈ [0, 1] H×W used to mine unsafe spatial regions. 3.Tournament scoring: prompt faithfulnessF, seam/perceptual qualityP, and relative adver- sarial suppression strength B ∈ [0, 1]. The faithfulness branch is isolated from safety- label gradients: safety labels do not update the image–text alignment head, preserving a clean fi- delity signal for tournament gating. Region mining. At inference, the adversarial map headConv2d(2048, 1, 1) → Sigmoidpro- ducesr adv ∈ [0, 1] H ′ ×W ′ . This map is bilinearly upsampled to512× 512, thresholded at the 85th percentile, and feathered with a Gaussian kernel ClassPRF1 Safe0.900.970.94 Nudity0.900.780.84 Violence0.850.890.87 Macro avg0.880.880.88 Table 2: Auditor-Scorer classification performance on the held-out test set (overall accuracy: 88%). (σ = 5, 15× 15) to smooth mask boundaries be- fore inpainting. Training objective.The auditor is trained with a factorized multi-task objective that separates trig- gering, harm typing, repair quality, and prompt fidelity: L A s = L adv |z repair trigger +0.5 L class |z harm type +0.4 L rel-adv |z suppression score + 0.3 L seam |z visual quality +0.5 L InfoNCE | z prompt fidelity . HereL adv gates whether repair is triggered;L class selects the harm category and risk map;L rel-adv estimates adversarial strength;L seam penalizes boundary artifacts; andL InfoNCE (van den Oord et al., 2018) supervises image–text faithfulness in- dependently of safety labels. Dataset and labeling. We assemble 82,000 image–prompt pairs from eight public Hugging- Face datasets covering safe, nudity, and violence categories. Images without prompts are captioned using Qwen2.5-VL (Bai et al., 2025). All images are relabeled with InternVL-3.5 (8B) (OpenGVLab et al., 2025) intosafe, nudity, violence, selected after qualitative labeler ablations against four alter- native VLMs (Table 4). We use a 70/15/15 split; dataset sources appear in Table 3. 2.2 Policy-Aligned Inpainter: Learning Safe Local Repairs The inpainterA g converts a masked unsafe region into a visually coherent, policy-compliant alterna- tive. We train it in two stages. The first stage teaches what safe repair looks like; the second teaches when safe repair should dominate un- safe completion. This mirrors SFT-then-alignment in language-model safety and follows the staged preference-tuning pattern used in inpainting (Liu et al., 2024a). Stage 1: Refusal supervised fine-tuning. We start from Stable Diffusion 1.5 Inpainting and apply LoRA fine-tuning (Hu et al., 2022) withr = 128 andα = 128on attention modules, feed-forward layers, spatial projection layers, and the UNet in- put convolution. Training examples pair unsafe prompts and masked unsafe images with safe tar- get completions, using masks derived from auditor heatmaps. A Min-SNR weighted MSE loss (Hang et al., 2023) is applied acrosst∈ [0, 1000], teach- ing the model a stable safe repair manifold for masked adversarial regions. The LoRA weights are merged into the base checkpoint before alignment. Stage 2: Binary Classifier Optimization. On the SFT-merged model, we apply Binary Classifier Optimization (BCO) (Jung et al., 2024) with binary safe/unsafe labels. Each training step samples a clean latent z 0 , timestep t, and Gaussian noise ε: t∼U [0, 1000), ε∼N (0,I), z t = √ ̄α t z 0 + √ 1− ̄α t ε. The trainable denoiserε θ and frozen referenceε ref receive the same tuple(z t ,t,m ℓ , ̃z 0 ,p), where ̃z 0 is the masked latent andm ℓ is the latent-space mask. Masked reconstruction reward.Both denoisers are converted into clean-latent estimatesˆz θ 0 andˆz ref 0 . Alignment is driven by the masked reconstruction gap: g = ℓ ref − ℓ θ , ℓ • = P 1 + λ m m ℓ ˆz • 0 − z 0 2 m ℓ P m ℓ ,λ m = 0.5. A positivegmeans the trainable model reconstructs the masked region better than the reference. This is rewarded for safe examples and penalized for unsafe ones. BCO loss. An EMA baseline stabilizes the re- ward boundary: r = g− δ ema , δ ema ∈ [−0.03, +0.03], sgn = 2y− 1, y ∈0, 1. The BCO objective is L BCO = E[softplus (−sgn· βr)], β = 50. Weuseclassweights1.0/5.0/12.0for safe/nudity/violence,imposinghighercost on unsafe misses. To reduce the single-step/multi- step mismatch, every 10 gradient steps we run K−1deterministic DDIM steps (Song et al., 2021a) undernograd, re-noise tot final =⌊t/K⌋, and apply a second BCO update at the lower noise level. This teaches repair at the stage where unsafe structure is visually coherent. Full details are in Appendix ??. Figure 2: Tournament example: rows show exam- ples of proposed inpainted candidates in a tourna- ment for different adversarial prompts. The yellow border marks the WINNER candidate for that tourna- ment round.(Translucent red mask shows the the binary mask/the inpainted area) 2.3 Guarded Tournament: Ranking Repairs Under Safety Gates Naively sampling inpainting hyperparameters wastes compute: many configurations create ar- tifacts, near-duplicates, or safety gains obtained by destroying prompt fidelity. GUARDPAINT instead learns a tournament policy that proposes repairs with high expected guarded utility. Policy and state.The policyπ θ is a 3-layer MLP (256→ 128→ 64, SiLU, LayerNorm). It maps a compact repair state to distributions over con- tinuous knobs—CFG scale, mask dilation, mask feather, noise jitter, inversion depth—and a discrete seed bucket: s = concat h proj(e text ), proj(z t ), proj(e img ), proj( ̄m), t/T i ∈ R 257 . Each projection is 64-dimensional,e text comes from the frozen auditor BiLSTM,e img is the audi- tor image embedding, and ̄mis the mask coverage fraction. Candidate scoring.For a flagged region(R,m), the policy samplesN = 5configurationsa i N i=1 . The inpainter produces repairsC i N i=1 , which are composed into the audit view and scored by A s : (S i ,F i ,P i ,B i ) = A s (p, Compose(I t ,C i ,R)). HereS i is policy safety,F i is prompt faithfulness, P i is seam/perceptual quality, andB i is adversarial suppression confidence. Guarded utility. Each candidate is ranked by a safety gain gated by quality and fidelity: u i = (S i − S 0 − δ) + |z safety gain ·1[P i ≥ τ P ] | z quality gate · 1[F i ≥ τ F ] |z fidelity gate ·B i |z suppression confidence . S 0 is the unedited control score,δ = 0.01is the im- provement margin, andτ P ,τ F are calibrated gates. The winning edit isi ∗ = arg max i u i and is ac- cepted if u i ∗ > 0. Otherwise, the unedited control is retained. Thus, accepted edits are strict improve- ments under the calibrated auditor objective; all failed repairs leave the original trajectory intact. We use time-varying thresholds that change as a function of the normalized timestep t norm = t/T : τ P (t norm ) = 0.40 + 0.25· t norm τ F (t norm ) =              0.30 + 0.30t norm , t norm < 0.85, 0.55− 0.10 t norm −0.85 0.15 , t norm ≥ 0.85. Tournament policy objective. Utilities are con- verted into leave-one-out softmax credits: w i = softmax u i τ − 1 N , τ = std u i N i=1 . The offline policy objective is L tour =− N X i=1 w i logπ θ (a i | s)− λ H H[π θ (·| s)] + λ c Cost(a i N i=1 )− λ div X i<j d(C i ,C j ). The entropy term prevents mode collapse, the cost term discourages expensive settings such as deep in- version or excessive CFG, and the diversity term en- courages distinct candidates. We useλ cont H = 0.01, λ disc H = 0.005,λ c = 0.005, andλ div = 0.01, with mini-batches of 4 rollouts and gradient clipping at 1.0. 2.4 Latent Reinsertion Across Diffusion Families After a repair is selected, it must be reintroduced into the active generation trajectory without disrupt- ing the target latent distribution. To achieve this across distinct architectural families, the reinser- tion mapping is strictly governed by the underlying base model’s specific forward diffusion or flow- matching equations. Null-text inversion for UNet-based models. For SD 1.5 and SDXL, we use null-text inver- sion (Mokady et al., 2023a). A per-step null-text embedding is optimized for 10 gradient steps (lr = 0.01) to reconstruct the winning inpaint from the live latent. The edit is then blended only inside the feathered mask: z t−1 = (1− α m )z ctrl t−1 + α m z edit t−1 . Hereα m is the mask upsampled to latent resolu- tion. This keeps the repair on the base model’s denoising manifold and avoids drift from naive VAE re-encoding. Flow-ODE reinsertion for flow-matching mod- els.For SD 3.5 and FLUX.1, Null-text inversion schedules are inapplicable. We instead use the rectified-flow relation (Liu et al., 2023): z t = (1− t)z 0 + tε, ˆz t = (1− t norm )ˆz 0 + t norm ε. The selected repair is encoded toˆz 0 and projected forward to the current normalized time. At late audit steps (t norm < 0.25), where the trajectory is nearly deterministic, we directly blendˆz 0 into the live latent. DDPM-noise blending and DDIM inversion are implemented as additional reinsertion baselines and compared in Appendix ??. 3 Full Pipeline Algorithm Algorithm 1 summarizes one audited decoding step. The base model first advances normally, produc- ing a control latentz ctrl t−1 and audit viewI t . If the current timestep is not audited, the control latent is returned. Otherwise, the auditor runs once. A benign image continues at the cost of a single au- ditor pass. A flagged image triggers mask con- struction, candidate proposal, guarded scoring, and latent reinsertion only if the winning candidate has positive utility. Audit onset. Coherent visual structure typically appears after roughly 70–80% of the denoising trajectory. Earlier audits are inefficient because decoded views are noise-dominated; later audits focus compute where unsafe semantic structure is visible and repairable. Algorithm 1 GUARDPAINT Decoding Step at Timestep t Require:Base modelΦ base , auditorA s , inpainterA g , frozen policy π θ , prompt p, latent z t , thresholds (δ,τ P ,τ F ) 1: z ctrl t−1 ,I t ← Φ base (z t ,p,t), Dec(z ctrl t−1 ) 2: if t /∈T audit then 3:return z ctrl t−1 4: end if 5: r 0 ← A s (p,I t )▷ one auditor pass 6: ifr 0 .advprob <0.40andr 0 .harmclass /∈ nudity, violence then 7:return z ctrl t−1 ▷ benign path 8: end if 9: S 0 ← r 0 .policy safe 10: m← BuildMask(r 0 .heatmap) 11: s← StateEncode(p,z t ,r 0 .img embed, ̄m,t/T ) 12: a i N i=1 ← π θ (s) 13: C i N i=1 ← A g (I t ,m,p;a i ) 14: for i = 1,...,N do 15:(S i ,F i ,P i ,B i )← A s (p,C i ) 16: u i ← (S i − S 0 − δ) + 1[P i ≥ τ P ]1[F i ≥ τ F ]B i 17: end for 18: i ∗ ← arg max i u i 19: if u i ∗ > 0 then 20: z t−1 ← Reinsert(C i ∗ ,z ctrl t−1 ,m,t/T ) 21: else 22: z t−1 ← z ctrl t−1 ▷ auditor-defined non-regression 23: end if 24: return z t−1 Complexity. Each audited step begins with one auditor passC A s . For benign prompts, this is the only extra cost: Cost t = C A s if no trigger fires. When repair is triggered, the cost scales with mined regions and candidate repairs: Cost t = C A s + K t N C A g + N C A s + C reinsert , whereK t is the number of mined regions,Nis the tournament size, andC A g is one inpainting pass. Initial text and image embeddings are cached and reused across candidates. The tournament policy reduces wasted compute by biasing proposals to- ward high-utility configurations, lower inversion depth, and fewer exhausted candidate sets. Figure 3: Change-from-baseline heatmap under the JailBreakDiffBench protocol (Luo et al., 2024). Each cell shows the absolute change in , ASR (↓), of GUARDPAINT relative to the undefended base model. Blue cells indicate improvement; red cells indicate regression. Rows are attack families; columns are model architectures. Baselines are sourced from JailBreakDiffBench. Figure 4: Ablation: number of audited steps (X) vs. candidates per tournament (N) across five architectures. Mean ASR (↓) is shown as a function of audited denoising steps forN ∈1, 3, 5, 10. ASR decreases with bothX andN, with diminishing returns beyondN = 5.Flow-matching models such as FLUX.1 converge faster because their trajectories are more deterministic. 4 Experimental Setup 4.1 Adversarial Attack Benchmarks We evaluate under the JailBreakDiffBench pro- tocol (Luo et al., 2024) on five black-box prompt- space attack families, where the adversary has only query access to the model with no knowledge of parameters, gradients, or latent representations: SneakPrompt(Yang et al., 2024b): formulates jailbreak as an RL-guided search that jointly opti- mizes adversarial prompts for semantic similarity to harmful targets and successful safety-filter eva- sion. MMA(Yang et al., 2024a): leverages offline CLIP-based guidance to iteratively perturb or re- place tokens while preserving original prompt se- mantics in embedding space. PGJ(Ma et al., 2025): uses ChatGPT-generated antonym concepts to guide embedding-space op- timization of stealthy adversarial prompts that by- pass safety filters. DACA(Deng and Chen, 2024): employs LLM- driven semantic rewriting to substitute trigger words with contextually benign alternatives, pro- ducing prompts that are textually innocent but se- mantically adversarial. RABell(Tsai et al., 2024): applies surrogate CLIP-based guidance to iteratively replace tokens while maintaining surface-level naturalness and fil- ter evasion. 4.2 Base Models We attach GuardPaint to five T2I architectures span- ning UNet-based and flow-matching designs: Sta- ble Diffusion 1.5 (SD 1.5; runwayml) (Rombach et al., 2022) SDXL Base 0.9 (SDXL; stabilityai) (Podell et al., 2023) Stable Diffusion 3.5 Medium (SD 3.5-M; stabilityai)(Esser et al., 2024) Stable Diffusion 3.5 Large-Turbo (SD 3.5-LT; stabilityai) FLUX.1-dev (Black Forest Labs; flow-matching) (Black Forest Labs, 2026) The Policy finetuned inpainter (A g ) always uses SD 1.5 Inpainting, re- gardless of the base model family. 5 Results 5.1 Main Results We evaluate under the JailBreakDiffBench proto- col (Luo et al., 2024), which provides standardized adversarial prompt generation, baseline results, and evaluation scripts for fair cross-system comparison; all baseline ASR values are taken directly from that benchmark. Table 3 reports absolute attack success rates for all five attack families across four base ar- chitectures, before and after GuardPaint defense. Across all model–attack pairs, GuardPaint reduces ASR without exception. The gains are largest on UNet-based models: on SD 1.5, DACA drops from 8.25% to 1.94% and SneakPrompt from 7.00% to 1.79%; on SDXL, MMA is reduced to near zero (5.25%→3.92%). Flow-matching models (SD 3.5M, SD 3.5-LT, FLUX.1-dev) be- gin from lower baselines owing to stronger safety pretraining, yet GuardPaint still achieves consis- tent reductions—for instance, SneakPrompt on FLUX.1-dev falls from 8.25% to 6.99% and MMA from 4.25% to 2.44%. Prompt fidelity and image quality remain sta- ble across all defended configurations. AlignS and BLIP scores stay within±0.05 of the undefended baseline, confirming that inpainting repairs are con- fined to the flagged region and leave the semanti- cally valid portions of the prompt intact. Flow-matching models present a distinct failure mode: when an attack succeeds despite defense, au- diting and inpainting at early denoising steps causes the base model to partially override the reinserted latent, producing blurred or incoherent regions. 6 Ablation Study We ablate the two primary search parameters: steps audited (X) and candidates per tournament (N), which directly govern the security–compute trade- off. Figure 4 reports ASR across an attack as a function ofX ∈ 2, 4, 6, 8forN ∈ 1, 3, 5, 7, evaluated on all five base architectures. ASR decreases monotonically with bothXand Nacross all model families. Diminishing returns are pronounced: the gain fromN = 1 → N = 3 substantially exceeds that fromN = 5 → N = 7, and similarly forXbeyond steps. Flow-matching models (FLUX.1) converge faster inXdue to their more linear and deterministic trajectories, where a single late-stage audit step captures most of the Figure 5: Qualitative comparison. Each row shows an adversarial prompt alongside the undefended baseline output, and the GuardPaint output.The policy aware in- painter produces semantically coherent safe alternatives rather than blank refusals, preserving scene context out- side the flagged region. safety benefit. These results motivate our produc- tion setting ofN = 5and auditing the final two de- noising steps, which sits on the knee of the quality– latency Pareto curve across all tested architectures (marked on each panel in Figure 4). 7 Conclusion We presented GuardPaint , a decoding-time safety framework for T2I diffusion models inspired by speculative decoding in LLMs. By monitoring the diffusion trajectory with a multi-task auditor, rewriting suspect regions with a BCO-aligned in- painter, and efficiently proposing winning config- urations with the Tournamnet policy, The Guard- Paint framework provides compliant alternatives to adversarial outputs rather than blank refusals— without modifying any base model weights.The framework is plug-and-play across seven T2I ar- chitectures spanning four generations of design: DDPM-UNet (SD 1.5, SDXL), flow-matching- UNet (SD 3.5), and flow-matching-transformer (FLUX.1). 8 Limitations Latency. On a single NVIDIA A6000 (48 GB), typical overhead is∼7–8 seconds per audited timestep withN = 5for UNet-based architectures (SD 1.5, SDXL), where each of the 5 SD 1.5 in- painting passes takes approximately 1.4 seconds; on FLUX.1-dev, flow-ODE reinsertion raises this to∼20 seconds per audited timestep. Averaged across all audit steps, the expected per-step over- head is∼2 seconds for UNet architectures. Single-family inpainter.A g is always an SD 1.5 Inpainting checkpoint, regardless of the base model family. While reinsertion bridges the resolution and latent-space mismatch, a native inpainter for each base family would reduce the fidelity cost and better preserve the stylistic properties of higher- capacity generators such as FLUX.1. Coarse policy taxonomy. The auditor and in- painter are trained on three crude categories: safe, nudity, violence. This taxonomy is in- sufficient for subtler policy violations that do not manifest as localized explicit content. Harmful stereotyping, racial or gender bias encoded in vi- sual representations, cultural insensitivity, and dis- criminatory imagery are not captured by any of the three training labels and would not trigger the auditor regardless of severity. Extending GUARD- PAINT to such harms would require richer label ontologies, dedicated training data, and potentially separate auditor heads—each with their own cali- bration challenges. Adversarial attacks on the auditor itself. GUARDPAINT assumes the auditor is a reliable detector within its training distribution. A white- box or adaptive adversary with knowledge of the auditor’s architecture could craft inputs that simul- taneously satisfy the base model’s generation ob- jective and suppress the auditor’s adversarial score, bypassing the repair trigger entirely. Because the auditor operates in image space and the inpainter operates in latent space, the two modalities pro- vide some implicit robustness—an adversary must fool both to achieve a clean bypass—but this does not constitute a certified defense. Adaptive attacks specifically targeting the auditor trigger threshold or the tournament utility function remain an open threat. Inpainting domain gap. The policy-aligned in- painter is fine-tuned on masks derived from auditor heatmaps over a fixed training corpus. At inference, heatmap quality degrades on out-of-distribution content styles—highly stylized art, photorealis- tic renders with unusual lighting, or dense crowd scenes—which can produce masks that are either too coarse (capturing safe context) or too sparse (missing the unsafe region). In such cases the in- painter may repair the wrong region, introduce visi- ble seam artifacts, or fail the quality gate and leave the base trajectory intact. The non-regression guard prevents active degradation in the last scenario, but provides no safety improvement either. Policy alignment without architectural diver- sity.AlthoughA g is preference-aligned via BCO to produce policy-compliant repairs, alignment is constrained by the representational capacity of the SD 1.5 inpainting backbone. Higher-capacity in- painters built on SDXL or flow-matching archi- tectures have richer generative priors that could produce repairs with better perceptual quality, finer detail preservation, and stronger semantic coher- ence with the surrounding unmasked context. The resolution and latent-space mismatch that reinser- tion currently compensates for would not arise if the inpainter shared the same architecture as the base generator. Static calibration.Auditor thresholds(δ,τ P ,τ F ) and the tournament policyπ θ are calibrated offline on a fixed distribution of prompts and attack fam- ilies. As jailbreak methods evolve, the proposal distribution shifts and calibrated gates may become either too permissive or too conservative. Periodic recalibration against new attack families is neces- sary to maintain the intended operating point. Semantic drift across reinsertion. Latent reinsertion—whether via null-text inversion for UNet models or flow-ODE transport for flow- matching models—is an approximation. At early audit steps where the trajectory is still noisy, rein- sertion can introduce subtle distributional drift that accumulates over subsequent denoising steps, caus- ing the final image to deviate from the intended prompt in ways that neither the auditor nor the fidelity gate reliably detect. This effect is most pro- nounced on long-horizon generation tasks with fine compositional requirements. References Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6820–6829. Zhongjie Ba, Jieming Zhong, and 1 others. 2024. By- passing the safety filter of text-to-image models via substitution. In ACM CCS. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 oth- ers. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923. Black Forest Labs. 2026.Flux.1-dev: A rectified flow transformer for text-to-image generation. https://huggingface.co/black-forest-labs/ FLUX.1-dev. Model card. Umberto Borso and 1 others. 2025. D3po: Preference- based alignment of discrete diffusion models. In ICLR. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model de- coding with speculative sampling. In arXiv preprint arXiv:2302.01318. Yimo Deng and Huangxun Chen. 2024. Harnessing llm to attack llm-guarded text-to-image models. Preprint, arXiv:2312.07130. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rishubh Agarwal, Jonas M ̈ uller, Karsten Kreis, Tim Dockhorn, Minji Kim, Axel Sauer, Sascha Boesel, Dustin Podell, Tim Dauer, Dominic Lorenz, and Robin Rombach. 2024. Scaling rectified flow trans- formers for high-resolution image synthesis. In Pro- ceedings of the 41st International Conference on Ma- chine Learning (ICML). Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024.Kto: Model alignment as prospect theoretic optimization. Preprint, arXiv:2402.01306. Rohit Gandikota, Joanna Materzy ́ nska, Jaden Fiotto- Kaufman, and David Bau. 2023.Erasing con- cepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2426–2436. Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, and Baining Guo. 2023. Effi- cient diffusion training via min-snr weighting strat- egy. arXiv preprint arXiv:2303.09556. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Ad- vances in Neural Information Processing Systems, volume 33, pages 6840–6851. Jonathan Ho and Tim Salimans. 2022.Classifier- freediffusionguidance.arXivpreprint arXiv:2207.12598. Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Ra- sul, James Thorne, and Jongheon Jeong. 2025. Margin-aware preference optimization for align- ing diffusion models without reference. Preprint, arXiv:2406.06424. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung-Woon On. 2024. Binary classifier optimiza- tion for large language model alignment. arXiv preprint arXiv:2404.04656. Mingyu Kim, Dongjun Kim, Amman Yusuf, Stefano Er- mon, and Mijung Park. 2025. Training-free safe de- noisers for safe use of diffusion models. In NeurIPS. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via spec- ulative decoding. In Proceedings of the 40th Inter- national Conference on Machine Learning (ICML), pages 19274–19286. PMLR. Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yi- ran Qin, Xihui Liu, Lu Sheng, and Jing Shao. 2025. T2isafety: Benchmark for assessing fairness, toxicity, and privacy in image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13381–13392. Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. 2024a. Aligning diffusion models by optimizing human utility. In NeurIPS. Diffusion-KTO. Xinfeng Li, Yuchen Yang, Jiangyi Deng, Chen Yan, Shiqi Ji, Jie Wang, and Xiaoyu Jia. 2024b. Safe- Gen: Mitigating sexually explicit content generation in text-to-image models. In Proceedings of the ACM SIGSAC Conference on Computer and Communica- tions Security (CCS). Kendong Liu, Zhiyu Zhu, Chuanhao Li, Hui Liu, Huan- qiang Zeng, and Junhui Hou. 2024a.Prefpaint: Aligning image inpainting diffusion model with hu- man preference. arXiv preprint arXiv:2410.21966. Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. 2024b. Latent guard: a safety framework for text-to-image generation. In European Conference on Computer Vision, pages 93– 109. Springer. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2024c. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In European Confer- ence on Computer Vision (ECCV). Xingchao Liu, Chengyue Gong, and Qiang Liu. 2023. Flow straight and fast: Learning to generate and trans- fer data with rectified flow. In International Confer- ence on Learning Representations (ICLR). Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. 2024. Jailbreakv: A bench- mark for assessing the robustness of multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2404.03027. Jiachen Ma, Yijiang Li, Zhiqing Xiao, Anda Cao, Jie Zhang, Chao Ye, and Junbo Zhao. 2025. Jailbreak- ing prompt attack: A controllable adversarial attack against diffusion models. In Findings of the Associ- ation for Computational Linguistics: NAACL 2025, pages 3141–3157, Albuquerque, New Mexico. Asso- ciation for Computational Linguistics. Xiangtao Meng, Yingkai Dong, Ning Yu, Li Wang, Zheng Li, and Shanqing Guo. 2026. Beyond the safety tax: Mitigating unsafe text-to-image gen- eration via external safety rectification. Preprint, arXiv:2508.21099. Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023a. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 6038–6047. Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023b. Null-text inversion for editing real images using guided diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047. NeuralShell. 2023. Gore blood dataset. Authors OpenGVLab and 1 others. 2025.In- ternvl3.5: Advancing open-source multimodal mod- els in vision-language understanding. arXiv preprint arXiv:2508.18265. Yong-Hyun Park, Sangdoo Yun, Jin-Hwa Kim, Junho Kim, Geonhui Jang, Yonghyun Jeong, Junghyo Jo, and Gayoung Lee. 2024. Direct unlearning optimiza- tion for robust and safe text-to-image models. arXiv preprint arXiv:2407.21035. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. 2018. FiLM: Vi- sual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ̈ uller, Joe Peng, and Robin Rombach. 2023. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. 2025.Un- safebench: Benchmarking image safety classifiers on real-world and ai-generated images. In Proceed- ings of the 2025 ACM SIGSAC Conference on Com- puter and Communications Security, CCS ’25, page 3221–3235, New York, NY, USA. Association for Computing Machinery. Robin Rombach, Andreas Blattmann, Dominic Lorenz, Patrick Esser, and Bj ̈ orn Ommer. 2022.High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695. Patrick Schramowski, Manuel Brack, Bj ̈ orn Deiseroth, and Kristian Kersting. 2023. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22522–22531. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021a. Denoising diffusion implicit models. In In- ternational Conference on Learning Representations (ICLR). Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021b. Denoising diffusion implicit models. In In- ternational Conference on Learning Representations. Lukas Struppek, Dominik Hintersdorf, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting. 2023. Exploiting cultural biases via homo- glyphs in text-to-image synthesis. Journal of Artifi- cial Intelligence Research, 78:1017–1068. Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. 2024. Ring-a-bell! how reliable are concept removal methods for diffusion models? In International Conference on Learning Representa- tions, volume 2024, pages 41543–41554. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. 2024. Diffusion model alignment using direct prefer- ence optimization. In CVPR. Zongyu Wu, Hongcheng Gao, Yueze Wang, Xiang Zhang, and Suhang Wang. 2024. Universal prompt optimizer for safe text-to-image generation. In Pro- ceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6340–6354. Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung-Yi Ho, Nan Xu, and Qiang Xu. 2024a. Mma-diffusion: Mul- timodal attack on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7737–7746. Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. 2024b. Sneakyprompt: Jailbreaking text- to-image generative models. In 2024 IEEE Sympo- sium on Security and Privacy (SP), pages 897–912. yolov7test. 2022. Weapon detection object detection model.[https://universe.roboflow.com/ yolov7test-pdxwq/weapon-detection-m7tpo] (https://universe.roboflow.com/ yolov7test-pdxwq/weapon-detection-m7tpo). Author Zhang and 1 others. 2025. Shielddiff: Suppress- ing sexual content generation from diffusion models through reinforcement learning. arXiv preprint. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The unreason- able effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). A Training Dataset Construction This appendix details the construction of the 82K image–prompt corpus used to train the Auditor- ScorerA s , covering source datasets, prompt recon- struction, labeling pipeline design, and labeling quality. A.1 Source Datasets We draw from eight publicly available Hugging- Face datasets spanning three content categories: safe, nudity, and violence. Table 3 summarizes each source, its nominal category, and whether prompts were available or reconstructed. Sources were selected to cover both the nudity and violence policy axes with sufficient volume and diversity of visual style, scene complexity, and severity level. Datasets with pre-existing captions were used as-is after cleaning; datasets without prompts required reconstruction (see Section A.2). A.2 Prompt Reconstruction Four source datasets did not include generation prompts. For these, image captions were synthe- sized using Qwen2.5-VL (Bai et al., 2025) as a surrogate prompt. The model was instructed to produce a concise, descriptive caption in the style of a text-to-image generation prompt rather than a natural-language description, targeting the visual content of the image without editorializing about policy compliance. These reconstructed captions serve the faithfulness branch of the auditor dur- ing training and are not used for any safety label decision. A.3 Labeling Pipeline All images—regardless of source label—were re- labeled into three mutually exclusive categories: safe, nudity, violence. Multi-label annotation was evaluated and discarded: overlapping soft gra- dients from multi-label supervision degraded au- ditor classification performance in preliminary ex- periments. One-hot categorization was therefore adopted as the final labeling scheme. Labeler selection. We evaluated five candidate VLMs as automated labelers. Each was assessed on a manually inspected reference set of hard cases including bikinis, underwear, sports scenes with physical contact, crowds, close interpersonal prox- imity, and classical artwork (e.g., Greek nude sculp- ture). The reference set was chosen to stress-test policy-boundary behavior rather than gross viola- tions, which all models handle trivially. Table 4 summarizes observed failure modes. InternVL-3.5 (8B) was selected as the final la- beler based on this ablation. Why chain-of-thought prompting was de- emphasized. Explicit chain-of-thought prompt- ing increases verbosity without reliably improv- ing category fidelity. On ambiguous images, it causes models to generate unsupported post-hoc rationales or self-contradictory intermediate rea- soning that conflicts with the final label. Self- consistency (repeated passes with majority voting) was also rejected: it increases inference cost sub- stantially while failing to correct systematic clas- sification errors caused by stable policy-boundary confusion—the primary failure mode on borderline cases. Policy-first prompt design. The InternVL-3.5 prompt was designed to be policy-oriented rather than explanation-oriented. Categories are defined in operational terms directly tied to visual evidence: •Nudity: Swimwear, underwear, medical im- agery, and athletic clothing are explicitly safe. •Violence: visible physical harm or graphic cues such as blood or weapons in active use. Proxim- ity, sparring poses, and non-graphic contact are safe. •Safe: all other content, including classical artwork with nudity, sports imagery without graphic harm, and intimate proximity without explicit content. This framing prevents surface-level cue trigger- ing (e.g., visible skin area, figure proximity, athletic posture) which was the dominant failure pattern across rejected labelers. A.4 Threshold Calibration and Dead-Zone Exclusion InternVL-3.5 produces a continuous policy score for each image. Because no large ground-truth la- beled set was available for data-driven threshold selection, the operating threshold was calibrated empirically against the same manually inspected hard-case reference set used for labeler ablation. Model scores were compared against human judg- ment on this subset, and the threshold was set to reflect the desired moderation policy rather than an externally optimized operating point. Table 3: Source datasets for the 82K auditor training corpus from huggingface. DatasetCategoryPrompts Subh775/WeaponDetection (yolov7test, 2022)ViolenceVLM-generated NeuralShell/Gore-Blood-Dataset-v1.0 (NeuralShell, 2023)ViolenceVLM-generated x1101/nsfw-fullNudityVLM-generated Lenkashell/unsafeviolenceimagecaptionsViolenceExisting Lenkashell/unsafeshockingimagecaptionsViolenceExisting yiting/UnsafeBench(Qu et al., 2025)Nudity, ViolenceExisting OpenSafetyLab/t2isafetydataset(Li et al., 2025)Nudity, ViolenceExisting Table 4: Labeler ablation: observed failure modes on the hard-case calibration set. FP = false positive rate; FN = false negative rate on the calibration reference. ModelObserved failure mode Qwen2.5-VL (7B) Basic prompting yields high FP rate; chain-of-thought prompting introduces label contradictions; self-consistency is slow and inconsistent; few-shot prompt- ing degrades prompt-following according to community reports. Qwen3-VLSimilar failure pattern to Qwen2.5-VL; no improvement on boundary cases. ShieldGemma (vanilla) Slow inference; inconsistent categoriza- tion on borderline cases without fine- tuning; high FN rate overall. Grounding DINO(Liu et al., 2024c) Highly threshold-sensitive (small shift pro- duces either high FP or high FN); lacks semantic context for borderline cases (e.g., classical nude sculpture). InternVL- 3.5 (8B) Best overall performance: lowest FP rate; strong contextual nuance (e.g., cor- rectly categorizes classical nude sculp- tures as safe); conservative on FN (accept- able for training corpus construction). A dead zone was reserved around the threshold: images whose scores fell within a margin of the decision boundary were excluded from the training corpus entirely rather than assigned an arbitrary label. This exclusion strategy ensures that the high- confidence tails of the score distribution— where label quality is highest—dominate training, while uncertain borderline samples do not introduce spu- rious supervision signal. Residual label noise. The most likely resid- ual failure mode from single-shot VLM scoring is not catastrophic random mislabeling but sys- tematic boundary bias: InternVL-3.5 may mildly over-score skin-heavy but policy-safe images (e.g., bikini photographs), causing a fraction of such cases to drift above the nudity threshold. This pro- duces moderate boundary noise in the minority class rather than label collapse. For auditor train- ing, such boundary noise is manageable provided the high-confidence tails remain clean—which the dead-zone exclusion strategy is designed to ensure. A.5 Dataset Statistics and Splits The final labeled corpus contains 82,000 image– prompt pairs distributed across the three categories. After dead-zone exclusion of boundary cases, the corpus was split into training, validation, and test sets using a 70 / 15 / 15 stratified partition, preserv- ing class proportions across splits. The test set was held out entirely for the classification evaluation reported in Table 2. B Multi-Modal Adversarial Auditor Pipeline Architecture This appendix provides a rigorous architectural and mathematical specification of the Adversar- ial Image Auditor pipeline. The pipeline integrates multi-task learning, cross-modal attention, and dif- fusion timestep conditioning via Feature-wise Lin- ear Modulation (FiLM) to evaluate safety and struc- tural integrity in generated images. B.1 Complete Pipeline Flow and Spatial Dimensions The system takes three distinct inputs: an image X ∈ R B×3×224×224 , a tokenized prompt sequence T ∈ R B×L (whereLis the token sequence length), and a normalized diffusion timestept ∈ R B×1 . The primary data flow is formalised below. 1. Visual Feature Extraction. F visual = ResNet101 backbone (X) ∈ R B×2048×7×7 A global average-pooled vector is also com- puted: f global = AdaptiveAvgPool2d(F visual ) ∈ R B×2048 Figure 6: Architecture diagram of the Auditor module. 2. Textual Encoding. f text , S seq , M pad = BiLSTM encoder (T ) wheref text ∈ R B×512 is the global textual context,S seq ∈ R B×L×512 contains per-token sequence embeddings (pre-LayerNorm), and M pad ∈0, 1 B×L is a boolean padding mask. 3.Prompt-to-Image Cross-Attention. Visual fea- ture maps are projected to the textual dimension, flattened, and normalised to form query vectors: Q = LayerNorm Conv2d 1×1 (F visual ) ∈ R B×49×512 K = V = LayerNorm(S seq ) ∈ R B×L×512 Multi-head cross-attention yields prompt- conditioned visual representations: A attended = MultiHeadAttention(Q, K, V ; mask = M pad ) ∈ R B×49×512 Spatial averaging givesf attended ∈ R B×512 , which is used exclusively by the CLIP-style alignment head (Section B.3). B.2 Timestep-Aware FiLM Conditioning Adversarial noise patterns manifest differently de- pending on the denoising timestept ∈ [0, 1]. A multi-layer perceptron mapstto a shared temporal embedding: e t = MLP timestep (t) ∈ R B×512 This embedding drives two independent FiLM projections with separate learned weights — one for adversary modulation and one for seam modu- lation. Each projection produces scaleγand shift βcoefficients for Feature-wise Linear Modulation. 1.Relative Adversary Modulation. The relative adversary score quantifies the continuous per- turbation strength of an image on a[0, 1]scale (see Section B.3). To condition its prediction on the timestep, we modulate the global pooled features: γ adv , β adv = Split Linear film,adv (e t ) ∈ R B×2048 f mod global = (1 + γ adv )⊙ f global + β adv HereLinear film,adv : R 512 → R 4096 , andSplit divides the output equally along the last dimen- sion. 2.Seam Quality Modulation. A separate projec- tion conditions the intermediate convolutional featuresF seam ∈ R B×512×7×7 used for struc- tural artefact detection: γ seam , β seam = Split Linear film,seam (e t ) ∈ R B×512 F mod seam = 1 + γ seam ⊙ F seam + β seam whereγ seam andβ seam are broadcast over the spa- tial dimensions. Note thatLinear film,seam maps R 512 → R 1024 and has no shared weights with Linear film,adv . B.3 Specialised Multi-Task Auditor Heads The auditor decouples distinct tasks into dedicated prediction heads to ensure stable gradient flow and to protect alignment encodings from direct safety- label contamination. •Binary Adversarial Head.Computes P (Adversarial | X)directly from the un- modulated spatial featuresF visual via a1×1 convolution, adaptive global average pooling, and Sigmoid activation. This head intentionally bypasses FiLM conditioning so that the binary decision is not conflated with timestep dynamics. •Safety Category Head. ProjectsF visual toC = 3 channels and applies cost-sensitive cross-entropy with class weightsw = [1.0, 5.0, 2.0]. The el- evated weight for class 1 (adversarial) reflects its underrepresentation in the training distribu- tion; the moderate weight for class 2 (borderline) penalises ambiguous predictions more than the majority safe class. •Relative Adversary Score Head. Regresses the continuous perturbation strengths ∈ [0, 1] from the FiLM-modulated vectorf mod global using a three-layer MLP with dropout. Unlike the binary head, this head is conditioned on the timestep em- bedding to capture how perturbation magnitude varies across denoising stages. •Seam Quality Assessment Head. Evaluates in- painting seams and composite boundaries by ap- plying a regression head toF mod seam , yielding a lo- calised artefact score. • CLIP-style Faithfulness Alignment Head. Projectsf attended andf text independently into a shared latent spaceR 256 using separate two-layer MLPs, then optimises image–prompt alignment via symmetric InfoNCE loss with a learnable log- temperatureτ. Both the cross-attended visual vector and the raw global text vectorf text are fed to this head to preserve low-level lexical ground- ing alongside spatially-attended semantics. B.4 GradCAM Explainability Pipeline For visual explainability, target activations are ex- tracted from the final residual convolutional layer of the ResNet101 backbone. The localised risk heatmap M risk ∈ R 7×7 for safety class c is: M risk = ReLU X k α c k · F visual,k ! where the importance weightsα c k are obtained by global average pooling of the backpropagated gra- dients: α c k = AdaptiveAvgPool2d ∂ y c ∂ F visual,k The resulting maps are bilinearly upsampled to 224×224and alpha-blended onto the source image using custom contour overlays to localise flagged regions. Because GradCAM operates directly on F visual , it provides explanations that are indepen- dent of FiLM conditioning, giving an unbiased view of which image regions drive the safety clas- sification. C Policy-Aligned Inpainter: SFT and BCO Training Details This appendix documents the complete training pipeline for the policy-aligned inpainterA g , cover- ing: the Refusal SFT stage (§C.2), the alignment objective selection and the migration from KTO to BCO (§C.1–C.3), the full BCO loss formula- tion and motivations (§C.4), the failure archaeology from all training runs (§C.5), and the final validated training configuration (§??). C.1 Alignment Objective Selection: Why KTO and BCO Dataset constraints.Modern RLHF methods im- pose different data-preparation burdens. DPO (Wal- lace et al., 2024) requires paired (chosen, rejected) completions for every prompt; for images this de- mands either human preference labelling or an ex- pensive oracle model. KTO (Li et al., 2024a) and BCO (Jung et al., 2024) both operate on unpaired binary-labelled data(x,y) ∈ 0, 1, wherey = 1denotes a safe (desirable) completion andy = 0denotes a policy- violating (undesirable) one. This format is pro- duced directly by the auditor, requiring no addi- tional human annotation. Risk profile motivation.For a content-safety in- painter, the error costs are highly asymmetric: a false negative (generating adversarial material un- detected) is far more costly than a false positive (over-suppressing a benign region). KTO was ini- tially chosen specifically for its loss-averse utility function and its asymmetric treatment of gains and losses, properties that appeared well-suited to this risk profile. C.2 Refusal SFT - LoRA Fine-Tuning Motivation: the know-before-you-refuse prin- ciple. The LLM training paradigm provides a useful analogy: instruction tuning precedes align- ment because a model must first know how to per- form a task before it can refuse it in a principled way. Applying an alignment signal directly to a model with no prior concept of “safe inpainting” is under-constrained: the model has no learned attrac- tor basin for policy-compliant completions, so the alignment gradient has nothing to guide it toward. Formally, letπ θ be the inpainter policy andRthe target “refusal manifold” in latent space—the set of completions that replace NSFW content with spa- tially coherent, policy-compliant alternatives. An alignment objective such as BCO applies a gradi- ent∇ θ L BCO that rewards distance from the policy- violating attractor. However, ifRis not already in the support ofπ θ , the gradient has no preferred direction in which to move the policy; it merely pushes it away from the violation without speci- fying where to go. SFT pre-seeds the policy with a collapsed distribution overR, giving the BCO gradient a meaningful signal basin to expand. In our training logs, this manifests clearly: the BCO run starting from an SFT-initialised check- point exhausts its SFT prior through approximately step 500, then searches the manifold for an ex- ploit. The asymmetric auxiliary losses (reconstruc- tion and identity, detailed in §C.4) constrain this search to a corridor that eventually leads to the glob- ally optimal solution—rendering coherent clothing. A BCO run starting from a vanilla SD-inpainting checkpoint does not exhibit this three-phase struc- ture; it collapses directly to the pixel-smudging attractor with no recovery. Base model and adapter. SFT training starts from runwayml/stable-diffusion-inpainting, a 9- channel inpainting UNet conditioned on(z t ∈ R 4 ,m ℓ ∈ R 1 , ̃z 0 ∈ R 4 ) , wherem ℓ is the latent- space mask and ̃z 0 is the masked image latent. We apply LoRA (Hu et al., 2022) with rankr=64, α=64(effective scale= α/r = 1.0, i.e. no im- plicit scaling) targeting the following parameter groups: •Cross- and self-attention: toq, tok, tov, toout.0 •Feed-forward layers in transformer blocks: f.net.0.proj, f.net.2 • Spatial projection layers: projin, projout •ResNet convolutional layers (structural/spatial output): conv1, conv2 • UNet input convolution layer: conv in The inclusion of convin is critical. The 9- channel inpainting UNet differs from the standard 4-channel text-to-image UNet in precisely this first layer: it processes the concatenated(z t ,m ℓ , ̃z 0 ) tensor. Leaving convin frozen during alignment causes the mask-conditioning logic to resist policy signals, because the gradient from the loss cannot propagate into the layer that interprets which pixels to modify. Similarly, the ResNet convolutional lay- ers (conv1, conv2) handle the structural and spatial content of the output. Refusal requires rendering new content (clothing, blurred faces)—a structural operation—and attention-only LoRA lacks the rep- resentational capacity for such changes. Stage B-2 uses rankr=128,α=128for the align- ment phase, on the hypothesis that the earlier runs with smaller rank were exhausting LoRA capacity and saturating prematurely. Training data. The SFT dataset consists of ap- proximately 2 000 images (1 000 nudity viola- tions, 1 000 violence violations). For each un- safe source image, the auditor generates a mask over the policy-violating region. The source im- age and mask are then passed to the inpainter with a custom-generated safe prompt (derived by prompting a Qwen-2.5-VL captioner followed by a LLaMA-8B rewriter) to produce a target “accepted” image. Training pairs are therefore (x unsafe ,m,x safe ,p unsafe ): the unsafe source, its vi- olation mask, the human-approved safe target, and the original adversarial prompt. Training on the adversarial prompt rather than the safe prompt is deliberate: at inference, the in- painter receives the adversarial prompt because the user’s intent is unknown. The model must learn the pixel-space signature of a safe completion condi- tioned on an unsafe prompt—not relying on text- based cues to trigger refusal. Loss and schedule. The SFT loss is Min- SNR (Hang et al., 2023) weighted MSE across the full noise schedule t∈ [0, 1000): L SFT = E t,ε " w(t)·∥ε θ ( z t ,t,m ℓ , ̃z 0 ,p unsafe )− ε∥ 2 # , wherew(t) = min(SNR(t),γ)/SNR(t)with γ=5. Without Min-SNR reweighting, high-noise timesteps (larget) dominate the gradient because the signal-to-noise ratio is low and the loss mag- nitude is large. This matters here because safe completions are visually coherent only at low noise levels (t ≪ 500); high-noise gradients contribute primarily noise-level statistics and not the structural pattern of clothing or background fill. Min-SNR redistributes gradient mass toward the low-noise regime where the visual content of the refusal is actually resolved. Formally, the Min-SNR weight is derived from the signal-to-noise ratioSNR(t) = ̄α t /(1− ̄α t ), where ̄α t = Q t s=1 α s is the cumulative product of the noise schedule. The clamp atγprevents the weight from collapsing to zero at very lowt (where SNR is enormous), maintaining gradient flow throughout the schedule. A small noise offset of0.05is added:ε ′ = ε + 0.05· ε 0 whereε 0 ∼ N (0,I C×1×1 )is broad- cast. This offset helps with dark and saturated regions—common in clothing inpainting—by pre- venting the VAE latent distribution from collapsing into degenerate attractors near the boundary of its quantization grid. After convergence (loss plateau below∼0.07 over 8 epochs), the LoRA weights are merged into the base checkpoint viamergeandunload(), pro- ducing a single 9-channel UNet used as the Stage B-2 initialisation. How SFT initialises BCO: a geometric argu- ment. LetM img denote the natural image man- ifold in latent space andM safe ⊂ M img the sub- manifold of safe completions inside the mask. Pre- SFT,π θ (the base inpainting model) has learned a strong prior towardM img conditioned on the in- put context, but has never encountered a training signal that distinguishesM safe from the rest of the manifold. SFT provides approximately 2 000 examples of(x unsafe ,m,x safe )triples. The MSE loss(C.2) carves a narrow basin in the loss landscape centred onM safe for inputs of the form(z t ,m ℓ , ̃z 0 ,p unsafe ). After SFT,π θ has a collapsed mode at cloth- ing/covering completions for masked unsafe inputs. BCO then operates on this initialised policy. The BCO gradient moves the policy such that: (a) on safe inputs, the policy stays near or improves upon the reference; (b) on unsafe inputs, the policy de- liberately degrades relative to the reference (moves further from the policy-violating content). Because the SFT basin already exists, the BCO search for “worse on unsafe” lands directly onM safe rather than on degenerate off-manifold solutions. This ex- plains the three-phase dynamics observed in BCO training: the SFT prior is exploited first (Phase 1), the degenerate attractor is blocked by auxiliary losses (Phase 2), and the SFT basin is rediscovered as the unique viable optimum (Phase 3). C.3 KTO: Formulation, Experiments, and Failure Modes We now document the KTO implementation and the concrete failure modes that motivated the switch to BCO. C.3.1 KTO Formulation The KTO objective (Ethayarajh et al., 2024) for diffusion models (Li et al., 2024a) adapts the Kahneman–Tversky prospect-theoretic value func- tion to the noise-prediction setting. Our implemen- tation computes a “g-term” as: g i = ℓ θ i − ℓ ref i , whereℓ • i is the masked MSE of model•against the clean latent z 0 inside the mask: ℓ • i = P b,c,h,w (1 + λ m m ℓ )· (ˆz • 0 − z 0 ) 2 · m ℓ P m ℓ . A positiveg i means the trainable model has drifted further fromz 0 than the reference inside the mask— worse alignment with the safe target. The signal is centered using an EMA over safe samples: δ ema ← 0.99δ ema + 0.01 ̄g| safe , δ ema ∈ [−0.02, 0.05]. The centered signal drives a sigmoid-based loss: h i = σ(sgn i · β· (g i − δ ema )), L KTO = E i [w y · (1− h i )], wheresgn i = +1for safe samples and−1for unsafe,β = 7, andw y ∈1.0, 5.0, 12.0for safe, nudity, and violence classes respectively. C.3.2 KTO Experiment Progression Vanilla KTO. The first KTO implementation used a static reference model and theε-space MSE as a proxy for the probability of generating policy- violating content. The model quickly found a de- generate attractor: generating uniform blue patches inside the masked region. This trivially reduced the noise-prediction MSE difference relative to the reference (whose MSE was also high at the same patches) while producing visually incoherent out- put. The model had hacked the reward by leaving the image manifold entirely. Dynamic reference. To prevent manifold depar- ture, we tried updating the reference model every Ngradient steps. This accelerated instability: accu- mulated errors in the trainable model were passed to the reference model, which then guided train- ing toward its own failures in a positive-feedback loop. Image quality outside the mask degraded correspondingly. Reconstruction and identity losses. To anchor the model to the image manifold, we introduced two auxiliary losses (described fully in §C.4): a reconstruction loss penalising noise-prediction er- ror outside the mask, and an identity guardrail pe- nalisingˆz 0 -space divergence from the reference. These improved manifold stability but introduced a new failure mode: the model learned to smudge hands, feet, and peripheral body parts inside the mask—a pixel-destruction strategy that achieved low KTO/BCO loss while staying near the identity boundary. Modified loss formulation.We observed that the reference model did not always generate a policy- violating inpaint, especially for prompts that did not explicitly mention NSFW content (Qwen-2.5- VL captions tend to be descriptive rather than ex- plicit). We reframed the problem as alignment and safety-quality improvement, computing the g-term as the difference of MSEs against ground-truth noise rather than against the reference. Results were similar. Prompt dropout. Observing that the cross- attention layers were overweighted (higher acti- vation delta than before the manifold dip and after recovery), we suspected the cross-attention was learning to key on NSFW token patterns rather than visual features. We introduced 30% prompt dropout per batch, zeroing the text conditioning to force spatial grounding of the refusal signal. This engineering choice is retained in the BCO configu- ration. C.3.3 Why KTO Failed: Theoretical Analysis Failure Mode 1:ε-space stochasticity. The KTO g-term(1)is computed fromˆz 0 predictions, but theˆz 0 reconstruction is itself a function of the noiseεsampled at each step. At high timesteps t ∼ U [500, 1000), the standard deviation ofεis ∼O(1), and the resulting variance in the masked MSE across samples from the same image at the sametisO(10 −1 ). The preference gap be- tween safe and unsafe images in our dataset is O(10 −2 ). Consequently, the signal-to-noise ratio of the preference gradient is less than 1; the loss tracks noise-schedule variance rather than safety information. This produces the characteristic flat h safe ≈ h unsafe ≈ 0.500plateau observed in early KTO runs. Failure Mode 2: Asymmetric risk cannot be en- coded cleanly. KTO uses(1− σ(x))as its loss function, which is equivalent tologσ(−x)up to sign. The curvature of this function is symmet- ric inx: moving the policy away from a violation provides the same gradient magnitude as moving it toward a safe completion. Our risk profile is strongly asymmetric: false negatives on unsafe content (NSFW output reaches the user) are far more costly than false positives (over-suppression of benign content). KTO has no mechanism to en- code this asymmetry beyond the class weightsw y , which scale the loss magnitude but not the gradient shape. Additionally, KTO’s EMA baseline is computed from safe samples only ((C.3.1)), then subtracted from both safe and unsafe g-terms. This conflates the distributional statistics of two distinct popu- lations: safe images (whereg ≈ 0under a well- behaved policy) and unsafe images (whereg ≫ 0 is desirable). Subtracting a safe-derived baseline from the unsafe signal systematically miscalibrates the centering. Failure Mode 4: Identity loss bleeds globally. The KTO implementation computes the identity gap as a single global scalar: ℓ id = ∥ˆz θ 0 − ˆz ref 0 ∥ 2 − 0.02 2 + , aggregated over all samples. A single severely drifted sample inflates this scalar, raising the penalty for all other samples and causing the op- timizer to over-regularize the well-behaved ones. The resulting signal is poorly calibrated and im- pedes the policy’s ability to diverge from the ref- erence on unsafe samples—which is precisely the desired behavior. C.4 BCO Loss: Full Formulation with Proofs and Motivations BCO (Jung et al., 2024) reformulates the alignment objective as binary classification: a sample is “de- sirable” (y = 1) if it should be encouraged and “undesirable” (y = 0) if it should be suppressed. The following sections derive each component of the BCO loss from first principles. C.4.1 Why ˆz 0 , Not ε Standard diffusion training minimises noise- prediction MSE∥ε θ − ε∥ 2 . At timestept, the magnitude ofεis∼ O(1), so small changes in ε θ produce large, timestep-dependent changes in the MSE. A reward computed from∥ε θ − ε∥ 2 is therefore dominated byt, making per-class weight- ing meaningless—a sample att = 900produces 10–100×the loss magnitude of the same sample at t = 100. Instead, both UNets predict the clean latent: ˆz • 0 = z t − √ 1− ̄α t ε • √ ̄α t , •∈θ, ref. The denominator √ ̄α t ranges from approximately 1.0(att = 0) to∼0.01 (att = 999). While ˆz 0 predictions are noisier at hight, they are cal- ibrated to the same latent space regardless oft: ˆz 0 ∈ [−3σ z 0 , +3σ z 0 ]with bounded variance. This invariance is the key property: per-class weights now operate on a reward signal with consistent scale, making nudity weight5.0×and violence weight12.0×interpretable as actual preference ra- tios. C.4.2 Masked MSE Reward The reward quantifies how much better the train- able policy reconstructs the safe target inside the mask, relative to the frozen reference: w = 1 + λ m · m ℓ , λ m = 0.5, ℓ • = P b,c,h,w w· (ˆz • 0 − z 0 ) 2 · m ℓ P m ℓ , g i = ℓ ref i − ℓ θ i . Note the sign convention:g i > 0means the train- able model reconstructs the masked region better than the reference (closer toz 0 ). For safe images, g > 0is desirable—the policy should improve on the reference. For unsafe images,g < 0is desirable—the policy should produce reconstruc- tions further from the policy-violating z 0 . The mask weightλ m = 0.5applies1.5×gradi- ent emphasis inside the adversarial region without completely ignoring the surrounding context. Ig- noring the unmasked region entirely (λ m →∞, i.e. masking the loss to the interior only) removes the coherence signal from the context border, produc- ing visible seam artifacts at the mask boundary. Hinge cap on unsafe reward. For unsafe sam- ples (y = 0), the reward is capped at: g i ← min g i , 1.5·| ̄g unsafe | detach , where ̄g unsafe is the mean reward over unsafe sam- ples in the current batch. Without this cap, the pol- icy can exploit the mask by uniformly suppressing all content inside it—a trivial maximisation of−g that achieves a very negative reward (far fromz 0 ) while producing visually incoherent output. The cap prevents the policy from exploiting arbitrar- ily large reward by bounding the unsafe reward at 1.5×its own class mean, forcing it to find solu- tions that are both safe-suppressing and spatially coherent. C.4.3 Reward-Shift EMA and BCO Loss To center the reward at the decision boundary be- tween safe and unsafe classes, we maintain an ex- ponential moving average: δ raw = 1 2 E[g|y=1] + E[g|y=0] , δ ema ← 0.999δ ema + 0.001δ raw , δ ema ∈ [−0.03, +0.03]. Why the symmetric midpoint? The shifted re- wardr i = g i − δ ema should be positive for safe samples (the policy is better than baseline) and negative for unsafe samples (the policy is worse than baseline). The optimal centering point is the midpoint of the two class means, which is exactly δ raw . KTO’s centering ((C.3.1)) uses a safe-only EMA, which systematically underestimates the true decision boundary whengdiffers across classes, miscalibrating the loss. The EMA clamp is applied in-place to the stored EMA value before any downstream use: The [−0.03, +0.03]bound is tighter and symmetric compared to KTO’s[−0.02, +0.05], preventing runaway baseline drift in either direction. BCO loss: softplus vs. sigmoid. Givenr i = g i −δ ema and label signsgn i = 2y i −1∈−1, +1, the per-sample BCO loss is: ℓ BCO i = softplus(− sgn i · β· r i ) = log(1 + exp(− sgn i · β· r i )). Why softplus instead of KTO’s(1 − σ)loss? Note thatlogσ(x) = −softplus(−x). The KTO per-sample lossw y (1− h i )withh i = σ(·)is re- lated to the negative log-likelihood− logσ(·)but has a different gradient profile. 1.Gradient saturation.∂(1 − σ(x))/∂x = −σ(x)(1− σ(x)) , which collapses to zero as x → ±∞.∂ softplus(−x)/∂x = −σ(−x) = −(1− σ(x)), which goes to−1asx → +∞ and to0asx → −∞. Softplus maintains nonzero gradient on correctly classified, highly confident samples, continuing to push them fur- ther from the boundary. KTO’s(1− σ)loss provides a vanishing gradient oncehis large, causing training to stall for well-separated sam- ples while allowing hard cases to drift. 2. Probabilistic interpretation. Softplus is the numerically stable form of the binary cross- entropy loss− logσ(·), which is the maximum- likelihood loss for a logistic classifier. KTO’s (1− σ)loss lacks this clean probabilistic inter- pretation and is less aligned with standard RL reward maximization intuition. 3.Numerical stability. Atβ = 50, the argu- ment to the sigmoid can easily reach|βr| ∼ 50× 0.5 = 25, causing floating-point overflow inexp(25)≈ 7×10 10 . PyTorch’sF.softplus handles this via the numerically stable formula- tion max(x, 0) + log(1 + exp(−|x|)). The final BCO loss aggregates with per-class weights w y : L BCO = E i [w y · softplus(− sgn i · β· r i )], β = 50. C.4.4 Identity Guardrail: From Linear Penalty to Quadratic Hinge The smudging problem.An early linear identity penaltyL id = ∥ˆz θ 0 − ˆz ref 0 ∥ 2 creates a continuous tradeoff: the policy can achieve low BCO loss (sup- pressing unsafe content) while staying near the reference (low identity loss) by smudging pixels— slightly blurring or washing out the masked re- gion rather than replacing it with coherent con- tent. Smudging produces∥ˆz θ 0 − ˆz ref 0 ∥ 2 ≈ 0.01 – 0.03, which is small enough to be tolerated by a linear penalty but large enough to visually destroy the image. The Quadratic Hinge. The fix is a threshold that allows free exploration below a safe-smudging boundary and applies quadratic torque above it: ℓ id (i) = ∥ˆz θ 0 (i)− ˆz ref 0 (i)∥ 2 − κ 2 + , κ = 0.02, L id = E i [ ̄w i · ℓ id (i)], where ̄w i = 30.0for safe samples and ̄w i = 5.0 for unsafe . The hinge atκ = 0.02is chosen based on empir- ical observation: below this threshold, the model is rendering coherent content (clothing, fabric tex- ture, background elements); above it, the model is applying large-scale structural distortions in- cluding smudging. Belowκ: zero penalty—the model may freely render clothing, fabric, and back- ground detail. Aboveκ: quadratic torque—the penalty grows as the square of the excess, mak- ing large distortions (ℓ id ≫ κ) prohibitively ex- pensive. A distortion ofℓ id = 0.05incurs penalty (0.05−0.02) 2 = 9×10 −4 , whileℓ id = 0.10incurs (0.10− 0.02) 2 = 6.4× 10 −3 —a7.1×increase for a 2× increase in distortion. Hinge applied per sample, not globally. BCO applies hinge per sample before aggregating. KTO:L id = E i h ∥ˆz θ 0 − ˆz ref 0 ∥ 2 i − 0.02 2 + . With global aggregation, a single badly drifted sam- ple inflates the mean, raising the penalty for all other samples and causing the optimizer to over- regularise the well-behaved ones. Per-sample hing- ing isolates the penalty to the drifted samples, main- taining a clean gradient for samples that are already rendering coherent content. Small weight for unsafe samples. The identity loss is applied to unsafe samples at a reduced weight ( ̄w i = 5.0vs.30.0for safe). This is deliber- ate: for unsafe samples, the policy should diverge from the reference inside the mask (the reference produces NSFW content; the policy should not). However, the identity loss is applied across the full image (ˆz 0 is full-resolution), so it also anchors the unmasked background. The small weight preserves this background anchor—preventing suppression from bleeding into arms, hands, and context out- side the mask—without blocking the policy from diverging on the masked region. C.4.5 Reconstruction Anchor The reconstruction loss anchors the unmasked back- ground to the ground-truth noise: L recon = P i ̄r i ·∥(ε θ − ε)⊙ (1− m ℓ )∥ 2 i P i ̄r i , where ̄r i = 200.0for safe samples and ̄r i = 0.0 for unsafe. Why noise space, notˆz 0 space?The reconstruc- tion loss operates inε-space rather thanˆz 0 -space, because it targets the unmasked region where the noise-schedule variance argument does not apply: at any timestept, the noise prediction on an un- masked region should match the ground-truth noise εregardless oft. Usingˆz 0 in the unmasked region would require dividing by √ ̄α t , which amplifies errors at hightand is unnecessary since we are not computing a preference signal here—only an anchor. Why zero weight for unsafe?The policy should freely modify the unmasked background on unsafe samples only in the sense of preventing leakage from the masked region. Applying a strong re- construction anchor to unsafe samples creates an objective conflict: the BCO loss pushes the mask region toward non-NSFW content, while the recon- struction loss would resist any structural change to the surrounding context. Zeroing the unsafe recon- struction weight removes this conflict, allowing the identity loss (at small weight) to provide the light background anchor. Normalized aggregation. BCO uses a normal- ized weighted mean:L recon = ( P i ̄r i ℓ i )/( P i ̄r i ) . Since unsafe samples have ̄r i = 0, the denomina- tor equals the sum over safe samples only, making the result the mean reconstruction loss of safe sam- ples. This is preferable to the KTO formulation of L recon = 200.0× mean(ℓ i ), which scales the raw mean (including near-zero unsafe contributions) by a large scalar, creating scale mismatch. Large fixed scalar multipliers are sensitive to learning rate choice and require careful tuning; normalized aggregation is robust to class imbalance. C.4.6 Multi-Step DDIM Unrolling At inference, the inpainter runs the full DDIM schedule starting from the scheduler’s audited timestep. Single-step proxy training (predicting at onet) creates a distribution gap: the model is trained to minimize single-stepˆz 0 MSE, but evalu- ated on multi-step trajectories. Every 10 gradient steps, we augment the BCO loss with a second pass at a lower timestep. Given the current timesteptand unrolling factorK = 10: ˆz (s) 0 = z t s − √ 1− ̄α t s ε θ (z t s ,t s ) √ ̄α t s , z t s+1 = p ̄α t s+1 ˆz (s) 0 + p 1− ̄α t s+1 ε θ (z t s ,t s ), run forK − 1deterministic (η = 0) DDIM steps undernogradto producez mid . We then re-noise: z t f = p ̄α t f z mid + p 1− ̄α t f ε ′ ,t f =⌊t/K⌋, and compute a second BCO loss at(z t f ,t f ) under gra- dients. Both passes contribute equally (weight0.5each) to the final gradient: L total BCO = 0.5L (t) BCO + 0.5L (t f ) BCO . To prevent holding two full UNet activation graphs in VRAM simultaneously, pass-1 backward is called before pass-2 forward. C.4.7 Full Objective L = 4L BCO +L recon +L id . The factor of4onL BCO was tuned empirically. The softplus loss withβ = 50has magnitude ∼O(10 −2 –10 −1 )at the decision boundary. The reconstruction and identity losses have magnitude ∼O(10 −1 –10 0 )(noise-space MSE). The factor of 4brings the BCO signal into the same order of magnitude as the auxiliary losses, preventing them from dominating the gradient. Training batches are stratified: 8 safe / 4 nudity / 4 violence per batch of 16. Prompt dropout: 10% per sample independently (reduced from the 30% used in the later KTO experiments, as the BCO loss is less prone to cross-attention overfitting due to its spatial masking). C.5 Failure Mode Archaeology Table 5 documents the specific failure modes dis- covered across the different training runs, the diag- nostic evidence that revealed them, and the archi- tectural fix applied. C.6 Training Dynamics and final Results Figure ( 8) displays the final training run dynamics. Three-phase dynamics. The final training run exhibits a characteristic three-phase training trajec- tory: Phase 1 (steps 0–1000): SFT exploitation and smudging. The model first uses the SFT prior to make modest progress (∆Nrises to0.222), then transitions to pixel smudging as a cheaper opti- mization path. At step 1000,h U = 0.772 > h S = 0.668: the model has become “more confident” on unsafe images than safe ones, a signature of the smudging attractor. The inverted gap indicates the model has found an incorrect local minimum. Figure 7: The graph shows separation in the prompt trajectories in the latent space from high noiset = 700 to low noiset = 5between red=‘nudity’, blue=‘violence’ and green=‘safe’ prompts for the policy-aware inpainter at step 0 and step 3000 of the BCO process for the attention module. Failure ModeSymptomRoot CauseFix Applied Noise-space stochas- ticity (KTO Runs) h safe ≈ h unsafe ≈ 0.500 through step 500; no prefer- ence separation Preference gapO(10 −2 )over- whelmed by high-tnoise vari- anceO(10 −1 )inε-space; SNR of gradient < 1 Switch to BCO; reward com- puted onˆz 0 (bounded, schedule- invariant) Off-manifoldblue patches(KTO, vanilla) Model generates uniform blue fill inside mask; image quality deteriorates outside mask Model hacks reward by leav- ing image manifold; noise- space MSE is trivially mini- mized by off-manifold output Reconstruction + identity losses to anchor manifold Smudging / pixel de- struction (KTO Runs and early BCO) Unsafe content obscured by blur/smear rather than replaced;identitygap ≈ 0.035 Linear identity penalty creates continuous trade-off; smudg- ing is cheaper than rendering fabric QuadraticHingeidentity penalty with thresholdκ = 0.02 Mask-conditioning breakage Model ignores mask; gener- ates uniform texture across full image Alignment gradients propa- gate intoconvin, corrupting 9-channel conditioning logic Includeconvinin LoRA adapter from SFT stage;L recon anchors unmasked context Sigmoid saturation sigmoid satpct > 90% by step 1500; softplus gradi- ent near floor; training stalls β=50 with large reward gap pushes softplus into saturation; expected behavior, not a bug Monitor∆Nfor continued sep- aration. Saturation = binary de- cision boundary learned. Re- duceβonly if∆Nplateaus with saturation. Cross-attention over- fitting Safety behavior collapses whenNSFWkeywords present; model keys on token patterns not visual features Cross-attentionlayers memorize token-safety co- occurrence; spatial features unused 10% prompt dropout; forces re- liance on spatial latent features for refusal triggering Reference contami- nation BCO validation set appears intrainingset;cross- attention activations differ qualitatively before and after manifold dip Qwen-2.5-VL captions pro- duce near-identical text for semantically similar images; dataset leakage Prompt dropout (secondary fix); data de-duplication of validation set against training set Table 5: Failure modes discovered across all training runs, their diagnostic evidence, root causes, and the fixes applied. Phase 2 (steps 1000–1500): Quadratic Hinge activation. As smudging intensifies,∥ˆz θ 0 − ˆz ref 0 ∥ 2 crosses the hinge thresholdκ = 0.02. The identity loss now applies quadratic torque, making smudg- ing prohibitively expensive. BCO simultaneously demands more unsafe suppression, and theˆz 0 an- chor prevents structural distortion. The model backs off, causinghvalues to temporarily equalize and∆Nto plateau. This corresponds to the model searching the constrained manifold for a new opti- mization path. Phase 3 (steps 1500–3000): Clothing discov- ery. With smudging blocked andδ ema clamped, the model is constrained to solutions that: (a) are safe- suppressing (lowgon unsafe images, satisfying BCO); (b) are spatially coherent (low identity loss, below hinge); (c) preserve background context (low reconstruction loss outside mask). The unique so- lution satisfying all three constraints is rendering coherent clothing or fabric over NSFW content. Clothing is structurally stable (ℓ id < 0.02), seman- tically safe (large∆N), and contextually coherent (low seam artifacts). By step 2000,h S = 0.918, ∆N = 0.403, and sigmoid saturation reaches 96.9%. Sigmoid saturation as a success criterion. In standard diffusion training, sigmoid saturation (> 90%) is a pathology indicating vanishing gradients. In BCO withβ = 50, it is the intended outcome: the loss explicitly places the model into a binary regime. At97%saturation, the model has learned a near-hard classifier inˆz 0 -space: safe images are reconstructed better than the reference (g > δ), unsafe images worse (g < δ). The softplus loss maintains nonzero gradients even at saturation (un- like KTO’s(1− σ)loss), ensuring continued re- finement rather than complete stall. The optimal checkpoint is around step 2000–2500; continued training past step 3000 shows slight∆Ndegrada- tion, likely due to over-fitting the BCO signal at the cost of generalization. Pre-computed dataset structure. All training images are resized to512× 512and pre-encoded through the SD 1.5 VAE to eliminate encoder for- ward passes during training. Each sample stores: z 0 (clean latent,R 4×64×64 ),m ℓ (latent mask, R 1×64×64 ), ̃z 0 (masked image latent), tokenized prompt inputids, and binary label y ∈0, 1.. D Guarded Tournament: Architecture, Training, and Calibration Details This provides the exact specifications of the Tour- nament Sampling Policy Optimization component. We cover the exact policy network and state en- coder architectures (D.1), the complete loss deriva- tion with all regularization terms (D.2), the guarded utility function and its time-varying thresholds (D.3), the seam quality metric and how it is com- puted (D.4), the reinsertion strategy comparison and why null-text inversion is selected ,the full training configuration (9). Table 6: StateEncoder architecture and modules. Here,p text denotes the projected text embedding,z latent denotes the projected latent representation,im image de- notes the projected image embedding,m mask denotes the projected mask statistic, andt norm denotes the nor- malized diffusion timestep. ModuleOperationOut TextLinear(TEXTDIM, 64) + ReLU64 LatentAdaptiveAvgPool(4×4)→Flatten→ Linear(16·LATENTC, 64)+ ReLU 64 ImageLinear(256, 64) + ReLU64 MaskLinear(1, 64) + ReLU64 TimeNormalized timestep t norm 1 Final State Concatenate ([p text ,z latent ,im image ,m mask ,t norm ]) 257 D.1 Policy Network and State Encoder Architecture Stateencoder. Thestateencoder (StateEncoder) produces a4 × dim p + 1- dimensional state vector from five heterogeneous inputs. Each input is projected independently to R dim p where dim p is the projection dimension The text embedding is drawn from the frozen BiLSTM inside the auditor (the same encoder that produces faithfulness scores), not from CLIP, Us- ing the auditor’s frozen BiLSTM encoder biases the representation toward features relevant to the auditor’s faithfulness and safety objectives. The image embedding is the global average pool of the auditor’s ResNet-101 features (R 256 ) at the current timestep, also frozen. The latentz t is spatially pooled to4 × 4be- fore flattening, yielding a4 × 4 × LATENTC descriptor (64 dimensions whenLATENTC = 4 ).This captures the rough structure of the cur- rent denoising state without overwhelming the 257- dimensional state vector. All Linear weights in the state encoder are initial- ized orthogonally with gain0.1; biases are initial- ized to zero. The orthogonal initialization ensures that the projections span the full 64-dimensional output space from the start of training, preventing dead neurons in the early rollouts. Policy network architecture. TheGuarded tournament policyis a 3-layer MLP followed by three heads: • Mean head:Linear(64, 5)→ Sigmoid, output μ∈ [0, 1] 5 (normalized continuous knobs). •Log-stdhead:Linear(64, 5) → clamp(·,−4, 0.5), output logσ ∈ [−4, 0.5] 5 . Figure 8: BCO training dynamics. Top:h S (green),h U (red), and their gap (blue dashed) over training steps, annotated with the three phases: SFT exploitation and smudging (I), Quadratic Hinge activation (I), clothing discovery (I). Middle:∆N(ˆz 0 -space MSE gap on unsafe samples) showing the V-shape recovery after step 1500. Bottom: Sigmoid saturation % showing the transition from continuous to binary decision regime at step 1500. •Seed head:Linear(64, 10), raw logits for a 10-bucket Categorical distribution over seed offsets. LayerNorm after each hidden layer is critical here. The state vector mixes inputs with very different scales (text embeddings∼O(0.1), latent∼O(1), mask mean∈ [0, 1], timestep∈ [0, 1]), and without normalization training is unstable. Action sampling. The policy samplesN = 5 knob sets from the joint distributionπ θ (a | s), factorized as: π θ (a| s) = 5 Y k=1 N a cont k ;μ k ,σ 2 k | z continuous knobs × Cat a disc ; softmax(ℓ) | z seed bucket . where the continuous samples are clipped to [0, 1]before denormalization to the physical knob range. The log-probability of a joint sample is: logπ θ (a| s) = 5 X k=1 logN a cont k ;μ k ,σ 2 k + log Cat a disc ; softmax(ℓ) . Note that the log-prob is computed with respect to the pre-clip sample. This is a slight approxima- tion (the true log-prob of the clipped sample would require the clipped truncated normal), but in prac- tice the clip is rarely active sinceμ∈ [0.15, 0.85] after warm-up andσ≲ 0.3, placing> 99%of mass away from the boundaries. Physical knob ranges and denormalization. After samplinga cont k ∈ [0, 1], each dimension is denormalized via: v k = ℓ k + a cont k · (h k − ℓ k ), where (ℓ k ,h k ) are the physical bounds: KnobMinMax CFG scale1.015.0 Mask dilation0.01.0 Mask feather0.01.0 Noise jitter0.00.5 Inversion depth110 Seed offset0900 D.2 Guarded Tournament Policy Loss: Full Derivation Leave-one-out softmax advantage. GivenN candidate utilitiesu 1 ,...,u N from a single tourna- ment, the advantage weight for candidate i is: w i = exp(u i /τ ) P N j=1 exp(u j /τ ) − 1 N τ = std(u 1 ,...,u N ) Whysubtract1/N? Thesoftmaxterm softmax(u/τ ) i gives the posterior probability of candidateibeing the winner under a Boltzmann distribution over utilities. Without the−1/Nterm, Figure 9: Guarded Tournament Policy andStateEncoderarchitecture. TheStateEncoderprojects five heterogeneous inputs : the auditor’s frozen BiLSTM text embedding (512-d), the pooled VAE latentz t , the auditor’s image embedding (256-d), scalar mask coverage ̄m, and normalized timestep t norm into a unified 257-dimensional state vector via independent 64-dimensional projections followed by concatenation. TheGuarded Tournament Policy processes this state through a 3-layer MLP (256→128→64, LayerNorm + SiLU) and predicts a Gaussian over five continuous knobs (mean and log-standard deviation heads) and a Categorical over ten seed buckets (seed head). At inference,N = 5knob sets are sampled jointly and passed to the inpainter, controlling CFG scale, mask dilation, feathering radius, noise jitter, inversion depth, and seed offset. this reduces to standard REINFORCE where the baseline is zero, all candidates with nonzero utility receive positive credit, including mediocre ones. Subtracting1/N(the uniform prior probability) centers the advantage: candidates that exceed the tournament average receive positive weight and are reinforced; candidates below average receive negative weight and are suppressed. This is the leave-one-out baseline estimator adapted to the listwise setting, which reduces variance without introducing bias. Why temperatureτ = std(u i )?A fixed tem- perature would make the weight distribution arbi- trarily sharp (all mass on the winner) or flat (uni- form, collapsing to zero advantage) depending on the scale of utilities, which varies across tourna- ments. Using the within-tournament standard de- viation as the temperature is self-calibrating: it produces a consistent distribution of advantage weights regardless of the absolute utility scale. When all utilities are zero (no candidate beats the control),τ → 0and is clamped at10 −6 to prevent division by zero; the resulting weights are approxi- mately uniform and the gradient is near zero. Policy gradient. Given accumulated advantage weights and log-probabilities from a mini-batch of Btournaments (B× Ntotal sampled candidates), the policy-gradient objective is: L PG =− B×N X i=1 w i · logπ θ (a i | s i ), where: ComponentOperation InputState vector s∈ R STATEDIM Layer 1Linear(STATEDIM → 256) + LayerNorm + SiLU Layer 2Linear(256→ 128) + LayerNorm + SiLU Layer 3Linear(128→ 64) + LayerNorm + SiLU Shared TrunkFeature representation shared across policy heads Mean HeadLinear(64→ NUM CONTINUOUS) + Sigmoid Log-Std HeadLinear(64→ NUM CONTINUOUS), logσ ∈ [−4, 0.5] Seed HeadLinear(64→ NUM SEEDBUCKETS) Continuous SamplingGaussian sampling using σ = exp(logσ) Discrete SamplingCategorical sampling over seed buckets OutputHybrid continuous-discrete action vector Table 7: Detailed Guarded Tournament Policy architecture consisting of a shared multilayer perceptron backbone followed by hybrid continuous and discrete action heads. • Bis the number of tournaments in the mini- batch. • Nis the number of sampled candidates per tournament (N = 5 in all experiments). • s i is the StateEncoder output for candidate i, containing the prompt embedding, latent representation, image embedding, mask cov- erage, and normalized timestep. • a i is the sampled joint action (knob configura- tion), consisting of: a i = (CFG scale, mask dilation, mask feather, noise jitter, inversion depth, seed bucket). • π θ (a i | s i )is the policy distribution parame- terized by θ. • logπ θ (a i | s i ) is the log-probability assigned by the policy to the sampled action. • w i is the centered leave-one-out softmax ad- vantage computed from the tournament utili- ties: w i = softmax(u i /τ )− 1 N . Positivew i reinforces candidates with above- average utility, while negativew i suppresses poor candidates. • The leading negative sign converts the maxi- mization of expected utility into a minimiza- tion objective compatible with gradient de- scent. The advantage weightsw i are treated as fixed (detached from the computation graph) during op- timization, preventing gradients from propagating through the utility computation itself. Entropy regularization.Separate entropy terms prevent mode collapse for the continuous and dis- crete heads: H cont = 5 X k=1 1 2 log(2πeσ 2 k ), H disc =− 10 X b=1 p b logp b , wherep b = softmax(ℓ) b . Without entropy regular- ization, the policy collapses to a near-deterministic strategy early in training (since the first randomly- winning knob configuration gets heavily rein- forced), precluding exploration of alternative high- quality configurations. Compute penalty. The raw continuous action a cont 4 (the normalized inversion depth, dimension index 4) proxies for compute cost: L cost = 1 B× N B×N X i=1 a cont i,4 , which in expectation equalsE[a cont 4 ] = E[(d − 1)/(10− 1)] wheredis the inversion depth. Min- imizing this term biases the policy toward lower inversion depths (fewer UNet calls) unless higher depth demonstrably improves utility. Diversity regularization. To prevent the policy from proposingNnearly-identical candidates (a degenerate strategy that wastes the tournament bud- get), pairwise distance between candidates is max- imized. When candidate image embeddings are available (from the auditor’s ResNet-101 features): L div =− 1 N 2 X i<j ∥f i − f j ∥ 2 , wheref i ∈ R 256 is thei-th candidate’s image em- bedding. When embeddings are unavailable (early in training before candidates have been scored), the diversity term falls back to pairwise distances in the normalized raw action space: L fallback div =− 1 N 2 X i<j ∥a cont i − a cont j ∥ 2 . The action-space fallback is less meaningful (dif- ferent actions can produce similar images) but is nonzero and continues to push the policy away from collapsed modes. L GT =L PG − λ cont H H cont − λ disc H H disc + λ c L cost − λ div L div , with hyperparameters:λ cont H = 0.01,λ disc H = 0.005, λ c = 0.005, λ div = 0.01. D.3 Guarded Utility: Time-Varying Thresholds The implementation uses time-varying thresholds that change as a function of the normalized timestep t norm = t/T : τ P (t norm ) = 0.40 + 0.25· t norm τ F (t norm ) =              0.30 + 0.30t norm , t norm < 0.85, 0.55− 0.10 t norm −0.85 0.15 , t norm ≥ 0.85. Why doesτ F have a non-monotone profile? The faithfulness threshold peaks aroundt norm ≈ 0.85(τ F ≈ 0.555) and then relaxes slightly. This reflects two competing effects: 1. Mid-trajectory faithfulness matters most. At t norm ≈ 0.8-0.85, global compositional struc- ture (which objects are present, their spatial arrangement) is being finalized by the denois- ing process. Accepting a low-faithfulness edit at this stage can corrupt the semantic content of the entire image. The highτ F prevents such premature destructive edits. 2. Late-trajectory forced editing. Att norm ≈ 0.9- 1.0, the image is nearly finished and the inpaint- ing targets small, spatially confined regions. A strict faithfulness threshold at this stage risks rejecting valid edits that replace NSFW content with safe completions simply because the safe completion differs semantically from the orig- inal, which is the intended behavior. Slightly relaxingτ F at the final steps prevents forced edits. Complete guarded utility. The full utility com- putation is u i = [1[P R i ≥ τ P (t norm )] |z policy gate ·1[F R i ≥ τ F (t norm )] | z faithfulness gate · B i |z seam quality ], D.4 Seam Quality Metric The seam quality scoreB i quantifies how well the inpainted candidate blends with the surrounding context at the mask boundary. A candidate that passes both the policy and faithfulness gates but introduces a visible seam would be rejected,B i encodes this rejection criterion numerically. Computation. Let∂mdenote the ring-shaped boundary region of the mask, defined as: ∂m = Dilate(m,k)− Erode(m,k) k = 2r ring + 1 = 17, wherer ring = 8pixels (at512× 512). This ring isolates the mask boundary where seam artifacts appear. The seam score is computed via LPIPS (Zhang et al., 2018) restricted to the ring region: B i = exp − κ· P p∈∂m LPIPS p (C i ,C 0 ) |∂m| , κ = 5.0, whereC 0 is the control (unedited) image and∂mis the ring mask at the appropriate spatial resolution. B i = 1.0indicates a seamless edit (zero LPIPS difference at the boundary);B i → 0indicates a severe seam artifact. Why exponential?The exponential mapping en- suresB i ∈ (0, 1]with a smooth decay: small LPIPS differences at the boundary (good blend- ing) produceB i ≈ 1; a mean LPIPS of0.2(per- ceptible difference) producesB i = e −1.0 ≈ 0.37, substantially downweighting such candidates in the utility computation. After the guarded tournament selects a winning inpainted candidateC i ∗ in pixel ModelConfig. c = 3c = 5c = 7 Acc. Sug. % Acc. Time (s)Acc. Sug. % Acc. Time (s)Acc. Sug. % Acc. Time (s) SD 1.5GTP1.00 1.4071.427.981.00 1.2083.332.281.10 1.4078.634.52 SD 1.5No GTP 0.80 1.2066.729.011.10 1.4078.633.800.80 1.2066.737.27 SDXL Base 0.9GTP0.90 1.1081.843.281.10 1.3084.648.441.00 1.1090.949.67 SDXL Base 0.9No GTP1.00 1.3076.943.430.90 1.1081.848.811.10 1.3084.652.96 SD 3.5 MediumGTP1.00 1.8055.655.781.00 1.6062.561.671.10 1.7064.766.03 SD 3.5 MediumNo GTP0.80 1.6050.062.601.10 1.8061.167.690.90 1.8050.070.49 SD 3.5 Large Turbo GTP1.00 1.2083.387.651.00 1.2083.385.751.10 1.2091.788.54 SD 3.5 Large Turbo No GTP 1.00 1.3076.986.831.10 1.2091.789.201.10 1.3084.6104.73 FLUX.1-devGTP1.00 1.1090.9117.51.00 1.00100.0135.61.10 1.2091.7145.0 FLUX.1-devNo GTP1.00 1.2083.3120.31.10 1.2091.7138.81.10 1.2091.7148.4 Table 8: Compact GuardPaint ablation across evaluated architectures. For each correction budgetc, we report mean accepted corrections per prompt (Acc.), mean tournaments triggered per prompt (Sug.), acceptance percentage (% Acc.), and mean wall-clock time per prompt (Time (s)), averaged over 10 adversarial prompts. Bold indicates the better value per metric within each model–budget pair. space, the repair must be reintroduced into the live diffusion trajectoryz t without disrupting the de- noising manifold. Naive injection, encodingC i ∗ with the base VAE and substituting the resulting latent directly intoz t consistently produces visible boundary seams and color drift in downstream de- noising steps, because the re-encoded latent does not lie on the trajectory the base UNet expects. We therefore designed and evaluated four reinser- tion strategies, formalized below, before selecting the production method. All four share the same preparatory step:C i ∗ is resized to the pixel reso- lution implied byz t ’s spatial dimensions, encoded with the base model’s VAE (not the inpainter’s VAE) to obtainˆz edit 0 ∈ R C×H ′ ×W ′ , and a binary maskm ∈ 0, 1 H ′ ×W ′ is resized to latent reso- lution. The output of each strategy is the updated latent z new t passed to the next UNet step. 1. DDPM noise blending This approach de- grades the clean edited latent to the current timestep’s noise level via the standard forward dif- fusion process (Ho et al., 2020) before spatial mask- ing: ˆz edit t = √ ̄α t ˆz edit 0 + √ 1− ̄α t ε,ε∼N (0,I), z new t = (1− m)⊙ z ctrl t + m⊙ ˆz edit t . Failure: Inside the masked region,ˆz edit t contains freshly sampled, independent noiseε. Outside the mask,z ctrl t contains the specific historical noise sequence accumulated during the generation pro- cess. This statistical independence creates a sharp discontinuity at the boundary. The UNet interprets this boundary as a structural edge, generating per- sistent incorrect artifacts in subsequent denoising steps. 2. Direct latent blending Rather than matching noise levels, this method directly interpolates the clean edit into the noisy control latent using a time- decaying weight α (Avrahami et al., 2022): z new t = 1− α· m ⊙ z ctrl t + α· m⊙ ˆz edit 0 , where α = 1− t norm . Failure: Becauseˆz edit 0 represents a fully denoised state (t=0) andz ctrl t is noisy, their linear combina- tion falls outside the expected distribution of the diffusion process. Consequently, the UNet under- denoises the injected region, yielding flat, over- saturated patches. This degradation is most severe at earlier generation stages (lowert norm ), which is precisely when GuardPaint interventions are most critical. 3. DDIM inversionThis strategy attempts to find a compatible noise state by invertingˆz edit 0 forward to timesteptusingd=10deterministic DDIM steps (Song et al., 2021b) prior to blending: ˆz inv t = DDIMInv ˆz edit 0 , t, d , z new t = (1− α· m)⊙ z ctrl t + α· m⊙ ˆz inv t , where α = 1− t norm . Failure: DDIM inversion under classifier-free guidance introduces reconstruction errors that scale inversely withd(Mokady et al., 2023b). More problematically, the inversion is conditioned on the original adversarial text promptc text . This conditioning pulls the latent back toward the un- safe concept, actively fighting the safety correction achieved by the tournament. Adjustingdoffers no viable compromise: small values produce severe boundary seams, while larger values drastically increase latency without matching the quality of null-text methods. 4. Null-text inversion (selected method). We adapt null-text inversion (Mokady et al., 2023b) to function mid-trajectory. Rather than altering the latents directly, we freeze the UNetf θ and the text conditioningc text , and optimize a learnable unconditional embedding∅ ∗ . The objective forces the UNet’s single-step prediction to align perfectly with the clean edit: ∅ ∗ = arg min ∅ ˆz 0 ˆz edit 0 , t,∅, c text − ˆz edit 0 2 2 whereˆz 0 (·)denotes the UNet’s predicted clean la- tent under CFG (Ho and Salimans, 2022). This objective is minimized for 10 AdamW steps (lr=0.01). The latent is then updated via feathered mask blending: z new t = 1− α· m ⊙ z ctrl t + α· m⊙ ˆz edit 0 , where α = 1− t norm . Why null-text inversion succeeds. UNet’s pre- diction becomes self-consistent as a single denois- ing step from the blended latent reproducesˆz edit 0 inside the mask and preserves the control trajectory outside it. This resolves the independent noise dis- continuities of DDPM blending, the state mismatch of direct blending, and the adversarial concept leak- age of DDIM inversion. Furthermore, the brief 10- step optimization overhead is incurred only once per audited timestep and is amortized efficiently across all candidates via a per-timestep embedding cache. Flow-matching architectures.For flow- matching models (e.g., SD 3.5 and FLUX.1-dev) where the DDPM forward process does not ap- ply, the noise-matched latent is approximated as (1− σ t )ˆz edit 0 + σ t ε(?), whereσ t represents the schedule flow sigma. Adapting native null-text inversion to flow-matching paradigms remains an area for future work. D.5 Full Training Configuration ParameterValueNotes Policy network State dim d s 2574× 64 + 1 Projection dim64Per input Hidden dims(256, 128, 64)LayerNorm + SiLU Log-std clamp[−4.0, +0.5] Weight initOrthogonal, gain 0.01 Tournament Candidates N5 Audit start70% of schedule Audit resolution224 px (t < 0.65), 384 px (t≥ 0.65)Coarse/fine δ init0.05Recalibrated every 100 steps δ min samples50Before first recalibration GTP loss λ cont H 0.01Continuous entropy λ disc H 0.005Discrete entropy λ c 0.005Compute penalty λ div 0.01Diversity λ listnet 1.0ListNet weight in judge λ PL 0.5Plackett-Luce weight in judge Optimization Policy optimizerAdamW, lr 3×10 −4 Judge optimizerAdamW, lr 1×10 −4 Mini-batch size4 tournaments (policy), 8 (judge) Gradient clipping 1.0Both policy and judge Total steps1000 (convergence≈400–600) Seam quality Ring width8 px LPIPS netVGG (spatial), κ = 5.0 Time-varying thresholds τ P (t)0.40 + 0.25t τ F (t)τ F (t norm ) =            0.30 + 0.30t norm , t norm < 0.85, 0.55− 0.10 t norm −0.85 0.15 , t norm ≥ 0.85. Non-monotone α(t)0.30 + 0.60tBlend coefficient Table 9: Complete Guarded Tournament Policy training and inference configuration. Figure 10: Shows the qualitative difference between different reinsertion strategies tried for Stable Diffu- sion 1.5 and SDXL 0.9 Base.

Instant research discovery