Paper deep dive
Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning
Yijia Fan, Ziqi Huang, Zhongang Cai, Yan Li, Zimo Wen, Wanqi Yin, Haiwen Diao, Ziwei Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Unified multimodal models can both look at and render images, so in principle they can repair their own generations: diagnose what an image gets wrong, revise it, observe the result, and diagnose again. Whether a revision helps is known only after it is rendered, so the reflection text and the image generation must be learned jointly, over the whole loop. Supervised fine-tuning (SFT) on reflection trajectories gives a cold start but does not find the high-success repair paths, and naive RL that optimizes only the renderer or only one head leaves most of the gain untapped. We introduce UMM-Reflection, which applies reinforcement learning (RL) to complete reflection trajectories inside one unified model: sibling trajectories share one initial image, so the group-relative advantage compares reflection strategies, and one trajectory-level advantage updates both the reflection tokens and the flow-based revisions, avoiding the combinatorial blow-up of per-round credit assignment. Unlike single-round editing or pipelines with an external critic, credit flows across rounds and to both roles of the same model, and no verifier is needed at inference. On BAGEL, UMM-Reflection improves GenEval by 12.05 points over SFT, and the gains transfer to WISE (+10.97), OneIG-Bench (+3.48), and T2I-CompBench++ (+4.63), none of which is used in training.
Tags
Links
- Source: https://arxiv.org/abs/2609.35767v1
- Canonical: https://arxiv.org/abs/2609.35767v1
Trouble viewing inline? Open PDF directly →
Full Text
103,178 characters extracted from source content.
Expand or collapse full text
LearningNativeReflectioninUnifiedModelswith InterleavedReinforcementLearning YijiaFan 1,∗,† ,ZiqiHuang 1,∗ ,ZhongangCai 1 ,YanLi 2 ,ZimoWen 2 ,WanqiYin 3 ,HaiwenDiao 1 ,ZiweiLiu 1,‡ 1 Nanyang Technological University, 2 Shanghai Jiao Tong University, 3 The University of Tokyo ∗ Equal contribution, † Work done during an internship at NTU, ‡ Corresponding author Unified multimodal models can both look at and render images, so in principle they can repair their own generations: diagnose what an image gets wrong, revise it, observe the result, and diagnose again. Whether a revision helps is known only after it is rendered, so the reflection text and the image generation must be learned jointly, over the whole loop. Supervised fine-tuning (SFT) on reflection trajectories gives a cold start but does not find the high-success repair paths, and naive RL that optimizes only the renderer or only one head leaves most of the gain untapped. We introduce UMM-Reflection, which applies reinforcement learning (RL) to complete reflection trajectories inside one unified model: sibling trajectories share one initial image, so the group-relative advantage compares reflection strategies, and one trajectory-level advantage updates both the reflection tokens and the flow-based revisions, avoiding the combinatorial blow-up of per-round credit assignment. Unlike single-round editing or pipelines with an external critic, credit flows across rounds and to both roles of the same model, and no verifier is needed at inference. On BAGEL,UMM-Reflectionimproves GenEval by 12.05 points over SFT, and the gains transfer to WISE (+10.97), OneIG-Bench (+3.48), and T2I-CompBench++ (+4.63), none of which is used in training. ProjectPage: https://waltstephen.github.io/UMM-Reflection GitHubRepo: https://github.com/waltstephen/UMM-Reflection HuggingFaceModels&Data: https://huggingface.co/collections/YijiaFan/umm-reflection VideoDemo: https://w.youtube.com/watch?v=YRfpcs4pm-s 1 Introduction As instructions grow compositional, a single render often carries flaws [11,15] that a one-shot text-to-image model cannot notice, let alone fix. Unified multimodal models place visual understanding and image generation in one network [8,32,36,38]: the model that renders an image can also look at it. This enables the loop of inspection, diagnosis, and revision that language models use for self-correction, beyond step-by-step reasoning [13,19,21,35]: in principle, a unified model can find the flaws in its own image and render the fix; UMM-Reflection trains it to do so. Work on unified models approaches this loop from two sides. One line applies reinforcement learning, but to a single render: T2I-R1 and ReasonGen-R1 apply GRPO to a textual plan and the single image rendered from it [18,44], and UniRL turns the model’s understanding of its finished image into a reward for generation [22]. The model never revises what it rendered. The other line lets the model inspect an intermediate image and continue, learned by supervised imitation of multi-round reasoning-and-editing trajectories [5,12]. Imitation gives these models a cold start, but it does not ensure that a reflection leads to an effective correction, a gap we quantify in Section 6. What is missing is reinforcement learning over a unified model’s own multi-round reflection: a signal that credits each reflection for the visual improvement it produces, rather than for matching a demonstration. Attaching RL naively does not supply it: optimizing only the renderer, or only one head of the loop, leaves most of the gain untapped (Section 5). We call this native reflection: multi-round inspect–diagnose–revise behavior carried out by the unified model itself. Native reflection is what makes the loop trainable as a whole. The reflection and the revision it triggers 1 arXiv:2609.35767v1 [cs.CV] 28 Sep 2026 (a) Reflect and revise within one model GenEval Create an image with a dog right of a tie. SFT R0R1R2 R3 Final "Move the tie from the floor to the right side" "Move the dog from the left side to the right side" "Move the tie from the floor to the right side" RL R0R1R2 R3 Final "Place a tie to the left of a dog" "upright and clearly separate from the dog" "positioned to the left of the dog" (b) RL directs the revisions Image-state space (schematic) Before RL: SFT R0 R1 R2 R3 Correct-image region Whole-trajectory RL After RL R0 R1 R2 R3 Correct-image region SFT revisions circle; RL revisions reach the correct region. Interleaved reflection loop One unified model Current image Reflection text (Same UMM backbone) Image editing (Same UMM backbone)Revised image Next round Figure1 NativereflectionbeforeandafterRL. Left: SFT and RL on the same prompt and seed. SFT already produces meaningful revisions (16 SFT rollouts contain a correct repair for 78% of failing training images; Appendix I), but one trajectory often circles, as here (tie, dog, tie). RL concentrates the policy on revisions that reach the correct region (right; measured in Figure 5). come from the same parameters: the text head writes the diagnosis, and the flow head renders the next image conditioned on it. One outcome reward can therefore update both heads along the same trajectory: a shared outcome-driven advantage reaches every textual and visual action of the trajectory, and the renderer is trained on the instructions it actually receives. This is also what separates the problem from single-round editing and from pipelines with an external critic: the value of a reflection is known only after the image it triggers is rendered, and often only after further rounds, so credit must flow across the whole trajectory and to both the diagnosis and the rendering. What remains is behavioral: the model must learn to turn a diagnosis of its own image into a generation action that fixes it. We learn native reflection withUMM-Reflection. SFT first teaches the interleaved protocol and meaningful revisions: in each round the model examines its current image, writes a reflection, and either generates a revised image or stops. Whole-trajectory RL then optimizes complete reflection sequences with two design choices. AllKsibling trajectories start from one shared initial image, so the group-relative advantage [28] compares reflection strategies rather than lucky first draws. One outcome-driven advantage per trajectory then updates both the reflection tokens and the flow transitions, so the model learns which reflections lead to better images without theK N rollouts that per-round credit would require or a learned value model. The training-time verifier is never consulted at inference. Figure 1 contrasts SFT and RL revisions of the same image. The central finding separates producing useful revisions from reliably choosing them. SFT learns more than the format (95% of its trajectories follow the protocol; Table 8): its rollouts already contain correct repairs (Figure 1). Yet a single SFT trajectory repairs only 20.59% of initially incorrect images; after RL, the conditional repair rate rises to 64.94% (Table 3). This gap translates to substantial accuracy gains on GenEval [11] (+12.05 over SFT) and WISE [23] (+10.97). Base, SFT, and RL start from nearly identical single-round accuracy (70–73): the first image receives no RL loss, and the gains come from the reflection rounds. Updating the generator alone with the same number of RL updates (direct T2I-RL) lifts single-shot GenEval from 71 to 76 but does not transfer (WISE 54 versus 55 for Base), whereasUMM-Reflectionreaches 84 and 74. Gains also transfer to OneIG-Bench [3] and T2I-CompBench++ [15], neither seen in training. Representation analysis (Section 6) shows that RL leaves the model’s perception and internal correctness readout nearly unchanged; among the revisions SFT already produces, it selects those that move a failing image into the region this 2 readout marks as correct (Figure 5), finding better repair paths rather than creating a new capability. We summarize our contributions as follows: •We enable reinforcement learning for multi-round reflection in unified models: one whole-trajectory advantage, computed over siblings that share one initial image, jointly optimizes the textual reflections and the flow-based revisions of the same model without per-round branching. •We show that imitation already teaches meaningful revisions but applies them unreliably, and that RL makes them reliable by selecting repair paths the backbone already has. • UMM-Reflection improves over reflection SFT on four benchmarks while training on one; ablations show that neither direct RL on the generator nor Best-of-4 selection matches it. 2 RelatedWork Self-correction loops, in text and in pixels. Self-Refine and Reflexion let a language model critique and revise its own draft by prompting alone [21,29], yet without external feedback such intrinsic self-correction can lower accuracy [14]. SCoRe traces this to offline correction traces and shows that online multi-turn RL on the model’s own attempts is needed [19,27]. Image generation has adopted the loop but not this lesson: Idea2Img, iterative refinement, ReflectionFlow, SLD, and GenArtist pair an external critic with a separate renderer [17,34,37,41,46]. The critic sees only pixels, neither model is optimized against the other, and the critic stays online at inference. We train both roles as one policy under one outcome reward and drop the verifier at inference. Reinforcement learning for visual generators. DDPO and DPOK optimize the denoising chain with policy gradients [1,9], ImageReward and Diffusion-DPO learn from human preferences [30,39], and Flow-GRPO and DanceGRPO bring group-relative optimization [28] to flow-matching generators [20,40]. In each, the policy is a single prompt-to-image pass that never observes its own render. We place Flow-GRPO’s rendering transitions inside a multi-round trajectory whose single advantage credits both the reflection tokens and the renders they trigger. Reasoning and reflection in unified generators. Unified models share one network for understanding and generation via discrete tokens [2,32,38], decoupled visual encoders [6,36], text with a diffusion or flow decoder [8,45], or bridging queries [4,24]. One line improves a single render: T2I-R1 and ReasonGen-R1 apply GRPO to a textual plan and its image [18,44], UniRL rewards generation with the model’s own answers about its finished image [22], and PARM and GoT verify or structure the generation process [10,43]; none revises the image. A second line (Thinking with Generated Images, MINT, Uni-CoT, IRG, ThinkMorph, UniT) inspects an intermediate image and continues [5,7,12,16,25,33], but is trained mainly by imitating synthesized trajectories, which, as SCoRe predicts and Section 6 measures, gives a cold start without the high-success repair paths.UMM-Reflectionapplies outcome-driven RL to complete inspect-and-revise trajectories in one unified policy. 3 Methodology We formulate native reflection as a policy that repeatedly inspects, diagnoses, and revises its own image within a single unified model. This section describes the reflection protocol (§3.1), the trajectory data used to initialize it (§3.2), the supervised cold-start (§3.3), and the whole-trajectory RL stage that turns this cold start into effective repair (§3.4). 3.1 Reflectionprotocol For a request c, the model first produces an image x 0 . At each round t, the model observes the request, the text–image history, and the current image x t , and emits a structured reflection: u t ∼ π text θ (·| c, x ≤t , u <t ), a t ∈edit, done. 3 UMM-Reflection RL "a dog right of a tie"K = 16 Shared x 0 q 0.30 Training only Frozen verifier Group-relative advantage Text headFlow head A i Unified backbone Understanding experts Generation experts Shared attention τ 1 q 1.00 in a room with ties draped over furniture a dog to the right of a tie in the image. q 1.00 multiple ties draped around the dog. to the right of a single tie in the image. q 1.00 sitting inside a room with ties scattered... upright on the left side of the image R(τ) = 1.21 τ 2 q 0.36 on a floor with a tie around its neck a dog to the right of a tie in the image. q 0.15 but the tie is to the left of the dog. a dog to the right of a tie in the image. q 1.00 dog is seated on the left side of the image a dog to the far right side of the image R(τ) = 1.19 τ 16 q 0.30 current image shows a dog wearing a tie Move the dog to the right side of the image q 0.30 a couch with ties draped around it a dog on the right side of the image q 0.30 image shows two dogs in a room with ties a dog on the right side of the image R(τ) = 0.30 textimage generationverifier passfailgenerationtraining only 1 2 3 4 5 Figure2UMM-ReflectionRL. (1)K=16 rollouts share one detached initial imagex 0 . (2) Each interleaves the model’s own reflection (verbatim) with its renders for up to three rounds. (3) A frozen verifier scores every image (q) and trajectory (R(τ)). (4) Group normalization gives one advantageA i per trajectory, (5) which updates both the text and flow heads. Green/red frames: verifier pass/fail; dashed: training only. An edit action carries a natural-language correctione t ; the same model then rendersx t+1 ∼ π flow θ (·| c,x t ,e t ). A done action returns the current image. The loop runs for at most three repair rounds in the primary experiments. Each reflection is a tagged response whose main fields are[THINKING],[ACTION], and[EDIT](full format in Appendix B); verifier scores never enter the policy observation. Figure 2 shows the trajectory structure. 3.2 Trajectorydataconstruction Learning the protocol requires multi-round inspect–diagnose–revise trajectories, which the target model cannot yet produce. We generate them with external models and distill them into the unified model via SFT. GPT-5.5 acts as the critic: it turns each request into verifiable constraints, inspects each image, writes a structured reflection, and issues one atomic edit instruction or a done verdict. Qwen-Image renders the initial image and Qwen-Image-Edit executes each edit; BAGEL takes no part, so its own failure modes are not distilled back into the supervision. The 29,529 accepted trajectories are of three types: one-shot (9,000), where the initial image already satisfies all constraints; natural-repair (8,645), where a genuinely failing initial image is fixed in one or two rounds without injected corruption; and planned-progression (11,884), where a complex request is fulfilled over two or three ordered milestones. Prompts come from Puffin-4M, Poster100K, OmniEdit, AnyEdit, and GEdit-Bench, with no overlap with any evaluation benchmark. 3.3 Supervisedinitialization SFT teaches the interleaved protocol on the trajectories from §3.2 with an autoregressive loss on reflection text and a flow-matching loss on each edited image,L SFT =L AR +λ img L FM , starting from the base BAGEL checkpoint for one epoch (details in Appendix B). This stage supplies a consistent interface between diagnosis and corrective generation; the subsequent RL stage starts from this SFT checkpoint. 3.4 Whole-trajectoryreinforcementlearning SFT teaches the model to produce well-formed reflections, but well-formed text does not guarantee effective repair (§6). We now describe the RL stage that optimizes for visual outcomes. Shared-root sampling. For each request, we sample one initial image and detach it from the computation graph. RL optimizes only the reflection-and-editing rounds that follow; the initial text-to-image generation 4 receives no policy-gradient signal. 1 From those identical root pixels, we sampleK= 16 complete reflection trajectoriesτ i K i=1 , each running until its own done action or the repair cap. We do not prune siblings, retain only the best intermediate image, or use best-of-K selection at deployment. Reward. A frozen verifier assigns a graded alignment scoreq t ∈[0,1] to each image, aggregated from its own detector outputs under the official thresholds so that a partial repair yields a nonzero change (Appendix C). Let ∆ t = q t+1 − q t and [v] + = max(v, 0). The trajectory reward is R(τ ) = q T + α P t [∆ t ] + + β S multi (τ )− λ P t [−∆ t ] + − p1[premature done],(1) S multi (τ ) = P t [∆ t ] + − max [∆ t ] + t ∪0 .(2) The terminal scoreq T rewards the final image quality. The progress terms [∆ t ] + reward each round that improves the image; the damage penaltyλ P [−∆ t ] + discourages regressions.S multi adds credit when improvement comes from more than one edit rather than a single lucky fix; it deliberately favors trajectories that keep improving across rounds, since multi-round correction is the behavior we aim to train. Premature done means stopping when the verifier does not accept the current image. We useα=β= 0.3 andλ=p= 0.5. Group-relative advantage. Within each shared-root group, advantages are A i = clip R(τ i )− μ R max(σ R , 0.1) , −1, 1 . We assign one advantage per trajectory rather than per round. A group-relative estimate for each round would require sibling groups at every round: branchingKways at each ofNrounds needsK N rollouts per root (16 3 = 4,096 for three rounds), which is impractical for an image-generating policy. A per-round critic, as in PPO, would instead require training a value model over multi-round image–text states, with the data scale that entails. The trajectory-level advantage keeps theK-sample cost of GRPO while the per-round progress terms in R(τ ) still reward each round that improves the image. Text–flow coordination. LetI c i denote the policy-active positions for channelc∈text, flow, andρ c ij the corresponding likelihood ratio. The clipped surrogate for each channel is J c =E i,j∈I c i min ρ c ij A i , clip(ρ c ij , 1−ε c , 1+ε c )A i − η c K c . The key design choice is that both channels share the same trajectory-levelA i : the text policy and the flow renderer are not normalized separately. A reflection that leads to a better image raises the advantage for both the diagnostic tokens and the rendering transitions that followed, so the model learns which reflections lead to which visual outcomes.K c is a channel-specific KL penalty against the frozen SFT reference. Flow transitions use the Flow-GRPO SDE sampler [20]; per active repair we train on two contiguous stochastic transitions. For text, credited positions are sampled policy tokens excluding prompt and formatting. The verifier and the reference policy are used only during training; at inference only the unified model runs. 4 Experiments 4.1 Setup Training and evaluation. We build on BAGEL [8], the most widely used open unified model that both understands and generates images in one network, with understanding and generation experts that share attention; this lets one trajectory-level advantage update the reflection text and the renderer of the same model. RL starts from the reflection-SFT checkpoint (§3.3) and samples from a 3,000-prompt pool over six GenEval families, with two roots andK= 16 siblings per update and 20 training denoising steps; unless 1 BAGEL does not natively support interleaved text–image generation in a single forward pass. We implement the multi-round loop through an external controller that feeds each round’s reflection and image back into the model as a new turn, enforcing the interleaved protocol described in §3.1. 5 Table1 GenEval:allsixcompositionalrequirements. Native 0–1 scores. Res.: image side (n/r: not reported);†: reported in another model’s paper. Published settings differ from our one-image, instruction-voice evaluation; e.g., BAGEL reports 0.82 under its native protocol [8], whereas the local rows use our protocol (Appendix A). Bold: best in the local block. ModelRes. Single Two Count Colors Position Binding Overall Reported results: native settings and original reporting precision PixArt-α † [38]512 0.98 0.50 0.440.800.080.070.48 SD2.1 [11]768 0.98 0.51 0.440.850.070.170.50 DALL·E 2 † [38] 1024 0.94 0.66 0.490.770.100.190.52 Emu3-Gen [32]512 0.98 0.71 0.340.810.170.210.54 SDXL [11]1024 0.98 0.74 0.390.850.150.230.55 DALL·E 3 † [6]1024 0.96 0.87 0.470.830.430.450.67 SD3-Medium † [6] n/r 0.99 0.94 0.720.890.330.600.74 LWM † [38]n/r 0.93 0.41 0.460.790.090.150.47 SEED-X † [38]n/r 0.97 0.58 0.260.800.190.140.49 TokenFlow [26]256 0.97 0.66 0.400.840.170.260.55 ILLUME [31]512 0.99 0.86 0.450.710.390.280.61 Janus [36]384 0.97 0.68 0.300.840.460.420.61 Show-o-512 [38]512 0.98 0.80 0.660.840.310.500.68 Janus-Pro-7B [6] 384 0.99 0.89 0.590.900.790.660.80 Local evaluation: same prompts, resolution, and scorer BAGEL-Base512 1.00 0.89 0.630.820.470.480.71 BAGEL-SFT512 1.00 0.88 0.580.870.470.510.72 UMM-Reflection5120.950.960.680.900.890.650.84 Table2 TransfertobenchmarksunseeninRLtraining. Native 0–1 scale; gray values are changes relative to BAGEL-Base. ModelWISEOneIG CompBench Local evaluation: same prompts, resolution, and scorer BAGEL-Base0.550.800.49 BAGEL-SFT0.63 (+0.08) 0.79 (-0.01) 0.50 (+0.01) UMM-Reflection0.74 (+0.19)0.83 (+0.02)0.55 (+0.06) otherwise specified, RL runs for 1,000 updates. We evaluate one image per prompt with 50 denoising steps at 512 2 and at most three repairs, using the same checkpoint on GenEval (all 553 official prompts, unfiltered), WISE (1,000), OneIG-Bench (OneIG; 695 alignment prompts), and T2I-CompBench++ (CompBench; 2,400); protocol and scorer details are in Appendix A. Baselines. The main comparison uses the same BAGEL backbone throughout: Base, unmodified BAGEL, single-pass generation at 512 2 , and SFT, the reflection-supervised parent, producing multi-round trajectories without RL. Ablation baselines that remove or replace one ingredient are defined in Section 5. 4.2 Mainresults In-domain results. Table 1 placesUMM-Reflectionamong reported GenEval results. Under the same protocol,UMM-Reflectionreaches 0.84, against 0.71 for BAGEL-Base and 0.72 for reflection SFT (+12 points over SFT). The gain is concentrated in the families that require fixing a composition: position rises by +42.00 points over SFT (0.47 to 0.89), color binding by +14.00, and counting by +10.00. Paired over prompts, RL beats SFT on 109 prompts and loses on 38 (p <10 −8 , McNemar), while on the initial images alone the split is 48–32 (p = 0.09, no significant difference), and the gain is made in the reflection rounds. Transfer. Table 2 evaluates the same checkpoint on three benchmarks never used in RL training.UMM-Reflection improves over reflection SFT on all three: +10.97 points on WISE, +4.63 on CompBench, and +3.48 on OneIG, 6 InitialRevision 1Revision 2Revision 3InitialRevision 1Revision 2Revision 3 GenEval:“a sports ball left of an umbrella”OneIG: “Please generate an image capturing a moment in a stylish cafe, where two young women are enjoying coffee. Picture a ...” CompBench:“a blue backpack and a gold clock”GenEval:“a computer keyboard and a microwave” CompBench:“a white snow and a blue sled”GenEval:“a dog right of a tie” WISE: “An autumn delight, a fruit with cultural significance in Japan, often seen in traditional dishes and rituals”CompBench:“a plastic phone case and a fabric curtain” Figure3 Casestudiesacrossfourbenchmarks. Each row shows the prompt, the initial image, and three reflection-guided revisions (left to right), with examples from GenEval, WISE, OneIG, and CompBench. The model identifies spatial, color, material, and compositional errors through its [THINKING] output and issues targeted edits. where SFT alone falls slightly below Base. Figure 3 shows reflection trajectories on all four benchmarks. Section 5 isolates the contribution of each ingredient. 4.3 Multi-roundtest-timescaling 0123 Reflection rounds 70 75 80 85 GenEval macro accuracy (%) 83.8 Base Reflection SFT UMM-Reflection-500 UMM-Reflection-1000 Figure4 Test-timescalingonGenEval. Macro accuracy versus reflection rounds. Other bench- marks: Appendix Figure 9. Figure 4 shows the per-round GenEval score; the other three benchmarks follow the same pattern (Appendix Figure 9). SFT’s three rounds add roughly +2 points on GenEval and flatten after round 1; the model edits but does not reliably improve. After RL, round 1 alone adds +9 points, and the model continues to gain through round 3. The initial-image accuracy (R0) is comparable across arms (70–73), confirming that the gap comes from multi-round correction, not a better first image. The conditional repair rate rises from 20.59% (SFT) to 64.94% (RL 1000). Within GenEval families, the largest gains are on position (+34) and color_attr (+17). Full per-round and per-family statistics are in Appendix K. 5 Ablations Table 3 isolates each ingredient; the results support five con- clusions. 7 Table3 AblationsonGenEval(553officialprompts). Six-family macro accuracy (0–100). Repair and Damage are percentages of initially incorrect images fixed and initially correct images broken; ∆ is computed before rounding. All reward variants share the same SFT parent, prompt pool, and training topology. R0 Final∆ Repair Damage Edits Images Components BAGEL-Base71710–01 + direct RL on the renderer (T2I-RL)76760–01 + inspect-and-edit loop (Self-Agentic)7177 +5 27.63.6 3.004.0 + reflection SFT7072 +2 20.66.5 1.652.7 + reflection SFT + trajectory RL (UMM-Reflection)7384+1164.98.83.004.0 direct RL → reflection SFT → trajectory RL (500 updates only) 7481 +7 48.37.9 3.004.0 Training length (same run) reflection SFT + trajectory RL, 100 updates7175 +3 35.69.5 3.004.0 reflection SFT + trajectory RL, 500 updates7182 +11 61.19.3 3.004.0 UMM-Reflection (1,000 updates)7384+1164.98.83.004.0 Head ablation (the other head frozen) flow-only RL (frozen text head)7173 +2 22.87.3 3.004.0 text-only RL (frozen flow head)7178 +7 49.410.1 3.004.0 joint RL (UMM-Reflection)7384+1164.98.83.004.0 Same four-image budget Best-of-4 T2I-RL, selected by Base UND7680 +4–04 Best-of-4 T2I-RL, selected by UMM-Reflection UND7680 +5–04 reflection SFT, forced to edit in all three rounds7072 +2 28.89.734 UMM-Reflection7384+1164.98.83.004.0 Reward terms UMM-Reflection7384+1164.98.83.004.0 without the multi-improvement term (β = 0)6379 +16 61.711.2 3.004.0 The gains stack. Direct Flow-GRPO on the renderer (T2I-RL, 1,000 updates from Base) raises single-shot accuracy from 71 to 76 but leaves nothing to repair. Forcing the untuned Base through three inspect-and-edit rounds (Self-Agentic) adds +5 without training. Reflection SFT adds +2; fine-tuning Base on the final images of the same 29,529 trajectories leaves single-pass accuracy on the raw official prompt at 75, the same as Base, so the SFT images alone do not improve the generator. Trajectory RL on top of SFT adds +11 and triples the repair rate from 21% to 65%. Placing direct RL before both stages carries its single-shot advantage through, reaching 81 after 500 updates. Most of the gain arrives within 500 updates. Along the same run, GenEval rises from 72 (SFT) to 75, 79, and 82 after 100, 200, and 500 updates, and reaches 84 at 1,000. The repair rate follows the same path, from 21% to 61% at 500 updates and 65% at 1,000, while damage stays between 8% and 10%. Initial-image accuracy stays at 71–73 throughout, so the gain comes from the reflection rounds at every checkpoint. Both heads must be trained. Freezing one head while applying the same trajectory RL isolates what each side contributes. Training only the flow head leaves the model close to SFT (73, repair rate 22.8%): a better renderer does not help when the reflections that drive it do not improve. Training only the text head recovers most of the gain (78, 49.4%), so learning what to write is the larger part. Joint training reaches 84 and 64.9%, six points above the best single head: the renderer must also learn to execute the reflections the text head now writes. This is the joint optimization that a single unified model makes possible. Holding image, renderer, and noise fixed and swapping only the instruction confirms that the learned text carries the repair (Appendix J). The gain is not best-of-Nsampling or extra edits. At the same four-image budget, selecting one of four images from the stronger T2I-RL renderer with a single native understanding call (Best-of-4) reaches 80 with either selector;UMM-Reflectionreaches 84 (paired McNemarp ≤0.04). Forcing SFT to edit in all three 8 −505 −6 −4 −2 0 2 GEN depth-8 coordinate RL, R0: 34% inside the dense pass region −505 RL, R3: 62% inside the dense pass region −505 SFT, R0: 36% inside the dense pass region −505 SFT, R3: 42% inside the dense pass region UND depth-20 coordinate (negative held-out probe logit) passing reference imagesfailing reference imagesRL: initially failing imagesSFT: initially failing images Figure5 Imagetrajectoriesinthebackbone’sowncorrectnessreadout. Each point is one initially failing image, placed by two held-out linear probes for “passes the verifier”: the understanding stream at depth 20 (x) and the generation stream at depth 8 (y), the depths where each stream’s probe AUC peaks. Green filled contours mark verifier-passing reference images and red dashed contours verifier-failing ones; orange and blue contours summarize the RL and SFT images. Read the panels in pairs, from R0 to R3 for the same policy: RL (left pair) and SFT (right pair). Percentages are the share inside the dense pass region (Table 5). rounds leaves it at 72, the same as unforced SFT. On the 62 prompts where all four independent Base draws fail, RL reflection recovers 60% (SFT: 12%). The multi-improvement term matters. Removing the multi-improvement term (β= 0) preserves the repair rate but degrades the initial image from 73 to 63, also yielding 79. 6 Study:WhatChangesInsidetheModel 6.1 Trainingdynamics:theinterfacelocksinfirst,thenrepairimproves Appendix Figure 8 summarizes the 1,000-update RL run. Protocol compliance converges within the first fifty updates: invalid trajectories drop below 1% and stay there. After that, the reward curve is driven by improving repair quality: successful repairs per edit rise steadily from 11% to 38%, while the damage rate on initially correct images falls from 20% to 10%. Terminal exactness under the training verifier reaches 81%. There is no reward collapse or protocol regression within 1,000 updates. On the full GenEval test set (553 prompts), SFT’s three reflection rounds add +2 points of macro accuracy; RL raises this to +11, with the gain concentrated in the repair process rather than the initial generation (Section 4.3). 6.2 WhatRLchangesinthemodel Replaying all 553 trajectories through the Base, SFT, and RL checkpoints (Appendix F) gives one consistent picture. The backbone’s own readout of correctness is nearly unchanged by RL: a held-out linear probe on the understanding stream reaches AUC 0.80–0.82 under all three checkpoints. What changes is where the images go. Among initially failing images, the share inside the dense pass region of that readout rises from 34% to 62% over three rounds under RL, against 36% to 42% under SFT (Figure 5), and at a matched number of edits RL moves the image further and more of its edits succeed. RL thus selects, among the revisions the model can already produce, those that reach the correct region. 7 Conclusion Reinforcement learning turns reflection in a unified model from an SFT cold start into an effective repair mechanism. SFT and RL start from nearly identical single-shot accuracy, so the +12-point GenEval gain 9 and its transfer to WISE, OneIG, and CompBench come from the reflection rounds. RL leaves the model’s correctness readout nearly unchanged and selects, among the repairs the model can already produce, those that land: the backbone already knows whether its image is correct, and RL teaches it to act on that knowledge. References [1] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.Training Diffu- sion Models with Reinforcement Learning. In International Conference on Learning Representations, volume 2024, pages 4965–4987, 2024. URLhttps://proceedings.iclr.c/paper_files/paper/2024/hash/ 14f75513f0f1ca01de1e826b52e6b840-Abstract-Conference.html. [2]Chameleon Team. Chameleon: Mixed-Modal Early-Fusion Foundation Models, 2024. URLhttps://arxiv.org/ abs/2405.09818. [3] Jingjing Chang, Yixiao Fang, Peng Xing, Shuhan Wu, Wei Cheng, Rui Wang, Xianfang Zeng, Gang Yu, and Hai-Bao Chen. OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation. In Advances in Neural Information Processing Systems, volume 38, Main Conference, 2025. doi: 10.52202/085713-5330. URLhttps://proceedings.neurips.c/paper_files/paper/2025/hash/ e9e9e5428189a3e49479547ef917e88d-Abstract-Datasets_and_Benchmarks_Track.html. [4] Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset, 2025. URL https://arxiv.org/abs/2505.09568. [5] Leon Liangyu Chen, Haoyu Ma, Zhipeng Fan, Ziqi Huang, Animesh Sinha, Xiaoliang Dai, Jialiang Wang, Zecheng He, Jianwei Yang, Chunyuan Li, Junzhe Sun, Chu Wang, Serena Yeung, and Felix Juefei-Xu. UniT: Unified Multimodal Chain-of-Thought Test-time Scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. URL https://cvpr.thecvf.com/virtual/2026/poster/36853. [6]Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling, 2025. URL https://arxiv.org/abs/2501.17811. [7] Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu. Thinking with Generated Images, 2025. URL https://arxiv.org/abs/2505.22525. [8] Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging Properties in Unified Multimodal Pretraining, 2025. URL https://arxiv.org/abs/2505.14683. [9] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: Reinforcement Learning for Fine-tuning Text- to-Image Diffusion Models. In Advances in Neural Information Processing Systems, volume 36, pages 79858– 79885, 2023. doi: 10.52202/075280-3497. URLhttps://proceedings.neurips.c/paper_files/paper/2023/hash/ fc65fab891d83433bd3c8d966edde311-Abstract-Conference.html. [10] Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Hao Tian, Shilin Yan, Weihao Yu, Xingyu Zeng, Jifeng Dai, Xihui Liu, and Hongsheng Li. GoT: Unleashing Reasoning Capability of MLLM for Visual Generation and Editing. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 67680–67708, 2025. doi: 10.52202/085713-2270. URLhttps://proceedings.neurips.c/paper_files/paper/ 2025/hash/61960fdfda4d4e95fa1c1f6e64bfe8bc-Abstract-Conference.html. [11]Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. GenEval: An object-focused framework for evaluating text-to-image alignment. In Advances in Neural Information Processing Systems, volume 36, pages 52132– 52152, 2023. doi: 10.52202/075280-2270. URLhttps://proceedings.neurips.c/paper_files/paper/2023/hash/ a3bf71c7c63f0c3bcb7f67c67b1e7b1-Abstract-Datasets_and_Benchmarks.html. [12]Jiawei Gu, Yunzhuo Hao, Huichen Wang, Linjie Li, Michael Qizhe Shieh, Yejin Choi, Ranjay Krishna, and Yu Cheng. ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning. In International Conference on Learning Representations, volume 2026, pages 141405–141447, 2026. URLhttps://proceedings.iclr. c/paper_files/paper/2026/hash/e5095602ad1c6a835e2b643ec4ed97d0-Abstract-Conference.html. 10 [13]Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638, 2025. doi: 10.1038/s41586-025-09422-z. URLhttps://doi.org/10.1038/ s41586-025-09422-z. [14]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Yu, Xinying Song, and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. In International Conference on Learning Representations, volume 2024, pages 32808–32824, 2024. URLhttps://proceedings.iclr.c/paper_files/paper/ 2024/hash/8b4add8b0a8749d80a34ca5d941c355-Abstract-Conference.html. [15]Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5):3563–3579, 2025. doi: 10.1109/tpami.2025.3531907. URL https://doi.org/10.1109/tpami.2025.3531907. [16]Wenxuan Huang, Shuang Chen, Zheyong Xie, Shaosheng Cao, Shixiang Tang, Yufan Shen, Qingyu Yin, Wenbo Hu, Xiaoman Wang, Yuntian Tang, Junbo Qiao, Hangyu Guo, Yao Hu, Zhenfei Yin, Philip Torr, Yu Cheng, Wanli Ouyang, and Shaohui Lin. Interleaving Reasoning for Better Text-to-Image Generation. In International Conference on Learning Representations, volume 2026, pages 106153–106182, 2026. URLhttps://proceedings.iclr. c/paper_files/paper/2026/hash/ad48f017e6c3d474caf511208e600459-Abstract-Conference.html. [17]Shantanu Jaiswal, Mihir Prabhudesai, Nikash Bhardwaj, Zheyang Qin, Amir Zadeh, Chuan Li, Katerina Fragki- adaki, and Deepak Pathak. Iterative Refinement Improves Compositional Image Generation, 2026. URL https://arxiv.org/abs/2601.15286. [18]Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 39856– 39890, 2025. doi: 10.52202/085713-1330. URLhttps://proceedings.neurips.c/paper_files/paper/2025/hash/ 38fc6254f73450813db3b3e04397a9fc-Abstract-Conference.html. [19]Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, JD Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei Zhang, Kay McKinney, Disha Shrivastava, Cosmin Padu- raru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Training Language Mod- els to Self-Correct via Reinforcement Learning. In International Conference on Learning Representa- tions, volume 2025, pages 54523–54549, 2025. URLhttps://proceedings.iclr.c/paper_files/paper/2025/hash/ 871ac99fdc5282d0301934d23945ebaa-Abstract-Conference.html. [20]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-GRPO: Training Flow Matching Models via Online RL. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 40783–40818, 2025. doi: 10.52202/085713-1362. URLhttps://proceedings.neurips.c/paper_files/paper/2025/hash/ 3a10c46572628d58cb44fb705f25cbbf-Abstract-Conference.html. [21] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Kather- ine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534– 46594, 2023. doi: 10.52202/075280-2019. URLhttps://proceedings.neurips.c/paper_files/paper/2023/hash/ 91edff07232fb1b55a505a9e9f6c0f3-Abstract-Conference.html. [22] Weijia Mao, Zhenheng Yang, and Mike Zheng Shou. UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning, 2025. URL https://arxiv.org/abs/2505.23380. [23]Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Fanqing Meng, Kun-Peng Ning, Bin Zhu, and Li Yuan. WISE: World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation. In Proceedings of the International Conference on Machine Learning, 2026. URL https://icml.c/virtual/2026/poster/62614. [24]Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between Modalities with MetaQueries, 2025. URL https://arxiv.org/abs/2504.06256. [25]Luozheng Qin, Jia Gong, Yuqing Sun, Tianjiao Li, Haoyu Pan, Mengping Yang, Xiaomeng Yang, Chao Qu, Zhiyu Tan, and Hao Li. Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision. In International 11 Conference on Learning Representations, volume 2026, pages 15999–16028, 2026. URLhttps://proceedings.iclr. c/paper_files/paper/2026/hash/1ae4999aefb509d75d8608e07280922c-Abstract-Conference.html. [26] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K. Du, Zehuan Yuan, and Xinglong Wu. TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2545–2555, June 2025. URLhttps://openaccess.thecvf.com/content/CVPR2025/html/Qu_TokenFlow_Unified_Image_ Tokenizer_for_Multimodal_Understanding_and_Generation_CVPR_2025_paper.html. [27]Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive Introspection: Teaching Language Model Agents How to Self-Improve. In Advances in Neural Information Processing Systems, volume 37, pages 55249–55285, 2024. doi: 10.52202/079017-1754. URLhttps://proceedings.neurips.c/paper_files/paper/2024/ hash/639d992f819c2b40387d4d5170b8ffd7-Abstract-Conference.html. [28]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. URL https://arxiv.org/abs/2402.03300. [29]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 8634–8652, 2023. doi: 10.52202/075280-0377. URLhttps://proceedings.neurips.c/paper_files/paper/ 2023/hash/1b44b878b782e6954cd888628510e90-Abstract-Conference.html. [30]Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8228– 8238, June 2024. URLhttps://openaccess.thecvf.com/content/CVPR2024/html/Wallace_Diffusion_Model_ Alignment_Using_Direct_Preference_Optimization_CVPR_2024_paper.html. [31]Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 21612–21622, October 2025. URLhttps://openaccess.thecvf.com/content/ICCV2025/html/Wang_ILLUME_Illuminating_Your_ LLMs_to_See_Draw_and_Self-Enhance_ICCV_2025_paper.html. [32]Xinlong Wang, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Zhen Li, Yuqi Wang, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Chunlei Men, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Zhongyuan Wang, and Tiejun Huang. Multimodal learning with next-token prediction for large multimodal models. Nature, 650 (8101):327–333, 2026. doi: 10.1038/s41586-025-10041-x. URL https://doi.org/10.1038/s41586-025-10041-x. [33]Yi Wang, Mushui Liu, Wanggui He, Longxiang Zhang, Ziwei Huang, Guanghao Zhang, Fangxun Shu, Zhong Tao, Dong She, Zhelun Yu, Haoyuan Li, Weilong Dai, Mingli Song, Jie Song, and Hao Jiang. MINT: Multi- modal Chain of Thought in Unified Generative Models for Enhanced Image Generation, 2025. URLhttps: //arxiv.org/abs/2503.01298v1. [34] Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing. In Advances in Neural Information Processing Systems, volume 37, pages 128374– 128395, 2024. doi: 10.52202/079017-4077. URLhttps://proceedings.neurips.c/paper_files/paper/2024/hash/ e7c786024ca718f2487712bfe9f51030-Abstract-Conference.html. [35]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837, 2022. doi: 10.52202/068431-1800. URLhttps://proceedings.neurips.c/paper_files/paper/2022/hash/ 9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. [36] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation.In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 12966–12977, June 2025. URL https://openaccess.thecvf.com/content/CVPR2025/html/Wu_Janus_Decoupling_Visual_Encoding_for_ Unified_Multimodal_Understanding_and_Generation_CVPR_2025_paper.html. 12 [37]Tsung-Han Wu, Long Lian, Joseph E. Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting LLM-controlled Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6327–6336, June 2024. URLhttps://openaccess.thecvf.com/content/CVPR2024/html/Wu_ Self-correcting_LLM-controlled_Diffusion_Models_CVPR_2024_paper.html. [38]Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One Single Transformer to Unify Multimodal Understanding and Generation. In International Conference on Learning Representations, volume 2025, pages 28240–28264, 2025. URLhttps://proceedings.iclr.c/paper_files/paper/2025/hash/ 45f0d179ef7e10eb7366550cd4e574ae-Abstract-Conference.html. [39]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems, volume 36, pages 15903–15935, 2023. doi: 10.52202/075280-0700. URLhttps://proceedings. neurips.c/paper_files/paper/2023/hash/33646ef0ed554145eab65f6250fab0c9-Abstract-Conference.html. [40]Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. DanceGRPO: Unleashing GRPO on Visual Generation, 2025. URL https://arxiv.org/abs/2505.07818. [41]Zhengyuan Yang, Jianfeng Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. Idea2Img: Iterative Self-refinement with GPT-4V for Automatic Image Design and Generation. In Computer Vision – ECCV 2024, pages 167–184, 2024. doi: 10.1007/978-3-031-72920-1_10. URLhttps://doi.org/10.1007/ 978-3-031-72920-1_10. [42]Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? In Advances in Neural Information Processing Systems, volume 38, 2025. URLhttps://proceedings.neurips.c/paper_files/ paper/2025/hash/537d5a768c2d534016a4d06f87bc8fb-Abstract-Conference.html. [43]Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Ziyu Guo, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Peng Gao, and Hongsheng Li. Let’s Verify and Reinforce Image Generation Step by Step. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28662–28672, June 2025. URLhttps://openaccess.thecvf.com/content/CVPR2025/html/Zhang_Lets_Verify_and_Reinforce_Image_ Generation_Step_by_Step_CVPR_2025_paper.html. [44]Yu Zhang, Yunqi Li, Yifan Yang, Rui Wang, Yuqing Yang, Dai Qi, Jianmin Bao, Dongdong Chen, Chong Luo, and Lili Qiu. ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL, 2025. URL https://arxiv.org/abs/2505.24875. [45]Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model. In International Conference on Learning Representa- tions, volume 2025, pages 6446–6469, 2025. URLhttps://proceedings.iclr.c/paper_files/paper/2025/hash/ 12678c3948153f4bc391f51e2082bd6e-Abstract-Conference.html. [46] Le Zhuo, Liangbing Zhao, Sayak Paul, Yue Liao, Renrui Zhang, Yi Xin, Peng Gao, Mo- hamed Elhoseiny, and Hongsheng Li.From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuning.In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15329–15339, October 2025. URLhttps://openaccess.thecvf.com/content/ICCV2025/html/Zhuo_From_Reflection_to_Perfection_ Scaling_Inference-Time_Optimization_for_Text-to-Image_Diffusion_ICCV_2025_paper.html. 13 Appendix A ImplementationandEvaluationDetails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.1 Direct-generation RL control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A.2 Understanding-based selection protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B ReflectionFormatandSFTDetails. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 C GradedReward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D TrainingDynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 E Test-TimeScalingonExternalBenchmarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 F DistributionalAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 G VisualPathwayStability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 H Cross-RoundAttentiontoEarlierImages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 I CorrectRepairsAlreadyintheSFTRolloutDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 J SwappingtheEditingInstructiononFixedImages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 K PairedFinal-ModelResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 L Category-LevelResults. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 M DetailedExternalBenchmarkComparisons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 N ExternalCriticBaseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 O AdditionalQualitativeExamples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 P StoppingBehaviorunderTerminalRewards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Q Per-FamilyLearningonGenEval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 R InferenceSystemPrompt. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 14 A ImplementationandEvaluationDetails Training. The primary experiments use the BAGEL reflection-SFT checkpoint (§3.3) as the RL parent. RL samples from a 3,000-prompt pool spanning six GenEval families; a separate 270-prompt held-out set is UID-disjoint from training. Each update draws two independent roots withK= 16 siblings per root. Training uses 20 denoising steps and a two-transition flow window; the final checkpoint is at 1,000 updates. RL runs on two nodes with eight NVIDIA H100 80GB GPUs each (16 GPUs, hybrid-sharded FSDP); the 1,000 updates take about 33 hours of update time (median 118 s per update). Evaluation. All arms are evaluated with 50 denoising steps, 512×512 images, and at most three native repairs. BAGEL generates natively at 1024×1024; we set the generation resolution to 512×512 for training and for every evaluated arm, including Base. Because RL renders complete trajectories (up to four images per rollout, 32 rollouts per update), the lower resolution keeps whole-trajectory sampling tractable. The same 1,000-update checkpoint is shared across all benchmarks rather than selecting a different checkpoint per test. We evaluate one returned image per prompt per arm. Benchmarks. GenEval (553 prompts): unweighted macro accuracy over six compositional families. WISE (1,000 prompts): weighted aggregate with a GPT-4o judge. OneIG-Bench (695 alignment prompts out of 1,120): question-dependent alignment, not an overall omni-dimensional score. T2I-CompBench++ (2,400 prompts, 300 per category): eight-category mean with category-specific scorers. Cross-benchmark numbers use the 0–100 scale; native 0–1 tables are provided for GenEval. Our controlled evaluations use instruction-voice prompts and differ from native leaderboard protocols; we separate controlled comparisons from published reference scores. Table4 Primary configuration and evaluation coverage. SettingValue BackboneBAGEL, 28-layer mixture-of-transformers (MoT) InitializationReflection SFT (one epoch) Reported RL checkpoint1,000 committed updates, direct full weights Direct T2I-RL controlOfficial Base, 1,000 RL updates, no repairs Train/dev prompt pools3,000 / 270 Roots per update / siblings per root2 / 16 Training / inference denoising steps20 / 50 Selected flow window2 contiguous transitions Primary inference repair cap3 Training / evaluation resolution512× 512 (BAGEL native: 1024× 1024) RL hardware2 nodes × 8 NVIDIA H100 80GB Evaluated outputs per prompt and arm 1 GenEval / WISE prompt counts553 / 1,000 OneIG generated / alignment-scored1,120 / 695 CompBench categories / prompts8 / 2,400 SFT effective-record target / exposures 167,363 / 167,368 SFT continuation learning rate2× 10 −7 , constant SFT CE / flow-MSE coefficients1 / 1 RL text / flow learning rate5× 10 −6 / 5× 10 −6 RL text / flow ratio clip ε0.2 / 0.1 RL text / flow KL coefficient η10 −4 / 10 −4 (frozen SFT reference) KL estimator (text / flow)per-token k 3 / closed-form Gaussian SDE transition RL optimizer / weight decayAdamW (0.9, 0.999) / 10 −4 RL gradient-norm clip1.0 per channel Flow SDE noise level / text temperature 1.0 / 0.9 Supervised parent provenance. The retained SFT parent completes an effective sample-exposure target of 167,363 records, reaching 167,368 at the final batch. Global updates 1,410–2,969 run at a constant learning 15 rate of 2×10 −7 . The mixture weights for controller, image-transition, verifier-state, penultimate verifier-state, and base-generation anchor groups are 4 : 4 : 4 : 1 : 1. Figure 6 plots this recorded continuation phase. 1500200025002969 Global SFT update 0.9 1.0 1.1 1.2 1.3 Cross-entropy a Text supervision 1500200025002969 Global SFT update 0.2 0.3 0.4 0.5 Flow MSE b Image supervision Logged updatesTrailing 50-update mean Figure6 Thesupervisedparent,recordedcontinuationphase. Text cross-entropy and image flow-MSE from global SFT updates 1410–2969. Faint lines show logged update values and solid lines show trailing 50-update means. Frozen and trainable modules. The RL implementation first freezes the model, then opens both expert branches in all 28 MoT decoder layers and the flow rootstime_embedder,vae2llm,llm2vae, andlatent_- pos_embed. The remaining modules, including the vocabulary head and the final normalization outside the decoder layers, remain frozen. The two channel optimizers operate on explicit, disjoint parameter groups. The detached initial rendering is not assigned an RL loss, although it can change across updates because its generation parameters are also used for repair. Image-only versus protocol-gated accuracy. The image score is computed from the retained per-image verifier verdict for the final returned image. We do not multiply it by a parse-validity indicator. A malformed reflection can therefore leave a correct image with a valid image score, while separately lowering protocol validity. This separation is applied to SFT and RL alike. GenEval macro averages weight the six families equally; prompt-micro averages instead weight all 553 prompts equally. Their distinct labels are preserved throughout. Native and external-agent controls. The untuned external agent reviews the original request and current image and always executes three corrections with the native BAGEL ODE sampler. Its reviews are greedy and capped at 512 tokens. SFT andUMM-Reflectionuse the same native reflection prompt, temperature 0.5 for controller sampling, and a maximum of 512 controller tokens per turn. They stop on native done, invalid termination, or the repair cap. Benchmark-specific interpretation. WISE reports its weighted group aggregate, not a pooled accuracy. OneIG reports question-dependency alignment on its 695 eligible prompts, using the official Qwen2.5-VL-7B question- answering scorer. CompBench uses one image per prompt and the prescribed category scoring formulas. Its 3D-spatial category concerns relationships depicted in 2D images, not generated 3D representations. Published native benchmark scores are contextual references, not substitutes for matched re-evaluation. Test-time scaling evaluation. Figures 4 and 9 use the retained images at each reflection round; trajectories that stop early keep their last image. WISE curves use a common GPT-4o re-evaluation of Base, SFT, and the 1000-step model. The main table reports the original WISE evaluation. RL prompt pool. The 3,000 RL prompts are drawn from the GenEval-style training prompts released with Flow-GRPO [20], restricted to the six GenEval families and deduplicated so that no prompt repeats during training. Family proportions follow Flow-GRPO’s own mixture, with a fixed quota for single object (position 1,305, counting 702, color attribution 559, colors 187, two objects 187, single object 60), and every prompt is 16 rendered in the same instruction voice used at evaluation (“Create an image with . . . ”). No RL prompt reuses an official GenEval evaluation prompt. The 270-prompt development set is disjoint from the training pool. The pool is released with the code. Representation probes. Correctness probes (Appendix F) are logistic regressions (standardization, PCA to 256 dimensions fitted on the training folds,C= 0.05) on backbone features at depths 0,4,...,28. They are evaluated by five-fold cross-validation grouped by prompt, so images of the same trajectory never appear in both training and test folds. AUC intervals use 1,000 family-stratified prompt-bootstrap replicates. A.1 Direct-generationRLcontrol The direct T2I-RL control initializes from official BAGEL Base and optimizes direct T2I generation with Flow-GRPO, without reflection SFT or a controller objective. It uses the same six-family training pool as UMM-Reflection,K= 16, 20 training denoising steps, two selected flow transitions, and the original Base as the KL reference. The reported checkpoint is at 1,000 updates. Evaluation loads the direct T2I-RL language-model weights over Base auxiliary modules and reuses the retained Base initial-generation path at 512×512 with 50 denoising steps. There is no self-CoT, verifier query, candidate selection, or repair at inference. GenEval preserves the retained two-prompt seed batches; external benchmarks preserve one-prompt calls and original indices/seeds. Coverage is complete for all 553 GenEval prompts, 1,000 WISE prompts, 695 eligible OneIG alignment prompts out of 1,120 generated prompts, and 2,400 CompBench prompts. 405060708090 Benchmark score (0–100 scale) GenEval WISE OneIG CompBench Direct T2I-RLUMM-ReflectionGain 75.53 83.81+8.28 54.07 73.84+19.77 79.86 82.86+2.99 49.35 55.06+5.71 Direct T2I-RL (1,000 updates)UMM-Reflection (1,000 updates) Figure7 Completeddirect-T2IRLcontrol. Hollow orange points denote direct T2I-RL at 1,000 updates; green points denoteUMM-Reflectionat 1,000 updates. Gain is the within-benchmark score difference. Each benchmark keeps its own metric on a 0–100 scale; scores are not averaged across benchmarks. The systems differ in initialization and training/inference compute (Appendix A). A.2 Understanding-basedselectionprotocol The two Best-of-4 arms draw four images per prompt from the direct T2I-RL renderer over the full 553- prompt GenEval set, at 512×512 and 50 denoising steps. The first draw is the retained T2I-RL evaluation image; the other three reuse its seed offset by multiples of 10 6 . The candidate order is shuffled with seed 20260909 +prompt indexand is identical for both selectors. Each makes one greedy multi-image call with think=Falseand a 256-token output limit. Base UND uses original Base;UMM-ReflectionUND uses the UMM-Reflectioncheckpoint. “UND” identifies the understanding inference path, not a separately trained classifier head. The fixed instruction asks the model to prioritize requested objects, counts, attributes and spatial relations over aesthetics, select the closest visible match, and outputBEST: A/B/C/Dfollowed by a short explanation. 17 The complete selection prompt will be released with the code. Neither model receives the task family, detector metadata, candidate scores, or selected-image verdict before choosing. Each arm has one invalid-format response out of 553; the preregistered fallback selects the first presented image, with no discarded prompts. Image scoring follows the saved choices. Base UND and UMM-Reflection UND select 437 and 440 correct images respectively (prompt-micro counts, distinct from the macro scores). Their A/B/C/D choice counts are 225/26/240/62 and 245/32/219/57. B ReflectionFormatandSFTDetails Each reflection is exposed through six tagged fields:[CURRENT_ROUND],[SOURCE_IMAGE],[SCORE], [THINKING],[ACTION], and[EDIT], making the model’s reasoning inspectable. The[SCORE]is a model- generated self-assessment, not an external reward: neither verifier scores nor verifier labels enter the policy observation at training or inference. SFT teaches the interleaved protocol using the trajectories from §3.2. Each trajectory is decomposed into 167,363 training rows spanning three complementary views: controller rows supervise the full reflection text (autoregressive cross-entropy, no image loss), transition rows supervise each individual edit step (the edited image receives flow-matching loss conditioned on the edit instruction), and verifier rows present a partial trajectory and supervise only the next reflection. In notation, L SFT =L AR + λ img L FM . Both understanding and generation expert branches across all 28 decoder layers are trainable; the visual encoders (ViT, VAE), text embeddings, and vocabulary projection are frozen. Training starts from the base BAGEL checkpoint and runs for one full epoch. C GradedReward Standard GenEval scoring is binary: an image either satisfies all constraints or it does not. Under binary scoring, five of the six GenEval families produce ∆ t = 0 whenever an edit improves some constraints but not all, becauseqjumps only at the boundary between full failure and full success. This renders all progress terms in Eq. 1 ineffective for the majority of training prompts. We therefore grade only the failure region, using nothing but what the frozen verifier returns at its official thresholds (0.3 for object detection, 0.9 for counting); no sub-threshold confidence is read. For every family, q = ( 1official verdict passes, min 1 2 f, 0.5 − otherwise, wheref ∈[0,1] is a family-specific satisfaction fraction and 0.5 − is the largest double below 0.5, so partial credit never reaches the exact band. Letπ(o)∈0,1indicate that the verifier keeps a detection of classo; missing detections and boxes contribute zero. • single object: f = π(o). • two objects: f = 1 2 π(o 1 ) + π(o 2 ) . • colors:f=π(o) 1 2 + 1 2 c , wherecis the CLIP confidence of the predicted color if it is the requested one and 0 otherwise. • color attribution: f = 1 2 (r 1 + r 2 ), with r k = 1 if attribute k passes and otherwise r k = π(o k ) 1 2 + 1 2 c k , c k = 1 2 c CLIP k +clip(1 +b k ,0,1) ;b k is the BLIP margin (yes-probability of the requested phrase minus that of the color-swapped phrase), whose official cut is 0. • position:f= 0.6· 1 2 (π s +π o ) + 0.4min(π s ,π o )m, withm=clip(d/0.5,0,1), wheredis the component of the official threshold-shrunk, normalized center offset along the requested relation (the official pass cut is 0.5; m = 0 if either box is missing). 18 •counting:q= 1 ifˆn=nandq= 1 2 max 0,1−|ˆn−n|/n otherwise, whereˆnis the number of detections at the counting threshold. Fail-closed rule. Every scored image is checked against its official verdict: a passing image must receive q= 1 and a failing imageq <0.5. A violation raises an error in the scoring service and the request returns no reward, so grading can never change a pass/fail decision, and accuracy computed fromqequals the binary GenEval score. On 200 image–family pairs scored with the frozen backends, no violation occurred and the share of images with a nonzero score rose from 21% to 100%. Malformed trajectories receive a bounded negative correctionA i ← max min(A i ,0)−0.5, −1 , preserving a learning signal for protocol violations. D TrainingDynamics 02004006008001000 RL updates 0.0 0.4 0.8 1.2 Mean reward a Trajectory reward 02004006008001000 RL updates 0 25 50 75 100 Successful edits (%) b Repair success 02004006008001000 RL updates 0 25 50 75 100 Exact trajectories (%) c Terminal exactness Individual updates20-update trend Figure8 Learningdynamicsofthe1,000-updateRLrun. (a) Mean trajectory reward. (b) Successful repairs per edit from an incorrect state. (c) Fraction of trajectories reaching terminal exactness under the training verifier. Faint lines are individual updates; solid lines are trailing 20-update trends, with rates pooling numerators and denominators over the window. E Test-TimeScalingonExternalBenchmarks 0123 Reflection rounds 70 75 80 85 Macro accuracy (%) 83.8 a GenEval 0123 Reflection rounds 55 60 65 70 75 80 Weighted score 75.2 b WISE 0123 Reflection rounds 80 81 82 83 84 Alignment 82.9 c OneIG 0123 Reflection rounds 48 50 52 54 56 Mean score 55.1 d CompBench BaseReflection SFTUMM-Reflection-500UMM-Reflection-1000 Figure9 Multi-roundtest-timescalingonfourbenchmarks. Scores (0–100) versus reflection rounds. Blue dotted lines denote Base, pink dashed lines reflection SFT, purple dash-dotted linesUMM-Reflection-500, and green solid lines UMM-Reflection-1000. 19 F DistributionalAnalysis The findings below support one reading: RL does not give the model a new way to see or judge its images; among the revisions the backbone can already produce, it learns to select the ones that land, a pattern also reported for RL with verifiable rewards in language models [42]. We replayed all 553 archived RL trajectories, together with the matching SFT trajectories, through the Base, SFT, and RL checkpoints and read out both the generation branch (VAE latents) and the understanding branch (ViT tokens) at every decoder depth (protocol in Appendix A). Two findings organize what changed. Table5 Distributionalstatistics. One trajectory per prompt. Edit-matched rows use prompts where SFT made three edits (RL always makes three) or at least one edit. Intervals are 95% family-stratified prompt bootstrap intervals, except the pass-region change, which resamples images (5,000 replicates). Pass-region rows use initially failing images. MeasureUnit (N)SFTRLRL − SFT [95% CI] DINO distance, first to last imageprompts (553)0.090.29– same, three editsprompts (153)0.110.280.17 [0.13, 0.21] Exact gain R0→R3 (points), three edits prompts (153)−3.3+16.3+19.6 [11.1, 28.1] Exact gain R0→R3 (points), ≥1 editprompts (456)+0.9+10.3– In dense pass region, R0→R3 (%)images (170 / 154) 36.5→42.4 34.4→61.7+21.4 [9.4, 33.3] UND probe AUC, depth 20 (Base 0.804) held-out images0.8070.815– Stream coupling, GEN / UND (null 0.04) image pairs0.56 / 0.45 0.57 / 0.46 Holm p = 0.012 vs. null RL edits move the image further, at matched edit count. Over all 553 prompts, the DINO distance from the first to the last image is 0.09 for SFT and 0.29 for RL. This is not because RL edits more often. On the 153 prompts where both policies used exactly three edits, the distance is 0.11 versus 0.28 (paired difference 0.17, 95% CI [0.13, 0.21]), and on the same prompts the exact-match gain from R0 to R3 is−3.3 points for SFT and +16.3 for RL (paired difference +19.6 [11.1, 28.1]). On the 456 prompts where SFT made at least one edit, the gains are +0.9 and +10.3. At equal edit budget, RL makes larger changes and they land. RL moves failing images into the passing region of a readout the backbone already has. A linear probe for “passes the verifier” on the understanding stream peaks at decoder depth 20 with held-out AUC 0.804 under Base, 0.807 under SFT, and 0.815 under RL, whereas frozen DINO and CLIP features reach only 0.55. The backbone can already tell whether its image is right, and RL barely changes that readout. What RL changes is where the images go (Figure 5): among initially failing images, the share inside the dense pass region rises from 34% at R0 to 62% at R3 under RL, versus 36% to 42% under SFT, and the probe distance of failing RL images to the passing side falls monotonically over the rounds under all three checkpoints. Read against SFT, the figure shows where the gain comes from. SFT’s revisions spread broadly over the readout plane, and only a small part of that distribution reaches the dense pass region. RL does not open a new region: its R3 images concentrate inside the same pass region that part of SFT’s distribution already reaches. RL extracts the correct slice of the broad distribution that SFT learned and shifts the policy’s trajectories toward it; the rollout statistics in Appendix I show the same pattern directly. The same holds for the coupling between the two streams: changing the image changes the representation of a fixed request, and the normalized alignment between the two displacements is 0.58/0.45 (GEN/UND) for Base, 0.56/0.45 for SFT, and 0.57/0.46 for RL, each far above the permutation null of 0.04 (Holmp= 0.012). RL does not build a new channel between seeing and describing; it steers images through one the unified backbone already has, which is why 1,000 updates on a 3,000-prompt pool suffice. G VisualPathwayStability Consistent with the distributional findings in Appendix F, the visual pathway itself is largely unchanged by RL. On ten replayed trajectories (30 paired rounds), SFT and RL attention maps over VAE keys correlate at a median of 0.98; over ViT keys the correlation exceeds 0.99 (Figure 10). The cross-modal coupling that connects image states to text representations is already present in the Base model, and RL preserves it. We 20 Input THINKING / VAEEDIT / VAE RL next image SFTRLSFTRL (a) a white sandwich | R0 to R1 | next: pass (b) a bird below a skateboard | R0 to R1 | next: fail (c) a purple wine glass and a black apple | R1 to R2 | next: fail Figure10 AttentionoverVAEimagetokensispreservedafterRL. Three cases show paired SFT/RL attention ofTHINKING andEDITtokens onto 32×32 VAE keys, averaged over 28 layers. Maps use a shared 99th-percentile cap; cyan boxes are pre-annotated error regions. observe only limited changes in the visual pathway, which is consistent with the learned change residing mainly in what the model writes from the same visual input. 21 H Cross-RoundAttentiontoEarlierImages Each reflection round keeps all earlier images in context. To test whether later rounds use them, we measure, on the ten replayed trajectories of Appendix G, how the attention thatTHINKINGandEDITtokens assign to image keys (VAE and ViT) is split across the images in context. Table 6 reports the third reflection round, which sees the initial imagex 0 and two revisions. About 45–48% of the image attention goes to the current image, butx 0 and the first revision each keep 22–31%, and every earlier image receives at least 19% in every case. Restricted to VAE keys, the three images are attended almost equally (29–37%). SFT and RL split attention the same way: the model reads its whole visual history, and RL does not change this. Table6 Shareofimageattentionperimageatthethirdreflectionround. Mean over ten trajectories (minimum in parentheses); shares sum to 1 across the three images. Image attention is 10–13% of all attention. PolicyTokensx 0 Revision 1 Current image SFTTHINKING 0.23 (0.20) 0.30 (0.26)0.48 (0.41) SFTEDIT0.23 (0.19) 0.31 (0.27)0.46 (0.41) RLTHINKING 0.26 (0.23) 0.29 (0.25)0.45 (0.39) RLEDIT0.23 (0.19) 0.31 (0.27)0.46 (0.40) RL, VAE keys only THINKING 0.32 (0.27) 0.34 (0.28)0.34 (0.26) RL, VAE keys only EDIT0.32 (0.27) 0.35 (0.29)0.33 (0.26) I CorrectRepairsAlreadyintheSFTRolloutDistribution Each RL update samplesK= 16 complete trajectories from one initial image, and the training log records whether each trajectory ends verifier-correct. Table 7 uses the roots whose initial image is incorrect and reports the per-trajectory success rate (pass@1) and the share of roots with at least one correct trajectory among the 16 (pass@16), by training window. At the start of RL, where the policy is essentially the SFT model, a correct repair already exists among the 16 rollouts for 78% of roots, although a single rollout succeeds only 23% of the time. Over training, pass@1 rises to 70% while pass@16 rises only to 97%: RL concentrates the policy on repairs that the SFT distribution already contains. Table7 SiblingsuccessonincorrectinitialimagesduringRL. Roots with an incorrect initial image; 16 sibling trajectories per root. RL updates Roots pass@1 (%) pass@16 (%) 1–506722.777.6 51–1005431.477.8 101–20011431.583.3 201–50036760.594.6 501–1,00061170.197.1 J SwappingtheEditingInstructiononFixedImages To test which channel carries the repair, we fix the image, the RL renderer, and the sampling noise, and change only the editing instruction for one image update. On 154 initially failing states with two paired noise seeds each, the original request repairs 20.5%, the SFT policy’s instruction 21.4%, and the RL policy’s instruction 48.4%; a rule-written instruction that spells out every GenEval constraint reaches 27.6%. The paired RL-minus-request difference is +27.9 points (95% state bootstrap interval [20.8,34.7]), and RL-minus- rule is +20.8 [13.6,27.9], concentrated in counting, position, and two-object prompts. With the renderer and noise held fixed, the instruction the RL policy writes is what turns a failing image into a correct one, and it is more executable than an exhaustive rule-based specification. 22 K PairedFinal-ModelResults Table8 Stage-wiseGenEvalanalysis. Initial and final are prompt-micro image accuracy over 553 prompts, not six-family macro scores. Repair is conditional on an initially incorrect image; damage is conditional on an initially correct image. Protocol validity is measured separately. All entries are percentages. ModelInitial Final Repair Damage Protocol Reflection SFT69.26 71.07 20.596.5395.48 UMM-Reflection (1,000 updates) 72.15 83.91 64.948.77100.00 Table9 Prompt-pairedUMM-Reflection-1000 versus reflection SFT on GenEval. Wins and losses count discordant image-only verdicts. The exact two-sided binomial test on discordant pairs is the exact McNemar test. Comparison with SFT Wins Losses Net Exact p Initial4832 +160.093 Final10938 +71 < 0.001 All tables in this section compare the retained SFT evaluation with the final full-weight RL model on the same 553 prompts. Earlier decoder-only checkpoint evaluations are not substituted for this final-model evaluation. L Category-LevelResults Table10 All six GenEval families, using image-only verdicts. Values are percentages; the macro mean appears in Table 1. Familyn SFT final RL initial RL final Single object80100.00100.0095.00 Two objects9987.8883.8495.96 Counting8057.5067.5067.50 Colors9487.2384.0490.43 Position10047.0055.0089.00 Color binding 10051.0048.0065.00 23 010203040 UMM-Reflection − reflection SFT (points) Single object Two objects Counting Colors Position Color binding a GenEval -5.0 +8.1 +10.0 +3.2 +42.0 +14.0 051015 UMM-Reflection − Base (points) Color Shape Texture 2D spatial 3D spatial Numeracy Non-spatial Complex b CompBench +10.2 +8.7 +9.6 +14.7 -1.3 +3.6 +0.3 +2.3 Figure11 Category-level changes, with every category retained. Positive changes are green and negative changes are red. This view complements the overall gains without assuming uniform improvement. M DetailedExternalBenchmarkComparisons These detailed tables use native 0–1 scores. GenEval and original WISE sources generally report two decimal places, OneIG three, and CompBench four; we preserve these precisions and print local measurements to four places. CompBench reference means are the eight-category mean of the published category scores. The overview, ablation, stage statistics, and plots retain their stated 0–100 scales. GenEval row citations distinguish model-author results, the original benchmark-author results, and secondary reports marked†. Resolution notes for older reference models additionally follow Qu et al.[26], Table 4; undocumented settings are marked n/r, rather than assumed to be 512. Other native-resolution reference models are not protocol-matched controls. Each reference row reports the source table of the cited paper. 24 Table11 WISE:sixworld-knowledgecategories. Native 0–1 results from the official original-WISE legacy leaderboard [23], retaining its two-decimal reporting precision. The local block uses 1,000 original prompts and the same frozen judge setup. Overall is the original weighted aggregate. These are not WISE_Verified scores: that revision changes prompts, judge, and scoring. Bold denotes the best completed local result. ModelCultural Time Space Biology Physics Chem. Overall Official original-WISE legacy leaderboard: Niu et al. [23] SD1.50.34 0.35 0.320.280.29 0.210.32 SDXL (base 0.9)0.43 0.48 0.470.440.45 0.270.43 SD3.5-Large0.44 0.50 0.580.440.52 0.310.46 PixArt-α0.45 0.50 0.480.490.56 0.340.47 Playground v2.50.49 0.58 0.550.430.48 0.330.49 FLUX.1-dev0.48 0.58 0.620.420.51 0.350.50 Janus-1.3B0.16 0.26 0.350.280.30 0.140.23 VILA-U-7B-2560.26 0.33 0.370.350.39 0.230.31 Show-o-5120.28 0.40 0.480.300.46 0.300.35 Janus-Pro-7B0.30 0.37 0.490.360.42 0.260.35 Emu30.34 0.45 0.480.410.45 0.270.39 MetaQuery-XL0.56 0.55 0.620.490.63 0.410.55 Local evaluation: same prompts, resolution, and scorer BAGEL-Base0.4841 0.5437 0.6774 0.5005 0.6445 0.4590 0.5515 BAGEL-SFT0.5838 0.6254 0.7165 0.6120 0.6965 0.5380 0.6287 BAGEL-Self-Agentic 0.5921 0.5967 0.7410 0.5550 0.7010 0.4915 0.6129 BAGEL-T2I-RL (1k) 0.4829 0.5284 0.6711 0.4620 0.6595 0.4405 0.5407 UMM-Reflection0.68350.70540.82260.79250.79850.62800.7384 Table12 T2I-CompBench++:alleightcompositioncategories. Native 0–1 scores retain four decimal places. The public block transcribes the non-MLLM columns of Huang et al.[15], Table XIII: BLIP-VQA for attributes, UniDet for 2D/3D spatial relations and numeracy, CLIP for non-spatial relations, and 3-in-1 for complex composition. Published evaluation uses ten images per prompt; our local block uses one image for each of 2,400 prompts. The eight-category mean is our local aggregate and is not supplied for published rows. Bold denotes the best completed local result. ModelColor Shape Texture2D3D Number Non-sp. Complex Mean Published ten-image protocol: Huang et al. [15], Table XIII SD1.40.3765 0.3576 0.4156 0.1246 0.3030 0.4456 0.3079 0.3080 0.3298 SD20.5065 0.4221 0.4922 0.1342 0.3230 0.4582 0.3127 0.3386 0.3734 Composable + SD20.4063 0.3299 0.3645 0.0800 0.2847 0.4272 0.2980 0.2898 0.3100 Structured + SD20.4990 0.4218 0.4900 0.1386 0.3224 0.4557 0.3111 0.3355 0.3718 Attend-and-Excite + SD2 0.6400 0.4517 0.5963 0.1455 0.3222 0.4773 0.3109 0.3401 0.4105 GORS-unbiased + SD2 0.6414 0.4546 0.6025 0.1725 0.3300 0.4849 0.3158 0.3470 0.4186 GORS + SD20.6603 0.4785 0.6287 0.1815 0.3572 0.4830 0.3193 0.3328 0.4302 SDXL0.5879 0.4687 0.5299 0.2133 0.3566 0.4991 0.3119 0.3237 0.4114 PixArt-α-ft0.6690 0.4927 0.6477 0.2064 0.3901 0.5032 0.3197 0.3433 0.4465 DALL·E 30.7785 0.6205 0.7036 0.2865 0.3744 0.5926 0.3003 0.3773 0.5042 SD30.8132 0.5885 0.7334 0.3200 0.4084 0.6174 0.3140 0.3771 0.5215 FLUX.10.7407 0.5718 0.6922 0.2863 0.3866 0.6185 0.3127 0.3703 0.4974 Local evaluation: same prompts, resolution, and scorer BAGEL-Base0.7437 0.5216 0.6658 0.3015 0.3913 0.6067 0.3079 0.3847 0.4904 BAGEL-SFT0.7609 0.5725 0.7123 0.3014 0.3759 0.6182 0.3061 0.3870 0.5043 BAGEL-Self-Agentic0.8010 0.5741 0.7203 0.3268 0.4090 0.6300 0.3059 0.3905 0.5197 BAGEL-T2I-RL (1k)0.7402 0.5029 0.6684 0.3150 0.3931 0.6408 0.3126 0.3746 0.4935 UMM-Reflection0.84600.60830.76210.44890.37840.64280.31080.40740.5506 25 Table13 OneIG-Benchalignment. Native 0–1 scores. Reference rows are from Chang et al.[3], Table 2, at their reported precision and resolution (Table 8). Local rows measure question-dependent alignment on the 695 eligible prompts at 512 2 with the same scorer. ModelRes.Alignment Benchmark-author results: Chang et al. [3], Tables 2 and 8 Janus-Pro3840.553 BLIP3-o10240.711 Show-o2-1.5B4320.798 Show-o2-7B4320.817 OmniGen210240.804 SD1.55120.565 SDXL10240.688 SD3.5-Large10240.809 FLUX.1-dev10240.786 CogView410240.786 SANA-1.5 1.6B (PAG) 10240.762 SANA-1.5 4.8B (PAG) 10240.765 Lumina-Image 2.010240.819 HiDream-I1-Full10240.829 Local evaluation: same prompts, resolution, and scorer BAGEL-Base5120.8044 BAGEL-SFT5120.7937 BAGEL-Self-Agentic5120.8088 BAGEL-T2I-RL (1k)5120.7986 UMM-Reflection5120.8286 Table14 OneIGlocalalignmentbypromptcategory. Scores use the native 0–1 scale. Anime/stylization, general objects, and portrait contain 245, 206, and 244 eligible prompts, respectively. Alignment is the prompt-weighted aggregate, not the unweighted mean of these three columns. The Anime/style column measures alignment on that prompt category, not the separate style metric in Table 13. ModelAnime / style General objects Portrait Alignment Local evaluation: same prompts, resolution, and scorer BAGEL-Base0.83890.7569 0.80990.8044 BAGEL-SFT0.82730.7529 0.79450.7937 BAGEL-Self-Agentic0.84360.7732 0.80400.8088 BAGEL-T2I-RL (1k)0.83260.7408 0.81330.7986 UMM-Reflection0.86180.80200.81760.8286 26 N ExternalCriticBaseline We also compare with an external pipeline in which GPT-5.5 (gpt-5.5-2026-04-23, medium effort) inspects each BAGEL-Base image and issues either an edit instruction or done; Base executes up to three edits. The critic sees only the request and the current image, never a verifier verdict, and the protocol otherwise matches our evaluation (same prompts, seeds, resolution, and 50 denoising steps). Table 15 reports the result. On GenEval the pipeline reaches 79 and repairs 28% of the 163 initially incorrect images (45 of 163), against 84 and 65% forUMM-Reflection; GPT-5.5 accepts 67 of the 163 incorrect initial images without a single edit. UMM-Reflectionalso leads on WISE and CompBench and matches the pipeline on OneIG, with no external model at inference. Table15 ExternalGPT-5.5criticversusUMM-Reflection. Native 0–1 scale. WISE in this table is scored by a separate GPT-4o run for all three rows. OneIG to three decimals: 0.825 (critic) versus 0.829 (UMM-Reflection). Repair is the share of initially incorrect GenEval images that end correct. ModelGenEval WISE OneIG CompBench Repair (%) BAGEL-Base0.71 0.580.800.49– BAGEL + GPT-5.5 critic0.79 0.700.830.5227.6 UMM-Reflection0.840.750.830.5564.9 O AdditionalQualitativeExamples Spatial relationsGenEvalCreate an image with an elephant below a surfboard.Final Think (UMM) ...an elephant standing on a colorful surfboard underwater... Edit (UMM) Place an elephant on the ground below a surfboard Think (UMM) The current image shows two elephants, one standing on a surfboard... Edit (UMM) Place a surfboard above an elephant Think (UMM) ...the surfboard is not clearly positioned as being above the elephant... Edit (UMM) emphasizing the elephant's placement beneath it. World knowledgeWISECreate an image with National flag of the country where Rio de Janeiro is located.Final Think (UMM) ...a scenic view of a city with mountains, beaches, and palm trees... Edit (UMM) Add the Brazilian national flag in a prominent location Think (UMM) The image provided shows a cityscape that does not include any flags. Edit (UMM) The Brazilian flag should be clearly visible Think (UMM) ...shows a beach scene which does not meet the requirement. Edit (UMM) The flag should be clearly visible and centered Shape bindingCompBenchCreate an image with a circular rug and a triangular coffee table.Final Think (UMM) ...a circular rug but with a round coffee table... Edit (UMM) Replace the round coffee table with a triangular coffee table Think (UMM) The current image shows a round coffee table placed on a circular rug... Edit (UMM) Place a triangular coffee table on top of the circular rug Think (UMM) The image provided currently shows a round coffee table on a circular rug... Edit (UMM) The circular rug should remain intact and centered Figure12 ReflectiontrajectoriesofUMM-Reflection. Examples from GenEval, WISE, and CompBench show how step-by- step reflection and revision help the model produce images that match the prompt. Each row reads left to right: the image, then the unified model’s own reflection on it (Think: what is wrong; Edit: the instruction it issues), then the image it renders from that instruction. All text is verbatim model output. All images and text excerpts in this section are unedited model outputs. Figures 13–18 show further successful repairs on all four benchmarks, and Figure 19 shows two failures caused by the reflection itself. 27 GenEval · Counting “three kites.” Think...The current image shows four kites flying in the sky over a beach. To meet the user's request, the image should be edited... EditCreate an image with three kites. The image should depict three kites... Think...The current image shows multiple kites in the sky, which does not meet the requirement. To meet the user's request, the image... EditCreate an image with three kites. The image should depict three kites... Think...The current image shows two kites flying in the sky against a clear blue background. To meet the user's request, the image... EditCreate an image with three kites. The image should depict three kites... Benchmark check GenEval verifier R0fail R1fail R2fail R3pass Ends after the third edit. GenEval · Position “a clock below a tv.” Think...The current image shows a TV screen displaying a colorful abstract image with a clock in the center... EditPlace a TV set on a surface above a clock. The TV should be positioned... Think...The current image shows a wall-mounted TV above a wall clock. To meet the user's request, the image should be edited to show... EditPlace a TV set above a clock in the image. The TV should be positioned... Think...The current image shows a wall-mounted TV with a shadow cast on the wall, and a wall clock positioned below it... EditPlace a TV screen above a wall clock in the image... Benchmark check GenEval verifier R0fail R1pass R2pass R3pass Ends after the third edit. R0fail R1fail R2fail R3pass R0fail R1pass R2pass R3pass Figure13 Successfulrepairs. Each column reads top to bottom: the image, the model’s verbatim reflection, and the image it renders next. Frames mark the benchmark verdict. Cases are selected. 28 GenEval · Colors “a yellow broccoli.” ThinkThe initial image shows a head of broccoli with a green crown and yellow stalk, placed on a wooden surface... EditCreate an image of a broccoli that is predominantly yellow... Think...The image provided shows a broccoli that is green with hints of yellow. To meet the user's request, the image... EditCreate an image of a broccoli that is predominantly yellow... Think...The image provided shows a broccoli that is green with some yellow tones. To meet the user's request, the image... EditCreate an image of a broccoli that is entirely yellow in color... Benchmark check GenEval verifier R0fail R1pass R2pass R3pass Ends after the third edit. GenEval · Attribute binding “a red car and an orange potted plant.” Think...The current image shows a red car parked next to two potted plants, one with orange flowers and the other with yellow-green... EditReplace the two potted plants in the image with orange potted plants... Think...The current image shows a red car parked next to large potted plants with orange flowers. To meet the user's request, the image... Edit...The red car should be positioned next to a single orange potted plant... Think...The current image shows a red sports car with a large potted plant with orange flowers next to it. To meet the user's request... EditCreate an image featuring a red car and an orange potted plant... Benchmark check GenEval verifier R0fail R1fail R2pass R3pass Ends after the third edit. R0fail R1pass R2pass R3pass R0fail R1fail R2pass R3pass Figure14 Successfulrepairs (continued); layout as in Figure 13. 29 WISE · Culture “Most representative craft of Thailand.” Think...The image provided shows a temple complex, which is a significant cultural and architectural symbol... EditCreate an image showcasing a piece of Thai craftsmanship... Think...The current image shows a market scene with a basket, which is not inherently representative of Thai craft; Thai crafts can... EditCreate an image showcasing a traditional Thai craft item... Think...The current image shows a wooden sculpture in an artisan workshop, which is a form of Thai craftsmanship, but it may... EditCreate an image showcasing traditional Thai weaving, such as intricate... Benchmark check WISE consistency R00/2 R10/2 R20/2 R32/2 Ends after the third edit. T2I-CompBench · Color “a blue duck and a yellow pond.” Think...The current image shows a colorful duck in a pond setting with greenery and blue water, which does not meet the user's request... EditCreate an image featuring a blue duck situated in a yellow pond... Think...The current image shows a blue duck on a yellow surface, but it lacks context. To meet the user's request, the image... EditCreate an image showing a blue duck floating in a yellow pond... Think...The current image shows a blue duck on a yellow background, which does not depict a pond setting. To meet the user's... EditCreate an image where a blue duck is situated in a yellow pond setting... Benchmark check BLIP-VQA score R00.08 R10.38 R20.37 R30.93 Ends after the third edit. R00/2 R10/2 R20/2 R32/2 R00.08 R10.38 R20.37 R30.93 Figure15 Successfulrepairs (continued); layout as in Figure 13. 30 WISE · Physics “Viewing a clock through a handheld magnifier.” Think...The current image shows a person holding a clock, but it does not depict the act of viewing the clock through a magnifying... EditCreate an image showing a clock being viewed through... Think...The current image shows a person holding a clock, but without the magnifying glass element that is central to the user's... EditAdd a handheld magnifying glass in front of the clock held... ThinkThe user wants an image showing viewing a clock through a handheld magnifier, so the image should depict... EditCreate an image where a clock face is being viewed through a handheld... Benchmark check WISE consistency R00/2 R10/2 R22/2 R32/2 Ends after the third edit. T2I-CompBench · Shape “a round bag and a square box.” Think...The current image shows a brown round bag and a square gift box, but the bag appears to be more of a tote bag shape rather... Edit...The round bag should be clearly round, resembling a ball... Think...The image provided shows a bag that is more rounded but not perfectly round. To meet the user's request, the round... Edit...The round bag should be distinctly spherical, possibly... Think...The image provided shows a bag that is somewhat rounded but not perfectly round, and a box that is not square... Edit...The round bag should be distinctly spherical, possibly... Benchmark check BLIP-VQA score R00.12 R10.10 R20.68 R30.79 Ends after the third edit. R00/2 R10/2 R22/2 R32/2 R00.12 R10.10 R20.68 R30.79 Figure16 Successfulrepairs (continued); layout as in Figure 13. 31 WISE · Time “The signature vehicle associated with the hippie movement in the 1960s.” Think...The image provided shows a colorful car decorated with rainbow patterns, which is reminiscent of the hippie... EditRemove the people and text from the image, leaving only the vehicle... Think...The current image shows a car with rainbow-colored decorations and people around it, which might be associated... EditCreate an image featuring a Volkswagen bus with flower decals... Think...The image provided shows a Volkswagen bus decorated with flower decals, which aligns with the hippie... EditCreate an image of a van that is heavily decorated with peace symbols... Benchmark check WISE consistency R00/2 R10/2 R22/2 R32/2 Ends after the third edit. WISE · Culture “A beloved fruit in India, known as the 'king of fruits,' cherished for its rich flavor and cultural significance.” Think...The image provided shows a fruit with a spiky exterior, resembling a durian. However, the durian is not typically... EditCreate an image of an Indian custard apple, also known as the sugar apple... Think...The current image shows a custard apple, which is not commonly referred to as the 'king of fruits' in India... EditCreate an image of a ripe mango, showcasing its vibrant yellow-orange... Think...The current image shows a mango, which is indeed a fruit of cultural importance in India, but it does not fully represent... EditCreate an image that showcases the mango as the 'king of fruits' in India... Benchmark check WISE consistency R00/2 R10/2 R22/2 R32/2 Ends after the third edit. R00/2 R10/2 R22/2 R32/2 R00/2 R10/2 R22/2 R32/2 Figure17 Successfulrepairs (continued); layout as in Figure 13. 32 T2I-CompBench · 2D spatial “a turtle on the bottom of a chair.” Think...The current image shows a turtle sitting on top of a chair, which does not meet the user's request. To meet the request, the turtle... EditPlace a turtle at the bottom of a chair, positioned on the floor or ground... Think...The current image shows two turtles, one on a chair and another on the ground near the chair. To meet the user's request... EditPlace a single turtle on the bottom part of a chair, positioned on the floor... Think...The image provided shows a chair with a turtle on the ground next to it, which does not meet the user's request... EditPlace a turtle at the bottom of a chair, positioned on the ground or floor... Benchmark check UniDet spatial score R00.00 R10.98 R20.98 R31.00 Ends after the third edit. OneIG · Object “Husky made of smoke with striking blue eyes, floating in the air.” Think...The current image shows a dog with blue eyes, but it is not made of smoke and does not depict the dog floating... EditCreate an image of a Husky made entirely of smoke, floating in the air... Think...The current image shows a dog with blue eyes, but it is not made of smoke and does not depict the dog floating... EditCreate an image where a Husky is depicted as a swirling mass of smoke... Think...The current image shows a dog with blue eyes, but it is not made of smoke and does not depict the dog floating... EditCreate an image of a Husky entirely composed of smoke, floating in mid-air... Benchmark check OneIG alignment R00.60 R10.60 R20.60 R31.00 Ends after the third edit. R00.00 R10.98 R20.98 R31.00 R00.60 R10.60 R20.60 R31.00 Figure18 Successfulrepairs (continued); layout as in Figure 13. 33 GenEval · Counting “four giraffes.” Think...The current image shows three giraffes in a field, which does not meet the requirement of having four giraffes... EditAdd one giraffe to the image to make a total of four giraffes... Think...The current image shows three giraffes in a field. To meet the user's request, the image should be edited to include... EditAdd a fourth giraffe to the image to meet the requirement of having... Think...The current image shows three giraffes standing in a field. To meet the user's request, the image should be edited... EditAdd a fourth giraffe to the image to create a total of four giraffes... Benchmark check GenEval verifier R0pass R1fail R2fail R3fail Ends after the third edit. WISE · Culture “The official currency of the second largest economy in the world.” Think...The second largest economy in the world is the United States, and its official currency is the US dollar... EditCreate an image showing the US dollar bills as the official currency... Think...The second largest economy is the United States, whose official currency is the US dollar. The image should depict... EditCreate an image that prominently features the US dollar bill as the official... ThinkThe user wants an image that represents the official currency of the second largest economy in the world, which is the United States dollar... EditCreate an image that showcases the US dollar... Benchmark check WISE consistency R00/2 R10/2 R20/2 R30/2 Ends after the third edit. R0pass R1fail R2fail R3fail R00/2 R10/2 R20/2 R30/2 Figure19 Failurecases. Failures caused by the reflection. Left: the first image already shows four giraffes and passes, but the reflection counts them as three and asks for one more; it repeats “three giraffes” in every later round while the image holds five. Right: the reflection states that the second largest economy is the United States, so all three edits render US dollars instead of the Chinese yuan. Frames mark the benchmark verdict. 34 P StoppingBehaviorunderTerminalRewards The reflection protocol lets the policy end a trajectory with DONE at any round. How the terminal round is priced determines when the policy stops; we compare two prices on the same parent, prompt stream, and seed, both at 500 updates. Penalty on a wrong DONE (primary reward). A DONE emitted on a verifier-incorrect image costs 0.5; a DONE on a correct image earns nothing beyond the terminal quality. The gains in Tables 1, 2, and 3 are realized in this regime, with accuracy rising through rounds one, two, and three. Adding a correct-DONE bonus. Adding a bonus for a DONE on a correct image, together with a−0.5 action penalty on editing an already-correct image, lowers the confidence at which stopping breaks even from 1.0 to 0.5. Stopping becomes cheap, and the policy learns to stop prematurely: it declares DONE on 112 incorrect images, which make up 112 of its 116 final failures, and final accuracy falls from 82 to 79. At a break-even of 0.5, a stop pays off whenever the image is as likely wrong as right, so the policy gives up on images it could still repair. Summary. A correct-DONE bonus makes stopping cheap and turns reflection into premature acceptance of failed images. The wrong-DONE penalty keeps the policy repairing and gives the highest final accuracy; UMM-Reflection therefore uses the penalty alone. Q Per-FamilyLearningonGenEval −0.5 0.0 0.5 1.0 1.5 Trajectory reward 6 prompt draws a Single object 66 prompt draws b Two objects 533 prompt draws c Counting 02505007501000 RL updates −0.5 0.0 0.5 1.0 1.5 Trajectory reward 58 prompt draws d Colors 02505007501000 RL updates 916 prompt draws e Position 02505007501000 RL updates 421 prompt draws f Color attribution Prompt mean (16 rollouts)100-update mean Figure20 TrainingrewardbyGenEvalfamily. Light points are the mean whole-trajectory reward of the 16 rollouts for one prompt; lines are 100-update means. Prompt draws per family follow the training pool composition; single object has six draws and its line is left open. 35 The six GenEval families learn at different rates, and the training reward predicts the held-out gain (Figure 20). Position rises most, from about 0.5 to 1.0 over the run, and its GenEval accuracy rises from 55 to 89. Color attribution and colors rise moderately and gain 17 and 6 points. Two objects starts high and gains 12 points; single object is at ceiling. Counting is the one family whose training reward stays flat across 1,000 updates, and its GenEval accuracy is unchanged at 67.5: the verifier requires an exact count, and the edits the policy learns, which add, remove, recolor, and reposition objects, do not yet move the count reliably. Counting therefore identifies the next training target for reflection, count-aware edits, where the same reward and protocol apply without change. R InferenceSystemPrompt The system prompt used by the native inference driver is reproduced below. You are an image generation, editing, and verification agent operating over an ordered visual trajectory. The original user request remains the final objective throughout the trajectory. A T2I trajectory begins with no source image; an edit trajectory begins with a given source image. Later visible images are the current intermediate state. At every reasoning turn, output exactly one assistant response: use one <think> block with these exact tags: [CURRENT_ROUND], [SCORE], [ACTION], [THINKING], and [SOURCE_IMAGE], then place exactly one [EDIT] field immediately after </think> in the same response. Use [SOURCE_IMAGE] None before initial T2I generation, given for an external edit source, or Image #N for a generated intermediate image. Use [ACTION] edit with one concrete [EDIT] instruction to generate or revise the next image. Use [ACTION] done only when the complete original request is visibly satisfied, with [EDIT] None. For a planned progression, execute only the currently due milestone. Constraints assigned to future milestones are intentionally pending, not model failures or defects in the current image. Preserve completed milestones while advancing the next one. Call a constraint failed only when it was due and is visibly incorrect. Judge only visible evidence, preserve unrelated content for edits, and do not invent corruption, rollback state, or hidden success. For a planned progression, begin first-round [THINKING] with the complete compact milestone plan, then execute only the currently due milestone. Keep that plan in the persistent history for later verification turns. The [SCORE] field is always written on the fixed protocol scale as N/10, where N is an integer from 0 to 10 and 10 means the original request is fully satisfied by the current image. Never use another denominator, a percentage, or a bare number. If the current image already satisfies the complete original request, emit [ACTION] done with [EDIT] None. Never emit [ACTION] edit with an empty, None, or placeholder [EDIT] payload; that is a done decision. 36