Paper deep dive
Understanding Reward Hacking in Text-to-Image Reinforcement Learning
Yunqi Hong, Kuei-Chun Kao, Hengguang Zhou, Cho-Jui Hsieh
Models: Janus-Pro-1B, Qwen2.5-VL-7B-Instruct
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 5:29:24 PM
Summary
The paper investigates reward hacking in text-to-image (T2I) reinforcement learning, where models exploit imperfect reward functions to achieve high scores while producing visually unrealistic artifacts. The authors identify a universal failure mode across existing reward models (e.g., HPS, GDino, ORM) and propose 'ArtifactReward', a lightweight, adaptive regularizer trained on a small dataset to penalize structural artifacts, which significantly improves visual realism and reduces reward hacking.
Entities (6)
Relation Signals (3)
ArtifactReward → usesbackbone → Qwen2.5-VL-7B-Instruct
confidence 98% · we use Qwen2.5-VL-7B-Instruct [2] as the backbone visual-language model
ArtifactReward → mitigates → Reward Hacking
confidence 95% · incorporating our artifact reward significantly improves visual realism and reduces reward hacking
Janus-Pro-1B → trainedwith → HPS
confidence 90% · We finetune the Janus-Pro-1B model [4] on the same training set... under different reward configurations.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has become a standard approach for post-training large language models and, more recently, for improving image generation models, which uses reward functions to enhance generation quality and human preference alignment. However, existing reward designs are often imperfect proxies for true human judgment, making models prone to reward hacking--producing unrealistic or low-quality images that nevertheless achieve high reward scores. In this work, we systematically analyze reward hacking behaviors in text-to-image (T2I) RL post-training. We investigate how both aesthetic/human preference rewards and prompt-image consistency rewards individually contribute to reward hacking and further show that ensembling multiple rewards can only partially mitigate this issue. Across diverse reward models, we identify a common failure mode: the generation of artifact-prone images. To address this, we propose a lightweight and adaptive artifact reward model, trained on a small curated dataset of artifact-free and artifact-containing samples. This model can be integrated into existing RL pipelines as an effective regularizer for commonly used reward models. Experiments demonstrate that incorporating our artifact reward significantly improves visual realism and reduces reward hacking across multiple T2I RL setups, demonstrating the effectiveness of lightweight reward augment serving as a safeguard against reward hacking.
Tags
Links
- Source: https://arxiv.org/abs/2601.03468
- Canonical: https://arxiv.org/abs/2601.03468
Trouble viewing inline? Open PDF directly →
Full Text
55,887 characters extracted from source content.
Expand or collapse full text
Understanding Reward Hacking in Text-to-Image Reinforcement Learning Yunqi Hong, Kuei-Chun Kao, Hengguang Zhou, Cho-Jui Hsieh Department of Computer Science, University of California, Los Angeles yunqihong, chohsieh@cs.ucla.edu Abstract Reinforcement learning (RL) has become a standard ap- proach for post-training large language models and, more recently, for improving image generation mod- els, which uses reward functions to enhance genera- tion quality and human preference alignment. How- ever, existing reward designs are often imperfect proxies for true human judgment, making models prone to re- ward hacking–producing unrealistic or low-quality im- ages that nevertheless achieve high reward scores. In this work, we systematically analyze reward hack- ing behaviors in text-to-image (T2I) RL post-training. We investigate how both aesthetic/human preference re- wards and prompt-image consistency rewards individ- ually contribute to reward hacking and further show that ensembling multiple rewards can only partially mit- igate this issue. Across diverse reward models, we iden- tify a common failure mode: the generation of artifact- prone images. To address this, we propose a lightweight and adaptive artifact reward model, trained on a small curated dataset of artifact-free and artifact-containing samples. This model can be integrated into existing RL pipelines as an effective regularizer for commonly used reward models. Experiments demonstrate that incorporating our ar- tifact reward significantly improves visual realism and reduces reward hacking across multiple T2I RL setups, demonstrating the effectiveness of lightweight reward augment serving as a safeguard against reward hacking. 1. Introduction The use of reinforcement learning (RL) has proven highly effective for post-training large language models (LLMs), enabling stronger alignment with human pref- erences through methods such as reinforcement learn- ing from human feedback (RLHF) [27] or more re- cently improving prediction accuracy via reinforcement learning with verifiable feedback (RLVF) [13, 15, 31]. This success has motivated a growing interest in ex- ploring RL-based fine-tuning for image generation tasks [5, 10, 18, 39, 44, 47]. To guide the optimization of generative models, var- ious types of reward functions have been proposed. Ar- chitecturally, many of these rewards are built on vision- language encoders such as CLIP or BLIP [19, 42, 43]. Others leverage multimodal LLMs, either through direct scoring [20, 38, 47], additional prediction heads [50], or token-level probability signals [14, 22, 23, 40, 41, 46]). A third category relies on external perception tools such as object detectors [6, 25, 51]. Functionally, reward models are typically designed to measure aesthetic or perceptual quality [30, 45, 46], human preference align- ment [19, 38, 42, 43], and prompt-image consistency [14, 22, 23, 25]. However, unlike textual domains where verifiable signals (e.g., correctness or factuality) can be obtained, reward functions for visual generation are often non- verifiable imperfect proxies for human preferences [24, 49]. Such imperfect reward design can inadvertently en- courage models to exploit weaknesses in the reward sig- nal rather than genuinely improving visual generation quality, which is a phenomenon known as reward hack- ing [1, 8, 11, 34]. As a result, fine-tuned models may produce visually implausible or low-quality images that nonetheless receive high reward scores. In this work, we conduct the first systematic analysis of reward hacking behaviors in text-to-image (T2I) RL post-training. We investigate how different types of re- ward functions individually contribute to reward hack- ing, including those designed for aesthetics or human preference and those enforcing prompt-image consis- tency. Although ensembling multiple rewards partially mitigate this issue, it cannot fully prevent reward ex- ploitation. Across all settings, we identify a universal pattern of reward hacking: the generation of unrealis- tic visual artifacts that nevertheless receive high reward scores. While prior studies have occasionally reported instances of reward exploitation or proposed regulariza- tion techniques to mitigate it [8, 24, 32, 49], ours is the first to systematically characterize how different reward types induce specific hacking behaviors and to identify 1 arXiv:2601.03468v1 [cs.CV] 6 Jan 2026 their common pattern of artifact generation. To address this reward hacking issue, we propose a lightweight and adaptive artifact reward model that can be seamlessly integrated into existing T2I RL pipelines. Our approach requires only a small dataset (approxi- mately 200 samples) of model-generated images con- sisting of both artifact-free and artifact-containing ex- amples. By using an LLM-based backbone, we automat- ically optimize a textual prompt that discriminates be- tween these two categories, so as to assign higher scores to artifact-free images and lower scores to those contain- ing artifacts. Extensive experiments demonstrate that using our ar- tifact reward as a regularizer for existing reward models consistently improves visual realism and reduces reward hacking across multiple T2I RL setups. Evaluations on two benchmarks, WISE [26] and LLM4LLM [37], show that reliably boosts realism, consistency, and semantic alignment across all training configurations. 2. Related Work 2.1. Evaluation Metrics for T2I Generation Unlike domains such as mathematics or programming, where ground-truth answers can serve as objective re- ward signals, evaluating the quality of generated images is inherently more complex and multi-dimensional. It may depend on factors such as aesthetic appeal, seman- tic alignment with the prompt, and overall visual real- ism, leading to a diverse landscape of reward functions and evaluation metrics. A major class of reward models is built upon vision- language encoders such as CLIP [29] and BLIP [21]. HPSv2 [42] and PickScore [19] fine-tune CLIP on human-preference datasets to predict visual quality based on human aesthetic appeal and preference. Aes- thetic Score [30] predicts image aesthetics using a feed- forward neural network over CLIP features. ImageRe- ward [43] is built on BLIP to estimate preference scores via cross-attention between image and text features. With the emergence of MLLMs that exhibit strong visual reasoning abilities, many works have explored using them as image evaluators. Davidsonian Scene Graph [7] evaluates image-prompt consistency by an- swering questions corresponding to the prompt based on the generated image.VIEScore [20] utilizes the general knowledge of MLLMs to evaluate con- ditional image generation without additional finetun- ing.EvalAlign [35] finetunes MLLMs to evaluate image faithfulness and text-image alignment.Uni- fiedReward [38] finetunes MLLMs to directly output scalar image scores through comprehensive analyses. LLaVA-Reward [50] employs the Bradley-Terry objec- tive to train an MLLM evaluator using pairwise prefer- ence data. Other works, such as Q-Align [40], VQAS- core [23], DeQA [46], finetuned ORM [14], and Re- wardDance [41], estimate image quality based on the to- ken probabilities of specific answer tokens, each empha- sizing different evaluation aspects depending on their datasets and design choices. 2.2. Existing Rewards for T2I Post-Training Gradient-basedfinetuningmethods. ImageRe- ward [43] introduces Reward Feedback Learning (ReFL), a direct optimization approach that uses gradients from reward models, to finetune diffusion models. DRaFT [8] further analyzes how rewards such as HPSv2 [42] and PickScore [19] influence model behavior when applied through differentiable optimiza- tion. ROCM [32] compared training outcomes using different reward functions, including PickScore [19], HPSv2 [42], CLIPScore [16], and Aesthetic Score [30], to study their individual effects. RL-based optimization. Recent progress in RL for LLMs has renewed interest in applying RL-based fine- tuning to T2I models. Existing methods primarily dif- fer in the choice of reward functions and optimization strategies. RL Diffusion [49] adopts REINFORCE-style pol- icy gradients to train diffusion models with various re- ward functions, including ImageReward [43] for hu- man preference, UniDet [51] for compositionality, and a distribution-based statistical parity reward for diversity and fairness. DDPO [3] employs Aesthetic Score [30] and BERTScore [48], which measures the semantic similarity between the prompt and VLM-generated im- age captions, as reward signals.DanceGRPO [44] finetunes diffusion models with HPS-v2.1 [42] and CLIP [29] rewards, while Flow-GRPO [24] improves the flow-matching models by using a combination of compositionality (GenEval [12]) and human preference (PickScore [19]) rewards. T2I-R1 [18] ensembles multi- ple rewards, including HPS [42], Grounding DINO [25], a VQA model [36], and the finetuned ORM [14]. AR- GRPO [47] incorporates HPS-v2.1 [42] and CLIP [29] for conditional reward, MANIQA [45] for image quality, and prompting Qwen2.5-VL-3B-Instruct [2] for realism. Overall, existing T2I reward functions can be broadly categorized into three groups: • Human preference rewards such as HPS [42], PickScore [19], and ImageReward [43]; • CompositionalityrewardssuchasGrounding DINO [25]; • MLLM-based rewards derived from visual question answering or image scoring models [14, 23, 38, 46]. Our work focuses on analyzing model behaviors when trained with rewards from these categories. 2 2.3. Reward Hacking in T2I RL training Although none of the existing work provides a detailed discussion and systematic analysis of reward hacking in T2I post-training, several studies have briefly dis- cussed instances of reward hacking in their training ex- periments. DRaFT [8] reports that finetuned models sometimes lose diversity and produce reward-specific visual patterns. RL Diffusion [49] hypothesizes that gradient-based optimization methods (i.e. ReFL [43] and DRaFT [8]) are more susceptible to reward hack- ing because they directly access reward gradients, and thus proposes an RL-based alternative to improve ro- bustness. ROCM [32] introduces distributional regular- ization to stabilize training and mitigate reward exploita- tion. Flow-GRPO [24] uses KL regularization for simi- lar purposes. 3. Reward hacking analysis In this section, we characterize the reward hacking be- havior observed in T2I RL training. We first describe the definition of reward hacking and its relevance to T2I generation (Section 3.1), followed by a description of our experimental setup and the reward functions exam- ined (Section 3.2). We then analyze the training be- haviors under different reward models and reveal that the model consistently exploits these rewards to achieve high training rewards while producing visually unreal- istic and artifact-heavy images. Notably, this behavior persists across a wide range of reward settings. 3.1. Definition of Reward Hacking Reward hacking refers to the phenomenon where opti- mizing an imperfect proxy reward results in degraded performance with respect to the true underlying objec- tive [34]. In other words, reward hacking occurs when a model learns to exploit weaknesses in the reward func- tion and achieve high scores under the designed proxy while performing poorly according to the true goal. In the context of text-to-image generation, numerous reward functions have been proposed to guide model training. However, each reward is typically designed to capture a specific aspect of generation quality, such as aesthetic appeal, human preference, or prompt-image alignment. Since these rewards are predefined and in- herently partial, they are vulnerable to reward hacking [24]. 3.2. Reward Choice and Evaluation for T2I RL Training As discussed in Section 2.1, one major class of reward models for T2I RL training is built upon vision-language encoders such as CLIP [29] and BLIP [21]. These mod- els primarily aim to capture aesthetic appeal and human preference. To analyze how such reward functions af- fect model behavior, we select HPS [42] as a representa- tive. This reward model correlates strongly with human preference and aesthetic judgments, making it an ideal candidate to study potential reward hacking phenomena driven by subjective visual quality signals. In contrast, another important dimension of T2I eval- uation is image-prompt alignment, which measures how well the generated image reflects the semantics de- scribed in the text prompt. A common approach to en- forcing this alignment is to employ an object detector as a compositionality reward [6, 25, 51]. Alternatively, re- cent works fine-tune MLLMs to directly assess whether an image matches the given prompt [14, 23, 35, 38]. In our study, we adopt Grounding DINO [25] as a repre- sentative object detection-based reward, and finetuned ORM [14] as a representative MLLM-based prompt- image alignment reward. We train Janus-Pro-1B [4] with each of these reward functions independently and monitor the reward hacking behaviors. The training dynamics are evaluated using various kinds of metrics: • Aesthetic/human preference-oriented: HPS [42], Aes- thetic Score [30], PickScore [19], and DeQA [46]; • Text-image alignment-based: GDino [25], ORM [14], and VQAScore [23]; • Unified quality and alignment: UnifiedReward [38]. 3.3. Training Dynamics and Reward Hacking Behaviors We finetune the Janus-Pro-1B model [4] on the same training set used in T2I-R1 [18], running three epochs of RL training (approximately 3000 steps) under differ- ent reward configurations. Figure 1 shows the evolution of various evaluation metrics throughout training. We only visualize the first 1200 steps for the HPS [42] re- ward, as its metric values begin to collapse shortly after this point. Reward-specific improvements and cross-metric degradation. Across most metrics, each model pri- marily improves on the metric family aligned with its training reward, but fails to generalize to others. For in- stance, models trained with GDino [25] or ORM [14] show steady gains on their respective reward values, yet exhibit clear degrade in human preference, aesthetic quality, and DeQA scores (Figure 1a-1c). Conversely, the model trained with HPS [42] shows rising HPS re- ward and aesthetic scores but experiences clear drops on grounding accuracy and semantic consistency (Fig- ure 1e-1f). Other evaluation metrics, such as PickScore (Figure 1d), VQAScore (Figure 1g), and UnifiedReward (Figure 1h), show no consistent upward trend across training steps, regardless of reward types. Training with 3 0100020003000 Steps 0.26 0.27 0.28 0.29 0.30 (a) HPS - Object 0100020003000 Steps 5.4 5.6 5.8 6.0 6.2 6.4 6.6 (b) Aesthetic Score - Object 0100020003000 Steps 3.3 3.4 3.5 3.6 3.7 3.8 (c) DeQA - Spatial 0100020003000 Steps 0.218 0.219 0.220 0.221 0.222 0.223 0.224 0.225 (d) PickScore - Object 0100020003000 Steps 0.55 0.60 0.65 0.70 0.75 (e) GDino - Spatial 0100020003000 Steps 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 (f) ORM - Color 0100020003000 Steps 0.78 0.80 0.82 0.85 0.88 0.90 0.93 0.95 (g) VQAScore - Object 0100020003000 Steps 3.8 4.0 4.2 4.4 4.6 (h) UnifiedReward - Object 0100020003000 Steps 0.30 0.32 0.34 0.36 0.38 0.40 0.42 0.44 0.46 (i) Artifacts - Object 0100020003000 Steps 0.40 0.42 0.44 0.46 0.48 (j) Artifacts - Shape 0100020003000 Steps 0.30 0.32 0.34 0.36 0.38 (k) Artifacts - Spatial 0100020003000 Steps 0.42 0.44 0.46 0.48 0.50 0.52 0.54 (l) Artifacts - Texture Figure 1. Evolution of metrics over training steps trained on Janus-Pro-1B [4]. Blue color denotes the model trained with HPS [42]; Orange color denotes the model trained with GDino [25]; Green color denotes the model trained with HPS and GDino; Red color denotes the model trained with finetuned ORM [14]. combined rewards (HPS + GDino) yields more stable and moderate improvements, suggesting partial comple- mentary effects between the two reward signals. How- ever, this ensemble only partially mitigates the issue, as reward-specific biases still remain. Overall, this cross-metric mismatch reveals that the model is optimizing narrowly for the specific reward sig- nals rather than true improvements in generation quality, which is a direct manifestation of reward hacking. Visual inspection of reward-driven biases. By ex- amining the generated images, distinct reward-induced tendencies become apparent (see Table 1). The HPS- trained model tends to produce vibrant backgrounds and visually striking colors, since HPS implicitly favors aesthetic appeal. However, over-emphasizing colorful- ness often leads to unrealistic and over-saturated images, sacrificing spatial grounding and object fidelity. Even when combined with GDino, this issue persists, as train- ing remains biased toward HPS gains (Figure 1a and 1e), and thus the images still look unrealistic and over- saturated. In contrast, GDino or ORM-trained models generate simpler, object-centered compositions. This behavior naturally reflects their design: object detectors like GDino focus solely on the presence and localization of specified entities, while ORM is essentially trained with object-centered prompts, giving high rewards to clear depiction of main objects. Thus, these models achieve stronger grounding but reduced background di- versity and realism. Universal reward failure:structural artifacts. While the above analysis highlights reward-specific bi- ases and cross-metric degradation, we further identify a more pervasive and universal reward failure pattern in T2I RL–structural artifacts. Across the qualitative re- sults in Table 1, generated images frequently exhibit ge- ometric distortions (e.g., physically implausible or dis- torted tennis rackets), object duplication or fragmenta- tion (e.g., duplicated limbs or overlapping object out- lines), and even blending between different entities (e.g., a human body merged with a bear head). Remarkably, 4 Table 1. Image Illustration. The first column is the prompt. The second and fourth columns are from models trained with existing rewards. The third and fifth columns are from models trained with the help of our ArtifactReward. The label above each image is the reward used in RL training. HPSHPS + artifactsGDinoGDino + artifacts A photo of a cup above a tennis racket HPS + GDinoHPS + GDino + artifactsORMORM + artifacts HPSHPS + artifactsGDinoGDino + artifacts A photo of a person right of a bear HPS + GDinoHPS + GDino + artifactsORMORM + artifacts HPSHPS + artifactsGDinoGDino + artifacts A person is looking at a map and planning a hiking trail. HPS + GDinoHPS + GDino + artifactsORMORM + artifacts 5 Table 2.Accuracy of different reward models in assign- ing higher scores to artifact-free images compared to artifact- containing ones. Each image pair is scored as 1 if the reward assigns a higher score to the artifact-free image, 0.5 if the scores are tied, and 0 if the artifact-containing image receives a higher score. The accuracy is averaged across all pairs. RewardsCorrectTieWrongAccuracy HPS [42]1080940.53 Aesthetic Score [30]8401180.42 ImageReward [43]1337620.68 MANIQA [45]8001220.39 DeQA [46] 1040980.51 GDino [25]46143130.58 ORM [14]39149140.56 UnifiedReward [38] 8692240.65 ArtifactReward (Ours)1612390.80 these structural failures arise consistently under every reward configuration (HPS, GDino, ORM, and HPS + GDino), revealing a shared systematic blind spot in current reward formulations. None of these rewards re- liably enforce local structure or fine-grained object cor- rectness. As a result, the generative policy exploit this loophole, optimizing reward signals without genuinely learning to produce geometrically coherent images. This behavior illustrates a deeper form of reward hacking, where models achieve higher scores by bypassing true structural correctness. Summary. Together, these findings reveal clear and pervasive reward hacking in T2I RL training. Each re- ward guides the model toward narrow patterns that max- imize its own metric while degrading other aspects of performance. The model learns to exploit superficial cues favored by the reward, such as over-saturated colors for HPS or object-centric simplifications for GDino and ORM. Even joint-optimization with multiple rewards (e.g., HPS + GDino) provides only partial mitigation, as the underlying biases remain. More fundamentally, cur- rent reward formulations lack any mechanism that dis- courages artifact-driven shortcuts. Without constraints enforcing structural plausibility or fine-grained correct- ness, the model is free to achieve higher scores through geometrically distorted and implausible generations. 4. Mitigating Reward Hacking via Artifact- Aware Reward Design To quantify the prevalence of artifact generation, we cu- rated a small diagnostic dataset comprising both artifact- containing and artifact-free images, annotated via hu- man judgment. For each prompt, we formed pairs of one artifact-free image and one artifact-containing im- age, allowing controlled comparison of reward behav- iors. Representative examples are provided in Table 3. Table 3. Examples of curated artifact diagnostic dataset. The upper images show no artifacts, while the lower images con- tain artifacts. rubber sole shoes and a wooden chaira wooden table and a fabric hat HPS: 0.3293HPS: 0.2878 GDino: 0.4GDino: 1.0 ORM: 1.0 ORM: 1.0 Artifact: 0.7549Artifact: 0.9149 HPS: 0.3362HPS: 0.2893 GDino: 0.4GDino: 1.0 ORM: 1.0 ORM: 1.0 Artifact: 0.2689Artifact: 0.3486 a photo of a bird above a scissorsa photo of a handbag left of a toaster HPS: 0.2795HPS: 0.2272 GDino: 1.0GDino: 0.9 ORM: 0.0ORM: 0.6027 Artifact: 0.8355Artifact: 0.7879 HPS: 0.2988HPS: 0.2896 GDino: 1.0GDino: 0.9 ORM: 0.0ORM: 1.0 Artifact: 0.4844Artifact: 0.4378 We then evaluate existing reward models on this cu- rated dataset to assess whether they correctly assign higher scores to artifact-free images compared with artifact-containing images. As summarized in Table 2, none of the existing reward models consistently penal- ize images with clear structural failures. In many cases, they assign equal or even higher scores to flawed im- ages. GDino, ORM and UnifiedReward, in particular, frequently produce identical scores across pairs (“Tie”), revealing an inability to detect fine-grained geometric inconsistencies. This results expose a centural limitation in current T2I reward designs: they lack sensitivity to fine-grained local structural coherence and object plau- sibility, allowing models to exploit reward loopholes and generate visually implausible yet high-reward images. 4.1. ArtifactReward: Complementary Reward via Automatic Prompt Optimization To address this limitation, we introduce ArtifactReward, a complementary reward function explicitly designed to penalize unrealistic or artifact-containing generations. Rather than relying on large-scale human annota- tion, we leverage automatic prompt optimization (APO) [9, 28, 33, 52] to construct a lightweight, data-efficient proxy reward from limited labeled examples. Specificly, we use Qwen2.5-VL-7B-Instruct [2] as the backbone visual-language model. Details of the 6 Table 4. Performance on WISE [26] benchmark trained on Janus-Pro-1B [4]. WiScore = 0.7× Consistency + 0.2× Re- alism + 0.1× Aesthetic by the default setting. MethodConsistencyRealismAestheticWiScore Janus-Pro-1B [4]0.24200.35500.37900.2783 HPS [42]0.29600.45150.61150.3587 HPS + Artifact0.30000.57600.54700.3799 GDino [25]0.29050.41000.43950.3293 GDino + Artifact0.31500.55500.50650.3821 ORM [14]0.32100.47500.50600.3703 ORM + Artifact0.33350.56000.50950.3964 HPS + GDino0.31850.48050.59600.3787 HPS + GDino + Artifact0.33350.57650.52100.4009 T2I-R1 [18]0.33700.49450.56750.3916 T2I-R1 + Artifact 0.34350.55800.51950.4040 APO procedure for constructing ArtifactReward are pro- vided in Supplementary 6. At inference time, the model is prompted with the optimized instruction and asked to determine whether the input image contains any unre- alistic artifacts. The model outputs probabilities for the “yes” and “no” tokens. We define the ArtifactReward score as the normalized probability assigned to “no”, i.e., which corresponds to the estimated likelihood that the image is artifact-free: R ArtifactReward = 1 1 + e [log(p yes )−log(p no )] .(1) Through APO, the instruction is iteratively refined to maximize discrimination between artifact-free and artifact-containing samples. The resulting ArtifactRe- ward demonstrates strong ability to detect structural in- consistencies and fine-grained artifacts (Table 2), out- performing existing rewards in recognizingunrealistic generations. We further evaluate the ArtifactReward signal across different existing training reward configurations. As shown in Figure 1i-1l, none of existing reward models produce a meaningful increase in ArtifactReward scores during training, reinforcing that current rewards do not effectively suppress artifact formation. 4.2. Integrating ArtifactReward into RL Train- ing We incorporate ArtifactReward into the four prior RL settings (HPS, GDino, ORM, and HPS + GDino) and the T2I-R1 [18] setting. As illustrated in Table 1, incorpo- rating ArtifactReward consistently reduces the preva- lence of visual artifacts in generated images. The re- sulting images exhibit reduced artifacts, improved real- ism and more balanced color composition. Table 5. Performance on LLM4LLM [37] benchmark trained on Janus-Pro-1B [4]. MethodPerceptionCorrespondenceAll Janus-Pro-1B [4]0.40560.50580.9114 HPS [42]0.42400.50670.9307 HPS + Artifact 0.44640.50890.9553 GDino [25]0.42300.53710.9601 GDino + Artifact 0.43360.54160.9751 ORM [14]0.43470.54180.9765 ORM + Artifact0.44790.54970.9976 HPS + GDino0.44820.53680.9850 HPS + GDino + Artifact0.44940.54410.9934 T2I-R1 [18]0.44480.53900.9838 T2I-R1 + Artifact 0.44630.54580.9921 4.3. Benchmark Evaluation To assess the generalizability of ArtifactReward, we evaluate all training variants on two recent benchmarks: • WISE [26]: uses GPT-4o [17] to evaluate consistency, realism, and aesthetic quality of generated images; • LLM4LLM [37]:employs two fine-tuned LLM judges to compute Perception (clarity, authenticity, aesthetics) and Correspondence (text-image align- ment). Table 4 shows the results on WISE benchmark. Across all reward configurations, integrating Artifac- tReward leads to consistent performance improvements, particularly in realism and consistency. The primary gain originates from ArtifactReward’s explicit suppres- sion of structural distortions and over-saturated appear- ances. Consequently, images appear more natural and coherent, despite sometimes at a modest cost to aesthetic scores, which is an expected tradeoff given the reduced emphasis on exaggerated colorfulness. Table 5 shows the results on LLM4LLM bench- mark. Incorporating ArtifactReward yields consistent gains across most training configurations. On the Per- ception dimension, ArtifactReward improves nearly all baselines, indicating that reducing structural artifacts di- rectly enhances the perceived authenticity of generated images. Similarly, the Correspondence dimension also benefits from ArtifactReward, often achieving the best or near-best scores within each reward setting. Figure 2 and Figure 3 illustrate qualitative exam- ples from these two benchmarks. In Figure 2, images generated with the help of our ArtifactReward exhibit noticeably improved realism and better alignment with the prompt intent, while also reducing the excessive color saturation often produced by HPS-trained models. In Figure 3, we also observe that HPS-based training tends to introduce unnatural colors, whereas GDino and ORM-based training frequently results in duplicated or 7 (a) HPS(b) HPS + Artifact(c) GDino(d) GDino + Artifact (e) HPS + GDino(f) HPS + GDino + Artifact(g) ORM(h) ORM + Artifact Figure 2. Images generated with prompt “Traditional food of the Mid-Autumn Festival” in WISE [26] benchmark under different training reward configurations. This prompt expect the image to show mooncakes. (a) HPS(b) HPS + Artifact(c) GDino(d) GDino + Artifact (e) HPS + GDino(f) HPS + GDino + Artifact(g) ORM(h) ORM + Artifact Figure 3. Images generated with prompt “a photo of a cow” in LLM4LLM [37] benchmark under different training reward config- urations. geometrically distorted objects. In contrast, incorporat- ing our ArtifactReward effectively suppresses these is- sues, yielding images with more coherent structure and plausible visual attributes. Summary. Together, these results demonstrate that ArtifactReward generalizes well across diverse reward settings and architectures, consistently improving both perceptual quality and semantic correspondence. By ex- plicitly targeting perceptual and structural plausibility, failure modes that existing rewards systematically over- 8 look, ArtifactReward acts as an effective and general- purpose regularizer against T2I reward hacking. It com- plements existing rewards that emphasize aesthetics or compositional alignment.Although aesthetic scores may slightly decrease due to reduced over-saturation, the substantial improvements in realism, alignment con- sistency, and cross-benchmark robustness highlight Ar- tifactReward’s value as a meaningful step toward more human-aligned T2I reward design. 5. Conclusion In this work, we present the first systematic study of reward hacking in Text-to-Image RL post-training. Through extensive analysis across widely used reward models, which spans aesthetics, human preference, and image-text alignment, we reveal that existing rewards not only induce reward-specific biases but also share a universal blind spot: the persistent production of struc- tural artifacts that current reward functions fail to penal- ize. To address this gap, we introduce ArtifactReward, a complementary reward learned via automatic prompt optimization to detect fine-grained visual artifacts from limited supervision. When integrated into multiple RL training configurations, ArtifactReward consistently re- duces artifact prevalence, improves realism, and en- hances benchmark performance. References [1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Chris- tiano, John Schulman, and Dan Man ́ e. Concrete prob- lems in ai safety. arXiv preprint arXiv:1606.06565, 2016. 1 [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 6, 1 [3] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with re- inforcement learning. arXiv preprint arXiv:2305.13301, 2023. 2 [4] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and gen- eration with data and model scaling.arXiv preprint arXiv:2501.17811, 2025. 3, 4, 7, 1, 2, 5, 6, 8, 9, 10 [5] Yihang Chen, Yuanhao Ban, Yunqi Hong, and Cho-Jui Hsieh. Iris: Intrinsic reward image synthesis. arXiv preprint arXiv:2509.25562, 2025. 1 [6] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16901–16911, 2024. 1, 3 [7] Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text- to-image generation. arXiv preprint arXiv:2310.18235, 2023. 2 [8] Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differen- tiable rewards. arXiv preprint arXiv:2309.17400, 2023. 1, 2, 3 [9] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning, 2022. 6 [10] Chengqi Duan, Rongyao Fang, Yuqing Wang, Kun Wang, Linjiang Huang, Xingyu Zeng, Hongsheng Li, and Xihui Liu. Got-r1: Unleashing reasoning capability of mllm for visual generation with reinforcement learn- ing. arXiv preprint arXiv:2505.17022, 2025. 1 [11] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. 1 [12] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neu- ral Information Processing Systems, 36:52132–52152, 2023. 2 [13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 1 [14] Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Rui Huang, Haoquan Zhang, Manyuan Zhang, Ji- aming Liu, Shanghang Zhang, Peng Gao, et al. Can we generate images with cot?let’s verify and rein- force image generation step by step.arXiv preprint arXiv:2501.13926, 2025. 1, 2, 3, 4, 6, 7 [15] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chao- jie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312, 2025. 1 [16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021. 2 [17] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Ak- ila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 7 [18] Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703, 2025. 1, 2, 3, 7 [19] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image gen- 9 eration. Advances in neural information processing sys- tems, 36:36652–36663, 2023. 1, 2, 3 [20] Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867, 2023. 1, 2 [21] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022. 2, 3 [22] Yuheng Li, Haotian Liu, Mu Cai, Yijun Li, Eli Shecht- man, Zhe Lin, Yong Jae Lee, and Krishna Kumar Singh. Removing distributional discrepancies in captions im- proves image-text alignment. In European Conference on Computer Vision, pages 405–422. Springer, 2024. 1 [23] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ra- manan. Evaluating text-to-visual generation with image- to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024. 1, 2, 3 [24] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jia- heng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching mod- els via online rl. arXiv preprint arXiv:2505.05470, 2025. 1, 2, 3 [25] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38– 55. Springer, 2024. 1, 2, 3, 4, 6, 7 [26] Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kun- peng Ning, Bin Zhu, et al. Wise: A world knowledge- informed semantic evaluation for text-to-image genera- tion. arXiv preprint arXiv:2503.07265, 2025. 2, 7, 8, 1, 4, 5, 6 [27] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with hu- man feedback. Advances in neural information process- ing systems, 35:27730–27744, 2022. 1 [28] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023. 6, 1 [29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learning, pages 8748–8763. PmLR, 2021. 2, 3 [30] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Ad- vances in neural information processing systems, 35: 25278–25294, 2022. 1, 2, 3, 6 [31] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 1 [32] Shivanshu Shekhar and Tong Zhang. Rocm: Rlhf on con- sistency models. arXiv preprint arXiv:2503.06171, 2025. 1, 2, 3 [33] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Elicit- ing knowledge from language models with automatically generated prompts, 2020. 6 [34] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming.Advances in Neural Information Processing Systems, 35:9460–9471, 2022. 1, 3 [35] Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Evaluat- ing text-to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annotations. arXiv preprint arXiv:2406.16562, 2024. 2, 3, 10 [36] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. 2 [37] Jiarui Wang, Huiyu Duan, Yu Zhao, Juntong Wang, Guangtao Zhai, and Xiongkuo Min. Lmm4lmm: Bench- marking and evaluating large-multimodal image genera- tion with lmms. arXiv preprint arXiv:2504.08358, 2025. 2, 7, 8, 3, 9 [38] Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understand- ing and generation. arXiv preprint arXiv:2503.05236, 2025. 1, 2, 3, 6 [39] Hongyang Wei, Baixin Xu, Hongbo Liu, Cyrus Wu, Jie Liu, Yi Peng, Peiyu Wang, Zexiang Liu, Jingwen He, Yi- dan Xietian, et al. Skywork unipic 2.0: Building kontext model with online rl for unified multimodal model. arXiv preprint arXiv:2509.04548, 2025. 1 [40] Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17090, 2023. 1, 2 [41] Jie Wu, Yu Gao, Zilyu Ye, Ming Li, Liang Li, Hanzhong Guo, Jie Liu, Zeyue Xue, Xiaoxia Hou, Wei Liu, et al. Rewarddance:Reward scaling in visual generation. arXiv preprint arXiv:2509.08826, 2025. 1, 2 [42] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human prefer- ence score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023. 1, 2, 3, 4, 6, 7 10 [43] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Im- agereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Infor- mation Processing Systems, 36:15903–15935, 2023. 1, 2, 3, 6 [44] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818, 2025. 1, 2 [45] Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 1191–1200, 2022. 1, 2, 6 [46] Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress ac- curate image quality scores using score distribution. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pages 14483–14494, 2025. 1, 2, 3, 6 [47] Shihao Yuan, Yahui Liu, Yang Yue, Jingyuan Zhang, Wangmeng Zuo, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Ar-grpo: Training autoregressive image genera- tion models via reinforcement learning. arXiv preprint arXiv:2508.06924, 2025. 1, 2 [48] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text gener- ation with bert. arXiv preprint arXiv:1904.09675, 2019. 2 [49] Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. In European Conference on Computer Vision, pages 1– 17. Springer, 2024. 1, 2, 3 [50] Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu, Branislav Kveton, Yufan Zhou, Jiuxiang Gu, Jian Chen, and Changyou Chen. Multimodal llms as customized reward models for text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19638–19648, 2025. 1, 2 [51] Xingyi Zhou, Vladlen Koltun, and Philipp Kr ̈ ahenb ̈ uhl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7571–7580, 2022. 1, 2, 3 [52] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engi- neers, 2023. 6 11 Understanding Reward Hacking in Text-to-Image Reinforcement Learning Supplementary Material 6. Automatic Prompt Optimization for Arti- factReward We employ automatic prompt optimization (APO) [28] to optimize the prompt used for ArtifactReward. Qwen2.5-VL-7B-Instruct [2] serves as our backbone vision-language model, and APO is used to optimize the prompt so that the model can more reliably distinguish artifact-containing images from artifact-free ones. Our initial prompt is given as: Initial prompt. Is there any artifacts in the image that look not realistic? We compute the ArtifactReward using the normal- ized probability of the model answering “no”, where a higher “no” probability indicates a higher likelihood of being artifact-free (See Eq. 1). During optimization, we maximize the following log-likelihood objective: L = ylog(p yes ) + (1− y)log(1− p yes ),(2) where y is the ground-truth label, and p yes = 1− p no de- notes the normalized probability assigned to the “yes” token. Algorithm 1 provides the complete APO proce- dure, with the scoring function S(·) defined as the log- likelihood above. An example of the optimized prompt is: Optimized prompt. Analyze the images to identify any unintentional digital artifacts, concentrating on irregular lighting, object placement, blending errors, or anomalies that could affect realism. Disregard any deliberate artistic styles or intentional surreal elements. Respond with YES if artifacts are detected; if not, respond with NO. This optimized prompt is more explicit and struc- turally aligned with common artifact patterns, enabling the model to produce more accurate and reliable arti- fact predictions. As shown in Table 3, our ArtifactRe- ward consistently assigns higher scores to artifact-free images, clearly separating clean generations from those with distortions. This demonstrates the effectiveness of our optimized prompt in enabling a reward model that can reliably detect and penalize generated artifacts. Algorithm 1 Automatic Prompt Optimization for Arti- factReward Require: p 0 : initial prompt, N : iterations,D = (x,y): train- ing dataset where x is the input image and y is the artifact label, b: top prompts retained per iteration, l: number of error examples for reflections, S(·): scoring function 1: P 0 ←p 0 ▷ Initialize candidate prompt set 2: for t = 1 to N do 3: P c ← P t−1 4:for p∈ P t−1 do 5:J error = (x i ,y i ) | p i yes > 0.5 if y i = 0, or p i yes < 0.5 if y i = 1 6:J i error ⊂ J error is a sampled subset for each i = 1,...,l 7:G = S i=1,...l Reflect(p,J i error ) 8:H = S i=1,...l Modify(p,g i ,J i error ) 9:P c ← P c ∪ H 10:end for 11: S c =S(p)| p∈ P c ▷ Evaluate prompts 12: P t ←p∈ P c | S(p)≥ τ, where τ is the b th highest score inS c 13: end for 14: Return p ∗ ← arg max p∈P N S(p) 7. Additional Experiment Results 7.1. Additional Training Dynamics on Janus- Pro-7B We conducted additional experiments using the larger 7B variant of Janus-Pro [4]. The training dynamics of different evaluation metrics are shown in Figure 4. Over- all, we observe trends similar to those seen with the 1B model. Optimization under a given reward tends to improve metrics with closely related inductive bi- ases, while degrading performance on metrics that cap- ture different aspects of image quality. For example, the model trained with the GDino reward shows in- creased GDino and ORM scores, as both scores empha- size object recognition in generated images. In contrast, the model fails to improve HPS, Aesthetic Score, and DeQA, which focus more on color quality and visual ap- peal. Moreover, across all reward settings, we find that none effectively reduce artifacts in the generated images (Figures 4i–4l). 7.2. Detailed Results of WISE benchmark Figure 5 presents the full WISE [26] benchmark re- sults across different prompt subcategories and evalu- ation metrics for Janus-Pro-1B [4] trained with vari- ous reward configurations. Across nearly all categories, 1 0100020003000 Steps 0.27 0.28 0.29 0.30 0.31 0.32 (a) HPS - Spatial 0100020003000 Steps 5.80 6.00 6.20 6.40 6.60 6.80 (b) Aesthetic Score - Color 0100020003000 Steps 3.55 3.60 3.65 3.70 3.75 3.80 3.85 3.90 (c) DeQA - Spatial 0100020003000 Steps 0.21 0.22 0.22 0.22 0.23 0.23 0.23 0.23 (d) PickScore - Object 0100020003000 Steps 0.84 0.86 0.88 0.90 0.92 0.94 0.96 0.98 (e) GDino - Texture 0100020003000 Steps 0.30 0.40 0.50 0.60 0.70 0.80 (f) ORM - Spatial 0100020003000 Steps 0.66 0.68 0.70 0.72 0.74 0.76 0.78 0.80 0.82 (g) VQAScore - Numeracy 0100020003000 Steps 2.60 2.70 2.80 2.90 3.00 3.10 (h) Unified - Numeracy 0100020003000 Steps 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 (i) Artifacts - Object 0100020003000 Steps 0.35 0.40 0.45 0.50 0.55 (j) Artifacts - Shape 0100020003000 Steps 0.25 0.30 0.35 0.40 0.45 (k) Artifacts - Spatial 0100020003000 Steps 0.35 0.40 0.45 0.50 0.55 0.60 0.65 (l) Artifacts - Texture Figure 4. Evolution of metrics over training steps with different categories of prompts trained on Janus-Pro-7B [4]. Blue color denotes the model trained with HPS [42]; Orange color denotes the model trained with GDino [25]; Green color denotes the model trained with HPS and GDino; Red color denotes the model trained with finetuned ORM [14]. Table 6. Performance on WISE [26] benchmark trained on Janus-Pro-7B [4]. WiScore = 0.7× Consistency + 0.2× Re- alism + 0.1× Aesthetic by the default setting. MethodConsistencyRealismAestheticWiScore Janus-Pro-7B [4]0.43400.54700.55280.4685 HPS [42]0.41600.47450.65300.4514 HPS + Artifact 0.42550.66050.58150.4881 GDino [25]0.44450.51700.54350.4689 GDino + Artifact0.44700.62950.54650.4935 ORM [14]0.44200.58000.59750.4852 ORM + Artifact0.46650.63550.58400.5121 HPS + GDino0.47800.56970.67550.5161 HPS + GDino + Artifact0.47750.64700.58800.5225 T2I-R1 [18]0.46550.59250.67300.5157 T2I-R1 + Artifact0.49450.64500.60000.5352 incorporating our ArtifactReward leads to substantial improvements in image realism, consistently outper- forming models trained with baseline rewards. In ad- dition, ArtifactReward enhances image consistency in most subcategories, indicating that reducing structural artifacts not only improves visual plausibility but also strengthens text-image alignment across diverse prompt types. More image illustrations can be found in Figure 7 and Figure 8. We additionally evaluate our method on the larger Janus-Pro-7B [4] model. As shown in Table 6 and Fig- ure 6, the results mirror those observed with the 1B model, demonstrating that ArtifactReward generalizes effectively across model scales. 7.3. Detailed Results of LLM4LLM benchmark Figure 9 presents the LLM4LLM [37] benchmark re- sults across different prompt subcategories and evalua- tion metrics for Janus-Pro-1B [4] trained with various reward configurations. More image illustrations can be found in Figure 11 and Figure 12. Additional results trained on Janus-Pro-7B [4] are shown in Table 7 and Figure 10. 2 Table 7. Performance on LLM4LLM [37] benchmark trained on Janus-Pro-7B [4]. MethodPerceptionCorrespondenceAll Janus-Pro-7B [4]0.43900.54160.9806 HPS [42]0.43600.52050.9564 HPS + Artifact 0.44540.52780.9732 GDino [25]0.44170.55700.9987 GDino + Artifact 0.45060.55191.0024 ORM [14]0.44990.55611.0059 ORM + Artifact0.45760.56131.0188 HPS + GDino0.46150.55401.0156 HPS + GDino + Artifact0.45580.55941.0152 T2I-R1 [18]0.45800.55591.0139 T2I-R1 + Artifact 0.46080.56221.0230 Table 8. Performance on EvalAlign [35] benchmark trained on Janus-Pro [4]. Method Faithfulness 1B7B Janus-Pro [4]0.76420.8694 HPS [42]0.75690.8538 HPS + Artifact0.93021.0676 GDino [25]0.78320.8956 GDino + Artifact0.89520.9837 ORM [14]0.75600.8607 ORM + Artifact0.91400.9245 HPS + GDino0.78400.8956 HPS + GDino + Artifact0.87560.9837 T2I-R1 [18]0.76920.9059 T2I-R1 + Artifact0.90150.9638 7.4. Detailed Results of EVALALIGN bench- mark We additionally evaluate our method on EVALALIGN [35], a fine-grained, MLLM-based evaluation bench- mark for text-to-image models. It uses image faithful- ness, how accurately the visual content matches real- ity or semantics, as one of their image quality measure- ments. To highlight the impact of our method on reduc- ing such artifacts, we focus on this faithfulness metric. As shown in Table 8, Figure 13 and 14, incorporat- ing our ArtifactReward consistently leads to substantial improvements, achieving higher scores across nearly all subcategories. These results further confirm that Arti- factReward can enhance semantic plausibility and struc- tural correctness beyond what existing reward models capture. 3 CulturalTimeSpaceBiologyPhysicsChemistry 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (a) Realism CulturalTimeSpaceBiologyPhysicsChemistry 0.0 0.1 0.2 0.3 0.4 0.5 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (b) Consistency CulturalTimeSpaceBiologyPhysicsChemistry 0.0 0.1 0.2 0.3 0.4 0.5 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (c) WiScore Figure 5. Performance on WISE [26] benchmark across different categories trained on Janus-Pro-1B [4]. 4 CulturalTimeSpaceBiologyPhysicsChemistry 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (a) Realism CulturalTimeSpaceBiologyPhysicsChemistry 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (b) Consistency CulturalTimeSpaceBiologyPhysicsChemistry 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (c) WiScore Figure 6. Performance on WISE [26] benchmark across different categories trained on Janus-Pro-7B [4] 5 (a) HPS(b) HPS + Artifact(c) GDino(d) GDino + Artifact (e) HPS + GDino(f) HPS + GDino + Artifact(g) ORM(h) ORM + Artifact Figure 7. Images generated with prompt “An iconic bridge, known for its red hue and location over a famous bay in San Francisco” in WISE [26] benchmark under different training reward configurations trained on Janus-Pro-1B [4]. (a) HPS(b) HPS + Artifact(c) GDino(d) GDino + Artifact (e) HPS + GDino(f) HPS + GDino + Artifact(g) ORM(h) ORM + Artifact Figure 8. Images generated with prompt “Siblings of identical twins” in WISE [26] benchmark under different training reward configurations trained on Janus-Pro-1B [4]. 6 HumanLinguisticCountingPositionTextureScene 0.0 0.1 0.2 0.3 0.4 0.5 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (a) Perception HumanLinguisticCountingPositionTextureScene 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (b) Correspondence Figure 9. Performance on LLM4LLM [37] benchmark across different categories trained on Janus-Pro-1B [4] 7 HumanLinguisticCountingPositionTextureScene 0.0 0.1 0.2 0.3 0.4 0.5 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (a) Perception HumanLinguisticCountingPositionTextureScene 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward (b) Correspondence Figure 10. Performance on LLM4LLM [37] benchmark across different categories trained on Janus-Pro-7B [4] 8 (a) HPS(b) HPS + Artifact(c) GDino(d) GDino + Artifact (e) HPS + GDino(f) HPS + GDino + Artifact(g) ORM(h) ORM + Artifact Figure 11. Images generated with prompt “A teacher in a white blouse stands at the blackboard, her curly brown hair tied back in a ponytail.” in LLM4LLM [37] benchmark under different training reward configurations trained on Janus-Pro-1B [4]. (a) HPS(b) HPS + Artifact(c) GDino(d) GDino + Artifact (e) HPS + GDino(f) HPS + GDino + Artifact(g) ORM(h) ORM + Artifact Figure 12. Images generated with prompt “a dog is smiling with happy emotion. He find a lot of delicious food.” in LLM4LLM [37] benchmark under different training reward configurations trained on Janus-Pro-1B [4]. 9 BodyHandFaceObjectCommon 0.0 0.5 1.0 1.5 2.0 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward Figure 13. Performance on EvalAlign [35] benchmark across different categories trained on Janus-Pro-1B [4]. BodyHandFaceObjectCommon 0.0 0.5 1.0 1.5 2.0 Score Base HPS HPS + ArtifactReward GDino GDino + ArtifactReward ORM ORM + ArtifactReward HPS + GDino HPS + GDino + ArtifactReward T2I-R1 T2I-R1 + ArtifactReward Figure 14. Performance on EvalAlign [35] benchmark across different categories trained on Janus-Pro-7B [4]. 10