Paper deep dive
Personalize at Test Time: Learning User Preferences for Image Generation
Jiamu Bai, Jiaming Hu, Yanhong Wu, Zellux Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/8/2026, 6:42:54 AM
Summary
The paper introduces PAT (Personalized at Test Time), a method for aligning diffusion models with individual user preferences without fine-tuning the model. It uses an autoencoder to compress visual attributes into 50 shared dimensions, trains an evaluator to score images on these dimensions, and estimates user-specific weights via the Bradley-Terry model using pairwise preference feedback. This allows for data-efficient personalization at inference time using Direct Noise Optimization.
Entities (8)
Relation Signals (7)
PAT → uses → Autoencoder
confidence 95% · First, we use an autoencoder to compress hundreds of visual attributes into 50 attribute-anchored preference dimensions
PAT → uses → Bradley-Terry Model
confidence 95% · estimating the user-specific weights by maximizing the likelihood of their observed pairwise preferences under the Bradley-Terry model.
Autoencoder → reducesdimensionalityof → Visual Attributes
confidence 90% · compress hundreds of visual attributes into 50 attribute-anchored preference dimensions
PAT → uses → Direct Noise Optimization
confidence 90% · we use Direct Noise Optimization (DNO) to guide generation without updating the diffusion model.
Qwen2.5-VL-72B → annotates → Pick-a-Pic v1
confidence 85% · we annotate the pool with Qwen2.5-VL-72B using the ViPer attribute taxonomy
SigLIP → encodes → Prompt and Image
confidence 85% · A frozen SigLIP model encodes the caption and image
PAT → achievesaccuracyon → Pick-a-Pic v1
confidence 80% · Experiments on real-user preference data show that our approach achieves approximately 77% held-out pairwise preference prediction accuracy
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion models can generate high-quality images, yet aligning their outputs with individual user preferences remains challenging. A key bottleneck is accurately modeling diverse user preferences from limited feedback. Existing approaches often rely on labor-intensive manual preference annotations or vision-language models (VLM) to extract preference information from user interaction histories, introducing substantial annotation or computational costs that limit scalability. We propose an approach that learns personalized reward models directly from users' historical image preference pairs. First, we use an autoencoder to compress hundreds of visual attributes into 50 attribute-anchored preference dimensions and train an evaluator to score images along these dimensions. We then represent each user's preferences as a linear combination of the shared dimension scores, estimating the user-specific weights by maximizing the likelihood of their observed pairwise preferences under the Bradley-Terry model. This formulation reduces per-user adaptation to optimizing a low-dimensional weight vector, simplifying optimization and enabling data-efficient personalization from sparse feedback. The learned personalized rewards guide image generation at inference time while keeping the diffusion model frozen. Experiments on real-user preference data show that our approach achieves approximately 77% held-out pairwise preference prediction accuracy and improves the alignment of generated images with individual user preferences.
Tags
Links
- Source: https://arxiv.org/abs/2610.09015v1
- Canonical: https://arxiv.org/abs/2610.09015v1
Trouble viewing inline? Open PDF directly →
Full Text
74,133 characters extracted from source content.
Expand or collapse full text
Personalize at Test Time: Learning User Preferences for Image Generation Jiamu Bai† †thanks: Work done during Jiamu Bai’s internship at Meta. $†$: co-first author. Affiliation: Penn State University Jiaming Hu† Affiliation: Boston University Yanhong Wu Affiliation: Meta Inc. Zellux Wang Affiliation: Meta Inc. Abstract Diffusion models can generate high-quality images, yet aligning their outputs with individual user preferences remains challenging. A key bottleneck is accurately modeling diverse user preferences from limited feedback. Existing approaches often rely on labor-intensive manual preference annotations or vision-language models(VLM) to extract preference information from user interaction histories, introducing substantial annotation or computational costs that limit scalability. We propose an approach that learns personalized reward models directly from users’ historical image preference pairs. First, we use an autoencoder to compress hundreds of visual attributes into 50 attribute-anchored preference dimensions and train an evaluator to score images along these dimensions. We then represent each user’s preferences as a linear combination of the shared dimension scores, estimating the user-specific weights by maximizing the likelihood of their observed pairwise preferences under the Bradley–Terry model. This formulation reduces per-user adaptation to optimizing a low-dimensional weight vector, simplifying optimization and enabling data-efficient personalization from sparse feedback. The learned personalized rewards guide image generation at inference time while keeping the diffusion model frozen. Experiments on real-user preference data show that our approach achieves approximately 77% held-out pairwise preference prediction accuracy and improves the alignment of generated images with individual user preferences. 1 Introduction Text-to-image diffusion models have made remarkable progress in visual fidelity, prompt adherence, and compositional generation, powering increasingly capable systems such as Stable Diffusion (Podell et al., 2024; Rombach et al., 2022), DALL-E (Ramesh et al., 2021), and Seedream (Seedream et al., 2025). To adapt these general-purpose models to downstream requirements, prior work has explored supervised or reinforcement-learning-based fine-tuning, the injection of additional conditioning signals, and reward-guided inference (Wallace et al., 2024; Prabhudesai et al., 2023). These approaches have proven effective for applications including subject-driven generation, style customization, and general preference alignment. Despite their effectiveness in many applications, personalizing diffusion models to individual users remains challenging, as it requires accurately modeling each user’s preferences from limited feedback. Existing approaches generally obtain user preferences in two ways: using vision-language models to summarize a user’s historical preference records into a profile or embedding (Dang et al., 2025; Kim et al., 2025a), or collecting explicit human annotations of users’ likes and dislikes and converting them into structured visual preferences to guide generation (Salehi et al., 2024). These approaches have two important limitations. First, incorporating preference representations requires additional fine-tuning of the diffusion model or training of a specialized adapter, introducing substantial computational cost and limiting scalability across different generative models. Second, acquiring reliable preference representations remains difficult: profiles inferred by vision-language models may be inaccurate when user histories are short, long, or heterogeneous, while collecting detailed human annotations imposes considerable burden on users. In this work, we propose PAT(Personalized at Test Time for diffusion models), which instantiates a personalized reward model at test time directly from a user’s historical preference feedback. For each pair of images generated for a prompt, a user only needs to indicate which image they prefer. Our approach assumes that user preferences admit a low-dimensional latent representation, with individual differences captured by the weights each user assigns to shared preference dimensions. We therefore formulate personalization as a two-stage problem: first, identifying a compact, shared visual attribute space that captures diverse preference-relevant attributes; and second, quantifying each user’s preferences along these dimensions. To identify the shared preference dimensions, we draw on the collection of over 500 visual attributes compiled by Salehi et al. (2024). While these attributes provide broad coverage of visual preferences, estimating a separate weight for every attribute would require fitting hundreds of user-specific parameters from only a small number of pairwise observations, making preference estimation prone to overfitting. We therefore seek a compact shared representation that preserves meaningful variation across visual attributes while reducing the number of parameters to be estimated for each user. To this end, we use an autoencoder to derive a low-dimensional set of shared preference dimensions grounded in these attributes. Unlike covariance-based linear methods such as PCA (Pearson, 1901; Hotelling, 1933) and factor analysis(Spearman, 1961; Thurstone, 1931), the autoencoder can capture a nonlinear low-dimensional representation that better preserves interactions among visual attributes. The advantage is supported by our empirical results in Table 4. An evaluator then maps a prompt–image pair (c,x)(c,x) to its scores sθ(c,x)∈ℝds_θ(c,x) ^d along these dimensions, providing a shared basis for evaluating the visual characteristics of images. To capture each user’s individual preferences, we construct a personalized reward model that can be efficiently learned at test time. We quantify the reward of user u as a linear combination ru(c,x)=λu⊤sθ(c,x)r_u(c,x)= _u s_θ(c,x), where λu _u is the user-specific preference weight. Given the user’s preference history represented as a collection of n prompt–image preference pairs (ck,xk+,xk−)1n(c_k,x_k^+,x_k^-)_1^n, we estimate the user-specific weights using the Bradley–Terry objective, λu∗=argmax∑k=1muλulogσ(λu⊤[sθ(ck,xk+)−sθ(ck,xk−)]). _u^*= _ _u _k=1^m_u σ ( _u [s_θ(c_k,x_k^+)-s_θ(c_k,x_k^-) ] ). (1) This factorization reduces test-time personalization to estimating a low-dimensional weight vector, keeping per-user optimization compact and enabling data-efficient learning from sparse feedback. The resulting reward is used to guide generation at inference time while the underlying diffusion model remains frozen. To improve generalization for users with very few preference pairs, we further introduce a mixed-effects parameterization λu=G+bu _u=G+b_u, where G captures population-level preferences shared across users and bub_u captures user-specific deviations. By pooling preference information through G, this formulation further enables more reliable personalization in few-shot settings. We evaluate our approach on real-user preference data, achieving approximately 78%78\% held-out pairwise preference prediction accuracy. When transferred to personalized image generation, our method achieves a 63.0%63.0\% win rate against the non-personalized generator. Our main contributions are as follows: • We introduce the first mathematical formulation of user profiling for personalized image generation based directly on offline pairwise preferences. Our formulation represents each user by a low-dimensional preference vector over shared visual dimensions, turning personalization into a tractable parameter-estimation problem. • We propose PAT, an end-to-end pipeline that fits a compact personalized reward and uses it to guide a frozen diffusion model through test-time noise optimization, requiring neither per-user model fine-tuning nor separate personalized generators. • We empirically demonstrate that our approach accurately models real users’ preferences under limited feedback and that the learned personalized rewards transfer effectively from preference prediction to personalized image generation. Figure 1: Overview of PAT. We learn a shared low-dimensional visual preference space and evaluator, fit a personalized reward from each user’s pairwise preference history, and use the reward to guide a frozen diffusion model at inference time. 2 Related Work Diffusion Model Alignment. A growing body of work aligns pretrained diffusion models with downstream objectives such as human preferences, aesthetic quality, and safety-awareness. One class of approaches fine-tunes the diffusion model using reward supervision, thereby incorporating the desired objective into the model parameters (Dai et al., 2023; Betker et al., 2023; Clark et al., 2023; Prabhudesai et al., 2023; Fan et al., 2023; Black et al., 2023). Another class performs alignment at inference time, steering the denoising trajectory or directly optimizing the latent variables while keeping the pretrained diffusion model frozen (Jiang et al., 2026; Goyal et al., 2026; Tang et al., 2024; Yeh et al., 2024; Kim et al., ; Zhai et al., 2025). More recently, reinforcement learning has been introduced to diffusion model alignment by treating denoising as a sequential decision-making process. Existing methods employ policy-optimization algorithms such as GRPO (Zheng et al., 2026; Liu et al., 2026; Xue et al., 2025; Li et al., 2025a; Li et al., 2025b; Li et al., 2026c), as well as RLHF and direct preference optimization objectives such as DPO (Wallace et al., 2024; Jiang et al., 2026; Liang et al., 2024; Wang et al., 2025; Lee et al., 2025b; Gu et al., 2024; Yang et al., 2024a; Zhu et al., 2025), to improve generation according to human feedback or learned rewards. While most of these methods perform alignment by updating the diffusion model during training, a small number of concurrent works, besides ours, have also explored alignment at inference time. Jin et al. (2025) and Xie et al. (2026) investigate training-free inference-time reward transfer from RL aligned-base difference between tuned model and base model. Chen et al. (2026) treats intermediate flow latents as implicit policies and optimizes them at test time using PRDP and GRPO objectives. Personalization for Generative Models. Personalization has been extensively studied for large language models through approaches such as user-profile conditioning, memory augmentation, parameter-efficient adaptation, and personalized reward modeling (Yang et al., 2024b; Li et al., 2026a; Poddar et al., 2024; Wang et al., 2024; Song et al., 2026; Yu et al., 2026; Wang et al., 2026a; Wu et al., 2026; Tan et al., 2026; Barreto et al., 2025; Cai et al., 2026; Iyer et al., ; Chen et al., 2025; Shenfeld et al., 2025; Kim et al., 2025b). Personalizing diffusion models poses additional challenges. An LLM can jointly interpret heterogeneous user histories, extract preference-relevant information, and incorporate the resulting profile into generation through natural-language conditioning. In contrast, a diffusion model primarily serves as a generator and does not natively provide a comparable interface for reasoning over user histories or constructing explicit user models. Existing approaches therefore commonly rely on external vision-language models to summarize a user’s prior choices or encode them into preference representations, while others use human-defined attributes or manual annotations to characterize the relevant dimensions of variation (Dang et al., 2025; Lee et al., 2025a; Salehi et al., 2024; Kim et al., 2025a; Wang et al., 2026b; Li et al., 2026b). Our approach makes diffusion personalization more scalable by directly learning shared preference dimensions and user-specific rewards from pairwise feedback. It requires neither external VLM-based nor manual analysis of each user’s history, and personalizes generation at inference time without fine-tuning the underlying diffusion model. 3 Methodology 3.1 Problem Formulation For each user u, we observe a small set of offline preference pairs, u=(ci,xi+,xi−)i=1muD_u=\(c_i,x_i^+,x_i^-)\_i=1^m_u, where in i-th image pair, cic_i is a text prompt and xi+x_i^+ is preferred to xi−x_i^-. We assume that user preferences lie in a shared k-dimensional space, represented by an evaluator s(c,x)∈ℝks(c,x) ^k. The personalized reward is a linear combination ru(c,x)=λu⊤s(c,x),λu∈ℝk,r_u(c,x)= _u s(c,x), _u ^k, (2) where λu _u is the only user-specific parameter. At inference time, we aim to generate images maximizing the personal reward ru(c,x)r_u(c,x). 3.2 Learning an AutoEncoder Preference Evaluator Preference Dataset. We construct the data for learning the shared preference evaluator from the training split of Pick-a-Pic v1 (Kirstain et al., 2023), where each comparison consists of a text prompt c, two generated images (x+,x−)(x^+,x^-), and the image preferred by the user. We sample a dimension-learning pool of 50,000 images from light raters. To obtain structured supervision describing the visual properties of each image, we annotate the pool with Qwen2.5-VL-72B (Bai et al., 2025) using the ViPer attribute taxonomy (Salehi et al., 2024). The taxonomy contains approximately 512 attributes organized into 21 visual categories, including art style, color palette, composition, and mood. For each image, Qwen2.5-VL-72B assigns a binary label indicating whether an attribute is present in the image. Aggregating these annotations produces a binary attribute matrix with size 50,000×51250,000× 512, which is subsequently used to train the autoencoder. Autoencoder for dimensionality reduction. Assigning a separate preference weight to each of the 512 attributes would require estimating a high-dimensional user vector from limited feedback, making reliable estimation difficult. We therefore learn a shared representation with k≪512k 512 dimensions, reducing the number of user-specific parameters to estimate. The bottleneck dimension k controls the trade-off between representing diverse preferences and learning them from sparse feedback. To effectively reduce dimension, we use an autoencoder to capture potentially nonlinear attribute dependencies. Specifically, let ai∈0,1512a_i∈\0,1\^512 denote the attribute vector of image xix_i in the annotated pool of N=50,000N=50,000 images. An encoder EηE_η produces a latent representation, and a decoder DψD_ψ reconstructs the attribute probabilities: zi=Eη(ai)∈ℝk,a^i=Dψ(zi)∈(0,1)512.z_i=E_η(a_i) ^k, a_i=D_ψ(z_i)∈(0,1)^512. (3) We learn the encoder and decoder parameters (η,ψ)(η,ψ) by minimizing ℒAE=1N∑i=1Nℓrec(ai,a^i),L_AE= 1N _i=1^N _rec\! (a_i, a_i ), (4) where ℓrec _rec is the binary cross-entropy loss across attributes. After training, we discard the decoder and use ziz_i as shared supervision targets for the evaluator sθ(ci,xi)s_θ(c_i,x_i). These compact scores encode visual attribute structure rather than user preferences; each user’s preferences are subsequently learned through the weights λu _u over the evaluator’s k outputs. Evaluator for each dimension. We train a differentiable evaluator sθ(c,x)∈ℝks_θ(c,x) ^k that combines general human preference learning with dimension-specific visual attribute supervision. A frozen SigLIP model encodes the caption and image, whose representations are concatenated and passed to MLP scoring heads with output dimension of k. Following the pairwise preference-learning principle of PickScore (Kirstain et al., 2023), each head learns to assign higher scores to human-preferred images. We additionally supervise each head using the corresponding autoencoder latent dimension: for each dimension, we supervise the top 33% of pairs by absolute latent difference using the sign of that difference as the ordering target, then apply an attribute-ranking loss only to image pairs with different binary labels in that dimension. This auxiliary objective encourages the heads to retain distinct visual attribute signals while learning general preferences. We train the scoring heads on around 36k image pairs while keeping the SigLIP encoders and autoencoder fixed. The resulting scores form a shared basis for personalized reward fitting. Details are provided in Appendix A. 3.3 Personalized Reward Learning Given the shared evaluator and user u’s offline preference pairs uD_u, we infer the weight vector λu _u that best explains the user’s choices. Under the Bradley–Terry model, the probability of preferring xi+x_i^+ to xi−x_i^- is Pu(xi+≻xi−∣ci)=σ(ru(ci,xi+)−ru(ci,xi−))=σ(λu⊤Δsi),P_u(x_i^+ x_i^- c_i)=σ\! (r_u(c_i,x_i^+)-r_u(c_i,x_i^-) )=σ\! ( _u s_i ), (5) where σ(a)=(1+e−a)−1σ(a)=(1+e^-a)^-1 and Δsi=s(ci,xi+)−s(ci,xi−) s_i=s(c_i,x_i^+)-s(c_i,x_i^-). The basic regularized maximum-likelihood estimator is λ^u=argmaxλ∈ℝk∑i=1mulogσ(λ⊤Δsi)−γ‖λ‖22, λ_u= _λ ^k \ _i=1^m_u σ(λ s_i)-γ\|λ\|_2^2 \, (6) where γ controls regularization and mum_u is the number of total preference pairs for user u. This formulation reduces personalization to estimating a low-dimensional vector, but its effectiveness still depends on the shared evaluator providing meaningful preference coordinates. Alternating evaluator and weight optimization. The collected image preference pairs can further train the shared evaluator to better reflect preferences across users. We alternate between fitting each user’s λu _u with the evaluator fixed and updating the evaluator heads with the user weights fixed. Repeating these two steps allows the shared preference representation and user-specific weights to adapt together, while the evaluator’s image and text encoders and the diffusion model remain frozen. Mixed-effects preference modeling. We further extend Equation 6 to a mixed-effect model, parameterizing each user’s weights as λu=G+bu _u=G+b_u. Here, G∈ℝkG ^k is a shared fixed effect capturing population-level preferences, and bu∈ℝkb_u ^k is a user-specific random effect capturing deviations from this shared component. With an isotropic Gaussian prior bu∼(0,I)b_u (0,I), we jointly estimate the shared component and user deviations by maximizing the regularized log-likelihood: (G^,b^u)=argmaxG,bu∑u∈train∑i=1mulogσ((G+bu)⊤Δsu,i)−γb∑u∈train‖bu‖22, ( G,\ b_u\)= _G,\b_u\\; _u _train _i=1^m_u σ\! ((G+b_u) s_u,i )- _b _u _train\|b_u\|_2^2, (7) where trainU_train denotes the set of training users, and γb _b controls the allowed user-level variation, and mum_u is the number of total preference pairs for user u. This mixed-effect model is particularly useful when each user provides only a few preference pairs. The shared component G is learned across users, allowing data-sparse users to benefit from population-level evidence, while the penalty on bub_u limits unsupported individual deviations. This can reduce the variance of user-specific estimates and improve preference prediction under limited feedback. 3.4 Inference-Time Generation with Direct Noise Optimization After learning the personalized reward rur_u, we use Direct Noise Optimization (DNO) (Tang et al., 2024) to guide generation without updating the diffusion model. Let Mϕ(c,xT)M_φ(c,x_T) denote the image produced by the frozen diffusion model ϕφ from prompt c and initial noise xT∼(0,I)x_T (0,I). Given a user’s personal reward model ru(c,x)r_u(c,x), we aim to find the locally optimal noise xT⋆x_T such that xT⋆=argmaxxTru(c,Mϕ(c,xT)).x_T = _x_Tr_u(c,M_φ(c,x_T)). The optimal noise is derived from gradient ascent. At the k-th optimization step, the k+1k+1 noise xTk+1x_T^k+1 is updated by, xTk+1:=xTk+ℓ∇xTru(c,Mϕ(c,xTk)).x_T^k+1:=x_T^k+ _x_Tr_u(c,M_φ(c,x_T^k)). (8) Note that while Tang et al. (2024) mainly uses an SDE sampler and optimizes over both initial noise and step-wise injected noises, we use ODE sampler thus only the initial noise is optimized. Additional implementation details are provided in Appendix D. 4 Experiments Our experiments evaluate two central questions: (1) whether the learned personalized reward accurately represents a user’s preferences, and (2) whether this reward can be used to generate images that match those preferences. We first study a controlled toy setting with manually specified synthetic preference profiles, and then extend the evaluation to real user preferences from Pick-a-Pic. 4.1 Toy Example Setting Goal, data, and evaluation. We begin with a controlled toy experiment involving five synthetic users, each represented by a manually specified ground-truth preference vector λ⋆λ (Table 5). Their preferences are defined over five candidate reward dimensions: clip (Radford et al., 2021), aes (Schuhmann, 2022), color (Hasler and Suesstrunk, 2003), hps (Wu et al., 2023), and pickscore (Kirstain et al., 2023). These scores capture complementary aspects of visual appearance and text–image alignment; details are provided in Appendix B.1. We repeat the experiment three times, sampling 200 captioned image pairs from Pick-a-Pic v1 in each repetition and splitting them into 100 training and 100 test pairs. After standardizing each reward dimension over the sampled images, we construct synthetic preference labels by selecting the image with the higher ground-truth weighted reward, (λ⋆)⊤r(c,x)(λ ) r(c,x). We then estimate λ from the training preferences by maximizing the Bradley–Terry likelihood with regularization 0.1. Evaluation considers the preference prediction accuracy on the held-out pairs. We compare the learned multi-dimensional reward with the ground-truth reward and a best one-hot baseline, which uses the highest-performing individual reward dimension available in each configuration. This comparison tests whether learning a combination of reward signals better captures a user’s preferences than relying on a single evaluator. Table 1: Held-out pairwise preference prediction accuracy(%) for the synthetic configurations. Best one-hot uses the strongest individual reward dimension. The learned combination achieves higher accuracy. Config PAT Best one-hot Uniform 90.0 ± 4.3 77.0 ± 1.6 Normal-1 90.7 ± 1.2 80.3 ± 3.3 Normal-2 88.7 ± 3.6 79.0 ± 3.7 Heavy-1 91.0 ± 5.7 81.7 ± 4.8 Heavy-2 90.7 ± 4.4 81.0 ± 5.4 Table 2: Held-out pairwise preference accuracy (%) on 631 Pick-a-Pic users, grouped by the number of training pairs. Each user has 50 test pairs. Bold indicates the best accuracy in each group. Training pairs 11–20 21–30 31–60 61++ PAT 73.1 ± 1.3 74.6 ± 0.9 77.1 ± 0.3 78.2 ± 0.1 w/o Mixed Effect 73.1 ± 1.4 73.5 ± 0.9 76.4 ± 0.4 77.9 ± 0.1 w/o Alternating Training 71.2± 1.1 72.3± 0.9 75.2± 0.7 75.4± 0.2 w/o Alternating Training & Mixed Effect 71.0± 1.0 71.6± 0.7 74.7± 0.7 74.7± 0.3 Universal λ 67.6± 0.4 66.9± 0.6 69.3± 0.2 70.3± 0.3 w/ Frozen Evaluator 66.9± 0.4 65.3± 0.6 65.1± 0.8 66.0± 0.4 Results. As shown in Table 2, our learned reward combination outperforms the best single reward dimension across all five synthetic preference configurations, achieving around 90% held-out accuracy. These results demonstrate that fitting user-specific weights from limited pairwise feedback better captures diverse preference profiles than relying on any individual metric. Figure 2 qualitatively illustrates how different preference weights affect generation under the same caption. Increasing the weight on colorfulness produces more vivid, saturated images, while other weight configurations yield more muted palettes and different architectural compositions. These examples show that varying the preference weights enables different visual styles without changing the text prompt. Figure 2: Qualitative examples of preference-controlled generation under a fixed prompt. Each image is generated using a different weight vector λ over (clip, aes, color, hps), producing distinct visual styles. Corners emphasize individual reward dimensions, while intermediate positions combine multiple dimensions. The corresponding weights and metric scores are shown beside each image. 4.2 Real User: Personalized Weight Modeling We next extend our evaluation from synthetic preferences to real user choices using the training split of Pick-a-Pic (Kirstain et al., 2023). We evaluate 631 users, holding out 50 preference pairs per user for testing and using their remaining pairs for training. To examine performance under different amounts of personal feedback, we groups users by their training-set sizes, with 66, 55, 120, and 390 users in the four groups, respectively. We measure held-out pairwise preference accuracy to assess how well each method predicts a user’s unseen choices. We repeat the train-test split and model user preference for 3 times and report the average held-out pair test accuracy. We evaluate variants of PAT to examine the contribution of each pipeline component. The full method combines alternating evaluator and user-weight optimization, hierarchical preference modeling, and user-specific personalization. w/o mixed effect replaces the joint parameterization λu=G+bu _u=G+b_u with separate per-user weight fitting regularized toward a fitted global preference vector, pushing back to an objective as Equation 6. w/o Alternating Training freezes the shared evaluator during user-weight fitting. We also remove both components to evaluate their combined contribution. Universal λ replaces user-specific weights with a shared population-level vector. Finally, Universal λ w/ Frozen Evaluator uses a fixed evaluator and a shared weight vector, removing all three components. Table 2 shows that PAT achieves strong preference prediction performance across different amounts of user feedback. Personalization provides substantial gains: compared with using shared population-level weights, user-specific modeling improves accuracy by 5.5%5.5\%–7.9%7.9\% across all four groups, highlighting the importance of capturing individual preferences. Mixed-effects modeling is particularly useful when feedback is limited: it improves accuracy by 1.1%1.1\% and 0.7%0.7\% percentage points in the 20–30 and 30–60 groups, respectively, while matching independent fitting in the smallest group. Alternating training further improves accuracy across all groups, supporting joint adaptation of the evaluator and user-specific weights. Table 3: Effect of autoencoder dimension k on held-out pairwise preference accuracy (%) for 631 users. Variants follow the component ablations in Table 2. Bold indicates the best accuracy at each dimensionality. Variant k=20 k=40 k=50 k=80 k=100 PAT 75.28 76.88 77.14 77.17 77.10 w/o Mixed Effect 75.14 76.93 76.73 76.85 76.74 w/o Alternating Training 72.99 74.80 74.65 74.60 74.51 w/o Alternating Training & Mixed Effect 72.60 74.24 74.04 74.23 74.36 Universal λ 69.58 70.36 69.53 69.45 69.53 w/ Frozen Evaluator 64.72 65.88 65.86 65.51 65.47 Table 4: General-purpose image-quality and prompt-alignment metrics for PAT and vanilla SD3.5. Win rate is the fraction of matched pairs in which our image receives a higher score. Metric Baseline PAT PAT win rate PickScore 0.2207 0.2203 47.6% HPS 0.2817 0.2817 52.0% CLIP 0.2825 0.2813 49.1% AES 5.913 5.856 45.4% 4.3 Ablation Studies Autoencoder vs. other dimension reduction methods. We compare the autoencoder with two linear dimension reduction methods:(a) a correlation-based baseline that groups correlated visual attributes and averages their binary labels to construct shared preference dimensions and (b) Principle Component Analysis(PCA) that uses SVD to factorize the binary label matrix and use the top k principle dimension as shared preference dimensions. As detailed in Appendix B.5, the autoencoder achieves highest held-out pairwise accuracy at every tested dimensionality. At k=50k=50, accuracy improves from 75.39%75.39\% to 77.14%77.14\%. These results support Autoencoder approach constructs a more effective basis for personalization than other linear dimension reduction methods. Preference-space dimensionality. Table 4 examines the effect of the autoencoder dimension k. Increasing k from 20 to 50 improves PAT from 75.28%75.28\% to 77.14%77.14\% accuracy, suggesting that additional dimensions help capture heterogeneous user preferences. Beyond k=50k=50, accuracy plateaus, reaching 77.17%77.17\% and 77.10%77.10\% at k=80k=80 and k=100k=100, respectively. These results suggest that 50 dimensions are sufficient to capture the preference variation needed for accurate prediction in this setting, with further increases providing no additional benefit for the full method. Regularization strength. We vary the regularization coefficient γ to examine its effect on preference estimation, with detailed results provided in Appendix B.4. For the sake of notation brevity, we use γ to represent the regularization strength including mixed-in effect. Held-out pairwise accuracy peaks with γ=0.03γ=0.03. This trend suggests that moderate regularization helps generalization, whereas excessive regularization restricts the model’s ability to capture user-specific preferences. User Profile SD 3.5 SD 3.5+Profile PPD PAT Visually, they favor clean, centered compositions with strong subject isolation, soft natural textures, warm light, lush greens, creamy whites, and controlled saturated accents. a mature poodle toy Visually, they prefer earthy textures, architectural intricacy, and compositions … consistently reject overly glossy, cartoonish, exaggerated, or poster-like imagery; extreme close-ups, simplified fantasy shapes… a Majestic Ancient Temple, Moody, Fantasy Landscape, Volumetric Lighting Digital painting, Marc Allante, Studio Ghibli, dramatic, Intricate detail They like refined palettes, stylized realism, clean graphic shapes, purposeful costume/armor design, and compositions that place the subject in a meaningful environment. Illustration of two men sitting at a table eating food and drinking wine Aesthetically, they favor clean, centered compositions with a strong iconic subject, … reject cluttered scenes, busy narrative cartoons, muddy or overly literal landscapes. Cow made of Cactus oil painting Figure 3: Qualitative comparison of personalization methods. For each user, we compare images generated by vanilla SD3.5, Profile-as-Prompt, PPD, and PAT under the same prompt. We highlight the user preferences in green and dispreferences in red. More qualitative results are in Appendix E. 4.4 Real User: Personalized Image Generation Experimental setup. We evaluate whether the learned personalized rewards improve image generation beyond ranking existing images. We include all 631 users and generate 10 images per user for each method. We use Stable Diffusion 3.5 (SD3.5) as the base generator, with 30 sampling steps and a classifier-free guidance scale of 4.5. For PAT, we optimize the initial noise using Direct Noise Optimization (DNO) with the learned personalized reward, while keeping the diffusion model frozen. DNO runs for 30 optimization steps with a learning rate of 0.005. We use GPT-5.5 as the judge under a blind, profile-aware evaluation protocol, comparing each method’s outputs against vanilla SD3.5 for alignment with the user’s preferences. latex We first provide GPT-5.5 with 10 image pairs and the user’s recorded preferences, one pair for each test prompt, to construct a user preference profile. We then use this profile to guide GPT-5.5 in judging which generated image better aligns with the user’s preferences. The prompt templates are provided in Appendix C. Baselines. We compare against our reproduction of Personalized Preference Fine-tuning of Diffusion Models (PPD). For each user, PPD randomly selects four historical prompt–image pairs and uses a 72B Qwen vision-language model to extract the user’s preferences. The resulting VLM embeddings serve as additional conditioning signals for diffusion model fine-tuning, and the texts summarized from VLM are treated as user profile. At generation time, we set PPD’s user-signal guidance scale to 1. We also include a Profile-as-Prompt baseline, which appends the textual user profile produced by PPD’s VLM to the original generation prompt, directly conditioning SD3.5 on the preference description. Generation results. Images generated by PAT are preferred over vanilla SD3.5 in 63.0% of comparisons, compared with 58.5% for PPD and 55.7% for Profile-as-Prompt. Our method therefore improves the win rate against the shared baseline by 4.5% and 7.3%, respectively. These results demonstrate that personalized rewards learned from offline preference pairs transfer effectively to generation-time optimization, achieving the highest win rate among the evaluated methods while keeping the diffusion model frozen. Image-quality control. To assess whether the personalized generation gains reflect a general improvement in image quality, we compare PAT with vanilla SD3.5 over the image pairs generated from the same prompts. We evaluate each pair using four general-purpose quality and alignment metrics: PickScore, HPS, CLIP, and AES. As shown in Table 4, average scores remain similar across methods, while metric-based win rates range from 45.4%45.4\% to 52.0%52.0\%, showing no consistent advantage for PAT on these metrics. Together with the 63.0%63.0\% win rate under profile-aware evaluation, these results suggest that our gains reflect better alignment with individual preferences rather than a broad increase in generic image quality. 5 Conclusion We introduced PAT, a framework for personalizing diffusion models at test time from historical pairwise preferences. By combining shared, attribute-grounded reward dimensions with user-specific weights, PAT enables preference estimation from limited feedback and guides generation while keeping the diffusion model frozen. Experiments demonstrate improved preference prediction and better personalized generation win rate over vanilla SD3.5, without consistent gains in generic quality metrics. These results support compact personalized rewards as an effective interface between individual preferences and inference-time image generation. AI Use Statement We used generative AI tools to assist with manuscript editing, organizing experimental results, preparing figures, providing feedback on research methodology and experiments, and summarizing existing literature. We also used LLMs to assist with implementing our method and employed a vision-language model as a judge in part of our experimental evaluation. Generative AI tools were not used to originate research ideas, formulate the methodology, design experiments, or search for related literature. The authors reviewed all AI-assisted work and verified the correctness of LLM-generated code. We take responsibility for the final content of this work, including all text, claims, results, and artifacts produced with the assistance of generative AI. Ethics Statement This work uses existing public preference data from Pick-a-Pic and involves no new participant recruitment or manual preference annotation. Automated visual attribute annotations and model-based generation evaluations may reflect biases in the underlying models and data. Personalized generation may reinforce biased preferences or produce harmful content, while user histories and learned preference vectors may reveal sensitive information. Practical deployment should therefore protect preference data and retain appropriate content-safety safeguards. Reproducibility Statement Appendix A details the evaluator architecture and training objectives, Appendix A provides dataset descriptions, reward metrics, and ablation details. We will publicly release our code and experimental configurations to support reproduction and further research. References Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin Qwen2.5-vl technical report. External Links: 2502.13923, Link Cited by: §3.2. Barreto et al. (2025) A. Barreto, V. Dumoulin, Y. Mao, M. Rowland, N. Perez-Nieves, B. Shahriari, Y. Dauphin, D. Precup, and H. Larochelle Capturing individual human preferences with reward features. Advances in Neural Information Processing Systems. Cited by: §2. Betker et al. (2023) J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf 2 (3), p. 8. Cited by: §2. Black et al. (2023) K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301. Cited by: §2. Cai et al. (2026) H. Cai, Y. Li, T. Yu, F. Zhu, W. Wang, F. Feng, and W. Li One adapts to any: meta reward modeling for personalized llm alignment. arXiv preprint arXiv:2601.18731. Cited by: §2. Chen et al. (2026) J. Chen, C. Huang, Q. Huang, Y. Tu, Z. Zheng, and X. Huang Test-time reinforcement learning for flow matching. In Forty-third International Conference on Machine Learning, Cited by: §2. Chen et al. (2025) R. Chen, X. Zhang, M. Luo, W. Chai, and Z. Liu Pad: personalized alignment of llms at decoding-time. In International Conference on Learning Representations, Vol. 2025, p. 9014–9039. Cited by: §2. Clark et al. (2023) K. Clark, P. Vicol, K. Swersky, and D. J. Fleet Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400. Cited by: §2. Dai et al. (2023) X. Dai, J. Hou, C. Ma, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807. Cited by: §2. Dang et al. (2025) M. Dang, A. Singh, L. Zhou, S. Ermon, and J. Song Personalized preference fine-tuning of diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8020–8030. Cited by: §1, §2. Fan et al. (2023) Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee Dpok: reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems 36, p. 79858–79885. Cited by: §2. Goyal et al. (2026) M. Goyal, A. Singh, and H. Jamali-Rad Unified control for inference-time guidance of denoising diffusion models. In 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 5437–5446. Cited by: §2. Gu et al. (2024) Y. Gu, Z. Wang, Y. Yin, Y. Xie, and M. Zhou Diffusion-rpo: aligning diffusion models through relative preference optimization. arXiv preprint arXiv:2406.06382. Cited by: §2. Hasler and Suesstrunk (2003) D. Hasler and S. E. Suesstrunk Measuring colorfulness in natural images. In Human vision and electronic imaging VIII, Vol. 5007, p. 87–95. Cited by: §B.1, §4.1. Hotelling (1933) H. Hotelling Analysis of a complex of statistical variables into principal components.. Journal of educational psychology 24 (6), p. 417. Cited by: §1. [16] V. Iyer, M. Gheini, C. Federmann, L. Tan, and A. Birch Personalized reward modelling from implicit user preferences. Cited by: §2. Jiang et al. (2026) Z. Jiang, Y. Wen, and Z. Liu Rethinking preference alignment for diffusion models with classifier-free guidance. arXiv preprint arXiv:2602.18799. Cited by: §2. Jin et al. (2025) L. Jin, Z. Qiu, J. Liu, Z. Diao, L. Qiao, N. Ding, A. Lamb, and X. Qiu Inference-time alignment control for diffusion models with reinforcement learning guidance. arXiv preprint arXiv:2508.21016. Cited by: §2. Kim et al. (2025a) H. Kim, S. Ahn, and Y. Seo Draw your mind: personalized generation via condition-level modeling in text-to-image diffusion models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 17171–17180. Cited by: §1, §2. Kim et al. (2025b) M. Kim, K. Lee, S. Joo, H. Lee, T. Thonet, and K. Jung Drift: decoding-time personalized alignments with implicit user preferences.. In EMNLP (Findings), p. 6107–6126. Cited by: §2. [21] S. Kim, M. Kim, and D. Park Test-time alignment of diffusion models without reward over-optimization. In The Thirteenth International Conference on Learning Representations, Cited by: §2. Kirstain et al. (2023) Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy Pick-a-pic: an open dataset of user preferences for text-to-image generation. Advances in neural information processing systems 36, p. 36652–36663. Cited by: Appendix A, §B.1, §B.2, §3.2, §3.2, §4.1, §4.2. Lee et al. (2025a) J. Lee, R. Heo, and D. Lee Personalized reward modeling for text-to-image generation. arXiv preprint arXiv:2511.19458. Cited by: §2. Lee et al. (2025b) K. Lee, X. Li, Q. Wang, J. He, J. Ke, M. Yang, I. Essa, J. Shin, F. Yang, and Y. Li Calibrated multi-preference optimization for aligning diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 18465–18475. Cited by: §2. Li et al. (2025a) J. Li, Y. Cui, T. Huang, W. Kong, Y. Cheng, C. Zeng, Y. Ma, C. Fan, M. Yang, Z. Zhong, et al. Mixgrpo: unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802. Cited by: §2. Li et al. (2026a) X. Li, R. Jin, J. Zhu, R. Guo, and Z. Liu Aplaud: adaptive personalized low-rank decomposition for user-specific llm. arXiv preprint arXiv:2609.04738. Cited by: §2. Li et al. (2026b) Y. Li, S. Yang, X. Han, W. Wang, J. Dong, Y. Lyu, and Z. Xue Instant preference alignment for text-to-image diffusion models. IEEE Transactions on Visualization and Computer Graphics. Cited by: §2. Li et al. (2026c) Y. Li, Q. Li, C. Bai, X. Luo, Z. Xue, W. Qin, M. Wang, Y. Wang, and S. Zhang AEGPO: adaptive entropy-guided policy optimization for diffusion models. arXiv preprint arXiv:2602.06825. Cited by: §2. Li et al. (2025b) Z. Li, Z. Liu, Q. Zhang, B. Lin, F. Wu, S. Yuan, Z. Yan, Y. Ye, W. Yu, Y. Niu, et al. Uniworld-v2: reinforce image editing with diffusion negative-aware finetuning and mllm implicit feedback. arXiv preprint arXiv:2510.16888. Cited by: §2. Liang et al. (2024) Z. Liang, Y. Yuan, S. Gu, B. Chen, T. Hang, J. Li, and L. Zheng Step-aware preference optimization: aligning preference with denoising performance at each step. arXiv preprint arXiv:2406.04314 2 (5), p. 7. Cited by: §2. Liu et al. (2026) J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang Flow-grpo: training flow matching models via online rl. Advances in neural information processing systems 38, p. 40783–40818. Cited by: §2. Pearson (1901) K. Pearson LIII. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science 2 (11), p. 559–572. Cited by: §1. Poddar et al. (2024) S. Poddar, Y. Wan, H. Ivison, A. Gupta, and N. Jaques Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems 37, p. 52516–52544. Cited by: §2. Podell et al. (2024) D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach Sdxl: improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, Vol. 2024, p. 1862–1874. Cited by: §1. Prabhudesai et al. (2023) M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki Aligning text-to-image diffusion models with reward backpropagation. Cited by: §1, §2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §B.1, §4.1. Ramesh et al. (2021) A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever Zero-shot text-to-image generation. In International conference on machine learning, p. 8821–8831. Cited by: §1. Rombach et al. (2022) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 10674–10685. Cited by: §1. Salehi et al. (2024) S. Salehi, M. Shafiei, T. Yeo, R. Bachmann, and A. Zamir Viper: visual personalization of generative models via individual preference learning. In European Conference on Computer Vision, p. 391–406. Cited by: §1, §1, §2, §3.2. Schuhmann (2022) C. Schuhmann LAION-aesthetics. Note: https://laion.ai/blog/laion-aesthetics/ Cited by: §B.1, §4.1. Seedream et al. (2025) T. Seedream, Y. Chen, Y. Gao, L. Gong, M. Guo, Q. Guo, Z. Guo, X. Hou, W. Huang, Y. Huang, et al. Seedream 4.0: toward next-generation multimodal image generation. arXiv preprint arXiv:2509.20427. Cited by: §1. Shenfeld et al. (2025) I. Shenfeld, F. Faltings, P. Agrawal, and A. Pacchiano Language model personalization via reward factorization. arXiv preprint arXiv:2503.06358. Cited by: §2. Song et al. (2026) Y. Song, J. Wu, W. Zhang, C. Shen, S. Yuan, W. Lu, J. Wang, Y. Wang, N. Dutt, and A. M. Rahmani CARD: cluster-level adaptation with reward-guided decoding for personalized text generation. arXiv preprint arXiv:2601.06352. Cited by: §2. Spearman (1961) C. Spearman ” General intelligence” objectively determined and measured.. Cited by: §1. Tan et al. (2026) Z. Tan, Z. Zhang, H. Wen, Z. Li, R. Zhang, P. Chen, F. Mo, Z. Liu, Q. Zeng, Q. Yin, et al. Instant personalized large language model adaptation via hypernetwork. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 23557–23580. Cited by: §2. Tang et al. (2024) Z. Tang, J. Peng, J. Tang, M. Hong, F. Wang, and T. Chang Tuning-free alignment of diffusion models with direct noise optimization. In ICML 2024 Workshop on Structured Probabilistic Inference \\ &\ Generative Modeling, Cited by: Appendix D, Appendix D, §2, §3.4, §3.4. Thurstone (1931) L. L. Thurstone Multiple factor analysis.. Psychological review 38 (5), p. 406. Cited by: §1. Wallace et al. (2024) B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik Diffusion model alignment using direct preference optimization. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8228–8238. Cited by: §1, §2. Wang et al. (2025) F. Wang, Y. Shui, J. Piao, K. Sun, and H. Li Diffusion-npo: negative preference optimization for better preference aligned generation of diffusion models. arXiv preprint arXiv:2505.11245. Cited by: §2. Wang et al. (2024) K. Wang, R. Kidambi, R. Sullivan, A. Agarwal, C. Dann, A. Michi, M. Gelmi, Y. Li, R. Gupta, K. A. Dubey, et al. Conditional language policy: a general framework for steerable multi-objective finetuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 2153–2186. Cited by: §2. Wang et al. (2026a) X. Wang, J. Han, Z. Wang, Q. Zeng, X. Wang, R. Wang, and J. Li Learning to adapt cross-domain preferences via meta-lora for llm personalization. arXiv preprint arXiv:2608.12389. Cited by: §2. Wang et al. (2026b) Z. Wang, Y. Wei, X. Zhou, T. Zhang, T. Liang, Y. Bai, H. Zhang, and W. Zuo Premier: personalized preference modulation with learnable user embedding in text-to-image generation. arXiv preprint arXiv:2603.20725. Cited by: §2. Wu et al. (2023) X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341. Cited by: §B.1, §4.1. Wu et al. (2026) Y. Wu, C. Wang, Y. Bai, X. Zhao, Y. Zhang, and F. Feng PrefReward: learning user preference matrix for personalized text generation. arXiv preprint arXiv:2607.21067. Cited by: §2. Xie et al. (2026) X. Xie, F. Zhang, and D. Gong Test-time weak-to-strong alignment: transferring implicit rewards from weak to strong flow models. arXiv preprint arXiv:2609.05968. Cited by: §2. Xue et al. (2025) Z. Xue, J. Wu, Y. Gao, F. Kong, L. Zhu, M. Chen, Z. Liu, W. Liu, Q. Guo, W. Huang, et al. Dancegrpo: unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818. Cited by: §2. Yang et al. (2024a) K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, W. Shen, X. Zhu, and X. Li Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8941–8951. Cited by: §2. Yang et al. (2024b) R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen Rewards-in-context: multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207. Cited by: §2. Yeh et al. (2024) P. Yeh, K. Lee, and J. Chen Training-free diffusion model alignment with sampling demons. arXiv preprint arXiv:2410.05760. Cited by: §2. Yu et al. (2026) X. Yu, H. Xing, and L. Xue EXACT: explicit attribute-guided decoding-time personalization. arXiv preprint arXiv:2602.17695. Cited by: §2. Zhai et al. (2025) K. Zhai, U. Singh, A. Thatipelli, S. Chakraborty, A. K. Sahu, F. Huang, A. S. Bedi, and M. Shah Mira: towards mitigating reward hacking in inference-time alignment of t2i diffusion models. arXiv preprint arXiv:2510.01549. Cited by: §2. Zheng et al. (2026) K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M. Liu Diffusionnft: online diffusion reinforcement with forward process. In International Conference on Learning Representations, Vol. 2026, p. 134129–134150. Cited by: §2. Zhu et al. (2025) H. Zhu, T. Xiao, and V. G. Honavar DSPO: direct score preference optimization for diffusion model alignment.. Cited by: §2. Appendix A Evaluator Training Architecture and supervision. Our evaluator produces a vector of scores sθ(c,x)=[sθ,1(c,x),…,sθ,k(c,x)]⊤s_θ(c,x)=[s_θ,1(c,x),…,s_θ,k(c,x)] for a caption–image pair (c,x)(c,x). We encode the caption and image using frozen SigLIP encoders and concatenate their representations into a 2,304-dimensional feature vector(1152 dimension for both image and caption embeddings). This vector is passed through an MLP consisting of a linear layer with 512 hidden units, a GELU activation, and a linear output layer producing k scalar scores. The output dimensions share the hidden representation but have separate output weights. We train the MLP on approximately 36,000 image pairs using two sources of supervision: human pairwise preferences and dimension-specific targets derived from the pretrained autoencoder. The former provides a common preference-learning objective across dimensions, while the latter encourages distinct visual attribute responses. Human preference supervision. Let (ci,xi+,xi−)(c_i,x_i^+,x_i^-) denote a training pair, where xi+x_i^+ is preferred to xi−x_i^- by the user. For dimension j, we define the predicted score difference as dij=sθ,j(ci,xi+)−sθ,j(ci,xi−).d_ij=s_θ,j(c_i,x_i^+)-s_θ,j(c_i,x_i^-). (9) Following the pairwise preference-learning principle of PickScore (Kirstain et al., 2023), we encourage each output dimension to assign a higher score to the preferred image. We use the binary cross-entropy loss ℓBCE(d,y)=−ylogσ(d)−(1−y)log(1−σ(d)), _BCE(d,y)=-y σ(d)-(1-y) (1-σ(d) ), (10) where d∈ℝd is a predicted score difference, y∈0,1y∈\0,1\ indicates whether the first image should receive a higher score than the second, and σ is the sigmoid function. Since each pair is ordered by human preference, the target for this objective is always 11. For a minibatch ℬB, the preference loss is Lprefer=1|ℬ|k∑i∈ℬ∑j=1kℓBCE(dij,1).L_prefer= 1|B|k _i _j=1^k _BCE(d_ij,1). (11) Visual attribute supervision. To construct dimension-specific supervision, we use the fixed encoder of the pretrained autoencoder: zi+=Eη(a(xi+)),zi−=Eη(a(xi−)),z_i^+=E_η(a(x_i^+)), z_i^-=E_η(a(x_i^-)), (12) where a(x)∈0,1512a(x)∈\0,1\^512 is the binary visual attribute annotation of image x, EηE_η is the autoencoder encoder with fixed parameters η, and zi+,zi−∈ℝkz_i^+,z_i^- ^k are the corresponding continuous latent representations. We first compute the latent difference for each pair and dimension: δij=zij+−zij−. _ij=z_ij^+-z_ij^-. (13) For each dimension j, we rank the training pairs by |δij|| _ij| and retain the top 33% for attribute supervision. Let ℐjI_j denote the indices of the retained pairs. We define the supervision mask and binary ordering target as mij=[i∈ℐj],yijattr=[δij>0].m_ij=1[i _j], y_ij^attr=1[ _ij>0]. (14) Thus, pair selection is based on the magnitude of the continuous latent difference, while the binary target records its direction. We do not binarize the individual images’ latent values. This procedure focuses attribute supervision on pairs with relatively large differences within each dimension. For a minibatch ℬB, the masked attribute loss is Lattr=∑i∈ℬ∑j=1kmijℓBCE(dij,yijattr)max(1,∑i∈ℬ∑j=1kmij).L_attr= _i _j=1^km_ij\, _BCE(d_ij,y_ij^attr) (1, _i _j=1^km_ij ). (15) Pairs outside ℐjI_j contribute no attribute loss for dimension j, but still contribute to the human preference loss. We optimize the sum of the two losses: Leval=Lprefer+Lattr.L_eval=L_prefer+L_attr. (16) Together, these objectives encourage the evaluator to balance general preference prediction with dimension-specific visual attribute supervision. Only the MLP parameters are updated; the SigLIP encoders and autoencoder remain fixed during this training stage. The autoencoder outputs define supervision masks and targets, whereas both losses operate on the evaluator’s predicted score differences. The evaluator therefore learns continuous, attribute-grounded preference scores rather than directly reconstructing the autoencoder latent values. These scores are subsequently combined using user-specific weights to construct personalized rewards. Appendix B More Details about Experiments B.1 Reward Metrics Our controlled experiments define synthetic user preferences over five reward dimensions: clip, aes, color, hps, and pick. These scores measure text–image alignment, visual aesthetics, colorfulness, and learned human preferences. Although the dimensions may be correlated, they provide distinct signals that can be combined to construct different preference profiles. CLIP similarity (clip). CLIP (Radford et al., 2021) learns a shared image–text embedding space through contrastive training on image–text pairs. We use the similarity between the embeddings of a generated image and its corresponding prompt to measure semantic alignment. Higher scores indicate stronger correspondence between the image content and the text description. This dimension captures prompt relevance, although it does not directly measure visual quality or individual user preferences. Aesthetic score (aes). The aesthetic predictor (Schuhmann, 2022) estimates an image’s aesthetic appeal using a prediction head applied to its CLIP image embedding. It is trained to predict human aesthetic ratings and provides an image-only quality signal, without conditioning on the prompt. This dimension captures visual appeal that may not be reflected by text–image similarity alone. Colorfulness (color). The colorfulness metric of Hasler and Suesstrunk (2003) quantifies image colorfulness using statistics of red–green and yellow–blue opponent channels. It combines the variability and mean magnitude of these color signals to produce an image-only score. Unlike the learned reward models, this metric measures a specific visual property without preference supervision. Higher values indicate more colorful images, which need not be preferred by every user; its contribution to the synthetic reward is determined by the user’s weight vector. Human Preference Score (hps). Human Preference Score v2 (Wu et al., 2023) is a CLIP-based scoring model fine-tuned on the Human Preference Dataset v2 (HPD v2). Given a prompt and an image, it predicts a score intended to reflect human preferences over generated images. Compared with the original CLIP similarity, this score incorporates explicit human preference supervision and provides a broader signal of perceived image quality and prompt compatibility. PickScore (pickscore). PickScore (Kirstain et al., 2023) is a CLIP-based preference model trained on pairwise human feedback from Pick-a-Pic. It assigns a scalar score to a prompt–image pair using the scaled similarity between its learned text and image embeddings. Training encourages the relative scores of competing images to match observed human choices. Like HPS, PickScore provides a population-level preference signal; the two models are trained on different preference datasets and may therefore rank the same images differently. Combining reward dimensions. Because the five metrics have different numerical scales, we standardize each dimension over the sampled images before constructing synthetic preferences, as described in Section 4.1. Let r~j(c,x) r_j(c,x) denote the standardized score of dimension j. For a synthetic user with ground-truth weights λ⋆λ , the reward is rλ⋆(c,x)=∑j=15λj⋆r~j(c,x).r_λ (c,x)= _j=1^5 _j r_j(c,x). For each image pair associated with the same prompt, we assign the synthetic preference label to the image with the higher weighted reward. This construction allows us to evaluate preference learning against a known reward function. B.2 Pick-a-Pic Dataset Pick-a-Pic (Kirstain et al., 2023) is an open dataset of real user preferences for text-to-image generation, collected through a web application in which users submit prompts and compare generated images. Each comparison associates a text prompt with two candidate images and the user’s preference feedback. The dataset also permits ties and includes user identifiers, enabling preference observations to be grouped into individual user histories. Its prompts and judgments arise from users interacting with image generation systems, providing a setting for studying heterogeneous visual preferences. We use Pick-a-Pic v1 in two experimental settings. In the controlled toy experiments, it supplies captioned image pairs, while preference labels are constructed from the synthetic weighted reward rather than the original human choices. Each repetition samples 200 pairs, with 100 used for fitting the preference weights and 100 for evaluation. In the real-user experiments, we use the recorded human choices to fit and evaluate personalized rewards. Following Section 4.2, we select the 631 users with the largest numbers of preference pairs from the training split, hold out 50 pairs per user for evaluation, and use the remaining pairs for training. These experiments assess, respectively, recovery of controlled synthetic preferences and prediction of unseen choices from real users. B.3 Toy Example λ Weights Table 5 reports the ground-truth and fitted preference weights for the five synthetic users introduced in Section 4.1, with fitted weights averaged over three independently sampled datasets. The configurations cover both broadly distributed preferences (uniform and half-normal) and more concentrated preferences (Pareto). Weight recovery is strongest for the heavy-1 configuration, with a reported cosine similarity of 0.990, and the fitted vector preserves its emphasis on the third and fourth reward dimensions. Recovery is less accurate for the uniform and half-normal configurations, where the fitted weights place greater emphasis on the fifth dimension. These results highlight that recovering the exact preference weights remains challenging with limited pairwise feedback, even in this controlled setting. Table 5: Ground-truth and fitted preference weights for five synthetic users with uniform, half-normal, or Pareto weight configurations. Each row reports the ground-truth vector λ⋆λ , the fitted vector λ λ averaged over three datasets, and the reported cosine similarity between fitted and ground-truth weights (higher is better). Config Family Ground-truth λ⋆λ Fitted λ λ (mean over 3 datasets) cos(λ^,λ⋆) ( λ,λ ) uniform 5-dim uniform (0.20,0.20,0.20,0.20,0.20)(0.20,0.20,0.20,0.20,0.20) (0.12,0.09,0.16,0.22,0.41)(0.12,0.09,0.16,0.22,0.41) 0.869 normal-1 half-normal (0.22,0.17,0.19,0.18,0.24)(0.22,0.17,0.19,0.18,0.24) (0.15,0.06,0.13,0.19,0.47)(0.15,0.06,0.13,0.19,0.47) 0.870 normal-2 half-normal (0.24,0.11,0.18,0.25,0.22)(0.24,0.11,0.18,0.25,0.22) (0.17,0.00,0.12,0.27,0.44)(0.17,0.00,0.12,0.27,0.44) 0.883 heavy-1 Pareto (0.06,0.03,0.44,0.40,0.07)(0.06,0.03,0.44,0.40,0.07) (0.00,0.00,0.51,0.39,0.10)(0.00,0.00,0.51,0.39,0.10) 0.990 heavy-2 Pareto (0.38,0.06,0.10,0.26,0.20)(0.38,0.06,0.10,0.26,0.20) (0.34,0.00,0.04,0.25,0.37)(0.34,0.00,0.04,0.25,0.37) 0.939 B.4 Ablation on Regularization Figure 4 examines the effect of the regularization coefficient γ on held-out pairwise preference accuracy. Among the tested values, accuracy peaks at approximately 77.3% with γ=0.03γ=0.03. Increasing γ beyond this value progressively reduces accuracy, reaching approximately 74.7% at γ=1γ=1. This pattern suggests that moderate regularization provides a useful balance between fitting observed preferences and generalizing to unseen comparisons, whereas excessive regularization limits the model’s ability to capture user-specific preferences. Figure 4: Ablation on the regularization coefficient γ. Table 6: Comparison of correlation-based dimension reduction, PCA based dimension reduction, and an autoencoder across four preference-space dimensions k. Values are reported as percentages. k Correlation PCA Autoencoder 20 74.52% 74.41% 75.28% 30 74.81% 75.61% 76.29% 40 74.87% 76.05% 76.88% 50 75.39% 76.63% 77.14% B.5 Dimension Reduction with Correlation Correlation-based dimension construction. Let a(x)∈0,1512a(x)∈0,1^512 denote the attribute annotation vector for image x. We first compute pairwise correlations between attribute dimensions across the annotated images, then group highly correlated attributes into k clusters, denoted by 1,…,kC_1,…,C_k. Each cluster defines one shared preference dimension. For an image x, we compute its score along dimension j by averaging the binary labels of the attributes belonging to that cluster: zjcorr(x)=1|j|∑ℓ∈jaℓ(x),j=1,…,k.z_j^corr(x)= 1|C_j| _ _ja_ (x), j=1,…,k. For example, if a cluster contains ten attributes and four are present in an image, its score is 0.40.4. This aggregation converts the original binary attribute vector into a compact representation zcorr(x)∈[0,1]kz^corr(x)∈[0,1]^k, where each coordinate measures the fraction of attributes present within a correlated group. Principal component analysis. We also consider PCA as a linear alternative to the autoencoder for constructing the shared attribute representation. We use the same annotated image pool, with each image represented by its binary attribute vector a(x)∈0,1512a(x)∈\0,1\^512. Let a¯ a denote the mean attribute vector over this pool. We center the attribute vectors and compute the k principal directions with the largest eigenvalues of the empirical covariance matrix. Collecting these directions as the columns of Wk∈ℝ512×kW_k ^512× k, we represent each image as zPCA(x)=Wk⊤(a(x)−a¯)∈ℝk.z^PCA(x)=W_k (a(x)- a ) ^k. (17) The mean vector and projection matrix are estimated from the representation-learning pool and subsequently held fixed. Unlike correlation-based grouping, which averages attributes within each cluster, PCA constructs orthogonal linear combinations that retain the greatest attribute variance at the chosen dimensionality. In contrast to the autoencoder, its representation is restricted to a linear projection. Evaluator and personalized reward. For both alternatives, we replace the autoencoder latent targets with the corresponding k-dimensional representation while retaining the evaluator architecture and training objectives described in Appendix A. Specifically, we compute the representation difference between the two images in each training pair. For each dimension, we select the top 33% of pairs by absolute difference and use the sign of the difference as the binary attribute-ordering target. The evaluator is trained using this dimension-specific supervision together with human preference supervision. We then fit user-specific weights under the same personalized reward-learning procedure: ru(c,x)=λu⊤sθ(c,x).r_u(c,x)= _u s_θ(c,x). (18) This comparison varies the construction of the shared attribute representation while keeping the downstream evaluator and personalization procedure unchanged. Appendix C Prompts for Profile-Aware Evaluation Table 7 presents the two-stage prompting protocol used for profile-aware evaluation. First, GPT-5.5 constructs a user preference profile from 10 image pairs, one for each test prompt. Within each pair, the preferred image is followed by the dispreferred image, allowing the model to summarize both favored and rejected visual characteristics. The profile covers subject matter and stylistic preferences while explicitly excluding judgments of image quality or rendering fidelity. Second, GPT-5.5 receives this profile, an evaluation caption, and two generated images, and selects the image that better matches the user’s preferences. In the templates, k and n are set to 10 and 20, respectively; profile and caption are replaced with the generated profile and evaluation caption. Table 7: Prompt templates for user profiling and GPT-5.5 evaluation. The first prompt constructs a taste profile from preferred–dispreferred image pairs. The second compares two generated images using that profile. Here, k is the number of preference pairs and n is the number of attached images (n=2kn=2k); profile and caption denote the resulting profile and evaluation caption. Stage Prompt template 1. User profiling You are profiling one user’s personal visual taste from k preference examples. n images are attached in THIS EXACT ORDER, as k consecutive pairs: image 1 = PREFERRED, image 2 = DISPREFERRED (same caption) image 3 = PREFERRED, image 4 = DISPREFERRED (same caption) … and so on for all k pairs. Within each pair the two images answer the same prompt, so what separates them is taste, not subject matter. Study what the preferred image has that the dispreferred one lacks, and what the dispreferred one has that the user rejected. Write a concise profile of this user’s taste, in two short paragraphs: 1. TOPICS/subjects they are drawn to (what the images are of). 2. Their VISUAL/AESTHETIC preferences, described using these aspects where relevant: Art Style, Brushstrokes, Color Palette, Composition, Skill, Detail, Hues, Saturation, Line, Shape, Value, Pattern, Texture, Artistic Medium, Mood, Perspective, Depth, Movement, Form, Juxtaposition, Iconography. State what they favour AND what they consistently reject. IMPORTANT: describe only CONTENT topics and STYLISTIC preferences. Do NOT mention image quality, resolution, sharpness, realism-as-quality, rendering fidelity, artifacts, or which images are ’better made’ — those are not part of the user’s taste. Output only the profile text. 2. Pairwise judging Here is a profile of a user’s visual taste (topics + stylistic preferences): profile Two new AI-generated images are attached: img_P (first) and img_Q (second), both for the caption “caption”. Based ONLY on how well each image matches this user’s TOPICS and STYLISTIC preferences above — NOT on raw image quality or which looks better made — which image would this user more likely prefer? Answer with “P” or “Q”. Appendix D Image Generation with Direct Noise Optimization Deterministic sampling process. We apply Direct Noise Optimization (DNO) (Tang et al., 2024) to maximize the learned personalized reward at inference time while keeping the image generator frozen. Stable Diffusion 3.5 uses a flow-matching model with a deterministic ODE-based sampler in our implementation. Once the prompt, initial latent, and sampling configuration are fixed, no additional randomness is introduced during denoising. Consequently, the complete generation process can be represented as a deterministic mapping x=Mϕ(c,xT),x=M_φ(c,x_T), (19) where c is the text prompt, x is the clean image, xT∈ℝdx_T ^d is the initial latent noise, and ϕφ denotes the frozen diffusion generator, including its denoising sampler and VAE decoder. The initial latent is sampled from the Gaussian prior, xT(0)∼(0,I)x_T^(0) (0,I). This setting differs from the main stochastic formulation studied by Tang et al. (2024), which jointly optimizes the initial noise and the additional noises injected at individual sampling steps. Because our deterministic sampler introduces no step-wise noise, its output is fully determined by xTx_T, and the initial latent is the only optimization variable. Noise optimization. For a user u with personalized reward function ru(c,x)r_u(c,x), the unregularized DNO objective is maxxTru(c,Mϕ(c,xT)). _x_T r_u\! (c,M_φ(c,x_T) ). (20) At each optimization step, we run the complete differentiable sampling trajectory from the current initial latent and evaluate the personalized reward on the resulting image. Gradients are then backpropagated through the frozen generator to xTx_T. Although gradients pass through MϕM_φ and rur_u, all parameters of the generator and personalized reward model remain fixed; only xTx_T is updated. Starting from xT(0)∼(0,I)x_T^(0) (0,I), a gradient-ascent update can be written as xT(k+1)=xT(k)+ℓ∇xTru(c,Mϕ(c,xT))|xT=xT(k),x_T^(k+1)=x_T^(k)+ . _x_Tr_u\! (c,M_φ(c,x_T) ) |_x_T=x_T^(k), (21) where ℓ denotes the optimization step size. In practice, we perform this update using Adam for 30 optimization steps with learning rate 0.005. Appendix E More Qualitative Results We add more qualitative examples in Figure 5. In general, PAT generates images that are closer to user profile. User Profile SD 3.5 SD 3.5+Profile PPD PAT They favor cooler or earthy palettes with selective strong accents, textured landscapes… They consistently reject overly polished fantasy spectacle, high-saturation blue/green casts photo of a dirt road in a forest, pine trees Visually, they favor clean, polished, high-saturation images with bright eyes, direct engagement, centered compositions, warm or vivid palettes, smooth surfaces, and clear character silhouettes. a dog riding bicycle Aesthetically, they prefer vivid color, warm natural light, …. They favor recognizable forms with emotional immediacy, saturated blues / oranges, and compositions centered on a single living figure. A cactus playing a guitar on a beach Visually, they favor ornate, composed images with strong framing, clear central forms, full-scene context, rich pattern, decorative linework, and a sense of depth. concept art of a beautiful english garden, 4k, vivid colors, intricate details very realistiv, cinematic lighting, volumetric lighting Aesthetically, they favor vivid, strong value contrast, glowing bokeh or star patterns, and theatrical central compositions… reject latter, quieter, greyer, more literal scenes; sparse compositions.. sorrow They favor grounded compositions with environmental context, gentle depth, tactile textures, and stylized anime/painterly looks when the subject is fantasy. Egirl with red hair, gorgeous, high-quality, beautiful Aesthetically, they favor saturated color, strong lighting, warm highlights, rich contrast, …. They reject flatter, emptier, more muted or gray compositions …. concept art, drawing of fantasy ballroom at night with faceless dancers Figure 5: More qualitative comparison of personalization methods.