Paper deep dive
PAPA: Online Personalized Active Preference Alignment
Anindya Sarkar, Nasik Muhammad Nafi, Isaac Lyngaas, Muralikrishnan Gopalakrishnan Meena, Yevgeniy Vorobeychik
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 7/5/2026, 9:14:36 AM
Summary
The paper introduces PAPA (Personalized Active Preference Alignment), a novel method for the online fine-tuning of diffusion models to align with user preferences without requiring a parameterized reward model. Unlike traditional RL-based methods that rely on large-scale static datasets or separate reward models, PAPA uses real-time user feedback and a variational inference framework to directly optimize the diffusion model. The authors also propose EPAPA (Enhanced-PAPA), a pruning-style fine-tuning strategy that reduces computational costs and accelerates the process. The method is designed to balance exploration and exploitation, ensuring sample diversity and efficiency in high-dimensional spaces like personalized recommendation and image generation.
Entities (7)
Relation Signals (4)
PAPA → isatypeof → Personalized Active Preference Alignment
confidence 100% · In this work, we introduce Personalized Active Preference Alignment (PAPA)...
PAPA → optimizes → Diffusion Models
confidence 100% · directly optimizing the diffusion model using real-time user feedback.
PAPA → uses → Variational Inference
confidence 100% · PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework.
EPAPA → enhances → PAPA
confidence 95% · we propose an enhanced fine-tuning strategy, referred to as EPAPA, that requires less computational budget and accelerates the fine-tuning process
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences-initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data-something that is often not feasible in practice. In this work, we introduce Personalized Active Preference Alignment PAPA, a novel method that bypasses the requirement for a parametrized reward model by directly optimizing the diffusion model using real-time user feedback. PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework. We demonstrate PAPA's effectiveness through extensive experiments and ablation studies across diverse class-conditioned and fine-grained alignment tasks. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy, referred to as EPAPA, that requires less computational budget and accelerates the fine-tuning process, further boosting PAPA's suitability for real-world deployment. Our code is made publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.00486v1
- Canonical: https://arxiv.org/abs/2607.00486v1
Trouble viewing inline? Open PDF directly →
Full Text
104,270 characters extracted from source content.
Expand or collapse full text
decorations.pathmorphing PAPA: Online Personalized Active Preference Alignment Anindya Sarkar ∗ Washington University in St. Louis anindya@wustl.edu Nasik Muhammad Nafi Oak Ridge National Laboratory nafinm@ornl.gov Equal Contribution. Isaac Lyngaas Oak Ridge National Laboratory lyngaasir@ornl.gov Muralikrishnan Gopalakrishnan Meena Oak Ridge National Laboratory gopalakrishm@ornl.gov Yevgeniy Vorobeychik Washington University in St. Louis yvorobeychik@wustl.edu Abstract Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences—initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data—something that is often not feasible in practice. In this work, we introduce Personalized Active Preference Alignment (PAPA), a novel method that bypasses the requirement for a parametrized reward model by directly optimizing the diffusion model using real-time user feedback. PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework. We demonstrate PAPA’s effectiveness through extensive experiments and ablation studies across diverse class-conditioned and fine-grained alignment tasks. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy, referred to as EPAPA, that requires less computational budget and accelerates the fine-tuning process, further boosting PAPA’s suitability for real-world deployment. Our code is made publicly available at https://github.com/NasikNafi/papa. †footnotetext: Accepted at ECML PKDD 2026 1 Introduction Diffusion models are deep generative models that generate data by reversing a diffusion process, excelling at capturing complex spaces like natural image manifolds. However, in applications like personalized product recommendations, the goal is to steer generation toward items that align with individual user preferences, which are revealed over time through user activity. A similar challenge arises in other domains as well. For instance, in image generation, diffusion models are trained on vast datasets scraped from the internet, but practical applications often require images with high aesthetic quality. In fact, many other scenarios share this general structure, such as drug discovery, where the goal is to guide generation toward compounds with high bioactivity. This can be framed as a reinforcement learning (RL) problem, where the objective is to fine-tune the diffusion model to maximize a reward that reflects the desired properties of the user’s preferences. However, these methods rely on extensive preference data to learn the reward model, making them ineffective in scenarios like personalized recommendation platforms, where large-scale preference data for each user is unavailable but can be gathered through costly interactive feedback. Figure 1: Overview of the considered personalized active preference alignment problem. The challenge is twofold: Firstly, achieving this objective requires efficient exploration. However, in high-dimensional spaces, such as those of natural images, this goes beyond simply discovering new regions. It also necessitates respecting the structural constraints of the problem. For instance, in areas like product recommendation, valid solutions—such as realistic-looking products—are typically confined to a lower-dimensional manifold within a much larger design space. Therefore, an effective, feedback-efficient fine-tuning method must explore this space while staying within the feasible area, as venturing outside would lead to wasteful invalid queries. Moreover, fine-tuning the diffusion model to aggressively optimize based on the preferences collected so far can reduce sample diversity. This is because human preferences are often multimodal, and the model, if overly focused on a narrow set of preferences, may fail to capture the full spectrum of diverse user preferences, leading to a less varied sample generation. Therefore, efficient exploration is crucial to maintaining the quality of generated samples and ensuring greater diversity in sample generation. Secondly, a key challenge in many applications is the high cost of acquiring feedback for the ground-truth reward function. For instance, in a product recommendation system, determining user preferences requires subjective human judgment, which is both costly and time-consuming. This challenge is further compounded by the need for the model to not only explore new options but also to exploit the information it has gathered to generate samples that align with the user’s preferences. If the model continues to explore without producing samples that meet the user’s expectations, it risks disengaging the user. In a nutshell, the model must strike a balance between exploration and exploitation—effectively generating preference-aligned samples while minimizing costly reward queries. While several recent works have proposed RL-based fine-tuning methods for diffusion models [2, 8, 27], none directly tackle the challenge of feedback efficiency in an online setting. [28] introduced a framework that accounts for the online nature of feedback but still relies on a separate parameterized reward model for optimization. Our goal instead is to develop a feedback-efficient online fine-tuning approach that entirely eliminates the need for a separate reward model, instead directly fine-tuning the diffusion model using real-time user feedback. Although direct preference optimization [16] has enabled efficient offline alignment, these methods frequently falter in online interactive settings, due to its reliance on pre-existing large-scale preference data. Critically, prior approaches, such as [22], fail to disentangle sample quality and diversity from alignment itself, resulting in poor performance when preference spans multiple, heterogeneous classes intrinsic to real human feedback. To this end, we propose a principled feedback-efficient online fine-tuning method for diffusion models, derived using the tools of variational inference, aimed at Personalized Active Preference Alignment (PAPA). We showcase the effectiveness of PAPA through comprehensive experiments and ablation studies across various domains. Moreover, we propose a pruning-style fine-tuning strategy paired with a sampling approach that leverages both the pre-trained and fine-tuned models. This strategy significantly reduces the training computation requirements of PAPA, thereby accelerating the fine-tuning process. The effectiveness of this approach is supported by both theoretical insights and empirical results, which highlight two key observations: (i) the pre-trained diffusion model retains strong zero-shot denoising capabilities at low noise levels, and (i) overfitting can occur when using the fine-tuned model for denoising at low noise levels. Below, we summarize key contributions: • We introduce PAPA, a novel online fine-tuning method for diffusion models that enables feedback-efficient preference alignment. • We also propose EPAPA, Enhanced-PAPA, a pruning-style fine-tuning strategy that accelerates the fine-tuning process while reducing computational costs without affecting the performance. • We validate the effectiveness of each component of PAPA and EPAPA through comprehensive quantitative and qualitative ablation studies across diverse preference alignment settings. • Compared to prior methods, PAPA adapts and aligns more quickly to fine-grained alignment tasks, making it well-suited for real-world interactive alignment scenarios. Notably, PAPA aligns preferences spanning across diverse classes, a capability lacking in previous methods. 2 Related Work Fine-Tune Diffusion model with RL: fine-tuning generative models with human feedback, such as user preferences, has become increasingly popular [20, 26]. Many prior studies have examined fine-tuning diffusion models by optimizing reward functions through methods like supervised learning [15], control-based techniques [1, 13], or policy-gradient [2]. However, these methods often rely on a static reward model, treating rewards as either fixed ground truth or not allowing for online feedback queries. In contrast, our approach focuses on an online setting that enables interactive preference learning, where the user preferences are initially unknown and gradually revealed through sequential user feedback. [6] presents a general online learning approach for generative models. However, their method is not specifically tailored for diffusion models, and it relies on a separate reward model to select high-quality samples for fine-tuning. These key differences are what fundamentally set our work apart. More recently, [28] proposed a feedback-efficient online fine-tuning approach for diffusion models. However, their method still relies on learning a separate reward model. The SFT method introduced by [7] applied RL to diffusion models to improve the performance of existing fast DDPM samplers. ReFL [30] leverages the RLHF framework. It first trains a model based on human preferences, and then fine-tunes the diffusion model through reinforcement learning. DDPO [2] frames the denoising process of diffusion models as a Markov Decision Process (MDP) in order to fine-tune the models using multiple reward functions. All of these models require a powerful reward model, which in turn necessitates a large dataset of images and comprehensive human evaluations. Direct Preference Optimization: In RL, exploring policies based on preferences rather than explicit rewards has gained attention through various methods. Preference-based Reinforcement Learning [16] learns from binary preferences derived from a hidden scoring function, rather than explicit rewards. The DPO approach [22] was recently proposed to fine-tune LLMs directly using preferences. It capitalizes on the relationship between reward functions and optimal policies, effectively tackling the challenge of reward maximization within a single phase of policy training. The D3PO method [31] was recently introduced to fine-tune diffusion models directly. It operates similarly to DPO but is more cost-effective and reduces computational overhead for diffusion model training. However, these methods are not designed for online settings; they assume access to pre-existing large-scale pairwise preference data to optimize parameters. 3 Preliminaries Denoising diffusion models (DMs) generate samples from a learned target distribution by progressively denoising white Gaussian noise. More formally, diffusion models generate samples by reversing a forward diffusion process with T steps that start from a data point x0x_0 and evolve as xt=αtxt−1+(1−αt)ϵ~tx_t= _tx_t-1+ (1- _t) ε_t, t=1,…,Tt=1,…,T, where ϵ~t\ ε_t\ are i.i.d standard Gaussian vectors. Samples from this forward diffusion process can be alternatively expressed as xt=α¯tx0+(1−α¯t)ϵt,ϵt∼(0,I)x_t= α_tx_0+ (1- α_t) _t, _t (0,I) (1) where α¯t=∏s=1tαs α_t= _s=1^t _s. αt\ _t\ are chosen such that α¯t\ α_t\ forms a monotonic sequence with α¯T≈0 α_T≈ 0. This ensures that the density pxTp_x_T is close to the normal distribution (0,I)N(0,I). The reverse diffusion process is learned by modeling the distribution of xt−1x_t-1 given xtx_t as Gaussian with mean [xt−1∣xt]=1αt(xt−1−αt1−α¯tϵθ(xt,t))E[x_t-1 x_t]= 1 _t (x_t- 1- _t 1- α_t _θ(x_t,t) ) (2) and covariance σtI _tI, where ϵθ(⋅,⋅) _θ(·,·) is a neural network and σt\ _t\ are fixed hyperparameters. Training is performed by minimizing the ELBO loss, which simplifies to a series of MSE terms as defined below: L(θ)=∑t=1Tx0,ϵt[‖ϵθ(xt,t)−ϵt‖22].L(θ)= _t=1^TE_x_0, _t [ \| _θ(x_t,t)- _t \|^2_2 ]. (3) Upon reaching the optimal solution, the neural network approximates the posterior mean, which depends on the timestep as ϵθ(xt,t)=[ϵt∣xt=xt]. _θ(x_t,t)=E[ _t x_t=x_t]. To generate samples, DMs sample xT∼(0,I)x_T (0,I) and then iteratively follow the learned reverse probabilities to produce a sample of x0x_0. Specifically, at each timestep t, the DM takes xtx_t as input and predicts the noise ϵt _t from which xt−1x_t-1 is obtained by sampling from the reverse distribution [12]. 4 Problem Formulation Consider a pre-trained Diffusion model, such as DDPM, denoted as gθ∗g_θ^*, with initial parameters θ∗∈Θ⊆ℝdθ^*∈ ^d, where Θ represents the full parameter space. The model has been trained on a specific dataset D, consisting of m independent and identically distributed (i.i.d.) samples xii=1m\x_i\_i=1^m drawn from a distribution XP_X over the data space X. The goal of the model is to learn the underlying data distribution XP_X. For instance, when the goal is to generate natural images, the cardinality of |X||X| is enormous. Although the raw design space is vast, the actual feasible and meaningful solutions typically lie within a complex, yet potentially low-dimensional manifold embedded in X, denoted as feasX_feas. We consider feedback-efficient online fine-tuning of gθ∗g_θ^* for personalized active preference alignment. Specifically, we work in a setting where we do not have any data with feedback initially, but we have a pre-trained diffusion model gθ∗g_θ^* trained on XP_X. We aim to fine-tune gθ∗g_θ^* to produce a sequence of new models gθtftg_θ^ft_t over the period of online interaction so as to maximize the expected user reward maxgθtft∑t=1t=ℬxt∼gθtft[r(xt)], _\g_θ^ft_t\ _t=1^t=BE_x_t g_θ^ft_t[r(x_t)], (4) where r(x)r(x) is the (unknown) user binary reward function with r(x)=1r(x)=1 if the user likes (approves) and r(x)=0r(x)=0 if they dislike the input x, gθtftg_θ^ft_t is the fine-tuned diffusion model used to sample xtx_t at the t’th online interaction step, and ℬB is the number of input queries we can make to the user to elicit associated rewards. We assume that r(x)=0r(x)=0 whenever x is outside the support of the feasible (or meaningful) set of inputs feasX_feas. By leveraging the sequence of fine-tuned models, we aim to efficiently explore the vast sample space X to uncover the user’s diverse preferences, while also exploiting the gathered information to generate samples that align with those preferences. Next, we present our proposed approach to tackle the problem at hand. 5 Algorithm At the very first step of the online interaction process, we leverage the pre-trained diffusion model gθ∗g_θ^* to generate samples 1D^1, since no user feedback data is available at the outset. Based on the recommended samples 1D^1, the user provides binary feedback on each sample to express their preferences. We denote the set of preferred and non-preferred samples as pD_p and npD_np respectively, where 1=p∪npD^1=\D_p _np\. A natural objective for fine-tuning the diffusion model is to maximize the probability of generating samples from pD_p. To this end, we propose a deceptively simple loss function, ℒp(θ,p)L^p(θ,D_p), derived by optimizing the variational lower bound on the negative log-likelihood of the preferred data pD_p, following the approach in [12]. The definition of ℒp(θ,p)L^p(θ,D_p) is as follows: = = γ(∑x0∈p∑t=1T1−αtαt(1−α¯t−1)‖ϵ0−ϵθ(xt,t)‖2)⏟Preference Aligner γ ( _x_0 _p _t=1^T 1- _t _t(1- α_t-1) \| _0- _θ(x_t,t) \|^2 )_Preference Aligner (5) We now demonstrate that optimizing ℒp(θ,p)L^p(θ,D_p) guarantees continuous, monotonic improvement in preference alignment at each fine-tuning step. Theorem 1. Let (x0)P(x_0) be the data distribution from the pre-trained diffusion model, and w(x0)w(x_0) a non-negative weighting function integrable with respect to (x0)P(x_0), proportional to the preference alignment objective. Assume the diffusion model gθg_θ perfectly learns the distribution implied by the preference alignment loss term 5 at each online interaction step, as defined below: ∑x0∈pγ⏟w(x0)(∑t=1T1−αtαt(1−α¯t−1)‖ϵ0−ϵθ(xt,t)‖2) \ _x_0 _pγ\_$w(x_0)$ ( _t=1^T 1- _t _t(1- α_t-1) \| _0- _θ(x_t,t) \|^2 ) Then, the learned data distribution after H online interaction steps is given by: θH(xo)=w(x0)H(x0)NH;NH=∫w(x0)H(x0)x0P^H_θ(x_o)= w(x_0)^HP(x_0)N_H;N_H= w(x_0)^HP(x_0)\,dx_0 Remark: Here, NHN_H is the normalization constant that ensures θH(xo)P^H_θ(x_o) is a valid probability distribution. We provide detailed proof of Th. 1 in the Appendix. According to Th. 1, fine-tuning the diffusion model leads to a strict improvement in preference alignment, assuming w(x0)>0w(x_0)>0. Interestingly, as H→∞H→∞, the learned distribution θH(xo)P^H_θ(x_o) converges to a Dirac-delta function that perfectly aligns with one of the user’s preference (Proof is in the Appendix). Since human preferences are inherently multimodal, ℒp(θ,p)L^p(θ,D_p) alone is insufficient for effective active alignment, highlighting the necessity of a diversity-enhancing objective. Furthermore, to effectively solve the active preference alignment problem, a sample-efficient approach is essential, one that fully leverages all available user feedback. ℒp(θ,p)L^p(θ,D_p), however, overlooks non-preference data, which is crucial for improving the efficiency of preference alignment learning. According to neuroscientists [5], in order to actively learn something new, the brain often needs to “forget” or weaken certain existing neural connections. This is because our brain has limited capacity and must prioritize storing the most relevant information, a process known as “memory consolidation”. Building on these insights, we propose a novel objective function that utilizes non-preference data for active memory consolidation and addresses all the challenges mentioned (such as diversity) in a unified manner. To accomplish this, we draw insights from the Bayesian inference literature. According to [3, 9], fine-tuned parameters θft1θ^1_ft are a sample from the fine-tuned model’s parameter posterior distribution (θ∣p)P(θ _p) i.e., θft1∼(θ∣p)θ^1_ft (θ _p). Similarly, the pre-trained model’s parameters θ∗∼(θ∣p,np)θ^* (θ _p,D_np). Motivated by this, at time step 1, we aim to approximate the posterior distribution of the fine-tuned model’s parameters (θ∣p)P(θ _p) as follows: (θ∣p,np) (θ _p,D_np) ∝(p,np∣θ)(θ) (D_p,D_np θ)P(θ) (6) ∝(np∣θ)(p∣θ)(θ) (D_np θ)P(D_p θ)P(θ) ∝(np∣θ)(θ∣p) (D_np θ)P(θ _p) Eqn. 6 directly follows from Bayes’ rule, neglecting the normalizing constant and assuming independence between pD_p and npD_np. As evident from Eqn. 6, that the posterior distribution (θ∣p)P(θ _p) is intractable, necessitating an approximation in the form Mapp((θ∣p)≈ϕ∗(θ)Mapp(P(θ _p)≈φ^*(θ). Here, Mapp(.)Mapp(.) denotes a projection function that transforms an intractable, unnormalized distribution into a normalized one. We employ variational KL-divergence minimization as Mapp(.)Mapp(.) in our approach, as previous studies [4] have shown it outperforms other inference methods [29, 19, 14] for complex models. Therefore, our approach is framed as a variational KL-divergence minimization over a collection of plausible approximate posterior distributions ψ as follows: ϕ∗(θ)=argminϕ(θ)∈ψKL[ϕ(θ)∥Z⋅(θ∣p,np)(np∣θ)] φ^*(θ)= _φ(θ)∈ψ\,D_KL [φ(θ)\,\|\,Z· P(θ _p,D_np)P(D_np θ) ] (7) Here, Z represents the intractable normalization constant, which does not depend on the parameter θ. Next, we present a theoretical result that provides the foundation for deriving a tractable optimization objective for Eqn. 7. Theorem 2. Assume a Gaussian mean-field approximation in the parameter space, i.e., if the variational prior distribution is ϕ(θ)=∏i=1d(θi,σi2)φ(θ)= _i=1^dN( _i, _i^2) and the posterior distribution with full data is (θ∣np,p)=∏i=1d(μi∗,σi∗2)P(θ _np,D_p)= _i=1^dN( _i^*, _i^*2), then the following holds: KL[ϕ(θ)∥Z⋅(θ∣np,p)(np∣θ)]≳ _KL [φ(θ)\,\|\,Z· P(θ _np,D_p)P(D_np θ) ] (8) −∑x0∈np∑t=2T(1−αt)αt⋅(1−α¯t−1)‖ϵ0−ϵθ(xt,t)‖2 - _x_0 _np _t=2^T (1- _t) _t·(1- α_t-1) \| _0- _θ(x_t,t) \|^2 +∑i=1d[(θi−μi∗)22σi∗2−12+logσi∗σi+σi22σi2]. + _i=1^d [ ( _i- _i^*)^22 _i^*2- 12+log _i^* _i+ _i^22 _i^2 ]. Figure 2: An overview of PAPA at step t. We present the proof in Appendix. Finally, when the variational prior distribution is ϕ(θ)=∏i=1d(θi,σi2)φ(θ)= _i=1^dN( _i, _i^2) and the posterior distribution with full data is (θ∣p,np)=∏i=1d(μ∗,σ∗2)P(θ _p,D_np)= _i=1^dN(μ^*,σ^*2), Eqn. 7 results in the minimization of the following loss function, which we define as the personalized active preference alignment loss ℒnp(θ,θ∗,)L^np(θ,θ^*,D) = = −(∑x0∈np∑t=1T1−αtαt(1−α¯t−1)‖ϵ0−ϵθ(xt,t)‖2)⏟Preference Alignment Enhancer(ℒPAE(θ,np)) - ( _x_0 _np _t=1^T 1- _t _t(1- α_t-1) \| _0- _θ(x_t,t) \|^2 )_Preference Alignment Enhancer(L^PAE(θ,D_np)) (9) +β∑i=1d(θi−μi∗)22σi∗2⏟Quality Preserver & Diversity Enhancer(ℒQPDE(θ∗,θ,p)) + β _i=1^d ( _i- _i^*)^22 _i^*2_Quality Preserver \& Diversity Enhancer(L^QPDE(θ^*,θ,D_p)) The minimization of the ℒnp(θ,θ∗,)L^np(θ,θ^*,D) loss function is equivalent to minimizing Eqn. 7 by realizing that the first term on the right-hand side of the inequality in Theorem 8 resembles the preference alignment enhancer term from the first part of the loss function ℒnp(θ,θ∗,)L^np(θ,θ^*,D) as defined in Eqn. 9. Furthermore, the second term in Theorem 8 appears to be the same as the quality preserver and Diversity Enhancer term in Equation 9 if we assume σi=σi∗ _i=σ^*_i. Thus we utilize the proposed loss function ℒnp(θ,θ∗,)L^np(θ,θ^*,D) for optimizing the pre-trained model during the fine-tuning process. ℒnp(θ,θ∗,)L^np(θ,θ^*,D) consists of two main components: the first, called the “Preference Alignment Enhancer,” minimizes the log-likelihood of the non-preferred data, hence aiding in preference alignment, while the second, the “Quality Preserver and Diversity Enhancer,” penalizes deviations in the model’s parameters to prevent excessive divergence from their pre-trained values during the fine-tuning process, thus ensuring quality preservation. It also boosts sample diversity by letting the model enjoy the core strengths of the pre-trained model, like its diversity, by staying close to its original parameters θ∗θ^*. The sequence αt:t∈T\ _t:t∈ T\ represents the noise scheduler of the diffusion model, where α¯t=∏i=1tαi α_t= _i=1^t _i. Here, ϵ0 _0 denotes the true added noise, ϵθ(xt,t) _θ(x_t,t) is the model’s predicted noise at time t given the noisy sample xtx_t, and d is the dimension of the parameter space. To implement “Quality Preserver and Diversity Enhancer”, we align the representations of the pre-trained (ϵθ∗ _θ^*) and fine-tuned denoisers (ϵθ _θ), both processing the same noisy sample (xtx_t), as defined below: ℒQPDE(θ∗,θ,p)=∑x0∈p∑t=1T∥ϵθ∗(xt,t))−ϵθ(xt,t)∥2 ^QPDE(θ^*,θ,D_p)= _x_0 _p _t=1^T \| _θ^*(x_t,t))- _θ(x_t,t) \|^2 (10) It is important to note that, we consider solely samples from the preference set (pD_p) for this objective, as optimizing it for non-preference set elements (npD_np) contradicts the preference alignment enhancer goal. Finally, our proposed method, PAPA, integrates the preference alignment objectives, ℒPAE(.)L^PAE(.) and ℒP(.)L^P(.), driving exploitation by optimizing the model based on the accumulated preference data, alongside ℒQPDE(.)L^QPDE(.), which fosters exploration by ensuring the fine-tune model remains close to the pre-trained model. We define our final ℒPAPA(.)L_PAPA(.) objective below: ℒp(θ,p)+ℒPAE(θ,np)⏟Exploitation+β⋅ℒQPDE(θ,θ∗,p)⏟Exploration L^p(θ,D_p)+L^PAE(θ,D_np)_Exploitation+β· L^QPDE(θ,θ^*,D_p)_Exploration (11) Algorithm 1 Personalized Active Preference Alignment 1:Input: p=∅D_p= , np=∅D_np= , Pre-trained diffusion model parameter θ∗θ^*, ℬB, α η, γ, Fine-tune model parameter θ1θ^1. 2:Initialize: θ1←θ∗θ^1←θ^* 3:for each interaction step t=1t=1 to ℬB do 4: Generate samples t=pt∪nptD^t=D^t_p ^t_np following a series of reverse-diffusion steps with θtθ^t as the denoiser. 5: Collect user’s preference for samples in tD^t & update 6:p←p∪t∖nptD_p _p∪\D^t ^t_np; np←np∪t∖ptD_np _np∪\D^t ^t_p\. 7: Compute the loss function ℒnp(θt,θ∗,)L^np(θ^t,θ^*,D) and ℒp(θt,p)L^p(θ^t,D_p) as defined in 9 and 5 respectively. 8: Update the fine-tune diffusion model parameters: 9:θt+1←θt−η∇θℒPAPA(θt,θ∗,p,np)θ^t+1←θ^t-η _θL_PAPA( _t,θ^*,D_p,D_np). 10:end for 11:Output: Final parameters θℬ _B. Where β controls the balance between exploration and exploitation. Importantly, β introduces a crucial degree of freedom, allowing independent control over alignment, sample quality, and diversity. This explicit control enables PAPA to remain efficient even when preferences span diverse classes. We outline the PAPA algorithm in 1 and provide an illustrative overview in Fig. 2. Next, we present EPAPA, an enhanced version of PAPA that accelerates the fine-tuning process while reducing the computational cost. [34] observed that a pre-trained diffusion model serves as an effective universal denoiser for lightly corrupted data, adept at identifying and correcting subtle distortions. This capability results in enhanced generation quality when the fine-tuned model is replaced by the original pre-trained one, especially in scenarios with low distortion. However, the suboptimal performance of fine-tuned models for lightly corrupted data points to potential issues such as overfitting, mode collapse, or catastrophic forgetting. Experiments conducted with varying denoising steps demonstrate that the fine-tuning objective should emphasize high-level shaping, which is tied to domain-specific characteristics. Building on this insight, we propose a sampling strategy that leverages the pre-trained model, which excels at low-level denoising, with the fine-tuned model, which specializes in high-level shaping skills. The inverse relationship between the optimality of the pre-trained and fine-tuned diffusion models is formally justified by the following Theorem. Figure 3: Alignment results for diverse preference set from Fashion MNIST dataset. Figure 4: Alignment with different preference set from CIFAR-10 and MNIST. Theorem 3. Suppose a diffusion model with limt→0αt=1andlimt→Tαt=0 _t→ 0 _t=1 _t→ T _t=0 over finite samples. Then the ideal denoiser F satisfies: (1) limt→0F(xt)=argminx0‖x0−xt‖ _t→ 0F(x_t)= _x_0\|x_0-x_t\|, i.e., the closest sample in the dataset. (2) limt→TF(xt)=x0∼pD(x0)[x0] _t→ TF(x_t)=E_x_0 p_D(x_0)[x_0], i.e., the mean of the data distribution. Remark: Derivation for the above Theorem can be found in [34]. We have added the proof in the Appendix for completeness. According to Th. 3, as t→0t→ 0, a model trained on a dataset D can effectively perform zero-shot denoising within the vicinity of the support set supp()supp(D). As the training dataset grows in scale, the coverage of supp()supp(D) broadens, empowering diffusion models to serve as robust zero-shot denoisers for data associated with small t. This implies that a pre-trained diffusion model, trained on a large-scale dataset, is particularly adept at low-level denoising, as lightly corrupted samples remain close to the support of D. Interestingly, as t→Tt→ T, the diffusion model’s generalization power is profoundly shaped by the distribution distance distdist([x0],ft[x0]E_D[x_0],E_D^ft[x_0]), where D is the dataset used for pre-training and ftD^ft represents the fine-tuned dataset, defined as ft=p∪npD^ft=D_p _np. This insight underscores the need for a fine-tuned diffusion model that excels at denoising heavily corrupted data, honing high-level shaping capabilities. Inspired by these observations and theoretical insights, we propose EPAPA, which utilizes the low-level denoising capabilities of a pre-trained model during the reverse diffusion phase, allowing the fine-tuning process (as defined in 9, 5) to focus solely on handling higher levels of noise. Specifically, we exclude the denoising loss terms from both loss functions, ℒnp(θt,θ∗,)L^np(θ^t,θ^*,D) and ℒp(θt,p)L^p(θ^t,D_p), as defined in Eqn. 9 and 5 respectively, for values of t≤Kt≤ K, where 1≪K≪T1 K T. This omission speeds up the fine-tuning process and cuts down on computational costs significantly, making it a more efficient alternative to PAPA. Furthermore, by concentrating exclusively on higher noise levels during the fine-tuning process, we empower the fine-tuned diffusion model to specialize in denoising heavily corrupted data, thereby enhancing its high-level shaping capabilities. As a result, during the reverse diffusion process, we employ the fine-tuned model as the denoiser for higher noise levels (larger t) and the pre-trained model for lower noise levels. Our sampling process, which utilizes both models, is outlined in 2. Moreover, by utilizing a pre-trained model, EPAPA inherently promotes both diversity and quality by inheriting the properties of the pre-trained model. 6 Experiments and Results Algorithm 2 Sampling Strategy of EPAPA 1:Input: xT∼(0,I)x_T (0,I), Pre-trained model θ∗θ^*, Fine-tuned model θ, αtt=0T,σtt=0T\ _t\_t=0^T,\ _t\_t=0^T, K, z∼(0,I)z (0,I). 2:for t=T,T−1,…,1t=T,T-1,…,1 do 3: if t≥Kt≥ K then 4: Calculate ϵθ(xt,t) _θ(x_t,t) using the fine-tuned model. 5: Update xt−1x_t-1 according to: xt−1=1αt(xt−1−αt1−αt¯ϵθ(xt,t))+σtz,x_t-1= 1 _t (x_t- 1- _t 1- _t _θ(x_t,t) )+ _tz, 6: else 7: Calculate ϵθ∗(xt,t) _θ^*(x_t,t) using the pre-trained model. 8: Update xt−1x_t-1 according to: xt−1=1αt(xt−1−αt1−αt¯ϵθ∗(xt,t))+σtz,x_t-1= 1 _t (x_t- 1- _t 1- _t _θ^*(x_t,t) )+ _tz, 9: end if 10:end for 11:Output: Generated sample x0x_0. Table 1: Quality & diversity analysis with diverse pref. sets for MNIST & F-MNIST. MNIST: set s1s_1 Method FID↓ IS↑ Base 39.23± 2.3 2.0±.02 D3PO 41.1± 7.9 2.0±.03 PAPA 25.01± 5.2 2.0±.03 EPAPA 18.87± 5.5 2.0±.05 MNIST: set s2s_2 FID↓ IS↑ 30.09± 12.6 1.8±.08 59.9± 39.8 2.0±.07 24.08± 12.9 1.9±.05 22.98± 12.1 1.9±.06 F-MNIST: set s1s_1 FID↓ IS↑ 119.07± 2.2 2.6±.04 154.33± 50.8 3.4±.54 80.70± 31.4 2.8±.41 36.72± 16.9 2.8±.34 F-MNIST: set s2s_2 FID↓ IS↑ 102.39± 2.38 2.8± 0.06 111.62± 10.1 3.6± 0.11 72.76± 14.2 2.7± 0.54 40.67± 8.8 2.4± 0.30 Figure 5: Success rates across interaction steps for different preference sets from MNIST (left two) and Fashion MNIST (right two). Evaluation Metric Since PAPA aims to align with user preferences while maintaining sample diversity and preserving sample quality, we evaluate its performance using the following metrics, each addressing a specific aspect of the assessment. We evaluate preference alignment using Success Rate (SR), which is the proportion of generated samples that are preferred by the user. Specifically, in the t-th interaction step, SR is calculated as: SR=ptpt+nptSR= D^t_pD^t_p+D^t_np. For a comprehensive evaluation of both the diversity and quality of the generated samples throughout the fine-tuning process, we rely on FID [11] and IS [24] metrics. The FID score is calculated by comparing the generated samples from the fine-tuned model with real samples produced by the pre-trained diffusion model trained solely on the preference set as the real distribution. We compute the FID and IS scores after each fine-tuning step throughout the interaction phase. Once we accumulate these scores, we calculate their mean and standard deviation to summarize the model’s performance. At each online step, we generate 5K samples for evaluation but present only 8 for user feedback, reflecting real-world scenarios where users interact with a limited subset of products before making a decision. This approach models the feedback process with a minimal sample set, aligning with the constraints of actual user interactions. We employ a pre-trained, near-accurate class prediction model or score models as a surrogate for real human feedback. Following sections present details on feedback models and evaluations. Figure 6: Images with Aesthetic Quality and Compressibility as preference. Figure 7: Image Reward as preference. Datasets and Baselines We assess the performance of PAPA and EPAPA across: (i) MNIST, (i) Fashion-MNIST, and (i) CIFAR-10. We further validate PAPA on preference alignment tasks that are hard to capture through text prompts alone, such as image compressibility and aesthetic quality [31]. We demonstrate that our method effectively improves prompt-image alignment through Image Reward [30] on Art subset of Parti Prompt dataset [32]. We benchmark against D3PO [31], a SOTA reward-free baseline. 6.1 Class-targeted Alignment We first evaluate performance on two Fashion-MNIST preference sets with complementary challenges: s1∈Boot,Sandal,Sneakers1∈\Boot,Sandal,Sneaker\, and s2∈Dress,Trousers2∈\Dress,Trouser\. Figure 5 (right) reports SR for both cases, showing that EPAPA’s SR steadily improves with fine-tuning and additional preference data, outperforming the pre-trained Base model. Similar results on MNIST Fig. 5 (left) with diverse preference sets s1∈1,4,7,9s1∈\1,4,7,9\, where the target set is small, and s2∈0,2,3,4,5,6,8,9s2∈\0,2,3,4,5,6,8,9\, which covers most classes, further highlight PAPA’s advantage. Notably, while D3PO performs competitively on s2s2, it fails on s1s1, indicating PAPA’s robustness under larger distribution shifts. D3PO struggles to align diverse human preferences primarily because, unlike PAPA, it lacks an explicit mechanism to independently control sample quality and diversity apart from alignment. Additional analysis in the appendix explaining why D3PO struggles to align diverse preferences. We evaluate sample quality and diversity using FID and IS, with results presented in Table 1. The result indicates that our approach effectively maintains high quality (as our approach achieves a similar IS score compared to Base (trained on pref. set)) while ensuring diversity within the preferred set (as suggested by lower FID), as visualized in Figure 4. These empirical outcomes also reveal the relative benefits of EPAPA compared to PAPA. Therefore, unless stated otherwise, we use EPAPA with K=400K=400 throughout the paper. Additionally, we evaluate our approach on CIFAR-10 and observe consistent results (in Appendix). Fig. 4 and Fig. 4 showcases EPAPA-generated samples, further confirming its efficacy in producing preference-aligned samples. Qualitative comparisons and more generated samples are in the Appendix. Figure 8: Reward comparison on fine-grained objective and prompt-image alignment. Table 2: Effect of ℒQPDEL^QPDE (top), K (bottom). Obj. FID↓ IS↑ ℒnoqpL^noqp 39.93 ± 18.3 2.8 ± 0.38 ℒPAPAL_PAPA 36.72 ± 16.9 2.9 ± 0.34 K FID↓ IS↑ 100 50.62± 13.25 2.9± 0.30 400 36.72± 16.9 2.9± 0.34 700 435.40± 106.5 1.4± 0.70 Figure 9: Ablation ℒPAEL^PAE. Figure 10: Qualitative performance comparison across variants with individual components removed. Results validate the necessity of each component. 6.2 Fine Grained Preference Alignment We further evaluate the efficacy of PAPA on fine-grained preference alignment tasks with pre-defined quantifiable objectives. We consider three diverse alignment tasks: a) Compressibility, in which an image with a smaller size is regarded as better; b) Aesthetic Quality, in which we use the LAION aesthetic score predictor [25] to automatically assign aesthetic ratings to images, enabling objective reward assignment based on visual quality without requiring human evaluation; c) Prompt-Image Alignment, that assesses how well the generated images align with the given text prompts. We use Image Reward [30], a general-purpose text-to-image human preference score, to generate feedback and evaluation. For these analysis, we use Stable Diffusion v1.5 as the base model and compare its performance with D3PO [31]. Our experimental findings, as reported in Fig. 8, indicate that PAPA rapidly adapts to the preference with comparatively fewer online feedback responses than D3PO (we train on 10-20 times less number of samples), justifying its suitability for sophisticated preference alignment in interactive environments. In Figs. 7 and 7, we qualitatively compare samples generated from identical Gaussian noises to ensure a fair comparison between the two models. PAPA consistently better reflects the qualitative criteria of the preference objective and is less prone to deformities. For the compressibility preference, although it is quantified via file size, qualitatively it corresponds to images with simplified subject details and smoother, often single-color backgrounds while preserving overall structure. In Fig. 7 (rabbit and bear examples), D3PO retains fine details and often introduces structural distortions, whereas PAPA produces coherent, meaningful shapes and removes superfluous details leading to smaller-size images. For the aesthetic preference, which is measured via the LAION aesthetic score [25], PAPA generates images with sharp subject details, blurred backgrounds (bokeh), appealing composition, and harmonious colors. In Fig. 7 (horse and fox examples), D3PO exhibits various deformities and lack of vibrancy, while PAPA preserves body structure, adds stylistic components, enhances fine details, and more clearly separates foreground from background, demonstrating rapid adaptation across fine-grained alignment tasks. Figure 7 shows that PAPA generates more semantically aligned images in text-to-image setting. It is evident from the generated images that PAPA outperforms D3PO considering artistic style, object instantiation, spatial arrangement, concept realization, and fidelity. 6.3 Analysis and Ablation ℒQPDEL^QPDE as Quality Preserver & Sample Diversity: To analyze the role of ℒQPDEL^QPDE in preserving quality and diversity, we compare our approach with a variant, ℒnoqpL^noqp, which is identical to ours but excludes the ℒQPDEL^QPDE loss term from ℒnpL^np as defined in Eqn. 9. For this comparison, we select \Sandal, Sneaker, Boot\ as the preference set, with results shown in Table 2 (top). Omitting the ℒQPDEL^QPDE term from ℒnpL^np leads to a notable decline in both FID and IS scores. Qualitative visualizations, such as the comparison between Figures 10(a) and 10(c), further highlight how the quality-preserving term maintains the structure of generated samples while ensuring sample diversity. Additionally, we observe that excluding ℒQPDEL^QPDE does not affect SR, as both ℒnoqpL^noqp and ℒPAPAL_PAPA achieves similar SR across the interaction steps, as discussed in the Appendix with visualizations. ℒPAEL^PAE as Preference Alignment Enhancer: To assess the impact of ℒPAEL^PAE on preference alignment, we compare our approach with a variant, ℒnopaL^nopa, which is identical to ours but omits the ℒPAEL^PAE term from ℒnpL^np as defined in Eqn. 9. For this comparison, we select \Sneaker, Sandal, Boot\ as the preference set, with results presented in Figure 9. Excluding ℒPAEL^PAE from ℒnpL^np results in a consistent and substantial decline in SR, underscoring the importance of ℒPAEL^PAE as the preference-alignment enhancer. Qualitative visualizations (see Figures 10(b) and 10(c)) further demonstrate how the inclusion of this term helps produce samples that better align with the user preferences. Effect of K: To realize the optimal value of K, we conduct experiments with different choices of K, and compare the performance in terms of FID and IS scores. For this analysis, we select \Sandal, Sneaker, Boot\ as the preference set, with results in Table 2 (down). Our experiments reveal that both extremely high and low values of K are ineffective. A very high K is problematic because pre-trained diffusion models struggle to denoise effectively at elevated noise levels. Conversely, a very low K is also not ideal, as these fine-tuned models are optimized for higher noise levels and struggle with low noise levels. Therefore, an optimal range for K lies in the middle, where the model performs most effectively. More analysis on the effect of K, β, and PAPA’s generalizability under non-binary feedback is in Appendix. 7 Conclusion We introduce a novel feedback-efficient approach for fine-tuning diffusion models to achieve personalized active preference alignment. It enables diffusion models to align actively with diverse human preferences while preserving the quality of generated samples. We validate its effectiveness through comprehensive experiments and ablation studies across diverse class-conditioned and fine-grained alignment tasks. We hope this work will pave the way for future research into personalized active preference alignment. Deploying PAPA across a range of scientific problems, from drug discovery to materials generation, would be a potential direction to explore. Acknowledgments: This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Advanced Scientific Computing Research programs in the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This work was partially supported by the NSF (IIS-2214141), ARO (W911NF-25-1-0059), ONR (N000142412663), Foresight Institute, and Amazon. We thank William Hsu and Beocat High-Performance Computing (HPC) cluster at Kansas State University for initial compute support. This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan ( https://w.energy.gov/doe-public-access-plan ). References Ajay et al. [2022] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022. Black et al. [2023] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023. Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613–1622. PMLR, 2015. Bui et al. [2016] Thang Bui, Daniel Hernández-Lobato, Jose Hernandez-Lobato, Yingzhen Li, and Richard Turner. Deep gaussian processes for regression using approximate expectation propagation. In International conference on machine learning, pages 1472–1481. PMLR, 2016. Davis and Zhong [2017] Ronald L Davis and Yi Zhong. The biology of forgetting—a perspective. Neuron, 95(3):490–503, 2017. Dong et al. [2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. Fan and Lee [2023] Ying Fan and Kangwook Lee. Optimizing ddpm sampling with shortcut fine-tuning. arXiv preprint arXiv:2301.13362, 2023. Fan et al. [2024] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024. Ghahramani and Attias [2000] Zoubin Ghahramani and H Attias. Online variational bayesian learning. In Slides from talk presented at NIPS workshop on Online Learning, 2000. He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. Janner et al. [2022] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022. Knoblauch et al. [2022] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on bayes’ rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(132):1–109, 2022. Lee et al. [2023] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023. Liu et al. [2023] Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Zero-shot preference learning for offline rl via optimal transport. arXiv preprint arXiv:2306.03615, 2023. Lu et al. [2022] C Lu, Y Zhou, F Bao, J Chen, and C Li. A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Proc. Adv. Neural Inf. Process. Syst., New Orleans, United States, pages 1–31, 2022. MacKay [2003] David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003. Nguyen et al. [2017] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10628, 2017. Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Peng et al. [2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part I 18, pages 234–241. Springer, 2015. Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294, 2022. Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Uehara et al. [2024a] Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous-time diffusion models as entropy-regularized control. arXiv preprint arXiv:2402.15194, 2024a. Uehara et al. [2024b] Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Sergey Levine, and Tommaso Biancalani. Feedback efficient online fine-tuning of diffusion models. arXiv preprint arXiv:2402.16359, 2024b. Wild et al. [2022] Veit David Wild, Robert Hu, and Dino Sejdinovic. Generalized variational inference in function spaces: Gaussian measures meet bayesian deep learning. Advances in Neural Information Processing Systems, 35:3716–3730, 2022. Xu et al. [2024] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024. Yang et al. [2024] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941–8951, 2024. [32] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research. Zeiler [2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. Zhong et al. [2024] Jincheng Zhong, Xingzhuo Guo, Jiaxiang Dong, and Mingsheng Long. Diffusion tuning: Transferring diffusion models via chain of forgetting. arXiv preprint arXiv:2406.00773, 2024. PAPA: Online Personalized Active Preference Alignment Appendix A Omitted Proofs A.1 Proof of Theorem 4.1 Proof. Assume the distribution induced by the pre-trained generative model (x0)P(x_0). Given the standard DDPM loss function: ℒ(θ)=x0∼(x0)[∑t=1T1−αtαt(1−α¯t−1)‖ϵ0−ϵθ(xt,t)‖2]⏟Let’s denote it as L(x0)L(θ)=E_x_0 (x_0) [ _t=1^T 1- _t _t(1- α_t-1) \| _0- _θ(x_t,t) \|^2 ]_Let's denote it as $L(x_0)$ Here, L(x0)L(x_0) is a loss function defined for sample x0x_0. Now, if we fine-tune the parameters of the diffusion model (i.e., θ) with the objective defined in 1, then according to [21] the induced data distribution induced(x0)P^induced(x_0) can be defined as: induced(x0)∝(x0)exp(−γL(x0))P^induced(x_0) (x_0) (-γ L(x_0)) with γ>0γ>0 being a positive constant. Now, consider a weighted probability distribution weighted(x0)=w(x0)(x0)P^weighted(x_0)=w(x_0)P(x_0). In that case, we can represent the induced distribution induced(x0)P^induced(x_0) in terms of (x0)P(x_0), as follows: induced(x0)∝w(x0)(x0)exp(−γL(x0))P^induced(x_0) w(x_0)P(x_0) (-γ L(x_0)) We derive 3, by observing that ℒ(θ)=x0∼weighted(x0)[L(x0)]L(θ)=E_x_0 ^weighted(x_0)[L(x_0)] This, in turn, implies that induced(x0)∝weighted(x0)⏟w(x0)(x0)exp(−γL(x0))P^induced(x_0) P^weighted(x_0)_$w(x_0)P(x_0)$ (-γ L(x_0)) We can re-write ℒp(θ)L^p(θ) as follows: =t∼U[0,1],x0∼(x0),xt∼pt(xt|x0)[w(x0)⋅1−αtαt(1−α¯t−1)⏟= c > 0∥ϵθ(xt,t)−ϵ0)∥2]=E_t U[0,1],x_0 (x_0),x_t p_t(x_t|x_0) [w(x_0)· 1- _t _t(1- α_t-1)_= c > 0\| _θ(x_t,t)- _0)\|^2 ] Where w(x0)=γifx0∈P,elsew(x0)=0.w(x_0)=γ x_0 _P, w(x_0)=0. Note that, γ>0γ>0. Assuming, there is at least a single element in pD_p, implies w(x0)>0w(x_0)>0. ℒp(θ)=∫01∫w(x0)(x0)∫∥ϵθ(xt;t)−ϵ0)∥2t(xt|x0)dxtdx0dtL^p(θ)= _0^1 _Xw(x_0)P(x_0) _X\| _θ(x_t;t)- _0)\|^2P_t(x_t|x_0)\,dx_t\,dx_0\,dt Then, we can define the reweighted distribution pinduced(x0)p^induced(x_0) as: pinduced(x0)=w(x0)(x0)Z,whereZ=∫w(x0)(x0)x0p^induced(x_0)= w(x_0)P(x_0)Z, Z= w(x_0)P(x_0)\,dx_0 Since w(x0)≥0w(x_0)≥ 0 and Z<∞Z<∞, pinduced(x0)p^induced(x_0) is a valid pdf over X. Substituting pinduced(x0)p^induced(x_0) (from 6) into the loss function in 5, we have: ℒp(θ)=Z∫01∫pinduced(x0)∫∥ϵθ(xt;t)−ϵ0)∥2t(xt|x0)dxtdx0dtL^p(θ)=Z _0^1 _Xp^induced(x_0) _X\| _θ(x_t;t)- _0)\|^2P_t(x_t|x_0)\,dx_t\,dx_0\,dt We can re-write the above expression as: =Z⋅t∼U[0,1],x0∼induced(x0),xt∼pt(xt|x0)[∥ϵθ(xt,t)−ϵ0)∥2]=Z·E_t U[0,1],x_0 ^induced(x_0),x_t p_t(x_t|x_0) [\| _θ(x_t,t)- _0)\|^2 ]\>\>\>\> Therefore, the gradient of the above expression is: ℒp(θ)=(Z)t∼U[0,1],x0∼induced(x0),xt∼pt(xt|x0)[∇θ∥ϵθ(xt,t)−ϵ0)∥2]L^p(θ)=(Z)\>E_t U[0,1],x_0 ^induced(x_0),x_t p_t(x_t|x_0) [ _θ\| _θ(x_t,t)- _0)\|^2 ] Note that the normalizing factor Z does not depend on the optimization variable θ. Hence, it does not affect the optimization process. Therefore, minimizing ℒp(θ)L^p(θ) is equivalent to minimizing the expected loss under the distribution induced(x0)P^induced(x_0). Following a similar result as in Equation 2, we can express the learned data distribution by optimizing Equation 9, denoted as θ1(x0)P^1_θ(x_0) as follows: θ1(x0)∝induced(x0)exp(−γℒp(x0;θ))P^1_θ(x_0) ^induced(x_0) (- ^p(x_0;θ)) Utilizing the relation from Equation 6, we can write: θ1(x0)∝w(x0)(x0)exp(−γℒp(x0;θ))P^1_θ(x_0) w(x_0)P(x_0) (- ^p(x_0;θ)) Assuming the ℒp(x0;θ)L^p(x_0;θ) loss converges to 0 after the fine-tuning step, we can write the above expression as: θ1(x0)∝w(x0)(x0)P^1_θ(x_0) w(x_0)P(x_0) By normalizing, with normalization constant Z, we can write: θ1(x0)=w(x0)(x0)ZP^1_θ(x_0)= w(x_0)P(x_0)Z We can now iteratively repeat these steps, and by induction, express the learned data distribution after H update steps (i.e., H online interaction steps) as follows: θH(x0)=w(x0)H(x0)ZH,where,ZH=∫w(x0)H(x0)x0P^H_θ(x_0)= w(x_0)^HP(x_0)Z^H,where,Z^H= _Xw(x_0)^HP(x_0)dx_0 This completes the proof. ∎ A.2 Limiting case: as H→∞H→∞ Proof. (Limiting Case). Let’s now consider an interesting limiting case where H→∞H→∞ : Assume w(x0)w(x_0) attains its maximum at x0∗x_0^*. We define ϕ(x0)=w(x0)w(x0∗)φ(x_0)= w(x_0)w(x_0^*) which implies ϕ(x0∗)=1φ(x_0^*)=1 since w(x0∗)w(x0∗)=1 w(x_0^*)w(x_0^*)=1, and 0≤ϕ(x0)<10≤φ(x_0)<1 for x0≠x0∗x_0≠ x_0^*, since w(x0)<w(x0∗)w(x_0)<w(x_0^*). Then, we can rewrite θH(x0)P^H_θ(x_0) using ϕ(x0)φ(x_0) as: θH(x0)=[w(x0∗)]Hϕ(x0)H(x0)ZHP^H_θ(x_0)= [w(x_0^*)]^Hφ(x_0)^HP(x_0)Z^H Then, for x0≠x0∗x_0≠ x_0^*, we have ϕ(x0)H→0φ(x_0)^H→ 0 as H→∞H→∞ since ϕ(x0)<1φ(x_0)<1. Thus, θH(x0)→0asH→∞,∀x0≠x0∗P^H_θ(x_0)→ 0\>\>\>\>as\>H→∞,∀ x_0≠ x_0^* And for x0=x0∗x_0=x_0^*, we have ϕ(x0∗)H=1φ(x_0^*)^H=1, and θH(x0∗)=[w(x0∗)]H(x0∗)ZHP^H_θ(x_0^*)= [w(x_0^*)]^HP(x_0^*)Z^H. The normalization constant can be written as: ZH=∫w(x0)H(x0)x0=[w(x0∗)]H∫ϕ(x0)H(x0)x0Z^H= _Xw(x_0)^HP(x_0)\,dx_0=[w(x^*_0)]^H _Xφ(x_0)^HP(x_0)dx_0 Similarly, we can obtain ZH≈[w(x0∗)]H(x0∗)Z^H≈[w(x_0^*)]^HP(x_0^*). As H→∞H→∞: Then, we have the limit behavior: For x0≠x0∗x_0≠ x_0^*: θH(x0)=[w(x0∗)]Hϕ(x0)H(x0)[w(x0∗)]H(x0∗)=ϕ(x0)H(x0)(x0∗)→0.P^H_θ(x_0)= [w(x_0^*)]^Hφ(x_0)^HP(x_0)[w(x_0^*)]^HP(x_0^*) = φ(x_0)^HP(x_0)P(x_0^*)→ 0. For x0=x0∗x_0=x_0^*: θH(x0∗)=[w(x0∗)]H(x0∗)[w(x0∗)]H(x0∗)=1.P^H_θ(x_0^*)= [w(x_0^*)]^HP(x_0^*)[w(x_0^*)]^HP(x_0^*) =1. Therefore, we conclude: limH→∞θH(x0)=δ(x0−x0∗). _H→∞P^H_θ(x_0)=δ(x_0-x_0^*). ∎ A.3 Proof of Theorem 4.2 Proof. The objective function in equation 7 (in the main paper) can be expressed as follows: DKL(ϕ(θ)∥Z⋅(θ|p,np)(np∣θ))=ϕ(θ)[lnϕ(θ)(np∣θ)Z⋅(θ|p,np)]D_KL (φ(θ)\, \,Z· P(θ|D_p,D_np)P(D_np θ) )=E_φ(θ) [ φ(θ)P(D_np θ)Z·P(θ|D_p,D_np) ] =ϕ(θ)[lnϕ(θ)(θ|p,np)]+ϕ(θ)[ln(np|θ)]; (we ignore Z as it is independent of θ)=E_φ(θ) [ φ(θ)P(θ|D_p,D_np) ]+E_φ(θ) [ (D_np|θ) ]; (we ignore $Z$ as it is independent of $θ$) =ϕ(θ)[lnϕ(θ)(θ|p,np)]+ϕ(θ)[∑x0∈npln(x0|θ)⏟assuming i.i.d assumption on the data]=E_φ(θ) [ φ(θ)P(θ|D_p,D_np) ]+E_φ(θ) [ _x_0 _np (x_0|θ)_assuming i.i.d assumption on the data ] =DKL(ϕ(θ)∥(θ|p,np))⏟term-I+ϕ(θ)[∑x0∈npln(x0|θ)]⏟term-I= D_KL (φ(θ)\, \,P(θ|D_p,D_np) )_term-I+ E_φ(θ) [ _x_0 _np (x_0|θ) ]_term-I Now, let’s focus on term-I. The term-I in Eq. (4) with the parameter prior distribution ϕ(θ)=∏i=1d(θi,σ2)φ(θ)= _i=1^dN( _i,σ^2) and the posterior distribution with full data (θ|p,np)=∏i=1d(μ∗,σ∗2)P(θ|D_p,D_np)= _i=1^dN(μ^*,σ^*2) becomes: DKL(ϕ(θ)∥(θ|np,p))=∑i=1d(lnσi∗σi+σi2+(θi−μi∗)22σi∗2−12)D_KL(φ(θ) (θ|D_np,D_p))= _i=1^d ( _i^* _i+ σ^2_i+( _i- _i^*)^22 _i^*2- 12 ) Equation (5) is derived using the following well-known standard lemma, which is presented in [18] and stated as follows: Lemma 4. The Kullback-Leibler divergence for two multivariate normal distributions can be expressed as follows: DKL((x;μx,Σx)∥(y;μy,Σy))=12( D_KL\! (N(x; _x, _x) (y; _y, _y) )= 12 ( log|Σy|−log|Σx|−d _y - _x -d +tr(Σy−1Σx) +tr( _y^-1 _x) +(μy−μx)⊤Σy−1(μy−μx)) +( _y- _x) _y^-1( _y- _x) ) Next, term−IIterm-I can be expressed using Monte Carlo estimation as follows: ϕ(θ)[∑xo∈npln(x0|θ)]≈1B∑b=1Bln(x0|θm)E_φ(θ) [ _x_o _np (x_0|θ) ]≈ 1B _b=1^B (x_0| _m) ≥1B∑b=1B[−∑x0∈np∑t=2Tq(xt∣xo)DKL[(q(xt−1|xt,x0)∥pθ(xt−1|xt))]]≥ 1B _b=1^B [- _x_0 _np _t=2^TE_q(x_t x_o)D_KL[ (q(x_t-1|x_t,x_0)\|p_θ(x_t-1|x_t) )] ] Equation (7) is derived using the following Lemma [17]: Lemma 5. The log-likelihood under the backward diffusion process kernel is given by: lnpθ(x0)≥−∑t=2Tq(xt∣x0)[DKL(q(xt−1∣xt,x0)∥pθ(xt−1∣xt))] p_θ(x_0)≥- _t=2^TE_q(x_t x_0) [D_KL (q(x_t-1 x_t,x_0)\|p_θ(x_t-1 x_t) ) ] Proof. Let x0x_0 denote the true data sample. In order to increase the log-likelihood of the data, we maximize the ELBO as follows: lnp(x0)=ln∫p(x0:T)x1:T p(x_0)= p(x_0:T)\,dx_1:T\, =ln∫p(x0:T)q(x1:T∣x0)q(x1:T∣x0)x1:T= p(x_0:T)q(x_1:T x_0)\,q(x_1:T x_0)dx_1:T\, =lnq(x1:T∣xo)p(x0:T)q(x1:T∣x0)= _q(x_1:T x_o) p(x_0:T)q(x_1:T x_0) ≥q(x1:T∣xo)[lnp(x0:T)q(x1:T∣x0)](by applying Jensen’s inequality) _q(x_1:T x_o) [ p(x_0:T)q(x_1:T x_0) ] (by applying Jensen's inequality) =q(x1:T∣xo)[lnp(xT)∏t=1Tpθ(xt−1∣xt)∏t=1Tq(xt∣xt−1)]=E_q(x_1:T x_o) [ p(x_T) _t=1^Tp_θ(x_t-1 x_t) _t=1^Tq(x_t x_t-1) ] (Utilizing markovian property of forward process) =q(x1:T∣xo)[lnp(xT)pθ(x0∣x1)∏t=2Tpθ(xt−1∣xt)q(x1∣x0)∏t=2Tq(xt∣xt−1)]=E_q(x_1:T x_o) [ p(x_T)p_θ(x_0 x_1) _t=2^Tp_θ(x_t-1 x_t)q(x_1 x_0) _t=2^Tq(x_t x_t-1) ] =q(x1:T∣xo)[lnp(xT)pθ(x0∣x1)∏t=2Tpθ(xt−1∣xt)q(x1∣x0)∏t=2Tq(xt∣xt−1,x0)]=E_q(x_1:T x_o) [ p(x_T)p_θ(x_0 x_1) _t=2^Tp_θ(x_t-1 x_t)q(x_1 x_0) _t=2^Tq(x_t x_t-1,x_0) ] =q(x1:T∣xo)[lnp(xT)pθ(x0∣x1)q(x1∣x0)+ln∏t=2Tpθ(xt−1∣xt)q(xt∣xt−1,x0)]=E_q(x_1:T x_o) [ p(x_T)p_θ(x_0 x_1)q(x_1 x_0)+ _t=2^T p_θ(x_t-1 x_t)q(x_t x_t-1,x_0) ] =q(x1:T∣xo)[lnp(xT)pθ(x0∣x1)q(x1∣x0)+ln∏t=2Tpθ(xt−1∣xt)q(xt−1∣xt,x0)q(xt∣x0)q(xt−1∣x0)](Bayes’ Rule)=E_q(x_1:T x_o) [ p(x_T)p_θ(x_0 x_1)q(x_1 x_0)+ _t=2^T p_θ(x_t-1 x_t) q(x_t-1 x_t,x_0)q(x_t x_0)q(x_t-1 x_0) ] (Bayes' Rule) =q(x1:T∣xo)[lnp(xT)pθ(x0∣x1)q(x1∣x0)+ln∏t=2Tpθ(xt−1∣xt)q(xt−1∣xt,x0)+lnq(x1∣x0)q(xT∣x0)]=E_q(x_1:T x_o) [ p(x_T)p_θ(x_0 x_1)q(x_1 x_0)+ _t=2^T p_θ(x_t-1 x_t)q(x_t-1 x_t,x_0)+ q(x_1 x_0)q(x_T x_0) ] =q(x1:T∣xo)[lnp(xT)pθ(x0∣x1)q(xT∣x0)+∑t=2Tlnpθ(xt−1∣xt)q(xt−1∣xt,x0)]=E_q(x_1:T x_o) [ p(x_T)p_θ(x_0 x_1)q(x_T x_0)+ _t=2^T p_θ(x_t-1 x_t)q(x_t-1 x_t,x_0) ] =q(x1:T∣x0)[lnpθ(x0∣x1)]+q(x1:T∣x0)[lnp(xT)q(xT∣x0)] =E_q(x_1:T x_0)[ p_θ(x_0 x_1)]+E_q(x_1:T x_0) [ p(x_T)q(x_T x_0) ] +∑t=2Tq(x1:T∣x0)[lnpθ(xt−1∣xt)q(xt−1∣xt,x0)] + _t=2^TE_q(x_1:T x_0) [ p_θ(x_t-1 x_t)q(x_t-1 x_t,x_0) ] =q(x1∣x0)[lnpθ(x0∣x1)]+q(xT∣x0)[lnp(xT)q(xT∣x0)] =E_q(x_1 x_0)[ p_θ(x_0 x_1)]+E_q(x_T x_0) [ p(x_T)q(x_T x_0) ] +∑t=2Tq(xt,xt−1∣x0)[lnpθ(xt−1∣xt)q(xt−1∣xt,x0)] + _t=2^TE_q(x_t,x_t-1 x_0) [ p_θ(x_t-1 x_t)q(x_t-1 x_t,x_0) ] =q(x1∣x0)[lnpθ(x0∣x1)]−DKL(q(xT∣x0)||p(xT)) =E_q(x_1 x_0)[ p_θ(x_0 x_1)]-D_KL(q(x_T x_0)||p(x_T)) −∑t=2Tq(xt∣x0)[DKL(q(xt−1∣xt,x0)||pθ(xt−1∣xt))] - _t=2^TE_q(x_t x_0)[D_KL(q(x_t-1 x_t,x_0)||p_θ(x_t-1 x_t))] =−∑t=2Tq(xt∣x0)[DKL(q(xt−1∣xt,x0)||pθ(xt−1∣xt))]=- _t=2^TE_q(x_t x_0)[D_KL(q(x_t-1 x_t,x_0)||p_θ(x_t-1 x_t))] (Other terms can be ignored as they are insignificant.) ∎ Before continuing with term−IIterm-I, we present a well-known lemma, which can be found in any diffusion model literature, such as in [17]: Lemma 6. Assuming that in the forward diffusion process, the transition kernel is denoted as q(xt|xt−1)q(x_t|x_t-1), with the joint posterior distribution given by q(x1:T|x0)=∏t=1Tq(xt|xt−1),q(x_1:T|x_0)= _t=1^Tq(x_t|x_t-1), where each q(xt|xt−1)=(xt;αtxt−1,(1−αt)I)q(x_t|x_t-1)=N(x_t; _tx_t-1,(1- _t)I). Similarly, for the backward diffusion process, the transition kernel is denoted as p(xt−1|xt)p(x_t-1|x_t), with the joint distribution p(x0:T)=p(xT)∏t=1Tpθ(xt−1|xt),p(x_0:T)=p(x_T) _t=1^Tp_θ(x_t-1|x_t), where p(xT)=(xT;0,I)p(x_T)=N(x_T;0,I). Thus, after optimizing the diffusion model, the sampling procedure proceeds by sampling Gaussian noise from p(xT)p(x_T) and iteratively applying the denoising transitions pθ(xt−1|xt)p_θ(x_t-1|x_t) for T steps to generate a new sample x0x_0. Assuming all the transition kernels are Gaussian, the following holds: • q(xt−1|xt,x0)=(xt−1;μq(t),σq2(t)I)withμq(t)=1αtxt−1−αt1−αt¯αtϵ0q(x_t-1|x_t,x_0)=N(x_t-1; _q(t), _q^2(t)I) _q(t)= 1 _tx_t- 1- _t 1- _t _t _0 • pθ(xt−1|xt)=(xt−1;μθ(t),σq2(t)I)w/μθ(t)=1αtxt−1−αt1−αt¯αt⋅ϵθ(xt,t)p_θ(x_t-1|x_t)=N(x_t-1; _θ(t), _q^2(t)I) / _θ(t)= 1 _tx_t- 1- _t 1- _t _t· _θ(x_t,t) • σq2(t)=(1−αt)(1−α¯t−1)(1−α¯t) _q^2(t)= (1- _t)(1- α_t-1)(1- α_t) Now, utilizing the results of Lemma .1 and .3, we can rewrite expression 7 as follows: ϕ(θ)[∑xo∈npln(x0|θ)]≥ _φ(θ) [ _x_o _np (x_0|θ) ]≥ 1B∑b=1B[−∑x0∈np∑t=2Tq(xt∣xo)DKL[(q(xt−1|xt,x0)∥pθ(xt−1|xt))]] 1B _b=1^B [- _x_0 _np _t=2^TE_q(x_t x_o)D_KL[ (q(x_t-1|x_t,x_0)\|p_θ(x_t-1|x_t) )] ] ≥1B∑b=1B[−∑x0∈np∑t=2Tq(xt∣xo)DKL((xt−1;μq,Σq(t))∥(xt−1;μθ,Σq(t)))]≥ 1B _b=1^B [- _x_0 _np _t=2^TE_q(x_t x_o)D_KL (N(x_t-1; _q, _q(t)) (x_t-1; _θ, _q(t)) ) ] ≥ ≥ 1B∑b=1B[−∑x0∈np∑t=2Tq(xt∣xo)∥12σq2(t)(1αtxt−1−αt1−αt¯αtϵθ(xt,t) 1B _b=1^B [- _x_0 _np _t=2^TE_q(x_t x_o) \| 12 _q^2(t) ( 1 _tx_t- 1- _t 1- _t _t\, _θ(x_t,t) −1αtxt+1−αt1−αt¯αtϵ0)∥22] 128.0374pt- 1 _tx_t+ 1- _t 1- _t _t\, _0 ) \|_2^2 ] ≥1B∑b=1B[−∑x0∈np∑t=2Tq(xt∣xo)(1−αt)22σq2(t)(1−α¯t)(αt)‖ϵ0−ϵθ(xt,t)‖22]≥ 1B _b=1^B [- _x_0 _np _t=2^TE_q(x_t x_o) (1- _t)^22 _q^2(t)(1- α_t)( _t) \| _0- _θ(x_t,t) \|^2_2 ] (By rearranging the terms.) ≥1B∑b=1B[−∑x0∈np∑t=2Tq(xt∣xo)(1−αt)(1−α¯t−1)(αt)‖ϵ0−ϵθ(xt,t)‖22]≥ 1B _b=1^B [- _x_0 _np _t=2^TE_q(x_t x_o) (1- _t)(1- α_t-1)( _t) \| _0- _θ(x_t,t) \|^2_2 ] (Using the expression of σq2(t) _q^2(t).) ≥−∑x0∈np∑t=2T(1−αt)(1−α¯t−1)(αt)‖ϵ0−ϵθ(xt,t)‖22(Using the expression of σq2(t).)≥- _x_0 _np _t=2^T (1- _t)(1- α_t-1)( _t) \| _0- _θ(x_t,t) \|^2_2(Using the expression of $ _q^2(t)$.) By combining Equation 5 and Equation 12, we can write: KL[ϕ(θ)∥Z⋅(θ∣np,p)(np∣θ)]≳ _KL [φ(θ)\,\|\,Z· P(θ _np,D_p)P(D_np θ) ] −∑x0∈np∑t=2T(1−αt)αt⋅(1−α¯t−1)‖ϵ0−ϵθ(xt,t)‖2 - _x_0 _np _t=2^T (1- _t) _t·(1- α_t-1) \| _0- _θ(x_t,t) \|^2 +∑i=1d[(θi−μi∗)22σi∗2−12+logσi∗σi+σi22σi2]. + _i=1^d [ ( _i- _i^*)^22 _i^*2- 12+log _i^* _i+ _i^22 _i^2 ]. ∎ A.4 Proof of Theorem 4.3 Proof. Let’s assume that the dataset consists of finite bounded samples =x(1),x(2),…,x(n)D=\x^(1),x^(2),…,x^(n)\, and f is the denoiser designed to minimize L(θ)L(θ) as defined below: L(θ)=t,x0,ϵ[‖ϵ−fθ(αtx0+(1−αt)ϵ)‖2]L(θ)=E_t,x_0,ε [ \|ε-f_θ ( _tx_0+ (1- _t)ε ) \|^2 ] Where x0∼x_0 represents real samples, ϵ∼(0,I)ε (0,I) denotes the noise signal, and xt=αtx0+1−αtϵx_t= _tx_0+ 1- _tε is the perturbed sample at timestep t. Sampling from diffusion models follows a Markov chain, which iteratively denoises from xT∼(0,I)x_T (0,I) to x0x_0. For convenience, we convert the denoiser into an x0x_0-parameterization by rearranging the terms and defining F(xt)=xt−1−αtf(xt)αtF(x_t)= x_t- 1- _tf(x_t) _t, and the objective becomes: L=t,x0,xt[‖x0−F(xt)‖2]L=E_t,x_0,x_t [\|x_0-F(x_t)\|^2 ] An ideal denoiser F should minimize the value F(xt)F(x_t) for all t,xtt,x_t, implying an objective for F(xt)F(x_t): Lt,xt(F(xt))=x0∼p(x0|xt)[‖x0−F(xt)‖2].L_t,x_t(F(x_t))=E_x_0 p(x_0|x_t) [\|x_0-F(x_t)\|^2 ]. By taking the derivative, it holds that 0=∇F(xt)Lt,xt(F(xt))=x0∼p(x0|xt)[−2(x0−F(xt))].0= _F(x_t)L_t,x_t(F(x_t))=E_x_0 p(x_0|x_t)[-2(x_0-F(x_t))]. And finally, F(xt)=x0∼p(x0|xt)[x0].F(x_t)=E_x_0 p(x_0|x_t)[x_0]. That is, F(xt)=∫x0x0⋅p(x0|xt)x0.F(x_t)= _x_0x_0· p(x_0|x_t)\,dx_0. Using Bayes’ rule, we can rewrite, F(xt)=∫x0x0⋅p(x0)p(xt|x0)dx0p(xt).F(x_t)= _x_0 x_0· p_D(x_0)p(x_t|x_0)\,dx_0p_D(x_t). Or equivalently, F(xt)=∫x0x0⋅p(x0)p(xt|x0)x0∫x0p(x0)p(xt|x0)x0.F(x_t)= _x_0x_0· p_D(x_0)p(x_t|x_0)\,dx_0 _x_0p_D(x_0)p(x_t|x_0)\,dx_0. Using a normal distribution, F(xt)=∫x0(xt;αtx0,(1−αt)I)⋅x0⋅p(x0)x0∫x0(xt;αtx0,(1−αt)I)⋅p(x0)x0.F(x_t)= _x_0N(x_t; _tx_0,(1- _t)I)· x_0· p_D(x_0)\,dx_0 _x_0N(x_t; _tx_0,(1- _t)I)· p_D(x_0)\,dx_0. Using Monte-Carlo estimates, we can rewrite the above expression as: F(xt)=∑x0∈(xt;αtx0,(1−αt)I)⋅x0∑x0∈(xt;αtx0,(1−αt)I).F(x_t)= _x_0 N(x_t; _tx_0,(1- _t)I)· x_0 _x_0 N(x_t; _tx_0,(1- _t)I). Case when t→Tt→ T As t→Tt→ T, αt→0 _t→ 0, and thus (xt;αtx0,(1−αt)I)→(xt;0,I)N(x_t; _tx_0,(1- _t)I) (x_t;0,I), which is a constant for varying x0x_0. Bringing this back to Eq. (21), it follows that F(xt)=1n∑x0∈x0,F(x_t)= 1n _x_0 x_0, Case when t→0t→ 0. As t→0t→ 0, αt→1 _t→ 1. For simplicity, assume that the closest sample to tx_t is unique. Let closest=argmin0∈‖αt0−t‖2,x_0_closest= _x_0 \| _tx_0-x_t \|^2, d=min0∈D∖closest(‖αt0−t‖2−‖αtclosest−t‖2)>0,d= _x_0∈ D \x_0_closest\ ( \| _tx_0-x_t \|^2- \| _tx_0_closest-x_t \|^2 )>0, 0≤‖∑x0∈(xt;αt0,(1−αt))⋅∑x0∈(xt;αtclosest,(1−αt))−closest‖0≤ \| _x_0 N(x_t; _tx_0,(1- _t)I)·x_0 _x_0 N(x_t; _tx_0_closest,(1- _t)I)-x_0_closest \| ≤∑∈∖closest‖12π(1−αt)exp(−‖αt0−t‖2+‖αtclosest−t‖22(1−αt))‖≤ _x_0 \x_0_closest\ \| 1 2π(1- _t) ( - \| _tx_0-x_t \|^2+ \| _tx_0_closest-x_t \|^22(1- _t) ) \| ≤∑∈∖closest‖12π(1−αt)exp(−d2(1−αt))‖→0≤ _x_0 \x_0_closest\ \| 1 2π(1- _t) (- d2(1- _t) ) \|→ 0 By setting, αt _t → 1, ∑0∈(t;αt0,(1−αt))⋅0∑x0∈(;αtclosest,(1−αt))→closest _x_0 N (x_t; _tx_0,(1- _t)I )·x_0 _x_0 N (x; _tx_0_closest,(1- _t)I ) _0_closest Similarly, we can write, ∑0∈(t;αt0,(1−αt))∑x0∈(;αtclosest,(1−αt))→1 _x_0 N (x_t; _tx_0,(1- _t)I ) _x_0 N (x; _tx_0_closest,(1- _t)I )→ 1 Hence, F(t)F(x_t) →closest _0_closest (utilizing the relation in 28). It completes the proof. ∎ Appendix B Additional Results On Fine Grained Preference Alignment Task Figure 11: Reward Comparison on Fine-Grained Alignment Tasks. We further evaluate the efficacy of PAPA on a more fine-grained preference alignment task with pre-defined quantifiable objectives. We consider two diverse alignment tasks: a) Compressibility, in which an image with a smaller size is regarded as better; b) Aesthetic Quality, in which we use the LAION aesthetic score predictor [25] to automatically assign aesthetic ratings to images, enabling objective reward assignment based on visual quality without requiring human evaluation. For this analysis, we use Stable Diffusion v1.5 as the base model and compare its performance with D3PO [31]. Our experimental findings, as reported in 11, indicate that PAPA rapidly adapts to the preference with comparatively fewer online feedback responses than D3PO, justifying its suitability for fine-grained preference alignment in interactive environments. Appendix C Additional Visualizations of PAPA on Fine-Grained Alignment Task Figure 12 shows additional examples of images generated by PAPA for the fine-grained alignment task. Figure 12: Additional visualizations of PAPA on the Fine-Grained Alignment Task. Appendix D Qualitative Comparisons of Proposed PAPA with D3PO Figure 13: Additional visualizations of PAPA and D3PO in generating rabbit and deer images with compressibility as the objective. Figure 14: Additional visualizations of PAPA and D3PO in generating horse and fox images with the high aesthetic score objective. Detailed comparison reveals that PAPA outperforms D3PO starting from early stages of interaction. Here, we present visualizations of generated images by D3PO and proposed PAPA across gradually increasing epochs. Figure 13 presents samples generated with compressibility as a preference for prompts ’rabbit’ and ’deer’. Figure 14 presents generated examples for prompts ’horse’ and ’fox’ when aesthetic score is used as a preference. Appendix E Insufficiency of Existing D3PO in Active Preference Alignment Tasks Figure 15: Instability of current preference alignment approach D3PO. Our extensive experimental analysis in the main paper shows D3PO’s ineffectiveness in personalized active preference alignment. This section analyzes the underlying reasons. We initiate our analysis by conducting an experiment in which we systematically vary the value of β. This allows us to observe how adjustments to β—which governs the strength of the preference alignment loss (as proposed in [31]), as formally described below—affect the performance of preference alignment. ℒi(θ)=−(si,σw,σl)[logρ(βlogπθ(aiw|siw)πref(aiw|siw)−βlogπθ(ail|sil)πref(ail|sil))]L_i(θ)=-E_(s_i, _w, _l) [ ρ (β _θ(a_i^w|s_i^w) _ref(a_i^w|s_i^w)-β _θ(a_i^l|s_i^l) _ref(a_i^l|s_i^l) ) ] (11) We present our empirical results in Figure 15. Our observations indicate that increasing the value of β leads to a decline in the success rate of D3PO, since a lower β imposes a stronger emphasis on the preference alignment objective. Conversely, a higher β relaxes this alignment, resulting in a notably reduced success rate, particularly when compared to PAPA and EPAPA. Intriguingly, we find an inverse relationship between the success rate and the quality of generated samples: high success rates correspond to lower sample quality, and vice versa. As the influence of preference alignment diminishes (i.e., as β increases), the model remains closer to the base or reference model in parameter space, leading to improved sample quality but reduced alignment. Distinct from PAPA, D3PO lacks an independent parameter that controls sample quality without affecting the strength of the preference alignment objective. Consequently, unlike PAPA, D3PO cannot attain strong preference alignment without a substantial loss in the quality and diversity of generated samples. Appendix F Results with Cifar-10 In this section, we analyze our proposed approach using the Cifar-10 dataset. For pre-training the DDPM model on Cifar-10, we use the same architecture and hyperparameters specified in the open-source code available here111https://github.com/openai/improved-diffusion. Table 3: Comparison of FID and IS scores on CIFAR-10 with preference set Pleane, Car, Truck, Ship. CIFAR-10: Preference set s1s_1 Method FID↓ IS↑ Base 124.4 ± 6.0 6.6 ± 0.2 PAPA 104.7 ± 5.0 5.3 ± 0.5 EPAPA 95.9 ± 5.7 5.3 ± 0.5 Table 4: Success rate (SR) comparisons with Cifar-10 across interaction steps. Figure 16: Additional Visualizations on the Efficacy of EPAPA on generating images aligning with the preference set. We evaluate performance using different user preference sets. Table 4 presents the results based on the SR metric, with preference s1∈Car, Truck, Plane, Ships_1∈\Car, Truck, Plane, Ship\. The empirical results indicate that as fine-tuning advances and more user preference data is gathered, the SR significantly improves compared to the pre-trained model (denoted as Base), highlighting the model’s increasing capability to precisely align with user preferences. Moreover, we observe that EPAPA consistently outperforms PAPA, consistent with our findings across other datasets. This improvement underscores the effectiveness of EPAPA’s sampling strategy, which combines the pre-trained diffusion model—adept at low-level denoising—with the fine-tuned model, which excels in high-level shaping. We assess sample quality and diversity using FID and IS metrics, with results summarized in Table 4. The results demonstrate that our approach successfully preserves high quality, as evidenced by achieving an IS score comparable to the Base model (trained on the preference set), while promoting diversity within the preferred set, indicated by a lower FID. Overall, these empirical findings highlight the impact of our proposed approaches in tackling the active preference alignment problem. Appendix G Qualitative Visualization from Cifar-10 In this section, we provide a side-by-side visualization comparing samples produced by our proposed method and those generated by a baseline—specifically, a pretrained diffusion model trained on the CIFAR-10 dataset. For this qualitative analysis, we select a preference set s∈Car,Truck,Airplane,Ships∈\Car,Truck,Airplane,Ship\, as illustrated in Figure 16. The results reveal that EPAPA produces samples closely aligned with user preferences, whereas the baseline diffusion model frequently generates samples outside the specified preferences. Additionally, samples from EPAPA maintain both quality and diversity within the chosen preference set. These further visualizations from CIFAR-10 underscore the effectiveness of our approach in a standard computer vision context. Appendix H Effect of β This section examines the impact of β on performance. To this end, we perform experiments with different values of β and evaluate the results based on Success Rate, FID, and IS scores. For this analysis, we select \ Boot, Sandal, Sneaker \ as the preference set. The comparison of FID and IS scores is presented in Table 6, while the Success Rate comparison is shown in Table 6. Our empirical results reveal that selecting a high value for β improves sample quality but significantly hinders preference alignment. On the other hand, a very low β greatly enhances preference alignment, but at the expense of sample quality. Therefore, a mid-range value like 0.009 strikes the ideal balance, offering improved preference alignment while maintaining high sample quality—making it the most effective choice for practical applications. Table 5: Comparison of FID and IS scores for different values of β. Eval: Effect of β β FID↓ IS↑ 0.0 39.93± 18.3 2.8± 0.38 0.001 47.98± 15.53 2.3± 0.40 0.005 238.90± 74.02 2.4± 0.43 0.009 36.72± 16.9 2.9± 0.34 0.05 401.70± 115.58 1.7± 0.84 Table 6: Analyzing the Effect of β. Appendix I Impact of LQPDEL^QPDE on Success Rate In the main paper (see Section 6), we examine the effect of ℒQPDEL^QPDE on the quality and diversity of the generated samples. Here, we focus on its impact on the success rate. For this analysis, we select \ Boot, Sandal, Sneaker\ as the preference set and present the results in Figure 17. We observe that the model fine-tuned with and without ℒQPDEL^QPDE achieves similar success rates. This suggests that while ℒQPDEL^QPDE does not directly influence the success rate, it significantly enhances the diversity and quality of the generated samples. Figure 17: Analyzing ℒQPDEL^QPDE. Appendix J Qualitative Analysis on the Importance of LQPDEL^QPDE In the main paper (see Section 6), we study the role of ℒQPDEL^QPDE in improving sample quality and diversity. In this section, we provide additional qualitative results that further highlight the impact of ℒQPDEL^QPDE on both sample quality and diversity. In Figure 18, we compare the samples generated by models fine-tuned with and without ℒQPDEL^QPDE. For this comparison, we choose \ Boot, Sandal, Sneaker\ as the preference set. We observe a noticeable improvement in both the quality and diversity of the generated samples when ℒQPDEL^QPDE is included in the objective. Figure 18: Visualizing the importance of ℒQPDEL^QPDE on Sample Quality &\& Diversity. Appendix K Visualizations of Insufficiency of LpL^p In the main paper (see Section 5), we argue that the ℒpL^p objective alone is insufficient for effectively addressing active preference alignment problems. Through a series of ablation studies, we emphasize the critical role of other components within the PAPA and EPAPA frameworks (see Section 6). To illustrate the limitations of ℒpL^p, we provide a visualization showing its inability to generate diverse samples within the preference set. The results, shown in Figure 19, use \ Boot, Sneaker, Sandal\ as the preference set. We observe that the model trained with the ℒpL^p objective predominantly generates samples from a single preferred class (i.e., Boot), underscoring its failure to produce a diverse range of samples, which is also supported by Theorem 1. Figure 19: Insufficiency of ℒpL^p. Appendix L Insufficiency of LnpL^np One might wonder whether ℒnpL^np alone is sufficient to address the active preference alignment task. However, it turns out that ℒnpL^np by itself is inadequate for solving this problem. It faces challenges similar to those encountered when the model is fine-tuned using only ℒpL^p. As depicted in Figure 20, we observe that when fine-tuned solely with ℒnpL^np, the model gradually forgets everything and eventually generates entirely black images. This suggests that the objective essentially leads to forgetting the non-preferred set derived from the user’s feedback. Thus, relying exclusively on ℒnpL^np during fine-tuning causes the model to unlearn prior knowledge. This observation underscores the critical role of ℒpL^p in maintaining a balance between relearning and forgetting ("memory consolidation"), which is essential for effective active preference alignment. Figure 20: Insufficiency of ℒnpL^np. Appendix M Generalization of PAPA under Non-Binary Feedback Figure 21: Visualizations of generated samples with EPAPA under non-binary feedback settings. While our current formulation treats all preferred samples equally, the Preference Aligner objective (Eqn. 5) is flexible and can be naturally extended to handle richer feedback signals, such as ratings or auxiliary inputs. In particular, normalized rating scores can be incorporated as weighting coefficients in the ℒPAPAL_PAPA term, allowing us to prioritize certain preferences more strongly than others. To explore this idea, we conducted a controlled experiment where one preferred class (e.g., Sandal) was assigned a higher normalized preference score of 0.9, compared to other preference classes (Boot with a preference score of 0.2, and Sneaker with 0.5). As feedback accumulated, the model adapted accordingly—generating noticeably more sandal samples, in alignment with the stronger preference signal. For a visualization of the generated samples, please see Figure 21 (right). In Figure 21 (left), we depict the visualization of generated samples with a different normalized preference score across the same preference set. These additional results not only demonstrate the adaptability of PAPA to more expressive feedback modalities but also highlight its potential to serve as a broader framework for personalized generative modeling. We believe incorporating fine-grained feedback is an exciting avenue for future work, and PAPA is well-positioned to support it. Appendix N Qualitative Comparisons with different K Figure 22: Visualizations with Different Values of K. In the main paper (see Section 6), we provide a quantitative analysis of the performance for different values of K. Here, we offer a qualitative comparison, with the results shown in Figure 22. We choose \ boot, Sandal, Sneaker \ as the preference set for this analysis. As shown in Figure 22, the best results are achieved with K=400K=400, while larger values of K lead to a decline in sample quality. These findings align with our hypothesis, reinforcing the importance of using the fine-tuned model as a denoiser for higher noise levels, and the pre-trained model for lower noise levels, to enhance performance. Appendix O Additional Results on the Effects of K As analyzed in the main paper (see Section 6.3), our experiments with Fashion-MNIST show that extreme values of K are suboptimal: large K values lead to poor denoising by pre-trained diffusion models, while very small K values result in weaker alignment due to insufficient guidance. This motivates choosing K in the mid-range of the reverse diffusion steps, where the model achieves a balance between generative quality and preference alignment. While the optimal K may vary slightly across domains, we find that values near the midpoint of the reverse diffusion step generalize well in practice. To further validate this, we conducted additional experiments on MNIST and report our results in the following Table 7. We observe a consistent trend with Fashion-MNIST—optimal performance is achieved when the value of K lies near the midpoint of the diffusion trajectory. We present the result in the following table, and our empirical outcomes are consistent with the result we observed with the other dataset (see Table 2 in the main paper). Table 7: Effect of K. K FID↓ IS↑ 100 18.87± 5.5 2.0± 0.04 400 16.25± 3.03 2.0± 0.05 700 30.95± 4.02 2.0± 0.03 Appendix P Stability Analysis of Our Proposed Approach To assess the stability of our proposed method, we conducted experiments across 3 independent random trials. The following Figure 23 presents a success rate plot showing the mean and standard deviation calculated from these trials. The solid lines in the plot denote the mean, while the shaded regions denote the standard deviation. These results further reinforce the efficacy and stability of our proposed method. Figure 23: Visualizations of success rate across different choices of preference sets using three independent trials. Appendix Q Details of the Predictive Model Used as the Proxy for User Feedback In our work, instead of relying on user-provided feedback for preferences, we use a pre-trained classifier model to predict the class of the generated image and provide binary feedback based on whether the predicted class belongs to the user’s preferred set. If the predicted class matches, feedback is positive; otherwise, it is negative. The classifier utilizes a custom neural network architecture with two 2D convolutional layers ([32, 64] filters), followed by two fully connected layers. Max-pooling is applied between the last convolutional layer and the first fully connected layer. ReLU activations are used after each layer, and dropout layers (with rates of 0.25 and 0.5) help prevent overfitting. We use cross-entropy loss to optimize the parameters of the classifier and leverage Adadelta [33] as the optimizer and StepLR as the learning rate scheduler. Models trained on the MNIST and Fashion-MNIST datasets achieve accuracies of 98% and 94%, respectively, making them reliable substitutes for human feedback in simulating user interaction. Nonetheless, human preferences are often nuanced and intricate, frequently surpassing what a typical classifier can capture. In our work, we adopt a pre-trained classifier as a practical and controlled proxy for user feedback, particularly to facilitate reproducible experiments in the absence of large-scale human-in-the-loop data. We see this as a foundational step toward more realistic preference modeling. Notably, our framework is modular by design and can seamlessly integrate richer forms of feedback—such as human responses or learned preference models—as they become available. Appendix R Additional Visualizations of Preference Alignment with Diverse Preference Set Here, we present additional comparative visualizations of EPAPA, Base, and D3PO across diverse preference sets. We present the visualizations in Figure 24, again reinforcing the effectiveness of our proposed approach. Figure 24: Additional visualizations of generated images using proposed EPAPA compared to base model and existing approach D3PO. Appendix S Comparative Visualizations of PAPA and EPAPA In the main paper, we offer a quantitative analysis of PAPA and EPAPA across various experimental settings (see Section 6.1). Here, we provide a qualitative comparison between PAPA and EPAPA through visualizations, which are presented in Figure 25. These qualitative visualizations indicate that the quality and diversity of the samples generated by EPAPA are notably superior to those produced by PAPA. The enhanced performance of EPAPA can be primarily attributed to its effective use of the pre-trained model during the reverse diffusion process at low noise levels. Figure 25: Comparative visualizations of PAPA and EPAPA. Appendix T Additional Examples of Generated Samples Using a Different Preference Set from CIFAR-10 In this section, we present visualizations with s2∈Horse,Deer,Cat,Dogs_2∈\Horse,Deer,Cat,Dog\ as the preference set, as depicted in Figure 26. These additional visualizations further reinforce the efficacy of EPAPA in generating samples aligning with diverse preference sets. Figure 26: Additional visualizations on the efficacy of EPAPA on generating images aligning with the preference set. Appendix U Details of Computing Resource and Hyperparameters Our diffusion model employs a UNet [23] architecture as its backbone. For MNIST experiments, the network comprises six ResNet [10] layers and four attention heads, providing a robust framework for image generation tasks. Our implementation is configurable, allowing seamless adaptation for both grayscale and RGB images at higher resolutions. We incorporate a sinusoidal time embedding to effectively encode the time step of the diffusion process. All experiments were conducted on Linux-based servers equipped with AMD Instinct MI250X GPUs with 64 GB memory each and NVIDIA GTX 2080 with 8GB memory. To efficiently utilize multiple GPUs within a single training run, we leverage Data Distributed Parallel (DDP) for scalable computation. Our code is publicly available at https://github.com/NasikNafi/papa. Here are the details of the hyperparameters used in the experiments for MNIST: Hyperparameter Values # num of timesteps 1000 noise scheduler β start 0.0001 noise scheduler β end 0.02 optimizer ADAM learning rate 5e-4 # num of ResNet blocks 6 # num of attention heads 4 K for EPAPA 400 QPDE coefficient β 0.009 time embedding dimension 128 total samples 300 # num of interaction step 30 # num of samples per interaction 10 Table 8: Hyperparameters used for training and evaluation Appendix V Future Work In this work, we introduce a foundational framework for active preference alignment and conduct a comprehensive analysis of its core components, rigorously validating their roles and interactions using standard computer vision datasets. Looking forward, we are eager to expand this framework to tackle significant scientific challenges — such as active drug discovery and the generation of new metals and molecules — while dynamically accommodating target property preferences as they are sequentially revealed, thereby broadening the influence of active preference alignment in critical scientific domains.