Paper deep dive
Direct Preference Density Alignment for Conversational Audio Equalization
Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe, Pablo Martinez Nuevo, Zheng-Hua Tan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/16/2026, 9:07:46 AM
Summary
This paper introduces Direct Preference Density Alignment (DPDA), a proxy-free framework for aligning Large Language Models (LLMs) in continuous control tasks, specifically conversational audio equalization. DPDA constructs non-parametric preference density maps from large-scale user data to serve as an empirical reward surface, eliminating the need for unstable learned reward models. The authors propose a hybrid pipeline combining Group Relative Policy Optimization (GRPO) for structural stability and Direct Preference Optimization (DPO) for precise preference refinement. Experiments show that a 1.5B-parameter model using this method achieves perceptual parity with GPT-4o mini in blind listening tests while using significantly less inference compute.
Entities (8)
Relation Signals (8)
Direct Preference Density Alignment → appliedto → Audio Equalization
confidence 95% · We instantiate this framework on a subjective audio equalization task
Direct Preference Density Alignment → combines → GRPO
confidence 95% · Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO.
Direct Preference Density Alignment → combines → DPO
confidence 95% · Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO.
Direct Preference Density Alignment → achievesparitywith → GPT-4o-mini
confidence 92% · enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline
Qwen2.5-Instruct → usedin → Direct Preference Density Alignment
confidence 90% · We utilize the Qwen2.5-Instruct family... as our base models
Direct Preference Density Alignment → uses → Reflective-KDE
confidence 90% · we adopt Reflective-KDE [27]... to correct this, we adopt Reflective-KDE
GRPO → provides → structural stability
confidence 88% · GRPO excels at structural stability (learning how to format)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.
Tags
Links
- Source: https://arxiv.org/abs/2609.12607v1
- Canonical: https://arxiv.org/abs/2609.12607v1
Trouble viewing inline? Open PDF directly →
Full Text
52,354 characters extracted from source content.
Expand or collapse full text
Direct Preference Density Alignment for Conversational Audio Equalization Ioannis Stylianou Affiliation: Department of Electronic Systems Affiliation: Aalborg University Affiliation: Aalborg, Denmark Email: iost@es.aau.dk Sven Ewan Shepstone Affiliation: Bang & Olufsen A/S Affiliation: Struer, Denmark Email: ssh@bang-olufsen.dk Jon Francombe Affiliation: Bang & Olufsen A/S Affiliation: Struer, Denmark Email: jofr@bang-olufsen.dk Pablo Martinez Nuevo Affiliation: Bang & Olufsen A/S Affiliation: Struer, Denmark Email: pmn@bang-olufsen.dk Zheng-Hua Tan Affiliation: Department of Electronic Systems Affiliation: Aalborg University Affiliation: Aalborg, Denmark Email: zt@es.aau.dk Abstract Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute. 1 Introduction Aligning Large Language Models (LLMs) to subjective continuous-control tasks remains a significant open challenge. Modern Reinforcement Learning from Human Feedback (RLHF) [23] pipelines rely heavily on Proximal Policy Optimization (PPO) and learned proxy Reward Models (RMs). However, RMs are highly susceptible to “reward hacking” [13] and PPO is extremely resource-intensive, requiring four separate LLMs (Trained, Reference, Reward, and Value models) during training. Recent advancements attempt to mitigate this compute burden. Group Relative Policy Optimization (GRPO) [30] eliminates the value model, while offline methods like Direct Preference Optimization (DPO) [25] eliminate both the value and reward models. However, we demonstrate that by shifting from online Reinforcement Learning (RL) to offline “contrastive” learning, DPO loses active exploration. In compact spaces [12], this lack of environmental (or programmatic) grounding leads to format collapse, where the model forgets valid syntactical constraints. To resolve this, we propose a proxy-free, online RL framework for continuous parameter estimation. Instead of training an unstable proxy RM, we leverage population data to construct static, non-parametric preference surfaces. The models then navigate the surface like a topographical map, exploring different regions to eventually find the “peaks” of highest human satisfaction. We instantiate this framework on a subjective audio equalization task, mapping abstract natural language (e.g., “make it warmer”) to loudspeaker parameters. Crucially, this empirical reward surface allows us to combine the online structural grounding of GRPO with the targeted offline refinement of DPO, outperforming either of the two algorithms in isolation, as well as Supervised Fine-Tuning (SFT) and In-Context Learning (ICL) baselines. Our contributions are summarized as follows: • Proxy-free online RL framework: We introduce a paradigm for aligning LLMs that completely eliminates the instability and memory overhead of learned reward models, while preserving the ability for online exploration. • Hybrid alignment strategy: Our approach seamlessly bridges online structural grounding (GRPO) and offline preference sharpening (DPO), overcoming the inherent limitations of each method when used in isolation. • Improved distillation: Through this GRPO+DPO pipeline, we successfully distill subjective population preferences into a 1.5B parameter model, achieving perceptual parity with GPT-4o mini in a blind A/B listening test. 2 Related Work Our work positions itself at the intersection of semantic audio control, preference alignment, and RL in continuous action spaces. Traditionally, mapping natural language to audio parameters has been treated as a supervised regression problem. Early systems relied on fixed vocabulary-to-parameter mappings [24], while recent approaches employ LLMs to interpret unconstrained text and estimate equalization (EQ) and reverberation (reverb) parameters [7, 8]. Crucially, these regression-based methods implicitly favor a single parameter realization at inference time. Prior work [31] challenged this assumption, proposing that audio descriptors like “warm” map to distributions rather than point estimates. However, that study relied on small-scale supervised learning (N=11N=11 per prompt). Our work scales this distributional premise (N≈3,000N≈ 3,000 per prompt) and transitions from supervised matching to RL, enabling on-policy optimization that fully exploits densely annotated data. The standard LLM RL-tuning pipeline [23] typically relies on PPO [26] optimized against a learned reward model. However, PPO requires training a separate Value Network to estimate the expected return. In subjective multimodal domains like audio preference, fitting a Value Network introduces an additional proxy objective, compounding the risk of reward misspecification [5]. To circumvent the instability of critics and reward models, recent work has shifted toward direct alignment methods. Approaches like DPO, Identity Preference Optimization (IPO) [1], Kahneman-Tversky Optimization (KTO) [10] and other variants [6, 20, 16], optimize the policy directly from preference data without an explicit reward function. However, these methods are blind to the probability mass assigned to completions outside the preference dataset [36, 15]. As a result, the model may increase the probability of malformed or constraint-violating outputs during training, even though such outputs are never explicitly preferred. As we demonstrate, this lack of active exploration makes them brittle when the model must actively learn to maintain strict formatting constraints. While extensions like C-DPO [22] attempt to mitigate this by integrating explicit penalty margins, the lack of active exploration remains a bottleneck in continuous control tasks where the model must inherently learn the structural manifold. GRPO [30] offers a middle ground between the heavy machinery of PPO and the static nature of DPO. Originally designed for reasoning tasks, GRPO removes the value network and instead uses the average reward of a group of generated rollouts as the baseline. By generating its own data during training, GRPO provides the exploration necessary to learn format constraints. Other variants such as Dr.GRPO [21] and DAPO [37] have proposed advanced advantage estimation and regularization techniques to improve upon this foundation. However, in this study we deliberately evaluate the vanilla instantiations of both DPO and GRPO. By omitting explicitly constrained decoding, customized validity penalties, or specific algorithmic extensions, we isolate the fundamental structural trade-off: the formatting failures of pure offline contrastive learning versus on-policy exploration of online methods. To mitigate the individual weaknesses of online and offline methods, recent works in reasoning and multimodal generation have begun exploring hybrid pipelines that sequence online exploration with offline refinement [19, 34, 33]. Building on this emerging paradigm, our work adapts this hybrid stabilization strategy to continuous parameter estimation. Crucially, we do so while completely eliminating the reliance on the learned verifiers or reward models used in these prior works. 3 Preliminaries and reward surface construction To effectively align an LLM with subjective auditory preferences, we must first define the control space and construct a reliable reward signal that captures the nuance of human perception without relying on unstable proxy networks. 3.1 The Beosonic control space We ground our study in the “Beosonic” interface [3], a continuous two-dimensional control plane ∈[−6,6]2X∈[-6,6]^2. This space reduces the EQ transfer function to 2 parameters, with each corresponding to the intensity of a distinct equalization filter (±6± 6 gain), as illustrated in Figure 1. With this approach we effectively represent audio equalization through an interpretable 2D parameter space. Unlike discrete selection tasks, the model must predict a continuous coordinate x∈[−6,6]2x∈[-6,6]^2. Figure 1: Filters of the EQ controller. Horizontal movement yields a “smile curve” (left); vertical applies a linear adjustment (right). 3.2 Dataset and motivation Our reward signal is the product of a large-scale comparison dataset, originally collected to evaluate the framework proposed in [31]. In this deployment, participants were presented with natural language prompts paired with audio variations generated by four distinct strategies: a deterministic Zero-Shot LLM (T2B), two distributional baselines (LoRA, RAG), and a Random agent. For each variation, users provided binary feedback indicating whether the audio satisfied the prompt. Figure 2: Total user selection counts by model type. The Zero-Shot baseline (T2B) outperforms other models (LoRA, RAG), indicating that previous attempts at distributional modeling failed to capture preference modes. The resulting dataset comprises N≈90,000N≈ 90,000 interaction events (consisting of both selections and rejections). An analysis of the positive selections (Figure 2) reveals a critical insight: The prompt-engineered Zero-Shot baseline (T2B) achieved the highest selection count (14,08714,087), significantly outperforming the more complex distributional models. This outcome suggests that the prior distributional approaches, having been trained on extremely sparse data (N=11N=11), failed to generalize effectively. This observation motivates our RL approach: We require an objective that can explicitly optimize for the peaks of human satisfaction (Exploitation) while maintaining sufficient coverage of the space (Exploration). 3.3 Probability of selection To construct the reward function R(x|p)R(x|p) for a specific prompt p, we cannot simply estimate the density of high-rated points, as this would be biased by the sampling distribution of the generating models (e.g., if a model samples the center frequently, high density there might differ from high preference). We therefore aim to estimate the Probability of Selection P(Selected|x)P(Selected|x) for a given coordinate x. Let allD_all be the set of all offered coordinates for prompt p, and pref⊆allD_pref _all be the same set weighted by binary selection events. We estimate two densities using Reflective-KDE (defined in Subsection 3.4) 1. f^total(x) f_total(x): The density of the proposal distribution (where the points came from; P(x)P(x)). 2. f^pref(x) f_pref(x): The density of the users’ preferred regions (P(x|Selected)P(x|Selected)). By applying Bayes’ theorem, the Probability of Selection is proportional to the ratio of the positive density to the total offered density: S(x) S(x) =P(Selected∣x)=P(x∣Selected)P(Selected)P(x)∝f^pref(x)f^total(x)+ϵ =P(Selected x)= P(x )\,P(Selected)P(x) f_pref(x) f_total(x)+ε (1) where ϵε ensures numerical stability. This ratio effectively cancels out the sampling bias, isolating the underlying population preference. 3.4 Reflective kernel density estimation Standard KDE suffers from boundary bias in bounded spaces, underestimating density near the edges (where user preferences often cluster, e.g., “Max Bass”). To correct this, we adopt Reflective-KDE [27], augmenting the dataset by mirroring each point xix_i across the boundaries of X into the 8 surrounding neighbor squares, creating a 3×33× 3 grid of “ghost” points. The density estimate becomes: f^(x)=19nh2∑i=1n∑k=08K(x−Tk(xi)h) f(x)= 19nh^2 _i=1^n _k=0^8K ( x-T_k(x_i)h ) (2) where TkT_k represents the reflection mappings (T0T_0 is the Identity) and K is a Gaussian kernel. We set the bandwidth h=0.25⋅hScotth=0.25· h_Scott. This scaling factor on Scott’s Rule [28] accounts for two specific properties of our space: a 0.5 factor to adjust for the extra point mass introduced by the reflective boundary, and an additional 0.5 factor to counteract the known tendency of Scott’s Rule to over-smooth non-convex spaces. An example of the effect of different bandwidths is provided in Appendix C. 3.5 Global normalization and augmentation Figure 3: Non-Parametric Reward. (a) Sampling bias. (b) Total density f^total f_total. (c) Preferred density f^pref f_pref. (d) Final normalized Reward Surface R(x). To ensure the reward signal is consistent across different prompts, we perform a global normalization pass. We compute the global maximum (SmaxS_max) and minimum (SminS_min) density ratios across the entire dataset. The final reward R(x)R(x) used for RL training is scaled to [−1,1][-1,1]: R(x)=2⋅S(x)−SminSmax−Smin−1R(x)=2· S(x)-S_minS_max-S_min-1 (3) The whole reward-surface construction process is exemplified in Figure 3. Finally, we employ semantic data augmentation. We assume that synonymous prompts (e.g., “Make it warmer” and “Increase warmth”) share an identical preference manifold. We aggregate data for synonymous clusters and map them to the same reward surface. 3.6 Evaluation splits To assess generalization, we partition the prompts into three sets; a training set containing approximately 80%80\% of the prompts used for policy updates, a 10% validation set consisting of the randomly held-out prompts drawn from the same distribution, and a manually isolated 10% Out-Of-Distribution (OOD) set containing a cluster of prompts related to “Vocal Clarity.” These OOD concepts are entirely unseen during training, allowing to evaluate the model’s ability to extrapolate its learned preference geometry to novel semantic inputs. 4 Alignment strategies in continuous spaces Having constructed a non-parametric reward surface R(x)R(x), our goal is to align an LLM policy πθ _θ to maximize the expected return x∼πθ[R(x)]E_x _θ[R(x)]. In this section, we analyze the structural limitations of standard offline alignment in continuous spaces and propose a hybrid pipeline that utilizes an online-aligned model to synthesize its own high-precision preference dataset. 4.1 The challenge of continuous formatting Unlike standard chat models, our agent must output valid coordinates in a strict format (e.g., [1.5, -3.2]). A critical failure mode in this domain is format collapse, where the model drifts into generating conversational text or invalid syntax. We hypothesize that offline methods (i.e., DPO) are particularly susceptible to this. Since DPO is limited to the specific negative examples in the static dataset, it cannot penalize emergent structural failures. As the policy shifts, it may drift into generating novel malformed syntaxes that were not present in the pre-collected data, effectively bypassing the optimization constraints. Conversely, online methods (like GRPO) generate their own predictions during training. This allows the environment to provide immediate negative feedback (R=−1.0R=-1.0) for syntax errors, effectively “grounding” the model in the constraints of the control interface. 4.2 Baseline 1: direct preference optimization DPO trains the policy by increasing the margin between preferred responses and dispreferred ones: ℒDPO=−(p,xw,xl)∼[logσ(r^θ(p,xw)−r^θ(p,xl))]L_DPO=-E_(p,x_w,x_l) [ σ ( r_θ(p,x_w)- r_θ(p,x_l) ) ] (4) where the implicit reward is defined as: r^θ(p,x)=βlogπθ(x|p)πref(x|p). r_θ(p,x)=β _θ(x|p) _ref(x|p). While efficient, DPO relies heavily on the Kullback-Leibler (KL)-divergence constraint (implicitly controlled by β) to prevent the model from deviating from the reference policy πref _ref. In our domain, we construct the dataset D using the reward map R(x)R(x), where xwx_w and xlx_l are pairs of coordinates such that R(xw|p)>R(xl|p)R(x_w|p)>R(x_l|p). More details are provided in Subsection 4.4. 4.3 Baseline 2: group relative policy optimization We adopt GRPO as our online framework. Unlike PPO, GRPO eliminates the value model. For each prompt p, the policy generates a group of G outputs x1,…,xG\x_1,…,x_G\. By setting ρi(θ)=πθ(xi|p)πold(xi|p) _i(θ)= _θ(x_i|p) _old(x_i|p), the objective maximizes: GRPO(θ)=p∼P(P),xii=1G∼πθold[1G∑i=1G(min(ρi(θ)Ai,clip(ρi(θ),1−ϵ,1+ϵ)Ai)⏟Clipped Surrogate Objective−βKL(πθ||πref)⏟KL Penalty)] splitJ_GRPO(θ)=E_ subarraycp P( P),\x_i\_i=1^G _ _old subarray [ 1G _i=1^G ( ( _i(θ)A_i,clip( _i(θ),1-ε,1+ε)A_i )_Clipped Surrogate Objective- _KL( _θ|| _ref)_KL Penalty& ) ] split (5) The advantage AiA_i is computed by normalizing the rewards within the group. Crucially, if the model generates invalid format, the reward function returns R=−1.0R=-1.0. Since this penalty is included in the group, the model learns to avoid invalid syntax to achieve a positive advantage relative to its peers. 4.4 Proposed method: Hybrid GRPO+DPO pipeline We observe that while GRPO excels at structural stability (learning how to format), it may fail to find all the local maxima of the preference density (learning exactly what different people want to hear). To resolve this, we propose a two-stage pipeline that combines the stability of online RL with the precision of offline RL. Stage 1: Structural Alignment (GRPO). We first train πθ _θ using GRPO against the Density Map. This transforms the generalist LLM into a robust control agent (πGRPO _GRPO) that reliably outputs coordinates in the correct manifold. Stage 2: Synthetic Preference Mining & Peak Injection. We construct a synthetic DPO dataset synD_syn using πGRPO _GRPO as the generator. To ensure the dataset steers the model toward the local optima, we employ a peak injection strategy: 1. On-Policy Mining: We sample completions from πGRPO _GRPO. Those with high rewards (R(x)>0.8R(x)>0.8) are added to the positive pool; malformed or low-reward outputs (R(x)<−0.2R(x)<-0.2) are added to the negative pool. 2. Peak Injection: If πGRPO _GRPO fails to discover the top-K local maxima of the Density Map R(x)R(x) for a given prompt, we analytically find the peak coordinates via grid search on R(x)R(x) and inject them into the positive pool as “Golden Truths.” The process is exactly the same as in standard DPO (Subsection 4.2), with πref _ref as the generator instead of πGRPO _GRPO. Stage 3: Refinement (DPO). We tune the model using DPO on synD_syn, setting πref=πGRPO _ref= _GRPO. By initializing with the GRPO model and training on injected expert peaks with relatively high β values, the hybrid model achieves both high format compliance and improved preference matching. 5 Experiments and results We evaluate our alignment strategies on the OOD test set described in Section 3.6. This set contains semantic concepts unseen during training, requiring the model to generalize the geometric understanding of the preference space. 5.1 Experimental setup Models: We utilize the Qwen2.5-Instruct family (0.5B and 1.5B parameters) [32] as our base models due to their strong instruction-following capabilities and open weights (Apache License 2.0). Baselines: We compare against ICL baselines using Qwen2.5 and GPT-4o mini. To establish a rigorous performance floor, we inject domain knowledge into these baselines via a sophisticated system prompt containing an explicit semantic-to-coordinate mapping dictionary (e.g., “warm” →[0,−6]→[0,-6]), derived from audio attribute clustering literature [11]. In contrast, our RL-tuned models are agnostic to these expert definitions; they rely on a minimal system prompt and must learn the semantic-to-acoustic relationships purely from the reward signal. Metrics: Greedy Mean Relative Reward (GMRR): We evaluate the models’ performance on their most probable (greedy) completion for each prompt. Additionally, while training utilizes globally normalized rewards, evaluation requires measuring optimality relative to the specific prompt. We therefore revert to local normalization: For each prompt, the reward is rescaled such that the maximum of the density map is 1.01.0 and the minimum is 0.00.0. GMRR(x)=S(x)−Smin,localSmax,local−Smin,local.GMRR(x)= S(x)-S_min,localS_max,local-S_min,local. (6) Under this metric, a score of 1.01.0 indicates the model found the absolute peak of human preference for that specific query, while the absolute low or malformed outputs are assigned 0.00.0. Format Success Rate (Parse Rate): The percentage of generated outputs that strictly adhere to the coordinate format [x, y]. 5.2 The stability gap: Online vs. offline Our first experiment investigates the structural stability of the alignment algorithms. We sweep the KL-divergence penalty coefficient β∈0.0,0.1,0.5,1.0β∈\0.0,0.1,0.5,1.0\ for both DPO (Offline) and GRPO (Online) and measure the Format Success Rate on the OOD test set. As shown in Table 1, a critical failure mode emerges for DPO. A low β value allows the model to deviate more aggressively from its initial state. During training, the objective of the algorithm is to maximize the margin between the positive and negative examples in the pre-developed preference dataset. This approach indeed increases the relative probability of generating a preferred example, but only compared to generating a dis-preferred one. It is likely, especially with low KL regularization, that the overall probability of generating a positive example decreases during training. However, the training objective is still met as long as the overall probability of the negative counterpart decreases even faster. A different way to view this is that offline DPO tends to “tunnel-vision” on the two examples within each preference dataset entry, while the probabilities of all other completions are updated without any supervision. In contrast, GRPO maintains a 100% format success rate even at β=0.0β=0.0. Because the online agent generates its own data during training, any format deviation results in an immediate penalty (R=−1R=-1), forcing the policy to remain within the valid syntax manifold. Interestingly, excessively large β values can over-constrain the model, hindering its ability to follow strict structure. Table 1: GMRR (± STD) on the OOD test set and format success rate (%). The top section compares the GRPO and DPO methods for different β values. The bottom section provides ICL reference scores. 12-GRPO DPO β Model GMRR Rate GMRR Rate 0.0 Qwen2.5-0.5B 0.44 ± 0.08 100% 0.00 0% Qwen2.5-1.5B 0.52 ± 0.10 100% 0.00 0% 0.1 Qwen2.5-0.5B 0.49 ± 0.14 100% 0.31 ± 0.27 47% Qwen2.5-1.5B 0.53 ± 0.09 100% 0.00 0% 0.5 Qwen2.5-0.5B 0.52 ± 0.10 100% 0.33 ± 0.22 81% Qwen2.5-1.5B 0.53 ± 0.09 100% 0.32 ± 0.27 69% 1.0 Qwen2.5-0.5B 0.50 ± 0.10 94% 0.30 ± 0.25 98% Qwen2.5-1.5B 0.51 ± 0.09 100% 0.07 ± 0.23 10% ICL Reference Models GMRR Rate Qwen2.5-0.5B 0.25 ± 0.24 54% Qwen2.5-1.5B 0.49 ± 0.08 100% GPT-4o mini 0.56 ± 0.07 100% 5.3 Scaling exploration We further analyze the impact of the group size G (number of rollouts per prompt) on GRPO performance (Table 2). Increasing G consistently improves the GMRR, saturating between G=8G=8 and G=16G=16. However, this improvement is relatively small, with even the 2-Rollout models outperforming their sophisticated ICL counterparts. This observation aligns with recent studies [35], which question the benefits of larger samples sizes relative to the computational burden. Table 2: Performance comparison of Qwen2.5 models under the GRPO algorithm with varying numbers of rollouts (sample size) and fixed β value (0.5). Metrics shown are Greedy Mean Relative Reward (± Standard Deviation) and format success rate (%). 2 Rollouts 4 Rollouts 8 Rollouts 16 Rollouts 32 Rollouts Model GMRR Parse Rate GMRR Parse Rate GMRR Parse Rate GMRR Parse Rate GMRR Parse Rate Qwen2.5-0.5B 0.47 ± 0.17 81% 0.50 ± 0.14 93% 0.53 ± 0.10 100% 0.51 ± 0.10 100% 0.48 ± 0.14 96% Qwen2.5-1.5B 0.50 ± 0.08 100% 0.51 ± 0.09 100% 0.51 ± 0.09 100% 0.52 ± 0.09 100% 0.52 ± 0.09 100% 5.4 Hybrid pipeline performance Finally, we evaluate the proposed hybrid GRPO+DPO pipeline. We select the best GRPO configuration (β=0.5β=0.5) as the Reference and use DPO (β=1.0β=1.0) on the synthetically mined “Peak Injection” dataset (Subsection 4.4) across different rollouts. Results in Table 3 demonstrate that for the 0.5B model, GMRR jumps from 0.530.53 (Base GRPO) to 0.56 (hybrid), matching the GPT-4o baseline at the cost of some inconsistency (97% Parse Rate). The 1.5B model achieves a GMRR of 0.60 without compromising format compliance. However, the hybrid approach exhibits performance fluctuations across rollout configurations. To confirm this reflects hyperparameter sensitivity rather than algorithmic instability, we evaluated the optimal configuration across four independent seeds, achieving a consistent mean GMRR of 0.58 (see Appendix E). Table 3: Performance comparison of Qwen2.5 models using GRPO (Baseline) versus the combined GRPO + DPO algorithm. The β values for GRPO and DPO are fixed at 0.5 and 1.0 respectively. Model Rollouts Approach GMRR Parse Rate (%) Observation 2 GRPO 0.47 ± 0.17 81 GRPO+DPO 0.44 ± 0.32 88 Drop in GMRR, improvement in parsing rate 4 GRPO 0.50 ± 0.14 93 GRPO+DPO 0.56 ± 0.26 73 Preference sharpened, but stability drops 8 GRPO 0.53 ± 0.10 100 GRPO+DPO 0.48 ± 0.25 100 Drop in GMRR 16 GRPO 0.51 ± 0.10 100 GRPO+DPO 0.49 ± 0.32 74 Drop in both metrics 32 GRPO 0.48 ± 0.14 96 Qwen2.5-0.5B GRPO+DPO 0.56 ± 0.18 97 Improvement in both metrics 2 GRPO 0.50 ± 0.08 100 GRPO+DPO 0.27 ± 0.24 100 GMRR Drop 4 GRPO 0.51 ± 0.09 100 GRPO+DPO 0.56 ± 0.18 78 Improvement but stability degradation hurts GMRR 8 GRPO 0.51 ± 0.09 100 GRPO+DPO 0.40 ± 0.17 100 GMRR Drop 16 GRPO 0.52 ± 0.09 100 GRPO+DPO 0.60 ± 0.17 100 Strongest Improvement (GMRR & Rate) 32 GRPO 0.52 ± 0.09 100 Qwen2.5-1.5B GRPO+DPO 0.48 ± 0.13 100 GMRR Drop To assess if the preference manifold could be learned via supervision, we also trained an SFT baseline with Low Rank Adaptation [14] (LoRA) on the “peaks” of the training prompts. This model achieved a GMRR of 0.45, performing worse than the base model with In-Context Learning (0.49). This suggests that SFT leads to overfitting, degrading the model’s ability to extrapolate to OOD semantic concepts. Table 4 provides a performance summary of the different methods for Qwen2.5-1.5B. 6 Subjective evaluation While our objective metrics demonstrate the convergence of the proposed RL approach, the ultimate validation of an audio system lies in listener perception. We conducted a blind A/B listening test to compare our best-performing hybrid model (Qwen2.5-1.5B) against the baseline (GPT-4o mini via In-Context Learning). Both models controlled the exact same equalization system, ensuring that any perceived differences result purely from the parameters selected by the LLMs. Table 4: Performance summary of Qwen2.5-1.5B on the OOD Test Set. GRPO improves generalization compared to SFT via online exploration, while the hybrid GRPO+DPO pipeline achieves the highest performance. Model Approach GMRR Rate 1.5B SFT (Peak) 0.45 ± 0.12 100% 1.5B ICL 0.49 ± 0.08 100% 1.5B GRPO 0.53 ± 0.09 100% 1.5B Hybrid 0.60 ± 0.17 100% 6.1 Unconstrained data collection and protocol To ensure the evaluation reflected real-world interaction, we utilized a two-phase study with human listeners. First, we recruited 12 participants for a prompt-collection phase. We employed an unconstrained elicitation protocol: Participants listened to audio clips and were asked to request any changes they desired without being informed of the system’s technical limitations (i.e., that it was an equalizer). This design choice was deliberate to avoid priming bias, ensuring that users did not subconsciously tailor their requests to perceived system capabilities (e.g., only asking for “more bass”). As expected, this yielded highly Out-Of-Distribution requests (e.g., volume control, removing instruments, or changing speaker identity). After filtering out these functional system requests, a set of 30 prompts was retained. In the second phase, 11 of the original participants returned to evaluate the models’ responses on these 30 prompts. For each prompt, participants listened to audio processed by both the hybrid RL model and the GPT-4o mini baseline in a randomized, blinded interface. They rated how well the resulting audio satisfied the original request on a 5-point Likert scale (1=“Very Poorly” to 5=“Very Well”). Table 5: Subjective evaluation on Likert score (± Standard Error). The p-value is calculated via the Linear Mixed-Effects Model. Method Score (1–5) Win/Tie % p-value GPT-4o 2.92 ±0.06± 0.06 72.1% 0.095 Hybrid 3.04 ±0.06± 0.06 77.9% 6.2 Quantitative results The aggregate results (Table 5) indicate that our specialized RL model performs at parity with the previously-best generalist baseline. More specifically: Mean Preference Score: The 1.5B RL model achieved a mean rating of 3.04, outperforming the GPT-4o mini baseline (2.92). This suggests that the RL model handles ambiguous, OOD prompts gracefully. Win/Tie Rate: In head-to-head comparisons, the RL model was preferred or rated as equivalent to the baseline in 77.9% of the 330 trials (11 participants × 30 prompts). Statistical Analysis: Because a total of 660 evaluations (two per trial) were presented individually in randomized order, we analyzed the results using a Linear Mixed-Effects Model (LMM) with crossed random effects for both participant and prompt to account for repeated measures [2]. The LMM revealed a non-significant main effect between the models (p=0.095p=0.095). To formally establish perceptual parity between the 1.5B model and the baseline, we subsequently conducted an equivalence test using the Two One-Sided Tests (TOST) procedure [18]. Detailed methodology, the full LMM summary matrix, and the TOST confidence intervals demonstrating statistical parity are provided in Appendix A. The critical implication of these results is efficiency: The proposed Density-Based RL pipeline successfully distilled the necessary audio control capabilities into a model that is not utilizing expert knowledge and is orders of magnitude smaller than its competitor, achieving equivalent user satisfaction without the need for large inference compute or proprietary APIs. 7 Discussion and limitations Our results suggest a paradigm shift for LLM alignment: Rather than training unstable proxy reward models, we can leverage the “wisdom of the crowd” directly via non-parametric density maps. However, this approach entails specific constraints that merit discussion. The curse of dimensionality Our Reflective-KDE approach is highly effective for low-dimensional spaces (e.g., the 2D Beosonic interface). However, KDE suffers from the curse of dimensionality [4]. As the control space expands (e.g., additional acoustic parameters or a 6-DoF robotic arm), the data required to populate a dense reward map grows exponentially. Scaling this framework to high-dimensional action spaces will likely require latent variable models (e.g., Variational Auto Encoders [17]) to compress the control space into a tractable lower-dimensional manifold before applying our density-based RL. Text-only conditioning The current system aligns audio parameters solely based on the semantic intent of the text prompt (e.g., “Make it warm”). It does not analyze the input audio signal (P(Selected|x,AudioID)P(Selected|x,Audio_ID)). While this enables a lightweight, sparse (responds when requested), privacy-preserving inference (no audio upload required), it limits the system’s context awareness. A “warm” adjustment for a podcast could differ significantly from a “warm” adjustment for techno music. Future iterations could integrate audio embeddings (e.g., CLAP [9] features) into the policy to enable content-adaptive equalization. Consensus vs. personalization Finally, while our density map method effectively models the population’s preference space, it does not yet solve the problem of individual personalization. By optimizing for the peaks of the density map, the hybrid model acts as a “Mode-Seeking” agent–it generates the setting that could satisfy any user. It does not yet have the capacity to adapt to user-specific hearing profiles. Extending this framework to conditioned density estimation (P(Selected|x,UserID)P(Selected|x,User_ID)) remains a promising and natural avenue for future research. 8 Conclusion In this work, we presented a proxy-free reinforcement learning framework for continuous parameter estimation. By applying Reflective Kernel Density Estimation to large-scale user data (N≈90,000N≈ 90,000), we constructed non-parametric preference density maps that serve as a stable, empirical reward surface. This approach completely eliminates the memory overhead and reward-hacking vulnerabilities of learned proxy reward models, while preserving the ability to perform true online exploration. Crucially, this continuous reward surface provides the flexibility to employ and combine distinct alignment methodologies. It seamlessly enables both the active, on-policy exploration of GRPO and the targeted, contrastive optimization of DPO. By leveraging this capability to build a hybrid pipeline, we successfully aligned a highly efficient 1.5B-parameter model, achieving perceptual parity with a heavily prompt-engineered GPT-4o mini baseline in a blind A/B listening test. References [1] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos (2023) A general theoretical paradigm to understand learning from human preferences. External Links: 2310.12036, Link Cited by: §2. [2] R. H. Baayen, D. J. Davidson, and D. M. Bates (2008) Mixed-effects modeling with crossed random effects for subjects and items. Journal of memory and language 59 (4), p. 390–412. Cited by: §6.2. [3] Bang&Olufsen (2023) Beosound beovision theatre: technical sound guide. Cited by: §3.1. [4] R. Bellman (1957) Dynamic programming, princeton univ. Press Princeton, New Jersey 39. Cited by: §7. [5] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. Wang, S. Marks, C. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Bıyık, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell (2023) Open problems and fundamental limitations of reinforcement learning from human feedback. External Links: 2307.15217, Link Cited by: §2. [6] S. R. Chowdhury, A. Kini, and N. Natarajan (2024) Provably robust dpo: aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409. Cited by: §2. [7] A. Chu, P. O’Reilly, J. Barnett, and B. Pardo (2025) Text2fx: harnessing clap embeddings for text-guided audio effects. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §2. [8] S. Doh, J. Koo, M. A. Martínez-Ramírez, W. Liao, J. Nam, and Y. Mitsufuji (2025) Can large language models predict audio effects parameters from natural language?. arXiv preprint arXiv:2505.20770. Cited by: §2. [9] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang (2023) CLAP learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §7. [10] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) KTO: model alignment as prospect theoretic optimization. External Links: 2402.01306, Link Cited by: §2. [11] J. Francombe, T. Brookes, and R. Mason (2017) Automatic text clustering for audio attribute elicitation experiment responses. In AES 143rd Convention, Cited by: §5.1. [12] M. M. Fréchet (1906) Sur quelques points du calcul fonctionnel. Rendiconti del Circolo Matematico di Palermo (1884-1940) 22 (1), p. 1–72. Cited by: §1. [13] L. Gao, J. Schulman, and J. Hilton (2023) Scaling laws for reward model overoptimization. In International Conference on Machine Learning, p. 10835–10866. Cited by: §1. [14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §5.4. [15] H. Ivison, Y. Wang, J. Liu, Z. Wu, V. Pyatkin, N. Lambert, N. A. Smith, Y. Choi, and H. Hajishirzi (2024) Unpacking dpo and ppo: disentangling best practices for learning from preference feedback. Advances in neural information processing systems 37, p. 36602–36633. Cited by: §2. [16] H. Ji, C. Lu, Y. Niu, P. Ke, H. Wang, J. Zhu, J. Tang, and M. Huang (2024) Towards efficient exact optimization of language model alignment. arXiv preprint arXiv:2402.00856. Cited by: §2. [17] D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §7. [18] D. Lakens (2017) Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social psychological and personality science 8 (4), p. 355–362. Cited by: §A.1, §6.2. [19] Y. Li, X. Chen, Z. Li, Z. Liu, L. Wang, W. Luo, B. Hu, and M. Zhang (2025) Veripo: cultivating long reasoning in video-llms via verifier-gudied iterative policy optimization. arXiv preprint arXiv:2505.19000. Cited by: §2. [20] T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu (2023) Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657. Cited by: §2. [21] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: a critical perspective. External Links: 2503.20783, Link Cited by: §2. [22] Z. Liu, X. Sun, and Z. Zheng (2024) Enhancing llm safety via constrained direct preference optimization. External Links: 2403.02475, Link Cited by: §2. [23] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1, §2. [24] B. Pardo, D. Little, and D. Gergle (2012) Building a personalized audio equalizer interface with transfer learning and active learning. In Proceedings of the second international ACM workshop on Music information retrieval with user-centered and multimodal strategies, p. 13–18. Cited by: §2. [25] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §1. [26] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §2. [27] E. F. Schuster (1985) Incorporating support constraints into nonparametric estimators of densities. Communications in Statistics-Theory and methods 14 (5), p. 1123–1136. Cited by: §3.4. [28] D. W. Scott (2010) Scott’s rule. Wiley Interdisciplinary Reviews: Computational Statistics 2 (4), p. 497–502. Cited by: §3.4. [29] S. Seabold J. Perktold et al. (2010) Statsmodels: econometric and statistical modeling with python.. scipy 7 (1), p. 92–96. Cited by: Appendix A. [30] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2. [31] I. Stylianou, J. Francombe, P. Mart´ ınez-Nuevo, S. E. Shepstone, and Z. Tan (2026) One prompt, many sounds: modeling listener variability in llm-based equalization. IEEE Journal of Selected Topics in Signal Processing, p. 1–13. External Links: ISSN 1941-0484, Link, Document Cited by: §2, §3.2. [32] Q. Team (2024) Qwen2.5: a party of foundation models. External Links: Link Cited by: §5.1. [33] C. Tong, Z. Guo, R. Zhang, W. Shan, X. Wei, Z. Xing, H. Li, and P. Heng (2025) Delving into rl for image generation with cot: a study on dpo vs. grpo. arXiv preprint arXiv:2505.17017. Cited by: §2. [34] S. Wu, J. Xie, Y. Zhang, A. Chen, K. Zhang, Y. Su, and Y. Xiao (2025) Arm: adaptive reasoning model. arXiv preprint arXiv:2505.20258. Cited by: §2. [35] Y. Wu, L. Ma, L. Ding, M. Li, X. Wang, K. Chen, Z. Su, Z. Zhang, C. Huang, Y. Zhang, M. Coates, and J. Nie (2025) It takes two: your grpo is secretly dpo. External Links: 2510.00977, Link Cited by: §5.3. [36] S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu (2024) Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719. Cited by: §2. [37] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §2. Appendix A Extended statistical analysis of perceptual parity To evaluate the subjective listening test results presented in Section 6.2, we transitioned from a naive paired hypothesis test to a Linear Mixed-Effects Model (LMM). The 660 user evaluations were presented individually and in randomized order to prevent anchoring bias. Because each participant rated both model outputs for each prompt, the dataset contains crossed random effects. We fit an LMM using the statsmodels library [29], defining the subjective score (1–5 Likert scale) as the dependent variable and the model variation as the fixed effect. We included crossed random intercepts for both the user_id and the prompt to explicitly model the variance introduced by individual rater harshness and the inherent difficulty of specific audio requests. The full regression results are provided in Table 6. Table 6: Linear Mixed-Effects Model regression results Model: MixedLM Dependent Variable: score No. Observations: 660 Method: REML Log-Likelihood: -905.9961 Converged: Yes Scale: 0.7866 Coef. Std.Err. z P>|z| [0.025 0.975] Intercept (Baseline) 2.921 0.190 15.364 0.000 2.549 3.294 Variation [RL Model] 0.115 0.069 1.668 0.095 -0.020 0.250 prompt Var 0.247 0.086 user_id Var 0.281 0.150 The model converged successfully, yielding a non-significant main effect for the model variation (p=0.095p=0.095). The variance component estimates indicate that prompt-level variability (σ2=0.247σ^2=0.247) was comparable in magnitude to participant-level variability (σ2=0.281σ^2=0.281), validating our choice to use a crossed random-effects structure. A.1 Two One-Sided Tests (TOST) for Equivalence A non-significant p-value indicates a failure to reject the null hypothesis of no difference, but it does not formally prove that the two models are equivalent. To evaluate statistical equivalence, we conducted an equivalence test using the Two One-Sided Tests (TOST) framework [18]. We defined the margin of practical equivalence (Smallest Effect Size of Interest, or Δ ) as ±0.25± 0.25 points on our 5-point Likert scale. The empirical standard deviation of the subjective ratings was approximately 0.88 (SD ≈0.88≈ 0.88). Under the common half-standard-deviation heuristic for practical significance, this corresponds to a margin of approximately ±0.44± 0.44 rating points. Our chosen equivalence margin of Δ=0.25 =0.25 is therefore more conservative. In the TOST procedure at an α=0.05α=0.05 significance level, equivalence is established if the 90%90\% Confidence Interval (CI) of the mean difference falls entirely within the [−Δ,+Δ][- ,+ ] bounds. Using the estimates for the Mean Difference (+0.115+0.115), Standard Error (0.0690.069) and Critical value for the 90% CI (z0.95=1.645z_0.95=1.645), the 90%90\% Confidence Interval is calculated as: CI90%=0.115±(1.645×0.069)=[+0.002,+0.229]CI_90\%=0.115±(1.645× 0.069)=[+0.002,+0.229] (7) The entire 90%90\% Confidence Interval [+0.002,+0.229][+0.002,+0.229] falls within our equivalence bounds of [−0.25,+0.25][-0.25,+0.25], we reject the presence of a practically meaningful difference (pequivalence<0.05p_equivalence<0.05). These results support that the perceptual quality of the 1.5B RL model is statistically equivalent to the GPT-4o mini baseline within the predefined equivalence margin of ±0.25± 0.25 rating points. Appendix B Subjective evaluation user interface To ensure the transparency of our subjective evaluation (Section 6), we provide screenshots of the custom user interface presented to the participants. The study was conducted in two distinct phases: an initial unconstrained prompt elicitation phase (Figure 4) and a subsequent blind A/B model comparison phase (Figure 5). (a) Introduction screen for the prompt elicitation phase, explaining the task to the user. (b) Main user interface for the prompt elicitation phase. Participants listened to an audio clip and could type any desired change into the text box. Figure 4: Screenshots from Phase 1 (Prompt Elicitation) of the subjective evaluation study. (a) Introduction screen for the blind A/B comparison phase. (b) Main user interface for the A/B comparison. For each of the previously elicited prompts, users listened to two versions (“Original” and “Altered”) and rated them on a 5-point Likert scale. Figure 5: Screenshots from Phase 2 (Blind A/B Comparison) of the subjective evaluation study. Appendix C Ablation on KDE bandwidth selection To validate our choice of the bandwidth scaling factor for the Reflective-KDE (h=0.25⋅hScotth=0.25· h_Scott) introduced in Section 3.4, we provide a visual ablation. Standard density estimators, such as Scott’s Rule, typically assume a unimodal Gaussian distribution. While effective for simple datasets, applying an unscaled Scott’s Rule to the highly multimodal and subjective preference data found in continuous audio control results in excessive smoothing. This smoothing destroys the local topological features representing specific user preferences. Figure 6 demonstrates the effect of varying this scaling factor on the final non-parametric reward surface for a representative prompt. To contextualize the density estimation, the raw data heatmap (bottom-right) displays both the density of user evaluations (integers within cells) and the average preference score (cell color) across the discretized space. By scaling the bandwidth by 0.250.25, we successfully counteract over-smoothing, interpolating high-confidence preference clusters while avoiding overfitting to isolated, low-sample noise. Figure 6: Effect of KDE bandwidth scaling on the preference density. The bottom-right panel displays a heatmap of the raw data for a representative prompt; cell colors indicate the aggregated preference score, while the integers inside the cells denote the number of user evaluations in that region. The remaining three panels show the resulting normalized reward surfaces using different bandwidth scaling factors. An under-smoothed factor (0.1, top-left) overfits to individual low-count data points, creating a noisy and fragmented reward signal. An over-smoothed factor (1.0, bottom-left) assumes unimodality, merging distinct preference modes into a single, uninformative gradient. Our chosen setting (0.25, top-right) provides an optimal balance, capturing the essential topological features and local maxima of the underlying preference manifold without overfitting to sparse noise. Appendix D Experimental Details To facilitate full reproducibility of our alignment pipeline, Table 7 provides a comprehensive overview of the exact hyperparameters, hardware configurations, and methodological settings used throughout the training and evaluation phases. Table 7: Hyperparameters and Methodological Details for Reproducibility Parameter Value / Description General & Infrastructure Base Models Qwen/Qwen2.5-0.5B, 1.5B-Instruct Precision bfloat16 (Model weights and gradients) Hardware 1× NVIDIA L40 (48GB VRAM) Distributed Training None (Single GPU) Optimizer AdamW (adamw_torch) LR Scheduler Linear decay (0 warmup steps) Training Duration (GRPO) ≈ 6 hours per model Training Duration (DPO) ≈ 1 hour per model Reward Surface & Reflective-KDE Global Normalization Range [−1.0,1.0][-1.0,1.0] Bandwidth Scaling (h) 0.25⋅hScott0.25· h_Scott Numerical Stability Epsilon (ϵε) 10−710^-7 Malformed Format Penalty −1.0-1.0 Stage 1: GRPO Training Learning Rate 10−510^-5 Max Training Steps 10001000 Per-Device Train Batch Size 44 Gradient Accumulation Steps 3232 KL Penalty (β) 0.50.5 (Optimal), Swept 0.0,0.1,0.5,1.0\0.0,0.1,0.5,1.0\ Number of Rollouts (G) 1616 (Optimal), Swept 2,4,8,16,32\2,4,8,16,32\ Stage 2: Synthetic Data & DPO Refinement Injected Peaks (K) 1010 Total DPO Pairs per Prompt 2020 (10 Positive, 10 Negative) Learning Rate 5×10−65× 10^-6 Max Training Steps 10001000 Per-Device Train Batch Size 11 Gradient Accumulation Steps 1212 KL Penalty (β) 1.01.0 (Optimal), Swept 0.0,0.1,0.5,1.0\0.0,0.1,0.5,1.0\ Evaluation & Inference Decoding Strategy Greedy (do_sample=False, num_beams=1) Max New Tokens 3232 Appendix E Multi-seed experiments As noted in Section 5.4, the performance of the hybrid pipeline fluctuates depending on the number of rollouts (G) used during the initial GRPO phase. To determine whether this stems from fundamental algorithmic instability or mere hyperparameter sensitivity, we evaluated our best configuration (G=16G=16) across four independent training seeds. The model achieved a relatively consistent mean GMRR of 0.58±0.0160.58± 0.016 with a 100% parse rate. Table 8: Multi-seed performance of the optimal Qwen2.5-1.5B Hybrid model (G=16G=16). The low standard deviation (±0.016± 0.016) demonstrates high training stability. Main model (GMRR = 0.60) not included. Training Run GMRR Parse Rate Seed 1 0.60 100% Seed 2 0.58 100% Seed 3 0.56 100% Seed 4 0.58 100% Mean (± Std. Dev.) 0.58 ± 0.016 100%