Paper deep dive
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
Yang Qu, Yusheng Han, Chengjia Feng, Handan Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/13/2026, 3:45:37 AM
Summary
The paper introduces LSC-DPO (Learning-Signal-Controlled Direct Preference Optimization), a method that dynamically regulates the DPO coefficient beta to maintain the local sensitivity (learning signal) of the loss function near a target regime. This approach addresses the issue where standard DPO loss sensitivity decays as training progresses. The authors derive a stability condition for this control mechanism and a signal-budget compensation rule to handle initialization variance. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH demonstrate that LSC-DPO outperforms DPO and other preference optimization baselines like beta-DPO and epsilon-DPO.
Entities (12)
Relation Signals (9)
sigmoid factor → characterizes → local sensitivity
confidence 95% · identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective.
LSC-DPO → evaluatedon → AlpacaEval 2
confidence 95% · Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO
LSC-DPO → evaluatedon → Anthropic-HH
confidence 95% · Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO
LSC-DPO → evaluatedon → MT-Bench
confidence 95% · Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO
LSC-DPO → improvesover → DPO
confidence 95% · Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines.
LSC-DPO → uses → sigmoid factor
confidence 95% · We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO)...
LSC-DPO → outperforms → epsilon-DPO
confidence 90% · It also consistently outperforms β-DPO and ε-DPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.
Tags
Links
- Source: https://arxiv.org/abs/2610.07592v1
- Canonical: https://arxiv.org/abs/2610.07592v1
Trouble viewing inline? Open PDF directly →
Full Text
83,038 characters extracted from source content.
Expand or collapse full text
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization Yang Qu Affiliation: Northeastern University Yusheng Han Affiliation: Northeastern University Chengjia FengHandan Liu* Email: qu.yang1, han.yus, feng.chen, h.liu@northeastern.edu*Corresponding author Affiliation: Northeastern University Abstract Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-H show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.11 1 Code is available at https://github.com/handan-liu-lab/lsc-dpo. 1 Introduction Large language models (LLMs) have achieved remarkable performance across a broad range of tasks. However, their outputs are not always aligned with human preferences. Reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022; Stiennon et al., 2020) has become a widely used paradigm for improving alignment. In the standard RLHF pipeline, approaches such as Proximal Policy Optimization (PPO) (Schulman et al., 2017) optimize a policy based on a learned reward model that captures human preferences, while a KL penalty constrains the policy from deviating excessively from a reference model. Recent work has explored reward-model-free approaches to reduce the computational cost of RLHF. Direct Preference Optimization (DPO) (Rafailov et al., 2023) directly optimizes a policy from offline preference data without explicitly training a reward model. In DPO, preference learning is formulated through the policy-reference log-ratio difference between chosen and rejected responses. Under the Bradley–Terry model (Bradley and Terry, 1952), the scaled log-ratio difference can be interpreted as an implicit reward difference. The coefficient β scales this quantity and thereby affects the resulting optimization dynamics. A central question in DPO is how to characterize and regulate its evolving optimization state during training. The DPO objective contains a logistic loss term whose sensitivity decreases as optimization progresses (Rafailov et al., 2023; Rafailov et al., 2024; Gorbatovski et al., 2024). As this sensitivity decreases, the same preference pairs receive progressively smaller gradient weight, weakening their contribution to subsequent policy updates. Importantly, the initial choice of the coefficient β also affects the sensitivity of the logistic loss term and can therefore lead to different optimization trajectories and downstream performance (Rafailov et al., 2023). These observations motivate a principled mechanism for characterizing and regulating variations in the training state. In this paper, we analyze the DPO objective from a loss-level geometric perspective and identify a sigmoid factor that characterizes the local sensitivity of the objective during training. This factor reflects both the evolving optimization state and the effect of the coefficient β, making it a useful quantity for diagnosing and regulating DPO training. Based on this observation, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal to keep it close to a target level during preference optimization. We further derive a local stability condition for tracking the desired learning-signal regime. Empirically, LSC-DPO achieves strong performance compared with existing preference-optimization methods (Rafailov et al., 2023; Wu et al., 2024b; Lee et al., 2025; Park et al., 2024; Azar et al., 2024; Ethayarajh et al., 2024; Xu et al., 2024; Hong et al., 2024; Zhao et al., 2023; Meng et al., 2024) across standard alignment benchmarks and preference-optimization experiments. We further use the learning-signal view to analyze sensitivity to the initial choice of β and derive a signal-budget compensation rule that reduces initialization-induced performance degradation and improves robustness across different initial coefficients. Our contributions are summarized in three aspects. (1) We provide a loss-level geometric characterization of DPO, showing that the sigmoid factor serves as a learning signal for tracking the local sensitivity of the objective and the evolving training state. (2) We propose LSC-DPO, a learning-signal-controlled preference optimization method that dynamically adjusts the DPO coefficient according to the evolving training state. We further derive a local stability condition that characterizes stable tracking around the target learning-signal regime. Empirically, LSC-DPO achieves strong performance compared with existing preference-optimization baselines across standard alignment benchmarks. (3) We investigate the relationship between the initial coefficient and learning-signal dynamics, and derive a signal-budget compensation rule that reduces initialization-induced performance degradation and improves robustness across different initial coefficients. 2 Preliminaries In RLHF, a policy is optimized against a reward function while being regularized toward a reference policy (Christiano et al., 2017; Jaques et al., 2017; Jaques et al., 2020): maxπθx∼,y∼πθ(⋅|x)[r(x,y)]−βx∼[KL(πθ(⋅|x)∥πref(⋅|x))], _ _θ\;E_x ,\,y _θ(·|x) [r(x,y) ]-β\,E_x [D_KL ( _θ(·|x)\,\|\, _ref(·|x) ) ], (1) where πθ _θ denotes the policy and πref _ref denotes the reference model. DPO (Rafailov et al., 2023) uses the closed-form solution (Ziegler et al., 2019; Peng et al., 2019; Peters and Schaal, 2007; Go et al., 2023; Rafailov et al., 2023) to reparameterize the reward function as: r(x,y)=βlogπθ(y∣x)πref(y∣x)+βlogZ(x),r(x,y)=β _θ(y x) _ref(y x)+β Z(x), (2) where Z(x)Z(x) is the partition function. Substituting the reparameterized reward into the Bradley–Terry model (Bradley and Terry, 1952) yields the DPO objective: ℒDPO(πθ;πref)=−(x,y+,y−)[logσ( _DPO( _θ; _ref)=-E_(x,y_+,y_-) [ σ ( βlogπθ(y+∣x)πref(y+∣x)−βlogπθ(y−∣x)πref(y−∣x))], β _θ(y_+ x) _ref(y_+ x)-β _θ(y_- x) _ref(y_- x) ) ], (3) where y+y_+ and y−y_- denote the chosen and rejected responses, respectively. The DPO objective depends on the difference between the policy log-ratio of the chosen and rejected responses and the corresponding reference model log-ratio. We define this difference as the preference margin: m=logπθ(y+∣x)πθ(y−∣x)−logπref(y+∣x)πref(y−∣x).m= _θ(y_+ x) _θ(y_- x)- _ref(y_+ x) _ref(y_- x). (4) This margin measures how much more the policy favors the chosen response over the rejected response, relative to the reference model. We further define the scaled preference margin as z=βm.z=β m. (5) Using the margin formulation above, the DPO objective for a single preference pair can be written compactly as ℓDPO=−logσ(βm)=−logσ(z). _DPO=- σ(β m)=- σ(z). (6) Thus, the optimization behavior of DPO depends on the scaled margin z=βmz=β m, which couples the evolving preference margin with the coefficient β. 3 Learning-Signal-Controlled Direct Preference Optimization In this section, we introduce Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO). We first identify the sigmoid factor σ(−z)σ(-z) in the DPO objective as a learning signal that reflects the local sensitivity of the DPO loss during training and illustrate how different initial choices of β affect the evolution of this signal. We then use this signal to characterize the training state of DPO and derive an update rule that regulates the signal during preference optimization. We analyze the stability of the resulting dynamics in log-space. 3.1 Learning Signal in DPO To understand the training behavior of DPO, we analyze the local structure of the DPO objective in Eq. 6. The gradient can be written as ∇θℓ=−σ(−zθ)∇θzθ. _θ =-σ(-z_θ) _θz_θ. (7) Taking another derivative gives the Hessian of the DPO objective: ∇θ2ℓ=σ(zθ)σ(−zθ)∇θzθ∇θzθ⊤−σ(−zθ)∇θ2zθ. _θ^2 =σ(z_θ)σ(-z_θ) _θz_θ _θz_θ -σ(-z_θ) _θ^2z_θ. (8) Figure 1: Illustration of the DPO loss geometry as a function of the scaled margin z=βmz=β m. The learning signal σ(−z)σ(-z) and curvature factor c(z)=σ(z)σ(−z)c(z)=σ(z)σ(-z) decrease as z grows. Equations 7 and 8 show that σ(−zθ)σ(-z_θ) plays a central role in the training dynamics of DPO. From the gradient perspective, σ(−zθ)σ(-z_θ) directly measures the first-order sensitivity of the DPO loss. From the curvature perspective, the Hessian contains two scaled terms: the outer-product term is weighted by σ(zθ)σ(−zθ)σ(z_θ)σ(-z_θ), while the second-order term is weighted by σ(−zθ)σ(-z_θ). During training, zθz_θ typically increases for preference pairs that become better aligned with the preference ordering. As zθz_θ grows, the sigmoid factor σ(−zθ)σ(-z_θ) decreases. Since these factors scale the Hessian terms that determine local curvature, their decrease indicates reduced local sensitivity of the objective and a weaker preference-learning tendency. Figure 1 illustrates this loss geometry. We therefore use σ(−zθ)σ(-z_θ) as a learning signal for DPO training. When this signal is relatively large, the DPO loss remains more sensitive to changes in the scaled margin, corresponding to a high-signal regime. As zθz_θ increases, σ(−zθ)σ(-z_θ) decreases and the loss becomes progressively less sensitive to further changes in the scaled margin. We refer to this as a low-signal regime, which progressively approaches a lower-curvature region as zθz_θ grows further. Figures 2(a) and 2(b) further show that σ(−zθ)σ(-z_θ) evolves systematically over training. It remains relatively high during the early stage and gradually decreases before stabilizing at a lower level. This evolution is consistent with our interpretation of σ(−zθ)σ(-z_θ) as reflecting the local responsiveness of the DPO loss during training. The figures further show that different choices of the DPO coefficient lead to markedly different learning-signal trajectories σ(−z)σ(-z). This observation suggests that the learning-signal dynamics are related to the coefficient setting and motivates our later analysis of coefficient initialization in Section 4.4. (a) H-harmless-base (b) H-helpful-base Figure 2: σ(−z)σ(-z) under fixed β values during DPO training with Llama3-8B on two Anthropic-H subsets: (a) harmless-base and (b) helpful-base. EMA-smoothed trajectories are shown for readability; raw trajectories are provided in Appendix A. 3.2 Learning-Signal-Controlled Update Since σ(−z)σ(-z) reflects the learning signal in DPO, we aim to keep it near a target level rather than allowing it to decay rapidly during training. Because σ(−z)σ(-z) decreases monotonically with the scaled margin z, regulating z provides a direct way to control the signal. Since z=βmz=β m, we use β as the control variable. At training step k, we compute the per-example preference margins on the current batch and use their mean as the batch-level margin mkm_k. We then obtain the corresponding scaled margin: zk=βkmk.z_k= _km_k. (9) Thus, the controller computes the signal from the scaled batch-mean margin zkz_k. We then measure its deviation from a target scaled margin z∗z^*: uk=zk−z∗.u_k=z_k-z^*. (10) The target z∗z^* determines the learning-signal regime that LSC-DPO tracks, equivalently corresponding to the target signal level σ(−z∗)σ(-z^*). LSC-DPO updates β using the following multiplicative rule: βk+1=βkexp(−ηuk)=βkexp[−η(βkmk−z∗)], _k+1= _k (-η u_k)= _k [-η( _km_k-z^*) ], (11) where η>0η>0 controls the adjustment rate. The multiplicative form preserves the positivity of β and implements feedback control over the scaled margin. In the positive-margin regime, when zk>z∗z_k>z^*, σ(−zk)σ(-z_k) is below the target level and the update decreases βk _k, moving the scaled margin downward. Conversely, when zk<z∗z_k<z^*, the update increases βk _k, moving the scaled margin upward toward the target. The complete procedure is provided in Algorithm 1 in Appendix B. 3.3 Stability Analysis of the Update Process We analyze the LSC-DPO update in log-space to characterize controller-level tracking of the target z∗z^*, rather than convergence of the full policy optimization problem. Starting from Eq. 11, let xk:=logβk.x_k:= _k. (12) Substituting βk=exk _k=e^x_k into Eq. 11 gives xk+1=xk−η(exkmk−z∗).x_k+1=x_k-η (e^x_km_k-z^* ). (13) Since the margin statistic mkm_k evolves during training, the induced system is non-autonomous. Here, mkm_k denotes the realized batch-level margin at step k and may depend on the current policy, minibatch sampling, and previous coefficient updates. Accordingly, our analysis is conditional on the realized margin sequence and characterizes controller-level tracking rather than joint convergence of the full policy-training dynamics. Conditioned on mkm_k, the step-dependent update map is Fk(x)=x−η(exmk−z∗).F_k(x)=x-η (e^xm_k-z^* ). (14) Instantaneous equilibrium. For a realized mkm_k, the map FkF_k admits an instantaneous fixed point xkf=logz∗mk,βkf=z∗mk,x_k^f= z^*m_k, _k^f= z^*m_k, (15) provided that mk>0m_k>0 and z∗>0z^*>0. Because mkm_k evolves during training, this equilibrium moves over time. We next characterize its local stability. Local stability. Linearizing the step-dependent map FkF_k around the instantaneous equilibrium xkfx_k^f gives Fk′(xkf)=1−ηz∗.F_k (x_k^f)=1-η z^*. (16) Hence, the instantaneous equilibrium is locally asymptotically stable whenever 0<ηz∗<2.0<η z^*<2. (17) Notably, this condition is independent of the realized margin mkm_k. The formal proposition and proof are provided in Appendix C.1. Bounded tracking. Because mkm_k evolves during training, the corresponding equilibrium xkfx_k^f also moves over time. Under local contraction and bounded equilibrium drift, the resulting tracking error satisfies |xk−xkf|≤ρk−T|xT−xTf|+δ¯1−ρ,k≥T,|x_k-x_k^f|≤ρ^k-T|x_T-x_T^f|+ δ1-ρ, k≥ T, (18) where ρ∈(0,1)ρ∈(0,1) denotes a local contraction factor and δ¯ δ bounds the step-to-step motion of the instantaneous equilibrium. The first term decays geometrically, while the second captures the persistent error induced by the moving target. Thus, when the realized margin evolves sufficiently gradually, the controller remains close to the target scaled-margin regime and keeps the learning signal near σ(−z∗)σ(-z^*). The complete theorem, assumptions, and proof are provided in Appendix C.2. 4 Experiments In this section, we evaluate LSC-DPO on standard alignment benchmarks and analyze its training dynamics. We first compare LSC-DPO with representative preference-optimization baselines on AlpacaEval 2 (Dubois et al., 2024) and MT-Bench (Zheng et al., 2023b), and then study its performance and learning-signal dynamics on Anthropic-H (Bai et al., 2022). We further validate the proposed learning-signal control through optimization-dynamics analyses and comparisons with predetermined β schedules. Finally, we investigate robustness to the initial DPO coefficient and the proposed signal-budget compensation rule. Additional analyses of target-signal selection and hyperparameters are provided in Appendix J, control stability in Appendix G, and downstream capability evaluations in Appendix K. 4.1 Experimental Setup Models and training datasets. We evaluate on two open-source models, meta-llama/Meta-Llama-3-8B (Llama3-8B) (Grattafiori et al., 2024) and Qwen/Qwen3-8B-Base (Qwen3-8B) (Yang et al., 2025). We first perform supervised fine-tuning on UltraChat-200k (Ding et al., 2023), and then perform preference optimization on UltraFeedback (Cui et al., 2023). We additionally use the harmless-base and helpful-base subsets of Anthropic-H (Bai et al., 2022) for controlled analyses. Full training settings are provided in Appendix E.1. Evaluation benchmarks and baselines. For the UltraFeedback experiments, we evaluate on AlpacaEval 2 (Dubois et al., 2024) using win rate and length-controlled (LC) win rate, and on MT-Bench (Zheng et al., 2023b). On Anthropic-H, we report pairwise win rates on both harmless-base and helpful-base. We compare our method with representative preference-optimization baselines, including DPO (Rafailov et al., 2023), β-DPO (Wu et al., 2024b), ε -DPO (Lee et al., 2025), R-DPO (Park et al., 2024), IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), CPO (Xu et al., 2024), ORPO (Hong et al., 2024), SLiC-HF (Zhao et al., 2023), and SimPO (Meng et al., 2024). For the controlled Anthropic-H analyses, we focus on DPO, β-DPO, and ε -DPO as the most relevant coefficient-control baselines. Detailed evaluation protocols and baseline configurations are provided in Appendix E.2. 4.2 Main Results on Standard Alignment Benchmarks Table 1: Evaluation results on AlpacaEval 2 and MT-Bench. For AlpacaEval 2, we report both win rate (WR) and length-controlled win rate (LC). For MT-Bench, we report the average score on a 1–10 scale. Method Llama3-8B Qwen3-8B AlpacaEval 2 MT-Bench AlpacaEval 2 MT-Bench LC (%) WR (%) Score LC (%) WR (%) Score SFT 6.8 3.7 6.3 8.5 4.8 6.9 DPO 19.2 17.7 6.6 17.2 13.1 7.6 ε -DPO 19.6 17.6 6.9 19.4 14.0 7.6 β-DPO 16.6 16.5 6.8 16.4 10.3 7.2 CPO 11.2 10.3 6.4 16.5 14.5 7.6 IPO 8.6 8.5 6.6 14.9 9.2 7.4 KTO 16.9 15.1 6.8 17.2 11.3 7.4 ORPO 6.4 5.3 5.5 9.4 6.8 6.9 R-DPO 19.5 18.2 6.8 19.3 13.2 7.6 SLiC-HF 10.1 8.7 5.9 12.5 10.5 7.1 SimPO 19.7 19.2 6.9 16.8 10.9 7.4 LSC-DPO 22.7 21.1 6.9 22.1 16.1 7.8 LSC-DPO achieves the strongest overall performance. Table 1 reports that LSC-DPO achieves the best AlpacaEval 2 performance on both Llama3-8B and Qwen3-8B, while attaining top or tied-top MT-Bench scores. Compared with DPO, LSC-DPO improves the length-controlled win rate from 19.2 to 22.7 on Llama3-8B and from 17.2 to 22.1 on Qwen3-8B, with similar gains in standard win rate. It also consistently outperforms β-DPO and ε -DPO and remains competitive with broader preference-optimization baselines, including R-DPO, KTO, CPO, ORPO, SLiC-HF, IPO, and SimPO. We next use Anthropic-H for controlled comparisons of coefficient-regulation methods and their learning-signal behavior. (a) H-harmless-base (b) H-helpful-base Figure 3: Performance comparison on Anthropic-H. We report win rates of DPO, β-DPO, ε -DPO, and LSC-DPO using Llama3-8B and Qwen3-8B. Error bars denote approximate 95% confidence intervals computed from prompt-level GPT-4 judgment results. (a) harmless-base (b) helpful-base Figure 4: Learning signal σ(−z)σ(-z) on Anthropic-H using Llama3-8B. The corresponding raw curves and Qwen results are provided in Appendix F. Overall performance on Anthropic-H. Figures 3(a) and 3(b) show that LSC-DPO achieves the best pairwise win rates across both model families and Anthropic-H subsets, reaching approximately 79% on harmless-base and 82–87% on helpful-base. It consistently outperforms DPO, β-DPO, and ε -DPO, further supporting the effectiveness of learning-signal control under controlled preference-optimization settings. Learning-signal dynamics. Figures 4(a) and 4(b) show that DPO exhibits a sustained signal decay and eventually moves toward a low-signal regime, β-DPO shows larger fluctuations, and ε -DPO partially recovers after an initial decline. In contrast, LSC-DPO maintains a higher and more stable signal near its target regime across most of training. These relative dynamics are consistent with the performance trends in Figure 3; on helpful-base, for instance, ε -DPO maintains a lower signal than β-DPO for much of training before recovering later, aligning with its lower win rate. Overall, these results support the intended learning-signal regulation behavior of LSC-DPO. 4.3 Mechanistic Validation of Learning-Signal Control Table 2: Optimization diagnostics before and after LSC-DPO control activation. We report the mean curvature factor and mean pre-clipping gradient norm. Before control (steps 1–83) After control (steps 84–477) Method Curvature Grad. norm Curvature Grad. norm DPO 0.24993 29.861 0.23724 90.330 LSC-DPO 0.24993 29.896 0.24371 73.373 Table 3: Comparison with predetermined β schedules. Method LC (%) WR (%) DPO 19.2 17.7 Linear β schedule 19.7 18.4 Exponential β schedule 20.3 18.5 Cosine β schedule 19.1 18.1 LSC-DPO 22.7 21.1 Learning signal and realized optimization dynamics. Table 2 shows that DPO and LSC-DPO are nearly indistinguishable before control activation: both have the same mean curvature factor and almost identical pre-clipping gradient norms. After control activation, LSC-DPO maintains a larger curvature factor than DPO. The slower reduction of this quantity indicates that LSC-DPO keeps the objective farther from the flatter, low-curvature regime. At the same time, its pre-clipping gradient norm is lower, showing that this increased loss-level responsiveness is not obtained by simply amplifying the overall parameter-gradient magnitude. Together, these results support our interpretation of σ(−z)σ(-z) as a meaningful loss-level training signal. Full training trajectories are provided in Appendix H.1. Feedback versus predetermined β schedules. Table 3 compares LSC-DPO with several predetermined schedules. Linear, exponential, and cosine schedules yield only modest changes relative to DPO, whereas LSC-DPO achieves substantially higher LC and WR. This suggests that the improvement cannot be explained solely by using a time-varying β schedule and supports the role of state-dependent learning-signal feedback in LSC-DPO. 4.4 Initialization Robustness and Signal Compensation Figure 5: Learning-signal trajectories under different initial β0 _0. Curves are EMA-smoothed. Initialization sensitivity. Table 4 shows substantial sensitivity to β0 _0 across all methods. LSC-DPO achieves the highest performance at every tested initialization and degrades less severely, but still exhibits a 7.5-point LC spread. Figure 5 further shows how different β0 _0 values affect the learning-signal trajectory σ(−z)σ(-z). Although LSC-DPO eventually regulates the trajectories toward a similar controlled level, larger β0 _0 causes a faster decay of σ(−z)σ(-z) during the early stage of training. Table 4: Initialization sensitivity on AlpacaEval 2. We report length-controlled (LC) win rate (%). † denotes the reference setting. Full LC/WR results are provided in Appendix I.1. Initial coefficient β0 _0 Method 0.01† 0.05 0.10 0.30 0.80 DPO 19.2 10.4 8.7 8.3 6.6 β-DPO 16.6 10.5 10.8 7.3 6.9 ε -DPO 19.6 15.6 12.9 8.8 6.9 LSC-DPO 22.7 19.6 19.1 15.3 15.2 Signal-budget compensation. We hypothesize that the residual performance variation across different initial coefficients is related to differences in the cumulative learning signal during the transient stage. For an arbitrary anchor run a, we define the cumulative signal deficit as DH(β0)=∑t=1H[σ(−zta)−σ(−zt(β0))],D_H( _0)= _t=1^H [σ(-z_t^a)-σ(-z_t^( _0)) ], (19) where H denotes the end of the transient phase. A positive DH(β0)D_H( _0) indicates that the run accumulates a smaller learning signal than the anchor during this stage. We compensate for this deficit over the remaining T−HT-H steps by setting σ(−zcomp∗)=σ(−za∗)+DH(β0)T−H,zcomp∗=log1−σ(−zcomp∗)σ(−zcomp∗),σ(-z_comp^*)=σ(-z_a^*)+ D_H( _0)T-H, z_comp^*= 1-σ(-z_comp^*)σ(-z_comp^*), (20) where zcomp∗z_comp^* denotes the compensated target scaled margin corresponding to the compensated learning-signal target. Thus, a larger early-stage deficit leads to a higher compensated learning-signal target and, equivalently, a smaller z∗z^*. Under ideal post-transient tracking, this rule matches the cumulative learning signal of the anchor run. We evaluate below whether this compensation also reduces the corresponding performance gap. Table 5: Effect of full signal-budget compensation on AlpacaEval 2. We report length-controlled (LC) win rate (%). Baseline Full compensation β0 _0 z∗z^* LC zcomp∗z_comp^* LC (Δ)\;( ) 0.05 0.400 19.6 0.359 21.5 (+1.9) 0.10 0.400 19.1 0.348 21.1 (+2.0) 0.30 0.400 15.3 0.333 20.4 (+5.1) 0.80 0.400 15.2 0.324 19.7 (+4.5) Compensation results. Table 5 shows that signal-budget compensation improves the LC win rate for all tested non-anchor initializations, with gains of up to 5.15.1 points, while reducing the LC spread from 4.44.4 to 1.81.8 points. These results indicate that adjusting the learning signal can partially compensate for the performance differences induced by coefficient initialization. Overall, the compensation substantially reduces, but does not eliminate, initialization sensitivity. Full results and additional analyses are provided in Appendix I.2. 5 Related Work Reinforcement learning from human feedback. Reinforcement learning from human feedback (RLHF) has become a central approach for aligning large language models with human preferences and values (Bai et al., 2022; Christiano et al., 2017; Ouyang et al., 2022; Stiennon et al., 2020; Ziegler et al., 2019). A typical RLHF pipeline first collects preference comparisons, trains a reward model from human feedback (Chen et al., 2024; Gao et al., 2023; Havrilla et al., 2024; Luo et al., 2023), and then optimizes the policy with a reinforcement learning algorithm (Anthony et al., 2017; Schulman et al., 2017). Proximal policy optimization (PPO) is commonly used for this policy optimization stage (Schulman et al., 2017). Preference-based alignment has also been applied to improve helpfulness and safety, reduce toxic outputs, improve factuality, and control multiple objectives (Dai et al., 2023; Korbak et al., 2023; Zheng et al., 2023a; Tian et al., 2024; Wang et al., 2024). Direct preference optimization and variants. DPO (Rafailov et al., 2023) has motivated a broad family of direct alignment methods that optimize language models from offline preference data without training an explicit reward model. Representative extensions include IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), RRHF (Yuan et al., 2023), R-DPO (Park et al., 2024), ODPO (Amini et al., 2024), CPO (Xu et al., 2024), ORPO (Hong et al., 2024), SLiC-HF (Zhao et al., 2023), and SimPO (Meng et al., 2024). These methods extend preference optimization through different objective designs, reference-model formulations, and robustness mechanisms. DPO training dynamics and coefficient control. Recent work has studied DPO optimization from training-dynamics and coefficient-control perspectives. Direct alignment methods can also exhibit overoptimization behavior as the policy moves away from the reference model (Rafailov et al., 2024). A related line of work studies this training behavior and mitigates it by updating the reference policy during training (Gorbatovski et al., 2024). In the coefficient-control direction, β-DPO (Wu et al., 2024b) updates β using empirical reward-margin statistics, while ε -DPO (Lee et al., 2025) perturbs the coefficient to reduce sensitivity to a fixed β. Related margin-adaptive approaches include AlphaDPO (Wu et al., 2024a) and γ-PO (Sun et al., 2025). 6 Conclusion In this paper, we introduced Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), a feedback-control approach that regulates the learning signal of DPO during preference optimization. By analyzing the DPO objective from a loss-level geometric perspective, we identified the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective and used it to design a control rule that regulates DPO training dynamics around a target learning-signal level. Our log-space analysis shows that, under a simple local stability condition, the feedback update can stably track the target learning-signal regime. Empirical results further show that LSC-DPO maintains more sustained learning-signal dynamics and achieves stronger alignment performance. We further showed that different coefficient initializations are related to learning-signal dynamics and derived a signal-budget compensation rule that substantially reduces the resulting performance variation. Limitations and Future Work. Although the best-performing learning-signal levels are similar across the model families studied in this work, we do not yet provide a complete theory explaining why these target ranges emerge or when they transfer across models and datasets. In addition, although signal-budget compensation substantially reduces initialization-induced performance variation, it does not completely eliminate the remaining performance gap. Future work may develop a more complete theory of target learning-signal selection and signal-budget dynamics, and extend learning-signal control beyond DPO to other preference-optimization objectives. AI use statement Generative AI tools were used in this work to edit the manuscript for grammar and readability, to check the formatting and the bibliography against the original sources, and to assist with writing and debugging the training and evaluation scripts. Generative AI tools were not used to generate synthetic data sets, to develop the theoretical framework, or to formulate or prove the mathematical claims; the remaining tasks with required disclosure are not applicable to this work. All AI-assisted text was reviewed and revised by the authors, AI-assisted code was verified and tested by the authors, and every citation was checked against the original publication. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI. Ethics statement This work studies preference optimization for language-model alignment. All experiments use publicly available datasets (UltraChat-200k, UltraFeedback, and Anthropic-H) and open-weight base models under their respective licenses, and no human-subject data were collected. The Anthropic-H harmless-base subset contains prompts designed to elicit harmful responses; we use it only to evaluate harmlessness. As with any preference-optimization method, LSC-DPO can in principle be used to align a model toward undesirable preferences; our method does not change this risk relative to DPO. We do not release any new dataset. Reproducibility statement The complete LSC-DPO procedure is given in Algorithm 1 (Appendix B). Models, datasets, hyperparameters, and compute are listed in Appendix E.1, and the evaluation protocols, judge models, and decoding settings in Appendix E.2. The assumptions and full proofs of the stability results are in Appendices C and D. The predetermined β schedules and the signal-budget compensation targets are specified in Appendices H.2 and I.2. References Amini et al. (2024) A. Amini, T. Vieira, and R. Cotterell Direct preference optimization with an offset. In Findings of the Association for Computational Linguistics: ACL 2024, p. 9954–9972. Cited by: §5. Anthony et al. (2017) T. Anthony, Z. Tian, and D. Barber Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems 30. Cited by: §5. Azar et al. (2024) M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. Cited by: §E.2, §1, §4.1, §5. Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §E.1, §4.1, §4, §5. Bradley and Terry (1952) R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §1, §2. Chen et al. (2024) L. Chen, C. Zhu, J. Chen, D. Soselia, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro ODIN: disentangled reward mitigates hacking in RLHF. In International Conference on Machine Learning, Vol. 235, p. 7935–7952. Cited by: §5. Christiano et al. (2017) P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems 30. Cited by: §1, §2, §5. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: Appendix K. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix K. Cui et al. (2023) G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, et al. UltraFeedback: boosting language models with scaled AI feedback. arXiv preprint arXiv:2310.01377. Cited by: §E.1, §4.1. Dai et al. (2023) J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang Safe RLHF: safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773. Cited by: §5. Ding et al. (2023) N. Ding, Y. Chen, B. Xu, Y. Qin, S. Hu, Z. Liu, M. Sun, and B. Zhou Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 3029–3051. Cited by: §E.1, §4.1. Dubois et al. (2024) Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto Length-controlled AlpacaEval: a simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. Cited by: §E.2, §4.1, §4. Ethayarajh et al. (2024) K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela Model alignment as prospect theoretic optimization. In International Conference on Machine Learning, Vol. 235, p. 12634–12651. Cited by: §E.2, §1, §4.1, §5. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In International Conference on Machine Learning, p. 10835–10866. Cited by: §5. Go et al. (2023) D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman Aligning language models with preferences through f-divergence minimization. In International Conference on Machine Learning, Vol. 202, p. 11546–11583. Cited by: §2. Gorbatovski et al. (2024) A. Gorbatovski, B. Shaposhnikov, A. Malakhov, N. Surnachev, Y. Aksenov, I. Maksimov, N. Balagansky, and D. Gavrilov Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656. Cited by: §1, §5. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.1. Havrilla et al. (2024) A. Havrilla, S. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, and R. Raileanu GLoRe: when, where, and how to improve LLM reasoning via global and local refinements. arXiv preprint arXiv:2402.10963. Cited by: §5. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: Appendix K. Hong et al. (2024) J. Hong, N. Lee, and J. Thorne ORPO: monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 11170–11189. Cited by: §E.2, §1, §4.1, §5. Jaques et al. (2017) N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck Sequence tutor: conservative fine-tuning of sequence generation models with KL-control. In International Conference on Machine Learning, p. 1645–1654. Cited by: §2. Jaques et al. (2020) N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 3985–4003. Cited by: §2. Korbak et al. (2023) T. Korbak, K. Shi, A. Chen, R. V. Bhalerao, C. Buckley, J. Phang, S. R. Bowman, and E. Perez Pretraining language models with human preferences. In International Conference on Machine Learning, p. 17506–17533. Cited by: §5. Lee et al. (2025) S. Lee, J. Han, H. Song, S. J. Choi, H. Lee, and Y. Yu KL penalty control via perturbation for direct preference optimization. Advances in Neural Information Processing Systems 38, p. 168095–168121. Cited by: Appendix K, §E.1, §E.2, §E.2, §1, §4.1, §5. Li et al. (2024) T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. arXiv preprint arXiv:2406.11939. Cited by: §E.2. Lin et al. (2022) S. Lin, J. Hilton, and O. Evans TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252. Cited by: Appendix K. Luo et al. (2023) H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang WizardMath: empowering mathematical reasoning for large language models via reinforced Evol-Instruct. arXiv preprint arXiv:2308.09583. Cited by: §5. Meng et al. (2024) Y. Meng, M. Xia, and D. Chen SimPO: simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37, p. 124198–124235. Cited by: Appendix K, §E.1, §E.2, §E.2, §1, §4.1, §5. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, p. 27730–27744. Cited by: §1, §5. Park et al. (2024) R. Park, R. Rafailov, S. Ermon, and C. Finn Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, p. 4998–5017. Cited by: §E.2, §1, §4.1, §5. Peng et al. (2019) X. B. Peng, A. Kumar, G. Zhang, and S. Levine Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. Cited by: §2. Peters and Schaal (2007) J. Peters and S. Schaal Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, p. 745–750. Cited by: §2. Rafailov et al. (2024) R. Rafailov, Y. Chittepu, R. Park, H. Sikchi, J. Hejna, W. B. Knox, C. Finn, and S. Niekum Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems 37, p. 126207–126242. Cited by: §1, §5. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36, p. 53728–53741. Cited by: §E.1, §E.2, §E.2, §1, §1, §1, §2, §4.1, §5. Sakaguchi et al. (2021) K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial Winograd schema challenge at scale. Communications of the ACM 64 (9), p. 99–106. External Links: Document Cited by: Appendix K. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §5. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33, p. 3008–3021. Cited by: §1, §5. Sun et al. (2025) J. Sun, J. Wu, J. Wu, Z. Zhu, X. Lu, J. Zhou, L. Ma, and X. Wang Robust preference optimization via dynamic target margins. In Findings of the Association for Computational Linguistics: ACL 2025, p. 5399–5416. Cited by: §5. Tian et al. (2024) K. Tian, E. Mitchell, H. Yao, C. D. Manning, and C. Finn Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations, Cited by: §5. Wang et al. (2024) H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang Arithmetic control of LLMs for diverse user preferences: directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8642–8655. Cited by: §5. Wu et al. (2024a) J. Wu, X. Wang, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He AlphaDPO: adaptive reward margin for direct preference optimization. arXiv preprint arXiv:2410.10148. Cited by: §5. Wu et al. (2024b) J. Wu, Y. Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He β-DPO: direct preference optimization with dynamic β. Advances in Neural Information Processing Systems 37, p. 129944–129966. Cited by: §E.1, §E.2, §E.2, §1, §4.1, §5. Xu et al. (2024) H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim Contrastive preference optimization: pushing the boundaries of LLM performance in machine translation. In International Conference on Machine Learning, p. 55204–55224. Cited by: §E.2, §1, §4.1, §5. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1. Yuan et al. (2023) Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang RRHF: rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302. Cited by: §5. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4791–4800. Cited by: Appendix K. Zhao et al. (2023) Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu SLiC-HF: sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. Cited by: §E.2, §1, §4.1, §5. Zheng et al. (2023a) C. Zheng, P. Ke, Z. Zhang, and M. Huang Click: controllable text generation with sequence likelihood contrastive learning. In Findings of the Association for Computational Linguistics: ACL 2023, p. 1022–1040. Cited by: §5. Zheng et al. (2023b) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, p. 46595–46623. Cited by: §E.2, §4.1, §4. Ziegler et al. (2019) D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. Cited by: §2, §5. Appendix A Learning Signal σ(−z)σ(-z) under Fixed β We report the raw per-step trajectories of the learning signal under fixed-β DPO with different β values. Figures 6(a) and 6(b) show that σ(−z)σ(-z) generally decreases during training on both Anthropic-H subsets. (a) harmless-base (b) helpful-base Figure 6: Learning signal under DPO. Raw per-step trajectories of σ(−z)σ(-z) are shown for different fixed β values on Anthropic-H: (a) harmless-base and (b) helpful-base. Appendix B LSC-DPO Algorithm Algorithm 1 summarizes the complete training procedure of LSC-DPO, including computation of the learning signal, controller activation, and the subsequent coefficient update described in Section 3.2. Algorithm 1 Learning-Signal-Controlled Direct Preference Optimization 1: policy πθ _θ, reference πref _ref, initial β0 _0, z∗z^*, η, τstart _start 2: β←β0β← _0; control is inactive 3: while not converged do 4: Sample a batch of preference triples (x,y+,y−)∼(x,y_+,y_-) 5: Compute margin m, scaled margin z=βmz=β m, and signal σ(−z)σ(-z) 6: Activate control if σ(−z)≤τstartσ(-z)≤ _start 7: If control is active, update β←βexp[−η(z−z∗)]β←β [-η(z-z^*)] 8: Update πθ _θ using the DPO loss with the current β 9: end while 10: return aligned policy πθ _θ Appendix C Proofs for the Stability Analysis In this appendix, we provide the formal statements and detailed proofs for the local stability and bounded tracking results summarized in Section 3.3. Recall that the log-space update is xk+1=Fk(xk),Fk(x):=x−η(exmk−z∗),x_k+1=F_k(x_k), F_k(x):=x-η (e^xm_k-z^* ), (21) and that the instantaneous equilibrium is xkf=logz∗mk.x_k^f= z^*m_k. (22) C.1 Proof of Local Stability The formal local-stability result is stated as follows. Proposition 1 (Local stability). Assume mk>0m_k>0 and z∗>0z^*>0. The instantaneous equilibrium xkf=log(z∗/mk)x_k^f= (z^*/m_k) is locally asymptotically stable whenever 0<ηz∗<2.0<η z^*<2. (23) The corresponding equilibrium linearization factor is ρ0:=|1−ηz∗|<1. _0:=|1-η z^*|<1. (24) This result follows by linearizing FkF_k around xkfx_k^f, where Fk′(xkf)=1−ηz∗.F_k (x_k^f)=1-η z^*. (25) Proof. Since mk>0m_k>0 and z∗>0z^*>0, the instantaneous equilibrium is well defined and satisfies exkfmk=z∗.e^x_k^fm_k=z^*. (26) The derivative of the step-dependent map is Fk′(x)=1−ηexmk.F_k (x)=1-η e^xm_k. (27) Evaluating this derivative at the equilibrium gives Fk′(xkf)=1−ηexkfmk=1−ηz∗.F_k (x_k^f)=1-η e^x_k^fm_k=1-η z^*. (28) For a one-dimensional discrete-time dynamical system, a fixed point is locally asymptotically stable if the derivative at the fixed point has magnitude smaller than one. Hence, |Fk′(xkf)|<1⟺|1−ηz∗|<1.|F_k (x_k^f)|<1 |1-η z^*|<1. (29) Solving this inequality yields 0<ηz∗<2.0<η z^*<2. (30) Under this condition, the equilibrium linearization factor is ρ0:=|Fk′(xkf)|=|1−ηz∗|<1, _0:=|F_k (x_k^f)|=|1-η z^*|<1, (31) which proves the proposition. ∎ C.2 Bounded Tracking Theorem and Proof Theorem 1 (Bounded tracking). Assume the local stability condition in Eq. 17 holds, and define ρ0:=|1−ηz∗|<1. _0:=|1-η z^*|<1. For any ρ satisfying ρ0<ρ<1, _0<ρ<1, there exists a radius r>0r>0 such that, for every k, |Fk′(x)|≤ρwhenever |x−xkf|≤r.|F_k (x)|≤ρ |x-x_k^f|≤ r. (32) Suppose that there exists a step T such that |xT−xTf|≤r,|x_T-x_T^f|≤ r, (33) and that, for some constant δ¯≥0 δ≥ 0 and all k≥Tk≥ T, |xk+1f−xkf|≤δ¯,δ¯≤(1−ρ)r.|x_k+1^f-x_k^f|≤ δ, δ≤(1-ρ)r. (34) Then the iterates remain within the local contraction neighborhood, |xk−xkf|≤r,∀k≥T,|x_k-x_k^f|≤ r, ∀ k≥ T, and satisfy |xk−xkf|≤ρk−T|xT−xTf|+δ¯1−ρ,∀k≥T.|x_k-x_k^f|≤ρ^k-T|x_T-x_T^f|+ δ1-ρ, ∀ k≥ T. (35) The complete proof is given below. An interpretation of the resulting tracking bound is provided in Appendix C.3. We define the tracking error by ek:=xk−xkf.e_k:=x_k-x_k^f. (36) Lemma 1 (One-step tracking recursion). For each step k, there exists some ξk _k between xkx_k and xkfx_k^f such that ek+1=Fk′(ξk)ek+(xkf−xk+1f).e_k+1=F_k ( _k)e_k+ (x_k^f-x_k+1^f ). (37) Consequently, |ek+1|≤|Fk′(ξk)||ek|+|xk+1f−xkf|.|e_k+1|≤|F_k ( _k)|\,|e_k|+|x_k+1^f-x_k^f|. (38) Proof. By definition, ek+1=xk+1−xk+1f.e_k+1=x_k+1-x_k+1^f. (39) Using the update xk+1=Fk(xk)x_k+1=F_k(x_k) and adding and subtracting xkfx_k^f, we obtain ek+1 e_k+1 =Fk(xk)−xk+1f =F_k(x_k)-x_k+1^f =(Fk(xk)−Fk(xkf))+(xkf−xk+1f), = (F_k(x_k)-F_k(x_k^f) )+ (x_k^f-x_k+1^f ), (40) where we used Fk(xkf)=xkfF_k(x_k^f)=x_k^f. By the mean value theorem, there exists some ξk _k between xkx_k and xkfx_k^f such that Fk(xk)−Fk(xkf)=Fk′(ξk)(xk−xkf)=Fk′(ξk)ek.F_k(x_k)-F_k(x_k^f)=F_k ( _k)(x_k-x_k^f)=F_k ( _k)e_k. (41) Substituting this identity gives Eq. 37. Taking absolute values gives Eq. 38. ∎ Proof of Theorem 1. From Proposition 1, the derivative at the instantaneous equilibrium satisfies ρ0:=|Fk′(xkf)|=|1−ηz∗|<1. _0:=|F_k (x_k^f)|=|1-η z^*|<1. (42) Choose any ρ such that ρ0<ρ<1. _0<ρ<1. (43) For a displacement e from the instantaneous equilibrium, Fk′(xkf+e)=1−ηexkf+emk=1−ηz∗e.F_k (x_k^f+e)=1-η e^x_k^f+em_k=1-η z^*e^e. (44) The right-hand side is independent of mkm_k. Since it is continuous in e and equals 1−ηz∗1-η z^* at e=0e=0, there exists a radius r>0r>0 such that |Fk′(xkf+e)|≤ρ,for all |e|≤r|F_k (x_k^f+e)|≤ρ, all |e|≤ r (45) and for all k≥Tk≥ T. We now show that this neighborhood is invariant. By assumption, |eT|≤r.|e_T|≤ r. (46) Suppose inductively that |ek|≤r|e_k|≤ r. Since ξk _k lies between xkx_k and xkfx_k^f, it also satisfies |ξk−xkf|≤|xk−xkf|=|ek|≤r.| _k-x_k^f|≤|x_k-x_k^f|=|e_k|≤ r. (47) Therefore, |Fk′(ξk)|≤ρ.|F_k ( _k)|≤ρ. (48) Using Lemma 1 and the bounded-drift assumption gives |ek+1|≤ρ|ek|+δ¯.|e_k+1|≤ρ|e_k|+ δ. (49) Since δ¯≤(1−ρ)r δ≤(1-ρ)r, |ek+1|≤ρr+(1−ρ)r=r.|e_k+1|≤ρ r+(1-ρ)r=r. (50) Hence, by induction, |ek|≤r,∀k≥T.|e_k|≤ r, ∀ k≥ T. (51) Thus, the local contraction bound used in Eq. 49 remains valid along the entire tracked trajectory. Unrolling Eq. 49 gives |ek|≤ρk−T|eT|+δ¯∑j=0k−T−1ρj.|e_k|≤ρ^k-T|e_T|+ δ _j=0^k-T-1ρ^j. (52) Since 0≤ρ<10≤ρ<1, ∑j=0k−T−1ρj=1−ρk−T1−ρ≤11−ρ. _j=0^k-T-1ρ^j= 1-ρ^k-T1-ρ≤ 11-ρ. (53) Substituting this bound into Eq. 52 yields |ek|≤ρk−T|eT|+δ¯1−ρ,∀k≥T.|e_k|≤ρ^k-T|e_T|+ δ1-ρ, ∀ k≥ T. (54) Since ek=xk−xkfe_k=x_k-x_k^f, this proves the theorem. ∎ C.3 Interpretation of the Bound The bound |ek|≤ρk−T|eT|+δ¯1−ρ|e_k|≤ρ^k-T|e_T|+ δ1-ρ (55) contains two terms. The first term, ρk−T|eT|ρ^k-T|e_T|, is a transient error that decays geometrically. The second term, δ¯/(1−ρ) δ/(1-ρ), is a persistent tracking-error bound controlled by the maximum equilibrium drift. Therefore, the log-space β dynamics remains close to the instantaneous equilibrium trajectory as long as the equilibrium drift is bounded and the trajectory stays within the local contraction neighborhood. Appendix D A Sufficient Condition for Bounded Equilibrium Drift The bounded tracking theorem assumes that the moving equilibrium satisfies |xk+1f−xkf|≤δ¯.|x_k+1^f-x_k^f|≤ δ. We show that this condition follows from a simple regularity condition on the margin statistic. Lemma 2 (Sufficient condition for bounded equilibrium drift). Assume that there exist constants T∈ℕT , mmin>0m_ >0, and ε≥0 ≥ 0 such that for all k≥Tk≥ T, mk≥mmin,|mk+1−mk|≤ε.m_k≥ m_ , |m_k+1-m_k|≤ . Then |xk+1f−xkf|≤εmmin,∀k≥T.|x_k+1^f-x_k^f|≤ m_ , ∀ k≥ T. Proof. Using xkf=log(z∗/mk)=logz∗−logmkx_k^f= (z^*/m_k)= z^*- m_k, we have |xk+1f−xkf|=|logmk+1−logmk|.|x_k+1^f-x_k^f|=| m_k+1- m_k|. By the mean value theorem, for some ζk _k between mkm_k and mk+1m_k+1, |logmk+1−logmk|=1ζk|mk+1−mk|.| m_k+1- m_k|= 1 _k|m_k+1-m_k|. Since mk,mk+1≥mmin>0m_k,m_k+1≥ m_ >0, we have ζk≥mmin _k≥ m_ . Therefore, |xk+1f−xkf|≤1mmin|mk+1−mk|≤εmmin.|x_k+1^f-x_k^f|≤ 1m_ |m_k+1-m_k|≤ m_ . ∎ This lemma provides an explicit bound on the equilibrium drift whenever the margin statistic stays bounded away from zero and changes gradually across training steps. In particular, the drift condition in Theorem 1 is satisfied whenever ε/mmin≤(1−ρ)r /m_ ≤(1-ρ)r. Appendix E Implementation Details E.1 Models and Training Datasets UltraFeedback experiments. For the UltraFeedback experiments, we follow the SimPO (Meng et al., 2024) training setup. We first perform supervised fine-tuning of the Meta-Llama-3-8B22 2 https://huggingface.co/meta-llama/Meta-Llama-3-8B; Llama 3 Community License. and Qwen3-8B-Base33 3 https://huggingface.co/Qwen/Qwen3-8B-Base; Apache 2.0 License. models on UltraChat-200k44 4 https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k; MIT License. (Ding et al., 2023); the resulting SFT checkpoints are used as the policy initializations and reference models for preference optimization on the UltraFeedback binarized55 5 https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized; MIT License. (Cui et al., 2023) dataset. We also follow the SimPO protocol for common training settings, including the learning rate, scheduler, and optimizer, and keep them fixed across experiments. Table 6 reports these fixed training settings and the LSC-DPO control parameters, with bold entries denoting the main settings used in Table 1. Table 6: Training configuration of LSC-DPO on UltraFeedback. Configuration Llama3-8B Qwen3-8B Model meta-llama/Meta-Llama-3-8B Qwen/Qwen3-8B-Base Optimizer AdamW AdamW Epoch 1 1 Batch size 128 128 Learning rate 5e-7 5e-7 Scheduler cosine cosine Warmup ratio 0.1 0.1 Weight decay 0 0 Precision BF16 BF16 β0 _0 0.01 0.01 η 0.1 0.1 z∗z^* 0.20,0.30,0.35,0.40,\0.20,0.30,0.35,0.40, 0.45,0.50,0.60,0.700.45,0.50,0.60,0.70\ 0.20,0.30,0.35,0.40,\0.20,0.30,0.35,0.40, 0.45,0.50,0.60,0.700.45,0.50,0.60,0.70\ τstart _start 0.40,0.43,0.45\0.40,0.43,0.45\ 0.40,0.43,0.45\0.40,0.43,0.45\ Device 4×H200 GPUs 4×H200 GPUs Anthropic-H experiments. For the Anthropic-H (Bai et al., 2022)66 6 https://huggingface.co/datasets/Anthropic/h-rlhf; MIT License. experiments, we use both the harmless-base and helpful-base subsets. We follow DPO (Rafailov et al., 2023) and β-DPO (Wu et al., 2024b) for the batch size (64) and single epoch, ε -DPO (Lee et al., 2025) for the optimizer, scheduler and warmup, and keep the learning rate of 5×10−75× 10^-7 shared with our UltraFeedback runs. For method-specific hyperparameters of β-DPO and ε -DPO, we use the recommended values reported in the respective papers. Tables 7 and 8 summarize the SFT-stage and preference-optimization-stage configurations, respectively. Table 7: SFT training configuration for Anthropic-H. Configuration Llama3-8B Qwen3-8B Model meta-llama/Meta-Llama-3-8B Qwen/Qwen3-8B-Base Epochs 1 1 Batch size 64 64 Scheduler cosine cosine Warmup ratio 0.1 0.1 Weight decay 0 0 Learning rate 2e-5 2e-5 Precision BF16 BF16 Device 4×H200 GPUs 4×H200 GPUs Table 8: Training configurations of LSC-DPO on Anthropic-H. Configuration Llama3-8B Qwen3-8B Model meta-llama/Meta-Llama-3-8B Qwen/Qwen3-8B-Base Epochs 1 1 Batch size 64 64 Scheduler cosine cosine Warmup ratio 0.1 0.1 Weight decay 0 0 Learning rate 5e-7 5e-7 Precision BF16 BF16 β0 _0 0.1 0.1 τstart _start 0.40,0.43,0.45,0.48,0.50\0.40,0.43,0.45,0.48,0.50\ 0.40,0.43,0.45,0.48,0.50\0.40,0.43,0.45,0.48,0.50\ z∗z^* 0.35,0.40,0.45,0.60,0.80,0.85\0.35,0.40,0.45,0.60,0.80,0.85\ 0.35,0.40,0.45,0.60,0.80,0.85\0.35,0.40,0.45,0.60,0.80,0.85\ η 0.01, 0.05, 0.1, 0.3, 0.5, 1, 5, 8 0.01, 0.05, 0.1, 0.3, 0.5, 1, 5, 8 Device 4×H200 GPUs 4×H200 GPUs Computation environment. All training experiments in this paper are based on the Hugging Face alignment-handbook77 7 https://github.com/huggingface/alignment-handbook; Apache 2.0 License. and TRL88 8 https://github.com/huggingface/trl; Apache 2.0 License.. In our setup with 4 H200 GPUs, each LSC-DPO training run on UltraFeedback completes within two hours, while each run on the Anthropic-H helpful-base or harmless-base subset completes within one hour. E.2 Evaluation Benchmarks and Baselines UltraFeedback evaluation. For the UltraFeedback experiments, we follow the SimPO open-alignment evaluation protocol (Meng et al., 2024) and the official benchmark settings of AlpacaEval 2 (Dubois et al., 2024) and MT-Bench (Zheng et al., 2023b). We compare LSC-DPO with SFT, DPO (Rafailov et al., 2023), β-DPO (Wu et al., 2024b), ε -DPO (Lee et al., 2025), R-DPO (Park et al., 2024), IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), CPO (Xu et al., 2024), ORPO (Hong et al., 2024), SLiC-HF (Zhao et al., 2023), and SimPO (Meng et al., 2024). All methods are evaluated with the same generation settings and evaluation pipeline to ensure a fair comparison. On AlpacaEval 2, we report both the raw win rate and the length-controlled win rate, with the length-controlled metric used as the primary metric because preference optimization can increase response length. On MT-Bench, we report the average score over both turns. Anthropic-H evaluation. For the Anthropic-H experiments, we follow the evaluation protocol used in DPO (Rafailov et al., 2023) and β-DPO (Wu et al., 2024b). We evaluate harmless-base and helpful-base separately by comparing each model response with the H chosen response, using gpt-4-0613 as the judge with randomized response order. We report the win rate of the model response. All methods use the same decoding configuration, with temperature 0.70.7 and top-p sampling (p=0.9p=0.9), to ensure a fair comparison. Control-parameter selection. We selected the LSC-DPO control parameters using a separate Anthropic-H development subset before the main AlpacaEval 2 evaluation. This preliminary study identified a coarse candidate region around z∗∈[0.35,0.45]z^*∈[0.35,0.45] and τstart∈[0.40,0.45] _start∈[0.40,0.45]. The AlpacaEval 2 results in Appendix J.1 are therefore reported as a target-sensitivity analysis rather than as an unrestricted benchmark-specific hyperparameter search. After the target and training settings were fixed, the same trained checkpoints were evaluated on MT-Bench without additional tuning. Arena-Hard reproducibility considerations. We also considered Arena-Hard (Li et al., 2024) as an additional open-ended benchmark, but do not include it in the main evaluation because prior Arena-Hard v0.1 results for SimPO (Meng et al., 2024) and ε -DPO (Lee et al., 2025) used gpt-4-1106-preview as the judge. OpenAI has since retired this model, so the original protocol cannot be reproduced in our evaluation environment. To examine how the released checkpoints behave under currently available judging setups, we re-evaluated SimPO’s released Llama3-8B-base preference-optimization checkpoints under two settings: Arena-Hard v0.1 judged with gpt-4.1, and Arena-Hard v2.0 hard-prompt judged with gpt-4.1. Table 9 shows that all evaluated preference-optimization methods score below the SFT baseline under these updated settings, in contrast to their behavior on AlpacaEval 2 and MT-Bench. The relative ordering also differs substantially from prior Arena-Hard reports, suggesting that results obtained under the updated judging setups may not be directly comparable to those reported under the original protocol. We therefore use AlpacaEval 2 and MT-Bench as the main evaluations. Table 9: Re-evaluation of the released Llama3-8B-base preference-optimization checkpoints from the SimPO study on Arena-Hard. Panel (a) reports Arena-Hard v0.1 results using gpt-4.1; Panel (b) reports Arena-Hard v2.0 hard-prompt results using gpt-4.1. The original v0.1 judge, gpt-4-1106-preview, is no longer available in our evaluation environment. Model Score (%) 95% CI (%) gpt-4-0314 (ref.) 50.0 (−0.0/+0.0)(-0.0/+0.0) SFT 26.4 (−1.9/+1.7)(-1.9/+1.7) SFT + CPO 22.4 (−1.3/+1.6)(-1.3/+1.6) SFT + SimPO 20.1 (−1.7/+2.1)(-1.7/+2.1) SFT + DPO 16.3 (−1.4/+1.6)(-1.4/+1.6) SFT + R-DPO 14.5 (−1.3/+1.3)(-1.3/+1.3) SFT + IPO 13.4 (−1.3/+1.5)(-1.3/+1.5) SFT + ORPO 12.4 (−1.2/+1.2)(-1.2/+1.2) SFT + KTO 10.3 (−1.1/+1.1)(-1.1/+1.1) SFT + SLiC-HF 7.7 (−0.9/+0.8)(-0.9/+0.8) (a) Arena-Hard v0.1, judge: gpt-4.1. Model Score (%) 95% CI (%) o3-mini (ref.) 50.0 (−0.0/+0.0)(-0.0/+0.0) SFT 9.2 (−0.8/+0.8)(-0.8/+0.8) SFT + CPO 7.4 (−0.7/+0.9)(-0.7/+0.9) SFT + DPO 2.3 (−0.4/+0.4)(-0.4/+0.4) SFT + SimPO 1.8 (−0.4/+0.4)(-0.4/+0.4) SFT + IPO 1.4 (−0.3/+0.4)(-0.3/+0.4) SFT + R-DPO 1.4 (−0.3/+0.3)(-0.3/+0.3) SFT + ORPO 1.1 (−0.4/+0.4)(-0.4/+0.4) SFT + KTO 0.7 (−0.2/+0.3)(-0.2/+0.3) SFT + SLiC-HF 0.5 (−0.2/+0.2)(-0.2/+0.2) (b) Arena-Hard v2.0 hard-prompt, judge: gpt-4.1. Appendix F Additional Learning-Signal Trajectories on Anthropic-H This appendix provides additional learning-signal trajectories for the Anthropic-H analysis in Section 4.2. We first report the raw per-step trajectories corresponding to the EMA-smoothed Llama3-8B results, and then provide the corresponding Qwen3-8B results as a cross-model comparison. F.1 Raw Learning-Signal Trajectories on Llama3-8B Figures 7(a) and 7(b) show the raw per-step trajectories together with their EMA-smoothed trends for Llama3-8B. Compared with the smoothed curves reported in Section 4.2, the raw trajectories make the short-term fluctuations in the learning signal more visible, especially for β-DPO. This behavior suggests that dynamically adjusting β based on margin statistics can introduce large short-term fluctuations in the learning signal. In contrast, LSC-DPO maintains a more stable smoothed trajectory while avoiding the persistent low-signal regime observed under DPO, particularly on the helpful-base subset. (a) harmless-base (b) helpful-base Figure 7: Raw and EMA-smoothed trajectories of the learning signal σ(−z)σ(-z) on Anthropic-H with Llama3-8B. Faint curves show raw per-step values, while bold curves show EMA-smoothed trends. F.2 Learning-Signal Trajectories on Qwen3-8B Figures 8 and 9 report the corresponding learning-signal trajectories for Qwen3-8B. The same qualitative pattern observed with Llama3-8B also appears for Qwen3-8B. DPO drives the learning signal toward a lower regime as training proceeds, while β-DPO exhibits larger fluctuations. ε -DPO partially recovers the signal in later stages. Across both harmless-base and helpful-base, LSC-DPO maintains a higher and more stable learning signal, providing additional evidence that the proposed control rule is not specific to a single model family. (a) harmless-base (b) helpful-base Figure 8: Raw and EMA-smoothed trajectories of the learning signal σ(−z)σ(-z) on Anthropic-H with Qwen3-8B. Faint curves denote raw per-step values, while bold curves denote EMA-smoothed trends. (a) harmless-base (b) helpful-base Figure 9: EMA-smoothed trajectories of the learning signal σ(−z)σ(-z) on Anthropic-H with Qwen3-8B. Appendix G Stability under Different Update Rates We further examine the stability behavior of LSC-DPO by varying the control rate η while fixing the target scaled margin to z∗=0.4z^*=0.4. Under this setting, the target learning signal is σ(−z∗)≈0.4013σ(-z^*)≈ 0.4013, and changing η directly changes the stability-relevant product ηz∗η z^* in Proposition 1. Figures 11 and 11 show the EMA-smoothed trajectories of the learning signal σ(−z)σ(-z) on Anthropic-H subsets. Very small values of η react slowly to deviations from the target signal. For example, when η=0.01η=0.01, the signal often drops below the target level and recovers only gradually. In contrast, moderate values such as η=0.3η=0.3, 0.50.5, and 1.01.0 provide more responsive correction and keep the signal close to the target after the initial adjustment phase across both model families and subsets. With z∗=0.4z^*=0.4, the local stability condition 0<ηz∗<20<η z^*<2 reduces to η<5η<5. At η=5η=5, which lies exactly on the boundary with |1−ηz∗|=1|1-η z^*|=1, the local contraction is lost and the learning-signal trajectory exhibits substantially larger fluctuations. In contrast, η=8η=8 violates the stability condition and fails to reliably track the target signal. These results support the feedback-control interpretation of LSC-DPO: appropriately bounded control rates maintain the learning signal near the target regime, whereas overly large rates can destabilize the signal dynamics. (a) harmless-base (b) helpful-base Figure 10: Learning signal σ(−z)σ(-z) trajectories of LSC-DPO under different control rates η with Llama3-8B on Anthropic-H subsets. The dashed line marks the target signal σ(−z∗)≈0.4013σ(-z^*)≈ 0.4013. (a) harmless-base (b) helpful-base Figure 11: Learning signal σ(−z)σ(-z) trajectories of LSC-DPO under different control rates η with Qwen3-8B on Anthropic-H subsets. The dashed line marks the target signal σ(−z∗)≈0.4013σ(-z^*)≈ 0.4013. Appendix H Additional Mechanistic Validation of Learning-Signal Control The mechanistic experiments in Section 4.3 use the same Llama3-8B SFT checkpoint and UltraFeedback training configuration as the main experiments. Unless otherwise specified, all training and evaluation settings are kept fixed across the compared methods. H.1 Learning Signal and Realized Optimization Dynamics We compute the curvature factor as ct=σ(−zt)(1−σ(−zt))c_t=σ(-z_t)(1-σ(-z_t)) and report the gradient norm measured before gradient clipping. LSC-DPO control is activated at step 84. For visualization, solid curves show EMA-smoothed trajectories, whereas all phase-averaged statistics in Table 2 are computed from the original unsmoothed measurements. Figure 12: Optimization diagnostics for DPO and LSC-DPO throughout training. Left: curvature factor. Right: pre-clipping gradient norm. The vertical dashed line marks LSC-DPO control activation at step 84. Before control activation, DPO and LSC-DPO follow nearly identical trajectories. After activation, LSC-DPO consistently maintains a larger curvature factor, whereas DPO continues moving toward a lower-curvature regime. The pre-clipping gradient trajectories also separate after control activation. These full trajectories confirm that the phase-averaged differences reported in Table 2 persist throughout the controlled stage rather than arising from a small number of training steps. H.2 Feedback versus Predetermined β Schedules For the predetermined-schedule baselines in Table 3, we vary β according to linear, cosine, and exponential schedules fixed before training. Let t=0,…,T−1t=0,…,T-1 index the β updates and ft=t/(T−1)f_t=t/(T-1). All schedules start from β0 _0 and end at βend _end, and are defined as βtlin _t^lin =β0+ft(βend−β0), = _0+f_t( _end- _0), (56) βtcos _t^cos =β0+12(1−cos(πft))(βend−β0), = _0+ 12 (1- (π f_t) )( _end- _0), (57) βtexp _t^exp =exp(logβ0+ft(logβend−logβ0)). = ( _0+f_t( _end- _0) ). (58) All predetermined schedules use the same initial and final coefficient values as LSC-DPO, differ only in how β evolves between these endpoints, and update the coefficient once per gathered microbatch. Appendix I Additional Results on Initialization Robustness and Signal Compensation The experiments in Section 4.4 use the same Llama3-8B, UltraFeedback, training, and evaluation settings as those in Section 4.3; only β0 _0 and, where applicable, the compensation target are varied. I.1 Initialization Sensitivity: Full LC/WR Results Table 10 reports the complete AlpacaEval 2 results for the initialization-sensitivity study, including both length-controlled (LC) win rate and standard win rate (WR). The β0=0.01 _0=0.01 configuration corresponds to the reference setting used in the main experiments. Table 10: Full initialization-sensitivity results on AlpacaEval 2. We report length-controlled win rate (LC) and standard win rate (WR), both in percentage points. † denotes the reference setting. β0=0.01† _0=0.01 β0=0.05 _0=0.05 β0=0.10 _0=0.10 β0=0.30 _0=0.30 β0=0.80 _0=0.80 Method LC WR LC WR LC WR LC WR LC WR DPO 19.2 17.7 10.4 7.7 8.7 5.8 8.3 4.8 6.6 3.9 β-DPO 16.6 16.5 10.5 7.8 10.8 7.8 7.3 4.7 6.9 3.8 ε -DPO 19.6 17.6 15.6 13.2 12.9 9.8 8.8 5.9 6.9 4.5 LSC-DPO 22.7 21.1 19.6 18.3 19.1 17.2 15.3 14.1 15.2 13.4 I.2 Additional Signal-Budget Compensation Results We provide additional results for the signal-budget compensation experiments described in Section 4.4. The β0=0.01 _0=0.01 run is used as the anchor configuration, and the baseline LSC-DPO target is z∗=0.400z^*=0.400. For each non-anchor initialization, the full-compensation target is determined by the signal-budget rule in Eq. 20. For β0=0.10 _0=0.10 and β0=0.80 _0=0.80, we additionally evaluate a half-compensation setting, obtained by applying half of the target shift from the baseline z∗=0.400z^*=0.400 toward the corresponding full-compensation target. All other training and evaluation settings are kept unchanged. Table 11: Detailed signal-budget compensation results on AlpacaEval 2. LC and WR denote length-controlled and standard win rates, respectively. Setting z∗z^* LC (%) WR (%) β0=0.05 _0=0.05 0.359 21.5 20.1 β0=0.10 _0=0.10 (half) 0.374 20.7 18.9 β0=0.10 _0=0.10 (full) 0.348 21.1 19.5 β0=0.30 _0=0.30 0.333 20.4 19.4 β0=0.80 _0=0.80 (half) 0.362 17.4 15.8 β0=0.80 _0=0.80 (full) 0.324 19.7 18.3 Compared with the corresponding uncompensated runs reported in Table 10, full compensation improves both LC and WR across all tested non-anchor initializations. For the two initializations where both half and full compensation are evaluated, half compensation yields intermediate performance, while full compensation produces larger improvements. This pattern is consistent with the intended effect of progressively correcting the initialization-induced learning-signal deficit. To directly examine whether the target adjustment produces the intended signal-level correction, we compare the cumulative learning-signal recovery under half and full compensation. The required recovery is defined relative to the anchor run and represents the cumulative signal deficit that must be recovered to match its signal budget. Figure 13: Cumulative learning-signal recovery under half and full signal-budget compensation for β0=0.10 _0=0.10 and β0=0.80 _0=0.80. The dashed curves denote the recovery required to match the anchor run (β0=0.01 _0=0.01), while the solid curves show the recovery achieved by half and full compensation. Full compensation closely matches or slightly exceeds the required recovery, whereas half compensation recovers approximately half of the deficit. Figure 13 shows that the realized cumulative recovery follows the intended compensation strength. Full compensation approaches the required cumulative recovery by the end of training, whereas half compensation produces a substantially smaller correction. These results verify that the target adjustment acts on the cumulative learning signal in the intended direction. Matching the cumulative signal budget does not imply identical optimization trajectories or policies, but provides a direct mechanism-level check of the proposed compensation rule. Appendix J Ablation Analysis of Parameters J.1 Target Learning Signal Table 12 analyzes the target learning signal in LSC-DPO on AlpacaEval 2. In our update rule, the target scaled margin z∗z^* specifies the target signal level σ(−z∗)σ(-z^*) and therefore determines the learning-signal regime that the method attempts to track. We also include DPO as a reference point; its reported signal is the final stabilized value observed during training rather than a controlled target. The results indicate that stronger performance is achieved in an intermediate target-signal region rather than at either extreme. When z∗z^* is too small, the target signal remains high and the loss stays in a relatively high-sensitivity, high-curvature region. The empirical results show that maintaining an overly high target signal does not yield the best performance. Conversely, when z∗z^* is too large, the target signal becomes small and the sigmoid loss moves toward a flatter, lower-sensitivity region. For Llama3-8B, increasing z∗z^* from 0.400.40 to 0.700.70 reduces the LC win rate from 22.722.7 to 19.519.5; for Qwen3-8B, increasing z∗z^* from 0.350.35 to 0.700.70 reduces the LC win rate from 22.122.1 to 13.713.7. The best results are obtained at moderate target signal levels. Llama3-8B performs best at z∗=0.40z^*=0.40, corresponding to σ(−z∗)=0.4013σ(-z^*)=0.4013, while Qwen3-8B performs best at z∗=0.35z^*=0.35, corresponding to σ(−z∗)=0.4134σ(-z^*)=0.4134. Compared with DPO, the best LSC-DPO settings improve the LC win rate from 19.219.2 to 22.722.7 on Llama3-8B and from 17.217.2 to 22.122.1 on Qwen3-8B. Overall, these results suggest that LSC-DPO benefits from an intermediate target learning signal: in our experiments, the strongest settings concentrate around σ(−z∗)≈0.40σ(-z^*)≈ 0.40–0.410.41, where the loss retains meaningful local responsiveness without staying too close to the overly high-signal region or drifting into saturation. Table 12: Target learning signal on AlpacaEval 2. The DPO row reports the final stabilized signal under fixed-β training. Scores are official AlpacaEval 2 point estimates; standard errors for WR/LC are typically 1.01.0–1.21.2 percentage points in our runs. Model z∗z^* / Method Target signal σ(−z∗)σ(-z^*) WR (%) LC (%) Llama3-8B 0.20 0.4502 3.8 4.3 0.30 0.4256 12.8 14.7 0.35 0.4134 20.1 20.9 0.40 0.4013 21.1 22.7 0.45 0.3894 19.7 21.1 0.50 0.3775 18.6 20.7 0.60 0.3543 18.7 20.2 0.70 0.3318 17.5 19.5 DPO 0.3429 17.7 19.2 Qwen3-8B 0.20 0.4502 12.9 18.6 0.30 0.4256 15.2 21.1 0.35 0.4134 16.1 22.1 0.40 0.4013 13.5 18.9 0.45 0.3894 12.9 18.6 0.50 0.3775 12.3 17.8 0.60 0.3543 10.9 18.2 0.70 0.3318 8.8 13.7 DPO 0.3787 13.1 17.2 J.2 Ablation of Control Parameters We further study the parameters of LSC-DPO on Anthropic-H using Llama3-8B and Qwen3-8B. Figures 14 and 15 vary the target scaled margin z∗z^*, the control rate η, and the start threshold τstart _start. The target scaled margin z∗z^* specifies the desired learning-signal regime through σ(−z∗)σ(-z^*). Moderate values of z∗z^* generally achieve stronger win rates, whereas overly large values reduce performance, consistent with the view that a small target signal moves the loss toward a lower-sensitivity regime. The control rate η determines the responsiveness of the feedback update. Moderate values perform well, while overly large rates can degrade performance, consistent with the stability dependence on ηz∗η z^* in Proposition 1. The start threshold τstart _start determines when the dynamic update is activated. Compared with z∗z^* and η, performance is less sensitive to τstart _start within the tested range, although moderate activation thresholds tend to perform best. Overall, these ablations suggest that LSC-DPO tolerates moderate variations in its control parameters, while benefiting from target margins and control rates that maintain a responsive learning-signal regime. (a) Target scaled margin z∗z^* (b) Control rate η (c) Start threshold τstart _start Figure 14: Ablation analysis of key control parameters with Llama3-8B on Anthropic-H subsets: (a) target scaled margin z∗z^*, (b) control rate η, and (c) start threshold τstart _start. The star marks the default setting used in the main experiments. (a) Target scaled margin z∗z^* (b) Control rate η (c) Start threshold τstart _start Figure 15: Ablation analysis of key control parameters with Qwen3-8B on Anthropic-H subsets: (a) target scaled margin z∗z^*, (b) control rate η, and (c) start threshold τstart _start. The star marks the default setting used in the main experiments. Appendix K Downstream Evaluation Following the downstream evaluation tasks used in SimPO (Meng et al., 2024) and ε -DPO (Lee et al., 2025), we further evaluate whether preference optimization preserves general task capabilities beyond open-ended chatbot benchmarks. We consider MMLU (Hendrycks et al., 2020), ARC-Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), TruthfulQA (Lin et al., 2022), WinoGrande (Sakaguchi et al., 2021), and GSM8K (Cobbe et al., 2021), covering knowledge, reading comprehension, commonsense reasoning, truthfulness, and mathematical reasoning. Tables 13 and 14 report the results for Llama3-8B and Qwen3-8B, respectively. Overall, LSC-DPO maintains downstream task performance at a level comparable to strong preference-optimization baselines and does not exhibit a consistent degradation relative to SFT. The effect of preference optimization varies across tasks and model families, consistent with prior observations that downstream task changes can depend on the pretrained model, preference data, and optimization objective. Notably, on GSM8K, LSC-DPO does not show a score drop relative to the SFT initialization on either model family, suggesting that the alignment improvements do not come from a substantial loss of mathematical reasoning capability. As an additional reproducibility check, we also evaluate the released SimPO checkpoints on the same set of downstream tasks under the same evaluation protocol and obtain highly similar results. This consistency strengthens the reliability of our capability-preservation conclusion. We therefore interpret these downstream results mainly as a capability-preservation check rather than as the primary evidence for alignment improvement. Table 13: Downstream capability evaluation on Llama3-8B. Scores are reported as percentages. Avg. denotes the mean over the displayed task scores. Method MMLU ARC-C HellaSwag TruthfulQA WinoGrande GSM8K Avg. SFT 62.54 60.24 81.30 45.50 74.43 49.20 62.20 DPO 63.15 64.51 83.14 53.12 75.14 54.97 65.67 β-DPO 62.92 63.99 83.04 51.37 75.14 53.45 64.99 ε -DPO 62.98 64.33 83.29 52.94 75.14 53.15 65.30 IPO 62.96 62.63 81.68 49.02 73.88 54.36 64.09 CPO 62.73 60.32 81.44 52.97 75.06 48.22 63.46 KTO 62.88 64.51 83.39 53.93 75.85 53.37 65.65 ORPO 62.80 55.29 80.70 47.72 74.03 44.35 60.82 SLiC-HF 62.69 57.85 80.57 50.82 74.35 46.47 62.13 R-DPO 62.41 61.95 80.91 43.74 74.35 39.58 60.49 SimPO 62.85 65.78 83.12 58.48 74.25 52.54 66.17 LSC-DPO 63.02 65.25 83.34 52.57 74.82 53.98 65.50 Table 14: Downstream capability evaluation on Qwen3-8B. Scores are reported as percentages. Avg. denotes the mean over the displayed task scores. Method MMLU ARC-C HellaSwag TruthfulQA WinoGrande GSM8K Avg. SFT 75.99 67.49 80.17 53.87 75.85 82.18 72.59 DPO 76.54 68.69 81.30 57.39 76.01 84.46 74.06 β-DPO 76.48 68.77 81.13 56.40 76.72 83.40 73.82 ε -DPO 76.63 68.34 81.44 57.57 76.01 84.53 74.09 IPO 76.48 67.83 80.92 56.30 76.09 84.15 73.63 CPO 75.63 68.09 80.03 56.72 75.61 83.55 73.27 KTO 76.41 68.77 81.12 57.37 76.09 84.99 74.12 ORPO 75.84 66.38 79.66 53.83 75.22 67.48 69.73 SLiC-HF 75.01 67.06 79.33 55.03 74.51 79.98 71.82 R-DPO 75.92 68.00 80.60 54.07 75.93 84.23 73.13 SimPO 76.37 68.69 81.24 56.73 76.80 84.84 74.11 LSC-DPO 76.50 68.52 81.38 57.50 76.01 84.53 74.07