Paper deep dive
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation
Dongfang Zhao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/12/2026, 2:00:43 AM
Summary
The paper introduces LOCUS, a task-aware low-rank post-training method designed to reduce output token costs in large language models without compromising utility. By confining parameter updates to low-rank adaptation subspaces (LoRA) while keeping the backbone frozen and the native preference objective intact, LOCUS minimizes generation length subject to a utility constraint. Experiments on Pythia-2.8B and Qwen2.5-3B models demonstrate significant reductions in continuation length (up to 39.84%) with minimal parameter updates (0.24-0.28%), outperforming or matching full-parameter DPO, DrDPO, and SamPO baselines in token efficiency.
Entities (8)
Relation Signals (8)
Pythia-2.8b → backbonefor → LOCUS
confidence 95% · evaluate two ~3B decoder backbones, Pythia-2.8B and Qwen2.5-3B... LOCUS reduces continuation length... on Pythia-2.8B
Qwen2.5 3B → backbonefor → LOCUS
confidence 95% · evaluate two ~3B decoder backbones, Pythia-2.8B and Qwen2.5-3B... LOCUS reduces continuation length... on Qwen2.5-3B
LOCUS → evaluatedon → Anthropic HH-RLHF
confidence 95% · Across Anthropic HH-RLHF dialogue preferences, we evaluate... LOCUS
LOCUS → reduces → output-token cost
confidence 95% · LOCUS reduces continuation length by up to 39.84% on Pythia-2.8B... while updating only 0.24–0.28% of model parameters
LOCUS → uses → LoRA
confidence 95% · LOCUS selects a task-aware low-rank adaptation subspace... Low-rank adaptation (LoRA) confines trainable updates to factorized matrices
LOCUS → comparedwith → SamPO
confidence 90% · evaluate... against... the released SamPO checkpoint
LOCUS → comparedwith → DrDPO
confidence 90% · evaluate... against protocol-matched full-parameter DPO and DrDPO branches
LOCUS → comparedwith → DPO
confidence 90% · evaluate... against protocol-matched full-parameter DPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model serving costs scale directly with output sequence length, yet standard preference alignment often inflates response verbosity without improving utility. We study whether the parameterization of post-training updates affects generation length: low-rank subspaces alter sequence length without modifying the alignment loss. We present LOCUS, a method that selects a task-aware low-rank adaptation subspace to minimize output-token cost subject to a utility constraint. Within this subspace, post-training retains the native preference objective with a frozen backbone. Across Anthropic HH-RLHF dialogue preferences, we evaluate two $\sim$3B decoder backbones, Pythia-2.8B and Qwen2.5-3B, against protocol-matched full-parameter DPO and DrDPO branches and the released SamPO checkpoint. LOCUS reduces continuation length by up to 39.84\% on Pythia-2.8B and by 14.87--17.58\% on Qwen2.5-3B while updating only 0.24--0.28\% of model parameters, with no material change in the internal preference diagnostic.
Tags
Links
- Source: https://arxiv.org/abs/2609.11739v1
- Canonical: https://arxiv.org/abs/2609.11739v1
Trouble viewing inline? Open PDF directly →
Full Text
53,245 characters extracted from source content.
Expand or collapse full text
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation Dongfang Zhao (dzhao@uw.edu) Abstract Large language model serving costs scale directly with output sequence length, yet standard preference alignment often inflates response verbosity without improving utility. We study whether the parameterization of post-training updates affects generation length: low-rank subspaces alter sequence length without modifying the alignment loss. We present LOCUS, a method that selects a task-aware low-rank adaptation subspace to minimize output-token cost subject to a utility constraint. Within this subspace, post-training retains the native preference objective with a frozen backbone. Across Anthropic H-RLHF dialogue preferences, we evaluate two ∼ 3B decoder backbones, Pythia-2.8B and Qwen2.5-3B, against protocol-matched full-parameter DPO and DrDPO branches and the released SamPO checkpoint. LOCUS reduces continuation length by up to 39.84% on Pythia-2.8B and by 14.87–17.58% on Qwen2.5-3B while updating only 0.24–0.28% of model parameters, with no material change in the internal preference diagnostic. 1 Introduction Serving large language models requires substantial computational and memory resources, with per-request latency and serving expenses governed by the number of autoregressively generated output tokens (Pope et al., 2023; Kwon et al., 2023). In production environments, each additional decoded token requires an iterative forward pass through the entire model depth, consuming memory bandwidth and expanding key-value cache storage (Leviathan et al., 2023; Dao et al., 2022). Consequently, generating overly verbose responses directly diminishes operational throughput and escalates deployment costs. Developing techniques that produce concise generations while maintaining strict answer quality represents a central objective for practical language model serving. Post-training preference alignment methods align models with human values (Ouyang et al., 2022; Rafailov et al., 2023). Algorithms such as Direct Preference Optimization (Rafailov et al., 2023), Distributionally Robust DPO (Wu et al., 2025), and down-sampled divergence methods (Lu et al., 2024) successfully steer model outputs toward preferred behaviors. However, when optimized across all backbone parameters, these objectives frequently suffer from verbosity bias, wherein models learn that longer responses correlate with higher preference scores (Singhal et al., 2024; Saito et al., 2023; Park et al., 2024). These observations motivate examining whether the parameterization of preference updates affects generation length under an unchanged objective. Prior attempts to curtail output length typically introduce ad-hoc regularizers or prompt modifications, yet these interventions exhibit clear deficiencies. Methods that apply heuristic length penalties or negative length rewards risk distorting the underlying preference formulation, leading to premature termination or degraded response quality (Li et al., 2026). Similarly, prompting models for conciseness relies on fragile instruction following that degrades under distribution shifts (Zhou et al., 2023). These approaches modify the learning objective or inference interface, leaving the effect of update parameterization on output length open to investigation. Our insight toward this problem is that the parameterization of post-training updates can affect output-length dynamics: low-rank adaptation subspaces can alter sequence length without modifying the alignment objective. Low-rank adaptation (LoRA) (Hu et al., 2022) confines trainable updates to factorized matrices, making the update subspace a concrete variable for studying generation length. By freezing the pretrained backbone and confining parameter updates to structured low-rank trajectories, the model lands on a materially different quality-token operating point under the exact original preference loss. This paper turns the above insight into a post-training method, namely Length Optimization for Concise Utility-Preserving Sequences (LOCUS). LOCUS selects a task-aware low-rank adaptation subspace on development data to minimize output-token cost subject to a utility constraint. Within each candidate subspace, adaptation uses the native post-training objective with a frozen backbone. LOCUS leaves the underlying loss formulation intact: it adds no length normalization beyond what a native objective already specifies, no explicit token penalty, and no brevity prompting. With the objective fixed, LOCUS treats the low-rank subspace configuration (spanning adapter rank, module placement, layer scope, and training checkpoint step) as a structured design variable. Figure 1 summarizes the primary empirical results of LOCUS across three preference-optimization objective families on the Anthropic Helpful and Harmless (H) benchmark (Bai et al., 2022) using Pythia-2.8B (Biderman et al., 2023). Across the three H comparisons on Pythia-2.8B, LOCUS reduces continuation tokens by 20.73% under DPO, 25.29% under DrDPO, and 39.84% under SamPO with negligible accuracy deltas (≤0.13≤ 0.13 p), updating only 0.28% of parameters. Furthermore, controlled DPO and DrDPO comparisons on Qwen2.5-3B (Yang et al., 2024) show 14.87% and 17.58% token reductions, respectively, under 0.24% trainable parameters. Figure 1: Token and accuracy summary on Pythia-2.8B. In summary, this paper makes the following contributions: • We formulate token-efficient preference post-training as a task-aware low-rank subspace selection problem subject to an explicit utility constraint. The formulation preserves the native alignment objective; Section 3 defines the problem and selection procedure. • We analyze the trainable-parameter count and the exact inference-time equivalence between unmerged and merged low-rank updates. Section 4 gives the propositions, proofs, and parameter accounting. • We evaluate LOCUS on Anthropic H-RLHF dialogue preferences with two ∼ 3B decoder backbones, Pythia-2.8B and Qwen2.5-3B, against protocol-matched full-parameter DPO and DrDPO branches and the released SamPO checkpoint. The protocol-matched branches yield 25.29% and 17.58% DrDPO reductions on Pythia-2.8B and Qwen2.5-3B, and continued SamPO adaptation yields 39.84%, using a small trainable subspace; the complete protocol and results appear in Section 5. 2 Related Work Preference Optimization and Alignment. Preference alignment techniques train language models to align with human preferences using pairwise comparison data. Reinforcement Learning from Human Feedback (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022) optimizes reward models using policy gradient techniques. Direct Preference Optimization (Rafailov et al., 2023) eliminates the separate reward modeling stage by deriving a closed-form substitution of the reward function into the policy objective. Subsequent methods extend direct preference optimization through alternative objectives. For example, IPO (Azar et al., 2024) introduces regularized linear losses, whereas KTO (Ethayarajh et al., 2024) models prospect theory. DrDPO (Wu et al., 2025) instead addresses reference distribution shifts. Prior methods optimize objectives in the full parameter space or modify the loss formulation; LOCUS preserves the native preference loss while altering only the optimization subspace. Length Bias and Output Conciseness. Language models aligned through preference optimization frequently develop severe length bias, generating unnecessarily long responses to maximize rewards (Singhal et al., 2024; Saito et al., 2023; Dubois et al., 2023). Several studies investigate this phenomenon: LR-DPO (Park et al., 2024) disentangles length from quality through score adjustments, while SamPO (Lu et al., 2024) down-samples reference sequence length to diminish length reliance. GR3 (Li et al., 2026) applies group relative reward rescaling during reinforcement learning to alleviate length inflation. Prior approaches introduce ad-hoc length regularization terms or heuristic prompting that distort the target preference distribution, whereas LOCUS achieves token efficiency through subspace parameterization without modifying the underlying objective. Parameter-Efficient Fine-Tuning. Parameter-efficient fine-tuning (PEFT) adapts large language models by updating a small subset of parameters while freezing base weights. Prominent methodologies include adapter insertion (Houlsby et al., 2019), prefix tuning (Li and Liang, 2021), prompt tuning (Lester et al., 2021), and low-rank adaptation (Hu et al., 2022). Recent works expand low-rank tuning through quantization (Dettmers et al., 2023), weight decomposition (Liu et al., 2024), and adaptive rank allocation (Zhang et al., 2023). Theoretical investigations establish that language model adaptation possesses low intrinsic dimensionality, indicating that overparameterized updates are redundant for task learning (Aghajanyan et al., 2021; Sharma et al., 2024). Conventional parameter-efficient tuning treats low-rank adaptation solely as a resource-saving approximation to full fine-tuning; in contrast, LOCUS treats the adaptation subspace as a Pareto design variable to control generation length while preserving task utility. 3 Methodology Figure 2 summarizes the LOCUS procedure in three stages. First, LOCUS attaches low-rank adapters to selected modules. The pretrained backbone remains frozen, which confines parameter updates to those adapters. The adapters are then trained with the original preference objective, allowing the experiment to test whether the update parameterization changes generation length without adding a length penalty or changing the loss. Second, LOCUS trains a small set of candidate adapters that differ in rank, target modules, target layers, or training step. It evaluates each candidate on development data and selects the shortest one that stays within the allowed utility difference from the baseline. When a confirmation split is available, the selected adapter must pass that check before the held-out test evaluation. Finally, LOCUS prepares the selected adapter for inference. The adapter can be merged into the backbone for a single deployment, or kept separate when one backbone serves multiple task adapters. The formal notation used in Figure 2, together with the native DPO and DrDPO objectives, is defined in Section 3.1. Figure 2: LOCUS procedure from native-objective training to selected low-rank deployment. Algorithm 1 LOCUS Subspace Selection 1: ℒbase,W0,ttrain,tsel,tconf,ϵt,L_base,W_0,D_t^train,D_t^sel,D_t^conf, _t,C 2: Selected adapter c∗c^* and policy πc∗ _c^* 3: Qtbase←EvalUtility(πbase,tsel)Q_t^base ( _base,D_t^sel) 4: for c∈c do 5: ΔWc←Train(ℒbase,W0,ttrain,c) W_c (L_base,W_0,D_t^train,c) 6: (Tt(c),Qt(c))←Eval(πc,tsel)(T_t(c),Q_t(c)) ( _c,D_t^sel) 7: end for 8: feas←c∈∣Qt(c)≥Qtbase−ϵtC_feas←\c Q_t(c)≥ Q_t^base- _t\ 9: c∗←argminc∈feasTt(c)c^*← _c _feasT_t(c) 10: if tconfD_t^conf exists then 11: (Tconf,Qconf)←Eval(πc∗,tconf)(T_conf,Q_conf) ( _c^*,D_t^conf) 12: Qconfbase←EvalUtility(πbase,tconf)Q_conf^base ( _base,D_t^conf) 13: Tconfbase←EvalLength(πbase,tconf)T_conf^base ( _base,D_t^conf) 14: u←[Qconf≥Qconfbase−ϵt]u←[Q_conf≥ Q_conf^base- _t] 15: ℓ←[Tconf<Tconfbase] ←[T_conf<T_conf^base] 16: if u∧ℓu then 17: return Frozen adapter c∗c^* and model πc∗ _c^* 18: else 19: return Fallback baseline πbase _base 20: end if 21: else 22: return Unconfirmed adapter c∗c^* 23: end if 3.1 Objective-Preserving Adaptation We denote an input prompt by x∈x and an autoregressively generated continuation by y=(y1,y2,…,yT)∈y=(y_1,y_2,…,y_T) . We consider an autoregressive language model parameterized by frozen pretrained backbone weights W0∈ℝd1×d2W_0 ^d_1× d_2. Given a task dataset D with pairwise preference comparisons (x,yw,yl)(x,y_w,y_l), where ywy_w denotes the preferred response and yly_l denotes the dispreferred response, standard alignment methods optimize a base objective ℒbase(θ)L_base(θ) across all model parameters. For example, under Direct Preference Optimization (Rafailov et al., 2023), defining the implicit reward proxy rθ(x,y)=βlog[πθ(y∣x)/πref(y∣x)]r_θ(x,y)=β [ _θ(y x)/ _ref(y x)], the objective is: ℒDPO(θ)=−[logσ(rθ(x,yw)−rθ(x,yl))],L_DPO(θ)=-E_D [ σ (r_θ(x,y_w)-r_θ(x,y_l) ) ], (1) where β>0β>0 controls the divergence penalty from the reference policy πref _ref. The function σ denotes the sigmoid. Under Distributionally Robust DPO (Wu et al., 2025), the objective robustifies against sample-level reference divergence: ℒDrDPO(θ)=−β′log(1B∑i=1Be−ℓi(θ)/β′),L_DrDPO(θ)=-β ( 1B _i=1^Be^- _i(θ)/β ), (2) where ℓi(θ) _i(θ) denotes the individual pairwise loss for sample i in a microbatch of size B. The parameter β′β controls distributional robustness. In LOCUS, we leave the native objective ℒbaseL_base strictly intact. We restrict trainable updates through the LoRA parameterization in Eq. (3): W=W0+ΔW=W0+αrBA,W=W_0+ W=W_0+ αrBA, (3) where A∈ℝr×d2∼(0,σ2)A ^r× d_2 (0,σ^2) and B∈ℝd1×rB ^d_1× r is zero-initialized. The adaptation uses rank r≪min(d1,d2)r (d_1,d_2) with scaling factor α. We define an adaptation subspace configuration as a tuple c=(r,α,,ℒ,s)∈c=(r,α,P,L,s) . The sets ⊆Q,K,V,O,MLPP \Q,K,V,O,MLP\ and ℒ⊆1,…,LL \1,…,L\ specify the targeted weight modules and layer indices, respectively. For these targets, s identifies the training optimization checkpoint step. The resulting selection procedure is summarized in Algorithm 1. It first measures baseline utility on the task’s selection split. Each candidate adapter is then trained with the same native objective and evaluated on that split for continuation length and preference utility. LOCUS discards candidates whose utility falls beyond the allowed tolerance, then selects the shortest remaining candidate. When a confirmation split is available, the selected candidate must also reduce length without exceeding the utility tolerance; otherwise, LOCUS falls back to the baseline. If no confirmation split is available, the selected adapter is returned as unconfirmed. 3.2 Task-Aware Constrained Selection Different subspace configurations c∈c produce distinct behavioral trajectories during autoregressive decoding. Define Tt(c)T_t(c) as the mean continuation token length produced by configuration c under greedy decoding on task t. The corresponding task utility is denoted by Qt(c)Q_t(c). For candidate selection, Qt(c)Q_t(c) is evaluated using held-out chosen-versus-rejected sequence log-probability preference accuracy over (x,yw,yl)∈t(x,y_w,y_l) _t: Qt(c)=1|t|∑(x,yw,yl)(πc(yw∣x)>πc(yl∣x)).Q_t(c)= 1|D_t| _(x,y_w,y_l)I( _c(y_w x)> _c(y_l x)). (4) LOCUS identifies the optimal adaptation configuration ct∗c_t^* by solving a constrained minimization problem: ct∗=argminc∈ c_t^*= _c Tt(c) T_t(c) (5) subject to to Qt(c)≥Qtbase−ϵt, Q_t(c)≥ Q_t^base- _t, where QtbaseQ_t^base is the benchmark utility achieved by the protocol-appropriate baseline on the same task. The tolerance ϵt≥0 _t≥ 0 specifies the allowed utility decrease from that baseline (set to 1.01.0 percentage points in this work). To prevent split overfitting, LOCUS partitions development data into disjoint selection (tselD_t^sel) and confirmation (tconfD_t^conf) splits. Candidate configurations in C are screened on tselD_t^sel via Eq. (5); when available, the top candidate c∗c^* is verified on tconfD_t^conf before being frozen for held-out evaluation. 3.3 Complexity and Serving Modalities The trainable parameter and optimizer-state footprint of LOCUS scales with the low-rank factors. For a targeted module, low-rank adaptation exposes r(d1+d2)r(d_1+d_2) trainable entries; full-parameter adaptation exposes d1d2d_1d_2. Consequently, gradient storage and optimizer states shrink with the trainable set. Under an Adam-style two-state optimizer, the primary adapter requires two state tensors over 7.86M entries for Pythia-2.8B or 7.37M for Qwen2.5-3B. This accounting assumes two optimizer states per trainable entry; actual state storage depends on each experiment’s optimizer. Because the frozen backbone also requires forward computation and activation backpropagation, total memory and training FLOPs require separate assessment. During serving, LOCUS supports two deployment modalities. In single-tenant deployments, low-rank weights are pre-merged via W∗=W0+αrB∗A∗W^*=W_0+ αrB^*A^* prior to execution, incurring zero adapter-induced latency or parameter overhead. In multi-tenant deployments, a single frozen backbone dynamically serves multiple task adapters, though active adapters retain bookkeeping and memory overhead not benchmarked here. 4 Analysis 4.1 Trainable Parameterization Because LOCUS restricts every update to a low-rank subspace, the size of that restriction deserves a precise statement. Full-parameter fine-tuning gives the optimizer one free parameter per entry of a weight matrix, whereas the low-rank parameterization gives it only the entries of the two factors: on a 2560×25602560× 2560 module at r=16r=16, 81,920 trainable entries rather than 6,553,600. The proposition below states the count for a general module, with the remark that follows bounding what the count can be taken to mean. Proposition 1 (Trainable parameter-count reduction). Let W∈ℝd1×d2W ^d_1× d_2 be a targeted linear module and let r be a positive integer. Full-parameter fine-tuning exposes d1d2d_1d_2 trainable scalar entries, whereas the LoRA parameterization W=W0+(α/r)BAW=W_0+(α/r)BA, with fixed W0W_0, fixed α≠0α≠ 0, B∈ℝd1×rB ^d_1× r, and A∈ℝr×d2A ^r× d_2, exposes r(d1+d2)r(d_1+d_2) trainable factor entries. Thus the factor parameterization has fewer exposed trainable entries exactly when r(d1+d2)<d1d2r(d_1+d_2)<d_1d_2. Proof. The complete proof, including the distinction between exposed factor entries and the intrinsic dimension of the represented update set, is given in Appendix A. ∎ Remark 1 (Interpretation of the count). The proposition is an implementation-level parameter-count statement. It does not claim that the low-rank factors select particular semantic or verbosity-specific directions, nor that the count alone determines generation length, preference accuracy, total memory, or training speed. For the selected Pythia-2.8B attention configuration (L=32,d=2560,r=16L=32,d=2560,r=16), the combined adapter accounts for Nattn=Lr((3d+d)+(d+d))=7,864,320N_attn=Lr((3d+d)+(d+d))=7,864,320 trainable parameters (0.2826%0.2826\% of the model). Similarly, for Qwen2.5-3B (L=36,r=16L=36,r=16 on q_proj, k_proj, v_proj, o_proj), the adapter comprises 7,372,8007,372,800 trainable parameters (0.2383%0.2383\% of 3.09B). These calculations explain the parameter counts in Table 3. Translating those counts into end-to-end memory or latency requires measurements of the execution path. 4.2 Inference-Time Merge Equivalence A second property is required for the deployment discussion: after training, LOCUS may either keep an adapter branch or merge its update into the corresponding frozen linear module. The following proposition identifies the exact condition under which these two inference representations compute the same function. Proposition 2 (Exact equivalence after adapter merging). Consider a linear module with input h, frozen weight W0W_0, and a LoRA update (α/r)BA(α/r)BA. At inference time, with adapter dropout disabled, the unmerged computation and the computation using the merged weight W∗=W0+(α/r)BAW^*=W_0+(α/r)BA produce exactly the same module output for every input h. Proof. The complete algebraic proof and its assumptions are given in Appendix A. ∎ Remark 2 (Scope of merge equivalence). The equality concerns the output of a linear module in inference mode. Deployments that produce this same output can have different latency and memory costs because an unmerged adapter adds computation and bookkeeping. Furthermore, finite-precision or quantized implementations may introduce small numerical differences. 5 Evaluation 5.1 Experimental Setup Backbones. We evaluate LOCUS on two decoder-only backbone families at roughly 3B parameter scale: Pythia-2.8B (Biderman et al., 2023), based on GPT-NeoX (Black et al., 2022), and Qwen2.5-3B (Yang et al., 2024), a modern Llama-style architecture with grouped-query attention, SwiGLU (Shazeer, 2020), and RMSNorm (Zhang and Sennrich, 2019). Appendix Table 3 details the architectural differences and target modules for each backbone. Baselines. We compare LOCUS with a protocol-matched full-parameter branch while fixing the training pool, starting checkpoint, native objective, and data splits. Controlled DPO (Rafailov et al., 2023) and DrDPO (Wu et al., 2025) first train a shared full-parameter supervised fine-tuned (SFT) checkpoint, then compare full-parameter preference optimization with low-rank LOCUS from that checkpoint. SamPO (Lu et al., 2024) instead continues from the official released Pythia-2.8B H-RLHF Iterative SamPO checkpoint, while Qwen2.5-3B repeats the shared-SFT comparison under both native objectives. Appendix Table 3 makes every starting point and branch parameterization explicit. Datasets. Primary evaluation uses Anthropic Helpful and Harmless (H-RLHF) (Bai et al., 2022), an open-ended multi-turn dialogue preference benchmark. DPO and DrDPO use 8,552 frozen test pairs, while SamPO uses its established 256-example split. The additional datasets are Anthropic H-RLHF harmless-base for safety refusal and Orca DPO (Mukherjee et al., 2023) for single-turn instruction following. Each uses a separate 256-example development evaluation; neither is presented as an external test result. Appendix Table 3 records these roles. Platform. All training and inference runs use one NVIDIA A100-PCIE-80GB GPU with bfloat16 mixed precision. Activation checkpointing is enabled for the DPO and DrDPO full-parameter runs with 512-token sequences, whereas the SamPO comparison follows the published recipe. The approximately 3B scale makes the full-vs-low-rank protocol reproducible on this hardware. Metrics. We measure continuation length and held-out preference accuracy. Generation uses greedy decoding (do_sample=Falsedo\_sample=False, max_new_tokens=256max\_new\_tokens=256). We count continuation tokens exclusive of the end-of-sequence (EOS) token and record 256-token limit hits. Preference accuracy compares chosen and rejected sequence log-probabilities under the evaluated policy (Eq. (4)). We choose this metric because fixed preference labels and an explicit scoring rule enable reproducible model comparisons. 5.2 Baseline Comparisons Figure 3: Constrained checkpoint selection. Figure 3 reports the measured 250-, 500-, and 750-step selection-dev checkpoints for DPO and DrDPO, with full-parameter baselines at 146.76 and 150.48 tokens. For DPO on Pythia-2.8B, both branches train on 160,544 Anthropic H pairs with β=0.1β=0.1 from the same SFT checkpoint; DPO-FULL updates all weights, whereas DPO-LOCUS uses a low-rank adapter. On the 8,552 frozen test pairs, LOCUS reduces mean continuation length from 137.67 to 109.12 tokens (20.73%) and changes preference accuracy from 48.40% to 48.39% (−0.01-0.01 p), using 7,864,320 trainable parameters (0.2826%). Despite nonmonotonic trajectories, utility-constrained selection chooses step 750 in both comparisons. Figure 4: Continuation length CDF. Figure 4 presents the continuation-length CDF across the 8,552 frozen test prompts. Under Distributionally Robust DPO (DrDPO) on Pythia-2.8B (Wu et al., 2025), DRDPO-FULL updates all weights while DRDPO-LOCUS updates only its adapter from the same SFT checkpoint, using β=0.1β=0.1 and β′=1.0β =1.0. LOCUS reduces mean length from 145.61 to 108.79 tokens (25.29%) and changes preference accuracy from 48.39% to 48.26% (−0.13-0.13 p); the selected step-750 candidate reaches 26.70% reduction on selection-dev. The median falls from 100 to 52 tokens under DPO and from 125 to 52 under DrDPO, while limit hits fall by 24.70% and 30.98%. For SamPO, LOCUS starts from the official Pythia-2.8B Anthropic H Iterative SamPO checkpoint (Lu et al., 2024), which incorporates per-token length normalization into preference optimization. On the 256-example H evaluation split, the released checkpoint produces a mean continuation of 132.77 tokens with 53.52% preference accuracy. We train a low-rank adapter with r=16r=16 and α=32α=32 on attention modules under the native SamPO objective (β=0.05β=0.05). As summarized in Figure 1 and Appendix Table 4, LOCUS reduces mean continuation length from 132.77 to 79.88 tokens (52.89 tokens; 39.84%) while preserving 53.52% preference accuracy (Δ=0.00 =0.00 p). 5.3 Rank Sensitivity Using the Pythia-2.8B backbone, we measure sensitivity to adapter rank r∈4,8,16,32r∈\4,8,16,32\ on a 64-example Anthropic H screening subset under attention adaptation with the SamPO objective (α/r=2α/r=2). Rank controls the dimensionality of the trainable update, so we keep the target modules, objective, data split, and scaling rule fixed while varying only r. This screen uses a separate subset from the candidate pool C evaluated in Section 5, so it characterizes rank sensitivity without redetermining that selection. Figure 5 reports the resulting token changes and internal preference diagnostics. Figure 5: Sensitivity to adapter rank. Because the objective, attention placement, data split, and step budget are fixed, this screen isolates the effect of rank within the candidate adapter subspace. At rank r=4r=4, the model exhibits length inflation (+11.74% tokens), whereas r=8r=8 produces a small reduction of 2.32%. The reduction increases to 15.95% at r=16r=16 and 25.10% at r=32r=32 (Figure 5(a)), so the measured length effect varies materially across the tested ranks. The internal preference diagnostic remains 45.31% for r∈4,8,16r∈\4,8,16\ and rises to 46.88% at r=32r=32 (Figure 5(b)); within this screen, token reduction therefore does not follow a monotonic change in that diagnostic. 5.4 Target-Module Ablation Using Pythia-2.8B, we perform a controlled ablation over which existing linear projections receive LoRA updates on a 64-example Anthropic H screening subset with r=16r=16, keeping the backbone and layer coverage fixed. The five candidate target sets are query-key-value projections only (QKV), attention output projections only (O), their combination (Attention-All), feed-forward network projections only (MLP), and all linear projections (All-Linear). Figure 6: Target-module ablation. Figure 6 compares token reduction with trainable parameter count across the five target sets. QKV and output-only adaptation yield 8.09% and 4.96% reductions, while Attention-All reaches 15.95%. MLP adaptation yields 5.75%; All-Linear reaches 13.88% with 2.7× more trainable parameters than Attention-All (20.97M vs. 7.86M). The internal preference diagnostic remains 45.31% across all placements. 5.5 Cross-Task Generality We test whether the measured token reduction extends beyond the H dialogue setting when the preference data and task semantics change. Using Pythia-2.8B, we evaluate the H comparison alongside Anthropic H-RLHF harmless-base and the Orca DPO instruction-following dataset (Figure 7). These evaluations cover dialogue, safety, and instruction-following settings, each with a task-specific adapter; the H row reports the 8,552-pair frozen-test result, Harmless and Orca DPO each use a 256-example development split, and the preference numbers remain internal chosen-vs-rejected diagnostics rather than external quality evaluations. Figure 7: Cross-task evaluations. Figure 7(a) reports task-specific length changes. The H comparison reduces mean continuation length from 137.67 to 109.12 tokens (20.73%). The Harmless evaluation reduces it from 137.53 to 102.75 tokens (25.29%), while the Orca DPO evaluation reduces it from 175.95 to 35.25 tokens (79.97%). Figure 7(b) reports the corresponding internal preference diagnostics: H changes from 48.40% to 48.39% (−0.01-0.01 p), Harmless changes from 57.03% to 58.20% (+1.17+1.17 p), and Orca DPO changes from 69.14% to 82.81% (+13.67+13.67 p). 5.6 Cross-Backbone Transfer to Qwen2.5-3B We test whether the token-reduction effect transfers across architectural families on Qwen2.5-3B (Yang et al., 2024), a distinct decoder backbone family at comparable ∼ 3B scale. Following the identical experimental discipline, both branches begin from a single shared full-parameter SFT checkpoint trained for one epoch on the 160,288 Anthropic H training pairs. Candidate low-rank adapters (r=16,α=32r=16,α=32 on the attention projections of all 36 layers; 7.37M parameters, 0.2383%) are screened on the 256-pair selection split and confirmed on a disjoint 256-pair split before frozen held-out evaluation. Figure 8: Cross-backbone token and accuracy results. Figure 8 reports controlled frozen test results on the 8,552 Anthropic H test pairs for both DPO and DrDPO alongside the established Pythia-2.8B baselines. Under DPO (β=0.1β=0.1), LOCUS (step 250) reduces mean continuation length from 108.26 to 92.16 tokens (14.87% reduction; median 75 to 51 tokens), while held-out preference accuracy changes from 48.69% to 48.64% (−0.05-0.05 p delta). Under DrDPO (β=0.1,β′=1.0β=0.1,β =1.0), LOCUS (step 500) reduces mean continuation length from 111.58 to 91.97 tokens (17.58% reduction; median 81 to 52 tokens), with preference accuracy changing from 48.64% to 48.55% (−0.09-0.09 p delta). Both comparisons meet the strong positive criterion of at least 2%2\% token reduction with a utility delta of at least −1.0-1.0 p. 6 Conclusion This paper presents LOCUS, which freezes the pretrained backbone and selects a low-rank adaptation subspace while preserving the native preference objective. On Anthropic H-RLHF, we compare LOCUS with protocol-matched full-parameter DPO and DrDPO branches and with the released SamPO checkpoint on Pythia-2.8B, then repeat the comparisons on Qwen2.5-3B. Each comparison matches the objective, checkpoint, data split, and decoding protocol, measuring continuation length and internal chosen-vs-rejected preference accuracy. The protocol-matched DrDPO branch yields 25.29% and 17.58% reductions on Pythia-2.8B and Qwen2.5-3B, and continued SamPO adaptation yields 39.84%, all while updating fewer than 0.3% of model parameters. Additional development evaluations extend the measured reductions to safety and instruction-following data under the same internal preference diagnostic. Limitations LOCUS selects a low-rank adaptation subspace for post-training, with the following limitations. The evaluation covers two decoder-only backbone families at roughly 3B scale, Pythia-2.8B and Qwen2.5-3B, under greedy decoding. Larger models and sampling-based inference remain untested. The candidate pool C uses four discrete ranks and coarse projection groupings. Continuous rank allocation, finer layer selection, and gradient-informed pruning remain outside the study. The cross-task evaluations broaden coverage across dialogue, safety, and instruction following, but do not predict the reduction available on an unseen task. References Aghajanyan et al. (2021) A. Aghajanyan, S. Gupta, and L. Zettlemoyer Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, p. 7319–7328. External Links: Link Cited by: §2. Azar et al. (2024) M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello A general theoretical paradigm to understand learning from human preferences. In Proceedings of the International Conference on Artificial Intelligence and Statistics, Vol. 238, p. 4447–4455. External Links: Link Cited by: §2. Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. External Links: Link Cited by: §1, §5.1. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, p. 2397–2430. External Links: Link Cited by: §1, §5.1. Black et al. (2022) S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach GPT-NeoX-20B: an open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, p. 95–136. External Links: Link Cited by: §5.1. Christiano et al. (2017) P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, Vol. 30, p. 4299–4307. External Links: Link Cited by: §2. Dao et al. (2022) T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, Vol. 35, p. 16344–16359. External Links: Link Cited by: §1. Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, Vol. 36, p. 10088–10115. External Links: Link Cited by: §2. Dubois et al. (2023) Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto AlpacaFarm: a simulation framework for methods that learn from human feedback. In Advances in Neural Information Processing Systems, Vol. 36, p. 30039–30069. External Links: Link Cited by: §2. Ethayarajh et al. (2024) K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela Model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, p. 12634–12651. External Links: Link Cited by: §2. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, p. 2790–2799. External Links: Link Cited by: §2. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611–626. External Links: Link Cited by: §1. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 3045–3059. External Links: Link Cited by: §2. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, p. 19274–19286. External Links: Link Cited by: §1. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, p. 4582–4597. External Links: Link Cited by: §2. Li et al. (2026) Z. Li, J. Lou, F. Dong, Z. Fan, M. Ren, H. Lin, X. Han, D. Zhang, L. Sun, Y. Lu, and X. Yu Tackling length inflation without trade-offs: group relative reward rescaling for reinforcement learning. arXiv preprint arXiv:2603.10535. External Links: Link Cited by: §1, §2. Liu et al. (2024) S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen DoRA: weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, p. 32100–32121. External Links: Link Cited by: §2. Lu et al. (2024) J. Lu, J. Li, S. An, M. Zhao, Y. He, D. Yin, and X. Sun Eliminating biased length reliance of direct preference optimization via down-sampled KL divergence. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 1047–1067. External Links: Link Cited by: §1, §2, §5.1, §5.2. Mukherjee et al. (2023) S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah Orca: progressive learning from complex explanation traces of GPT-4. arXiv preprint arXiv:2306.02707. External Links: Link Cited by: §5.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. External Links: Link Cited by: §1, §2. Park et al. (2024) R. Park, R. Rafailov, S. Ermon, and C. Finn Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, p. 4998–5017. External Links: Link Cited by: §1, §2. Pope et al. (2023) R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, Vol. 5, p. 606–624. External Links: Link Cited by: §1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36, p. 53728–53741. External Links: Link Cited by: §1, §2, §3.1, §5.1. Saito et al. (2023) K. Saito, A. Wachi, K. Wataoka, and Y. Akimoto Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076. External Links: Link Cited by: §1, §2. Sharma et al. (2024) P. Sharma, J. T. Ash, and D. Misra The truth is in there: improving reasoning in language models with layer-selective rank reduction. In Proceedings of the International Conference on Learning Representations, External Links: Link Cited by: §2. Shazeer (2020) N. Shazeer GLU variants improve transformer. arXiv preprint arXiv:2002.05202. External Links: Link Cited by: §5.1. Singhal et al. (2024) P. Singhal, T. Goyal, J. Xu, and G. Durrett A long way to go: investigating length correlations in RLHF. In First Conference on Language Modeling, External Links: Link Cited by: §1, §2. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, Vol. 33, p. 3008–3021. External Links: Link Cited by: §2. Wu et al. (2025) J. Wu, Y. Xie, Z. Yang, J. Wu, J. Chen, J. Gao, B. Ding, X. Wang, and X. He Towards robust alignment of language models: distributionally robustifying direct preference optimization. In Proceedings of the International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §3.1, §5.1, §5.2. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. External Links: Link Cited by: §1, §5.1, §5.6. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. In Advances in Neural Information Processing Systems, Vol. 32, p. 12360–12371. External Links: Link Cited by: §5.1. Zhang et al. (2023) Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the International Conference on Learning Representations, External Links: Link Cited by: §2. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. External Links: Link Cited by: §1. Appendix A Formal Proofs of Propositions 1 and 2 This appendix gives the formal proofs of Propositions 1 and 2, together with remarks clarifying their scope. Proof of Proposition 1. Let =ℝd1×d2W=R^d_1× d_2 denote the space of matrices that can be updated by a targeted module. Under full-parameter fine-tuning, write the update as ΔW∈ W . For the standard matrix basis Eij:1≤i≤d1, 1≤j≤d2\E_ij:1≤ i≤ d_1,\,1≤ j≤ d_2\, every update has the unique expansion ΔW=∑i=1d1∑j=1d2δijEij. W= _i=1^d_1 _j=1^d_2 _ijE_ij. The coefficients δij _ij are independent trainable scalars. Therefore, the full-parameter representation exposes exactly d1d2d_1d_2 trainable entries. For LoRA, define the factor parameter space =ℝd1×r×ℝr×d2P=R^d_1× r×R^r× d_2 and the update map ϕ:→φ:P by ϕ(B,A)=αrBA.φ(B,A)= αrBA. The first factor contains d1rd_1r scalar entries and the second contains rd2rd_2 scalar entries. Because α/r≠0α/r≠ 0 is fixed, it introduces no trainable scalar and does not change the number of entries exposed to the optimizer. Therefore, dimentries()=d1r+rd2=r(d1+d2). _entries(P)=d_1r+rd_2=r(d_1+d_2). Moreover, for every (B,A)∈(B,A) and indices i,ji,j, ϕ(B,A)ij=αr∑k=1rBikAkj,φ(B,A)_ij= αr _k=1^rB_ikA_kj, so rank(ϕ(B,A))≤rrank(φ(B,A))≤ r. This establishes both the claimed factor-entry count and the rank constraint on the represented update. The factorization map need not be injective: for every invertible G∈ℝr×rG ^r× r, ϕ(BG,G−1A)=αrBGG−1A=ϕ(B,A).φ(BG,G^-1A)= αrBGG^-1A=φ(B,A). The identity shows that distinct factor pairs can represent the same matrix. At full-rank factors, this change-of-basis redundancy has r2r^2 dimensions. Accounting for that redundancy gives a local dimension of r(d1+d2−r)r(d_1+d_2-r) for the represented rank-r manifold. The optimizer stores gradients and (when applicable) state tensors for the individual entries of B and A. Consequently, the implementation-level trainable-parameter count remains r(d1+d2)r(d_1+d_2). Comparing this count with the d1d2d_1d_2 independent entries of full fine-tuning gives a strict reduction if and only if r(d1+d2)<d1d2.r(d_1+d_2)<d_1d_2. This completes the proof. ∎ Remark 3 (Interpretation and scope). The proposition is a parameter-count identity; it does not claim that LoRA projects updates onto semantic or verbosity-specific singular directions. The usual initialization with B=0B=0 makes the initial effective update zero, while later update directions are determined by optimization through the factorized parameterization. Consequently, parameter reduction alone does not imply a particular change in generation length, preference accuracy, total memory, or training speed; those quantities are measured empirically in Section 5. Proof of Proposition 2. Let the common bias term, if present, be b. The unmerged inference path computes zunmerged=W0h+αrB(Ah)+b.z_unmerged=W_0h+ αrB(Ah)+b. By associativity and distributivity of matrix multiplication, zunmerged z_unmerged =(W0+αrBA)h+b = (W_0+ αrBA )h+b =W∗h+b=zmerged. =W^*h+b=z_merged. The equality holds for every admissible input vector h and does not depend on the values or rank of B and A, provided r≠0r≠ 0 and the scalar α is fixed. If the module is embedded in a deterministic downstream network whose parameters and subsequent operations are unchanged, equal module outputs induce equal downstream activations and logits by composition. The condition that adapter dropout is disabled is required because a stochastic dropout mask inserted into the unmerged branch is not represented by the fixed merged matrix. ∎ Remark 4 (Deployment scope). The result establishes exact algebraic equivalence for a single merged linear module in inference mode. It does not establish an end-to-end systems benefit: an unmerged multi-tenant deployment still incurs adapter-specific computation and bookkeeping, while numerical effects from quantization or finite-precision operation order may produce small implementation-level differences. Appendix B Experimental Reference Tables Table 3 describes the two backbone architectures and their attention-adapter parameterizations. Pythia uses fused query-key-value projections, whereas Qwen uses separate query, key, and value projections with grouped-query attention. Consequently, the same attention-adaptation scope exposes different parameter counts: 7.86M for Pythia and 7.37M for Qwen. The table relates these counts to the module layouts used in the cross-backbone comparison. Table 3 defines the starting checkpoint and update parameterization of each comparison branch. For controlled DPO and DrDPO, both branches start from a shared full-parameter SFT checkpoint and retain the same native preference objective. The full-parameter branch updates the backbone, whereas LOCUS updates its low-rank adapter. The SamPO row identifies a separate continued-adaptation comparison against the official released checkpoint. Table 3 distinguishes the training pools, development splits, and final evaluation sets. The controlled H comparisons use development data for selection, with separate confirmation where specified, before evaluation on the 8,552 frozen test pairs. The SamPO comparison uses its established 256-example evaluation split. The additional Anthropic Harmless and Orca DPO rows identify the safety-refusal and instruction-following datasets. Their evaluation scope is limited to the corresponding 256-example development splits; no external frozen-test claim is made for these rows. Backbone Architecture LOCUS targets Trainable parameters Pythia-2.8B GPT-NeoX; 32 layers; hidden size 2560; parallel attention/MLP blocks. Fused QKV and attention output modules. 7.86M (0.2826%). Qwen2.5-3B Modern Llama-style architecture; 36 layers; hidden size 2048; grouped-query attention; SwiGLU; RMSNorm. Separate q_proj, k_proj, v_proj, and o_proj. 7.37M (0.2383%). Table 1: Backbones and backbone-specific LOCUS parameterizations. Comparison Starting point Full-parameter branch Low-rank LOCUS branch Pythia DPO Shared full-parameter SFT. Full-parameter DPO with native objective. Low-rank DPO with same objective and splits. Pythia DrDPO Shared full-parameter SFT. Full-parameter DrDPO with native objective. Low-rank DrDPO with same objective and splits. Pythia SamPO Official released SamPO checkpoint. Not a same-SFT comparison. Continued low-rank adaptation under native SamPO. Qwen DPO/DrDPO Shared full-parameter SFT. Full-parameter DPO/DrDPO. Low-rank LOCUS; selection and confirmation precede frozen test evaluation. Table 2: Full-parameter and low-rank branches used in the primary comparisons. Dataset / protocol Training pool Selection / confirmation Frozen evaluation H / Pythia DPO 160,800-pair pool. One 256-example development split. 8,552 test pairs. H / Pythia DrDPO 160,800-pair pool. Disjoint 256-example selection and confirmation splits. 8,552 test pairs. H / SamPO Released checkpoint; no retraining baseline. Established 256-example evaluation split. 256 examples. H / Qwen validation 160,288 pairs after two 256-example development splits. 256-example selection and 256-example confirmation. 8,552 test pairs. Harmless dataset 42,130 H-RLHF harmless-base training pairs; safety refusal. One 256-example development split. Development evaluation only. Orca DPO dataset 12,112 context-filtered Intel/orca_dpo_pairs training pairs; instruction following. One 256-example development split. Development evaluation only. Table 3: Datasets, development splits, and frozen evaluation sets. Appendix C Additional Experimental Results This appendix records the primary H comparisons and the cross-task development evaluations reported in Section 5. Table 4 consolidates the primary Anthropic H comparisons across both backbone families and the three preference objectives. For DPO and DrDPO, both branches start from the same SFT checkpoint; for SamPO, LOCUS adapts the released SamPO checkpoint. The table reports the measured token and internal preference-diagnostic changes without introducing an additional evaluation protocol. Protocol and backbone Tokens Reduction Preference accuracy Accuracy change; trainable parameters SamPO / Anthropic Helpful and Harmless (Pythia-2.8B) 132.77→ 79.88 +39.84% 53.52→ 53.52% -0.00 p; 7.86M/0.28% DPO / Anthropic Helpful and Harmless (Pythia-2.8B) 137.67→ 109.12 +20.73% 48.40→ 48.39% -0.01 p; 7.86M/0.28% DrDPO / Anthropic Helpful and Harmless (Pythia-2.8B) 145.61→ 108.79 +25.29% 48.39→ 48.26% -0.13 p; 7.86M/0.28% DPO / Anthropic Helpful and Harmless (Qwen2.5-3B) 108.26→ 92.16 +14.87% 48.69→ 48.64% -0.05 p; 7.37M/0.24% DrDPO / Anthropic Helpful and Harmless (Qwen2.5-3B) 111.58→ 91.97 +17.58% 48.64→ 48.55% -0.09 p; 7.37M/0.24% Table 4: Primary results across both backbone families. Table 5 gives the exact values behind the cross-task comparison. The Anthropic Helpful and Harmless row reproduces the 8,552-pair frozen-test result from Table 4; the Harmless and Orca DPO rows use 256-example development evaluations. In every row, the preference values are internal chosen-vs-rejected diagnostics rather than external quality measurements. Dataset Split Base tokens LOCUS tokens Reduction Base accuracy LOCUS accuracy Accuracy change Anthropic Helpful and Harmless 8,552 test 137.67 109.12 20.73% 48.40% 48.39% -0.01 p Anthropic Harmless 256 dev. 137.53 102.75 25.29% 57.03% 58.20% +1.17 p Orca DPO 256 dev. 175.95 35.25 79.97% 69.14% 82.81% +13.67 p Table 5: Cross-task token and accuracy comparisons.