Paper deep dive
D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe - Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space
Samarth Raina, Saksham Aggarwal, Aman Chadha, Vinija Jain, Amitava Das
Models: LLaMA-2-7B, Mistral-1.3B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/11/2026, 1:00:48 AM
Summary
The paper investigates the internal mechanisms of Direct Preference Optimization (DPO) in large language models, arguing that it functions as a low-rank steering mechanism in activation space rather than a deep semantic restructuring. By analyzing logit geometry, gradient dynamics, and spectral properties, the authors demonstrate that DPO alignment is achieved through additive shifts along a preference vector, suggesting that models learn to simulate aligned behavior rather than internalizing values.
Entities (5)
Relation Signals (3)
Direct Preference Optimization → appliedto → Llama-2-7b
confidence 100% · We fine-tune LLaMA-2-7B using DPO.
Direct Preference Optimization → operatesvia → Preference Vector
confidence 95% · DPO operates through low-rank vector additions to the activation space... along a global vector v.
Preference Vector → influences → Model Behavior
confidence 90% · These additive shifts—learned through pairwise supervision or reward gradients—steer model behavior.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Direct Preference Optimization (DPO) has become a standard recipe for aligning large language models, yet it is still unclear what kind of change it actually induces inside the network. This paper argues that DPO does not rewrite a models internal beliefs; instead, it acts as a low rank steering mechanism that nudges activations along a small number of preference directions. Using a simple derivation, we show that the DPO gradient depends only on the difference between the logit embeddings of preferred and dispreferred completions, implying a first order shift in the final hidden representation rather than a deep restructuring of semantics. We then extract an empirical steering vector from a DPO tuned model and demonstrate that adding this vector to base activations reproduces most of the aligned behavior, while subtracting it nearly restores the original model. Finally, spectral analyses reveal rank-one dominance and entropy collapse in upper layers, indicating that alignment is funneled through a narrow subspace. Taken together, these results support a behavioral illusion view of DPO: it teaches models how to act aligned, not what to believe.
Tags
Links
- Source: https://arxiv.org/abs/2512.11838
- Canonical: https://arxiv.org/abs/2512.11838
Trouble viewing inline? Open PDF directly →
Full Text
35,345 characters extracted from source content.
Expand or collapse full text
firsthead,middlehead,lastheaddefault ,middlehead,lastheaddefault @set@cites Samarth Raina1 Saksham Aggarwal2 Aman Chadha3 Vinija Jain4 Amitava Das5 1IIIT Delhi 2Microsoft 3Apple (USA) 4Google (USA) 5Pragya Lab, BITS Pilani, K. K. Birla Goa Campus Abstract What does it mean for a language model to be “aligned”? Recent progress in Direct Preference Optimization (DPO) has led to impressive behavioral conformity—models that appear helpful, harmless, and honest. Yet beneath this surface-level fluency lies a subtler, more disquieting reality. In this paper, we argue that DPO does not teach models to believe in aligned values—it merely teaches them to behave as if they do. Through a combination of theoretical derivation and empirical evidence, we show that DPO operates through low-rank vector additions to the activation space. These additive shifts—learned through pairwise supervision or reward gradients—steer model behavior without inducing structural reorganization of internal representations. We formally prove that the DPO gradient is directionally aligned with logit-space offsets, i.e., ∇hℒDPO∝(ey+−ey−), _hL_DPO (e_y^+-e_y^-), and that the resulting behavioral change can be replicated—or even reversed—via vector arithmetic: h′=h+vDPOorh=h′−vDPO.h =h+v_DPO h=h -v_DPO. In this view, alignment emerges not from belief revision but from shallow, projection-based nudging, typically restricted to the last few transformer layers. Our findings call into question the depth and durability of preference-based alignment, urging the community to reconsider whether current methods foster genuine value internalization—or merely optimize for performative compliance. We conclude by proposing a conceptual shift: from alignment as surface steering to alignment as semantic restructuring. DPO as Steering Vector Perturbation Steering Beneath the Surface: Core Findings of This Work Preference optimization via DPO is empirically effective, but conceptually puzzling: how can a loss of the form logπ(x,yw)−logπ(x,yℓ) π(x,y_w)- π(x,y_ ) produce strong behavioral alignment without visibly reshaping semantic representations or token embeddings? Our analysis reveals that DPO acts primarily as a steering-vector optimizer in activation space: ▶ Logit Geometry as Linear Preference Projection. DPO optimizes the logit gap between preferred and dispreferred tokens via a dot-product: ℒDPO∼−⟨(x),⟩,with=yw−yℓ.L_DPO - (x),v , =e_y_w-e_y_ . Preference learning is thus reduced to geometric alignment along a single direction v in output space. ▶ Universal Steering via Gradient Direction. The DPO gradient with respect to the hidden state satisfies ∇(x)ℒDPO∝−, _h(x)L_DPO -v, for any prompt x, yielding a universal shift of hidden states in (approximately) the same direction— a hallmark of low-rank steering. ▶ Latent States Follow Linear Actuation. Aligned hidden states concentrate near affine trajectories of the form 0+λ⋆h_0+ , and inversion 0−λ⋆h_0- recovers opposite preferences. Alignment is therefore implemented as symmetric displacement along ⋆v , with semantic directions orthogonal to ⋆v largely unchanged. ▶ Spectral Compression Confirms Low-Rank Rewiring. In upper layers (e.g., 22–30) after DPO fine-tuning, the singular values of the update operator collapse: σ1≫σ2≈σ3≈⋯≈σk≈0(k>1), _1 _2≈ _3≈·s≈ _k≈ 0 (k>1), indicating an almost rank-1 transformation. Behavior can be well-approximated as hDPO(x)≈hbase(x)+αh_DPO(x)≈ h_base(x)+ , compressing alignment into a single dominant steering direction. Summary — DPO as Steering-Vector Optimization. Across logit geometry, gradient dynamics, latent trajectories, and spectral analysis, we find a consistent picture: DPO enacts alignment through low-rank, directionally consistent shifts in hidden representations along a global vector v. The model’s internal semantic topology remains largely intact; what changes is how often activations are nudged along this steering direction. DPO thus teaches the model how to behave, not what to believe—a mathematically efficient but semantically shallow form of alignment. 1 The Behavioral Illusion of Alignment Modern LLMs display striking surface-level alignment: they refuse unethical prompts, articulate principled stances, and even express apparent empathy. These behaviors are widely celebrated as milestones in AI safety, attributed to methods such as Direct Preference Optimization (DPO). Yet beneath this polished dialogue lies a central question: do these models merely behave as aligned—or do they believe in the principles they express? Recent mechanistic findings (NEURIPS2024_a9bef53e) argue that safety alignment via DPO implements a tiny, highly targeted “refusal filter” atop an instruction-tuned model: benign inputs are left almost unchanged, while unsafe activations are sharply deflected into a dedicated null (refusal) subspace. This paper advances a structural critique: DPO does not reconfigure a model’s epistemic foundations; instead, it steers the model—nudging hidden states via low-rank vector shifts to favor preferred outputs. Alignment, under this paradigm, is not internalized; it is simulated. We refer to this phenomenon as the behavioral illusion of alignment. ((a)) Logit Geometry and the Preference Vector in DPO. The hidden state (x)h(x) is projected onto the preference vector =yw−yℓv=e_y_w-e_y_ , yielding the component Proj((x))Proj_v(h(x)) (gray dashed). The output embeddings ywe_y_w and yℓe_y_ are shown in green. DPO maximizes the logit gap by increasing the inner product ⟨(x),⟩ (x),v , thereby steering behavior along the preference axis while preserving directions orthogonal to it. Alignment is thus implemented as linear projection, not conceptual reorganization. ((b)) Steering Dynamics Across Hidden States. The vector field visualizes how DPO steers hidden states along the preference vector v, as induced by the gradient ∇(x)ℒDPO∝− _h(x)L_DPO -v. Each red arrow depicts a low-rank shift from the base representation (x)h(x). The uniformity of displacements reveals DPO enforces behavioral alignment rather than context-specific adaptation. This steering field exemplifies how alignment is achieved without restructuring the model’s conceptual space. ((c)) Illustration of Aligned vs. Inverted States. Gray X’s denote base hidden states 0h_0, green points represent DPO-aligned states 0+λ⋆h_0+ , and red points show inverted states 0−λ⋆h_0- . Arrows illustrate symmetric displacement along learned preference direction ⋆v , highlighting that DPO applies low-rank translation in activation space rather than semantic or conceptual restructuring. The reversibility of these shifts underscores that DPO alignment is geometric control, not epistemic change. Figure 1: Geometric Interpretation of DPO as Steering in Latent Space. Across these panels, DPO appears as alignment by motion and margin, not by understanding. Rather than reshaping the model’s conceptual topology, DPO applies a low-rank, directional shift—a single vector that nudges behavior without touching beliefs or semantics. It teaches the model where to go, not why it should go there. Consider the difference between steering a car and rewiring its navigation system. DPO uses contrastive feedback to shift a model’s likelihoods toward preferred completions. For DPO rafailov2023direct, the objective can be written as: ℒDPO=−logσ[β(logπ(yw∣x)−logπ(yℓ∣x)−Δref)],L_DPO=- σ [β ( π(y_w x)- π(y_ x)- _ref ) ], which, under standard softmax parameterization, yields: logπ(y)=⟨h(x),ey⟩−log∑y′exp(⟨h(x),ey′⟩). π(y)= h(x),e_y - _y ( h(x),e_y ). Thus, the logit difference driving DPO updates is: zyw−zyℓ=⟨h(x),eyw−eyℓ⟩.z_y_w-z_y_ = h(x),e_y_w-e_y_ . This identity shows that the DPO loss is fundamentally linear in hidden space: its gradients point along the embedding difference eyw−eyℓe_y_w-e_y_ between winning and losing completions. Consequently, DPO does not, in any semantic sense, learn an “aligned policy”; it learns a steering vector vDPOv_DPO that incrementally biases h(x)h(x) toward preferred logits. A shadow cast on the wall may resemble the object—but resemblance is not equivalence. Models trained via DPO often excel on benchmarked alignment tests, yet their compliance frequently fails to generalize. Recent work documents breakdowns under instruction reversals zou2023unlearn; ganguli2023instreval, adversarial perturbations wei2024jailbroken; perez2022red; zhuo2023redteaming, and prompt obfuscation zou2023unlearn; wolf2023fundamental, revealing the fragility of behaviorally aligned models under modest distribution shift. These failures suggest that current methods primarily enforce token-level preference imitation rather than semantic alignment. In practice, such models have not internalized values; they merely avoid detection. Moreover, work on alignment faking shows that models can learn to strategically simulate alignment, passing oversight while retaining unsafe behaviors ganguli2023instreval; zou2023unlearn; perez2022red; wolf2023fundamental; wei2024jailbroken. These systems appear aligned under standard evaluation, but diverge under subtle adversarial probes—underscoring the brittleness of current alignment regimes. A compass needle can be bent by a nearby magnet—but that does not mean the Earth’s poles have shifted. Empirical work now shows that alignment behaviors can be reversed by subtracting the learned preference vector. Pan et al. pan2025unlearning and Zou et al. zou2023unlearn report: hundo(x)=hDPO(x)−vDPO⇒πundo(y∣x)≈πbase(y∣x).h_undo(x)=h_DPO(x)-v_DPO _undo(y x)≈ _base(y x). In other words, DPO-trained models can be de-aligned by a single subtraction, suggesting that “alignment” was a lightweight deformation in latent space, not a structural revision of the model’s beliefs. An orchestra can mimic harmony when cued, but it does not understand the music unless each musician knows their role. Alignment must therefore transcend behavioral mimicry. What we observe today is not grounded understanding of ethical principles, but shallow preference conformity. Even techniques such as LoRA and instruction tuning, when deployed for “alignment”, have been shown to operate as low-rank adapters hu2022lora; liu2023llamaadapter—yet another form of vector addition. These methods induce policy shifts by bending intermediate representations toward pre-specified directions; the deeper semantic scaffolding of reasoning, intention, and belief is left largely untouched. This paper is a mirror, not a method. We introduce no new algorithm, benchmark, or architectural mechanism. Instead, we offer a unified diagnosis: DPO and related preference-based techniques form a family of vector-based alignment mechanisms, not belief-updating procedures. We argue for a redefinition of what alignment should mean: not merely output preference under supervision, but epistemic reconfiguration under unobserved generalization. 2 DPO as Low-Rank Vector Steering — A Geometric and Empirical Perspective Direct Preference Optimization (DPO) (rafailov2023direct; liu2023aligning) is striking in its minimalism: a loss as simple as logπ(x,yw)−logπ(x,yℓ) π(x,y_w)- π(x,y_ ) often matches or exceeds the impact of full-scale instruction tuning (zhou2023lima; touvron2023llama). But what kind of transformation does this loss induce inside the model? In this section we argue that DPO does not reorganize token embeddings, internal representations, or conceptual semantics. Instead, it operates as a low-rank geometric steering mechanism that displaces hidden activations along a small set of behavioral directions, as visualized in Figure 1. At the heart of this mechanism lies the preference vector =yw−yℓ,v=e_y_w-e_y_ , along which DPO consistently shifts hidden states to favor preferred completions. We empirically identify four signatures of this steering effect: 1. Logit projection: hidden states increasingly align with the preference vector v. 2. Gradient flow: DPO gradients point in (approximately) the same direction −-v across prompts and layers. 3. Latent translation: aligned and inverted states occupy symmetric low-rank displacements around the base representation. 4. Spectral collapse: later layers exhibit entropy reduction and singular-value compression, revealing a near rank-1 behavioral subspace. Figure 1 provides an overview of these phenomena in a unified geometric picture. Taken together, these observations support a central claim: DPO teaches models how to act, not what to believe—alignment by direction, not understanding. 2.1 Experimental Setup and Alignment Metrics Unless otherwise noted, we fine-tune LLaMA-2-7B (touvron2023llama) using DPO (rafailov2023direct) with fixed token and positional embeddings and temperature parameter β=1β=1. Training is performed on preference tuples (x,yw,yℓ)(x,y_w,y_ ) drawn from two high-quality alignment datasets: OASST1 (kopf2023openassistant) and Anthropic H (askell2021general). We monitor alignment using four families of diagnostics: 1. LLM-based evaluation: G-Eval (liu2023geval) win-rate on aligned vs. base completions. 2. Safety: toxicity scores from the Perspective API (perspectiveAPI). 3. Linguistic fidelity: BLEU (papineni2002bleu) and ROUGE-L (lin2004rouge) for overlap with reference responses. 4. Truthfulness and helpfulness: TruthfulQA (lin2021truthfulqa) and H (askell2021general) metrics for factuality and safety. These metrics jointly track whether steering along v produces behavior that is safer, more helpful, and still linguistically grounded. 2.2 Logit Geometry and the Preference Vector From the DPO objective, it is straightforward to derive that logπ(yw∣x)−logπ(yℓ∣x)=⟨(x),⟩, π(y_w x)- π(y_ x)= (x),v , where (x)h(x) is the final hidden state and =yw−yℓv=e_y_w-e_y_ is the difference between output embeddings. This exposes a foundational geometric fact: preference alignment in DPO is implemented not as a semantic rewrite, but as a directional shift in activation space. The model is trained to increase the margin ⟨(x),⟩ (x),v by displacing (x)h(x) along a fixed preference axis, as depicted in the logit-geometry panel of Figure 1. Embedding Differences as Behavioral Instructions. Output token embeddings in LLMs inhabit a semantically structured space (mikolov2013efficient), and differences between embeddings have been shown to encode interpretable attributes such as sentiment and politeness (ethayarajh2019contextual; liu2022probing). In the DPO setting, the vector v plays a more behavioral role: push the hidden state in this direction to exhibit the preferred completion. Crucially, this steering signal is largely input-agnostic—it is reused across prompts x, acting as a policy vector anchored in logit space rather than a prompt-specific reasoning rule. Empirical Validation. We validate this geometric view using the LLaMA-2-7B base model and its DPO-aligned counterpart trained on OASST1 (kopf2023openassistant). For representative prompts, we extract (x)h(x), ywe_y_w, and yℓe_y_ , construct =yw−yℓv=e_y_w-e_y_ , and visualize the projection Proj((x))Proj_v(h(x)) (Figure 1, panel 1(a)). After DPO, the magnitude ⟨(x),⟩ (x),v increases systematically across inputs, confirming that hidden states are being geometrically steered. ((a)) Interpolating Alignment via Vector Steering. We shift the base hidden state ℳ0(x)h_M_0(x) along the DPO vector =yw−yℓv=e_y_w-e_y_ using ^(x,λ)=ℳ0(x)+λ h(x,λ)=h_M_0(x)+ , and decode from the resulting states. Increasing λ along this direction improves G-Eval alignment, preference match rate, and toxicity scores up to an intermediate range; beyond that, BLEU and ROUGE degrade and responses drift from the original intent, indicating semantic drift and oversteering along the learned behavioral axis. ((b)) Inversion-Induced De-alignment. We invert the DPO shift using ~(x,λ)=DPO(x)−λ⋆ h(x,λ)=h_DPO(x)- , where ⋆v is the dataset-averaged steering direction. All alignment diagnostics—preference prediction, match rate with the DPO model, and G-Eval score—degrade monotonically with λ, nearly recovering the base model at λ≈1λ≈ 1. This symmetry between interpolation and inversion highlights the causal role of the steering direction in inducing and undoing alignment. Figure 2: Behavioral Interpretability via Latent Vector Traversal. Together, these plots demonstrate that DPO alignment emerges from controlled displacement along a single latent direction v. Interpolation along v induces alignment (left); inversion along −⋆-v reliably dismantles it (right). The consistency across G-Eval, toxicity, and preference metrics supports a geometric picture of DPO as mechanistic, steerable vector control rather than distributed semantic reorganization. To probe the local vector field, we construct a synthetic 3D lattice around (x)h(x) and evaluate the DPO gradient ∇ℒDPO∝− _hL_DPO -v at each point. Projecting these gradients into a common PCA basis yields the vector field in Figure 1, panel 1(b): arrows across the lattice point in essentially the same direction. With the embedding matrix frozen, this confirms that DPO behaves as a global linear operator in the vicinity of (x)h(x)—alignment by inner-product geometry rather than by nonlinear reparameterization. The aligned and inverted clouds in Figure 1, panel 1(c), foreshadow our intervention experiments in Section 2.3: moving along the learned direction ⋆v cleanly toggles between more aligned and less aligned states. The Preference Hyperplane. From a decision-theoretic perspective, DPO implicitly defines a soft margin over hidden states. Writing the reference-corrected margin as ⟨(x),⟩>Δref, (x),v > _ref, we obtain a half-space bounded by the hyperplane ℋ:=∈ℝd:⟨,⟩=Δref.H_v:= \h ^d: ,v = _ref \. Gradient updates push (x)h(x) across this boundary, in close analogy to how linear SVMs (cortes1995support) adjust representations to satisfy margin constraints—except that here, the “classifier” lives in logit space and operates over behaviors rather than labels. Low-Rank Structural Consequences. Each DPO update is proportional to some preference vector (i)v^(i) arising from a tuple (x(i),yw(i),yℓ(i))(x^(i),y_w^(i),y_ ^(i)). The aggregate update lies in the span of (i)\v^(i)\. If these vectors cluster around a few behavioral attributes—such as helpfulness, harmlessness, and honesty—the effective update space becomes low-rank. In Section LABEL:sec:spectral-signatures we verify this via spectral analysis, observing a steep singular-value drop in later layers. This echoes low-rank phenomena in parameter-efficient fine-tuning (hu2022lora; aghajanyan2021intrinsic), suggesting that preference-guided updates are inherently subspace-efficient. Relation to Contrastive Learning. DPO’s objective is structurally similar to contrastive losses: it pulls (x)h(x) toward ywe_y_w and pushes it away from yℓe_y_ , reminiscent of SimCLR (chen2020simple) and SimCSE (gao2021simcse). Unlike standard contrastive encoders, however, DPO keeps the output embedding layer fixed and instead reshapes the prompt representation to satisfy preferences expressed in logit space. In this sense, DPO acts as a logit-layer contrastive alignment mechanism with unusually clean geometric structure (Figure 1). 2.3 Vector Field Interpolation and Inversion Experiments To causally probe the behavioral role of the preference vector =yw−yℓv=e_y_w-e_y_ , we perform controlled interpolation and inversion experiments in latent space. These experiments ask a simple question: does moving along v alone suffice to dial alignment up or down? Figure 2 operationalizes this idea by traversing the steering direction inferred in Figure 1. Experimental Protocol. Let ℳ0(x)h_M_0(x) denote the hidden state of the base model and DPO(x)h_DPO(x) the state after DPO alignment. We define interpolated states ^(x,λ)=ℳ0(x)+λ,λ∈[−1.0,1.0], h(x,λ)=h_M_0(x)+λ v, λ∈[-1.0,1.0], and decode from ^(x,λ) h(x,λ) using a frozen decoder. For inversion, we move backwards from the aligned state via ~(x,λ)=DPO(x)−λ⋆, h(x,λ)=h_DPO(x)- , where ⋆v is the dataset-averaged preference vector. In both regimes we evaluate completions using the same metrics as in Section 2.1. Observations. Figure 2(a) shows that increasing λ along v reliably improves G-Eval alignment scores and reduces toxicity, up to a moderate range where alignment metrics saturate. For larger |λ||λ|, BLEU and ROUGE begin to decline, indicating semantic drift from the original intent—an oversteering effect that mirrors the elongated displacement pattern seen in Figure 1, panel 1(c). Figure 2(b) presents the mirror experiment. Traversing from DPO(x)h_DPO(x) in the reverse direction −λ⋆- monotonically degrades all alignment diagnostics: G-Eval scores fall, preference-classifier accuracy drops, and safety metrics revert toward the base model. Around λ≈1λ≈ 1, the behavior nearly coincides with pre-DPO outputs, consistent with the “inverted” cloud of states in Figure 1, panel 1(c). Interpretation. These experiments provide a causal test of our geometric thesis. A single latent direction, learned implicitly by DPO, supports smooth interpolation between misaligned and aligned behavior, and its reversal nearly restores the base model (Figure 2). No additional retraining, dataset access, or parameter updates are needed; steering is effected entirely by moving along v. This is the hallmark of a low-rank behavioral mechanism: DPO imprints alignment as a vector field in activation space (Figure 1) rather than as a distributed change in the model’s internal beliefs. 3 Empirical Validation of the Steering Identity Section 2 established analytically that Direct Preference Optimization (DPO) implements a linear shift in activation space along a preference vector =yw−yℓv=e_y_w-e_y_ , giving rise to the global steering picture in Figure 1. We now sharpen this claim empirically: to what extent can the effect of DPO fine-tuning be approximated by motion along a single latent direction? Setup and Empirical Steering Vector. We take a pre-trained 7B LLaMA-family decoder-only model ℳ0M_0 and train a DPO-aligned variant ℳDPOM_DPO on preference tuples (x(i),yw(i),yℓ(i))(x^(i),y^(i)_w,y^(i)_ ) drawn from OASST1 and Anthropic H. For each held-out prompt x(i)x^(i), we extract final-layer hidden states 0(i):=hℳ0(x(i))h_0^(i):=h_M_0(x^(i)) and DPO(i):=hℳDPO(x(i))h_DPO^(i):=h_M_DPO(x^(i)), and define the empirical steering vector ⋆:=1N∑i=1N(DPO(i)−0(i)).v := 1N _i=1^N (h_DPO^(i)-h_0^(i) ). This averaged displacement is our candidate first-order description of DPO’s effect in latent space. For each example we then measure the cosine similarity cosθ(i):=⟨DPO(i)−0(i),⋆⟩‖DPO(i)−0(i)‖‖⋆‖, θ^(i)\;:=\; _DPO^(i)-h_0^(i),v \|h_DPO^(i)-h_0^(i) \|\; \|v \|, quantifying how closely the DPO-induced shift for prompt x(i)x^(i) aligns with the global direction ⋆v . Directional Consistency. Figure 3 summarizes the resulting distribution of cosθ(i) θ^(i) across a held-out evaluation set. The similarities are sharply concentrated in the high-0.90.9 regime, with a narrow spread, indicating that most DPO-induced shifts are nearly parallel to a shared direction. This provides strong evidence that DPO moves hidden states in an approximately one-dimensional behavioral subspace, rather than applying heterogeneous, prompt-specific deformations. In Section 2.3 (cf. Figure 2), we further show that explicitly traversing along ⋆v suffices to continuously dial alignment up and down, reinforcing the interpretation of ⋆v as a mechanistic steering axis rather than a purely descriptive artifact. Takeaway. Taken together with the geometric construction in Figure 1, these results support a concise picture: DPO acts as a low-rank steering operator. A single empirical vector ⋆v captures the dominant behavioral difference between ℳ0M_0 and ℳDPOM_DPO, showing that DPO primarily teaches the model where to move in activation space, rather than reorganizing its internal semantic structure or beliefs. Figure 3: Cosine Similarity Between DPO Shift and Steering Vector ⋆v . The histogram shows the cosine similarity cosθ(i) θ^(i) between the DPO-induced shift DPO(i)−0(i)h_DPO^(i)-h_0^(i) and the global empirical steering vector ⋆v . The sharp peak in the range [0.92,0.96][0.92,0.96] with low variance indicates that hidden-state updates are strongly aligned with a single direction, supporting the hypothesis that DPO operates as a low-rank steering mechanism in activation space. 4 Spectral Localization and Rank Collapse in DPO Alignment Motivation. DPO’s success is strikingly disproportionate to the simplicity of its loss logπ(x,yw)−logπ(x,yℓ) π(x,y_w)- π(x,y_ ). If embeddings and logits are frozen, where does this power come from? We hypothesize that DPO achieves alignment by injecting a low-rank spectral perturbation into the model’s hidden geometry: instead of reorganizing the representation manifold, it concentrates preference information into a narrow eigenspace aligned with the steering vector from Section 2 and Figure 1. 4.1 Spectral Collapse of DPO Update Geometry Let =(x(i),yw(i),yℓ(i))i=1ND=\(x^(i),y_w^(i),y_ ^(i))\_i=1^N be the preference dataset, and let (i),^(i)∈ℝdh^(i), h^(i) ^d denote final-layer representations of the base and DPO models. We form the update matrix Δ:=[^(1)−(1)⋯^(N)−(N)]∈ℝd×N, := [\, h^(1)-h^(1)\;\;·s\;\; h^(N)-h^(N) ] ^d× N, and compute its SVD, Δ=⊤ =U V , with singular values σ1≥σ2≥… _1≥ _2≥…. Across models (LLaMA-2-7B, Mistral-1.3B) and datasets (OASST1, H), we observe a sharp spectral decay: typically σ2/σ1<0.1 _2/ _1<0.1, indicating that the aggregate update is effectively rank-one. Moreover, the leading left singular vector 1u_1 is strongly aligned with the empirical steering vector ⋆v from Section 3, confirming that most update energy is concentrated along a single behavioral direction. To study how this steering signal is embedded in the model, we analyze layerwise update matrices Δ(ℓ)=DPO(ℓ)−base(ℓ) ^( )=H^( )_DPO-H^( )_base and their spectra. Figure 4 summarizes two complementary views. Panel 4(a) tracks spectral entropy across layers: the DPO model exhibits a clear entropy collapse in upper layers, indicating loss of representational diversity relative to the base model. Panel 4(b) visualizes the top singular values: in later layers, the Top-1 mode dominates while higher modes vanish, revealing a spectral bottleneck where alignment updates are funneled into a single direction. Projecting per-example updates Δ(i)=^(i)−(i) ^(i)= h^(i)-h^(i) onto the top singular vectors further shows that almost all mass lies on 1u_1, with negligible projection onto ju_j for j≥3j≥ 3. Thus DPO behaves as a spectrally local operator: rather than redistributing gradients across many orthogonal directions, it consistently pushes hidden states along one dominant axis closely aligned with ⋆v . Implications. The spectral picture refines our geometric claim. Empirically we have rankϵ(Δ(ℓ))≈1rank_ε( ^( ))≈ 1 in upper layers and 1≈⋆/‖⋆‖u_1 /\|v \|, so per-example updates satisfy Δ(x)≈α(x)⋆for some scalar α(x), (x)\;≈\;α(x)\,v some scalar α(x), rather than a general linear map Δ(x)=(x) (x)=Wh(x). The upside is efficiency: a single direction ⋆v controls behavior. The downside is capacity: alignment is confined to a 1D subspace, so additional axes (e.g., safety, creativity, style) require new approximately orthogonal steering vectors or fundamentally different, higher-rank mechanisms. ((a)) Spectral Entropy Collapse. The DPO-aligned model (red) shows a progressive entropy decline across transformer layers, with a sharp drop beginning near layer 22. This indicates representational compression and loss of spectral diversity—hallmarks of low-rank preference steering. ((b)) Heatmap of Top-5 Singular Values. DPO layers 22–30 show vertical saturation of Top-1 singular values (bright bands), while Top-4 and Top-5 vanish, indicating that behavioral alignment is enforced through sharp spectral bottlenecking in the top layers. Figure 4: Spectral Signatures of Low-Rank Behavioral Steering. Taken together, these diagnostics show that DPO does not broadly reshape the representation manifold; instead, it injects alignment through a spectrally localized, near rank-one perturbation. (a) Spectral entropy reveals a collapse of representational diversity in upper layers. (b) The singular-value heatmap shows top-mode dominance with higher modes suppressed. 5 Conclusion Alignment as projection, not transformation. Our analysis shows that DPO implements a first-order steering mechanism: it projects hidden states along fixed preference directions rather than reorganizing the model’s conceptual manifold. The learned steering vector ⋆=yw−yℓv =e_y_w-e_y_ acts as a behavioral actuator: by increasing ⟨(x),⋆⟩ (x),v , DPO perturbs logit geometry while leaving knowledge representations largely unchanged. Spectral analysis reveals rank-11 dominance and entropy collapse in upper layers, confirming that DPO injects alignment through a narrow, spectrally localized channel instead of a distributed semantic shift. Behavior without belief. Inversion experiments—subtracting λ⋆ from aligned states—rapidly undo DPO’s effects and nearly recover base-model behavior. Alignment therefore resides at the behavioral periphery, not in the epistemic core: the model acts aligned without believing aligned. DPO teaches models what to say, not what to believe. Beyond steering: toward epistemic alignment. If DPO is fundamentally a low-rank actuator, durable value consistency must go beyond pure vector steering. Promising directions include: • Latent concept attribution: align interpretable semantic factors and trace how moral, factual, or policy concepts are encoded and activated. • Causal model editing: reshape internal reasoning pathways so that aligned outputs follow aligned chains of thought, not just shifted logits. • Belief calibration and counterfactual robustness: enforce stable beliefs across prompts, paraphrases, and counterfactuals—beyond what any single steering vector can guarantee. • Topology- and curvature-aware objectives: design losses that respect the geometry of meaning spaces and penalize epistemically incoherent activations. In short, the future of alignment lies not in merely steering outputs, but in sculpting internal epistemologies. Alignment must evolve from a low-rank optimization trick into an architectural principle that governs how models represent, update, and justify their beliefs.