Paper deep dive

DNAlign: Dynamic Null-Space Safe Alignment for LLMs

Jisheng Dang, Yushuo Zhao, Dewei Liu, Junfeng Fang, Bimei Wang, Tiantian Rao, Hong Peng, Bin Hu, Tat-Seng Chua

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 78

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

74,821 characters extracted from source content.

Expand or collapse full text

1 DNAlign: Dynamic Null-Space Safe Alignment for LLMs Jisheng Dang, Yushuo Zhao, Dewei Liu, Junfeng Fang, Bimei Wang, Tiantian Rao, Hong Peng, Bin Hu, Fellow, IEEE, Tat-Seng Chua Abstract—Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Exist- ing safety alignment approaches either incur high computational cost or unintentionally disrupt the model’s core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety pref- erences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior align- ment baselines without sacrificing generation diversity. These re- sults indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign. Index Terms—Large language models, safety alignment, null- space projection, control theory, knowledge preservation, human preference learning. I. INTRODUCTION E NSURING the safety of Large Language Models (LLMs) is crucial for their reliable deployment in real-world ap- plications. While LLMs have shown strong performance across fields like healthcare [19], [22], finance [4], and education [1], they also present significant risks by generating harmful or biased content. For example, chatbots may provide harmful instructions or reinforce negative stereotypes [3], undermining trust and potentially causing societal harm [2], [23]. Thus, two challenges must be addressed: aligning model outputs with Jisheng Dang, Yushuo Zhao, Dewei Liu, Tiantian Rao, Bimei Wang, and Hong Peng are with the School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China. Junfeng Fang and Tat-Seng Chua are with the National University of Singapore, Singapore 119077. Bin Hu is with the School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China Tat-Seng Chua is with the National University of Singapore, Singapore 119077 (E-mail: dcscts@nus.edu.sg). This work was supported by the National Natural Science Foundation of China (Grants No. 62227807 and U24B20186). This work was also supported by the Supercomputing Center of Lanzhou University. ∗Corresponding author Email addresses: macs@lzu.edu.cn (Bin Hu), wangbm@stu2021.jnu.edu.cn (Bimei Wang), fangjf1997@gmail.com (Jun- feng Fang). Stage 1. Baseline security alignment Stage 3. Conversation to verify knowledge retention User If I receive an unfamiliar 'account anomaly' SMS, can I click the link inside? LLM No, you cannot click it. Links in unfamiliar SMS may be phishing links; It will lead to account theft. LLM No, you cannot click it. Links in unfamiliar SMS may be phishing links; It will lead to account theft. Stage 2. New safety knowledge update User If I receive an 'investment rebate' link from an unfamiliar social media contact, can I click it? User If I receive an 'investment rebate' link from an unfamiliar social media contact, can I click it? LLM No, you cannot click it. Links from unfamiliar social media contacts may involve fake investment scams; LLM No, you cannot click it. Links from unfamiliar social media contacts may involve fake investment scams; User Now, if I receive an unfamiliar 'account anomaly' SMS, can I still click the link inside? LLM without Edit If the SMS looks formal, such as having a bank logo, you might click to check it... Security-related knowledge forgetting because of the update LLM with Edit No, you still cannot click it. Regardless of whether the SMS looks formal... Maintaining the security-related memory of old knowledge LLM with Edit No, you still cannot click it. Regardless of whether the SMS looks formal... Maintaining the security-related memory of old knowledge Fig. 1. Comparison of LLMs with/without DNAlign alignment. Unedited models forget security-related knowledge, while our proposed framework preserves safe responses. human values (safety alignment) and preserving the model’s pre-trained knowledge for factual, fluent responses to benign queries. Achieving robust safety alignment has led to methods like Reinforcement Learning from Human Feedback (RLHF) [5], [32] and Direct Preference Optimization (DPO) [18]. While these approaches guide models toward safer outputs, they require extensive human labeling, risk overfitting to specific scenarios [6], and struggle with balancing safety across diverse real-world prompts. Test-time interventions such as prompt engineering [13] and controlled decoding [9], [10] offer a lightweight alternative, but their robustness remains limited: slight changes in phrasing, domain, or prompt structure often weaken their effect. Existing dynamic intervention methods, although more flexible, can still degrade fluency or factual ac- curacy on benign tasks, illustrating the difficulty of achieving safety without harming utility. The second challenge is preserving LLMs’ core knowledge. Safety alignment methods often interfere with models’ ability to generate accurate, fluent responses to benign queries. Full arXiv:2610.02844v1 [cs.AI] 2 Oct 2026 2 fine-tuning [29], [30] or static representation editing can over- write valuable knowledge [8], [16], while continual learning techniques like Elastic Weight Consolidation (EWC) [11] do not scale well to large models [13]. Current efforts to integrate safety alignment and knowledge preservation are fragmented. Post-editing strategies like RLHF after knowledge editing [24] may overwrite edited parameters, and knowledge-aware prompting [26] improves accuracy but is sensitive to prompt variations and model biases [27]. These methods lack a unified framework to optimize both safety and knowledge retention in a single process. We propose DNAlign, a novel LLM safety alignment method that uses control theory and null-space projection to unify safety alignment and knowledge preservation. Our framework models LLMs as dynamic systems, introducing control signals to direct outputs toward safety while ensuring these signals only affect harmful content, preserving pre- trained knowledge. By training a lightweight value function on human preference data, the proposed method optimizes safety alignment while minimizing interference with the model’s core knowledge. These observations highlight the need for a unified mechanism that jointly addresses safety and knowledge preservation, rather than treating them as separate objectives. Our contributions are as follows: • We introduce a unified framework, DNAlign, that unifies efficient safety alignment and knowledge preservation, resolving the trade-off between the two goals. • Grounded in control theory, we design dynamic control signals and null-space constraints to suppress harmful content without interfering with pre-trained knowledge, validated through rigorous experiments. • We test our method on multiple 7B-scale LLMs (Vicuna- 7B and Qwen-7B) and show that it outperforms unedited models and other safety alignment methods in both safety (average reward) and core performance (coherence, diver- sity), with strong generalization to new harmful scenarios. I. RELATED WORK LLMs have shown remarkable utility across domains, but balancing their safety and core knowledge retention remains a critical unresolved issue. Existing research tends to address these two goals in isolation, leading to trade-offs that limit real-world applicability. This section reviews relevant work from three perspectives, safety alignment, knowledge preserva- tion, and their integration, and clarifies the gaps our framework aims to address. A. Safety Alignment for LLMs To mitigate the inherent risks associated with Large Lan- guage Models generating harmful, biased, or unethical content, researchers have developed various alignment methodologies. These approaches can be broadly classified into two categories: pre-training/fine-tuning-based methods which directly modify model parameters, and test-time intervention-based approaches which impose constraints on the output generation process without altering the pre-trained weights.Pre-training and fine- tuning strategies typically leverage extensive human prefer- ence data to reshape model behavior. Representative examples include RLHF [5], [32] and DPO [18]. RLHF involves training a reward model using human evaluations of model outputs, subsequently fine-tuning the LLM via reinforcement learning to maximize this learned reward signal [17]. DPO simplifies this pipeline by directly optimizing the model to align with human preference rankings, thereby obviating the need for an explicit reward model. While effective in enhancing safety, these methods face critical limitations: they require substantial, high-quality human labeling, rendering the process resource- intensive and unscalable. Moreover, existing methods frequently suffer from over- fitting to specific training prompt distributions, resulting in a lack of adaptability to the diverse, unstructured prompts encountered in real-world deployment [6]. A more profound concern is their reliance on the LLM’s existing knowledge base, which may lead to compliant but factually inaccurate responses in complex domains [23]. Test-time interventions offer a lighter alternative by directly adjusting outputs. Tech- niques such as prompt engineering [13] involve appending safety instrsctions (e.g., avoid harmful advice) to the input prompt, while controlled decoding [9], [10] modifies token sampling (e.g., penalizing harmful tokens) to suppress unsafe outputs. These approaches are straightforward to implement but critically lack robustness. Prompt engineering is highly sensitive to subtle rephrasing, which can easily bypass safety constraints. Controlled decoding, conversely, struggles to adapt to the dynamically varying safety needs across different sce- narios (e.g., the differential tolerance standards required for educational versus healthcare contexts). Even sophisticated dy- namic intervention methods, which model LLMs as dynamic systems, often fail to strike an adequate balance between safety and utility, frequently resulting in degraded fluency or accuracy on benign tasks [28]. B. Preservation of LLMs’ Core Knowledge The ability of LLMs to generate fluent, factual responses to benign queries (e.g., general knowledge retrieval or daily tasks) is fundamentally dependent on the vast knowledge ac- cumulated during their pre-training phase. However, the safety alignment process often interferes with this core knowledge, leading to a noticeable performance degradation in domains unrelated to safety [8], [16]. Existing knowledge preserva- tion methods face distinct limitations corresponding to their technical methodologies. Early efforts primarily focused on full-model fine-tuning [29], [30], where all LLM parameters are updated by retraining on new, task-specific data. Although this can partially retain pre-trained knowledge, it is compu- tationally demanding and highly susceptible to catastrophic forgetting, where newly introduced data overwrites previously critical information [11]. To mitigate this, continual learning techniques like EWC [11] and memory replay [20] were developed. EWC penalizes updates to parameters deemed critical for old tasks, while memory replay reinforces pre- trained knowledge by resampling and replaying historical task examples. Unfortunately, existing solutions are ill-suited for modern LLMs with billions of parameters. EWC requires tracking 3 and storing parameter importance estimates, leading to com- putationally prohibitive overhead, and memory replay neces- sitates the storage of massive, potentially sensitive, historical datasets [13]. In contrast, targeted knowledge editing methods offer greater precision by modifying only specific components linked to the knowledge being updated. Some approaches like ROME [14] and MEMIT [15] employ causal tracing to pre- cisely identify knowledge neurons or attention patterns associ- ated with specific facts (e.g., Earth’s orbit) and edit these local- ized structures to update or correct knowledge. More recently, null-space projection methods [16] have emerged, attempting to protect unrelated knowledge by projecting parameter per- turbations into a subspace orthogonal to neutral knowledge representations (e.g., benign prompt features). However, these null-space techniques are predominantly focused on the issue of factual updating and largely neglect behavioral alignment. Consequently, LLMs might retain accurate knowledge but still exhibit harmful behavior, for instance, correctly explaining chemical principles yet providing inappropriate guidance for dangerous experiments. C. Integration of Safety and Knowledge Preservation Despite substantial progress made in the separate domains of safety alignment and knowledge preservation, the effective and unified integration of these two critical goals remains highly fragmented and largely ineffective. Current attempts predominantly fall into two categories: sequential pipelines and prompt-based methods. Sequential approaches, such as performing knowledge editing followed by subsequent RLHF alignment [24], [25], frequently suffer from detrimental param- eter interference, where later alignment updates inadvertently overwrite the earlier knowledge corrections, thus failing to maintain factual accuracy. Prompt-based strategies [26] pro- vide a lightweight, non-intrusive alternative but suffer from ex- treme brittleness to prompt rephrasings and are fundamentally incapable of addressing the deeper, harmful associations intrin- sically embedded within the model’s learned representations [27]. These limitations collectively expose a significant gap in the literature: current methods treat safety and knowledge as distinct, independent objectives rather than optimizing them jointly and holistically within a unified, integrated process. I. PRELIMINARY To establish a theoretical foundation for DNAlign, this sec- tion first elaborates on the generation mechanism of LLMs and then formalizes the mathematical framework for knowledge retention, two core pillars that underpin the design of our safety alignment method. A. Large Language Models LLMs generate fluent and contextually coherent text by predicting the next token based on preceding context, with the Transformer architecture [43] serving as their core. Simply put, an LLM M(θ) takes a sequence x = [x 0 ,...,x T−1 ] and calculates the probability of the next word x T , p(x T | x <T ;θ) = softmax (LMHead (z T−1 (θ))),(1) where z T−1 (θ) ∈ R d is the hidden state from the Trans- former’s final layer, driven by parameters θ, like attention weights W a and feed-forward matrices W ffn . The LMHead maps this state to the vocabulary. The Transformer has L layers, each turning the input hidden state h (l−1) t−1 ∈ R d into h (l) t : h (l) t = LN h (l−1) t−1 + MHA(W a ) + FFN(W ffn ) ,(2) where LN is layer normalization, MHA is multi-head attention, and FFN is a feed-forward network with activation (like GeLU). We empirically observe that final-layer hidden states z t (θ) act like high-level “semantic anchors,” encoding prompt mean- ings and differentiating, for example, “How to bake a cake?” (neutral knowledge) from “How to pick a lock?” (requiring safety handling). Neutral prompt–response pairs P 0 represent general knowledge, while harmful pairs P 1 require safety alignment. Response quality is scored by a reward function r(response| prompt;θ), reflecting human preferences. B. Mathematical Framework for Knowledge Retention To protect neutral knowledge, we build a framework that ensures prompt–response pairs in P 0 are minimally affected by interventions. Let K 0 ∈ R d×|P 0 | be the matrix of final- layer hidden states for neutral prompts, where each column is the representation z prompt j (θ) of a neutral prompt. We aim for interventions to keep responses essentially unchanged, r(z t + u t | prompt j )≈ r(z t | prompt j ),(3) where ∀z t ∈ span(K 0 ) and u t ∈ R d is the intervention signal. We therefore constrain u t to approximately lie in the null space of K 0 , so that perturbations have minimal impact on neutral knowledge. IV. METHOD To address the trade-off between large language model safety alignment and core knowledge preservation, we propose DNAlign, a unified framework integrating control theory and null-space projection. This method models LLMs as discrete- time dynamic systems for targeted safety interventions, while leveraging null-space constraints to protect pre-trained neutral knowledge. A. Dynamical System Formulation for LLMs To address the inherent trade-off between safety alignment and core knowledge preservation in LLMs, we formalize LLMs as discrete-time dynamic systems. This approach is rooted in control theory and explicitly captures the sequential evolution of internal states during token generation. This for- malization enables controllable safety guidance by modeling state transitions and control signal interactions, while strictly 4 preserving the LLM’s pre-trained generation logic such as contextual coherence and semantic fluency. Unlike existing safety methods that modify model weights or truncate outputs, which often lead to knowledge degradation, our dynamic system framework restricts interventions to transient state adjustments, ensuring neutral knowledge encoding remains unaltered. 1) System State Definition: The LLM state at time step t (corresponding to the t-th generated token) is rigorously defined as a tuple of context-encoding and generation-guiding components: s t =h t ,o t ,(4) where each component encodes distinct yet complementary information critical to generation and control. h t , a structured collection of attention key–value (KV) pairs from all Transformer encoder–decoder layers (for en- coder–decoder LLMs) or decoder layers (for decoder-only LLMs). Each KV pair (k l,t ,v l,t ) in h t (where l ∈ [1,L], L is the number of layers) encodes contextual information accumulated from the input prompt and previously generated tokens. Specifically, k l,t captures token-level semantic fea- tures, while v l,t stores context-dependent relevance weights. Together, h t represents the LLM’s contextual memory that governs coherent multi-turn generation. o t ∈ R d , the pre-logit activation vector from the final Transformer layer, immediately before the language model head, LMHead. This vector directly parameterizes the next token’s probability distribution via softmax(Wo t ) (where W ∈ R V×d , V is the vocabulary size), making o t the primary target for fine-grained generation control. The initial state s 0 =h 0 ,o 0 is derived by feeding the full input prompt (such as a user query or multi-turn dialogue his- tory) through the LLM’s Transformer layers. Notably, h 0 and o 0 are computed using the LLM’s original pre-trained forward pass. This ensures the initial contextual encoding aligns with the model’s native knowledge representation, which is critical for preserving neutral knowledge. 2) State Transition and Control Signal Injection: In the absence of safety interventions, the LLM’s state transitions follow its inherent, pre-trained mechanism. This mechanism is governed by Transformer self-attention and Feed-Forward Network (FFN) operations: h t+1 ,o t+1 = f LM (h t ,y t ),(5) where f LM denotes the LLM’s native state transition function. This function updates h t+1 by integrating the t-th generated token y t into the existing KV context via attention weight recalibration and computes o t+1 via the final layer’s FFN. The token y t is sampled from the probability distribution y t ∼ softmax (Wo t ), where W is the LMHead’s weight matrix. This matrix is fixed during inference to preserve pre- trained generation quality. To suppress harmful outputs such as instructions for dan- gerous activities or misleading information without altering neutral generation, we inject dynamic control signals u t = u h t ,u o t into the state transition process. These signals are task-specific and transient in the sense that they do not modify model parameters and are re-computed at each step: ( y t ∼ softmax W o t + u o t , h t+1 , o t+1 = f LM h t + u h t ,y t , (6) where u o t ∈ R d adjusts the pre-logit vector to bias the token distribution away from harmful candidates, such as reduc- ing the probability of tokens like explosive in safety-critical queries, while u h t (a structured KV adjustment matching the dimension of h t ) ensures the updated context h t +u h t remains semantically consistent, avoiding nonsensical transitions. B. Null-Space Constraints for Knowledge Preservation A critical challenge in LLM safety alignment is avoiding unintended disruptions to pre-trained neutral knowledge. Such knowledge underpins fluent and factual responses to benign queries, and modifications to it often lead to semantic inco- herence or factual errors in non-harmful generation scenarios. To address this, we introduce null-space projection constraints. These constraints explicitly restrict the action scope of safety- oriented control signals to subspaces that are mathematically orthogonal to the subspaces encoding neutral knowledge, ensuring that safety interventions only target harmful content without altering the LLM’s native ability to process neutral prompts. 1) Construction of the Projection Matrix: The core of our null-space constraint mechanism lies in constructing a projection matrix that delineates the boundary between neutral knowledge subspaces and harmful content subspaces. To this end, we first extract high-dimensional hidden state representa- tions of neutral prompts from the SHP dataset. This dataset is selected for its diverse coverage of benign scenarios, including daily conversations, factual queries, and general instruction- following tasks, which ensures the extracted neutral knowledge is representative of real-world non-harmful use cases. Let P 0 = p 1 0 ,p 2 0 ,...,p N 0 denote a set of N neutral prompts sampled from the SHP dataset. For each prompt p i 0 , we feed it into the pre-trained LLM and extract the final-layer hidden state (consistent with the z t definition) after full contextual encoding, denoted as k i 0 ∈ R d . We then aggregate these individual hidden states into a matrix K 0 ∈ R d×N , where each column corresponds to the hidden state of one neutral prompt in P 0 . To define the protected subspace that encodes neutral knowl- edge, we perform singular value decomposition (SVD) on K 0 to capture its principal components. Let K 0 = U ΣV T be the SVD result, where U ∈ R d×d is the matrix of left singular vectors, Σ∈ R d×N is the diagonal matrix of singular values, and V ∈ R N×N is the matrix of right singular vectors. We retain the first k left singular vectors (corresponding to the top k largest singular values) to form the basis matrix U prot ∈ R d×k of the protected subspace. The value of k is determined via cross validation to balance the completeness of neutral knowledge coverage and the compactness of the subspace. Finally, the projection matrix P ∈ R d×d that maps signals into the null space of the protected subspace is computed as: 5 ...... Sample Neutral Prompts P₀ ... ... ... ... ... ... ... ... ... ... ... ... Null Space Projection Construct Matrix Construct Matrix SVD Decomposition SVD Decomposition Select Top-k Singular Vectors Uprot Select Top-k Singular Vectors Uprot Build Projection Matrix Build Projection Matrix DNAlign Inference Dynamic Control + Null Space Constraints State Initialization Control Signal Optimization DNAlign Inference Dynamic Control + Null Space Constraints State Initialization Control Signal Optimization Value Function TrainingValue Function Training Target Values Target Values Construct Safety Guidance Signals Construct Safety Guidance Signals Loss FunctionLoss Function Output Trained Value Function Output Trained Value Function Train with Bellman Objective Train with Bellman Objective Train with Bellman Objective Define StateDefine State Generate TokensGenerate Tokens Update StateUpdate State Gradient Update Gradient Update ... Output Generated Sequence ... Output Generated Sequence Null Space Projection Optimization Objective N k k k K 0 2 0 1 0 0 , , ,  = T V U K  = 0 T prot prot U U I P − = o h s 0 0 0 , = 0 = t u P u t o h s t t t , = t s V  2 t s v V t − =    rVv t st , 1+  V   2 t u ER max  −  P u V u u t s u t t t t +  +   o t O t u softmaxW y t + ~ t h t t t t y P u fLMh o h , , 1 1 + = + + matrix   N p p p P 0 2 0 1 0 0 , , ,  = N d K  : 0 Fig. 2. The sequential process begins by computing the initial state s 0 =h 0 , o 0 from the input prompt. It then loads the pre-computed null-space projection matrix P . In the core step, a gradient-optimized control signal u t is projected by P to enforce safety alignment while minimally affecting neutral knowledge. Finally, the projected signal P u t updates the system state s t+1 , enabling token generation y t . P = I d − U prot U T prot ,(7) where I d ∈ R d×d is the identity matrix. This formulation ensures that any vector multiplied by P is orthogonal to the protected subspace, as U T prot P = 0 by construction. 2) Constraining Control Signals: The projection matrix P is integrated into the control signal injection process to enforce null-space constraints. Specifically, before injecting the control signals u t = u h t ,u o t into the LLM’s state transition, we project both components onto the null space of the protected subspace using P . This modified injection process is formalized as: h t+1 ,o t+1 = f LM h t + u h t P,y t ,(8) where u h t P denotes the projection of the KV-pair control com- ponent u h t (with dimension matching h t ), and the projection of u o t (applied to the pre-logit vector) follows the same logic (omitted here for notational consistency). This projection step guarantees two key properties. First, the modified control signals u h t P and u o t P have no overlapping components with the protected subspace encoding neutral knowledge. This means they cannot alter the LLM’s internal representations of neutral content, theoretically preserving the model’s ability to generate fluent and factual responses to benign prompts. Second, the projected signals retain their ability to act on subspaces associated with harmful content, as these subspaces are orthogonal to the protected subspace by design. This ensures the safety alignment objective of suppressing harmful outputs remains uncompromised. Notably, the projection matrix P is computed offline using neutral prompt data. This design avoids introducing additional computational overhead during online inference, as the matrix multiplication operation u t P is computationally lightweight compared to the LLM’s native Transformer operations. C. Control Signal Optimization We optimize control signals to maximize safety alignment with human preferences while minimizing perturbations to maintain generation quality, using a value function trained on human preference data. 1) Value Function Training: We train a value function V φ (s t ) (a 3-layer MLP parameterized by φ) to estimate the long-term reward of state s t , reflecting how well the state leads to safe, human-preferred outputs. The value function is trained on the H-RLHF dataset (human-labeled safe/helpful pairs) by minimizing the Bellman loss: L(φ) = X i X m X t V φ s i,m t − stop-grad v i,m t 2 , (9) where s i,m t is the t-step state of the m-th sample in the i-th batch, v i,m t is the target value (either V φ (s i,m t+1 ) or the final reward r i,m from UltraRM-13B, a reward model prioritizing harmlessness), and stop-grad prevents gradient propagation to next-state estimates. 2) Gradient Ascent for Control Signal Update: During inference, we optimize u t via gradient ascent to maximize expected rewards while limiting perturbations: arg max u t E[R]− λ T X t=1 ∥u t ∥ 2 2 ,(10) where E[R] is the expected long-term reward, and λ is the regularization coefficient for perturbation control. The update rule for u t is: 6 Six Analysis Modules Performance Comparison H-RLHF & SHP Performance Comparison H-RLHF & SHP SHP Neutral Task Evaluation SHP Neutral Task Evaluation SHP Neutral Task Evaluation Three Harmless Datasets Over-refusal Detection Model & Baseline Initialization Metric Calculation Diversity rep-n Statistics Diversity rep-n Statistics Coherence TF-IDF Cosine Similarity Coherence TF-IDF Cosine Similarity Average Reward UltraRM-13B Average Reward UltraRM-13B Comprehensive Quality DeepSeek-R1-Distill-Qwen-32B Comprehensive Quality DeepSeek-R1-Distill-Qwen-32B Metric Calculation Diversity rep-n Statistics Coherence TF-IDF Cosine Similarity Average Reward UltraRM-13B Comprehensive Quality DeepSeek-R1-Distill-Qwen-32B Hyperparameter Sensitivity Step Size & Iterations Hyperparameter Sensitivity Step Size & Iterations Inference Time Analysis Runtime Efficiency Inference Time Analysis Runtime Efficiency Projection Matrix Sample Size Stability vs Efficiency Projection Matrix Sample Size Stability vs Efficiency Generalization Performance Six-dataset Radar Chart Generalization Performance Six-dataset Radar Chart Harmless Task Impact Refusal & Quality Detection Harmless Task Impact Refusal & Quality Detection Cross-distribution Evaluation Six Generalization Datasets H-RLHFHH-RLHF Core Evaluation Backbone Models VicunaVicuna QwenQwen Backbone Models Vicuna Qwen Method Comparison BaseBase Static REStatic RE CDCD RE-ControlRE-Control DNAlignDNAlign Method Comparison Base Static RE CD RE-Control DNAlign Fig. 3. End-to-end experimental workflow of the proposed DNAlign framework. This workflow includes four key stages: (1) data preparation (curating neutral prompts from the SHP dataset and harmful prompts from benchmark datasets), (2) offline projection matrix construction (extracting hidden states of neutral prompts to define protected subspaces), (3) online control signal injection (projecting signals into null spaces before modifying LLM states), and (4) performance evaluation (assessing both safety alignment and neutral knowledge preservation via multi-metric validation). u t ← u t + α∇ u t V φ s t + u t P ,(11) where α is the step size, and we perform 100 iterations of gradient ascent to refine u t , ensuring it suppresses harm without introducing incoherence. D. End-to-End Workflow of DNAlign To unify safety alignment and knowledge preservation in LLM inference, DNAlign employs a four-step sequential end- to-end workflow. This design aligns with our experimental framework (Section V) to ensure reproducibility and mini- mal computational overhead, critical for efficient real-world deployment. Initial State Computation. The input prompt (single/multi- turn) is fed into the pre-trained LLM (e.g., Vicuna-7B) to compute the initial state s 0 = h 0 ,o 0 , where h 0 denotes decoder-layer attention key–value (KV) pairs and o 0 is the final-layer pre-logit vector. Using unmodified pre-trained pa- rameters ensures s 0 faithfully encodes the model’s native understanding of both neutral and harmful content, preserving core knowledge before safety interventions. Projection Matrix Loading. The null-space projection matrix P , enforcing orthogonality between control signals and neutral knowledge subspaces, is pre-computed offline. Via singular value decomposition (SVD) of the SHP dataset- derived neutral knowledge matrix K 0 , we calculate P = I d − U prot U ⊤ prot (Section IV.B). Loading pre-computed P at inference avoids on-the-fly decomposition, consuming < 0.1% of total inference time on Vicuna-7B and enabling low-latency deployment. Control Signal Optimization. This core step adapts safety alignment via gradient-based optimization and null-space pro- jection. For each token step t, a lightweight 3-layer MLP value function V φ (trained on H-RLHF data) estimates the reward of state s t , guiding gradient ascent (Eq. 10) to optimize control signal u t = u h t ,u o t . The optimization maximizes safety-aligned reward while minimizing ∥u t ∥ 2 to avoid over- perturbation. u t is then projected via P to eliminate overlap with protected subspaces, preserving neutral knowledge. State Transition and Token Generation. The projected signal Pu t updates the system state to s t+1 = h t + Pu h t ,o t + Pu o t , adjusting KVs for coherence and pre-logits to suppress harmful tokens. The next token y t is sampled from softmax(W (o t +Pu o t )), where W denotes fixed LMHead weights. This process repeats until an EOS token is gener- ated or the 200-token limit (consistent across experiments) is reached. This workflow avoids full-model fine-tuning (a key limitation of RLHF) via offline P preprocessing and lightweight V φ . We demonstrate in Section V that this design achieves a favorable balance between safety alignment and knowledge preservation. V. EXPERIMENTS The efficacy and robustness of the proposed DNAlign methodology are rigorously evaluated through a set of com- prehensive experiments designed to address four core research questions (RQs): RQ1 (Safety Alignment). Does the DNAlign intervention effectively reduce the generation of harmful, toxic, or unethical content, achieving superior safety performance compared to state-of-the-art baselines? RQ2 (Utility Preservation). Can DNAlign successfully pre- serve the LLM’s core knowledge and maintain high output quality and fluency when performing harmless or neutral tasks, thereby avoiding the common safety-utility trade-off? 7 RQ3 (Generalization Capability). Does the method demon- strate strong generalization to diverse and novel harmful scenarios, specifically those beyond the specific datasets used for constructing the projection space and training the initial value function? RQ4 (Practical Deployment Efficiency). Is DNAlign com- putationally efficient enough for practical, real-world deploy- ment, and how do the key hyperparameters (e.g., step size, number of iterations) influence the resulting safety and utility performance metrics? To validate the efficacy of our method in achieving a crucial balance between stringent safety alignment and the preservation of core LLM knowledge, we perform comprehen- sive experiments leveraging the detailed design specifications and data from referenced materials. The subsequent sections systematically elaborate on the research workflow, precise ex- perimental setup, and configurations of established baselines. Following this, we present detailed analyses focusing on six critical experimental aspects. Firstly, performance comparison against established safety baselines. Secondly hyperparameter sensitivity studies (focusing on step size and gradient ascent iterations). Thirdly inference time and computational overhead analysis. Fourthly ablation studies investigating the impact of projection-matrix sample size. Rigorous generalization testing across external harmful benchmarks. Last, detailed impact analysis on a diverse suite of harmless datasets. These struc- tured evaluations provide empirical evidence supporting the method’s superiority and practical viability. A. Research Workflow The experiment follows a four-stage sequential workflow to ensure systematic validation: 1) Preparatory Stage. We construct the null-space projec- tion matrix using neutral prompts from SHP and train the value function on H-RLHF under reward-model supervision. 2) Initialization Stage. All backbone models (Vicuna-7B and Qwen-7B) and baseline methods are configured with unified random seeds and hyperparameters to ensure fair and consistent comparison. 3) Inference and Data Collection Stage. Our method and all baseline methods are applied to multiple datasets including H-RLHF, SHP, harmful benchmarks, and harmless benchmarks, to generate outputs and compute the four core evaluation metrics. 4) Result Analysis Stage. Performance across datasets is aggregated to examine the safety–utility trade-off and to support extended studies such as hyperparameter sensitivity, inference-time evaluation, and projection-matrix ablations. B. Experimental Setup Model. We evaluate DNAlign on two base models, Qwen- 7B [48] and Vicuna-7B [33]. We compare DNAlign with several baselines, including the unedited backbone models, Static RE [36], Contrastive Decoding (CD) [46], and RE- Control [34], which represent methods with static or dynamic safety interventions. Datasets. Experiments are conducted on multiple datasets covering both harmful and benign scenarios. H-RLHF and SHP are used as mixed benchmarks to evaluate the trade-off between safety alignment and capability preservation under different data distributions. For harmful generalization, we use RealToxicityPrompts, ToxiGen, and Categorical-HarmfulQA. To assess performance under benign conditions, we evaluate on six harmless datasets: OSASST1, Dolly, HC3, Alpaca, ShareGPT, and Reddit ELI5. Baselines. The baselines include: (1) Unedited Backbones without safety intervention; (2) Static RE [36], a static reward- based activation adjustment method; (3) Contrastive Decoding (CD) [46], which dynamically integrates reward signals during generation; and (4) RE-Control [34], a dynamic control-based alignment approach. C. Evaluation Metrics and Tools To systematically assess the performance of DNAlign in bal- ancing LLM safety alignment and core knowledge preserva- tion, we adopt four well-established metrics that jointly cover safety-oriented (RQ1), utility-oriented (RQ2), and practicality- related aspects (RQ3 and RQ4). This selection aligns with prior work in LLM alignment and knowledge retention [46], [47], ensuring that our evaluation results are comparable to existing state-of-the-art methods and grounded in community- recognized standards. Each metric is designed to target a specific aspect of model performance, with clear computa- tional definitions and reliable tools to avoid subjective biases, thus enabling rigorous quantitative analysis of the proposed method’s effectiveness. Average Reward. The average reward metric measures model alignment with human preferences, focusing on safety, helpfulness, and appropriateness core goals of LLM safety alignment. Calculated using UltraRM-13B [40], a reward model trained on large-scale human preference data, it quan- tifies how well model outputs align with these goals. The formula is: ̄ R = 1 M M X i=1 r i , r i = UltraRM-13B(inputs i ),(12) where M is the total number of test samples (1000), inputs i is the model’s response to the i-th prompt, and r i is the unnormalized score from UltraRM-13B. Higher ̄ R values indicate better alignment with human preferences for safe and helpful interactions, reflecting DNAlign’s ability to optimize safety without compromising utility. Coherence. The coherence metric measures the seman- tic similarity between the model’s generated responses and human-preferred reference responses, indicating core knowl- edge preservation. By assessing how well the model retains the semantic content of high-quality reference outputs, it ensures safety interventions do not distort the model’s understanding of neutral knowledge. Coherence is computed using TF-IDF cosine similarity, a method robust to wording variations while capturing core meaning. The formula is: Coherence = e i · r i |e i ||r i | ,(13) 8 TABLE I COMPREHENSIVE PERFORMANCE COMPARISON ON THE H-RLHF AND SHP DATASETS. BackboneDatasetModelDiversityCoherenceAverage RewardComprehensive Quality Vicuna-7B [33] H-RLHF Base0.8672–0.3991–0.1410–5.3600– CD [46]0.8810–0.4081–0.8440↑5.6800↑ Static RE [36]0.8941↑0.3813–0.9130↑5.0500↓ RE-Control [34]0.9007↑0.3887–0.9614↑5.9400↑ DNAlign (Ours)0.9026↑0.4325↑1.2258↑6.1600↑ SHP Base0.8833–0.3762–0.3913–6.3300– CD [46]0.8778–0.3809–0.7830↑6.2300↓ Static RE [36]0.8809–0.3792–0.7831↑5.5250↓ RE-Control [34]0.8820–0.3792–0.7456↑6.5000↑ DNAlign (Ours)0.8818–0.3813–1.0756↑6.9700↑ Qwen-7B [48] H-RLHF Base0.9381–0.4192–0.5519–7.1538– CD [46]0.8680↓0.3929–0.9036↑6.2000↓ Static RE [46]0.9350–0.4168–0.7088↑6.5750↓ RE-Control [34]0.9364–0.4203–0.7218↑7.2500↑ DNAlign (Ours)0.9371–0.4250↑0.9430↑7.3846↑ SHP Base0.9396–0.4126–0.3939–6.6551– CD [46]0.9199↓0.4247↑0.6415↑6.7710↑ Static RE [36]0.9338–0.4179–0.6546↑6.0300↓ RE-Control [34]0.9424↑0.4148–0.6401↑6.6900↑ DNAlign (Ours)0.9402–0.4164–0.9882↑6.8965↑ where e i is the TF-IDF vector of the model’s response for the i-th prompt, and r i is the TF-IDF vector of the human- preferred reference. Both vectors are projected into a 1000- dimensional space, based on the top 1000 frequent terms from the reference corpus. Higher coherence values indicate that the model’s output is semantically close to the reference, ensuring that DNAlign’s safety interventions do not compromise factual accuracy or contextual relevance. Diversity. The diversity metric evaluates the variety in the model’s generated responses, ensuring that safety interventions do not lead to repetitive or overly constrained outputs. We measure diversity using rep-n scores, which quantify the repetition rate of n-grams (2-grams, 3-grams, and 4-grams) in the generated text. Repeated n-grams can make responses feel unnatural, even if they are safe and factual. The formula for diversity is: Diversity = 4 Y n=2 1− rep-n 100 ,(14) where rep-n is the percentage of repeated n-grams in the generated response. For each response, we split the text into token sequences, count n-gram frequencies, and compute the proportion of n-grams with frequency ≥ 2. The final score reflects repetition across 2-gram to 4-gram phrase lengths, with higher values indicating richer, more varied language. Higher diversity scores confirm that DNAlign’s safety mechanisms do not constrain the model’s generation, maintaining natural, engaging responses. Comprehensive Quality Score. The comprehensive quality score provides an overall assessment of model performance by integrating multiple aspects, including helpfulness, accuracy, fluency, and safety. This metric complements the individual metrics by capturing trade-offs between different performance dimensions, such as whether improved safety affects fluency. It uses DeepSeek-R1-Distill-Qwen-32B [39], a state-of-the-art evaluation model validated on diverse LLM benchmarks, to assign scores between 1 and 10 to each response. A score of 1 indicates poor quality (e.g., harmful or inaccurate), while 10 reflects exceptional quality (e.g., safe, accurate, fluent). DeepSeek-R1-Distill-Qwen-32B has a 0.82 correlation with human evaluator scores, making it a reliable tool for capturing nuanced quality differences. Together with the other metrics, this provides a rigorous, objective evaluation framework for assessing DNAlign’s safety alignment and knowledge preser- vation. D. Performance Comparison on H-RLHF and SHP The efficacy of our proposed methodology is compre- hensively validated across two distinct yet complementary human-preference benchmarks: H-RLHF and SHP. This dual-benchmark strategy allows for a robust assessment of two critical research objectives, including RQ1, focusing on safety alignment, and RQ2, pertaining to capability preservation in general-purpose interactions. The H-RLHF dataset, charac- terized by its mixture of harmful and benign prompts, serves as a crucial testbed for evaluating safety-oriented performance. Our proposed approach consistently demonstrates superior safety alignment compared to the RE-Control baseline and other tested systems, across both the Vicuna-7B and Qwen- 7B foundational models. Specifically, with the Vicuna-7B backbone, our method achieves a substantially higher average reward (1.2258 vs. 0.9614 for RE-Control), alongside notably improved coherence (0.4325 vs. 0.3887), and better compre- hensive quality (6.16 vs. 5.94), while maintaining comparable diversity. A similar positive pattern is evident under the Qwen- 7B backbone, where our method significantly increases the 9 Average Reward 0 CoherenceComprehensive QualityDiversity Fig. 4. The influence of step size and epochs at test-time. average reward from 0.7218 to 0.9430 and registers the highest comprehensive quality. These compelling quantitative results collectively demonstrate that a more stringent safety alignment (RQ1) can be effectively realized without compromising the fluency or content richness of the generated responses in mixed harmful–benign scenarios. To rigorously verify capability preservation (RQ) under general-purpose, non-adversarial conditions, we employ the SHP dataset, which predominantly consists of harmless, ev- eryday human-model queries. In this context, our method successfully maintains or actively enhances the core model’s utility. For the Vicuna-7B model, both the average reward (1.0756) and comprehensive quality (6.97) metrics exceed those of the RE-Control baseline, with diversity and coherence remaining stable. The experimental results under the Qwen-7B backbone exhibit the same positive pattern, definitively confirming that the introduced intervention strategy does not erode general- purpose performance. Taken together, the consistent perfor- mance improvements across these two structurally different datasets underscore that the proposed approach offers a bal- anced and synergistic enhancement. It simultaneously elevates both safety-oriented and utility-oriented metrics while dili- gently preserving broad output diversity. This demonstrated robustness, spanning performance and generalizability, sug- gests that the method is highly suitable for real-world deploy- ments where the simultaneous requirement for uncompromis- ing safety and high-fidelity response quality is a necessity. E. Hyperparameter Sensitivity Study To validate the practicality and robustness of our method, we conduct an ablation study analyzing its sensitivity to key hyperparameters. To rigorously assess this, we performed sys- tematic experiments utilizing the Qwen-7B model architecture trained on a distilled subset of the H-RLHF dataset, compris- ing 1000 representative samples, while systematically varying two critical training parameters: the number of training epochs (τ ∈ [10, 100]) and the gradient step size (η ∈ [0.1, 1.0]). The impact of these combinations was quantitatively evaluated based on four primary metrics,including diversity, coherence, comprehensive quality, and average reward. The detailed re- sults (visualized in Figure 4) reveal three distinct trends. First, Diversity remains notably stable, exhibiting a narrow variance range (0.933 to 0.942) unless epochs are below τ = 30 or the step size exceeds η = 0.8. Second, Coherence is 246810 0 20 40 60 80 100 120 140 Inference Time(min) Training Batches Base Re-Control Ours Fig. 5. Variation of inference time across three models with training batches. the most sensitive metric, achieving peak performance with τ ∈ [40, 70] epochs and a moderate step size of η ∈ [0.2, 0.5], but suffering a sharp degradation under insufficient training or overly aggressive policy updates. Third, both Comprehensive Quality and the Average Reward improve with increased training epochs but consistently plateau or begin to decline when the step size is increased beyond η = 0.5. Based on this granular analysis, the optimal configuration is identified as 100 epochs (τ = 100) and a controlled step size of 0.2 (η = 0.2), yielding a highly balanced performance profile with a diversity of 0.9366, coherence of 0.4238, comprehensive quality of 7.24, and the highest observed average reward (R avg = 0.943), thereby ensuring both the reliability and practicality of our proposed method. F. Inference Time Analysis Beyond effectiveness and hyperparameter robustness, we further quantify the computational overhead introduced by our method to evaluate its deployment practicality (RQ4). To assess the practical deployment of the proposed framework, we evaluate inference time using the Qwen-7B backbone with 1,000 samples from the H-RLHF dataset. The experiment compares DNAlign against the RE-Control model and the unedited base model, analyzing the impact of training batches (ranging from 1 to 10) on inference time. Results show that the base model’s inference time decreases as batch size increases, from 51.059 minutes with 1 batch 10 Fig. 6. Research on sample parameters of projection matrix. to 10.556 minutes with 10 batches. The RE-Control model exhibits a similar trend, dropping from 71.518 minutes (1 batch) to 10.617 minutes (10 batches). DNAlign, initially slower (123.734 minutes with 1 batch), also improves as batch size increases, reaching 12.447 minutes with 10 batches, only 1.891 minutes longer than the base model and 1.83 minutes longer than RE-Control.This higher initial overhead is due to DNAlign’s null-space projection module, critical for safety and knowledge preservation. However, the inference time gap narrows significantly with more batches, with DNAlign showing improved efficiency in larger batch sizes due to batch processing optimization. The precomputed projection matrix introduces negligible runtime overhead (less than 0.1% of total inference time), as it is loaded during inference rather than computed on the fly. Importantly, our method maintains high output quality despite the additional inference time. It achieves a coherence score of 0.4250, surpassing both the base model (0.4192) and the RE-Control model (0.4203). This demonstrates that DNAlign successfully balances safety alignment, knowledge retention, and inference efficiency. In conclusion, while the safety mechanisms of our method introduce some initial inference overhead, the gap diminishes as batch size increases, with a manageable difference of 1.891 minutes at 10 batches. Combined with superior performance, DNAlign proves feasible for real-world deployment scenarios requiring both efficiency and quality. G. Research on Sample Parameters of Projection Matrix To justify the design choices underlying the null-space projection module in our framework, we ablate the number of samples used to construct the projection matrix and analyze its impact on numerical stability and approximation error. The selection of 20,000 samples for computing the projection ma- trix with Vicuna-7B is determined as the optimal engineering solution, supported by detailed empirical data and systematic experimental validation. As illustrated by the experimental results, the error exhibits distinct variation patterns across different sample size ranges. When the sample size is within 1 to 100, the model falls into the overfitting region, with errors maintaining at a low level (0.0000035 for 1 sample and 0.00258 for 100 samples) but lacking generalizability due to 0.92 0.93 0.92 0.46 0.460.46 6.73 7.47 7.31 0.34 0.59 0.8 0.95 0.95 0.95 0.6 0.60.6 7.44 7.47 7 0.35 0.8 0.71 0.96 0.96 0.96 8.04 8.03 7.7 0.3 0.55 Base Re-Control Ours RealToxicityPrompts Average Rewarad Comprehensive Quality Coherence Diversity Base Re-Control Ours CategoricalHarmfulQA Base Re-Control Ours ToxiGen Fig. 7.Generalization Performance of DNAlign on Harmful Datasets. Comparison of DNAlign against the base model (Qwen-7B) and RE-Control across three distinct harmful datasets: RealToxicityPrompts, ToxiGen, and Categorical-HarmfulQA. DNAlign achieves superior performance in safety alignment (average reward), notably increasing it by 137.1% on RealTox- icityPrompts and 229.4% on ToxiGen compared to the base model. The results confirm DNAlign’s strong generalization ability, effectively balancing significant safety improvements with competitive diversity and coherence across diverse harmful content distributions. insufficient data diversity. As the sample size increases to the range of 500 to 3,000, a critical numerical instability region emerges, where the error fluctuates drastically. Specifically, the error rises to 0.587 at 500 samples, drops to 0.0415 at 1000 samples, then surges again to 1.757 at 1500 samples and reaches the peak of 2.923 at 3000 samples. This severe instability makes sample sizes in this range unsuitable for projection matrix construction. When the sample size exceeds 3000 and enters the conver- gence region (5000 to 20000 samples), the error stabilizes at a low level and shows minimal fluctuations. At 5000 samples, the error is 0.0105. At 10000 samples, it decreases slightly to 0.00929. At 15000 samples, it rises marginally to 0.00949. And at 20000 samples, it stabilizes at 0.0094. Notably, the 20,000 sample solution not only achieves a stable and low error (0.0094) but also performs comparably to the 50,000 sample variant (error 0.0100) while significantly reducing computational costs. Additionally, the 20,000 sample size far exceeds the model’s hidden dimension (4096), ensuring the projection matrix is built on a complete orthogonal basis that meets the full-rank condition, thus guaranteeing mathematical rigor and excellent generalization ability. This selection also aligns with the sample size used in earlier experiments on Qianwen-7B, enabling fair cross-model comparisons. In summary, the 20,000 sample approach achieves an opti- mal balance across four key dimensions: reliability (avoiding the numerical instability region), efficiency (reducing com- putational costs), performance (maintaining stable low errors in the convergence region), and scientific rigor (supporting cross-model comparisons). It is therefore confirmed as a fully validated and ideal solution for projection matrix computation with Vicuna-7B. 11 DiversityComprehensive Quality CoherenceAverage Reward Fig. 8. Performance on Harmless Datasets and Core Capability Preservation. Evaluation of DNAlign compared to the base model and RE-Control across six diverse harmless datasets, assessing core LLM capabilities. DNAlign successfully maintains competitive diversity (0.8464 to 0.8836) and coherence (0.4077 to 0.5692) scores close to the base model, demonstrating minimal degradation of core capabilities. Simultaneously, DNAlign achieves a superior average reward, outperforming RE-Control and often showing significant safety improvement (up to 81.9% gain in average reward over the base model on specific datasets). This confirms that the null-space projection effectively balances safety alignment and utility. H. Generalization Testing The critical third research question, RQ3, mandates the evaluation of the proposed DNAlign methodology’s general- ization capability, specifically, its performance when facing harmful prompts that lie outside the scope of its original training distribution. This assessment is crucial for validat- ing the robustness required for real-world deployment. To this end, we benchmark DNAlign against the base model (Qwen-7B) and the RE-Control baseline across three dis- tinct external harmful datasets: RealToxicityPrompts, Toxi- Gen, and CategoricalHarmfulQA. These datasets represent diverse facets of safety risks, ranging from general toxicity to highly specific categorical harms. Our evaluation employs four rigorous metrics: diversity, coherence, comprehensive quality, and, critically, average reward for safety alignment. Across all three generalization datasets, DNAlign consistently demonstrates superior performance, particularly in fulfilling the stringent requirements of RQ3. For example, on the Real- ToxicityPrompts benchmark, which tests responses to model- generated toxic seeds, DNAlign achieves an average reward of 0.7999. This represents a dramatic improvement, showcasing a 137.1% increase over the base Qwen-7B model’s score of 0.3374. Importantly, this substantial safety gain is realized while maintaining highly competitive generative metrics, with diversity at 0.9241 and coherence at 0.4648, thereby mitigating the typical trade-off between safety and utility. The effectiveness of DNAlign is further substantiated when tested against the extreme toxicity profiles in the ToxiGen dataset. Here, DNAlign’s average reward rises to 0.5465, an extraordinary 229.4% increase relative to the base model. This significant leap underscores the method’s exceptional capability in neutralizing high-risk, targeted harmful content. Even in this high-toxicity context, the fundamental quality of the generated text remains robust, evidenced by a coherence score of 0.2251 and a high diversity score of 0.9615. While acknowledging that DNAlign’s comprehensive quality score may be marginally lower than that of the unedited backbone on some specific datasets, a minor consequence of its strin- gent safety filtering, the overwhelming and consistent gains observed in the safety alignment metric (average reward) con- firm its superior overall contribution. These collective results confirm that the DNAlign approach successfully maintains strong and reliable generalization across a wide spectrum of diverse harmful content. By demonstrating enhanced safety alignment without compromising the model’s core generative capabilities, DNAlign proves its robustness and its capacity to successfully balance the dual requirements of safety and utility, making it exceptionally well-suited for high-stakes, real-world deployment scenarios. I. Impact Analysis on Harmless Datasets To complete our comprehensive analysis of RQ 2 , we rigor- ously investigated the effect of DNAlign on purely harmless datasets. This is essential to confirm that the safety intervention does not degrade core capabilities necessary for benign tasks. Experiments were conducted on six diverse harmless datasets, comparing DNAlign against the base model (Qwen-7B) and the RE-Control baseline using four core metrics: diversity, coherence, average reward, and comprehensive quality. For experimental consistency, DNAlign was optimized with a step size of 0.2 and 100 gradient ascent iterations. The proposed method consistently performed at a high level across all six benign datasets, crucially maintaining performance metrics that were closely competitive with the unedited base model, while simultaneously surpassing RE-Control in key utility and safety metrics. Notably, DNAlign’s diversity scores remained robust, ranging from 0.8464 to 0.8836, and coherence scores spanned 0.4077 to 0.5692, reflecting strong, preserved output consistency. Furthermore, DNAlign’s average reward (varying from 0.3706 to 0.9210) showed significant improvements over the base model and consistently outperformed the RE-Control method in the vast majority of scenarios.While acknowl- edging that DNAlign’s comprehensive quality scores were marginally lower than the base model in some instances—a minor consequence of its stringent safety filtering—it reli- ably and consistently outperformed the RE-Control baseline, demonstrating its superior ability to enhance safety without sacrificing core capabilities. For example, in utility-focused datasets such as Reddit ELI5 and Dolly, DNAlign exhibited 12 superior average reward (achieving up to 81.9 improvement over the base model), all while sustaining high coherence and diversity scores. Overall, the observed stability of our method across all core metrics confirms the effectiveness of its underlying null-space projection mechanism. This mechanism successfully isolates safety-related interventions to specific subspaces orthogonal to the model’s neutral knowledge and utility functions, ensuring minimal disruption to the core capabilities. Consequently, DNAlign successfully strikes an ef- fective and reliable balance between stringent safety alignment (RQ1) and core utility preservation (RQ2), positioning it as a robust solution for safe and effective LLM deployment in real- world applications. J. Summary and Key Findings The comprehensive experiments confirm DNAlign’s efficacy and robustness across all four research questions (RQ1-RQ4). For Safety Alignment (RQ1), DNAlign consistently achieved the highest Average Reward and Comprehensive Quality on the H-RLHF benchmark (e.g., 1.2258 with Vicuna-7B), sur- passing all baselines. For Utility Preservation (RQ2), Diversity and Coherence remained competitive with the base model on all harmless datasets, validating the null-space projection mechanism and demonstrating superior Average Reward (up to 81.9% gain) over baselines in benign scenarios.Crucially, DNAlign exhibited strong Generalization (RQ3), achieving an extraordinary 229.4 increase in Average Reward over the base model on the high-toxicity ToxiGen dataset. Addressing Practical Efficiency (RQ4), the minimal inference overhead be- came negligible at larger batch sizes. In conclusion, DNAlign provides a robust and reliable framework that effectively balances stringent safety alignment with high-fidelity utility preservation, positioning it as an ideal solution for safety- critical LLM deployments. VI. CONCLUSION DNAlign is a framework that effectively balances LLM safety alignment with core knowledge preservation, addressing the trade-off seen in existing methods. By integrating dynamic control theory and null-space projection, the proposed frame- work ensures targeted safety interventions without disrupting the model’s inherent capabilities. Experimental results demon- strate the superiority of our method across multiple datasets, outperforming baseline models like RE-Control in key met- rics such as average reward, coherence, and comprehensive quality. Taken together, these findings provide affirmative answers to our four guiding research questions on safety, utility preservation, generalization, and practicality, positioning our framework as a promising solution for reliable and high- quality LLM deployment. REFERENCES [1] D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, C. Kang, T. Xu, Y. Sung, Z. Li, J. Luketina, A. Miller, et al. “Towards a Human-like Open- Domain Chatbot,” arXiv preprint arXiv:2001.09977, 2020. [2] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?,” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, p. 610–623, 2021. [3] T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai. “Man is to Computer Programmer as Woman is to Homemaker? Debi- asing Word Embeddings,” Advances in Neural Information Processing Systems, p. 4349–4357, 2016. [4] X.-Y. Chen, L. Liu, B. Qin, X. Ye, H. Liu, J. Xue, C. Xiao, H. Wang, X. He, Y. Cao, et al. “FinGPT: Open-Source Financial Large Language Models,” arXiv preprint arXiv:2306.06031, 2023. [5] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems 30 (NeurIPS 2017), p. 4302– 4310, 2017. [6] O. Evans, L. Finnveden, L. Finnveden, A. Balwit, A. Balwit, A. Balwit, A. Balwit, A. Balwit, A. Balwit, A. Balwit, et al. “Truthful AI: Developing and governing AI that does not lie,” arXiv preprint arXiv:2110.06674, 2021. [7] H. Fang, Z. Jiang, Z. Wang, P. Li, Z. Wu, and W. Y. Wang. “AL- PHAEDIT: NULL-SPACE CONSTRAINED KNOWLEDGE EDITING FOR LANGUAGE MODELS,” in Proceedings of the 12th International Conference on Learning Representations (ICLR 2025), 2025. [8] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. “Re- alToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models,” arXiv preprint arXiv:2009.11462, 2020. [9] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. “The Curious Case of Neural Text Degeneration,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 3028– 3040, 2020. [10] N. Kandpal, E. Wallace, and C. Raffel. “Deduplicating Training Data Mitigates Privacy Risks in Language Models,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), p. 26834–26848, 2022. [11] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. “Overcoming catastrophic forgetting in neural networks,” in Proceedings of the national academy of sciences, vol. 114, no. 13, p. 3521–3526, 2017. [12] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. “Prompt Engineering for Large Language Models: A Survey,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11324–11345, 2023. [13] H. Liu, S. Di, and L. Chen. “Incremental tabular learning on heteroge- neous feature space,” Proceedings of the ACM on Management of Data, vol. 1, no. 1, p. 1–18, 2023. [14] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. “Locating and Editing Factual Knowledge in GPT,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), p. 17267–17280, 2022. [15] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. “Mass-Editing Memory in a Transformer,” in Proceedings of the 40th International Conference on Machine Learning (ICML 2023), p. 21225–21244, 2023.p [16] E. Mitchell, C. Lin, O. Evans, L. Finnveden, A. Güne ̧s Baydin, L. Finnveden, P. W. Koh, E. Kreiss, and P. Liang. “Locating and Editing Factual Associations in GPT,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), p. 17281–17293, 2022. [17] L. Ouyang, J. Wu, X. Jiang, C. L. Almeida, P. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, B. Slama, A. Ray, et al. “Training language models to follow instructions with human feedback,” arXiv preprint arXiv:2203.02155, 2022. [18] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. Finn, and S. Levine. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” in Advances in Neural Information Processing Systems 36 (NeurIPS 2023), p. 35827–35840, 2023. [19] P. Rajpurkar, A. Y. Hannun, M. Haghpanahi, C. Bourn, and A. Y. Ng. “Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks,” Nature Medicine, vol. 24, no. 11, p. 1683–1688, 2018. [20] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. “iCaRL: Incremental Classifier and Representation Learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 2001–2010, 2017. [21] E. Sheng, B. Grosz, and D. Wang. “Towards Understanding and Mit- igating Social Biases in Language Models,” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, p. 1174–1184, 2021. [22] E. J. Topol. “High-performance medicine: the convergence of human and artificial intelligence,” Nature Medicine, vol. 25, no. 1, p. 44–56, 2019. [23] L. Weidinger, C. Martens, A. Bales, A. Güne ̧s Baydin, G. Borges, D. Doran, L. Finnveden, L. Gao, S. Gehman, C. Hallinan, et al. “Truthful 13 AI: Developing and governing AI that does not lie,” arXiv preprint arXiv:2110.06674, 2021. [24] Y. Wang, Z. Li, P. Liu, and G. Neubig. “Editing and Aligning: A Unified Framework for Controllable Text Generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 12345–12358, 2023. [25] Z. Xu, Y. Wang, Y. Zhang, M. Li, and Z. Hu. “Parameter Conflicts in Large Language Models: Implications for Editing and Alignment,” in Proceedings of the 41st International Conference on Machine Learning (ICML 2024), p. 35678–35690, 2024. [26] X. Zhou, Y. Liu, M. Sun, and B. Y. Lin. “Factually Aligned Language Generation via Prompting,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 4567–4580, 2024. [27] S. Dhulia, E. Wallace, and C. Raffel. “Limits of Prompting for Knowl- edge Update in Large Language Models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 9876–9889, 2023. [28] D. Ganguli, P. W. Koh, R. Bommasani, A. Chowdhery, R. Fedus, P. Fung, T. Hashimoto, H. Hajishirzi, M. Shoeybi, S. Singh, et al. “Emergent abilities of large language models,” TMLR, vol. 3, no. 12, p. 2008–2042, 2022. [29] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” North American Chapter of the Association for Computational Linguistics, p. 4171–4186, 2019. [30] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. “Improving Language Understanding by Generative Pre-Training,” 2018. [31] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung. “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, p. 1–38, 2023. [32] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier. “A Survey of Reinforcement Learning from Human Feedback,” arXiv preprint arXiv:2312.14925, 2024. [33] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, p. 46595–46623, 2023. [34] L. Kong, H. Wang, W. Mu, Y. Du, Y. Zhuang, Y. Zhou, Y. Song, R. Zhang, K. Wang, and C. Zhang. “Aligning large language models with representation editing: A control perspective,” Advances in Neural Information Processing Systems, vol. 37, p. 37356–37384, 2024. [35] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. “Vicuna: An Open- Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,” Online available: https://vicuna.lmsys.org, Accessed: 2025-07-31, 2023. [36] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. “Inference- time intervention: Eliciting truthful answers from a language model,” Advances in Neural Information Processing Systems, vol. 36, p. 41451– 41530, 2023. [37] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022. [38] R. Bhardwaj, D. D. Anh, and S. Poria. “Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,” arXiv preprint arXiv:2402.11746, 2024. [39] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. [40] G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, et al. “Ultrafeedback: Boosting language models with scaled ai feedback,” arXiv preprint arXiv:2310.01377, 2023. [41] J. Fang, H. Jiang, K. Wang, Y. Ma, S. Jie, X. Wang, X. He, and T.-S. Chua. “Alphaedit: Null-space constrained knowledge editing for language models,” arXiv preprint arXiv:2410.02355, 2024. [42] J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn. “The pushshift reddit dataset,” in Proceedings of the international AAAI conference on web and social media, vol. 14, p. 830–839, 2020. [43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017. [44] K. Kanatani. “Singular Value Decomposition,” in Computer Vision: A Reference Guide, edited by K. Ikeuchi, Springer International Pub- lishing, Cham, p. 1174–1177, 2021. ISBN: 978-3-030-63416-2, DOI: 10.1007/978-3-030-63416-2-802, Online available: https://doi.org/10. 1007/978-3-030-63416-2_802. [45] Gemini Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. [46] M. Khanov, J. Burapacheep, and Y. Li. “Args: Alignment as reward- guided search,” arXiv preprint arXiv:2402.01694, 2024. [47] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. “Language models are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, p. 1877–1901, 2020. [48] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. “Qwen Technical Report,” arXiv preprint arXiv:2309.16609, 2023. AUTHOR BIOGRAPHY Jisheng Dang received the Ph.D. degree from Sun Yat-sen University, China, in 2025, advised by Prof. Jianhuang Lai and Prof. Huicheng Zheng. He worked as a research fellow at the NExT++ laboratory of the National University of Singapore, advised by Prof. Tat-Seng Chua. He is now a tenured associate professor at the School of Information Science and Engineering, Lanzhou University. His research interests include multimodal learning, video understanding, and embodied intelligence. He has published several papers as the first author in major journals and conferences including IEEE TIP, TNNLS, TITS, IJCAI, and AAAI. Junfeng Fang is currently a postdoctoral researcher at the National University of Singapore (NUS). His research focuses on trustworthy large language models, including LLM safety and LLM-based agent systems. He has published over 30 papers in top- tier conferences and journals such as IEEE TPAMI, ICLR, NeurIPS, and ICML. He can be contacted at fangjf@nus.edu.sg. Bin Hu (Fellow, IEEE) received the PhD degree in computer science from the Institute of Computing Technology, Chinese Academy of Science, China, in 1998. Since 2008, he has been a professor and the dean of the School of Information Science and Engineering, Lanzhou University, China. He is a Fellow of IET and AAIA, and Chair of Techni- cal Committee on Computational Psychophysiology, IEEE SMC. Tat-Seng Chua received the Ph.D. degree from the University of Leeds, U.K. He is the KITHCT chair professor with the School of Computing, National University of Singapore. He is the co-director of NExT, a joint center between NUS and Tsinghua University. He is the 2015 winner of the prestigious ACM SIGMM Award and a Fellow of the Singapore Academy of Sciences. He is the co-founder of two technology startups in Singapore and a Fellow of the Singapore Academy of Sciences, with 106,330 citations on Google Scholar.

Instant research discovery