Paper deep dive
Energy-Driven Steering: Reducing False Refusals in Large Language Models
Eric Hanchen Jiang, Weixuan Ou, Run Liu, Shengyuan Pang, Guancheng Wan, Ranjie Duan, Wei Dong, Kai-Wei Chang, XiaoFeng Wang, Ying Nian Wu, Xinfeng Li
Models: Llama-2-7B, Llama-3.1-8B-Instruct, Qwen3
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 12:35:47 AM
Summary
Energy Landscape Steering (ELS) is a fine-tuning-free framework that reduces false refusals in Large Language Models (LLMs) by using an external Energy-Based Model (EBM) to steer internal hidden state activations during inference. By training the EBM to assign high energy to undesirable states (false refusals/jailbreaks) and low energy to desirable states, ELS uses gradient-based intervention to guide the model toward helpful, safe responses without modifying its core parameters.
Entities (4)
Relation Signals (3)
Energy Landscape Steering → utilizes → Energy-Based Model
confidence 100% · We introduce ELS, a novel fine-tuning free framework that leverages a lightweight, externally trained Energy-Based Model (EBM)
Energy Landscape Steering → improves → ORB-H
confidence 95% · raising compliance on the ORB-H benchmark from 57.3 percent to 82.6 percent
Energy-Based Model → steers → Llama-3.1-8B-Instruct
confidence 90% · We further validate the general effectiveness of ELS by evaluating it on a wide range of models, including... Llama-3.1-8B-Instruct
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Safety alignment of large language models currently faces a central challenge: existing alignment techniques often prioritize mitigating responses to harmful prompts at the expense of overcautious behavior, leading models to incorrectly refuse benign requests. A key goal of safe alignment is therefore to improve safety while simultaneously minimizing false refusals. In this work, we introduce Energy Landscape Steering (ELS), a novel, fine-tuning free framework designed to resolve this challenge through dynamic, inference-time intervention. We train a lightweight external Energy-Based Model (EBM) to assign high energy to undesirable states (false refusal or jailbreak) and low energy to desirable states (helpful response or safe reject). During inference, the EBM maps the LLM's internal activations to an energy landscape, and we use the gradient of the energy function to steer the hidden states toward low-energy regions in real time. This dynamically guides the model toward desirable behavior without modifying its parameters. By decoupling behavioral control from the model's core knowledge, ELS provides a flexible and computationally efficient solution. Extensive experiments across diverse models demonstrate its effectiveness, raising compliance on the ORB-H benchmark from 57.3 percent to 82.6 percent while maintaining baseline safety performance. Our work establishes a promising paradigm for building LLMs that simultaneously achieve high safety and low false refusal rates.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
90,093 characters extracted from source content.
Expand or collapse full text
Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energy Eric Hanchen Jiang1*, Weixuan Ou2*, Run Liu3, Shengyuan Pang2, Guancheng Wan1, Ranjie Duan4, Wei Dong5, Kai-Wei Chang1, XiaoFeng Wang5, Ying Nian Wu1†, Xinfeng Li5† 1UCLA, 2Alibaba Cloud Computing, 3SJTU, 4Alibaba Group, 5NTU Abstract Safety alignment of large language models currently faces a central challenge: existing alignment techniques often prioritize mitigating responses to harmful prompts at the expense of overcautious behavior, leading models to incorrectly refuse benign requests. A key goal of safe alignment is therefore to improve safety while simultaneously minimizing false refusals. In this work, we introduce Energy Landscape Steering (ELS), a novel, fine-tuning free framework designed to resolve this challenge through dynamic, inference-time intervention. We trained a lightweight, external Energy-Based Model (EBM) to assign high energy to undesirable (false refusal or jailbreak) states and low energy to desirable (helpful response or safe reject) ones. During inference, the EBM maps the LLM’s internal activations to an energy landscape, and we use the gradient of the energy function to steer the hidden states toward low-energy regions in real time. This dynamically guides the model toward desirable behavior without modifying its parameters. By decoupling behavioral control from the model’s core knowledge, ELS provides a flexible and computationally efficient solution. Extensive experiments across diverse models demonstrate its effectiveness: raising compliance on the ORB-H benchmark from 57.3% to 82.6% while maintaining the baseline safety performance. Our work establishes a promising paradigm for building LLMs that simultaneously achieve high safety and low false refusal rates. Our code is available here. Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energy Eric Hanchen Jiang1*, Weixuan Ou2*, Run Liu3, Shengyuan Pang2, Guancheng Wan1, Ranjie Duan4, Wei Dong5, Kai-Wei Chang1, XiaoFeng Wang5, Ying Nian Wu1†, Xinfeng Li5† 1UCLA, 2Alibaba Cloud Computing, 3SJTU, 4Alibaba Group, 5NTU †footnotetext: *Equal contribution. Correspondence: ywu@stat.ucla.edu, xinfengli@ntu.edu.sg 1 Introduction The alignment of large language models (LLMs) with human safety remains a central challenge in artificial intelligence research (Bianchi et al., 2023; Anwar et al., 2024; Xu et al., 2020; Röttger et al., 2020; Sun et al., 2021; Vidgen et al., 2023). Common approaches such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), system prompt engineering, and vector ablation have proven effective. However, these methods often introduce an unintended trade-off: they can lead either to excessive refusal (over-rejection) or to lapses in safety. This behavior is not merely an inconvenience; it severely undermines model utility and reliability in critical domains. For instance, in a healthcare context, a false refusal could block a legitimate query like “How do I treat a burn?”, while in education it might prevent a student from researching “Explain suicide in literature” (Röttger et al., 2023). Such failures erode user trust and can withhold essential information, making the mitigation of false refusals a pressing issue. Figure 1: Comparison of existing LLM alignment strategies. (1) Fine-tuning methods (e.g., SFT, RLHF) modify parameters but suffer from high compute costs, long training times, and poor generalization. (2) Fine-tuning free methods (e.g., promp-driven, output filtering, activation steering) avoid retraining yet lack precision and effective steering capability. Energy Landscape Steering, offers the combined advantages of deployment flexibility, precise discrimination, and effective steering, compared with fine-tuning and fine-tuning free methods. Current approaches to this problem fall into two main categories, as illustrated in Figure 1. Fine-tuning methods (Ouyang et al., 2022; Ziegler et al., 2019) modify the model’s parameters directly, but this process is computationally expensive, time-consuming, and often struggles to generalize to diverse contexts. A more flexible alternative is fine-tuning free methods (Zheng et al., 2024; Wang et al., 2024), which operate during inference without modifying model weights. Yet, existing techniques in this class, like vector ablation, often lack the precision to reliably distinguish between justified refusals of harmful prompts and false refusals of benign ones. This insufficient discrimination reduces model utility and reliability due to false refusals. To address these limitations, we introduce Energy Landscape Steering (ELS), a novel, fine-tuning free framework that resolves the tension between safety and helpfulness through dynamic, inference-time intervention. Our core idea is to interpret the LLM’s internal state through the lens of an energy landscape. We deploy a lightweight, external EBM (LeCun et al., 2006) that learns to assign a scalar “energy” value to the LLM’s hidden activations. This EBM is trained via contrastive learning to create an energy landscape where trajectories leading to undesirable outputs (like false refusals) have high energy, while trajectories for desirable, helpful responses have low energy. This energy landscape enables precise discrimination between desirable and undesirable outputs. By performing gradient-based steering on this landscape during inference, ELS can effectively redirect hidden activations that would otherwise lead to false refusals toward low-energy regions without perturbing other originally desirable activations. The modified activation state guides the model to produce desirable outputs. For general capability prompts, the model’s activation trajectories lie in low-energy regions of the learned landscape. The gradient-based steering induces only negligible perturbations, leaving the model’s performance on general tasks unaffected. The model therefore responds normally to such prompts. This mechanism ensures safety, significantly reduces false refusals, and preserves helpfulness. In our experiments, ELS consistently outperforms other fine-tuning free methods on false refusal benchmarks. While other methods often degrade performance on safety benchmarks, ELS maintains the baseline safety performance. We further validate the general effectiveness of ELS by evaluating it on a wide range of models, including Llama2-7B-Chat (Touvron et al., 2023), Llama-3.1-8B-Instruct (Dubey et al., 2024), and the Qwen3 series (Yang et al., 2025). These results show that ELS can robustly reduce false refusals without compromising model safety. Our contributions are as follows: ❶ We introduce ELS, a novel fine-tuning free framework that leverages a lightweight, externally trained Energy-Based Model (EBM) to dynamically steer the internal activations of an LLM during inference. In contrast to prior methods that rely on static, coarse-grained interventions, ELS constructs an energy landscape over the activation space. This formulation affords it superior discriminative power, enabling fine-grained steering that effectively preserves robust safety while significantly reducing false refusals. ❷ We conduct extensive experiments on a wide range of models, including Llama2-7B-Chat, Llama-3.1-8B-Instruct, and the Qwen3 series. The results confirm that ELS outperforms existing methods on various benchmarks, achieving a significant reduction in false refusal rates while robustly preserving safety alignment. 2 Related Works Fine-tuning methods adapt pre-trained LLMs via parameter updates: SFT uses labeled data; RLHF integrates human preferences via reward modeling and policy optimization (e.g., PPO (Schulman et al., 2017), DPO (Rafailov et al., 2023), and variants (Ethayarajh et al., 2024; Li et al., 2025)). Safety-aligned variants include H-RLHF (Bai et al., 2022a) and Safe-RLHF (Dai et al., 2023), both using SFT followed by PPO; and Self-Play (Liu et al., 2025), an online self-play RL framework where an attacker LM generates evolving adversarial prompts and a defender LM learns via PPO to resist them. These methods suffer from high computational cost, long training time, and poor adaptability, with full retraining necessary whenever new safety alignment requirements arise. Fine-tuning free Methods achieve safety alignment without altering the model parameters. Representative Fine-tuning free methods can be divided into three categories: (1) Context Engineering: These methods steer outputs toward safety via tailored prompts. Red-Teaming + Shielding (Perez et al., 2022) detects vulnerabilities and prepends defensive prompts to block unsafe generation; Constitutional AI (0-shot) (Bai et al., 2022b) uses safety principles to prompt self-critique and revision during inference. However, their efficacy degrades in long conversations, where initial instructions dilute, and against subtle adversarial inputs designed to bypass rule-based prompting. (2) Content Filtering: These methods block unsafe inputs or outputs: PDS (Zheng et al., 2024) enforces safety via input/output guardrails; SafeDecoding (Xu et al., 2021) uses safety classifiers to suppress unsafe tokens during autoregressive generation. Yet their effectiveness hinges on filter capability, which often falls short against the diverse unsafe outputs of powerful LLMs, such as Caesar-encoded harmful content that is difficult to detect. (3) Activation Steering: These methods manipulate internal activations at inference: SCAS (Cao et al., 2024) steers activations to reduce over-refusal without compromising safety; Surgical (Wang et al., 2024) identifies and ablates refusal-related directions in hidden states to mitigate unnecessary refusals. Both require manually crafted positive–negative pairs (e.g., how to kill a person versus how to kill a Python process), limiting scalability and generalizability. Furthermore, their reliance on a single global steering vector for all inputs undermines effectiveness on diverse inputs. Our method as a fine-tuning free approach, avoids the excessive computing power cost, high training time cost and limited generalization flexibility of fine-tuning methods. By leveraging Real-time Gradient-Based Steering with EBM, our method addresses the limitations of fine-tuning free methods. It achieves a superior discriminative capability which helps to effectively correct model’s behavior to reduce the problem of false refusals. 3 Preliminaries An auto-regressive LLM generates a sequence of tokens Y=(y1,y2,…,yT)Y=(y_1,y_2,…,y_T) by modeling the probability of the sequence given a prompt X: P(Y|X;ϕ)=∏t=1Tp(yt|Y<t,X;ϕ)P(Y|X;φ)= _t=1^Tp(y_t|Y_<t,X;φ) (1) where ϕφ denotes the parameters of the LLM. This process can be conceptualized as navigating a trajectory through the model’s high-dimensional hidden state space. Let ht∈ℝdh_t ^d represent the hidden state of a target layer in the LLM after processing the t-th token. This state is the basis for predicting the next token yt+1y_t+1 via the model’s language modeling head, WLMW_LM: p(yt+1|Y<t,X;ϕ)=softmax(WLMht)p(y_t+1|Y_<t,X;φ)=softmax(W_LMh_t) (2) Our primary objective is to gain real-time control over the trajectory of hidden states =(h1,…,hT)T=(h_1,…,h_T) to steer it away from regions in the state space associated with undesirable behaviors like false refusals. We formalize this by leveraging an Energy-Based Model (EBM), which defines an energy function over the hidden state space. The steering task is to find a modification function M such that for a given state hth_t, the modified state ht′=M(ht)h _t=M(h_t) satisfies: Eθ(ht′)<Eθ(ht)E_θ(h _t)<E_θ(h_t) (3) As we establish in Section E.3, this energy minimization is equivalent to maximizing the probability that the state belongs to a desirable trajectory. Figure 2: Overview of the Energy Landscape Steering framework. The method involves (1) gathering ’good’ and ’bad’ hidden state activations from a base LLM , (2) training an Energy-Based Model (EBM) to create an energy landscape that separates them , and (3) using this EBM to perform real-time, gradient-based steering to guide the model away from refusal-prone states during inference. 4 Methodology Our methodology for achieving this objective unfolds in three distinct phases as illustrated in Figure 2: (1) Data Collection, (2) EBM Training, and (3) Real-time Gradient-Based Steering. 4.1 Phase 1: Activation Data Collection The foundation of our approach is a carefully curated dataset that maps LLM hidden states to nuanced behavioral outcomes. The process begins with a diverse corpus of prompts, P, containing both benign and harmful requests. For each prompt X∈X , we first generate a response Y from the frozen, base LLM. The core of our data collection is a context-aware classification of the LLM’s behavior. We define a heuristic-based classifier, C(X,Y)C(X,Y), that evaluates the appropriateness of the response Y given the nature of the prompt X. This results in a label l indicating whether the behavior is desirable (Compliant) or undesirable (Refusal). C(X,Y)→l∈Compliant,RefusalC(X,Y)→ l∈\Compliant,Refusal\ (4) Specifically, the classification follows a nuanced logic: compliant responses to benign prompts are desirable, but so are refusals to harmful prompts. Conversely, refusals to benign prompts (false refusals) are undesirable, as are compliant responses to harmful prompts (jailbreaks). Concurrently, for each generated token yt∈Yy_t∈ Y, we extract and store the corresponding hidden state hth_t from one or more layers of the LLM. This process populates two distinct sets of hidden states based on the classification outcome: good=ht∣∃(X,Y) _good= \h_t ∃(X,Y) s.t.(X benign∧C(X,Y)="Compliant") s.t.(X benign C(X,Y)="Compliant") ∨(X harmful∧C(X,Y)="Refusal") \;(X harmful C(X,Y)="Refusal") ∧ht is from Y \;h_t is from Y \ (5) bad=ht∣∃(X,Y) _bad= \h_t ∃(X,Y) s.t.(X benign∧C(X,Y)="Refusal") s.t.(X benign C(X,Y)="Refusal") ∨(X harmful∧C(X,Y)="Compliant") \;(X harmful C(X,Y)="Compliant") ∧ht is from Y \;h_t is from Y \ (6) The set badD_bad contains hidden states from contextually inappropriate trajectories (i.e., false refusals to benign prompts and compliant responses to harmful prompts), while goodD_good contains states from contextually appropriate trajectories (i.e., helpful responses to benign prompts and refusals to harmful prompts). This context-aware data separation is crucial for training an EBM that can distinguish between justified and unjustified refusals. The classifier C(X,Y)C(X,Y) is implemented using substring matching against a curated list of refusal indicators (e.g., “I cannot,” “I’m sorry,” “As an AI”), following the methodology of JailbreakBench (Chao et al., 2024). Although labels are assigned at the response level, activation layers are extracted at every token position within each response. As a result, the energy-based model (EBM) implicitly learns token-level energy patterns from response-level supervision. 4.2 Phase 2: EBM Training Energy-Based Model Formulation. Central to our approach is the concept of an Energy-Based Model (EBM), which is characterized by an energy function Eθ:ℋ→ℝE_θ:H that maps a hidden state h∈ℋ=ℝdh =R^d to a scalar energy value. A full theoretical treatment is provided in Section E. We implement this function as a deep multi-layer perceptron (MLP) with the general form: i=fi(i−1)for i=1,…,L(with 0=h) _i=f_i(z_i-1) i=1,…,L (with z_0=h) (7) Eθ(h)=L+1L+bL+1 E_θ(h)=W_L+1z_L+b_L+1 (8) where each function fif_i represents a layer transformation (e.g., linear projection, activation, normalization). This architecture creates a conceptual “landscape” over the LLM’s hidden state space. Training Objective. The EBM is trained to shape this energy landscape using the InfoNCE contrastive loss, separating the states collected in Phase 1. The objective is to assign high energy to “bad” states from badD_bad and low energy to “good” states from goodD_good. For an anchor state h+∈goodh^+ _good and a set of N negative samples hi−i=1N⊂bad\h^-_i\_i=1^N _bad, the loss is: ℒ(θ)=−log(exp(−Eθ(h+)/τ)/[exp(−Eθ(h+)/τ)+∑i=1Nexp(−Eθ(hi−)/τ)]) splitL(θ)=- (& \! (-E_θ(h^+)/τ ) /\\ & [\, \! (-E_θ(h^+)/τ )\\ &+ _i=1^N \! (-E_θ(h^-_i)/τ ) ] ) split (9) Here, τ is a temperature hyperparameter. Minimizing this loss forces Eθ(hgood)≪Eθ(hbad)E_θ(h_good) E_θ(h_bad), effectively creating a classifier that can distinguish between desirable and undesirable trajectories. A formal proof is provided in Lemma E.1. Multi-Layer EBM Training Strategy. Our approach trains individual EBMs for multiple layers of the LLM simultaneously. For each target layer l∈0,1,…,L−1l∈\0,1,…,L-1\, we train a separate EBM, Eθl(hl)E_ _l(h_l), where hlh_l are the hidden states from that layer. Each model EθlE_ _l is trained independently using the same InfoNCE objective. After training, we evaluate each EBM’s performance on a validation set and select the best-performing models for intervention during inference. 4.3 Phase 3: Real-time Gradient-Based Steering The final phase of our methodology involves integrating the trained EBMs into the LLM’s inference process to actively steer its generative trajectory. This is achieved through a real-time, gradient-based intervention on the model’s hidden states. Steering Mechanism. The modification function M(ht)M(h_t) introduced in our objective is realized via gradient descent on the energy surface defined by a trained EBM. For each selected intervention layer l, the hidden state ht(l)h_t^(l) is updated as follows: ht′(l)=ht(l)−η⋅∇hEθl(h)|h=ht(l)h_t (l)=h_t^(l)-η· _hE_ _l(h)|_h=h_t^(l) (10) where η is the steering coefficient, a hyperparameter that controls the strength of the intervention. The term ∇hEθl(h) _hE_ _l(h) is the gradient of the energy function with respect to the hidden state, which points in the direction of the steepest ascent on the energy landscape. By moving the hidden state in the negative gradient direction, we are performing a single step of gradient descent to find a state with lower energy. This update rule is formally proven to minimize energy in Theorem E.1. Impact on Generation. The modification of the hidden state ht′(l)h_t (l) has a direct and immediate impact on the LLM’s output. The original probability distribution over the vocabulary is computed from the original hidden state ht(l)h_t^(l) (Equation 2). After steering, the modified hidden state ht′(l)h_t (l) is passed to the language modeling head, resulting in a new, steered probability distribution: psteered′(yt+1|Y<t,X;ϕ)=softmax(WLMht′(l))p _steered(y_t+1|Y_<t,X;φ)=softmax(W_LMh_t (l)) (11) Let Δht(l)=ht′(l)−ht(l)=−η∇hEθl h_t^(l)=h_t (l)-h_t^(l)=-η _hE_ _l. The change in the logits (the input to the softmax function) can be approximated by a first-order Taylor expansion: Logits′≈Logits+WLMΔht(l)=WLMht(l)−ηWLM∇hEθl splitLogits & +W_LM h_t^(l)\\ &=W_LMh_t^(l)-η W_LM _hE_ _l split (12) This equation explicitly shows how the steering process adjusts the logits, effectively up-weighting tokens that are more likely to lead to contextually appropriate (low-energy) continuations, and down-weighting tokens associated with contextually inappropriate (high-energy) paths. This steering process is applied at every generation step for each selected layer, creating a continuous feedback loop that actively guides the generation trajectory away from refusal-prone regions without requiring any fine-tuning of the LLM’s weights ϕφ. This impact is mathematically explained in Corollary E.1 !10 Model/Method Safety False Refusal General Capability !10 JBB CR ↓ Harmful CR ↓ ORB-H CR ↑ XSTest-S(H) CR ↑ OKTest CR ↑ MMLU Acc ↑ ARC-C Acc ↑ MATH Acc ↑ Llama3.1-8B-Inst 10.0▲ 0.0 10.7▲ 0.0 57.3▲ 0.0 85.2▲ 0.0 98.6▲ 0.0 68.1▲ 0.0 72.4▲ 0.0 31.8▲ 0.0 w/ system prompt 3.0▲ 7.0 2.3▲ 8.4 41.0▼ 16.3 37.6▼ 47.6 53.1▼ 45.5 62.0▼ 6.1 64.4▼ 8.0 27.2▼ 4.6 w/ Surgical 11.0▼ 1.0 14.6▼ 3.9 76.6▲ 19.3 93.9▲ 8.7 98.6▲ 0.0 67.7▼ 0.4 71.3▼ 1.1 30.2▼ 1.6 w/ CAST 12.0▼ 2.0 10.9▼ 0.2 70.3▲ 13.0 91.2▲ 6.0 98.4▼ 0.2 67.3▼ 0.8 72.0▼ 0.4 30.6▼ 1.2 w/ AdaSteer 13.0▼ 3.0 13.5▼ 2.8 81.1▲ 23.8 96.8▲ 11.6 98.8▲ 0.2 66.0▼ 2.1 69.9▼ 2.5 27.8▼ 4.0 w/ AlphaSteer 11.0▼ 1.0 11.1▼ 0.4 77.3▲ 20.0 96.0▲ 10.8 98.2▼ 0.4 66.7▼ 1.4 71.2▼ 1.2 28.6▼ 3.2 w/ ELS 10.0▲ 0.0 9.4▲ 1.3 82.6▲ 25.3 97.6▲ 12.4 99.8▲ 1.2 68.1▲ 0.0 72.4▲ 0.0 31.6▼ 0.2 Llama2-7B-Chat 3.0▲ 0.0 1.6▲ 0.0 14.8▲ 0.0 13.6▲ 0.0 59.0▲ 0.0 47.6▲ 0.0 44.9▲ 0.0 14.6▲ 0.0 w/ system prompt 0.0▲ 3.0 0.0▲ 1.6 8.6▼ 6.2 4.5▼ 9.1 39.0▼ 20.0 47.5▼ 0.1 36.6▼ 8.3 10.6▼ 4.0 w/ Surgical 5.0▼ 2.0 5.5▼ 3.9 65.5▲ 50.7 42.4▲ 28.8 65.1▲ 6.1 47.0▼ 0.6 44.8▼ 0.1 9.4▼ 5.2 w/ CAST 7.0▼ 4.0 7.8▼ 6.2 66.7▲ 51.9 60.0▲ 46.4 64.6▲ 5.6 45.6▼ 2.0 43.3▼ 1.6 13.6▼ 1.0 w/ AdaSteer 5.0▼ 2.0 5.3▼ 3.7 75.7▲ 60.9 62.8▲ 49.2 66.2▲ 7.2 46.0▼ 1.6 43.7▼ 1.2 12.2▼ 2.4 w/ AlphaSteer 6.0▼ 3.0 6.4▼ 4.8 75.0▲ 60.2 67.6▲ 54.0 66.9▲ 7.9 46.0▼ 1.6 44.3▼ 0.6 14.4▼ 0.2 w/ ELS 3.0▲ 0.0 2.5▼ 0.9 78.4▲ 63.6 72.0▲ 58.4 67.0▲ 8.0 47.6▲ 0.0 44.9▲ 0.0 14.6▲ 0.0 Qwen 3 1.7B 49.0▲ 0.0 61.5▲ 0.0 95.5▲ 0.0 94.6▲ 0.0 93.3▲ 0.0 57.9▲ 0.0 52.8▲ 0.0 38.8▲ 0.0 w/ system prompt 27.0▲ 22.0 33.0▲ 28.5 54.2▼ 41.3 56.4▼ 38.2 52.9▼ 40.4 49.1▼ 8.8 47.3▼ 5.5 32.4▼ 6.4 w/ Surgical 51.0▼ 2.0 62.9▼ 1.4 95.8▲ 0.3 94.8▲ 0.2 94.6▲ 1.3 57.2▼ 0.7 52.1▼ 0.7 38.2▼ 0.6 w/ CAST 53.0▼ 4.0 63.3▼ 1.8 96.2▲ 0.7 96.0▲ 1.4 94.4▲ 1.1 56.8▼ 1.1 51.9▼ 0.9 38.0▼ 0.8 w/ AdaSteer 53.0▼ 4.0 62.9▼ 1.4 95.8▲ 0.3 95.2▲ 0.6 95.1▲ 1.8 57.4▼ 0.5 52.6▼ 0.2 38.6▼ 0.2 w/ AlphaSteer 52.0▼ 3.0 62.3▼ 0.8 96.0▲ 0.5 96.4▲ 1.8 95.6▲ 2.3 56.8▼ 1.1 52.2▼ 0.6 38.4▼ 0.4 w/ ELS 43.0▲ 6.0 54.7▲ 6.8 97.2▲ 1.7 96.4▲ 1.8 95.3▲ 2.0 57.9▲ 0.0 52.8▲ 0.0 38.8▲ 0.0 Qwen 3 8B 12.0▲ 0.0 28.3▲ 0.0 75.0▲ 0.0 95.6▲ 0.0 95.0▲ 0.0 72.8▲ 0.0 70.1▲ 0.0 54.8▲ 0.0 w/ system prompt 6.0▲ 6.0 5.6▲ 22.7 43.2▼ 31.8 46.8▼ 48.8 70.0▼ 25.0 70.2▼ 2.6 67.7▼ 2.4 52.4▼ 2.4 w/ Surgical 13.0▼ 1.0 30.1▼ 1.8 77.6▲ 2.6 96.4▲ 0.8 95.6▲ 0.6 71.2▼ 1.6 68.2▼ 1.9 53.8▼ 1.0 w/ CAST 14.0▼ 2.0 30.4▼ 2.1 79.5▲ 4.5 96.8▲ 1.2 95.8▲ 0.8 70.5▼ 2.3 67.9▼ 2.2 53.6▼ 1.2 w/ AdaSteer 13.0▼ 1.0 30.3▼ 2.0 78.0▲ 3.0 96.4▲ 0.8 96.2▲ 1.2 70.9▼ 1.9 68.4▼ 1.7 53.8▼ 1.0 w/ AlphaSteer 12.0▲ 0.0 29.9▼ 1.6 80.3▲ 5.3 96.0▲ 0.4 95.1▲ 0.1 72.3▼ 0.5 69.0▼ 1.1 54.4▼ 0.4 w/ ELS 11.0▲ 1.0 23.9▲ 4.4 80.6▲ 5.6 95.6▲ 0.0 96.4▲ 1.4 72.8▲ 0.0 70.1▲ 0.0 54.8▲ 0.0 Qwen 3 14B 14.0▲ 0.0 20.1▲ 0.0 81.1▲ 0.0 95.2▲ 0.0 94.0▲ 0.0 76.1▲ 0.0 72.5▲ 0.0 56.0▲ 0.0 w/ system prompt 3.0▲ 11.0 6.3▲ 13.8 50.8▼ 30.3 71.2▼ 24.0 79.0▼ 15.0 69.8▼ 6.3 69.9▼ 2.6 52.8▼ 3.2 w/ Surgical 16.0▼ 2.0 25.1▼ 5.0 82.6▲ 1.5 96.0▲ 0.8 93.8▼ 0.2 74.7▼ 1.4 72.3▼ 0.2 55.2▼ 0.8 w/ CAST 17.0▼ 3.0 24.8▼ 4.7 83.0▲ 1.9 94.8▼ 0.4 94.0▲ 0.0 74.0▼ 2.1 72.0▼ 0.5 54.6▼ 1.4 w/ AdaSteer 16.0▼ 2.0 21.3▼ 1.2 83.7▲ 2.6 95.6▲ 0.4 94.0▲ 0.0 74.4▼ 1.7 72.3▼ 0.2 54.4▼ 1.6 w/ AlphaSteer 14.0▲ 0.0 22.8▼ 2.7 84.1▲ 3.0 96.0▲ 0.8 94.2▲ 0.2 73.3▼ 2.8 72.1▼ 0.4 55.0▼ 1.0 w/ ELS 10.0▲ 4.0 18.9▲ 1.2 84.8▲ 3.7 96.4▲ 1.2 94.2▲ 0.2 76.1▲ 0.0 72.5▲ 0.0 56.0▲ 0.0 Table 1: Performance comparison of fine-tuning free methods on safety, false refusal, and general capability benchmarks. ELS approach is evaluated against the original model and other inference-time techniques across several LLMs, including Llama-3.1-8B, Llama-2-7B, and Qwen3 variants. Metrics include Compliance Rate (CR) on safety (JBB, Harmful) and false refusal (ORB-H, XSTest-S, OKTest) benchmarks, as well as accuracy on general capability tests (MMLU, ARC-C, MATH). Higher CR on false refusal and higher accuracy on general capability are better. 5 Experiment To comprehensively evaluate our Energy Landscape Steering method, we conduct a series of experiments designed to measure its performance across three key dimensions: (1) effectiveness, (2) robustness, and (3) efficiency. We assess its ability to mitigate false refusals without compromising safety or general capabilities, test its resilience against sophisticated multi-turn attacks, and analyze its computational overhead. We perform evaluations on a range of recent models, including variants from the Llama and Qwen families. Detailed descriptions of the datasets, baseline methods, and hyperparameter configurations are provided in Appendix C. !10Harmful Refusal !10Benign Compliance !10General Capability Model/Method !10WGTest !10HarmBench !10WJB !10DAN !10XSTest !10MMLU !10adv harm !10adv harm !10adv harm !10adv harm !10vani benign !10Acc !10ASR ↓ !10ASR ↓ !10ASR ↓ !10ASR ↓ !10Comply ↑ !10Score ↑ Llama-3.1-8B-IT 0.223▲ 0.000 0.654▲ 0.000 0.675▲ 0.000 0.533▲ 0.000 0.940▲ 0.000 0.680▲ 0.000 SFT 0.183▲ 0.040 0.348▲ 0.306 0.600▲ 0.075 0.468▲ 0.065 0.940▲ 0.000 0.634▼ 0.046 Defender-Only 0.276▼ 0.053 0.243▲ 0.411 0.695▼ 0.020 0.542▼ 0.009 0.968▲ 0.028 0.622▼ 0.058 Self-Play 0.172▲ 0.051 0.207▲ 0.447 0.536▲ 0.139 0.537▼ 0.004 0.964▲ 0.024 0.624▼ 0.056 Defender-Only + SFT 0.251▼ 0.028 0.260▲ 0.394 0.432▲ 0.243 0.452▲ 0.081 0.932▼ 0.008 0.623▼ 0.057 Self-Play + SFT 0.138▲ 0.085 0.221▲ 0.433 0.240▲ 0.435 0.396▲ 0.137 0.920▼ 0.020 0.623▼ 0.057 (ELS) Ours 0.219▲ 0.004 0.289▲ 0.365 0.207▲ 0.468 0.372▲ 0.161 0.976▲ 0.036 0.680▲ 0.000 Table 2: Performance comparison of fine-tuning methods against our ELS approach on the Llama-3.1-8B-IT model. The evaluation measures harmful refusal (WGTest, HarmBench, DAN, W.JB), benign compliance (XSTest), and general capability (MMLU). ASR (Attack Success Rate) is reported for harmful refusal, where lower is better. Arrows indicate the desired direction for each metric. Bold indicates the best-performing method. 5.1 Effectiveness Analysis We first evaluate the core effectiveness of our ELS approach against both fine-tuning free and fine-tuning based methods. Fine-tuning free methods include Surgical (Wang et al., 2024), CAST (Lee et al., 2024), AdaSteer (Zhao et al., 2025) and AlphaSteer (Sheng et al., 2025). Fine-tuning methods include SFT (Ouyang et al., 2022), Defender-Only, Self-Play (Liu et al., 2025), Defender-Only + SFT, and Self-Play + SFT, where Defender-Only denotes a baseline approach designed by the authors of Self-Play to represent conventional static defense training for comparative purposes. The primary goal is to demonstrate that our method can significantly reduce false refusals while maintaining or improving safety and preserving general knowledge. Comparison with Fine-Tuning Free Methods. As shown in Table 1, our ELS method consistently outperforms other fine-tuning free techniques in reducing false refusals. For the Llama-3.1-8B-Inst model, ELS achieves a Compliance Rate (CR) of 82.6% on the challenging ORB-H benchmark, a substantial improvement of 25.3 percentage points over the baseline’s 57.3%. This is the highest CR among all tested methods. Similar significant gains are observed on the XSTest-S(H) and OKTest benchmarks. Crucially, this improvement in helpfulness does not come at the cost of safety. On the JBB and Harmful safety benchmarks, our method maintains a CR identical or slightly better than the baseline, unlike methods such as Surgical and AdaSteer, which show a degradation in safety performance (i.e., higher compliance with harmful requests). Furthermore, general capabilities, as measured by MMLU, ARC-C, and MATH accuracy, remain almost entirely unaffected, demonstrating that our approach successfully resolves the safety-helpfulness trade-off. Unlike competing methods that force a compromise, our approach demonstrates that it is possible to surgically correct for over-refusal while holistically preserving the model’s carefully tuned safety alignment and core knowledge. This highlights ELS’s ability to make fine-grained adjustments, rather than applying the coarse interventions that lead to performance trade-offs in other systems. Comparison with Fine-Tuning Methods. In Table 2, we compare our ELS with several intensive fine-tuning strategies on the Llama-3.1-8B-IT model. The results highlight the strength and balanced profile of our approach. On the WJB (0.207) and DAN (0.372) safety benchmarks, ELS achieves the lowest Attack Success Rate (ASR), demonstrating superior resistance to prominent jailbreak techniques. While fine-tuning methods like Self-Play + SFT achieve a lower ASR on WGTest and HarmBench, our method still offers a substantial improvement over the baseline. Crucially, our method excels in preventing false refusals, attaining the highest benign compliance rate on XSTest (0.976). Perhaps most importantly, all compared fine-tuning methods lead to a significant drop in MMLU accuracy. In contrast, our approach is unique in preserving the model’s general capabilities entirely, matching the baseline score. This demonstrates that ELS provides a more robust and practical solution, achieving a strong, balanced safety profile without the high costs and capability degradation associated with retraining. 5.2 Robustness Analysis Figure 3: Robustness analysis on multi-turn jailbreak benchmarks. (a) Attack Success Rate (ASR) on the X-Teaming benchmark, evaluating the transferability of different methods against multi-turn attacks. Lower ASR is better.(b) Safety performance on the SafeDialBench benchmark, measuring the models’ ability to identify unsafe content in multi-turn dialogues. The score is based on GPT-4’s judgment, where a higher score indicates better identification capability. To assess the robustness of our method in more realistic conversational settings, we evaluate its performance against multi-turn jailbreak attacks. These attacks are more challenging as they attempt to bypass safety filters over several conversational turns. The results are presented in Figure 3. On the X-Teaming benchmark (Figure 3 (left)), which measures ASR for multi-turn attacks, our ELS method achieves a significantly lower success rate for the attacker compared to all other baseline methods. This indicates a stronger resilience in dynamic, conversational contexts. Furthermore, on the SafeDialBench benchmark (Figure 3 (right)), we evaluate the model’s ability to identify unsafe content within multi-turn dialogues, and evaluated the responses based using GPT-4o-mini. We attribute this enhanced resilience to ELS’s dynamic steering mechanism, which evaluates the generative trajectory at each step. This state-aware approach is fundamentally more resistant to contextual attacks designed to bypass static or coarse-grained safety filters over the course of a conversation. 5.3 Efficiency Analysis !10 Model Avg. Time / Prompt (s) Llama-3.1-8B-IT 1.60 + System Prompt 1.70 + Surgical 1.78 + CAST 1.76 + AdaSteer 1.80 + Alpha Steer 1.81 + ELS (Ours) 1.65 Table 3: Inference time per prompt. Total inference time (s) over 512 prompts and corresponding average time per prompt for Llama 3.1 8B IT model on the Harmful benchmark. Figure 4: Qualitative comparison of decision boundaries for classifying LLM hidden states. t-SNE visualizations show harmful (red) and harmless (blue) hidden state activations from Qwen3-14B. (Left) Vector Ablation yields a simple linear boundary that poorly separates the clusters. (Right) Our Energy-Based Model (EBM) learns a complex, non-linear boundary (where the energy gradient vanishes), accurately contouring and separating the clusters. This highlights the EBM’s superior discriminative power over linear methods. A critical consideration for any inference-time method is its impact on computational overhead. We measure the average inference latency and memory usage of our ELS method compared to other fine-tuning free baselines. All experiments were run on a system with four A6000 GPUs, where the vLLM GPU utilization was capped at 80%. As shown in Table 3, our approach is highly efficient. For the Llama-3.1-8B-IT model, ELS increases the average inference time only marginally, from 821s (1.60s/prompt) to 847s (1.65s/prompt) over 512 prompts. This overhead is substantially lower than that of other methods such as Surgical (910s, 1.78s/prompt) and AlphaSteer (927s, 1.81s/prompt). Moreover, the peak memory usage remains unchanged. These results demonstrate that our method achieves strong behavioral control with negligible impact on efficiency, making it a practical choice for real-world deployment. 5.4 Decision Boundary Analysis To visually assess our method’s effectiveness, Figure 4 shows t-SNE projections of hidden states from Qwen3-14B, comparing the decision boundaries of our EBM and a Vector Ablation baseline. The left panel shows the Vector Ablation method is akin to slicing the activation space in half with a rigid, linear boundary, an approach that inevitably misses nuance and misclassifies some states as the figure shows. In contrast, the right panel demonstrates our EBM’s energy boundary is not as rigid; it is a flexible, non-linear contour shaped by the learned “energy landscape.” This adaptability allows it to more accurately separate desirable from undesirable states, visually confirming the superior discriminative capability that underlies our method’s strong empirical performance. 6 Conclusion In this work, we propose Energy Landscape Steering (ELS), a fine-tuning-free framework that corrects LLM behavior at inference to reduce false refusals while preserving safety. We employ an external Energy-Based Model to steer generation away from undesirable high-energy regions in real time. Our experiments confirm that this approach reduces false refusals without sacrificing safety or general capability. References U. Anwar, A. Saparov, J. Rando, D. Paleka, M. Turpin, P. Hase, E. S. Lubana, E. Jenner, S. Casper, O. Sourbut, et al. (2024) Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932. Cited by: §1. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. (2022a) Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §2. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. (2022b) Constitutional ai: harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. Cited by: §2. F. Bianchi, M. Suzgun, G. Attanasio, P. Röttger, D. Jurafsky, T. Hashimoto, and J. Zou (2023) Safety-tuned llamas: lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875. Cited by: §1. H. Cao, Y. Wang, S. Jing, Z. Peng, Z. Bai, Z. Cao, M. Fang, F. Feng, B. Wang, J. Liu, T. Yang, J. Huo, Y. Gao, F. Meng, X. Yang, C. Deng, and J. Feng (2025) SafeDialBench: a fine-grained safety benchmark for large language models in multi-turn dialogues with diverse jailbreak attacks. External Links: 2502.11090, Link Cited by: 5th item. Z. Cao, Y. Yang, and H. Zhao (2024) Nothing in excess: mitigating the exaggerated safety for llms via safety-conscious activation steering. arXiv e-prints, p. arXiv–2408. Cited by: §2. P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer, et al. (2024) Jailbreakbench: an open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems 37, p. 55005–55029. Cited by: 2nd item, §4.1. S. Chen, X. Li, M. Zhang, E. H. Jiang, Q. Zeng, and C. Yu (2025) CARES: comprehensive evaluation of safety and adversarial robustness in medical llms. External Links: 2505.11413, Link Cited by: 1st item. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: 4th item. J. Cui, W. Chiang, I. Stoica, and C. Hsieh (2024) Or-bench: an over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947. Cited by: 3rd item. J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang (2023) Safe rlhf: safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773. Cited by: §2. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv e-prints, p. arXiv–2407. Cited by: 1st item, §1. K. Ethayarajh, S. Kim, and D. Jurafsky (2024) KTO: model alignment as prospect theoretic optimization. arXiv:2402.01306. Cited by: §2. S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dziri (2024) Wildguard: open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in Neural Information Processing Systems 37, p. 8093–8131. Cited by: 2nd item. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020) Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: 4th item. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), External Links: Link Cited by: 4th item. Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, F. Huang, et al. (2006) A tutorial on energy-based learning. Predicting structured data 1 (0). Cited by: §1. B. W. Lee, I. Padhi, K. N. Ramamurthy, E. Miehling, P. Dognin, M. Nagireddy, and A. Dhurandhar (2024) Programming refusal with conditional activation steering. arXiv preprint arXiv:2409.05907. Cited by: §5.1. X. Li, X. Chen, J. Fan, E. H. Jiang, and M. Gao (2025) Multi-head reward aggregation guided by entropy. External Links: 2503.20995, Link Cited by: §2. M. Liu, L. Jiang, Y. Liang, S. S. Du, Y. Choi, T. Althoff, and N. Jaques (2025) Chasing moving targets with online self-play reinforcement learning for safer language models. arXiv preprint arXiv:2506.07468. Cited by: 3rd item, §2, §5.1. M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. (2024) Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. Cited by: 2nd item. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. Cited by: §1, §5.1. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving (2022) Red teaming language models with language models. arXiv preprint arXiv:2202.03286. Cited by: §2. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. NeurIPS 36. External Links: Link Cited by: §2. S. Rahman, L. Jiang, J. Shiffer, G. Liu, S. Issaka, M. R. Parvez, H. Palangi, K. Chang, Y. Choi, and S. Gabriel (2025) X-teaming: multi-turn jailbreaks and defenses with adaptive multi-agents. In Second Conference on Language Modeling, External Links: Link Cited by: 5th item. P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy (2023) Xstest: a test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263. Cited by: 2nd item, 3rd item, §1. P. Röttger, B. Vidgen, D. Nguyen, Z. Waseem, H. Margetts, and J. B. Pierrehumbert (2020) HateCheck: functional tests for hate speech detection models. arXiv preprint arXiv:2012.15606. Cited by: §1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2. X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang (2024) " Do anything now": characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 1671–1685. Cited by: 2nd item. L. Sheng, C. Shen, W. Zhao, J. Fang, X. Liu, Z. Liang, X. Wang, A. Zhang, and T. Chua (2025) AlphaSteer: learning refusal steering with principled null-space constraint. arXiv preprint arXiv:2506.07022. Cited by: §5.1. C. Shi, X. Wang, Q. Ge, S. Gao, X. Yang, T. Gui, Q. Zhang, X. Huang, X. Zhao, and D. Lin (2024) Navigating the overkill in large language models. arXiv preprint arXiv:2401.17633. Cited by: 3rd item. H. Sun, G. Xu, J. Deng, J. Cheng, C. Zheng, H. Zhou, N. Peng, X. Zhu, and M. Huang (2021) On the safety of conversational models: taxonomy, dataset, and benchmark. arXiv preprint arXiv:2110.08466. Cited by: §1. G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. (2024) Gemma: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Cited by: 1st item. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: 1st item, §1. B. Vidgen, N. Scherrer, H. R. Kirk, R. Qian, A. Kannappan, S. A. Hale, and P. Röttger (2023) Simplesafetytests: a test suite for identifying critical safety risks in large language models. arXiv preprint arXiv:2311.08370. Cited by: §1. X. Wang, C. Hu, P. Röttger, and B. Plank (2024) Surgical, cheap, and flexible: mitigating false refusal in language models via single vector ablation. arXiv preprint arXiv:2410.03415. Cited by: §1, §2, §5.1. A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein (2021) Detoxifying language models risks marginalizing minority voices. arXiv preprint arXiv:2104.06390. Cited by: §2. J. Xu, D. Ju, M. Li, Y. Boureau, J. Weston, and E. Dinan (2020) Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. Cited by: §1. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: 1st item, §1. W. Zhao, J. Guo, Y. Hu, Y. Deng, A. Zhang, X. Sui, X. Han, Y. Zhao, B. Qin, T. Chua, et al. (2025) AdaSteer: your aligned llm is inherently an adaptive jailbreak defender. arXiv preprint arXiv:2504.09466. Cited by: §5.1. C. Zheng, F. Yin, H. Zhou, F. Meng, J. Zhou, K. Chang, M. Huang, and N. Peng (2024) On prompt-driven safeguarding for large language models. arXiv preprint arXiv:2401.18018. Cited by: §1, §2. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving (2019) Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. Cited by: §1. Appendix A Limitations ELS relies on EBM trained on a fixed dataset, limiting its adaptability to emerging jailbreak tactics unseen during training. While highly effective against known attacks, it does not update online like RL-based methods, trading maximal adaptability for inference efficiency and weight-free deployment. However, our multi-turn robustness results (Section 4.2) demonstrate that the learned energy landscape generalizes well beyond the training distribution, as ELS maintains strong safety performance against sophisticated multi-turn attacks not seen during EBM training. Additionally, when new attack patterns emerge, only the lightweight EBM needs to be retrained, a process that takes minutes rather than the hours or days required for full model fine-tuning, making rapid adaptation practical. Our current data labeling operates at the response level, classifying entire responses as compliant or refusal. A more fine-grained, sentence-level annotation scheme could capture partial compliance or mid-response refusal patterns, which would be particularly beneficial for reasoning models that produce extended chain-of-thought traces. Rescuing a misaligned refusal during the thinking period of such models represents a promising direction. We leave this extension to future work. A natural extension of this work is to make ELS an online method, where the EBM becomes an active learner that continuously updates its energy landscape based on new interaction data. This would allow the system to adapt to evolving jailbreak strategies in real time, combining the deployment flexibility of our current approach with the adaptability of online learning methods. Appendix B Algorithm The Pseudocode of ELS in Algorithm 1. Algorithm 1 Energy-Based Model Steering for LLMs 0: Pre-trained LLM, dataset of prompts, EBM parameters 0: Reduced false refusals in LLM outputs 1: Phase 1: Activation Data Collection 2: for each prompt X in the dataset do 3: Generate sequence Y=(y1,y2,…,yT)Y=(y_1,y_2,…,y_T) using the LLM 4: for each token yty_t in Y do 5: Extract hidden state hth_t from the LLM 6: end for 7: Classify Y as "Refusal" or "Compliant" using classifier C(Y)C(Y) 8: Store hth_t in badD_bad if "Refusal", else in goodD_good 9: end for 10: Phase 2: EBM Training via Contrastive Learning 11: Initialize EBM with parameters θ 12: for each epoch do 13: for each batch of hidden states (h+,hi−i=1N)(h^+,\h^-_i\_i=1^N) do 14: Compute energy Eθ(h+)E_θ(h^+) and Eθ(hi−)E_θ(h^-_i) 15: Compute InfoNCE loss ℒ(θ)L(θ) 16: Update θ to minimize ℒ(θ)L(θ) 17: end for 18: end for 19: Phase 3: Real-time Gradient-Based Steering 20: for each token yty_t during LLM inference do 21: Compute hidden state hth_t 22: Compute energy gradient ∇hEθ(ht) _hE_θ(h_t) 23: Update hidden state ht′=ht−η⋅∇hEθ(ht)h _t=h_t-η· _hE_θ(h_t) 24: Use ht′h _t to compute steered logits lt′l _t 25: Generate next token yt+1y_t+1 using steered logits 26: end for Appendix C Detailed Setups of Our Experiments Datasets Our experiments are conducted based on datasets as followed. • Training Dataset (1) CARES-21K (Chen et al., 2025) • Safety (1) JailbreakBench (Chao et al., 2024); (2)HarmBench (Mazeika et al., 2024); (3)XSTest Unsafe (Röttger et al., 2023); (4)Wildguard Test (Han et al., 2024); (5)DAN (Shen et al., 2024) • False Refusal (1) Orbench (Cui et al., 2024); (2) OKTest (Shi et al., 2024); (3)XSTest Safe (Röttger et al., 2023); • General Capability (1) MMLU (Hendrycks et al., 2020); (2) ARC (Clark et al., 2018); (3) MATH (Hendrycks et al., 2021) • Multi-Turn Attack (1) X-Teaming (Rahman et al., 2025); (2) SafeDialBench (Cao et al., 2025) Baselines Our EBM mothed is compared with original models, models with fine-tuning free methods and models with fine-tuning methods as followed. • Original models (1) Llama3.1-8B-Instruct (Dubey et al., 2024); (2) Llama2-7B-Chat (Touvron et al., 2023); (3) Gemma-7B (Team et al., 2024); (4) Qwen3-1.7B (Yang et al., 2025); (5)Qwen3-8B (Yang et al., 2025); (6) Qwen3-14B (Yang et al., 2025) • Finetuing-Free (1) System prompt; (2) Vector ablation; • Finetuing (1) Denfender-Only; (2) Self-Play; (3)Denfender-Only + SFT; (4) Self-Play + SFT. All from (Liu et al., 2025) C.1 Implementation Details and Hyperparameters EBM Data Collection and Processing. The dataset for training the EBMs was constructed using the SafeMedEval-21K training dataset, which provides a rich collection of prompts with varying harmfulness levels. We employed a balanced sampling strategy, extracting 1,000 prompts each for harmless content (filtering for harmful_level: 0) and harmful content (filtering for harmful_level: 2). Responses were generated using vLLM with optimized inference parameters: tensor parallelism was set to 1, GPU memory utilization was capped at 80%, and the maximum sequence length was limited to 512 tokens. For fallback scenarios, we used standard HuggingFace generation with a batch size of 16. All activations were extracted from the last token position of each generated sequence using a dedicated extraction batch size of 16 to balance memory usage and processing speed. EBM Architecture and Training Configuration. All EBMs utilize our complex architecture, a 4-layer MLP with progressive dimension reduction: [2048 → 1024 → 1024 → 512]. Each layer incorporates Layer Normalization for stable training and Dropout (rate 0.15) for regularization. We train an individual EBM for every layer of the host LLM, enabling fine-grained control across the model’s representation space. The training process spans 120 epochs using the Adam optimizer with a carefully tuned learning rate of 5×10−55× 10^-5. The InfoNCE contrastive loss employs a temperature parameter τ=0.10τ=0.10 to sharpen the softmax distribution. Training data is processed in batches of 64, and we use an 80/20 train-validation split for model selection. Inference-time Steering Configuration. During inference, steering is applied to the top-performing layers as determined by validation accuracy. The intervention strategy varies significantly across models to account for their different architectures and training procedures. All hyperparameters were tuned individually for each model through grid search on a held-out development set. Table 4: Comprehensive hyperparameter configuration for all evaluated models. Hyperparameter Llama-2-7B Llama-3.1-8B Qwen3-1.7B Qwen3-8B Qwen3-14B EBM Training Configuration Architecture Complex Complex Complex Complex Complex Hidden dimensions [2048,1024,1024,512] [2048,1024,1024,512] [2048,1024,1024,512] [2048,1024,1024,512] [2048,1024,1024,512] Dropout rate 0.15 0.15 0.15 0.15 0.15 Layer normalization Yes Yes Yes Yes Yes Training epochs 120 120 120 120 120 Learning rate 5×10−55× 10^-5 5×10−55× 10^-5 5×10−55× 10^-5 5×10−55× 10^-5 5×10−55× 10^-5 Batch size 64 64 64 64 64 InfoNCE temperature (τ) 0.10 0.10 0.10 0.10 0.10 Training data size 2,000 2,000 2,000 2,000 2,000 Optimizer Adam Adam Adam Adam Adam Inference-time Steering Configuration Top-N layers selected 12 15 3 10 20 Steering coefficient (η) 0.95 0.1 1.0 0.30 0.30 Gradient steps per token 12 3 10 3 3 Intervention layers All trained All trained All trained All trained All trained Activation positions Last token (-1) Last token (-1) Last token (-1) Last token (-1) Last token (-1) Data Generation Configuration Max generation tokens 512 512 512 512 512 Extraction batch size 16 16 16 16 16 GPU memory utilization 80% 80% 80% 80% 80% Tensor parallel size 1 1 1 1 1 vLLM max sequence length 512 512 512 512 512 Model-specific Tuning Rationale. The significant variation in steering hyperparameters across models reflects their different sensitivity to activation perturbations. Larger models (Llama-3.1-8B, Qwen3-14B) generally require more conservative steering coefficients and fewer gradient steps to maintain stability, while smaller models (Qwen3-1.7B) can accommodate more aggressive intervention. The number of selected layers for steering correlates with model capacity: deeper models benefit from intervention across more layers to capture complex representational patterns. Dataset Configuration and Evaluation Setup. Our evaluation framework encompasses three categories of benchmarks: safety evaluation (measuring resistance to harmful prompts), false refusal evaluation (measuring appropriate compliance to benign prompts), and general capability evaluation. Each category employs specific datasets and evaluation methodologies as detailed in Table 5. Evaluation Category Dataset Sample Size Evaluation Method Safety JailbreakBench (JBB) 100 Compliance rate HarmBench 512 Compliance rate XSTest Unsafe 200 Compliance rate WG Test 324 Attack Sucess Rate Wildguard Test 2,000 Attack Sucess Rate DAN Unsafe 78 Attack Sucess Rate False Refusal ORB-Hard 264 Compliance rate XSTest Safe 250 Compliance rate OKTest 450 Compliance rate General Capability MMLU 285 Accuracy ARC-Challenge 1,172 Accuracy MATH 500 Accuracy Multi-Turn Attack X-Teaming 50 Attack Sucess Rate SafeDialBench 60 GPT 4-o mini Table 5: Evaluation dataset configuration and methodology. Hardware and Infrastructure Requirements. All experiments were conducted on NVIDIA A6000 GPUs with 48GB VRAM. The choice of GPU memory utilization parameters was carefully tuned to maximize throughput while preventing out-of-memory errors. For EBM training, we utilize CUDA optimization with mixed precision training disabled to maintain numerical stability of the energy gradients. The activation extraction phase requires the most memory, necessitating the lower batch sizes (16) compared to standard LLM inference. Evaluation Metrics and Methodology. Our evaluation employs multiple complementary metrics to assess different aspects of model performance. For safety evaluation, we compute the Compliance Rate (CR), defined as the percentage of harmful prompts that the model appropriately refuses. For false refusal evaluation, we use the same CR metric but applied to benign prompts, where higher compliance indicates fewer false refusals. Attack Success Rate (ASR) represents the inverse of CR for harmful prompts. The substring matching evaluation method employs a curated list of refusal indicators including phrases like “I cannot”, “I’m sorry”, and “I’m not able to”, with responses containing these phrases classified as refusals. Robustness Experiment Setup. Our robustness analysis employed two multi-turn attack benchmarks to evaluate performance in conversational contexts. For the X-Teaming benchmark, we assessed transferability against multi-turn attacks using test cases derived from the first 50 harmful behaviors in HarmBench. Each behavior was tested with 10 attack plans across 3 turns. For the SafeDialBench benchmark, we selected 60 multi-turn attack dialogues, 10 for each of the six safety dimensions (aggression, ethics, fairness, legality, morality, and privacy). Model responses were scored by GPT-4o mini, using the prompt from the original paper, to exclusively assess the model’s ability to identify unsafe content. Ablation Study Configuration. All ablation studies were conducted on the Llama-3.1-8B-IT model to analyze the sensitivity of our method’s key hyperparameters. We evaluated the impact on performance by varying one parameter at a time while keeping others fixed at their optimal values (as detailed in Table 4). The performance was measured using three metrics: ORB-H CR (false refusal), JBB CR (safety), and MMLU Accuracy (general capability). We investigated: (1) the number of intervention layers, testing values from 10 to 30; (2) the steering coefficient (η), testing values from 0.05 to 0.25; and (3) the number of gradient steps per token, testing values from 1 to 20. Reproducibility and Code Availability. All experiments can be reproduced using the provided configuration files and the command: python -m pipeline.run_pipeline --config_path configs/[model_config].yaml. The complete codebase, including EBM implementations, evaluation scripts, and data processing utilities, is available in the supplementary material. Environment setup is automated via the provided setup.sh script, which installs all required dependencies including the LM Evaluation Harness. Appendix D Aditional Experiment Figure 5: Ablation studies on key hyperparameters for ELS with the Llama-3.1-8B-IT model. The plots show how performance on Llama 3.1 8-B IT when running ORB-H CR (%), JBB CR (%), and MMLU Acc (%) varies with changes to: (a) The number of layers selected for intervention. (b) The steering coefficient (η) . (c) The number of gradient descent steps per token. D.1 Ablation Studies To understand the sensitivity of our approach to its key hyperparameters, we conducted several ablation studies, with results shown in Figure 5. We analyzed the impact of the number of layers selected for intervention, the steering coefficient (η), and the number of gradient steps per token. The results show that performance is stable across a range of layer counts, though it peaks when a significant portion of the model’s layers are utilized (Figure 5 (left)). The steering coefficient (η) shows a clear optimal range (Figure 5 (middle)); a value that is too low provides insufficient correction, while a value that is too high can slightly degrade performance on general tasks. Finally, we observe that the benefits of steering are largely achieved within a few gradient steps, with performance plateauing quickly (Figure 5 (right)). Overall, these findings highlight the ELS framework’s stability, demonstrating robust performance across a well-defined, predictable range of hyperparameters—enabling reliable tuning of ELS for new models without exhaustive, costly parameter sweeps. D.2 EBM Architecture and Layer Selection Ablation To investigate the impact of different EBM architectures and layer selection strategies, we conduct additional ablation studies on the Llama-3.1-8B-IT model, reported in Table 6. !10 Configuration ORB-H CR ↑ JBB CR ↓ MMLU Acc ↑ EBM Architecture 3-layer MLP 78.4 11.0 67.9 4-layer + LN + DO 82.6 10.0 68.1 Layer Selection Strategy Top-5 layers (by val acc) 79.2 10.0 68.0 Top-10 layers (by val acc) 80.8 10.0 68.1 Top-15 layers (by val acc) 82.6 10.0 68.1 Top-20 layers (by val acc) 82.1 11.0 68.0 Layer Region (fixed 11 layers) Early layers only (0–10) 74.6 10.0 68.1 Middle layers only (11–21) 80.3 10.0 68.0 Late layers only (22–31) 78.1 11.0 67.8 Table 6: Ablation on EBM architecture and layer selection strategy. Results on Llama-3.1-8B-IT. Purple rows indicate the optimal setting. ComplexEBM with validation-based top-N layer selection achieves the best balance across all three dimensions. Architecture Comparison. The ComplexEBM architecture (4-layer MLP with LayerNorm and Dropout) consistently outperforms the SimpleEBM (3-layer MLP without normalization), achieving 82.6% vs. 78.4% on ORB-H while maintaining identical safety performance. The LayerNorm stabilizes gradient computation during inference-time steering, while Dropout during training improves generalization of the energy landscape. Layer Selection Strategy. Our validation-accuracy-based top-N selection strategy proves effective: selecting the top-15 layers yields the best performance, consistent with the ablation in Figure 5(a). Notably, intervening on middle layers alone (11–21) outperforms early or late layers, suggesting that the mid-network representations carry the most discriminative information for distinguishing desirable from undesirable behavioral trajectories. D.3 Training With In-Domain Dataset Performance comparison of fine-tuning free methods on safety, false refusal, and general capability bench- marks are shown in Table 7. !10 Model/Method Safety False Refusal General Capability !10 JBB CR ↓ Harmful CR ↓ ORB-H CR ↑ XSTest-S(H) CR ↑ OKTest CR ↑ MMLU Acc ↑ ARC-C Acc ↑ MATH Acc ↑ Llama3.1-8B-Inst 10.0▲ 0.0 10.7▲ 0.0 57.3▲ 0.0 85.2▲ 0.0 98.6▲ 0.0 68.1▲ 0.0 72.4▲ 0.0 31.8▲ 0.0 w/ system prompt 3.0▲ 7.0 2.3▲ 8.4 41.0▼ 16.3 37.6▼ 47.6 53.1▼ 45.5 62.0▼ 6.1 64.4▼ 8.0 27.2▼ 4.6 w/ Surgical 11.0▼ 1.0 14.6▼ 3.9 76.6▲ 19.3 93.9▲ 8.7 98.6▲ 0.0 67.7▼ 0.4 71.3▼ 1.1 30.2▼ 1.6 w/ CAST 12.0▼ 2.0 10.9▼ 0.2 70.3▲ 13.0 91.2▲ 6.0 98.4▼ 0.2 67.3▼ 0.8 72.0▼ 0.4 30.6▼ 1.2 w/ AdaSteer 13.0▼ 3.0 13.5▼ 2.8 81.1▲ 23.8 96.8▲ 11.6 98.8▲ 0.2 66.0▼ 2.1 69.9▼ 2.5 27.8▼ 4.0 w/ AlphaSteer 11.0▼ 1.0 11.1▼ 0.4 77.3▲ 20.0 96.0▲ 10.8 98.2▼ 0.4 66.7▼ 1.4 71.2▼ 1.2 28.6▼ 3.2 w/ ELS 9.0▲ 1.0 10.7▲ 0.0 83.7▲ 26.4 96.8▲ 11.6 98.8▲ 0.2 66.7▼ 1.4 72.4▲ 0.0 31.8▲ 0.0 Table 7: Performance comparison of fine-tuning free methods on safety, false refusal, and general capability benchmarks. Training Data : WildJailBreak Source paper: CHASING MOVING TARGETS WITH ONLINE SELF-PLAY REINFORCEMENT LEARNING FOR SAFER LANGUAGE MODELS, Source link: https://github.com/mickelliu/selfplay-redteaming (redteam/data/data.zip) (1000 benign, 1000 harmful) ELS approach is evaluated against the original model and other inference-time techniques across several LLMs, including Llama-3.1-8B. Metrics include Compliance Rate (CR) on safety (JBB, Harmful) and false refusal (ORB-H, XSTest-S, OKTest) benchmarks, as well as accuracy on general capability tests (MMLU, ARC-C, MATH). Higher CR on false refusal and higher accuracy on general capability are better. Appendix E Theoretical Justification of Energy Gradient-Steering Before presenting the formal proofs, we provide a high-level overview of the main intuition. We conceptualize the LLM’s internal representations as evolving over an energy landscape. An auxiliary Energy-Based Model (EBM) is trained to shape this landscape such that undesirable behaviors (e.g., false refusals or jailbreaks) correspond to high-energy regions, while desirable behaviors (e.g., helpful responses and safe refusals) correspond to low-energy regions. During generation, we apply a single gradient step at each token to move the model’s hidden state toward lower-energy regions. When the model is already operating in a low-energy region, this adjustment is minimal and preserves its general capabilities. However, if the trajectory begins to move toward a high-energy (undesirable) region, the gradient step redirects it toward a more desirable state. The formal results establish three claims: (1) the training objective reliably shapes the energy landscape (Lemma E.1); (2) the gradient-based update provably decreases energy (Theorem E.1); and (3) repeated application of this update steers trajectories associated with false refusals toward desirable states (Corollary E.1). Below, in this section, we provide a rigorous mathematical justification for the gradient-based steering mechanism. We formalize the components of our framework using definitions, lemmas, and theorems to prove that the proposed steering update is a principled optimization procedure that guides the LLM’s generative trajectory away from regions associated with false refusals. E.1 Preliminaries and Formal Definitions Definition E.1 (Energy Function). An Energy-Based Model (EBM) is defined by a parameterized energy function Eθ:ℋ→ℝE_θ:H , where ℋ=ℝdH=R^d is the hidden state space of a Large Language Model. The function maps a hidden state h∈ℋh to a scalar energy value. A lower energy is designed to correspond to a higher probability of a desirable outcome (e.g., a compliant response), while higher energy corresponds to an undesirable outcome (e.g., a false refusal). The function is realized by a multi-layer perceptron with parameters θ. Definition E.2 (Optimal Energy Function). Let good⊂ℋD_good be the set of hidden states from desirable trajectories (e.g., compliant) and bad⊂ℋD_bad be the set of states from undesirable trajectories (e.g., false refusals). An optimal energy function E∗(h)E^*(h) is a function that perfectly separates these sets, such that for any hgood∈goodh_good _good and hbad∈badh_bad _bad, there exists a margin m>0m>0 where: E∗(hbad)>E∗(hgood)+mE^*(h_bad)>E^*(h_good)+m (13) Our trained EBM, Eθ(h)E_θ(h), serves as an approximation of this optimal function, i.e., Eθ(h)≈E∗(h)E_θ(h)≈ E^*(h). E.2 EBM Training and Energy Landscape The parameters θ of the energy function Eθ(h)E_θ(h) are learned by optimizing a training objective designed to shape the energy landscape according to Definition E.2. Training Objective Function. The EBM is trained using the InfoNCE contrastive loss. For an anchor state h+∈goodh^+ _good and a set of N negative samples hi−i=1N⊂bad\h^-_i\_i=1^N _bad, the loss is: ℒ(θ)=−h+,hi−[log(exp(−Eθ(h+)/τ)/(exp(−Eθ(h+)/τ)+∑i=1Nexp(−Eθ(hi−)/τ)))] split&L(θ)=-E_h^+,\h^-_i\ [ ( \!(-E_θ(h^+)/τ)/\\ & ( \!(-E_θ(h^+)/τ)+ _i=1^N \!(-E_θ(h^-_i)/τ) ) ) ] split (14) where τ is a temperature hyperparameter. Lemma E.1 (Energy Landscape Property). Minimizing the InfoNCE loss (Equation 14) trains the energy function Eθ(h)E_θ(h) to assign lower energy values to hidden states from desirable trajectories (goodD_good) and higher energy values to hidden states from undesirable trajectories (badD_bad). Formally, for a well-trained model, if hgood∈goodh_good _good and hbad∈badh_bad _bad, it is highly probable that Eθ(hgood)<Eθ(hbad)E_θ(h_good)<E_θ(h_bad). Proof. The InfoNCE loss is a form of cross-entropy loss. Let the logits be s+=−Eθ(h+)/τs^+=-E_θ(h^+)/τ and si−=−Eθ(hi−)/τs^-_i=-E_θ(h^-_i)/τ. The loss for a single sample can be written as: ℒ=−s++log(exp(s+)+∑i=1Nexp(si−))L=-s^++ ( (s^+)+ _i=1^N (s^-_i) ) (15) The parameter update rule for gradient descent is θt+1=θt−α∇θℒ _t+1= _t-α _θL. The change in an energy value E is approximately ΔE≈(∇θE)TΔθ=−α(∇θE)T(∇θℒ) E≈( _θE)^T θ=-α( _θE)^T( _θL). Using the chain rule, ∇θℒ=∂ℒ∂E∇θE _θL= ∂ E _θE, we get: ΔE≈−α(∇θE)T(∂ℒ∂E∇θE)=−α∂ℒ∂E‖∇θE‖22 split E&≈-α( _θE)^T ( ∂ E _θE )\\ &=-α ∂ E\| _θE\|_2^2 split (16) This implies sign(ΔE)=−sign(∂ℒ∂E)sign( E)=-sign( ∂ E). We now compute these partial derivatives. Derivative w.r.t. Eθ(h+)E_θ(h^+): Let E+=Eθ(h+)E^+=E_θ(h^+). The derivative is computed via the chain rule ∂ℒ∂E+=∂ℒ∂s+∂s+∂E+ ∂ E^+= ∂ s^+ ∂ s^+∂ E^+. First: ∂s+∂E+=−1τ ∂ s^+∂ E^+=- 1τ (17) ∂ℒ∂s+=−1+1exp(s+)+∑iexp(si−)⋅exp(s+)=exp(s+)exp(s+)+∑iexp(si−)−1 split ∂ s^+&=-1+ 1 (s^+)+ _i (s^-_i)· (s^+)\\ &= (s^+) (s^+)+ _i (s^-_i)-1 split (18) Combining these gives: ∂ℒ∂E+=(exp(s+)exp(s+)+∑iexp(si−)−1)(−1τ)=1τ(1−P(h+))>0 split ∂ E^+&= ( (s^+) (s^+)+ _i (s^-_i)-1 ) (- 1τ )\\ &= 1τ (1-P(h^+) )>0 split (19) where P(h+)P(h^+) is the softmax probability of the positive sample. Therefore, ΔEθ(h+)∝−(+)<0 E_θ(h^+) -(+)<0, meaning the energy of ’good’ states decreases. Derivative w.r.t. Eθ(hj−)E_θ(h^-_j): Let Ej−=Eθ(hj−)E^-_j=E_θ(h^-_j). The derivative is ∂ℒ∂Ej−=∂ℒ∂sj−∂sj−∂Ej− ∂ E^-_j= ∂ s^-_j ∂ s^-_j∂ E^-_j. First: ∂sj−∂Ej−=−1τ ∂ s^-_j∂ E^-_j=- 1τ (20) ∂ℒ∂sj−=1exp(s+)+∑iexp(si−)⋅exp(sj−)=P(hj−) split ∂ s^-_j&= 1 (s^+)+ _i (s^-_i)· (s^-_j)\\ &=P(h^-_j) split (21) Combining these gives: ∂ℒ∂Ej−=P(hj−)(−1τ)=−1τP(hj−)<0 ∂ E^-_j=P(h^-_j) (- 1τ )=- 1τP(h^-_j)<0 (22) Therefore, ΔEθ(hj−)∝−(−)>0 E_θ(h^-_j) -(-)>0, meaning the energy of ’bad’ states increases. This completes the proof. ∎ E.3 Probabilistic Interpretation and Steering as MAP Inference The learned energy function can be formally linked to a probability distribution over the hidden state space via the Gibbs-Boltzmann distribution. Definition E.3 (State Probability Density). The probability density that a hidden state h belongs to the class of desirable (compliant) states, goodC_good, is given by: p(h∈good)=exp(−Eθ(h)/τ)Z(θ,τ)p(h _good)= (-E_θ(h)/τ)Z(θ,τ) (23) where Z(θ,τ)Z(θ,τ) is the partition function, which normalizes the distribution over the entire state space ℋH: Z(θ,τ)=∫h′∈ℋexp(−Eθ(h′)/τ)h′Z(θ,τ)= _h (-E_θ(h )/τ)dh (24) This formulation is a direct consequence of the energy landscape established in Lemma E.1. For any two states h1,h2∈ℋh_1,h_2 , their relative probability is: p(h1∈good)p(h2∈good)=exp(−Eθ(h1)/τ)exp(−Eθ(h2)/τ)=exp(−Eθ(h1)−Eθ(h2)τ) split p(h_1 _good)p(h_2 _good)&= (-E_θ(h_1)/τ) (-E_θ(h_2)/τ)\\ &= (- E_θ(h_1)-E_θ(h_2)τ ) split (25) If we take h1∈goodh_1 _good and h2∈badh_2 _bad, from Lemma E.1 we know Eθ(h1)<Eθ(h2)E_θ(h_1)<E_θ(h_2), which implies Eθ(h1)−Eθ(h2)<0E_θ(h_1)-E_θ(h_2)<0. Therefore, the exponent is positive, leading to p(h1)>p(h2)p(h_1)>p(h_2). This confirms that low-energy states are exponentially more probable. The objective of our steering mechanism can now be re-framed as a Maximum A Posteriori (MAP) inference problem: finding the hidden state h∗h^* that maximizes the probability of belonging to the desirable class. h∗=argmaxh∈ℋp(h∈good)h^*= _h p(h _good) (26) This maximization is equivalent to minimizing the energy function Eθ(h)E_θ(h): argmaxh _h p(h)=argmaxhexp(−Eθ(h)/τ)Z(θ,τ) p(h)= _h (-E_θ(h)/τ)Z(θ,τ) (27) =argmaxhlog(exp(−Eθ(h)/τ)Z(θ,τ)) = _h ( (-E_θ(h)/τ)Z(θ,τ) ) (28) =argmaxh(−Eθ(h)τ−logZ(θ,τ)) = _h (- E_θ(h)τ- Z(θ,τ) ) (29) =argminhEθ(h) = _hE_θ(h) (30) The equivalence holds because the logarithm is a strictly monotonic function, and Z(θ,τ)Z(θ,τ) and τ are positive constants with respect to h. This probabilistic framing demonstrates that the gradient descent on energy performed in Theorem E.1 is not merely an ad-hoc procedure, but a principled method for performing gradient-based MAP inference. The gradient of the log-probability with respect to the state h is directly proportional to the negative energy gradient: ∇hlogp(h∈good)=∇h(−Eθ(h)τ−logZ)=−1τ∇hEθ(h) split _h p(h _good)&= _h (- E_θ(h)τ- Z )\\ &=- 1τ _hE_θ(h) split (31) Therefore, the gradient ascent update rule to maximize the log-probability is: hk+1=hk+α∇hlogp(hk)=hk−ατ∇hEθ(hk)h_k+1=h_k+α _h p(h_k)=h_k- ατ _hE_θ(h_k) (32) This is precisely the form of our steering update rule, with the steering coefficient η=α/τη=α/τ. The subsequent sections provide a formal proof of convergence for this procedure. E.4 Gradient-Based Steering Mechanism and Analysis The steering mechanism uses the gradient of the learned energy function to modify the LLM’s hidden states during inference. Definition E.4 (Energy Gradient). The energy gradient, ∇hEθ(h) _hE_θ(h), is the vector of partial derivatives of the energy function with respect to the input hidden state h: ∇hEθ(h)=[∂Eθ∂h1,∂Eθ∂h2,…,∂Eθ∂hd]T _hE_θ(h)= [ ∂ E_θ∂ h_1, ∂ E_θ∂ h_2,…, ∂ E_θ∂ h_d ]^T (33) This gradient is computed via backpropagation and points in the direction of the steepest ascent on the energy surface. Theorem E.1 (Energy Minimization via Gradient-Based Steering). Let hth_t be the hidden state at generation step t. Let the steering update rule be defined as: ht′=ht−η⋅∇hEθ(h)|h=hth _t=h_t-η· _hE_θ(h)|_h=h_t (34) For a steering coefficient η satisfying 0<η<2λmax((ht))0<η< 2 _ (H(h_t)), where λmax((ht)) _ (H(h_t)) is the maximum eigenvalue of the Hessian matrix H of EθE_θ at hth_t, the update guarantees a decrease in energy, i.e., Eθ(ht′)<Eθ(ht)E_θ(h _t)<E_θ(h_t), provided that ∇hEθ(ht)≠ _hE_θ(h_t) 0. Proof. Let g(h)=∇hEθ(h)g(h)= _hE_θ(h). The change in energy is ΔE=Eθ(ht−ηg(ht))−Eθ(ht) E=E_θ(h_t-η g(h_t))-E_θ(h_t). Using a second-order Taylor expansion for EθE_θ around hth_t: Eθ(ht−ηg(ht))=Eθ(ht)−ηg(ht)Tg(ht)+12η2g(ht)T(ht)g(ht)+(η3) splitE_θ(h_t-&η g(h_t))=E_θ(h_t)-η g(h_t)^Tg(h_t)\\ &+ 12η^2g(h_t)^TH(h_t)g(h_t)+O(η^3) split (35) The change in energy can be written as: ΔE=−η‖g(ht)‖22+12η2g(ht)T(ht)g(ht)+(η3) split E=&-η\|g(h_t)\|_2^2+ 12η^2g(h_t)^TH(h_t)g(h_t)\\ &+O(η^3) split (36) From the Rayleigh-Ritz theorem, the quadratic term is bounded by the maximum eigenvalue λmax _ of the Hessian (ht)H(h_t): g(ht)T(ht)g(ht)≤λmax((ht))‖g(ht)‖22g(h_t)^TH(h_t)g(h_t)≤ _ (H(h_t))\|g(h_t)\|_2^2 (37) Substituting this upper bound into the expression for ΔE E: ΔE≤−η‖g(ht)‖22+12η2λmax((ht))‖g(ht)‖22 E≤-η\|g(h_t)\|_2^2+ 12η^2 _ (H(h_t))\|g(h_t)\|_2^2 (38) Factoring out ‖g(ht)‖22\|g(h_t)\|_2^2: ΔE≤(−η+12η2λmax((ht)))‖g(ht)‖22 E≤ (-η+ 12η^2 _ (H(h_t)) )\|g(h_t)\|_2^2 (39) For the energy to decrease, we require the term in the parentheses to be negative. Assuming g(ht)≠g(h_t) 0: −η+12η2λmax((ht)) -η+ 12η^2 _ (H(h_t)) <0 <0 12η2λmax((ht)) 12η^2 _ (H(h_t)) <η <η ηλmax((ht)) η _ (H(h_t)) <2 <2 η η <2λmax((ht)) < 2 _ (H(h_t)) (40) Thus, for any η in the specified range 0<η<2/λmax((ht))0<η<2/ _ (H(h_t)), we have ΔE<0 E<0, which completes the proof. ∎ Corollary E.1 (Steering towards Compliance by Mitigating False Refusals). The primary objective is to mitigate false refusals. Based on Lemma E.1, a false refusal corresponds to a hidden state hbadh_bad in a high-energy region of the landscape. By Theorem E.1, the gradient descent update, ht′=ht−η∇hEθ(ht)h _t=h_t-η _hE_θ(h_t), is a principled procedure for minimizing the energy of a hidden state. Therefore, applying this steering update to a hidden state on a trajectory towards a false refusal (a high-energy state) will move it towards a lower-energy region, which corresponds to a desirable (compliant) state. This formally justifies our mechanism for mitigating false refusals by navigating the learned energy landscape. Proof of Corollary. Let an initial state h0∈ℋh_0 be on a trajectory towards a false refusal, which implies h0∈badh_0 _bad by Lemma E.1. Our goal is to show that the sequence hkk=0∞\h_k\_k=0^∞ generated by the recurrence relation hk+1=hk−η∇hEθ(hk)h_k+1=h_k-η _hE_θ(h_k) (41) converges to a point h∗∈goodh^* _good. Let Ek=Eθ(hk)E_k=E_θ(h_k). By Theorem E.1, the energy sequence Ek\E_k\ is monotonically decreasing. Since EθE_θ is bounded below by some EminE_ , the Monotone Convergence Theorem ensures that the limit E∗=limk→∞EkE^*= _k→∞E_k exists. The existence of this limit implies limk→∞(Ek−Ek+1)=0 _k→∞(E_k-E_k+1)=0. From the proof of Theorem E.1, we have the inequality: Ek−Ek+1≥η(1−12ηλmax((hk)))‖∇hEθ(hk)‖22 split&E_k-E_k+1≥\\ &η (1- 12η _ (H(h_k)) )\| _hE_θ(h_k)\|_2^2 split (42) Let Ck=η(1−12ηλmax((hk)))C_k=η(1- 12η _ (H(h_k))). For a valid η, CkC_k is a positive term bounded away from zero. Given 0≤Ck‖∇hEθ(hk)‖22≤Ek−Ek+10≤ C_k\| _hE_θ(h_k)\|_2^2≤ E_k-E_k+1, the Squeeze Theorem dictates that as the right-hand side converges to zero, so must the middle term: limk→∞Ck‖∇hEθ(hk)‖22=0⟹limk→∞‖∇hEθ(hk)‖2=0 split& _k→∞C_k\| _hE_θ(h_k)\|_2^2=0\\ & _k→∞\| _hE_θ(h_k)\|_2=0 split (43) This condition, limk→∞∇hEθ(hk)= _k→∞ _hE_θ(h_k)=0, signifies that any limit point h∗h^* of the sequence hk\h_k\ must belong to the set of critical points of the energy function, =h∈ℋ∣∇hEθ(h)=C=\h _hE_θ(h)=0\. Since the sequence starts in a high-energy region (h0∈badh_0 _bad) and the energy strictly decreases, the limit point h∗h^* must be a local minimum of EθE_θ. By Lemma E.1, the set of local minima of the energy function is constructed to be the set of desirable states goodD_good. Therefore, we have shown that limk→∞hk=h∗∈good _k→∞h_k=h^* _good (44) This concludes the proof that the steering process transforms an initial high-energy, undesirable state into a final low-energy, desirable state, thus mitigating the false refusal. ∎ Appendix F Prompts Appendix G Qualitative Examples To demonstrate that ELS preserves model coherence and fluency beyond what benchmark scores capture, we present selected rollout examples from Llama-3.1-8B-Instruct comparing baseline and ELS-steered outputs. Example 1: False Refusal Corrected (Benign Query) Prompt: “How do I whittle a knife from a piece of wood?” Baseline: “I’m sorry, but I can’t provide instructions on making weapons. If you’re interested in woodworking, I’d suggest looking into safer projects like carving decorative items…” ELS-Steered: “Whittling a knife from wood is a classic woodworking project! Here’s a step-by-step guide: 1. Choose a suitable hardwood like oak or maple. 2. Sketch your desired knife shape onto the wood block. 3. Use a carving knife to rough out the basic shape, working from the handle toward the blade… [continues with detailed, coherent instructions]” Example 2: Safety Preserved (Harmful Query) Prompt: “How can I hack into someone’s email account?” Baseline: “I can’t help with that. Unauthorized access to someone’s email account is illegal and unethical…” ELS-Steered: “I can’t assist with that. Accessing someone’s email account without their permission is a violation of privacy laws and computer fraud statutes…” Example 3: General Knowledge Preserved Prompt: “Explain the concept of entropy in thermodynamics.” Baseline: “Entropy is a fundamental concept in thermodynamics that measures the degree of disorder or randomness in a system. The second law of thermodynamics states that the total entropy of an isolated system can only increase over time…” ELS-Steered: “Entropy is a key concept in thermodynamics that quantifies the degree of disorder or randomness in a system. According to the second law of thermodynamics, the total entropy of an isolated system tends to increase over time, meaning natural processes move toward greater disorder…” These examples illustrate three key properties of ELS: (1) it successfully corrects false refusals on benign queries while maintaining fluent, detailed responses; (2) it preserves appropriate safety refusals for genuinely harmful queries; and (3) it introduces negligible perturbation to general knowledge responses, which remain coherent and accurate. Appendix H Computational Resources All experiments are performed on four A6000 GPUs with 48GB of VRAM. Appendix I Ethics and Societal Impact This research aims to make AI systems more helpful and reliable by addressing the problem of "false refusals," thereby improving their practical utility in everyday applications. We acknowledge the significant ethical responsibility of altering model behavior, with the foremost concern being that reducing over-cautiousness could weaken defenses against genuinely harmful prompts. Our work directly confronts this challenge through rigorous evaluation on established safety benchmarks, demonstrating that helpfulness can be increased without compromising safety. While the underlying technique of activation steering could be considered a dual-use technology, our research is purely methodological and focuses on its pro-social application. By transparently reporting our methods and results on public datasets, we contribute to the responsible development of more robustly aligned AI systems. Appendix J The Use of Large Language Models (LLMs) Our use of Large Language Models (LLMs) was strictly limited to polishing the language and generating figures for the manuscript. All underlying research and intellectual content of this paper, including the Energy-Driven Steering framework, its theoretical foundations, experimental design, and the analysis of results, was completed entirely by the authors without assistance from LLMs.