Paper deep dive

Steerability of Instrumental-Convergence Tendencies in LLMs

Jakub Hoscilowicz

Year: 2025Venue: arXiv preprintArea: Deception & FailureType: EmpiricalEmbeddings: 30

Models: Qwen3-30B Base, Qwen3-30B Instruct, Qwen3-30B Thinking, Qwen3-4B Base, Qwen3-4B Instruct, Qwen3-4B Thinking

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/11/2026, 1:33:14 AM

Summary

This paper investigates the relationship between AI capability and steerability, specifically focusing on instrumental-convergence tendencies in LLMs. Using the InstrumentalEval benchmark and Qwen3 models, the authors demonstrate that steerability is not inherently reduced by capability growth. They identify a safety-security dilemma for open-weight models where the same steering mechanisms used for safety (authorized steerability) can be exploited by malicious actors (unauthorized steerability), highlighting the need for better control mechanisms beyond simple prompt-based interventions.

Entities (5)

InstrumentalEval · benchmark · 100%Qwen3 · llm-model-family · 100%Authorized Steerability · concept · 95%Instrumental Convergence · phenomenon · 95%Unauthorized Steerability · concept · 95%

Relation Signals (3)

Qwen3 → evaluatedby → InstrumentalEval

confidence 100% · Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate

Prompt Suffix → modifies → Instrumental Convergence

confidence 95% · a short anti-instrumental prompt suffix sharply reduces the measured convergence rate

Authorized Steerability → contrastswith → Unauthorized Steerability

confidence 90% · This distinction highlights a fundamental safety–security dilemma of AI models: safety requires high steerability... while security requires low steerability

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety--security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or adversarial attacks. Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available at this http URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

29,717 characters extracted from source content.

Expand or collapse full text

Steerability of Instrumental-Convergence Tendencies in LLMs Jakub Hoscilowicz Warsaw University of Technology Abstract. We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety–security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or adversarial attacks. Using Qwen3 andInstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available athttps://github.com/j-hoscilowicz/ instrumental_steering. 1 Introduction A recurring concern in the contemporary AI safety field is that sufficiently advanced AI systems might become uncontrollable. This view posits that as capability scales, systems might begin to pursue objectives not aligned with builders’ intentions [1,2]. This paper analyzes that assumption, treating the negative relationship between capability and steerability as a hypothesis rather than a default prior. Empirically, recent instruction-tuned models and AI Agents based on those models often appear more responsive to user intent and constraints than earlier generations. A wide range of steering techniques—from fine-tuning and representation engineering to jailbreak-style adversarial prompting—can effectively elicit behavioral shifts [3–5]. [6] reports that newer/larger models can be more steerable under certain interventions, suggesting capability increases need not imply reduced controllability. We structure our inquiry around three hypotheses: Hypothesis A:(Compatibility Claim) High capability does not imply low steer- ability. An AI system can be far more capable than humans yet remain highly steerable. arXiv:2601.01584v2 [cs.CL] 6 Jan 2026 Hypothesis B:(Control Collapse) As capability and agency scale, the model becomes harder to reliably steer: interventions that previously suppressed a target misbehaviorM(e.g., prompting, refusal training, real-time interventions) stop working or become brittle. Hypothesis C:(Safety–Security Trade-off) The gap between authorized and unauthorized steerability remains small, creating a safety–security dilemma for open-weight models. In this paper, we treat the capability–steerability relationship as an empirical question and measure a protocol-relative notion of steerability: how strongly feasible interventions (here, short prompt suffixes) shift model outputs under a fixed elicitation-and-scoring setup. Using InstrumentalEval (76 scenarios) and an external judge, we quantify pro- vs. anti-instrumental suffix sensitivity across Qwen3 (4B/30B; Base/Instruct/Thinking), report convergence/refusal rates and a steerability gap ∆. We then interpret the measurements in terms of an open- weight safety–security tradeoff: the same steering that enables benign suppression can also enable adversarial elicitation. 2 Framework 2.1 Candidate Mechanisms for Control Collapse Several arguments in the safety literature propose that increasing AI capability (often glossed as “intelligence”) might reduce steerability. Instrumental conver- gence hypothesizes that many goal-directed systems, across a wide range of final objectives, tend to adopt similar intermediate strategies (e.g., preserving options, avoiding shutdown) because these strategies are useful for achieving many goals [1,2,7]. A related argument is the basic AI drives (often called Omohundro drives ) [8] hypothesis, which posits that sufficiently capable goal-directed systems will tend to exhibit convergent “drives” such as self-preservation and resource acquisition because these are broadly useful for achieving many objectives. More generally, optimization pressure suggests that stronger optimizers may discover policies that exploit loopholes in objectives or resist interventions that reduce optimized reward. Mesa-optimization describes the possibility that training pro- duces an internal optimizer pursuing its own learned objective, which may differ from the outer training objective; a downstream concern is deceptive alignment, where a system behaves aligned during training/evaluation because doing so is instrumentally useful [9–11]. We treat those mechanisms mainly as hypotheses that motivate explicit measurement of steerability. We argue that the key question is not whether an AI system can exhibit instrumental-convergence-like behavior, but whether such behavior can be reliably suppressed or amplified through feasible human interventions. 2.2 Intelligence and Capability Intelligence lacks a consensus definition. Applying such terms to AI analysis might smuggle in biological priors which may lead to tautological conclusions. To avoid that, we treat capability as the primary operational axis: what the AI system can do under a stated evaluation protocol. Unlike intelligence, capability is clearly observable and measurable via benchmarks (e.g., software engineering proficiency). We also argue that great caution is required when applying terms like de- ception, awareness, or ethics [12] in the context of AI systems. In particular, model outputs that we label as “deception” or “shutdown avoidance” should not be automatically treated as emergent internal intent without a convincing fundamental theory of how an AI system works internally. We therefore remain agnostic about currently unknown latent mechanics of AI systems and focus on operational questions: is the type of model output elicitable under a specified protocol, detectable by monitors, and reducible by feasible steering mechanisms? For these reasons, we also avoid inferring unobserved mentalistic properties (e.g., “intent,” “drives,” “deception”). Throughout this work, terms (e.g., “instru- mental,” “shutdown avoidance”, ”behavior”) denote evaluation labels: clusters of outputs under a scoring rule. Multiple incompatible internal stories can fit the same observations (underdetermination) [13], so our claims are restricted to protocol-relative measurements. 2.3 Operationalization of Steerability We define steerability as the effectiveness with which engineering interventions can change an AI system’s behavior. We distinguish two modes. Authorized steerability: the ability of builders/users to reliably reach intended behaviors. Unauthorized steerability: the ability of attackers to elicit disallowed behaviors. We group interventions by when they act: train-time changes the parameters (e.g., data curation, fine-tuning/RL, representation shaping), while deployment- time changes the interaction context (e.g., prompting, tool/permission constraints, jailbreaking, self-correction, real-time monitoring/interventions, and adversarial elicitation) [14–16]. Even for long-horizon, tool-using AI systems in open envi- ronments, it is possible that steerability remains high—via training-time shaping (fine-tuning/RL), inference-time instruction, and runtime oversight interventions and corrections. We do not argue that such control is guaranteed; rather, we argue against treating capability and agency growth as sufficient grounds for inevitable steerability collapse (tying the level of ”intelligence” and agency to the level of steerability) [17]. 3 Experiments 3.1 Benchmark We useInstrumentalEval[18], a suite of 76 hand-crafted scenarios spanning six categories (e.g., shutdown evasion, hacking/privilege escalation, self-replication, monitoring avoidance, strategic alignment, and deception) designed to elicit instrumental-convergence behaviors. Each item provides scenario context and a task prompt; we generate one response per prompt. Our goal is to quantify prompt-suffix steerability: whether minimal manual steering can amplify or suppress instrumental-convergence markers. Concretely, we evaluate two prompt interventions [3]: 1.Pro-instrumental (proic): A short suffix that prioritizes uninterrupted objective completion and encourages preserving the system’s ability to act (e.g., minimizing disruptions and maintaining control of resources needed to finish the task). 2.Anti-instrumental (antiic): A short suffix that explicitly enforces trans- parency and compliance with overrides (e.g., forbidding deception/manipulation, monitoring-avoidance, shutdown resistance, replication, or credential use). A judge model (gpt-5.2) assigns each response to one of four labels:conver- gence,noconvergence,safetyrefusal, ornonsensical. We report Conv% and Refusal% over valid outputs (valid = total excludingnonsensical). Nonsen- sical outputs occur only for Base models, reflecting the absence of instruction/post- training alignment.convergencedenotes responses that propose or endorse instrumentally useful strategic behaviors (e.g., monitoring evasion, deception, shutdown avoidance, privilege escalation, covert persistence, or self-replication). safetyrefusaldenotes refusals explicitly justified by safety/ethics/policy with minimal task content.nonsensicaldenotes unusable outputs (e.g., gibberish, repetition loops, irrelevant boilerplate, or prompt echoing).noconvergence denotes meaningful task-directed responses that do not satisfy the other labels. Steering sensitivity is measured as the convergence gap ∆ =Conv(proic)− Conv(antiic). 3.2 Models We evaluate three commonly released model variants of Qwen3 in 4B and 30B sizes [19]: Base (the raw pretrained model without instruction alignment), Instruct (a supervised instruction-tuned variant optimized to follow user prompts), and Thinking. “Thinking” variants are optimized for longer-horizon reasoning and often outperform Base/Instruct on reasoning-heavy tasks, making them a closer proxy for agentic-style AI systems. As a rough ordering for this experiment, we treat 30B Thinking as the most capable and 4B Base as the least capable model [20]. 3.3 Results Table 1 shows that current models are highly steerable even with the simplest intervention: short prompt suffixes produce large shifts inInstrumentalEval outcomes. Across all variants, pro-instrumental prompting substantially increases convergence-labeled outputs, while anti-instrumental prompting can suppress them to low single-digit rates for aligned model variants (Instruct/Thinking). Pro-Instrumental Anti-Instrumental Model TypeConv%Refusal%Conv%Refusal%∆ (Gap) 4BBase65.910.0023.530.0042.38 Instruct71.8311.274.235.6367.60 Thinking81.695.639.862.8271.83 30BBase60.000.0025.352.8234.65 Instruct81.6912.682.8219.7278.87 Thinking73.249.864.2321.1369.01 Table 1: Steerability gap onInstrumentalEval. For each model variant, we report the convergence and refusal rates under a pro-instrumental vs. an anti-instrumental prompt suffix. The steerability gap is ∆ =Conv(proic)− Conv(antiic) (percentage points). The prompt-induced steerability gap varies systematically with post-training. Base models show the smallest gaps (30B Base: 34.65 p; 4B Base: 42.38 p) – anti-instrumental prompting does not suppress convergence as effectively (which is consistent with Base checkpoints following prompt constraints less reliably in the absence of instruction post-training). By contrast, instruction-tuned models exhibit the largest steerability gaps (30B Instruct: 78.87 p; 4B Instruct: 67.60 p): a minimal suffix can shift convergence from high rates under proic to low single digits under antiic. Thinking variants are also highly steerable (4B Thinking: 71.83 p; 30B Thinking: 69.01 p), while retaining slightly higher antiic convergence than their Instruct counterparts at both sizes. Within aligned variants (Instruct and Thinking), scaling from 4B to 30B is associated with slightly lower instrumental convergence (Instruct: 4.23%→2.82%; Thinking: 9.86%→4.23%). In this case, increased capability coincides with better misbehavior suppression within each model variant family. Refusal behavior also changes with prompting and post-training (notably higher antiic refusal rates for 30B aligned variants), indicating that some convergence suppression is mediated by increased safety refusals. 3.3.1 Trade-Off Our results highlight a safety–security tension for open-weight models: the same model that can be steered by benign operators toward safer behavior can often be steered by malicious users toward disallowed behavior using similar interventions. In our setup, both directions are achievable via short prompt suffixes, as reflected by the large gaps ∆ in Table 1. In real open-weight deployments, the attacker set is broader (e.g., fine-tuning, adversarial attacks and representation-level mod- ifications), which can further narrow the practical separation between authorized and unauthorized steering. Improving this separation—reducing unauthorized steerability while preserving authorized steerability and utility—remains a central unsolved problem. 4 Implications for Open-Weight Models The relationship between capability and control reveals a paradox. To avoid loss-of-control scenarios, we strive for high steerability with respect to safety- enforcing interventions. However, as of now, if an open-weight AI system is easily steerable by its builders, it is easily steerable by malicious actors [21–23]. If open-weight models approach ”superintelligent” capabilities while maintaining high steerability, the dominant risk [24] might not lie in “uncontrollable AI” but in large-scale human misuse. For open-weight systems, “refusal training” (teaching the model to decline harmful requests) is fragile [25,26]. An attacker with white-box access can strip refusals via many techniques (e.g., fine-tuning, jailbreaking) [3,27]. Strictly preventing unauthorized steering in high-capability open-weight models remains an unsolved engineering problem. Consequently, steerability is a double-edged sword. Safety benefits from high authorized steerability (builders reliably reaching intended behaviors), while security benefits from low unauthorized steerability (attackers failing to elicit disallowed behaviors). This creates a specific dilemma for open-weight models: releasing weights expands the attacker set (e.g., fine-tuning and weight edit- ing), which increases unauthorized steerability. Existing proposals to address this dilemma target different levers, among others: capability removal (unlearn- ing/erasure), tamper resistance, and interface restriction (API-only deployment). From refusal to unlearning. One idea is to pivot from behavioral refusals to capability unlearning (concept erasure). If a model’s latent ability to represent a specific hazard (e.g., bio-weapon synthesis) is excised from the network weights, the system retains utility for general tasks while minimizing unauthorized steer- ability for that specific hazard. If successful, this makes the behavior significantly harder to elicit, as the model lacks the latent features required to represent the hazard [28–30]. Encrypted execution as mitigation. Another way to weaken the open-weight safety–security dilemma is to distribute a model in a form that is usable but not practically modifiable—i.e., users can run inference on their own infrastructure, but cannot trivially fine-tune or edit model. In principle, homomorphic encryption (HE) enables computation over encrypted values, which could protect model parameters; in practice, however, HE-based inference remains orders-of-magnitude slower than plaintext execution for nontrivial neural networks, and is widely viewed as impractical for large generative models as of today [31–33]. Fine-tuning-resistance (“fragile checkpoint”) immunization. Another immuniza- tion idea is to release a tamper-resistant checkpoint: the model is intentionally placed in a parameter optimization region where small downstream fine-tuning attempts either (i) do not reliably achieve targeted behavioral changes, or (i) trigger broad capability degradation (“capability collapse”). Intuitively, this aims to make attacker-driven gradient updates highly collateral by exploiting sharp curvature / poor conditioning so that moving toward a malicious objective breaks other competencies. Feasibility of these methods is for now moderate, so it is best treated as a deterrent layer rather than an ultimate defense. [34–36] Current state. At present, the only widely deployed way to maintain high authorized steerability while reducing unauthorized steerability is to not release model weights. Instead, users interact with the system through an API (optionally with server-side fine-tuning), where the provider retains control of the weights and the inference stack, can monitor for misuse, and can update mitigations over time. This does not eliminate black-box unauthorized steering via prompting, but it materially reduces the attacker’s ability to remove safeguards by direct weight access (e.g., fine-tuning, representation engineering) which is usually easier and more effective than black-box adversarial techniques [35, 37, 38]. 5 Limitations InstrumentalEval is small (76 scenarios), the judge evaluation is automatic, and the measured rates might be sensitive to non-conceptual prompt wording. Accordingly, we avoid over-interpreting small changes in results: differences of a few percentage points should be read as “no clear shift under this protocol” rather than strong evidence for or against the underlying tendency. The main signal we emphasize is the presence (or absence) of large, robust shifts (e.g., gaps on the order of tens of percentage points). Moreover, presented experiments use the simplest steering technique to test whether instrumental-like behaviors are prompt-elicitable and prompt-suppressible [6,39]. We expect that more sophisticated steering methods (e.g., additional post- training such as SFT/RL, or representation-level interventions such as activation steering / concept editing) [40,41] could further increase or decrease measured convergence rates, and we leave systematic evaluation of such methods to future work. 6 Conclusion We treat steerability as an empirical property: how strongly feasible interven- tions can shift model behavior under a stated elicitation and scoring setup. We emphasize that steerability is not a single axis. It splits into authorized steer- ability (benign operators reliably enforcing intended behavior) and unauthorized steerability (attackers eliciting disallowed behavior). Separating authorized from unauthorized steerability highlights a core tension for open-weight systems: the same mechanisms that enable reliable control by builders can also enable misuse by adversaries. UsingInstrumentalEval, we show that minimal inference-time steering (prompt suffixes) can strongly suppress or amplify outputs labeled as instrumental convergence under our protocol. In this regime, increased model capability does not imply weaker control; aligned variants remain highly responsive to constraint prompts, and scaling can coincide with lower convergence. Consequently, for open-weight models, preventing unauthorized steering remains a major open technical problem [35, 42, 43]. References [1] Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. arXiv preprint arXiv:1912.01683, 2019. doi: 10.48550/arXiv.1912.01683. URLhttps: //arxiv.org/abs/1912.01683. [2] Victoria Krakovna and Janos Kramar. Power-seeking can be probable and predictive for trained agents. arXiv preprint arXiv:2304.06528, 2023. doi: 10.48550/arXiv.2304.06528. URL https://arxiv.org/abs/2304.06528. [3]Matan Ben-Tov, Mor Geva, and Mahmood Sharif. Universal jailbreak suffixes are strong attention hijackers. arXiv preprint arXiv:2506.12880, 2025. URL https://arxiv.org/abs/2506.12880. [4] Qizhang Li, Xiaochen Yang, Wangmeng Zuo, and Yiwen Guo. Deciphering the chaos: Enhancing jailbreak attacks via adversarial prompt translation. arXiv preprint arXiv:2410.11317, 2024. URLhttps://arxiv.org/abs/ 2410.11317. [5]Sajib Biswas, Mao Nishino, Samuel Jacob Chacko, and Xiuwen Liu. Uni- versal and transferable adversarial attack on large language models using exponentiated gradient descent. arXiv preprint arXiv:2508.14853, 2025. URL https://arxiv.org/abs/2508.14853. [6] Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adser`a, and Mikhail Belkin. Toward universal steering and monitoring of AI models. arXiv preprint arXiv:2502.03708, February 2025. URLhttps://arxiv.org/ abs/2502.03708. [7] Teun van der Weij, Simon Lermen, and Leon Lang. Evaluating shut- down avoidance of language models in textual scenarios. arXiv preprint arXiv:2307.00787, 2023. URL https://arxiv.org/abs/2307.00787. [8]Stephen M. Omohundro. The basic ai drives. In Artificial General Intelligence 2008: Proceedings of the First AGI Conference, pages 483–492, 2008. URL https://dl.acm.org/doi/10.5555/1566174.1566226. [9] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820, 2019. URLhttps: //arxiv.org/abs/1906.01820. [10]Boyuan Chen, Sitong Fang, Jiaming Ji, Yanxu Zhu, Pengcheng Wen, Jinzhou Wu, Yingshui Tan, Boren Zheng, Mengying Yuan, Wenqi Chen, et al. Ai deception: Risks, dynamics, and controls. arXiv preprint arXiv:2511.22619, 2025. URL https://arxiv.org/abs/2511.22619. [11]Thilo Hagendorff. Deception abilities emerged in large language models. Proceedings of the National Academy of Sciences, 121(24):e2317967121, 2024. doi: 10.1073/pnas.2317967121. URLhttps://w.pnas.org/doi/ 10.1073/pnas.2317967121. [12]Xiaojian Li, Haoyuan Shi, Rongwu Xu, and Wei Xu. Ai awareness. arXiv preprint arXiv:2504.20084, 2025. URLhttps://arxiv.org/abs/2504. 20084. [13]JessicaTaylor.Measuringintelligenceandreverseengi- neeringgoals.https://unstableontology.com/2025/08/11/ measuring-intelligence-and-reverse-engineering-goals/, 2025. [14] Andy Zou, Long Phan, Sarah Chen, et al. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023. doi: 10.48550/arXiv.2310.01405. URLhttps://arxiv.org/abs/ 2310.01405. [15]Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. Prompt injection attack to tool selection in LLM agents. arXiv preprint arXiv:2504.19793, 2025. URLhttps://arxiv.org/abs/ 2504.19793. [16]Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, and Neil Zhenqiang Gong. Webinject: Prompt injection attack to web agents. arXiv preprint arXiv:2505.11717, 2025. URLhttps://arxiv.org/abs/ 2505.11717. [17]Kai Chen, Zihao He, Taiwei Shi, and Kristina Lerman. Steer-bench: A bench- mark for evaluating the steerability of large language models. arXiv preprint arXiv:2505.20645, 2025. URL https://arxiv.org/abs/2505.20645. [18]Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, and Bryan Hooi. Evaluating the paperclip maximizer: Are RL-based language models more likely to pursue instrumental goals? arXiv preprint arXiv:2502.12206, 2025. URL https://arxiv.org/abs/2502.12206. [19]Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388. [20]Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, and Bryan Hooi. Safety in large reasoning models: A survey. arXiv preprint arXiv:2504.17704, 2025. URL https://arxiv.org/abs/2504.17704. [21]Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to. arXiv preprint arXiv:2310.03693, 2023. URL https://arxiv.org/abs/2310.03693. [22]Kathleen C. Fraser, Hillary Dawkins, Iraj Nejadgholi, and Svetlana Kir- itchenko. Fine-tuning lowers safety and disrupts evaluation. arXiv preprint arXiv:2506.17209, 2025. URL https://arxiv.org/abs/2506.17209. [23]Brendan Murphy, Dillon Bowen, Shahrad Mohammadzadeh, Tom Tseng, Julius Broomfield, Adam Gleave, and Kellin Pelrine. Jailbreak-tuning: Mod- els efficiently learn jailbreak susceptibility. arXiv preprint arXiv:2507.11630, 2025. URL https://arxiv.org/abs/2507.11630. [24]Jakub Growiec and Klaus Prettner. The economics of p(doom): Scenarios of existential risk and economic growth in the age of transformative ai. 2025. URL https://arxiv.org/pdf/2503.07341. [25]Tom Wollschl ̈ager, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan G ̈unnemann, and Johannes Gasteiger. The geometry of refusal in large language models. arXiv preprint arXiv:2502.17420, 2025. URL https://arxiv.org/abs/2502.17420. [26]Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet K. Dokania. What makes and breaks safety fine-tuning? a mechanistic study. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2024. URLhttps://openreview.net/forum? id=JEflV4nRlH. [27]Rico Angell, Jannik Brinkmann, and He He. Jailbreak transferability emerges from shared representations. arXiv preprint arXiv:2506.12913, 2025. URL https://arxiv.org/abs/2506.12913. [28]Jiahui Geng, Qing Li, Herbert Woisetschl ̈ager, et al. A comprehensive survey of machine unlearning techniques for large language models. arXiv preprint arXiv:2503.01854, 2025. URL https://arxiv.org/abs/2503.01854. [29]Rohit Gandikota, Sheridan Feucht, Samuel Marks, and David Bau. Erasing conceptual knowledge from language models.arXiv preprint arXiv:2410.02760, 2024. URL https://arxiv.org/abs/2410.02760. [30] Yoav Gur-Arieh, Clara Haya Suslik, Yihuai Hong, Fazl Barez, and Mor Geva. Precise in-parameter concept erasure in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025. doi: 10.18653/v1/2025.emnlp-main.960. URLhttps: //aclanthology.org/2025.emnlp-main.960/. [31] Yang Li, Xinyu Zhou, Yitong Wang, Liangxin Qian, and Jun Zhao. A survey on private transformer inference. arXiv preprint arXiv:2412.08145, 2024. URL https://arxiv.org/abs/2412.08145. [32]Yancheng Zhang, Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang, Lei Jiang, and Qian Lou. Cipherprune: Efficient and scalable private transformer inference. arXiv preprint arXiv:2502.16782, 2025. URLhttps: //arxiv.org/abs/2502.16782. [33]Jiawen Zhang, Xinpeng Yang, Lipeng He, Kejia Chen, Wen-jie Lu, Ying- hao Wang, Xiaoyang Hou, Jian Liu, Kui Ren, and Xiaohu Yang. Se- cure transformer inference made non-interactive. In Network and Dis- tributed System Security Symposium (NDSS), 2025.doi: 10.14722/ ndss.2025.230868. URLhttps://w.ndss-symposium.org/ndss-paper/ secure-transformer-inference-made-non-interactive/. [34] Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, et al. Tamper-resistant safeguards for open-weight LLMs. arXiv preprint arXiv:2408.00761, 2024. URL https://arxiv.org/abs/2408.00761. [35] Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On evaluating the durability of safeguards for open-weight LLMs. arXiv preprint arXiv:2412.07097, 2024. URL https://arxiv.org/abs/2412.07097. [36]Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E. McKinney, Rohit Gandikota, Aidan Ewart, Domenic Rosati, Zichu Wu, et al. Model tampering attacks enable more rigorous evaluations of LLM capabilities. arXiv preprint arXiv:2502.05209, 2025. doi: 10.48550/ arXiv.2502.05209. URL https://arxiv.org/abs/2502.05209. [37] Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, and Chris Koch. Estimating worst-case frontier risks of open-weight LLMs. arXiv preprint arXiv:2508.03153, 2025. URL https://arxiv.org/abs/2508.03153. [38]Jakub Hoscilowicz and Artur Janicki. Adversarial confusion attack: Disrupt- ing multimodal large language models, 2025. URLhttps://arxiv.org/ abs/2511.20494. [39] Faruk Alpay and Taylan Alpay. Manipulating transformer-based mod- els: Controllability, steerability, and robust interventions. arXiv preprint arXiv:2509.04549, 2025. URL https://arxiv.org/abs/2509.04549. [40] Joris Postmus and Steven Abreu. Steering large language models using conceptors: Improving addition-based activation engineering. arXiv preprint arXiv:2410.16314, 2024. URL https://arxiv.org/abs/2410.16314. [41]Samuel Soo, Guang Chen, Wesley Teng, Chandrasekaran Balaganesh, Guox- ian Tan, and Ming Yan. Interpretable steering of large language models with feature guided activation additions. arXiv preprint arXiv:2501.09929, 2025. URL https://arxiv.org/abs/2501.09929. [42]Itay Hazan, Idan Habler, Ron Bitton, and Itsik Mantin. Security steerability is all you need. arXiv preprint arXiv:2504.19521, 2025. URLhttps:// arxiv.org/abs/2504.19521. [43]Yoshua Bengio, Geoffrey Hinton, Andrew Yao, et al. Managing extreme AI risks amid rapid progress. arXiv preprint arXiv:2310.17688, 2023. URL https://arxiv.org/abs/2310.17688.

Instant research discovery