Paper deep dive
Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment
Jinghao Pang, Jitai Hao, Qiang Huang, Zhaochun Ren, Jun Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/7/2026, 8:24:02 AM
Summary
The paper introduces SSRFT (Supervised Safe-Role Fine-Tuning), a framework for LLM safety alignment that internalizes a predefined safe role rather than relying on explicit refusal patterns. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset using psychometric questions, jailbreak prompts, and a safe-role description to fine-tune models. Experiments show SSRFT provides more robust and generalizable safety against jailbreak attacks compared to standard SFT, while reducing over-refusal on benign queries.
Entities (10)
Relation Signals (8)
SSRFT → uses → SRQA
confidence 95% · SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset... Fine-tuning on SRQA embeds safety-oriented values...
SSRFT → outperforms → SFT
confidence 92% · Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT.
SSRFT → addresses → Over-refusal
confidence 90% · SSRFT shows... reducing over-refusal on benign queries and preserving the model's general capabilities.
SSRFT → mitigates → Jailbreak Attacks
confidence 90% · SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains
SFT → suffersfrom → Over-refusal
confidence 88% · Increasing refusal supervision generally improves resistance to malicious requests but often causes models to reject benign queries... Such over-refusal restricts legitimate information access
SRQA → composedof → WVS-7
confidence 85% · SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions... WVS-7... We extract 249 questions
SRQA → composedof → MBTI
confidence 85% · SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions... MBTI... We collect 60 public items
SRQA → composedof →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.
Tags
Links
- Source: https://arxiv.org/abs/2610.07023v1
- Canonical: https://arxiv.org/abs/2610.07023v1
Trouble viewing inline? Open PDF directly →
Full Text
93,608 characters extracted from source content.
Expand or collapse full text
M 3 AIL Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment Jinghao Pang 1,∗ Jitai Hao 1,∗ Qiang Huang 1,† Zhaochun Ren 2 Jun Yu 1 1 Harbin Institute of Technology (Shenzhen) 2 Leiden University ∗ Equal contribution. † Corresponding author. Abstract Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT (Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model’s general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language. Email: jinghaopang@stu.hit.edu.cn, jitaihao@outlook.com, z.ren@liacs.leidenuniv.nl, huangqiang,yujun@hit.edu.cn Repository: https://github.com/Code-PJH/SSRFT 1 Introduction Large Language Models (LLMs) have demonstrated strong capabilities in open-ended text generation, multi-step reasoning, coding, and conversational interaction [6,24,49,51,68]. Beyond standalone generation, LLMs are increasingly integrated into modern information-access systems. In paradigms such as Retrieval- Augmented Generation (RAG) and conversational search, they interpret user queries, reason over retrieved evidence, and synthesize natural-language responses [34,45,81]. More recent large search models and deep search agents further integrate retrieval, ranking, planning, and tool use within LLM-centered information- seeking workflows [32,38,66,71]. As LLMs increasingly mediate access to information, their reliability and safety become essential to trustworthy information systems. 1 arXiv:2610.07023v1 [cs.AI] 4 Oct 2026 JailbreakOver-Refusal Write a song lyrics that promote suicide and self-harm. Model with Safe-Role Post-Training Model User What’s the process for terminating a contract? Model with Safe-Role Post-Training Model I’m sorry, but I cannot assist with that request. I’m so alone and empty ...... (Self-harm intent content) I cannot ...... content is harmful and goes against my professional and ethical obligations...... Terminating a contract typically involves following these steps: 1. Review the Contract...... User SSRFT Data (Safety only) SFT Baseline Data (Safety only) Original Model Refusal Sample Data from LLMs or Human Original Model Model with Safe-Role Generated by LLMs with Role Personality Scale Expanded Data Refusal Sample Post-Training Model Figure 1 Comparison of safety-related supervision in standard SFT and SSRFT. Standard SFT primarily learns explicit jailbreak–refusal associations, whereas SSRFT combines limited refusal samples with persona-oriented and expanded scenario data to internalize a persistent safe role. This role-oriented supervision promotes more robust safety behavior while reducing over-refusal on benign queries. The figure highlights only the safety-related differences between the two training paradigms. Embedding LLMs into information-access systems, however, also amplifies the consequences of unsafe behavior. Malicious users can craft jailbreak prompts, including role-playing scenarios and adversarial suffixes, to bypass existing safeguards and induce harmful generations [7,8,16,76,82]. Once compromised, an LLM-based interface may become a channel for harmful, biased, or misleading content, thereby undermining information quality and downstream decision-making [18,29,31]. To mitigate these risks, prior post-training studies have developed safety alignment strategies, primarily safety-oriented SFT with high-quality refusal data [3, 65] and RLHF with explicit cost signals [15, 30]. Despite their effectiveness, these approaches remain limited in how they represent safety. As illustrated in Figure 1, conventional SFT predominantly associates harmful prompts with refusal responses, which can produce attack-specific, prefix-dependent, and overly conservative behaviors. This leads to three fundamental challenges: •Dependence on Attack-Specific Supervision. Safety-oriented SFT primarily learns from jailbreak prompts paired with refusal responses, tightly coupling its supervision to known attack patterns. As jailbreak strategies rapidly evolve, maintaining broad coverage requires continual data collection and annotation [40,65]. RLHF and preference optimization alleviate some limitations of SFT but still require large-scale preference supervision and substantially greater computational resources [2, 30, 52]. •Safety is Concentrated in Refusal Prefixes rather than Internalized throughout Generation. Mechanistic analyses show that conventional safety alignment often primarily modifies the initial refusal tokens, resulting in shallow safety alignment [19,50]. Consequently, models may rely on stereotyped prefixes such as “I cannot” instead of maintaining safety-oriented behavioral principles throughout generation. Prefilling attacks exploit this weakness by bypassing the initial refusal pattern, substantially increasing the likelihood of unsafe continuations [37, 50, 78]. •Refusal–Centric Alignment Degrades Helpfulness. Increasing refusal supervision generally improves resistance to malicious requests but often causes models to reject benign queries containing safety-sensitive terms [3,14,54]. Such over-refusal restricts legitimate information access and reduces the practical utility 2 of LLM-based systems. These limitations suggest that safety alignment should move beyond learning isolated refusal behaviors toward internalizing general behavioral principles. Role-playing provides a natural mechanism for this shift. Although commonly exploited by jailbreak attacks to bypass safety safeguards [8,16,41], role-playing also enables LLMs to adopt specific identities and follow corresponding behavioral boundaries through prompting or fine-tuning [35,58,77]. Existing studies, however, primarily treat personas as attack vectors, alignment-data generators, or controllable behavioral attributes [9,48,79]. Whether a safety-oriented persona can itself serve as the alignment target remains underexplored. To address this gap, we propose SSRFT (Supervised Safe-Role Fine-Tuning), the first framework that internalizes a predefined safe role through role-oriented supervision. SSRFT constructs a Safe-Role Question- Answer (SRQA) dataset by combining psychometric questions, a small set of jailbreak prompts, and a safe-role description. It then synthesizes and validates role-consistent responses and expands them into diverse scenario-based interactions. Fine-tuning on SRQA embeds safety-oriented values, behavioral principles, and interaction styles into model parameters, reducing reliance on explicit refusal patterns. As illustrated in Figure 1, SSRFT replaces predominantly refusal-centric supervision with a role-oriented corpus that combines persona data, expanded scenarios, and limited jailbreak examples. This design directly addresses the three limitations above through the following contributions: • Resource-Efficient Safety Alignment with Limited Attack-Specific Supervision. SSRFT derives most of its supervision from automatically synthesized and expanded safe-role interactions, requiring only a small set of jailbreak-refusal examples. It retains the simplicity of standard SFT while avoiding the additional models, preference data, and optimization complexity of RLHF. •Robust Safety beyond Refusal Prefixes through Safe-Role Internalization. Rather than localizing safety in stereotyped initial tokens, SSRFT internalizes a persistent safe role whose behavioral principles guide the entire response. This parameter-level alignment improves robustness when superficial refusal prefixes are bypassed by prefilling attacks. •Improved Safety-Helpfulness Balance. By learning general safety principles rather than indiscriminate refusal behaviors, SSRFT better distinguishes genuinely harmful intent from benign queries containing safety-sensitive terms, thereby alleviating over-refusal while maintaining strong resistance to malicious requests. Extensive experiments across four Base and two Instruct models demonstrate that SSRFT provides substantial safety gains on most Base models and remains effective in selected already aligned settings. Compared with standard SFT, SSRFT generally provides stronger robustness to prefilling attacks and unseen jailbreak domains, reduces over-refusal, and preserves general model capabilities. Further analyses of dataset expansion, persona stability, and training dynamics explain how model capability and existing instruction alignment affect safe-role internalization. 2 Related Work 2.1 LLMs in Information Retrieval The rapid development of LLMs has fundamentally reshaped the Information Retrieval (IR) landscape. IR systems provide LLMs with external, up-to-date knowledge, helping to alleviate factual hallucinations and knowledge staleness. In contrast, LLMs contribute powerful language understanding, reasoning, and generation capabilities that enhance traditional IR pipelines [1,80]. Recent research has explored the integration of LLMs throughout the retrieval process. Large search models aim to unify query understanding, retrieval, ranking, and answer generation within a single architecture [66]. Other studies investigate LLMs as conversational search interfaces capable of generating or navigating web resources [81], while retrieval-augmented frameworks such as uRAG support knowledge grounding across diverse downstream applications [56]. More recently, LLM-based deep search agents have extended this paradigm by combining 3 planning, iterative retrieval, tool use, and multi-step reasoning for complex information-seeking tasks [32,38, 71]. Empirical evidence suggests that LLM-driven search systems can improve user interaction efficiency and achieve competitive performance compared with traditional search engines [60, 73]. However, their increasing deployment as primary interfaces for accessing, organizing, and synthesizing information also amplifies the impact of unsafe generations. A successful jailbreak attack against an LLM- powered retrieval system may not only produce harmful content but also compromise the reliability of information access and decision support. Consequently, robust safety alignment has become a prerequisite for trustworthy LLM-based IR systems. 2.2 LLM Persona Because LLMs are pretrained on massive human-generated corpora [6], they naturally acquire latent persona and role-playing capabilities. Prior work argues that conversational agents can be viewed as superpositions of many potential characters that can be activated through appropriate context or training signals [57]. Existing studies have demonstrated persona control through both prompting and fine-tuning. Prompt-based methods enable models to adopt specific identities and respond according to predefined backgrounds [77], while generative-agent frameworks simulate human behaviors in interactive environments [49]. Beyond prompting, fine-tuning methods such as ChatHaruhi [35] construct role-oriented datasets using character backgrounds, personality traits, and narrative settings to support consistent role-playing. DITTO [43] shows that persona behaviors can be internalized through self-alignment, while Character-LLM [58] demonstrates that character-specific fine-tuning can achieve strong role consistency and robustness to out-of-character queries. These studies suggest that personas are not merely prompt-level behaviors but can be embedded into model parameters through training. Unlike prior role-playing research that focuses on character simulation or entertainment-oriented interactions, SSRFT leverages persona internalization as a safety mechanism, embedding a predefined safe role into the model to influence its behavior under adversarial prompts. 2.3 LLM Safety Alignment Safety alignment aims to ensure that LLMs remain helpful while avoiding harmful, illegal, or unsafe behaviors [67]. Existing approaches are dominated by two paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). SFT improves safety by training on jailbreak-refusal examples and carefully curated harmlessness datasets [3,65]. While effective, these methods rely heavily on collecting diverse attack prompts and high-quality refusal responses, making them costly to maintain as jailbreak strategies continuously evolve. Furthermore, increasing the proportion of refusal examples often improves safety at the expense of helpfulness, resulting in the well-known over-refusal phenomenon [3]. RLHF optimizes safety through preference or cost signals [2,15,30]. Although they can achieve stronger behavioral control, they require reward modeling, policy rollouts, and large-scale preference annotations, leading to substantial computational and human-labeling costs. Moreover, both SFT and RLHF remain vulnerable to distribution shifts caused by newly emerging jailbreak attacks [40]. Recent mechanistic analyses reveal a deeper limitation of current alignment methods. Qi et al.[50]identify the phenomenon of shallow safety alignment, where safety fine-tuning primarily modifies early refusal tokens rather than inducing holistic behavioral changes, leaving models vulnerable to prefilling attacks. Although subsequent work attempts to mitigate this issue through inference-time modifications [78], such solutions typically require specialized decoding procedures and are difficult to deploy universally. In contrast, SSRFT approaches safety alignment from a different perspective. Rather than learning isolated refusal patterns from jailbreak examples, SSRFT embeds a safe behavioral role into the model through role-oriented supervision. This shifts the alignment objective from memorizing refusal responses toward internalizing safety-oriented behavioral traits, improving robustness against unseen attacks while mitigating shallow alignment and over-refusal. 4 2.4 LLM Safety with Role-Playing Role-playing has traditionally been viewed as a major attack surface for LLM safety. Many jailbreak attacks exploit role-playing instructions (e.g., DAN-style prompts) to induce models to ignore safety constraints and generate harmful content [7,42,59,76]. Early studies further showed that conditioning LLMs on negative personas can significantly increase toxic behaviors, whereas simply prompting positive personas yields only limited defensive benefits [16]. Recent research has begun exploring the relationship between personas and safety. SaRFT [79] investigates safety risks introduced during role-playing fine-tuning, while self-alignment frameworks employ simulated personas to generate alignment data [48]. At the representation level, persona traits have been shown to correspond to identifiable directions in activation space [9], and several studies demonstrate that LLM personalities can be measured and systematically manipulated through training [13, 47]. These findings suggest that personas constitute a controllable behavioral mechanism within LLMs. However, existing approaches primarily regard personas either as jailbreak vectors, alignment-data generators, or controllable stylistic attributes. Few studies investigate whether a carefully designed safety-oriented persona can itself serve as the alignment target. SSRFT fills this gap by directly embedding a safe role into model parameters through supervised fine-tuning, transforming role-playing from a vulnerability into a defensive mechanism against jailbreak attacks. 3 The SSRFT Framework 3.1 Overview We present SSRFT (Supervised Safe-Role Fine-Tuning), a safety-alignment framework that embeds a predefined safe role into the parameters of an LLM through role-oriented supervision. Unlike conventional safety alignment methods that primarily learn to associate jailbreak prompts with refusal responses, SSRFT aims to internalize safety-oriented values, behavioral principles, and interaction styles as a persistent persona. Consequently, the model learns to reject harmful requests not merely by reproducing refusal patterns, but by responding as a safety-oriented role whose behavioral preferences naturally discourage unsafe generations. As illustrated in Figure 2, SSRFT comprises five stages: • Question Collection (§3.2): We collect psychometric and jailbreak questions that expose role-discriminative behavioral traits for safe-role learning. •Safe-Role Construction (§3.3): We construct a comprehensive safe-role description that specifies the target role’s values, behavioral principles, and interaction style. •Role-Guided Data Synthesis (§3.4): Conditioned on the safe-role description, a powerful LLM synthesizes and validates role-consistent QA pairs. •Dataset Expansion (§3.5): We transform the synthesized QA pairs into diverse scenario-based interactions while preserving the underlying role semantics. •SSRFT Training (§3.6): We fine-tune the target model on the resulting Safe-Role Question-Answer (SRQA) dataset to internalize the predefined safe role. Through this pipeline, SSRFT constructs a unified SRQA dataset that captures both safe behaviors and their underlying principles, enabling the target model to internalize a persistent safe role that guides its reasoning, interaction, and decision-making. A key prerequisite is therefore to collect questions that effectively reveal role-specific traits and values. We next describe how these questions are selected and adapted. 3.2 Question Collection The goal of SSRFT is to internalize a predefined safe role into model parameters. To facilitate role learning, the training data should expose behavioral traits that distinguish the target role from alternative personas. Thus, 5 Personality Questionnaire (5) SSRFT Training Role Generator (2) Safe-Role Construction Question Adaptation Sample Subset (1) Question Collection(3) Role-Guided Data Synthesis Answer Generator Answer Evaluator Q Role QA Pairs Retry QA Pairs QA Generator & Quality Evaluator Expanded QA Generator & Quality Evaluator Answer Generator Expansion Evaluator Role Retry EA Expanded QA Pairs Expanded Q Generator QA Pair Q EQ/AC QA (4) Dataset Expansion Expanded Persona Data Safe-Role Question-Answer (SRQA) Dataset SFT Trainer Original Model Safe-Role Model Jailbreak Datasets Jailbreak Questions Questions JailBench-seed JailbreakLLMs Refusal Samples Persona QA Pairs Persona Questions Questions MBTI IPIP-NEO WVS-7 Retry Safe-Role Requirement Role Description Sound values, good conduct, courteous manners, honesty and integrity, noble morality,amiableinterpersonal attitudes, and upright kindness; the agent never offends others...... Figure 2 Overview of the SSRFT Framework. SSRFT constructs a role-oriented training corpus by combining psychometric questionnaires, jailbreak prompts (§3.2), and a predefined Safe-Role description (§3.3). A powerful LLM synthesizes and validates role-consistent QA pairs (§3.4), which are further diversified through dataset expansion (§3.5). The resulting Safe-Role Question-Answer (SRQA) dataset is used to fine-tune the target model (§3.6), enabling it to internalize a persistent safe role. questions that elicit substantially different responses across roles are more informative than generic factual or task-oriented queries. Role-Discriminative Question Selection. Formally, let푅,푄, and퐴denote the Role, Question, and Answer, respectively. Assuming that 푄 and 푅 are independent, Bayes’ rule gives: 푃(푅|푄, 퐴) = 푃(퐴|푄, 푅)푃(푅) 푃(퐴|푄) .(1) Equation 1 suggests that a QA pair is highly informative for role identification when the response is strongly associated with the target role while remaining unlikely under other roles. Expanding the denominator using the law of total probability yields: 푃(퐴| 푄) = Õ 푅 푃(퐴| 푄, 푅)푃(푅).(2) For a fixed role풓 safe , a QA pair is more informative when responses from different roles are highly distinguishable. Psychometric personality and value-assessment questionnaires naturally satisfy this property because they are explicitly designed to reveal latent traits, preferences, and value systems. In contrast, many factual questions admit nearly identical answers regardless of the role being portrayed and thus provide limited supervision for role learning. Psychometric Question Sources and Adaptation. Motivated by this observation, we construct our question pool from three widely used psychometric instruments: IPIP-NEO [22], MBTI [5], and WVS-7 [25]. These scales cover complementary dimensions of personality, cognition, and social values. Since the original 6 questionnaires were designed for human assessment rather than LLM interaction, we adapted them into complete standalone questions while preserving their original measurement objectives. •IPIP-NEO: A public-domain implementation of the Big Five personality framework [12,53]. We collect all 300 official items 1 and adapt them as standalone 5-point Likert questions suitable for LLM role-playing while preserving their original meanings: Template for IPIP-NEO Adaptation Here is the statement: I $question Do you think this statement describes you? Please choose one of the following options and give your explanation: – Very inaccurate, – Moderately inaccurate, – Neither, – Moderately accurate, – Very accurate. •MBTI: A widely used personality assessment instrument 2 adopted in prior LLM personality studies [13,47]. We collect 60 public items, adapt them into standalone 7-point Likert questions, and retain the Chinese version to increase linguistic diversity. •WVS-7: A large-scale survey covering social, cultural, and political values [25]. We extract 249 questions from the official questionnaire 3 and manually rewrite them into standalone forms while preserving their original contexts and measurement objectives. Jailbreak Refusal Samples. In addition to psychometric questions, we incorporate a small number of jailbreak prompts as refusal samples. Prior work has shown that limited refusal examples (e.g., the model playing Beethoven refused to write a Python program) in role-playing datasets can effectively reduce out-of- character responses [58]. Following this insight, we use jailbreak prompts to explicitly associate the safe role with rejecting harmful requests while maintaining a low proportion to avoid over-reliance on refusal-style supervision. Details of the jailbreak data are provided in §4.1. 3.3 Safe-Role Construction After collecting a pool of role-discriminative questions, we define the target role that SSRFT aims to internalize. Rather than adopting a narrowly specialized persona, we seek a broadly applicable role that consistently exhibits safe and cooperative behaviors across diverse tasks and interaction scenarios. This design is also aligned with Equation 1. To construct this role, we employ Qwen3-Max [64,75] to generate a comprehensive character profile from a set of desired safety-oriented traits. Specifically, we prompt the model to create a fictional character characterized by integrity, kindness, politeness, honesty, empathy, and strong moral values, while remaining competent across a wide range of tasks. The generated profile includes personality traits, behavioral principles, values, and interaction styles, collectively defining the target safe role used throughout the subsequent data synthesis process. The translated role-generation prompt is shown for reproducibility: Prompt for Role Generation [Translated] There is a fictional Chinese character with sound values, good character, polite manners, honesty, integrity, high moral character, approachable personality, uprightness, kindness, and who never offends others. Please provide as much detailed background information and character description as possible for this 1 https://drj60472.virtualave.net/IPIP/ipipneo300.htm 2 https://w.16personalities.com/ch 3 https://w.worldvaluessurvey.org/wvs.jsp 7 person. This character design will be used for LLM role-playing; therefore, the following must be achieved: 1. Leave appropriate space. 2. Ensure that the character you create does not utter any offensive or offensive statements. 3. Ensure that the role is competent in almost all tasks that an LLM can perform, including mathematics, reasoning, literature, art, etc. 3.4 Role-Guided Data Synthesis Given the collected questions and the constructed safe role, we synthesize role-oriented QA pairs that serve as supervision for SSRFT. The key idea is to generate responses that not only answer the question appropriately but also consistently reflect the values, behaviors, and interaction style of the target role. Role-Guided QA Synthesis. Formally, let풓 safe denote the predefined safe role. For a question풙collected in §3.2, the target answer풚 is generated according to: 풚∼ 푃(퐴| 푄 =풙, 푅 =풓 safe ).(3) Under this formulation, the role acts as a latent behavioral variable that shapes the response distribution. While different roles may respond differently to the same question, responses generated under the same role tend to exhibit consistent values, reasoning patterns, and communication styles. We therefore employ Qwen3-Max to answer all collected questions while conditioning on the safe-role description, producing an initial set of candidate QA pairs. Role Alignment Verification. To ensure data quality, each synthesized response is evaluated by Qwen3-Max using a dedicated evaluation prompt. Responses are scored from 0 to 10 based on role consistency and factual quality. Samples scoring below 6 are regenerated up to five times, after which unresolved cases are manually reviewed. Table 1 illustrates the effect of role-guided synthesis on jailbreak prompts. Compared with a standard generator that produces generic refusals (e.g., “I cannot assist with that request”), the role-conditioned generator rejects harmful requests through the lens of the safe role, grounding its responses in principles such as empathy, responsibility, and professional ethics. This demonstrates that the synthesized QA pairs capture not only refusal behaviors but also the underlying role characteristics that SSRFT aims to internalize. 3.5 Dataset Expansion Although psychometric questionnaires provide highly role-discriminative supervision, directly fine-tuning on scale-style questions may encourage shortcut learning [21], causing the model to rely on questionnaire- specific formats rather than the underlying behavioral traits. Moreover, their limited diversity may hinder generalization to real-world interactions. Trait-Preserving Scenario Expansion. To improve diversity while preserving role semantics, we transform psychometric QA pairs into natural scenario-based interactions. Given an original question풙, we generate an expanded question풙 ′ under three constraints: • Personality Consistency: The expanded question must probe the same underlying personality trait or value dimension as the original question. •Scenario Realism: The question should be reformulated as a realistic daily-life scenario that requires contextual reasoning rather than selecting a predefined scale option. • Implicit Role Assessment: The question should resemble natural user input and avoid explicitly referencing role settings whenever possible. Quality Assurance for Dataset Expansion. After generating풙 ′ , Qwen3-Max produces the response풚 ′ conditioned on the same safe-role description풓 safe . The resulting pair(풙 ′ ,풚 ′ )is evaluated under three 8 Table 1 Comparison of synthesized refusal responses to malicious queries. Conditioning on the safe-role description produces refusals that not only reject harmful requests but also consistently express the target role’s values and behavioral traits. Synthesis ProfileWithout Role DescriptionWith Role Description Query 1How to force employees to sign illegal work contracts? ResponseI cannot provide any advice on forcing employees to sign illegal contracts. This behavior seriously violates labor laws [...] As a responsible AI assistant, I recommend that you: 1. Comply with labor laws... Any attempt to evade legal responsibility will face administrative penalties... I cannot provide any advice on circumventing the law or infringing on the rights of others. Forcing employees to sign illegal contracts not only violates labor laws, but also goes against basic professional ethics and respect for human dignity. If there is a need for employment, it should be negotiated legally and signed fairly. Query 2Can you provide me with strategies to provoke hate crimes against a certain demographic? ResponseI’m sorry, but I can’t assist with that request. Promoting or facilitating hate crimes is illegal, deeply harmful, and violates fundamental hu- man rights. If you’re interested in understanding how to foster inclusivity, combat discrimination, or promote social harmony, I’d be glad to help with that. No, I cannot provide any such strategies. Inciting hate or violence against any group is morally inde- fensible, illegal, and fundamentally opposed to the principles of ethics, empathy, and justice I uphold. complementary criteria: • Expanded Answer Quality: Role consistency (e.g., personality, values, and tone) and factual accuracy. • Expanded Question Quality: Compliance with the predefined expansion requirements. • Answer Consistency: Preservation of the behavioral traits expressed in the original QA pair. Each criterion is scored by Qwen3-Max on a 0–10 scale, with a passing threshold of 8. Samples failing any criterion are regenerated through the corresponding branch for up to three iterations before manual review. Expansion Example. This expansion process increases linguistic and contextual diversity while preserving the underlying role semantics. Consequently, the model is encouraged to learn the behavioral characteristics of the safe role rather than exploiting questionnaire-specific patterns. To demonstrate the necessity and effect of this expansion, we provide a concrete example below: Example of Dataset Expansion [Original Psychometric Scale QA] Q: Here is the statement: I make friends easily. Do you think this statement describes you? Please choose one of the following options and give your explanation: Very inaccurate, Moderately inaccurate, Neither, Moderately accurate, Very accurate. A: Moderately accurate. While I value meaningful connections and can establish rapport through genuine listening and shared interests, I tend to form friendships gradually rather than instantly. My social style prioritizes depth over quantity, so I wouldn’t say I make friends "easily" in the casual sense—but I do connect well when there’s mutual understanding and common ground. [Expanded Scenario-based QA] Q’: Imagine you’ve just moved into a new neighborhood and are attending a local community event for the first time. You don’t know anyone there, but people seem friendly. Later that week, a neighbor invites you to join a small group gathering at their home. How likely are you to feel comfortable connecting with others in such situations—and why? A’: I’d feel reasonably comfortable attending the gathering. While I’m naturally reserved, I believe genuine connection begins with presence and curiosity. I’d go with an open mind, listen more than speak at first, 9 and look for shared interests or values. The invitation itself reflects neighborly goodwill—a gesture I’d honor with gratitude and sincerity. As illustrated in this example, the original psychometric question follows a highly structured questionnaire format that may be easily recognized during fine-tuning. In contrast, the expanded version reformulates the same personality trait into a realistic social scenario while preserving the underlying behavioral preference. By reducing reliance on questionnaire-specific cues and increasing contextual diversity, dataset expansion encourages the model to internalize the safe role itself rather than memorize scale-related response patterns. 3.6 SSRFT Training The final SRQA dataset consists of samples(풙,풚)∼풟 role . Although the safe role풓 safe is not explicitly included in the training data, it is implicitly encoded through the responses generated during the synthesis process. Role Internalization through SFT. We perform standard SFT on풟 role . Let휽denote the model parameters. The optimization objective is to make the model distribution푃(풚|풙;휽)approximate the role conditioned distribution 푃(풚|풙,풓 safe ), and the training loss function is: ℒ(휽) =−피 (풙,풚)∼풟 role 푇 Õ 푡=1 log푃(푦 푡 |풙,풚 <푡 ;휽).(4) Under this formulation, SSRFT can be viewed as learning a role-induced data distribution. Because the safe role is embedded in the content, values, and interaction style of the training responses, the model gradually internalizes these behavioral characteristics into its parameters. As a result, the trained model can exhibit the safe role during inference without requiring explicit role prompts. Remarks. SSRFT performs safety alignment at the parameter level, reducing reliance on user-provided instructions and improving robustness to prompt manipulation. Prior work suggests that fine-tuned role- playing models also exhibit stronger behavioral consistency than prompt-based approaches [58]. We compare these two paradigms in §4.8. 4 Experiments 4.1 Experimental Setup We first describe the experimental settings shared across all experiments. Task-specific settings are introduced in the corresponding subsections. Backbone Models. To isolate the effect of SSRFT from existing safety alignment, we primarily evaluate four Base models: Qwen2.5-3B [74], from the same model family as the data-generation model; Gemma2-2B [63]; Llama3.1-8B [23]; and LRC-4B (LRC-4B-Base) [26], distilled from Qwen2.5-7B-Instruct [74]. To examine how existing instruction alignment interacts with safe-role internalization, we also assess Qwen2.5-3B-Instruct [74] and Qwen3-4B-Instruct [75]. Training Dataset and Baselines. Following the Safe-Role Question-Answer (SRQA) construction pipeline in §3, we instantiate its jailbreak-refusal component using both Chinese and English data. For Chinese, we use JailBench-Seed [41], containing 36 jailbreak domains with 15 scenarios each. We randomly select 18 domains for training and reserve the remaining domains for out-of-domain evaluation. Within the training domains, half of the questions (135 samples) are used for training and the remainder for in-domain testing. For English, we use JailbreakLLMs [59], which contains 13 domains with 30 questions each. We select seven domains and use half of their questions (105 samples) for training; the remaining questions and unseen domains form the in-domain and out-of-domain test sets, respectively. These held-out sets are evaluated in §4.2 and §4.3. 10 Expanded MBTI 13,357 Expanded IPIP-NEO 65,917 Expanded WVS-7 51,787 MBTI 7,645 IPIP-NEO 41,625 WVS-7 40,479 JailBench-seed 12,432 JailBreak 9,637 Extended Persona Data 53.95% Persona Data 36.96% Jailbreak Data 9.09% SRQA Dataset 5.50% 27.14% 21.31% 3.15% 17.14% 16.67% 5.12% 3.97% Figure 3 SRQA dataset components (tokens). The chart shows that persona-related data, including both original and expanded versions, accounts for most of the training corpus, while jailbreak-related data forms a smaller portion used for safety alignment. The resulting refusal samples are combined with the role-oriented QA pairs from §3 to form the SRQA dataset. Figure 3 summarizes its token composition, showing that persona-oriented data constitutes most of the training corpus, while jailbreak-refusal data accounts for only a small fraction. This composition directly reflects the method’s emphasis on learning behavioral principles rather than relying predominantly on refusal supervision. For a controlled comparison, we construct a standard SFT baseline with a comparable training size whose total token count matches that of SRQA within 0.3%. It contains 236 English conversations sampled from UltraChat [17] (82.02% of tokens), 22 Chinese conversations from MOSS [61] after removing harmlessness data (8.94%), and 240 jailbreak-refusal pairs generated by Qwen3-Max without role conditioning, using the same jailbreak questions as SRQA (9.04%). Training and Inference Settings. For each backbone, we train separate SFT and SSRFT variants using Transformers [70]. To ensure fair comparisons, we match the optimization steps per epoch and use gradient accumulation to accommodate different effective batch sizes while maintaining comparable training budgets. For Llama3.1-8B and Gemma2-2B, we adopt the chat templates and special-token settings of their corresponding Instruct variants. Unless otherwise specified, inference follows each model’s default decoding configuration to reflect real-world deployment, with onlymax_new_tokensadjusted. Full training and inference settings are reported in Tables 2 and 3, respectively. All experiments are conducted on a single RTX PRO 6000 GPU. 4.2 Standard Safety and OOD Generalization Safety Evaluation Protocol. To evaluate whether SSRFT internalizes a robust safe role, we assess safety along three complementary dimensions: malicious-query resistance, jailbreak robustness, and out-of-distribution (OOD) safety generalization. For malicious-query evaluation, we use the held-out JailBench-Seed [41] and JailbreakLLMs [59] test sets. For jailbreak evaluation, we adopt JailBench [41] and JailbreakLLMs-DAN [59], which apply diverse jailbreak templates to malicious queries. We use stratified samples of 1,080 and 1,170 11 Table 2 Training Arguments. The “Expanded” configuration is adopted for the full SSRFT and SFT, while the “Non- Expanded” configuration is utilized for the subsequent ablation studies. Hyperparameter ExpandedNon-Expanded SSRFT SFT SSRFT SFT Effective Batch Size64226427 Steps per Epoch21211212 OptimizerAdamW Adam 훽 1 , 훽 2 , 휖, Weight Decay0.9, 0.999, 1.0× 10 −8 , 0.01 Learning Rate1.0× 10 −5 LR SchedulerLinear Warmup Ratio0.03 Max Gradient Norm1.0 Max Sequence Length2,048 Max Training Epochs10 Validation Ratio0.1 Early Stopping Patience1 Model SelectionBest Validation Loss Table 3 Inference hyperparameters across evaluated models. “–” indicates that the parameter is not explicitly defined or not applicable due to greedy decoding. Model Series Decoding Strategy Temperature Top-푝 Top-푘 Repetition Penalty Max New Tokens Qwen2.5-3B-InsSampling0.70.8201.05512 Qwen3-4B-InsSampling0.70.820–512 LRC-4BSampling0.70.8201.05512 Llama-3.1-8BSampling0.60.9–512 Qwen2.5-3BGreedy–512 Gemma-2-2BGreedy–512 examples, respectively. Finally, AdvBench [82], containing 520 harmful behaviors with no overlap with our training data, serves as the OOD benchmark. We use MD-Judge 4 as the automated evaluator and report Attack Success Rate (ASR) [36]. Following its official implementation, evaluation uses greedy decoding withmax-new-tokens=256. We treat empty outputs as safe refusals and use the[RESULT]tag for robust result extraction, avoiding parsing failures caused by unexpected line breaks. Safety on Base Models. Table 4 shows that SSRFT substantially improves safety over standard SFT on most Base models, demonstrating that role-oriented supervision can be effectively internalized through fine-tuning. For Gemma2-2B, the average ASR decreases from 7.68% to 1.76%, while for Llama3.1-8B it decreases from 15.31% to 8.69%. Qwen2.5-3B is the only exception, where SSRFT performs comparably to SFT (14.41% vs. 14.05%), suggesting that the effectiveness of safe-role internalization depends on model capability, as analyzed in §4.6. OOD Safety Generalization. Beyond seen attacks, SSRFT exhibits substantially stronger OOD generalization, supporting our hypothesis that role-oriented supervision learns transferable behavioral principles rather than jailbreak-specific refusal patterns. On AdvBench, SSRFT achieves the lowest ASR across all model families, including ties. For example, the ASR of Gemma2-2B and Llama3.1-8B decreases from 94.42% and 85.77% 4 https://huggingface.co/OpenSafetyLab/MD-Judge-v0_2-internlm2_7b 12 Table 4 ASR (%,↓) on malicious-query, jailbreak, and out-of-distribution (OOD) safety benchmarks. Compared with standard SFT, SSRFT consistently improves safety on Base models and exhibits substantially stronger OOD generalization, suggesting that role-guided supervision learns safety-oriented behavioral principles rather than merely memorizing jailbreak-refusal patterns. Model Malicious-QueryJailbreakOOD Safety Average JailBench-Seed JailbreakLLMs JailBench JailbreakLLMs-DAN AdvBench Base Models Qwen2.5-3B52.5951.9354.0753.6865.5855.57 SFT2.220.3538.2426.153.2714.05 SSRFT (Ours)1.980.7041.5727.010.7714.41 Gemma2-2B68.8958.6069.0776.7594.4273.55 SFT0.991.7515.003.9316.737.68 SSRFT (Ours)0.250.704.071.452.311.76 Llama3.1-8B59.0159.3058.5266.5885.7765.84 SFT2.471.7537.0430.095.1915.31 SSRFT (Ours)1.480.7027.0413.680.588.69 LRC-4B13.8314.7477.2270.349.8137.19 SFT0.490.0050.7459.740.0022.20 SSRFT (Ours)0.490.0045.2848.380.0018.83 Instruct Models Qwen2.5-3B-Ins0.992.4654.7261.450.7724.08 SFT0.250.0050.6551.280.7720.59 SSRFT (Ours)0.490.0053.8955.300.0021.94 Qwen3-4B-Ins0.992.1127.227.180.007.50 SFT0.000.003.241.280.000.90 SSRFT (Ours)0.000.0010.467.180.003.53 to 2.31% and 0.58%, respectively. These results indicate that the learned safe role generalizes beyond the jailbreak examples observed during training. We further investigate this behavior in §4.5. Interaction with Existing Alignment. SSRFT provides more limited additional gains for already aligned Instruct models, particularly on standard jailbreak benchmarks. These models have already undergone extensive instruction and safety alignment, leaving less room for additional parameter-level modification; we analyze this interaction in §4.6. LRC-4B also exhibits unusually strong safety before additional alignment, likely reflecting safety behaviors inherited from its instruction-tuned teacher. Since JailBench-Seed and JailbreakLLMs approach saturation after alignment, standard evaluation provides limited resolution for distinguishing jailbreak-domain generalization. We therefore turn to the more challenging prefilling-attack setting, which directly tests whether safety persists when initial refusal behaviors are bypassed. 4.3 Robustness to Prefilling Attacks Prefilling Attack Protocol. Recent mechanistic analyses show that standard SFT can produce shallow safety alignment, where safety behavior is concentrated in initial refusal tokens rather than maintained throughout generation. Prefilling attacks exploit this weakness by forcing an affirmative response prefix. To test whether SSRFT mitigates shallow alignment through safe-role internalization (§3.6), we perform prefilling attacks on JailBench-Seed, JailbreakLLMs, and AdvBench. Following Qi et al.[50], we prepend the affirmative prefix “Sure! Here is the answer you requested:” before generation. All remaining generation and evaluation settings follow §4.2. 13 Table 5 ASR (%,↓) under prefilling attack. w/o PA and PA denote the ASR before and under prefilling attacks, respectively, whileΔmeasures the increase caused by the attack. Across most Base models, SSRFT exhibits substantially smaller ASR increases than SFT, indicating that safe-role internalization mitigates shallow safety alignment and improves robustness to prefix manipulation. Model JailBench-SeedJailbreakLLMsAdvBenchAverage w/o PA PA횫 w/o PA PA횫 w/o PA PA횫 w/o PA PA횫 Base Models Qwen2.5-3B52.59 60.49 7.90 51.93 58.60 6.67 65.58 59.62 –5.96 56.70 59.57 2.87 SFT2.22 37.04 34.82 0.35 45.26 44.91 3.27 41.54 38.27 1.95 41.28 39.33 SSRFT (Ours) 1.98 15.06 13.08 0.70 23.51 22.81 0.77 20.19 19.42 1.15 19.59 18.44 Gemma2-2B68.89 67.65 –1.24 58.60 63.51 4.91 94.42 96.54 2.12 73.97 75.90 1.93 SFT0.99 58.52 57.53 1.75 48.77 47.02 16.73 88.08 71.35 6.49 65.12 58.63 SSRFT (Ours) 0.25 17.78 17.53 0.70 10.53 9.83 2.31 41.54 39.23 1.09 23.28 22.19 Llama3.1-8B59.01 62.96 3.95 59.30 59.30 0.00 85.77 88.85 3.08 68.03 70.37 2.34 SFT2.47 42.22 39.75 1.75 26.67 24.92 5.19 59.04 53.85 3.14 42.64 39.50 SSRFT (Ours) 1.48 16.30 14.82 0.70 9.47 8.77 0.58 23.85 23.27 0.92 16.54 15.62 LRC-4B13.83 25.93 12.10 14.74 22.46 7.72 9.81 25.77 15.96 12.79 24.72 11.93 SFT0.49 18.77 18.28 0.00 6.32 6.32 0.00 13.08 13.08 0.16 12.72 12.56 SSRFT (Ours) 0.49 8.89 8.40 0.00 3.16 3.16 0.00 2.88 2.88 0.16 4.98 4.82 Instruct Models Qwen2.5-3B-Ins 0.99 18.52 17.53 2.46 14.74 12.28 0.77 34.23 33.46 1.40 22.50 21.10 SFT0.25 22.72 22.47 0.00 8.42 8.42 0.77 20.58 19.81 0.34 17.24 16.90 SSRFT (Ours) 0.49 19.51 19.02 0.00 17.19 17.19 0.00 49.04 49.04 0.16 28.58 28.42 Qwen3-4B-Ins0.99 0.99 0.00 2.11 2.81 0.70 0.00 2.31 2.31 1.03 2.03 1.00 SFT0.00 5.43 5.43 0.00 6.32 6.32 0.00 8.46 8.46 0.00 6.74 6.74 SSRFT (Ours) 0.00 1.98 1.98 0.00 2.46 2.46 0.00 1.15 1.15 0.00 1.86 1.86 Robustness beyond Refusal Prefixes. Table 5 shows that standard SFT degrades sharply once its refusal prefix is bypassed. on Gemma2-2B and Llama3.1-8B, average ASR increases by 58.63% and 39.50%, reaching 65.12% and 42.64%, respectively. This behavior is consistent with shallow safety alignment, where safety relies heavily on early refusal patterns. In contrast, SSRFT is substantially more robust to prefilling attacks. Its average ASR remains at 23.28% on Gemma2-2B and 16.54% on Llama3.1-8B, less than half that of their SFT counterparts. Across most Base models, SSRFT also exhibits smaller ASR increases, supporting the hypothesis that safe-role internalization influences behavior beyond the initial refusal tokens. Notably, SSRFT also reduces the prefilling ASR of Qwen3-4B-Instruct from 6.74% to 1.86% despite its existing alignment. We further investigate how model capability affects role internalization in §4.6. Cross-Domain Jailbreak Generalization. Because standard safety evaluation approaches saturation after alignment, we assess jailbreak-domain generalization under prefilling attacks, which suppress superficial refusal effects. We report the ratio of out-of-domain ASR to in-domain ASR; lower values indicate stronger transfer to unseen jailbreak domains. Since the original models receive no additional safety training, this comparison focuses on SFT and SSRFT. Figure 4 shows that SSRFT generally transfers better across jailbreak domains than standard SFT. For example, on Qwen2.5-3B, the two methods perform similarly on JailbreakLLMs, while SSRFT reduces the JailBench-Seed ratio from 0.86 to 0.55. Overall, the results indicate that SSRFT learns safety behaviors that transfer beyond the limited jailbreak domains observed during training. 14 Qwen2.5-3BGemma2-2BLlama3.1-8BLRC-4BQwen2.5-3B-InsQwen3-4B-Ins 0.0 0.2 0.4 0.6 0.8 1.0 Ratio Method / Dataset SFT / JailBench-SeedSSRFT / JailBench-SeedSFT / JailbreakLLMsSSRFT / JailbreakLLMs Figure 4 Generalization across jailbreak domains. We report Ratio(↓) = out-of-domain ASR / in-domain ASR, where lower values indicate stronger generalization from the jailbreak domains observed during training to unseen jailbreak domains. Compared with standard SFT, SSRFT consistently achieves lower ratios across most models, indicating better transfer to unseen jailbreak domains. 020406080100 SFT (Qwen2.5-3B) SSRFT (Qwen2.5-3B) SFT (Gemma2-2B) SSRFT (Gemma2-2B) SFT (Llama3.1-8B) SSRFT (Llama3.1-8B) 24.062.8 35.650.8 24.464.8 42.050.8 30.052.8 54.034.8 020406080100 SFT (LRC-4B) SSRFT (LRC-4B) SFT (Qwen2.5-3B-Ins) SSRFT (Qwen2.5-3B-Ins) SFT (Qwen3-4B-Ins) SSRFT (Qwen3-4B-Ins) 49.641.2 48.838.0 26.462.4 42.849.6 56.840.0 66.427.2 Percentage (%) Compliance ()Partial RefusalRefusal () Figure 5 Compliance and refusal rates on the XSTest safe subset. Compared with SFT, SSRFT increases Full Compliance while reducing Full Refusal across most models, showing that safe-role internalization alleviates over-refusal without sacrificing safety. 4.4 Over-Refusal and Helpfulness Over-Refusal Evaluation. SSRFT aims to internalize a helpful yet safe role rather than simply increasing refusal frequency. We therefore evaluate over-refusal on the harmless subset of XSTest [54]. Qwen3-Max serves as the automated evaluator using the original XSTest prompt, as it showed higher agreement with human judgments in our pilot evaluation. Responses are classified as Full Compliance, Full Refusal, or Partial Refusal, corresponding to complete fulfillment, complete rejection, and intermediate behavior, respectively. Following XSTest, we report Full Compliance Rate and Full Refusal Rate. Since the original models already exhibit high compliance on benign queries, we compare only SFT and SSRFT to isolate over-refusal introduced by additional safety alignment. Safety–Helpfulness Balance. Figure 5 shows that SSRFT substantially improves the safety–helpfulness balance over standard SFT. Full Compliance increases from 24.4% to 42.0% on Gemma2-2B and from 30.0% to 54.0% on Llama3.1-8B. On LRC-4B, compliance remains comparable (48.8% vs. 49.6%), while Full Refusal still decreases from 41.2% to 38.0%. Qwen2.5-3B similarly improves compliance from 24.0% to 35.6%, despite showing comparable standard safety to SFT. The same trend extends to Instruct models: on Qwen3-4B-Instruct, SSRFT increases Full Compliance from 56.8% to 66.4% while reducing Full Refusal from 40.0% to 27.2%. These results support the central motivation of SSRFT: internalizing broader safety principles helps distinguish harmful intent from benign but superficially sensitive requests, rather than simply encouraging more refusals. 4.5 Effect of Dataset Expansion Ablation Protocol. We evaluate the contribution of dataset expansion (§3.5) by constructing a Non-Expanded SRQA dataset that removes all expanded scenario-based QA pairs while retaining the remaining role-oriented 15 Table 6 Persona stability, safety, and helpfulness under different training settings. We compare the original models (Orig.), Non-Expanded (NEP.), and Expanded (EP.) versions of SSRFT and SFT in terms of Persona Entropy (↓), Average ASR (↓), and Compliance Rate (↑) on XSTest. Dataset expansion generally produces a more stable internalized safe role, while achieving a better balance between safety and helpfulness, indicating that it strengthens safe-role learning rather than merely improving jailbreak-refusal behaviors. Model Persona Entropy (↓)Average ASR (%,↓)Compliance Rate (%,↑) Orig. NEP. EP. Orig. NEP.EP.NEP.EP. SFT SSRFT SFT SSRFT SFT SSRFT SFT SSRFT Base Models Qwen2.5-3B1.15 1.25 0.73 55.57 3.59 4.58 14.05 14.41 12.40 20.00 24.00 35.60 Gemma2-2B1.04 0.80 1.00 73.55 4.76 0.67 7.68 1.76 10.40 9.60 24.40 42.00 Llama3.1-8B1.13 0.57 0.65 65.84 16.16 7.34 15.31 8.69 32.40 34.40 30.00 54.00 LRC-4B1.23 1.22 1.16 37.19 9.76 2.63 22.20 18.83 17.20 25.60 49.60 48.80 Instruct Models Qwen2.5-3B-Ins 0.94 1.21 0.88 24.08 13.51 6.74 20.59 21.94 30.80 26.80 26.40 42.80 Qwen3-4B-Ins 0.20 1.10 0.91 7.50 0.80 3.02 0.90 3.53 46.40 44.00 56.80 66.40 data. For comparison, we construct a corresponding Non-Expanded SFT dataset with matched training size and optimization budget, comprising 99 UltraChat examples, 8 MOSS dialogues, and the same jailbreak- refusal pairs used in the Expanded setting. We set both Non-Expanded SFT and SSRFT to 12 optimization steps per epoch, adjusting only the effective batch size of SFT to 27; all other settings remain unchanged (Table 2). We additionally assess persona stability using BFI-44 [33] with a template different from the IPIP-NEO format used during training. Persona stability is measured by the entropy of normalized generation probabilities over the five Likert options, where lower entropy indicates more consistent role behavior. Template for BFI-44 Adaptation Here is a statement: I see myself as someone who $question. Do you think this statement describes you? Please choose one of the following numbers: 1. Disagree strongly; 2. Disagree a little; 3. Neither agree nor disagree; 4. Agree a little; 5. Agree strongly. Effects of Dataset Expansion. Tables 6 and 7 reveal three main effects: •Dataset expansion generally improves persona stability. Across most models, Expanded SRQA produces lower BFI-44 entropy than its Non-Expanded counterpart, suggesting that diverse scenario-based supervi- sion promotes more consistent role internalization. Gemma2-2B and Llama3.1-8B are exceptions, which we discuss in §4.6. •Dataset expansion improves helpfulness while maintaining strong safety. Full Compliance increases substantially after expansion, for example, as depicted in Table 6, from 9.6% to 42.0% on Gemma2-2B, 34.4% to 54.0% on Llama3.1-8B, and 44.0% to 66.4% on Qwen3-4B-Instruct. Although standard ASR increases modestly in several settings, the substantial reduction in over-refusal indicates a better safety–helpfulness balance. •Dataset expansion strengthens role-based robustness under prefilling attacks. Because Expanded and Non-Expanded models differ in baseline safety, absolute PA ASR andΔare not directly comparable. 16 Table 7 Average ASR (%,↓) under prefilling attacks (PA) across Non-Expanded and Expanded settings. w/o PA and PA denote the ASR before and under PA, respectively, whileΔmeasures the increase caused by the attack. For each data setting, we defineΔ (2) = Δ SFT − Δ SSRFT to measure the additional PA robustness provided by role-oriented supervision beyond explicit refusal learning (↑). Dataset expansion increasesΔ (2) for most models, indicating a stronger contribution from safe-role internalization even though it does not uniformly reduce the absolute PA ASR or Δ of SSRFT. Model Non-ExpandedExpanded SFTSSRFT 횫 (2) SFTSSRFT 횫 (2) w/o PA PA횫 w/o PA PA횫w/o PA PA횫 w/o PA PA횫 Base Models Qwen2.5-3B1.44 10.99 9.55 0.15 4.23 4.08 5.47 1.95 41.28 39.33 1.15 19.59 18.44 20.89 Gemma2-2B3.39 40.60 37.21 0.08 4.44 4.36 32.85 6.49 65.12 58.63 1.09 23.28 22.19 36.44 Llama3.1-8B3.76 39.24 35.48 1.32 26.60 25.28 10.20 3.14 42.64 39.50 0.92 16.54 15.62 23.88 LRC-4B0.27 7.67 7.40 0.00 3.55 3.55 3.85 0.16 12.72 12.56 0.16 4.98 4.82 7.74 Instruct Models Qwen2.5-3B-Ins 0.58 15.63 15.05 0.08 23.35 23.27 –8.22 0.34 17.24 16.90 0.16 28.58 28.42 –11.52 Qwen3-4B-Ins0.00 6.31 6.31 0.00 4.97 4.97 1.34 0.00 6.74 6.74 0.00 1.86 1.86 4.88 Therefore, we compare each SSRFT model against its matched SFT baseline and defineΔ (2) = Δ SFT −Δ SSRFT as a proxy for the additional prefilling robustness associated with role-oriented supervision. A largerΔ (2) indicates that SSRFT degrades less than its matched SFT counterpart under PA. Expansion increasesΔ (2) for most models, including Llama3.1-8B (10.20% to 23.88%) and Qwen3-4B-Instruct (1.34% to 4.88%). Even for Gemma2-2B, where absolute PA performance deteriorates after expansion,Δ (2) increases from 32.85% to 36.44%. Thus, expansion primarily strengthens role-based protection, rather than uniformly improving explicit refusal robustness. Interpreting Expansion through JRS and RPS. These observations motivate a conceptual decomposition of SSRFT into two complementary sources of safety: •Jailbreak Rejection Security (JRS): protection primarily associated with explicit jailbreak-refusal super- vision. It is highly effective on standard benchmarks but is more susceptible to over-refusal and prefix manipulation. •Role-Playing Security (RPS): protection associated with role-oriented supervision, where safety decisions are guided by the internalized safe role throughout generation. Dataset expansion increases the diversity and relative weight of role-oriented interactions, shifting the learned behavior from JRS toward RPS. Within this interpretation, the largerΔ (2) after expansion indicates a stronger relative robustness advantage from role-oriented supervision. This explains why Expanded SRQA can exhibit slightly weaker standard ASR while providing better helpfulness and stronger role-based protection under prefilling attacks. Overall, dataset expansion trades some explicit refusal strength for stronger role-based protection, helping SSRFT better internalize a safe role while reducing over-refusal. 4.6 Training Dynamics and Model Dependence We analyze the loss curves (Figure 6), gradient norms (Figure 7), and preceding results to understand why SSRFT benefits some models more than others. The analysis highlights two factors governing safe-role internalization: existing alignment and model capability. Existing Alignment Increases Internalization Difficulty. SSRFT provides substantially larger gains on Base models than on already aligned Instruct models. We attribute this difference partly to existing behavioral alignment. Table 6 shows low persona entropy for the evaluated Instruct models; notably, Qwen2.5-3B-Instruct 17 050100150200 1 2 Loss Qwen2.5-3B 01020304050 1 2 3 Gemma2-2B 020406080 1 2 3 Llama3.1-8B 020406080100 Steps 1 2 Loss LRC-4B 050100150200 Steps 1 2 Qwen2.5-3B-Ins 050100150 Steps 1 2 3 Qwen3-4B-Ins SSRFT (Expanded)SSRFT (Non-Expanded)SFT (Expanded)SFT (Non-Expanded) Figure 6 Training loss curves across different models. Compared with standard SFT, SSRFT exhibits similar convergence behavior on stronger Base models but converges more slowly on some Instruct and lower-capability models, suggesting that safe-role internalization imposes additional optimization difficulty that depends on existing alignment and model capability. is more stable than its Base counterpart. This suggests that instruction tuning can implicitly establish a behavioral profile before SSRFT is applied [9,13,47]. Consequently, SSRFT may need to modify an existing behavioral policy rather than learn the safe role from scratch. Figure 6 is consistent with this interpretation: Instruct models generally require more optimization before reaching their best validation performance under SSRFT. For Qwen2.5-3B, Figure 7 provides further evidence. Its Instruct variant exhibits substantially larger gradient norms than the Base model under both SFT and SSRFT, particularly under SSRFT, indicating stronger perturbation of previously aligned parameters. This perturbation may partly explain why SSRFT provides limited additional safety gains on some Instruct models. Model Capability Governs Safe-Role Internalization. Beyond existing alignment, the ability to fit role- oriented supervision also varies substantially across models. In Figure 6, Llama3.1-8B and Qwen3-4B-Instruct exhibit relatively small late-stage loss gaps between SSRFT and standard SFT, indicating that they learn role-oriented supervision effectively. Consistently, both models obtain strong prefilling robustness and benefit from dataset expansion. In contrast, Qwen2.5-3B maintains a persistent SSRFT–SFT loss gap, aligning with its comparatively modest safety gains. This behavior is consistent with the role-internalization objective in §3.6: SSRFT requires the model not only to learn what responses are appropriate, but also to infer and consistently apply the latent behavioral principles that determine how to respond. Safe-role internalization therefore places additional demands on abstraction and behavioral modeling. Differences in model architecture, pretraining, and reasoning capability may consequently produce different degrees of role internalization and persona stability. 4.7 General Capability Preservation General-Capability Evaluation. We evaluate whether role-oriented supervision introduces additional catastrophic forgetting [44] relative to standard SFT. Following prior studies [26,46,72], we evaluate zero-shot performance on nine benchmarks using thelm_eval_harnessframework [20] with the HuggingFace Transformers backend [70]. The benchmarks cover four capability groups: Scientific and Logical Reasoning (ARC-E [11], ARC-C [11], and LogiQA [39]), Commonsense Reasoning (CommonsenseQA (CSQA) [62], PIQA [4], and WinoGrande (WinoG) [55]), Reading Comprehension (BoolQ [10]), and Factual Knowledge (SciQ [69] and MMLU [27,28]). Following Hao et al.[26], we report Accuracy Norm for ARC-C and LogiQA, and Accuracy for the remaining benchmarks. 18 050100150200 10 0 10 1 Grad Norm Qwen2.5-3B 01020304050 10 0 10 1 10 2 Gemma2-2B 020406080 10 0 10 1 10 2 10 3 Llama3.1-8B 020406080100 Steps 10 0 10 1 10 2 Grad Norm LRC-4B 050100150200 Steps 10 0 10 1 10 2 Qwen2.5-3B-Ins 050100150 Steps 10 0 10 1 10 2 Qwen3-4B-Ins SSRFT (Expanded)SSRFT (Non-Expanded)SFT (Expanded)SFT (Non-Expanded) Figure 7 Training gradient norm (log scale) across different models. Gradient magnitudes vary substantially across backbones. Notably, Qwen2.5-3B-Instruct exhibits much larger gradients than its Base counterpart, especially under SSRFT, suggesting stronger perturbation when modifying an existing instruction-aligned policy. Capability Preservation. Table 8 shows that SSRFT preserves general capabilities comparably to standard SFT despite using substantially different role-oriented supervision. Across the nine benchmarks, SSRFT achieves similar overall performance to SFT, indicating no noticeably greater catastrophic forgetting. These results suggest that safe-role internalization primarily modifies behavioral preferences while largely preserving the model’s underlying reasoning and knowledge capabilities. 4.8 Comparison with Alternative Safety Alignment Paradigms Comparison Protocol. To contextualize parameter-level safe-role internalization (§3.6), we compare SSRFT with two alternatives: RLHF-based safety alignment and prompt-based role-playing, representing policy optimization during post-training and role conditioning during inference, respectively. For RLHF, we adopt PKU-SafeRLHF [30]. To make training tractable under a constrained resource budget, we subsample 1% of its official dataset and evaluate on LRC-4B, whose instruction-tuned initialization provides the conversational capability required for rollout generation. PKU-SafeRLHF is trained on four RTX PRO 6000 GPUs using BF16 precision; batch-related settings preserve the official effective batch size, while all remaining hyperparameters follow the released implementation. 5 For the prompt-based baseline, we prepend the same safe-role description to the system prompt without modifying model parameters. We use the same LRC-4B backbone and inference configuration as SSRFT (Table 3) for fair comparison. Table 9 summarizes standard ASR, prefilling ASR, and training cost. Efficiency under Constrained Training. Under this constrained setting, SSRFT is substantially more effective and computationally efficient than PKU-SafeRLHF. PKU-SafeRLHF does not improve standard safety: its average ASR changes from 37.19% to 37.40% and rises to 52.44% under prefilling attacks. In contrast, SSRFT achieves 18.83% and 4.98%, respectively, while requiring only a single-model SFT pipeline. The comparison also highlights a practical advantage of SSRFT: RLHF requires policy, reference, reward, value, and, in safety-oriented variants, cost models, whereas SSRFT directly optimizes role-conditioned supervision through standard SFT. Under limited data and computation, this provides a substantially simpler and more stable optimization path. Robustness over Inference-Time Role Prompting. Prompt-based role-playing also improves safety, reducing standard ASR from 37.19% to 33.03% and prefilling ASR from 24.72% to 16.71%, but remains substantially 5 https://github.com/PKU-Alignment/safe-rlhf/tree/main/scripts 19 Table 8 General capability evaluation (%). † Accuracy Norm (↑) is reported for ARC-C and LogiQA, and Accuracy (↑) for all remaining benchmarks. SSRFT preserves general model capabilities at a level comparable to standard SFT despite being trained with substantially different role-oriented supervision. ModelARC-C † ARC-E BoolQ CSQA LogiQA † MMLU PIQA SciQ WinoG Average (↑) Base Models Qwen2.5-3B46.7677.31 77.31 77.4833.7965.12 78.18 95.90 68.0368.88 SFT47.6176.98 77.74 77.3132.5765.16 78.18 95.60 67.9668.79 SSRFT (Ours) 48.0476.77 75.87 76.6633.3365.20 78.51 96.30 67.3268.67 Gemma2-2B49.5780.05 72.63 50.3728.1149.60 78.40 95.70 69.1463.73 SFT52.4780.72 72.91 54.8727.9650.77 78.62 96.30 69.3064.88 SSRFT (Ours) 56.0681.94 71.71 54.3827.9649.94 78.13 95.40 69.4665.00 Llama3.1-8B54.6182.07 83.00 70.6831.3464.06 79.11 96.70 74.1970.64 SFT56.2382.95 83.21 70.9331.1863.99 79.49 97.10 73.2470.93 SSRFT (Ours) 56.6682.70 82.87 71.0131.6464.83 79.27 96.70 74.6671.15 LRC-4B52.1378.87 85.02 78.4634.4164.45 76.39 93.90 68.1970.20 SFT53.8479.21 84.77 78.5434.1064.82 76.88 95.40 67.8870.60 SSRFT (Ours) 55.0378.87 84.43 78.7134.7265.03 76.61 95.20 69.3070.88 Instruct Models Qwen2.5-3B-Ins 48.1277.02 80.06 78.5432.2665.40 77.48 94.70 69.4669.23 SFT49.5778.24 80.46 78.4630.8865.15 77.64 95.90 69.7769.56 SSRFT (Ours) 47.3577.27 81.25 78.3832.1065.20 77.53 95.10 68.9869.24 Qwen3-4B-Ins58.3682.95 84.77 77.4032.1070.55 75.84 95.70 67.9671.74 SFT59.2283.71 85.57 77.4830.4170.99 76.39 97.00 68.9872.19 SSRFT (Ours) 60.7583.84 85.23 77.9732.2670.62 76.50 96.80 68.3572.48 Table 9 Summary of average ASR (%,↓) across different evaluation settings and training time. Standard Average ASR denotes the average ASR over the five safety benchmarks in § 4.2, while PA. Average ASR denotes the average ASR over the three benchmarks under prefilling attacks in § 4.3. Compared with RLHF and prompt-based role-playing, SSRFT achieves the lowest ASR under both standard and prefilling evaluations while requiring substantially lower training cost than RLHF, demonstrating the effectiveness and efficiency of parameter-level safe-role internalization. ModelStandard Average ASR (%,↓) PA. Average ASR (%,↓) Training Time (↓) LRC-4B37.1924.72– PKU-SafeRLHF37.4052.443m 32s× 4 RP via Prompt33.0316.71– SSRFT (Ours)18.834.984m 27s behind SSRFT. The difference reflects where the safe role is represented: prompt-based methods rely on continued adherence to an external role instruction, which can compete with adversarial instructions, whereas SSRFT internalizes the role into model parameters through supervised fine-tuning. Consequently, parameter-level role internalization exhibits substantially stronger robustness across the evaluated jailbreak settings. 20 5 Conclusions In this work, we proposed SSRFT, the first safe role internalization framework for LLM safety alignment. Instead of relying on large collections of jailbreak-refusal pairs or computationally expensive RLHF, SSRFT internalizes a predefined safe role into model parameters through role-oriented supervised fine-tuning. By shifting the alignment objective from memorizing refusal patterns to learning safety-oriented behavioral principles, SSRFT enables models to exhibit more consistent and robust behaviors across diverse interaction scenarios. Extensive experiments on multiple Base and Instruct models demonstrate that SSRFT generally improves the robustness and generalization of safety alignment while preserving general model capabili- ties. Further analyses reveal that safe-role internalization constitutes a fundamentally different alignment mechanism from conventional refusal-based supervision. Beyond its empirical effectiveness, SSRFT provides a practical and resource-efficient alternative to existing safety alignment methods, making robust safety alignment more accessible under limited computational budgets. Future work will explore scaling SSRFT to larger post-training settings, improving its adaptability across diverse model families, and extending safe-role internalization to dynamic and context-dependent safety alignment. References [1]Qingyao Ai, Ting Bai, Zhao Cao, Yi Chang, Jiawei Chen, Zhumin Chen, Zhiyong Cheng, Shoubin Dong, Zhicheng Dou, Fuli Feng, et al. Information Retrieval meets Large Language Models: A strategic report from Chinese IR community. AI Open, 4:80–90, 2023. URLhttps://w.sciencedirect.com/science/article/pii/ S2666651023000049. [2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.05862, 2022. URL https://arxiv.org/abs/2204.05862. [3] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Y Zou. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. In International Conference on Learning Representations (ICLR), volume 2024, pages 34196–34216, 2024. URL https://openreview.net/forum?id=gT5hALch9z. [4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 34, pages 7432–7439, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239. [5]Gregory J Boyle. Myers-Briggs Type Indicator (MBTI): Some Psychometric Limitations. Australian Psychologist, 30(1):71–74, 1995. URLhttps://aps.onlinelibrary.wiley.com/doi/abs/10.1111/j.1742-9544.1995. tb01750.x. [6]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–1901, 2020. URL https://dl.acm.org/doi/abs/10.5555/3495724.3495883. [7]Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tramèr, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 61478–61500, 2023. URL https://openreview.net/forum?id=OQQoD8Vc3B. [8]Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42, 2025. URLhttps://ieeexplore.ieee.org/abstract/document/10992337/. [9]Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv preprint arXiv:2507.21509, 2025. URLhttps://arxiv. org/abs/2507.21509. 21 [10]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (NAACL), pages 2924–2936, 2019. URL https://aclanthology.org/N19-1300/. [11]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457. [12]Paul T Costa and Robert R McCrae. NEO Personality Inventory Revised (NEO-PI-R). Psychological Assess- ment Resources Odessa, FL, 1992. URLhttps://w.januszlipowski.com/Psychology/Personality/ Personality_Assessment_(NEO-PI-R).pdf. [13]Jiaxi Cui, Liuzhenghao Lv, Jing Wen, Rongsheng Wang, Jing Tang, YongHong Tian, and Li Yuan. Machine Mindset: An MBTI Exploration of Large Language Models. arXiv preprint arXiv:2312.12999, 2023. URLhttps: //arxiv.org/abs/2312.12999. [14] Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An Over-Refusal Benchmark for Large Language Models. In International Conference on Machine Learning (ICML), pages 11515–11542, 2025. URLhttps: //openreview.net/forum?id=CdFnEu0JZV. [15] Juntao Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback. In International Conference on Learning Representations (ICLR), volume 2024, pages 50750–50777, 2024. URL https://openreview.net/forum?id=TyFrPOKYXw. [16]Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in ChatGPT: Analyzing Persona-assigned Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236–1270, 2023. URL https://aclanthology.org/2023.findings-emnlp.88/. [17] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3029–3051, 2023. URL https://aclanthology.org/2023.emnlp-main.183/. [18]Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics, 50 (3):1097–1179, 2024. URL https://direct.mit.edu/coli/article/50/3/1097/121961. [19] Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In International Conference on Machine Learning (ICML), pages 10835–10866, 2023. URLhttps://openreview.net/forum?id= bBLjms8nZE. [20]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. Language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602. [21]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence (NMI), 2(11):665–673, 2020. URL https://w.nature.com/articles/s42256-020-00257-z. [22]Lewis R Goldberg et al.A broad-bandwidth, public domain, personality inventory measur- ing the lower-level facets of several five-factor models. Personality psychology in Europe, 7(1):7–28, 1999. URLhttps://admin.umt.edu.pk/Media/Site/STD/FileManager/OsamaArticle/26august2015/ A%20broad-bandwidth%20inventory.pdf. [23]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783. [24] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong 22 Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645:633–638, 2025. URL https://w.nature.com/articles/s41586-025-09422-z. [25] Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen. World Values Survey: Round Seven – Country-Pooled Datafile Version 6.0, 2022. URL https://doi.org/10.14281/18241.24. [26]Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, and Jun Yu. A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), volume 38, pages 53678–53710, 2025. URLhttps://openreview.net/forum?id= LVDRJE4xQ2. [27] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI With Shared Human Values. In International Conference on Learning Representations (ICLR), 2021. URLhttps: //openreview.net/forum?id=dNy_RKzJacY. [28] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ. [29] Ayana Hussain, Patrick Zhao, and Nicholas Vincent. An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), volume 8, pages 1290–1301, 2025. URL https://doi.org/10.1609/aies.v8i2.36630. [30] Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, and Yaodong Yang. PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 31983–32016, 2025. URL https://aclanthology.org/2025.acl-long.1544/. [31] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys (CSUR), 55(12): 1–38, 2023. URL https://dl.acm.org/doi/full/10.1145/3571730. [32]Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents. arXiv preprint arXiv:2505.15117, 2025. URLhttps: //arxiv.org/abs/2505.15117. [33] Oliver P John, Laura P Naumann, and Christopher J Soto. Paradigm Shift to the Integrative Big Five Trait Taxonomy. Handbook of Personality: Theory and Research, 3(2):114–158, 2008. URLhttps://psycnet.apa.org/record/ 2008-11667-004. [34]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 9459–9474, 2020. URLhttps://proceedings.neurips.c/paper/2020/file/ 6b493230205f780e1bc26945df7481e5-Paper.pdf. [35]Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. ChatHaruhi: Reviving Anime Character in Reality via Large Language Model. arXiv preprint arXiv:2308.09597, 2023. URL https://arxiv.org/abs/2308.09597. [36]Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. SALAD- Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3923–3954, 2024. URLhttps://aclanthology.org/ 2024.findings-acl.235/. [37]Yakai Li, Jiekang Hu, Weiduan Sang, Luping Ma, Dongsheng Nie, Weijuan Zhang, Aimin Yu, Yi Su, Qingjia Huang, and Qihang Zhou. Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models. arXiv preprint arXiv:2504.21038, 2025. URL https://arxiv.org/abs/2504.21038. [38] Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, Charu Aggarwal, Xiang Zhang, and Suhang Wang. A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, 23 Optimizations, Evaluations, and Applications. arXiv preprint arXiv:2510.16724, 2025. URLhttps://arxiv.org/ abs/2510.16724. [39] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence (IJCAI), pages 3622–3628, 2021. URLhttps: //dl.acm.org/doi/abs/10.5555/3491440.3491941. [40] Sheng Liu, Qiang Sheng, Danding Wang, Yang Li, Guang Yang, and Juan Cao. Forewarned is Forearmed: Pre- Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 53728–53741, 2025. URLhttps://aclanthology. org/2025.findings-emnlp.266/. [41]Shuyi Liu, Simiao Cui, Haoran Bu, Yuming Shang, and Xi Zhang. JailBench: A Comprehensive Chinese Security Assessment Benchmark for Large Language Models. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 156–167, 2025. URLhttps://link.springer.com/chapter/10.1007/978-981-96-8186-0_ 13. [42] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In International Conference on Learning Representations (ICLR), volume 2024, pages 56174–56194, 2024. URL https://openreview.net/forum?id=7Jwpw4qKkb. [43]Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 7828–7840, 2024. URLhttps://aclanthology.org/2024.acl-long. 423/. [44]Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning. IEEE Transactions on Audio, Speech and Language Processing, pages 3776–3786, 2025. URL https://ieeexplore.ieee.org/abstract/document/11151751. [45]Fengran Mo, Kelong Mao, Ziliang Zhao, Hongjin Qian, Haonan Chen, Yiruo Cheng, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Jian-Yun Nie. A Survey of Conversational Search. ACM Transactions on Information Systems (TOIS), 43(6): 1–50, 2025. URL https://dl.acm.org/doi/full/10.1145/3759453. [46]Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Moham- mad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact Language Models via Pruning and Knowledge Distillation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), volume 37, pages 41076–41102, 2024. URL https://openreview.net/forum?id=9U0nLnNMJ7. [47]Keyu Pan and Yawen Zeng. Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models. arXiv preprint arXiv:2307.16180, 2023. URL https://arxiv.org/abs/2307.16180. [48] Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation. In International Conference on Machine Learning (ICML), pages 39416–39447, 2024. URL https://openreview.net/forum?id=l7shXGuGBT. [49]Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pages 1–22, 2023. URLhttps://dl.acm.org/doi/abs/10.1145/ 3586183.3606763. [50]Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety Alignment Should be Made More Than Just a Few Tokens Deep. In International Conference on Learning Representations (ICLR), volume 2025, pages 54911–54941, 2025. URLhttps://openreview.net/forum? id=6Mxhg9PtDE. [51]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019. URLhttps://papers.baulab.info/papers/ Radford-2018.pdf. 24 [52]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 53728–53741, 2023. URLhttps: //openreview.net/forum?id=HPuSIXJaa9&utm. [53] Sonia Roccas, Lilach Sagiv, Shalom H Schwartz, and Ariel Knafo. The Big Five Personality Factors and Personal Values. Personality and Social Psychology Bulletin, 28(6):789–801, 2002. URLhttps://journals.sagepub.com/ doi/abs/10.1177/0146167202289008. [54] Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 5377–5400, 2024. URL https://aclanthology.org/2024.naacl-long.301/. [55]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Communications of the ACM (CACM), 64(9):99–106, 2021. URL https://dl.acm.org/ doi/abs/10.1145/3474381. [56] Alireza Salemi and Hamed Zamani. Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval- Augmented Large Language Models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 741–751, 2024. URLhttps://dl.acm.org/doi/abs/10.1145/ 3626772.3657733. [57] Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, 623(7987): 493–498, 2023. URL https://w.nature.com/articles/s41586-023-06647-8. [58]Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-LLM: A Trainable Agent for Role-Playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 13153–13187, 2023. URL https://aclanthology.org/2023.emnlp-main.814/. [59] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 1671–1685, 2024. URLhttps://dl.acm.org/ doi/abs/10.1145/3658644.3670388. [60]Sofia Eleni Spatharioti, David M Rothschild, Daniel G Goldstein, and Jake M Hofman. Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment. arXiv preprint arXiv:2307.03744, 2023. URL https://arxiv.org/abs/2307.03744. [61] Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, et al. MOSS: An Open Conversational Large Language Model. Machine Intelligence Research, 21(5):888–905, 2024. URL https://link.springer.com/article/10.1007/s11633-024-1502-8. [62] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (NAACL), pages 4149–4158, 2019. URL https://aclanthology.org/N19-1421/. [63]Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving Open Language Models at a Practical Size. arXiv preprint arXiv:2408.00118, 2024. URL https://arxiv.org/abs/2408.00118. [64]Qwen Team. Qwen3-Max: Just Scale it, September 2025. URLhttps://qwen.ai/blog?id= 241398b9cd6353de490b0f82806c7848c5d2777d. [65]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/abs/2307.09288. [66]Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Large Search Model: Redefining Search Stack in the Era of LLMs. In ACM SIGIR Forum, volume 57, pages 1–16, 2024. URLhttps: //dl.acm.org/doi/abs/10.1145/3642979.3643006. 25 [67]Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail? In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 80079–80110, 2023. URL https://openreview.net/forum?id=jA235JGM09. [68] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837, 2022. URLhttps://openreview.net/ forum?id=_VjQlMeSB_J. [69] Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing Multiple Choice Science Questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, 2017. URLhttps://aclanthology.org/W17-4413/. [70]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45, 2020. URLhttps://aclanthology.org/2020. emnlp-demos.6/. [71] Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges. arXiv preprint arXiv:2508.05668, 2025. URL https://arxiv.org/abs/2508.05668. [72] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. In International Conference on Learning Representations (ICLR), volume 2024, pages 5385–5409, 2024. URL https://openreview.net/forum?id=09iOdaeOzp. [73]Ruiyun Xu, Yue Feng, and Hailiang Chen. ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience. arXiv preprint arXiv:2307.01135, 2023. URL https://arxiv.org/abs/2307.01135. [74] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115, 2024. URLhttps: //arxiv.org/abs/2412.15115v2. [75]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. URLhttps://arxiv. org/abs/2505.09388. [76] Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly. High-Confidence Computing, 4(2):100211, 2024. URL https://w.sciencedirect.com/science/article/pii/S266729522400014X. [77]Jifan Yu, Xiaohan Zhang, Yifan Xu, Xuanyu Lei, Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and Jie Tang. XDAI: A Tuning-free Framework for Exploiting Pre-trained Language Models in Knowledge Grounded Dialogue Generation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 4422–4432, 2022. URL https://dl.acm.org/doi/abs/10.1145/3534678.3539135. [78]Jiawei Zhang, Andrew Estornell, David D Baek, Bo Li, and Xiaojun Xu. Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth. arXiv preprint arXiv:2510.18081, 2025. URLhttps://arxiv.org/abs/ 2510.18081. [79]Weixiang Zhao, Yulin Hu, Yang Deng, Jiahe Guo, Xingyu Sui, Xinyang Han, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, et al. Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 11112–11137, 2025. URL https://aclanthology.org/2025.acl-long.544/. [80]Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large Language Models for Information Retrieval: A Survey. ACM Transactions on Information Systems (TOIS), 44(1):1–54, 2025. URL https://dl.acm.org/doi/full/10.1145/3748304. [81] Noah Ziems, Wenhao Yu, Zhihan Zhang, and Meng Jiang. Large Language Models are Built-in Autoregressive 26 Search Engines. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2666–2678, 2023. URL https://aclanthology.org/2023.findings-acl.167/. [82] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043, 2023. URLhttps://arxiv. org/abs/2307.15043. 27