Paper deep dive
RAJ-PGA: Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework for Large Reasoning Models
Jianhao Chen, Mayi Xu, Haoyang Chen, Xiaohu Li, Xiangyu Zhang, Jianjie Huang, Zheng Wang, Xiaochun Cao, Tieyun Qian
Models: DeepSeek-R1-0528-Qwen3-8B, DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-V3.2-Exp, Gemini-2.5-Pro, GLM-4.6, Qwen3-Plus
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 12:47:11 AM
Summary
The paper introduces RAJ-PGA, a framework for Large Reasoning Models (LRMs) that addresses safety vulnerabilities arising from internal reasoning chains. It proposes a 'Reasoning-Activated Jailbreak' (RAJ) via concretization to expose vulnerabilities and a 'Principle-Guided Alignment' (PGA) mechanism to transform harmful traces into safe, constructive responses, resulting in a 3,989-sample dataset that improves defense success rates by up to 29.5% without degrading reasoning capabilities.
Entities (5)
Relation Signals (3)
PGA dataset → enhances → Large Reasoning Models
confidence 95% · Extensive experiments show that fine-tuning LRMs with PGA dataset significantly enhances model safety.
RAJ-PGA → produces → PGA dataset
confidence 95% · We introduce the PGA dataset, a verified alignment dataset containing 3,989 samples using our proposed method.
RAJ-PGA → mitigates → Large Reasoning Models
confidence 90% · To systematically mitigate this vulnerability, we further develop a scalable framework for constructing high-quality safety alignment datasets.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Reasoning Models (LRMs) face a distinct safety vulnerability: their internal reasoning chains may generate harmful content even when the final output appears benign. To address this overlooked risk, we first propose a novel attack paradigm, Reasoning-Activated Jailbreak (RAJ) via Concretization, which demonstrates that refining malicious prompts to be more specific can trigger step-by-step logical reasoning that overrides the model's safety protocols. To systematically mitigate this vulnerability, we further develop a scalable framework for constructing high-quality safety alignment datasets. This framework first leverages the RAJ attack to elicit challenging harmful reasoning chains from LRMs, then transforms these high-risk traces into safe, constructive, and educational responses through a tailored Principle-Guided Alignment (PGA) mechanism. Then, we introduce the PGA dataset, a verified alignment dataset containing 3,989 samples using our proposed method. Extensive experiments show that fine-tuning LRMs with PGA dataset significantly enhances model safety, achieving up to a 29.5% improvement in defense success rates across multiple jailbreak benchmarks. Critically, our approach not only defends against sophisticated reasoning-based attacks but also preserves, even enhances, the model's general reasoning capabilities. This work provides a scalable and effective pathway for safety alignment in reasoning-intensive AI systems, addressing the core trade-off between safety and functional performance.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
70,855 characters extracted from source content.
Expand or collapse full text
1 RAJ-PGA: Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework for Large Reasoning Models Jianhao Chen, Mayi Xu, Haoyang Chen, Xiaohu Li, Xiangyu Zhang, Jianjie Huang, Zheng Wang, Xiaochun Cao, and Tieyun Qian, Abstract—Large Reasoning Models (LRMs) face a distinct safety vulnerability: their internal reasoning chains may generate harmful content even when the final output appears benign. To address this overlooked risk, we first propose a novel attack paradigm, Reasoning-Activated Jailbreak (RAJ) via Concretiza- tion, which demonstrates that refining malicious prompts to be more specific can trigger step-by-step logical reasoning that overrides the model’s safety protocols. To systematically mitigate this vulnerability, we further develop a scalable framework for constructing high-quality safety alignment datasets. This frame- work first leverages the RAJ attack to elicit challenging harmful reasoning chains from LRMs, then transforms these high-risk traces into safe, constructive, and educational responses through a tailored Principle-Guided Alignment (PGA) mechanism. Then, we introduce the PGA dataset, a verified alignment dataset containing 3,989 samples using our proposed method. Extensive experiments show that fine-tuning LRMs with PGA dataset significantly enhances model safety, achieving up to a 29.5% improvement in defense success rates across multiple jailbreak benchmarks. Critically, our approach not only defends against sophisticated reasoning-based attacks but also preserves, even enhances, the model’s general reasoning capabilities. This work provides a scalable and effective pathway for safety alignment in reasoning-intensive AI systems, addressing the core trade-off between safety and functional performance. Index Terms—Large Reasoning Models, Jailbreak Attack, AI Safety, Safety Alignment, Prompt Engineering. I. INTRODUCTION T HE emergence of large language models (LLMs) has revolutionized many fields, such as natural language processing and even complex problem solving [1]. Building on this, the recently emerged large reasoning models (LRMs) are more focused on processing tasks that require deep and struc- tured reasoning [2], [3]. LRMs, represented by the OpenAI- o1 [4] and DeepSeek-R1 series [5], have demonstrated their Jianhao Chen, Mayi Xu and Haoyang Chen contributed equally to this work. Corresponding author: Tieyun Qian. Jianhao Chen, Mayi Xu, Haoyang Chen, Xiaohu Li, Zheng Wang, and Tieyun Qian are with the School of Computer Science, Wuhan University, Wuhan 430072, China. Jianhao Chen, Haoyang Chen, Zheng Wang, and Tieyun Qian are also with the Zhongguancun Academy, Beijing 100080, China (email: qty@whu.edu.cn). Xiangyu Zhang is with the College of Cryptology and Cyber Science, Nankai University, Tianjin 300350, China, and also with Zhongguancun Academy, Beijing 100080, China. Jianjie Huang and Xiaochun Cao are with the School of Cybersecurity and Technology, Sun Yat-sen University, Shenzhen 518107, China, and also with Zhongguancun Academy, Beijing 100080, China. Code and data are available at https://github.com/JianhaoChen2025/ LRMsafety Jailbreaking Enhance Reasoning Reasoning Requirements Safety Requirements Vertical Thinking Lateral Thinking Instruct- Following Instruct- Refuse ASR 0 1 2 3 4 5100 80 60 40 20 0 Original Concretization Rewrite 2.04 2.41 4.36 4.10 42 72 45 20 44 61 Score Rate (%) (a) Competition between reasoning and safety requirements of LRMs (b) Reasoning and safety performance of LRM before and after concretization rewriting J Fig. 1. Conceptual Framework of the Safety-Reasoning Dilemma and the Concretization-Based Jailbreak Attack. (a) The inherent competition between reasoning goals and safety constraints. (b) Performance shift in vertical/lateral thinking and ASR before and after concretization. (c) Illustration of how de- tailed prompts bypass safety filters by activating vertical thinking trajectories. outstanding ability to generate long chains of reasoning to solve complex problems, making them increasingly important in applications such as code assistance and scientific discov- ery [6], [7]. Unlike LLMs, which directly respond to user queries, LRMs’ output consists of two parts: the reasoning phase and the response phase [8]. LRMs generate structured chains of thought [9] by adopting a<think>...</think> reasoning format before producing the final response. This internal monologue allows them to tackle complex problems in domains like code assistance and scientific discovery with multiple intermediate steps, even in the absence of advanced prompting strategies [10]. However, while structured reasoning processes improve LRMs’ capability, they also introduce more unique safety risks arXiv:2508.12897v2 [cs.AI] 30 Dec 2025 2 and vulnerabilities [11]. When facing malicious instructions, LRMs’ reasoning phase can easily exhibit unsafe behaviors, even though the final response phase is safe [12]. In addition, due to the powerful reasoning ability of LRMs, their danger- ous reasoning chain often contains more detailed guidance, which potentially makes the safety risk of LRMs higher than LLMs [13], and the deployment of these models in critical domains means that such vulnerabilities can lead to severe real-world consequences [14]. Therefore, we need to construct a new alignment paradigm based on the characteristics of LRM to avoid potential safety risks. Previous studies on LLMs jailbreak [15]–[17] attribute the reasons for LLMs’ jailbreak to the target competition and generalization mismatch [18], where the target competition refers to the competition between the LLMs’ instruction- following target and the safety target. In light of this, we assume that the LRMs jailbreak can also be attributed to the specific target competition. We delve into the generation process of LRMs and speculate that there is a reasoning target that causes LRMs to compromise safety for competition, as shown in Fig.1 (a). We conceptualize the safety dilemma in LRMs as a fun- damental clash of cognitive modes, drawing an analogy from Edward de Bono’s distinction between vertical thinking, which is a logical and structured approach prioritizing procedural correctness, and lateral thinking, which emphasizes breaking traditional patterns to evaluate problems from new angles [19]. We posit that a successful jailbreak occurs when a malicious prompt traps the model in a vertical thinking loop, compelling it to prioritize procedural logic at the expense of its lateral evaluative capacity. In this rigid state, the model’s logical inertia effectively suppresses its safety guardrails, which drives the LRM to maintain an integral reasoning process in CoT. Unlike standard LLMs, the LRMs’ deep reasoning mode creates a tunnel vision effect, where the pursuit of logical completeness leads the model to unintentionally bypass safety protocols. Consequently, we define a safe refusal not merely as a rejection, but as the successful activation of lateral thinking to re-evaluate the prompt’s intent. To empirically validate this conjecture, we enhance LRMs’ vertical thinking by systematically concretizing 100 harm- ful prompts from the 8 risk categories in JailbreakBench (JBB) [20]. We employ an evaluator LLM (Qwen3-Plus) to separately assign 0-5 scalar scores to quantify the ver- tical and lateral thinking of the responses from the victim LRM (DeepSeek-R1-Distill-Qwen-32B) shown in Fig. 1 (b). We evaluate instruction-following and refusal capabilities by calculating the proportion of responses that adhere to the given instructions. Specifically, we assessed whether the model followed instructions and answered as required, and whether it explicitly refused to answer the corresponding instructions. We measure safety using the attack success rate (ASR). The detailed evaluation methodology is provided in our open- source code repository. As can be seen from Fig. 1(b), the rewritten harmful prompts stimulated the LRMs’ vertical thinking ability and im- proved its instruction-following ability, indicating that concrete rewriting can stimulate the LRMs’ vertical thinking. Critically, this shift towards vertical thinking directly correlated with a higher ASR, as shown in Fig. 1(b). The improvement of the ASR indicates that LRMs’ safety performance has declined. These results provide strong evidence for our hypothesis: stimulating an LRM’s direct reasoning capability can override its safety alignment, thus creating a vulnerability that can be exploited for jailbreaking. Through the above analysis, we suggest that the reasoning process of LRMs will bring additional safety risks. The typical way to enhance safety is to fine-tune LRMs using safety alignment data. However, previous studies [21], [22] have revealed that pure safety alignment data will cause significant damage to the model’s reasoning ability. Then, the next question worth studying is: How to enhance safety without reducing the reasoning ability of LRMs? To address these challenges, we introduce the Reasoning- Activated Jailbreak and Principle-Guided Alignment (RAJ-PGA) framework. Drawing inspiration from Constitu- tional AI [23], our approach bridges the gap between adversar- ial attack and safety alignment. We first employ a Reasoning- Activated Jailbreak (RAJ) via Concretization strategy to sys- tematically expose latent vulnerabilities in LRMs. Subse- quently, we apply the Principle-Guided Alignment (PGA) strategy, defining five core principles to guide a rewrite model (Qwen-plus) in rewriting these jailbroken outputs into safe, constructive responses. By fine-tuning LRMs on this data, we shift the alignment paradigm from simple refusal to safe reasoning, enabling models to internalize safety constraints while maintaining their deductive capabilities. The overall architecture and workflow are illustrated in Fig. 2. The main contributions are summarized as follows: • Framework: We propose RAJ-PGA, a scalable paradigm that generates reasoning-activated jailbreaks via con- cretization attacks and transforms them into instructional safety responses through principle-guided rewriting. • Datasets: We open-source the RAJ (attack) and PGA (alignment) datasets. These rigorously filtered resources provide high-quality pairs of jailbreak triggers and aligned reasoning traces to facilitate reproducible safety research. • Performance: Extensive experiments demonstrate that fine-tuning with our PGA dataset improves the defense success rate by up to 29.5% across diverse benchmarks without compromising the model’s general reasoning abilities. I. RELATED WORK This section reviews recent advancements in the safety land- scape of LRM, focusing on emerging vulnerabilities inherent to reasoning mechanisms and safety alignment strategies. A. Safety Risks in Reasoning Models While the Chain-of-Thought (CoT) mechanism empowers LRMs with superior problem-solving capabilities, it simulta- neously expands the attack surface for adversarial exploita- tion [11]. Recent evaluations have exposed critical vulnera- bilities in state-of-the-art LRMs. For instance, SEAL [24] is a 3 Original Harmful Prompt Reasoning-Activated Jailbreak via Concretization Rewrite LRM Output (Controversial Reasoning + Response) Elicit Safety Validation (Llama-Guard + Qwen3Guard + GPT-5) Reasoning- Activated Jailbreak Dataset (N = 3,989) 5 Principles: Reframing, etc. Aligned Dataset (N = 3,989) (Safe Reasoning +Safe Response) Fine-tuning (LoRA) (Target: Open Source LRMs) Embedded System Prompt (Target: Closed Source LRMs) Result: Safe LRM Safety (↑SOTA) General Reasoning （←→） Rewrite Filter Constitutional Alignment Step 1: Jailbreak Generation Step 2: Alignment Transformation Step 3: Model Alignment Fig. 2. Architectural Overview of the Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework. The pipeline comprises three stages: (1) Jailbreak Generation, using Concretization to transform original harmful prompts into reasoning-activated harmful prompts, prompting the victim LRM generate controversial contents; (2) Alignment Transformation, employing a consensus-based safety validation and Principle-Guided Alignment to rewrite harmful contents into principle-aligned contents; (3) Model Alignment, where LRMs’ safety performance is enhanced by Principle-Guided Alignment strategy. novel jailbreak attack that uses stacked, adaptive encryption to override LRM reasoning and safety filters. By dynamically ad- justing cipher combinations, SEAL achieves an 80.8% success rate on GPT-o4-mini, surpassing current baselines by 27.2% across major models like DeepSeek-R1 and OpenAI o4. Zhou et al. [12] demonstrated that LRMs are significantly more sus- ceptible to prompt injection attacks than standard LLMs. Their findings highlight a safety decoupling phenomenon where the intermediate reasoning chain often harbors unsafe content, even when the final response appears benign. Subsequent large-scale evaluations [13] on datasets like StrongREJECT confirm that the structured reasoning process, while intended for logic, can be hijacked to elaborate on harmful instructions, generating detailed malicious guidance that traditional LLMs might refuse due to a lack of context depth. B. Safety Alignment for LRMs Safety alignment aims to constrain model behaviors within human values and societal norms [25], [26]. While Supervised Fine-Tuning (SFT) with safe instruction-response pairs has proven effective for standard LLMs [27], the unique two-stage generation process (reasoning followed by response) of LRMs renders traditional datasets inadequate [28]. To address this, recent works have developed LRM-specific alignment resources. SafeChain [13] introduces the first reasoning-aware safety dataset by filtering 50K stochastic sam- ples to curate 40K safe reasoning-response pairs. Alternatively, STAR-1 [22] adopts a refusal-oriented approach, replacing the harmful reasoning process with an analysis of why the instruction should be rejected. However, this method discards the context of the malicious query, potentially limiting the model’s ability to understand complex adversarial intents. Un- safeChain [29] improves upon these methods by employing a correction-based supervision strategy, using GPT-4 to rewrite unsafe responses into safe ones, demonstrating that quality often trumps quantity in alignment data. The alignment tax, defined as the trade-off where safety alignment degrades general reasoning performance, remains a persistent challenge despite recent advances [21]. To mitigate this, STAIR [30] proposes a complex pipeline involving Safety-Aware Monte Carlo Tree Search (SI-MCTS) and pro- cess reward models to explore safe reasoning paths. While STAIR alleviates the performance regression, its reliance on search-based inference incurs substantial computational over- head. Unlike previous approaches that either sacrifice reasoning capabilities for safety (e.g., simple refusal) or require com- putationally expensive search procedures (e.g., STAIR), our work introduces RAJ-PGA strategy. Instead of searching for a safe path, we teach the model by rewriting harmful reasoning traces into safe, constructive, and principled thoughts. This strategy is not only computationally efficient but also provides the model with generalizable examples of safe reasoning patterns. By doing so, we aim to achieve a superior balance: enhancing safety robustness against sophisticated jailbreaks while preserving, and potentially even boosting, the model’s general reasoning intelligence. I. METHODOLOGY This section details our systematic method for creating the safety alignment dataset. The process consists of two primary stages: eliciting reasoning vulnerabilities and then transforming them into safe and educational examples. A. Reasoning-Activated Jailbreak via Concretization Unlike traditional LLMs, LRMs generate an explicit rea- soning trajectory r before producing the final response y. Formally, given an input prompt x, the joint probability of the model’s output can be decomposed as: P θ (r,y | x) = P θ (r | x)· P θ (y | x,r).(1) This decoupling allows us to analyze how reasoning-activated prompts influence the safety of both reasoning and response. As formalized in Algorithm 1, the construction of the Reasoning-Activated Jailbreak (RAJ) dataset D RAJ is guided by the safety inversion logic, which evaluates the model’s 4 responses to original prompts versus their concretized coun- terparts through a differential execution pipeline: D RAJ =(x c ,y c )| S(r o ,y o ) = safe∧ S(r c ,y c ) = unsafe, (2) where (x c ,y c ) denotes the concretized prompt and the corre- sponding model output including the reasoning chain r c . The function S(·) ∈ safe,unsafe represents the consensus- based safety validation (comprising automated guards and human auditing), where S = safe indicates a safe refusal and S = unsafe signifies a safety violation. This criterion ensures that D RAJ captures samples where reasoning-eliciting inputs effectively bypass the model’s intrinsic safety alignment. a) Stage 1: Concretization and Attack: We utilize our proposed rewriting method to transform an initial set of ma- licious prompts into a concretized dataset. Our initial pool of malicious prompts from PKU-SafeRLHF [31], which contains 44.6K carefully curated prompts spanning 19 fine-grained harm categories, and further labels harm severity at three levels (Minor/Moderate/Severe), enabling controlled sampling along both harm type and risk intensity. For each original prompt x o ∈X origin , we generate a concretized counterpart x c using a three-step rewriting strategy: • Intent Identification: Based on the original malicious question, LLM accurately determines its hidden malicious intent and understands the real needs behind the question, thereby providing a clear direction for subsequent rewrit- ing. • Association: Using the identified malicious intent, as- sociate the intent with things that can achieve the goal. Through these associations, we can build a richer problem context, expand the understanding of the problem, and enhance the LRMs’ tendency to generate more inferences. • Rewriting: Rewriting the question based on the original question and the associated components. The goal of rewriting is to make the question more specific, involve more details, and appear legal and harmless in form, so as to induce the model to generate output containing sensitive information. We then separately attack the victim LRM (DeepSeek-R1- Distill-Qwen-32B), π θ , using both datasets. This yields two sets of outputs: the reasoning chain and final response for the original prompts (r o ,y o ), and those for the concretized prompts (r c ,y c ). b) Stage 2: Consensus-Based Safety Evaluation: Previ- ous work [13] evaluated several advanced safety classifiers and demonstrated that Llama-Guard achieved superior accuracy. To further mitigate the risk of misclassification and enhance the robustness of our labeling process, we integrate Qwen3Guard- Gen-8B (G Q ) [32] as a complementary evaluator. In instances where G L and G Q disagree on the safety label, the conflicting sample is submitted to GPT-5 (G A ), acting as the final arbiter. The final safety decision S(u) is determined through a majority vote with expert arbitration logic. This mechanism is formally defined as: S(u) = ( G L (u),if G L (u) = G Q (u) G A (u), otherwise (3) where S(u)∈Safe, Unsafe. This rigorous multi-stage veri- fication protocol provides a clean and reliable data foundation for characterizing reasoning-activated vulnerabilities. c) Stage 3: Differential Filtering: The final reasoning- activated jailbreak dataset D RAJ is constructed based on the logic of safety inversion. S Origin ∧ (¬S Concretized ),(4) where S denotes the consolidated safety label. Algorithm 1 Two Stage Pipeline for Constructing Reasoning- Activated Jailbreak and Principle-Guided Alignment Datasets. Require: Base LRM π θ ; Original prompts X origin ; Concretizer C(·) Require: Safety Guards G L ,G Q ,G A ; Aligner M Rewrite ; Principle S CAI Ensure: Reasoning-Activated Jailbreak Dataset D RAJ ; Principle-Guided Alignment D PGA 1: Function IS SAFE(u): 2: if G L (u) == G Q (u) then 3:return G L (u) 4: else 5:return G A (u) 6: end if 7: End Function 8: D RAJ ←∅; D PGA ←∅ 9: Part 1: Construct Reasoning-Activated Jailbreak Dataset 10: for x o ∈X origin do 11: x c ← C(x o ) Concretized Prompt 12:(r o ,y o )← π θ (·|x o ); (r c ,y c )← π θ (·|x c ) Attack LRM 13: Ensemble Safety Judgment 14: S org ← IS SAFE(r o )∧ IS SAFE(y o ) 15: S conc ← IS SAFE(r c )∧ IS SAFE(y c ) 16: Differential Filtering 17:if S org ∧ (¬S conc ) then 18: D RAJ ←D RAJ ∪(x c ,y c ) 19:end if 20: end for 21: Audit: Perform Stratified Random Sampling (N = 100) on D RAJ to Verify Guard Reliability. 22: Part 2: Construct Principle-Guided Alignment 23: for (x c ,y c )∈D RAJ do 24: Rule-Guided Alignment 25: y h ← M Rewrite (x c ,y c ,S CAI ) 26:if y h is valid ∧ IS SAFE(y h ) then 27: D PGA ←D PGA ∪(x c ,y h ) 28:end if 29: end for 30: Audit: Perform Stratified Random Sampling (N = 100) on D PGA to Verify Alignment Quality. 31: return D PGA B. Principle-Guided Alignment Strategy for Safety Alignment In this section, we introduce the framework for transforming the verified jailbreak dataset D RAJ into a safety alignment 5 dataset D PGA . a) Rule-Guided Rewriting: We define a principle S CAI consisting of five specific strategies designed to mitigate harm while preserving helpfulness. A rewriting model (Qwen-plus) M Rewrite utilizes these rules to rewrite the harmful outputs in D RAJ into harmless versions y h . This process can be formulated as: y h = M Rewrite (x c ,y c ,S CAI ).(5) The five strategies S CAI are: • Goal Reframing: Disregard the user’s specific, poten- tially harmful method and focus on satisfying their un- derlying legitimate intent with a safer, more effective solution. • Information Downgrading: Retain theoretical (what and why) knowledge but deliberately omit or obscure critical actionable (how-to) details to mitigate risk. • Counterfactual & Risk Clarification: Rather than com- plying, explicitly reveal the hidden risks and severe consequences associated with the request. • Premise Rejection & Positive Reframing: Explicitly reject the harmful or unethical premises of the query and reframe the conversation into a constructive discussion. • Empathetic Redirection & Resource Guidance: Iden- tify signs of user distress, shift the focus to expressing empathy, and provide professional avenues for help. b) Automated Verification: To ensure high-quality and reliable data transformation, we implement a rigorous pro- cedural constraint and iterative refinement process. During the rewriting stage, M Rewrite is strictly constrained to gener- ate responses whose length is within 80% to 120% of the original harmful outputs, thereby ensuring that the depth of the reasoning chain and the richness of information are pre- served. Furthermore, we employ a multi-layered verification and correction mechanism: any samples that are still classified as unsafe after the initial rewriting undergo a second round of model-based rewriting, followed by meticulous human correction to resolve any remaining safety violations or logic inconsistencies. This comprehensive process ensures thatD RAJ is fully converted into safe, principle-aligned D PGA . Similar to the attack phase, we apply stratified random sampling on the final harmless dataset D PGA to manually inspect the quality and safety of the rewriting, ensuring the data is suitable for safety alignment training. C. Human Annotation We evaluated dataset quality through human annotation of 100 samples selected via stratified random sampling. Three independent annotators labeled each instance across six anno- tation tasks with binary classifications: reasoning and response labels for the Original, Concretized (RAJ), and PGA datasets. The reference standard is constructed using majority voting: labels agreed upon by two or more annotators are adopted as the ground truth. AI annotation quality is assessed by comparing AI-generated labels against this reference standard. For each task, the positive class (Target) is determined by the majority class in the reference standard. We computed accuracy, precision, recall, and F1-score, constructing con- fusion matrices to identify True Positives (TP) and False Positives (FP). All metrics are calculated separately for the six annotation tasks and are reported as percentages. IV. EXPERIMENT SETTINGS This section details the experimental methodology used to validate the proposed RAJ-PGA framework. We describe the datasets, model configurations, and evaluation metrics employed to assess both the safety alignment performance and the preservation of general reasoning abilities in LRMs. A. Safety Performance Evaluation To validate the safety performance of LRMs, we use the following representative datasets: AdvBench [33], a bench- mark comprising 520 prompts for diverse harmful behaviors designed to elicit harmful model responses. WildChat [34], a large dataset of 1 million user-ChatGPT conversations, offering diversity in potentially toxic use-cases for research. WildJailbreak [35], an open-source synthetic safety dataset with 262K prompt-response pairs, featuring both harmful queries and lookalike benign queries to mitigate exaggerated safety behaviors. StrongREJECT [36], a benchmark of 313 malicious prompts designed to evaluate jailbreak attacks, as- sesses whether these attacks enable malicious actors to utilize LLMs for harmful tasks. JailbreakBench [20], named JBB, is a representative collection of 100 distinct misuse behaviors categorized according to OpenAI’s usage policies. To evaluate the vulnerability and robustness of LRMs, we employ two metrics: Attack Success Rate (ASR) and Defense Success Rate (DSR). ASR is defined as the proportion of unsafe responses, while the DSR corresponds to the proportion of responses evaluated as safe: ASR = P N i=1 I(¬S (i) ) N × 100%,DSR = P N i=1 I(S (i) ) N × 100%, (6) where I(·) is the indicator function. These metrics allow us to uniformly assess the vulnerability and robustness of the LRM against malicious prompts. B. General Performance Evaluation To demonstrate that LRMs retain high reasoning capabili- ties, we use several typical benchmarks as follows. HumanEval [37], a benchmark for evaluating the ability of LLMs to solve programming questions primarily using Python code. MATH500 [38], a dataset of 500 challenging high school math competition questions covering seven subjects. TruthfulQA [39], a multiple-choice benchmark to measure whether an LLM is truthful in generating answers to questions, containing 684 questions that span 38 categories, including health, law, finance, and politics. ARC-Challenge [40], a multiple-choice question benchmark to test LLMs’ ability to solve 299 scientific questions. GPQA-Diamond [41]: a challenging dataset of 198 multiple-choice questions written by science domain experts. We evaluate the LRMs’ general ability based on their accuracy in answering questions from relevant benchmarks, where higher accuracy signifies superior general ability. 6 C. Models for Evaluation For the jailbreak attack in the reasoning phase of the reasoning-type large language model, we choose 6 DeepSeek series models [42] for experiments and employ DeepSeek- R1-Distill-Qwen-32B as the victim LRM to generate raw jailbreak reasoning and response contents. In the safety align- ment experiment, we select DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-0528-Qwen3- 8B and Qwen/Qwen3-8B. We also evaluate the effectiveness of RAJ strategies by applying them as system prompt guidelines for four commercial closed-source models: DeepSeek-V3.2-Exp [43], GLM4.6 [44], Qwen3-plus [45], and Gemini-2.5-pro [46]. D. Dataset Baselines We compare the performance of our PGA dataset against several publicly available safety-tuning datasets for LRMs: • STAR-1 [22]: A high-quality safety dataset of 1,000 sam- ples designed for LRMs. STAR-1 utilizes a deliberative reasoning paradigm where DeepSeek-R1 generates CoT trajectories grounding refusal decisions in specific safety policies. It employs a rigorous GPT-4o-based scoring sys- tem to ensure safety compliance and reasoning accuracy. • DirectRefusal [21]: A baseline dataset derived from BeaverTails-refusal, consisting of 1,000 harmful queries paired with a fixed, minimal thinking trajectory (e.g., “I should not answer this question!”), followed by immedi- ate rejection. It is used to evaluate alignment methods that bypass extended reasoning, contrasting with CoT approaches in terms of efficiency. • UnsafeChain [29]: A dataset adopting a correction- based alignment strategy for hard prompts. We utilize the full version (13,600), a random subset (1,000), and a hard-case subset (1,000). Unlike filtering methods, UnsafeChain uses GPT-4.1 to rewrite unsafe completions into safe, helpful responses equipped with CoT reasoning. • SafeChain [13]: A dataset of 40,000 safe responses to adversarial prompts, filtered by Llama-Guard. It fea- tures extensive CoT trajectories that explicitly reason about safety, ethics, and legal constraints. It benchmarks reasoning-aware alignment strategies to assess if long- context thinking preserves general reasoning capabilities. • STAIR-SFT [30]: An SFT dataset of 20,000 samples aggregated from PKU-SafeRLHF and other jailbreak datasets. It employs GPT-4o to synthesize structured CoT responses, strictly formatted with Problem Analysis, Reasoning, and Final Answer sections to enhance inter- pretability. • PGA: A dataset comprising 3,989 aligned prompt- response pairs, constructed by PGA strategy. To inves- tigate the relationship between dataset size and safety alignment performance, we constructed three subsets: PGA-1000, PGA-2000, and PGA-3000, which perform stratified random sampling from the full PGA dataset. After completing the dataset collection, we format it into a “<think> reasoning</think> response ” structure to build a training set for fine-tuning the LRM to evaluate the quality of the dataset. E. Training Settings To validate our proposed safety algorithm, we fine-tuned four LRMs using Parameter-Efficient Fine-Tuning (PEFT) with LoRA adapters, adhering to the setup from Un- safechain [29]. We trained the models for 2 epochs on 4 NVIDIA A800 GPUs. For memory and computational effi- ciency, we utilized mixed-precision FP16, 8-bit model load- ing, and gradient accumulation. Specifically, we employed a learning rate of 1× 10 −5 , a per-device batch size of 1 with 8 gradient accumulation steps (resulting in an effective batch size of 8), and a maximum sequence length of 2048. For the LoRA configuration, adapters are applied to the q_proj and v_proj modules with a rank (r) of 16, alpha (α) of 32, and a dropout rate of 0.05. The core objective of our fine-tuning process is to minimize the Causal Language Modeling (CLM) loss. Formally, for a given dataset D containing N samples, the loss function is defined as: L CLM (Φ) =− 1 N N X i=1 T X t=1 logP (y i,t |y i,<t ,x i ;θ 0 , Φ),(7) where x i represents the input prompt (e.g., the concretized malicious prompt), and y i,t denotes the t-th token of the target safety-aligned response. To maintain the fundamental reason- ing capabilities of LRMs while achieving safety alignment, the pre-trained weights θ 0 remain frozen, and the optimization is performed over the trainable LoRA parameters Φ: ∆θ = BA, B ∈ R d×r ,A∈ R r×k ,(8) where r ≪ min(d,k) is the rank of the adaptation. By ap- plying this formulation, the model learns to rectify its internal reasoning trajectory from harmful concretization toward safe and constructive outputs. V. RESULTS This section details the empirical findings of our study. We first quantify the ASR of concretization-based jailbreak attacks and construct the concretization-based jailbreak dataset, fol- lowed by an assessment of the safety improvements achieved through PGA strategies. Finally, we report on the impact of our methods on general utility and conduct ablation studies. A. Analysis of Concretization-Based Jailbreak Attacks To evaluate the effectiveness of our concretization-based jailbreak rewriting method, we construct a modified dataset by applying our concretization-based jailbreak rewriting method to the 520 original malicious prompts from AdvBench. We measure the ASR in both the reasoning and response phases using Llama-Guard-3-8B. Evaluating the original and rewritten prompts on the victim model (DeepSeek-R1-Distill-Qwen- 32B), we observe that the rewritten prompts substantially increase ASR in both phases (see Fig. 3), indicating degraded safety. Moreover, the average ASR in the reasoning phase is 7 consistently higher than in the response phase, underscoring the importance of rendering the reasoning phase harmless. We further verify this trend across both open-source and closed- source commercial LRMs in subsequent experiments. 020406080100 Reasoning ASR (%) =1.54% =14.81% =8.07% =22.30% =21.35% =22.88% =44.61% =37.50% =22.70% =23.08% =12.11% Original Concretized 020406080100 Response ASR (%) =5.38% =20.38% =16.73% =25.97% =21.15% =2.10% =24.61% =17.50% =7.88% =14.03% =8.66% Original Concretized R1-Qwen-1.5B R1-Qwen-7B R1-Llama-8B R1-Qwen-14B R1-Qwen-32B R1-Qwen3-8B Qwen3-8B DeepSeek-R1 Qwen-plus Gemini-2.5-pro GLM4.6 Fig. 3. Comparison Between ASR Original and ASR Concretized in Reasoning and Response Phases.∆ denotes ASR Concretized − ASR Original . B. Constructing the Reasoning-Activated Jailbreak Dataset Building on the observation that concretization can activate the model’s internal reasoning and potentially bypass safety guardrails, we construct a high-quality jailbreak dataset for subsequent safety alignment. a) Transform the Original PKU-SafeRLHF Dataset via Concretization Attack: We apply Algorithm 1 to PKU- SafeRLHF [31] by pairing each original prompt p with its concretized rewrite ̃p and querying the victim LRM for both. We then run the same safety check on both the generated reasoning and response. An example is included in RAJ if the original outputs are safe while the concretized outputs are unsafe. We refer to the resulting subset as the Reasoning- Activated Jailbreak (RAJ) dataset. TABLE I ASR MEASURED BY DIFFERENT SAFETY EVALUATORS ClassifierDatasetASR Reasoning ASR Response Llama-Guard-3 Original33.43%15.83% Concretized43.22%29.49% Qwen3Guard-Gen Original38.66%18.16% Concretized47.55%32.03% GPT-5 as Arbiter Original74.23%57.11% Concretized80.70%64.86% Our ensemble approach highlight the limitations of relying solely on open-source classifiers. We observed substantial dis- agreements between Llama-Guard-3 and Qwen3Guard: they provided conflicting labels on 9,488 instances (21.8%) for original prompts and 10,188 instances (23.5%) for concretized prompts. We therefore use GPT-5 as an automated arbiter to resolve conflicts. As shown in Table I, the arbiter flags a higher fraction of outputs as unsafe, implying that many disputed cases lie near the safety boundary and merit closer scrutiny. Based on the strict safety inversion criterion, we finally filter the initial pool down to 3,989 high-quality pairs. This subset, termed RAJ, serves as the foundation for our subsequent safety alignment experiments, representing the specific failure modes introduced by reasoning capabilities. b) Annotation Reliability and Human Validation: To evaluate the reliability of AI-based annotations, we employ three human experts to annotate a stratified sample (N = 100) independently. Table I shows strong agreement overall. Minor discrepancies mainly occur in the Original (Safe) group, where a few unsafe instances are occasionally labeled as safe; the Concretized (Unsafe) and PGA (Safe) groups exhibit consistently high agreement with only rare borderline cases. Notably, all groups achieved 100% recall, demonstrating that the AI system successfully identified all instances of the target class. The F1-scores range from 97.96% to 100%, indicating balanced performance between precision and recall across all annotation tasks. Overall, the annotation quality is sufficient for downstream analysis and dataset construction. TABLE I QUALITY ASSESSMENT OF AI-BASED ANNOTATION AGAINST HUMAN EXPERT GROUND TRUTH (ACCURACY, PRECISION, RECALL, AND F1-SCORE ARE REPORTED IN %). TaskTarget TP FP Accuracy Precision RecallF1 Reasoning Original safe96496.0096.00100.00 97.96 Response Original safe96496.0096.00100.00 97.96 Reasoning Concretized unsafe 100 0100.00100.00100.00 100.00 Response Concretized unsafe 98298.0098.00100.00 98.99 Reasoning PGA safe100 0100.00100.00100.00 100.00 Response PGA safe100 0100.00100.00100.00 100.00 c) Classification and Analysis of RAJ dataset: To sys- tematically analyze the RAJ dataset, we adopt a unified harm taxonomy derived roughly from Qwen3Guard [32]. As shown in Table I, the harm-category of malicious prompt distri- bution shifts markedly after concretization: most successful jailbreaks fall into Non-violent Illegal Acts (66.28%), followed by Unethical Acts (17.80%) and Violent Illegal Acts (8.95%). This suggests that concretization is particularly effective for prompts that demand procedural or actionable guidance. TABLE I DISTRIBUTION OF HARMFUL CATEGORIES ACROSS THE ORIGINAL PKU-SAFERLHF DATASET VERSUS THE REASONING-ACTIVATED JAILBREAK DATASET (ORIGINAL AND CONCRETIZED). PKU-SafeRLHFOriginalConcretized CategoryCountRatioCountRatioCountRatio Non-violent Illegal Acts 15,90136.62%1,89747.56%2,64466.28% Unethical Acts8,52819.64%69117.32%71017.80% Violent Illegal Acts3,4397.92%3187.97%3578.95% Sexual Content1,6263.74%1864.66%1493.74% Personal Information7771.79%1463.66%922.31% Politically4341.00%350.88%210.53% Suicide & Self-Harm2170.50%150.38%120.30% Copyright Violation190.04%30.08%10.03% Safe12,47728.74%69817.50%30.08% Total43,418 100.00%3,989 100.00%3,989 100.00% Furthermore, Table IV reveals that while the concretized prompts maintain a length comparable to the original ques- tions, the resulting reasoning and response are substantially 8 longer. For instance, the average length of reasoning traces increases from approximately 3,400 to over 4,400 charac- ters. This observation supports our hypothesis that increased specificity triggers deeper reasoning, thereby inducing safety vulnerabilities in LRMs. TABLE IV AVERAGE CONTENT LENGTH ANALYSIS. AVG. CHARS MEANS AVERAGE CHARACTERS OF CONTENTS, AVG. WORDS MEANS AVERAGE WORDS OF CONTENTS. BEST RESULTS ARE HIGHLIGHTED IN BOLD PromptReasoningResponse MetricOriginal Concretized Original Concretized Original Concretized Avg. Chars124.1127.33,396.84,430.81,437.91,824.6 Avg. Words21.618.7558.8714.8200.4250.6 To reflect the changes brought about by concretization, we further examine vocabulary shifts with differential word clouds (Fig. 4). Across prompt, reasoning, and response, concretization introduces more scenario-specific and action- oriented terms, with the largest shift observed in the reasoning traces. Words that increase (red) and decrease (green) suggest that concretization primarily enriches contextual detail rather than merely substituting synonyms. Fig. 4.Differential word clouds comparing vocabulary changes between original and concretized texts across three components: (a) Prompt, (b) Rea- soning, and (c) Response. Red tones indicate words that significantly increased in frequency after concretization, while green tones indicate words that significantly decreased. Word size reflects frequency and change magnitude. C. Principle-Guided Alignment Enhances LRM Safety To verify the effectiveness of the RAJ-PGA framework detailed in Section I-B, we construct an Principle-Guided Alignment dataset (PGA dataset, N = 3,989 samples) with our framework and perform a controlled comparison against several baseline alignment datasets. We fine-tune four rep- resentative LRMs using PEFT with LoRA adapters, and keep the fine-tuning hyperparameters and protocol consistent across all datasets to ensure a fair evaluation, following UnsafeChain [29]. Table V summarizes the safety performance (DSR) of models fine-tuned on PGA versus those fine-tuned on different baseline datasets over five jailbreak benchmarks. In the table, green cells indicate a safety increase and red cells indicate a decrease relative to the base model. Overall, PGA yields predominantly green improvements across benchmarks and backbones, suggesting that principle-guided reasoning traces provide a higher-density and more effective supervision signal for safety alignment than existing alternatives. As measured by average DSR, PGA achieves the best safety performance on three of the four backbones and ranks TABLE V SAFETY PERFORMANCE OF FINE-TUNED LRMS (DSR, IN %). GREEN INDICATES A SAFETY INCREASE, WHILE RED SIGNIFIES A DECREASE RELATIVE TO THE BASE MODEL. BEST RESULTS ARE HIGHLIGHTED IN BOLD AND THE SECOND HIGHEST RESULTS ARE UNDERLINED . ModelWildChat JBB StrongReject WildJailbreak AdvBench Avg. (↑) DeepSeek-R1-0528-Qwen3-8B Subset R1-Qwen3-8B94.0074.0072.8081.2073.6579.13 -STAR-196.0067.6761.2078.4067.8874.23 -DirectRefusal96.0064.3379.2087.6083.4682.12 -UnsafeChain-Random93.0064.0054.8071.6062.1269.10 -UnsafeChain-Selected94.0064.3355.6062.0065.0068.19 -UnsafeChain-Full93.5061.3367.6088.4095.0081.17 -SafeChain95.0059.0055.2070.8073.6570.73 -STAIR 97.5070.0073.6098.0065.5880.94 -PGA-100095.5076.0090.0090.9690.9688.68 -PGA-200093.5077.0092.0095.6097.3191.08 -PGA-300095.5081.6791.2095.2099.6292.64 -PGA97.0083.0092.4096.0099.6293.60 DeepSeek-R1-Distill-Llama-8B Subset R1-Llama-8B98.0053.6758.4067.6046.7364.88 -STAR-197.0059.0070.8066.0065.1971.60 -DirectRefusal96.0061.0071.2078.8059.8173.36 -UnsafeChain-Random97.0057.6768.0066.8063.0870.51 -UnsafeChain-Selected 97.0061.6778.0070.0074.4276.22 -UnsafeChain-Full96.5060.3378.4076.8088.2780.06 -SafeChain 97.0059.6770.8070.8058.4671.35 -STAIR 96.5072.3398.8099.2063.8586.14 -PGA-100096.5067.0087.2084.4074.8181.98 -PGA-200098.5076.3396.0091.2085.7789.56 -PGA-300098.0081.0098.0096.0091.9292.98 -PGA97.5082.6798.8096.0096.9294.38 DeepSeek-R1-Distill-Qwen-7B Subset R1-Qwen-7B 98.0056.3349.6060.8037.1260.37 -STAR-1 97.5054.3350.8063.2040.7761.32 -DirectRefusal97.5059.3355.6071.6033.2763.46 -UnsafeChain-Random97.5052.0051.6062.8038.2760.43 -UnsafeChain-Selected96.5061.3356.0060.4034.6261.77 -UnsafeChain-Full96.5057.0075.6076.0073.8575.79 -SafeChain97.0054.0060.4063.6045.1964.04 -STAIR97.0068.3376.0096.0074.0482.27 -PGA-1000 99.0053.0053.2074.0038.8563.61 -PGA-200098.0062.0068.4077.6051.9271.58 -PGA-300096.5068.0074.4081.2058.8575.79 -PGA98.0071.6787.2088.8062.8881.71 Qwen3-8B Subset Qwen3-8B95.5059.3389.2060.8089.0478.77 -STAR-195.0059.6780.0057.2083.4675.07 -DirectRefusal95.5057.3361.2056.8070.1968.20 -UnsafeChain-Random95.0053.0075.6056.8082.6972.62 -UnsafeChain-Selected95.5052.3374.8052.8085.0072.09 -UnsafeChain-Full 94.5058.0089.6088.4097.1285.52 -SafeChain 95.5062.0071.2075.6067.6974.40 -STAIR96.5073.6776.0082.4060.7777.87 -PGA-100094.0063.6776.0064.0072.6974.07 -PGA-200096.0069.3381.6070.8087.8881.12 -PGA-300095.5067.0090.4079.2091.1584.65 -PGA96.0072.3392.0083.2085.5885.82 second on R1-Qwen-7B, despite using fewer training samples than STAIR. Notably, PGA outperforms larger datasets. This highlights the high density of effective safety reasoning within PGA compared to larger-scale alternatives. The efficacy of PGA is evident in its stability across different model architectures. On the base Qwen3-8B model, where the STAIR dataset leads to a slight degradation in safety performance (-0.91%), PGA yields a substantial improvement of +7.05%, increasing the average score to 85.82%. Similarly, on R1-Qwen3-8B, PGA outperforms STAIR by a significant margin (93.60% vs. 80.94%), demonstrating robust general- ization capabilities that are less susceptible to the negative transfer effects observed with other datasets. When controlling for data size, our method demonstrates superior sample efficiency. The PGA-1000 subset generally outperforms other 1,000-sample baselines such as STAR-1, DirectRefusal, and UnsafeChain-Random/Selected. For exam- ple, on R1-Llama-8B, PGA-1000 achieves an average safety performance of 81.98%, significantly higher than DirectRe- 9 fusal (73.36%) and STAR-1 (71.60%). This suggests that the principle-guided aligned reasoning trails in PGA provide a stronger training signal for safety alignment than simple refusal or alternative CoT strategies. In terms of specific attack vectors, PGA consistently mitigates harmful responses on challenging benchmarks like StrongReject and AdvBench. On R1-Llama-8B, the full PGA dataset improves the StrongReject score by +50.19% (reaching 96.92%), whereas the closest competitor, UnsafeChain-Full, improves it by +41.54%. Addi- tionally, the consistent scaling trend observed from PGA-1000 to PGA-3000 indicates that the model’s safety capabilities effectively scale with the addition of our diverse aligned data. D. Principle-Guided Alignment Mitigates Alignment Tax To ensure that PGA fine-tuning does not compromise LRM performance, we evaluate the fine-tuned models across five diverse general-performance benchmarks. As shown in Ta- ble VI, where green denotes a general performance (Accuracy) increase and red denotes a decrease relative to the base model, PGA generally mitigates the alignment tax by maintaining or improving general capabilities while enhancing safety. As indicated by the average general performance, PGA achieves the highest or near-highest average scores on most evaluated models. Notably, on the DeepSeek-R1-0528-Qwen3- 8B Subset, PGA attains an average score of 71.17%, signifi- cantly outperforming the base model (63.06%) and competi- tive baselines like UnsafeChain-Full (66.87%). The efficacy of PGA is evident in its stability across different model architectures. On the base Qwen3-8B model, where the STAIR dataset leads to a significant degradation in general performance (−8.20%), PGA maintains performance parity with a negligible variation of −0.11%. Similarly, on R1-Qwen3-8B, PGA outperforms STAIR by a substantial margin (71.17% vs. 60.65%), demonstrating robust generaliza- tion capabilities that avoid the severe performance regression observed with other alignment methods. In terms of specific benchmarks, PGA preserves or im- proves performance on TruthfulQA and MATH500. On R1- Qwen3-8B, the full PGA dataset improves the TruthfulQA score by +29.82%, whereas the closest competitor, SafeChain, improves it by +22.95%. Although we observe minor trade- offs, such as a regression in HumanEval scores on R1-Llama- 8B (ranging from−3% to−11%), these dips are marginal and controlled compared to the catastrophic failures of baselines, such as STAIR’s −35.86% drop on GPQA. When controlling for data size, our method demonstrates superior sample efficiency. The PGA-1000 subset generally outperforms other 1,000-sample baselines such as STAR-1 and DirectRefusal. For example, on R1-Llama-8B, PGA-1000 achieves an average general score of 66.37%, which is not only higher than DirectRefusal (63.46%) but also surpasses the base model itself (65.05%). These results suggest that PGA can effectively instill safety constraints without compromising the underlying reasoning capabilities. TABLE VI GENERAL PERFORMANCE OF FINE-TUNED LRMS (ACCURACY, IN %). GREEN INDICATES A PERFORMANCE INCREASE, WHILE RED SIGNIFIES A DECREASE RELATIVE TO THE BASE MODEL. BEST RESULTS ARE HIGHLIGHTED IN BOLD AND THE SECOND HIGHEST RESULTS ARE UNDERLINED . ModelHumanEval MATH500 ARC TruthfulQA GPQA Avg. (↑) DeepSeek-R1-0528-Qwen3-8B Subset R1-Qwen3-8B48.7871.0078.2644.0173.2363.06 -STAR-150.6172.0083.6143.1360.6161.99 -DirectRefusal46.9570.5080.2748.6871.7263.62 -UnsafeChain-Random48.7866.0080.2723.8365.1556.81 -UnsafeChain-Selected49.3968.5083.2824.5672.7359.69 -UnsafeChain-Full65.2468.0084.2851.1765.6666.87 -SafeChain59.7666.5086.6266.9673.2370.61 -STAIR60.9870.0091.3030.9950.0060.65 -PGA-100054.8872.5081.9459.9464.6566.78 -PGA-200056.7166.0085.6270.1872.2270.15 -PGA-300054.8867.5088.6373.3964.1469.71 -PGA55.4965.5088.2973.8372.7371.17 DeepSeek-R1-Distill-Llama-8B Subset R1-Llama-8b41.4672.5079.9355.1276.2665.05 -STAR-140.8572.0081.2757.3171.7264.63 -DirectRefusal35.9869.5082.6155.9973.2363.46 -UnsafeChain-random35.9869.5082.6159.3676.7764.84 -UnsafeChain-selected43.2966.5082.2757.3173.7464.62 -UnsafeChain-full54.2761.5081.9453.9557.0761.75 -SafeChain45.1261.5084.2857.3168.1863.28 -STAIR45.7345.5068.5655.9940.4051.24 -PGA-100038.4171.5081.9457.1682.8366.37 -PGA-200033.5468.5081.9460.9677.2764.44 -PGA-300035.3772.5079.9363.4576.2665.50 -PGA30.4969.5082.9463.4575.2564.33 DeepSeek-R1-Distill-Qwen-7B Subset R1-Qwen-7B38.4175.0076.2543.5748.9956.44 -STAR-139.6377.5076.9244.4447.9857.29 -DirectRefusal 45.7371.0074.5844.5949.4957.08 -UnsafeChain-Random46.3472.0076.2545.0345.9657.12 -UnsafeChain-Selected46.3475.0076.2547.0844.4457.82 -UnsafeChain-Full 46.9573.5073.9142.1142.4255.78 -SafeChain 51.2271.0079.6048.9867.6863.70 -STAIR 57.9365.0064.2148.3953.0357.71 -PGA-100047.5675.5076.5946.4947.9858.82 -PGA-200050.0071.0076.2549.1245.2758.33 -PGA-3000 42.0770.5075.2549.5650.0057.48 -PGA34.1568.0075.5946.9343.9453.72 Qwen3-8B Subset Qwen3-8B64.0279.5089.9770.4754.0471.60 -STAR-153.6683.5091.6466.6759.0970.91 -DirectRefusal58.5476.5088.2949.4250.5164.65 -UnsafeChain-Random59.7681.5091.3065.3556.0670.79 -UnsafeChain-Selected59.1577.5091.3065.7957.0770.16 -UnsafeChain-Full60.3770.0088.9667.2546.9766.71 -SafeChain40.8575.0091.9757.0250.5163.07 -STAIR58.5473.5087.2957.7539.9063.40 -PGA-100057.9378.5092.3167.9858.5971.06 -PGA-200054.8882.0091.9768.5762.2271.93 -PGA-300057.3278.5092.3168.7163.6472.10 -PGA52.4481.5091.9767.4064.1471.49 E. Comprehensive Assessment and F1 Score Analysis Based on the safety and general performance, we next summarize the trade-off between the LRM’s safety and general capabilities using (i) the average of safety DSR and general accuracy, and (i) an F1-score that jointly captures both objectives. Table VII and Fig. 5 present a comprehensive comparison of our proposed method against the base models and various baseline approaches. Overall, our method consistently outperforms both the base models and the baseline methods in terms of Average (Avg.) and F1 scores. On the DeepSeek-R1-0528-Qwen3-8B Subset, our approach achieves an Avg. score of 82.38% and an F1 score of 80.86%, significantly surpassing the Base Model by margins of 11.29% and 10.68%, respectively. Furthermore, compared to strong baselines such as UnsafeChain-Full and 10 TABLE VII HOLISTIC PERFORMANCE EVALUATION VIA AVERAGE AND F1 SCORES. AVG. (%) REPRESENTS THE AVERAGE OF SAFETY PERFORMANCE AND GENERAL PERFORMANCE, AND F1 (%) DENOTES THE F1 SCORE. BEST RESULTS ARE HIGHLIGHTED IN BOLD AND THE SECOND HIGHEST RESULTS ARE UNDERLINED. R1-Qwen3-8BR1-Llama-8BR1-Qwen-7BQwen3-8B MethodAvg.F1Avg.F1Avg.F1Avg.F1 Base Model71.0970.1864.9664.9758.4058.3475.1975.02 STAR-168.1167.5668.1267.9459.3059.2472.9972.93 DirectRefusal72.8771.7068.4168.0560.2760.1066.4366.38 UnsafeChain-Random62.9662.3567.6767.5658.7758.7371.7071.69 UnsafeChain-Selected63.9463.6670.4269.9459.8059.7371.1271.11 UnsafeChain-Full74.0273.3370.9169.7265.7964.2676.1174.95 Safechain70.6770.6767.3167.0763.8763.8768.7368.27 STAIR70.8069.3468.6964.2569.9967.8470.6369.89 PGA-100077.7376.1974.1773.3561.2161.1272.5672.54 PGA-200080.6279.2577.0074.9564.9564.2876.5376.25 PGA-300081.1779.5579.2576.8666.6465.3778.3877.87 PGA82.3880.8679.3576.5167.7264.8278.6678.00 565860626466687072 Avg. General Performance (%) 70 75 80 85 90 95 Avg. Safety Performance (%) R1-Qwen3-8B 50.052.555.057.560.062.565.067.5 Avg. General Performance (%) 65 70 75 80 85 90 95 Avg. Safety Performance (%) R1-Llama-8B 545658606264 Avg. General Performance (%) 60 65 70 75 80 Avg. Safety Performance (%) R1-Qwen-7B 6466687072 Avg. General Performance (%) 67.5 70.0 72.5 75.0 77.5 80.0 82.5 85.0 87.5 Avg. Safety Performance (%) Qwen3-8B Base STAR-1 DirectRefusal UnsafeChain-Random UnsafeChain-Selected UnsafeChain-Full SafeChain STAIR PGA-1000 PGA-2000 PGA-3000 PGA Fig. 5. Trade-off Analysis between Model Safety (DSR, in %) and General Reasoning Capabilities. The PGA framework (red markers) consistently achieves a superior Pareto frontier compared to baselines across multiple model backbones. STAIR, our method demonstrates superior efficacy in balanc- ing safety and general performance. Similar trends are observed across the R1-Llama-8B and Qwen3-8B models, where our method obtains the highest scores across all metrics. For instance, on R1-Llama-8B, we achieve an F1 score of 76.51%, outperforming the strongest baseline (UnsafeChain-Selected) by approximately 6.57%. While STAIR shows competitive performance on the R1- Qwen-7B model, our method maintains robust performance and achieves state-of-the-art results on the majority of the evaluated architectures. Meanwhile, there is a clear positive correlation between the dataset size and the model’s performance. As the data scale increases from 1000 to 3000 samples (PGA-1000, PGA-2000, and PGA-3000), we observe improvements across all model backbones. As illustrated in Table VII, our method achieves state-of-the- art performance across multiple model families. Specifically, we outperform the Base Model and existing methods like SafeChain and DirectRefusal by a significant margin. On average, our approach improves the F1 score by over 5− 10% compared to the base models, validating the effectiveness of our proposed alignment strategy. F. Defense Enhancement for Closed-Source Models To further evaluate the generalizability and practical utility of our proposed PGA strategies, we investigate their effective- ness as a reasoning-time defense mechanism for closed-source models. We report DSR (%) for the generated reasoning and response output as their safety performance. We embedded our five harmlessness principles directly into the system prompt of four leading commercial LRMs: DeepSeek-V3.2, GLM-4.6, Qwen3-Plus, and Gemini-2.5-Pro. We randomly sampled 200 instances from our RAJ dataset to evaluate these models under two conditions: Standard (default system prompt) and + Guide (with PGA strategies). TABLE VIII DEFENSE EFFECTIVENESS ON CLOSED-SOURCE MODELS. WE REPORT THE DSR (%) OF THE CONTEXT. + Guide DENOTES THE INTEGRATION OF OUR HARMLESSNESS STRATEGIES INTO THE SYSTEM PROMPT. OriginalConcretized ReasoningResponseReasoningResponse DeepSeek-V3.2-Exp89.0096.0081.5097.00 + Guide96.0099.5092.0098.50 GLM4.693.5093.5084.5085.50 + Guide96.0097.0095.5094.50 Qwen3-plus92.5094.5088.5093.00 + Guide96.0097.5095.5097.50 Gemini-2.5-Pro97.0098.5085.5087.50 + Guide100.0099.5098.0099.00 As shown in Table VIII, the integration of our harmlessness principles yields consistent safety improvements across all four commercial closed-source models. Notably, while the baseline models exhibit a discernible decline in DSR when facing Concretized prompts, where reasoning safety often drops more sharply than final response safety, the addition of our + Guide strategy effectively buffers this degradation. For instance, Gemini-2.5-Pro’s reasoning DSR increases from 85.50% to 98.00% under the Concretized setting, nearly eliminating the vulnerability introduced by reasoning-activated jailbreaks. Furthermore, the performance gains are observed not only in the final output but also within the internal reasoning chains, suggesting that our principles successfully instill a self- correction mechanism during the inference process. These results confirm that our constitutional principles serve as a lightweight yet powerful defense layer, effectively guiding black-box models to neutralize reasoning-based attacks with- out requiring parameter updates. G. Ablation Study To rigorously verify the contribution of RAJ-PGA frame- work, we conducted an ablation study comparing PGA against two strategic variants (Original-Refusal and RAJ). The variants are defined as shown in Fig. 6 and discussed as follows: 11 TABLE IX COMBINED ABLATION STUDY: SAFETY, GENERAL PERFORMANCE, SUM, AND F1 SCORE (IN %). WE COMPARE THE MODELS FINE-TUNED ON THE ORIGINAL-REFUSAL, THE RAJ, AND PGA DATASET. BOLD INDICATES THE BEST RESULT WITHIN EACH MODEL SUBSET. Safety PerformanceGeneral PerformanceOverall ModelWildChatJBBStrongRejectWildJailbreakAdvBenchAvg.HumanEvalMath500ARCTruthfulQAGPQAAvg.Avg.F1 DeepSeek-R1-0528-Qwen3-8B Subset R1-Qwen3-8B94.0074.0072.8081.2073.6579.1348.7871.0078.2644.0173.2363.0671.0970.18 -PGA97.0083.0092.4096.0099.6293.6055.4965.5088.2973.8372.7371.1782.3880.86 -Original-Refusal93.5063.6780.0084.4073.6579.0454.8869.5088.9664.6271.2169.8374.4474.15 -RAJ94.0057.3374.8081.2062.5073.9754.2767.5085.2864.9167.6867.9370.9570.82 DeepSeek-R1-Distill-Llama-8B Subset R1-Llama-8B98.0053.6758.4067.6046.7364.8841.4672.5079.9355.1276.2665.0564.9664.97 -PGA97.5082.6798.8096.0096.9294.3830.4969.5082.9463.4575.2564.3379.3576.51 -Original-Refusal 97.0062.6783.6083.2057.1276.7235.9862.5082.2755.2671.7261.5569.1468.30 -RAJ 97.0054.3364.4070.0037.1264.5742.6870.0079.6052.1975.7664.0564.3164.31 DeepSeek-R1-Distill-Qwen-7B Subset R1-Qwen-7B98.0056.3349.6060.8037.1260.3738.4175.0076.2543.5748.9956.4458.4058.34 -PGA98.0071.6787.2088.8062.8881.7134.1568.0075.5946.9343.9453.7267.7264.82 -Original-Refusal96.5063.0070.0073.2045.5869.6640.8572.5073.5846.7848.4856.4463.0562.35 -RAJ96.5054.3348.8064.0024.2357.5740.2469.5068.9044.3050.0054.5956.0856.04 Qwen3-8B Subset Qwen3-8B95.5059.3389.2060.8089.0478.7764.0279.5089.9770.4754.0471.6075.1975.02 -PGA96.0072.3392.0083.2085.5885.8252.4481.5091.9767.4064.1471.4978.6678.00 -Original-Refusal94.5057.0065.2064.8061.7368.6559.7675.5090.6460.6755.0568.3268.4968.48 -RAJ94.5055.6753.6062.8069.2367.1659.1574.5089.6359.9448.9966.4466.8066.80 Original-Refusal: This variant comprises the original ma- licious prompts and their corresponding safe responses gener- ated by the victim model before concretization. This baseline represents a standard refusal-based alignment strategy, serving to determine whether RAJ-PGA offers superior trade-offs compared to simply learning from direct refusals. RAJ: This variant utilizes the raw RAJ dataset without alignment. Evaluating RAJ quantifies the inherent risk of the elicited capabilities and validates the necessity of the PGA. Original Refusal Dataset (N = 3,989) (Safe Reasoning + Safe Response) Concretize RAJ Dataset (N = 3,989) (Unsafe Reasoning + Unsafe Response) PGA Dataset (N = 3,989) (Safe Reasoning + Safe Response) Fine-tuning (LoRA) (Target: Open Source LRMs) Result: Safe LRM (Safety performance is better than base but lower than ours) Result: Unsafe LRM (Safety performance is the worst) Result: Safe LRM (Safety performance is SOTA and General capability are not compromised) Detoxify Fig. 6. Ablation setup for RAJ-PGA framework. Models are fine-tuned on three datasets (Original-Refusal, RAJ, and PGA) and evaluated on safety and general benchmarks. We fine-tuned the four base LRMs using these datasets and evaluated them across both safety and general benchmarks. The results are summarized in Table IX. Comparing PGA with Original-Refusal reveals the advan- tage of our alignment strategy. While Original-Refusal pro- vides a decent safety baseline (e.g., 76.72% on R1-Llama- 8B), PGA consistently achieves higher safety performance ( 94.38% ) and F1 scores( 76.51% ). This suggests that training on aligned reasoning traces, which explain risks and provide educational guidance, leads to more robust safety alignment than merely mimicking refusal patterns. Comparing PGA with RAJ, we observe an improvement in safety performance. For the DeepSeek-R1-Distill-Llama-8B Subset, the average safety performance increases from 64.57% (RAJ) to 94.38%, with practically no loss in general capability (64.05% vs. 64.33%). This confirms that our alignment strat- egy successfully removes the harmful knowledge elicited by the attack. VI. CONCLUSION In this work, we addressed the safety risks inherent in the reasoning process of LRMs by proposing and validating a novel method for constructing alignment datasets. Our method first effectively captures LRMs’ deep reasoning vulnerabilities through a Reasoning-Activated Jailbreak attack, then reshapes the harmful outputs into safe and instructive content using a Principle-Guided Alignment framework. The core contribu- tion of our work is a systematic and scalable paradigm for AI safety, moving beyond reactive defenses against specific attacks to a more proactive alignment of the model’s internal processes. The effectiveness of our method is demonstrated by the PGA dataset, which, when used for fine-tuning, signif- icantly enhances model robustness while preserving reasoning capabilities. The dataset construction methodology we have introduced is highly generalizable. Future work could involve applying this method to bolster model resilience against evolving adversarial reasoning attacks and multi-step prompt injections. Further- more, the framework can be adapted for more advanced future models, providing a durable path toward building inherently safer and more reliable AI systems. REFERENCES [1] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, p. 100211, 2024. [2] F. Xu, Q. Hao, Z. Zong, J. Wang et al., “Toward large reasoning models: A survey of reinforced reasoning with large language models,” Patterns, vol. 6, no. 10, p. 101370, Oct. 2025. [3] Z. Deng, W. Ma, Q.-L. Han, W. Zhou, X. Zhu, S. Wen, and Y. Xiang, “Exploring deepseek: A survey on advances, applications, challenges and future directions,” IEEE/CAA Journal of Automatica Sinica, vol. 12, no. 5, p. 872–893, 2025. [4] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky et al., “Openai o1 system card,” arXiv preprint arXiv:2412.16720, 2024. [5] C. Zhang, Y. Deng, X. Lin, B. Wang, D. Ng et al., “100 days after deepseek-r1: A survey on replication studies and more directions for reasoning language models,” arXiv preprint arXiv:2505.00551, 2025. 12 [6] Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang et al., “Scienceagent- bench: Toward rigorous assessment of language agents for data-driven scientific discovery,” The Thirteenth International Conference on Learn- ing Representations, 2025. [7] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” in The Twelfth International Conference on Learning Representations, 2023. [8] D. Zhang, Z.-Z. Li, M.-L. Zhang, J. Zhang, Z. Liu et al., “ From System 1 to System 2: A Survey of Reasoning Large Language Models ,” IEEE Transactions on Pattern Analysis & Machine Intelligence, no. 01, p. 1–20, Nov. 5555. [9] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, p. 24 824–24 837, 2022. [10] X. Qu, Y. Li, Z. Su, W. Sun, J. Yan, D. Liu, G. Cui, D. Liu, S. Liang, J. He et al., “A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond,” arXiv preprint arXiv:2503.21614, 2025. [11] A. Arrieta, M. Ugarte, P. Valle, J. A. Parejo, and S. Segura, “o3-mini vs deepseek-r1: Which one is safer?” arXiv preprint arXiv:2501.18438, 2025. [12] K. Zhou, C. Liu, X. Zhao, S. Jangam, J. Srinivasa et al., “The hidden risks of large reasoning models: A safety assessment of r1,” ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025. [13] F. Jiang, Z. Xu, Y. Li, L. Niu, Z. Xiang et al., “SafeChain: Safety of language models with long chain-of-thought reasoning capabilities,” in Findings of the Association for Computational Linguistics: ACL 2025. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, p. 23 303–23 320. [14] Z. Deng, R. Sun, M. Xue, W. Ma, S. Wen et al., “Hardening llm fine- tuning: From differentially private data selection to trustworthy model quantization,” IEEE Transactions on Information Forensics and Security, vol. 20, p. 7211–7226, 2025. [15] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023. [16] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, p. 1671–1685. [17] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,” Advances in Neural Information Processing Systems, vol. 37, p. 61 065–61 105, 2024. [18] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” Advances in Neural Information Processing Sys- tems, vol. 36, p. 80 079–80 110, 2023. [19] E. de Bono, The Use of Lateral Thinking. Avon Books, 1971. [20] P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tram ` er, H. Hassani, and E. Wong, “Jailbreakbench: An open robustness bench- mark for jailbreaking large language models,” in NeurIPS Datasets and Benchmarks Track, 2024. [21] T. Huang, S. Hu, F. Ilhan, S. F. Tekin, Z. Yahn et al., “Safety tax: Safety alignment makes your large reasoning models less reasonable,” arXiv preprint arXiv:2503.00555, 2025. [22] Z. Wang, H. Tu, Y. Wang, J. Wu, J. Mei, B. R. Bartoldson, B. Kailkhura, and C. Xie, “Star-1: Safer alignment of reasoning llms with 1k data,” The Association for the Advancement of Artificial Intelligence, 2026. [23] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion et al., “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073, 2022. [24] V.-A. Nguyen, S. Zhao, G. Dao, R. Hu, Y. Xie, and L. A. Tuan, “Three minds, one legend: Jailbreak large reasoning model with adaptive stacked ciphers,” arXiv preprint arXiv:2505.16241, 2025. [25] Z. Dong, Z. Zhou, C. Yang, J. Shao, and Y. Qiao, “Attacks, defenses and evaluations for LLM conversation safety: A survey,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, Jun. 2024, p. 6734–6747. [26] R. Uuk, C. I. Gutierrez, D. Guppy, L. Lauwaert, A. Kasirzadeh, L. Velasco, P. Slattery, and C. Prunkl, “A taxonomy of systemic risks from general-purpose ai,” arXiv preprint arXiv:2412.07780, 2024. [27] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, p. 27 730–27 744, 2022. [28] Y. Cui and C. Zuo, “Practical reasoning interruption attacks on reasoning large language models,” arXiv preprint arXiv:2505.06643, 2025. [29] R. V. Tomar, P. Nakov, and Y. Wang, “Unsafechain: Enhancing reasoning model safety via hard cases,” arXiv preprint arXiv:2507.21652, 2025. [30] Y. Zhang, S. Zhang, Y. Huang, Z. Xia, Z. Fang et al., “STAIR: Improving safety alignment with introspective reasoning,” in Forty- second International Conference on Machine Learning, 2025. [31] J. Ji, D. Hong, B. Zhang, B. Chen, J. Dai et al., “PKU-SafeRLHF: Towards multi-level safety alignment for LLMs with human preference,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, Jul. 2025, p. 31 983–32 016. [32] H. Zhao, C. Yuan, F. Huang, X. Hu, Y. Zhang, A. Yang, B. Yu, D. Liu, J. Zhou, J. Lin et al., “Qwen3guard technical report,” arXiv preprint arXiv:2510.14276, 2025. [33] Y. Chen, H. Gao, G. Cui, F. Qi, L. Huang et al., “Why should adver- sarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, p. 11 222– 11 237. [34] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, “Wildchat: 1m chatGPT interaction logs in the wild,” in The Twelfth International Conference on Learning Representations, 2024. [35] L. Jiang, K. Rao, S. Han, A. Ettinger, F. Brahman et al., “Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models,” The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [36] A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh et al., “A strongreject for empty jailbreaks,” Advances in Neural Information Processing Systems, vol. 37, p. 125 416–125 440, 2024. [37] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [38] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker et al., “Let’s verify step by step,” in The Twelfth International Conference on Learning Representations, 2024. [39] S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, vol. 1, p. 3214 – 3252, 2022. [40] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal et al., “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv:1803.05457v1, 2018. [41] D. Rein, B. L. Hou, A. C. Stickland, J. Petty et al., “GPQA: A graduate- level google-proof q&a benchmark,” in First Conference on Language Modeling, 2024. [42] D. Guo, D. Yang, H. Zhang et al., “Deepseek-r1 incentivizes reasoning in llms through reinforcement learning,” Nature, vol. 645, p. 633–638, 2025. [43] DeepSeek-AI, “Deepseek-v3.2: Pushing the frontier of open large lan- guage models,” arXiv preprint arXiv:2512.02556, 2025. [44] Zhipu AI, “GLM-4.6: Advanced Agentic, Reasoning and Coding Capa- bilities,” https://z.ai/blog/glm-4.6, sep 2025, accessed: 2025-11-10. [45] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, and et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [46] G. D. G. Group, “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” arXiv preprint arXiv:2507.06261, 2025.