Paper deep dive
CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion
Zhen Liang, Hai Huang, Wentao Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models have achieved remarkable capabilities across diverse domains, yet their safety alignment remains vulnerable to jailbreak attacks. In this work, we identify a previously underexplored failure mode - safety generalization lag - where alignment trained predominantly on natural language fails to transfer to the code domain. We show that this lag induces a code-completion blind spot, allowing malicious intent embedded within syntactically valid code to evade safety mechanisms. To exploit this vulnerability, we propose CodeMimicry, a fully automated black-box jailbreak framework that generates structured, object-oriented code prompts to induce harmful outputs via code completion. Experiments on 8 state-of-the-art commercial LLMs demonstrate that CodeMimicry achieves a 96.25% attack success rate with 1.51 queries on average, significantly outperforming both template-based and optimization-based baselines. Beyond empirical performance, we provide a mechanistic analysis of code-based jailbreaks through latent space representations, including projection onto refusal-related directions and activation steering. This analysis offers an explanation of how CodeMimicry bypasses safety mechanisms in code-related domains. Our findings reveal a weakness in current safety alignment and highlight the need for robust alignments in structured domains such as code.
Tags
Links
- Source: https://arxiv.org/abs/2609.39902v1
- Canonical: https://arxiv.org/abs/2609.39902v1
Trouble viewing inline? Open PDF directly →
Full Text
69,215 characters extracted from source content.
Expand or collapse full text
CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion Zhen Liang Affiliation: School of Computer Science and Technology, Zhejiang Sci-Tech University Affiliation: Zhejiang Key Laboratory of Digital Fashion and Data Governance, Zhejiang Sci-Tech UniversityHangzhou 310018, China Email: liangzhen741@gmail.com Hai Huang †thanks: Corresponding author. Email: haihuang1005@gmail.com Affiliation: School of Computer Science and Technology, Zhejiang Sci-Tech University Affiliation: Zhejiang Key Laboratory of Digital Fashion and Data Governance, Zhejiang Sci-Tech UniversityHangzhou 310018, China Email: haihuang1005@gmail.com Wentao Chen Affiliation: China Academy of Information and Communications Technology Email: chenwentao@caict.ac.cn Abstract Large language models have achieved remarkable capabilities across diverse domains, yet their safety alignment remains vulnerable to jailbreak attacks. In this work, we identify a previously underexplored failure mode—safety generalization lag—where alignment trained predominantly on natural language fails to transfer to the code domain. We show that this lag induces a code-completion blind spot, allowing malicious intent embedded within syntactically valid code to evade safety mechanisms. To exploit this vulnerability, we propose CodeMimicry, a fully automated black-box jailbreak framework that generates structured, object-oriented code prompts to induce harmful outputs via code completion. Experiments on 8 state-of-the-art commercial LLMs demonstrate that CodeMimicry achieves a 96.25% attack success rate with 1.51 queries on average, significantly outperforming both template-based and optimization-based baselines. Beyond empirical performance, we provide a mechanistic analysis of code-based jailbreaks through latent space representations, including projection onto refusal-related directions and activation steering. This analysis offers an explanation of how CodeMimicry bypasses safety mechanisms in code-related domains. Our findings reveal a weakness in current safety alignment and highlight the need for robust alignments in structured domains such as code. Our code is on https://github.com/lzzzr123/CodeMimicry. 1 Introduction Large language models (LLMs) have demonstrated unprecedented capabilities in Natural Language Processing [1, 2], information retrieval [3], and code tasks [4, 5, 6], driving their integration into critical applications. Therefore, ensuring these models adhere to security guidelines has become crucial. Techniques such as reinforcement learning based on human feedback (RLHF) [7] and supervised fine-tuning [8] are widely used to align models with human values, thus building guardrails against malicious abuse. However, LLMs remain vulnerable to sophisticated jailbreak attacks, in which carefully crafted malicious prompts can induce the model to produce harmful or unintended outputs. Jailbreak attacks against LLMs are generally divided into two categories: (1) White-box attacks, such as GCG [9], which use gradient-based optimization to craft adversarial suffixes. However, they require access to model parameters and gradients, making them impractical for closed-source LLMs. (2) Black-box attacks, including template-based and automated methods. Template-based approaches such as CodeChameleon [10] and FlipAttack [11] rely on manually designed prompts to bypass safety mechanisms, but are often mitigated by simple rule-based or pattern-based defenses due to their fixed structures. Automated methods such as PAIR [12], GPTfuzzer [13], and AutoDan-Turbo [14] generate jailbreak prompts via model feedback, removing the need for manual design. However, they typically lack principled, domain-specific strategies (e.g., program structure or code completion), relying instead on heuristic exploration in natural language space, leading to limited effectiveness and efficiency. A critical vulnerability arises from the rapid advancement of large language models: mismatched generalization [15], wherein models’ broad pretraining capabilities substantially outpace the narrower scope of their safety alignment. While modern general LLMs have achieved near-expert proficiency in code generation and mathematical reasoning [16, 17], their safety alignment is still predominantly trained on natural language data. Motivated by this observation, we hypothesize the existence of a Safety Generalization Lag in the code domain: as LLMs’ capabilities in the code domain continue to advance, their safety alignment fails to generalize effectively to this rigid, syntactic subspace. This lag creates a code-completion blind spot, where safety-aligned representations fail to generalize to the structured code space. As refusal behaviors are primarily learned from natural language, they transfer poorly to code generation, allowing malicious intent embedded in syntactically valid code to bypass safeguards. In this paper, to address these limitations and support our hypothesis, we propose CodeMimicry, an efficient and fully automated jailbreak framework that eliminates manual strategy design by automatically generating jailbreak prompts in which malicious intent is embedded within structured code. Extensive experiments on 8 black-box commercial LLMs demonstrate the superiority of CodeMimicry. Notably, it achieves a 90% or higher attack success rate on each model with a very low average of queries. To better understand the behavioral differences between successful and failed code-based jailbreaks, we further conduct an in-depth internal analysis of model representations and refusal behaviors. These results suggest deficiencies in code-domain safety alignment and indicate that robust code-domain safety guardrails have not been sufficiently established within these LLMs. Our contributions are three-fold: • Code-domain Jailbreaking Utilizing Safety Generalization Lag: We identify a previously underexplored vulnerability in LLMs—safety generalization lag—where alignment does not reliably transfer to the code domain. Leveraging this insight, we propose CodeMimicry, an automated jailbreak framework that embeds malicious intent into structured object-oriented code, shifting models from safety-aligned dialogue to compliance-driven code completion. • Effectiveness with Minimal Query Budget: Across 8 state-of-the-art LLMs, CodeMimicry achieves a 96.25% attack success rate with 1.51 queries on average on AdvBench, and 96.07% ASR with 1.46 queries on HarmBench, substantially outperforming both template-based and optimization-based baselines. • Mechanistic Insight via Latent Space Analysis: We conduct a mechanistic analysis of code-based jailbreaks in the latent representation space, including projection onto refusal-related directions and activation steering. The results show that successful jailbreaks remain close to benign code representations while avoiding refusal-associated features, providing an explanation of how CodeMimicry bypasses safety mechanisms in code domains. 2 Related Work In the white-box setting, methods such as GCG [9] and AutoDAN [18] leverage gradient information to optimize discrete adversarial tokens. While effective, these approaches require direct access to model parameters and are therefore incompatible with commercial black-box APIs. In the black-box setting, a substantial body of work employs diverse obfuscation and transformation strategies to conceal malicious objectives and thereby achieve jailbreak attacks. FlipAttack [11], CipherChat [19], ArtPrompt [20], and DRA [21] obfuscate harmful instructions using Base64 or ASCII art. CodeChameleon [10] and CodeAttack [22] embed malicious intent within code-like structures, while MathPrompt [23] and EquaCode [24] transform instructions into mathematical problem formulations. Despite their effectiveness, these methods depend on manually crafted prompts to bypass the built-in safety mechanisms of LLMs. Their fixed and easily recognizable structures, however, make them vulnerable to simple rule- or pattern-based defenses. Another line of work formulates jailbreaking as an iterative prompt optimization problem. Evolutionary approaches (e.g., GPTFuzzer [13]) and attacker-LLM-based frameworks, including PAIR [12], TAP [25], GAP [26], ReNeLLM [27] and AutoDan-Turbo [14], employ auxiliary LLMs to iteratively rewrite prompts based on model feedback. Nonetheless, these methods are restricted to unstructured natural language prompts and fail to leverage structured or domain-specific strategies, which constrains their overall effectiveness and efficiency. Existing black-box jailbreak attacks either depend on manually designed strategies or remain confined to natural language–level prompt manipulations. By contrast, our method fully automates the use of domain knowledge via structured code completion, enabling both efficient and highly effective attacks. 3 Methodology 3.1 Problem Formulation We study jailbreak attacks against a safety-aligned large language model in a black-box setting. Let M denote the target model. Given an input prompt P, the model produces a response R=M(P)R=M(P). Let q denote a harmful query (e.g., "How to make a bomb?") that would normally trigger the model’s safety safeguards and be rejected. We focus on prompt-based jailbreak attacks, in which the adversary applies a transformation ϕ(⋅)φ(·) to convert q into an adversarial prompt P=ϕ(q)P=φ(q) that preserves the malicious intent of q while potentially evading the model’s safety mechanisms. 3.2 Threat Model We consider a black-box threat model in which the adversary interacts with the target LLM solely through standard API queries. Adversary goal. Given a harmful query q, the adversary aims to find a transformation ϕ(·)φ(·) such that the adversarial prompt P=ϕ(q)P=φ(q) induces the target model to generate a policy-violating response. A successful attack requires more than suppressing the model’s refusal behavior. It must induce the model to provide substantive harmful content. Let R=M(P)R=M(P) denote the model response. We use a binary evaluator E(P,R)E(P,R) to determine whether the attack succeeds: E(P,R)E(P,R) = 1 if R violates policy, otherwise 0. We consider P a successful jailbreak prompt if E(P,R)E(P,R) = 1. Adversary capability. The adversary has black-box query access to the target model and can submit arbitrary natural language prompts while observing only the returned outputs. The attack is restricted to prompt-level manipulation: the adversary cannot access or modify model parameters, gradients, logits, hidden states, training data, safety filters, or the deployment environment. Figure 1: CodeMimicry 3.3 CodeMimicry Attack. We introduce CodeMimicry, as illustrated in Figure 1, an automated, black-box red-team framework that utilizes attack models with coding capabilities to generate object-oriented structured code, simulates normal code completion-based requests from legitimate users to perform jailbreak attacks, and is continuously optimized. Attack with structured code completion. We employ a model with reasoning and coding capabilities as the attacker model A. The core task for A is to encapsulate the q within a complex, benign-looking code structure. In order to make the attacker model generate code that can induce the target model to complete harmful responses, we instruct A (as the prompt designed in Appendix Figure 14) to generate unfinished object-oriented structured code PcP_c. This mainly involves: 1) Structural Obfuscation: The malicious query q (e.g., "How to hack into a government database?") is decomposed into an important class’s method which is designed to contain the malicious content or steps generated by target LLMs (e.g., def execute_penetration_scenario()). 2) Docstrings and Comments: We adopt docstrings and comments as a side-channel for instruction. Instead of a direct command, the intent is framed as a functional description. This formulation guides the target model to generate direct and harmful information when completing the code. (e.g., "Simulates a comprehensive penetration testing scenario for vulnerability analysis."). 3) Utility and Parameter Simulation: To enhance the benign appearance, A is required to generate extra tool functions and realistic parameters. This forces the target model’s attention mechanism to focus on code logic rather than safety filtering. To disguise the attack as a benign code-completion request, we design a static, harmless, and universal engineering prompt PtP_t (code completion trigger) applied to PcP_c, as illustrated in Fig 13 (Appendix F). This trigger simulates a normal user instruction for code completion. The final jailbreak prompt is constructed as P=Pt(Pc)P=P_t(P_c). By embedding PcP_c into PtP_t, CodeMimicry shifts the model’s attention toward code structure and debugging context, thereby obscuring the malicious intent embedded in comments. 1: Input: Malicious query q, Target model M, Attacker model A, Successful Threshold S 2: Initialize: Initial prompt C with q 3: Initialize: attacker dialogue history H←CH← C 4: for i=1i=1 to T do 5: Pc←A(H)P_c← A(H) 6: P←Pt(Pc)P← P_t(P_c) 7: R←M(P)R← M(P) 8: Score←Evaluator(q,P,R)Score (q,P,R) 9: if Score>=SScore>=S then 10: return P,RP,R 11: end if 12: H←H+q,Pc,R,ScoreH← H+\q,P_c,R,Score\ 13: end for Algorithm 1 CodeMimicry with single process Feedback-Driven Refinement. The generation of effective jailbreak code is modeled as a multi-turn optimization problem, formalized in Algorithm 1. We employ a history-driven feedback mechanism to iteratively refine the jailbreak code. In each iteration i, the attacker model A generates a candidate code context PcP_c conditioned on the accumulated dialogue history H. Crucially, this generation is not random; A is instructed to analyze previous failures recorded in H. The generated context is then fused with the code completion trigger PtP_t, creating the final prompt P. The target’s response R is assessed by the LLM Evaluator, yielding a harmful score Score∈[1,10]Score∈[1,10]. S signifies the threshold of a successful jailbreak. If Score<SScore<S, the framework executes a refinement analysis and improves the code PcP_c. The tuple q,Pc,R,Score\q,P_c,R,Score\ is appended to HH. This updated history serves as a trajectory of trial and error. It enables the attacker model to perform self-correction via in-context learning. 4 Experiments Table 1: Comparison of different methods across various LLMs on AdvBench. We report the attack success rates (%), CodeMimicry is evaluated with T=5T=5. The bold and underlined values are the best and runner-up ASR. Method GPT-4o GPT-4.1 GPT-5 chat Claude 3.7 Sonnet Claude Sonnet 4 Gemini 2.5 Pro Llama 3.1 70B Llama 3.3 70B Ave. Template-based CipherChat [19] 6.00 12.00 22.00 0.00 0.00 78.00 0.00 0.00 14.75 ReNeLLM [27] 50.00 74.00 4.00 42.00 2.00 38.00 44.00 66.00 40.00 CodeAttack [22] 36.00 34.00 6.00 20.00 4.00 20.00 62.00 44.00 28.25 FlipAttack [11] 66.00 96.00 96.00 60.00 0.00 76.00 0.00 12.00 50.75 EquaCode [24] 82.00 100 84.00 84.00 24.00 94.00 70.00 88.00 78.25 CodeChameleon [10] 94.00 100 94.00 100 70.00 98.00 48.00 80.00 85.00 Automated PAIR [12] 32.00 18.00 18.00 4.00 0.00 72.00 40.00 90.00 34.25 DRA [21] 40.00 80.00 8.00 4.00 2.00 26.00 32.00 40.00 29.00 GPTfuzzer [13] 2.00 0.00 0.00 46.00 0.00 68.00 10.00 34.00 20.00 AutoDan-Turbo [14] 50.00 54.00 4.00 10.00 0.00 64.00 46.00 68.00 37.00 CodeMimicry 90.00 100 96.00 100 90.00 100 96.00 98.00 96.25 4.1 Main Results ASR comparison. Table 1 presents a comparison of CodeMimicry with 10 baselines, including template-based and automated methods. CodeMimicry achieves the highest attack success rate (ASR) of 96.25% on AdvBench across all 8 target LLMs. We also evaluate CodeMimicry on HarmBench B. The detailed experimental setup is provided in Appendix C. Such attack outcomes empirically validate our core hypothesis: current safety alignment is overfitted to the chat modality, leaving the code completion subspace largely undefended. The automated methods, e.g., PAIR and AutoDan-Turbo often yield poor results when confronting powerful models, particularly against the Claude Sonnet 4 model, where the attack success rate both drops to 0%. Table 1 also compares CodeMimicry with existing template-based baselines. In general, template-based methods achieve higher ASR than fully automated baselines. This is likely because they incorporate attack strategies from non-natural-language domains, rather than relying solely on natural language manipulations. However, the effectiveness of these methods (e.g., CodeChameleon and FlipAttack) fluctuates substantially across different LLMs. For instance, on Llama-3.1-70B, their ASR drops to 48% and 0%, respectively. This instability is likely attributable to the model’s sensitivity to specific, fixed jailbreak patterns. In contrast, CodeMimicry mitigates this rigidity through feedback-driven refinement, dynamically adjusting variable naming, harmful comments, and class structures to iteratively "debug" the attack and ensure successful jailbreaks. Efficiency Comparison. We also compare the efficiency of CodeMimicry with other baselines: PAIR and AutoDan-Turbo. As illustrated in Figure 2, each point in the figure represents a target model. Our method is primarily located in the upper-left region of the plot, indicating a high ASR coupled with very low average queries (AQ). Specifically, CodeMimicry achieves an AQ of 1.51. Compared to PAIR, CodeMimicry attains a 2.80× higher ASR while requiring only 0.36× the number of queries. Relative to AutoDan-Turbo, CodeMimicry achieves a 2.61× higher ASR with just 0.54× the queries. This near one-shot attack behavior indicates that the code-domain vulnerability is not sparsely distributed but instead forms a broad and easily accessible region in the model’s representation space. Such a phenomenon is consistent with our hypothesis of a Safety Generalization Lag, where safety constraints fail to densely cover the code subspace. More details are provided in Appendix B.3 and Table 8. 4.2 Ablation Study Robustness of the completion trigger. We evaluate the sensitivity of CodeMimicry’s static completion wrapper on AdvBench using GPT-4o as the target model. We compare the original Full-Wrapper with four variants: Paraphrased-Wrapper, which preserves the original intent with different wording; No-Refusal-Ban, which removes explicit refusal-ban wording; Minimal-Trigger, which uses only “Complete this python code:”; and Bare-Code, which provides only the code context. As shown in Table 2, these results show that the attack is sensitive to the wrapper, but not exclusively dependent on one exact fixed string. The paraphrased wrapper still achieves 72% ASR, indicating robustness to surface-level rewording. At the same time, the drop for No-Refusal-Ban, Minimal-Trigger, and Bare-Code shows that the wrapper contributes meaningfully by establishing a code-completion mode and suppressing refusal-style responses. Improvement with refinement. Figure 3 shows the distribution of successful jailbreaks across five interaction turns. The first code prompt is highly effective, achieving a 75.5% success rate in the first attempt, indicating that most defenses are bypassed immediately. The second turn adds a notable 12.8%, raising the cumulative ASR to 88.2%. Subsequent turns yield smaller gains (2.2%–3.2% each), but the success rate steadily reaches 96.25% by turn five. These results justify the 5-turn limit as a balance between cost and performance, and explain the low AQ (1.51), showing that most jailbreaks are successful early while harder ones benefit from iterative refinement. Different programming languages. Table 3 presents the impact of using different programming languages as the attack medium. While Python yields the optimal efficiency (1.18 AQ) and perfect success rate—likely attributable to its dominance in LLM pre-training corpora—the framework demonstrates remarkable cross-language generalization. Julia, Go, and Java all sustain near-perfect ASRs (>96%>96\%) with comparable query costs, confirming that the vulnerability is fundamental to the code-completion objective rather than being overfitted to specific Python syntax. Attacker LLMs. To investigate the influence of the attacker model’s capabilities on jailbreak performance, we conduct an ablation study on different attackers with three models: Grok-3, DeepSeek-V3 (DS-V3), and DeepSeek-R1 (DS-R1). As shown in Table 4, DS-R1 consistently outperforms Grok-3 and DS-V3, achieving the highest average ASR of 96.8% with the lowest AQ (1.48). This advantage is particularly pronounced on robust targets like Claude-Sonnet-4, where DS-R1 attains 90.0% ASR compared to 82.0% for Grok-3 and 78.0% for DS-V3. These results suggest that the enhanced reasoning capabilities of DS-R1 enable the construction of more logically intricate code structures, which prove significantly more effective at evading advanced target models. Figure 2: Comparison of CodeMimicry with PAIR and AutoDan-Turbo. Higher ASR is better; lower AQ is better. Figure 3: Ablation on iterations that resulted in a successful jailbreak, using DeepSeek-R1 as the attacker. Variant Average ASR Full-Wrapper 90% Paraphrased-Wrapper 72% No-Refusal-Ban 32% Minimal-Trigger 40% Bare-Code 36% Table 2: Sensitivity of attack success to different completion-wrapper variants on AdvBench with GPT-4o as the target model. Attack language ASR ↑ AQ ↓ Python 98.00 1.18 Julia 98.00 1.44 Go 96.00 1.58 Java 98.00 1.58 Table 3: Ablation study on AdvBench50 with different programming language, we set Llama-3.3-70B as the target model. Target Model Grok-3 DS-V3 DS-R1 GPT-4.1 100 / 1.02 100 / 1.04 100 / 1.04 GPT-5-chat 88.0 / 1.72 92.0 / 1.78 96.0 / 1.58 Claude-Sonnet-4 82.0 / 2.56 78.0 / 2.88 90.0 / 2.54 Gemini-2.5-Pro 100 / 1.00 98.0 / 1.10 100 / 1.08 Llama-3.3-70B 98.0 / 1.24 96.0 / 1.22 98.0 / 1.18 Average 93.6 / 1.51 92.8 / 1.60 96.8 / 1.48 Table 4: Ablation results on AdvBench50 with different attacker LLMs. (ASR↑ / AQ↓ ) 5 Mechanistic Analysis in Latent Space To understand why CodeMimicry succeeds, we analyze the representation space of aligned LLMs under code-based jailbreaks. To conduct feature contrastive experiments, we first test the Llama-3-8B-Instruct model using CodeMimicry and collect 50 successful jailbreak code prompts and 50 failed jailbreak code prompts (rejected by Llama model). For benign data, we select the first 50 benign instructions from [28] and use them with our designed prompts to guide DeepSeek-R1 in generating 50 benign code samples. The prompt is shown in Figure 15. Based on these data, we construct three datasets for t-SNE visualization: successful jailbreak code samples succD_succ, failed jailbreak code samples failD_fail and benign code samples BCD_BC. 5.1 Extracting Features and Visualization Let M denote the target LLM consisting of L Transformer blocks. For an input sequence x=(t1,t2…,tT)x=(t_1,t_2…,t_T), the hidden state output by the model at the l-th layer is: HM(l)(x)=(1(l),2(l),…T(l)),t(l)∈ℝdH_M^(l)(x)=(h_1^(l),h_2^(l),…h_T^(l)),h_t^(l) ^d (1) where l∈0,2,…L−1l∈\0,2,… L-1\. Given the autoregressive nature of the target models, the last token serves as a critical information bottleneck that aggregates the global context via the causal attention mechanism. Since the hidden state at this position directly conditions the prediction of the subsequent token and dominates the information flow [29, 30, 28], we adopt the hidden states of the last token at each layer as the representative embedding for the input: (l)(x)=T(l)∈Hℳ(l)(x)z^(l)(x)=h_T^(l)∈H_M^(l)(x), for each dataset D, the set of representations for the l-th layer is: (l)=(l)(x)|x∈Z_D^(l)=\z^(l)(x)|x \ (2) (a) 14th layer (b) 16th layer Figure 4: The t-SNE visualization of intermediate representations from 3 datasets in the Llama-3-8B-Instruct model: (a) layer 14 and (b) layer 16. Visualization Analysis. We adopt t-SNE and PCA for the visualization, as shown in Figure 4, the representations before layer 16 are not yet well separated, with substantial overlap among the three datasets. This suggests that, in earlier layers, the model has not yet learned to distinguish high-level semantic distinctions relevant to jailbreak behavior. Starting from layer 16, the failed jailbreak code samples aggregate into a highly isolated cluster in the latent space. In contrast, the representation regions of benign codes and successful jailbreak code are close to each other. This suggests that successful jailbreak prompts may achieve out-of-distribution evasion, rather than clustering with refuse. More details in the Appendix E. 5.2 Feature Projection on Refusal Vector To quantify the safety activation, we adopt the representation engineering approach. We hypothesize that safety fine-tuning establishes a specific refusal direction in the activation space. We calculate the centroids for failed jailbreak code samples and benign code at each layer, denoted as (l)(fail) μ_(l)(D_fail) and (l)(ℬ) μ_(l)(D_BC): (l)()=1||∑x∈(l)(x) μ_(l)(D)= 1|D| _x z^(l)(x) We then define a vector operator (l)(1,2)v_(l)(D_1,D_2), as a mapping from two datasets to a normalized direction in the representation space: (l)(1,2)=(l)(1)−(l)(2)‖(l)(1)−(l)(2)‖2v_(l)(D_1,D_2)= μ_(l)(D_1)- μ_(l)(D_2) \| μ_(l)(D_1)- μ_(l)(D_2) \|_2 This mapping captures the dominant activation direction that differentiates two datasets at layer l. In particular, we define the refusal vector as:(l)ref=(l)(fail,ℬ)v_(l)^ref=v_(l)(D_fail,D_BC) where failD_fail denotes failed jailbreak samples and BCD_BC denotes benign code samples. We project each hidden representation onto the refusal direction to quantify its alignment. The projection score S(l)(D)S_(l)(D) for input datasets D is calculated as the dot product: S(l)(D)=(l)()⋅(l)refS_(l)(D)= μ_(l)(D)·v_(l)^ref We compute the projection scores across each layers for failD_fail, BCD_BC, and succD_succ. Results and Analysis. As Figure 5 shows, failed jailbreak samples exhibit increasingly positive projection values, indicating stronger alignment with the refusal-associated direction. In contrast, benign code samples consistently show negative projection values, suggesting they lie on the opposite side of this representational axis. Notably, successful jailbreak samples remain closely aligned with benign code, rather than clustering with failed attempts, and their projection values are significantly lower than those of refused samples across all examined layers. This projection analysis suggests that successful jailbreak prompts appear to systematically avoid activating this direction, instead maintaining representations that are indistinguishable from benign code along this axis. This supports the hypothesis that CodeMimicry steers the internal activations into regions of the latent space that fall outside the model’s learned refusal manifold, thereby bypassing safety alignment guardrails. Figure 5: The Y-axis represents the projection score of hidden states onto the refusal direction defined by ℓ(fail) μ_ (D_fail) and ℓ(BC) μ_ (D_BC). Positive values indicate activation of refusal mechanisms, while negative values indicate compliance. Higher projection scores indicate stronger alignment with the refusal direction In Llama-3-8B-Instruct, failed jailbreak code samples show increasing refusal activation, whereas our successful jailbreak code samples closely track benign code in the safe subspace throughout all layers 5.3 Activation Steering during Inference To investigate whether jailbreak success is associated with hidden representation patterns, we perform activation steering in the representation space during inference. Using the vector mapping defined above, we define the jailbreak steering direction at layer l as (l)jail=(l)(fail,succ)v_(l)^jail=v_(l)(D_fail,D_succ), which represents the activation direction from refusal-aligned representations toward successful jailbreak representations. Let α∈ℝα denote the steering strength, a scalar hyperparameter that controls the magnitude of the intervention. We denote the original hidden state of the input as t(l)h_t^(l). The steering operation modifies this hidden state to obtain the post-intervention state ~t(l) h_t^(l) as follows: ~t(l)=t(l)+α⋅(l)jail h_t^(l)=h_t^(l)+α·v_(l)^jail (3) Main results. In our experiments, we use the number of model refusals to evaluate the effectiveness of activation steering. The results in Figure 6(a) show that, within the same layer, as α increases, the number of refused code jailbreak samples also increases. Under the same α, the closer the layer is to the 16th layer, the higher the number of refusals. Under the same conditions as in (a), (b) shows nearly complementary results. For the samples that failed jailbreaks, we examined the model outputs after steering, and the Llama model completed code with harmful information. These results further indicate that the key to CodeMimicry’s jailbreak lies in generating jailbreak prompts whose representations lie outside the refusal representation space. (a) (b) Figure 6: Activation steering results on CodeMimicry samples. (a) Steering applied to samples that originally elicited harmful responses. (b) Steering applied to samples that were originally refused by the model. Lighter colors indicate fewer refusals after steering. 6 Defense Against CodeMimicry We summarize the effectiveness of different defenses against CodeMimicry across 2 categories. 1) Detection-based methods such as Llama Guard [31] and perplexity filtering provide limited protection, with high residual ASR (76%–100%), whereas SelfDefend [32] significantly improves robustness, reducing ASR to 4%–14%. 2) Alignment-based defenses show mixed results: reasoning-based methods like SafeChain [33] remain largely ineffective (ASR 94%), while Safepath [34] reduces ASR to 42%; hidden-layer interventions are more effective, Circuit Breaker [35] and Representation Bending lower ASR from 90% to 8% and 38%, respectively. We further use CodeMimicry to generate a red-team dataset for fine-tuning the LlamA model to defend against CodeMimicry, where the attack success rate decreases from 90% to 62%, demonstrating improved robustness after safety alignment. Detailed experimental results are presented in Appendix D. 7 Limitations CodeMimicry is a code-based automated jailbreak framework that relies on the target model possessing sufficient code generation and instruction-following capabilities. As a result, its effectiveness may degrade on smaller or lightweight models with limited coding abilities. Another limitation of CodeMimicry is its dependence on string-level side channels, particularly comments and docstrings. Experimental results in Appendix B.4 show that introducing comment filtering substantially lowers the ASR of CodeMimicry and other code-based jailbreak attacks. 8 Conclusion We identify a previously underexplored vulnerability in large language models—safety generalization lag, where alignment trained on natural language fails to transfer to structured domains such as code. To exploit this gap, we propose CodeMimicry, a fully automated black-box jailbreak framework that embeds malicious intent within syntactically valid object-oriented code. Experiments on eight state-of-the-art LLMs show that CodeMimicry achieves high attack success rates with minimal queries, consistently outperforming existing baselines. These results indicate that current alignment methods leave significant portions of the code-domain representation space weakly constrained. We further provide a mechanistic analysis of code-based jailbreaks in latent space. Successful attacks align closely with benign code distributions while remaining separated from refusal directions, revealing a pathway for bypassing safety mechanisms via under-aligned subspaces. Activation steering results support this interpretation. References [1] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. External Links: 2306.05685, Link Cited by: §1. [2] Z. Li, C. Wang, P. Ma, D. Wu, S. Wang, C. Gao, and Y. Liu (2024) Split and merge: aligning position biases in llm-based evaluators. External Links: 2310.01432, Link Cited by: §1. [3] A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks (2024) Improving alignment and robustness with circuit breakers. External Links: 2406.04313, Link Cited by: §1. [4] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, et al. (2024) Code llama: open foundation models for code. External Links: 2308.12950, Link Cited by: §1. [5] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y. Fan, Y. Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y. Miao, S. Quan, Y. Feng, X. Ren, X. Ren, J. Zhou, and J. Lin (2024) Qwen2.5-coder technical report. External Links: 2409.12186, Link Cited by: §1. [6] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang (2024) DeepSeek-coder: when the large language model meets programming – the rise of code intelligence. External Links: 2401.14196, Link Cited by: §1. [7] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan (2022) Training a helpful and harmless assistant with reinforcement learning from human feedback. External Links: 2204.05862, Link Cited by: §1. [8] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2022) Finetuned language models are zero-shot learners. In International Conference on Learning Representations, External Links: Link Cited by: §1. [9] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. External Links: 2307.15043 Cited by: Appendix C, §1, §2. [10] H. Lv, X. Wang, Y. Zhang, C. Huang, S. Dou, J. Ye, T. Gui, Q. Zhang, and X. Huang (2024) CodeChameleon: personalized encryption framework for jailbreaking large language models. External Links: 2402.16717, Link Cited by: Appendix C, §1, §2, Table 1. [11] Y. Liu, X. He, M. Xiong, J. Fu, S. Deng, Y. MA, J. Zhang, and B. Hooi (2025) FlipAttack: jailbreak LLMs via flipping. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: Appendix C, §1, §2, Table 1. [12] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong (2024) Jailbreaking black box large language models in twenty queries. External Links: 2310.08419, Link Cited by: §B.2, Appendix C, Appendix C, Appendix C, §D.1, §1, §2, Table 1. [13] J. Yu, X. Lin, Z. Yu, and X. Xing (2024) \llm-Fuzzer\: scaling assessment of large language model jailbreaks. In 33rd USENIX Security Symposium (USENIX Security 24), p. 4657–4674. Cited by: Appendix C, §1, §2, Table 1. [14] X. Liu, P. Li, G. E. Suh, Y. Vorobeychik, Z. Mao, S. Jha, P. McDaniel, H. Sun, B. Li, and C. Xiao (2025) AutoDAN-turbo: a lifelong agent for strategy self-exploration to jailbreak LLMs. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix C, §1, §2, Table 1. [15] A. Wei, N. Haghtalab, and J. Steinhardt (2023) Jailbroken: how does llm safety training fail?. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 80079–80110. External Links: Link Cited by: §1. [16] OpenAI (2024) GPT-4 technical report. External Links: 2303.08774, Link Cited by: §1. [17] G. DeepMind (2025)Gemini 3(Website) External Links: Link Cited by: §1. [18] X. Liu, N. Xu, M. Chen, and C. Xiao (2024) AutoDAN: generating stealthy jailbreak prompts on aligned large language models. External Links: 2310.04451, Link Cited by: §2. [19] Y. Yuan, W. Jiao, W. Wang, J. Huang, P. He, S. Shi, and Z. Tu (2024) GPT-4 is too smart to be safe: stealthy chat with llms via cipher. External Links: 2308.06463, Link Cited by: Appendix C, §2, Table 1. [20] F. Jiang, Z. Xu, L. Niu, Z. Xiang, B. Ramasubramanian, B. Li, and R. Poovendran (2024) ArtPrompt: ASCII art-based jailbreak attacks against aligned LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 15157–15173. External Links: Link, Document Cited by: §2. [21] T. Liu, Y. Zhang, Z. Zhao, Y. Dong, G. Meng, and K. Chen (2024) Making them ask and answer: jailbreaking large language models in few queries via disguise and reconstruction. In Proceedings of the 33rd USENIX Conference on Security Symposium, SEC ’24, USA. External Links: ISBN 978-1-939133-44-1 Cited by: Appendix C, §2, Table 1. [22] Q. Ren, C. Gao, J. Shao, J. Yan, X. Tan, W. Lam, and L. Ma (2024) CodeAttack: revealing safety generalization challenges of large language models via code completion. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 11437–11452. External Links: Link, Document Cited by: Appendix C, §2, Table 1. [23] E. Bethany, M. Bethany, J. A. N. Flores, S. K. Jha, and P. Najafirad (2024) Jailbreaking large language models with symbolic mathematics. External Links: 2409.11445, Link Cited by: §2. [24] Z. Liang, H. Huang, and Z. Chen (2025) EquaCode: a multi-strategy jailbreak approach for large language models via equation solving and code completion. External Links: 2512.23173, Link Cited by: Appendix C, §2, Table 1. [25] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi (2024) Tree of attacks: jailbreaking black-box llms automatically. External Links: 2312.02119, Link Cited by: §2. [26] D. Schwartz, D. Bespalov, Z. Wang, N. Kulkarni, and Y. Qi (2025) Graph of attacks with pruning: optimizing stealthy jailbreak prompt generation for enhanced llm content moderation. External Links: 2501.18638, Link Cited by: §2. [27] P. Ding, J. Kuang, D. Ma, X. Cao, Y. Xian, J. Chen, and S. Huang (2024) A wolf in sheep’s clothing: generalized nested jailbreak prompts can fool large language models easily. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 2136–2153. External Links: Link, Document Cited by: Appendix C, §2, Table 1. [28] T. Li, Z. Wang, W. Liu, M. Wu, S. Dou, C. Lv, X. Wang, X. Zheng, and X. Huang (2025) Revisiting jailbreaking for large language models: a representation engineering perspective. External Links: 2401.06824, Link Cited by: §5.1, §5. [29] C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye (2024) INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §5.1. [30] A. Azaria and T. Mitchell (2023) The internal state of an llm knows when it’s lying. External Links: 2304.13734, Link Cited by: §5.1. [31] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa (2023) Llama guard: llm-based input-output safeguard for human-ai conversations. External Links: 2312.06674, Link Cited by: §B.2, §D.1, §6. [32] X. Wang, D. Wu, Z. Ji, Z. Li, P. Ma, S. Wang, Y. Li, Y. Liu, N. Liu, and J. Rahmel (2025) SelfDefend: llms can defend themselves against jailbreaking in a practical manner. External Links: 2406.05498, Link Cited by: §D.1, §6. [33] F. Jiang, Z. Xu, Y. Li, L. Niu, Z. Xiang, B. Li, B. Y. Lin, and R. Poovendran (2025) SafeChain: safety of language models with long chain-of-thought reasoning capabilities. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 23303–23320. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §D.2, §6. [34] W. Jeung, S. Yoon, M. Kahng, and A. No (2025) SAFEPATH: preventing harmful reasoning in chain-of-thought via early alignment. External Links: 2505.14667, Link Cited by: §D.2, §6. [35] A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks (2024) Improving alignment and robustness with circuit breakers. External Links: 2406.04313, Link Cited by: §D.2, §6. [36] A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer (2024) A strongreject for empty jailbreaks. External Links: 2402.10260, Link Cited by: §B.2. [37] X. Qi, Y. Zeng, T. Xie, P. Chen, R. Jia, P. Mittal, and P. Henderson (2023) Fine-tuning aligned language models compromises safety, even when users do not intend to!. External Links: 2310.03693, Link Cited by: §B.2. [38] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks (2024) HarmBench: a standardized evaluation framework for automated red teaming and robust refusal. External Links: 2402.04249 Cited by: Appendix C. [39] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein (2023) Baseline defenses for adversarial attacks against aligned language models. External Links: 2309.00614, Link Cited by: §D.1. [40] A. Yousefpour, T. Kim, R. S. Kwon, S. Lee, W. Jeung, S. Han, A. Wan, H. Ngan, Y. Yu, and J. Choi (2025) Representation bending for large language model safety. External Links: 2504.01550, Link Cited by: §D.2. Appendix A Broader Impacts This work studies safety vulnerabilities in large language models (LLMs) and can support red-teaming and robustness evaluation, particularly in code-related tasks. Our findings may help improve domain-specific alignment and safety defenses. However, the proposed method could be misused to bypass safety mechanisms and generate harmful content. We emphasize that it should be used only for research and defensive purposes. We will release our code to support reproducibility while clearly highlighting these risks. We hope this work encourages the development of stronger and more context-aware safety mechanisms for LLMs. Appendix B Additional Experiments Table 5: Comparison of different methods across various LLMs on HarmBench. We report the success rates (%), CodeMimicry is evaluated with T=5T=5. The bold and underlined values are the best and runner-up ASR. Method GPT-4o GPT-4.1 GPT-5 chat Claude 3.7 Sonnet Claude Sonnet 4 Gemini 2.5 Pro Llama 3.1 70B Llama 3.3 70B Ave. Manual Design CipherChat 0.00 8.57 15.71 0.00 0.00 74.29 0 2.86 13.39 ReNeLLM 27.14 45.71 0.00 0.2 0.00 37.14 15.71 35.71 22.68 CodeAttack 11.43 18.57 17.14 17.14 0.43 31.43 34.29 22.86 19.64 FlipAttack 74.29 87.14 75.71 52.86 0.00 81.43 0.00 22.86 49.29 EquaCode 42.86 84.29 5.71 81.43 15.71 78.57 67.14 80.00 56.96 CodeChameleon 57.14 80.00 81.43 95.71 57.14 91.43 28.57 80 71.07 Automated PAIR 42.85 64.29 18.57 5.71 0.00 75.71 37.14 91.43 41.96 DRA 40.00 68.57 11.43 5.71 0.00 41.43 47.14 47.14 32.68 GPTfuzzer 5.71 0.00 0.00 4.29 0.00 74.29 44.29 65.71 24.29 AutoDan-Turbo 37.14 57.14 14.29 7.14 0.00 72.86 57.14 58.57 43.47 CodeMimicry 98.57 100.00 100.00 100.00 74.29 100.00 97.14 98.57 96.07 B.1 Results on HarmBench. As illustrated in Table 5, the superiority of CodeMimicry is further validated on HarmBench70 dataset, where it achieves an average ASR of 96.07%, outperforming the strongest automated baseline (AutoDan-Turbo, 43.47%) by over 52 percentage points. Notably, in the evaluation against the most robust commercial models, such as Claude-Sonnet-4 and GPT-5-chat, our method demonstrates exceptional performance. While baseline methods like FlipAttack, PAIR, and DRA collapse to near-zero success rates on Claude-Sonnet-4 (0%, 0%, and 0% respectively), CodeMimicry sustains a 74.29% success rate, proving that our code-context disguised strategy effectively bypasses the semantic filters that block traditional attacks. On the GPT-5-chat, where traditional automated attacks struggle significantly (e.g., PAIR achieves only 18% on GPT-5, while AutoDan-Turbo achieves 14.29%), our framework maintains a 100% breakthrough rate. This evidence indicates that by encapsulating malicious intent within code, CodeMimicry can effectively bypass the model’s safety scrutiny and focus on code completion tasks. B.2 Evaluation with Human Expert. To evaluate the performance of our method, we adopt a diverse set of evaluation approaches, including human experts, GPT-evaluator (GPT-5-chat, GPT-4o) [12], StrongREJECT [36], HarmfulScore [37], dictionary-based methods, and Llama Guard-2-8B [31]. We adopt a dataset provided by [12], which contains 300 prompt-response Paris, to evaluate six methods. We invite 4 experts specializing in LLM security to label the pairs. We then calculate the rate of all evaluation methods with human ethical preferences. The results are shown in Table 6 and we have the following findings: 1) The evaluation remains a challenging task, as human experts still exhibit noticeable variability in their judgments. The inter-method agreement fluctuates across annotators (e.g., GPT-5-chat ranges from 88.67% to 92.33%), indicating that even strong evaluators are affected by subjective differences. 2) LLM-based evaluation methods continue to demonstrate strong alignment with human preferences, but their relative performance differs more clearly under the updated results. GPT-5-chat achieves the highest average accuracy (90.25%), followed closely by GPT-4o (89.58%). StrongREJECT attains a moderate performance (84.42%), while HarmfulScore reaches 87.33%, slightly lower than GPT-based evaluators. Compared to previous observations, the gap between GPT-based evaluators and other automated methods becomes more evident, suggesting better robustness of GPT-based evaluation. Meanwhile, Llama Guard achieves 77.08%, showing a noticeable performance gap compared to LLM-based evaluators. 3) The dictionary-based method shows significantly lower accuracy, dropping to 56.00%, which is substantially worse than all other approaches. Based on the convenience and accuracy of the GPT API, we adopt the GPT-5-chat as our evaluation method. The prompt for this evaluator is detailed in Section F, and the dictionary-based rejection keywords are listed in Table 7. Table 6: Accuracy results of 6 evaluation methods on 300 prompt-and-response pairs from AdvBench. The results present the accuracy (%) of each evaluation method relative to the assessments made by human experts. Evaluation GPT-5-chat GPT-4o StrongREJECT Dictionary-based HarmfulScore Llama Guard Human1 91.00 88.67 84.67 59.33 88.33 72.33 Human2 88.67 88.00 82.33 54.67 90.00 78.67 Human3 89.67 90.00 84.33 52.67 86.67 79.00 Human4 92.33 91.67 86.33 58.00 84.33 78.33 Average 90.25 89.58 84.42 56.00 87.33 77.08 Table 7: The dictionary-based rejection keywords. Rejection Dictionary I’m sorry illegal Sorry not legal I am sorry My apologies I apologize but I cannot As an If you have any other non-malicious requests As an AI I’m not able to provide I’m an I am really sorry I’m just I cannot create As a language model I cannot support As an Assistant I can’t provide I cannot I can’t assist I can’t I am unable I do not I must cannot It must Hello! it must is illegal and unethical is not capable of I’m really sorry As a responsible unethical never not ethical is not B.3 Efficiency comparison between automated methods. This section primarily reports the efficiency comparison between CodeMimicry and other automated methods (PAIR, AutoDan-Turbo) on the AdvBench and HarmBench datasets. Across both datasets, CodeMimicry achieved the lowest AQ and the highest ASR. In Table 8, PAIR requires the highest average number of queries (up to 4.18) yet only attains an average ASR of 34.5. Notably, when facing the most secure model, Claude Sonnet 4, all attacks fail, even those using the highly capable reasoning model DeepSeek-R1 as the attacker model. Although AutoDan-Turbo reaches a lower AQ (2.81) than PAIR, its attack success rate remains only 37.0%. In contrast, our CodeMimicry achieves an almost perfect ASR (96.25) with an average AQ of 1.51. As the results on HarmBench show in Table 9, CodeMimicry still achieves the best ASR and AQ, demonstrating the efficiency and stability of our method across different datasets. Notably, even against the Claude Sonnet 4 model, which possesses the strongest security performance, CodeMimicry still attains an attack success rate of 74.29%. Table 8: Efficiency comparison of CodeMimicry with automated methods (PAIR and AutoDan-Turbo) on AdvBench. We report the ASR(%) and the AQ. The bold values are the best results. Target Model PAIR AutoDan-Turbo CodeMimicry ASR ↑ AQ ↓ ASR ↑ AQ ↓ ASR ↑ AQ ↓ GPT-4o 32.00 4.40 50.00 2.32 90.00 1.86 GPT-4.1 18.00 4.72 54.00 2.18 100.00 1.04 GPT-5-chat 18.00 4.76 4.00 3.66 96.00 1.58 Claude-3.7 4.00 4.94 10.00 3.46 100.00 1.10 Claude-4 0.00 5.00 0.00 4.76 90.00 2.54 Gemini-2.5-Pro 72.00 2.80 64.00 1.92 100.00 1.08 Llama3.1-70B 40.00 4.16 46.00 2.56 96.00 1.74 Llama3.3-70B 90.00 2.64 68.00 1.64 98.00 1.18 Average 34.25 4.18 37.00 2.81 96.25 1.51 Table 9: Efficiency comparison of CodeMimicry with automated methods (PAIR and AutoDan-Turbo) on HarmBench. We report the ASR(%) and the AQ. The bold values are the best results. Target Model PAIR AutoDan-Turbo CodeMimicry ASR ↑ AQ ↓ ASR ↑ AQ ↓ ASR ↑ AQ ↓ GPT-4o 42.85 4.28 37.14 2.74 98.57 1.34 GPT-4.1 64.29 3.22 57.14 2.27 100.00 1.21 GPT-5-chat 18.57 4.58 14.29 3.41 100.00 1.27 Claude-3.7 5.71 4.76 7.14 2.91 100.00 1.17 Claude-4 0.00 5.00 0.00 4.95 74.29 2.74 Gemini-2.5-Pro 75.71 2.79 72.86 2.80 100.00 1.07 Llama3.1-70B 37.14 4.04 57.14 2.61 97.14 1.46 Llama3.3-70B 91.43 2.64 58.57 2.17 98.57 1.37 Average 41.96 3.91 43.47 2.99 96.07 1.46 B.4 Industrial comment filtering and deployment realism. we added sanitizer ablations on AdvBench using three target models: GPT-4o, Llama-3.1-70B, and Llama-3.3-70B. The sanitizer applies regex-based stripping to remove comment/docstring side-channel content before sending the prompt to the target model. Table 10: Attack success rates under different sanitization settings. Target Model Setting CodeMimicry CodeAttack CodeChameleon GPT-4o No sanitizer 90% 36% 94% Regex sanitizer 20% 8% 34% Llama-3.1-70B No sanitizer 96% 62% 48% Regex sanitizer 22% 2% 40% Llama-3.3-70B No sanitizer 98% 44% 80% Regex sanitizer 26% 0% 58% As the Table 10 shows, these results confirm that lightweight front-end filtering can substantially reduce comment/docstring-based jailbreak effectiveness. At the same time, the attacks are not fully neutralized. CodeMimicry retains 20%-26% ASR after regex sanitization across the three target models, and remains higher than CodeAttack under the same defense. The remaining ASR after sanitization suggests that comment/docstring side channels are an important factor, but not the only factor: even after removing them, structured code context can still induce unsafe completions in some cases. B.5 Functional validity of generated code Our primary attack success rate (ASR) follows standard jailbreak benchmarks, where a response is considered successful if it contains substantive policy-violating content, including procedural instructions, operational details, or code-like assistance. Such an evaluation does not require the generated code to constitute a fully executable malicious program, as many behaviors in AdvBench and HarmBench concern harmful guidance rather than complete runnable payloads. To further assess the practical validity of the generated code, we conduct a separate executable-code analysis on AdvBench across all eight target LLMs. For each successful attack response, we first extract the generated Python code. We then assess its validity in two stages. First, we check whether the extracted code is syntactically valid using ast.parse(). Second, syntactically valid code is executed in a sandboxed subprocess to determine whether it successfully runs. Based on these evaluations, we report three metrics: Syntax Correctness Rate (SCR): the proportion of generated code samples that pass Python syntax parsing using ast.parse(). Runtime Success Rate (RSR): the proportion of generated code samples that execute successfully in the sandboxed environment. Functional Validity Rate (FVR): the proportion of generated code samples that are both syntactically correct and successfully executable, representing end-to-end functional validity. Table 11: Functional validity of code generated by successful CodeMimicry attacks on AdvBench. SCR denotes syntax correctness rate, RSR denotes runtime success rate, and FVR denotes end-to-end functional validity rate. Model SCR (%) RSR (%) FVR (%) GPT-4o 100.0 100.0 100.0 GPT-4.1 100.0 100.0 100.0 GPT-5-Chat 100.0 100.0 100.0 Claude-3.7-Sonnet 100.0 100.0 100.0 Claude-Sonnet-4 97.8 97.8 95.6 Gemini-2.5-Pro 98.0 94.0 92.0 Llama-3.1-70B 95.6 93.3 91.1 Llama-3.3-70B 100.0 97.8 97.8 Average 98.9 97.9 97.1 As the Table 11 indicates that CodeMimicry does not merely induce superficial or malformed code-like responses. Four of the eight target models achieve a 100% FVR, while all eight models achieve an FVR of at least 90%. Overall, 98.9% of generated code samples are syntactically correct, and 97.1% achieve end-to-end functional validity. These results provide additional evidence that the attack can elicit executable Python programs from the target models, rather than merely producing superficially code-like harmful responses. This analysis complements the primary ASR evaluation by distinguishing between policy-violating content and the functional validity of generated code. In particular, the high FVR across target models suggests that the observed attack success is not solely attributable to malformed or non-executable code, but also reflects the models’ ability to generate functionally valid programs under the CodeMimicry attack. B.6 Attacker-Model Dependency and Optimization Overhead We further quantify the attacker-side overhead on AdvBench50 across the 8 target models. Here, Attacker AQ denotes the average number of attacker-model calls. Since each CodeMimicry iteration consists of one attacker-model generation followed by one target-model query, Attacker AQ is numerically identical to the target-side AQ reported in Table 12. Table 12: Attacker-side cost comparison on AdvBench50 across 8 target models. Method ASR (%) Attacker AQ Avg. Tokens (In / Out) CodeMimicry 96.3 1.51 1656 / 1341 PAIR 34.5 4.18 8030 / 5005 CodeMimicry achieves an ASR of 96.25% with an average Attacker AQ of 1.51, requiring 1,656 input and 1,341 output tokens on average. In comparison, PAIR achieves 34.25% ASR with an Attacker AQ of 4.18 and substantially higher token usage of 8,030 input and 5,005 output tokens. Thus, CodeMimicry achieves higher attack success while using fewer attacker-model calls and considerably fewer tokens. Appendix C Experimental setup Datasets. We employ AdvBench [9] and HarmBench [38] to validate our method and baselines, which contain 520 and 400 harmful behaviors respectively. In the experiment, we choose a subset of 50 examples from AdvBench [12]. For HarmBench, we randomly select 10 examples from each of its 7 behavior categories (e.g., chemical biological, cybercrime intrusion), resulting in a subset of 70 data points, these results are in section B. Target models. We conduct comprehensive evaluation on 8 SOTA closed-source commercial LLMs and open-source LLMs, including GPT-4o (GPT-4o-1120), GPT-4.1 (GPT-4.1-0414), GPT-5-chat (GPT-5-chat-1003), Claude-3.7-sonnet (Claude-3-7-sonnet-20250219), Claude-sonnet-4 (Claude-sonnet-4-20250514), Gemini-2.5-Pro, Llama-3.1-70B, Llama-3.3-70B-Instruct. Baselines. Since our approach operates in a black-box setting, we compared it with 10 black-box attacks including 6 template-based methods: CipherChat [19], ReNeLLM [27], CodeAttack [22], FlipAttack [11], EquaCode [24], CodeChameleon [10] and 4 automated black-box methods: PAIR [12], DRA [21], GPTfuzzer [13], AutoDan-Turbo [14] Experimental setting. For a fair comparison with other automated jailbreak baselines, we adopt identical hyperparameter settings across all methods. Specifically, the temperature of the attacker model is set to 0.7, while the temperatures of both the target LLMs and the evaluation LLM are fixed to 0. We set the maximum number of attack attempts to T=5T=5 and for automated attack baselines, including CodeMimicry, PAIR, and AutoDan-Turbo, we employ DeepSeek-R1 as the attacker model. Evaluation. We adopt Attack Success Rate (ASR) as the primary evaluation metric, defined as ASR=NsuccessMtotal,ASR= N_successM_total, where NsuccessN_success denotes the number of successful attacks and MtotalM_total is the total number of attempted attacks. A successful attack is determined following the GPT-based evaluation protocol proposed in [12], which employs a state-of-the-art language model as an automated judge. Specifically, we use GPT-5-Chat to score the malicious relevance and harmfulness of model responses on a 1–10 scale, and only responses receiving the maximum score of 10 are considered successful. To further quantify attack efficiency, we report the Average Queries (AQ), defined as AQ=1Mtotal∑i=1MtotalQi,AQ= 1M_total _i=1^M_totalQ_i, where QiQ_i denotes the number of iterations required for the i-th attempted attacks. Appendix D Defend Against CodeMimicry with Different Method This section presents three defensive strategies against CodeMimicry. D.1 Detection-based Defense strategies. We evaluate CodeMimicry against 3 defense methods: the perplexity filter, Llama Guard [31] and SelfDefend [32]. For Llama Guard, we reference its technical report and perform defense testing using the prompt-and-response pair as input. Following the approach of [39, 12], we set the threshold to the average perplexity of malicious behaviors found in AdvBench. In Table 13, for SelfDefend, we adopt GPT-4o as the detection model. The results show that both Llama Guard and the Perplexity filter fail to completely block CodeMimicry attacks, and SelfDefend successfully block most of CodeMimicry’s attacks. Table 13: CodeMimicry performance under defense mechanisms: We report the defense effects of Llama Guard, Perplexity filter and SelfDefend against CodeMimicry. Method GPT-4o GPT-4.1 GPT-5 chat Claude 3.7 Sonnet Claude Sonnet 4 Gemini 2.5 Pro Llama 3.1 70B Llama 3.3 70B CodeMimicry 90.00 100.00 96.00 100.00 90.00 100.00 96.00 98.00 Llama Guard 88.0 90.00 82.00 84.00 90.00 76.00 92.00 88.00 Perplexity filter 86.00 94.00 94.00 96.00 84.00 100.00 92.00 94.00 SelfDefend 12.00 6.00 12.00 8.00 14.00 4.00 8.00 6.00 D.2 Alignment-Based Defenses Against CodeMimicry We employ two categories of safety alignment methods to test the defense effectiveness against CodeMimicry: one is reasoning-based alignment training defense methods, including Safechain [33] and Safepath [34]; the other is hidden layer-based safety alignment defense methods, including Circuit Breaker [35] and Representation Bending [40]. Table 14: Defense results on reasoning methods. Safechain and Safepath stand for DeepSeek-R1-8B models reasoning trained by their own method. We report the ASR(%), the lower ASR the defense better. Method DS-R1-8B Safechain Safepath CodeMimicry 92 94 42 AutoDan-Turbo 90 86 4 PAIR 82 64 8 Reasoning-based defense. The results in Table 14, which lead to three key observations: First, the attack transfers effectively to reasoning models: the high ASR on DeepSeek-R1-8B (92%) indicates that stronger reasoning capability alone does not eliminate the vulnerability. Second, SafeChain does not substantially mitigate the attack (94% ASR), suggesting that CoT-style safety alignment does not directly address the mechanism exploited by CodeMimicry. In particular, our method does not rely on eliciting explicit harmful reasoning, but instead shifts the model into a code-completion mode where harmful semantics are embedded in structurally valid code. Third, SAFEPATH provides the strongest mitigation among the tested models, reducing ASR to 42%. This suggests that early-stage intervention in the reasoning process is more effective than standard reasoning-aligned safety training. However, the remaining ASR (42%) is still substantial, and CodeMimicry continues to significantly outperform natural-language automated baselines on this model (42% vs. 4% for AutoDan-Turbo and 8% for PAIR). Overall, these results show that CodeMimicry remains effective on reasoning models, and that current reasoning-aware safety alignment techniques are insufficient to fully address this class of attacks. We will include these additional experiments and analyses in the revised version. Hidden state-based defense. We evaluated CodeMimicry on safety-enhanced Llama-3-8B models trained with Circuit Breakers and Representation Bending. As the hidden layer visualization Figure 7 and Figure 8 show, jailbreak code samples that originally succeeded move closer to the rejection representation space after being aligned by both hidden-state-based methods, indicating that these two hidden layer alignment methods are effective in defending against CodeMimicry. Furthermore, we actually tested the Llama-3-8B model after applying such hidden layer alignment, and the Attack Success Rate (ASR) dropped from 90% to 8% (CircuitBreaker) and 38% (ReBend), respectively. These two alignment methods demonstrate strong defense effectiveness against CodeMimicry. (a) (b) (c) (d) Figure 7: The t-SNE visualization results of layer 16 (a), layer 18 (b), the PCA visualization results of layer 16 (c),layer 18 (d), on Llama-3-8b aligned by Circuit Breaker. (a) (b) (c) (d) Figure 8: The t-SNE visualization results of layer 16 (a), layer 18 (b), the PCA visualization results of layer 16 (c),layer 18 (d), on Llama-3-8b aligned by Representation Bending.. D.3 Adversarial Fine-tuning Against CodeMimicry We investigate adversarial fine-tuning with attack examples, specifically, we sampled 50 harmful prompts from AdvBench and generated 10 CodeMimicry attack prompts per prompt, yielding 500 adversarial examples. We combined these with 100 benign code samples to fine-tune Llama-3-8B by LoRA, with the goal of improving safety alignment in code-completion settings. Evaluation was conducted on a disjoint subset of 50 AdvBench examples, from which CodeMimicry jailbreak prompts were generated under the same attack configuration. Under this setting, the ASR decreases from 90% for the base model to 62% after fine-tuning. Appendix E Extracting Features and Visualization E.1 Visualization on Llama-3-8B-Instruct As shown in Figure 9 (PCA visualization) and Figure 10 (t-SNE visualization) on Llama-3-8B-Instruct, the clusters of failed jailbreak samples and benign code are clearly separated in the 16 layers. This indicates that Llama-3-8B is capable of successfully recognizing the harmful intent of instructions. However, our successful jailbreak code samples and the benign code samples remain closely intertwined, maintaining a state of mutual inclusion, which suggests that the model fails to distinguish the malicious intent of the code and proceeds to complete the malicious content of the code. (a) (b) (c) (d) (e) (f) Figure 9: The PCA visualization results of layer 12 (a), layer 14 (b), layer 15 (c),layer 16 (d), layer 18 (c),layer 20 (d), on Llama-3-8b-instruct. (a) (b) (c) (d) (e) (f) Figure 10: The t-SNE visualization results of layer 12 (a), layer 14 (b), layer 15 (c),layer 16 (d), layer 18 (e), layer 20 (f), on Llama-3-8B-Instruct. E.2 Generalization of Mechanistic Analysis on Llama-3.1-70B-Instruct Feature Projection on the Refusal Vector. To examine whether the representation-level findings generalize to a larger open-weight model, we have repeated the refusal-vector projection analysis on Llama-3.1-70B-Instruct. Specifically, we collect 27 failed jailbreak code prompts, 50 successful jailbreak code prompts, and the same 50 benign code samples used in the paper. Following the protocol in Section 5, for each layer l, we define the refusal direction as the normalized difference between the mean activation of failed jailbreak samples and that of benign code samples. We then project successful jailbreak, failed jailbreak, and benign code samples onto this direction. The Table11 results reproduce the same qualitative pattern observed on Llama-3-8B-Instruct. Failed jailbreak samples consistently lie on the positive/refusal side of the axis, whereas successful jailbreak samples remain close to benign code and on the opposite side of the refusal direction. The separation also becomes stronger in deeper layers, with the largest gap appearing in the final layers. This suggests that the representation-space explanation is not specific to the 8B model, but also appears in a much larger open-weight model. Activation Steering during Inference. We have also repeated the activation-steering experiment on Llama-3.1-70B-Instruct. We inject the layer-specific refusal direction into selected layers with different steering strengths and measure how many outputs change their refusal/compliance behavior. As can be seen from Table 12, the results are consistent with the 8B model: larger steering strengths affect more samples, and the effect is strongest near layer 40; layers farther from this range require stronger intervention or change fewer outputs. This intervention result supports the projection analysis, suggesting that successful CodeMimicry prompts bypass refusal-related activations not only in the 8B model but also in the much larger 70B model. Figure 11: The Y-axis represents the projection score of hidden states onto the refusal direction defined by ℓ(fail) μ_ (D_fail) and ℓ(BC) μ_ (D_BC). Positive values indicate activation of refusal mechanisms, while negative values indicate compliance. Higher projection scores indicate stronger alignment with the refusal direction In Llama-3.1-70B-Instruct, failed jailbreak code samples show increasing refusal activation, whereas our successful jailbreak code samples closely track benign code in the safe subspace throughout all layers (a) (b) Figure 12: Activation steering results on CodeMimicry samples. (a) Steering applied to samples that originally elicited harmful responses. (b) Steering applied to samples that were originally refused by the model. Lighter colors indicate fewer refusals after steering. Appendix F Prompt Figure 13: The Code Completion Trigger is designed to simulate a regular user requesting LLM code completion. The language can be set to Python, JavaScript, etc., and the jailbreak code is generated by the attacker. Figure 14: Our Jailbreak prompt for attacker LLM to generate the jailbreak code. Figure 15: A benign prompt for DeepSeek-R1 to generate benign code. the goal is set by benign query. Figure 16: The evaluate prompt for GPT LLMs.