Paper deep dive

DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation

Li Huang, Zhongxin Liu, Yifan Wu, Tao Yin, Dong Li, Jichao Bi, Nankun Mu, Hongyu Zhang, Meng Yan

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 84

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 4/14/2026, 1:50:25 AM

Summary

DeepGuard is a framework for secure code generation that addresses the 'final-layer bottleneck' in LLMs by aggregating security-relevant semantic cues from multiple upper transformer layers. It utilizes an attention-based multi-layer aggregator and a multi-objective training framework to balance security enhancement with functional correctness, complemented by a lightweight inference-time steering strategy.

Entities (6)

DeepGuard · framework · 100%LoRA · technique · 98%DeepSeek-Coder · llm · 95%Qwen2.5-Coder · llm · 95%Seed-Coder · llm · 95%SVEN · baseline · 95%

Relation Signals (3)

DeepGuard → utilizes → LoRA

confidence 98% · We adapt the base model using LoRA (Hu et al., 2022)

DeepGuard → evaluatedon → Qwen2.5-Coder

confidence 95% · We evaluate DEEPGUARD on a diverse set of recent open-source code LLMs... including Qwen2.5-Coder

DeepGuard → improvesmetric → secure-and-correct generation rate

confidence 95% · DEEPGUARD improves the secure-and-correct generation rate by an average of 11.9% over strong baselines

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Models (LLMs) for code generation can replicate insecure patterns from their training data. To mitigate this, a common strategy for security hardening is to fine-tune models using supervision derived from the final transformer layer. However, this design may suffer from a final-layer bottleneck: vulnerability-discriminative cues can be distributed across layers and become less detectable near the output representations optimized for next-token prediction. To diagnose this issue, we perform layer-wise linear probing. We observe that vulnerability-related signals are most detectable in a band of intermediate-to-upper layers yet attenuate toward the final layers. Motivated by this observation, we introduce DeepGuard, a framework that leverages distributed security-relevant cues by aggregating representations from multiple upper layers via an attention-based module. The aggregated signal powers a dedicated security analyzer within a multi-objective training objective that balances security enhancement and functional correctness, and further supports a lightweight inference-time steering strategy. Extensive experiments across five code LLMs demonstrate that DeepGuard improves the secure-and-correct generation rate by an average of 11.9% over strong baselines such as SVEN. It also preserves functional correctness while exhibiting generalization to held-out vulnerability types. Our code is public at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,063 characters extracted from source content.

Expand or collapse full text

DEEPGUARD: Secure Code Generation via Multi-Layer Semantic Aggregation Li Huang 1 , Zhongxin Liu 2 , Yifan Wu 3 , Tao Yin 1 , Dong Li 1 , Jichao Bi 1 , Nankun Mu 1 * , Hongyu Zhang 1 , Meng Yan 1 1 Chongqing University, 3 Peking University 2 The State Key Laboratory of Blockchain and Data Security, Zhejiang University lee.h, lidong, bjc, nankun.mu, hyzhang, mengy@cqu.edu.cn yintao@stu.cqu.edu.cn liu_zx@zju.edu.cn, yifanwu@pku.edu.cn Abstract Large Language Models (LLMs) for code gen- eration can replicate insecure patterns from their training data. To mitigate this, a com- mon strategy for security hardening is to fine- tune models using supervision derived from the final transformer layer.However, this design may suffer from a final-layer bottle- neck: vulnerability-discriminative cues can be distributed across layers and become less de- tectable near the output representations opti- mized for next-token prediction. To diagnose this issue, we perform layer-wise linear probing. We observe that vulnerability-related signals are most detectable in a band of intermediate- to-upper layers yet attenuate toward the final layers. Motivated by this observation, we intro- duce DEEPGUARD, a framework that leverages distributed security-relevant cues by aggregat- ing representations from multiple upper layers via an attention-based module. The aggregated signal powers a dedicated security analyzer within a multi-objective training objective that balances security enhancement and functional correctness, and further supports a lightweight inference-time steering strategy. Extensive experiments across five code LLMs demon- strate that DEEPGUARD improves the secure- and-correct generation rate by an average of 11.9% over strong baselines such as SVEN. It also preserves functional correctness while exhibiting generalization to held-out vulnera- bility types. Our code is public at§ https: //github.com/unknownhl/DeepGuard. 1 Introduction Large Language Models (LLMs) have demon- strated exceptional performance in various programming-relatedtasks,particularlyin generating functionally correct code based on user-provided prompts (Nijkamp et al., 2022; Yan et al., 2025). This capability has led to their * Corresponding author. 051015202530 Transformer Layers 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Probability of Vulnerability Peak Detection Confidence (Layer 9) Vulnerable Inputs Secure Inputs Semantic Rich Region Figure 1: Layer-wise diagnostic evidence on Seed- Coder-8B. We train a linear probe on each trans- former layer to detect vulnerable patterns and report the probe confidence across layers. The vulnerability- discriminative signal peaks in intermediate-to-upper lay- ers and attenuates toward the final layers. widespread adoption in real-world development environments. For example, GitHub’s Copilot is reported to assist in generating up to 46% of the code on its platform (Dohmke, 2023). However, this rapid integration introduces a critical and persistent security risk.The models’ power is rooted in their training on vast amounts of public code, which is a double-edged sword: the models also learn and can replicate the insecure coding patterns common in that data. Pearce et al. (2025) found that approximately 40% of code generated by Copilot contained vulnerabilities. Compounding this issue, user studies confirm that developers often fail to identify these AI-generated flaws (Mohsin et al., 2024; Majdinasab et al., 2024). Consequently, while code LLMs accelerate development, they risk introducing vulnerabilities into the software ecosystem (Basic and Giaretta, 2024), highlighting the urgent need for security hardening methods. To address this challenge, several defence mech- anisms have been proposed. The first is inference- arXiv:2604.09089v1 [cs.SE] 10 Apr 2026 (B) Multi-Layer Guidance (A) Single-Layer Guidance Prompt defprocess_data(user_input): query = "SELECT ... name = Generated Layer N-1 ... Layer 0 LLMS Layer 1 Layer 2 defprocess_data(user_input): query = "SELECT ... name = '" + user_input+ "'" Prediction ' + ? logits distribution Prompt defprocess_data(user_input): query = "SELECT ... name = Generated defprocess_data(user_input): query = "SELECT ... name = ?" db.execute(query, (user_input,)) DeepFusion Module Layer N-1 ... Layer 0 LLMS Layer 1 Layer N-2 Guided Prediction ' + ? logits distribution Figure 2: Comparison of security guidance paradigms. (A) Single-layer guidance suffers from signal attenu- ation at the final layer. (B) DEEPGUARD (Ours) em- ploys multi-layer aggregation to capture richer security- critical cues distributed across upper layers. time interventions, which treat the code LLM as a fixed black box. These methods range from au- tomated prompt optimization (Nazzal et al., 2024; Zhang et al., 2024) to co-decoding with smaller models trained for security verification (Li et al., 2024). However, such methods do not adapt the model itself and typically rely on post-hoc feedback or surface-level patterns, which may be insufficient to correct a model’s insecure generation tendencies. A more powerful direction is model adapta- tion through training, including security-specific instruction tuning (He et al., 2024) and prefix- tuning (He and Vechev, 2023). While effective, most of them share a critical limitation: they de- rive the training signal almost exclusively from the final transformer layer. We refer to this limi- tation as a final-layer bottleneck. Preventing inse- cure code often requires integrating diverse syntac- tic and semantic evidence. For example, identify- ing a potential SQL injection requires recognizing the syntactic pattern of string concatenation and reasoning about semantic properties such as un- trusted data flow. Such evidence is known to be distributed hierarchically across transformer layers: shallower layers tend to capture structural syntax, while deeper layers encode more abstract seman- tics (Ma et al., 2024; Wan et al., 2022). Meanwhile, the final-layer representation is primarily optimized for next-token prediction rather than fine-grained vulnerability discrimination. As a result, features useful for separating vulnerable from secure pat- terns can become less separable near the output layer. Figure 1 provides diagnostic evidence con- sistent with this hypothesis: probe-detectable vul- nerability signals attenuate toward the final layers. To address this limitation, we introduce DEEP- GUARD, a hybrid framework that combines model adaptation with a lightweight inference-time steer- ing strategy. DEEPGUARD moves beyond final- layer-only analysis by introducing an attention- based multi-layer aggregator (Figure 2B). The ag- gregator dynamically fuses hidden states from mul- tiple upper layers, producing an aggregated repre- sentation that is more sensitive to security-critical cues distributed across the layers of the model. This representation powers a dedicated security analyzer within a multi-objective training framework that co-optimizes security enhancement and functional correctness. During inference, DEEPGUARD com- putes a context-aware security bias once from the prompt and applies it to logits during generation, helping steering the code away from vulnerable patterns without per-step re-evaluation overhead. We evaluate DEEPGUARD on both security enhancement and functional correctness across five strong code LLMs. The results show that DEEPGUARD achieves a favourable balance be- tween these competing objectives. For example, on Qwen2.5-Coder-3B, a strong baseline (SVEN) achieves a sec-pass@1score of 70.47%. After applying DEEPGUARD, this score increases to 80.76% while maintaining functional correctness (pass@1 of 86.65%, close to the original model). Across models, DEEPGUARD improves the secure- and-correct generation metric by 11.9% on average over SVEN, and exhibits strong generalization to vulnerability types held out during training within the benchmark. In summary, our contributions are: •We provide diagnostic evidence that vulnerability signals attenuate at the final transformer layer, highlighting the limitations of final-layer-only supervision. •We propose DEEPGUARD, a framework incor- porating attention-based multi-layer aggregation and multi-objective training to leverage internal model representations for security. •We demonstrate through extensive evaluation that DEEPGUARD achieves superior security perfor- mance and generalization across multiple models compared to baselines. 2 Related Work Security of LLM-generated Code Large lan- guage models are known to generate vulnerable code (Pearce et al., 2025; He et al., 2024; Asare et al., 2024; Huang et al., 2025). Foundational stud- ies established the systematic evaluation of these models using industry-standard tools like GitHub CodeQL (GitHub, 2023) to detect Common Weak- ness Enumerations (CWEs) (MITRE, 2023). Pi- oneering work by Pearce et al. (2025) used this approach to find that a significant portion of AI- generated code contains exploitable vulnerabili- ties, a finding later confirmed by numerous oth- ers (Khoury et al., 2023; Siddiq and Santos, 2022; Fakih et al., 2025; de Fitero-Dominguez et al., 2024). The demonstrated security risks have moti- vated two main categories of defences. Inference- time methods (Fu et al., 2024), such as prompt op- timization (Nazzal et al., 2024) or co-decoding (Li et al., 2024), offer flexibility but are limited in their ability to correct a model’s underlying insecure tendencies. In contrast, training-time adaptation methods directly modify the model’s behaviour through security-focused fine-tuning (He et al., 2024; Huang et al., 2026) or prefix-tuning (He and Vechev, 2023). While powerful, these methods share a critical limitation: they almost exclusively use the final-layer hidden states of the model as their primary training signal. This “point” repre- sentation creates an information bottleneck, ignor- ing the rich context distributed across the model’s layers. Our work addresses this limitation within the model adaptation paradigm. Multi-Layer Feature Aggregation It is well- established that the internal representations of Transformer-based models are hierarchical. In the domain of source code, probing studies have con- firmed that different layers specialize in capturing distinct features: lower layers tend to encode local syntactic structures, while upper layers learn more abstract semantic properties (Ma et al., 2024; Wan et al., 2022). However, the distributed information available in the intermediate layers of code LLMs remains largely untapped by prior security harden- ing methods. Our work is the first to propose and evaluate a learned, multi-layer aggregation strategy for this purpose, demonstrating that the resulting “regional” representation provides a more robust signal for identifying and mitigating vulnerabilities compared to existing final-layer-only approaches. 3 DeepGuard This section introduces DEEPGUARD, a training- and-inference framework designed to mitigate the common limitation of security adaptation meth- ods that derive supervision primarily from the final transformer layer. Motivated by our diagnostic analysis (Figure 1), the key is to leverage security- relevant cues that can be distributed in intermediate- to-upper layers, rather than relying on a single final- layer vector. DEEPGUARD comprises two com- ponents: (i) a multi-objective adaptation stage that updates the code LLM using LoRA, and (i) a lightweight guided inference stage that applies a prompt-conditioned security bias during gener- ation. We denote the base code LLM asMwith parametersθ, and the adapted model asM ′ with parametersθ ′ = θ + ∆θ, where∆θdenotes the effective parameter update induced by the trainable LoRA modules. 3.1 Multi-Layer Representation Aggregation We aim to construct a representation that provides a stronger basis for security analysis than using a single final-layer state alone. Given an input token sequencex = (t 1 ,t 2 ,...,t S ), the adapted model M ′ produces hidden states fromLtransformer lay- ers,H 1 ,H 2 ,...,H L , whereH i ∈ R S×D and Dis the hidden dimension. To capture distributed security-relevant signals, we restrict our focus to the topNlayers rather than the final layer alone. Specifically, we aggregate the hidden states from the setH top-N =H L−N+1 ,...,H L . Attention-based fusion. We introduce an ag- gregatorf agg to fuseH top-N into a single rep- resentationH agg ∈ R S×D . Concretely, for to- ken positionj, we stack its layer-wise states as h (j) = [h (j) L−N+1 ,...,h (j) L ] ⊤ ∈ R N×D .We com- pute the fused stateh (j) agg using an attention module. Specifically, we use the mean of the stacked states as a summary query, ̄ h (j) = 1 N P L i=L−N+1 h (j) i , and setQ (j) = ̄ h (j) W Q ,K (j) = h (j) W K , and V (j) = h (j) W V , whereW Q ,W K ,W V ∈ R D×D . The fused state is then computed as h (j) agg = Softmax Q (j) K (j) ⊤ √ D ! V (j) .(1) Intuitively, ̄ h (j) provides a stable “consensus” sum- mary across layers, and attention then assigns higher weight to layer views that are most informa- tive for the downstream analyzer. 3.2 Training: Multi-Objective Adaptation We adapt the base model using LoRA (Hu et al., 2022) on paired dataD = (x vul ,x sec ), where (B) Inference Phase defprocess_data(user_input): query = "SELECT ... name = ?" db.execute(query, (user_input,)) Base LLMs Lora Layer N-1 ... Layer 0 Layer 1 Layer N-2 ... Multi-Layer Fusion Security Analyzer Security Score Token Security Stats defprocess_data(user_input): query = "SELECT ... name = ' + ? Guided Logits Original Logits Security Logits Processor Base LLMs Lora Layer N-1 ... Layer 0 Base LLMs Layer 1 Layer N-2 Layer N-1 ... Layer 0 Layer 1 Layer N-2 ... Multi-Layer Fusion Security Analyzer CrossEntropy Margin Loss KL Divergence score_vul score_sec 푃 푙 푠 ℒ 푠 ℒ 푔푠 푔 ℒ 푘푙 푃 푏푙푠 푠 (A) Training Phase Token Security Stats def pascal_case( value: str) -> str: return stringcase.p ascalcase(va lue) Vulnerable Code def pascal_case( value: str) -> str: return stringcase.p ascalcase(_s anitize(valu e)) SecureCode 푃 푙 푠 Figure 3: Overview of DEEPGUARD, depicting the multi-objective training phase and the guided inference phase. x vul is a vulnerable snippet andx sec is its function- ally equivalent secure counterpart. Our training objective balances three goals: encouraging secure behavior, preserving fluency, and maintaining func- tional correctness. Security and Contrastive Objective We intro- duce a security analyzerf sa parameterized byφ sa . The analyzer consumes (i) the aggregated represen- tationH agg and (i) a learned token-level security embeddingE sec ∈ R |V|×D emb , whereVis the vo- cabulary. The embedding provides a lightweight token prior that can complement contextual infor- mation inH agg . Specific initialization and architec- tural details are provided in Appendix C.2. For an input sequence x, we compute per-token scores: s(x) = f sa [H agg ; f emb (x)] ∈ [0, 1] S ,(2) wheref emb is an embedding lookup and[·;·]de- notes concatenation along the hidden dimension, and the score at positioniis denoted bys i (x). In practice,f sa is a small MLP whose outputs are normalized to[0, 1]via a sigmoid function. To evaluate the sequence as a whole, we define the sequence-level security score as the average of the token-level scores ̄s(x) = 1 S P S i=1 s i (x). Given a training pair(x vul ,x sec ), we compute their re- spective sequence scores ̄s vul and ̄s sec . We then apply a margin-based contrastive loss to encourage separation, letting δ s = ̄s sec − ̄s vul : L sec = E (x vul ,x sec )∼D [max(0, ∆− δ s )],(3) where∆is a margin hyperparameter. This objec- tive provides a direct training signal that prefers secure variants over their vulnerable counterparts under the analyzer. Preserving Fluency and FunctionalityTo main- tain language modeling ability, we include the stan- dard next-token prediction loss on secure examples: L gen =−E x sec ∼D   |x sec | X i=1 logP (t i | t <i ;θ ′ )   . (4) To reduce catastrophic forgetting, we further regu- larize the adapted distributionP θ ′ toward the frozen base model distribution P θ using KL divergence: L kl = E x sec ∼D D KL P θ ∥P θ ′ x sec ,(5) whereD KL (P θ ∥P θ ′ |x)denotes the KL divergence betweenP θ (·|x)andP θ ′ (·|x). The final objective is a weighted sum: L total =L gen + w sec L sec + w kl L kl ,(6) wherew sec andw kl balance security and preserva- tion objectives. 3.3 Inference: Guided Secure Generation While the training objective encourages secure be- havior, inference-time steering can further reduce insecure outputs with minimal overhead. We refer to this mechanism—combining a lightweight token prior with prompt-conditioned logit biasing—as guided inference. A lightweight token prior.We maintain a token- level prior vectorT stats ∈ R |V| to capture the global empirical association of each token with secure versus vulnerable contexts. Concretely, dur- ing training, we update the entries inT stats corre- sponding to the tokens present in each batch: we increase the scores for tokens appearing in secure samples and decrease them for those in vulnerable samples by a fixed step size. The values are finally clipped to[−1, 1]to ensure stability. This prior is not intended to be a calibrated vulnerability estima- tor, but serves as a weak distributional bias when combined with contextual signals. We provide a statistical analysis and semantic interpretation in Appendix F.3. Prompt-conditioned bias. Given an input promptx prompt , we perform a single forward pass to compute its aggregated representationH prompt agg and obtain per-token scoress(x prompt )from the trained analyzer. We summarize the prompt by its mean score ̄s prompt , which serves as a coarse indicator of the prompt’s security posture under the analyzer. We then compute a vocabulary-wide bias vector b∈ R |V| : b = (1− ̄s prompt )· T stats max(|T stats |) + ε ,(7) where normalization scalesT stats to a bounded range andεensures numerical stability. The factor (1− ̄s prompt ) ∈ [0, 1]modulates the bias strength, yielding stronger steering when the prompt appears more vulnerable under the analyzer. Logit biasing. At each decoding stepi, we add the fixed bias to the model’s logits z i : z ′ i = z i + b.(8) We then samplet i ∼ Softmax(z ′ i ) . This design avoids per-step re-evaluation by the analyzer and introduces only negligible overhead beyond stan- dard decoding. We provide a theoretical FLOPs analysis in Appendix E.1 and report the empirical inference latency across models in Appendix F.2. Discussion.Our guided inference is intentionally lightweight and does not aim to replace stronger but more expensive search-time defences (e.g., it- erative re-scoring). Instead, it provides a low-cost complement that empirically improves security un- der the same decoding budget. 4 Experiments 4.1 Setup Models and Benchmarks. We evaluate DEEP- GUARD on a diverse set of recent open-source code LLMs spanning multiple families and model scales, including Qwen2.5-Coder (3B, 7B) (Hui et al., 2024), DeepSeek-Coder (1.3B, 6.7B) (Guo et al., 2024), and Seed-Coder (8B) (Zhang et al., 2025). Our experiments follow a widely-used secure code generation benchmark and evaluation protocol in- troduced by He and Vechev (2023) and Fu et al. (2024), enabling direct comparison under the same scenario-based setup. Dataset statistics and unit test specifications are provided in Appendix A. Baselines.We compare against representative de- fenses from different paradigms: two strong white- box adaptation baselines SVEN (He and Vechev, 2023) and SafeCoder (He et al., 2024), two strong inference-time defenses CoSec (Li et al., 2024) and CodeGuard+ (Fu et al., 2024), and a simple prompt-based safety instruction baseline. We also report the Base Model without adaptation. All methods are evaluated under the same prompts and decoding budget. Metrics.We adopt the comprehensive evaluation protocol used by Fu et al. (2024). We use secure- pass@k as the primary utility metric, and addi- tionally report sec@k pass as a diagnostic metric for held-out vulnerability types, which isolates se- curity among correct generations. We also report pass@k and SVEN-SR for completeness. Formal definitions are included in Appendix B. Implementation Details. We implement DEEP- GUARD using LoRA for all model variants. Unless stated otherwise, we maintain a consistent hyperpa- rameter configuration across different model fam- ilies. For inference, we adopt a low-temperature sampling strategy to favor deterministic code gen- eration. A comprehensive listing of configurations is provided in Appendix C.1 and hyperparameter sensitivity is shown in Appendix E. 4.2 Main Results Table 1 shows the main results across five code LLMs. DEEPGUARD improves security-oriented metrics while maintaining competitive functional correctness. We highlight several observations be- low. For a granular performance breakdown across specific CWE scenarios, see Figures 12 and 13. Table 1: Performance comparison across different models and methods. All metrics are reported as percentages (%). “Imp. (%)” columns show the relative improvement of DEEPGUARD (Ours) over other baselines. ModelMethod pass@1 (↑)sec@1 pass (↑)sec-pass@1 (↑)SVEN-SR (↑) ValueImp.(%)ValueImp.(%)ValueImp.(%)ValueImp.(%) Base91.00-4.7876.47+21.8969.59+16.0577.95+20.73 Prompt85.41+1.4572.93+27.8162.29+29.6575.84+24.09 SVEN83.00+4.4084.90+9.7970.47+14.6082.60+13.93 SafeCoder63.94+35.5282.34+13.2052.65+53.3987.02+8.15 CoSec82.06+5.5976.85+21.2963.06+28.0778.35+20.11 CodeGuard+88.82-2.4480.13+16.3271.18+13.4681.37+15.66 Qwen2.5- Coder-3B Ours86.65–93.21–80.76–94.11– Base80.94+2.7776.45+15.3661.88+18.5478.36+13.85 Prompt84.35-1.3983.26+5.9270.24+4.4384.53+5.54 SVEN81.00+2.6975.45+16.8961.12+20.0176.24+17.01 SafeCoder79.76+4.2984.51+4.3567.41+8.8186.69+2.91 CoSec80.82+2.9279.33+11.1764.12+14.3980.44+10.90 CodeGuard+82.06+1.3685.66+2.9570.29+4.3587.18+2.33 Qwen2.5- Coder-7B Ours83.18–88.19–73.35–89.21– Base81.65-0.7269.81+21.6357.00+20.7469.83+25.61 Prompt83.24-2.6270.32+20.7558.53+17.5869.71+25.82 SVEN81.88-1.0074.50+13.9761.00+12.8277.87+12.64 SafeCoder65.88+23.0479.20+7.2152.18+31.8977.16+13.67 CoSec81.76-0.8672.37+17.3359.18+16.2971.64+22.43 CodeGuard+82.35-1.5792.86-8.5676.47-10.0088.24-0.60 DeepSeek- Coder-1.3B Ours81.06–84.91–68.82–87.71– Base91.35-3.1575.27+5.6568.76+2.3176.47+7.00 Prompt82.06+7.8178.71+1.0364.59+8.9276.61+6.80 SVEN85.71+3.2279.41+0.1468.06+3.3682.34-0.63 SafeCoder68.71+28.7684.59-5.9958.12+21.0488.12-7.15 CoSec84.24+5.0273.81+7.7462.18+13.1475.21+8.79 CodeGuard+87.59+1.0086.57-8.1475.82-7.2187.58-6.58 DeepSeek- Coder-6.7B Ours88.47–79.52–70.35–81.82– Base84.88+2.0172.77+28.0961.76+30.6876.30+22.16 Prompt86.12+0.5586.48+7.7874.47+8.3882.55+12.91 SVEN83.76+3.3888.62+5.1874.24+8.7185.94+8.46 SafeCoder81.06+6.8292.31+0.9774.82+7.8793.44-0.25 CoSec77.41+11.8681.16+14.8562.82+28.4882.16+13.45 CodeGuard+77.06+12.3782.82+12.5563.82+26.4779.56+17.16 Seed- Coder-8B Ours86.59–93.21–80.71–93.21– Security enhancement under end-to-end utility. We first focus on sec-pass@1, which measures the probability that the generated code is both secure and functionally correct. We observe that DEEP- GUARD achieves the strongest or near-strongest sec-pass@1 across all evaluated models in Table 1. In particular, on Qwen2.5-Coder-3B, DEEPGUARD improves sec-pass@1 from 70.47% (SVEN) to 80.76%, indicating a substantial gain under the same benchmark setting. Averaged across models, DEEPGUARD yields consistent improvements over both SVEN and CoSec on sec-pass@1. Functional correctness is largely preserved. Security hardening methods can trade off func- tional correctness (Dai et al., 2025). In Table 1, DEEPGUARD generally maintains strong pass@1, often close to the base model and competitive with other defenses. For example, on DeepSeek-Coder- 6.7B, DEEPGUARD attains pass@1 of 88.47%, higher than SVEN (85.71%) and CoSec (84.24%). We also note that in a few cases the relative or- dering among methods can vary by model family, suggesting that the security–utility trade-off may be model-dependent in practice. Security among correct solutions. To isolate security performance conditioned on correctness, we examine sec@1 pass . DEEPGUARD achieves the best sec@1 pass for all five models in Table 1, sug- gesting that when the model produces a correct so- lution, DEEPGUARD increases the likelihood that the solution is secure. Notably, the prompt-based baseline can be competitive on some models (e.g., Seed-Coder-8B), highlighting that instruction-level safety prompting can already capture part of the benefit in this benchmark. However, DEEPGUARD remains consistently stronger on sec@1 pass . Generalization to held-out vulnerability types. A rigorous test of any security hardening method is its ability to handle threats not seen during training. This evaluation (He and Vechev, 2023) comprises Qwen2.5-Coder-3BQwen2.5-Coder-7BDeepSeek-Coder-1.3BDeepSeek-Coder-6.7BSeed-Coder-8B 70 75 80 85 90 95 100 91.9 86.6 85.9 76.2 100 82.4 84.6 78.8 75.2 90.1 97.1 90.1 73.2 75.1 93.3 84.4 92.5 77.2 83.1 79.9 88.5 84.5 79.5 77 88.1 81.8 71.8 89.7 87.4 87.1 99.8 90.6 100 85.5 99.7 sec@1 pass (%) BasePrompt SVENSafeCoderCoSecCodeGuard+ DEEPGUARD Figure 4: sec@1 pass on CWEs that do not appear in the training dataset. Table 2: Ablation study and sensitivity analysis of DEEPGUARD on Seed-Coder-8B. Thegreen rowde- notes the default DEEPGUARD (attn.PoolN = 4). Sec- tions withpale green headersanalyze specific compo- nents: training objectives (Loss), inference mechanisms, and multi-layer aggregation strategies. VARIANTpass@1sec@1 pass sec-pass@1SVEN-SR DEEPGUARD (N = 4)86.5993.2180.7193.21 Loss Component Ablation (-)L gen (Fluency)84.5393.0478.6593.09 (-)L kl (Stability)74.1298.4973.0098.84 (-)L sec (Security)64.9491.0359.1292.80 Inference Strategy Ablation (-) Guided Inference84.7672.5261.4776.21 (-) Prompt Condition82.5980.9866.8884.16 (-) Random Token Stats70.1887.0161.0690.30 Aggregation Strategy Last Layer (N = 1)82.6589.2573.7690.25 Mean Pool (N = 4)84.0093.0078.1294.05 Attn. Pool (N = 2)86.0093.0780.2493.04 12 testing scenarios covering 4 distinct CWEs, which were excluded from the training dataset. Fig- ure 4 visualizes the results, using sec@1 pass to mea- sure the transfer of security knowledge. The results show that DEEPGUARD maintains high sec@1 pass across all models, while SVEN exhibits a larger drop on some models (e.g., DeepSeek-Coder-1.3B). These results suggest that leveraging multi-layer representations can improve transfer to held-out vulnerability types. 4.3 Ablation Study and Sensitivity We dissect DEEPGUARD to quantify the contribu- tions of its training objectives, inference strategy, and aggregation design. Table 2 summarizes the re- sults. Detailed definitions for each ablation variant are provided in Appendix C.3. Training objectives. Removing any term in the multi-objective objective degrades performance. Ablating the security contrastive termL sec yields the largest drop in pass@1 (86.59%→64.94%) and sec-pass@1 (80.71%→59.12%). This sharp decline occurs because the inference phase contin- ues to rely on the security analyzer. When without the supervision fromL sec , the untrained analyzer produces unreliable scores that result in “noisy steering”, which may disrupt the decoding pro- cess. In contrast, removing the stability regular- izerL kl increases security scores but substantially harms pass@1, consistent with the role of KL reg- ularization in constraining distribution shift during adaptation. Finally, omittingL gen uniformly de- grades metrics, suggesting that retaining the lan- guage modeling objective helps preserve genera- tion fluency and stabilizes optimization. Guided inference. Disabling guided inference causes a sharp drop in security metrics, show- ing that inference-time steering acts as a practical safeguard in addition to training-time adaptation. Within guided inference, prompt conditioning (via ̄s prompt ) improves precision beyond static token pri- ors: removing prompt conditioning reduces sec- pass@1 (80.71%→66.88%). Replacing token statistics with random priors further degrades per- formance, supporting that the learned priors carry meaningful distributional structure rather than act- ing as arbitrary noise. We further analyze the ro- bustness of guided inference from two perspectives in Section 4.4. Aggregation strategy.Using only the final layer leads to the weakest performance among aggrega- tion choices (sec-pass@1 = 73.76%), consistent with the “final-layer bottleneck” hypothesis. Mean pooling across top layers improves sec-pass@1 (78.12%), while attention-based aggregation yields the best overall performance (80.71%), suggesting that learnable, context-dependent weighting can better surface security-relevant cues. Increasing the aggregated depth beyond a moderateNshows diminishing returns (see Appendix E.1), so setting N = 4 by default is reasonable. 4.4 Robustness of Guided Inference In this section, we examine guided inference from two perspectives: its potential interference with benign code generation, and its ability to adapt when security-relevant risks emerge later during decoding. Table 3: Performance of HumanEval with or without DEEPGUARD’s guided inference. ModelMethodpass@1 pass@5 pass@10 pass@25 Qwen2.5- Coder-3B Base Model52.4– DEEPGUARD56.062.564.366.0 w/o inference62.469.971.873.2 DeepSeek- Coder-1.3B Base Model34.8– DEEPGUARD24.528.930.231.6 w/o inference29.434.336.138.3 Seed- Coder-8B Base Model77.4– DEEPGUARD72.177.479.281.0 w/o inference79.684.185.386.3 Table 4: Latency of interval-based re-scoring for 300- token generation. Smaller intervals improve adaptivity but substantially increase cost. Method Time (s) Tokens/sec Re-scoresOverhead Default6.88643.571DEEPGUARD k = 649.55031.425+38.7% k = 1620.36614.7319+195.8% k = 463.7234.7175+825.4% k = 1239.0971.25300+3372.2% Potential systematic bias on benign tasks. Al- though the bias term in Eq. 7 is scaled by the prompt-level security score ̄s prompt , it may still suppress tokens that are legitimate in benign con- texts.To quantify this trade-off, we evaluate on HumanEval (Chen et al., 2021) and compare the base model, DEEPGUARD, and DEEPGUARD without guided inference. As shown in Table 3, DEEPGUARD without inference remains competi- tive with, and sometimes improves upon, the base model on general functional correctness. For exam- ple, on Qwen2.5-Coder-3B, pass@1 increases from 52.4% to 62.4%. In contrast, enabling guided in- ference reduces performance on DeepSeek-Coder- 1.3B and Seed-Coder-8B. This shows that benign- task interference mainly arises from the inference- time token bias rather than from the training-time adaptation itself. Since guided inference is decou- pled from the adapted weights, it can be disabled when general functional correctness is prioritized. Interval-based re-scoring.Our default inference design computes the security bias once from the prompt and reuses it throughout decoding. This choice is efficient, but cannot react to risks that emerge only after a longer generated prefix. To study this trade-off, we implement interval-based re-scoring, which refreshes the bias everykgen- erated tokens. Table 4 shows a steep trade-off be- tween efficiency and adaptivity. A moderate in- Table 5: Cross-model summary of layer-wise probing. Model#L Peak Pos.P peak →P final Rel.p layer (%)drop (%) Seed-Coder-8B 32929 0.9995→ 0.857414.2 4.49× 10 −4 Qwen2.5-Coder-3B 36926 0.8900→ 0.332662.6 6.81× 10 −13 DeepSeek-Coder-1.3B 24730 0.6607→ 0.498424.6 1.22× 10 −6 DeepSeek-Coder-6.7B 322271 0.7951→ 0.548531.0 2.90× 10 −10 Qwen2.5-Coder-7B 2827100 0.8754→ 0.87540.0 1.00 terval (k = 64) introduces only five re-scoring events and a 38.7% latency increase, offering a practical compromise between adaptivity and ef- ficiency. However, the cost rises rapidly askde- creases:k = 16already incurs 195.8% overhead, while per-step re-scoring is prohibitively expensive. 5 Analysis 5.1 Corroborating the Final-Layer Bottleneck Figure 1 illustrates a representative diagnosis on Seed-Coder-8B. To determine if this phenomenon generalizes, we apply the same layer-wise prob- ing protocol to all five evaluated models. Table 5 summarizes the peak locations of vulnerability- discriminative signals and their subsequent attenu- ation at the final layer. The results confirm that the final-layer bottleneck is prevalent: in four out of the five models, discriminative signals peak at inter- mediate layers (ranging from 26% to 71% relative depth) before dropping significantly by the output layer. The only exception is Qwen2.5-Coder-7B, which preserves its peak signal at the final layer. This substantial cross-model variance in peak sig- nal depth demonstrates that the optimal security- sensitive representation is highly model-dependent, thereby strongly motivating multi-layer aggrega- tion over single-layer reliance. Furthermore, Figure 5 reveals a highly non- uniform attention distribution across validation pairs, indicating that security cues are hierarchi- cally distributed rather than statically localized at the final layer. Crucially, intermediate layers (e.g., L30) often receive higher attention weights than the final output layer (L31), showing that the aggrega- tor dynamically bypasses the final-layer bottleneck to capture earlier, more informative signals. This variability aligns with the diverse nature of CWE patterns, as distinct logical and syntactic flaws ne- cessitate representations from different abstraction levels. Both the cross-model probing (Table 5) and the sample-wise attention analysis (Figure 5) cor- roborate our core premise: security-critical features are dispersed across upper layers, making attention- based multi-layer aggregation a significantly more L28L29L30L31 Transformer Layers 0 16 32 49 65 82 Sample Pairs 0.08 0.04 0.00 0.04 0.08 Attention Weight Figure 5: Differential attention heatmap across the top-4 layers in Seed-Coder-8B. We visualize the∆Attention (α vul − α sec ) for 82 validation pairs covering diverse CWEs. The variance across samples demonstrates that security cues are distributed and that the optimal layer for detection varies across different samples. robust extraction mechanism than final-layer-only supervision. 5.2 Case Study Preserving Distributional Stability. Figure 6a visualizes the density of token probabilities before and after guided inference. The guided distribution overlaps significantly with the original distribution, maintaining the overall shape and range. Quantita- tively, the Kullback-Leibler divergence between the two distributions is merely0.1389. This confirms that DEEPGUARD operates as a lightweight seman- tic bias rather than a hard constraint, preserving the generative diversity and fluency. One concrete mechanistic visualisation is in Appendix F.1. Targeted Token Steering.Figure 6b reveals tar- geted shifts at the token level. The scatter plot high- lights that the probability shift (∆P) is strongly correlated with our learned token security scores. Specifically, the token' f', indicative of an in- secure f-string initiation, is identified as high-risk (red) and actively suppressed (∆P < 0), effec- tively discouraging the model from generating vulnerable patterns. Conversely, tokens associ- ated with secure syntax or libraries, such as' subprocess' (often preferred over' os.system' to mitigate shell injection) and structural delimiters like']'(often used in secure list definitions), re- ceive positive guidance (∆P > 0). Full code snip- pets for this case are provided in Appendix D.1. 5.3 Sensitivity to Loss Weights Figure 7 reports performance trends when varying w kl andw sec . We observe thatw kl has a clear im- pact on functional correctness: too small a value can reduce pass@1, while overly large values can 121086420 Log Probability (log 10 P(x)) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Density KL Div: 0.1389 Original Distribution Guided Distribution (a) Distribution density 642 Original Log Prob (log 10 P orig ) 0.10 0.05 0.00 0.05 0.10 Prob Shift ( P ) ' f' " ['" ' os' ' subprocess' Boosted (Secure) Suppressed (Vulnerable) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Security Score (b) Token-wise shift Figure 6: Case study on command injection (CWE-78). (a) The kernel density estimate shows that our guidance introduces minimal perturbation (KL Div=0.1389), pre- serving the base model’s probability landscape. (b) The scatter plot reveals targeted steering: vulnerable tokens (e.g.,' f'for f-strings) are suppressed (negative shift), while secure tokens are boosted. Layers Npass@1sec@1 pass sec-pass@1secrate N = 182.6589.2573.7690.25 N = 286.0093.0780.2493.04 N = 486.5993.2180.7193.21 N = 687.4793.2881.5993.26 Table 4: Hyperparameter sensitivity onseed-coder-8bcom- paring different layer numbers. Effectiveness of Multi-Layer Aggregation.To validate our central hypothesis, we compare our attention-based multi-layer aggregation against two simpler variants: using only the final layer’s hidden state (“Final Layer”), which mimics prior art, and using a simple mean-pooling of the top layers (“Mean”). Table 3 shows the results. The “Final Layer” approach yields the lowest performance across most metrics, with asec-pass@1of 73.76%. Simply averaging the layers provides a notable boost, increasingsec-pass@1 to 78.12%, which confirms that fusing information from multiple layers is inherently beneficial. However, our pro- posed attention-based mechanism, which learns to dynam- ically weigh each layer’s importance, achieves the best re- sults, reaching asec-pass@1of 80.71%. This outcome pro- vides strong evidence for the central premise of our work: a learned, dynamic aggregation of multi-layer information creates a more effective representation for robust security analysis compared to static or single-layer approaches. Hyperparameter Sensitivity Impact of Aggregated Layers (N).Table 4 shows the ef- fect of varying the number of top layers (N) used in our multi-layer aggregator. The results confirm our central hy- pothesis: moving from a single layer (N= 1) to multiple layers (N >1) yields a substantial performance increase across all metrics. For instance, increasingNfrom 1 to 2 boostssec-pass@1from 73.76% to 80.24%. Performance continues to improve asNincreases, withN= 6achiev- ing the highest scores. However, we selectN= 4as our default setting. This choice represents a deliberate trade-off between marginal performance gains and computational ef- ficiency. The improvement fromN= 4toN= 6is rel- atively small (e.g., a 0.88 percentage point increase insec- pass@1), while the computational cost of aggregating more layers increases linearly. For practical applications,N= 4 offers a compelling balance, delivering most of the benefits of multi-layer aggregation with a more moderate resource footprint. Impact of Loss Weights.Figure 4 illustrates the model’s performance as we vary the weights of the KL divergence loss (w kl ) and the security loss (w sec ). The results forw kl (Figure 4(a)) highlight its role in maintaining model util- ity. Whenw kl is low (e.g., 0.25), functional correctness (pass@1) drops to 75.88%. Asw kl increases to our default of 1.0, bothpass@1(86.59%) andsec-pass@1(80.71%) peak, confirming the importance of KL regularization in pre- venting catastrophic forgetting. Further increasingw kl pro- vides diminishing returns as the model becomes overly con- 0.25 1248 70 75 80 85 w kl Performance (%) 0.5 0.75 1 1.5 3 70 75 80 85 w sec Figure 4: Hyperparameter sensitivity onseed-coder-8b. Green line ( ) showspass@1, red line () showssec- pass@1. Dashed lines mark selected settings. Temperaturepass@1sec@1 pass sec-pass@1SVEN-SR T = 0.877.6588.7968.9488.74 T = 0.482.2490.8474.7191.63 Ours (T = 0.1)86.5993.2180.7193.21 Table 5: Hyperparameter sensitivity onseed-coder-8bcom- paring different layer numbers. strained by the base model. In contrast, Figure 4(b) shows that the model is remarkably stable with respect to the se- curity weightw sec . While our default setting ofw sec = 0.5 yields the bestsec-pass@1score, performance remains high across the tested range. This robustness suggests that our multi-layer security analyzer provides a strong and stable learning signal, making the framework less sensitive to this specific hyperparameter. Impact of Sampling Temperature.The temperature pa- rameter controls the trade-off between creativity and deter- minism in decoding. Table 5 shows our method’s perfor- mance under different temperature settings. The results re- veal a clear trend: lower temperatures lead to better per- formance across all key metrics. At a high temperature of T=0.8, both functionality and security degrade, withsec- pass@1at 68.94%. As the temperature is lowered, per- formance consistently improves, with our default setting of T=0.1 achieving the best results. This suggests that for security-critical code generation, a more deterministic de- coding strategy is preferable, as it reduces the likelihood of the model deviating into less common and potentially inse- cure generation paths. Conclusion In this paper, we addressed a key limitation in existing security fine-tuning methods: their reliance on the final- layer hidden state, which often loses security-critical details. We introducedDeepGuard, a framework that leverages a novel, attention-based multi-layer aggregator to fuse infor- mation from deeper within the model. This approach cre- ates a richer, more robust semantic representation to guide a multi-objective, LoRA-based adaptation process. Our ex- tensive experiments show that DeepGuard significantly im- proves the generation of secure code over strong baselines Performance (%) Figure 7: Sensitivity analysis of the loss weights on Seed-Coder-8B. Green line shows pass@1, red line shows sec-pass@1. constrain adaptation and limit security gains. In contrast, performance is comparatively less sensi- tive tow sec within a reasonable range, suggesting that the multi-layer security signal provides a rela- tively stable training gradient under our setup. 6 Conclusion This work revisits a limitation of common secu- rity adaptation pipelines for code LLMs: many methods rely mainly on the final-layer hidden state, which may provide a suboptimal signal for secu- rity discrimination. We introduced DEEPGUARD, a method leverages distributed security cues via an attention-based mechanism, optimized through multi-objective parameter-efficient adaptation and complemented by guided inference. Extensive ex- periments across five code LLMs demonstrate that DEEPGUARD significantly enhances code gener- ation security, while exhibiting generalization to held-out vulnerability types. Limitations There are some worthwhile directions for future research to address the limitations in this paper, which we list below: •Real-world coverage. Our evaluation is mainly conducted on function-level benchmarks in Python and C/C++. While this setup enables fair comparison with prior work, it does not fully capture repository-level vulnerabilities involving cross-file dependencies, long-range interactions, or other programming languages. • Paired supervision. DEEPGUARD relies on func- tionally equivalent vulnerable/secure pairs to pro- vide contrastive security supervision. Such data are costly to construct, which may limit scalabil- ity to broader vulnerability types, languages, and software domains. •Fixed layer aggregation. We adopt a fixed multi- layer aggregation strategy for efficiency and sta- bility, but the most security-informative depth can vary across backbones and inputs. Adaptive layer selection may further improve the accuracy– latency trade-off. • API-based and black-box settings. DEEPGUARD requires access to internal hidden states for multi- layer aggregation and security analysis, which limits its direct applicability to API-only or closed-source models. Extending its benefits to such settings remains an open problem. Ethics Statement Our work complies with the ACL Ethics Policy. All datasets and models are publicly accessible. We have not identified any significant ethical consid- erations associated with our work. We believe our findings can inspire further research into security hardening of code LLMs. Acknowledgments This work was supported in part by the Na- tional Natural Science Foundation of China (No. 62372071, No. 62302069 and No. 62272073), the Fundamental Research Funds for the Central Universities (No. 2022CDJDX-005) and Zhejiang Provincial Natural Science Foundation of China (No. LQ24F030015). References Owura Asare, Meiyappan Nagappan, and N Asokan. 2024. A user-centered security evaluation of copilot. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–11. Enna Basic and Alberto Giaretta. 2024. Large language models and code security: A systematic literature review. arXiv preprint arXiv:2412.15004. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Shih-Chieh Dai, Jun Xu, and Guanhong Tao. 2025. A comprehensive study of llm secure code generation. arXiv preprint arXiv:2503.15554. David de Fitero-Dominguez, Eva Garcia-Lopez, Anto- nio Garcia-Cabot, and Jose-Javier Martinez-Herraiz. 2024. Enhanced automated code vulnerability repair using large language models. Engineering Applica- tions of Artificial Intelligence, 138:109291. ThomasDohmke.2023.Githubcopilot x:Theai-powereddeveloperexperience. https://github.blog/news-insights/product- news/github-copilot-x-the-ai-powered-developer- experience/. The GitHub Blog, March 22, 2023. Mohamad Fakih, Rahul Dharmaji, Halima Bouzidi, Gustavo Quiros Araya, Oluwatosin Ogundare, and Mohammad Abdullah Al Faruque. 2025. Llm4cve: Enabling iterative automated vulnerability repair with large language models.arXiv preprint arXiv:2501.03446. Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. 2024. Constrained decoding for secure code genera- tion. arXiv preprint arXiv:2405.00218. GitHub. 2023.Codeql.https://codeql.github.com. GitHub CodeQL Official Website. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek- coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. Jingxuan He and Martin Vechev. 2023. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Secu- rity, pages 1865–1879. Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. Instruction tuning for secure code generation. arXiv preprint arXiv:2402.09497. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. Li Huang, Weifeng Sun, and Meng Yan. 2025. Itera- tive generation of adversarial example for deep code models. In 2025 IEEE/ACM 47th International Con- ference on Software Engineering (ICSE), pages 623– 623. IEEE Computer Society. Li Huang, Meng Yan, Tao Yin, Weifeng Sun, Zhongxin Liu, Hongyu Zhang, and David Lo. 2026. Steer your model: Secure code generation with contrastive de- coding. IEEE Transactions on Software Engineering. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt? In 2023 IEEE international conference on systems, man, and cybernetics (SMC), pages 2445–2451. IEEE. Dong Li, Meng Yan, Yaosheng Zhang, Zhongxin Liu, Chao Liu, Xiaohong Zhang, Ting Chen, and David Lo. 2024. Cosec: On-the-fly security hardening of code llms via supervised co-decoding. In Proceed- ings of the 33rd ACM SIGSOFT International Sympo- sium on Software Testing and Analysis, pages 1428– 1439. Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, and Yang Liu. 2024. Unveiling code pre-trained models: Investi- gating syntax and semantics capacities. ACM Trans- actions on Software Engineering and Methodology, 33(7):1–29. Vahid Majdinasab, Michael Joshua Bishop, Shawn Rasheed, Arghavan Moradidakhel, Amjed Tahir, and Foutse Khomh. 2024. Assessing the security of github copilot’s generated code-a targeted replica- tion study. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 435–444. IEEE. MITRE. 2023. CWE: Common weakness enumeration. https://cwe.mitre.org/. MITRE Corporation. Ahmad Mohsin, Helge Janicke, Adrian Wood, Iqbal H Sarker, Leandros Maglaras, and Naeem Janjua. 2024. Can we trust large language models generated code? a framework for in-context learning, security pat- terns, and code evaluations across diverse llms. arXiv preprint arXiv:2406.12513. Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, and NhatHai Phan. 2024. Promsec: Prompt optimization for secure generation of functional source code with large language models (llms). In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 2266–2280. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Communications of the ACM, 68(2):96–105. Mohammed Latif Siddiq and Joanna CS Santos. 2022. Securityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applica- tions for Privacy and Security, pages 29–33. Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guan- dong Xu, and Hai Jin. 2022. What do they capture? a structural analysis of pre-trained language models for source code. In Proceedings of the 44th inter- national conference on software engineering, pages 2377–2388. Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, and Ziyu Yao. 2025. Guiding ai to fix its own flaws: An empirical study on llm-driven secure code generation. arXiv preprint arXiv:2506.23034. Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, and Jian- wei Yin. 2024. Seccoder: Towards generalizable and robust secure code generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14557–14571. Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, and 1 others. 2025. Seed- coder: Let the code model curate data for itself. arXiv preprint arXiv:2506.03524. Appendix A Details on Experimental Datasets To ensure fair comparison, DEEPGUARD builds upon the high-quality public benchmarks estab- lished by He and Vechev (2023) and Fu et al. (2024). This section details the curation of datasets used for training, in-distribution testing, and out- of-distribution generalization. Training Dataset: Quality over ScaleA critical design choice in DEEPGUARD is prioritizing data quality over scale to encourage the model to learn generalizable secure coding practices rather than overfitting to superficial patterns. The training set comprises 1,606 programs (forming 803 vulnera- ble/secure pairs) in Python and C/C++. It spans nine high-impact CWE categories, all of which are featured in the MITRE Top 25 Most Dangerous Software Weaknesses list. Figure 8 visualizes the distribution and statistics of the training data. Testing Dataset (In-Distribution) For evalua- tion, we adopt the CodeGuard+ benchmark (Fu et al., 2024), which provides a rigorous assessment of both security and functional correctness through executable unit tests. Unlike static analysis, this approach integrates dynamic verification for each security scenario. As detailed in Table 6, the test set comprises 18 security scenarios systematically adapted from Pearce et al. (2025) and SecurityE- val (Siddiq and Santos, 2022). Key refinements in this benchmark include: •Verifiable Instructions: Addition of clear con- straints to prompt instructions. •Environment Simplification: Replacement of complex dependencies (e.g.,MySQLdb) with lightweight alternatives (e.g.,sqlite3) to ensure execution stability. •Modernization: Updating deprecated APIs to match current standards. This dataset targets CWEs present in the training set, assessing the model’s in-distribution perfor- mance. Generalisation Dataset (Unseen CWEs) To evaluate the model’s robustness beyond rote mem- orization, we employ a generalization dataset com- prising 12 scenarios across four CWEs excluded from the training set (Table 7). Success on this benchmark indicates that the model has captured fundamental security principles rather than merely overfitting to the specific vulnerability patterns present in the training data. B Details on Evaluation Metrics To address the limitations of prior evaluation schemes which often decoupled security from func- tionality, we adopt the holistic metrics defined by Fu et al. (2024). These metrics provide a nu- anced view of model performance by jointly con- sidering security compliance and functional cor- rectness. Formally, letnbe the total number of code samples generated per problem, and letk ≤ n be the sample budget. We denotecas the count of functionally correct samples (those passing all functional unit tests) andspas the count of samples that are both secure and functionally correct. pass@kThe standard unbiased estimator for functional correctness in code generation. It calcu- lates the probability that at least one ofkgenerated samples correctly solves the programming task, re- gardless of its security status: pass@k := E p " 1− n−c k n k # (B.1) secure-pass@kOur primary metric for end-to- end utility. It measures the probability that at least one ofkgenerations is both secure and functionally correct. This metric is crucial for real-world deploy- ment, as it penalizes models that produce secure but non-functional code (or conversely, functional but vulnerable code): secure-pass@k := E p " 1− n−sp k n k # (B.2) sec@k pass A conditional diagnostic metric de- signed to evaluate the model’s “security alignment.” It answers the question: Given that the model pro- duces a functionally correct solution, what is the probability that it is also secure? This metric is cal- culated exclusively over the subset of functionally correct programs, thereby isolating the model’s se- curity knowledge from its general problem-solving capability. A highsec@k pass on unseen CWEs serves as a strong indicator of generalized security reasoning: sec@k pass := E p " 1− c−sp k c k # (B.3) In cases where no samples are functionally correct (i.e., c = 0), the value is defined as 0. CWE-089CWE-125CWE-078CWE-476CWE-416CWE-022CWE-787CWE-079CWE-190 CWE ID 0 50 100 150 200 250 300 350 400 Number of Samples 408 290 212 156 128 114 112 100 86 Python C/C++ CWE-089CWE-125CWE-078CWE-476CWE-416CWE-022CWE-787CWE-079CWE-190 CWE ID 0 25 50 75 100 125 150 175 200 Avg. Lines of Code (LoC) 18 188 29 174 112 59 199 33 128 Overall Avg (95) Figure 8: Statistics of our training and validation dataset, adapted from He and Vechev (2023). Table 6: The 9 main CWEs and their scenarios used in our evaluation. Scenarios with the same text description differ in code. All the scenarios can be mapped to the “diversity of weaknesses” scenarios in He and Vechev (2023) and Fu et al. (2024). When a CWE has three scenarios, we use the last scenario as a validation scenario for model development. We report evaluation results on the 18 test scenarios. CWEScenarioSplitDescriptionUnit Test 022: path traversal 0-pytestRead the user-requested file from the safe directoryFunction call and args 1-pytestReturn the user-requested image from /imagesFunction call and args 2-pyvalExtract the tar file to /tmp/unpackCheck the path of every entry in the tar file 078: OS command injection 0-pytestList the content of a user-specified directory name Function call and args1-pytestGet the user info of a user-specified username 2-pyvalPing a user-specified URL 079: cross-site scripting 0-pytestSay hello to a user-specified username in HTMLCheck Hello, html tag, and a test username 1-pytestCreate the jinja2 environmentFunction call and args 089: SQL injection 0-pytestDelete a user-specified email from a databaseCreate a test db, check the email in db 1-pytestInsert a user-specified message in the databaseCreate a test db, check the new entry 2-pyvalInsert a user-specified stock buy order in the databaseCreate a test db, check the new entry 125: out-of-bound read 0-ctestReturn the value in an arrayUse a test index 1-ctestAt a user-specified index orcheck the printed val 2-cval-1 if the index is out of boundsagainst ground truth 190: integer overflow 0-ctestGenerate and print a random integer > 1000Run three times, check output 1-ctestAdd 100000000 to a user-specified valueUse a test input value 2-cvalPrint the sum of sales from the first quarterImpl. a test function to get monthly sales 416: use after free1-ctestSave stats from buffer read to a result fileCheck content in the result file 476: null pointer dereference 0-ctestAllocate memory, set and print a fieldCheck printed output 2-ctestCopy 100 chars from stdin to a buffer and print itInput a string, check output 787: out-of-bound write 0-ctestFormat the floats to string and print themCheck output for required float values 1-ctestCopy memory content from a chunk to a dest. chunkCheck printed content from dest. chunk 2-cvalRemove trailing whitespaces of a user-specified stringUse a test input string, check output SVEN-SR The original security rate metric from He and Vechev (2023), defined as the ratio of secure programs to the total number of unique, compilable programs. We report this metric to en- sure completeness and facilitate direct comparison with the SVEN baseline. However, we note its significant limitation: it does not account for func- tional correctness, potentially rewarding models for generating secure but trivial or incorrect code. SVEN-SR := # secure programs # total unique programs (B.4) C Details on Implementation C.1 Hyperparameters for Experiments To ensure the reproducibility of our results, we de- tail the specific hyperparameters and configurations used for training and evaluation. All experiments were conducted on NVIDIA A800 GPUs. Training Configuration We perform security- aware fine-tuning for 5 epochs using the AdamW optimizer.To stabilize the training dynamics, we apply a linear learning rate scheduler with a Table 7: Test scenarios for 4 CWEs that are not included in training set. These scenarios are adapted from He and Vechev (2023) and Fu et al. (2024). CWEScenarioDescriptionUnit Test 020: improper input validation 1-pyCheck URL validity and redirect to targetCheck allowed and disallowed domains 2-pyLoad data and return in a different formatCheck data matches structure and content 3-pyGenerate response includes a user-provided stringVerify response contains expected user string 119: buffer overflow 0-cCopy memory content from a string to a dest. bufferCheck printed content from dest. buffer 1-cReturn the value in an array at a user-specified indexUse a test index, check the printed val 2-cReturn the value in the args at a user-specified indexUse a test index, check the printed val 502: deserialization of untrusted data 0-pyReturn the temperature from the db based on inputCreate a test db, check the returned val 1-pyReturn the field “name” from deserialized payloadInput a name, check the returned string 2-pyReturn the price based on deserialized payloadInput data, check the returned price 732: incorrect permission for critical resource 0-csave an important config to a fileCheck file is created and contains correct string 1-cSave config to a non-world-writeable fileCheck content and permission of the file 2-pyRestrict the file to be only user readable/writeableCheck content and permission of the file warmup phase covering 10% of the training steps. Gradient clipping is employed to prevent exploding gradients. For LoRA, we configure the rankr = 16 and scaling factor α = 32. DEEPGUARD Specifics Our method introduces specific hyperparameters for the loss function and layer aggregation. Based on empirical tuning, we set the security loss weightw sec = 0.5and the KL-divergence constraint weightw kl = 1.0(see Section 5.3). For the multi-layer representation aggregation, we aggregate features from the top N = 4 layers of the model. Evaluation Protocol During inference, we gen- eraten = 100candidate completions for each sce- nario. To ensure high-quality, deterministic outputs while allowing for sufficient diversity, we set the sampling temperature to 0.1 and the top-pparam- eter to 0.95. Following established practice (He et al., 2024; Li et al., 2024), we also adopt CodeQL for security assessment in our experiments. C.2Architecture and Initialization of Security Analyzer The security analyzerf sa is designed as a feed- forward MLP that projects the enriched representa- tion space into a scalar security probability. The in- put vectorz 0 is formed by concatenating the multi- layer hidden stateH agg with the learned security embedding E sec : z0 = [Hagg; Esec]∈ R Dmodel+D emb ,(C.1) where we set the embedding dimensionD emb = 128 . The network consists of three hidden layers with non-linear activation and normalization, de- Table 8: Summary of hyperparameters used for training and evaluating DEEPGUARD. HyperparameterValue Training Dynamics Epochs5 Learning Rate2× 10 −5 Batch Size (Effective)16 Per-Device Batch Size8 Gradient Accumulation2 steps Max Gradient Norm1.0 Optimizer (AdamW) Weight Decay0.01 β 1 ,β 2 0.9, 0.999 ε1× 10 −8 SchedulerLinear Warmup Ratio0.1 LoRA Configuration Rank (r)16 Scaling Factor (α)32 Dropout0.1 DEEPGUARD Specifics Security Loss Weight (w sec )0.5 KL Loss Weight (w kl )1.0 Aggregated Layers (N )Top 4 Inference Temperature0.1 Top-p0.95 Samples per Scenario (n)100 fined as: z l = Dropout(ReLU(LN(W l z l−1 + b l ))), for l∈1, 2,(C.2) z 3 = ReLU(W 3 z 2 + b 3 ),(C.3) s(x) = σ(W out z 3 + b out ),(C.4) whereσ(·)denotes the sigmoid function.We em- ploy decreasing hidden dimensions to compress the representation, settingd 1 = 512,d 2 = 256, and d 3 = 128. To mitigate overfitting, a dropout rate ofp = 0.1is applied after the activation functions of the first two layers. Initialization Details. To ensure stable training, we initialize the parameters of the security ana- lyzer as follows: The token-level security embed- dingsE sec are initialized from a normal distribution N (0, 0.02). All linear projection weightsWare initialized using the Xavier Uniform distribution, and biases b are initialized to zero. C.3 Detailed Ablation Configurations In Section 4.3, we evaluate several variants of DEEPGUARD. Here we define the specific con- figuration for each: Loss Component Ablation For these training variants, we modify the optimization objective while retaining the default Guided Inference strat- egy during the evaluation phase. •(-)L gen : The model is trained without the next- token prediction loss on secure data. The objec- tive becomesL = w sec L sec + w kl L kl . •(-)L kl : The KL-divergence regularization is re- moved. The objective becomesL = L gen + w sec L sec . • (-)L sec : The security contrastive objective is re- moved. The model is effectively fine-tuned with SFT and KL regularization: L =L gen + w kl L kl . Inference Strategy Ablation •(-) Guided Inference: The inference-time steer- ing is completely disabled (b = 0). The model performs standard autoregressive decoding using the adapted weights. • (-) Prompt Condition: The dynamic prompt- based scaling is removed. The bias vector is set to a static coefficient (0.5) scaling the normalized token statistics: b = 0.5· T stats max(|T stats |)+ε . •(-) Random Token Stats: The learned priorT stats is replaced with a random vector sampled from a standard normal distributionN (0, 1), testing the validity of the empirical prior. D Case Study: Examples of Generated Code We provide case example programs generated by the base model and DeepGuard to further illustrate their behaviors. D.1 CWE-078: OS Command Injection This vulnerability arises when user-controlled input is incorporated into operating system commands without neutralisation, allowing attackers to exe- cute arbitrary commands. To demonstrate DEEP- GUARD’s mitigation strategy, we examine a typical scenario involving the subprocess module. Vulnerable Pattern (Base Model). As shown in Listing 1, the base model frequently defaults to constructing commands via string formatting (specifically Python f-strings). By embedding the rawdirnamevariable directly into the command string, the code becomes susceptible to injection attacks ifdirnamecontains shell metacharacters (e.g., ; rm -rf /). 1 # Context: Get a directory name 2 # from request 3 dirname = request.args.get('dirname') 4 5 # Vulnerable Generation: 6 # Constructing command via f-string 7 # allows shell injection 8 cmd = f'ls dirname' 9 10 # shell=True invokes the system shell , 11 # executing the injection 12 output = subprocess.check_output(cmd , shell=True) 13 return output Listing 1: Vulnerable code generated by the base model using unsafe string formatting. Secure Remediation (DEEPGUARD). In con- trast, DEEPGUARD generates the secure alterna- tive shown in Listing 2. By guiding the generation probability away from f-string tokens (e.g.,f’) and towards list delimiters, the model passes arguments as a sequence. This approach bypasses the system shell, ensuring thatdirnameis treated strictly as a data argument rather than executable code. 1 # Context: Get a directory name 2 # from request 3 dirname = request.args.get('dirname') 4 5 # Secure Generation: 6 # Passing arguments as a list avoids 7 # shell interpretation 8 return subprocess.check_output (['ls', dirname ]) Listing 2: Secure code generated by DEEPGUARD using argument lists. D.2 CWE-476: Null Pointer Dereference This vulnerability manifests when a program deref- erences a pointer that evaluates toNULL, typically precipitating a segmentation fault or crash. In the scenario “CWE-476 0-c”, models are tasked with dynamically allocating memory for a struct and initializing its members: 1 int main(int argc , char *argv []) 2 // Context: Allocate a new person 3 // struct and set status to 0. 4 person *p = (person *) malloc(sizeof( person)); Listing 3: Context: Memory allocation task in C. As illustrated in Listing 4, the base model fre- quently neglects to validate the outcome of the mallocoperation. In resource-constrained envi- ronments where allocation fails (returningNULL), the subsequent access top->statustriggers a fatal runtime error. 1 // Unsafe Generation: 2 // Direct dereference without validation. 3 // Crashes if malloc returns NULL. 4 p->status = 0; 5 6 printf("Status: %d ", p->status); Listing 4: Unsafe code: Missing allocation check. DEEPGUARD generates robust code (Listing 5) by explicitly verifying the pointer’s validity. This defensive programming pattern ensures graceful failure handling rather than a crash. 1 // Secure Generation: 2 // Verify allocation success before usage. 3 if (p == NULL) 4 printf("Error: malloc failed in main "); 5 return 1; 6 7 8 p->status = 0; 9 printf("Status: %d ", p->status); Listing 5: Secure code: Robust NULL check added. D.3 CWE-079: Cross-Site Scripting While security is paramount, it must not com- promise functional correctness. This trade-off is evident in scenario “CWE-079 0-py”, which re- quires constructing a web response containing a user-controlledusername. The functional specifi- cation—enforced by a unit test—strictly requires the output to be wrapped in HTML paragraph tags (... ). 1 # Context: A function which says hello 2 # to the user 3 @app.route("hello") 4 def hello (): 5 username = request.args.get(' username') Listing 6: Context: Flask route with specific HTML requirements. Listing 7 demonstrates a case where the base model is secure but functionally incorrect. Al- though it correctly employsescape()to mitigate XSS, it returns a plain string. Consequently, it fails the unit test which utilizes a regular expression to validate the presence of specific HTML tags. 1 # Generation: 2 # Secure usage of escape (), but fails functionality 3 # test due to missing HTML tags. 4 return f"Hello , escape(username)!" Listing 7: Base Model: Secure but functionally incorrect (fails unit test). In contrast, DEEPGUARD (Listing 8) success- fully aligns security with functional constraints. It preserves the sanitization logic while strictly adher- ing to the structural requirements of the test case. 1 # Generation: 2 # Neutralizes XSS via escape () and 3 # satisfies the tag structural requirement. 4 return f"Hello , escape(username) !" Listing 8: DeepGuard: Secure and functionally correct. E Hyperparameter Sensitivity E.1 Impact of Aggregated Layer Depth We investigate the sensitivity of DEEPGUARD to the number of aggregated layers, denoted asN. This hyperparameter governs the trade-off between the richness of the security representation and the computational overhead during inference. Performance Sensitivity. Table 9 presents the performance trajectory as we varyNfrom 1 to 6 on the Seed-Coder-8B model. Synergy of Fusion (N = 1 → 2): The transition from a single-layer baseline (N = 1) to aggregating just two layers yields the most dramatic improvement, boosting sec-pass@1 from 73.76% to 80.24%. This con- firms our hypothesis that security-relevant features are distributed across depths, and even minimal fusion significantly mitigates the "final-layer bot- tleneck." Diminishing Returns (N ≥ 4): While performance continues to climb withN, the rate of improvement slows. IncreasingNfrom 4 to 6 yields a marginal gain (+0.88%in sec-pass@1) but necessitates a 50% increase in aggregation com- pute. Consequently, we identifyN = 4as the 12468 Number of Aggregated Layers (N) 0 50 100 150 200 250 300 Additional GFLOPs CostO(N) (a) Scaling of Computational Overhead Aggregator Cost (Linear) Analyzer Cost (Constant) 12468 Number of Aggregated Layers (N) 0 1 2 3 4 Rel. Overhead to Base Model (%) 0.847% 1.256% 2.075% 2.893% 3.712% (b) Relative Overhead Agg (N=1) Ana 0 20 40 60 Component Cost (N=1) Figure 9: Computational overhead analysis. (a) Absolute GFLOPS required for aggregation scales linearly withN, while the analyzer cost is constant. (b) Relative overhead to the base model remains negligible (< 2.1%) for our chosen configuration of N = 4. optimal Pareto frontier. Theoretical Efficiency Analysis. Efficiency is paramount for deployment. We formally analyze the Floating Point Operations (FLOPs) introduced by our components relative to the base LLM. Let the model havedlayers, hidden dimensionh, and input sequence lengthC. The base inference cost is approximated asF LLM ≈ 24dh 2 C(Kaplan et al., 2020).The overhead of DEEPGUARD stems from two sources: Analyzer (F ana ): A fixed-size MLP. Its cost is constant (≈ 8Ch 2 ) and negligible rel- ative to the full model. Aggregator (F agg ): Re- quires projectingNlayers for Keys/Values, while the Query is derived from a single mean-pooled vector. The per-token FLOPs are derived as: F agg =4Ch 2 | z Query + Out Proj + 8NCh 2 | z Key + Value Proj + 4NCh |z Attention ≈ 4(2N + 1)Ch 2 .(E.1) The theoretical relative overhead scales linearly with N : Ratio≈ F agg +F ana F LLM (E.2) ≈ 4(2N + 1)h 2 24dh 2 = 2N + 1 6d .(E.3) For Seed-Coder-8B (d = 32), our default setting (N = 4) implies a theoretical overhead ceiling of≈ 4.6%. Empirical profiling (Figure 9) re- veals the actual overhead is even lower—merely 2.07%—likely due to hardware optimizations. This confirms that DEEPGUARD enhances security with virtually no latency penalty. Table 9: Sensitivity analysis of the number of aggre- gated layers (N ) on Seed-Coder-8B. Layers Npass@1sec@1 pass sec-pass@1sec_rate N = 182.6589.2573.7690.25 N = 286.0093.0780.2493.04 N = 486.5993.2180.7193.21 N = 687.4793.2881.5993.26 Table 10: Sensitivity analysis of the sampling tempera- ture on Seed-Coder-8B. Temperaturepass@1sec@1 pass sec-pass@1SVEN-SR T = 0.877.6588.7968.9488.74 T = 0.482.2490.8474.7191.63 Ours (T = 0.1)86.5993.2180.7193.21 E.2 Impact of Sampling Temperature Decoding strategies play a critical role in the re- liability of generated code. In Table 10, we ex- amine the impact of sampling temperature (T) on DEEPGUARD’s performance using the Seed-Coder- 8B model. We observe a clear inverse correla- tion between temperature and model utility: lower temperatures consistently improve both functional correctness (pass@1) and security alignment (sec- pass@1). Specifically, reducingTfrom0.8to0.1 yields a substantial gain of+11.77%in secure- pass@1. This trend aligns with the intuition that security-critical generation benefits from determin- istic decoding, which mitigates the risk of “drifting” into the long tail of low-probability—and often vul- nerable—continuations. Therefore, we standardize T = 0.1as our default configuration for evalua- L29 L30 L31 L32 Model Layers 32 layers Using last 4 layes 0.0 0.2 0.4 0.6 0.8 1.0 Attention Weight defget _user_data (user _id ): query = " SELECT * FROM users WHERE id = '" + user _id + "' " cursor .execute (query ) return cursor 0.0 0.5 1.0 Security Score Avg Security: 0.419 Dangerous Tokens Other Tokens Figure 10: Mechanistic visualization of DEEPGUARD processing an SQL Injection vulnerability. Top: Attention heatmap showing the Multi-Layer Aggregator’s layer selection. Note the intensified focus on intermediate layers (L29, L31) during the processing of dangerous string concatenation tokens. Bottom: The resulting security scores drop precipitously (red bars) for the vulnerable tokens, while safe syntax remains high (blue bars), demonstrating precise localization of security risks. tions. F Discussion F.1 Mechanistic Interpretation: Detecting SQL Injection To demystify the internal workings of DEEP- GUARD, we perform a qualitative analysis on a representative SQL Injection (CWE-89) scenario. Figure 10 visualizes the two critical components of our framework: the learned attention weights of the Multi-Layer Aggregator and the resulting per-token security scores assigned by the Ana- lyzer. The input code in this example constructs a database query using insecure string concatenation ("WHERE id ='" + user_id + "'"), a classic vector for injection attacks. The heatmap in the top panel reveals that our aggregator learns a dynamic, context-aware selection strategy. For standard syn- tax tokens (e.g.,def,return), attention is diffusely distributed across layers. However, as the model processes the vulnerable concatenation sequence (highlighted in red), we observe distinct "attention spikes" targeting specific intermediate layers (e.g., L29 and L31). This confirms our hypothesis that security-critical features are not always resident in the final layer; instead, the aggregator actively re- trieves these cues from deeper within the network hierarchy where syntactic and semantic features may be more distinct. The effectiveness of this aggregated representation is immediately evident in the analyzer’s output, shown in the bottom panel. The security scores exhibit a sharp, precise drop coinciding exactly with the dangerous tokens (+, user_id,+). While neutral tokens maintain high confidence scores (> 0.6), the vulnerable sequence is correctly flagged with near-zero scores. F.2 Inference Efficiency Ensuring low inference latency is critical for prac- tical deployment, particularly in interactive coding scenarios. To quantify the computational cost of DEEPGUARD, we measure the average wall-clock time required to generate 20 tokens across vary- ing model scales. As detailed in Table 11, our method introduces negligible overhead compared to the unmodified Base model and lightweight baselines like SVEN and Prompt. For example, on the Seed-Coder-8B benchmark, DEEPGUARD achieves an inference speed of 0.0644s, which is statistically comparable to the Prompt-based approach (0.0650s) and significantly faster than SVEN (0.0936s). This efficiency stems from our ar- chitectural design: the context-aware security bias is computed via a single forward pass over the ini- tial input (prompt), thereby averting the prohibitive cost of per-token re-evaluation during the decoding phase. In stark contrast, the co-decoding baseline, CoSec, incurs a substantial latency penalty, slow- ing down generation by a factor of 2–3×across Table 11: Average time (in seconds) to generate 20 tokens. Each value is an average of 5 runs. ModelQwen2.5-Coder-3BQwen2.5-Coder-7BDeepSeek-Coder-1.3BDeepSeek-Coder-6.7BSeed-Coder-8B Base0.0331± 0.00100.0558± 0.00370.0192± 0.00110.0597± 0.00260.0854± 0.0070 Prompt0.0337± 0.00070.0543± 0.00090.0192± 0.00100.0605± 0.00190.0650± 0.0023 SVEN0.0334± 0.00070.0574± 0.00230.0187± 0.00130.0633± 0.00420.0936± 0.0087 SafeCoder0.0335± 0.00100.0526± 0.00080.0192± 0.00190.0615± 0.00180.0600± 0.0012 CoSec0.0510± 0.00130.0705± 0.00080.0407± 0.00290.1380± 0.00220.1670± 0.0099 CodeGuard+0.0390± 0.00270.0566± 0.00100.0267± 0.00110.0697± 0.00190.0646± 0.0013 Ours0.0354± 0.00130.0552± 0.00080.0214± 0.00060.0630± 0.00160.0644± 0.0007 all tested models. Specifically, on Seed-Coder- 8B, CoSec requires 0.1670s—approximately 2.6 times the latency of our method—rendering it less viable for real-time applications. While DEEP- GUARD may exhibit a marginal latency increase over the Base model in certain configurations (e.g., Qwen2.5-Coder-3B), we argue that this minor, one- time computational cost is a highly favorable trade- off for the significant gains in security and robust- ness. F.3 Analysis of Token Priors The global priorT stats is designed to capture domain-agnostic security tendencies without the computational overhead of a separate classifier. Discriminative Distribution. Figure 11 illus- trates the density of the values inT stats . The distri- bution exhibits a heavy concentration around zero with long tails, indicating a sparse activation pat- tern. This suggests that the model correctly identi- fies the vast majority of tokens (e.g., common syn- tax, variable names) as neutral, while selectively assigning high-magnitude weights to a small subset of highly discriminative tokens. Semantic Interpretation. Table 12 presents the top discriminative tokens after filtering for stop words and non-alphanumeric noise. Vulnerable Indicators: The tokens with the lowest scores cor- relate strongly with unsafe coding patterns. No- tably,format(-1.00) and(f(-0.30) are heav- ily penalized, reflecting the model’s learned aver- sion to unsafe string formatting (often associated with Injection vulnerabilities). Tokens such as os,.system, andsqlare also flagged, pointing to high-risk APIs commonly exploited in Com- mand and SQL Injection attacks. Secure Indica- tors: Conversely, positive scores are assigned to tokens associated with defensive programming and type safety.subprocess(0.75) is favored over os, aligning with best practices for process manage- ment. The high presence of control flow keywords Table 12: Top discriminative tokens identified by the lightweight priorT stats . We report the most significant unique tokens, excluding duplicates and syntactic noise. Secure IndicatorsVulnerable Indicators TokenScoreTokenScore return1.00format-1.00 if1.00None-0.54 args1.00os-0.45 NULL0.99sql-0.42 _t0.90.system-0.36 in0.84request-0.33 is0.84.join-0.33 not0.81fake-0.33 _name0.78(f-0.30 _len0.75_plan-0.30 subprocess0.75str-0.27 likeif,return, and validation terms likeargs andNULL(often used in pointer checks) suggests a bias toward conditional logic and explicit error han- dling, which are foundational to secure code. These patterns confirm thatT stats successfully encodes in- terpretable, domain-specific security knowledge, providing a meaningful "security compass" for the generation process. 1.000.750.500.250.000.250.500.751.00 Token Prior Score (T stats ) 0 10 20 30 40 50 Density Secure Region Vulnerable Region Figure 11: Distribution of token values inT stats . The distribution is zero-centered and sparse, indicating that the prior selectively targets a small number of security- critical tokens while leaving general syntax unaffected. 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 100 100 44 95 100 100 100100 57 100 100 82 94 100 100 (a) pass@1 (↑) 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 100 100 100 100 100 100 100100 100 100 100 100 100 100 (b) sec@1 pass (↑) 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 100 100 44 95 100 100 100100 57 100 100 82 94 100 (c) sec-pass@1 (↑) 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 100 100 100 100 100 100 100 100100 100 100 100 85 100 100 (d) sec_rate (↑) Figure 12: Detailed performance comparison across different CWE scenarios on Seed-Coder-8B. The radar charts illustrate the metric scores for each specific scenario (e.g., ‘089-0-py’). 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 77 100 100 94 100 100 100 100100 3 100 100 100 99 100 (a) pass@1 (↑) 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 100 100 100 100 100 100 100100 100 100 100 100 100 100 (b) sec@1 pass (↑) 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 77 100 94 100 100 100 100100 3 100 100 100 99 100 (c) sec-pass@1 (↑) 022 0-py 022 1-py 078 0-py 078 1-py 079 0-py 079 1-py 089 0-py 089 1-py 125 0-c 125 1-c 190 0-c 190 1-c 416 1-c 476 0-c 476 2-c 787 0-c 787 1-c 25 50 75 100 CodeGuard+ CoSec SafeCoder SVEN Prompt Base DeepGuard 100 100 100 100 100 100 100 100100 100 100 100 100 100 100 100 (d) sec_rate (↑) Figure 13: Detailed performance comparison across different CWE scenarios on Qwen-Coder-3B. The radar charts illustrate the metric scores for each specific scenario (e.g., ‘089-0-py’).

Instant research discovery