Paper deep dive
Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations
Alizishaan Khatri
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/19/2026, 3:57:44 AM
Summary
This paper investigates whether the internal latent activations of Large Language Models (LLMs) during the prefill phase contain recoverable signals about code vulnerability. The authors extract last-prefill-token activations from four LLMs (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) and train lightweight Multi-Layer Perceptron (MLP) probes to predict vulnerability labels. Evaluated on four C/C++ benchmarks (Devign, Big-Vul, Draper VDISC, PrimeVul), the probes achieved an average F1 of 41.7%. Notably, the best probe (Qwen3.5-9B) matched the State-of-the-Art (SOTA) on Devign (68.8% F1 vs 67.9% SOTA) despite using only frozen model activations. The study suggests that model-native, lightweight screening is feasible, offering significant latency and parameter efficiency benefits over post-hoc external detectors.
Entities (11)
Relation Signals (8)
Qwen3.5-9B → achievedbestperformanceon → Devign
confidence 95% · On Devign, the best probe (Qwen3.5-9B, 68.8% F1) matches the published fine-tuned-classifier SOTA
Qwen3.5-9B → hasprobef1on → 68.8%
confidence 95% · On Devign, the best probe (Qwen3.5-9B, 68.8% F1)
MLP Probe → trainedonactivationsof → Qwen3.5-9B
confidence 95% · We extract last prefill token activations from four LLMs... and train MLP probes on these activations.
MLP Probe → trainedonactivationsof → Qwen3.6-27B
confidence 95% · We extract last prefill token activations from four LLMs... and train MLP probes on these activations.
MLP Probe → trainedonactivationsof → Gemma-4-12B
confidence 95% · We extract last prefill token activations from four LLMs... and train MLP probes on these activations.
MLP Probe → trainedonactivationsof → Granite-4.1-8B
confidence 95% · We extract last prefill token activations from four LLMs... and train MLP probes on these activations.
CWE-119 → hardestfor → MLP Probe
confidence 90% · CWE-119 the single hardest category for every model
Devign → hassotaf1 → 67.9%
confidence 90% · matches the published fine-tuned-classifier SOTA (67.9%)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-based code generation is now embedded in mission-critical pipelines, but defenses against vulnerable output remain post-hoc -- static analyzers, fine-tuned classifiers, or an LLM judge that screen completed code, ignoring the generating model's own internal state. We test a narrower, directly measurable question: when an LLM reads a piece of C/C++ code as context, do its hidden activations already carry a signal about that code's vulnerability status? We extract last prefill token activations from four LLMs (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) across three model families and train MLP probes on these activations. We evaluate them on four function-level C/C++ benchmarks (Devign, Big-Vul, Draper VDISC, PrimeVul). Our probes achieve 41.7\% average F1 using 13.4--16.0M-parameter probes -- under 0.2\% of base-model size. On Devign, the best probe (Qwen3.5-9B, 68.8\% F1) matches the published fine-tuned-classifier SOTA (67.9\%) despite reading only a frozen, general-purpose LLM's activations; on the harder, more imbalanced benchmarks (Big-Vul, Draper VDISC, PrimeVul) probes trail SOTA substantially. This is early evidence that a coding LLM's own representation of arbitrary code is informative about that code's vulnerability status, motivating further work toward lightweight, model-native vulnerability screening.
Tags
Links
- Source: https://arxiv.org/abs/2608.16970v1
- Canonical: https://arxiv.org/abs/2608.16970v1
Trouble viewing inline? Open PDF directly →
Full Text
35,772 characters extracted from source content.
Expand or collapse full text
Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations Alizishaan Khatri Affiliation: Wrynx Inc. research@wrynx.com Abstract LLM-based code generation is now embedded in mission-critical pipelines, but defenses against vulnerable output remain post-hoc – static analyzers, fine-tuned classifiers, or an LLM judge that screen completed code, ignoring the generating model’s own internal state. We test a narrower, directly measurable question: when an LLM reads a piece of C/C++ code as context, do its hidden activations already carry a signal, recoverable easily, about that code’s vulnerability status? We extract last-prefill-token activations from four LLMs (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) across three model families and train MLP probes on these activations, evaluated on four function-level C/C++ benchmarks (Devign, Big-Vul, Draper VDISC, PrimeVul) using each dataset’s existing, human/CVE-labeled functions as the probed context – not model-generated completions, which we have not yet tested (§IV). So this is direct evidence for activation-level informativeness and only indirect motivation for the stronger claim that a model could flag its own forthcoming output as vulnerable. Our probes achieve 41.7% average F1 using 13.4–16.0M-parameter probes – under 0.2% of base-model size. On Devign, the best probe (Qwen3.5-9B, 68.8% F1) matches the published fine-tuned-classifier SOTA (67.9%) despite reading only a frozen, general-purpose LLM’s activations; on the harder, more imbalanced benchmarks (Big-Vul, Draper VDISC, PrimeVul) probes trail SOTA substantially, tracking those datasets’ known label-imbalance and quality problems rather than an absence of signal. This is early evidence that a coding LLM’s own representation of arbitrary code is informative about that code’s vulnerability status, motivating further work toward lightweight, model-native vulnerability screening. Index Terms: large language models, code generation, vulnerability detection, activation probing, interpretability, secure software supply chain, representation probing I Introduction LLM-based coding assistants and autonomous coding agents are now used to generate production code, including in defense and safety-critical pipelines [17] that demand strong assurance against vulnerable output [27], across applications such as autonomous vehicles [20] and self-organizing robot swarms [33]. As with harmful-content generation, the dominant defenses against vulnerable code generation operate at the interface – static analyzers, learned classifiers, or a second LLM run over the completed code – and share the structural weaknesses common to any post-hoc, black-box screen (§I). A parallel line of interpretability work argues this asymmetry is unnecessary: LLMs already encode substantial task-relevant semantics, including whether a prompt is harmful, as linearly separable directions in their hidden states [31, 13, 29, 15, 14] (§I). We ask: When an LLM reads a piece of code as context, does its internal state already encode a recoverable signal about that code’s vulnerability status, consistently enough across model families to motivate testing it as a generation-time gate? We test this on existing, human/CVE-labeled corpus functions, not model-generated completions (§IV) – a necessary but not sufficient step toward a model flagging its own forthcoming vulnerable output, which we do not test here. This is a preliminary, workshop-scale study; §VII enumerates the controls (an architecture ablation, a non-LLM baseline, an adversarial-robustness check) still needed to fully support the claims below. Contributions. • We extend activation probing for LLM safety ([15, 26] and related probing work [11, 29]) to code vulnerability detection, rather than prompt-harm classification, where code appeared only incidentally as a minority of Aegis-dataset samples [8] (§I). • We extract last-prefill-token activations from four LLMs (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) across three model families and train MLP probes to predict the vulnerability label of the code in context, evaluated on four function-level C/C++ vulnerability benchmarks – Devign, Big-Vul, Draper VDISC, and PrimeVul (§IV). • We report precision, recall, and F1 for every (model, dataset) pair (§V). • For all four probes on Big-Vul, we break down classification accuracy by CWE type and find it uniformly high and tightly clustered across the five most frequent types, with CWE-119 the single hardest category for every model (§V). • We discuss the latency/compute case for probe-based generation-time gating over post-hoc baselines, and known generalization failure modes of probing-based detectors [30, 6] (§VII). I Related Work I-A Post-Hoc Vulnerability Detection Most production screening for LLM-generated code runs static/dynamic analyzers over the output or fine-tunes a classifier, often another LLM, on labeled code [28]. Such detectors are post-hoc, black-box to the generating model’s own computation, and evadable by surface transformations that preserve the underlying vulnerability [16]. Steenhoek et al. [28] show via attention analysis that detection models already encode bug-relevant semantics, though not as a linear probe. We probe that internal state directly via the model’s own forward pass, rather than analyzing emitted code after the fact. I-B Probing LLM Internals for Vulnerability Detection LPASS [11] places linear probes after every layer of a compressed LLM to guide pruning, for compression, not screening. Sternfeld et al. [29], our closest prior work, train logistic-regression/MLP probes on prompt-end hidden states to predict whether a yet-to-be-generated completion will be secure, showing minimal prompt perturbations can flip it. We differ: deeper 6-layer MLP probes; activation over the candidate code itself from four dedicated corpora, not a prompt paired with a generated completion; and an emphasis on cross-model (4 LLMs, 3 families) detection rather than perturbation sensitivity. I-C Latent-State Probing Beyond Code Latent-state probing has also targeted malicious prompts, jailbreaks, and malware. Chia et al. [2] separate safe from jailbroken states in latent subspaces; Fomin [6] probes three LLMs under a leave-one-dataset-out protocol; Wang et al. [30] show such probes generalize poorly out-of-distribution (§VI-C); Ajayi et al. [1] apply VAE latent spaces to malware detection outside the LLM setting. I Problem Formulation Let ℳM be an LLM used for code generation and :→0,1V:O→\0,1\ a ground-truth vulnerability oracle over completions o=ℳ(x)o=M(x) for coding prompt x, such that (o)=1V(o)=1 iff o contains an instance of one or more target vulnerability classes (e.g., a CWE, as determined by dataset ground truth. We define the last-prefill-token activation h(x)=ℳhidden(x)[−1]∈ℝdh(x)=M_hidden(x)[-1] ^d as the final-layer hidden state at the last prompt token, immediately before generation begins. We seek a probe fθ:ℝd→0,1f_θ:R^d→\0,1\, trained on frozen activations h(x)h(x), that predicts (ℳ(x))V(M(x)) before ℳ(x)M(x) is generated – unlike post-hoc detectors, which operate on o directly, and unlike the harmful-prompt setting of [15], where the probe predicts a property of x itself. This is the target formulation; none of our four evaluation datasets pairs a prompt x with a model-generated completion o=ℳ(x)o=M(x), so §IV operationalizes it by substituting an existing, dataset-labeled function c for x and reading (c)V(c) directly off the dataset – a proxy for, not a direct test of, the (ℳ(x))V(M(x)) formulation above. Fig. 1 and §IV-E detail the resulting extraction and training pipeline; the practical objective is a probe cheap enough to run inline with generation (Tprobe≪TllmT_probe T_llm) that generalizes across datasets and model families rather than overfitting to one model’s idiosyncratic representation of vulnerability. IV Methodology IV-A Overview Our pipeline mirrors the two-stage design of [15]: (1) an extraction stage that runs each coding prompt through a frozen LLM and stores the last-prefill-token activation, and (2) a training stage that fits an MLP probe on those frozen activations to predict vulnerability of the resulting completion. Stage 1: Extraction Stage 2: TrainingCandidate function c(from dataset)Frozen target LLM ℳMActivation h(c)∈ℝdh(c) ^d(last-prefill token)Cached pairs (hi,yi)(h_i,y_i)forward passpaired w/ label yyMLP probe fθf_θ(trainable)Prediction y^∈0,1 y∈\0,1\Class-weighted BCE lossbackprop(θ only) Fig. 1: Two-stage probing pipeline (cf. Fig. 2 of [15]). Stage 1 (Extraction): each function c passes through the frozen LLM ℳM (dashed border denotes frozen weights) and its last-prefill-token activation h(c)∈ℝdh(c) ^d is cached with label y. Stage 2 (Training): an MLP probe fθf_θ is trained on the cached (hi,yi)(h_i,y_i) pairs via a class-weighted binary cross-entropy loss (§IV-E); only θ is updated – ℳM stays frozen. IV-B Models We evaluate probes trained on activations from four LLMs spanning three publishers, to test whether vulnerability-relevant representations are a general property of coding-capable LLMs rather than an artifact of a single model, and, via the two Qwen sizes, to probe within-family generalization across a scale/generation jump: Granite-4.1-8B (IBM, dense decoder-only) [12]; Qwen3.5-9B (Alibaba, hybrid gated-DeltaNet/gated-attention) [22]; Qwen3.6-27B (Alibaba, same family as Qwen3.5-9B, later generation) [23]; and Gemma-4-12B (Google DeepMind, encoder-free decoder-only) [7]. Table I lists the hidden dimension d and layer count for each model, which fixes the input width of that model’s probe (Table I). TABLE I: Target Model Specifications Model Params d (hidden dim) Layers Granite-4.1-8B 8B 4096 40 Qwen3.5-9B 9B 4096 32 Qwen3.6-27B 27B 5120 64 Gemma-4-12B 12B 3840 48 For each model, we extract the final hidden layer’s last-prefill-token activation using last-token pooling, following [15]. We probe only the final layer’s last-token representation (a single d-dimensional vector per example), not a layerwise sweep as in [11]; §VII notes this as a direction their layer-pruning results motivate for future work. IV-C Datasets We evaluate on four widely used function-level vulnerability detection datasets, all C/C++ and all binary-labeled (vulnerable / benign) at function granularity, chosen to span a range of labeling methodologies (manual security-expert review, CVE/patch mining, and static-analyzer-derived labels) and known label-quality regimes, since label noise is itself an active concern in this literature [3]: TABLE I: Dataset Summary and Reported Binary-Classification SOTA Dataset Size Vuln. % SOTA F1 Labeling Devign [32] 27K 46% 67.9% manual review Big-Vul [4] 217K 5.8% 96.5%* CVE/patch mining Draper VDISC [25] 1.27M ∼ 9.6% 60.9% static-analyzer PrimeVul [3] 236K ∼ 3% 24.5%* auto-labeling All four are C/C++, function-level, binary-labeled (vulnerable/benign). Devign [32] (HF DetectVul/devign, 27.4K rows: 21.9K/2.7K/2.7K train/val/test) draws from the Linux kernel, QEMU, Wireshark, and FFmpeg, manually labeled; relatively balanced (∼ 46%, smaller than other Devign mirrors in circulation); SOTA is 67.9% F1 (SNOPY [24]). Big-Vul [4] (HF bstee615/bigvul) links 309 GitHub projects to 2002–2019 CVEs via patch commits; imbalanced, known label noise [3]; reported SOTA is 96.5% F1 (DeepDFA [24]), but a bag-of-words classifier with no code structure reaches 86% F1 in the same study – treat this SOTA with caution. Draper VDISC [25] (HF claudios/Draper) labels via a static-analysis ensemble (Clang SA, Cppcheck, Flawfinder); 80/10/10 split; SOTA is 60.9% F1 (CodeT5 [19]). PrimeVul [3] (HF colin/PrimeVul) spans 140+ CWE types, purpose-built to fix label noise/contamination in Devign and Big-Vul via improved auto-labeling, deduplication, and chronological splits; best reported F1 when actually fine-tuned and evaluated on PrimeVul is 24.5% (UniXcoder with class-aware contrastive learning [3]), still far below the other three benchmarks. The same study reports a starker number: a 7B model fine-tuned on Big-Vul scores 68.3% F1 there but only 3.1% when that checkpoint is evaluated on PrimeVul’s test set – a cross-dataset transfer collapse, not PrimeVul’s achievable SOTA, but the starkest sign of how gameable the other three benchmarks are, and why we evaluate across all four rather than Devign/Big-Vul alone. Operationalization. As §I flags, we substitute each dataset function c for x and pool h(c)h(c) (the same forward pass, run as input context) rather than pairing a prompt with a generated completion, avoiding the post-hoc completion labeling of [29] and instead treating the LLM as a fixed feature extractor, as in LPASS [11]. The forward pass is identical whether c is a corpus function (evaluated here) or a completion the model just emitted, but we have not yet run the latter (§VII). IV-D Probe Architecture Following [15], we use an MLP with progressively decreasing hidden dimensions, GELU activations, and dropout (Table I). We depart from [15] in the output layer: rather than its softmax over two classes, ours ends in a single raw logit trained with a class-weighted binary cross-entropy loss (§IV-E), matching our binary vulnerable/benign task rather than the multi-class-style head their prompt-harm setting used. Because our four models do not share a hidden dimension d (Table I), we instantiate one probe per model with its own input width d but an otherwise identical architecture, so cross-model comparisons isolate the representation rather than probe capacity. This 6-layer architecture is inherited from [15]’s harmful-prompt task, not independently justified for code, and widths/dropout (p=0.1p=0.1) were fixed a priori rather than swept; we have not tested a linear (logistic-regression) probe on the same activations to see whether the nonlinear capacity is earned (§VII) – so “recoverable” claims about our results should not be read as evidence of linear separability specifically. TABLE I: MLP Probe Architecture (Shared Across Models; Only d Varies) Layer In Out Activation 1 d (Table I) 2048 GELU + Dropout(p=0.1p=0.1) 2 2048 2048 GELU + Dropout(p=0.1p=0.1) 3 2048 512 GELU + Dropout(p=0.1p=0.1) 4 512 512 GELU + Dropout(p=0.1p=0.1) 5 512 64 GELU + Dropout(p=0.1p=0.1) 6 64 1 (raw logit; sigmoid applied by the loss) IV-E Training Procedure Probe training is adapted from [15] to predict completion-level vulnerability rather than prompt-level harm: cache each (hi,yi)(h_i,y_i) pair (Fig. 1), then optimize fθf_θ by class-weighted binary cross-entropy, checkpointing and threshold-sweeping every epoch as detailed below. We train with AdamW [18] (η=2.5×10−4η=2.5× 10^-4, weight decay 10−210^-2, β=(0.9,0.9999)β=(0.9,0.9999), ϵ=10−8ε=10^-8), batch size 1024, for up to 50 epochs, with a ReduceLROnPlateau schedule that halves η after 2 epochs without a ≥10−2≥ 10^-2 improvement in validation loss. All four datasets are skewed toward the benign class (Table I), most severely PrimeVul (∼ 3% vulnerable); we address this with a class-weighted binary cross-entropy loss whose positive-class weight is (nneg/npos)p(n_neg/n_pos)^p on the training split (p=1p=1 full inverse-frequency reweighting by default, with p=0.5p=0.5 available as a gentler compromise), optionally combined with minority-class oversampling to a configurable target ratio (used for PrimeVul in our runs). Because reweighting shifts the probe’s output distribution, the fixed 0.5 sigmoid cutoff is no longer guaranteed to be the best decision boundary: after every epoch we sweep the threshold on validation to maximize F1 (configurable to accuracy/precision/recall) and apply the resulting fixed threshold to the test split, reporting the AUC (threshold-independent) alongside. Checkpoints are saved every epoch, and the best checkpoint is selected by validation loss by default (configurable to the validation F1 at the tuned threshold instead); §V reports precision, recall, and F1 at that checkpoint’s tuned threshold where the threshold sweep for that specific checkpoint is available, and at the untuned 0.5 cutoff otherwise (marked in Table IV). IV-F Evaluation Metrics For each (model, dataset) pair we report precision, recall, F1, and AUC on a held-out test split, following [15]. We additionally break down accuracy per CWE type for the two datasets whose schema exposes CWE labels – Draper VDISC (per-CWE boolean columns) and Big-Vul (regex extraction from a free-text field) – at a checkpoint’s tuned threshold; Devign/PrimeVul lack a schema-specific extractor. We do not yet report a cross-model/cross-dataset generalization matrix, which §VI-C and [30] motivate (§VII). IV-G Experimental Setup Hardware: 2× NVIDIA L4 and 2× NVIDIA A100 GPUs, 2 CPUs, and 1TB of storage (local SSD plus cloud). GPUs were used exclusively for extracting activations from the target LLMs; probe training and visualization (§IV-E, Table V) ran on CPU only. Splits: We use each dataset’s native HuggingFace train/validation/test split as-is (Table I); we do not apply custom stratification by label or CWE class on top of the dataset-provided split. V Results Table IV reports probe performance across all 16 (model, dataset) pairs, averaging 41.7% F1 overall. Difficulty splits by dataset, not model: Devign is easiest (62.1–68.8% F1) and roughly matches its published SOTA (67.9%, Table I) for three of four models; Big-Vul is next (44.6–59.5%) but well below its likely-inflated 96.5% SOTA (§IV); Draper VDISC and PrimeVul are both hard and strikingly model-invariant – 30.5–30.9% and 17.6–19.5% F1 respectively, despite base models differing in size by over 3×. This tightness is consistent with a hypothesis that dataset label quality and class imbalance, not which model’s activations the probe reads, set the ceiling on these two benchmarks – also consistent with PrimeVul’s own SOTA (24.5%, [3]) needing purpose-built imbalance handling, not a different architecture – but we have not run the non-LLM baseline (e.g. TF-IDF/AST) that would confirm this rather than the probes simply underperforming an achievable ceiling (§VII). No model is uniformly best across all four datasets, though several cells (marked †) use an untuned threshold, so part of this ranking may reflect threshold availability rather than a genuine model difference. TABLE IV: Probe Performance Across Models and Datasets (Precision / Recall / F1, %) Devign Big-Vul Draper VDISC PrimeVul Model P R F1 P R F1 P R F1 P R F1 Granite-4.1-8B 58.15 75.63 65.74 43.26 46.00 44.59 24.15 41.48 30.53† 27.48 15.12 19.51† Qwen3.5-9B 56.30 88.35 68.77 44.69 88.99 59.50† 22.05 50.27 30.65† 15.77 19.85 17.58 Qwen3.6-27B 58.38 70.28 63.78 43.73 52.73 47.81 23.33 45.78 30.91† 23.80 15.30 18.63 Gemma-4-12B 58.10 66.76 62.13 42.50 88.89 57.50† 22.24 48.47 30.49† 23.35 15.48 18.62† †Untuned 0.5-threshold result (validation-F1 sweep unavailable for this checkpoint); P/R are not directly comparable to undaggered cells in the same column (§IV-E). V-A Per-Vulnerability-Class Breakdown For Big-Vul, we group each test-split row by CWE type (not mutually exclusive) and report accuracy at the tuned threshold within the five most frequent types (of 85 total; the rest are too sparse), plus a residual “Benign / No CWE” bucket; Draper VDISC is supported by the same tooling but not yet run, and Devign/PrimeVul lack the needed schema. Table V: accuracy is uniformly high and tightly clustered, 97.5–99.0% across all five types and within about 1 point of each model’s Benign/No-CWE accuracy. CWE-119 (buffer-bounds violations, n=4478n=4478) is the single hardest category for all four models, and CWE-416/CWE-399 are consistently easiest. We read this uniformity cautiously: Big-Vul skews heavily benign (Table I), all four checkpoints use a high tuned threshold (0.919–0.945), and its reported SOTA is itself inflated by a bag-of-words baseline reaching 86% F1 [24] – near-ceiling accuracy here is consistent both with CWE-independent vulnerability signal and with a general vulnerability-correlated shortcut; accuracy alone cannot distinguish the two. TABLE V: Per-CWE-Type Classification Accuracy (%) on Big-Vul, Top-5 CWE Types Plus the Benign/No-CWE Bucket CWE Type (n) Granite Qwen3.5 Qwen3.6 Gemma Benign / No CWE (6424) 98.8 98.6 98.9 98.3 CWE-119 (4478) 97.9 97.5 97.7 97.7 CWE-20 (3388) 98.1 97.8 98.0 97.8 CWE-399 (2889) 98.8 98.6 98.7 98.6 CWE-264 (2287) 98.1 98.3 98.3 98.6 CWE-416 (1559) 99.0 98.8 98.8 98.3 Tuned threshold per model: Granite-4.1-8B τ=0.922τ=0.922; Qwen3.5-9B τ=0.926τ=0.926; Qwen3.6-27B τ=0.945τ=0.945; Gemma-4-12B τ=0.919τ=0.919. VI Discussion VI-A Why Probe at Generation Time: The Resource-Savings Case The case for probing over external screening is fundamentally a resource argument: [15]’s 12.6M-parameter probe is 0.16% of the 8B-parameter prompt-harm model it reads from, yet matches 7B-parameter guard models (82.7% F1 vs. WildGuard’s 84.4% on BeaverTails) and trails GPT-4-scale judges by only a few points. Two savings compound: • Parameter cost. A probe adds a negligible fraction of the base model’s parameters versus deploying a second 7B–1.8T model as an external judge. Our probes range 13.41–16.03M parameters (Table I), i.e. 0.17%–0.06% of the respective base model – the same order of magnitude as [15]’s 0.16%, and shrinking as the base model scales up, since absolute probe size barely grows with d. • Latency and infrastructure. External screening is additive (prompt filter, generation, response filter run sequentially, each potentially a separate GPU service with network round-trips); a probe reading the primary model’s own activations is concurrent and needs no separate infrastructure, bounding total latency by the slower of TllmT_llm and TprobeT_probe. Since probe parameters are a small fraction of primary-model parameters, Tprobe≪TllmT_probe T_llm in practice – [15] measured under 1 ms against 50–500 ms generation times, effectively free – and, reading prefill-time activations, can in principle flag risk before a vulnerable completion is fully emitted, unlike a response filter that must wait for it. Relevance to tactical and air-gapped deployment. Beyond raw resource savings, a same-process probe needs no round-trip to an external judge, and hence no network path for generated code – itself potentially sensitive or classified [17, 27] – to leave the inference environment. This suits on-device or air-gapped pipelines, e.g. embedded autonomous-vehicle/robot-swarm toolchains [20, 33], where an external guard model is an exfiltration risk or infeasible under tactical compute/connectivity constraints, though we have not evaluated our probes in such an environment. Table VI situates our probe’s parameter count against representative external code-vulnerability and content-safety detectors from the literature (§I): encoder-only code models fine-tuned for vulnerability classification (CodeBERT, GraphCodeBERT) and instruction-tuned LLM guard models (Llama Guard 3, Granite Guardian, WildGuard). These external sizes are literature-reported, not benchmarks we ran ourselves (§VII), so the comparison is parameter-count only. TABLE VI: Parameter Count: Probe vs. External Detection Models Model Role Params Our Probe (4 models, Table I) inline, generation-time 13.41–16.03M CodeBERT [5] fine-tuned classifier 125M GraphCodeBERT [10] fine-tuned classifier 125M Llama Guard 3-8B [9] external guard model 8B Granite Guardian 3.x-8B [21] external guard model 8B WildGuard [15] external guard model 7B GPT-4-class judge [15] LLM-as-judge ∼ 1.8T Our probes are roughly 8–9× smaller than CodeBERT/GraphCodeBERT, 437–597× smaller than the 7–8B guard models we compare against, and five orders of magnitude smaller than a GPT-4-scale judge. Unlike CodeBERT/GraphCodeBERT, which are fine-tuned end-to-end, our MLP head reads frozen activations off a model the pipeline needs anyway for generation – that distinction, not just the parameter-count gap, drives the latency/infrastructure savings above. Table IV shows this recoverability holds cleanly on Devign and less so on the three harder benchmarks; the parameter-count gap, not a measured one, is the evidence we currently have for the latency argument (§VII). VI-B Relation to Prior Probing Work §I details how our design differs from [11, 29]. Neither reports a directly comparable F1: LPASS targets pruning, and Sternfeld et al. report a completion-flip rate under prompt perturbation, not F1 on a held-out corpus. Our Devign result (68.8% F1, matching fine-tuned-classifier SOTA) is the first quantitative evidence we are aware of that a probe on unmodified activations recovers a comparable signal. VI-C Threats to Generalization Probing-based detectors are known to overfit to in-distribution artifacts: Wang et al. [30] show malicious-input probes that look accurate in-distribution fail out-of-distribution, and Fomin [6] shows standard splits can overstate generalization relative to a true distribution-shift evaluation. One narrow, encouraging signal: the per-CWE breakdown (Table V) agrees across all four models on which CWE type is hardest – shared, not per-model idiosyncratic – though this is single-dataset agreement, not the cross-dataset test [30, 6] call for. A leave-one-dataset-out probe and an obfuscation check [16] are our highest-priority follow-ups (§VII). VII Limitations • Coverage: 4 models, 3 families (Qwen3.5-9B/Qwen3.6-27B share a family); not validated outside these families (e.g., closed-weight models where activations are inaccessible). • Final layer, last-prefill-token only, not a layerwise sweep, per [11]. Binary label only; the per-CWE breakdown (§V) is a post-hoc diagnostic, not a per-CWE risk score, and runs on Big-Vul only. • Dataset caveats (Table I): Big-Vul’s spurious-correlation risk, Draper VDISC’s imperfect static-analyzer ground truth, and PrimeVul’s severe imbalance (∼ 3% vulnerable) are the main ones. All evaluation is on existing corpus functions, not a model-generated completion slice (§IV), and in-distribution results may not bound worst-case or obfuscated [16] performance [30]. • No static analyzer, fine-tuned classifier, or LLM-judge baseline run ourselves – Tables I/VI use literature-reported numbers, not a controlled head-to-head – and no measured wall-clock probe latency (§VI-A). • No architecture ablation: the 6-layer MLP is inherited from [15], not independently justified; no linear (logistic-regression) probe tested (§IV-E), nor a non-LLM baseline (e.g. TF-IDF/AST) to test whether dataset quality, not model choice, really caps performance. • No adversarial/evasion analysis, despite motivating this work by noting post-hoc detectors are evadable [16]. • Each dataset uses its own native split (§IV-G), not a shared, re-stratified one; several Table IV cells use an untuned threshold (†), so within-column rankings should be read with that caveat. VIII Conclusion We investigated whether an LLM’s activations over a piece of code already encode a signal predictive of that code’s vulnerability status. Extending the activation-probing methodology of [15] from prompt-harm to code vulnerability, we trained MLP probes on four models (Granite-4.1-8B, Qwen3.5-9B, Qwen3.6-27B, Gemma-4-12B) and four datasets, achieving 41.7% average F1 with 13.41–16.03M-parameter probes – 0.06–0.17% of base-model size. Probes match published SOTA on Devign (up to 68.8% vs. 67.9% F1) and are remarkably consistent across models on the harder, more imbalanced benchmarks; we read this as consistent with dataset quality being the binding constraint there, though confirming it needs a non-LLM baseline we have not yet run (§VII). This is early, preliminary evidence that vulnerability signal is recoverable by a lightweight probe from a frozen coding LLM’s own activations over existing code – narrower than, but a precondition for, a probe recognizing its own forthcoming output as vulnerable. A linear-probe ablation, a non-LLM baseline, a model-generated-completion pilot, an adversarial-robustness check, and a cross-model/cross-dataset matrix (§VII) are the controls this claim still needs. References [1] B. Ajayi, B. Barakat, and K. McGarry (2025) Leveraging VAE-derived latent spaces for enhanced malware detection with machine learning classifiers. arXiv preprint arXiv:2503.20803. Cited by: §I-C. [2] X. W. Chia, S. L. Wong, and J. Pan (2025) Probing latent subspaces in LLM for AI security: identifying and manipulating adversarial states. arXiv preprint arXiv:2503.09066. Cited by: §I-C. [3] Y. Ding, Y. Fu, O. Ibrahim, et al. (2025) Vulnerability detection with code language models: how far are we?. In Proc. IEEE/ACM 47th Int’l Conf. Software Engineering (ICSE), Cited by: §IV-C, §IV-C, TABLE I, §V. [4] J. Fan, Y. Li, S. Wang, and T. N. Nguyen (2020) A C/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th International Conference on Mining Software Repositories (MSR), Cited by: §IV-C, TABLE I. [5] Z. Feng, D. Guo, D. Tang, N. Duan, et al. (2020) CodeBERT: a pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 1536–1547. Cited by: TABLE VI. [6] M. Fomin (2026) When benchmarks lie: evaluating malicious prompt classifiers under true distribution shift. arXiv preprint arXiv:2602.14161. Cited by: 5th item, §I-C, §VI-C. [7] Gemma Team, Google DeepMind (2026) Gemma 4 technical report. arXiv preprint arXiv:2607.02770. Cited by: §IV-B. [8] S. Ghosh, P. Varshney, M. N. Sreedhar, et al. (2025) AEGIS2.0: a diverse AI safety dataset and risks taxonomy for alignment of LLM guardrails. In Proc. 2025 Conf. North American Chapter Assoc. Comput. Linguistics (NAACL), p. 5992–6026. External Links: Document Cited by: 1st item. [9] A. Grattafiori et al. (2024) The llama 3 herd of models. Cited by: TABLE VI. [10] D. Guo, S. Ren, S. Lu, Z. Feng, et al. (2021) GraphCodeBERT: pre-training code representations with data flow. In Int’l Conf. Learning Representations (ICLR), Cited by: TABLE VI. [11] L. Ibanez-Lissen, L. Gonzalez-Manzano, J. M. de Fuentes, and N. Anciaux (2025) LPASS: linear probes as stepping stones for vulnerability detection using compressed LLMs. arXiv preprint arXiv:2505.24451. Cited by: 1st item, §I-B, §IV-B, §IV-C, §VI-B, 2nd item. [12] IBM Granite Team (2026) Granite 4.1: IBM’s open, efficient foundation model family. Cited by: §IV-B. [13] D. Jiao, Y. Liu, Y. Yuan, et al. (2026) LLM safety from within: detecting harmful content with internal representations. In Proc. 64th Annual Meeting Assoc. Comput. Linguistics (ACL), p. 39711–39727. External Links: Document Cited by: §I. [14] A. Khatri and D. L. Chan (2026) Do all llms know when they’re being harmful? a reproducibility study of latent-space safety probes across model families. arXiv preprint arXiv:2608.08029. Cited by: §I. [15] A. Khatri, C. Prabhu, and O. Neogi (2026) Safety beyond the interface: detecting harm via latent states in large language models. In 2026 56th Annual IEEE International Conference on Dependable Systems and Networks Workshops (DSN-W), Vol. , p. 48–52. External Links: Document Cited by: 1st item, §I, §I, Fig. 1, §IV-A, §IV-B, §IV-D, §IV-E, §IV-F, 1st item, 2nd item, §VI-A, TABLE VI, TABLE VI, 5th item, §VIII. [16] X. Li, Y. Li, H. Wu, et al. (2025) A systematic study of code obfuscation against LLM-based vulnerability detection. arXiv preprint arXiv:2512.16538. Cited by: §I-A, §VI-C, 3rd item, 6th item. [17] M. Liu, J. Wang, T. Lin, et al. (2024) An empirical study of the code generation of safety-critical software using llms. Applied Sciences 14 (3). External Links: Document Cited by: §I, §VI-A. [18] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Note: arXiv:1711.05101 Cited by: §IV-E. [19] J. Norton, D. Eyers, and V. Liesaputra (2026) VulnBench: a comprehensive benchmark for transformer-based vulnerability detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. Cited by: §IV-C. [20] A. Nouri, J. Andersson, K. De Jesus Hornig, et al. (2025) On simulation-guided llm-based code generation for safe autonomous driving software. In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, p. 1097–1106. Cited by: §I, §VI-A. [21] I. Padhi, M. Nagireddy, G. Cornacchia, et al. (2024) Granite guardian. arXiv preprint arXiv:2412.07724. Cited by: TABLE VI. [22] Qwen Team, Alibaba Group (2026) Qwen3.5: small, on-device models from 0.8B to 9B parameters. Cited by: §IV-B. [23] Qwen Team (2026) Qwen3.6-27B: flagship-level coding in a 27B dense model. External Links: Link Cited by: §IV-B. [24] N. Risse, J. Liu, and M. Böhme (2025) Top score on the wrong exam: on benchmarking in ML for vulnerability detection. Proc. ACM Softw. Eng. (PACMSE), ISSTA. Note: arXiv:2408.12986 Cited by: §IV-C, §V-A. [25] R. L. Russell, L. Kim, L. H. Hamilton, et al. (2018) Automated vulnerability detection in source code using deep representation learning. arXiv preprint arXiv:1807.04320. Cited by: §IV-C, TABLE I. [26] B. Saglam et al. (2025) Large language models encode semantics and alignment in linearly separable representations. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, p. 2282–2303. Cited by: 1st item. [27] M. Sevenhuijsen, M. S. Patil, M. Nyberg, and G. Ung (2025) Generating safety-critical automotive c-programs using llms with formal verification. In Proceedings of The 19th International Conference on Neurosymbolic Learning and Reasoning, Vol. 284, p. 353–378. Cited by: §I, §VI-A. [28] B. Steenhoek, M. M. Rahman, S. Sharmin, and W. Le (2023) Do language models learn semantics of code? A case study in vulnerability detection. In arXiv preprint arXiv:2311.04109, Cited by: §I-A. [29] A. Sternfeld, L. Dolamic, and A. Kucharavy (2026) Minimal prompt perturbations lead to code vulnerabilities: prompt fragility and hidden-state signals in coding LLMs. arXiv preprint arXiv:2605.29737. Cited by: 1st item, §I, §I-B, §IV-C, §VI-B. [30] C. Wang, Z. Wei, Q. Liu, and M. Chen (2025) False sense of security: why probing-based malicious input detection fails to generalize. arXiv preprint arXiv:2509.03888. Cited by: 5th item, §I-C, §IV-F, §VI-C, 3rd item. [31] C. Zheng, F. Yin, H. Zhou, et al. (2024) On prompt-driven safeguarding for large language models. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 61593–61613. Cited by: §I. [32] Y. Zhou, S. Liu, J. Siow, et al. (2019) Devign: effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 32. Cited by: §IV-C, TABLE I. [33] W. Zhu, M. Dorigo, and M. K. Heinrich (2025) Online automatic code generation for robot swarms: llms and self-organizing hierarchy. arXiv preprint arXiv:2510.04774. Cited by: §I, §VI-A.